Month: April 2022

Article originally posted on InfoQ. Visit InfoQ

CRM software company Salesforce have revealed their approach to service reliability using service-level indicators and objectives (SLIs and SLOs). After building a platform to monitor SLOs, they saw massive adoption with 1,200 services onboarded in the first year.  The platform provides service owners with deep and actionable insights into how to improve or maintain the health of their services, to find dips in SLIs, to find dependent services that weren’t meeting their own SLOs, and overall provide a better understanding of customers’ experience with their services.

Building a platform to monitor service reliablility abstracts away organizational complexities and toil, allowing teams to focus on driving business value. Tripti Sheth talks through how it was crucial for Salesforce to agree a definition of ‘highly reliable’ across a range of tech stacks, and across the many products and individual supporting services and products within the organisation. This led to them being able to frame reliability in terms of SLIs and SLOs.

As documented by Google Cloud, Site Reliability Engineering (SRE) begins with the idea that availability is a prerequisite for success. Service-Level Objectives (SLOs) are a precise numerical target for service availability. A Service-Level Agreement (SLA) defines a promise to a service user that the SLO will be met over a specific time period, and Service-Level indicators (SLIs) are direct measurements of the service’s performance. These generally accepted definitions are often used to show customer experience in a clear, quantitative and actionable way.

In the past, Salesforce’s teams had assembled SLOs manually, meaning that updating these metrics and reporting on them was a time-consuming and error-prone task. Additionally, different teams would calculate and store these values in different ways, preventing the company from gaining a clear picture of customer experience.

Forming a standardized view of service availability was crucial, and Salesforce approached this in three areas:

Standardised Measurements: Salesforce used a previously established SLO framework based on five readings of request rate, errors, availability, duration/latency, and saturation (READS) to define standardised measurement of product and service health.

Standardised Tooling: a dedicated SLO platform for hosting the definitions of SLIs, SLOs and services, including ownership, health thresholds and alert configurations. This metadata is held in a single data store, with long-term storage and retention to give visibility of historical health trends. Automated alerts can be set up based on the data collected.

Standardised Visualisation: as soon as a new service is added to the platform, an out-of-the-box standard view of metrics is generated, with the standard READS SLIs and any custom SLIs added for that specific service. The visualisation includes a dedicated Grafana dashboard for realtime monitoring which is automatically generated and populated by real-time data. Also, the service is added to the service analytics dashboard which is regularly reviewed to drive conversations about service health and availability.

The combination of these three areas creates many benefits:

• Confidence that SLOs are calculated in a standardized way
• Insights from visualized SLI and SLO metrics
• Using granular targets on SLOs to judge if a service is meeting expectations
• Alerting on SLI and SLO metrics
• Correlation of breaches with incidents
• Identification of service dependencies

The SLO platform architecture comprises multiple components. It is centered around a service registry and configuration store – keeping service ownership information, service statuses and service-specific configuration, and data on SLIs, SLOs and the thresholds required for triggering alerting. Peripheral to this are data stores for change and release information, collected for future use in correlating changes with SLO breaches, and a time-series monitoring platform and pipelines for collecting and aggregating metrics.

The unified service health dashboard has become a focal point for operational reviews. The team have used these metrics to trigger architectural reviews, and stimulated discussions around strategic investments and tactical improvements.

Future work will enable a more comprehensive view of the dependencies for a service – with the goal of pinpointing exactly where a failure occurs and minimising recovery times. Furthermore, having collected these data per service, and with a realistic view of its dependent service, Salesforce will be able to set realistic SLIs across the entire stack.

The full article with further detail is available on Medium.

Posted on mongodb google news. Visit mongodb google news

Fiera Capital Corp lowered its position in shares of MongoDB, Inc. (NASDAQ:MDB – Get Rating) by 11.5% in the fourth quarter, according to its most recent 13F filing with the Securities & Exchange Commission. The institutional investor owned 81,627 shares of the company’s stock after selling 10,621 shares during the quarter. Fiera Capital Corp owned approximately 0.12% of MongoDB worth $43,209,000 at the end of the most recent quarter.

Other institutional investors have also recently made changes to their positions in the company. Arlington Partners LLC bought a new position in MongoDB during the 4th quarter worth approximately $30,000. Winch Advisory Services LLC raised its holdings in shares of MongoDB by 54.2% in the third quarter. Winch Advisory Services LLC now owns 74 shares of the company’s stock valued at $35,000 after purchasing an additional 26 shares during the last quarter. HBC Financial Services PLLC raised its holdings in shares of MongoDB by 3,233.3% in the fourth quarter. HBC Financial Services PLLC now owns 400 shares of the company’s stock valued at $39,000 after purchasing an additional 388 shares during the last quarter. First Horizon Advisors Inc. raised its holdings in shares of MongoDB by 194.2% in the third quarter. First Horizon Advisors Inc. now owns 203 shares of the company’s stock valued at $78,000 after purchasing an additional 134 shares during the last quarter. Finally, Lindbrook Capital LLC raised its holdings in shares of MongoDB by 56.5% in the third quarter. Lindbrook Capital LLC now owns 180 shares of the company’s stock valued at $85,000 after purchasing an additional 65 shares during the last quarter. Institutional investors and hedge funds own 89.57% of the company’s stock.

A number of equities analysts recently weighed in on the stock. UBS Group upgraded shares of MongoDB from a “neutral” rating to a “buy” rating and boosted their target price for the stock from $345.00 to $450.00 in a report on Friday, March 18th. Royal Bank of Canada assumed coverage on shares of MongoDB in a research report on Tuesday, March 1st. They issued an “outperform” rating and a $505.00 target price for the company. Barclays cut their price objective on shares of MongoDB from $556.00 to $410.00 and set an “overweight” rating for the company in a research report on Wednesday, March 9th. Stifel Nicolaus dropped their target price on shares of MongoDB from $550.00 to $425.00 in a report on Wednesday, March 9th. Finally, Zacks Investment Research downgraded shares of MongoDB from a “hold” rating to a “sell” rating in a research report on Thursday, February 3rd. One investment analyst has rated the stock with a sell rating, one has issued a hold rating and fifteen have assigned a buy rating to the stock. According to MarketBeat, the company presently has an average rating of “Buy” and a consensus target price of $496.72.

In other MongoDB news, CEO Dev Ittycheria sold 35,000 shares of the stock in a transaction that occurred on Wednesday, April 6th. The stock was sold at an average price of $412.38, for a total transaction of $14,433,300.00. Following the transaction, the chief executive officer now owns 204,744 shares in the company, valued at $84,432,330.72. The sale was disclosed in a document filed with the Securities & Exchange Commission, which is accessible through the SEC website. Also, Director Charles M. Hazard, Jr. sold 1,666 shares of the firm’s stock in a transaction that occurred on Tuesday, February 1st. The shares were sold at an average price of $406.70, for a total value of $677,562.20. The disclosure for this sale can be found here. Insiders sold 145,833 shares of company stock worth $57,329,693 in the last 90 days. Company insiders own 7.40% of the company’s stock.

Shares of MDB traded down $22.72 during mid-day trading on Friday, hitting $354.93. 801,656 shares of the company’s stock traded hands, compared to its average volume of 991,906. The company has a quick ratio of 4.02, a current ratio of 4.02 and a debt-to-equity ratio of 1.70. The business’s 50-day moving average price is $383.70 and its 200-day moving average price is $445.17. The firm has a market capitalization of $23.98 billion, a P/E ratio of -74.88 and a beta of 0.84. MongoDB, Inc. has a 1 year low of $238.01 and a 1 year high of $590.00.

MongoDB (NASDAQ:MDB – Get Rating) last posted its quarterly earnings results on Tuesday, March 8th. The company reported ($0.09) earnings per share for the quarter, beating the Thomson Reuters’ consensus estimate of ($1.26) by $1.17. The business had revenue of $266.50 million for the quarter, compared to the consensus estimate of $243.42 million. MongoDB had a negative return on equity of 66.70% and a negative net margin of 35.12%. The company’s quarterly revenue was up 55.8% compared to the same quarter last year. During the same quarter in the previous year, the business earned ($1.01) EPS. Sell-side analysts anticipate that MongoDB, Inc. will post -5.48 earnings per share for the current fiscal year.

MongoDB Profile (Get Rating)

MongoDB, Inc provides general purpose database platform worldwide. The company offers MongoDB Enterprise Advanced, a commercial database server for enterprise customers to run in the cloud, on-premise, or in a hybrid environment; MongoDB Atlas, a hosted multi-cloud database-as-a-service solution; and Community Server, a free-to-download version of its database, which includes the functionality that developers need to get started with MongoDB.

Read More

Want to see what other hedge funds are holding MDB? Visit HoldingsChannel.com to get the latest 13F filings and insider trades for MongoDB, Inc. (NASDAQ:MDB – Get Rating).

Receive News & Ratings for MongoDB Daily – Enter your email address below to receive a concise daily summary of the latest news and analysts’ ratings for MongoDB and related companies with MarketBeat.com’s FREE daily email newsletter.

Article originally posted on mongodb google news. Visit mongodb google news

Posted on mongodb google news. Visit mongodb google news

In last trading session, MongoDB Inc. (NASDAQ:MDB) saw 0.8 million shares changing hands with its beta currently measuring 0.89. Company’s recent per share price level of $354.93 trading at -$22.72 or -6.02% at ring of the bell on the day assigns it a market valuation of $26.31B. That closing price of MDB’s stock is at a discount of -66.23% from its 52-week high price of $590.00 and is indicating a premium of 32.94% from its 52-week low price of $238.01. Taking a look at company’s average trading volume of 1.24 million if we extend that period to 3-months.

For MongoDB Inc. (MDB), analysts’ consensus is at an average recommendation of a Buy while assigning it a mean rating of 1.90. Splitting up the data highlights that, out of 20 analysts covering the stock, 0 rated the stock as a Sell while 1 recommended an Overweight rating for the stock. 4 suggested the stock as a Hold whereas 15 see the stock as a Buy. 0 analyst(s) advised it as an Underweight. The company is expected to be making an EPS of -$0.22 in the current quarter.

Upright in the red during last session for losing -6.02%, in the last five days MDB remained trading in the red while hitting it’s week-highest on Friday, 04/29/22 when the stock touched $354.93 price level, adding 8.51% to its value on the day. MongoDB Inc.’s shares saw a change of -32.95% in year-to-date performance and have moved -4.34% in past 5-day. MongoDB Inc. (NASDAQ:MDB) showed a performance of -17.50% in past 30-days.

Wall Street analysts have assigned a consensus price target of $463.20 to the stock, which implies a rise of 23.37% to its current value. Analysts have been projecting $325.00 as a low price target for the stock while placing it at a high target of $650.00. It follows that stock’s current price would drop -83.13% in reaching the projected high whereas dropping to the targeted low would mean a gain of 8.43% for stock’s current value.

MongoDB Inc. (MDB) estimates and forecasts

Statistics highlight that MongoDB Inc. is scoring comparatively higher than the scores of other players of the relevant industry. The company lost -29.66% of value to its shares in past 6 months, showing an annual growth rate of 33.90% while that of industry is 5.70. Apart from that, the company came raising its revenue forecast for fiscal year 2022. The company is estimating its revenue growth to increase by 33.30% in the current quarter and calculating -13.30% decrease in the next quarter. This year revenue growth is estimated to rise 43.80% from the last financial year’s standing.

12 industry analysts have given their estimates about the company’s current quarter revenue by setting an average figure of $241.76 million for the same. And 13 analysts are in estimates of company making revenue of $253.75 million in the next quarter that will end on Apr 2022.

Weighing up company’s earnings over the past 5-year and in the next 5-year periods, we find the company posting an annual earnings growth rate of 3.70% during past 5 years.

MDB Dividends

MongoDB Inc. is more likely to be releasing its next quarterly report between March 07 and March 11 and investors are confident in the company announcing better current-quarter dividends despite the fact that it has been facing issues arising out of mounting debt.

MongoDB Inc. (NASDAQ:MDB)’s Major holders

Insiders are in possession of 7.58% of company’s total shares while institution are holding 90.05 percent of that, with stock having share float percentage of 97.44%. Investors also watch the number of corporate investors in a company very closely, which is 90.05% institutions for MongoDB Inc. that are currently holding shares of the company. Capital World Investors is the top institutional holder at MDB for having 6.8 million shares of worth $3.21 billion. And as of Sep 29, 2021, it was holding 10.19% of the company’s outstanding shares.

The second largest institutional holder is Price (T.Rowe) Associates Inc, which was holding about 6.68 million shares on Sep 29, 2021. The number of shares represents firm’s hold over 10.01% of outstanding shares, having a total worth of $3.15 billion.

On the other hand, Growth Fund Of America Inc and Smallcap World Fund are the top two Mutual Funds which own company’s shares. As of Nov 29, 2021, the former fund manager was holding 4.81 million shares of worth $2.39 billion or 7.20% of the total outstanding shares. The later fund manager was in possession of 1.91 million shares on Sep 29, 2021, making its stake of worth around $898.76 million in the company or a holder of 2.86% of company’s stock.

Article originally posted on mongodb google news. Visit mongodb google news

Article originally posted on InfoQ. Visit InfoQ

The latest release of GitHub official GUI client app for macOS and Windows, GitHub Desktop 3.0 brings new features that, while not striking on the surface, may improve collaboration and development workflow, including new notifications and improved checks UI.

GitHub recently added support to re-run failed jobs in a GitHub Action without having to re-run the whole Action. This possibility can save a lot of time with complex workflows that take a long time to execute through, says GitHub.

Our users build complex workflows that often rely on multiple jobs and dependencies. Features like job matrices can trigger multiple variant jobs with just a few lines of YAML. However, these workflows can take a while to run, so when a small part fails, it is frustrating to have to wait to run it again.

GitHub Desktop 3.0 improves on the initial implementation of this feature by making it possible to re-run a single check, instead of running all failed jobs. This feature is available under the drop-down menu associate to a PR. Checks can be re-run only when the spawning Action has completed its execution, so the re-run button is disabled while an Action is still in progress.

To improve developer workflow with checks, GitHub Desktop 3.0 also introduces a new notification when a check fails. This is useful, according to GitHub, to speed up the pull request review process while reducing the risk that your teammates start reviewing it before it is final.

Not addressing the issue promptly could result in your teammates reviewing code different than what you intended to merge, or needing to ask them for an additional approval once the checks pass again.

The new notification will lead you to a dialog with more detail about the failed checks and provides a button to re-run those checks if you suspect the problem could be not related to code. Otherwise, you can switch to the PR branch with a click and start working on the fix.

To improve team collaboration, GitHub Desktop includes new notifications for a number of key events, such as a teammate requesting a change or adding a comment. Additionally, you will be notified when a PR has been approved.

As a last remark about notifications, GitHub Desktop 3.0 introduces a new setting to only enable high-signal pull request notifications. When this is enabled, GitHub Desktop does not send a notification for every single event happening in repositories you collaborate but only for significant, PR-related events.

Finally, GitHub Desktop 3.0 also deals with an annoying warning about the default merge policy to use when pulling from a repo by defaulting to merging (as opposed to rebasing).

Posted on mongodb google news. Visit mongodb google news

MongoDB (NASDAQ:MDB – Get Rating) and CCC Intelligent Solutions (NYSE:CCCS – Get Rating) are both computer and technology companies, but which is the superior stock? We will contrast the two businesses based on the strength of their profitability, institutional ownership, risk, valuation, analyst recommendations, dividends and earnings.

Insider and Institutional Ownership

89.6% of MongoDB shares are owned by institutional investors. Comparatively, 86.2% of CCC Intelligent Solutions shares are owned by institutional investors. 7.4% of MongoDB shares are owned by insiders. Strong institutional ownership is an indication that endowments, hedge funds and large money managers believe a company is poised for long-term growth.

Analyst Recommendations

This is a breakdown of recent ratings and price targets for MongoDB and CCC Intelligent Solutions, as reported by MarketBeat.com.

MongoDB currently has a consensus price target of $492.13, suggesting a potential upside of 38.66%. CCC Intelligent Solutions has a consensus price target of $11.25, suggesting a potential upside of 21.89%. Given MongoDB’s stronger consensus rating and higher probable upside, analysts clearly believe MongoDB is more favorable than CCC Intelligent Solutions.

Earnings and Valuation

This table compares MongoDB and CCC Intelligent Solutions’ top-line revenue, earnings per share and valuation.

CCC Intelligent Solutions has lower revenue, but higher earnings than MongoDB.

Profitability

This table compares MongoDB and CCC Intelligent Solutions’ net margins, return on equity and return on assets.

Volatility and Risk

MongoDB has a beta of 0.84, indicating that its stock price is 16% less volatile than the S&P 500. Comparatively, CCC Intelligent Solutions has a beta of 0.92, indicating that its stock price is 8% less volatile than the S&P 500.

Summary

MongoDB beats CCC Intelligent Solutions on 7 of the 12 factors compared between the two stocks.

About MongoDB (Get Rating)

MongoDB, Inc. provides general purpose database platform worldwide. The company offers MongoDB Enterprise Advanced, a commercial database server for enterprise customers to run in the cloud, on-premise, or in a hybrid environment; MongoDB Atlas, a hosted multi-cloud database-as-a-service solution; and Community Server, a free-to-download version of its database, which includes the functionality that developers need to get started with MongoDB. It also provides professional services comprising consulting and training. The company was formerly known as 10gen, Inc. and changed its name to MongoDB, Inc. in August 2013. MongoDB, Inc. was incorporated in 2007 and is headquartered in New York, New York.

About CCC Intelligent Solutions (Get Rating)

CCC Intelligent Solutions Holdings Inc. provides cloud, mobile, AI, telematics, hyperscale technologies, and applications for the property and casualty insurance economy. It SaaS platform digitizes mission-critical AI-enabled workflows, facilitates commerce, and connects businesses across the insurance economy, including insurance carriers, collision repairers, parts suppliers, automotive manufactures, financial institution, and others. The company offers CCC Insurance solutions, including CCC workflow, CCC estimating, CCC total loss, CCC AI and analytics, and CCC casualty; CCC Repair solutions, such as CCC network management, CCC repair workflow, and CCC repair quality; CCC Other Ecosystem solutions, comprising CCC parts solutions, CCC automotive manufacturer solutions, CCC lender solutions, and CCC payments; and CCC International solutions. CCC Intelligent Solutions Holdings Inc. was founded in 1980 and is headquartered in Chicago, Illinois.

Receive News & Ratings for MongoDB Daily – Enter your email address below to receive a concise daily summary of the latest news and analysts’ ratings for MongoDB and related companies with MarketBeat.com’s FREE daily email newsletter.

Article originally posted on mongodb google news. Visit mongodb google news

Posted on mongodb google news. Visit mongodb google news

New Jersey, United States,- Mr Accuracy Reports published new research on Global Database as a Service (DBaaS) Provider covering micro level of analysis by competitors and key business segments (2022-2029). The Global Database as a Service (DBaaS) Provider explores comprehensive study on various segments like opportunities, size, development, innovation, sales and overall growth of major players. The research is carried out on primary and secondary statistics sources and it consists both qualitative and quantitative detailing.

Some of the Major Key players profiled in the study are IBM, Zoho Creator, Ninox, AWS, Oracle, MongoDB Atlas, Beats, Azure, Aiven, Kintone, Fusioo, Google Cloud Bigtable, SAP, DataStax, Caspio

Get PDF Sample Report + All Related Table and Graphs @: https://www.mraccuracyreports.com/report-sample/401251

Various factors are responsible for the market’s growth trajectory, which are studied at length in the report. In addition, the report lists down the restraints that are posing threat to the global Database as a Service (DBaaS) Provider market. This report is a consolidation of primary and secondary research, which provides market size, share, dynamics, and forecast for various segments and sub-segments considering the macro and micro environmental factors. It also gauges the bargaining power of suppliers and buyers, threat from new entrants and product substitute, and the degree of competition prevailing in the market.

Global Database as a Service (DBaaS) Provider Market Segmentation:

Database as a Service (DBaaS) Provider Segmentation by Type:

Cloud-based, On-premise.

Database as a Service (DBaaS) Provider Segmentation by Application:

Large Enterprises, SMEs

Key market aspects are illuminated in the report:

Executive Summary: It covers a summary of the most vital studies, the Global Database as a Service (DBaaS) Provider market increasing rate, modest circumstances, market trends, drivers and problems as well as macroscopic pointers.

Study Analysis: Covers major companies, vital market segments, the scope of the products offered in the Global Database as a Service (DBaaS) Provider market, the years measured and the study points.

Company Profile: Each Firm well-defined in this segment is screened based on a products, value, SWOT analysis, their ability and other significant features.

Manufacture by region: This Global Database as a Service (DBaaS) Provider report offers data on imports and exports, sales, production and key companies in all studied regional markets

Market Segmentation: By Geographical Analysis

The Middle East and Africa (GCC Countries and Egypt)
North America (the United States, Mexico, and Canada)
South America (Brazil etc.)
Europe (Turkey, Germany, Russia UK, Italy, France, etc.)
Asia-Pacific (Vietnam, China, Malaysia, Japan, Philippines, Korea, Thailand, India, Indonesia, and Australia)

The cost analysis of the Global Database as a Service (DBaaS) Provider Market has been performed while keeping in view manufacturing expenses, labor cost, and raw materials and their market concentration rate, suppliers, and price trend. Other factors such as Supply chain, downstream buyers, and sourcing strategy have been assessed to provide a complete and in-depth view of the market. Buyers of the report will also be exposed to a study on market positioning with factors such as target client, brand strategy, and price strategy taken into consideration.

Key questions answered in the report include:

• who are the key market players in the Database as a Service (DBaaS) Provider Market?
• Which are the major regions for dissimilar trades that are expected to eyewitness astonishing growth for the Database as a Service (DBaaS) Provider Market?
• What are the regional growth trends and the leading revenue-generating regions for the Database as a Service (DBaaS) Provider Market?
• What will be the market size and the growth rate by the end of the forecast period?
• What are the key Database as a Service (DBaaS) Provider Market trends impacting the growth of the market?
• What are the major Product Types of Database as a Service (DBaaS) Provider?
• What are the major applications of Database as a Service (DBaaS) Provider?
• Which Database as a Service (DBaaS) Provider Services technologies will top the market in next 7 years?

Please click here today to buy full report @ https://www.mraccuracyreports.com/checkout/401251

Table of Contents

Global Database as a Service (DBaaS) Provider Market Research Report 2022 – 2029

Chapter 1 Database as a Service (DBaaS) Provider Market Overview

Chapter 2 Global Economic Impact on Industry

Chapter 3 Global Market Competition by Manufacturers

Chapter 4 Global Production, Revenue (Value) by Region

Chapter 5 Global Supply (Production), Consumption, Export, Import by Regions

Chapter 6 Global Production, Revenue (Value), Price Trend by Type

Chapter 7 Global Market Analysis by Application

Chapter 8 Manufacturing Cost Analysis

Chapter 9 Industrial Chain, Sourcing Strategy and Downstream Buyers

Chapter 10 Marketing Strategy Analysis, Distributors/Traders

Chapter 11 Market Effect Factors Analysis

Chapter 12 Global Database as a Service (DBaaS) Provider Market Forecast

If you have any special requirements, please let us know and we will offer you the report as you want. you can also get individual chapter wise section or region wise report version like North America, Europe or Asia.

Article originally posted on mongodb google news. Visit mongodb google news

Article originally posted on InfoQ. Visit InfoQ

Transcript

Breck: I work in real-time systems, and have for a while. I think Astrid’s talk, talked about thinking of the electrical grid as a distributed system, and how are we going to manage all these renewable devices on the electrical grid. That’s inherently a real-time problem. Our shift to these really event-based real-time systems is out of necessity. It’s not just a fad. The same thing in the work that I do. If we think of ride sharing, or if we think of financial transactions, these things are all very real-time, event-based systems. We’ve had a lot of success in the transition to these systems. Often, we don’t hear about some of the real hard parts, especially in living with these systems, I think over time. That’s what I want to focus this panel discussion on.

Background

I’m Colin Breck. I lead the cloud platforms team at Tesla, in the Energy Division, and mainly focused on real-time services for battery charging, and discharging, solar power generation, and vehicle charging.

Bykov: I’m Sergey Bykov. Currently I work at Temporal Technologies, at a startup that is focusing on building a workflow engine for code-first workflows to automate a lot of things. Before joining Temporal, I spent like forever at Microsoft, there is group sec from servers embedded, eventually go into cloud and building cloud technologies with an angle on actors, and streaming, and highly biased towards gaming scenarios, or like event basis. Very important.

Roestenburg: My name is Raymond Roestenburg. I’m a tech lead at Lightbend. I’m a member of the Akka team, where I work on Akka serverless, Akka platform, and related technologies. I’m specialized in event-based systems, highly scalable distributed systems, and data streaming, in those areas. I’ve been working with event-based streaming for about 15 years since. I counted the other day, I’ve been using the JVM for 25 years or so. That was a quarter of a century. Also, I wrote a book on Akka, “Akka in Action.” There’s a second edition now available for early access. That’s what I do nowadays.

Batch Systems and Real-Time Systems

Breck: I think there was lots of talk about the Lambda architecture a few years ago, where we would have parallel batch systems and build time systems. There’s a lot of tension in that, especially maintaining data models and code that does aggregations or these kinds of things, in two parallel systems, is a real burden. Even say, answering certain queries across those two systems becomes a huge problem. I think there was maybe a notion that, as we move to purely event-based systems, purely real-time systems that we can do away with the batch system. In my experience, that isn’t the case. We’re still living with batch systems. Do you also see that? Do you have opinions on how batch systems and real-time systems are maybe complementary?

Roestenburg: I think they are complementary, in many cases. I think Lambda in some shape or form is still used. People obviously want to move away from the complexity of these systems. For instance, in machine learning, it makes a lot of sense to have training on your larger historical data, and then do the inference or scoring on the streaming information that passes by. That’s a very useful way for using a Lambda like architecture. There’s obviously, as we know, lots of work in trying to figure out a better architecture that is simpler to use and that removes some duplication in the code. It’s very well known, written about in the Kappa architecture, which is very often used with Kafka, there is a Delta architecture, which is something that Databricks is doing with Delta tables. They are basically making it very easy for you to do upserts on top of very large historical data. Upserting real-time information on top of existing data.

At the same time, it also depends on how you look at streaming. You can look at streaming from an in-application operation, or as a data oriented ETL pipeline. There’s quite a few different areas in which you could use streaming, of course. If we’re talking about streaming between microservices, for me, it’s quite different from the streaming case where you’re doing ETL and you’re basically purifying data, and doing ML Ops or machine learning type systems.

Bykov: I generally agree. I think these are complementary scenarios, and they come from optimizing for different things. The batch systems have always been much more efficient than granularly your per-event, that is more expensive, but the latency requirements or capabilities go way down. It can be much more contextualized. It can be more rapid than reacting, whether it is device operation, some alerts, or a financial transaction like fraud prevention. Another factor I see is that there are different kinds of tooling. For people that do Hadoop, Hive queries, or machine learning, it’s a different skill than extending queries in more real-time systems. Again, they’re complementary.

None of these technologies die. I was at Costco the other day, and I was solving some problems, I saw they still use an AS/400 for the backend. I’m sure Lambda will stay for a long time. I think Kappa may be like more what I’ve been saying recently, where there is some ingestion layer where things go through the streaming system, but then they diverge, and one fork of it goes into more real-time the other goes literally into like blob to create batch for the system. Some cross between Lambda and Kappa, I think it’s more what’s popular.

Data and Code Governance in Batch and Event-based Systems

Breck: I definitely see in our work, things like late reporting IoT devices, and doing certain types of aggregation, say like a daily rollup and these kinds of things, are just much easier in batch systems, or more reliable in batch systems, easier to iterate on in batch systems than in purely event-based systems. I think a lot of the tension there is how do you maintain data models or code that maybe you’re doing aggregations or derivations and those kinds of things? Do you have experience or have you seen people having success maybe managing data governance or code governance across systems when they’re doing one thing in an event-based system, in a real-time system, but then also bringing that maybe to their data warehouse or their batch systems?

Roestenburg: What I’ve seen so far is that that’s very often separate systems. It’s easier to do it that way. Very often the real-time processing, because of what you just said, the possibilities for aggregation are different. If you’re having batch or you can do more query like things. There’s far more state available. The moment you are doing stateful processing, it makes a lot of sense to have batch or micro-batch. For instance, Spark on top of Parquet, those kind of systems. Where on the real-time side, you keep stuff in memory, but it’s more transient. It’s more, you’re processing, you keep some information, but you can’t bring the whole world in there because then it will take too long to stay in real-time. The actual logic will be different. Even though it might seem like you’re duplicating, what’s being duplicated is very often the schemas and the data formats. There is a need to combine both the real-time and the historical data, so there’s a similarity there. I think the logic is very often quite different.

Bykov: I agree with that. In developer experience, if you make a change, speaking about schema evolution and evolution of systems, for real-time you have different constraints with rollout of the new version of your streaming system. You cannot usually go much further back and reprocess. While in the batch system, you can say, let’s go a month back, or even a year in some cases, and reprocess everything.

The other difference I saw is that real-time systems are much less concerned with data protection and governance laws. In the batch system, you have to think upfront how you’re going to delete this data, or how you’re going to prove that you handle it correctly. In real-time, they’re more ephemeral, and you have aggregates, or some products persisted, there are less objects to have.

Breck: I think this is a particularly interesting subject area, actually, you have a lot of decisions being made in real-time systems that then aren’t necessarily represented historically. You can imagine in some operational technology, where an operator or an algorithm makes a certain decision based on real-time data, but that’s actually not persisted. You can’t actually audit or evaluate how the system made its decision, I think it’s actually quite interesting.

State of the Art for Workflow Management in Event-based Systems

One of the challenges I keep seeing over again, in event-based systems is that there’s a chain of events that happen when we get an event. You can see that in Astrid’s talk as well, there’s a time component to things. It’s not just we’ve made a decision to discharge a battery, but we want it to follow this profile over time, and maybe have other triggers in the system that are dependent on it. It blows up into this, essentially workflow. There’s a workflow defined through a series of events that trigger it. Of course, there’s implications there for state management, whether that’s state of devices, or whether that’s some notion of timing and retries, or giving up on things, or event windowing, watermarks that are using streaming systems, these things. I know that’s close to your heart Sergey, and what you have been working on recently. What do you think the state of the art is for workflow management in event-based systems?

Bykov: I’m not sure I can claim something as state of the art. I think what you point into is the key question, because data needs to be processed in the context. That context is like a user or a device, some session. Within that context, you have recent history, you have timing, which is much harder to reason about in the context of millions of users or devices. This is where you trade efficiency of batching for less efficient processing, but much more easy to understand and reason about it, and evolve the processing. You have watermarks and some events that happened a certain time ago, this is where that goes into this, you have workloads that can call workflow business process, which is some set of rules that say, ok, if this happens three times, and then if time more than x, do this action. Then you can change that logic. Somebody reviewing this change will understand, ok, we just did this way. Maybe I’m biased towards more like the real-time and more contextualized processing and small scale versus batch, but that’s where I’m thinking about.

Developer Experience in Event-based Workflows

Breck: Maybe talking about the developer experience there is like, there are certain engines for managing workflows. I haven’t seen developers get that excited about using these and they often become quite constraining after a certain point. You get a POC, it is ok. Especially if you’re driving real-time services, like imagine bidding into the energy market and these kinds of things. Those workflow engines haven’t served people that well. Then I’ve also seen people that don’t even go near that stuff, and they’re essentially building their own workflow engine from scratch, and dealing with all these hard problems. How do you see that evolving? Do you think the developer experience in terms of event-based workflow will get better and we’ll be building on top of platforms that serve those needs?

Bykov: This is a hot topic, actually. I think the contention is between code-first versus declarative or data-driven definitional workflows. I think I’m squarely in the code-first camp because there, the developer writes code, and then can reason and can debug code, what’s going on, versus things that define for a simple XML, like XAML, or YAML, all those things. They look simple, like you said. In POC, we have three steps, and then add more. Then they evolve over time, and before you know, we have this huge blob of some ad hoc DSL, and it’s very difficult to figure out what went wrong. Again, developer experience, and especially dealing with failures, I think, is paramount and is often underestimated when people get into this, “I’ll just write some DSL and handle it.” Then two years later, somebody who inherited this system, they cannot manage it. It’s like, leave it as-is. My bias is squarely towards code-first, or code based, to be developer friendly.

Roestenburg: For me as well, I’ve always been in the code-first camp. I would agree with you there, Sergey. I’ve seen a lot of systems where people start using a few Kafka topics at first, and then they get more Kafka topics. Then over time, it becomes very difficult to track where everything goes. You can obviously add all kinds of logging, all kinds of tracing stuff. It’s definitely not a simple thing. Being able to easily debug and have a good developer experience is, I think, something that the future hopefully brings us.

Bykov: On the very same visit to Costco when I saw the AS/400, I ran into an old friend of mine who is working at this huge payment processing company, and he was selling exactly the solar array you were mentioning. He’s reviewing architecture of systems and services, and people ask what’s wrong. He sees this aggregate, starts with Kafka topic, and then gets into a dead-letter queue. Then there’s three layers of dead-letter queues and some ad hoc logic to deal with that. Because yes, it’s simple from the start, before you get into these real life cases.

Dealing with Complexity in Event-based Systems

Breck: I’m definitely dealing with this, you transition fully into these event-based architectures, often largely around some messaging system at the core. Each new problem you have, like wanting a dead-letter queue or something like that, you produce back to Kafka, something like that. Then you tack on one more layer. It reminds me a little bit of the third talk in the track of this microservice sprawl comes first. Then you need to figure out solutions for data governance, for federation, for solving the hard problems. Yes, I’m not sure we’ve quite got there yet with event-based systems. How do you see that evolving maybe? Say, a company who has standardized on Kafka, and it’s pretty easy to create a new service against it or create a new topic and eventually that becomes fairly intertwined. It’s hard to tell where events are going. It’s hard to change the system over time. Are there companies that are doing that well? What are the techniques they’re using?

Roestenburg: There are different things that you can do, though, eventually, you have to build up a picture of what’s going on. You can imagine a whole bunch of topics with processing parts in between, and you need to understand where things go. One of the things that you can do is in the metadata or the header information, similar to CloudEvents, for instance, you can put contextual information about what the message was, but you would still have to extract it from every topic and then build up a bigger picture. The same thing you can do with log tracing systems where you can see which components called which. This is definitely not easy to do. You have to start doing it from the beginning. This is one of the difficult issue with event-based systems, from the beginning, you have to build things in, because otherwise you can’t later on see what’s going on.

For instance, one very simple thing, if you have a processor that reads from a topic and writes to another topic, when you’re processing from that topic, and you get any corruption, basically the only option you have is to drop the message and to continue, because otherwise your system is stuck, you have a poisoned queue. The dead-letter queues that you mentioned, very often are not very useful, because it was corrupted data. How are you going to read this? It needs a lot of human intervention. You would have to go to previous versions of code, see what it was maybe. By that time your system is already a lot further. These interactions become very difficult.

Yes, I think building up more context per message and keeping this context with the message or inside the message itself in the domain of itself, or as metadata on the message, can help you to later on create a picture of what’s going on. As you can understand, the moment you lose a message, that’s something you don’t see. It’s very difficult to then see where it’d actually go. I haven’t seen any stream lineage tools or something that really can just very automatically trace back to where things began and how it fanned out, or something. I haven’t seen that yet so far.

Bykov: We see a bunch of things with customers, where we started referring to them on a queues and duct tape architecture. Where the queues by themselves are great, but this challenge of when there’s a poisoned message, or just pure failure to talk to another system, which may be down or maybe there’s some interrupt and retries, and exponential backups, backpressure. These set of concerns are pretty standard when you go from one stage to another or from one microservice to another. The challenge there is to have a simple to use, unambiguous set of tools for developers, which will be very clear what the behavior of the system would be without writing code for this duct tape. Have duct tape be part of the system, and then you can reason about what’s going on. You can trace what’s going on by looking at standardized, unified way of, here’s the progress of this processing, within context. I think context comes first.

Visibility into State Management & Workflow Progress for Millions of Instances

Breck: Maybe back to that context, let’s maybe think of an event-based workflow that has lots of this context, maybe applied to an IoT scenario, when you have millions of them. How do we get visibility into state management, workflow progress, those kinds of things, when that is defined not as a single instance, but say millions of them?

Bykov: It depends on the scenario. In some scenarios, like in IoT, there may be transient state that it’s ok because there’ll be subsequent update, maybe like a minute later, that will invalidate the previous state anyway. It’s not important to alert in this case. Or like telemetry, another example, where there may be some hiccup in telemetry, some metrics, and generally at high volume it’s ok because they keep being reissued. There are different scenarios like payment transactions, where it’s absolutely unacceptable to handle them, then you need to alert. Then you need to stop processing, and maybe even like page a developer or an engineer, to go figure out.

Using CloudEvents to Solve Metadata Problems

Breck: Are you seeing anybody use CloudEvents to solve the metadata problems you’re just talking about of traceability of events, or correlating events to certain actions?

Bykov: I don’t. Maybe people do that, but I have not myself seen that.

Roestenburg: Yes, we are using CloudEvents in Akka serverless, for instance, and there is some additional information that we keep, but it’s not necessarily meant, at this point, in any case, to extend for this information. You could use Kafka’s normal headers as well. It’s not necessarily CloudEvents, you could use anything. Although it is handy to have a particular format that you keep for all your stuff, whatever you’re doing, so you can very easily inspect, ok, if I put everything in CloudEvents, I can very easily see at least some of these metrics, some of these headers. The other thing you could obviously do is, in your stateful processing, keep information about what’s going on and then provide that in telemetry again. It’s very case by case basis, potential in what you want to do.

Bykov: I think Telemetry is key. If you emit enough telemetry, then you can separate systemic issues versus some one-off things, and then you can decide where you want to invest time.

Breck: That’s been my experience as I think CloudEvents sounds very interesting, but I haven’t seen much adoption of it. I’ve seen more what you’re mentioning, Ray, with using Kafka headers, or propagating request IDs, even all the way through Kafka. You can even tie a gRPC, or an HTTP request to an event that ended up in Kafka that got processed later on to do something.

Roestenburg: You could do it inside your messages or outside as a header, it depends. The nice thing about if you put on the headers is that they are always readable so you wouldn’t get a corrupt header, or something really horrible would happen to Kafka maybe. That’s the benefit of using these envelopes.

Breck: I’ve seen people wanting to do transactional things, actually using the Kafka offsets and propagating those as metadata. Especially when you run an event to another topic, to another topic, because if you maintain the original offset, then you can actually do some transactional things against that, which is interesting.

Roestenburg: We’ve done that but mostly from the topic where you’re producing or where you’re processing the topic, that’s where you do is in your writes to the output, so when you write to any output, even to a database. In a transaction, you write to offset where you left off, so in that way you can do effectively once processing within that service. It’s very hard. I don’t know how to do that over many services. One of those hops is fine, and so you can build these services that can die at any time, continue where they left off. You might have to deal with some duplication. In the case where you write the offsets to the output where you’re writing, if that has a transactional model, then you can be effectively once basically, for processing.

Bykov: Same here, I’ve seen it within the scope of a service, but crossing service boundaries are usually different mechanism.

Developer Experience and Ecosystems

Breck: We’re going to talk a bit about developer experience and ecosystems. A lot of event-based systems, they have their own libraries, or they have their own developer experience. I think of Kafka and Kafka streams and the systems people build around that. Most of my experience in the last five years has been using Akka Streams, even around Kafka, because I can interface with all sorts of different systems, and I’m not just tied into one ecosystem. Do you think that’s a model that will continue to evolve in libraries in lots of different languages, or do you think there are certain advantages of having ecosystem specific libraries? Do you have opinions there? What are you seeing develop?

Bykov: My opinion here is that what’s most important is establishing patterns, and Kafka did, it established. It’s essentially the partition log. It’s not even the queuing system. For example, when I had to explain years ago, what Azure Event Hubs is, it’s essentially hosted Kafka. It’s the same thing, like the Kinesis. Establishing this pattern, and then people can use different services like that in the same manner. Then switching from library [inaudible 00:29:46] is speaking the same language. In terms of abstractions, I think it’s easy. Trying to unify it into one layer that hides everything, I think that’s dangerous. I think we made that mistake early in streaming, back in the days when we provided a single interface for queuing systems with very different semantics, like log-based versus SQS as your queue versus even in-memory like TCP, direct messages. I think that’s a dangerous thing to try to hide that.

Roestenburg: I definitely agree with that. I think it’s very good to take advantage of these very specific libraries. Streaming is not streaming, there are so many different things you can do. You can do what you said with Akka Streams, which obviously I like. It’s a particular kind of in-application streaming that you do there. The moment you want to do data oriented streaming, it’s a whole different thing. For instance, Akka Streams doesn’t have things like windowing, watermarks. It’s possible to build those things, but there’s far better support in Spark, or Flink, which are completely different models, so you basically need specific libraries. I don’t think it’s a place inside a programming language. That’s a danger as well to expect that of a programming language. It’s very specific needs, if you look further than just memory constraints, processing of data.

Apache Beam

Breck: Are you seeing any use of Apache Beam as the model for solving some of those problems?

Roestenburg: I sadly have not. I’ve always been hoping to see Apache Beam being used more. I do see things like Flink. The actual standards of Apache Beam being used, is not something I actually know. I was expecting it. That’s a very large standard for these things.

The Evolution of Developer Experience for Building Event-based Systems

Breck: Let’s talk maybe about the evolution of these systems. I think a lot of events-first systems, streaming systems, people have built those. I have lots of experience with actors over the last many years. I know both of you have a lot of experience with actor-like. I know Sergey, you wrote an article saying it’s such a loaded term, it’s hard to use sometimes. Most of what I build, we run it ourselves. We’re building these distributed, stateful actor systems that model entities, and we’re building it ourselves. It seems like, more recently, we’re starting to see this evolution on top of Cloud platforms with Azure Durable Functions, or the Cloudflare Durable Objects. It seems like this location transparent, stateful actor model is almost becoming the way of programming in the cloud in the future. Do you agree? What do you think that evolution is in terms of developer experience for building event-based systems?

Bykov: That’s been my belief that it would come to that serverless hosted, cloud hosted systems based on the actor or context oriented computations. I think it’s inevitable because there’s this data gravity. Yes, you can deploy and run your own replicated databases in the whole estate, but most people don’t want to. They want to outsource this problem. When it comes to hosting your compute, of course, we can all run on Kubernetes clusters. Do we want to, becomes the question if it’s cost effective to run it in the cloud provider. Then before you know, you outsource most of this platform, you can start to focus on the actual application. I think that’s inevitable for a lot of scenarios, if not for most, that would be much easier, more economical to just run it in the cloud.

Roestenburg: In all cases, you want to focus on the business. A lot of these systems, which are wonderful, they are also very hard to run. Kafka is quite notorious in this respect. You can get these distributed logs, Pulsar, Kafka, you can get them run for you. Nowadays even with infinite storage, tiered storage, those things are very hard to set up yourself. Even if you would be able to manage a Kafka cluster, how are you going to set up all of this extra stuff that you need to not run out of space? A similar thing you can see with Google Spanner where there’s also an infinite amount of storage. There, I think it’s very interesting that people love transactions in the sense of simplicity. That comes back as well in what Databricks is doing with the Delta Lake stuff. I think people are moving towards, ok, it’s very interesting, all these technologies, but I want to focus on a simple model, and have a lot of the Ops taken care of for me, instead of having to deal with all these complexities.

Bykov: One other thing I forgot to mention is compliance. If you hosted yourself, getting SOC 2 compliance, or other compliance is just a pain. Yes, if it’s outsourced, then you just present their SOC 2 documents and you’re good.

Roestenburg: Even security to some extent will be better arranged than what you are going to do yourself. Although there’s still obviously always a big open problem there.

The Saga Pattern

Breck: Maybe back to workflows. I hope most people are familiar with the Saga pattern, the notion of compensating transactions and taking care of things when they fail. Is that actually being successfully applied? My impression is that most people that try this, learn that it’s actually really difficult to apply the Saga pattern correctly. That there is really no such thing as rolling back the world. You can compensate in certain ways sometimes, but you can’t go backwards.

Bykov: I’m not sure I agree with you actually. Yes, it’s compensation. You’re not rolling the clock back, like in this classic example of reservations, yes, you can cancel through reservations if one of them failed or got declined. I think there is nothing wrong with Saga as a pattern. I think Saga is a restricted case of a general workflow. It’s like what do you want to do if this step in your business logic fails. Saga just says, you unwind what you allocated before. In a more general case, you might want to do something else. I don’t see that as necessarily special, separate case that’s different from a normal workflow.

Breck: What I’ve seen is depending on the state that you arrive in you can’t actually compensate. The reason you can’t move forward is also the reason you can’t compensate. It’s almost like head-of-line blocking again, you’re stuck.

Bykov: That’s fine. Because in this case, like in a general workflow logic, you need to have a step or a case for notifying. Saying, this thing needs to be manually handled, or call manager, or something. It’s a more general escape hatch for something that did not progress automatically on the predefined simple logic.

Roestenburg: I think one of the issues is that you really have to keep it in the business domain, in the logic of the business. Modeling things as Sagas purely in the business domain makes sense, because you can have these different scenarios, you compensate or you continue. The really tricky part is, if you might be confused with the fact that you could use Sagas for intermittent errors or something, or technical errors. The service wasn’t there, and then you might have already called it. You might have already reserved a table, you just didn’t get a response back. On the business level, where you have more context, you can then say, this workflow ends because something went wrong and I’m just going to get someone to look at it. If you try to solve technical problems with the Saga, then you’re going to be in a big problem.

See more presentations with transcripts

Article originally posted on InfoQ. Visit InfoQ

Transcript

Chopra: My name is Tejas Chopra. I’ll be talking about Netflix Drive. I work on infrastructure solutions, specifically software that deals with billions of assets and exabyte scale data that is generated and managed by Netflix studios and platforms.

Outline

We’ll go over a brief explanation of what Netflix Drive is. Some of the motivations for creating a software such as Netflix Drive. The design and the architecture, lifecycle of a typical instance of Netflix Drive, and some learnings from this process.

What Is Netflix Drive?

You can think of Netflix Drive as an edge software that runs on studio artists’ workstations. It’s a multi-interface, multi-OS cloud file system, and it is intended to provide the look and feel of a typical POSIX file system. In addition to that, it also behaves like a microservice in that it has REST endpoints. It has backend actions that are leveraged by a lot of the workflows and automated use cases where users and applications are not directly dealing with files and folders. Both the interfaces, the REST endpoint as well as the POSIX interface can be leveraged together for a Netflix Drive instance. They are not mutually exclusive. The other main aspect of Netflix Drive is that it is a generic framework. We intended it to be generic so that there can be different types of data and metadata stores that can be plugged into the Netflix Drive framework. You could imagine Netflix Drive working on the cloud datastores and metadata stores, as well as hybrid datastores and metadata stores. An example could be, you could have Netflix Drive with DynamoDB as the metadata store backend, and S3 as a datastore backend. You could also have MongoDB and Ceph Storage as the backend datastores and metadata stores for Netflix Drive. It has event alerting backends also configured to be a part of the framework, and that eventing, alerting is a first-class citizen in Netflix Drive.

Motivations

Let us get into what are some of the motivations of having Netflix Drive. Netflix is in general pioneering the idea of studio in a cloud. The idea is to give artists the ability to work from different corners of the world, on creating stories and assets to entertain the world. In order to do so, the platform layer needs to provide a distributed, scalable, and performant infrastructure. At Netflix, assets, which you can think of are a collection of files and folders, have data and metadata that are stored and managed by disparate systems and services. Starting at the point of ingestion, where data is produced right out of the camera, till the point where the data eventually makes its way to movies and shows, these assets get tagged with a variety of metadata by different systems based on the workflow of the creative process. At the edge, where artists work with assets, the artists’ application and the artists themselves expect a file and folder interface so that there can be a seamless access to these assets. We wanted to make working with studio applications a seamless experience for our artists. This is not just restricted to artists, and it can actually extend to more than just the studio use case as well. A great example is all the asset transformations that happen during the rendering of content for which Netflix Drive is being used today.

The other thing is that studio workflows have a need to move assets across various stages of the creative iterations. At each stage, a different set of metadata gets tagged with an asset. We needed a system that could provide the ability to add and support attaching different forms of metadata with the data. Along with that, there is also a level of dynamic access control, which can change per stage, which projects only a certain section of assets to the applications, users, or workflows. Looking at all of these considerations, we came up with the design of Netflix Drive, which can be leveraged in multiple scenarios. It can be used as a simple POSIX file system that can store the data on cloud and retrieve data from cloud, but also has a much richer control interface. It is a foundational piece of storage infrastructure to support a lot of Netflix studios and platforms’ needs.

Netflix Drive Architecture

Let us dive a bit into the architecture of Netflix Drive. Netflix Drive actually has multiple types of interfaces. The POSIX interface just allows simple file system operations, such as creating a file, deleting a file, opening a file, renames, moving, close, all of that. The other interface is the API interface. It provides control interface and a controlled IO interface. We also have events and telemetry as a first-class citizen of the Netflix Drive architecture. The idea is that different types of event backends can be plugged into the Netflix Drive framework. A great example of where this may be used is audit logs that keep a track of all the actions that have been performed on a file or a set of files by different users. We’ve also abstracted out the data transfer layer. This layer abstracts the movement of data from the different types of interfaces that are trying to move the data. It deals with bringing the files into a Netflix Drive mount point on an artist workstation or machine, and pushing files to the cloud as well.

Getting a bit deeper into the POSIX interface, it deals with the data and metadata operations on Netflix Drive. All the files that are stored in Netflix Drive, get read, write, create and other requests from different applications, users, or there could be separate scripts and workflows that do these operations. This is similar to any live file system that you use.

The API interface is of particular interest to a lot of workflow management tools or agents. This exposes some form of control operations on Netflix Drive. The idea is that a lot of these workflows that are used in studio actually have some notion and awareness of assets or files. They want to control the projection of these assets on the namespace. A simple example is, when Netflix Drive starts up on a user’s machine, the workflow tools will only allow a subset of the large corpus of data to be available to the user to view initially. That is managed by these APIs. They are also available to perform dynamic operations, such as uploading a particular file to cloud, or downloading a specific set of assets dynamically and showing them up, and attaching them at specified points in the namespace.

Events have telemetry information. You could have a situation where you want audit logs, metrics, and updates to all be consumed by services that run in cloud. Making it a generic framework allows different types of event backends to get plugged into the Netflix Drive ecosystem.

Data transfer layer is an abstraction that deals with transferring data out of Netflix Drive to multiple tiers of storage. Netflix Drive does not deal with sending the data in line to cloud. This is because of performance reasons. The expectation is that Netflix Drive will perform as close as possible to a local file system. What we do is we leverage local storage, if available, to store the files. Then we have strategies to move the data from the local storage to cloud. There are two typical ways in which our data is moved to cloud. First is dynamically issuing APIs that are done by the control interface to allow workflows to move a subset of the assets to cloud. The other is auto-sync, which is an ability to automatically sync all the files that are there in local storage to cloud. You can think of this the same way that Google Drive tries to store your file to cloud. Here, we have different types of tiers in cloud storage as well. We have peculiarly called out media cache and Baggins here. Media cache is a region-aware caching tier that brings data closer to the edge at Netflix, and Baggins is our layer on top of S3 that deals with chunking and encrypting content.

Overall, picture of Netflix Drive architecture looks as follows. You have the POSIX interface that has data and metadata operations. The API interface that deals with different types of control operations. The event interface that tracks all the state change updates. In fact, events is also used to build on top of Netflix Drive. Notions of having shared files and folders can be built using this event interface. Then, finally, we have the data transfer interface that abstracts moving the bits in and out of Netflix Drive to cloud.

Netflix Drive Anatomy

Let us now discuss some of the underpinnings of the design that go back to the motivation by discussing the anatomy of Netflix Drive. Here are some terminology that we’ll use. CDrive is a studio asset-aware metadata store that is used in Netflix. Baggins is Netflix’s S3 datastore layer that deals with chunking content and encrypting content before pushing it to S3. Media cache is an S3 region-aware caching tier whose intention is to bring the data closer to the applications and users. Intrepid is an internally developed high leverage transport protocol used by a bunch of Netflix applications and services to transfer data from one service to another.

This is a picture of the Netflix Drive interface, or Netflix Drive in general. We have the interface layer, which is the top layer, and this has all the FUSE file handlers alongside the REST endpoints. The middle layer is the storage backend layer. One thing to note is that Netflix Drive provides a framework where you can plug and play different types of storage backends. Here we have the abstract metadata interface and the abstract data interface. In our first iteration, we have used CDrive as our metadata store, and Baggins and S3 as our datastore. Finally, we have the Intrepid layer, which is the transport layer that transfers the bits from and to Netflix Drive. One thing to note is that Intrepid is not just used to transport the data, but here it is also used to transfer some aspects of the metadata store as well. This is needed to save some state of the metadata store on cloud.

To look at it in another way, we have the abstraction layers in Netflix Drive, so you have the libfuse, because we are using a FUSE based file system, that handles the different types of file system operations. You initially start the Netflix Drive and bootstrap it with a manifest. You have your REST APIs and control interface as well. Your abstraction layer abstracts the default metadata stores and the datastores. You can have different types of data and metadata stores here. In this particular example, we have the CockroachDB adapter as the metadata store, and an S3 adapter as the datastore. We can also use different types of transfer protocols, and they are also a plug and play interface in Netflix Drive. The protocol layer that is used can be REST or gRPC. Finally, you have the actual storage of data.

This here shows the different services and how they are split between workstation and cloud. You have the typical Netflix Drive API and POSIX interface on the workstation machine that sends the bits and bytes to the transport agent and library. You have a bunch of services on cloud as well. Namely, they’re your metadata store, which is CDrive in our case. You have a media cache, which is a middle caching tier of storage. You finally have object storage in S3. Netflix Drive on your local workstation will talk to the metadata store and the datastore using the transport agent and the library.

One thing to note here is that we also use local storage to cache the read and the write, and to absorb a lot of the performance that the users expect out of Netflix Drive. Security is a first-class citizen in Netflix Drive. We wanted to provide two-factor authentication on Netflix Drive. The reason is that a bunch of these cloud services are actually used by a lot of applications, they front all of the corpus of assets in Netflix. It is essential to make these assets secure, and to only allow users that have proper permissioning to view the subset of assets that they are allowed to use and view.

Typical Lifecycle

Let us discuss a typical lifecycle of Netflix Drive and some runtime aspects of it. Given the ability of Netflix Drive to dynamically present namespaces and bring together disparate datastores and metadata stores, it is essential to discuss the lifecycle. This may not be true in typical file systems where you do not necessarily have a typical stream of events that happen in the life cycle. In Netflix Drive’s case, we initially bootstrap the Netflix Drive using a manifest. An initial manifest could be an empty manifest as well. You have the ability to allow workstations or workflows to download some assets from cloud and preload and hydrate the Netflix Drive mount point with this content. The workflows and the artists would then modify these assets. They will periodically either snapshot using explicit APIs, or leveraging the auto-sync feature of Netflix Drive and upload these assets back to cloud. This is how a typical Netflix Drive instance will run.

Let us get into the bootstrapping part of it. Netflix Drive, typically during the bootstrap process expects a mount point to be specified. Some amount of user identity, for authentication and authorization. Location of the local storage, where the files will be cached. The endpoints. The metadata store endpoint and the datastore endpoint. Optional fields for preloading content. Also, persona. Netflix Drive is envisioned to be used by different types of applications and workflows. Persona gives Netflix Drive its flavor when working for applications. For example, a particular application may specifically rely on the REST control interface because they are aware of the assets, and so they will explicitly use APIs to upload files to cloud. Some other application may not necessarily be aware of when they want to upload the files to cloud, so they would rely on the auto-sync feature of Netflix Drive to upload files in the background to cloud. That is defined by the persona of Netflix Drive.

Here is a sample bootstrap manifest. A particular Netflix Drive mount point can have several Netflix Drive instances that are separate from each other. You have a local file store, which is the local storage used by Netflix Drive to cache the files. The instances get manifested under the mount. In this case, we have two separate instances, a dynamic instance and a user instance, with different backend datastores and metadata stores. In the first instance, which is a dynamic instance, you have a Redis metadata store, and an S3 datastore. You will also uniquely identify a workspace for data persistence. Then in the second one, you have CockroachDB as a metadata store and Ceph as a datastore, again.

Namespace of Netflix Drive is all the files that are viewed inside Netflix Drive. There are two options to actually create namespace. Netflix Drive can create the namespace statically at bootstrap time, where you can specify the exact files and folders that you need to pre-download and hydrate your current instance with. For this, you present a file session and a Netflix Drive container information. You have workflows that can prepopulate your Netflix Drive mount point with some files, so that the subsequent workflows can then be built on top of it. The other way to hydrate a namespace is to explicitly call Netflix Drive APIs in the REST interface. In this case, we use the stage API to stage the files and pull them from cloud, and attach them to specific locations in our namespace. One thing to note is that both these interfaces are not mutually exclusive.

Update

Let us now get into some of the Netflix Drive operations. Modifications for Netflix Drive content can happen through POSIX interface or REST APIs. File system POSIX operations that can modify a file would be open, rename, move, read, write, and close. There could also be a subset of REST APIs that can be used to modify a file. For example, staging a file, which pulls the file from cloud. Checkpointing a file. Saving a file, which actually explicitly uploads the file to cloud. An example of how a file is uploaded to cloud is using the publish API. We have the ability to autosave files, which would periodically checkpoint the files to cloud and the ability to also have an explicit save. The explicit safe would be an API that is invoked by different types of workflows to publish content.

A great example of seeing where these different APIs can be used is the case where artists are working on a lot of ephemeral data. A lot of this data does not have to make to cloud because it’s a work in progress. In that case, for those workflows, explicit save is the right call. Because once they are sure of the data, and they want to publish it to cloud to be used by subsequent artists or in subsequent workflows, that’s when they would invoke this API. That would pick the files. It will snapshot the files in Netflix Drive mount point, and then pick them and deliver it to cloud, and store it in cloud under the appropriate namespace. That is where you can see a difference between autosaving, which is like a Google Drive way of saving files, and an explicit save that is called by artists or workflows.

Learnings

Given that Netflix Drive is used in multiple personas by different types of workflows, here are some of the learnings that we had while developing Netflix Drive. The number one learning was that there were several points of making different choices for our architecture. We intended it to be a generic framework that can have any datastore and metadata store plugged into it. Also, a lot of our architectural choices were dictated by the performance and the latency aspects of files, workflows, and artists’ workstation, and artists’ experience that we wanted to provide using Netflix Drive. A great example is we used FUSE based file system. We implemented a lot of the code of Netflix Drive using C++. We compared other languages, and we thought that C++ gave us the best performance results when compared to other languages, and performance was a critical feature of Netflix Drive that we wanted to provide.

The other is designing a generic framework for several operating systems is very difficult. In our case, we support Netflix Drive on CentOS, OS X, and Windows. We leverage FUSE file system. We had to then investigate a lot of the alternatives for FUSE based file systems on these different operating systems. That also multiplied our testing matrix, our supportability matrix.

The third learning is that the key to scalability is handling metadata. In our case, because we work with disparate backends, and we have different layers of caching and tiering, we actually rely heavily on metadata operations being cached in Netflix Drive. That gives us a great performance for a lot of the studio applications and workflows that are very metadata heavy. Having multiple tiers of storage can definitely provide performance benefits. When we designed Netflix Drive, we did not restrict ourselves to just the local storage or cloud storage, we in fact wanted it to be built in a way that different tiers of storage can easily leverage Netflix Drive framework and be added as a backend for Netflix Drive. That came through in our design, in our architecture, and in our code.

Having a stacked approach to software architecture was very critical for Netflix Drive. A great example is again, the idea of shared namespaces. We are currently working on the ability to share files between different workstations or between different artists. This is built on top of our eventing framework, which is again, part of the Netflix Drive architecture itself. When one Netflix Drive has a file that is added to the namespace, it generates an event, which is consumed by different cloud services, and is also then using the REST interface of the subsequent Netflix Drive to inject that file into the Netflix Drive instances namespace. This is how you can build on top of existing primitives of Netflix Drive.

Resources

If you would like to learn more about Netflix Drive, we have a tech blog available on the Netflix tech blog channel.

Questions and Answers

Watson: Being an application you built natively on the cloud, what was your biggest challenge around scalability? I assume everything wasn’t just flawless day one. Did you have to make some technical tradeoffs to achieve the scale you wanted given how many assets Netflix handles?

Chopra: We are targeting Netflix Drive to serve exabytes of data and billions of assets. Designing for scalability was one of the cornerstones of the architecture itself. When we think in terms of scaling a solution on cloud, oftentimes, we think that the bottleneck would be the datastore. Actually, it is the metadata store that becomes the bottleneck. We focused a lot on the metadata management, how we could reduce the amount of calls that are done to metadata stores. Caching a lot of that data on the Netflix Drive locally, was something that gave us great performance.

The other thing is, in terms of the datastore itself, we explored having file systems in the cloud, like EFS. With file systems, you cannot scale beyond a point, it impacts your performance. If you really want to serve billions of assets, you need to use some form of an object store and not a file store. That meant that our files and folders which our artists are used to had to be translated into objects. The simplest thing to do is have a one is to one mapping between every file and an object. That is very simplistic, because sometimes file sizes may be bigger than the maximum object size supported. You really want the ability to have a file mapped to multiple objects, and therein lies the essence of deduplication as well. Because if you change a pixel in the file, you will then only change the object that has that chunk of the file. Building that translation layer was a tradeoff and was something that we did for scalability. These are examples of how we designed for the cloud, thinking about scalability challenges there.

Watson: How would you compare Netflix Drive with other cloud storage drive systems like Google Drive?

Chopra: That was one of the first things that we were asked when we were designing Netflix Drive. We compared the performance of Google Drive with Netflix Drive. The advantage that Netflix Drive has is because we have a tiered architecture of storage, where we leverage our media cache. Media cache is nothing but a media store, a caching layer that is closer to the user and the application. We also cache a lot of data on the local file store. Google Drive doesn’t do that. We could always get local file system performance compared to Google Drive. If you remove the local file system from the picture, even then, because we have our media caches that are spread across the globe, we can still perform much better than Google Drive.

Watson: It probably doesn’t help to log your data in S3 as well, I don’t know if Google Drive does well.

What prompted you to invent Netflix Drive over, for example, using the AWS File Storage Gateway?

Chopra: One of our cornerstones for thinking about Netflix Drive is security. We wanted to design a system that could provide our artists that are globally distributed, a secure way of accessing only the data that is relevant to them. We investigated AWS File Storage Gateway, but the performance and the security aspects of it did not match our requirements. The second thing is that the file Storage Gateway, I don’t think, when we investigated it, could translate it into objects on the backend. For us, we really wanted the power to have these objects closer to the user in our media caches, and control where these objects are stored. A great example is the case where, let’s say multiple artists are working on an asset. These assets, if every iteration of it is stored in the cloud, your cloud costs will explode. What we wanted to do was enable these assets to be stored in media caches, which is something that we own, and only the final copy goes to the cloud. That way, we can leverage a hybrid infrastructure, control what gets pushed to the cloud versus what stays locally, or in shared storage. Those are parameters that were not provided by File Storage Gateway to us.

Watson: We have similar challenges at Pinterest, where people will say, why don’t you use EKS? Ninety percent of the work we do is integrating our paths around the cloud offering. It’s like you still have to do all that work with security.

How many requests can your system handle in an hour? Do you have any sense of like rough metrics?

Chopra: I ran an Fio to compare Netflix Drive with other alternatives, and we perform much better than Google Drive. Google Drive, I think, has a lot of limits on the number of folders, the number of files that you can put in, Netflix Drive has no such limits.

Would EFS work as a substitute to some extent for Netflix Drive?

EFS could have worked as a substitute, but EFS will not scale beyond a point. Because, in general, if you have to design scalable systems, you have to pick object stores over file stores. Object stores give you more scalability, and you can also cache pieces of objects. Also, deduplication is a challenge because if you change a pixel of a file, then you have to probably have that synced to EFS, and have it stored as a file in multiple tiers of caches. That is not something that we felt was giving us the performance that we needed, so we went with object stores.

Why was CockroachDB selected as a database? Was there a specific use case for it? Did you guys compare it with other options?

We actually have a lot of databases that are built inside Netflix, and we have layers on top of different databases. Cockroach database was the first one that we picked because it gave us a seamless interface to interact with it. We wanted to leverage something that was built in-house, and it is scalable horizontally. It’s a SQL database that is horizontally scalable. That’s the reason why we went with that. Also, you could deploy it on-premise or on cloud. If we ever wanted to go into a hybrid situation, we always had the option of doing that. That being said, CockroachDB was the first one that we picked, but our code is written in a way that we could have picked any one. We also built a lot of security primitives, event mechanisms, all of that for CockroachDB. We wanted to just leverage what is already built, so we went with CockroachDB.

Watson: Having been at Netflix myself for quite a few years, I know you have so many partners there. Have your partners in the industry adopted use of the drive? I assume you have to work with different publishing houses and producers.

Chopra: At this point we are working with artists that are in Netflix, and they have adopted Netflix Drive. We haven’t reached out to the publishers as yet. The independent artists work with our tools that actually would be supported by Netflix Drive. Those tools will be a part of Netflix Drive.

You’ve described how big objects are split, and every small bit of it is referenced in a file, and then, that there was a challenge deduplicating, how is that different from symlinking?

In object stores, you have versioning. Every time you change, or mutate a small part of an object, you create a new version of the object. If you imagine a big file, but only a small pixel of it gets changed, that means that in a traditional sense, if your file was mapped to an object, you would have to write the entire file again as an object, or send the entire bits again. You cannot just send the delta and apply that delta on cloud stores. By chunking, you actually reduce the size of the object that you have to send over to the cloud. Choosing the appropriate chunk size is more of an art than a science, because now if you have many smaller chunks, you will just have to manage a lot of data and a lot of translation logic, and your metadata will just grow. The other aspect is also encryption, because today we encrypt at a chunk granularity. If you have many chunks, you will have to then have so many encryption keys, and manage the metadata for that. That is how it is challenging to deduplicate content also, in such scenarios.

Do you use copy-on-write techniques in Netflix Drive to optimize your storage footprint?

At this point, we do not use that. We could definitely explore using copy-on-write to optimize the storage footprint. Yes, that’s a good suggestion.

What is the usual chunk size?

Chunk size, it depends. I think if I remember correctly, it’s in megabytes, like 64 megabytes is what I remember last seeing. It is configurable. Configurable in the sense that it is still statically assigned at the start, but we can change that number to suit our performance needs.

Watson: How are security controls managed across regions, it’s usually an external API or built into the drive?

Chopra: It’s actually built into our CRDB, as a layer on top of CockroachDB that we have. Netflix Drive leverages that layer to handle all the security primitives. There are several security services that are built within Netflix, so we leverage those at this point. We don’t have external APIs that we can plug in to at this point. We plan to abstract them out as well, whenever we release our open source version so that anyone can build those pluggable modules as well.

Watson: You mentioned open sourcing this, and I know open sourcing is always a hurdle and takes a lot of effort. Do you have a general sense as to when this would be open source?

Chopra: We are working towards it for next year. We hope to have it open source next year because we have a lot of other folks that are trying to build studios in the cloud, reach out to us. They want to use Netflix Drive as well, the open source version of it, and build pluggable modules for their use cases. We do intend to prioritize this and get it out next year.

C++ for performance reasons. Have you considered using Rust?

We did not consider using Rust because I think one of the reasons was that a lot of FUSE file system support was not at that time present in Rust. Secondly, C++ was a choice. First of all, you pick a language for the performance reasons. The second thing is you pick a language that you’re more familiar with. I’m not familiar with Rust. I have not used it. It would take me some time to get used to it and explore it and use it in the best way possible. That would have meant lesser time devoted to the development of Netflix Drive. There is nothing that stops us from investigating it in the future, when we’ve made it a framework and we release it out there.

See more presentations with transcripts

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

(Wright-Studio/Shutterstock)

Rockset today announced launched a new integration into SQL Server that will allow customers to continuously index production data as it hits the relational database and deliver sub-second analytics on that data.

Rockset is a relational database designed to enable customers to run real-time analytics on fast moving data. One of the key elements of the offering is its Converged Index, which indexes every field in customers’ data, including structured data but also JSON, geo and time-series data. Once it’s indexed, Rockset employs SQL to query the data, which can include joins, windowing functions, sorts, aggregations, and order-bys, among others.

The new integration with SQL Server will enable customers to move away from batch-oriented ETL workflows and instead use change data capture (CDC) functionality to get at fresh data as soon as it’s entered into the relational database, the company says.

The combination of CDC with the Rockset index gives customers a new cpaabilty to analyze fast-moving data, says Shruti Bhat, the chief product officer at Rockset.

“While leading cloud providers now provide ways to tap into changes as they occur in the form of database CDC streams, most warehouses are not equipped to consume and analyze these CDC streams in real-time” Bhat says in a press release. “As a real-time analytics platform built for the cloud, Rockset is fully mutable to elegantly handles upserts and stay in sync with SQL Server tables, in real-time.”

Rockset says the new integration will enable analysts to query data generated in SQL Server using visualization tools like Tableau, Retool, Redash, and Superset. It will also enable customers to build new applications that incorporate real-time analytics in the areas of e-commerce, logistics, delivery tracking, gaming leaderboards, fraud detection systems, health and fitness trackers, and social media newsfeeds, the company says.

The San Mateo, California company boasts a collection of integrations and connectors for a variety of data sources, including data buses like Apache Kafka and Amazon Kinesis; NoSQL databases like MongoDB and DynamoDB; and relational databases like PostgreSQL and, now, SQL Server.

Rockset was founded by former Facebook engineers, including some of the team that developed Newsfeed and Search. Venkat Venkataramani, the company’s co-founder and CEO, is also a Datanami Person to Watch for 2022.

Related Items:

Did Rockset Just Solve Real-Time Analytics?

Investors Continue to Target Real-Time Analytics

Rockset, SQL Cloud Service, Emerges from Stealth

Article originally posted on InfoQ. Visit InfoQ

Subscribe on:

Apple Podcasts
Google Podcasts
Soundcloud
Spotify
Overcast
Podcast Feed

What are the major trends that matter right now in software development and technical leadership?
Deep-dive into
Java,
Microservices,
ML,
Blockchain,
APIs,
Data Pipelines & Data Mesh,
WebAssembly,
Frontends
and many more topics at
QCon Plus
this May 10-20.
Find practical inspiration (not product pitches) from software leaders deep in the trenches creating software, scaling architectures and fine-tuning their technical leadership to help you make the right decisions.
Save your spot now!

.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.