Month: November 2024

Posted on mongodb google news. Visit mongodb google news

Article originally posted on mongodb google news. Visit mongodb google news

Posted on mongodb google news. Visit mongodb google news

Healthcare of Ontario Pension Plan Trust Fund bought a new position in MongoDB, Inc. (NASDAQ:MDB – Free Report) in the third quarter, according to the company in its most recent Form 13F filing with the SEC. The fund bought 94,824 shares of the company’s stock, valued at approximately $25,636,000. Healthcare of Ontario Pension Plan Trust Fund owned about 0.13% of MongoDB as of its most recent filing with the SEC.

A number of other hedge funds have also bought and sold shares of MDB. Jennison Associates LLC increased its position in shares of MongoDB by 23.6% during the 3rd quarter. Jennison Associates LLC now owns 3,102,024 shares of the company’s stock valued at $838,632,000 after purchasing an additional 592,038 shares during the last quarter. Swedbank AB increased its position in shares of MongoDB by 156.3% during the 2nd quarter. Swedbank AB now owns 656,993 shares of the company’s stock valued at $164,222,000 after purchasing an additional 400,705 shares during the last quarter. Westfield Capital Management Co. LP increased its position in shares of MongoDB by 1.5% during the 3rd quarter. Westfield Capital Management Co. LP now owns 496,248 shares of the company’s stock valued at $134,161,000 after purchasing an additional 7,526 shares during the last quarter. Thrivent Financial for Lutherans increased its position in shares of MongoDB by 1,098.1% during the 2nd quarter. Thrivent Financial for Lutherans now owns 424,402 shares of the company’s stock valued at $106,084,000 after purchasing an additional 388,979 shares during the last quarter. Finally, Blair William & Co. IL increased its holdings in MongoDB by 16.4% in the 2nd quarter. Blair William & Co. IL now owns 315,830 shares of the company’s stock worth $78,945,000 after acquiring an additional 44,608 shares in the last quarter. Hedge funds and other institutional investors own 89.29% of the company’s stock.

Analyst Upgrades and Downgrades

Several equities research analysts recently issued reports on MDB shares. Citigroup increased their target price on shares of MongoDB from $350.00 to $400.00 and gave the stock a “buy” rating in a report on Tuesday, September 3rd. Truist Financial increased their target price on shares of MongoDB from $300.00 to $320.00 and gave the stock a “buy” rating in a report on Friday, August 30th. DA Davidson increased their target price on shares of MongoDB from $330.00 to $340.00 and gave the stock a “buy” rating in a report on Friday, October 11th. Wells Fargo & Company increased their target price on shares of MongoDB from $300.00 to $350.00 and gave the stock an “overweight” rating in a report on Friday, August 30th. Finally, Mizuho increased their target price on shares of MongoDB from $250.00 to $275.00 and gave the stock a “neutral” rating in a report on Friday, August 30th. One analyst has rated the stock with a sell rating, five have issued a hold rating, nineteen have issued a buy rating and one has assigned a strong buy rating to the company. Based on data from MarketBeat.com, the stock currently has an average rating of “Moderate Buy” and an average price target of $340.29.

Read Our Latest Stock Report on MongoDB

MongoDB Price Performance

MDB traded up $0.32 during trading on Friday, reaching $324.92. The company had a trading volume of 49,481 shares, compared to its average volume of 1,448,312. The business has a 50 day simple moving average of $282.65 and a 200-day simple moving average of $272.02. MongoDB, Inc. has a 52 week low of $212.74 and a 52 week high of $509.62. The company has a quick ratio of 5.03, a current ratio of 5.03 and a debt-to-equity ratio of 0.84.

MongoDB (NASDAQ:MDB – Get Free Report) last issued its quarterly earnings results on Thursday, August 29th. The company reported $0.70 earnings per share (EPS) for the quarter, topping analysts’ consensus estimates of $0.49 by $0.21. The company had revenue of $478.11 million during the quarter, compared to the consensus estimate of $465.03 million. MongoDB had a negative return on equity of 15.06% and a negative net margin of 12.08%. MongoDB’s revenue was up 12.8% on a year-over-year basis. During the same period last year, the company earned ($0.63) earnings per share. As a group, research analysts predict that MongoDB, Inc. will post -2.37 earnings per share for the current year.

Insider Buying and Selling

In other MongoDB news, CRO Cedric Pech sold 302 shares of the stock in a transaction on Wednesday, October 2nd. The stock was sold at an average price of $256.25, for a total value of $77,387.50. Following the sale, the executive now owns 33,440 shares in the company, valued at $8,569,000. This trade represents a 0.90 % decrease in their ownership of the stock. The transaction was disclosed in a document filed with the SEC, which is available at this hyperlink. Also, Director Dwight A. Merriman sold 2,000 shares of the stock in a transaction on Monday, November 25th. The stock was sold at an average price of $349.17, for a total value of $698,340.00. Following the completion of the sale, the director now owns 1,124,006 shares in the company, valued at approximately $392,469,175.02. The trade was a 0.18 % decrease in their position. The disclosure for this sale can be found here. Insiders sold a total of 26,600 shares of company stock worth $7,442,189 over the last three months. 3.60% of the stock is owned by insiders.

MongoDB Profile

(Free Report)

MongoDB, Inc, together with its subsidiaries, provides general purpose database platform worldwide. The company provides MongoDB Atlas, a hosted multi-cloud database-as-a-service solution; MongoDB Enterprise Advanced, a commercial database server for enterprise customers to run in the cloud, on-premises, or in a hybrid environment; and Community Server, a free-to-download version of its database, which includes the functionality that developers need to get started with MongoDB.

See Also

Before you consider MongoDB, you’ll want to hear this.

MarketBeat keeps track of Wall Street’s top-rated and best performing research analysts and the stocks they recommend to their clients on a daily basis. MarketBeat has identified the five stocks that top analysts are quietly whispering to their clients to buy now before the broader market catches on… and MongoDB wasn’t on the list.

While MongoDB currently has a “Moderate Buy” rating among analysts, top-rated analysts believe these five stocks are better buys.

View The Five Stocks Here

With average gains of 150% since the start of 2023, now is the time to give these stocks a look and pump up your 2024 portfolio.

Get This Free Report

Article originally posted on mongodb google news. Visit mongodb google news

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

Picture this: It’s Double 11 (Singles’ Day), China’s colossal online shopping frenzy and millions are hitting ‘buy’ simultaneously. Behind the scenes, a distributed database called OceanBase is handling the massive traffic spikeensuring every transaction proceeds without a hitch.

OceanBase is not just built to handle peak shopping traffic; it was specifically designed to overcome the limitations of traditional monolithic databases. With its distributed architecture, OceanBase ensures ultra-low-latency transactions, high throughput, and seamless scalability — essential for industries that need to handle large-scale data and real-time applications. Its ability to process millions of transactions per second and its robust fault tolerance and elastic scaling capabilities have made it a newly emerged innovative force for businesses that demand reliability even during peak loads.

From crisis to creation

OceanBase’s origin story is one of necessity and innovation. Rewind to 2013: Alipay, already a leader in the payments world, faced a scaling crisis as its monolithic Oracle database struggled to handle the rising volume of transactions.

However, the technical challenges extended beyond just the database. “The entire technical infrastructure was a massive fabric under extreme stress,” explains OceanBase.  

Recognizing that throwing money at the database and hardware was not sustainable, Alipay developed a fully distributed database, OceanBase, from the ground up to resolve performance bottlenecks and scale effortlessly.

In 2019, OceanBase made history by becoming the first distributed database to top the TPC-C benchmark with a score of 60 million tpmC, outperforming Oracle on standard x86 servers. It then shattered its record in 2020 with 707 million tpmC. This marked a turning point, as it was the first time a distributed database claimed the top spot in the TPC-C benchmark, traditionally dominated by monolithic systems.

Open source, enterprise-ready

Today, OceanBase serves customers from a wide range of industries, from banks and insurers to telecom operators and retailers. Its open-source approach has been a driving force behind its growing popularity, making it accessible to developers worldwide and accelerating collaboration within the community.

“Open source is essential for expanding our global reach,” OceanBase says. Now, the open-source version of OceanBase supports technology exploration and collaboration and accelerates further innovations, while the enterprise version focuses on advanced security features, catering to the rigorous demands of financial institutions and customers from other sectors.

What sets OceanBase apart is its ability to deliver seamless, unified capabilities for mission-critical workloads, ensuring consistency and efficiency across transactional, analytical, and AI-driven operations.

Its hybrid TP and AP architecture ensures high performance for transactional and analytical workloads, eliminating the need for separate systems. With multi-model integration, OceanBase supports relational, JSON, Key Value and more, enabling businesses to handle diverse data types efficiently within a single database. Furthermore, its vector hybrid search capabilities empower AI-driven workloads, bringing generative AI and recommendation system applications into a unified database environment.

Beyond its unified design and performance, OceanBase is also secure, cost-efficient, and flexible. Security is paramount for financial institutions. OceanBase offers transparent data encryption, granular access controls, and comprehensive audit trails backed by SOC2 and PCI DSS certifications.

Additionally, OceanBase caters to a wide range of deployment scenarios, whether fully on-premises, cloud-native, or somewhere in between. This multi-cloud approach supports businesses to tailor their technical stack to meet specific needs while avoiding vendor lock-in.

Database for AI and hybrid workloads

OceanBase’s roadmap is designed to meet the growing complexity of modern data-driven workloads. Key upcoming features include enhanced Hybrid Transactional/Analytical Processing (HTAP) for real-time insights, a high-performance NoSQL KV store offering a reliable alternative to HBase, and vector hybrid search to support demanding generative AI and recommendation workloads efficiently.

As organizations increasingly rely on AI and hybrid workloads, the role of the database is more critical than ever. OceanBase is not just adapting to this shift but actively driving innovation, ensuring that businesses can seamlessly manage transactional, analytical, and AI-driven operations within a unified platform.

Image credit: iStockphoto/jullasart somdok

Article originally posted on InfoQ. Visit InfoQ

Thoughtworks recently published their Technology Radar Volume 31, providing an opinionated guide to the current technology landscape.

As per the Technology Radar, Generative AI and Large Language Models (LLMs) dominate, with a focus on their responsible use in software development. AI-powered coding tools are evolving, necessitating a balance between AI assistance and human expertise.

Rust is gaining prominence in systems programming, with many new tools being written in it. WebAssembly (WASM) 1.0’s support by major browsers is opening new possibilities for cross-platform development. The report also notes rapid growth in the ecosystem of tools supporting language models, including guardrails, evaluation frameworks, and vector databases.

In the Techniques quadrant, notable items in the Adopt ring include 1% canary releases, component testing, continuous deployment, and retrieval-augmented generation (RAG). The Radar stresses the need to balance AI innovation with proven engineering practices, maintaining crucial software development techniques like unit testing and architectural fitness functions.

For Platforms, the Radar highlights tools like Databricks Unity Catalog, FastChat, and GCP Vertex AI Agent Builder in the Trial ring. It also assesses emerging platforms such as Azure AI Search, large vision model platforms such as V7, Nvidia Deepstream SDK and Roboflow, along with SpinKube. This quadrant highlights the rapid growth in tools supporting language models, including those for guardrails, evaluations, agent building, and vector databases, indicating a significant shift towards AI-centric platform development.

The Tools section underscores the importance of having a robust toolkit that combines AI capabilities with reliable software development utilities. The Radar recommends adopting Bruno, K9s, and visual regression testing tools like BackstopJS. It suggests trialing AWS Control Tower, ClickHouse, and pgvector, among others, reflecting a focus on cloud management, data processing, and AI-related database technologies.

For Languages and Frameworks, dbt and Testcontainers are recommended for adoption. The Trial ring includes CAP, CARLA, and LlamaIndex, reflecting the growing interest in AI and machine learning frameworks.

The Technology Radar also highlighted the growing interest in small language models (SLMs) as an alternative to large language models (LLMs) for certain applications, noting their potential for better performance in specific contexts and their ability to run on edge devices. This edition drew a parallel between the current rapid growth of AI technologies and the explosive expansion of the JavaScript ecosystem around 2015.

Overall, the Technology Radar Vol 31 reflects a technology landscape heavily influenced by AI and machine learning advancements, while also emphasizing the continued importance of solid software engineering practices. Created by Thoughtworks’ Technology Advisory Board, the technology Radar provides valuable insights twice-yearly for developers, architects, and technology leaders navigating the rapidly evolving tech ecosystem, offering guidance on which technologies to adopt, trial, assess, or approach with caution.

The Thoughtworks Technology Radar is available in two formats for readers: an interactive online version accessible through the website, and a downloadable PDF document.

Article originally posted on InfoQ. Visit InfoQ

AWS recently announced a new integration between AWS Amplify Hosting and Amazon Simple Storage Service (S3), enabling users to deploy static websites from S3 quickly. This integration streamlines the hosting process, allowing developers to deploy static sites stored in S3 and deliver content over AWS’s global content delivery network (CDN) with just a few clicks according to the company.

AWS Amplify Hosting, a fully managed hosting solution for static sites, now offers users an efficient method to publish websites using S3. The integration leverages Amazon CloudFront as the underlying CDN to provide fast, reliable access to website content worldwide. Amplify Hosting handles custom domain setup, SSL configuration, URL redirects, and deployment through a globally available CDN, ensuring optimal performance and security for hosted sites.

Setting up a static website using this new integration begins with an S3 bucket. Users can configure their S3 bucket to store website content, then link it with Amplify Hosting through the S3 console. From there, a new “Create Amplify app” option in the Static Website Hosting section guides users directly to Amplify, where they can configure app details like the application name and branch name. Once saved, Amplify instantly deploys the site, making it accessible on the web in seconds. Subsequent updates to the site content in S3 can be quickly published by selecting the “Deploy updates” button in the Amplify console, keeping the process seamless and efficient.

(Source: AWS News blog post)

This integration benefits developers by simplifying deployments, enabling rapid updates, and eliminating the need for complex configuration. For developers looking for programmatic deployment, the AWS Command Line Interface (CLI) offers an alternative way to deploy updates by specifying parameters like APP_ID and BRANCH_NAME.

Alternatively, according to the respondent on a Reddit thread, users could opt for Cloudflare:

If your webpage is static, you might consider using Cloudflare – it would probably be cheaper than the AWS solution.

Or using S3 and GitLab CI, according to a tweet by DrInTech:

Hello everyone! I just completed a project to host a static portfolio website, leveraging a highly accessible and secure architecture. And the best part? It costs only about $0.014 per month!

Lastly, Amplify Hosting integration with Amazon S3 is available in AWS Regions where Amplify Hosting is available and pricing details for S3 and hosting on the respective pricing pages.

Article originally posted on InfoQ. Visit InfoQ

.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.

Article originally posted on InfoQ. Visit InfoQ

Recently, AWS CodeBuild introduced support for managed GitLab self-hosted runners, towards advancement in continuous integration and continuous delivery (CI/CD) capabilities. This new feature allows customers to configure their CodeBuild projects to receive and execute GitLab CI/CD job events directly on CodeBuild’s ephemeral hosts.

The integration offers several key benefits including Native AWS Integration, Compute Flexibility and Global Availability. GitLab jobs can now seamlessly integrate with AWS services, leveraging features such as IAM, AWS Secrets Manager, AWS CloudTrail, and Amazon VPC. This integration enhances security and convenience for the users.

Furthermore, customers gain access to all compute platforms offered by CodeBuild, including Lambda functions, GPU-enhanced instances, and Arm-based instances. This flexibility allows for optimized resource allocation based on specific job requirements.The integration is available in all regions where CodeBuild is offered.

To implement this feature, users need to set up webhooks in their CodeBuild projects and update their GitLab CI YAML files to utilize self-managed runners hosted on CodeBuild machines.

The setup process involves connecting CodeBuild to GitLab using OAuth, which requires additional permissions such as create_runner and manage_runner.

It’s important to note that CodeBuild will only process GitLab CI/CD pipeline job events if a webhook has filter groups containing the WORKFLOW_JOB_QUEUED event filter. The buildspec in CodeBuild projects will be ignored unless buildspec-override:true is added as a label, as CodeBuild overrides it to set up the self-managed runner.

When a GitLab CI/CD pipeline run occurs, CodeBuild receives the job events through the webhook and starts a build to run an ephemeral GitLab runner for each job in the pipeline. Once the job is completed, the runner and associated build process are immediately terminated.

As a side, GitLab has been in the news since earlier this year as they planned to introduce CI Steps, which are reusable and composable pieces of a job that can be referenced in pipeline configurations. These steps will be integrated into the CI/CD Catalog, allowing users to publish, unpublish, search, and consume steps similarly to how they use components.

Moreover, GitLab is working on providing users with better visibility into component usage across various project pipelines. This will help users identify outdated versions and take prompt corrective actions, promoting better version control and project alignment.

AWS CodeBuild has been in the news as well, as they added support for Mac Builds. Engineers can build artifacts on managed Apple M2 instances that run on macOS 14 Sonoma. Few weeks ago, AWS CodeBuild enabled customers to configure automatic retries for their builds, reducing manual intervention upon build failures. They have also added support for building Windows docker images in reserved fleets.

Such developments demonstrate the ongoing evolution of CI/CD tools and practices, with a focus on improving integration, flexibility, and ease of use for DevOps teams.

Article originally posted on InfoQ. Visit InfoQ

Transcript

Pencea: I bet you’re wondering what this picture is doing on a tech conference. These are two German academics. They started to build a dictionary, but they actually became famous, because along the way, they collected a lot of folk stories. The reason they are here is partly because they were my idol when I was a child. I thought there was nothing better to do than to listen to stories and collect them. My family still makes fun of me because I ended up in tech after that. The way I see it, it’s not such a big difference. Basically, we still collect folk stories in tech, but we don’t call them folk stories, we call them best practices. Or we go to conferences to learn about them, basically to learn how other people screwed up, so that we don’t do the same. After we collect all these stories, we put them all together. We call them developer experience, and we try to improve that. This brings us to the talk that we have, improving developer experience using automated data CI/CD pipelines. My name is Simona Pencea. I am a software engineer at Xata.

Ványi: I’m Noémi Ványi. I’m also a software engineer at the backend team of Xata. Together, we will be walking through the data developer experience improvements we’ve been focused on recently.

Pencea: We have two topics on the agenda. The first one, testing with separate data branches covers the idea that when you create a PR, you maybe want to test your PR using a separate data environment that contains potentially a separate database. The second one, zero downtime migrations, covers the idea that we want to improve the developer experience when merging changes that include schema changes, without having any downtime. Basically, zero downtime migrations. For that, we developed an open-source tool called pgroll. Going through the first one, I will be covering several topics. Basically, I will start by going through the code development flow that we focused on. The testing improvements that we had in mind. How we ensured we have data available in those new testing environments. How we ensured that data is safe to use.

Code Development Workflow

This is probably very familiar to all of you. It’s basically what I do every day. When I’m developing, I’m starting with the local dev. I have my local dev data. It’s fake data. I’m trying to create a good local dev dataset when I’m testing my stuff. I’m trying to think about corner cases, and cover them all. The moment I’m happy with what I have in my local dev, I’m using the dev environment. This is an environment that is shared between us developers. It’s located in the cloud, and it also has a dataset. This is the dev dataset. This is also fake data, but it’s crowdfunded from all the developers that use this environment. There is a chance that you find something that it’s not in the local dev. Once everything is tested, my PR is approved. I’m merging it. I reach staging.

In staging, there is another dataset which is closer to the real life, basically, because it’s from beta testing or from demos and so on. It’s not the real thing. The real thing is only in prod, and this is the production data. This is basically the final test. The moment my code reaches prod, it may fail, even though I did my best to try with everything else. In my mind, I would like to get my hands on the production dataset somehow without breaking anything, if possible, to test it before I reach production, so that I minimize the chance of bugs.

Data Testing Improvements – Using Production Data

This is what led to this. Can we use production data to do testing with it? We’ve all received those emails sometimes, that say, test email, and I wasn’t a test user. Production data would bring a lot of value when used for testing. If we go through the pros, the main thing is, it’s real data. It’s what real users created. It’s basically the most valuable data we have. It’s also large. It’s probably the largest dataset you have, if we don’t count load test generated data and so on. It’s fast in the way that you don’t have to write a script or populate a database. It’s already there, you can use it. There are cons to this. There are privacy issues. It’s production data: there’s private information, private health information. I probably don’t even have permission from my users to use the data for testing. Or, am I storing it in the right storage? Is this storage with the right settings so that I’m not breaking GDPR or some other privacy laws.

Privacy issues are a big con. The second thing, as you can see, large is also a con, because a large dataset does not mean a complete dataset. Normally, all the users will use your product in the most common way, and then you’ll have some outliers which give you the weird bugs and so on. Having a large dataset while testing may prevent you from seeing those corner cases, because they are better covered. Refreshing takes time because of the size. Basically, if somebody changes the data with another PR or something, you need to refresh everything, and then it takes longer than if you have a small subset. Also, because of another PR, you can get into data incompatibility. Basically, you can get into a state where your test breaks, but it’s not because of your PR. It’s because something broke, or something changed, and now it’s not compatible anymore.

If we look at the cons, it’s basically two categories that we can take from those. The first ones are related to data privacy. Then the second ones are related to the size of the dataset. That gives us our requirements. The first one would be, we would like to use production data but in a safe way, and, if possible, fast. Since we want to do a CI/CD pipeline, let’s make it automated. I don’t want to run a script by hand or something. Let’s have the full experience. Let’s start with the automated part. It’s very hard to cover all the ways software developers work. What we did first was to target a simplification, like considering GitHub as a standard workflow, because the majority of developers will use GitHub. One of the things GitHub gives to you is a notification when a PR gets created. Our idea was, we can use that notification, we can hook up to it. Then we can create what we call a database branch, which is basically a separate database, but with the same schema as the source branch, when a GitHub PR gets created. Then after creation, you can copy the data after it. Having this in place would give you the automation part of the workflow.

Let’s see how we could use the production data. We said we want to have a fast copy and also have it complete. I’ll say what that means. Copying takes time. There is no way around it. You copy data, it takes time. You can hack it. Basically, you can have a preemptive copy. You copy the data before anyone needs it, so when they need it, it’s already there. Preemptive copying means I will just have a lot of datasets around, just in case somebody uses it, and then, I have to keep everything in sync. That didn’t really fly with us. We can do Copy on Write, which basically means you copy at the last minute before data is actually used, so before that, all the pointers point to the old data. The problem with using Copy on Write for this specific case is that it did not leave us any way into which we could potentially change the data to make it safe. If I Copy on Write, it’s basically the same data. I will just change the one that I’m editing, but the rest of it is the same one.

For instance, if I want to anonymize an email or something, I will not be able to do it with Copy on Write. Then, you have the boring option, which is, basically, you don’t copy all the data, you just copy a part of the data. This is what we went for, even though it was boring. Let’s see about the second thing. We wanted to have a complete dataset. I’ll go back a bit, and consider the case of a relational database where you have links as a data type. Having a complete dataset means all the links will be resolved inside of this dataset. If I copy all the data, that’s obviously a complete dataset, but if I copy a subset, there is no guarantee it will be complete unless I make it so. The problem with having a complete dataset by following the links is it sounds like an NP-complete problem, and that’s because it is an NP-complete problem. If I want to copy a subset of a bigger data, and to find it of a certain size, I would actually need to find all the subsets that respect that rule, and then select the best one. That would mean a lot of time. In our case, we did not want the best dataset that has exactly the size that we have in mind. We were happy with having something around that size. In that case, we can just go with the first dataset that we can construct that follows this completeness with all the links being resolved in size.

Data Copy (Deep Dive)

The problem with constructing this complete subset is, where do we start? How do we advance? How do we know we got to the end, basically? The where do we start part is solvable, if we think about the relationships between the tables as a graph, and then we apply a topological sort on it. We list the tables based on their degrees of independence. In this case, this is an example. t7 is the most independent, then we have t1, t2, t3, and we can see that if we remove these two things, the degrees of independence for t2 and t3 are immediately increased because the links are gone. We have something like that. Then we go further up. Here we have the special case of a cycle, because you can’t point back with links to the same table that pointed to you. In this case, we can break the cycle, because going back we see the only way to reach this cycle is through t5.

Basically, we need to first reach t5 and then t6. This is what I call the anatomy of the schema. We can see this is the order in which we need to go through the tables when we collect records. In order to answer the other two questions, things get a bit more complicated, because the schema is not enough. The problem with the schema not being enough for these cases is because, first of all, it will tell you what’s possible, but it doesn’t have to be mandatory, unless you have a constraint. Usually, a link can also be empty. If you reach a point where you run into a link that points to nothing, that doesn’t mean you should stop. You need to go and exhaustively find the next potential record to add to the set. Basically, if you imagine it in 3D, you need to project this static analysis that we did on individual rows. The thing that you cannot see through the static analysis from the beginning is that you can have several records from one table pointing to the same one in another table. The first one will take everything with it, and the second one will bring nothing.

Then you might be tempted to stop searching, because you think, I didn’t make any progress, so then the set is complete, which is not true. You need to exhaustively look until the end of the set. These are just a few of the things that, when building this thing, need to be kept on the back of the mind, basically. We need to always allow full cycle before determining that no progress was made. When we select the next record, we should consider the fact that it might have been already brought into the set, and we shouldn’t necessarily stop at that point.

We talked about at the beginning how we want to have this production data, but have it safe to use. This is the last step that we are still working on. It is a bit more fluffy. The problem with masking the data is that, for some fields, you know exactly what they are. It’s an email, then, sure, it’s private data. What if it’s free text, then what? If it’s free text, you don’t know what’s inside. The assumption is it could be private data. The approach here was to provide several possibilities on how to redact data and allow the user to choose, because the user has the context and they should be able to select based on the use case. The idea of having, for instance, a full reduction or a partial reduction, is that, sure you can apply that, but it will break your aggregations.

For instance, if I have an aggregation by username, like my Gmail address, and I want to know how many items I have assigned to my email address, if I redact the username and it will be, **.gmail, then I get aggregations on any Gmail address that has items in my table. The most complete would be a full transformation. The problem with full transformation is that it takes up a lot of memory, because you need to keep the map with the initial item and the changed item. Depending on the use case, you might not need this because it’s more complex to maintain. Of course, if there is a field that has sensitive data and you don’t need it for your specific test case, you can just remove it. The problem with removing a field is that that would basically mean you’re changing the schema, so you’re doing a migration, and that normally causes issues. In our case, we have a solution for the migrations, so you can feel free to use it.

Zero Downtime Migrations

Ványi: In this section of the presentation, I would like to talk about, what do we mean by zero downtime. What challenges do we face when we are migrating the data layer? I will talk about the expand-contract pattern and how we implemented it in PostgreSQL. What do I mean when I say zero downtime? It sounds so nice. Obviously, downtime cannot be zero because of physics, but the user can perceive it as zero. They can usually tolerate around 20 milliseconds of latency. Here I talk about planned maintenance, not service outages. Unfortunately, we rarely have any control over service outages, but we can always plan for our application updates.

Challenges of Data Migrations

Let’s look at the challenges we might face during these data migrations. Some migrations require locking, unfortunately. These can be table, read, write locks, meaning no one can access the table. They cannot read. They cannot write. In case of high availability applications, that is unacceptable. There are other migrations that might rely on read, write locks. Those are a bit better, and we can live with that. Also, it’s something we want to avoid. Also, when there is a data change, obviously we have to update the application as well, and the new instance, it has to start and run basically at the same time as the old application is running. This means that the database that we are using, it has to be in two states at the same time. Because there are two application versions interacting with our database, we must make sure, for example, if we introduce a new constraint, that it is enforced in both the existing records and on the new data as well.

Based on these challenges, we can come up with a list of requirements. The database must serve both the old schema and the new schema to the application, because we are running the old application and the new application at the same time. Schema changes are not allowed to block database clients, meaning we cannot allow our applications to be blocked because someone is updating the schema. The data integrity must be preserved. For example, if we introduce a new data constraint, it must be enforced on the old records as well. When we have different schema versions live at the same time, they cannot interfere with each other. For example, when the old application is interacting with the database, we cannot yet enforce the new constraints, because it would break the old application. Finally, as we are interacting with two application versions at the same time, we must make sure that the data is still consistent.

Expand-Contract Pattern

The expand-contract pattern can help us with this. It can minimize downtime during these data migrations. It consists of three phases. The first phase is expand. This is the phase when we add new changes to our schema. We expand the schema. The next step is migrate. That is when we start our new application version. Maybe test it. Maybe we feel lucky, we don’t test it at all. At this point, we can also shut down the old application version. Finally, we contract. This is the third and last phase. We remove the unused and the old parts from the schema. This comes with several benefits.

In this case, the changes do not block the client applications, because we constantly add new things to the existing schema. The database has to be forward compatible, meaning it has to support the new application version, but at the same time, it has to support the old application version, so the database is both forward and backwards compatible with the application versions. Let’s look at a very simple example, renaming a column. It means that here we have to create the new column, basically with a new name, and copy the contents of the old column. Then we migrate our application and delete our column with the old name. It’s very straightforward. We can deploy this change using, for example, the blue-green deployments. Here, the old application is still live, interacting with our table through the old view. At the same time, we can deploy our new application version which interacts through another new view with the same table. Then we realize that everything’s passing. We can shut down the old application and remove the view, and everything just works out fine.

Implementation

Let’s see how we implemented in Postgres. First, I would like to explain why we chose PostgreSQL in the first place. Postgres is well known, open source. It’s been developed for 30 years now. The DDL statements are transactional, meaning, if one of these statements fail, it can be rolled back easily. Row level locking. They mostly rely on row level locking. Unfortunately, there are a few read, write locks, but we can usually work around those. For example, if you are adding a nonvolatile default value, the table is not rewritten. Instead, the value is added to the metadata of the table. The old records are updated when the whole record is updated. It doesn’t really work all the time. Let’s look at the building blocks that Postgres provides. We are going to use three building blocks, DDL statements, obviously, to alter the schema.

Views, to expose the different schema version to the different application versions. Triggers and functions to migrate the old data, and on failure, to roll back the migrations. Let’s look at a bit more complex example. We have an existing column, and we want to add the NOT NULL constraint to it. It seems simple, but it can be tricky because Postgres does a table scan, meaning it locks the table, and no one can update or read the table, because it goes through all of the records and checks if any of the existing records violate the NOT NULL constraint. If it finds a record that violates this constraint, then the statement returns an error, unfortunately. We can work around it. If we add NOT VALID to this constraint, the table scan escaped. Here we add the new column and set NOT NULL constraint and add NOT VALID to it, so we are not blocking the database clients.

We also create triggers that move the old values from the columns. It is possible that some of the old records don’t yet have values, and in this case, we need to add some default value or any backfill value we want, and then we migrate our app. We need to complete the migration, obviously. We need to clean up the trigger, the view we added, so the applications can interact with the table and the old column. Also, we must remember to remove NOT VALID from the original constraint. We can do it because the migration migrated the old values, and we know that all of the new records, or new values are there, and every record satisfies the constraint.

It all seemed quite tedious to do this all the time, and that’s why we created pgroll. It’s an open-source command line tool, but it is written in Go, so you can also use it as a library. It is used to manage safe and reversible migrations using the expand-contract pattern. I would like to walk you through how to use it. Basically, pgroll is running a Postgres instance, so you need one running somewhere. After you installed and initialized it, you can start creating your migrations. You can define migrations using JSON files. I will show you an example. Once you have your migration, you run a start command. Then it creates a new view, and you can interact with it through your new application. You can test it. Then you can also shut down your old application. You run the complete command. pgroll removes all of these leftover views and triggers for you. This is the JSON example I was just talking about.

Let’s say that we have a user’s column that has an ID field, name field, and a description, and we want to make sure that the description is always there, so we put a NOT NULL constraint on it. In this case, you have to define a name. For the migration, it will be the name of the view, or the schema in Postgres. We define a list of operations. We are altering a column. The table is obviously named users. The description field, we no longer allow null values in the column. This is the interesting part. This is the up migration. It contains what to do when we are migrating the old values. In this case, it means that if the description is missing, we add the text description for and insert the name. Or if the data is there, we just move it to the new column. The down migration defines what to do when there is an error and we want to roll back. In this case, we keep the old value, meaning, if the value was missing, it’s a null, and if there was something, we keep it there.

Here is the start command. Let’s see in psql what just happened. We have a user’s table with these three columns, but you can see that pgroll added a new column. Remember, there is this migration ongoing right now. In the old description column, there are records that do not yet satisfy the constraint. In the new description the backfill value is already there for us to use. We can inspect what schemas are in the database. We can notice that there is this create_users_table, that’s the old schema version. The new one is the user_description_set_nullable, which is the name of the migration we just provided in our JSON. Let’s try to insert some values into this table. We are inserting two values. The first one is basically how the new application version is behaving. The description is not empty. In the second record, we are mimicking what the old application is doing. Here the description is NULL. Let’s say that we succeeded. We can try to query this table.

From the old app’s point of view, we can set the search path to the old schema version and perform the following query so we can inspect what happened after we inserted these values. This is what we get back. The description for Alice is, this is Alice, and for Bob it’s NULL because the old application doesn’t enforce the constraint. Let’s change the search path again to the new schema version and perform the same query. Here we can see that we have the description for Alice. Notice that Bob has a description. It is the default description or default migration we provided in the JSON file. Then we might complete the migration using the complete command, and we can see that the old schema is cleaned up. Also, the intermediary column is also removed, and the triggers, functions, everything is removed. Check out pgroll. It’s open source. It takes care of mostly everything. There is no need to manually create new views, functions, new columns, nothing. After you complete your migrations, it cleans up after itself. It is still under development, so there are a few missing features. For example, few missing migrations. We do not yet support adding comments, unfortunately, or batched migrations.

Takeaways

Pencea: Basically, what we presented so far were bits and pieces from this puzzle that we want to build the CI/CD data pipeline. What we imagined when we created this was, somebody creates a PR. Then, test environment with a test database with valid data that’s also safe to use, gets created for them. Then the tests are run. Everything is validated, PR is merged. Then it goes through the pipeline, and nobody has to take care or worry about migrations, because we can do the changes and everything.

Ványi: The migrations are done without downtime. If your pull request is merged, it goes through the testing pipeline, and if everything passes, that’s so nice. We can clean up after ourselves and remove the old schema. If maybe there is a test failure or something is not working correctly, we can roll back anytime, because the old schema is still kept around just in case. As we just demonstrated or told you about, there are still some work left for us, but we already have some building blocks that you can integrate into your CI/CD pipeline. You can create a test database on the fly using GitHub notifications, fill it with safe and relevant data to test. You can create schema changes and merge them back and forth without worrying about data migrations. You can deploy and roll back your application without any downtime.

Questions and Answers

Participant 1: Does pgroll take care of keeping the metadata of every migration done: is started, ongoing, finished?

Ványi: Yes, there is a migration site. Also, you can store, obviously, your migrations file in Git if you want to control it, but pgroll has its own bookkeeping for past migrations.

Participant 2: For the copying of the data from production, was that for local tests, local dev, or the dev? How did you control costs around constantly copying that data, standing up databases, and tearing them back down?

Pencea: It’s usually for something that sits in the cloud, so not for the local dev.

Participant 2: How did you control cost if you’re constantly standing up a near production size database?

Pencea: What we use internally is data branching. We don’t start a new instance every time. We have a separate schema inside a bigger database. Also, what we offer right now is copying 10k of data, it’s not much in terms of storage. We figured it should be enough for testing purposes.

Participant 3: I saw in your JSON file that you can do migrations that pgroll knows about like, is set nullable to false? Can you also do pure SQL migrations?

Ványi: Yes. We don’t yet support every migration. If there is anything missing, you can always work around it by using raw SQL migrations. In this case, you can shoot yourself in the foot, because, for example, in case of NOT NULL, we take care of the skipping of the table scan for you. When you are writing your own raw SQL migration, you have to be careful not to block your table and the database access.

Participant 4: It’s always amazed me that these databases don’t do safer actions for these very common use cases. Have you ever talked to the Postgres project on improving the actual experience of just adding a new column, or something? It should be pretty simple.

Ványi: We’ve been trying to have conversations about it, but it is a very mature project, and it is somewhat hard to change such a fundamental part of this database. Constraints are like the basic building block for Postgres, and it’s not as easy to just make it more safe. There is always some story behind it.

Pencea: I think developer experience was not necessarily something that people were concerned about, up until recently. I feel like sometimes it was actually the opposite, if it was harder, you looked cooler, or you looked like a hacker. It wasn’t exactly something that people would optimize for. I think it’s something that everybody should work towards, because now everybody has an ergonomic chair or something, and nobody questions that, but we should work towards the same thing about developer experience, because it’s ergonomics in the end.

Participant 5: In a company, assuming they are adopting pgroll, all these scripts can grow in number, so at some point you have to apply all of them, I suppose, in order. Is there any sequence number, any indication, like how to apply these. Because some of them might be serial, some of them can be parallelized. Is there any plan to give direction on the execution? I’ve seen there is a number in the script file name, are you following that as a sequence number, or when you’re then developing your batching feature, you can add a sequence number inside?

Ványi: Do we follow some sequence number when we are running migrations?
Yes and no. pgroll maintains its own table or bookkeeping, where it knows what was the last migration, what is coming next? The number in the file name is not only for pgroll, but also for us.

Participant 6: When you have very breaking migrations using pgroll, let’s say you need to rename a column, or even changing its type, which you basically replicate a new column and then copying over the data. How do you deal with very large tables, say, millions of rows? Because you could end up having even some performance issues with copying these large amounts of data.

Ványi: How do we deal with tables that are basically big? How do we make sure that it doesn’t impact the performance of the database?

For example, in case of moving the values to the new column, we are creating triggers that move the data in batches. It’s not like everything is copied in one go, and you cannot really use your Postgres database because it is busy copying the old data. We try to minimize and distribute the load on the database.

Participant 7: I know you were using the small batches to copy the records from the existing column to the new column. Once you copy all the records, only then you will remove the old column. There is a cost with that.

See more presentations with transcripts

Article originally posted on InfoQ. Visit InfoQ

DORA can help to drive sustainable change, depending on how it is used by teams and the way it is supported in a company. According to Carlo Beschi, getting good data for the DORA keys can be challenging. Teams can use DORA reports for continuous improvement by analysing the data and taking actions.

Carlo Beschi spoke about using DORA for sustainable improvement at Agile Cambridge.

Doing DORA surveys in your company can help you reflect on how you are doing software delivery and operation as Beschi explained in Experiences from Doing DORA Surveys Internally in Software Companies. The way you design and run the surveys, and how you analyze the results, largely impact the benefits that you can get out of them.

Treatwell’s first DORA implementation in 2020 focused on getting DORA metrics from the tools. They set up a team that sits between their Platform Engineering team and their “delivery teams” (aka product teams, aka stream aligned teams), called CDA – Continuous Delivery Acceleration team. Half of their time is invested in making other developers and teams life better, and the other half is about getting DORA metrics from the tools:

We get halfway there, as we manage to get deployment frequency and lead time for changes for almost all of our services running in production, and when the team starts to dig into “change failure rate”, Covid kicks in and the company is sold.

DORA can help to drive sustainable change, but it depends on the people who lead and contribute to it, and how they approach it, as Beschi learned. DORA is just a tool, a framework, that you can use to:

• Lightweight assess your teams and organisation
• Play back the results, inspire reflection and action
• Check again a few months / one year later, maybe with the same assessment, to see if / how much “the needle has moved”

Beschi mentioned that teams use the DORA reports as part of their continuous improvement. The debrief about the report is not too different from a team retrospective, one that brings in this perspective and information, and from which the team defines a set of actions, that are then listed, prioritised, and executed.

He has seen benefits from using DORA in terms of aligning people on “this is what building and running good software nowadays looks like”, and “this is the way the best in the industry work, and a standard we aim for”. Beschi suggested focusing the conversation on the capabilities, much more than on the DORA measures:

I’ve had some good conversations, in small groups and teams, starting from the DORA definition of a capability. The sense of “industry standard” helped move away from “I think this” and “you think that”.

Beschi mentioned the advice and recommendations from the DORA community on “let the teams decide, let the teams pick, let the teams define their own ambition and pace, in terms of improvement”. This helps in keeping the change sustainable, he stated.

When it comes to meeting the expectations of senior stakeholders, when your CTO is the sponsor of a DORA initiative then there might be “pushback” on teams making decisions, and expectations regarding their “return of investment” on doing the survey, aiming to have more things change, quicker, Beschi added.

A proper implementation of DORA is far from trivial, Beschi argued. The most effective ones rely on a combination of data gathered automatically from your system alongside qualitative data gathered by surveying (in a scientific way) your developers. Getting good data quickly from the systems is easier said than done.

When it comes to getting data from your systems for the four DORA keys, while there has been some good progress in the tooling available (both open and commercial) it still requires effort to integrate any of them in your own ecosystem. The quality of your data is critical.

Start ups and scale ups are not necessarily very disciplined when it comes to consistent usage of their incident management processes – and this impacts a lot the accuracy of your “failure change rate” and “response time” measures, Beschi mentioned.

Beschi mentioned several resources for companies that are interested in using DORA:

• The DORA website, where you can self-serve all DORA key assets and find the State of DevOps reports
• The DORA community has a mailing list and bi-weekly vídeo calls
• The Accelerate book

In the community you will find a group of passionate and experienced practitioners, very open, sharing their stories “from the trenches” and very willing to onboard others, Beschi concluded.

Article originally posted on InfoQ. Visit InfoQ

.NET Aspire 9.0 is now generally available, following the earlier release of version 9.0 Release Candidate 1 (RC1). This release brings several features aimed at improving cloud-native application development on both AWS and Azure. It supports .NET 8 (LTS) and .NET 9 (STS).

A key update in Aspire 9.0 is the integration of AWS CDK, enabling developers to define and manage AWS resources such as DynamoDB tables, S3 buckets, and Cognito user pools directly within their Aspire projects. This integration simplifies the process of provisioning cloud resources by embedding infrastructure as code into the same environment used for developing the application itself. These resources are automatically deployed to an AWS account, and the references are included seamlessly within the application.

Azure integration has been upgraded in Aspire 9.0. It now offers preview support for Azure Functions, making it easier for developers to build serverless applications. Additionally, there are more configuration options for Azure Container Apps, giving developers better control over their cloud resources. Aspire 9.0 also introduces Microsoft Entra ID for authentication in Azure PostgreSQL and Azure Redis, boosting security and simplifying identity management.

In addition to cloud integrations, Aspire 9.0 introduces a self-contained SDK that eliminates the need for additional .NET workloads during project setup. This change addresses the issues faced by developers in previous versions, where managing different .NET versions could lead to conflicts or versioning problems. 

Aspire Dashboard also receives several improvements in this release. It is now fully mobile-responsive, allowing users to manage their resources on various devices. Features like starting, stopping, and restarting individual resources are now available, giving developers finer control over their applications without restarting the entire environment. The dashboard provides better insights into the health of resources, including improved health check functionality that helps monitor application stability.

Furthermore, telemetry and monitoring have been enhanced with expanded filtering options and multi-instance tracking, enabling better debugging in complex application environments. The new support for OpenTelemetry Protocol also allows developers to collect both client-side and server-side telemetry data for more comprehensive performance monitoring.

Lastly, resource orchestration has been improved with new commands like WaitFor and WaitForCompletion, which help manage resource dependencies by ensuring that services are fully initialized before dependent services are started. This is useful for applications with intricate dependencies, ensuring smoother deployments and more reliable application performance.

Community feedback highlights how much Aspire’s development experience has been appreciated. One Reddit user noted:

It is super convenient, and I am a big fan of Aspire and how far it has come in such a short time.

Full release details and upgrade instructions are available in the .NET Aspire documentation.