Category: Uncategorized

Article originally posted on InfoQ. Visit InfoQ

Trouble seeing, pain, and stiffness are some of the things that can make it harder to program as you age, as Kate Gregory discovered. But they aren’t inevitable, and there are solutions like changing fonts, using glasses, and rearranging the office layout that can help. Some older programmers’ mental concerns included a lack of motivation and cynical feelings. According to Gregory, exercising and getting good sleep help to keep the abilities needed as a developer.

Kate Gregory gave a talk about continuing to program as people age at NDC Tech Town.

There’s a lot of pressure to stop programming as you get older, Gregory said. Many people move into management, or change careers entirely, or retire. She wants to still write code, and she wanted to explore what it is about getting older that can make it harder to remain a programmer:

As I looked into it, I came to understand there are things you can do in your 20s, 30s, and so on that will make a difference, not just in the moment, but accumulate over time to give you a much healthier and happier old age.

Gregory did a survey to explore ageing for programmers as part of preparing her talk, and hundreds of developers, of all ages, responded. She asked about body issues, mind and brain issues, discrimination, and more:

The number one body concern of older programmers was eyesight, which surprised me, along with pain and stiffness. The number one mental concern was a lack of motivation and feeling cynical.

Sometimes a difference in your body can make it harder to work, but not all body differences actually impede writing code, Gregory said. She gave an example for people who find stairs hard. If you work in an office with all the meeting rooms on a separate floor from the offices, and all the coffee areas on another floor entirely, going up and down those stairs all the time may be a big issue for you, she said. That’s not so much a problem with your body as it is a problem with your workplace, Gregory argued.

Specifically for eyesight, that can make it hard to see a screen and so on, the survey respondents reported changing font, having glasses, and some people had laser eye surgery:

It’s perfectly normal to have different glasses for different tasks, in my case for coding and for driving.

Gregory suggested getting plenty of Vitamin A to help your night vision.

The respondents didn’t report taking action around mood issues, perhaps because they didn’t know that they could do anything about them, Gregory said. She decided to do some research on this:

My research shows it’s not typical to get grumpier and nastier as you age: most people’s personalities intensify rather than flip.

Gregory mentioned that some people react badly to losses and disappointments during their life and can end up pretty miserable: the more control you have over things like your career and your health, the more you can avoid that situation.

People who believe they can still do most of what they want to do are happier than those who have had to give up a lot, Gregory concluded.

InfoQ interviewed Kate Gregory about things developers can do to continue to program as they age.

InfoQ: How can programmers keep their health and condition in good shape?

Kate Gregory: I’m sad to report that study after study shows you have to exercise. I think we all wish that wasn’t true. But being active improves your stamina, lowers your pain and stiffness, boosts your immune system, and even helps you to learn!

You should also eat fruits and vegetables, to help your night vision, and gain the benefits that Vitamin C and fibre bring.

InfoQ: How important is getting good sleep?

Gregory: It’s absolutely vital. One study on sleep and subjective age found just one month of good sleep made people feel almost six years younger – and just two terrible nights made them feel almost five years older. Sleep is when we heal and often when I solve a lot of my problems.

It’s normal to need less as you age, but it’s also normal to need more — people who need more sleep aren’t lazy or sluggish, they just have a physical need for more sleep. Get as much as you need and you’ll be able to do all you want in the hours you’re awake.

Posted on mongodb google news. Visit mongodb google news

Anthosa Consulting, a forward-thinking boutique strategy consulting firm, is proud to announce its strategic partnerships with ten pioneering technology providers: AWS, Centreon, Databricks, Dynatrace, MongoDB, Prevedere, Resourcely, Snowflake, WSO2, Zabbix, and Zededa.

Article originally posted on mongodb google news. Visit mongodb google news

Posted on mongodb google news. Visit mongodb google news

National Pension Service decreased its position in MongoDB, Inc. (NASDAQ:MDB – Free Report) by 1.4% during the 3rd quarter, according to its most recent filing with the SEC. The institutional investor owned 131,005 shares of the company’s stock after selling 1,810 shares during the period. National Pension Service owned about 0.18% of MongoDB worth $35,417,000 as of its most recent filing with the SEC.

A number of other large investors have also recently bought and sold shares of the company. Swedbank AB boosted its holdings in shares of MongoDB by 156.3% in the 2nd quarter. Swedbank AB now owns 656,993 shares of the company’s stock valued at $164,222,000 after buying an additional 400,705 shares during the last quarter. Thrivent Financial for Lutherans raised its holdings in shares of MongoDB by 1,098.1% during the 2nd quarter. Thrivent Financial for Lutherans now owns 424,402 shares of the company’s stock worth $106,084,000 after buying an additional 388,979 shares in the last quarter. Clearbridge Investments LLC boosted its holdings in shares of MongoDB by 109.0% during the 1st quarter. Clearbridge Investments LLC now owns 445,084 shares of the company’s stock valued at $159,625,000 after acquiring an additional 232,101 shares in the last quarter. Point72 Asset Management L.P. acquired a new stake in shares of MongoDB during the second quarter valued at about $52,131,000. Finally, Renaissance Technologies LLC grew its holdings in MongoDB by 828.9% during the 2nd quarter. Renaissance Technologies LLC now owns 183,000 shares of the company’s stock worth $45,743,000 after acquiring an additional 163,300 shares during the last quarter. Institutional investors and hedge funds own 89.29% of the company’s stock.

Wall Street Analysts Forecast Growth

Several research analysts recently issued reports on the company. Citigroup raised their target price on MongoDB from $350.00 to $400.00 and gave the stock a “buy” rating in a research report on Tuesday, September 3rd. Scotiabank raised their price target on shares of MongoDB from $250.00 to $295.00 and gave the stock a “sector perform” rating in a report on Friday, August 30th. UBS Group upped their price objective on shares of MongoDB from $250.00 to $275.00 and gave the company a “neutral” rating in a research note on Friday, August 30th. Royal Bank of Canada reaffirmed an “outperform” rating and set a $350.00 target price on shares of MongoDB in a research note on Friday, August 30th. Finally, Wedbush raised shares of MongoDB to a “strong-buy” rating in a research note on Thursday, October 17th. One investment analyst has rated the stock with a sell rating, five have issued a hold rating, twenty have assigned a buy rating and one has given a strong buy rating to the stock. According to data from MarketBeat.com, the stock presently has an average rating of “Moderate Buy” and an average price target of $337.96.

Read Our Latest Report on MongoDB

Insider Buying and Selling at MongoDB

In other MongoDB news, Director Dwight A. Merriman sold 3,000 shares of the firm’s stock in a transaction dated Tuesday, September 3rd. The stock was sold at an average price of $290.79, for a total transaction of $872,370.00. Following the sale, the director now directly owns 1,135,006 shares of the company’s stock, valued at $330,048,394.74. This trade represents a 0.00 % decrease in their ownership of the stock. The transaction was disclosed in a legal filing with the Securities & Exchange Commission, which is available through the SEC website. In other news, Director Dwight A. Merriman sold 3,000 shares of MongoDB stock in a transaction that occurred on Tuesday, September 3rd. The stock was sold at an average price of $290.79, for a total value of $872,370.00. Following the completion of the transaction, the director now owns 1,135,006 shares in the company, valued at approximately $330,048,394.74. The trade was a 0.00 % decrease in their position. The sale was disclosed in a document filed with the Securities & Exchange Commission, which can be accessed through the SEC website. Also, Director Dwight A. Merriman sold 1,000 shares of MongoDB stock in a transaction on Friday, August 30th. The stock was sold at an average price of $290.40, for a total transaction of $290,400.00. Following the transaction, the director now directly owns 1,138,006 shares of the company’s stock, valued at $330,476,942.40. The trade was a 0.00 % decrease in their ownership of the stock. The disclosure for this sale can be found here. Over the last 90 days, insiders have sold 24,281 shares of company stock valued at $6,657,121. 3.60% of the stock is owned by company insiders.

MongoDB Stock Up 2.5 %

MDB traded up $7.03 on Thursday, hitting $288.01. The company had a trading volume of 93,961 shares, compared to its average volume of 1,424,771. The stock has a fifty day simple moving average of $276.77 and a 200-day simple moving average of $277.95. The company has a debt-to-equity ratio of 0.84, a current ratio of 5.03 and a quick ratio of 5.03. The firm has a market capitalization of $21.28 billion, a P/E ratio of -93.04 and a beta of 1.15. MongoDB, Inc. has a 52-week low of $212.74 and a 52-week high of $509.62.

MongoDB (NASDAQ:MDB – Get Free Report) last released its quarterly earnings results on Thursday, August 29th. The company reported $0.70 earnings per share for the quarter, beating the consensus estimate of $0.49 by $0.21. MongoDB had a negative return on equity of 15.06% and a negative net margin of 12.08%. The company had revenue of $478.11 million for the quarter, compared to analyst estimates of $465.03 million. During the same period in the previous year, the company posted ($0.63) earnings per share. The firm’s revenue was up 12.8% on a year-over-year basis. As a group, equities analysts expect that MongoDB, Inc. will post -2.39 EPS for the current fiscal year.

MongoDB Company Profile

(Free Report)

MongoDB, Inc, together with its subsidiaries, provides general purpose database platform worldwide. The company provides MongoDB Atlas, a hosted multi-cloud database-as-a-service solution; MongoDB Enterprise Advanced, a commercial database server for enterprise customers to run in the cloud, on-premises, or in a hybrid environment; and Community Server, a free-to-download version of its database, which includes the functionality that developers need to get started with MongoDB.

Featured Stories

Before you consider MongoDB, you’ll want to hear this.

MarketBeat keeps track of Wall Street’s top-rated and best performing research analysts and the stocks they recommend to their clients on a daily basis. MarketBeat has identified the five stocks that top analysts are quietly whispering to their clients to buy now before the broader market catches on… and MongoDB wasn’t on the list.

While MongoDB currently has a “Moderate Buy” rating among analysts, top-rated analysts believe these five stocks are better buys.

View The Five Stocks Here

As the AI market heats up, investors who have a vision for artificial intelligence have the potential to see real returns. Learn about the industry as a whole as well as seven companies that are getting work done with the power of AI.

Get This Free Report

Article originally posted on mongodb google news. Visit mongodb google news

Posted on mongodb google news. Visit mongodb google news

How 400 Scandalous Videos of Equatorial Guinea’s Baltasar Ebang Engonga Were Discovered and Leaked

As the world continues to discuss the scandal involving Baltasar Ebang Engonga, Equatorial Guinea’s National Financial Investigation Agency director, new details have surfaced on how 400 videos of him with various women made their way onto the internet.

Equatorial Guinea’s Vice President, Teodoro Nguema Obiang Mangue, announced on X that Engonga has been suspended and is currently under investigation.

Also read: WATCH: Videos Of Prophet Israel Ford’s Two Wives Having Lula Lula Leak

How 400 Videos of Equatorial Guinea’s Baltasar Ebang Engonga Were Leaked

Engonga kept the videos on his computer and reports indicate that public investigators stumbled upon them while examining it as part of a broader inquiry into corruption and embezzlement of public funds. They uncovered around 400 explicit videos involving Engonga and multiple women, including some who are wives or relatives of high-ranking government officials.

Although it remains unclear who leaked the footage or their motives, some videos surfaced on social media, sparking shock and outrage across in Equatorial Guinea and the world over.

 It remains unclear whether the encounters depicted were consensual or if any of the women involved have filed formal complaints against Engonga.

Also read: Hopewell Chin’ono Weighs in on Viral Lula Lula Scandal Involving Equatorial Guinea Politician Baltasar Ebang Engonga

Baltasar Ebang Engonga Could Face Charges for Lula Lula

Prosecutor General Anatolio Nzang Nguema noted that authorities are investigating whether Engonga may have used these relationships to knowingly spread a disease, which could lead to public health endangerment charges.

“The authorities want to establish whether the man deliberately used these relationships to spread a possible disease among the population,” Prosecutor General Anatolio Nzang Nguema said.

In response to the scandal, Vice President Mangue announced plans to install surveillance cameras in government offices to “eradicate improper and illicit behaviour.” He emphasized that anyone caught engaging in any form of sexual activity in the workplace will be dismissed immediately.

Follow Us on Google News for Immediate Updates

Article originally posted on mongodb google news. Visit mongodb google news

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

This post is co-written with TaeHun Yoon and Changsoon Kim from Channel Corporation.

Channel Corporation is a B2B software as a service (SaaS) startup that operates the all-in-one artificial intelligence (AI) messenger Channel Talk. In Part 1 of this series, we introduced our motivation for NoSQL adoption, technical problems with business growth, and considerations for migration from PostgreSQL to Amazon DynamoDB. In this post, we share our experience integrating with other services to solve areas that couldn’t be addressed with DynamoDB alone.

Background: Structured and unstructured data retrieval problems

There are some key differences between relational data design and NoSQL. DynamoDB excels at efficient data retrieval for well-defined access patterns, offering optimized performance for specific query types. Given this characteristic, the following Channel Corporation use cases were difficult to solve with DynamoDB alone:

• Searching for structured data with various filtering conditions (message search) – Channel Talk allows users to quickly search for a conversation. Users should be able to search by adding various filters such as assignee, team, and follower, as illustrated in the following screenshot.

• Searching for unstructured data (customer data search) – Customer data has a different schema depending on what information the Channel Corporation’s customer enters. This schema is not fixed, and at the time of writing, it has the characteristic of being able to quickly search up to 1 million customer data records using multiple fields.

Channel Corporation decided that it would be more effective and efficient to use a purpose-built search service such as Amazon OpenSearch Service with DynamoDB to solve these two problems. To do this, data synchronization between DynamoDB and other services is required. DynamoDB has zero-ETL integration with Amazon OpenSearch Service, but Channel Corporation currently uses self-managed extract, transform, and load(ETL) pipeline with the following reasons:

• It was necessary to transfer only the minimum information necessary for searchable data. Plus, we needed to ignore changes in the value of a specific attribute, and our logic compared the changes between the existing and current value.
• In order to ensure idempotence in search service, there were cases where records had to be changed and inserted using soft delete instead of directly deleting them.

Integrating streams with DynamoDB

A series of sequential events ordered in time is called a stream. DynamoDB provides two ways to run change data capture (CDC) as a stream:

• Amazon DynamoDB Streams captures a time-ordered sequence of item-level modifications in a DynamoDB table and stores this information in a log for up to 24 hours
• Amazon Kinesis Data Streams captures item-level modifications in a DynamoDB table and replicates them to a Kinesis data stream

Channel Corporation was able to use these services to solve the message and customer data search issues by passing the changed data of DynamoDB to a service that can search well through a stream.

The following diagram illustrates the workflow using DynamoDB Streams.

This solution offers the following characteristics:

• Reading from DynamoDB Streams is free for AWS Lambda based consumers
• It offers a deduplicated, time-ordered sequence of item-level modifications

However, it has the following considerations:

• You need to handle failures in Lambda, the stream processing layer
• There is a possibility of missing events if the starting position is set to LATEST

The following diagram illustrates the workflow using Kinesis Data Streams.

This solution offers the following characteristics:

However, consider the following:

• It needs to handle failures in Lambda, the stream processing layer
• There is a possibility of missing events if the starting position is set to LATEST
• There is a possibility of reverse or duplicated events

To understand this in more detail, let’s look at how data is passed from DynamoDB to each stream and how Lambda performs operations on the streams.

DynamoDB Streams

DynamoDB sends changes that are performed on each item within a partition to the corresponding shard by maintaining the order of the changes made. This process makes sure a given key is present in at most one shard and its order is maintained.

Kinesis Data Streams

Data records in Kinesis Data Streams may appear in a different order than when item changes occurred, and the same item notification may appear in the stream more than once. You can check the ApproximateCreationDateTime attribute to identify the order that the item modifications occurred in, and to identify duplicate records.

How Lambda performs operations on streams

An event source mapping is a Lambda resource that reads items from stream- and queue-based services and invokes a function with batches of records. You can use an event source mapping to process events from streams or queues of services that don’t directly invoke Lambda functions. Let’s approach the characteristics of each solution and problems again with the understanding that Lambda is not invoked directly from DynamoDB Streams, but through that resource.

You can handle basic retry processing by setting the MaximumRecordAgeInSeconds and MaximumRetryAttempts values ​​of the event source mapping configuration. However, failures that can’t be resolved by only retry can occur due to various reasons, such as bugs in your Lambda code or issues faced during deployment.

When you look at the event source mapping resource, you can configure the On-failure destination setting to forward notifications about the records that could be processed to an Amazon Simple Queue Service (Amazon SQS) queue or Amazon Simple Notification Service (Amazon SNS) topic. The following example shows an invocation record for a DynamoDB stream:

{
    "requestContext": {
        "requestId": "316aa6d0-8154-xmpl-9af7-85d5f4a6bc81",
        "functionArn": "arn:aws:lambda:us-east-2:123456789012:function:myfunction",
        "condition": "RetryAttemptsExhausted",
        "approximateInvokeCount": 1
    },
    "responseContext": {
        "statusCode": 200,
        "executedVersion": "$LATEST",
        "functionError": "Unhandled"
    },
    "version": "1.0",
    "timestamp": "2019-11-14T00:13:49.717Z",
    "DDBStreamBatchInfo": {
        "shardId": "shardId-00000001573689847184-864758bb",
        "startSequenceNumber": "800000000003126276362",
        "endSequenceNumber": "800000000003126276362",
        "approximateArrivalOfFirstRecord": "2019-11-14T00:13:19Z",
        "approximateArrivalOfLastRecord": "2019-11-14T00:13:19Z",
        "batchSize": 1,
        "streamArn": "arn:aws:dynamodb:us-east-2:123456789012:table/mytable/stream/2019-11-14T00:04:06.388"
    }
}

Based on the preceding information, you should retrieve the record from DynamoDB Streams and try again. When retrieving the record from DynamoDB Streams and processing it, not all events are delivered in chronological order, so out of order delivery of events is possible.

When the BatchSize is more than 1 and some items fail to process due to transient failures, Lambda retries the entire batch, including previously processed records. Without appropriate handling, this can result in duplicate events.

In addition, if the starting position of the event source mapping is set to LATEST, some events may be missed.

Sometimes values set in the event source mapping, such as MaximumRetryAttempts and MaximumRecordAgeInSeconds, need to be changed depending on error handling and situations, unlike the initial settings. In this case, some records may be missed unintentionally.

If the starting position is changed to TRIM_HORIZON to solve this, all data in DynamoDB Streams will be delivered to the event consumer from the beginning, which may result in duplicate processing of events.

Problem-solving with DynamoDB Streams and Kinesis Data Streams

We believe that both DynamoDB Streams and Kinesis Data Streams have the ability to solve similar problems. For this post, we discuss the following use cases:

• Is it possible to write all stream processing functions idempotently?
• Can we retry when a problem occurs in a Lamdba function?

Idempotence

One of the most important things in stream processing is to design your logic idempotent. You should validate incoming events and determine if they’ve been processed before. If you write your event consumer idempotently, many problems can be solved.

For example, let’s look at a situation where events appear in the stream out of order.

As shown in the preceding figure, data integrity can be broken due to processing of events out of order.

To solve this, if all events occurring in create, update, and delete operations are performed in chronological order, there will be no problem because each state is the same final state.

In other words, if the last current state is the result of the most recent event, the preceding problem can be easily solved.

To do this, let’s rewrite the implementation, assuming that updates are only performed if the timestamp representing the time is greater, so that only events after the current state are performed.

In this case, out of order delivery occurred, but because it’s an event in the past rather than the current state, it is not performed, so the same result value can be obtained. DynamoDB allows optimistic locking with version number, and the version number automatically increases each time an item is updated. An update or delete request is possible only if the client-side object version matches the corresponding item version number in the DynamoDB table. If you use this characteristic, you can solve the problem.

If we perform the same logic as before, we have no issues with create and update operations, but there is a case where it becomes a problem for deletes.

Even in the case of deletes, the problem can be solved by using soft delete instead of hard delete for records in the service to enforce the order of occurrence of events. For example, if A is deleted and there is information about when it was deleted, we can use the information and drop the event.

Retry strategy for failure

Now, let’s assume that all logic is written idempotently and talk about whether you can retry when a problem occurs.

Both Kinesis Data Streams and DynamoDB Streams can use the On-failure destination option and redeliver past data to stream consumers. However, the strategies for the two streams may be different:

• DynamoDB Streams – DynamoDB Streams provides LATEST and TRIM_HORIZON in the starting position of the event source mapping. This means that in order to get records from a specific point in time again, a separate application should exist to read and reprocess from a specific sequence number in a specific shard to the desired point in time.
• Kinesis Data Streams – Kinesis Data Streams provides five options, including AT_TIMESTAMP, in the starting position of the event source mapping. This feature allows us to go back to the point just before the problem occurred, update only the event source mapping, and redeploy to resolve the problem.

Channel Corporation’s choice

We looked at the cases that can arise when synchronizing data with other services using the two streams provided by DynamoDB. It is difficult to say that it is unconditionally better to use a specific stream because the pros and cons of the two streams are different in terms of operational considerations and cost. Therefore, Channel Corporation uses both streams based on specific criteria.

In the following use cases, we use DynamoDB Streams:

• When it’s important that events occur in chronological order
• When it’s okay to have higher error recovery costs when a problem occurs

In the following use cases, we use Kinesis Data Streams:

• When it’s important to recover quickly from a desired point in time when a problem occurs
• When there is a case where more than two Lambda functions need to process the stream simultaneously

Online schema change strategy using DynamoDB Streams

As another example of using streams, Channel Corporation uses DynamoDB Streams to perform online schema change. The same approach can be used to migrate between different AWS accounts. The following diagram illustrates the workflow.

This workflow includes the following steps:

1. The first step consists of two parts:
   1. Create a new table with a new schema in DynamoDB.
   2. Deploy a Lambda function that consumes the DynamoDB Streams event of the old table and converts it to the new table schema.
2. Read the historical data before the Lambda function was deployed and change it to the new schema.
3. Deploy new API servers.

This process enables us to perform live schema change even when there are significant schema changes. In Step 2, there are various ways to input data to the new table, such as Amazon EMR, AWS Glue, or a custom application.

When you need to insert data from a specific point in time into a new DynamoDB table, there are also many things to care about due to idempotence. To simplify this, Channel Corporation creates a pipeline as illustrated in the preceding figure and increases the version of all existing items by 1. In this case, all changed items are moved to DynamoDB Streams and Lambda can process them to a new schema, so you can transfer data to the new table without much concern.

Conclusion

With DynamoDB, scaling is nearly infinite, and dependencies with various downstream services are eliminated. For Channel Corporation, the combination of DynamoDB and Kinesis Data Streams offers a robust solution for application deployment. This pairing enables quick recovery from a specific point in time if issues arise during deployment. As a result, developers can confidently perform deployments at any time, knowing they have a reliable fallback mechanism in place. Finally, we can implement an online schema change strategy leveraging one of the streaming options for DynamoDB to remove legacy tables and efficiently manage tables.

Consider implementing a similar solution for your own use case, and leave your questions in the comments section.

About the Authors

TaeHun (Clsan) Yoon, a seasoned professional in the realm of computer engineering, spearheads innovative solutions as the CTO (Chief Technology Officer) at Channel Corp. With a keen focus on enhancing the chatting experience, TaeHun and his team are dedicated to resolving diverse challenges encountered by customers.

Changsoon (CS) Kim is a DevOps Engineer at Channel Corp. He is interested in efficiently resolving issues between development and operations.

Sungbae Park is Account Manager in the AWS Startup Team and a regular member of AWS SaaS TFC (Technical Field Communities). He is currently focusing on helping B2B software startups grow and succeed with AWS. Sungbae previously worked as Partner Development Manager making mutual growth with MSP, SI, and ISV partners.

Hyuk Lee is a Sr. DynamoDB Specialist Solutions Architect based in South Korea. He loves helping customers modernize their architecture using the capabilities of Amazon DynamoDB.

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

This post is co-written with TaeHun Yoon and Changsoon Kim from Channel Corporation.

Channel Corporation is a B2B software as a service (SaaS) startup that operates the all-in-one artificial intelligence (AI) messenger Channel Talk. Channel Corporation’s vision is “to solve all problems between customers and companies,” and its first product, Channel Talk, emerged to solve communication issues between customers and companies. Channel Talk is an all-in-one product that provides live chat for customer service, chatbots, voice calls, customer relationship management (CRM) marketing, and in-house messaging. More than 150,000 customers are using Channel Talk in 22 countries, including Korea, Japan, and the US, and more than 70 million calls are made every month.

This two-part blog series starts by presenting the motivation and considerations for migrating from RDBMS to NoSQL. Part 2 covers how Channel Corporation uses streams to implement event-driven architecture.

In this post, we discuss the motivation behind Channel Corporation’s architecture modernization with Amazon DynamoDB, the reason behind choosing DynamoDB, and the four major considerations before migrating from Amazon Relational Database Service (Amazon RDS) for PostgreSQL.

Background: Business growth and emergence of new problems

Channel Talk was initially divided into two components: Team Chat and User Chat. The Team Chat feature, shown in the following screenshot, helps communication within a team.

The User Chat feature, shown in the following screenshot, helps communication between customers and companies.

As the business has grown 2–5 times every year since 2018, the number of requests per second (RPS) in the chat service also began to increase rapidly. Previously, we ran the chat service with Amazon RDS for PostgreSQL, so we naturally scaled up instance types when we needed higher performance.

However, we had to modify our scale-up strategy when we started a marketing service that sent campaign messages and one-time messages. A campaign message is a function that automatically sends messages based on rules when a customer performs a specific action. For example, sending a welcome message when a customer signs up or a discount coupon when viewing a pricing page. A one-time message is a function that sends messages to target customers just once at a desired time. For example, it can be used to issue limited discount coupons to customers who are currently online.

Channel Talk’s main workload was chat-based one-on-one customer consultation, so it wasn’t difficult to predict the peak traffic. However, the marketing service was expected to create lots of traffic spikes in multiple tables in an instant when a customer ran a sale event or sent one-time messages to a large number of customers. As a result, Channel Corporation needed a new scalable database, and there were four major considerations:

• Cost inefficiency – Instance-based services need to select the instance size based on the peak, which the marketing service was expected to add high variability in.
• Inter-table load propagation – High loads on specific tables and services could affect performance of the entire RDS instance.
• Support to implement efficient ways of performing operations that we did in PostgreSQL.
• Prepare a migration strategy aligned with our requirements and acceptable downtime.

Reasons to choose DynamoDB

In addition to solving the four problems we mentioned, Channel Corporation chose DynamoDB for three main reasons:

• The ability to implement event sourcing to other AWS services, a common event-driven architecture pattern
• Rich integration with other AWS services
• ACID transactions

Our approaches to solving the four problems

The first and second problems related with the marketing service could be solved simply by choosing DynamoDB.

We were able to handle the cost inefficiency problem flexibly because DynamoDB provided on-demand capacity mode. This mode instantly accommodated up to double the previous peak traffic, as long as we didn’t exceed double our previous peak within 30 minutes. Also, a table could be pre-warmed if necessary. In addition, provisioned capacity mode provided DynamoDB auto scaling, so it was possible to run steady workloads and set upper limits to prevent overuse. The following two figures show write traffic patterns when one-time bulk messages are generated in user_chat and message DynamoDB tables.

The second problem, inter-table load propagation, could be quickly solved because the load of one table didn’t affect other tables in DynamoDB. Therefore, we could improve our architecture to handle flexible traffic while operating the entire service with stability.

To address the third problem, we examined the key features of Channel Talk chat to see if existing PostgreSQL operations could be replaced with DynamoDB. The following screenshot illustrates this process.

The main features of Channel Talk chat are abstracted as follows:

• Chat room (Chat) – There are messages in a chat room
• Participation information (ChatSession) – Information such as the number of unread messages and the last read time in a specific chat room is recorded

The number of unread messages in the participation information is called a ChatBadge. Because a specific user can belong to multiple chat rooms at the same time, the total number of unread messages in multiple chat rooms is called a ManagerBadge.

These components have two requirements that operate separately:

• Atomicity – When a message occurs in each chat room, ChatBadge is atomically increased for receivers other than the sender
• Count – The sum of all ChatBadges that occurred in each chat room becomes ManagerBadge

Previously, because we used PostgreSQL, ChatBadge used atomic operations, and ManagerBadge was just a sum of ChatBadges. However, performing ad-hoc count of records after each user message would not scale and performance is likely to degrade over time as number of chat rooms grew for users. To solve this problem, we decided to use DynamoDB transactions. The following screenshot shows the attributes before the table design change.

The following screenshot shows the attributes after the table design change.

When a message is created in a particular chat room, the following operations are performed on each participant except the sender:

1. Create an UpdateItem that increases the participant’s ChatBadge.
2. Create an UpdateItem that increases the participant’s ManagerBadge.
3. Handle the two UpdateItem operations using a TransactWriteItems API.

This way, we can make sure the sum of ChatBadges becomes ManagerBadge, and we can query each of them with a time complexity of O(1). Of course, this approach can cause transaction conflicts in situations where messages are continuously generated simultaneously, but because most of Channel Talk’s workload is one-on-one conversations for customer consultations, there aren’t many cases where simultaneous messages occur. In addition, when a large number of messages occur simultaneously, we have a procedure to handle them.

As shown in the following figure, when a conflict occurs, we use the exponential backoff retry strategy, and because DynamoDB transactions support idempotence when using the ClientRequestToken parameter, even if the same request is submitted multiple times due to problems such as connection timeouts, we can accurately manage ChatBadge and ManagerBadge within 10 minutes, which solved the third problem.

Lastly, the migration was done by stopping API servers during the early morning hours. Borrowing the segmenting idea of ​​DynamoDB parallel scan, we hashed the PostgreSQL ID and migrated them using the BatchWriteItem API in parallel for each segment using multiple threads.

Conclusion

DynamoDB provides nearly unlimited throughput and storage, schema flexibility, DynamoDB Streams to enable event-driven architectures, and ACID transactions for multiple tables. Since we adopted DynamoDB, Channel Talk has been operating its business by continuously adding new features without large-scale infrastructure changes in the chat business area. We gained the following main advantages using DynamoDB:

• We have been able to continuously add new features without improving the infrastructure or code structure. For example, even when a feature that would rapidly increase the number of messages is released, such as the ability for bots to generate messages based on rules, we have been able to operate the service without infrastructure issues.
• The cost model of DynamoDB, which allows us to pay only for what we use without separate instance costs, makes it straightforward to change the database cost, which was previously a fixed cost, into a variable cost. Because DynamoDB costs are also decreased when traffic decreases, there is no need to modify the architecture to reduce costs even when the business situation is not good, and the same is true when traffic increases.
• As the business area continues to expand, we are solving various problems by utilizing DynamoDB in areas that require horizontal scalability, such as measuring the usage of all Channel Talk customers in addition to chat.

In Part 2 of this series, we discuss how we integrated our solution with other services to solve areas what couldn’t be solved with DynamoDB alone.

About the Authors

TaeHun (Clsan) Yoon, a seasoned professional in the realm of computer engineering, spearheads innovative solutions as the CTO (Chief Technology Officer) at Channel Corp. With a keen focus on enhancing the chatting experience, TaeHun and his team are dedicated to resolving diverse challenges encountered by customers.

Changsoon (CS) Kim is a DevOps Engineer at Channel Corp. He is interested in efficiently resolving issues between development and operations.

Sungbae Park is Account Manager in the AWS Startup Team and a regular member of AWS SaaS TFC (Technical Field Communities). He is currently focusing on helping B2B software startups grow and succeed with AWS. Sungbae previously worked as Partner Development Manager making mutual growth with MSP, SI, and ISV partners.

Hyuk Lee is a Sr. DynamoDB Specialist Solutions Architect based in South Korea. He loves helping customers modernize their architecture using the capabilities of Amazon DynamoDB.

Posted on mongodb google news. Visit mongodb google news

Dwight Merriman, a director at MongoDB , Inc. (NASDAQ:MDB), recently sold 3,000 shares of the company’s Class A common stock. The shares were sold at a price of $269.57 each, resulting in a total transaction value of $808,710. Following this sale, Merriman holds 1,127,006 shares directly. Additionally, Merriman has indirect ownership of 89,063 shares through the Dwight A. Merriman Charitable Foundation and 522,896 shares held in a trust for his children. The sale was conducted under a Rule 10b5-1 trading plan.

In other recent news, MongoDB, Inc. has announced the full redemption of its 0.25% Convertible Senior Notes due in 2026, with the total principal amount being $1,149,972,000. The redemption is scheduled for December 16, 2024, allowing noteholders to convert their notes into MongoDB’s Common Stock before the conversion deadline. This strategic financial move is part of MongoDB’s recent developments.

MongoDB has also reported a 13% year-over-year revenue increase in its second quarter, amounting to $478 million, primarily due to the success of its Atlas (NYSE:ATCO) and Enterprise Advanced offerings. This performance led to an addition of over 1,500 new customers, bringing MongoDB’s total customer base to over 50,700.

In response to these developments, several analyst firms including DA Davidson, Piper Sandler, and KeyBanc Capital Markets have raised their price targets for MongoDB, reflecting confidence in the company’s continued growth. MongoDB’s management anticipates third-quarter revenue to range between $493 million and $497 million, with the full fiscal year 2025 revenue projected to be between $1.92 billion and $1.93 billion. These projections are based on MongoDB’s recent performance and analyst expectations.

While Dwight Merriman’s recent stock sale might raise eyebrows, it’s essential to consider MongoDB’s overall financial health and market position. According to InvestingPro data, MongoDB boasts a substantial market capitalization of $20.82 billion, reflecting its significant presence in the database software market.

InvestingPro Tips reveal that MongoDB holds more cash than debt on its balance sheet, indicating a strong financial position. This liquidity strength is further supported by the fact that the company’s liquid assets exceed its short-term obligations. These factors suggest that despite the insider sale, MongoDB maintains a solid financial foundation.

The company’s revenue growth remains robust, with a 22.37% increase over the last twelve months, reaching $1.82 billion. This growth trajectory aligns with another InvestingPro Tip indicating that net income is expected to grow this year. Moreover, 22 analysts have revised their earnings upwards for the upcoming period, signaling positive sentiment about MongoDB’s future performance.

It’s worth noting that while MongoDB is not currently profitable over the last twelve months, analysts predict the company will turn profitable this year. This expectation, coupled with the company’s strong revenue growth, may explain why it’s trading at a high revenue valuation multiple.

For investors seeking a more comprehensive analysis, InvestingPro offers 11 additional tips for MongoDB, providing a deeper understanding of the company’s financial health and market position.

This article was generated with the support of AI and reviewed by an editor. For more information see our T&C.

Article originally posted on mongodb google news. Visit mongodb google news

Posted on mongodb google news. Visit mongodb google news

Anthosa Consulting has significantly ramped up its network of partners through new alliances with ten technology providers.

The fresh partnerships are with well-known tech vendors in the areas of cloud, data management, observability, and infrastructure automation: AWS, Databricks, Dynatrace, MongoDB, Resourcely, Snowflake, WSO2, Zabbix, Centreon and Zededa.

Chamara Somaratne, the Founder of Anthosa Consulting, said that the series of new partnerships comes in response to growing client demand for support on their digital transformation projects. “Our partnerships with these leading technology providers reinforce Anthosa Consulting’s commitment to delivering client-centric, value-driven solutions.”

“These alliances empower Anthosa Consulting to accelerate digital transformation further and enable AI adoption, assisting clients in modernising their infrastructure and streamlining operations for enhanced competitiveness and agility. It also ensures that we build a robust, end-to-end solution for our clients.”

An overview of the partnerships and their scope:

• AWS: For scalable cloud solutions that support AI and machine learning integration.
• Centreon, Dynatrace, & Zabbix: Comprehensive observability and monitoring solutions that enhance system resilience and improve operational visibility.
• Databricks & Snowflake: Enhancing data processing and analytics capabilities, thereby enabling deeper insights and data-driven decision-making.
• MongoDB: Delivering scalable, flexible database solutions that are essential for modern applications.
• Resourcely: Facilitating secure and compliant infrastructure provisioning.
• WSO2 (Choreo Internal Developer Platform): Driving microservices architecture to modernise legacy systems while supporting flexible, API-first architectures that accelerate digital transformation.
• Zededa: Enabling edge computing and decentralised digital ecosystems, providing organisations with adaptive, scalable infrastructure solutions.

Melbourne-headquartered Anthosa Consulting is a boutique management consulting firm specialising in digital transformation.

Article originally posted on mongodb google news. Visit mongodb google news

Article originally posted on InfoQ. Visit InfoQ

Transcript

Peterson: I’m Erik Peterson. I’m the CTO of a Boston startup called CloudZero. I started that company August 8, 2016, been almost done an 8-year journey. I’ve been building in the cloud, really, since the beginning, since this thing existed. It’s been my love affair. I started in the security industry. Spent a lot of time in that world. When the cloud came around, I just got to work, started building, and I haven’t looked back. There was one big problem that I had personally, which was, all the money that I saw my engineering team spending on that without a care in the world. That bothered me. I decided to solve that problem by starting CloudZero. I want to talk about that problem. I want to talk about how it relates to all of us as engineers. I want to get into the details.

Every Engineering Decision is a Buying Decision

I want to start with this statement. It’s a very important statement, I think, that needs to change how we all think of the world. I want to frame this whole conversation around this, which is that every engineering decision is a buying decision. This is this giant wave that is coming at us that we cannot escape. Our only option is to surf it. Otherwise, it’s going to crush us. Every single person who writes a line of code in the cloud, you are spending money. If you’re traveling and you’re out there and you write an expense report, there will be more scrutiny on that expense report for that $20 lobster dinner, maybe $50 lobster dinner, than there will be for the line of code that spends a million dollars. That’s the challenge that we all have a responsibility to focus on.

I think this problem is an existential threat to the cloud itself. I don’t think the cloud actually is going to be successful. We’ve all heard the naysayers. I love it. I don’t think the cloud is going to happen in our lifetime. It’s still relatively small given all things IT, if it doesn’t make strong economic sense. That’s our responsibility as engineers. Unfortunately, this is where we’re living today. Anybody remember the days before DevOps? You threw it over the wall, you gave it to ops, everything was good, you went home. This is the reality that most of us are living in right now. It works fine for us. It’s finance’s problems now. More importantly, it’s actually our business’s problem. If we can’t build profitable systems, some of the companies that we work for might not exist tomorrow. It’s pretty obvious, this is going to cost you a lot. We’re still trying to figure this out.

What Is a Million-Dollar Line of Code?

What is a million-dollar line of code? My talk in San Francisco at QCon last year focused on just what a million-dollar line of code looks like. I wanted to give you all just a quick example. Anyone want to spot a million-dollar line of code in this little slice of Python? It’s hard to see. It’s pretty obvious, or maybe it’s not. I’ve got some code that’s writing 1000 bytes, sending it to DynamoDB. No problem.

Problem is, there’s a limit. 1024 means if you go over that limit, you’re spending twice as much to make a write operation in DynamoDB. That means, because I’m off by just a few bytes, this code costs twice as much than it should. Easy to make that mistake, very hard to understand the context of that. How do I fix it? Just shrink it down. Get rid of that big long key. Make the timestamp a little bit easier to read. Cut my cost for this code in half. That was an actual million-dollar line of code running in production, running every day, handling billions of transactions, cut in half. That’s the power of what us as engineers have at our fingertips. The question of course, is, how am I going to go about finding these lines of code? It’s a little bit hard. That’s what we’re going to get into.

I’ve got a few axioms that I’m going to try to frame the conversation about. The first one is that only observed costs are accurate. We’ve all tried to estimate costs. We’ve probably used a cost calculator at some point. We’ve gone to the thing on the cloud provider’s website and maybe even did a little math in a spreadsheet. None of that was really accurate, though, when it got to production, was it? First, a little word from our Lord and Savior, from this fine young gentleman, Donald Knuth. He was very, I think, ahead of his time in many ways. If we don’t, many of us have this tattooed on our minds already, premature optimization is the root of all evil. As we go through this, I want us to think about, when does it make sense to actually start thinking about optimizing when trying to find these million-dollar lines of code?

Cloud Cost Challenges

Here are some of the challenges. The majority of cloud computing costs, they come from usage, not infrastructure. I may spin up 100 EC2 instances. Sure, that costs money, but what am I doing with them? What API calls are they making? The same piece of code can be both cheap and bankrupt me. That example I showed you, run once, costs a few pennies, run a billion times, costs a lot. The behavior of my application and how the world is interacting with it matters a lot, and unfortunately, it’s probably just something like 3% of your code that accounts for 90% or 97% of that cost. The reality is that emergent behavior is what rules the cloud. It’s not possible, actually, to predict the cost of a change.

When I have a small change happening by a junior engineer who changes the timestamp to be a couple bytes larger than it should be, our marketing runs a campaign that suddenly sends a million people to my website that I wasn’t expecting, or ops goes in and sets logs to debug, all of these things create downstream effects that I cannot predict. You will never be able to predict it. You can make an educated guess. You can try to forecast what you’re going to spend your money on based on past performance, but it is not indicative of what’s going to happen in the future, and eventually it will be wrong. It will almost certainly bite you at the least appropriate moment.

One of the ways to get around this, and what I’ve seen folks do that makes solid sense, but surprisingly, it’s not implemented everywhere. We track quality. We track performance. We track staging and verify that everything is looking good before we push to production, or maybe you’re like me, I like to test in production. We don’t track the cost of that environment before we pull the trigger. My recommendation for this now that we’re observing our costs, is we pick a threshold, pick a number, up, down by 10%, start somewhere.

Track the before and after. You can get really detailed. Get into the unit metrics. We’re going to talk about that. Get into the cost of the services I’m using: the compute cost, the API transaction volume. How much log data am I producing? Or, just start simple at the beginning. Deploy all that to an isolated account, track the total cost, 24 hours later, if your cost is up by 10%, you have a problem. I’m sure a lot of people have worked with or know Twilio. This is no different than what Twilio does today, to make sure that they keep their margins in check. Engineering team writes a bunch of new code, pushes up their compute cost, that means their cost per API transaction goes up, profit goes down. If those costs exceed your threshold, roll back your changes.

Axiom 2, an observed system is a more cost effective one. I thought of all ways I could throw in a Schrödinger’s cat example into this, but the reality is, and this seems like obvious wisdom, but so few of us are doing it, is that you’ve got to actually pay attention if you’re going to save money. Most engineers are not part of the cost conversation. Finance is in the mix. Maybe they’re not even giving you access. You cannot build a well-architected system in the cloud without access to this information. It’s like they asked you to make the website more performant, they’re not going to tell you how many milliseconds a transaction takes. You’ve got to demand for this if you want to build a cost-effective system.

What Actually Costs You Money?

What actually costs you money? Not so obvious. Somebody turns on debug logging, suddenly you have terabytes of data flowing into logs. Maybe it’s going into CloudWatch if I’m using AWS. Maybe it’s going into Datadog, even more expensive. Maybe it’s going someplace else, to a Slack channel, destroying some poor intern’s soul. It’s possible. All of that costs money. The semi-obvious, API calls cost money. You might be writing a ton of data to S3 and not thinking much about it. The API calls to S3 cost a lot of money. Maybe you have a billion calls just to check to see if the file is there, and you write the file once, that is a very expensive piece of code. If it’s running slow, classic performance problems, of course, you use more compute.

Compute is a commodity. In some cases it’s scarce, particularly if we’re trying to buy those big AI instances these days. Things like reading and writing data, network traffic, even cross-region traffic, all this costs money. Maybe we already knew this, but they’re hard to observe. The thing is, this should be obvious, like I said, but isn’t, is that unobserved systems when you are not paying attention to the costs, are less cost effective. What does that mean, cost effective? There are a couple different ways to think about that, because your cloud bill going up isn’t necessarily a bad thing, if your return on that investment is good. For every dollar I spend, if I make $10 back, you might not be actually worried about spending money. The importance is efficiency, and how good am I serving up the value that my product makes, or builds, or delivers to the world, and what am I getting in return for that?

I’m sure some of you have tried to calculate these things and guessed. Remember what I said earlier? Sooner or later, the emergent properties of the cloud are going to get you. Here’s the proof. This is just one example across a lot of data that we’ve been tracking. The only thing that was introduced into this environment very intentionally was the ability for every single engineer on the team to suddenly know the cost of their actions. When they checked in some code, saw it run, they’d get a little note: cost went up, cost went down. Nothing else changed. Nobody told them what to do. Nobody forced them to do anything. All we did was create visibility. It was a fascinating experiment. From that action alone, we saw costs decrease and level off.

Actually now, what we see is this continues to grow as they go up, but we’re tracking something called unit cost and unit economics, which I’ll talk about. If your finance team says, “Yes, we’re looking at it. We got it all covered”. It doesn’t count. The cost must be observable to the people who are actually building the systems. Imagine every stack trace for any bug that you ever created you had to wait a month before you got it, or somebody gatekeeped it away from you, we would all be doing different jobs. The same is true for cost. The cost must be observable by those building the system.

You might be wondering if this is going to slow us all down. Maybe this is going to destroy our ability to innovate. Maybe premature optimization is the root of all evil, and this is just premature optimization. We’re just piling more work on top of every developer who’s already doing a million things. The truth is, constraints drive innovation. Cost constraints are practically the only constraint that we have today in the cloud. Think about it, thanks to the cloud, we all have infinite scale. I can bring up 1000 machines if I wanted to. I can write a single line of code to do that. I can put it in a Terraform template.

I have infinite scale, but I do not have infinite wallet. Constraints drive innovation. I’m sure some people are familiar with the theory of constraints and other thoughts around this in the world of manufacturing, perhaps. If not, go check it out. Think about some of the hardest systems or most amazing systems that have ever been built in the world under extreme constraints. My favorite is the Apollo 11 Guidance Computer. It’s ridiculously constrained in terms of the resources that it had. I have a billion of those on my wrist right now. It put a person on the moon. Constraints drive innovation, and we’ve lost our sight on these constraints.

The Cloud Innovator’s Dilemma

This is what happens. We’re living this, what I call the cloud innovator’s dilemma. We all rush out, we adopt new technology. We start building rather rapidly on this, until somebody somewhere panics. I like to call that guy Mr. Bill Panic, he’s right there. All innovation halts. This is actually more disruptive to velocity than anything else there is. Who here loves being called out of the flow moment that they’re in to go fix a bunch of bugs, to go deal with build panic? Cost cutting begins, the Dark Ages returns. Stuff starts going crazy. Nobody is building anything cool, and you’re wondering, when am I going to be able to get back to innovation again? The same curve happened in the security industry over a decade ago. If you remember, in the software security world, security was the responsibility of the security team, and that didn’t work out so well over time.

Eventually, companies like Microsoft found themselves getting hacked practically every single day. They were getting tipped over. It was becoming a national security issue. Eventually, Bill Gates had to write the Trusted Computing memo that he wrote back in, I think, 2001, and he said, Microsoft’s going to stop innovating until we solve the security problem. In fact, we’re going to innovate on security. Some people thought that might have gone on for a few months. It went on for two years before a lot of people got back to building new software. That’s how dangerous this dilemma can be, until we get back to innovating, and then the cycle repeats itself.

How do we solve this cloud innovator’s dilemma? Donald Knuth also said that we should not pass up our opportunities in that critical 3%. I think we need to end that cycle and leverage cost as a non-functional requirement. Non-functional requirements are super powerful in the world of software design. They help us understand what our stakeholders want. They help us understand what our business wants. I believe very strongly that cost belongs in that 3% as a design constraint. Those constraints, remember, drive innovation. Think about some of the best systems that have ever been built out there. I know this for a fact from my own experience. One of the reasons I started CloudZero was because very early on in my cloud journey, I went to our CFO. His name was Ed Goldfinger.

It was a perfect name for a CFO. I thought I was living in a James Bond film. I went to Ed and I said, “Ed, I need the company credit card because I’m just slightly smart enough to know that I probably shouldn’t put my credit card number into that box on the AWS website”. He said, “No problem, Erik. We trust you. We hired you guys to go build some cool stuff. Here’s the credit card, but you’ve only got a budget of $3,000”. I thought that was insane. I thought we would spend that in probably 2 seconds. I really didn’t have anything to base my beliefs on, because we were just getting started. I said, challenge accepted. Then we started innovating. What happened is we actually threw out the entire design that we were working on, where we came back and treated it as a non-functional requirement, this is back in 2009 and 2010, and we ended up building probably what to this day I consider a truly cloud native system.

A system that can only exist in the cloud. We love to talk about cloud native as something that can move from place to place. I think cloud native means cloud only. The reason we had to build that way was because we had to leverage the full power of what we had. We had to sequence workloads, and turn things on and off, and be very dynamic about how we were doing it. For that $3,000 we did some pretty amazing things. We built a system that generated millions of dollars in revenue, for $3,000. That kind of efficiency is possible when you embrace it. If that constraint of $3,000 hadn’t existed for me, we would have never done it. We would have gone and spent probably a million dollars building that thing.

That system can never move back to the data center, because it was built only for the cloud. These are the things that prevent build panic. I want all of you to think about runaway cloud cost as a software defect from this point forward. It’s not something that finance has to deal with. It’s your code, your designs, your architecture, the decisions that you made, those buying decisions that are responsible for that cost. Runaway cloud cost is a software defect.

The Possible Side Effects of Cost Constraints

Like I said, there are some constraints here, or there are some side effects. You might need to say goodbye to the data center forever. I know that might sound a little scary, but you’re already locked into something, no matter where you’re going. Lock-in is a lie. You’re building to the lowest common denominator. You’re not embracing the true power. You’re spending a lot of money to do it. This is even true if you use Kubernetes or whatever new container thing comes out tomorrow. You’ve got to embrace some of the managed services and other capabilities that the cloud offers you so that you can get out of the mode of running and building things that you can let somebody else do more efficiently for you.

Give your cloud provider the undifferentiated heavy lifting. You’re probably going to have to do something more than just EC2. I know that might be tough, but think about it, for $3,000 you could build a system that generates millions of dollars in revenue if you embrace it. If you’ve been building in the cloud for a while, your developer velocity might not take a hit, but it probably will. People are going to have to learn new things. It’s a great community, all of us. If you’re new to this, it’ll take a little hit. If it’s not improving, try harder. Squeeze that constraint. Hang on to the reality that constraints drive innovation and will push the team to think harder and faster about what they’re building. I assure you, they will thank you for it. We love solving hard problems.

There’s somebody who agrees with this as well, it’s the CTO of AWS, Dr. Werner Vogels. He first started talking about cost as a positive property of the cloud in 2013. He got up on stage, he gave his keynote. There were five properties of the cloud, one of them was cost goes down. I think at that time, he was thinking the same way, that cost when treated as a constraint, will drive innovation. Unfortunately, him and all of AWS got too excited about lift and shift and all the money they were making, and they forgot this journey. I think it’s horrible. Lift and shift was probably one of the greatest lies that the cloud providers ever told. I don’t blame them. They allowed the conversation to shift away from the amazing things that we could build if we treated it as a more important property of the systems that we were building.

In 2023, Werner, 10 years later, almost to the day, got back up on stage last year at re:Invent, and he shared these thoughts with the rest of the world. I hope it continues down this path. I hope all of the cloud providers continue down this path. I hope all of us realize the importance of this. It really does drive some amazing innovations. We think about a lot of ways where we’re just living in a world where, it works, why would I want to mess with it? Ask ourselves, does it really work for the business? Does it really work for the companies that we’re supporting? Are we building a more profitable system so that we can hire larger engineering teams and build more things? Chances are, if we’re not thinking about it, we probably are not.

That leads me to axiom 4, how do we understand what actually makes sense to the business? Sometimes when we look at our cloud bill, it’s a big number. Yes, I’d like to spend as much as a new Lamborghini every single day. What does that mean? Is that good? Is that bad? Is it scary? Cost is a horrible metric, just a big, scary number. How do I know what’s good? It’s lacking any context. The only reason we know response times down to the millisecond for websites are good or bad is because, maybe Google published their site rankings, and said, if you get your latency down, it’s better. They set the standard. Somebody needs to set the standard. Your businesses set the standard. It’s called a margin. If you’re in the software business, you’re probably wanting to achieve something around 80% on your profit margins. The way you get at that is to understand the unit economics of the products that you’re building.

Back in 2009 and 2010, when I was building that system, I had nothing to go off of in terms of how to think, how we’re going to stay into this budget. I was building a security scanning tool, and I had $3,000. I figured out how many sites I needed to scan, divided it by the budget I had, came up with something that said, you can only spend a few pennies per scan. That’s what drove how we started iterating on that architecture.

Leverage Unit Economics as the Bridge between Finance and Engineering

This is the bridge between you and finance. If you’ve ever seen those people sneaking around, wondering what we’re all spending our money on, and they speak a scary language sometimes. I think engineering and finance are two long-lost siblings. They just grew up in different parts of the world. They speak completely different languages. They’re two very highly technical, specialized disciplines. You have more in common than you actually realize. They’re just as scared of you. They really don’t know what Kubernetes is. They really don’t understand the cloud. They might be trying. What they want are metrics that actually make sense to them in terms of improving the bottom line, because you’ve got these two worlds of data that need to come together if you’re going to build a truly optimized, profitable business.

You all as engineers are the key. What is unit economics anyways? Do I have to go back to school? Do I need to go get an economics degree? Hopefully not. I’m going to give you a crash course. It’s very simple, as we can tell here. It’s a system of using objective measurements to ensure your business is maximizing profit while delivering and developing cloud-based software and services. Perfect. No problem. We’re all ready to go. It’s going to take some data. It’s going to take a little bit of thought about how we calculate this. Let’s get into it. A unit metric is a representation of the rate of change for a demand driver. A demand driver can be something like, I’m DoorDash, and I deliver meals. People want to order meals.

What’s my cost to deliver a meal when I look at my cloud infrastructure? I’m a credit card company. People want to do a transaction. What’s the cost to do a credit card transaction? What’s the cost for someone to sign up? These are demand drivers. These are things that drive our cost. It’s the other side of the equation. The goal of a good unit metric is to present that incremental consumption of some resource in terms of that demand driver. How much money am I consuming to deliver that meal? Looks something like this: consumption on the top, demand driver on the bottom. This is my unit metric. Some great things that we just focus on unit cost: total cloud cost, cloud cost per service, cloud cost per feature, per customer, per microservice.

Break it up. I can take other drivers. How much vCPU compute am I using? How many gigabytes am I storing? I can map that against how many daily active users. I was at the FinOps X conference. FinOps is an emerging discipline that’s growing pretty fast. If you don’t know about it, there’s probably close to 15,000 people now in that community that exploded out of nowhere. It’s designed around building this bridge between finance and engineering. They had their conference in San Diego. I got up on the stage with my good friend Vitor from Duolingo. How many people use Duolingo? They know exactly what their cost per daily active user is. Because we’ve been working together on it, so that they can build a more cost-effective app, when you sit down at night to make sure you freeze that streak.

This is how you need to think about it. $25 million spent on EC2, who cares? It’s a worthless metric. It’s pointless. It’s not helpful. Maybe it’s curious. Maybe it sounds good at parties. If you’re in the business of doing transactions, 25 cents per transaction, when you sell those transactions for $10, is a very profitable business. You can go spend $25 million all day, every day, and no one will care. In fact, everyone will be happy. In fact, finance might even come to you and ask, how can we spend more money? Which means more toys for all of us. That’s what I was thinking back then when I had that impossible challenge. I want to go play with the cloud toys, I’m going to have to make this work. I want to equip all of you to do that. Don’t think about how much money you’re spending. That cost number is not useful. Think about the unit metric for your business. Start looking for the data that’s going to allow you to calculate that.

The last piece I want to leave you all with is that engineering is responsible for the cloud cost. Just like the security industry went through that transformation so many years ago, we all are in the midst of this transformation right now, whether it was the economy, or COVID, or both. That was this cloud world’s Pearl Harbor moment. That was Bill Gates’s Trusted Computing memo that said, we’re going to halt innovation until we figure this security thing out. That was really the before and after for the application security industry. After that, all the companies started to understand, I have to adopt security as part of my software development lifecycle.

Today that’s the way it’s done. There are tools built for all of us to bring security into that development process. We need to bring cost into our development process. We’re very early days. We’re still trying to figure this out. Some of the challenges though, like I said, is that it’s, in many ways, almost impossible to calculate what they’re going to cost until we’ve observed them. I can take a Terraform template or a CloudFormation template, and I could add up all the resources that are going to be created, and I would spit out a number. Until I know what the code and the users are doing that run on that system, I might not have a very accurate estimate of what that all is going to cost. Engineering has to be responsible for cloud costs.

Runaway Cost is a Software Defect

We have to think about runaway cost as a software defect. How many people remember this chart? We’ve been living it since the days of IBM and Raft and Rational and who knows what? Those were good days. Maybe not. It’s been pounded in our heads. The further you get away from the moment the code was created, the harder it is to fix that bug, almost exponentially harder. Maybe it is. It’s a math thing. Now thanks to the cloud, it actually costs a lot more. In 2022, somebody crazy came up with this number, which I have a still hard time believing, but all the bugs that we wrote, all those quality issues that we were responsible for, may have cost the economy $2.4 trillion, if you can believe that.

A lot of that stuff was probably built in the cloud, which means a lot of that stuff spent a lot of money in the cloud. I don’t think that number is even included in here. If you take Gartner’s projection on what the world’s cloud spend was in 2022, which is somewhere around $500 billion. When you take Flexera’s report that says about 30% of all cloud spend is waste, that means about $143 billion should be added to that number. Our software defects caused that to happen. I think it would be really amazing if we all walked around with a number above our heads indicating our total lifetime cloud spend. That’s probably a Black Mirror episode or something. It’s scary.

How many people have a million-dollar number hanging over their head? Anybody want to admit that? How many people have a multimillion-dollar number hanging over their head? How many people have written bugs that spent $30,000? No regrets. That was one of the challenges those years ago when Ed was beating me up at that company. I’d get dragged into his office once we got more of the engineering team engaged, and he would be like, what happened to those good old days when we were just spending $3,000? Because I’d love to say that that story continued, but eventually we started doing lift and shift when we figured out that it costs a lot more. Then I felt like I was going to the principal’s office every week. It was painful.

Sometimes I had to admit to him that it was my code, that single line of code, maybe even a million-dollar line of code, that was responsible for all this. It cut me deep. I wanted everyone on the team to feel that they could be part of making that never happen again. It’s hard, because we need to catch these things sometimes when there’s only a few dollars. You look at it running in staging, it’s only costing $3 to run, but break that out, and that’s where unit economics come in handy. How many users are we expecting? How many transactions are we expecting? What things are we expecting my application to do? Let’s multiply that $3 by whatever that number is, and discover, it might be a few million dollars, that change that we just made, if we let it run. I’ve seen single lines of code cost over $50 million annualized. One line of code. That’s how powerful this is.

It’s hard to find those lines of code because we have to watch them running in the real world. If we’re breaking all these things out, we can zero in on the parts of the system that matter. If we are correlating them to the check-ins that we make and we make it part of our engineering process, then it should be very easy for us to zero in on. In fact, I think it is.

Buy Better or Build Better?

There’s two ways to think about that. Most of us are probably living in an organization, or maybe have been living in an organization where it’s finance-led optimization. Finance-led optimization is really about buying better. This comes from the world of procurement, where we all started. Discount optimizations, buying RIs. Maybe you’re doing this as well. Maybe you have a shared responsibility. Fantastic. A lot of organizations, they’re still learning their way. Budgeting, forecasting, trying to understand what’s going to happen the old way, probably being very frustrated that none of the old ways are really working. What’s more powerful is on the other side, building better.

Let’s write some better code that’s going to improve our product and cost to serve for unit economics. Let’s respond to anomalies when we see a change in the environment that triggers that runaway cost. Let’s address the root cause. Let’s treat it like an incident. Let’s go about reducing waste. We’ve got a love-hate relationship with rightsizing, because a lot of times the opportunity cost is greater than the time spent on it. It’s also a smell, that your architecture, you might still be too in love with that architecture. You might be treating it like pets. The problem is that all of that should be automated, if you’re building the right systems. You shouldn’t have to be thinking about that architecture, so when the next compute architecture comes out, it should be very easy for me to upgrade to it.

That challenge is not solved after the system is deployed. It’s solved during design. You have to realize, everything you build will eventually rot and decay and go away. Another favorite quote of mine from Werner was that everything fails all the time, and in the cloud, it’s guaranteed to happen all of the time. I almost felt like it was a feature, created some evolutionary pressure on the environment in the systems we were building so that we would create more efficient systems, or at least more resilient systems. We’ve still pushed against it. We need to do both of these, because there’s only so much that finance can do. We’ve got to partner with them. We’ve got to go back to that bridge.

Key Takeaways

I’ll give you these five axioms. Take this with you. Every engineering decision is a buying decision. The world has flipped. The challenge for us is to build systems with the least amount of resource consumption now. It’s not just good for what we’re trying to do as businesses, but it’s good for the planet. We used to see optimization as maximizing the use of what I was given in the data center. You gave me 10 servers. If I didn’t use those 10 servers, I was being wasteful. In the cloud, I have to solve a much harder problem. How do I use the smallest amount? It’s a very almost unsolvable problem in some ways.

Questions and Answers

Participant 1: You mentioned how it’s easy to go through and determine the commits that may have caused these issues. How do you handle scenarios where it may take a couple days to figure out the real cost, because it has to go into production, has to get traffic, in that time window, maybe 200 commits have gone in?

Peterson: You’re going to have to calibrate over time. You probably won’t get it right the first time. It’s like old load testing. In the good old days, like, we tested it with this load then we pushed into production. That fell over, and you realized you were synthesizing the wrong load, or you’re doing something that wasn’t testing real-world scenarios. Like anything, you’re going to have to iterate. I like to say that you should commit and iterate over time. The mechanical parts of this is that you really should try to carve out how you’re going to break apart your spend allocated, or attributed to the teams or the products or the features. It’s hard to do this with just tagging. You might need to take other methods and things like that. That’s your first task.

How can I break this out and isolate the system that I’m observing? That might mean you need to stand up and account for it. That might mean you need to tag it specifically. Maybe you have to apply a couple different techniques. Really focus in on that first. Then once you’re really clear on how you’ve been able to isolate that, now you’ve got staging versus production, figure out what moves the needle for it. Then those become your demand drivers that you’re going to simulate against that environment before you push to production. Go through those processes. You got to walk before you run.

Eventually, when you get there, you’ll be able to understand if you should break the build and roll back within 24 hours or less. That 24 hours exists only because the cloud providers themselves typically don’t deliver the data fast enough. One day I hope to break that, we call it the cloud cost sound barrier. We’ll figure out how to break that soon actually.

Participant 2: Are there any services that allow you or help you in tracking cost, the way that you describe them, by units?

Peterson: If you haven’t done any of this, go look at what your cloud provider could give you for free before you get out of the gate. Amazon, Google, Microsoft, they all have a center for FinOps or cloud cost within their products. Sometimes you’ll find certain limitations of what you can do there. It’s the best place to get your feet wet. Then, when you get to the stage where you’re struggling with some of that stuff, I’ll steer you in the right direction. I’ll even tell you to go use a different product if I think it’s better for you.

Participant 3: I’m curious on how the fourth and the fifth axiom correlates, the unit economics and even engineering being responsible. Because, let’s say my company does 20 cents per order, taking the example of DoorDash. If you translate that at a company level, even though an engineer uses some extra cloud compute a day, it wouldn’t really granulate to a cent per order. How do engineers feel responsible for that unit economics?

Peterson: This is where the partnership with finance is probably important, because they can maybe help you get some of that data to build that up. One very important task after you start going down the path of building out some of the metrics that you should be tracking is to understand that those metrics are even correlated to how your company makes money. It’s one thing to think about it as a correlation against cost, but more importantly, correlated against how you make money. You may discover that your company has a horrible pricing model, that your code is unbelievably efficient, but you’re just not selling it correctly, and it’s not even your problem.

It’s a very liberating day because you can go to finance and be like, “We are badass, we got this covered. It’s all those salespeople are just selling it wrong. You got to raise the price”. You got to figure out how it correlates against how you make money, not just against your cost. Some of these metrics are very helpful though, like cost per vCPU, can give you a very quick idea of just how efficiently your code is running against a whole cluster of machines. It’s not a bad metric to start with. Think about how your company makes money.

See more presentations with transcripts

Article originally posted on InfoQ. Visit InfoQ

The Uno Platform has released its latest update, version 5.5, with new features, bug fixes and performance improvements. The new release aims to simplify and speed up the packaging and distribution of cross-platform desktop applications.

Uno platform is an alternative UI platform for building multi-device applications in C# and XAML. It was launched in 2018, after years of internal use by a Canadian company nventive. It allows developers to write applications for Windows, iOS, Android, WebAssembly, macOS and Linux. It is released under Apache 2.0 open-source license on GitHub.

A key focus of the 5.5 release is revamping of the desktop app packaging. In previous releases, Uno had simplified the packaging of mobile apps on iOS and Android, as well as web applications. However, the experience of packaging desktop apps on Windows, macOS and Linux was fragmentary, having to do manual steps for each of the platforms.

In the new release, the dotnet publish command has been extended to detect the TFM (target framework moniker) in the Uno application and to generate executables in each platform format: ClickOnce for Windows, Snap packages for Linux, and .app packages for macOS. In the near future, Uno will support other packaging formats such as dmg, pkg or flatpack.

For example, the command to create an Linux Snap package is:

dotnet publish -f net8.0-desktop -p:SelfContained=true -p:PackageFormat=snap

The command to create an .app package for macOS is:

dotnet publish -f net8.0-desktop -r osx-arm64 -p:SelfContained=true -p:PackageFormat=app

Another big improvement is the replacement of a custom WebAssembly build process and its dependencies by the standard .NET 9 SDK. The WASM apps can now use build enhancements and targets, such as Hot Reload, incremental builds, AOT builds or direct VS Code support. The mobile and desktop apps were already moved to .NET 9 SDK in Uno 5.3.

For developers ready to upgrade to Uno Platform 5.5, a detailed migration guide has been provided.

In response to feedback from both enterprise clients and the broader community, the 5.5 release includes several notable enhancements aimed at making the platform more robust and accessible. Among these improvements are:

• WebView2 control is now available for WebAssembly apps
• New GLCanvasElement control for direct access to OpenGL 3D content with hardware acceleration, leveraging Silk.Net library
• In a similar fashion, new SKCanvasElement integrates with Skia Desktop hardware rendering pipeline, skipping the software generation that happened before
• The ComboBox control new supports IsEditable property for providing user-typed value in the control

The full list of changes for the 5.5 release is available on the UNO GitHub repository.

According to .NET developers’ discussions on social media, the main advantage of the Uno platform over MAUI or Xamarin is being the only cross-platform .NET framework to support writing WebAssembly and Linux applications in C# and XAML.