Author: Renato Losio

The security team at Datadog recently disclosed a security issue on AWS where non-production endpoints were used as an attack surface to silently perform permission enumeration. AWS has since remediated these specific bypasses.

According to the announcement, Datadog identified two new scenarios in June 2023 for bypassing AWS CloudTrail: using certain non-production endpoints with API actions that access account-level information, and using API calls that generate multiple events in CloudTrail. Nick Frichette, staff security researcher at Datadog, explains:

We determined that non-production AWS API endpoints could be used for permission enumeration without logging to CloudTrail. Since our initial public disclosure of this technique, we have collaborated closely with AWS to illustrate how adversaries could leverage this method to stealthily assess the privileges of compromised credentials.

The research highlights that attackers can exploit misconfigurations and vulnerabilities in these non-production endpoints, often overlooked in security measures, to gain unauthorized access, escalate privileges, and potentially compromise production environments.

Since the issue was reported to AWS, the cloud provider has remediated two specific bypasses, releasing a fix last September for the CloudTrail bypass in AWS Cost Explorer (ce:GetCostAndUsage) and the one in Route 53 (route53resolver:ListFirewallConfigs). The AWS Security Outreach team requested delaying the publication of Datadog’s findings until additional mitigation processes were rolled out. Corey Quinn, chief cloud economist at The Duckbill Group, comments:

AWS Security dragged its feet for 11 months on this vulnerability disclosure. What the hell is going on over there?

Frichette emphasizes instead the importance of securing all endpoints, including those considered low-risk or used for testing and development, to prevent security breaches:

While this specific example is no longer vulnerable, it is important to note that there are thousands of non-production endpoints, any number of which could exhibit similar behavior.Aside from bypassing CloudTrail, non-production endpoints have an additional potential use case when it comes to defense evasion: event source obfuscation.

The cloud provider acknowledges the vulnerability:

For isolated non-production endpoints that do not log to CloudTrail but are otherwise callable with normal credentials and exhibit normal IAM permission behavior, AWS considers the CloudTrail logging bypass of such endpoints also to be a security issue.

At the same AWS highlights that not every endpoint must be remediated:

Non-production endpoints that have access to production resources but generate CloudTrail events that do not match the events generated by the standard endpoint will not be remediated unless it is unclear what service and operation is involved.

Along with the announcement, Datadog released a video documenting how the vulnerabilities were discovered. This is not the only disclosure from Datadog this year; the recent article “Amplified exposure: How AWS flaws made Amplify IAM roles vulnerable to takeover” covers a vulnerability in AWS Amplify that exposed IAM roles associated with projects.

InfoQ Dev Summit Munich is a two-day in-person software development conference for senior software engineers, architects, and team leaders in the Bavarian capital on September 26th and 27th. The sessions will cover critical topics such as generative AI and platform engineering, with use cases from the German automotive, banking, and telecommunication industries.

For practitioners by practitioners, the conference is designed for senior software engineers, software architects, and team leaders in Europe. What can an attendee expect in Munich? Thanks to 22 technical talks from senior software practitioners, the conference provides actionable insights and practical advice on today’s developer priorities with sessions ranging from machine learning to security, from site reliability engineering to scaling Java applications.

Generative AI and bringing machine learning models to production will be major themes of the two days. Olalekan Elesin, engineering director at HRS Group and AWS Machine Learning Hero, will show how to elevate the developer experience with generative AI capabilities on the cloud, while Andrey Cheptsov, founder of dstack, will explain how to leverage open-source LLMs for production. Ines Montani, CEO at Explosion and core developer of spaCy, will demonstrate instead how to take LLMs out of the black box:

I’ll show some practical solutions for using the latest state-of-the-art models in real-world applications and distilling their knowledge into smaller and faster components that you can run and maintain in-house.

Performance for system and software architects will be another central topic, with the speed and efficiency of WebAssembly showcased by Danielle Lancashire, principal software engineer at Fermyon. Gunnar Morling, the developer behind the One Billion Row Challenge, will discuss the tricks employed by the fastest solutions:

Parallelization and efficient memory access, optimized parsing routines using SIMD and SWAR, as well as custom map implementations are just some of the topics which we are going to discuss.

The 1BRC went viral within the Java community earlier this year, with results showing that Java can process a file with one billion rows in two seconds. Thanks to sessions by Johannes Bechberger, a JVM developer at SAP, and Markus Kett, CEO at MicroStream, Java developers will also learn how to build a fast firewall with eBPF and create ultra-fast in-memory database applications.

While the conference in Munich shares the same format as the InfoQ Dev Summit Boston, it will spotlight themes of local significance, with speakers presenting use cases from German automotive, banking, and telecommunication industries.

With attendees and speakers from around the world, all presentations and conference activities at the first InfoQ conference in the DACH region will be held in English.

The topics of the two keynote sessions will be announced soon. Join us and your fellow senior software practitioners in Munich this September!

AWS has recently announced the general availability of Amplify Gen 2, a “code-first developer experience” that enables TypeScript and JavaScript developers to build and deploy fullstack applications on AWS. Since its public preview at the re:Invent conference, Amplify Gen 2 has added new features, including support for TypeScript functions with environment variables, storage support, and custom domains.

AWS Amplify helps developers host web applications and build and connect to a cloud backend. The new version allows developers to write requirements such as data models, business logic, and authorization rules in TypeScript, with the cloud infrastructure automatically provisioned without explicit infrastructure definitions. Rene Brandel, principal product manager – technical at AWS, and Ali Spittel, head of developer advocacy and education at AWS Amplify, write:

With Amplify Gen 2, every part of your app’s cloud backend is defined in TypeScript. Need an Auth backend? TypeScript. Data backend? TypeScript. Storage backend? TypeScript. Everything is defined in TypeScript.

According to AWS, the new release includes a code-first developer experience that avoids the friction of CLI tools and enables AI assistance from services like Amazon Q Developer. It provides tools and features to help developers configure and integrate various AWS services to support use cases such as user authentication, real-time data, and file storage. In the announcement of the preview, Nikhil Swaminathan, head of product at AWS Amplify, explained the main difference between the first version and the new one:

The first generation of the tooling offered a tooling-first experience, using a CLI/Console-based interactive workflow to create a backend. Gen 2 transitions to a code-first DX, allowing developers to succinctly express app requirements like data models, business logic, and authorization rules in TypeScript.

Originally launched in 2017, AWS Amplify has evolved to include the Amplify CLI, a command line tool to help developers build app backends, Amplify Studio, a visual interface, and Amplify Hosting, a service to deploy server-side rendered apps, static apps, and Single Page Applications (SPAs). Brandel and Spittel describe the main benefit of the new DX:

TypeScript is the popular programming language that spans across the spectrum from frontend to backend. The beauty of the TypeScript community is that your options are effectively endless. You can compose and recompose, pick and choose, adjust, modify, and iterate on “just the right stack” for your use case.

In the article “AWS Amplify in 2024 is not the Amplify you grew up with“, Michael Liendo senior developer advocate at AWS, explains:

Amplify Gen 2 in your CLI and code editor also brings enhancements to the Hosting platform as well! For starters, it’s now possible to have wildcard subdomains on your domain name. This, along with enhanced server-side rendering support means it’s now possible to have true multi-tenant applications hosted on Amplify.

In a separate article, the cloud provider describes the progression of full-stack web development and shows the development evolution with AWS Amplify over the past few years. Allen Helton, ecosystem engineer at Momento, writes in his newsletter:

I tried this out the other day and can say with confidence – this is an extremely easy way to build full-stack applications. I love how easy the team has made it to build small to medium-sized apps.

The general availability includes integration guides for AI/ML services (Bedrock, Translate, Polly, and Rekognition) and Auth enhancements such as support for multiple OIDC providers, and user groups, and granting access to other AWS resources.

AWS has recently announced the general availability of Amplify Gen 2, a “code-first developer experience” that enables TypeScript and JavaScript developers to build and deploy fullstack applications on AWS. Since its public preview at the re:Invent conference, Amplify Gen 2 has added new features, including support for TypeScript functions with environment variables, storage support, and custom domains.

AWS Amplify helps developers host web applications and build and connect to a cloud backend. The new version allows developers to write requirements such as data models, business logic, and authorization rules in TypeScript, with the cloud infrastructure automatically provisioned without explicit infrastructure definitions. Rene Brandel, principal product manager – technical at AWS, and Ali Spittel, head of developer advocacy and education at AWS Amplify, write:

With Amplify Gen 2, every part of your app’s cloud backend is defined in TypeScript. Need an Auth backend? TypeScript. Data backend? TypeScript. Storage backend? TypeScript. Everything is defined in TypeScript.

According to AWS, the new release includes a code-first developer experience that avoids the friction of CLI tools and enables AI assistance from services like Amazon Q Developer. It provides tools and features to help developers configure and integrate various AWS services to support use cases such as user authentication, real-time data, and file storage. In the announcement of the preview, Nikhil Swaminathan, head of product at AWS Amplify, explained the main difference between the first version and the new one:

The first generation of the tooling offered a tooling-first experience, using a CLI/Console-based interactive workflow to create a backend. Gen 2 transitions to a code-first DX, allowing developers to succinctly express app requirements like data models, business logic, and authorization rules in TypeScript.

Originally launched in 2017, AWS Amplify has evolved to include the Amplify CLI, a command line tool to help developers build app backends, Amplify Studio, a visual interface, and Amplify Hosting, a service to deploy server-side rendered apps, static apps, and Single Page Applications (SPAs). Brandel and Spittel describe the main benefit of the new DX:

TypeScript is the popular programming language that spans across the spectrum from frontend to backend. The beauty of the TypeScript community is that your options are effectively endless. You can compose and recompose, pick and choose, adjust, modify, and iterate on “just the right stack” for your use case.

In the article “AWS Amplify in 2024 is not the Amplify you grew up with“, Michael Liendo senior developer advocate at AWS, explains:

Amplify Gen 2 in your CLI and code editor also brings enhancements to the Hosting platform as well! For starters, it’s now possible to have wildcard subdomains on your domain name. This, along with enhanced server-side rendering support means it’s now possible to have true multi-tenant applications hosted on Amplify.

In a separate article, the cloud provider describes the progression of full-stack web development and shows the development evolution with AWS Amplify over the past few years. Allen Helton, ecosystem engineer at Momento, writes in his newsletter:

I tried this out the other day and can say with confidence – this is an extremely easy way to build full-stack applications. I love how easy the team has made it to build small to medium-sized apps.

The general availability includes integration guides for AI/ML services (Bedrock, Translate, Polly, and Rekognition) and Auth enhancements such as support for multiple OIDC providers, and user groups, and granting access to other AWS resources.

During the recent Google Next ’24 conference, Google unveiled Axion, its first custom Arm-based CPUs designed for data centers. Utilizing the Arm Neoverse V2 CPU architecture, the new processor will be available to customers later this year.

According to the cloud provider, Axion will deliver 30% better performance than the fastest general-purpose Arm-based instances available in the cloud. Additionally, it promises up to 50% better performance and up to 60% better energy efficiency than comparable current-generation x86-based instances. Amin Vahdat, engineering fellow and VP at Google, writes:

Axion is underpinned by Titanium, a system of purpose-built custom silicon microcontrollers and tiered scale-out offloads. Titanium offloads take care of platform operations like networking and security, so Axion processors have more capacity and improved performance for customer workloads. Titanium also offloads storage I/O processing to Hyperdisk, our new block storage service that decouples performance from instance size and that can be dynamically provisioned in real time.

Even if Google claims “giant leaps in performance for general-purpose workloads” no details or benchmarks have been provided. Google is the latest of the cloud hyperscalers developing an Arm-based custom processor, with Axion now competing with AWS Graviton, already at its fourth generation, and the recently announced Microsoft Cobalt. Matthew Kimball, technology analyst, writes:

So, the three largest cloud providers (AWS, Microsoft Azure, Google Cloud) have all become first-party silicon designers. And of course, Oracle Cloud is utilizing Ampere at scale. If you had any doubts about Arm’s standing in the datacenter CPU market – put them to rest.

Axion is built on the standard Armv9 architecture and instruction set and Google promises that customers will be able to use Axion in many managed services including Compute Engine, Kubernetes Engine, Dataproc, Dataflow, and Cloud Batch. Vahdat adds:

Google also has a rich history of contributions to the Arm ecosystem. We built and open sourced Android, Kubernetes, Tensorflow and the Go language, and worked closely with Arm and industry partners to optimize them for the Arm architecture.

On Hacker News, user soulbadguy comments:

Another interesting challenge for intel (and AMD to a lesser extent). Between the share of compute moving to AI accelerator (and NVIDIA), and most cloud providers having in-house custom chips, I wonder how Intel will position itself in the next 5 to 10 years.

Virtual machines based on Axion processors will only be available in preview in the coming months. User mtmail adds:

Eight testimonials and it’s clear the companies haven’t been able to test anything yet.

While the new processors are not yet available to customers, Google claims that several internal services, including BigTable, Spanner, BigQuery, Blobstore, Pub/Sub, Google Earth Engine, and the YouTube Ads platform, have already been migrated to Arm-based servers. No pricing information has been provided.

The creators of DBOS have recently introduced DBOS Cloud, a transactional serverless application platform tailored for TypeScript developers. With all state information stored in a highly available DBMS, this new platform assures transactional serverless computing, offering reliable execution alongside so-called “time travel” capabilities.

Dubbed as “the world’s first cloud-native operating system” and a “database alternative to Kubernetes”, DBOS (DataBase oriented Operating System) implements operating system services in SQL, operating atop a high-performance distributed, transactional, partitioned fault-tolerant database. Michael Stonebraker, computer scientist and a Turing Award laureate, writes:

The idea for DBOS (DataBase oriented Operating System) originated 3 years ago with my realization that the state an operating system must maintain (files, processes, threads, messages, etc.) has increased in size by about 6 orders of magnitude since I began using Unix on a PDP-11/40 in 1973. As such, storing the OS state is a database problem. Also, Linux is legacy code at the present time and is having difficulty making forward progress. For example, there is no multi-node version of Linux, requiring people to run an orchestrator such as Kubernetes.

According to the authors, DBOS Cloud automatically logs every step an application takes and each change it makes in the database. Two distinct features of DBOS Cloud are reliable execution and time travel: in the event of an interruption to the code running on a DBOS program, it automatically resumes from the point of interruption without re-executing any previously completed work. Stonebraker adds:

Providing such guarantees yourself is months of work, but in DBOS, they’re built into every program (…) You can step through past executions to reproduce rare bugs and even run new code against historical state.

Furthermore, a “time travel debugger” enables developers to replay any DBOS Cloud trace locally on a laptop, observe past application states, and test code changes. In the future, the team plans to release a time travel functionality for disaster recovery, enabling developers to roll back an application and its data to any previous state. Jeremy Daly, CEO and founder of Ampt, comments:

What makes this super interesting is that the project was founded by Dr. Mike Stonebraker (he created Ingress, PostgreSQL, and VoltDB) and Matei Zaharia, the creator of Apache Spark.

Peter Zaitsev, founder at Percona and open source advocate, agrees but adds:

Mike Stonebraker does not seem to be a big fan of Open Source for his companies in recent years.

While the team has released an open-source DBOS TypeScript SDK, unlike Ingress, PostgreSQL, VoltDB, or Apache Spark, DBOS itself is not open-source. In a thread on Hacker News, Peter Kraft, co-founder at DBOS, explains:

Under the hood of our cloud platform (mostly Go + SQL), we’re building on ideas from the academic project to provide new features like reliable execution/time travel, but like a good OS, we want to hide that complexity from users.

A free tier and a programming guide covering idempotency and workflow execution on the serverless computing platform are now available. The free tier offers fixed resources per application (a Firecracker microVM with 512 MB of RAM and 1 vCPU) that scale to zero when not in use.

Redis has recently announced a change in their license by transitioning from the open-source BSD to the more restrictive Server Side Public License (SSPLv1). The move has promptly led to a fork initiated by former maintainers and reignited discussions surrounding the sustainability of open-source initiatives.

Starting with Redis 7.4, all subsequent versions will be dual-licensed under the Redis Source Available License (RSALv2) and SSPLv1, foregoing distribution under an approved open-source license. Rowan Trollope, CEO of Redis, explains:

The new source-available licenses allow us to sustainably provide permissive use of our source code (…) Under the new license, cloud service providers hosting Redis offerings will no longer be permitted to use the source code of Redis free of charge.

Started in 2009 by Salvatore Sanfilippo, Redis has emerged as the most popular in-memory storage solution, used as a distributed, in-memory key–value database and cache. With the adoption of the more restrictive SSPL license created by MongoDB, Redis will not be an open-source project anymore. In a follow-up article to address some of the concerns, Trollope and Yiftach Shoolman, co-founder and president at Redis, address some of the concerns and announce new features:

We’re going back to our roots as the world’s fastest real-time data platform and are proud to announce the acquisition of Speedb, the world’s fastest data storage engine. Over the past two years, we’ve been working directly with Speedb, integrating it as the default storage engine in the Redis Enterprise auto-tiering functionality launched recently in version 7.2.

Reflecting on Redis’s journey, Khawaja Shams, co-founder and CEO at Momento, and Tony Valderrama, head of product at Momento, argue that “Redis did not create Redis” and “how Garantia Data pulled off the biggest heist in open source history.”

The licensing transition has raised concerns in the community and Madelyn Olson, principal engineer at Amazon ElastiCache and formerly a member of the Redis open source governance, writes:

I’ve gotten together with various former Redis contributors and we’ve started working on a fork. We are all unhappy with the license change and are looking to build a new truly open community to fill the void left by Redis. Come join us!

Similar changes in licenses by ElasticSearch triggered official forks from AWS, with the cloud provider recently building GLIDE, a new Redis client. To provide clarity on the new project, Olson clarifies:

Is this an AWS fork? AWS employs me, but this is just me trying to keep the continuity with the community. AWS is aware of what I’m doing and is preparing their own response.

Werner Vogels, CTO at Amazon, adds:

I am excited about the actions Madelyn Olson and other core Redis maintainers are taking. BTW, this is Madelyn taking action, not an official AWS announcement. She should get serious credit for her bias for action. Expect more news soon.

The new license applies to the upcoming 7.4 release, with all previous versions continuing to be open-source under the BSD license.

AWS has recently announced free egress traffic for customers leaving the cloud and withdrawing their data from the AWS infrastructure. This initiative follows the guidelines of the European Data Act and is designed to help customers switch to alternative cloud providers or on-premises data centers.

Cloud providers usually offer free data ingress into the cloud, but costs associated with data transfer out to the internet (DTO) can be substantial: previously, AWS provided 100 GB per month free egress from AWS regions to the internet as part of the Free Tier but any data beyond this allocation incurred charges. With the recent announcement, customers now have the option to request free DTO for additional data by contacting support. Upon approval of the request, AWS will issue temporary credits based on the total volume of data stored across AWS services. Sébastien Stormacq, principal developer advocate at AWS, explains:

It’s necessary to go through support because you make hundreds of millions of data transfers each day, and we generally do not know if the data transferred out to the internet is a normal part of your business or a one-time transfer as part of a switch to another cloud provider or on-premises.

Under mounting pressure from European regulators, AWS is not the only provider introducing a waiver on DTO charges. Earlier this year, Google introduced a comparable option, while Microsoft announced this week free DTO when leaving Azure. Stormacq adds:

The waiver on data transfer out to the internet charges also follows the direction set by the European Data Act and it is available to all AWS customers around the world and from any AWS region.

Credits apply exclusively to DTO charges associated with moving away from the cloud but there is no requirement to close an account or migrate all hosted workloads to qualify. Furthermore, the waiver does not apply to data out from services like CloudFront, Direct Connect, Snow Family, or Global Accelerator. Gergely Orosz, author of the The Pragmatic Engineer newsletter, comments:

Europe gets a lot of flak for how they regulate more and more of tech (in Europe, of course). Regulation sets new rules, often ones that companies wouldn’t follow if not required. AWS just dropped their atrocious egress fees, globally, because of an EU regulation.

Corey Quinn, chief cloud economist at The Duckbill Group, disagrees:

This is almost entirely done for optics; if someone’s debating moving a workload off of AWS (this happens less frequently than the tech press would have you believe) they aren’t stopping because the egress fee is expensive; it’s ~3 months of storing the data in S3. Instead, the fee hurts when customers are doing their usual business and sending traffic to customers, not “when they’re leaving the cloud.” This change does nothing to fix that core pain, but it may look like it does to regulators if they’re not on the ball.

AWS has published a FAQ page with further details on how customers can apply for the waiver. In a separate announcement, AWS explains how it supports the Fair Software Licensing Principles for customers changing provider.

Recently, a former employee of F5 and main contributor of the Nginx project announced the fork Freenginx of the popular web server. The new project was started to address a security dispute and wants to be a drop-in replacement of Nginx, run by developers rather than corporate entities. Maxim Dounin, formerly principal software engineer at F5, provides insights into the fork:

Unfortunately, some new non-technical management at F5 recently decided that they know better how to run open-source projects. In particular, they decided to interfere with the security policy Nginx has used for years, ignoring both the policy and the developers’ position.

Originally developed by Igor Sysoev and currently maintained by F5, Nginx is open-source software for web serving, reverse proxying, caching, load balancing, and media streaming. According to the Web Server Survey, even two decades after its initial launch, Ngnix remains the leading web server serving 23.21% of all sites. In a popular thread on Hacker News, user sevg points out:

Worth noting that there are only two active “core” devs, Maxim Dounin (the OP) and Roman Arutyunyan. Maxim is the biggest contributor that is still active. Maxim and Roman account for basically 99% of the current development. So this is a pretty impactful fork.

In his announcement on the nginx-devel mailing list, Dounin highlights the initial dispute that prompted Nginx to issue a security patch addressing two critical vulnerabilities. He adds:

I am no longer able to control which changes are made in Nginx within F5, and I no longer see Nginx as a free and open-source project developed and maintained for the public good. As such, starting from today, I will no longer participate in Nginx development as run by F5. Instead, I am starting an alternative project, which is going to be run by developers, and not corporate.

Freenginx is not the first fork and drop-in replacement for Nginx: Angie was created by other Russian Nginx developers when F5 left Russia in 2020 and it is currently managed by the Russian company Web Server. Diogo Baeder, lead backend developer at DeepOpinion, comments:

Nginx is an incredible software and platform, but I was wondering if it wouldn’t be time to just bite the bullet and create a more modern solution based on Rust. Having a solution that follows a similar model, “understands” the Nginx configuration language, and reaches similar performance levels, but with the memory safety and wide community adoption of Rust could lead to an amazing new project – perhaps even doing to Nginx what Nginx did to Apache HTTP.

Vincentz Petzholtz, network engineer and architect, is less optimistic and adds:

Sometimes a fork is all you can do when a project is taken into a difficult direction. In the end, the users will vote with the adoption and install-base.

The first release version is Freenginx-1.25.4 under the same BSD license as Nginx. Dounin has provided access to a read-only Mercurial repository, discarding for now the option to migrate to GitHub. The project started a new developer mailing list.

The recent update to Amazon CloudWatch Logs introduces support for account-level subscription filtering. With this enhancement, developers can now access a real-time feed of CloudWatch Logs from all logs groups and have it delivered to a single destination for further processing.

The implementation of a single account-level subscription filter enables customers to deliver real-time log events that are ingested into Amazon CloudWatch Logs to a Kinesis Data Stream, Amazon Kinesis Data Firehose, or AWS Lambda for custom processing, analysis, or redirection to alternative destinations. It is possible to set an account-level subscription policy that includes a subset of log groups in the account.

All the logs that are sent to a receiving service through an account-level subscription policy are base64 encoded and compressed as gzip files. Designed to reduce the overhead of managing large and complex AWS deployments, the account-level subscription filter applies to both existing log groups and any future log groups that match the configuration. Jeremy Daly, CEO and founder of Ampt, comments:

This is a dream come true for those of us who wrestle with tens (if not hundreds) of log group subscription filters.

The put-account-policy API can be used to set the CloudWatch Logs account-level subscription. For example, using the AWS CLI, the following command sends all log data to the helloWorld Lambda function, excluding the group names LogGroupToExclude1 and LogGroupToExclude2.

aws logs put-account-policy 
	--policy-name "ExamplePolicyLambda" 
	--policy-type "SUBSCRIPTION_FILTER_POLICY" 
	--policy-document '{"DestinationArn": "arn:aws:lambda:region:123456789012:function:helloWorld", "FilterPattern": "Test", "Distribution": "Random"}' 
	--selection-criteria 'LogGroupName NOT IN ["LogGroupToExclude1", "LogGroupToExclude2"]' 
	--scope "ALL"

Source: AWS documentation.

Regardless of the chosen destination, AWS stresses the importance of evaluating upfront the volume of log data that will be generated to avoid throttling. Developers should make sure that the Kinesis Data Firehose stream or Lambda function can handle the volume, or that the Kinesis Data Streams has enough shards. With Kinesis Data Streams, throttled deliverables are retried for up to 24 hours and then dropped.

The cloud provider warns as well about the risk of infinite recursive loops with subscription filters, triggering large increases in ingestion billing. The team provides advice on recursion prevention:

To mitigate this risk, we recommend that you use selection criteria in your account-level subscription filters to exclude log groups that ingest log data from resources that are part of the subscription delivery workflow (…) When excluding log groups, consider the following AWS services that produce logs and may be a part of your subscription delivery workflows: Amazon EC2 with Fargate, Lambda, AWS Step Functions, and Amazon VPC flow logs that are enabled for CloudWatch Logs.

Referring to the new features added at re:Invent, and the recent announcement that CloudWatch alarms now support AWS Lambda functions as an action for state changes, Ran Isenberg, principal software architect at CyberArk, notes:

CloudWatch is on a roll lately and is making up lost ground in comparison to third-party observability tools.

CloudWatch Logs Account-level Subscription Filter is available in all AWS commercial regions except Israel and Canada West.

Each AWS account can create one account-level subscription filter.