Author: Steef-Jan Wiggers

Google has unveiled its seventh-generation Tensor Processing Unit (TPU), Ironwood, at Google Cloud Next 25. Ironwood is Google’s most performant and scalable custom AI accelerator to date and the first TPU designed specifically for inference workloads.

Google emphasizes that Ironwood is designed to power what they call the “age of inference,” marking a shift from responsive AI models to proactive models that generate insights and interpretations. The company states that AI agents will use Ironwood to retrieve and generate data, delivering insights and answers.

A respondent in a Reddit thread on the announcement said:

Google has a huge advantage over OpenAI because it already has the infrastructure to do things like making its own chips. Currently, it looks like Google is running away with the game.

Ironwood scales up to 9,216 liquid-cooled chips, connected with Inter-Chip Interconnect (ICI) networking, and is a key component of Google Cloud’s AI Hypercomputer architecture. Developers can leverage Google’s own Pathways software stack to utilize the combined computing power of tens of thousands of Ironwood TPUs.

The company states, “Ironwood is our most powerful, capable, and energy-efficient TPU yet. And it’s purpose-built to power thinking, inferential AI models at scale.”

Furthermore, the company highlights that Ironwood is designed to manage the computation and communication demands of large language models (LLMs), mixture of experts (MoEs), and advanced reasoning tasks. Ironwood minimizes data movement and latency on-chip and uses a low-latency, high-bandwidth ICI network for coordinated communication at scale.

Ironwood will be available for Google Cloud customers in 256-chip and 9,216-chip configurations. The company claims that a 9,216-chip Ironwood pod delivers more than 24x the compute power of the El Capitan supercomputer, with 42.5 Exaflops compared to El Capitan’s 1.7 Exaflops per pod. Each Ironwood chip boasts a peak compute of 4,614 TFLOPS.

Ironwood also features an enhanced SparseCore, a specialized accelerator for processing ultra-large embeddings, expanding its applicability beyond traditional AI domains to finance and science.

Other key features of Ironwood include:

• 2x improvement in power efficiency compared to the previous generation, Trillium.
• 192 GB of high-bandwidth memory (HBM) per chip, 6x that of Trillium.
• 1.2 TBps bidirectional ICI bandwidth, 1.5x that of Trillium.
• 7.37 TB/s of HBM bandwidth per chip, 4.5x that of Trillium.

(Source: Google blog post)

Regarding the last feature, a respondent on another Reddit thread commented:

Tera? Terabytes? 7.4 Terabytes? And I’m over here praying that AMD gives us a Strix variant with at least 500GB of bandwidth in the next year or two…

While NVIDIA remains a dominant player in the AI accelerator market, a respondent in another Reddit thread commented:

I don’t think it will affect Nvidia much, but Google is going to be able to serve their AI at much lower cost than the competition because they are more vertically integrated, and that is pretty much already happening.

In addition, in yet another Reddit thread, a correspondent commented:

The specs are pretty absurd. Shame Google won’t sell these chips, a lot of large companies need their own hardware, but Google only offers cloud services with the hardware. Feels like this is the future, though, when somebody starts cranking out these kinds of chips for sale.

And finally, Davit tweeted:

Google just revealed Ironwood TPU v7 at Cloud Next, and nobody’s talking about the massive potential here: If Google wanted, they could spin out TPUs as a separate business and become NVIDIA’s biggest competitor overnight.

These chips are that good. The arms race in AI silicon is intensifying, but few recognize how powerful Google’s position actually is. While everyone focuses on NVIDIA’s dominance, Google has quietly built chip infrastructure that could reshape the entire AI hardware market if it decides to go all-in.

Google states that Ironwood provides increased computation power, memory capacity, ICI networking advancements, and reliability. These advancements, combined with improved power efficiency, will enable customers to handle demanding training and serving workloads with high performance and low latency. Google also notes that leading models like Gemini 2.5 and AlphaFold run on TPUs.

The announcement also highlighted that Google DeepMind has been using AI to aid in the design process for TPUs. An AI method called AlphaChip been used to accelerate and optimize chip design, resulting in what Google describes as “superhuman chip layouts” used in the last three generations of Google’s TPUs.

Earlier, Google reported that AlphaChip had also been used to design other chips across Alphabet, such as Google Axion Processors, and had been adopted by companies like MediaTek to accelerate their chip development. Google believes that AlphaChip has the potential to optimize every stage of the chip design cycle and transform chip design for custom hardware.

Microsoft has announced the Public Preview of the open-source Azure MCP Server, a new tool designed to enhance the capabilities of AI agents by providing access to Azure resources. The Azure MCP Server allows AI agents to interact with Azure services such as file storage, databases, and logs, and execute CLI commands.

Model Context Protocol (MCP) is an open protocol that standardizes the interaction between AI agents and external resources. The Azure MCP Server implements this protocol, exposing Azure services to AI systems. According to Microsoft, this enables developers to build context-aware agents for their Azure resources. For instance, agents can now query Azure Cosmos DB using natural language, access Azure Storage files, and analyze Azure Log Analytics logs.

The Public Preview of the Azure MCP Server includes support for the following Azure services and tools:

• Azure Cosmos DB (NoSQL): List accounts, databases, containers, and items; execute SQL queries.
• Azure Storage: List accounts and blob containers/blobs; manage blob containers and blobs; list and query tables; get container properties and metadata.
• Azure Monitor (Log Analytics): List workspaces and tables; query logs using Kusto Query Language (KQL); configure monitoring.
• Azure App Configuration: List stores; manage key-value pairs and labeled configurations; lock/unlock settings.
• Azure Resource Groups: List and manage resource groups.
• Azure CLI: Execute commands directly, with full functionality and JSON output.
• Azure Developer CLI (azd): Execute commands directly, supporting template discovery, initialization, provisioning, and deployment.

(Source: Medium blog post)

Brian Veldman concluded in a Medium blog post on the Azure MCP Server:

From now on, I can use the Azure MCP Server to interact with the Azure services within my subscription. This is especially helpful in troubleshooting scenarios, such as analyzing logs.

Microsoft states that this functionality allows agents to operate on Azure services, manage cloud resources, and deploy applications. Yet, more open-source projects, such as the Azure CLI MCP Server, are available on GitHub, leveraging the MCP Server for Azure resources. Julion Dubois, a principal manager, Java Developer Relations, tweets on the mentions the Azure CLI MCP Server:

It’s an MCP server that wraps the Azure CLI, so your LLM can directly send commands to Azure.

In addition, Madni Aghadi states on  via tweet on X:

“MCP is just hype” That’s what I thought until I saw 1000+ MCP servers built since its launch.

Any agent that supports the MCP client pattern, including GitHub Copilot Agent Mode and custom MCP clients, can use the Azure MCP Server.

• GitHub Copilot Agent Mode: The Azure MCP Server can be installed with GitHub Copilot in VS Code. Microsoft recommends combining the Azure MCP Server with the GitHub Copilot for Azure extension for an enhanced development experience.
• Custom MCP Clients/Agents: Agents must adopt the MCP client pattern to interact with the Azure MCP Server. Frameworks like Semantic Kernel can be used to build such agents. Microsoft provides a command (npx -y @azure/mcp@latest server start) to install and execute the server, and notes that the Azure MCP Server should work with any MCP client.

The Azure MCP Server follows similar moves by other cloud providers to enhance AI agent capabilities within their ecosystems:

Lastly, Microsoft plans to enhance the Azure MCP Server with more agent samples, documentation, Microsoft products, Azure service integrations, and additional features.

Google has announced that its Gemini models will be available on Google Distributed Cloud (GDC), bringing its advanced AI capabilities to on-premises environments. The public preview is slated for Q3 2025.

With this move, the company aims to allow organizations to leverage Gemini’s AI while adhering to strict regulatory, sovereignty, and data residency requirements. The company is collaborating with NVIDIA to make this possible by utilizing NVIDIA Blackwell systems, allowing customers to purchase the necessary hardware through Google or other channels.

Sachin Gupta, vice president and general manager of infrastructure and solutions at Google Cloud, said in a NVIDIA blog post:

By bringing our Gemini models on premises with NVIDIA Blackwell’s breakthrough performance and confidential computing capabilities, we’re enabling enterprises to unlock the full potential of agentic AI.

GDC is a fully-managed on-premises (available since 2021) and edge cloud solution offered in connected and air-gapped configurations. It can scale from a single server to hundreds of racks and provides Infrastructure-as-a-Service (IaaS), security, data, and AI services. GDC is designed to simplify infrastructure management, enabling developers to focus on building AI-powered applications, assistants, and agents.

According to Google, bringing Gemini to GDC will allow organizations to use advanced AI technology without compromising their need to keep data on-premises. The GDC air-gapped product already holds authorization for US Government Secret and Top Secret missions, providing high levels of security and compliance.

Keith Townsend stated in a LinkedIn post:

For security-conscious industries like manufacturing, this is a game-changer. Let’s say you’re running a complex OT environment. Machines generate massive volumes of telemetry—temperatures, vibration patterns, run times. With Distributed Gemini Flash, you can deploy lightweight agents on-prem, behind your firewall, to analyze that data in real time.

Gemini models are designed to deliver breakthrough AI performance. They can analyze million-token contexts, process diverse data formats (text, image, audio, and video), and operate across over 100 languages. The Gemini API is intended to simplify AI inferencing by abstracting away infrastructure, OS management, and model lifecycle management. Key features include:

• Retrieval Augmented Generation (RAG) to personalize and augment AI model output.
• Tools to automate information processing and knowledge extraction.
• Capabilities to create interactive conversational experiences.
• Tools to tailor agents for specific industry use cases.

In addition to Gemini, Google highlights that Vertex AI is already available on GDC. Vertex AI is a platform that accelerates AI application development, deployment, and management. It provides pre-trained APIs, generative AI building tools, RAG, and a built-in embeddings API with the AlloyDB vector database.

Lastly, the company also announced that Google Agentspace search will be available on GDC (public preview in Q3 2025). Google Agentspace search aims to provide enterprise knowledge workers with out-of-the-box capabilities to unify access to data in a secure, permissions-aware manner.

AWS has announced the open-source release of AWS Model Context Protocol (MCP) Servers for Code Assistants, a suite of specialized servers designed to enhance AI-powered code assistants with AWS best practices. According to the company, these servers leverage AI to provide context-aware guidance to accelerate development, improve code quality, and ensure adherence to security and cost optimization principles.

The open-source release of MCP Servers for Code Assistants bridges AI-powered coding assistants (like Amazon Q, Claude, and Cursor) and AWS services – it enables these assistants to understand the nuances of AWS, offering intelligent suggestions and automating tasks that would otherwise require manual effort and deep AWS expertise.

As the authors of an AWS blog post on the open source release describe:

Model Context Protocol (MCP) is a standardized open protocol that enables seamless interaction between large language models (LLMs), data sources, and tools. This protocol allows AI assistants to use specialized tooling and access domain-specific knowledge by extending the model’s capabilities beyond its built-in knowledge—all while keeping sensitive data local.

In addition, the blog post outlines the following key benefits:

• Accelerated Development: MCP Servers significantly reduce development time by providing ready-to-use code snippets and configurations based on AWS best practices.
• Enhanced Security: MCP Servers help developers implement secure configurations, ensuring IAM roles, encryption, and security policies align with AWS Well-Architected principles.
• Cost Optimization: The Cost Analysis MCP Server provides insights into AWS pricing, helping developers make informed decisions and avoid unnecessary expenses.
• Access to AWS Knowledge: MCP Servers seamlessly integrate with AWS documentation and knowledge bases, giving AI assistants access to information.
• Infrastructure as Code (IaC): The AWS CDK MCP Server automates the generation of IaC templates, simplifying infrastructure provisioning.

By leveraging AI to provide context-aware guidance, this open-source initiative has the potential to democratize AWS expertise and accelerate the adoption of secure and efficient cloud development patterns.

In a dev.to post, AWS Community Builder Arthur Schneider writes:

In today’s fast-paced tech world, we’re constantly looking for ways to accelerate development processes while improving quality. Especially in the AWS environment, where complexity increases with each new service, we need smarter tools that make our work easier. This is where MCP comes into play – a protocol that fundamentally changes the way we interact with AI models.

Lastly, the GitHub repository or Pypi package manager provides developers with example implementations to get started.

Apache Kafka has reached a significant milestone with the release of version 4.0, a major update that introduces a host of new features and improvements, most notably the default operation in KRaft mode, which, according to Confluent’s documentation, eliminates the dependency on Apache ZooKeeper..

For over a decade, ZooKeeper has served as the backbone of Kafka, and the community has expressed gratitude for its contributions. However, the move to KRaft by default in Kafka 4.0 streamlines deployment and management by removing the need to maintain a separate ZooKeeper ensemble.

(Source: Confluent documentation)

Lalit Moharana, an AWS Community Builder, posted on LinkedIn:

ZooKeeper is stepping aside as Apache Kafka adopts KRaft with the upcoming Kafka 4.0 release, marking the end of a 14-year partnership. This shift simplifies Kafka’s architecture by ditching the separate ZooKeeper system, boosting scalability, and paving the way for a self-sufficient future – all thanks to KRaft’s Raft protocol magic.

In addition:

Why the Change? ZooKeeper’s overhead and limits (think 100,000+ partitions) couldn’t keep up with Kafka’s growth. And:

KRaft Benefits: One system, millions of partitions, faster recovery – Kafka’s ready to soar!

Beyond the architectural shift, Kafka 4.0 brings the general availability of KIP-848, which introduces a next-generation consumer group protocol. This new protocol is designed to dramatically improve rebalance performance, reducing downtime and latency for consumer groups, especially in large-scale environments. By minimizing “stop-the-world” rebalances, Kafka aims to provide a more stable and responsive data streaming experience. The new protocol is enabled by default on the server side, with consumers needing to opt in by setting group.protocol=consumer.

In a Hacker News thread, a respondent commented:

One thing I immediately noticed after switching from SNS/SQS to Kafka was its speed. Messages seem to get sent/received almost immediately.

Furthermore, Kafka 4.0 offers early access to Queues for Kafka (KIP-932). This feature introduces the concept of “share groups” to enable cooperative consumption using regular Kafka topics, effectively allowing Kafka to support traditional queue semantics. While not a direct addition of a “queue” data structure, this enhancement expands Kafka’s versatility, making it suitable for a broader range of messaging use cases, particularly those requiring point-to-point messaging patterns akin to durable shared subscriptions.

In a LinkedIn post, Govindan Gopalan, an AI & Data Engineering Leader at IBM, concluded:

Early queue support (KIP-932) introduces point-to-point messaging, expanding Kafka’s use cases beyond traditional publish-subscribe workflows.

This major release marks a significant step forward in platform modernization. As part of its evolution, Kafka 4.0 has removed APIs deprecated for at least 12 months. Furthermore, it updates the minimum Java requirements, with Kafka Clients and Kafka Streams now requiring Java 11, and Kafka Brokers, Connect, and Tools requiring Java 17. This move encourages the adoption of newer Java features and aligns Kafka with more current technology stacks. The release also updates the minimum supported client and broker versions (KIP-896) and defines new baseline requirements for supported upgrade paths, as detailed in KIP-1124.

AWS has announced the availability of parallel test execution in AWS CodeBuild, a fully-managed continuous integration service. According to the company, this new feature significantly reduces build times by allowing test suites to run concurrently across multiple build compute environments.

The announcement highlights the growing challenge of lengthy test execution times in continuous integration (CI) pipelines as projects become increasingly complex. These long cycles can delay feature delivery, hinder developer productivity, and increase costs.

Thomas Fernandez wrote in a Semaphore blog post on parallel testing:

Parallel testing lets us do more while waiting less. It’s an essential tool to keep sharp and ready so we can always establish a fast feedback loop. 

With parallel test execution in CodeBuild, developers can now configure their build process to split test suites and run them in parallel across multiple independent build nodes. CodeBuild provides environment variables to identify the current node and the total number of nodes, enabling intelligent test distribution. The feature supports a sharding approach with two main strategies:

• Equal distribution: Sorts test files alphabetically and distributes them evenly across parallel environments.
• Stability: Uses a consistent hashing algorithm to maintain file-to-shard assignments even when test files are added or removed.

To enable parallel testing, developers configure the batch fanout section in their buildspec.xml file, specifying the desired level of parallelism. The pre-installed codebuild-tests-run utility is used in the build step to manage test execution and sharding based on the chosen strategy. A sample of a buildspec.yml that shows parallel test execution with Cucumber on a Linux platform looks like:

version: 0.2

batch:
  fast-fail: false
  build-fanout:
    parallelism: 5
    ignore-failure: false

phases:
  install:
    commands:
      - echo 'Installing Ruby dependencies'
      - gem install bundler
      - bundle install
  pre_build:
    commands:
      - echo 'prebuild'
  build:
    commands:
      - echo 'Running Cucumber Tests'
      - cucumber --init
      - |
        codebuild-tests-run 
         --test-command "cucumber" 
         --files-search "codebuild-glob-search '**/*.feature'"
  post_build:
    commands:
      - echo "Test execution completed"

CodeBuild also offers automatic merging of test reports from the parallel executions into a single, consolidated test summary. This simplifies result analysis by providing aggregated pass/fail statuses, test durations, and failure details in the CodeBuild console, via the AWS CLI, or through integration with other reporting tools.

(Source: AWS Documentation)

A demonstration highlighted in an AWS blog post on the feature showed a reduction in total test time from 35 minutes to 6 minutes (including environment provisioning) for a Python project with 1,800 tests when running on ten parallel compute environments.

Sébastien Stormacq, a principal developer advocate at AWS, wrote:

The 1,800 tests of the demo project take one second each to complete. When I run this test suite sequentially, it took 35 minutes to complete. When I run the test suite in parallel on ten compute environments, it took 6 minutes to complete, including the time to provision the environments. The parallel run took 17.9 percent of the time of the sequential run.

This new capability is compatible with all testing frameworks, and the AWS documentation provides examples for popular languages and frameworks like Django, Elixir, Go, Java (Maven), Javascript (Jest), Kotlin, PHPUnit, Pytest, Ruby (Cucumber), and Ruby (RSpec). For frameworks with specific requirements for test file lists, CodeBuild provides the CODEBUILD_CURRENT_SHARD_FILES environment variable, which contains a newline-separated list of test files for the current shard.

Parallel test execution in AWS CodeBuild is available today in all AWS regions where CodeBuild is offered, across all three compute modes: on-demand, reserved capacity, and AWS Lambda compute, with no additional cost beyond the standard CodeBuild pricing for the resources used.

In collaboration with NVIDIA, Microsoft has announced the integration of NVIDIA NIM microservices and the NVIDIA AgentIQ toolkit into Azure AI Foundry. This strategic move aims to significantly accelerate the development, deployment, and optimization of enterprise-grade AI agent applications, promising streamlined workflows, enhanced performance, and reduced infrastructure costs for developers.

The integration directly addresses the often lengthy enterprise AI project lifecycles, extending from nine to twelve months. By providing a more efficient and integrated development pipeline within Azure AI Foundry, leveraging NVIDIA’s accelerated computing and AI software, the goal is to enable faster time-to-market without compromising the sophistication or performance of AI solutions.

NVIDIA NIM (NVIDIA Inference Microservices), a key component of the NVIDIA AI Enterprise software suite, offers a collection of containerized microservices engineered for high-performance AI inferencing. Built upon robust technologies such as NVIDIA Triton Inference Server, TensorRT, TensorRT-LLM, and PyTorch, NIM microservices provide developers with zero-configuration deployment, seamless integration with the Azure ecosystem (including Azure AI Agent Service and Semantic Kernel), enterprise-grade reliability backed by NVIDIA AI Enterprise support, and the ability to tap into Azure’s NVIDIA-accelerated infrastructure for demanding workloads. Developers can readily deploy optimized models, including Llama-3-70B-NIM, directly from the Azure AI Foundry model catalog with just a few clicks, simplifying the initial setup and deployment phase.

Once NVIDIA NIM microservices are deployed, NVIDIA AgentIQ, an open-source toolkit, takes center stage in optimizing AI agent performance. AgentIQ is designed to seamlessly connect, profile, and fine-tune teams of AI agents, enabling systems to operate at peak efficiency.

Daron Yondem tweeted on X:

NVIDIA’s AgentIQ treats agents, tools, and workflows as simple function calls, aiming for true composability: build once, reuse everywhere.

The toolkit leverages real-time telemetry to analyze AI agent placement, dynamically adjusting resources to reduce latency and compute overhead. Furthermore, AgentIQ continuously collects and analyzes metadata—such as predicted output tokens per call, estimated time to following inference, and expected token lengths—to dynamically enhance agent performance and responsiveness. The direct integration with Azure AI Foundry Agent Service and Semantic Kernel further empowers developers to build agents with enhanced semantic reasoning and task execution capabilities, leading to more accurate and efficient agentic workflows.

(Source: Dev Blog post)

Drew McCombs, vice president of cloud and analytics at Epic, highlighted the practical benefits of this integration in an AI and Machine Learning blog post, stating:

The launch of NVIDIA NIM microservices in Azure AI Foundry offers Epic a secure and efficient way to deploy open-source generative AI models that improve patient care.

In addition, Guy Fighel, a VP and GM at New Relic, posted on LinkedIn:

NVIDIA #AgentIQ will likely become a leading strategy for enterprises adopting agentic development. Its ease of use, open-source nature, and optimization for NVIDIA hardware provide a competitive advantage by reducing development complexity, optimizing performance on NVIDIA GPUs, and integrating with cloud platforms like Microsoft Azure AI Foundry for scalability.

Microsoft has also announced the upcoming integration of NVIDIA Llama Nemotron Reason, a powerful AI model family designed for advanced reasoning in coding, complex math, and scientific problem-solving. Nemotron’s ability to understand user intent and seamlessly call tools promises to enhance further the capabilities of AI agents built on the Azure AI Foundry platform.

AWS has announced updates to Amazon EC2 On-Demand Capacity Reservations (ODCRs), introducing split, move, and modify functionalities to improve resource management and cost efficiency. These new features are designed to give users greater control over their reserved EC2 capacity, addressing the dynamic needs of modern cloud deployments.

The split feature enables users to divide existing ODCRs, creating new reservations from unused capacity. This allows for more granular resource allocation, significantly when demand fluctuates. Instead of maintaining a large, underutilized reservation, users can now create smaller, more targeted reservations.

(Source: AWS News blog post)

The move capability allows users to transfer unused capacity slots between existing ODCRs. This optimizes resource utilization across different reservations, preventing wasted capacity and reducing costs. Users can reallocate resources to where they are needed most, improving overall efficiency.

The modify feature allows users to change reservation attributes without disrupting running workloads. Users can adjust the instance quantity, switch between targeted and open reservation types, and modify the reservation’s end date. This eliminates the need to create new reservations for minor adjustments and reduces operational overhead.

“These new capabilities provide you with powerful tools for improved capacity management and resource usage, leading to more efficient operations and cost savings,” states the AWS blog post, highlighting the benefits of these updates. The enhancements aim to improve capacity management, offering greater flexibility and control. By optimizing resource utilization and minimizing disruptions, users can achieve cost savings and improve the overall efficiency of their EC2 deployments.

The company writes in the blog post:

The ability to dynamically adjust and share Capacity Reservations provides the flexibility you need while maintaining the stability necessary for your critical workloads.

While cloud providers like Microsoft Azure and Google Cloud Platform (GCP) offer similar capacity reservation mechanisms, the specific features differ. Azure’s Reserved Virtual Machine Instances (Reserved VMs) and GCP’s Committed Use Discounts (CUDs) provide cost savings for committed compute usage. However, the newly introduced AWS “split, move, and modify” features offer more granular control over reservations than Azure and GCP’s standard offerings.

Confluent Cloud for Apache Flink has introduced Flink Native Inference, Flink search, and built-in ML functions, offering a unified Flink SQL solution. These new features aim to simplify real-time AI development by eliminating workflow fragmentation, reducing costs, and enhancing security.

The new capabilities follow up on the GA release of the service last year. Adi Polak, a director at Confluent, elaborated on the pain points that Flink Native Inference, Flink search, and the built-in ML functions address for enterprise developers building AI applications:

Today, developers must use separate tools and languages to work with AI models and data processing pipelines, leading to complex and fragmented workflows. This results in a data pipeline sprawl, with data spread across many databases and systems. This lack of data integration and orchestration can lead to inefficiencies, increased networking and compute costs, and more development time, making it challenging to scale AI applications and see immediate value from AI investments.

The new features tackle these pain points by offering a unified, Flink SQL-based solution. Flink Native Inference allows any open-source AI model to run directly within Confluent Cloud. Polak explains:

These open-source and fine-tuned AI models are hosted inside Confluent Cloud, so there’s no need to download external files or manage the GPUs to run them. Instead of sending data to a remote model endpoint, Native Inference brings state-of-the-art open-source AI models to customers’ private data, allowing them to build event-driven AI systems using Flink SQL easily.

This approach eliminates network hops, enhances security by keeping data within the platform, and lowers latency for faster inference, which is crucial for real-time AI insights.

Flink search simplifies data enrichment across multiple vector databases, while the built-in ML functions make advanced data science capabilities accessible to a broader range of developers.

Regarding security, cost efficiency, and operational overhead, Polak emphasized the benefits of Confluent’s unified approach:

By unifying data processing and AI workloads, Confluent decreases organizations’ operational overhead by making it easier to develop and deploy AI applications faster. With Confluent Cloud’s fully-managed AI offering, companies can save costs with a hybrid orchestration of data processing workloads, more efficient use of CPUs and GPUs within a pre-defined compute budget, and benefit from having no cloud ingress or egress fees. There is also enhanced security since data never leaves Confluent Cloud, and proprietary data is not shared with third-party vendors or model providers.

According to the company, the new capabilities remove the complexity and fragmentation that have traditionally hindered real-time AI development – reducing the complexity of building real-time AI applications within enterprise environments.

GitHub has launched an AI-powered secret scanning feature within Copilot, integrated into GitHub Secret Protection, that leverages context analysis to improve the detection of leaked passwords in code significantly. This new approach addresses the shortcomings of traditional regular expression-based methods, which often miss varied password structures and generate numerous false positives.

According to a GitHub blog post detailing the development, the system now analyzes the usage and location of potential secrets to reduce irrelevant alerts and provide more accurate notifications critical to repository security. Sorin Moga, a senior software engineer at Sensis, commented on LinkedIn that this marks a new era in platform security, where AI not only assists in development, but also safeguards code integrity.

A key challenge identified during the private preview of GitHub’s AI-powered secret scanning was its struggle with unconventional file types and structures, highlighting the limitations of relying solely on the large language model’s (LLM) initial training data. GitHub’s initial approach involved “few-shot prompting” with GPT-3.5-Turbo, where the model was provided with examples to guide detection.

To address these early challenges, GitHub significantly enhanced its offline evaluation framework by incorporating feedback from private preview participants to diversify test cases and leveraging the GitHub Code Security team’s evaluation processes to build a more robust data collection pipeline. They even used GPT-4 to generate new test cases based on learnings from existing secret scanning alerts in open-source repositories. This improved evaluation allowed for better measurement of precision (reducing false positives) and recall (reducing false negatives).

GitHub experimented with various techniques to improve detection quality, including trying different LLM models (like GPT-4 as a confirming scanner), repeated prompting (“voting”), and diverse prompting strategies. Ultimately, they collaborated with Microsoft, adopting their MetaReflection technique, a form of offline reinforcement learning that blends Chain of Thought (CoT) and few-shot prompting to enhance precision.

As stated in the GitHub blog post:

We ultimately ended up using a combination of all these techniques and moved Copilot secret scanning into public preview, opening it widely to all GitHub Secret Protection customers.

To further validate these improvements and gain confidence for general availability, GitHub implemented a “mirror testing” framework. This involved testing prompt and filtering changes on a subset of repositories from the public preview. By rescanning these repositories with the latest improvements, GitHub could assess the impact on real alert volumes and false positive resolutions without affecting users.

This testing revealed a significant drop in both detections and false positives, with minimal impact on finding actual passwords, including a 94% reduction in false positives in some cases. The blog post concludes that:

This before-and-after comparison indicated that all the different changes we made during private and public preview led to increased precision without sacrificing recall, and that we were ready to provide a reliable and efficient detection mechanism to all GitHub Secret Protection customers.

The lessons learned during this development include prioritizing accuracy, using diverse test cases based on user feedback, managing resources effectively, and fostering collaboration. These learnings are also being applied to Copilot Autofix. Since the general availability launch, Copilot secret scanning has been part of security configurations, allowing users to manage which repositories are scanned.