Month: May 2024

Article originally posted on InfoQ. Visit InfoQ

At the annual Build conference, Microsoft announced the public preview of Azure Compute Fleet, a new service that streamlines the provisioning and management of Azure compute capacity across different virtual machine (VM) types, availability zones, and pricing models to achieve desired scale, performance, and cost.

Compute Fleet enables users to access Azure’s quick compute capacity in a specific region by launching a combination of Standard and Spot VMs at the best price and highest availability. Different pricing models, including Reserved Instances, Savings Plan, Spot, and Pay-As-You-Go (PYG) options, can help users achieve better price-performance ratios.

Microsoft’s Azure principal product manager, Rajeesh Ramachandran, wrote:

Tell us what you need, capacity and instance-wise, for Standard and Spot VM, and Compute Fleet will create both Standard and Spot VMs from a customized SKU list tailored to your workload requirements with the capability to maintain the Spot VM target capacity in the fleet.

Overview of Azure Compute Fleet (Source: Tech Community blog post)

Users can deploy up to 10,000 VMs with a single API, using Spot and Standard VM types together. Aidin Finn, an Azure MVP, tweeted:

Be handy for those times when I need 10k VMs with a single command.

Deploying Compute Fleet with a predefined fleet allocation strategy allows users to optimize their mix of VMs for capacity and cost efficiency or a balance of both. This will help alleviate concerns about the complexity of determining optimal VM pricing, available capacity, managing Spot evictions (instances that are closed down because another customer reserved them), and SKU availability.

Both AWS and Google Cloud Platform (GCP) provide services like Azure Compute Fleet for managing large deployments of virtual machines and instances across different environments. AWS offers AWS Systems Manager Fleet Manager and Amazon EC2 Fleet. The former provides a centralized interface to manage servers across AWS and on-premises environments. At the same time, the latter allows users to launch and manage a collection of EC2 instances across multiple instance types, availability zones, and pricing models through a single API call.

On the other hand, GCP offers Google Cloud Deployment Manager to automate the creation and management of Google Cloud resources.

Currently, Azure Compute Fleet is available in the following regions: West US, West US2, East US, and East US2. The documentation pages provide more details on Azure Compute Fleet.

Article originally posted on InfoQ. Visit InfoQ

At the annual Build conference, Microsoft announced the public preview of Azure Compute Fleet, a new service that streamlines the provisioning and management of Azure compute capacity across different virtual machine (VM) types, availability zones, and pricing models to achieve desired scale, performance, and cost.

Compute Fleet enables users to access Azure’s quick compute capacity in a specific region by launching a combination of Standard and Spot VMs at the best price and highest availability. Different pricing models, including Reserved Instances, Savings Plan, Spot, and Pay-As-You-Go (PYG) options, can help users achieve better price-performance ratios.

Microsoft’s Azure principal product manager, Rajeesh Ramachandran, wrote:

Tell us what you need, capacity and instance-wise, for Standard and Spot VM, and Compute Fleet will create both Standard and Spot VMs from a customized SKU list tailored to your workload requirements with the capability to maintain the Spot VM target capacity in the fleet.

Overview of Azure Compute Fleet (Source: Tech Community blog post)

Users can deploy up to 10,000 VMs with a single API, using Spot and Standard VM types together. Aidin Finn, an Azure MVP, tweeted:

Be handy for those times when I need 10k VMs with a single command.

Deploying Compute Fleet with a predefined fleet allocation strategy allows users to optimize their mix of VMs for capacity and cost efficiency or a balance of both. This will help alleviate concerns about the complexity of determining optimal VM pricing, available capacity, managing Spot evictions (instances that are closed down because another customer reserved them), and SKU availability.

Both AWS and Google Cloud Platform (GCP) provide services like Azure Compute Fleet for managing large deployments of virtual machines and instances across different environments. AWS offers AWS Systems Manager Fleet Manager and Amazon EC2 Fleet. The former provides a centralized interface to manage servers across AWS and on-premises environments. At the same time, the latter allows users to launch and manage a collection of EC2 instances across multiple instance types, availability zones, and pricing models through a single API call.

On the other hand, GCP offers Google Cloud Deployment Manager to automate the creation and management of Google Cloud resources.

Currently, Azure Compute Fleet is available in the following regions: West US, West US2, East US, and East US2. The documentation pages provide more details on Azure Compute Fleet.

Article originally posted on InfoQ. Visit InfoQ

With the recently released PGlite, a Wasm build of Postgres that is packaged into a TypeScript client library, developers can run Postgres queries in the browser with no extra dependencies. PGlite is used for reactive, real-time, local-first apps.

Developers could already run SQLite in the browser with WebAssembly. However, running PostgreSQL in the browser previously required spanning a virtual machine and there was no way to persist data. With PGlite, developers can run PostgreSQL queries in a JavaScript environment and persist data to storage in the file system (Node/Bun) or indexedDB (browser).

PGlite was created to support use cases associated with reactive, real-time, local-first applications. ElectricSQL, which self-describes as a local-first software platform for modern apps with instant reactivity, real-time multi-user collaboration, and conflict-free offline support, uses PGlite to sync on demand with a server. They explain:

Local-first is a new development paradigm where your app code talks directly to an embedded local database and data syncs in the background via active-active database replication. […] ElectricSQL gives you instant local-first for your Postgres. Think of it like “Hasura for local-first”.

PGlite includes parameterized SQL queries, an interactive transaction API, pl/pgsql support, web worker execution, and more. PGlite is 2.6MB gzipped. PGlite can be used in memory in the browser as follows:

const db = new PGlite()
await db.query("select 'Hello world' as message;")

Developers can also persist the database to indexedDB:

const db = new PGlite("idb://my-pgdata");

One Reddit developer emphasized testing as an additional use case for PGlite.

Code testing is a big one for me. I’m currently using in-memory SQLite for tests and I’m often running into differences between SQLite and Postgres (default values, JSON handling, etc). This could allow me to use the real thing without running a full Docker instance.

Another developer said:

No one using Postgres in the cloud is going to use this as an alternative, but there are at least two use cases where this could be very useful:

• You want your app to be local first (snappy, great offline support, etc) but sync data to a server. This is the ElectricSQL use case.

• You want a serious data store in-browser. SQLite via Wasm already fits this use case, but it’s nice to have options.

PGlite is open-source software under the Apache 2.0 license. It is being developed at ElectricSQL in collaboration with Neon. PGlite is still in the early stages and plans to support pgvector in future releases.

Article originally posted on InfoQ. Visit InfoQ

To ensure their Spanner database keeps working reliably, Google engineers use chaos testing to inject faults into production-like instances and stress the system’s ability to behave in a correct way in the face of unexpected failures.

As Google engineer James Corbett explains, Spanner builds on solid foundations provided by machines, disks, and networking hardware with a low rate of failure. This is not enough, though, to guarantee it works correctly under any circumstances, including bad memory or disks corrupting data, network failures, software errors, and so on.

According to Corbett, using fault-tolerant techniques is key to masking failures and achieving high reliability, including checksums to detect data corruption, data replication, using the Paxos algorithm for consensus, and others. But to make sure all of them work as expected, you need to exercise those techniques and make sure they are effective. This is where chaos testing comes into play, consisting of deliberately injecting faults into production-like instances at a much higher rate than they would arise in a production environment.

We run over a thousand system tests per week to validate that Spanner’s design and implementation actually mask faults and provide a highly reliable service. Each test creates a production-like instance of Spanner comprising hundreds of processes running on the same computing platform and using the same dependent systems (e.g., file system, lock service) as production Spanner.

The faults that are injected belong to several categories, including server crashes, file faults, RPC faults, memory/quota faults, and Cloud faults.

Server crashes may be produced by sending a SIGABRT signal at any time to trigger the recovery logic, which entails aborting all distributed transactions coordinated by the crashed server, forcing all clients accessing that server to fail over to a distinct one, and using a disk-based log of all operations to avoid losing any memory-only data.

File faults are injected by intercepting all file system calls and randomly changing their outcome, for example by returning an error code, corrupting the content on a read or write, or never returning to trigger a timeout.

Another area where a similar approach is followed is interprocess communication with RPC. In this case, RPC calls are intercepted to inject delays, return error codes, and simulate network partitions, remote system crashes, or bandwidth throttling.

Coming to memory faults, the Spanner team focuses on two specific behaviors: simulating a pushback state, whereby a server becomes overloaded and clients start redirecting their requests to less busy replicas, and leaking enough memory so the process is killed. Similarly, they simulate “quota exceeded” errors, whether they come from disk space, memory, or flash storage per user.

Injecting Cloud faults aims to test abnormal situations related to the Spanner API Front End Servers. In this case, a Spanner API Front End Server is crashed to force client sessions to migrate to other Spanner API Front End Servers and make sure this does not entail any impact for clients besides some additional latency.

Finally, Google engineers also simulate an entire region becoming unreachable due to several possible causes including file system or network outages, thus forcing Spanner to serve data from a quorum of other regions according to the Paxos algorithm.

Thanks to this approach based on a fault-tolerant design coupled with continuous chaos testing, Google efficiently validates Spanner’s reliability, concludes Corbett.

Article originally posted on InfoQ. Visit InfoQ

OpenAI recently announced the latest version of their GPT AI foundation model, GPT-4o. GPT-4o is faster than the previous version of GPT-4 and has improved capabilities in handling speech, vision, and multilingual tasks, outperforming all models except Google’s Gemini on several benchmarks.

The “o” in GPT-4o stands for “omni,” reflecting the model’s multi-modal capabilities. While previous versions of ChatGPT supported voice input and output, this used a pipeline of models: a distinct speech-to-text to provide input to GPT-4, followed by a text-to-speech model to convert GPT-4’s text output to voice. The new model was trained end-to-end to handle audio, vision, and text, which reduces latency and gives GPT-4o access to more information from the input as well as control over the output. OpenAI evaluated the model on a range of benchmarks, including common LLM benchmarks as well as their own AI safety standards. The company has also performed “extensive external red teaming” on the model to discover potential risks in its new modalities. According to OpenAI: 

We recognize that GPT-4o’s audio modalities present a variety of novel risks. Today we are publicly releasing text and image inputs and text outputs. Over the upcoming weeks and months, we’ll be working on the technical infrastructure, usability via post-training, and safety necessary to release the other modalities. For example, at launch, audio outputs will be limited to a selection of preset voices and will abide by our existing safety policies. We will share further details addressing the full range of GPT-4o’s modalities in the forthcoming system card.

OpenAI gave a demo of the GPT-4o and its capabilites in their recent Spring Update livestream hosted by CTO Mira Murati. Murati announced that the new model will be rolled out to free users, along with access to features such as custom GPTs and the GPT Store which were formerly available only to paid users. She also announced that GPT-4o would be available via the OpenAI API and claimed the model was 2x faster than GPT-4 Turbo, with 5x higher rate limits.

In a Hacker News discussion about the release, one user noted: 

The most impressive part is that the voice uses the right feelings and tonal language during the presentation. I’m not sure how much of that was that they had tested this over and over, but it is really hard to get that right so if they didn’t fake it in some way I’d say that is revolutionary.

OpenAI CEO Sam Altman echoed this sentiment in a blog post:

The new voice (and video) mode is the best computer interface I’ve ever used. It feels like AI from the movies; and it’s still a bit surprising to me that it’s real. Getting to human-level response times and expressiveness turns out to be a big change. The original ChatGPT showed a hint of what was possible with language interfaces; this new thing feels viscerally different. It is fast, smart, fun, natural, and helpful.

Along with GPT-4o, OpenAI released a new MacOs desktop app for ChatGPT. This app supports voice mode for conversing with the model, with the ability to screenshots to the discussion. OpenAI has also launched a simplified “look and feel” for the ChatGPT web interface.

Article originally posted on InfoQ. Visit InfoQ

JEP 467, Markdown Documentation Comments, has been promoted from Proposed to Target to Targeted for JDK 23. This feature proposes to enable JavaDoc documentation comments to be written in Markdown rather than a mix of HTML and JavaDoc @ tags. This will allow for documentation comments that are easier to write and read in source form.

This update’s primary goal is to simplify the process of writing and reading documentation comments in Java source code. By allowing Markdown, which is known for its simplicity and readability, developers can avoid the complexities associated with HTML and JavaDoc tags. Existing documentation comments will remain unaffected, ensuring backward compatibility.

Furthermore, the update extends the Compiler Tree API, allowing other tools that analyze documentation comments to handle Markdown content effectively.

It’s important to note that this update does not include automated conversion of existing documentation comments to Markdown syntax. Developers must manually update their documentation to take advantage of the new feature.

Java documentation comments traditionally use HTML and JavaDoc tags, a practical choice in 1995 but has since become less convenient. HTML is verbose and challenging to write by hand, especially for developers who may not be familiar with it. Inline JavaDoc tags, such as {@link} and {@code}, are cumbersome and often require referencing documentation for proper usage.

Markdown, in contrast, is a lightweight markup language that is easy to read and write. It supports simple document structures like paragraphs, lists, styled text, and links, making it a suitable replacement for HTML in documentation comments. Additionally, Markdown allows the inclusion of HTML for constructs that it does not directly support, providing flexibility while reducing complexity.

Consider the following JavaDoc comment for java.lang.Object.hashCode written in the traditional format:

/**
 * Returns a hash code value for the object. This method is
 * supported for the benefit of hash tables such as those provided by
 * {@link java.util.HashMap}.
 * 

 * The general contract of {@code hashCode} is:
 * 

 * 
Whenever it is invoked on the same object more than once during
 *     an execution of a Java application, the {@code hashCode} method
 *     must consistently return the same integer, provided no information
 *     used in {@code equals} comparisons on the object is modified.
 *     This integer need not remain consistent from one execution of an
 *     application to another execution of the same application.
 * 
If two objects are equal according to the {@link
 *     #equals(Object) equals} method, then calling the {@code
 *     hashCode} method on each of the two objects must produce the
 *     same integer result.
 * 
It is not required that if two objects are unequal
 *     according to the {@link #equals(Object) equals} method, then
 *     calling the {@code hashCode} method on each of the two objects
 *     must produce distinct integer results.  However, the programmer
 *     should be aware that producing distinct integer results for
 *     unequal objects may improve the performance of hash tables.
 * 
 *
 * @implSpec
 * As far as is reasonably practical, the {@code hashCode} method defined
 * by class {@code Object} returns distinct integers for distinct objects.
 *
 * @return  a hash code value for this object.
 * @see     java.lang.Object#equals(java.lang.Object)
 * @see     java.lang.System#identityHashCode
 */

This comment can be written in Markdown as follows:

/// Returns a hash code value for the object. This method is
/// supported for the benefit of hash tables such as those provided by
/// [java.util.HashMap].
///
/// The general contract of `hashCode` is:
///
///   - Whenever it is invoked on the same object more than once during
///     an execution of a Java application, the `hashCode` method
///     must consistently return the same integer, provided no information
///     used in `equals` comparisons on the object is modified.
///     This integer need not remain consistent from one execution of an
///     application to another execution of the same application.
///   - If two objects are equal according to the
///     [equals][#equals(Object)] method, then calling the
///     `hashCode` method on each of the two objects must produce the
///     same integer result.
///   - It is _not_ required that if two objects are unequal
///     according to the [equals][#equals(Object)] method, then
///     calling the `hashCode` method on each of the two objects
///     must produce distinct integer results.  However, the programmer
///     should be aware that producing distinct integer results for
///     unequal objects may improve the performance of hash tables.
///
/// @implSpec
/// As far as is reasonably practical, the `hashCode` method defined
/// by class `Object` returns distinct integers for distinct objects.
///
/// @return  a hash code value for this object.
/// @see     java.lang.Object#equals(java.lang.Object)
/// @see     java.lang.System#identityHashCode

Markdown documentation comments are indicated using /// at the beginning of each line instead of the traditional /** ... */ syntax. This helps to avoid conflicts with embedded /* ... */ comments in the examples, which is increasingly common in documentation comments.

The Markdown parser used in this implementation is the CommonMark variant, with enhancements to support linking to program elements and simple GFM (GitHub Flavored Markdown) pipe tables. JavaDoc tags can still be used within Markdown documentation comments, ensuring that existing JavaDoc features are retained.

The parsed documentation comments are represented by com.sun.source.doctree package in the Compiler Tree API. To handle uninterpreted text, a new type of tree node, RawTextTree, is introduced, with a new tree-node kind, DocTree.Kind.MARKDOWN, indicating Markdown content. The implementation leverages the commonmark-java library to transform Markdown to HTML.

The introduction of Markdown support for JavaDoc comments marks a significant improvement in Java’s documentation capabilities, making it more accessible and easier to maintain. This change is expected to enhance developer productivity and improve the overall readability of Java documentation.

Article originally posted on InfoQ. Visit InfoQ

An Australian superannuation fund manager, UniSuper, using Google Cloud for an Infrastructure-as-a-Service (IaaS) contract, found it had no disaster recovery (DR) recourse when its entire infrastructure subscription was deleted.

UniSuper had previously migrated its VMware-based hardware infrastructure from two data centers to Google Cloud, using the Google Cloud VMware Engine. As part of its private cloud contract, UniSuper had its services and data duplicated across two Google Cloud regions. However, this regional separation was effectively virtual because both regions lost copies due to an internal Google error. There was no external disaster recovery facility in place.

This led to an outage that affected over 620,000 UniSuper fund members, leaving them unable to access their superannuation accounts for over a week.

Although UniSuper had backup systems in place, the deletion affected both geographies, causing a complete loss of data. Fortunately, UniSuper had additional backups with another provider, which helped minimize data loss and speed up restoration.

In a joint statement by Google and UniSuper:

This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.

Yet, not everyone is convinced that it’s a one-of-a-kind issue; Daniel Compton, a software developer focusing on Clojure and large-scale systems, dedicated a blog post on the occurrence and concluded:

Given how little detail was communicated, it is difficult to make a conclusive statement about what happened, though I personally suspect UniSuper operator error was a large factor. Hopefully, APRA, Australia’s superannuation regulator, will investigate further and release a public report with more details.

And a respondent on a Hacker thread stated:

I really would like to hear the actual story here since it is basically impossible. It actually was “Google let customer data be completely lost in ~hours/days.” This is compounded by the bizarre announcements – UniSuper putting up TK quotes on their website, which Google doesn’t publish and also doesn’t dispute.

Furthermore, given the additional backups, a respondent on the same Hacker News commented:

Interestingly, the Australian financial services regulator (APRA) requires companies to have a multi-cloud plan for each of their applications. For example, a ‘company critical’ application needs to be capable of migrating to a secondary cloud service within four weeks.

I’m not sure how common this regulation is across industries in Australia or whether it’s common in other countries as well.

And lastly, a respondent on a Reddit thread wrote:

Customer support just isn’t in Google’s DNA. While this could have happened on any provider, this happens far more often on Google.

This story is a classic reminder of the rule of 1: 1 is 0, and 2 is 1. Thank goodness they could recover from a different provider.

Article originally posted on InfoQ. Visit InfoQ

It hasn’t even been two years since OpenAI announced ChatGPT which is the first mainstream Large Language Model from a generative pre-trained transformer to be made available to the public in a way that is very easy to use.

This release triggered lots of excitement and activity from Wall Street to the White House. Just about every fortune 500 company and tech start up is trying to figure out how to capitalize on LLMs. The developer ecosystem is ablaze with supporting tools and infrastructure such as Lang Chain that accelerates the integration of existing applications with LLMs.

At a recent tech conference in Denver, Eric Evans (creator of Domain-Driven Design) “encouraged everyone to start learning about LLMs and conducting experiments now, and sharing the results and learnings from those experiments with the community.”

This article documents my personal contribution to that effort. This research is not based on any requirements or direction from any of my employers past or present.

The Experiment

I decided to come up with an experiment that I could perform on each of the more prevalent LLMs that are available in order to make comparisons between them and to explore the boundaries of what LLMs are capable of at least for the near term. I am not interested in or even worried about how LLMs will replace coders. You still need experienced developers when using LLMs because you need to be particularly vigilant about reviewing the suggestions. I am more interested in how LLMs can help coders become more productive by automating the more time consuming and menial yet still very important parts of writing code. I am referring to unit tests, of course.

Many will claim that unit tests are not a good place for LLMs because, in theory, with Test Driven Development you write the tests first and then the code afterwards. I have had experience in a fair number of companies where the unit tests were almost an afterthought and the amount of code being covered by the tests was directly proportionate to the amount of time remaining in the sprint. If coders could write more unit tests faster with LLMs, then there would be more code coverage and higher quality. I can already hear the TDD die-hards up in arms. Sorry but that is just the cold hard truth of it. Besides, how are you going to mock the external dependencies in the tests without first knowing the internal details of the implementations?

I have an open source repository where I implement identical microservices in various programming languages and tech stacks. Each implementation includes access to MySql fronted by Redis.  I put each microservice on the same load test where I collect and analyze the performance data in order to make comparisons. I took a service class from the Java on Spring Boot implementation and scooped out all but three of the public, routable methods. Then I took the unit test code and removed all but one of the unit tests. I left the imports, setup and the annotation-based dependency injection in place. In the prompting, I ask for the other two unit tests to be generated. The entire prompt is 250 lines (about 1300 words) in length. Why did I use the Java on Spring Boot stack? There is a lot of code readily available online, without licensing restrictions, which was used as some of the training data for the LLMs. Spring Boot’s approach is heavily annotation-based which requires deeper understanding than simply studying the code itself.

I have approached this problem like it was a scientific experiment but it is not a very good one. An experiment is valuable only if it is reproducible but all of these technologies under evaluation are constantly evolving and the owning organizations spend a lot of money innovating on these products, and hopefully improving them. It is highly unlikely that you would arrive at the same results were you to conduct the same experiment today. Nevertheless, I parked a copy of the prompt that I used here just in case you wished to try.

OpenAI’s ChatGPT

It was the early days of ChatGPT, at the end of 2022, that brought transformer-based Large Language Models and OpenAI into the media spotlight. Of course I had to include ChatGPT in this experiment. There were a lot of rough patches in those early days: a small context window, low quality output, prompt forgetting, confident hallucinations. It’s in a much better place now. Like all of the other technologies evaluated here, I used the free version. There are concerns around the revealing of proprietary information via prompt leaking when using these commercial LLMs. That is why I based the experiment on open source. There is nothing proprietary to leak. Prompt leaking is unavoidable because your prompts are used to fine-tune the LLM which, over time, improves the quality of its answers in the future.

How did ChatGPT perform? It did okay. The explanation of the results was concise and accurate. The output was useful but it did have bugs, especially in the dependency injection and mocking areas. The test coverage was just okay. The unit test code had assertions for individual properties, not found, and not null. Even though there were bugs, I still considered the output to be useful because I felt that it would have taken me more time to type the code myself than to fix the bugs from the generated code.

Amazon CodeWhisperer

The next technology that I evaluated in this way was Amazon CodeWhisperer. This is a plugin for Visual Studio Code. The experience is basically a better statement completion. You start typing and it finishes the line. Sometimes it completes a block of code. You then choose to either accept or reject the proposed change. Most of the time, I accepted the change then made whatever corrections were required to fix any bugs in the generated code. I felt the most productive with CodeWhisperer.

I believe that CodeWhisperer is similar in approach to Github Copilot which I did not evaluate because that costs money whereas CodeWhisperer was free. When it comes to the internals of CodeWhisperer, Amazon keeps their cards close to the vest. It’s probably more than just an LLM but it does have that LLM feel to it. I suspect that CodeWhisperer was phoning home all the time because the IDE would freeze up often. You can disable CodeWhisperer which would result in the IDE becoming responsive again.

Code Llama

Ollama is an open source platform that allows you to build and run LLMs. It makes available a fair number of open source models that have already been pre-trained, including Meta’s Code Llama model. There is a lot of interest in these open source LLMs. Noted venture capitalist Bill Gurley identified Llama 2 (included in Ollama’s library of models) as OpenAI’s biggest threat at a summit in UCLA last year. Remember that prompt leaking issue that I mentioned earlier? Because you are hosting Ollama on VMs that are directly under your control, there is little possibility of prompt leaking.

Although it is not required, you are definitely going to want to run this on a system with a reasonably powered GPU. I don’t have a GPU on my personal laptop, so I provisioned a machine learning linux with CUDA system on an a2-highgpu-1g VM with a nvidia-tesla-a100 (312 TFLOPS) from the Google Cloud Platform to run the tests. More specifically, I used the codellama:34b model. From the meta blog that introduced this model, “Code Llama is a code-specialized version of Llama 2 that was created by further training Llama 2 on its code-specific datasets, sampling more data from that same dataset for longer. Essentially, Code Llama features enhanced coding capabilities, built on top of Llama 2.” I ran the same test with codellama:70b on a NVIDIA A100 40GB and did see a slight improvement in the code coverage of the resulting generated code. Provisioning that instance cost $3.67 per hour at the time of this writing.

The results were a little bit worse than ChatGPT but not too bad. There were compile errors, missing packages and imports, and mocking and dependency injection bugs. With the 34b model, the only code coverage was to assert not null. With the 70b model, that was replaced with an assertion that what was returned from the service call matched what got injected in the mock of the underlying DAO call. There is no explanation or online references included in the results with Code Llama. The generated output did include code comments that did not add much value.

Google Gemini

The last LLM that I performed this experiment on was Gemini which is Google’s rebranding of Bard. Like Code Llama, the generated output neglected to include the package statement or the imports which were available in the input. That was easy to fix. I am not sure if that was a mistake or simply a different interpretation of what I was asking for. Like all of the chat-based technologies, the output also had similar bugs with the dependency injection and mocking code. The code coverage was a little bit better as there were also tests for both cache hit and cache miss. The explanation was slightly better than ChatGPT and it cited the source that it used the most, which was not the open source repo where all of the code in the experiment came from. This was useful the same way that ChatGPT was useful. It took less time to fix the bugs than to code the two methods from scratch.

Conclusion

It can be quite challenging to obtain accurate and reliable numeric measurements for something as subjective as software quality. The following table attempts to summarize these findings in a numeric and therefore comparative way. The Myers Diff Algorithm is used to measure the number of lines of code added and deleted (a modified line is counted as both an add and a delete) needed to fix the bugs in the generated unit test code (because you are definitely going to have to fix the generated code). The Jacoco Code Coverage is the percentage of instructions (think Java byte code) covered by the unit tests divided by the total number of instructions (both covered and missed).

LLM-Based Generation of Unit Tests Results Summary By the Numbers

From these experiments, it became quite obvious that there was no Artificial General Intelligence present in the generation of any of this unit test code. The lack of any professional level comprehension of annotation based dependency injection and mocking made it clear to me that there was nothing deeply intelligent behind the figurative curtain. Quite simply, an LLM encodes a large number of documents tokenizing the input and capturing the context of that input based on its structure in the form of a transformer based neural network with a large number of weights known as a model.

When asked a question (such as a coding assignment), the model generates a response by predicting the most probable continuation or completion of the input. It considers the context provided by the input and generates a response that is coherent, relevant, and contextually appropriate but not necessarily correct. In that way, you can think of LLMs as a kind of complex, contextual form of search (albeit way more sophisticated than an Inverse Document Frequency or PageRank weighted stemmed term based search over skip lists that you find in web based search engines or Lucene circa 2020).

What I did find in the LLMs included in this experiment was a really good code search capability that turned out to be useful to an experienced developer. What was a disappointment for me with ChatGPT, Ollama, and Gemini was that I have been conditioned to expect human intelligence on the other side of a chat window. However, I have no such conditioning with statement completion. CodeWhisperer didn’t disappoint not because the AI was better but because the user experience did a better job at managing my expectations.

What’s Next?

There are also a few corporate concerns that may need to be addressed before adoption of unit test code completion style generative AI can exit the experimental phase.

I have already discussed prompt leaking. That should be a big issue for corporations because a lot of the corporation’s code will need to get included in the prompting and most corporations typically view their code as proprietary. If your prompting doesn’t go back to a model instance that is shared with other corporations, then you don’t have to worry about prompt leaking with other corporations. One option is to run the LLM locally (such as Ollama) which requires that every developer works on a machine with a sufficiently powered GPU. Another option is to subscribe to a single tenant, non-shared version of ChatGPT or Gemini. Yet another option is to turn off prompt driven fine tuning of the model entirely. The third option is currently available but not the second option.

The other concern is around cost. I suspect that generative AI pricing today is focused on increasing market share and does not yet cover all the costs. In order to transition from growth to profitability, those prices are going to have to increase. That NVIDIA A100 40GB I mentioned in the Code Llama section above costs around $10,000-ish today. There is also the question of power consumption. There is ongoing innovation in this area but typically, GPUs consume about three times as much power as CPUs. The single tenant approach is more secure but also more expensive because the vendor cannot benefit from economies of scale outside of building the foundation model. The improvement in productivity was only marginal. Cost is going to factor into the equation of long-term usage.

As I mentioned earlier, this space is moving fast. I did most of these experiments in early 2024. Since then, Amazon Code Whisperer got its own enterprise upsell. There are some early versions of both ChatGPT and Gemini fueled plugins for both IntelliJ and VS Code. Meta has released Llama 3 and, yes, it’s already available in Ollama. Where will we be this time next year? In terms of product placement, strategic positioning, government regulation, employee disruption, vendor winners and losers, who can say? Will there be AGI? Nope.

Article originally posted on InfoQ. Visit InfoQ

The Airbnb Tech Blog recently detailed how Airbnb enhances accessibility for users with vision difficulties. Through careful implementation of text resizing guidelines, Airbnb maintains web content, functionality, and a good user experience even as the text font size is doubled.

The Web Content Accessibility Guidelines are a set of standards and recommendations to make web content more accessible for individuals with disabilities. Mild visual disabilities are a fairly common occurrence. The CDC estimates the prevalence of the issue to be 3 out of 5 in America. Furthermore, as many Reddit users mention, not only users with a visual disability may reach out to text resizing capabilities of their browsers (or website, when available):

Anyone with a visual disability is likely to have a “large font size” option set in their browser. [lightmatter501]
And yes, people scale the font size with keyboard or mouse buttons. Because of accessibility mostly. [throwtheamiibosaway]
For some pages, I view the browser at 110% or 125% zoom since my monitor is 3440×1440 and sometimes the font is too small to read. I guess that still counts as accessibility. [Reddit user]
I just do it if I want to lean back in my chair for a while, or if I’m showing someone something, or if I’m sharing my screen on Zoom. [superluminary]
I change the minimum font size because of only one particular website I like to visit. But I am young and still have good eyes. [lontrachen]

The WCAG 1.4.4 Resize Text (Level AA) guideline requires that except for captions and images of text, text can be resized without assistive technology up to 200 percent without loss of content or functionality. Airbnb’s blog post details how they use font scaling as a complement to zoom scaling. Font scaling ties to the ability to increase or decrease text font size without necessarily affecting non-text elements of the page. Using browser zooming capabilities on the other hand scales all web content proportionally, which may lead to a suboptimal experience for some users.

The core idea consists of using em and rem CSS units instead of px units. px units are fixed and do not vary with the user-preferred font size. rem units on the other hand are relative to the font size of the root element. The root element defaults to 16px in many browsers, so 1rem is often equal to 16px. Setting font sizes with rem units is a good idea because it is designed to adapt to the user’s browser preferences. em is also a relative unit of measurement that unlike rem is relative to the font size of the parent element or the font-size of the nearest parent with a defined font size.

The blog article explains:

The choice between em and rem units often comes down to the level of control and predictability required for font scaling. While em units can be used, they can lead to cascading font size changes that may be difficult to manage, especially in complex layouts. In contrast, rem units provide a more consistent and predictable approach to font scaling, as they are always relative to the root element’s font size. […]

In the case of Airbnb, the team decided to prioritize the use of rem units specifically for font scaling, rather than scaling all elements proportionally.

The blog article additionally goes into detail about spreading the corresponding design choices across the entire codebase (which uses two different CSS-in-JS systems), ensuring designers and developers adopt the new approach, and solving cross-platform issues (e.g., Safari on mobile). Airbnb deemed the experience successful:

Choosing font scaling as the product accessibility strategy brought about a range of significant benefits that notably enhanced our platform’s overall user experience. Making that change using automation to convert to rem units made this transition easier. When looking at our overall issues count after these changes were site-wide, more than 80% of our existing Resize Text issues were resolved. Moreover, we are seeing fewer new issues since then.

Developers are invited to refer to the full article and review the detailed technical explanations and demos that are provided.

Article originally posted on InfoQ. Visit InfoQ

Subscribe on:

Apple Podcasts
Google Podcasts
Soundcloud
Spotify
Overcast
Podcast Feed

.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.