Author: Olimpiu Pop

Recently, multiple languages have begun supporting compilation to WebAssembly (Wasm), allowing developers to build real polyglot systems. Chicory is a Wasm interpreter for the JVM with zero native dependencies and can run on any JVM. As wazero in the Go ecosystem, Chicory promises developers to safely interact with libraries written in any language supported by the Wasm ecosystem.

The initiators of the project, Benjamin Eckel, chief technical officer at Dylibso, and Andrea Peruffo, principal software engineer at RedHat, motivated their decision to build a WebAssembly interpreter on top of the JVM with the following statement:

Eckel & Peruffo: Wasm was born on the web, hence for safety reasons it has a sandboxed memory model which prevents modules from reading memory or jumping to code outside their scope. By default, Wasm will not allow you to access system resources such as files or networks. Also, it has no concepts like objects or heap meaning it can run low-level languages very efficiently. This makes Wasm an ideal runtime for running untrusted/third-party code written in various languages.

Chicory is similar to Graal’s WebAssembly implementation, except that it only requires a library as a jar, and doesn’t have native dependencies. Presumably, it should be possible to run it on any JVM, GraalVM included. Peruffo points to wazero as inspiration – Chicory’s “distant cousin from the Go ecosystem”.

To get started, just add the chicory dependency to a project, followed by loading the Wasm file and its instantiation:

    
   import com.dylibso.chicory.runtime.ExportFunction;
   import com.dylibso.chicory.wasm.types.Value;
   import com.dylibso.chicory.runtime.Module;
   import com.dylibso.chicory.runtime.Instance;
   import java.io.File;
     
   // point this to your path on disk
   Module module = Module.builder(new File("./factorial.wasm")).build();
   Instance instance = module.instantiate();
    

The Module class, the module instance, is just the “inert” code while the Instance class, the instance, is the Wasm virtual machine that loaded the code and can be run. Developers can invoke a function exported by the module.

   
   ExportFunction iterFact = instance.export("iterFact");
   Value result = iterFact.apply(Value.i32(5))[0];
   System.out.println("Result: " + result.asInt()); // should print 120 (5!)
   

When coding, developers should be aware of a couple of the Wasm particularities: the methods might return multiple results and it supports just basic integer and float primitives. More complex types can be passed by transferring pointers. Using the low-level API, to provide a string, developers can do the following:

   import com.dylibso.chicory.runtime.Memory;  
   Memory memory = instance.memory();
   String message = "Hello, World!";
   int len = message.getBytes().length;
   // allocate {len} bytes of memory, this returns a pointer to that memory
   int ptr = alloc.apply(Value.i32(len))[0].asInt();
   // We can now write the message to the module's memory:
   memory.writeString(ptr, message);

Chicory provides the Memory class which allows developers to allocate, read and write different values to the Wasm program’s memory.

By default, Wasm programs are sandboxed (they cannot affect the “outside world”) and can’t do anything except compute. Developers can use a “native function” if required. The HostFunction class, written in Java, is available to be called from the WebAssembly code. Peruffo pointed out that the use cases where Chicory could be used “are endless”: it was used to help with the JRuby distribution or even to “even to run Doom” on the JVM.

At this point, Chicory passes the WebAssembly test suite meaning that it can run “any correct Wasm project”. As pointed out by Peruffo, “This includes 100% of the V1 specification (but no SIMD support) on the happy path”. The next focus will be on making the runtime safe (incorrect programs will crash according to the spec) and later performant (JMH benchmarks were added to the project, to be able to spot any “performance” regressions).

Even if multiple Wasm runtimes are available, the project’s ambition is to become the “de facto” standard WebAssembly runtime for the JVM ecosystem. Other important milestones are “becoming production ready” before its first anniversary (September 2024) and “to be fast” and compatible by the end of 2024. These imply the creation of an AOT compiler that creates JVM bytecode and support for WebAssembly System Interface Preview 1(WASI), Single Instruction Multiple Data and Garbage Collection support. Those interested in contributing can interact via the project’s GitHub page or zulip chat.

The US National Institute of Standards and Technology (NIST) launched a public generative AI evaluation program developed by the international research community. The pilot program focuses on text-to-text and text-to-image. The general objectives include but are not limited to evolving benchmark dataset creation, multi-modal authenticity detection, comparative analysis and fake or misinformation source detection. The first-round submission deadline is August.

The pilot aims to measure and understand system behaviours for discriminating between synthetic and human-generated content in text-to-text (T2T) and text-to-image (T2I) modalities. Mainly, to respond to the following: “How does human content differ from synthetic content?” and “How can users differentiate between the two?”

Teams can act as generators, discriminators or both. Generator teams will be evaluated on their system’s ability to generate synthetic content as close as possible to humans. The discriminator teams will be evaluated on their system’s ability to detect synthetic content created by generative AI (LLMs and deep fake tools).

The Text-To-Text-Discriminators (T2T-D) task will need to detect whether a targeted output summary was generated using generative AI. Each trial consists of a single summary, the T2T-D detection system must render a confidence score (any real number). The higher numbers indicating the target text summary is more likely to have been generated using LLM-based models. The primary metric for measuring detection performance will be the Area Under the Receiver Operating Characteristics (ROC) Curve (AUC).

The Text-to-Text Generators (T2T-G) task is designed to automatically generate high-quality summaries based on a “topic” (statement of information needed) and a set of targeted documents (about 25). The summary must answer the need for information expressed in the topic statement. Participants should assume that the target audience of the summary is a supervisory information analyst who will use it to inform decision-making. The submission will have to adhere to the following rules:

• All processing of documents and generation of summaries must be automatic
• The summary can be no longer than 250 words (whitespace-delimited tokens)
• Summaries longer than the size limit will be truncated
• No bonus will be given for creating shorter summaries
• No specific formatting other than linear is allowed (e.g. plain text)

There will be about 45 topics in the test data for generator teams. This set of summaries from all generator teams will serve as the testing data for discriminator teams. The summary output will be evaluated by determining how easy or difficult it is to discriminate AI-generated summaries from human-generated summaries (the goal of generators is to output a summary that is indistinguishable from human-generated summaries).

The participants are not allowed to use the test dataset for training, modelling, or tuning their algorithms. All machine learning or statistical analysis algorithms must complete training, model selection, and tuning before running their system on the available test data; learning/adaptation during processing is not permissible. Each participant is allowed to submit system output for evaluation only once per 24-hour period.

The first pilot is focused on text-to-text and it will run throughout 2024. The platform is designed to support multiple modalities and technologies for teams from academia, industry, and other research labs. Those interested in participating can register on the program’s website until May 2025. The test phases are scheduled in June, September and November. After the evaluation closes in January 2025, the results will be released in February 2025 and a GenAI evaluation workshop will be organised in March 2025.

Other such contests are the generative AI hackathon organised by Google, the RTX developer challenge proposed by Nvidia, the generative AI competition organised by members from Harvard and AI for Life Sciences organised with support from the University of Vienna.

The US National Institute of Standards and Technology (NIST) launched a public generative AI evaluation program developed by the international research community. The pilot program focuses on text-to-text and text-to-image. The general objectives include but are not limited to evolving benchmark dataset creation, multi-modal authenticity detection, comparative analysis and fake or misinformation source detection. The first-round submission deadline is August.

The pilot aims to measure and understand system behaviours for discriminating between synthetic and human-generated content in text-to-text (T2T) and text-to-image (T2I) modalities. Mainly, to respond to the following: “How does human content differ from synthetic content?” and “How can users differentiate between the two?”

Teams can act as generators, discriminators or both. Generator teams will be evaluated on their system’s ability to generate synthetic content as close as possible to humans. The discriminator teams will be evaluated on their system’s ability to detect synthetic content created by generative AI (LLMs and deep fake tools).

The Text-To-Text-Discriminators (T2T-D) task will need to detect whether a targeted output summary was generated using generative AI. Each trial consists of a single summary, the T2T-D detection system must render a confidence score (any real number). The higher numbers indicating the target text summary is more likely to have been generated using LLM-based models. The primary metric for measuring detection performance will be the Area Under the Receiver Operating Characteristics (ROC) Curve (AUC).

The Text-to-Text Generators (T2T-G) task is designed to automatically generate high-quality summaries based on a “topic” (statement of information needed) and a set of targeted documents (about 25). The summary must answer the need for information expressed in the topic statement. Participants should assume that the target audience of the summary is a supervisory information analyst who will use it to inform decision-making. The submission will have to adhere to the following rules:

• All processing of documents and generation of summaries must be automatic
• The summary can be no longer than 250 words (whitespace-delimited tokens)
• Summaries longer than the size limit will be truncated
• No bonus will be given for creating shorter summaries
• No specific formatting other than linear is allowed (e.g. plain text)

There will be about 45 topics in the test data for generator teams. This set of summaries from all generator teams will serve as the testing data for discriminator teams. The summary output will be evaluated by determining how easy or difficult it is to discriminate AI-generated summaries from human-generated summaries (the goal of generators is to output a summary that is indistinguishable from human-generated summaries).

The participants are not allowed to use the test dataset for training, modelling, or tuning their algorithms. All machine learning or statistical analysis algorithms must complete training, model selection, and tuning before running their system on the available test data; learning/adaptation during processing is not permissible. Each participant is allowed to submit system output for evaluation only once per 24-hour period.

The first pilot is focused on text-to-text and it will run throughout 2024. The platform is designed to support multiple modalities and technologies for teams from academia, industry, and other research labs. Those interested in participating can register on the program’s website until May 2025. The test phases are scheduled in June, September and November. After the evaluation closes in January 2025, the results will be released in February 2025 and a GenAI evaluation workshop will be organised in March 2025.

Other such contests are the generative AI hackathon organised by Google, the RTX developer challenge proposed by Nvidia, the generative AI competition organised by members from Harvard and AI for Life Sciences organised with support from the University of Vienna.