Author: Anthony Alford

Article originally posted on InfoQ. Visit InfoQ

Microsoft announced the release of ML.NET 2.0, the open-source machine learning framework for .NET. The release contains several updated natural language processing (NLP) APIs, including Tokenizers, Text Classification, and Sentence Similarity, as well as improved automated ML (AutoML) features.

Program manager Luis Quintanilla announced the release at the recent .NET Conf 2022. The updated NLP APIs are powered by TorchSharp, a .NET wrapper for the popular PyTorch deep learning framework. The release includes the EnglishRoberta tokenization model and a TorchSharp implementation of NAS-BERT, which is used by the Text Classification and Sentence Similarity APIs. Updates to AutoML include an API for automated data pre-processing and a set of APIs for running experiments to find the best models and hyperparameters. Quintanilla also announced a new release of the Model Builder tool for Visual Studio, which includes the new text classification scenario and advanced training options.

The Text Classification API, which was previewed earlier this year, is based on the NAS-BERT model published by Microsoft Research in 2021. This model was developed using neural architecture search (NAS), resulting in smaller models than the standard BERT model, while maintaining accuracy. Users can fine-tune the pre-trained NAS-BERT model with their own data, to fit their custom use cases. The Sentence Similarity API uses the same pre-trained model, but instead of being fine-tuned to classify an input string, the model takes two strings as input and outputs a score indicating the similarity of the meaning of the two inputs.

The AutoML APIs are based on Microsoft’s Fast Library for Automated Machine Learning & Tuning (FLAML). While the Featurizer API is designed for pre-processing, the rest of the APIs work together to search for the best set of hyperparameters. The Experiment API coordinates the optimization of a Sweepable pipeline over a Search Space using a Tuner. Devs can use the Sweepable API to define the training pipeline for hyperparameter optimization of their models; the Search Space API for configuring the range of the hyperparameter search space for that pipeline; and the Tuner API to choose a search algorithm for that space. The release includes several tuner algorithms, including basic grid and random searches as well as Bayesian and Frugal optimizers.

Quintanilla also gave viewers a preview of the ML.NET roadmap. Future plans for deep learning features include new scenarios and APIs for question answering, named-entity recognition, and object detection. There are also plans for TorchSharp integrations for custom scenarios and improvements to the ONNX integration. Other plans include upgrades to the LightGBM implementation and to the implementation of the IDataView interface, as well as improvements to the AutoML API.

At the end of his presentation, Quintanilla answered questions from the audience. One viewer asked about support for different vendors’ GPUs and accelerator libraries, and Quintanilla noted that currently only NVIDIA’s CUDA accelerator is supported. When another viewer asked whether ML.NET’s object detection algorithms would run fast enough to support a live video stream, Quintanilla replied:

We want to focus on performance. We’re introducing new deep learning scenarios and we realized that performance is key there, so performance is a focus for us going forward.

The ML.NET source code is available on GitHub.

Article originally posted on InfoQ. Visit InfoQ

Meta AI Research recently open-sourced CICERO, an AI that can beat most humans at the strategy game Diplomacy, a game that requires coordinating plans with other players. CICERO combines chatbot-like dialogue capabilities with a strategic reasoning, and recently placed first in an online Diplomacy tournament against human players.

CICERO was described in a paper published in the journal Science. CICERO uses a 2.7B parameter language model to handle dialogue between itself and other players. To determine its moves, CICERO’s planning algorithm uses the dialogue to help predict what other players are likely to do, as well as what other players think CICERO will do. In turn, the output of the planner provides intents for the dialogue model. To evaluate CICERO, the team entered it anonymously in 40 online Diplomacy games; the AI achieved a score more than double that of the human average. According to the Meta team,

While we’ve made significant headway in this work, both the ability to robustly align language models with specific intentions and the technical (and normative) challenge of deciding on those intentions remain open and important problems. By open sourcing the CICERO code, we hope that AI researchers can continue to build off our work in a responsible manner. We have made early steps towards detecting and removing toxic messages in this new domain by using our dialogue model for zero-shot classification. We hope Diplomacy can serve as a safe sandbox to advance research in human-AI interaction.

Diplomacy is a strategy board game where players must capture a majority of territories called supply centers to win. There is no random component in the game; instead, battles are determined by numerical superiority. This often requires players to cooperate, so the bulk of game play consists of players sending messages to each other to coordinate their actions. Occasionally players will engage in deceit; for example, promising to help another player, while actually planning to attack that player.

To be successful, therefore, an AI must not only generate messages of human-level quality; the messages must make sense given the state of the game board, and the messages must cause other players to trust the AI. To generate the dialogue, Meta used a pre-trained R2C2 language model that was fine-tuned on a dataset of almost 13M messages from online Diplomacy games. The generated dialogue is conditioned on the intents generated by a planning module; the intents are the most likely actions that message sender and receiver will take after reading that message.

CICERO’s planning module generates intents by predicting other players’ likely actions, given the state of the board and messages from those players, then choosing an optimal action for itself. To model the likely actions of the other players, CICERO uses an iterative planning algorithm called piKL which incorporates information from the dialogues with other players. To train the planning module, the Meta researchers used a self-play algorithm similar to that used by AlphaZero.

The Meta team entered CICERO into anonymous league play for online Diplomacy games. The AI played 40 games, including an 8-game tournament with 21 players; CICERO placed first in the tournament. For its entire 40 games, CICERO was ranked in the top 10 percent of players with an average score was 25.8%, while the average score of its 82 human opponents was 12.4%.

In a Twitter thread about the work, CICERO co-author Mike Lewis replied to a question about whether CICERO would “backstab” (that is, lie to) other players:

It’s designed to never intentionally backstab – all its messages correspond to actions it currently plans to take. However, sometimes it changes its mind…

The CICERO source code is available on GitHub.

Article originally posted on InfoQ. Visit InfoQ

Microsoft Research recently open-sourced FarmVibes.AI, a suite of ML models and tools for sustainable agriculture. FarmVibes.AI includes data processing workflows for fusing multiple sets of spatiotemporal and geospatial data, such as weather data and satellite and drone imagery.

The release was announced on the Microsoft Research blog. FarmVibes.AI is part of Microsoft’s Project FarmVibes, an effort to develop technologies for sustainable agriculture. The key idea in FarmVibes.AI is fusion of multiple data sources to improve the performance of AI models. The toolkit contains utilities for downloading and preprocessing public datasets of satellite imagery, weather, and terrain elevation. It also includes models for removing cloud cover from satellite images and for generating micro-climate forecasts. According to Microsoft:

In addition to research, we are making these tools available to the broader community. Scientists, researchers, and partners can build new workflows leveraging these AI models, to estimate farming practices, the amount of emissions, and the carbon sequestered in soil.

World population growth and climate change are two of the major concerns motivating Project FarmVibes. As the population grows, farmers will need to produce more food; yet farming is not only impacted by climate change, it is also considered to be one of its causes. Project FarmVibes aims to help farmers increase their yields while reducing their use of water and chemicals. The project builds on FarmBeats, a previous research effort that was released as an Azure Marketplace product in 2019.

The core of FarmVibes.AI is a Kubernetes-based computing cluster for executing workflows. The cluster has four components: a REST API for invoking workflows and monitoring results; an orchestration module for managing workflow execution; workers for processing chunks of data through the workflow; and a cache for storing reusable intermediate results. There is also a pre-built Python client for interacting with the REST API.

The system comes with several built-in workflows for data ingestion, data processing, machine learning, and farm-related AI. The data ingestion workflows can download and process nearly 30 publicly available geospatial datasets. The data processing workflows implement several statistical and transformative operations, such as thresholding and Normalized Difference Vegetation Index (NDVI). The ML and AI workflows implement several models for identifying features on a farm, such as crops or pavement, as well as “what-if” scenarios such as water conservation and carbon sequestration.

FarmVibes.AI also includes several example Jupyter notebooks demonstrating data fusion, model training, and inference. These notebooks showcase some of Microsoft’s agriculture-related AI research, including SpaceEye and DeepMC. SpaceEye is a deep-learning computer vision model that can “recover pixels occluded by clouds in satellite images.” This can improve the performance of downstream models that use satellite imagery as input; for example, models that identify crops in satellite images. DeepMC is a model that can make short-term predictions of microclimate parameters, including temperature, humidity, wind speed, and soil moisture, which can help farmers identify optimal times for planting and harvesting.

Besides FarmVibes.AI, Project FarmVibes also includes: FarmVibes.Connect, technology for networking remote farms; FarmVibes.Edge, an IoT solution for processing data locally instead of in the cloud; and FarmVibes.Bot, a chatbot interface for communicating with farmers. Although only the FarmVibes.AI source code is currently available on GitHub, Microsoft says that the other components “will be released to GitHub soon.”

Article originally posted on InfoQ. Visit InfoQ

Anaconda, makers of a Python distribution popular among data scientists, recently published a report on the results of their State of Data Science survey. The report summarizes responses from nearly 3,500 students, academics, and professionals from 133 countries, and covers topics about respondent demographics and jobs as well as trends within the community.

The report was announced on the Anaconda blog. The survey ran from April 25 until May 14 this year, with respondents gathered from social media, the Anaconda website, and the Anaconda email database. The survey begins with demographics questions, then moves to workplace topics, including job tasks, tools, and the future of work. The survey also drills deep into the use of open-source software (OSS) in the enterprise, with particular focus on contribution as well as security concerns. The report concludes with several key takeaways, such as concerns in industry about a data-science talent shortage.

According to Anaconda,

As with years prior, we conducted a survey to gather demographic information about our community, ascertain how that community works, and collect insights into big questions and trends that are top of mind within the community. As the impacts of COVID continue to linger and assimilate into our new normal, we decided to move away from covering COVID themes in our report and instead focus on more actionable issues within the data science, machine learning (ML), and artificial intelligence industries, like open-source security, the talent dilemma, ethics and bias, and more.

The report is organized into the following sections:

• The Face of Data Science: demographic data about the respondents
• Data Professionals at Work: data about the work environment
• Enterprise Adoption of Open Source: use and contribution to OSS and concerns about the OSS “supply chain”
• Popularity of Python: data about the adoption of various programming languages by the respondents
• Data Jobs and the Future of Work: data about job satisfaction, talent shortages, and the future of the workforce
• Big Questions and Trends: sentiments about innovation and government involvement

The section on enterprise adoption of OSS revealed a concern about the security of OSS projects. In particular, the recent Log4j incident was a “disruptive and far-reaching example” that caused nearly a quarter of the respondents to reduce their OSS usage. Although most companies represented in the responses still use OSS, nearly 8% do not, and of those more than half said it was due to security concerns, which is a 13% increase from last year. The results also showed a 13% year-over-year decrease in the number of respondents whose companies encourage them to contribute to OSS.

The section on big questions and trends looked at several topics, including blockers to innovation. A majority said that insufficient talent and insufficient investment in engineering and tooling were the biggest barriers in the enterprise. Perhaps related to this concern about talent shortages, respondents said the most important role of AutoML would be to enable non-experts to train models, and nearly 70% of respondents thought their governments should provide more funding for STEM education.

In a discussion on Twitter about the survey responses on the role of government in tech, industry analyst Lawrence Hecht pointed out:

[Respondents] mostly want the government to give them money. Only 35% want regulation of Big Tech. A follow-on chart shows that even more clearly that there is no desire for specific actions re: AI and tech regulation — more of a general angst.

Earlier this year, InfoQ covered Stanford University’s AI Index 2022 Annual Report, which identifies top trends in AI, including advances in technical achievements, a sharp increase in private investment, and more attention on ethical issues. More recently, AI investors Nathan Benaich and Ian Hogarth published their fifth annual State of AI Report, covering issues related AI research, industry, politics, and safety. At the 2022 Google Cloud Next conference, Kaggle presented the results of their annual State of Data Science and Machine Learning survey of 23,997 respondents from 173 countries; the survey questions covered areas related to respondent demographics, programming language and tools, machine learning, and cloud computing.

The Anaconda survey response raw data is available on the Anaconda Nucleus website.

Article originally posted on InfoQ. Visit InfoQ

PyTorch, the popular deep-learning framework developed by Meta AI Research, has now become an independent top-level project of the Linux Foundation. The project will be managed by the newly-chartered PyTorch Foundation, with support from several large companies including Meta, AWS, NVIDIA, AMD, Google, and Microsoft.

PyTorch co-creator Soumith Chintala announced the move on the PyTorch blog. The move is intended to make sure business decisions about the framework are open and transparent, and take into account the needs of the many PyTorch stakeholders. The new Foundation also formalizes the technical governance of the project, defining a hierarchical maintainer organization and processes for making technical decisions. The project leaders chose the Linux Foundation as the parent organization because of its experience managing large open-source projects with a diverse community of stakeholders. According to Chintala:

PyTorch started with a small group of contributors which have grown and diversified over the years, all bringing in new ideas and innovations that would not have been possible without our community. We want to continue the open-source spirit – for the community and by the community. Thank you to our contributors, maintainers, users, supporters and new foundation members. We look forward to the next chapter of PyTorch with the PyTorch Foundation.

Chintala and his colleagues began developing PyTorch in 2016 and released version 1.0 in 2018. The framework quickly gained popularity, especially among the academic researchers; it’s currently used by approximately 80% of researchers who contribute to major machine learning conferences. InfoQ covered the initial 1.0 release as well as many of the framework’s major releases since.

The PyTorch Foundation is overseen by a governing board with members from Meta as well as from other large companies who have invested in and contributed to the project: computer chip manufacturer NVIDIA and AMD; and cloud providers Amazon Web Services (AWS), Microsoft Azure, and Google Cloud. This board will “prioritize the continued growth of PyTorch’s vibrant community.” Technical oversight will remain with the contributors and small group of core maintainers who “drive the overall project direction.”

The community reaction to the move has largely been positive, but in a Hacker News discussion, one user pointed out the lack of academic representatives on the governing board:

As an academic myself who does research on PyTorch, I wonder if there should have been more academics involved in the guidance or governance of PyTorch, especially given how much the future of machine learning may depend on choices made by such frameworks. Maybe it is unfounded, but I fear that over-optimizing for large industry uses and forgetting what it is like to run a single network training on your desktop GPU might be detrimental to PyTorch and machine learning as a whole.

Other users wondered whether the move signaled a reduction in Meta’s investment in PyTorch. Chintala addressed these concerns on Reddit, saying that “Meta is not divesting the project, if anything it’s the opposite — they’re investing more and more into PyTorch.” Yann LeCun, a Meta AI researcher and deep-learning progenitor, expressed a similar opinion on Twitter:

No. More resources from Meta, and way more resources from contributors other than Meta, now that PyTorch is a perennially open community project…It ensures that continued support is not subject to resource allocation decisions in one company. With this structure, there will be support as long as there are users who care sufficiently.

The PyTorch Foundation charter and technical governance documents can be found on the PyTorch site. PyTorch project source code is available on GitHub.

Article originally posted on InfoQ. Visit InfoQ

Researchers from New York University, University of Washington, and Johns Hopkins University have published the results of the NLP Community Metasurvey, which compiles the opinions of 480 active NLP researchers about several issues in the natural language processing AI field. The survey also includes meta-questions about the perceived opinions of other researchers.

The goal of the survey was to learn about the actual distribution of beliefs in the NLP community as well as sociological beliefs—what researchers think other researchers think. The survey was targeted at researchers who have published at least two NLP papers in the last three years. The questions cover six categories related to NLP research in particular, as well as artificial general intelligence (AGI) and social concerns; the team chose questions that are frequently discussed in the community and are subjects of public disagreement. In the results, the team found that a large majority of respondents think NLP research will have a positive impact on the future, and a narrow majority believes that recent progress in large language models (LLM) are significant steps toward AGI. According to the survey team:

By necessity, we are covering a subjectively chosen set of questions and reducing many complex issues into simplified scales, but we hope that the outcome can serve as a reference point for community discussion and for future surveys. This is not the final word in any debate, but we hope it will spark new discussions as an initial study of the range of positions people hold and ways in which the community may mis-model itself.

The survey questions covered the following categories:

• State of the Field: the role of industry and the near-term possibility of an “AI winter”
• Scale, Inductive Bias, and Adjacent Fields: whether large scale is sufficient or if linguistic expertise is needed to solve NLP problems
• AGI and Major Risks: if NLP research is making progress toward AGI and if AGI is a risk society
• Language Understanding: whether language models actually understand language
• Promising Research Programs: is NLP research on the right track?
• Ethics: if NLP has a positive impact and whether certain research areas are ethical

In addition to specifying whether they agreed with a question or not, the respondents were asked to predict what percentage of other respondents would agree with the question. The goal of collecting these meta-responses was to help researchers understand sociological beliefs, since mistaken sociological beliefs can “slow down communication and lead to wasted effort.”

Questions about the role of scaling AI models showed “striking mismatches” between what NLP researchers actually believe compared to what they think community beliefs are. Survey respondents estimated that nearly 50% of researchers would agree that scaling can solve “practically any” problem, and that less than 40% would agree that linguistic theory and expert design would be needed to solve important problems. However, in a Twitter thread highlighting some of the results, lead author Julian Michael pointed out:

Less than 20% of the field thinks that scaling up existing techniques will be enough to solve all applied NLP problems. A majority thinks that insights from linguistics or cognitive science will be an important part of future progress.

In a Hacker News discussion about the limits of current AI technology, AI writer and researcher Gwern Branwen referred to the NLP survey results and defended scaling, saying:

AGI & scaling critics are still in the majority, despite posturing as an oppressed minority…If you believe in scaling, you are still in a small minority of researchers pursuing an unpopular and widely-criticized paradigm. (That it is still producing so many incredible results and appearing so dominant despite being so disliked and small is, IMO, to its credit and one of the best arguments for why new researchers should go into scaling – it is still underrated.)

While the survey paper contains charts and summaries of the data, the survey website notes that a web-based dashboard for exploring the results is “coming soon.”