Month: December 2021

Posted on mongodb google news. Visit mongodb google news

Latest research study from JCMR including most recent “Q1-2021” Global Open-Source Database Software Market by Manufacturers, Regions, Type and Application, Forecast to 2021-2029. The Open-Source Database Software Research report presents a complete assessment of the market and contains Future trend, Current Growth Factors, attentive opinions, facts, historical data, and statistically supported and industry validated market data. The Open-Source Database Software study is segmented by products type & Applications. The research study provides estimates for and Open-Source Database Software Market Forecast till 2029

Get Quick Free Sample Copy of Open-Source Database Software Report @: jcmarketresearch.com/report-details/1495022/sample
 

Key Companies/players: MySQL, Redis, MongoDB, Couchbase, Apache Hive, MariaDB, Neo4j, SQLite, Titan

Open-Source Database Software Report Application & Types as follwed: 

 Segment by Type
– Cloud-based
– On Premises
Segment by Application
– Large Enterprises
– SMEs

The research covers the current & Future market size of the Global Open-Source Database Software market & its growth rates based on 8 year history data. It also covers various types of Open-Source Database Software segmentation such as by geography [China, Japan, Korea, Taiwan, Southeast Asia, India & Australia].The Open-Source Database Software market competition is constantly growing higher with the rise in technological innovation and M&A activities in the Open-Source Database Software industry. Moreover, many local and regional vendors are offering specific application products for varied end-users.On the basis of attributes such as company overview, recent developments, strategies adopted by the Open-Source Database Software market leaders to ensure growth, sustainability, financial overview and recent developments.

Get the crucial Qualitative + Quantitative Open-Source Database Software Report @ jcmarketresearch.com/report-details/1495022/Open-Source-Database-Software

Stay up-to-date with global Open-Source Database Software market research offered by JCMR. Check how Open-Source Database Software key trends and emerging drivers are shaping Open-Source Database Software industry growth.global Open-Source Database Software market insights reports covers market characteristics, size and growth, segmentation, regional breakdowns, competitive landscape, shares, trend and strategies for Open-Source Database Software market. The Open-Source Database Software market characteristics section of the report defines and explain the Open-Source Database Software market. The Open-Source Database Software market size section gives the revenues, covering both the historic growth of the Open-Source Database Software market and forecasting the future.

In the Global Open-Source Database Software Industry Market Analysis & Forecast 2021-2029, the revenue is valued at USD XX million in 2021 and is expected to reach USD XX million by the end of 2029, growing at a CAGR of XX% between 2021 and 2029. The production is estimated at XX million in 2021 and is forecasted to reach XX million by the end of 2029, growing at a CAGR of XX% between 2021 and 2029.

Get Discount on Open-Source Database Software Report @ jcmarketresearch.com/report-details/1495022/discount

QueriesResolved in Open-Source Database Software report – Global Open-Source Database Software Market, 2021 by Manufacturers, Regions, Type and Application, Forecast to 2029

What will the Open-Source Database Software market size in 2029 & what will the growth rate?

What are the key Open-Source Database Software market trends?

What is driving Global Open-Source Database Software Market?

What are the challenges to Open-Source Database Software market growth?

Who are the key vendors in Global Open-Source Database Software Market space?

What are the key Open-Source Database Software market trends impacting the growth of the Global Open-Source Database Software Market?

What are the key outcomes of the five forces analysis of the Global Open-Source Database Software Market?

What are the Open-Source Database Software market opportunities and threats faced by the vendors in the Global Open-Source Database Software market?  Get in-depth details about factors influencing the Open-Source Database Software market shares of the Americas, APAC, and EMEA?

There are 15 Chapters to display the Global Open-Source Database Software market.

Chapter 1, to describe Definition, Specifications and Classification of Open-Source Database Software, Applications and Market Segments by Regions;

Chapter 2, to analyze the Open-Source Database Software Manufacturing Cost Structure, Raw Material and Suppliers, Manufacturing Process, Industry Chain Structure;

Chapter 3, to display the Open-Source Database Software Technical Data and Manufacturing Plants Analysis of , Capacity and Commercial Production Date, Manufacturing Plants Distribution, Export & Import, R&D Status and Technology Source, Raw Materials Sources Analysis;

Chapter 4, to show the Overall Open-Source Database Software Market Analysis, Capacity Analysis (Company Segment), Sales Analysis (Company Segment), Sales Price Analysis (Company Segment);

Chapter 5 and 6, to show the Regional Open-Source Database Software Market Analysis that includes North America, China, Europe, Southeast Asia, Japan & India, Open-Source Database Software Market Analysis by [Type];

Chapter 7 and 8, to analyze the Open-Source Database Software Market Analysis by [Application] Major Manufacturers Analysis of Open-Source Database Software;

Chapter 9, Open-Source Database Software Market Trend Analysis, Regional Open-Source Database Software Market Trend, Open-Source Database Software Market Trend by Product Types, Open-Source Database Software Market Trend by Applications;

Chapter 10, Open-Source Database Software Regional Marketing Type Analysis, International Trade Type Analysis, Supply Chain Analysis;

Chapter 11, Open-Source Database Software to analyze the Consumers Analysis of;

Chapter 12, to describe Open-Source Database Software Research Findings and Conclusion, Appendix, methodology and data source;

Chapter 13, 14 and 15, to describe Open-Source Database Software sales channel, distributors, traders, dealers, Research Findings and Conclusion, appendix and data source.

Buy this Open-Source Database Software research report @ jcmarketresearch.com/checkout/1495022

Reasons for Buying Open-Source Database Software Report

This Open-Source Database Software report provides pin-point analysis for changing competitive dynamics

Open-Source Database Software provides a forward looking perspective on different factors driving or restraining market growth

Open-Source Database Software provides a 8-year forecast assessed on the basis of how the market is predicted to grow

Open-Source Database Software helps in understanding the key product segments and their future

Open-Source Database Software provides pin point analysis of changing competition dynamics and keeps you ahead of competitors

Open-Source Database Software helps in making informed business decisions by having complete insights of market and by making in-depth analysis of market segments

Thanks for reading Open-Source Database Software article; you can also get individual chapter wise section or region wise report version like North America, Europe or Asia.

Find more research reports on Open-Source Database Software Industry. By JC Market Research.

About Author:

JCMR global research and market intelligence consulting organization is uniquely positioned to not only identify growth opportunities but to also empower and inspire you to create visionary growth strategies for futures, enabled by our extraordinary depth and breadth of thought leadership, research, tools, events and experience that assist you for making goals into a reality. Our understanding of the interplay between industry convergence, Mega Trends, technologies and market trends provides our clients with new business models and expansion opportunities. We are focused on identifying the “Accurate Forecast” in every industry we cover so our clients can reap the benefits of being early market entrants and can accomplish their “Goals & Objectives”.

Contact Us: https://jcmarketresearch.com/Contact-Details

JC Market Research

Mark Baxter (Head of Business Development)

Phone: +1 (925) 478-7203

Email: [email protected]

Article originally posted on mongodb google news. Visit mongodb google news

Article originally posted on Data Science Central. Visit Data Science Central

Companies today need to be nimble and ready in the face of constantly evolving technology and changing consumer preferences. To achieve this, organizations are switching to Amazon Web Services (AWS). It enables companies to rapidly deploy and scale technology that meets the growing (or shrinking) demand without having to invest in expensive IT infrastructure. This is an efficient and cost-effective solution that makes it easier for businesses of all sizes to drive innovation. And, by now though most organizations are aware of the value delivered by AWS yet many of them have an alarmingly low awareness regarding cloud security.

The security model of AWS cloud computing service is based on the shared responsibility tenet. It implies that AWS is in charge of cloud infrastructure security though the customer must strive to achieve desired security levels within the cloud environment. Here are some best practices to enable you on how to go about it.

1. Strategize: While the endeavors to achieve top-notch AWS cloud security are surely appreciable and advised, any attempts to achieve the required security levels without any planning will most likely end in disaster for the company. A precise cloud security strategy will not only keep the implementation process on track but also ensure desired levels of safety even as software is continually updated.
2. Cloud-native security solutions: Some companies may be tempted to simply make do with conventional security solutions. Resist the urge at all costs; instead, opt for a cloud-native security solution that also offers consistent support, can safeguard AWS workloads, etc.
3. Embrace cloud security controls: Ensuring security is not just AWS’ responsibility; i.e. companies must also do their bit, including the implementation of different measures to ensure high levels of security. To that end, companies must unambiguously outline user roles, regularly organize and execute privilege audits to ensure users that do not need access any longer do not have access, and leverage a robust password policy. Implementing other controls, including permission time-outs and multi-factor authentication, helps further alleviate risk.
4. Encryption is key: Most of us would think that encryption is now a standard procedure, but that is far from the truth, i.e. some companies often ignore this rather rudimentary requirement. Do not do that, please. Encryption is important for a variety of reasons, including the safety of sensitive data, regulatory compliance requirements, etc. AWS is highly conducive to encryption, actually; all companies need to do is enable native encryption to safeguard all the data stored on the Amazon S3 service.
5. Prepare a response plan: Of course, the hope and goal are to ensure zero attacks; unfortunately, reality often tends to be different. Hence, the prudent thing to do in this regard is to hope for the best and prepare for the worst. In the context of AWS cloud security, this means the company must assume that an attack will indeed occur at some point and based on that, put together a plan of action for what to do when the attack occurs. Having a response plan to a breach ready to go allows companies to cut down the time it takes them to respond to such a situation.

There is not even a modicum of doubt that AWS is a segment leader for plenty of reasons. So, it is understandable that this behemoth in the cloud computing sector implements and enforces class-leading security measures to safeguard its virtual environments as well as its many, many users. Unfortunately, these measures can still be undermined if users do not equally share in the responsibility of ensuring top-notch cloud security. If you have been wondering how to do that, we hope you find the aforementioned best practices handy. And if you want to take it up a few notches still, we highly recommend you look for a trusted name in the world of AWS software development services. Their expertise will lend your pursuit of security a variety of crucial benefits.

Posted on mongodb google news. Visit mongodb google news

sudo apt update

sudo apt install mongodb-server mongodb

mongo --version

sudo apt remove mongodb*

sudo apt purge mongodb

sudo apt autoremove

sudo rm -r /var/log/mongodb

sudo rm -r /var/lib/mongodb

sudo apt-get install gnupg

wget -qO - https://www.mongodb.org/static/pgp/server-5.0.asc | sudo apt-key add -

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/5.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/5.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/5.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list

sudo apt update

sudo apt clean
sudo apt-get install mongodb-org

sudo systemctl daemon-reload

sudo systemctl start mongod

sudo systemctl enable mongod

sudo systemctl status mongod

mongo --version

mongo

show dbs

use yourDbname

db.items.insert ({name:“name”})

db.items.find ()

exit;

Article originally posted on mongodb google news. Visit mongodb google news

Article originally posted on InfoQ. Visit InfoQ

With the year quickly coming to an end, whether we like it or not, we reflect on 2021. Probably one of the things worth being mentioned, especially in the Java world, is the release of JDK 17, the LTS version incorporating features like pattern matching for switches, a new vector API or sealed classes. The release bears the promise of out-of-the-box improved performance as Application and Class Data Sharing (AppCDS) is among the new features along with the improved performance of the new G1 Garbage Collector. Even though the release date was clearly set, the team at Minecraft decided to make the move earlier, migrating to JDK 16. This is even more interesting, as Minecraft is now part of Microsoft, so hence it will use the newly developed JRE. Given their head start with running Java 16 in production, InfoQ spoke to the Mojang Team to better understand their experience and gains.

InfoQ: What was the reasoning to migrate to Java 16, when Java 17 was just around the corner? Did you migrate to Java 17, or do you intend to? What was the version you migrated from?

Mojang Team:

We’d been keeping up to date with each Java release version as it came out (on a separate branch from our main branch). Java 16 gave us significant out-of-the-box performance gains with regards to startup times, memory management and code execution. It also gave our developers a boost in their productivity with new features like var, switch expressions, text blocks and more!

We’re currently evaluating Java 17 with Microsoft, since it is an LTS release it makes sense that we’ll roll it out sometime in the future, but we’re not committing to a concrete date yet.

InfoQ: Can you describe the migration process? How smooth was it? How long did the migration take?

Mojang Team:

As we’d been keeping up to date with each major version, we avoided the big bang migration and so it wasn’t too difficult. The earlier challenges in the migration came via moving from Java 8 to Java 11 due to the module system introducing stricter security controls around accessing JDK internals which some of our 3^rd party dependencies relied on at the time.

We recommend that any serious Java shop sets up their CI/CD to onboard each major release. Then you can incrementally update your dependencies and code to take advantage of the new features. Once you get into the habit it’s a relatively small update cost for a large gain each time!

InfoQ: You have been running JDK 16 in production for some time now. What improvements did you observe in comparison with the previous setup?

Mojang Team:

Lots of performance gains out-of-the-box with start-up times from Application and Class Data Sharing, memory efficiency gains from String Deduplication in the G1 Garbage Collector, a smoother experience because there are fewer long pauses from GC, and a host of other small improvements. It’s worth upgrading from an operational perspective!

InfoQ: Was there something that you expected to happen once using Java 16, but didn’t?

Mojang Team:

We didn’t have any failed expectations per se, it’s been very successful. One interesting thing was that we weren’t sure how quickly the mod community would take to using the new language features. The response has been overwhelmingly positive, it really matters to their developer productivity to have access to the latest features!

InfoQ: What Java build are you using? Is it the Microsoft one? If so, what was the inspiration to use it?

Mojang Team:

It is the Microsoft Build of OpenJDK. Microsoft has made major long-term investments into Java (such as the jClarity acquisition in 2019). They have a very capable Java Engineering Group who have already made significant contributions to Java itself (like the Windows ARM port). They also share our love for Minecraft, we know we’re in good hands.

InfoQ: What were the most challenging technical problems to tackle during the migration?

Mojang Team:

There were three main challenges:

1. One was just making sure we updated all the 3^rd party dependencies to versions that also support Java 16 such as the Lightweight Java Gaming Library (LWGL).
2. Making sure that changes in UI toolkits, fonts, sound etc. were still compatible.
3. Re-configuring the new Garbage Collectors and other JVM tuning parameters to get the most out of the new changes.

InfoQ: What would’ve you liked to know before the migration, but you didn’t?

Mojang Team:

There were several guides and blog posts out there, but not really a canonical source for migrating through each version. It wasn’t a blocker, but we did have to piece the information together a bit. The Microsoft guide for transitioning from Java 8 to 11 was helpful.

InfoQ: What do you have planned next?

Mojang Team:

From a purely Java perspective we’ll continue working with the Microsoft Java Engineering Group on testing each new Java version as they’re released. We’ll undoubtedly take some of those on once we’re satisfied with the stability, performance gains and developer productivity gains!

We can’t comment on any specific roadmap at this time but with Microsoft supporting the proposed two-year LTS cadence for OpenJDK vendors, it gives us the confidence that we can follow upgrades with a more regular cadence than we have in the past.

Even though for most of the companies, the LTS releases are targeted when upgrading, Minecraft, now a Microsoft Company, took a leap of faith when upgrading to Java 16. Their expectations in terms of performance gains with start-up times out-of-the-box due to AppCDS, memory efficiency gains from String Deduplication in the G1 Garbage Collector, and an altogether smoother experience due to fewer long pauses from GC.

Also the new language additions were welcomed by the mod community: “The response has been overwhelmingly positive, it really matters to their developer productivity to have access to the latest features!”. These promise to allow developers to write more elegant code, faster.

Posted on mongodb google news. Visit mongodb google news

Article originally posted on mongodb google news. Visit mongodb google news

Article originally posted on InfoQ. Visit InfoQ

GitLab 14.6 new Geo configuration streamlines the process of using the geographically closest replica to speed up clone and pull commands. It also introduces an activity list for GitLab’s Agent to log real-time events and brings support for .NET 6.

GitLab Geo replicas provide local, read-only access to GitLab instances so pull and clone operations can execute faster. Geo replicas are a feature relevant to larger, distributed organizations that work with large repositories. Previous to GitLab 14.6, users were responsible for using the right URL for the specific replica they wanted to use for a clone or pull, while targeting all write requests to the main repository.

GitLab 14.6 enables instead the use of a single, organization-wide URL that automatically uses the Geo replica closest to the user. Additionally, secondary replicas transparently proxy write requests to the main GitLab instance, thus removing any limitations on the kind of operations users can carry through against a replica.

GitLab’s Kubernetes Agent now includes the possibility of logging activity events. Initially, the activity list includes token creation and cluster connection events, but it will be extended in the near future with additional events, says GitLab.

GitLab 14.6 also improves its markdown editing experience by making the rich and textual markdown editing modes work seamlessly together, whereas it was previously required to save changes when switching from a mode to another.

As mentioned, GitLab 14.6 has got updates to support Microsoft latest .NET Core release, .NET 6. Specifically, GitLab’s static code analyzer for .NET, Security Code Scan, has been updated for the latest .NET version. Security Code Scan is able to detect a number of vulnerabilities, including SQL Injection, Cross-Site Scripting, Cross-Site Request Forgery, and others. Similarly, GitLab SAST has learned to detect .NET 6 project and apply the new scanning capabilities, provided you enabled the latest Security Code Scan in your .gitlab-ci.yml.

In addition to support for .NET 6, GitLab 14.6 code scanning updates many analyzers, including Spotbugs, PMD, Semgrep, and Gosec, which will be automatically used unless you pinned them to a specific version in .gitlab-ci.yml. Furthermore, scan policies for CI pipelines can now be defined independent of .gitlab-ci.yml.

GitLab 14.6 includes many more new features than what can be summarized here, so do not miss the official announcement for the full detail.

Posted on mongodb google news. Visit mongodb google news

“ 

Get Sample Report Buy Complete Report

Article originally posted on mongodb google news. Visit mongodb google news

Posted on mongodb google news. Visit mongodb google news

Global Document Databases Software Market (Pre-Post Covid-19) Size & Forecast Analysis till 2029: Global Document Databases Software research report on the Document Databases Software market is a product of a brief review and an extensive analysis of the realistic data collected from the Global Document Databases Software Market 2021. The data was gathered based on Document Databases Software manufacturing drifts and services & goods linked demands.

Download Free Sample Document Databases Software Report PDF @ jcmarketresearch.com/report-details/1420129/sample

         Due to the rising partnership activities of Document Databases Software industry key players over the projected period, North America accounted for the xxx million $ share on the Document Databases Software market in 2021

Top Document Databases Software Key players included in this Research: MongoDB, Amazon, ArangoDB, Azure Cosmos DB, Couchbase, MarkLogic, RethinkDB, CouchDB, SQL-RD, OrientDB, RavenDB, Redis

Major Types & Applications Present in Document Databases Software Market as followed:

Market segment by Type, the product can be split into
– Cloud Based
– Web Based

Market segment by Application, split into
– Large Enterprises
– SMEs

A flawless example of the latest developments and groundbreaking strategic changes allows our clients the opportunity to improve their decision-making skills. Ultimately this helps to work with perfect business solutions and execute innovative implementations. The Global Document Databases Software Market 2020-2029 Report highlights the latest trends, growth, new opportunities and latent tricks.

[We are currently offering Special Discount on Document Databases Software report because of Covid-19 please share you budget so we can help you to deliver our service]

In addition to the Document Databases Software related statistics, the larger part of the data obtained is presented in graphical form. The global Market Study Document Databases Software shows in detail the working of key market players, manufacturers, and distributors. The study also outlines the restrictions and factors influencing the global demand for Global Document Databases Software Market.

Special Discount on Document Databases Software Report Immediate Purchase @ jcmarketresearch.com/report-details/1420129/discount

Commonly Asked Questions:

• At what rate is the Document Databases Software market projected to grow?

The year-over-year growth for 2021 is estimated at XX% and the incremental growth of the market is anticipated to be $xxx million.

• Who are the top players in the Document Databases Software market?

MongoDB, Amazon, ArangoDB, Azure Cosmos DB, Couchbase, MarkLogic, RethinkDB, CouchDB, SQL-RD, OrientDB, RavenDB, Redis

• What are the key market drivers and challenges?

The demand for strengthening ASW capabilities is one of the major factors driving the Document Databases Software market.

• How big is the North America Document Databases Software market?

The North America region will contribute XX% of the Document Databases Software market share

Check feasibility and Get Customized for Document Databases Software Report @: jcmarketresearch.com/report-details/1420129/enquiry

This helps to understand the overall market and to recognize the growth opportunities in the global Document Databases Software Market. The report also includes a detailed profile and information of all the major Document Databases Software market players currently active in the global Document Databases Software Market. The companies covered in the report can be evaluated on the basis of their latest developments, financial and business overview, product portfolio, key trends in the Document Databases Software market, long-term and short-term business strategies by the companies in order to stay competitive in the Document Databases Software market.

Regions & Countries Mentioned in The Document Databases Software Market Report:

• Document Databases Software industry North America: United States, Canada, and Mexico.
• Document Databases Software industry South & Central America: Argentina, Chile, and Brazil.
• Document Databases Software industry Middle East & Africa: Saudi Arabia, UAE, Turkey, Egypt and South Africa.
• Document Databases Software industry Europe: UK, France, Italy, Germany, Spain, and Russia.
• Document Databases Software industry Asia-Pacific: India, China, Japan, South Korea, Indonesia, Singapore, and Australia.

The Document Databases Software report analyses various critical constraints, such as item price, production capacity, profit & loss statistics, and global market-influencing transportation & delivery channels. It also includes examining such important elements such as Document Databases Software market demands, trends, and product developments, various organizations, and global market effect processes.

Buy Full Copy Global Document Databases Software Report @ jcmarketresearch.com/checkout/1420129

A methodically organized Market Analysis study Document Databases Software is based on the primary and secondary tools. It illustrates the data gathered in a more communicative and descriptive manner encouraging the consumer to develop a well-structured strategy to grow and improve their companies in the expected time.

Find more research reports on Document Databases Software Industry. By JC Market Research.

About Author:

JCMR global research and market intelligence consulting organization is uniquely positioned to not only identify growth opportunities but to also empower and inspire you to create visionary growth strategies for futures, enabled by our extraordinary depth and breadth of thought leadership, research, tools, events and experience that assist you for making goals into a reality. Our understanding of the interplay between industry convergence, Mega Trends, technologies and market trends provides our clients with new business models and expansion opportunities. We are focused on identifying the “Accurate Forecast” in every industry we cover so our clients can reap the benefits of being early market entrants and can accomplish their “Goals & Objectives”.

Contact Us: https://jcmarketresearch.com/contact-us

JC MARKET RESEARCH

Mark Baxter (Head of Business Development)

Phone: +1 (925) 478-7203

Email: [email protected]

Connect with us at – LinkedIn 

www.jcmarketresearch.com

Article originally posted on mongodb google news. Visit mongodb google news

Posted on mongodb google news. Visit mongodb google news

“

Get Sample Report Buy Complete Report

Article originally posted on mongodb google news. Visit mongodb google news

Article originally posted on InfoQ. Visit InfoQ

Microsoft announced the release of SynapseML, an open-source library for creating and managing distributed machine learning (ML) pipelines. SynapseML runs on Apache Spark, provides a language-agnostic API abstraction over several datastores, and integrates with several existing ML technologies, including Open Neural Network Exchange (ONNX).

The release was announced in a blog post by software engineer Mark Hamilton. SynapseML runs on Apache Spark, taking advantage of Spark’s management of large-scale fault-tolerant compute clusters. The library has APIs for both Python and Java, with the ability to generate bindings for Java, R, and C#. SynapseML includes the HTTP on Spark module, allowing users efficient integration of web services into their pipelines, as well as pre-built wrappers for invoking several such services, including Azure Cognitive Services. Using ONNX, developers can deploy pre-trained models from Microsoft’s ONNX Model Hub, or convert models built in other frameworks, such as TensorFlow or PyTorch, to perform distributed inference on Spark. The Spark Serving module allows developers to expose their Spark pipelines as low-latency web services. According to Hamilton,

Our goal is to free developers from the hassle of worrying about the distributed implementation details and enable them to deploy them into a variety of databases, clusters, and languages without needing to change their code.

SynapseML includes integrations with several popular ML frameworks, including ONNX, CNTK, LightGBM, OpenCV, and Vowpal Wabbit. These integrations provide APIs that conform to the Transformer and Estimator abstractions defined by Spark’s ML pipelines.The ONNX and CNTK integrations allow users to import pre-trained deep learning models generated by other systems; users can then use the Spark cluster as a distributed inference engine. The OpenCV integration provides Transformer abstractions for manipulating image data within a ML pipeline, for example, flipping or blurring images. The LightGBM and Vowpal Wabbit integrations provide both Estimators for training models and Transformers for inference.

To support integration with Azure Cognitive Services, SynapseML includes an HTTP on Spark module, which is described in detail in a paper by Hamilton’s research team. HTTP on Spark provides an efficient way for Spark workers to call out to web services. It augments Spark’s parallelism model, allowing Spark workers to multi-task while waiting for HTTP requests to complete. This improves throughput without interfering with Spark’s native parallelism. To mitigate the risk of the cluster overwhelming a downstream web service, HTTP on Spark handles rate-limiting header responses to implement “back pressure” and re-tries. In addition to Transformer wrappers for nearly 50 Azure services, SynapseML also provides a generic HTTPTransformer that allows users to call arbitrary web services.

SynapseML includes several other features. Spark Serving extends Spark’s Structured Streaming engine, allowing any Structured Streaming job to be exposed as a web service. SynapseML also includes tools for responsible AI, such as data balance analysis and model explainability. The library includes support for AutoML features, such as finding the best-performing model using hyperparameter search, as well as Spark-native implementation of several models, including an anomaly-detection model for cyber security; an isolation forest model, which performs nonlinear outlier detection; and a conditional k-nearest-neighbor model.

Along with the open-source release, at their recent Ignite conference Microsoft announced the general availability of SynapseML on the Azure Synapse Analytics service. In response to a question on Twitter about the difference between SynapseML and Spark’s MLLib, Bart Czernicki, a principal technical architect at Microsoft, replied that the library was repackaged and optimized for Azure Synapse Analytics. In a separate tweet about SynapseML, ML researcher Moein Kareshk said:

Although I haven’t had a chance to benchmark the speed, its APIs are clear and easy to learn to prototype #ML products.

The SynapseML code is available on GitHub.