Month: July 2023

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

The evolution of database technology has been a fascinating journey, from the early days of hierarchical and network databases to the more recent innovations in NoSQL and NewSQL databases. As the world becomes increasingly data-driven, the need for efficient and scalable database solutions has never been more critical. In this context, the emergence of Artificial Intelligence (AI) document-oriented databases represents a significant leap forward in the ongoing quest for better ways to store, manage, and analyze data.

Document-oriented databases, a type of NoSQL database, store data in a semi-structured format, such as JSON or BSON. This flexible schema allows for the storage of complex and hierarchical data, making it well-suited for modern applications that deal with diverse and dynamic data types. The ability to scale horizontally and perform well in distributed environments has made document-oriented databases a popular choice for big data and real-time analytics applications.

The integration of AI into document-oriented databases has the potential to revolutionize the way we interact with data. By incorporating machine learning algorithms and natural language processing techniques, AI-powered databases can understand and interpret the content of documents, making it possible to perform complex queries and analysis without the need for manual data manipulation or pre-processing.

One of the key benefits of AI document-oriented databases is their ability to learn and adapt over time. As more data is ingested and analyzed, the database can refine its understanding of the content and relationships within the data, leading to more accurate and efficient query results. This self-learning capability is particularly valuable in industries where data is constantly changing and evolving, such as healthcare, finance, and e-commerce.

Another advantage of AI document-oriented databases is their ability to handle unstructured data. Traditional relational databases struggle with unstructured data, such as text, images, and video, as they require a fixed schema to store and organize data. In contrast, document-oriented databases can easily accommodate unstructured data, and with the addition of AI, they can also extract valuable insights from this data. For example, an AI-powered document-oriented database could analyze customer reviews to identify trends and sentiment, or process medical records to uncover patterns and correlations that may be indicative of a particular condition.

As AI document-oriented databases continue to evolve, we can expect to see even more advanced features and capabilities. For instance, AI algorithms could be used to automatically identify and tag relevant metadata, making it easier for users to search and filter documents based on specific criteria. Additionally, AI-powered databases could enable more sophisticated data visualization and reporting tools, allowing users to gain deeper insights into their data and make more informed decisions.

The integration of AI into document-oriented databases is not without its challenges, however. Ensuring data privacy and security is a top concern, as AI algorithms often require access to large amounts of sensitive data to function effectively. Additionally, the complexity of AI algorithms can make it difficult to troubleshoot and optimize database performance. As a result, organizations must carefully weigh the benefits and risks of adopting AI document-oriented databases and invest in the necessary infrastructure and expertise to support these advanced technologies.

In conclusion, the evolution of database technology has reached an exciting new frontier with the advent of AI document-oriented databases. By harnessing the power of AI, these databases have the potential to transform the way we store, manage, and analyze data, unlocking new possibilities for innovation and growth across a wide range of industries. As with any emerging technology, there are challenges to overcome, but the potential rewards are vast and will undoubtedly shape the future of database technology for years to come.

Article originally posted on InfoQ. Visit InfoQ

Spotify runs more than 250 online experiments annually on its Spotify Home platform, which are used by dozen of different teams. To accomplish running experiments at such scale, Spotify uses a number of different tools, explains Spotify Product Manager Nik Goyle.

At Spotify, our Home serves as a personalized surface where users retrieve familiar content and discover new content tailored to their preferences. […] Home experiments focus on pushing the boundaries of personalization, exploring innovative ways to tailor content and programming strategies.

At Spotify’s scale, it makes sense to develop custom tools to improve the efficiency of online experiments, especially A/B testing. One such tool is dubbed Home Config and is aimed at allowing developers and testers to fine-tune personalization strategies. Goyle describes it like a configuration-as-service tool, making it simple for technical as well as non-tech users to create personalized experiences to test out:

[Home Config] allows them to define parameters related to ranking, content, visual treatments, and more, ensuring a personalized and tailored experience for users.

Another essential component of Spotify’s solution is the Experimentation Platform, which enables releasing configurations created using Home Config into production and speeding-up the experimentation process.

It provides experimenters with a comprehensive interface to design, launch, and monitor experiments. With EP, experimenters can define experiment parameters, set up control and treatment groups, track metrics, and analyze results.

A third tool, Home QA, is a front-end application able to simulate Home requests to ensure every experiment does not break anything before launch.

Another critical dimension to Spotify approach to experiments is coordination. This is backed by two additional tools, one Experiment Tracker used to prioritize and monitor all experiments from a centralized location, and the Experiment Validation Assistant, which validate proposed A/B tests by checking they are not misconfigured and providing actionable data in an automated way.

EVA decreases overhead and ensures experiments adhere to predefined criteria. The results of EVA’s validations are shared in a designated Slack channel, enabling swift feedback and necessary adjustments.

While not all organization will have the resources to implement their own experimentation tools, Spotify’s approach illustrates important requirements for a successful strategy to carry through experiments at scale, including how to configure experiments, deploy and run them, ensure quality is preserved across them, and how to streamline communication. If you are interested in the full detail, do not miss the original article.

Article originally posted on InfoQ. Visit InfoQ

JEP 452, Key Encapsulation Mechanism API, has been promoted from Targeted to Completed for JDK 21. This JEP introduces an API for Key Encapsulation Mechanisms (KEMs), a modern encryption technique for securing symmetric keys using public key cryptography. The new API aims to enable applications to use KEM algorithms such as RSA Key Encapsulation Mechanism (RSA-KEM), Elliptic Curve Integrated Encryption Scheme (ECIES), and candidate KEM algorithms for the National Institute of Standards and Technology (NIST) Post-Quantum Cryptography standardization process.

KEMs are a crucial tool for defending against quantum attacks. The traditional technique of securing symmetric keys involves encrypting a randomly generated symmetric key with a public key. However, this requires padding and can be challenging to prove secure. KEMs, on the other hand, use properties of the public key to derive a related symmetric key, eliminating the need for padding.

The new API will also enable the use of KEMs in higher-level protocols such as Transport Level Security (TLS) and in cryptographic schemes such as Hybrid Public Key Encryption (HPKE, RFC 9180). It will allow security providers to implement KEM algorithms in either Java code or native code. However, it is not a goal to include key pair generation in the KEM API, as the existing KeyPairGenerator API is sufficient.

The KEM API consists of three functions: a key pair generation function, a key encapsulation function, and a key decapsulation function. The key pair generation function is covered by the existing KeyPairGenerator API, while the encapsulation and decapsulation functions are defined in a new class, KEM. Consider the following code snippets:

package javax.crypto;

public class DecapsulateException extends GeneralSecurityException;

public final class KEM {

    public static KEM getInstance(String alg)
        throws NoSuchAlgorithmException;
    public static KEM getInstance(String alg, Provider p)
        throws NoSuchAlgorithmException;
    public static KEM getInstance(String alg, String p)
        throws NoSuchAlgorithmException, NoSuchProviderException;

    public static final class Encapsulated {
        public Encapsulated(SecretKey key, byte[] encapsulation, byte[] params);
        public SecretKey key();
        public byte[] encapsulation();
        public byte[] params();
    }

    public static final class Encapsulator {
        String providerName();
        int secretSize();           // Size of the shared secret
        int encapsulationSize();    // Size of the key encapsulation message
        Encapsulated encapsulate();
        Encapsulated encapsulate(int from, int to, String algorithm);
    }

    public Encapsulator newEncapsulator(PublicKey pk)
            throws InvalidKeyException;
    public Encapsulator newEncapsulator(PublicKey pk, SecureRandom sr)
            throws InvalidKeyException;
    public Encapsulator newEncapsulator(PublicKey pk, AlgorithmParameterSpec spec,
                                        SecureRandom sr)
            throws InvalidAlgorithmParameterException, InvalidKeyException;

    public static final class Decapsulator {
        String providerName();
        int secretSize();           // Size of the shared secret
        int encapsulationSize();    // Size of the key encapsulation message
        SecretKey decapsulate(byte[] encapsulation) throws DecapsulateException;
        SecretKey decapsulate(byte[] encapsulation, int from, int to,
                              String algorithm)
                throws DecapsulateException;
    }

    public Decapsulator newDecapsulator(PrivateKey sk)
            throws InvalidKeyException;
    public Decapsulator newDecapsulator(PrivateKey sk, AlgorithmParameterSpec spec)
            throws InvalidAlgorithmParameterException, InvalidKeyException;

}

The getInstance methods create a new KEM object that implements the specified algorithm. The sender calls one of the newEncapsulator methods, which takes the receiver’s public key and returns an Encapsulator object. The sender can then call one of that object’s encapsulate methods to get an Encapsulated object, which contains a SecretKey and a key encapsulation message.

The receiver calls one of the newDecapsulator methods, which takes the receiver’s private key and returns a Decapsulator object. The receiver can then call one of that object’s decapsulate methods, which takes the received key encapsulation message and returns the shared secret.

The introduction of the KEM API in OpenJDK is a significant step forward in modern cryptography, providing a more secure and efficient way to secure symmetric keys using public key cryptography. It is expected to play a crucial role in defending against quantum attacks and enhancing the security of higher-level protocols and cryptographic schemes.

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

InfluxData has lost the data of customers using its services in Australia while users in Belgium are struggling to figure out if they can restore the last 100 days.

The vendor behind the InfluxDB time series DBMS has now apologized to customers caught out when it discontinued InfluxDB Cloud service in the two regions: AWS Sydney and GCP Belgium.

According to a blog from CTO Paul Dix, the company notified customers of its decision to discontinue these services for economic reason on February 23, April 6, and May 15. It also contacted customers for whom it had other details and updated the homepage of the UI for InfluxDB Cloud 2 in those regions with a notice that the service was going to be shut down on June 30.

“In hindsight, our assumption that the emails, sales outreach, and web notifications would be sufficient to ensure all users were aware of and acted on the notifications was overly optimistic,” he said.

“Our engineering team is looking into whether they can restore the last 100 days of data for GCP Belgium. It appears at this time that for AWS Sydney users, the data is no longer available.”

Users took to online forums to vent their frustration at the move.

On the vendor’s community forum, one said: “We were never informed about this. We have a running use case and are not in the habit of checking the documentation every week just in case our service gets cancelled without prior warning. InfluxData should also have seen that instances in these regions still had read and write access and informed all affected customers. This is highly unprofessional.”

On Hacker News, a user opined: “Your number one expectation as a cloud database provider is to keep data safe and recoverable.” Another said: “This is pretty much corporate suicide. I really don’t understand what they are trying to achieve with this and their attitude in this thread is baffling.”

CTO Dix, who responded to comments on the thread, initially appeared to be defensive. “I realize that it’s not ideal that we’ve shut down this system, but we made our best efforts to notify affected users and give them options to move over to other regions,” he said.

He later seemed more contrite. “It’s a terrible situation and we failed on many levels on this one. We will improve our process from here and conduct a more full postmortem.”

A number of comments on both threads pointed to InfluxData failing to carry out a “scream test,” where a service provider turns off access to a service, but does not kill the service itself. When those who are locked out “scream” via email or the phone, the company can tell them the services will be turned off for good at a later date, giving customers time to back up their data and migrate applications.

In the blog, Dix promised to do things differently in the future. He said the company would create a separate category of “Service Notification” emails that customers could not opt out of. He promised to improve email processes and clarity. The company would redouble its efforts to contact users who have not reduced their reads or writes within the 30 or 45 days before the end-of-life date for the region.

He also said the company would use a scream test and implement a 30-day data retention grace period, and publish a banner at the top of the status.influxdata.com page as soon as the initial notifications went out.

While it may be reassuring that InfluxData hopes to improve its customer communication next time it turns off a service, it will be little consolation to those who have lost data.

The Register has asked InfluxData to comment. ®

Posted on mongodb google news. Visit mongodb google news

Last month MongoDB announced its public preview of Vector Search among the updates to its developer platform of its Atlas database-as-a-service. The move means document database MongoDB joins Cassandra, PostgreSQL and SingleStore among the systems supporting similar features as the interest in putting large language models (LLMs) into production gathers pace.

LLMs have received a great deal of hype in the last six months, with OpenAI’s GPT 4.0 sucking up the lion’s share of media airtime. The idea is to extract some meaning – in the form of natural language question answering from a corpus of text. The relationships between words, sentences and other textual units are represented as multi-dimensional vectors (sometimes running into hundreds of dimensions), which are then resolved to find the most likely association.

Anticipating the boom in this form of analysis of text and other data, a group of vendors have developed specialist databases with architectures designed specifically for the task. The question is whether it is better to employ a database or use new features of a system already familiar to developers and enterprises, with a home marked out in the technology stack.

However, MongoDB argues that single-purpose databases for use cases like vector stores were often bolted on to existing technology stacks, and therefore led to greater administrative complexity and longer time to value. The approach also required developers to learn a new system.

Speaking to The Register, Ben Flast, product management lead for Vector Search, said high-dimensional vectors could be stored inside JSON documents around which MongoDB is designed.

“It’s quite straightforward to include these high-dimensional vectors inside of your documents,” he said. “As you look to add semantic search as a capability to your application and other new use cases around LLMs and chat bots arise, [you can] take that same data you were storing inside of your MongoDB deployment. You can embed or vectorize it, and add that vector to the individual documents and then create an index on it. We then manage all of the complexity behind the scenes in terms of having that index and supporting those queries.”

Other popular developer databases including open source relational system PostgreSQL and wide-column store Cassandra support similar features. Pgvector is an open source vector extension for similarity search for PostgreSQL. “Since on vector embeddings you can use AI tools for capturing relationships between objects (vector representations), you are also able to identify similarities between them in an easily computable and scalable manner,” according to database service provider Aiven.

The Cassandra features are available in DataStax’s database service Astra and are set for inclusion in open source Cassandra 5.0 later this year. Patrick McFadin, Apache Cassandra committer and developer relations veep at DataStax, told The Register: “Several new startups have created a business by constructing a specialized vector search database. However, this limited approach avoids other crucial data in AI workloads. These startups have taken a single function and attempted to transform it into a separate product.

“While vector search was once a niche industry requirement, these new products only fit those niche requirements. Nowadays, as vector search has become a mainstream requirement, mainstream databases are incorporating vector search as a feature for their developers.”

But for the specialists, it is a question of scale and performance, not developer convenience, that will ensure continuing demand for their approach.

Built by the team behind Amazon Sagemaker, Pinecone is designed to allow machine learning engineers to search through catalogues of embeddings, the continuous vector representations of separate variables fundamental to common ML algorithms. In April, it raised $100 million in series B funding, resulting in an estimated value of $750 million.

Speaking to The Register, Pinecone product veep Elan Dekel said that while every database is likely to feature some kind of vector support in the near future, they might not be the most effective approach for all use cases.

“If your use case is relatively small, then [a general purpose system is] probably enough,” he said. “But at some point, you’re going to realize that you’re starting to break the limits of the existing architecture. When you want to hit a real production scale, the retrofitting the existing solutions will mean the cost will explode to get this performance.

“If your use case is relatively small, or you don’t care about performance, you will be fine. There’ll be like this mid-tier of use cases where you can happily continue, but as you get to sort of real production scale, you will start to reach the limits of the existing systems. If you want high performance, support for high scale systems and you want it efficiently, at a reasonable cost, you’ll ultimately realise that you need a purpose-built database.”

Peter Zaitsev, an expert in MySQL performance and founder of database service company Percona, said there would not be not a single answer to the dilemma.

“Quite often, in the early stage, there are multiple technologies that appear on the market with slightly different approaches, features and focus, and it will take time for the market to settle,” he told The Register.

“In the end, I expect the SQL standard will include some things to support vector search applications, and we will have some unique extensions in various existing databases, whether relational, document and so on. Alongside these, we will get between three and five special purpose vector databases controlling 95 percent of the special purpose vector database market.”

Among specialist vector database systems, Pinecone is joined by Weaviate, Qdrant, Milvus, and Vespa.

Noel Yuhanna, veep and principal analyst with Forrester Research, said he was hearing positive feedback from organizations using these systems, which promise access control, high availability, transformation, query optimization, resource management, scalability, concurrency, and fast data queries that help support LLMs.

However, developer familiarity would be a strong draw towards most established databases supporting vector analysis.

“While native vector databases will stand out, having better performance and scale, we will likely see organizations also leveraging traditional databases with vector capabilities that need more integrated data comprising systems of record, systems of engagement, and vector data to deliver much richer LLM applications with less coding,” he said.

Poster child of the current LLM hype machine OpenAI was valued at around $29 billion earlier this year as it inhaled a $300 million investment. If business applications reflect anything like that kind of interest, the best supporting databases will rage for some time. ®

Article originally posted on mongodb google news. Visit mongodb google news

Posted on mongodb google news. Visit mongodb google news

Asset Management One Co. Ltd. boosted its stake in MongoDB, Inc. (NASDAQ:MDB – Free Report) by 7.1% during the first quarter, according to its most recent 13F filing with the Securities and Exchange Commission. The institutional investor owned 64,283 shares of the company’s stock after buying an additional 4,268 shares during the quarter. Asset Management One Co. Ltd. owned approximately 0.09% of MongoDB worth $14,782,000 at the end of the most recent reporting period.

A number of other hedge funds and other institutional investors have also recently bought and sold shares of MDB. Cherry Creek Investment Advisors Inc. lifted its position in shares of MongoDB by 1.5% during the 4th quarter. Cherry Creek Investment Advisors Inc. now owns 3,283 shares of the company’s stock worth $646,000 after purchasing an additional 50 shares during the last quarter. Allworth Financial LP lifted its position in shares of MongoDB by 12.9% during the 4th quarter. Allworth Financial LP now owns 508 shares of the company’s stock worth $100,000 after purchasing an additional 58 shares during the last quarter. Cetera Advisor Networks LLC lifted its position in shares of MongoDB by 7.4% during the 2nd quarter. Cetera Advisor Networks LLC now owns 860 shares of the company’s stock worth $223,000 after purchasing an additional 59 shares during the last quarter. First Republic Investment Management Inc. lifted its position in shares of MongoDB by 1.0% during the 4th quarter. First Republic Investment Management Inc. now owns 6,406 shares of the company’s stock worth $1,261,000 after purchasing an additional 61 shares during the last quarter. Finally, Janney Montgomery Scott LLC lifted its position in shares of MongoDB by 4.5% during the 4th quarter. Janney Montgomery Scott LLC now owns 1,512 shares of the company’s stock worth $298,000 after purchasing an additional 65 shares during the last quarter. 89.22% of the stock is owned by hedge funds and other institutional investors.

MongoDB Stock Up 1.5 %

NASDAQ MDB opened at $394.62 on Tuesday. MongoDB, Inc. has a 52-week low of $135.15 and a 52-week high of $418.70. The firm has a 50-day moving average price of $333.01 and a 200 day moving average price of $254.67. The company has a debt-to-equity ratio of 1.44, a current ratio of 4.19 and a quick ratio of 4.19.

MongoDB (NASDAQ:MDB – Get Free Report) last issued its earnings results on Thursday, June 1st. The company reported $0.56 earnings per share (EPS) for the quarter, beating analysts’ consensus estimates of $0.18 by $0.38. MongoDB had a negative net margin of 23.58% and a negative return on equity of 43.25%. The company had revenue of $368.28 million during the quarter, compared to analysts’ expectations of $347.77 million. During the same period last year, the company posted ($1.15) earnings per share. The firm’s quarterly revenue was up 29.0% compared to the same quarter last year. On average, equities research analysts predict that MongoDB, Inc. will post -2.8 earnings per share for the current fiscal year.

Insider Buying and Selling at MongoDB

In other MongoDB news, Director Dwight A. Merriman sold 2,000 shares of the stock in a transaction dated Thursday, May 4th. The shares were sold at an average price of $240.00, for a total value of $480,000.00. Following the completion of the transaction, the director now directly owns 1,223,954 shares of the company’s stock, valued at $293,748,960. The sale was disclosed in a filing with the SEC, which is available at this link. In other news, Director Dwight A. Merriman sold 2,000 shares of the firm’s stock in a transaction that occurred on Thursday, May 4th. The shares were sold at an average price of $240.00, for a total transaction of $480,000.00. Following the completion of the transaction, the director now directly owns 1,223,954 shares of the company’s stock, valued at approximately $293,748,960. The sale was disclosed in a filing with the SEC, which is available at the SEC website. Also, CEO Dev Ittycheria sold 50,000 shares of the firm’s stock in a transaction that occurred on Wednesday, July 5th. The stock was sold at an average price of $407.07, for a total transaction of $20,353,500.00. Following the transaction, the chief executive officer now directly owns 218,085 shares of the company’s stock, valued at $88,775,860.95. The disclosure for this sale can be found here. Over the last three months, insiders sold 116,821 shares of company stock valued at $41,133,220. Corporate insiders own 4.80% of the company’s stock.

Wall Street Analyst Weigh In

A number of equities research analysts recently weighed in on MDB shares. William Blair reissued an “outperform” rating on shares of MongoDB in a research report on Friday, June 2nd. Royal Bank of Canada boosted their price target on shares of MongoDB from $400.00 to $445.00 in a research report on Friday, June 23rd. 22nd Century Group reissued a “maintains” rating on shares of MongoDB in a research report on Monday, June 26th. Oppenheimer boosted their price target on shares of MongoDB from $270.00 to $430.00 in a research report on Friday, June 2nd. Finally, Stifel Nicolaus boosted their price target on shares of MongoDB from $375.00 to $420.00 in a research report on Friday, June 23rd. One equities research analyst has rated the stock with a sell rating, three have issued a hold rating and twenty-one have given a buy rating to the company. Based on data from MarketBeat.com, the company currently has a consensus rating of “Moderate Buy” and a consensus target price of $366.30.

About MongoDB

(Free Report)

MongoDB, Inc provides general purpose database platform worldwide. The company offers MongoDB Atlas, a hosted multi-cloud database-as-a-service solution; MongoDB Enterprise Advanced, a commercial database server for enterprise customers to run in the cloud, on-premise, or in a hybrid environment; and Community Server, a free-to-download version of its database, which includes the functionality that developers need to get started with MongoDB.

Featured Articles

Receive News & Ratings for MongoDB Daily – Enter your email address below to receive a concise daily summary of the latest news and analysts’ ratings for MongoDB and related companies with MarketBeat.com’s FREE daily email newsletter.

Article originally posted on mongodb google news. Visit mongodb google news

Article originally posted on InfoQ. Visit InfoQ

AWS recently announced the launch of the remote invoke feature in AWS Serverless Application Model (SAM) Command Line Interface (CLI) for AWS Lambda, allowing developers to quickly invoke their AWS Lambda functions deployed to the AWS cloud.

AWS SAM CLI is a developer tool that makes it easier to build, test, package, and deploy serverless applications. With the added remote invoke feature, developers can use the SAM CLI to test a Lambda function with a simple command sam remote invoke, which takes an event payload and Lambda logical id as input to invoke the Lambda function, then outputs the response. It supports multiple invocation modes such as response streaming, asynchronous invoke, dry run, and request response. With this feature, developers can speed up the build-deploy-test iteration loop along with the sam sync command.

According to the documentation, when a developer invokes a Lambda function with sam remote invoke, the AWS SAM CLI returns a response that contains the following:

• Request metadata – Metadata associated with the request. This includes a request ID and request start time.
• Lambda response – The response from your Lambda function after being invoked in the cloud.

In addition, a developer can use the –output option to configure the AWS SAM CLI output response in json or text.

sam remote invoke arn:aws:lambda:us-west-2:012345678910:function:sam-app-HelloWorldFunction-ohRFEn2RuAvp

Invoking Lambda Function                                                         

arn:aws:lambda:us-west-2:012345678910:function:sam-app-HelloWorldFunction-ohRFEn2RuAvp                                                                       

START RequestId: 9be48562-f991-4547-9f11-97f2f339b9f4 Version: $LATEST

END RequestId: 9be48562-f991-4547-9f11-97f2f339b9f4

REPORT RequestId: 9be48562-f991-4547-9f11-97f2f339b9f4  Duration: 5.45 ms       Billed Duration: 6 ms     Memory Size: 128 MB     Max Memory Used: 67 MB  Init Duration: 160.95 ms

{"statusCode":200,"body":"{"message":"hello world"}"}%

Eric Johnson, a Principal Developer Advocate at AWS, explains in an AWS Compute blog post how to test the potential sam remote invoke command:

To test the remote invoke feature, a small AWS SAM application comprises two AWS Lambda functions. The TranslateFunction takes a text string and translates it to the target language using the AI/ML service Amazon Translate. The StreamFunction generates data in a streaming format.

Using either of these approaches, AWS SAM will present the response from the Lambda function as if it was invoked through a service like Amazon API Gateway. However, AWS SAM also provides the option to obtain the raw response directly from the Python software development kit (SDK), boto3. This format includes additional details such as the invoked version, retry attempts, and other relevant information. To retrieve this output, execute the invocation with the extra parameter “–output” and set its value to “json.”

Source: https://aws.amazon.com/blogs/compute/testing-aws-lambda-functions-with-aws-sam-remote-invoke/

The AWS SDKs provide various choices for invoking Lambda functions through the Lambda service, with AWS SAM utilizing boto3 for remote invocation. AWS SAM introduces the “–parameter” flag to leverage the SDK options for Lambda function invocation, which can be used multiple times.

Additionally, while the sam logs feature in AWS SAM assists in viewing aggregated logs and traces from Amazon CloudWatch and AWS X-Ray, the new remote invoke function focuses on providing contextual logs specific to individual function invocations, catering to developers’ needs during testing.

Lastly, the remote invoke feature is available with SAM CLI version 1.88.0+; more details are available in the documentation.

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

• ScyllaDB NoSQL will provide scalable, resilient persistent storage required for building robust supply chain solutions over the distributed ledger technology (DLT).
• With data interoperability, scalable data stores, and permissionless feeless DLT, IOTA seeks to solve issues around cross-border trade and end-to-end supply chain traceability.

For years, the IOTA Foundation has been working on building robust and transparent supply chain solutions using distributed ledger technology (DLT). IOTA is now close to realizing this vision in East Africa amid the collaboration between the IOTA Foundation and TradeMark East Africa (TEMA).

At ScyllaDB Summit 2023, these two organizations joined hands to address the challenge surrounding the digitization of Kenya’s exporters,  airliners, and freight forwarders. As per TMEA, an African entrepreneur completes a total of 200 communications, including 96 paper documents, just for a single transaction.

With IOTA in the picture, TMEA proposes moving the key trade documents on the IOTA Tangle ledger and then sharing them with customs in the destination countries. This would expedite the export process while making African companies competitive globally.

However, the digitization of the supply chain comes with its own technical challenges. José Manuel Cantera, Technical Analyst & Project Lead at IOTA Foundation, talked of the three most pressing challenges with supply chain digitization.

• First, verifying the identity of multiple actors and systems involved in the supply chain is crucial. This includes suppliers, OEMs, food processors, brands, recycling agents, consumers, ports, carriers, ground transporters, inspectors/authorities, freight forwarders, customs, dealers, repairers, and others.
• Second, there are various relationships between these actors, which often cross borders and lack a central anchor or single source of truth. These relationships include not only business-to-business and business-to-consumer, but also business-to-government and government-to-government.
• Third, there are different functional needs to ensure trust among the actors through verifiable data. Traceability plays a vital role in achieving this. It enables compliance, product authenticity, transparency, and provenance for various applications. For instance, traceability is essential for ethical sourcing, food safety, and effective recalls.

Solving Cross-Border Trade and End-to-End Supply Chain Traceability

Cantera explains that cross-border trade operations “is a multilayered domain, and there are many different problems that have to be solved in different places”. The key processes involved here are financial procedures, transportation procedures, trade procedures, and regulator procedures.

The IOTA Foundation has initiated some key measures to address the complexity involved with the multi-layered cross-border trade. Cantera explained:

We are allowing different actors, different government agencies and the private actors (traders) to share documents and to verify documents in one shot. Whenever a consignment moves between East Africa and Europe, all the trade certificates, all the documents can be verified in one shot by the different actors, and the authenticity and the provenance of the documents can be traced properly. And as a result, the agility of the trade processes is improved. It’s more efficient and more effective.

Follow us for the latest crypto news!

Another major use case of the IOTA’s solution is end-to-end supply chain traceability.

When it comes to cross-border trade, one challenge is tracing the origin and history of trade items, ensuring reliability in sustainability claims. Traceability means being able to track the history, distribution, location, and application of products to verify claims related to human rights, labor, environment, and anti-corruption.

Let’s consider a bag of potato chips as an example. To trace its history, we need to know details such as the origin of the potatoes, the fertilizer used in their growth, the source of oils used in processing, and other relevant information. Each step along the supply chain is a critical event with key data elements describing who, what, when, where, why, and how.

In summary, traceability involves following the entire journey of trade items, including their origins, production processes, and various components, to ensure transparency and verification of claims.

IOTA Foundation To Address the Top Technical Challenges

In order to address the technical challenges, the IOTA Foundation shall be applying core technologies such as data interoperability, scalable data resources, and Scalable, permissionless feeless distributed ledger technology.

• Data Interoperability: For effective data interchange among different actors in various industries, standardized syntax and reference vocabularies are necessary to ensure semantic interoperability. Technologies such as W3C with JSON-LD, GS1 with EPCIS 2.0, and UN/CEFACT provide the means for data standardization and extensibility.
  IOTA also utilizes sectoral standards like DCSA, MOBI, and the Global Dialogue on Seafood Traceability to facilitate data interoperability. Notably, IOTA played a significant role in developing EPCIS 2.0, which enables stakeholders to share transactional information about objects using a JSON-based serialization format and REST APIs.
• Scalable Data Stores with ScyllaDB NoSQL: Storing and managing vast amounts of data associated with each supply chain event posed a scalability challenge. ScyllaDB emerged as a solution, offering advantages in terms of data scalability and retention. It combines the benefits of NoSQL and SQL, providing robust schemas for trusted and reliable data.
• Scalable, permissionless feeless distributed ledger technology: IOTA’s scalable, permissionless, and feeless distributed ledger technology played a crucial role in the solution developed by the IOTA Foundation. By combining the IOTA distributed ledger with protected storages like IPFS, they enabled functionalities such as data and document verifiability, auditability, and immutability in peer-to-peer interactions.

The IOTA Foundation is working on digitizing supply chains in East Africa using open-source distributed ledger technology and ScyllaDB NoSQL. They aim to track and trace trade items through distributed ledgers, making processes more efficient and effective. Challenges include verifying the identity of actors, establishing relationships, and maintaining trust through traceability.

Crypto News Flash does not endorse and is not responsible for or liable for any content, accuracy, quality, advertising, products, or other materials on this page. Readers should do their own research before taking any actions related to cryptocurrencies. Crypto News Flash is not responsible, directly or indirectly, for any damage or loss caused or alleged to be caused by or in connection with the use of or reliance on any content, goods, or services mentioned.

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

<meta name="x-tns-categories" content="Data / Kubernetes / Software Development“><meta name="x-tns-authors" content="Igor Domrev“>

2023-07-10 11:50:56

Running ScyllaDB NoSQL on Kubernetes with Spot Instances

sponsor-scylladb,sponsored-post-contributed,

A look at how and why visual testing vendor Visually.io chose ScyllaDB to replace MongoDB as its main production real-time database.

Jul 10th, 2023 11:50am by

Igor Domrev

Image via Shutterstock.

Serving more than 1 million operations per second with an average latency of a few milliseconds — while reading/writing real-time user-level data that can grow to billions of rows — is not a trivial task. It requires serious infrastructure that typically has a premium price tag and requires a team of experts to operate.

What if I tell you that all you need is a Kubernetes cluster and an open source database to achieve zero downtime failovers, single-digit millisecond-level response times, both vertical and horizontal scaling, data sharding per CPU core, fully distributed read/write ops and much more? In this article, I’ll share how my team at Visually.io used ScyllaDB to replace MongoDB as our main production real-time database.

ScyllaDB is an open source NoSQL database that’s API-compatible with Apache Cassandra (and also DynamoDB). It has all the advantages of a ring architecture masterless database while avoiding all the issues Cassandra is notorious for, including Java virtual machine  issues like stop-the-world garbage collection, a large memory footprint, slow startup, just-in-time warmup and complex configuration.

ScyllaDB comes with a production-ready Helm chart, Kubernetes operator, and a plug-and-play configuration. It’s open source and it works flawlessly on spot (volatile) instances that cost 1/4 of the regular cloud compute price.

Why ScyllaDB vs. MongoDB?

All that sounds amazing, but what’s wrong with MongoDB? It’s open source and supports data sharding. But MongoDB’s architecture is quite different. It has a single point of failure: If the coordinator goes down, the database starts a failover, and it’s unavailable during that time. In addition, achieving high availability requires that every MongoDB shard runs as a replica set (more nodes). The ring architecture shared by both Cassandra and ScyllaDB is superior in this sense. Moreover, ScyllaDB’s driver is shard-aware and knows to reach the precise node/CPU that’s responsible for the queried row, which allows true distribution.

But why are high availability and zero downtime failovers so important? If you plan to run on spot instances (1/4 of the compute price), you will experience frequent (daily) failovers because Kubernetes will constantly kill and re-create nodes, which will cause all pods/processes running on them to die, including your database.

Getting up and Running on Kubernetes

First, you’ll want to run ScyllaDB locally and play. Use its drivers and run some CQL (Cassandra Query Language) as described in the docs. I used the gocql driver. Remember that ScyllaDB drivers are shard-aware, and you need to connect to the shard-aware ScyllaDB port 19042 (not the default Cassandra port on 9042).

The ScyllaDB Kubernetes Operator repo contains three Helm charts:

• scylla: The database itself. It contains the ScyllaCluster CRD (Kubernetes custom resource definition), a YAML that configures the ScyllaDB cluster, its size, resources, file system and so on.
• scylla operator: Installs a Kubernetes controller that will take this YAML and creates from it a StatefulSet, services and other Kubernetes entities.
• scylla manager: Basically a singleton service that automates tasks. It is connected to all ScyllaDB nodes and can run clusterwide tasks such as a repair or a cloud storage backup.

I used Argo CD to install and configure the charts mentioned above. It allows GitOps mechanics rollbacks and provides visibility for the things happening in Kubernetes. (Argo CD is outside the scope of this article, but basically, instead of running a Helm install command, I will be clicking a few UI buttons and pushing a few YAMLs into a git repo).

Configuring the cluster

The configuration of the operator chart is pretty straightforward. The only thing you need to define is a Kubernetes nodeSelector and taint tolerations if you need them. Define on which k8s nodes the operator can run, then it’s plug and play.

Now, we’ll move on to ScyllaDB Manager. Let’s look at the Chart.yaml:

The dependencies directive declares that scylla-manager imports scylla chart, so when you install it, you install both of them. The manager configuration (values.yaml) has a section for ScyllaDB, where all the action takes place.

The key point regarding the above configuration is the xfs storageClassName, which is advised by ScyllaDB and provides better performance. The chart does not contain the storage class definition, but you can add it yourself:

<br /> apiVersion: storage.k8s.io/v1<br /> kind: StorageClass<br /> metadata:<br /> name: xfs-class<br /> provisioner: pd.csi.storage.gke.io<br /> parameters:<br /> type: pd-ssd<br /> csi.storage.k8s.io/fstype: xfs<br /> volumeBindingMode: WaitForFirstConsumer<br /> allowVolumeExpansion: true

1 2 3 4 5 6 7 8 9 10

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata:   name: xfs–class provisioner: pd.csi.storage.gke.io parameters:   type: pd–ssd   csi.storage.k8s.io/fstype: xfs volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true

Notice the allowVolumeExpansion flag. It will allow you to later increase the PersistentVolumeClaim (PVC) disk size seamlessly while the database is running. After Argo CD installed both charts, here is the result:

ScyllaDB Operator

ScyllaDB Operator is up and running. A thing to note here is that the operator itself is highly available and has two replicas of its own. It will now create the ScyllaDB cluster based on its CRD.

ScyllaDB cluster

In our case, the operator created a cluster of three nodes. Every pod is running the database itself, ScyllaDB Manager and operator clients. This helps replace “the team of experts” and automates administration and operation tasks.

Monitoring

No production database can exist without proper monitoring and alerting in place. ScyllaDB Operator achieves this with the Prometheus service monitor configuration.

<br /> scylla:<br /> …<br /> serviceMonitor:<br /> promRelease: staging-prometheus-operator<br /> create: true

scylla: …   serviceMonitor:       promRelease: staging–prometheus–operator       create: true

This flag causes the operator to create two service monitors.

ServiceMonitor

This will cause Prometheus to scrape the database metrics periodically, store them in a time series database and allow running promQL queries to define Grafana dashboards and alerts.

Dashboards

Grafana dashboards.

Grafana JSON dashboards can be found here. Here’s how to add them to the Helm charts that ScyllaDB provides.

To do that, we need to create Kubernetes ConfigMaps and label them as Grafana dashboards. Fortunately, Helm can help us with that.

<br /> {{- range $path, $_ := .Files.Glob “dashboards/scylla/*.json” }}<br /> {{- $filename := trimSuffix (ext $path) (base $path) }}<br /> apiVersion: v1<br /> kind: ConfigMap<br /> metadata:<br /> name: scylla-dashboard-{{ $filename }}<br /> namespace: monitoring<br /> labels:<br /> grafana_dashboard: “1”<br /> app.kubernetes.io/managed-by: {{ $.Release.Name }}<br /> app.kubernetes.io/instance: {{ $.Release.Name }}<br /> data:<br /> {{ base $path }}: |-<br /> {{ $.Files.Get $path | indent 4 }}<br /> —<br /> {{- end }}

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

{{- range $path, $_ :=  .Files.Glob  “dashboards/scylla/*.json” }} {{- $filename := trimSuffix (ext $path) (base $path) }} apiVersion: v1 kind: ConfigMap metadata:   name: scylla–dashboard-{{ $filename }}   namespace: monitoring   labels:     grafana_dashboard: “1”     app.kubernetes.io/managed–by: {{ $.Release.Name }}     app.kubernetes.io/instance: {{ $.Release.Name }} data:   {{ base $path }}: |- {{ $.Files.Get $path | indent 4 }} — {{- end }}

The above snippet will result in five config maps being added to Kubernetes and labeled with grafana_dashboard: "1"(which will cause Grafana to mount them).

ScyllaDB overview dashboard – view 1

ScyllaDB overview dashboard – view 2

There are many graphs with nuanced metrics exported, which allows fine-grained monitoring for everything the database experiences. The following graph is very important; it describes all the failovers in the past 24 hours.

13 fails over in 24 hours

Every time Kubernetes kills a random spot instance, it then schedules a new ScyllaDB pod, which rejoins the cluster, without any downtime, in a couple of minutes.

We have been running ScyllaDB for almost a year now, and it works like clockwork. A useful tip here is to overprovision the node pool by one node at all times. This will, most likely, ensure that there is at least one available node that can be scheduled with the new database pod. It increases the price a bit, but it’s still much more cost-efficient than using regular nodes.

Failovers / RAM / CPU / latency

The above image shows that every time a ScyllaDB instance is killed, a short CPU spike occurs, the latency is increased by a couple of milliseconds, and the RAM is dropped since all the cache ScyllaDB was building in RAM disappears. This is a clear disadvantage of using spot instances. However, in our use case it’s worth trading short, very small latency spikes for a large compute price discount.

Conclusion

In conclusion, ScyllaDB proves to be an exceptional open source database that lives up to its promises. The fact that ScyllaDB is freely available as open source is truly remarkable. As a software developer, I have no affiliation with ScyllaDB, but I am grateful for the technology it provides. This article serves as a heartfelt thank you to the ScyllaDB community for its dedication to open source and for empowering developers like myself with such remarkable technology.

Created with Sketch.

Igor Domrev is a seasoned senior software developer with a Bachelor of Science in computer science and a decade of experience, including a successful stint at a startup that he joined in its early days that achieved unicorn status. Currently,…

Read more from Igor Domrev

TNS owner Insight Partners is an investor in: Pragma.

SHARE THIS STORY

THE NEW STACK UPDATE
A newsletter digest of the week’s most important stories & analyses.

Posted on mongodb google news. Visit mongodb google news

Worldwide Market Reports (WMR), a leading provider of market research and analysis, is proud to announce the release of its latest comprehensive market research report titled “HTAP-Enabling In-Memory Computing Technologies Market Research Report 2023 – Unveiling Key Insights and Trends.” This groundbreaking report offers valuable insights and trends in this sector, providing businesses and stakeholders with a deep understanding of the market dynamics and growth opportunities.

Backed by extensive data analysis and expert assessments, WMR conducted an in-depth study of the HTAP-Enabling In-Memory Computing Technologies market, exploring key factors influencing its growth, emerging trends, competitive landscape, and consumer preferences. The HTAP-Enabling In-Memory Computing Technologies market report offers a holistic view of the market, encompassing both qualitative and quantitative analysis, empowering businesses to make informed decisions and drive strategic growth initiatives.

“This HTAP-Enabling In-Memory Computing Technologies market research report is a game-changer for businesses operating in this sector,” said our Research Head. “It not only provides a comprehensive overview of the market but also sheds light on crucial insights that can help businesses identify untapped opportunities, understand consumer behavior, and stay ahead of the competition. We are thrilled to contribute to the industry’s knowledge base and support our clients in making data-driven decisions.”

Request a Sample Copy of Our Cutting-Edge HTAP-Enabling In-Memory Computing Technologies Market Report Today! @ https://www.worldwidemarketreports.com/sample/923383

The HTAP-Enabling In-Memory Computing Technologies Market report covers extensive competitive intelligence which includes the following data points:

Business Overview
Business Model
Financial Data
Financial – Existing
Financial – Funding
Product/Service Segment Analysis and specification
Recent Development and Company Strategy Analysis
SWOT Analysis

This Study Covers the Following Key Players:

❖ Microsoft
❖ IBM
❖ MongoDB
❖ SAP
❖ Aerospike
❖ DataStax
❖ GridGain

The HTAP-Enabling In-Memory Computing Technologies Market report covers a wide range of critical topics, including Types, Applications, and Regions. Its findings have the potential to shape industry strategies, influence investment decisions, and drive innovation.

HTAP-Enabling In-Memory Computing Technologies Market Segmentation Analysis:

By Type –

❖ Single Node Based
❖ Distributed Systems Based
❖ Hybrid Memory Structure Based

By Application –

❖ Retail
❖ Banks
❖ Logistics
❖ Others

The global analysis of the HTAP-Enabling In-Memory Computing Technologies market identifies various factors that impact its performance across different regions, including:

» North America (United States, Canada, and Mexico)
» Europe (Germany, France, UK, Russia, and Italy)
» Asia-Pacific (China, Japan, Korea, India, and Southeast Asia)
» South America (Brazil, Argentina, Colombia)
» Middle East and Africa (Saudi Arabia, UAE, Egypt, Nigeria, and South Africa)

Speak to Our Analyst and Gain Crucial Industry Insights that will Help Your Business Grow @ https://www.worldwidemarketreports.com/quiry/923383

Scope of the HTAP-Enabling In-Memory Computing Technologies Market:

The Global HTAP-Enabling In-Memory Computing Technologies market is anticipated to rise at a considerable rate during the forecast period, between 2023 and 2030. In 2022, the market is growing at a steady rate, and with the rising adoption of strategies by key players, the market is expected to rise over the projected horizon.

Key Benefits for Stakeholders:

1. The study represents a quantitative analysis of the present HTAP-Enabling In-Memory Computing Technologies Market trends, estimations, and dynamics of the market size from 2022 to 2030 to determine the most promising opportunities.
2. Porter’s five forces study emphasizes the importance of buyers and suppliers in assisting stakeholders to make profitable business decisions and expand their supplier-buyer network.
3. In-depth analysis, as well as the market size and segmentation, help you identify current HTAP-Enabling In-Memory Computing Technologies Market opportunities.
4. The largest countries in each region are mapped according to their revenue contribution to the market.
5. The HTAP-Enabling In-Memory Computing Technologies Market research report gives a thorough analysis of the current status of the HTAP-Enabling In-Memory Computing Technologies Market’s major players.

Reasons to Purchase HTAP-Enabling In-Memory Computing Technologies Market Report:

✦ Both current and future prospects for the HTAP-Enabling In-Memory Computing Technologies Market in developed and emerging markets.
✦ Analysis of various perspectives of the market with the help of Porter’s five forces analysis.
✦ During the forecast period, major regions are expected to see the most rapid increase.
✦ Identify the most recent advancements, HTAP-Enabling In-Memory Computing Technologies Market shares, and top market players’ strategies.

Embrace the Latest Insights: Get Your Hands on the Newest Edition of the HTAP-Enabling In-Memory Computing Technologies Market Report, Now at an Unbeatable Discount! (Up to 70%) @ https://www.worldwidemarketreports.com/promobuy/923383

Frequently Asked Questions (FAQs):

➟ What is the main driving factor for the growth of the global HTAP-Enabling In-Memory Computing Technologies market?
➟ What are the restraining factors of the market?
➟ Who are the key market players?
➟ Which region holds the biggest market share?
➟ What are the recent trends of the global HTAP-Enabling In-Memory Computing Technologies market?

» Contact Us:

Worldwide Market Reports,
Tel: U.S. +1-415-871-0703
U.K. +44-203-289-4040
Japan +81-50-5539-1737
Email: [email protected]
Website: https://www.worldwidemarketreports.com

Article originally posted on mongodb google news. Visit mongodb google news