Month: June 2023

Article originally posted on InfoQ. Visit InfoQ

Transcript

Wang: My name is Allen Wang. I’ve been building real time data infrastructure for the past seven years. First at Netflix for the Keystone Data Pipeline and currently at DoorDash, building a real time event processing system called Iguazu. The journey of building those systems give me an opportunity to find some common patterns in successful real time data architectures.

Successful Patterns in Building Realtime Event Processing

In my mind, there are four principles that have passed the test of time. First, decoupling stages. We’ll talk about what those stages are, and why, and how to make that decoupling happen. Second, leveraging stream processing framework. We will cover what stream processing framework can do for you and how you can pick one. Third, creating abstractions. For that, I will list the common abstractions that you can create to facilitate the adoption of your system. Finally, fine-grained failure isolation and scalability. Failures will happen, how do you isolate the failures and avoid bottleneck in scalability? I know those concepts may sound a bit abstract at this point, so I’m going to use a real time event processing system we created at DoorDash as an example, to give you some concrete ideas.

Real Time Events Use Case at DoorDash

First, let’s talk about use cases of real time events. At DoorDash, real time events are an important data source to gain insight into our business, and help us make data driven decisions, just to give you a few examples how real time events are used at DoorDash. First, almost all events need to be reliably transported to our data warehouse, Snowflake, or other online analytical datastores with low latency for business analysis. For example, the dasher assignment team relies on the assignment events to detect any bugs in their algorithm at near real time. The second use case is mobile application health monitoring. Some mobile events will be integrated with our time series metric backend for monitoring and alerting so that the teams can quickly identify issues in the latest mobile application releases. The third use case is sessionization. We will like to group user events into sessions and generate session attributes at real time so that we can better analyze the user behaviors and push real time recommendations.

Historically, DoorDash has a few data pipelines that get data from our legacy monolithic web application and ingest the data into Snowflake. Each pipeline is built differently and can only handle one event. They involve mixed data transports, multiple queuing systems, and multiple third-party data services, which make it really difficult to control the latency, maintain the cost, and identify scalability bottlenecks in operations. We started to rethink this approach and decided to build a new system to replace those legacy pipelines and address the future event processing needs that we anticipated. First, it should support heterogeneous data sources, including microservices, and mobile or web applications, and be able to deliver the events to different destinations. Second, it should provide low latency and reliable data ingest into our data warehouse with a reasonable cost. Third, real time events should be easily accessible for data consumers. We want to empower all teams to create their own processing logic and tap into the streams of real time data. Finally, to improve data quality, we want to have end-to-end schema enforcement and schema evolution.

Introducing Iguazu

Two years ago, we started the journey of creating from scratch a real time event processing system named Iguazu. The important design decisions we made is to shift the strategy from heavily relying on third party data services to leveraging open source frameworks that can be customized and better integrates with the DoorDash infrastructure. Fast forward to today, we scaled Iguazu from processing just a few billion events to hundreds of billions of events per day with four nines of delivery rate. Compared to the legacy pipelines, the end-to-end latency to Snowflake is reduced from a day to just a few minutes. This is the architecture overview of Iguazu. The data ingestion for microservices and mobile clients are enabled through our Iguazu Kafka proxy before it lands into Kafka. We use Apache Kafka for Pub/Sub and decoupling the producers from consumers. Once the data lands into Kafka, we use stream processing applications built on top of Apache Flink for data transformation. That is achieved through Flink’s data stream APIs and Flink SQL. After the stream process is done, the data is sent to different destinations, including S3 for data warehouse, and data lake integrations, Redis for real time features, and Chronosphere for operational metrics. In the following slides, I’ll discuss the details of each of those major components.

Simplify and Optimize Event Producing

Let’s first talk about producing events. The main focus is to make event producing as easy as possible and optimized for the workload of analytical data. To give more context, let me first explain the requirements for processing analytical data. First, analytical data usually comes with high volume and growth rate. As an example, in the process of making restaurant order on DoorDash, tens of thousands of analytical events are published. As we strive to improve user experience and delivery quality, more user actions are tracked at fine granularity, and these lead to the higher growth rate of analytical data volume compared with order volume. Therefore, compared with transactional data, analytical data requires higher scalability and cost efficiency. On the other hand, because analytical data are almost always processed and analyzed in aggregated fashion, minor data loss will not affect the quality of analysis. For example, if you collect 1 million results in an A/B test, and randomly drop 100 events from the results, most likely it will not affect the conclusion of the experiment. Typically, we have found a data loss of less than 0.1% will be acceptable for analytical events.

Leveraging Kafka REST Proxy

Let’s look at the measures that we have taken to ensure the efficiency and the scalability in analytical event publishing. As I mentioned in the overview of the Iguazu architecture, we choose Kafka as the central Pub/Sub system. One challenge we face is how we can enable average DoorDash service to easily produce events through Kafka. Even though Kafka has been out there for quite some time, there are still teams struggling with the producer internals, tuning the producer properties, and making the Kafka connection. The solution we created is to leverage the Confluent Kafka REST proxy. The proxy provides a central place where we can enhance and optimize event producing functionalities. It provides the abstractions over Kafka with HTTP interface, eliminating the need to configure Kafka connections from all services and making event publishing much easier.

The Kafka REST proxy provides all the basic features that we need out of the box. One critical feature is that it supports event batching. You probably know that batching is a key factor in improving efficiencies in data processing. Why would it specifically help improving efficiency in event publishing to Kafka? What we have found out is that the Kafka brokers workload is highly dependent on the rate of the producing requests. The higher the rate of producing requests, the higher the CPU utilization on the program, and more workers will likely be needed. How can you produce to your Kafka with high data volume better than low rate of produce requests? As a formula in the slide suggests, you need to increase the number of events per request, which essentially means increasing the batch size. Batching comes in with tradeoff of higher latency in event publishing. However, processing of analytical events are typically done in an asynchronous fashion and does not require sub-second latency in getting back the final results. Small increase in the event publishing latency is an acceptable tradeoff. With Kafka REST proxy, you can batch events in each request from client side and rely on the Kafka client leader proxy to further batch them before producing to the broker. The result is that you will get the nice effect of event batching across client instances and applications. Let’s say you have an application that publishes events at a very low rate, which would lead to inefficient batching. The proxy will be able to mix these low volume events with other high-volume events in one batch. It makes event publishing very efficient and greatly reduce the workload for the Kafka brokers.

Proxy Enhancements

The Kafka REST proxy provides all the basic features we need out of the box, but to further improve its performance and scalability, we added our own feature like multi-cluster producing, and asynchronous request processing. Multi-cluster producing means the same proxy can produce to multiple Kafka clusters. Each topic will be mapped to a cluster, and this ensures that we can scale beyond one Kafka cluster. It also enables us to migrate topics from one Kafka cluster to another, which helps to balance the workload and improve the cost efficiency of Kafka clusters. Asynchronous request processing means the proxy will respond to the produce request as soon as it pulls the Kafka records into the Kafka clients producer buffer without waiting for the broker’s acknowledgment. This has a few advantages. First, it significantly improves the performance of the proxy and helps reduce the back pressure on the proxy’s clients. Second, asynchronous request processing means the proxies spend less time block waiting for the broker’s response and more time for the proxy to process requests, which lead to better batching and throughput. Finally, we understand that this asynchronous mode means clients may not get the actual clear response from the broker and may lead to data loss. To mitigate this issue, we added automated retries on the proxy side on behalf of the client. The number of retries is configurable and each subsequent retry will be done on a randomly picked partition to maximize the chance of success. The result is minimal data loss of less than 0.001%, which is well in range of acceptable data loss level for analytical events.

Event Processing with Flink

Now that I have covered events producing, let’s focus on what we have done to facilitate event consuming. One important objective for Iguazu is to create a platform for easy data processing. Apache Flink’s layered API architecture fits perfectly with this objective. We choose Apache Flink, also because of its low latency processing, native support of processing based on event time, and fault tolerance and built-in integrations with a wide range of sources and sinks, including Kafka, Redis, Elasticsearch, and S3. Ultimately, we need to understand what stream processing framework can do for you. We’ll demonstrate that by looking at a simple Kafka consumer. This is a typical Kafka consumer. First, it gets records from Kafka in a loop. Then it will update a local state using the records, and it just retrieves and produce a result from the state. Finally, it will push the result to a downstream process, perhaps over the network.

On the first look, the code is really simple and does the job. However, questions will arise over time. First, note that the code does not commit Kafka offset, and this will likely lead to failures to provide any delivery guarantee when failures occur. Where should the offset commit be added to the code to provide the desired delivery guarantee be it at-least once, at-most once, or exactly-once. Second, the local state object is stored in memory and it will be lost when the consumer crashes. It should be persisted along with the Kafka offset so that the state can be accurately restored upon failure recovery. How can we persist and restore the state when necessary? Finally, the parallelism of Kafka consumer is limited by the number of partitions of the topic being consumed. What if the bottleneck of the application is in processing of the records or pushing the results to downstream, and we need higher parallelism than the number of partitions. Apparently, it takes more than a simple Kafka consumer to create a scalable and fault tolerant application. This is where stream processing framework like Flink shines. As shown in this code example, Flink helps you to achieve delivery guarantees, automatically persists and restores application state through checkpointing. It provides a flexible way to assign compute resources to different data operators. These are just a few examples that stream processing framework can offer.

One of the most important features from Flink is layered APIs. At the bottom, process function allows engineers to create highly customized code and have precise control on handling events, state, and time. The next level up, Flink offers data stream APIs with built-in high-level functions to support different aggregations and windowing, so that engineers can create a stream processing solution with just a few lines of code. On the top we have SQL and table APIs, which offer casual data users the opportunity to write Flink applications in a declarative way, using SQL instead of code. To help people at DoorDash leverage Flink, we have created a stream processing platform. Our platform provides a base frame Docker image with all the necessary configurations that are well integrated with the rest of the DoorDash infrastructure. Flink’s high availability setup and Flink internal metrics will be available out of the box. For better failure isolation and the ability to scale independently, each Flink job is deployed in a standalone mode as a separate Kubernetes service. We support two abstractions in our platform, data stream APIs for engineers and a Flink SQL for casual data users.

You may be wondering why Flink SQL is an important abstraction we want to leverage. Here’s a concrete example. Real time features are an important component in machine learning, for model training and prediction. For example, to predict an ETA of a DoorDash delivery order requires up to date store order count for each restaurant, which we call a real time feature. Traditionally, creating a real time feature requires coding history and parsing the application and transforms and arrays the events into real time features. Creating Flink applications requires a big learning curve for machine learning engineers and becomes a bottleneck when tens or hundreds of features need to be created. The application created often have a lot of boilerplate code that are replicated across multiple applications. Engineers also take the shortcut to bundle the calculation of multiple features in one application, which lacks failure isolation or the ability to allocate more resources to a specific feature calculation.

Riviera

To meet those challenges, we decided to create a SQL based DSL framework called Riviera, where all the necessary processing logic and wiring are captured in a YAML file. The YAML file creates a lot of high-level abstractions, for example, connecting to a Kafka source, and producing to certain things. To create a real time feature, engineers only need to create one YAML file. Riviera achieved great results. The time needed to develop a new feature is reduced from weeks to hours. The feature engineering code base is reduced by 70% after migrating to Riviera. Here’s a Rivera DSL example, to show how Flink SQL is used to calculate store order count for each restaurant, which is an important real time feature used in production for model prediction.

First, you need to specify the source sink and a schema used in the application. In this case, the source is a Kafka topic and a sink is a Redis cluster. The process logic is expressed as a SQL query. You can see that we used the built-in COUNT function to aggregate the order count. We also used hot window. The hot window indicates that you want to process the data that are received in the last 20 minutes and refresh the results every 20 seconds.

Event Format and Schema: Protocol between Producers and Consumers

In the above two sections, we covered event producing and consuming in Iguazu. However, without a unified event format, it’s still difficult for producers and consumers to understand each other. Here, we will discuss the event format, and schemas, which serve as a protocol between producers and consumers. From the very beginning, we defined a unified event format. The unified event format includes an envelope and payload. The payload contains the schema encoded event properties. The envelope contains context of the event, for example, event creation time, metadata, including encoding method and the references to the schema. It also includes a non-schematized JSON blob called custom attributes. This JSON section in the envelope gives users a choice, where they can store certain data and evolve them freely without going through the formal schema evolution. This flexibility proves to be useful at an early stage of event creation, where frequent adjustments of schema definition are expected. We created serialization libraries for both event producers and the consumers to interact with this standard event format. In Kafka, the event envelope is stored as a Kafka record header, and the schema encoded payload is stored as a record value. Our serialization library takes the responsibility of converting back and forth between the event API and the properly encoded Kafka record so that the applications can focus on their main logic.

We have leveraged Confluent schema registry for generic data processing. First, let me briefly introduce schema registry. As we know, schema is a contract between producers and consumers on data or events that they both interact with. To make sure producers and consumers agree on the schema, one simple way is to present the schema as a POJO class, which is available for both producers and consumers. However, there’s no guarantee that the producers and the consumers will have the same version of the POJO. To ensure that a change of the schema is propagated from the producer to the consumer, they must coordinate on when the new POJO class will be made available for each party. Schema registry helps to avoid this manual coordination between producers and consumers on schema changes. It is a standalone REST service to store schemas and serve the schema lookup requests from both producers and consumers using schema IDs. Where schema changes are registered with schema registry, it will enforce compatibility rules and reject incompatible schema changes.

To leverage the schema registry for generic data processing, schema ID is embedded in the event payload so that the downstream consumers can look it up from schema registry without relying on the object classes on the runtime class path. Both protobuf and Avro schemas are supported in our serialization library and the schema registry. We support protobuf schema, because almost all of our microservices are based on gRPC and protobuf supported by a central protobuf Git repository. To enforce this single source of truth, avoid duplicate schema definition and to ensure the smooth adoption of Iguazu, we decided to use the protobuf as the primary schema type. On the other hand, the Avro schema is still better supported than protobuf in most of the data frameworks. Where necessary, our serialization library takes the responsibility of seamlessly converting the protobuf message to Avro format and vice versa.

One decision we have to make is when we should allow schema update to happen. There are two choices, at build time or producers runtime. It is usually tempting to let producers freely update the schema at the time of event publishing. There are some risks associated with that. It may lead to data loss because any incompatible schema change will fail the schema registry update and cause runtime failures in event publishing. It will also lead to spikes of schema registry update requests causing potential scalability issues for the schema registry. Instead, it will be ideal to register and update the schema at build time to catch incompatible schema changes early in the development cycle, and reduce the update aka call volume to the schema registry.

One challenge we faced is how we can centrally automate the schema update at build time. The solution we created is to leverage the central repository that manages all of our protobuf schemas and integrate the schema registry update as part of its CI/CD process. When a protobuf definition is updated in the pull request, the CI process will validate the change with the schema registry and it will fail if the change is incompatible. After the CI passes, and the pull request is merged, the CD process will actually register or update the schema registry and publish the compiled protobuf JAR files. The CI/CD process not only eliminates the overhead of manual schema registration, but also guarantees the early detection of incompatible schema changes and the consistency between the [inaudible 00:26:34] protobuf class families and the schemas in the schema registry. Ultimately, this automation avoids schema update at runtime and the possible data loss due to incompatible schema changes.

Data Warehouse and Data Lake Integration

So far, we have talked about event producing, consuming, and event format. In this slide, I’ll give some details on our data warehouse integration. Data Warehouse integration is one of the key goals of Iguazu. Snowflake is our main data warehouse, and we expect events to be delivered to Snowflake with strong consistency and low latency. The data warehouse integration is implemented as a two-step process. In the first step, data is consumed by a Flink application from a Kafka topic and uploaded to S3 in the parquet format. This step leverages S3 to decouple the stream processing from Snowflake so that Snowflake failures will not impact stream processing. It will also provide a backfill mechanism from S3 given Kafka’s limited retention. Finally, having the parquet files on S3 enables date lake integrations as the parquet data can be easily converted to any desired table format. The implementation of uploading data to S3 is done through Flink’s StreamingFileSink. When completing an upload as part of Flink’s checkpoint, StreamingFileSink guarantees strong consistency and exactly once delivery. StreamingFileSink also allows customized bucketing on S3 which means you can flexibly partition the data using any fields. This optimization greatly facilitates the downstream consumers.

At the second step, data is copied from S3 to Snowflake tables via Snowpipe. Snowpipe is a Snowflake service to load data from external storage at near real time. Triggered by SQS messages, Snowpipe copies data from S3 as soon as they become available. Snowpipe also allows simple data transformation during the copy process. Given that Snowpipe is declarative and easy to use, it’s a good alternative compared to doing data transformation in stream processing. One important thing to note is that each event has its own stream processing application for S3 offload and its own Snowpipe. As a result, we can scale pipelines for each event individually and isolate the failures.

Working Towards a Self-Serve Platform

So far, we covered end-to-end data flows from clients to data warehouse, and here we want to discuss the operational aspect of Iguazu and see how we are making it a self-serve to reduce operational burdens. As you recall, to achieve failure isolation, each event in Iguazu has its own pipeline from Flink job to Snowpipe. This requires a lot more setup work and makes the operation a challenge. This diagram shows the complicated steps required to onboard a new event, including creation of the Kafka topic, schema registration, creation of a Flink stream processing job, and creation of the Snowflake objects. We really want to automate all these manual steps to improve operational efficiency. One challenge we face is how we can accomplish it under the infrastructure as code principle. At DoorDash, most of our operations, from creating Kafka topic, to setting up service configurations involve some pull request to different Terraform repositories and requires code review. How can we automate all these? To solve the issue, we first worked with our infrastructure team to set up the right pull approval process, and then automate the pull request using GitHub automation. Essentially, a GitHub app is created, where we can programmatically create and merge pull requests. We also leverage the Cadence workflow engine, and implemented the process as a reliable workflow. This pull automation reduced the event onboarding time from days to minutes. We get the best of both worlds, where we achieve automations, but also get versioning, all the necessary code reviews, and consistent state between our code and the infrastructure.

To get us one step closer to self-serve, we created high level UIs for a user to onboard the event. This screenshot shows a schema exploration UI, where users can search for a schema, using brackets, pick the right schema subject and version, and then onboard the event from there. This screenshot shows the Snowpipe integration UI where users can review and create Snowflake table schemas and Snowpipe schemas. Most importantly, we created a service called Minions that does the orchestration, and have it integrated with Slack so that we get notifications on each step it carries out. On this screenshot, you can see about 17 actions it has taken in order to onboard an event, including all the pull requests, launching the Flink job, and creating Snowflake objects.

Summary

Now that we covered the architecture and designs of all the major components in Iguazu, I’d like to once again review the four principles that I emphasized. The first principle is to decouple different stages. You want to use a Pub/Sub or messaging system to decouple data producers from consumers. This not only relieves the producers from back pressure, but also increases the data durability for downstream consumers. There are a couple of considerations you can use when choosing the right Pub/Sub or messaging system. First, it should be simple to use. I tend to choose the one that’s built with a single purpose and do it well. Secondly, it should have high durability and throughput guarantees. It should be highly scalable, even when there’s a lot of consumers and high data fanout. Similarly, stream processing should be decoupled from any downstream processes, including the data warehouse, using a cost-effective cloud storage or data lake. This guarantees that in case there’s any failures in the data warehouse, you don’t have to stop the stream processing job and you can always use the cheaper storage for backfill.

As a second principle, it pays to invest on the right stream processing framework. There are some considerations you can use when choosing the right stream processing framework. First, it should support multiple abstractions so that you can create different solutions according to the type of the audience. It should have integrations with sources and sinks you need, and support a variety of data formats like Avro, JSON, protobuf, or parquet, so that you can choose the right data format in the right stage of stream processing. The third principle is to create abstractions. In my opinion, abstraction is an important factor to facilitate the adoption of any complicated system. When it comes to the real time data infrastructure, there are a few common abstractions worth considering. For example, leveraging Kafka REST proxy, creating event API and serialization library, providing a DSL framework like Flink SQL, and providing high level UIs with orchestrations. Finally, you need to have fine-grained failure isolation and scalability. For that purpose, you should aim to create independent stream processing jobs or pipelines for each event. This avoids resource contention, isolates failures, and makes it easy to scale each pipeline independently. You also get the added benefits of being able to provide different SLAs for different events and easily attribute your costs at event level. Because you are creating a lot of independent services and pipelines to achieve failure isolation, it is crucial to create orchestration service to automate things and reduce the operational overhead.

Beyond the Architecture

Here are some final thoughts that go beyond the architecture. First, I find it important to build products using a platform mindset. Ad hoc solutions are not only inefficient, but also difficult to scale and operate. Secondly, picking the right framework and creating the right building blocks is crucial to ensure success. Once you have researched and find the sweet spots of those frameworks, you can easily create new products by combining a few building blocks. This dramatically reduces the time to building your products, and the effort to maintain the platform. I would like to use a Kafka REST proxy as an example. We use it initially for event publishing from internal microservices. When the time comes for us to develop a solution for mobile events, we found the proxy is also a good fit because it supports batching and JSON payload, which are important for mobile events. With a little investment, we made it work as a batch service, which save us a lot of development time.

Questions and Answers

Anand: Would you use a REST proxy if loss is not an option?

Wang: I mentioned that we added the extra capability to do asynchronous request processing. That is designed purely for analytical data, because it would introduce maybe very minor data loss, because we are not giving the actual broker acknowledgments back to the client. You can also use the REST proxy in a different way, where it would give precisely what the broker acknowledgment is. Basically, you’re adding a date hop in the middle, but client would know definitely whether this producing request is successful or not, or it can timeout, sometimes it can timeout. Let’s say, you send something to the proxy and the proxy crashed, and then you don’t hear anything back from the proxy, but on the client side, you will know it timed out, then you will try to do some kind of retry.

Anand: I’m assuming REST requests coming to the REST proxy, you accumulate it in some buffer, but you’re responding back to the REST clients if they’re mobile or whatever, that you’ve received the data, but it’s uncommitted. Therefore, you can do larger writes and get a higher throughput sense.

Wang: Yes, of course, that also had to be changed, if you are really looking for lossless data publishing. It’s not desirable to have that behavior.

Anand: Also curious about alternatives to Flink. One person is using Akka. They’re moving off Akka towards Flink, and they were wondering what your alternative is.

Wang: I haven’t used Akka before. Flink is the primary one that I use. I think Flink probably provides more data processing capability than Akka including all the abstractions it provides, the SQL support, and the fault tolerance built on top of Flink’s internal state. I would definitely recommend using these kinds of frameworks that provide these capabilities.

Anand: Also curious about alternatives to Flink. We used to use Airflow plus Beam on data flow to build a similar framework. Did your team evaluate Beam? Why choose Flink? We usually hear about Beam and Flink, because Beam is a tier above.

Wang: I think Beam does not add a lot of advantages in streaming.

Anand: Beam is just an API layer, and it can be used with Spark streaming or with Flink.

Wang: Unless you want to have the capability of being able to migrate your workload to a completely different stream processing engine, Beam is one of those choices you can use. I also know that some of the built-in functionalities on streaming parts and edges are not transferable to another. I think most likely they’re not. You are using certain built-in functionalities of those stream processing frameworks, which is not exposed with Beam. It’s actually hard to migrate to a different one. I think Beam is a very good concept, but in reality, I feel it’s actually more efficient to stick to the native stream processing framework?

Anand: It’s a single dialect if the two engines below have the same functionality, but the two engines are diverging and adding their own, so having a single dialect is tough.

How do you handle retries in the event processing platform? Are there any persistence to the problematic events with an eventual reprocessing?

Wang: The retries, I’m sure what is that context? In both the event publishing stage, and event processing stage or consuming stage, you can do retries. Let’s say in publishing stage, of course, when you try to publish something, the broker gives you a negative response, you can always retry.

Anand: What they’re saying is, how do you handle retries in the event processing platform? The next one is, if you have a bad message, do you persist these away, like in a dead letter queue, with an eventual reprocessing of it once it’s patched?

Wang: In the context of data processing or consuming the data, if you push something to the downstream and that failed, or you’re processing something that failed, you can still do a retry, but I think dead letter queue is probably the ultimate answer. That you can push the data back to a different Kafka topic, for example, and have the application consumed again from that dead letter queue. You can add a lot of controls on like, how many times you want to retry, and how long you want to keep retrying. I think dead letter queue is essential if you really want to minimize your data loss.

Anand: Apache Kafka is a popular choice, especially for real time event stream processing, I’m curious of presenter’s opinion on simpler event services such as Amazon SNS. How does one decide if you truly need something as powerful as Kafka, or perhaps a combination of SNS and SQS for added message durability? Could they be enough for a use case?

Wang: I think SNS as a queue has helped different kinds of use cases, they are more like point-to-point messaging system. Kafka on the other side, it’s really helped emphasize on streaming. For example, in Kafka, you always consume a batch of messages and your offset commit is on the base level batch, not individual messages. Kafka has another great advantage where you can have different consumers consume from the same queue or same topic and it will not affect each other. Data fanout is a very good use case of Kafka. Having data fanout is a little bit difficult to deal with, with SNS and SQS.

Anand: I’ve used SNS, SQS together. SNS is topic based, but if a consumer joins late, they don’t get historical data, so then you basically have like multiple consumer groups as SQS topics. Then each consumer group has to link to that SQS topic. What it didn’t work for, for me, is Kafka, when the producer wants a durable, it doesn’t matter how many consumer groups you add later, different apps. With SNS, SQS, you end up adding the SQS topic, or the SQS queue for another consumer group, but there’s no historical data. I think that’s the difference. Kafka supports that, SNS, SQS doesn’t actually solve that problem.

Are there any plans in exploring Pulsar to replace Kafka? What are your thoughts on Pulsar if you already explored it?

Wang: We actually explored Pulsar in my previous job at Netflix. At that time, Pulsar was not super mature, it still had some stability issues. In the end, I think Kafka is still simple to use. I think Pulsar has tried to do a lot of things. It’s tried to accomplish both, supporting stream processing and also supporting this point-to-point messaging. It tried to accommodate both. It becomes a bit more complicated or complex architecture, than Kafka. For example, I think Pulsar relies on another service or open source framework called BookKeeper, Apache BookKeeper. There’s multiple layers of services in between. We think that deployment installation operation could be a headache, so we choose Kafka, we want to stick with Kafka for its simplicity.

Anand: You have the option of doing go time versus directly hooked in. I think that option is only at publish time, like producer time.

See more presentations with transcripts

Article originally posted on InfoQ. Visit InfoQ

Transcript

Gelobter: What I want to do here is talk a little bit about examples of how we really and truly can harness technology to solve what had been previously thought of as intractable problems. You can do that in public sector, but you can also do it in private sector. What I’d love for you all to take away from this is the idea of inspiration and aspiration, and how do you make the same practices that we use every day for things that also matter, that can have a significant impact at scale. That’s what we’ll be talking about.

I’m a black woman with a degree in computer science, which unfortunately, makes me somewhat of a unicorn. I’ll talk a little bit more about my background, just because we have gotten a lot more comfortable about talking around issues of race or gender. What we don’t necessarily talk about is socioeconomic differences. I come from a low-income background. While I do have a degree in computer science, it took me 24 years to get it. This is actually what would have been my 20th reunion. I actually graduated class of 2011. Turns out, it’s tough to pay for school when you have to work full time, and put yourself through college. I just want to really just emphasize and highlight all the differences and all the different places and things and identities and intersectionalities that affect how people show up. These are all things of course that you can and should be using as you are thinking about the products that you all are building. I have been fortunate enough to work on some pretty transformative technologies. I was a software engineer on Shockwave. Shockwave, in the early to mid-90s, literally introduced animation, multimedia interactivity to the web, like made the web move. I also helped launched Hulu. I ran digital at BET, the television network, Black Entertainment Television. Then I got a call from the White House, which is quite jarring when you are in the private sector and have always been in private sector. I was fortunate enough to go and serve in the Obama administration as the Chief Digital Service Officer for the U.S. Department of Education. That is making that leap from private sector into public sector. I’m back in private sector now, but still trying to take those same lessons learned and apply them to the scope and the magnitude of what I work on today and try to make systemic change. That’s the dream.

Energy & Impact of a Lean Tech Startup, at Federal Government Scale

Just to give you a little bit of a sense, I’m going to give you some examples of things that we’ve implemented both in the public sector, but then also private. What happened was in federal government, Todd Park was the CTO, I think at the time, of the United States of America, and so, President Obama, the idea is, if I can sit on my sofa in New York, and order a vegan burrito, and mescal, and weed, at this point and have it show up in 10 minutes. If I want to apply for SNAP benefits, for food stamps, it takes months and it’s super onerous, and super painful. How do we apply the things that we know how to do well in private sector, and bring them to the things that are really affecting the people who are most in need. That is the concept of what we tried to do when we were trying to bring best practices from private sector into government. That was the plan. It was actually founded based on healthcare.gov, or actually the failure and then the rescue of healthcare.gov. When healthcare.gov launched, it essentially imploded. The Affordable Care Act, it was a fundamental tenet of the thing that President Obama had successfully done. They implemented it and then it went spectacularly.

What ended up happening was they called in a group of folks from private sector, I think it was only six or seven people. They called it the ad hoc team. When they joined the healthcare.gov, trying to resolve the issue, healthcare.gov was registering 150 people per day. One hundred and fifty people per day were signing up for health insurance. By the time they left, 6 or 7 weeks later, it was 150,000 per day. Over time, tens of millions of people have gotten health care through it. What did they do? The challenge is, of course, there were 48 million people in the U.S. who did not have health care, and 45,000 people were literally dying every year for want of health care. What I’m trying to impress on you all is the difference that you can make using technology, like going from 150 people every day to 150,000 people every day to millions of people. You can literally save people’s lives.

What did the ad hoc team do? When they came on, the application took 76 screens to complete, and it took about 20 minutes to complete an application. By the time they left again, 6 weeks later, it was down to 16 screens, that’s worst case. The average time to complete an application was 9 minutes. The biggest issue was the login system, the authentication system, so identity management. It was responsible for the largest percentage of errors. It originally took $200 million to build, and then $70 million to operate and maintain it. What does that say? The max response time was 5 seconds, forever and ever. It was failing 25% of the time. They rebuilt it. This actually did take 6 months. It cost $4 million. To operate it and keep it running was also $4 million a year. Again, $200 million to $4 million. A max response time of 0.02 seconds. They actually broke the load tester. The error rates went down significantly. Again, that’s what you can do.

Immigration is another great example. Every year, there are 6 million applications on paper. It’s the I-90 form, the ability to renew or replace your Green Card. It was super painful. Again, all paper. Then businesses were popping up because it was so painful, so there were third parties who were like, if you pay me a bunch of money, I’ll do it for you. It should have just worked. USCIS spent 7 years building this process. They tried to digitize it. They spent $1.2 billion doing it. Then there was an independent study that showed that they’d made the process worse than the paper process. You know what they did? They re-upped with the same vendor for a 10-year contract for $2.5 billion. The definition of crazy is doing the same thing over again, expecting a different result. A U.S. Digital Service team went in and basically created a digital version. Again, a short amount of time. They moved from Waterfall to Agile. They moved to the public cloud. They implemented real-time monitoring. They also left the building and did a bunch of user research. Totally changed the game, super successfully, again, in a short amount of time for way less money.

There’s a ton of other examples. One is Veterans’ Medical Health Record interoperability. The challenge is somebody who serves in the military service, the Department of Defense has all their military records. They come back, they’re a veteran, and the systems at VA and at DOD do not talk to each other. There’s all kinds of challenges obviously, that come along with that. I will talk a little bit more about some of these examples. There’s this Police Data Initiative. This was in the wake of Mike Brown and Eric Garner being murdered by cops. There were two folks who worked, they were Presidential Innovation Fellows. They were engineers. They were like, what can we do? There is, in fact, a ton of data that talks about community and police interactions. There are ways to analyze and study them. They actually put this program into place called the Police Data Initiative. The concept was to really aggregate, harness the data, analyze the data, study the data and take learnings from that, so we could start to actually identify where there were problems and where improvements could be made.

Similarly, there is a crowdsourcing program for mapping, put out by the State Department. On April 25, 2015, there was a 7.8 magnitude earthquake that struck Nepal at 11:56 a.m. More than 5000 people died. Again, this idea of like, how can you actually make a difference in people’s lives? Within 48 hours, there was a mapping program where volunteers could come on and actually map, where there were cleared roads, where an ambulance might be able to get through, where a helicopter might be able to land. That is the idea of actually helping the first responders get to where they needed to be with information and data that we could provide them. Those are all a couple of different ways that the federal government has tried to make significant change.

College Scorecard

I specifically work on something called College Scorecard. This was my first project at ed in the White House. A college degree from a 4-year granting institution is worth more than a million dollars over the course of your lifetime. People who have a college degree, the poverty level is about 3%. If you don’t have a college degree, only a high school diploma, it’s about 12%. This makes a really important difference. President Obama had already made an announcement where he wanted to actually make this change. He wanted to help students, their parents make informed choice about college. As opposed to looking at these rating systems, it was like how much money your alumni donates, or how many new buildings a college might have. The idea is to look at metrics of accessibility, affordability, and outcomes. The challenge is, is the people who need most access to the data, who need to figure out the best school that is for them, folks from underserved communities, from low-income backgrounds. They don’t have exposure and access to this information. Their point was to try to get that into their hands however we could.

I started at the White House, I think it was April 6, 2015. President Obama had announced this in August of 2014, and that it was going to be launched by the fall, by September, before the school year started in 2015. I show up in April. Apparently, the Secretary of Ed and the administrator of the U.S. Digital Service had had a meeting with President Obama the week before I joined. President Obama was like, where are we at with College Scorecard? He had gotten out a sheet of paper and was drawing what it needed to be. I was like, I do not know, when designer in chief became the most important role for the President of the United States. I wasn’t at that meeting. All I know is both of them were like, don’t worry about it, Lisa starts next week, she’ll figure it out. Again, no pressure, I got this. The problem is when basically you are handed a drawing or a sketch or a wireframe from the President of the United States, it can be a little bit daunting to say, that’s nice, and put it away. Because, turns out, that’s not how you build a good product.

Part of the reason why this was so daunting is that it was such a cross-departmental, cross-agency collaboration. Whether it was the Department of Education, Federal Student Aid, which is a different department, Domestic Policy Council, the White House, Council of Economic Advisors, Treasury, it was just a lot of folks all up in the mix here. The idea here was to deliver the most reliable and comprehensive dataset ever in the history of the Department of Education. It included all of these things. Question number one, of course, is, what should we actually build, when I put away the president’s wireframe? We of course took a design thinking approach to it. Let me tell you, number one job was to actually explain to the Department of Education what design thinking was, because it’s completely new and foreign. What we were trying to really do is bring new tools, new methodologies into the department. To really help them understand, this is about building a shared understanding of the customer. That this will actually help build things that will result in measurable change. It’s outcomes, not outputs. Doing this will make sure that we build the thing that will work, that will actually help people, not just a thing that we arbitrarily abstractly think should be done.

First Step is Empathize Mode

The first thing we did is empathize. We really did actually do this. First step is empathize. I had been in D.C. a week, and I was like, I don’t know anybody down here that’s got college age kids. How do I talk to some students? How do I talk to some users? I’m breaking out in hives, and somebody on my team is like, we could just go to the mall. I was like, that’s brilliant. The mall where the youth hang out, there’s movies. Of course, they’re just going to be lolling about there, I could hang out. She’s like, or the mall outside the building, the Washington Mall. It worked. We went outside and just accosted people on the Washington Mall coming out of the Air and Space Museum. It turns out, people will just talk to you. Walk up to them with a clipboard, they are absolutely willing to engage in conversation. We didn’t just talk to people who could afford, like there are people there on spring break. We didn’t just talk to people who could afford to come to D.C., on spring break. We went to a public school in Anacostia. We talked to schools in the South Bronx. We talked to schools in the Midwest. We did actually do more than just talk to people on the mall. That was a great first step.

We came back in super excited with what we’d learned, super jazzed, energized. Part of what we wanted to do is bring the department along. Everything was about, how do we help you all see what we’re doing, understand it, explain it? We sat down at the table, and I was like, look at what we did? Before I even got to the learnings, we went out to the mall, we talked to 15 people. They were like, “You talked to more than 9 people? That’s illegal.” I was like, what’s happening? There’s apparently a thing called the Paperwork Reduction Act. The attempt is to actually not get more information from citizens than is needed. People read it and interpret it differently. Some people read it like you weren’t allowed to ask the same set of questions from more than 9 people in a row. That’s of course not how user research works. I was like, am I going to PRA jail? We were fine, but learned a lot from the experience.

We did all of the classic things for empathize. We went out, we talked to people. People write in letters to the president. There’s a thing called 10 letters a day, so we actually got letters that people had written in. We interviewed 79 stakeholders and users across all these different departments, all kinds of different users. Some of the quotes were things like, from a parent, “It’s a full-time job to sort through this information and make a good decision.” This is hard. What was happening is, if you were in federal government, oftentimes, if you wanted to share data, they would sideways scan paper and make them into PDFs and be like, here you go, what’s the matter with that? The idea that an open data user would want to actually be able to access the data, not through a PDF. This is my favorite. We ended up talking to a bunch of students who came with their 4-H club. It turns out that students know how to comparison shop. This student says, “I comparison shop for rabbit harnesses. You can’t just buy the cheapest one or the nicest looking one. You have to make sure the material will hold. You want to make sure it’s not a rip-off.” They knew how to do it, but they didn’t know how to do it when it came to looking at colleges. They also talked about it like horse breeding, all of that stuff. When it came to making choices, and analyzing and getting advice about where to go to school, it never even occurred to them to apply the same lens to it. When you talk about insights that you’ve learned, it’s really interesting.

Extensive Research and Analysis

We did a ton of research, websites, landscapes, all of this stuff. My favorite thing is we actually mapped out the process by hand because paper wins. My favorite part of this process is, again, a lot of different moving pieces. The Federal Student Aid had to give information, including social security numbers over to the Department of Treasury, so that the Department of Treasury could actually look at what people were earning 5, 6, 10 years out of college. That’s super secure private information. It turns out, the most secure and safest way to do it was to put it on a hard drive, walk it over to the Department of Treasury. I had full access to the White House. I could give tours of the West Wing, literally. To get into treasury, it took me 24 hours of security checks and access and all of that stuff. Treasury is more secure than the White House. They have the hard drive in their hot little hands. It’s not like they could go into the building without it being a huge pin. What they did is they would call a deputy secretary, so secretary, deputy secretary, super senior, and they would come down and they would hand it through the fence. That’s what that is over there, that’s the fence where the hard drive went through. That’s not something we changed. We were like, we have to pick our battles. That is, in fact, the most efficient way to do it. You all should be doing that. It is what it is. There was an Open Data Executive Order.

Define Mode

Then we went on to, as you do, you define, now we have gotten a lot of input and information. Now we’re going to define what we’re going to do. We came up with this problem statement. We wanted to engage and educate potential college students of any age or background, and those that support and advise them to find schools best suited to them. The number one thing is, nothing on there says build a website. The number two thing is actually about 50% of the students in the U.S. are over 24. It’s not just kids. It’s not just teenagers. It’s not just people with parents. It is people with kids. It is people with full time jobs. Just trying to be really thoughtful and comprehensive about who we were solving the problem for.

S.M.A.R.T. Goals and Success Criteria

We did what we were supposed to do, S.M.A.R.T. goals: specific, measurable, achievable, realistic, and time based. One, we want to engage a diverse set of students and their supporters because it’s not just the actual student, but it’s who they’re getting advice from, maybe their school counselors. We wanted to educate not just the individuals, but also the marketplace as a whole. We want to change the narrative, change the conversation about what would make for a good school. We wanted to help you find the school that’s going to be best for you personally. Because here’s the thing, the Department of Education cannot say this is a good school, or this is a bad school. How do you compare a Harvard University with a Howard University, a historically black college university, which serves a different audience? How do you compare a Berklee School of Music to an MIT or a divinity school? Those are the things that it depends on the individual. It’s not that one is good and one is bad. The other thing, and it was a four for us, was to foster continuous improvement. This wasn’t going to be a one and done. The idea was to actually shift the way the Department of Education approached this and make sure that it would continue on and grow and improve. It’s build, measure, learn. There’s a lot of data, we did make them measurable and specific.

Why Us? The Ed Differentiator

The other thing that we asked flat out was like, should it be the Department of Education who does this? Maybe we should hand off the data to a nonprofit. Maybe somebody else should take this on, to make sure that it has longevity and continuity and less bureaucracy. It turns out, the Department of Education actually has data that nobody else has access to. The Department of Education can shape the national conversation. The Secretary of Ed can literally get every school counselor in the country on a conference call and share that message. A reporter we interviewed said, “Government brings with it a sense of authority and credibility,” because if you’re going to other third-party resources, you don’t necessarily know how accurate or truthful they are.

Drafting Preliminary Personas

We drafted preliminary personas. This is all stuff that I’m 100% sure that you all do in your day-to-day lives, but this can be applied for any problem. It wasn’t fancy. It’s paper on a wall. We did primary users which included students: high school and adult. Also, the parents, the school counselors, and advisors. In terms of changing the narrative and changing the conversation, it was really important to really think about media, researchers, civil rights groups, all of those things, if we’re actually going to change how people think about what’s important. All these other ones, institutions, accreditors, other report systems.

Idea Generation, Prototyping, and Testing

Then it was ideate. Now that we’ve defined the problem, now that we know what the problem is, how are we going to solve it? Two things, one, we created a consumer tool. Two, we actually decided that we wanted to open up the data. Our password for the entire project was, set the data free. Then we prototyped. We did all of the things that you all are doing. We did lo-fi prototypes all on paper. We even prototyped the data tool. We did actually define a preliminary API set, went out, and tested that. How do people respond, react? What were they looking for? What were they missing? Then we went and tested it. Same thing, we cut out a fake form out of cardboard. We created a long strip of paper that we could slide through it, and we went back out to the mall. People were literally using their thumb to move the paper through the phone. It was really amazing. We got to see what mattered to people. What people were looking at. What they were pausing on. What they were reading. What they were pretend clicking. It was great.

The Results

Where we ended, was, we did in fact create this consumer tool. I’ve been told this was the first website in government that was built mobile first. Because again, for our audience, a lot of people are on their phones, as opposed to on computers. The idea was to actually put front and center what were the most important things. What are the things that you should be looking at? Because when we were out there talking to students, we asked them a question like, who’s ever given you the best advice about where to go to school? Because it’s about trying to figure out what their mind frame is and who they’re talking to. They also didn’t know what they wanted to major in. It was like accounting and nursing, very concrete, specific things. Additionally, they all said that they were going to go to public schools, and we asked why. Ask why five times. It was because they had assumed that that would be the most cost-effective thing for them. It turns out, if you are from a low-income background, public schools are not necessarily the most cost-effective option.

If you look at the front screen here, programs, because people didn’t know what they wanted to major in: location, size, name. Front and center, we did not have public school or private school. We did have it if you wanted to dig deep in the search results, you could actually do an advanced search. Even though everybody said it, it wasn’t public or private, it was about cost. You would do a search, and the research results would be like, first thing is, here’s what the cost is going to be. We also did benchmarks against countywide averages, so you could actually see whether something was good or bad. Because if not, it’s super abstract. Again, looking at some of these other things, graduation rates, salary after attending. Also, if a college is under monitoring, that also is a red flag. We looked at a lot of detail. Again, we tried to present it in really accessible, usable ways. Graduation rates. Students who return after their first year, because that is really telling about how successful you’re going to be. If a college graduation rate is 15%, you should not go there because it means that you won’t graduate. Helping people understand the relative importance to your personal circumstances, is all that we were trying to do. Salary after attending. Students paying down their debt. One of the things you’ll see at the bottom, typical monthly loan payment. We’re out there talking to students and students are like, “I buy new kicks once a month for 150 bucks.” They understood the concept of money, but $23,000 seemed like, I don’t know what to do with that. If I can translate it down to, that’s a pair and a half of sneakers a month. It helps them put it into the real-world circumstances that resonate with them. Those are all the things that we really tried to do with this product.

The second part of it was opening up the data. This dataset, 7300 schools in the U.S., there were 1900 columns per year, and it went back 18 years. There was a bunch of data up in there. This is what it looked like before. This is obviously a tiniest of snapshots. The idea afterwards was to make it searchable, downloadable, API accessible. What are some of the data insights? How should you be thinking about it? Here are some questions that you might ask. Here’s why these two metrics might sound the same, but they are different. Trying to really make it understandable. I come from media, one of the things that we always talked about was, it’s really important to get your content out to your user wherever they are, on whatever device, at whatever time they’re looking for your content. That was the same thing that we wanted to do here, because not every student is going to come to an ed.gov website when they’re looking for colleges. What we needed to do was get it out to them in the hands in the places that they’re looking for it. We actually did a beta program. Here’s the thing about doing this in federal government, my heart grew like three sizes. We did this all very quickly. It was over the summer, when everybody is taking vacations, I think August. We basically called a bunch of people who had applications around data and we said, “We can’t tell you what we’re doing, but if you sign an NDA, would you be willing to join a beta program that is super-secret.” Because of the press and all that stuff, so we had to actually keep it pretty close, although we did build in the open, we just did it super discreetly. It was all publicly accessible.

We did this beta program as well. We started development, June 17th, and we launched on September 12th, so under 3 months, which, even for private sector is not too shabby. This is also apparently the first time that the president had uttered a URL since healthcare.gov. Again, no pressure, but it had to stand up and it had to work. It was fantastic. Not only did it launch, we got a million visitors in the first week. We launched the beta program, August 1st, this is September 12th, so in 6 weeks, people rolled up their sleeves, took in the data, and actually put it into their applications and stood up with us on September 12th with the new data. Then, thanked us for releasing the data. It was just really a beautiful experience.

How it Works

I’ll show you all a little bit about how it actually works. The Department of Education can’t say this is a good school, this is a bad school. It all depends on you and who you are and your individual characteristics. There is a program called ScholarMatch, which is actually based in the Bay Area, which is very specifically geared towards helping low-income students find the right college for them. This is a really good example. San Francisco State, San Jose State, very near to each other, hour-and-a half near each other. Undergraduate size is about the same. You look at the top-level stats, and they seem like they should be about the same kind of school. Percentage Pell Recipients is a way to tell income brackets. I’m a Pell Grant Recipient. If you come from a low-income background, that’s where you get Pell Grants. Diversity, tuition, and fees, it all looks pretty much the same. You dive into the data that we actually released, and if your income is under $48,000, actually, it’s no longer a $700 or $900 difference, it actually is a $3,500 difference. The graduation rates depending from what income bracket you come from, can vary pretty significantly. Once you start taking that into account based on your income, going to San Francisco State costs 50k, whereas San Jose State costs 71k. Those are the kinds of things where you can actually say, I know who you are, and I can help you figure out what data is going to help you make an informed choice.

This one is interesting. This is CollegeBoard. If you look, it’s repayment by average net price. The ones in green, the private nonprofits cost more, but are also the most repaid. Then public costs less and are more repaid than the private for profits. What it did was give people access to look at data, slice and dice it in all kinds of different ways. Somebody did this StartClass around Harvard versus the average Ivy League versus colleges in Massachusetts. It was awesome to see what people could do with it. College Scorecard went from being denigrated, people trash talked it. It was a brouhaha. It was an uproar. That was why we had to be so secretive about it. The New York Times says, the data has wrought a sea change in the way students and families evaluate prospective colleges. Google integrated College Scorecard data into search results, because turns out that is where people are looking for this information. College Scorecard was credited, within three years, with improving college graduation rates in the U.S. by a point and a half. That’s the dream. We’re trying to make real change. Key takeaways include, build for your user by talking to them. Get out of the building. Open up your data as much as you can.

tEQuitable

When I was leaving the administration, trying to decide what I wanted to be when I grew up, because it is a journey. I was like, if we can send a Tesla Roadster into outer space, create space debris, maybe we can use the same best practices, product development strategies, innovative approaches, right here on our home planet, to solve some of the issues for the underserved, the underrepresented, and the underestimated. From that I founded this company called tEQuitable. We’re using technology to make workplaces more equitable. Our mission is to really help companies create work culture that’s going to work for everyone. For example, if my boss makes a sexist crack, or tries to touch my hair, that’s not the totality of who they are. I’m not going to go to HR for that, because that feels like the nuclear option. I’m not trying to get them fired, but I would like the behaviors to stop. Flip side of it is if I feel like I’m being overly discriminated against, or harassed, then I do want the company to take immediate action. tEQuitable tries to help the employee in either of those situations, figure out what their next step should be, and how they can move forward. Simultaneously, oftentimes, companies don’t have a great sense for what’s happening on the ground, day-to-day. We try to provide data and insights back to them for that. The reason I’m talking about this is, again, I showed you examples in public sector, and now doing it in private sector, for the concept of the greater good. When I left government, I was like, first of all, I need to make some more money, so I’m going to go back to private sector. It turns out, it’s really hard to let go of the mission orientation. I am very grateful that I am still able to make what I hope is significant change using technology, so using those same concepts again.

Issues are Pervasive

Issues are pervasive within companies. There’s a study that talks about how 78% of employees report experiencing some form of unfair behavior or treatment. There’s also an EEOC report from 2016, that talks about how as many as 85% of women feel like they’re harassed at work, but as few as 6% of them report it. The takeaway here is, it doesn’t matter that HR are the most wonderful human beings with the best of intentions, if they’re not even hearing about these issues. It’s impossible for them to solve it, to try to get in front of them. Here’s the thing about this, is that, this is not just, we should do the right thing. It’s not just a moral issue. It actually really is also a business imperative. I’m hopeful that you’ve all seen the stats that talk about how racially diverse companies are more likely to have better financial returns, that having better racial diversity in exec team results in better earnings. Just generally, there’s a study that talks about if the workforce matched the talent pool, that tech could actually generate an extra $300 billion per year. There are real implications, financial implications, truthfully, for the GDP, around actually bringing more folks into tech innovation inclusion economy. More stats about inclusive and why it matters. Better business outcomes, more likely to be innovative and Agile, more likely to be high performing. Then the financial implications. That is why it’s so important. That’s the carrot.

Then the stick is, if you don’t do it, huge cost: higher turnover rate, lack of engagement, loss of productivity, lower diversity numbers, reputational damage, and not to mention lawsuit settlements. These are judgments that have become public. This is just the tip of the iceberg. They’re huge. So many of the stuff happens under cover of night in darkness, settlements that you never ever hear about. It is billions of dollars, these issues cost. It has real-world implications. If somebody is not feeling that they’re being treated fairly, they disengage, so it’s higher absenteeism, more errors, more turnover. Same thing, if somebody doesn’t feel comfortable in their workplace, it doesn’t just stop with them, it snowballs. It bleeds into their relationship with their coworkers and potentially even with customers. There are so many different facets and avenues where this thing can have a real impact.

What Employees are Looking For

What are employees looking for, especially around microaggressions, microinequities? I’m always mistaken for the other person of my race in my building, even though we look nothing alike. Or, I know you’re a third generation Asian American, but you speak English so good. It’s those kinds of things that can really affect the day-to-day work lives. Again, through user research, we went out and talked to, at this point, probably thousands of employees, in terms of what they’re looking for. The first thing is, sometimes they just need to feel heard. Sometimes it’s just one of that. They also really want to know that it’s not just all in their head, or it’s not just happening to them, they’d like to get advice on what they can do about it. They’d like for it to be fixed, but without it being their responsibility. Very specifically, most often we hear is, I don’t want it happening again to somebody else, if I can help prevent that. The reasons they don’t want to come forward to formal channels is, first of all, they lose control over whether something will be escalated. It disempowers them. They worry about repercussions, whether it’s immediate, or in the future, because it’s a small world out here these days. They don’t know whether something has actually crossed the line, because the line is in the eye of the beholder. It might be a gray area. You don’t want to be told, “You shouldn’t have taken it that way. She didn’t mean it that way.” They’re not necessarily trying to get somebody fired over these issues.

The Types of Issues Span a Broad Spectrum

What we try to do is help with a broad spectrum of issues, whether it has to do with gender or race or homophobia, transphobia, immigration status, parental status, ageism, ableism, all the isms, all of the phobias. Then for us, it’s not just about the severe issues. It’s not just, my boss groped me in the bathroom. It is the more subtle, insidious death by a thousand papercuts that can really affect people’s day-to-day work lives. It’s really about the interpersonal conflict. It could be things like, I joined a Zoom with 20 other people on it, my boss didn’t see me, and I heard him talking about me. It is just all the things that really can affect how you feel in your work environment. The idea of being able to put interventions into place can prevent issues from ever escalating. What we’re really trying to do is help companies get in front of and prevent issues, not just catch harassers on the back end after it’s too late.

The Ombudsman Model

We are based on the model of an ombudsman. Ombuds have been around for centuries, really, but in the U.S., since about the 1960s, primarily found in government, academia. There are some Fortune 500s like Coca-Cola, American Express, they all have teams of human ombuds. It’s really just a safe place where employees can come and figure out how to handle workplace conflict. We are a third-party tech enabled ombuds, and again, this idea of taking a principle that existed and amplifying it. We follow all the same principles and practices of any organizational ombuds. We are independent, so not beholden to the company’s management structure. We are confidential, meaning we’ll neither confirm nor deny someone has even spoken with us. We are impartial so we don’t advocate for the employee, we don’t advocate for the company, we advocate for fairness of process. We’re very specifically informal and off the record, so telling us something is not serving notice to the organization. This is a proven model. It’s been shown to reduce misconduct by 75%, and virtually eliminate all retaliation, and also to reduce external litigation costs by 50% to 80%. It’s not for nothing.

The idea is, how do you take a proven, tried, and true model, how do you then apply technology to it through user research, through design thinking, and try to affect change and make it accessible? Right now, like UCLA has 1 ombuds for every 15,000 employees. The U.S. State Department has 1 ombuds for every 25,000 employees. It turns out, then people are only coming for like big, egregious issues, as opposed to like, it’s not a bad thing to use the platform. It’s a really positive one. It’s helping to create a company culture, where if you see something, say something, do something. It’s that kind of a thing of like, how do you make it accessible and usable? The principles apply. It’s a proven model. For employees, we provide a sounding board where they can come get advice, explore their options, figure out what their next step should be. While they’re doing that, we gather data that we anonymize and aggregate. We use that to try to identify systemic issues within an organization’s culture, create a report back for the management team with actionable recommendations. We really try to work on both sides of the equation where we’re empowering and supporting employees, but we are also helping companies identify and address issues before they escalate, trying to create this virtuous cycle. As you may have seen from my background, systemic change is what I’m all about.

The tEQuitable Product

This is the product. The concept is really one of, foster a culture of healthy and open communication. We did a ton of user research, and some people are looking for a step one, step two, step three, step four. We do things like strategies on how to have a tough conversation, or we built a little conversation starter, which really just helps an employee build an I statement. When you did such and such, be objective here. How did it make you feel? Were you sad, angry, frustrated? We also have action modules, more off the beaten path. How to respond to a backhanded compliment, “You’re black, but you’re so articulate.” Those kinds of things. What can you say in the moment that might get heard but you still feel safe.

What we also heard from so many other people was, how do I know it’s not just me? Am I being too sensitive? We created a library of stories where people could see themselves reflected without having to take aggressive action. In this case, it’s, “He told me I’d be prettier if I smiled more.” Every story and narrative form covers four topics, what happened to me. How it made me feel, because that’s really the thing that resonates with people. “I walked away with a pit in the bottom of my stomach.” What I did about it, and then what the outcomes were. At the bottom of each of these, we actually ask the visitor two questions. One, did you see yourself reflected here? Did this content resonate with you? Two, do you now feel more confident in the next step that you’re going to take? Because ultimately, that’s what we’re trying to do. We’re trying to give the employee back some agency, because so often, they feel like the control was taken out of their hands. Then, at any point, they can also schedule an appointment to talk to a professional ombuds. The idea here is to really meet the visitor wherever it is that they are.

For us, it really is about envisioning a future where everyone can bring their whole best selves to work. How do we use technology to actually empower both the employees but also the companies, while maintaining privacy and security? Disenfranchised communities must be welcomed into the innovation economy. Tech is the fastest growing sector in the U.S. Today, there are 500,000 open jobs in IT. By 2024, there will be 1.8 million that are unfilled. Until we actually can start bringing in other folks who can take advantage of this and start using their skills to apply towards this, it’s going to affect the U.S. economy. Then also, in tech, these are jobs that pay more than a lot of other things. The wealth gap is so disparate between white families and black families. If things continue the way they are, it’ll take 228 years for the black wealth gap compared to white to go away. That is the idea is like, how do you actually start bringing folks in and pay them well enough so that we can start making societal level change? That is what we’re trying to do. It’s really why this is so important.

The thing that’s so important about doing this, though, is we’re all engineers, we’re all leaders in engineering. How do you actually take the skill that you have? It’s no longer about, how do you sit in a dark cubicle, and do the thing that you’re trying, like this little area that is your focus. You can actually take your superpowers, your skills, and apply them across industries, across communities, across segments. We should all be thinking about being activists in any way that we can. I would just encourage you all to think about. What I tried to give you was a bunch of different examples of the ways that you really can make change, just being a technologist. That is what I will leave you with is, go forth and use your powers for good.

See more presentations with transcripts

Article originally posted on InfoQ. Visit InfoQ

When dealing with an environment that feels complex, people commonly look for ways to reduce variability and increase control for dealing with complex systems. An alternative approach is to embrace complexity by acknowledging that it exists, asking questions and listening, and constructing a shared understanding based on different perspectives. This lets us improve how we adapt on an ongoing basis.

Fred Hebert will give a talk about embracing complexity at QCon New York 2023. This conference will be held from June 13-15.

According to Hebert, initiatives for reducing variability and increasing control tend to come from the top of the organisation, and by their nature, they tend to communicate a need to make plans successful by increasing predictability. This can work fine so long as this variability is accidental and non-essential, but can backfire in other situations:

Whenever that variability is there for a good reason, as an adaptation to a complex landscape or dynamic conditions, then trying to rein it in is just likely to make you less successful regardless of the desired outcome behind trying to increase control.

According to Hebert, we have to get a better understanding of what is going on first. And that understanding must be done with the intent of knowing what our limitations might be as a centralised decision-maker, not as a way to just control things better:

Find out where in the organisation people might be better located to act on inputs—usually they’ll be people in direct contact with customers, code, and able to act on events as they happen—and who else might need that information.

The first step in embracing complexity is to acknowledge it exists, and that it requires different tools. Some of the things that Hebert suggests doing are:

Rather than using metrics and indicators as self-contained signals, see them as pointers towards where to make deep dives.

Rather than trying to nudge people on what to do by adding carrots and sticks, figure out what drives their decision-making and see if anything can be made easier for them.

According to Hebert, we should ask questions and listen, and don’t try to simplify too much, and don’t put too much importance on finding an objective explanation for everything that went on, but construct an understanding based on the varied perspectives of people we work with.

InfoQ interviewed Fred Hebert about dealing with complexity.

InfoQ: As trying to control complexity doesn’t work, what’s the alternative?

Fred Hebert: Become a facilitator and aligner for decentralised control, rather than a decider who ultimately is a bottleneck people will learn to bypass.

A common pattern I’ve noticed where the same mechanism can be used for control or empowerment (but rarely both at once) is around Request For Comments (RFCs) or Architecture Decision Records (ADRs). Organisations often ask for people to design new components upfront, and the committee doing the review acts as a gatekeeper that aims to prevent mistakes. In almost all cases I’ve seen this put in place, people who perceive the process as a blocker end up doing the work regardless and then just half-ass writing a document to cover their tracks after everything has been put in motion. The process isn’t helpful. But it could be oriented toward empowerment and trust: making expertise from other departments available, and using it as an information propagation mechanism more than a way to prevent mistakes.

InfoQ: How can we embrace complexity, can you give an example?

Hebert: A classic example here is one of counting incidents and tying a team’s bonuses (or reprimands) to the absence of incidents (or their presence). Incidents are already difficult to deal with, often tied to long-standing decisions within the organisation, and just adding more variables to the equation is likely to change little aside from driving people to mess with the data they report to meet the new objectives without making the existing ones worse.

InfoQ: What’s your advice for fostering a richer view of systems?

Hebert: Trying to build a clean, truthful, axiomatic view of events tends to discard a lot of the messy details about what is going on. Sometimes it’s because these messy details are hard to measure, they’re opinions, or out of the realm of things you control, but they still influence how people navigate their part of the system.

Embracing that mess will likely lead to views that are far richer in terms of the insights it generates.

Article originally posted on InfoQ. Visit InfoQ

Earlier this week, Uno Platform released version 4.9 of their framework for building native mobile, desktop, and WebAssembly apps. The latest version brings the MediaPlayerElement control and WebView2 support which are two community long awaited features. Also, beyond these additions, the release includes over 100 other improvements and enhancements, including the native host support for Skia.

The first notable feature is a community of highly anticipated MediaPlayerElement control from Windows UI to iOS, Android, and Mac Catalyst platforms through the Uno Platform. Additionally, Uno Platform 4.9 expands its reach by introducing support for Web and Linux targets, with the Linux target utilizing libVLC for media rendering. Detailed instructions on utilizing the MediaPlayerElement can be found in the official documentation.

Regarding the MediaPlayerElement control, Uno Platform Team states the following:

Whether your target is iOS, Android, Web, Mac, Linux, or Windows itself, this guarantees consistent and seamless media playback experiences across devices.

Another bigger feature is that support for the WebView2 control has been introduced for Android, iOS, and Mac Catalyst platforms. This control enables users to navigate to external web content and display custom HTML content, providing versatile functionality. It also facilitates communication between C# and JavaScript, allowing for powerful embedding scenarios such as interacting with JavaScript charting or data grid components. The WebView2 control replaces the previous WebView “1” in WinUI, offering an expanded feature set and ensuring a seamless migration for existing scenarios.

Furthermore, another big addition is regarding the Native Host Support for Skia. It enables the integration of native controls for enhanced platform integration. This new feature empowers the ContentControl component to accept native instances, such as GTK Widgets or WPF FrameworkElements, and render them on the Uno Platform’s surface, utilizing the allocated size of the associated ContentControl.

Notably, the original blog post reports that this capability has been used successfully in the implementation of MediaPlayerElement support, ensuring the precise placement of video surfaces. Embedding Native Controls in Skia Apps documentation article is also available for developers to explore and learn more about.

In the original blog post, the Uno Platform team provided the code example which will create a new Gtk CheckButton control displayed on top of the Uno Platform Skia canvas.

Other notable changes and improvements include indexer and MVVM toolkit support for x:Bind expressions, UIElement.ActualOffset support, and ms-appdata support for loading app-packaged SVGs through SvgImageSource. The update also introduces significant performance improvements for WebAssembly using JSImport/JSExport, as well as enhanced performance and memory optimizations in the XAML Generator. Notably, TextBlock rendering for Skia heads has undergone performance enhancements. Additionally, Visual Studio 2019’s Uno Platform Solution Templates are now being deprecated. These updates collectively contribute to a more robust and efficient development experience with Uno Platform 4.9.

Lastly, developers interested in learning more about the Uno Platform can visit the official website for very detailed documentation which contains how-tos and tutorials about the platform, alongside the official GitHub repository, and a more detailed full list of improvements is available at the release changelog.

Article originally posted on InfoQ. Visit InfoQ

Subscribe on:

Apple Podcasts
Google Podcasts
Soundcloud
Spotify
Overcast
Podcast Feed

.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.

Article originally posted on InfoQ. Visit InfoQ

At the end of May, Microsoft introduced the Azure Monitor OpenTelemetry Distro. Azure Monitor and all Azure SDKs support OpenTelemetry, integrating with APM systems such as Prometheus and Grafana. Microsoft Azure provides the Azure Monitor OpenTelemetry Distro, facilitating the deployment of this technology and increasing the observability of applications.

Microsoft Azure uses OpenTelemetry technology as the industry standard for monitoring and transmitting telemetry. They retrofit Azure Monitor and SDKs to support OpenTelemetry, ensuring compatibility with any language. This vendor-independent approach integrates data with various application performance monitoring (APM) systems, including popular open-source projects such as Prometheus, Grafana, Jaeger & Zipkin, and the Azure platform’s native monitoring solution, Azure Monitor.

In order to simplify the deployment of OpenTelemetry technology, the Azure platform is introducing the ‘Azure Monitor OpenTelemetry Distro’. Initially focused on Application Insights, the distribution will extend its reach to other scenarios within Azure Monitor. It provides a comprehensive solution packaged as a NuGet package Azure.Monitor.OpenTelemetry.AspNetCore and supported by a corresponding GitHub project. By offering this distribution, the Azure platform aims to enhance observability and make it easier for developers and organisations to use OpenTelemetry in Azure Monitor.

Making the implementation of OpenTelemetry easier for customers poses a challenge for observability platforms like Azure Monitor. Currently, developers face a learning curve as they learn the basics of OpenTelemetry, select the appropriate instrumentation libraries, configure their setup and ensure compatibility with existing APM systems. This process involves managing multiple components and tracking various moving parts. In addition, OpenTelemetry offers an extensive API surface with numerous instrumentation libraries to meet a variety of observability requirements. To address these issues, the Azure platform aims to provide developers with a more accessible entry point and recommended monitoring practices for Azure-based ASP.NET Core web applications.

For ease of enabling OpenTelemetry, Distro provides methods to quickly connect Azure Monitor, including Application Insights, with a single line of code: 

var builder = WebApplication.CreateBuilder(args); 
builder.Services.AddOpenTelemetry().UseAzureMonitor(); 
var application = builder.Build(); 
application.Run(); 

One line of code gives full capabilities of the Azure platform such as including popular libraries for collecting traces, metrics and logs, correlated traces from Application Insights, AAD authentication, standard Application Insights metrics and automatic discovery of Azure resources. New features such as Live Metrics will be added soon. The ultimate goal is to provide customers with the best experience and full functionality of OpenTelemetry through this distribution.

Getting started with Distro requires several steps, which are described in detail on Azure Monitor Application Insights OpenTelemetry Enablement Docs.

Article originally posted on InfoQ. Visit InfoQ

The latest JBang release provides a preview of Java 21 and more integration with the OpenAI plugin via the quickstart port to Java. JBang has also recently introduced the integration of ChatGPT via the CLI, allowing developers to create JBang scripts using ChatGPT.

JBang is a lightweight Java-based scripting tool that allows developers to write and run Java code directly from the command line without requiring an entire Java project or compilation steps. It was developed to simplify the execution of Java code more interactively and conveniently, similar to languages like Python.

Java developers can now execute Java code with JBang using the upcoming release of Java 21, scheduled for September 19, 2023. Furthermore, developers can also request via CLI to generate code with OpenAI’s ChatGPT or even create plugins in Java for ChatGPT.

To use JBang with Java 21 preview features, simply run the following command:

$ jbang --java 21 --enable-preview -c 'print(STR."First arg is {args[0]}!")' JBang

The command will download the early-access release of OpenJDK 21 in case there isn’t one on the system. Furthermore, it will allow Java developers to use the preview features like String templates.

The enhancements introduced also bring some exciting ergonomics, e.g., a developer doesn’t necessarily need to figure out all the intrinsic details of an API. In the following example, the command creates Java code that takes an image, adds text, and then writes it to the filesystem. Usually, anyone learning OpenCV in Java would need to know the intrinsic details of image processing and its components and concepts to understand dependencies before even beginning to write any code.

$ jbang --preview --verbose init ImageProcessor.java "Write a Java program that processes an input image https://shaaf.dev/images/testImage.jpg and add text Hello world on it using OpenCV to produce an output image called output.png"

In the above command:

• The --preview argument for using preview features, e.g., ChatGPT
• The init ImageProcessor.java argument will create a new ImageProcessor.java file in the directory where the command executes.
• The text “Write…” requests to create a Java program that uses OpenCV.

Other mundane tasks, for example calling a REST API via a Java Client and showing the results, have become single-line commands.

$ jbang init jokes.java "print a random chuck norris joke fetched from the API service providing free jokes API."

$ jbang jokes.java

Or to fetch information and process it further:

$ jbang --verbose init gh.java "connect to GitHub and fetch all repositories under jbang and collect how many stars they have"

In conjunction with the 0.107.0 release, Max Anderson, a distinguished engineer at Red Hat and creator of JBang, also introduced a port for the OpenAI plugin Quickstart, allowing users to write plugins for ChatGPT and test them locally with JBang.

InfoQ spoke to Max Andersen on this release and the future of Java and AI.

InfoQ: What does JBang integration to OpenAI’s ChatGPT bring to the Java community?

Andersen: JBang today has default templates for creating a starting Java project. However, one still needs to know their names and what they do; secondly, they are limited. With JBang OpenAI support, you can use natural language to outline your ideas and wishes and get a good starting point. This is much faster than doing a web search to try and collect all the required components. Since JBang has always been about reducing the ceremony around Java projects, this was a natural extension of JBang. It’s a great “cure” against writer’s block – it lets me get started. It lets anyone get started with Java as a newbie or as an expert and lets you try new things.

InfoQ: Does the Java community need more integration into Artificial Intelligence libraries and tools?

Andersen: Python is dominant in the data modeling and processing parts of Artificial Intelligence – and for a good reason, as Python is very good as a data manipulation DSL. However, Java can shine in being used to do things way more efficiently and utilize the models coming out of AI. In this case, loading models using native code or even, in the future, pure Java would be helpful.There is also a lot of cross-pollination from which the Java and Python ecosystems could benefit. For example, I recently realized that Jupyter Notebooks could be used as a basis for Java/JBang tutorials.

InfoQ: Are there any specific features or integrations you are working on for AI?

Andersen: For JBang itself, similar to what was done with the jbang init command, I haven’t explored much more yet, besides upgrading it to GPT-4 instead of GPT-3. I’ve used OpenAI in a Quarkus CLI command called quarkus explain that allows you to use OpenAI in a project and ask it to explain individual files, whether that is .java, pom.xml, or an application.properties file. It works remarkably well.

InfoQ: What’s next for JBang and Artificial Intelligence?

Andersen: I used JBang to make an OpenAI plugin using Quarkus: I was quite pleased seeing that it reduces the code by about 30-40% compared to similar Python code. I want to use the JBang approach to illustrate Java is not as verbose and complex as many think. Java has matured a lot since the early 2010s.

Java and Artificial Intelligence (AI) are becoming better integrated, and Java core is moving in a direction that further supports some of these ideas. The recent inclusion of the Foreign Function & Memory API (Third Preview) and the Vector API (Sixth Incubator) is a step towards integrating with other libraries and enabling further innovation in this space.

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

“

The global Information Technology Application Innovation Databases Market report is a thorough research that gives significant insights into the keyword business, including market trends, development prospects, competitive landscape, and the influence of COVID-19. This review outlines the essential topics, tools and techniques employed, services given, market value, and why investors should consider investing in this report.

Request a sample report @ https://www.orbisresearch.com/contacts/request-sample/6987653

Contents of the Global Information Technology Application Innovation Databases Market Report:

● Market overview and definition
● Market size, growth predictions, and CAGR analysis
● Key market players and their market shares
● Market dynamics, including drivers, restrictions, and opportunities
● Competitive analysis and strategic suggestions
● Trends and advancements in the Information Technology Application Innovation Databases industry
● Impact of COVID-19 on the market
● Market segmentation based on several characteristics
● Regional analysis and market outlook
● Future prospects and investment possibilities

Tools and Techniques Used to Create the Report:

The production of the global Information Technology Application Innovation Databases Market report included a combination of primary and secondary research:

● Primary research: Surveys, interviews, and data collecting from industry experts, market players, and consumers.
● Secondary research: Analysis of existing sources such as industry studies, market databases, corporate websites, and published publications.

Key Players in the Information Technology Application Innovation Databases market:

Oracle
DB2
Microsoft SQL Server
Microsoft Access
MySQL
SQL
Amazon Aurora
IBM
RERADATA
Amazon REDSHIFT
NoSql
Cloudant
Hbase
Mongo DB
Couch Base
Neo4j
Google Cloud Datastore
Bigtable
Cloudera
Elasticsearch
Databricks

Information Technology Application Innovation Databases market Segmentation by Type:

RDB
NoSQL

Information Technology Application Innovation Databases market Segmentation by Application:

Smart Government Affairs
Information Security
Industry Digitalization
Digital Industrialization
Others

Direct Purchase the report @ https://www.orbisresearch.com/contact/purchase-single-user/6987653

Services Offered by the Global Information Technology Application Innovation Databases Market Report: The global Information Technology Application Innovation Databases Market report offers the following services to its readers:

● Market research and analysis
● Market sizing and growth projections
● Competitor analysis and benchmarking
● Trend analysis and forecasting
● Market segmentation and targeting
● Strategic advice for firms and investors
● Insights into emerging possibilities and difficulties
● Access to extensive data and information for decision-making

Market Value and CAGR of the Global Information Technology Application Innovation Databases Market Report:

The market value and compound annual growth rate (CAGR) of the global Information Technology Application Innovation Databases Market report will vary on the specific report and the investigated time period. It is advisable to refer to the report itself to receive precise and up-to-date statistics.

COVID-19 Impact and Market Status of the Global Information Technology Application Innovation Databases Market:

The report explores the impact of the COVID-19 pandemic on the Information Technology Application Innovation Databases market, including changes in consumer behavior, supply chain interruptions, and market dynamics. It presents a study of the current market status considering the continuing pandemic and other pertinent considerations.

Do You Have Any Query Or Specific Requirement? Ask to Our Industry Expert @ https://www.orbisresearch.com/contacts/enquiry-before-buying/6987653

Why Investors Should Consider Investing in the Global Information Technology Application Innovation Databases Market Report:

Investors might gain from investing in the global Information Technology Application Innovation Databases Market report for the following reasons:

● Gain a complete grasp of the Information Technology Application Innovation Databases business and its prospective development opportunities.
● Identify investment possibilities, market trends, and upcoming technology.
● Make educated decisions based on market information and strategic suggestions.
● Stay current with the effect of COVID-19 and other market influencers.
● Access vital data and research for determining market feasibility and profitability.

About Us

Orbis Research (orbisresearch.com) is a single point aid for all your market research requirements. We have a vast database of reports from leading publishers and authors across the globe. We specialize in delivering customized reports as per the requirements of our clients. We have complete information about our publishers and hence are sure about the accuracy of the industries and verticals of their specialization. This helps our clients to map their needs and we produce the perfect required market research study for our clients.

Contact Us:

Hector Costello
Senior Manager – Client Engagements
4144N Central Expressway,
Suite 600, Dallas,
Texas – 75204, U.S.A.
Phone No.: USA: +1 (972)-591-8191 | IND: +91 895 659 5155
Email ID: sales@orbisresearch.com

“

Article originally posted on InfoQ. Visit InfoQ

Carbon neutrality means that the total amount of emissions is either eliminated, neutralized, or compensated. Net-zero is a target that doesn’t include compensation and puts more emphasis on avoiding carbon emissions. Many products, data centers, or companies, are already carbon neutral, but few have reached net-zero.

Martin Lippert spoke about sustainability at OOP 2023 Digital.

There is no exact or clear definition of carbon neutral and net-zero, but we have a well-established common understanding of them, Lippert mentioned. He distinguished between two ways to deal with emissions: eliminating emissions and offsetting emissions:

Eliminating emissions means to not emit carbon into the atmosphere in the first place, you basically avoid creating carbon dioxide (or equivalent greenhouse gasses) and avoid emitting them into the atmosphere.

Offsetting carbon emissions means that you continue to emit carbon into the atmosphere, but you either compensate for those emissions (e.g. via carbon certificates) or you try to remove those emissions from the atmosphere again over time – which is often referred to as neutralizing emissions.

Lippert explained that carbon neutral means that the total amount of emissions is either eliminated (avoided), neutralized (removed again), or compensated (e.g. via carbon certificates) – or a combination of all three ways. But you don’t know which way was chosen to achieve carbon neutral – as long as the total sum is the same as the amount of carbon emissions that you caused, he added. So it might be the case that a company did not do anything to eliminate emissions, but solely bought certificates to compensate for those emissions. That would still result in a carbon neutral banner, Lippert said.

In contrast to that, net-zero takes the compensation part out of this equation and puts a lot more emphasis on the elimination (avoidance) part, Lippert explained. It usually means that you first try to eliminate as many carbon emissions as possible – and neutralize (remove later) the remaining emissions that you can’t eliminate, he said.

According to Lippert, net-zero is a much stronger goal when it comes to reducing carbon emissions – and probably the reason why you see many products, data centers, or companies already being carbon neutral, but not so many net-zero – at least not yet.

The problem with offsetting approaches is that you – more or less – outsource the problem and continue to emit carbon, Lippert argued. If you emit, for example, 1 metric ton of carbon dioxide into the atmosphere today, it contributes to climate change now. Planted trees – a widely used practice to achieve carbon neutrality – will need between 40 and 80 years to remove that carbon from the atmosphere again. While planting trees is usually a good idea, it doesn’t really compensate for your emissions in a meaningful time – especially if you continue to produce carbon emissions, Lippert concluded.

InfoQ interviewed Martin Lippert about offsetting emissions and net-zero.

InfoQ: Offsetting emissions by either neutralizing or compensating sounds a bit vague. Can you elaborate on what this really means?

Martin Lippert: The fundamental idea of “neutralizing” emissions is that you try to remove the emissions that you have put into the atmosphere from the atmosphere again – or at least emissions of the same amount. Planting trees is a widely known example of this. Those trees extract carbon dioxide from the atmosphere. And because it is hard for individuals and companies to plant those trees themselves, they can sponsor those activities via organizations. The underlying idea is – if you can’t avoid the emissions in the first place – try to at least remove them again at a later time.

The story behind compensation is somewhat different. The underlying idea here is: if you can’t avoid the emissions on your end, help someone else to avoid emissions (in the same amount). A famous example is to sponsor modern cookstoves in developing countries. Those modern cookstoves emit far less greenhouse gas emissions than the ones people would continue to use if they would have no chance to get those modern ones. So this helps to avoid emissions somewhere else that would occur otherwise.

Both variants are often realized via so-called carbon offsets that you can buy via organizations that then invest the money into those projects.

InfoQ: You mentioned in your talk that offsetting approaches do not really solve the problem. Why?

Lippert: Buying carbon offsets sounds like an easy way to deal with your own carbon emissions and feels like you don’t do anything wrong anymore when you buy those carbon offsets. The term carbon neutral underlines that impression. But even though those offsets are – most of the time – well invested into good ideas and projects, they don’t change the fact that you’ve put carbon into the atmosphere and that this carbon doesn’t go away anytime soon and accelerates climate change. We really need to focus on avoiding emissions in the first place.

Posted on nosqlgooglealerts. Visit nosqlgooglealerts

<meta name="x-tns-categories" content="AI / Data / Machine Learning“><meta name="x-tns-authors" content="Chris J. Preimesberger“>

2023-06-07 10:07:25

DataStax Adds Vector Search to Astra DB on Google Cloud

DataStax is working with the Google Cloud AI/ML Center of Excellence as part of the Built with Google AI program to enable Google Cloud’s generative AI offerings to improve the capabilities of customers using DataStax.

Jun 7th, 2023 10:07am by

Chris J. Preimesberger

With so much data piling up everywhere, loaded database nodes are becoming a serious challenge for users to search faster and more accurately to find what they are seeking.

DataStax, which makes a real-time database cloud service built upon open source Apache Cassandra, announced today that its Database as a Service (DBaaS), Astra DB, now supports vector search. This is fast becoming an essential capability for enabling databases to provide long-term memory for AI applications using large language models (LLMs) and other AI use cases.

DataStax is working with the Google Cloud AI/ML Center of Excellence as part of the Built with Google AI program to enable Google Cloud’s generative AI offerings to improve the capabilities of customers using DataStax.

Vector search can be difficult to explain to non-mathematics-type people. It uses machine learning to convert unstructured data, such as text and images, into a numeric representation within the database called a vector. This vector representation captures the meaning and context of the data, allowing for more accurate and relevant search results. It also is able to recognize and connect similar vectors in the database within the context of the query in order to produce more accurate results.

Vector search is often used for semantic search, a type of search that looks for items that are related in meaning, rather than just those that contain the same keywords. For example, a vector search engine could be used to find songs that are similar to a user’s favorite song, even if they don’t share any of the same keywords.

‘Vector Search Is Magic’

“Vector search is magic because it understands what you meant vs. what you said (in a query),” DataStax CPO Ed Anuff told The New Stack. “The more complex a piece of content is, turning it into a vector becomes a much more efficient way of finding this similarity without having to try to guess which keywords are (exactly) right.

“Let’s imagine that I have a database of all of the articles you’ve written. The process of turning each one of your articles into a vector is done through an LLM (large language model), and it looks through the entirety of each article. It figures out what are the most important pieces of an article, and the vector that it produces gets to the essence of it in a concise way. For example, even though you might have used the word ‘Cassandra’ many times in an article, it knows the LLM when it transforms into the vector. It knows that your article is about an open-source database – not about the Cassandra constellation or a performance artist named Cassandra,” Anuff said.

Developers create vectors with simple API calls, and they query those vectors on simple API calls. “But they can now put this powerful capability to work. So that’s why vectorization is such a powerful aspect of this,” Anuff said.

Some of the benefits of using vector databases include:

• Scalability: They can scale to handle large amounts of data.
• Flexibility: They can be used to store and manage a variety of data types, including structured, unstructured and semi-structured data.
• Performance: They can provide high performance for queries on large datasets.

Vector search is also used for image search. In this case, the vectors represent the features of an image, such as its color, texture, and shape. This allows for more accurate and relevant image search results, such as finding images that are similar to a user-uploaded image.

DataStax is launching the new vector search tool and other new features via a NoSQL copilot — a Google Cloud Gen AI-powered chatbot that helps DataStax customers develop AI applications on Astra DB. DataStax and Google Cloud are releasing CassIO, an open source plugin to LangChain that enables Google Cloud’s Vertex AI service to combine with Cassandra for caching, vector search, and chat history retrieval.

Designed for Real-Time AI Projects

Coming on the heels of the introduction of vector search into Cassandra, the availability of this new tool in the pay-as-you-go Astra DB service is designed to enable developers to leverage the massively scalable Cassandra database for their LLM, AI assistant, and real-time generative AI projects, Anuff said.

“Vector search is a key part of the new AI stack; every developer building for AI needs to make their data easily queryable by AI agents,” Anuff said. “Astra DB is not only built for global scale and availability, but it supports the most stringent enterprise-level requirements for managing sensitive data including HIPAA, PCI, and PII regulations. It’s an ideal option for both startups and enterprises that manage sensitive user information and want to build impactful generative AI applications.”

Vector search enables developers to search by using “embeddings”; for example, Google Cloud’s API for text embedding, which can represent semantic concepts as vectors to search unstructured datasets, such as text and images. Embeddings are tools that enable search in natural language across a large corpus of data, in different formats, in order to extract the most relevant pieces of data.

New Capabilities in the Tool

In addition, DataStax has partnered with Google Cloud on several new capabilities:

• CassIO: The CassIO open source library enables the addition of Cassandra into popular generative AI SDKs such as LangChain.
• Google Cloud BigQuery Integration: New integration enables Google Cloud users to seamlessly import and export data from Cassandra into BigQuery straight from their Google Cloud Console to create and serve ML features in real time.
• Google Cloud DataFlow Integration: New integration pipes real-time data to and from Cassandra for serving real-time features to ML models, integrating with other analytics systems such as BigQuery, and real-time monitoring of generative AI model performance.

Goldman Sachs Research estimates that the generative AI software market could grow to $150 billion, compared to $685 billion for the global software industry.

Vector search is available today as a non-production use public preview in the serverless Astra DB cloud database. It will initially be available exclusively on Google Cloud, with availability on other public clouds to follow. Developers can get started immediately by signing up for Astra.

Created with Sketch.

TNS owner Insight Partners is an investor in: The New Stack, Real.

SHARE THIS STORY

THE NEW STACK UPDATE
A newsletter digest of the week’s most important stories & analyses.