Article originally posted on InfoQ. Visit InfoQ

Today’s modern data architecture stacks look significantly different from the data architecture models from only a few years ago.

Data streaming and stream processing have become the core components of modern data architecture. Real time data streams are being managed as first-class citizens in data processing analytics solutions. Some companies are even shifting their architecture and technology thinking from “everything’s at rest” to “everything’s in motion.”

Change data capture (CDC) has become a critical design pattern in data engineering use cases. CDC can be used in event-driven microservices based applications, along with data streaming to implement robust solutions.

The emphasis on data streams  is also driving innovations in the data governance space such as the stream catalog and stream lineage.

Data mesh architecture, which has been getting a lot of attention recently, is built on four solid principles: domain ownership, data as a product, self-serve data infrastructure platform, and federated governance. Data mesh is expected to have a huge impact on the overall data management programs and initiatives in organizations.

Similar to many compute services in the cloud platforms, data storage services and databases now support serverless models where you only pay for what you use.

On the security and regulatory compliance side, data residency and data sovereignty  are getting a lot of attention to ensure the consumers’ data is protected and privacy is maintained throughout the life of the data.

Next-generation data engineering innovations will build on these recent trends to provide even more robust, secure, highly available and resilient data solutions to the development community.

In the InfoQ “Data Engineering Innovations” eMag, you’ll find up-to-date case studies and real-world data engineering solutions from technology SME’s and leading data practitioners in the industry.

Free download