Augmented data management: Data fabric versus data mesh

Gartner defines a data fabric as “a design concept that serves as an integrated layer of data and connecting processes. A data fabric utilizes continuous analytics over existing, discoverable and inferenced metadata to support the design, deployment and utilization of integrated and reusable datasets across all environments, including hybrid and multicloud platforms.” [1]

The data fabric architectural approach can simplify data access in an organization and facilitate self-service data consumption at scale. This approach breaks down data silos, allowing for new opportunities to shape data governance, data integration, single customer views and trustworthy AI implementations among other common industry use cases.

Since its uniquely metadata-driven, the abstraction layer of a data fabric makes it easier to model, integrate and query any data sources, build data pipelines, and integrate data in real-time. A data fabric also streamlines deriving insights from data through better data observability and data quality by automating manual tasks across data platforms using machine learning. This improves data engineering productivity and time-to-value for data consumers.

What’s a data mesh?

According to Forrester, “A data mesh is a decentralized sociotechnical approach to share, access and manage analytical data in complex and large-scale environments—within or across organizations using.” [2]

The data mesh architecture is an approach that aligns data sources by business domains, or functions, with data owners. With data ownership decentralization, data owners can create data products for their respective domains, meaning data consumers, both data scientist and business users, can use a combination of these data products for data analytics and data science.

The value of the data mesh approach is that it shifts the creation of data products to subject matter experts upstream who know the business domains best compared to relying on data engineers to cleanse and integrate data products downstream.

Furthermore, the data mesh accelerates re-use of data products by enabling a publish-and-subscribe model and leveraging APIs, which makes it easier for data consumers to get the data products they need including reliable updates.

How does a data fabric relate to a data mesh?

A data fabric and data mesh can co-exist. When it comes to data management, a data fabric provides the capabilities needed to implement and take full advantage of a data mesh by automating many of the tasks required to create data products and manage the lifecycle of data products. By using the flexibility of a data fabric foundation, you can implement a data mesh, continuing to take advantage of a use case centric data architecture regardless if your data resides on premises or in the cloud. In fact, there’s three ways a data fabric enables the implementation of a data mesh:

1. Provides data owners data products creation capabilities like cataloging data assets, transforming assets into products and following federated governance policies
2. Enable data owners and data consumers to use the data products in various ways such as publishing data products to the catalog, searching and find data products, and querying or visualizing data products leveraging data virtualization or using APIs.
3. Use insights from data fabric metadata to automate tasks by learning from patterns as part of the data product creation process or as part of the process of monitoring data product

A data fabric lays the foundation for data mesh.

Do you want to know more and keep on the discussion? Join the IBM team and Partners on our booth during Big data Expo event on Sep 14 and 15 !

1 “Data Fabric Architecture is Key to Modernizing Data Management and Data Integration” Gartner. 11 May 2021.

2. https://www.forrester.com/blogs/exposing-the-data-mesh-blind-side/