Combining data mesh and data fabric to drive greater data consumption

Organizations are increasingly focused on controlling and harnessing their data. As the volume and range of information they generate increases, they understand the need to successfully manage data and make it available for consumption at scale, by both business employees and AI models. 

This means they need to ensure that their data stacks are flexible, agile and able to meet future needs. Multiple modern data architectures have been proposed to ensure that data is both well-managed and easily-available, including data fabric and data mesh. Traditionally seen as opposing approaches, they actually complement each other, providing a combination of benefits when implemented together. Drawing on presentations at the Gartner Data & Analytics Summit in London, this blog explains the two architectures and outlines how they can integrate to create a future-proof approach that puts data at the heart of the organization and its operations.

As the number of data sources and systems have grown, organizations have built technical data stacks to effectively manage their data and turn it to business advantage. Beginning with enterprise data warehouses, these architectures have become more wide-ranging and complex as the volume of data and number of systems has increased exponentially. Data is now stored in silos across the organization, and beyond, and covers a wide range of types, from ERP and CRM to real-time web and Internet of Things (IoT) sensor data. 

To turn this data into value it needs to be made available to everyone (human and AI) within the business in a timely manner, and in trustworthy formats that they can easily consume, without requiring technical skills or support. At the same time data has to meet governance and compliance requirements, protecting sensitive information such as customer and financial details. 

Essentially, organizations need to connect the growing demand for data with effective supply in a way that balances flexibility with consistency and robust governance. This requires autonomous, agile, scalable data architectures, with data fabric and data mesh emerging as key approaches.

What is data fabric?

The data fabric approach aims to unify data across silos for knowledge discovery and to simplify data understanding. It works by creating a network or “fabric” that connects various data sources, allowing them to be integrated, transformed, and shared seamlessly. Key to this is the deployment of active metadata, which enables real-time, bidirectional data exchange across various tools in the data ecosystem by utilizing open APIs and semantic inference where systems understand the underlying meaning of data and use this to create insights and actions.

Gartner “Data fabric serves as a backbone for knowledge discovery. It observes the data residing in disparate applications and makes them ready for business exploration.”

Data fabric enables organizations to automate complex and repetitive data tasks and to improve data trust and use. However, many companies face challenges around immature metadata and difficulties in ensuring that their data is consumed across the business.

What is data mesh?

Data mesh decentralizes data ownership by empowering business units and departments through a federated approach. The aim is to assign responsibilities for particular data assets to those that are closest to them, but backed up by agreed, company-wide governance and metadata standards to ensure interoperability. There is more on data mesh in this previous Opendatasoft blog.

There are two key aspects of data mesh that drive data consumption and compliance – data products and federated governance. Data products are specific, actionable, continually-updated and high-value data assets that are provided in a ready-to-consume format to users, and serve a specific business need for a large group of people. When made available via a data product marketplace they connect non-technical users with the data they need to work more effectively and efficiently, maximizing value from data. Federated governance makes data owners within the business responsible for specific data assets, while following company-wide governance standards. 

Data mesh architectures rely on having strong data skills across the organization and a culture where different departments understand the need to work together collectively to turn data into value.

While they have often been painted as competing approaches, Gartner views them as increasingly complementary:

• Data fabric: Provides unified design architecture, active metadata and augmented data management
• Data mesh: Provides distributed delivery of data through data products, federated governance and maximizes the involvement of data practitioners across the organization

Organizations increasingly share the same view. In the 2024 Gartner Evolution of Data Management Survey 13% of companies surveyed had implemented both fabric and mesh. Already nearly 64% of organizations have decentralized data management teams in some shape or form, while 51% have federated governance. With over half of organizations aiming to deploy both approaches over the next two-three years, the majority are likely to harness elements of fabric and mesh together.

Gartner “By 2028, 80% of autonomous data products supporting “AI-ready data” use cases will emerge from a fabric and mesh complementary architecture.”

How can organizations successfully combine data mesh and data fabric to drive greater data consumption and more effective data management? According to Gartner, it starts by understanding the importance of decentralization, and following these six key best practices:

1 Invest in unified design and distributed delivery

Data mesh and data fabric have different roles in data management, and need to be deployed together to maximize their strengths. Businesses should use fabric as their foundational data management design pattern, organizing their data, while relying on mesh to optimize data delivery to users, whether human or AI.

2 Combine active metadata and federated governance

Given the complexity of data stacks and the growth of data assets, organizations need to be able to automate management while guaranteeing data is protected and secure. Embracing active metadata within data fabric architectures helps organizations to augment data management and increase efficiency. At the same time federating data governance involves data owners and helps ensure greater compliance and better data management. Gartner found that combining federated data governance and decentralized data management delivers a 2.5x greater impact on investment.

3 Bring together humans and AI

When it comes to data management, AI technologies such as knowledge graphs within data fabric deliver benefits in analysis and understanding. However, they work best when deployed in conjunction with human experts. Creating human/machine feedback loops provides a way of combining the strengths of both people and AI to optimize data management and maximize consumption and value.

4 Appoint data product managers

Old approaches where technical data experts answered individual queries from the business for reports or information simply doesn’t scale. Data products and data marketplaces empower business users to self-serve by providing them with seamless access to trusted, understandable and usable data. Deploying data products increases impact by 2.9x according to Gartner but requires an internal shift in mindset. Data owners need to become product focused, collaborating with their customers to create products that evolve to meet their specific needs, backed up by data contracts and company-wide data product templates to maximize efficiency.

5 Evolve your data architecture into an integrated and cohesive data ecosystem

Organizations now have increasingly complex data management stacks, with many business-critical solutions relying on current data flows. It is vital to move away from a rigid data architecture to a more flexible fabric-based approach that enables a more agile approach. However, evolution rather than revolution is vital – it is important to reuse as much as possible of the existing architecture, learning how to make it more flexible, rather than simply ripping it out and replacing it with costly new solutions.

6 Focus on skills as well as technology and strategy

Maximizing data consumption and increasing value is about much more than technology. It relies on people, ensuring they have the right skills and that culture supports greater data sharing. At a technical level it requires data owners within business units to be upskilled to manage data as products, broadening their capabilities and ensuring they have customer-focused mindsets. Ensuring that data architecture, strategy and operations meet business needs is paramount. Gartner recommends creating fusion teams that bring together data, technical and business people to guide and drive progress around this area. Finally, data democratization requires everyone within the organization to be comfortable and happy to use data, building data literacy skills so that they actively consume data products within their working lives.

Data leaders such as Chief Data Officers understand the imperative to increase data consumption to maximize data value. As Gartner states, this means creating flexible, automated and agile data architectures that better manage growing data volumes, enabling sharing and good governance. Data marketplaces complete this architecture, providing self-service access to data for business users through an intuitive, e-commerce style experience that enables data products and other data assets to be consumed at scale by all. This helps build data-driven businesses that are more effective, efficient, and innovative, now and moving forward.

Want to learn more about how Opendatasoft helps create your agile data architecture? Contact us to speak to our experts.