Data Mesh: Pushback to Centralized Repositories, Data Lakes, and Data Warehouses

Data is empowering and is changing the dynamics at many organizations. Greater access to data levels the business hierarchy. Organizations are growing flatter and staff members are being entrusted with greater authority and more decision capability.

Data itself though has seen a trend of centralization. Concepts like data warehousing and data lakes have tried to aggregate central stores of information. The two trends, organizational flattening and data centralization, have reached a point of incompatibility.

Gil Feig, co-founder of Merge API, said that “the notion of storing all data together within a centralized platform creates bottlenecks where everyone is largely dependent on everyone else. Data mesh addresses this head-on.”

The term Data Mesh was introduced by Zhamak Dehghani of Thoughtworks. Instead of centralization, Dehghani advocates a decentralized architecture where individual groups are able to access data more easily on a self-serve basis using an interface tailored to their usage and needs.

Dehghani said that “data warehousing, which we used for about 40 years, had been problematic at scale. This notion that we can get data from all different complex domains and put them in one model thousands of tables and thousands of reports, then we can really use that in an agile and nimble way, has been an unfulfilled promise. The way we have divided our organization and technology around these two separate [data] planes and the way we are integrating them through this ETL pipeline is the source of trouble just to start with. These data pipelines are very fragile and keeping them happy and healthy is very challenging.

Dehghani said that “in order to decentralize the monolithic data platform, we need to reverse how we think about data, it’s locality and ownership. Instead of flowing the data from domains into a centrally owned data lake or platform, domains need to host and serve their domain datasets in an easily consumable way.”

A Data Mesh uses decentralized architecture yet still has overall patchwork to communicate between the different data sets.

Justin Borgman, Starburst CEO, said that “the reason I find a data mesh particularly attractive is simply that it reflects to me what reality naturally looks like. It is so rare for a customer to truly implement the enterprise data warehouse to its fullest extent. To be able to actually have everything that you need in one place is, practically speaking, very challenging to do.”