Data Mesh Simplified: A Reflection Of My Thoughts On Data Mesh

This is the republished article from https://www.dataengineeringweekly.com/. Subscribe to Data Engineering Weekly for your weekly data engineering news in the industry.

The Rise of Data Mesh

Data Mesh is a set of data engineering principles coined by Zhamak Dehghani from ThoughtWorks. I highly recommend reading How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh and Data Mesh Principles and Logical Architecture.

The sad state of data engineering

Now the fundamental question you may ask. Why Data Mesh and Why now? To understand Data Mesh, we need to understand the current state of the data engineering world. It may not directly apply to your organization, but most of the data infrastructure remains in this sad state.

Wait, Don’t Data Lake solved It?

A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is without having to structure the data.

  1. The consumers down the line have no domain understanding of the producer and struggle to understand the data lake data.
  2. The consumers then connect with the data producer to understand the data. At that point, the producer side’s domain expertise depends on human knowledge that may or may not available.

How Data Mesh Solve It?

Data Mesh is an enterprise data platform principle that converges the principles from Distributed Domain Driven Architecture, Self-serve Platform Design, and Thinking Data as a Product.

How Data Lake Different From Data Mesh?

But, There Is Always A Catch

Data Mesh sounds very cool, but there is always a catch. This is a great Twitter thread summarizing the challenges in adoption.

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

Conclusion

Data Mesh is not a technology or a storage solution instead of a set of principles to streamline its data management. As Gwen, Sriram, Kishore, and Vinoth pointed out, it is an invisible structure in most organizations and requires proper tooling to enable the Data Mesh principles.

I break things for living.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store