The article is an extraction of my thoughts on the recent data mesh discussions over Twitter. You can subscribe to my weekly data engineering newsletter to follow the latest updates on the data community.
Last week Data Twitter had some interesting discussions around Data Mesh. I share a few of my thoughts on the data mesh discussion here.
Almost five years ago, I met Dave (not the real name) at one of the tech conferences. Dave is the principal architect for one of the largest retail banks in Canada. We were exchanging common challenges in the data ingestion, observability and data silo. I’m fascinated and explains how the ingestion framework I worked on has in-build observability and scalable architecture. Dave responds to me with a smiling face, “Ananth, do you think a large bank with the capital at disposal can’t buy one of these systems. I don’t have a data silo, but a people silo. People hold on to their data as a negotiation tool, so all data problem becomes a trade-off resulting inefficient workaround. How will you fix the people silo and free the data?”
The people silo problem is still valid in most organizations. IMO I don’t see any scalability issue with a monolithic architecture where storage and compute can scale independently. The multi-tenant centralized storage with a clear separation of concern can scale with proper tooling. DBT is solving the silo with the domain view structure, but the instrumentation part is still challenging.
I genuinely believe a concept like Data Mesh and domain ownership much-needed one to validate data systems similar to the CAP theorem for distributed systems.
CAP theorem is not perfect eitherBut it is good enough to validate. Any vague and misleading concept leads to multiple interpretations that will only result in chaotic culture.
I believe the goal of Data Mesh is to spread the democratize data accessibility and break organization silo. A reference architecture for data mesh and a clear demonstration of how domain ownership brings accountability can simplify the concept by miles.