home..
Data Mesh
June 2023
Some job desciption mentioned experience with data mesh concept is an advantage, so I decided to figure out what the data mesh means (Sounds like a 3-D mesh in computer vision for me…)
- In which case do we need a data mesh architecture?
In my previous experience, I only worked on four types in the structure of data analytics:
- In an NGO for a so-called “data management and impact assessment” position, I, as a volunteer intern, gathered information about policies and related statistics from government or other official websites. Organizing all the number in a report, and then hand them in directly. It is a very traditional way to utilize data for policy decision way back into 2017.
- In a tech-startup company, a mix of database and python/excel is used. Since the final products that they delivered to customers are mostly in the form of excel table and word report, so the necessary tools for a data analyst are basically mysql, excel, R/python. But a comprehensive framework is lack, leading to a strong reliance on the closely collaboration among colleagues.
- In the academical environment, on the other hand, independent works are more prevelant. Weekly meetup with professor is the time to align with the joint goal. The rest of the time is utilized to papar and github repository reading and coding. VS code plus github is a powerful method to work in academic, preventing from any overlaps. It’s clear and efficient.
- Back into the industry, I fortunately gained a great chance to be familiar with some commercial platforms, like services from Google and Microsoft. Azure DevOps Server, Azure Synapse (especially from Data Lake Storage to Notebook) and PowerBI jointly accelerates the process from pipeline to dashboard, unleashing potential of data and algorithms.