Data Mesh: Another Fad or a Viable Concept for Data Architecture?
Explore How To Implement a Successful Data Mesh Strategy by Decentralizing Data Products to a Domain Level
-
November 22, 2022
Downloads
Download Article
-
Data architecture has been the subject of many conceptual movements in the past 20 years, primarily in response to frustrations with data accessibility and usability for many businesses. Strategies to date, from centralised monolithic data warehouses to democratised data lakes, have delivered mixed results. With the right approach, a data mesh can be the solution to finally unlock the power of data.
Why Is Change Needed?
Consider the weaknesses of traditional centralised data architecture design:
- Central data engineers and analysts are expected to be experts in all domains.
- Centralised monoliths of data become difficult to maintain quickly.
- Centralised data functions are typically overwhelmed with demand for services.
- There is a disconnect among domain experts, data engineers and data scientists.
- Data governance is difficult to enforce centrally.
What Is a Data Mesh?
A data mesh is a concept that places full ownership of data and provision of data products into domains.
Domains are operational teams or departments (for example, marketing, finance, sales) that are responsible for curating raw data into usable, quality-assured datasets and reusable products. Their expertise is ‘closer’ to that of the data subject, enabling them to provide appropriate context which can be moulded into data for a) their own use and b) other domains to leverage.
These domains are responsible for maintaining directories and being the authority for definitions relating to their own department. This information is then housed in a ‘data marketplace’, a digital platform where users can both share data and subscribe to data across domains.
Subsequently, central data functions are able to shift focus from providing data solutions to providing data lakes and self-serve platforms.
Is a Data Mesh for Everyone?
Implementation of a data mesh is not for everyone. It requires highly data-savvy skill sets within organisational domains. To determine whether an organisation is ‘mesh-ready’, leaders need to evaluate their compatibility against five key criteria.
- Culture: For most organisations, a cultural change is required in each domain’s relationship with data. Domains must buy into a vision for delivering self-built datasets and products for their own use but also curated for the wider organisation.
- Ownership: Domains must assume a position of total ownership and authority over their data, how it’s defined and how everyone uses it appropriately.
- Skills: It’s important to acquire data-savvy talent to leverage self-serve tools appropriately and drive a healthy, sustainable data marketplace.
- Change: Executive support is required to change the fundamental building blocks and organisational structure of data. The responsibilities and initiatives within both central data functions and domains have an entirely new complexion.
- Investment: Implementation of a data mesh is transformational and therefore requires a strong commitment to the costs associated with the delivery of advanced data platforms and the significant operating model changes required.
Delivering a Successful Data Mesh
A data mesh can provide huge freedom for domains, but falling short on implementation controls has risks. Caution must be taken when moving towards a federated model as it could encourage data silos, fuel shadow IT and duplication of efforts.
To mitigate these risks and maximise the power of a data mesh, transformation programmes must shape their priorities around four key principles in order to be successful.
Domain Ownership
Placing ownership of data into domains comes with new resourcing requirements. Data management skills are required to curate and productise data in a quality- assured manner. Securing these skillsets — by either acquiring new talent or reskilling existing employees — and implementing these new responsibilities into domains are arguably the single largest factor in delivering a successful data mesh.
Adoption of and buy-in to a data mesh policy are critical for both the launch and the continued upkeep of the programme. For example, a policy might dictate that domains share products on an internal marketplace and always provide an accompanying glossary and a data quality scorecard.
Data as a Product
A step change in mentality is required, from treating data as a by-product of operations within an individual department to treating data as a reusable and sharable asset for all departments. To achieve this, it is important to define from the outset what constitutes a product.
A data product is any format of prepared data, from simple files to advanced functional software, which is:
- Valuable: Must deliver benefit.
- Usable: Easily consumable, quality assured.
- Available: Readily accessible and sharable.
Modern technologies can provide a digital data marketplace experience which directly enables the successful sharing and adoption of products. Delivery in this high-performing marketplace should focus on the following product characteristics:
- Navigable: Simple to explore and discover.
- Understandable: Accompanied with simple, concise glossaries
- Governed: Having quality-assured assets only.
- Appropriately Sized: Not cluttered with excessive volumes of files and apps.
Self-Serve Platforms
Historically, domains have struggled to build and maintain their own data products due to a lack of data accessibility or technical enablement, or the absence of the required skillsets. As a result, there is typically a handoff point from domains to data engineers, analysts and Business Intelligence (BI) resource. Domains feel separated from delivery, while the multi-team approach causes further disconnect among all parties. Delivery of well-equipped self-serve platforms is critical to eliminating these challenges.
Best-in-class self-serve platforms are:
- User Friendly: Data tools must be usable within domains and tailored for domain skillsets where advanced Developers don’t typically reside.
- Governed: Strong controls to manage alignment with data policies.
- Secure: Highest security standards, access controls and data leakage prevention.
- Data Protected by Design: Appropriate data hosting architecture design; automation of data masking, retention rules and overall data life-cycle management.
Domain Governance
Given that a data mesh accelerates the production and sharing of data assets, it is critical that clear governance controls are set in place, including the following:
- Clear Ownership Framework: Ownership of data assigned at the lowest reasonable level, split between executive ownership and ‘operationally responsibility’ across employees.
- Design Assurance: Centrally defined set of minimum delivery standards with a light-touch approval board.
- Data Policy: Strong controls to manage alignment with data policies.
For expert guidance on implementing a data mesh, contact our leadership team from FTI Consulting’s Digital Science practice.
Related Insights
Published
November 22, 2022
Key Contacts