Big Data Market Segment LS
Big Data Market Segment RS
Monday, 21 December 2020 14:20

Data mesh decentralises custodianship while maintaining governance

ThoughtWorks data and AI practice lead Dave Colls ThoughtWorks data and AI practice lead Dave Colls

The data mesh concept is similar to microservices but for data rather than code, explains ThoughtWorks data and AI practice lead Dave Colls.

Centralised data platforms present various problems, says Colls. They can be bottlenecks, which limits innovation – a criticism that has previously been applied to other aspects of centralised IT.

And data scientists don't necessarily understand business requirements, while people working on the business side tend to lack data skills. This latter point helps explain why siloed projects tend to fail.

But data mesh recognises and meets the need of data consumers by decentralising ownership to those close to the production or use of the specific data set, with a governance layer to take care of issues including lineage, discovery, access, change management, and the reuse of patterns and resources.

In this way, data mesh improves speed and quality, Colls says.

Importantly, adopting data mesh does not require a big bang approach. "It can be done iteratively in small steps," he says.

You can carve off an individual use case and use that to prove the idea is relevant and can deliver business value.

So an organisation might start with product and customer data, plus relevant telemetry and third-party data. Once they have been exposed in consistent formats, they can be built into downstream applications.

For example, a retailer might make POS data available more widely, including to live analytics. Or geospatial data could be exposed in realtime and in various formats giving users a choice between freshness and consistency.

Such projects are mostly about "encapsulating existing data with a consistent way of accessing it," and making it easy to access through "a series of mini data warehouses," although "the data mesh concept of a 'data product' provides different and additional capabilities over a data warehouse."

Traditional data warehouses tend to be modelled with a particular view in mind, but data mesh's decentralised approach mean these mini data warehouses can support different requirements. For instance, security and marketing departments have very different views of web traffic data.

This approach is beginning to be reflected in conventional data stores from a variety of major vendors, where transaction processing and analytics applications are combined.

Discoverability – "knowing what data you have" – is an important consideration.

According to Colls, there is lots of 'dark data' in many organisations, ie, data that is not discoverable or easy to access.

This can lead to duplicated effort or squandered resources, but cataloguing and maintaining data becomes an overwhelming task if done centrally.

The data mesh model accommodates a network of data stewards responsible for cataloguing and maintaining particular datasets and the associated metadata, and explaining what it can be used for.

This is as much an organisational problem as a technical problem, he explains, as the organisation and the individual stewards need to see the value of good custodianship.

There's a parallel with the DevOps mindset, Colls suggests. That can be seen as cooperation to make data available more quickly and reliably.

Governance is necessary, but it is important to balance the freedom to work according to the specific circumstances with the need to ensure the community works as an ecosystem, says Colls.

So it's necessary to define the key interfaces in terms of the data <I>and</I> the governance controls, and then do the implementation appropriately.

Service levels must be taken into consideration, as they need to reflect the needs of data consumers in other parts of the organisation. Architectural forums and fitness functions can play a part, but evidence-based decisions are essential. It is important that teams are evaluated not just in terms of the data functions they deliver but also the achievement of agreed service levels.

ThoughtWorks has completed data mesh projects that were built on blob storage, relational databases and streaming services, both on-premises and in the cloud.

There can be challenges around resource limits. Colls warns, especially as billing models may limit the elasticity of the underlying services.

And while there are efficiencies in reusing data resources, it is important to leave each originating team with the flexibility they need when working within their own domain.

ThoughtWorks' engagements with clients has led to the development of a shared set of data mesh principles and architectures. The company now wants to engage with others to develop standards and tooling to help broader adoption.

This has the potential to increase speed when working with data, and improve the quality of outputs, he says.

"We're trying to avoid this being a proprietary thing."

Industries adopting or intending to adopt data mesh include retail, financial services, health, consumer products and media.

"As each industry becomes more digitised, there's more data to work with," says Colls.

Subscribe to ITWIRE UPDATE Newsletter here


It's all about Webinars.

Marketing budgets are now focused on Webinars combined with Lead Generation.

If you wish to promote a Webinar we recommend at least a 3 to 4 week campaign prior to your event.

The iTWire campaign will include extensive adverts on our News Site and prominent Newsletter promotion and Promotional News & Editorial. Plus a video interview of the key speaker on iTWire TV which will be used in Promotional Posts on the iTWire Home Page.

Now we are coming out of Lockdown iTWire will be focussed to assisting with your webinatrs and campaigns and assassistance via part payments and extended terms, a Webinar Business Booster Pack and other supportive programs. We can also create your adverts and written content plus coordinate your video interview.

We look forward to discussing your campaign goals with you. Please click the button below.



iTWire TV offers a unique value to the Tech Sector by providing a range of video interviews, news, views and reviews, and also provides the opportunity for vendors to promote your company and your marketing messages.

We work with you to develop the message and conduct the interview or product review in a safe and collaborative way. Unlike other Tech YouTube channels, we create a story around your message and post that on the homepage of ITWire, linking to your message.

In addition, your interview post message can be displayed in up to 7 different post displays on our the site to drive traffic and readers to your video content and downloads. This can be a significant Lead Generation opportunity for your business.

We also provide 3 videos in one recording/sitting if you require so that you have a series of videos to promote to your customers. Your sales team can add your emails to sales collateral and to the footer of their sales and marketing emails.

See the latest in Tech News, Views, Interviews, Reviews, Product Promos and Events. Plus funny videos from our readers and customers.


Stephen Withers

Stephen Withers is one of Australia¹s most experienced IT journalists, having begun his career in the days of 8-bit 'microcomputers'. He covers the gamut from gadgets to enterprise systems. In previous lives he has been an academic, a systems programmer, an IT support manager, and an online services manager. Stephen holds an honours degree in Management Sciences and a PhD in Industrial and Business Studies.

Share News tips for the iTWire Journalists? Your tip will be anonymous




Guest Opinion

Guest Reviews

Guest Research

Guest Research & Case Studies

Channel News