Friday, 02 March 2018 09:22

Collecting data with a purpose to avoid the 'Big Data Trap'

In the age of Big Data, the rule seems to be “Collect everything now, figure out what to do with it later”. Unfortunately, this is often a costly mistake. Cloud storage is getting cheaper by the day – so you certainly can do it – but the real question is why should you do it?

By kyeling, December 18, 2017

Without a strategy for data, the methodology of getting a lot of storage space, gathering all the possible data and then setting up a team to analyse it can easily turn you into a data hoarder. Although it feels great to brag about how many Tb of data we have stored, we tend to forget that having more Tb of data will limit how fast and flexible we can be when the time comes to analyse it: quality over quantity still applies to data analysis.

Firstly, we need to go back to why we collect data. It is the first step in a process that also involves transforming that data into a format suitable for analysis followed by a methodical examination of relationships, patterns and trends to find answers to questions that are usually not immediately obvious. Collecting data is after all just a means to an end.

Also, deciding what data to track can be hard as it’s not always obvious how certain variables can be used in the future or even what the business goals will be down the line. The fact is, we are constantly expanding our analysis and enriching our datasets, so we may easily overlook the relevance of new data as it becomes available. In my experience as a consultant and CDO I have found myself in situations where I was invited to join or execute a “dream project” involving “Big Data” just to find out that the data collected was often duplicated, redundant or simply not fit for the purpose. After spending years and large sums collecting and storing this data, they realized that doing so without first having a clear use case was an expensive mistake.

Other usual mistakes that I find include collecting data that:

  •     Cannot be linked to the existing datasets by any universal identifier/key or used to enrich existing datasets;
  •     Is not needed to solve any underlying business problem or cannot be associated to any of the goals or strategic priorities;
  •     Has a large cost to collect, store and analyse but has no clear value proposition;
  •     Is collected erratically or generates datasets that are unbalanced in ways that cannot be tracked and corrected, generating biased analysis and leading to wrong conclusions
  •     Has no associated plan on when and how to be used, what are the desired results to be achieved and how they can be measured.

The best way to prevent falling into the “Big Data trap” is to start by defining a strategy for data, setting clear business goals and assessing the value to the business, then working backwards to find out what kind of analysis and datasets we need to best achieve these goals, pointing us towards what we need to collect and analyse both now and into the future.

After all, if the reason we need information it to make better decisions, understanding which ones are the most important and deliver the best return to our businesses will guarantee the relevance of our projects, contributions towards the overall success and ultimately improve the satisfaction of both our teams and stakeholders.

Read 33888 times

Subscribe to ITWIRE UPDATE Newsletter here


The much awaited iTWire Shop is now open to our readers.

Visit the iTWire Shop, a leading destination for stylish accessories, gear & gadgets, lifestyle products and everyday portable office essentials, drones, zoom lenses for smartphones, software and online training.

PLUS Big Brands include: Apple, Lenovo, LG, Samsung, Sennheiser and many more.

Products available for any country.

We hope you enjoy and find value in the much anticipated iTWire Shop.



iTWire TV offers a unique value to the Tech Sector by providing a range of video interviews, news, views and reviews, and also provides the opportunity for vendors to promote your company and your marketing messages.

We work with you to develop the message and conduct the interview or product review in a safe and collaborative way. Unlike other Tech YouTube channels, we create a story around your message and post that on the homepage of ITWire, linking to your message.

In addition, your interview post message can be displayed in up to 7 different post displays on our the site to drive traffic and readers to your video content and downloads. This can be a significant Lead Generation opportunity for your business.

We also provide 3 videos in one recording/sitting if you require so that you have a series of videos to promote to your customers. Your sales team can add your emails to sales collateral and to the footer of their sales and marketing emails.

See the latest in Tech News, Views, Interviews, Reviews, Product Promos and Events. Plus funny videos from our readers and customers.



Related items

Share News tips for the iTWire Journalists? Your tip will be anonymous




Guest Opinion

Guest Interviews

Guest Reviews

Guest Research

Guest Research & Case Studies

Channel News