There’s clearly strength in numbers but how can we sort out what’s relevant?

Leveraging the vast amounts of data that has been collected over the years is something that the world of business has been buzzing over for quite a time now. We’re told that we are living in the era of ”Big Data” and to remember to put our ‘white science-coats’ on when crunching the numbers.

“We’re really just getting under way. But the march of quantification, made possible by enormous new sources of data, will sweep through academia, business, and government. There is no area that is going to be untouched,” said Gary King, director of Harvard’s Institute for Quantitative Social Science, as quoted in the New York Times as early as 2012.

Still only a handful of companies seem able to connect the data dots in such a way that it simplifies their business efforts instead of making them more complicated.

The more data we have, the more likely we are to drown in it.

If some data is better than no data, and a lot of data is better than some data, then shouldn’t an enormous chunk of data be the best of all?

Being more evidence-led is generally a good thing and the insights from the generated data have helped us to take enormous leaps in various fields of research and business. It’s clear that the ever-increasing amount of data generates new opportunities, but the problem lies in hearing the useful patterns and signals from amongst all the noise. As Nassim Nicholas Taleb, Professor of Risk Engineering and author of The Black Swan, puts it: ”The more data we have, the more likely we are to drown in it”.

It’s the pattern, stupid

By its nature, ’Big Data’ refers to examining behavior on the level of mass. Still, one could easily get the illusion that collecting vast amounts of customer data is synonymous with accurate prediction of customer behavior at the individual level. Predicting behavior even at the population-level is hard and hitting accurate consumer behavior predictions at the individual level can be a daunting, if not an impossible task. The customer-level data, that companies have access to, can be fundamentally biased.

It turns out that the real value of having big quantities of customer data resides in spotting useful patterns, something that the human mind is exceptionally good at. ‘Selective attention’ as some experts call it. However, the people-centric approach does not scale well. Machines already exceed humans in terms of raw computing power and are well on their way to becoming artificially intelligent thinkers and pattern spotters, as described by AI pioneer Ray Kurzweil in How to Create a Mind: The Secret of Human Thought Revealed.

So it’s the patterns in data that matter – and they matter a lot! One of the highest valued companies in the big data pattern recognition business is Palantir Technologies, the partly CIA-funded Silicon Valley company chaired by none other than the PayPal co-founder and technologist Peter Thiel. Palantir’s software consumes huge amounts of data from different sources, integrates it and gets unlikely patterns to emerge from the data. Still, even Thiel is somewhat skeptical when it comes to big data.

“If you hear the words ‘cloud computing,’ ‘big data,’ ‘machine learning,’ you just think fraud and run away as fast as you can. You should be scared of buzzwords,” Inc Magazine quoted Thiel as saying at the Post Seed Conference.

What’s the return on investment?

What successful pattern recognition means for companies is that a clear business objective must be set before crunching the numbers. Just going through a pile of data trying to find ’something’ is more than likely just a worthless fishing expedition.

Even if more valuable insights sometimes lie in asking the right questions of smaller data sets, the fact that there’s more and more data to use along with the computing power to crunch it, seems to be a productive asset for companies. Professor Erik Brynjolfsson from Massachusetts Institute of Technology and two of his colleagues from MIT and Wharton concluded in their 2011 research that companies that use data and business analytics to guide decision making are more productive and experience higher returns on equity than competitors that don’t.

”By analyzing a large sample of firms, we find that DDD (’data driven decision making’) is indeed associated with higher productivity and market value, and that there is some evidence that DDD is associated with certain measures of profitability (ROE, asset utilization),” the researchers wrote. However, there are also findings that say the Big Data ROI is tough to measure.

Until a company learns how to use data and analysis to support its operating decisions, it will not be in a position to benefit from big data.

In the middle of all the hype and big corporate Big Data investment decisions, the real reminder is that the value of data lies more in the questions asked of it than in the data itself. As Jeanne Ross, Cynthia Beath and Anne Quaadgras put it in their Harvard Business Review article: ”Until a company learns how to use data and analysis to support its operating decisions, it will not be in a position to benefit from big data.”

---

References: Brynjolfsson, Erik, Lorin M. Hitt, and Heekyung Hellen Kim. “Strength in Numbers: How Does Data-Driven Decisionmaking Affect Firm Performance?.” (2011).
Available here.

Photo credit: Shutterstock