Is Big Data About to Be Usurped By Smart Data?

  • Richard Thelwell
  • October 22, 2015

smart data usurp big dataThese days, Big Data is, well, big. Browse the Internet, or open the business pages, and examples abound. And, to be frank, not a little hype, as well. But one question you don’t hear asked is this: is Big Data necessary? Or would Smart Data be better, instead?

Two people who would doubtless say ‘yes’ to that question are Edwina Dunn and her husband Clive Humby. And if those names sound vaguely familiar, that’s because they’re founders of data analytics business DunnHumby—the firm that in 1994 sparked a retail revolution by applying analytics to point of sale data at Tesco.

And their presentation to Tesco’s board of directors, following a three-month data analytics pilot study, has since passed into analytics folklore. Famously, it prompted Tesco’s then chairman, Lord MacLaurin, to observe:

“What scares me about this is that you know more about my customers after three months, than I know after 30 years.”

What followed was the launch of the Tesco Clubcard—in order to link point-of-sale data to customer demographic data—and a doubling of Tesco’s market share in just twelve months.

Smart samples

Think of Tesco, with its 38 million Clubcard customers, and its 3,500 UK stores, each with multiple checkouts busily capturing point of sale data, and it’s not difficult to think in terms of Big Data.

But in fact, what we now call Big Data wasn’t a part of the analytics innovation pioneered by DunnHumby. For a start, back in 1994, the technologies that we now associate with Big Data simply weren’t available.

smart data tesco clubcard
Smart data helped pioneer Tesco’s successful clubcard program.

Instead, DunnHumby realised that you didn’t actually need Big Data. What you needed was Smart Data. In other words, base your conclusions not on gargantuan analyses of all the data, but instead on skilfully-designed samples of data.

99% accuracy is good enough

On its own, Tesco had thought that it needed to analyse all the data it was proposing to collect with Clubcard—a conclusion that led its internal IT department to cost the task in terms of tens of millions of pounds, and a three-year rollout.

DunnHumby, on the other hand, advised capturing samples of customer data, and then extrapolating from those samples. This significantly reduced the scale of the task.

“We brought in a statistician’s mindset, which was ‘some of the data, some of the time’,” Ms Dunn has explained. “That is what the technology people at the time didn’t grasp.”

Put another way, it was possible to study a 10% sample and still achieve between 95% and 99% accuracy.

Smart Data, in short, and not Big Data.

Faster time-to-benefit

So what should ordinary businesses take away from this? Simply this, we believe: it’s not necessary to have Big Data scale, before you can get Big Data benefits.

Now to some, that is heresy. Seeing Big Data as some sort of ‘Ironman Challenge’, they make a point of boasting of the size the datasets they are handling, and the computer and software resource that is engaged on the task.

Needless to say, it’s an attitude that pretty much predisposes business to increased costs, and lengthened timescales. That said, such businesses can at least boast of truly Big Data volumes.

smart data big data too much
Big Data is great, but it can often be expensive and time consuming to handle.

But we’d sooner boast of Smart Data benefits, achieved at a fraction of the cost, and delivered in a fraction of the time.

Sampling commonsense

Now, again, to some people there’s an element of heresy here. Mention ‘sampling’, and they get suspicious. At least with Big Data, they argue, you’re analysing all the data—so the prospect of sampling bias is minimal.

So let’s go back to retail analytics, for a moment, to debunk that particular myth.

For a retailer like Tesco, with 3,500 stores, you’re clearly talking Big Data. But suppose you take just one superstore, and extrapolate from that? Or two, or three, or whatever number you’re comfortable with?

Is that really such a dumb idea? Of course not. It’s a smart idea—and Smart Data.

Because it would be distinctly odd if shoppers in Brighton, Bristol, Bedford or Blackpool had shopping habits that were wildly different from national norms.

Why wait?

Today, exploiting Smart Data—as opposed to Big Data—has never been easier.

And the Cloud—together with a new generation of self-service analytics and reporting software—can provide most of what’s required.

Put another way, a potential Big Data project that a business may have been mulling has never been easier to bring to fruition.

Do it with Big Data, by all means. But it’s probably smarter to do it with Smart Data.

To find out how you can make smart data work for you, download our free guide below