Up close and personal: big data, supermarket style

What’s your favourite meal? A juicy, rare steak with some hot, crispy chips? How would you feel if your local supermarket buzzed your smartphone with a tempting promotion on steak and chips next time you walked past it. Would you be startled?

You shouldn’t be. They only sent you the offer because you told them it was your favourite meal, even if you didn’t realise you were doing it at the time. They’ve been tracking your purchasing habits for years anyway, but earlier on today you updated your Facebook status, hankering after steak and chips - informing them that not only do you like steak and chips, but you also fancy it for dinner tonight.

Then your smartphone location GPS data told them, in real time, that you were approaching one of their stores. And once inside, the same GPS data tracked what aisles you were in. What better time to send you a targeted promotion offering you some money off? Or to buzz you details of a second deal, on an expensive Pinot Noir (which you tweeted about last night).

Welcome to big data, supermarket style. The principle is the same as underpinned the data revolution sparked by Clubcard in 1995 - gather detailed information about your customers so you can understand them better and sell them more product. But big data raises the bar because it offers retailers details they can’t get from their existing loyalty schemes. That’s why it’s so ‘big’. So why aren’t more retailers going big on big data?

On the face of it, big data looks like a no-brainer. As Chris Withers, smarter commerce lead at IBM UK and Ireland, says: “The future is to collate information from outside your own enterprise and analyse what you can derive from the resulting combination. Whether you are data-scraping Facebook likes, product reviews, chatter about brands, or monitoring locations, that combination of internal and external data is the transformational step - it’s where the real return is.”

The virtue of big data is that it allows you to create far more accurate consumer profiles, he adds. “The old adage is that 50% of marketing is wastage but the trouble is, you don’t know which 50% is which,” he explains. “With big data you can start to figure that out. And there are so many use cases. Big data can be truly revolutionary in terms of wallet share, margins and inventory if retailers can master that unstructured data to augment their customer profiles.”

In theory, it couldn’t be simpler. Tesco could take data from a Clubcard used by a family of four, say, and augment it using big data taken from Twitter, Facebook etc into four granular sets of individual data - which could then be used to generate four sets of personalised promotions. For example, if it already knew the family bought red and white wine, big data could help it establish that mum likes red and dad likes white.

In practice, however, it’s not that straightforward. Huge volumes of data are created every day and sifting through it to find the gold isn’t easy. In the two minutes since you started reading this article, 70,000 people ‘liked’ a brand or organisation on Facebook, Google performed 4,000,000 web searches and 200,000 people tweeted tweets.

“Everyone is grappling with social media,” says Paul Winsor, retail industry director at Teradata. “If you look at social media data, the volumes created are ridiculous. But although there is a lot of useful information out there about brands, there is also lots of noise you don’t need that divulges nothing about their shopping habits.”

That noise is one of the big challenges of big data. It’s chaotic and messy compared with the information derived from a neat grocery shopping session. “Online shopping is far richer in terms of valuable data about customers individual preferences than social media,” says Winsor of online shopping. “And supermarkets are struggling with that online grocery customer data right now, let alone the rest of it.”

But social media data is still valuable, says Cindy Etsell, commercial marketing manager at SAS. Especially once you clean it up. “If you put garbage in, you get garbage out the other side,” she says. “The key is to extract the really useful information, harness it and use it effectively. You just have to make sure it is clean so you can talk to it, and there are so few retailers able to do that. Social media is just adding to the pile. The mobile area is adding to it too - it’s becoming more and more difficult to handle the increasing volumes of data involved. A lot of retailers don’t know where to start. It’s a mega challenge.”

One that retailers can begin to meet by applying high performance analytics to their data. The fun starts once the data starts “talking back”, as she puts it. “I want to walk into a store and get a message saying ‘Great to see you again, Cindy. I know four weeks ago you bought laundry detergent, so you’re probably running low. Here’s a special price on it today.’”

In short, a retailer can start targeting customers with personalised offers it knows they really want. So Sainsbury’s could use its Nectar data to establish that a shopper loves pizza, for example, and use external big data to establish that they drive past Sainsbury’s every day at around 6.30pm. Then it could combine the two data sets and ping that shopper an offer for pizza at 6.15 pm when his stomach starts to rumble and he starts to think about stopping off somewhere to pick up some dinner.

It’s genius. But it’s also a little bit creepy, which is where the other big issue with big data lies for many people. “Location data can be used to get customers into the store and prompt purchasing, but it also raises ethical dilemmas,” says Duncan Ross, director of data science at Teradata. “You can learn an incredible amount about people through this rich big data set that is generated nowadays, but perhaps it’s not a case of can I, but should I?”

Recent studies in the US indicated that people were prepared to divulge really personal data for just 60c, he says: “It’s surprisingly little.” But US retailer Target learned the hard way what happens if you get too close - and it was just using the customer data it had already, rather than any supplementary data.

While retailers are generating “actionable insight” from their analysis of big data, says Winsor, acting on it is proving difficult, often because the infrastructure is simply not there. “For a lot of retailers, the customer insight department is Bob on the third floor,” he says. “I’m exaggerating, but there are companies out there that have surprised me with the lack of organisation in place. There is a definite skills shortage. It needs investment and organisation. Whatever happens in the next five years, the lack of data is not going to be a problem. It’s the management of that data and how retailers use it. The challenge is going to be getting properly set up so they are ready to use it.”

“Big data can be truly revolutionary in terms of wallet share, margins and inventory if retailers can master unstructured data”

The good news is that there’s a wealth of data that if used effectively through relevant, personalised, well timed and therefore welcome promotions, will boost customer retention - and retail revenues, says Etsell.

“A retailer might have millions of customers with millions of variables, which can create huge amounts of scrambled data,” she elaborates. “But we can analyse it in three minutes. A 1% lift in retention for a retailer with millions of customers could equate to £120m in additional revenue every year, so making these improvements can have a major effect on sales. And retailers are struggling right now, they are pushing cost out of their stores, their warehouses, their people and the supply chain. So now is the time to get in there.”

Etsell believes a “gradual turnaround” is taking place in terms of retailers realising the potential of big data. “Then it’s going to go exponential,” she says. “If you know your customers inside out, that is the store of the future - one based on thorough data analysis of your customers’ personal preferences. Customers want you to talk to them personally and companies that don’t get it right will be left behind. And if you aren’t going to rise to the occasion, then what is the point?”

Tesco won’t be caught napping. In June, CEO Philip Clarke told the Consumer Goods Forum that in-store Wi-Fi would let it combine Clubcard data with smartphone data to target shoppers with “suggestions about what to buy” and “point out offers” as they wander the aisles. The era of “mass personalisation” had arrived, he said, offering up “the most valuable asset of all” as a reward - shopper loyalty. Tesco would also be “embracing social media”, he added. “We’re not simply following the customer, we’re standing alongside them, making suggestions, being their loyal partner. And that, ladies and gentlemen, is the way to thrive.”

In an environment where most retailers are currently focused on surviving rather than thriving, big data could make the big difference.