Editor | On 14, Jun 2018

Darrell Mann

“HURRICANE FRANCES was on its way, barreling across the Caribbean, threatening a direct hit on Florida’s Atlantic coast. Residents made for higher ground, but far away, in Bentonville, Ark., executives at Wal-Mart Stores decided that the situation offered a great opportunity for one of their newest data-driven weapons, something that the company calls predictive technology. A week ahead of the storm’s landfall, Linda M. Dillman, Wal-Mart’s chief information officer, pressed her staff to come up with forecasts based on what had happened when Hurricane Charley struck several weeks earlier. Backed by the trillions of bytes’ worth of shopper history that is stored in Wal-Mart’s computer network, she felt that the company could “start predicting what’s going to happen, instead of waiting for it to happen,” as she put it. The experts mined the data and found that the stores would indeed need certain products — and not just the usual flashlights. “We didn’t know in the past that strawberry Pop-Tarts increase in sales, like seven times their normal sales rate, ahead of a hurricane,” Ms. Dillman said.
New York Times, November 14, 2004.

The Wal-Mart Pop-Tart-Hurricane ‘prediction’ story has been used almost endlessly by the Big Data Analytics industry since Wal-Mart went public back in 2004. Mainly as a proof-point of the value being delivered by the industry’s new found ability to acquire, store and analyse terabytes worth of data. Here’s a typical example of some of the hyperbole surrounding the story:

Figure 1: BDA Predicts Pop Tart Sales? 

Pop Tart sales rise before hurricanes. Apparently, ‘nobody could have made that hypothesis’. Except, of course, if they decided to engage their brain and spend more than a couple of minutes thinking about the problem. No need to go and interview people that have been involved in hurricanes in the past. Just think.

But, of course, computers mean we don’t need to think any more. We just have to collect lots of data and then ask the computer to find interesting correlations. In the case of Wal-Mart in 2004 inevitably the ‘data’ they had access to was structured numerical data. A mathematician’s delight. Mathematicians love numbers. Give them enough numbers and enough computing power and they’ll find all the correlations you’ll ever want. And then several more you probably don’t.

Sometimes these correlations can be useful. Oftentimes they’re not. In the case of pre-hurricane behaviour the Wal-Mart team’s ‘discovery’ about Pop Tart sales increases turned out to be a useful one. Hurricanes represent a fairly specific contextual framework around which to go and look for correlations. ‘What correlates with hurricanes?’ is, in the context of looking for correlations, a pretty good question to ask. If Pop Tart sales increased by a factor of seven every time there’s a hurricane forecast, then it would be fairly sensible to assume that the same thing will happen next time there is a hurricane. Or at least it would until such times as someone stops analyzing the numbers, starts thinking, and innovates.

Such is the problem with optimization thinking. Wal-Mart’s ‘prediction’ allowed them to anticipate the next Pop Tart surge by looking at weather forecasts, and then act upon the prediction to make sure the supermarket shelves were suitably seven-times over-stocked in anticipation. This is called using data to optimize your operations. The key word being optimize. As in ‘not innovate’.

Here’s the main problem with the BDA industry right now, it is caught in this kind of optimization-thinking prediction trap. Great to be able to optimize your ordering and delivery logistics to sell more Pop Tarts, but settling for ‘optimization’ almost inevitably means you’ve missed a much bigger innovation opportunity.

True enough, analyzing all that data meant no-one had to think, but I suspect that if Linda Dillman had’ve asked her staff to not trawl through the terabytes of data, but instead to sit down together in a room for an hour and brainstorm what people want when there’s a hurricane on the way, one suspects they might have come up with some rather stronger ideas.

By all means use the Pop Tart data to inform the session, but then instead of accepting the information at face value, what if the team had started to think about the characteristics of Pop Tarts that made them valuable in a hurricane. Things like:

• Can be eaten hot or cold
• High calorie content
• Physically compact
• Long shelf-life
• Comfort food
• A slightly sinful treat for parents to give to the kids when they’re in need of cheering up

Having done this, what they could then have done is said to themselves, ‘are there other products that meet these criteria?’ In which case they might well have identified a whole bunch of other foodstuffs that customers could be encouraged to purchase. The fact that the customer ‘so-far’ hadn’t connected such foodstuffs to hurricane emergency, comfort food means there was a messaging innovation opportunity.

Even better, they could’ve gone back to the food industry and put out a specification for new product designs. Then we might have had some proper innovation.

Funnily enough, I’m not desperately interested in Pop Tarts. Or the Wal-Mart story. Except for the way in which it highlights some of the main problems of the Big Data Analytics world. They published the story as a big success, when in reality, from an innovation opportunity perspective, it was (and still is) a thoroughly lost opportunity. Two points in particular are worth highlighting and remembering next time your BDA team comes to you with their latest ‘prediction’ revelation:

1. Numerical data – irrespective of how many terabytes of the stuff you might have in your possession – allows you to only optimize what you’re already doing. Optimization is the opposite of innovation. Numerical data allows analysts to find correlations. Correlation begets optimization. Correlation, though, has nothing to do with causation. And causation is what you need if you’re going to innovate. In the Pop-Tart story, the causal links between the product and the hurricane is all the hidden stuff like ‘comfort food’. What the data analysis should have prompted was a search for causal links between hurricanes and comfort food. And then, having found those causal links, found better ways than Pop Tarts to bridge them. Like self-heating chocolate drinks maybe? Or beer? Causation begets innovation.
2. Recognizing that the BDA analysts are probably allergic to the idea of thinking (‘doesn’t the computer do that for me?’), there are ways to automatically capture causal links these days. Unfortunately, those ways don’t involve analyzing numbers. Numbers correlate. In order to find causal links, you need to examine the un-structured narrative, and ‘read between the lines’. You need to listen to what people are frustrated about, angry about and fearful of, and then see what those emotions lead to. And if that sounds like a job for PanSensic and Perception Mapping, you’re probably right. The reason we’ve built PanSensic is because we come from the innovation world not the optimization world. PanSensic is designed specifically to reveal causality. And, to repeat again, causation begets innovation. It’s the important stuff. Stuff the BDA analysts, unfortunately for them, don’t get yet.