Editor | On 15, May 2018

Darrell Mann

If you ever need to see evidence of how the Operational Excellence world optimizes itself into a cul-de-sac from which it can never seemingly return, you need look no further than the world of the Net Promoter Score (NPS). AKA ‘the ultimate question’. If you’ve ever been in a restaurant, or on a plane, or in a British hospital, and been asked the ‘how likely are you to recommend us to a friend or family?’ question, you already know where this article is going.
The whole shebang started, like a lot of management fads, with an article in a learned journal, in this case the Harvard Business Review. Enough people like the article that the author is encouraged to write a book. And then enough people buy the book that the publisher’s demand a follow-up. Something like this:

Figure 1: Evolution Of A Management Fad

Despite the irony that an ‘ultimate question’ would ever need a version 2.0, it’s easy to see how managers fell into the trap. The whole thing starts from the author’s discovery of a correlation between the share price of an organization and how much the customers of that organization said nice things about them. The key word in that sentence being ‘correlation’. The second key element being the slightly more subtle idea that something that sounds inherently obvious (‘successful companies have happy customers’, duh) now had actual data to ‘prove’ that it was true. Not only that, but now managers had access to a standard measurement scale that held the potential to benchmark themselves against others, and – perhaps more importantly – to monitor their own progress. CEOs spend a lot of their time worrying about things like share-price, and so anything that purports to help them to ‘manage’ the unmanageable found a highly receptive audience. CEOs quickly came to love NPS. It allowed them to feel more in control of the business. And a weapon to beat-up business unit heads when they could see the NPS heading in the wrong direction.

So far so good. But not, sadly, if you were one of the beaten-up managers. Very quickly the people tasked with actually looking after customers grew to hate NPS. The problem with managing businesses with the one ‘ultimate question’ score is that it knowing that your score had dropped three-points this quarter offered up no insight at all into why. Or, more importantly, what you might need to do to get things back on track next month.

Reducing complex situations – like ‘customers’ – down to a single number sounds great if you’re the CEO. Sadly, unless you’ve accidentally hit upon the fundamental ‘DNA’ from which the complexity emerges, your attempt is destined to end in failure and frustration.

As soon as we see a book called ‘The Ultimate Question 2.0’, we can be fairly certain that said DNA has not been revealed. Author, Fred Reichheld, himself has been forced in recent years to concede that his research too has fallen into the frequently fallen into trap of mistaking correlation and causation.

A big part of the ‘ultimate question’ fallacy is that people asked the question rapidly become jaded. The first time I see the question as a customer, I’m likely to find it intriguing. It makes me think about my friends and family, and how I think they might enjoy what I’ve just experienced. It ticks all the ABC emotional boxes: I’m in control (Autonomy), it connects me to my (Belonging) tribe, and, hey, all I have to do a rank my experience on a Likert Scale, so I feel Competent.

But then, the second time I’m exposed to the question, it’s less intriguing than it was the first time. From a Kano perspective, the Exciter is already a lot less exciting. By the time I’ve seen the bloody question for a fifth or sixth time, I’m thoroughly dis-enchanted by the whole charade. The damn restaurant hasn’t got any better than the first time I visited, so why do I even bother? Or, because by asking me yet again, they’re starting to annoy me, why don’t I just lower the score. That’ll show them.

And so the downward spiral of irrelevance begins.

Let’s call it a contradiction. A conflict between a providers desire to understand what their customers think about them, and their inability to acquire a meaningful measurement. Or at least not an easy measurement. All NPS has done, it seems, is to trade meaning for convenience. Whereas, if Reichheld et al had sat down with the intention of actually solving the contradiction rather than trading off, they might have ended up somewhere else entirely.

Somewhere like PanSensic, perhaps? Not that this is intended to be an advertisement for our tools, just that things like the Net Promoter Score cause us to remember why we got into the measurement game in the first place.

Job one is to make use of existing, ideally ‘free’, data, rather than bludgeoning customers into answering questions they have no interest in answering. Job two is to make sure that the data is unstructured narrative in sufficient quantity that I’m able to have enough lines to read between, and to establish how representative, relevant, reliable, and congruent the words are.

Once I’ve got those two things, I’m well on the way to being able to conduct a sentiment analysis like the exemplar shown in Figure 2, an analysis of the blog content of a host of American truckers (a cohort, it turns out, that seems to find an awful lot of time on their hands to write about their lives!):

Figure 2: Exemplar Sentiment Analysis

This kind of analysis is standard fare as far as PanSensic is concerned. The steps from this analysis to a Net Promoter Score are then fairly short and sweet. Firstly, we need to calculate some kind of ratio between the positive and negative emotions being expressed. Second, we need to somehow cross-calibrate these ratio scores to some NPS data. This step is rather more troublesome, since it is far from clear that there is such a thing as NPS data that is in any way truthful due to the fundamental flaws in the whole system. In this regard, our main hope for the future is that, having given the world a basic scale of measurement, the method of scoring an organization on that scale has the chance to evolve away from the ‘ultimate question’ fallacy.

The best we’ve been able to do so far is to simply take lots of NPS data – such as the Figure 3 analysis taken from a cross-section of Australian customer perspectives – scrape lots of customer narrative around the different product domains being measured for the same time period, and use a machine learning algorithm to find the appropriate weighting of each of the sentiment analysis figures in Figure 2 relative to one another.

Figure 3: Typical NPS Data Acquired Through ‘Ultimate Question’ Survey Means

Do this enough times, and hey presto, what you end up with is effectively an automatic NPS tracker. Just point it at whatever social media narrative you determine is relevant (Trip Advisor reviews, Facebook, etc) and watch what your actual Not Promoter Score is doing – Figure 4. I say, ‘Not’ to avoid infringing on Dr Reichheld’s copyright and also because, one of the things we’re finding is that the reality we’re finding is that most customers who say they’re likely to recommend your amazing restaurant or not-so-smart-watch to their family and friends don’t actually mean it.

Figure 4: Post Machine-Learning Training Of Not Promoter Score Algorithm

Then comes the best bit. The bit that solves the CEO versus manager contradiction. The automated Not Promoter Score calculator gives the CEO the magic number he or she is looking for. But rather than being the blunt instrument then used to bludgeon the customer interfacing managers with, PanSensic allows them to drill-down into the sentiment analysis and work out what their customers are frustrated and angry about – Figure 5 – such that they now have a much clearer insight into what needs to be done to get the scores moving in the right direction. And thus allow the CEO to get a restful night’s sleep.

Figure 5: Frustration Drill-Down Example – US Truck Driver Frustrations