Editor | On 05, Jan 2020

Darrell Mann

“When we try to pick out anything by itself we find that it is bound fast by a thousand invisible cords that cannot be broken, to everything in the universe.”
John Muir

“To demand or preach mechanical precision, even in principle, in a field incapable of it is to be blind and to mislead others.”
Isaiah Berlin

Here’s a problem we’ve been grappling with for years now. How to measure the Return On Investment of innovation activities? It’s a challenge we’ve been given several times by clients, and its one we’ve heart-ached a lot ourselves in relation to the impact we are having with our clients. Eagle-eyed viewers of the SI website may have noticed that we’ve finally, in the past few months, become brave enough to start publishing our cumulative impact since we started back in 1998: how much top-line new revenue we’ve helped clients achieve; how much bottom-line savings we’ve been involved in, and how much CO2 have we helped clients save. The top-line number is over $5B now, which means that, given our small size, any traditional measure of ROI (i.e. some form of ratio of benefits delivered over costs) becomes near infinite. Which in turn tends to look ridiculous. Which then causes prospective new clients to be doubtful about our abilities.

The bigger innovation ROI problem, however, is one of complexity. Any and every innovation project fundamentally finds itself in the middle of a maelstrom of complex issues. This in turn creates a whole series of problems regarding what costs and what benefits can rightly be attributable to any individual innovation project? Which costs should be included? Which ones should be ‘externalised’? No-one, to the best of our knowledge – and especially the Big Five consulting companies, who, I imagine, get asked to make innovation ROI calculation all the time by their clients – has found a meaningful way to create ‘the right way’ to make the calculation. Hence, again based on my experience, the ROI ‘calculation’ typically proceeds something like this:

Client: the powers that be need us to calculate the ROI on this project.
(BFC*): great, I’m sure we can help.
Client: you are a lifesaver.
(BFC): its why we’re here. No biggie. So, I assume you’ve already made your own calculations? What do you think the answer is?
Client: well, depending on how you use the numbers, somewhere between 40 and 45%. Obviously, for tax purposes, we prefer the lower number.
(BFC): (nodding sagely) Sure. Give us the numbers and your calculation and we’ll go away and do our version for you.
(Client): my hero.(BFC disappears for 4 weeks, into which they cram approximately 2 hours of work.)

Next meeting:

(BFC): Phew, that was a tough one.
(Client): What did you come up with?
(BFC): Well, according to us, you were a little bit out with your calculations.
(Client): Oh, no.
(BFC): Mmm. Not to worry. We make the ROI 38.7%.
(Client): Wow. That sounds good. How did you do that?
(BFC): It’s all written up in this 50-page report.
(Client): my hero.

(BFC disappears, sends client a bill for 160 hours work.) …And everyone lives happily ever after.

(* BFC: Big Five Consultant)

Well, except the whole thing has been a fiction of course. A fiction that, in the long run, serves only to perpetuate the enormous level of dysfunction in the innovation world.

In defense of both client and to some extent the BFC, though, the ROI challenge is a tough nut to crack. Here are a few examples of the sorts of challenges that need to be addressed:

1. 1. The innovation team at a large multi-national spend three years creating what rapidly becomes a ‘billion-dollar brand’ new product for the company. Ten years later, the product family continues to generate a billion dollars a year. Is it right that the ROI calculation takes credit for all ten years? For how much longer should the innovation team continue to be accruing ROI credit? For as long as the product stays on the market?
   2. A multi-national energy company is on the verge of signing the approvals for a $300M capital expenditure project, when the innovation team discovers, thanks to a recently completed piece of R&D, that the same outcomes can be delivered for about $300K. The $300M approval is rescinded. How much of the resulting 99% saving is then attributable to the innovation project as ROI?
   3. The innovation team at a large consumer electronics organisation is tasked with reducing the manufacture cost of a particularly challenging, but critical component within the company’s main product range. They come up with a solution that saves the company just short of $100M in the first year. The reduced costs trigger an increase in sales of the product, that corresponds to a further $150M savings in the second year. For how long does the team keep accruing ROI credit as a result of their work?
   4. The innovation team in a Tier 1 white goods sector supplier derives a breakthrough product solution that rapidly comes to cannibalise sales of their existing product, such that within three years, the new product has completely displaced the old. The new product necessitated the building of a completely new production-line costing close to $50M, but on the plus side, it possesses significantly improved performance, however, so as word spreads around the industry, their market share by year three has increased by 10%.
   5. The advertising department for a multi-national confectionery company completes a piece of market research that reveals a significant insight into changing consumer behavior. The insight requires no change to the product, but results in a breakthrough advertising campaign that sees incremental sales increase of 30% in the first year, another 15% in year 2, and another 10% in year three. How to calculate the ROI of this marketing innovation?
   6. The R&D department of a multi-national aerospace company develop a patented, breakthrough new engine nacelle design that saves about 5% fuel burn, reduces weight by around 20%, and reduces noise by around 6dB. For fifteen years, industry ignores the solution, and so the technology sits on the shelf. Sixteen years after the solution has been proven, finally, the market is deemed to be ready for the novel nacelle. It proves to be an enormous success very quickly, generating around $80M in new revenue in the first year of introduction. However, by the end of this year, the patent expires and this allows competitors to launch their own ‘me-too’ version of the design that in effect makes the ‘new’ design into the new ‘standard’ and effectively, therefore, becomes a commodity.
   7. The R&D function of a multi-national chemical company is informed that their budget will be reduced by 40% and that they therefore need to prioritise which projects they will continue and which they will put on hold. In theory, they should continue the projects destined to deliver the highest ROI, but how best to make this kind of predictive calculation?

All the time we examine examples like these, the search is on for ‘the’ simple, first-principles calculation model that fits all cases. To date, I have to say, we’ve not been able to find such a thing. A bit like attempts to find ‘the theory of everything’, what makes the job so difficult is the fact that the world is full of discontinuities, and until such times as it becomes possible to resolve the contradictions that exist between each side of such discontinuities, the best we can hope for is to manage them. What ‘manage’ means in the case of innovation ROI calculations is that we need a different calculation for each side of each different discontinuity we can find.

Before we try and unravel that story, it is always useful to understand our purpose. Why is it important to measure innovation ROI?

When we’ve asked this question of clients, once we get beyond the ‘because my boss told me to’ level answer, we usually find ourselves in a quagmire of confusion. ‘We need to demonstrate our effectiveness.’ ‘We need to understand opportunities for improvement.’ ‘The Senior Leadership Team need the information to know where to invest their budgets.’ These statements might seem clear, but in reality – when we recognize the complex environments in which we are expected to operate – they invariably turn out to be anything but. Reference 1 has much to say on this subject, as well as offering up a host of case studies in which good measurement intentions resulted in some very unhappy outcomes.

From a TRIZ/SI perspective, the only real value of an innovation ROI measurement is the one that enables the vii) scenario from earlier to be addressed in a meaningful way. In other words, the only true value of an ROI calculation is if it offers a meaningful predictive capability. We need leading indicators. Unfortunately, however, the vast majority of ROI calculations are the opposite, in that the provide a lagging indication of how well we did or didn’t do in the past. Ultimately this type of after-the-fact data is only useful as a means of validating or enabling us to improve our future prediction capability. And even then, when we take into account the ‘never stepping in the same river twice’ characteristics of complex systems, it is difficult to connect the metrics of a unique past project to the likely future performance of the unique next project.

That said, it also needs to be remembered that most Senior Leadership Teams have a quite traditional view of ROI calculations. The ‘expect’ lagging indicators because that’s all they’ve ever had. And when they do force R&D managers to project future returns, everyone knows a game of random numbers is about to start, and that when the eventual results are in, no-one is going to attempt to close the loop and adjust the prediction ‘model’ used to make the initial estimates. Now we find ourselves back in BFC territory, and the only thing that’s happened now is we’re going to find ourselves spending even more money in order to produce the same random numbers. Check out Reference 2, for example, to gauge the level of randomness on offer to start-ups and investors.

This might sound like a skeptical view of the ROI measurement industry. Probably because there is good cause to be skeptical.

That skepticism, however, should not be a hindrance to an attempt to do better. And, again applying TRIZ/SI thinking, doing better means knowing our Ideal Final Result destination, and then putting in place measurement methods that are both appropriate to the current level of capability and have the possibility to evolve in the direction of the IFR.

So, what is that Innovation ROI Ideal Final Result?

We think it should be a variant on this definition:

And, in terms of the likely evolution trajectory as a calculation method advances towards this end goal, it seems important to match the prevailing Innovation Capability Maturity of the enterprise making the calculation. In other words, the ROI calculation made by an enterprise at the Level-1 beginnings of their Innovation Capability building journey need a different ROI calculation to a Level 2, 3, 4 or 5 enterprise.

In the second part of this story, we will delve into the specifics of what we think the different ROI calculations look like for the five different ICMM Levels. Before we get there, however, it is worth exploring the key elements that ultimately need to be embraced in order to perform meaningful innovation ROI calculations.

Innovation ROI Is A Function Of…

1. 1. Industry Pulse Rate: the rate of discontinuous shifts (i.e. s-curve jumps) at the system and sub-system level for the technical, process and business elements of an enterprise. In high pulse rate industries like consumer electronics, innovation attempts are required on a constant basis, but the time they have available to generate revenue is likely to be limited before a replacement solution arrives. Conversely, in slow pulse rate industries like Mining, an innovation attempt occurring just after a successful step-change may not have the opportunity to generate any revenue until the next step-change, which may be thirty years away.
   2. Level Of Invention: here we think about the five Levels of Invention used within the TRIZ world. A Level 1 invention is ‘obvious’, while a Level 5 invention is ‘world-changing. The relevance to an ROI calculation is ‘if we didn’t have this idea now, how long would it take for someone to derive this idea?’ This can make for a particularly contentious calculation, especially when thinking about how TRIZ/SI removes so much of the randomness and trial-and-error from the ideation process. TRIZ/SI will get innovation teams to good answers faster than any trial-and-error process, but trial-and-error would have got there eventually. Irrespective of whether TRIZ/SI was used or not, the ‘how much sooner’ question remains. We can safely hypothesize that a Level 1 invention would be discovered by others more quickly than a Level 5 invention. As yet, we have found no way to validate a meaningful way to quantify how much more quickly, but the algorithm we use connects Level of invention to the aforementioned Industry Pulse Rate:Level 1 – saves 0.1xpulse-rate (up to a maximum of 1 year)
      Level 2 – saves 0.2xpulse-rate (up to a maximum of 2 years)
      Level 3 – saves 0.5xpulse rate
      Level 4 – saves 1 pulse rate
      Level 5 – saves 2 pulse rates(so, taking example i) from the earlier scenarios, if the invention that created the billion dollar industry was Level 4 (it was!), and the industry pulse rate was around 3 years (it is), then the innovation ROI calculation should accrue 1×3 years of the mature-level revenue generated)
   3. Patentability & Patent Quality: a patentable solution is ‘worth’ more than one that is not since it prevents competitors from intruding into a market space; a ‘high quality’ patent is also worth more than a ‘weak’ one, since it is more difficult for competitors to design around. Looked at from a different perspective, if we spend time and effort ring-fencing competitor patents so that they are unable to evolve their current solution, this also has a likely ROI value. This parameter is inherently coupled to Industry Pulse Rate and Level of Invention, but also requires inclusion in its own right.
   4. Innovation Project Effort Expended: in theory, one of the simplest parts of the ROI calculation to make – how much time and budget did the team spend in order to achieve the eventual tangible benefits to the enterprise? In practice, however, very easy to evolve into a rats-nest of a calculation taking into account sunk-cost effects, lost-opportunity costs (‘how much better off would we have been if we had conducted Project B instead of Project A?’) and a host of intangible bias effects that kept us from or got us to solutions slower or faster than we anticipated.
   5. Type Of Innovation: whether an innovation attempt represents a sustaining solution that cannibalizes existing business but grows market share, or is a disruption, or an exaptation, or comes from a pioneer versus ‘fast-follower’, a technical or business step-change will all affect the ROI calculation. ‘Business’ innovations tend to be cheaper to execute than ‘technical’ ones, but sooner or later an enterprise that devotes all of its effort to business innovation is going to suffer when the technology they are exploiting is no longer fit for purpose.
   6. CapEx/OpEx Split: the need or otherwise for a large capital expenditure in order to productionise an innovative solution may also need to be taken into account. This will again be a calculation that is influenced by Industry Pulse Rate.
   7. Transition Costs: most innovations will not only require training and education of personnel to do things the ‘new way’, but will also very likely need to take into account the cost of unlearning the old ways of doing things.
   8. Managing The Unknowns: as we’ve described elsewhere (Reference 3), a big part of any innovation project involves the successful management of the inevitable unknowns surrounding a project. Answering each ‘unknown’ has both a cost and a value to a project. Even if a project ultimately ‘fails’, the ‘unknowns’ that have been answered are still very likely to have a positive value in so far as the knowledge gained informs future projects.
   9. Multiplier Effects: Perhaps the trickiest part of any ROI calculation and intimately connected to complex systems theory and particularly aspects relating to Butterfly Effect-like non-linearities in which a tiny intervention in one aspect of a new solution has the ability to create enormous impacts in surrounding domains. The Brexit slogan ‘take back control’ cost little more than a few hours of Dominic Cummings’ time, but the impact on the UK economy could in many ways be argued have been profound. If it weren’t for that fact that some of the more capable of the world’s innovators are increasingly able to build models of the world that enable the synthesis and modelling of Butterfly Effect non-linearities, we, like most accountants, would most likely look to ignore the impact of multiplier effects on ROI calculations. The fact that such models do exist means that – for high ICMM Level enterprises at least – multiplier effects are becoming one of the most important aspects of the innovation story.

More on that story in Part #2, when we describe how Innovation ROI calculations might best be made – which factors should and should not be included – for each of the ICMM Levels.

References

1. Muller, J.Z., ‘The Tyranny Of Metrics’, Princeton University Press, 2018.
2. https://thecorporatestartupbook.com/calculator.
3. Systematic Innovation Ezine, ‘Managing The (Autonomous Vehicle) Unknowns’, Issue 203, February 2019.