Will my innovation be a success? – 3 Methods for evaluating innovations
3 Methods for evaluating the chances of success of innovation ideas
Innovations are one of the key drivers for the growth and survival of companies. However, as various studies show, there is a very high probability that an innovation idea will not fulfill the goals associated with it. For example, between 70 and 90% of all newly launched products fail. Almost 2/3 of all startup investments lose money. According to a recent study by CB Insights, one of the main reasons for the failure of startups is that there was no sufficient market need for the developed solution. In view of these figures, the following will first show why the classic approach of creating a business plan is not sufficient or can even be a waste of time and resources. Subsequently, the Bayes theorem, innovation options and Monte Carlo simulations are presented as three powerful but simple methods for evaluating the chances of success of innovations in the early development phases.
Index
Why classic business plans are not enough for innovations
The starting point for many innovation projects is the creation of a business plan. In this plan, the concept of an innovation idea is presented in written form. The business plan not only includes a description of the idea and the essential framework conditions (e.g., market and competition), but also contains a corresponding financial plan. The aim of a business plan is to provide a realistic picture of the prospects of success of an innovation idea in order to obtain sufficient resources – in particular funding and personnel – for its realization.
Particularly for innovations that are characterized by a high degree of novelty and a problem that is not precisely defined, however, classic business plans are not suitable, especially at the beginning of the project, for the following reasons:
- Since the development of innovations is directed into the future, the innovation process is characterized by uncertainty. On the one hand, uncertainty results from the fact that events in the future do not necessarily follow the course of past events. Second, knowledge about the future is always incomplete [1]. Consequently, the essential problem of many business plans is that – despite all the accuracy and time and resources spent on their preparation – they do not sufficiently take into account the uncertainty of an innovation idea. Rather, traditional business plans attempt to manage uncertainty and risk of innovations by planning as precisely as possible. It is true that the creation of worst-case and best-case scenarios often attempts to map the uncertainty and the associated risk. However, this usually leaves open how probable the individual scenarios are. Thus, business plans also do not show the probability that the innovation idea will be a success.
- Another problem with traditional business plans is that they are not testable [2]. It is not possible to tell at the outset whether an innovation idea is a good or a bad one. Therefore, it is first necessary to reveal the key influencing factors and assumptions that are crucial for the idea to be successful. Then, the key assumptions are tested through experiments. Traditional business plans evade such testing because they typically focus on lagging indicators that map the outcome of a previously performed activity. However, commonly used lagging indicators such as sales or contribution margin can only be tested once the innovation idea has already been introduced to the market. However, since a large part of the necessary investments and costs have already been incurred by this time, it is too late to review the innovation idea at this point.
- Finally, business plans are usually static, i.e., they are not usually adjusted after they have been created and approved. A business plan thus reflects the state of knowledge in the initial phase of an innovation idea, when the degree of uncertainty is greatest. By contrast, new information and findings, e.g. from experiments conducted, are not incorporated into the business plan. During the further phases of the innovation project, the business plan therefore does not provide any updated information about the probability of success of the idea. In addition, there is a risk that cognitive biases based on approved business plans – in particular confirmation bias and false conclusions from sunk, irreversible costs (sunk cost fallacy) – limit the willingness to implement corrective measures after failed experiments.
Against this background, three methods are presented below – Bayes’ theorem, innovation options and Monte Carlo simulations – which provide information about the chances of success of an innovation idea in early innovation phases and explicitly take uncertainty into account. In addition, these methods are also suitable for tracking and documenting the changed information statuses in the further course of the innovation project.
Probabilities of success based on Bayes’ theorem
Bayes’ theorem – a classic in mathematics education – is a calculation rule from probability theory that is used to calculate the (conditional) probability of an event. This means that based on prior knowledge about conditions related to the event, the Bayes formula can be used to determine the probability of the event itself. At its core, this involves updating a probability estimate in light of new data. For example, prior to the start of a soccer game, the probability that Team A will win the game may be estimated as 50%. However, if Team A is leading by three goals two minutes before the end of the game, hardly anyone will bet against Team A winning because the probability of victory is assessed at almost 100%.
In his YouTube series “The Math of Success” Savoia illustrates the application of Bayes’ theorem for determining the probability of success of innovations. The starting point is the question of how likely the success of an innovation idea is based on the data from his own previously conducted experiments. An innovation is a success if the actual results are at least equal to the expected results. The expected results are formulated by Savoia in terms of the so-called XYZ hypothesis: “At least X% of target market Y will Z”, where Z describes how this percentage of the target market will react to the innovation idea [3].
To calculate the probability of success based on available data P(S|D), information about the following four probabilities is required. To keep their determination simple, Savoia uses for each of these four probabilities five possibilities of occurrence from “Very unlikely” to “Very likely”, which he evaluates with percentage values from 10% to 90%:
- P(S): Probability that the innovation will be a success (S) before data from an experiment are available.
- P(F): probability that the innovation will fail (F) before data from an experiment are available.
- P(D|S): probability of data (D) from an experiment if the XYZ hypothesis is correct.
- P(D|F): Probability for data (D) from an experiment if the XYZ hypothesis is false.
Finally, based on these four probabilities, the following formula can be used to calculate the probability of success of the innovation idea:
Calculating the probability of success of the innovation idea
P(S|D)=(P(S)*P(D|S)) / (P(S)*P(D|S)+(P(F)*P(D|F))
Particularly in the early development phases of innovation ideas, the special value of Bayes’ theorem lies not only in its simplicity, but also in the fact that after each experiment conducted, the learning progress can be evaluated by an updated calculation of the probability of success. The aim here is to achieve a predefined minimum value for the probability of success by conducting experiments, above which there is sufficient certainty that the innovation can be a success. Only once this threshold value has been reached are larger investments in the innovation project approved.
Calculation of return on investment with innovation options
The innovation options method is based on the approach of viewing innovations as real options. A real option is an initial investment that grants the right, but not the obligation, to make a subsequent investment when additional information is available. Similar to the purchase of a financial option, the real option serves to limit the loss in cases of high uncertainty. For this purpose, investments are staggered in such a way that under poor conditions no further investments are made and thus no further losses are incurred. In contrast, under favorable conditions, further investments can be made [4].
The starting point of innovation options is the question of how the economic benefit of an innovation idea can be determined in order to obtain the budget required to implement the idea. In the face of criticism of widely used valuation methods (such as net present value) that fail under conditions of high uncertainty, Binetti developed the innovation options method [5].
The basis of innovation options is the trinomial tree developed by Boyle, a grid-based computational model. Here, each node in the lattice represents a possible value of the innovation option at a particular point in time. At each node, three possible paths exist: an upward path, a downward path, and a middle path. In the case of innovations, the paths represent possible learning progress or the results of experiments. They are the three possible answers to the question, “Given what we have learned since the last evaluation point, are we in a better, worse, or about the same position as before?” The initial node of the grid thus represents the initial value of the innovation option at time zero, i.e., the profitability of the innovation idea without any additional information or implementation actions. The budget for the initial implementation steps of the idea must therefore not be larger than this number.
In addition to the detailed description of the procedure for calculating innovation options, Binetti provides a calculation tool. Therefore, we will not go into further mathematical details at this point. With this tool, the calculation of an innovation option requires only the following information:
- The sales potential of the innovation idea in the best-case scenario,
- The estimated budget to bring the innovation to market, and
- The terms of the option, i.e., the term and the number of iterations during the term.
Innovation options focus on evaluating the uncertainty associated with innovation, including the possibility that the innovation idea will fail. Instead of immediately releasing the entire budget for the development of an innovation idea, innovation options encourage a step-by-step approach. Investments are only made in a next step if there is a possibility of a positive return on investment. If, on the other hand, none of the three paths leads to a positive value at a certain point in time, it makes no sense to continue investing in the implementation of the idea. Instead, the project should be terminated.
Financial models with Monte-Carlo-Simulation
Monte Carlo simulation is a simulation method based on random numbers, with which the possible results of uncertain events or influencing factors are estimated. The core is a very large number of similar random experiments. In each experiment random numbers are generated for the uncertain factors and the resulting results are calculated. For the generation of the random numbers, probability distributions are used, e.g. a uniform or normal distribution. Due to their flexibility, Monte Carlo simulations are suitable for solving complex problems in a very wide range of applications.
Despite its advantages and its widespread use in other areas, Monte Carlo simulation has so far been used comparatively little for evaluating the chances of success of innovations. For example, Euchner and Ganguly illustrate how tire manufacturer Goodyear uses Monte Carlo simulations to identify and prioritize risks of innovation concepts as part of its business model innovation process [6]. Kromer and Eilert apply Monte Carlo simulations based on McClure’s Pirate Metrics funnel to simulate the growth and financial performance of startups [7].
In both cases, a hypothesis-based financial model is built at an early stage of innovation development. However, in order to integrate the uncertainty of the influencing factors into the model, ranges are defined instead of individual values – as is the case in the business plan. This means that a lower and an upper value are determined for which there is sufficient certainty that the real value will lie in this range. A very large number of simulations are then performed, with a new random value generated in each simulation based on the defined range. Since an outcome is calculated for each individual simulation, the result of the Monte Carlo simulations is a frequency distribution of potential outcomes. This makes it possible to calculate the probability of each outcome and thus answer, for example, the question “How likely is it to make a profit of at least X euros?” In addition, sensitivity analyses can be performed to identify those factors that have the greatest influence on the outcome. The results of the Monte Carlo simulation provide the basis for subsequently testing the most important assumptions through experiments and reducing the associated uncertainty.
Although creating Monte Carlo simulation may sound complicated and time-consuming at first, initial models can be created with relatively little effort in common spreadsheets (e.g., Microsoft Excel or Google Sheets). Admittedly, a lot of information is still missing at this early stage of innovation development and many or even all assumptions may be wrong. Nevertheless, creating the model at this stage is very valuable. For one thing, it helps to make transparent and obvious the hidden assumptions that are critical to innovation success. On the other hand, creating the model requires a clear thinking through of the planned business model and a first quantification of the assumptions with estimated ranges.
At the same time, it is important at this stage not to spend too much effort on creating a model that is as realistic as possible. A simple model for mapping the essential mechanisms of the business model is sufficient in the beginning. Only as the level of information increases and the uncertainty of the innovation idea is progressively reduced should the model be expanded.
Conclusion on the evaluation of the success of ideas
Especially at the beginning of an innovation project, it is characterized by a high degree of uncertainty. For this reason, it makes little sense to invest a lot of effort in the creation of a business plan. This is all the more true because the business plan does not answer the question of how likely it is that the innovation idea will be a success.
To answer this question, the Bayes theorem, innovation options and Monte Carlo simulations were presented as three methods, each of which has a different focus and can be used individually or in combination. While Bayes’ theorem focuses on determining the probability of achieving a defined goal, innovation options focus on determining the return on investment. Monte Carlo simulations offer the greatest flexibility, as they can be used to determine both financial target variables and their probabilities, as well as the key influencing factors. Despite these different focuses, the three methods have in common that they:
- allow the assessment of the chances of success of innovation ideas with little effort,
- enable the quantification of information statuses that improve in the further course of the project, and
- form a basis for go/no-go decisions regarding further investments in the innovation project.
References
[1] Harri Jalonen, The Uncertainty of Innovation: A Systematic Review of the Literature, Journal of Management Research, Vol. 4, No. 1, 2012.
[2] Tristan Kromer / Elijah Eilert, Innovation Accounting: The Failure of The Business Case, https://thefutureshapers.com/innovation-accounting-the-failure-of-the-business-case, 31.08.2021.
[3] Alberto Savoia, The Right It: Why So Many Ideas Fail and How to Make Sure Yours Succeed, 2019.
[4] Rita McGrath, Falling Forward: Real Options Reasoning and Entrepreneurial Failure, in: Academy of Management Review, Vol. 24, No. 1, 1999.
[5] David Binetti, How to Calculate an Innovation Option, https://blog.innovation-options.com/how-to-calculate-an-innovation-option-d673369e8658, 8.12.2017 und David Binetti, Measuring Learning in Dollars, https://blog.innovation-options.com/innovation-options-a-framework-for-evaluating-innovation-in-larger-organizations-968bd43f59f6, 23.04.2015.
[6] Jim Euchner / Abhijit Ganguly, Business Model Innovation in Practice, in: Research-Technology Management, November-December 2014.
[7] Tristan Kromer / Elijah Eilert, Innovation Accounting in Practice, https://innovationmetrics.co/innovation-accounting-in-practice/.
Comments are closed.