Confidence Meter
The Confidence Meter is an estimation tool that helps product managers evaluate the reliability and credibility of proposed improvement ideas based on the strength of the supporting evidence.
Itamar Gilad invented the model while working at Google, “to cut through long debates and battles of opinions”. A visual template is available from Gilad’s web site, and the model is described in detail in his book Evidence Guided (along with examples of usage with the ICE scoring in the GIST framework) .
Levels of confidence
The model introduces a numeric score from 0.01 to 10, intended to be used as a multiplier for other scoring components such as expected value. In effect, this produces a result very where low-confidence value estimates are divided by a 100, very high confidence value estimates are multiplied by a factor of 5-10, changing the factors by several orders of magnitude.
| Evidence | Examples | Confidence Score |
|---|---|---|
| Self-conviction | 0.01 | |
| Pitch-deck | 0.03 | |
| Thematic support | aligned with strategy or trends… | 0.03-0.1 |
| Other’s opinion | experts, investors, press… | 0.1-0.2 |
| Estimates and plans | feasibility evaluation, business case… | 0.4-0.5 |
| Anecdotal evidence | 1-3 interested customers, some competitors have such features… | 0.5-1 |
| Market data | surveys, smoke tests, most competitors have such features… | 1-3 |
| User or customer evidence | lots of product data, interviews with 20+ users, usability study, MVP | 3 |
| Test results | A/B tests, longitudinal user studies … | 4-5 |
| Launch data | information collected from production systems | 10 |
When to use the Confidence Meter
The Confidence Meter encourages teams to prioritize ideas supported by stronger evidence, reducing risks from making decisions based on intuition. It is very useful for quick idea evaluation, from two perspectives. First, it’s fairy quick and simple. Second, because it systematically categorises different levels of confidence, making it possible to compare and evaluate the validity of different ideas. In this way, the Confidence Meter is most commonly used as a component of estimation and scoring techniques for idea prioritisation, such as ICE.
By proposing several typical levels of confidence with examples of evidence at each level, this model also encourages teams to perform further research and shows ways of increasing the confidence score. For example, if someone strongly believes in an idea but it was rejected because of low overall ICE score, the person proposing it can gather evidence expected to reach one of the higher levels in the Confidence Meter model, and propose the idea again.