The age-old debate in product management. How do you measure the success of your widget? Is it a measure of how many widgets you can produce? Or is it a measure of how many widgets consumers use? That can be a complex question even though it can be stated quite simply as “Do you measure success in terms of output, or outcomes?”
To understand the crux of the question, it’s important to define terms. I’m a huge fan of clarity in language. Output is the widget, the thing(s) your team delivers. It could be a physical object, a service, or a feature, the point remains the same regardless. The measure of outcomes is about how the widget impacted the user. It’s represented by a change in user behavior that is a result of the widget you made.
This begs the question, if outcomes are important, don’t you have to start with output? The simple and complex answer is sort of. You can produce all the widgets in the world but if it’s not something your consumers want to buy or use, then the output doesn’t mean the product was successful, no matter how efficiently you produced it.
Output
The measure of output ends at the end. It measures how many, and how often delivery occurs. I want to be clear that I am not saying that measuring output is not important, it most definitely has value when you are looking at operational efficiency and cost of production. It does not consider the value delivered to the end user though and that means it’s missing half the equation.
Outcomes
This measure begins at the beginning and ends after the sale and delivery. Measuring for outcomes puts the user experience in the drivers seat, the basis for user-centered design. It also means that the value proposition’s success is evaluated across the life cycle. It starts with ensuring that you understand the problem to be solved. It means your value proposition should be testable. Testing is not an afterthought; it is an integral part of the process. It takes tough decisions, even on products you think are amazing. The data may show that users are no longer aligned with your value proposition, and you can’t fall prey to the sunk cost fallacy. You must be prepared to walk away, pivoting to the next idea. The testing doesn’t stop at the point of sale, it continues through the end user experience and how that drives change and impact.
How do you tell what you are measuring?
Output is measured by activity or delivery. How many tasks were completed? How many features were delivered? How often are products delivered?
Outcomes are measured by impact, most frequently by looking at product usage, product adoption, and user retention.
Why is this important?
According to the 2019 Feature Adoption Report produced by Pendo, 56% of features are rarely used and 24% are never used. Pendo also found that 12% of features generated 80% of the daily usage volume for the average software product.[1] These are startling numbers though certainly not surprising.
Outlook
The good news is that in ProductPlan’s 2024 State of Product Management Report, they found that outcome related measures, like revenue, retention, and usage are being used as primary measures of organizational success.[2]
A move toward outcome focused metrics starts with defining the right problem that users are looking to have solved, then testing your value proposition. That testing doesn’t happen once but should be completed regularly during the development of the product to ensure the solution continues to align with what users want. It also means being prepared to pivot to a new idea or a new solution if the test results indicate the current solution is off track. In the past, being able to measure outcomes was challenging. Using a framework like Alex Cowan’s Hypothesis-Driven Development, testing is at the forefront of each stage. The result is that measurement is built in, questions and thresholds are established early, and testing occurs throughout the product development lifecycle.
Ultimately, the user should be at the center of our entire process to give companies the greatest opportunity for success. Otherwise, it is highly likely that any success is a matter of happenstance and not repeatable.
What do you think? How does your company measure success?
[1] (Pendo, 2019)
https://www.pendo.io/resources/the-2019-feature-adoption-report/?utm_source=chatgpt.com
[2] (ProductPlan, 2024)
My Life in Practice 
Leave a comment