26th September, 2021
Week 39/52
Photo by engin akyurt on Unsplash
As you know, one of my favourite questions to ask founders or anybody promoting initiatives in the ecosystem is:
How will you know it's working?
It's important we try to answer in advance, otherwise it's too easy to just shoot an arrow and then draw the bullseye around whatever we hit. But there are always a long list of different metrics to consider and it’s common to feel overwhelmed.
So today, let’s consider what we measure, when we measure it and who we measure it for…
🤹 Juggle
When learning to juggle it’s easy to assume that the important thing is catching. But actually the key is throwing.1
This isn’t intuitive but makes sense when we think about it: if we can learn to accurately launch the balls (or flaming torches, bowling pins or chickens) we’re trying to juggle so that they land where we can catch them effortlessly - e.g. without needing us to lunge and without distracting our attention - then catching takes care of itself.
There are four different things we can try to measure:
We can measure inputs. Those are the things we start with.
We can measure activities. Those are the things we do.
We can measure outputs. Those are the immediate results.
We can measure outcomes. This is the longer term impact or lasting change.
Once we understand these layers, the temptation is always to go later: don’t just measure inputs measure outputs; don’t just measure outputs measure outcomes; don’t just measure outcomes measure sustained outcomes or systems of change etc.
There are two big problems with this:
When we reach for outcomes it can be hard to isolate the specific things we do from the results we get, because there are so many other factors that can influence those results over time. It requires us to show both that something happened, and that it happened because of the things we did.
Also, possibly even more importantly, we have to wait much longer before we get any answer. Sometimes it can be years or decades before outcomes are obvious. In the meantime there is nothing to give confidence that things are on track. Without the feedback loop created by scrutiny and consequences we are unlikely to meet our expectations.2
For something that is well established and has been done previously, it's absolutely correct to ask about the outcomes rather than just measuring inputs or activity. But when we think about measuring something brand new and (as yet) unproven that doesn't work. There is too much uncertainty.
So maybe we need to be inspired by novice jugglers?
That means clearly identifying the important activity that we believe will lead to the outcomes we want over time, if we do a good job. And then, expand on exactly what we mean by "do a good job" (e.g. throwing so that the ball lands where we can catch it easily). That gives us something specific to start measuring immediately.
This approach forces us to articulate up-front what we think is hard and what "good" means; it creates a much shorter and easily measured feedback loop, so we can track our progress and improvement on a scale over time (which is much better than a binary success/fail measure); it eliminates more external variables; and it explicitly acknowledges the leap of faith we're taking.3
Remember, we use evidence to identify problems but we need to use experiments to solve problems. And, if we already know it's going to work it's not an experiment
#️⃣ Intersect
You can only make as well as you can measure
—Joseph Whitworth
There are so many different ways to measure a start-up. It's easy to drown in metrics. How do we know what we should focus on?
Here are two different ways we can divide and conquer this problem...
Three Machines
Consider the different people working on the team and the responsibilities they have.
There are broadly three specialities:4
Product - those responsible for developing, maintaining and operating the product or service;
Customer - those responsible for marketing, selling, on-boarding and supporting customers to use the product or service; and
Company - those responsible for recruiting, retaining, running and (not least) funding the team that makes everything else possible.
Three Phases
Or, consider the progression of a start-up from early-stage to successful mature business.
Again, there are broadly three stages:5
Build - in the very beginning we need to focus on developing a product that solves a real and valuable problem and attract some initial customers.
Grow - then, our focus shifts to overcoming obscurity and everything we need to do to promote and sell the product to more and more customers.
Profit - finally, our focus shifts to doing all of those things efficiently and ensuring that the amount we spend on each customer is relative to the amount we earn.
Nine Intersections
We can use these two groupings to create a 3x3 matrix, and use that to help us think about the important measures of success for each team at each stage.
What's at the intersections?
We can work down each column in this framework to consider the progression for each area of the business...
Product Team
The first job is to develop a product or service that solves a problem. Perhaps we stumble across that immediately, but more likely we need to work hard while customer numbers are small to understand if people are actually using what we're building in the way we anticipated, and adjust accordingly.6 As the number of customers grows we can start to segment, and understand what is attracting our best customers and also what are the gaps for those customers we don’t retain.7 Later, as we get beyond early adopters we need to pay more attention to things like customer churn and the cost of servicing each customer, so we can invest in the right areas.
We progress from measuring active users to measuring delighted users to finally measuring loyal users.8
Customer Team
In the early-stages we're looking for initial customers who are willing to use the product right away, even though it's only half built. We need to understand how customers move through our sales funnel - e.g. using free trials or a land-and-expand approach. To move beyond one-by-one sales we need to experiment with the different sales channels that are available and understand the costs associated with each. We need to learn what generates growth - perhaps existing customers spread the word for us, but more likely we need to buy it. Maybe there are also opportunities to increase the revenue we earn from existing customers. Later, with the benefit of more time, we can track the lifetime value of different customer cohorts and use that detail to tune our sales process.
We progress from measuring sales conversions to measuring sales margins to finally measuring sales efficiency.
Company Team
At the beginning cash is typically the most critical constraint. Either we need to ensure that the amount we're spending is relative to the initial revenue we're generating, or we need to raise external capital to fund the difference through the early-stages. As the team grows we need to focus on team culture (in simple terms, the things we do repeatedly) and make sure we're attracting the right kind of people to fill the different roles we have and keep them engaged and happy. Eventually we also need to consider how those who have invested time and money in creating the business will get a return.
We progress from measuring cashflow to measuring team engagement to measuring shareholder returns.
Walk → Run → Fly
We can also work across each row in this framework to quickly identify the basket of metrics that are likely to be most important at each stage of the business...
In the early-stage we measure active users, our sales funnel and cashflows.
As growth accelerates we measure net promoters, net new revenue and cost-to-acquire, plus team engagement.
And, as the business matures we measure customer retention and churn, lifetime value (or perhaps more usefully average contract value) and return on investment.
This is an embarrassingly simplistic framework, but hopefully points in a useful direction. I say this every week, but mean it even more this time: I'd love your feedback on this - what's missing, what's wrong, what's useful? Let me know!
Related: Metrics Maturity Model
🧑⚖️ Judge
Trying to manage without metrics is like flying blind.
But equally managing entirely by metrics is misguided.
It's common to hear people claim they are led by data. They say they do what the data tells them to do. But it's never as simple as that in my experience. Data can answer our questions, but we still need to ask them.
It's also common to manage using incentive schemes based on measurable results. But, whenever we do that we leave ourselves open to people gaming the system or chasing selfish or short-term results at the expense of long-term or collective outcomes. So every measure needs a counter-measure to ensure that the resulting benefits aren't weighed down by greater associated costs.
Often the biggest and most important decisions we need to make come down to subjective judgement. In those moments we should make sure we have all of the data we can get our hands on to inform our decision. But let's not pretend that the numbers will make difficult decisions for us.
Why It’s Almost Impossible to Juggle 15 Balls, by WIRED
Top Three is a weekly collection of things I notice in 2021. I’m writing it for myself, and will include a lot of half-formed work-in-progress, but please feel free to follow along and share it if it’s interesting to you.
See Ceri Evans' definitions from Perform Under Pressure:
Expectations = what standard have we set ourselves?
Scrutiny = how are we going to know if we have achieved those standards?
Consequences = what happens if we do/don’t achieve those standards?
Of course, there is always a risk is that we choose the wrong thing. But at least this way we have something concrete to be wrong about. And, as long as we acknowledge that when it happens, we can learn from being wrong and try to be less wrong next time.
See: The Three Machines by Brad Feld.
See: Lean Analytics by Alistair Croll & Benjamin Yoskovitz, specifically chapter 5 where these three phases are described as:
Product Development
Customer Growth
Unit Economics
They are themselves expanding on the stages of a start-up that Eric Ries described in his book The Lean Startup: make something that is sticky, then make something that is viral and then finally make something that generates revenue.
A common trap at this stage is to be overly influenced by what potential customers say they will do, only to later discover they don't do that at all once we've actually built what they said they wanted.
See: Net Promoter Score.
As per Jimmy Guterman,
“There are only two industries that refer to their customers as users: high tech and illegal drugs”.