Feedback Loops

Top Three Summer Series - Part 2

Dec 15, 2024

Welcome to the 2024/25 Top Three Summer Series. This is the second of six short posts to be published through December and January, leading up to the launch of my new book How To Be Wrong in early 2025. Enjoy!

Aim … Fire 8th December
Feedback Loops 15th December

If you would like to get these posts delivered directly to your inbox as soon as they are published, please subscribe using the link below…

Last week we started this series by looking at the three components of a plan:

Location
Destination
Route

We talked about the importance of honestly describing our current reality, and of breaking goals down into performance and process components.

This week, let’s focus on the third bit … the route.

You can only see as far as your headlights,
but you can make the whole trip that way.
- E.L. Doctorow

🔗 Activity vs Outcomes

One of the things I like to say to the teams I work with is:

What matters in the end is the work we complete, not the time we spend completing it

In other words - focus on the output not the inputs.

Every visitor to Sydney marvels at the Opera House. All these years later, not many people remember how painful the construction was.1

On the other hand, it’s easy to measure hours worked and very difficult to objectively measure the quality of work produced.

Every manager needs to manage the trade-off between good, fast and cheap.

So how do we resolve that paradox?

One of the tricks I’ve learned is to first just measure activity. This might sound counterintuitive, if we’re focused on outcomes. But it’s surprising how often going through the process of describing exactly what specific things we need to do and then just asking if we did those things highlights that we either (a) don’t know exactly what we need to do; or (b) do know what we need to do, but still don’t do it.

For example, let’s imagine we’re responsible for a currently unreliable software system. We might say the outcome we want is “improved reliability” and then specifically define that based on number of outages or uptime. What activities do we need to complete in order to achieve those goals? We might decide to, say, put real-time monitoring in place for all critical systems, setup an on-call support roster so there is always somebody available to deal with issues that occur out of office hours, conduct debriefs of all material outages to identify the root cause and propose fixes, etc. Then we can simply start by asking: did we do those things?2

If we’re not completing the activities we believe will lead to the outcomes we want, then we’re unlikely to achieve those outcomes. Those activities are a pre-condition of success.

Of course, it’s also possible that we do complete all of those activities and still don’t achieve the outcome. That’s the risk we take whenever we plan. But the only way we get to discover that is by doing the work.

What we’re really talking about here is tightening the feedback loops - so we don’t wait until the end of the project and then ask if the route we took was correct or not, we have some much more immediate milestones we can use to measure our progress.

Next time somebody tells you to focus on the outcomes, use that as an opportunity to discuss what activities you each believe will lead to those outcomes. You might be surprised how often that is a revealing discussion.

🔄 Iterate

Nobody ever remembers the first version of a product. In fact, the only time we ever go back and look at the first version of a product is when it has proved, over time, to be very successful. All successful products evolve, rather than emerging fully formed and perfect. It’s always interesting to compare the early rough and, with the benefit of hindsight, slightly embarrassing first version.

For example, this is what Trade Me looked like when I started working on it in early 2000. There was quite a bit of work to do!

To quote Arthur van Hoff:

"Over the years, I've learned that the first idea you have is irrelevant. It's just a catalyst for you to get started. Then you figure out what's wrong with it and you go through phases of denial, panic, regret. And then you finally have a better idea and the second idea is always the important one."

This approach is fine when we’re working on software. But how can we apply it when the things we’re making are more solid and difficult to change or where the investment required is more binary. For example, what is the “minimum viable product” for new roads or public transport infrastructure?

I recall this in-the-weeds post from Simon Lyall:

A minimal viable light rail for Auckland

This was July 2022, shortly after the then-Government had announced their intention to spend $15 billion building a light-rail line to Auckland airport.3

By comparison, his suggestion might have seemed unambitious and incremental. He proposed starting much smaller - building just over 3km initially, with no tunnels or other complications, but designed with future expansion in mind. However, compared to the nothing that was actually built before the project was cancelled it sounds like an amazing base that could have been built upon in subsequent iterations.

It’s a common mistake to assume that we have one shot for glory, and so need to make our plans as big and ambitious as possible. But the alternative approach of planning our route in small more easily completed steps might actually get us much further in the end.

Trust your future self to handle future problems. The version of you that will deal with difficult futures will have more life experience than you do now.

Divide and conquer.

🧭 Roughly Right

One of the most dangerous things to do when planning a route is to prioritise speed over direction. We need to do the opposite: after all, if we’re going the wrong way then our speed is irrelevant.

We’ve gotten used to map apps giving us precise estimated arrival times. They achieve this by building detailed network diagrams of all of the possible routes, and weaving in real-time traffic data (every time you ask a map for directions you are both getting advice from the app and providing data to the app to inform other users).

But when we’re planning any non-trivial work it’s unlikely we’ll be able to be as complete in understanding the possible routes. We can’t just plug in a destination and then choose the fastest direction. The first few steps are much more likely a leap into the unknown.

This is why having regular feedback loops is so critical. They help us validate that we're still headed in roughly the right direction, rather than precisely the wrong one.4 Small steps in approximately the right direction, validated by customer feedback, are far more valuable than perfectly planned giant leaps. The goal isn't to avoid being wrong - it's to discover quickly when we are wrong and adjust accordingly.

This is counter-intuitive. But trying to be precisely right about an uncertain future is impossible. Being roughly right, and maintaining the ability to course-correct based on feedback, is much more likely to get us there.

A useful feedback loop is:

Short - so we can adjust immediately (the further we’ve gone in the wrong direction before we get this feedback the more difficult and expensive it will be to course correct)
Focussed - ideally shining a spotlight on things we can influence directly
Relevant - linked to our ultimate destination

For example, imagine writing a book. There are two options: wait until each chapter is perfect before showing it to anybody; or share work-in-progress with a small group of people whose feedback you respect and ask them to help you make it better. Doing the latter quickly highlights positives (the stories that are - literally - remarkable) and negatives (concepts that need clarification or sections where the narrative has lost momentum). To narrow in, ask specific questions. I’ve learned it’s useful to give context when asking for feedback, so people reading know if they are seeing an early draft, where thoughts on the structure and flow are useful, or a more complete piece of writing, where highlighting small details like spelling mistakes are useful.

It’s always uncomfortable sharing incomplete work. But short, focussed and relevant feedback loops make the final product much better. The opposite is ultimately much more uncomfortable.

The goal isn't to make perfect predictions about the future, but rather to build systems that help us learn and adapt more quickly.

When planning a route, feedback loops are the key.

How To Be Wrong will be published in early 2025. Pre-sales are available at https://howtobewrongbook.com.

If you know somebody who would find these posts useful please share it with them using the link below…

The Sydney Opera House was designed by Jørn Utzon, who in 1957 won a competition to select an architect for the project. In April 1962 it was estimated the build would be completed within three years at the latest. It actually took 11 more years - construction was finally completed in 1973. The total cost was A$102 million, over 1,300% more than the original budget. Utzon quit the project in 1966.

Hat tip to Ben Gracewood who planted this idea in my mind when he shared this link: Making policy for a low-trust world

That announcement itself came nearly five years after this was first promised, in the heat of an election campaign.

We can think of right + wrong vs rough/crude + precise as two dimensions of a quadrant diagram. We think we need to be precise and right, and that is ideal. But we also need to consider the cost and effort of precision. Better to be roughly correct than precisely wrong.