OVO Tech Blog

Experiment-driven product development

Introduction

Matteo Di Vito

Matteo Di Vito


ux

Experiment-driven product development

Posted by Matteo Di Vito on .
Featured

ux

Experiment-driven product development

Posted by Matteo Di Vito on .
Shoe with superpowers

“We value responding to change
over following a plan”

If you create digital products, these words should sound familiar. They’re from the Agile Manifesto and represent a core principle of the philosophy.

In the world of digital product development, we sometimes forget the benefits of applying an agile methodology to our workflow isn’t about velocity or speed of delivery. The original intent was to tackle a much bigger issue: uncertainty.

Who are the people we're building for? What do they really need? Is our solution right for them? Will we produce something valuable for them and our business? These questions haunt us daily, and one way to reduce uncertainty is to form a plan.

Plans are great. They give you a sense of security, and something your team can agree on and rally around. But they can only describe a vision of the future: a theory, a prediction of how things might be. Circumstances change, often unpredictably, and our knowledge is always limited. Plans are like grand ideologies: they try to explain every detail of the problem, but they’re based on ideas rather than facts. And they’re always disrupted by the next grand thinker.

A plan is often the best method we have to get something done. In industrial design, for example, you have to commit to forming a plan because once you send your design to the factory you’re committed to it. If the design needs to change, the machines will need to start again to produce expensive new iterations of your product.

Digital products have superpowers

A similar approach was traditionally applied in software development, where teams would attempt to design a perfect solution in advance, so engineers could build and release it in one go. We called it waterfall because once something was complete, it was hard to return to an earlier stage and change things. The cascade only moves forward.

But digital products are special; they have a built-in superpower.

Let’s imagine a digital product as a pair of shoes for a moment… but they’re different from normal shoes: every time they’re used, the team which created them receives information about how they’re being used. Also, this team can update how the shoes look and work as many times as they like at very little expense.

When someone buys these shoes, they find them okay but not the most comfortable, and the colour isn’t their favourite. The product team behind the shoes are aware of the discomfort and the displeasure. So they start thinking about ways to make the shoes more comfortable and come up with a tweaked design. The team pushes a button—and voila! the shoes are suddenly a bit more comfortable, and a bit better looking.

This is how digital products work. The window we have on how people use our products, and our ability to react to what we learn about their needs is what makes digital product development different from creating physical products.

This method is a humble one. It accepts we don’t have all the answers when we start working on an idea. We need to accumulate knowledge from the information we collect by building and testing that idea. A bit like how scientists develop an experiment...

The scientific method

Science also makes progress through testing and observation.

Scientists begin with a prediction—an educated guess about the way something works. They design experiments to create conditions where a specific outcome can only be observed if their original assumptions are true; they draw conclusions from whatever it is they observe. And whether the experiment succeeds or fails they always learn something new, which informs a new hypothesis and the cycle starts again.

This is simplified of course, but does it sound familiar…?

Scientific world and product world

Applying the scientific method to digital product development

If we continuously improve the comfort and attractiveness of our special shoes, we know people will be happy with them, hopefully using them more and recommending them to others. If these are the outcomes we desire, we need to look for signals that show our changes are having the effect we want.

As a product development team we need to constantly observe and respond to change. The shoe’s owner may want to use them differently from how the team imagined, perhaps only in certain weather or seasons for example.

We call these attempts at improvement experiments.

Each experiment starts with an hypothesis — an assumption about how to improve our shoes. It’s not enough to follow customer’s comments alone: they might tell us the shoes hurt in one spot, but they might not know why or how to fix it. It’s our job to figure it out.

So the team changes the design and makes a small update to increase the comfort. Then they wait and watch for signals: has the discomfort gone? are they wearing the shoes for longer or more frequently?

If nothing changes, or the signals are negative, then it wasn’t the right solution. Our assumption was wrong and we need to explore other ideas instead. If there’s positive signals, we’re on the right path! If we continue with this approach, we’ll hopefully arrive at a shoe that fits the wearer perfectly, and is highly desirable. Job done!


Some lessons I’ve learned...


Our shoe analogy makes sense and is easy to apply — for the one customer in our example. In reality, we serve many people at the same time, all with different contexts and needs.

Here’s some principles which have worked for our team:

  • You can’t improve a digital product if you don’t know what you’re trying to achieve first. Define the problems to solve, set your desired outcomes, and build around them. You may be listening to the people you build for, but without clear goals, you can’t connect that feedback to your outcomes. Make a straightforward plan to get you started.
  • Don’t aim to create things quickly only. If we build a poor solution fast, we may end up making things worse than they were before.
  • Small changes are better than big ones. Would you be happy if your shoes dramatically changed shape, fit, and colour every few months? Slow, steady, incremental changes are easier to adapt to.
  • Look for things that people do which help you achieve your team's goal. For example, did you observe that people that read a blog post are more likely to sign up? Try to enforce that positive behaviour by creating a more powerful connection between what triggered the behaviour and the goal. In our case, it could be creating a direct link between the blog and the sign up flow, or facilitate access to blog articles to newcomers. These would form the basis of the hypotheses, the first building block for your experiments.
  • When we hear the word ‘experiments’, many people think about A/B tests. This is a valid way to conduct an experiment, but only one of many methods to detect behaviour changes. There’s no rule of thumb for detecting positive signals, so your team should experiment with how you conduct experiments!

Final thoughts

Final thoughts

A good plan is always useful at the beginning, especially when there’s lots of uncertainty. But our plans should evolve as we learn more about what we’re trying to achieve.

Product development, whether physical or digital, is never finished. We might decide to stop improving it, but that doesn’t mean it couldn’t be improved. Products are shaped around behaviours; behaviours come from people; and people are always changing. If we can detect those changes, we can continuously shape our products to fit new behaviours.

Further reading & watching

📖 Outcomes Over Output: Why customer behavior is the key metric for business success
By Joshua Seiden

🎤 Keynote - Impact Mapping
By Gojko Adzic

📖 Running Lean: Iterate from Plan A to a Plan That Works
By Ash Maurya

📜 Lean Product Management Manifesto:
By Melissa Perri

Matteo Di Vito

Matteo Di Vito

http://matteodivito.com

View Comments...