Experiment Design for Sloths

Launchrock, the enabler of so many experiments – excellent and terrible ones. The screenshot is from a particularly clumsy one of my own.

I’m putting together some guidelines on experiment design for a startup team I’m working with. I’ve done this in the past and I’ve usually gone overboard. Mostly because I’ve read some great book or blog on how to design excellent experiments. So I would get all excited and summarize those great books in powerpoint tomes claiming to be quick guides to excellent experiment design.

But the reality is that most people aren’t trying to run excellent experiments. They’re mostly just trying to avoid running shitty ones. And they definitely don’t want a 37 page deck on how to do either. So I’ve been cutting the fluff to find a few questions to ask your startup buddy the next time she says “we’ll just run a test”. Questions most people should be able to remember even after hundreds of foosball games. They’re pretty straightforward and would make decent fridge magnets:

1. What is the question you want to answer?
2. What is the decision you want to make?
3. When will you evaluate?

What is the question you want to answer?

Replies often go something like “I want to try out X” (not a question), “I want to know how well X works” (again…) or when pressed “How well does X work?” No. Keep asking why until you have a question that can be answered. A good question will typically be structured something like:

• How will [doing something] (e.g. adding a cat video)
• for [someone] (e.g visitors coming from Linkedin)
• change [something]? (e.g. rate of lead submissions)

What is the decision you want to make?

“I want to see how it works” – not a decision. “I want to see if we should keep doing X or not” – better. “If the conversion rate is more than 5% higher we’ll make it permanent for all Linkedin users” – now we’re getting somewhere. Sometimes an open-ended experiment is needed to explore something, but mostly it’s someone being too lazy to take the data interpretation discussion up front.

If you don’t know which decision you are going to make, you risk doing a whole lot of work only to realize down the line that the results, however interesting, aren’t helping you decide on anything.  At least that’s what I’ve experienced. So tell me what it is you’re going to do that you can’t do without this experiment?

When will you evaluate?

Tests and experiments are notorious for being launched with great enthusiasm and then ignored with equally great determination. Just pick an end date and put it on your calendar. Ideally other peoples’ calendars as well. Find the question, identify the decision, schedule the evaluation. Please, just schedule it. Everything else will fall in place.

Sing with me: Question, decision, date. Question, decision, date.