05 October 2010

Hypothesis Testing & Software

What is Hypothesis Testing?

Hypothesis testing is used by Eric Ries in the context of the Lean Startup. Hypothesis testing encompasses:
  1. Proposition - for example: How many users will pre-order our product?
  2. Test - e.g., a website with a picture of the product and an accompanying pre-order button.
  3. Results - Evaluate the number of pre-orders.
  4. Action - Refine the test (e.g., A/B testing or different market segment) or tweak business model (see pivoting).
Does Hypothesis Testing Apply to Iterative Software Development?

A shortcoming I repeatedly see in Agile software development is what I derisively call bogus prioritization. That is, the business or product owner sets the direction of the development team based on seat-of-the-pants guesses of what's valuable, rather than on something more verifiable.

I propose that we - the iterative software development community - think of each release of new features as a business proposition, much like a startup company. Part of the Lean Startup philosophy is continuous testing and refinement as you converge to a better business model. Can we do the same with each software release?

We might ask,
  • What is a hypothesis and how best to test it with real customers?
  • That is, how do we make a bold hypotheses like "mCommerce will double eStore revenue" testable?
  • What hypotheses are we hoping to get answers to in this release?
  • How might prototyping of features be used to test hypotheses?
  • Will test-marketing feature sets be off-putting to our loyal customers? To new customers?
Believe those who seek the truth. Doubt those who find it.
~ Derek Sivers