Hypothesis testing is used by Eric Ries in the context of the Lean Startup. Hypothesis testing encompasses:
- Proposition - for example: How many users will pre-order our product?
- Test - e.g., a website with a picture of the product and an accompanying pre-order button.
- Results - Evaluate the number of pre-orders.
- Action - Refine the test (e.g., A/B testing or different market segment) or tweak business model (see pivoting).
A shortcoming I repeatedly see in Agile software development is what I derisively call bogus prioritization. That is, the business or product owner sets the direction of the development team based on seat-of-the-pants guesses of what's valuable, rather than on something more verifiable.
I propose that we - the iterative software development community - think of each release of new features as a business proposition, much like a startup company. Part of the Lean Startup philosophy is continuous testing and refinement as you converge to a better business model. Can we do the same with each software release?
We might ask,
- What is a hypothesis and how best to test it with real customers?
- That is, how do we make a bold hypotheses like "mCommerce will double eStore revenue" testable?
- What hypotheses are we hoping to get answers to in this release?
- How might prototyping of features be used to test hypotheses?
- Will test-marketing feature sets be off-putting to our loyal customers? To new customers?
Believe those who seek the truth. Doubt those who find it.
~ Derek Sivers
I like this line of thinking. It parallels a practice that I rarely see in software product companies - Set Development.
ReplyDeleteSet Development - producing a set of solutions to the problem and then choosing from the set of known solutions a best solution to the problem.
So few software companies take advantage of the nature of their chosen media - they are not carving Mt Rushmore - they are building SOFTware. How many of us get the solution correct on the first attempt? How many know the correct direction to travel when lost? Explorers know they must look in many directions to find the best path.
The Hypothesis: Of the set of solutions we will present in the next iteration, one will stand out as the best solution.
If true - you have found the path. If false - perhaps another round of exploration is in order.
David,
ReplyDeleteThanks for introducing me to the term "Set Development".
I've noticed that some web apps let users select a "new" interface with lots of client-side control behaviors or continue using a "classic" interface with old-school html controls posted to the server (e.g., Yahoo Fantasy Baseball gives users this choice & Twitter did that with their new UI).
I like the idea of giving customers a choice of solution sets for the purpose of collecting use patterns. Then, at some point, acting on that data. To me, that approach would be superior to a Product Owner declaring the "new" better, simply based on personal whim.