Experiment types in product development

Being able to efficiently and effectively test assumptions can make or break a product. Tests help you understand if the solution at hand is useable or solves the problem you've identified. The earlier and more effectively tests are run in the development period, the more likely you are to avoid waisting time building useless products.

There are several common experiment categories in product development.

• Prototype tests are when you share a prototype or mockup of a product and observe the user interacting with it. You might assign them tasks. For example, given an interactive website prototype you might ask users to "book a hotel room in Fiji") and observe their ability to carry out the task successfully.
• False door tests are when you add to your live product elements which indicate that a feature exists (even if it's not really complete yet) to see engagement, get signup or monitor interaction. For example, you could add a button to the website that says "Click here to subscribe for more articles" before you even have this functionality. If and when users click this button, you can take them to a page that says "coming soon" and even accept email addresses so that you can follow-up later.
• Ops hacks are when you have the button as above, but you're doing whatever work the button implies might be done automatically, manually behind the scenes. In services companies, this is what the work would be before a "tech transformation" automates that work.
• Concierge tests are when offer to do the thing you're testing for them, hand-in-glove, so that you can both learn how it's done, figure out a way to scale, and determine if there's value there. This is similar to an ops-hack, except the user knows that you're doing this by hand for them.
• User research is when you conduct interviews, observational studies, and surveys to learn more about the customers' problems, workflows, and day-to-day. The Mom Test is the best book (138 pages) to read on how to do this well. While this doesn't test a solution explicitly, this is the most critical and cheapest test to run to help determine what a solution might be.
• A/B testing is a way to test a series of changes to a product and compare the performance of the product across these changes. It often requires a lot of users' interactions to provide a strong signal, and requires some degree of infrastructure that can be challenging or expensive to set up. More simply, a change can be made directly and the performance monitored for regression without an A/B test as well.

Please send me a note if I should add some other test types to this list. It's not meant to be exhaustive or the best place to learn about each of these methods, but I'd like to cover the major types of tests such that someone can go from here to learning what they need to to carry out these tests.