Product Validation vs. Research – A Follow Up

The discussions around product validation and research have provoked some excellent debate in the community and at Terem. 

Getting tight on the definition and use of validation in product is something that has emerged for me recently as the phrase has gained traction and popular use. Given the popularity of validation, having a common understanding becomes essential to success in teams. It also becomes essential because it helps you operate better and improve the success of what you’re doing.

I’ve even personally shifted on my use of the word. For example, I disagree with my own use of the word “validate” in my article, the 6 Steps to Validate JTBDs with Surveys. If I were to write it again, I would title it 6 steps to research JTBDs with surveys because you can’t observe interaction with your product through a survey (more on this later). 

To continue the debate and help tighten what product validation means, I’ve compiled a list of the questions or statements about what product validation is (or isn’t) that came out of the initial post I wrote on product validation. I’ve then put my thoughts down on each for the purpose of bringing further clarity to what product validation is. Further debate and discussion are more than welcome.

Background

Before I jump into the specific questions or statements on what product validation is (or isn’t), let’s cover some background.

Broad Clarifications

There were some themes in the discussions around product validation that are worth addressing as themes.

1. “But product validation is more than UI/UX”: Yes, it is. The definition proposed above covers more than UI/UX. Interaction can be making (or not making) a purchase, for instance.
2. It’s not that you do research or validation, but rather that not all research is validating. This is an excellent clarification that Angus McDonald (Senior Product Manager at Terem) shared.
3. Validation isn’t a binary outcome. Product validation is a continuum of confidence that changes over time (hat tip to Anthony Murphy). 

Clarifying Product Validation

Now, to clarify the definition, let’s go through the statements and questions around product validation.

#1 – There are plenty of B2B products where the end-users experience is not the reason they bought. User behaviour testing isn’t needed to get validation.

Yes, user behaviour testing might not be needed for a B2B product to be validated, but customer behaviour is (i.e. making a purchase). This fits with the proposed definition for product validation as a customer interacting or failing to interact with your product. 

What I would add is, even a purchase can’t be taken as validation. Better product validation – remembering validation is a confidence continuum – would be making the sale and seeing the user interaction you expect of a successful user/customer.

I’ve personally fallen afoul of this. I’ve personally sold business software to a senior manager (who was both purchaser and user), yet they failed to use the product. We had to issue a refund two months later. That is, I thought we had validation because of customer behaviour, but I hadn’t accounted for user behaviour as well.

#2 – Legal validation might be just seeking a legal opinion that the way your software works is legal.

Confidence that your product will be legal is needed in regulated industries. However, I’d argue that a legal opinion isn’t validation that your product is legal. Yes, it gives you confidence, but it’s research rather than validation because you aren’t observing any behaviour. 

So what is legal validation? It’s an interesting question. Here are some thoughts:

• Airbnb and Uber provide an interesting set of case studies in validating, whether a product or business is legal. 
• The legality of something changes over time – just like customer needs – based on new laws and changing public sentiment.
• The closest you can get to (in)validation is a court judgement or a legal body decision (police, regulator). 

This is one that needs further thought at another time.

#3 – Product validation is when the product is selling. When the user need finds a product fit, the market begins to pull the product out of the business. However, many companies have crashed and burned with huge sales, but a bad product.

This isn’t product invalidation, but business model invalidation. Distinguishing between product validation and business model validation will help you isolate your learnings, which will provide you with more concrete actions to improve. Often, product validation is within your control in larger organisations, but the business model might not be.

There is certainly some overlap between business model validation and product validation, but this is a topic for another day. 

The above is really an extension of an excellent discussion between Alex Freeman (Head of Commercial at Mondo Ventures) and Lynton Pipkorn (Group Innovation Manager at Suncorp Group) on LinkedIn.

#4 – It depends on the hypothesis you are testing. 

Thought: You could validate desirability with click-throughs on a landing page, you could validate feasibility with a prototype that works, and you could validate viability by selling something where your cost of sale is lower than your revenue.

I want to unpack this statement because it’s a strong one. The key strength to it, with regards to validation, is that it looks for observable behaviour/actions (click-throughs by a customer, a working prototype, an actual sale). This is a statement by someone that knows validation well. 

Where it usually gets a bit murky, and where the major driver for me writing about product validation is, is that the strength of the statement above seems to get diluted a bit too often. Validating viability becomes “let’s validate this by asking someone who did something similar what their margins are”. Validating desirability becomes (in its weakest form) “let’s validate this by interviewing a bunch of people we know and see if they like it.”

The bigger the decision (e.g. a pivot, or green-light on a new product), the more you need the distinction and clarity of thought. The clearer you are on what validation really means in your situation, the better the decision you can make.

A Final Thought

If you take one thing from this, it’s to ask questions about what validation means for you with your product, and really work out what matters for you.

---