Continuous Product Discovery with Painted Door Tests

Great products evolve as the user’s needs and motivations evolve. How do you keep products competitive and loved by users? A typical answer to this question involves customer feedback, sales requests, and competitor research. The outcome of this revolves around what “we know today”. But how do we find out what customers will need in the future?

Painted door test (fake door experiments or smoke tests) is a strategy to test new ideas or ‘big bets’. This approach gauges the user’s interest in a feature, offer, or promotion without fully developing a solution. It simulates features without the functionality, often achieved by a single click on a call-to-action or URL.

Example: Are customers of an online rental interested in a subscription model? Will users of a social networking app find value in a live-streaming feature? 

Source: Convert

Painted door tests are low effort but highly strategic to explore a user’s interest in something new and unique. Instead of experimenting with features or user flows, these tests are an “illusion experiment”.

PDTs are a great example of “Minimum Viable Experiments”.

Why do it? Instead of spending a huge engineering budget on something with less surety, you can be more intentional with development. Product teams struggle with the backlog and stakeholder requests daily. Developing something radical backed by market research but not requested by customers or stakeholders could potentially harm frequent valuable delivery. 

When to Conduct a Painted Door Test

Imagine you are a product manager at Duolingo. You know the impact of personalization on CLV and User Satisfaction. But is this an opportunity for Duo learners? Do they have a strong desire for personalized lessons, learning styles, or notifications? The development cost and marketing effort are too high to get stakeholders onboard readily. You may also want to assess user interest and translate it into a business opportunity. 

Enter the Painted Door test. 

Add a CTA, “Interested in learning at your own pace?” in the app, and upon clicking it, users can either see a video demo of what’s coming next or a message that says, ”Thanks for showing interest. This will be available soon”. The number of clicks is the quickest way to see whether users have a desire for personalization. 

The CTA button is a Painted Door. The click is the “reaction” of users.

Painted door tests help invest in case building instead of engineering efforts. Consider doing this when you come across questions or goals like:

1. I want to explore entering a new market or niche.
2. Why should I build something that is complex but might help users?
3. I want to innovate and introduce something that delights my customers.
4. I don’t know what my users are thinking or looking for.
5. I have identified a problem but want to derisk the solution.

How to Conduct the Painted Door Test

Step 1: Define the Idea

Clearly explain the concept that you want users to react to. Is it a new feature, a new product category, or a pricing model? How does it impact the existing user journey? At what point do users interact with it? Create a mindmap or thinking tree to visualize the idea. 

Protip: Consider the impact of this idea on existing technology and user journey.   

Step 2: Define the Facade

Work on figuring out how you would display it to users. Is it going to be a button that they can click, a toggle that can be enabled, or a message that is displayed as a pop-up?

Protip: At this point get all cross-functional stakeholders on one page to communicate the change and set expectations about the impact it may drive for different teams.

Step 3: Develop and Track

Once the facade is agreed upon, now it's time to develop. In parallel, decide how you will track the results using tools like Amplitude, etc. What exactly will you track? # of clicks, submissions, or support tickets? Given the significance or sensitivity of the facade will all users participate in the test or a certain percentage? 

Protip: In step one you have identified what journey and steps your test will impact. Track impact on primary metrics of those journeys. 

Example: Is the new test negatively or positively impacting signups?

Step 4: Analyze and Interpret

Give the facade to run for a considerable time period that helps you gather statistical significance. Document all subjective and quantitative findings from the tests that may impact the product or other teams.

Protip: Sometimes users react passively to a change which can be monitored by heatmaps. Is the user curious about the new change or completely ignores it? Analyze whether the user hovers over the new change for a considerable time. This can help decide whether to keep the test running or improve the messaging or positioning of the facade.

Step 5: Use Insights to Inform Decisions

After concluding the test and documenting the findings, now teams can translate it into strategies and decisions. Did the test drive have negative, positive, or no results? Do we want to validate further or move to solutions? 

Protip: Consider communicating the findings to a broader audience. You may have uncovered learnings about users that have the potential to improve marketing messaging or customer support levers.

Considerations while Running a Painted Door Test

Painted door tests don’t offer real value to the end user as there's no output based on the expectations that have been set with a CTA or button. This can negatively impact customer satisfaction. Another result of this activity could be an increase in the number of support tickets inquiring about the offer or feature. It is important to focus on satisfactory messaging for the end users, and clearly communicate expected impact to cross-functional teams involved in the product.

How does a Painted Door Differ from A/B Tests?

While Painted Door tests focus on identifying a user’s interest in a specific feature or offer, A/B tests are focused on comparing two versions of a specific element to see which performs better. Between the two the goals are also different. With A/B tests measure impact on a specific metric while Painted Door tests measure interactions. 

In product teams, A/B tests are used at the final stage of product delivery while Painted Door tests are valuable during early stages of development.

For Product Managers

When used in correct circumstances, Smoke tests are an effective way to empathize with customers by encouraging you to dive deep into the user's thought process. It's a great tool to learn from behaviors and apply learnings to prioritization and roadmap. 

In The End

Things are changing really fast out there, so knowing what customers need before they ask is a big advantage. Painted Door Tests offer a low-risk way to test ideas, gather data, and reduce guesswork. It helps teams validate concepts before heavy investment. By using this approach, product teams can turn insights into action and stay ahead of the competition.