You have a killer idea. You practice your elevator pitch and all of your friends and family are convinced that you have found the Next Big Thing. You build, nurture, and launch it only to find that it falls flat in the real world. What went wrong?
User behavior is unpredictable. People’s preferences, opinions, and awareness are constantly evolving. So, how do you make sure your great ideas are truly great?
We recently tested a personalization feature to change the display order of consumer-written content on product pages, displaying the most relevant opinions for the individual viewer first. The feature evolved over five rounds of testing with real consumers in our user research lab, specifically designed to guide users through the shopping experience without biasing their opinions. We soon learned that our initial assumptions didn’t hold up – which led to a revised feature that has increased conversion for early pilot clients by 3.6-3.9%. Here’s how we got there.
Shape the challenge through early research
We first brought 25 shoppers into the lab equipped with eye-tracking software, and collected viewing patterns as they shopped online. The generated heatmap images highlighted three top focus areas for consumers on a product page:
- Photos: Images are the first thing shoppers look at.
- Total cost: Consumers look for costs, including tax and shipping.
- Consumer content: They actively seek out consumer feedback, and most often concentrate on the first few reviews displayed.
Understanding that the first few consumer opinions a shopper sees are the most impactful, we recognized that changing display order of those opinions may positively impact conversion.
Analyze big data to test assumptions
To evaluate this assumption beyond a limited test group, we analyzed the massive amounts of usage data collected across the network and discovered that a low percentage of product page viewers click through to see more consumer opinions beyond the first page. The data confirmed the results of the eye tracking: consumers often read only the first few that appear on a product page. Our takeaway: the most important consumer-written content needs to be highlighted immediately and easily found.
Functioning prototypes are worth a thousand… pictures
Our next step was to develop prototypes we could show to users in the lab. Prototypes for personalizing review display ranged from personalized ratings based on the user’s demographics, to highlighting snippets of content, to seeing reviews by friends.
Each of the four tests yielded the same response. While people thought these features were interesting, they kept telling us what they want most. “It’s interesting, but I’m unique and want to see all the reviews to decide for myself. I want the full story.” Additionally, we found that visitors seek out critical commentary in the form of reviews with low star ratings to feel like they have seen the full, balanced story.
We thus determined the most relevant content needed to include rich text, helpful votes, recency, photos, some critical opinions, and relevant keywords from their search.
Confirm the story with A/B testing
The final step in our process was to build the feature and test it on live pilot clients – gathering usage data from real shopping consumers in their natural shopping environment, free from the lab. We crafted an A/B test in which certain shoppers saw product pages with the more relevant opinions displayed first, while others simply saw the default review order.
Our design hypothesis for personalization was confirmed. The A/B tests on multiple client pilots showed that the relevance sort consistently outperformed others. In fact, we saw a 3.6-3.9% improvement in conversion across these early pilot clients.
Product design and user research can help you find true winning ideas faster by understanding what people want and need. The next time you have that killer idea people will love, watch some of them try it out. You’ll be amazed at what you learn.