Sort and Measure Method: How To Do User Research from Product Reviews for eCommerce A/B Tests

It’s well established now that elite conversion optimization firms and experts do extensive user research to generate high-quality test ideas. Without this research, companies will resort to just “gut instinct” and guesses, instead of having a list of evidence-backed hypotheses.

Download Free: A/B Testing Guide

But the typical suite of conversion optimization “user research” tactics falls into a small set of familiar categories:

• Use heatmaps/scrollmaps.
• Survey users on site.
• Watch session recordings.
• Do live user testing.

These are perfectly fine methods of learning about your user. This article, however, will focus on a treasure trove of user feedback that most ecommerce sites are underutilizing despite having abundant amounts of it: product reviews.

Product reviews give you a window into the psychology of the most qualified customers ever, those who purchased.

You can learn:

• What features or benefits do they value?
• What features or benefits do they not value?
• Why did they purchase?
• What words and phrases do they use to describe the products and benefits?

It’s also free and fast; the reviews already exist on your site; and there is no need to pay for participants or wait to collect data.

But the key to successfully using product reviews is in having a method to analyze them quantitatively. So you can extract actionable, numerical takeaways, not just vague high-level ideas.

Here, we’re going to tell a story about a quantitative method of analyzing open-ended feedback like product reviews. We call this method the Sort and Measure method. The information you can extract from it helps uncover which benefits are most valued by customers which can help shape copy and site design choices. The ideas you extract from applying this method can lead to better, more data-backed AB test hypotheses.

Analyzing Product Reviews to Improve Product Page Copy

In this case study, we analyzed hundreds of product reviews for the site Amerisleep, a multimillion dollar online mattress company in the United States.

Using our observation, we were able to quantitatively rank how important different benefits were to the users—comfort, pain, relative temperature regulation, environmental manufacturing, cost, returns, shipping, and more.

Knowing how important these benefits were allowed us to craft detailed, long form product detail pages with confidence. Instead of guessing, we had real data that told us which benefits mattered and which did not.

If you have a 7- or 8-figure ecommerce brand, it’s possible that you can get the same results with this technique. Let’s meet the company.

Meet Amerisleep: An Online Mattress Leader with Multiple Product Benefits

Online mattresses is an ultra-competitive space right now. Amerisleep is a leading player in this space. It has a high-end mattress that it feels has several advantages:

• (Comfort) Specially designed memory foam
• (Options) Available in 5 different firmness levels
• (Environmental) Manufactured in an environmentally friendly process
• (Temperature) Designed to avoid overheating
• (Cost) More affordable than retail memory foam mattresses
• (Returns) They have a great “sleep trial” policy

Having lots of benefits sounds great, but it leads to unanticipated site design challenges.

The Problem: Which Benefit to Emphasize?

Imagine you are the VP of Ecommerce for Amerisleep and also in charge of AB testing new product detail page copy to improve sales.

It’s nice to have a lot of product benefits, but it means that you have some predicaments when improving your product detail pages:

• Which of these benefits should you emphasize?
• If you list all of them, will it overwhelm the shopper?
• Do shoppers even care about [insert benefit]?
• Which benefits should you put at the top of the page, and which down below?

How People Solve This: On-Site Surveys

How would you solve this problem and understand which benefits are more important than others?

The most typical ecommerce user research method for extracting this information is ask users in an on-site survey. This is a useful method and can work. You use a Hotjar, Qualaroo or VWO style poll and ask shoppers which benefits they care about.

• You could ask an open-ended question: What are you looking for in a mattress?

• You could ask a multiple-choice question: Which of these benefits of a mattress is most important to you?

This is a solid method of starting to understand your users. If you do this consistently, in our experience, you’d be ahead of a lot of competitors.

But this method has a key drawback—you don’t know how qualified the respondents are.

This is especially an issue for high-end brands. What if you get lot of responses about the price being too high, but the respondents weren’t your target customers and weren’t ever going to buy from you anyways?

This is where product reviews become extremely useful.

Product Reviews: You Already Have the Answers on Your Site

A great way to solve this problem is to survey only users who have bought the mattress. You can do this on the success page by asking: What were the key reasons you bought your mattress from us today? But the vast majority of successful ecommerce brands already have these survey results on their sites: these are in the form of product reviews.

Product reviews are exactly what benefits users consider important: it’s a survey of only customers who bought, and it’s open-ended and in their own words.

Note: The case study in this article used a buyer verification process for leaving reviews. If such a process doesn’t exist, it is possible that non-buyers could leave reviews. But in our experience, most stores have too few reviews, not too many.

Reviews also have a tendency of being brutally honest (something many store owners know too well), as many will include criticism of what they didn’t like and what could be improved, even though they bought anyways.

Download Free: A/B Testing Guide

Sort and Measure Method: Extract Quantitative Answers to Make Decisions

But how are you going to turn product reviews into actionable information to inform your product page redesign or any other AB test on your site?

Enter the Sort and Measure Method. Here’s what it is, in a nutshell.

Casual Browsing: A Recipe for Personal Biases

Typically, what happens with survey data at an ecommerce company is that it’s emailed around and discussed a bit, and people then interpret it how they want.

One person says: “I told you, most of the reviewers were talking about back pain relief!”

Another employee says: “Well, what really struck me were the ones saying our environmental manufacturing was important.”

The debate can be never-ending. Most companies end up going with decisions based on either who is the loudest or who is the most senior.

For example, even before we used this technique, we did a large redesign and rewrite of their product detail pages. Using VWO Testing to A/B test the redesign, we got a great, 14% lift in checkouts – a lift that’s worth 7 figures annually.

Although this was a great result, we had email exchanges like this between me and the copywriting consultant we partnered with, debating where to put emphasis. Here is an email I wrote to him about a certain benefit I felt wasn’t that important:

Here is his email to me:

Like I said, these debates are never-ending and get you nowhere. That’s why, while this test was running, we started using the Sort and Measure Method. Before the test was even over, we had a quantitative understanding of which benefits did and did not matter to Amerisleep customers (not prospects, but customers who actually bought).

Now, we don’t have to spend additional time and resources A/B testing features on our eCommerce store, and see benefits that we know very few customers care about.
We don’t need to have endless email debates about what matters—we have data.

The Sort and Measure Method requires some manual effort, but it can be outsourced and the results are worth it. Here’s how it works.

Step 1: Sort

You list the product reviews in a spreadsheet:

Then you go through them one by one, create categories for common themes, and mark which categories they fall into.

It’s totally fine for a product review to be “sorted” into multiple categories.

As you work your way through the list, you’ll notice new categories and refine the list a bit, that’s okay. You’ll need to take a second pass to get it right.

Step 2: Measure

Now you just total up how many times certain categories are mentioned and you have your results, which any spreadsheet can do for you easily.

Here are the results for our analysis of Amerisleep’s product reviews, except that we’ve removed the benefit, to maintain data confidentiality.

(Yes, reviews are public information; and you can do this analysis for Amerisleep yourself, so it’s not really confidential…but will you? Probably not.)

Think about how useful this is. For the two smallest bars, we were able to clearly agree that this was not something happy customers talked about often.

That means:

• We don’t need to have mentions of those benefits take up a lot of space, or be above the fold.
• We should not prioritize AB testing to optimize sections that talk about these benefits.
• We should not prioritize testing these benefits in the ad copy.

Benefits of the Sort and Measure Method

We were able to use this method to understand which benefits were valuable to customers and which were not. This understanding has allowed us to focus our testing efforts on the copy that emphasizes the benefits that matter to our customers. Thus, the Sort and Measure Method has become a useful tool in our user research arsenal.

Try this method yourself on your own store. Doing so will give you information on improving site functionality, help you identify friction points that could be hurting sales, and help you create better data-backed AB test hypotheses.