We’ve talked to several advisors and prospects who expect to see a “U” shape in the distribution of their review submissions by ratings. In other words, there should be an equal number of “1” ratings as there is “5” ratings, displaying the extremes of customer opinion. We weren’t sure what to expect when Bazaarvoice got off the ground a year ago.

We didn’t know what to expect then, but we do now.

Across many clients in diverse industries this “U” curve turns out to be more like a “J” curve…almost a reverse "L" (see below). The average rating across all clients is 4.3 out of 5 stars. The distribution looks like a J, where there are more 1s than 2s, but far more 4s and 5s than the lower ratings.

Why is this? Aren’t people more likely to share their word of mouth about bad experiences? Perhaps they are more likely to share negative opinions when they have personal experiences with a company (service, sales) than the product they buy?

And perhaps customers are interested in sharing their opinion about great products they buy, because there are so many mediocre products. So there’s some satisfaction in sharing the news when we find a product we love.

We’ll learn more and share more here. But in the meantime, this “J” curve is part of the answer to the conern: “What about negative reviews?”

27 Responses to “Ratings “J Curve””

  1. motorolaVE440

    If the product quality is maintained then you can get a J curve in every business, with respect to users reviews. But once the product gets famous, the product sellers gives a cheaper quality in sake of brand.
    http://www.cdmacellulars.com/

  2. motorolaVE440

    If the product quality is maintained then you can get a J curve in every business, with respect to users reviews. But once the product gets famous, the product sellers gives a cheaper quality in sake of brand.
    http://www.cdmacellulars.com/

  3. An obvious explanation for the J Curve.

    If people are influenced by user generated reviews and I believe that they are, than customers are more likely to buy items with good reviews. Thus it stands to reason that future customers will also likely be satisfied with their purchase. In this situation the accurate representation of a product and its relative price should normally drive a J Curve.

    Also it is most likely that products or companies whose customer satisfaction is inconsistent would typically not use a feedback system that would demonstrate their relative poor satisfaction levels.

  4. An obvious explanation for the J Curve.

    If people are influenced by user generated reviews and I believe that they are, than customers are more likely to buy items with good reviews. Thus it stands to reason that future customers will also likely be satisfied with their purchase. In this situation the accurate representation of a product and its relative price should normally drive a J Curve.

    Also it is most likely that products or companies whose customer satisfaction is inconsistent would typically not use a feedback system that would demonstrate their relative poor satisfaction levels.

  5. In my experience, the take-away from a U-shape curve is not that your respondents are more “balanced”, but that your quality issues are rampant – inconsistent process, customer service, and product quality to name a few. The more the J-curve looks like a U, quality is root cause because it indicates high levels of variation among customer experiences. L’s say you’re not meeting customer expectations, while U’s suggest you are completely out of control. Conversely, the more hockey-stick the J (upward to positive scores), the greater an indication that prodcut quality, customer service, and internal processes are in sync. Virtuous cycles–by definition–work.

    That said, you can’t rest on your laurels. I like the way Ben suggests variation among “happy” responses to tease out additional shades of gray, and using that as a segue for getting additional feedback.

  6. In my experience, the take-away from a U-shape curve is not that your respondents are more “balanced”, but that your quality issues are rampant – inconsistent process, customer service, and product quality to name a few. The more the J-curve looks like a U, quality is root cause because it indicates high levels of variation among customer experiences. L’s say you’re not meeting customer expectations, while U’s suggest you are completely out of control. Conversely, the more hockey-stick the J (upward to positive scores), the greater an indication that prodcut quality, customer service, and internal processes are in sync. Virtuous cycles–by definition–work.

    That said, you can’t rest on your laurels. I like the way Ben suggests variation among “happy” responses to tease out additional shades of gray, and using that as a segue for getting additional feedback.

  7. Product reviews are great for the webmasters. They get a lot of content on the site free. Either by way of their own reviews, which are anyway copied from some other source, or by way of reviews written by readers. This helps a lot in SEO.
    When you reads posts on any site, there is so much of contradicting views that one is bound to get confused. Moreover, you do not know which review is genuine.
    I think, if you are really unhappy with a product, writing a reviews vent out your frustrations for the product. Although, you are sure that not even 1% of the site visitors will not read it. The same holds true for happy buyers of products.

  8. Product reviews are great for the webmasters. They get a lot of content on the site free. Either by way of their own reviews, which are anyway copied from some other source, or by way of reviews written by readers. This helps a lot in SEO.
    When you reads posts on any site, there is so much of contradicting views that one is bound to get confused. Moreover, you do not know which review is genuine.
    I think, if you are really unhappy with a product, writing a reviews vent out your frustrations for the product. Although, you are sure that not even 1% of the site visitors will not read it. The same holds true for happy buyers of products.

  9. I wonder if it would be more of a U shape if the reviews were anonymous. Perhaps people are less likely to give negative ratings or sumbit any ratings if they know their identity will be revealed. E.g., how many times have you given you waitress negative feedback on those restaurant surveys that occassionally accompany the bill? I’m running a small experiment myself WRT rating people anonymously online (www.tomslist.net). There’s only a couple hundred ratings so far and the distribution looks a bit more U-ish but it’s still quite early. We’ll see how it goes.

  10. I wonder if it would be more of a U shape if the reviews were anonymous. Perhaps people are less likely to give negative ratings or sumbit any ratings if they know their identity will be revealed. E.g., how many times have you given you waitress negative feedback on those restaurant surveys that occassionally accompany the bill? I’m running a small experiment myself WRT rating people anonymously online (www.tomslist.net). There’s only a couple hundred ratings so far and the distribution looks a bit more U-ish but it’s still quite early. We’ll see how it goes.

  11. Sam,

    We have approx. 18,000 user reviews of mobile phones contributed to our site and aggregated from other sources. Of those 18k 25% of reviewers gave their phone full marks, 10/10.
    More on that here if you’re interested: http://www.cellphones.ca/1779

  12. Sam,

    We have approx. 18,000 user reviews of mobile phones contributed to our site and aggregated from other sources. Of those 18k 25% of reviewers gave their phone full marks, 10/10.
    More on that here if you’re interested: http://www.cellphones.ca/1779

  13. Hi – I originally left this comment at a blog that links here, but am reposting here as I thought you might find it useful.

    This is as much of a question anchoring problem as it is a problem of certain customers responding. I’m a survey researcher, and attended a presentation recently which covered this issue (I’ve hunted, but cannot find a relevant link, sorry). The gist was that more useful, distinguishing, information could be obtained by chaninging the response options from somthing like:

    Extremely good

    Extremely bad

    To something like:
    Way beyond expectations
    Better than expected for a good firm
    About what I expect from a good firm
    Not as good as expected for a good firm
    Way below expectations

    You’d need to play around with the wording, but the effect should be to gravitate answers to the midpoint, so that the extremes actually serve their purpose: to discriminate between the average, the really good and the really bad.

    Of course, if the aim is only to weed out the really bad, then this may not matter much – the bad should show up regardless.

  14. Hi – I originally left this comment at a blog that links here, but am reposting here as I thought you might find it useful.

    This is as much of a question anchoring problem as it is a problem of certain customers responding. I’m a survey researcher, and attended a presentation recently which covered this issue (I’ve hunted, but cannot find a relevant link, sorry). The gist was that more useful, distinguishing, information could be obtained by chaninging the response options from somthing like:

    Extremely good

    Extremely bad

    To something like:
    Way beyond expectations
    Better than expected for a good firm
    About what I expect from a good firm
    Not as good as expected for a good firm
    Way below expectations

    You’d need to play around with the wording, but the effect should be to gravitate answers to the midpoint, so that the extremes actually serve their purpose: to discriminate between the average, the really good and the really bad.

    Of course, if the aim is only to weed out the really bad, then this may not matter much – the bad should show up regardless.

  15. Thanks for sharing – very counter-intuitive. And very good to know when you’re pushing to enable more openness online.

  16. Thanks for sharing – very counter-intuitive. And very good to know when you’re pushing to enable more openness online.

  17. Coincidentally, a couple weeks after we announced the J-Curve KellerFay came out with a study that found 63% of word of mouth (online or offline) is positive. The conventional wisdom that the majority of word of mouth is negative is no longer conventional nor wisdom!

  18. I think the negative reviewers will be less likely to rate but will provide bad word of mouth in the real world.

    This provides two possible dangers – firstly passive viewers will see the J curve effect and discount its veracity, secondly and more importantly the absence of low ratings may breed a false sense of achievement in those being rated. My philosophy has always been to look for the dissatisfied and determine if their dissatisfaction is valid – that way lies progress.

  19. I think the negative reviewers will be less likely to rate but will provide bad word of mouth in the real world.

    This provides two possible dangers – firstly passive viewers will see the J curve effect and discount its veracity, secondly and more importantly the absence of low ratings may breed a false sense of achievement in those being rated. My philosophy has always been to look for the dissatisfied and determine if their dissatisfaction is valid – that way lies progress.

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>