What's the half-life of a Hong Kong restaurant review?

One of my favorite new interview questions to give potential product managers for our team is:  “What’s the best restaurant in Hong Kong?” On the surface, it’s a simple question – but it’s a wonderfully nuanced and complex question at the core.  I’ll get the puzzled look, and they’ll tell me that they’ve never been to Hong Kong. Then I’ll explain what I’m really asking: Given a set of data, how exactly would one determine the best overall restaurant in Hong Kong? Here are the some of the ideas that have informed the best responses.

Idea 1: Find a formula that gracefully combines volume + rating

Most candidates start with a simple answer:  “It’s the restaurant with the best overall average rating.” This idea is quickly shown to be naïve with a simple illustration:

  • Restaurant one has one review, which is 5 stars, for a 5.00 average rating overall
  • Restaurant two has 100 reviews, 99 are 5 stars, 1 is 4 stars, for a 4.99 average overall
Thomas Bayes, bad assThe key here is to find an approach that degrades gracefully – you don’t want to make up artificial rules, but rather find an angle that will allow you to compare restaurants with any number of reviews.  What you’re after is a way to apply the logic of that voice in your head that tells you to value insights drawn for more data over insights drawn from less data. Luckily, we can develop an average that is a “truer reflection of how the [restaurants] are rated in relation to each other.” If you’re interested in how this is done, Thomas Bayes, a British minister that died nearly 250 years ago, is your man.

Idea 2: Determine the half-life

Let’s again compare two restaurants:

  • Restaurant one has 20 reviews.  First 10 are 1 star, most recent 10 are 5 star. 2.5 average rating overall
  • Restaurant two has 20 reviews.  First 10 are 5 star, most recent 10 are 1 star.  2.5 average rating overall

Obviously, you’d want to go to restaurant one.  Maybe there’s been a change of ownership for the better and restaurant one is worth visiting now.  The ten most recent ratings are better predictors of the experience you’ll have now than the ten ratings that precede them. So, we need to dial down the weight we give to earlier ratings.

The basic way to do this would be to introduce a notion of decay, or half-life to the reviews.  Simply put, the most recent reviews should count more than the older reviews (but the older reviews should still count for something).

Idea 3: Factor in reviewer experience and helpfulness

Which of these would you rely more on?

  • Review A:    Written by someone that is writing their first review
  • Review B:    Written by someone that has contributed 20 reviews, and received thousands of helpfulness votes

Many would choose the content from the veteran reviewer because of the positive social signals attached to it and the sense that “practice makes perfect.”

But other factors could go into this decision as well. How close is the reviewer to you on the interest graph? Does the reviewer’s language indicate that they’re being as objective as possible, or that they have dietary restrictions or preferences that aren’t relevant to many others (statistically speaking)? Do we assign more value to reviews that don’t express preexisting expectations?

Idea 4: Personalize the answer

When you get down to it, there isn’t one single answer.  The best restaurant for a college student isn’t the same as the best restaurant for a foodie.  Based on implicit and explicit behavior provided by the user, you will want to filter down and personalize the answer.

So, if a user has filled out profile data indicating they are a “food snob”, then we may elect to feature restaurants rated highly by others with related terms in their profiles. Or, if we see that their IP is in the US and they’re looking at reviews of Hong Kong restaurants, we may give preference to reviews from other American travelers to Hong Kong.

Of course, users should always have the option to remove personalization layers from their experience. At a certain point, personalization can become limiting rather than empowering, and users should be able to determine for themselves where that line is.

Data creative

So, what’s the best restaurant in Hong Kong? Depends whom you ask. When you ask a hundred product managers or data analysts, you’ll get a hundred different convincing answers. If a candidate is able to make it all the way through this thought process, we’re in for an amazingly engaging conversation. It’s not about the answer they give me, but how they’re getting there. It’s far more important that they are “data creative,” because it means that they’ll be able to build the best possible experience for our clients and their customers.

If you’re interested in joining the Product Management team, check out our openings.

Customer Intelligence
  • hongkonghotel

    Amazing reviews you have here, i’ m sure this can attract lots of readers over the net for the cool write up. Keep it up

  • http://blog.bazaarvoice.com Ian Greenleigh

    Great piece, Manish.