One of my favorite new interview questions to give potential product managers for our team is: “What’s the best restaurant in Hong Kong?” On the surface, it’s a simple question – but it’s a wonderfully nuanced and complex question at the core. I’ll get the puzzled look, and they’ll tell me that they’ve never been to Hong Kong. Then I’ll explain what I’m really asking: Given a set of data, how exactly would one determine the best overall restaurant in Hong Kong? Here are the some of the ideas that have informed the best responses.
Idea 1: Find a formula that gracefully combines volume + rating
Most candidates start with a simple answer: “It’s the restaurant with the best overall average rating.” This idea is quickly shown to be naïve with a simple illustration:
- Restaurant one has one review, which is 5 stars, for a 5.00 average rating overall
- Restaurant two has 100 reviews, 99 are 5 stars, 1 is 4 stars, for a 4.99 average overall
Let’s again compare two restaurants:
- Restaurant one has 20 reviews. First 10 are 1 star, most recent 10 are 5 star. 2.5 average rating overall
- Restaurant two has 20 reviews. First 10 are 5 star, most recent 10 are 1 star. 2.5 average rating overall
Obviously, you’d want to go to restaurant one. Maybe there’s been a change of ownership for the better and restaurant one is worth visiting now. The ten most recent ratings are better predictors of the experience you’ll have now than the ten ratings that precede them. So, we need to dial down the weight we give to earlier ratings.
The basic way to do this would be to introduce a notion of decay, or half-life to the reviews. Simply put, the most recent reviews should count more than the older reviews (but the older reviews should still count for something).
Idea 3: Factor in reviewer experience and helpfulness
Which of these would you rely more on?
- Review A: Written by someone that is writing their first review
- Review B: Written by someone that has contributed 20 reviews, and received thousands of helpfulness votes
Many would choose the content from the veteran reviewer because of the positive social signals attached to it and the sense that “practice makes perfect.”
But other factors could go into this decision as well. How close is the reviewer to you on the interest graph? Does the reviewer’s language indicate that they’re being as objective as possible, or that they have dietary restrictions or preferences that aren’t relevant to many others (statistically speaking)? Do we assign more value to reviews that don’t express preexisting expectations?
Idea 4: Personalize the answer
When you get down to it, there isn’t one single answer. The best restaurant for a college student isn’t the same as the best restaurant for a foodie. Based on implicit and explicit behavior provided by the user, you will want to filter down and personalize the answer.
So, if a user has filled out profile data indicating they are a “food snob”, then we may elect to feature restaurants rated highly by others with related terms in their profiles. Or, if we see that their IP is in the US and they’re looking at reviews of Hong Kong restaurants, we may give preference to reviews from other American travelers to Hong Kong.
Of course, users should always have the option to remove personalization layers from their experience. At a certain point, personalization can become limiting rather than empowering, and users should be able to determine for themselves where that line is.
So, what’s the best restaurant in Hong Kong? Depends whom you ask. When you ask a hundred product managers or data analysts, you’ll get a hundred different convincing answers. If a candidate is able to make it all the way through this thought process, we’re in for an amazingly engaging conversation. It’s not about the answer they give me, but how they’re getting there. It’s far more important that they are “data creative,” because it means that they’ll be able to build the best possible experience for our clients and their customers.
If you’re interested in joining the Product Management team, check out our openings.