Signup or Login

Track sales that interest you and monitor your favorite hotels by signing in to EveryLodge.


Crowdsourcing Hotel Ratings: Even More Problematic Than Usually Supposed

In a recent piece for the Wall Street Journal, Scott McCartney describes the “Big Flaws in Hotel Rankings.” It’s no secret that on sites like TripAdvisor, some hotels post favorable reviews of themselves — and unfavorable reviews of their competitors. TripAdvisor has a number of ways of protecting against this trickery. We were interested to learn from Mr. McCartney’s piece that this includes posing as freelance contractors and answering job ads that the hotels post in an attempt to find people to post favorable reviews. Other sites, such as Expedia and Priceline, reduce the likelihood of hotels influencing their ratings by only allowing users who have actually booked a stay at a particular hotel to review that hotel. This is a significant issue, but it is not the only reason that crowdsourcing hotel ratings is problematic. There are other — and we think bigger — flaws.

Suppose that sites like TripAdvisor succeed in rooting out fraudulent reviews. (This seems unlikely to happen; as user-generated rating sites become increasingly important, hotels will go to greater lengths to manipulate the figures, and their efforts will become even more difficult to detect, at least in a systematic and cost-effective manner.) Even if fraud is eliminated, user-generated hotel ratings will still be consistently unreliable. Here’s why.

First, the people with the greatest incentive to post reviews are people who have had marginal experiences. The next time you’re on TripAdvisor, read some of the negative reviews closely. Many will deal with issues that are unlikely to impact your own experience at the hotel. There’s the user who trashed the Four Seasons at Carmelo because a flood delayed his arrival at the resort. There’s the user who thought the Ritz in London deserved one star because she was denied entry to the hotel bar for failing to meet the dress code. (The dress code ruled out jeans but she thought an exception ought to have been made because hers were by Armani.) There are people who complain about their experiences with companies and services unaffiliated with the hotel. About cell phone reception. And even about the weather.

The most significant category of negative reviews, however, has to do with service. “The front desk was unwelcoming.” “The bellboy was too slow.” “I was overcharged and the staff was rude when I complained.” Unlike our previous examples, these complaints are legitimate. The problem is that they are overrepresented in online reviews, because being treated poorly causes  emotional responses (anger, embarrassment, defensiveness) that are much more likely to prompt a guest to take the time to log on and review a hotel. Sure, service is important, but it is only one of many factors that influence how much we enjoy a hotel stay. In negative TripAdvisor reviews, other factors like location, facilities and comfort are often not given the attention they deserve.

Given a sufficient number of reviews, this problem will disappear. Hotels in places where it is hot in summer should all be equally affected by people complaining about the heat in summer. And the number of complaints about service should roughly correspond with the quality of service. But there are not nearly enough reviews on TripAdvisor for this kind of representativeness. If a hotel only has 50 reviews, one scathing review from a heat-intolerant guest or someone who experienced unusually bad service is often enough to bump that hotel several places in TripAdvisor’s rankings.

Secondly, guests who write reviews online rarely have any basis for comparing the hotel with other similar properties. Choosing a hotel is necessarily a comparative exercise. We want to find the best hotel. To do this we need to know how the hotel compares to its peers across factors that we care about (distance to tourist attractions, cleanliness, facilities, noise, etc.)  Perhaps with the exception of seasoned business travelers who have tried lots of hotels in a particular city, users are poorly equipped to make these comparisons. It’s all very well complaining about how much street noise there is in a particular hotel if all similarly situated hotels are equally affected. This comparative exercise is best carried out by professionals. For this reason, we believe that guide books remain the best way of working out where to stay.

This is not to say that sites like TripAdvisor don’t have a useful function. “Candid traveller photos” are a great way of seeing what a hotel’s rooms actually look like. And if you spend enough time reading the reviews — and filtering out those that are the result of some kind of marginal experience that will probably not bear on your own stay — you can often get a good sense of a particular hotel. Just avoid looking at the numbers. “Ranked #27 of 347 hotels” is a deliciously definite claim. But it is very likely wrong.

– Andrew Nicol @aknicol

comments powered by Disqus