Saturday, May 16, 2009

The Web's Dirty Little Secret: Flawed Measurement

Close readers of this space will recall that when I wrote about the decision by the Seattle Post Intelligencer to go web only, I reported widely different numbers for its monthly unique visitors. had only 30,000, but the PI itself said that Nielsen reported 1.8 million unique visitors, and its own internal numbers had 4 million uniques.

Big difference, huh.

Today's New York Times
tells a similar story for a much bigger site, hulu.

"Does Hulu, the Web’s most popular place for TV viewing, reach nine million people a month or 42 million? Millions of dollars in advertising revenue may hinge on the answer. But no one seems to know for sure how big the site’s audience is."

Nine million or 42 million????? That's like saying the circulation of the New York Times might be 800,000 or 4 million? You'd think you might pay a different price for advertising, depending on which one it was!

It's a problem that the industry has been wrestling with for some time. "The web has been around for more than 10 years as a medium, and it's been called the most measurable medium in history. Yet, web publishers at large media sites such as or have been dismayed to see that their direct counts of web visitors measured from their servers vary so wildly" from those reported by various measurement firms, says Mark Glaser.

It turns out that web measurement firms rely on two basic approaches, panel and census based. In simple english, the panel approach is similar to what has always been used for television or radio--track the habits of a supposedly random representative group and then extrapolate to the entire population. The census approach tries to track actual usage from the servers (but then has overstatement problems of its own because it might include counts for automated search spiders, spyware traffic and the like).

This may seem like inside baseball, but if I'm a media buyer and I can chose between a site that has 500,000 unique visitors or ten million, for the same price, it would be an easy decision, except that it's the same site.

The wildly divergent numbers demonstrate the nascency of the market for online video measurement. It’s “still the wild wild West,” said Rob Davis, a leader of the interactive video practice at OgilvyInteractive.

We are still at the infant stage for new media, still figuring out what the data means, how "efficient" it really is, and what it's worth. In the meantime, you can trust that the good sales people will be telling the best story--the difference from yesteryear is that instead of having to rely on the numbers brought in by the friendly radio ad sales person, you can go up to the web yourself and find whatever numbers you want.

1 comment:

  1. Great observation. Hits always tend to be an ambiguous area depending on where you look. I don't think there is going to be a solution for this problem either - you need to have a standardized service that is providing data for everyone. This won't happen though.

    Instead, I think there should be different types of measurement to consider when deciding between Company A and Company B.