Second Life ratings highlight growing web analytics issue

Stan Beer
Monday, 07 May 2007 06:52

Opinion and Analysis

The social networking virtual world called Second Life has caught on like wildfire in Europe, while North Americans have been far more reserved in their adoption of the service, according to web analytics firm Comscore. However, the most recent stats from Comscore highlight a growing issue concerning the use of sample panels for determining the number of web site visits.

According to Comscore, a relatively modest 1.3 million logged-in to Second Life in March 2007, of which 61% were Europeans, while just 19% were from the US. In fact, Comscore reckons that the number of active German Second Life residents exceeds those in the US. This is despite the fact that Second Life was launched in the US in 2002 by San Francisco based Linden Labs.

Comscore, like rival analytics firms such as Nielsen Netratings, bases its findings on the surfing habits of sample panels of volunteer users, rather than tracking actual hits to sites. The method, similar to how TV ratings are compiled, was developed more than 70 years ago and has come under increasing scrutiny recently, with massively popular social networking site Digg reporting huge discrepancies between its own stats based on server logs and those supplied by Comscore.

According to a report in BPM Today, Digg's own stats, independently audited and confirmed by web analytics firm WebSideStory, showed the site was getting 15 million visitors a month. Comscore, however, put the figure at between one and two million. Technorati, a popular site that tracks searches of blogs, reported similar order of magnitude discrepancies between its own logs and those of analytics firms using sample ratings systems .

Needless to say, the advertising rates of a site that gets an audience 15 million visitors a month differs significantly from a site an order of magnitude lower on the scale of visitations. Such discrepancies have prompted calls from web publishers for both Comscore and Nielsen Netratings to voluntarily have their ratings systems audited by an independent auditor.

On April 20, the Interactive Advertising Bureau (IAB), which represents hundreds of US web publishers sent a letter to both comScore and Nielsen Netratings, requesting them to submit to a third-party audit of their measurement processes.

In an open letter to both Comscore and Nielsen Netratings, the IAB stated:

"The discrepancies exist between the audience measurements of comScore and NNR (Nielsen Netratings) and those of the server logs of the IAB’s own members. Further compounding these differences are the disparities between comScore’s and NNR’s own measurement results. All measurement companies that report audience metrics have a material impact on interactive marketing and decision-making. Therefore, transparency into these methodologies is critical to maintaining advertisers’ confidence in interactive, particularly now, as marketers allocate more budget to the platform.
 
"Without these audits, the industry has no way of knowing whether these deviations in measurement result from inconsistent counting or from outdated measurement methodologies, such as the panels developed in the 1930s and still relied on today."

“To persist in using panels that potentially undercount or ignore the diverse populations that are the future of consumer marketing is to deny marketers the insights they need to build their businesses,” wrote IAB President and CEO Randall Rothenberg in an open letter to Magid M. Abraham, the President and CEO of Comscore, Inc., and William Pulver, the President and CEO Nielsen NetRatings. “And it certainly appears to us as if these audiences are being undercounted or disregarded.”

As a web publisher, it seems strange to us that there are still parties prepared to put their faith in statistics gathered from volunteer sample audiences in an era where technology is widely available to track every single visitation to a site.

We would certainly be interested to know the composition of such volunteer samples. What percentage are children, technolophiles, games freaks, business technology users, technology decision makers? What incentives are there for members of these samples to be part of a web analytics firm's panel? How many refusals does an analytics firm get from ordinary web surfers before it finds someone who is prepared to sacrifice their privacy to be part of a panel? Hands up anyone who is or knows someone who is part of a web analytics panel.

The problem of course for the analytics firms that use panel ratings systems is that the advent of the Internet and its associated technology has brought the barriers to their businesses down. Advertisers no longer have to rely on sample audiences when every single visit to a site can be tracked and recorded.

A message to advertisers: Digg definitely gets much more than one or two million visitors a month. We know because we track the traffic Digg sends to us whenever we make it to their front page!