Vanity searching in vain

Tom Foremski has an interesting post this week about the apparent shrinking scale of Google’s search results.

It turns out that searches on his own name are yielding fewer and fewer results over time.  That understandably surprised Foremski since he’s both prolific and widely referenced.

Foremski is mostly concerned with the fact that this is happening.  “Clearly,” he concludes, “the number of results that Google claims is bogus.”

“What’s going on?” he then asks, and his commenters suggest a number of likely explanations in response.

But whatever the reason, Foremski’s observation makes clear that Google search statistics have dubious validity as measures of scale. And yet journalists, myself included, use them all the time as a shorthand way of measuring impact.

Our model is citations. These certainly have validity when measured against complete and comprehensively indexed sets of data. An example would be a full archive of a particular newspaper’s articles.  You can clearly state that a particular word was used, say, in the New York Times x times in 2005 and y times in 2010.

Citations are also important in science where they help measure impact of a researcher’s work. It genuinely means something to say someone is the most cited psychologist, for example, in the world.

Google, I’m sure, doesn’t mind us thinking that it, too, offers a pretty-much complete and comprehensive data set along with a true count of the number of citations in that data set.

But both the comprehensiveness of the set and the reliability of the count seem to be in fairly constant flux.

There’s solace here for Foremski — his vanity search results may not truly measure his reach (and, along with it, his impact) online.

For the rest of us there’s a lesson, too: that we need to swallow the citation counts we find on Google with a healthy dose of salt.  And I guess we need to be looking for new kind of vanity index to use from now on.