Nosy Search Engines Could Give Better Results

It’s no secret that major search engines try, data protection laws permitting, to use demographic information to provide more accurately-targeted advertisements. Now researchers believe the same details could be used for producing more relevant search results.

Ingmar Weber and Carlos Castillo of Yahoo’s research center in Barcelona say there’s a clear pattern between demographics and how people search. They say the difference isn’t necessarily what terms people use to search with, but rather what they mean by those terms. (That’s the problem which techniques known as semantic search try to solve.)

One example they give is the term “wagner”. They found that American women searching for the term were more likely to be looking for the composer, while American men were more likely to be looking for a spray-paint manufacturer.

Of course, that’s something of a generalization, and still provides no guarantees of getting things right. But the researchers believe they can improve the odds.

To test their theory they used the query logs of “a major US-based web search engine” (no prizes for guessing which), the birth year, gender and ZIP code of 28 million registered users, and details from the US census such as the average income in each ZIP code.

They then took a sample of around half a million searches from registered users and then checked the first result they clicked on from the resulting list. They also took the age and gender of each searcher and added in other demographic details from the census (based on their ZIP code).

From this information, the researchers developed a formula which could cross-reference a search request with the user’s demographics. They calculated that, if used on their sample searches, it would have got the “right” site (that is, the one the searcher actually clicked on and didn’t return) to the top of the results list seven percent more often than the search engine currently manages.

The research also threw up a few other findings which certainly back up the expectations you’d have through stereotypes. For example, people in ZIP codes with a high proportion of university graduates are more likely to type longer queries and then only follow one link from the results page (suggesting they got the right page the first time.)

Meanwhile the oldest 20% of people were most likely to search for a URL rather than a phrase. (As the researchers rather gently put it, “such queries are generally more appropriately placed directly in the URL bar of a browser.)