Since my last post, I’ve taken a job on the MSN Search team. It’s a pretty interesting problem space to work in, but it’s tough to stay objective about which search engine is the best one around. I want to see us ship such a cool product that everyone prefers us hands down, but before I wanted that, I was quite a Google fan.
So in order to keep my sanity, and so I can tell exactly when it’s time to start telling everyone I know that we have not only shipped a great search engine (which I think we can already claim with what’s running at search.msn.com today) but that we’ve shipped the best one in town, I’ve written myself a little tool that lets me compare how useful MSN’s results are to me vs. how useful Google’s are.
I won’t go into details in this post on how it works, but it has taught me at least one interesting thing: as I use MSN Search more over time, Google’s results are losing their relevance. Sounds a little strange, and possibly even un-objective, I know. Here’s what I mean: when I used to search for something, say “flight simulator”, the results that Google returned would emphasize the sites that it had scored as most relevant, using the scoring algorithm that they have developed over time. Over time, I learned which of those sites I considered most relevant. So as I scanned search results for a different, but related query, say “flight simulator add-on airplanes”, I would notice the results that pointed to my favorite sites. The sites that were my favorites because they were the best ones I had been introduced to. Not necessarily because they were the best sites out there.
This meant that when I started using MSN Search for flightsim queries, it had a handicap to overcome–namely that I was expecting results from a certain set of sites. It returned some from them, but it also returned many other results as well. Those others, at least initially, appeared to be “less relevant” simply because I wasn’t accustomed to them. Overall, Google was formerly beating MSN Search for me about 2 times in 3, but the margin has recently slipped to only 3 in 5. To the best of my knowledge, it’s not because Google’s returning worse results…