The IBM computer “Watson” has established a clear lead after two days of its gameshow challenge against Jeopardy champions Ken Jennings and Brad Rutter. But thankfully for those fearing 2011 may be the year the machines take over, it has made a couple of flubs.
The three-episode challenge is an attempt to test IBM’s abilities to create a machine capable of sophisticated reasoning rather than pure data storage and processing. The format of Jeopardy (contestants guess the question from a supplied answer) is a particular test of this. Not only does the contestant have to figure out which elements of the clue are most significant, but there is inherently no single “correct” question. This means the contestant first has to simultaneously consider a range of possibilities, but must then make a judgment about which is most likely to be the one the quiz setter was thinking of.
As we noted last week, though, it’s not just a game: the results of the show will help IBM refine the process for other situations such as those that involve diagnosing a problem working only from known symptoms, whether that be medical advice or tech support.
After the first day of competition, things were tight: Watson and Rutter were tied on $5,000 apiece, with Jennings lagging on $2,000. But despite its impressive performance, one limitation of Watson emerged when it repeated an incorrect answer that had already been offered by another contestant.
Come day two, though, and Watson hit its stride: answering 14 questions correctly out of 15, it ended the episode on $35,734 compared to $10,400 for Rutter and $4,800 for Jennings.
It’s the question that Watson got wrong that’s attracting the most attention though. Offered the clue “Its largest airport is named for a World War II hero; its second largest, for a World War II battle”, it guessed Toronto rather than the correct reply of Chicago. That wouldn’t have been so embarrassing had the category not been US Cities.
Though that might seem like a total duh-brain move, some have offered possible explanations. Stephen Baker of the Huffington Post speculated that Watson may have been influenced by data sources that lumped in Toronto with US cities, such as the Blue Jays baseball team playing in the American League, or authors embarking on US city book tours that include the short hop across the border.
It’s also worth noting the mistake was only uncovered because the particular rules of the round meant an answer had to be provided. Watson estimated the probability of Toronto being correct at just 14% which, under normal circumstances, would have meant it wouldn’t have buzzed in with an answer.