Clive Thompson has a wonderful article in the NY Times Magazine on Watson, the supercomputer programmed to excel at Jeopardy. Thompson delves into the clever heuristics used to generate singular answers to ambiguous questions. (Watson relies on massive amounts of parallel processing, so that “he” is running thousands of Google searches simultaneously.) While Watson’s performance is certainly impressive, I thought the most interesting part of the story involved the failings of the machine. It’s easy to rhapsodize about the ever escalating speed of microchips, but it turns out that Watson is often too slow at ringing the buzzer:
In more than 20 games I witnessed between Watson and former “Jeopardy!” players, humans frequently beat Watson to the buzzer. Their advantage lay in the way the game is set up. On “Jeopardy!” when a new clue is given, it pops up on screen visible to all. (Watson gets the text electronically at the same moment.) But contestants are not allowed to hit the buzzer until the host is finished reading the question aloud; on average, it takes the host about six or seven seconds to read the clue.
Players use this precious interval to figure out whether or not they have enough confidence in their answers to hazard hitting the buzzer. After all, buzzing carries a risk: someone who wins the buzz on a $1,000 question but answers it incorrectly loses $1,000.
Often those six or seven seconds weren’t enough time for Watson. The humans reacted more quickly. For example, in one game an $800 clue was “In Poland, pick up some kalafjor if you crave this broccoli relative.” A human contestant jumped on the buzzer as soon as he could. Watson, meanwhile, was still processing. Its top five answers hadn’t appeared on the screen yet. When these finally came up, I could see why it took so long. Something about the question had confused the computer, and its answers came with mere slivers of confidence. The top two were “vegetable” and “cabbage”; the correct answer — “cauliflower” — was the third guess.
To avoid losing money — Watson doesn’t care about the money, obviously; winnings are simply a way for I.B.M. to see how fast and accurately its system is performing — Ferrucci’s team has programmed Watson generally not to buzz until it arrives at an answer with a high confidence level. In this regard, Watson is actually at a disadvantage, because the best “Jeopardy!” players regularly hit the buzzer as soon as it’s possible to do so, even if it’s before they’ve figured out the clue. “Jeopardy!” rules give them five seconds to answer after winning the buzz. So long as they have a good feeling in their gut, they’ll pounce on the buzzer, trusting that in those few extra seconds the answer will pop into their heads. Ferrucci told me that the best human contestants he had brought in to play against Watson were amazingly fast. “They can buzz in 10 milliseconds,” he said, sounding astonished. “Zero milliseconds!”
This anecdote highlights one of the most impressive talents of the human mind. We don’t just know things – we know we know them, which leads to feelings of knowing. I’ve written about this before, but one of my favorite examples of such feelings is when a word is on the tip of the tongue. Perhaps it occurs when you run into an old acquaintance whose name you can’t remember, although you know that it begins with the letter “J.” Or perhaps you struggle to recall the title of a recent movie, even though you can describe the plot in perfect detail.
As this exercise shows, our memory is more than simply a database of facts. We also catalogue that we have the information somewhere. In some sort of ‘meta’ fashion, our brain remembers that we learned a fact, even if it can not remember the fact itself.
This kind of makes sense since in the wild, you do not necessarily have the seconds needed to retrieve complete memories and facts. Simply having a gut feeling that there is a useful climbing tree nearby may be enough to start running and may provide valuable milliseconds to keep one alive.
But it is fascinating to think that for many memories that not only is the data stored somewhere but also there is also another location that remembers that we have that memory. I imagine that is why mnemonics that can be so helpful. They help us actualize the meta-memory, allowing us to recover it faster. Perhaps some of them are useful, not because they produce the memory as much as make it easier to access the meta-memory trail.
This may take IBM quite some time to deal with because it essentially requires a lot of pre-work done on every memory, something that may be difficult for computers.
So, while the computer may be able to be the average person, it has a hard time with real Jeopardy champions, most likely because they have very well-developed meta-memory.