A Chronicle article – No Computer Left Behind – by Daniel J. Cohen and Roy Rosenzweig of GMU (requires an account), discusses how access to Internet information makes multiple-choice tests redundant, and in that addresses the issue of how to trust information on the web.
Computer scientists have an optimistic answer for worried scholars. They argue that the enormous scale and linked nature of the Web make it possible for it to be “right” in the aggregate while sometimes very wrong on specific pages. The Web “has enticed millions of users to type in trillions of characters to create billions of Web pages of on average low-quality contents,” write the computer scientists Rudi Cilibrasi and Paul Vitányi in a 2004 essay.Yet, they continue, “the sheer mass of the information available about almost every conceivable topic makes it likely that extremes will cancel and the majority or average is meaningful in a low-quality approximate sense.” In other words, although the Web includes many poorly written and erroneous pages, taken as a whole the medium actually does quite a good job encoding meaningful data.
“Good enough” is good enough for multiple-choice tests. Now what we need is a way of comparing all that information to get a good enough answer. Google does it for translations. And George Mason is experimenting with it with H-Bot, a historical software agent. Give it a historical question, it uses a set of algorithms to compare documents concerning the subjects embedded in the question to return an accurate answer.
Right now H-Bot can only answer questions for which the responses are dates or simple definitions of the sort you would find in the glossary of a history textbook. For example, H-Bot is fairly good at responding to queries such as “What was the gold standard?”, “Who was Lao-Tse?”, “When did Charles Lindbergh fly to Paris?”, and “When was Nelson Mandela born?” The software can also answer, with a lower degree of success, more difficult “who” questions such as “Who discovered nitrogen?” It cannot currently answer questions that begin with “how” or “where,” or (unsurprisingly) the most interpretive of all historical queries, “why.” In the future, however, H-Bot should be able to answer more difficult types of questions as well as address the more complicated problem of disambiguation—that is, telling apart a question about Charles V the Holy Roman Emperor (1500-1558) from one about Charles V the French king (1338-1380). To be sure, H-Bot is a work in progress, a young student eager to learn. But given that its main programming has been done without an extensive commitment of time or resources by a history professor and a (very talented) high-school student, Simon Kornblith, rather than a team of engineers at Google or MIT, and given that a greater investment would undoubtedly increase H-Bot’s accuracy, one suspects that the software’s underlying principles are indicative of the promise of the Web as a storehouse of information. (Web of Lies)
My first thought was to the state of knowledge underlying the algorithm. Wouldn’t it take a discerning expert to determine the accuracy of the response? Sure it might. H-Bot provides the target that can be checked by the expert. But after a while, even the expert comes to trust the algorithm.
Cohen and Rosenzweig finally argue that once we trust the factual accuracy of look up information, we can set aside multiple-choice tests and move on to more interesting – deeper and more significant – questions.
Now that newer technology threatens the humble technology of the multiple-choice exam, we have an opportunity to return to some of the broader and deeper measures of understanding in history — and other subjects.
I’d look forward to this because it means we’d have to redesign exams. Change the form of the exam and you change what’s taught.
Listening to Dancing Shoes from “Whatever People Say I Am That’s What I Am Not” by Arctic Monkeys