Wizard of Oz experiment
Encyclopedia
In the field of human-computer interaction, a Wizard of Oz experiment is a research
experiment in which subjects interact with a computer
system that subjects believe to be autonomous, but which is actually being operated or partially operated by an unseen human being.
, human factors, ergonomics
, linguistics
, and usability engineering
to describe a testing or iterative design methodology wherein an experimenter (the “wizard”), in a laboratory setting, simulates the behavior of a theoretical intelligent computer application (often by going into another room and intercepting all communications between participant and system). Sometimes this is done with the participant’s a-priori knowledge and sometimes it is a low-level deceit employed to manage the participant’s expectations and encourage natural behaviors.
For example, a test participant may think he or she is communicating with a computer using a speech interface, when the participant’s words are actually being secretly entered into the computer by a person in another room (the “wizard”) and processed as a text stream, rather than as an audio stream. The missing system functionality that the wizard provides may be implemented in later versions of the system (or may even be speculative capabilities that current-day systems do not have), but its precise details are generally considered irrelevant to the study. In testing situations, the goal of such experiments may be to observe the use and effectiveness of a proposed user interface
by the test participants, rather than to measure the quality of an entire system.
. (His dissertation advisor was the late Professor Alphonse Chapanis, the “Godfather of Human Factors and Engineering Psychology”). Amusingly enough, in addition to some one-way mirrors and such, there literally was a blackout curtain separating Jeff, as the “Wizard”, from view by the participant during the study.
The “Experimenter-in-the-Loop” technique had been pioneered at Chapanis’ Communications Research Lab at Johns Hopkins as early as 1975 (J.F. Kelley arrived in 1978). W. Randolph Ford used the experimenter-in-the-loop technique with his innovative CHECKBOOK program wherein he obtained language samples in a naturalistic setting. In Ford’s method, a preliminary version of the natural language processing system would be placed in front of the user. When the user entered a syntax that was not recognized, they would receive a “could you rephrase that?” prompt from the software. After the session, the algorithms for processing the newly obtained samples would be created or enhanced and another session would take place. This approach led to the eventual development of his natural language processing technique, "Multi-Stage Pattern Reduction". Dr. Ford's recollection was that Dr. Kelley did in fact coin the phrase "Wizard of Oz Paradigm" but that the technique had been employed in at least two separate studies before Dr. Kelley had started conducting studies at the Johns Hopkins Telecommunications Lab.
In that employment the experimenter (the “Wizard”) sat at a terminal in an adjacent room separated by a one-way mirror so the subject could be observed. Every input from the user was processed correctly by a combination of software processing and real-time experimenter intervention. As the process was repeated in subsequent sessions, more and more software components were added so that the experimenter had less and less to do during each session until asymptote was reached on phrase/word dictionary growth and the experimenter could “go get a cup of coffee” during the session (which at this point was a cross-validation of the final system’s unattended performance).
A final point: Dr. Kelley's recollection of the coinage of the term is backed up by that of the late Professor Al Chapanis. In their 1985 University of Michigan technical report, Green and Wei-Haas state the following:
The first appearance of the "Wizard of Oz" name in print was in Jeff Kelley's thesis (Kelley, 1983a, 1983b, 1984a). It is thought the name was coined in response to a question at a graduate seminar at Hopkins (Chapanis, 1984; Kelley, 1984b). "What happens if the subject sees the experimenter [behind the "curtain" in an adjacent room acting as the computer]?" Kelley answered, "Well, that's just like what happened to Dorothy in the Wizard of Oz." And so the name stuck. (cited by permission)
There is also a passing reference to planned use of the "Wizard of Oz experiments" in a 1982 proceedings paper by Ford and Smith.
One fact, presented in Kelley's dissertation, about the etymology of the term in this context: Dr. Kelley did originally have a definition for the “OZ” acronym (aside from the obvious parallels with the 1900 book The Wonderful Wizard of Oz
by L. Frank Baum). “Offline Zero” was a reference to the fact that an experimenter (the “Wizard”) was interpreting the users’ inputs in real time during the simulation phase.
The thinking current among many computer scientists and linguists at the time was that, in order for a computer to be able to “understand” natural language enough to be able to assist in useful tasks, the software would have to be attached to a formidable “dictionary” having a large number of categories for each word. The categories would enable a very complex parsing algorithm to unravel the ambiguities inherent in naturally produced language. The daunting task of creating such a dictionary led many to believe that computers simply would never truly “understand” language until they could be “raised” and “experience life” as humans, since humans seem to apply a life’s worth of experiences to the interpretation of language.
The key enabling factor for the first use of the OZ method was that the system was designed to work in a single context (calendar-keeping), which constrained the complexity of language encountered from users to the extent where a simple language processing model was sufficient to meet the goals of the application. The processing model was a two-pass keyword/keyphrase matching approach, based loosely on the algorithms employed in Weizenbaum’s famous Eliza
program. By inducing participants to generate language samples in the context of solving an actual task (using a computer that they believed actually understood what they were typing), the variety and complexity of the lexical structures gathered was greatly reduced and simple keyword matching algorithms could be developed to address the actual language collected.
This first use of OZ was in the context of an Iterative design
approach. In the early development sessions, the experimenter simulated the system in toto, performing all the database queries and composing all the responses to the participants by hand. As the process matured, the experimenter was able to replace human interventions, piece by piece, with newly-created developed code (which, at each phase, was designed to accurately process all the inputs that were generated in preceding steps). By the end of the process, the experimenter was able to observe the sessions in a “hands-off” mode (and measure the recognition rates of the completed program).
OZ was important because it addressed the obvious criticism:
The answer turned out to be:
In the 23 years that followed initial publication, the OZ method has been employed in a wide variety of settings, notably in the prototyping and usability testing of proposed user interface designs in advance of having actual application software in place.
story, in which an ordinary man hides behind a curtain and pretends, through the use of “amplifying” technology, to be a powerful wizard.
In David Lodge
’s novel Small World
, a university lecturer in English literature
is introduced to a computer program named ELIZA, which he believes is capable of conducting a coherent conversation with him. It transpires that a computer lecturer is operating the computer and providing all the responses.
Research
Research can be defined as the scientific search for knowledge, or as any systematic investigation, to establish novel facts, solve new or existing problems, prove new ideas, or develop new theories, usually using a scientific method...
experiment in which subjects interact with a computer
Computer
A computer is a programmable machine designed to sequentially and automatically carry out a sequence of arithmetic or logical operations. The particular sequence of operations can be changed readily, allowing the computer to solve more than one kind of problem...
system that subjects believe to be autonomous, but which is actually being operated or partially operated by an unseen human being.
Concept
The term Wizard of Oz (originally OZ Paradigm) has come into common usage in the fields of experimental psychologyExperimental psychology
Experimental psychology is a methodological approach, rather than a subject, and encompasses varied fields within psychology. Experimental psychologists have traditionally conducted research, published articles, and taught classes on neuroscience, developmental psychology, sensation, perception,...
, human factors, ergonomics
Ergonomics
Ergonomics is the study of designing equipment and devices that fit the human body, its movements, and its cognitive abilities.The International Ergonomics Association defines ergonomics as follows:...
, linguistics
Linguistics
Linguistics is the scientific study of human language. Linguistics can be broadly broken into three categories or subfields of study: language form, language meaning, and language in context....
, and usability engineering
Usability engineering
Usability engineering is a field that is concerned generally with human-computer interaction and specifically with making human-computer interfaces that have high usability or user friendliness...
to describe a testing or iterative design methodology wherein an experimenter (the “wizard”), in a laboratory setting, simulates the behavior of a theoretical intelligent computer application (often by going into another room and intercepting all communications between participant and system). Sometimes this is done with the participant’s a-priori knowledge and sometimes it is a low-level deceit employed to manage the participant’s expectations and encourage natural behaviors.
For example, a test participant may think he or she is communicating with a computer using a speech interface, when the participant’s words are actually being secretly entered into the computer by a person in another room (the “wizard”) and processed as a text stream, rather than as an audio stream. The missing system functionality that the wizard provides may be implemented in later versions of the system (or may even be speculative capabilities that current-day systems do not have), but its precise details are generally considered irrelevant to the study. In testing situations, the goal of such experiments may be to observe the use and effectiveness of a proposed user interface
User interface
The user interface, in the industrial design field of human–machine interaction, is the space where interaction between humans and machines occurs. The goal of interaction between a human and a machine at the user interface is effective operation and control of the machine, and feedback from the...
by the test participants, rather than to measure the quality of an entire system.
Origin
John F. (“Jeff”) Kelley coined the terms “Wizard of OZ” and “OZ Paradigm” for this purpose circa 1980 to describe the method he developed during his dissertation work at The Johns Hopkins UniversityJohns Hopkins University
The Johns Hopkins University, commonly referred to as Johns Hopkins, JHU, or simply Hopkins, is a private research university based in Baltimore, Maryland, United States...
. (His dissertation advisor was the late Professor Alphonse Chapanis, the “Godfather of Human Factors and Engineering Psychology”). Amusingly enough, in addition to some one-way mirrors and such, there literally was a blackout curtain separating Jeff, as the “Wizard”, from view by the participant during the study.
The “Experimenter-in-the-Loop” technique had been pioneered at Chapanis’ Communications Research Lab at Johns Hopkins as early as 1975 (J.F. Kelley arrived in 1978). W. Randolph Ford used the experimenter-in-the-loop technique with his innovative CHECKBOOK program wherein he obtained language samples in a naturalistic setting. In Ford’s method, a preliminary version of the natural language processing system would be placed in front of the user. When the user entered a syntax that was not recognized, they would receive a “could you rephrase that?” prompt from the software. After the session, the algorithms for processing the newly obtained samples would be created or enhanced and another session would take place. This approach led to the eventual development of his natural language processing technique, "Multi-Stage Pattern Reduction". Dr. Ford's recollection was that Dr. Kelley did in fact coin the phrase "Wizard of Oz Paradigm" but that the technique had been employed in at least two separate studies before Dr. Kelley had started conducting studies at the Johns Hopkins Telecommunications Lab.
In that employment the experimenter (the “Wizard”) sat at a terminal in an adjacent room separated by a one-way mirror so the subject could be observed. Every input from the user was processed correctly by a combination of software processing and real-time experimenter intervention. As the process was repeated in subsequent sessions, more and more software components were added so that the experimenter had less and less to do during each session until asymptote was reached on phrase/word dictionary growth and the experimenter could “go get a cup of coffee” during the session (which at this point was a cross-validation of the final system’s unattended performance).
A final point: Dr. Kelley's recollection of the coinage of the term is backed up by that of the late Professor Al Chapanis. In their 1985 University of Michigan technical report, Green and Wei-Haas state the following:
The first appearance of the "Wizard of Oz" name in print was in Jeff Kelley's thesis (Kelley, 1983a, 1983b, 1984a). It is thought the name was coined in response to a question at a graduate seminar at Hopkins (Chapanis, 1984; Kelley, 1984b). "What happens if the subject sees the experimenter [behind the "curtain" in an adjacent room acting as the computer]?" Kelley answered, "Well, that's just like what happened to Dorothy in the Wizard of Oz." And so the name stuck. (cited by permission)
There is also a passing reference to planned use of the "Wizard of Oz experiments" in a 1982 proceedings paper by Ford and Smith.
One fact, presented in Kelley's dissertation, about the etymology of the term in this context: Dr. Kelley did originally have a definition for the “OZ” acronym (aside from the obvious parallels with the 1900 book The Wonderful Wizard of Oz
The Wonderful Wizard of Oz
The Wonderful Wizard of Oz is a children's novel written by L. Frank Baum and illustrated by W. W. Denslow. Originally published by the George M. Hill Company in Chicago on May 17, 1900, it has since been reprinted numerous times, most often under the name The Wizard of Oz, which is the name of...
by L. Frank Baum). “Offline Zero” was a reference to the fact that an experimenter (the “Wizard”) was interpreting the users’ inputs in real time during the simulation phase.
Significance
The Wizard of OZ method (unlike the eponymous “wizard” in the film) is very powerful. In its original application, Dr. Kelley was able to create a simple keyboard-input natural language recognition system that far exceeded the recognition rates of any of the far more complex systems of the day.The thinking current among many computer scientists and linguists at the time was that, in order for a computer to be able to “understand” natural language enough to be able to assist in useful tasks, the software would have to be attached to a formidable “dictionary” having a large number of categories for each word. The categories would enable a very complex parsing algorithm to unravel the ambiguities inherent in naturally produced language. The daunting task of creating such a dictionary led many to believe that computers simply would never truly “understand” language until they could be “raised” and “experience life” as humans, since humans seem to apply a life’s worth of experiences to the interpretation of language.
The key enabling factor for the first use of the OZ method was that the system was designed to work in a single context (calendar-keeping), which constrained the complexity of language encountered from users to the extent where a simple language processing model was sufficient to meet the goals of the application. The processing model was a two-pass keyword/keyphrase matching approach, based loosely on the algorithms employed in Weizenbaum’s famous Eliza
ELIZA
ELIZA is a computer program and an early example of primitive natural language processing. ELIZA operated by processing users' responses to scripts, the most famous of which was DOCTOR, a simulation of a Rogerian psychotherapist. Using almost no information about human thought or emotion, DOCTOR...
program. By inducing participants to generate language samples in the context of solving an actual task (using a computer that they believed actually understood what they were typing), the variety and complexity of the lexical structures gathered was greatly reduced and simple keyword matching algorithms could be developed to address the actual language collected.
This first use of OZ was in the context of an Iterative design
Iterative design
Iterative design is a design methodology based on a cyclic process of prototyping, testing, analyzing, and refining a product or process. Based on the results of testing the most recent iteration of a design, changes and refinements are made. This process is intended to ultimately improve the...
approach. In the early development sessions, the experimenter simulated the system in toto, performing all the database queries and composing all the responses to the participants by hand. As the process matured, the experimenter was able to replace human interventions, piece by piece, with newly-created developed code (which, at each phase, was designed to accurately process all the inputs that were generated in preceding steps). By the end of the process, the experimenter was able to observe the sessions in a “hands-off” mode (and measure the recognition rates of the completed program).
OZ was important because it addressed the obvious criticism:
Who can afford to use an iterative method to build a separate natural language system (dictionaries, syntax) for each new context? Wouldn’t you be forever adding new structures and algorithms to handle each new batch of inputs?
The answer turned out to be:
By using an empirical approach like OZ, anyone can afford to do this; Dr. Kelley’s dictionary and syntax growth reached asymptote (achieving from 86% to 97% recognition rates, depending on the measurements employed) after only 16 experimental trials and the resulting program, with dictionaries, was less than 300k of code.
In the 23 years that followed initial publication, the OZ method has been employed in a wide variety of settings, notably in the prototyping and usability testing of proposed user interface designs in advance of having actual application software in place.
Fictional references
The name of the experiment comes from The Wonderful Wizard of OzThe Wonderful Wizard of Oz
The Wonderful Wizard of Oz is a children's novel written by L. Frank Baum and illustrated by W. W. Denslow. Originally published by the George M. Hill Company in Chicago on May 17, 1900, it has since been reprinted numerous times, most often under the name The Wizard of Oz, which is the name of...
story, in which an ordinary man hides behind a curtain and pretends, through the use of “amplifying” technology, to be a powerful wizard.
In David Lodge
David Lodge (author)
David John Lodge CBE, is an English author.In his novels, Lodge often satirises academia in general and the humanities in particular. He was brought up Catholic and has described himself as an "agnostic Catholic". Many of his characters are Catholic and their Catholicism is a major theme...
’s novel Small World
Small World: An Academic Romance
Small World: An Academic Romance is a humorous "campus novel" by the British writer David Lodge. It is a sequel to Lodge's 1975 novel, Changing Places....
, a university lecturer in English literature
English literature
English literature is the literature written in the English language, including literature composed in English by writers not necessarily from England; for example, Robert Burns was Scottish, James Joyce was Irish, Joseph Conrad was Polish, Dylan Thomas was Welsh, Edgar Allan Poe was American, J....
is introduced to a computer program named ELIZA, which he believes is capable of conducting a coherent conversation with him. It transpires that a computer lecturer is operating the computer and providing all the responses.