wizard of oz support throughout an iterative design...

© 2005 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for

creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained

from the IEEE.

For more information, please see www.ieee.org/portal/pages/about/documentation/copyright/polilink.html.

MOBILE AND UBIQUITOUS SYSTEMS www.computer.org/pervasive

Wizard of Oz Support throughout an Iterative Design Process

Steven Dow, Blair MacIntyre, Jaemin Lee, Christopher Oezbek, Jay David Bolter, and Maribeth Gandy

Vol. 4, No. 4

October–December 2005

This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright

holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be

reposted without the explicit permission of the copyright holder.

18 PERVASIVEcomputing Published by the IEEE CS and IEEE ComSoc ■ 1536-1268/05/$20.00 © 2005 IEEE

R A P I D P R O T O T Y P I N G

Wizard of Oz Supportthroughout an IterativeDesign Process

The Wizard of Oz prototypingapproach, widely used in human-computer interaction research, isparticularly useful in exploring userinterfaces for pervasive, ubiquitous,

or mixed-reality systems that combine complexsensing and intelligent control logic. The vastdesign space for such nontraditional interfacesprovides many possibilities for user interaction

through one or more modalitiesand often requires challenginghardware and software imple-mentations. The WOz methodhelps designers avoid gettinglocked into a particular designor working under an incorrectset of assumptions about user

preferences, because it lets them explore and eval-uate designs before investing the considerabledevelopment time needed to build a completeprototype.

As with other throwaway tools, most WOzinterfaces aren’t conceived to evolve with the sys-tem. So, designers tend to use WOz studies once(or perhaps twice) during a system’s evolution, insharp contrast to other evaluation methods thatmight be used repeatedly as the system evolves.This is unfortunate, as WOz studies can helpdesigners evaluate partially complete systems andadvance the underlying technology’s design.

One reason for the method’s limited use is theeffort required to engineer a successful WOzinterface and integrate it with an incomplete sys-

tem. To address this problem, we built explicitWOz support into the Designer’s AugmentedReality Toolkit,1 a design environment for aug-mented and mixed-reality systems. Becausedesigners using DART will find it easier to createand reuse WOz interfaces throughout the designcycle, they’ll be more likely to use this evaluationmethod.

Based on our work and our survey of the liter-ature (see the sidebar), we believe opportunitiesexist to better exploit the WOz strategy. This arti-cle expands on our initial work2 by introducinga preliminary framework for wizard roles, pro-viding more detail about wizard tool support inDART, and supporting post-WOz data analysis.If WOz tools become a fluid component of a nat-ural design process, wizards can take on variousroles with differing levels of responsibility forfacilitating interaction between computing sys-tems and users.

WOz roles in iterative designInteractive system design typically involves an

iterative process of brainstorming, prototyping,development, user testing, and evaluation. Thisisn’t a clear-cut process; it often iterates throughmany cycles before reaching a final system. Fig-ure 1 illustrates how we can use WOz through-out the design process, as a system evolves. Inpractice, the development cycle is much morecomplicated, including the likelihood that thevision for the system will change at various points,altering the technology development curve.

A case study of a location-aware audio experience using the authors’DART design environment shows how the Wizard of Oz prototypingmethod can work throughout an iterative design process.

Steven Dow, Blair MacIntyre,Jaemin Lee, Christopher Oezbek,Jay David Bolter, and Maribeth GandyGeorgia Institute of Technology

In WOz studies, a wizard operatorgenerally plays some role in a work-in-progress computer system—he or shemight simulate sensor data, contextualinformation, or system intelligence. Weuse the complementary word puppet torefer to the mocked-up user experiencethat the wizard controls through the wiz-ard interface. During a WOz-based eval-uation, the wizard essentially becomes acrutch for simulating the envisionedinterface and interactions before the sys-tem works entirely. As technology devel-opment progresses, the gap filled by thewizard shrinks.

Traditionally, a wizard operator playsone of two roles. A controller fully sim-ulates an unbuilt system component(perhaps a sensor or system intelligence),whereas a supervisor oversees a work-ing system and, if necessary, overridesdecisions that the system or user makes.The blue and red dots in figure 1 repre-sent examples of previous work with theWOz method.

A less common role, that of modera-tor, lies between the controller andsupervisor roles. When a technology orsystem component is working but is stillnot trusted, instead of hooking it into thesystem, you can take on the moderatorrole to observe this component’s outputand then send that output as input to therest of the system. However, because themoderator can override the system com-ponent or sensor before the outputreaches the rest of the system, a moder-ated evaluation can still give you the

envisioned user experience. This lets youobtain useful evaluation feedback whilecollecting a rich set of metadata for eval-uating the development of the systemcomponent or sensor in question (assum-ing the system logs the evaluation ses-sion, and the wizard operator implicitlytags the log when he or she either fol-lows the action the component suggestsor overrides it). By using the moderatorrole to ensure that the user receives theenvisioned experience, you can performa much more realistic evaluation despitethe underlying system behaving unex-pectedly (perhaps behaving erraticallyor completely failing). By analyzingwhat was happening when the wizarddiverged from system decisions, the eval-uation might also uncover false designassumptions. Conducting WOz studiesat intermediate stages of system devel-opment helps refine the user interactionwhile informing the technology devel-opment, especially if you collect user

interaction and wizard operator inputdata during the moderated experience.

Although we’d like to claim that thewizard’s role evolves naturally from con-troller to moderator to supervisor witha progressively diminishing cognitiveload, the design process isn’t thisstraightforward. Nor do these descrip-tions represent an exhaustive list of spe-cific roles; rather, we intend them toreveal the spectrum of possibilities forWOz prototyping.

WOz support tools in DART We built DART for prototyping a

wide range of applications in aug-mented reality, mixed reality, and ubiq-uitous computing. Our goal was tomake it easier for designers to integratelive video, tracking technology, andother sensor data within new mediaapplications. One key decision was tointegrate these emerging technologiesinto an established and familiar com-

OCTOBER–DECEMBER 2005 PERVASIVEcomputing 19

Stat

e of

tech

nolo

gy

Envisioned system

Finalsystem

Current implementation

Time

Initial prototype

WOz

WOz

WOzWOz

Figure 1. One general view of design evolution and the role WOz simulationcan play in facilitating evaluation of anenvisioned experience. The WOz methodhas traditionally been used early to simulate undeveloped technology (examples indicated as red dots) or nearthe end of design, when the wizard cansupervise a full implementation or simulate a small piece of the technology(blue dots). We can also use WOz simulation in the middle ground as ameans for transitioning to a final design.

Our goal was to make it easier for designers to

integrate live video, tracking technology, and

other sensor data within new media applications.

mercial media design environment,Macromedia Director.

DART comprises

• a Director extension called the DARTXtra, written in C++, to handle low-level interaction with sensors, track-ers, and cameras; and

• behaviors, written in Director’s cus-tom scripting language Lingo, formanipulating common high-leveldesign features.

DART behaviors are abstractions of sys-tem entities that can be configured usingsimple parameter windows or cus-tomized by modifying the Lingo code

directly. We seek to support learnabilitythrough these accessible high-levelbehaviors that help familiarize designerswith the environment and its constraints,and we hope to sustain learning by let-ting designers peel away layers ofabstraction and work directly with codeand raw data. At its core, DART imbuesthe philosophy of rapid prototyping anddesign iteration by enabling designers toflexibly coevolve application logic andmedia content.

Event architecture and networkingDART uses a simple event broad-

cast/subscription model (we use theabstractions of cues and actions) to com-

municate between behaviors. A cue fireswhen changes happen in the system: ahigh-level user interaction (for example,a position cue—“user near site A”) or achange in internal state (for example, atime cue—“2 minutes elapsed”). DARTexecutes an action in response to a par-ticular cue being fired, typically changingmedia content or the application state.The system labels cues and actions usingunique event names such as myEvent1;the designer can set up cues to respondto sensor values (for example, when theGPS device returns a value within somerange) and link them to output actions(for example, playing sound file A). Anaction must subscribe to a cue, using the

20 PERVASIVEcomputing www.computer.org/pervasive


WOz techniques have been used in various ways, depending on

the technologies available and the project’s specific goals.

Its most common role in human-computer interface develop-

ment places responsibility on the wizard to fully control a missing

piece of the technology infrastructure (from simulating the entire

system to simulating one sensor). Used early in design as a light-

weight prototyping method, the WOz method presents users with

rough sketches of interface ideas, even when it’s unclear what the

underlying technology should be. In their study, Scott Hudson

and his colleagues wanted to understand users’ willingness to be

interrupted in an office environment.1 Their WOz study helped

them identify appropriate sensors for the space. Stephen Voida

and his colleagues used WOz to study basic interaction techniques

for a projector/camera-based augmented-reality environment.2

They wanted to understand user behavior and preferred modes of

interaction unconstrained by technology-imposed limitations,

such as special gloves or highly constrained movements to aid a

computer vision subsystem.

Further in the design process, after identifying appropriate tech-

nologies, you still might find them too difficult to implement,

especially for testing speculative user interface issues. In the tool

Topiary, the wizard plays the role of a location sensor (for example,

a GPS) during the design of location-based applications.3 In Kent

Lyons’ evaluation of dual-purpose speech, the WOz method

enabled him to explore a larger portion of the design space by

analyzing unrestricted speech input for novices.4 Quan Tran and

his colleagues used a wizard to simulate vision technology during

the development of the Cook’s Collage, a memory aid for elders in

a kitchen environment.5 Using WOz, prototypes such as this can

mature into sophisticated applications that help researchers test

interaction theories without spending time over-engineering a

complex underlying system that might not be needed in the final

product.

Even in finished applications where the system does most of the

work, a wizard can play a variety of roles. In Alice’s Adventures in

New Media, a wizard operator simulates a gesture recognizer as

input to a sophisticated, agent-based narrative engine as part of

an augmented-reality experience.6

All these examples position the wizard as a controller, whether

it’s early or late in the design process. In the mixed-reality perfor-

mance Desert Rain, the wizard plays the supervisor role, helping

participants through the experience on a case-by-case basis.7 Wiz-

ards can also play a dedicated role, add intelligence beyond the

current possibilities of computing, or simply monitor an

experience to help when problems arise, much like an amusement

park ride operator.

Other authors have pointed out general considerations for WOz

studies. An early article by Nils Dahlback and his colleagues re-

vealed a need for careful design of WOz simulations in natural-lan-

guage dialogue systems (although their observations are generally

applicable); researchers must pay attention to nonsimulated parts

of the system, the constraints of the user task, and guidelines for

the wizard.8 As Lennart Molin indicates, user awareness of correct

and incorrect wizard behavior can taint an evaluation and compro-

mise the results.9 Reflecting on their use of the WOz method, David

Maulsby and his colleagues state that a designer benefits greatly by

becoming the wizard operator and that formal models for the sys-

tem’s behavior are necessary for keeping the simulation honest.10

In the Suede system for prototyping and evaluating speech user

interfaces, Scott Klemmer and his colleagues reveal the need for

The Wizard of Oz in HCI

cue’s event name to complete the broad-cast/subscription connection. Multiplecues might trigger one action, or multi-ple actions might subscribe to one cue(see figure 2a). This loose coupling facil-itates rapid prototyping; for example,you can replace an action that’s initiallytriggered using a keyboard cue with anexternal device button cue or a moresophisticated cue. Our earlier workdetails DART’s networking and distrib-uted shared memory system.1

WOz prototyping toolsThe WOz tools in DART leverage the

event broadcast/subscription architec-ture. To enable WOz communication,

the designer adds the “Puppet of Oz”behavior to the DART application onthe machine that is delivering the userexperience and adds the “Wizard of Oz”behavior with the puppet machine’s IPaddress to the DART application on aremote wizard machine. The two high-level behaviors establish the networkingconnection and enable the wizardmachine to trigger any actions currentlyavailable in the user application by sim-ply firing actions locally using the samecue names. The WOz tools employ twoforms of communication: cue broadcast(from wizard to puppet) and action listnotification (from puppet to wizard) (seefigure 2b). By using common naming

conventions, the designer can triggercues locally or through a remote wizard.

Leveraging the continuous notificationof available actions on the puppet, DARTcan automatically generate a WOz inter-face consisting of GUI buttons that cor-respond to the list of possible actions onthe puppet. DART maintains an actionsubscription list on the puppet during run-time, forwarding subscription list changesto the wizard (events can be added andremoved at runtime). The wizard gener-ates a generic button interface (using asimple layout algorithm) and labels eachbutton with the unique event name. Asthe puppet application runs, the wizardautomatically refreshes the correspond-


simulating system error to realistically evaluate user performance

during WOz studies.11 The work on Suede supports our argument

that WOz studies should simulate the envisioned interaction, even

when generating the envisioned system requires modification to

the wizard input. These considerations all point to the need for

careful and appropriate experimental design in WOz simulations.

We should plan WOz user studies to answer existing design ques-

tions, and the wizard’s role in the experimental design should be

well defined and consistent.

Several research projects emphasize WOz simulation in develop-

ment environments, but generally limit the design space to one

interface domain (such as speech recognition in the Suede tool or

location simulation in Topiary3,10). The Neimo project developed a

WOz testing environment for studying multimodal systems where

one or more wizard operators can supplement missing system func-

tions.12 Our work on DART focuses on the rapid prototyping of

more general applications in mixed-reality and ubiquitous comput-

ing.13 While DART isn’t appropriate for all pervasive computing sys-

tems, it illustrates how a judicious choice of programming model

can enable WOz tools to be integrated into a general-purpose

design environment.

REFERENCES

1. S. Hudson et al., “Predicting Human Interruptibility with Sensors: A Wiz-ard of Oz Feasibility Study,” Proc. SIGCHI Conf. Human Factors in Comput-ing Systems (CHI 03), ACM Press, 2003, pp. 257–264.

2. S. Voida et al., “A Study on the Manipulation of 2D Objects in a Projec-tor/Camera-Based Augmented Reality Environment,” Proc. SIGCHI Conf.Human Factors in Computing Systems (CHI 05), ACM Press, 2005, pp.611–620.

3. Y. Li, J. Hong, and J. Landay, “Topiary: A Tool for Prototyping Location-Enhanced Applications,” Proc. ACM Symp. User Interface Software andTechnology (UIST 04), ACM Press, 2004, pp. 217–226.

4. K. Lyons, Improving Support of Conversations by Enhancing Mobile Com-puter Input, doctoral dissertation, College of Computing, Georgia Inst.of Technology, 2005.

5. Q. Tran and E. Mynatt, What Was I Cooking? Towards Deja Vu Displays ofEveryday Memory, tech. report GIT-GVU-TR-03-33, Georgia Inst. Tech-nology, 2003.

6. B. MacIntyre et al., “Augmented Reality as a New Media Experience,”Proc. Int’l Symp. Augmented Reality (ISAR 01), IEEE CS Press, 2001, pp.197–206.

7. B. Koleva et al., “Orchestrating a Mixed Reality Performance,” Proc.SIGCHI Conf. Human Factors in Computing Systems (CHI 01), ACM Press,2001, pp. 38–45.

8. N. Dahlback, A. Jonsson, and A. Lars, “Wizard of Oz Studies: Why andHow,” Proc. Int’l Workshop Intelligent User Interfaces (IUI 93), ACM Press,1993, pp. 193–200.

9. L. Molin, “Wizard of Oz Prototyping for Cooperative Interaction Designof Graphical User Interfaces,” Proc. SIGCHI Nordic Conf. Human-ComputerInteraction (NordiCHI 04), ACM Press, 2004, pp. 425–428.

10. D. Maulsby, S. Greenberg, and R. Mander, “Prototyping an IntelligentAgent through Wizard of Oz,” Proc. ACM SIGCHI Conf. Human Factors inComputing Systems (CHI 93), ACM Press, 1993, pp. 277–284.

11. S. Klemmer et al., “Suede: A Wizard of Oz Prototyping Tool for SpeechUser Interfaces,” Proc. ACM Symp. User Interface Software and Technol-ogy (UIST 00), ACM Press, 2000, pp. 1–10.

12. S. Balbo, J. Coutaz, and D. Salber, “Towards Automatic Evaluation ofMultimodal User Interfaces,” Proc. Int’l Workshop Intelligent User Inter-faces (IUI 93), ACM Press, 1993, pp. 201–208.

13. B. MacIntyre et al., “DART: A Toolkit for Rapid Design Exploration ofAugmented Reality Experiences,” Proc. ACM Symp. User Interface Soft-ware and Technology (UIST 04), ACM Press, 2004, pp. 197–206.

ing set of buttons. By supporting auto-matic wizard generation in a high-levelbehavior (WizardButtonAuto), we aimto lower the barrier for using WOz pro-totyping as an evaluation strategy.

You can also customize the wizardinterface to control the puppet usingbuilt-in Director widgets and DARTbehaviors. For example, you can createa custom button in Director and thenplace a WizardButton behavior on topof the graphical element (one parame-ter of the behavior is a specific actionname to launch on the puppet). Or, youcan place an overhead map image in theapplication and attach a MapTrackingbehavior that generates synthetic GPSreports, which appear to the puppetapplication to be real GPS reports. Youcan program your own behaviors inLingo, taking advantage of the net-working infrastructure and DARTarchitecture to tailor the wizard inter-face. The wizard can send cues locallyto subscribers or broadcast them onlyto the puppet.

One strategy we’ve used is to integratepart of the puppet user interface into thewizard interface so that the wizard oper-ator will experience the same application

state as the user. In The Voices of Oak-land project, described later, we use thisstrategy to let the wizard listen to thesame audio segments as the user, fol-lowing along with the user and decidingwhat content to display. Because youdevelop both interfaces in DART, andbecause DART uses a model of inde-pendent actors communicating viaevents, you can easily port any codedeveloped on either the wizard or thepuppet interface to the other.

Visualization toolsDART supports the ability to capture

data from video cameras, trackers,analogs, buttons, as well as wizard inputfor later analysis and playback. DARTstores the time-synchronized data inDirector casts (collections of Lingoscripts, text files, and other media inDirector), which can be imported laterinto any DART application. Using inte-grated playback facilities in DART, youcan replay the sensor data as it happenedduring the experiment. The applicationlogic responds to replayed data just as itwould to live data, so in playback mode,you’ll perceive exactly what the user per-ceived during the experience.

Replaying an experience is one visual-ization strategy; DART also includes toolsfor visualizing data textually and graph-ically. Observers simply show a text print-out of the data based on current time.DataGraphs behaviors format the replaydata into a graphical image. These high-level behaviors require you to specify cer-tain parameters, such as the specific replaydata set and the graph boundaries (suchas axis assignment, maximum values, andsize of data points). You can attach mul-tiple DataGraphs behaviors to the sameimage so that you can visualize multipledata streams on one graph. DART’s visu-alization tools can show both static,cumulative data as well as dynamic datavalues at any particular time on theabstract clock. The abstract clock can becontrolled independently from Director’sclock by DART’s TimeSlider behavior orusing custom widgets. For the Voices ofOakland experience, we visualized eachparticipant’s GPS data on the same imageto get an overview of user movement(described in more detail later).

Although you can move the numberscollected from WOz experiments toMatlab, Excel, or another commongraphing tool, integrating these facilities



Puppet of Oz (user) Wizard of Oz (remote operator)

Cues Actions Cues Actions

myEvent1

myEvent3

myEvent2

myEvent3

myEvent1

myEvent2

myEvent2

myEvent3

myEvent1

myEvent2

myEvent2

myEvent3

myEvent1

myEvent3

myEvent2

myEvent3

Cue broadcast

Action listnotification

Cues (such as timeCue,positionCue, and buttonCue)

(b)(a)

Cues Actions

myEvent1

myEvent3

myEvent2

myEvent3

myEvent1

myEvent2

myEvent2

myEvent3

Actions (such as startMediaAction,and pauseMedia)

Automatic wizardgeneration

DART

Eve

nt a

rchi

tect

ure

Figure 2. (a) A standard event-based architecture in DART. (b) Integration of WOz tools into the event-based model. Cues fired onthe wizard machine are broadcast locally and to the puppet. The puppet machine notifies the wizard about available actions at runtime to automatically generate a basic button UI on the wizard interface.

in the DART environment has certainbenefits. One advantage is the ability tovisualize live data in parallel with previ-ously collected data, enabling real-timeanalysis of user performance. Embeddedvisualization takes advantage of DART’sabstract clock controls. DART replaysdata on an abstract clock that you canpause, unpause, set to a particular time,rewind, fast-forward, and so on. By sup-porting lightweight visualization toolswithin the design environment, you canperform new sorts of analysis whilemaintaining consistency with the rest ofthe design environment.

Case study: A mixed-realityexperience in the OaklandCemetery

A case study of the Voices of Oaklandproject, an audio location-based experi-ence in Atlanta’s historic Oakland Ceme-tery, illustrates the value of flexible wiz-ard prototyping and analysis tools.Visitors wander through the space andhear an unfolding drama about Atlanta’shistory through its inhabitants’ voices(see figure 3). We describe our designconsiderations, including issues of storystyle (dramatic vs. commentary), storyarc (linear vs. nonlinear), agency (uservs. system control), medium (visual vs.nonvisual media), and technology (forinstance, location tracking) elsewhere.3

As we described earlier, a wizard canhave three roles or levels of responsibil-ity during a design process. In our firstiteration of the Voices of Oakland expe-rience, the wizard fully controlled theaudio segments the participant heard. Inthe second iteration, we included but-tons for the participant to select content.We routed button presses to a graphicaldisplay in the wizard interface, but themoderator wizard still had to choose theappropriate audio segments on the basisof this feedback. In the third iteration,we handed over full responsibility to the

system for presenting audio based onlocation and user button interaction,with the wizard supervising the experi-ence. The DART WOz tools facilitatedeasy transition between each change inthe wizard interface.

First design iteration: Wizard as controller

In the first generation of the Voices ofOakland project, the wizard operatorwas integral for simulating several par-ticipants’ experience. We started withrough audio content and a vague idea ofwhere and when we wanted those sto-ries to be presented as the user movedthrough the space. To facilitate the con-troller wizard role, we developed twomodes of interaction:

• a map-based interface, where the wiz-ard tracks the user’s position and trig-gers content when the user moves neara red content zone (see figure 4a, left);and

• a button-based interface, where eachbutton triggers a specific audio seg-ment to play (figure 4a, right).

The map-based interface took severaldays to create because we had to con-figure the map image to correspond withthe correct GPS coordinates and link theaudio content to particular locations inthe cemetery. We generated the button-based interface quickly using DART’sWizardButtonAuto behavior.

In our pilot study, we evaluated thewizard interface as well as the participantexperience. Our evaluation involved twowizards (script writers) and two users.Aside from demonstrating how the wiz-ard interface worked, we gave the wiz-ard operators no specific instructions onhow to create the visitor’s experience;they could simulate the GPS tracking ordirectly trigger the media content bypushing buttons. During the study, wefound that the wizard operators preferredthe buttons rather than the map, osten-


Figure 3. A participant experiencing theVoices of Oakland system in Atlanta’sOakland Cemetery.

In our first iteration of the Voices of Oakland

experience, the wizard fully controlled the audio

segments the participant heard.

sibly because it put them closer to thecontent and enabled them to create amore compelling visitor experience.(However, because the location-basedimplementation was fairly crude at thispoint, this observation should be takenwith a grain of salt.) The tour participantsprovided feedback about the type of con-tent and length of audio segments, andwe incorporated many ideas into the nextiteration.

Second design iteration: Wizard asmoderator

After the initial pilot testing, we made

several changes to the experience and thewizard’s role. We divided the script intoshort audio dialogues and organized theminto categories—life stories, history, andarchitecture. To simulate user control, wegave the user a handheld controller withbuttons corresponding to the content cat-egories. The visitor could listen to shortaudio stories within a category and thendig deeper, or choose a different categorywhile at any particular grave site.

The new wizard interface (figure 4b)incorporated Director’s built-in GUI wid-gets and a set of buttons (near the bot-tom of 4b) for guiding the visitor if the

primary instructions were inadequate.We chose to route the user’s button inter-action through the wizard interface andsituate the wizard as a moderator. Wehad questions about the users’ expecta-tions about the interaction, and wewanted to control for interface issuessuch as multiple button pressing whileproviding an engaging experience. Themoderator could see the button interac-tion in the wizard interface (and theuser’s position in the physical space) anddetermine the appropriate content.Although it took longer to create this cus-tom wizard interface, the DART WOztools enabled us to easily coevolve thewizard application to match the new userinteraction and to support the moderatorwizard role.

During an outdoor Fall festival in thecemetery, we informally tested the expe-rience with about 15 members of the pub-lic. For the most part, our four differentwizard operators simply relayed user but-ton presses, indicating that buttons weresufficient for controlling the audio con-tent at each grave. We didn’t record rawGPS data or try to visualize the results, inpart because we weren’t doing a formalevaluation—although we realized laterthat it would have been useful.

Third design iteration: Wizard assupervisor

For the third version of the Voices ofOakland, we allowed participant inter-action with the button device to directlyaffect content changes. The wizard inter-face remained the same (figure 4b), butthe wizard’s role shifted to supervisor.



Figure 4. (a) Wizard interface used forthe first design iteration. The controllerwizard could simulate user position on amap (left) or trigger audio contentdirectly using buttons (right). (b) Wizardinterface used in iterations 2 and 3. Thewizard observes the user’s button interaction in the six circles in the upperleft and activates audio content usingbuttons (story segments on the right,navigational segments at the bottom).

(a)

(b)


The wizard could observe the user’s but-ton presses and override the user’s con-tent choices, if necessary. The wizard stillcommunicated with the system when aparticipant was in range of a particulargrave and controlled the ancillary navi-gational content as needed.

We used DART’s capture facilities torecord all the sensor data (GPS, headrotation, and button presses) and wiz-ard actions during a formal user evalu-ation. Even though we weren’t using theGPS and head rotation data in the appli-cation, we collected the data to help usclose the gap between the wizard-simu-lated experience and a working applica-tion. Using DART’s replay and visual-ization tools, we created interactivegraphs to help analyze the WOz study(see figure 5).

The DART visualization tools are aninitial attempt at supporting WOz studyanalysis. Figure 5 shows multiple par-ticipants’ GPS data above a satelliteimage of the cemetery. For each partici-pant, we could print the head rotationvalues and graph the button interactionover time. We included the TimeSliderin our visualization so that we couldscrub the time values and see dynamicupdates in the graphs. Our visualizationhelped us understand how GPS and headrotation data might be useful in our sys-tem design. (While a graphical view ofthe rotation values might have been use-ful, we opted for a simpler textual rep-resentation until we clearly needed thegraphical view.) More details about thestudy results are available elsewhere.3

During future WOz evaluations, we also

plan to explore visualizing prior partic-ipant data during the study of live par-ticipants for real-time predictive feed-back on user behavior.

Explicit support for the WOzsimulation method is certainlyimportant in pervasive comput-ing prototyping environments.

By providing easy-to-use high-level toolsfor wizard interface creation and datavisualization, we hope to lower the bar-rier for designers and researchers toexplore a variety of potential uses ofWOz simulation throughout the designcycle. We also hope this leads to a morecomprehensive framework of possiblewizard roles.

We learned many lessons during ourexperimentation with WOz simulationin the Oakland project. Clearly, thenature of the application will dictatewhat facilities are needed. The ability toeasily try different WOz approaches wasuseful, given the design space we wereexploring. If we were doing visual aug-mented reality or spatialized audiowhere accurate location was integral, themap-based wizard interface might havebeen more useful, but we might not havehad as many other useful options forwizard controls.

Automatic wizard interface generationappears to be most useful for early

design exploration. We used the auto-matically generated interface during ourearly informal studies but not duringour formal WOz studies, because cus-tomized WOz features were more use-ful. The embedded visualization toolsare well suited for intermediate stagesof design and novel wizard roles, wherewe can learn a lot about users’ preferredinteractions and the technology’s limi-tations. The rapid-prototyping philoso-phy of DART and our WOz toolsencourages designers to explore variousroles with different degrees of responsi-bility throughout a design process.

Over the past decade, researchershave raised many significant considera-tions that designers must be consciousof when designing WOz experiments.In particular, they should design the wiz-ard interface with the wizard operators’perceptual, cognitive, and motor skillsin mind, just as with any user interface.When there’s less demand on the wiz-ard, the wizard can pay more attentionto the user and the environment, per-form necessary activities more easily,and contribute observations to the eval-uation of the experience. For the WOzmethod to be effective, designers mustbe able to bridge the gap between thewizard’s role and actual system imple-mentation. We believe that integratingWOz prototyping tools into the pro-gramming environment is vital to sup-porting this transition.

Figure 5. Visualizing the Voices ofOakland experience with a few DARTbehaviors (DataGraphs, Observers,TimeSlider) inside Director: (a) GPS datafor five participants, with dynamic circlesshowing the users’ position at a particulartime; (b) textual representations of GPSlocation and head rotation data; (c)graph of button interaction over time;(d) TimeSlider for controlling DART’sabstract time.

REFERENCES1. B. MacIntyre et al., “DART: A Toolkit for

Rapid Design Exploration of AugmentedReality Experiences,” Proc. ACM Symp.User Interface Software and Technology(UIST 04), ACM Press, 2004, pp. 197–206.

2. S. Dow et al., “Wizard of Oz Interfaces forMixed Reality Applications,” ExtendedAbstracts SIGCHI Conf. Human Factors inComputing Systems (CHI 05), ACM Press,2005, pp. 1339–1343.

3. S. Dow et al., “Exploring Spatial Narrativesand Mixed Reality Experiences in OaklandCemetery,” Proc. ACM SIGCHI Conf.Advances in Computer Entertainment(ACE 05), ACM Press, 2005, pp. 51–60.

For more information on this or any other comput-ing topic, please visit our Digital Library at www.computer.org/publications/dlib.



the AUTHORS

Steven Dow is a PhD student in human-centered computing in the College of Com-puting at the Georgia Institute of Technology. His research interests include designtools, human-computer interaction, ubiquitous computing, mixed reality, and expe-rience design. He received his MS in human-computer interaction from the GeorgiaInstitute of Technology. Contact him at the College of Computing, GVU Center,Georgia Inst. of Technology, Atlanta, GA 30332-0280; [email protected].

Blair MacIntyre is an assistant professor in the Georgia Institute of Technology’s Col-lege of Computing and the Graphics, Visualization, and Usability Center. His researchinterests include understanding how to create highly interactive augmented-realityenvironments, especially those that use personal displays to augment a user’s percep-tion of his or her environment. He received his PhD in computer science from Colum-bia University. Contact him at the College of Computing, GVU Center, Georgia Inst.Technology, Atlanta, GA 30332-0280; [email protected].

Jaemin Lee is a freelance usability specialist. Her research interests include HCI,ubiquitous computing, and context-aware computing. She received her MS inhuman-computer interaction from the Georgia Institute of Technology. Contact herat 1550 Iron Point Rd., Apt. 221, Folsom, CA 95630; [email protected].

Christopher Oezbek is a PhD student in software engineering at the Free UniversityBerlin. His research interests include documentation processes, API usability and dis-covery, and the open source development process. He received his MS in comput-ing from the Georgia Institute of Technology. Contact him at the Working GroupSoftware Eng., Free Univ. Berlin, Takustr. 9, D-14195 Berlin, Germany; [email protected].

Jay David Bolter is the Wesley Chair of New Media at the Georgia Institute of Tech-nology. Along with the Augmented Environments Lab at Georgia Tech, he is helpingto build augmented-reality systems to stage dramatic experiences for entertainmentand education. He also wrote several books on hypertext, media theory, and digitaldesign. He received his PhD in classics from the University of Toronto. Contact himat the School of Literature, Communication, and Culture, Georgia Inst. Technology,Atlanta, GA 30332-0165; [email protected].

Maribeth Gandy is a research scientist with the Interactive Media Technology Cen-ter at the Georgia Institute of Technology and a doctoral student there with a focuson augmented reality and HCI. Her other research interests include mobile,wearable, and ubiquitous computing; universal design; and computer audio. Shereceived her MS in computer science from the Georgia Institute of Technology. Con-tact her at the Interactive Media Technology Center, Georgia Inst. of Technology,Atlanta, GA 30332-0280; [email protected].

Ensure that your networks operate safely and provide

critical services even in theface of attacks. Develop lasting

security solutions, with thispeer-reviewed publication.

Top security professionals in the field share information youcan rely on:

• Wireless Security • Securing the Enterprise• Designing for Security

Infrastructure Security • Privacy Issues • Legal Issues • Cybercrime • Digital Rights Management • Intellectual Property

Protection and Piracy • The Security Profession • Education

Order your subscription today.

www.computer.org/security/

BE SECURE.

DON’TRUNTHERISK.

BE SECURE.

DON’TRUNTHERISK.

wizard of oz support throughout an iterative design...

Documents