9/10/04

Dear Sir/Madam:

We would like to submit an interactive installation to the CHI 2005 Interactivity program.

Authors:Bill Tomlinson, Man Lok Yau, Jessica O’Connell, Ksatria Williams, So Yamaoka

Contact information:Bill Tomlinson430A Computer Science BuildingUniversity of California, IrvineIrvine, CA 92697-3425Tel: 949-824-9333Cell: 310-529-6056Fax: 949-824-4056Email: wmt@uci.edu

Title of Submission:The Protohuman Project

Category of Submission:Creative and Artistic Innovation

Thank you for your consideration!

Best regards,Bill Tomlinson and the Social Code Group at UCI

1

The Protohuman Project: A Mobile Interface for Interactingwith Communities of Autonomous Characters

Bill Tomlinson, Man Lok Yau, Jessica O’Connell, Ksatria Williams, So YamaokaACE (Arts Computation Engineering) program

University of California, Irvine430A Computer Science Building / Irvine, CA 92697-3425

wmt@uci.edu

ABSTRACTThis paper presents a novel and intuitive paradigm forinteracting with autonomous animated characters. Thisparadigm utilizes a mobile device to allow people totransport characters among different virtual environments.The central metaphor in this paradigm is that virtual spaceis like land and real space is like water for virtualcharacters. The tangible interface described here serves as avirtual raft with which people may carry characters across asea of real space from one virtual island to another. Byincreasing participants’ physical engagement with theautonomous characters, this interaction paradigmcontributes to the believability of those characters.

Author KeywordsTangible interfaces, Intuitive interfaces, Mobile devices,autonomous characters, interactive animation

ACM Classification KeywordsH5.2. Information interfaces and presentation (e.g., HCI):User Interfaces; I.3.7 Three Dimensional Graphics andRealism: Animation

INTRODUCTIONThe Protohuman Project is a multidisciplinary undertakingseeking to create communities of believable autonomouscharacters (Fig. 1) that inhabit heterogeneous networks ofcomputational devices. In particular, the project isinterested in allowing the characters to break the plane ofthe traditional desktop screen. To this end, we havedesigned an interactive installation featuring a noveltangible paradigm for interacting with the characters. Thisparadigm involves the use of a mobile device, such as aTablet PC or handheld computer, as a “virtual raft” (Fig. 2)by which a character may be transported among several

virtual worlds. By enabling the character on the raft to reactin real time to the raft’s motion in real space, thisinstallation encourages participants to become physicallyengaged with virtual characters. We believe that thisphysical engagement can lead to an increase in thebelievability of the characters.

Figure 1: A community of autonomous characters gathersaround a camp fire in a screen-based virtual island.

Figure 2: These characters may jump from a desktopscreen onto a mobile device with a "virtual raft" on it and

be carried around in physical space.

2

INTERACTIONIn the interactive installation, each of three fixed computerscreens displays a virtual environment inhabited by a smallcommunity of animated characters. These characters exhibitsimple autonomous behavior such as gathering around acentral campfire or approaching people who come up totheir screen. The three screens serve as islands – themembers of one community roam freely on their ownscreen. The islands are separated by a gulf of real space thatis not traversable by synthetic agents without assistance.The primary interaction for human participants is to helpthe characters migrate among the three islands. People cando this by means of a virtual “raft” – a Tablet PC that canbe moved in physical space by a human interactor. Whenthis virtual raft is brought near one of the islands, acharacter may jump off the island onto the raft (Fig. 3).

Once the character jumps on, it then needs to balance on theraft. The raft contains accelerometers that detect the motionof the raft in two axes. The participant may give thecharacter a smooth ride by carrying the raft carefully, ormay simulate stormy seas by tilting and swaying the raft.The smoothness of the ride on the raft affects the emotionalstate of the characters, which is expressed through the styleof their animation. When the person brings the raft up toanother island, the character may then jump off. Throughthis mechanism, the person may determine which charactersshould inhabit each of the different islands.

Once a character has been transported to another island, itbegins to interact with the other individuals on that island.Currently these interactions are very simple; a goal of theproject to date has been to create an interactive platformthrough which researchers may explore autonomous agentcommunication and learning as individuals are introducedto and removed from different communities.

TECHNOLOGYThe technology in this installation builds on a variety ofprevious research projects. The core interface technology is

a form of tangible interface [3], allowing a participant tophysically transport a virtual character. The interface is alsoinspired by toys such as Tamagotchi [1], where virtualcharacters are embodied on handheld devices, and computergames such as Animal Crossing [5], where characters maybe transported from one virtual environment to another on anon-interactive memory stick. The project described here isunique from these predecessors in that it uses 3D graphics,accelerometer data and high-speed networking to createbelievable characters on the mobile device itself.

The characters in this installation use simple machine vision[2] to detect humans and the virtual raft when theyapproach their island. The characters’ autonomous behavioris derived from the work of Bruce Blumberg’s SyntheticCharacters Group at the MIT Media Lab [4]. The system iswritten in Java and uses JOGL for the graphics [6].

INNOVATION AND RELEVANCEA great deal of research has contributed to the creation ofbelievable animated characters. Researchers have madecharacters look more believable with lifelike hair, skin,clothes etc., sound more believable through expressivespeech synthesis, and act more believable through artificialintelligence and autonomous agent techniques. The goal ofthe project described here is to create an interface toautonomous characters that will contribute to theirbelievability as well. By allowing the character to jumpinstantaneously from desktop to mobile device and toanimate similarly on both sides of the jump, the systemmaintains the illusion of believability across theseheterogeneous devices.

ACKNOWLEDGMENTSWe thank the ACE (Arts Computation Engineering)program, the Donald Bren School of Information &Computer Science and the Claire Trevor School of the Artsat UCI and Cal-(IT)2 for supporting this project.

REFERENCES1. Bandai, "Tamagotchi." Japan, 1996.2. S.-C. Cheung and C. Kamath, "Robust techniques for

background subtraction in urban traffic video," VideoCommunications and Image Processing, SPIE ElectronicImaging, San Jose, 2004.

3. H. Ishii and B. Ullmer, "Tangible bits: towards seamlessinterfaces between people, bits and atoms," inProceedings of the SIGCHI conference on Human factorsin computing systems: ACM Press, 1997, pp. 234--241.

4. D. Isla, R. Burke, M. Downie, and B. Blumberg, "ALayered Brain Architecture for Synthetic Creatures,"Proceedings of the International Joint Conferences onArtificial Intelligence (IJCAI), Seattle, WA, 2001.

5. Nintendo, "Animal Crossing." Japan, 2002.6. D. Twilleager, J. Kesselman, A. Goldberg, D. Petersen, J.

C. Soto, and C. Melissinos, "Java technologies forgames," Comput. Entertain., vol. 2, pp. 18--18, 2004.

Figure 3: When the raft comes close to an island,characters may jump on or off it.

The Protohuman ProjectBill Tomlinson, Man Lok Yau, Jessica O’Connell, Ksatria Williams, So Yamaoka

ACE (Arts Computation Engineering) programUniversity of California, Irvine

430A Computer Science BuildingIrvine, CA 92697-3425

wmt@uci.edu

DESCRIPTION OF THE WORKThe Protohuman Project is an interactive installation in which human participants interact withseveral communities of autonomous animated characters. These “protohumans” live on threevirtual islands – one on each of three computer screens arranged around the installation space (seeFigure 1). The protohumans have a variety ofautonomous behaviors. When no real peopleare near their screen-based island, they gatheraround the camp fire and drum together (seeFigure 2). When one or more humanparticipants walk in front of their island, thecharacters stop interacting with each other andwill instead approach the people (see Figure 3).In addition to interacting with the protohumansdirectly through their computer vision systems,participants are able to carry a mobile device,which represents a virtual raft, around theinstallation space. The protohumans are notable to migrate among the three islands withouthuman assistance. When a participant bringsthe virtual raft near one of the islands, aprotohuman from that island may jump onto it(see Figure 4). While the character is on theraft, it has to try to balance and keep fromfalling over when the participant tilts themobile device (see Figures 5, 6 and 7). Theparticipant can then carry the character to adifferent island. Each island has its own firecolor (see Figure 8); the crowns of theprotohumans correspond to the fire on their“birth island”. When the raft is close enoughto this other island, the character can jump off(see Figure 9). It will now begin to interactwith the other characters already residing onthat island (see Figure 10).

In the Chamber at CHI 2005, this installationwill be presented in its fully functional form sothat attendees are able to interact with it. Theinstallation is sufficiently robust that it will notrequire the creators to supervise it. Rather,student volunteers will simply need to switch

Figure 2: A community ofautonomous animated protohumansgathers around a camp fire on one of

the screen-based islands.

Figure 1: A schematic of theinstallation space. Three virtual

islands run on the three monitors, anda human participant carries a mobile

device – the virtual raft.

the battery in the mobile device every fewhours, and insure that the device is not removedfrom the installation space.

WHY IT IS APPROPRIATEThe goal of this project is to demonstrate thatthe interface between people and autonomouscharacters can have a significant impact on thebelievability of the characters. Livedemonstrations in the Chamber will fullyexhibit the interactivity and believability in away that would otherwise be difficult to showin a normal presentation environment.

DESCRIPTION OF SYSTEMThis system presents a novel and intuitiveparadigm for interacting with autonomousanimated characters. This paradigm utilizes amobile device to allow people to transportcharacters among different vir tualenvironments. The central metaphor in thisparadigm is that virtual space is like land andreal space is like water for virtual characters.The tangible interface serves as a virtual raftwith which people may carry characters acrossa sea of real space from one virtual island toanother. By increasing participants’ physicalengagement with the autonomous characters,this interaction paradigm contributes to thebelievability of those characters.

PROBLEM IT ADDRESSESA great deal of research has contributed to thecreation of believable animated characters.Researchers have made characters look morebelievable with lifelike hair, skin, clothes etc.,sound more believable through expressivespeech synthesis, and act more believablethrough artificial intelligence and autonomousagent techniques. The goal of the projectdescribed here is to create an interface toautonomous characters that will contribute totheir believability as well. By allowing thecharacter to jump instantaneously from desktopto mobile device and to animate similarly onboth sides of the jump, the system maintainsthe illusion of believability across theseheterogeneous devices.

Figure 3: When participants walk upto an island, the protohumans

approach them.

Figure 4: When the raft is broughtnear one of the islands, a protohuman

may jump onto it.

Figure 5: The character then needs tobalance on the raft.

RELEVANCE TO CHI COMMUNITYThis installation presents a novel interactionparadigm that is relevant to the CHIcommunity for several reasons. The paradigminvolves a tangible interface, through whichparticipants may become physically engagedwith the characters. The interaction paradigmis intuitive, drawing on participants’understanding of the land/water dichotomy, acentral theme in many different cultures. Theproject demonstrates that an interface mayhave a substantial impact on the believabilityof autonomous characters, making them seemmore real because they share an awareness ofthe same physical space that people inhabit.This project draws inspiration from a range ofprevious research in human computerinteraction, in particular regarding tangibleinterfaces and mobile devices. This project isunique from its predecessors in that it uses 3Dgraphics, accelerometer data and high-speednetworking to create believable characters onthe mobile device itself.

COMMERCIAL STATUSThis installation is an academic prototype, andhas not been commercialized. It wasdeveloped with the support of the ACE (ArtsComputation Engineering) program, theDonald Bren School of Information &Computer Science and the Claire TrevorSchool of the Arts at the University ofCalifornia, Irvine.

EQUIPMENT/SUPPORTFOR INSTALLATIONThis installation runs on the followingequipment:• 3 desktop PCs (Windows 2000/XP,

>=2.4GHz, >=512MB RAM, OpenGLcompatible graphics card with >= 64MBvideo RAM, speakers)

• 3 flat panel displays (>=18”) or projectorsand screens (rear projection only)

• 3 USB webcams (with 3 USB extensioncables, at least 6 feet each)

• 1 Toshiba M200 Tablet PC• Wired network to the three desktops &

wireless network to the Tablet PC• Assorted cables, keyboards, mice, etc.

Figure 8: The camp fire on eachisland is a different color – red, green

and blue.

Figure 7: A screenshot of aprotohuman balancing on the raft.

Figure 6: A participant tilts thevirtual raft.

The authors will provide the Tablet PC andwebcams. We request that the conferenceprovide the three desktop machines, monitors andnetworking equipment. If this equipment requestpresents a significant problem, please contact BillTomlinson (wmt@uci.edu) to discuss otheroptions.

DESCRIPTION OF PRESENTERSThe presentation will be delivered by the first andsecond authors of the project – Bill Tomlinsonand Man Lok Yau.

Bill Tomlinson is an Assistant Professor ofInformatics and Drama at the University ofCalifornia, Irvine, where he teaches in the ACE(Arts Computation Engineering) graduateprogram. He is a researcher and animator ofautonomous computational characters. Previousinteractive projects have been shown atSIGGRAPH (97, 98, 99, 01, 02), Ars Electronica(02), the ZKM Future Cinema exhibition (02-03)and other venues, and have been reviewed byCNN, the Wall Street Journal, SculptureMagazine, Scientific American Frontiers, the LATimes, Wired.com and the BBC. In addition hisanimated film, Shaft of Light, screened at theSundance Film Festival and was distributed bythe Anti-Defamation League in its Anti-Bias/Diversity Catalog. He holds an A.B. inBiology from Harvard College, an M.F.A. inExperimental Animation from CalArts, and S.M.and Ph.D. degrees from the MIT Media Lab.

Man Lok (Simon) Yau is a Computer Scienceundergraduate at the University of California, Irvine. His research interests are artificialintelligence and human-computer interaction. He is a fellow of the Summer UndergraduateResearch Program (SURP) at UC Irvine.

EQUIPMENT/SUPPORTFOR PRESENTATIONThe presenters will provide their own notebook computers. We request that the conferenceprovide a projector and external speakers that may be hooked up to the notebooks.

Figure 10: The protohuman from thered island now begins to interact with

the residents of the blue island.

Figure 9: When the raft is closeenough, the protohuman jumps off of

it onto the blue island.

The Protohuman Project: Presentation OutlineBill Tomlinson, Man Lok Yau, Jessica O’Connell, Ksatria Williams, So Yamaoka

ACE (Arts Computation Engineering) programUniversity of California, Irvine

430A Computer Science Building / Irvine, CA 92697-3425wmt@uci.edu

1. Overview of the WorkPresenter: TomlinsonLength: 6 minutes

1.1. Description of Installation1.2. Land/Water Metaphor1.3. Relevance to CHI community

2. VideoLength: 3 minutes

3. Technical Details3.1. How the Virtual Raft WorksPresenter: TomlinsonLength: 3 minutes

3.1.1. Accelerometers3.1.2. Animation3.1.3. Water effect

3.2. How the Jump from Desktop to Mobile Device WorksPresenter: YauLength: 3 minutes

3.2.1. Networking3.2.2. Animation3.2.3. Timing

4. Demonstration of Character Balancing on RaftLength: 5 minutesPresenters: Tomlinson & Yau

5. Question & AnswerLength: 10 minutesPresenters: Tomlinson & Yau