Supporting Collaborative Awareness in Tele-immersion

byKevin M. Curry

Thesis submitted to the faculty of theVirginia Polytechnic Institute and State University

in partial fulfillment of the requirements for the degree of

Master of Sciencein

Computer Science and Applications

© Kevin M. Curry and VPI & SU, 1999

Acknowledgements VT-CAVE staff:

John Kelso; resident guru, “throw” and “catch”Kent Swartz; the need to study awareness, research partnerDr. Ronald Kriz; advocate, sponsor, mentor

Thesis Committee:Dr. Mary-Beth Rosson (chair); guidance in CSCW, “how is the CAVEa collaborative tool?”Dr. John Carroll; research focus, scenario-based design Dr. Ronald Kriz

Acknowledgements, cont’d. Research Community:

Dr. Jason Leigh; CAVERNsoft, LIMBO, and lots of encouragementDr. Stuart Levy; voice-recognition

Family:Dennis “Pat” Curry; military modeling and simulationCarolyn Curry; encouragement, tolerance, and support

Presentation Outline

1. Introduction to Tele-immersion, Connection w/ CSCW2. Research Background, Motivations, and Goals3. Focus on Collaborative Awareness4. Software5. Conclusion and Comments on Future Work

CHAPTER ONE

Introduction

1. Introduction to Tele-immersion

• Coined in 1996 by Tom DeFanti at an Electronic VisualizationLaboratory (EVL) Workshop“The union of networked VR and video in the context ofsignificant computing and data mining.” [Leigh, et. al., 1997]

• Or, as Leigh describes it:“the [creation of persistent VEs]enabling multiple, globally -situated participants to collaborate over high-speed and high -bandwith networks connected to heterogenous super-computingresources and large data stores.” [‘97, p. 1 of 9]

• Early stages of research focused on system performance

Effects of latency, lag, jitter, etc. on usability

System performance always an issue, but interaction,

communication and collaboration are the issues we

deal with today.

• Well over 100 organizations worldwide, using some kind of Immersive Projection Technology (IPT)

Human factors have quickly grown to include CSCW

The Connection with CSCW

The earlier descriptions of Tele-immersion lead to an obviousconnection with CSCW which researchers can leverage...

Why this is significant: Limited experience with human-factors in Tele-immersion Experience with collaborative virtual environments (CVEs)

gives us a little more to go on, but... Many challenges encountered in Tele-immersion deal with

abstract concepts that have been studied in CSCW(e.g., awareness, persistence, shared manipulation)

Research Scope

• Interaction and usability in the context of CSCW; focuson computer-mediated collaboration via Tele-immersion

• Many candidate domains of use:• Scientific Visualization and Design• Military Simulation and Training• Education and Distance Learning• Architectural and Interior Design

• Perhaps several collaborative issues common among all Collaborative Awareness is definitely one!

Approach to Following Research

1. Extensive survey of literature CSCW, CVEs, Tele-immersion

2. Feedback from CAVE users Seasoned CAVErs, novice CAVErs Programmers, scientists, designers Industry Specialists LOTS of random DEMOS

3. Scenario-based design and Claims Analysis Validated by domain specialists

4. Software to demonstrate concepts CAVE Collaborative Console

CHAPTER TWO

Background, Motivations, & Goals

2. Background, Motivations, Goals

Interest is growing beyond research, into governmentand industry Caterpillar, GM, John Deere, NASA, Searle,

Motorola, Mercedes, Military

Tele-immersion is not without its share of detractors Cost: $1.2M (USD) for early CAVEs

+ maintenance+ application development

Just another multi-media system?

Despite these, and other challenges, the way we interactwith and through computers is changing Tele-immersion is being applied in practice with

increasing numbers

Let’s start tackling fundamental usability issues! VR, by definition, is supposed to enhance interaction

Why Tele-immersion?Some Domain-independent Benefits

• Geographically Distributed Team Global Tele-immersion STAR-TAP, i-Grid, vBNS, Abelene

• Geographically Distributed Resources Remote access to super-computing centers

and one-of-a-kind facilities

• Support for Large, Multi-dimensional Data Sets Unique and necessary visualization techniques

not found in other systems

• High Levels of Presence and Fidelity Visiting “inaccessible” environments

• Uniques Support for Mutiple Views Personal, multiple subjective, shared

• Gesturing and Non-verbal Cues Potential to reduce need for artificial cues

• Physical Attributes (compared to other VR systems) Perceive body more naturally than w/ HMDs Glasses weigh less and are less intrusive

Some Domain-Specific Examples

Figure 1: Collaborative Engineering Environment -

NASA scientists working in front of an I-Wall[Source: ISE homepage, http://ise.larc.nasa.gov]

NASA Intelligent Synthesis Environment (ISE) Collaborative Engineering Environment (CEE)

Scientific Analysis & Engineering

Systems Simulation

Figure 2: NRL Battlefiled Visualization and Simulation

[Source: http://www.evl.uic.edu/cavern/users/nrl.html]

Naval Research Laboratory (above) Searle Pharmaceutical: VRTIME

Architectural & Interior Design

Figure 3: Immersive Interior Design Review (Chris Turner)

[Source: http://www.sv.vt.edu/future/vt-cave/VT/]

Virginia Tech Interior Design Futures Lab Star 2000, Rome Rebuilt

Education & Training

Figure 3: An Avatar in the NICE Garden

[Source: http://www.ice.eecs.uic.edu/~nice]

NICE, Foundations of the Hellenistic World Virtual Cockpit (pilot training)

Configuration of the Collaborative Session

VT-CAVE Set-up Original CAVE with Flock of Birds trackers,

stereoscopic shutter glasses, and wand Immersa-desk w/ trackers, glasses, and wand (and keyboard) CAVE Simulator on SGI Workstations (Octane, O2) -

no trackers, defaults to monoscopic, mouse and keyboardinstead of wand

Other IPTs: VR-CUBE, I-Wall, VisionDome, HMDs

The ìGulf of Executionî in the CAVE(from Norman)

ï The wand will never be much more than a point-and-click,or grab-drag-and-drop device (and usable GUIs are not very pervasive in the community)

ï Some limited success with wireless PDAs [Watsen, et. al.]

and ìpalettesî [Williams, et. al.]

But how realistic are these interaction techniques? Joysticks, machine levers, haptic feedback are more

in line with our goals

CAVERNsoft and LIMBO

CAVERNsoftï Hybrid networking and database APIï TCP/IP, UDP, Multi-cast (reliable & unreliable channels)ï Manages efficient distribution of shared dataï The so-called ìbackboneî for Tele-immersion

LIMBOï Provides basic avatar managerï Load and co-manipulate shared modelsï ìHooksî for internet telephone

Scenarios of Use

CHAPTER THREE

The Focus on Awareness

3. The Focus on Awareness

Collaborative awareness is fundamental to collaborative interaction in Tele-immersion Interaction does not begin until some level of awareness

is established (does not have to be mutual) Cooperation requires mutual awareness - how do I perceive

the actions of others? How do they perceive my actions? Awareness about the environment affects collaboration -

How do I perceive changes in the environment?

CSCW Revisited

ï Collaborative awareness has recognized importance in CSCW Ellis, et. al., 1991 -- ìgroup focus and distraction,î

ìnotificationî Begole, 1998 -- many of his ìGroupware Principlesî

deal with awareness Many others in between...

ï ...Specifically, we are interested in findings related to CVEs Benford, Bowers, Greenhalgh, Snowdon, and others Conceptual models start to mesh with Tele-immersion

The Spatial Model of Interaction [Benford, et. al., ë94, Greenhalgh and Benford, ë97]

Some preliminary observations:ï Awareness achieved by numerous means in multiple media

with individual and combined implicationsï Awareness can be mutual or one-wayï Awareness can have wide or narrow scope of interest and

extended or limited rangeï An individualís desire to be noticed can be enhanced or

restricted, changing they way others perceive that individual

The Spatial Model captures these things nicely...

The main components of the Spatial Model are:

Medium the mode of communication; text, audio, videoÖ

Aura portion of space for which interaction is enabled

and allowed

Focus what the observer is trying to perceive

Nimbus the range at which a person can broadcast his or

her own presence so as to attract or deter attention

Adapters objects that modify aura, focus, and nimbus in

order to customize interaction

Other Terms: [Benford, et. al. ë96]

Transportation - the degree to which user feels removed

from the local, physical environment

Fidelity - the degree to which the VE mimics the real world

Spatiality - ìthe degree to which [VEs] supportÖcontainment,

topology, navigation, and shared frame of reference

Extending the Spatial Model(Extensions, Assumptions, Qualifications)

ï Even inanimate objects have aura focus and nimbus

ï ìInteractionî varies in complexity, from face-to-face to justthe mere sight of another object

ï Intersection of auras only creates a potential for interaction Individual foci and nimbi determine actual interaction

ï The VE may default to mimic true spatiality, but we forseedesires to alterand relax this restriction (i.e., exploit adapters)

Gradient Awareness

Awareness is treated as a scaleable concept comprised of multiplecomponents in multiple media...

Totally Aware - able to perceive all possible cues throughall available media

Totally Unaware - unable to perceive any cues

Partially Aware - some cues available, possibly limited by medium

Global Awareness - able to perceive any part of the world, from anyperspective, at any time

Encapsulated Awareness (local, situational)- restricted, perhaps by

spatiality, to awareness of the particular area being ìoccupiedî

6-Category Classification of Awareness

1. Action Awareness2. Attention Awareness3. Environmental Awareness4. Location Awareness5. Awareness of Presence6. Views

Awareness of Presence

ï 2 halves of presence:

1. Presence of others - does it feel like you are sharing space

with others? How do we know others are present? Embodiment & peripheral indicators (e.g., participant list)

2. Presence in the environment - recall ìTransportationî -

does you feel like an occupant of some virtual space?

We are focusing on the 1st half

(An often maligned word) we generally refer to presence asthe sense of ìbeing there.î

Avatarsï Range from texture-mapped GIFs with minimal tracking to

3D video streams of an actual personï Can be an effective embodiment that fulfills other needs

with respect to awareness

Peripheralsï Participant list is the best example

Soundï Multi-directional sound increases fidelity - accounts for

distance, direction, and amplitude

Attention Awareness

Attention awareness deals with focus & range of perception

ï Some synchronous collaborative utilities have absolutely nobuilt-in support for awareness (e.g., UNIX talk)

ï Bowers, et. al., demonstrated the impact of poor support forattention awareness with a Conversation Analysis of MASSIVE

Seems to be one of the more elusive goals of Tele-immersion We can track head orientation, but we still are not

making ìeye contact.î

Awareness & Synchronization

Switch Management

ï Need to synchronize embodiment with actual focus, and in somecases, even participation [Bowers, et. al., ë96] Switching ìCorpsingî

ï How do we handle cases when remote participants are switchingtheir focus of attention to something in the real world? A switch is really and interruption, is it not? Bowers, et. al., call for ìswitch managementî Not so much an issue of how to switch, but how others are

made aware of the switch How much artificiality (awkwardness) will this impose? Real world interruptions are awkward by nature

Action Awareness

Attention awareness concerns the ability to knowwhat others are doing

Physical gesturing - conveying the idea of ìover thereî naturally

Avatars - changing their appearance to convey userís action state ìMortalsî & ìdietiesî in CALVIN [Leigh, et. al.]

ï This is in the context of synchronous collaboration

Environmental Awareness

Environmental awareness handles change in a dynamic environment

Not the same as environmental presence

ï Change can occur as a result of someoneís actions, or thethe VE might change on itís own (e.g., goes from day to night We can be aware of environmental change without

necessarily being aware of user action (e.g. change overthe course of asynchronous session)

This will come up again under persistent VEs

Location Awareness

Location awareness is knowledge of position in the VE

ï Knowing where you are as well as knowing where others are

One othe the more easily implemented forms of awareness

Spatial location is just 3 floats for (x, y, z)

Orientation can be added for the cost of 3 more floats

ï There are other elements of location that may be of interest Physical location of remote user Location in modelsís time (e.g. CAVE5D)

Views

ï Point-of-view is what visualization is all about

ï 3D immersion provides unique frames of reference forviewing graphical data

ï This can be leveraged in the collaborative environment

We start with a personal view through our own eyes

We can also have multiple subjective views

Camera positions (like in graphics apps)

Or we can have shared views (WISIWYS)

This can be difficult to implement in 3Dbecause of the added orientation

Other Awareness IssuesSynchronous Collaboration and Awarenessï Many conventional systems require artificial forms of

expression - for example, : ^ ) ; ^ ) : ^ (ï Avatars in Tele-immersion, while they donít currently support

these particular forms of expression, do support gesture Pointing, waving, nodding, etc.

Asynchronous Collaboration and Awarenessï Again, environmental change over asynchronous sessions

Persistent Tele-immersion

Navigation and Movementï We have several modes for movement in Tele-immersion,

which are not necessarily natural or realistic ìWalkingî w/ the wand Flying (wand or joystick) Teleporting Tethering

ï Awareness is tied to all of these - especially last 2 Static teleports should be well marked and should tell you

where you are about to go; transition should be smooth Dynamic teleports allow you to jump to any location

Location could be another user - notification!

Social Interaction and Group Behaviorï Items like: floor control, privacy, etiquette

Whiteboard example - who controls the board? Itís fairly obvious in Tele-immersion!

How is privacy conveyed? Person is known to be in environment, but is

broadcasting intention to stay private Person is not even known to be in environment

CHAPTER FOUR

The CAVE Collaborative Console

Input Device

Input TranslationProtocol

Keyboard

Voice

Wand Perl & C progs.

LIMBO

CAVE CollaborativeConsole

Radar

Console Item

MyViewRadar

MapViewRadar

WorldViewRadar

ParticipantList

------------------------

Status Panel--- -------- --------- --- --- -

...

Console Objects - Performer-based (C/C++)•Radar - ThreeSpaceRadar, FloorPlanRadar

•Lists - ParticipantList, ObjectList

Console Utilities•teleport - to location, to person, to object

•tether/detach - to person, to moving object•view - anything from a specified camera_position(FRONT | BACK | LEFT | RIGHT | TOP |

BOTTOM)

Console Utilities•savePath - save (and then load) someone’s

•mediaCapture - record a movie of the session•TAPE - record the session on VHS

Http://bleen.sv.vt.edu/~kcurry/ccc.html

CHAPTER FIVE

Conclusions & Future Work

This thesis continues upon the foundations setby CSCW and CVE research in two ways:1. Supported by the literature survey and local

experience, it offers greater understanding ofthe impact of collaborative awareness on collaborative interaction

2. Software developed in conjunction with the research demonstrates the utility of thisunderstanding

Summary of Results

1. Survey and examination of collaborative awareness in immersive VEs

2. The quality of Tele-immersion can beimproved by recognizing the connection with CSCW and CVEs

3. Basic user requirements for collaborative awarenesscan be easily supported

4. A Tele-immersive application can be designed to supporta flexible range of input devices and an easily customizable interface

5. A new system can be integrated with existing supportsystems while require only the minimally expected changes

References

Abrams, H., Watsen, K., Zyda, M., “Three-tiered Interest Management for Large-scale

Virtual Environments,” _______.

Benford, S., Brown, C., Reynard, G., Greenhalgh, C., “Shared Spaces: Transportation,

Artificiality, and Spatiality,” CSCW ’96: Computer-Supported Cooperative

Work, 1996, pp. 77-86.

Benford, S., Bowers, J., Fahlèn, L., Greenhalgh, C., “Managing Mutual Awareness in

Collaborative Virtual Environments,” Proceedings VRST’94, August 1994.

Bowers, J., O’ Brien, J., Pycock, J., “Practically Accomplishing Immersion: Cooperation

in and for Virtual Environments,” Computer Supported Cooperative Work ‘96,

1996.

Curry, K., “Supporting Collaborative Awareness in Tele-immersion,” Proceeding of the

3rd Annual Immersive Projection Technologies Workshop, 1999, pp. 253-261.

Ellis, C.A., Gibbs, S.J., Rein, G.L., “Groupware: Some Issues and Experiences,”Communications of the ACM, Vo l. 34, No. 1, January 1991, pp. 39-58.

Finholt, T. A., Olson, G. M., “From Laboratories to Collaboratories: A New

Organizational Form for Scientific Collaboration,” Psychological Science, Vo l. 8,

No. 1, January 1997, pp. 28-35.

Goldin, D., “Tools for the Future,” Remarks Delivered at the Annual NASA Employee

Conference, January, 1998.

Greenhalgh, C., Benford, S., “Boundaries, Awa reness and Interaction in Collaborative

Virtual Environments,” Proceedings of the 6th International WET-ICE, June 1997.

Greenhalgh, C., Benford, S., “MASSIVE: A Collaborative Virtual System for Tele-

conferencing,” ACM Transactions of Computer-Human Interfaces, Vo l. 2. No. 3,

Sept. 1995, pp. 239-261.

Grinter, R.E., “From Workplace to Development: What Have We Learned So Far and

Where Do We Go?”, Group ’97: Proceedings of the International ACM

SIGGROUP Conference on Computer-Supported Cooperative Work, 1997, pp.

231-240.

Isaacs, E.A., Tang, J.C., Morris, T., “Piazza: A Desktop Environment Supporting

Impromptu and Planned Interactions,” Computer Supported Cooperative Work ’96,19996, pp. 315-324.

Johnson, A., Leigh, J., DeFanti, T., Brown, M., Sandin, D., “CAVERN: the CAVE

Research Network,” Proceedings of 1st International Symposium on Multimedia

Virtual Laboratory. March 1998, pp. 15-27.

Jones, K.C., Cygnus, M.W., Storch, R.L., Farnsworth, K.D., "A Virtual Reality for

Manufacturing Simulation," Proceedings of the 1993 Winter Simulation

Conference, 1993, pp. 882-887.

Kriz, R.D., Farkas, D., Batra, R.C., “ Integrating Simulation Research Into Curriculum

Modules on Mechanical Behavior of Materials: From the Atomistic to the

Continuum," Presented at the MRS 1998 Workshop, Submitted for review to the

Journal of Materials Education, January 1999.

Leigh, J., Park, K., Kenyon, R., Wong, H., “Preliminary STAR TAP Tele-immersion

Experiments between Chicago and Singapore,” Proceedings HPCAsia’98, 1998.

Leigh, J., DeFanti, T., “CAVERN: A Distributed Architecture for Supporting Scaleable

Persistence and Interoperability in Collaborative Virtual Environments,” Journal

of Virtual Reality Research, Development and Applications, Vol. 2.2, December

1997 (1996), pp. 217-237.

Leigh, J., DeFanti, T., Johnson, A., Brown, M., Sandin, D., “Global Tele-immersion:

Better Than Being There,” Proceedings of ICAT '97, Dec, 1997.

Leigh, J., Johnson, A., DeFanti, T., “CALVIN: an Immersimedia Design Environment

Utilizing Heterogeneous Perspectives,” Proceedings of the IEEE International

Conference on Multimedia Computing and Systems'96, 1996.

Middleton, W.E., McIntyre, III, R.T., O'Keefe, IV, J., "Virtual Reality and Analytical

Simulation of the Soldier," Proceedings of the 1993 Winter Simulation

Conference, 1993, pp. 1048-1052.

Pedersen, E. R., McCall, K., Moran, T. P. , Halsz, F.G., “Tivoli: An Electronic

Whiteboard for Informal Workgroup Meetings,” INTERCHI ’93, pp. 391-398.

Roseman, M., Greenberg, S., “Building Real-Time Groupware with GroupKit, A

Groupware Toolkit,” ACM Transactions of Computer-Human Interaction, Vo l. 3,

No. 1, March, 1996, pp. 66-106.

Roussos, M., Johnson, A., Leigh, J., “The NICE Project: Narrative, Immersive,

Constructionist/ Collaborative, Environments for Learning in Virtual Reality,”

Proceedings of the ED-MEDIA/ED-TELECOM, 1997a, pp. 917-922.

Roussos, M., Johnson, A., Leigh, J., Vasilakas, C., Barnes, C., Moher, T., “NICE:Combining Constructionism, Narrative, and Collaboration in a Virtual LearningEnvironment,” Computer Graphics, Vo l. 31, Num. 3, ACM SIGRAPH, 1997b, pp. 62-63.

Roussos, M., Johnson, A., Moher, T., Leigh, J., Vasilakas, C., Barnes, C., “Learning and

Building Together in an Immersive Virtual World,” Presence, 1997c.

Sagar, M.A., Bullivant, D., Mallinson, G.D., Hunter, P.J., "A Virtual Environment and

Model of the Eye for Surgical Simulation,", SIGRAPH '94: Proceedings of the

21st Annual Conference on Computer Graphics, 1994, pp. 205-212.

Scharm, H., Breining, R. "How the Automotive Industry Uses Immersive Projection

Technology," Proceedings of the 3rd Annual Immersive Projection Technologies

Workshop, 1999, pp. 133-144.

Schiefele, J., Dörr, K., Olbert, M., Kubbat, W., "Stereoscopic Projection Screens and

Virtual Cockpit Simulation for Pilot Training," Proceedings of the 3rd Annual

Immersive Projection Technologies Workshop, 1999, pp. 211-222.

Snowdon, D., Jää-Aro, K., “A Subjective Virtual Environment for Collaborative

Information Visualization,” Virtual Reality Universe'97, April 2-5, 1997.

Watsen, K., Darken, R.P., Capps, M.V. , "A Handheld Computer as an Interaction Device

to a Virtual Environment," Proceedings of the 3rd Annual Immersive Projection

Technologies Workshop, 1999, pp. 51-57.

Wheless, G.H., Lascara, C.M., Leigh, J., Kapoor, A, Johnson, A.E., DeFanti, T.A.,

“CAVE6D: A Tool for Collaborative Immersive Visualization of Environmental

Data,” submitted to IEEE Visualization 1998.

Williams, G., Faste, H., McDowall, I., Bolas, M. "Physical Presence - Palettes in Virtual

Space," Proceedings of the 3rd Annual Immersive Projection Technologies

Workshop, 1999, pp. 65-75.

CITED URLs

1. http://www.ncsa.uiuc.edu/VR/cavernus/users.html

2. http://www.evl.uic.edu/cavern/users/nrl.html

3. http://www.evl.uic.edu/cavern/users/vrtime.html

4. http://www.fhw.gr/fhw/en/projects/3d.html

5. http://www.evl.uic.edu/cavern/cavernsoft/

6. http://www.evl.uic.edu/cavern/cavernsoft/li mbo.html

7. http://ise.larc.nasa.gov/ISE_Overview/sld051.htm

8. http://scv.bu.edu/HiPArt/intro.html

9. http://www.ice.eecs.uic.edu/~nice/

10. http://www.ncsa.uiuc.edu/VR/cavernus

11. http://bleen.sv.vt.edu/~kcurry/voice.html