frontiers in user- computer interaction brown university and the nsf science and technology center...
TRANSCRIPT
Frontiers in User-Computer Interaction
Brown University and the NSF Science and Technology Center
for Computer Graphics and Scientific Visualization
June 1999
Andries van Dam
Roadmap for Talk
•Introduction– background and goals of UIs
– pros and cons of WIMP (Windows, Icons, Menus, and Point-and-click) GUIs (Graphical User Interface), e.g., MS Windows
– motivation for post-WIMP UIs
•Post-WIMP interaction: From gesture and speech recognition to Multi-modal and Perceptual UIs (PUIs)
•Research problems
• Note: Interaction is hard to describe with static slides, even with video. You really must experience it…
Computer vs. Human Performance
ComputeCompute GraphicsGraphics
Computing Capacity Human Capacity
t
Goal: increase bandwidth to the brain
UI
t
•None! UIs are a necessary evil
•Want human-style interactions with other humans, intelligent assistants, objects in the real and/or virtual world
– intelligent assistant (IA) may be manifested as an agent with a “social interface”/avatar (see Apple’s Knowledge Navigator video)
– prediction: when IAs become vastly superior to what is available today, they will mediate the majority of transactions
The Ultimate User Interface
Intelligent Assistants: Understand You and Your Context
•Combine continuous speech recognition and natural language understanding, knowledge management and processing
•May have customizable social interfaces/avatars
•May run both synchronously (under direct user control) and asynchronously (communicating with other agents)
Today’s Best UI
•Goal: Transparency
– minimal cognitive processing, relying mostly on motor skills and subconscious perception
– focus on task, not technology (e.g., bicycle, car, and our Sketch examples. MS PowerPoint is a negative example!)
– ease of learning is not the same as ease of use
– context determines how much of an investment the user is willing to make (e.g., playing a musical instrument)
What to Use Moore’s Law for?•Computers are now fast enough for most applications
•Increasingly, we can afford to concentrate on human productivity rather than machine productivity (8 CDs for Office 2000!)
•How? Use compute power to make dramatic improvements in user interfaces and capabilities (new modes of expression)– today, majority of code in desktop productivity
applications is in the GUI (e.g., MS Office), and we’re only tinkering
– need radical departure, e.g., Raj Reddy in the early 80s proposed SILK interfaces -- not yet hereospeech, images, language, knowledge base
History of UIs: Four Generations
1960’s Batch
1970’s Command Line on # CRT
1980-90’s WIMP GUI
1999? Post-WIMP
Each paradigm shift led tolarge increase in user base
DOS Command Line Interface
GUIs Reduce Typing
1984 Apple Macintosh Screen
An Unanticipated and Revolutionary Result of GUIs
Even young children who can’t yet read and write are productive computer users
Advantages of WIMP GUIs
•Xerox PARC’s legacy provides a “standard” => ease of x for users
•But not everyone can or wants to use a mouse...
•Layers of support software => ease of implementation, maintainability
– toolkits
– interface builders
– UIMS
Limitations of WIMP GUI
•Imposes sequential “ping-pong” dialog model: mouse and keyboard input, 2D graphics (sound?) output
– deterministic and discrete
– difficult to handle simultaneous input, even two mice
– pure WIMP doesn’t use other senses: hearing, touch, ...
– >50% of our neurons in visual cortex, but as humans it is very difficult for us to communicate without speech, sound...
•Not usable for immersive VR (e.g., headmounted display or cave) where you are “in” the scene: no keyboard, mouse…
Impedance-matchingLimitations of WIMP GUI
Limited Vision(Flat, 2D)
No Speech
No Gestures
One HandTied BehindBack
Limited Audio
Limited Tactile
Brown’s 2D NotePads: Simple Post-WIMP UIs via Gestures•Both based on today’s tablet technology; visual
plus tactile feedback
•NotePad– for note-taking, annotation
– extend WIMP GUI with appropriate marking menus and gestures
– designed for both stand-alone use and integration into a larger environment such as a Cave
•Musical Notepad– for musical notation and simple composition
– purely gestural
– electronic music paper integrated with full computer editing and MIDI facilities
Environment of Tomorrow: Push/Pull• Driven by Moore’s Law, clever device engineering (e.g.,
micro- and nano-technology) we’ll have a profusion of form factors: – wearable computers, information appliances, e-books (based on
digital paper)
– smart rooms, furniture, clothing, jewelry…(Gershenfeld’s “When Things Start to Think”)
– office/home (inexpensive) Virtual Reality
– micro- and even nano-devices sensors & actuators/“displays”
• Ubiquitous/pervasive computing with many devices per user
– personal computing won’t depend on “the personal computer” – users will be mobile, location- and device-independent, to concentrate on task
• Next century: bionic humans, neural implants; human- machine fusion? (Kurzweil’s “Age of Spiritual Machines”)
Rear-Projected ActiveDesk
UNC Vision of Augmented Reality
Augmented Reality Prototype
MIT’s Wearable Computing Group (1998)
Augmented Reality: LCD Projection (1998)
•Thad Starner, MIT Media Lab, Wearable Computing group
•Micro Optical Corporation, 320x240x8 LCD Display
•High- resolution Virtual Retinal Display (laser-based) at Dr. Furness’ HITLab @ U. of Washington
Post-WIMP User Interface Characteristics (1/2)
•Multiple simultaneous devices, sensory channels, or
humans
– 2-handed, with gesture recognition (e.g., Brown’s Sketching
demos)
– multi-modal UI: multiple integrated and reinforcing modes
ooutput: display + spatial sound
o input: gesture + speech recognition
– perceptual UI: goal is to imitate multi-modal human-human dialog
o transparent, passive sensing -- computer is aware of and models user state (location, position, and possibly even mood) so it can respond intelligently
– multiple humans can collaborate, compete, or be adversarial…
Post-WIMP User Interface Characteristics (2/2)
•High bandwidth, continuous input
– body part tracking (head, gaze, hand...), e.g., for immersive VR (which requires head-tracked, wide-field-of-view, low latency, stereo display)
– gesture and speech recognition => probabilistic disambiguation
oex: handwriting, gesture recognition in 2D and 3D/VR
•Autonomous objects in active world are common
•Examples: “Put that there” demo from the 80s at MIT Media Lab, many VR demos
• UI spectrum
– direct control (direct manipulation, by-example...)
– indirect control (smart X, agents, social interfaces)
• WIMP enhanced by
– 3D GUI “widgets” (e.g., 3D translate/rotate/scale boxes with handles)
– speech & gesture recognition (tablets, wands, gloves, body and scene extraction via video-based computer vision technology…)
– haptics (e.g., force feedback joysticks, augmented mice)
– smart X, agents/wizards
WIMP GUIs Will Be Augmented, Not Replaced
From HCI to HHI (Human-Human Interaction)
Note: each human typically controls many devices and
user interfaces
Multiple, Interconnected Devices and UIs per User (1/3)
•Office application– wall displays + personal notepad that provides private
space in addition to any shared space(s)
– user is video-tracked for identification, location, gaze, gesture
– user’s speech is decoded: continuous speech recognition + natural language understanding + intelligent information processing allows dialog with an intelligent assistant
– furniture: chair is instrumented to help detect posture, adjust to the user’s preferred position
– user may wear prostheses (e.g., hearing aid), health monitors (pulse and blood pressure to monitor stress)
Multiple, Interconnected Devices and UIs per User (2/3)•Health-care application
– scaled-down office computing environment for the home
– prostheses (today, heart pacemakers, hearing aids, cochlear implants, voice boxes, artificial joints and organs…)
– electro-chemical monitors, probes (increasingly less obtrusive)
osmart toilet to monitor bodily wastes
– avatars of
ohealth-care providers
o family and friends, support group
Multiple, Interconnected Devices and UIs per User (3/3)•Telecollaboration for multi-disciplinary industrial
design– immersive VR environment
– emphasis on small team collaboration
– scene composed of CAD objects and objects in the environment, including co-workers
o requires combination of geometry-based rendering (polygons, surfaces) and image-based rendering of geometry acquired via scene extraction based on computer vision techniques
o this is a major paradigm shift in computer graphics
– huge distributed systems problems, e.g., minimizing latency and its effects (cybersickness!)
The Cave for Immersive VR
•8’ cubicle with projection screens as walls and floor
A Vision of Telecollaboration: A Normal Office
Henry Fuchs, chief architect of this vision, in his office at UNC, Chapel Hill
A Vision of Telecollaboration: Overlapped Projected Displays
Synchronized cameras, and projectors use imperceptible structured light to calibrate the room and its contents (depth and surface reflection properties of each pixel)
A Vision of Telecollaboration: Seeing and Manipulating Objects
Designing a new head-mounted display (electro-mechanical-optical team design)
Shared CAD & Visualization
Modeling
Image Generation
Scene Acquisition
Interaction
Networking
Tracking
Rendering
Displays
A Vision of Telecollaboration: “Being There” Together
Near-term Post-WIMP Office VR with Co-workers
Designing for the Future (1/2)
•Unlike with WIMP GUI for desktop applications, “one size fits all” doesn’t work: need many specialized UI’s and styles
•Design space is vastly larger and more complex, and needs a multidisciplinary aproach to create a design discipline– Science: how do humans see/hear/…/understand/remember?
opsychophysics, perceptual psychology, cognitive science, social science, engineering and computer science (how to sense, store, manipulate, integrate and present efficiently)
– Applied arts: Communication (newspapers, magazines, advertising…), Storytelling (art, theater, film and video, opera…), Design (graphic, industrial, architectural…), Education
Designing for the Future (2/2)•3D virtual worlds are far more complex than 2D “worlds,”
and they are not the same as the real world. For example,– physical space and virtual space needn’t have the same metrics
– rules for behavior can be simpler or more complex than real-world
– social interfaces are not the same as people
– cybersickness in VR (latency-induced) isn’t the same as motion sickness in the real world
•We should avoid the trap of designing “horseless carriages” – these are new media, neither the real world nor the 2D desktop
– they will have their own idioms and metaphors, design discipline, and development environments
Some Research Challenges for Post-WIMP Interfaces (1/3)•Much better device technology, greatly improving
– accuracy, comfort, safety (avoid cybersickness)
– both input (e.g., pointers, trackers) and output (e.g., visual and aural displays, haptics for force and tactile feedback)
– non-obtrusiveness, spatial and temporal resolution, robustness (for handling, mutual interference, etc.)
– cost effectiveness
•Metaphors for navigation, selection, manipulation, feedback– some based on the real world, others not (like
teleportation)
– animation for feedback
Some Research Challenges for Post-WIMP Interfaces (2/3)•Augmentation of display with other sensory
channels (e.g., haptics, sound); “unification” algorithms for mutual disambiguation and reinforcement, e.g., for gesture + voice
•Scaling of interaction techniques
•Platform independence (desktop vs. VR; small-team vs. large-scale simulations vs. online communities) – how can we factor out commonality?
•Integration of direct control with indirect control (e.g., through agents)
Some Research Challenges for Post-WIMP Interfaces (3/3)
•Powerful integrated development environment
– visual programming and debugging tools?!?
•Usability studies, especially for longterm effects (negative training, safety...)
•Design discipline for post-WIMP UI design
– interdisciplinary
– based on verifiable theories, metrics
– produce design handbooks with design patterns (rules) and exemplars of good design
Where are We Today?
Resources•Basic UI textbooks – oriented towards traditional
WIMP world but useful foundation material– Laurel, B. “The Art of Human-Computer Interface Design”,
1990– Newman, W. M. and Lamming, M. G. "Interactive System
Design", 1995– Preece, Jenny. "Human-Computer Interaction", 1994– Shneiderman, B. "Designing the User Interface", 1997
•EC/NSF Research Frontiers in Virtual Environments and Human-Centered Computing Workshop (June 1999) report – to be published in ACM Computer Graphics SIGGRAPH
issue, and IEEE Computer Graphics and Applications
•Resources Website: – http://www.cs.brown.edu/research/graphics/UIResources/
– further information about EC/NSF Workshop report publication will be posted here
Acknowledgements
•Bill Buxton (SGI Alias|Wavefront)
•Tom Furness (University of Washington Human Interface Technology Laboratory)
•Ben Shneiderman (University of Maryland)
•Rosemary M. Simpson (Brown University)
•Mathew Turk (Microsoft Research)
•Turner Whitted (Microsoft Research)
•THE END