research and outlook of the machine learning and … · technology able to fully immerse the user...

Interaction with Virtual Environments

Emanuele Ruffaldi

Scuola Superiore Sant’Anna, Pisa

February 20th , 2012

© 2012 Scuola Superiore Sant’Anna

Course Details

• Instructor: Emanuele Ruffaldi

• Email: [email protected]

• Office Hours:

– Mondays 15:00-16:00

– Wednesdays: 15:00-16:00

• Course Website:

http://www.percro.org/corsi/iVE12/

mailto:[email protected]



Course Objectives and Requisites

• Objectives

– Explore human-machine interaction with

applications in VE

– Efficient use of interaction devices

– Practical Use

• Prerequisites

– Networked Virtual Environments

• A plus for special effects

– Some programming skills


Course Topics

1. Fundamentals of Interaction – Basics of VE and Interaction

– Capture Techniques (sensing devices from wearable to Kinect)

– Interaction Metaphors (what can be do in interaction)

2. Physics Based Interaction – Physics Simulation (rigid body mostly)

– Elements of Haptics (basics of devices and haptic rendering)

– Manipulation of Objects

3. Multi-user Interaction – Navigation

– Locomotion

– Virtual Humans

– Architectures for Multi-user (complements NVE)


Materials and Tools

• Materials

– Material provided along the course

– No specific book

– http://www.percro.org/corsi/iVE12/

• Tools

– Most action will be performed using XVR

already covered by NVE



VIRTUAL ENVIRONMENTS

AND INTERACTION


What are Virtual Environments?

• A synthetic environment that is elicited by

means of technology

• A VE is a medium

• A VE can be generated by low

technology, even a book

[Ellis 1994]


VE and Virtual Reality?

• Virtual Reality refers to a framework of

technology able to fully immerse the user

mainly in the visual and audio domains

[For the History of VR see NVE Course]


Virtuality Contiuum

[Milgram 1994]

Reality Augmented

Reality

Augmented

Virtuality

Virtual

Reality


Virtuality Continuum and Mobiles

http://research.nokia.com/files/NTI_MAR

A_-_June_2009.pdf

[Milgram 1994]

http://research.nokia.com/files/NTI_MARA_-_June_2009.pdf





VE Interaction

Real

World

Human Operator

Sensors

Effectors

Real

Entities


Real

World

VE Interaction – Full Immersion

Virtual

Environment Human Operator

Sensors

Effectors

Effector

Sensor


Real

World

VE Interaction – Mixing Real with Virtual

Virtual

Environment Human Operator

Sensors

Effectors

Effector

Sensor

Real

Entities


Classic Augmented Reality


Outdoor AR


The ultimate interface


Pillars of Virtual Environments

Presence

Place Illusion

Plausibility

Presence is the

mental feeling of

being in an

environment

Immersion as the

perceptual feeling

of the virtual

environment

Place illusion as the

sensation of being

there

Plausibility as the

understanding that

whas is perceived is

Happening realli

Interaction as the

ability of controlling

the environment


Different roles of Self and Immersion

[Garau 2003]


Characteristics of Interaction

• Sensorial Modality and Extension

• Physicality of the Interaction

– Typically Device Mediation

– More recently Natural Interaction

• Directness of the Interaction

– Direct: we act directly on the entities of the

VE

– Mediated: we act indirectly by means of a

virtual human (avatar) or by means of an

object (proxy)


From Interface to Natural Interaction

• Less control on the environment

• Reduced precision

• …

[VPL Data Suit]


Direct vs Mediated Interaction

It is not just a matter of point of

view, in most of the cases the

mediated interaction induces

chenges in the way the VE is

perceived


Interaction Continuum

No-interaction Device Based Natural Interaction

Desktop Devices Wired Wireless Environmental

Mouse

Joystick

Desktop

Haptics Wearable

Haptics

BCI

Gesture

Recognition

Speech

Recognition Motion

Capture Touch

Screens

Moda

lity


Physically Mediated Interaction

Classic Robot

Teleoperated System

Networked Teleoperation

Haptics in Virtual Environment

Haptic Collaborative

Virtual Environment

Augmented Reality with Haptics


Elements of Interaction

• World Knowledge

• Interface Intelligence

World

Unmodeled Where/What Where+What

Completely

Modeled

Pure

Sensing On board

Processing

World

Model

User

Model


Uses of Interaction in Virtual Environments

• Navigation

– How to move in the VE

• Selection

– How to select object

• Manipulation

– How to modify


FAMILIES OF INTERACTION

DEVICES


Interaction Device Characteristics

• Transmission Delay

• Bandwidth

• Resolution

• Dynamic Range

• Signal/Noise

• Some characteristics are technology

limited, others are perception limited


Tracking Entities

• Organization by Technology

– Mechanical

– Acoustic

– Optical

– Magnetic

– Inertial

– Hybrid


Mechanical Tracking


Acoustic Tracking

• Use time of flight between emitter and multiple receiver

• Triangulation allows to extract position and orientation

• Used also for measuring relative motion of limbs

• They are affected by shadowing, temperature and noise. The most negative feature is latency

– But the absolute position can be integrated with faster inertial trackers


Magnetic Tracking

• Based on electric current induces by moving magnetic fields

• Triangulation of relative position

• Good accuracy and speed

• Low on sensitivity to metallic materials. Limited

• Most know is the Polhemus


Inertial Tracking

• Small electronics measuring acceleration

and rotations

– Recently pushed by the integration in Mobile

Phones

• Affected by Drift and usually integrated

with absolute measures


Optical Tracking • Two major families:

– Marker Based

– Markerless

• Camera requirements: – Single camera

– Stereo and Multiple View Camera

– Depth camera


Body Tracking

Animazoo

XSens

[Vlasic07] CLIP

CLIP


Full Body Locomotion


Motion Feedback


Full Body Tracking


Acquiring the World

I/O Brush, Ishi, MIT WHaT, Pai, British Columbia


Haptics


Hiding Reality

[Cosco 2009]


Integration Issues

• Different modalities can be integrated in a

single multimodal experiences

1) Issues of synchronization

2) Advantage of modality dominance


Brain Computer Interfaces

• Adoption of brain measurements for

controlling a VE or a robotic system


Evoked Potentials vs Motor Imagery

• Evoked Potentials: One of the most applied approaches is the one of Evoked Potentials in which the choice of the user is identified in a set of options by fast flashing all the available options. E.g. P300 or SSVEP

• Motor imagery: use the EEG signals for detecting changes in the electric patterns

• Trends: reduced setup time and complexit of the capturing, improved modeling of the acquired data

[Guger 2010]


Physiological Interfaces

• Advanced body sensing with the support

of Machine Learning. Skinput

[Harrison2010]

CLIP


Body as a Surface – Sixth Sense

CLIP


Tangible Interfaces

• Exploit the possibility of using small physical objects, in most of the cases purposely designed. E.g. also RFID. Everything becomes an interface

• This requires a better model for describing interaction in digital-real world


MVC vs MCRpd Models

[Ullmer 2001]

Model View Controller of GUI MCRpd model of TUI


Challenges of Interaction

• Hardware Challenges – Portability and Natureleness

• Software Challenges – Interfacing the Interface

– Hardware Abstraction

– Application adaptation

• Modality Challenges – Metaphor Selection

– Usability

– Learning Curve

– World Modeling

– Evaluation of Interaction


References

• Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems, 77(12), 1321-1329. The Institute of Electronics, Information and Communication Engineers

• Ellis, S. R. (1994). What are virtual environments? Computer Graphics and Applications, IEEE, 14(1), 17-22. IEEE

• Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos Trans R Soc Lond (B).

• Carrozzino, M., & Bergamasco, M. (2010). Beyond virtual museums: Experiencing immersive virtual reality in real museums. Journal of Cultural Heritage, 11(4), 452-458. Elsevier.

• Guger, C., & others. (2011). State of the Art in BCI Research: BCI Award 2010. Recent Advances in Brain-Computer Interface Systems. Retrieved from http://www.intechopen.com/articles/show/title/state-of-the-art-in-bci-research-bci-award-2010

• Cosco, F. I., Garre, C., Bruno, F., Muzzupappa, M., & Otaduy, M. A. (2009). Augmented touch without visual obtrusion. Mixed and Augmented Reality, 2009. ISMAR 2009. 8th IEEE International Symposium on (pp. 99-102).

• Ruffaldi, E., Tripicchio, P., Avizzano, C. A., & Bergamasco, M. (2011). Haptic Rendering of Juggling with Encountered Type Interfaces. PRESENCE: Teleoperators and Virtual Environments, 20(5), 480-501. MIT Press.

• Cohn, G., Morris, D., Patel, S. N., & Tan, D. S. (2011). Your noise is my command: sensing gestures using the body as an antenna. Proceedings of the 2011 annual conference on Human factors in computing systems (pp. 791-800).

• Harrison, C., Tan, D., & Morris, D. (2010). Skinput: appropriating the body as an input surface. Proceedings of the 28th international conference on Human factors in computing systems (pp. 453-462).

• Ullmer, B., & Ishii, H. (2000). Emerging frameworks for tangible user interfaces. IBM systems journal, 39(3.4), 915-931. IBM.

• Vlasic, D., Adelsberger, R., Vannucci, G., Barnwell, J., Gross, M., Matusik, W., & Popović, J. (2007). Practical motion capture in everyday surroundings. ACM Transactions on Graphics (TOG), 26(3), 35. ACM.

• Garau, M., Slater, M., Vinayagamoorthy, V., Brogni, A., Steed, A., & Sasse, M. A. (2003). The impact of avatar realism and eye gaze control on perceived quality of communication in a shared immersive virtual environment. Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 529-536).

http://www.intechopen.com/articles/show/title/state-of-the-art-in-bci-research-bci-award-2010



















research and outlook of the machine learning and … · technology able to fully immerse the user...

Documents