immersive media vision create seamless immersive environment for distributed, interactive real and...

12
Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses with fidelity approaching the limits of human perception Relevant to any type of human interaction: entertainment, education, communications

Upload: kenia-boatwright

Post on 30-Mar-2015

228 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses

Immersive Media Vision Create seamless immersive environment for distributed,

interactive real and virtual events Reproduce audio, video and other senses with fidelity

approaching the limits of human perception

Relevant to any type of human interaction: entertainment, education, communications

Page 2: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses

IMSC Entertainment Vision:RMI

A fusion of internet browsing with a theater-like immersive experience

HD Video at up to 45 Mbits/sec 10.2 ch Immersive Audio (16 Mbits/sec)

Streaming over the Internet on-demand to a mouse click (20 min prog)

RMI demonstrates IMSC innovations in several critical technical areas:

Immersed in a college football game

Doctors assisting in a remote procedure

Business people negotiating like they

are in the same room

Students visiting an aquarium a thousand

miles away

Streaming media

servers

Immersive audio

capture and

rendering

Network protocols for error

correction Synchronization

Applications

Page 3: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses

Premier Demonstration May 9, 2002

Cross-country streaming from ISI-East in Virginia

NY Times coverage in “Circuits” section

NBC-TV and KTLA http://imsc.usc.edu/rmi/ http://www.east.isi.edu/NGI-

S/

Page 4: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses

Internet2 Fall 2002 Member Meeting

Audio: 10.2 channel, immersive sound systemVideo: HDTV 1920x1080i

New World Symphony, Miami Beach

550 seat USC Bing Theater

Page 5: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses

University of Maryland Server

USC

USC Client

New World Symphony

Server

Georgia Tech Server

RMI Experimental Setup Synchronized immersive audio and HDTV streamed playback

from Yima server over Internet2 Control of end-to-end process: capturing, network interface,

transmission, rendering CENIC 2003 “Gigabit or Bust” Award

Page 6: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses

Remote Master Class

LA Philharmonic cellist located at our USC lab Student at New World Symphony in Miami Beach Three hour session with MPEG2 video and 10.2 immersive audio Teacher reports that student "was really there" with immersive audio Many psychophysical, perceptual and artistic tests to be done

Page 7: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses

Recent Developments...

Two-way audio communication to New World Symphony in Miami Beach, Florida 8-channel audio – low latency

Live video capture/transmission (DVCam) Near term prospects

RMI theater at Inha University (Korea) 2-way live HD-video/immersive audio

streaming for Internet collaboration

Page 8: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses

Distributed Immersive Performance Create seamless immersive interactive environment Distributed musicians, conductor (active) and audience

(passive) Q: Why a musical performance?

A: A distributed orchestra is one of the most challenging immersive, interactive environments

Scenario: Multiple soloists scheduled in different cities Under served areas can be reached

Page 9: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses
Page 10: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses

IMSC Communication VisionInterAct!

UWB console captures 3D model and problem details

PDA provides spatial array of participants and sonic

translation of details

Immersive workstation coordinates video, avatars, graphics, audio, and haptics to facilitate problem resolution

Page 11: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses

Research Highlights

immersive audio multichannel and HRTF approaches - holistic DSP approach

streaming servers distributed and scalable architecture

computer vision computational framework for grouping based on tensor voting, tracking for augmented realities and SFX

graphics & animation 3D DSP mesh processing, compression, mesh operations, hair modeling and animation

multimodal emotive interfaces Speech and dialog, vision sensing of body and hands, facial expressions analysis and expressive avatars

virtual reality -- chair of VR2003 in LA applications to psychology (ADD diagnosis), haptics applications, and user studies

perception & cognitive modeling theoretical foundations for understanding benefits of media immersion

UWB wireless leadership in technology, FCC regulation, and commercialization

IMSC has produced ground breaking fundamental research in:

Page 12: Immersive Media Vision Create seamless immersive environment for distributed, interactive real and virtual events Reproduce audio, video and other senses

Sensory Interfaces (SI)

Robust Vision Systems Segmentation, motion, 3D body and hand tracking, tensor

voting, activity analysis, recognition and tracking Speech Recognition and Synthesis

Emotive dialog, translation, emotion analysis Immersive Audio

Sonic visualization, spatialization, autocalibration Facial Analysis -- Gestures and Animation

Expression signatures for emotion and recognition Video fusion with 3D models for Surveillance

Structure extraction from LiDAR and texture projection Digital Geometry (3D) Processing

Compression, surface and topology smoothing, mesh simplification