VR Inputs and OutputsVR Inputs and Outputs

Output DevicesOutput Devices

VisualVisual AuditoryAuditory HapticHaptic

Visual DevicesVisual Devices

DimensionsDimensions• Stereoscopic/monoscopicStereoscopic/monoscopic• Image resolutionImage resolution• Field of viewField of view• Display techDisplay tech• Ergonomic factorsErgonomic factors• CostCost

Human EyeHuman Eye

126 million 126 million photoreceptorsphotoreceptors

Eye-gaze Eye-gaze technology not technology not useful yetuseful yet• CalibrationCalibration• ErrorError• Overlaying controlOverlaying control

IPDIPD• 53-75 mm53-75 mm

Head-tracked DisplaysHead-tracked Displays

Displays in which the user’s head is Displays in which the user’s head is tracked and the image display screen tracked and the image display screen is located at a fixed location in is located at a fixed location in physical space.physical space.

Examples?Examples?• CRTCRT• Virtual Workbench or ImmersaDeskVirtual Workbench or ImmersaDesk• CAVECAVE• Many large screen displaysMany large screen displays

CRT HTD (Fishtank VR)CRT HTD (Fishtank VR)

3D Glasses

Head Tracker

Stationary Display

Stereoscopic DisplayStereoscopic Display

User

C

AB

Screen

C

A

B

Standard Display Stereoscopic Display

Virtual WorkbenchVirtual Workbench3D Glasses

3D Display

3D Object

CAVECAVE

Any Projected Large Stereoscopic ScreenAny Projected Large Stereoscopic Screen

CharacteristicsCharacteristics

LargeLarge Projection-Based (except for Fishtank Projection-Based (except for Fishtank

VR)VR) StereoscopicStereoscopic Head TrackedHead Tracked Stationary Display Screen(s)Stationary Display Screen(s) Let’s try to identify the pros and cons Let’s try to identify the pros and cons

and application domainsand application domains

3D Glasses

3D Display

3D Object

Large Screen ProjectionLarge Screen Projection

InfrastructureInfrastructure• Front Projection Front Projection

(user occlusion)(user occlusion)• Back Projection Back Projection

(takes up space)(takes up space)

Large Screen ProjectionLarge Screen Projection

AdvantagesAdvantages Viewer not isolatedViewer not isolated CollaborationCollaboration Minimal physical gearMinimal physical gear Good resolutionGood resolution Large field of viewLarge field of view

DisadvantageDisadvantagess One person “sweet-One person “sweet-

spot” spot” Real objects may Real objects may

occlude virtual objectsocclude virtual objects SynchronizationSynchronization LightLight

• ReflectionReflection• BleedingBleeding

Equipment footprintEquipment footprint AlignmentAlignment

Head-Mounted DisplayHead-Mounted Display

Visually Coupled Systems Visually Coupled Systems

Operator uses natural Operator uses natural visual and motor skills visual and motor skills for system controlfor system control

Basic ComponentsBasic Components• An immersive visual An immersive visual

display (HMD, large display (HMD, large screen projection screen projection (CAVE), dome (CAVE), dome projection)projection)

• Tracking system for Tracking system for head and/or eye motionhead and/or eye motion

HMD Optical SystemHMD Optical System

Image SourceImage Source• CRT or Flat Panel (LCD)CRT or Flat Panel (LCD)• Video, See–Through or Video-Pass ThroughVideo, See–Through or Video-Pass Through

HMD Optical HMD Optical SystemSystem

Image SourceImage Source• CRT or Flat Panel (LCD)CRT or Flat Panel (LCD)• Video, See–Through or Video, See–Through or

Video-Pass ThroughVideo-Pass Through

Head-Mounted Displays Optical Head-Mounted Displays Optical SystemSystem

Mounting ApparatusMounting Apparatus• What are some factors?What are some factors?

EyeglassesEyeglasses WeightWeight EarphonesEarphones TrackersTrackers

Field of ViewField of ViewMonocular FOV is the angular subtense (usually expressed in degrees) of the displayed image as measured from the pupil of one eye.

Total FOV is the total angular size of the displayed image visible to both eyes.

Binocular(or stereoscopic) FOV refers to the part of the displayed image visible to both eyes.

FOV may be measured horizontally, vertically or diagonally.

Focal Length & DiopterFocal Length & Diopter

Focal LengthFocal Length - The - The distance from the distance from the surface of a lens at surface of a lens at which rays of light which rays of light converge.converge.

Diopter Diopter - The power - The power of a lens. Equal to of a lens. Equal to 1/(focal length of the 1/(focal length of the lens measured in lens measured in meters)meters)

OcularityOcularityOcularityOcularity MonocularMonocular - HMD - HMD

image goes to only image goes to only one eye.one eye.

Biocular Biocular - Same HMD - Same HMD image to both eyes.image to both eyes.

Binocular Binocular (stereoscopic) (stereoscopic) - - Different but matched Different but matched images to each eye.images to each eye.

IPDIPD

Interpupillary Interpupillary Distance (IPD)Distance (IPD)

IPD is the horizontal IPD is the horizontal distance between a distance between a user's eyes.user's eyes.

IPD is the distance IPD is the distance between the two between the two optical axes in a optical axes in a binocular view binocular view system.system.

Vignetting and Eye ReliefVignetting and Eye Relief

VignettingVignetting The blocking or The blocking or

redirecting of light rays redirecting of light rays as they pass through as they pass through the optical system.the optical system.

Eye Relief DistanceEye Relief Distance Distance from the last Distance from the last

optical surface in the optical surface in the HMD optical system to HMD optical system to the front surface of the the front surface of the eye.eye.

Basic Eye Basic Eye

Cornea

CrystallineLens

Fovea

RetinaOpticNerve

Properties of the EyeProperties of the Eye

Approximate Field of ViewApproximate Field of View• 120 degrees vertical120 degrees vertical• 150 degrees horizontal (one eye)150 degrees horizontal (one eye)• 200 degrees horizontal (both eyes)200 degrees horizontal (both eyes)

AcuityAcuity• 30 cycles per degree (20/20 Snellen 30 cycles per degree (20/20 Snellen

acuity).acuity).

Simple FormulasSimple Formulas

Visual Resolution in Cycles per degree Visual Resolution in Cycles per degree (V(Vresres) = Number of pixels /2 (FoV in ) = Number of pixels /2 (FoV in degrees)degrees)

Example:Example: (1024 pixels per line)/(2*40 (1024 pixels per line)/(2*40 degrees) = Horizontal resolution of degrees) = Horizontal resolution of 12.8 cycles per degree12.8 cycles per degree

To convert to Snellen acuity (as in To convert to Snellen acuity (as in 20/xx)20/xx)

VVresres = 600/xx (20/47) = 600/xx (20/47)

Optical SystemOptical System

Move image to a distance that can Move image to a distance that can be easily accommodated by the eye.be easily accommodated by the eye.

Magnify the imageMagnify the image

Simple Magnifier HMD DesignSimple Magnifier HMD Design

p

q

Eyepiece(one or more

lenses)Display

(Image Source)

Eye f Image

1/p + 1/q = 1/f where1/p + 1/q = 1/f wherep = object distance (distance from image source to eyepiece)p = object distance (distance from image source to eyepiece)q = image distance (distance of image from the lens)q = image distance (distance of image from the lens)f = focal length of the lensf = focal length of the lens

Virtual Image Virtual Image

Lens Display

VirtualImage

ResolutionResolution

(low) 160 x 120 color pixels per eye(low) 160 x 120 color pixels per eye (current low) 640x480 – 1024x768(current low) 640x480 – 1024x768 (high) 1000+ x 1000+(high) 1000+ x 1000+

Note that resolution and FOV are Note that resolution and FOV are independentindependent

Another important factor: pixel densityAnother important factor: pixel density• Pixels per degree of FOVPixels per degree of FOV

How can we use the make up of the eye to How can we use the make up of the eye to better leverage resolution?better leverage resolution?

Basic Eye Basic Eye

Cornea

CrystallineLens

Fovea

RetinaOpticNerve

LEEP OpticsLEEP Optics

Large Expanse Large Expanse Extra PerspectiveExtra Perspective

Very wide FOV for Very wide FOV for stereoscopic imagesstereoscopic images• Higher resolution in Higher resolution in

the middle of FOVthe middle of FOV• Lower resolution on Lower resolution on

the peripherythe periphery Pincushion Pincushion

distortiondistortion

Fresnel LensFresnel Lens

A lens consisting of a A lens consisting of a concentric series of concentric series of simple lens sectionssimple lens sections

Result is a thin lens with a Result is a thin lens with a short focal length and short focal length and large diameterlarge diameter

More even resolution More even resolution distributiondistribution

Less distortionLess distortion from lanternroom.comfrom lanternroom.com

Distortion in LEEP OpticsDistortion in LEEP Optics

A rectangle Maps to this

How would you correct this?

To correct for distortionTo correct for distortion Predistort imagePredistort image This is a pixel-based This is a pixel-based

distortiondistortion Graphics rendering uses Graphics rendering uses

linear interpolation!linear interpolation! Too slow on most systemsToo slow on most systems Pixel shaders!Pixel shaders! Render to TextureRender to Texture

Distorted Field of ViewDistorted Field of View

Your computational model (computer Your computational model (computer graphics) assumes some field of graphics) assumes some field of view.view.

Scan converter may over or Scan converter may over or underscan, not all of your graphics underscan, not all of your graphics image may appear on the screen.image may appear on the screen.

Are the display screens aligned Are the display screens aligned perpendicular to your optical axis?perpendicular to your optical axis?

Distance alongz-axis

Distorted FoV Distorted FoV (cont.)(cont.)

Compound Microscope HMD Compound Microscope HMD DesignDesign

Relay lens produces a real image of the display Relay lens produces a real image of the display image source (screen) at some intermediate image source (screen) at some intermediate location in the optical train. The eyepiece is then location in the optical train. The eyepiece is then used to produce an observable virtual image of used to produce an observable virtual image of this intermediate image.this intermediate image.

Relay Lens

ImageIntermediateReal Image

Eyepiece

ExitPupil

Exit PupilExit Pupil The area in back of the optics from which The area in back of the optics from which

the entire image can be seen. Important if the entire image can be seen. Important if IPD not adjustable.IPD not adjustable.

Compound microscope optical systems Compound microscope optical systems have a real exit pupil.have a real exit pupil.

Simple magnifier optical systems do not Simple magnifier optical systems do not have an exit pupil.have an exit pupil.

Relay Lens

ImageIntermediateReal Image

Eyepiece

ExitPupil

Virtual Research V8 HMDVirtual Research V8 HMD DisplayDisplay

• Dual 1.3” diagonal Dual 1.3” diagonal Active Matrix LCDActive Matrix LCD

• Resolution per eye: 640 Resolution per eye: 640 x 480x 480

• focal length = 1mfocal length = 1m OpticalOptical

• Field of view: 60° Field of view: 60° diagonaldiagonal

SolveSolve• What is the cycles per What is the cycles per

degree?degree?• What is its horizontal What is its horizontal

and vertical field of and vertical field of view?view?

Pros/ConsPros/Cons

Characteristics of Characteristics of HMDsHMDs

ImmersiveImmersive• You are inside the computer You are inside the computer

worldworld• Can interact with real world Can interact with real world

(mouse, keyboard, people)(mouse, keyboard, people)• Mask out real world (including Mask out real world (including

body)body) ErgonomicsErgonomics

• Headborn weightHeadborn weight• Length of useLength of use• Cue conflict Cue conflict

accommodation vs. parallaxaccommodation vs. parallax PerspectivePerspective

• Resolution and field of viewResolution and field of view• TetheredTethered• AvatarsAvatars

Multi-tile and see-through HMDsMulti-tile and see-through HMDs

Hand Mounted DisplaysHand Mounted Displays

BinocularsBinoculars• 1280x10241280x1024

$19900$19900

Floor Supported Floor Supported DisplaysDisplays

Articulated mechanical arm Articulated mechanical arm • Offload weight Offload weight • 0.2 ms Latency0.2 ms Latency• High accuracyHigh accuracy• 0.1 degree orientation error0.1 degree orientation error

Pole in the wayPole in the way Limited Space (6’ diam, 3’ Limited Space (6’ diam, 3’

hgt)hgt) Good FOVGood FOV Good resolution Good resolution

(1280x1024)(1280x1024) Fakespace Boom3CFakespace Boom3C WindowsVRWindowsVR

• DifferencesDifferences StereoStereo TrackingTracking

• AdvantagesAdvantages

Desk Supported DisplaysDesk Supported Displays Fishtank VRFishtank VR Autostereoscopic Autostereoscopic

DisplaysDisplays Addresses weight Addresses weight

fatiguefatigue Tracking?Tracking? DTI 2018XL Virtual DTI 2018XL Virtual

WindowWindow Elsa Ecomo4DElsa Ecomo4D

Monitor Large Monitor Large Volume DisplaysVolume Displays

Multiple peopleMultiple people Monitor Based LVDMonitor Based LVD

• Fishtank VRFishtank VR• With limited FOV, what With limited FOV, what

is a possible solution?is a possible solution? Exaggerate trackingExaggerate tracking More monitors (synch, More monitors (synch,

bezel) bezel) Most use active stereo Most use active stereo

(shutterglasses)(shutterglasses)• 29-32% transmittance29-32% transmittance• $100-800$100-800• Can be used for Can be used for

prolonged periodsprolonged periods

Projector LVDProjector LVD WorkbenchWorkbench CAVECAVE

• $300-500k with SGI$300-500k with SGI• $100k with PC clusters$100k with PC clusters• Issues?Issues?

Large wall displaysLarge wall displays

Sound DisplaysSound Displays What are good goals for VR Sound DisplaysWhat are good goals for VR Sound Displays Compare importance to:Compare importance to:

• MoviesMovies• Video GamesVideo Games

What role does sound play?What role does sound play?• InteractivityInteractivity• ImmersionImmersion• Perceived image quality(!)Perceived image quality(!)

DimensionsDimensions• Mono/Stereo/”virtual sound”Mono/Stereo/”virtual sound”

TrackerTracker• OcclusionOcclusion• What does it take to uniquely place a sound source?What does it take to uniquely place a sound source?

3D Sound3D Sound Azimuth CuesAzimuth Cues

• Interaural Time DifferenceInteraural Time Difference head radius/(speed of sound(head radius/(speed of sound(θθ+sin +sin θθ))))

• Interaural Intensity DifferenceInteraural Intensity Difference High frequencyHigh frequency Head shadow effectHead shadow effect

Elevation CuesElevation Cues• How are ears modeled?How are ears modeled?• If simple holes what is the problem?If simple holes what is the problem?• Pinna effects sound propagationPinna effects sound propagation

Frequency attenuation and amplificationFrequency attenuation and amplification

3D Sound3D Sound Range CuesRange Cues

• IntensityIntensity• Motion ParallaxMotion Parallax

As the user moves his/her head, the As the user moves his/her head, the changes in azimuthchanges in azimuth

Head Response Transfer FunctionsHead Response Transfer Functions• Two microphones at the earsTwo microphones at the ears• Record sounds (HR impulse Record sounds (HR impulse

responses)responses)• Corresponding Fourier transforms Corresponding Fourier transforms

(HRTFs)(HRTFs)• Depends on sound frequencyDepends on sound frequency• No two people are exactly the sameNo two people are exactly the same

Given HRTFsGiven HRTFs• Take soundTake sound• Apply filter Apply filter • Get two signals that Get two signals that shouldshould

replicate the sound to the userreplicate the sound to the user However what’s the effect of using However what’s the effect of using

another person’s HRTF?another person’s HRTF?

Sound Display HardwareSound Display Hardware Offload compuational loadOffload compuational load ConvolvotronConvolvotron

• Crystal RiverCrystal River• First virtual 3D sound First virtual 3D sound

GeneratorGenerator• NASA 1988NASA 1988• Real-time DSPReal-time DSP• Calculate new HRTFs givenCalculate new HRTFs given

SoundSound Head position (tracker)Head position (tracker)

AudigyAudigy Soundblaster EAX/ Aureal A3DSoundblaster EAX/ Aureal A3D

• Simple geometrySimple geometry• Occlusion, reflectionOcclusion, reflection

Tracking?Tracking? High-end SDKs - $14kHigh-end SDKs - $14k

Speaker LayoutsSpeaker Layouts

StereoStereo Quad speakersQuad speakers 5.1 surround sound (6.1/7.1)5.1 surround sound (6.1/7.1) Multiple users?Multiple users?

Haptic FeedbackHaptic Feedback Hapthai – touchHapthai – touch Greatly improves realismGreatly improves realism When is it needed?When is it needed?

• Other cues occluded/obstructedOther cues occluded/obstructed• Required for task performanceRequired for task performance• High bandwidth!High bandwidth!

Why are hands and wrist the most Why are hands and wrist the most important?important?• High density of touch receptorsHigh density of touch receptors• Full body are still in research and Full body are still in research and

not very usable (back, body)not very usable (back, body) Two kinds of feedbackTwo kinds of feedback

• Touch Feedback – information on Touch Feedback – information on surface geometry, roughness, surface geometry, roughness, temperature, etc. Does not resist temperature, etc. Does not resist user contactuser contact

• Force Feedback – information on Force Feedback – information on compliance, weight, and inertia. compliance, weight, and inertia. Actively resists contact motionActively resists contact motion

Passive HapticsPassive Haptics

Not controlled by Not controlled by systemsystem

ProsPros• CheapCheap• Large scaleLarge scale• AccurateAccurate

ConsCons• Not dynamicNot dynamic• Limited useLimited use

Active HapticsActive Haptics Actively resists contact Actively resists contact

motionmotion Dimensions?Dimensions?

• Force resistanceForce resistance• Frequency ResponseFrequency Response• Degrees of FreedomDegrees of Freedom• LatencyLatency• IntrusivenessIntrusiveness• SafetySafety• ComfortComfort• PortabilityPortability

Human Haptic System SensorsHuman Haptic System Sensors Meissner and PacinianMeissner and Pacinian

• Quick (high frequency) Quick (high frequency) sensors (50-300 Hz)sensors (50-300 Hz)

• Vibration, accelerationVibration, acceleration• Low resolutionLow resolution

Merkel and Ruffini - Slow Merkel and Ruffini - Slow (low frequency) adapting (low frequency) adapting (0-10 Hz)(0-10 Hz)• Constant, surfaces, edgesConstant, surfaces, edges• High resolutionHigh resolution

Thermoreceptors – Thermoreceptors – temperaturetemperature

Proprioceptors/KinesthesiaProprioceptors/Kinesthesia• Perception of own body Perception of own body

motionmotion Page 96Page 96

Tactile Feedback InterfacesTactile Feedback Interfaces

Goal: Stimulate skin tactile receptorsGoal: Stimulate skin tactile receptors How?How?

• Air bellowsAir bellows• JetsJets• Actuators (commercial)Actuators (commercial)• Micropin arraysMicropin arrays• Electrical (research)Electrical (research)• Neuromuscular stimulations (research)Neuromuscular stimulations (research)

Tactile MouseTactile Mouse

Logitch iFeel Logitch iFeel MouseMouse

Electrical ActuatorElectrical Actuator• Shakes up and Shakes up and

down (do not down (do not disturb XY motion)disturb XY motion)

• Mouse over buttonsMouse over buttons Haptic BumpHaptic Bump Rumble PackRumble Pack

CyberTouch GloveCyberTouch Glove CyberGloveCyberGlove Immersion CorporationImmersion Corporation Six Vibrotactile actuatorsSix Vibrotactile actuators

• Back of fingerBack of finger• PalmPalm

Off-centered actuator Off-centered actuator motormotor

Rotation speed=frequency Rotation speed=frequency of vibration (0-125 Hz)of vibration (0-125 Hz)

When tracked virtual hand When tracked virtual hand intersects with virtual intersects with virtual object, send signal to glove object, send signal to glove to vibrateto vibrate

$15000$15000

Force Feedback DevicesForce Feedback Devices

Attempts to stop the user’s motion if Attempts to stop the user’s motion if necessarynecessary

Larger ActuatorsLarger Actuators GroundedGrounded Mechanical bandwidth Mechanical bandwidth - perceived - perceived

frequency of a haptic interaction (in frequency of a haptic interaction (in Hz)Hz)

Control bandwidthControl bandwidth – system – system control bandwidth to devicecontrol bandwidth to device

Force Feedback JoysticksForce Feedback Joysticks

WingMan Force 3DWingMan Force 3D Inexpensive ($60)Inexpensive ($60) Actuators that can Actuators that can

move the joystick move the joystick given system given system commandscommands

Max 3.3 N of forceMax 3.3 N of force Force feedback Force feedback

driving wheeldriving wheel

SensAble PHANToM ArmSensAble PHANToM Arm

3 and 6 DOF force feedback 3 and 6 DOF force feedback devicedevice

Most popular (1000+) in VR Most popular (1000+) in VR researchresearch

3 DC motors3 DC motors Little interitia/frictionLittle interitia/friction 6.4 N (1.7 continuous)6.4 N (1.7 continuous) Control loop = 1000 HzControl loop = 1000 Hz $1500 – Omni$1500 – Omni $16000 – standard Desktop$16000 – standard Desktop $56000 – 6 DOF high end$56000 – 6 DOF high end

Other Force FeedbackOther Force Feedback

Large ForcesLarge Forces• Haptic Master ArmHaptic Master Arm

250 N250 N

• Sarcos ArmSarcos Arm• http://www.princeton.ed

u/~jmelli/papers/ch4-haptics.pdf

GlovesGloves• CyberGraspCyberGrasp

$39000$39000

• CyberForceCyberForce $56000$56000

Why is it hard?Why is it hard?

Control bandwidthControl bandwidth Mechanical bandwidthMechanical bandwidth ForcesForces HygieneHygiene Gender differencesGender differences PortabilityPortability EncumbranceEncumbrance Solutions?Solutions?

• Specific engineeringSpecific engineering