effects of surface characteristics on alignment of real and graphic objects in stereoscopic...

Effects of Surface Characteristics on Alignment of Real and Graphic

Objects in Stereoscopic Augmented Reality Environments

Ming Hou

Ergonomics in Teleoperation and Control (ETC) Laboratory

Department of Mechanical and Industrial Engineering

University of Toronto

January 6th, 2003

3D Measurement in Unmodelled World

Virtual Tape Measure

Problem:

Unknown relationship between real and virtual objects in Augmented Reality displays

Research on Real World Targets

Video ImageVirtual Pointer

LINE

AREA

VOLUME

Hemisphere

Cylinder

Pseudo-Transparency Phenomenon

Real ObjectReal Object

(in Video)

Virtual pointer inVirtual pointer in frontfront of surfaceof surface

VirtualVirtual pointer pointer behind behind realreal surfacesurface

““Transparency EffectTransparency Effect””Breakdown of Fusion Breakdown of Fusion + Fusion Conflict + Fusion Conflict

Fuse Real Surface Fuse Virtual Pointer

Virtual pointer inVirtual pointer in frontfront of surfaceof surface

Virtual Virtual pointer pointer behindbehind real surface real surface

Conflict between Binocular Disparity and Occlusion Cues

Theory of Surface Interaction: Fusion Breakdown depends on Texture Density

Low Texture Density High Texture Density

Virtual pointer inVirtual pointer in frontfront of surfaceof surface

Virtual Virtual pointerpointer behindbehind real surface real surface

Research Motivation

Is this conflict really significant?

If yes, can it be used as an extra cue for detecting interactions between real and virtual objects,

and thus locating the real objects in AR environment more easily and accurately?

Main Hypotheses #1Main Hypotheses #1

More accurate to indicate position More accurate to indicate position on (curved) surface with on (curved) surface with high high texture density (HTD) (HTD) than with than with low low texture density (LTD) (LTD)

HTDHTD better than better than LTDLTD

Main Hypotheses #2Main Hypotheses #2

OrientationOrientation of observer relative to of observer relative to target surfacetarget surface will have influence will have influence

CentreCentre different fromdifferent from Off-centreOff-centre

Top ViewReal Surface

Virtual Pointer

Stereo Camera

Top ViewReal Surface

Virtual Pointer

Stereo Camera

Main Hypotheses #3Main Hypotheses #3

FormForm of Virtual Pointer ( of Virtual Pointer (VPVP) ) will have impact on alignment will have impact on alignment performanceperformance

LINE LINE AREAAREA VOLUMEVOLUME

Main Hypotheses #4Main Hypotheses #4

BBinocular disparityinocular disparity (i.e. crossed (i.e. crossed vs 0 vs uncrossed) will affect vs 0 vs uncrossed) will affect alignment performancealignment performance

No disparity (ND) > Crossed (C) or Uncrossed (UC)No disparity (ND) > Crossed (C) or Uncrossed (UC)

NDNDCC UCUC

Experimental Investigation ofAR “ Surface Effects ”

Stereo CameraStereo Camera

Indigo 2Indigo 2

CylinderCylinder

BarrierBarrier

VirtualVirtual PointerPointer

Stereoscopic AR DisplayStereoscopic AR Display

SpaceballSpaceball

Methodology

Expt. #Factorial Design

Independent Variables

Dependent Variables (Measurement)

#1 2x2x3x3

Texture Density Surface Orientation Binocular Disparity

VP Orientation

Placement ErrorPlacement Error between perceived between perceived target and its actual target and its actual

position on real surfaceposition on real surface

#2 2x2x3x3

Texture Density Surface Orientation Binocular Disparity

VP Form

Placement Error Placement Error ++ Confidence Rating + Confidence Rating +

Preference RatingPreference Rating

#3 5x5Texture Density

Surface Orientation

Placement ErrorPlacement Error ++ Angular ErrorAngular Error between between estimated normal and estimated normal and

real surface normalreal surface normal

6 Images

(15 paired comparisons)

Virtual Pointer FormLINE AREA VOLUME

Texture

Density

High

Low

Subjective Comparisons

Ease of Use

Transparency

Ease of Fusion

Methodology

Expt. #Factorial Design

Independent Variables

Dependent Variables (Measurement)

Expt. #1

2x2x3x3

Texture Density Surface Orientation Binocular Disparity

VP Orientation

Placement ErrorPlacement Error between perceived between perceived target and its actual target and its actual

position on real surfaceposition on real surface

Expt. #2

2x2x3x3

Texture Density Surface Orientation Binocular Disparity

VP Form

Placement Error Placement Error ++ Confidence Rating + Confidence Rating +

Preference RatingPreference Rating

Expt. #3

5x5Texture Density

Surface Orientation

Placement ErrorPlacement Error ++ Angular ErrorAngular Error between between estimated normal and estimated normal and

real surface normalreal surface normal

Placement Error vs Texture Density - 1

Main Experimental Results - 1

VP Placement Error vs Texture Density(Error Bar = 95% CI, F(1,11) = 11.14, P = 0.007)

-0.4

0

0.4

0.8

1.2

1.6

High Low

Texture Density

Mean

Pla

cem

en

t E

rro

r (c

m) Farther

Closer

\ \ Cylinder Surface \ \

Placement Error along Surface Normal vs

Texture Density(Error Bar = 95% CI, F(4,40) = 41.34, P < 0.001)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0 10 20 30 40

Texture Density (%)

Me

an

Pla

ce

me

nt

Erro

r (

cm

)

Placement Error vs Texture Density - 2

Main Experimental Results - 2

Placement Error vs Surface Orientation - 1

Main Experimental Results - 3

VP Placement Error vs Target Position (Error Bar = 95% CI, F(1,11) = 98.19, P < 0.001)

-0.4

0

0.4

0.8

1.2

1.6

Centre Right

Target Position (relative to observer)

Mean

Pla

cem

en

t E

rro

r (c

m)

Closer

Farther

Placement Error along Surface Normal vs

Target Position(Error Bar = 95% CI, F(4,40) = 11.90, P < 0.001)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

1 2 3 4 5

Target Position on the Sphere

Mean

Pla

cem

en

t E

rro

r (c

m)

Placement Error vs Surface Orientation - 2

Main Experimental Results - 4

1 2 3 4 5

Altitude Error vs Target Position on Hemisphere

(Error Bar = 95% CI, F(4, 40) = 36.72, P < 0.001)

0

5

10

15

20

1 2 3 4 5

Target Position

Mean

Alt

itu

de E

rro

r

(deg

ree)

Angular Error vs Surface Orientation

Main Experimental Results - 5

1 2 3 4 5

Subjective Comparison Resultsw.r.t. Ease of Use and Ease of Fusion

Volume VP better than other VPs, regardless of texture density

w.r.t. Transparency

Volume VP + Highly textured least transparent combination

Main Experimental Results - 6

Texture Density High better than Low

Surface Orientation

Centre different from Off-Centre

Summary of Results - 1

VP Form VOLUME > AREA > LINE (subjective comparisons)

Binocular Disparity

Summary of Results - 2

NDNDCC UCUC

Conclusions - 1Conclusions - 1

Perceptual conflict does exist when real and virtual objects interact in 3D AR environments (a model proposed)

Perceptual conflict can be used as extraextra depth cue to indicate interaction between real and virtual objects

Optimal density value for Random Dot texture pattern was found as Engineering Solution for accurate 3D measurement

R-V Interaction Process ModelR-V Interaction Process Model

behind

Localisation Achieved

Fusiondifficulty

No fusion difficulty No

Yes

(Transparency)

at

at

behind

(Breakdown/Conflict)

Perception (Stereo Matching)

AR Display

User

External WorldDisplay of virtual pointer (VP)

superimposed upon real object surface VP Controller

Cognition / Decision Making(Stereo Matching + Cue Conflict Resolution)

VP behind, or at, real

surface?

VP behind, or at, real

surface?

Stereo Matching :

VP in front ofreal surface?

FusedImage?

Ad

just

men

t

Conclusions - 1Conclusions - 1

Perceptual conflict does exist when real and virtual objects interact in 3D AR environments (a model proposed)

Perceptual conflict can be used as extraextra depth cue to indicate interaction between real and virtual objects

Optimal density value for Random Dot texture pattern found as Engineering Solution for accurate 3D measurement

Conclusions - 2Conclusions - 2Target position does affect alignment

task: centrally located targets benefit performance, but have disadvantage when along the line of sight

Volumetric stereo graphic cursor (more fusable features along three dimensions) is subjectively the most favoured VP

“Pseudo-transparency” contributes literature of depth perception cues (shape-from-texture and stereo)

Implications for AR Interface DesignImplications for AR Interface Design

Random Dot Stereogram enhances 3D alignment performance

Perceptual conflicts can be used as extraextra depth cue to detect real object position

3D VP better than other VPs

Perceptual errors always exist

LimitationsLimitations Implementation

Display Mode : stationary display without motion parallax and motion

perspective

Binocular Disparity : confounded with size cue and resolution

Scope

VP Design: line thickness of wire-frame VP

Texture Pattern: square Random Dot pattern

Future Work/Impact - 1Future Work/Impact - 1

Near Term Research

Projected lighting : more practical Motion parallax with Video-HMD Computational vision may alleviate

some error (being investigated)Simulated Projector

Stereo Cameras

Projected lighting with random dot

texture pattern

Future Work/Impact - 2 Long Term Interests

Integration of Computer Assisted Object Detection in AR Displays See-through HMD AR for Dismounted Soldiers in the Battlefield (e.g., Perceptual Conflicts, Navigational Aids, etc.)

Other Human Computer Interaction (HCI) Topics (e.g., Integration of Electronic Information

with Human-Machine Systems, etc.)

AcknowledgementAcknowledgement• Dr. Julius Grodski at Defence Research &

Development Canada (DRDC) – Toronto, Prof. Allison B. Sekuler and Prof. Paul Milgram at University of Toronto

• Dr. Stephen Ellis at NASA and Prof. Stanley Hamstra at University of Toronto

• Natural Sciences and Engineering Research Council (NSERC) Doctoral Scholarship

• Institute of Robotics and Intelligent Systems (IRIS), Canada

• Ontario Graduate Scholarship (OGS)

Experimental #1 Main ResultVP Placement Error vs Surface Texture

(Error Bar = 95% CI, F(1,9) = 619.71, P< 0.001)

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

High Low

Texture Density

Pla

ce

me

nt

Err

or

(cm

)Farther

Closer

\ \ Cylinder Surface \ \

VP Form and Orientation in 1st Experiment :

Diagonal Horizontal

Vertical

Observer

Interaction in Experiment # 1Interaction in Experiment # 1

Interaction between Surface Texture (High vs Low) and Target Position (Center vs Right)

VP Placement vs Texture and Target Position

(Error Bar = +/- 1SD, F(1,9) = 246.33, P< 0.001)

-11-10-9-8-7-6-5-4-3-2-101

Centre Right

Angular Displacement of Target Normal

Pla

cem

ent E

rro

r (c

m)

High

Low

Paired Comparison Result

Ease of Use

Transparency

Ease of Fusion

Image # 1 2 3 4 5 6

Mean Z score

0 1.15 1.71 2.26 3.16

Image # 2 1 3 6 5 4

0 0.25 0.89 2.97 3.82 4.37

Image # 1 2 3 5 4 6

0 0.69 0.73 2.08 2.14 2.26

Measurement of Placement Error along Surface Normal

Estimated Target

Positive Error

Real Surface Target

Surface Normal

Positive error shows the estimated target is inside the sphere along surface normal

Placement Error vs Texture Density for Experiments 2 and 3

Placement Error along Surface Normal vs

Texture Density(Error Bar = 95% CI, F(4,40) = 41.34, P < 0.001)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0 10 20 30 40

Texture Density (%)

Me

an

Pla

ce

me

nt

Erro

r (

cm

)

Experiment 2Experiment 3

Definition of Angular Error

X

Y

P

N

T

Z

O

Q

RS

'

'

Altitude Error : Angular distance between estimated normal TP and real surface normal TN

Azimuth Error : Angular Error between horizontal projection OR and OS

'

'

Angular Bias in Spherical Coordinate

Y

Bias Area

S

E

N

X

Z

T

Angular bias tilted upwards from real surface normal (TN)

Example of Distribution of Altitude and Azimuth

Angular bias between estimated and real surface normal

Altitude Error vs Azimuth Error at Target 5

-30

-20

-10

0

10

20

30

40

50

60

-180 -135 -90 -45 0 45 90 135 180

Azimuth Error (degree)

Alt

itu

de

Err

or

(deg

ree)

Side View

Stereo Cameras

Ground Truth Measurements in Real SceneCylinder Stimulus

Real Distance in Depth (Z)

Cylinder Calibration objectIron plate

Top View

Real World Origin (0,0,0)

Registration Verification: Measurement of a Pin and a Cube

Calibration Target

Stereo Cameras

Calibration Cube

VP Resolution for Experiment 2 VP Resolution and Accuracy TestsVP Resolution (Percentage of Real Distance

vs One Pixel Error of VP in Display)

0.00%

1.00%

2.00%

3.00%

4.00%

5.00%

6.00%

7.00%

8.00%

0 1 2 3 4 5

Distance of Object from Cameras (m)

Perc

en

tag

e o

f D

ista

nce (

D)

for

1

Pix

el C

han

ge in

Ho

rizo

nta

l D

isp

ari

ty

Retinal Disparity in Stereoscopic Display for One Pixel Separation (exaggerated)

b

a

C’

d

ZD

XD

2e

P = 1 pixel separation

Apparent position of point C’, due to 1 pixel horizontal disparity

StereoscopicDisplay Monitor

Viewer’sEye

b /2

)2

)2(2)(2

d

Pearctg

d

earctgba

Psychophysical Standard for Texture Density Control

• Spatial scale (size)

• Homogeneity (spatial regularity, density is approximately constant over the surface)

• Isotropy (no orientation bias, equally to be oriented in all directions) – compression

Practical Augmented Reality Example

Distance between point 1 and point 2 is 7mm

Coordinate of Point 1 (2.3, 14.7, 96.2)

Coordinate of Point 2 (1.8, 14.4, 95.8)

Virtual Tape Measure for Minimally Invasive

Surgery