detail to attention: exploiting visual tasks for selective rendering

30
Detail to attention: Exploiting Visual Tasks for Selective Rendering Kirsten Cater Kirsten Cater 1 , , Alan Chalmers 1 and Greg Ward 2 1 University of Bristol, UK 2 Anyhere Software, USA

Upload: ilori

Post on 09-Jan-2016

29 views

Category:

Documents


0 download

DESCRIPTION

Detail to attention: Exploiting Visual Tasks for Selective Rendering. Kirsten Cater 1 , Alan Chalmers 1 and Greg Ward 2 1 University of Bristol, UK 2 Anyhere Software, USA. Contents of Presentation. Introduction – Realistic Computer Graphics Flaws in the Human Visual System - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Detail to attention: Exploiting Visual Tasks for Selective

Rendering

Detail to attention: Exploiting Visual Tasks for Selective

Rendering

Kirsten Cater Kirsten Cater 11, ,

Alan Chalmers 1 and Greg Ward 2

1 University of Bristol, UK

2 Anyhere Software, USA

Page 2: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Contents of PresentationContents of Presentation

• Introduction – Realistic Computer Graphics

• Flaws in the Human Visual System

• Magic Trick!

• Previous Work

• Counting Teapots Experiment

• Perceptual Rendering Framework

• Example Animation

• Conclusions

Page 3: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Realistic Computer GraphicsRealistic Computer Graphics

• Major challenge achieve realism at interactive ratesMajor challenge achieve realism at interactive rates

• How do we keep computational costs realistic?How do we keep computational costs realistic?

Page 4: Detail to attention: Exploiting Visual Tasks for Selective Rendering

The Human Visual SystemThe Human Visual System

Good but not perfect!

Flaws in the human visual system:

• Change Blindness

• Inattentional Blindness

Avoid wasting computational time

Page 5: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Magic trick to Demonstrate Inattentional Blindness

Magic trick to Demonstrate Inattentional Blindness

Please choose one of the six cards below.Please choose one of the six cards below.

Focus on that card you have chosen.Focus on that card you have chosen.

Page 6: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Magic trick (2)Magic trick (2)

Can you still remember your card?Can you still remember your card?

Page 7: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Magic trick (3)Magic trick (3)

Here are the remaining five cards, is your card there?Here are the remaining five cards, is your card there?

Did I guess right? Or is it an illusion?Did I guess right? Or is it an illusion?

Page 8: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Magic trick – The ExplanationMagic trick – The Explanation

• You just experienced Inattentional Blindness

• None of the original six cards was displayed!

Page 9: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Previous Work

Bottom-up Processing – Stimulus Driven

Previous Work

Bottom-up Processing – Stimulus Driven

• Perceptual Metrics – Daly, Myszkowski et al., etc.

• Peripheral Vision Models – McConkie, Loschky et al., Watson et al., etc.

• Saliency Models – Yee et al., Itti & Koch, etc.

Page 10: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Top-down processing – Task DrivenTop-down processing – Task Driven

Yarbus (1967) recorded observers’ Yarbus (1967) recorded observers’ fixations and saccades whilst fixations and saccades whilst answering specific questions given to answering specific questions given to them about the scene they were them about the scene they were viewing.viewing.

Saccadic patterns produced were Saccadic patterns produced were dependent on the question that had dependent on the question that had been asked.been asked.

Theory – if we don’t perceive parts of Theory – if we don’t perceive parts of the scene what’s the point rendering it the scene what’s the point rendering it to such a high level of fidelity?!to such a high level of fidelity?!

Page 11: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Counting Teapots ExperimentCounting Teapots ExperimentCounting Teapots ExperimentCounting Teapots Experiment• Inattentional Blindness – more than just peripheral vision

methods – don’t notice an object’s quality if it’s not related to task at hand, even if observers fixate on it.

• Top-down processing i.e. task driven unlike saliency models which are stimulus driven, bottom-up processing.

Experiment: Get participants to count teapots for two images then ask questions on any observed differences between the images. Also jogged participants’ memory getting them to choose which image they saw from a choice of two.

Page 12: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Counting Teapots ExperimentCounting Teapots ExperimentCounting Teapots ExperimentCounting Teapots Experiment

3072 x 30723072 x 3072

1024 x 10241024 x 1024

Page 13: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Counting Teapots Experiment –Counting Teapots Experiment – Pre-Exp to find the Detectable ResolutionPre-Exp to find the Detectable ResolutionCounting Teapots Experiment –Counting Teapots Experiment – Pre-Exp to find the Detectable ResolutionPre-Exp to find the Detectable Resolution

Page 14: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Counting Teapots ExperimentCounting Teapots ExperimentCounting Teapots ExperimentCounting Teapots Experiment

Page 15: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Eye Tracking VerificationEye Tracking VerificationEye Tracking VerificationEye Tracking Verification

Page 16: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Eye Tracking Verification with VDPEye Tracking Verification with VDPEye Tracking Verification with VDPEye Tracking Verification with VDP

Page 17: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Counting Teapots ConclusionCounting Teapots Conclusion

Failure to distinguish difference in rendering quality Failure to distinguish difference in rendering quality between the teapots (selectively rendered to high between the teapots (selectively rendered to high quality) and the other low quality objects is NOT quality) and the other low quality objects is NOT due to purely peripheral vision effects.due to purely peripheral vision effects.

The observers fixate on the low quality objects, but The observers fixate on the low quality objects, but because these objects are not relevant to the task at because these objects are not relevant to the task at hand they fail to notice the reduction in rendering hand they fail to notice the reduction in rendering quality!quality!

Page 18: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Perceptual Rendering FrameworkPerceptual Rendering Framework

Use results/theory from experiment to design a Use results/theory from experiment to design a ““Just in timeJust in time” animation system.” animation system.

Exploits inattentional blindness and IBR.Exploits inattentional blindness and IBR.

Generalizes to other rendering techniquesGeneralizes to other rendering techniques

• Demonstration system uses Radiance

• Potential for real-time applications

Error visibility tied to attention and motion.Error visibility tied to attention and motion.

Page 19: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Input:• Task• Geometry• Lighting• View

High-level Vision Model

Geometric Entity

Ranking

Object Map & Motion

Lookup Task Map

Current Frame & Error Estimate

Frame Ready?

Output Frame

Contrast Sensitivity

ModelError

Conspicuity Map

Refine Frame

No

Yes

Last Frame

Iterate

Rendering FrameworkRendering Framework

First Order

Render

Page 20: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Example Frame w/ Task ObjectsExample Frame w/ Task Objects

Page 21: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Example AnimationExample Animation

The following animation was rendered at two The following animation was rendered at two minutes per frame on a 2000 model G3 laptop minutes per frame on a 2000 model G3 laptop computer.computer.

Many artifacts are intentionally visible, but less so if Many artifacts are intentionally visible, but less so if you are performing the task.you are performing the task.

DVDDVDMainMain BonusBonus

Page 22: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Error Map EstimationError Map Estimation

Stochastic errors may be estimated from Stochastic errors may be estimated from neighborhood samples.neighborhood samples.

Systematic error bounds may be estimated Systematic error bounds may be estimated from knowledge of algorithm behavior.from knowledge of algorithm behavior.

Estimate accuracy is not critical for good Estimate accuracy is not critical for good performance.performance.

Page 23: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Initial Error EstimateInitial Error Estimate

Page 24: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Image-based Refinement PassImage-based Refinement Pass

Since we know exact motion, IBR works very Since we know exact motion, IBR works very well in this framework.well in this framework.

Select image values from previous frameSelect image values from previous frame• Criteria include coherence, accuracy, agreement

Replace current sample and degrade errorReplace current sample and degrade error• Error degradation results in sample retirement

Page 25: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Contrast Sensitivity ModelContrast Sensitivity Model

Additional samples are directed based on Daly’s CSF model:

k = 6.1+ 7.3 log(c2vR /3)3

ρ max = 45.9 /(c2vR +2)

c0 =1.14, c1 = 0.67, c2 =1.7 for CRT at 100 cd/m2

⎟⎟⎠

⎞⎜⎜⎝

⎛−⋅⋅⋅⋅=

max

12120

4exp)2(),(

ρ

πρπρρ

ccvcckvCSF RR

where: is spatial frequencyvR is retinal velocity

Page 26: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Error Conspicuity ModelError Conspicuity Model

vR = v I −min(v I ⋅S /Smax +vmin , vmax )

Retinal velocity depends on task-level saliency:

where: vI = local pixel velocity (from motion map) S = task-level saliency for this region Smax = max. saliency in this frame, but not less than 1/0.82 vmin = 0.15°/sec (eye drift velocity) vmax = 80°/sec (movement-tracking limit)

EC = S ⋅max(E ⋅CSF /ND−1, 0) Error Conspicuity where: E = relative error estimate for this pixel ND = noticeable difference threshold

Page 27: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Error Conspicuity MapError Conspicuity Map

Page 28: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Implementation ExampleImplementation Example

Compared to a standard Compared to a standard rendering that finished in the rendering that finished in the same time, our framework same time, our framework produced better quality on task produced better quality on task objects.objects.

Rendering the same high quality Rendering the same high quality over the entire frame would over the entire frame would take about 7 times longer using take about 7 times longer using the standard method.the standard method.

Framew

ork renderingStandard rendering

Page 29: Detail to attention: Exploiting Visual Tasks for Selective Rendering

Overall ConclusionOverall Conclusion

By taking advantage of flaws in the human visual By taking advantage of flaws in the human visual system we can dramatically save on computational system we can dramatically save on computational time by selectively rendering the scene where the time by selectively rendering the scene where the user is not attending. user is not attending.

Thus for VR applications where the task is known a Thus for VR applications where the task is known a priori the computational savings, by exploiting these priori the computational savings, by exploiting these flaws, can be dramatic.flaws, can be dramatic.

Page 30: Detail to attention: Exploiting Visual Tasks for Selective Rendering

The End! Thank you The End! Thank you

[email protected], [email protected],

[email protected]