schepens eye research institute development of an objective, language-based measure of perceived...

Schepens Eye Research Institute

Development of an Objective, Language-Based Measure of

Perceived Video QualityDaniel R. Saunders,

Peter J. Bex,and Russell L. Woods

Schepens Eye Research Institute and Harvard Medical SchoolBoston, Massachusetts


• Dedicated to fighting blindness and visual impairment• Treatments and assistive technology for eye disorders

• Example: Age-related Macular Degeneration– Affects at least 1.75 million in the U.S.– Central vision loss, leading to reduced

contrast sensitivity and visual acuity– No cure

The Schepens Eye Research Institute

Daniel Saunders


Digital video enhancement

Daniel Saunders

Contrast enhancement (Peli and Peli, 1984; DigiVision CE-3000)

QuickTime™ and aH.264 decompressor

are needed to see this picture.


Digital video enhancement

Daniel Saunders

Digital magnification

QuickTime™ and aVideo decompressor

are needed to see this picture.


How to evaluate enhancements?• Easy to measure preference:

– Dual monitor– Rating

Adaptive contrast enhancement has been shown to be preferred in many cases. (Kim et al., 2004; Fullerton et al., 2008)

But subjective.

An objective measure would derive from how much it helps people perform a task. Reading, driving, watching TV.

Daniel Saunders


Information transfer• Principle: The better the perceived quality of a video

clip, the more (and more accurate) information the viewer should be able to report about it.

• Impaired perception should result in reduced amount of accurate information.

• Use of multiple choice quiz questions has problems– Creation of questions is subjective– Labour intensive– Also may have low sensitivity (Peli, 2005)

Daniel Saunders


Using free recall to measure information transfer

Daniel Saunders

“Describe this video as if to someone who hasn't seen it.”

Standard set of 200 video clips, 30 seconds each, from Hollywood films and nature documentaries.


Collecting “normal” responses• 4000 responses from Amazon.com Mechanical Turk

workers (20 per clip) working from home• 2400 responses from normally-sighted in a lab

setting, stratified by age and gender (in progress)

Daniel Saunders

A spotlight displays the Batman symbol. Batman arrives on a rooftop and a beautiful redhead appears from behind the

spotlight wearing a black negligee and robe. She flirtatiously walks over to the caped crusader and they

converse.

Batman swoops on. As he lands a woman comes out and starts talking to him. There is some plan afoot that Batman

must thwart and she seems helpful by her look.

Batman flies onto a roof top and starts talking to a blonde woman.

The scene is very dark. We first see a bat spotlight, then Batman flies into the room. A woman with long hair comes

out and speaks with Batman.

It is a very dark room with only a "Batman" Logo shining into the night. This is how Batman is called upon. Once he arrives, it is in a darken room with a very seductress Nicole

Kidman to greet him there.

It starts out with a huge bright spotlight with Batman's stencil on it shining up into the night. It's a very dark scene,

when Batman arrives. He is then approached by a young woman with long wavy red hair in a black robe and slinky

nightgown. There is a slight breeze blowing her hair.

A giant floodlight has a cutout of Batman in it. Batman appears out of nowhere and walks by the spotlight. A

woman wearing a sexy dress walks into the room and talks with Batman.

Examples:


Evaluate responses in terms of their discrepancy

Daniel Saunders

A spotlight displays the Batman symbol. Batman arrives on a rooftop and a beautiful redhead

appears from behind the spotlight wearing a black negligee and robe. She flirtatiously walks over to

the caped crusader and they converse.

Batman swoops on. As he lands a woman comes out and starts talking to him. There is some plan afoot that Batman must thwart and she seems

helpful by her look.

Batman flies onto a roof top and starts talking to a blonde woman.

The scene is very dark. We first see a bat spotlight, then Batman flies into the room. A

woman with long hair comes out and speaks with Batman.

It is a very dark room with only a "Batman" Logo shining into the night. This is how Batman is called upon. Once he arrives, it is in a darken room with a very seductress Nicole Kidman to

greet him there.

It starts out with a huge bright spotlight with Batman's stencil on it shining up into the night. It's

a very dark scene, when Batman arrives. He is then approached by a young woman with long

wavy red hair in a black robe and slinky nightgown. There is a slight breeze blowing her

hair.

A giant floodlight has a cutout of Batman in it. Batman appears out of nowhere and walks by the

spotlight. A woman wearing a sexy dress walks into the room and talks with Batman.

• Score for an individual is average distance of their responses from the normally-sighted responses.

• A high score indicates poor vision/perceived video quality.


Text passage similarity metrics• Baseline: shared word count (common word

removed)– Performance (Mechanical Turk): Mean shared words

within a clip = 6.8, outside a clip = 1.7. Accuracy at matching responses to clips = 95%.

• Latent Semantic Analysis could deal with synonyms etc. Used in essay grading and finding articles by matching abstracts. – Not our wheelhouse!

Daniel Saunders


Assessing validity of the measurePerceived video quality = Perception + video quality.

Measure can be used to assess video, with applications in e.g. video compression algorithms.

Future studies:• Normally-sighted participants watch blurred video.

– Greater blur should cause larger discrepancy scores– Within-subject design, so compensates for variability

• Visually-impaired participants watch enhanced video.

Daniel Saunders

Schepens Eye Research Institute Daniel Saunders

What we have: • An application (evaluating the benefit of video

enhancement to the visually impaired).• A dataset (free descriptions of 200 video clips).

What we need: A metric of text passage discrepancy. (LSA?)

Thanks!


Mechanical Turk for data collection

Daniel Saunders 13 / 31

• 3000 responses in 30 days


No objective measure of perceived video quality

• Most are based on ratings or side-by-side comparisons

• Relevant to perception as well as to videos.



Age-related macular degeneration• Affects more than 1.75 million people in the United

States(Friedman et al., 2004), 3 million by 2020.• Causes 50% of all cases of blindness in the United

States (WHO, 2002).• No cure, only aids.• Which aids really help? (the same question as

perceived visual quality)



TV is an important activity• How can we help people watch TV?• How can we evaluate whether we’ve helped?


20% of seniors who went in for a visual impairment exam said that one of their main motivations was watching TV.

69% of people with central field loss reported at least “some” difficulty with watching TV

schepens eye research institute development of an objective, language-based measure of perceived...

Documents

batman flies

batman logo

cutout of batman

batman symbol

batman swoops

dark room

progress daniel saunders

discrepancy daniel saunders