Towards a unique subjective experiment dataset for 3DTV – « 3DTV phase 1 »

Marcus Barkowsky

Overview

Purpose and scope Presentation of (additional) SRCs Presentation of (some) HRCs Timeline Examples of subjective test methodologies and

conditions Discussion

13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1

- 2 -

Purpose and scope

The scope of this database is to Provide a unique set of degraded (+reference)

sequences Free for distribution Free for publication Well documented Allowing comparisons!

Idea: Collecting a large number of “indices” for the same set of sequences in order to correlate the results and to discover the “ground truth”

13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1

- 3 -

Purpose and scope

This database allows for Cross - lab analysis: Run the same subjective experiment in

several labs Cross - methodology: Use different subjective test

methodologies Cross - environment: Use different environments for the

subjective experiment, e.g. displays, backgrounds Cross – scales: Asking observers to rate on various scales such

as discomfort, depth quality, preference to 2D, etc. in order to find the relation to “Overall Quality of Experience”

Comparing the effect of training of observers Evaluating additional inter-observer differences using for

example questionnaires, observer characterization tests (prism facility, 3D acuity, sensitivity to asymmetric conditions)

13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1

- 4 -

Out of this scope

The following points are out of the scope of this evaluation: Length of sequences

13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1

- 5 -

Presentation of additional SRC material database

NAMA3DS1:Nantes-Madrid 3D Stereoscopic Video Database 1 Content captured with Panasonic 3DA1 Most of the content captured uncompressed, otherwise

~20Mbit/s per view Camera parameters mostly available Depth map estimation available but not satisfying

Involved in the shooting: From UPM: Jesús Gutiérrez, Narciso García From IRCCyN/IVC: Matthieu Urvoy, Marcus Barkowsky,

Romain Cousseau, Yohann Pitrey, Yao Koudota, Junle Wang, Patrick Le Callet

13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1

- 6 -

Presentation of additional SRC material

13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1

Depth map

Filtered depth map

- 7 -

Presentation of (some) HRCs

HRCs currently planned for the database: Multiview Video Coding (MVC) MPEG-4 part 10 / H.264 (AVC) Transmission distortions Resolution reduction Framerate reduction Asymmetric conditions

Other ideas: Conversion S3D 2D+Z S3D Geometric distortions <input welcome!>

13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1

- 8 -

Selection of HRCs

How to select the PVSs for a unique subjective experiment ( ~ 160 sequences) from the large set of PVS? Spread the range of

Different content Different visual degradations (coding, blocking, blurring, color

distortions, geometric distortions, …) Different scale dimensions (video quality, discomfort, depth,

…) Suggestion: Perform a voting for PVSs in different 3D labs

13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1

- 9 -

Timeline

Preparation of SRC sequences 2 sets are already available

Identification of conditions in large HRC set End of January?

Distribution of PVSs Finish: End of February

Selection of PVSs Conference Call mid-March

Unique dataset Available end of March

Subjective experiments start Analyze the results, prepare a report Determine impact on Recommendations

13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1

- 10 -

Dimensions of subjective experiment

Ideas of different test methodologies and test setups Methodologies

ACR-HR DSCQS Paired Comparison (subset methods) SAMVIQ

Scales Quality of Experience Video Quality Naturalness Visual Discomfort

Environment Different screens Standardized / Home environment

13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1

- 11 -

Questions

Adding other SRC sequences NICT license open enough for free distribution?

Who would like to contribute more/different HRCs in that timeframe?

Interest in participation in the PVS selection? Who would be willing to run subjective experiments?

With well-prepared setup, e.g. 3D Blu-Ray? Controlling their own lab setup and experiment method?

13/12/2011Marcus Barkowsky: Unique dataset 3DTV phase 1

- 12 -