Progress and Future Plansfor

VQEGVQEG

Arthur WebsterU.S. Department of Commerce, NTIA

Institute for Telecommunication Sciences (webster@its.bldrdoc.gov)

ETSI STQ Workshop “Effect of transmission performance on Multimedia Quality of Service”

17-19 June 2008 - Prague, Czech Republic

From Turin to Kyoto 1997 - 2008

� First VQEG meeting in Turin, Italy 1997

⇒ Participants from ITU-T SG12, SG9, ITU-R WP11E

From Turin to Kyoto 1997 - 2008

� Most recent meeting in Kyoto, Japan 2008

⇒ Current focus is on Hybrid Methods, Multimedia, HDTV

Video Quality Expert’s Group (VQEG)� Founded 1997

� ITU-T SG 12, SG 9, and ITU-R 11E (now 6G) video experts � Website ( www.vqeg.org )� FTP server ( ftp://vqeg.its.bldrdoc.gov/ )

� Primary mission� Validate objective video/multimedia quality metrics� Report results to ITU and other standards organizations

VQEG does not develop or publish standards� VQEG does not develop or publish standards� Conducts tests and reports results� Provides expertise on video/multimedia quality

� VQEG’s Independent Lab Group (ILG) oversees and conducts tests� Combines resources of laboratory organizations worldwide� ILG members are not technology proponents in the same test

⇒International — focus on video/multimedia quality

Characteristics of VQEG

� Unofficial, with links to several standards organizations� Technical group, open to all� No dues, no staff, all volunteer effort� 650 email accounts on the main VQEG email list� Physical meetings have 20 to 40 participants

Advantages and disadvantages� Advantages and disadvantages� Documents and information open to all� Dependence upon volunteers to do all the work� Participants from industry, academia, government labs, and

standards organizations

⇒ Liaisons with ITU-T, ATIS, IEC, ETSI STQ

Organizations Studying Objective Methods for Video/Multimedia Quality Assessment

ITU-TJRG-MMQA

ITU-TStudy Group 12

ITU-TStudy Group 9

ITU-RWP 6G

VQEG

JRG-MMQA

ATIS PRQC, IIF/QoSMETSI STQ,

IEC/TC100/TA11

⇒ VQEG brings together SDOs, academia, industry

VQEG’s Board

Name Organizaton Position

Arthur Webster NTIA/ITS VQEG Co-Chair

Filippo Speranza CRC VQEG Co-Chair

Kjell Brunnstrom Acreo ILG Co-Chair

Greg Cermak Verizon ILG Co-Chair

Alex Bourret British Telecom RRNR-TV Co-Chair

Chulhee Lee Yonsei University RRNR-TV Co-Chair

Kjell Brunnstrom Acreo Multimedia Co-Chair

David Hands British Telecom Multimedia Co-Chair

⇒Co-Chairs Coordinate areas of work in VQEG

David Hands British Telecom Multimedia Co-Chair

Vittorio Baroncini FUB HDTV Co-Chair

Phil Corriveau INTEL HDTV Co-Chair

Margaret Pinson NTIA/ITS HDTV Co-Chair

Ricardo Pastrana France Telecom Tools and Subjective Labs Setup Co-Chair

Patrick Le Callet IRCCyN Tools and Subjective Labs Setup Co-Chair

Yves Dhondt Ghent University Tools and Subjective Labs Setup Co-Chair

Chulhee Lee Yonsei University Hybrid Perceptual/Bitstream Co-Chair

Jens Berger SwissQual Hybrid Perceptual/Bitstream Co-Chair

Patrick Le Callet IRCCyN Joint Effort Group Co-Chair

Alex Bourret British Telecom Joint Effort Group Co-Chair

Types of Objective Video Quality Assessment Methods (ITU-T Recommendation J.143)

� Full Reference (FR)� Full access to source video� Accurate but not as useful for in-service� Scene and technology independent

� Reduced Reference (RR)� Compares low bandwidth quality features from SRC & PVS streams

In-service monitoring� In-service monitoring� Scene and technology independent� Often as accurate as FR, degrades gracefully as RR BW is reduced

� No Reference (NR)� Access to source video not needed� In-service but not as accurate � Not scene or technology independent

⇒ Each type has valid applications and strong proponents

Full Reference TV (FRTV) Phase 1

� From 1997 to 2000� 8 subjective Labs, 10 objective proponents� 20 source sequences (split 525 & 625)� 32 video systems� Results:

� No clear “winner”� Recommendation ITU-T J.144 (March 2001)� Recommendation ITU-T J.144 (March 2001)

� 8 methods documented in non-normative appendices� VQEG FRTV Phase I Report

� Huge database publicly available for research� 20 source (3.5G) 320 processed sequences (54.5G)� Subjective ratings

� See: Objective Perceptual Assessment of Video Quality: Full Reference Television� http://www.itu.int/ITU-T/studygroups/com09/docs/tutorial_opavc.pdf� (Documents from FRTV Phase I and Phase II)

Full Reference TV (FRTV) Phase 2

� From 2000 to 2004� 3 subjective Labs, 8 objective Proponents (2 withdrew)� 26 source sequences (split 525 & 625)� 24 video systems� Results:

� Four models did better than PSNR for both 525 and 625:� NTIA/ITS (USA)� British Telecom (UK)� Yonsei University / SK Telecom (Korea)� CPqD (Brazil)

� These four are normative models included in Recommendations:� ITU-T J.144 2004 (ITU-T SG9)� ITU-R BT.1683 2004 (ITU-R SG6)

� Video database not available to the public� Many sequences restricted

Multimedia Test Phase I (video only)

� Active from 2004 to 2008� 5 proponents, 10 independent test labs, 7 other subjective test labs� 41 subjective tests (completed March 2008)

� 386 source video clips (VGA, CIF, QCIF)� 5320 processed sequences � Evaluated by 984 viewers

� Results:� Results:� Four normative models are included in Draft Recommendation

� ITU-T J.mmvofr (Full Reference) (SG9 will launch AAP LC in July)� Psytechnics� Opticom� Yonsei University� NTT

� One proponent’s models included in Draft Recommendation: � ITU-T J.mmredref (Reduced Reference) (SG9 will launch AAP LC in July)

� Yonsei University

� Video database not available to the public

Joint Rapporteur Group on Multimedia Quality Assessment (JRG-MMQA)

� Founded 2004� Between ITU-T SG9 and SG12 � Multimedia has both audio and video

� Audio and network experts from ITU-T SG12� Audio and network experts from ITU-T SG12� Video experts from ITU-T SG9� Many meetings have been held– usually as part of

VQEG meetings

� Multimedia Phase II will have audio and video� Hybrid may have audio as well as video

Multimedia Quality Standards

� From ITU-T J.148: “Requirements for an objective perceptual multimedia quality model”

Audio Source Audio Quality

Differential Multimedia quality

Aq

Aq ( Vq )

Mutimedia quality Differential Delay

Multimedia quality integration function

Video Source

Video Quality

Task

Vq

Mutimedia quality

Vq ( Aq )

⇒ Framework for ongoing VQEG multimedia testing

Representative (Current) ITU Projects

ITU-T SG 9� J.mmvofr – AAP Last Call July 2008� J.mmredref – AAP Last Call July 2008� J.redref (SDTV – reduced reference) – late 2008� J.mmnoref (MM no reference) – late 2008?� J.noref (SDTV – no reference) – late 2008?

Recs based on HDTV tests – 2009� Recs based on HDTV tests – 2009� Recs based on hybrid bitstream / perceptual tests – late 2009

ITU-T SG 12� G.1050 (IP/MM transmission performance model)� G.OMVAS (Quality planning model for IPTV)� P.NAMS (Quality monitoring model for IPTV)

⇒ Complementary work is being done in ITU-R SG 6

RRNR-TV Test

� Begun 2000; restarted 2005� May finish by September 2008

� Test plan: ftp://vqeg.its.bldrdoc.gov/Documents/Projects/rrnr-tv/RRNR-tv_draft_2.1_changes_highlighted.doc

� 4 proponents, 2 subjective tests � 2 subjective tests � 24 source video clips (525-line, 625-line)� ~296 processed sequences � Evaluated by 48 viewers

� Results:� Recommendation possible by September 2008

� Video database not available to the public.

HDTV Test

� Begun 2004; due to finish mid 2009� Test plan:

ftp://vqeg.its.bldrdoc.gov/Documents/Projects/hdtv/VQEG_HDTV_testplan_v2_0.doc

� ~7 proponents, ~7 proponents, � ~7 subjective tests (1 per proponent, maybe ILG)

� 9 source video clips (1080i) per proponent� 153 processed sequences per proponent� Evaluated by 24 viewers per test

� Results:� Expected by late 2009

� Video database probably not available to the public.

Hybrid Perceptual/Bitstream Test

� Begun 2006; could finish late 2009� Test plan:

ftp://vqeg.its.bldrdoc.gov/Documents/Projects/hybrid/VQEG_hybrid_testplan_v1.1a.doc

� Early stages, much interest, not much work done yet � Early stages, much interest, not much work done yet � Results:

� Expected by September 2009

� Video database not determined

Multimedia Phase II Test

� Not started, probably will begin September 2008� Test plan: None� Early stages, much interest, not much happening yet � Results:

Expected by September 2010� Expected by September 2010

� Audio/Video database not determined� Possible that current standardized models for audio and

video quality assessment will be used� May only need a combining function for audiovisual quality.� Still will require a validation test

VQEG Timeline and Future Activities

Multimedia – I : Video OnlyMultimedia – II : Video & Audio

HDTV

Multimedia - IQCIF

HYBRID

Multimedia - II3D?

2000 20041997 1999 2005 20061996 1998 2001 2002 2003 2007 2008

FRTV-I

FRTV-II

RRNR-TV

Multimedia - I

VGA

CIF

2009 2010 2011

Other VQEG Efforts

� Tools and Subjective Labs Setup Group� Collects, develops, or locates quality assessment software

tools� Makes them available on VQEG ftpsite

� Prefer freeware be contributed� Provides info for

� Subjective lab setup� Lab equipment (monitors, etc.)

� Joint Effort Group� New group, started 2007� Goal to jointly develop quality assessment models

� Combine the best parts of models

⇒ Joint Development of models very attractive

Summary

� VQEG combines expertise of academia, industry, government labs, and SDOs

� Conducts tests and reports results to SDOs and email list

� Major projects completed:� FRTV-I, FRTV-II, MM-I� FRTV-I, FRTV-II, MM-I

� Current and Future Projects:� RRNR-TV, HDTV, Hybrid, MM-II

� Joint development effort:� Potentially wider participation� Possibly better models

⇒ Future solutions will require coordination among SDOs