1 ip cablecom and mediacom 2004 eurescom p905 (aquavit) audio and audiovisual quality for mobile...

1

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 2004

EURESCOM P905 (AQUAVIT)EURESCOM P905 (AQUAVIT)Audio and audiovisual quality Audio and audiovisual quality

for mobile servicesfor mobile services

Harald Klaus – T-SystemsRapporteur of Q9/12

Harald.Klaus@T-Systems.com

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20042

OutlineOutline

Personal ProfileFramework and Objectives of AQUAVITTest beds of AQUAVIT Test methods developed in AQUAVITResults of InvestigationsConclusions

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20043

Personal ProfilePersonal Profile

Harald KlausStudied electrotechnical engineering

at the Technical University of BerlinResearch assistant for speech recognition

applications and speech quality assessmentHead of quality assessment of tele-services

at T-Systems Nova Berkom in BerlinRapporteur in ITU-T Study Group 12 for instrumental

speech quality assessment methodsRapporteur in ETSI Technical Committee STQ

4

The EURESCOM Project P905 The EURESCOM Project P905 (AQUAVIT)(AQUAVIT)

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20045

Framework of EURESCOM P905Framework of EURESCOM P905

AQUAVIT = Assessment of Quality for Audio-Visual Signals over IP and UMTSStart: January 1999End: December 20003 Partners...

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20046

Objectives of AQUAVITObjectives of AQUAVIT

Development of test beds for IP and UMTSDevelopment of subjective and objective test

methodsQuality assessment of typical audio-video

transmission scenarios for IP and UMTSCorrelation between subjective and objective

quality evaluations

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20047

Transm

ission errors

Access Unit Access UnitAccess Unit

Sync. Layer

SL-packetized stream

AU AU AU AU AU

Elementary Stream

Compression Layer

Elementary Stream

Encoders

MP

EG

-4

TestbedsTestbeds

Coding of signals for UMTS on send side

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20048

Executive

Dem

ux

Application

MediaStream BIFSDecoder

Root SceneObject

MediaStream

MediaObject

DecoderMediaStream

T

Presenter

VisualRenderer

AudioRenderer

T

Testbeds (2)Testbeds (2)Decoding of signals for UMTS on receive

side

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20049

Testbeds (3)Testbeds (3)

IP TestbedRunning on standard PCs

in realtimeAdjustable set of

parameters forCoding/DecodingBit rateBit and Frame Errors Error distribution

Windows user interface

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200410

Sequence I

MO

S

Time

just one vote for the whole sequence

a) ordinary way of voting

b) simulation of continuos voting

SI Part a SI Part b SI Part c SI Part d SI Part e SI Part f

Six different votes for the sequence

MO

S

Time

Subjective test methodsSubjective test methodsSpeech quality evaluationSpeech quality evaluation

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200411

Subjective test methodsSubjective test methodsSpeech quality evaluationSpeech quality evaluation

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200412

Subjective test methodsSubjective test methodsAudio quality evaluationAudio quality evaluation

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200413

Subjective test methodsSubjective test methodsVideo & A/V quality evaluationVideo & A/V quality evaluation

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200414

• Questionnaire

• Conjoint analysis

• Focus groups

Subjective test methodsSubjective test methodsAcceptability testsAcceptability tests

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200415

Objective test methodsObjective test methods

TOSQA (Telecommunication objective speech quality assessment)

PEAQ (Perceptual evaluation of audio quality)

A new video quality assessment model developed by British Telecom

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200416

Experiment 1narrow band & wideband

speech

= 96.6%

1

2

3

4

5

1 2 3 4 5MOS

TM

OS

1

2

3

4

5

1 2 3 4 5MOS

TM

OS

Experiment 2aOnly narrow band

speech

= 95.4%

1

2

3

4

5

1 2 3 4 5MOS

TM

OS

1

2

3

4

5

1 2 3 4 5MOS

TM

OS

Experiment 2bOnly wideband

speech

= 98.2%

1

2

3

4

5

1 2 3 4 5MOS

TM

OS

1

2

3

4

5

1 2 3 4 5MOS

TM

OS

Results of investigations Results of investigations Correlation of subjective and objective speech Correlation of subjective and objective speech quality (TOSQA)quality (TOSQA)

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200417

010

2030

4050

6070

8090

100

Mushra

Peaq

010

2030

4050

6070

8090

100

Mushra

Peaq

Results of investigations Results of investigations Subjective test results - audio qualitySubjective test results - audio quality

Low quality conditions (average of all sequences)

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200418

0

20

40

60

80

100

120

Mushra

Peaq

0

20

40

60

80

100

120

Mushra

Peaq

Results of investigations Results of investigations Subjective test results - audio qualitySubjective test results - audio quality

High quality conditions (average of all sequences)

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200419

Results of investigations Results of investigations Acceptability test resultsAcceptability test results

Quality and cost are the equally important main factors to the population questioned.

Benefit of mobility does not compensate for any deficit in quality.

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200420

ConclusionsConclusions

AQUAVIT developed a bundle of improved subjective quality methods for assessment ofSpeech quality (narrowband and wideband)Audio quality (intermediate and high quality)Video and A/V quality (intermediate and high q.)

All tests use the same set of materialResults of all tests can be directly used for

comparisons

IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200421

T-Systems Nova BerkomHarald KlausHead of Quality Assessment ===!"§==Systems=

User-centred Quality Engineering.Quality assessmentat T-Systems Nova.Consulting

Test conceptsand test design Quality Monitoringand Reporting Process Optimization

Speech, audioand A/V quality measures According to international

recommendationsandstandards Optimized, efficienttestprocedures Tests withrepresentative user groups Intrusiveand non-intrusive

measurement methods Measurements with Head-and-Torso

simulators (HATS)

Our Labs High quality accordingto ITU

requirements for auditoryand instrumental

quality assessment for recordings usingHATS for speechand audioproduction

New test New test services services of T-System of T-System Nova are Nova are based on based on AQUAVIT AQUAVIT methodsmethodsand resultsand results