eurescom p905 (aquavit) audio and audiovisual quality for mobile services
DESCRIPTION
EURESCOM P905 (AQUAVIT) Audio and audiovisual quality for mobile services. Harald Klaus – T-Systems Rapporteur of Q9/12 [email protected]. IP Cablecom and MEDIACOM 2004. Outline. Personal Profile Framework and Objectives of AQUAVIT Test beds of AQUAVIT - PowerPoint PPT PresentationTRANSCRIPT
1
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 2004
EURESCOM P905 (AQUAVIT)EURESCOM P905 (AQUAVIT)Audio and audiovisual quality Audio and audiovisual quality
for mobile servicesfor mobile services
Harald Klaus – T-SystemsRapporteur of Q9/12
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20042
OutlineOutline
Personal ProfileFramework and Objectives of AQUAVITTest beds of AQUAVIT Test methods developed in AQUAVITResults of InvestigationsConclusions
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20043
Personal ProfilePersonal Profile
Harald KlausStudied electrotechnical engineering
at the Technical University of BerlinResearch assistant for speech recognition
applications and speech quality assessmentHead of quality assessment of tele-services
at T-Systems Nova Berkom in BerlinRapporteur in ITU-T Study Group 12 for instrumental
speech quality assessment methodsRapporteur in ETSI Technical Committee STQ
4
The EURESCOM Project P905 The EURESCOM Project P905 (AQUAVIT)(AQUAVIT)
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20045
Framework of EURESCOM P905Framework of EURESCOM P905
AQUAVIT = Assessment of Quality for Audio-Visual Signals over IP and UMTSStart: January 1999End: December 20003 Partners...
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20046
Objectives of AQUAVITObjectives of AQUAVIT
Development of test beds for IP and UMTSDevelopment of subjective and objective test
methodsQuality assessment of typical audio-video
transmission scenarios for IP and UMTSCorrelation between subjective and objective
quality evaluations
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20047
Transm
ission errors
Access Unit Access UnitAccess Unit
Sync. Layer
SL-packetized stream
AU AU AU AU AU
Elementary Stream
Compression Layer
Elementary Stream
Encoders
MP
EG
-4
TestbedsTestbeds
Coding of signals for UMTS on send side
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20048
Executive
Dem
ux
Application
MediaStream BIFSDecoder
Root SceneObject
MediaStream
MediaObject
DecoderMediaStream
T
Presenter
VisualRenderer
AudioRenderer
T
Testbeds (2)Testbeds (2)Decoding of signals for UMTS on receive
side
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 20049
Testbeds (3)Testbeds (3)
IP TestbedRunning on standard PCs
in realtimeAdjustable set of
parameters forCoding/DecodingBit rateBit and Frame Errors Error distribution
Windows user interface
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200410
Sequence I
MO
S
Time
just one vote for the whole sequence
a) ordinary way of voting
b) simulation of continuos voting
SI Part a SI Part b SI Part c SI Part d SI Part e SI Part f
Six different votes for the sequence
MO
S
Time
Subjective test methodsSubjective test methodsSpeech quality evaluationSpeech quality evaluation
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200411
Subjective test methodsSubjective test methodsSpeech quality evaluationSpeech quality evaluation
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200412
Subjective test methodsSubjective test methodsAudio quality evaluationAudio quality evaluation
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200413
Subjective test methodsSubjective test methodsVideo & A/V quality evaluationVideo & A/V quality evaluation
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200414
• Questionnaire
• Conjoint analysis
• Focus groups
Subjective test methodsSubjective test methodsAcceptability testsAcceptability tests
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200415
Objective test methodsObjective test methods
TOSQA (Telecommunication objective speech quality assessment)
PEAQ (Perceptual evaluation of audio quality)
A new video quality assessment model developed by British Telecom
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200416
Experiment 1narrow band & wideband
speech
= 96.6%
1
2
3
4
5
1 2 3 4 5MOS
TM
OS
1
2
3
4
5
1 2 3 4 5MOS
TM
OS
Experiment 2aOnly narrow band
speech
= 95.4%
1
2
3
4
5
1 2 3 4 5MOS
TM
OS
1
2
3
4
5
1 2 3 4 5MOS
TM
OS
Experiment 2bOnly wideband
speech
= 98.2%
1
2
3
4
5
1 2 3 4 5MOS
TM
OS
1
2
3
4
5
1 2 3 4 5MOS
TM
OS
Results of investigations Results of investigations Correlation of subjective and objective speech Correlation of subjective and objective speech quality (TOSQA)quality (TOSQA)
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200417
010
2030
4050
6070
8090
100
Mushra
Peaq
010
2030
4050
6070
8090
100
Mushra
Peaq
Results of investigations Results of investigations Subjective test results - audio qualitySubjective test results - audio quality
Low quality conditions (average of all sequences)
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200418
0
20
40
60
80
100
120
Mushra
Peaq
0
20
40
60
80
100
120
Mushra
Peaq
Results of investigations Results of investigations Subjective test results - audio qualitySubjective test results - audio quality
High quality conditions (average of all sequences)
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200419
Results of investigations Results of investigations Acceptability test resultsAcceptability test results
Quality and cost are the equally important main factors to the population questioned.
Benefit of mobility does not compensate for any deficit in quality.
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200420
ConclusionsConclusions
AQUAVIT developed a bundle of improved subjective quality methods for assessment ofSpeech quality (narrowband and wideband)Audio quality (intermediate and high quality)Video and A/V quality (intermediate and high q.)
All tests use the same set of materialResults of all tests can be directly used for
comparisons
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 200421
T-Systems Nova BerkomHarald KlausHead of Quality Assessment ===!"§==Systems=
User-centred Quality Engineering.Quality assessmentat T-Systems Nova.Consulting
Test conceptsand test design Quality Monitoringand Reporting Process Optimization
Speech, audioand A/V quality measures According to international
recommendationsandstandards Optimized, efficienttestprocedures Tests withrepresentative user groups Intrusiveand non-intrusive
measurement methods Measurements with Head-and-Torso
simulators (HATS)
Our Labs High quality accordingto ITU
requirements for auditoryand instrumental
quality assessment for recordings usingHATS for speechand audioproduction
New test New test services services of T-System of T-System Nova are Nova are based on based on AQUAVIT AQUAVIT methodsmethodsand resultsand results