Magnus Schäfer

HEAD acoustics GmbH

ETSI STQ Workshop on Multimedia Quality

in Virtual, Augmented and other Realities

Sophia Antipolis, 10.05.17

Approaches for Testing and Evaluating

Car Audio Systems

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 2

For most car drivers and passengers today, the enjoyment of

music is an inherent part of their driving experience.

Continually increasing customer expectations regarding audio

quality

How to evaluate sound quality as perceived by the user?

Auditory testing

Instrumental evaluation

Introduction

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 3

Auditory Test Design – Recordings

Auditory evaluation of car audio systems

Concept is based on real music playback and DF-equalized

binaural recordings of head and torso simulator (HATS)

Measurements were conducted using common cars and their

original audio systems

Real (professional) music tracks as test stimuli: rock, pop, classic,

jazz, r&b, … Playback via CD, 7 tracks per DUT

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 4

Auditory Test Design – Procedure

Several auditory test methods and scales of different standards are

available for testing audio quality:

ACR, DCR, CCR, MUSHRA, …

Focus on large amount of test conditions

ACR test with 5-point scale acc. to ITU-T P.800

Loudness normalization to 23 sone (N5 acc. to ISO 532-1)

Hearing-adequate playback of binaural recordings

Anchors, arbitrary level offsets and other files included

161 samples in overall

Usually tests are conducted in a listening laboratory

Psychologic aspect of the listening test so far not considered:

Do test subjects behave differently in a real car?

Is the quality perception the same inside a real car?

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 5

Testing Environments – Laboratory

Everything is under control (light, seat, air condition, …)

very „clean“ environment

Idle noise < 20 dB(A)

Several terminals can be utilized simultaneously

But is it too unrealistic / artificial for car audio evaluation?

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 6

Testing Environments – Driving simulator

More „lively“ environment: street is projected on a screen in front of

the car (interaction possible but not part of the current evaluation)

Idle noise < 20 dB(A)

Only one test person at once

Not always accessible for subjective testing

More realistic for auditory evaluation?

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 7

Results of Auditory Test

Comparison of testing environments (details: Reimes et al. DAGA 2017)

Results highly correlate (RMSEStimuli = 0.236, RMSEDUT = 0.103)!

Level of significance of differences (ANOVA / t-Test) …

DUT 𝑝 = 0.84

Stimuli 𝑝 = 0.48

Votes 𝑝 = 0.046∗ (Cohen’s d: 𝑑 = 0.06 ≪ 0.2 not relevant)

No systematic under- or overestimation of perceived quality

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 8

Record audio signal at driver position

Comparison between input signal and recording

Focus here for illustration: Spatial properties

Approach for Instrumental Evaluation

Evaluation procedure

Audio recording

Binaural hearing model

Input and output

Analysis of differences

Mapping to perceptual scale

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 9

Coincidence-based model

Cross-correlation with inhibition [Lindemann86]

Frequency and lateralization weighting [SchäferBahramVary12]

Output: Correlogram

Additional derivative outputs possible

Binaural Hearing Model – Structure

Weighting and Averaging

Inhibited cross-correlation

Monaural ear modelMonaural ear model

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 10

Example: Correlograms for two car audio systems

Reference input signal

Classical music

Binaural Hearing Model – Output

Good performance Bad performance

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 11

Difference (and absolute values thereof) between correlograms of

recording and reference

Mean

Median

Percentiles

5th, 90th and 95th

Combination of several metrics

Random Forest Regression

Metrics and Regression

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 12

Entire listening test results used for training

Trained combination of all metrics can reproduce auditory results

Regression Results

memorize?

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 13

Listening test results split 50-50 between training and evaluation

Better than individual metrics – but far from satisfactory

Regression Results

Approaches for Testing and Evaluating Car Audio SystemsMagnus Schäfer 14

Conclusions

Setup of auditory testing for car audio systems based on binaural

recordings and hearing-adequate playback

Large auditory evaluation was conducted with 161 stimuli in two

test environments with 45 participants

Results between listening environments provide high correlation

No systematic differences can be observed in this study

Further auditory tests can be conducted in listening lab!

Spatial analysis of recordings from different car audio systems

Binaural Hearing Model

Neither individual metrics nor trained regression provide

satisfactory overall quality estimation

Using spatial properties is not enough

Magnus Schäfer

Telecom Research & Standardization

telecom@head-acoustics.de

www.head-acoustics.de © Copyright HEAD acoustics GmbH