an evolutionary approach to speech quality estimation using genetic programming
DESCRIPTION
An Evolutionary Approach to Speech Quality Estimation Using Genetic ProgrammingTRANSCRIPT
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
AN EVOLUTIONARY APPROACH TO SPEECH
QUALITY ESTIMATIONUSING GENETIC PROGRAMMING
A. Raja1 A. Azad2 C. Flanagan1 C. Ryan2
1Wireless Access Research CentreDepartment of Electronic and Computer Engineering
2Bio-Computing and Developmental SystemsDepartment of Computer Science and Information Sysmtems
University of Limerick, Limerick, Ireland
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OUTLINE
1 MOTIVATIONThe Problem of Speech Quality AssessmentResearch Goal
2 VOIP SIMULATION ENVIRONMENTSimulation SystemNetwork Traffic Characteristics
3 GP EXPERIMENTS
4 TEST RESULTS
5 CONCLUSIONS
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OUTLINE
1 MOTIVATIONThe Problem of Speech Quality AssessmentResearch Goal
2 VOIP SIMULATION ENVIRONMENTSimulation SystemNetwork Traffic Characteristics
3 GP EXPERIMENTS
4 TEST RESULTS
5 CONCLUSIONS
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OUTLINE
1 MOTIVATIONThe Problem of Speech Quality AssessmentResearch Goal
2 VOIP SIMULATION ENVIRONMENTSimulation SystemNetwork Traffic Characteristics
3 GP EXPERIMENTS
4 TEST RESULTS
5 CONCLUSIONS
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OUTLINE
1 MOTIVATIONThe Problem of Speech Quality AssessmentResearch Goal
2 VOIP SIMULATION ENVIRONMENTSimulation SystemNetwork Traffic Characteristics
3 GP EXPERIMENTS
4 TEST RESULTS
5 CONCLUSIONS
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OUTLINE
1 MOTIVATIONThe Problem of Speech Quality AssessmentResearch Goal
2 VOIP SIMULATION ENVIRONMENTSimulation SystemNetwork Traffic Characteristics
3 GP EXPERIMENTS
4 TEST RESULTS
5 CONCLUSIONS
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
SPEECH QUALITY ASSESSMENT METHODOLOGIES
Two approaches to speech quality Assessment1 Subjective Assessment2 Objective Assessment
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
SUBJECTIVE ASSESSMENT OF SPEECH QUALITY
Speech quality is estimated by humans.Advantage – Reliable results.Limitations
1 Expensive2 Time Consuming3 Laborious4 Lack of Repeatability
Mean Opinion Score (MOS) is the measure of quality.1 – bad5 – Excellent
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OBJECTIVE ASSESSMENT OF SPEECH QUALITY
A computer automated fast and reliable program is used toassay human perception of speech qualityTwo approaches:
1 Intrusive Assessment2 Non-Intrusive Assessment
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OBJECTIVE ASSESSMENT OF SPEECH QUALITYINTRUSIVE ASSESSMENT
The signal under test is compared against a correspondingreference signal.Advantages:
1 The most reliable artificial means of estimating speechquality
2 Tests can be repeated easilyLimitations:
1 Consumes considerable computing resources.2 Is not useful for continuous monitoring of quality due to
requirement of a reference signal.
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OBJECTIVE ASSESSMENT OF SPEECH QUALITYINTRUSIVE ASSESSMENT
The signal under test is compared against a correspondingreference signal.Advantages:
1 The most reliable artificial means of estimating speechquality
2 Tests can be repeated easilyLimitations:
1 Consumes considerable computing resources.2 Is not useful for continuous monitoring of quality due to
requirement of a reference signal.
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OBJECTIVE ASSESSMENT OF SPEECH QUALITYITU-T P.862 (PESQ)
PESQ algorithm is the current ITU-T Recommendation forintrusive speech quality estimation.The speech signal is mapped from time domain totime-frequency representation using the psychophysicalequivalents of frequency and intensity.
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OBJECTIVE ASSESSMENT OF SPEECH QUALITYITU-T P.862 (PESQ)
It has shown a high correlation with various ITU-Tbenchmark tests.For 30 ITU-T subjective tests the Pearson’s CorrelationCoefficient (R) was 0.935
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OBJECTIVE ASSESSMENT OF SPEECH QUALITYNON-INTRUSIVE ASSESSMENT
A challenging problem since a reference is not available.Two approaches exist
1 Signal-based models2 Parametric models
Signal-based modelsRecent approaches are based on emulating
1 Human speech production model2 Psychoacoustic processing of human ear
ITU-T P.563 is the current Recommendation.
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OBJECTIVE ASSESSMENT OF SPEECH QUALITYNON-INTRUSIVE ASSESSMENT
A challenging problem since a reference is not available.Two approaches exist
1 Signal-based models2 Parametric models
Signal-based modelsRecent approaches are based on emulating
1 Human speech production model2 Psychoacoustic processing of human ear
ITU-T P.563 is the current Recommendation.
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
OBJECTIVE ASSESSMENT OF SPEECH QUALITYPARAMETRIC MEASUREMENT OF VOIP QUALITY
Functions of transport layer metrics and other measurablequantities.Cogent metrics may be:
Packet Loss RateVariable delay – jitterEnd-to-end delay. . .
Aimed at Real-time and continuous evaluation of quality
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
RESEARCH GOAL
Derivation of a VoIP listening Quality estimation model as afunction of transport layer metrics.Genetic Programming based Symbolic Regression is usedUsing the PESQ algorithm as the reference system
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
VOIP SIMULATION ENVIRONMENT
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
NETWORK TRAFFIC PARAMETERS
No. Parameter Name Abbreviation1 Bit-rate (kbps) br2 mean loss rate mlr3 mean burst length mbl4 Packetization Interval (ms) PI5 Frame duration (ms) fd
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
NETWORK TRAFFIC SCENARIOS
No. Parameter Range1 br G.729 (8 kbps), G.723.1 (6.3 kbps),
AMR 7.4 and 12.2 kbps2 mlr [0,2.5,3.5,. . . 15,20,25,. . . 40]%3 mbl 10, 50, 60, 70 and 80%4 PI 10-60 ms5 fd 10, 20, 30 ms
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
EXPERIMENTAL SETUP
GPLabFour GP Experiments were performed with variousconfigurationsCommonalities
Each experiment constituted 50 runsEach Run spanned 50 generations
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
GP EXPERIMENTSCOMMON PARAMETERS
Parameter ValueInitial Population Size 300Selection LPP TournamentTournament Size 2Genetic Operators Crossover and Subtree MutationInitial Operator probabilities 0.5 initial, adaptive onwardsSurvival Half ElitismFunction Set +, -, *, /, sin, cos, log2, log10,
loge, sqrt, power,Terminal Set Random numbers [0.0 . . . 1.0]
Integers [2 . . . 10]. mlrVAD,mblVAD, PI, br , fd
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
GP EXPERIMENTSEXPERIMENTAL DETAILS
Experiment 1:Fitness function – Mean Squared Error MSE
Experiment 2:Linear Scaling MSEs
MSEs(y , t) = 1/nn∑i
(ti − (a + byi))2 (1)
a = t − by ,b =cov(t , y)
var(y)(2)
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
GP EXPERIMENTSEXPERIMENTAL DETAILS
Experiment 1:Fitness function – Mean Squared Error MSE
Experiment 2:Linear Scaling MSEs
MSEs(y , t) = 1/nn∑i
(ti − (a + byi))2 (1)
a = t − by ,b =cov(t , y)
var(y)(2)
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
GP EXPERIMENTSEXPERIMENTAL DETAILS
Experiments 3 and 4Selection criterion based on Gustafson et al. was usedMating takes place between dissimilar individuals
Experiment 4:The Maximum tree depth was reduced to 7 from 17
The results were treated to Mann-Whitney-Wilcoxon Testfor significance AnalysisExperiment 4 was found to be significantly better overall.
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
GP EXPERIMENTSEXPERIMENTAL DETAILS
Experiments 3 and 4Selection criterion based on Gustafson et al. was usedMating takes place between dissimilar individuals
Experiment 4:The Maximum tree depth was reduced to 7 from 17
The results were treated to Mann-Whitney-Wilcoxon Testfor significance AnalysisExperiment 4 was found to be significantly better overall.
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
ON DATA COLLECTION
Nortel ND speech database containing high quality signalswith speech from 2 male and 2 female speakers was usedfor analysis.A total of 3360 distorted speech files were created for eachcombination of network traffic parameters.
1177 35% were used for training503 15% were used for testing1680 50% were used for speaker independent validation
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
VOIP QUALITY MONITORING MODELS
MOS − LQOGP = −2.46 × log(cos(log(br)) + mlrVAD
×(br + fd/10)) + 3.17 (3)
MOS − LQOGP = −2.99 × cos(0.91×√
sin(mlrVAD)
+mlrVAD + 8) + 4.20 (4)
Equation (3) Equation(4)Data MSEs σ MSEs σ
Training 0.0370 0.9634 0.0520 0.9481Testing 0.0387 0.9646 0.0541 0.9501Validation 0.0382 0.9688 0.0541 0.9531
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
SCATTER PLOTS
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
SCATTER PLOTSON PERFORMANCE OF ITU-T P.563
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
CONCLUSIONS
1 The model is a good approximation to PESQ.2 Suitable for real-time and non-intrusive estimation of
speech quality whereas PESQ is NOT.3 Simple models; depend on 3 and 1 variable respectively.4 Performs significantly better than ITU-T P.563
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
FUTURE GOALS
To include wide band codecs in the research.To develop a unified quality estimation model for narrowand wide band telephony
Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions
QUESTIONS