1

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

AbstractPerception of speech under adverse listening conditions may be improved by processing it to incorporate properties of clear speech. It needs automated detection of stop land-marks and enhancement of bursts and transition segments. A technique for accurate detection of stop landmarks in continuous speech based on parameters derived from Gaussian mixture modeling (GMM) of log magnitude spectrum is presented. Applying the technique on sentences from the TIMIT database resulted in burst detection rates of 98, 97, 95, 90, and 73 % at temporal accuracies of 30, 20, 15, 10, and 5 ms respectively.

2

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

1. INTRODUCTION

Acoustic LandmarksRegions with concentration of phonetic information, important for speech perception

Stop Landmarks Closure Release burst Onset of voicing

Closure ▲ Release burst ▲ ▲ Onset of voicing/apa/

3

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

Problems in Stop PerceptionPerception of transient sounds with low intensityseverely affected by noise / hearing impairment

Clear Speech Style adapted by speakers under noisy conditions (~17 % more intelligible than conversational speech) Acoustic landmarks modified in duration & intensity

◄ Conversational▼Clear

‘the book tells a story’

4

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

Speech Intelligibility Enhancement Using Properties of Clear Speech

Automated detection of landmarks with Good temporal accuracy High detection rate and low false detections

Modification of speech characteristics aroundthe stop landmarks

5

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

Some Earlier Landmark Detection Techniques

Liu (1996): Rate-of-rise measures of parameters from a set of fixed spectral bands. Detection rate: 84 % at 20-30 ms, ~50 % at 5-10 ms.

Niyogi & Sondhi (2002): Optimal filtering approach with log energy, log energy in the band > 3 kHz & Wiener entropy. Detection rate 90 % at 20 ms.

6

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

ObjectiveDetection of stop landmarks using Gaussian mixture modeling (GMM) of speech spectrum

▪ for improving the temporal accuracy of detection and reducing insertion errors

▪ with adaptation to speech variability

▪ for enhancing burst and transition segments to improve speech intelligibility under adverse listening conditions

7

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

2. GAUSSIAN MIXTURE MODELING OF SHORT-TIME SPEECH SPECTRUM

Approximation of spectrum using a weighted sum of Gaussian functions

Means Variances Mixture weights

Good spectral approximation with 4 or 5 Gaussians (approximating the spectral resonances)

Adaptive to speech variability

1( ) ( , , )

Mn gn gn gn

gS k w G k

8

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

Spectral Modeling Short-time log magnitude spectrum of speech signal (S.R. = 10 kHz)

6 ms Hanning windowed frames (for suppressing the harmonic structure)

1 frame per ms (for tracking abrupt variations) 512-point DFT

Estimation of GMM parameters using Expectation Maximization (EM) algorithm

9

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

Estimation of GMM Parameters Spectrum treated as histogram with rectangular

bins placed at each frequency index

Iterative computation of parameters as maximum likelihood estimates Initialization

Means: Average formant frequencies [600, 1200, 2400, 3600 Hz]

Variances: Extreme formant bandwidths [160, 200, 300, 400 Hz]

Mixture weights: Equal for all Gaussians

Number of iterations: ≤ 12

10

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

Example: Modeling for a segment of vowel /a/

Modeling of a segment of vowel /a/: (a) windowed segment of 6 ms, (b) log magnitude spectrum (in dB), (c) smoothened spectrum (in dB), (d) GMM approximated spectrum with dotted lines indicating the individual Gaussian components.

Ag(n)

g(n)g(n)

11

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

3. DETECTION OF STOP LANDMARKSDetection based on

Rate of change (ROC) of GMM parameters

Voicing onset offset detector Spectral flatness measure

^( ) ( ( ))( )( )

g n g

g

g

A n S nnn

12

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

SpectralFlatnessMeasure

VoicingOnset /Offset

Detection GMM ParameterEstimation

Landmark DetectionSFM(n)+g, -g peaks

Speech Signal

ROC(n)

Median Smoothing &ROC computation

Windowing & Log. mag.Spectrum Computation

GMM-ROC

13

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

GMM Rate of Change Ag, g, g smoothened by 30-point median filter ROC: First difference (time step = 2 ms)

ROC Peak → Possible location of burst onset

( ) ( ) ( ) ( ) /c Ar n r n r n r n R

4

1( ) ( ) ( )A g g s

gr n A n A n n

4

1( ) ( ) ( )g g s

gr n n n n

4

1( ) ( ) ( )g g s

gr n n n n

14

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

Voicing Onset-Offset Detection [Liu, 1996] Energy variations E(n) in 0:400 Hz band (6 ms Hanning windowed segments, every 1 ms) Rate-of-rise re(n) with 26 ms time-step Voicing onset [+g]: re(n) +9dB Voicing offset [-g]: re(n) -9dB

Spectral Flatness Measure [Skowronski & Harris, 2006]

(20 ms Hanning windowed segments, every 1 ms) Fricative segments: SFM 1 Voiced segments: SFM 0

2// 2 / 22 2

1 1SFM( ) | ( ) | (2 / ) | ( ) |

NN Nn n

k kn X k N X k

15

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

Stop Landmark Detection For a voicing onset [+g] or voicing offset [-g] at t,

locate the preceding [+g] or [-g] If [-g] at t0,select GMM ROC peak at tb during (t0-50, t ms), Else select GMM ROC peak at tb during (t-50, t ms) as the burst candidate.

A burst is declared, if {SFM > 0.5 for 1 ms during (tb-15, tb+15 ms)} and {each of the norm. ampl. A2, A3, A4 < 0.5 for at least 10 ms during (t0, tb)}.

For burst at tb, closure is located at t0, and voicing onset at t.

16

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

/apa/: Waveform (a), Gaussian parameter tracks (b: 1st, c: 2nd, d: 3rd, e: 4th).

(a)

(b)

(c)

(d)

(e)

Ag(n)

g(n)

g(n)

17

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

/apa/: Waveform (a), Spectrogram (b), GMM spectrogram (c), Gaussian ROC (d)

(a)

(b)

(c)

(d)

18

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

50 100 150 200 250 300 350 ms-101

50 100 150 200 250 300 350-20

020

50 100 150 200 250 300 3500

0.51

50 100 150 200 250 300 3500

0.51

50 100 150 200 250 300 3500

0.51

Time(ms)

-g+g

(a)

(b)

(c)

(d)

(e)A2

A4

A3

/apa/: Waveform (a), -g, +g peaks (b), SFM (c), GMM ROC (d), Normalized Gaussian amplitudes for Gaussian 2, 3, 4 (e)

tb

tt0

ROC peak

SFM

19

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

4. TEST RESULTSComparison with manually labeled landmarks VCV utterances

▪ Stops /b/, /d/, /g/, /p/, /t/, /k/ & vowels /a/, /i/, /u/ ▪ 10 speakers (5 F, 5 M)

TIMIT sentences▪ 50 sentences▪ 5 speakers (3 F, 2 M)

20

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

0

20

40

60

80

100

5 10 15 20 30Temporal Accuracy (ms)

Dete

ctio

n (%

)

Closure Burst Voicing Onset

36

9076

64

92 93

73

93 9883

96 98 999893

Det. Rates for VCV Utterances

21

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

0

20

40

60

80

100

5 10 15 20 30Temporal Accuracy (ms)

Dete

ctio

n (%

)Closure Burst Voicing Onset

Det. Rates for TIMIT Sentences

Insertions : 13 % (Clicks, glottal stops : 8 %, Vowel-semivowel : 4 %, Stop to /l/, /r/ : 1 %)

19

73

45 40

90

7163

9580

919790

98 9682

22

A. R

. Jay

an, P

. C. P

and e

y, E

E De

p t. ,

IIT

Bom

b ay

5. CONCLUSIONDetection rate obtained using GMM based technique: comparable to other methods at 20-30 ms temporal accuracy, better at 10-15 ms.