1

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

AbstractPerception of speech under adverse listening conditions may be improved by processing it to incorporate properties of clear speech. It needs automated detection of stop land-marks and enhancement of bursts and transition segments. A technique for accurate detection of stop landmarks in continuous speech based on parameters derived from Gaussian mixture modeling (GMM) of log magnitude spectrum is presented. Applying the technique on sentences from the TIMIT database resulted in burst detection rates of 98, 97, 95, 90, and 73 % at temporal accuracies of 30, 20, 15, 10, and 5 ms respectively.

2

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

1. INTRODUCTION

Acoustic LandmarksRegions with concentration of phonetic information, important for speech perception

Stop Landmarks Closure Release burst Onset of voicing

Closure ▲ Release burst ▲ ▲ Onset of voicing/apa/

3

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

Problems in Stop PerceptionPerception of transient sounds with low intensityseverely affected by noise / hearing impairment

Clear Speech Style adapted by speakers under noisy conditions (~17 % more intelligible than conversational speech) Acoustic landmarks modified in duration & intensity

◄ Conversational

▼Clear

‘the book tells a story’

4

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

Speech Intelligibility Enhancement Using Properties of Clear Speech

Automated detection of landmarks with Good temporal accuracy High detection rate and low false detections

Modification of speech characteristics aroundthe stop landmarks

5

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

Some Earlier Landmark Detection Techniques

Liu (1996): Rate-of-rise measures of parameters from a set of fixed spectral bands. Detection rate: 84 % at 20-30 ms, ~50 % at 5-10 ms.

Niyogi & Sondhi (2002): Optimal filtering approach with log energy, log energy in the band > 3 kHz & Wiener entropy. Detection rate 90 % at 20 ms.

6

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

Objective

Detection of stop landmarks using Gaussian mixture modeling (GMM) of speech spectrum

▪ for improving the temporal accuracy of detection and reducing insertion errors

▪ with adaptation to speech variability

▪ for enhancing burst and transition segments to improve speech intelligibility under adverse listening conditions

7

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

2. GAUSSIAN MIXTURE MODELING OF SHORT-TIME SPEECH SPECTRUM

Approximation of spectrum using a weighted sum of Gaussian functions

Means Variances Mixture weights

Good spectral approximation with 4 or 5 Gaussians (approximating the spectral resonances)

Adaptive to speech variability

1( ) ( , , )

M

n gn gn gng

S k w G k

8

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

Spectral Modeling

Short-time log magnitude spectrum of speech signal (S.R. = 10 kHz)

6 ms Hanning windowed frames (for suppressing the harmonic structure)

1 frame per ms (for tracking abrupt variations)

512-point DFT

Estimation of GMM parameters

using Expectation Maximization (EM) algorithm

9

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

Estimation of GMM Parameters

Spectrum treated as histogram with rectangular bins placed at each frequency index

Iterative computation of parameters as maximum likelihood estimates Initialization

Means: Average formant frequencies [600, 1200, 2400, 3600 Hz]

Variances: Extreme formant bandwidths [160, 200, 300, 400 Hz]

Mixture weights: Equal for all Gaussians

Number of iterations: ≤ 12

10

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

Example: Modeling for a segment of vowel /a/

Modeling of a segment of vowel /a/: (a) windowed segment of 6 ms, (b) log magnitude spectrum (in dB), (c) smoothened spectrum (in dB), (d) GMM approximated spectrum with dotted lines indicating the individual Gaussian components.

Ag(n)

g(n)

g(n)

11

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

3. DETECTION OF STOP LANDMARKS

Detection based on Rate of change (ROC) of GMM parameters

Voicing onset offset detector Spectral flatness measure

^

( ) ( ( ))( )( )

g n g

g

g

A n S nnn

12

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

SpectralFlatnessMeasure

VoicingOnset /Offset

Detection GMM ParameterEstimation

Landmark DetectionSFM(n)+g, -g peaks

Speech Signal

ROC(n)

Median Smoothing &ROC computation

Windowing & Log. mag.Spectrum Computation

GMM-ROC

13

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

GMM Rate of Change Ag, g, g smoothened by 30-point median filter ROC: First difference (time step = 2 ms)

ROC Peak → Possible location of burst onset

( ) ( ) ( ) ( ) /c Ar n r n r n r n R

4

1( ) ( ) ( )A g g s

gr n A n A n n

4

1( ) ( ) ( )g g s

gr n n n n

4

1( ) ( ) ( )g g s

gr n n n n

14

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

Voicing Onset-Offset Detection [Liu, 1996] Energy variations E(n) in 0:400 Hz band (6 ms Hanning windowed segments, every 1 ms)

Rate-of-rise re(n) with 26 ms time-step Voicing onset [+g]: re(n) +9dB Voicing offset [-g]: re(n) -9dB

Spectral Flatness Measure [Skowronski & Harris, 2006]

(20 ms Hanning windowed segments, every 1 ms)

Fricative segments: SFM 1 Voiced segments: SFM 0

2// 2 / 22 2

1 1SFM( ) | ( ) | (2 / ) | ( ) |

NN N

n nk k

n X k N X k

15

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

Stop Landmark Detection

For a voicing onset [+g] or voicing offset [-g] at t, locate the preceding [+g] or [-g] If [-g] at t0,select GMM ROC peak at tb during (t0-50, t ms), Else select GMM ROC peak at tb during (t-50, t ms) as the burst candidate.

A burst is declared, if {SFM > 0.5 for 1 ms during (tb-15, tb+15 ms)} and {each of the norm. ampl. A2, A3, A4 < 0.5 for at least 10 ms during (t0, tb)}.

For burst at tb, closure is located at t0, and voicing onset at t.

16

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

/apa/: Waveform (a), Gaussian parameter tracks (b: 1st, c: 2nd, d: 3rd, e: 4th).

(a)

(b)

(c)

(d)

(e)

Ag(n)

g(n)

g(n)

17

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

/apa/: Waveform (a), Spectrogram (b), GMM spectrogram (c), Gaussian ROC (d)

(a)

(b)

(c)

(d)

18

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

50 100 150 200 250 300 350 ms-101

50 100 150 200 250 300 350-20

020

50 100 150 200 250 300 3500

0.51

50 100 150 200 250 300 3500

0.51

50 100 150 200 250 300 3500

0.5

1

Time(ms)

-g+g

(a)

(b)

(c)

(d)

(e)

A2

A4

A3

/apa/: Waveform (a), -g, +g peaks (b), SFM (c), GMM ROC (d), Normalized Gaussian amplitudes for Gaussian 2, 3, 4 (e)

tb

tt0

ROC peak

SFM

19

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

4. TEST RESULTS

Comparison with manually labeled landmarks

VCV utterances ▪ Stops /b/, /d/, /g/, /p/, /t/, /k/ & vowels /a/, /i/, /u/ ▪ 10 speakers (5 F, 5 M)

TIMIT sentences▪ 50 sentences▪ 5 speakers (3 F, 2 M)

20

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

0

20

40

60

80

100

5 10 15 20 30Temporal Accuracy (ms)

Det

ecti

on

(%

)

Closure Burst Voicing Onset

36

9076

64

92 93

73

93 9883

96 98 999893

Det. Rates for VCV Utterances

21

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

0

20

40

60

80

100

5 10 15 20 30Temporal Accuracy (ms)

Det

ecti

on

(%

)Closure Burst Voicing Onset

Det. Rates for TIMIT Sentences

Insertions : 13 % (Clicks, glottal stops : 8 %, Vowel-semivowel : 4 %, Stop to /l/, /r/ : 1 %)

19

73

45 40

90

7163

95

809197

9098 96

82

22

A. R

. J a

yan, P. C

. Pandey, EE D

ept.

, IIT

Bom

bay

5. CONCLUSION

Detection rate obtained using GMM based technique: comparable to other methods at 20-30 ms temporal accuracy, better at 10-15 ms.