variational sampling approaches to word confusability

Variational sampling approachesto word confusability

John R. Hershey, Peder A. Olsen and Ramesh A. Gopinath

{jrhershe,pederao,rameshg}@us.ibm.com

IBM, T. J. Watson Research Center

Information Theory and Applications, 2/1-2007 – p.1/31

Abstract

In speech recognition it is often useful to determine how confusable two words are. For

speech models this comes down to computing the Bayes error between two HMMs. This

problem is analytically and numerically intractable. A common alternative, that is numeri-

cally approachable, uses the KL divergence in place of the Bayes error. We present new

approaches to approximating the KL divergence, that combine variational methods with

importance sampling. The Bhattacharyya distance – a closer cousin of the Bayes error –

turns out to be even more amenable to our approach. Our experiments demonstrate an

improvement of orders of magnitude in accuracy over conventional methods.

Outline

• Acoustic Confusability

Outline

• Acoustic Confusability• Divergence Measures for Distributions

Outline

• Acoustic Confusability• Divergence Measures for Distributions• KL Divergence: Prior Art

Outline

• Acoustic Confusability• Divergence Measures for Distributions• KL Divergence: Prior Art• KL Divergence: Variational Approximations

Outline

• Acoustic Confusability• Divergence Measures for Distributions• KL Divergence: Prior Art• KL Divergence: Variational Approximations• KL Divergence: Empirical Evaluations

Outline

• Acoustic Confusability• Divergence Measures for Distributions• KL Divergence: Prior Art• KL Divergence: Variational Approximations• KL Divergence: Empirical Evaluations• Bhattacharyya: Monte Carlo Approximation

Outline

• Acoustic Confusability• Divergence Measures for Distributions• KL Divergence: Prior Art• KL Divergence: Variational Approximations• KL Divergence: Empirical Evaluations• Bhattacharyya: Monte Carlo Approximation• Bhattacharyya: Variational Approximation

Outline

• Acoustic Confusability• Divergence Measures for Distributions• KL Divergence: Prior Art• KL Divergence: Variational Approximations• KL Divergence: Empirical Evaluations• Bhattacharyya: Monte Carlo Approximation• Bhattacharyya: Variational Approximation• Bhattacharyya: Variational Monte Carlo Approximation

Outline

• Acoustic Confusability• Divergence Measures for Distributions• KL Divergence: Prior Art• KL Divergence: Variational Approximations• KL Divergence: Empirical Evaluations• Bhattacharyya: Monte Carlo Approximation• Bhattacharyya: Variational Approximation• Bhattacharyya: Variational Monte Carlo Approximation• Bhattacharyya: Empirical Evaluations

Outline

• Acoustic Confusability• Divergence Measures for Distributions• KL Divergence: Prior Art• KL Divergence: Variational Approximations• KL Divergence: Empirical Evaluations• Bhattacharyya: Monte Carlo Approximation• Bhattacharyya: Variational Approximation• Bhattacharyya: Variational Monte Carlo Approximation• Bhattacharyya: Empirical Evaluations• Future Directions

A Toy Version of the Confusability Problem

−5 −4 −3 −2 −1 0 1 2 3 4 50

probability density plot

N(x;−2,1)

• X is an acoustic feature vector representing a speech class, say "E", with pdff(x) = N (x;−2, 1).

−5 −4 −3 −2 −1 0 1 2 3 4 50

N(x;−2,1)

N(x;2,1)

• Another speech class, "O", is described by pdf g(x) = N (x; 2, 1).

−5 −4 −3 −2 −1 0 1 2 3 4 50

N(x;−2,1)

N(x;2,1)

• The asymmetric error is the probability that one class, "O", will be mistaken for theother, "E", when classifying according to Ae(f, g) =

Rf(x)1g(x)≥f(x)(x)dx.

−5 −4 −3 −2 −1 0 1 2 3 4 50

N(x;−2,1)

N(x;2,1)

• The Bayes Error is the total classification error Be(f, g) =

Rmin(f(x), g(x))dx.

−5 −4 −3 −2 −1 0 1 2 3 4 50

N(x;−2,1)

N(x;2,1)

(fg)1/2

• The Bayes Error is the total classification error Be(f, g) =

Rmin(f(x), g(x))dx.

Word Models

A word is modeled using its pronunciation(s) and an HMM. Asan example the word DIAL has a pronunciation D AY AX L andHMM F .

D AY AX L

Word Models

D AY AX L

CALL has a pronunciation K AO L and HMM G

K AO L

Word Models

D AY AX L

CALL has a pronunciation K AO L and HMM G

K AO L

Each node in the HMM has a GMM associated with it. The word

confusability is the Bayes error Be(F,G). This quantity is too hard

to compute!!

The Edit Distance

DIAL CALL edit op. cost

D K substitution 1

AY ins/del 1

AX AO substitution 1

L L none 0

Total cost 3

The Edit Distance

DIAL CALL edit op. cost

D K substitution 1

AY ins/del 1

AX AO substitution 1

L L none 0

Total cost 3

The edit distance is the shortest path in the graph:D AY AX L

1 1 1 1

1 1 1 1 1

1 1 1 1

1 1 1 0

Better ways

Other techniques use weights on the edges. Acousticperplexity and Average Divergence Distance are variantsto this paradigm that use approximations to the KL

divergence as weights.

Bayes Error

We use Bayes Error approximations for each pair of GMMs inthe Cartesian HMM products:

D : KI

D : AO

AY : K

AY : AO

AY : L

AX : K

AX : AO

AX : L

L : AO

D : AO

AY : K

AY : AO

AY : L

AX : K

AX : AO

AX : L

L : AO

Gaussian Mixture Models

Each node in the Cartesian HMM product corresponds to a pairof Gaussian Mixture models f and g. We write

f(x) =∑

πafa(x), where fa(x) = N (x;µa; Σa),

f(x) =∑

g(x) =∑

ωbgb(x), where gb(x) = N (x;µb; Σb).

f(x) =∑

g(x) =∑

ωbgb(x), where gb(x) = N (x;µb; Σb).

The high dimensionality of x ∈ Rd, d = 39, makes numerical

integration difficult.

Bayes, Bhattacharyya, Chernoff and Kullback-Leibler

• Bayes error: Be(f, g) =∫

min(f(x), g(x))dx

• Kullback Leibler divergence: D(f‖g) =∫

f(x) log f(x)g(x)dx

min(f(x), g(x))dx

f(x) log f(x)g(x)dx

• Bhattacharyya distance: B(f, g) =∫√

f(x)g(x)dx

min(f(x), g(x))dx

f(x) log f(x)g(x)dx

f(x)g(x)dx

• Chernoff distance: C(f, g) = max0≤s≤1Cs(f, g) andCs(f, g) =

f(x)sg(x)1−sdx, 0 ≤ s ≤ 1.

min(f(x), g(x))dx

f(x) log f(x)g(x)dx

f(x)g(x)dx

• Chernoff distance: C(f, g) = max0≤s≤1Cs(f, g) andCs(f, g) =

f(x)sg(x)1−sdx, 0 ≤ s ≤ 1.

Why these? — For a pair of single Gaussians f and g we can

compute D(f‖g), B(f, g) and Cs(f, g) analytically.

Connections

• Perimeter divergence (power mean):

Pα(f, g) =

Z �1

2f(x)α +

2g(x)α

� 1α

We have Be(f, g) = P−∞(f, g), B(f, g) = P0(f, g).

Connections

Pα(f, g) =

Z �1

2f(x)α +

2g(x)α

� 1α

• Rényi generalised divergence of order s:

Ds(f‖g) =1

s− 1log

Zf(x)sg(x)1−sdx.

We have D1(f‖g) = D(f‖g) and Ds(f‖g) = 1s−1

logCs(f‖g).

Connections

Pα(f, g) =

Z �1

2f(x)α +

2g(x)α

� 1α

• Rényi generalised divergence of order s:

Ds(f‖g) =1

s− 1log

Zf(x)sg(x)1−sdx.

We have D1(f‖g) = D(f‖g) and Ds(f‖g) = 1s−1

logCs(f‖g).• Generalisation:

Gα,s(f‖g) =1

s− 1log

Z(sf(x)α + (1 − s)g(x)α)

Gα,s(f‖g) connects logBe(f, g), D(f‖g), logB(f, g) and Cs(f, g).

Relations and Inequalities

• Gα, 12

(f‖g) = −2 logPα(f, g), G0,s(f‖g) = Ds(f‖g) and

G−∞,s(f‖g) = Be(f, g).

• Gα, 12

G−∞,s(f‖g) = Be(f, g).

• Pα(f, g) ≤ Pβ(f, g) for α ≤ β andB(f, g) ≤

Pα(f, g)P−α(f, g).

• Gα, 12

G−∞,s(f‖g) = Be(f, g).

• Pα(f, f) = 1, Gα,s(f, f) = 0, P−∞(f, g) + P∞(f, g) = 2,Be(f, g) = 2 − P∞(f, g).

• Gα, 12

G−∞,s(f‖g) = Be(f, g).

• B(f, g) = B(g, f), Be(f, g) = Be(g, f), D(f‖g) 6= D(g‖f).

• Gα, 12

G−∞,s(f‖g) = Be(f, g).

• B(f, g) = B(g, f), Be(f, g) = Be(g, f), D(f‖g) 6= D(g‖f).

• D(f‖g) ≥ 2 logB(f, g) ≥ 2 logBe(f, g),Be(f, g) ≤ B(f, g) ≤

Be(f, g)(2 −Be(f, g)),Be(f, g) ≤ C(f, g) ≤ B(f, g), C0(f, g) = C1(f, g) = 1 andB(f, g) = C1/2(f, g).

and so on.

The KL Divergence of a GMM

• Monte Carlo sampling : Draw n samples from f . Then

D(f‖g) ≈ 1

logf(xi)

with error O(1/√n).

The KL Divergence of a GMM

• Monte Carlo sampling : Draw n samples from f . Then

D(f‖g) ≈ 1

logf(xi)

with error O(1/√n).

• Gaussian Approximation: Approximate f with a gaussian fwhose mean and covariance matches the total mean andcovariance of f . Same for g and g, then useD(f‖g) ≈ D(f‖g)

µf =∑

πaµa

Σf =∑

πa(Σa + (µa − µf )(µa − µf )T ).

Unscented Approximation

It is possible to pick 2d “sigma” points {xa,k}2dk=1 such that

fa(x)h(x)dx = 12d

∑2dk=1 h(xa,k) is exact for all quadratic

functions h. One choice of sigma points is

xa,k = µa +√

dλa,k ea,k

xa,d+k = µa −√

dλa,k ea,k,

This is akin to gaussian quadrature.

Matched Bound Approximation

Match the closest pairs of gaussians

m(a) = argminb(D(fa‖gb) − log(ωb)).

Goldberger’s approximate formula is:

D(f‖g) ≈ DGoldberger(f‖g) =∑

D(fa‖gm(a)) + logπa

ωm(a)

Analogous to the chain rule for relative entropy.

Matched Bound Approximation

Match the closest pairs of gaussians

m(a) = argminb(D(fa‖gb) − log(ωb)).

Goldberger’s approximate formula is:

D(f‖g) ≈ DGoldberger(f‖g) =∑

D(fa‖gm(a)) + logπa

ωm(a)

Analogous to the chain rule for relative entropy.

Min Approximation: D(f‖g) ≈ mina,bD(fa‖gb) is an approximation

in the same spirit.

Variational Approximation

Let 1 =∑

b φb|a be free parameters

f log g =∑

fa log∑

φb|aωbgbφb|a

Introduce the variational parameters

Let 1 =∑

f log g =∑

fa log∑

φb|aωbgbφb|a

≥∑

φb|a logωbgbφb|a

Use Jensen’s inequality to interchange log and∑

b φb|a

Let 1 =∑

f log g =∑

fa log∑

φb|aωbgbφb|a

≥∑

φb|a logωbgbφb|a

πa∑

log(ωb/φb|a) +

fa log gb

Simplify expression after using Jensen

Let 1 =∑

f log g =∑

fa log∑

φb|aωbgbφb|a

≥∑

φb|a logωbgbφb|a

πa∑

log(ωb/φb|a) +

fa log gb

Maximize over φb|a and do the same for∫

f log f to get

Let 1 =∑

f log g =∑

fa log∑

φb|aωbgbφb|a

≥∑

φb|a logωbgbφb|a

πa∑

log(ωb/φb|a) +

fa log gb

Maximize over φb|a and do the same for∫

f log f to get

D(f‖g) ≈ Dvar(f‖g) =∑

πa log

a′ πa′e−D(fa‖fa′ )

b ωbe−D(fa‖gb)

Variational Upper Bound

Variational parameters:∑

b φb|a = πa and∑

a ψa|b = ωb.Gaussian replication:

f =∑

a πafa =∑

ab φb|afag =

b ωbgb =∑

ba ψa|bgb.

a ψa|b = ωb.Gaussian replication:

f =∑

a πafa =∑

ab φb|afag =

b ωbgb =∑

ba ψa|bgb.

D(f‖g) =

f log(f/g)

= −∫

ab ψa|bgbf

Introduce the variational parameters ψa|b.

a ψa|b = ωb.

D(f‖g) = −∫

ab ψa|bgbf

= −∫

φb|afaf

ψa|bgbφb|afa

Prepare to use Jensen’s inequality.

a ψa|b = ωb.

D(f‖g) = −∫

ab φb|afa∑

ab ψa|bgb

= −∫

φb|afaf

ψa|bgbφb|afa

≤∑

fa log

ψa|bgbφb|afa

Interchange log and∑

abφb|afa

a ψa|b = ωb.

D(f‖g) = −∫

ab φb|afa∑

ab ψa|bgb

≤∑

fa log

ψa|bgbφb|afa

= D(φ‖ψ) +∑

φb|aD(fa‖gb)

The chain rule for relative entropy for mixtures with unequal num-

ber of components!!

Optimize variational bound

D(f‖g) ≤ D(φ‖ψ) +∑

φb|aD(fa‖gb)

with respect to constraints∑

a ψa|b = ωb.

D(f‖g) ≤ D(φ‖ψ) +∑

φb|aD(fa‖gb)

a ψa|b = ωb.Fix φ, find optimal value for ψ

ψa|b =ωbφb|a∑

a′ φb|a′

Fix ψ, find optimal value for φ:

φb|a =πaψa|be−D(fa‖gb)

b′ ψa|b′e−D(fa‖gb′ )

Iterate a few times to find optimal solution!

Comparison of KL divergence methods (1)

Histograms of difference between closed form approximationsand Monte Carlo Sampling with 1 million samples:

−60 −50 −40 −30 −20 −10 0 10 200

Deviation from DMC(1M)

bability

zeroproduct

mingaussian

variational

Comparison of KL divergence methods (2)

Histograms of difference between various methodsand Monte Carlo Sampling with 1 million samples:

−5 −4 −3 −2 −1 0 1 2 3 4 50

Deviation from DMC(1M)

babilit

MC(2dn)

unscentedgoldberger

variationalupper

Summary of KL divergence methods

• Monte Carlo Sampling – arbitrary accuracy, arbitrary cost

• Gaussian Approximation – not cheap, not good, closed form

• Min Approximation – cheap, but not good

• Unscented Approximation – not as good as MC per cost

• Goldberger Approximation – cheap, good, not closed form

• Variational Approximation – cheap, good, closed form

• Variational Upper Bound – cheap, good, upper-bound

Comparison of KL, Bhattacharyya, Bayes

Bhattacharyya distance

The Bhattacharyya distance, B(f, g) =∫ √

fg, can be estimateusing Monte Carlo sampling from an arbitrary distribution h:

B(f, g) ≈ Bh =1

f(xi)g(xi)

h(xi),

where {xi}ni=1 are sampled from h.

B(f, g) ≈ Bh =1

f(xi)g(xi)

h(xi),

where {xi}ni=1 are sampled from h. The estimators areunbiased, E[Bh] = B(f, g), with variance

var(Bh) =1

h− (B(f, g))2

B(f, g) ≈ Bh =1

f(xi)g(xi)

h(xi),

where {xi}ni=1 are sampled from h. The estimators areunbiased, E[Bh] = B(f, g), with variance

var(Bh) =1

h− (B(f, g))2

h = f gives var(Bf ) = 1−(B(f,g))2

h = f+g2 gives var(B f+g

) =2fg

f+g−(B(f,g))2

And var(B f+g

) ≤ var(Bf ) (Harmonic–Arithmetic inequality).

Best sampling distribution

We can find the “best” sampling distribution h by minimizing thevariance of Bh subject to the constraints h ≥ 0 and

h = 1.

The solution is h =√fg√fg

, var(Bh) = 0.

h = 1.

, var(Bh) = 0.

Unfortunately, using this h requires

• Computing the quantity, B(f, g) =∫ √

fg, that we are tryingto compute in the first place.

• Sample from√fg.

h = 1.

, var(Bh) = 0.

Unfortunately, using this h requires

• Computing the quantity, B(f, g) =∫ √

fg, that we are tryingto compute in the first place.

• Sample from√fg.

We will use variational techniques to “approximate”√fg with

some unnormalized h that can be analytically integrated to give a

genuine pdf h.

Bhattacharyya Variational Upper Bound

f =∑

a πafa =∑

ab φb|afa∑

b φb|a = πa

g =∑

b ωbgb =∑

ba ψa|bgb∑

a ψa|b = ωb.

f =∑

a πafa =∑

ab φb|afa∑

b φb|a = πa

g =∑

b ωbgb =∑

ba ψa|bgb∑

a ψa|b = ωb.

B(f, g) =

ab ψa|bgbf

Introduce the variational parameters ψa|b.

f =∑

a πafa =∑

ab φb|afa∑

b φb|a = πa

g =∑

b ωbgb =∑

ba ψa|bgb∑

a ψa|b = ωb.

B(f, g) =

ab ψa|bgbf

φb|afaf

ψa|bgbφb|afa

Prepare to use Jensen’s inequality.

f =∑

a πafa =∑

ab φb|afa∑

b φb|a = πa

g =∑

b ωbgb =∑

ba ψa|bgb∑

a ψa|b = ωb.

B(f, g) =

ab ψa|bgbf

φb|afaf

ψa|bgbφb|afa

≥∑

ψa|bgbφb|afa

Interchange√· and

abφb|afa

f =∑

a πafa =∑

ab φb|afa∑

b φb|a = πa

g =∑

b ωbgb =∑

ba ψa|bgb∑

a ψa|b = ωb.

B(f, g) =

ab ψa|bgbf

≥∑

ψa|bgbφb|afa

φb|aψa|bB(fa, gb)

An inequality linking the mixture Bhattacharyya distance to the

component distances!!

B(f, g) ≥∑

φb|aψa|bB(fa, gb)

a ψa|b = ωb.

B(f, g) ≥∑

φb|aψa|bB(fa, gb)

a ψa|b = ωb.Fix φ, find optimal value for ψ

ψa|b =ωbφb|a(B(fa, gb))

a′ φb|a′(B(fa′ , gb))2.

Fix ψ, find optimal value for φ:

φb|a =πaψa|b(B(fa, gb))

b′ ψa|b′(B(fa, gb′))2.

Iterate to find optimal solution!Information Theory and Applications, 2/1-2007 – p.26/31

Variational Monte Carlo Sampling

Write the variational estimate

V (f, g) =

φb|aψa|bB(fa, gb) =

qφb|aψa|b

pfagb =

Here h =

φb|aψa|b

√fagb is an unnormalized approximation of the optimal

sampling distribution,√fg/

R √fg.

R RX q RXp

V (f, g) =

qφb|aψa|b

pfagb =

Here h =

φb|aψa|b

R √fg.

h = h/

h is a GMM since hab =√fagb/

R √fagb is a gaussian and

πabhab, where πab =q

φb|aψa|b

R √fagb

V (f, g)

V (f, g) =

qφb|aψa|b

pfagb =

Here h =

φb|aψa|b

R √fg.

h = h/

h is a GMM since hab =√fagb/

R √fagb is a gaussian and

πabhab, where πab =q

φb|aψa|b

R √fagb

V (f, g)

Thus, drawing samples {xi}ni=1 from h the estimate

pf(xi)g(xi)

is unbiased and in experiments is seen to be far superior to sampling from (f + g)/2.

Bhattacharyya Distance: Monte Carlo estimation

−5 −4 −3 −2 −1 0 1 2 3 4 50

3.5Bhattacharyya: Importance sampling from f (x)

bability

Deviation from Bhattacharyya estimate with 1M samples

• Importance sampling from f(x): Slow convergence.

−5 −4 −3 −2 −1 0 1 2 3 4 50

bability

−5 −4 −3 −2 −1 0 1 2 3 4 50

bability

101001000

−5 −4 −3 −2 −1 0 1 2 3 4 50

bability

10100100010K

−5 −4 −3 −2 −1 0 1 2 3 4 50

bability

10100100010K100K

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

Importance sampling from f(x)+g(x)2

bability

f 100K10

• Importance sampling from f(x)+g(x)2 : Better convergence.

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

bability

f 100K10100

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

bability

f 100K101001000

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

bability

f 100K10100100010K

−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 10

bability

f 100K10100100010K100K

−0.2 −0.15 −0.1 −0.05 0 0.05 0.1 0.15 0.20

120Variational importance sampling

bability

(f+g)/2 100K10

• Variational importance sampling: Fast convergence.Information Theory and Applications, 2/1-2007 – p.28/31

−0.2 −0.15 −0.1 −0.05 0 0.05 0.1 0.15 0.20

bability

(f+g)/2 100K10100

−0.2 −0.15 −0.1 −0.05 0 0.05 0.1 0.15 0.20

bability

(f+g)/2 100K101001000

−0.2 −0.15 −0.1 −0.05 0 0.05 0.1 0.15 0.20

bability

(f+g)/2 100K10100100010K

−0.2 −0.15 −0.1 −0.05 0 0.05 0.1 0.15 0.20

bability

(f+g)/2 100K10100100010K100K

−0.05 −0.04 −0.03 −0.02 −0.01 0 0.01 0.02 0.03 0.04 0.050

bability

f 100K(f+g)/2 100Kvariational 100K

Variational Monte Carlo Sampling: KL-Divergence

Future Directions

• HMM variational KL divergence

Future Directions

• HMM variational KL divergence• HMM variational Bhattacharyya

Future Directions

• HMM variational KL divergence• HMM variational Bhattacharyya• Variational Chernoff distance

Future Directions

• HMM variational KL divergence• HMM variational Bhattacharyya• Variational Chernoff distance• Variational sampling of Bayes error using Chernoff

approximation

Future Directions

approximation• Discriminative training, using Bhattacharyya divergence.

Future Directions

approximation• Discriminative training, using Bhattacharyya divergence.• Acoustic confusability using Bhattacharyya divergence

Future Directions

approximation• Discriminative training, using Bhattacharyya divergence.• Acoustic confusability using Bhattacharyya divergence• Clustering of HMMs

Future Directions

approximation• Discriminative training, using Bhattacharyya divergence.• Acoustic confusability using Bhattacharyya divergence• Clustering of HMMs

variational sampling approaches to word confusability

Documents

variational autoencoder

variational calculus

information kullback-liebler divergence variational...

sampling methods on manifolds and their view from...

approximate inference with graph neural networks...hence, in...

variational networks: connecting variational methods … ·...

unicode – the world standard for text and emoji · page 3...

variational principles

variational bayesian inference - kay brodersen ·...

variational autoencoders

methods gibbs sampling and variational · gibbs sampling...

variational inference & variational...

variational energy

a choice of variational stacks: exploring variational data...

understanding and accelerating particle-based variational...

beyond-the-standard-model contributions to rare b … · b...

the variational principle -...

ladder variational...

explaining variational approximationsexplaining variational...

optimum and efﬁcient sampling for variational quantum...