41. automatic language recognition via spectral automatic laccc.inaoep.mx/~villasen/bib/automatic...

811

Automatic La41. Automatic Language Recognition Via Spectraland Token Based ApproachesD. A. Reynolds, W. M. Campbell, W. Shen, E. Singer

Automatic language recognition from speechconsists of algorithms and techniques thatmodel and classify the language being spoken.Current state-of-the-art language recogni-tion systems fall into two broad categories:spectral- and token-sequence-based approaches.In this chapter, we describe algorithms forextracting features and models representingthese types of language cues and systemsfor making recognition decisions using oneor more of these language cues. A perfor-mance assessment of these systems is alsoprovided, in terms of both accuracy andcomputation considerations, using the Na-tional Institute of Science and Technology(NIST) language recognition evaluation bench-marks.

41.1 Automatic Language Recognition ........... 811

41.2 Spectral Based Methods ........................ 81241.2.1 Shifted Delta Cepstral Features ...... 81241.2.2 Classifiers ................................... 813

41.3 Token-Based Methods .......................... 81541.3.1 Tokens ....................................... 81541.3.2 Classifiers ................................... 816

41.4 System Fusion ...................................... 81841.4.1 Methods .................................... 81941.4.2 Output Scores ............................. 820

41.5 Performance Assessment ....................... 82041.5.1 Task and Corpus .......................... 82041.5.2 Systems ..................................... 82141.5.3 Results ....................................... 82241.5.4 Computational Considerations ...... 822

41.6 Summary ............................................. 823

References .................................................. 823

41.1 Automatic Language Recognition

Automatic language recognition from speech consists ofalgorithms and techniques that model and classify thelanguage being spoken. The term recognition can referto either identification of the spoken language from a setof languages known to the system, or detection of thepresence of a particular language out of a larger set oflanguages unknown to the system. As with other infor-mation conveyed by the speech signal, cues about thelanguage being spoken occur at several levels. At thetop level the words being spoken to communicate a co-herent message are the basis of what defines a language.However, relying on automatic extraction of the wordspresupposes the availability of a reliable speech recog-nizer in the languages of interest, which is often not thecase in many practical applications requiring languagerecognition. Further, language recognition is often usedas a low computation pre-processor to determine whichhigher computation speech recognizer should be run, if

any. Fortunately, cues about the language are also trans-mitted at lower levels in the speech signal, such as inthe phoneme inventory and co-occurrences (phonotac-tics), the rhythm and timing (prosodics) and the acousticsounds (spectral characteristics), that are more amenableto automatic extraction and modeling.

Current state-of-the-art language recognition sys-tems are based on processing these low-level cues andfall into two broad categories: spectral based and tokensequence based. This chapter describes algorithms forextracting features and models representing these typesof language cues and systems for making recognitiondecisions using one or more of these language cues. Ad-ditionally, a performance assessment of these systemsis provided, in terms of both accuracy and computationconsiderations, using the National Institute of Scienceand Technology (NIST) language recognition evaluationbenchmarks.

PartG

41

812 Part G Language Recognition

41.2 Spectral Based Methods

Spectral-based methods for language recognition op-erate by extracting measurements of the short-termspeech spectrum over fixed analysis frames and thenmodeling characteristics of these measurements, or fea-tures, for each language to be recognized. Classificationtechniques that have proved successful for languagerecognition are generative, via Gaussian mixture models(GMMs), and discriminative, via support vector ma-chines (SVMs). This section first describes the spectralfeature extraction process and then the GMM and SVMclassifiers used for recognition.

41.2.1 Shifted Delta Cepstral Features

The features used for language recognition systems arebased on the cepstral coefficients derived from a mel-scale filterbank analysis typically used in other speechprocessing tasks such as speech and speaker recognition.A block diagram of the filterbank feature extraction sys-tem is shown in Fig. 41.1. The feature extraction consistsof the following steps. Every 10 ms the speech signalis multiplied by a Hamming window with a durationof 20 ms to produce a short-time speech segment foranalysis. The discrete Fourier spectrum is obtained viaa fast Fourier transform (FFT) from which the mag-nitude squared spectrum is computed. The magnitudespectrum is multiplied by a pre-emphasis filter to em-phasize the high-frequency portion of the spectrum andthe result is put through a bank of triangular filters. Thefilterbank used is similar to that in [41.1] and simulatescritical band filtering with a set of triangular bandpassfilters that operate directly on the magnitude spectrum.The critical band warping is done by approximating the

��

��

�

��

� ��

�

��

�

��

��

��

��

��

��

��

��!"��# ��$%&� ��

�� '��

(�� )��

��

��

Fig. 41.1 Mel-scale filterbank cepstral feature extraction

mel-frequency scale, which is linear up to 1000 Hz andlogarithmic above 1000 Hz. The center frequencies ofthe triangular filters follow a uniform 100 Hz mel-scalespacing, and the bandwidths are set so the lower andupper passband frequencies of a filter lie on the centerfrequencies of the adjacent filters, giving equal band-widths on the mel-scale but increasing bandwidths on thelinear frequency scale. The number of filters is selectedto cover the signal bandwidth [0, fs/2] Hz, where fs isthe sampling frequency. For 8 kHz sampled telephonespeech, there are 24 filters.

The log energy output of each filter is then used asan element of a filterbank feature vector and the short-term mel-scale cepstral feature vector is obtained fromthe inverse cosine transform of the filterbank vector. Tomodel short-term speech dynamics, the static cepstralvector is augmented by difference (delta) and accel-eration (delta–delta) cepstra computed across severalframes.

Since the speech used for training and testing lan-guage recognition systems can come from a variety ofsources (involving different microphones and channels),it is important to apply some form of channel com-pensation to the features. Typical channel compensationtechniques include blind deconvolution via RASTA (rel-ative spectral) filtering [41.2] and per-utterance featurenormalization to zero mean and unit variance. More-sophisticated compensation techniques, such as featuremapping [41.3], that explicitly model channel effects arealso successfully used.

One of the significant advances in performing lan-guage recognition using GMMs was the discovery ofa better feature set for language recognition [41.4]. Theimproved feature set, the shifted delta cepstral (SDC) co-efficients, are an extension of delta-cepstral coefficients.

SDC coefficients are calculated as shownin Fig. 41.2. SDC coefficients are specified by fourparameters, conventionally written as N-d-P-k. Fora frame of data at time t, a set of MFCCs are calculated,i. e.,

c0(t), c1(t), . . . , cN−1(t) . (41.1)

Note that the coefficient c0(t) is used. The parameterd determines the spread across which deltas are calcu-lated, and the parameter P determines the gaps betweensuccessive delta computations. For a given time t,

Δc(t, i) = c(t + iP +d)− c(t + iP −d) (41.2)

represents an intermediate calculation, where i =0, 1, . . . , k. The SDC coefficients are then k stacked

PartG

41.2

Automatic Language Recognition Via Spectral and Token Based Approaches 41.2 Spectral Based Methods 813

�� $�*�%

��

+

�+�

�,�

�� $�*�%

�+�� +�

+

�+�+�

�,�

Fig. 41.2 Shifted delta cepstral coefficients

versions of (41.2),

SDC(t) =

⎛⎜⎜⎜⎜⎝

Δc(t, 0)

Δc(t, 1)...

Δc(t, k −1)

⎞⎟⎟⎟⎟⎠

. (41.3)

Prior to the use of SDC coefficients, GMM-basedlanguage recognition was less accurate than alternateapproaches [41.5]. SDC coefficients capture variationover many frames of data; e.g., the systems describedin this chapter use 21 consecutive frames of cepstralcoefficients. This long-term analysis might explain theeffectiveness of the SDC features in capturing language-specific information.

41.2.2 Classifiers

Current state-of-the-art language recognition systemsuse either or both generative GMM-based classifiers anddiscriminative SVM-based classifiers.

Gaussian Mixture ModelsThe approach for a GMM classifier is to model the dis-tribution of cepstral features for each language to berecognized. A language-specific GMM is given by

p(x|λ) =M∑

i=1

πiNi (x) , (41.4)

where x is a D-dimensional feature vector, Ni (x) are thecomponent densities, and πi are the mixture weights.Each component density is a D-variate Gaussian func-tion of the form

Ni (x) = 1

(2π)D/2|Σ i |1/2

× exp

{−1

2(x−μi )

T Σ−1i (x−μi )

},

(41.5)

with mean vector μi and covariance matrix Σ i . Themixture weights satisfy the constraint

M∑i=1

πi = 1 , (41.6)

which ensures that the mixture is a true probability den-sity function. The complete language-specific Gaussianmixture density is parameterized by the mean vec-tors, covariance matrices, and mixture weights fromall component densities and is represented by the no-tation

λ = {πi , μi , Σ i} i = 1, . . . , M . (41.7)

For language recognition using SDC features, di-agonal covariances and a mixture order of M = 2048are used. It was empirically determined [41.4] thathigher-order GMMs provided substantial performanceimprovements when using SDC features.

The GMM parameters are estimated via theexpectation-maximization (EM) algorithm [41.6] or byBayesian adaptation from a universal background model(UBM) [41.7], as is done in speaker recognition (seePart F of this Handbook). Adapting from a backgroundmodel has computational advantages in that it speedsup both model training and likelihood computations.For language recognition, the UBM is typically con-structed by training a GMM using an aggregation ofthe training data from all available languages. Recently,discriminative training of GMM parameters based onmaximum mutual information optimization has yieldedvery promising results [41.8].

For a general language recognition task, a GMM,λl , is trained for each of the desired languages,l = 1, . . . , L , and the likelihood of a sequence of fea-ture vectors, X = (x1, . . . , xT ), extracted from a testutterance is computed as

p(X|λl) =T∏

t=1

p(xt |λl) . (41.8)

The model likelihood is computed assuming indepen-dence between the feature vectors which means thetemporal ordering of the feature vectors is unimportant.However, the SDC feature vectors xt were themselvesextracted over a multiframe time span and thereby en-code local temporal sequence information.

The likelihood scores from the set of language mod-els are then used by the back-end fusion/decision systemto compute likelihood ratios or to fuse with other systemscores (Sect. 41.4).

PartG

41.2


Support Vector MachinesSupport vector machines (SVMs) are flexible classifiersthat have recently shown promise in language recogni-tion. SVMs rely on separating data in high-dimensionalspaces using the maximum margin concept [41.9]. AnSVM, f (Z), is constructed from sums of a kernel func-tion K (·, ·),

f (Z) =∑

i

αi K (Z, Zi )+d , (41.9)

where Z is the input feature vector,∑

i αi = 0, andαi �= 0. The Zi are support vectors and are obtainedfrom the training set by an optimization process [41.10].The ideal classifier values are either 1 or −1, dependingon whether the corresponding support vector is in class 0or class 1. For classification, a class decision is based onwhether the value f (Z) is above or below a threshold.

SVM kernels provide a method for compar-ing sequences of feature vectors (e.g., shifted deltacepstral coefficients). Given sequences of featurevectors from two utterances, X = (x1, · · · , xTx ) andY = (y1, · · · , yTy ), the kernel produces a comparisonthat provides discrimination between the languages ofthe utterances. If the utterances are from the same lan-guage, a large positive value is desired; otherwise, thekernel should produce a large negative value.

The general setup for training a language recognitionsystem using support vector machines is a one-versus-all strategy as shown in Fig. 41.3. For the example ofEnglish in the figure, class 1 contains only target (En-glish) language data and class 0 contains all of thenontarget (non-English) language data pooled together.Training proceeds using a standard SVM training toolwith a sequence kernel module (e.g., [41.10]). The re-sulting process produces a model for recognizing thetarget language.

-��

(�� !�

�./!��

01&�!#��

2��"��!��!��

2��"��!��!��

.�� !��!��

.�� !��!�

-�� !��!�-�� !��!�

��-�� !��!�

(�� !�

Fig. 41.3 Training a support vector machine for languagerecognition

A useful kernel for language recognition is thegeneralized linear discriminant sequence (GLDS) ker-nel [41.11]. The GLDS kernel is simply an inner productbetween average high-dimension expansions of a set ofbasis functions b j (·),

b(x) = (b1(x) · · · bK (x))T , (41.10)

where K is the number of basis functions. Typically,monomials up to a given degree are used as basisfunctions. An example basis is

b(x) =

⎛⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎝

1

x1

x2...

xn

x21...

xi11 xi2

2 . . . xinn

...

⎞⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎠

. (41.11)

The formula for the kernel is

KGLDS(X, Y) = bTx R−1by . (41.12)

The mapping X → bx is defined as

X → 1

Tx

Tx∑t=1

b(xt) . (41.13)

The vector by is defined in an analogous manner to(41.13). R is a correlation matrix derived from a largeset of data from multiple languages and many speakersand is usually diagonal.

In a general language recognition task, a set of SVMlanguage models { fl} are trained to represent the targetlanguages l = 1, · · · , L . The sequence of vectors X isextracted from a test utterance, mapped to an averageexpansion using (41.13), and converted to a set of scoressl = fl(bx). The SVM scores sl are usually transformedto s′

l using a likelihood-ratio-type calculation

s′l = sl − log

⎛⎝ 1

M −1

∑j �=l

e−s j

⎞⎠ . (41.14)

The transformation (41.14) assumes that SVM scoresbehave like log-likelihood ratios (see [41.12, 13] forrelated discussions).

PartG

41.2

Automatic Language Recognition Via Spectral and Token Based Approaches 41.3 Token-Based Methods 815

Several computational simplifications are availablefor implementing an SVM system for language recogni-tion. For instance, language recognition scoring can be

reduced to a single inner product. Also, language modeltraining can be simplified to a linear kernel in Fig. 41.3.The reader is referred to [41.13] for additional details.

41.3 Token-Based Methods

Unlike acoustic approaches in which fixed-duration win-dows of the speech signal are used to extract features forclassification, token-based approaches segment the in-put speech signal into logical units or tokens. Theseunits could be based on a phonetic segmentation (e.g.,phones), or a data-driven segmentation (e.g., GMMmixture component tokens [41.14]). Features are thenextracted from the token stream, and classification is per-formed using a language model. Figure 41.4 illustratesthe process.

This basic architecture allows for a variety of token-based approaches, but the phoneme recognition followedby language modeling (PRLM) approach originally de-scribed in [41.15, 16] has been shown to work well forlanguage recognition. The remainder of this section willfocus on PRLM and related techniques that model thephonotactic properties (phone stream dependencies) oflanguages. Other token-based techniques that should benoted include the work of [41.14] in which abstract to-kens were used in place of phonetic units to achieve nearstate-of-the-art language recognition performance.

41.3.1 Tokens

In the PRLM framework, the tokenizer is a phone rec-ognizer (PR) that converts an input speech signal intoa sequence of phonetic units (phones). Each phone and

�# �

��

��

Fig. 41.6 HMM:typical HMMtopology forphonetic units

its surrounding contexts are then combined to formphone n-grams, the features used for scoring. These fea-tures are scored against n-gram language models thatcorrespond to specific target languages to make languagedecisions. Intuitively, these features represent the phono-tactic dependencies (the rules governing the distributionof phonetic sounds within a language) between indi-vidual phonetic units. In an early work [41.17], Houseand Neuberg used transcribed data to demonstrate thefeasibility of exploiting phonotactics to perform auto-matic language identification. Figure 41.5 illustrates theprocess.

Phone RecognitionPhone recognition is typically accomplished using hid-den Markov models (HMMs) to represent phone units.Figure 41.6 shows a standard topology for differentphonetic units. During recognition, these models arecombined to allow any phone to transition to any otherphone with equal probability. This configuration is of-ten referred to as an open phone loop or null grammar,as shown in Fig. 41.7.

The phone decoding process is based on a sequenceof observations X = (x1, · · · , xT ), where T is the num-ber of speech frames. The observation xt is a vectorof acoustic features, usually mel-frequency cepstrumcoefficients (MFCC) [41.1] or perceptual linear predic-

3�#��

3�#�� 45 6! �!�7��!� ��

��8�� (�� '��

��

�� 45 69 �! �9�7�79��!��9��9�!�9�!�9� ��

Fig. 41.4 Token-based language recognition

:��

3�#�� 45 6! �!�7��!� ��

� ��'��

-�� !1/ ��

�� 45 69 � �9�7�79��

(�� !1/

;��!1/

�� !1/

Fig. 41.5 PRLM: phone recognition followed by language modeling

PartG

41.3


�# �

��

��

Fig. 41.7 Open phone loop (null grammar) with phoneHMMs

tion (PLP) coefficients [41.18]). These vectors may alsoinclude dynamic features (i. e., first and second differ-ences). Using standard assumptions (see Part E of thisHandbook), the probability of this sequence with respectto the language model λ is

p(X|λ) =∑∀s

T∏t=1

p(xt |st, λ) · P(st |st−1, λ) , (41.15)

where s = s1, · · · , st, · · · , sT is a hypothesized se-quence of states at times t = 1, · · · , T . Each state st ofthe model has an observation probability p(xt |st, λ) anda series of transition probabilities between all states andst . For simplicity, a bigram model P(st |st−1, λ) has beenselected for the state transition model. With a null gram-mar, the probabilities P(st |st−1, λ) are uniform for fixedst−1. Typically, the observation probability p(xt |st, λ)is modeled as a mixture of Gaussians resulting ina continuous-density HMM.

Given a sequence of observation vectors X, thephone recognizer attempts to decode the maximumlikelihood sequence of HMM states corresponding to X:

argmaxs

P(s|X) . (41.16)

Applying Bayes’ rule and recognizing that the maxi-mization is independent of p(X):

argmaxs

p(X|s)p(s)

p(X)= argmax

sp(X|s)P(s) . (41.17)

Equation (41.17) is the Viterbi approximation of (41.15);the goal is to maximize the following probability

p(X|s, λ)P(s|λ) =T∏

t=1

p(xt |st, λ) · P(st |st−1, λ)(41.18)

by selection of the state sequence s.The Viterbi algorithm is generally used to search

the space of possible state sequences and a variety ofpruning strategies can be applied to keep the searchproblem tractable. For language recognition, context in-dependent phonetic units are typically used for decoding(see [41.19] as an exception). For a more-detailed de-scription of HMM models of speech refer to Part E ofthis Handbook.

Phone Recognizer TrainingA vital consideration in the training of phone recognizersis the availability of phonetically or orthographi-cally labeled speech for multiple languages. Thesephone transcripts can be obtained from manual la-beling by a trained listener or automatic labelingfrom a word transcript and pronunciation dictio-nary. As discussed later, the languages of thephone recognizers need not be those of the tar-get languages. Phonetically transcribed corpora arerelatively uncommon, and for a number of yearswere available only as part of the Oregon Gradu-ate Institute multilanguage telephone speech (OGI-TS)corpus [41.20], which contained phonetically hand-labeled speech for six languages (English, German,Hindi, Japanese, Mandarin, and Spanish) collectedover telephone channels. More recently, investiga-tors have employed other corpora for training phonerecognizers, including Switchboard (English) [41.21](e.g., [41.22, 23]), CallHome (Arabic, Japanese, Man-darin, and Spanish) [41.21] (e.g., [41.22, 23]), andSpeechDat-East (Czech, Hungarian, Polish, Russian,and Slovak)d [41.24] (e.g., [41.8]).

41.3.2 Classifiers

Using the sequence of phone tokens, a number of dif-ferent classification techniques can be applied to makea language recognition decision. Two popular tech-niques are described here in detail: PRLM, a generativemodel of phonetic n-gram sequences, and PR-SVM-LATT (phone recognition followed by lattice-basedsupport vector machines), a discriminative approach tolanguage recognition. Both approaches assume that lan-guages differ in their phonotactic characteristics andthat these differences will cause a phonetic recogni-tion system to produce different distributions of tokensequences.

PartG

41.3

Automatic Language Recognition Via Spectral and Token Based Approaches 41.3 Token-Based Methods 817

PRLMThe basic PRLM decision rule is described by

L∗ = argmaxL

PL (W |X) , (41.19)

where X are the acoustic observations, W is the hypoth-esized token sequence W = w1, . . . , wN , and L∗ is theoptimal language decision. The term PL (W |X) can bedecomposed using Bayes’ rule:

PL (W |X) =same ∀L︷︸︸︷p(X|W) ·PL (W)

p(X)︸︷︷︸same ∀L

. (41.20)

For a given tokenizer, the terms p(X|W) and p(X)are the same across language models, since X is knownand W is determined from the phone recognition de-coding process described above. As such, the followingsimplified PRLM decision rule can be applied:

L∗ = argmaxL

PL (W) . (41.21)

As (41.21) suggests, only language model scores areused to make a language recognition decision. Phonerecognition acts solely to discretize the input speechsignal.

In real implementations PL (W) is approximated byan n-gram language model of fixed order,

PL (W) ≈N∏

i=1

language model︷︸︸︷PL (wi |wi−1 . . . wi−(n−1)) . (41.22)

Here, the probability PL (wi |wi−1 . . . wi−(n−1)) =PL (wi ), is a look up of the frequency of occurrenceof n-gram wi = wi−n+1wi−n+2 · · ·wi in language L’straining data. n-grams are a simple yet effective wayof capturing the context preceding a token. For the lan-guage recognition problem, n-grams of order two andthree (i. e., bigrams and trigrams) are commonly em-ployed. Increasing the n-gram order should theoreticallyincrease the ability of language recognition modelingtechniques to incorporate longer term dependencies, butin practice these models are difficult to estimate giventhe paucity of training data. As the order n increases, thenumber of possible unique n-grams increases exponen-tially as |W |n , where |W | is the size of the tokenizer’sphone inventory. Standard smoothing techniques havebeen shown to help mitigate the n-gram estimationproblem for language recognition tasks [41.25]. Otherlanguage modeling techniques, such as binary decisiontrees, have also been used successfully to model n-gramsequences with n > 3 [41.26].

Lattice-Based PRLM. As stated above, the standard 1-best PRLM model relies solely on the probability ofthe phone token sequence in its decision rule. This actsas an approximation of the acoustic hypothesis spacegenerated by the underlying HMM phonetic models fora given input.

In [41.22], a better approximation using an exten-sion of the 1-best PRLM model is proposed in whichposterior probabilities of phone tokens are incorporatedinto the estimation of the language models and the obser-vation likelihoods. In this model, estimates of expectedcounts derived from phone lattices are used in place of1-best counts during LM training and scoring.

To derive this extension, the standard 1-best lan-guage modeling equation can be reformulated in logform as

log PL (W)

=∑∀w

C(w) log PL (wi |wi−1, . . . , wi−n+1) , (41.23)

where

w = wiwi−1 . . . wi−n+1 (41.24)

and C(w) is the count of the n-gram w in the sequence W .In (41.23), the sum is performed over the unique n-gramsw in W .

In the lattice formulation, n-gram counts from the1-best hypothesis are replaced with expected countsfrom a phone lattice generated by the decoding of aninput message:

EL[ log PL (W)] =∑∀w

EL[C(w)

]log PL (wi |wi−1 . . . , wi−n+1) .

Like the 1-best hypothesis, the phone lattice is onlyan approximation of the acoustic hypothesis spacefor a given speech utterance. There is evidence thatboth the quality of the phonetic models and thephone lattices used for language recognition are im-portant [41.19,27]. Many methods of generating latticeshave been proposed in the automatic speech recognition(ASR) literature [41.28–30]. The Viterbi n-best tech-nique with a fixed number of trace-backs per state isoften used [41.31].

Parallel PRLM (PPRLM)An important aspect of the PRLM method, impliedby (41.21), is that the operations of tokenization (by thephone recognizer) and language modeling are decou-pled. As a consequence, it is not necessary for the phone

PartG

41.3


��

&��

<��

��

�� /��

��

��

0��!��! �� !�� '��!<��

��!��

0��!��!�� !�� '��!/��

��!��

� �

1�#��

��

1�#��

Fig. 41.8 PPRLM: parallel phone recognition followed by languagemodeling

recognizer to be trained in any of the target languages aslong as the tokenization of the target languages is suffi-ciently distinct. In a natural extension of this observation,Zissman [41.15] proposed running phone recognizersfor multiple languages in parallel in order to providemore-diversified coverage of the phones encounteredin target languages. The resulting system configuration,commonly known as parallel PRLM (PPRLM), employsa bank of phone recognizers, each of which is followedby a set of language models trained from the correspond-ing token sequences that it generates. As with PRLM,the multiple phone recognizers are trained using phonet-ically or orthographically labeled speech, but need notbe of any of the target languages. An example is shownin Fig. 41.8 where parallel Japanese and Mandarin phonerecognizers are used to tokenize speech in a German ver-sus Spanish language recognition task. The next sectionwill describe methods by which the individual outputsare fused to produce a final score.

Language recognition systems based on PPRLMwere found to outperform their competitors [41.5] anddominated the field of language recognition for a numberof years. More recently, employing banks of lattice-based phone recognizers has led to further languagerecognition performance improvement [41.22]. Efforts

have also been made to reduce the computation load ofa PPRLM system by employing a single multilingualphone recognizer but to date these approaches have notproved to be more effective than PPRLM [41.32].

PPR-SVM-LATTThe PPR-SVM-LATT approach is an application ofSVM classification for token streams [41.33]. For thePPR-SVM-LATT classifier, the phone token sequencefor each utterance X in a language corpus is used tocompute a vector of n-gram probabilities,

b =

⎛⎜⎜⎜⎜⎝

c1 p(w1|X)

c2 p(w2|X)...

cN p(wN |X)

⎞⎟⎟⎟⎟⎠

. (41.25)

The probability p(wi |X) represents the frequency of oc-currence of n-gram wi in the utterance X, and N = |W |is the number of possible n-grams. The n-gram probabil-ities can be computed from counts from the 1-best phonesequence or from expected counts from phone lattices.Experiments have determined that expected counts fromlattices produce better performance.

The values ci in (41.25) are weights that normal-ize the components of the vector to a common scale.Typically, a weighting such as the term-frequency loglikelihood ratio (TFLLR) [41.33] can be applied. ForTFLLR,

ci = 1√p(wi |{X j})

, (41.26)

the probability p(wi |{X j}) is calculated across all utter-ances of all languages in the training corpus.

After converting utterances to vectors, a simple lin-ear kernel, K (b1, b2) = bT

1 b2, is used, where the bi arethe expansion of two utterances as in (41.25). Train-ing and scoring are done in a manner analogous to thecepstral SVM.

41.4 System Fusion

The previous sections of this chapter described a vari-ety of methods available for implementing a languagerecognition system. In practice, additional performanceimprovements can be obtained by running multiple sys-tems in parallel and fusing the individual scores usinga back-end. If the recognizers do not behave uniformly

(i. e., their errors are not completely correlated) thencombining the scores may make it possible to exploitcomplementary information that exists in the scores.Score fusion was first adopted as a means of combiningthe scores produced by a PPRLM language recognizerand has since been applied to fusing scores of differ-

PartG

41.4

Automatic Language Recognition Via Spectral and Token Based Approaches 41.4 System Fusion 819

ent types of recognizers, such as those described inSects. 41.2 and 41.3.

41.4.1 Methods

For purpose of this discussion it is convenient to divideapproaches to fusion into two types, rule based and clas-sifier based. Rule-based fusion applies a fixed formulato the scores, such as a weighted-sum rule or productrule, where weights are either uniform or are chosenempirically using a set of development data. One ex-ample of a rule-based method for combining the scoresin a PPRLM system will be discussed below. The sec-ond approach to fusion utilizes a development data set totrain a classifier from a feature vector constructed fromthe individual system scores. In theory, the joint statisticsamong the scores (elements of the feature vector) canbe exploited by the classifier to produce overall perfor-mance that is superior to that achieved with a rule-basedback.end.

Product-Rule FusionA method for fusing the scores of a PPRLM languagerecognition system using the product rule was pro-posed by Zissman [41.5]. Assume a PPRLM languagerecognizer that uses K phone recognizers to detect L lan-guages. For each test input, a set of KL (linear) scoresis generated, with each phone recognizer producing Lscores. For an input utterance X, single per-languagescores y(X|l) are computed by first normalizing the like-lihoods sk,l produced by phone recognizer k and thenmultiplying the normalized per-language values acrossall phone recognizers:

y(X|l) =K∏k

sk,l∑Ll sk,l

. (41.27)

The normalization step puts scores across phone rec-ognizers into a common range, and the multiplicationstep creates a final score. Probabilistically, the first stepturns the raw phone recognizer scores into posteriors(with an assumption of equal priors) and the secondstep computes a joint probability of the observed dataunder the assumption that the information sources (in-dividual PRLM systems) are independent. A potentialdrawback of product-rule fusion is that a single scoreclose to zero may have an undesirably large impacton the final result. Despite the apparent inaccuracy ofthe independence assumption, the product-rule methodproved to be an effective way of combining PPRLMscores [41.5].

Classifier FusionClassifier-based fusion for PPRLM language recog-nition systems was originally proposed by Yan andBernard [41.34] using neural networks and by Ziss-man [41.35] using Gaussian classifiers, and these authorsreported that performance was superior to that obtainedwith the product-rule approach. A block diagram ofa Gaussian-based fusion approach for language recog-nition is shown in Fig. 41.9. The method consists oflinear discriminant analysis (LDA) normalization fol-lowed by Gaussian classification. Although there can beno guarantee that this particular classifier-based fusionarchitecture produces optimum results in all languagerecognition applications, extensive development testinghas shown that the technique produces reliably superiorperformance. Other investigators have reported resultsusing neural network-based back-end fusion [41.22].

The generic block diagram in Fig. 41.9 shows a setof core language recognizers (e.g., GMM, SVM, PRLM,PPRLM), each of which produces a score or score vectorfor each language hypothesis given an input utterance.These scores are used to form the elements of a fea-ture vector that is transformed via linear discriminantanalysis, a method by which the feature vector is pro-jected into a reduced dimensional space in such a wayas to maximize interclass separability and minimize in-traclass variability. In the figure, the transformed spaceis of dimension L −1, where L is the number of classes(languages) represented in the back-end training data.The transformed vector also has the desirable prop-erty that its features are decorrelated. It is importantto recognize that the individual recognizers may pro-duce different sized score vectors and that the scoresmay represent likelihoods of any languages, not just thetarget languages.

��

=��!��*�

��*�

��

0�� !�

=��!�

=��!�

��

0�� !� ��

0�� !� ��

Fig. 41.9 Gaussian-based fusion for language recognition

PartG

41.4


The transformed feature vectors are then processedby a parallel bank of Gaussian classifiers. Each Gaussianhas input dimension L −1, a diagonal grand covariance(i. e., pooled across all the back-end training data), andproduces an output y(X|l).

41.4.2 Output Scores

The form of the final system scores depends on thenature of the language recognition application. Forclosed-set identification, the index of the maximumscore, maxl y(X|l), produced using either the rule- orclassifier-based method is sufficient to identify the win-ning language hypothesis. For a detection task, wheredecisions are made using a fixed threshold, the outputscore must be formed in such a way as to be consis-tently meaningful across utterances. Typically, this isaccomplished by interpreting the (linear) scores y(X|l)as likelihoods and forming an estimated likelihood ratior(X|l) between the target language score and the sum(or average) of the others:

r(X|l) = y(X|l)1

L−1

∑Lj �=l y(X| j)

. (41.28)

The estimated likelihood ratio can then be used to seta threshold to achieve a specific application dependentoperating point.

Language recognition applications often require thatscores be reported as posterior probabilities rather thanas likelihoods or likelihood ratios. This may be thecase when the downstream consumer (human or ma-chine) requires confidence scores or scores that havebeen mapped from unconstrained likelihoods or likeli-hood ratios to a normalized range (0–1 or 0–100). Inprinciple, the scores y(X|l) produced by the back-end

1!= �/1:! �

��

��

/1:!�

/1:!��>!$�?�%

�>!$� ?�%

�>!$�?�%

��$ %

��$ %

��$ %

��

��

��

Fig. 41.10 Posterior probability estimation

fusion for input utterance X can be used directly to gen-erate posterior probabilities P(l) for each target languagel via Bayes’ rule:

P(l|X) = y(X|l)Pl∑Ll y(X|l)Pl

, (41.29)

where Pl represents the prior probability of target lan-guage l. In practice, posteriors estimated this way maybe unreliable due to the inaccurate assumptions made inconstructing the overall system. Rather than treating thescores as true likelihoods, it is preferable to view themas raw scores and to estimate posteriors using anotherclassifier.

A block diagram of a method that has been founduseful for posterior probability estimation is shownin Fig. 41.10. The fusion back-end log-likelihood ratioestimates log r(X|l) are treated as raw inputs to a bankof language-dependent single-input logistic regressionclassifiers (implemented using LNKnet [41.36] as mul-tilayer perceptrons) that produce estimates P(l|X) ofthe posterior probabilities. These estimates can thenbe interpreted by humans as meaningful measures ofconfidence.

41.5 Performance Assessment

The language recognition approaches described in thischapter have been subject to extensive development, andthis section describes their evaluation under conditionsthat follow protocols established by the US NationalInstitute of Standards and Technology (NIST). Since1996, NIST has coordinated several evaluations withthe goal of assessing the existing state-of-the-art inlanguage recognition technology. The most recent lan-guage recognition evaluation (LRE 2005) protocol willform the basis of the performance results presented in

this section. A description of the 2005 NIST languagerecognition evaluation plan can be found in [41.37].

41.5.1 Task and Corpus

The task for LRE 2005 was the detection of the pres-ence of one of seven target languages (English, Hindi,Japanese, Korean, Mandarin, Spanish, Tamil) in tele-phone speech utterances with nominal durations of30, 10, and 3 seconds selected from unspecified lan-

PartG

41.5

Automatic Language Recognition Via Spectral and Token Based Approaches 41.5 Performance Assessment 821

guages and sources. The evaluation material contained3,662 segments for each nominal duration. AlthoughNIST evaluated and reported performance results un-der several conditions, its primary evaluation conditionconsisted of trials for which the segments had 30 seconddurations, were from one of the seven target languages,and were from material collected by the Oregon HealthSciences University (OHSU). Unless otherwise stated,the results presented here will be for the NIST pri-mary evaluation condition. For a complete descriptionof LRE 2005, see [41.38].

Training data for the systems described below wasdrawn from four telephone speech corpora: CallFriend,CallHome, Mixer, and Fisher, and cover 13 languages(for more information on these corpora, see [41.21]).CallFriend and CallHome contain conversations be-tween acquaintances, while Mixer and Fisher containconversations between strangers. For a detailed descrip-tion of the use of this material for training the recognizersand the fusion back-end, see [41.23].

41.5.2 Systems

The fully fused system contained five core languagerecognizer components. For each recognizer, nonspeechframes were removed from the train and test utterancesusing a GMM-based speech activity detector.

GMMThe Gaussian mixture model (GMM)-based languagerecognizer used 2048 mixture components, shifted deltacepstral (SDC) coefficients, feature normalization acrossutterances to zero mean and unit variance on a per-feature basis, and feature mapping [41.3]. The SDCparameters N-d-P-k were set to 7-1-3-7 (Sect. 41.2.1),resulting in a 49-dimension feature vector computedover a 21-frame (210 ms) time span. In addition, gender-dependent language models were trained for each ofthe 13 languages in the training material. A back-ground model was trained by pooling all the trainingmaterial, and gender-dependent target language modelswere trained by adapting from the background model.This method of training permits fast scoring duringrecognition and was proposed by [41.39] for languagerecognition. The GMM system produced a total of 26scores per test utterance.

SVMThe support vector machine (SVM)-based languagerecognizer used a generalized linear discriminant se-quence (GLDS) kernel, expansion into feature space

using monomials up to degree three, and SDC coeffi-cients derived as for the GMM recognizer. The featureswere normalized across utterances to zero mean and unitvariance on a per-feature basis. Thirteen language mod-els were constructed from the training material and 13scores were produced per test utterance.

PPRLMThe PPRLM-based language recognizer used six OGI-trained phone recognizers, a 39-dimensional MFCCfeature vector containing cepstra, delta, and doubledelta coefficients, and trigram language modeling. Eachphone recognizer’s token sequences were used to train13 language models, resulting in an output feature vectorof dimension 78. The phone recognizers and languagemodels were trained using Cambridge University en-gineering department’s hidden Markov model toolkit(HTK) [41.40].

PPRLM-LATTA PPRLM based language recognizer using lattices togenerate probabilistic counts of phone frequencies forbigram and trigram language modeling. Phone recogniz-ers were trained using material from CallHome (Arabic,Japanese, Mandarin) and Switchboard English (landlineand cellular). A total of seven phone recognizers weretrained, resulting in 91 (7 × 13) scores per input utter-ance. The tokenizers were selected based on results ofdevelopment testing conducted in conjunction with LRE2005 ( [41.41]).

PPR-SVM-LATTThe PPR-SVM-LATT-based language recognizer usedlattices to generate probabilistic counts of phone fre-quencies that are then classified with support vectormachines. Probability vectors were constructed from un-igrams and bigrams, and a linear kernel with TFLLRweighting was applied. Lattices of phones were pro-duced using HMM models trained on six OGI languages,as in PPRLM. A total of 78 (6×13) scores were producedper input utterance.

The scores of the core language recognizers werestacked into a 286-dimension feature vector and fedto the fusion back-end for dimension reduction (lin-ear discriminant analysis) and classification (sevenGaussians with pooled diagonal covariances). The out-puts of the back-end were treated as estimated targetlanguage likelihoods and log likelihood ratios wereformed by taking the log of the quantity in (41.28).These scores were treated as the final outputs of thesystem.

PartG

41.5


��

��!��

��

��

��

��

�

�

�

�

�

� �!"�#

Fig. 41.11 Performance (percentage EER) of spectral, to-ken, and fused language recognition systems on the 2005NIST LRE primary condition test

41.5.3 Results

NIST language recognition evaluations require partic-ipants to provide two outputs for each test utterance,a hard decision and an arbitrarily scaled likelihoodscore. The hard decisions are used by NIST to evalu-ate a cost function while the scores are used to sweepout a detection error trade-off (DET) curve [41.42],a variant of the familiar receiver operating character-istic (ROC) curve. To compare detection performanceacross competing systems, it is convenient to use a sum-mary statistic such as the equal error rate (EER), andthis statistic has been adopted for comparison of thelanguage recognition systems described in this sec-tion. Results for the NIST primary condition describedabove, given in percentage EER, are shown in Fig. 41.11.The recognizers have been categorized according towhether they are spectral (GMM or SVM) or token based(PPRLM, PPRLM-LATT, PPR-SVM-LATT). Individu-ally, the best performing recognizer is PPRLM-LATT,while the SVM system produces the most errors. Theremaining systems (GMM, PPRLM, and PPR-SVM-LATT) perform comparably. It is worth noting thatfusing the scores of the spectral systems (SVM andGMM) results in a large performance gain [41.23]. Fus-ing the scores of all five recognizers produces the bestoverall result.

Figure 41.12 shows the DET plots for five-recognizer-system fusion for the LRE 2005 OHSUtarget language test segments with durations 30, 10,and 3 seconds. Table Table 41.1 gives the correspondingEERs for these DET curves. It is apparent that lan-guage recognition performance becomes increasinglychallenging as the test segment duration decreases, al-

�$

�$

��!%�%&�&�'!!(!(��!��)!��

��!%�%&�&�'!!(!)&��!��

��$� �$* � * � ��

�$*

�$�

�

*

�

��

��

+��

��+�

Fig. 41.12 Performance of five-system fusion on theLRE 2005 OHSU target language test segments for du-rations 30 s (solid line), 10 s (dashed line), and 3 s (dottedline). Filled circles indicate equal error points

though recent work has demonstrated that performanceat shorter durations can be substantially improved [41.8].

41.5.4 Computational Considerations

Although the focus of the discussion of language recog-nition systems has been performance, any practicalimplementation of such a system needs to account forcomputational complexity. A comparison of process-

Table 41.1 Performance, measured as percentage equal er-ror rate (EER) of full-fusion system for 30 s, 10 s, and 3 stest utterances

Duration EER(%)

30 4.3

10 10.3

3 21.2

Table 41.2 Processing speed of selected language recog-nizers as measured with a 3 GHz Xeon processor with2 GB RAM running Linux. The recognition task involved12 target languages and five minute audio file. The PPRLMsystem contained six OGI-trained phone recognizers (nolattices). All factors refer to speeds faster than real time (RT)

Reognizer Speed

GMM 17 RT

SVM 52 RT

PPRLM 2 RT

PartG

41.5

Automatic Language Recognition Via Spectral and Token Based Approaches References 823

ing speed for three types of recognizers is shown inTable 41.2. Note that all factors indicate the degreeto which the algorithms are faster than real time. Allmeasurements were made using 12 language modelsand a single 5 minute audio file. The platform was

a 3 GHz Xeon processor with 2 GB RAM and LinuxOS. The GMM, SVM, and PPRLM systems were de-scribed in previous sections of this chapter. The PPRLMlanguage recognizer used six OGI tokenizers and nolattices.

41.6 Summary

This chapter has presented a variety of methods forimplementing automatic language recognition systems.Two fundamental approaches, based on either the spec-tral or phonotactic properties of speech signals, weresuccessfully exploited for the language recognition task.For the spectral systems, these included GMM and SVMmodeling using frame-based shifted delta cepstral coeffi-cients. For phonotactic systems, variations of the classicparallel phone recognition followed by language model-ing were described. Performance gains can be achievedby fusing the scores of multiple systems operating simul-

taneously, and a framework for implementing classifierbased fusion was described. Finally, performance re-sults were presented for the spectral, token, and fusedsystems using the standardized protocols developed byNIST. For the latest advances in the field of automaticlanguage recognition, the reader is referred to the mostrecent proceedings of the International Conference onAcoustics, Speech, and Signal Processing (ICASSP), theOdyssey Speaker and Language Recognition Workshop,and the International Conference on Spoken LanguageProcessing (Interspeech – ICSLP).

References

41.1 S. Davis, P. Mermelstein: Comparison of ParametricRepresentations for Monosyllabic Word Recogni-tion in Continuously Spoken Sentences, IEEE Trans.Acoust. Speech Signal Process. (1980) pp. 357–366

41.2 H. Hermansky, N. Morgan, A. Bayya, P. Kohn: Com-pensation for the Effect of the CommunicationChannel in Auditory-Like Analysis of Speech, Proc.Eurospeech (1991) pp. 1367–1371

41.3 D.A. Reynolds: Channel robust speaker verificationvia feature mapping, Proceedings of the, Interna-tional Conference on Acoustics Speech and SignalProcessing (2003) pp. II–53, –56

41.4 P.A. Torres-Carrasquillo, E. Singer, M.A. Kohler,R.J. Greene, D.A. Reynolds, J.R. Deller Jr.: Ap-proaches to language identification using Gaussianmixture models and shifted delta cepstral features,International Conference on Spoken Language Pro-cessing (2002) pp. 89–92

41.5 M. Zissman: Comparison of Four Approaches toAutomatic Language Identification of TelephoneSpeech, IEEE Trans. Speech Audio Process. 4(1), 31–44 (1996)

41.6 A. Dempster, N. Laird, D. Rubin: Maximum likeli-hood from incomplete data via the EM algorithm,J. R. Stat. Soc. 39, 1–38 (1977)

41.7 D.A. Reynolds, T.F. Quatieri, R. Dunn: Speaker Ver-ification Using Adapted Gaussian Mixture Models,Digital Signal Process. 10(1-3), 19–41 (2000)

41.8 P. Matejka, L. Burget, P. Schwarz, J. Cernocky:Brno University of Technology System for NIST 2005

Language Recognition Evaluation, Proc. IEEE 2006Odyssey: The Speaker and Language RecognitionWorkshop (2006)

41.9 V.N. Vapnik: Statistical Learning Theory (Wiley,New York 1998)

41.10 R. Collobert, S. Bengio: SVMTorch: Support VectorMachines for Large-Scale Regression Problems, J.Mach. Learn. Res. 1, 143–160 (2001)

41.11 W.M. Campbell: Generalized Linear DiscriminantSequence Kernels for Speaker Recognition, Pro-ceedings of the International Conference onAcoustics Speech and Signal Processing (2002)pp. 161–164

41.12 J.C. Platt: Probabilities for SV Machines. In:Advances in Large Margin Classifiers, ed. byA.J. Smola, P.L. Bartlett, B. Schölkopf, D. Schuur-mans (MIT, Cambridge 2000) pp. 61–74

41.13 W.M. Campbell, J.P. Campbell, D.A. Reynolds,E. Singer, P.A. Torres-Carasquillo: Support vectormachines for speaker and language recognition,Comput. Speech. Lang. 20(2-3), 210–229 (2006)

41.14 P.A. Torres-Carrasquillo, D.A. Reynolds, J.R. DellerJr.: Language Identification Using Gaussian MixtureModel Tokenization, Proceedings of the Interna-tional Conference on Acoustics Speech and SignalProcessing (2002) pp. 757–760

41.15 M. Zissman, E. Singer: Automatic Language Iden-tification of Telephone Speech Messages UsingPhoneme Recognition and n -Gram Modeling,Proceedings of the International Conference on

PartG

41


Acoustics Speech and Signal Processing (1994)pp. 305–308

41.16 Y. Yan, E. Barnard: An Approach to AutomaticLanguage Identification Based on Language-Dependent Phone Recognition, Proceedings of theInternational Conference on Acoustics Speech andSignal Processing (1995) pp. 3511–3514

41.17 A. House, E. Neuberg: Toward automatic identifica-tion of the language of an utterance. I, Preliminarymethodological considerations, J. Acoust. Soc. Am.62, 708–713 (1977)

41.18 H. Hermansky: Perceptual linear predictive (PLP)analysis for speech, J. Acoust Soc. Am. 87, 1738–1752(1990)

41.19 D. Zhu, M. Adda-Decker, F. Antoine: Differ-ent Size Multilingual Phone Inventories andContext-Dependent Acoustic Models for LanguageIdentification, Proc. Interspeech (2005) pp. 2833–2836

41.20 Y.K. Muthusamy, R.A. Cole, B.T. Oshika: TheOGI Multi-language Telephone Speech Corpus,International Conference on Spoken Language Pro-cessing (1992) pp. 895–898

41.21 Linguistic Data Consortium, http://www.ldc.upenn.edu/ (2007)

41.22 J.L. Gauvain, A. Messaoudi, H. Schwenk: LanguageRecognition Using Phoneme Lattices, InternationalConference on Spoken Language Processing (2004)pp. 2833–2836

41.23 W. Campbell, T. Gleason, J. Navritil, D. Reynolds,W. Shen, E. Singer, P. Torres-Carrasquillo: Ad-vanced Language Recognition using Cepstra andPhonotactics: MITLL System Performance on theNIST 2005 Language Recognition Evaluation, Proc.IEEE 2006 Odyssey: The Speaker and LanguageRecognition Workshop (2006)

41.24 Eastern European Speech Databases for Creation ofVoice Driven Teleservices, http://www.fee.vutbr.cz/SPEECHDAT-E/ (2007)

41.25 E. Singer, P.A. Torres-Carrasquillo, T.P. Gleason,W.M. Campbell, D.A. Reynolds: Acoustic, Phonetic,and Discriminative Approaches to Automatic Lan-guage Identification, Proceedings of Eurospeech(2003) pp. 1345–1348

41.26 J. Navratil: Recent advances in phonotactic lan-guage recognition using binary-decision trees,International Conference on Spoken Language Pro-cessing, Vol. 2 (2006)

41.27 P. Matejka, P. Schwarz, J. Cernocky, P. Chytil:Phonotactic Language Identification using HighQuality Phoneme Recognition, Proceedings of In-terspeech (2005) pp. 2833–2836

41.28 F. Weng, A. Stolcke, A. Sankar: New Developmentsin Lattice-based Search Strategies in SRI’s H4 sys-tem, Proceedings of DARPA Speech RecognitionWorkshop (1998) p. 100

41.29 H. Ney, X. Aubert: A word graph algorithm forlarge vocabulary, continuous speech recognition,International Conference on Spoken Language Pro-cessing (1994) pp. 1355–1358

41.30 F. Weng, A. Stolcke, A. Sankar: Efficient Lat-tice Representation and Generation, InternationalConference on Spoken Language Processing (1998)pp. 100–100

41.31 R. Schwartz, S. Austin: A Comparison of several ap-proximate algorithms for finding multiple (n-best)sentence hypotheses, Proceedings of the Interna-tional Conference on Acoustics Speech and SignalProcessing (1991) pp. 701–704

41.32 J. Navratil: Automatic Language Identification. In:Multilingual Speech Processing, ed. by T.S. Pitts-burgh, K. Kirchhoff (Academic, New York 2006)

41.33 W.M. Campbell, J.P. Campbell, D.A. Reynolds,D.A. Jones, T.R. Leek: High-Level Speaker Verifica-tion with Support Vector Machines, Proceedings ofthe International Conference on Acoustics Speechand Signal Processing (2004) pp. I–73–I–76

41.34 Y. Yan, E. Barnard: Experiments for an Approach toLanguage Identification with Conversational Tele-phone Speech, Proceedings of the InternationalConference on Acoustics Speech and Signal Pro-cessing (1996) pp. 789–792

41.35 M.A. Zissman: Predicting, Diagnosing and Im-proving Automatic Language Identification Perfor-mance, Proceedings of Eurospeech (1997) pp. 51–54

41.36 R. Lippman, L. Kukolich: LNKnet Pattern Recogni-tion Software Package, http://www.ll.mit.edu/IST/lnknet/ (2007)

41.37 A. Martin, G. Doddington: The 2005 NIST LanguageRecognition Evaluation Plan, http://www.nist.gov/speech/tests/lang/2005/LRE05EvalPlan-v5-2.pdf(2005)

41.38 A.F. Martin, A.N. Le: The Current State of Lan-guage Recognition: NIST 2005 Evaluation Results,Proc. IEEE 2006 Odyssey: The Speaker and LanguageRecognition Workshop (2006)

41.39 E. Wong, S. Sridharan: Methods to improve Gaus-sian mixture model based language identificationsystem, International Conference on Spoken Lan-guage Processing (2002) pp. 93–96

41.40 Hidden Markov Model Toolkit, http://htk.eng.cam.ac.uk/ (2007)

41.41 W. Shen, W. Campbell, T. Gleason, D. Reynolds,E. Singer: Experiments with Lattice-based PPRLMLanguage Identification, Proc. IEEE 2006 Odyssey:The Speaker and Language Recognition Workshop(2006)

41.42 A. Martin, G. Doddington, T. Kamm, M. Ordowski,M. Przybocki: The DET Curve in Assessment of Detec-tion Task Performance, Proceedings of Eurospeech(1997) pp. 1895–1898

PartG

41

41. automatic language recognition via spectral automatic laccc.inaoep.mx/~villasen/bib/automatic...

Documents