nineoneone: recognizing and classifying speech for handling minority language emergency calls
DESCRIPTION
NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls. Udhay Nallasamy, Alan W Black, Tanja Schultz, Robert Frederking, [Jerry Weltman, Julio Schrodel] May 2008. Outline. Overview System design ASR design MT design Current results ASR results - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/1.jpg)
NineOneOne:Recognizing and Classifying Speech
for Handling Minority Language Emergency Calls
Udhay Nallasamy, Alan W Black,
Tanja Schultz, Robert Frederking,
[Jerry Weltman, Julio Schrodel]
May 2008
![Page 2: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/2.jpg)
Outline
• Overview– System design– ASR design– MT design
• Current results– ASR results– Classification-for-MT results
• Future plans
![Page 3: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/3.jpg)
Project overview
• Problem: Spanish 9-1-1 calls handled in slow, unreliable fashion
• Tech base: SR/MT far from perfect, but usable in limited domains
• Science goal: Speech MT that really gets used– 9-1-1 as likeliest route:
• Naturally limited, important, civilian domain• Interested user partner who will really try it
– (vs. Diplomat experience…)
![Page 4: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/4.jpg)
Domain Challenges/Opportunities
• Challenges:– Real-time required– Random phones– Background noise– Stressed speech– Multiple dialects– Cascading errors
• Opportunities:– Speech data source– Strong task constraints– One-sided speech– Human-in-the-loop– Perfection not required
![Page 5: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/5.jpg)
System flow
Spanish caller
English Text
I need ambulance
Spanish speech
English Dispatcher
Nueve uno uno, ¿cuál es su emergencia?
Necessito una ambulancia
SpanishASR
DAClassifier
Spanish-To-English MT
DispatchBoard
SpanishTTS
![Page 6: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/6.jpg)
Overall system design
• Spanish to English: [no TTS!]– Spanish speech recognized– Spanish text classified (context-dependent?) into
DomainActs, arguments spotted and translated– Resulting text displayed to dispatcher
• English to Spanish: [no ASR!]– Dispatcher selects output from tree, typing/editing arg
• Very simple “Phraselator”-style MT
– System synthesizes Spanish output• Very simple limited-domain synthesizer
• HCI work: keeping human in the loop!– role-playing & shadow use
![Page 7: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/7.jpg)
“Ayudame” system mock-up
• We plan to interface with call-takers via web browser
• Initial planned user scenario follows
• The first version will certainly be wrong– One of the axioms of HCI
• But iterating through user tests is the best way to get to the right design
![Page 8: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/8.jpg)
![Page 9: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/9.jpg)
![Page 10: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/10.jpg)
Technical plans: ASR
• ASR challenges:– Disfluencies– Noisy emotional speech– Multiple dialects, some English
• Planned approaches:– Noisy-channel model [Honal et al,
Eurospeech03]– Articulatory features– Multilingual grammars, multilingual front end
![Page 11: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/11.jpg)
Technical plans: MT
• MT challenges:– Disfluencies in speech– ASR errors– Accuracy/transparency vs. Development costs
• Planned approaches: adapt and extend– Domain Act classification from Nespole!
• Shallow interlingua, speaker intent (not literal)• Report-fire, Request-ambulance, Don’t-know, …
– Transfer rule system from Avenue– (Both NSF-funded.)
![Page 12: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/12.jpg)
Nespole! Parsing and Analysis Approach
• Goal: A portable and robust analyzer for task-oriented human-to-human speech, parsing utterances into interlingua representations
• Our earlier systems used full semantic grammars to parse complete DAs– Useful for parsing spoken language in restricted domains
– Difficult to port to new domains
• Nespole! focus was on improving portability to new domains (and new languages)
• Approach: Continue to use semantic grammars to parse domain-independent phrase-level arguments and train classifiers to identify DAs
![Page 13: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/13.jpg)
Example Nespole! representation
• Hello. I would like to take a vacation in Val di Fiemme.
• c:greeting (greeting=hello)
c:give-information+disposition+trip
(disposition=(who=i, desire),
visit-spec=(identifiability=no, vacation),
location=(place-name=val_di_fiemme))
![Page 14: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/14.jpg)
MT differences from Nespole!
• Hypothesis: Simpler domain can allow simpler (less expensive) MT approach
• DA classification done without prior parsing– We may add argument-recognizers as
features, but still cheaper than parsing
• After DA classification, identify, parse, and translate simple arguments (addresses, phone numbers, etc.)
![Page 15: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/15.jpg)
Currently-funded NineOneOne work
• Full proposal was not funded– But SGER was funded
• Build targeted ASR from 9-1-1 call data• Build classification part of MT system• Evaluate on unseen data, hopefully
demonstrating sufficient ASR and classification quality to get follow-on
• 18 months, began May 2006– No-Cost Extension to 24 months
![Page 16: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/16.jpg)
Spanish ASR Details
• Janus Recognition Toolkit (JRTk)• CI models initialized from GlobalPhone
data (39 Phones)• CD models are 3 state, semi-continuous
models with 32 gaussians per state• LM trained on Global Phone text corpus
(Spanish news – 1.5 million words)• LM is interpolated with the training data
transcriptions
![Page 17: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/17.jpg)
ASR Evaluation
• Training data – 50 calls (4 hours of speech)
• Dev set – 10 calls (for LM interpolation)
• Test set – 15 calls (1 hour of speech)
• Vocabulary size – 65K words
• Test set perplexity – 96.7
• Accuracy of ASR on test set – 76.5%– Good for spontaneous, multi-speaker
telephone speech
![Page 18: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/18.jpg)
Utterance Classification/Eval
• Can we automatically classify utterances into DAs?
• Manually classified turns into DAs– 10 labels, 845 labelled turns
• WEKA toolkit SVM with simple bag-of- words binary features
• Evaluated using 10-fold cross-validation • Overall accuracy 60.1%
– But increases to 68.8% ignoring “Others”
![Page 19: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/19.jpg)
Initial DA Classification
Tag Frequency Accuracy (%) Acc. w/o “Others” (%)
Giving Name 80 57.50 67.6
Giving Address 118 38.98 63.0
Giving Phone Number 29 48.28 63.6
Req. Ambulance 8 62.50 83.3
Req. Fire Service 11 54.55 75.0
Req. Police 24 41.67 62.5
Report Injury/Urgency 61 39.34 72.7
Yes 119 52.94 71.6
No 24 54.17 81.2
Others 371 75.74 ----
![Page 20: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/20.jpg)
DA classification caveat
• But DA classification was done on human transcriptions (also human utterance segmentation)
• Classifier accuracy on current ASR transcriptions is 40% (49% w/o “Others”)
• Probably needs to be better than that
![Page 21: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/21.jpg)
Future work
• Improving ASR
• Improving classification on real output:– More labelled training data– Use discourse context in classification– “Query expansion” via synsets from Spanish
EuroWordNet– Engineered phone-number-recognizer etc.
• Partial (simpler) return to Nespole! approach
– Better ASR/classifier matching
• Building and user-testing full pilot system
![Page 22: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/22.jpg)
Questions?
http://www.cs.cmu.edu/~911/
![Page 23: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/23.jpg)
![Page 24: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/24.jpg)
Class confusion matrixGN GA GP RA RF RP IU Y N O
GN 46 9 1 3 9 12
GA 9 46 2 3 13 45
GP 8 14 7
RA 1 5 2
RF 6 2 3
RP 1 10 5 8
IU 3 6 24 28
Y 9 11 5 63 31
N 3 13 8
O 12 27 2 3 4 7 25 10 281
![Page 25: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/25.jpg)
Argument Parsing
• Parse utterances using phrase-level grammars
• Nespole! used SOUP Parser (Gavaldà, 2000): Stochastic, chart-based, top-down robust parser designed for real-time analysis of spoken language
• Separate grammars based on the type of phrases that the grammar is intended to cover
![Page 26: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/26.jpg)
Domain Action Classification
• Identify the DA for each SDU using trainable classifiers
• Nespole! used two TiMBL (k-NN) classifiers:– Speech act– Concept sequence
• Binary features indicate presence or absence of arguments and pseudo-arguments
![Page 27: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/27.jpg)
Current status: March 2008 (1)
(At end of extended SGER…)
• Local Spanish transcribers transcribing HIPAA-sanitized 9-1-1 recordings
• CMU grad student (Udhay)– managing transcribers via internal website, – built and evaluated ASR and utt. classifier,– building labelling webpage, prototype, etc.
• Volunteer grad student (Weltman, LSU) analyzing, refining, and using classifier labels
![Page 28: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/28.jpg)
Current status: March 2008 (2)
• “SGER worked.”• Paper on ASR and classification accepted to
LREC-2008• Two additional 9-1-1 centers sending us data• Submitted follow-on small NSF proposal in
December 07: really build and user-test pilot– Letters of Support from three 9-1-1 centers
• Will submit to COLING workshop on safety-critical MT systems
![Page 29: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/29.jpg)
Additional Police Partners
• Julio Schrodel (CCPD) successes:– Mesa PD, Arizona– Charlotte-Mecklenburg PD, NC
• Much larger cities than Cape Coral– (Each is now bigger than Pittsburgh!)
• Uncompressed recordings!• Much larger, more automated 9-1-1
operations– Call-taker vs. dispatcher– User-defined call types logged
![Page 30: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/30.jpg)
Acquired data, as of 3/08
• Miami-Dade County: 5 audio cassettes!• St. Petersburg: 1 call!!
Call Center Calls
CCPD 140
CMPD 392
MesaPD 50
![Page 31: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/31.jpg)
Transcription Status
• VerbMobil transcription conventions
• TransEdit software (developed by Susi Burger and Uwe Meier)
• Transcribed calls: – 97 calls from Cape Coral PD– 13 calls from Charlotte
• Transcribed calls playback time: 9.7 hours
![Page 32: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/32.jpg)
LSU work: Better DA tags
• Manually analyzed 30 calls to find DAs with widest coverage
• Current proposal adds 25 new DAs• Created guidelines for tagging. E.g.:
– If the caller answers an open-ended question with multiple pieces of information, tag each piece of information
• Currently underway: Use web-based tagging tool to manually tag the calls
• Determine inter-tagger agreement
![Page 33: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/33.jpg)
Sample of Proposed Additional DAs Tag Example of utterance
Describing Residence “The left side of a duplex”
Giving Location “I’m outside of the house”,
“The corner of Vine and Hollywood”
Describing Vehicle “A white Ford Ranger”,
“License Plate ALV-325”
Giving Age “She is 3-years-old”
“Born on April 18, 1973”
Describing Clothing “He was wearing a white sweater and black shorts”
Giving Quantity “Only two”
“There are 3 of them right now”
Describing Conflict “He came back and started threatening me”
“My ex-husband won’t leave me alone”
Giving Medical Symptoms “He’s having chest pains”
Asking For Instructions “Should I pull over?”
Asking About Response “How long will it take someone to get here?”
![Page 34: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/34.jpg)
Project origin• Contacted by Julio Schrodel of Cape Coral
PD (CCPD) in late 2003– Looking for technological solution to shortage
of Spanish translation for 9-1-1 calls
• Visited CCPD in December 2003– CCPD very interested in cooperating– Promised us access to 9-1-1 recordings
• Designed system, wrote proposal– CCPD letter in support of proposal– Funded starting May 2006
• (SGER, only for ASR and preparatory work)
![Page 35: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/35.jpg)
![Page 36: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/36.jpg)
Articulatory features
• Model phone as a bundle of articulatory features such as voiced or bilabial
• Less fragmentation of training data
• More robust in handling hyper-articulation– Error-rate reduction of 25% [Metze et al,
ICSLP02]
• Multilingual/crosslingual articulatory features for multilingual settings– Error-rate reduction of 12.3% [Stuecker et al,
ICASSP03]
![Page 37: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/37.jpg)
Grammars plus N-grams
• Grammar-based concept recognition
• Multilingual grammars plus n-grams for efficient multi-lingual decoding [Fuegen et al, ASRU03]
• Multilingual acoustic models
![Page 38: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/38.jpg)
Interchange Format
• Interchange Format (IF) is a shallow semantic interlingua for task-oriented domains
• Utterances represented as sequences of semantic dialog units (SDUs)
• IF representation consists of four parts– Speaker– Speech Act– Concepts– Arguments
speaker : speech act +concept* +arguments*
} Domain Action
![Page 39: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/39.jpg)
![Page 40: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/40.jpg)
Hybrid Analysis Approach
Hello. I would like to take a vacation in Val di Fiemme.c:greeting (greeting=hello)c:give-information+disposition+trip (disposition=(who=i, desire), visit-spec=(identifiability=no, vacation), location=(place-name=val_di_fiemme))
hello i would like to take a vacation in val di fiemme
SDU1 SDU2
greeting= disposition= visit-spec= location=
hello i would like to take a vacation in val di fiemme
greeting give-information+disposition+trip
greeting= disposition= visit-spec= location=
hello i would like to take a vacation in val di fiemme
![Page 41: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/41.jpg)
Hybrid Analysis Approach
Text
Argument
Parser
Text
Arguments
SDU
Segmenter
Text
Arguments
SDUs
DA
Classifier
IF
Use a combination of grammar-based phrase-level parsing and machine learning to produce interlingua (IF) representations
![Page 42: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/42.jpg)
Grammars (1)
• Argument grammar– Identifies arguments defined in the IFs[arg:activity-spec=]
(*[object-ref=any] *[modifier=good] [biking])
– Covers "any good biking", "any biking", "good biking", "biking", plus synonyms for all 3 words
• Pseudo-argument grammar– Groups common phrases with similar meanings into
classess[=arrival=] (*is *usually arriving)
– Covers "arriving", "is arriving", "usually arriving", "is usually arriving", plus synonyms
![Page 43: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/43.jpg)
Grammars (2)
• Cross-domain grammar– Identifies simple domain-independent DAss[greeting]
([greeting=first_meeting] *[greet:to-whom=])
– Covers "nice to meet you", "nice to meet you donna", "nice to meet you sir", plus synonyms
• Shared grammar– Contains low-level rules accessible by all
other grammars
![Page 44: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/44.jpg)
Using the IF Specification
• Use knowledge of the IF specification during DA classification– Ensure that only legal DAs are produced– Guarantee that the DA and arguments
combine to form a valid IF representation
• Strategy: Find the best DA that licenses the most arguments– Trust parser to reliably label arguments– Retaining detailed argument information is
important for translation
![Page 45: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/45.jpg)
Avenue Transfer Rule Formalism (I)
Type information
Part-of-speech/constituent information
Alignments
x-side constraints
y-side constraints
xy-constraints,
e.g. ((Y1 AGR) = (X1 AGR))
;SL: the old man, TL: ha-ish ha-zaqen
NP::NP [DET ADJ N] -> [DET N DET ADJ]((X1::Y1)(X1::Y3)(X2::Y4)(X3::Y2)
((X1 AGR) = *3-SING)((X1 DEF = *DEF)((X3 AGR) = *3-SING)((X3 COUNT) = +)
((Y1 DEF) = *DEF)((Y3 DEF) = *DEF)((Y2 AGR) = *3-SING)((Y2 GENDER) = (Y4 GENDER)))
![Page 46: NineOneOne: Recognizing and Classifying Speech for Handling Minority Language Emergency Calls](https://reader036.vdocument.in/reader036/viewer/2022070405/56814027550346895dab8813/html5/thumbnails/46.jpg)
Avenue Transfer Rule Formalism (II)
Value constraints
Agreement constraints
;SL: the old man, TL: ha-ish ha-zaqen
NP::NP [DET ADJ N] -> [DET N DET ADJ]((X1::Y1)(X1::Y3)(X2::Y4)(X3::Y2)
((X1 AGR) = *3-SING)((X1 DEF = *DEF)((X3 AGR) = *3-SING)((X3 COUNT) = +)
((Y1 DEF) = *DEF)((Y3 DEF) = *DEF)((Y2 AGR) = *3-SING)((Y2 GENDER) = (Y4 GENDER)))