uses for automatic speech recognition with diverse english speakers
DESCRIPTION
Uses for Automatic Speech Recognition with Diverse English Speakers. 2002 American Speech-Language-Hearing Association Annual Convention Atlanta, Georgia World Congress Center, Room: A314, Saturday, Nov 23 2002 4:30PM – 5:30PM - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/1.jpg)
Uses for Automatic Speech Recognition with Diverse English Speakers
2002 American Speech-Language-Hearing Association Annual Convention
Atlanta, Georgia World Congress Center, Room: A314, Saturday, Nov 23 2002 4:30PM – 5:30PM
Presenters/Authors: Kathleen Eilers Crandall, Ph.D., Paula M. Brown, Ph.D., Donna E. Gustina, and Stephen S. Campbell
National Technical Institute for the DeafRochester Institute of Technology
![Page 2: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/2.jpg)
Seminar – PresentersKathleen Eilers crandall,
Ph.D.Department of English, National Technical Institute for the Deaf, Rochester Institute of Technology
Paula M. Brown, Ph.D., CCC-SLP Department of Speech and Language, National Technical Institute for the Deaf, Rochester Institute of Technology
![Page 3: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/3.jpg)
The Glossograph
• Fay wrote about an experimental mechanical device used to transcribe human speech, and said,
• “… it is not unreasonable to hope that some instrument will yet be contrived …“
Fay, E.A. (1883). The glossograph. American Annals of the Deaf, 28, 67-69.
![Page 4: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/4.jpg)
Sci-Fi or Reality?
"The pen was an archaic instrument, seldom used even for signatures...Apart from very short notes, it was usual to dictate everything into the speak-write…” (Nineteen eighty-four. Orwell, 1949)
![Page 5: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/5.jpg)
Two Projects
• Teacher use of ASR:– English Classroom/Lab Project
• Student use of ASR:– Speech Project
Funded by a grant from the Parsons Foundation of California
![Page 6: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/6.jpg)
English Classroom/Lab Project
![Page 7: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/7.jpg)
English Classroom/Lab Project
Purpose
Investigate direct use of ASR by classroom teacher to learn:
• Is acceptable recognition level attained?
• Under what conditions?– Style of speaking– Communication mode– Language complexity
![Page 8: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/8.jpg)
Related Work
Use of ASR by an intermediary • Intermediary, a ‘captionist,’ re-speaks
professor’s words into a computer• Intermediary summarizes professor’s
words into a computer (‘interpreted speech’)
• Intermediary may use C-print (a shorthand typing system) in combination with ASR http://cprint.rit.edu/
![Page 9: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/9.jpg)
Related Work
Use of ASR by the primary speaker
• iCommunicator™ http://www.myicommunicator.com/product_info.html
• Liberated Learning Environment http://www.liberatedlearning.com (St. Mary’s University, Halifax, Nova Scotia)
![Page 10: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/10.jpg)
Speech Project
![Page 11: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/11.jpg)
Speech Project Intent
• Can ASR become better than a naïve listener?
• Can ASR serve as an effective and motivating feedback system?
![Page 12: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/12.jpg)
Speech Project How ASR Is Used Educationally
Visual displays provide feedback regarding speech production
• Natural way of learning
• Expect feedback to reflect accuracy– Assume if don’t get right picture, you were
wrong
![Page 13: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/13.jpg)
English Classroom/Lab Project
![Page 14: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/14.jpg)
English Classroom/Lab Project
Teacher -- Students• Teacher -- Speaker
– Native speaker of American English– User of ASL as a second language – Trained the ASR equipment
• Students -- Readers – Young adult college students who are deaf or hard-of-
hearing– Reading and writing skills at the lowest quartile of
entering students– Enrolled in basic level English language reading and
writing courses
![Page 15: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/15.jpg)
English Classroom/Lab Project
Evaluation Procedures
• ASR Software: – Dragon Naturally Speaking– IBM ViaVoice– Microsoft Office
• Speaking styles: – Spontaneous conversation– Dictation-like speech
• Communication modes:– Speaking– Simultaneously speaking and signing
![Page 16: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/16.jpg)
English Classroom/Lab
Teacher stationControl systemSmart Board & LCD Projector
Student Stations
![Page 17: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/17.jpg)
English Classroom/Lab Project
Accuracy Needs
• Vary by population and message predictability– New vs. Known information– Fluent readers vs.
Language learners– Reading for pleasure vs. Reading to master new
information
• CLOZE research and prediction of missing information
![Page 18: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/18.jpg)
English Classroom/Lab Project
Results: ASR Software
75%
80%
85%
90%
95%
100%
Dragon ViaVoice XP
Conversation
Dictation
![Page 19: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/19.jpg)
English Classroom/Lab Project
Results: Communication Mode
80%
82%
84%
86%
88%
90%
92%
94%
96%
98%
Simultaneous Commmunication Speech Only
Conversation
Dictation
![Page 20: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/20.jpg)
English Classroom/Lab Project
Results: Language Complexity
82%
84%
86%
88%
90%
92%
94%
96%
98%
< 7th Grade > 7th Grade
Conversation
Dictation
![Page 21: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/21.jpg)
English Classroom/Lab Project
Correcting Text
• Error correction– What to correct – When to correct– How to correct
![Page 22: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/22.jpg)
Multitasking Demands
• Normal tasks for speaker/teacher– Formulating ideas relevant to topic– Attending to learning needs of students – Meeting lipreading and sign language needs
• Added tasks for speaker/teacher – Speaking to produce readable ASR text– Monitoring text– Making corrections
![Page 23: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/23.jpg)
Speech Project
![Page 24: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/24.jpg)
Speech Project
Training Sequence
• Read a paragraph
• Correct and train recognition errors
• Reread paragraph
• Correct and train recognition errors
• Create transfer paragraph or spontaneous speech
• Correct and train recognition errors
![Page 25: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/25.jpg)
Recognition Accuracy
0%
10%
20%30%
40%
50%
60%
70%
80%90%
100%
M Intel F semi-intel F quasi-intel
![Page 26: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/26.jpg)
Improvement Across Sessions
0%
10%
20%
30%
40%
50%
60%
70%
80%
time 1 time 2 time 3 time 4 time 5
![Page 27: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/27.jpg)
Improvement Within Session
65%
70%
75%
80%
85%
90%
95%
Reading 1 Reading 2 Reading 3 Spon Sp
![Page 28: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/28.jpg)
Speech Project
Improvement Evaluated
• Improvement across sessions
• Improvement within a session– Improvement with speaker training– Improvement with ASR training
![Page 29: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/29.jpg)
RecommendationsDiscussionQuestions
![Page 30: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/30.jpg)
Grammatical Correctness
• Is ASR accuracy affected by the grammatical correctness of the user’s speech?
• Student written responses spoken as written: Accuracy – 93.8%
• Student written responses spoken after corrected: Accuracy - 94.3%
![Page 31: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/31.jpg)
Style of Speaking
1. Style of speaking that more closely resembles dictation approaches a usable accuracy rate.
2. Lowering the complexity does not improve accuracy.
![Page 32: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/32.jpg)
Conditions of Use
Direct use of ASR by a language teacher --Useful only under very controlled conditions.• Illustrating the generation of written
language • Demonstrating the use of notes and
outlines to produce written text• Translating selected sign language
utterances into English text during discussions
![Page 33: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/33.jpg)
ASR: Classroom Use
Prepared Outline
Student’s Screen
Teacher’s Screen
![Page 34: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/34.jpg)
Considerations• Training
– Critical to reach over 90% accuracy– Training with conversation
• Corrections– Familiarity with strategies – Dictate, Spell, Right click
• Equipment– Microphone headsets - design, comfort, and size– Demand on computer processor– Effect of optional settings
![Page 35: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/35.jpg)
Language Processing
Teaching/Learning Issues:• Does ASR promote the learning of reading
and writing for Deaf and Hard-of-Hearing students?
• How do students process this information?• Do students attend to multiple inputs?• Can teachers attend to this many tasks
effectively?
![Page 36: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/36.jpg)
More Questions
• Who is at fault?– Speaker or ASR receiver?
• Acceptability of input– Various voices– Nontypical speakers
• User friendliness– Want immediate use
![Page 37: Uses for Automatic Speech Recognition with Diverse English Speakers](https://reader030.vdocument.in/reader030/viewer/2022033106/568137e4550346895d9f91e5/html5/thumbnails/37.jpg)
PresentersKathleen Eilers Crandall, Ph.D.Department of English
National Technical Institute for the Deaf
Rochester Institute of Technology Lyndon Baines Johnson Building -
2264
Phone: (585) 475-5111
Fax: (585) 475-6500
Email: [email protected]
Web: http://www.rit.edu/~kecncp
Paula M. Brown, Ph. D., CCC-SLP
Department of Speech and Language
National Technical Institute for the Deaf
Rochester Institute of Technology Lyndon Baines Johnson Building -
3851
Phone: (585) 475-6593 V/TDD
Fax: (585) 475-6500
Email: [email protected]
Web: http://www.rit.edu/~462www/