STARDUST – Speech Training And Recognition for Dysarthric Users of
Assistive Technology
Mark Hawley et alBarnsley District General Hospital and
University of Sheffield
STARDUST
• To develop speech-driven environmental control and voice output communication devices for people with dysarthria
– To develop a reliable small vocabulary speech recogniser for dysarthric speakers
– To develop a computer training program to help to stabilise the speech of dysarthric speakers
Research Team
• Department of Medical Physics and Clinical Engineering, Barnsley District General Hospital (Mark Hawley, Simon Brownsell, Stuart Cunningham)
• Institute of General Practice and Primary Care, University of Sheffield (Pam Enderby, Mark Parker, Rebecca Palmer)
• Department of Computer Science, University of Sheffield (Phil Green, Nassos Hatzis, James Carmichael)
• Project funded by Dept of Health New and Emerging Applications of Technology (NEAT) programme
Dysarthria
A neurological motor speech impairment characterised by slow, weak, imprecise and/or uncoordinated
movements of the speech musculature.
Speech is often difficult to understand (unintelligible) and variable (inconsistent)
Frequently associated with other physical disabilities
Severe = <40% intelligible
Speech-input writing programmes
• Normal speech - with recognition training can get >90% recognition rates (Rose and Galdo, 1999)
• Mild dysarthric speech - 10-15% lower recognition rates (Ferrier, 1992)
• Recognition declines as speech deteriorates - by 30-50% for single words (Thomas-Stonell, 1998, Hawley 2002)
Performance of a commercial speaker-dependent recogniser
(in ‘ideal’ conditions)
Recognition rateN=6
Dysarthric subject 1 60%
Dysarthric subject 2 80%
‘Normal’ control 98%
Difficult speech recognition problem
• Dysarthric speech– different to ‘normal’ models– more variable than ‘normal’ speech, both
between and within speakers– difficult to collect large corpus of speech
STARDUST
• To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria
• To develop a reliable small vocabulary speech recogniser for dysarthric speakers
• To develop a computer training program to help to stabilise the speech of dysarthric speakers(ie improve consistency) and improve recognition
Intelligibility and Consistency
• ‘Normal’ speech will be almost 100% intelligible and with few articulatory differences over time (consistent).
• ‘Severe’ dysarthria may be completely unintelligible to the naïve listener and will show high variability (inconsistent)– but may show consistency of key elements which will
make it more intelligible to the familiar listener.
• STARDUST is concerned with consistency
Training program
• Visual feedback to improve consistency at word level– Quantitative – Real time
• To be used by the client alone or with carer or therapist
• Training tool records speech - used to build recogniser
Training program set-up
• Record 10 examples of each word to be trained• Program builds models of words based on
examples• For each word, program selects example that best
matches its model (the best-fit recording)• Program feeds back a measure of the match
between last utterance and model
Outcome of speech training(preliminary data)
0 2 4 6 8 10 12 14 16 18 20 22-69
-68
-67
-66
-65
-64
Session Number
Mean L
og P
robabili
ty
0 1 2 3 4 5 6 7 8 9-74
-73
-72
-71
-70
-69
-68
-67
-66
-65
-64
Session Number
Mean L
og P
robabili
ty
In a group of 5 users, 3 showed an upward trend, 2 showed no upward trend
STARDUST
• To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria
• To develop a reliable small vocabulary speech recogniser for dysarthric speakers
• To develop a computer training program to help to stabilise the speech of dysarthric speakers
Recognition technology
• Small vocabulary
• Speaker dependent
• uses hidden Markov models• based on HTK (University of Cambridge)
STARDUST recogniser performance (N=number of words used for training)
IntelligibilitySingle words-sentences
STARDUSTrecogniser
N=6
STARDUSTrecogniser
N=20
STARDUSTrecogniser
N=28
CommercialspeechrecognitionECSN=6
Subject 1 0% - 0% 64% 80% 85% 60%
Subject 2 22% - 34% 100% 100% 100% 80%
Control 100% 100% 100% 100% 98%
STARDUST
• To develop demonstrators of speech-driven environmental control and voice output communication devices for people with dysarthria
• To develop a reliable small vocabulary speech recogniser for dysarthric speakers
• To develop a computer training program to help to stabilise the speech of dysarthric speakers
Vocabulary mapping
• One to one (word to phrase) mapping– ‘Want’ = I need something, could you help me?
• Pseudo-grammatical combinations– ‘Want ... drink’ = Could I have a drink, please?
• Coding– ‘3…6…4’ = I went to Spain for my holidays
– nm possible combinations, where n is no of words in vocab, m is length of vocabulary string
Work in progress
• Test systems in home-based field trials– acceptability– usability (eg speed of access)– accuracy– reliability– practicality
Work in progress
• Remove switch activation of recogniser
• Increase vocabularies to test limits of recogniser
• Develop tools for clinicians to build and test individual configurations
STARDUST - conclusions
• Recogniser that recognises severely dysarthric speech
• Computer-based training program to improve recognition and consistency – word level (and sub-word level in future)
– collects lots of speech data for recogniser
• Developed demonstrators of environmental control and voice-output device– next step to test in real usage