esat/psi-speech hugo van hammeesat/psi-speech hugo van hamme nsf workshop may 2015 4 july 2012...
TRANSCRIPT
![Page 1: ESAT/PSI-Speech Hugo Van hammeESAT/PSI-Speech Hugo Van hamme NSF workshop May 2015 4 July 2012 Leuven KU Leuven introduction 2 Leuven - university • Associatie K.U.Leuven – University](https://reader033.vdocument.in/reader033/viewer/2022052815/60a6054b0e66e846b135592d/html5/thumbnails/1.jpg)
ESAT/PSI-Speech
Hugo Van hamme
NSF workshop
May 2015
4 July 2012
![Page 2: ESAT/PSI-Speech Hugo Van hammeESAT/PSI-Speech Hugo Van hamme NSF workshop May 2015 4 July 2012 Leuven KU Leuven introduction 2 Leuven - university • Associatie K.U.Leuven – University](https://reader033.vdocument.in/reader033/viewer/2022052815/60a6054b0e66e846b135592d/html5/thumbnails/2.jpg)
Leuven
KU Leuven introduction2
![Page 3: ESAT/PSI-Speech Hugo Van hammeESAT/PSI-Speech Hugo Van hamme NSF workshop May 2015 4 July 2012 Leuven KU Leuven introduction 2 Leuven - university • Associatie K.U.Leuven – University](https://reader033.vdocument.in/reader033/viewer/2022052815/60a6054b0e66e846b135592d/html5/thumbnails/3.jpg)
Leuven - university
• Associatie K.U.Leuven– University + 12 university colleges
– 85000 students, 600 programs
• KU Leuven– 35000 students 350 programs
• Faculty of Engineering– 2000 Students (Ba+Ma), 60 programs
• Department of Electrical Engineering (ESAT)- 150 Ma students, 6 programs
- 270 PhD students and postdocs …
- 35 FTE permanent staff
Centre for Processing of Speech and Images (PSI)- 37 PhD students and postdocs
- 8 FTE permanent staff
- Speech research group- 12 PhD students and 1.4 postdocs
- 5 Master students
- 2.3 FTE permanent staff- Patrick Wambacq
- Dirk Van Compernolle
- Hugo Van hamme KU Leuven introduction3
![Page 4: ESAT/PSI-Speech Hugo Van hammeESAT/PSI-Speech Hugo Van hamme NSF workshop May 2015 4 July 2012 Leuven KU Leuven introduction 2 Leuven - university • Associatie K.U.Leuven – University](https://reader033.vdocument.in/reader033/viewer/2022052815/60a6054b0e66e846b135592d/html5/thumbnails/4.jpg)
• noise robustness – speech enhancement – source separation – source localization
• new paradigms for speech recognition – episodic models
• build and consolidate digital infrastructure for the Dutch language
• speaker properties (text-independent): ID, language, dialect, age, height
• acoustic environment modeling – ADL recognition
• zero-resource ASR - language acquisition by machines
• speech assessment - education
ESAT/PSI-Speech research areas
![Page 5: ESAT/PSI-Speech Hugo Van hammeESAT/PSI-Speech Hugo Van hamme NSF workshop May 2015 4 July 2012 Leuven KU Leuven introduction 2 Leuven - university • Associatie K.U.Leuven – University](https://reader033.vdocument.in/reader033/viewer/2022052815/60a6054b0e66e846b135592d/html5/thumbnails/5.jpg)
Speech assessment
• Reading tutor (dyslexia) / trainer after CI fitting
• Assess native (?) pronunciation, reading/respeak tracking
• Children’s speech, hesitant, poorly articulated
![Page 6: ESAT/PSI-Speech Hugo Van hammeESAT/PSI-Speech Hugo Van hamme NSF workshop May 2015 4 July 2012 Leuven KU Leuven introduction 2 Leuven - university • Associatie K.U.Leuven – University](https://reader033.vdocument.in/reader033/viewer/2022052815/60a6054b0e66e846b135592d/html5/thumbnails/6.jpg)
Zero-resource speech recognition
Why ?
• Assistive technologies:
– people with limited fine motor control
– alternative to scanning
– cope with dysarthric voices
• Huge inter-speaker variation
• Timing, extraneous sounds
• Dialects
• Long-term: interacting with robots
– “Fetch a Hoegaarden Grand Cru from the fridge”
– “Get my red slippers”
– “Open the garden window for me”
![Page 7: ESAT/PSI-Speech Hugo Van hammeESAT/PSI-Speech Hugo Van hamme NSF workshop May 2015 4 July 2012 Leuven KU Leuven introduction 2 Leuven - university • Associatie K.U.Leuven – University](https://reader033.vdocument.in/reader033/viewer/2022052815/60a6054b0e66e846b135592d/html5/thumbnails/7.jpg)
What’s different ?
• Learn acoustic model and language model from
examples with noisy, high-level supervision information
– Not like traditional ASR
– Not like the zero-resource challenge (IS15)
• Our first steps:
– Home automation
– “open the kitchen door”, “kitchen door open”
– Learn from demonstrations = weak supervision
• Learn acoustic model, vocabulary and grammar (ASR)
• Learn mapping to semantic frames (NLU)
![Page 8: ESAT/PSI-Speech Hugo Van hammeESAT/PSI-Speech Hugo Van hamme NSF workshop May 2015 4 July 2012 Leuven KU Leuven introduction 2 Leuven - university • Associatie K.U.Leuven – University](https://reader033.vdocument.in/reader033/viewer/2022052815/60a6054b0e66e846b135592d/html5/thumbnails/8.jpg)
VIVOCA results
![Page 9: ESAT/PSI-Speech Hugo Van hammeESAT/PSI-Speech Hugo Van hamme NSF workshop May 2015 4 July 2012 Leuven KU Leuven introduction 2 Leuven - university • Associatie K.U.Leuven – University](https://reader033.vdocument.in/reader033/viewer/2022052815/60a6054b0e66e846b135592d/html5/thumbnails/9.jpg)
Work ahead
• Larger vocabularies
– How does a word spurt come about ?
• Faster learning
– Ideally from one example
• More complex instructions and semantic representations
– Continuous state space
– Dynamic representation of semantics
– Uncertainty in meaning
– ...
– Related to many actual research topics in robotics
![Page 10: ESAT/PSI-Speech Hugo Van hammeESAT/PSI-Speech Hugo Van hamme NSF workshop May 2015 4 July 2012 Leuven KU Leuven introduction 2 Leuven - university • Associatie K.U.Leuven – University](https://reader033.vdocument.in/reader033/viewer/2022052815/60a6054b0e66e846b135592d/html5/thumbnails/10.jpg)
What’s needed ?
Speech assessment
• Investment in non-native and regional accent data
• Getting government involved is hard (budget cuts etc.)
Zero-resource ASR
• Interaction data
– grow complexity of task
– Limited reuse from one task to the next
• Understanding by community of relevance of the problem
– Cfr. reviewer instructions for IS15 Zero-resource Challenge
• Investment attitude in Europe/Belgium
• Industrial interest is growing internationally
![Page 11: ESAT/PSI-Speech Hugo Van hammeESAT/PSI-Speech Hugo Van hamme NSF workshop May 2015 4 July 2012 Leuven KU Leuven introduction 2 Leuven - university • Associatie K.U.Leuven – University](https://reader033.vdocument.in/reader033/viewer/2022052815/60a6054b0e66e846b135592d/html5/thumbnails/11.jpg)
Questions ?