towards using machine learning to detect dialogic instruction
DESCRIPTION
Towards using machine learning to detect dialogic instruction. Andrew Olney University of Memphis. Classroom Discourse. Human Measurement. Human Measurement. Project Goal. Replace trained human coders with artificial intelligence Automatic Speech Recognition - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/1.jpg)
Towards using machine learning to detect dialogic instruction
Andrew OlneyUniversity of Memphis
![Page 2: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/2.jpg)
Classroom Discourse
![Page 3: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/3.jpg)
Human Measurement
![Page 4: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/4.jpg)
Human Measurement
![Page 5: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/5.jpg)
Project Goal
• Replace trained human coders with artificial intelligence– Automatic Speech Recognition– Natural Language Understanding
![Page 6: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/6.jpg)
Methodology
• Collect new measurements with advanced sensors– Microphones and Kinects
• Build models using archival data– Machine learning using human codes
![Page 7: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/7.jpg)
New Measurements
• ~400 minutes of audio
![Page 8: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/8.jpg)
Speech Recognition
• Assessed systems– Microsoft Speech– Google Speech– Sphinx 4:WSJ– Sphinx 4:HUB4
![Page 9: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/9.jpg)
Speech Recognition: Overall
![Page 10: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/10.jpg)
Speech Recognition: By Class
![Page 11: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/11.jpg)
Speech Recognition: By Teacher
![Page 12: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/12.jpg)
Kinects
• Microphone array
• Difference in time reception = angle
![Page 13: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/13.jpg)
Heatmaps
![Page 14: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/14.jpg)
Models using Archival Data
• Question-level codes• Text• Speaker• Part of multiple response• Cognitive Level• Authenticity• Uptake
• Machine learning • Observed Question Properties Hidden Question
Properties
![Page 15: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/15.jpg)
Models using Archival Data
• Isolated question vs. Question event
• Is it possible to recover these variables from the isolated question?
![Page 16: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/16.jpg)
Models using Archival Data
![Page 17: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/17.jpg)
Models using Archival Data
Text Who can tell me the important word that you have to put down on the word list for Amelia J. Bloomer.
Speaker Teacher
Part of multiple
No
Cognitive level
Recitation
Authentic NoUptake No
![Page 18: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/18.jpg)
Authenticity: Hidden
• Open ended questions
• Leads to open-ended discussion– “What was your reaction to the end of the story?”
![Page 19: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/19.jpg)
Uptake: Hidden
• Question about something that another person has said previously
• Emphasizes the importance of students’ contribution– “Why did you say that?”
![Page 20: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/20.jpg)
Question: Observed
• Olney, et al. 2003– Based on part of speech, keywords, etc.– 16 question categories (Graesser & Person, 1994; Graesser,
Person, & Huber, 1992)
– Rule Based System
• Different kinds of questions might lead to different levels of uptake and authenticity
![Page 21: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/21.jpg)
Methodology
• Using machine learning– Predict Authenticity from question text– Predict Uptake from question text
• Compare these models with– Original human codes (coding in context)– New human codes (coding in isolation)
![Page 22: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/22.jpg)
Archival Data: In Context
• Partnership for Literacy Study (Partnership)– 7th and 8th grade English and language
arts– 120 classrooms in 23 schools (observed
twice)– 29,673 instances (25,711 utilized)
• Raters agreed on question properties approximately 80% of the time
![Page 23: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/23.jpg)
New Ratings: In Isolation
• Four human raters
• Uniform sampling (100 questions per code)
• Randomly ordered isolated questions
![Page 24: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/24.jpg)
Machine Learning: Features
• Binary: does word appear in the question?
• Casual consequent words: results, effects, etc.
• Procedural, Feature specification, Negation, Meta-communication, Metacognition, Comparison, Goal orientation, Judgmental Definition, Enablement, Interpretation, Example, Quantification, Casual antecedent, Disjunction
![Page 25: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/25.jpg)
Machine Learning: Features
• Positional: does word appear in – Beginning, Middle, End?
• Part of speech tags: Determiner, Noun, Pronoun, Adjective, Adverb, and Verb
• Question stems: Do/Have, Be, Modal, What,
How, Why
![Page 26: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/26.jpg)
Machine Learning: Model
• J48 Decision Tree in WEKA (Witten, et al., 2011)
• Evenly balanced classes
• Separate models for Authenticity and Uptake
• Both using features described
![Page 27: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/27.jpg)
Results: In Isolation
• Inter-rater reliability (kappa)
Authenticity
Uptake
![Page 28: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/28.jpg)
Results: In Isolation vs. In Context
• Authenticity
- Kappa Recognition Rate
R1 0.13 56%
R2 0.17 58%
R3 0.25 62%
R4 0.10 55%
Model 0.34 67%
![Page 29: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/29.jpg)
Results: In Isolation vs. In Context
• Uptake
Kappa Recognition rate
R1 0.22 61%
R2 0.25 62%
R3 0.30 65%
R4 0.23 61%
Model 0.46 73%
![Page 30: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/30.jpg)
Results: In Isolation, All Data
• 10 fold cross validation
Models Kappa Accuracy
Authenticity 0.28 64%
Uptake 0.24 62%
![Page 31: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/31.jpg)
Results: Unpacking the Model
• Correlation-based Feature Subset Selection (CFS)– Authenticity
Judgmental keywords, WH words, Enablement keywords, and “what”
– UptakeNegation keywords, Judgmental keywords, and “why”
![Page 32: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/32.jpg)
Results: Unpacking the Model
• Authenticity:“Do you think enterprising people always need to be audacious?”“Did you find it helpful?”“Do you think it needed to go on the next ten lines?”
• Uptake:“Why do you think he wants to help the little boy?”“You think he can't get help, Can you expand on that?”“Like if I make a connection to my life and not to all
three of them do you think that that might help?”
![Page 33: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/33.jpg)
Results: Key-Keywords
• Judgmental keywords
![Page 34: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/34.jpg)
Results: Key-Keywords
• Enablement keywords
![Page 35: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/35.jpg)
Results: Key-Keywords
• Negation keywords
![Page 36: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/36.jpg)
Conclusion
• Models >= Human on isolated questions
• But still 20% gap between isolated and full– Gap applies to model AND humans
• What cues are missing? Future work!!!
![Page 37: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/37.jpg)
Team
University of Memphis
Andrew Olney Art Graesser Borhan Samei Sean Kelly Laura Northrop
University of Wisconsin
Martin Nystrand Marci Glaus Xiaoyi Sun Sidney D’Mello Nathan Blanchard
University of Notre Dame
(PI)
University of Pittsburgh
![Page 38: Towards using machine learning to detect dialogic instruction](https://reader036.vdocument.in/reader036/viewer/2022062721/56813623550346895d9d9983/html5/thumbnails/38.jpg)
Questions?
http://www.english.wisc.edu/nystrand/class.html
http://andrewmolney.name
This research was supported by the Institute of Education Sciences (IES), U.S. Department of Education, through Grant R305A130030. Any opinions, findings and conclusions, or recommendations expressed are those of the authors and do not necessarily reflect the views of IES