lecturing with digital ink richard anderson university of washington
Post on 21-Dec-2015
213 views
TRANSCRIPT
![Page 1: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/1.jpg)
Lecturing with Digital Ink
Richard AndersonUniversity of Washington
![Page 2: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/2.jpg)
Lessons learned from the Classroom Presenter project
Classroom Pedagogy Teaching with ink
HCI Ink based presentation
Multimedia Analysis of lecture artifacts
![Page 3: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/3.jpg)
Classroom Presenter
Integration of slides and digital ink using Tablet PC
Key ideas: Ink overlay on images Distributed application
Many other systems also support ink and slides
![Page 4: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/4.jpg)
Classroom Presenter as a distributed application Designed as
distributed application for distance learning
Enables many scenarios
Mobility Walking and talking
Sharing materials with students
Note taking Classroom interaction
Student submissions
![Page 5: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/5.jpg)
Deployments
Estimated use in at least 100 courses Wide use inside of computer science Push for adoption outside of CS Lecture archives from UW
Professional Master’s Program Several hundred hours of recorded audio,
video, and ink.
![Page 6: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/6.jpg)
Distance Learning Classes
![Page 7: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/7.jpg)
“Typical ink usage”
![Page 8: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/8.jpg)
“Typical ink usage”
![Page 9: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/9.jpg)
Planning for ink usage
![Page 10: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/10.jpg)
Ink use in presentation
Cognitive load Only limited attention available for
computer while lecturing Linkage with speech
Close tie between ink and speech
![Page 11: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/11.jpg)
Cognitive load Limited feature use
Even color change unusual User interface must be simple (and
robust) Cannot give feedback to user Many actions appear to minimize
mental effort Color change only for contrast Reliance on screen erase
![Page 12: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/12.jpg)
Understanding Attentional Marks Properties
Brief, simple markings Occur with speech Augment meaning of speech Ad hoc form
Is there a linguistic context in which to understand these marks?
![Page 13: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/13.jpg)
Spontaneous Hand Gestures
Spontaneous Hand gestures [McNeill]: are synchronous w/speech are co-expressive w/speech lack standard of form
Attentional marks share these properties.
![Page 14: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/14.jpg)
Gesture Types: Iconic
![Page 15: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/15.jpg)
Gesture Types: Deictic & Cohesive
![Page 16: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/16.jpg)
Analysis of digital ink Understand ink usage Motivation: inform development of
ink based applications Archiving
Search, Summarization, Transcription Lecture based
Improved rendering, note taking, accessibility
![Page 17: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/17.jpg)
Ink classification Textual Diagrammatic Attentional
% of strokes % of episodes
B C B+C B C B+C
Attentional 49 53 51 77 74 76
Diagram 9 7 8 8 8 8
Writing 41 38 40 14 16 15
Other 1 2 1 2 2 2
Coding of six hours of lecture
![Page 18: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/18.jpg)
Goals
Understand usage “in the wild” Cannot expect lecturers to modify
behavior Determine opportunities for
automatic analysis Identify challenges
![Page 19: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/19.jpg)
Methodology
Study of recorded classes Best data set: Professional
Master’s Program Distance courses Audio, Video, Ink archives HCI, Compilers, Programming
Languages, AI, Transaction Processing
![Page 20: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/20.jpg)
Attentional ink Problem – content
matching Identify slide content
referred to by ink Study
Implement basic algorithms to match attention marks to slide content
Compare results with human coders
![Page 21: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/21.jpg)
Attentional ink
Determine the lecturer’s intent:
Determine level to parse the content
![Page 22: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/22.jpg)
Attentional ink Challenges
Recognition of attentional ink on text
Difficult example:
![Page 23: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/23.jpg)
Handwriting
How well does handwriting recognition work on “typical” instructor writing? Domain has many challenges
![Page 24: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/24.jpg)
Recognition Study Studied isolated
words/phrases written on slides
Removed non-textual ink
Fed through the Microsoft Handwriting Recognizer
No training
![Page 25: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/25.jpg)
Recognition Examples The Good:
The Bad:
The Ugly:
![Page 26: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/26.jpg)
Handwriting Reco Results
Exact Alternate
Close None
Prof. A
16 (88%) 1 (6%) 0 (0%) 1 (6%)
Prof. B
146 (59%)
26 (10%) 6 (2%) 71 (29%)
Prof. C
18 (42%) 5 (11%) 1 (3%) 19 (44%)
Prof. D
262 (61%)
45 (11%) 9 (2%)111
(26%)
Prof. E
408 (79%)
46 (9%) 2 <(1%) 58 (11%)
Total 850 (68%)
123 (10%)
18 (1%)260
(21%)
![Page 27: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/27.jpg)
Joint Writing and Speech Recognition Can we use handwriting recognition
with speech recognition together to improve accuracy?
Co-expression of ink and speech Are written words spoken as well?
Can speech disambiguate handwriting?
Can handwriting disambiguate speech?
![Page 28: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/28.jpg)
Examples Difficult for Speech and Ink Recognition
Difficult Written Abbreviations
Speech/Ink Used to Disambiguate Ink/Speech
![Page 29: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/29.jpg)
Experiment Examined instances of isolated word
writing Selected word writing episodes at random
but uniformly from the various instructors Generated transcripts manually from the
audio Checked whether the instructor spoke the
exact word written Measured the time between the written
and spoken word
![Page 30: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/30.jpg)
Speech/Text Co-occurrence Results
Exact Approx None Simul 0-2s > 2s
A 1 (100%) 0 (0%) 0 (0%) 1 (100%) 0 (0%) 0 (0%)
B 9 (75%) 3 (25%) 0 (0%) 12 (100%) 0 (0%) 0 (0%)
C 9 (82%) 2 (18%) 0 (0%) 10 (91%) 1 (9%) 0 (0%)
D 12 (86%) 2 (14%) 0 (0%) 10 (71%) 4 (29%) 0 (0%)
E 9 (56%) 7 (44%) 0 (0%) 7 (44%) 4 (25%) 5 (31%)
Total 40 (74%) 14 (26%) 0 (0%) 40 (74%) 9 (17%) 5 (9%)
![Page 31: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/31.jpg)
Activity Recognition Identifying slide corrections
![Page 32: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/32.jpg)
Example Results
![Page 33: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/33.jpg)
Diagrammatic ink
How do instructors use diagrams Basic legibility Observed behaviors
Diagram phasing Locality of expression
![Page 34: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/34.jpg)
![Page 35: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/35.jpg)
Typical diagram Basic, irregular
shapes Difficult labels Attentional ink
![Page 36: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/36.jpg)
More examples
![Page 37: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/37.jpg)
Zipf diagram
![Page 38: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/38.jpg)
Stroke order
![Page 39: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/39.jpg)
Diagram phasing
![Page 40: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/40.jpg)
More phasing
![Page 41: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/41.jpg)
Top arrows: “Not there”
Separate wins indicated together
Locality in diagrams
![Page 42: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/42.jpg)
Summary
Pedagogy with ink How is ink used in conjunction with
content and speech to express information
Presentation with ink Low attention task
Analysis of ink usage Extracting meaning from archived
lectures
![Page 43: Lecturing with Digital Ink Richard Anderson University of Washington](https://reader031.vdocument.in/reader031/viewer/2022032522/56649d6c5503460f94a4bce7/html5/thumbnails/43.jpg)
Resources
cs.washington.edu/education/dl/presenter/ Software Downloads Papers
Contact info Richard Anderson,
[email protected] Ruth Anderson, [email protected] Craig Prince, [email protected]