discovering real-world usage for a multimodal math search interface
DESCRIPTION
Thesis Defense Presentation by Keita (Del Valle) Wangari for Masters of Science in Human-Computer Interaction at Rochester Institute of Technology.TRANSCRIPT
![Page 1: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/1.jpg)
Discovering Real World Usage for a
Multimodal Math Search InterfaceThesis Defense | Keita Del Valle Wangari
Rochester Institute of Technology
December 16, 2013
![Page 2: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/2.jpg)
THE PROBLEMThe Intention Gap.
![Page 3: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/3.jpg)
![Page 4: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/4.jpg)
![Page 5: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/5.jpg)
![Page 6: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/6.jpg)
How do you “Google it”?
![Page 7: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/7.jpg)
Query options
• n choose k (verbal expression)
• binomial coefficient (related term/concept)
• {{n}choose{k}} (LaTeX)
• …(template editor)
![Page 8: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/8.jpg)
Target Audience
Non-experts in the math domain:
• less likely to know math expression names
• less likely to know a math encoding language
• less likely to be familiar with math template editors or the
sites that use them
![Page 9: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/9.jpg)
Intention Gap
Intention Gap
(Zha et al, 2010)
![Page 10: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/10.jpg)
Intention Gap
![Page 11: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/11.jpg)
To use math expressions in search, current search engines
require knowing expression names or using a structure
editor or encoding language (e.g., LaTeX) to enter
expressions. For people who are not math experts, this can
lead to an “intention gap” between the math query they
wish to express, and what the interface will allow.
Problem Statement
![Page 12: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/12.jpg)
THE STUDY GOALS
![Page 13: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/13.jpg)
min
(Sasarak et al, 2012)
![Page 14: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/14.jpg)
Demo 1: drawn input
![Page 15: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/15.jpg)
Demo 2: typed input
![Page 16: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/16.jpg)
Demo 3: search
![Page 17: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/17.jpg)
Study goals
1. To observe whether min changes user search behavior
2. Discover real-world scenarios for math search interfaces
![Page 18: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/18.jpg)
What do we know about...
• …the visual aspects of math?
• …inputting math on a computer?
• …math search?
![Page 19: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/19.jpg)
PREVIOUS WORKPerceiving, inputting, & searching for math.
![Page 20: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/20.jpg)
Perceiving math
• Appearance affects reasoning – cognitive illusion (Landy &
Goldstone, 2007)
• Grounded in visual structure (Landy & Goldstone, 2007)
• Designed to encourage us to think visually (Anthony, Yang &
Koedinger, 2005)
• Diagrammatic – derive meaning from layout (Landy &
Goldstone, 2007)
![Page 21: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/21.jpg)
Inputting math
• Typed input not optimal (Anthony, Yang & Koedinger, 2005)
• Handwriting most natural and satisfactory (Anthony, Yang &
Koedinger, 2005)
• Equation editors tedious (Smithies, Novins & Arvo, 2001)
![Page 22: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/22.jpg)
Math search interfaces
• Text keywords
• Expressions coded in LaTeX, TeX, MathML
• Expressions built using a template-based editor
![Page 24: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/24.jpg)
Editor input in MathFind(Munavalli & Miner, 2006)
![Page 25: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/25.jpg)
Zhao et al study
• written math expression
not useful as a search term
• doubt value of query-by-
expression capability
• prefer inputting LaTeX
• text most viable form of
searching
• specialized input
modalities unwieldy (Zhao, Kan, & Theng, 2008)
![Page 26: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/26.jpg)
min
1. To observe whether min changes user search behavior
2. Discover real-world scenarios for math search interfaces
![Page 27: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/27.jpg)
THE STUDYDesign & execution.
![Page 28: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/28.jpg)
Design considerations
• Observational – min is in prototype phase
• Peer-assist style – reduce math anxiety
• Math professor input – ensure tasks are level-appropriate
• Pilot – test and refine the protocol
![Page 29: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/29.jpg)
Participants
• 16 participants
• 18 or older
• first- or second-year college math course @ RIT
• Beginner or Intermediate level in math knowledge
• Comfortable or Very Comfortable using the internet
• Recruited via email
![Page 30: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/30.jpg)
The test session
• In the Usability Lab,
Golisano Hall @ RIT
• 1 hour duration
• 1 moderator & 1
observer
• Recorded
• $20 compensation
![Page 31: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/31.jpg)
Artifacts
• Screener Survey (online)
• Orientation Script (printout)
• Consent Form (printout)
• Background Survey (online)
• Task Sheets (printouts)
• Pre-demo Questionnaire (printout)
• Post-Study Questionnaire (online)
• Post-Study Interview Sheet (printout)
• Project Information Sheet (printout)
![Page 32: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/32.jpg)
Task Expressions
•
![Page 33: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/33.jpg)
Task Keywords
• polynomials
• Pascal‟s triangle
• binomial coefficients
• prime counting function
![Page 34: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/34.jpg)
Tasks 1 & 2
•
![Page 35: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/35.jpg)
Tasks 3 & 4
•
![Page 36: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/36.jpg)
Counterbalanced
Group Task Order
1 1 2 3 4
2 4 1 2 3
3 3 4 1 2
4 2 3 4 1
![Page 37: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/37.jpg)
Search Conditions
• Text books, notes, websites, and/or online search
• Online search only without the min interface
• min interface only
• Online search only with the option of using the min
interface
![Page 38: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/38.jpg)
Introducing min
• In between search condition 2 & 3
• Participant impression noted first
• Keyboard & mouse-drawn modalities demoed
• Upload modality described
• All tools and search function demoed
![Page 39: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/39.jpg)
min – hands-on use
• Supports diagrammatic aspects of math notation
• Affords preferred handwritten method of math input
![Page 40: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/40.jpg)
1. Does min change behavior?
Metrics
• Expression use in search query
• Query length
• # of query reformulations
• Task time
![Page 41: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/41.jpg)
2. Real-world use?
Metrics
• Input modality used
• Self-rated task success
• Satisfaction ratings
• Participant ideas
![Page 42: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/42.jpg)
What else can we observe?
![Page 43: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/43.jpg)
THE RESULTSPrimary findings.
![Page 44: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/44.jpg)
Goal 1
1. To observe whether min changes user search behavior
• Yes.
![Page 45: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/45.jpg)
Expression Use
2
0
16
10
0
12 16
0
2
4
0
2
4
6
8
10
12
14
16
18
Cond. 1: no
min, open resources
Cond. 2: no
min, search
Cond. 3: min req. Cond. 4: min (min
opt.)
Cond. 4: no min (min
opt)
n=14 n=16 n=16 n=12 n=4
To
tal
Nu
mb
er o
f In
itia
l Q
uer
ies
Without Expression
With Expression
![Page 46: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/46.jpg)
Average Time by Condition
112
209
133 132133
96
136
108
343
289
517
112
158173
405
164
79
50
0
29
0
180
360
540
720
900
prime counting function polynomial pascals triangle binomial coefficient
Sec
on
ds
Condition 1 - no min, open resources
Condition 2 - no min, online search
Condition 3 - min, min req.
Condition 4 - min canvas, min opt.
Condition 4 - no min canvas, min opt.
n=3n=3
n=3n=1
n=3n=1
n=3n=1
n=4 n=0
![Page 47: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/47.jpg)
Task Success
0
4
8
12
16
20
24
28
32
Successful Somewhat successful Unsuccessful
Without min
With min
![Page 48: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/48.jpg)
Finding a resource
Without min
• submit search query then “cherry pick” from search
results
With min
• submit search query to multiple databases
![Page 49: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/49.jpg)
Reformulating
Without min
• when reformulating a search query containing an
expression, modifications made to the expression, as well
as any text keywords
With min
• when reformulating a search query containing an
expression, modifications made only to the keywords.
![Page 50: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/50.jpg)
Goal 2
2. Discover real-world scenarios for math search interfaces
• Yes.
![Page 51: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/51.jpg)
Real-world use
• 12 out of 16 participants (75%) identified
scenarios where they would use min or could have used
min in the past.
![Page 52: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/52.jpg)
Participant Comment
“when they start to get nasty and use a lot of Greek
letters, it's hard to search a Greek letter”
![Page 53: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/53.jpg)
Participant Comment
“for more complex problems…even over wolfram
alpha, a lot easier to put problems in and can still search
wolfram”
![Page 54: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/54.jpg)
Participant Comment
“if searching for something with a radical or some weird
symbol that‟s really hard to enter that in… I like that you
can draw it knows what you're talking about and can
detect it and you can search it right there … I don't have
to Google and type in the term in place of the symbol”
![Page 55: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/55.jpg)
Participant Comment
“it would be really nice when you have a really long
equation … like when using wolfram alpha a lot of the
equations I put in there you have to put like 10
parentheses in it just to get it to work and it ends up taking
at least 10 minutes to make sure you have it right so this
would be nice to be able to actually just draw it out and
have it recognize what you draw”
![Page 56: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/56.jpg)
Participant Comment
“thought it was neat you can just write it in because it's
hard to Google or wolfram alpha equations”
![Page 57: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/57.jpg)
CONCLUSIONDiscussing the results.
![Page 58: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/58.jpg)
Increased Expression Use
• The affordance of the interface
• The novelty of the interface
• The ability for the interface to bridge the “intention gap”
![Page 59: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/59.jpg)
Intention Gap
![Page 60: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/60.jpg)
Intention Gap – bridged?
“I was so surprised when it picked up on 4 choose 2.”
“Like 4 choose 2 – that‟s really hard to „write‟ but it knew what I
meant and it accurately translated what I was trying to say to it.”
![Page 61: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/61.jpg)
122.75152.13
134.86 121.57
0
60
120
180
240
300
360
420
480
540
600
660
720
780
Sec
on
ds
263.71242.00
461.00
134.29
0
60
120
180
240
300
360
420
480
540
600
660
720
780
Sec
on
ds
Task Time by Expression
Without min With min
![Page 62: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/62.jpg)
FUTURE WORKmin development & areas to explore.
![Page 63: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/63.jpg)
min improvements
• Typed and recognized expressions are now rendered
using the online MathJax service
• Handwritten strokes are now hidden after recognition
• Now a button on toolbar brings up correction menu
• Now allows operator shorthand in text input such as „x^2‟
![Page 64: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/64.jpg)
min improvements – demo 1
![Page 65: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/65.jpg)
min improvements – demo 2
![Page 66: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/66.jpg)
Ideas for future work
• testing the improved min interface
• expressions with complex structure and notation
• experimental comparisons tests
• different populations (e.g., age, education, income)
• usage patterns over time
![Page 67: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/67.jpg)
Ideas for future work
• field use rather than lab use
• actual success rather than perceived success
• other domains that use diagrammatic terms and non-
keyboard characters
![Page 68: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/68.jpg)
AcknowledgementsFor expertise, guidance and support.
![Page 69: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/69.jpg)
Dr. Zanibbi | Primary Advisor
![Page 70: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/70.jpg)
Drs. Yacci & Rozanski | Advisors
![Page 71: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/71.jpg)
Dr. Agarwal | Task design
![Page 72: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/72.jpg)
Awelemdy Orakue | Assistant
![Page 73: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/73.jpg)
DPRL members | Technical Support
![Page 74: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/74.jpg)
AAUW | 2009 CD Grant
![Page 75: Discovering Real-World Usage for a Multimodal Math Search Interface](https://reader033.vdocument.in/reader033/viewer/2022042816/55961ef91a28ab920e8b47b1/html5/thumbnails/75.jpg)
Thank you.