Enriching Web Information Scent for Blind Users
Markel Vigo1, Barbara Leporini2, and Fabio Paternò2
1 University of the Basque Country
2 Human Interfaces in Information Systems
1 Laboratory of HCI for Special Needs
2 Italian National Research Council
Hypothesis: annotating links with the accessibility level of the page where they point would increase user orientation
“Visually impaired users need to be warned of obstacles because their reliance on cues is higher than for sighted users”- Goble et al.
”Detecting and notifying users about barriers improves user orientation”- Harper et al.
Goal: use of accessibility assessment results in web navigation scenarios
Information Scent:
Thus, we aim at enriching information scent using accessibility assessment results for screen reader users.
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
1. Introduction
1. Visual or textual cues provided on a Web site to suggest what information its links may contain.2. The perceived usefulness of a page based on such information.
Hypothesis: annotating links with the accessibility level of the page that the link point to increases user orientation
“Visually impaired users need to be warned of obstacles because their reliance on cues is higher than for sighted users”- Goble et al.
”Detecting and notifying users about barriers improves user orientation”- Harper et al.
Goal: deployment of accessibility assessment results in navigation scenarios
Information Scent:
Thus, we aim at enriching information scent using accessibility assessment results
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
1. Introduction
1. Visual or textual cues provided on a Web site to suggest what information it or its links may contain.2. The perceived usefulness of a page based on such information.
not accessible
highly accessible
fairly accessible
…calls for
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
2. Challenges
• automatic evaluation due to efficiency needs- based on guideline review tools- be aware of tool limitations- make assumptions, take risks
• user-tailored assessment- current assessment techniques address all user groups- adaptive evaluation and measurement- quantitative scores for accuracy and discrimination power
Goal: annotation of links with accessibility assessment results in navigation scenarios.
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
3. User-Tailored Assessment: Evaluation
• Web Accessibility: subset of WCAG 1.0 - following the classification by Brajnik for the BW method
- developed the Accessibility Checker for Blind users (ACB)
- ACB checks lack of tags, attributes and appropriate combination of them
Accessibility is measured in terms of conformance to web guidelines for blind users.
• Web Usability: Usability Guidelines for the Blind (UGB) - 4 principles: structure and arrangement, content appropriateness,
multimodal output, consistency - automatic guideline review tool: Magenta- Magenta checks adequate content of tags and attributes, arrangement of
headings or shortcuts
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
3. User-Tailored Assessment: Evaluation
• Resolving guideline set conflicts/overlap - Addressed- Addressed but not implemented
- Complementary
- Contradictory
Accessibility is measured in terms of conformance to web guidelines for blind users.
ACB
Magenta
www.foo.com
Accessibility report
Usabilityreport
Dependencies solver
Exclusively accessibility issues
Exclusively usability issues
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
3. User-Tailored Assessment: Measurement
• Calculate failure-rates for each test case, earl:TestCase• Using aggregation methods • Considering issue typology:- automatic issues (earl:automatic) yield earl:passed or earl:fail- recommendations- semi-automatic issues (earl:semiAuto)
ACB
Magenta
www.foo.com
Accessibility report
Usabilityreport
Dependencies solver
Exclusively accessibility issues
Exclusively usability issues
Metrics Calculation Component
accesibility score
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
3. User-Tailored Assessment: Measurement
Traditional aggregation:
€
E=W1E1 +.. +WiEi +.. +WnEn
Logic Scores Preferences:
€
E= W1E1ρ( d)
+.. +WiEiρ( d)
+.. +WnEnρ(d) ⎛
⎝ ⎜
⎞ ⎠ ⎟1ρ( d)
where W: weights and E: evaluation results
where ρ(d) are values selected upon the required logical relationship between evaluation results.
d=0 conjunction 0< d <0.5 quasiconjunction: simultaneity in satisfying all the evaluationsd=0.5 arithmetic mean0.5< d <1 quasidisjunction: penalizes only when all evaluations are not satisfied d=1 disjunction
Only intermediate values are applied
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
3. User-Tailored Assessment: Measurement• Example: a checkpoint implements 3 test cases
test cases
checkpoints
T1 T2 T3
guidelines
0.3
0.25
- strong quasiconjuction among earl:automatic
T1earl:automatic= 1
T2earl:automatic= 0.25
T3earl:semi-automatic= 0
- medium quasidisjunction among earl:semi-automatic
0.25
1 0.25 0
final score
- weak quasiconjunction is applied at checkpoint level
?
- between guidelines, mean value
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
4. User-testing• Preliminary evaluation for annotated links • Experimental settings- 16 experienced blind users. Age M=43, sd=11- JAWS on Internet Explorer- Remote usability testing
- Log analysis of interaction sequence, timing, keyboard and mouse actions
- Post-task forms and post-test questionnaire
• Two tasks- Browsing by navigating - Searching by navigating
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
4. User-testing: browsing by navigating
• Definition: casual/aimless browsing deciding at each step where to go next
• Goal: observe users with no/vague target in mind
• Two sites with 10 links- Top ten search results for “Pisa” and “Firenze” keywords
- Results were heterogeneous links wrt topic,
- Following a pattern: wikipedia, local university, soccer team and so on
- One site was manually annotated with accessibility scores and relevance scores
- Relevance based on ranking {very relevant, relevant, medium, low, irrelevant}
- 5 min free browsing. They had to write a report about what they learned
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
4. Results for browsing by navigating
• In the page without annotations,- 9 users proceeded sequentially. Kendall τ=[0.8-1.0] at most p<0.03
• In the page with annotations,- Only 2 users proceeded sequentially. Kendall τ=1.0 at most p<0.05
- None followed the sequence of most accessible links
- None followed the path based on relevance
- Some proceeded dichotomously
- However, when aggregating accessibility scores of visited pages, 7 points over the median are obtained
- This, can be interpreted as if the users browsed within the subset of more accessible pages according to random/preference criteria
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
4. User-testing: searching by navigating
• Definition: look for a target by sequentially deciding at each step where to go next
• Goal: observe users with a specific target in mind
• Two sites with 10 links- Top ten search results for “Accommodations in Pisa”
- Results were homogeneous links wrt topic
- One site was manually annotated with accessibility scores
- Relevance was not considered
- Two tasks: (1) given a telephone number (2) address
- They had to write down the name of the hotel
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
4. Results for searching by navigating
• In the page without annotations,- Only 2 users proceeded sequentially. Kendall τ=1.0 at most p<0.05
- One user proceeded inversely. Kendal τ=-1.0 at most p<0.02
• In the page with annotations,- Only 2 users proceeded sequentially. Kendall τ=1.0 at most p<0.05
- One user followed the most accessible path. Kendal τ=1.0 at most p<0.02
- Again, when aggregating accessibility scores of visited pages 6 points over the median are obtained
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
5. Results: gathered informal comments • 8 users appreciated accessibility scores in links• The suitability of scores was more controversial
…some were satisfied: - “values adequately reflect accessibility level”- “scores are useful”, “scores are interesting”- “accessibility scores seem correct”- “navigation is better if scores are included”
…while other were not: - “strange validation”- “scores are not very coherent”
…other changed their minds: - “I’m doubtful about accessibility criteria” “links with accessibility scores are
useful”- “scores don’t convey the actual difference in accessibility level” “scores make
navigation smoother and more instinctive” - “scores seem random””I missed accessibility scores in this task”
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
5. Results: post-test questionnaire
• “Scores are useful for the browsing task”
1 2 3 4 5
0
2
4
6
8
10
• “Scores are useful for the searching task”
• “Scores are correlated with actual accessibility level in browsing task”
• “Scores are correlated with actual accessibility level in searching task”
1 2 3 4 5
0
2
4
6
8
10
1 2 3 4 5
0
2
4
6
8
10
1 2 3 4 5
0
2
4
6
8
10
• 5 point Likert-scale {1: totally disagree – 5: totally agree}
Enriching Web Information Scent for Blind Users ACM ASSETS 2009
6. Conclusions: scores
• Most users state that scores are useful to a certain extent- Although there is not an agreement- “The perception of accessibility depends on each user and their
particular computer settings”
• There is no agreement on the type of scores they prefer- 50% for qualitative and quantitative
• Considering the informal comments it seems that the annotation technique prevails over scores
Enriching Web Information Scent for Blind Users
6. Conclusions: annotation technique
• In the searching scenario users do search within the subset of most accessible links • In the browsing scenario users change paradigm- From sequential browsing to random in the subset of most
accessible links- Subjective scores are more balanced than in the searching
scenario
• When directly enquired, users state that accessibility annotations would be useful in those scenarios where the topic of the linked pages would be similar
Enriching Web Information Scent for Blind Users
6. Conclusions: annotation technique • It seems that annotation technique would better fit in an scenario where:
- They are browsing casually - and topics of linked pages are similar
• For instance on the leaf nodes of a web directory
Enriching Web Information Scent for Blind Users
Markel Vigo1, Barbara Leporini2, and Fabio Paternò2
1 University of the Basque Country
2 Human Interfaces in Information Systems
1 Laboratory of HCI for Special Needs
2 Italian National Research Council
Questions?