guideline aggregation: web accessibility
DESCRIPTION
Web site evaluation methodologies and validation engines take theview that all accessibility guidelines must be met to gaincompliance. Problems exist in this regard as contradictions withinthe rule set may arise, and the type of impairment or its severityis not isolated. The Barrier Walkthrough (BW) method goes someway toaddressing these issues by enabling barrier types derived fromguidelines to be applied to different user categories such as motoror hearing impairment, etc. In this paper, we use set theory tocreate a validation scheme for older users by combining barriertypes specific to motor impaired and low vision users,thereby creating a new ``older users'' category from the results ofthis set addition. To evaluate this approach, we have conducted a BWstudy with four pages, 19 expert and 48 non-expert judges. Thisstudy shows that the BW generates reliable data for the proposedaggregated user category and shows how experts and non-expertsevaluate pages differently. The study also highlights a limitationof the BW by showing that a better aggregated user category wouldhave been created by having a severity level of disability fordifferent impairment types. By extending the BW with theseimpairment levels, we argue that the BW would become more useful forvalidating Web pages when dealing with users which multipledisabilities and thus we would be able to create a ``PersonalisedValidation and Repair'' method.TRANSCRIPT
![Page 1: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/1.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Guideline Aggregation: Web AccessibilityEvaluation for Older Users
Giorgio Brajnik (1), Yeliz Yesilada (2), Simon Harper (2)
(1) Dip. di Matematica e InformaticaUniversity of Udine, Italy
www.dimi.uniud.it/giorgio(2)School of Computer Science
University of ManchesterManchester, UK
W4A 2009
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 2: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/2.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
The problem with analytic evaluation methods
I conformance reviews (eg. wrt WCAG20) arenon-contextualized, not specific
I evaluators are not guided into assessing consequences ofviolations
I there’s no reliable way to rate severity of violations
Our approach
1. Provide context to evaluators: focus on specific barriersand user categories (eg. blind, motor impaired, cognitivelyimpaired, low vision, ...)
2. Provide more formalized ways to rate severity
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 3: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/3.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Multiple impairments
How to cope with multiple impairments and combinatorialexplosion?
I eg. older people
I Dynamic Aggregation:
1. do the evaluation for primitive categories2. and then aggregate3. eg. barriers for older people = barriers for low vision∪ those for motor impaired ∪ ...
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 4: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/4.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Barrier Walkthrough
1. Analytic method; similar to "heuristic walkthrough"2. Based on barriers (ako "vulnerability points")3. Failure modes are contextualized within usage scenarios4. This helps evaluators in rating severity = F(impact,
persistence) in {1,2,3}5. See http://www.dimi.uniud.it/giorgio/
projects/bw/bw.html
(Brajnik, ICCHP 2006; ASSETS 2007)
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 5: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/5.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Example of a barrier
Rich images lacking equivalent text
I Users: Blind persons using a screen readerI Cause: The page contains some image that provides
information (e.g. a diagram, histogram, picture, drawing,graph) but only in a graphical format; no equivalent textualdescription appears in the page.
I Failure mode: The user, even if s/he perceives that thereis an important image, has no way to get the information itcontains. In addition s/he spends time and effort trying tofind out where in the page or site that information is buried.
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 6: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/6.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Experiment
GoalTo explore which conclusions are invariant wrt aggregation.
I Do certain differences among sites disappear?I How does reliability change?I How does correctness of evaluations change?I How does the difference b/w expert/non-expert change?
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 7: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/7.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Plan
Mixed design experiment
I 19 experts + 51 non-experts applying BW; 61 barrier types(within-subj)
I 2 primitive user categories: low vision, motor impaired(within-subj)
I 1 aggregated category: older adults = union of individualbarriers found for primitive categories
I 4 pages (1 page/subject, between-subj): IMDB.com,Facebook.com, novascotiaquilts.com, Sam’s Chop House
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 8: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/8.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Spreadsheet
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 9: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/9.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
True Barriers Types
Correct ratingsthose where the majority of experts agreed on their severity
Results:I Experts: 27 out of 61 barrier types ("ambiguous links",
"functional images w/o text", "inflexible layout", "missinginternal links", ...)
I Non-experts: 24 out of those 27 (missed: "forms w/olabels", "moving content", "no css support")
I Certain barriers are specific for specific user categories
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 10: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/10.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Reliability
Reproducibilitygiven (barrier type, user group, page)rep = 1− sd
M if positive; 1 if M = 0; 0 otherwisewhere M, sd are mean/std.dev of weighted severity
Agreementgiven (user group, page)on all barrier types compute the ICC (Intraclass CorrelationCoefficient – relative and absolute consistency)
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 11: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/11.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Reproducibility
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 12: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/12.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Reproducibility
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 13: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/13.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Mean weighted severities
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 14: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/14.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Correctness
I Error rate E = IC+I
I Accuracy = % ofreported barriers thatare correct
I Sensitivity = % ofcorrect barriers that arereported
I F.measure = 2A·SA+S
Ratings:
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 15: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/15.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Error rates
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 16: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/16.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
F-measure
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 17: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/17.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Invariant properties
1. Aggregation does not worsen the problem of missedbarriers
2. Reliability: experts are consistently more reliable; samepattern across pages
3. Severities: experts are more judgmental; ranks of pagesdo not change
4. Quality: error rates maintain a similar difference (expert vsnon-experts)
5. Quality: F-measure conf. intervals shrink; they keep samerelationship
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 18: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/18.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Conclusions
1. Aggregation seems to work: it enables contextualizedevaluations and leads to results that are potentially valid
2. It could be extended to cope with degrees of impairment
Limitations
1. We did not validate our conclusions against anindependent assessment
2. We don’t know if the same conclusions would hold for anyset of primitive user categories
Questions?
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users
![Page 19: Guideline Aggregation: Web Accessibility](https://reader036.vdocument.in/reader036/viewer/2022081809/548942bdb47959140d8b5939/html5/thumbnails/19.jpg)
The problem Barrier walkthrough Experimental plan Results Discussion and conclusions
Evaluation framework
I based on reliability (reproducibility + agreement),correctness (error rate, accuracy, sensitivity andF-measure)
I is viableI is discriminatory
It can be used to assess pros and cons of an evaluationmethod.
c© Brajnik, Yesilada, Harper Guideline Aggregation: Web Accessibility Evaluation for Older Users