1 + 1 = YouMeasuring the comprehensibility of metaphors

for configuring backup authentication

Stuart Schechter Robert W. ReederSymposium on Usable Privacy and Security (SOUPS),

2009

Presented by:Payas Gupta

Why backup authentication?

• No good backup authentication system till date

• Commonly used– Email based authentication (Gmail)– Personal authentication questions

(Hotmail)

Email based authentication

• Popular but of limited use

Why good backup authentication?

• Cases like - Republican vice president candidate Sarah Palin’s Yahoo! account

• 20% of users forget their answers within six months

• 17% of answers guessed by acquaintances

• 13% of answers were easily cracked by 5 most popular responses

One-size-fits-all

• One-size-fits-all approach will never work

• Some have little to protect• Some have critical information

• More tasks and if she fails then?• Less tasks and if attackers gets access

then?

Exam Metaphor

Evidence Scale Metaphor

Screenshots presented

• Windows Live ID – password reset

• SplendMail– Short exam P5 form– Short exam P10 form– Long exam– Evidence scale form

Windows Live ID password reset

Short exam P5 form

Short exam P10 form

Long exam form

Evidence scale form

Questions accompanying screenshots

• One-or-both questions for both the Live ID and P5 form.– To change her password, will [Windows

Live|SplendMail] require Jane to establish her identity using both the e-mail address and the question, or is one of the two enough?

• Answer options – one, probably one, not sure, probably both, both

–Will Jane be able to change her password after performing all of the following actions (and only those actions) to prove her identity to [Windows Live|SplendMail]?

• Answer options - yes, probably, not sure, probably not, and no.

Mechanism comprehension

• Answer options - definitely A, probably A, not sure, probably B, and definitely B.

Zero-centered five point scale

• Integers from -2 to 2 scale• For example, if the correct answer to a

question was yes, a participant would receive – 2 points yes– 1 for probably yes– 0 for not sure– -1 for probably not– -2 for no.

Participant Demographics

• Total 18– Age between 30 and 48– 7 female 11 male– 12 have Live Id accounts

Hypothesis 1

• When presented short exam P5, which describes how each authentication mechanism will be used, Live ID users are better able to comprehend the use of these mechanisms than when presented with Live ID's password-reset settings form.

Hypothesis 2

• Live ID users comprehend the evidentiary requirements of authentication in the short exam form as well as they do for Live ID's current password reset settings form.

2b

2c

2d

Hypothesis 3

• Comprehension of the exam metaphor decreases as more authentication mechanisms are configured.

Hypothesis 4

• The evidence scale form, which does not require mental math, is more comprehensible than the exam form, which does.

Hypothesis 5

• Users prefer the exam form to the evidence scale form, or vice versa.

Result: Hypothesis 1

• When presented short exam P5, which describes how each authentication mechanism will be used, Live ID users are better able to comprehend the use of these mechanisms than when presented with Live ID's password-reset settings form.Not Significant

Result: Hypothesis 2

• Live ID users comprehend the evidentiary requirements of authentication in the short exam form as well as they do for Live ID's current password reset settings form.

Strongly significant

Somewhat Significant

• The difference in the mean scores for the hardest questions about the exam and the mean scores and the three simple questions about the Live ID screenshot is statistically significant -- in favour of the exam.

Result: Hypothesis 3

• Comprehension of the exam metaphor decreases as more authentication mechanisms are configured.

Result: Hypothesis 3

• 18 participants– only 5 of 54 total responses on the longer

exam were imperfect (either incorrect or not provided with full confidence).

• Not only was there no evidence of a decrease in comprehension, but the learning effect might well have increased comprehension.

Not Significant

Result: Hypothesis 4

• The evidence scale form, which does not require mental math, is more comprehensible than the exam form, which does.

Reject This: Exam form more

comprehensible

Result: Hypothesis 5

• Users prefer the exam form to the evidence scale form, or vice versa.– Exam was better

Limitation/Future Work

• How points would be assigned to authentication tasks in the exam metaphor?

• The effect of user demographics?

Conclusion

• Looked into--If users cannot understand whether one or both of two tasks is required to authenticate, how could they be expected to understand which of 5 tasks would be sufficient

• User authentication is, after all, a complex process.

• 15 of our 18 participants answered all eight questions about the exam metaphor perfectly.

Conclusion cont…

• Only one of the 18 participants missed more than one question.