Why Johnny Can’t EncryptWhy Johnny Can’t Encrypt

A Usability Evaluation of A Usability Evaluation of GPG 5.0GPG 5.0

Presented by Yin ShiPresented by Yin Shi

OverviewOverview

IntroductionIntroduction Understanding the ProblemUnderstanding the Problem Cognitive WalkthroughCognitive Walkthrough User TestUser Test ConclusionConclusion

IntroductionIntroduction Effective security requires a different usability Effective security requires a different usability

standardstandard Security mechanisms are effective only when Security mechanisms are effective only when

used correctlyused correctly– Matt Bishop claimed that configuration errors are Matt Bishop claimed that configuration errors are

the cause of more than 90% of all computer security the cause of more than 90% of all computer security failuresfailures

Making security usable will require the Making security usable will require the development of domain-specific user interface development of domain-specific user interface design principles and techniquesdesign principles and techniques

Choose PGP 5.0 for our case studyChoose PGP 5.0 for our case study– Designed by general consumer software standardsDesigned by general consumer software standards– ““Significantly improved graphical user interface Significantly improved graphical user interface

makes complex mathematical cryptograph makes complex mathematical cryptograph accessible for novice computer users.”accessible for novice computer users.”

Understanding the ProblemUnderstanding the Problem

Defining Usability for SecurityDefining Usability for Security– Definition: Security software is usable if the Definition: Security software is usable if the

people who are expected to use it:people who are expected to use it:Are reliably made aware of the security tasks Are reliably made aware of the security tasks

they need to performthey need to performAre able to figure out how to successfully Are able to figure out how to successfully

perform those tasksperform those tasksDon’t make dangerous errorsDon’t make dangerous errorsAre sufficiently comfortable with the interface to Are sufficiently comfortable with the interface to

continue using itcontinue using it

Understanding the ProblemUnderstanding the Problem Problematic Properties of SecurityProblematic Properties of Security

– Five inherent properties of securityFive inherent properties of securityThe unmotivated user propertyThe unmotivated user propertyThe abstraction propertyThe abstraction propertyThe lack of feedback propertyThe lack of feedback propertyThe barn door propertyThe barn door propertyThe weakest link PropertyThe weakest link Property

A Usability Standard for PGPA Usability Standard for PGP– Need for privacy and authenticationNeed for privacy and authentication– What needs to be doneWhat needs to be done– How to do it and avoid dangerous errorsHow to do it and avoid dangerous errors

Evaluation MethodsEvaluation Methods Two MethodsTwo Methods

– An informal cognitive walkthroughAn informal cognitive walkthrough– A user test performed in a laboratoryA user test performed in a laboratory

Cognitive WalkthroughCognitive Walkthrough Visual Metaphors (keys)Visual Metaphors (keys)

– PGP’s user interface relies on graphical depictions of PGP’s user interface relies on graphical depictions of keys and lockskeys and locks

– ImprovementsImprovements An extension of the metaphor to distinguish public An extension of the metaphor to distinguish public

keys for encryption and private keys for decryption keys for encryption and private keys for decryption Different icons for public and private keysDifferent icons for public and private keys

Cognitive WalkthroughCognitive Walkthrough Visual Metaphors (signatures)Visual Metaphors (signatures)

– The icon of the blue quill pen is used to The icon of the blue quill pen is used to indicate signing is problematicindicate signing is problematic

– Quill pen icon will not help user understand Quill pen icon will not help user understand they need to use their private keys to they need to use their private keys to generate signaturesgenerate signatures

– ImprovementsImprovementsKeep quill pen to represent signing, but modify it Keep quill pen to represent signing, but modify it

to show a private key as the nib of the pento show a private key as the nib of the penUse some entirely different icon for signaturesUse some entirely different icon for signatures

Cognitive WalkthroughCognitive Walkthrough Different Key TypesDifferent Key Types

– Originally, PGP used the RSA algorithm for Originally, PGP used the RSA algorithm for encryption and signingencryption and signing

– PGP 5.0 uses the Diffie-Hellman/DSS PGP 5.0 uses the Diffie-Hellman/DSS algorithmalgorithm

– PGP 5.0 can handle RSA keys, but other PGP 5.0 can handle RSA keys, but other version PGP can’t handle DSS keysversion PGP can’t handle DSS keys

– Lack of forward compatibilityLack of forward compatibilityRecipients with RSA keys can’t decrypt itRecipients with RSA keys can’t decrypt itRecipients with RSA keys can’t verify signaturesRecipients with RSA keys can’t verify signatures

– PGP 5.0 alerts its users to this compatibility PGP 5.0 alerts its users to this compatibility issues in two waysissues in two ways

Cognitive WalkthroughCognitive Walkthrough Different Key TypesDifferent Key Types

– Uses different icons to depict the different key typesUses different icons to depict the different key types– When user attempt to encrypt documents using mixed When user attempt to encrypt documents using mixed

key types, a warning message is showedkey types, a warning message is showed– ImprovementImprovement

Double-clicking on a key pops up a Key properties windowDouble-clicking on a key pops up a Key properties window

Cognitive WalkthroughCognitive Walkthrough Metaphor of choosing peopleMetaphor of choosing people Human icons obscure the key type informationHuman icons obscure the key type information Better to display multiple keys that person ownsBetter to display multiple keys that person owns

Cognitive WalkthroughCognitive Walkthrough Key Server Key Server

– Are publicly accessible databasesAre publicly accessible databases– PGP offers three key server operations under the Keys PGP offers three key server operations under the Keys

pull-down menupull-down menu

Cognitive WalkthroughCognitive Walkthrough Problems with the presentation of the Problems with the presentation of the

Key ServerKey Server– Users may not realize that it exists Users may not realize that it exists

No representation of it in the top level of No representation of it in the top level of PGPkeys displayPGPkeys display

– PGPkeys keeps no records of key server PGPkeys keeps no records of key server accessaccess

– PGP’s key revocation operation does not PGP’s key revocation operation does not send the resulting revocation certificate to send the resulting revocation certificate to the key serverthe key server

Cognitive WalkthroughCognitive Walkthrough Key Management PolicyKey Management Policy

– Two ratings for each public keyTwo ratings for each public keyValidity – how sure the user is that the key is Validity – how sure the user is that the key is

safe to encrypt withsafe to encrypt withTrust – how much faith the user has in the keyTrust – how much faith the user has in the key

– May not realize PGP can automatically sets May not realize PGP can automatically sets the validity rating of a key based on the validity rating of a key based on whether it has been signed by a certain whether it has been signed by a certain number of sufficiently trusted keys.number of sufficiently trusted keys.

Cognitive WalkthroughCognitive Walkthrough Irreversible ActionsIrreversible Actions

– Accidentally deleting the private keyAccidentally deleting the private key– Accidentally publicizing a keyAccidentally publicizing a key– Accidentally revoking a keyAccidentally revoking a key– Forgetting the passphraseForgetting the passphrase– Failing to back up the key ringsFailing to back up the key rings

ConsistencyConsistency– encodingencoding

Too Much InformationToo Much Information– PGPkeys application presents the user with too PGPkeys application presents the user with too

much information to make sense ofmuch information to make sense of Owner’s name, validity, trust level, creation date, and Owner’s name, validity, trust level, creation date, and

sizesize Nothing to help the user figure out which parts of the Nothing to help the user figure out which parts of the

display are the most important to pay attention todisplay are the most important to pay attention to

User TestUser Test Test DesignTest Design Initial task is to send the secret message Initial task is to send the secret message

to the team members in a signed and to the team members in a signed and encrypted emailencrypted email

Main stepsMain steps– Generate a key pair, get the public keysGenerate a key pair, get the public keys– Make their own public key available to team Make their own public key available to team

membersmembers– Type the secret message into an emailsType the secret message into an emails– Sign the email using private key, encrypt the Sign the email using private key, encrypt the

email using the team member’s public keysemail using the team member’s public keys

User TestUser Test One of the member had an RSA keyOne of the member had an RSA key Participant would encounter mixed key Participant would encounter mixed key

types warning messagetypes warning message Each of the five campaign members was Each of the five campaign members was

represented by a dummy email account represented by a dummy email account and a key pair:and a key pair:– These were accessible to the test monitor These were accessible to the test monitor

through a network laptopthrough a network laptop The test monitor could send email to the The test monitor could send email to the

participant from the appropriate dummy participant from the appropriate dummy accountaccount

User Test (Results)User Test (Results) Avoiding dangerous errorsAvoiding dangerous errors

– Three of them accidentally emailed the secret Three of them accidentally emailed the secret without encryptionwithout encryption

– One forgot her passphraseOne forgot her passphrase Figuring out how to encrypt with any keyFiguring out how to encrypt with any key

– One couldn’t figure out how to encrypt at allOne couldn’t figure out how to encrypt at all– A reconfiguration of PGP may requiredA reconfiguration of PGP may required– Another one kept sending unencrypted test Another one kept sending unencrypted test

messages, and finally succeeded after being messages, and finally succeeded after being prompted to use the PGP plug in buttonsprompted to use the PGP plug in buttons

User Test (Results)User Test (Results) Figuring out the correct key to encrypt Figuring out the correct key to encrypt

withwith– 11 participants figured out how to encrypt, 11 participants figured out how to encrypt,

but failed to understand the public key modelbut failed to understand the public key model– Another one so completely misunderstood Another one so completely misunderstood

the model that he generated key pairs for the model that he generated key pairs for each team member rather than for himselfeach team member rather than for himself

Decrypting an email messageDecrypting an email message– Five participants received encrypted emailFive participants received encrypted email– One can’t figure how to decrypt itOne can’t figure how to decrypt it– Two took a very hard time to figure it outTwo took a very hard time to figure it out– Other two were able to decrypt without any Other two were able to decrypt without any

problemproblem

User Test (Results)User Test (Results) Publishing the public keyPublishing the public key

– Ten could make their public key available to Ten could make their public key available to the team membersthe team members

– Two never addressed key distributionTwo never addressed key distribution– Those ten, five sent their keys to key serverThose ten, five sent their keys to key server– Three emailed to the team membersThree emailed to the team members– Other two did bothOther two did both

Getting other people’s public keysGetting other people’s public keys– Eight successfully got the team members’ Eight successfully got the team members’

public keyspublic keys– The others either never seemed aware they The others either never seemed aware they

need other people’s public key, or they did need other people’s public key, or they did know how to get itknow how to get it

User Test (Results)User Test (Results) Handing the mixed key types problemHanding the mixed key types problem

– Only four managed to send encrypted email correctlyOnly four managed to send encrypted email correctly– One didn’t have mixed key types problemOne didn’t have mixed key types problem– The other three received a reply email for The other three received a reply email for

complaining that they couldn’t decrypt emailcomplaining that they couldn’t decrypt email Signing an email messageSigning an email message Verifying a signature on an email messageVerifying a signature on an email message Creating a backup revocation certificateCreating a backup revocation certificate

– Only three participants managed to successfully Only three participants managed to successfully send encrypted email and decrypt a replysend encrypted email and decrypt a reply

– In response to direct prompting for backupIn response to direct prompting for backup One didn’t send the key pair to the key serverOne didn’t send the key pair to the key server One sent email to the campaign managerOne sent email to the campaign manager One simply ignored the promptOne simply ignored the prompt

User Test (Results)User Test (Results) Deciding whether to trust keys from the Deciding whether to trust keys from the

key serverkey server– Of the eight participants, only three Of the eight participants, only three

expressed some concern over if they should expressed some concern over if they should trust the keystrust the keys

– None of the three made use of the validity None of the three made use of the validity and trust labeling provided by PGPKeysand trust labeling provided by PGPKeys

Conclusion/QuestionsConclusion/Questions

PGP 5.0’s user interface does not come even PGP 5.0’s user interface does not come even reasonably close to achieving our usability reasonably close to achieving our usability standardstandard

It does not make public key encryption of It does not make public key encryption of electronic mail manageable for average electronic mail manageable for average computer userscomputer users

Public work on usability evaluation in a Public work on usability evaluation in a security context would be extremely valuablesecurity context would be extremely valuable

We expect to find better design strategiesWe expect to find better design strategies