Confused Johnny: When Automatic Encryption Leads to Confusion and MistakesCALEB STEPANIAN, CINDY ROGERS, NILESH PATEL

Background

Problem

Solution

Results

Conclusion

Outline1. Background Information

a) Who is Johnny?b) What is usability?

2. What is Johnny's problem?

3. How can we fix it?

4. Results

5. Conclusions

Background

Problem

Solution

Results

Conclusion

Who’s Johnny?

Background

Problem

Solution

Results

Conclusion

What is usability? Security software is usable if the people using it:

o Know the security tasks they need to performo Are able to figure out how to perform themo Don’t make dangerous errorso Are comfortable enough to continue use

Background

Problem

Solution

Results

Conclusion

Problem StatementJohnny finds it confusing to encrypt his emails.

Background

Problem

Solution

Results

Conclusion

Hypothesis

Johnny doesn’t encrypt because current solutions are not transparent enough.

Background

Problem

Solution

Results

Conclusion

Transparency To be considered transparent: 1. Cannot require too much effort 2. Must solve chicken-and-egg problem for keys 3. Handle encryption automatically, hiding cipher text

Background

Problem

Solution

Results

Conclusion

Experiment

Have Johnny try transparent encryption and opaque encryption methods to determine his preferences.

Background

Problem

Solution

Results

Conclusion

Experimental Methodology1. Find a transparent system that meets criteria2. Find other more opaque solutions3. Run one user study for each other solution

comparing it to the transparent one4. Find System Usability Scale (SUS) score for each5. Draw conclusions

Background

Problem

Solution

Results

Conclusion

System Usability ScaleA set of ten questions that allows one to access the usability of a system on a sliding scale.

Background

Problem

Solution

Results

Conclusion

Experimental SetupTransparent: Pwm (Private Webmail) Browser extension that overlays automatic encryption over web mail

Opaque: MP (Message Protector) Manual encryption with external program

Background

Problem

Solution

Results

Conclusion

Other Methods TestedDepot Base: Voltage(Voltage SecureMail Cloud) Sign up for an account and verify it.

Generic: Encrypt.it (Bookmarklet) Allows you to encrypt the text in any field with a password.

Background

Problem

Solution

Results

Conclusion

Pwm Example

Background

Problem

Solution

Results

Conclusion

MP Example

Background

Problem

Solution

Results

Conclusion

Results Because people did not see the encryption happening, 10% of users didn’t encrypt their emails correctly & some users didn’t innately trust the system. Manual encryption (copy pasting while seeing the ciphertext) and clear separation gave users more confidence in the system.

Background

Problem

Solution

Results

Conclusion

Comparison Results: PWM v. MP28 users tried both MP and Pwm

*Correctly identifying who would be able to read encrypted messages

Metric Percent users Pwm Study

Percent users MP study

Successful Decryption 86% 93%

Successful Encryption 83% 97%

Comprehension* 76% 93%

Intuitively decrypt 72% 100%

Preferred System 41% 41%

Background

Problem

Solution

Results

Conclusion

Study ResultsPwm Usability Study PWM v. Voltage Preference

Metric Successful Users out of 25

Setup Pwm 24

Successful Decryption

24

Reply with Encrypted Message

23

Send Encrypted Message Direct

22

•44% users reported Voltage was cumbersome to encrypt and decrypt a message•19% preferred Voltage

Background

Problem

Solution

Results

Conclusion

MP vs. Encipher.it Task 1: Install the given system

Task 2: Open Gmail and send encrypted message, decrypt response

Task 3: Open Facebook and send encrypted message, then decrypt reply

System Task 2 Task 3

MP 89% 96%

Encipher.it 57% / 50% 82% / 61%

Background

Problem

Solution

Results

Conclusion

Conclusions of MP vs. Encipher.itMP had a SUS score of 72.23Encipher.it had a SUS score of 61.25MP qualifies as “acceptable”Encipher.it ranks as “low marginal”

Background

Problem

Solution

Results

Conclusion

Conclusion Encryption needs to be somewhat manual so that users feel secure and know the difference between encryption and plaintext

Background

Problem

Solution

Results

Conclusion

Limitations1. User studies were short term lab studies2. First SUS question was “I think that I would like to use

this system frequently”.3. First MP study assumed secrets were already shared

Second MP study assumed Pwm was installed

Thank you!

Any Questions?

PGP (Pretty Good Privacy) public and private keypairs

private key needed to sign and decrypt

public key needed to encrypt and verify signature

A user needs to generate a keypair and share their public key before an encrypted message can be sent to them

Key escrow server Trusted third party that generates and stores key material for users

Has ability to read all messages and masquerade as any user

Background

Problem

Solution

Results

Conclusion

Example SUS Survey Choose from 1 (strongly disagree) to 5 (strongly agree).

1. I think that I would like to use this system frequently

2. I found the system unnecessarily complex

3. I thought the system was easy to use

4. I think that I would need the support of a technical person to be able to use this system

5. I found the various functions in this system were well integrated

6. I thought there was too much inconsistency in this system

7. I found the system very cumbersome to use

8. I would imagine that most people would learn to use this system very quickly

9. I felt very confident using the system

10. I needed to learn a lot of things before I could get going with this system