confused johnny when automatic encryption leads to confusion and mistakes scott ruoti, nathan kim,...
TRANSCRIPT
![Page 1: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/1.jpg)
Confused JohnnyWHEN AUTOMATIC ENCRYPTION
LEADS TO CONFUSION AND MISTAKESScott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons
Internet Security Research Lab
Computer Science Department
Brigham Young University
![Page 2: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/2.jpg)
Confused Johnnyo E-mail encryption for the masses
o We developed a system maximizing usability◦ Made everything transparent
o Johnny became confused
o Designed another system with manual encryption◦ This helped Johnny gain clarity
![Page 3: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/3.jpg)
Encrypted E-mailo Exists, but largely goes unused
o S/MIME, PGP◦ Tools available
o “Why Johnny can't encrypt: A usability evaluation of PGP 5.0”◦ Whitten and Tygar, 8th USENIX Security Symposium (1999)◦ Later research confirmed findings
o What can be done?
![Page 4: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/4.jpg)
Usability Issueso Users resist change
◦ Users are using webmail◦ If security is difficult users will forgo it
o Key management is confusing◦ Hierarchical, web-of-trust◦ Recipient must already have key◦ Chicken and egg problem
o Cryptography is complicated◦ Unclear which properties are provided◦ Unclear which properties are needed
![Page 5: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/5.jpg)
Private Webmail (Pwm)o Pronounced “Poem”
o Adds end-to-end encryption to existing webmail systems◦ Gmail, Hotmail, Yahoo! Mail◦ Runs on all modern browsers
o Designed to maximize usability
o Provide good-enough security◦ Improvement for those already sending sensitive e-mail
![Page 6: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/6.jpg)
Users Resist Changeo Security overlays
◦ Integrates tightly with existing webmail systems◦ Users do not need to learn yet-another-system
o Tightly integrates with existing systems◦ Replaces small portions of the interface◦ Displayed using iFrames
o Functionally transparent◦ Low barrier to adoption
o Visually distinctive◦ Easy to identify
![Page 7: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/7.jpg)
Usability Fixeso Users resist change
◦ Focus on bootstrapping first-time users◦ Helpful instructions in e-mail◦ Bookmarklet-based installation
o Key management is confusing◦ Key escrow based on IBE◦ Simple Authentication for the Web (EBIA)◦ No user interaction required
o Cryptography is complicated◦ Encryption is automatically handled by Pwm◦ Users never interact with ciphertext
![Page 8: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/8.jpg)
Pwm: Walkthrough
![Page 9: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/9.jpg)
Pwm: Walkthrough
![Page 10: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/10.jpg)
Pwm: Walkthrough
![Page 11: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/11.jpg)
Pwm: Walkthrough
![Page 12: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/12.jpg)
Pwm: Walkthrough
![Page 13: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/13.jpg)
Pwm User Studieso Two studies
o First study measured usability of Pwm◦ Also evaluated bookmarklets for use during installation
o Second study compared Pwm to Voltage Secure Mail Cloud◦ Voltage Secure Mail Cloud is an existing depot-based secure email system◦ Pwm was run using a browser extension
o Evaluation◦ Pre- and post-survey questionnaire◦ Monitored participants actions for unrecognized mistakes◦ Post-survey interviews
![Page 14: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/14.jpg)
System Usability Scaleo Brook (1996)
o Ten questions◦ Alternate negative and positive◦ Give a single number for usability
o Bangor compared scores for hundreds of systems
![Page 15: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/15.jpg)
76
71
63
![Page 16: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/16.jpg)
SUS Score Comparison
![Page 17: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/17.jpg)
Success?o Results are very promising
◦ Very positive reception◦ Users indicated they wanted to begin using it
o Not without problems
o Small number sent e-mail without encryption
o Participants were confused about security◦ Wanted to see more details◦ Unsure of who could read e-mails
![Page 18: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/18.jpg)
Where to go from here?o Simple solutions was to fix UI issues
o One author (Nathan Kim) had a different idea◦ Manual encryption◦ Decoupled interface
o Mocked up these ideas◦ Message Protector (MP)◦ Simple Interface◦ Direct handling of ciphertext◦ Implied key management
![Page 19: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/19.jpg)
MP: Walkthrough
![Page 20: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/20.jpg)
MP: Walkthrough
![Page 21: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/21.jpg)
MP: Walkthrough
![Page 22: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/22.jpg)
MP: Walkthrough
![Page 23: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/23.jpg)
First MP User Studyo Evaluated MP using SUS
o Compared against Encipher.it◦ Bookmarklet-based encryption system◦ Works in Gmail and Facebook
o Evaluation◦ Pre- and post-survey questionnaire◦ Monitored participants actions for unrecognized mistakes◦ Post-survey interviews◦ The system usability scale
o Evaluated comprehension◦ Survey included questions about comprehension◦ How to use the system◦ Who could read messages
![Page 24: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/24.jpg)
61
72
![Page 25: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/25.jpg)
Second MP User Studyo Surprising usability results
◦ Participants had a positive reaction to seeing ciphertext◦ Similar SUS score to MP
o Ran a second study comparing MP to Pwm◦ Modeled after the first MP study
![Page 26: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/26.jpg)
76
74
![Page 27: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/27.jpg)
SUS Score Comparison
![Page 28: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/28.jpg)
Other resultso MP improved users comprehension
◦ Clearly understood how to use system◦ Clearly understood who could read messages
o Usability scores nearly identical to Pwm
o Participants preferred manual encryption of MP
o Participants preferred tight integration of Pwm
![Page 29: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/29.jpg)
Study limitationso MP studies ignore bootstrapping new users
◦ Studies assumed software pre-installed◦ Bootstrapping is a key component of Pwm’s design◦ Not fully representative of overall usability
o Short-term studies
o SUS question unclear◦ “I think that I would like to use this system frequently.”◦ Participants ranked low even when enthusiastic about the system◦ Relevant to security studies
![Page 30: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/30.jpg)
Related Workso “Johnny 2: a user test of key continuity management with s/mime and outlook express.”
◦ Garfinkel and Miller, SOUPS 2005◦ Applied automatic key management to e-mail◦ Allowed great success
o “Helping Johnny 2.0 to encrypt his Facebook conversations.”◦ Fahl, et al., SOUPS 2012◦ Encrypted Facebook communication◦ Explored manual vs. automatic encryption◦ Invisibility security not trusted by users
![Page 31: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/31.jpg)
Conclusiono Pwm was a success
◦ Participants largely succeeded at using encrypted e-mail◦ Participants had high praise for Pwm◦ Succeeding in being easy for new users
o Pwm wasn’t perfect◦ Security was too transparent◦ Caused users to be confused and make mistakes
o Mocked up a system using manual encryption◦ Users enjoyed manual encryption◦ Wished it was tightly integrated with the browser
o A combination of approaches is needed to solve the problem
![Page 32: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/32.jpg)
Conclusion: Future Worko Manual encryption in Pwm
◦ Don’t automatically send encrypted email◦ “Encrypt” button which puts ciphertext in compose window
o Sidebar◦ Browser sidebar allowing for manual encryption◦ Can be used on any site◦ Fallback for when Pwm has an error
o Long-term studies◦ Larger populations◦ Real tasks
![Page 33: Confused Johnny WHEN AUTOMATIC ENCRYPTION LEADS TO CONFUSION AND MISTAKES Scott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent Seamons Internet](https://reader035.vdocument.in/reader035/viewer/2022062314/56649f0d5503460f94c2116f/html5/thumbnails/33.jpg)
Questions?