c mu u sable p rivacy and s ecurity laboratory user interfaces and algorithms for fighting phishing...
TRANSCRIPT
![Page 1: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/1.jpg)
CMU Usable Privacy and Security Laboratoryhttp://cups.cs.cmu.edu/
User Interfaces and Algorithms for Fighting Phishing
Steve ShengDoctoral Candidate, Carnegie Mellon UniversityPresented at IIS seminar, 1/30/2008
![Page 2: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/2.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Everyday Privacy and Security Everyday Privacy and Security ProblemProblem
2
![Page 3: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/3.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
This entire processknown as phishing
3
![Page 4: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/4.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Still a growing problemStill a growing problem
Estimated 1 in 122 emails are phishing
Average 31,000 unique phishing sites reported each month in 2007
Estimated 3.5 million people have fallen for phishing in 2006
Estimated $ 350m – $ 2b direct loss a year
More profitable to phish than rob the bank!
4
![Page 5: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/5.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Project: Supporting Trust Project: Supporting Trust DecisionsDecisions Goal: help people make better online trust
decisions• Currently focusing on anti-phishing
Large multi-disciplinary team project at CMU• Computer science, human-computer interaction,
public policy, social and decision sciences, CERT
![Page 6: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/6.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Our Multi-Pronged ApproachOur Multi-Pronged Approach
Human side• Interviews to understand decision-making
• PhishGuru embedded training
• Anti-Phishing Phil game
• Understanding effectiveness of browser warnings
Computer side• PILFER email anti-phishing filter
• CANTINA web anti-phishing algorithm
Automate where possible, support where necessary
![Page 7: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/7.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Our Multi-Pronged ApproachOur Multi-Pronged Approach
Human side• Interviews to understand decision-making
• PhishGuru embedded training
• Anti-Phishing Phil game
• Understanding effectiveness of browser warnings
Computer side• PILFER email anti-phishing filter
• CANTINA web anti-phishing algorithm
What do users know about phishing?
![Page 8: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/8.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Interview StudyInterview Study
Interviewed 40 Internet users (35 non-experts) “Mental models” interviews included email
role play and open ended questions Brief overview of results (see paper for details)
J. Downs, M. Holbrook, and L. Cranor. Decision Strategies and Susceptibility to Phishing. In Proceedings of the 2006 Symposium On Usable Privacy and Security, 12-14 July 2006, Pittsburgh, PA.
![Page 9: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/9.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Little knowledge of phishingLittle knowledge of phishing
Only about half knew the meaning of term “phishing”
55% say that they had never noticed an unexpected or strange-looking URL
55% reported being cautious when asked for sensitive financial information• But very few reported being suspicious of email
asking for passwords
Knowledge of financial phish reduced likelihood of falling for these scams• But did not transfer to other scams, such as an
amazon.com password phish
9
![Page 10: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/10.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Naive Evaluation StrategiesNaive Evaluation Strategies
The most frequent strategies don’t help much in identifying phish• This email appears to be for me
• It’s normal to hear from companies you do business with
• Reputable companies will send emails
“I will probably give them the information that they asked for. And I would assume that I had already given them that information at some point so I will feel comfortable giving it to them again.”
![Page 11: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/11.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Summary of FindingsSummary of Findings
People generally not good at identifying scams they haven’t specifically seen before
People don’t use good strategies to protect themselves
Large-scale survey across multiple cities in the US confirm finding
Downs, J. S., Holbrook, M. B., and Cranor, L. F. Behavioral Response to Phishing. In eCrime ’07: Proceedings of the 2007 e-Crime Researchers summit
![Page 12: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/12.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
OutlineOutline
Human side• Interviews to understand decision-making
• PhishGuru embedded training
• Anti-Phishing Phil game
• Understanding effectiveness of browser warnings
Computer side• PILFER email anti-phishing filter
• CANTINA web anti-phishing algorithm
Can we train people not to fall for phish?
![Page 13: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/13.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Web Site Training StudyWeb Site Training Study
Laboratory study of 28 non-expert computer users
Asked participants to evaluate 20 web sites• Control group evaluated 10 web sites, took 15 min break to
read email or play solitaire, evaluated 10 more web sites
• Experimental group same as above, but spent 15 min break reading web-based training materials
Experimental group performed significantly better identifying phish after training• Less reliance on “professional-looking” designs
• Looking at and understanding URLs
• Web site asks for too much information
People can learn from web-based training materials,
if only we could get them to read them!
![Page 14: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/14.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
How Do We Get People How Do We Get People Trained?Trained? Most people don’t proactively look for training
materials on the web
Companies send “security notice” emails to employees and/or customers
We hypothesized these tend to be ignored• Too much to read
• People don’t consider them relevant
• People think they already know how to protect themselves
Led us to idea of embedded training
![Page 15: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/15.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Embedded TrainingEmbedded Training
Can we “train” people during their normal use of email to avoid phishing attacks? • Periodically, people get sent a training email
• Training email looks like a phishing attack
• If person falls for it, intervention warns and highlights what cues to look for in succinct and engaging format
P. Kumaraguru, Y. Rhee, A. Acquisti, L. Cranor, J. Hong, and E. Nunge. Protecting People from Phishing: The Design and Evaluation of an Embedded Training Email System. CHI 2007.
![Page 16: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/16.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Subject: Revision to Your Amazon.com Information
Please login and enter your information
http://www.amazon.com/exec/obidos/sign-in.html
Embedded training exampleEmbedded training example
![Page 17: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/17.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Intervention #1 – DiagramIntervention #1 – Diagram
![Page 18: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/18.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Intervention #1 – DiagramIntervention #1 – Diagram
Explains why they are seeing this message
![Page 19: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/19.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Intervention #1 – DiagramIntervention #1 – Diagram
Explains what aphishing scam is
![Page 20: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/20.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Intervention #1 – DiagramIntervention #1 – DiagramExplains how to identifya phishing scam
![Page 21: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/21.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Intervention #1 – DiagramIntervention #1 – DiagramExplains simple thingsyou can do to protect self
![Page 22: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/22.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Intervention #2 – Comic Intervention #2 – Comic StripStrip
![Page 23: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/23.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Summary of Evaluation ResultsSummary of Evaluation Results
Study setup: Role play as Bobby Smith at Cognix Inc going through companies emails• 10 participants in each condition, screened for novice
Evaluation I: Lab study comparing our prototypes to standard security notices• Existing practice of security notices is ineffective
• Embedded training is effective
• Comic strip intervention worked best
Evaluation II: • Have to fall for phishing email to be effective?
• How well do people retain knowledge?
![Page 24: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/24.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Results of Evaluation #2Results of Evaluation #2 Have to fall for phishing email to be effective?
How well do people retain knowledge after a week?
![Page 25: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/25.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Results of Evaluation #2Results of Evaluation #2 Have to fall for phishing email to be effective?
How well do people retain knowledge after a week?
0.07
0.18
0.64
0.14
0.04
0.68
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
before immediate delay
Training set
Mean
co
rrectn
ess
Non-embedded condition Embedded condition
Cor
rect
ness
![Page 26: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/26.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Results of Evaluation #2Results of Evaluation #2 Have to fall for phishing email to be effective?
How well do people retain knowledge after a week?
0.07
0.18
0.64
0.14
0.04
0.68
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
before immediate delay
Training set
Mean
co
rrectn
ess
Non-embedded condition Embedded condition
Cor
rect
ness
![Page 27: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/27.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Anti-Phishing PhilAnti-Phishing Phil
A game to teach people not to fall for phish• Embedded training focuses on email
• Our game focuses on web browser
Goals• How to parse URLs
• Where to look for URLs
• Use search engines for help
S. Sheng, B. Magnien, P. Kumaraguru, A. Acquisti, L. Cranor, J. Hong, and E. Nunge. Anti-Phishing Phil: The Design and Evaluation of a Game That Teaches People Not to Fall for Phish. In Proceedings of the 2007 Symposium On Usable Privacy and Security, Pittsburgh, PA, July 18-20, 2007.
27
![Page 28: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/28.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Anti-Phishing PhilAnti-Phishing Phil
![Page 29: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/29.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 30: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/30.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 31: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/31.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 32: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/32.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 33: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/33.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 34: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/34.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Summary of Evaluation of Anti-Summary of Evaluation of Anti-Phishing PhilPhishing Phil Test participants’ ability to identify phishing
web sites before and after training up to 15 min• 10 web sites before training, 10 after, randomized order
Evaluation I: Lab study• How do Phil perform with
existing training materials?
Evaluation II: Online study• How well do people retain what they
learned?
![Page 35: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/35.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
ResultsResults
Phil had the best performance overall, with lowest false positives
Novice users improve by 47%, intermediate users by 25%
People remembered what they learned one week after the training
Over 52,000 people played the game in the last three months
35
![Page 36: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/36.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Game resultsGame results
29.7%
59.9%
92.5%
77.5%
84.6%
93.5%
76.8%
85.0%
93.9%
0%
20%
40%
60%
80%
100%
Novice (N = 46) Intermediate (N =256)
Expert (N = 372)
tota
l co
rrec
tnes
s
Pre testPost testOne Week Later
36
![Page 37: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/37.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 37
![Page 38: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/38.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 39: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/39.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Teaching users about phishing attacks
can be a reality!
![Page 40: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/40.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
OutlineOutline
Human side• Interviews to understand decision-making
• PhishGuru embedded training
• Anti-Phishing Phil game
• Understanding effectiveness of browser warnings
Computer side• PILFER email anti-phishing filter
• CANTINA web anti-phishing algorithm
Do people see, understand, and believe web browser warnings?
![Page 41: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/41.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
ScreenshotsScreenshots
Internet Explorer – Passive Warning
![Page 42: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/42.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
ScreenshotsScreenshots
Internet Explorer – Active Block
![Page 43: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/43.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
ScreenshotsScreenshots
Mozilla FireFox – Active Block
![Page 44: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/44.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
How Effective are these How Effective are these Warnings?Warnings? Tested four conditions
• FireFox Active Block
• IE Active Block
• IE Passive Warning
• Control (no warnings or blocks)
“Shopping Study”• Setup some fake phishing pages and added to blacklists
• Users were phished after purchases
• Real email accounts and personal information
• Spoofing eBay and Amazon (2 phish/user)
• We observed them interact with the warnings
![Page 45: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/45.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
How Effective are these How Effective are these Warnings?Warnings?
![Page 46: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/46.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
How Effective are these How Effective are these Warnings?Warnings?
![Page 47: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/47.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Discussion of Phish WarningsDiscussion of Phish Warnings
Nearly everyone will fall for highly contextual phish
Passive IE warning failed for many reasons• Didn’t interrupt the main task
• Slow to appear (up to 5 seconds)
• Not clear what the right action was
• Looked too much like other ignorable warnings (habituation)
• Bug in implementation, any keystroke dismisses
![Page 48: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/48.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
ScreenshotsScreenshots
Internet Explorer – Passive Warning
![Page 49: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/49.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Discussion of Phish WarningsDiscussion of Phish Warnings
Active IE warnings• Most saw but did not believe it
“Since it gave me the option of still proceeding to the website, I figured it couldn’t be that bad”
• Some element of habituation (looks like other warnings)
• Saw two pathological cases
Egelman, S, Cranor, L, Hong, J. You’ve been Warned. In CHI 2008.
![Page 50: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/50.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
ScreenshotsScreenshots
Internet Explorer – Active Block
![Page 51: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/51.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
OutlineOutline
Human side• Interviews to understand decision-making
• PhishGuru embedded training
• Anti-Phishing Phil game
• Understanding effectiveness of browser warnings
Computer side• PILFER email anti-phishing filter
• CANTINA web anti-phishing algorithm
Can we automatically detect phish emails?
![Page 52: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/52.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
PILFER Email Anti-Phishing FilterPILFER Email Anti-Phishing Filter
Philosophy: automate where possible, support where necessary
Goal: Create email filter that detects phishing emails• Spam filters well-explored, but how good for phishing?
• Can we create a custom filter for phishing?
I. Fette, N. Sadeh, A. Tomasic. Learning to Detect Phishing Emails. In W W W 2007.
![Page 53: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/53.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
PILFER Email Anti-Phishing FilterPILFER Email Anti-Phishing Filter
Heuristics combined in SVM• IP addresses in link (http://128.23.34.45/blah)
• Age of linked-to domains (younger domains likely phishing)
• Non-matching URLs (ex. most links point to PayPal)
• “Click here to restore your account”
• HTML email
• Number of links
• Number of domain names in links
• Number of dots in URLs (http://www.paypal.update.example.com/update.cgi)
• JavaScript
• SpamAssassin rating
![Page 54: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/54.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
PILFER EvaluationPILFER Evaluation
Ham corpora from SpamAssassin (2002 and 2003)• 6950 good emails
Phishingcorpus• 860 phishing emails
![Page 55: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/55.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
PILFER EvaluationPILFER Evaluation
![Page 56: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/56.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
PILFER EvaluationPILFER Evaluation
PILFER now implemented as SpamAssassin filter
![Page 57: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/57.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
OutlineOutline
Human side• Interviews to understand decision-making
• PhishGuru embedded training
• Anti-Phishing Phil game
• Understanding effectiveness of browser warnings
Computer side• PILFER email anti-phishing filter
• CANTINA web anti-phishing algorithm
How good is phish detection for web sites?Can we do better?
![Page 58: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/58.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Lots of Phish Detection Lots of Phish Detection AlgorithmsAlgorithms Dozens of anti-phishing toolbars offered
• Built into security software suites
• Offered by ISPs
• Free downloads (132 on download.com)
• Built into latest version of popular web browsers
![Page 59: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/59.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Lots of Phish Detection Lots of Phish Detection AlgorithmsAlgorithms Dozens of anti-phishing toolbars offered
• Built into security software suites
• Offered by ISPs
• Free downloads (132 on download.com)
• Built into latest version of popular web browsers
But how well do they detect phish?• Short answer: still room for improvement
![Page 60: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/60.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Testing the ToolbarsTesting the Toolbars
November 2006: Automated evaluation of 10 toolbars• Used phishtank.com and APWG as source of phishing URLs
• Evaluated 100 phish and 510 legitimate sites
Y. Zhang, S. Egelman, L. Cranor, J. Hong. Phinding Phish: An Evaluation of Anti-Phishing Toolbars. NDSS 2006.
![Page 61: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/61.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Testbed System ArchitectureTestbed System Architecture
![Page 62: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/62.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
ResultsResults
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 1 2 12 24
Time (hours)
Phi
shin
g si
tes
corr
ectly
iden
tifie
d
SpoofGuardEarthLinkNetcraftGoogleIE7CloudmarkTrustWatcheBayNetscapeMcAfee
38% false positives
1% false positives
PhishTank
![Page 63: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/63.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
ResultsResults
Only one toolbar >90% accuracy (but high false positives)
Several catch 70-85% of phish with few false positives
![Page 64: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/64.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
ResultsResults
Only one toolbar >90% accuracy (but high false positives)
Several catch 70-85% of phish with few false positives
Can we do better?• Can we use search engines to help find phish?
Y. Zhang, J. Hong, L. Cranor. CANTINA: A Content-Based Approach to Detecting Phishing Web Sites. In W W W 2007.
![Page 65: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/65.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Robust HyperlinksRobust Hyperlinks
Developed by Phelps and Wilensky to solve “404 not found” problem
Key idea was to add a lexical signature to URLs that could be fed to a search engine if URL failed• Ex. http://abc.com/page.html?sig=“word1+word2+...+word5”
How to generate signature?• Found that TF-IDF was fairly effective
Informal evaluation found five words was sufficient for most web pages
![Page 66: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/66.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Adapting TF-IDF for Anti-Adapting TF-IDF for Anti-PhishingPhishing Can same basic approach be used for anti-phishing?
• Scammers often directly copy web pages
• With Google search engine, fake should have low page rank
Fake Real
![Page 67: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/67.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
How CANTINA WorksHow CANTINA Works
Given a web page, calculate TF-IDF score for each word in that page
Take five words with highest TF-IDF weights
Feed these five words into a search engine (Google)
If domain name of current web page is in top N search results, we consider it legitimate • N=30 worked well
• No improvement by increasing N
Later, added some heuristics to reduce false positives
![Page 68: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/68.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Fake
eBay, user, sign, help, forgot
![Page 69: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/69.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Real
eBay, user, sign, help, forgot
![Page 70: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/70.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 71: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/71.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 72: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/72.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Evaluating CANTINAEvaluating CANTINAPhishTank
![Page 73: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/73.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Weaknesses in CANTINAWeaknesses in CANTINA
Bad guys may try to subvert search engines
Only works if legitimate page is indexed• Intranets
May be confused if same login page in multiple places
![Page 74: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/74.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
SummarySummary
Whirlwind tour of our work on anti-phishing• Human side: how people make decisions, training, UIs• Computer side: better algorithms for detecting phish
More info about our work at cups.cs.cmu.edu
![Page 75: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/75.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
AcknowledgmentsAcknowledgments
Alessandro Acquisti
Lorrie Cranor
Sven Dietrich
Julie Downs
Mandy Holbrook
Norman Sadeh
Anthony Tomasic
Umut Topkara
Supported by NSF, ARO, CyLab, Portugal Telecom
• Serge Egelman• Ian Fette• Ponnurangam
Kumaraguru• Bryant Magnien• Elizabeth Nunge• Yong Rhee• Steve Sheng• Yue Zhang
![Page 76: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/76.jpg)
CMU Usable Privacy and Security Laboratoryhttp://cups.cs.cmu.edu/
Steve ShengEngineering and Public
![Page 77: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/77.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 78: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/78.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Is it phish?Is it phish?
Our label
Yes No
Yes True positive False negative
No False positive True negative
![Page 79: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/79.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 80: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/80.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 81: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/81.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 82: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/82.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Minimal Knowledge of Lock IconMinimal Knowledge of Lock Icon
“I think that it means secured, it symbolizes some kind of security, somehow.”
85% of participants were aware of lock icon
Only 40% of those knew that it was supposed to be in the browser chrome
Only 35% had noticed https, and many of those did not know what it meant
![Page 83: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/83.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Solution SpaceSolution Space
83
![Page 84: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/84.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Phishing continues to evolvePhishing continues to evolve
Spear-phishing on the rise for US military and other organizations aiming sensitive information
Voice over IP phishing becoming more prevalent
Phishing techniques continue to evolve
84
![Page 85: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/85.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Research ProblemResearch Problem
As phishing continues to evolve, what can and should stakeholders do to better fight it?
85
![Page 86: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/86.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Summary of Summary of Thesis StatementThesis Statement Identify phishing stakeholders and their stakes
Find gaps in the countermeasures pursued by each stakeholder
Generate and evaluate policy options to better fight phishing now and in the future
Case studies on the effectiveness of anti-phishing toolbars and game-based anti-phishing education
86
![Page 87: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/87.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
OverviewOverview
Case study: Antiphishing toolbars
Case study: Antiphishing toolbars
Case study: Anti-phishing Phil
Case study: Anti-phishing Phil
87
![Page 88: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/88.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
OutlineOutline
Background• Relevant literature
• Prior Work
Public Policy Analysis
Case Study in Anti-phishing toolbars
Case Study in User Education
Schedule
88
![Page 89: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/89.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
OutlineOutline
Background• Relevant literature
• Prior Work
Public Policy Analysis
Case Study in Anti-phishing toolbars
Case Study in User Education
Schedule
89
![Page 90: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/90.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
StakeholdersStakeholdersConsumers --Organizations US Military, Universities, CorporationsFinancial Institutions Bank of America, Citibank, Wachovia, PaypalMerchants eBay, AmazonInternet Service Providers SBC, Comcast, AOLEmail Providers Gmail, YahooMail, Hotmail, Outlook, ThunderbirdBrowsers Internet Explorer, Firefox, Safari, Opera, Netscape.DNS authorities Verisign, various NICsSoftware Vendors Google, Microsoft, Symantec, RSA, MarkMonitorLaw Enforcements Federal Bureau of Investigation(FBI), CERT, Secret
Service, Identity Theft Divisions in Law enforcements
Government Regulators Federal Financial Institutions Examination Council (FFIEC), Federal Trade Commission (FTC)
Academic Institutions Carnegie Mellon University, Indiana UniversityIndustry Consortium Financial Services Technology Council(FSTC), Anti-
Phishing Working Group (APWG), Messaging Anti-Abuse Working Group(MAAWG)
Direct stakeholders Indirect stakeholders
PrimaryVictimsSecondary VictimsVendors
Enforcement
Oversight / Coordination / Research
Market based
90
![Page 91: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/91.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Phishing CountermeasuresPhishing Countermeasures
91
Education
![Page 92: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/92.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Economics of information Economics of information Security - ExternalitiesSecurity - Externalities
Does successfully combating phishing depends on the efforts of the laziest and most cowardly family? or the most valiant knight? or sum of efforts?
If it is all of above, which part requires what kinds of efforts?
92
![Page 93: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/93.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
HypothesisHypothesis
Consumers are the weakest link
The problem can be solved if a solution has ubitiquous coverage and near perfect performance, and browsers are the most likely candidate.• In which case, phishers will use other channels
Effective law enforcements require the sum of all efforts
93
![Page 94: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/94.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Longitudinal TrendsLongitudinal Trends
94
![Page 95: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/95.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Estimating problemsEstimating problems
95
![Page 96: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/96.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Estimating CountermeasuresEstimating Countermeasures
PhishingEmails
MailGateways
MailClients
End UserAction
Webbrowsers
User enters info
userknowledge
client filters warninggateway filters
MailStorage
storage filters
web mailclients
bank fraud
fraud detection
authenticationsystem
lawenforcements
1) What advantages, constraints does each stakeholder have in their phishing countermeasures? 2) What kind of solutions best fit each type of stakeholder?
96
![Page 97: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/97.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Understanding ConstraintsUnderstanding Constraints
97
![Page 98: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/98.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Expert InterviewsExpert Interviews
Goal: To further understand current and future phishing threats, relevant countermeasures, and with an eye on tomorrow, countermeasures should be put in place.
12 experts from industry associations, academia, industry, law enforcements, and volunteer organizations
98
![Page 99: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/99.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Expert InterviewsExpert Interviews
Sectors Examples Number to Interview
Industry Associations Anti-Phishing Working Group (APWG), Messaging Anti-abuse Working Group (MAAWG), Financial Services Technology Council (FSTC)
2-3 officers
Industry Microsoft, Google, RSA, Symantec, MarkMonitor, McAfee, MessageLabs , and CloudMark
3-6 experts
Law Enforcements Federal Bureau of Investigation (FBI), Secret Service, CERT
2-4 experts
Academia CMU and other institutions 3-5 faculty
Volunteer Organizations PhishTank, CastleCorps 2 experts
99
![Page 100: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/100.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
High Level QuestionsHigh Level Questions
Phishing threats• What do you think the current state of phishing?
• How phishing are costing various stakeholders?
• What kinds of attacks would likely to happen in the near future and long term?
Countermeasures• What kinds of solutions are stakeholders adopting?
• What are some effective ways to combat phishing?
• In light of the evolving phishing threats, what are some of the most promising ways?
• Is there anything missing in the countermeasures?
100
![Page 101: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/101.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
High Level QuestionsHigh Level Questions
Policy Related• Who is the best position to solve the problem? and what
kind of solutions you see are lacking?
• What additional investments are needed?
• How should we prioritize our spending on prevention, detection, shutdown, and education?
• Where are we wasting our money at?
101
![Page 102: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/102.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
MethodologyMethodology
Semi-structured interviews
Refine the objective and questions; outline a design; draft the interview questions; pilot test with 3 CMU experts, iterate on it more based on the results
Conduct interviews from May 2008 to October 2008
Follow up surveys with some organizations
102
![Page 103: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/103.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
OverviewOverview
103
![Page 104: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/104.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Gap AnalysisGap Analysis
Map countermeasures with attack vectors
Contrast stakeholders actions with expert analysis and recommendations
104
![Page 105: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/105.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Gap AnalysisGap Analysis
Attack Prevention Detection Warning Block / Shutdown
Website
Instant Messaging
Auto Dialer
News, Chat Room, Blog
Bulletin Board
Wireless LANs
P2P or Interactive
Games
Malware
105
![Page 106: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/106.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Phishing Life CyclePhishing Life Cycle
Source: Financial Service Technology Consortium, 2005
106
![Page 107: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/107.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Phishing Life CyclePhishing Life Cycle
107
![Page 108: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/108.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Why People fallWhy People fall
People judge a website's legitimacy by its “look and feel” (Rachna et al. 2006, Wu et al. 06)
Many do not understand or trust web browser indicator (Downs et al. 2007)
Awareness do not link to different behaviors or strategies (Downs et al. 2007)
Perceived severity of the consequences does not predict their behaviors (Downs et al. 2007)
108
![Page 109: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/109.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Cost of PhishingCost of Phishing
Direct costs• Consumers lose money, banking fraud
• Estimated 350 – 2 billion
Indirect costs• Erosion of consumer trust
• Impact on brand name
• Increase in customer call centers
Opportunity costs
109
![Page 110: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/110.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Recent DevelopmentsRecent Developments
VOIP phishing
Spear phishing
Rock phish and fast flux
110
![Page 111: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/111.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/ 111
![Page 112: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/112.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Phil Online User StudyPhil Online User Study
Conducted in 9/25 – 10/10
Validate Lab study results
Test for retention of knowledge
Condition• Control: N = 2702 (12 websites + game)
• Game: N = 2021 (674 complete one week later) (6 website + game + 6 website + 6 website one week later)
112
![Page 113: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/113.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Game resultsGame results
29.7%
59.9%
92.5%
77.5%
84.6%
93.5%
76.8%
85.0%
93.9%
0%
20%
40%
60%
80%
100%
Novice (N = 46) Intermediate (N =256)
Expert (N = 372)
tota
l co
rrec
tnes
s
Pre testPost testOne Week Later
113
![Page 114: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/114.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Misidentifying Legitimate SitesMisidentifying Legitimate Sites
42.0%
30.1%
5.5%
11.2%7.9%
2.8%
12.3%
8.4%
2.5%
0%
10%
20%
30%
40%
50%
Novice (N = 46) Intermediate (N =256)
Expert (N = 372)
Fal
se P
osi
tive
Pre test
Post test
One Week Later
114
![Page 115: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/115.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Falling for PhishingFalling for Phishing
28.3%
10.0%
2.0%
11.2%
7.4%
3.7%
10.9%
7.1%
3.7%
0%
10%
20%
30%
40%
Novice (N = 46) Intermediate (N= 256)
Expert (N = 372)
Fal
se N
egat
ive
Rat
e
Pre test
Post test
One Week Later
115
![Page 116: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/116.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Comparing Control with GameComparing Control with Game
Control group performance• Pre test score: 70.9%
• Post test score:67.1%
The effect is not due to simply showing the quiz. (p<0.0001, N = 4674) (2 sample t test on (Score_post – Score_pre))
116
![Page 117: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/117.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Signal detection theory Signal detection theory to measure learningto measure learning Users are learning well in the game
• d’_pre = 1.49, d’_post = 2.46 (p<0.001).
The improvement is not due to becoming more suspicious, in fact the reverse it true. • C’_pre = -0.352, C’_post = 0.016. (p<0.001)
117
![Page 118: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/118.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Intervention #2 – Comic Intervention #2 – Comic StripStrip
![Page 119: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/119.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Intervention #2 – Comic Intervention #2 – Comic StripStrip
![Page 120: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/120.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Little Knowledge of PhishingLittle Knowledge of Phishing
Only about half knew meaning of the term “phishing”
“Something to do with the band Phish, I take it.”
![Page 121: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/121.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Little Attention Paid to URLsLittle Attention Paid to URLs
Only 55% of participants said they had ever noticed an unexpected or strange-looking URL
Most did not consider them to be suspicious
![Page 122: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/122.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Some Knowledge of ScamsSome Knowledge of Scams
55% of participants reported being cautious when email asks for sensitive financial info• But very few reported being suspicious of email
asking for passwords
Knowledge of financial phish reduced likelihood of falling for these scams• But did not transfer to other scams, such as an
amazon.com password phish
![Page 123: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/123.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Embedded Training Evaluation Embedded Training Evaluation #1#1 Lab study comparing our prototypes to
standard security notices• Group A – eBay, PayPal notices
• Group B – Diagram that explains phishing
• Group C – Comic strip that tells a story
10 participants in each condition (30 total)• Screened so we only have novices
Go through 19 emails, 4 phishing attacks scattered throughout, 2 training emails too• Role play as Bobby Smith at Cognix Inc
![Page 124: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/124.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Embedded Training ResultsEmbedded Training Results
0102030405060708090
100
Emails which had links in them
Pe
rce
nta
ge
of
use
rs w
ho
clic
ke
d
on
a li
nk
Group A Group B Group C
![Page 125: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/125.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Embedded Training ResultsEmbedded Training Results
Existing practice of security notices is ineffective
Diagram intervention somewhat better• Though people still fell for final phish
Comic strip intervention worked best• Statistically significant
• Combination of less text, graphics, story?
![Page 126: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/126.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
Evaluation #2Evaluation #2
New questions:• Have to fall for phishing email to be effective?
• How well do people retain knowledge?
Roughly same experiment as before• Role play as Bobby Smith at Cognix Inc, go thru 16 emails
• Embedded condition means have to fall for our email
• Non-embedded means we just send the comic strip
• Also had people come back after 1 week
Kumaraguru, P., Rhee, Y., Sheng, S., Hasan, S., Acquisti, A., Cranor, L. F., and Hong, J. Getting users to pay attention to anti-phishing education: evaluation of retention and transfer. In eCrime ’07: Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit
![Page 127: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/127.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
![Page 128: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/128.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
A Science of A Science of WarningsWarnings
See the warning?
Understand?
Believe it?
Motivated?
Planning on refining this model for computer warnings
![Page 129: C MU U sable P rivacy and S ecurity Laboratory User Interfaces and Algorithms for Fighting Phishing Steve Sheng Doctoral Candidate,](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ee85503460f94bf93b6/html5/thumbnails/129.jpg)
Steve Sheng• CMU Usable Privacy and Security Laboratory • http://cups.cs.cmu.edu/
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 1 2 12 24
Time (hours)
Phis
hin
g s
ites c
orr
ectly identified
SpoofGuard
EarthLink
Netcraft
Firefox w/Google
IE7
Cloudmark
TrustWatch
eBay
Netscape
CallingID
Firefox
APWG