05-daniel votipka-hackers vs testers a comparison of ...dvotipka/slides/05-danielvotipka-hac… ·...
TRANSCRIPT
![Page 1: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/1.jpg)
Hackers vs Testers: A Comparison of Software Vulnerability Discovery
ProcessesDaniel Votipka, Rock A. Stevens, Elissa M. Redmiles,
Jeremy Hu, and Michelle L. Mazurek
22 May 2018
![Page 2: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/2.jpg)
VULNERABILITY DISCOVERY
!2
![Page 3: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/3.jpg)
!3
Testers: • Functionality • Performance • Security
VULNERABILITY DISCOVERY
Generalists
![Page 4: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/4.jpg)
!4
Hackers: • Security Team • Contracted Review • Bug Bounty
VULNERABILITY DISCOVERY
Experts
![Page 5: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/5.jpg)
CHALLENGES
!5
•Timeliness •Cognitive Diversity •Communication
![Page 6: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/6.jpg)
RESEARCH QUESTIONS
1. How do testers and hackers search for vulnerabilities?
2. What are the differences between testers and hackers?
!6
Interview study: • Task Analysis • Tools, Skills, and Communities
![Page 7: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/7.jpg)
RECRUITMENTHacker Groups: • Bug Bounty Programs • Top Hacking Teams
Tester Groups: • Meetup and LinkedIn • IEEE and AST • Ministry of Testing
!7
106 total groups
![Page 8: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/8.jpg)
PARTICIPANTS
!8
10 15
0-3 26-50
Participants
Vulnerabilities Found
Vulnerability Finding Time 5-10 hrs/w 10-20 hrs/w
![Page 9: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/9.jpg)
RESEARCH QUESTIONS
1. How do testers and hackers search for vulnerabilities?
2. What are the differences between testers and hackers?
!9
![Page 10: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/10.jpg)
RESEARCH QUESTIONS
1. How do testers and hackers search for vulnerabilities?
2. What are the differences between testers and hackers?
!10
![Page 11: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/11.jpg)
!11
Info Gathering
Program Understanding
Attack Surface
Exploration
Vulnerability Recognition
Reporting
HACKER AND TESTER PROCESS
![Page 12: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/12.jpg)
!12
Info Gathering
Program Understanding
Attack Surface
Exploration
Vulnerability Recognition
Reporting
Information Gathering • Build context prior to reading or
executing code • Example actions:
• Identifying libraries • Update history • Previous bug reports
“There were…other functional issues, so I figured that was probably where there was most likely to be security
issues as well. Bugs tend to cluster.”
![Page 13: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/13.jpg)
!13
Info Gathering
Program Understanding
Attack Surface
Exploration
Vulnerability Recognition
Reporting
Program Understanding • Determine how the program operates
• Interaction between components • Interaction with the environment
“You’re touching a little bit everything, and then you are organizing that into
a structure in your head.”
![Page 14: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/14.jpg)
!14
Info Gathering
Program Understanding
Attack Surface
Exploration
Vulnerability Recognition
Reporting
Attack Surface • Identify how user interacts with program • Direct and indirect inputs
![Page 15: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/15.jpg)
!15
Info Gathering
Program Understanding
Attack Surface
Exploration
Vulnerability Recognition
Reporting
Exploration • Possible inputs to the attack surface • Example actions:
• Fuzzing • Reading code
![Page 16: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/16.jpg)
!16
Info Gathering
Program Understanding
Attack Surface
Exploration
Vulnerability Recognition
Reporting
Vulnerability Recognition • Notice a problem when exploring • Typically described as intuition-based
![Page 17: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/17.jpg)
!17
Info Gathering
Program Understanding
Attack Surface
Exploration
Vulnerability Recognition
Reporting
Reporting • Tell developers about the problem • Advocate for remediation • Critical aspects:
• Make report understandable • Importance of fixing
“You do have to [convince] someone that there’s a risk. …It’s quite timely [time consuming], running a ticket.”
![Page 18: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/18.jpg)
RESEARCH QUESTIONS
1. How do testers and hackers search for vulnerabilities?
2. What are the differences between testers and hackers?
!18
![Page 19: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/19.jpg)
!19
Info Gathering
Program Understanding
Attack Surface
Exploration
Vulnerability Recognition
Reporting
Vulnerability Discovery
Experience
Access to Development
Process
Underlying System
Knowledge
Motivation
![Page 20: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/20.jpg)
!20
Info Gathering
Program Understanding
Attack Surface
Exploration
Vulnerability Recognition
Reporting
Vulnerability Discovery
Experience
Access to Development
Process
Underlying System
Knowledge
Motivation
![Page 21: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/21.jpg)
Attack Surface
!21
Exploration
Vulnerability Recognition
Vulnerability Discovery Experience
Exploration • Informs test case selection
Attack Surface • More likely to consider
indirect input paths
Vulnerability Recognition • Know vulnerability patterns • List of common vulnerabilities
“As soon as I found the LinkedIn problem, I made sure to test [FB and Twitter input] to make sure [they were processed correctly]. And if we did allow login with
another 3rd party in the future, I would check that too.”
![Page 22: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/22.jpg)
!22
Employment
Hacking Exercises
Community
Bug Reports
Attack Surface
Exploration
Vulnerability Recognition
Vulnerability Discovery Experience
![Page 23: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/23.jpg)
!23
Employment
Hacking Exercises
Community
Bug Reports
Vulnerability Discovery Experience
AMOUNT OF EXPERIENCE
![Page 24: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/24.jpg)
!24
Info Gathering
Program Understanding
Reporting
Access to Development
Process
Internal • Communicate with
developers • Documentation
External • Reverse engineering • Develop exploits
“You can give feedback to your developers. . . .You’re coming back with information, and then they react on it.”
“It’s hard to ignore certain details once you know about
certain areas already.”
![Page 25: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/25.jpg)
RECOMMENDATIONS
!25
‣ Provide training in known contexts • Hire hackers into the testing team • Bug report-based exercises
‣ Improve hacker communication • Single point of contact
![Page 26: 05-Daniel Votipka-Hackers vs Testers A Comparison of ...dvotipka/slides/05-DanielVotipka-Hac… · • Hacker/company communication dvotipka@cs.umd.edu vulnstudy.cs.umd.edu Questions:](https://reader033.vdocument.in/reader033/viewer/2022042218/5ec4b3edd8ff1a5b1147f220/html5/thumbnails/26.jpg)
SUMMARY
!26
• Similar processes • Impacted by:
• Vulnerability Discovery Experience • Underlying System Knowledge, • Access to the Development Process • Motivation
• Biggest difference in amount of experience and relationship with the developers
Recommendations: • Training in a known context • Hacker/company communication
[email protected] vulnstudy.cs.umd.edu
Questions: