05-daniel votipka-hackers vs testers a comparison of ...dvotipka/slides/05-danielvotipka-hac… ·...

Hackers vs Testers: A Comparison of Software Vulnerability Discovery

ProcessesDaniel Votipka, Rock A. Stevens, Elissa M. Redmiles,

Jeremy Hu, and Michelle L. Mazurek

22 May 2018

VULNERABILITY DISCOVERY

!2

!3

Testers: • Functionality • Performance • Security

VULNERABILITY DISCOVERY

Generalists

!4

Hackers: • Security Team • Contracted Review • Bug Bounty

VULNERABILITY DISCOVERY

Experts

CHALLENGES

!5

•Timeliness •Cognitive Diversity •Communication

RESEARCH QUESTIONS

1. How do testers and hackers search for vulnerabilities?

2. What are the differences between testers and hackers?

!6

Interview study: • Task Analysis • Tools, Skills, and Communities

RECRUITMENTHacker Groups: • Bug Bounty Programs • Top Hacking Teams

Tester Groups: • Meetup and LinkedIn • IEEE and AST • Ministry of Testing

!7

106 total groups

PARTICIPANTS

!8

10 15

0-3 26-50

Participants

Vulnerabilities Found

Vulnerability Finding Time 5-10 hrs/w 10-20 hrs/w

RESEARCH QUESTIONS

1. How do testers and hackers search for vulnerabilities?

2. What are the differences between testers and hackers?

!9

RESEARCH QUESTIONS

1. How do testers and hackers search for vulnerabilities?

2. What are the differences between testers and hackers?

!10

!11

Info Gathering

Program Understanding

Attack Surface

Exploration

Vulnerability Recognition

Reporting

HACKER AND TESTER PROCESS

!12

Info Gathering

Program Understanding

Attack Surface

Exploration

Vulnerability Recognition

Reporting

Information Gathering • Build context prior to reading or

executing code • Example actions:

• Identifying libraries • Update history • Previous bug reports

“There were…other functional issues, so I figured that was probably where there was most likely to be security

issues as well. Bugs tend to cluster.”

!13

Info Gathering

Program Understanding

Attack Surface

Exploration

Vulnerability Recognition

Reporting

Program Understanding • Determine how the program operates

• Interaction between components • Interaction with the environment

“You’re touching a little bit everything, and then you are organizing that into

a structure in your head.”

!14

Info Gathering

Program Understanding

Attack Surface

Exploration

Vulnerability Recognition

Reporting

Attack Surface • Identify how user interacts with program • Direct and indirect inputs

!15

Info Gathering

Program Understanding

Attack Surface

Exploration

Vulnerability Recognition

Reporting

Exploration • Possible inputs to the attack surface • Example actions:

• Fuzzing • Reading code

!16

Info Gathering

Program Understanding

Attack Surface

Exploration

Vulnerability Recognition

Reporting

Vulnerability Recognition • Notice a problem when exploring • Typically described as intuition-based

!17

Info Gathering

Program Understanding

Attack Surface

Exploration

Vulnerability Recognition

Reporting

Reporting • Tell developers about the problem • Advocate for remediation • Critical aspects:

• Make report understandable • Importance of fixing

“You do have to [convince] someone that there’s a risk. …It’s quite timely [time consuming], running a ticket.”

RESEARCH QUESTIONS

1. How do testers and hackers search for vulnerabilities?

2. What are the differences between testers and hackers?

!18

!19

Info Gathering

Program Understanding

Attack Surface

Exploration

Vulnerability Recognition

Reporting

Vulnerability Discovery

Experience

Access to Development

Process

Underlying System

Knowledge

Motivation

!20

Info Gathering

Program Understanding

Attack Surface

Exploration

Vulnerability Recognition

Reporting

Vulnerability Discovery

Experience

Access to Development

Process

Underlying System

Knowledge

Motivation

Attack Surface

!21

Exploration

Vulnerability Recognition

Vulnerability Discovery Experience

Exploration • Informs test case selection

Attack Surface • More likely to consider

indirect input paths

Vulnerability Recognition • Know vulnerability patterns • List of common vulnerabilities

“As soon as I found the LinkedIn problem, I made sure to test [FB and Twitter input] to make sure [they were processed correctly]. And if we did allow login with

another 3rd party in the future, I would check that too.”

!22

Employment

Hacking Exercises

Community

Bug Reports

Attack Surface

Exploration

Vulnerability Recognition

Vulnerability Discovery Experience

!23

Employment

Hacking Exercises

Community

Bug Reports

Vulnerability Discovery Experience

AMOUNT OF EXPERIENCE

!24

Info Gathering

Program Understanding

Reporting

Access to Development

Process

Internal • Communicate with

developers • Documentation

External • Reverse engineering • Develop exploits

“You can give feedback to your developers. . . .You’re coming back with information, and then they react on it.”

“It’s hard to ignore certain details once you know about

certain areas already.”

RECOMMENDATIONS

!25

‣ Provide training in known contexts • Hire hackers into the testing team • Bug report-based exercises

‣ Improve hacker communication • Single point of contact

SUMMARY

!26

• Similar processes • Impacted by:

• Vulnerability Discovery Experience • Underlying System Knowledge, • Access to the Development Process • Motivation

• Biggest difference in amount of experience and relationship with the developers

Recommendations: • Training in a known context • Hacker/company communication

dvotipka@cs.umd.edu vulnstudy.cs.umd.edu

Questions: