fix what matters: a data driven approach to vulnerability management
DESCRIPTION
Data driven approach to vulnerability management in information security using live breach and vulnerability data.TRANSCRIPT
![Page 1: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/1.jpg)
Fix What Matters Michael Roytman
SIRAcon October 21, 2013
![Page 2: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/2.jpg)
Why You Should(n’t) Listen
• Naive Grad Student Not Too Long Ago• Still Plays With Legos• Barely Passed Regression Analysis
• MS Operations Research, Georgia Tech
Michael Roytman• Data Scientist, Risk I/O
• Fraud Detection, Large Bank
![Page 3: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/3.jpg)
Roadmap
• The Struggle
• What’s Good?
• Data Driven Insights• Framework
• Decision-Making
• What’s Bad?
![Page 4: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/4.jpg)
Starting From Scratch
“It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories,
instead of theories to suit facts.”
-Sir Arthur Conan Doyle, 1887
![Page 5: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/5.jpg)
Starting From Scratch
![Page 6: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/6.jpg)
Starting From Scratch
Academia!• GScholar!• JSTOR!• IEEE!• ProQuest!
InfoSec Blogs!• CSIOs!• Pen Testers!• Threat Reports!• SOTI/DBIR!!
Twitter!• Thought Leaders (you
know who you are)!• BlackHats!• Vuln Researchers!
Primary Sources!• MITRE!• OSVDB!• NIST CVSS
Committee(s)!• Internal Message
Boards for ^!Text
CISOs
![Page 7: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/7.jpg)
Data Fundamentalism
Don’t Ignore What a Vulnerability Is: Creation Bias !
(http://blog.risk.io/2013/04/data-fundamentalism/) !
Jerico/Sushidude @ BlackHat !
(https://www.blackhat.com/us-13/briefings.html#Martin)!
Luca Allodi - CVSS DDOS !
(http://disi.unitn.it/~allodi/allodi-12-badgers.pdf):!
![Page 8: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/8.jpg)
Data Fundamentalism - What’s The Big Deal?
!
”Since 2006 Vulnerabilities have declined by 26 percent.” !(http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf)!
!
!
“The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ”!
(http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf)!!
!
![Page 9: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/9.jpg)
What’s Good?
Bad For Vulnerability Statistics:!!
NVD, OSVDB, ExploitDB, CVSS, Patches, Microsoft Reports, etc, et al, and so on. !
Good For Vulnerability Statistics:!!
Vulnerabilities. !
![Page 10: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/10.jpg)
Data Is Everything And Everything Is Data.
![Page 11: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/11.jpg)
What’s Good?
![Page 12: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/12.jpg)
What’s Good?
![Page 13: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/13.jpg)
What’s Good?
![Page 14: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/14.jpg)
What’s Good?
![Page 15: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/15.jpg)
What’s Good?
![Page 16: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/16.jpg)
What’s Good?
![Page 17: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/17.jpg)
Counterterrorism
Known Groups
Surveillance
Threat Intel, Analysts
Targets, Layouts
Past Incidents, Close Calls
![Page 18: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/18.jpg)
What’s Good?
![Page 19: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/19.jpg)
Uh, Sports?
Opposing Teams, Specific Players
Gameplay
Scouting Reports, Gametape
Roster, Player Skills
Learning from Losing
![Page 20: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/20.jpg)
InfoSec?
![Page 21: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/21.jpg)
Defend Like You’ve Done It Before
Groups, Motivations
Exploits
Vulnerability Definitions
Asset Topology, Actual Vulns on System
Learning from Breaches
![Page 22: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/22.jpg)
Work With What You’ve Got:
Akamai, Safenet
ExploitDB, Metasploit
NVD, MITRE
![Page 23: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/23.jpg)
Add Some Spice
![Page 24: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/24.jpg)
Show Me The Money
23,000,000 Vulnerabilities!
Across 1,000,000 Assets!
Representing 9,500 Companies!
Using 22 Unique Scanners!
![Page 25: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/25.jpg)
Whatchu Know About Dat?(a)
!
Duplication
Vulnerability Density
Remediation
![Page 26: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/26.jpg)
Duplication
0
225,000
450,000
675,000
900,000
1,125,000
1,350,000
1,575,000
1,800,000
2,025,000
2,250,000
2 or more scanners 3 or more 4 or more 5 or more 6 or more
![Page 27: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/27.jpg)
Duplication
We Have: F(Number of Scanners) => Number of Duplicate Vulnerabilities
We Want: F(Number of Scanners) => Vulnerability Coverage
Make Decisions At The Margins!
<---------Good Luck!
0.0
25.0
50.0
75.0
100.0
0 1 2 3 4 5 6
![Page 28: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/28.jpg)
Density
Type of Asset ~Count
Hostname 20,000
Netbios 1000
IP Address 200,000
File 10,000
Url 5,000
Hostname
Netbios
IP
File
Url
0.0 22.5 45.0 67.5 90.0
![Page 29: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/29.jpg)
CVSS And Remediation Metrics
0.0
350.0
700.0
1050.0
1400.0
1 2 3 4 5 6 7 8 9 10
Average Time To Close By Severity Oldest Vulnerability By Severity
![Page 30: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/30.jpg)
CVSS And Remediation - Lessons From A CISORemediation/Lack Thereof, by CVSS
1 2 3 4 5 6 7 8 9 10
NVD Distribution by CVSS
![Page 31: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/31.jpg)
The Kicker - Live Breach Data
1,500,000 !Vulnerabilities Related to Live Breaches Recorded!
June, July 2013 !
![Page 32: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/32.jpg)
CVSS And Remediation - Nope
0.0
1750.0
3500.0
5250.0
7000.0
1 2 3 4 5 6 7 8 9 10
Oldest Breached Vulnerability By Severity
![Page 33: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/33.jpg)
CVSS - A VERY General Guide For Remediation - Yep
0.0
40000.0
80000.0
120000.0
160000.0
1 2 3 4 5 6 7 8 9 10
Open Vulns With Breaches Occuring By Severity
![Page 34: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/34.jpg)
The One Billion Dollar Question
Probability(You Will Be Breached On A Particular Open Vulnerability)?
1.98%=(Open Vulnerabilities | Breaches Occurred On Their CVE)/(Total Open Vulnerabilities)
![Page 35: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/35.jpg)
I Love It When You Call Me Big Data
Probability A Vulnerability Having Property X Has Observed Breaches
RANDOM VULN
CVSS 10
CVSS 9
CVSS 8
CVSS 6
CVSS 7
CVSS 5
CVSS 4
Has Patch
0.00000 0.01000 0.02000 0.03000 0.04000
![Page 36: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/36.jpg)
What’s the Alternative?
![Page 37: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/37.jpg)
I Love It When You Call Me Big Data
Probability A Vulnerability Having Property X Has Observed Breaches
Random Vuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0 0.1 0.2 0.2 0.3
![Page 38: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/38.jpg)
Data Is Everything And Everything Is Data.
![Page 39: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/39.jpg)
Be Better Than The Gap
![Page 40: Fix What Matters: A Data Driven Approach to Vulnerability Management](https://reader033.vdocument.in/reader033/viewer/2022052522/554beb38b4c9056b348b4f8e/html5/thumbnails/40.jpg)
I Love It When You Call Me Big Data
Spray and Pray => 2% !
CVSS 10 => 4% !
Metasploit + ExploitDB => 30% !
A Good Model That’s Not Built By One Kid Without Hadoop => ???!