bsides 2014, fix what matters: why cvss sucks & how to do it better, by data scientist michael...
DESCRIPTION
Why using CVSS for vulnerability management is nuts. How to fix the vulnerabilities that truly matter, and how to create and measure an effective security practice.TRANSCRIPT
![Page 1: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/1.jpg)
Fix What Matters: !
Why CVSS Sucks And How To
Do Better
![Page 2: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/2.jpg)
Once Jailbroke an Iphone 3G
Michael Roytman
Proud Owner of Remote Controlled AirplaneRecently a Naive Grad Student
Data Scientist, Risk I/ODoes Not Wake Up Before 11 CST
qualifications:
![Page 3: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/3.jpg)
PART 1: !
YOU SUCK AT YOUR JOB
!
(and don’t even know it yet)
![Page 4: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/4.jpg)
Why Are We Here?
Empirical Failures of CVSSProper Remediation Frameworks (Yeah, they exist)
CVSS SUCKS
Analytical Failures of CVSS
(+Data Driven Alternatives)
![Page 5: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/5.jpg)
Remove the Threat
RemediationAccept the Risk
Repair the Vulnerability
![Page 6: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/6.jpg)
C(ommon) V(ulnerability) S(coring) S(ystem)
“CVSS is designed to rank information system vulnerabilities”
Exploitability/Temporal (Likelihood)
Impact/Environmental (Severity)
The Good: Open, Standardized Scores
![Page 7: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/7.jpg)
“It is a capital mistake to theorize before one has data.
!
!
!
Insensibly, one begins to twist facts to suit theories, instead of
theories to suit facts.”
![Page 8: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/8.jpg)
FAIL: Data FundamentalismDon’t Ignore What a Vulnerability Is: Creation Bias http://blog.risk.io/2013/04/data-fundamentalism/ !
Jerico/Sushidude @ BlackHat https://www.blackhat.com/us-13/briefings.html#Martin
!
Luca Allodi - CVSS DDOS http://disi.unitn.it/~allodi/allodi-12-badgers.pdf
![Page 9: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/9.jpg)
F1: Data FundamentalismSince 2006 Vulnerabilities have declined by 26 percent.” http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf !
!
The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ” http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf
![Page 10: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/10.jpg)
FAIL 2: A Priori Modeling“Following up my previous email, I have tweaked my equation to try to achieve better separation between adjacent scores and to have CCC have a perfect (storm) 10 score...There is probably a way to optimize the problem numerically, but doing trial and error gives one plausible set of parameters...except that the scores of 9.21 and 9.54 are still too close together. I can adjust x.3 and x.7 to get a better separation . . .”
![Page 11: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/11.jpg)
F3: Logical InconsistencyTemporal Scores Hurt Decision Making
Report Confidence is Useless
Base Rate Fallacy
![Page 12: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/12.jpg)
F4: Stochastic Ignorance
Attackers Change Tactics Daily
![Page 13: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/13.jpg)
F4: Stochastic Ignorance
![Page 14: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/14.jpg)
Empirical Failures of CVSSObjective: Remediate the riskiest vulnerabilities
Constraint: Can’t measure impact/priority
Need:
MOAR DATA!!!
![Page 15: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/15.jpg)
Repair the Vulnerability
![Page 16: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/16.jpg)
I Love It When You Call Me Big Data50,000,000 Live Vulnerabilities
1,500,000 Assets
2,000 Organizations
![Page 17: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/17.jpg)
I Love It When You Call Me Big Data
3,000,000 Breaches
![Page 18: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/18.jpg)
Baseline AllthethingsProbability (You Will Be Breached On A Particular Open Vulnerability)?
=(Open Vulnerabilities | Breaches Occurred On Their CVE) /(Total Open Vulnerabilities)
2%
![Page 19: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/19.jpg)
Probability A Vuln Having Property X Has Observed Breaches
RANDOM VULN
CVSS 10
CVSS 9
CVSS 8
CVSS 6
CVSS 7
CVSS 5
CVSS 4
Has Patch
0.000 0.010 0.020 0.030 0.040
![Page 20: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/20.jpg)
PART 2: !
FIX WHAT MATTERS
![Page 21: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/21.jpg)
Proper Framework
Know which vulnerabilities put you most at risk.
![Page 22: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/22.jpg)
![Page 23: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/23.jpg)
![Page 24: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/24.jpg)
![Page 25: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/25.jpg)
![Page 26: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/26.jpg)
![Page 27: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/27.jpg)
![Page 28: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/28.jpg)
Counterterrorism
Known Groups
Surveillance
Threat Intel, Analysts
Targets, Layouts
Past Incidents, Close Calls
![Page 29: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/29.jpg)
![Page 30: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/30.jpg)
Uh, Sports?
Opposing Teams, Specific Players
Gameplay
Scouting Reports, Gametape
Roster, Player Skills
Learning from Losing
![Page 31: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/31.jpg)
InfoSec?
![Page 32: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/32.jpg)
Defend Like You’ve Done It Before
Groups, Motivations
Exploits
Vulnerability Definitions
Asset Topology, Actual Vulns on System
Learning from Breaches
![Page 33: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/33.jpg)
Work With What You’ve Got:
Akamai, Safenet
ExploitDB, Metasploit
NVD, MITRE
![Page 34: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/34.jpg)
Bad Alternatives
Why Don’t I Just Patch The Important Assets?
![Page 35: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/35.jpg)
Good Alternatives
![Page 36: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/36.jpg)
Probability A Vuln Having Property X Has Observed Breaches
Random Vuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0 0.1 0.2 0.2 0.3
![Page 37: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/37.jpg)
Data Is Everything And Everything Is Data
![Page 38: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/38.jpg)
Data Is Everything And Everything Is Data
![Page 39: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/39.jpg)
Be Better Than The Gap
![Page 40: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/40.jpg)
Data is Everything and Everything is DataSpray and Pray = 2%
CVSS 10 = 4%
Metasploit and Exploit DB = 30%
![Page 41: BSides 2014, Fix What Matters: Why CVSS Sucks & How To Do It Better, by Data Scientist Michael Roytman](https://reader034.vdocument.in/reader034/viewer/2022042714/5537f93b550346e93a8b465a/html5/thumbnails/41.jpg)
Holler!www.risk.io@mroytman