bsidessf 2014 fix what matters
DESCRIPTION
Why using CVSS for vulnerability management is nuts. How to fix the vulnerabilities that truly matter, and how to create and measure an effective security practice.TRANSCRIPT
![Page 1: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/1.jpg)
Fix What Matters: !
Why CVSS Sucks And How To
Do Better
![Page 2: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/2.jpg)
Once Jailbroke an Iphone 3G
Michael Roytman
Proud Owner of Remote Controlled AirplaneRecently a Naive Grad Student
Data Scientist, Risk I/ODoes Not Wake Up Before 11 CST
qualifications:
![Page 3: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/3.jpg)
PART 1: !
YOU SUCK AT YOUR JOB
!
(and don’t even know it yet)
![Page 4: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/4.jpg)
Why Are We Here?
Empirical Failures of CVSSProper Remediation Frameworks (Yeah, they exist)
CVSS SUCKS
Analytical Failures of CVSS
(+Data Driven Alternatives)
![Page 5: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/5.jpg)
Remove the Threat
RemediationAccept the Risk
Repair the Vulnerability
![Page 6: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/6.jpg)
C(ommon) V(ulnerability) S(coring) S(ystem)
“CVSS is designed to rank information system vulnerabilities”
Exploitability/Temporal (Likelihood)
Impact/Environmental (Severity)
The Good: Open, Standardized Scores
![Page 7: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/7.jpg)
“It is a capital mistake to theorize before one has data.
!
!
!
Insensibly, one begins to twist facts to suit theories, instead of
theories to suit facts.”
![Page 8: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/8.jpg)
FAIL: Data FundamentalismDon’t Ignore What a Vulnerability Is: Creation Bias http://blog.risk.io/2013/04/data-fundamentalism/ !
Jerico/Sushidude @ BlackHat https://www.blackhat.com/us-13/briefings.html#Martin
!
Luca Allodi - CVSS DDOS http://disi.unitn.it/~allodi/allodi-12-badgers.pdf
![Page 9: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/9.jpg)
F1: Data FundamentalismSince 2006 Vulnerabilities have declined by 26 percent.” http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf !
!
The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ” http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf
![Page 10: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/10.jpg)
FAIL 2: A Priori Modeling“Following up my previous email, I have tweaked my equation to try to achieve better separation between adjacent scores and to have CCC have a perfect (storm) 10 score...There is probably a way to optimize the problem numerically, but doing trial and error gives one plausible set of parameters...except that the scores of 9.21 and 9.54 are still too close together. I can adjust x.3 and x.7 to get a better separation . . .”
![Page 11: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/11.jpg)
F3: Logical InconsistencyTemporal Scores Hurt Decision Making
Report Confidence is Useless
Base Rate Fallacy
![Page 12: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/12.jpg)
F4: Stochastic Ignorance
Attackers Change Tactics Daily
![Page 13: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/13.jpg)
F4: Stochastic Ignorance
![Page 14: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/14.jpg)
Empirical Failures of CVSSObjective: Remediate the riskiest vulnerabilities
Constraint: Can’t measure impact/priority
Need:
MOAR DATA!!!
![Page 15: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/15.jpg)
Repair the Vulnerability
![Page 16: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/16.jpg)
I Love It When You Call Me Big Data50,000,000 Live Vulnerabilities
1,500,000 Assets
2,000 Organizations
![Page 17: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/17.jpg)
I Love It When You Call Me Big Data
3,000,000 Breaches
![Page 18: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/18.jpg)
Baseline AllthethingsProbability (You Will Be Breached On A Particular Open Vulnerability)?
=(Open Vulnerabilities | Breaches Occurred On Their CVE) /(Total Open Vulnerabilities)
2%
![Page 19: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/19.jpg)
Probability A Vuln Having Property X Has Observed Breaches
RANDOM VULN
CVSS 10
CVSS 9
CVSS 8
CVSS 6
CVSS 7
CVSS 5
CVSS 4
Has Patch
0.000 0.010 0.020 0.030 0.040
![Page 20: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/20.jpg)
PART 2: !
FIX WHAT MATTERS
![Page 21: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/21.jpg)
Proper Framework
Know which vulnerabilities put you most at risk.
![Page 22: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/22.jpg)
![Page 23: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/23.jpg)
![Page 24: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/24.jpg)
![Page 25: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/25.jpg)
![Page 26: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/26.jpg)
![Page 27: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/27.jpg)
![Page 28: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/28.jpg)
Counterterrorism
Known Groups
Surveillance
Threat Intel, Analysts
Targets, Layouts
Past Incidents, Close Calls
![Page 29: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/29.jpg)
![Page 30: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/30.jpg)
Uh, Sports?
Opposing Teams, Specific Players
Gameplay
Scouting Reports, Gametape
Roster, Player Skills
Learning from Losing
![Page 31: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/31.jpg)
InfoSec?
![Page 32: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/32.jpg)
Defend Like You’ve Done It Before
Groups, Motivations
Exploits
Vulnerability Definitions
Asset Topology, Actual Vulns on System
Learning from Breaches
![Page 33: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/33.jpg)
Work With What You’ve Got:
Akamai, Safenet
ExploitDB, Metasploit
NVD, MITRE
![Page 34: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/34.jpg)
Bad Alternatives
Why Don’t I Just Patch The Important Assets?
![Page 35: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/35.jpg)
Good Alternatives
![Page 36: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/36.jpg)
Probability A Vuln Having Property X Has Observed Breaches
Random Vuln
CVSS 10
Exploit DB
Metasploit
MSP+EDB
0.0 0.1 0.2 0.2 0.3
![Page 37: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/37.jpg)
Data Is Everything And Everything Is Data
![Page 38: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/38.jpg)
Data Is Everything And Everything Is Data
![Page 39: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/39.jpg)
Be Better Than The Gap
![Page 40: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/40.jpg)
Data is Everything and Everything is DataSpray and Pray = 2%
CVSS 10 = 4%
Metasploit and Exploit DB = 30%
![Page 41: BsidesSF 2014 Fix What Matters](https://reader037.vdocument.in/reader037/viewer/2022110118/554beb7eb4c9055a368b4e7c/html5/thumbnails/41.jpg)
Holler!www.risk.io@mroytman