strata san jose 2016 - reduce false positives in security
TRANSCRIPT
![Page 1: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/1.jpg)
Powerball Predictor
Photo Credit: Sean McGrath
Crystal ball tells me with 99% accuracy if a powerball prediction is a winner.
![Page 2: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/2.jpg)
Powerball Predictor
Photo Credit: Sean McGrath
● ~300 million samples.● ~ 3 million false positives.● 1 true positive.
![Page 3: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/3.jpg)
Powerball Predictor
Photo Credit: Sean McGrath
The overwhelming majority of tickets are not winners.
Failing to recognize this is falling victim to the base rate fallacy.
![Page 4: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/4.jpg)
Security Crystal Ball
Photo Credit: Sean McGrath
The overwhelming majority of log entries and data points do not represent fraud and intrusions.
Failing to recognize this is falling victim to the base rate fallacy.
![Page 5: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/5.jpg)
![Page 6: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/6.jpg)
FRAUD Intrusion
Detection
System
Source: MXLabs
![Page 7: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/7.jpg)
Base Rate Fallacy
![Page 8: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/8.jpg)
Why False Positives?
![Page 9: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/9.jpg)
Case Study: Outlier Detection
Using an outlier detection system to identify fraudsters within the environment.
![Page 10: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/10.jpg)
For a set of generating mechanisms find the unusual ones.
![Page 11: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/11.jpg)
Example Time Series
![Page 12: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/12.jpg)
Photo credit SuperCar-RoadTrip.fr under Creative Commons Attribution 2.0
Change in the data over time in unforeseen ways.
Concept Drift
![Page 13: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/13.jpg)
Solution: Feedback Loop
![Page 14: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/14.jpg)
Explicit Feedback Loop
Photo credit Alan Levine under Creative Commons Attribution 2.0
![Page 15: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/15.jpg)
Explicit Feedback Loop
Photo credit Alan Levine under Creative Commons Attribution 2.0
Implicit Feedback Loop
![Page 16: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/16.jpg)
Fraud: Takeaways
- Concept Drift is a shift in behavior.- Feedback combats concept drift.- Implicit Feedback > Explicit Feedback
![Page 17: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/17.jpg)
IDS: Anatomy of Successful Detection
![Page 18: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/18.jpg)
Context: Security Analyst
![Page 19: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/19.jpg)
Red team Kill Chain
![Page 20: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/20.jpg)
Blue team Kill Chain
![Page 21: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/21.jpg)
False positives: Lose Ability to Triage
![Page 22: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/22.jpg)
Fact: You cannot salvage a false positive with Contextual Info or Visualization
![Page 23: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/23.jpg)
What is a Successful detection?
Properties + Frameworks
![Page 24: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/24.jpg)
Successful detection captures Adversary TTP from Sensor data ignoring Expected activity
Source: @MSwannMSFT
![Page 25: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/25.jpg)
Properties of a Successful Detection
Adaptability
Credible
Interpretability
Actionable
![Page 26: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/26.jpg)
Basic Advanced
Less Useful
More U
seful
Sophistication of Algorithms
Usefulness of A
lerts
Secu
rity
Dom
ain
Kno
wle
dge
Framework for a Successful detection
![Page 27: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/27.jpg)
Basic Advanced
Less Useful
More U
seful
Sophistication of Algorithms
Usefulness of A
lerts
Secu
rity
Dom
ain
Kno
wle
dge
Outlier
![Page 28: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/28.jpg)
Basic Advanced
Less Useful
More U
seful
Sophistication of Algorithms
Usefulness of A
lerts
Secu
rity
Dom
ain
Kno
wle
dge
Outlier
Anomaly
Increase Complexity
![Page 29: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/29.jpg)
Basic Advanced
Less Useful
More U
seful
Sophistication of Algorithms
Usefulness of A
lerts
Secu
rity
Dom
ain
Kno
wle
dge
Outlier
AnomalyIncrease Complexity
Security InterestingAlerts
Incr
e ase
Dom
ain
Kno
wle
dgeSuccessful
Detections incorporate Domain Knowledge Alerts
![Page 30: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/30.jpg)
How to encode Domain Knowledge: Embrace Rules
• Business Heuristics to filter out the “Security interesting anomalies”
• Rules can take many forms: •TI feeds •IOCs, IOAs•TTPs
• Rules are awesome • Credible, Interpretable, Adaptable (to some
extent), Actionable!• Highest Precision • Highest Recall
![Page 31: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/31.jpg)
Three ways to combine ML and Rules
![Page 32: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/32.jpg)
Three Ways to combine Rules and ML 1.Above Machine Learning Systems
a.Business Heuristics to filter alerts i. “For account _foo_, only raise sev 2 alerts until March 28th, 2016”,
![Page 33: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/33.jpg)
Work by Dan Mace et. al, Microsoft
![Page 34: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/34.jpg)
2. Below Machine Learning Systemsa. Featurizations - “If IP address present in List of malicious IP dataset, flag 1”b. Utilizes Threat Intel feeds (Cymru, Virus total, FireEye)
![Page 35: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/35.jpg)
3: Combining Rules and Machine Learning together using Markov Logic Networks
Initial Ideas given by Vinod Nair, MSR
![Page 36: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/36.jpg)
Intuition
•Rules alone place a set of hard constraintson the set of possible worlds•Let’s make them soft constraints:When a world violates a formula,It becomes less probable, not impossible•Give each formula a weight(Higher weight ⇒ Stronger constraint)
Source: Lectures by Pedro Domingos
![Page 37: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/37.jpg)
Interactive logons from service accounts causes attack
Similar service accounts tend to have similar logon behavior
Example: Service Accounts
Domain Knowledge
![Page 38: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/38.jpg)
Example: Service Accounts
Encode as First Order Logic
![Page 39: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/39.jpg)
Example: Service Accounts
1.5
1.1
Example: Service Accounts
AssociateEach Rule With the Learned Weight
![Page 40: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/40.jpg)
Example: Service Accounts
1.5
1.1
Attack(A)
InteractiveLogon(A)
InteractiveLogon(B)
Attack(B)
Example: Service Accounts
Consider two service accounts: A,B
![Page 41: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/41.jpg)
Example: Service Accounts
1.5
1.1
Attack(A)
InteractiveLogon(A)
InteractiveLogon(B)
Attack(B)Similar(A,
B)
Similar(B,A)
Similar(A,A)
Similar(B,B)
![Page 42: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/42.jpg)
Example: Service Accounts
1.5
1.1
Attack(A)
InteractiveLogon(A)
InteractiveLogon(B)
Attack(B)Similar(A,
B)
Similar(B,A)
Similar(A,A)
Similar(B,B)
![Page 43: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/43.jpg)
Example: Service Accounts
1.5
1.1
Attack(A)
InteractiveLogon(A)
InteractiveLogon(B)
Attack(B)Similar(A,
B)
Similar(B,A)
Similar(A,A)
Similar(B,B)
![Page 44: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/44.jpg)
•How to learn the structure? •Begin with hand-coded rules•Use Inductive Logic Programming, but need to infer arbitrary clause
•How to learn the weights? •For generative learning, depend on pseudolikelihood
•Checkout Alchemy -- http://alchemy.cs.washington.edu/
![Page 45: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/45.jpg)
Call for Action - After the conference • One Week
•Review •@CodyRioux - IPython Notebook•@Ram_ssk - Follow Up material
•Think comprehensively about Rules
• One Month •Ask your data scientists to literature review section
•Implement the rules on TOP of ML systems
• One quarter•Implement a feedback system to capture training data
•Implement all TI feeds within an ML System
•Play with Alchemy
![Page 46: Strata San Jose 2016 - Reduce False Positives in Security](https://reader036.vdocument.in/reader036/viewer/2022070523/58ecfbb41a28abb6788b4571/html5/thumbnails/46.jpg)
Literature● The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection
(Alexsson, 1999)
● Enhancing Performance Prediction Robustness by Combining Analytical Modeling
and Machine Learning (Didona et al., 2015)
● Richardson, Matthew, and Pedro Domingos. "Markov logic networks."Machine
learning 62.1-2 (2006): 107-136.