secure360 on risk
DESCRIPTION
Jay Jacobs & I co presented on Risk and Risk Management at the wonderful Secure360 conference this springTRANSCRIPT
Challenging Conventional Wisdom: A New Approach to Risk ManagementAlex HuttonJay Jacobs
What’s this about?
We think you’re getting bad information!
We think our industry can do better!
We think this will make us “more secure!”
Security is now so essential a concern that we can no longer use adjectives and adverbs but must instead use numbers. – Dan Geer
How are you making decisions now?
What’s the quality of those decisions?
Effective Decisions need quality data, models, execution
Our vendors and standards aren’t
helping us:-(
hey, why are you getting lousy information from standards and vendors?
hey, why are you getting lousy information from standards and vendors?
The science of information security & risk management is hard
1. Pseudo Science & Proto Science
2. Models & Data
3. Complexity
hey, why are you getting lousy information from standards and vendors?
The science of information security & risk management is hard
1. Pseudo Science & Proto Science
2. Models & Data
3. Complexity
State of the Industry (a)(Thomas Kuhn is way smarter than we are)
proto-science
somewhat random fact gathering (mainly of readily accessible data)
a“morass”of interesting, trivial, irrelevant observations
a variety of theories (that are spawned from what he calls philosophical speculation) that provide little guidance to data gathering
State of the Industry (b)At our present skill in measurement of security, we generally have an ordinal scale at best, not an interval scale and certainly not a ratio scale. In plain terms, this means we can say whether X is better than Y but how much better and compared to what is not so easy. – More from Dan Geer
If Science is based on inductive observations to derive meaning and understanding and measurement on quality (ratio) scales, how about InfoSec?
Where do we sit in the family of sciences?
We’re the Crazy Uncle with tinfoil hat antennae used to talk to the space aliens of Regulus V, has 47 cats, and who too frequently (but benignly) forgets to wear pants.
Take, for example, CVSS
“the Base Equation multiplies Impact by 0.6 and Exploitability by 0.4”
= ShinyJet Engine X Peanut Butter
adding onewilly-nilly doesn’t suddenly transformordinal rankings into ratio values.
decimals aren’t magic.
hey, why are you getting lousy information from standards and vendors?
The science of information security & risk management is hard
1. Pseudo Science & Proto Science
2. Models & Data
3. Complexity
20
Data must exist in order to feed our models...
... but creating the right models are dependent on understanding what data is useful!
Data, Models, Execution: Garbage in-Garbage Out
Data, Models, Execution: Treat Data Poorly
Data, Models, Execution: Adapting to Situations
hey, why are you getting lousy information from standards and vendors?
The science of information security & risk management is hard
1. Pseudo Science & Proto Science
2. Models & Data
3. Complexity
These “risk” statements you’re making...
I don’t think you’re doing it right.
- (Chillin’ Friederich Hayek)
“Given Newton's laws and the current position
and velocity of every particle in the universe,
it was possible, in principle, to predict
everything for all time.”
-- Simon-Pierre LaPlace, 1814
A Comforting Thought...
8
4 4
2 2 2 2
Reductionism
8
4 4
2 2 2 2
Functionalism
?
?
Asset
Comp. Comp.
Sub. Sub.
Attribute
Attribute
Attribute
Attribute
Reductionism
Functionalism
Awww man......even if it were the case that the natural laws had no longer any secret for us, we could still only know the initial situation approximately. ... small differences in the initial conditions produce very great ones in the final phenomenon. A small error in the former will produce an enormous error in the latter. Prediction becomes impossible...
-- Henri Poincare, 1887
13
5 6
2 2 2 2
Holism
Complexity non-linear
Systems Approach
Complex systems contain changing mixtures of failures latent within them.
The complexity of these systems makes it impossible for them to run without multiple flaws being present.
... individually insufficient to cause failure
...failures change constantly because of changing technology, work organization, and efforts to eradicate failures.
Complex systems run in degraded mode.
“How Complex Systems Fail” - Richard Cook
Security is a characteristic of systems and not of their components
Security is an emergent property of systems; it does not reside in a person, device or department of an organization or system.
... it is not a feature that is separate from the other components of the system.
...the state of Security in any system isalways dynamic
“How Complex Systems Fail” - Richard Cook
We may want to rethink our approach.
36
Overcoming the problem
• Medicine uses an “Evidence-Based” approach to solving problems in the complex system that is the body.
• Dr. Peter Tippett (MD, PhD) applies Evidence-Based principles to Information Security.
threat landscape
asset landscape
impact landscape
controls landscape
risk
Suggested context:Capability to manage(skills, resources, decision quality…)
What to study: Sources of Knowledge
How: Data Quality in Evidence-Based Practice
Evidence level D Evidence level C Evidence level B Evidence level A
Evidence level A“Expert opinion without explicit cri8cal appraisal, or based on physiology, bench research or first principles.”
Case-‐series study or extrapola8ons from level B studies.
Consistent Retrospec8ve Cohort, Exploratory Cohort, Ecological Study, Outcomes Research, case-‐control study; or extrapola8ons from level A studies.
Consistent Randomized Controlled Clinical Trial, cohort study, all or none, clinical decision rule validated in different popula8ons.
beNer
Evidence-Based Risk ManagementState of Nature State of Knowledge State of Wisdom
Evidence level D Lists Feeling like we’ve done something
Evidence level C Simple derived values with ad-hoc modeling
Outcomes with ad-hoc deductive selections
Evidence level B Formal Modeling Decision making constructs
Evidence level A
State of Nature State of Knowledge State of Wisdom
Evidence level D Lists Feeling like we’ve done something
Evidence level C Simple derived values with ad-hoc modeling
Outcomes with ad-hoc deductive selections
Evidence level B Formal Modeling Decision making constructs
Evidence level A
Evidence-Based Risk Management
State of Nature State of Knowledge State of Wisdom
Evidence level D Lists Feeling like we’ve done something
Evidence level C Simple derived values with ad-hoc modeling
Outcomes with ad-hoc deductive selections
Evidence level B Formal Modeling Decision making constructs
Evidence level A
You are here
Evidence-Based Risk Management
So How Do We Change?
DataModels…
Standards
START WITH THE
OUTCOMES!
Two True Security Outcomes:
Success and Failure
Knowing Success in InfoSec is hard
- Known Success (anti-Threat ops)- Unknown success (controls work
without us knowing)- Dumb luck (We’re not targeted, but our
neighbor is)
Getting the outcomes:Success
Getting the outcomes:Success
stronger processes result in fewer availability incidents
Getting the outcomes- Successes:
- Existences of processes- Operational (performance) metrics- Maturity ratings
WHAT WE WANT ARE PATTERNS!
Knowing Failure is (somewhat) easier
Getting The Outcomes: Failures
VERIS | Verizon Enterprise Risk and Information Sharing
VERIS takes the incident narrative and creates metrics (risk determinants)
A free (as in beer*) framework created for metrics, modeling, and compara8ve analy8cs.
A security incident (or threat scenario) is modeled as a series of events. Every event is comprised of the following 4 A’s:
Agent: Whose acLons affected the asset
AcLon: What acLons affected the asset
Asset: Which assets were affected
AOribute: How the asset was affected
VERIS | Verizon Enterprise Risk and Information Sharing
INCIDENT REPORT“An attacker from a Russian IP address
initiated multiple SQL injection attacks against a public-facing web application. They were able to introduce keyloggers and network sniffers onto internal systems. The keyloggers captured several domain credentials which the attackers used to further infiltrate the corporate network. The packet sniffers captured data for several months which the attacker periodically returned to collect…”
VERIS takes this :
and…
…and translates it to this…Event 1Agent: External (Org crime)Action: Hacking (SQLi)Asset: Server (Web server, Database)Attribute: IntegrityEvent 2Agent: External (Org crime)Action: Malware (Keylogger)Asset: Server (Web server)Attribute: ConfidentialityEvent 3Agent: External (Org crime)Action: Hacking (Use of stolen creds)Asset: Server, Network (multiple)Attribute: Confidentiality, IntegrityEvent 4…
1 2 3 4> > > >
patterns!
√∫∑
Framework
Models Data=
∩
Framework
Data
√∫∑Models=
∩
Framework
Data
Data √∫∑Models=
∩Process
ProcessProcess
Process
Using your metrics program
- Identify & Measure your processes- Identify & Measure your failures- Get into loss factors (ABC)- Share data- Support data sharing efforts
Bring it Home: your metrics program
Bring it Home: your metrics programor
Bring it Home: your metrics programorThe Amazing Technicolor Scorecard
Priority #1: no more surrogate data
Priority #1: (meaning) no more risk analysts*
Priority #1: (really) create data analysts
Data analysts need to focus on quality data, models, execution
State of Nature State of Knowledge State of Wisdom
Evidence level D Lists Feeling like we’ve done something
Evidence level C Simple derived values with ad-hoc modeling
Outcomes with ad-hoc deductive selections
Evidence level B Formal Modeling Decision making constructs
Evidence level A
Evidence-Based Risk Management
threat landscape
risk
A balanced scorecard of sorts
asset landscape
impact landscape
controls landscape
Where to look? The Two True Security Outcomes:
Success and Failure
Failures:threat landscape
asset landscape
impact landscape
controls landscape
incidents, red/blue team
vulnerabilities, misconfigurations, unknowns...
gaps in coverage, known lack of effectiveness, known underskilled/utilized...
Cost-Based Accounting around incidents, cost of operations, etc...
Successes:threat landscape
asset landscape
impact landscape
controls landscape
intel, red/blue teams, SIEM
vulnerabilities, misconfigurations, unknowns, skills, training
positive threat outcomes (tOps), skills, training
ROI? ROSI? (ducks to avoid tomatoes)
What to look? Two types of data to find:
Focus initially on Visibility, then look to find Variability.
How to look? The GQM Approach:
For each “where” for each “what” use the following “how”
How to look? The GQM Approach:
For each “where” for each “what”, start by using GQM as “how.”
Goal, Question, Metric
Conceptual level (goal) goals defined for an object for a variety of reasons, with respect to various models, from various points of view.
Operational level (question)
questions are used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal.
Quantitative level (metric)
metrics, based on the models, is associated with every question in order to answer it in a measurable way.
Victor Basili
The Book You Should Buy(Jay & Alex aren’t getting a kickback, in case you’re wondering)
GQM for Fun & Profit
Goals establishwhat we want to accomplish.
Questions help us understand how to meet the goal. They address context.
Metrics identify the measurements that are needed to answer the questions.
Goal 1 Goal 2
Q1 Q2 Q3 Q4 Q5
M1 M2 M3 M4 M5 M6 M7
GQM for Fun & Profit
Execution
Models
Data
Goal 1 Goal 2
Q1 Q2 Q3 Q4 Q5
M1 M2 M3 M4 M5 M6 M7
data about defined success and failures
models of assets, controls, threats contributing to impact
execution by data analysts...Feeding standards, audits and governance
Using your metrics program
- Identify & Measure your processes- Identify & Measure your failures- Get into loss factors (ABC)- Share data- Support data sharing efforts
Using your metrics program
- Identify & Measure your processes- Identify & Measure your failures- Get into loss factors (ABC)- Share data- Support data sharing efforts
Security is now so essential a concern that we can no longer use adjectives and adverbs but must instead use numbers. – Dan Geer
Questions?Jay Jacobs@[email protected]
Alex Hutton@[email protected]
threat landscape
asset landscape
impact landscape
controls landscape
risk
Prioritize
De-prioritize
Approaching the system as a system
threat landscape
asset landscape
impact landscape
controls landscape
risk
Suggested context:Capability to manage(skills, resources, decision quality…)
Data Sharing:
- Sources:- Qualify this Intel according to
framework- Treat with appropriate data quality
listings (let models shape the certainty)
Get Into Accounting
- Use existing models that take advantage of accounting concepts (ABC) to Talk to the LOBs
Using your metrics program
- Identify & Measure your processes- Identify & Measure your failures- Share data- Support data sharing efforts- Get into loss factors (ABC)
Challenging Conventional Wisdom
Conventional Wisdom may not be wrong- Question current practices - Seek Evidence and Feedback