deepsec 2014 - the measured cso
Post on 15-Jul-2015
592 Views
Preview:
TRANSCRIPT
THE MEASURED CSOALEX HUTTON -‐ A TOO BIG TO FAIL BANK
@ALEXHUTTON
SECTION 1: BACKGROUND Who am I? What is this topic? Where are we? How did we get here?
SECTION 2: ON THE ROLE OF THE CSO What is a CSO? What do they do? What is success? How do they get there?
SECTION 3: BECOMING MEASURED What does that mean? What do we need? How do we do it? Where shall we go?
SECTION 1: BACKGROUND Who am I? What is this topic? Where are we? How did we get here?
SECTION 2: ON THE ROLE OF THE CSO What is a CSO? What do they do? What is success? How do they get there?
SECTION 3: BECOMING MEASURED What does that mean? What do we need? How do we do it? Where shall we go?
SECTION 1: BACKGROUND Who am I? What is this topic? Where are we? How did we get here?
SECTION 2: ON THE ROLE OF THE CSO What is a CSO? What do they do? What is success? How do they get there?
SECTION 3: BECOMING MEASURED What does that mean? What do we need? How do we do it? Where shall we go?
SECTION 1: BACKGROUND Who am I? What is this topic? Where are we? How did we get here?
1.1 WHO AM I
• Security Engineer
• Security Product Management
• E-Commerce Site Design / Manager
• Risk Consultant
• OCTAVE / NIST
• FAIR
• Verizon DBIR
• IANS Faculty
• Director, Operations / Technology Risk
• Director, Information Security
1.1 WHO AM I
1.2 WHAT IS THIS TOPIC
“…when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be.”
William Thomson, 1st Baron Kelvin & Measurement Badass
The Journey Towards Knowledge (and therefore, security)
1.2 WHAT IS THIS TOPIC
WHERE ARE WE (OUR INDUSTRY)
Security is now so essential a concern that
we can no longer use adjectives and adverbs
but must instead use numbers.
Dan Geer, Security Badass
Unfortunately…
Science is based on inductive observations to derive meaning and understanding and measurement on quality (ratio) scales, so what about InfoSec?
Where do we sit in the family of sciences?
We’re the Crazy Uncle with tinfoil hat antennae used to talk to the space aliens of Regulus V, has 47 cats, and who too frequently (but benignly) forgets to wear pants.
Take, for example, CVSS
“the Base Equation multiplies Impact by 0.6 and Exploitability by 0.4”
= ShinyJet Engine X Peanut Butter
“the Base Equation multiplies Impact by 0.6 and Exploitability by 0.4”
20
adding onewilly-nilly doesn’t suddenly transformordinal rankings into ratio values.
decimals aren’t magic.
At our present skill in measurement of security, we
generally have an ordinal scale at best, not an interval scale
and certainly not a ratio scale. In plain terms, this means we
can say whether X is better than Y but how much better and
compared to what is not so easy.
– Again, Baddss Dan Geer
State of the Industry- proto-science - somewhat random fact
gathering (mainly of readily accessible data)
- a“morass”of interesting, trivial, irrelevant observations
- a variety of theories (that are spawned from what he calls philosophical speculation) that provide little guidance to data gatheringThomas Kuhn Philosophy of Science Badass
1.3 HOW DID WE GET HERE
1.3 HOW DID WE GET HERE
The tragedy of two mistakes
FIRST MISTAKE: LIMITING OURSELVES(security is an engineering issue?)
• OSI Model (original version)
• OSI Model (SOA Remix)
• OSI Model (Mika’s 12” Extended Dance Version)
10: Religion Operator Layer
SECOND MISTAKE: BLIND LEADING THE BLIND
BLIND MAN 1: THE FUD FACTORY
FUD FACTORY EXAMPLE - MOBILE VS WEB
Google Trend: Web Security Mobile Malware
#RSAC
36
Clustering of over 5,000 incidents
Espionage
Point of Sale
Skimming Devices
Theft/Loss
Error
EmployeeMisuse
Web Applications
DBIR Top Patterns:
Web Only:
Web Applications
In FinServ vs. All Industries
DBIR Global Representation of Assets in Cases:
DBIR Global Representation of Assets in Cases:
NHTCU investigation into groups using mobile malware showed that in less than a year’s time, five variations of mobile malware for one specific bank could be detected. Modest estimates suggest that criminals gained around €50,000 per week using this specific form of mobile malware, harvesting over 4,000 user credentials from 8,500 infected bank customers in just a few months. Mobile malware does not move the needle in our stats as we focus on organizational security incidents as opposed to consumer device compromises.
DBIR Global Representation of Assets in Cases:
NHTCU investigation into groups using mobile malware showed that in less than a year’s time, five variations of mobile malware for one specific bank could be detected. Modest estimates suggest that criminals gained around €50,000 per week using this specific form of mobile malware, harvesting over 4,000 user credentials from 8,500 infected bank customers in just a few months. Mobile malware does not move the needle in our stats as we focus on organizational security incidents as opposed to consumer device compromises.
BLIND MAN 2: THE ACCOUNTING-CONSULTANCY INDUSTRIAL COMPLEX
Complex (adaptive)Systemsa system composed of interconnected parts that as a whole exhibit one or more properties not obvious from the properties of the individual parts
These “risk” statements you’re making...
I don’t think you’re doing it right.
- (Chillin’ Friederich Hayek)
BLIND MAN 3: OUR BROKEN MODELS
“the Base Equation multiplies Impact by 0.6 and Exploitability by 0.4”
ROYTMAN: ON VULNERABILITIES
ROYTMAN: ON VULNERABILITIES
A CSO MUST BECOME “MEASURED” TO ESCAPE THE MISTAKES OF THE PAST AND PUSH INTO THE FUTURE
SECTION 2: ON THE ROLE OF THE CSO What is a CSO? What do they do? What is success? How do they get there?
• What Is a CISO (throne of blood image
WHAT IS A CSO
• What Is a CISO (throne of blood image
WHAT IS A MEASURED CSO
W.E. DEMING
Father of Total Quality Management and inspiration that drove the Japanese “post-war economic miracle.”
IT WAS NO MIRACLE. What Deming taught the Japanese was “management by fact.”
• Improvements to the system are never ending.
• The only people who really know where the real potentials for improvement are the workers.
• The system is always changing.
• There are countless ways for the system to go wrong.
• Statistics (metrics) are used to focus the conversation on fact and improvement
• Goals for quality are cross-silo
• Theories for improvements are implemented and tested.
• The management uses the workers as essential "instruments" in understanding what is.
A MEASURED CSO:
• Relies on metrics, data, intel for good decisions,
• Invests in improvements to People, Process and Technology,
• Puts innovation for improvements to the system (improvements = security, cost) in the hands of the operator,
• Ensures that there is a feedback loop for effectiveness initiatives, and
• Works tirelessly within the bureaucracy to improve all aspects of the system.
THE MEASURED CSO’S MISSION:
• To provide the best and least-cost security for shareholders, and continuity of employment for his workers.
• We, as an industry, know that “best” and”least-cost” are not necessarily contradictory
• We also have a HUGE continuity issue
THE MEASURED CSO USES METRICS TO IMPROVE THE SYSTEM.
WHAT IS THAT SYSTEM - That which Defends (Detects, Responds, & Prevents).
THE MEASURED CSO USES METRICS TO:
• Develop and improve the People, Process, and Technology to Defend
• Plan / Build / Manage those defenses
THE SYSTEMS FOR DEVELOPING METRICS ARE MORE IMPORTANT THAN THE SYSTEMS OF DOGMA THAT DEFINE “STANDARDS” OF OPERATION.
THE SYSTEMS FOR DEVELOPING METRICS ARE MORE IMPORTANT THAN THE SYSTEMS OF DOGMA THAT DEFINE “STANDARDS” OF OPERATION.
Sorry, ISACA
THE SYSTEMS FOR DEVELOPING METRICS ARE MORE IMPORTANT THAN THE SYSTEMS OF DOGMA THAT DEFINE “STANDARDS” OF OPERATION.
• There are two systems which the CSO must manage across (at least 4 audiences)
• Those that support “defend”
• Those that support Plan/Build/Manage
MEASURED CSO SYSTEM 1: THE METRICS AND MODELS THAT “DEFEND”
EPIDEMIOLOGY
EPIDEMIOLOGYRisk Factors (Determinants) Variables associated with increased frequency of event.
Risk Markers Variable that is quantitatively associated with a disease or other outcome, but direct alteration of the risk marker does not necessarily alter the risk of the outcome.
Correlation vs. Causation Risk factors or determinants are correlational and not necessarily causal, because correlation does not prove causation.
EPIDEMIOLOGYRisk Factors (Determinants) Variables associated with increased frequency of event.
Risk Markers Variable that is quantitatively associated with a disease or other outcome, but direct alteration of the risk marker does not necessarily alter the risk of the outcome.
Correlation vs. Causation -Risk factors or determinants are correlational and not necessarily causal, because correlation does not prove causation.
THE MEANS TO FIND PATTERNS
Example of a medical approach:Dr. Peter Tippett & Verizon DBIR
A security incident (or threat scenario) is modeled as a series of events. Every event is comprised of the following 4 A’s:
Agent: Whose actions affected the asset Action: What actions affected the asset Asset: Which assets were affected Attribute: How the asset was affected
VERIS (Vocabulary for Event Recording & Incident Sharing)
70
72
Object-Oriented Modeling
VERIS (Vocabulary for Event Recording & Incident Sharing)
73
1 2 3 4 5 >" >" >" >"Incident as a chain of events >"
Object-Oriented Modeling
VERIS (Vocabulary for Event Recording & Incident Sharing)
74
1 2 3 4 5 >" >" >" >"Incident as a chain of events >"
A “Pattern”
VERIS: Classification of Events by Risk Factor
Complex System?
VERIS FOUND PATTERNS!
#RSAC
36
Clustering of over 5,000 incidents
Espionage
Point of Sale
Skimming Devices
Theft/Loss
Error
EmployeeMisuse
Web Applications
DBIR Top Patterns:
THE KEY TO THE MEASURED CSO SYSTEM 1: FRAMEWORK, DATA, MODELS
√∫∑
Framework
Models Data=
∩
VERIS+
actor information
asset information
impact information
controls information
risk
Classifying sets of security information
√∫∑
Framework
Models Data=
∩Data Warehousing+
82
Apache Storm
83
Data MapReduce Process Analytics & Reporting
Threat Intel FeedsControl DataControl LogsSystem Logs
Event History & Loss Loss Distribu8on Dev. B.I.A.
Control DataControl LogsSystem Logs
Configuration DataVulnerability DataHR InformationProcess Behaviors
XMLCSVEDI
LOGSQL
JSONText
BinaryObjects
create map
reduce
TraditionalRDBMSSystems
Workflow
Analytics
Reporting
Models suggesting IO
C= true
88
1 2 3 4 5 >" >" >" >"Incident as a chain of events >"
89
1 2 3 4 5 >" >" >" >"Incident as a chain of events >" X X X
90
Example of data enrichment:
Asset Intel : Vendor-owned SaaS application
√∫∑
Framework
Models Data=
∩
MEASURED CSO SYSTEM 1: THE METRICS AND MODELS THAT “DEFEND” AGAINST THREAT PATTERNS.
(real and anticipated or forecasted)
MEASURED CSO SYSTEM 2: THE METRICS NEEDED TO PLAN/BUILD/MANAGE SYSTEMS (OPERATIONS)
THE MEASURED CSO MUST ALSO INCLUDE A KEEN UNDERSTANDING AND PARTNERSHIP WITH IT OPERATIONS
THE MICROMORT A one in a million chance of death Ronald A. Howard
Activities that increase the death risk by roughly one micromort, and their associated cause of death (wikipedia):
Traveling 6 miles by motorbike (accident) Traveling 17 miles by walking (accident) Traveling 10 miles by bicycle (accident) Traveling 230 miles (370 km) by car (accident) Traveling 1000 miles (1600 km) by jet (accident) Traveling 6000 miles (9656 km) by train (accident) Traveling 12,000 miles (19,000 km) by jet in the United States (terrorism) Increase in death risk for other activities on a per event basis:
Hang gliding – 8 micromorts per trip Ecstacy (MDMA) – 0.5 micromorts per tablet (most cases involve other drugs)
Modern Risk Management is not only bad at describing risk, but it also is focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)
Modern Risk Management is not only bad at describing risk, but it also is focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)
Ecstacy (MDMA) – 0.5 micromorts per tablet (most cases involve other drugs)
Modern Risk Management is not only bad at describing risk, but it also is focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)
Modern Risk Management is not only bad at describing risk, but it also is focused on reporting its own version “micromorts”Inefficiently.
Traveling 10 miles by bicycle (accident)
The Measured CSO must know where IT is overweight, smoking ecstasy, while riding a rocket-powered bicycle on the railing of a bridge.
DATA: VISIBLE OPS FOR SECURITY
104
Example of data enrichment:
Asset Intel : Vendor-owned SaaS application
SECTION 3: BECOMING MEASURED What does that mean? What do we need? How do we do it? Where shall we go?
106
MOST METRICS PROGRAMS
If we consider a single metric as a building block
108
It should be used by the CSO to paint a picture of the security program
109
Whose context is the whole of IT.
110
But because we gather what is most readily available - most metrics programs look like my living room.
How does the measured CSO get context?
GOAL, QUESTION, METRICConceptual level (goal) goals defined for an object for a variety of reasons, with respect to various models, from various points of view.
Operational level (question)
questions are used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal.
Quantitative level (metric)
metrics, based on the models, is associated with every question in order to answer it in a measurable way.
Victor Basili
GQM FOR FUN & PROFIT
Goals establishwhat we want to accomplish.
Questions help us understand how to meet the goal. They address context.
Metrics identify the measurements that are needed to answer the questions.
Goal 1 Goal 2
Q1 Q2 Q3 Q4 Q5
M1 M2 M3 M4 M5 M6 M7
Execution
Models
Data
Goal 1 Goal 2
Q1 Q2 Q3 Q4 Q5
M1 M2 M3 M4 M5 M6 M7
GQM FOR FUN & PROFIT
GQM EXAMPLE: PATCH MANAGEMENT
Patching Scorecard
Goal 1: Comprehensive
Goal 2: Timely
Goal 3: Cost Efficient
GQM EXAMPLE: PATCH MANAGEMENT
Patching Scorecard
Goal 1: Comprehensive
Goal 2: Timely
Goal 3: Cost Efficient
% Coverage by Business Units
%Coverage by Asset category
%Coverage by Risk
Unix
Windows Server
DesktopOS
Components
Likelihood
Impact
Most Significant Failures
Repeat Offenders
By Asset Category
By Location (DMZ, Semi-Pub, Internal)
By Business Unit
By Asset Category
By Location (DMZ, Semi-Pub, Internal)
By Business Unit
GQM EXAMPLE: PATCH MANAGEMENT
Patching Scorecard
Goal 1: Comprehensive
Goal 2: Timely
Goal 3: Cost Efficient
What should our Priorities be for timeliness?
What is Policy for timeliness?
What other Considerations for Timeliness?
What is time to patch like for assets with worst Likelihoods?
What is time to patch like for assets with worst Impacts?
What % are Late by
What are our Repeat Offenders?
likelihood
Impact
by asset category
by business unit
by risk
UNIX
Windows Server
Desktop
likelihood
impact
GQM EXAMPLE: PATCH MANAGEMENT
Patching Scorecard
Goal 1: Comprehensive
Goal 2: Timely
Goal 3: Cost Efficient
Cost
Risk Reduction
Hour per Asset spent PatchingBy Asset Category
By Location (DMZ, Semi-Pub, Internal)
By Cost Per Hour
Hour per Asset, by ALE per Hour
Hour per asset category
GQM EXAMPLE: PATCH MANAGEMENT
• The Measured CSO creates a scorecard of KRI’s & KPI’s that Includes:
• Historical values
• “Triggers”
• “Thresholds”
(each of these?) aren’t perfect, but establish a hypothesis for testing & optimization.
Now you’re ready to come correct, my Bias!
- (Chillin’ Friederich Hayek)
MEASURED CSO FRAMEWORK FOR GQM: NIST CSF
NIST CSF
Identify
Protect
Detect
Respond
Recover
Asset Management
Business Environment
risk assessment
risk management strategy
Governance
Access Control
Awareness and Traininig
Data Security
Information Protection Processes and Procedures
Maintenance
Protective Technology
Anomalies and Events
Security Continuous Monitoring
Detection Processes
Response Planning
Response Communications
Response Analysis
Response Mitigation
Response Improvements
Recovery Planning
Improvements
Communications
SECTION 3: BECOMING MEASURED What does that mean? What do we need? How do we do it? Where shall we go?
√∫∑
Framework
Models Data=
∩
124
Example of data enrichment:
Asset Intel : Vendor-owned SaaS application
ETL AND STORE ALL THE THINGS!!!
126
Data MapReduce Process Analytics & Reporting
Threat Intel FeedsControl DataControl LogsSystem Logs
Event History & Loss Loss Distribu8on Dev. B.I.A.
Control DataControl LogsSystem Logs
Configuration DataVulnerability DataHR InformationProcess Behaviors
XMLCSVEDI
LOGSQL
JSONText
BinaryObjects
create map
reduce
TraditionalRDBMSSystems
Workflow
Analytics
Reporting
Models suggesting IO
C= true
“If you do not know how to ask the right question, you discover nothing.”
RESOURCESFOR GQM AND MICROMORTS -‐ WIKIPEDIA FOR DBIR DATA, THE VERIZON DBIR FOR DEMING QUOTES, THE WORKS OF MYRON TRIBUS:
http://www.qla.com.au/papersTribus/Oslo3.pdf http://www.unreasonable-‐learners.com/wp-‐content/uploads/2011/03/Germ-‐Theory-‐of-‐Management-‐Myron-‐Tribus1.pdf
http://www.qla.com.au/papersTribus/DEMINGS_.PDF
top related