rsa conference 2012 security metrics

28
Session ID: Session Classification: John D. Johnson John Deere Presenting Metrics to the Executive Team SEM-003 Intermediate

Upload: john-johnson

Post on 11-Nov-2014

1.927 views

Category:

Documents


5 download

DESCRIPTION

Presentation on delivering security metrics to executives for half day seminar at RSA Conference 2012.

TRANSCRIPT

Page 1: RSA Conference 2012 Security Metrics

Session ID: Session Classification:

John D. Johnson John Deere

Presenting Metrics to the Executive Team

SEM-003 Intermediate

Page 2: RSA Conference 2012 Security Metrics

Questions:

§  How do we define security metrics? §  How are security metrics useful? §  Where do get the information, and how do we

turn it into something meaningful? §  How do we present security metrics to our

management? §  Building a security metrics program §  Group Discussion: What works for you?

2  

Page 3: RSA Conference 2012 Security Metrics

Metrics In Real Life…

3  

Page 4: RSA Conference 2012 Security Metrics

Measurements & Metrics

§  Performance metrics measure how well an organization performs §  Drives process improvements and demonstrates

value-add §  Metrics can show how we compare to our peers

§  Metrics can help us break out of the cycle that comes from relying on products from vendors to rescue us from new threats: Detect à Report à Prioritize à Remediate

4  

Page 5: RSA Conference 2012 Security Metrics

Security Metrics

§  Make security metrics more meaningful to stakeholders §  We need to learn to ask the right questions, if our

results are going to be meaningful §  The best metrics are SMART: Specific, Measurable,

Attainable, Repeatable & Time-Dependent §  This is an inherently difficult problem

§  What is meaningful to stakeholders? §  Can we make metrics more quantitative? §  What can we measure? §  What are our peers doing?

5  

Page 6: RSA Conference 2012 Security Metrics

Motivations

§  Various Motivations for Developing Metrics §  Regulations - Compliance §  Audits (both internal and external) §  Money (security is rarely a profit center) §  Responding to new threats §  Enabling new technology and business processes §  Awareness: Making executives aware of trends

§  Example Compliance Metrics: §  Manager sign-off on access controls §  A&A control artifacts §  Audit reports/findings (number, severity, BU) §  Exception reporting/tracking §  PCI Compliance status, dates

6  

Page 7: RSA Conference 2012 Security Metrics

Example Security Metrics §  Application Security

§  # Applications, % Critical Applications, Risk Assessment Coverage, Security Testing Coverage

§  Configuration Change Management §  Mean-Time to Complete Changes, % Changes w/Security Review, % Changes w/Security

Exceptions

§  Financial §  Infosec Budget as % of IT Budget, Infosec Budget Allocation

§  Incident Management §  Mean-Time to Incident Discovery, Incident Rate, % Incidents Detected by Controls, Mean-

Time Between Security Incidents, Mean-Time to Recovery

§  Patch Management §  Patch Policy Compliance, Patch Management Coverage, Mean-Time to Patch

§  Vulnerability Management §  Vulnerability Scan Coverage, % Systems w/o Known Severe Vulnerabilities, Mean-Time to

Mitigate Vulnerabilities, # Known Vulnerability Instances

7  

* Source: Center for Internet Security

Page 8: RSA Conference 2012 Security Metrics

Gathering Data

§  Data can be qualitative or quantitative §  Data can be coarse-grained or fine-grained §  Data can involve ordinal or cardinal numbers §  Less mature programs often have historical data to use

§  Coarse-grained, qualitative, requires interpretation §  Examples: Audit findings, incident reports, viruses…

§  More mature programs use multiple data sources §  Data from different sources can provide context, it is

important to consider the type of meta data that can be gathered to add value later on

8  

Page 9: RSA Conference 2012 Security Metrics

Modeling Data

§  Some good standard assessment frameworks can be used to provide a standard taxonomy for describing risk

§  Common frameworks allow data to be shared and compared between companies

§  Good models allow better analysis of complex risk scenarios

§  Examples: CAPEC, FAIR and VERIS §  Example of Industry Data: Verizon DBIR

9  

Page 10: RSA Conference 2012 Security Metrics

Operational, Tactical & Strategic Metrics

§  Operational plans lead to accomplishing tactical plans, which in turn lead to accomplishing strategic plans (which in turn are aligned with business objectives).

§  Tactical & Operational: IDS, Forensics, Help Desk Tickets, Time to Patch, Viruses Blocked, Support, Change Management…

§  Strategic Metrics: Overall Compliance, Compared to Baseline, Identifies Gaps in Program, Shows Business Alignment & Value

10  

Page 11: RSA Conference 2012 Security Metrics

Learn Where Others Succeed & Fail

11  

§  Successful security leaders overcome confirmation bias and compare notes more often with peers

§  Standards and frameworks help a company establish a baseline

§  Results need to be translated into a context that is relevant for your business

§  Be aware that executives may downplay the significance of industry data and feel their company is the exception to the rule

Page 12: RSA Conference 2012 Security Metrics

Good or Bad?

12  

Page 13: RSA Conference 2012 Security Metrics

Good or Bad?

13  

© Pedro Monteiro of the What Type blog

Page 14: RSA Conference 2012 Security Metrics

Good or Bad?

14  

Page 15: RSA Conference 2012 Security Metrics

Good or Bad?

15  

Page 16: RSA Conference 2012 Security Metrics

Good or Bad?

16  

Page 17: RSA Conference 2012 Security Metrics

Good or Bad?

17  

Page 18: RSA Conference 2012 Security Metrics

Good or Bad?

18  

Applied Security Visualization, Raffael Marty

Page 19: RSA Conference 2012 Security Metrics

Good or Bad?

19  

Applied Security Visualization, Raffael Marty

Page 20: RSA Conference 2012 Security Metrics

Good or Bad?

20  

http://www.pentest-standard.org

Page 21: RSA Conference 2012 Security Metrics

Clear, Concise, Contextual

21  

© 2010 Institute of Operational Risk

Page 22: RSA Conference 2012 Security Metrics

Presenting to Executives

22  

© 2010 Institute of Operational Risk

Page 23: RSA Conference 2012 Security Metrics

Security Metrics for Management

§  Find a way to add business value §  Meeting regulatory requirements §  Consolidation of tools, reduction of resources §  Demonstrate reduced costs by reduction in help desk cases §  Business leaders take the loss of IP seriously §  Have security seen as a business enabler. New technologies

come with risks, but they may also lead to new innovations and competitive advantage.

§  Explain it in language business leaders understand §  Make presentations clear & concise §  Avoid IT jargon §  Provide the information executives need to make informed

decisions

23  

Page 24: RSA Conference 2012 Security Metrics

Building a Security Metrics Program

§  Decide on your goals and objectives at the onset §  Long-term and short-term goals

§  Identify key metrics (SMART) to generate §  Will these be qualitative or quantitative? §  Will these be manual or automated? §  Will these be based on a standard framework, or vetted against peers, or use

some other model? §  Will these be tactical, operational, strategic or business metrics?

§  Establish a baseline and targets

§  Determine how best to present metrics in a consistent way, for audience and frequency

§  Get stakeholder buy-in and feedback; deliver balanced scorecard

§  Develop a process for continuous improvement

24  

Page 25: RSA Conference 2012 Security Metrics

References §  CAPEC, http://capec.mitre.org

§  Verizon DBIR, http://www.verizonbusiness.com/go/2011dbir

§  Verizon VERIS Framework, https://www2.icsalabs.com/veris/

§  FAIR Framework, http://fairwiki.riskmanagementinsight.com/

§  Center for Internet Security, Security Metrics, http://benchmarks.cisecurity.org/en-us/?route=downloads.metrics

§  Trustwave SpiderLabs Global Security Report, https://www.trustwave.com/GSR

§  Ponemon Institute, http://www.ponemon.org

§  Security Metrics: Replacing Fear, Uncertainty, and Doubt, Andrew Jaquith (2007)

§  Metrics and Methods for Security Risk Management, Carl Young (2010)

§  Security Metrics, A Beginner’s Guide, Caroline Wong (2011)

§  Applied Security Visualization, Raffael Marty (2008)

§  The Visual Display of Quantitative Information, Edward Tufte (2001)

25  

Page 26: RSA Conference 2012 Security Metrics

References §  New School Security Blog, http://newschoolsecurity.com/

§  SecurityMetrics.org, http://securitymetrics.org/

§  A Few Good Metrics, http://www.csoonline.com/read/070105/metrics.html

§  Measuring Security, Dan Geer, http://geer.tinho.net/measuringsecurity.tutorial.pdf

§  CIS Consensus Security Metrics v1.0.0, https://community.cisecurity.org/download/?redir=/metrics/CIS_Security_Metrics_v1.0.0.pdf

§  Performance Measurement Guide for Information Security, http://csrc.nist.gov/publications/nistpubs/800-55-Rev1/SP800-55-rev1.pdf

§  Directions in Security Metrics Research, http://csrc.nist.gov/publications/drafts/nistir-7564/Draft-NISTIR-7564.pdf

§  A Guide to Security Metrics, http://www.sans.org/reading_room/whitepapers/auditing/a_guide_to_security_metrics_55

§  Patch Management and the Need for Metrics, http://www.sans.org/reading_room/whitepapers/bestprac/1461.php

26  

Page 27: RSA Conference 2012 Security Metrics

References §  The Security Metrics Collection,

http://www.csoonline.com/article/455463/The_Security_Metrics_Collection

§  Implementing a Network Security Metrics Program, http://www.giac.org/certified_professionals/practicals/gsec/1641.php

§  Choosing the Right Metric, http://www.juiceanalytics.com/writing/choosing-rightmetric/

§  Web Metrics Demystified, http://www.kaushik.net/avinash/2007/12/webmetrics-demystified.html

§  Blogs about: Security Metrics, http://en.wordpress.com/tag/security-metrics/

§  Standardizing metrics and their presentation, http://www.unifiedcompliance.com/it_compliance/metrics/reporting_standards/standardizing_metrics_and_thei.html

§  Getting to a Useful Set of Security Metrics, http://www.cert.org/podcast/show/20080902kreitner.html

§  Dashboards by Example, http://www.enterprise-dashboard.com/

§  Excel Charting Tips, http://peltiertech.com/Excel/Charts/index.html

27  

Page 28: RSA Conference 2012 Security Metrics

Group Discussion

28