application assessment metrics

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

PRESENTED BY: Yvette du Toit

Agenda!

•  Background!•  Approach!•  Examples!•  Challenges with Application Security Metrics!•  Q&A!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

Background!

•  As Security Consultants we write reports!–  Test, analyse, write up findings, submit to client!

•  Issues still remain open – why?!–  Reports not say enough!–  Question value report offer!

•  Solution – metrics / visualisation!–  Graphs, colour, size etc!

•  First – letʼs take a look at what reports say…!–  Qualitative ratings!–  Best practice!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

What do Reports Say?!

•  2007 - 2011!•  Many words….!•  Content (Exec Summary, Technical Summary, Conclusion)!•  Are actions effective?!•  What would be more valuable – comparison (time & peers)!•  How do we use metrics?!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

Pages Words Assessments 638 224587 Re-Tests 137 28164 Total 775 252751

Approach!

•  Metrics – definition!–  Quantifiable!–  Characteristics!

•  3 Metric Veterans:!–  Jacquith - “those that support decision making about risk for the

purpose of managing that risk” !–  Marty – “a picture paints a thousand log records”!–  Godin: “just because something is easy to measure doesnʼt mean

itʼs important”!•  NB: To measure what is important & that will yield “useful”

information!–  Examples of metrics not necessarily useful!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!

•  Definition!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

Useful?!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!

•  Example!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

•  Metrics can be misleading!

Useful?!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!

•  Example!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

•  Metrics are not always 100% useful!

Approach!

•  Why?: illustrate useful information!–  Recurring issues!–  Time required to compromise!–  Top 10 list!–  Effectiveness of remediation!–  Benchmarking!

•  Who? 7 organisations in financial sector!•  When? 3 ½ years!•  How? Data capture process!

–  Marco Slaviero (Head of R&D)!–  Spreadsheet for data capture!–  Report meta-data (project length, frameworks, dates etc.)!–  Findings categorised (pre-defined list of vulns)!–  Findings ranked (Impact, EoE, Threat metric)!

•  Normalisation !–  Allows for comparison across time and peers !

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!

•  Introduction!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

Annual Distribution of Project (Days)!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

SensePost Metrics Proposal!

•  Metrics extracted from report data:!–  Timelines (plotting projects on timeline)!–  Basic counts and statistics (uncover counts)!

•  Number of projects!•  Number of days!•  Number of words and pages in report!

–  Threat metrics (Findings per threat level)!–  Bug class metrics (Findings across categories) !–  Top 10 list !–  Re-Test Metrics!–  Benchmarks (comparison to peers)!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!

•  Our Metrics!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

SensePost Metrics in Action: Timelines!

•  Useful?!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

! "#$%&! '()*&!

!"#$%&'()'*++%++#%,-+' ./0' 112304'

!"#$%&'()'5%67%+-+' 8/4' 108.2'

7(-9:' 443' 131438'

SensePost Metrics in Action: Threat Metrics!

•  Useful?!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

SensePost Metrics in Action: Bug Classes!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Useful?!•  See 56% of findings occur in Top 11 bug classes!•  2008 Anomaly (No Re-Tests) !

•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

SensePost Metrics in Action: Top 10!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Useful? !•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

SensePost Metrics in Action: Re-Test!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Useful?!•  29% Critical and 42% High-risk issues remain open !

•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

SensePost Metrics in Action: Benchmarks!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Useful?!•  Our client positioned 3rd (not highlighted here)!

•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

Challenges!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

•  Bug counts vs bug classes!–  Bug counts – number of findings!–  Bug classes – categories!–  2 applications scenario (10 findings 1 bug class vs 1 finding in 10 bug classes)!

•  Depth vs breadth!–  Each occurrence – depth!–  Each bug class - breadth!

Q&A!

ASSESSMENTS • VULNERABILITY MANAGEMENT • CONSULTING • TRAINING

Agenda!

•  Background!•  Approach!•  Examples!•  Challenges with

Application Security Metrics!

•  Q&A!

•  Thank you!•  Longer paper – mail me!•  Email: yvette@sensepost.com!•  Contact: +27 79 509 8913!