www.clearpointmetrics.com metrics 101 may, 2006. © 2005-2006 clearpoint metrics proprietary &...

www.clearpointmetrics.com

Metrics 101

May, 2006

© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential 2© 2005-2006 CLEARPOINT METRICS Proprietary & Confidential

Outline

What is a metric ? What makes a metric good ? What makes metrics so hard ? How are metrics used ? How are metrics mapped to the business ? What are some examples of good metrics ? What are some easy, automatable metrics ? What are some good ways to organize metrics ? Where do metrics get their data ? What is a well-managed metric ? What are key components of a well-managed metric ? What is the lifecycle of a metric ? What does a purpose-built tool for designing metrics look like ? What is a default view of metric ? What can a custom scorecard on a specific topic look like ?


What is a Metric ?

Measurement – Generated by counting Measurements provide quantitative observations of

discrete factors, in isolation

Metric – Derived through analysis applied to measurements Provide quantitative data about a target process or

asset in order to achieve an explicit purpose Truly useful metrics provide the insight needed to

make better decisions.


What Makes a Metric Good ?

Flexibility Measurement technique Sources employed Publication mechanism

Accuracy Repeatable Auditable

Context Correlates measurements across multiple sources Comparability across multiple dimensions (e.g. time, location) Relevance to the Business (e.g. organization, applications, LoB’s) Assignable to someone (e.g. accountability)

Transparency Assumptions well documented Data sources explicitly identified Models, analytics, algorithms completely and unambiguously defined


What Makes Metrics so Hard ?

RouterCisco3COMLucentNortelJuniper

FirewallCheckPointJuniperCiscoSymantec

AntivirusSymantecMcAfeeTrend MicroPandaSophos

Web CacheInktomiPersistenceF5, Cisco

Web ServerApacheIISNetscapeiPlanet

NetworkSystemMgtHP OpenViewIBM TivoliCA UnicenterCiscoRemedy

App ServerBEAOracleWebspere

ApplicationMiddlewareBEAWebsphereTibco

ERPSAPPeopleSoftOracleUser-written

CRMSeibelPeopleSoftOracle

DatabasesOracleSQL ServerDB2

DMZ Middle Tier Back-End

Partners & Suppliers

Heterogeneous and dispersed silo’s of vital IT information Never the same for any two organizations Difficult to fuse together silo’s and map results to a business context Challenging to express Exacting to communicate effectively


How are Metrics Used ?

To answer strategic questions: Is security getting better or worse? What is the value of a specific security investment? Where is the point of diminishing returns on a security

investment? What options exist and what are their consequences in terms of

security or operational risk? Are risk avoidance policies being followed?

To justify allocation of resources To drive positive change, awareness, accountability To provide hard quantitative evidence of the existence,

execution, coverage, effectiveness of controls To make better decisions


How are Metrics Mapped to the Business ?

80-20 Rule: Generic versus Specific Some Techniques:

Fusion across islands of data Comparative analysis and decomposition of values across

business-relevant categories such as Line-of-Business, office location, or asset class

Thresholds and goal-attainment models Coverage, efficiency, and effectiveness models Correlation between related metrics Complexity and variability models Weighting models for risk

Often done manually, but often (at least partially) amenable to automation.

Automation expands scope & capacity while improving accuracy, regularity, accountability, and repeatability


What are Some Examples of Good Metrics ?

1. Latency between employee termination and de-provisioning of all access to applications Context: IAM Efficiency Use: SSO Project Evaluation KPIs: Accounts/User, Support Time/Account, Support Cost/User –

min, max, mean, variance

2. Vulnerability Scan Coverage for past 12 months Context: Vulnerability Management Effectiveness Use: Comparison of two scanners KPIs: PercentScanned/Scan/Scanner – min, max, mean, variance

3. False Positive Rate for Event Manager for past 365 days Context: Event Management Accuracy Use: Cost analysis of Incident Response Center workload KPIs: Percent False Positives/Day (signal/noise), Cost/Event


What are Some Easy, Automatable Metrics ?

Incidents1. # of incidents and severity2. Mean time to resolve incidents…Hosts7. % hosts adhering to policy8. # of workstations using server

ports10.% of hosts that are portable (i.e.

laptops)…Users21.# of power users by division and

trending23.# adhering to password aging policy25.% with VPN access by user type…Perimeter35.# of internet facing hosts36.# of open ports and intended

application37.# of unused firewall rules…

IT Systems Configuration Turbulence78.# new hosts per time period79.# additional open ports/services per time

period80.# additional users per time period…

Patch Management55.# of vulnerabilities identified by system type56.Latency from patch release to patch

application57.Number of patches applied per time period…Storage & Backup66.% of hosts with managed backups67.hours of backup gap by system type and purpose71.# of restore requests (type/who/critical?)…

Virus Management89.% systems with AV systems90.AV signature age by all dimensions95.# inbound/outbound viruses at perimeter …

Vulnerability Management97. Average time period between scans99. % Machines scanned100.# Vuln’s identified by by type / patch level..

Threats35.# attacks by severity36.# false positives by severity37.# incidents by severity…


What are Some Ways to Organize Metrics ?

ThreatsCovers active monitoring and defenses against attacks, such as anti-virus and IDS systemsIncludes metrics around topics such as:

Security Event Management Host Anti-virus systems Network Anti-virus systems IDS Systems Incident Response

Compliance & RiskCovers compliance dimensions across other metrics, such as compliance of financial systems for SOX.Includes metrics around topics such as:

Coverage of Network Management Coverage of automated scanning Coverage of IDS and Anti-virus Risk weighting of systems

VulnerabilitiesCovers vulnerabilities inherent in hosts, such as discovery and remediation of known exploitable vulnerabilities and the application of patches.Includes metrics around topics such as:

Vulnerability Identification and Scanning

Vulnerability Remediation Patch Identification Patch Application

Identify and Access ManagementCovers user access and authentication to the organization’s systems.Includes metrics around topics such as:

Password age and strength User roles and permissions External Access


Where Do Metrics Get Their Data ?

Generic CSV Files Spreadsheets JDBC/ODBC LDAP

Threat Managers ArcSight eSecurity Symantec DeepSight

Vulnerability Managers Qualys QualysGuard ISS Site Protector Tenable/Nessus

Anti-Virus, Anti-Spam Trend Micro McAfee ePO Symantec AV

Network and System Managers Tivoli MSFT MOM HP OpenView

Identity and Access Management MSFT IAM Series Tivoli TIM, TAM, FIM CA/Netegrity Site Minder Symantec L0pht Crack John the Ripper

Incident Management Systems Remedy ARS Peregrine Service Center JourneyX Timekeeper

Human Resource Managers PeopleSoft SAP

Asset Managers MSFT SMS Tivoli Management Framework CA Unicenter Asset Manager LANDesk Management Suite

Application/Storage Security Mgrs Fortify Vontu


Metrics can be Data Producers, too

RiskMgmt.

AssetMgmt. Financial

Mgmt.

SecurityOps

Regulatory& Compliance

HR CRM

Vuln & Threat Info

Sources

NSM PolicyRespCenter &

Help Desk

Regulations


What is a Well-Managed Metric ?

A Well-Managed Metric has … Architecture

Components Interfaces to data providers and metrics consumers

Dynamic Concept of Operation Flow of work: Sequencing, Precedence, Schedules Contingency handling

Life Cycle & Operations Management Crisply defined stages Auditable transitions between stages Administration: Fault, Configuration, Accounting/Auditing,

Performance, & Security

Portability with respect to-- Data providers, data consumers Publication method


What are Key Components of a Well-Managed Metric ?

General Information Author Description Version Annotations: taxonomies, keywords

Data Sources Reference definition Interface drivers Mapping from at least two sources to reference definition

Business Logic Workflow orchestration Models, Analytics, Calculations

Persistence Logic Schedule Publication Logic

Schedule Notification criteria (Default) Scorecard visualization


MxStudio

Deploy

Metric & Scorecard

Description Package

MXServer

Metrics Result DB

MxPublisher

Publish

Create Calculate Communicate

What is the Lifecycle of a Well-Managed Metric ?

How often will this metric be computed?

What sources will provide the data?

What is the workflow to collect, compute and store results?

Where will results be accumulated over time?

Deliverable: Results

What question will this metric answer?

What data does it need? What calculations and models

will it use? What results will it produce? How should results be

visualized and published?Deliverable: Atomic Package

How often will scorecard editions be published?

What do they look like? Who can see what? Where will scorecards be

delivered? What is the workflow for

annotating & approving scorecards before pub?

Deliverable: Scorecards


What Does a Metric Authoring Tool Look Like ?

Hierarchically Organized Metric Catalog

Metric Editors:• General Information• Workflow (shown)• Schedule• Tester/Debugger

Analysis Wizards• Aggregation• Filtration• Transformation• Built-in Functions• Escape to Javascript

Hierarchically Organized Catalogs for Data Sources and Canned Analytics


What is a Default Visualization of Metric Results

Hierarchically Organized Scorecard Catalog

Hierarchically Organized Metrics Results

Default Visualization of Metric Results in Chart or Tablular Format

Display Controls, e.g. chart type and time period


What can a Custom Scorecard Look Like ?

www.clearpointmetrics.com metrics 101 may, 2006. © 2005-2006 clearpoint metrics proprietary &...

Documents

useful metrics

automatable metrics

examples of good metrics

metric good

isolation metric

wellmanaged metric

better decisions slide

defined slide