37150665 deloitte kpi and measuring security

You Cant Manage It If You Cant Measure It

ISACAMarch 2006

Deloitte & Touche LLP and affiliated entities.

Agenda

Do you know how well your information security program is working?

Key Performance Indicator (KPI)

Key Performance Index (KPX)

Information Collection

Examples

Summary


What do we have to be worried about?

The time between the discovery of a vulnerability and the potential exploit is diminishing from months to days if not hours


IT Security Governance Maturity Model

The Maturity Model is sponsored by the IT Governance Institute

It is used to rank the maturity of an organizations practices and standards against industry best practices and standards

It can be used to help guide an organization on the areas that will improve their overall information security posture


How do you know if you have an information security program that effectively manages risks?

Obtain a high score on an ISO 17799 assessment?

Complete regular, active penetration tests with no discovered vulnerabilities?

Have an acceptably low # of security incidents reported using the Incident Response process?

Have an effective virus program (few or no infections and any infections are managed effectively with little interruption)?

Have Measurable Service Level Expectations (SLE) that are consistently being achieved?

Have an effective IDS program (# and type of alerts are being managed effectively, little impact on the business, in line or better than industry benchmarks)?

Obtain certification against an information security reference standard (ISO 27001)?


There are several problems to avoid when establishing an information security measurement program

Lack of management commitment

Measuring too much, too soon

Measuring too little, too late

Measuring the wrong things

Imprecise metrics definitions

Using metrics data to evaluate individuals

Using metrics to motivate, rather than to understand

Collecting data that is not used

Lack of communication and training

Misinterpreting metrics data


Key Performance Indicators (KPIs) can help determine the current status of the information security program

A key performance indicator is a measure of a particular organizational performance activity, or an important indicator of a precise health condition of an organization

Used as an indication of the current state of a component of the business to take the surprise out of risk

To be effective, the KPI must be defined as succinctly as possible

Can be measured as an improvement from a known state or a reference standard


A Key Performance Indicator . . .

Must be something that can be measured and continued to be measured

Must be precise, meaningful and understandable

Must be relevant to the business

May be required by legislation and/or Regulations

Must have a measurement index that has meaning

Must have an appropriate life (Stickiness)

Should be tied to the organizations vision and strategy


Types of Key Performance Indicators (KPIs)

Threshold when an index reaches set targets or falls into set ranges

e.g., ETS scores on defined risks

Milestone when a specific condition is reached

e.g., certification

Quantitative measure of value (number, time, $, etc.) e.g., number of reported security incidents, lost time due to

viruses

Qualitative measure of acceptability or health e.g., survey ratings, rating of risks


Examples of Key Performance Indicators

Awareness

Knowledge of policies, standards and procedures (surveys and tests)

Risk Assessment

Depth and breadth of regular risk assessments across the enterprise (When was the last assessment? Qualitative measure of the risks, risk index)

Risk Management

Number of incidents reported, amount of loss incurred, number of situations managed

Audit

Noted deficiencies against the policy and standards (measured year over year)

Benchmarks and Certification

Maintaining/following IT security certifications such as FIPS 140-1, ISO 27001, ISO 15408 (Common Criteria)


Possible Non-Risk Key Performance Indicators (KPIs)

People Training & Certifications

Competence Turnover

Technology Currency

Cost management

Compliance / licensing

Investment Trends per area

Effectiveness & Return on Investment Key Risk Indicator experience vs. cost

Productivity

Missed Deadlines


KPIs can be used to measure the Effectiveness of Investment (EOI)

A Return on Investment (ROI) for information security is difficult to measure since risk, and especially risk reduction, is challenging to quantify in terms of dollars.

The Effectiveness of Investment (EOI) could be the comparison of the effectiveness of the security measures with the value of the investment.

For example, the number and impact of viruses and worms can be compared with the investment in virus detection technology and support programs.

A collection of KPIs could be used to measure the EOI for information security


A Key Performance Index (KPX) is a summary or correlation of one or more KPIs that provides an indication of the overall performance of a defined area of the security program

May prompt the organization to change strategic direction in information security

Levels may be triggered by a variety of factors

Must be meaningful and understandable

Must be relevant to the business

Must have a measurement index that has meaning

Must have an appropriate life (Stickiness) and

Should be tied to the organizations vision and strategy


Example KPI Format

Any additional information or comments? Is this a requirement from legislation or regulations?

Comments

___ Day ___ Week ___ Month

___ Quarter ___ Year ___ Year+

Frequency

Any potential tools used to support the measurement and reporting process?

Tools

Method used to measure the KPI Method

What does it apply to?Unit/Dept

__ Low __ Medium __ HighEffort

__ Quantitative ___ Qualitative ___ Milestone ___ ThresholdType

Who is this KPI relevant to?Stakeholder

What are the objectives of the KPI what is it measuring? Why is it important?

Objective

Description of the KPI what does it address?Description

Short name or title for the KPIKPI Name


Example Key Performance Indicator (KPI)

Need to have confidence in the detection and reporting mechanisms to be able to measure changes to the index over time. A lower index will then mean less risk

Comments

___ Day _X_ Week _X_ Month _X_ Quarter _X_ Year ___ Year+Frequency

IDS and/or security management/reporting softwareTools

Count number of reported security incidents/events at low, medium and high severity over the past week

Method

Information SecurityUnit/Dept

__ Low _X_ Medium __ HighEffort

_X_ Quantitative ___ Qualitative ___ Milestone ___ ThresholdType

CSIO, CIO, Operations Management, Technology ManagementStakeholder

A measure of the relative size and effectiveness of the organizations risk management processes

Objective

Provides a relative index on the current number of reported security incidents/events at differing security levels for the recent reporting week

Description

Weekly Reported Security IncidentsKPI Name


Example Key Performance Index (KPX)

Need to have confidence in the detection and reporting mechanisms to be able to measure changes to the index over time. A lower index will then mean less risk

Comments

___ Day _X_ Week _X_ Month _X_ Quarter _X_ Year ___ Year+Frequency

IDS and/or security management/reporting softwareTools

Count number of reported security incidents/events at low, medium and high severity over a defined time frame

Method

Core SystemsUnit/Dept

__ Low _X_ Medium __ HighEffort

_X_ Quantitative ___ Qualitative ___ Milestone ___ ThresholdType

CSIO,CIOStakeholder

A measure of the relative size and effectiveness of the organizations risk management processes

Objective

Provides a relative index on the current number of reported security incidents/events at differing security levels within a specified time frame

Description

Information Security Risk Management IndexKPI Name


Several automated tools can provide a view of security incidents and trends


Security Incidents - Advanced Forensic Tools


The Information Security Program should include a reporting mechanism that provides a single point of reference for concise, executive-level information for business and technology owners.

The dashboard aims to transform data from operations to actionable information for decision makers

Sample Security DashboardOperator Event View

Reports

Incident Tracking (Ticketing System)

Geographic Threat View

Trend View

Advanced Forensic Tools

Geographical Dashboard View


An analysis of security incidents will contribute to the current status of the Information Security Program


Keep track of each area of concern that is the object of a KPI or KPX definition

Any additional information or comments?Comments

Any required acknowledgement or reporting for this KPI?Reporting

How does the KPI(s) map to the individual performance goals?Map KPI(s) to Performance Goals

What summary index(s) can be defined that is a high-level representation of one or more KPIs that are vitally important to the organization?

KPX(s)

What Key Performance Indicators(s) should be defined for this objective?KPI(s)

What are the measurements that may be available to report on this area?Measurements

What are the key control objectives and controls that should be in place for the organization? The controls should be based on international reference standards

Key Control Objectives and

Controls

What is the main objective how is it measured? Why is it important?Objective

What is the Vision and Mission statement that directs IT security?Vision/Mission

Topic -

Deloitte & Touche LLP and affiliated entities.Presentation Name (View / Header and Footer)

An example KPI for Inappropriate Use

Inappropirate Use - KPXThe impact of recorded

inappropriate use events compared to the amount of

IT security awareness training per person.

KPI - 1Number of verified

instances of inappropriate use over a set time

period. (weekly or by reporting period)

KPI - 2

Impact of inappropriate use events to the business in terms of resources and or loss over time (weekly or

by reporting time)

KPI - 3Number of verified

inappropriate use events compared with the number of IT security awareness training days per person

compared over time

Measurement - 1Number of inappropriate use cases opened and

verified

Measurement - 2Amount of service lost to

inappropriate use

Measurement -3Number of IT security

awareness training days


An example KPX for Inappropriate Use

KPX


An example KPI for Intrusion Detection

KPI - 4Cost of the IDS program in relation to the number and impact of detected events

Measurement - 4The number of systems with active monitoring capabilities

KPI - 3Number of IDS program failures Measurement - 5

Number of Sensors per network segment

Measurement - 6Cost of the hardware and/orsoftware to implement intrusiondetection sensors

IDS KPXThe measureable amount of productivity loss attributedto intrusions in relation to thethe number of events and thecost of the IDS program.

KPI - 1Average amount of Loss (productivity time) per intrusion within a set time period (weekly or per reporting period).

Measurement - 1Number of incidents of intrusions detected and reported

Measurement - 3Amount of downtime or productivity loss caused by intrusion incidents.

Measurement - 2Number of incidents of intrusions impacting the organization that were not reported

KPI - 2Number of events caught andprevented by the IDS within aset time period


An example KPX for Threat ManagementIntrusion Detection System (IDS)

Number of Resolved Major and Catastrophic Incidents Over Time

Time/ Reporting Period

# of Resolved Major and

Catastrophic Incidents

11 22

33

Number of Major and Catastrophic Incidents Over Time

Time/ Reporting Period

# of Major and


High Risk IncidentsHigh Risk Incidents

Critical IncidentsCritical Incidents

Average Time to Resolve a Number of Major and Catastrophic Incidents

Average Time to Resolve Major and Catastrophic Incidents

# of Resolved Major and


Major Incidents

CatastrophicIncidents

Number of Resolved Major and Catastrophic Incidents

Major IncidentsMajor Incidents

CatastrophicIncidentsCatastrophicIncidents

Number of Resolved Major and Catastrophic Incidents

>4410hrs/month/ system productiv ity loss>10hrs/month/ system productiv ity loss


Summary

A good collection of Key Performance Indicators will provide an overview of the current status of risk management within the organization Use the collection of KPIs as an information security dashboard

The KPIs can be used to help comply with legislative or regulatory requirementsProvide the information that can be used for reporting purposes

The KPIs must be carefully selected and defined to be useful Must be meaningful and measurable

Effective KPIs can be used to demonstrate good management of risk For example, KPIs may provide a financial institution the ability to

reduce the percentage of reserve required to offset operational risk defined by the Basel II Accord

Questions?Glen Bruce, [email protected]

Member ofDeloitte Touche Tohmatsu


Deloitte, one of Canada's leading professional services firms, provides audit, tax, consulting, and financial advisory services through more than 6,100 people in 47 offices. Deloitte operates in Qubec as Samson Blair/Deloitte & Touche s.e.n.c.r.l. The firm is dedicated to helping its clients and its people excel. Deloitte is the Canadian member firm of Deloitte Touche Tohmatsu.

Deloitte refers to one or more of Deloitte Touche Tohmatsu, a Swiss Verein, its member firms, and their respective subsidiaries and affiliates. As a Swiss Verein (association), neither Deloitte Touche Tohmatsu nor any of its member firms has any liability for each other's acts or omissions. Each of the member firms is a separate and independent legal entity operating under the names "Deloitte," "Deloitte & Touche," "Deloitte Touche Tohmatsu," or other related names. Services are provided by the member firms or their subsidiaries or affiliates and not by the Deloitte Touche Tohmatsu Verein.

37150665 deloitte kpi and measuring security

Documents

affiliated entities

information security

strategy deloitte touche

hours deloitte touche

types of key performance

security incidents

effective ids program

effective virus program