improving metrics in cyber resiliency - cloud security alliance · 2019-08-15 · improving metrics...

Improving Metrics in Cyber Resiliency (ETIF/ETIT) © Copyright 2017, Cloud Security Alliance. All rights reserved 1

Improving Metrics in Cyber Resiliency

(With ETIF/ETIT)


© 2017 Cloud Security Alliance – All Rights Reserved All rights reserved. You may download, store, display on your computer, view, print, and link to the Improving Metrics in Cyber Resiliency (ETIF/ETIT) white paper at https://cloudsecurityalliance.org/download/improving-metrics-in-cyber-reciliency subject to the following: (a) the Report may be used solely for your personal, informational, non-commercial use; (b) the Report may not be modified or altered in any way;(c) the Report may not be redistributed; and (d) the trademark, copyright or other notices may not be removed. You may quote portions of the Report as permitted by the Fair Use provisions of the United States Copyright Act, provided that you attribute the portions to the Improving Metrics in Cyber Resiliency (ETIF/ETIT) white paper.

https://cloudsecurityalliance.org/download/connected-vehicle-security

https://cloudsecurityalliance.org/download/connected-vehicle-security


Acknowledgements

Lead Authors

Dr. Senthil Arul

Dr. Shimon Modi

Contributors

Josep Bardallo

Ramon Codina

Bernd Jaeger

Courtney Keogh

Paul Lanois

Daniel Miller

Michael Roza

Duncan Sparrell

John Yeoh

Stephen Lumpe (design)


Table of Contents

Improving Cyber Resiliency by Defining, Measuring and Reducing Elapsed Time to Identify

Failure (ETIF) and Elapsed Time to Identify Threat (ETIT)

Abstract

Introduction

Modified Cyber Resiliency Model

Conclusions and future work

References


Improving Cyber Resiliency by Defining, Measuring and Reducing Elapsed Time to Identify Failure (ETIF) and Elapsed Time to Identify Threat (ETIT)

AbstractCyber resiliency is important as it gives us “the ability to prepare and plan for, absorb, recover

from, or more successfully adapt to actual or potential adverse effects.” Despite billions of

dollars being spent on cybersecurity, information systems data breaches are increasing year

after year. To reverse this trend, it is essential to develop metrics and processes to measure (1)

threats before they become cyberattacks, (2) recovery of lost functionality after a cyberattack.

This paper introduces two essential metrics: Elapsed Time to Identify Failure (ETIF) and Elapsed

Time to Identify Threat (ETIT). Measuring them and developing processes to lower the values of

ETIF and ETIT would improve the resiliency of an information system. The paper also discusses

challenges associated with measuring ETIF and ETIT and proposes that the measurement and

reporting of ETIF and ETIT could be transferred from companies whose systems encounter

cyberattacks to companies who are in the Intrusion Detection System (IDS) space. Transferring

the responsibility for measuring and detecting cyber intrusions to the IDS community would

bring superior algorithms that are needed to detect anomalies and improve cyber resiliency.


IntroductionPreventing cyberattacks is an important objective. However, achieving such a lofty objective

is a challenge. Case in point, cyberattacks, data breaches, and spending on cybersecurity are

increasing year over year. According to ITRC survey study1, a 27.5% increase in cyberattacks

occurred in 2014 compared to 2013. These cyberattacks resulted in the exposure of over

80 million components of personally identifiable information (PII) including social security

numbers, driver’s license numbers, medical records, etc.1. In 2014, McAfee2 estimated that

the annual cost of cybercrime could be as high as $575 billion globally. Target Corporation’s

(NYSE:TGT) data breach in 2014 resulted in $1 billion in losses and the company incurred $88

million in breach related expenses3. According to Ponemon Institute4, the average cost of a

data breach for an American company is $5.4 million and the average per capita cost is $188.

In the USA, spending on cybersecurity reached more than $70 billion in 20145. The Yahoo data

breach, which is considered to be the largest discovered in history with over 500 million users

affected, has reduced the acquisition price by $350 million.6

The flexibility, scalability, availability, lower cost, etc., are driving the corporate information

infrastructure to migrate to the cloud from local servers and the operational assets information

such as financial, human resources, production, logistics, etc., are being stored remotely. The

cloud services could introduce new cyberattack vectors due to (1) lack of direct control of

cloud service providers on the shared infrastructure and dependency on third party developed

capabilities7, (2) ease in procuring and accessing cloud services allows nefarious users to hack

into data of other users in a multi-tenant cloud architecture8, and (3) Man In Cloud Attack –

involves the theft of user tokens which cloud platforms use to verify individual devices without

requiring logins during each update and sync9., etc. These new attack vectors would not only

compromise the information assets but also result in a detrimental impact to the operations

of the whole enterprise. Therefore, the discussions presented in this paper on improving

information system resiliency could be extended to the operational systems’ resiliency as well.

The cyber ecosystem is non-discriminant in the sense that good and harmful information

coexists harmoniously. The internet and transmission protocols by which the information

travels within the cyber ecosystem is also non-discriminant in the sense that good and


harmful information is transmitted across

cyberspace without discrimination as to

priority or hierarchy. Thus, the opportunity

to discriminate between good and harmful

information would not occur in the cyber

ecosystem. Discrimination must occur within

one’s own Information Technology/Operational

Technology (IT/OT) infrastructure, and must

occur in the security infrastructure layers.

As shown in Figure 1, the first opportunity

to discriminate between good and harmful

information occurs at the security layers

between the cyber ecosystem and IT/OT

infrastructure. The second opportunity occurs

at the security infrastructure layers part of the

information system.

If a cyberattack is identified and captured at

the security layers in the IT/OT infrastructure,

then the information system is protected.

However, if a cyberattack is identified by the security layers residing in the information system

(i.e., harmful information already past the IT/OT infrastructure), then the information system

is compromised. The loss of function resulting from a cyberattack depends on the kind of

function that the attack targets (logistics, tactics, etc.) and on the type of information that is

available and exposed (trade secrets, personally identifiable information, etc.). For example, a

2014 cyberattack on Sony Pictures (NYSE:SNE)6 disabled computers, leaked upcoming movies,

and completely paralyzed the operation for a short period of time whereas a cyberattack7 on

the Pentagon’s Joint Forces e-mail system resulted in the shutdown of the e-mail system for a

short period of time but only locally, without affecting rest of the Pentagon’s e-mail systems.

IT/OT Infrastructure

Cyber Ecosystem

Information System

Application 1

Layered security such as Intrusion Detection, Authentication, and Malware Detection.

Information System

Application 2

Information System

Application 3

Layered Security Layered Security Layered Security

Figure 1Depiction of various layered security in

the IT/OT infrastructure


The National Academy of Sciences8 (NAS) defines resiliency as “the ability to prepare and

plan for, absorb, recover from, or more successfully adapt to actual or potential adverse

effects”. Bruneau et al.9 developed a framework to quantitatively assess the resiliency of a

community after an earthquake by measuring Quality of Infrastructure, Q(t), over a period of

time. The value of Q(t) ranges from 0% to 100% where 0% means no service available and

100% means no degradation

in service. At time t0, an

earthquake event occurred

and dropped Q(t) from 100%

to 50% instantaneously. It took

time t1 to fully recover, Figure

2. This approach is based

on the notion that Quality

of Infrastructure, Q(t), of a

community is affected after

an earthquake and it takes

a certain amount of time to

fully recover from it. Zobel et

al.10 applied the resiliency of

community after an earthquake framework to quantify cyber resiliency after a cyberattack.

They analyzed many types of cyberattacks: (1) slow-onset single event, (2) sudden-onset single

event (3) slow-onset multi event, (4) sudden-onset multi event. An information system performs

many functions; they were all combined into a single function defined as Quality of Service, Q(t). The

functional value for Q(t) ranged between 0 and 1; value of zero (0) represents unable to perform

the intended function and value of one (1) represents fully performing the intended functions.

This paper expands the cyber resiliency model presented by Zobel et al. 10. It introduces two

new variables, namely Elapsed Time to Identify Failure (ETIF) and Elapsed Time to Identify

Threat (ETIT) and qualitatively presents a compelling case that the measurement and

publication of ETIF and ETIT would spur innovation in the IDS space, and aid in the overall

improvement of the resiliency of information systems.

100

t0 t1

0.0

Qua

lity

of S

ervi

ce, Q

(t)

Time, t

Figure 2Resiliency of community after an earthquake (adopted from Bruneau et al.)

Q’(t)


Modified Cyber Resiliency ModelFigure 3a graphically represents the Zobel et al.10 characterization of a resiliency profile

for a slow-onset single-event cyberattack. In this case, the loss of quality of service occurs

gradually over time, a virus gradually propagating across the system unnoticed. The start of the

cyberattack in the timescale is represented as t0start, the end of the cyberattack is represented

as t0end, and the corresponding of loss of functionality is Q’(t). The time to recover the full

functionality from t0start is T. The triangular area with the base as T and the height as Q’(t)

represent the loss of resiliency. The smaller the area under the resiliency curve the system

is more resilient. That is, the cyberattack did not impact the functional performance, or the

system recovered quickly and/or a combination of both.

Figure 3b represents the modified cyber resiliency model. In a typical cyberattack, the failure

is identified by studying the anomaly/degradation of intended functions at the IDS layer.

Therefore, the modified resiliency model starts with the time at which the failure was identified

ti (in Zobel et al. model is t0end) and the corresponding loss of function is Q’(t). Immediately

after the identification of a failure, an investigation would start to identify the root causes of

failure and to implement recovery actions to restore full functionality. The investigation would

1.0

T

0.0t0

start ts ti tet0end

1.0

0.0

Qua

lity

of S

ervi

ce, Q

(t)

Loss of Function

Recovery of Function

Qua

lity

of S

ervi

ce, Q

(t)

Loss of Reliliency

Time, t Time, t

Figure 3a represents the original cyber resiliency model for slow-onset single-event (adopted from Zobel

et al. slow-onset multi-event resiliency profile) and Figure 3b represents modified cyber resiliency model.

Please note - ETIF and Recovery time may vary

Figure 3a Figure 3b

ETIF

T

Q’(t)Q’(t)


identify the time at which the cyberattack started, ts (in Zobel et al.’s model is t0start). The line

connecting ti to ts represent the loss of function. The duration between the start of the failure

and the identification of the failure is defined as the Elapsed Time to Identify Failure (ETIF). The

recovery actions would fix the failure, necessary to achieve full functionality. The time at which

full functionality is achieved is represented as te. The line connecting ti and te represents the

recovery function. Therefore, the modified resiliency model includes ETIF and the slopes of loss

and recovery of functions, Figure 3b.

Cyberattacks occur when the information system vulnerabilities are exposed and exploited.

For a given information system, these vulnerabilities are embedded into the architecture of the

software and the hardware and characterize the attack surface15.

As noted earlier, resilience of a system is characterized by its ability to achieve its intended

functions despite disruptions. From that perspective, the bounding function on how a system

is attacked and recovers is dependent on the attack surface. Thus, the functions representing

resiliency, in this case ETIF, are bounded by the attack surface. These functions can be linear or

non-linear; for illustrative purposes, the graphical information presented in this paper show linear

relationships. However, discussions are certainly valid for both linear and non-linear functions.

100

D

A E C G

B

F

0.0

Q”(t)ETIF”

ETIF’

ETIF’”

Q’(t)

Q*(t)

ts ti” te” te’ te’”ti’

Qua

lity

of S

ervi

ce, Q

(t)

Time, t

Figure 4Illustrating the role of ETIF in cyber resiliency


ETIF measures the loss of resiliency of an information system. If ETIF moves from ETIF’ to ETIF’’,

Figure 4, then the loss of resiliency area would reduce from triangle ∆ABC to triangle ∆ADE,

the loss of quality of service, Q(t), would reduce from Q’(t) to Q’’(t), and the recovery time (te)

would reduce from te’ to te’’. Extending this argument, the loss of resiliency would reduce to

zero when ETIF approaches zero. That is, the start of the cyberattack (t0) and the identification

of the cyberattack (ti) occurred at the same time (i.e., the cyberattack was nullified at the IT/OT

infrastructure.)

If ETIF moves from ETIF’ to ETIF’’’, Figure 4, then the loss of resiliency area would increase from

∆ABC to triangle ∆AFG, the loss of quality of service, Q(t), would increase from Q’(t) to Q’’’(t), and

the recovery time (te) would increase from te’ to te’’’. However, if ETIF continue to increase, at

some point the loss of resiliency reaches a level beyond which the system is not recoverable.

This condition would be reached either when the quality of service, (Q(t), reached a point of

no recovery, Q*(t), or when the cost of recovery is more than that to build a new information

system, Figure 4. An example of this condition would be a security breach at a power

generation company causing an electric generator to have a catastrophic failure resulting in a

total loss of functionality. In this case recovery may not be possible.


No

1

2

3

May-14

Dec-14

as early as Nov 2013

29-Jan-15

29-Jan-15

24-Nov-14

> 7 months

> 1 month

1 year

Premera Blue Cross11

Anthem Blue Cross12

Sony Pictures6

Company/Organization

Cyberattack Identified Date

Elapsed Time to Identify Failure (ETIF)Cyberattack Date

4

5

6

as early as July 20, 2014

as early as April 2014

as early as mid-June 2014

20-Oct-14

2-Sep-14

Mid-August 2014

> 3 months

> 4 month

> 2 months

Staples13

Home Depot14

JP Morgan15

7 Apr-14 1-Jul-14 3 monthsCommunity Health16

8

9

10

28-Jan-15

17-Feb-15

May-14

30-Jan-15

17-Feb-15

1-May-15

3 days

3 hours

1 Year

Toys R Us17

University of Maryland18

Office of Personnel Management (OPM)19

Table 1Calculated ETIF based on recent cyberattacks

DiscussionTable 1 shows some of the recent cyberattacks and the ETIF for each of the attacks. The ETIF

ranged from as low as three (3) hours (University of Maryland) to as long as one (1) year (Sony

Pictures & the US Government’s Office of Personnel Management). The actual cyberattack date

and time, as mentioned before, is determined by forensic analysis. The forensic analysis process

is not standard across all industries and as such the data in the public domain is not easily

comparable. As of today, there is no agreed definition of what activities would characterize the

start of a cyberattack. For example, in some cases, certain viruses are dormant in the system

before becoming malignant and in other cases there are multiple attacks over a period of time.

Corporations are concerned about confidentiality, competitive pressures, litigation, image, etc.,

and are therefore reluctant to disclose the specific details of a cyberattack including the date of

the attack and the date of the identification. Given the lack of standardization in publicly available

data on start of a cyberattack, ts, and on the identification of a cyberattack ti, it is currently difficult

to consistently and specifically calculate ETIF and hence cyber resiliency.


If skilled companies in the IDS calculate and report ETIF instead of the corporations

experiencing the cyberattack , it would enable standardization of the forensic process and

development of the tools necessary to clearly define, measure and calculate the start of a

cyberattack. By reporting ETIF instead of the start of cyberattack, ts, and the identification

of cyberattack, ti, corporations may feel less exposed to litigation and negative publicity.

Publishing ETIF and reducing the value of ETIF could spur competition in IDS space to

develop advanced algorithms to identify anomalies in the quality of service data. However,

for ETIF to be meaningful and to be an effective metric for improving cyber resiliency, clear

definitions for ti and te, and agreed upon processes and tools need to be established to

measure ti and te.

The discussion until now has been focused on the loss of resiliency of an information system.

However, an ideal state would be to have no loss of resiliency. That is, the threat of a failure is

identified early enough before it becomes an attack. Such an outcome is possible if there was

a mechanism to disseminate information about the adverse events and the threats among

various entities on a real-time basis. Ideally, if an entity that experienced failure could share the

system’s vulnerability with other entities, analysis of such shared failure event would be critical

in detecting the presence of a threat or recover quickly from the failure. We define this metric

as Elapsed Time to Identify Threat (ETIT). The effectiveness of such a metric can be measured

by its ability to move ti (time to identify failure) closer to ts (start time of the failure). ETIT is

critical in changing limits on the loss and recovery functions and thus impacting the quality of

service. If there is an ability for early identification of the threat that is causing the failure, then

the overall time to recovery and hence the loss of resiliency could be reduced.

It is conceivable that different information systems could have the same loss of resiliency

(i.e., same area of the triangle) but with different slopes for loss and recovery functions. This

is illustrated in Figure 5 where System 1 and System 2 have the same loss of resiliency but

System 1 has much steeper loss and recovery functions and it fully recovers to perform

its full intended function much sooner than System 2. From a user/customer perspective,

System 1 would be preferred in the sense that it recovered to perform its fully intended

function in a much shorter duration than System 2. Therefore, in addition to the loss of


resiliency, the slope of the loss and recovery functions would become important. The steeper

the slope for the loss and recovery functions and shorter the duration between ts and ti (i.e.,

ETIF), the smaller the loss of resiliency.

100

0.0

Qua

lity

of S

ervi

ce, Q

(t)

Time, t

System 1

System 2

Figure 5Comparing two different information systems with the same loss of resiliency with

different Elapsed Time to Identify Failure (ETIF), loss and recovery functions


Conclusions and Future WorkA modified information system resiliency model was presented and introduced two new

variables: Elapsed Time to Identify Failure (ETIF) and Elapsed Time to Identify Threat (ETIT). The

model qualitatively demonstrated that the lower the ETIF, the smaller the loss of resiliency for

an information system. Challenges in calculating ETIF in recent cyber attacks showed difficulty

in measuring the start of the cyberattack (ts) and the identification of cyberattack (ti).

By transferring the responsibility of calculation and publication of ETIF from the companies that

experience cyberattacks to the companies that are in IDS space could bring standardization

and continuous improvement in measuring and reporting ETIF. In addition, it could encourage

competition in the IDS space and would bring innovative and superior algorithms to find

anomalies in the data quality thus improving cyber resiliency.

With the growth in cloud computing, information about the operational assets is being stored

away from the local servers. Although the benefits of cloud computing are obvious, such

decoupling could result in poor operational resiliency if the information asset is compromised.

Therefore, to keep the operational resiliency unaffected, it is essential to bolster information

asset resiliency in the cloud. Hence, a technical framework, along with appropriate regulatory

framework, needs to be created to enable the measurement and reporting of ETIF and ETIT.

The Cybersecurity Act of 201516 passed by the US Congress, aims at setting a regulatory

framework for inter-government and private company to government exchanges. NIST has

released a special publication outlining the building blocks of an incident exchange program.17

Future work would entail architecting a framework that allows a set of cooperative systems to

aggregate, summarize, and utilize ETIF and ETIT to improve resiliency.


References

1. http://www.idtheftcenter.org/ITRC-Surveys-Studies/2014databreaches.html, 2014

2. “Net losses: Estimating the global cost of cybercrime Economic impact of cybercrime II”,

Center for strategic and international studies, June 2014

3. Perlroth, N., Harris, E. A., “Cyberattack insurance a challenge for business”, http://www.

nytimes.com/2014/06/09/business/cyberattack-insurance-a-challenge-for-business.html?_

r=0 , June 2014

4. “2013 cost of data breach study: global analysis”, Research sponsored by Symantec and

independently conducted by Ponemon Institute LLC, May 2013

5. http://www.gartner.com/newsroom/id/2828722, August, 2014

6. http://money.cnn.com/2017/02/21/technology/yahoo-verizon-deal/

7. https://blogs.microsoft.com/microsoftsecure/2013/01/17/cloud-services-building-

resiliency-and-business-continuity/

8. http://www.levelcloud.net/why-levelcloud/cloud-education-center/advantages-and-

disadvantages-of-cloud-computing/

9. https://www.incapsula.com/blog/top-10-cloud-security-concerns.html

10. https://en.wikipedia.org/wiki/Sony_Pictures_Entertainment_hack

11. http://www.cnn.com/2015/08/05/politics/joint-staff-email-hack-vulnerability/

12. “Disaster resiliency: A national imperative”, The National Academies Press, 2012

13. Bruneau M., et al., “A framework to quantitatively assess and enhance the seismic resiliency

of communities”, Earthquake Spectra, 19(4), 733-752, 2003

14. Zobel C.W., et al. “Quantifying cyberinfrastructure resilience against multi-event attacks”,

Decision Sciences, Volume 43, No. 4, 687-709, August, 2012

15. Mandhata, P., Wing, J., “Measuring a System’s Attack Surface”, CMU-DS-04-102, January

2014

16. https://www.congress.gov/bill/114th-congress/senate-bill/754

17. Johnson, C, et al., “Guide to cyber threat information sharing”, NIST special publication (800-

150), April, 2016.

http://www.idtheftcenter.org/ITRC-Surveys-Studies/2014databreaches.html

http://www.nytimes.com/2014/06/09/business/cyberattack-insurance-a-challenge-for-business.html?_r=0



http://www.gartner.com/newsroom/id/2828722

http://money.cnn.com/2017/02/21/technology/yahoo-verizon-deal/

https://blogs.microsoft.com/microsoftsecure/2013/01/17/cloud-services-building-resiliency-and-business-continuity/

https://blogs.microsoft.com/microsoftsecure/2013/01/17/cloud-services-building-resiliency-and-business-continuity/

http://www.levelcloud.net/why-levelcloud/cloud-education-center/advantages-and-disadvantages-of-cloud-computing/

http://www.levelcloud.net/why-levelcloud/cloud-education-center/advantages-and-disadvantages-of-cloud-computing/

https://www.incapsula.com/blog/top-10-cloud-security-concerns.html

https://en.wikipedia.org/wiki/Sony_Pictures_Entertainment_hack

http://www.cnn.com/2015/08/05/politics/joint-staff-email-hack-vulnerability/

https://www.congress.gov/bill/114th-congress/senate-bill/754


18. http://www.npr.org/sections/alltechconsidered/2015/03/18/393868160/premera-blue-

cross-cyberattack-exposed-millions-of-customer-records

19. https://www.anthemfacts.com/

20. http://www.technobuffalo.com/2014/12/22/staples-1-16-million-credit-cards-affected-in-

cyber-attacks/

21. http://www.huffingtonpost.com/2014/09/18/home-depot-hack_n_5845378.html

22. http://www.wsj.com/articles/j-p-morgan-says-about-76-million-households-affected-by-

cyber-breach-1412283372

23. http://www.forbes.com/sites/danmunro/2014/08/18/cyber-attack-nets-4-5-million-records-

from-large-hospital-system/

24. http://www.scmagazine.com/attacks-attempt-to-access-rewardsrus-accounts/

article/401160/

25. http://www.diamondbackonline.com/news/umd-social-security-numbers-compromised/

article_b8236dea-99b6-11e3-92eb-0017a43b2370.html

26. http://www.politico.com/story/2015/07/federal-government-cyber-attack-breach-21-million-

people-affect-119918

http://www.npr.org/sections/alltechconsidered/2015/03/18/393868160/premera-blue-cross-cyberattack-exposed-millions-of-customer-records

http://www.npr.org/sections/alltechconsidered/2015/03/18/393868160/premera-blue-cross-cyberattack-exposed-millions-of-customer-records

https://www.anthemfacts.com/

http://www.technobuffalo.com/2014/12/22/staples-1-16-million-credit-cards-affected-in-cyber-attacks/

http://www.technobuffalo.com/2014/12/22/staples-1-16-million-credit-cards-affected-in-cyber-attacks/

http://www.huffingtonpost.com/2014/09/18/home-depot-hack_n_5845378.html

http://www.wsj.com/articles/j-p-morgan-says-about-76-million-households-affected-by-cyber-breach-1412283372

http://www.wsj.com/articles/j-p-morgan-says-about-76-million-households-affected-by-cyber-breach-1412283372

http://www.forbes.com/sites/danmunro/2014/08/18/cyber-attack-nets-4-5-million-records-from-large-hospital-system/

http://www.forbes.com/sites/danmunro/2014/08/18/cyber-attack-nets-4-5-million-records-from-large-hospital-system/

http://www.scmagazine.com/attacks-attempt-to-access-rewardsrus-accounts/article/401160/

http://www.scmagazine.com/attacks-attempt-to-access-rewardsrus-accounts/article/401160/

http://www.diamondbackonline.com/news/umd-social-security-numbers-compromised/article_b8236dea-99b6-11e3-92eb-0017a43b2370.html

http://www.diamondbackonline.com/news/umd-social-security-numbers-compromised/article_b8236dea-99b6-11e3-92eb-0017a43b2370.html

http://www.politico.com/story/2015/07/federal-government-cyber-attack-breach-21-million-people-affect-119918

http://www.politico.com/story/2015/07/federal-government-cyber-attack-breach-21-million-people-affect-119918

improving metrics in cyber resiliency - cloud security alliance · 2019-08-15 · improving metrics...

Documents