improving the effectiveness of · pdf fileimproving the effectiveness of csirts ... the...

October 2014

Global Cyber Security Capacity Centre: Draft working paper

Improving the effectiveness of CSIRTs

Maria Bada Global Cyber Security Capacity Centre,

University of Oxford

Sadie Creese Global Cyber Security Capacity Centre,


Michael Goldsmith Global Cyber Security Capacity Centre,


Chris Mitchell Global Cyber Security Capacity Centre, Royal Holloway, University of London

Elizabeth Phillips

Oxford University's Centre for Doctoral Training (CDT) Worcester College

Global Cyber Security Capacity Centre: Draft Working Paper

Bada, Creese, Goldsmith, Mitchell & Phillips Improving the Effectiveness of CSIRTs 2

Acknowledgements

For the completion of this report the Global Cyber Security Capacity Centre has worked closely with

a group of experts from National Organisations, Industry and Academia. Listing the organisations (in

no particular order): UK-CERT, FIRST, ITU, OECD, ENISA, and MICROSOFT.

The drafting of this report would not have been possible without the feedback and cooperation

kindly provided by a large number of experts.

At the individual level, the authors would like to thank Mr. Damir Rajnovic, Liaison Officer at FIRST,

for his feedback by reviewing the survey and sharing his valuable knowledge and information. Also,

special thanks to Ms. Rosheen Awotar-Mauree, Cybersecurity Officer at ITU, Mr. Michael Murray,

Technical Manager CSIRT Operations at Carnegie Mellon University, Ms. Robin Ruefle technical staff

of the CERT Program at the Software Engineering Institute (SEI) at Carnegie Mellon University for

their useful comments and provide of information.

Moreover, a special thank you to Mr. Chris Gibson, Director at UK-CERT, Mr. David Pollington,

Director of International Security Relations at Microsoft, Mr. Aaaron Martin, Technology policy

analyst at OECD, Mr. Marco Thorbruegge, Head of Unit Operational Security at ENISA, for their

valuable contribution.

The authors are in particular grateful for the valued contributions made by the Centre’s Working

Group 5 Members: Professor Chris Hankin, Mr. John Mallery, Professor David Pym, Professor Richard

Clayton, Professor Steve Furnell, Mr. Ian Glover, Professor Mike Walker, Dr Richard Horne, Professor

Michael Mainelli, Mr. Paul Hopkins, Mr. Bruno Brunskill, Dr Steve Marsh, Mr. John Madelin and Mr.

Steve Purser.



Contents

Abstract ................................................................................................................................................... 4

1. Introduction ........................................................................................................................................ 5

1.1 Scope and purpose ........................................................................................................................ 5

1.2 Structure of the paper .................................................................................................................. 5

1.3 Audience ....................................................................................................................................... 5

2. CSIRTs - An introduction ..................................................................................................................... 6

2.1. The role and purpose of CSIRTs ................................................................................................... 6

2.2. Terminology ................................................................................................................................. 6

2.3. Services provided by CSIRT teams ............................................................................................... 7

2.4 Categories of CSIRTs ...................................................................................................................... 8

2.5 Sectors of CSIRT operation ............................................................................................................ 8

2.6 Building a new CSIRT ..................................................................................................................... 8

2.7 Determining the authority ............................................................................................................ 9

3. Factors in assessing the effectiveness of a CSIRT ............................................................................. 10

3.1 Cooperation ................................................................................................................................ 10

3.2 Information sharing .................................................................................................................... 12

3.3. Trust Issues................................................................................................................................. 14

3.4 Other Issues ................................................................................................................................ 15

3.5 Measurement Types for Computer Security Incident Response ................................................ 15

3.6 Measures of the effectiveness of a CSIRT ................................................................................... 17

4. Ways of improving the effectiveness of a CSIRT ............................................................................... 19

5. Questions to be answered ................................................................................................................ 21

6. Next steps ......................................................................................................................................... 22

6.1. Methodology .............................................................................................................................. 22

6.1.1. Scope - Purpose .................................................................................................................. 22

6.1.2. Research Method ................................................................................................................ 22

6.1.3. Participants ......................................................................................................................... 23

7. Summary and conclusions ................................................................................................................ 24

References ............................................................................................................................................ 25

Additional Resources on CSIRTs ............................................................................................................ 27

Annex A ................................................................................................................................................. 30

Annex B ................................................................................................................................................. 37



Improving the effectiveness of CSIRTs

Maria Bada Global Cyber Security Capacity Centre, University of Oxford

[email protected]

Sadie Creese Global Cyber Security Capacity Centre, University of Oxford

[email protected]

Michael Goldsmith Global Cyber Security Capacity Centre, University of Oxford

[email protected]

Chris Mitchell Global Cyber Security Capacity Centre, Royal Holloway, University of London,

[email protected]

Elizabeth Phillips Oxford University's Centre for Doctoral Training (CDT), Worcester College

[email protected]

Abstract Following the pioneering work at Carnegie-Mellon University in the US, national Computer Emergency Response Teams (CERTs) have been established worldwide to try to address the ever-growing threats to information systems and their use. The problem they are designed to address is clearly real and formidable, although relatively little has been done to measure how effective such national responses are in mitigating the threats posed by cyber-criminals and state-sponsored cyber-attacks. The goal of this paper is to take a first step towards developing metrics which can be used to measure the effectiveness of CSIRTs. A primary motive for doing so is to enable more effective CSIRTs to be implemented, which focus on activities with the maximum impact on threat mitigation. More specifically, this paper aims to identify the ways in which a CSIRT might be deemed to be effective, and possible approaches towards developing CSIRT effectiveness metrics. It also identifies the issues that need to be addressed to realise the goal. Issues such as cooperation, data sharing and trust are discussed as crucial components of an effective CSIRT. Existing measurement types of computer security incident response (NIST, Carnegie Mellon's Software Engineering Institute) are presented before defining a set of suggested direct and indirect measures of the effectiveness of a CSIRT.

mailto:[email protected]







1. Introduction

1.1 Scope and purpose The primary mission of a Computer Security Incident Response Team (CSIRT) is to help other organizations to handle incidents occurring in computer networks, as well as provide a wider set of services. Apart from their main mission, CSIRTs need to be able to adapt to a continuous changing environment and present the flexibility to deal any unexpected incident. Today’s challenges have an impact on the effectiveness of CSIRTs’. CSIRTs have less time to react to a new unexpected threat or cyber-attack. This is the reason that there is need for quick and accurate notification of an incident. CSIRTs need effective methods to collaborate and share information, efficient mechanisms to triage incoming information and requirements for policies and procedures that are established and understood. It is a fact, that many obstacles can affect their effectiveness. This paper, discusses the subject of the effectiveness and assessment of CSIRTs. Before someone can present ways of improving the effectiveness of a CSIRT, it is vital to understand how to assess its effectiveness. Well-defined metrics are essential to determine which security practices are worth the investment. The results of this assessment will lead also to the improvement of CSIRT processes.

1.2 Structure of the paper Section 2 of this paper describes the role and purpose of CSIRTs, the services they provide, as well as information on the different sectors of CSIRT cooperation and of building a new CSIRT. Section 3, of the paper, discusses the sensitive topic of the assessment of CSIRT’s effectiveness. Issues such as cooperation, data sharing and trust are crucial in order a CSIRT to accoplish high levels of performance. Existing measurement types of computer security incident response (NIST, Carnegie Mellon's Software Engineering Institute) are presented before defining a set of direct and indirect measures. Following the types of metrics for the effectiveness of CSIRTs, this paper suggests ways for improving the performance of CSIRTs, in Section 4. As a result, questions and obstacles are being identified in Section 5 and next steps for the issue of CSIRT effectiveness are suggested in Section 6.

1.3 Audience

This paper is written primarily for Computer Security Incident Response Team (CSIRT) experts, Chief

Information Officers (CIOs), Senior Agency Information Security Officers (SAISOs) and Information

System Security Officers (ISSOs). The measures presented can be used both within government and

industry contexts.



2. CSIRTs - An introduction This section presents the role and purpose of CSIRTs, the services they provide and the various sectors they can operate in. Moreover, the basic principles of building a new effective CSIRT, as well as the importance of the parameters within which the CSIRT will be able to act are being presented. In order to be able to tackle any type of cybersecurity incident we need the capacity to be available at least in some organizational form, in particular a CSIRT. These are single organizations that present information to end users as well as organizations with the country.

2.1. The role and purpose of CSIRTs The name Computer Emergency Response Team is the historic designation for the first team (CERT/CC)1 at Carnegie Mellon University (CMU). CERT is now a registered service mark of Carnegie Mellon University that is licensed to other teams around the world. Some teams took on the more generic name of CSIRT (Computer Security Incident Response Team) to point out the task of handling computer security incidents instead of other tech support work. According to OECD recommendations of the Council on the Protection of Critical Information Infrastructure, member countries should demonstrate government leadership and commitment to protect critical information infrastructure by developing an incident response capability, such as a computer security incident response team (CERT/CSIRTs) in charge of monitoring, warning, alerting and carrying out recovery measures for CII, and mechanisms to foster closer cooperation and communications among those involved in incident response (OECD, 2008)2.

2.2. Terminology

CSIRT stands for Computer Security Incident Response Team. Various abbreviations for the same sort of terms exist:

CERT or CERT/CC (Computer Emergency Response Team / Coordination Centre)

CSIRT (Computer Security Incident Response Team)

IRT (Incident Response Team)

CIRT (Computer Incident Response Team)

SERT (Security Emergency Response Team)

WARPs (Warning Advice and Reporting Points) At the moment both terms (CERT and CSIRT) are used synonymously. In this document the term CSIRT will be used. The history of CSIRTs is linked to the existence of malware, especially computer worms and viruses. Whenever a new technology arrives, its misuse is not long in following. The first worm in the IBM VNET was covered up. Shortly after, a worm hit the Internet on 3 November 1988, when the so-called Morris Worm paralysed a good percentage of it. This led to the formation of the CERT/CC at Carnegie Mellon University under a U.S. Government contract. With the massive growth in the use of information and communications technologies over the subsequent years, the now-generic term "CSIRT" refers to an essential part of most large organisations' structures.

1 http://www.cert.org/ 2 OECD, Recommendation of the Council on the Protection of Critical Information Infrastructures, OECD

Ministerial Meeting on the Future of the Internet Economy, Seoul Korea June 2008. Retrieved form:

http://www.oecd.org/sti/40825404.pdf

http://www.cert.org/




2.3. Services provided by CSIRT teams CSIRT teams provide various services such as reactive as well as proactive. Also, part of their purpose is Artifact handling and security quality management. These services need to be realistic and reflect the financial, labour and technical resources available to a nation. A more analytical list of the CSIRT services is presented below (Table 1). A CSIRT needs to act as a focal point for incident reporting and to be easily reached by users. A CSIRT has three essential attributes a) a central location in relation to its constituency b) an educational role with regard to computer security c) an incident handling role (Javaid, 2013). The accumulated experience of the personnel in a CSIRT is crucial, both in terms of responding to incidents and of educating others. CSIRTs3 have as their main responsibility to detect and inform about vulnerabilities, to make patches available to organisations and to the general public, to provide technical assistance in dealing with computer incidents and to coordinate response in emergencies. CSIRTs can operate on a nation-wide basis, inside or outside of the governmental sector. Table 1. Services provided by CSIRT’s

CSIRT Services list from CERT/CC4 The European Commission5 has presented also the requirements and tasks of a Computer emergency Response Team (CERT). ENISA6 also released on November 2013 a report titled Good practice guide for CERTs in the area of Industrial Control Systems - Computer Emergency Response Capabilities considerations for ICS. This report, “builds upon the current practice of CERTs with responsibilities for ICS networks, and also on the earlier work of ENISA on a baseline capabilities

3 OECD, Studies in Risk Management, Norway Information Security, 2006. Retrieved from:

http://www.oecd.org/norway/36100106.pdf 4 CSIRT Services list from CERT/CC: http://www.cert.org/csirts/services.html 5 European Commission, 2013 http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2013:0048:FIN:EN:PDF 6 ENISA, Good practice guide for CERTs in the area of Industrial Control Systems - Computer Emergency Response Capabilities considerations for ICS, December 2013. http://www.enisa.europa.eu/activities/cert/support/baseline-capabilities/ics-cerc/good-practice-guide-for-certs-in-the- area-of-industrial-control-systems/at_download/fullReport

Reactive Services

Proactive Services

Artifact

Handling

Security Quality

Management

Alerts and Warnings Incident Handling Incident Analysis Incident Analysis Incident Response Support Incident Response Coordination Incident Response on Site Vulnerability Handling Vulnerability Analysis Vulnerability Response Vulnerability Response Coordination

Announcements Technology Watch Security Audits or Assessments Configuration and Maintenance of Security Development of Security Tools Intrusion Detection Services Security Related Information Dissemination

Artifact Analysis Artifact Response Artifact Response Coordination

Risk analysis Business Continuity and Disaster Recovery Security Consulting Awareness Building Education/Training Product Evaluation or Certification

http://www.oecd.org/norway/36100106.pdf

http://www.cert.org/csirts/services.html

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2013:0048:FIN:EN:PDF

http://www.enisa.europa.eu/activities/cert/support/baseline-capabilities/ics-cerc/good-practice-guide-for-certs-in-the-%20%20%20%20%20%20area-of-industrial-control-systems/at_download/fullReport

http://www.enisa.europa.eu/activities/cert/support/baseline-capabilities/ics-cerc/good-practice-guide-for-certs-in-the-%20%20%20%20%20%20area-of-industrial-control-systems/at_download/fullReport



scheme for national/ governmental (n/g) CERTs,” without prescribing which entity should provide these services for the EU. The good practices guide divides ICS-CERC provisions into four categories: mandate capabilities, technical operational capabilities, organisational operational capabilities, and co-operational capabilities.

2.4 Categories of CSIRTs

General categories of CSIRTs include7:

Internal or organizational CSIRTs - provide incident handling services to their parent organization (e.g. a university).

National CSIRTs – coordinate and facilitate the handling of incidents for a particular country, or economy.

Analysis Centers – focus on synthesizing data from various sources to determine trends and patterns in incident activity. This information can then be used to help predict future activity or provide early warning when current activity matches a set of previously determined characteristics.

Vendor Teams – coordinate with organizations who report and track vulnerabilities.

Incident Response Providers – provide incident handling services as a product to other organizations. These are sometimes referred to as Managed Security Service Providers (MSSPs).

Various global and regional organizations devoted to incident management collaboration and coordination have been created. This includes organizations such as the:

Forum of Incident Response and Security Teams8

2.5 Sectors of CSIRT operation There can be more than one CSIRT in a country serving the interest of various constituencies for example the academic, banking sectors, the commercial sector, CIP/CIIP Sector, governmental/national sector, military, energy sector, financial sector and within organisation. These CSIRTs are focussed on and provide services and support to their defined constituency for the prevention of, handling, and response to cybersecurity incidents. However it is also possible for a country to designate an entity as a national CSIRT to serve a principle entity serving Government or government-related organisations.

2.6 Building a new CSIRT In order to create an effective CSIRT, Carnegie Mellon University (CMC, J Haller, 2011) believe that there are four core principles all CSIRTs must have:

Technical Excellence: The National CSIRT should have the most up to date resources and advice and in order to maintain this advantage, the advice they give must be sound which requires high levels of technical excellence. This may lead to the CSIRT only being initially with a small number of good quality capabilities rather than lots of poor quality capabilities.

7 Creating and Managing Computer Security Incident Handling Teams (CSIRTs), CERT Training and Education Networked

Systems Survivability Software Engineering Institute Carnegie Mellon University, 2008. http://www.first.org/conference/2008/papers/killcrece-georgia-slides.pdf

8 http://www.first.org/

http://www.first.org/conference/2008/papers/killcrece-georgia-slides.pdf

http://www.first.org/



Trust: If the organizations and end users do not explicitly trust the CSIRT then they will be unable to share data with the CSIRT and will not be able to use all the facilities on offer. The trust is crucial for partner organisations and the organisations themselves would want confirmation that the CSIRT can handle sensitive information responsibly.

Resource Efficiency: The CSIRT must be constantly adapting by analysing potential new threats and their potential impact. This will then help to steer the allocation of funding sources to test, which treats and incidents are truly of interest to the CSIRT.

Cooperation: The CSIRT should cooperate as fully as possible (taking into account the sensitivity of some of their clients’ data) with national stakeholders, government and other National CSIRTs so that the knowledge can be shared and they can collaborate on complex problems.

Before the real work begins, it is crucial to identify key partners and Sponsors to ensure the financial security of the CSIRT. After this has been established, it is then necessary to determine any limiting factors such as time commitment, skill level of staff and the physical infrastructure available9.

2.7 Determining the authority Depending on the purpose of the CSIRT and the its sponsor, the CSIRT may be capable of prescribing or mandating particular actions after cyber-attacks and may be able to enforce other security measures. However, in some instances government approval/advice may be required first before conducting any action. The parameters within which the CSIRT will be able to act will depend on the specific nation’s laws, cultures and customs. The precise nature of the CSIRT may determine the level of cooperation and sharing of sensitive data as some organizations may be reluctant to disclose information if they believe the CSIRT to be too self-governing.

9 Grobler Marthie and Bryk Harri, 2010. http://icsa.cs.up.ac.za/issa/2010/Proceedings/Full/17_Paper.pdf

http://icsa.cs.up.ac.za/issa/2010/Proceedings/Full/17_Paper.pdf



3. Factors in assessing the effectiveness of a CSIRT Before we can consider ways of improving the effectiveness of a CSIRT it is vital to understand how we might assess its effectiveness. Well-defined metrics are essential to determine which security practices are worth the investment. The CSIRT will need to develop a mechanism to evaluate the effectiveness of its practise. This should be done in conjunction with management and the constituency. The results of this assessment will lead also to the improvement of the processes of a CSIRT.

These measures concentrate on the results of assessments concerning information security controls. Effectiveness as well as efficiency measures address two aspects of security control implementation results: the robustness of the result itself (effectiveness) and the timelines of the result (efficiency). These measures can provide important information for security decision makers in order to improve the performance of CSIRTs, and they help by determine the effectiveness of security controls.

By measuring the effectiveness of information security, there can be10:

a) Increase of accountability. Measures can help by identifying specific security controls that are implemented incorrectly or are ineffective.

b) Improvement of Information Security Effectiveness. Measures of information security can determine the effectiveness of implemented information security processes, procedures by interrelating results of various activities and events to security controls and investments.

c) Demonstration of Compliance. Organizations can demonstrate compliance with applicable laws and regulations by maintaining an information security measurement program.

International Telecommunication Union (ITU),11 in collaboration with IMPACT, is helping countries to establish their National Computer Incident Response Team (CIRT), which serves as a national focus point for coordinating cybersecurity incident response to cyber-attacks in the country. The objective of the Assessment of a CSIRT is to define the readiness to implement a national CSIRT. Part of their assessment includes the incident response capabilities of a country and the existence of an intrusion detection service offered to the constituents. Issues such as cooperation, data sharing and trust are crucial in order a CSIRT to accomplish high levels of performance. Exisitng measurement types of computer security incident response (NIST, Carnegie Mellon's Software Engineering Institute) are presented before defining a set of direct and indirect measures.

3.1 Cooperation

The OECD report (2005)12, presents as major survey finding between member states the importance of international cooperation for fostering a culture of security and the role of regional facilitating

10 NIST, Performance Measurement Guide for Information Security, Special Publication 800-55 Revision 1, 2008.

http://csrc.nist.gov/publications/nistpubs/800-55-Rev1/SP800-55-rev1.pdf 11 ITU http://www.itu.int/en/ITU-D/Cybersecurity/Pages/Organizational-Structures.aspx ITU, National Cybersecurity Strategy Guide. Frederick Wamala, September 2011. Retrieved from:

http://www.itu.int/ITU-D/cyb/cybersecurity/docs/ITUNationalCybersecurityStrategyGuide.pdf 12 OECD, The Promotion of a Culture of Security for Information Systems and Networks in OECD Countries, Working Party on Information Security and Privacy, December 2005. Retrieved from: http://www.oecd.org/internet/ieconomy/35884541.pdf

http://csrc.nist.gov/publications/nistpubs/800-55-Rev1/SP800-55-rev1.pdf

http://www.itu.int/en/ITU-D/Cybersecurity/Pages/Organizational-Structures.aspx

http://www.itu.int/ITU-D/cyb/cybersecurity/docs/ITUNationalCybersecurityStrategyGuide.pdf

http://www.oecd.org/internet/ieconomy/35884541.pdf



interactions and exchanges. International cooperation is consolidated in the area of cybercrime and Computer Security Incident Response Teams (CSIRTs). International cooperation is considered an integral and important part of the activities on national CSIRTs, and several countries have already established operational networks through which they exchange information and best practices. Most countries cooperate at the regional (European TF-CSIRT and EGC, APCERT) or global level (FIRST). ENISA13 while discussing the subject of the effectiveness of CSIRT’s, addressed the topic of multi various cooperation between CSIRTs. In order to understand the role of a cooperation between CSIRTs, ENISA discusses the benefits that a cooperation would offer in improving the services offered by CSIRTs. Also, ENISA focuses in possible barriers that can inhibit such a process. Specifically, four areas of benefits from a possible cooperation were identified.

Incident Handling, is a very important benefit resulting from the cooperation between CSIRTs. The information exchanged during incident handling is very sensitive and contains data such as electronic evidence, analysis of malicious code, attacked organizations etc. Exchanging these data in a secure way, reacting and providing information can result in setting up a long term exchange of incident data. This can lead to improving the quality of the incident handling process.

Project conducting, stems from cooperation between CSIRTs. Cooperation leads to an understanding of common interests and goals. An example is the eCSIRT.net project14.

Resource and information sharing, such as knowledge and experience sharing, staff exchange, technology sharing can offer CSIRTs the opportunity to exchange not only technical solutions and techniques but also actual software.

Social networking, contributes in building trusted relationships between CSIRTs. Workshops, conferences and meetings can be a first step to closer cooperation.

Cooperation between CSIRTs can be further facilitated by various stakeholders whose mission is to foster such a cooperation (See above Section 4). Also, ENISA15 while providing an overview of existing mechanisms supporting Computer Security Incident Response Teams (CSIRTs) to deploy capabilities necessary for their operations and their maturity level, introduced these mechanisms according to the CSIRT maturity levels based on eight criteria including requirements that CSIRTs must meet.

Type of approach (organisation model) An organisation's approach is an important determinant of the relationships that it establishes with CSIRTs and the services that it provides. The approach that these organisations employ provide information about matters such as: how an organisation interacts with CSIRTs, both members and non-members, what member or associated CSIRTs can expect from an organisation, what role an organisation envisions itself playing in the broader CSIRT community, how an organisation sees itself growing and evolving over time, where an organisation can obtain operational and substantive input and how an organisation secures funding.

Requirements for CSIRTs A given organisation’s mechanism may impose requirements on CSIRTs seeking to utilise the mechanism or to associate with the organisation. An organisation must also decide which

13 ENISA, 2006, CERT_cooperation_ENISA.pdf 14 eCSIRT.net project, http://www.ecsirt.net/ 15 ENISA https://www.enisa.europa.eu/activities/cert/support/files/updated-recommendations-2012

http://www.enisa.europa.eu/activities/cert/background/coop/files/cert-cooperation-and-its-further-facilitation-by-relevant-stakeholders/at_download/fullReport

http://www.ecsirt.net/

https://www.enisa.europa.eu/activities/cert/support/files/updated-recommendations-2012



specific requirements it will impose on CSIRTs in its mechanisms. An organization can create its own unique list of requirements that it expects a partner or member to meet, it can use existing good practices guide for CSIRTs and require a partner or member to meet either all of these practices or a certain percentage of them or, leave the decision about cooperation with or involvement for a CSIRT to its existing members, and allowing them to decide by vote on whether the CSIRT meets its standards.

The European Commission16 has presented also the requirements and tasks of a CSIRT. Part of the requirements of a CSIRT is its cooperation with the private sector. In practise data protection, data retention and obligations to work with law enforcement constitute the greatest set of challenges for cross-border CSIRT cooperation17. Barriers to cross-border CSIRT cooperation are (Silicki and Maj, 2008)18:

Lack of service level agreement between CSIRTs (lack of rules for strict reaction time can slow down cooperation)

Differences in legal systems (the legal regime of each country affects sharing of data)

Lack of standards of CSIRT cooperation

Insufficient organisational, political and financial support

3.2 Information sharing

CSIRTs play an important role at national level, in terms of Critical Information Infrastructure Protection. Information on cyber security should be shared because this information a) can lead governments and policy-makers to better formulate policy, b) is necessary for industry risk management, corporate governance and compliance and c) is necessary in order citizens to take appropriate measures. ENISA19 has dealt with the issue of the threat and incident information exchange and sharing practices used among CSIRTs in Europe, especially, but not limited to, national/governmental CSIRTs. It identifies the functional and technical gaps that limit threat intelligence exchange between national/governmental CSIRTs and their counterparts in Europe, as well as other CSIRTs within their respective countries. Interactions between CSIRTs can include asking other teams for advice, disseminating knowledge of

problems and working cooparatively to resolve an incident affecting one or more of the CSIRT’s

constituencies. Response teams have to decide what kinds of agreements can exist between them so

as to share but safeguard information, whether this information can be disclosed and to whom. A

peering agreement though, refers to simple co-operation between CSIRTs, while a team contacts

another and asks for help and advice.20

16 European Commission, 2013 http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2013:0048:FIN:EN:PDF 17 ENISA, A flair for sharing – encouraging information exchange between CERTs, A study into the legal and regulatory

aspects of information sharing and cross-border collaboration of national/governmental CERTs in Europe, November 2011. http://www.enisa.europa.eu/activities/cert/support/fight-against-cybercrime/legal-information-sharing

18 Silicki, K., Maj, M. (2008). 19 ENISA, Detect, SHARE, Protect - Solutions for Improving Threat Data Exchange among CERTs, 2013, https://www.enisa.europa.eu/activities/cert/support/data-sharing 20 Brownlee, N., and Guttman, E. Expectations for Computer Security Incident Response. Best Current Practice, ISF, Network Working Group RFC 2350, June 1998. Retrieved from: http://tools.ietf.org/html/draft-ietf-grip-framework-irt-04

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2013:0048:FIN:EN:PDF

http://www.enisa.europa.eu/activities/cert/support/fight-against-cybercrime/legal-information-sharing

https://www.enisa.europa.eu/activities/cert/support/data-sharing



Local as well as global detection of threats when accompanied by trusted forms of information exchange, can lead to global prevention of cyber-attacks. Sharing the information of the successful identification of an incident is very important, since it saves time and effort but also increases the cooperation between CSIRTs. This is why, secure and effective information exchange is so crucial. The key problems for effective information sharing are legal and technical barriers as well as lack of interest from cybersecurity stakeholders. Moreover, often the heavy workload can lead to problems in effective information sharing due to lack of time. As a result, the lack of post-processing of important incidents leads to ineffective information sharing. ENISA21 presented various issues regarding information sharing. The barriers to cooperation between CSIRTs are a) poor quality of information b) poor management of information sharing c) misaligned incentive stemming from reputational risks d) uncertainty about senior level awareness of cyber security e) dis-incentivised private sector to disclose information due to possible reputational damage. A study conducted by ENISA (2011)22, tried to identify whether there are any national or

international legal and regulatory factors which could affect cross-border information sharing

between CSIRTs. The study showed that their competences can be affected by their national laws

and their own statutes or operating rules, depending on the legal basis of their formation. A national

CSIRT may have a clearer legal basis for collection and process of personal data relating to suspicious

activities than a private sector CSIRT.

ENISA defines basic requirements for improved communications interoperable with existing solutions in order to improve information sharing. Better utilisation of current communication tools and practices is needed. Local detection of incidents accompanied with trusted forms of information exchange, can ultimately lead to improved prevention of cyber incidents on a global scale. The European Commission (Article 9)23 includes measures to a secure information sharing system for Member States. These include the exchange of sensitive information through a secure infrastructure at national level and the existence of adequate technical, financial and human resources and processes for their competent authority and CSIRT. The Information Sharing Framework (ISF, MACCSA, 2013)24 provides the guidance to establish the capability to increase an organisation’s cyber Situational Awareness, enabled by sharing information across a trusted community of interest, to achieve Collaborative Cyber Situational Awareness (CCSA). It includes an Information Sharing Model and an Information Management Model. For effective decision making, collaborative governance, federated access control and management of information quality are required. Preparation and good decision-making depend on quality CCSA. Good CCSA depends on sharing information quickly about the status of cyber security controls, potential threats and vulnerabilities, alerts and incidents. Cyber threat and incident information is shared today between working-level organisations, such as CSIRTs, without employing any

21 ENISA (2010), Incentives and Barriers to Information Sharing. http://www.enisa.europa.eu/activities/Resilience-and-

CIIP/public-privatepartnership/information-sharingexchange/incentives-and-barriers-to-information-sharing 22 ENISA, A flair for sharing – encouraging information exchange between CERTs, A study into the legal and regulatory

aspects of information sharing and cross-border collaboration of national/governmental CERTs in Europe, November 2011. http://www.enisa.europa.eu/activities/cert/support/fight-against-cybercrime/legal-information-sharing

23 European Commission, Article 9, 2013 http://eurlex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2013:0048:FIN:EN:PDF

24 MACCSA, Multinational Alliance for Collaborative Cyber Situational Awareness. Information Sharing Framework (ISF), 20

November 2013, version 2.4. https://www.terena.org/mail-archives/refeds/pdfjJz1CRtYC4.pdf

http://www.enisa.europa.eu/activities/Resilience-and-%20CIIP/public-privatepartnership/information-sharingexchange/incentives-and-barriers-to-information-sharing

http://www.enisa.europa.eu/activities/Resilience-and-%20CIIP/public-privatepartnership/information-sharingexchange/incentives-and-barriers-to-information-sharing


http://eurlex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2013:0048:FIN:EN:PDF

https://www.terena.org/mail-archives/refeds/pdfjJz1CRtYC4.pdf



standards-based approach or methodology, or much security automation. A top-down standards-based Information Sharing Model is proposed focusing on security controls based on the policies, procedures and mechanisms for federated trust. It includes Taxonomies for information interoperability, from ENISA (European Network and Information Security Agency), US DHS (Dept of Homeland Security) and IETF (Internet Engineering Task Force). The Information Sharing Model describes the means required for sharing information, proactive and reactive, alerts and warnings, best practices, information on security quality management and proactive artefact handling. The Information Management Model is focused on ensuring the quality of the shared information, which is vital to good decisions. Information needs to be timely and accurate with the right degree of richness. The ISF proposes a mesh of compliant Hubs, Nodes and CSIRTs to coordinate information sharing and maximise CCSA. The Model is based on existing federated secure collaboration capabilities in defence, intelligence and industry, comprising independent entities bound together by Information Sharing Agreements and further united by collaborative, community-centric governance authorities. According to this model, an Information Sharing Agreement25 can be made between two or more

collaborating organisations which describe verification and compliance methodologies. This

agreement can define: a) the scope and type of information to be shared b) how will this

information be used, shared and stored c) the roles and responsibilities of the organisations d) what

access control model or policy will be used e) the procedures and supporting legal and policy

documents to be followed to enforce compliance f) what legal timeframes exist, that can affect the

use of information g) what taxonomies and data labels will be used.

Collaborating organisations need a taxonomy so that they can understand and use information. The

taxonomy will be used for the preparation, the exchange and use of information. The Taxonomy is a

set of agreed definitions for data and rules for their use.

3.3. Trust Issues

Trust can be one of the biggest obstacles to enhanced and effective communication between CSIRTs but also between CSIRTs and other stakeholders. Lack of trust between stakeholders can lead even to lack of sharing security incident information. This component is of vital importance for the previous ones mentioned, cooperation and information sharing. According to Messenger (2005)26 trust in public/private partnerships has a very significant role which can be enhanced through frequency of contact between counterpart individuals, identification and sharing of common intentions and objectives, or technical credibility of technical staff. A first model to build trust between organisations is creating internship positions from one team to the other. Also, the discussion of incident triage, case transfer flow and which cases should be handled to the police enhances the collaboration between teams and increases the level of trust between them. At an international level, trust can again be built through regular meetings, between individuals involved. Also, encryption is very important for trustworthy communication and it should be highlighted from each counterparty27.

25 MACCSA, Multinational Alliance for Collaborative Cyber Situational Awareness. Information Sharing Framework (ISF), 20

November 2013, version 2.4. https://www.terena.org/mail-archives/refeds/pdfjJz1CRtYC4.pdf 26 Messenger, M. (2005)




According to the Information Sharing Framework (ISF, MACCSA, 2013) Trust depends on an AAA

Model: Authentication (Are you who you claim you are?), Authorisation (Do you have permission to

undertake the activities?) and Accountability (Can you evidence compliance in any court of law?).

3.4 Other Issues

The effectiveness of CSIRTs can be limited due to the growing work load and limited resources (Gonzalez and Kossakowski, 2005).28 To proactively avoid incidents, more interaction and information exchange regarding vulnerabilities, solutions, workarounds between vendors, other CSIRTs and other organisations, will also be necessary to make the impact of proactive efforts more effective. The typical work overload situation in a CSIRT, limits its effectiveness and can lead it into the “Capability Trap” that forces the CSIRT to work harder and harder. This pattern further reduces the capability of a CSIRT to improve. A CSIRT that has over-stretched its resources over a long time period must be prepared to go through a worse-before-better scenario to escape the “Capability Trap”. ‘’Such a transition process can be quite painful to the CSIRT and its surrounding environment, for example, through adjustments to scope of service to release resources for improvement’’ (Gonzalez and Kossakowski, 2005).

3.5 Measurement Types for Computer Security Incident Response

The issues of determining how secure an organization or any other environment is and how can someone truly test the quality of the organization’s security program after a crisis, are crucial. Organizations have to ask themselves29,

How many resources do they need

How can they justify the cost of the new measures

When can an organization know it is safe

What is the organization’s posture and how can it be compared to others with best practise standards

The key to security metrics is obtaining measurements that have the following ideal characteristics:

They should measure organizationally meaningful things

They should be reproducible

They should be objective and unbiased

They should be able to measure some type of progression toward a goal There are existing publications which refer to how we can measure the performance and create accountability for the capabilities of a CSIRT.

27 ENISA The Fight Against Cybercrime – Cooperation between CERTs and Law Enforcement Agencies to fight against

cybercrime, 2012. http://www.enisa.europa.eu/activities/cert/support/fight-against-cybercrime/supporting-fight-against-cybercrime/cooperation-between-certs-and-law-enforcement-agencies-in-the-fight-against-cybercrime-a-first-collection-of-practices

28 Gonzalez J. Johannes Wiik Jose, Kossakowski Klaus-Peter, 2005 http://resources.sei.cmu.edu/asset_files/WhitePaper/2005_019_001_53057.pdf 29 Chapin and Akridge, 2005.

http://www.enisa.europa.eu/activities/cert/support/fight-against-cybercrime/supporting-fight-against-cybercrime/cooperation-between-certs-and-law-enforcement-agencies-in-the-fight-against-cybercrime-a-first-collection-of-practices



http://resources.sei.cmu.edu/asset_files/WhitePaper/2005_019_001_53057.pdf



The NIST Special Publication 800-55 Revision 1 (2008)30 defined measurement types for information security as implementation, effectiveness/efficiency, and impact. The authors established that these are measurement types but they are actually purposes or the drive for measuring information security.

In another NIST publication, NIST Special Publication 800-61 Revision 1 (2008)31 possible metrics were proposed a) the number of incidents handled, b) time per incident, c) objective assessment of each incident, and d) subjective assessment of each incident. These metrics are very practical but suggest only a small portion of possible metrics and measurement types for measuring CSIRT.

A technical report from Carnegie Mellon's Software Engineering Institute (SEI 2007),32 measured incident management based on common functions and processes within CSIRT work flow.

Sritapan, et al (2011),33 developed a metrics framework for incident response to serve as an internal analysis, in order to support the incident reporting improvement and strengthen the security posture for an organization’ s mission.

The report on Improving the Evidence Base for Information Security and Privacy Policies (OECD, 2012)34 indicates that many CSIRTs already generate statistics based on their daily activities, including statistics on the number of alerts and warnings issued or incidents handled. CSIRTs also collect data or potentially have access to data that could be used to generate statistics on other relevant phenomena if appropriate guidance was available. However, the international comparability of these existing and potential statistics raises many challenges. Developing more robust statistical indicators so as to improve the international comparability of CSIRT statistics would enable better informed cybersecurity policy making.

The report ‘’Improving the International Comparability of Statistics Produced by Computer

Security Incident Response Teams’’ (OECD, 2014)35 presents the ability of CSIRTs to report

data about their constituencies, the size of the networks and users under their

responsibility, organisational capacity and incidents, as well as information on the quality of

these responses. The main objective of the OECD project is to improve the international

comparability of statistics produced by CSIRTs, in order to support statistical indicators to

inform cybersecurity policy making. The statistical indicators, which were tested refer to a)

the capacity of a national CSIRT (budget, personnel, skills, automation, requests for

assistance, targeted mitigation, co-operation) and b) the incidents being handled by the

CSIRT (phishing, breach of availability, malware). It is also proposed the OECD to further

explore other pertinent topics such as the most pressing vulnerabilities increasing

cybersecurity risk, the severity of new malware variants and the perceived economic and

social impacts of cybersecurity risks. Although, the results confirm that the proposed

30 NIST Special Publication 800-55 Revision 1, 2008. http://csrc.nist.gov/publications/nistpubs/800-55-Rev1/SP800-55-rev1.pdf 31 NIST Special Publication 800-61 Revision 2, 2012. http://csrc.nist.gov/publications/nistpubs/800-61rev2/SP800-61rev2.pdf 32 SEI 2007, http://www.dtic.mil/dtic/tr/fulltext/u2/a468688.pdf 33 Sritapan, Vincent, Stewart, Walter, Zhu, Jake, Rohm, C. E. Tapie, 2011. http://web.a.ebscohost.com/abstract?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=15435970&AN=8

2214678&h=iy9i2kF2J%2bamTVTRwH%2bIlaWSOV%2fzVtCuvbcifQhPd4SUP%2f1GDHbnx%2fg%2b2PWuhW0%2fQ4Br5JAbo0fVtpLnjIscYQ%3d%3d&crl=c

34 OECD, Improving the Evidence Base for Information Security and Privacy Policies: Understanding the Opportunities and Challenges related to Measuring Information Security, Privacy and the Protection of Children Online, 2012, Paris. 35 OECD, Improving the International Comparability of Statistics Produced by Computer Security Incident Response Teams, 18 June 2014.


http://csrc.nist.gov/publications/nistpubs/800-61rev2/SP800-61rev2.pdf

http://www.dtic.mil/dtic/tr/fulltext/u2/a468688.pdf

http://web.a.ebscohost.com/abstract?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=15435970&AN=82214678&h=iy9i2kF2J%2bamTVTRwH%2bIlaWSOV%2fzVtCuvbcifQhPd4SUP%2f1GDHbnx%2fg%2b2PWuhW0%2fQ4Br5JAbo0fVtpLnjIscYQ%3d%3d&crl=c





statistical indicators are good candidates for inclusion in the statistical guide, further

research is needed in order to determine the suitability of the remaining indicators.

3.6 Measures of the effectiveness of a CSIRT

o Impact measures36

These measures are used in order to assess the impact of a CSIRT’s mission. Examples of indirect

measures would be:

Volume of information output by the CSIRT. Number of accesses of the information provided by the CSIRT. Number of links to the CSIRT from web pages. Ranking of the CSIRT web pages on search engines. Enquiries to the CSIRT from its constituency. Amount of information reported to constituency about computer security issues or ongoing activity. Ranking of target region on lists of cybersecurity health. Amount of money lost to cybersecurity attacks in region (normalised to take account of size of the region).

o Incident Response Quality: Number of reported incidents Number of incidents handing Speed of initial response to front line of event Speed of identification of incident nature / attack characteristics Ability to remove threat from system Digital forensics capability Ability to achieve work-through attack status in face of incidents Stakeholder level of awareness (communications ability) Ability to cooperate with other CSIRT teams in support of investigations and

prosecutions (the latter requiring the evidence capability) Response time or time-to-live of an incident (high performers have short mean times). Percentage of security incidents that exploited existing vulnerabilities with known

solutions, patches, or workarounds37. The intent is to measure the percentage of successful attacks that were handled in accordance with policy, defined procedures, and in-place processes in a disciplined repeatable, predictable manner. This assumes that well-defined processes for incident management exist.

Percentage of security incidents that were managed in accordance with established policies, procedures, and processes38.

Percentage of vulnerability assessment findings that have been addressed. Mean times between incidents39 (high performers have long mean times). Frequency and damage caused by security breaches time to recover from a security breach.

36 CSIRTs, Software Engineering Institute Carnegie Mellon University, 2008. http://www.first.org/conference/2008/papers/killcrece-georgia-slides.pdf 37 http://net.educause.edu/ir/library/pdf/CSD3661.pdf

http://resources.sei.cmu.edu/asset_files/Podcast/2008_016_102_67465.pdf 38 http://www.cert.org/ 39 ITPI http://www.itpi.org/


http://net.educause.edu/ir/library/pdf/CSD3661.pdf

http://resources.sei.cmu.edu/asset_files/Podcast/2008_016_102_67465.pdf


http://www.itpi.org/



Number of vulnerability exploits for organisations and/or individuals in the target audience for the CSIRT. Percentage of systems with critical information assets or functions that have been assessed for vulnerabilities in accordance with policy since the last reporting period.

o Situational Awareness Capability: Access to threat and attack data feeds Synthesis of data feeds into single data model (indicator of fusion capability) Support for threat and attack intelligence capability Translation into information for distribution to stakeholder community Translation into actionable information for incident response Integration of feedback into refinement of architectures and best practices Involvement in disaster recovery planning

o Measures on general capability of CSIRTs

Effectiveness of a Government to support a CSIRT policy Existence of a portal on CSIRTs Number of staff members Existence of enough funding Existence of specialised legal and RP experts among staff members Existence of specialised personnel in reverse engineering or digital forensics Security posture of the organization

o Outreach Mission

Education/Training of staff members Training in specialised technical aspects Promotion of stakeholders’ awareness of existing national CSIRTs and their

responsibilities



4. Ways of improving the effectiveness of a CSIRT Since threats are becoming more complicated, CSIRTs have to be in a continuous improvement

position.40 Evaluation of the progress of a CSIRT can be accomplished through the revision of the

design and implementation plans, the evaluation of its capabilities and services and of course

through feedback mechanisms from its constituency, internally, and externally. After, finding out

what works and what doesn’t, a CSIRT team can better build improvement plans. ITPI's work reveals

that disciplined change and configuration management are predictors of a stable and security

computing environment.

Although security experts say that they can identify security incidents within hours, it takes about a month to work through the entire process of incident investigation, service restoration and verification. The identification of a security incident is only a small part of the overall process of handling that incident. Investment is critical for effective cyber incident response programs. Also, a crucial aspect is that usually management is largely unaware of cyber security threats41. Many organizations do not invest in CSIRTs because, often they are viewed as responsive rather than preventative. Moreover, limited budget leads to spenditure of money on actions which will stop breaches from occurring in the first place. However, incident response teams don’t merely clean up security breaches, they seek to understand them. The effectiveness of CSIRTs might be prohibited because of the different objectives between CSIRTs and law enforcement. For example, a CSIRT needs to meet its operational obligations and respond to a security incident while the main focus of the law enforcement might be to shut down the system to preserve evidence (Sommer, 2009)42. Usually, CSIRT programs are made up of experienced and credentialed experts, but lack full-time staff. Many organizations do not have a way to measure the effectiveness of their incident response teams, or to assess the readiness of their incident response teams on an ongoing basis. Also, most organizations keep incident response in-house and are not sharing threat intelligence and indicators. Another crucial problem, is that organizations do not have a pre-defined public relations and analyst relations plan that they can put into motion in the event of a material data loss that needs to be publicly disclosed or a multi-disciplinary insider threat management program43. Improvement of the effectiveness of a CSIRT could be fostered through:

The improvement of awareness of CSIRTs in target audience. This might be done by various

ways, such as web sites, conferences, white papers. The primary audiences such as the

technical staff and management of governmental and other institutions that operate a CSIRT

or any group or team that handles information or network security incidents and other

security professionals, can benefit from learning more about the work and mission of CSIRTs.

Improving means of communication to target audience through multiple communication channels can improve the effectiveness of CSIRTs. The accumulated contact between the

40 Creating and Managing Computer Security Incident Handling Teams (CSIRTs), CERT Training and Education Networked

Systems Survivability Software Engineering Institute Carnegie Mellon University, 2008. http://www.first.org/conference/2008/papers/killcrece-georgia-slides.pdf 41 Ponemon Institute LLC, January 2014. http://www.lancope.com/ponemon-incident-response/ 42 Sommer, P.M. (2009) 43 Ponemon Institute LLC, January 2014. http://www.lancope.com/ponemon-incident-response/


http://www.lancope.com/ponemon-incident-response/




CSIRT and its partners such as users, vendors, investigative forces, requires continuity to be successful.

Improvement of the flow of vulnerability information to CSIRTs. As mentioned above (Section 5) trust and cooperation between CSIRTs can be crucial components of its effectiveness enhancement.

Improving use of information provided by CSIRTs. As mentioned, (Section 5) trust is of vital importance in information sharing.

Improving trust in CSIRTs to ensure that (a) as much information is provided to CSIRTs as possible, and (b) take-up (action on) of information provided by a CSIRT is maximised.

Identify ways to support operational coordination between CSIRTs.

Improving cooperation between CSIRTs to improve information exchange (thereby maximising information available to CSIRTs). ENISA,44 presented the National Cyber Security Strategies: Practical Guide for Development and Execution, (Practical Guide) released in December 2012. In this document, “ENISA has identified a set of concrete actions, which if implemented will lead to a coherent and holistic national cyber-security strategy.” The definition of the role of the existing national CSIRTs as well as the cooperation and share of information between CSIRTs at national and international level, are presented as an important part of the development of a clear governance structure.

Sharing threat indicators with third-party organizations to foster collaboration.

Possible regulation and/or legislation to make organisations take action on CSIRT warnings and/or increase their liability so they feel obliged to take warnings seriously.

Better enforcement of existing legislation (including data privacy legislation) to force organisations to take privacy and security seriously.

Ensure that EU-level legislation takes account of the scope of national/governmental CSIRTs (Data Protection Directive 95/46/EC for the use of personal data against terrorism and crime).

Designate national/governmental CSIRTs on a specific regulatory basis to provide them with a clearer mandate.

A specific threshold that incidents must pass according to agreed indicators in order they to be addressed by national/governmental CSIRT.

Investigate measures to encourage cross-border information exchange (confidentiality charters or means to limit liability of CSIRT incident response activities).

Existence of enough resources. Growing work load and limited resources (Gonzalez and Kossakowski, 2005)45 can deter the effectiveness of a CSIRT.

Training bodies and Institutions, such as Centres of Excellence, Partnerships Training and Education Centres, Partnership for Peace Consortium of Defence Academies and Security Studies Institutes, can further facilitate the effectiveness of CSIRTs.

Further research into the mechanisms of cross-border CSIRT cooperation in order to explore the processes of cross-border incident response.

44 ENISA, National Cyber Security Strategies, Practical Guide on Development and Execution, December 2012.

http://www.enisa.europa.eu/activities/Resilience-and-CIIP/national-cyber-security-strategies-ncsss/national-cyber-security-strategies-an-implementation-guide/at_download/fullReport

45 Gonzalez J. Johannes Wiik Jose, Kossakowski Klaus-Peter, 2005 http://resources.sei.cmu.edu/asset_files/WhitePaper/2005_019_001_53057.pdf






5. Questions to be answered This section identifies a few questions which need to be answered and obstacles which have to be

overcome in order to improve the effectiveness of CSIRTs.

The assessment of the effectiveness of CSIRTs has to take into account a country’s resources, policy and legislation. It is obvious, that CSIRTs differ. It is obvious that the assessment measures of the effectiveness of CSIRTs differ and are not precise. This can be accomplished through trying to conduct measurements for existing CSIRTs and translating the results of these measures into user-friendly business communications. As mentioned above, limited resources and the heavy workload of CSIRTs can affect their effectiveness. Ways so that a CSIRT can handle a growing work load and also stay effective need to be defined and discovered. CSIRTs need long term endurance as well as brilliance in order to prove their effectiveness. Their work can be seen as a marathon and usually it takes years before they are recognised. Consistent training of CSIRT stuff as well as the continuous building of a network of experts who can provide advice and help is needed. Also, sharing threat indicators with third-party organizations can foster collaboration. It is crucial for an organization to build an incident response team consisting of experienced, full-time members, to assess the readiness of incident response team members on an ongoing basis, to create clearly defined rules of engagement for the incident response team, to have meaningful operational metrics to gauge the overall effectiveness of incident response, be able to translate the results of these measures into user-friendly business communications, involve multi-disciplinary areas of the organization in the incident response process and invest in technologies that support the collection of information to identify potential threats.46

Critical Questions:

Who funds the CSIRT?

Is there an independent body that oversees the National CSIRT?

What roles and responsibilities have been identified for the national CSIRT operating partners?

Is there consistent training of the stuff?

Is there a network of experts assisting the CSIRT?

How does a CSIRT handle the workload?

Is there communication and cooperation with other CSIRT teams?

Which is the level of awareness of multiple stakeholders

46 Ponemon Institute LLC, January 2014. http://www.lancope.com/ponemon-incident-response/




6. Next steps The most crucial step to be taken, is to define precise metrics, which will be able to assess the effectiveness of the services of a CSIRT. More work is needed in this domain. This can be accomplished through conducting case studies and meta-analysis of existing ones from various organisations. This way, there can be a better understanding of what has worked so far. Another topic which has to be further explored, is CSIRT training. Specific definition of the organizations who offer training, the kind of training and the costs, can help more developing countries to plan creating a CSIRT or evolve the effectiveness of their CSIRTs. In order to explore the issue of the assessment of the effectiveness of a CSIRT further, currently the Global Cyber Security Capacity Centre is conducting a research project using two methods i) online survey (see Annex A) and ii) interviews of experts working for CSIRTs, both in national and private sectors. Methodology is being described below and details can be seen at Annex A and Annex B of this paper. Experts can share their knowledge and personal experience on metrics which currently are being used. Also they can discuss their effectiveness in capability enhancement. Moreover, a very interesting issue is the reason why various metrics might have been considered but have eventually been discounted. Some metrics might have never been considered and this can also provide us with further ideas to explore.

6.1. Methodology

6.1.1. Scope - Purpose The primary mission of a Computer Security Incident Response Team (CSIRT) is to help other organizations to handle incidents occurring in computer networks, as well as provide a wider set of services. Apart from their main mission, CSIRTs need to be able to adapt to a continuous changing environment and present the flexibility to deal any unexpected incident. Today’s challenges have an impact on the effectiveness of CSIRTs’. CSIRTs have less time to react to a new unexpected threat or cyber-attack. This is the reason that there is need for quick and accurate notification of an incident. CSIRTs need effective methods to collaborate and share information, efficient mechanisms to triage incoming information and requirements for policies and procedures that are established and understood. It is a fact, that many obstacles can affect their effectiveness. Well-defined metrics are essential to determine which security practices are worth the investment. The results of this assessment will lead also to the improvement of CSIRT processes. The goal of the project ‘’Assessment of the effectiveness of CSIRTs’’ is to contribute towards developing metrics which can be used to measure the effectiveness of CSIRTs. Specifically our aims are to identify: 1) the ways in which a CSIRT might be considered to be effective 2) the issues which may limit the performance of a CSIRT and 3) approaches towards developing CSIRT effectiveness metrics.

6.1.2. Research Method The research project will include two methods i) online survey (see Annex A) and ii) interviews (see Annex B). More specifically, the online survey includes 51 questions on various factors determining



the effectiveness of CSIRTs and participants can answer the questionnaire through a link. The goal of the interviews is to gain a deeper insight on the experience of experts working for CSIRTs. Questions asked during the interviews and online survey, ask for the personal knowledge and experience of participants, regarding CSIRTs. The online survey should take approximately 30-35 minutes to complete. The duration of interviews can vary. The findings of the survey will be written up and published as a paper of the Oxford Global Cyber Security Capacity Centre which will be distributed widely and openly. Participants can ask questions about the study before they decide whether to participate. Participants have a right to withdraw at any time without prejudice and without providing a reason. In the event of withdrawal, existing, already provided, data will not be included in the analysis or be used in many way. This project has been reviewed by, and received ethics clearance through, the University of Oxford Central University Research Ethics Committee ((Ref No: SSD/CUREC1A/14-127, Annex C).

6.1.3. Participants The participants taking part in this study, will be experts within the existing CSIRT community, currently working in a CSIRT environment, or who have done so in the past, or have been involved in the creation of a CSIRT. If a participant is currently not working for a CSIRT, then he/she should answer each question referring to the last CSIRT he/she has worked for. They will be recruited by personal contact, email or telephone. Prior to taking part in the study, participants will be required to read and sign a consent form that informs them of the project, the study, its goals and how their information and feedback will be treated and used.



7. Summary and conclusions

This paper, discusses the subject of the effectiveness and assessment of CSIRTs. CSIRTs have been established worldwide in order to address the various threats to information systems and their use. Little is currently being done in order to measure their effectiveness. It is vital to understand how to assess the effectiveness of CSIRTs before starting to discuss about improving their effectiveness. The goal of this paper is to take a first step towards developing metrics which can be used to measure the effectiveness of CSIRTs. In order this to be accomplished, CSIRTs need to focus on activities with the maximum impact on threat mitigation. More specifically, this paper aims to identify the ways in which a CSIRT might be deemed to be effective, and possible approaches towards developing CSIRT effectiveness metrics. It also identifies the issues that need to be addressed to realise the goal. First of all, this paper describes the role and purpose of CSIRTs, the services they provide, as well as information on the different sectors of CSIRT cooperation. Also, the paper discusses the sensitive topic of the assessment of the effectiveness of CSIRT’s. Issues such as cooperation, data sharing and trust are discussed as crucial components of an effective CSIRT. Existing measurement types of the performance of CSIRTs (NIST, Carnegie Mellon's Software Engineering Institute) are presented, followed by a set of suggested direct and indirect measures of the effectiveness of a CSIRT. Measurements such as, the number of reported incidents, the number of incidents handing, the percentage of security incidents that were managed, the percentage of vulnerability assessment findings that have been addressed, the response time or time-to-live of an incident, the time spent in handling of incidents, the mean times between incidents, the speed of identification of an incident, the ability to achieve work-through attack status in face of incidents, the volume of information output by the CSIRT, the number of accesses of the information provided by the CSIRT, the ranking of the CSIRT web pages on search engines and the enquiries to the CSIRT from its constituency, are some examples of possible metrics to be used. The assessment of the effectiveness of CSIRTs, can lead to the improvement of CSIRT processes.

Suggested steps in order to improve the effectiveness of CSIRTs include, improvement of awareness

of CSIRTs in target audience, improvement of the flow of vulnerability information to CSIRTs,

improving use of information provided by CSIRTs, improving trust in CSIRTs to ensure that as much

information is provided as possible, better enforcement of existing legislation and of course

existence of enough resources.

This paper identifies also a few critical questions, which need to be answered in order to help the

improvement of the performance of CSIRTs. Issues such as the funding source of a CSIRT, the

consistent training of the stuff, the communication between CSIRTs, the existence of a network of

cooperating experts are critical.

As a conclusion, it is obvious that further research work is needed in this field. Proving the

effectiveness of CSIRTs can be a long term procedure. Experts working in Computer Emergency

Response Teams, need to share their knowledge and experience with a wider network of experts in

order to accomplish a higher level of capability enhancement.



References

Brownlee, N., and Guttman, E. Expectations for Computer Security Incident Response. Best Current Practice,

ISF, Network Working Group RFC 2350, June 1998. Retrieved from: http://tools.ietf.org/html/draft-ietf-grip-framework-irt-04

Carnegie Mellon University, 2008, CERT’s Podcasts: Security for Business Leaders, Show Notes. Retrieved from http://resources.sei.cmu.edu/asset_files/Podcast/2008_016_102_67465.pdf

Chapin A. David, Akridge Steven, How can security be measured? Information Systems Control Journal, 2, 2005.

http://www.isaca.org/Journal/Past-Issues/2005/Volume-2/Documents/jpdf052-how-can-security.pdf CERT. Retrieved from http://www.cert.org/ Corporate Information Security Working Group, Report of the Best Practices and Metrics Teams,

Subcommittee on Technology, Information Policy, Intergovernmental Relations and the Census, Government Reform Committee, United States, House of Representatives, November 2004. Retrieved from http://net.educause.edu/ir/library/pdf/CSD3661.pdf

CSIRT Services list from CERT/CC. Retrieved from http://www.cert.org/csirts/services.html Cyber Security Incident Response – Are we as prepared as we think? Ponemon Institute LLC, January 2014.

Retrieved from http://www.lancope.com/ponemon-incident-response/ eCSIRT.net project. Retrieved from http://www.ecsirt.net/ ENISA, Good practice guide for CERTs in the area of Industrial Control Systems - Computer Emergency

Response Capabilities considerations for ICS, December 2013. Retrieved from http://www.enisa.europa.eu/activities/cert/support/baseline-capabilities/ics-cerc/good-practice-guide-for-certs-in-the-area-of-industrial-control-systems/at_download/fullReport

ENISA, Detect, SHARE, Protect - Solutions for Improving Threat Data Exchange among CERTs, 2013. Retrieved

from https://www.enisa.europa.eu/activities/cert/support/data-sharing ENISA. CERT community - Recognition mechanisms and schemes, 2013. Retrieved from CERT communities.PDF

https://www.enisa.europa.eu/activities/cert/support/baseline-capabilities/cert-community-recognition-mechanisms-and-schemes

ENISA, Baseline Capabilities of National/Governmental CERTs, 2012. Retrieved from https://www.enisa.europa.eu/activities/cert/support/files/updated-recommendations-2012

ENISA, National Cyber Security Strategies, Practical Guide on Development and Execution, December 2012.

Retrieved from http://www.enisa.europa.eu/activities/Resilience-and-CIIP/national-cyber-security-strategies-ncsss/national-cyber-security-strategies-an-implementation-guide/at_download/fullReport

ENISA, The Fight Against Cybercrime – Cooperation between CERTs and Law Enforcement Agencies to fight

against cybercrime – A first collection of practices, 2012. Retrieved from: http://www.enisa.europa.eu/activities/cert/support/fight-againstcybercrime/supporting-fight-against-cybercrime/cooperation-between-certs-and-law-enforcement-agencies-in-the-fight-against-cybercrime-a-first-collection-of-practices

ENISA, A flair for sharing – encouraging information exchange between CERTs, A study into the legal and

regulatory aspects of information sharing and cross-border collaboration of national/governmental CERTs in Europe, November 2011. Retrieved from: http://www.enisa.europa.eu/activities/cert/support/fight-against-cybercrime/legal-information-sharing

http://tools.ietf.org/html/draft-ietf-grip-framework-irt-04

http://tools.ietf.org/html/draft-ietf-grip-framework-irt-04

http://resources.sei.cmu.edu/asset_files/Podcast/2008_016_102_67465.pdf

http://www.isaca.org/Journal/Past-Issues/2005/Volume-2/Documents/jpdf052-how-can-security.pdf


http://net.educause.edu/ir/library/pdf/CSD3661.pdf

http://www.cert.org/csirts/services.html


http://www.ecsirt.net/

http://www.enisa.europa.eu/activities/cert/support/baseline-capabilities/ics-cerc/good-practice-guide-for-certs-in-the-area-of-industrial-control-systems/at_download/fullReport

http://www.enisa.europa.eu/activities/cert/support/baseline-capabilities/ics-cerc/good-practice-guide-for-certs-in-the-area-of-industrial-control-systems/at_download/fullReport

https://www.enisa.europa.eu/activities/cert/support/data-sharing

http://www.enisa.europa.eu/activities/cert/support/baseline-capabilities/cert-community-recognition-mechanisms-and-schemes/at_download/fullReport



https://www.enisa.europa.eu/activities/cert/support/files/updated-recommendations-2012



http://www.enisa.europa.eu/activities/cert/support/fight-againstcybercrime/supporting-fight-against-cybercrime/cooperation-between-certs-and-law-enforcement-agencies-in-the-fight-against-cybercrime-a-first-collection-of-practices







ENISA (2010), Incentives and Barriers to Information Sharing: Retrieved from:

http://www.enisa.europa.eu/activities/Resilience-and-CIIP/public-privatepartnership/information-sharingexchange/incentives-and-barriers-to-information-sharing

ENISA. CERT cooperation and its further facilitation by relevant stakeholders, 2006. Retrieved from

CERT_cooperation_ENISA.pdf European Commission (EC), Proposal for a Directive of the European Parliament and of the council concerning

measures to ensure a high common level of network and information security across the Union, Brussels, 7.2.2013, COM (2013) 48 final. Retrieved from http://eurlex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2013:0048:FIN:EN:PDF

FIRST, Creating and Managing Computer Security Incident Handling Teams (CSIRTs), CERT Training and

Education Networked Systems Survivability Software Engineering Institute Carnegie Mellon University, 2008. Retrieved from http://www.first.org/conference/2008/papers/killcrece-georgia-slides.pdf

IT Process Institute (ITPI). Retrieved from http://www.itpi.org/ ITU, Retrieved from http://www.itu.int/en/ITU-D/Cybersecurity/Pages/Organizational-Structures.aspx ITU, National Cybersecurity Strategy Guide. Frederick Wamala, September 2011. Retrieved from:

http://www.itu.int/ITU-D/cyb/cybersecurity/docs/ITUNationalCybersecurityStrategyGuide.pdf MACCSA, Multinational Alliance for Collaborative Cyber Situational Awareness. Information Sharing

Framework (ISF), 20 November 2013, version 2.4. Retrieved from: https://www.terena.org/mail-archives/refeds/pdfjJz1CRtYC4.pdf

Messenger, M. (2005), Why would I tell you? Perceived influences for disclosure decisions by senior professionals in inter organisation sharing forums, Unpublished Masters dissertation, University of London Birkbeck School of Management and Organisational Psychology.

NIST Special Publication 800-55 Revision 1, Performance Measurement Guide for Information Security,

Elizabeth Chew, Marianne Swanson, Kevin Stine, Nadya Bartol, Anthony Brown, and Will Robinson, July 2008. Retrieved from http://csrc.nist.gov/publications/nistpubs/800-55-Rev1/SP800-55-rev1.pdf

NIST Special Publication 800-61 Revision 2, Computer Security Incident Handling Guide, Recommendations of

the National Institute of Standards and Technology, Paul Cichonski, Tom Millar, Tim Grance, Karen Scarfone, August 2012. Retrieved from http://csrc.nist.gov/publications/nistpubs/800-61rev2/SP800-61rev2.pdf

OECD, Organisation for Economic Co-operation and Development, Directorate for Science, Technology and

Industry, Committee for Information, Computer and Communications Policy. Improving the International Comparability of Statistics Produced by Computer Security Incident Response Teams, 18 June 2014.

OECD, Organisation for Economic Co-operation and Development, Improving the Evidence Base for

Information Security and Privacy Policies: Understanding the Opportunities and Challenges related to Measuring Information Security, Privacy and the Protection of Children Online, OECD Digital Economy Papers, no. 214, OECD, 2012, Paris.

OECD, Recommendation of the Council on the Protection of Critical Information Infrastructures, OECD

Ministerial Meeting on the Future of the Internet Economy, Seoul Korea June 2008. Retrieved form: http://www.oecd.org/sti/40825404.pdf



http://www.enisa.europa.eu/activities/cert/background/coop/files/cert-cooperation-and-its-further-facilitation-by-relevant-stakeholders/at_download/fullReport

http://eurlex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2013:0048:FIN:EN:PDF


http://www.itpi.org/

http://www.itu.int/en/ITU-D/Cybersecurity/Pages/Organizational-Structures.aspx

http://www.itu.int/ITU-D/cyb/cybersecurity/docs/ITUNationalCybersecurityStrategyGuide.pdf









OECD, The Promotion of a Culture of Security for Information Systems and Networks in OECD Countries, Working Party on Information Security and Privacy, December 2005. Retrieved from: http://www.oecd.org/internet/ieconomy/35884541.pdf

OECD, Studies in Risk Management, Norway Information Security, 2006. Retrieved from:


Ponemon Institute LLC, Cyber Security Incident Response – Are we as prepared as we think?, January 2014.

Retrieved from: http://www.lancope.com/ponemon-incident-response/ Silicki, K., Maj, M. (2008), Barriers to CSIRTs cooperation. Challenge in practice – the CLOSER Project, 20th FIRST

Annual Conference, Vancouver, Canada. Software Engineering Institute (SEI), Incident Management Capability Metrics Version 0.1, Audrey Dorofee

Georgia Killcrece Robin Ruefle Mark Zajicek April 2007, TECHNICAL REPORT CMU/SEI-2007-TR-008 ESC-TR-2007-008 CERT Program. Retrieved from http://www.dtic.mil/dtic/tr/fulltext/u2/a468688.pdf

Sommer, P.M. (2009). Directors and Corporate Advisors, Guide to Digital Investigations and Evidence (2nd edn),

Swindon, UK: Information Assurance Advisory Council. Sritapan, Vincent, Stewart, Walter, Zhu, Jake, Rohm, C. E. Tapie, Developing a Metrics Framework for the

Federal Government in Computer Security Incident Response. Communications of the IIMA, 2011, 11 (3), p55-73. Retrieved from http://web.a.ebscohost.com/abstract?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=15435970&AN=82214678&h=iy9i2kF2J%2bamTVTRwH%2bIlaWSOV%2fzVtCuvbcifQhPd4SUP%2f1GDHbnx%2fg%2b2PWuhW0%2fQ4Br5JAbo0fVtpLnjIscYQ%3d%3d&crl=c

Wiik Johannes, Gonzalez J. Jose, Kossakowski Klaus-Peter, Limits to Effectiveness in Computer Security Incident

Response Teams, Software Engineering Institute, Twenty Third International Conference of the System Dynamics Society, 2005. Retrieved from http://resources.sei.cmu.edu/asset_files/WhitePaper/2005_019_001_53057.pdf

Additional Resources on CSIRTs

AP-CERT, Asia Pacific Computer emergency Response Team, Retrieved from http://www.apcert.org/ CERT EU. Retrieved from http://cert.europa.eu/cert/ CERT UK. Retrieved from https://www.cert.gov.uk/ GovCertUK Incident Response Guidelines. Retrieved from

http://www.cesg.gov.uk/publications/Documents/incident_response_guidelines.pdf GovCertUK Information packs. Retrieved from

http://www.cesg.gov.uk/awarenesstraining/PET/Pages/index.aspx Q-CERT, Qatar Computer Emergency Response Team. Retrieved from http://www.qcert.org/ Kenya Computer Incident Response Team Coordination Centre (KE-CIRT CC). Retrieved from

http://www.cck.go.ke/industry/information_security/ke-cirt-cc/ MyCERT Malaysia. Retrieved from http://www.mycert.org.my/en/

http://www.oecd.org/internet/ieconomy/35884541.pdf



http://www.dtic.mil/dtic/tr/fulltext/u2/a468688.pdf





http://www.apcert.org/

http://cert.europa.eu/cert/

https://www.cert.gov.uk/

http://www.cesg.gov.uk/publications/Documents/incident_response_guidelines.pdf

http://www.cesg.gov.uk/awarenesstraining/PET/Pages/index.aspx

http://www.qcert.org/

http://www.cck.go.ke/industry/information_security/ke-cirt-cc/

http://www.mycert.org.my/en/



OIC-CERT, Organization of the Islamic Conference-Computer-Emergency Response Team. Retrieved from http://www.oic-cert.net/v1/index.html

The European Governent CERTs Group. Retrieved from http://www.egc-group.org/ UK’s Cyber Security Strategy. Retrieved from

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/60961/uk-cyber-security-strategy-final.pdf

US-CERT. Retrieved from https://www.us-cert.gov/about-us Forming an Incident Response Team http://www.auscert.org.au/render.html?it=2252&cid=1920 Avoiding the Trial-by-Fire Approach to Security Incidents. http://www.sei.cmu.edu/news-at-sei/columns/security_matters/1999/mar/security_matters.htm Site Security Handbook. http://www.ietf.org/rfc/rfc2196.txt Expectations for Computer Security Incident Response. http://www.ietf.org/rfc/rfc2350.txt Internet Security Glossary. http://www.ietf.org/rfc/rfc2828.txt Terena TF-CSIRT Guide to Setting up a CSIRT. http://www.terena.org/activities/tf-csirt/archive/acert7.html ENISA Step-by-Step Guide to Setting Up a CSIRT. http://www.enisa.europe.eu/cert_guide/downloads/CSIRT_setting_up_guide_ENISA.pdf GOVCERT.NL CERT-IN-A-BOX. http://www.govcert.nl/render.html?it=69 Computer Security Incident Handling Guide, National Institute of Standards and Technology (NIST SP 800-61). http://www.csrc.nist.gov/publications/nistpubs/800-61/sp800-61.pdf APWG, The Anti-Phishing Working Group. Retrieved from http://www.antiphishing.org/ Arora, Ashish and Telang, Rahul and Xu, Hao, Optimal Policy for Software Vulnerability Disclosure, 2005.

Retrieved from http://dx.doi.org/10.2139/ssrn.669023 The Central and Eastern European Networking Association (CEENET). Retrieved from http://www.ceenet.org/ Communications-Electronics Security Group, Retrieved from https://www.cesg.gov.uk ENISA CERT Inventory – Inventory of CERT teams and activities in Europe, Version 2.12b, January 2014.

Retrieved from http://www.enisa.europa.eu/activities/cert/background/inv/files/inventory-of-cert-activities-in-europe

FIRST, Forum of Incident Response and Security Teams. Retrieved from http://www.first.org/ ITU, Developing national CSIRT capabilities – A case study of Tunisian CERT. Retrieved from

http://www.itu.int/ITU-D/cyb/events/2009/tunis/docs/elmir-ansi-csirt-june-09.pdf

Grobler Marthie and Bryk Harri, Common Challenges Faced During the Establishment of a CSIRT, IEEE, 2010. Retrieved from http://icsa.cs.up.ac.za/issa/2010/Proceedings/Full/17_Paper.pdf

Javaid, Muhammad Adeel, Benchmarks for Setting Up CERT (September 10, 2013). Available at SSRN:

http://dx.doi.org/10.2139/ssrn.2389061

http://www.oic-cert.net/v1/index.html

http://www.egc-group.org/



https://www.us-cert.gov/about-us

http://www.auscert.org.au/render.html?it=2252&cid=1920

http://www.sei.cmu.edu/news-at-sei/columns/security_matters/1999/mar/security_matters.htm

http://www.ietf.org/rfc/rfc2196.txt



http://www.terena.org/activities/tf-csirt/archive/acert7.html

http://www.enisa.europe.eu/cert_guide/downloads/CSIRT_setting_up_guide_ENISA.pdf

http://www.govcert.nl/render.html?it=69

http://www.csrc.nist.gov/publications/nistpubs/800-61/sp800-61.pdf

http://www.antiphishing.org/


http://www.ceenet.org/

https://www.cesg.gov.uk/

http://www.enisa.europa.eu/activities/cert/background/inv/files/inventory-of-cert-activities-in-europe

http://www.enisa.europa.eu/activities/cert/background/inv/files/inventory-of-cert-activities-in-europe

http://www.first.org/

http://www.itu.int/ITU-D/cyb/events/2009/tunis/docs/elmir-ansi-csirt-june-09.pdf

http://icsa.cs.up.ac.za/issa/2010/Proceedings/Full/17_Paper.pdf




The Honeynet Project. Retrieved from http://www.honeynet.org/about Kang, Jerry, Information Privacy in Cyberspace Transactions. Stanford Law Review, Vol. 50, p. 1193, 1998.

Available at SSRN: http://ssrn.com/abstract=631723 Mwende Njiraini, Establishing a National Computer Incident Response Team (CSIRT) in Africa: Kenyan case

study, 2011. Retrieved from http://api.ning.com/files/CiKtTdA9zz-bW-rycYFYBYPsPW3M6MW83isAbwDQEvM7UoZt7B9oQ9xLbNk*fZbBJxfUnVWV7k6nkQYcmpAtpSNljYOyGZT/EstablishingaNationalComputerSecurityIncidentResponseTeamCSIRTinAfricaAKenyanCaseStudy.pdf

NATO Computer Incident Response Capability - Technical Centre (NCIRC TC). Retrieved from

http://www.ncirc.nato.int/ Solove, Daniel J., A Taxonomy of Privacy. University of Pennsylvania Law Review, Vol. 154, No. 3, p. 477,

January 2006; GWU Law School Public Law Research Paper No. 129. Available at SSRN: http://ssrn.com/abstract=667622

Spiekermann Sarah, Cranor Faith Lorrie, Engineering Privacy, IEEE Transactions on Software Engineering, Vol.

35, Nr. 1, 2009. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1085333 Spiekermann, Sarah and Berendt, Bettina and Grossklags, Jens, E-Privacy in 2nd Generation E-Commerce:

Privacy Preferences versus Actual Behavior. Available at SSRN: http://ssrn.com/abstract=761107 Strandburg, Katherine J., Privacy, Rationality, and Temptation: A Theory of Willpower Norms. Rutgers Law

Review, Vol. 57, No. 4, Spring 2005. Available at SSRN: http://ssrn.com/abstract=755284 TERENA, Trans-European Research and Education Networking Association. Retrieved from

http://www.terena.org/ Trim Peter and Youl Youm Heung, Korea-UK Collaboration in Cyber Security: From Issues and Challenges to

Sustainable Partnership Report Submitted to the Korean Government and the UK Government, March, 2014, British Embassy Seoul: Republic of Korea. Retrieved from http://www.iaac.org.uk/ItemFiles/ReportTrimYoumCyberSecurityMarch14.pdf

Trusted Introducer,(TI). Retrieved from http://www.trusted-introducer.org/

http://www.honeynet.org/about

http://ssrn.com/abstract=631723

http://api.ning.com/files/CiKtTdA9zz-bW-rycYFYBYPsPW3M6MW83isAbwDQEvM7UoZt7B9oQ9xLbNk*fZbBJxfUnVWV7k6nkQYcmpAtpSNljYOyGZT/EstablishingaNationalComputerSecurityIncidentResponseTeamCSIRTinAfricaAKenyanCaseStudy.pdf



http://www.ncirc.nato.int/


http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1085333



http://www.terena.org/

http://www.iaac.org.uk/ItemFiles/ReportTrimYoumCyberSecurityMarch14.pdf



Annex A

Online Survey

Questionnaire for the assessment of the effectiveness of CSIRTs

Dear participant,

This questionnaire is part of a survey being conducted by the University of Oxford’s Global Cyber

Security Capacity Centre (GCSCC). The goal of this survey is to contribute towards developing metrics

which can be used to measure the effectiveness of CSIRTs. Specifically our aims are to identify: 1)

the ways in which a CSIRT might be considered to be effective 2) the issues which may limit the

performance of a CSIRT and 3) approaches towards developing CSIRT effectiveness metrics.

Please answer each question based on your personal knowledge and experience – as a professional

working in a CSIRT environment, or who has done so in the past, or as someone who has been

involved in the creation of a CSIRT. If you are currently not working for a CSIRT, then answer each

question referring to the last CSIRT you have worked for. The findings of the survey will be written

up and published as a paper of the Global Cyber Security Capacity Centre which we will be

distributed widely and openly. The questionnaire is anonymous and your answers will be kept

completely confidential. No personal information such as name or organization details, will be

presented or published.

This project has been reviewed by and received ethics clearance through, the University of Oxford

Central University Research Ethics Committee (Ref No: SSD/CUREC1A/14-127). To take part in this

research you must first give consent. Participation in this research is entirely voluntary and

participants should note that they are free to withdraw at any time without giving any reason and

without being penalised or disadvantaged in any way. Any information or personal details gathered

in the study will be strictly confidential. No personal information such as name or organization

details, will be presented or published. The data collected may be used in future published research.

Data collected may be processed manually and with the aid of computer software. All data will be

treated as personal under the 1998 Data Protection Act, and will be stored securely. Research data

and records will be retained for as long as they are of continuing value to the researcher and the

wider research community. Please, use Mozilla Firefox, Google Chrome, Apple Safari and Internet

Explorer, for better view of this questionnaire.

If you have a concern about any aspect of this project, please speak to the relevant researcher Dr.

Maria Bada, (+44 (0) 1865 287366, [email protected]) Professor Michael Goldsmith (+44 (0)

1865 610746, [email protected]) or Professor Chris Mitchell ([email protected])

who will do their best to answer your query. The researcher and supervisors should acknowledge

your concern within 10 working days and give you an indication of how she intends to deal with it. If

you remain unhappy or wish to make a formal complaint, please contact the chair of the Research

Ethics Committee at the University of Oxford (Chair, Social Sciences & Humanities Inter-Divisional

Research Ethics Committee; Email: [email protected]; Address: Research Services, University of

Oxford, Wellington Square, Oxford OX1 2JD). The chair will seek to resolve the matter in a

reasonably expeditious manner.

Thank you for participating in this survey!

Sincerely, Oxford Global Cyber Security Capacity Centre



I confirm that I have freely agreed to participate in the Assessment of the effectiveness of CSIRTs research project. I have been briefed on what this involves and I agree to the use of the findings as described above. I hereby assign the copyright in my contribution to the ‘’University of Oxford’’. Yes, I accept to participate

No, I do not wish to participate

Q2 In which way are you involved with a CSIRT? Currently Working for a CSIRT Worked for a CSIRT in the past Have been involved in the creation of a CSIRT Other ____________________ Q3 Please indicate the type of CSIRT/CSIRTs you have been involved with (Multiple Choice). Internal or Organizational CSIRTs National CSIRTs Incident Response Providers Analysis Centers Vendor Team Other ____________________ Q4 How many years of experience do you have in CSIRTs? Q5 Please indicate the country where the CSIRT/CSIRTs you currently work for operates. Q6 Approximately how many staff members are there, in the CSIRT you currently work for? Q7 Do employees in the CSIRT you work for, receive training in specialised aspects? Yes No Don't Know Q8 If yes, how many of employees received training in specialised technical aspects, during the last year? Q9 If no training is received, please state the reasons that apply: Insufficient funding Insufficient time Courses unavailable Lack of interest from staff members Other ____________________ Q10 Is there a hierarchy of authority in place, to the CSIRT you work for? Yes No Don't Know



Q11 Is the CSIRT you work for, characterised by a positive work environment, which includes trust and goal congruence? Yes No Don't Know Q12 Please indicate the nature of capabilities of the CSIRT/CSIRTs you currently work for. How well do you think that these capabilities are performed? (1= Low Levels to 5= High Levels) ______ Mandate and strategy ______ Operation ______ Services portfolio ______ Risk assessment ______ Communication ______ Cooperation Q13 According to your opinion, what level of expertise is desirable for each of these capabilities? (1= Low Levels to 5= High Levels) ______ Mandate and strategy ______ Operation ______ Services portfolio ______ Risk assessment ______ Communication ______ Cooperation Q14 Do you believe that there are missing technical capabilities? Please indicate. Q15 How is the impact of the CSIRT you work for, being measured? Please select all that apply. Number of website visits Number of links to CSIRT website from other pages Number of twitter followers Number of people or organisations on mailing lists Number of organisations requesting supporting from the CSIRT Number of organisations providing positive feedback on the utility of the support received from

the CSIRT Assessment via survey of stakeholders Assessment via interviews with stakeholders Other ____________________ Q16 Please provide an overall assessment of the impact of the CSIRT/CSIRTs that you currently work for, where 1= Low Impact to 5= High Impact. Q17 How big is the audience of the CSIRT? Please indicate (Multiple Answer) National Organisational International



Q18 Which Policies/Frameworks does the CSIRT you work for follow? FIRST (Global Forum for Incident Response and Security Teams) TI (Trusted Introducer) RFC 2350 (Request for Comments Memoranda) ENISA ISO 27035 Other ____________________ Q19 How many security incidents occur monthly, on average, within the domain covered by your CSIRT/CSIRTs you are associated with? 10s 100s 1000s 100,000s Millions Billions Q20 Please indicate the nature of the three (3) most frequent security incidents: Unauthorised Access Denial of Service Malicious Code Improper Usage Scans/Probes/Attempted Access Forensics Malware Financial Audit Other ____________________ Q21 What percentage of reported incidents are being handled, on average? Reported Incidents % Q22 What percentage of vulnerability assessment findings are being addressed? Vulnerability Assessment Findings % Q23 Is the CSIRT you work for, in charge of patching vulnerabilities? Yes No Don't Know Q24 If yes, what percentage of vulnerabilities are being patched? Vulnerabilities in % Q25 Does the CSIRT you work for, forward vulnerability information reports to your constituency? Yes No Don't Know Q26 How many reports, on average, are you forwarding?



Q27 Do you select vulnerabilities only for products being used by your constituency or do you forward all? Select only vulnerabilities being used by the constituency Forward all vulnerability reports Q28 Do you translate vulnerability reports? Yes No Don't Know Q29 Please indicate the speed of reaction to incidents of the CSIRT you work for.

< 24 Hours < 7 Days 2 Weeks 3 Weeks Longer

How long does it take for an event

to be recognised

as an incident?

How long does it take to determine the type of

the incident?

Q30 Do you think that this time frame is within your expectations or not? Please indicate 1= Slow to 5= Quick ______ Speed of event recognition ______ Determination of the type of the incident Q31 Does the CSIRT you work for have the ability to maintain the normal work flow while an incident is being handled? Yes No Don't Know Q32 Is there automated reaction/reporting to certain events? Yes No Don't Know Q33 What type of incident reporting is automated? Unauthorised Access Denial of Service Malicious Code Scans/Prob/ Attempted Access Investigation group



Q34 Is there a pre-defined public relations and analyst relations plan that you can put into action in the event of a material data loss that needs to be publicly disclosed? Yes No Don't Know Q35 Does the CSIRT/CSIRTs you are associated with, communicate with other CSIRTs? (Information sharing) Yes No Don't Know Q36 Does the CSIRT/CSIRTs you are associated with, cooperate with other CSIRTs? (e.g. Conducting joint projects). Yes No Don't Know Q37 If yes, please indicate, in which way the CSIRT cooperates with other CSIRTs. Q38 Are you satisfied with the current level of cooperation between CSIRTs? Yes No Don't Know Q39 Does the CSIRT you work for cooperate with other CSIRTs at National, European or International level? Yes No Don't know Q40 Please, indicate the degree of cooperation between CSIRTs at National Level, between EU Member States and at International Level where 1= Low Levels to 5= High Levels. ______ National Level ______ Between EU Member States ______ International Level Q41 Are you familiar with the relevant national and international legal frameworks? (privacy rules, data retention rules, national security obligations) Yes No Don't Know



Q42 According to your knowledge and personal experience, which metrics are currently in use to assess the effectiveness of the CSIRT? Impact measures Incident response quality Situational awareness capability General capability of CSIRT Outreach capability Other ____________________ Q43 In your opinion, how effective are these metrics in capability enhancement? ______ Effectiveness of Metrics Q44 In your knowledge and personal experience, which metrics are most useful for judging effectiveness? Q45 According to your knowledge and personal experience, are there metrics which have been considered but discounted? Please indicate. Impact measures Incident response quality Situational awareness capability General capability of CSIRT Outreach capability Other ____________________ Q46 Why do you think that they were discounted? Q47 Are you aware of metrics which have never been considered? Yes No Don't Know Q48 Please indicate these metrics. Q49 What do you think is essential for an effective CSIRT? Please provide factors which can enhance the effectiveness of a CSIRT. Q50 Please indicate the top three (3) CSIRTs according to your opinion. What are the characteristics or capabilities on which you have based your ranking? ______ CSIRT 1: ______ CSIRT 2: ______ CSIRT 3: Q51 Would you like to provide us with your email, in case further clarifications are needed, but also in order to receive first results from this survey? Yes ____________________ No



Annex B

Interview

Participant Consent for Research Project

‘’The assessment of the effectiveness of CSIRTs’’ If you consent to being interviewed and to any data gathered being processed as outlined below, please date the form, in the spaces provided, and return it to Dr. Maria Bada ([email protected]).

This project - ‘The assessment of the effectiveness of CSIRTs’ - is being conducted by the Oxford Global Cyber Security Capacity Centre at the UNIVERSITY of OXFORD.

All data will be treated as personal under the 1998 Data Protection Act, and will be stored securely.

Interviews will be recorded by the research team.

Copies of interview tapes and transcripts will be offered to the ESRC Data Archive.

A copy of your interview transcript will be provided, free of charge, on request.

Participants have a right to withdraw at any time without prejudice and without providing a reason. In the event of withdrawal, existing, already provided, data will not be included in the analysis or be used in any way.

Data collected may be processed manually and with the aid of computer software.

Please indicate, by ticking ONE of the boxes below, whether you are willing to be identified, and whether we may quote your words directly, in reports and publications arising from this research.

I or my organisation, may be identified in reports made available outside the research teams and the ESRC, and in publications.

Neither I, nor my organisation, may be identified in reports made available outside the research teams and the ESRC, nor in any publications. My words may be quoted provided that they are anonymised.

Neither I, nor my organisation, may be identified in reports made available outside the research teams and the ESRC, nor in any publications. My words may not be quoted.

Confirmation and consent I confirm that I have freely agreed to participate in the ‘’Assessment of the effectiveness of CSIRTs’’ research project. I have been briefed on what this involves and I agree to the use of the findings as described above. I hereby assign the copyright in my contribution to the University of Oxford. Participant Signature: _____________ Name:_________________ Date: _____ I confirm, for the project team, that we agree to keep the undertakings in this contract. Researcher Signature: _____________ Name:_________________ Date: ______




Q2 In which way are you involved with a CSIRT? (Are you currently working for a CSIRT, have you worked for a CSIRT in the past, or have you been involved in the creation of a CSIRT? Q3 In what type of CSIRT/CSIRTs have you been involved with? Internal or Organizational CSIRTs National CSIRTs Incident Response Providers Analysis Centers Vendor Team Other ____________________ Q4 How many years of experience do you have in CSIRTs? Q5 In which country does the CSIRT/CSIRTs you currently work for operates. Q6 Approximately how many staff members are there, in the CSIRT you currently work for? Q7 Do employees in the CSIRT you work for, receive training in specialised aspects? Q8 If yes, how many of employees received training in specialised technical aspects, during the last year? Q9 If no training is received, please state the reasons that apply: Insufficient funding Insufficient time Courses unavailable Lack of interest from staff members Other ____________________ Q10 Is there a hierarchy of authority in place, to the CSIRT you work for? Q11 Is the CSIRT you work for, characterised by a positive work environment, which includes trust and goal congruence? Q12 Please indicate the nature of capabilities of the CSIRT/CSIRTs you currently work for. How well do you think that these capabilities are performed? (1= Low Levels to 5= High Levels) ______ Mandate and strategy ______ Operation ______ Services portfolio ______ Risk assessment ______ Communication ______ Cooperation



Q13 According to your opinion, what level of expertise is desirable for each of these capabilities? (1= Low Levels to 5= High Levels) ______ Mandate and strategy ______ Operation ______ Services portfolio ______ Risk assessment ______ Communication ______ Cooperation Q14 Do you believe that there are missing technical capabilities? Q15 How is the impact of the CSIRT you work for, being measured? Number of website visits Number of links to CSIRT website from other pages Number of twitter followers Number of people or organisations on mailing lists Number of organisations requesting supporting from the CSIRT Number of organisations providing positive feedback on the utility of the support received from

the CSIRT Assessment via survey of stakeholders Assessment via interviews with stakeholders Other ____________________ Q16 Please provide an overall assessment of the impact of the CSIRT/CSIRTs that you currently work for, where 1= Low Impact to 5= High Impact. Q17 How big is the audience of the CSIRT? Please indicate (Multiple Answer) National Organisational International Q18 Which Policies/Frameworks does the CSIRT you work for follow? FIRST (Global Forum for Incident Response and Security Teams) TI (Trusted Introducer) RFC 2350 (Request for Comments Memoranda) ENISA ISO 27035 Other ____________________ Q19 How many security incidents occur monthly, on average, within the domain covered by your CSIRT/CSIRTs you are associated with? 10s 100s 1000s 100,000s Millions Billions



Q20 Please indicate the nature of the three (3) most frequent security incidents: Unauthorised Access Denial of Service Malicious Code Improper Usage Scans/Probes/Attempted Access Forensics Malware Financial Audit Other ____________________ Q21 What percentage of reported incidents are being handled, on average? Q22 What percentage of vulnerability assessment findings are being addressed? Q23 Is the CSIRT you work for, in charge of patching vulnerabilities? Q24 If yes, what percentage of vulnerabilities are being patched? Q25 Does the CSIRT you work for, forward vulnerability information reports to your constituency? Q26 How many reports, on average, are you forwarding? Q27 Do you select vulnerabilities only for products being used by your constituency or do you forward all? Select only vulnerabilities being used by the constituency Forward all vulnerability reports Q28 Do you translate vulnerability reports? Q29a. How long does it take for an event to be recognised as an incident? (24 hours, & Days, 2 weeks, 3 weeks, longer) Q29 b. How long does it take to determine the type of the incident? (24 hours, & Days, 2 weeks, 3 weeks, longer) Q30 Do you think that this time frame is within your expectations or not? Please indicate 1= Slow to 5= Quick ______ Speed of event recognition ______ Determination of the type of the incident Q31 Does the CSIRT you work for have the ability to maintain the normal work flow while an incident is being handled? Q32 Is there automated reaction/reporting to certain events?



Q33 What type of incident reporting is automated? Unauthorised Access Denial of Service Malicious Code Scans/Prob/ Attempted Access Investigation group Q34 Is there a pre-defined public relations and analyst relations plan that you can put into action in the event of a material data loss that needs to be publicly disclosed? Q35 Does the CSIRT/CSIRTs you are associated with, communicate with other CSIRTs? (Information sharing) Q36 Does the CSIRT/CSIRTs you are associated with, cooperate with other CSIRTs? (e.g. Conducting joint projects). Q37 If yes, please indicate, in which way the CSIRT cooperates with other CSIRTs. Q38 Are you satisfied with the current level of cooperation between CSIRTs? Q39 Does the CSIRT you work for cooperate with other CSIRTs at National, European or International level? Q40 Please, indicate the degree of cooperation between CSIRTs at National Level, between EU Member States and at International Level where 1= Low Levels to 5= High Levels. ______ National Level ______ Between EU Member States ______ International Level Q41 Are you familiar with the relevant national and international legal frameworks? (privacy rules, data retention rules, national security obligations) Q42 According to your knowledge and personal experience, which metrics are currently in use to assess the effectiveness of the CSIRT? Impact measures Incident response quality Situational awareness capability General capability of CSIRT Outreach capability Other ____________________ Q43 In your opinion, how effective are these metrics in capability enhancement? Q44 In your knowledge and personal experience, which metrics are most useful for judging effectiveness?



Q45 According to your knowledge and personal experience, are there metrics which have been considered but discounted? Please indicate. Impact measures Incident response quality Situational awareness capability General capability of CSIRT Outreach capability Other ____________________ Q46 Why do you think that they were discounted? Q47 Are you aware of metrics which have never been considered? Q48 Please indicate these metrics. Q49 What do you think is essential for an effective CSIRT? Please provide factors which can enhance the effectiveness of a CSIRT. Q50 Please indicate the top three (3) CSIRTs according to your opinion. What are the characteristics or capabilities on which you have based your ranking? ______ CSIRT 1: ______ CSIRT 2: ______ CSIRT 3: