cert exercises handbook

8/9/2019 Cert Exercises Handbook

1/88

CERT Exercises

Handbook

December 08


2/88

Legal notice

Notice must be taken that this publication represents the views and interpretations of theauthors and editors, unless it is stated otherwise. This publication should not be construedto be an action of ENISA or the ENISA bodies unless adopted pursuant to the ENISARegulation (EC) No 460/2004. This publication does not necessarily represent state-of the-art and it might be updated from time to time.

Third party sources are quoted as appropriate. ENISA is not responsible for the content ofthe external sources including external websites referenced in this publication.

This publication is intended for educational and information purposes only. Neither ENISAnor any person acting on its behalf is responsible for the use that might be made of theinformation contained in this publication.

All rights reserved. No part of this publication may be reproduced, stored in a retrievalsystem or transmitted in any form or by any means, electronic mechanical, photocopying,recording, or otherwise without the prior written permission of ENISA, or as expresslypermitted by Law or under terms agreed with the appropriate rights organisations. Sourcemust be acknowledged at all times. Enquiries for reproduction can be sent to the contactaddress quoted in this publication.

European Network and information Security Agency (ENISA), 2008

Acknowledgements

ENISA wants to thank all institutions and persons who contributed to this document. Aspecial Thank you goes to the following contributors:

Anna Felkner, Tomasz Grudziecki, Przemyslaw Jaroszewski, Piotr Kijewski, Miroslaw Maj,Marcin Mielniczek, Elzbieta Nowicka, Cezary Rzewuski, Krzysztof Silicki, Rafal Tarlowskifrom NASK/CERT Polska, who produced the first version of this document as consultants

The countless people who reviewed this document


3/88

Table of contents

Exercise 1: Triage and Basic Incident Handling 2

Exercise 2: Incident Handling Procedure Testing 8

Exercise 3: Recruitment of CERT Staff 12

Exercise 4: Developing CERT Infrastructure 18

Exercise 5: Vulnerability Handling 23

Exercise 6: Writing Security Advisories 29

Exercise 7: Network Forensics 37

Exercise 8: Establishing External Contacts 58

Exercise 9: Large Scale Incident Handling 61

Exercise 10: Automation in Incident Handling 73

Exercise 11: Incident Handling in Live Role Playing 77

Exercise 12: Cooperation with Law Enforcement Agencies 81

CERT Exercises Handbook 1


4/88

Exercise 1Triage and Basic Incident Handling

CERT Exercises Handbook

The exercise is aimed at incident handlers at any level of experience. Itrequires a good understanding of Internet topology and services.

2 hours, 25 minutes

Introduction to the exercise 10 min.

Task 1-9: Incident report analysis, classification 60 min.and prioritisation

Discussion 60 min.

Exercise summary and wrap-up 15 min.

Once a year for new team members or members reassigned to incident

response.

This exercise can be used with real reports as an intra-team exercise forall incident handlers in a CERT. In this case, the goal is to make sure thereis a consistency between the classification and prioritisation of reports bydifferent team members.

Targeted

Audience

Total Duration

Time Schedule

Frequency

GENERAL DESCRIPTIONThe exercise simulates the initial phases of incident handling with 10 real-life incident reports.These phases include: verification of the report (did the incident actually occur?); interpretation (what actually happened?); determination of the scope of incident (what are the actual and possible consequences for your

constituency and others?); classification; and prioritisation (based on the previous factors).

The students will try to complete these phases for each of the reports. Discrepancies between theirresults will then be discussed.

Before conducting the exercise, read through all the reports and key answers. If students comefrom an already established team or teams, ask them to provide the classification scheme they usein everyday work. You may decide to use those schemes rather than the ones suggested in theexercises, but it is important that all students use the same scheme as it provides common groundfor a discussion. You may also consider using real-life examples from your own experience insteadof some of the cases provided in the students book. The guidelines on anonymising data for thepurposes of this exercise are as follows:

10/8 are networks located in Utopia 10.187/16 are networks of Utopia NREN .ut is Utopias top-level domain

They were consequently used in the reports included on the LiveDVD.

Main Objective

This exercise provides students with experience of real-life incidentreports, their ambiguity and complexity. After finishing the exercise theyshould understand what to focus on during initial analysis, how differentfactors may affect priorities and how to communicate with reporters aswell as third parties. During the exercise, they will apply a givenclassification scheme to incidents the purpose of this part of the exerciseis to work on the consistent classification of disputable cases (eg, worm vscanning) across team members and possibly to suggest a clearer, more

unambiguous classification scheme for the team.


5/88

EXERCISE COURSEThe course of this exercise is as follows. All discussions should be moderated by the trainer.

Introduction to the exerciseDivide students in small groups (2-3 people). Ask them to open the IceDove mail client containedon the LiveDVD. There are nine incident reports in the Inbox. The toolset contains guidelines for thestudents as well as the proposed classification scheme1:


Description / Examples

Unsolicited bulk e-mail, which means that therecipient has not granted verifiable permission forthe message to be sent and that the message issent as part of a larger collection of messages, allhaving an identical content.

Discrediting, or discrimination against, somebody(ie, cyberstalking)

Child pornography, glorification of violence, ...

Software that is intentionally included or insertedin a system for a harmful purpose. A userinteraction is normally necessary to activate thecode.

Attacks that send requests to a system to discoverweak points. This includes also some kinds oftesting processes to gather information abouthosts, services and accounts. Examples: fingerd,DNS querying, ICMP, SMTP (EXPN, RCPT, ).

Observing and recording network traffic(wiretapping).

Gathering information from a human being in anon-technical way (eg, lies, tricks, bribes, orthreats).

An attempt to compromise a system or to disruptany service by exploiting vulnerabilities with astandardised identifier such as a CVE name (eg,buffer overflow, backdoors, cross side scripting,etc).

Multiple login attempts (Guessing or crackingpasswords, brute force).

An attempt using an unknown exploit.

Incident Class(mandatoryinput field)

AbusiveContent

Malicious Code

InformationGathering

IntrusionAttempts

Incident Type(optional but desiredinput field)

Spam

Harassment

Child/Sexual/Violence/...

Virus

Worm

Trojan

Spyware

Dialler

Scanning

Sniffing

Social Engineering

Exploiting knownVulnerabilities

Login Attempts

New Attack Signature

1 This classification was developed during the eCSIRT.net project on CERT cooperation and common statistics.More information can be found athttp://www.ecsirt.net/cec/service/documents/wp4-clearinghouse-policy-v12.html#HEAD6


6/88

Ask the students to analyse the reports, to describe the situation and the possible ways in which itmay be mitigated, and to apply the classification scheme and prioritise incidents, giving thempriority ranks of 1, 2 or 3, with 1 as the top priority.

Allow 60-75 minutes for resolution. During that time, make sure you are available to answer anyquestions which may arise. Do not give hints and clues yourself answer fully and correctly onlywhen asked.

Keys to the exercise

Task 1 UKSUtopia InspectionsThis may seem like a regular spam report. On closer analysis it turns out that apparently somebodyat [email protected] a message to a mailing list informing co-workers about somescheduled maintenance. One of the addresses bounced and the bounce message was reported asspam. Clearly, this is a misunderstanding and the report is void.


A successful compromise of a system orapplication (service). This could have been causedremotely by a known or a new vulnerability, butalso by an unauthorised local access.

In this kind of an attack, a system is bombardedwith so many packets that the operations aredelayed or the system crashes. Examples of aremote DoS are SYS-a, PING-flooding or E-mailbombing (DDoS: TFN, Trinity, etc). However,

availability can also be affected by local actions(eg, destruction, disruption of power supply, etc).

Besides the local abuse of data and systems,information security can be endangered by asuccessful account or application compromise.Furthermore, attacks that intercept and accessinformation during transmission (wiretapping,spoofing or hijacking) are possible.

Using resources for unauthorised purposes,including profit-making ventures (eg, the use ofe-mail to participate in illegal chain letters forprofit or pyramid schemes).

Selling or installing copies of unlicensedcommercial software or other copyright protectedmaterials (Warez).

Type of attacks in which one entity illegitimatelyassumes the identity of another in order to benefitfrom it.

If the number of incidents in this categoryincreases, it is an indication that the classificationscheme needs to be revised.

Intrusions

Availability

InformationSecurity

Fraud

Other

Privileged AccountCompromise

Unprivileged AccountCompromise

Application Compromise

DoS

DDoS

Sabotage

Unauthorised access toinformation

Unauthorisedmodification ofinformation

Unauthorised use ofresources

Copyright

Masquerade

All incidents which don'tfit in one of the givencategories should be putinto this class.


7/88

Task 2 Abuse: 10.187.137.4The report speaks about a DDoS attack in which a host from the constituency of Utopia CERT takespart. The first thing to do should be to determine whether the address was spoofed or if we aredealing with a real problem in our network. Since the logs come from a web server and show fullHTTP requests, TCP connection must have been established and communication was bi-directional. Insuch a case, IP spoofing would require the hackers to hijack BGP prefixes of the network which isprobably too much effort when botnets are readily available. In any case the suggested follow-up isto check flows and the state of the machine in question.

Task 3 [SpamCop (http://www.company.ut/) id:3091085703]3-4 June-Workshops forManagersThis is a regular spam complaint forwarded via the SpamCop service. The complaint reaches UtopiaCERT because the website advertised in the e-mail is within your constituency. Possible follow-up

depends on legal situation of spam in a given country. In some cases, even when the sending of bulkcommercial e-mails is prohibited by law, each message must be individually reported by its recipientto appropriate authorities which effectively makes the law unenforceable. In such cases the role ofthe CSIRT is minimal and is limited to advising users and possibly registering the report for statisticalpurposes.

Task 4 [CERTPT #56817] Unauthorised access attempt registeredThis is a report from another CERT, containing logs of unauthorised login attempts. Within theproposed classification scheme it may be suggested that these kinds of brute-force attempts, whichfit logically as login attempts, may be signs of worm activity. This is okay if you are confident thatthis is typical worm behaviour (eg, known wide-spread infections with similar patterns havingoccurred recently) and the same classification is used consistently within the team.

Note that the logs do not concern Utopia CERT directly. Instead, the hosts listed are from a differentprovider in Utopia, so the Utopia CERT will play the role as coordinator. Moreover, *.internetdsl.* inhostnames suggests dynamic addressing so it would be vital to provide the ISP with full logs alongwith timestamps. Lack of the address of the attacked host could be a problem if the timestamps arenot synchronised and also in the case of NAT. Note that all timestamps are in GMT, so time-zoneoffset must be taken into account.

Task 5 Incident 10.187.21.203This is a report from an automated monitoring and reporting system which notifies you aboutscanning activity from one of the hosts in your constituency. Notice that the scans are concentratedaround well known ports used by worms (TCP 135, 137, 139 and 445). This may not necessarilyindicate worm activity (possibly multiple infections at the same time), so again arguments can beraised for both the scanning and worm classification of the activity.

Task 6 [SpamCop (http://www.bigoil.ut/cgibin/internet.exe/portal/ep/home.do?tabId=0)id:3120641650]----BIGOIL CO. Search (Immediate Part-Time JOB for At first sight this looks just like another spam report related to a spamvertised (advertised by Spammessages) website of a company located in Utopia. In reality this is a financial scam similar toNigerian scams, where the name, brand and a website of an existing and reputable company areabused in a fictional story of some shady business. Suggested classification is fraud, because socialengineering relates more to reconnaissance and gathering information useful for further attack.

Task 7 Incident 10.187.108.39Another report from an automated system. This time, along with scanning patterns, somedescriptions of IDS signatures are provided. The same kind of attack across multiple hosts in asubnet makes it likely to be related to the activity of a worm such as MSBlaster of lovSan (theseworms were targeting port 135 tcp).



8/88

Task 8 Bank Phish Site [211889] - Please Reply ((NOTE - THIS SITE(s) HAS BEEN UPSINCE 3/07. WE HAVE SENT 4 NOTICES TO SHUT IT DOWN - PLEASE DO SO))A phishing case where the site is apparently using fast flux technology to make it harder to shut itdown. Several copies are reported to exist in Utopia and the Utopia CERT is asked for assistance intaking them down. If possible, the appropriate ISPs should be asked to retain any evidence ofmalicious activities such as connection logs from the machines. However, this can be problematicwhere home-user machines are parts of a botnet. Additional actions might include re-examining thedomain from time to time as new IPs may pop up on the list of zombies hosting the website inquestion.

Task 9 [MBL# 89603] Malware Block List AlertA malicious file is hosted somewhere under .ut domain. The report does not indicate if the host itselfis also located in Utopia, so the first step would be to resolve the domain name. There are a few

scenarios to try with such incidents. If the website where the malware was injected (3q.ut in thiscase) seems legitimate itself, you should try contacting the company which owns it and inform themof the problems. Many companies will do enough to fix the problem just for the sake of saving theirreputation. Another path to try would be the hosting company, as in many cases the websitesowners outsource website administration and will need to contact the administrators anyway. If thefeeling is that the malware is hosted intentionally (or at least knowingly), the best thing to do is tocontact the ISP straightaway, possibly bringing the police into the loop.

DiscussionWhen the time is due, ask one person from each team to state clearly: their view of the situation; how would they proceed, whom would they contact; what type of incident they are dealing with (using the proposed classification scheme); and

what priority would they assign to the incident and why.

At this time, do not comment on the results. Write them all down on a whiteboard for everyone tosee.

When you have collected all the answers, discuss each case, focusing on those which receivedvarious grades of priority or different classification from different groups. Sometimes the very samereport is ranked as very important by one group and given a very low priority by others. This isokay as long as the groups can provide justifications for their rankings. Be open to arguments anddescribe cases from your own experience where applicable.

Summary of the exerciseSome points to use for wrap-up and conclusions in the summary:

Most classification schemes are not perfect; probably none are. Creating a classification schemespecifically for a given team can make the choices more obvious initially, but it will have to beupdated from time to time. On the other hand, using one classification scheme over a longerperiod of time and sharing it with other teams would allow for the comparison of statistics.

When an incident type is ambiguous, it is not the name of the class that matters. More importantis how you describe this class in your statistics. And the most important thing is consistency, somake sure that all incident handlers classify similar incidents in the same way. Regular meetingsand ad hocdiscussions should help resolve any discrepancies.

Priority is not a function of just one variable the incident type. Some groups might haveclassified a report in the same way, but give them different priorities based on additionalknowledge or assumptions such as it is a widespread worm. In real life, it is vital to know these

factors and collect any necessary information to avoid confusion.



9/88

EVALUATION METRICSAs stated above, there are no single correct answers in this exercise. Some cases can be moredisputable than others. Following the key provided above and the suggested answers below, makesure that the students have not missed some important spots that may not be obvious in the firstplace and have correctly identified the nature of the problem. It is also vital that, when justifyingthe priorities applied to the reports, students take into account not just the type of incident but alsoits scope and relevance to the constituency.

The table below contains suggested classification and prioritisation for the exercise:


Comments

This is not an incident

If the attack is not ongoing, the priority may be lowered.

1.1.1

1.1.2

Worm, if worm activity is high or other evidence is available.

1.1.3

1.1.4

Active phishing and malware distribution sites should betreated with higher than usual priority.

See above. It may be suggested that the classificationscheme should be expanded to include drive-by-downloadinfections and other malware distribution mechanisms.

Task

1

2

3

4

5

6

7

8

9

Classification

None

DDoS

Spam

Login attempts

Scanning

Fraud

Worm

Masquerade

Malicious Code

Priority

N/A

1

3

2

2

3

2

1

1


10/88

Exercise 2Incident Handling Procedure Testing


In this exercise participants will have the opportunity to learn the mostimportant information about incident handling. It will give them an idea onhow to organise this process in their teams in the most efficient way.

This exercise is especially aimed at novice CERT members. It can also bedelivered to more experienced members to provide them with anopportunity to review their existing procedures and learn new methods ofincident handling which will enable them to organise their work in a more

efficient way.

3 hours, 10 minutes


Task 1: Developing incident handling procedures 60 min.

Task 2: Resolving critical problems in incident handling 70 min.

Summary of the exercise and evaluation 30 min.

It is most important that this exercise be conducted with new CSIRTmembers or even with candidates. It could also be conducted periodically,

to give more experienced team members the opportunity to evaluate andimprove their existing procedures.

Main Objective

TargetedAudience

Total Duration

Time Schedule

Frequency

GENERAL DESCRIPTIONThe purpose of this exercise is: To familiarise participants with the basic set of activities relating to incident handling (IH)

processes; To teach a correct sequence of activities during the IH process; To point out and provide knowledge about the most important parts of the IH procedure which

critically influence the success of the process; To familiarise participants with all possible players in the IH process; and To provide participants with basic knowledge about the most effective methods of cooperation

between CSIRT and key incident handling players.


Introduction to the exerciseAt the beginning of the exercise you present the students with some general information about theprocedure for handling incidents. Define the most important part of the general procedure andexplain the general sequences of a proper procedure. Also, at this stage, identify the mostimportant players in the procedure.

To present a general concept of the incident handling procedure workflow you should describe themost important parts of it: reception of notification, verification of notification, analysis, processingof notification, and solution.


11/88

You can use the following schema:

Give students just a general overview of these phases. Do not explain in detail what kinds ofactivities are included in particular phases of the incident handling procedure, because this task willbe part of the students work. You should also mention incident handling procedure players, such as: Your CERT, A reporter (an individual / an organisation), A victim (an individual / an organisation), An attacker (an individual / an organisation), LEA (Law Enforcement Agencies), ISP (Internet Service Providers), and Other CERTs

When introducing these players, do not appoint a particular role for each of them as doing so will bea part of the students work.

Task 1 Developing an incident handling procedureAfter the initial, general presentation of the topic, continue with tasks for the students.

Divide the students into groups of 3-4 people. There should be at least two groups and preferablyno more then four. Each group undertakes the following task:

Provide the students with the content ofTask 1.

Using the objects of an incident handling procedure, form a complete incident handlingprocedure. Create the proper sequence of activities, build relations between them, andindicate the directions of the work flows. Additionally, extend the procedure with yourproposals for activities using the blank objects.

After forming a procedure, identify activities which require communication with externalparties. For each of them point out the recommended means of communication (eg, anormal e-mail, a phone call, an encrypted e-mail, etc).

Analyse your procedure. Specify the critical elements and identify the potential problemswhich could appear during execution of the procedures.

Use Appendix 1 for this task.


RECEPTION OF NOTIFICATION

VERIFICATION OF NOTIFICATION

ANALYSIS

PROCESSING OF NOTIFICATION

SOLUTION


12/88

Give the students 30-45 minutes to complete this task. During that time, make sure you areavailable to answer any questions that may arise. Do not give hints and clues yourself answerfully and correctly only when asked. After the time is up, give each group 5-10 minutes to presenttheir procedure proposal. List all critical elements presented by each group on the whiteboard.During the presentation all students can ask questions. After the presentation they can askquestions or make comments, but they should avoid making a final evaluation of the procedures.

Task 2 Resolving critical problems in incident handlingAfter all procedures have been presented by the students, ask the whole group the followingquestion:

Which procedure did you like the most and which one would you rather improve?

They must make a choice and explain their decision.

After they have had a discussion relating to their choices (about 30 minutes), ask the students topresent their ideas on how to deal with the most critical parts of a procedure according to the list ofproblems identified by the groups during Task 1. Together with the students, create a list of the fivemost significant problems. This becomes Task 2 for the groups. You can form new groups for thistask or you can keep the existing ones. Your decision can be influenced by the activity of the groupsso far.

Provide the students with the content ofTask 2:

Please write down the most critical parts of the procedure identified by the groups andthe trainer. Provide your ideas on how to deal with them in order to mitigate relatedrisks and propose proactive activities for avoiding such problems.

They have 20-30 minutes to discuss the problems within their groups and to present their solutionsfor mitigating these problems and the proactive actions needed to avoid them. After each groupspresentation there is a short (5-10 minutes) discussion.

Summary of the exercise and evaluationThe whole exercise finishes with the summary made by you. Use the following schema tosummarise the exercise: repetition of the main objectives of the exercise; description of the students tasks and a short evaluation of their execution (see Evaluation Metrics

for this exercise); description of the main parts of the incident handling procedure and the key players; enumeration of the means of communication in an incident handling procedure and a general

description of the pros and cons of these means in terms of the efficiency and safety of theincident handling procedure; and

summarisation of the problems identified by the students and the methods they would use tomitigate them.

The trainer should advise those students who already have their own incident handling procedure intheir teams to self-evaluate the procedures.

EVALUATION METRICSAn intermediate method of evaluation could be a cross-evaluation by the teams of participants.During such an evaluation they would analyse the proposal for a procedure prepared by a differentteam and try to compare it to their own. They would try to point out the faults and good points ofthe proposal. In the end, the opinions are discussed and an arbitrary evaluation by the trainer is

presented.

As more measurable factors in the evaluation, the following aspects could be checked:

Have the students pointed out all key players in the incident handling process?



13/88

[Answer]This is a relatively easy task as you mentioned these players in the introduction to the exercise. Youshould explain that most of the incident handling traffic is exchanged between CERT teams. At thispoint you should also explain that the type and importance of a player is an important factor inincident handling prioritisation. A special player is a LEA which is usually a police department. It isimportant that your procedure corresponds to the law describing how the LEA must proceed.

Your CERT A reporter (an individual / an organisation) A victim (an individual / an organisation) An attacker (an individual / an organisation) LEA (Law Enforcement Agencies) ISP (Internet Service Providers) Other CERTs

Have the students pointed all the key activities of the incident handling process?

[Answer]The most important ones are:

proper determination as to whether a report constitutes an incident or not; identification of the real victim and the attacker in an incident, bearing in mind that the

attacker identified in your report might not be the real one; activities related to alerting all interested parties about a threat related to your incident; and cooperation with the ISPs of the victim and the attacker for collecting and saving evidence of

the incident.

Have the students enumerated correctly the most important means of communication andmatched them to the relevant parts of the incident handling procedure?

[Answer]The most important relationships and means of communication are: CERT incident reporter

E-mail (do everything you can to ensure encrypted contact) Telephone (ask a reporter to confirm a report by sending an e-mail) Web form (ensure encryption for this means (SSL))

CERT LEA Telephone (this means is mainly used by LEAs for obtaining initial information and

consultation; use it widely as it is a very good educational system) Official letter (this is an official document, so draft and deliver it according to the law in yourcountry)

CERT ISP

E-mail (the most common partners and means in the incident handling process, use therelationship actively, try to develop a trusted schema for getting a prompt answer, and useencryption for all confidential information)

REFERENCESTo summarise the exercise, the trainer uses the references for graphical and descriptive informationabout incident handling procedure. He should also propose the following references: ENISA,A step-by-step approach on how to setup a CSIRT Doing Incident Handling,

http://www.enisa.europa.eu/cert_guide/pages/08_03.htm

CERT Coordination Center Incident Reporting Guidelines,http://www.cert.org/tech_tips/incident_reporting.html

Christopher Alberts (Carnegie Mellon University), Georgia Killcrece (Carnegie Mellon University),Robin Ruefle (Carnegie Mellon University), and Mark Zajicek (Carnegie Mellon University) Defining Incident Management Processes for CSIRTs: A Work in Progresshttp://www.sei.cmu.edu/pub/documents/04.reports/pdf/04tr015.pdf



14/88


Exercise 3Recruitment of CERT Staff

To raise the ability of CERT managers to optimally recruit staff for theirCERT teams

CERT managers who are responsible for recruiting staff

6 hours, 20 minutes


Task 1: Writing job advertisements for recruiting CERT staff 90 min.

Task 2: Analysing and choosing candidates to be interviewed 90 min.

Task 3: Interviewing chosen candidates 120 min.

Task 4: Final selection of the best candidates 30 min.

Summary of the exercise 30 min.

It is recommended that the exercise be performed once by CERTmanagers whose tasks cover the recruitment of the staff and thereafterevery three years.

Main Objective

TargetedAudience

Total Duration

Time Schedule

Frequency

GENERAL DESCRIPTIONThe purpose of the exercise is to improve the ability of CERT managers to optimally recruit staff fortheir CERT teams. Students will learn: What staff is essential for a CERT team; What kinds of professional experience and/or qualifications, as well as personal abilities, are

essential to fulfil the main roles and responsibilities of a CERT; What kinds of questions should be asked during a job interview; and How to choose the most suitable candidates for a CERT team.

In particular, the exercise is intended to deliver a collection of tips on how to recognise andunderstand a candidates attitude towards many aspects (technical, ethical and organisational) ofnetwork security.

The trainer should be an experienced CERT manager who has conducted many interviews withcandidates and managers, or who has managed an incident response team in the past.


Introduction to the exerciseTo begin with, ask students what kind of staff they have in their CERT teams and what differentroles need to be fulfilled within their teams. Next, describe the typical organisational structures of aCERT team (independent business model, organisation embedded model, campus model, etc) [1]and the typical services that a CERT provides (incident handling, alerts and warnings, vulnerabilityhandling) [2]. Also explain that, despite the differences between several CERT models, the teams

staff should include the following members: General manager, who manages a CERT team, Technical staff, ie, staff to operate CERT services, and Researchers, ie, staff to undertake research.

Additionally, some consultants can assist the general manager in his work; these would include alegal specialistwho deals with legal issues and preserves evidence in the event of a lawsuit.


15/88

The number of people to be hired depends on the scale of the CERT services which will be providedand its financial resources but, roughly speaking, the operational technical team should consist ofone technical manager plus two technicians and the research team of one research manager plustwo researchers.

The CERT team leader should have a background in security and experience in the work involved inresilience crisis management in the field. Staff of an operational technical team should be securityspecialists who can deliver the specialised CERT services for handling and responding to IT incidents.Researchers should have a good background in network security, experience in security projects andpublications in the field of network security.

Task 1 Writing job advertisements for recruiting CERT staffAt the beginning, tell students that determining the key competencies required of the future staff for

their team will have a significant influence on the effectiveness of each service provided by theteam; it will also motivate the work-place in a way that enables everybody to exchange ideas, worktogether and improve their skills. All this will affect the teams success in the future. Recruiting theright staff requires careful identification of features that are important for a team as a whole, but italso requires taking note of the individual skills of candidates.

Step 1: Ask students to prepare job advertisements (blank templates are included in the studentsexercise book). This step should take at most 45 minutes.

Task of groups 1-2 (technicians) is to write a job offer for a technical position. The main tasks ofan employee holding this position will include: Handling and responding to network security incidents Operating the CERT early warning and alerting system for a CERT constituency Writing security advisories Writing news about security threats Preparing CERT reports Carrying out security audits

Task of groups 3-4 (researchers) is to write a job offer for a research position. The main tasks ofan employee holding this position will include: Participation in projects related to the security network Carrying out research on new methods for the detection and analysis of malicious software Development of the concept of IT projects to pursue new solutions Cooperation with software engineers in the implementation of proposed solutions Testing developed applications Writing technical documentation Development of IT security polices

Step 2: Each group presents its job advertisement proposal to the others. (It may be displayed soeverybody can see it.) This step should take at most 30 minutes.

Job advertisements for the technician position may include the following main requirements: Good knowledge of issues related to security on the Internet Good knowledge of mechanisms of TCP / IP and most common network services Good knowledge of Windows operating systems Very good knowledge of Linux (administration will be an advantage) Knowledge of programming languages: Perl, PHP Good knowledge of at least one foreign language Responsibility The ability to work in a team The ability to transfer knowledge

High personal culture (diplomacy) CommunicativeAdditional advantages could be: Two years administration experience Membership of IT security organisations



16/88

Job advertisements for the researcherposition may include the following main requirements: Higher education or related level of education (MSc) Very good knowledge of network security issues, in particular the risks involved in network

monitoring and the analysis of malicious software Experience with new technologies (more than one) such as honeypots, client honeypots, systems

IDS / IPS / WAF, the sandbox, darknets, and early warning systems Very good knowledge of TCP / IP Good knowledge of Linux Practical knowledge of C / C or Java, and scripting languages Practical knowledge of relational databases Ability to think analytically Ability to work both in a group and on his or her own Knowledge of at least one foreign language

Good writing skillsAdditional advantages will be: Experience in research projects in the field of IT security Experience in project management

The CERT team can offer: Participation in innovative international projects in cooperation with world-renowned IT companies

and institutions Ability to pursue their own research interests Access to information about the latest events relating to the propagation of threats in networks Participation in international working groups and conferences IT security training

In general, according to ENISA [4], the technical qualifications of CERT technical staff shouldinclude: Broad knowledge of Internet technology and protocols Knowledge of Linux and Unix systems (depending on the equipment of the constituency) Knowledge of Windows systems (depending on the equipment of the constituency) Knowledge of network infrastructure equipment (router, switches, DNS, proxy, mail, etc) Knowledge of Internet applications (SMTP, HTTP(s), FTP, telnet, SSH, etc) Knowledge of security threats (DDoS, phishing, defacing, sniffing, etc) Knowledge of risk assessment and the practical implementation of security measures

The personal abilities of CERT technical staff should include: Flexibility, creativity and a good team spirit Strong analytical skills Ability to explain difficult technical matters in simple words

A good feeling for confidentiality and working in a procedural manner Good organisational skills Ability to handle stress well Strong communication and writing skills Open mindedness and a willingness to learn

There are also additional competencies that could be considered: Willing to work 24x7 or on call duty (depending on the service model) Maximum travelling distance (in case of emergency, availability in the office; maximum travelling

time) Level of education Experience in working in the field of IT security

Required CERT staff qualities are also described in [3] and [5].

Step 3: This phase could be summed up as a discussion of the skills that have the highest priorityfor each position. Ask students to make their own lists of the highest priorities for the competenciesof an ideal candidate for a particular position. Write all the ideas on the whiteboard. Afterwards, ifany items considered to be important were missed by students, add them to the list.

CERT Exercises Handbook4


17/88

It should be stressed here that both technical knowledge and the skills connected to the personalityof a candidate are very important. Skills such as communication abilities, language fluency, personalhabits, friendliness, and optimism are essential for contacts and teamwork. Also, motivation, theability to work hard under pressure, resistance to stress, as well as attitude to ethical issues, havehigh priority in this kind of work.

Task 2 Analysing and choosing candidates to be interviewedStudents from groups of the same profile form one group, so from now on there is one techniciansand one researchers group.

Step 1: Distribute a collection of 6 CVs (having selected them earlier from a 12 CV collectionincluded in the LiveDVD in/usr/share/exercises/03_RCS/adds/ directory) to each group. It isassumed that all candidates have passed computer literacy tests at the level required to be a

member of a CERT team. Students from each group analyse all the CVs and try to match them withthe prepared job offers. In parallel, students write short opinions about all the candidates (strongand weak points, pros and cons in several aspects). At the end of this step, each group decideswhich two candidates should be interviewed. This step should take about 45 min.

Step 2: Each group presents its opinions about the candidates and justifies their choice (for eachCV). Ask questions, comment on the students opinions and try to show aspects potentially missedby the students.

Task 3 Interviewing chosen candidatesThis phase is devoted to interviews. Each interview should not exceed 15 minutes.

Step 1: First, let the students become familiar with the code of conduct (CoC) in the TF CERT ([7],

included in the exercise package). Afterwards, based on the CoC as well as on the prepared jobadvertisement and the CVs of the chosen candidates, the groups propose up to 20 interviewquestions (5 general, 5 technical, 10 others) that they would like to ask particular candidates oftheir choice.

Step 2: Each group presents their interview questions to the others and explains which of them aremost important. Propose a few questions (including some in respect of CoC if missed by students)and let the students decide which of them they consider important. At this stage, do not commenton their choice.

Step 3: Each group decides on a set of about 10 questions to be put to a chosen candidate. Duringthe presentation, ask each group to assess the validity of some the questions.

Step 4: Ask for volunteers from each group to play the roles of the chosen candidates. (Notice that

the number of candidates chosen to be interviewed may vary between two and four.) If there are novolunteers, you need to choose them. Students from technicians will play the role of candidates forthe research position and, analogically, students from researchers will play the role of candidates forthe technical position. Volunteers receive copies of the CVs and have 15 minutes for preparation. Atthe same time, the rest of a group has a break. For volunteers information only: advise them togive answers which cannot be unambiguously interpreted easily. Suggest to them that they pretendto have different personal abilities than they actually have.

Step 5: After a break, the students start interviewing the selected candidates. Every group joins allthe interview sessions. If it happens that both groups have chosen the same candidate (ie, sameCV), this candidate is interviewed by both groups in one interview, responding to the questions ofthe technicians and the researchers. After each interview the group discusses the candidatesanswers and shares its opinions. Summarise them and encourage students to ask additionalquestions if needed.

Interview questions I. Large collections of general job interview questions are available at [6].Examples of some general questions regarding work history, experience, expectations from the newjob and company, interests, the future, etc, that should be asked of candidates may include:



18/88


1. Please introduce yourself.

2. What were your expectations for the job and to what extent were they met?

3. What were your responsibilities?

4. What major challenges and problems did you face? How did you handle them? Which was the

most or least rewarding?

5. What was your biggest accomplishment or failure in this position?

6. Questions about his or her supervisors and co-workers. Who was your best boss and who was

the worst?

7. Why are you leaving your job?

8. How do you handle stress and pressure?

9. What motivates you?

10. Do you prefer to work independently or in a team? Give some examples of teamwork.

11. If you know your boss is 100% wrong about something, how would you handle it?

12. What interests you about this job?

13. What do you know about this company?

14. Why do you want to work here?

15. Is there anything I haven't told you about the job or company that you would like to know?

16. What are your goals for the next five years or ten years?

17. Tell us about your hobby. (Speak about it in a foreign language.)

Facultative Questions:

1. What are your salary expectations?

2. What have you been doing since your last job?3. Why were you fired? (if applicable)

4. Do you take work home with you?

5. Are you willing to travel?

Interview questions II. More specific questions regarding technical qualifications and personalabilities may include:

Technical issues

1. How does Snort work? What is the working principle of network intrusion detection systems?

2. What is the difference between low- and high-interaction honeypots? What honeypots do you

know?

3. What is the difference between TCP and UDP protocols? Name a few services that use TCP

and UDP.

4. What examples of network worms do you know? What are the methods for their propagation?

5. How should information about new vulnerabilities or warnings of new threats be published?

6. What are the most common motivations behind black-hat hacking?

7. Why would anyone want to infect a home-user computer?

8. What is phishing? What techniques can be used to phish?

9. What is a botnet? How can you take it down?

10. What are countermeasures against DDoS attacks?

Ethical/general issues

1. What would you do if you discovered a publicly-unknown software vulnerability?

2. What do you think about ethical hacking? Have you ever done it?

3. What do you understand by the concept of ethics in the security industry?

4. What national or international security organisations do you know?

5. What is the biggest threat and/or the most popular type of incident on the network handled

by CERTs nowadays (according to the statistics from the annual CERT report)?


19/88


Moreover, a collection of interview questions should include a few questions referring to thecandidates CVs.

Task 4 Final selection of the best candidatesAfter all the interviews, ask the students to prepare their own opinions about all the candidates andto make their selections (with justifications). Then, ask them to vote for the candidates.

After all the presentations, begin a discussion with the following questions: Which candidates answers convinced them to choose that candidate (if any was selected)? Do

the other students have similar feelings about this? Which candidates answers convinced them to reject that candidate (if any was rejected)? Do the

others have similar feelings about this?

Summary of the exercise

As a summary of this exercise, you can ask students the following: What do they think are the most useful abilities for becoming part of a CERT team? How do they imagine an ideal candidate (technical qualifications, personal abilities and other

competencies) for different roles within a CERT team? On the other hand, what do they consider problematic about some recruited staff in their daily

work?

Also, you can ask students where would be the best place to publish their job offers. Moreover,where and how they would seek candidates? You can also ask for other possibilities for recruiting.

Encourage students to exchange their opinions, to ask questions, and to give their feedback aboutthe exercise.

Moreover, you can mention also that a candidate who has just graduated from university can beconsidered for a position as a junior IT specialist researcher. This candidate should have, however,

some past experience in Internet security activities such as script kiddies, research groups andwriting security news, etc.

EVALUATION METRICSEvaluate the offers and the prepared interview questions, as well as the reasons for choosing orrejecting the candidates. Did the students consider the appropriate skills for each position in their job offers (technical,

personal, ethical)? Did the students propose adequate questions for conducting the interviews?

Were the students opinions about candidates and selections adequately and sufficiently justified?

REFERENCES[1] CERT organisational structure, http://www.enisa.europa.eu/cert_goodPractices/pages/04_02.htm

[2] CERT services. http://www.cert.org/csirts/services.html, (2008)

[3] CERT/CC. Staffing Your Computer Security Incident Response Team What Basic Skills AreNeeded? http://www.cert.org/csirts/csirt-staffing.html

[4] ENISA. CERT team roles and staffing.http://www.enisa.europa.eu/cert_goodPractices/pages/04_03.htm

[5] Handbook for Computer Security Incident Response Teams (CERTs), CERT/CC document.http://www.cert.org/archive/pdf/csirt-handbook.pdf [Staff issues, p.166-171]

[6] Large collections of various interview questions.http://jobsearch.about.com/od/interviewquestionsanswers/a/interviewquest.htm

http://www.jobinterviewquestions.org/

[7] The European CERT Network. Code of Conduct.http://www.ecsirt.net/service/eCERT-WP2-CoC-20021209.pdf


20/88


Exercise 4Developing CERT Infrastructure

To learn what kind of software and hardware solutions could be used toprovide a particular CERT service for a constituency.

Technical and management CERT staff.

Roughly 3 hours


Task 1: Incident handling incident analysis 45 min.

Task 2: Further 3-5 services 90 min.


The exercise should be carried out when a new team is being establishedor plans to expand its services.

Main Objective

TargetedAudience

Total Duration

Time Schedule

Frequency

GENERAL DESCRIPTIONThe purpose of this exercise is to learn what kind of software and hardware solutions could be usedto provide a particular CERT service for a constituency. By doing this exercise, students will learnabout the connection between a set of services defined for their team and available IT solutions.

This will help them to provide their services more easily and more effectively.

As a trainer, you should become familiar with the CSIRT services base, listed by the CERT/CC CSIRTat http://www.cert.org/csirts/services.html. This will be the basis of the discussion. It isrecommended that, for every service, the trainer should compose a list of freely available(as well as commercial, if needed) software solutions needed to provide the service.

All discussions should be moderated by the trainer.

EXERCISE COURSEThe course of this exercise is as follows.

Introduction to the exerciseAt the beginning, introduce students to the exercise, outlining what its main tasks are and how theexercise will be carried out. This exercise consists of two main tasks:

TASK 1: Step by step example: Incident handling incident analysis; and

TASK 2: A further 3-5 scenarios.

At the beginning the students should receive a short introduction to the CSIRT services base, listedby the CERT/CC CSIRT on the website: http://www.cert.org/csirts/services.html. The next taskwould be to challenge the students to create a concept for providing these services using aproposed hardware and software infrastructure. You should give an example of a step-by-stepexercise to get the students to understand how to proceed. In this exercise, the incident handling incident analysis service is chosen. Further scenarios will depend on what you and the studentsagree upon.


21/88

Task 1 Discuss the proposed infrastructure for the incident handling incident analysisserviceHand out the two diagrams shown below to the students. Your goal is to discuss them with thestudents, asking the students to point out the strengths and weaknesses of the proposed solutions.You should lead the students by asking them questions, and step by step bring them closer topossible answers. Note, that the answers do not have to be the same as in this example, but shouldcover a similar set of aspects. The questions are presented below.

Listed below are possible questions that could be asked regarding the incident handling service.Note that these are just suggestions and not an attempt at enumerating every possible issue. Theanswers are just examples as well and may not cover every issue. You should carefully thinkthrough the issues below and come up with additional answers or answers of your own, so that youwill be able to moderate the discussion accordingly.

Incidents could be reported via several ways or channels. Which of them should be maintained byCERT teams as a minimum? The most basic channel is via the Internet. Usually CERT teams use e-mail or/and web-page

forms. Also telephone and fax should be available as a minimum. Every team should have apublicly available PGP key.

What tools can be used to better organise teamwork and information flow especially forincidents reported via the Internet? A possible open source incident handling system that could be used is Request Tracker for

Incident Response (RTIR: http://bestpractical.com/rtir/). If students do not know about RTIR,you could give a short overview of this tool. Look at the RTIR requirements.

A mail server is needed. If you use Linux, free ones include Postfix or Sendmail. All mails targeted at the incident response centre should be passed through no anti-spam oranti-virus rules should block traffic, or if they do, they should do it in a manner that enablesthe analysis of such traffic (look also at the question: how to secure CERT infrastructure?).

A web server will be useful: Apache is a possible choice. A large information display in the incident response centre, which everyone can see, is a good

idea: it could be a projector which projects information onto a wall or screen or LCD/plasmadisplays. Information about current threats could be displayed here. What are the possiblesources of such information?

How to better organise teamwork in respect of telephone and fax? There should be an established position of duty officer of the day. Every team member should

hold this position interchangeably. The duty officer is responsible for, amongst other things,answering calls and faxes.

How phone calls are to be handled outside working hours should also be addressed. Some new fax-machines can turn faxes into documents and send them via e-mail.



22/88

Where to store incident reports and why is this so important? Every result of incident handling could be potential evidence. Every incident (report, analysis

and the effect of the investigation) and information gathered should be documented and safelystored. All e-mails or other electronic data must be stored in a safe way on server(s) (withbackup and HA cluster). All faxes must be stored in a safe place (for example in a safe-box). Ifyou have the means, you should record your calls. This gathering of information and evidencemust be done in a way that documents a provable chain of custody that is admissible in acourt of law under the rules of evidence[1].

How to prevent a failure or outage of Internet or telephone connections and servers (hardware)? There should be a backup Internet connection (via another autonomous ISP). A backup telephone line (for example via GSM operator) is also a good idea. To eliminate single points of failure, failover clusters should be deployed (critical services such

as incident handling servers should consist of redundant nodes). To minimise downtime and maximise availability, servers should be equipped with hot-swap

RAID arrays and be connected to a UPS system. Making regular backups is extremely important. Automatic backup system/scripts can be used.

Created copies should be periodically verified to see whether they are usable.

How to monitor your network for the failure or outage of servers, Internet connections, etc? A network monitoring system should be deployed to warn about failures or service status

changes (open source solutions such as Nagios, Argus, Munin, and OpenNMS can be used).This information should be displayed on an information displayer (projector or LCD/plasmadisplays).

How to respond to network failures? Emergency procedures should be developed in case of a network failure.

How to secure all CERT infrastructures? Firewall(s) how many, IDS, IPS? An antivirus filter should be integrated with the mail server; AV protection with the latest virus

definitions is highly recommended for workstations. (Please note that AV protection should notblock incident reports because they may contain malware samples sent intentionally.)

The physical security of critical network elements should be assured. Physical security should also cover confidential papers, faxes, documents, etc. Use a safe-box. Server hardening delivers another layer of protection one can use kernel patches (ie, PaX,

Exec Shield, SE Linux, LIDS, grsecurity), hardening scripts (Bastille Linux), kernel-level packetfiltering (netfilter), and host-based intrusion detection systems (OSSEC, tripwire).

Sometimes incident analysis requires going outside the network centre or lab. What tools are

helpful in working remotely? Laptop Mobile phone Portable HDD or flash drive with large storage space PDA with Internet connection and e-mail client, web browser, etc, connected via VPN?

What basic software should you have for incident handling in the context of the first questions? For handling an incident via e-mail you should have an e-mail client installed. (A possible free

one is Mozilla Thunderbird.) For handling an incident via RTIR you should have Internet browsers installed. (Possible free

ones are Mozilla Firefox and Opera.)



23/88

What basic software do you need to perform incident analysis in the context of? network forensics:

Tools for obtaining information about addresses, domain names, etc (CLI: whois, dig, host;there are also web-based online versions of these tools.)

Tools for analysing pcap files (CLI: tcpdump, GUI: Wireshark) Tools for analysing netflow data (CLI: nfdump, GUI: nfsen) Lab isolated with firewall: subnet and hosts

computer forensics: Tools for data preservation (hardware: DriveBlocker, etc, ???) Tools for data analysis (EnCase, etc, ???) Isolated lab: hosts and subnet

malware/binary analysis Isolated and monitored lab: host or subnet (with different types of operating systems; an

IDS/IPS will be useful to identify malware: Snorts) Virtual environment (software: VirtualBox, Vmware) Reverse engineering tools

The checklists below could help you judge how well the students ideas and solutions comply withthe main assumptions.


Assumptions

Backup Internet connection from other ISP

Firewall(s) (how many), IDS, IPS, etc.

Web server (HA cluster)

Mail server

Incident handling server (HA cluster) for example for RTIR

Central database (HA cluster)

Backup server

Services available from the Internet are separated from internal network by situating

them in demilitarised zone (DMZ)

Internal services such as backup, database and incident handling servers, as well as

team workstations, are located behind firewall

Lab subnet isolated with firewall

Servers should be equipped with hot-swap RAID arrays and connected to UPS system

Fax machine

Telephone

Shredder

Printer

Established position of duty officer

Filing cabinets

Safe-box

Info displayer projector or LCD/plasma displays

Extra tools

Discussion/consultation table

Screen/board

Tools for outside work:

Mobile phone

PDA

Laptop Portable HDD

yes/no

1.1.5

1.1.6

1.1.7

1.1.8

1.1.91.1.10

1.1.11

1.1.12

1.1.13

1.1.14

1.1.15

1.1.16

1.1.17

1.1.18

1.1.19

1.1.20

1.1.21

1.1.22

1.1.23

1.1.24

1.1.25

1.1.26

1.1.27

1.1.281.1.29


24/88

Task 2 Discuss the proposed infrastructure for a further 3-5 servicesOnce the first task has been completed, a set of services should be chosen, partly by the trainer,and partly by the students. The set chosen should include services from all main categories such asreactive services, proactive services and security quality management services. About 3-5 servicesshould be chosen.

In a manner similar to the previous exercise, the students should create a concept of providingthose particular services using a hardware and software infrastructure. They should design anetwork environment, including computers, network devices and connections between them. It isimportant that the students face the task of the separation of the services in relation to theircriticality. It is advisable that the trainer prepares, for each service, a basic set of solutions (as inthe example exercise) in order to facilitate discussion. A checklist would be useful to evaluateproposals. How could the topology presented in the first task be extended to accommodate the new

services?

Summary of the exerciseSummarise the exercise. By going through so many services, you have established with yourstudents quite large infrastructures. Compare these infrastructures with the one you initially thoughtof. Did the discussion contribute anything? If you have carried out this exercise before, how was theoutcome different this time?

Encourage students to exchange their opinions, ask questions, and give their feedback about theexercise.

EVALUATION METRICS

Evaluating the results of this exercise. The main criteria should be how active the students wereduring the discussions. Did they introduce new ideas? Use the checklists you prepared beforehandto track what students missed.

REFERENCES[1] CSIRT services, http://www.cert.org/csirts/services.html.



25/88


Exercise 5Vulnerability Handling

To provide a practical overview of the vulnerability handling process andhow vulnerabilities reported to a CERT team should be handled. Also, toprovide some hands-on experience with difficult situations that may arisethrough the role of coordinator.

Managers and incident handlers

3 hours, 10 minutes [optionally 4 hours, 10 minutes]


Task 1: Responsibilities of a CERT team in a vulnerability case 30 min.

Task 2: Vulnerability disclosure advantages and disadvantages 30 min.

Task 3: Designing a vulnerability disclosure policy 45 min.

Task 4: Introducing CERT coordination in a vulnerability case 45 min.

Task 5: Identification of vulnerability handling phases [optional] [30 min.]

Task 6: Coordination of single and multiple vendor cases [30 min.][optional]


It is recommended that this exercise be performed when a CERT team isbeing set up and when there is a significant personnel change within aCERT team. As not many CERTs have a full vulnerability handling service,it should be performed each time a team decides to introduce this serviceor recognises that it is treated by its constituency as a provider of thisservice.

Main Objective

TargetedAudience

Total Duration

Time Schedule

Frequency

GENERAL DESCRIPTIONThe objective of the exercise is to provide a practical overview of the vulnerability handling process

and how vulnerabilities reported to a CERT team should be handled. Students will learn: Who the key players are, and the main phases of the vulnerability handling process; The main responsibilities of a CERT team involved in a vulnerability case; How to design a vulnerability disclosure policy suitable for their CERT; and How to deal with difficult situations that may arise through their role as a coordinator.

This exercise, in particular, will focus on giving some starting points (also for reading anddiscussion) to be prepared for handling unexpected and challenging problems that may arise whena vulnerability is reported to a CERT team. It is also intended to highlight the issues to beconsidered by a CERT in communicating and in resolving vulnerability cases.

In practice, vulnerability handling requires technical knowledge of vulnerabilities and some incidenthandling experience, as well as familiarity with social engineering techniques, high-levelcommunication practices and risk management skills.



26/88

Introduction to the exerciseAt the beginning, introduce the students to the exercise, providing them with information on howlong the exercise will take and what its main parts are. During this part, provide the students withsome general information about the vulnerability handling process as described below.

Generally, the vulnerability handling process includes:(1) analysis of a reported vulnerability (ie, technical verification of a suspected vulnerability andidentification of the means for exploiting it);(2) vulnerability repairing (ie, installing patches to limit or prevent the exploitation); and(3) response coordination (ie, developing a vulnerability disclosure strategy) [1].

A vulnerabilitycase that involves CERT in the

coordinating role, assumes two parties, such as anexternal non-affiliated evaluator who discovers anovel software failure and a software vendorresponsible for the product concerned [3].

For these three main actors (see Fig 1), thevulnerability process assumes the following roles: Evaluator (reporting role), who reports the

vulnerability discovered to a vendor or a CERTteam;

Vendor (repairing role), who is responsible forfixing the vulnerability (eg, by a patch); and

CERT team (coordinating role), whichestablishes and maintains the communication

link between the reporter and the repairer. TheCERT teams role is to advise on how to resolvea vulnerability case.

Task 1 Discussion: Responsibilities of a CERT team in a vulnerability caseAt the beginning, sketch a typical vulnerability case as follows:

After a vulnerability is reported to a CERT team, the evaluator is asked to provide details aboutthe identified vulnerability. Once this has been received, the CERT team asks the vendor toprovide information about how their products are affected by the vulnerability reported. Thevendor is then responsible for assessing the impact and severity of the vulnerability (ie, whocould be affected and how) and for preparing a patch. He can also quantify the costs and benefitsof vulnerability disclosure. After the patch is ready, both an evaluator and a CERT team canevaluate the final fix.

Next, ask students to identify a CERTs main responsibilities and activities in a vulnerability case,keeping in mind that a CERT team acts as an independent coordination centre. In particular askthem: (a) What responsibilities do they have as coordinator towards the vendor? (b) What responsibilities do they have towards the vulnerability reporter?

Moderate the discussion, record the students ideas on the whiteboard, and (if needed) complete thelist provided by the students with the following information regarding the CERTs activities andresponsibilities: Providing efficient communication between all involved parties (also using the CERTs existing

security contacts); Providing vulnerability verification; Evaluating the vulnerability assessment impact provided by the vendor;

Independent identification of the scope of a vulnerability; Analysing the interests of all parties involved; Considering the advantages and disadvantages of disclosure; Determining when to disclose the vulnerability; Evaluating the final vulnerability fix; and Developing an appropriate strategy for disclosure.

CERT Exercises Handbook4

Fig 1: Three main actors and their roles inthe vulnerability process


27/88

Task 2 Discussion: Vulnerability disclosure advantages and disadvantagesVulnerability disclosure is perhaps the most controversial aspect of the vulnerability handlingprocess. You should mention that various discussions are underway, but so far there is noagreement upon standards or processes in this area [8, page 133]. Also, there is no standard policyon how to deal with vulnerability once it has been found, eg, should it be kept a secret or bepublicly disclosed? Therefore, before making any decisions, it is necessary to consider differentaspects of disclosing information about a vulnerability, such as: Why, who, or what information should be disclosed? When or where should the vulnerability be disclosed? What factors influence the timing of disclosure?

You should stress here that there is a real dispute regarding whetherand whya vulnerability shouldbe disclosed [7] and ask students to think why this is so. Ask students to think about any pros and

cons of full disclosure of a vulnerability and to write down their ideas in their work book.

Advantages: Some advocate that disclosure stimulates vendors to fix vulnerabilities. Some alsobelieve that the release of the details of a vulnerability motivates other vendors to make moretests of their software and make it more secure. Furthermore, some claim that there is only asmall chance (about 8%) that the same vulnerability will be identified independently by malicioushackers and white hat hackers.

Disadvantages: Others think that disclosure significantly increases the risk of exploitation, withall the consequences that could involve, eg, the loss of millions of visitors (eg, the DoS attack onYahoos site in 2000). The problem also concerns the quality of the patch particularly ifdeveloped under severe time pressure which can be insufficient to prevent the exploitation. Buteven the best patch protects only customers who keep their software up to date and lesssecurity-conscious users will still be at risk of being attacked by malicious hackers.

Task 3 Designing a vulnerability disclosure policyBegin with a general discussion about vulnerability disclosure policies. Ask the students What does responsible vulnerability disclosure mean to them?

Next, split the students into a few groups and ask them to develop a general vulnerability disclosurepolicy they believe proper for their CERT. When the groups are ready, everybody should discusswhat should be the main parts of a vulnerability handling policy. Issues addressed in their policiesshould include (a) and (b).

Give an example of so-called grace periods (ie, the amount of time given to the affected vendor todevelop a security update before the details are published) which are different for different CERTs.When CERT/CC, for example, is notified about a potential vulnerability, it contacts the softwarevendor and gives it a 45-day period to develop a patch [6]. After that time, that CERT makes theinformation public. However, the goal of CERT/CC policy is to balance the need of the public to be

informed of security vulnerabilities with the vendors need for time to respond effectively. The finaldetermination of a publication schedule is always based on the best interests of the communityoverall [6].

Stress that, as each vulnerability case is unique, it may require a quite different management policy.Also, since there are various actors and interests in the vulnerability process, there are thereforealso different viewpoints regarding the disclosure of the vulnerability.

Next, present some real examples of vulnerability handling and disclosure policies [4, 5, and 6].Details of the viewpoints regarding each of the roles in the vulnerability process can be found in [4](RFPolicy reporters perspective), [5] (CISCO policy vendors perspective) and [6] (CERT/CCpolicy coordinator perspective), which are also discussed in references [2] and [7]. Discuss withthe students the aspects of these policies they find acceptable or unacceptable for theirconstituency. It is important to present the vulnerability handling process from different points of

view. This will give the students information about the complexity of the process as well as allowthem to understand controversial issues relating to this process.

Emphasise that developing a vulnerability disclosure policy, and handling and managementstrategies are tricky tasks that require careful analysis based on real-case scenarios, best practicepolicies, privacy laws, and vendors policies.



28/88

Task 4 Role-playing game: Introducing CERT coordination in a vulnerability caseDuring the role-playing game students firstly receive a case-study, a story about vulnerabilityhandling related by you or described in a brief by you and read by students. Then the initialscenario of a game is presented.

You should pay attention to the following rules during the role-playing game. (You should also getthe students familiar with them.) A game leader is the trainer. A game leader has an absolute power to shape, modify and adjust a game scenario, eg:

can stop an action and introduce new factors and new conditions; can rewind an action to change factors or conditions or actions already performed; and can accelerate an action to avoid valueless activities.

All students must fit their actions to what the trainer has decided.

Students can communicate during a role-playing game only as players, not as students. (Forexample, they are not allowed to comment on an action unless the trainer changes it.)

A main purpose for the trainer is to achieve the goals of the exercises.

Now you tell the story about how vulnerability should not be handled. (You can name it One Day atBlack Hat.) You can also give the students the story in a written form.

Lynn was initially represented at the conference by noted cyber law attorney Jennifer Granick. Thelawsuit filed by Cisco and ISS was settled with a permanent injunction against both Lynn and BlackHat preventing further disclosure of information on the exploit. [9]

Now you start a game. It has obligatory and optional players:

Obligatory players are: the hacker, the large ISP, and CERT (two possibilities: CERT inside ISP, CERT outside ISP).

Optional players are (when there are too many students for one obligatory group, but too few fortwo): vendor of vulnerable hardware, law enforcement, and vulnerability auction company like WabiSabiLabi [10].

Try to fit the roles to the students. You should consider getting acquainted with the roles beforehandand assigning them to people according to their personalities and future work as closely as possible.Consequently, if the exercise is used as a part of a longer, multi-day training it should be scheduled

towards the end of the course. This way the students will be able to become more familiar with eachother and with the trainer.

Game scenario:The hacker reports a very serious remote administration vulnerability in a hardware device of alarge ISP to a CERT and wants money and credit on the ISPs webpage for providing the details. Thevulnerability is easily exploitable and renders hardware useless without hard/hand reboot. Initialdirect contact between the hacker and the ISP has failed the ISP feels threatened and is willing toprosecute the hacker.

You explain the task:The ISPs main goals are to prevent any disclosure and to get the details of the vulnerability. Ifthe CERT is located inside the ISP, care should be taken to show how difficult internal companyrelations can be CERT v PR, network engineers v management, and so on. Students should be

split into small groups. It is recommended that each player is represented by one student. Thegoal is to resolve the incident in a manner satisfactory to all. The trainer is responsible formoderating the discussion.



29/88

Initiate the game. Involve all students in it as soon as possible; then let them improvise. Theyshould contact each other the best option in a role-playing game is to stage phone calls anddiscuss a topic and move the case towards a solution.

It is important that they use only the information dedicated for each role. Phone tapping isforbidden.

Task 5 Identification of vulnerability handling phases [optional, if needed or there is aspecial interest from the students]During this post role-playing activity, students are given the task of identifying as many activitiesand processes as possible. This is achieved through a brain-storming session with the trainer as agroup leader. Afterwards, the whole group is divided into three parts and each of them must assesshow important the particular processes are for the group. A factor which makes groups different is

that they represent, during the assessment, the different vulnerability handling players: vendors,vulnerability researcher (evaluator) and a CERT team. This should produce different results andmake students aware how differently the vulnerability handling process looks from differentperspectives and interests.

Task 6 Coordination of a single and multiple vendor case [optional, if needed or there is aspecial interest from the students]Ask the students to think about aspects that differ in various real cases of vulnerability.

The possible variants of vulnerability cases can differ both in terms of the number of actors and theroles of each actor, as well as the different kinds of sources of a vulnerability report; it may be awhite-hat hacker, a malicious hacker, a security professional, or an internal group in a company.There can also be multiple vendors or subcontractors involved in a single case. If more than onevendor is affected, who releases an advisory? Should an advisory be internal or public? Each activitytaken should be accompanied by a careful risk management plan and should be documented at eachstage of the vulnerability process.

Now, focus on one aspect, ie, a vulnerability case that involves multiple vendors. Ask the studentsto think about possible complications, both general and those from the point of view of a CERT teamacting as a coordinator.

When students are ready with their ideas, mention that over 60% of software vulnerabilities affectcustomers of multiple vendors. Multiple vendors add complexity to the original model concerning amonopolist vendor in two ways: (1) first, competition among vendors may lead to a competitiveeffort in shortening the patching time. This may or may not be a good thing. (2) Second, theearliest disclosure may be set by either CERT or one of the vendors. This may mean that thedisclosure policy of the CERT might become somewhat irrelevant.

Summary of the exerciseNow, it is time to summarise the exercise. Encourage students to exchange their opinions, askquestions, and give their feedback about the exercise.

It should be taken into account that communication in a vulnerability case may concern differentactors with potentially conflicting roles. For example, the goal of an evaluatorcould be getting somebenefits or credits from a vendor in return for providing details of a failure discovered. The goal of avendorwill be to minimise the cost of disclosure. In any case, a CERT should aim at balancing theinterests of the parties in determining when to publicly disclose.

CERT teams have already proved to be invaluable for tackling complicated vulnerability processes,thanks to the effective identification of multi-vendor cases and the building of test laboratories, aswell as the reduction of communication overheads [4]. And, although different cases may require

different response solutions, the CERT goals should be always the same: (1) focus on repairing avulnerability as fast as possible to prevent a situation which could escalate to a crisis, (2)responsible disclosure that mitigates the vulnerability, and (3) a strategy that optimally satisfies theinterest of all involved parties.



30/88

Individual vulnerability cases may however require different response strategies. Appropriatestrategies should be developed, based on knowledge of cases already resolved and existing bestpractices [2]. Direct students to resources they may find interesting or which could provide themwith more details about the vulnerability handling process.

EVALUATION METRICSTo evaluate the outcome and performance of the exercise, the trainer answers the followingquestions: Did the students identify the most important responsibilities of a CERT team in a vulnerability

case? Did the students recognise the most important advantages and disadvantages of vulnerability

disclosure? Did the students fail to address any obvious issues in their vulnerability policy?

If any aspect of a real vulnerability policy was found unacceptable by a student, was there goodreasoning behind it?

How engaged were all sides in the role-playing scenario? Did the students identify the most problematic issues in the coordination of a multiple vendor

case?

REFERENCES[1] CERT services. http://www.cert.org/CERTs/services.html (2008)

[2] Shepherd S A, Vulnerability Disclosure: How Do We Define Responsible Disclosure?SANS, GIACSEC Practical (2003) https://www2.sans.org/reading_room/whitepapers/threats/932.php

[3] Laakso M, Takanen A, Rning J, The Vulnerability Process: a tiger team approach to resolvingvulnerability cases, in proceedings of the 11th FIRST Conference on Computer Security Incident

Handling and Response. Brisbane (1999)

[4] Rain Forest Puppy Full Disclosure Policy(RFPolicy) v2.0http://www.wiretrip.net/rfp/policy.html (2008)

[5] CISCO CISCO Security Vulnerability Policyhttp://www.cisco.com/en/US/products/products_security_vulnerability_policy.html

[6] CERT-CC The CERT Coordination Center Vulnerability Disclosure Policyhttp://www.cert.org/kb/vul_disclosure.html (2008)

[7] Laakso M, Takanen A, Rning J, Introducing constructive vulnerability disclosures, in proceedingsof the 13th FIRST Conference on Computer Security Incident Handling. Toulouse (2001)

[8] Killcrece G, Kossakowski K P, Ruefle R, Zajicek M, State of the Practice of Computer SecurityIncident Response Teams (CERTs), Technical ReportCMU/SEI-2003-TR-001 (2003)

[9] Micheal Lynn case story. http://en.wikipedia.org/wiki/Michael_Lynn

[10] WabiSabiLabi, http://www.wslabi.com/wabisabilabi/home.do

Further reading:Vulnerability disclosure policies: Descriptive list of publications regarding different incident response steps and processes can be

found in [8] (Appendix B, pages 149-153)

Ethical issues: paying for vulnerability discovery or not? Offering a bounty for security bugs, available at

http://news.cnet.com/Offering-a-bounty-for-security-bugs/2100-7350_3-5802411.html Zero Day Initiative, http://www.zerodayinitiative.com/advisories/disclosure_policy/ Bug Finders: should they be paid?available at

http://www.wired.com/science/discoveries/news/2002/08/54450?currentPage=2 Microsoft approach, available at http://blogs.zdnet.com/security/?p=130



31/88


Exercise 6Writing Security Advisories

The objective of the exercise is to provide a practical overview of whatconstitutes a good and a bad advisory publication for a CERT constituency.

Technical and management CERT staff

About 4 hours


PART 1

Task 1: Identifying key points in an advisory 30 min.

Task 2: Step-by-step comparison of real advisories 30 min.

Task 3: Comparison of real security advisories by students 60 min.

PART 2

Task 1: CVSS basics and tools 30 min.

Task 2: CVSS vectors and metrics of the DNS CVE-2008-1447 30 min.

vulnerability

Task 3: Calculation of CVSS scores by students themselves 30 min.


This exercise should be carried out when the CERT is first set up or newmembers who are responsible for writing security advisories join the team.

Main Objective

TargetedAudience

Total Duration

Time Schedule

Frequency

GENERAL DESCRIPTIONThe objective of the exercise is to provide a practical overview of what constitutes a good and a badadvisory publication for a CSIRT constituency. After completion of the exercise, the students will: Understand how to write good security advisories;

Understand the specifics of their constituency and its influence on the content of securityadvisories;

Be able to create their own template for security advisories; Have learned how to judge the severity level of an advisory; and Have learned the basics of CVSS.

Before carrying out this exercise, read this handbook carefully. The handbook lists specificadvisories that have been published by real organisations in the past CERTs, vendors, etc. You areencouraged to become familiar with them. You can also add new advisories to the exercise.

To fully carry out the exercise, you will need to give students access to the CERT Exercise BookLiveDVD. Students should be asked to boot their laptops from this DVD and select this exercise,which will contain instructions on how to proceed. The DVD will contain all the examples of theadvisories mentioned in this exercise. A short presentation as an introduction would be beneficial.

For the comparison of real advisories carried out by the students themselves, it is advisable thatyou provide each student with a printout of the checklist. Students should have paper and pencils.A whiteboard would be useful. CVSS training sessions require access to the LiveDVD or the Internet.


32/88

EXERCISE COURSEThe course of this exercise is as follows:

Introduction to the exerciseAs the trainer, you are expected to give a general introduction on the topic of writing securityadvisori

cert exercises handbook

Documents