S

Ts

DMa

b

c

a

ARRA

KEESTPV

1

ce

i

H

0h

Resuscitation 84 (2013) 992–998

Contents lists available at ScienceDirect

Resuscitation

journa l homepage: www.e lsev ier .com/ locate / resusc i ta t ion

imulation and education

actical and operational response to major incidents: Feasibility and reliability ofkills assessment using novel virtual environments�

aniel Cohena,∗, Nick Sevdalisa, Vishal Patela, Michael Taylora, Henry Leea, Mick Vokesb,ick Heysb, David Taylora, Nicola Batrickc, Ara Darzia

Division of Surgery, Department of Surgery and Cancer, Imperial College London, London, United KingdomAmbulance HART, Defence CBRN Centre, Winterbourne Gunner, Salisbury, Wiltshire, United KingdomDepartment of Emergency Medicine, St. Mary’s Hospital, Imperial College Healthcare Trust, London, United Kingdom

r t i c l e i n f o

rticle history:eceived 19 September 2012eceived in revised form 1 December 2012ccepted 12 December 2012

eywords:mergency preparednessducationimulationraumarehospital responseirtual worlds

a b s t r a c t

Objectives: To determine feasibility and reliability of skills assessment in a multi-agency, triple-site majorincident response exercise carried out in a virtual world environment.Methods: Skills assessment was carried out across three scenarios. The pre-hospital scenario requiredparamedics to triage and treat casualties at the site of an explosion. Technical skills assessment formswere developed using training syllabus competencies and national guidelines identified by pre-hospitalresponse experts. Non-technical skills were assessed using a seven-point scale previously developedfor use by pre-hospital paramedics. The two in-hospital scenarios, focusing on a trauma team leaderand a silver/clinical major incident co-ordinator, utilised the validated Trauma-NOTECHS scale to assessfive domains of performance. Technical competencies were assessed using an ATLS-style competencyscale for the trauma scenario. For the silver scenario, the assessment document was developed usingcompetencies described from a similar role description in a real-life hospital major incident plan. Thetechnical and non-technical performance of all participants was assessed live by two experts in eachof the three scenarios and inter-assessor reliability was computed. Participants also self-assessed theirperformance using identical proformas immediately after the scenarios were completed. Self and expertassessments were correlated (assessment cross-validation).Results: Twenty-three participants underwent all scenarios and assessments. Performance assessmentswere feasible for both experts as well as the participants. Non-technical performance was generallyscored higher than technical performance. Very good inter-rater reliability was obtained between expertraters across all scenarios and both technical and non-technical aspects of performance (reliability range0.59–0.90, Ps < 0.01). Significant positive correlations were found between self and expert assessment

in technical skills across all three scenarios (correlation range 0.52–0.84, Ps < 0.05), although no suchcorrelations were observed in non-technical skills.Conclusions: This study establishes feasibility and reliability of virtual environment technical and non-technical skills assessment in major incident scenarios for the first time. The development for furtherscenarios and validated assessment scales will enable major incident planners to utilise virtual technolo-gies for improved major incident preparation and training.

. Introduction

A major incident can be defined as “any event whose impactannot be handled within routine service arrangements.”1 Suchvents may be caused by natural disasters such as flooding, ter-

� A Spanish translated version of the abstract of this article appears as Appendixn the final online version at http://dx.doi.org/10.1016/j.resuscitation.2012.12.011.∗ Corresponding author at: Division of Surgery, Imperial College London, St. Mary’sospital Campus, Praed Street, London W2 1NY, United Kingdom.

E-mail address: daniel.cohen@imperial.ac.uk (D. Cohen).

300-9572/$ – see front matter © 2012 Elsevier Ireland Ltd. All rights reserved.ttp://dx.doi.org/10.1016/j.resuscitation.2012.12.011

© 2012 Elsevier Ireland Ltd. All rights reserved.

rorist acts or transportation accidents. Across the globe, legislationand guidance has been put in place to improve the emergency pre-paredness knowledge and response capability of individuals, teamsand organisations, both at a local and national level.2,3

Individuals and teams involved in pre-hospital and in-hospitalresponse must be proficient in multiple skills, and organisationsmust have adequate procedures in place to function effectivelyin major incident situations. A relevant set of standardised com-

petencies has recently been proposed for healthcare personnelinvolved in managing major incidents, although there is a lackof validated assessment methods.4,5 The skills required for effec-tive individual or team response may be very different from those

citatio

peacepael

acapfSEeoawi

oofioiaiq“trntiaaeea

ivbdvhatupmp

2

2

bted

D. Cohen et al. / Resus

racticed on a day to day basis, and these require frequent upkeep tonsure continued competence.6,7 Inadequate skills, competenciesnd preparation in healthcare personnel within the resource-onstrained, time-pressurised, stressful and potentially hazardousnvironment of a major incident cause deterioration of response,oor decision-making and inadequate performance.8–11 System-tic training and preparation are therefore important to ensure anffective and safe response, at both an individual and organisationalevel.

In both the United Kingdom and United States, designatedcute responders are required to undertake command, control andommunications preparedness exercises at regular intervals.2,3 Inddition, there are many highly regarded courses teaching therinciples of major incident response and trauma managementor clinicians, including Major Incident Medical Management andupport (MIMMS), Advanced Trauma Life Support (ATLS) and theMERGO tabletop exercise.12–15 Despite this, there is increasingvidence that the emergency preparedness of many healthcarerganisations is far from adequate. Acute care staff, many of whomre not based in the emergency environment, may not be familiarith organisational or local response plans, nor are they confident

n their individual or team roles should a major incident arise.16–18

It is currently uncertain to what extent current training meth-ds improve major incident response or outcomes.6,19 Analysesf current preparedness training reveal a number of consistentndings. Firstly, current major incident exercises focus primarilyn testing existing organisational plans, rather than the learn-ng requirements of individuals or teams; consequently feedbacknd learning opportunities are lost.20 Secondly, access to exercisess difficult and costly, and consequently few clinicians have ade-uate experience.21–23 Live exercises, whilst accepted to be thegold-standard” are challenging to organise, expensive and disrup-ive; tabletop exercises, which lack immersiveness, typically onlyun every six months for relatively few members of staff; man-equin and equipment-based drills are often unable contextualisehe stressors of a major incident environment.24,25 Thirdly, theres little standardisation in the design and assessment of exercisesnd a lack of standardised competencies makes formal assessmentnd feedback difficult.5,19,26,27 Lastly, accurate data capture in anyxercise is challenging given the often subjective nature of thevaluations, and requirement for on-site expert assessors or laterudiovisual recording and playback.28

In the light of the above, it is not surprising that the need formproved, cost-effective, immersive and accessible training pro-ision for multidisciplinary, multilevel emergency responders haseen consistently identified by high-level incident reports and aca-emic review articles.4,29–31 To address these shortcomings in aalid, clinically appropriate and cost-effective manner, our groupas recently established the feasibility of the development andcceptability of use of low-cost simulated environments for mul-idisciplinary professionals to prepare for major incident responsesing state of the art virtual environments.32 Here we report theotential of this technology to be utilised for formal skills assess-ent of personnel involved in managing a major incident at both

re-hospital and in-hospital settings.

. Methods

.1. Virtual environments

Virtual environments are 3-dimensional, online computer-

ased environments in which individuals interact using voice orext in real time by using personalised avatars. Virtual worldnvironments are being increasingly used in healthcare training,ue to their potential long-term cost effectiveness, easy online

n 84 (2013) 992–998 993

accessibility, and ability to record and reproduce simulated train-ing exercises.33,34 In this study, two-low cost virtual Environmentswere utilised: Second Life (www.secondlife.com) and OpenSimu-lator (www.opensimulator.org).

2.2. Scenario design and implementation

Three virtual scenarios were developed, based around an explo-sion at a sports stadium resulting in multiple seriously injuredcasualties (Table 1). The detailed design and development of thescenarios has been recently reported in this Journal.32 In brief, thethree scenarios were; on-scene pre-hospital response for ambu-lance staff; a trauma team working in a resuscitation room andfinally a command and control scenario for the Emergency Depart-ment ‘silver commander’ – which represents a high level leadershiprole encompassing overall clinical and managerial control of theEmergency Department during a major incident response. All sce-narios were tested at St. Mary’s Hospital Campus, Imperial CollegeLondon using HP Probook 4530s Notebooks containing a dedi-cated AMD Radeon Graphics Card. Expert assessors observed thescenarios live on screen and assessed technical and non-technicalperformance in a dedicated observation room. All participants com-pleted technical and non-technical skills self-assessment formsafter the scenarios were completed.

The sections that follow report the actions required of partici-pants in each scenario across the major incident pathway and theassessments that were applied. Details of the assessments can befound in Table 2.

2.3. Scenario 1 – pre-hospital response

Actions. Two versions of the scenario were created with inputfrom ambulance HART trainers to enable each participant to per-form team leader and triage paramedic roles. The team leader gavea briefing to the triage paramedic before entering the inner cor-don. The triage paramedic worked as a pair with an actor avatar tolocate, triage and treat the casualties and subsequently hand overthe care of the patient to a colleague at the casualty collection point.The two ambulance HART paramedics were physically isolated; allcommunication was via radio channels through the virtual Envi-ronments software. A ‘point and click’ interface was developed forinteracting with the casualties, who exhibited realistic injuries andsome of whom could communicate verbally. Deceased casualtieswere also included.

Assessments (Table 2, top panel). The quality and complete-ness of the pre-triage briefing were independently assessed by asenior HART trainer (MV) and a clinician (DC) according to a stan-dardised ‘9-point briefing’ scale extracted from the current HARTtraining syllabus. Triage, treatment and handover performancewas assessed using identified competencies defined by two HARTtrainers (MH and MV) based on the HART syllabus and nationalambulance service guidelines.35 Non-technical skills were assessedby a human factors expert (NS) and HART trainer (MV).

2.4. Scenario 2 – in-hospital trauma resuscitation

Actions. Each participant played the role of team leader in atrauma resuscitation scenario. The patient, a physiologically “real”avatar, was a 30-year-old male with a traumatic blast-related leftlower limb amputation, distracting from a cervical spine injury.By communicating verbally through the virtual environment, theteam leader could take a history and ask the team to examine the

patient, request investigations and perform interventions as neces-sary. Patient appearance and the monitored observations changedas a result of investigations or interventions as clinically appropri-ate.

994 D. Cohen et al. / Resuscitation 84 (2013) 992–998

Table 1Overview of the three major incident scenario settings and actions (see Cohen et al. for detailed description32).

Pre-hospital: ambulancehazardous area responseteam (HART)

Scenario setting:A blast outside a sports stadium; suspicion of “dirty” element to bomb.Inner cordon established on scenario commencing. Estimates of tennon-mobile casualties at scene, with diverse injury profiles includingairway obstruction, crush injuries and amputations. A point and clickinterface was developed to allow Ambulance HART to interact withboth casualties and their equipment.Actions at scene:The Ambulance HART team leader receives a written briefing fromscene commander. The team leader gives 9-point briefing to team andinstructs team into appropriate personal protective equipment. HARTtriaging team admitted into hot zone by Fire Incident Officer. HARTlocate, triage and treat casualties according to realistic resourceconstraints, before handover at the casualty collection point.Virtual environment:OpenSimulator

In-hospital: traumaresuscitation

Scenario setting:A casualty from a bomb blast is treated by a trauma team in theresuscitation room of a receiving hospital. The physiologicallyresponsive casualty, based on a purpose-built structured data model,can undergo simultaneous investigation and treatment by all traumateam members.Actions at scene:The trauma team leader instructs a trauma team to investigate andtreat the casualty according to realistic resource constraints, usingweb-based menus via a split-screen. The trauma team, consisting of ananaesthetist, surgeon and emergency department doctor, are playedby medically-qualified actors. Clinical decisions and pre-definedevents alter the state of the patient physiology, which is representedby real-time physiology monitoring on-screen.Virtual environment:Second Life

In-hospital: ‘silvercommand’ (overall controlof Emergency Departmentwith clinical andmanagerialresponsibilities)

Scenario setting:The Silver commander/Clinical Major Incident Co-ordinator mustmanage the after-effects of a bomb-blast has resulting in a suddeninflux of casualties to a trauma centre. The four-bed resuscitationroom is full; the condition of existing casualties deteriorate and newcasualties arrive.Actions at scene:The Silver commander must liaise with trauma team leaders in theresuscitation room, specialist services (such as radiology, theatres andintensive care), triage and other areas of the emergency department toensure smooth running of the major incident. Virtual telephones andrunners were developed in Second Life to aid communication.Virtual environment:Second Life

namctosAAwa(bd(

2

t

Assessments (Table 2, middle panel). Both clinical (technical) andon-technical skills (e.g. communication, leadership, etc.) weressessed. The ATLS instructor manual contains assessment andanagement forms for each of the scenarios utilised on the ATLS

ourse.12 Participants are assessed against 6–10 critical clinicalreatment decisions and a further 6–14 observed manoeuvresr decisions, which are scenario dependent. For this feasibilitytudy, a modified form was created using the principles of theTLS template and approved as being fit for purpose by twoTLS instructors and Emergency Medicine Consultants. Scenariosere observed and independently/blindly assessed in real time by

n ATLS instructor and course director (NB) and ATLS providerDC). Non-technical skills were independently/blindly assessedy a human factors expert (NS) and ATLS instructor and courseirector (NB) using the recently validated Trauma NOTECHS scaleT-NOTECHS).36

.5. Scenario 3 – in-hospital silver command

Actions. The scenario was designed to stress the combined role ofhe clinical major incident coordinator and silver commander in a

hospital major incident response, as described above. The scenariowas scripted by an experienced NHS major incident planner (NB)based on previous tabletop exercises and real-life major incidents(including the London bombings of July 7th 2005). The participantswere required to liaise with trauma teams and multidisciplinarycolleagues in the resuscitation room and other areas of the hospitalto ensure the safe running of the emergency department in a majorincident. The scenario took place entirely within a virtual environ-ment; a virtual telephone and runner were available if required.Clinician actors played the roles of Emergency Department per-sonnel and other specialties.

Assessments (Table 2, bottom panel). The technical skills assess-ment document was created using the hospital major incidentplan from St. Mary’s Hospital, London, UK; this has been usedin a number of major incidents, notably the Edgware road bomb(of July 7th 2005) and the Paddington Rail crash (1999), and isrevised and updated regularly. Key performance metrics were

extracted from the role descriptions in the major incident plan andapproved independently by an experienced major incident plannerand an Emergency Department consultant. Five “critical decisions”were rated as performed (yes/no) and a further 12 performance

D. Cohen et al. / Resuscitation 84 (2013) 992–998 995

Table 2Skills assessments across virtual worlds simulations.

Scenario Technical skills assessments Non-technical skills assessments

Assessment tools Assessment content Assessment tools Assessment content

Pre-hospital ambulanceHART

9-Point briefingassessment;Competency areasdefined by HARTtraining syllabus(yes/no scale)

1. Introduction to the scene2. General scene description3. Current situation4. Aims of operation5. Execution of tasks6. Command and control7. Communications8. Logistics9. Timings

A seven-point competency tool,developed by an organisationalpsychologist for the initial HARTrecruitment process; assessed on1–4 Likert scales(1 = unsatisfactory, 4 = excellent)using clearly defined exemplarbehaviours.

1. Team orientationand development2. Self-discipline andregulation3. Communication4. Adaptability5. Capability6. Planning orientation7. Decision making

Triage, treatment andhandover assessment;Competencies definedby HART trainingsyllabus and UKAmbulance ServiceJRCALC guidelines(yes/no scale)35

Appropriate triage category,medical interventions – these varydepending on the clinical pictureof each patient (e.g. tourniquet,oxygen), appropriate handover ofpatient including relevant historyand interventions

In-hospitaltrauma resus

Assessment documentbased on principles ofATLS checklist;competencies relevantto scenario identified bytwo ATLS trainers(yes/no scale)

Eighteen competencies assessed,including airway management,cervical spine immobilisation,treatment of haemorrhagic shockinterpretation of investigations.

T-NOTECHS (The TraumaNon-Technical Skills Scale);performance assessed on likertscale ranging from 1 to 5 usingclearly defined exemplarbehaviours.36

1. Leadership2. Cooperation andresource management3. Communication andinteraction4. Assessment anddecision making5. Situationawareness/Coping withstress

In-hospitalSilverCommand

Assessment protocolbased on hospital majorincident plan; relevantcompetenciesdeveloped byexperienced majorincident planner

Two sets of performance metricsassessed:1. Ensuring appropriate care ofcritically ill patients inresource-constrained environment(yes/no scale)2. Appropriate interactions withcolleagues in specialities withinand outside of the emergencydepartment, including Intensive

adiolo

maS4

ssme

2

Cct

TT

N

Care, Acute Wards, RTheatres

etrics relating to actions involving Radiology, Intensive Care, The-tres, the Emergency Department, the Acute Medical Unit andpecialties were assessed on 1–4 Likert scales (1 = not performed,= satisfactory performance).

Non-technical performance was assessed using the T-NOTECHScale as described for scenario 2. Both technical and non-technicalkills were independently and blindly assessed by an experiencedajor incident planner (NB), a clinician (DC) and a human factors

xpert (NS).

.6. Statistical analysis

All analyses were carried out using SPSS version 19.0 (SPSS Inc.,hicago, IL). Descriptive statistics were computed using total per-entage scores (%) for all measures to allow comparisons betweenhe different assessment metrics. No inferential analyses were

able 3echnical and non-technical assessment scores for Ambulance HART personnel (Pre-hosp

Pre-hospital ambulance HART (n = 12)

Technical skills9 Point briefing (tool range: 0–30)Triage and treatment (tool range: 0–51)

Non-technical skills Global score (tool range 7–28)

ote: Scores converted to percentages (%) and rounded up to the nearest percentage poin

gy and

carried out on these data due to small sample sizes. Inter-raterreliability between the expert assessors was calculated using intr-aclass correlation coefficients (ICCs) separately for the technicaland non-technical skill scores. ICCs of 0.70 or higher typically indi-cate acceptable levels of inter-assessor agreement. The relationshipbetween expert assessment and self-assessment of performanceby participants was investigated using Spearman’s rho betweenexpert raters’ and participants’ own scores (to test for cross-validation). For all analyses, statistical significance was set atP < 0.05 or lower.

3. Results

3.1. Participants’ demographics

In total 23 participants completed the three scenarios, consist-ing of 12 ambulance HART practitioners, six surgical registrars who

ital scenario).

Expert assessment(median % score,range)

Self assessment(median % score, range)

63% (50–80%) 68% (40–83%)59% (52–64%) 61% (51–82%)79% (29–93%) 73% (50–86%)

t for ease of comparison.

996 D. Cohen et al. / Resuscitation 84 (2013) 992–998

Table 4Technical and non-technical assessment scores for the hospital-based scenarios (trauma resus and silver command scenarios).

Trauma resus (n = 6) Silver command (n = 5)

Expert assessment(median % score, range)

Self assessment(median % score, range)

Expert assessment(median % score, range)

Self assessment(median % score, range)

Technical skills (tool rangetrauma resus 0–18, silvercommand 12–53)

61% (58–64%) 69% (33–94%) 69% (66–79%) 55% (43–70%)

Non-technical skills (tool range 85% (75–91%) 68% (60–100%) 90% (72–98%) 72% (48–76%)

N e poin

aDtcaAs

3d

aa(5apacmIntss(

3

pes(ams

TR

5–25)

ote: Scores converted to percentages (%) and rounded up to the nearest percentag

ct as trauma team leaders in their current role and five Emergencyepartment Consultants, all of whom could be expected to fulfill

he role of silver commander in a major incident (with some havingarried out this role in the recent past). All scenarios were observednd assessed live by expert assessors and were recorded in full.ll participants completed the technical and non-technical skillself-assessments immediately after the scenarios ended.

.2. Technical and non-technical skills across scenarios:escriptive analyses

Pre-hospital participant performance is shown in Table 3. Self-ssessment of technical performance in 9-point briefing and triagend treatment was higher than what the expert assessors scoredself-assessments of 68% and 61% vs expert assessments of 63% and9% respectively). This finding was reversed in the non-technicalssessment, where participants under-scored themselves com-ared to the expert scores (73% vs 79%). Variability in expertssessments was generally lower in the technical assessmentsompared to the non-technical ones, whereas participants wereore variable in their own self-assessment of their technical skills.

n both the Trauma Resus and Silver Command hospital sce-arios experts rated non-technical performance higher than theechnical performance. Finally, as in the pre-hospital scenario,elf-assessment of technical skills was overall higher than theelf assessment of non-technical skills compared to expert scoresTable 4).

.3. Inter-assessor reliability analyses

Inter-rater reliability analyses are reported in Table 5 (leftanel). There were significant and strong correlations between thexpert assessors across both the technical elements of the threecenarios (ICC range 0.73–0.90) and the non-technical elements

ICC range 0.59–0.89). This is a very strong result, as it suggestsdequate reliability to carry out skills and performance assess-ents in virtual Environments for such scenarios for five of the

ix skills categories (those where ICC was higher than 0.70), with

able 5eliability in expert scoring (intraclass coefficients) and correlation between expert and s

Pre-hospital ambulance HART scenario Technical skillsNon technical skills

In-hospital: trauma resus scenario Technical skillsNon technical skills

In-hospital: silver command scenario Technical skillsNon-technical skills

* P < 0.05.** P < 0.01.

*** P < 0.001.

t for ease of comparison.

the remaining category (non-technical performance of the SilverCommander) approaching this level.

3.4. Relationship between expert and self skill assessments

Correlations between expert and self-assessments are reportedin Table 5 (right panel). Significant and positive correlations werefound between expert and self-assessed technical performance forall three scenarios – ranging from 0.52 in the pre-hospital scenarioto 0.84 in Silver command scenario. Although not as high as theexpert reliabilities, these correlations suggest that the participantsdid have some insight into their levels of technical performance. Incontrast, there were no significant correlations between expert andself-assessments of non-technical performance in any of the threescenarios. This finding complements the discrepancies obtained indescriptive analyses of expert vs self scoring – we return to this inSection 4.

4. Discussion

Successful major incident response requires the application ofa complex skill set by multi-disciplinary teams working in unpre-dictable and hazardous environments. Currently available trainingand preparation modalities are compromised by cost, accessi-bility and difficulties with data capture to assess and debriefperformance.20 In a previous study, we reported the feasibilityand applicability of training via state of the art ‘virtual worlds’environments.32 This study has demonstrated the feasibility of util-ising these immersive, low-cost virtual training environments forskills assessment as an adjunct to existing training and assessmenttools in major incident preparation. Formal performance metrics(technical and non-technical) were successfully utilised within vir-tual environments across multiple disciplines, overcoming someof the shortfalls of current methodologies. This was achieved byutilising validated real-world assessment tools for non-technical

skills, and the creation of assessment tools for technical skillsbased on expert-defined, appropriate performance metrics. Previ-ous research has demonstrated the feasibility of utilising virtualworlds and gaming platforms for different components of major

elf-assessment (Pearson r coefficients) across scenarios and skill domains.

Agreement between experts’assessments (inter-raterreliability; ICC)

Agreement between expert andself assessments (cross-validation;Spearman’s rho r)

0.90*** 0.52*

0.89** 0.480.73*** 0.59*

0.59** 0.400.85*** 0.84***

0.68*** 0.46

citatio

iir

watbaiwstmattrtao

nipnitmsfctiwtclfwi

vanaTbftutdcm

4

whnavm

D. Cohen et al. / Resus

ncident preparation.34,37 To the best of our knowledge, this studys the first to attempt to formalise skills assessment in virtual envi-onments.

The requirement for testing organisational response frame-orks during exercises does not often allow for individual skills

ssessment and feedback. Furthermore, data capture and analysis isime consuming and resource intensive. Lack of high-quality feed-ack to individuals, teams and organisations has been identifieds a significant barrier to future skill development and outcomemprovement.20,25 During the virtual environment exercises that

e report here, the feasibility of both technical and non-technicalkills assessment was successfully demonstrated across opera-ional and tactical response levels, including both clinical and

anagerial roles. Formal (i.e. systematic and reliable) individualnd team skills assessment is therefore feasible in these virtualraining environments. In addition, the tactical response theme ofhe scenarios (ambulance HART team leader and Silver commandoles) enabled elements of real organisational major incident planso be tested (derived from our own hospital); although not formallyssessed here, this could improve individual and team awarenessf existing emergency response plans.

The role of non-technical skills, such as teamwork and commu-ication, in reducing error and improving patient safety is being

ncreasingly recognised in healthcare. Suboptimal non-technicalerformance is known to impact on the performance of tech-ical skills and may lead to adverse events, especially in high

ntensity environments.38–40 The training and assessment of non-echnical skills such as leadership and situational awareness in

ajor incident response, where responders may work in highlytressful, rapidly changing, unfamiliar environments, is there-ore paramount, although under-represented in current trainingurricula.36,41 In this study, reliable, expert assessment of non-echnical skills was demonstrated. However, as has been shownn other healthcare specialties, non-technical skill self-assessment

as only moderately correlated with expert scores.39,42 Formalraining and general awareness of non-technical skills in health-are has only emerged in the past decade – thus these skills areikely less well embedded into the clinical mindset. There is there-ore a need for improved training and awareness of this skill set –hich is now possible via numerous validated tools and modules,

ncluding those reported here.Technical skill assessment in the virtual setting focuses on indi-

idual or team decision-making, rather than an ability to performtask, which is assumed. For example, in the pre-hospital sce-

ario the decision to apply a tourniquet to an amputated limb wasssessed, rather than the ability to apply the tourniquet correctly.he decision making processes in major incident settings haveeen identified as being often deficient, yet crucial to a success-ul response.23,43,44 Unfortunately, few clinicians are adequatelyrained to perform such tasks in stressful large-scale crisis sit-ations where resources may be constrained and conventionalreatment pathways are inappropriate.17,41,44,45 This study hasemonstrated that virtual environment simulation can provide aontextualised, immersive setting in which such decisions can beade and assessed.

.1. Limitations

The results of this study, whilst promising, must be interpretedith some caution. Firstly, whilst the feasibility of skills assessmentas been demonstrated, the construct validity of the scenarios has

ot been established, so further validation work is necessary usinglarger sample size in an appropriately powered study. Predictivealidity would be difficult to establish given the random nature ofajor incident occurrence.28 Secondly, the scenario assessments

n 84 (2013) 992–998 997

were performed using a variety of metrics that had been faceand content validated by expert trainers, but not tested elsewhereprior to these scenarios (with the exception of the validated T-NOTECHS). Thirdly, this study was primarily aimed at individualand team assessment, rather than that organisational response.Determining the organisational preparedness of acute respondersis challenging; both questionnaire and observational methods havebeen developed for this purpose but are not yet validated or inwidespread use.5,46 Incorporating a full-scale organisational pre-paredness exercise into a virtual environment (including strategic“gold” command) may enable wider access, improved understand-ing of incident plans and encourage the further developmentof appropriate metrics to assess organisational response. Suchorganisation-level exercises are available and have been used bythe military.47 Further research is required to determine whethersimilar exercises can be delivered in virtual environments – andindeed whether they can improve organisational response and per-formance in major incidents. A step towards this would be thedevelopment of multiple trauma scenarios that could run syn-chronously with a silver-command scenario, which would betterreflect the reality of major incident response.

4.2. Implications

Virtual environments have advantages over current liveor tabletop exercises. For example, our scenarios were allrecorded in full from scene-generic and participant-specific views,using straightforward and inexpensive computer technology.The difficulties of live assessment in fast-paced, geographicallyspread, multi-person emergency scenarios have been previouslydocumented.48–50 Provision of accurate participant feedback hasbeen identified as the most important feature of simulation-basedmedical education.51 Review of scenario recordings may enableimproved trainer-led assessment and development, by providingevidence-based feedback on actions.50 With suitable training, par-ticipants may also be able to self-assess their performance or that oftheir peers using video review, which would limit the requirementfor expert trainers to be present at each scenario. Furthermore, theability to modify and rerun scenarios could provide a mechanismfor analysis of skill improvement over time and provide increas-ingly challenging environments as practitioners gain additionalskills which can be contextualised into the major incident setting.In addition, virtual environments allow individuals and teams thatare physically distant to exercise together in an immersive envi-ronment. Further validation of exercise assessment metrics wouldenable individuals, teams and organisations to benchmark theirperformance against, and learn from other peer groups nationally,and potentially around the world, by using the virtual environmentas a live educational platform. Lastly, the scenarios demonstrated inthis study demonstrate the future generic training potential of thevirtual platform, especially for resuscitation and other emergencymedical settings.

5. Conclusions

Major incident training and assessment is complex, encompass-ing multiple competencies and disciplines both inside and outsideof the hospital setting. By demonstrating the feasibility of technicaland non-technical skills assessment in virtual environments, thisstudy has implications for major incident trainers and planners.

Improving the accessibility, immersiveness and feedback potentialof major incident exercising is necessary; this study has demon-strated the potential for virtual environments to achieve these goalsas an adjunct to existing training and assessment methods. Further

9 citatio

io

F

Arb

C

A

i2

R

1

1

1

1

1

1

1

1

1

1

2

2

222

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

4

4

4

4

4

4

4

4

4

4

98 D. Cohen et al. / Resus

nvestigation is needed to determine the optimal use and roll-outf these training and assessment environments.

unding

This research was funded by a grant from the Health Protectiongency, United Kingdom. Nick Sevdalis is affiliated with the Impe-ial Centre for Patient Safety and Service Quality, which is fundedy the National Institute for Health Research (UK).

onflict of interest statement

There are no conflicts of interest.

ppendix A. Supplementary data

Supplementary data associated with this article can be found,n the online version, at http://dx.doi.org/10.1016/j.resuscitation.012.12.011.

eferences

1. Department of Health Emergency Preparedness Division. NHS emergency plan-ning guidance 2005. London: DoH; 2005.

2. Civil Contingencies Secretariat. Civil contingencies act 2004. London: CabinetOffice; 2004.

3. US Department of Homeland Security. National Response Framework. Wash-ington DC; 2008.

4. Schultz CH, Koenig KL, Whiteside M, Murray R. Development of national stan-dardized all-hazard disaster core competencies for acute care physicians, nurses,and EMS professionals. Ann Emerg Med 2012;59:196–208.

5. Kaji AH, Langford V, Lewis RJ. Assessing hospital disaster preparedness: a com-parison of an on-site survey, directly observed drill performance, and videoanalysis of teamwork. Ann Emerg Med 2008;52:195–201.

6. Ryan J, Montgomery H. The London attacks – preparedness: terrorism and themedical response. N Engl J Med 2005;353:543–5.

7. Chaput CJ, Deluhery MR, Stake CE, Martens KA, Cichon ME. Disaster training forprehospital providers. Prehosp Emerg Care 2007;11:458–65.

8. Milsten A. Hospital responses to acute-onset disasters: a review. PrehospitalDisaster Med 2000;15:32–45.

9. Tokuda Y, Kikuchi M, Takahashi O, Stein GH. Prehospital management of sarinnerve gas terrorism in urban settings: 10 years of progress after the Tokyosubway sarin attack. Resuscitation 2006;68:193–202.

0. Banauch G, McLaughlin M, Hirschhorn R, Corrigan M, Kelly K, Prezant D. Injuriesand illnesses among New York City Fire Department rescue workers afterresponding to the World Trade Center attacks (Reprinted from MMWR, vol 51,pg 1–5, 2002). J Am Med Assoc 2002;288:1581–4.

1. Barkemeyer BM. Practicing neonatology in a blackout: the University HospitalNICU in the midst of Hurricane Katrina: caring for children without power orwater. Pediatrics 2006;117(Pt 3):S369–74.

2. American College of Surgeons. Advanced Trauma Life Support (ATLS) instructormanual. Chicago, IL: American College of Surgeons; 1997.

3. Carley S, Mackway-Jones K. Major incident medical management and support:the practical approach in the hospital. London/Oxford: BMJ/Blackwell; 2005.

4. Hodgetts TJ, Porter C. Major incident management system: the scene aid mem-oire for major incident medical management and support. London: BMJ; 2002.

5. Lennquist S. The emergotrain system for training and testing disaster prepared-ness: 15 years of experience. Int J Disaster Med 2003;1:25–34.

6. Candiotti KA, Kamat A, Barach P, Nhuch F, Lubarsky D, Birnbach DJ. Emergencypreparedness for biological and chemical incidents: a survey of anesthesiologyresidency programs in the United States. Anesth Analg 2005;101:1135–40 (tableof contents).

7. Galante JM, Jacoby RC, Anderson JT. Are surgical residents prepared for masscasualty incidents? J Surg Res 2006;132:85–91.

8. Wong K, Turner PS, Boppana A, et al. Preparation for the next major incident:

are we ready? Emerg Med J 2006;23:709–12.

9. Williams J, Nocera M, Casteel C. The effectiveness of disaster training for healthcare workers: a systematic review. Ann Emerg Med 2008;52:211–22, e1–e2.

0. Cohen D, Sevdalis N, Patel V, Taylor D, Batrick N, Darzi A. Major incident prepa-ration for acute hospitals: current state of the art, training needs analysis,

5

5

n 84 (2013) 992–998

and the role of novel Virtual Worlds simulation technologies. J Emerg Med2012;43:1029–37.

1. Wimbush S, Davies G, Lockey D. The presentation and management of victims ofchemical and biological agents: a survey of knowledge of UK clinicians. Resus-citation 2003;58:289–92.

2. Waeckerle JF. Disaster planning and response. N Engl J Med 1991;324:815–21.3. Burstein JL. The myths of disaster education. Ann Emerg Med 2006;47:50–2.4. Idrose AM, Adnan WA, Villa GF, Abdullah AH. The use of classroom training and

simulation in the training of medical responders for airport disaster. Emerg MedJ 2007;24:7–11.

5. Dausey DJ, Buehler JW, Lurie N. Designing and conducting tabletop exercises toassess public health preparedness for manmade and naturally occurring biolog-ical threats. BMC Public Health 2007;7:92.

6. Burstein JL. Smoke and shadows: measuring hospital disaster preparedness. AnnEmerg Med 2008;52:230–1.

7. Daily E, Padjen P, Birnbaum M. A review of competencies developed for dis-aster healthcare providers: limitations of current processes and applicability.Prehospital Disaster Med 2010;25:387–95.

8. Hsu EB, Jenckes MW, Catlett CL, et al. Effectiveness of hospital staff mass-casualtyincident training methods: a systematic literature review. Prehospital DisasterMed 2004;19:191–9.

9. Hallett H. Report under Rule 43 of The Coroner’s Rules 1984 Coroner’s Inquestsinto the London bombings of 7th July 2005. London; 2011.

0. Brevard SB, Weintraub SL, Aiken JB, et al. Analysis of disaster response plans andthe aftermath of hurricane katrina: lessons learned from a level I trauma center.J Trauma – Inj Infect Crit Care 2008;65:1126–32.

1. Lavery GG, Horan E. Clinical review: communication and logistics in the responseto the 1998 terrorist bombing in Omagh. North Ir Crit Care 2005;9:401–8.

2. Cohen D, Sevdalis N, Taylor D, et al. Emergency preparedness in the 21st cen-tury: training and preparation modules in virtual environments. Resuscitation2012;84:78–84.

3. Patel V, Aggarwal R, Osinibi E, Taylor D, Arora S, Darzi A. Operating room intro-duction for the novice. Am J Surg 2011;203:266–75.

4. Heinrichs WL, Youngblood P, Harter P, Kusumoto L, Dev P. Training healthcarepersonnel for mass-casualty incidents in a virtual emergency department: VEDII. Prehospital Disaster Med 2010;25:424–32.

5. Fisher J, Brown S, Cooke M, Joint Royal Colleges Ambulance Liaison Committee.UK Ambulance Service clinical practice guidelines. London: IHCD; 2006.

6. Steinemann S, Berg B, DiTullio A, et al. Assessing teamwork in the traumabay: introduction of a modified “NOTECHS” scale for trauma. Am J Surg2012;203:69–75.

7. Knight JF, Carley S, Tregunna B, et al. Serious gaming technology in major inci-dent triage training: a pragmatic controlled trial. Resuscitation 2010;81:1175–9.

8. Hull L, Arora S, Aggarwal R, Darzi A, Vincent C, Sevdalis N. The impact of non-technical skills on technical performance in surgery: a systematic review. J AmColl Surg 2012;214:214–30.

9. McKay A, Walker ST, Brett SJ, Vincent C, Sevdalis N. Team performance in resus-citation teams: comparison and critique of two recently developed scoring tools.Resuscitation 2012, http://dx.doi.org/10.1016/j.resuscitation.2012.04.015.

0. Reader T, Flin R, Lauche K, Cuthbertson BH. Non-technical skills in the intensivecare unit. Br J Anaesth 2006;96:551–9.

1. Uddin SG, Barnett DJ, Parker CL, Links JM, Alexander M. Emergency prepared-ness: addressing a residency training gap. Acad Med 2008;83:298–304.

2. Arora S, Miskovic D, Hull L, et al. Self vs expert assessment of technical andnon-technical skills in high fidelity simulation. Am J Surg 2011;202:500–6.

3. Inglesby TV. Observations from the Top Off exercise. Public Health Rep2001;116(Suppl. 2):64–8.

4. Hillsborough: the report of the Hillsborough independent panel. London: TheStationery Office; 2012.

5. Einav S, Feigenberg Z, Weissman C, et al. Evacuation priorities in mass casu-alty terror-related events: implications for contingency planning. Ann Surg2004;239:304–10.

6. Savoia E, Biddinger PD, Burstein J, Stoto MA. Inter-agency communication andoperations capabilities during a hospital functional exercise: reliability andvalidity of a measurement tool. Prehospital Disaster Med 2010;25:52–8.

7. Arora S, Sevdalis N. HOSPEX and concepts of simulation. J R Army Med Corps2008;154:202–5.

8. Green GB, Modi S, Lunney K, Thomas TL. Generic evaluation methods for disasterdrills in developing countries. Ann Emerg Med 2003;41:689–99.

9. von Wyl T, Zuercher M, Amsler F, Walter B, Ummenhofer W. Technical and non-technical skills can be reliably assessed during paramedic simulation training.Acta Anaesthesiol Scand 2009;53:121–7.

0. Georgiou A, Lockey DJ. The performance and assessment of hospital traumateams. Scand J Trauma Resusc Emerg Med 2010;18:66.

1. Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features anduses of high-fidelity medical simulations that lead to effective learning: a BEMEsystematic review. Med Teach 2005;27:10–28.