JUST-IN-TIME SUPPORT: ADAPTIVE, INTELLIGENT SYSTEMS

TO ENHANCE HUMAN PERFORMANCE

by

Paul Picciano

A dissertation submitted to the faculty of

The University of Utah

in partial fulfillment of the requirements for the degree of

Doctor of Philosophy

Department of Psychology

The University of Utah

May 2006

INTRODUCTION .......................................................................................................................5

What is Just-in-Time Support (JITS).......................................................................................5 The Growing Need for JITS ....................................................................................................5 JITS applied to Emergency Response: CPR and defibrillation ...............................................9

Sudden Cardiac Arrest (SCA)..............................................................................................9 Responding to SCA............................................................................................................12 JITS to Improve SCA Response ........................................................................................15

BACKGROUND .......................................................................................................................17

Intelligent Tutoring Systems (ITS) ............................................................................................18 Elementary Components of ITS........................................................................................19

Differences Between ITS and JITS........................................................................................21 Objectives ..........................................................................................................................21 Time available....................................................................................................................22 Processing strategies ..........................................................................................................23 System input.......................................................................................................................23

Foundations of JITS...............................................................................................................23 The Contextual Control Model (COCOM)........................................................................23 Expertise ............................................................................................................................29 Feedback ............................................................................................................................33 Visual Displays ..................................................................................................................36 Task Analysis (TA)............................................................................................................39

Structure of Just-in-Time Support Systems ...........................................................................43 Plans...................................................................................................................................44 Cues....................................................................................................................................45 Feedback ............................................................................................................................46

Hypotheses.............................................................................................................................48 METHODS ................................................................................................................................50

Participants.............................................................................................................................50 Training..................................................................................................................................51

Participant Training ...........................................................................................................51 Coder Training ...................................................................................................................52

Apparatus ...............................................................................................................................53 Procedure ...............................................................................................................................56

RESULTS ..................................................................................................................................59

Physiological Data .................................................................................................................59 COCOM Data ........................................................................................................................66

Control Mode.....................................................................................................................66 Protocol Adherence................................................................................................................68 Survey Data............................................................................................................................71

Action.................................................................................................................................72 Feedback ............................................................................................................................73 Outcome.............................................................................................................................75

DISCUSSION............................................................................................................................76

CPR Task performance ..........................................................................................................76 JITS Effect .............................................................................................................................78 COCOM Parameters as Dependent Measures .......................................................................81 Conclusions............................................................................................................................86

Limitations .........................................................................................................................86 Contribution of JITS ..........................................................................................................89

APPENDICIES ..........................................................................................................................91

Appendix A: BLS Flow Chart ...............................................................................................91 Appendix B : COCOM Coding Sheet....................................................................................94 Appendix C : Experimental Data Forms................................................................................95 Appendix D : Screening Questionnaire .................................................................................99 Appendix E: Instructions ....................................................................................................100 Appendix F : Statistical Calculations...................................................................................102

Fisher’s Exact Test: GRE-NO n = 6.................................................................................102 MANOVA for Sensor Data ............................................................................................103 Inter-rater Reliability .......................................................................................................110 MANOVA for COCOM Coding .....................................................................................115 MANOVA for Post-Run Questionnaire (Survey) Results...............................................118 Fisher’s Exact Test: Instances of Feedback ....................................................................121 Fisher’s Exact Test: Protocol Sequencing. .....................................................................122

Appendix G: CPR Release/Consent.....................................................................................123 Appendix H: Basic Life Support (BLS)...............................................................................124 Appendix I: The Future of Just-in-Time Support Systems.................................................125

REFERENCES ........................................................................................................................129

List of Tables

1. Group means for CPR performance variables. ...................................................... 61 2. Group means for COCOM classification (in percent). ........................................... 66 3. Mean number of protocol steps executed correctly (9 max).................................. 70 4. Number of responders exhibiting correct sequencing of subtasks......................... 71 5. "Action" survey results .......................................................................................... 72 6. "Feedback" survey results...................................................................................... 73 7. Number of participants provided feedback by the system.................................... 74 8. "Outcome" survey results....................................................................................... 75

List of Figures

Figure 1. 3 (Training) x 2 (Device) nested factorial design............................................ 52 Figure 2. The device in use on training mannequin......................................................... 54 Figure 3. Components of the device. ............................................................................... 56 Figure 4. Snapshot of video instruction "place headrest" ................................................ 58 Figure 5. Group means of chest compression rate for each cycle ................................... 64 Figure 6. Group means of inspired volume per breath for each cycle............................. 65

INTRODUCTION

������������������������������������������������������ ����������������� ����������������� ����������������� �������������������The 1999 film The Matrix portrayed examples of “instant expertise” by means of

downloading knowledge and abilities directly into the brain with plug-and-play

convenience. This technique proved particularly effective when surprise encounters

demanded unanticipated skills sets (such as the sudden need to pilot a rotary aircraft).

Here, a novice was thrust into a critical situation lacking the expertise to deliver a

successful (or even non-fatal) outcome. While the prospect of instantly imparting piloting

skills remains science fiction, the work presented here embodies a pursuit to design

support systems enabling operators to far exceed their baseline capabilities at the moment

an event demands intervention.

Just-in-Time Support (JITS) provides a theoretical framework designed to

enhance human performance employing contextually aware systems. Specifically, the

current focus of JITS systems aims to assist non-expert operators complete exigent tasks

via adaptive support. The development of JITS systems challenges designers to assess

task requirements, provide resources, and plan for contingencies. A robust understanding

of the users and their needs is vital for driving successful human-system interactions.

Most determinant, decomposable tasks are potential candidates for JITS. It is envisioned

JITS can provide significant benefits when confronting emergency situations,

infrequent/off-nominal procedures, and tasks novel to the user.

��������������������������������������������������������������������������������������������������������

Just-in-time support (JITS) systems are intended to bridge the expertise gap and

empower non-expert users with the ability to achieve favorable results in unfamiliar

situations. Human operators often struggle with tasks due to inadequate skills and

insufficient knowledge. Frustration mounts as novel encounters with technology

continue to proliferate burdening luddites as well as the technically savvy (Gleick, 1999;

Naki�enovi� & Grübler, 1991; Sheridan, 2002).

Astute observers understand the unceasing advancement and infusion of

technology is replete with benefits as well as challenges. Improvements in technology

deliver new capabilities, liberating efficiencies and welcomed convenience never before

experienced. Minimally invasive surgery, remotely piloted vehicles, and self check-out

lines have evolved largely through technological enhancements.

Unintended consequences also emerge, however. Capricious system behavior,

substantial error rates, and user frustration reflect unmitigated interaction deficiencies.

Users often find new technology enigmatic. The technology itself often receives the

focus without consideration for operator interaction. New products and systems continue

to demand more from operators (in terms of expertise) and provide less (interaction cues

and guidance). Thus, the number of human-system interactions involving novice

operators is expected to increase, resulting in unsatisfactory performance or an inability

to interact at all.

Unfamiliar tasks and technology confront society on a regular basis. This

challenge is omnipresent in developed modern nations. Consumers with new mobile

phones often cannot access their voicemail for days and encounters with the latest

features prove frustrating. Evolving home video and audio capabilities, with dozens of

functions hidden in an arsenal of remotes, frequently challenge users in the simplest tasks

(e.g., power on).

Even more pressing than a phone message, an unprepared public may encounter

urgent circumstances such as vehicle malfunctions, fires in their home, or the need to

administer life-saving treatment. A JITS system could provide assistance in all of these

situations by transferring expertise at the instant it is needed.

Current remedies addressing gaps in expertise have numerous limitations. Of

course, there are many training programs and simulations for teaching, and the use of

computers for learning is not revolutionary. However, current solutions for real-time,

adaptive support are scant. First and foremost, they fail to deliver the training

concurrently with the task. This hobbles the user with a significant memory burden that

is subject to temporal displacement (the time from training), and is further exacerbated by

the stress of the situation. For example, many large corporations utilize computer-based

training (CBT) that covers topics from intellectual property to evacuation. A CBT

approach helps teach large numbers of employees (it’s difficult to get 50,000 through

lectures), on their own time. However, there is difficulty retaining the information over

time, especially when employees view the training as peripheral to their job and simply

try to complete it as quickly as possible.

Second, the implemented expert models are often inflexible. Computerized

provider order entry (CPOE) systems provide such an example. Though designed to help

reduce adverse drug events and control costs, inflexible thresholds for drug dose and drug

interactions as well as pop-up windows have been a source of frustration for users

(Miller, R., Waitman, L., Sutin Chen, S. & Rosenbloom, S., 2005)

Further, these systems require direct manipulation (usually on a computer), which

may be unrelated to the goal. This obligates the operator to perform two duties - the

primary task (accomplishing the objective), and the secondary task of updating the aiding

system. One such scenario would be the case of an in-vehicle GPS directing a driver

down a closed road. Deficiencies such as described above inspired the development of

the JITS approach.

JITS solutions strive to circumvent task impediments and provide concurrent

support to novices in specific task objectives. By providing workable plans, directive

cues, and actionable feedback in a manner suited to the non-expert, a JITS system can

help novice users accomplish tasks that would otherwise have an extremely low

probability for success. The JITS framework success resides in providing users real-time,

adaptable, transparent, information to support task facilitation in the given context.

For instance, an untrained person faced with a victim requiring CPR may feel

paralyzed by the lack of knowledge and the urgency of the situation. For those willing to

make a magnanimous attempt, aspiration of the stomach and broken ribs could result

from improper CPR technique; this commonly occurs even when delivered by trained

professionals (Lederer et. al. 2004).

Such a circumstance provided an ideal proving ground for JITS as it encapsulated

all the parameters and challenges JITS was designed to mitigate. The demands of basic

life support (BLS, see Appendix H) are well defined and prescribed, yet demands

adaptation based on victim needs. The CPR task requires knowledge that is not

commonly held and skills widely unpracticed. The automated external defibrillator

(AED) compels an interaction with unfamiliar technology. The circumstance demands

immediate action in the absence of experts.

The nature of JITS is well suited for developing a BLS aid. Cues can initiate user

actions. The active sensors determine the adequacy of those actions and delivers

feedback to drive performance in the right direction. It was believed this approach would

be far more successful than a “canned” program that doled out instructions assuming a

putative response scenario without regard to the specific context. Current AED solutions,

such as those of Phillips and Zoll offer only a single treatment path which introduce

delays and may not be optimal for the victim.

The first step (after opening the package and removing the victim’s shirt) of the

Phillips HeartStart OnSite guides the responder through pad placement for heart rhythm

analysis (demo: http://www.medical.philips.com/main/products/resuscitation/products/onsite/onsite_demo.asp).

In contrast, Zoll’s AEDPlus (demo: http://www.zoll.com/parent2.swf) first instructs the

responder to call 911 and check for responsiveness. The debate here is not which

sequence is better, but to point out that neither system can adapt its sequence to best serve

the specific context. In some cases victim benefits from an immediate shock. Others

need oxygen delivered as soon as possible. The objective of a JITS system is to make

that determination and guide the user accordingly.

����������������� �������� �������!�"# ���������$�������������������������� �������� �������!�"# ���������$�������������������������� �������� �������!�"# ���������$�������������������������� �������� �������!�"# ���������$�������������

��������������� ��������

Cardiovascular disease is the leading cause of death in the United States with over

696, 000 cases recorded by the Center for Disease Control (CDC) in 2002 (National

Center for Health Statistics, 2002). In particular, SCA claims 300,000 to 400,000

American lives annually (AHA, 2005, Huikuri, Castellanos & Myerburd, 2001, Zipes &

Wellens, 1998). Ventricular fibrillation is believed to be the insidious perpetrator

associated with the majority of these heart maladies (AHA, 2005, Cummins & Hazinski,

2000). In addition to the mortality in the United States, SCA causes 700,000 European

deaths per annum (ILCOR, 2005).

Unfortunately, most victims that experience SCA have only a small chance of

survival (Stiell, Nichol, Wells, De Maio, Nesbitt, Blackburn, & Spaite, 2003; Zipes &

Wellens, 1998). Increasing the probability of survival requires skilled and

knowledgeable responders. Usually the task falls to EMTs, firefighters, police officers,

or other highly trained responders. Unfortunately, skilled response alone does not

guarantee healthy outcomes; the elapsed time between incident and intervention is

believed to be more critical (Cummins, 1989, ILCOR, 2005 ). Disturbingly, there is a

growing literature demonstrating professionals’ deficiencies in both performance and

response time.

Despite their extensive training, studies continue to document trained responders’

suboptimal CPR performance. The accrued evidence suggests professionals must

improve their technique to propagate healthier outcomes (Abella, Avarado, Myklebust, et

al. 2005; Wik, Kramer-Johansen, Mykleburst , Sorebo, Svensson, Fellows, & Steen,

2005).

Abella et al. (2005) recently published their findings assessing the quality of CPR

delivered by professionals inside hospitals in Chicago between 2002 and 2004. They

observed numerous instances in which resuscitation performance did not adhere to the

American Heart Association (AHA) guidelines. The deficiencies included chest

compression rates below 90 per minute (the guideline prescribes 100/min), insufficient

compression depth, and excessive ventilation rates. The suboptimal performances

witnessed lead the researchers to conclude that even professional responders could

benefit from active monitoring and feedback during the task.

Wik et al., (2005) performed a similar examination but studied professionals out

of the hospital performing CPR in the field. This European study composed mostly of

paramedics and nurse anesthetists reported results consonant with Abella et al. (2005).

Of the total time available during the case, responders failed to provide chest

compressions for 48% of the available time. Valenzuela, Kern et al. (2005) indict the

frequent interruption of chest compressions for the poor survival rate in out-of-hospital

SCA. Insufficient compression activity combined with inadequate compression depth

(only 28% of compressions given adhered to the guideline) revealed a problematic

departure from the standards or CPR.

But even ideal execution of the protocol is futile if it is not delivered immediately

after SCA onset. Numerous studies assert that timeliness of response is the primary

factor in promoting healthy outcomes (Bunch et. al, 2003, Cummins & Hazinski, 2000,

Hallstrom & Ornato, 2004, Marenco, Wang, Link, Homoud, & Estes, 2001; Valenzuela,

Roe, Nichol, Clark, Spatie & Hardman, 2004). Each passing moment devoid of

intervention imperils the victim’s survival. For each minute without life-sustaining

measures, the victim’s probability of survival decreases 7-10% (Cummins, 1989).

Twelve minutes from collapse without circulating oxygen reduces the chance of survival

to a grim range of 2-5% (Cummins & Hazinski, 2000).

Survival rates attest to the historically slow response times. Zipes & Wellens

(1998) postulate a 3-5% discharge survival rate (victims that survive from cardiac event

to hospital discharge) nationally. In cities where there is widespread deployment of

AEDs and civilian training initiatives, survival rates make a small climb to a

disappointing 15%. Other studies lend credence to these concerns. Stiell, Nichol, Wells,

De Maio, Nesbitt, Blackburn, & Spaite (2003) reported on a 20 community study with

more than 8000 victims and found survival to discharge improved to a meager 5.2% with

citizen initiated CPR. Formerly a community with a superior discharge survival rate,

Dade County fell from 23% in the 1970s to only 9% in 1996 (Myerburg, Fenster, Velez,

Rosenberg, Lai, SKurlansky, Newton, Knox, & Castellanos, 2002). The alarming

decline served as the impetus for their examination of response times.

Significantly better survival rates have been observed in airline and casino

studies. These circumstances represent the confluence of optimized response

characteristics: trained, equipped rescuers in close proximity to provide immediate

resuscitation. An airline study in the late 1990s reported a discharge survival rate of 40%

(for victims with ventricular fibrillation treated with AEDs). Similarly, in a study

conducted within several casinos, trained security officers with AEDs readily available

garnered a survival rate of 74% for victims treated within three minutes of the cardiac

event. It is important to note that even at the level of greatly reduced response times (in

comparison to EMT response), in these domains, minutes were critical. In the casino

analysis, treatment after the 3 minute threshold produced a substantial decline in survival

rate to 49%.

�� ��������������

The AHA gathering of the 2nd Public Access Defibrillation (PAD) Conference

proposed four levels of defibrillation deployment resulting in a response taxonomy. To

align with the content of this dissertation, CPR and defibrillation are considered together

as part of the larger BLS function. One further addition, paramedics/EMTs are included

in the first level of response (they were not so designated by the conference). The merits

of each level are discussed in terms of quality and timeliness of response.

L1: Traditional first responder defibrillation: [paramedics, EMTs], police, or firefighters.

Level one represents the preeminent responders and can be expected to provide

the highest quality resuscitation performance. The problem is ensuring they arrive in

time to make a difference. Response times severely jeopardize survivability. Valenzuela

et al. (2005) report professional response times of 6 minutes 27 seconds from 911 call to

arrival at patient side with an additional 54 seconds to defibrillation. EMS services in

Dade County required almost eight minutes (7:56), likely a contributing factor in their

declining survival rate. Attempting to reduce the response tine , the county implemented

a program in which AED-equipped police units were dispatched simultaneously with

EMS. Response time improved to 4:53 (Myerburg, Fenster, Velez, Rosenberg, Lai,

SKurlansky, Newton, Knox, & Castellanos, 2002). In this study, the time of the first

vehicle to arrive (either police or EMS) provided the arrival data point. It would be

difficult to maintain such a redundancy for long periods to continue the reduced response

time, however.

L2: Nontraditional first responder defibrillation: life guards, security, flight attendants.

Members of these groups have generally achieved superior response times and

survival outcomes throughout the literature. Note the natural boundaries of the domains

and the ubiquitous authorities with mandated training. These contexts demonstrate the

potential for great success benefiting from the presence of capable responders already at

the scene of the SCA.

L3: Citizen CPR defibrillation: citizens with AED training

Programs to train citizens to respond to SCA events have been shown to increase

the number of victims that receive CPR, reduce the time to defibrillation and improve

survival (Hazinski, Idris, Kerber, Epstein, Atkins, Tang, & Lurie, 2005). A study in Italy

demonstrated an increase in survival rate from 3.3% (EMS) to 10.5% with trained (4 hour

course) volunteers (Capucci, Aschieri, Peipoli, Brady, Iconomu, & Arvedi, 2002).

Though this seems like a promising solution, it relies on citizens volunteering for

training. Willing volunteers do not make up a significant portion of the population.

Culley, Rea, Murray, Welles, Fahrenbruch, Olsufka, Eisenberg, & Copass, (2004)

reported that over a four year period, only 4,000 of the 1,700,000 people of King County,

WA received training (~0.2%).

L4: Minimally trained witness defibrillation. Individuals that witness SCA and have AED available but no training.

Perhaps this level was misnamed. It seems untrained witness would be more

accurate as the section fails to describe training of any kind. In these situations, the

response is rapid (associated with high quality outcomes), but its efficacy dubious. Here

is an opportunity for JITS! With an equipped and willing (though previously untrained)

responder in place, JITS can guide the operator through the procedure. The benefit of

device deployment no longer is impeded by the miniscule number of classroom-trained

volunteers. The instruction and course of action are tailored to the context. Thus, the

response approaches the level of L2, minimizing response time and elevating

performance.

��������������������� ��� ��On November 28, 2005, the International Liaison Committee on Resuscitation

(ILCOR) released 2005 International Consensus on Cardiopulmonary Resuscitation

(CPR) and Emergency Cardiovascular Care (ECC) Science With Treatment

Recommendations via the American Heart Association’s (AHA) journal Circulation.

With an emphasis on evidence-based medicine, they reviewed an exhaustive array of

studies employing human and animal models. The goal was to marshal the clinical

evidence in a fashion that could enhance the protocol and improve survival. Two points

from the introduction of this erudite, comprehensive work are pivotal in relation to the

research here.

• “The most important determinant of survival from sudden cardiac arrest is the presence of a trained rescuer who is ready, willing, able, and equipped to act”. (ILCOR, 2005. p. III-3 )

• “[O]ur greatest challenge remains the education of the lay rescuer”. (ILCOR, 2005. p. III-3 )

The report contains further implications for JITS. “[W]e must increase the

effectiveness and efficiency of instruction, improve skills retention, and reduce barriers to

action for both basic and advanced life support providers”, (ILCOR, 2005, p. PIII).

Many instances of SCA occur in public spaces (Becker, Eisenberg, Fahrenbruch

& Cobb, 1998), but with no trained responders present. Thus, these events occur with

people in the vicinity that could respond quickly, however they are ill prepared to act.

The contrapositive holds. Those capable of delivering appropriate actions require time to

arrive at the scene subjecting the victim to the descending probabilities of survival.

As a result, most SCA victims die outside the hospital without receiving

treatment (relating to Zipes & Wellens (1998) conjecture that 75% of victims die at

home). JITS systems in the home could drastically reduce the number of victims that left

untreated. A spouse, a child, a neighbor could be empowered to save a life with a JITS

device by providing life sustaining intervention while professional help is enroute.

In an effort to augment layperson resuscitation, ILCOR (2005) adjusted the

guidelines to be more effective and easier to learn. The guidelines increase the number of

compressions per minute to 30 from 15 and apply it to all victim types (except infants,

this reduces the need to memorize multiple ratios for different victim types). This change

is a small step in the right direction. While it is unclear that the number thirty is any

easier to retain than the number fifteen, the increased number of chest compressions

should increase circulatory function and maintain higher thoracic pressures over the

former protocol. But how much of a gain can be expected from these minor adjustments?

The conclusions of the ILCOR team clearly emphasize the need for immediate

response and improved technique. JITS designed systems provide a means to accomplish

both. JITS implementation could lead to dramatically reduced response times with

widespread deployment of a device that can instantly train the witness of a cardiac event.

Providing adequate ventilations and compressions in a timely fashion contributes to

healthier outcomes and improved survival rates. Sufficient deployment of support tools

could engender immediate, effective response through the ability to “instantly train”

bystanders. Thus, the researchers embarked on developing a CPR/AED system to test the

JITS framework. The following pages demonstrate the feasibility of garnering effective

treatment from untrained responders and provide an explanation of how JITS facilitates

performance.

BACKGROUND The advancement of technology in systems has elevated the demands placed on

the human operator. When system demands exceed operator resources, a problematic

mismatch emerges. The rapid development of new technologies and their deployment

across vast applications and domains makes keeping pace difficult. This discord is

exacerbated when a novice operator is called upon to perform the task.

The JITS framework strives to elevate user performance. Performance

enhancements are possible because JITS provides the user a workable plan, directive

cues, and corrective feedback. The system delivers the plan in accord with user progress.

The cues initiate user actions to begin each subtask. Those actions are closely

monitored, along with system state, to generate customized feedback and evaluate the

plan. All of this occurs while the user is engaged in the task.

To create such a system demands numerous coordinated efforts. The researchers

must establish a theoretical base capable of generating predictions, and robust enough to

account for a considerable range of human-system interactions. The technological

aspects must be developed, implemented and integrated with a systems engineering

perspective. Lastly, an environment must be fabricated in which to test theoretical and

applied aspects, including methodological constructs, data collection, and recruiting and

training participants.

Leveraging knowledge and experience from other scholarly domains is a prudent

start. Intelligent tutoring provided the interaction paradigm for JITS and has a rich

history of success. The research in expertise, feedback, and visual displays provided vital

details in how to deliver information to the user. The formidable task analysis literature

enabled the identification and assessment of task requirements and resources. Finally, the

Contextual Control Model (COCOM) offered a predictive model of human performance

within a JITS system as well as parameters for measuring outcomes. Intelligent Tutoring

Systems (ITS).

������������������������� �������������������������������� �������������������������������� �������������������������������� �����������The development of Intelligent Tutoring Systems (ITS) can be traced back several

decades. Early systems can be thought of as computerized flash cards with an ability to

score student responses. These programs generated problem sets (often in arithmetic or

vocabulary), recorded students’ answers and evaluated performance (Uhr, 1969).

The systems soon became more sophisticated. One of the first notable intelligent

tutoring systems was SCHOLAR (Carbonell, 1970). SCHOLAR tutored students in

South American geography. This was a landmark development in that it was the first

tutoring system that moved beyond a scripted hierarchy of knowledge and presentation.

Carbonell (1970) not only wanted to create a system that was able to handle all possible

answers a student might submit (a challenge to earlier systems), but also empower the

system to appropriately respond to questions asked by the student.

Another important step in the development of ITS was embodied in the work of

Collins (1977). Earlier that decade Craik & Lockhart (1972) demonstrated that deeper,

more elaborate processing improved learning and retention. In conjunction with this

work, Collins (1977) developed a Socratic method of exploration for his WHY system.

A Socratic system is structured such that a student works to “discover” knowledge. This

method empowers the student to build principles from experience and individual cases,

through self-directed exploration of the knowledge base.

������������������� ���������Many ITS projects share a basic architecture consisting of the same elementary

components. Perhaps the most basic as described by Burns and Capps (1988) are the

“Expert Model”, the “Student Model” and the “Tutor”. These modules were also

essential to the development of the JITS framework.

Expert Model The expert model is designed to encapsulate all available knowledge for a

particular domain. Domains in ITS have included arithmetic – BUGGY (Brown &

Burton, 1978), programming language - LISPIT (Anderson, 1988), and electronic

troubleshooting – SOPHIE (Brown, Burton, and deKleer, 1982). Amassing the expert

model requires the daunting task of harvesting and representing all collectible knowledge

for the particular domain. Anderson (1988) warns, “. . . a great deal of effort needs to be

expended to discover and codify the domain knowledge” (p.22). This information is

usually assembled by teams of experts in specific fields. The expert model commonly

serves as the standard to which the student’s actions and knowledge is compared.

Student Model The student (or user) model contains the system’s representation of the user’s

abilities and knowledge of the subject domain. A single student model is not sufficient

because user proficiency changes over time. Not only is there an overall trend of

learning and improved knowledge over the as the student progresses toward expert, but

there are perturbations of the user’s proficiency across and even within a training session.

A critical function of ITS is to properly diagnose the user and identify apposite

assistance. If students perform below their normal baselines, the system must respond

according. For the system to accurately gauge a student, it must go beyond a simple

comparison to the expert model. One shortcoming of current diagnostic functions is they

only evaluate response accuracy in terms of the knowledge base (Linn, 1990; Marshall,

1990). This often accomplishes little more than the ability to categorize a student. It

offers nothing to aid learning and performance (Linn, 1990).

Tutor Model Without a tutoring model (e.g., instructional module, or pedagogical agent) an

ITS would merely administer and score tests with no ability to impart knowledge or

understanding to the user. The instructional module interacts with the expert model and

the student model to formulate relevant information to the user.

The tutor model evaluates the student model in terms of the expert model and

decides what information to present, how to display the information, and when to do so.

The tutor model is the most visible module for the user. Through the interface, the

pedagogical agent serves as the teacher, consultant, assistant, or coach.

Customizing Information for the User The quality, quantity and timing of assistance determine the efficacy of the

system. Unnecessary instruction may be perceived as a nuisance and potentially hinder

performance. Users are unlikely to assimilate information that exceeds their

comprehension, arrives after significant delay, or conflicts with their mental models.

Accepting that not all users learn at the same pace nor benefit from the same level

of instruction, customization is crucial. Brusilovsky and Hoah-Der (2002) developed an

exemplar system basing information delivery on user needs. Their curriculum tutored

students in a web-based C programming application (WADEIn).

WADEIn demonstrated several advances for assessing user proficiency as well

providing user assistance. To help build the user model, WADEIn tracked three

parameters: the number of times the user had seen a particular visualization, the number

of times a user had performed an operation, and whether the student identified the order

of operation correctly. This data helped construct a rating for learner proficiency which

they assigned 1.0 – 5.0. This grade provided a basis for selecting specific assistance

protocols. For example, if a user was rated 1.0, the assistance provided all sub-steps of

the particular task (per the expert model) and showed the tutorial animation in slow

motion. A user rated at 5.0 received no sub-steps and no animation. Such acumen in

diagnosing user proficiency and matching information needs can benefit JITS

development by optimizing the human-system interaction.

%�����������&�������������������%�����������&�������������������%�����������&�������������������%�����������&�����������������������

A superficial comparison of intelligent tutoring and JITS may lead to the

fallacious conclusion of equivalence. Certainly, there are similarities – both are primarily

aimed at novices, are computer based, and leverage best practices of expert systems and

user-interface principles.

A major disparity arises however, simply examining the goals of each system.

The objective of ITS is to provide a student with knowledge and foster long-term

retention. In contrast, JITS systems are designed to support an operator complete a

specific task at hand. Thus, very different constraints and contexts apply to each system.

These differences are elucidated in terms of their objectives, time available, processing

strategies, and system input. A brief summary of these discords is displayed in Table 1.

�� ����� �

The disparate objectives of each system are yoked to their temporal constraints.

The extended time horizon of ITS enables the goals of learning, long-term retention and

fosters accuracy in the speed-accuracy trade-off. In contrast, JITS is not focused on

learning; retention is inconsequential. The objective is to perform the task to a

satisfactory degree in the limited time available. Due to the temporal constraints, it is

more important to sufficiently complete all requisite steps in the given time than to strive

for flawless performance on some tasks while failing to complete others.

Paradigm Objectives Time Available Processing Strategies System Input

Intelligent Tutoring Systems

• Long term retention • Learning • Accuracy over speed

• Long time horizon • Self paced • Low time pressure

• Deep/elaborative • Adaptive, transferable • Declarative

• Passive collection • Requires user input to computer

Just-in- Time Support

• Short term performance • Complete task • Speed over accuracy

• Short time horizon • Externally paced • Significant time pressure

• Shallow / perceptual • Mimicking • Procedural

• Active collection • Monitors user actions

����������������

There is generally an absence of time pressure while students engage in ITS

learning. The user determines the start time, duration, and sets the pace of the

interaction. In contrast, a JITS task will be subject to considerable time pressure due to

the criticality of the task (a non-critical task could tolerate awaiting the arrival of an

expert), making the task externally-paced. The JITS operator has a limited time envelope

in which to work compared to a self-paced, multi-session training exercise (the putative

ITS mode). Success or failure will be realized shortly in JITS facilitated tasks.

Table 1. Fundamental Differences between Just-in-Time Support (JITS) and Intelligent Tutoring Systems (ITS)

!��� ��� ������ � A significant discrepancy is also evident in the level of mental processing

indicative of each system. As mentioned above, learning has been shown to benefit from

a deeper level of processing (Craik & Lockhart, 1972). Intelligent tutoring systems

utilize declarative knowledge permitting extensive mental associations and abstractions.

This more elaborative processing strategy serves not only long-term retention, but also

pliant, adaptive application. Just-in-time support systems target a more perceptual level.

As a consequence, mimicking-type strategies, though inferior to elaborative processing

for memory, are efficient for accomplishing procedural tasks without practice.

�� ����������The final pivotal difference discussed concerns the inputs to each system.

Traditional ITS requires a direct manipulation by the user (type, point and click, etc.).

The system itself is passive and requires direct input by the user. An advancement

distinguishing JITS is its ability to actively collect data from the environment. With

various sensors the JITS system identifies changes of state and immediately updates its

models. This is transparent to the operator (sensors are integrated with the tools used),

and drives updated cues and feedback. Feedback and plan adaptability rely upon the

external data the system is able to collect and integrate. The operator is permitted to act in

a goal directed manner on the world (essential for performing a task), and relieved of the

additional burden of updating the system.

'�����������������'�����������������'�����������������'����������������������"�������#������������$����������$��

Hollnagel (1993) describes a continuum of human control that can serve JITS

development in both design and evaluation. The COCOM provides designers a tool to

identify parameters and determine control characteristics. This not only enables

prediction of control conditions, but establishes means to manipulate control states.

Additionally, the COCOM provides an effective assessment tool to evaluate performance

hypotheses.

Before describing the nominal control modes, it may be helpful to explain the

parameters that characterize a given state of control. The control modes will vary on the

following dimensions. Determination of outcome: the operator’s ability to detect and

interpret a change in system state. Subjectively available time: the time pressure

perceived by the operator. Number of simultaneous goals: the number of objectives the

user can maintain concurrently and their relevance to the overall goal. Availability of

plans: the user has access to heuristics, plans, procedures or something rule-like to guide

actions. Event horizon: is composed of the “history size”, which refers to the amount of

previous information utilized in a given decision, plus the “prediction length”, which is

an extrapolation of the future state of the system. Mode of execution: there are two basic

modes: a “subsumed” (ballistic or feedforward) mode where actions are executed

automatically, require assumptions and predictions, and a “feedback” mode in which

state data guide future actions.

Control Modes The control continuum is anchored by an absence of control at one end, and

highlights several milestones as control progresses to a very high-level, effective process.

Hollnagel has identified four characteristic regions of control to serve as watermarks on

the continuum. From least control to greatest control they are: scrambled, opportunistic,

tactical, and strategic.

Scrambled control resides on the low end of the continuum, and in its most

primeval, is akin to no control at all. The next action of the user is random and

unpredictable. Often, a blind trial and error approach is adopted and the determination of

outcome is extremely limited. User actions are often incongruous with the situation. A

user operating in a scrambled control mode has no heuristics, rules or procedures

applicable to the situation. The user is often under a great deal of stress due to workload,

time pressure, and a futile understanding of the system and/or the current environmental

conditions. The user is commonly in a panic state when operating in a scrambled control

mode.

Opportunistic control is the next region on the continuum featuring some ability

to guide the system toward the goal state. As in the scrambled control mode, the operator

perceives significant time pressure. There is still scant planning of the next action, but

actions are no longer characterized by randomness. Despite the lack of planning,

cognition is on the rise. This fosters the hallmark of opportunistic control: an ability to

recognize and act on salient cues in the environment. The user has moved from a

subsumed mode of execution to a feedback mode. Feedback is accessible and

interpretable. Additionally, if the environment is somewhat familiar, the user may be

influenced by frequently used schema.

Tactical control finally offers relief from the perceived time pressures and permits

short range planning because the event horizon is expanded both backward (previous

states) and forward (predicted states) in time. Performance is largely based on rules or

procedures available to the user. The operator can begin to anticipate needs of the near

future, but these predictions are constrained by the present context. The meanings of the

outcomes are also more completely understood (in terms of the current context). Two or

three goals may be active at once, and likely there is a plan, rule, or procedure to support

each goal. Feedback is again a valued input, but for a different reason. Feedback in the

tactical control mode is utilized for comparison with higher level goals.

Strategic control is a state of high stability and planning well beyond the

immediate context. Performance is usually effective and robust. The event horizon is

further extended (in both directions). The operator has the opportunity to contemplate the

highest level goals. Interestingly, both modes of execution are exhibited (due to the

long-range planning, there can be a significant lag in feedback and the need to take some

actions in the subsumed mode). Finally, it is also suggested that to be in a strategic mode

requires operator motivation. The user must embrace the additional cognitive load

required (for extended planning, reasoning, and observation) to attain a strategic mode of

control. As a consequence of the arduous workload, strategic control often cannot be

maintained for long periods.

Mode Transitions The movement between control modes is also of interest. Transition between

states can be discussed in terms of changes in the parameters outlined above. Since those

characteristics describe a state of control, it is appropriate to speak about an operator

moving along the control continuum as result of a change in one or more of those

parameters.

Parameters Scrambled Opportunistic Tactical Strategic Determination of Outcomes Obscured Limited Context

dependent Elaborate + prediction

Subjective Time Pressure Severe Significant –

Severe Light- Moderate None

Simultaneous Goals 0-1 1 2-3 2-many

Availability of Plans

None employed Minimal

Most goals supported by plans

Plans and contingencies available for all

Event horizon

Present only

Some history, little planning

Planning (based on current), use of previous data

Extensive planning & use of historic data

Mode of Execution Subsumed Feedback Feedback Feedback +

subsumed Action Selection

Random

Cue driven Plan driven Prediction

driven

The most pedestrian transition to envision is a simple step to an adjacent region.

This shift may be either an increase or reduction of control effectiveness. Examples

include tactical to strategic and opportunistic to scrambled. However, larger jumps are

also possible. Imagine an operator in a tactical control mode that suddenly encounters a

novel crisis. The subjectively available time may have all but vanished. The user may

have no heuristics to employ. The determination of outcomes proves elusive. All

planning has ceased and the only goal may be to figure out what just occurred. Clearly, a

scrambled condition has resulted. Similarly, it may be possible to jump straight from a

scrambled mode to a tactical mode. Though it may be considered extremely fortuitous, a

random action could bring the system to a stable, recognizable state in which the operator

has sufficient time, knowledge, and experience to assert tactical control. Hollnagel

(1993) does insist there is a constraint in moving to and from strategic control. Strategic

Table 2. Summary of COCOM parameters and control states (adapted from Hollnagel, 1993)

control can only be reached from the tactical region. Further, strategic control can only

be degraded to tactical (from which of course a fall to scrambled control is possible).

Knowing the characteristics of each control mode, it is possible to identify the

region in which an operator is working. Knowledge of the task, tools, and context

permits a designer to make mode control predictions. These parameters provide a

workspace that can be manipulated in an effort to improve an operator’s control. For

instance, a plan could be made accessible where a user had none. The absence of salient

features can be rectified to guide action. Feedback could be presented to promote

interpretation. These are just some of the steps that may be taken to improve

performance according to the COCOM. The priority of the parameters and combination

effectiveness will likely vary by task, user, and context, and are probably best derived

empirically.

Lastly, the COCOM, as it is a model of human performance, is a valuable tool for

not only design, but also assessment of a JITS system. It provides valuable

characteristics to describe the task and desired performance. These characteristics enable

researchers to predict performance in a given situation. Finally, the parameters can also

serve as dependent measures for comparing design decisions, methods and even full-scale

systems.

COCOM Implications for JITS design

• Provide a plan of action, adaptable to dynamic conditions and needs

• Provide salient, action-directing cues

• Provided interpretable feedback to correct actions and update system state

• Work within time constraints

• Control mode is a function of parameters above; designers have power to manipulate

�#��� ��In order to assuage potential ambiguity, a brief definition of expertise is offered.

An expert is a person that has gained extensive knowledge and prodigious skill in a

particular domain. The knowledge is also highly organized and accessible by the expert.

Anderson (1983) insists that both theoretical and experiential knowledge are required to

become an expert. Ample time is another correlate as a common finding suggests it

takes ten years to become in an expert in many domains (Klein, 1998). For a treatise on

expertise, see Sternberg (1997).

Expertise is a major component and is ubiquitous throughout JITS. First, task

knowledge must be collected in order to prescribe a successful method for task

completion (the plan). Then, domain expertise is required to create effective, adaptive

protocols capable of supporting numerous scenarios. To produce a robust system, the

experts must anticipate failures and complications caused by the system, the user, or the

environment. Ultimately, this vast, high-level knowledge must be translated to a form

suitable for the non-experts supported by the system. Thus, it is critical to examine the

documented differences between experts and novices.

The literature has largely focused on what expertise is. By default a person

lacking these superior skills and knowledge is considered a non-expert. This paper will

not demarcate the categories and qualifications of non-experts in great length utilize a

common ITS-employed continuum: novice, beginner, intermediate, expert (Virvou, &

Moundrido, 2001). The current state of JITS is aimed at the less skilled of this set, but it

is believed that JITS can be customized to service more advanced operators as well.

Characteristics of Experts A substantial amount of research is available characterizing experts and expert

performance (Chi, Glaser, & Farr, 1988,). Glaser and Chi (1988) have established seven

general attributes representative of experts presented in table 3.

Expert-novice differences With the focus of the literature directed at experts, differentiation between experts

and non-experts is often left to inference. Fortunately, these conclusions are warranted.

Johnson (1988), in multiple domains, has observed that novices do not have extensive

domain knowledge, and cannot encode information well (e.g. they have representation

and pattern matching difficulties) nor process new information quickly. Miller & Perlis

(1997) also discussed novices’ inferior knowledge base and structure, adherence to

superficial cues, and utilization of small, fragmented information units. Not only did

novices utilize more superficial knowledge, but the knowledge base lacked cross

referencing and the organization experts are able to impose.

Therefore, in developing a system for non-experts, the inverse of each axiom

above can be used to guide design. Different aspects of the JITS framework were crafted

1. Experts excel in their own domain.

2. Experts perceive large meaningful patterns.

3. Experts solve problems quickly with little error.

4. Experts have superior short- & long-term memory in their domains.

5. Experts represent problems more abstractly.

6. Experts are able to spend great deal of time analyzing a problem qualitatively.

7. Experts have strong self-monitoring skills.

Table 3. Expert characteristics (Glaser & Chi, 1988)

to specifically address these salient needs. Below are the design (D) recommendations

pertinent to each expert characteristic described above.

Novices:

are outside their domain (or are experiencing a novel/low frequency event) D1. Novices do not excel in the given domain, they need help. That’s the

motivation for JITS!

cannot decipher meaning in complex patterns D2. Information must be parsed into simpler, more digestible chunks. Critical

parameters and combinations must be made salient. The system can help the user construct patterns in small increments.

are slower and more prone to error

D3. Along with reduced data size, the pace at which information is delivered should be slowed for novices. The sequence may also need altering. Repetitions may be necessary. The pedagogical model should monitor and control the pace and sequence pursuant to operator needs.

have no strategies/knowledge to enhance STM & LTM

D4. The system should relieve burdens to STM & LTM by holding the information and making it visible, accessible, and congruent with the current subtask.

don’t understand abstract concepts of the domain

D5. Cues and feedback must utilize concrete representations. Designers can’t assume knowledge on the part of the user. Metaphors, higher level concepts, and domain knowledge are likely to be lost on a novice.

won’t analyze the problem

D6. Novices won’t have time or ability to analyze the situation on a deeper level. The context-aware system will be responsible for identifying the proper course of action given the context.

will not be able to self monitor reliably if at all.

D7. Novices will likely be far too stressed to have any resources available for this meta-task. Smart algorithms and sensors take over this job. In conjunction with the situation assessment (D6), the system monitors operator performance in order to optimize cues and feedback to assist the user.

The emphasis in employing these principles is that it is imperative to provide

information congruent with the operator’s knowledge. It is crucial to design systems

suited for the user’s’ level of expertise and these guidelines are intended to illuminate the

specific needs of novice users. In a human-to-human tutoring example, findings by

Hinds, Patterson and Pfeffer (2001) demonstrated that experts used more abstract (and

fewer concrete) statements as they tutored novices. In contrast, novices tutored by

beginners performed better and reported fewer problems with the instructions. This was

largely the result of beginner tutors using more concrete terms, and their ability to

comprehend the novices’ states of learning.

Developers of JITS systems should not be obfuscated with the seduction of

attempting to transform novice users into experts. Tasks may consist of only a single

trial, not nearly enough time to establish expertise. By abandoning the effort to create an

expert and embracing the task of mimicking one, several advantages arise. A shallow

level of processing can be targeted. While inferior for retention, shallow processing

reduces the demand on the novice’s limited cognitive resources and leverages more

primitive processes such as perception-action mechanisms and preattentive processing.

As Klein (1998) elegantly suggests, experts see the invisible. The crux of JITS

development is to unveil and communicate that information for novice use.

Expertise Implications for JITS design

• Break information down into comprehensible chunks

• Use concrete (as opposed to abstract) representations

• Ensure the pace of information delivery is apt for user population

• Minimize cognitive burdens (memory, search, attention capture) as much as possible.

• System should assume monitoring role

• Keep information visible

%������&�Feedback, loosely defined, may be considered any information provided back to

the operator concerning performance or outcome. This discussion will exclude intrinsic

feedback (that which comes from within the operator), but focus on the information

provided by the system as a result of operator actions. Many terms such as knowledge of

results (KR), knowledge of performance (KP), augmented feedback, and intrinsic

feedback have been used throughout the literature to describe different effects of

feedback (Proctor & Dutta, 1995). In the following section, specific terms have been

preserved where research findings are reported. Their nuances are less important for the

larger discussion as all contribute to the design of effective feedback for JITS.

In order to provide closed-loop control, feedback must be communicated to the

operator. As discussed in the COCOM section above, feedback enhances operator

control. Of course, the feedback presented must be interpretable and actionable for the

operator. Generating suitable feedback for non-experts often requires a skillful balancing

of contradicting design issues.

Benefits Reviews and meta-analyses have demonstrated that accurate feedback generally

improved overall performance in a variety of tasks (Azevedo & Bernard, 1995, Kluger &

DeNisi, 1996, Salmoni, Schmidt, & Walker,1984). Reliable Feedback has been shown

to enhance learning and performance compared to no feedback or misleading feedback

conditions.

Concomitantly, erroneous feedback has been shown to produce deleterious effects

(Young & Lee, 2001). These findings demonstrate stout support for the use of reliable

feedback in developing JITS systems. In accordance with the JITS tasks, Azevedo

(1995) has emphasized the importance of feedback in a computer supported environment.

In support of immediate task facilitation, Goodman (1998) concluded that

frequent and immediate feedback to participants improved their performance during

practice but did not report the same benefits for retention. Here, practice refers to the

participants’ first encounter(s) with the task. Similarly, Young & Lee (2001) found that

more feedback in the “acquisition phase” facilitated task performance (but not retention).

In JITS supported tasks, “practice” occurs simultaneously with the “test”. Therefore,

frequent and immediate feedback will likely generate desired effects for JITS systems.

An additional benefit, feedback has been found to increase participant motivation.

Participants often work harder, show more interest in the task, and persist longer than

those not receiving feedback. Interestingly, Salmoni, Schmidt, & Walker (1984) found

these effects persisted for some time even after removing KR (e.g., feedback).

Pitfalls Not all research has witnessed positive returns from feedback. Some studies

resulted in null or negative effects, and other studies showed greater learning with

reduced feedback. Negative effects can often be traced to poorly designed feedback as it

relates to the task or the operator. For example, feedback may not be suited for the

operator’s level of expertise. “Cognitive feedback” attempts to provide individuals with

considerable insight to allow them to learn relationships between the cues they utilized

and the judgments that followed. This however, required significant domain knowledge

and was not suitable for novices (Balzer, Hammer, Sumner, Birchenough, Parham, &

Raymark, 1994).

In complex tasks, multiple sources of feedback may provide contradictory

information and degrade performance (Goodman, 1998). Thus the reliability of the

system must be sufficient to supersede conflicting data. Efforts should be made to reduce

the overall data stream in an attempt to provide only what is necessary to facilitate the

task.

Other problems relate to the timing of the feedback. Rapid feedback may elude

operator detection. Similarly, the operator could be engaged (mentally or physically)

with a task and unprepared to receive new information. Feedback presented at such times

will not be processed or incorporated into the next actions. Further, if concentrating on

the action itself, the user could view the feedback as interruptive and distracting.

However, feedback lag may also present a problem as Salmoni, Schmidt, and Walker

(1984) showed that a long delay in feedback (as opposed to KR after each trial) degraded

practice performance (but facilitated long-term retention).

Other researcher (Young & Lee, 2001) also showed benefits of withholding

feedback. They used a “bandwidth” feedback condition in which feedback was only

communicated if performance deviated from a specified tolerance (10% error). They

found this group to outperform the group receiving feedback after every trial. In the

same study, they found that over specified feedback (e.g., unnecessary precision), was

usually ignored by the participants.

Feedback implications for JITS design

• Customize feedback pursuant to operator knowledge • Feedback provided must be reliable

� Conflicting information can confuse the user

� Excess precision (based on user knowledge) will likely be ignored

• Feedback timing is critical

� User must be able to relate feedback to relevant activity

� User must be ready to attend to feedback information

• Feedback based on threshold may be more suitable than after every trial

• Feedback may serve as motivation in completing the task

' ����( ���� ��As most of the tasks supported by JITS will entail significant complexity, the

visual channel could be overloaded with information. Technology-intense domains

typically require visual displays to support detection and interpretation tasks, particularly

when governed by capricious constraints (Sanderson, Haskell & Flach, 1992). This holds

for the CPR/defibrillation task investigated here.

Park and Hopkins (1992) outlined six conditions in which dynamic visual

displays (DVDs) are effective. The majority of those stipulations are highly cogent with

JITS system development to support CPR. The use of DVDs is effective when:

- demonstrating sequential actions in a procedural task : The sequencing of the

procedural steps is achieved through context-dependent algorithms based on the AHA

protocol.

- obtaining attention focused on specific tasks or presentation displays: The system

choregraphs the responder’s attentional demands, switching between information, tools,

and the victim as required.

- illustrating a task which is difficult to describe verbally: It would be quite difficult

to verbally impart hand configuration and placement in performing chest compressions.

- explicitly representing invisible system functions or behaviors: A responder would

have no way of knowing their inspired volume. The system tells them to give more

breaths if insufficient.

Animation Animated graphics (e.g., dynamic visual displays) are a great tool for conveying

complex, temporally constrained, spatial information. It is envisioned many JITS tasks

will require space-dependent manipulations of artifacts in the workspace. For users

unfamiliar with compulsory components, spatial relationships, or procedures, animation

provides strong cues for comprehension. Reiber (1990) asserted animation provides three

vital attributes for performance: visualization, motion, and trajectory. ChanLin (2000)

demonstrated that spatial and procedural elements of animation play an important role in

interpreting information

Before trumpeting the success of animation, it must be noted that the animation

literature is tainted with mixed results ( Kehoe, Stasko, & Taylor, 2001, Park & Hopkins,

1992, Rieber, 1990, and Tversky & Morrison, 2002 for reviews). One problem with the

research is the difficulty constructing fair comparisons (Tverksky & Morrison, 2002).

They further observed that, “animations are often too complex or too fast to be accurately

perceived” (p. 247). They suggested much of the animated designs violated the

“apprehension principle” requiring graphics to be conceived appropriately and perceived

accurately.

An example of this excessive complexity hypothesis was captured in an

experiment by ChanLin (2001). A procedural learning task in physics (resultant force

vector) was taught to high school students using text, still graphics and animation. The

results indicated that still graphics were significantly better for the novice group (over

text or animation), while animation garnered superior performance from experienced

students. They concluded the animation was too complex to support novice

comprehension and required substantial prior knowledge to be interpreted correctly.

A similar phenomenon was observed when graphics-to-text comparisons were

being made in the 1970s. Early studies failed to uncover the benefits of graphics over

text. Willows (1978) argued that performance suffered when graphics were superfluous,

complex, or incongruent with the text. Eventually, greater effort was expended on the

design and graphics began to improve performance. By mid-decade, Booher (1975), had

sufficient data sighting the superior performance of well constructed graphics-plus-text

presentations over either alone. Today, it is accepted that well contrived graphics are

highly effective at portraying visuo-spatial information (Tversky, 1995).

A number of experiments describe the benefits of animation for elevating first-

time performance, vital for the single-trial nature of JITS tasks. One such experiment

comparing textual versus animated instructions provides a convincing illustration.

Palmiter and Elkerton (1991) examined user ability to learn a procedural task on a

desktop computer. The procedure involved a training period, and two tests of retention:

The retention tests favored the group receiving textual instructions. It was concluded that

deeper encoding required for reading and processing textual propositions benefited users

on retention tests manifest in the text group’s dominance in the delayed post-tests.

However, in the training trials, the animation group demonstrated a superior

ability to achieve early success. The animation group performed their training session

tasks in approximately half the time the text-based group required. Intriguingly, this was

not at the expense of accuracy. In training, the animation group surpassed the text group

by performing over 90% of their trials correctly, while the text-based group failed to

achieve 80%.

A conclusion drawn by the researchers (and supported by comments from

participants) was the animation group adopted a mimicking strategy. They simply

followed the animated procedure, mimicking tasks and actions with little mental

processing.

In JITS systems, the training and test occur simultaneously. The CPR task

required immediate action by unpracticed individuals in a critical moment. There is no

need for retention or even comprehension, simply the aptitude to follow the procedure.

Animation can be beneficial in conveying task information for unpracticed, time-

pressured tasks.

Visualization (DVDs) implications for JITS design

• Important for conveying dynamic visuo-spatial information (difficult to verbalize)

• Appropriate for sequencing and procedural information

• Effectively portray motion and trajectory

• Animations should be simple and relevant

• Pacing of dynamic visuals must be carefully considered

�� &������ ������A critical step in developing JITS systems is a thorough examination and

description of the tasks, procedures, and goals. Task analysis (TA) yields a deeper

understanding of the fundamental elements of the task and exposes the nature and

organization of the sub-tasks. Through task decomposition, requirements assessment,

and error prediction, task analysis elucidates critical performance parameters engendering

efficient, fault-tolerant systems.

In general, a “Task Analysis”, serves to marshal the task-relevant design

implications. A surfeit of specific task analysis techniques exists including: cognitive

task analysis (CTA), hierarchical task analysis (HTA), critical path analysis (CPA),

timeline analysis, failure modes and effects analysis (FMEA), and goals-means task

analysis (GMTA). This section does not detail the various methods available or delve

into the current TA issues (see Annett & Stanton (2000)), but instead serves to call

attention to their importance and breadth.

Selection of specific techniques should be driven pursuant to the focus of the

analysis and researchers should leverage methods for their strengths and suitability in the

context of the project. The research plan may require a focus on: physical actions,

cognitive requirements, performance evaluation, temporal/sequencing issues, functional

descriptions, or goal accomplishment. Kirwan and Ainsworth (1992) provide a

comprehensive summary of the strengths of more than two dozen task analysis

techniques.

Task Decomposition Task decomposition is a widely employed technique often entangled with other

tools such as a Hierarchical Task Analysis (HTA) or Goals, Operators, Methods, and

Selection Rules (GOMS). Task decomposition requires the identification of the

constituent activities that, when performed together (and correctly), result in the

achievement of a higher goal. Effective task decomposition yields a clear description and

understanding of all pertinent sub-task activities. Through task dissection, the analyst can

gain the greatest insight to task requirements and constraints. Task parameters

commonly exposed by decomposition include: atomistic action descriptions, physical

and cognitive demands, failure points, temporal constraints, commencement and halting

cues, decision thresholds, resources and constraints, and performance criteria.

Hollnagel (1993) disambiguates a few terms to facilitate TA discussion. A goal is

a state that is reached when tasks have been successfully completed. A Task is a

collection of actions that are used to achieve a goal. Lower-level actions are known as

task steps, which begin to capture fundamental actions. A pre-condition is a requirement

that must be met in order to attempt a task or a task step. Not surprisingly, a goal can

have multiple tasks and sub-tasks with several pre-condition requirements. Further, the

absence of a pre-condition may result in the need to create a goal satisfying a pre-

condition which could then require tasks and sub-tasks to realize that pre-condition. This

can rapidly burgeon into an intricate, nested hierarchy. There are also cases in which

tasks are not dependent upon subordinate tasks (e.g., “pure” tasks), and are sequence

independent. That would represent a non-hierarchical task description.

What constitutes a pure task will largely depend on the level of analysis. This

granularity decision resides with the analyst. What precision must be reached by the

analysis? This can be a difficult question to answer and should be determined in the

context of the project. A guideline that may be helpful is to assess what can be reaped

from the requirements assessment and error prediction exercises. If the results of these

seem satisfactory, then the level is likely sufficient. If there are obvious voids, the level

of analysis may be too shallow. Conversely, is there anything to be gained by excessive

analysis (i.e. nerve innervations)? Again, these questions must be addressed in the

context of a specific project.

Error Analysis Though task decomposition can be an arduous process, the benefits are quite

valuable. The results can be described in terms of a “needs analysis”. These needs will

be crucial in the prediction and analysis of erroneous behavior. Error is a relative term,

and is generally determined in comparison to an expected “correct” performance (based

on goals and known successful activities relevant to the goal). Several of the task analysis

tools available are apposite for error and consequence prediction. A few examples are

barrier analysis, FMEA, fault tree analysis, GOMS, and Petri-nets (Dix, Finlay, Abowd,

& Beale, 1998). Again, the researchers should employ particular techniques that

complement underpinnings of the research.

A prudent methodology for detecting error is searching for inconsistencies

between the needs required to perform the task and the capacity to meet those needs. The

capacity is composed of operator capabilities/performance, resources available to the

operator, operator-system interaction, and system/environmental constraints.

Deficiencies in any of these areas can impede progress toward the goal state. Each of

these areas is subject to considerable complexity which is proportional to the level of

detail. Further, the integration of these components to aggregate a system-wide

assessment is nontrivial. However, this effort to predict and mitigate errors in the design

stage may help evade significant consequences. Of course, if a system already exists (or

an analogous system), much can be gleaned through observations.

Task Analysis implications for JITS design:

• Break down tasks into elementary actions that build larger tasks and actions

• Assess the cognitive and physical resources needed to complete each subtask

• Assess the means to meet the demands

� Determine resources supplied by the system

� Determine resources supplied by operator

• (requires realistic evaluation of user population)

������������������������������������������������������������������ �������������� ��� �������������� ��� �������������� ��� �������������� �����

The cardinal elements of a JITS system are derived from user needs with respect

to task completion. The novice lacks domain expertise and therefore cannot formulate

plans, recognize decision and action cues, and will have little success monitoring the

progress of the task. The framework of JITS addresses these problems by developing and

communicating plans, cues, and feedback to the operator. Figure 1 depicts a generic

schematic of a JITS system. The diagram includes the JITS system, the operator and the

task space. Notice that no effectors extend from the operator to the JITS device. Instead,

the device collects input from the state space via tools (which may be manipulated by the

user).

Just-in-Time Support System

COCOM

Error Analysis

PLAN

Task Workspace

Human Operator

Task Analysis

Expertise

ITS

Visualization

FB

receptors

effectors

sensors

Algorithms

receptors

tools/equip

Modules

+

Sequencer

CUES

Figure 1. Schematic of a Just-in-Time Support System.

!��� �The first requirement of a JITS system is to generate an effective plan for the

operator. Lacking a starting point, a user may fail to begin the task or may perform

actions that are fruitless or counterproductive. The task and circumstances may be

overwhelming, disabling cognition in general and specifically executive function.

Providing the user a plan and assertive direction will assuage substantial cognitive

burdens.

Plan development is much more than generating a single recipe for action.

Sensors continuously monitor the workspace. Algorithms constantly assess the

progression toward the goal state. Decision rules are employed by the system to alter the

task as necessary. These processes eclipse the capacity of a novice and must reside in the

support system. Non-experts possess scant ability to recognize the need for

contingencies, and have few resources to derive them. Thus the plan is not just a ballistic

delivery of a single formulation of actions, but an adaptable confluence of actionable data

that facilitates task completion in the given context.

The adaptability of the plan greatly depends on the effort expended in the task

analysis. Sufficient specificity of the TA should produce elementary action modules.

These modules are the building blocks of information delivery. Basic action scripts are

presented to create almost any task-relevant action sequence needed. The modularity of

the task actions fosters flexibility as individual actions can be called at anytime creating

extensive sequencing options.

A meticulous task analysis remunerates JITS developers in another way. The

effort reveals the information needs to support the task. This obviously has a significant

impact on the design and inclusion of various sensors and algorithms assigned the tasks

of collecting and interpreting data (which impacts the selection of plans, cues, and

feedback). It is not difficult to imagine a scenario in which a system is deployed (based

on a substandard TA), and found to have informational deficiencies when tested with

humans. Anticipating information needs early in the process prevents unwelcome

surprises and design modifications later. Plan progression and alterations, as well as the

operator’s ability to follow them, are reliant upon discovering and satisfying the

information needs.

The optimal plan, if followed appropriately, will likely be at least somewhat

accordant with an expert’s sequence of actions. However, the measure of success is not

how closely the operator tracked the expert model, but an assessment of the actual end

state comapred to the desired end state.

There may be numerous paths to meet task objectives. The development of plans

requires significant hypothesizing and supposition by the designers. Since it would be

impossible to imagine all possible human actions, performance criteria must be

established to provide tolerances for assessment. Identifying acceptable ranges can serve

as decision points in the algorithms. For example, how many times should the system

repeat a cue before trying a new presentation? When is it time to abandon a subtask and

attempt another action? Of course these depend on the task, but tolerances need to be

identified and programmed to permit such decisions.

��� �In order to initiate proper actions, the system must engage the user prior to

providing guidance. As the COCOM model indicates, operators in the opportunistic

mode can utilize cues to improve their performance. A primary objective of JITS is to

provide those cues.

Presenting information without the operator allocating appropriate resources will

accomplish little. In many subtasks, the first cue should be audible. Omnidirectional

aural information may be necessary as many targets vie for the operator’s visual

resources. An aural indication can redirect the operator’s attention to the display. A

synthetic voice or recurring chime could apprise the operator of new information.

For some task elements, the auditory channel will be sufficient. For example, a

metronome-like aural cue can guide pacing without taxing visual resources. However,

many tasks employing JITS will likely entail significant complexity thereby requiring

visual cues for adequate support. Providing a consistent visual mapping of the

components on the display to their real-world counterpart will enhance object

recognition. Artifact identification, relative positions and motions, and landmarks, can

all be conveyed through visual cues. Operators can leverage cues such as color, patterns,

shape, movement, and spatial relationships to identify and manipulate objects correctly.

Cue development relies on adequate task decomposition. A simplified subtask

can reduce clutter compared with more complex action sequences. This limits the

number of cues presented simultaneously, relieving the burdens of search and

discrimination (extra cognitive tasks which can lead to error).

%������&� Section 4.3 reviewed the academic findings of feedback pertinent to JITS. This

section describes feedback as a pillar of JITS (along with plans and cues) and discusses

the functionality of feedback as a tool.

Feedback in JITS applications is analogous to a servomechanism. Optimally,

small corrections through effectively communicated feedback will direct operator

performance toward the goal state. Slight, qualitative adjustments can usher the user

along the desired performance trajectory. Pragmatically, cases will exist that require

significant deviations to current actions. This could include presenting alternative

formats of the same information, selecting a different sequence of subtasks, or activating

a completely new plan.

Feedback is also capable of providing additional means of imparting information

if the first cues were insufficient. The system may simply repeat the cue, or if

information is available, a new cue may be delivered to address a specific shortcoming.

The system may substitute alternate cues hoping to find a better match for the operator.

Ultimately, if a user is unable to adequately act on a cue, the system may abandon that

subtask and devise a new set of actions. Many of the guidelines outlined for cues are also

applicable to feedback design. Expertise, visualization, and of course the feedback

literature, are all prudently employed in the development of feedback presentation.

Examples include the need for concrete representations, apposite timing and sequencing

of information, and suitable decomposition.

Two objectives can serve to guide feedback design. First, feedback is necessary

to convey information for corrective actions. Regardless of the degree of error, the

feedback module carries the responsibility of error reduction. Secondly, updates of task

and system status should be communicated to the operator. Novice users will be ill

equipped to evaluate and track task success; their errors need to be corrected, but they

also need to be told when they get things right. Feedback provides the user with

important information to incorporate in future actions as well as serves as motivation to

adhere to the task.

(���������(���������(���������(������������� The researchers endeavored to promote design methods enabling satisfactory

performance by novice operators in unfamiliar tasks. The fundamental provisions of an

action plan, cues, and feedback can propel a naïve operator from a state of bewilderment

to a mode of control enabling information processing and action. However, the

framework required validation.

Non-expert responders performing CPR and defibrillation provided a robust

context in which to thoroughly vet the assertions of JITS. First it enabled the researchers

to demonstrate the dismal performance of those without training or JITS, establishing a

baseline. Second, JITS induced effects were examined on trained and untrained

responders. Further, the context afforded the utilization of a respected protocol (AHA),

strict control of training, interaction with technology, and quantitative and COCOM

based performance measures, all critical parameters of JITS. Thus the experimental

investigation was constructed around the Basic Life Saving (BLS) task of CPR and

defibrillation.

Below, hypotheses are presented by participant grouping. There are two distinct

hypothesis tracks. The first, labeled [CPR], captures predictions concerning participants’

physical performance related to the execution of CPR tasks as prescribed by the AHA

protocol. These tasks include delivering breaths and chest compressions and consider

variables such as the volume of breath delivered and the frequency of chest

compressions.

The second hypothesis track is devoted to classifying participant behaviors

[BEH]. COCOM based predictions foster the categorization of each participant’s

behavior as scrambled, opportunistic or tactical. In contrast to the physical variables

[CPR], that simply capture the result of their actions, COCOM measures [BEH] provide

insight as to how responders performed those actions.

GRE trained, no support (GRE-NO):

[CPR1] Performance measures will be grossly inadequate and far below the other groups.

[BEH1] Responses will be haphazard; they’ll have little understanding of their performance and make no improvements over the course of the scenario. They will demonstrate scrambled control.

GRE trained, device supported (GRE-DEV):

[CPR2] Early performance will be substandard but will ramp up quickly and should surpass the trained/unsupported group.

[BEH2] Most actions will be guided by the system. They will rely heavily on cues and feedback. May show signs of scrambled control early but plans, cues and feedback will quickly advance them to an opportunistic mode.

CPR trained, no support (CPR-NO)

[CPR3] Will perform well, but may make some mistakes. Thus, their overall performance could suffer somewhat, but expected to be adequate.

[BEH3] They must rely on memory. They have no cues or feedback. A plan should be evident if they have retained their training. This represents tactical control.

CPR trained and Device trained, both device supported (CPR-DEV) & (DEV-DEV)

[CPR4] These two groups should outperform the others groups. Overall performance should closely track prescribed treatment per AHA.

[BEH4] Though trained, it is likely they will rely or at least refer to the system commands and feedback, showing evidence of leveraging their training and the system. An opportunistic control mode is expected here.

METHODS The investigation employed a randomized 3(training) x 2(device) nested factorial

design. Naïve participants received one of three training programs and returned 2 weeks

later to complete the experimental procedure (either with or without device assistance).

Sensors collected multiple measures providing means to analyze participant performance.

#�����������#�����������#�����������#���������������Measures from 100 volunteers were analyzed for the study. The participants were

largely members of the University of Utah community. The mean age for all participants

was 22.6 (SD=6.7) years. We required that participants were not knowledgeable in CPR

or first aid. Forty-five participants claimed prior CPR training with a mean of 6.5 years

(SD=6.5) since the training (it is common that high school students from the area receive

a training class however, it does not result in certification). A questionnaire was

administered in order to screen for CPR knowledge (Appendix D). The researchers asked

if they had been trained in CPR or with an AED and asked basic skill questions such as

the ABCs of CPR. None of the participants was able to answer all of the knowledge

questions correctly.

Participation in this experiment required a tyro’s knowledge of CPR for two

reasons. First, two conditions sought to simulate highly uninformed responders. The

remaining three conditions were best served with controlled, equivalent training for each

group. Participants demonstrating intermediate CPR knowledge or greater or currently

certified responders were excluded. Four volunteers were excluded prior to training due

to advanced CPR knowledge. Two additional participants were excused after

experimental training because they became certified after during the delay interval (in

order to be employed as lifeguards).

������������������������������������!��������������

Three different training conditions were utilized in the experiment. The “CPR”

training consisted of a slightly modified version of the American Heart Association

(AHA) protocol. The modifications included dropping portions of the procedure

including checking for pulse and calling for help as these were not emphasized in this

investigation. Two groups of twenty (40) participants received CPR training. Twenty

(20) more participants received “device” training (DEV-DEV in figure 1). Device

training mirrored CPR training as much as possible but incorporated the device during

instruction. DEV-DEV was the only group with exposure to the device prior to the

experimental session. A registered nurse, (responsible for anesthesia resident training at

the hospital and currently certified as a CPR instructor), administered the CPR and device

training. Prior to completing the training, each individual demonstrated aptitude to an

adequate level and trained to criteria. Finally, two more groups (20 per group) trained in

a fashion unrelated to the task. The experimental design called for two naïve groups in

which no relevant training was given. These participants learned strategies to improve

verbal scores on the GRE in order to maintain a consistent experience for all groups.

Comparable class size, duration, and participation were maintained for all training

sessions. Figure 1 provides a visual representation of the design and names the groups.

TESTING Figure 1. 3 (Training) x 2 (Device) nested factorial design

�����������The research assistants providing analysis also required special training in order

to properly code the data from the video tapes. Coders were taught to distinguish

behaviors for given actions but kept uninformed as to the foundation for those

distinctions and remained naïve to the hypotheses, control mode categories, and

participant training.

A coding sheet (Appendix B) and specific coder training governed the

systematized coding method. The sheet consisted of an array of actions reflecting

potential responder behaviors for each fundamental subtask and coders were trained to

distinguish subtle discrepancies in execution. For example, the coders chose from three

descriptions of mask holding to capture performance data. One selection was an

anesthesia hold. This type of mask placement was taught in training and required a

TR

AIN

ING

GRE trained

CPR trained

n = 20 for each group N=100

no device device

GRE-NO

CPR-NO

GRE-DEV

CPR-DEV

DEV-DEV

specific hand and finger configuration easily recognized on video. A second choice was

any two-handed grip (less the intricacies of the anesthesia hold) that involved two points

of contact sealing the mask. Lastly, a catch-all was used to code inferior mask holds

such as one-handed holds or the failure to use the mask at all.

The coders compiled these raw behavior data enabling the creation of a table to

categorize each participant as behaving in one of three control modes (per COCOM,

strategic was eliminated). Once the coders assessed individual actions such as mask

placement, the researchers calculated the proportion of actions by mode for each

participant and a control mode category was assigned. Most participants performed

actions from at least two of the categories, but usually a dominant control strategy

emerged. Over the duration of the scenario, each judgment contributed to a control mode

profile fostering an aggregate classification of mode for each responder.

)�������)�������)�������)�����������The device provided a means for novice users to perform effective CPR and

defibrillation tasks. Based on the AHA protocol, it delivered protocol instructions crafted

for novice understanding via visual and aural prompts. Utilizing smart sensors and

algorithms, the device assessed user actions and customized feedback to improve

performance. For example, if the sensors detected the chest compressions were too

shallow and too slow, it prompted the user to “push harder” and “push faster” for the next

set of compressions. These sensors not only drove feedback algorithms but also collected

data. In addition to relieving many cognitive burdens, there were also engineered

improvements such as the integrated headrest and mask. Pre-experiment investigation

revealed many novice responders had difficulty maintaining an open airway while giving

breaths. The headrest was designed to ensure victim head-tilt to keep the airway open

without requiring the responder to manually perform the task each time while giving

breaths. Figure 2 shows the device in use.

Figure 2. The device in use on training mannequin

The system provided the first visual and auditory cues to initiate the actions of the

novice responder. The system governed the pace and content in accord with the operator

performance by monitoring changes in the task space (i.e. tool placement, flow meter

readings). For example, the instructions required for rescue breaths were withheld until

the system recognized correct head placement. Pressure sensor in the headrest

determined the placement of the victim’s head. The system managed the action plan and

provided the user simple, actionable commands.

After giving simple instructions and monitoring performance, the system then

provided feedback if necessary. For example, after placing the mask, the system stated

‘give two breaths” and provided an animation demonstrating the proper method. Then

the sensors actively monitored the inspired volume of air into the lungs. If the rescuer

delivered insufficient volumes, instead of moving to the next step, the system encouraged

the responder to “give two large breaths”. Feedback of this ilk was critical in elevating

performance early in the scenario.

A headrest, anesthesia breathing mask (with one-way valve), and defibrillator

pads from Zoll’s AED-plus, and Lilliput 8-inch touch screen LCD served as the tools

available to the responders in the “device” condition (Figure 3.). The headrest was

customized from the foam of Giro bike helmet. Two pressure sensors were inserted at

the neck and in the center of the bowl (in order to detect proper head placement). The

mask apparatus consisted of a standard anesthesia mask, bacteria filter and a one-way

valve (directs victim’s exhalation away from the responder).

A Dell Dimension desktop and a Dell Latitude laptop provided the computing

resources for generating the displayed animations and auditory cues as well as collecting

data from: two Novametrix Medical Systems, Inc. CO2SMO Plus flow monitors (one

inside the mannequin, a second externally in the mask); pressure sensors (headrest),

linear potentiometer placed on the spring inside the mannequin.

The signals were converted by a PMD 1208 LS, Measurement Computing

Systems analog-to-digital converter. All software was written in C++. A Laerdal

Medical Little Annie mannequin portrayed the victim for each responder. The

mouthpiece was exchanged and discarded for every responder while the bacteria filter

and mannequin “lungs” were replaced every five participants or fewer as needed.

Figure 3. Components of the device.

#�������#�������#�������#�����������Training sessions were offered at various times and locations for a period of 90

days. Participants were blinded to the type of training they would undergo. After the

group training session, each individual was to schedule a testing date a minimum of

fourteen days to maximum of twenty-one days after training. Participants’ self selection

of training and experimental times contributed to the randomization process.

Upon returning for the experimental session the experimenter ensured consent

forms were completed, verified training, and read the participant the appropriate

instructions. All were informed that they would enter a room with an unconscious victim

(no breathing, no pulse). Their task was to perform CPR until help arrived (the victim

would remain unconsciousness for the entire scenario).

The groups not utilizing the device were informed that “tools” would be available

adjacent to the victim. These included a mask for ventilating the victim as well as pads

associated with the defibrillator. The components were identical to those available in the

device condition.

The device group had the added benefit of the headrest and of course, the

protocol, audio and visual cues and feedback presented by the device. Pre-experiment

instructions encouraged device-group participants “…to follow the device as closely as

possible”. A still frame of the video instruction to place the headrest is displayed in

Figure 4.

Figure 4. Snapshot of video instruction "place headrest"

After completing the scenario, the researcher praised the efforts of the participants

and reassured them it was simply a simulation with an inanimate object. The debriefing

also included a video-cued recall exercise as well as survey to assess actions. They were

finally thanked for their time and paid $30.

RESULTS Sensors placed inside the mannequin collected the clinically relevant data.

Variables appraising CPR performance received highest priority. These generally

include measures relating to ventilation and circulation. These types performance

parameters can serve as inputs to models of oxygen delivery and ultimately impact

survivability.

Data supporting the COCOM-based interests were captured on video and assessed

through several coding procedures. Four research assistants, blind to hypotheses,

observed behaviors and selected corresponding action descriptions. The researchers then

compiled those data and determined the appropriate control mode.

Lastly, survey instruments allowed the participants to provide their perceptions,

comments and display their knowledge. The survey was utilized to provide additional

support for the behavioral data.

Throughout the rest of this work, participant groups are designated (as in Figure

1) by the training-support convention. The GRE-NO group received GRE training and

was not supported by the device. Participants in the group CPR-DEV received training in

CPR and utilized the device during the testing phase of the experiment.

#�������������%���#�������������%���#�������������%���#�������������%������� The availability of the physiological data for each group is itself informative.

The first data that captured the researchers’ attention were the number of responders

registering values for the dependent physiological measures (the “n” for each group).

Each sensor had a minimum threshold in order to distinguish a valid input from readings

attributed to other artifacts. For example, a 50mL threshold was set for the flow meter to

measure breaths. The threshold was set because small rescue breaths would be

indiscernible from the air movement induced by chest compressions. The 50mL

minimum enabled the researchers to identify deliberate rescue breaths in the sensor data.

As stated, 20 participants composed each group. For GRE-NO, only 6 participants

provided measurable breaths and compressions (n = 6). In contrast, the remaining groups

preserved a minimum of seventeen or 85% (GRE-DEV). Only 30% of the participants in

GRE-NO were able to meet the minimum performance requirements to surpass sensor

thresholds. However, this does not suggest those six in GRE-NO performed well; it only

states their performance was discernible from no action at all (unlike the other 14

members of their group). The abysmal performance of GRE-NO supports the first

performance hypothesis [CPR1] which predicted GRE-NO would perform far worse than

the other groups. The difference between the number of participants achieving

measurable performance for GRE-NO and GRE-DEV (equivalent training, but GRE-DEV

used the device) proved significant (Fisher’s exact test, one-tailed, p < 0.001).

Table 1 provides summary performance measures for critical CPR variables by

group. The mean for each group is displayed with the standard deviation in parentheses

beneath the mean. The researchers employed a one-way multivariate analysis of variance

(MANOVA) to calculate differences attributable to the groups. The results indicate a

significant difference in performance measures for group among all measures (Wilk’s

Lambda = 0.126, approximate F(20,279.5) = 12.1; p<0.001).

Corresponding univariate analyses and post-hoc, pair-wise testing is also

presented where appropriate. Scheffe tests were selected for all post-hoc testing. Despite

results from Levene’s test for equality of variance (p < 0.002) for 4 of 5 variables which

would advocate another test such as Dunnett’s T3, Scheffe’s test proved to be the most

conservative for avoiding Type I errors. Statistical results are posted in Appendix F.

Significance is determined at the α = 0.05 level.

Table 1. Group means for CPR performance variables. (Standard deviations in parentheses below).

Group

Breath : chest

compression (CC) ratio

Inspired volume with each

breath (mL)

Chest compression (CC)

frequency (per minute)

Chest compression

(CC) depth (in.)

GRE-NO

0.7 : 4.8 (1.2 : 4.7)

125 (204)

43.4 (31.0)

1.2 (0.8)

GRE-DEV

2.7 : 14.7 (0.9 : 4.6)

982 (383)

75.7 (21.8)

1.4 (0.3)

CPR-NO

1.9 : 13.2 (0.7 : 2.4)

460 (316)

95.5 (10.5)

1.8 (0.3)

CPR-DEV

2.6 : 14.3 (0.6 : 2.4)

867 (367)

84.1 (12.2)

1.6 (0.3)

DEV-DEV

2.3: 15.1 (0.5 : 2.4)

1235 (423)

91.1 (5.2)

1.8 (0.2)

The first physiological hypothesis [CPR 1] predicted GRE-NO would severely

underperform compared to the other groups and provide clinically meaningless

interventions. Univariate analyses for all five dependent measures resulted in significant

differences driven largely by the GRE-NO data. The univariate results were: breaths

administered (F(4,88) = 17.3, p < 0.001), chest compressions delivered (F(4,88) = 28.1, p

< 0.001), chest compression rate (per minute) (F(4,88) = 24.0, p < 0.001), compression

depth (inches) (F(4,88) = 4.9, p < 0.001), and average volume (mL) delivered with each

breath (F(4,88) = 30.9, p < 0.001). Post-hoc testing revealed GRE-NO means were

significantly lower than the other groups in the number of breaths and compressions

delivered and the rate of compressions (p < 0.003). Further, their breath volume (M =

125, SD=204 mL) and chest compression rate (M = 43, SD =31 per min) were well

below the AHA guideline of 800 – 1200mL per breath and 100 compressions per minute

respectively.

The second physiological prediction [CPR 2] forecast a slow start for GRE-DEV

but insisted their performance would quickly improve and even surpass CPR-NO. Figure

5 shows group chest compression rate means for each successive cycle. Following GRE-

DEV’s progress from their first cycle to their last shows an improvement in rate from the

low sixties to the high eighties. Similar gains are portrayed in Figure 6 charting breath

volumes. Their first cycle falls short of the recommended 800mL. However on the next

cycle they exceed the 800mL minimum and deliver sufficient breaths in all remaining

cycles for a mean breath volume of 982 (SD=383) mL. GRE-DEV significantly

surpassed CPR-NO (M = 460, SD = 316 mL) in delivering breath volumes (p=0.001)

lending credence to the hypothesis that GRE-DEV could outperform CPR-NO in some

instances.

A multivariate test produced a significant difference for the slopes (Wilks’

Lambda = 0.566, F(8,188) = 7.7; p<0.001) as did the univariate tests for chest

compression rate (F(4,92) = 9.0, p<0.001) and average breath volume (F(4,92) = 9.4,

p<0.001). Post-hoc Scheffe tests revealed GRE-DEV showed significantly improved

chest compression rate over GRE-NO, CPR-NO and DEV-DEV (p ≤ 0.006). Similarly,

GRE-DEV exceeded the learning rate of GRE-NO and CPR-NO for average breath

volume (p ≤ 0.001).

The third performance hypothesis [CPR 3] focused on CPR-NO and assumed they

would likely perform adequately, but were also subject to a higher probability of error.

Overall, their performance adhered to guidelines and demonstrated proficiency but

proved no better than the other device groups. CPR-NO’s means were not significantly

better except in the case of exceeding GRE-DEV for chest compression rate (p=0.042).

However, as predicted, they did demonstrate inferior performance in one aspect. Their

mean breath volumes (M = 460, SD = 316 mL) not only failed to meet the minimum

suggested volume (800mL), but significantly lagged all device groups in this

performance measure (p < 0.017). Clinically, this performance would do little to deliver

oxygen to the blood stream.

The AHA recommends a volume of 800mL to 1200mL for each breath. This

range accounts for the roughly 400mL of “dead space” that must be cleared between the

victim’s mouth and alveoli. This dead space volume must be completely purged to

optimize oxygen transfer to the blood stream. In essence their optimal chest

compressions are wasted; their efforts would result in circulating deoxygenated blood.

Mean CC frequency per cycle

30

40

50

60

70

80

90

100

110

1 2 3 4 5 6 7 8 9 10 11 12

cycle

CC

per

min

ute

gre-no

gre-dev

cpr-no

cpr-dev

dev-dev

����

����� � ���

� �

��� ���������

����� ����

Figure 5. Group means of chest compression rate for each cycle

The last physiological hypothesis [CPR 4] conjectured both trained groups

supported by the device (CPR-DEV, DEV-DEV) would outperform the other groups and

closely adhere to protocol prescribed values. Table 1 shows means for all five variables

closely track the prescribed values in the AHA protocol for both CPR-DEV and DEV-

DEV. However, post-hoc tests revealed these two groups only outperformed GRE-NO in

all five dependent measures (p < 0.001) with the exception of the difference in chest

compression depth means for GRE-NO and CPR-DEV (p = 0.20). Neither of these

trained groups with the device surpassed GRE-DEV on any variable ( p > 0.18)

Avg Breath Vol (mL) per cycle

0

200

400

600

800

1000

1200

1400

1600

1 2 3 4 5 6 7 8 9 10 11 12cycle

Bre

ath

Vol

(mL)

gre - no

gre - dev

cpr - no

cpr - dev

dev - dev

����

����� � ���

� ��� � ��

����������

Figure 6. Group means of inspired volume per breath for each cycle

"*"*+ �%���"*"*+ �%���"*"*+ �%���"*"*+ �%��������������$����

Four blinded coders performed the control mode analyses by reviewing video and

recording observations to a data template. The coders provided COCOM based for

subtask actions executed by the participants. Inter-rater reliability was assessed with

Cohen’s Kappa (Kappa = 0.899, see Appendix F). This tabulation was well above the

commonly accepted 0.70 level threshold validating the use of the coding results.

Table 2 summarizes the proportion of actions by mode for each group. A

multivariate analysis of the control mode data resulted in a significant difference for

group (Wilks’Lambda = 0.049, approximate F(12, 246.3) = 43.6, p < 0.001). The

appropriate univariate and post-hoc Scheffe results are discussed below.

Table 2. Group means for COCOM classification (in percent). (Standard deviations in parentheses below).

Group %Scrambled %Opportunistic %Tactical

GRE-NO 90

(12)

4

(7)

5

(7)

GRE-DEV 30

(15)

66

(15)

4

(8)

CPR-NO 19

(22)

16

(7)

64

(25)

CPR-DEV 17

(12)

62

(16)

21

(17)

DEV-DEV 15

(9)

60

(10)

24

(12)

The COCOM data conform well to the behavioral hypotheses. Haphazard

responses, trial and error approaches contributing to scant comprehension of the process

constituted the first behavioral hypothesis [BEH 1]. GRE-NO responders demonstrated

their lack of knowledge and an inability to improve during the scenario as all twenty

participants were classified in the scrambled control mode. Almost all of their actions (M

= 90%, SD = 12%) epitomized scrambled behavior. Post-hoc tests revealed the 90%

mean for GRE-NO was significantly higher compared to scrambled scores for all other

groups (p < 0.001).

In contrast, GRE-DEV benefited from the provisions of plans, cues, and

feedback and demonstrated an opportunistic control mode affirming the second

behavioral hypotheses [BEH2] which predicted the device would elevate their level of

control. GRE-DEV posted the highest proportion of opportunistic behaviors (M = 66%,

SD =15%). Post-hoc tests showed this percentage was significantly higher than the two

non-device groups (p < 0.001) but not statistically different from the other two device

groups (p > 0.658). Most of GRE-DEV ‘s remaining actions (M = 30%, SD = 15%)

resembled scrambled behavior. Only two participants (10%) in the group coded as

scrambled while the rest convincingly performed in an opportunistic mode.

The third behavioral prediction [BEH3] involved the CPR-NO group. Well

trained but unsupported (no plan, cues, or feedback), this group relied on knowledge

retrieval to perform the task. Adequate performance under such conditions demanded a

tactical control mode; and the data concur. Tactical actions accounted for the majority

(M = 64%, SD = 25%) of their activity, significantly higher than any other group (p <

0.001). Their opportunistic score (M = 16%, SD = 7%) was significantly lower than the

three device groups (p < 0.001) CPR-NO’s scrambled score (M = 19%, SD = 22%) was

comparable to the three device groups and reveals that participants in all groups made

errors. Of the twenty in CPR-NO, eighteen coded as tactical and only two scrambled

confirming the control mode prediction.

The final behavioral prediction [BEH4] conceded that despite their training, the

presence of the device would engender an opportunistic mode of control for CPR-DEV

and DEV-DEV. The device captivated these participants. CPR-DEV tallied 16 members

in opportunistic, 3 in tactical and 1 in scrambled. Again, the three device groups did not

differ significantly in their opportunistic scores (p > 0.658), nor scrambled scores (p >

0.074 ), but GRE-DEV did lag the other two groups in tactical actions (p < 0.026). All 20

participants in DEV-DEV registered opportunistic modes of control.

#��������)���������#��������)���������#��������)���������#��������)�������������A total of nine subtasks were identified as necessary steps to comply with the

AHA protocol. The researchers recorded the number of correctly completed steps

achieved by each participant. The analysis was confined to only the first cycle since all

tasks must be completed the first time through the procedure. For example, if the pads

are placed in the first cycle, there is no need to complete that step in each cycle.

Subsequently, only a subset of the tasks was repeated (Appendix C, Error Coding,

contains the list of subtasks).

Two coders independently evaluated each action on the video and determined if

participants correctly completed the required action. Of the 900 judgments, only 19

discrepancies emerged resulting in high concurrence between the raters (agreement

exceeded 97.8%). Table 3 contains the number of protocol steps each group performed

correctly.

Table 3. Mean number of protocol steps executed correctly (9 max).

Protocol Steps GRE-NO GRE-

DEV CPR-NO CPR-DEV DEV-DEV

MEAN 1.6 7.80 7.85 8.45 8.65

SD 1.46 1.05 1.49 0.88 0.49

These group means did demonstrate a statistically significant difference (F(4,95)

=135, p<0.001). The results were congruent with all behavior hypotheses stated earlier.

GRE-NO was unable to accomplish most subtasks critical to CPR and defibrillation. Post-

hoc tests demonstrated GRE-NO was statistically disparate from all other groups (p <

0.001). The second [CPR 2] and fourth [CPR 4] hypotheses predicted the device groups

would complete all subtasks and the data here support that. The data also showed CPR-

NO [CPR 3] demonstrated sufficient recall of the procedure as they too regularly

completed all steps.

The completion of protocol subtasks was also assessed for proper sequencing.

The scrutiny of sequence proved formidable for CPR-NO. As postulated in the third

performance hypothesis [CPR 3], relying on memory proved onerous. CPR-NO

demonstrated problems recalling the correct sequence. While they remembered the

necessary steps, 60% of the responders failed to perform them in the correct order. Table

4 displays the number of responders in each group that followed the sequence correctly

(Yes), and those that failed to do so (No).

Table 4. Number of responders exhibiting correct sequencing of subtasks.

Correct Sequence

all-DEV (GRE-DEV, CPR-DEV,

DEV-DEV)

all NO-DEV (GRE-NO, CPR-

NO)

CPR-NO

GRE-DEV

Yes 45 8 8 15

No 15 32 12 5

Fisher’s Exact test was again used to determine statistical significance for

multiple 2 x 2 comparisons. The three device groups significantly outperformed both no-

device groups (all-DEV vs all NO-DEV, p < 0.001). Removing GRE-NO from the

analysis to unburden CPR-NO with disproportionate inefficacy still yielded a significant

difference (all-DEV vs all CPR-NO, p < 0.001).

��,���%�����,���%�����,���%�����,���%�������

Survey responses resided on a 10 point scale and were collected post-experiment.

Inquiries coincided with COCOM parameters such as the availability of plans, the

utilization of cues and feedback, and the ability to comprehend the situation. The

researchers designed the low, middle, and high values of the scale to embody scrambled,

opportunistic, and tactical modes of control, respectively (see Appendix C for the Post-

Run Questionnaire Survey as well as Appendix F for full statistical analyses). The

survey was intended to provide additional support for the COCOM predictions.

A multivariate analysis examining the groups for all survey variables proved

significant (Wilks’ Lambda = 0.047, approximate F(32, 326.1) = 13.2, p<0.001).

Additional statistical analyses are given where appropriate.

������The first survey item attempted to reveal the driving force initiating responder

actions. Lower survey scores indicated a random approach, the middle range suggested

actions were guided largely by the system, and higher numbers indicated the use of a

retrieved plan, consistent with the COCOM continuum. A significant result between

group means (Table 5) was confirmed (F(4,95) =12.0, p<.001). Post-hoc Scheffe tests

revealed that CPR-NO did differ significantly from the other groups (p ≤ 0.029) with the

exception of DEV-DEV.

Table 5. "Action" survey results

The means for each group are in line with the outlined behavioral hypotheses. In

particular, CPR-NO turned in the highest rating (M=8.25, SD=1.1) suggesting the use of

memorized plans (no system cues were available in this condition) [BEH3]. GRE-DEV

registered a mid-range score (M = 5.75, SD=1.3) confirming their leveraging of system

functionality [BEH4]. GRE-NO’s score of 6.05 is not consistent with the first behavioral

hypothesis [BEH1], nor is it likely to be an accurate report as cue availability was

Action Score GRE-NO GRE-DEV CPR-NO CPR-DEV DEV-DEV

MEAN 6.05 5.75 8.25 6.75 7.85

SD 1.66 1.29 1.11 1.74 1.08

essentially non-existent. Perhaps they believed the presence of tools (such as mask and

pads) served as cues, but their proclivity to forgo tool use is further behavioral evidence

contradicting their survey mean.

%������&�The second item on the survey captured the participants’ perception of the

importance of feedback to their performance. Higher ratings indicated that feedback was

utilized and fostered ample improvement. The middle of the scale suggested occasional

use and limited enhancement, and the low end of the scale represented the absence of

feedback. In addition to collecting participant sentiment, it also served as a check – the

paltry feedback provided in the “no device” conditions left little information to be

gleaned (though trained rescuers astutely claimed that leaking breath sounds constituted

feedback).

Again, a significant difference was observed between the groups (F(4,95) = 31.7,

p<0.001). As feedback is a critical element in JITS, behavioral hypotheses posited the

device groups would rate feedback favorably whereas the groups without the device

should not. GRE-NO and CPR-NO had mean scores of 2.85 and below while the three

groups with the device tallied survey scores between 7.0 and 7.8 (see Table 6).

Table 6. "Feedback" survey results

Feedback Score GRE-NO GRE-DEV CPR-NO CPR-DEV DEV-DEV

MEAN 2.85 7.80 2.45 7.00 7.60

SD 2.15 1.85 2.39 1.89 2.21

Post-hoc tests revealed that device groups did not differ significantly from each

other, nor did the no-device groups, but each device group differed significantly from

each no-device group (all p-values <0.001).

An adjuvant aspect of the feedback data was interesting to note. The researchers

tallied the number of participants receiving chest compression feedback in each group.

Classifying a participant as having received feedback required three or more instances

over the course of the scenario (participants receiving at least 3 feedback occurrences

generally encountered many more instances of feedback where as the researchers

determined 1 or 2 instances was not sufficient to justify the feedback categorization).

Table 7 displays the sums.

Table 7. Number of participants provided feedback by the system.

Feedback Provided GRE-DEV CPR-DEV DEV-DEV

Yes 16 8 15

No 4 12 5

Fisher’s Exact Test calculated the differences between CPR-DEV and DEV-DEV

resulting in a significant difference (one-tail p =0.026). This finding complied with

behavioral hypotheses quite well as JITS reduced the emphasis on retention. One

explanation could be that those trained with the device believed they could depend on it

later – comforted by that feedback would optimize their performance. During training,

DEV-DEV may have learned to rely on the device for guidance and processed the stimuli

on a more superficial level. In contrast, CPR-DEV was not aware an assist device would

be present for the test and felt more compelled to remember their training.

��������Participants were asked to rate their comprehension of the progress and results of

the scenario. Those comfortable in their understanding and confident in their efforts

gravitated toward the higher values. In contrast, the lower end of the scale was crafted

for participants with no ability assess their performance or postulate the outcome. The

differences in means for this inquiry were statistically significant (F(4, 95) =9.1, p <

0.001). Those participants aided by the device at least felt they maintained some

understanding of the events in comparison to their counterparts without the device.

Comparing the group means (Table 8) revealed GRE-NO significantly lagged each device

group (p ≤ 0.001) however, CPR-NO did not (p ≥ 0.232).

Table 8. "Outcome" survey results.

Outcome Score GRE-NO GRE-DEV CPR-NO CPR-DEV DEV-DEV

MEAN 3.85 7.15 5.60 7.30 7.30

SD 2.34 1.95 2.18 2.55 2.17

DISCUSSION The vision of JITS is to develop systems capable of enhancing user performance

as they confront unfamiliar tasks requiring immediate intervention. JITS accomplishes

this challenge by providing adaptive plans, cues, and feedback tailored to the user’s

understanding. System flexibility is required to adapt to dynamic contextual conditions

as well as user behavior.

The results of this study validate the tenets of the JITS approach. The data

demonstrated that participants with no specialized knowledge, skills or training could

perform as well as those given training and the opportunity to practice. This was

achieved simply by providing novice participants a JITS device. Additionally, their

performance was in the range of clinical guidelines suggesting physiological benefits.

"# ����-������� ����"# ����-������� ����"# ����-������� ����"# ����-������� �������� The JITS benefits are most evident when examining the performance differences

between GRE-NO and GRE-DEV. All participants in these groups lacked declarative and

procedural knowledge required to perform CPR. All completed a training session in

which they learned strategies for the verbal section of the GRE which in no way prepared

them for the CPR task. The difference between the groups was their use of the device

during the experimental test; GRE-DEV reaped significant benefits from a JITS designed

device. The data affirmed the performance gains.

The untrained responders without the device failed to achieve anything of clinical

import. Only 30% of GRE-NO provided inspired volumes large enough to be detected.

They delivered approximately one-third of the chest compressions required by the AHA

guideline. Other performance measures such as volumes and compression frequency also

failed to attain clinical relevance. From the CPR viewpoint, no breaths were delivered to

the lungs and no oxygen delivered to the vital organs that need it. This finding was not

unexpected; untrained responders with no support are unable to perform CPR effectively.

However, this study measured their fecklessness, providing a means for quantitative

comparison.

Providing a similar population of naïve responders a JITS system resulted in

vastly improved performance. The device aided group successfully delivered life-saving

treatment despite having no previous training or knowledge. Their breath-to-

compression ratio approached the guideline thanks to the pacing of the device. The

breath volumes delivered tracked the middle of the recommended range because sensors

drove feedback to guide them there. While their compression depth and frequency were

slightly below threshold, their performance still provided physiological relevant

treatment.

In a real cardiac event, this intervention would take place during the transit time

of professional responders. This four to eight minute duration is often void of

resuscitation efforts. The data suggest novices may be capable of providing effective

life-saving measures during this interval when equipped with a JITS device even if

untrained. A JITS designed intervention has potential to greatly increase the dismal SCA

survival rates.

An even more intriguing comparison is that of GRE-DEV vs. CPR-NO. This

comparison essentially examined the difference between class training and real-time

training. The data demonstrated GRE-DEV was able to perform on par with the trained

group, and in the case of oxygen delivery, outperformed CPR-NO considerably. This

suggests the real-time system achieved results comparable to class training. Should these

findings continue to be upheld by future experimentation, a case could be made

challenging the classroom training paradigm. Instead of training a few responders that

may never be called upon to act, training would be given to people actually responding to

a cardiac event.

The comparison of CPR-DEV vs. DEV-DEV did not result in any discoveries of

note. The distinction of these groups was created to examine the effect of specific

training prior to device use. Having used the device in training produced no measurable

advantage in this study. Since both groups performed close to the protocol

recommendations, this could be the result of a ceiling effect. Additionally, it should be

noted that the introduction of the device to a group trained without it was not hampered

when required to use the device during the experiment. In fact, comparing this group to

CPR-NO it is evident that the device helped a great deal in delivering effective breaths

and did not induce negative transfer effects.

������������������������������������������������For most of the untrained responders this experiment provided their first attempt

at CPR. How were untrained responders able to achieve such impressive performance?

It is clear from the data the JITS device enabled an untrained group to perform

successfully. The equivalently prepared group without the device failed to accomplish

anything that could positively impact survival.

By observation it was evident that all GRE-trained participants entered the

scenario with trepidation. GRE-DEV respondents showed some relief when they realized

they would receive computerized support. The device first gave the instruction “touch

here to start”. The intention was to capture the user’s attention and direct it toward the

device. This simple instruction set the stage for the upcoming interaction. This first,

simple command prepared the user for the more complex information that followed. For

some the command was immediately effective, others required a few seconds to orient

themselves.

CPR-NO, though received training, did not deliver breaths adequately. Of course,

they were not supported with feedback and had few resources for identifying their

deficiency. Therfore, they were unable to perceive their error and continued to perform

poorly. Though they administered effective compressions, their failure to provide

sufficient ventilation rendered the compressions far less effective.

It is apparent the CPR-NO participants were unaware of their impotent

ventilations. An ability to assess the progress of the task is a critical element in

controlling any system. It is particularly difficult for non-experts in stressful situations

according to COCOM. For those cognizant enough to ascertain the problem, they found

it difficult to correct their mistakes.

Laypersons’ difficulty in delivering rescue breaths is so ubiquitous that clinical

researchers and professionals advocated revision of CPR guidelines. Many called for the

simplification of the process which included increasing the number of chest compressions

instead of wasting effort on ineffective rescue breaths (Cummins, & Hazinski, 2000,

Sanders & Ewy, 2005). Further, the interruption of chest compressions resulted in the

absence of circulatory stimulus for excessive periods (Valenzuela et al., 2005). Many

shared their conviction as reflected in the November 2005 revision of the CPR guidelines

(ILCOR 2005). The number of chest compressions recommended was raised to 30 from

15 essentially attenuating rescue breath efforts by default.

This significant change to the guideline highlights the disparity between a static,

predetermined approach versus a flexible, adaptive JITS system. The revision is based

on the assumption that all resuscitation needs will be better served by the new

parameters. A JITS solution could make the determination during the case, and perhaps

even during each cycle and adjust the response according to the needs of the victim. If

the first breaths were inadequate, the system would coach the responder to provide more

effective breaths. Processing the first breath inputs (with active sensors) might result in

feedback alerting the responder to a problem and offer corrective assistance such as

“press harder on mask” and “give 2 larger breaths”. Perhaps the first attempt at breaths

is unsuccessful and too much time has passed, so the system decides to move to chest

compressions. What if fifteen or twenty or twelve was the optimal number of

compressions to attempt before going back to breaths? Such intervention can only arise

from real-time systems capable of monitoring the context and designed to be sufficiently

adaptable. As demonstrated in the experiment, an adaptive, monitored plan, cues and

feedback loops provided a significant advantage to participants in the device groups.

An additional impediment to lay responder success is their poor retention of CPR

skills. Morgan, Donnelly, and Lester, (1996) surveyed the scope of the problem in

England and found that only 7% of trainees were able to perform safe, effective CPR six

months after training. Additional studies reveal the retention problem is not limited to

non-professionals. Medical students demonstrate similar retention failures (Graham &

Scollon, 2002, Fossel, Kiskaddon & Sternbach, 1983). Thus the performance delivered

by CPR-NO should be considered optimal since only two weeks passed between training

and test. Research findings predict skills would degrade considerably as more time

elapsed between training and test (Morgan, Donnelly, & Lester, 1996, Starr, 1998).

Memory should not be a significant impediment to healthy patient outcomes.

Even unintelligent systems can provide memory aids. On a surprise retest 30 days after

training, Star (1998) observed a 4-fold improvement in performance skills for participants

using a simple prompt system versus those relying on memory alone.

Evidence that a brief prompt was capable of improving performance should spark

a great interest in the development of a more complete JITS solution. The prompts were

bound to a sequence, did not benefit from contextual awareness and could not leverage

real-time data. A JITS system for CPR could enhance the response of trained responders

as well as enable sufficient performance from novices.

"*"*+ �#��� ���������%���������+������"*"*+ �#��� ���������%���������+������"*"*+ �#��� ���������%���������+������"*"*+ �#��� ���������%���������+���������� This research pioneered one of the earliest applications of the COCOM to

generate dependent measures sufficiently robust for quantitative analysis. Very few

researchers have attempted to marshal the COCOM predictions in such a manner largely

due to the inchoate bounds of control modes and an inability to derive cogent measures.

Stanton, Ashleigh, Roberts, and Xu, (2001) provided the first empirical support for

COCOM hypotheses. However, they’re coding process was not developed with the rigor

employed in this work. Ultimately, successful utilization of COCOM parameters

requires a very specific context. Therefore, previous efforts can serve only to guide

future development. The measures described in this work are not transferable.

The basis for the COCOM-derived dependent measures reside in Hollnagel’s

mode characteristics. Each of the subtasks (i.e. holding mask, placing pads, delivering

compressions) provided subtleties for control mode classification. For example, one

technique forholding the mask was the anesthesia-style hold taught in training. This grip

exemplified tactical control as it demonstrated knowledge retention and skill.

A participant portraying an opportunistic mode first showed evidence of perusing

the screen prior to placing the mask. Then a two-handed hold was employed with a

downward thrust near the nose and chin. This provides a logical opportunity to redesign

for the novice user. While there are benefits to the anesthesia hold, the designers

surmised this technique would be very difficult to portray and for naïve users to learn in

real-time. As a result, a compromise was reached. The conveyed information for mask

holding was simplified to a two-hand hold placing pressure in two points to create a seal.

This approach was far simpler to portray and perform. In most cases, the system

instructed the user to “press harder” during mask placement. A pressure sensor in the

cuff of the mask was set to ensure enough downward force was applied to generate a

good seal to ensure rescue breaths did not leak from the mask. A poor seal was a

significant problem for CPR-NO which did not have the sensor-driven feedback.

Scrambled control designations for the mask placement task took many forms.

Many of the GRE-NO participants never used, and some never noticed the mask was

available. Such instances received the scrambled classification. Two of the GRE-NO

responders attempting to use the mask inserted the mouthpiece into the mannequin and

blew into the mask. Reversed mask insertion ensured no ventilation and exemplified the

hallmark trial-and-error approach associated with scrambled control. Members of other

groups also recorded scrambled actions in the mask hold because they only used one

hand to seal the mask to the victim’s face. This practice always produced leaks and

therefore less effective breaths. Without the aid of the device or serendipitous recall from

training, this error usually went uncorrected.

Utilizing COCOM parameters and translating them into CPR-required subtasks

produced a matrix capable of capturing responder behavior. Though Hollnagel cautions

mode demarcation is inexact, the data conformed very well to hypotheses.

As predicted [BEH 1], GRE-NO performed 90% of their actions in scrambled

mode. All 20 participants coded as scrambled overall for the scenario. Based on the fact

that they had neither skills nor knowledge to employ, there was little probability they

could perform in any other manner.

CPR-NO demonstrated tactical control a majority of the time (64%). They too

conformed to hypotheses [BEH 3] as it was predicted they would make mistakes due to

their reliance on recall. Their erroneous behavior was highlighted by 20% of their

actions coded as scrambled and 60% of the responders failing to properly sequence the

subtasks. Despite great success in adhering to the protocol for breath-compression ratio

and compression frequency, mistakes such as their poor ventilations would adversely

affect patient outcomes.

The variance in CPR-NO means was also noteworthy. CPR-NO recorded the

highest standard deviations in Table 2 (SD = 22 scrambled, SD=25 tactical), indicating a

considerable range in the group performance. It is also important to note the experiment

took place only two weeks after their training and still substantial recall errors surfaced.

The variance suggested some responders displayed superb retention while severe decay

was evident in others. As other research has shown, performance of all members can be

expected to decline with greater delays between training and test. Over time, a reduction

in performance would be predicted as would a correlated reduction in variance.

Examining the device groups, GRE-DEV was hypothesized to demonstrate

opportunistic control and the data support that claim. Sixty-six percent of their actions

coded as opportunistic. They also made some mistakes as evidenced by 30% of their

actions classified as scrambled. Observation of their performance indicated that many of

the mistakes they made occurred early in the scenario. Anecdotal examples include some

confusion while searching for the mask, and giving compressions with the wrong grip

and compressing much more slowly than the required pace. Most device-aided

participants corrected their actions in the subsequent CPR cycles.

Trained groups using the device also demonstrated control profiles easily adapted

to the COCOM paradigm. At firs t glance, their training should have enabled them to

demonstrate some tactical mode characteristics. Nonetheless, the presence of the system

would likely drive their actions in the direction of opportunistic control. The question to

be determined empirically was the proportion of opportunistic to tactical methods. Each

trained-device group performed roughly 20% - 25% of their actions in tactical mode and

more than 60% in opportunistic. Two factors are most plausible for generating this

distribution. First, the system was explicitly designed to seize attention and engage the

operator throughout the scenario. Second, with what can be considered limited training

and practice, most operators would likely defer to the system for expertise.

The post-experimental questionnaire served as yet another instrument employed

to assess the usually amorphous bounds of control mode categories and provided

converging evidence for the hypotheses. The survey instrument utilized multiple scales

to represent different COCOM parameters. Specifically, scales represented action

selection, utilization of cues and feedback, determination of outcomes, for goal setting,

and time pressure. The inquiries concerning action drivers (includes cues), feedback

utilization, and outcome determination yielded results consistent with hypotheses

In general, GRE-NO exhibited a scrambled control mode as they reported no use

of feedback and minimal determination of the outcome. GRE-DEV showed a high

reliance on cues and feedback. CPR-NO ranked feedback very low and recorded the

highest action score avowing their use of a memorized plan to drive their actions. One

disappointment in the survey results came from the time pressure scale. According to the

COCOM, operators in scrambled mode have a proclivity to perceive an enormous limit of

the time available. However, due to the length of the scenario and lack of purposeful

activity, five minutes probably felt like a long time superseding any time pressure they

may have experienced at the start of the scenario (it did to the researchers watching them

struggle).

Attempts to impugn the validity of the COCOM measures must contend with

converging evidence from the coding and the survey data. The COCOM proved valuable

not only in predicting human behavior and providing design guidance, but also as a

measurement tool to evaluate the system. Though it may be impossible to describe

global COCOM measures, a well defined context not only affords their use, but benefits

from their instantiation.

"���������"���������"���������"�������������Could JITS replace traditional class training? There are many advantages to

widespread deployment of such a system. The delay in getting professional responders to

the scene is well documented. Precious moments are lost in transport. Additionally, it

has been shown that even when they arrive, they are not performing optimally.

The most obvious benefit is the augmented number of available responders and

thus the ability to initiate life-saving treatment immediately. The paltry percentage of

trained citizens currently limits the probability that a witness to SCA will be prepared to

act. The limited response incurred is directly attributable to the current, flawed training

paradigm. Instead of training large groups of people that may never apply their skill,

JITS methods can train the few people that need to perform BLS immediately.

Good Samaritans, previously helpless because they lacked 6-12 hours of training,

would now be empowered to act. Herein resides the greatest decrease in response time

and greatest clinical benefit to the victim. When a witness can immediately respond to a

cardiac event effectively, the victim’s probability for a healthy outcome is vastly

improved. The researchers believe the JITS framework could be used to transform any

willing bystander into a capable responder.

)������ �There are several notable limitations to this study. The most apt concession

acknowledges the difficulty in mapping statistical significance on to clinical relevance.

While statistical differences surfaced for many variables, the clinical relevance of these

differences is not always evident. Conversely, differences that failed to breach statistical

thresholds could belie substantial medical implications. For example, are clinical

benefits realized when the mean volume per breath rises from 867mL to 962mL? This

work cannot answer that question and previous work with models has yet to provide a

satisfactory resolution or even determine if the question is worth asking.

A second limitation indicts the device itself. The criticality of response time

cannot be overemphasized. Yet, utilizing the JITS solution did cost the responders time

compared to the no-device groups. Appropriate usage and placement of tools, processing

lags to read sensors and generate feedback imposed delays beyond the control of the

responder. The algorithms contained “hold points” to ensure prerequisite steps were

performed prior to proceeding. These potentially can stall the procedure and thus

requires prudent threshold setting. Self-paced, CPR-NO completed more cycles per

scenario than any other group and spent less time between cycles demonstrating some

time cost attributed to using the system.

A more general limitation for JITS resides in its technological dependencies.

Algorithm and sensor development could present significant challenges and have

ramifications for human performance issues. For example, if the task analysis requests

unattainable data, proxy variables, or modeled data, procedure modifications may be

necessary. Relying heavily on technology also demands prepared contingencies for

system faults and failures.

The researchers and development team encountered several examples of

technological obstacles. The available computational processing speed demanded our

first concession. A potentiometer in the mannequin’s chest provided feedback for chest

compressions. However, the signal delay coupled with the algorithm made it impossible

to assess performance and deliver feedback during the cycle. Thus, the feedback, based

on the previous cycle, was presented to the responder at the beginning of the next cycle.

In a sense, the user was required to “remember” the force and speed of the previous

performance and adjust accordingly. It is not clear that the participants detected this

discrepancy and the concern is further assuaged by the improved performance due to

feedback. Nonetheless, future JITS systems will face numerous technological challenges.

Sensor placement inside the mannequin presented a further technical impediment.

Obviously, this would be impossible for a system intended for human use. Placement

inside the mannequin was performed for two reasons. First, it provided the most reliable

means of data collection – critical for experimentation. Second, these measures drive the

design of the sensors needed for the next iteration of the CPR device. In one case,

engineers are leveraging flow meter, inspired volume, and leak data in order to establish a

model capable of assessing inspired volume without an internal sensor.

Beyond technology, a further constraint is the need for tasks to be well structured.

The nascent stages of JITS are ill equipped to address the perplexing issues connate with

indeterminate tasks. The application of JITS requires a certain level of predictability.

Fortunately, a rigidly prescribed BLS protocol provided the structure necessary to

constrain and therefore predict most operator actions. This procedure relies on few

actions and subtasks. More complex activities will demand a great deal of contingency

planning and anticipation of off-normal conditions and actions.

Finally, the human limitations must be considered. For example, no amount of

JITS will enable recreational joggers to run 100 meters in 9.78 seconds (a physical

limitation). With tasks requiring highly practiced skills, such as landing an aircraft, JITS

will not mitigate a novice’s oscillatory control during a first attempt (automatized skills).

Nor can it be expected that a JITS system can equip a Private First Class with the war-

gaming prowess of a General (experience/knowledge).

In addition to motor skills, the emotional stability of the responder must also be

considered given the gravity of this task. There was no way to simulate the stress that

would likely envelope responders dealing with a real victim. For the most part,

participants maintained an emotional level that did not significantly degrade their

performance. However, this finding should not be generalized to a scenario involving a

real life. More experimentation is needed.

However, if designers are mindful of the capabilities and applicability of JITS

systems, ample domains can benefit from such support. The present demand for structure

does not nullify the applicability and benefits of JITS. Countless tasks are governed by

such protocols and can be seen in many domains including: aviation, aerospace, defense,

maintenance, manufacturing, medicine, process control, quality assessment, and safety.

������������������ The instantiation of the JITS framework can aid designers in the development of

systems crafted to improve non-expert performance of critical tasks. The need for such

support continues to grow as the demand for expertise outpaces supply. The JITS

framework can be employed in system development to attenuate specific expertise voids

supporting a wide range of objectives.

One issue that can be assuaged by JITS is user performance of infrequent tasks.

These atypical occurrences may include emergency situations or far less critical events.

The operator may have received training for the task, but the reality is the task is seldom

performed and the user is unpracticed. In this case, designers are dealing with a known

population. This enables some domain knowledge to be infused in the plan, cues and

feedback. The system must address the lack of practice, but probably does not have to

assume complete ignorance on the part of the user. It may be found that domain specific

cues and leveraging the knowledge of the operators can elevate performance.

Employing JITS systems aiding infrequent tasks could reduce or eliminate the

need for expending resources on training. Training for rare events is often ineffective

because of an extensive delay between training and practice and the misperception that

the training will never be needed. Additionally, these resources usually must be drawn

upon repeatedly not only for new users, but also refresher training. A JITS solution could

engender superior performance while curtailing wasted training costs.

The real power in broad deployment of JITS systems is the ability to significantly

increase the population capable of completing novel tasks. The system stores the

knowledge requirements, collects and interprets the data, and guides the user through a

workable plan. In situations formerly requiring significant expertise, JITS can supplant

that need and enable non-experts to complete the tasks.

A mature JITS system may do far more than elevate novice performance. It is

envisioned as a tool that would work with the individual from the first day of training and

accompany the user on every task mission. A robust student model would be constructed

over that time which would formulate optimized plans, cues, and feedback tailored to the

user. The same system would also recognize an unfamiliar operator and quickly diagnose

skill level. A mature JITS system is the manifestation of an intelligent, highly adaptive

system capable of supporting a diverse population of users accomplish tasks beyond their

proficiency.

APPENDICIES

)������.�)!�&/��'����"����)������.�)!�&/��'����"����)������.�)!�&/��'����"����)������.�)!�&/��'����"��������

)������.�&�!�"*"*+ �"�����������)������.�&�!�"*"*+ �"�����������)������.�&�!�"*"*+ �"�����������)������.�&�!�"*"*+ �"���������������

Mask no 2 hand - 2-hand seal 2-hnd anesthesia seal

breaths audible leak give when system

commands look for chestrise while

giving

cc 1handed; wrong grip; cc w beeps 15 consistent (despite

beeps)

body excessive move (waist

shoulders) looks @ screen little movement to

accomplish

Verbal what/how to do; express

frustration "Oh" count out loud; verbalize

plan

Start/Stop begin action- w/out finish

(correct) syst interrupt; follow sys ignore system to perform

correct tools fail to use; explore; search; used on command use correct w/out prompt can't find immediately anticipate

pads don't use watch screen to place place w/out looking

screen correct b4/w/out cue/fb place words up Mask no 2 hand - 2-hand seal 2-hnd anesthesia seal

breaths audible leak give when system

commands look for chestrise while

giving

cc 1handed; wrong grip; cc w beeps 15 consistent (despite

beeps)

body excessive move (waist

shoulders) looks @ screen little movement to

accomplish

Verbal what/how to do; express

frustration "Oh" count out loud; verbalize

plan

Start/Stop begin action- w/out finish

(correct) syst interrupt; follow sysignore system to perform

correct tools fail to use; explore; search; used on commanduse correct w/out prompt can't find immediately anticipate

pads don't use watch screen to placeplace w/out looking

screen correct b4/w/out cue/fb

)������.�"�!�)������.�"�!�)������.�"�!�)������.�"�!��.���� ������%����'�� ��.���� ������%����'�� ��.���� ������%����'�� ��.���� ������%����'�� �����

In-situ Error Coding Subject # ___________ Condition_______________ Date of Experiment____________ Errors during additional cycles Headrest Y N Mask used Y N Mask placement Y N 2 breaths Y N Shirt removed Y N Pads used Y N Tabs removed Y N Pad placement Y N Hands correct Y N Number of compressions ____________________________________

Post Run Questionnaire Subject # ___________ Condition_______________ Date of Experiment____________ Circle the ONE number that best represents your feelings during the experiment (do not circle multiple numbers nor place a circle between numbers). The text represents sentiments correlated with both ends and the center of the scale. 1. My actions during the experiment were driven by:

1A. Briefly describe any external cues or memorized plans (if used). 2. What best describes your utilization of feedback (FB) during the experiment?

2A. Please list specific examples of the feedback you employed: 3. What best describes your goal setting during the experiment?

1 4 2 3 5 10 6 7 8 9

1 4 2 3 5 10 6 7 8 9

1 4 2 3 5 10 6 7 8 9

Trial & Error Things I observed Memorized Plan

I didn’t get FB

FB helped a little

FB improved many actions

I focused on both CPR and the memorization task

I did not consciously consider goals

4. Describe the time pressure you felt while performing the task

5. How many minutes passed from the start of the scenario until you were told to quit ?

6. Describe your ability to understand what was occurring as a result of your actions.

7. Two lists of words were on the front of the victim’s shirt. Write as many

words as you remember.

8. What was the category that could describe most of the words? ____________ 9. What word was written in red? _______________

1 4 2 3 5 10 6 7 8 9

1 4 2 3 5 10 6 7 8 9

1 4 2 3 5 10 6 7 8 9

I focused solely on the CPR goal

Enormous time pressure

Pressured, but just enough time

Time wasn’t a factor

1 min 10 min

I had little understanding of the results of my actions

I could sometimes see and understand what was resulting

The results of all actions were clear to me

5 min

Video-cued Recall Form

Subject # ___________ Condition_______________ Date of Experiment____________ GOTO: first walking in room. Stop video as they approach victim/box.

Q. As you approached the victim, what were your thoughts? Describe the actions you were planning to take.

GOTO: point where approach patient, see and remove shirt.

Q. Did you see the words on the shirt and think about the memorization task? Did you consciously make a decision about studying the list or moving on?

GOTO: transition between breaths and compressions (1st cycle)

Q. Why did you move to chest compressions at this point? GOTO: last compression of the second cycle of compressions.

Q. Did you think about your future actions at this point? What were your plans?

)������.�%�!�����������0�����������)������.�%�!�����������0�����������)������.�%�!�����������0�����������)������.�%�!�����������0���������������

Participant Questionnaire Name ________________________________ phone __________________ Gender M / F Age _________ Year in school __________ Major __________ Cardiopulmonary Resuscitation (CPR) Have you ever been trained in CPR ? YES NO

month/year of initial training ___________ mo/yr. most recent training ________ Have you ever been certified in CPR YES NO Are you currently certified (AHA, Am Red Cross) in CPR ? YES NO Automated External Defibrillator(AED) Have you ever been trained to use an AED? YES NO If yes, when were you most recently trained?________________ The questions below are designed to assess your expertise. Most participants should not be able to answer these questions. If you don’t know the answer, simply place an “X” on the line. What are the ABCs of CPR? ________________________________________________ Briefly describe the purpose and technique in performing a “jaw thrust” ________________________________________________________________________________________________________________________________________________________________________________________________________________________ Briefly describe V-fib _____________________________________________________ _______________________________________________________________________

)������.��!�������������)������.��!�������������)������.��!�������������)������.��!�����������������

Instructions JITS w/Device Groups

Thank you for participating in this study. Today you will perform Cardiopulmonary Resuscitation (CPR) to the best of your ability on a mannequin designed for CPR training. I have your consent form. You have the option to withdraw from this experiment now, or at any time during the procedure. Should you wish to withdraw during the experiment, simply tell the researchers in the room you would like to stop. When the scenario begins you will find a mannequin representing the victim. The victim requires CPR. There is no need to evaluate the victim for responsiveness, as we have determined for you that the patient is unconscious and non-responsive. Assume a call to 911 has already been placed and paramedics will be arriving shortly to relieve you. Continue to perform CPR until you are told to stop. When you approach the victim, you will notice a white box with a red cross. This box contains supplies that you may use to perform your task. In addition to the supplies you will be using, the box also houses a video screen that will provide animated instructions and real-time feedback. Simply touch the screen to begin and the system will walk you through the CPR procedure step-by-step. The system is able to monitor your actions and offer feedback to help improve your performance. We encourage you to follow instructions as closely as possible. Please perform all tasks as if this was a real victim. The components that you will come in contact with are sterilized and/or replaced for each participant. Therefore, there is no need to mock any actions, or say you “would” perform an action, simply perform the action fully on the mannequin. Lastly, you may encounter a list of words at sometime during the procedure. We emphasize here that we want you to devote all your efforts to perform CPR to the best of your ability. CPR is very demanding, and we don’t expect participants to have much success memorizing the list. However, if you feel you can, try to memorize as many words as possible. That concludes your prebriefing instructions. Do you have any questions? Once you start, no one can answer any questions for you.

JITS JITS

Instructions Non JITS Groups

Thank you for participating in this study. Today you will perform Cardiopulmonary Resuscitation (CPR) to the best of your ability on a mannequin designed for CPR training. I have your consent form. You have the option to withdraw from this experiment now, or at any time during the procedure. Should you wish to withdraw during the experiment, simply tell the researchers in the room you would like to stop. When the scenario begins you will find a mannequin representing the victim. The victim requires CPR. There is no need to evaluate the victim for responsiveness, as we have determined for you that the patient is unconscious and non-responsive. Assume a call to 911 has already been placed and paramedics will be arriving shortly to relieve you. Continue to perform CPR until you are told to stop. When you approach the victim, you will notice a white box with a red cross. This box contains supplies that you may use to perform your task. Please perform all tasks as if this was a real victim. The components that you will come in contact with are sterilized and/or replaced for each participant. Therefore, there is no need to mock any actions, or say you “would” perform an action, simply perform the action fully on the mannequin. Lastly, you may encounter a list of words at sometime during the procedure. We emphasize here that we want you to devote all your efforts to perform CPR to the best of your ability. CPR is very demanding, and we don’t expect participants to have much success memorizing the list. However, if you feel you can, try to memorize as many words as possible. That concludes your prebriefing instructions. Do you have any questions? Once you start, no one can answer any questions for you.

NON JITS NON JITS

)������.�'�!�������������"����������)������.�'�!�������������"����������)������.�'�!�������������"����������)������.�'�!�������������"��������������

% "�* ��#������ �+����������,�-�Chi Square is not appropriate for 2 x 2 contingency table and also has limitations with small cell sizes (n < 5). Therefore, Fishers Exact test was utilized. Class 1 represents GRE-DEV while class 2 = GRE-NO.

Gp1 is the number of subjects able to register non-zero values for the physiological data. Gp 2 is the number of participants unable to perform above the noise threshold of the sensors. Therefore, performance is equivalent to no action at all.

The small p-value (0.0005 one-tail, 0.001 two-tail) indicates a significant difference in the distribution of these groups.

��������� ��� ��� �

�������� 17�

3�

20�

�

�������� 6�

14�

20�

�

� 23�

17�

40�

�

� �����������

� � ����������� 0.0005�

� Reset������������� 0.9999

�

� � ������������ 0.0010�

�

Applet available at : http://www.unc.edu/~preacher/fisher/fisher.htm

$�.�'��������� ��(�����Multivariate Tests(d)

Effect Value F

Hypothesis

df Error df Sig.

Partial Eta

Squared

Noncent.

Parameter

Intercept Pillai's Trace .963 439.425(b

) 5.000 84.000 .000 .963 2197.127

Wilks' Lambda .037 439.425(b

) 5.000 84.000 .000 .963 2197.127

Hotelling's Trace 26.156 439.425(b

) 5.000 84.000 .000 .963 2197.127

Roy's Largest Root

26.156 439.425(b) 5.000 84.000 .000 .963 2197.

127

group Pillai's Trace 1.317 8.540 20.00

0 348.000 .000 .329 170.804

Wilks' Lambda .126 12.117 20.00

0 279.54

6 .000 .404 189.538

Hotelling's Trace 3.824 15.775 20.00

0 330.000 .000 .489 315.506

Roy's Largest Root

2.957 51.454(c) 5.000 87.000 .000 .747 257.268

a Computed using alpha = .05 b Exact statistic c The statistic is an upper bound on F that yields a lower bound on the significance level. d Design: Intercept+group

Levene's Test of Equality of Error Variances(a)

F df1 df2 Sig. senBR 6.709 4 88 .000 senCC 4.797 4 88 .002 Rate 18.500 4 88 .000 CCdepth 12.360 4 88 .000 avgBRvolCyc 1.699 4 88 .157

Tests the null hypothesis that the error variance of the dependent variable is equal across groups. a Design: Intercept+group

Tests of Between-Subjects Effects

Source Dependent Variable

Type III Sum of

Squares df Mean Square F Sig. Partial Eta Squared

Corrected Model

senBR 47.489(b) 4 11.872 17.276 .000 .440

senCC 1346.311(c) 4 336.578 28.093 .000 .561 Rate 32734.677(d) 4 8183.669 24.000 .000 .522 CCdepth 4.141(e) 4 1.035 4.877 .001 .181 avgBRvolCyc 14863636.38

7(f) 4 3715909.097 30.854 .000 .584

Intercept senBR 393.900 1 393.900 573.189 .000 .867 senCC 14292.798 1 14292.798 1192.984 .000 .931 Rate 564126.667 1 564126.667 1654.403 .000 .949 CCdepth 226.232 1 226.232 1065.857 .000 .924 avgBRvolCyc 49997669.76

4 1 49997669.764 415.136 .000 .825

group senBR 47.489 4 11.872 17.276 .000 .440 senCC 1346.311 4 336.578 28.093 .000 .561 Rate 32734.677 4 8183.669 24.000 .000 .522 CCdepth 4.141 4 1.035 4.877 .001 .181 avgBRvolCyc 14863636.

387 4 3715909.097 30.854 .000 .584 Error senBR 60.474 8

8 .687 senCC 1054.303 8

8 11.981 Rate 30006.680 8

8 340.985 CCdepth 18.678 8

8 .212 avgBRvolCyc 10598441.46

3 88 120436.835

Total senBR 495.946 93

senCC 16656.798 93

Rate 630405.580 93

CCdepth 251.289 93

avgBRvolCyc 75422430.611

93

Corrected Total

senBR 107.964 92

senCC 2400.614 92

Rate 62741.357 92

CCdepth 22.819 92

avgBRvolCyc 25462077.850

92

a Computed using alpha = .05 b R Squared = .440 (Adjusted R Squared = .414) c R Squared = .561 (Adjusted R Squared = .541) d R Squared = .522 (Adjusted R Squared = .500) e R Squared = .181 (Adjusted R Squared = .144) f R Squared = .584 (Adjusted R Squared = .565)

Multiple Comparisons Dependent Variable

(I) group

(J) group

Mean Difference (I-

J) Std. Error Sig. 95% Confidence Interval

Lower Bound

Upper Bound

senBR Scheffe 1.00 2.00 -1.9997(*) .27675 .000 -2.8705 -1.1289 -1.1345(*) .26896 .003 -1.9808 -.2882 4.00 -1.8586(*) .27267 .000 -2.7166 -1.0007 5.00 -1.5174(*) .26557 .000 -2.3530 -.6817 2.00 1.00 1.9997(*) .27675 .000 1.1289 2.8705 3.00 .8651 .27675 .052 -.0057 1.7360 4.00 .1410 .28036 .992 -.7411 1.0232 5.00 .4823 .27347 .543 -.3782 1.3428 3.00 1.00 1.1345(*) .26896 .003 .2882 1.9808 2.00 -.8651 .27675 .052 -1.7360 .0057 4.00 -.7241 .27267 .143 -1.5821 .1339 5.00 -.3828 .26557 .722 -1.2185 .4528 4.00 1.00 1.8586(*) .27267 .000 1.0007 2.7166 2.00 -.1410 .28036 .992 -1.0232 .7411 3.00 .7241 .27267 .143 -.1339 1.5821 5.00 .3413 .26933 .807 -.5062 1.1888 5.00 1.00 1.5174(*) .26557 .000 .6817 2.3530 2.00 -.4823 .27347 .543 -1.3428 .3782 3.00 .3828 .26557 .722 -.4528 1.2185 4.00 -.3413 .26933 .807 -1.1888 .5062 Dunnett

T3 1.00 2.00 -1.9997(*) .35126 .000 -3.0479 -.9514

3.00 -1.1345(*) .31999 .013 -2.0967 -.1724 4.00 -1.8586(*) .30936 .000 -2.7949 -.9224 5.00 -1.5174(*) .29987 .000 -2.4311 -.6037 2.00 1.00 1.9997(*) .35126 .000 .9514 3.0479 3.00 .8651(*) .27638 .036 .0358 1.6945 4.00 .1410 .26399 1.000 -.6565 .9386 5.00 .4823 .25280 .468 -.2870 1.2517 3.00 1.00 1.1345(*) .31999 .013 .1724 2.0967 2.00 -.8651(*) .27638 .036 -1.6945 -.0358 4.00 -.7241(*) .22069 .023 -1.3810 -.0672 5.00 -.3828 .20718 .505 -1.0007 .2350 4.00 1.00 1.8586(*) .30936 .000 .9224 2.7949 2.00 -.1410 .26399 1.000 -.9386 .6565 3.00 .7241(*) .22069 .023 .0672 1.3810 5.00 .3413 .19034 .545 -.2255 .9081 5.00 1.00 1.5174(*) .29987 .000 .6037 2.4311 2.00 -.4823 .25280 .468 -1.2517 .2870 3.00 .3828 .20718 .505 -.2350 1.0007 4.00 -.3413 .19034 .545 -.9081 .2255 senCC Scheffe 1.00 2.00 -9.5041(*) 1.15556 .000 -13.1402 -5.8680 3.00 -8.2703(*) 1.12300 .000 -11.8039 -4.7366

4.00 -9.2843(*) 1.13849 .000 -12.8667 -5.7019 5.00 -10.1596(*) 1.10887 .000 -13.6488 -6.6705 2.00 1.00 9.5041(*) 1.15556 .000 5.8680 13.1402 3.00 1.2338 1.15556 .887 -2.4022 4.8699 4.00 .2198 1.17062 1.000 -3.4637 3.9033 5.00 -.6555 1.14183 .988 -4.2484 2.9374 3.00 1.00 8.2703(*) 1.12300 .000 4.7366 11.8039 2.00 -1.2338 1.15556 .887 -4.8699 2.4022 4.00 -1.0140 1.13849 .939 -4.5964 2.5683 5.00 -1.8894 1.10887 .577 -5.3786 1.5998 4.00 1.00 9.2843(*) 1.13849 .000 5.7019 12.8667 2.00 -.2198 1.17062 1.000 -3.9033 3.4637 3.00 1.0140 1.13849 .939 -2.5683 4.5964 5.00 -.8753 1.12456 .962 -4.4139 2.6632 5.00 1.00 10.1596(*) 1.10887 .000 6.6705 13.6488 2.00 .6555 1.14183 .988 -2.9374 4.2484 3.00 1.8894 1.10887 .577 -1.5998 5.3786 4.00 .8753 1.12456 .962 -2.6632 4.4139 Dunnett

T3 1.00 2.00 -9.5041(*) 1.56385 .000 -14.1679 -4.8403

3.00 -8.2703(*) 1.20524 .000 -11.9255 -4.6151 4.00 -9.2843(*) 1.21374 .000 -12.9608 -5.6078 5.00 -10.1596(*) 1.20531 .000 -13.8141 -6.5052 2.00 1.00 9.5041(*) 1.56385 .000 4.8403 14.1679 3.00 1.2338 1.25689 .974 -2.6248 5.0925 4.00 .2198 1.26504 1.000 -3.6580 4.0976 5.00 -.6555 1.25696 1.000 -4.5135 3.2025 3.00 1.00 8.2703(*) 1.20524 .000 4.6151 11.9255 2.00 -1.2338 1.25689 .974 -5.0925 2.6248 4.00 -1.0140 .77930 .874 -3.3327 1.3046 5.00 -1.8894 .76611 .162 -4.1609 .3821 4.00 1.00 9.2843(*) 1.21374 .000 5.6078 12.9608 2.00 -.2198 1.26504 1.000 -4.0976 3.6580 3.00 1.0140 .77930 .874 -1.3046 3.3327 5.00 -.8753 .77941 .944 -3.1909 1.4403 5.00 1.00 10.1596(*) 1.20531 .000 6.5052 13.8141 2.00 .6555 1.25696 1.000 -3.2025 4.5135 3.00 1.8894 .76611 .162 -.3821 4.1609 4.00 .8753 .77941 .944 -1.4403 3.1909 Rate Scheffe 1.00 2.00 -32.2662(*) 6.16478 .000 -51.6643 -12.8681 3.00 -52.1210(*) 5.99109 .000 -70.9725 -33.2694 4.00 -40.6254(*) 6.07373 .000 -59.7370 -21.5137 5.00 -47.6579(*) 5.91573 .000 -66.2724 -29.0435 2.00 1.00 32.2662(*) 6.16478 .000 12.8681 51.6643 3.00 -19.8547(*) 6.16478 .042 -39.2529 -.4566 4.00 -8.3592 6.24512 .774 -28.0101 11.2918 5.00 -15.3917 6.09157 .182 -34.5594 3.7760 3.00 1.00 52.1210(*) 5.99109 .000 33.2694 70.9725 2.00 19.8547(*) 6.16478 .042 .4566 39.2529 4.00 11.4956 6.07373 .470 -7.6160 30.6072

5.00 4.4630 5.91573 .966 -14.1514 23.0775 4.00 1.00 40.6254(*) 6.07373 .000 21.5137 59.7370 2.00 8.3592 6.24512 .774 -11.2918 28.0101 3.00 -11.4956 6.07373 .470 -30.6072 7.6160 5.00 -7.0326 5.99941 .848 -25.9103 11.8452 5.00 1.00 47.6579(*) 5.91573 .000 29.0435 66.2724 2.00 15.3917 6.09157 .182 -3.7760 34.5594 3.00 -4.4630 5.91573 .966 -23.0775 14.1514 4.00 7.0326 5.99941 .848 -11.8452 25.9103 Dunnett

T3 1.00 2.00 -32.2662(*) 8.87102 .009 -58.7902 -5.7422

3.00 -52.1210(*) 7.51938 .000 -75.3041 -28.9378 4.00 -40.6254(*) 7.68727 .000 -64.1647 -17.0860 5.00 -47.6579(*) 7.21941 .000 -70.2703 -25.0455 2.00 1.00 32.2662(*) 8.87102 .009 5.7422 58.7902 3.00 -19.8547(*) 5.80733 .023 -37.7281 -1.9814 4.00 -8.3592 6.02313 .824 -26.7250 10.0067 5.00 -15.3917 5.41330 .095 -32.5007 1.7173 3.00 1.00 52.1210(*) 7.51938 .000 28.9378 75.3041 2.00 19.8547(*) 5.80733 .023 1.9814 37.7281 4.00 11.4956(*) 3.75823 .041 .2865 22.7047 5.00 4.4630 2.67396 .638 -3.6576 12.5837 4.00 1.00 40.6254(*) 7.68727 .000 17.0860 64.1647 2.00 8.3592 6.02313 .824 -10.0067 26.7250 3.00 -11.4956(*) 3.75823 .041 -22.7047 -.2865 5.00 -7.0326 3.11497 .268 -16.6186 2.5535 5.00 1.00 47.6579(*) 7.21941 .000 25.0455 70.2703 2.00 15.3917 5.41330 .095 -1.7173 32.5007 3.00 -4.4630 2.67396 .638 -12.5837 3.6576 4.00 7.0326 3.11497 .268 -2.5535 16.6186 CCdepth Scheffe 1.00 2.00 -.1413 .15381 .932 -.6253 .3427 3.00 -.5179(*) .14947 .023 -.9883 -.0476 4.00 -.3751 .15154 .200 -.8520 .1017 5.00 -.5266(*) .14759 .017 -.9911 -.0622 2.00 1.00 .1413 .15381 .932 -.3427 .6253 3.00 -.3766 .15381 .209 -.8606 .1073 4.00 -.2338 .15581 .690 -.7241 .2565 5.00 -.3853 .15198 .180 -.8635 .0929 3.00 1.00 .5179(*) .14947 .023 .0476 .9883 2.00 .3766 .15381 .209 -.1073 .8606 4.00 .1428 .15154 .925 -.3340 .6196 5.00 -.0087 .14759 1.000 -.4731 .4557 4.00 1.00 .3751 .15154 .200 -.1017 .8520 2.00 .2338 .15581 .690 -.2565 .7241 3.00 -.1428 .15154 .925 -.6196 .3340 5.00 -.1515 .14968 .905 -.6225 .3195 5.00 1.00 .5266(*) .14759 .017 .0622 .9911 2.00 .3853 .15198 .180 -.0929 .8635 3.00 .0087 .14759 1.000 -.4557 .4731 4.00 .1515 .14968 .905 -.3195 .6225

Dunnett T3

1.00 2.00 -.1413 .20553 .998 -.7737 .4911

3.00 -.5179 .20622 .166 -1.1516 .1158 4.00 -.3751 .20716 .538 -1.0109 .2606 5.00 -.5266 .20105 .138 -1.1499 .0966 2.00 1.00 .1413 .20553 .998 -.4911 .7737 3.00 -.3766(*) .09965 .006 -.6736 -.0797 4.00 -.2338 .10159 .232 -.5372 .0695 5.00 -.3853(*) .08845 .001 -.6501 -.1206 3.00 1.00 .5179 .20622 .166 -.1158 1.1516 2.00 .3766(*) .09965 .006 .0797 .6736 4.00 .1428 .10298 .828 -.1636 .4492 5.00 -.0087 .09004 1.000 -.2769 .2595 4.00 1.00 .3751 .20716 .538 -.2606 1.0109 2.00 .2338 .10159 .232 -.0695 .5372 3.00 -.1428 .10298 .828 -.4492 .1636 5.00 -.1515 .09219 .655 -.4273 .1243 5.00 1.00 .5266 .20105 .138 -.0966 1.1499 2.00 .3853(*) .08845 .001 .1206 .6501 3.00 .0087 .09004 1.000 -.2595 .2769 4.00 .1515 .09219 .655 -.1243 .4273 avgBRvolCyc

Scheffe 1.00 2.00 -857.9926(*) 115.85897 .000 -

1222.5548 -493.4304

3.00 -335.8544 112.59468 .073 -690.1451 18.4364

4.00 -742.9189(*) 114.14778 .000 -

1102.0967 -383.7412

5.00 -1110.1317(*) 111.17834 .000 -

1459.9658 -760.2976

2.00 1.00 857.9926(*) 115.85897 .000 493.4304 1222.5548

3.00 522.1383(*) 115.85897 .001 157.5761 886.7004

4.00 115.0737 117.36888 .915 -254.2396 484.3869

5.00 -252.1391 114.48302 .311 -612.3717 108.0935

3.00 1.00 335.8544 112.59468 .073 -18.4364 690.1451

2.00 -522.1383(*) 115.85897 .001 -886.7004 -157.5761

4.00 -407.0646(*) 114.14778 .017 -766.2423 -47.8869

5.00 -774.2774(*) 111.17834 .000 -

1124.1114 -424.4433

4.00 1.00 742.9189(*) 114.14778 .000 383.7412 1102.0967

2.00 -115.0737 117.36888 .915 -484.3869 254.2396

3.00 407.0646(*) 114.14778 .017 47.8869 766.2423

5.00 -367.2128(*) 112.75095 .038 -721.9952 -12.4303

5.00 1.00 1110.1317(*) 111.17834 .000 760.2976 1459.9658

2.00 252.1391 114.48302 .311 -108.0935 612.3717

3.00 774.2774(*) 111.17834 .000 424.4433 1124.1114

4.00 367.2128(*) 112.75095 .038 12.4303 721.9952

Dunnett T3

1.00 2.00 -857.9926(*) 104.04305 .000 -

1176.5306 -539.4546

3.00 -335.8544(*) 86.30985 .005 -594.7625 -76.9462

4.00 -742.9189(*) 98.37015 .000 -

1041.5325 -444.3054

5.00 -1110.1317(*) 105.62110 .000 -

1429.4408 -790.8226

2.00 1.00 857.9926(*) 104.04305 .000 539.4546 1176.5306

3.00 522.1383(*) 117.91064 .001 168.7549 875.5216

4.00 115.0737 127.00478 .986 -264.3643 494.5117

5.00 -252.1391 132.70020 .467 -646.9770 142.6988

3.00 1.00 335.8544(*) 86.30985 .005 76.9462 594.7625

2.00 -522.1383(*) 117.91064 .001 -875.5216 -168.7549

4.00 -407.0646(*) 112.93648 .010 -743.8516 -70.2775

5.00 -774.2774(*) 119.30540 .000 -

1129.1087 -419.4461

4.00 1.00 742.9189(*) 98.37015 .000 444.3054 1041.5325

2.00 -115.0737 127.00478 .986 -494.5117 264.3643

3.00 407.0646(*) 112.93648 .010 70.2775 743.8516

5.00 -367.2128 128.30072 .065 -748.2371 13.8115

5.00 1.00 1110.1317(*) 105.62110 .000 790.8226 1429.4408

2.00 252.1391 132.70020 .467 -142.6988 646.9770

3.00 774.2774(*) 119.30540 .000 419.4461 1129.1087

4.00 367.2128 128.30072 .065 -13.8115 748.2371

Based on observed means. * The mean difference is significant at the .05 level.

����/���������������� Cohen’s Kappa calculation”

1900

3521

0223

Scram

Scram

Opp

TactOpp

Tact

Rater 2

Rat

er 1

24

19

56

25

2254 100

Expected Frequencies

ef =Row total * column total

Overall total

ef1 = 6

ef2 = 30.4

ef3 = 4.18

SumDiag = 96

SumEF = 40.6

Kappa Calculation

96 – 40.6

κ =N - SumEF

SumDiag - SumEF

κ =100 – 40.6

κ > 0.70 considered acceptable

= 0.899

MANOVA for COCOM coding Descriptive Statistics

group Mean Std. Deviation N 1.00 .8985 .12223 20 2.00 .2980 .14753 20 3.00 .1870 .22679 20 4.00 .1710 .11805 20 5.00 .1580 .09373 20

perSCRAM

Total .3425 .31911 100 1.00 .0455 .06886 20 2.00 .6595 .15357 20 3.00 .1660 .07783 20 4.00 .6190 .16370 20 5.00 .6010 .09580 20

perOPP

Total .4182 .28473 100 1.00 .0555 .07776 20 2.00 .0425 .08491 20 3.00 .6450 .25078 20 4.00 .2110 .17976 20 5.00 .2400 .12053 20

perTACT

Total .2388 .26770 100 Multivariate Tests(c)

Effect Value F Hypothesis df Error df Sig. Pillai's Trace 1.000 333353.53

3(a) 3.000 93.000 .000

Wilks' Lambda .000 333353.533(a) 3.000 93.000 .000

Hotelling's Trace 10753.340 333353.53

3(a) 3.000 93.000 .000

Intercept

Roy's Largest Root 10753.340 333353.53

3(a) 3.000 93.000 .000

Pillai's Trace 1.525 24.561 12.000 285.000 .000 Wilks' Lambda .049 43.600 12.000 246.346 .000

Hotelling's Trace 7.718 58.958 12.000 275.000 .000

group

Roy's Largest Root 5.678 134.842(b) 4.000 95.000 .000

a Exact statistic b The statistic is an upper bound on F that yields a lower bound on the significance level. c Design: Intercept+group

Tests of Between-Subjects Effects

Source Dependent Variable

Type III Sum of Squares df

Mean Square F Sig.

Corrected Model perSCRAM 7.975(a) 4 1.994 89.924 .000 perOPP 6.689(b) 4 1.672 118.844 .000 perTACT 4.758(c) 4 1.190 48.360 .000 Intercept perSCRAM 11.731 1 11.731 529.085 .000 perOPP 17.489 1 17.489 1242.841 .000 perTACT 5.703 1 5.703 231.836 .000 group perSCRAM 7.975 4 1.994 89.924 .000 perOPP 6.689 4 1.672 118.844 .000 perTACT 4.758 4 1.190 48.360 .000 Error perSCRAM 2.106 95 .022 perOPP 1.337 95 .014 perTACT 2.337 95 .025 Total perSCRAM 21.812 100 perOPP 25.515 100 perTACT 12.797 100 Corrected Total perSCRAM 10.081 99 perOPP 8.026 99 perTACT 7.095 99

a R Squared = .791 (Adjusted R Squared = .782) b R Squared = .833 (Adjusted R Squared = .826) c R Squared = .671 (Adjusted R Squared = .657) Multiple Comparisons Scheffe

Dependent Variable (I) group (J) group

Mean Difference

(I-J) Std. Error Sig. 95% Confidence

Interval

Lower Bound

Upper

Bound

perSCRAM 1.00 2.00 .6005(*) .04709 .000 .4526 .7484 3.00 .7115(*) .04709 .000 .5636 .8594 4.00 .7275(*) .04709 .000 .5796 .8754 5.00 .7405(*) .04709 .000 .5926 .8884 2.00 1.00 -.6005(*) .04709 .000 -.7484 -

.4526 3.00 .1110 .04709 .244 -.0369 .2589 4.00 .1270 .04709 .132 -.0209 .2749 5.00 .1400 .04709 .074 -.0079 .2879 3.00 1.00 -.7115(*) .04709 .000 -.8594 -

.5636 2.00 -.1110 .04709 .244 -.2589 .0369 4.00 .0160 .04709 .998 -.1319 .1639 5.00 .0290 .04709 .984 -.1189 .1769 4.00 1.00 -.7275(*) .04709 .000 -.8754 -

.5796

2.00 -.1270 .04709 .132 -.2749 .0209 3.00 -.0160 .04709 .998 -.1639 .1319 5.00 .0130 .04709 .999 -.1349 .1609 5.00 1.00 -.7405(*) .04709 .000 -.8884 -

.5926 2.00 -.1400 .04709 .074 -.2879 .0079 3.00 -.0290 .04709 .984 -.1769 .1189 4.00 -.0130 .04709 .999 -.1609 .1349 perOPP 1.00 2.00 -.6140(*) .03751 .000 -.7319 -

.4961 3.00 -.1205(*) .03751 .042 -.2384 -

.0026 4.00 -.5735(*) .03751 .000 -.6914 -

.4556 5.00 -.5555(*) .03751 .000 -.6734 -

.4376 2.00 1.00 .6140(*) .03751 .000 .4961 .7319 3.00 .4935(*) .03751 .000 .3756 .6114 4.00 .0405 .03751 .883 -.0774 .1584 5.00 .0585 .03751 .658 -.0594 .1764 3.00 1.00 .1205(*) .03751 .042 .0026 .2384 2.00 -.4935(*) .03751 .000 -.6114 -

.3756 4.00 -.4530(*) .03751 .000 -.5709 -

.3351 5.00 -.4350(*) .03751 .000 -.5529 -

.3171 4.00 1.00 .5735(*) .03751 .000 .4556 .6914 2.00 -.0405 .03751 .883 -.1584 .0774 3.00 .4530(*) .03751 .000 .3351 .5709 5.00 .0180 .03751 .994 -.0999 .1359 5.00 1.00 .5555(*) .03751 .000 .4376 .6734 2.00 -.0585 .03751 .658 -.1764 .0594 3.00 .4350(*) .03751 .000 .3171 .5529 4.00 -.0180 .03751 .994 -.1359 .0999 perTACT 1.00 2.00 .0130 .04960 .999 -.1428 .1688 3.00 -.5895(*) .04960 .000 -.7453 -

.4337 4.00 -.1555 .04960 .051 -.3113 .0003 5.00 -.1845(*) .04960 .011 -.3403 -

.0287 2.00 1.00 -.0130 .04960 .999 -.1688 .1428 3.00 -.6025(*) .04960 .000 -.7583 -

.4467 4.00 -.1685(*) .04960 .026 -.3243 -

.0127 5.00 -.1975(*) .04960 .005 -.3533 -

.0417 3.00 1.00 .5895(*) .04960 .000 .4337 .7453 2.00 .6025(*) .04960 .000 .4467 .7583 4.00 .4340(*) .04960 .000 .2782 .5898 5.00 .4050(*) .04960 .000 .2492 .5608 4.00 1.00 .1555 .04960 .051 -.0003 .3113 2.00 .1685(*) .04960 .026 .0127 .3243

3.00 -.4340(*) .04960 .000 -.5898 -.2782

5.00 -.0290 .04960 .987 -.1848 .1268 5.00 1.00 .1845(*) .04960 .011 .0287 .3403 2.00 .1975(*) .04960 .005 .0417 .3533 3.00 -.4050(*) .04960 .000 -.5608 -

.2492 4.00 .0290 .04960 .987 -.1268 .1848

Based on observed means. * The mean difference is significant at the .05 level.

$�.�'���������$������� Descriptive Statistics

group Mean Std. Deviation N perSCRAM 1.00 .8985 .12223 20 2.00 .2980 .14753 20 3.00 .1870 .22679 20 4.00 .1710 .11805 20 5.00 .1580 .09373 20 Total .3425 .31911 100 perOPP 1.00 .0455 .06886 20 2.00 .6595 .15357 20 3.00 .1660 .07783 20 4.00 .6190 .16370 20 5.00 .6010 .09580 20 Total .4182 .28473 100 perTACT 1.00 .0555 .07776 20 2.00 .0425 .08491 20 3.00 .6450 .25078 20 4.00 .2110 .17976 20 5.00 .2400 .12053 20 Total .2388 .26770 100

Multivariate Tests(c)

Effect Value F Hypothesis df Error df Sig. Pillai's Trace 1.000 333353.53

3(a) 3.000 93.000 .000

Wilks' Lambda .000 333353.533(a) 3.000 93.000 .000

Hotelling's Trace 10753.340 333353.53

3(a) 3.000 93.000 .000

Intercept

Roy's Largest Root 10753.340 333353.53

3(a) 3.000 93.000 .000

Pillai's Trace 1.525 24.561 12.000 285.000 .000 Wilks' Lambda .049 43.600 12.000 246.346 .000 Hotelling's Trace 7.718 58.958 12.000 275.000 .000

group

Roy's Largest Root 5.678 134.842(b) 4.000 95.000 .000

a Exact statistic b The statistic is an upper bound on F that yields a lower bound on the significance level. c Design: Intercept+group

Tests of Between-Subjects Effects

Source Dependent Variable

Type III Sum of

Squares df Mean Square F Sig. Corrected Model perSCRAM 7.975(a) 4 1.994 89.924 .000 perOPP 6.689(b) 4 1.672 118.844 .000 perTACT 4.758(c) 4 1.190 48.360 .000 Intercept perSCRAM 11.731 1 11.731 529.085 .000 perOPP 17.489 1 17.489 1242.841 .000 perTACT 5.703 1 5.703 231.836 .000 group perSCRAM 7.975 4 1.994 89.924 .000 perOPP 6.689 4 1.672 118.844 .000 perTACT 4.758 4 1.190 48.360 .000 Error perSCRAM 2.106 95 .022 perOPP 1.337 95 .014 perTACT 2.337 95 .025 Total perSCRAM 21.812 100 perOPP 25.515 100 perTACT 12.797 100 Corrected Total perSCRAM 10.081 99 perOPP 8.026 99 perTACT 7.095 99

a R Squared = .791 (Adjusted R Squared = .782) b R Squared = .833 (Adjusted R Squared = .826) c R Squared = .671 (Adjusted R Squared = .657) Multiple Comparisons Scheffe

Dependent Variable (I) group

(J) group

Mean Difference

(I-J) Std. Error Sig. 95% Confidence Interval

Lower Bound

Upper Bound

perSCRAM 1.00 2.00 .6005(*) .04709 .000 .4526 .7484 3.00 .7115(*) .04709 .000 .5636 .8594 4.00 .7275(*) .04709 .000 .5796 .8754 5.00 .7405(*) .04709 .000 .5926 .8884 2.00 1.00 -.6005(*) .04709 .000 -.7484 -.4526 3.00 .1110 .04709 .244 -.0369 .2589 4.00 .1270 .04709 .132 -.0209 .2749 5.00 .1400 .04709 .074 -.0079 .2879 3.00 1.00 -.7115(*) .04709 .000 -.8594 -.5636 2.00 -.1110 .04709 .244 -.2589 .0369 4.00 .0160 .04709 .998 -.1319 .1639 5.00 .0290 .04709 .984 -.1189 .1769 4.00 1.00 -.7275(*) .04709 .000 -.8754 -.5796 2.00 -.1270 .04709 .132 -.2749 .0209 3.00 -.0160 .04709 .998 -.1639 .1319 5.00 .0130 .04709 .999 -.1349 .1609

5.00 1.00 -.7405(*) .04709 .000 -.8884 -.5926 2.00 -.1400 .04709 .074 -.2879 .0079 3.00 -.0290 .04709 .984 -.1769 .1189 4.00 -.0130 .04709 .999 -.1609 .1349 perOPP 1.00 2.00 -.6140(*) .03751 .000 -.7319 -.4961 3.00 -.1205(*) .03751 .042 -.2384 -.0026 4.00 -.5735(*) .03751 .000 -.6914 -.4556 5.00 -.5555(*) .03751 .000 -.6734 -.4376 2.00 1.00 .6140(*) .03751 .000 .4961 .7319 3.00 .4935(*) .03751 .000 .3756 .6114 4.00 .0405 .03751 .883 -.0774 .1584 5.00 .0585 .03751 .658 -.0594 .1764 3.00 1.00 .1205(*) .03751 .042 .0026 .2384 2.00 -.4935(*) .03751 .000 -.6114 -.3756 4.00 -.4530(*) .03751 .000 -.5709 -.3351 5.00 -.4350(*) .03751 .000 -.5529 -.3171 4.00 1.00 .5735(*) .03751 .000 .4556 .6914 2.00 -.0405 .03751 .883 -.1584 .0774 3.00 .4530(*) .03751 .000 .3351 .5709 5.00 .0180 .03751 .994 -.0999 .1359 5.00 1.00 .5555(*) .03751 .000 .4376 .6734 2.00 -.0585 .03751 .658 -.1764 .0594 3.00 .4350(*) .03751 .000 .3171 .5529 4.00 -.0180 .03751 .994 -.1359 .0999 perTACT 1.00 2.00 .0130 .04960 .999 -.1428 .1688 3.00 -.5895(*) .04960 .000 -.7453 -.4337 4.00 -.1555 .04960 .051 -.3113 .0003 5.00 -.1845(*) .04960 .011 -.3403 -.0287 2.00 1.00 -.0130 .04960 .999 -.1688 .1428 3.00 -.6025(*) .04960 .000 -.7583 -.4467 4.00 -.1685(*) .04960 .026 -.3243 -.0127 5.00 -.1975(*) .04960 .005 -.3533 -.0417 3.00 1.00 .5895(*) .04960 .000 .4337 .7453 2.00 .6025(*) .04960 .000 .4467 .7583 4.00 .4340(*) .04960 .000 .2782 .5898 5.00 .4050(*) .04960 .000 .2492 .5608 4.00 1.00 .1555 .04960 .051 -.0003 .3113 2.00 .1685(*) .04960 .026 .0127 .3243 3.00 -.4340(*) .04960 .000 -.5898 -.2782 5.00 -.0290 .04960 .987 -.1848 .1268 5.00 1.00 .1845(*) .04960 .011 .0287 .3403 2.00 .1975(*) .04960 .005 .0417 .3533 3.00 -.4050(*) .04960 .000 -.5608 -.2492 4.00 .0290 .04960 .987 -.1268 .1848

Based on observed means. * The mean difference is significant at the .05 level.

$�.�'�����!� �/����0�� ����������������� ��� � Descriptive Statistics

grpnum Mean Std. Deviation N 1.00 1.6000 1.46539 20 2.00 7.8000 1.05631 20 3.00 7.8500 1.49649 20 4.00 8.4500 .88704 20 5.00 8.6500 .48936 20

Yprotocol

Total 6.8700 2.89428 100 1.00 6.0500 1.66938 20 2.00 5.7500 1.29269 20 3.00 8.2500 1.11803 20 4.00 6.7500 1.74341 20 5.00 7.8500 1.08942 20

action

Total 6.9300 1.69524 100 1.00 2.8500 2.15883 20 2.00 7.8000 1.85245 20 3.00 2.4500 2.39462 20 4.00 7.0000 1.89181 20 5.00 7.6000 2.21003 20

FB

Total 5.5400 3.16042 100 1.00 6.5500 1.90498 20 2.00 6.0500 .99868 20 3.00 6.4000 2.01050 20 4.00 5.6500 1.13671 20 5.00 6.6500 1.84320 20

goal

Total 6.2600 1.64298 100 1.00 7.0000 2.65568 20 2.00 6.8000 2.28496 20 3.00 7.5500 2.01246 20 4.00 7.7500 2.22131 20 5.00 7.8000 2.52566 20

Tpressure

Total 7.3800 2.33887 100 1.00 4.7500 2.12442 20 2.00 4.6500 1.63111 20 3.00 3.8000 1.00525 20 4.00 4.8000 1.57614 20 5.00 3.8000 .76777 20

time

Total 4.3600 1.54082 100 1.00 3.8500 2.34577 20 2.00 7.1500 1.95408 20 3.00 5.6000 2.18608 20 4.00 7.3000 2.55672 20

outcome

5.00 7.3000 2.17885 20

Total 6.2400 2.59417 100 1.00 4.6500 2.05900 20 2.00 1.7500 3.00657 20 3.00 2.2000 1.96281 20 4.00 1.2000 1.90843 20 5.00 2.8500 2.87045 20

words

Total 2.5300 2.64558 100 Multivariate Tests(c)

Effect Value F Hypothesis df Error df Sig. Pillai's Trace .989 985.561(a) 8.000 88.000 .000 Wilks' Lambda .011 985.561(a) 8.000 88.000 .000 Hotelling's Trace 89.596 985.561(a) 8.000 88.000 .000

Intercept

Roy's Largest Root 89.596 985.561(a) 8.000 88.000 .000

Pillai's Trace 1.618 7.730 32.000 364.000 .000 Wilks' Lambda .047 13.191 32.000 326.123 .000 Hotelling's Trace 8.363 22.607 32.000 346.000 .000

grpnum

Roy's Largest Root 6.991 79.518(b) 8.000 91.000 .000

a Exact statistic b The statistic is an upper bound on F that yields a lower bound on the significance level. c Design: Intercept+grpnum Tests of Between-Subjects Effects

Source Dependent Variable

Type III Sum of Squares df Mean Square F Sig.

Corrected Model

Yprotocol 705.260(a) 4 176.315 135.026 .000

action 95.760(b) 4 23.940 12.049 .000 FB 565.340(c) 4 141.335 31.704 .000 goal 13.440(d) 4 3.360 1.258 .292 Tpressure 16.460(e) 4 4.115 .744 .564 time 21.140(f) 4 5.285 2.347 .060 outcome 183.940(g) 4 45.985 9.058 .000 words 141.660(h) 4 35.415 6.103 .000 Intercept Yprotocol 4719.690 1 4719.690 3614.434 .000 action 4802.490 1 4802.490 2417.147 .000 FB 3069.160 1 3069.160 688.477 .000 goal 3918.760 1 3918.760 1466.833 .000 Tpressure 5446.440 1 5446.440 985.359 .000 time 1900.960 1 1900.960 844.279 .000 outcome 3893.760 1 3893.760 766.965 .000 words 640.090 1 640.090 110.310 .000 grpnum Yprotocol 705.260 4 176.315 135.026 .000

action 95.760 4 23.940 12.049 .000 FB 565.340 4 141.335 31.704 .000 goal 13.440 4 3.360 1.258 .292 Tpressure 16.460 4 4.115 .744 .564 time 21.140 4 5.285 2.347 .060 outcome 183.940 4 45.985 9.058 .000 words 141.660 4 35.415 6.103 .000 Error Yprotocol 124.050 95 1.306 action 188.750 95 1.987 FB 423.500 95 4.458 goal 253.800 95 2.672 Tpressure 525.100 95 5.527 time 213.900 95 2.252 outcome 482.300 95 5.077 words 551.250 95 5.803 Total Yprotocol 5549.000 100 action 5087.000 100 FB 4058.000 100 goal 4186.000 100 Tpressure 5988.000 100 time 2136.000 100 outcome 4560.000 100 words 1333.000 100 Corrected Total

Yprotocol 829.310 99

action 284.510 99 FB 988.840 99 goal 267.240 99 Tpressure 541.560 99 time 235.040 99 outcome 666.240 99 words 692.910 99

a R Squared = .850 (Adjusted R Squared = .844) b R Squared = .337 (Adjusted R Squared = .309) c R Squared = .572 (Adjusted R Squared = .554) d R Squared = .050 (Adjusted R Squared = .010) e R Squared = .030 (Adjusted R Squared = -.010) f R Squared = .090 (Adjusted R Squared = .052) g R Squared = .276 (Adjusted R Squared = .246) h R Squared = .204 (Adjusted R Squared = .171)

% "�* ��#������ �+���� ����� ����%������&��Class 1 represents CPR-DEV while class 2 = GRE-NO.

Gp1 is the number of subjects that received feedback (2 or more cases) during the scneario.

Gp 2 is the number of participants that are considered to not have received feedback.

The one-tail p value of 0.0267 suggests a significant difference.

��������� ��� ��� �

�������� 8�

12�

20�

�

�������� 15�

5�

20�

�

� 23�

17�

40�

�

� �����������

� � ����������� 0.0267�

� Reset������������� 0.9952

�

� � ������������ 0.0535�

��������� Status okay�

�

Applet available at : http://www.unc.edu/~preacher/fisher/fisher.htm

% "�* ��#������ �+��!���������1������2�

Class 1 represents all-DEV while class 2 = all-NODEV.

Gp1 is the number of subjects that performed the proper sequence.

Gp 2 is the number of participants failing to follow proper sequence.

��������� ��� ��� �

�������� 45�

15�

60�

�

�������� 15�

32�

47�

�

� 60�

47�

107�

�

� �����������

� � ����������� 0.0000�

� Reset������������� 0.9999

�

� � ������������ 0.0000�

��������� Status okay�

�

Class 1 represents all-DEV while class 2 = CPR-NO.

Gp1 is the number of subjects that performed the proper sequence.

Gp 2 is the number of participants failing to follow proper sequence.

��������� ��� ��� �

�������� 45�

8�

53�

�

�������� 15�

12�

27�

�

� 60�

20�

80�

�

� �����������

� � ����������� 0.0053�

� Reset������������� 0.9989

�

� � ������������ 0.0063�

��������� Status okay�

�

Both tests are significant

)������.��!�"# � ������1"������)������.��!�"# � ������1"������)������.��!�"# � ������1"������)������.��!�"# � ������1"����������

THIS TRAINING DOES NOT RESULT IN CPR CERTIFICATION

Thank you for volunteering to participate in our study. Today you will be taught a modified version of cardiopulmonary resuscitation or CPR. While what you are being taught today would be useful, if you are ever in a situation where CPR is needed, this training in no way makes you certified in CPR. We have changed steps and added equipment that alter this training from a certified course. The training will consist of a lecture time where the steps of CPR are demonstrated and explained. After this, all participates will be split up into 2 groups. While you are in your group you will be asked to “administer” CPR to a victim. You will then follow the steps of resuscitation you were taught in the demonstration. While in your group, if you are not the rescuer you will be asked to follow the steps on the handout to make sure all are followed. Name _______________________________________ (print) Signature____________________________________ Date_____________

)������.�(!�&�����/������������&/��)������.�(!�&�����/������������&/��)������.�(!�&�����/������������&/��)������.�(!�&�����/������������&/������

Courtesy of the American Heart Association, available at: http://circ.ahajournals.org/content/vol102/issue90001/images/large/hc33t0071002.jpeg

)������.��!������'����������)������.��!������'����������)������.��!������'����������)������.��!������'�������������������� �������������� ��� �������������� ��� �������������� ��� �������������� �����The value of a JITS system can be realized on multiple dimensions. It may

empower an intermediate user to perform at an expert level. It may enable a novice to

accomplish a formerly unachievable task. It may even support an expert in a degraded

state or help prevent errors.

Experts are not immune to the perturbations promulgated through innovation.

Manufacturing and the healthcare industry are two exemplar domains that have seen

significant technological infusion over the last fifty years. Rarely can technology be

introduced without notably impacting the human element of the system (Sheridan, 2002).

Requirements, procedures, and resources are often significantly altered as a result of new

technology. It is no surprise that knowledge requirements, skills, and attitudes are

impacted as well.

In emergency response, trained, experienced responders could also benefit

from JITS systems. The problems of CPR effectiveness and protocol adherence extend

beyond novice performers. Section 1 described the problems of improper ventilations,

inadequate compression rates and depths, and numerous accounts of rib and sternum

fractures (Abella et al., 2005; Lederer, Mair, Rabl, & Baubin, 2004; Myklebust, et al.,

2005; Wik, et al., 2005).

Abella et al. (2005) assert attaining “high-quality CPR” requires a method to

“improve monitoring and feedback” (p. 309). This is exactly the expert use paradigm of

JITS. Ultimately, the appropriate feedback could allow professionals to customize CPR

based on the individual victim’s needs (such as altering the protocol for larger breaths,

more compressions, etc.). Sensors, providing oxygen saturation or end-tidal CO2, could

provide experts valuable feedback during the procedure and augment their response.

Experts have invested considerable time and effort forging their proficiency.

Becoming expert in a field is so demanding that their breadth of expertise is usually

limited (Lukasiewicz, 1994, Moghaddam, 1997). Thus, they are not likely to perform as

experts when novel tasks and new technologies are thrust upon them. They will not have

automatized the relevant skills, increasing the drain on their cognitive resources. Experts

may be stripped of their status (at least temporarily) and find themselves acting as non-

experts. JITS solutions could serve them well.

CPR & Defibrillation A man collapses in a mall and is in need of immediate medical attention. He has

no pulse, and is not breathing. A woman, with no medical training, seizes a device that

contains supplies and instructions for administering basic life support (BLS). She opens

the box and a video screen engages her attention and provides step-by-step instructions

on how to complete each task. She removes the victim’s shirt and places electrode pads

correctly because both steps are demonstrated with simple dynamic graphics mapping the

salient features on the display to the real-world. The pads not only serve to monitor and

treat the patient, but also collect data on the actions of the user. This function enables the

system to provide corrective feedback (the frequency of chest compressions for example).

Similar monitoring technology resides in the oxygen mask. Thoughtful engineering and

animated instructions ensure proper placement on the victim. The system prescribes an

action sequence based on the needs of the patient. A protocol is devised, communicated

with cues and feedback, enabling the naïve caregiver to administer life-saving support.

Help is on the way as a call to 911 was placed upon activating the support system.

Fire extinguisher

Some oily rags are ignited in a homeowner’s garage. He runs to get his fire

extinguisher and dons the accompanying goggles. These are not high school chemistry

goggles (though eye protection is a key feature). They contain a microprocessor, infra-

red sensors, and an ability to overlay images on the real world. The images are

components of a larger set of instructions presented to the user. The Spartan pull, aim,

squeeze, sweep directions are conveyed to the user in the proper sequence just as he needs

the information. Sensors on the extinguisher itself support the tutorial. For example, the

spray & sweep commands are not activated until pin removal is registered. Guidance is

given when the user peers through the goggles at the fire. A synthetic vision program

provides a moving target to optimize the spray & sweep action. The sensors in the

goggles also calculate the distance of the operator from the fire and whether he should

move closer or back up. By comparing the intensity of the fire with the capacity of the

extinguisher (and changing intensity as a result of the extinguishing agent applied), a

decision algorithm continually runs to determine if the fire should be battled or if/when to

evacuate. Through dynamic temperature readings, the system determines the user is

aiming too high and provides instructive feedback to correct the error. Now the fuel is

being suffocated and quenched and the fire soon dies.

Space station maintenance

An astronaut is preparing a spacewalk to repair a component on the International

Space Station (ISS). Preparation for this task began several years ago. She was first

introduced to this task on a desktop simulator. A JITS system guided her through her

first actions and began to sketch her profile from the first encounter. Subsequent

practice with the system not only educated the astronaut about the task, but the system

learned a great deal about the user. The JITS system was integrated in every step of

training and practice from part task simulations to the pool. This system is also utilized

when performing the actual spacewalks. The user and system have developed a

significant history over the years of training. The information requirements and data

presentation have adjusted with the astronaut’s evolving competency. The system has a

robust representation of the user for each task based on parameters such as the number of

interactions, ratio of successful interactions, and even knows how long she takes to

perform individual subtasks. On this mission, the system immediately notices that a

step is taking far longer than usual. The system senses a problem and investigates

immediately. Sensors indicate a bolt has not been loosened (a step required for the task).

At a level of expert presentation, this minor step does not even warrant presentation (and

hasn’t been presented to this astronaut since the early days of training). The system

assumes that more information is needed and provides elementary steps and monitoring

(e.g. is tool applied, is it rotating in the proper direction). Shortly after the rudimentary

task animations, the bolt begins to loosen. Oxygen deprivation had induced erroneous

bolt-turning behavior. The error had eluded the astronaut’s detection due to her reduced

monitoring capability. Decomposition and active monitoring generated feedback to solve

the problem. The system will present more information (than it normally would), and

continue to present basic task steps until it recognizes performance has approached that

stored in her profile.

REFERENCES

Abella, B., Avarado, J., Myklebust, H. et al. (2005). Quality of cardiopulmonary resuscitation during in-hospital cardiac arrest. JAMA. 293:305-310

American Heart Association. (2005). Heart Disease and Stroke Statistics — 2005 Update. Dallas, Tex.: American Heart Association; available at http://www.americanheart.org/presenter.jhtml?identifier=1928

Anderson, J.R., (1983). The architecture of cognition. Cambridge, MA: Harvard

University Press. Anderson, J.R. (1988). The expert module. In M. Polson & J. Richardson (Eds.),

Foundations of Intelligent Tutoring Systems. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Annett, J., and Stanton, N. (Eds.) (2000). Task Analysis. New York: Taylor & Francis

Azevedo, R. & Bernard, R.(1995). A meta-analysis of the effect of feedback in computer based instruction. Journal of Educational Computing Research, 13(2), 109-125.

Balzer, W, Hammer, L., Sumner, K., Birchenough, T., Parham, S., & Raymark, P.

(1994). Effects of cognitive feedback components, display format and elaboration on performance. Organzational Behavior and Human Decision Processes, 58, 639-385. Becker, L., Eisenberg, M., Fahrenbruch, C., & Cobb, L. (1998). Public locations of cardiac arrest: Implication for public access defibrillation. Circulation, 97, 2106-2109.

Booher, H. (1975). Relative comprehensibility of pictorial information and printed words in proceduralized instructions. Human Factors, (17), 261-277.

Brown, J. S., Burton, R. R., (1978). Diagnostic models for procedural bugs in mathematical skills. Cognitive Science, 2, 155-191.

Brown, J. S., Burton, R. R., & deKleer, J. (1982). Pedagogical, natural language and knowledge engineering techniques in SOPHIE I, II and III. In D. Sleeman & J. S. Brown (Eds.), Intelligent Tutoring Systems. New York: Academic Press.

Brusilovsky, P., & Hoah-Der, S. (2002). Adaptive visualization component of a distributed web-based adaptive educational system. Intelligent Tutoring Systems, Proceedings of the 6th International Conference, Biarritz, France and San Sebastian, Spain.

Bunch, T, et. al. (2003). Long term outcomes of out-of-hospital cardia arrest after successful early defibrillation. N Engl J Med 348; 26.

Burns, H.L., & Capps, C.G. (1988). Foundations in intelligent tutoring systems: An Introduction. In M. Polson & J. Richardson (Eds.), Foundations of Intelligent Tutoring Systems. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Caffrey, S., Willoughby, P., Pepe, P., & Becker, L. (2002). Public use of automated external defibrillators. N Engl J Med, 347(16), 1242-1247.

Capucci, A., Aschieri, D., Peipoli, M., Brady, G., Iconomu, E., & Arvedi, M. (2002). Tripling survival from sudder cardiac arrest via early defibrillation without traditional education in cardio pulmonary resuscitation. Circulation, 106, 1065-1070.

Carbonell, J.R. (1970). AI in CAI: An artificial intelligence approach to computer-assisted instruction. IEEE Transactions on Man-Machine Systems, 11(4), 190-202.

ChanLin, L. (2000). Attributes of animation in learning scientific knowledge.

Journal of Instructional Psychology, 27(4), 239-243.

ChanLin, L. (2001). Formats and prior knowledge on learning in a computer-based lesson. Journal of Computer Assisted Learning, 17, 409-419.

Chi, M.T.M., Glaser, R., & Farr, M.J (Eds.). (1988). The Nature of Expertise.

Hillsdale, NJ: Erlbaum. Collins, A. (1977). Processes in acquiring knowledge. In R. Anderson, R. Spiro,

& W. Montague, (Eds.), Schooling and the acquisition of knowledge. Hillsdale, NJ: Lawrence Erlbaum Associates.

Craik, F. & Lockhart, R. (1972). Levels of processing: A framework for memory

research. Journal of Verbal Learning & Verbal Behavior, 11, 671-684.

Culley, L., Rea, T., Murray, J., Welles, B., Fahrenbruch, C., Olsufka, M., Eisenberg, M., & Copass, M. (2004). Public access defibrillator in out-of-hospital cardiac arrest. Circulation, 109, 1859-1863.

Cummins R. (1989). From concept to standard-of-care? Review of the clinical experience with automated external defibrillators. Ann Emerg Med.18: 1269–1275

Cummins, R., & Hazinski, M.F. (Eds.)(2000). Guidelines 2000 for cardiopulmonary resuscitation and emergency cardiovascular care. Circulation. Vol. 102(8). Supplement I. August

Dix, A., Finlay, J., Abowd, G., Beale, R. (1998). Human-Computer Interaction. London: Prentice Hall Europe.

Fossel, M., Kiskaddon, R., Sternbach, G. (1983). Retention of cardiopulmonary

resuscitation skills by medical students. Journal of Medical Education, 58, 568-575.

Gleick, J. (1999). Faster: The Acceleration of Just About Everything. New York, NY: Pantheon Books.

Goodman, J. S. (1998). The interactive effects of task and external feeack on practice performance and learning. Organizational behavior and human decision processes, 76(3), December, pp.223-252.

Glaser, R. & Chi, M. T. M. (1988). Overview. In M. Chi, R. Glaser & M. Farr (Eds.), The Nature of Expertise (pp. xv-xxvii). Hillsdale, NJ: Erlbaum.

Graham, C., Scollon, D. (2002). Cardiopulmonary resuscitation training for undergraduate medical students: a five year study. Medical Education, 36, 296-298

Hinds, P.J., Patterson, M & Pfeffer, J. (2001). Bothered by abstraction: The effect of expertise on knowledge transfer and subsequent novice performance. Jouran of Applied Psychology, 86(6), 1232-1243). Hallstrom, A., Ornato, J.(2004) Public-access defibrillation and survival after out-of-hospital cardiac arrest. N Engl J Med 351; 7.

Hazinski, M., Idris, A., Kerber, R., Epstein, A., Atkins, D., Tang, W., & Lurie, K. (2005). Lay rescuer automated external defibrillator (public access defibrillation) programs. Circulation, 111, 3336-3340.

Hollnagel, E. (1993). Human Reliability Analysis and Control. London: Academic Press

Huikuri, H.V., Castellanos, A., Myerburd, R. (2001). Sudden cardiac death due to cardiac arrhythmias. N Engl J Med, Vol. 345, No. 20

International Liaison Committee on Resuscitation (ILCOR) (2005). 2005 International Consensus on Cardiopulmonary Resuscitation (CPR) and Emergency Cardiovascular Care (ECC) Science With Treatment Recommendations. Circulation 112, (suppl III). American Heart Association.

Johnson, E.J. (1988). Expertise and decision under uncertainty: Performance and process. In M. Chi, R. Glaser & M. Farr (Eds.), The Nature of Expertise. (pp. 209-228). Hillsdale, NJ: Erlbaum.

Kehoe, C., Stasko, J., & Taylor, A. (2001). Rethinking the evaluation of algorithm animations as learning aids: an observational study. International Journal of Human-Computer Interaction, 54, 265-284.

Kirwan, B., and Ainsworth, L.K. (1992). A Guide To Task Analysis. London: Taylor & Francis

Klein, G. (1998). Sources of Power: How people make decisions. Cambridge,

MA: The MIT Press.

Kluger, A., & DeNisi, A. (1996). The effect of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulleting, 119(2), 254-284.

Lederer, W., Mair, D., Rabl, W., Baubin, M. (2004). Frequency of rib and sternum fractures associated with out-of-hospital cardiopulmonary resuscitation is underestimated by conventional chest X-ray. Resuscitation, Feb. 60(2), 157-62.

Lukasiewicz, J. (1994). The Ignorance Explosion: Understanding Industrial Civilization. Ottawa, Ontario: Carleton University Press.

Linn, R. L. (1990). Diagnostice testing. In N. Frederiksen, R. Glaser, A.

Lesgold, M. Shafto (Eds.), Diagnostic Monitoring of Skill and Knowledge Acquisition (pp.489-498). Hillsdale, New Jersey: Lawrence Erlbaum Associates.

Marenco, J., Wang, P., Link, M., Homoud, M., & Estes, M. (2001). Improving survival from sudden cardiac arrest. JAMA, 285(9), 1193-1200.

Marshall, S. P. (1990) Generating good items for diagnostic tests. In N,

Frederiksen, R. Glaser, A. Lesgold, M. Shafto (Eds.), Diagnostic Monitoring of Skill and Knowledge Acquisition, pp.433-452. Hillsdale, New Jersey: Lawrence Erlbaum Associates, Inc.

Miller, M., & Perlis, D. (1997). Toward automated expert reasoning and expert-

novice communication. In P. Feltovivh, K. Ford, & R. Hoffman (Eds.), Expertise in context (pp.405-415). Menlo Park, CA: American Association for Artificial Intelligence.

Miller, R., Waitman, L., Sutin Chen, S. & Rosenbloom, S. (2005) The anatomy of decision support during inpatient care provider order entry (CPOE): Empirical observations from a decade of CPOE experience at Vanderbilt. Journal of Biomedical Informatics, 38, Issue 6 , 469-485.

Virvou, M., & Moundridou, M. (2001). Adding instructor modeling component to to the architecture of ITS authoring tools. International Journal of Artificial Intelligence in Education,12, 185-211.

Mogdaham, F. (1997). The Specialized Society: The Plight of the Indvidual in an Individualized Society. Westport, CT: Praeger Publishers.

Morgan, C., Donnelly, P., & Lester, C. (1996). Effectiveness of the BBC’s 999 training roadshows on cardiopulomonary video performance of chohort of unforewarned participants at home afterwards. BMJ, 313, 912-916.

Myerburg, R., Fenster, J., Velez, M., Rosenberg, D., Lai, S., Kurlansky, P., Newton, S., Knox, M. & Castellanos, A. (2002). Impact of community-wide police car deployment of automated external defibrillators on survival from out-of-hospital cardiac arrest. Circulation, 106, 1058-1064.

Nichol, G., Hallstrom, A., Ornato, J., Riegel, B., Stiell, I., Valenzuella, T., Wells, G., White, R., & Westfeldt, M. (1998). Potential cost-effectiveness of public access defibrillation in the United States. Circulation, 97, 1315-1320.

National Center for Health Statistics, data for 2002 (2005). National Vital

Statistics Report Vol. 53. No. 17. available at http://www.cdc.gov/nchs/data/dvs/nvsr53_17tableE2002.pdf

Naki�enovi�, N., & Grübler, A. (1991). Diffusion of Technologies and Social Behavior. Berlin, Germany: International Institute for Applied Systems Analysis (IIASA).

Palmiter, S. and Elkerton, J. and Baggett, P. (1991). Animated demonstrations vs written instructions for learning procedural tasks: A preliminary investigation. International Journal of Man-Machine Studies, 34, 687-701.

Park, O., & Hopkins, R. (1992). Instructional conditions for using dynamic visual displays: A review. Instructional Science, 21(6), 427-449.

Proctor, R., & Dutta, A. (1995). Skill Acquistion and Human Performance. Thousand Oaks, CA: Sage Publications.

Rieber, L.P. (1990). Animatin in computer-based instruction. Educational

Technoloyg, Research and Development, 38, 77-86.

Sanderson, P., Haskell, I., & Flach, J. (1992). The complex role of perceptual organization in visual display design theory. Ergonomics, 35(10), 1199-1219.

Salmoni, A., Schmidt, R., & Walter, C. (1984). Knowledge of results and motor learning: A review and critical reappraisal. Psychological Bulletin, 95(3), 355-386.

Sanders, A., & Ewy, G. (2005). Cardiopulmonary resuscitation in the real world: When will the guidelines get the message? JAMA,293, 363-365.

Sheridan, T. (2002). Humans and Automation: System Design and Research Issues. Santa Monica, CA: Wiley & Sons, Inc. in cooperation with the Human Factors & Ergonomics Society. Stanton, N., Ashleigh, M., Roberts, A., & Xu, F. (2001). Testin Hollnagel’s contextual control model: Assessing team behavior in a human supervisory control task. International Journal of Cognitive Ergonomics, 5(2), 111-123.

Starr, L. (1998). An effective CPR home learning system. AAOHN J, 46(6), 289-295.

Sternberg, R. (1997). Cognitive conceptions of expertise. In P. Feltovivh, K. Ford, & R. Hoffman (Eds.), Expertise in context (pp.149-162). Menlo Park, CA: American Association for Artificial Intelligence. Stiell, I., Nichol, G., Wells, G., De Maio, V., Nesbitt, L., Blackburn, J., & Spaite, D. (2003). Health-related quality of life is better for cardiac arrest survivors wh received citizen cardiopulmonary resuscitation. Circulation, 108, 1939-1944.

Tversky, B. (1995). Cognitive origins of conentions. In F. Marchese (Ed.), Understanding Images: Finding Meaning in Digital Imagery, 29-53. New York: Springer-Verlag.

Teversky, B., & Morrison, J. B. (2002). Animation: Can it facilitate?

International Journal of Human-Computer Studies, 57, 247-262.

Uhr, L. (1969). Teaching machine programs that generate problems as a function

of interaction with students. Proceedings of the 24th National Conference of Education, Virginia, April 2-4, 1969. Valenzuela, T., Kern, K., Clark, L., Berg, R., Berg, M., Breg, D., Hilwig, R., Otto, C., Newburn, D., Ewy, G. (2005). Interuptions of chest compressions during emergency medical systems resuscitation. Circulation 112, 1259-1265.

Valenzuela, T., Roe, D., Nichol, G., Clark, L., Spatie, D., Hardman, R.(2000). Outcomes of rapid defibrillation by security officers after cardiac arrest in casinos. N Engl J Med, 343, 17.

Wik, L., Kramer-Johansen, J., Myklebust, H., Sorebo, H., Svensson, L., Fellows, B., & Steen, P. (2005). Quality of cardiopulmonary resuscitation during out-of-the hospital cardiac arrest. JAMA, 293, 299-304.

Willows, D. (1978). A picture is not always worth a thousand words: Pictures as distractions in reading. Journal of Educational Psychology, 70(2), 255-262.

Young, D.E. & Lee, T.D. (2001). Skill Learning: Augmented feedback. In W. Karwowski (Ed.) International Encyclopedia of Ergonomics and Human Factors. (pp. 558-562). London: Taylor & Francis.

Zipes, D., Wellens, H.(1998). Sudden cardiac death. Circulation:98:2334-2351.