use of prognostic scoring systems to · section four: the use of inflammatory markers in addition...

Use of prognostic scoring

systems to predict outcomes of

critically ill patients

Dr. Kwok Ming HO

MBBS, Postgrad Dip (Echo), MPH, FRCP (Glasg), FANZCA, FJFICM

Staff Specialist, Intensive Care Unit

Royal Perth Hospital

This thesis is presented for the degree of Doctor of Philosophy of

The University of Western Australia

Schools of Medicine, Pharmacology, and Population Health

2008

Acknowledgement

I would like to thank Drs. Geoffrey Clarke and John Weekes for their part in

initiating the clinical database in the Intensive Care Unit of Royal Perth Hospital and also all

the consultants who have been recording the Acute Physiology and Chronic Health

Evaluation (APACHE) data for every admission to the Intensive Care Unit. I would like to

thank my PhD supervisors (Clinical Associate Professor Steven Webb, Clinical Professor

Geoffrey Dobb, Professor Matthew Knuiman, and Professor Judith Finn) and my colleagues

(Drs. Kok Yeng Lee and Simon Towler) in the Intensive Care Unit of Royal Perth Hospital

for their contributions to the studies generated from this thesis. I would also like to thank

BUPA Foundation for funding the cost of data linkage. Finally, I would like to express my

special thanks to my wife, Kayo, and my son, Akio, for their patience and support. Without

their support, completion of this thesis would not be possible.

Certification by the co-authors

The following co-authors certify that Dr. K.M. Ho is the main author in initiating the

original idea, designing and analysing the data, and drafting of the manuscripts and thesis.

All co-authors agree that the published manuscripts be included in this PhD thesis.

Name of co-author Signature Date

1. Prof. Matthew Knuiman

2. Prof. Judith Finn

3. Clin. A/Prof Steven Webb

4. Clin. A/Prof Geoffrey Dobb

5. Dr. Kok Y. Lee

6. Dr. Simon Towler

7. Ms Teresa Williams

CONTENT

Abbreviations P. 1

Summary (overview) of the thesis P. 2-4

Publications arising from the thesis P. 5-6

Section one: Background, rationale, materials, and methods of the study

Chapter 1. Background, motivation and rationale P. 7-13

Chapter 2. Characteristics of the cohort and statistical methods P. 14-28

Section two: Assessment of the APACHE II scoring system in an Australian context

Chapter 3. The worst first 24-hour and admission APACHE II P. 29-39

scoring system

Chapter 4. The use of the APACHE II scoring system for indigenous P. 40-46

patients

Chapter 5. Assessing calibration by meta-analytic techniques P. 47-55

Section three: Relationship between the APACHE II scoring system, organ failure scores,

and co-morbidities in determining hospital mortality and ICU readmission

Chapter 6. Comparing the APACHE II scoring system with organ failure

scores to predict hospital mortality P. 56-66

Chapter 7. Combining the APACHE II scoring system with Sequential

Organ Failure Assessment (SOFA) scores to predict hospital mortality P. 67-76

Chapter 8. Combining the APACHE II scoring system with co-morbidity

data to predict hospital mortality P. 77-85

Chapter 9. The effect of co-morbidity on risk of unplanned ICU

readmission P. 86-94

Chapter 10. Evaluating the APACHE II scoring system in predicting

hospital mortality of ICU readmissions P. 95-106

Section four: The use of inflammatory markers in addition to organ failure score and the

APACHE II scoring system in predicting post-ICU hospital mortality and ICU readmission

Chapter 11. Inflammatory markers and risk of unplanned ICU

readmission P. 107-15

Chapter 12. Inflammatory markers and prediction of hospital mortality P. 116-24

Section five: Predicting long term survival after hospital discharge

Chapter 13. The PREDICT model P. 125-35

Chapter 14. The effect of socioeconomic status on long term survival P. 136-43

Section six: Conclusion

Chapter 15. Summary and directions for future research P. 144-6

References P. 147-55

Appendices: Ethics approval forms and correspondence P. 156-60

1

Abbreviations

ACHS Australian Council of Healthcare Standards

APACHE Acute Physiology and Chronic Health Evaluation

ARIA Area of Remoteness Index of Australia

ANZICS Australian & New Zealand Intensive Care Society

CHIC Confidentiality of Health Information Committee

CI Confidence Interval

CRP C-reactive protein

DLU Data Linkage Unit

HMD Hospital Morbidity Databases

ICD International Classification of Diseases

ICU Intensive Care Unit

MPM Mortality Prediction Model

ROC Receiver Operating Characteristic

RPH Royal Perth Hospital

RPHICU Royal Perth Hospital Intensive Care Unit

SAPS Simplified Acute Physiology Score

SEIFA Socio-Economic Indices for Areas

SES Socioeconomic Status

SMR Standardised Mortality Ratio

SOFA Sequential Organ Failure Assessment

SPSS Statistical Package for the Social Sciences

SUPPORT Study to Understand Prognoses and Preferences for Outcomes

and Risks of Treatments

WA Western Australia

2

Summary (overview) of the thesis

This research thesis consists of five sections. Section one provides the

background information (chapter 1) and a description of characteristics of the cohort

and the methods of analysis (chapter 2).

The Acute Physiology and Chronic Health Evaluation (APACHE) II scoring

system is one of commonly used severity of illness scoring systems in many intensive

care units (ICUs). Section two of this thesis includes an assessment of the

performance of the APACHE II scoring system in an Australian context. First, the

performance of the APACHE II scoring system in predicting hospital mortality of

critically ill patients in an ICU of a tertiary university teaching hospital in Western

Australia was assessed (Chapter 3). Second, a simple modification of the traditional

APACHE II scoring system, the ‘admission APACHE II scoring system’, generated

by replacing the worst first 24-hour data by the ICU admission physiological and

laboratory data was assessed (Chapter 3). Indigenous and Aboriginal Australians

constitute a significant proportion of the population in Western Australia (3.2%) and

have marked social disadvantage when compared to other Australians. The difference

in the pattern of critical illness between indigenous and non-indigenous Australians

and also whether the performance of the APACHE II scoring system was comparable

between these two groups of critically ill patients in Western Australia was assessed

(Chapter 4).

Both discrimination and calibration are important indicators of the

performance of a prognostic scoring system. Meta-analytic techniques were used in

this thesis to illustrate the uniformity of fit in the calibration of the APACHE II

scoring system across different diagnostic and age subgroups and the results of these

3

techniques were compared with the results from assessment of the slope and intercept

of the calibration curve of the APACHE II scoring system (chapter 5).

There are other factors and alternative scoring systems that may be useful and

may improve the performance of the APACHE II scoring system if they are

incorporated. Section three of this thesis includes a comparison of the performance of

the APACHE II scoring system with two organ failure scoring systems (Chapter 6),

and an evaluation of whether the performance of the APACHE II scoring system

could be enhanced by incorporating organ failure data (Chapter 7) and more detailed

co-morbidity data (Chapter 8).

Unplanned ICU readmission is one of the quality indicators adopted by the

Australian Council of Healthcare Standards (ACHS). This undesirable in-hospital

outcome was further explored by assessing whether the APACHE II scoring system

and co-morbidity can be used to predict unplanned ICU readmission during the same

hospitalisation (Chapter 9). The APACHE II scoring system (and in fact all existing

ICU scoring systems) had excluded patients who were readmitted to the ICU during

the same hospitalisation, and as such, its performance in predicting mortality of ICU

readmission remained unknown. The use of the APACHE II scoring system in

patients readmitted to ICU during the same hospitalisation was evaluated and also

whether incorporating events prior to the ICU readmission to the APACHE II scoring

system would improve its ability to predict hospital mortality of ICU readmission was

assessed in chapter 10.

Whilst there have been a number of studies investigating predictors of post-

ICU in-hospital mortality none have investigated whether unresolved or latent

inflammation and sepsis may be an important predictor. Section four examines the

role of inflammatory markers measured at ICU discharge on predicting ICU re-

4

admission (Chapter 11) and in-hospital mortality during the same hospitalisation

(Chapter 12) and whether some of these inflammatory markers were more important

than organ failure score and the APACHE II scoring system in predicting these

outcomes.

Section five describes the development of a new prognostic scoring system

that can estimate median survival time and long term survival probabilities for

critically ill patients (Chapter 13). An assessment of the effects of other factors such

as socioeconomic status and Aboriginality on the long term survival of critically ill

patients in an Australian ICU was assessed (Chapter 14).

Section six provides the conclusions. Chapter 15 includes a summary and

discussion of the findings of this thesis and outlines possible future directions for further

research in this important aspect of intensive care medicine.

5

Publications arising from the thesis

Chapter 3.

Ho KM, Dobb GJ, Knuiman M, Finn J, Lee KY, Webb SA. A comparison of

admission and worst 24-hour Acute Physiology and Chronic Health Evaluation II

scores in predicting hospital mortality: a retrospective cohort study. Critical Care

2006;10:R4.

Chapter 4.

Ho KM, Finn J, Dobb GJ, Webb SA. The outcome of critically ill Indigenous patients.

Medical Journal of Australia 2006;184:496-9.

Chapter 5.

Ho KM. Forest and funnel plots illustrated the calibration of a prognostic model: a

descriptive study. Journal of Clinical Epidemiology 2007;60:746-51.

Chapter 6.

Ho KM, Lee KY, Williams T, Finn J, Knuiman M, Webb SA. Comparison of Acute

Physiology and Chronic Health Evaluation (APACHE) II score with organ failure

scores to predict hospital mortality. Anaesthesia 2007;62:466-73.

Chapter 7.

Ho KM. Combining sequential organ failure assessment (SOFA) score with acute

physiology and chronic health evaluation (APACHE) II score to predict hospital

mortality of critically ill patients. Anaesthesia & Intensive Care 2007;35:515-21.

Chapter 8.

Ho KM, Finn J, Knuiman M, Webb SA. Combining multiple comorbidities with

Acute Physiology Score to predict hospital mortality of critically ill patients: a linked

data cohort study. Anaesthesia 2007;62:1095-100.

http://www.ncbi.nlm.nih.gov/pubmed/16719746?ordinalpos=25&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_RVDocSum





6

Chapter 9.

Ho KM, Dobb GJ, Finn J, Knuiman M, Webb SA. The effect of co-morbidities on

risk of intensive care readmission during the same hospitalisation: a linked data cohort

study. Journal of Critical Care 2008 (published online in April 2008).

Chapter 10.

Ho KM, Knuiman M. Bayesian approach to predict hospital mortality of intensive

care readmissions during the same hospitalisation. Anaesthesia & Intensive Care

2008;36:38-45.

Chapter 11.

Ho KM, Dobb GJ, Lee KY, Towler SC, Webb SA. C-reactive protein concentration

as a predictor of intensive care unit readmission: a nested case-control study. Journal

of Critical Care 2006;21:259-65.

Chapter 12.

Ho KM, Lee KY, Dobb GJ, Webb SA. C-reactive protein concentration as a predictor

of in-hospital mortality after ICU discharge: a prospective cohort study. Intensive

Care Medicine 2008;34:481-7.

Chapter 13.

Ho KM, Knuiman M, Finn J, Webb SA. Estimating long-term survival of critically ill

patients: the PREDICT model. Public Library of Science One 2008;3:e3226.

Chapter 14.

Ho KM, Dobb GJ, Knuiman M, Finn J, Webb SA. The effect of socioeconomic

inequalities on outcomes of seriously ill patients: a linked data cohort study. Medical

Journal of Australia 2008;189:26-30.




7

Section one: Background, rationale, materials, and

methods of the study

Chapter 1. Background, motivation and rationale

The significance of intensive care services

ICUs are specially staffed and equipped hospital wards that provide advanced

life support for patients with life-threatening illnesses or after major surgery. The

major role of ICUs is to save lives that might otherwise be lost to acute life-

threatening illnesses such as severe infection, trauma, burns, drug overdose,

cerebrovascular accidents, or acute respiratory failure. Because of the staffing and

equipment requirements in ICUs, intensive care service is much more expensive than

many other health care services. There are more than 6,000 ICUs in the US, with

between 75,000 and 90,000 beds.1,2

The expenditures on health care were $2.1 trillion

in 2006 and accounted for 16% of the gross domestic product, of which

10% (of the

total health expenditures) is estimated to be spent on critically ill patients in the

United States.3

The cost of adult ICUs in the United Kingdom has been estimated at

£700 million, which represents 0.1% of GDP.4

In 2001, there were 172 ICUs in

Australia, providing a total of 1,272 beds to care for the 137,598 critically ill patients.5

The total cost of ICU care in Australia is not known. In 2003, the average costs of an

ICU day and total cost of ICU stay per patient in a tertiary ICU in Australia were

estimated to be about A$2,670 and A$9,852, respectively.6

Demand for ICU services is increasing,7 and at a rate that is higher than the

average for all health care services.2 Increase in treatment and monitoring technology,

patients’ expectations, and ageing population all contribute to this increased demand

for ICU services.7 The current data suggest that intensive care services are reasonably

8

cost effective when compared to other medical and surgical interventions,8-10

but this

may change significantly if intensive care services are provided without any rationing

or monitoring of the outcome data. Intensive care is indeed increasingly being

provided to many older and sicker patients, whom in the past were not treated in the

ICUs.11

Undesirable outcomes following critical illness

Although most patients survive their critical illness after intensive care therapy

without any undesirable events, unplanned ICU readmission and in-hospital death

after ICU discharge during the same hospitalisation are not uncommon in many

ICUs.12,13

Some patients may also die within a short period of time after their hospital

discharge or survive with a very poor quality of life.14

If modifiable risk factors of

these undesirable outcomes after critical illness can be identified, perhaps patient

outcomes can be improved by improving the process of care to reduce these risk

factors. From a clinical perspective, many patients and clinicians may also be

interested to know the risk factors of these poor outcomes, even if they are not

modifiable, when making difficult triage or treatment decision in ICUs. Furthermore,

survival after hospital discharge is increasingly being used as an end-point in

assessing cost effectiveness of expensive new technology and treatments in critical

illness.15

If tools that can predict short and long term outcomes after critical illness are

available, the accuracy of risk adjustment and cost effectiveness analysis of any new

treatments or technology can potentially be improved.

A quest for a cost effective high quality intensive care service coupled with an

increasing demand provides a strong rationale for improved modelling of prediction

of outcomes in ICUs.

9

The existing prediction or scoring systems in intensive care

Predictions and prognostications of outcomes of critically ill patients are often

made on a daily basis by many clinicians working in the ICUs or critical care

environments to triage ICU admissions and treatment.16-18

It will be desirable if

clinicians have accurate and consistent information regarding the patients likely

outcomes. Several prediction and prognostic scoring systems have been developed,

and these include scoring systems that evaluate co-morbidities,19

organ failure,20-22

or

a combination of factors including age, chronic health status, and severity of

physiological derangement. The latter type of scoring systems, such as the Acute

Physiology and Chronic Health Evaluation (APACHE) scoring system, Mortality

Prediction Model (MPM), and Simplified Acute Physiology Score (SAPS), are

commonly used in many ICUs for audit and research purposes.23-25

The performances of these scoring systems are quite variable, especially when

applied to different cohorts of critically ill patients in different ICUs. The APACHE

scoring system, initially described in 1981 in USA, is the most widely used method in

assessing severity of illness in ICUs. The APACHE II scoring system, published in

1985, is a revised version that uses the worst physiological measurements within the

first 24 hours of ICU admission, age and previous health status coupled with

diagnostic information to estimate the risk of hospital death.24

An increasing score

(range 0 to 71) has been shown to be closely correlated with the subsequent risk of

death across a range of life threatening diseases, with an area under the receiver

operator characteristic (ROC) curve in the initial validation study of 86%.24

The

APACHE III scoring system, published in 1991, is a further modification of the

APACHE scoring system and has incorporated more diagnostic categories to improve

its calibration.25

The performance of the APACHE III scoring system has

10

subsequently been evaluated in Brazil, the United Kingdom, Germany, US, and

Australia with variable results.26,27

The use of APACHE III scoring system was

further extended to predict 6-month survival of critically ill patients and the area

under receiver operating characteristic (ROC) curve in the validation cohort was

found to be 78%.14

In 2006, a further update on the APACHE scoring system was

made and the APACHE IV scoring system was published.28

Despite the availability of

the newer versions of the APACHE scoring system, the APACHE II scoring system

remains very popular and commonly used in many ICUs for audit and clinical

research purposes. This may be because of its ease of use and its long history in

clinical use that allows easy comparison between different ICUs or time periods.29,30

Perhaps somewhat surprising, the APACHE II scoring system has not been

thoroughly evaluated in any Australian ICUs and its applicability in predicting

hospital mortality of critically ill Aboriginal patients has also not been evaluated.

The APACHE II scoring system has significant limitations. First, the

APACHE II scoring system uses the worst physiological derangement of a patient

within the first 24 hours of ICU admission. In collecting the worst physiological and

laboratory data, the data collector has to compare all the physiological and laboratory

data over a 24-hour period to generate the correct (maximised) APACHE II score.31

Studies have shown that accuracy of the data can be very variable depending on

expertise of the data collector and the process of data collection can be very time

consuming. In using the APACHE II (or III and IV) scoring system as a tool to

stratify participants into different risk categories in a clinical trial in ICUs, enrolment

of the participants may have to be delayed to wait for complete collection of the first

24 hours of physiological and laboratory data. Alternatively, the worst physiological

and laboratory data until the point of enrolment within the first 24 hours of ICU

11

admission are used. Another possible and more straightforward alternative would be

to use the physiological and laboratory data at the time of ICU admission. This

approach is easier and the results may be potentially less variable because comparing

physiological and laboratory data obtained within the first 24 hours of ICU admission

is not necessary. The admission data have in fact been used by some studies to

calculate the APACHE II score for risk adjustment purposes.32,33

Whether this

modified use of the APACHE II scoring system is reliable or as accurate as the

original APACHE II scoring system has not been assessed.

Second, the APACHE II scoring system was primarily designed to estimate

hospital mortality of critically ill patients who have not been treated in an ICU during

the same hospitalisation. In the original cohort used to derive the APACHE II scoring

system, patients readmitted to the ICU during the same hospitalisation were

excluded.24

No other existing ICU scoring systems have studied this subgroup of

critically ill patients and there is currently no risk adjustment tool that can estimate the

risk of hospital death of these patients. The Australian Council of Healthcare

Standards (ACHS) has adopted unplanned ICU readmission within 72 hours of ICU

discharge as an indicator of the quality of care of an ICU because unplanned ICU

readmissions are associated with an increase in health care costs, patients morbidities

and mortality.34,35

Very little epidemiological data on this quality indicator or

unexpected death after ICU discharge during the same hospitalisation are available

from Australian ICUs. Whether the APACHE II scoring system, either alone or in

combination with other prognostic factors such as co-morbidities or organ failure, can

be used to predict these undesirable in-hospital outcomes has not been thoroughly

evaluated in an Australian ICU population.36,37

12

Third, it is possible that many patients and ICU clinicians make ICU triage

and treatment decisions based on their perception of the most likely long term

survival and quality of life following a critical illness.38,39

Epidemiological data on the

long term outcomes following critical illness are sparse. Most published

epidemiological data on long term outcomes of critically ill patients are limited by a

relatively short duration and significant loss to follow up, small sample size, cohort

with limited range of diagnoses, or absence of information on severity of acute illness

or pre-existing comorbidities.40

The use of the APACHE II and III scoring systems in predicting survival after

hospital discharge was evaluated by two studies. The SUPPORT investigators from

the United States of America and Wright et al. from United Kingdom published two

scoring systems that can provide an estimation of 6-month and 5-year survival

probabilities of critically ill patients, respectively.14,41

The latter scoring system

provides only three survival probabilities if a patient’s risk score falls into either <70,

70-80, or >90.41

As such, its utility is limited. There is currently no prognostic scoring

systems that can give long term survival estimate of more than 5 years.

Broad aims of the study

In this thesis, the strengths and weaknesses of one of the most commonly used

scoring systems in ICUs, the APACHE II scoring system, were evaluated in an

Australian context including its application to critically ill indigenous patients. An

assessment was made as to whether this scoring system can be further improved by

incorporating other data including organ failure and detailed co-morbidity data.

13

The incidence for two undesirable in-hospital outcomes, unplanned ICU

readmission and unexpected death after ICU discharge, was described and whether

the APACHE II scoring system was useful to predict these outcomes was assessed.

As a first attempt to develop a tool that can estimate long term survival

probabilities following critical illness, a new scoring system (the PREDICT model)

was developed. Evidence suggests that ethnicity and socio-economic status may have

a significant association with long term survival after some life-threatening

diseases.42,43

Whether Aboriginality, socioeconomic status, and accessibility to

essential services may have an independent association with long term survival

outcome of critically ill patients, over and beyond the usual biological factors such as

co-morbidities and severity of illness as measured by the APACHE II scoring system,

were assessed in this thesis.44

14

Section one: Background, rationale, materials, and

methods of the study

Chapter 2. Characteristics of the cohort and statistical methods

Patient selection and data sources

(1) Royal Perth Hospital Intensive Care Unit (RPHICU) databases

Royal Perth Hospital (RPH) intensive care unit is the largest ICU in Western

Australia and it provides over 40% of all intensive care services in Western Australia.

Since 1987, demographics and clinical information such as the admission APACHE II

score and its components on admission, admission diagnoses, daily assessment of

organ failure (RPH organ failure score), and daily administration of common

therapeutic modalities, have been collected for all patients admitted to the RPHICU.

Since 1989, the worst APACHE II scores within the first 24 hours of ICU admission

have been collected. Since 2004, daily Sequential Organ Failure Assessment (SOFA)

score have been collected. The data were collected prospectively on a pre-printed data

collection form by the senior medical staff of the unit and entered into a database

(.dbf) by one designated clerical officer.

Between 1987 and 2002 (i.e. 16 years), there were 26,021 RPHICU

admissions for 22,990 patients (over 3,000 patients had more than one ICU

admissions). The patient population was comprised of more males (67%) than

females, had a mean age of 57.2 (± standard deviation [SD] 17.4) years, with only

0.7% under the age of 16 years (RPH provides only limited paediatric services) and

4.8% were 80 years or older. The median length of stay in ICU and hospital was 2.6

and 12 days, respectively. The mean APACHE II score for ‘elective’ admissions was

15

10.0 (± SD 4.1), while the mean APACHE II score for ‘emergency’ patients was 13.5

(± SD 7.5). The overall crude ICU and hospital survival was 92.1% and 89.2%,

respectively.

(2) Western Australian Data Linkage

Details of each individual patient can theoretically be summarised in a more

comprehensive manner by linking all the available health and administrative

databases from different sources. Data Linkage WA at the Department of Health

collaborates with the Centre for Health Services Research at the University of

Western Australia, the Division of Health Sciences at Curtin University of

Technology, and the Telethon Institute for Child Health Research to provide

information for valuable medical and population health research. The unit was

established in 1995 to develop and maintain a system of linkages connecting data

about health events for individuals in WA. The unit manages the Western Australian

Data Linkage System which links the WA's core population health data sets.

Operations depend on access to personal identifying information derived from each of

the contributing data sources, but the actual health details are stored and managed

separately by delegated data custodians. These linkages are created and maintained

using rigorous, internationally accepted privacy preserving protocols, extensive

clerical review, and probabilistic matching.45

The probabilistic matching technique is

based on six Automatch (software package) passes. These six Automatch passes

include unit medical record number (unique only to teaching hospitals), surname &

first name, initial, data of birth, sex and address of the patient. Clerical checking of

additional information for possible matches that fall within a ‘grey area’ between

http://www.health.wa.gov.au/home/



http://www.sph.uwa.edu.au/go/sph

16

definite matches and definite non-matches is undertaken to improve the accuracy of

the data.

The quality of the WA hospital morbidity databases (HMD) linked data was

assessed by a sampling technique a few years ago, and both the percentage of invalid

links (false positives) and missed links (false negatives) were estimated to be 0.11%.46

The HMD have demographic information including date of birth, gender, clinical

diagnoses recorded in all public and private hospitalisations coded according to the

International Classification of Diseases (ICD-9-CM and ICD-10CM), and details of

the hospital length of stay and hospital discharge status of all patients in WA.47

The

WA Death Registry has information on date, causes (text and Australian Bureau of

Statistics (ABS) coded), and place of all deaths (e.g. hospital or residential address) in

WA.

The data used in this thesis were obtained by linking the RPHICU database

with the WA HMD and WA Death Registry. For consistency reasons, the first RPH

ICU admission on or following January 1st 1987 was classed as that person’s index

admission and taken as time zero for estimation of their survival time after hospital

discharge. The survival status of this cohort was assessed on 31st December 2003 and

the mean duration of follow-up of the cohort was about 6 years. A minimum of 12

months follow-up period was available for the whole cohort (22,990 patients), and 5-

year, 10-year, and 15-year follow-up was available for 18,048 patients, 11,265

patients, and 3,070 patients, respectively.

The RPHICU database contained information regarding the demographic

factors of the patients such as age, gender, and ethnicity, the APACHE II score,

APACHE II predicted mortality, elective/emergency status, source of admission,

admission diagnostic categories (as classified in the APACHE II scoring system),

17

length of ICU stay, and length of hospital stay. In this thesis, the Western Australian

Data Linkage Unit provided linked information from HMD and WA Death Registry

regarding specific co-morbidities (from which Charlson co-morbidity index was

estimated),19

socioeconomic factors, and long term mortality data of all ICU

admissions.

The Centre for Health Services Research at The University of Western

Australia developed the SPSS syntax based on the Dartmouth-Manitoba algorithm to

generate a ‘Charlson co-morbidity index’ for the patients in the ICU database.48

The

co-morbidities identified within HMD records where the hospital admission date was

within a five-year period prior to the index ICU admission were all included. A

relatively long ‘look back’ period was used in this thesis to try to capture all pre-

existing co-morbidities of the patients. It is possible, that an ICU patient may not have

had any previous WA hospitalisations prior to the index ICU admission, especially if

the patient was young (and previously healthy) or had been living in other states of

Australia or overseas prior to the index ICU admission. Those ICU patients with no

link to the WA HMD were assumed to have had no previous hospitalisations and their

Charlson co-morbidity index was estimated as zero.

The post code of each ICU patient’s usual place of residence was used to

classify patients into different socioeconomic groups, using the Socio-Economic

Indices for Areas (SEIFA) of the closest Census year to the year of index ICU

admission.49

The degree of accessibility to essential services was classified by Area of

Remoteness Index of Australia (ARIA) also using the post code of the patient’s usual

place of residence. In the ARIA system, accessibility to essential services is different

among the following categories, Major Cities of Australia; Inner Regional Australia;

Outer Regional Australia; Remote Australia; and Very Remote Australia.49,50

In the

18

studies of the long term outcome of critically ill patients, patients with residential

addresses outside Western Australia were excluded from analysis because follow up

on their survival outcome outside WA would be impossible.

The cohort of patients admitted to the RPHICU between 1987 and 2002 was

used to investigate the effects of demographic factors, socioeconomic status, co-

morbidities, severity of acute illness (as measured by the APACHE II scoring system)

on risk of ICU readmission, hospital mortality, and also long term survival outcome

after hospital discharge. For the studies that compared the performance of the

APACHE II scoring system with organ failure scoring systems and also studies on

predictors of unplanned ICU readmission and unexpected death after ICU discharge,

only the RPHICU data between 2004 and 2005 were used because the daily SOFA

score was not available before 2004.

(3) Comparing RPHICU cohort to patients in other Australian ICUs

RPHICU is staffed by fully trained intensivists who have postgraduate

specialist qualifications in intensive care medicine. The formal training of intensive

care specialists in Australia was started over 30 years ago in Australia and the quality

of intensive care services is in general very high when compared to other countries.51

RPHICU and most ICUs in Australia are often regarded as a ‘closed’ unit. The

intensive care team has a strong administrative and clinical role in deciding ICU

admission, treatment options, and discharge decisions on patients who are critically ill

in the hospital.51,52

Similar staff training and administrative model of the ICUs have

created very similar clinical practices and case mix, and a supportive environment for

multi-centre research and collaborations between different Australian ICUs.53-55

19

In this thesis the characteristics of patients in three diagnostic subgroups

including sepsis, community acquired pneumonia and non-operative trauma were

compared to assess whether the RPHICU cohort in 2001 and 2002 was comparable to

patients admitted to 55 other Australian ICUs during the same time period. These

three major diagnoses were selected because they were easily matched between the

APACHE II and III scoring system that was used for the RPHICU cohort and other

Australian ICUs, respectively. The comparisons showed that patients in the RPHICU

were younger and with less co-morbidities when compared to the patients in the other

Australian ICUs. The severity of acute illness (i.e. the APACHE II predicted

mortality) and in-hospital survival function were, however, comparable between the

RPHICU cohort and other Australian ICUs in patients with these three major

diagnoses (Table 1 to 3 and Figure 1 to 3).56

With these limitations in mind, not all the

results of this thesis may be generalisable to other Australian ICUs.

20

Table 1. Characteristics of severe sepsis or septic shock admissions to Royal Perth

Hospital ICU (RPHICU) and other Australian ICUs.

Variable RPHICU (n=111) Other Australian ICUs (n=1,429) P value

#

Age, yrs (SD) 54.6 (16.9) 60.1 (17.9) 0.001

Male / female, no. (%) 54 (48.6) / 57 (51.4) 792 (55.4) / 637 (44.6) 0.198

APACHE II score 22.0 22.0 0.900

(SD, median, IQR) (7.9, 22.0, 11.0) (9.8, 21.0, 13.7)

APACHE II predicted 45.7 45.6 0.776

mortality, % (23.2, 45.2, 37.6) (26.5, 41.6, 43.6)

(SD, median, IQR)

Chronic respiratory 2 (1.8) 126 (8.8) 0.006

disease, no. (%)

Chronic cardiovascular 1 (0.9) 140 (9.8) 0.001

disease, no. (%)

Chronic renal disease, 3 (2.7) 105 (7.3) 0.079

no. (%)

Chronic liver disease, 0 (0) 59 (4.1) 0.019

no. (%)

Immunosuppressed state, 7 (6.3) 185 (12.9) 0.05

no. (%)

Length of ICU stay, 9.9 5.1 0.001

days (SD, median, IQR) (13.1, 5.1, 7.0) (7.7, 2.4, 4.9)

Length of hospital stay, 26.4 17.3 0.001


ICU mortality, no. (%) 24 (21.6) 319 (23.0) 0.815

28-day in-hospital 28 (23.4) 355 (27.9) 0.582

mortality, no. (%)

Hospital mortality, 35 (31.5) 417 (30.7) 0.832

no. (%)

# P values were generated by either Mann-Whitney or chi-square test. IQR, interquartile range.

21

Figure 1. Kaplan Meier survival curve of the patients with severe sepsis or septic

shock from Royal Perth Hospital ICU (RPHICU) and other Australian ICUs.

25 20 15 10 5 0

Days since ICU admission

1.0

0.8

0.6

0.4

0.2

0.0

RPHICU

Other Australian ICUs

Cumulative survival

Survival difference between

the two cohorts was

insignificant, p=0.194

by log rank test

22

Table 2. Characteristics of pneumonia admissions to Royal Perth Hospital ICU

(RPHICU) and other Australian ICUs.

Variable RPHICU (n=82) Other Australian ICUs (n=1,066) P value #

Age, yrs (SD) 56.1 (15.7) 61.1 (17.8) 0.003

Male / female, no. (%) 47 (57.3) / 35 (42.7) 588 (55.2) / 477 (44.7) 0.731


(SD, median, IQR) (7.2, 20.0, 9.3) (8.1, 19.0, 10.0)


mortality, % (19.9, 35.5, 28.3) (22.2, 32.2, 31.0)

(SD, median, IQR)


disease, no. (%)

Chronic cardiovascular 1 (1.2) 93 (8.7) 0.011

disease, no. (%)

Chronic renal disease, 3 (3.7) 27 (2.5) 0.469

no. (%)


no. (%)

Immunosuppressed state, 5 (6.1) 101 (9.4) 0.427

no. (%)





ICU mortality, no. (%) 13 (15.9) 169 (16.2) 1.000

28-day in-hospital 18 (22.0) 190 (20.2) 0.671

mortality, no. (%)*

Hospital mortality,* 20 (24.4) 230 (23.0) 0.786

no. (%)

# P values were generated by either Mann-Whitney or chi-square test. * ICU and hospital mortality

outcome of other Australian ICUs cohort was available only in 1,039 and 997 patients, respectively.

IQR, interquartile range.

23

Figure 2. Kaplan Meier survival curve of the patients with pneumonia from Royal

Perth Hospital ICU (RPHICU) and other Australian ICUs.

25 20 15 10 5 0


1.0

0.8

0.6

0.4

0.2

0.0

RPHICU Other Australian ICUs

Cumulative survival


the two cohorts was insignificant, p=0.860

by log rank test

24

Table 3. Characteristics of non-operative head and multiple trauma admissions to

Royal Perth Hospital ICU (RPHICU) and other Australian ICUs.

Variable RPHICU (n=176) Other Australian ICUs (n=2,114) P value #

Age, yrs (SD) 35.9 (16.3) 42.6 (19.3) 0.001

Male / female, no. (%) 137 (77.8) / 39 (22.2) 1599 (75.6) / 515 (24.4) 0.583


(SD, median, IQR) (7.2, 13.0, 9.8) (7.6, 11.0, 10.0)

Glasgow Coma Scale 9.8 11.7 0.001

within first 24 hrs (4.7, 14.0, 7.0) (4.4, 15.0, 6.0)

(SD, median, IQR)


mortality, % (15.1, 6.3, 12.1) (14.1, 6.2, 12.4)

(SD)


disease, no. (%)

Chronic cardiovascular 0 (0) 43 (2.0) 0.074

disease, no. (%)

Chronic renal disease, 0 (0) 3 (0.1) 1.000

no. (%)


no. (%)

Immunosuppressed state, 0 (0) 38 (1.8) 0.113

no. (%)





ICU mortality, no. (%)* 17 (9.7) 163 (8.0) 0.472

28-day in-hospital 18 (10.2) 195 (9.7) 0.791

mortality, no. (%)*

Hospital mortality,* 20 (11.4) 210 (10.5) 0.701

no. (%)

# P values were generated by either Mann-Whitney or chi-square test. * ICU and hospital mortality

outcome of other Australian ICUs cohort was available only in 2,031 and 2,010 patients, respectively.

IQR, interquartile range.

25

Figure 3. Kaplan Meier survival curve of the patients with non-operative head and

multiple trauma from Royal Perth Hospital ICU (RPHICU) and other Australian

ICUs.

Cumulative survival

2520151050


1.0

0.8

0.6

0.4

0.2

0.0

RPHICU

Other Australian ICUs


the two cohorts was insignificant, p=0.470

by log rank test

Cumulative survival

26

Statistical Methods

A variety of standard statistical methods were used to model the effects of

different risk factors on outcomes of critically ill patients and to assess model

performance in this thesis.

Logistic regression was used to generate predicted probability (or risk) of any

predictive models that model on a categorical outcome variable. Propensity score

method was used to assess and adjust for the effect of selection bias in one of the

studies when missing data was a significant problem.57

The discrimination of a

scoring system was assessed by area under the receiver operating characteristic

(ROC) curve, and the difference in area under the ROC curves derived from the same

cases was assessed according to the method suggested by Hanley and McNeil.58

The calibration of a scoring system was assessed by the shape of the

calibration curve and Hosmer-Lemeshow Chi-square statistics.59

The slope and

intercept of the calibration curve derived from patients of different diagnostic

subgroups were computed to assess the uniformity of fit of the scoring system across

different diagnostic subgroups. The uniformity of fit of a prognostic model across

different subgroups of patients was then compared with the results of meta-analytic

techniques that utilised funnel and forest plots to assess calibration of the model.60

These latter techniques in assessing uniformity of fit in calibration across different

subgroups of patients have not been previously reported in the literature.

In developing a scoring system to predict median survival time and long term

survival probabilities of critically ill patients, a Cox proportional hazards regression

model was fitted.61

The proportional hazards assumption of the predictors in the Cox

model was checked by plotting the logarithm of the negative logarithm of the Kaplan

Meier survivor estimates. Predictors were pre-selected according to clinical

27

plausibility and also data from the literature instead of significance of the p-value of a

predictor in the univariable analysis.62,63

During the modelling process, categorising

continuous predictors was avoided and a non-linear relationship with hazard of death

was allowed by using a 6-knot restricted cubic spline function.62,63

The discrimination

performance of the Cox model was assessed with the c-index, which is a

generalisation of the c-statistic, that allowed for censored data and was computed and

adjusted for optimism (arising from using the same data to develop the model and

assess its performance) by a bootstrap technique to penalise for possible over-fitting,

with 200 re-samples and at least 200 patients per risk group.63,64

Model calibration of

the Cox model (similarity of predicted risks and proportions actually dying) was

assessed graphically and used a bootstrap re-sampling to construct a bias-corrected

calibration curve.63

Splitting of the sample into development and validation data sets

was not used in this thesis because this technique was regarded as data ‘inefficient’

and not as accurate as bootstrapping technique.63

A nomogram was presented to

illustrate how the Cox model can be used to generate median survival time and long

term survival probabilities of a heterogenous group of critically ill patients.63

Whenever possible, the overall performance of the scoring system was assessed by

Nagelkerke R2

and Brier’s score.65-67

A p-value less than 0.05 was regarded as significant and all tests were two-

tailed in this thesis. No adjustment was made for multiple comparisons in the

subgroup analyses because of the small sample size of the subgroups. All statistical

analyses were performed by SPSS statistical software (version 13.0 for Windows,

SPSS Inc. USA) and the Cox model was constructed by using the Design library in S-

PLUS software (version 8.0, 2007, Insightful Corp.; Seattle, Washington, USA).

28

There were a total of 26,021 admissions to RPHICU between 1987 and 2002

with a hospital survival rate of 89.2%. The power of the survival analysis of ICU

cohorts depends on the total number of outcome events, that is, mortality. Power

calculations for the survival analysis showed that a total of 200 events provides an

88% power to detect a relative risk (RR) of 1.25 for a continuous risk factor (eg

APACHE score or predicted mortality, Charlson co-morbidity index), a total of 200

events provided around 95% power to detect a relative risk of 2.0 for a binary risk

factor (eg the presence of a certain co-morbidity) that has a prevalence of 10%, and a

total of 250 events provides an 86% power to detect a RR of 1.5 for a binary risk

factor that has a prevalence between 25 and 50% (eg gender, received mechanical

ventilation). Therefore, the sample size of this cohort would be adequate for

predictive modelling with multiple predictive variables.

Ethics approval

The RPHICU database contains clinical data that are identified by patient

medical record numbers. Patient medical record numbers were used to retrieve the

data on inflammatory markers of some ICU admissions in two studies that evaluated

the association between inflammatory markers and outcomes after ICU discharge.

The conduct of the studies was reviewed and found to meet with the approval of the

Hospital Ethics Committee or representative on the basis that they were clinical

audits. For studies involving linked data and data from the Aboriginal patients, they

were approved by the WA Confidentiality of Health Information Committee (CHIC)

and the WA Aboriginal Health Information and Ethics Committee, respectively.

(Appendices on page 156-160).

29

Section two: Assessment of the APACHE II scoring

system in an Australian context

Chapter 3. The worst first 24-hour and admission APACHE II

scoring system

The APACHE II scoring system is widely used in many Australian ICUs for audit

and clinical research purposes.54,55

The APACHE II scoring system has, however, not

been thoroughly evaluated in an Australian ICU. In this chapter, the performance of the

APACHE II scoring system in a tertiary Australian ICU was evaluated. The results

showed that the performance of the APACHE II scoring system was similar to the

original APACHE II cohort and reports from other ICUs.24

The overall discrimination as

measured by area under the ROC curve was 0.85 with 95% confidence interval [CI]:

0.84-0.86 and the Standardised Mortality Ratio [SMR] was 0.84 with 95%CI: 0.80-0.88.

The performance of the APACHE II model in RPHICU also appeared to be stable

between 1993 and 2003 without significant changes over time as reported by some other

ICUs. This was different from what had been described in other ICUs. The possible

explanations may include the high quality data collection process of the RPHICU since

the inception of the database in 1987, stability in case mix at RPHICU, or changes in case

mix being offset by improvement in care.68

When the performance of the APACHE II scoring system in different diagnostic

subgroups was considered, the discrimination of the scoring system was least satisfactory

in two major diagnostic subgroups; the subgroup with sepsis, pneumonia, gastrointestinal

obstruction / perforation for which the area under ROC curve was 0.68 and also the

30

cardiac arrest subgroup for which the area under the ROC curve was 0.74. Using the

SMR as a guide to assess the model calibration across different subgroups, the APACHE

II scoring system appeared to be least well calibrated in the subgroups of patients with

multiple trauma (SMR 1.24, 95%CI: 1.17-1.31) and those transferred directly to RPHICU

from another hospital (SMR 0.71, 95%CI: 0.67-0.75).

The traditional APACHE II scoring system uses physiological data values

recorded as the worst values over the first 24 hours after ICU admission. The collection

of physiological data on admission only is logistically easier and is used by some ICUs.

The performance of a modified APACHE II scoring system using the admission

physiology and laboratory data to replace the worst first 24-hour data (the admission

APACHE II scoring system) was assessed and compared with the traditional worst first

24-hour APACHE II scoring system. The discrimination of the admission APACHE II

scoring system both overall and within different diagnostic subgroups was considered

satisfactory. The calibration of the admission APACHE II scoring system, as illustrated

by the calibration curve and SMR, was better than the worst first 24-hour APACHE II

scoring system in most patient subgroups. The only exception was the group of patients

with multiple trauma among whom the model calibration deteriorated by using the

admission physiology and laboratory data.

While there are some limitations with the admission APACHE II scoring

system as discussed in this study, the results showed that it is valid to use this modified

APACHE II scoring system as an alternative risk adjustment tool for critically ill non-

trauma patients.

31

In conclusion, the APACHE II scoring system had a satisfactory overall

performance in a major tertiary Australian ICU and its performance was stable between

1993 and 2003. Its performance was, however, not uniform across all different patient

subgroups. The scoring system’s discrimination appeared to be least satisfactory among

patients with sepsis, pneumonia, gastrointestinal obstruction / perforation and cardiac

arrest. Using admission physiology and laboratory data to replace the worst first 24-hour

data to generate the admission APACHE II predicted mortality (the admission APACHE

II scoring system) appeared to be a viable and simple alternative to the worst 24-hour

APACHE II scoring system as an audit and risk adjustment tool for critically ill non-

trauma patients.

The details of this study are contained in the following published article:

Ho KM, Dobb GJ, Knuiman M, Finn J, Lee KY, Webb SA. A comparison of

admission and worst 24-hour Acute Physiology and Chronic Health Evaluation II scores

in predicting hospital mortality: a retrospective cohort study. Critical Care 2006;10:R4.

Available online http://ccforum.com/content/10/1/R4

Open AccessVol 10 No 1ResearchA comparison of admission and worst 24-hour Acute Physiology and Chronic Health Evaluation II scores in predicting hospital mortality: a retrospective cohort studyKwok M Ho1,2,3, Geoffrey J Dobb4,5, Matthew Knuiman6, Judith Finn7, Kok Y Lee8 and Steven AR Webb8,9

1Consultant Intensivist Department of Intensive Care, Royal Perth Hospital, Wellington street, Perth, WA 6000, Australia2PhD candidate, School of Population Health, University of Western Australia, Crawley, Perth, WA 6009, Australia3PhD candidate, School of Medicine and Pharmacology, University of Western Australia, Crawley, Perth, WA 6009, Australia4Acting Head of the Department, Department of Intensive Care, Royal Perth Hospital, Wellington street, Perth, WA 6000, Australia5Associate Professor, School of Medicine and Pharmacology, University of Western Australia, Crawley, Perth, WA 6009, Australia6Professor, School of Population Health, University of Western Australia, Crawley, Perth, WA 6009, Australia7Senior Lecturer, School of Population Health, University of Western Australia, Crawley, Perth, WA 6009, Australia8Consultant Intensivist, Department of Intensive Care, Royal Perth Hospital, Wellington street, Perth, WA 6000, Australia9Senior Lecturer, School of Medicine and Pharmacology, University of Western Australia, Crawley, Perth, WA 6009, Australia

Corresponding author: Kwok M Ho, [email protected]

Received: 17 Aug 2005 Revisions requested: 26 Sep 2005 Revisions received: 6 Oct 2005 Accepted: 26 Oct 2005 Published: 25 Nov 2005

Critical Care 2006, 10:R4 (doi:10.1186/cc3913)This article is online at: http://ccforum.com/content/10/1/R4© 2005 Ho et al.; licensee BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Introduction The Acute Physiology and Chronic HealthEvaluation (APACHE) II score is widely used in the intensivecare unit (ICU) as a scoring system for research and clinicalaudit purposes. Physiological data for calculation of theAPACHE II score are derived from the worst values in the first24 hours after admission to the ICU. The collection ofphysiological data on admission only is probably logisticallyeasier, and this approach is used by some ICUs. This studycompares the performance of APACHE II scores calculatedusing admission data with those obtained from the worst valuesin the first 24 hours.

Materials and Methods This was a retrospective cohort studyusing prospectively collected data from a tertiary ICU. Therewere no missing physiological data and follow-up for mortalitywas available for all patients in the database. The admission andthe worst 24-hour physiological variables were used to generatethe admission APACHE II score and the worst 24-hourAPACHE II score, and the corresponding predicted mortality,respectively.

Results There were 11,107 noncardiac surgery ICU admissionsduring 11 years from 1 January 1993 to 31 December 2003.The mean admission and the worst 24-hour APACHE II scorewere 12.7 and 15.4, and the derived predicted mortalityestimates were 15.5% and 19.3%, respectively. The actualhospital mortality was 16.3%. The overall discrimination ability,as measured by the area under the receiver operatingcharacteristic curve, of the admission APACHE II model(83.8%, 95% confidence interval = 82.9–84.7) and the worst24-hour APACHE II model (84.6%, 95% confidence interval =83.7–85.5) was not significantly different (P = 1.00).

Conclusion Substitution of the worst 24-hour physiologicalvariables with the admission physiological variables to calculatethe admission APACHE II score maintains the overalldiscrimination ability of the traditional APACHE II model. Theadmission APACHE II model represents a potential alternativemodel to the worst 24-hour APACHE II model in critically illnontrauma patients.

IntroductionScoring systems such as Acute Physiology and ChronicHealth Evaluation (APACHE), the Therapeutic Intervention

Scoring System, and Mortality Probability Models (MPM) havebeen developed and used as quality assurance tools and forrisk stratification in research involving critically ill patients [1,2].

Page 1 of 8(page number not for citation purposes)

APACHE = Acute Physiology and Chronic Health Evaluation; CI = confidence interval; ICU = intensive care unit; MPM = Mortality Probability Models; SAPS = Simplified Acute Physiology Score.

http://ccforum.com/content/10/1/R4

http://creativecommons.org/licenses/by/2.0

http://www.biomedcentral.com/info/about/charter/

Critical Care Vol 10 No 1 Ho et al.

Each scoring system has its own strengths and weaknesses,and the choice depends on the system's ease of use andgoodness of fit for that particular intensive care unit (ICU) orpatient group.

The traditional APACHE II model utilises the worst values of12 physiological variables during the first 24 hours followingICU admission, along with an evaluation of the patient'schronic health and admission diagnosis to calculate theAPACHE II predicted mortality [3]. The APACHE II model hasbeen widely validated and used by many ICUs to classify theseverity of illness and to predict hospital mortality [2,4-7].APACHE II has now been modified to APACHE III; however,some studies have shown that APACHE III may underestimatethe number of deaths [8,9]. Although the APACHE II model isquite old, and other scoring systems have been developedusing more recent cohorts, APACHE II is still widely used forresearch and clinical audit purposes. APACHE II is easier touse than APACHE III and has been in use for a long period,which allows consistency [2,10].

A potential problem with these methods is that the worst 24-hour physiological data used to derive APACHE II scores andAPACHE III scores can be treatment-dependent and thereforeit may reflect poor clinical management rather than sickerpatients [11-13]. Collection of the admission physiologicalvariables rather than the worst 24-hour physiological variablesis a standard practice in some ICUs to calculate the APACHEII predicted mortality, and may theoretically overcome thispotential problem [14,15]. The use of only admission physio-logical variables may make data collection easier as the datacollector does not need to peruse all the blood tests and phys-iological variables over 24 hours to work out the worst score.However, the performance of APACHE II scores using admis-sion data has not been thoroughly assessed [3,16].

When the APACHE III scoring system was developed, theeffect of using admission physiological variables rather thanthe worst 24-hour physiological variables was assessed. Theabsolute difference between the mean scores, derived fromthe admission and worst 24-hour physiological data, was notstatistically significantly different from zero [16]. However, theproportion of missing values favoured the worst 24-hour val-ues over the admission values, as did the maximum explana-tory power. Some other scoring systems use only admissiondata (MPM II0 and Simplified Acute Physiology Score [SAPS]III), and it is therefore established that scoring systems usingphysiological data from the time of admission to the ICU canprovide valid assessment of the severity of illness and out-come prediction [17,18].

In the present study we evaluated the performance of theAPACHE II model using physiological data at the time of ICUadmission with the model using data obtained from the worstvalues in the first 24 hours.

Materials and methodsThis was a retrospective cohort study that utilised prospec-tively collected data. The study was conducted in the medical–surgical ICU at Royal Perth Hospital, an 800-bed universityteaching hospital. The 22-bed ICU is a 'closed' ICU thatadmits critically ill adult patients of all specialties and is staffedby fully trained intensivists. The unit database contains de-identified information for components of the APACHE II scorefor physiological data collected at admission and for the worstvalues in the first 24 hours – admission diagnosis and source,age, ethnicity, ICU mortality and hospital mortality. The admis-sion and the worst 24-hour physiological data were used togenerate the admission APACHE II score and the worst 24-hour APACHE II score, respectively. The admission APACHEII score and the worst 24-hour APACHE II score were thenused to calculate the admission APACHE II predicted mortal-ity (admission APACHE II model) and the worst 24-hour pre-dicted mortality (worst 24-hour APACHE II model), using thepublished APACHE II mortality prediction equation coeffi-cients [3].

The data were collected by the duty ICU consultant on papersheets and updated on a daily basis by the duty consultantwhile the patient remained in the ICU. After the patient wasdischarged from the ICU, the data were checked for transcrip-tion errors and completeness by a designated trained clericalstaff member using data from the computerised laboratorydatabase, going through the ICU vital signs flow chart againbefore the data were transferred to the computer. A total of 12consultants were involved in collecting data, of which sevenwere involved throughout the study period, using a standard-ised data dictionary. The worst 24-hour APACHE II score wasdetermined precisely as described by Knaus and colleagues[3].

Measurement of all 12 physiological variables on admissionand over the first 24 hours in the ICU was mandatory in theAPACHE data recording form. If the patient was anaesthe-tised before ICU admission, the Glasgow coma score wasassessed using the available clinical information prior toanaesthesia. Acute renal failure was defined as oliguria withurine output less than 135 ml over a consecutive 8-hour periodwith abnormal serum creatinine concentrations over 133µmol/l. Other than the Glasgow coma score and urinary out-put, pre-ICU physiological data were not used in the calcula-tion of APACHE II scores. Arterial blood gas measurementswere judged to be inappropriate in some patients, and in thesepatients the serum bicarbonate concentration was used to cal-culate the physiological score [3]. One data custodian wasresponsible for ensuring data quality throughout the studyperiod. The data were reviewed for internal consistency beforeannual lockdown, and there were no patients with missingphysiological data or who were lost to mortality follow-up. Thestudy utilised de-identified data only and was deemed to be a



'Clinical Audit' by the Hospital Ethics Committee and as suchthe need for formal ethics committee approval was waived.

The performance of the admission APACHE II model in pre-dicting hospital mortality was compared with the performanceof the worst 24-hour APACHE II model with respect to theirdiscrimination ability and calibration. Because the originalAPACHE II prediction model did not include cardiac surgicalpatients, we have included only the data from noncardiac sur-gery ICU admissions. All patients in the database in the studyperiod were considered, including those patients who diedwithin 24 hours of ICU admission.

The discrimination ability of each of the scoring systems wasassessed by the area under the receiver operating character-istic curve: above 90% was regarded as excellent, above 80%

was regarded as good, and below 80% was regarded as poorin this study. Calibration was assessed by comparing absoluteobserved mortality with predicted mortality in fixed risk strata(for example 0–0.099, 0.1–0.199, and so on) using the Hos-mer-Lemeshow chi-square H statistic. P < 0.05 in the Hos-mer-Lemeshow chi-square H statistical test infers a significantdeparture from the null hypothesis of good calibration. Therelationship between the admission APACHE II predicted hos-pital mortality risk and the worst 24-hour APACHE II predictedhospital mortality risk was assessed by the two-tailed Pearsoncorrelation coefficient. The ratio of total observed to predictedmortality is the standardised mortality ratio (SMR).

The discrimination ability was further analysed for differentdiagnostic and patient subgroups to test the uniformity of fit ofboth models. The diagnostic subgroups analysed included

Table 1

Characteristics of the cohort

Variables Mean (SD)

Age (years) 53.5 (19.5)

Male/female (%) 6,871/4,236 (61.9/38.1)

Admission source (%)

Operating room 4,885 (44.0)

Recovery room 638 (5.7)

Emergency department 2,976 (26.8)

Ward 1,481 (13.3)

Another hospital 1,127 (10.1)

Primary organ failure (%)

Cardiovascular 3,693 (33.2)

Neurological 3,893 (35.0)

Respiratory 2,682 (24.1)

Gastrointestinal 401 (3.6)

Renal 167 (1.5)

Metabolic 217 (2.0)

Haematological 49 (0.4)

ICU stay (days) 5.1 (7.8)

Hospital stay (days) 21.1 (29.3)

Admission APACHE II score 12.7 (7.3)

Worst 24-hour APACHE II score 15.4 (7.9)

Admission APACHE predicted mortality (%) 15.5 (19.1)

Worst 24-hour APACHE predicted mortality (%) 19.3 (22.1)

Actual ICU mortality (%) 12.0

Actual hospital mortality (%) 16.3

All data in parentheses are standard deviations unless stated otherwise. APACHE, Acute Physiology and Chronic Health Evaluation; ICU, intensive care unit; SD, standard deviation.



patients with different major diagnoses such as sepsis, pneu-monia, and gastrointestinal perforation or obstruction, intracra-nial haemorrhage, multiple trauma, cardiac arrest, and electivesurgery. The patient subgroups analysed included aboriginalpatients, patients transferred from another hospital, patientsadmitted to the ICU before or after early 1999, patients whostayed in the ICU longer than 24 hours, and patients who sur-vived longer than 24 hours of hospitalisation. P < 0.05 wasregarded as significant in all analyses and no adjustment wasmade for multiple comparisons in the subgroup analyses. Allstatistical analyses were performed by SPSS statistical soft-ware (version 11.0 for Windows; SPSS Inc., Chicago, IL,USA] and confidence intervals were generated by ConfidenceInterval Analysis (version 2.0.0; BMJ 2000, UK).

ResultsThe time for collecting and checking the admission physiolog-ical data manually required an average of 5 minutes per patient(range, 3–7 minutes), and the average for the worst 24-hourphysiological data was 20 minutes per patient (range, 10–40minutes). The time required to work out the worst 24-hourAPACHE II score was longer when more blood tests had beenperformed for the patient.

There were 11,107 noncardiac surgery ICU admissions in the11-year period from 1 January 1993 to 31 December 2003.The characteristics of the ICU cohort are presented in Table 1.The difference in the admission APACHE II score and theworst 24-hour APACHE II score was small in most patients(Figure 1). The mean admission APACHE II score and theworst 24-hour APACHE II scores were 12.7 and 15.4, and thederived predicted hospital mortality estimates were 15.5%and 19.3%, respectively. The admission APACHE II predictedmortality and the worst 24-hour APACHE II predicted mortalitywere closely correlated (Pearson correlation coefficient =0.955, P = 0.0001). The actual hospital mortality was 16.3%.The overall standardised mortality ratio was 1.05 (95% confi-dence interval [CI] = 1.00–1.10) and was 0.84 (95% CI =0.80–0.88) using the admission APACHE II predicted mortal-ity and the worst 24-hour APACHE II predicted mortality as thedenominator, respectively.

The overall discrimination abilities, as measured by the areaunder the receiver operating characteristic curve, of the admis-sion APACHE II model (83.8%, 95% CI = 82.9–84.7) and theworst 24-hour APACHE II model (84.6%, 95% CI = 83.7–85.5) with the entire cohort were not significantly different (P

Table 2

The discriminating ability of the admission Acute Physiology and Chronic Health Evaluation (APACHE) II model and the worst 24-hour APACHE II model to predict inhospital mortality in different diagnostic and patient subgroups

Different diagnostic and patient subgroups

Number of patients

Mean area under the ROC curve (%) (95% confidence interval)a

Standardised mortality ratio (95% confidence interval)

Admission model Worst 24-hour model Admission model Worst 24-hour model

Sepsis, pneumonia, gastrointestinal perforation or obstruction

1,474 68.3 (65.4–71.3) 68.5 (65.6–71.4) 0.94 (0.90–0.98) 0.77 (0.75–0.80)

Intracranial, subdural or subarachnoid haemorrhage

851 79.5 (76.3–82.7) 80.4 (77.2–83.5) 1.29 (1.22–1.36) 1.03 (0.98–1.08)

Multiple trauma 1,299 87.0 (84.1–89.9) 87.3 (84.4–90.1) 1.73 (1.63–1.84) 1.24 (1.17–1.31)

Cardiac arrest (nonoperative or intraoperative)

395 73.9 (69.1–78.8) 73.9 (69.0–78.8) 0.92 (0.88–0.96) 0.82 (0.79–0.85)

Elective surgery (excluding cardiac surgery)

3,012 78.6 (74.8–82.4) 80.8 (77.3–84.4) 1.04 (1.00–1.09) 0.79 (0.76–0.83)

Aboriginal patients 863 77.8 (74.2–81.4) 78.8 (75.2–82.3) 1.02 (0.95–1.09) 0.82 (0.77–0.87)

Patients transferred from another hospital

1,127 79.4 (76.3–82.4) 80.4 (77.4–83.5) 0.87 (0.82–0.92) 0.71 (0.67–0.75)

Patients admitted between 1993 and early 1999

5,553 85.4 (84.0–86.7) 86.1 (84.8–87.4) 1.05 (1.01–1.09) 0.85 (0.82–0.88)

Patients admitted between early 1999 and 2003

5,554 83.3 (82.0–84.5) 84.1 (82.8–85.3) 1.09 (1.06–1.13) 0.88 (0.86–0.91)

Patients stayed in the ICU longer than 24 hours

8,461 80.4 (79.2–81.5) 81.2 (80.1–82.3) 0.99 (0.97–1.02) 0.79 (0.77–0.81)

Patients survived longer than 24 hours of hospitalisation

10,733 82.2 (81.1–83.2) 83.0 (82.0–84.0) 0.93 (0.91–0.95) 0.74 (0.73–0.76)

aThere was no significant difference in the areas under the receiver operating characteristic (ROC) curves between the admission APACHE II model and the worst 24-hour APACHE II model (P = 1.00).



= 1.00) (Figure 2). The discrimination abilities of the admissionAPACHE II model and the worst 24-hour APACHE II modelwere also not significantly different within all subgroups ana-lysed (Table 2).

The Hosmer and Lemeshow goodness of fit chi-square H sta-tistic was 66.7 for the admission APACHE II model and was189.3 for the worst 24-hour APACHE II model indicating abetter fit for the admission APACHE II model but both P valueswere very small (P < 0.0001). The calibration curve of the two

APACHE II models is displayed in Figure 3 and shows the bet-ter fit of the admission APACHE II model especially in the highrisk strata. The overall correct classification rate (based onclassifying a patient to die if his/her predicted mortality riskexceeded 50%) for the admission APACHE II model and theworst 24-hour APACHE II model were both 85.4% (Table 3).

DiscussionThe advantages of the admission APACHE II modelOur results showed that the performance of the admissionAPACHE II model is no worse than the traditional worst 24-hour APACHE II model when there are no significant missingdata. These results were consistent with the results of otherstudies that assessed or utilised the admission APACHE IIscore to calculate the APACHE II predicted mortality [15-17].

The use of the admission APACHE II score to calculate theAPACHE II predicted mortality (admission APACHE II model)has a few potential advantages and may represent a viablealternative to the traditional APACHE II model. First, it canassess the risk of hospital death at ICU admission, as in theMPM II0 and SAPS III scoring systems that assess the risk ofhospital death at ICU admission [17,18]. The admissionAPACHE II model also shares these systems' advantages ofease of use, and, since they are independent of ICU treatment,may be more applicable for risk stratification in clinicalresearch and triage decisions [19]. The ability of a scoring sys-tem to stratify patient risk on admission to the ICU mayfacilitate stratification of patients into trials that assess earlyinterventions in critically ill patients.

Second, the data collection for the admission APACHE IImodel is less laborious than the worst 24-hour APACHE IImodel, as demonstrated in our data. It may also reduce errorsbecause it does not require perusal of a series of values toobtain the worst score. Nevertheless, this potential advantageis important only when a computerised information system isnot available and the data are collected manually.

Third, the admission APACHE II model may be a better reflec-tion of quality of care in the ICU because risk assessmentoccurs before any ICU therapy is instituted [12-14].

Finally, poor calibration with the worst 24-hour APACHE IImodel has been reported in many studies [20-22]. Our resultsconfirmed this problem of the worst 24-hour APACHE IImodel, with the predicted mortality being much higher than theactual mortality in the high-risk strata. The admission APACHEII model appeared to have reduced the overestimation of mor-tality in the high-risk strata and improved the calibration of theAPACHE II model in the present study. However, data oncalibration of the admission APACHE II model from otherstudies are lacking [15-17] and further studies in other set-tings will be needed to confirm this finding.

Figure 1

The difference in APACHE II scores using the admission and worst 24-hour physiological dataThe difference in APACHE II scores using the admission and worst 24-hour physiological data. AP, Acute Physiology and Chronic Health Evaluation.

Figure 2

The receiver operating characteristic (ROC) curves for the admission Acute Physiology and Chronic Health Evaluation (APACHE) II model and the worst 24-hour APACHE II model in predicting hospital mortalityThe receiver operating characteristic (ROC) curves for the admission Acute Physiology and Chronic Health Evaluation (APACHE) II model and the worst 24-hour APACHE II model in predicting hospital mortal-ity. Area under ROC curves: worst 24-hour APACHE II model, 84.6% (95% CI = 83.7–85.5); admission APACHE II model, 83.8% (95% CI = 82.9–84.7). No significant difference between the two areas under the ROC curves (P = 1.00).



Limitations of the admission APACHE II modelThe admission APACHE II model is a minor modification of theworst 24-hour APACHE II model and retains many intrinsicweaknesses and problems of the worst 24-hour APACHE IImodel. These weaknesses include errors arising from impre-cise principal diagnosis, lead time bias, and poor uniformity offit of the model. The admission APACHE II model, as withother ICU scoring systems such as the APACHE III model,needs an accurate diagnosis to accurately predict the hospitalmortality. The admission APACHE II model does not eliminatethis requirement.

The performance of the worst 24-hour APACHE II model isaffected by the source and timing of patient referral to the ICU,and it tends to underestimate the mortality of the patientsreferred from other ICUs or hospitals [23,24]. Our results weredifferent from these reports. This may be because manypatients were transferred from remote Western Australia andwere not fully resuscitated when they were admitted to theICU. The standardised mortality ratio of the patients trans-ferred from other hospitals, based on the admission APACHEII model in this study, was closer to unity than that of the worst24-hour APACHE II model (Table 2). The admission APACHEII model was associated with a lower lead time bias in thisstudy. The uniformity of fit in the discrimination ability of theadmission APACHE II model and the worst 24-hour APACHEII model was similarly poor in patients with sepsis, pneumonia,gastrointestinal perforation, and cardiac arrest, and also in theaboriginal patients. Both the worst 24-hour APACHE II modeland the APACHE III model were not well calibrated in predict-ing mortality in trauma patients [23,25,26]. Our results con-firmed this problem of the worst 24-hour APACHE II model,and the admission APACHE II model did not improve the per-formance of the worst 24-hour APACHE II model in this sub-group of patients.

Limitations of the studyThis was a single-centre study and these results may not begeneralisable to other ICUs [23]. Our observation that thestandardised mortality ratio calculated with the admissionphysiological variables was closer to unity than that calculatedwith the worst 24-hour values may be different in other units.Further evaluation of the admission APACHE II model in otherICUs is essential.

Also, this study did not directly compare the admissionAPACHE II model with other scoring systems that assess therisk of hospital mortality at ICU admission such as the MPM II0

Table 3

Classification table for the admission Acute Physiology and Chronic Health Evaluation (APACHE) II model and the worst 24-hour APACHE II model to predict hospital mortality

Observed hospital mortality Predicted hospital mortality

No (n) Yes (n) % correct

Using the worst 24-hour APACHE II model

No 8,899 394 95.8

Yes 1,229 585 32.2

Overall percentage 85.4

Using the admission APACHE II model

No 8,966 327 96.5

Yes 1,293 521 28.7

Overall percentage 85.4

The cutoff value is 0.50.

Figure 3

Calibration curves for the admission Acute Physiology and Chronic Health Evaluation (APACHE) II score and the worst 24-hour APACHE II score in predicting hospital mortality across different risk strataCalibration curves for the admission Acute Physiology and Chronic Health Evaluation (APACHE) II score and the worst 24-hour APACHE II score in predicting hospital mortality across different risk strata. The Hosmer-Lemeshow goodness of fit chi-square H statistic for the admis-sion APACHE II predicted mortality and for the worst 24-hour APACHE II predicted mortality were 66.9 and 189.3, respectively (both P < 0.0001).



and SAPS III models [17,18]. Whether the performance of theadmission APACHE II model is comparable with these scoringsystems remains uncertain and will be further investigated.

Critical illness is a dynamic process and therefore outcomeprediction based on a single time point such as ICU admis-sion, as in the admission APACHE II model, does not considerchanges in patients' clinical status over time and theirresponse to treatment. Serial predictions over a period of time,as in the APACHE III model, may improve prediction accuracyand clinical utilities, although acquiring these data continu-ously will be difficult in practice [27,28].

Finally, the admission APACHE II model, as with most otheroutcome prediction models, does not consider functional out-comes beyond survival [9].

ConclusionIn conclusion, substituting the worst 24-hour physiologicalvariables with the admission physiological variables to calcu-late the admission APACHE II score and the APACHE II pre-dicted mortality does not result in significantly worsecalibration or discrimination compared with the traditionalAPACHE II model. The admission APACHE II modelrepresents a potential alternative model to the worst 24-hourAPACHE II model in critically ill nontrauma patients.

Competing interestsThe authors declare that they have no competing interests.

Authors' contributionsKMH performed the statistical analysis and drafted the manu-script. GJD initiated the original idea of the study and helpedto draft the manuscript. MK, JF, and SARW helped analyse thedata and draft the manuscript. KYL was the data-collectionquality controller and helped to draft the manuscript. Allauthors read and approved the final manuscript.

AcknowledgementsThe authors would like to thank Dr Geoffrey Clarke and Dr John Weekes for their part in initiating the Royal Perth Hospital ICU database, and thank all ICU consultants who have been recording APACHE II data for every admission to the ICU. This study was solely funded by the Depart-ment of Intensive Care, Royal Perth Hospital.

References1. Knaus WA: APACHE 1978–2001: the development of a quality

assurance system based on prognosis: milestones and per-sonal reflections. Arch Surg 2002, 137:37-41.

2. Gunning K, Rowan K: ABC of intensive care: outcome data andscoring systems. BMJ 1999, 319:241-244.

3. Knaus WA, Draper EA, Wagner DP, Zimmerman JE: APACHE II: aseverity of disease classification system. Crit Care Med 1985,13:818-829.

4. Oh TE, Hutchinson R, Short S, Buckley T, Lin E, Leung D: Verifi-cation of the Acute Physiology and Chronic Health Evaluationscoring system in a Hong Kong intensive care unit. Crit CareMed 1993, 21:698-705.

5. Livingston BM, MacKirdy FN, Howie JC, Jones R, Norrie JD:Assessment of the performance of five intensive care scoringmodels within a large Scottish database. Crit Care Med 2000,28:1820-1827.

6. Breen D, Churches T, Hawker F, Torzillo PJ: Acute respiratoryfailure secondary to chronic obstructive pulmonary diseasetreated in the intensive care unit: a long term follow up study.Thorax 2002, 57:29-33.

7. Rowan KM, Kerr JH, Major E, McPherson K, Short A, Vessey MP:Intensive Care Society's Acute Physiology and Chronic HealthEvaluation (APACHE II) study in Britain and Ireland: a prospec-tive, multicenter, cohort study comparing two methods for pre-dicting outcome for adult intensive care patients. Crit CareMed 1994, 22:1392-1401.

8. Buist M, Gould T, Hagley S, Webb R: An analysis of excess mor-tality not predicted to occur by APACHE III in an Australianlevel III intensive care unit. Anaesth Intensive Care 2000,28:171-177.

9. Angus DC: Scoring system fatigue...and the search for a wayforward. Crit Care Med 2000, 28:2145-2146.

10. Konarzewski W: Continuing to use APACHE II scores ensuresconsistency. BMJ 2000, 321:383-384.

11. Shann F: Mortality prediction model is preferable to APACHE.BMJ 2000, 320:714.

12. Boyd O, Grounds RM: Physiological scoring systems and audit.Lancet 1993, 341:1573-1574.

13. Knaus W, Draper E, Wagner D: APACHE III study design: ana-lytic plan for evaluation of severity and outcome in intensivecare unit patients. Introduction. Crit Care Med 1989,17:S176-S180.

14. Khilnani G, Banga A, Sharma S: Predictors of mortality ofpatients with acute respiratory failure secondary to chronicobstructive pulmonary disease admitted to an intensive careunit: a one year study. BMC Pulm Med 2004, 4:12. it is a full arti-cle but no page span because it does not have printed version,only Internet version

15. Goel A, Pinckney RG, Littenberg B: APACHE II predicts long-term survival in COPD patients admitted to a general medicalward. J Gen Intern Med 2003, 18:824-830.

16. Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M,Bastos PG, Sirio CA, Murphy DJ, Lotring T, Damiano A, et al.:APACHE III prognostic system. Risk prediction of hospitalmortality for critically ill hospitalized adults. Chest 1991,100:1619-1636.

17. Metnitz PG, Moreno RP, Almeida E, Jordan B, Bauer P, CamposRA, Iapichino G, Edbrooke D, Capuzzo M, Le Gall JR, on behalf ofthe SAPS 3 Investigators: SAPS 3-From evaluation of thepatient to evaluation of the intensive care unit. Part 1: Objec-tives, methods and cohort description. Intensive Care Med2005, 31:1336-1344.

18. Lemeshow S, Teres D, Klar J, Avrunin JS, Gehlbach SH, RapoportJ: Mortality Probability Models (MPM II) based on an interna-tional cohort of intensive care unit patients. JAMA 1993,270:2478-2486.

19. Joynt GM, Gomersall CD, Tan P, Lee A, Cheng CA, Wong EL: Pro-spective evaluation of patients refused admission to an inten-sive care unit: triage, futility and outcome. Intensive Care Med2001, 27:1459-1465.

20. Carson SS, Bach PB: Predicting mortality in patients sufferingfrom prolonged critical illness: an assessment of four severity-of-illness measures. Chest 2001, 120:928-933.

Key messages

• Modifying the APACHE II model using admission physi-ological variables instead of worst 24-hour physiological variables to calculate the APACHE II score and pre-dicted mortality (admission APACHE II model) does not result in significantly worse calibration and discrimina-tion compared with the traditional APACHE II model in critically ill nontrauma patients.


http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=11772212



















































21. Tan IK: APACHE II and SAPS II are poorly calibrated in a HongKong intensive care unit. Ann Acad Med Singapore 1998,27:318-322.

22. Arabi Y, Al Shirawi N, Memish Z, Venkatesh S, Al-Shimemeri A:Assessment of six mortality prediction models in patientsadmitted with severe sepsis and septic shock to the intensivecare unit: a prospective cohort study. Crit Care 2003,7:R116-R122.

23. Cowen JS, Kelly MA: Errors and bias in using predictive scoringsystems. Crit Care Clin 1994, 10:53-72.

24. Combes A, Luyt CE, Trouillet JL, Chastre J, Gibert C: Adverseeffect on a referral intensive care unit's performance ofaccepting patients transferred from another intensive careunit. Crit Care Med 2005, 33:705-710.

25. Zimmerman JE, Wagner DP, Draper EA, Wright L, Alzola C, KnausWA: Evaluation of acute physiology and chronic health evalu-ation III predictions of hospital mortality in an independentdatabase. Crit Care Med 1998, 26:1317-1326.

26. Chawda MN, Hildebrand F, Pape HC, Giannoudis PV: Predictingoutcome after multiple trauma: which scoring system? Injury2004, 35:347-358.

27. Afessa B, Keegan MT, Mohammad Z, Finkielman JD, Peters SG:Identifying potentially ineffective care in the sickest critically illpatients on the third ICU day. Chest 2004, 126:1905-1909.

28. Wagner DP, Knaus WA, Harrell FE, Zimmerman JE, Watts C: Dailyprognostic estimates for critically ill adults in intensive careunits: results from a prospective, multicenter, inception cohortanalysis. Crit Care Med 1994, 22:1359-1372.























40



Chapter 4. The use of the APACHE II scoring system for the

indigenous patients

Indigenous Australians are over-represented in ICU admissions in the Northern

Territory (28% of the population but 45% of all ICU admissions),69

but there is little

information on their pattern of critical illness and outcomes from other parts of Australia.

The APACHE II scoring system has been used for risk adjustment purposes for many

critically ill patients and has a reasonable discrimination with the RPHICU cohort. This

study hypothesised that the APACHE II scoring system will have a similar performance

when applied to critically ill indigenous and non-indigenous patients.

This chapter examines and compares the patterns of critical illness of the

indigenous patients with the non-indigenous patients in the Western Australia RPHICU

cohort and assesses whether the APACHE II scoring system is a reasonable risk

adjustment tool for critically ill indigenous patients. First, indigenous Australians were

over-represented in RPHICU admissions comprising 3.2% of the population but 6.4% of

all ICU admissions. The pattern of their critical illness was also very different from other

patients. Compared with non-indigenous patients, the indigenous patients were younger

and more likely to have been transferred from another hospital. ICU admissions due to

respiratory or renal failure, sepsis, pneumonia, trauma, and cardiopulmonary arrest with a

higher severity of acute illness were more common among indigenous patients leading to

41

a significantly longer length of ICU stay. Chronic liver and renal diseases were also more

common among the indigenous patients.

Second, during the 11 year-period from 1993 to 2003 there was a progressive

increase in emergency ICU admissions among indigenous patients. If emergency ICU

admission is regarded as an “ambulance at the bottom of a cliff” with many patients

admitted to the ICU only after other layers of the health care system have failed to

reverse or prevent the critical illness, this result raised the concern that primary and

preventive care services for the indigenous patients might have deteriorated over this

period. Another possible explanation for the increase in emergency admissions would be

better access to intensive care service in WA.

Third, with the difference in sample size in mind, the APACHE II scoring system

has a similar calibration curve for indigenous and non-indigenous patients, especially in

patients with less severe acute diseases (predicted mortality <50%). The discrimination of

the model for critically ill indigenous patients (area under the ROC curve 0.79, 95%CI:

0.75-0.82) was, however, slightly less satisfactory than for non-indigenous patients. This

was most likely due to the different disease pattern of the indigenous patients, among

whom sepsis, pneumonia, cardiac arrest, and transfer from another hospital were very

common. As discussed in Chapter 3, the overall discrimination of the APACHE II

scoring system was also less satisfactory among these diagnostic and patient subgroups.

In conclusion, as for many other illnesses, the pattern of ICU utilisation differs

between indigenous and non-indigenous Australians. Critical illnesses requiring

emergency ICU admission are increasing among indigenous Western Australians. The

performance of the APACHE II scoring system appears to be less satisfactory among the

42

indigenous Australians, and this is most likely due to a larger proportion of them being

transferred from another hospital and the different disease pattern. Therefore, the

limitations of the APACHE II scoring system should be considered when the model is

used as a risk adjustment tool for critically ill indigenous Australians.

Further details of this study are contained in the following published article:

Ho KM, Finn J, Dobb GJ, Webb SA. The outcome of critically ill Indigenous

patients. Medical Journal of Australia 2006;184:496-9.


47



Chapter 5. Assessing calibration by meta-analytic techniques

It is important to understand that the performance of a scoring system may vary

significantly depending on the characteristics of the cohort among whom the model is

applied. This may explain why a prognostic scoring system can be reported to perform

very well in some ICUs and not so well in other ICUs. The results in Chapters 3 and 4

demonstrated that the performance of the APACHE II scoring system was less

satisfactory in some diagnostic and patient subgroups in the RPHICU cohort.

Most researchers use Standardised Mortality Ratio (SMR) of different subgroups

of patients to illustrate whether a prognostic scoring system is well calibrated across these

different subgroups of patients or with good uniformity of fit. This method generates a

complicated table of numerical data that can be difficult to interpret and hard to see clear

patterns.

Forest plots are commonly used in meta-analyses to combine quantitative results

from several different studies and funnel plots are useful in identifying publication bias.

In this chapter, the variation in the calibration of the APACHE II scoring system across

different diagnostic and patient subgroups is explored through the use of forest and

funnel plots. As discussed in Chapter 3, patients with multiple trauma and to a lesser

extent sepsis or pneumonia appeared to be least well-calibrated in the APACHE II

scoring system according to the SMR method. In this chapter the application of forest and

funnel plots to these data demonstrated visually that the calibration of the APACHE II

48

scoring system was poor for these subgroups of patients. The fact that the APACHE II

scoring system was poorly calibrated for patients with multiple trauma was also

confirmed by computing the slope and intercept of the calibration curve for these

subgroups of patients. The applications of the forest and funnel plots were further

expanded to other grouping classifications which revealed that the APACHE II scoring

system was not well calibrated for patients older than 80 years with the RPHICU patients.

This study demonstrates the feasibility and utility of using forest and funnel plots

in testing uniformity of fit in calibration of a scoring system as an alternative to

complicated tables of SMR data. It should, however, be noted that a type II error could

affect the interpretation of forest and funnel plots. That is, the ability of a funnel plot to

detect poor calibration among some subgroups of patients could be unreliable when the

sample size of the cohort is small, a problem similar to using Hosmer-Lemeshow chi-

square statistics to assess model calibration.

In conclusion, this study has introduced an alternative method to illustrate

uniformity of fit or variation in the calibration of a prognostic scoring system across

different subgroups of patients. The funnel plot illustrated visually that the APACHE II

scoring system was not well calibrated in patients with multiple trauma and patients older

than 80 years with the RPHICU cohort. This alternative approach may be useful in

situations when the sample size of the cohort is not too small.


Ho KM. Forest and funnel plots illustrated the calibration of a prognostic model: a

descriptive study. Journal of Clinical Epidemiology 2007;60:746-51.


56

Section three: Relationship between the APACHE II

scoring system, organ failure scores, and co-morbidities in

determining hospital mortality and ICU readmission

Chapter 6. Comparing the APACHE II scoring system with organ

failure scores to predict hospital mortality

It has been suggested that the intensity and duration of organ failure in critically

ill patients is important in determining their outcomes.20-22

The APACHE II prognostic

scoring system considers age, chronic health status, and physiological derangement

within the first 24 hours of ICU admission only. The APACHE II scoring system

considers two important prognostic factors (age and chronic health status) that most

organ failure scores do not consider. Conversely, most organ failure assessment scores

consider the response of the patients to treatment and the progression of organ failure

during the ICU stay that are not considered by the APACHE II score.24

The natural

question to ask is whether the APACHE II scoring system is better than some of these

organ assessment scores in predicting hospital mortality of critically ill patients.

Since the inception of the RPHICU database in 1987, daily organ failure

assessment has been recorded for every ICU admission by a locally developed RPHICU

organ failure score.70,71

There are also a number of other organ failure assessment scores

developed by different ICUs. Sequential Organ Failure Assessment (SOFA) score was

developed in 1996 after a consensus conference,20

and since then it has been evaluated

and used by many ICUs as a risk adjustment tool.21,22

In this chapter, the predictive

performance of the RPHICU organ failure score was assessed. Specifically, the

57

performance of the APACHE II scoring system and two organ failure scores (the

RPHICU organ failure score and the SOFA score) were compared in relation to

predicting hospital mortality of critically ill patients.

The results of this study showed that the APACHE II scoring system has better

discrimination, calibration and overall predictive performance, as assessed by the

Nagelkerke R2

or Brier’s score, than either the RPHICU organ failure score or SOFA

score in predicting hospital mortality. This result suggests that age and chronic health

status are important prognostic factors and should be considered in a prognostic scoring

system that predicts hospital mortality of critically ill patients. The locally developed

organ failure score (RPHICU organ failure score) had a reasonably good predictive

performance (area under ROC of the first day or cumulative 5-day RPHICU organ failure

score was 0.822 and 0.819, respectively) and its performance was indeed comparable to

the widely used SOFA score in determining hospital mortality. This latter result supports

the validity of using the RPHICU organ failure score (or intensity and duration of organ

failure as defined by the RPHICU organ failure score) as a predictor in the modelling of

outcomes of critically ill patients in the subsequent studies of this thesis.

In conclusion, the APACHE II scoring system is better than organ failure

assessment scores alone in predicting hospital mortality of critically ill patients. Age and

chronic health status are important factors in determining hospital mortality of critically

ill patients.

Further details of this study are contained in the followed published article:

58

Ho KM, Lee KY, Williams T, Finn J, Knuiman M, Webb SA. Comparison of

Acute Physiology and Chronic Health Evaluation (APACHE) II score with organ failure

scores to predict hospital mortality. Anaesthesia 2007;62:466-73.


67




Chapter 7. Combining the APACHE II scoring system with

Sequential Organ Failure Assessment (SOFA) scores to predict

hospital mortality

In Chapter 6, the APACHE II scoring system was shown to be better than two

organ failure scores in predicting hospital mortality. It is, however, possible that the two

scoring systems can be supplementary to each other because they consider the risk of

death of a critically ill patient by using data at different time points. Although both the

APACHE II and SOFA scoring system have been used together for risk adjustment in

many clinical trials, whether using both scoring systems together will improve the

accuracy of risk adjustment has never been assessed. If most of the information relevant

to the risk of death of a critically ill patient are already completely captured by the

APACHE II scoring system, it can be argued that the additional use of an organ failure

score for further risk adjustment will only increase the unnecessary work-load of data

collection. On the other hand, if the two scores can supplement each other then the use of

both scoring systems together may improve the accuracy of risk prediction and

adjustment.

The daily SOFA scores can be summarised into three scores; the Admission

SOFA score (i.e. SOFA score on the first day of ICU admission), Max SOFA

(summation of the maximum SOFA score of each organ failure during the entire ICU

68

stay), and the Delta SOFA (the difference between Max SOFA and Admission SOFA).21

In this chapter, combining each of these three SOFA scores with the APACHE II scoring

system was evaluated in relation to predicting hospital mortality of critically ill patients.

The results of this study showed that combining the Max SOFA (area under ROC

curve 0.875 vs. 0.858, P = 0.014; Nagelkerke R2: 0.411 vs. 0.371; Brier Score: 0.086 vs.

0.090) or Delta SOFA score (area under ROC curve 0.874 vs. 0.858, P = 0.003;

Nagelkerke R2: 0.412 vs. 0.371; Brier Score: 0.086 vs. 0.090) with the APACHE II score

did improve the discrimination and overall performance of the predictions when

compared with using the APACHE II scoring system alone, especially in the emergency

ICU admissions. This improvement was not apparent when the Admission SOFA was

combined with the APACHE II scoring system. These results suggest that intensity and

duration of organ failure do play a significant part in determining hospital mortality of

critically ill patients, over and beyond the risks associated with age, chronic health status,

and physiological derangement within the first 24 hours of ICU admission modelled by

the APACHE II scoring system. The improvement in performance was, however,

relatively small and may not be clinically significant when compared to the APACHE II

scoring system alone. With the exceptions of patients who are transferred from another

ICU or readmitted to ICU during same hospitalisation, it can be argued that most, but not

all, of the information relevant to mortality risk of a critically ill patient is captured by

age, chronic health status, and severity of physiology derangement at the onset of critical

illness.

In conclusion, intensity and duration of organ failure after the first 24 hours of

ICU admission does play a small part in determining mortality of critically ill patients.

69

Combining the Max SOFA or Delta SOFA score with the APACHE II scoring system

may improve the accuracy of risk adjustment if additional daily organ failure data

collection is possible. However, the degree of improvement is small and so the additional

data collection effort may not be justified or worthwhile.


Ho KM. Combining sequential organ failure assessment (SOFA) score with acute

physiology and chronic health evaluation (APACHE) II score to predict hospital

mortality of critically ill patients. Anaesthesia & Intensive Care 2007;35:515-21.


77




Chapter 8. Combining the APACHE II scoring system with co-

morbidity data to predict hospital mortality

Co-morbidities have been reported to be associated with hospital outcomes of

critically ill patients.19,24

The APACHE II scoring system assesses severe chronic health

condition in addition to age and physiological derangement in predicting mortality of

critically ill patients. There is a suggestion that the mortality risk associated with co-

morbidities may not be fully captured by the APACHE II scoring system. There are at

least three criticisms concerning how co-morbidities are considered in the APACHE II

model. First, only severe co-morbidities that are debilitating or requiring organ

supportive therapy (e.g. dialysis, home oxygen) are considered in the APACHE II scoring

system. Other co-morbidities such as diabetes mellitus, cerebrovascular accident, and

ischaemic heart disease are not considered. Second, the APACHE II scoring system does

not allow extra weighting if a patient has two or more severe co-morbidities. Third, the

APACHE II scoring system will give a larger co-morbidity weighting (or score) if a

patient is admitted after an emergency than following an elective surgical procedure.

There is in fact very little epidemiological data to support the decision to perform this

differential weighting.24

This chapter provides an assessment of the effects of severe co-morbidities in the

APACHE II scoring system and also whether incorporating more detailed co-morbidity

78

data, including minor co-morbidity from the Elixhauser co-morbidities and Charlson co-

morbidity index will improve the performance of the APACHE II scoring system.19,72

This was done overall and also for subgroups formed according to older or younger

patients (>75 vs <75 years old), elective surgery or emergency admission, and cardiac

surgery or non-cardiac surgery patients.

The results of this study showed that minor co-morbidities as described in the

Charlson co-morbidity index and Elixhauser co-morbidities were prevalent in critically ill

patients. Among 24,303 ICU admissions, 3,615 (14.9%), 10,223 (42.1%), and 11,597

(47.7%) patients had at least one co-morbidity as defined in the APACHE II scoring

system, Charlson co-morbidity index, and Elixhauser co-morbidities, respectively. While

the ability of co-morbidity alone to discriminate between hospital survivors and non-

survivors was poor (areas under ROC <0.610), severe co-morbidity was a significant

component in the APACHE II scoring system in predicting mortality of non-cardiac-

surgical admissions. Replacing the weighted co-morbidity data in the APACHE II

scoring system with other more comprehensive measures of co-morbidity, such as

Charlson co-morbidity index or counts of minor co-morbidities, did not significantly

improve the discrimination of the APACHE II scoring system in non-cardiac surgical

patients. For cardiac surgical patients neither the severe co-morbidity component in the

APACHE II scoring system, nor more comprehensive measures, contributed significantly

to the APACHE II scoring system in predicting their hospital mortality.

In conclusion, severe co-morbidity was a significant component of the APACHE

II scoring system for non-cardiac surgical patients. Further improvement in the predictive

performance of the APACHE II scoring system was not observed by incorporating more

79

detailed co-morbidity data. The APACHE II scoring system appeared to have captured

most, if not all, of the hospital mortality risk due to co-morbidity. Clinicians should not

put undue emphasis on the total number of co-morbidities when they make

prognostication on hospital mortality of critically ill patients.

Further details of the study are contained in the following published article:

Ho KM, Finn J, Knuiman M, Webb SA. Combining multiple comorbidities with

Acute Physiology Score to predict hospital mortality of critically ill patients: a linked

data cohort study. Anaesthesia 2007;62:1095-100.


86




Chapter 9. The effects of co-morbidity on risk of unplanned ICU

Readmission

The previous chapter examined co-morbidity as a predictor of hospital mortality.

In this chapter, the effects of co-morbidity on risk of unplanned ICU readmission are

examined.

It has been hypothesised that ICU readmission may represent an intermediate

event for an intrinsically sicker group of patients.73

If co-morbidities are a risk factor for

ICU readmission then this may explain why ICU readmission is associated with a higher

hospital mortality. It is also of interest to determine whether multiple minor co-

morbidities, as measured by the Charlson co-morbidity index but not by the APACHE II

scoring system, are a risk factor for ICU readmission and explain the excess mortality

associated with ICU readmission.12,73

Unplanned ICU readmission within 72 hours of ICU discharge is one of the

quality indicators adopted by the ACHS.34,35

As such, we had stratified the analyses into

patients who were readmitted within 72 hours or after 72 hours of ICU discharge in this

study. Our results showed that the total number (or count) of Charlson co-morbidities was

found to be an independent risk factor for late (>72 hours) but not early (< 72 hours)

unplanned ICU readmission during the same hospitalisation. The severity of illness on

87

first admission to ICU as measured by the APACHE II scoring system was not

independently associated with either early or late unplanned ICU readmission.

These results suggest that ICU readmission occurred because of factors that were

not primarily related to the severity of acute illness leading to the first ICU admission.

Co-morbidities not measured by the APACHE II scoring system might explain why some

patients were readmitted to the ICU after 72 hours of ICU discharge but not those who

were readmitted earlier.

The multivariate analyses showed that either early (<72 hours) or late (>72 hours)

ICU readmission and the severity of acute illness as measured by the APACHE II scoring

system were significant risk factors for hospital mortality. Co-morbidities were not a

significant independent risk factor of hospital mortality. These findings imply that co-

morbidities could not account for why patients with either early or late ICU readmissions

had higher hospital mortality. The excess mortality associated with ICU readmissions

could be due to factors or events that occurred after the onset of critical illness such as

nosocomial infections or complications (e.g. thromboembolism).


Ho KM, Dobb GJ, Finn J, Knuiman M, Webb SA. The effect of co-morbidities on

risk of intensive care readmission during the same hospitalisation: a linked data cohort

study. Journal of Critical Care 2008 (published online in April 2008).

95




Chapter 10. Evaluating the APACHE II scoring system for

predicting hospital mortality in ICU readmissions

The results in the previous chapter showed that co-morbidity could be a

significant risk factor for late ICU readmission, but it could not explain why ICU

readmission was associated with an increase in hospital mortality. There is currently no

risk adjustment scoring system for patients who are readmitted to the ICU during the

same hospitalisation. This study aimed to assess whether the APACHE II scoring system

was useful to predict mortality of ICU readmission when it was applied to patients during

their readmission.

In this chapter, the use of the APACHE II predicted mortality measured at the

time of ICU readmission (the Readmission APACHE II predicted mortality) together

with information collected prior to ICU readmission were assessed for their ability to

predict hospital mortality in ICU readmission during the same hospitalisation. The prior

information considered included the time interval between ICU discharge and

readmission, admission source, elective surgery status, length of stay of the first ICU

admission and the APACHE II predicted mortality for the first ICU admission. ICU

readmissions were stratified into two groups (7 days or >7days) to assess whether these

factors would have different effects. A relatively long period before readmission (7 days)

was chosen as a cut point in this study because any prior events before ICU readmission

96

are theoretically less likely to have a significant effect on the course of very late ICU

readmission (> days) when compared to earlier readmission (7 days).

Using two alternative approaches, logistic regression and multilevel likelihood

ratio, the information prior to ICU readmission was found to be not as important as the

Readmission APACHE II predicted mortality in determining the hospital mortality of

these patients. For patients who were readmitted after 7 days of ICU discharge, the

severity of acute illness leading to the first ICU admission had no relationship to their

subsequent hospital mortality (odds ratio 1.05, 95% confidence interval 0.27-1.25,

p=0.602). Combining the APACHE II predicted mortality of the first ICU admission with

the Readmission APACHE II predicted mortality also did not improve the

discrimination between survivors and non-survivors further (area under the ROC curve

0.694 vs 0.699, p=0.682). Other factors such as the time interval between ICU discharge

and readmission, admission source, elective surgery status, and length of stay of the first

ICU admission were also not significant in determining the hospital mortality of ICU

readmissions.

The Readmission APACHE II predicted mortality only had a moderate ability

to discriminate survivors and non-survivors and its performance was slightly better when

applied to ICU readmissions within 7 days of ICU discharge (area under ROC curve

0.785 vs 0.694). The results of this study have at least two significant clinical

implications. First, the mortality risk associated with ICU readmission might not be

related to the patient’s events prior to their readmission including factors related to their

first ICU admission during the same hospitalisation. It is highly possible that ICU

readmission occurred because of an adverse event closer to the time of readmission rather

97

than because they were intrinsically a sicker group of patients in their first ICU

admission.71

It is possible that some nosocomial complications such as pneumonia,

thromboembolism, catheter or wound-related infections may precipitate the readmissions.

This explanation also suggests that some ICU readmissions and their associated

attributable mortality could be potentially preventable if these precipitating events were

identified early and modified in time.

Second, the fact that the performance of the Readmission APACHE II predicted

mortality was not as good as the APACHE II scoring system when applied to the first

ICU admission (in all patients) implies that the model was not capturing the excess

mortality risk of ICU readmission. It is possible that the effect of a disease may have a

stronger mortality effect when it occurs in a hospital setting (that is, nosocomial event) or

when there is a delay in clinical recognition than when a similar disease that occurs

without recent critical illness. Also, a certain degree of physiological derangement from a

nosocomial complication (e.g. deep vein thrombosis, hospital acquired pneumonia,

wound infection, catheter related sepsis) may have a stronger mortality effect on a patient

who has not completely recovered from their critical illness. These findings suggest that

patients who are readmitted to ICU after a recent critical illness are very different from

patients without prior critical illness. Any prognostic model for ICU readmission should

consider a different weighting coefficient for a similar diagnosis and physiological

derangement when compared to patients without prior ICU admission.

In conclusion, the readmission diagnosis and physiological derangement within

the first 24 hours of the readmission explained most of the mortality risk of ICU

readmission. The modified use of the APACHE II model as the Readmission APACHE

98

II predicted mortality only had a moderate ability to discriminate survivors and non-

survivors in ICU readmissions. Events prior to ICU readmission only had a very small

effect on mortality of early ICU readmissions.


Ho KM, Knuiman M. Bayesian approach to predict hospital mortality of intensive

care readmissions during the same hospitalisation. Anaesthesia & Intensive Care

2008;36:38-45.


107

Section four: The use of inflammatory markers in

predicting hospital mortality and ICU readmission

Chapter 11. Inflammatory markers and risk of unplanned ICU

readmission

The results of Chapter 9 suggest that it is not patients who were sicker at the time

of their first ICU admission that have greatest risk of ICU readmission. The implication is

that factors associated with the management of the patient in ICU or adverse events

following first admission to ICU may be risk factors for ICU readmission. It is possible

that ICU readmission is more likely when the initial ICU discharge was pre-mature, that

is, the patient was discharged from the ICU when they had not adequately recovered from

their critical illness and were placed in a less intensive environment. For example,

nocturnal ICU discharge, discharge to a general ward instead of a high dependency unit,

persistent organ dysfunction (e.g. a high SOFA score on the day of discharge), and a lack

of follow-up team have been identified as risk factors for unplanned ICU readmission and

support the hypothesis that at least some ICU discharges may be premature and

potentially preventable.37,74,75

This study explores whether residual organ dysfunction (as measured by the

Discharge SOFA score) and/or an occult infection or inflammatory process as suggested

by an increase in serum inflammatory markers such as fibrinogen, white cell counts, and

C-reactive protein (CRP) at the time of ICU discharge may increase the risk of ICU

readmission during the same hospitalisation.76

In this study only inflammatory markers

routinely used in daily clinical practice in RPHICU were assessed. As the laboratory data

108

and SOFA scores were required, only the ICU readmissions in the most recent year

(2004) were assessed. The propensity score method was used to adjust for the potential

effects of selection bias due to missing laboratory data.

The results of this study showed that the SOFA score at ICU discharge was not

predictive of subsequent unplanned ICU readmission and CRP concentration at ICU

discharge may be a risk factor for unplanned ICU readmission. These results support the

hypothesis that some readmissions might be due to an occult infection or persistent

inflammation without obvious organ failure that may not be apparent to the ICU

clinicians before the ICU discharge. The strong association between CRP and unplanned

ICU readmission reported in our study was recently confirmed by a large prospective

cohort study in Germany.77


Ho KM, Dobb GJ, Lee KY, Towler SC, Webb SA. C-reactive protein

concentration as a predictor of intensive care unit readmission: a nested case-control

study. Journal of Critical Care 2006;21:259-65.


116

Section four: The use of inflammatory markers in

predicting hospital mortality and ICU readmission

Chapter 12. Inflammatory makers and prediction of hospital

mortality

In the last chapter, CRP concentration at ICU discharge was found to be

associated with unplanned ICU readmission. In this chapter, the use of CRP

concentration at ICU discharge to predict unexpected hospital mortality after ICU

discharge was examined.

This prospective study was based on the cohort of 603 patients first admitted to

RPH ICU between 1 June and 31 December 2005 and who survived to ICU discharge.

The incidence of unexpected death in this cohort was about 4.3%. The APACHE II

predicted mortality, different SOFA scores (Admission, Max, Delta, and Discharge

SOFA score), and CRP concentrations within 24 hours of ICU discharge were all related

to unexpected death after ICU discharge in the univariate analysis. Multivariate analysis

showed that Delta SOFA score (which signifies progression of organ failure during the

ICU stay) and CRP concentrations at ICU discharge were the only significant predictors

of unexpected hospital death.

The findings from this study and also from the previous study (Chapter 11)

suggest that persistent or new onset inflammation near ICU discharge is a risk factor for

undesirable in-hospital outcomes after ICU discharge.57

Most ICU clinicians decide when

to discharge their patients based on their clinical judgement on the risks and benefits of

keeping or discharging their patients from the ICU. There is, so far, no predictive model

that can be used to assist ICU clinicians in making this decision. If the results of this

117

study are confirmed by other ICUs, it is possible that the intensity of organ failure during

the ICU stay and markers of inflammation near ICU discharge are main determinants of

short term undesirable outcomes after ICU discharge. As such, these risk factors should

be considered when prognostic scoring systems that aim to assist ICU clinicians to

improve their discharge decisions are developed.


Ho KM, Lee KY, Dobb GJ, Webb SA. C-reactive protein concentration as a

predictor of in-hospital mortality after ICU discharge: a prospective cohort study.

Intensive Care Medicine 2008;34:481-7.


125

Section five: Predicting long term survival after hospital

discharge

Chapter 13. The PREDICT model

There are several potential uses for a tool that is capable of predicting survival

over longer time-periods. Long term survival after critical illness is increasingly being

recognised as an important outcome in assessing effectiveness of new therapy.15

In order

to control for confounding and bias in assessing long term survival of critically ill

patients in a clinical trial, a risk adjustment tool that can objectively estimate long term

survival is essential. From a clinical perspective, many patients and clinicians are also

interested in knowing the long term survival outcome after critical illness, in addition to

other information such as burden of treatment and quality of life after recovery, when

making treatment decisions in the ICU.38,39

There are, so far, only two long term survival

scoring systems and they are limited to estimation of 6-month to 5-year survival of

critically ill patients after hospital discharge.14,41

The RPHICU clinical database was linked to the Western Australian Data

Linkage System (WADLS) in order to obtain information on long term survival

following discharge from hospital. The WADLS contains all hospital admission and

death records for all people resident in Western Australia. The geographical isolation of

Western Australia and the low emigration rate indicate that long term survival data are

available on almost all RPHICU patients.76

In this study, a new prognostic model, based on seven pre-selected predictors

(APACHE II Predicted Risk, Existing Diseases, and Intensive Care Therapy: the

126

PREDICT model), to estimate median survival time and long term survival probabilities

of patients after their critical illness was developed. The seven pre-selected predictors

included age, gender, pre-existing disease (as measured by the Charlson co-morbidity

index), severity of acute illness (as measured by the APACHE II predicted mortality),

and intensity and duration of intensive care therapy. Among these seven predictors, we

found that age, co-morbidities, and severity of acute illness were most important in

determining long term survival of critically ill patients. These three predictors also had a

relatively linear relationship to the probability of long term survival and fulfilled the

proportional hazards assumption over a 15-year period. To facilitate the use of this

model, a nomogram was derived which allows estimation of a patient’s median survival

time and long term survival probabilities by summing scores of each of the seven

predictors.

The current prognostic model, the PREDICT model, provides a framework for

risk adjustment purposes in assessing long term survival of critically ill patients.

However, this prognostic model has some limitations. Firstly, patients’ wishes and the

anticipated quality of life before and after their critical illness are important factors in

making treatment decisions. The median survival time (or long term survival

probabilities) is only one of many factors that patients and clinicians may consider in

making treatment decisions. Furthermore, the c-statistic of this model is only about 0.76

and this leaves considerable uncertainty in its applicability in predicting long term

survival of individual patients. As such, the predicted survival probabilities of this

prognostic model should only be considered as an average estimate of patients with

similar characteristics and not be used for individual patients.

127

Secondly, evidence suggests that combining an objective prognostic scoring

system with physicians’ intuition of a patient’s prognosis may improve the accuracy of

outcome prediction.14,16

Whether combining this current prognostic model with

physicians’ intuition will improve its predictive performance further remains uncertain,

but this merits further investigation. Thirdly, although a large cohort of critically ill

patients was studied and the case-mix of this cohort was, to some extent, similar to other

Australian ICUs (Chapter 2)56

, validation of this model in other ICUs is essential to

assess its generalisability. Finally, the performance of the current model may be

improved if more predictors were considered in the model. The next chapter considers

other potential prognostic factors such as socioeconomic status, accessibility to essential

services, and indigenous status in relation to long term survival of critically ill patients.


Ho KM, Knuiman M, Finn J, Webb SA. Estimating long-term survival of critically ill

patients: the PREDICT model. Public Library of Science One 2008;3:e3226.

Estimating Long-Term Survival of Critically Ill Patients:The PREDICT ModelKwok M. Ho1,2*, Matthew Knuiman3, Judith Finn3, Steven A. Webb1,2

1 School of Population Health, University of Western Australia and Royal Perth Hospital, Perth, Western Australia, Australia, 2 School of Medicine, University of Western

Australia and Royal Perth Hospital, Perth, Western Australia, Australia, 3 School of Population Health, University of Western Australia, Crawley, Western Australia, Australia

Abstract

Background: Long-term survival outcome of critically ill patients is important in assessing effectiveness of new treatmentsand making treatment decisions. We developed a prognostic model for estimation of long-term survival of critically illpatients.

Methodology and Principal Findings: This was a retrospective linked data cohort study involving 11,930 critically illpatients who survived more than 5 days in a university teaching hospital in Western Australia. Older age, male gender, co-morbidities, severe acute illness as measured by Acute Physiology and Chronic Health Evaluation II predicted mortality, andmore days of vasopressor or inotropic support, mechanical ventilation, and hemofiltration within the first 5 days of intensivecare unit admission were associated with a worse long-term survival up to 15 years after the onset of critical illness. Amongthese seven pre-selected predictors, age (explained 50% of the variability of the model, hazard ratio [HR] between 80 and60 years old = 1.95) and co-morbidity (explained 27% of the variability, HR between Charlson co-morbidity index 5 and0 = 2.15) were the most important determinants. A nomogram based on the pre-selected predictors is provided to allowestimation of the median survival time and also the 1-year, 3-year, 5-year, 10-year, and 15-year survival probabilities for apatient. The discrimination (adjusted c-index = 0.757, 95% confidence interval 0.745–0.769) and calibration of thisprognostic model were acceptable.

Significance: Age, gender, co-morbidities, severity of acute illness, and the intensity and duration of intensive care therapycan be used to estimate long-term survival of critically ill patients. Age and co-morbidity are the most importantdeterminants of long-term prognosis of critically ill patients.

Citation: Ho KM, Knuiman M, Finn J, Webb SA (2008) Estimating Long-Term Survival of Critically Ill Patients: The PREDICT Model. PLoS ONE 3(9): e3226.doi:10.1371/journal.pone.0003226

Editor: Jeffrey A. Gold, Oregon Health & Science University, United States of America

Received July 27, 2008; Accepted August 25, 2008; Published September 17, 2008

Copyright: � 2008 Ho et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricteduse, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: A grant was received from BUPA to provide support for the cost of linking the databases used in this study. The funding agency has no involvement inthe study design, collection of data, analysis and interpretation of data, writing of the report, or in the decision to submit the article for publication.

Competing Interests: The authors have declared that no competing interests exist.

* E-mail: [email protected]

Introduction

Demand for intensive care unit (ICU) services is increasing [1],

and at a rate that is higher than the average for all health care

services [2]. Increase in treatment and monitoring technology,

patients’ expectations, and ageing population all contribute to this

increased demand for intensive care services [1]. Indeed, intensive

care is increasingly being provided to older and sicker patients,

whom in the past were not treated in intensive care [3]. Intensive

care services accounted for 10% of the US$2.1 trillion total health

expenditures on health care in the United States in 2006 [4] and

has been estimated to cost more than £700 million in the United

Kingdom in 1999 [5]. The cost of intensive care services coupled

with increasing demand provides the rationale for improved

modelling of outcomes of critically ill patients.

Long-term survival after critical illness is increasingly being

recognized as an important outcome in assessing effectiveness of

new therapy [6]. In order to control for confounding and bias in

assessing long-term survival of critically ill patients in a clinical

trial, a risk adjustment tool that can objectively estimate long-term

survival is essential. From a clinical perspective, many patients and

clinicians are also interested in knowing the long-term survival

outcome after critical illness, in addition to other information such

as burden of treatment and quality of life after recovery, when

making treatment decisions in the ICU. Although many clinicians

may foretell patient hospital survival outcome more accurately

than some objective prognostic models [7], treatment decisions

made by clinicians do vary considerably with their practice style

and work experience [8–10]. The strategy of continuing intensive

treatment for all patients until death will reduce the need for

patients and clinicians to make difficult treatment decisions and

may improve the survival time of some. This strategy is, however,

expensive and undesirable by imposing excessive burden of

treatment on those who have a very poor prognosis [11]. For

example, initiating acute renal replacement therapy in critically ill

patients with less than 10% probability of 6-month survival was

estimated to cost US$274,000 (£137,000) per quality-adjusted life

year saved [12].

The SUPPORT investigators from the United States and

Wright et al. from the United Kingdom published two prognostic

models that were based on age, severity of acute illness and

admission diagnosis to estimate 6-month and 5-year survival of

PLoS ONE | www.plosone.org 1 September 2008 | Volume 3 | Issue 9 | e3226

critically ill patients, respectively [13,14]. The utility of latter

model is, however, limited by its ability to classify 5-year survival

probabilities only into three risk categories when the calculated

risk score is either ,70, 70–80, or .80 [14]. This model also did

not consider the potential effect of detailed co-morbidity data on

long-term survival of critically ill patients beyond the usual

assessment included in the Acute Physiology and Chronic Health

Evaluation (APACHE) score [14,15]. There is currently no

prognostic model that is available to estimate the survival of

critically ill patients beyond 5 years after the onset of critical

illness. Furthermore, the relative importance of age, co-morbidity,

and severity of acute illness in determining long-term prognosis of

critically ill patients also remains unknown. In this study we

examined the long-term survival of 11,930 critically ill adult

patients who survived at least 5 days and developed a new

prognostic model (Predicted Risk, Existing Diseases, and Intensive

Care Therapy: the PREDICT model) to estimate their median

survival time and long-term survival probabilities.

Methods

The characteristics of the cohortThis cohort study utilized the clinical database of the ICU at

Royal Perth Hospital (RPH) in Western Australia. RPH is the

largest tertiary university teaching hospital in Western Australia

and the 22-bed ICU admits patients of all specialties except liver

transplantation and captures over 40% of all critically ill patients

in Western Australia. The database analyzed in this study includes

details of all ICU admissions between 1989 and 2002, including

demographic factors, admission diagnosis, admission source,

severity of acute illness as measured by APACHE II scores based

on the worst first 24-hour ICU data [15], daily organ failure

assessment and supportive therapy required [16], and ICU and

hospital survival outcome.

In this study the patients with a diagnosis excluded from the

original APACHE II cohort (e.g. coronary artery graft surgery,

burns, snake bite)[15], those who resided outside Western

Australia at the time of ICU admission (who could not be

followed for survival), readmissions after the first ICU readmission,

patients who were younger than 16 years old, and patients who

did not survive more than 5 days during their hospitalization of

the index ICU admission were excluded. The data were reviewed

for internal consistency annually, and there were no patients with

missing hospital mortality data. Some of the other details of this

cohort have been described in our previous publications [16–18].

The ICU clinical database was linked to the Western Australian

hospital morbidity and mortality databases using probabilistic

matching [16], providing information on patients’ co-morbidities

as recorded in all private and public hospital admissions including

any prior ICU admissions up to 5 years before the index ICU

admission. A relatively long five-year ‘look back’ period was

chosen in this study to capture all existing co-morbidities of each

patient. We ascertained the presence of co-morbidities in the

Charlson co-morbidity index (Table 1) using the published ICD-

9-CM and ICD-10-AM coding algorithms [16,19]. We did not

assign a co-morbidity to a patient if that condition was diagnosed

during the hospitalization of the index ICU admission. The

proportions of invalid (false positive) and missed links (false

negatives) between Western Australian hospital morbidity and

mortality databases were evaluated several years ago, and both

false positives and negatives were estimated to be 0.11% [20].

The survival status of the patients was assessed on 31 December

2003 and the length of follow up ranged from 1 year to 15 years

with an average of 6 years. Western Australia is geographically

isolated and has a very low rate of emigration (,0.03% in

2002)[16], and as such, lost to long-term survival follow-up by the

Western Australian mortality database is likely to be very low. The

data analyzed had the patient name and address removed and the

study was approved by the RPH Ethics Committee and the

Western Australian Confidentiality of Health Information Com-

mittee (CHIC).

Development of the modelThe prognostic model was fitted using Cox proportional

hazards regression [21], restricting predictors to factors that were

likely to be important predictors of long-term survival of

hospitalized patients [13,14,22,23]. These pre-selected factors

included age [14,22], gender [22], APACHE II predicted

mortality risk [13–15], Charlson co-morbidity index [19,23], days

of mechanical ventilation, acute renal replacement therapy or

hemofiltration, and vasopressor or inotropic therapy during the

first 5 days of the index ICU admission [13]. The APACHE II

predicted mortality was chosen as a measure of severity of acute

illness because it is widely used and summarizes the diagnosis,

acute physiologic derangement within the first 24 hours of ICU

admission, severe co-morbidities, and whether the ICU admission

is after elective or emergency surgery. Our previous study also

showed that the APACHE II predicted mortality has a very stable

performance in this cohort over the past 10–15 years [17].

Although age and severe co-morbidities are already used to

calculate the APACHE II predicted mortality [15], these two

factors may still have a profound effect on long-term survival over

and beyond the weightings used in the APACHE II predicted

mortality [14,22,23]. As such, both age and Charlson co-morbidity

index were used as separate predictors in additional to the

APACHE II predicted mortality in this prognostic model. These

seven predictors were also chosen because they are often recorded

Table 1. Charlson co-morbidity index component and itsweighting.

Co-morbidity Weighting

Myocardial infarction 1

Congestive heart failure 1

Peripheral vascular disease 1

Cerebrovascular disease 1

Dementia 1

Chronic pulmonary disease 1

Connective tissue disease 1

Peptic ulcer disease 1

Mild liver disease 1

Diabetes mellitus 1

Hemiplegia 2

Moderate or severe renal disease 2

Diabetes with end-organ damage 2

Any tumour 2

Leukemia 2

Lymphoma 2

Moderate to severe liver disease 3

Metastatic solid tumour 6

AIDS 6

doi:10.1371/journal.pone.0003226.t001

PREDICT Model


in the administrative databases of many ICUs, and as such, it is

possible for other ICUs to validate this model using their data [24].

The proportional hazards assumption of the continuous

predictors in the Cox model was assessed and found to be

acceptable (Figure 1a, 1b, 1c). During the modelling process, we

avoided categorizing continuous predictors [24,25] and allowed a

non-linear relationship with hazard of death using a 6-knot

restricted cubic spline function [25]. The relative contribution of

each predictor was assessed using the chi-square statistic minus the

degrees of freedom [25]. The discrimination performance of the

model was assessed with the c-index, which is a generalization of

the c-statistic or the area under the receiver-operating character-

istic curve, allowing for censored data [25,26]. In this study, a c-

index between 0.70 and 0.80 was regarded as acceptable and a c-

index above 0.80 was regarded as excellent [27]. Using the Design

library in S-PLUS software (version 8.0, 2007. Insightful Corp.,

Seattle, Washington, USA), the c-index was computed and

adjusted for optimism (arising from using the same data to

develop the model and assess its performance) by a bootstrap

technique to penalise for possible over-fitting, with 200 re-samples

and at least 200 patients per risk group [25,28]. The bootstrapping

technique was used in this study instead of splitting the data into

development and validation data set because this method is

regarded as most data ‘efficient’ and accurate in developing a

prognostic model [25]. Model calibration (similarity of predicted

risks and proportions actually dying) was assessed graphically and

used a bootstrap re-sampling to construct a bias-corrected

calibration curve and its slope [25,29]. Nagelkerke’s R2 (a

generalized measure of the percentage of the variance in survival

accounted for by the model) was computed to assess the overall

performance of the model [25,30]. The performance of the model

was assessed over the full 15 years of follow-up, when follow-up

was restricted to a maximum of 5 years for each patient, and also

when only patients admitted after 1997 were considered.

A nomogram was developed for the model that generates the

median survival time and selected annual survival probabilities by

adding up the scores for each of the seven predictors [25]. The

use of the nomogram and how each predictor may affect a

patient’s long-term prognosis is described for a selection of typical

patient scenarios. In particular, these scenarios were selected to

illustrate how the long-term prognosis of a patient can be

different from the short-term prognosis. Nevertheless, the results

of the nomogram should only be considered as an average

estimate of patients with similar characteristics and not be used

for individual patients.

Results

Characteristics of the cohortThe study cohort consisted of a heterogeneous group of

critically ill patients, with elective surgery including heart valve

surgery, urology, gastrointestinal and spinal surgery accounting for

36.2% of all ICU admissions. The emergency admissions consisted

of patients with multiple trauma (8.5%), isolated head trauma

(2.5%), acute myocardial infarction, congestive heart failure,

cardiac arrhythmias, or cardiogenic shock (7.4%), hypovolemic

or hemorrhagic shock (0.8%), drug overdoses (7.2%), subarach-

noid or intracranial hemorrhage (5.1%), sepsis (4.3%), pneumonia

or aspiration (3.7%), obstructive airway diseases (2.1%), cardiore-

spiratory arrest (4.0%), gastrointestinal hemorrhage, perforation or

obstruction (2.4%), and other medical and surgical emergencies.

Details of this cohort including demographic factors, co-morbid-

ities, severity of acute illness, and the length of ICU and hospital

stay are described in Table 2.

Figure 1. The proportional hazards assumption of the predictors in the Cox model was assessed by plotting the logarithm of thenegative logarithm of the Kaplan Meier survivor estimates and the assumption was found to be acceptable for the three pre-selected continuous predictors; APACHE II predicted mortality, Charlson co-morbidity index, and age. (a) Graph assessing theproportionality of hazards associated with severity of acute illness measured by the APACHE II predicted mortality risk categories (0–20%, 20–40%,40–60%, 60–80%, 80–100%). (b) Graph assessing the proportionality of hazards associated with co-morbidities measured by Charlson co-morbidityindex categories (0, 1, 2, 3, 4–5, .5). (c) Graph assessing the proportionality of hazards associated with age measured by age categories (16–30, 30–50, 50–60, 60–70, 70–80, .80 years old)doi:10.1371/journal.pone.0003226.g001

PREDICT Model


Effect of the Predictors on Hazard of DeathAmong all the seven pre-selected predictors in the model, age

(50%), co-morbidity as measured by Charlson co-morbidity index

(27%), and severity of acute illness as measured by the APACHE

II predicted mortality (20%) made the strongest contributions in

predicting survival time (Figure 2). After adjusting for other

predictors, the log hazard of death increased linearly with age,

Charlson co-morbidity index, and the number of days of

vasopressor or inotropic therapy, mechanical ventilation, or

hemofiltration therapy (Figure 3). The relationship between the

APACHE II predicted mortality and log hazard of death was non-

linear with a steep effect when the APACHE II predicted mortality

was less than 10% and a smaller effect when it was more than

10%. The estimated (adjusted) hazard ratios for the seven

predictors are summarized in Figure 4.

Clinical Application of the ModelFigure 5 presents the model in the form of a nomogram that

provides the median survival time and long-term survival

probabilities corresponding to a particular total score. The total

score for a patient is obtained by adding up the scores for each of

the seven predictors. We use the following hypothetical but typical

patients to illustrate how the nomogram is used and how the short-

term prognosis of a patient can be quite different from the long-

term prognosis.

Patient A:

A 40-year old male, without pre-existing co-morbidities (ie

Charlson co-morbidity index = 0), was admitted to the ICU

because of septic shock with an APACHE II predicted mortality of

80%. He required vasopressor or inotropic therapy, mechanical

ventilation, and hemofiltration therapy during the first 5 days in

the ICU.

The gender of this patient scores 5 points, age scores 28 points,

Charlson co-morbidity scores zero points, the APACHE II

predicted mortality or risk scores 30 points, 5 days of vasopressor

or inotropic therapy scores 7 points, 5 days of mechanically

ventilation scores 15 points, and 5 days of hemofiltration scores 20

points. The total score of this patient is therefore 105 which gives

an estimated median survival time of about 4 years, .70% 1-year

survival probability, .50% 3-year survival probability, .45% 5-

year survival probability, and .20% 10-year survival probability.

Patient B:

A 70-year old female, with chronic obstructive airway disease

and non-insulin dependent diabetes mellitus with no end-organ

damage (ie Charlson co-morbidity index = 2), was admitted to the

ICU because of severe community acquired pneumonia with an

APACHE II predicted mortality of 30%. She required vasopressor

or inotropic therapy and mechanical ventilation but not

hemofiltration during the first 5 days in the ICU.

The gender of this patient scores zero points, age scores 70

points, Charlson co-morbidity index scores 12 points, the

APACHE II predicted mortality scores 16 points, 5 days of

mechanical ventilation scores 15 points, and 5 days of vasopressor

or inotropic therapy scores 7 points. The total score of this patient

is therefore 120 which gives an estimated median survival time of

about 2 years, 60% 1-year survival probability, 40% 3-year

survival probability, 30% 5-year survival probability, and 10% 10-

year survival probability.

Patient C:

A 80-year old male, with a history of myocardial infarction,

congestive heart failure, peripheral vascular disease, cerebrovas-

cular disease, and dementia (ie Charlson co-morbidity index = 5),

was admitted to an ICU with bowel perforation and peritonitis

with an APACHE II predicted mortality of 30%. He required

vasopressor or inotropic therapy and mechanical ventilation but

not hemofiltration during the first 5 days in the ICU.

The gender of this patient scores 5 points, age scores 85 points,

Charlson co-morbidity index scores 30 points, the APACHE II

predicted mortality scores 16 points, 5 days of mechanical

ventilation scores 15 points, and 5 days of vasopressor or inotropic

therapy scores 7 points. The total score of this patient is therefore

158 which gives an estimated median survival time of ,0.5 years,

25% 1-year survival probability, and 10% 3-year survival

probability.

Discrimination and Calibration of the Prognostic ModelThe adjusted c-index for this prognostic model was 0.757 (95%

confidence interval 0.745–0.769), Nagelkerke’s R2 was 0.255 and

the bias-corrected calibration of the model over a 15-year period

was reasonable (slope of the calibration = 0.98)(Figure 6). The

Nagelkerke’s R2 remained unchanged and the adjusted c-index

only increased marginally when the analysis was restricted to a

maximum of 5 years follow up (c-index = 0.759, slope = 0.97) or

data after 1997 (c-index = 0.762, slope of the calibration = 0.97).

Table 2. Characteristics of the cohort (n = 11,930).

Variables

Mean (median,standard deviation),unless statedotherwise

Age, yrs 53.8 (57.0, 19.0)

Gender (male/female), no. (%) 7489 (62.8)/4441 (37.2)

Elective surgery admission, no. (%) 4318 (36.2)

APACHE II score 13.7 (13.0, 6.8)

APACHE II predicted mortality, % 14.5 (7.0, 17.8)

No. of APACHE co-morbidities 0.1 (0, 0.3)

(a) Cardiovascular, no. (%) 592 (5.0)

(b) Respiratory, no. (%) 210 (1.8)

(c) Renal, no. (%) 109 (0.9)

(d) Immunosuppressed, no. (%) 197 (1.7)

(e) Liver, no. (%) 76 (0.6)

No. of Charlson co-morbidities 0.8 (0, 1.2)

Charlson co-morbidity index 1.0 (0, 1.7)

Length of ICU stay, days 5.6 (3.0, 8.3)

Length of hospital stay, days 20.3 (13.0, 25.9)

No. of patients mechanically ventilated (%) # 8034 (67.3)

No. of patients on inotrope (%) # 3921 (32.9)

No. of patients on dialysis (%) # 608 (5.1)

No. of ICU survivor (%)* 11557 (96.9)

No. of hospital survivor (%)* 11101 (93.1)

No. of survivor/total no. of patients followed up (%)

(a) at 1-year 10334/11101 (93.1)

(b) at 3-year 8031/10019 (80.2)

(c) at 5-year 6109/8212 (74.4)

(d) at 10-year 2609/4238 (61.6)

(e) at 15-year 441/887 (49.7)

#During the first 5 days in ICU.*Excluding patients died within 5 days of ICU admission.ICU, intensive care unit.APACHE, Acute Physiology and Chronic Health Evaluation.doi:10.1371/journal.pone.0003226.t002

PREDICT Model


Figure 2. Contribution of each predictor in predicting the survival time in the Cox proportional hazards model.doi:10.1371/journal.pone.0003226.g002

Figure 3. The relationship between relative hazard and each predictor after adjusting for other predictors in the model.doi:10.1371/journal.pone.0003226.g003

PREDICT Model


Discussion

This study showed that age, gender, co-morbidities (Charlson

co-morbidity index), severity of acute illness (the APACHE II

predicted mortality), and duration of intensive care therapy or

organ support within the first 5 days of ICU admission are

important prognostic factors for long-term survival of critically ill

patients. To the best of our knowledge, this new prognostic model

(Predicted Risk, Existing Diseases, and Intensive Care Therapy:

the PREDICT model) is the first preliminary prognostic model

that can be used to estimate the median survival time and long-

term survival probabilities of critically ill patients up to 15 years

after the onset of critical illness.

The current prognostic model has confirmed that age, gender,

co-morbidities, severity of acute illness, and duration of intensive

care therapy or organ failure are important predictors of 6 months

to 5 years survival of hospitalized or critically ill patients

[13,14,19,22,23]. The current model is indeed built on the results

of these previous studies but further extended the significance of

these risk factors in predicting survival of critically ill patients

beyond 6 months to 5 years. This current model also demonstrat-

ed that most of these predictors have a relatively linear relationship

to the long-term survival probability. More importantly, our

results also showed that age and co-morbidities are the most

important determinants of long-term prognosis of critically ill

patients. This latter finding has at least two significant clinical

Figure 4. The estimated (adjusted) hazard ratios and multilevel confidence bars (0.70 as illustrated by the black bar to 0.99 asillustrated by the orange bar) for the effects of predictors in the model are summarized in the figure below. An increase of 20 years ofage and an increase in Charlson co-morbidity index from 0 to 5 approximately doubled the risk of death. Doubling the APACHE II predicted mortalityfrom 20% to 40% increased the relative risk of death by about 30 to 40%. Similarly, increased the number of days of intensive care therapy from 1 to 5increased the relative risk of death by between 10% and 50%.doi:10.1371/journal.pone.0003226.g004

Figure 5. Nomogram for predicting long-term survival probabilities and median survival time. Note: gender: 2 = female, 1 = male.Predicted.mortality = APACHE II predicted mortality in %.doi:10.1371/journal.pone.0003226.g005

PREDICT Model


implications. First, the factors that determine long-term survival of

a critically ill patient are different from those that affect short-term

prognosis. Previous evidence suggested that diagnosis and acute

physiological derangement of a patient are most important in

determining hospital survival [15,31]. In our three hypothetical

patients, Patient A has in fact the most severe form of acute critical

illness and worst short-term prognosis. Nevertheless, because this

patient is younger and has no co-morbidities, this patient has a

very reasonable and better long-term prognosis than Patient B and

C. If we use the prognostic model developed by Wright et al. [14]

to estimate the long-term survival of our three hypothetical

patients, Patient B will have the best 5-year prognosis (risk score is

estimated to be 68) followed by Patient C (risk score 75) and then

Patient A (risk score 87). The lack of detailed co-morbidity data

and a heavy emphasis on severity of acute illness in the model

developed by Wright et al. is the most likely explanation why our

results are different from theirs.

Many clinicians may intuitively consider the intensity of organ

failure as very important in affecting a patient’s prognosis [32,33].

Our findings suggest that the effect of acute organ failure on long-

term survival is not strong and mostly captured by age, co-

morbidities, and the APACHE II predicted mortality on admission

to ICU. Our previous studies have also showed that the intensity of

organ failure alone is not as important as the APACHE II score in

predicting hospital mortality [34,35]. Therefore, our findings

suggest that clinicians should be very careful not to place undue

emphasis on the severity of acute illness and intensity of organ

failure when making long-term prognostications of critically ill

patients.

Second, because the contributions by intensive care therapy are

relatively small when compared to age, Charlson co-morbidity

index, and the APACHE II predicted mortality, using the data

after the first 24 to 48 hours of ICU stay is unlikely to

underestimate the final total prediction score significantly (,20

points)(Figure 5). Therefore, early estimation of a slightly

‘optimistic’ long-term survival probability and median survival

time is feasible after the first 24 to 48 hours of ICU stay; and in

patients with either extremes of prognosis, this early estimation is

unlikely to be significantly different from the final prediction by

collecting all data after five days of intensive care therapy.

Nevertheless, the current prognostic model utilizes the APACHE

II predicted mortality after ICU admission as a predictor to

estimate long-term survival, as such, the model cannot be used, in

its current form, as a tool to triage ICU admission.

This study has significant limitations. First, patients’ wishes and

the anticipated quality of life before and after their critical illness

are important factors in making treatment decisions [36,37]. The

median survival time and long-term survival probabilities is only

one of the many factors that patients and clinicians may consider

in making treatment decisions. Furthermore, the c-statistics of this

model is only about 0.76 and this leaves considerable uncertainty

in its applicability in predicting long-term survival of individual

patients. As such, the predicted survival probabilities of this

prognostic model should only be considered as an average estimate

of patients with similar characteristics and should not be used for

individual patients. Second, evidence suggests that combining an

objective prognostic model with physicians’ intuition may improve

the accuracy of outcome prediction [13]. Whether combining this

current prognostic model with physicians’ intuition will improve its

predictive performance further remains uncertain, but this merits

further investigation. Third, although we studied a large cohort of

critically ill patients, and also the case-mix, severity of illness, and

Figure 6. Bootstrap estimate of calibration accuracy for 15-year estimates from the Cox proportional hazards model. Dotscorrespond to apparent predictive accuracy and x marks the bootstrap-corrected estimates.doi:10.1371/journal.pone.0003226.g006

PREDICT Model


in-hospital survival of this cohort is very similar to many other

ICUs in Australia [38], validation of this model by other ICUs that

have access to data linkage is essential to assess its generalizability.

Finally, although the APACHE II prognostic model is still widely

used for risk adjustment purposes in many ICUs [39,40], it is

possible that using newer prognostic models instead of the

APACHE II prognostic model may improve our current model

[41]. Similarly, the performance of the current model may be

improved if we consider more predictors in the model although

this will also increase the complexity of the model. In this regard,

we hope that the PREDICT model developed in this study will be

of value to others who aim to develop a new prognostic model to

enhance our understanding of long-term survival of critically ill

patients.

In summary, Age, gender, co-morbidities, severity of acute

illness, and the intensity and duration of intensive care therapy can

be used to estimate long-term survival of critically ill patients. Age

and co-morbidity are the most important determinants of the long-

term prognosis of critically ill patients. The current prognostic

model, the PREDICT model, provides a framework for

prognostications and risk adjustment when long-term survival of

critically ill patients is considered.

Author Contributions

Conceived and designed the experiments: KMH MK JF SAW. Analyzed

the data: KMH MK JF SAW. Contributed reagents/materials/analysis

tools: KMH. Wrote the paper: KMH MK JF SAW.

References

1. Acute Health Division DoHS (1997) Review of intensive care in Victoria [Phase

1 report]. Melbourne: Department of Human Services.2. Halpern NA, Bettes L, Greenstein R (1994) Federal and nationwide intensive

care units and healthcare costs: 1986–1992. Crit Care Med 22: 2001–2007.

3. Kvale R, Flaatten H (2002) Changes in intensive care from 1987 to 1997 - hasoutcome improved? A single centre study. Intensive Care Med 28: 1110–1116.

4. Poisal JA, Truffer C, Smith S, Sisko A, Cowan C, et al. (2007) Health spendingprojections through 2016: modest changes obscure part D’s impact. Health Aff

(Millwood) 26: w242–w253.

5. The Audit Commission (1999) Critical to Success. The place of efficient andeffective critical care services within the acute hospital. London: Audit

Commission for Local Authorities and the National Health Service in Englandand Wales.

6. Girard TD, Kress JP, Fuchs BD, Thomason JW, Schweickert WD, et al. (2008)Efficacy and safety of a paired sedation and ventilator weaning protocol for

mechanically ventilated patients in intensive care (Awakening and Breathing

Controlled trial): a randomised controlled trial. Lancet 371: 126–134.7. Sinuff T, Adhikari NK, Cook DJ, Schunemann HJ, Griffith LE, et al. (2006)

Mortality predictions in the intensive care unit: comparing physicians withscoring systems. Crit Care Med 34: 878–885.

8. Cook DJ, Guyatt GH, Jaeschke R, Reeve J, Spanier A, et al. (1995)

Determinants in Canadian health care workers of the decision to withdrawlife support from the critically ill. JAMA 273: 703–708.

9. Elstein AS, Christensen C, Cottrell JJ, Polson A, Ng M (1999) Effects ofprognosis, perceived benefit and decision style upon decision making in critical

care. Crit Care Med 27: 58–65.10. Garland A, Connors AF (2007) Physicians’ influence over decisions to forego life

support. J Palliat Med 10: 1298–1305.

11. Connors AF Jr (1999) The influence of prognosis on care decisions in thecritically ill. Crit Care Med 27: 5–6.

12. Hamel MB, Phillips RS, Davis RB, Desbiens N, Connors AF Jr, et al. (1997)Outcomes and cost-effectiveness of initiating dialysis and continuing aggressive

care in seriously ill hospitalized adults. SUPPORT Investigators. Study to

Understand Prognoses and Preferences for Outcomes and Risks of Treatments.Ann Intern Med 127: 195–202.

13. Knaus WA, Harrell FE Jr, Lynn J, Goldman L, Phillips RS, et al. (1995) TheSUPPORT prognostic model. Objective estimates of survival for seriously ill

hospitalized adults. Study to understand prognoses and preferences for outcomes

and risks of treatments. Ann Intern Med 122: 191–203.14. Wright JC, Plenderleith L, Ridley SA (2003) Long-term survival following

intensive care: subgroup analysis and comparison with the general population.Anaesthesia 58: 637–642.

15. Knaus WA, Draper EA, Wagner DP, Zimmerman JE (1985) APACHE II: aseverity of disease classification system. Crit Care Med 13: 818–829.

16. Williams TA, Dobb GJ, Finn JC, Knuiman M, Lee KY, et al. (2006) Data

linkage enables evaluation of long-term survival after intensive care. AnaesthIntensive Care 34: 307–315.

17. Ho KM, Dobb GJ, Knuiman M, Finn J, Lee KY, et al. (2006) A comparison ofadmission and worst 24-hour Acute Physiology and Chronic Health Evaluation

II scores in predicting hospital mortality: a retrospective cohort study. Crit Care

10: R4.18. Williams TA, Dobb GJ, Finn JC, Knuiman MW, Geelhoed E, et al. (2008)

Determinants of long-term survival after intensive care. Crit Care Med 36:1523–1530.

19. Charlson ME, Pompei P, Ales KL, MacKenzie CR (1987) A new method ofclassifying prognostic comorbidity in longitudinal studies: development and

validation. J Chronic Dis 40: 373–383.

20. Holman CD, Bass AJ, Rouse IL, Hobbs MS (1999) Population-based linkage ofhealth records in Western Australia: development of a health services research

linked database. ANZ J Public Health 23: 453–459.

21. Cox DR (1972) Regression models and life tables (with discussion). Journal of the

Royal Statistical Society 34: 187–220.

22. Lee SJ, Lindquist K, Segal MR, Covinsky KE (2006) Development and

validation of a prognostic index for 4-year mortality in older adults. JAMA 295:

801–808.

23. Pompei P, Charlson ME, Ales K, MacKenzie CR, Norton M (1991) Relating

patient characteristics at the time of admission to outcomes of hospitalization.

J Clin Epidemiol 44: 1063–1069.

24. Wyatt JC, Altman DG (1995) Prognostic models: clinically useful or quickly

forgotten? BMJ 311: 1539–1541.

25. Harrell FE Jr (2001) Regression Modeling Strategies. New York: Springer.

26. Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver

operating characteristic (ROC) curve. Radiology 143: 29–36.

27. Feringa HH, Bax JJ, Hoeks S, van Waning VH, Elhendy A, et al. (2007) A

prognostic risk index for long-term mortality in patients with peripheral arterial

disease. Arch Intern Med 167: 2482–2489.

28. Efron B, Tibshirani R (1993) An Introduction to the Bootstrap. New York:

Chapman & Hall.

29. Ho KM (2007) Forest and funnel plots illustrated the calibration of a prognostic

model: a descriptive study. J Clin Epidemiol 60: 746–751.

30. Nagelkerke NJ (1991) A note on a general definition of the coefficient of

determination. Biometrika 78: 691–692.

31. Ho KM, Finn J, Knuiman M, Webb SA (2007) Combining multiple

comorbidities with Acute Physiology Score to predict hospital mortality of

critically ill patients: a linked data cohort study. Anaesthesia 62: 1095–1100.

32. Cabre L, Mancebo J, Solsona JF, Saura P, Gich I, et al.; and the Bioethics

Working Group of the SEMICYUC (2005) Multicenter study of the multiple

organ dysfunction syndrome in intensive care units: the usefulness of Sequential

Organ Failure Assessment scores in decision making. Intensive Care Med 31:

927–933.

33. Nathens AB, Rivara FP, Wang J, Mackenzie EJ, Jurkovich GJ (2008) Variation

in the rates of do not resuscitate orders after major trauma and the impact of

intensive care unit environment. J Trauma 64: 81–88.

34. Ho KM, Lee KY, Williams T, Finn J, Knuiman M, et al. (2007) Comparison of

Acute Physiology and Chronic Health Evaluation (APACHE) II score with

organ failure scores to predict hospital mortality. Anaesthesia 62: 466–473.

35. Ho KM (2007) Combining sequential organ failure assessment (SOFA) score

with acute physiology and chronic health evaluation (APACHE) II score to

predict hospital mortality of critically ill patients. Anaesth Intensive Care 35:

515–521.

36. Ho KM, Liang J (2004) Withholding and withdrawal of therapy in New Zealand

intensive care units (ICUs): a survey of clinical directors. Anaesth Intensive Care

32: 781–786.

37. Ho KM, English S, Bell J (2005) The involvement of intensive care nurses in

end-of-life decisions: a nationwide survey. Intensive Care Med 31: 668–673.

38. Finfer S, Bellomo R, Lipman J, French C, Dobb G, et al. (2004) Adult-

population incidence of severe sepsis in Australian and New Zealand intensive

care units. Intensive Care Med 30: 589–596.

39. Harvey S, Harrison DA, Singer M, Ashcroft J, Jones CM, et al.; PAC-Man study

collaboration (2005) Assessment of the clinical effectiveness of pulmonary artery

catheters in management of patients in intensive care (PAC-Man): a randomised

controlled trial. Lancet 366: 472–477.

40. Finfer S, Bellomo R, Boyce N, French J, Myburgh J, et al. (2004) , SAFE Study

Investigators (2004) A comparison of albumin and saline for fluid resuscitation in

the intensive care unit. N Engl J Med 350: 2247–2256.

41. Zimmerman JE, Kramer AA, McNair DS, Malila FM (2006) Acute Physiology

and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for

today’s critically ill patients. Crit Care Med 34: 1297–1310.

PREDICT Model


136

Section five: Predicting long term survival after hospital

discharge

Chapter 14. The effect of socioeconomic status on long term

survival

Socioeconomic status (SES), as measured by individual-level indicators such as

education, income, and occupation or disadvantaged area-level indicators, is a

determinant of outcomes for many chronic diseases.78-81

Many seriously ill patients are

admitted to ICU only after other layers of the health care system have failed to prevent or

reverse the critical illness. It is possible that SES can have a significant effect on

outcomes of critical illnesses.82

So far, no prognostic scoring system for critically ill

patients has considered SES. This may be because it is difficult to classify patients into

different SES levels and perhaps also because such a prognostic scoring system would be

difficult to generalise or validate in other ICUs. Furthermore, the association between

SES and co-morbidities could also potentially confound the relationship between SES

and mortality outcome.

The current study hypothesised that SES of the patients can affect their hospital

and long term outcome after critical illness, over and above the usual biological

explanation such as severity of acute illness and co-morbidities. So as such, SES may

potentially affect the performance of a prognostic scoring system when it is applied to

cohorts with different SES. Specifically, this study assessed the effects of SES on

hospital and long term survival of critically or seriously ill patients, after adjustment for

demographic factors, pre-existing co-morbidities, type of ICU admission, severity of

137

acute illness, and geographical accessibility to essential services. In this study all data

from the RPHICU since 1988 were used in the modelling process initially. When the

APACHE II predicted mortality was used as a covariate in the latter stage of the

modelling, only patients with the APACHE II predicted mortality data since 1989 were

included.

The results of this study showed that SES was not significantly associated with

hospital mortality regardless of whether or not adjustments were made for age, elective

admission, co-morbidities, severity of acute illness, accessibility to essential services

(measured by ARIA), and indigenous status. SES was, however, independently

associated with the long term survival of the patients after adjusting for age, type of

admission, co-morbidities, severity of acute illness, ARIA, and indigenous status. In fact,

a progressive increase in risk of (long term) death was observed from SES group I (least

disadvantaged) to group VI (most disadvantaged). The relationship between SES and

long term survival remained unchanged and significant when the analysis was restricted

to hospital survivors only.

These findings suggest that SES is not a significant factor in determining hospital

mortality in RPHICU where universal free access to intensive care services was

available. SES was an important factor in determining patients’ long term survival after

critical illness. This study has confirmed that indigenous status is an important

determinant of long term survival after a critical illness. As such, SES and ethnicity may

need to be considered in the modelling of long term outcome of critically ill patients and

these factors may potentially affect the performance of a long term prognostic scoring

138

system when it is applied to cohorts with different SES and ethnicity backgrounds (e.g.

private hospital patients).

The reason why SES is important in determining long term survival remains

speculative. Although the relationship between long term survival and SES was apparent

even after adjustment for many potential confounders or predictors of long term survival,

there are still other factors that can affect long term survival of patients. Some possible

modifiable factors that can explain the relationship between SES and long term survival

may include financial and cultural barriers to specialist medical services, poor nutrition,

overcrowded accommodation, smoking or alcohol use, and physical inactivity. These

issues have significant public health implications and merit further investigation.


Ho KM, Dobb GJ, Knuiman M, Finn J, Webb SA. The effect of socioeconomic

inequalities on outcomes of seriously ill patients: a linked data cohort study. Medical

Journal of Australia 2008;189:26-30.

144

Section six: Conclusion

Chapter 15. Summary and directions for future research

The APACHE II scoring system was assessed within an Australian context and

found to perform well over the past 10 to 15 years, including when the model was applied

to critically ill indigenous patients. Its performance was also reasonable when modified to

use the admission physiological and laboratory data only and this simple modification of

the APACHE II scoring system (the Admission APACHE II scoring system) represents

a viable simpler alternative risk adjustment tool to the traditional APACHE II scoring

system. The APACHE II scoring system had a better performance than two organ failure

assessment scores (SOFA and RPHICU organ failure score), but its performance was

only marginally improved when combined with one of the organ failure scores (SOFA

score). The severe co-morbidity data modelled in the APACHE II also appeared to

capture most of the hospital mortality risk associated with co-morbidity and no

significant improvement in the model performance was achieved by incorporating

multiple severe or minor co-morbidities into the model.

The APACHE II scoring system has significant limitations. Firstly, using meta-

analytic techniques and also the slope and intercept of the calibration curve, the

APACHE II scoring system was found to be poorly calibrated in some subgroups of

patients such as patients with multiple trauma. This finding was consistent with other

studies.82

Secondly, the APACHE II scoring system had a limited ability to predict other

undesirable in-hospital outcomes such as unplanned ICU readmission or unexpected

death after ICU discharge in an institution where these events were uncommon. The

145

APACHE II scoring system did not perform well when applied to patients readmitted to

ICU during the same hospitalisation. While co-morbidity may be a risk factor for late

ICU readmission, it could not account for the excess hospital mortality associated with

ICU readmission.

Age, gender, co-morbidity, the severity of acute illness, as measured by the

APACHE II predicted mortality in the APACHE II scoring system, intensity and duration

of intensive care therapy, socioeconomic status, and indigenous status were all important

determinants of long term survival of patients after their critical illness. Using seven pre-

selected predictors, a new prognostic model, the PREDICT model, that can estimate

median survival time and long term survival probabilities of critically ill patients was

developed and presented. This is indeed the first prognostic model that can be used to

estimate median survival time and also long term (>5-year) survival probabilities after a

critical illness. This is a preliminary model and its performance needs to be validated (or

improved) by other ICUs where access to long term survival data is available. It is likely

that the PREDICT model may not be generalisable to all other ICUs because long term

survival of a patient can be affected by factors that are not considered in this model (eg

socioeconomic status and ethnicity).

There are more opportunities for further research in the use of prognostic scoring

systems to predict outcomes of critically ill patients. The APACHE II is still widely used

in many ICUs but has significant limitations. The latest version of the APACHE model

(version IV) was published in 2006 and a new Simplified Acute Physiology Score (SAPS

III) has also been published in 2005.28,84

It is likely that these newer prognostic models

will replace the older APACHE II scoring system with time. The studies presented in this

146

thesis have demonstrated how these new scoring systems can be assessed in an Australian

context and also in predicting other undesirable in-hospital outcomes such as unplanned

ICU readmission and unexpected death after ICU discharge. A new ICU discharge

prognostic scoring system, using predictors available at the time of ICU discharge such as

organ dysfunction or markers of inflammation, may also be useful to triage patients into

different risk categories before ICU discharge. Risk stratification of critically ill patients

before ICU discharge merits further investigation.

Long term outcomes of patients after critical illness are important, however,

survival is only one of the outcomes patients and clinicians will consider. Quality of life

after critical illness is also very important. A prognostic scoring system or model that can

estimate long term survival as well as quality of life will be extremely useful to patients

and clinicians in making difficult treatment decisions. In order to achieve this goal, we

need a population based multi-centre long term observational study with long term

follow-up visits to assess the quality of life after critical illness. This will require a

significant amount of resources but will generate a vast amount of useful information to

formulate our future health policy. Western Australia is geographically isolated and with

a very low emigration rate. The linkage of comprehensive ICU databases of all Western

Australia ICUs to long term survival information will put Western Australia in a very

unique position to achieve this important goal.

147

References of the whole thesis

(1) Angus DC, Sirio CA, Clermont G, Bion J. International comparisons of critical

care outcome and resource consumption. Crit Care Clin 1997;13:389-407.

(2) Halpern NA, Bettes L, Greenstein R. Federal and nationwide intensive care units

and healthcare costs: 1986-1992. Crit Care Med 1994;22:2001-7.

(3) Poisal JA, Truffer C, Smith S, et al. Health spending projections through 2016:

modest changes obscure part D's impact. Health Aff (Millwood) 2007;26:w242–53.

(4) The Audit Commission. Critical to Success. The place of efficient and effective

critical care services within the acute hospital. London: Audit Commission for Local

Authorities and the National Health Service in England and Wales, 1999.

(5) Clarke T, Hart LG. Review of ICU Resources & Activity 00/01: ANZICS, 2002.

(6) Rechner IJ, Lipman J. The costs of caring for patients in a tertiary referral

Australian intensive care unit. Anaesth Intensive Care 2005;33:477-82.

(7) Acute Health Division DoHS. Review of intensive care in Victoria [Phase 1

report]. Melbourne: Department of Human Services, 1997.

(8) Ridley S, Morris S. Cost effectiveness of adult intensive care in the UK.

Anaesthesia 2007;62:547-54.

(9) Colagiuri S, Walker AE. Using an economic model of diabetes to evaluate

prevention and care strategies in Australia. Health Aff (Millwood) 2008;27:256-68.

(10) Hynes N, Sultan S. A prospective clinical, economic, and quality-of-life analysis

comparing endovascular aneurysm repair (EVAR), open repair, and best medical

treatment in high-risk patients with abdominal aortic aneurysms suitable for EVAR:

the Irish patient trial. J Endovasc Ther 2007;14:763-76.

(11) Kvåle R, Flaatten H. Changes in intensive care from 1987 to 1997 - has outcome

improved? A single centre study. Intensive Care Med 2002;28:1110-6.


http://www.ncbi.nlm.nih.gov/sites/entrez?Db=pubmed&Cmd=Search&Term=%22Colagiuri%20S%22%5BAuthor%5D&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_DiscoveryPanel.Pubmed_RVAbstractPlus

http://www.ncbi.nlm.nih.gov/sites/entrez?Db=pubmed&Cmd=Search&Term=%22Walker%20AE%22%5BAuthor%5D&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_DiscoveryPanel.Pubmed_RVAbstractPlus


148

(12) Rosenberg AL, Watts C. Patients readmitted to ICUs: a systematic review of risk

factors and outcomes. Chest 2000;118:492-502.

(13) Azoulay E, Adrie C, De Lassence A, Pochard F, Moreau D, Thiery G, Cheval C,

Moine P, Garrouste-Orgeas M, Alberti C, Cohen Y, Timsit JF. Determinants of

postintensive care unit mortality: a prospective multicenter study. Crit Care Med

2003;31:428-32.

(14) Knaus WA, Harrell FE, Lynn J, et al. The SUPPORT Prognostic Model;

objective estimates of survival for seriously ill hospitalized adults. Ann Intern Med

1995;122:191-203.

(15) Girard TD, Kress JP, Fuchs BD, et al. Efficacy and safety of a paired sedation

and ventilator weaning protocol for mechanically ventilated patients in intensive care

(Awakening and Breathing Controlled trial): a randomised controlled trial. Lancet

2008;371:126-34.

(16) Rocker G, Cook D, Sjokvist P, et al. Clinician predictions of intensive care unit

mortality. Crit Care Med 2004;32:1149-54.

(17) Girbes ARJ. Dying at the end of your life. Intensive Care Med 2004;30:2143-4.

(18) Griffith L, Cook D, Hanna S, et al., for the Level of Care Investigators and the

Canadian Critical Care Trials Group. Clinician discomfort with life support plans for

mechanically ventilated patients. Intensive Care Med 2004;30:1783-90.

(19) Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying

prognostic comorbidity in longitudinal studies: development and validation. J Chronic

Dis 1987;40:373-83.

(20) Vincent JL, Moreno R, Takala J, et al. The SOFA (Sepsis-related Organ Failure

Assessment) score to describe organ dysfunction/failure. On behalf of the Working

http://www.ncbi.nlm.nih.gov.rplibresources.health.wa.gov.au/sites/entrez?Db=pubmed&Cmd=ShowDetailView&TermToSearch=12576947&ordinalpos=76&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_RVDocSum




http://www.ncbi.nlm.nih.gov.rplibresources.health.wa.gov.au/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=8844239&query_hl=4&itool=pubmed_docsum

149

Group on Sepsis-Related Problems of the European Society of Intensive Care

Medicine. Intensive Care Med 1996;22:707-10.

(21) Ferreira FL, Bota DP, Bross A, Melot C, Vincent JL. Serial evaluation of the

SOFA score to predict outcome in critically ill patients. JAMA 2001;286:1754-8.

(22) Kajdacsy-Balla Amaral AC, Andrade FM, Moreno R, Artigas A, Cantraine F,

Vincent JL. Use of the sequential organ failure assessment score as a severity score.

Intensive Care Med 2005;31:243-9.

(23) Gunning K, Rowan K. Outcome data and scoring systems. BMJ 1999;319:241-4.

(24) Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of

disease classification system. Crit Care Med 1985;13:818-29.

(25) Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG,

Sirio CA, Murphy DJ, Lotring T, Damiano A, et al. APACHE III prognostic system.

Risk prediction of hospital mortality for critically ill hospitalized adults. Chest 1991;

100:1619-36.

(26) Cook DA. Performance of APACHE III models in an Australian ICU. Chest

2000;118:1732-8.

(27) Buist MD, Gould T, Haglet S, Webb R. An analysis of excess mortality not

predicted to occur by Apache III in an Australian level III intensive care unit. Anaesth

Intensive Care 1999;28;171-7.

(28) Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and

Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's

critically ill patients. Crit Care Med 2006;34:1297-310.

(29) Konarzewski W. Continuing to use APACHE II scores ensures consistency. BMJ

2000;321:383-4.

(30) Knaus WA. APACHE 1978-2001: The development of a quality assurance




/entrez/query.fcgi?cmd=Retrieve&db=pubmed&dopt=Abstract&list_uids=1959406&query_hl=11



150

system based on prognosis. Milestones and personal reflections. Arch Surg

2002;137:37-41.

(31) Ledoux D, Finfer S, McKinley S. Impact of operator expertise on collection of

the APACHE II score and on the derived risk of death and standardized mortality

ratio. Anaesth Intensive Care 2005;33:585-90.

(32) Khilnani G, Banga A, Sharma S. Predictors of mortality of patients with acute

respiratory failure secondary to chronic obstructive pulmonary disease admitted to an

intensive care unit: A one year study. BMC Pulm Med 2004;4:12.

(33) Goel A, Pinckney RG, Littenberg B. APACHE II predicts long-term survival in

COPD patients admitted to a general medical ward. J Gen Intern Med 2003;18:824-

30.

(34) Duke G, Santamaria J, Shann F, et al. Outcome-based clinical indicators for

intensive care medicine. Anaesth Intensive Care 2005;33:303-10.

(35) The Australian Council on Healthcare Standards. ACHS clinical indicator

summary guide. An approach to demonstrating the dimensions of quality. http://

www.achs.org.au (accessed on 18 October 2008).

(36) Paratz J, Thomas P, Adsett J. Re-admission to intensive care: identification of

risk factors. Physiother Res Int 2005;10:154-63.

(37) Duke GJ, Green JV, Briedis JH. Night-shift discharge from intensive care unit

increases the mortality-risk of ICU survivors. Anaesth Intensive Care 2004;32:697-

701.

(38) Ho KM, Liang J. Withholding and withdrawal of therapy in New Zealand

intensive care units (ICUs): a survey of clinical directors. Anaesth Intensive Care

2004;32:781-6.


http://www.achs.org.au/

151

(39) Griffiths J, Fortune G, Barber V, Young JD. The prevalence of post traumatic

stress disorder in survivors of ICU treatment: a systematic review. Intensive Care

Med 2007;33:1506-18.

(40) Williams TA, Dobb GJ, Finn JC, Webb SA. Long-term survival from intensive

care: a review. Intensive Care Med 2005;31:1306-15.

(41) Wright JC, Plenderleith L, Ridley SA. Long-term survival following intensive

care: subgroup analysis and comparison with the general population. Anaesthesia

2003;58:637-42.

(42) Ding J, Diez Roux AV, Nieto FJ, et al. Racial disparity in long-term mortality

rate after hospitalisation for myocardial infarction: the Atherosclerosis Risk in

Communities study. Am Heart J 2003;146:459-64.

(43) Tonne C, Schwartz J, Mittleman M, et al. Long-term survival after acute

myocardial infarction is lower in more deprived neighborhoods. Circulation 2005;

111:3063-70.

(44) Trewin D. Information Paper: Outcomes of ABS views on remoteness

consultation, Australia. Canberra: Australian Bureau of Statistics, 2001. 30.

(45) Jaro MA. Probabilistic linkage of large public health data files. Stat Med

1995;14:491-8.

(46) Holman CD, Bass AJ, Rouse IL, Hobbs MS. Population-based linkage of health

records in Western Australia: development of a health services research linked

database. Aust N Z J Public Health 1999;23:453-9.

(47) Health information Centre (Health Statistics Section). Hospital Inpatient

Summary (HA22) Reference Manual. Perth: Health Department of Western Australia,

1996.


152

(48) Romano PS, Roos LL, Jollis JG. Adapting a clinical comorbidity index for use

with ICD-9-CM administrative data: differing perspectives. J Clin Epidemiol

1993;46:1075-1079; discussion 1081-90.

(49) Trewin D. Socio-economic indexes for areas, Australia 2001. Information paper

2039.0, census of population and housing. Australian Bureau of Statistics, Canberra.

(50) Australian Institute of Health and Welfare 2004. Rural, regional and remote

health: a guide to remoteness classifications. AIHW cat. No. PHE 53. Canberra:

AIHW.

(51) Dobb GJ. Intensive care in Australia and New Zealand. No nonsense "down

under". Crit Care Clin 1997;13:299-316.

(52) Worthley LI. The ideal intensive care unit: open, closed or somewhere between?

Crit Care Resusc 2007;9:219-20.

(53) Jones DA, Cooper DJ, Finfer SR, et al. Advancing intensive care research in

Australia and New Zealand: development of the binational ANZIC Research Centre.

Crit Care Resusc 2007;9:198-204.

(54) Finfer S, Bellomo R, Lipman J, French C, Dobb G, Myburgh J. Adult-population

incidence of severe sepsis in Australian and New Zealand intensive care units.

Intensive Care Med 2004;30:589-96.

(55) Finfer S, Bellomo R, Boyce N, French J, Myburgh J, Norton R; SAFE Study

Investigators. A comparison of albumin and saline for fluid resuscitation in the

intensive care unit. N Engl J Med 2004;350:2247-56.

(56) Taori G, Ho KM, George C, Webb SA, Bellomo R, Hart G. Optimal time frame

end-point for assessing survival of critically ill patients: a comparative cohort study

(submitted for publication).







153

(57) Ho KM, Dobb GJ, Lee KY, Towler SC, Webb SA. C-reactive protein

concentration as a predictor of intensive care unit readmission: a nested case-control

study. J Crit Care 2006;21:259-65.

(58) Hanley JA, McNeil BJ. A method of comparing the areas under receiver

operating characteristic curves derived from the same cases. Radiology 1983;148:839-

43.

(59) Lemeshow S, Hosmer DW. A review of goodness of fit statistics for use in the

development of logistic regression model. Am J Epidemiol 1982; 115:92-106.

(60) Ho KM. Forest and funnel plots illustrated the calibration of a prognostic model:

a descriptive study. J Clin Epidemiol 2007;60:746-51.

(61) Cox DR. Regression models and life tables (with discussion). Journal of the

Royal Statistical Society 1972;34:187-220.

(62) Wyatt JC, Altman DG. Prognostic models: clinically useful or quickly forgotten?

BMJ 1995;311:1539-41.

(63) Harrell FE Jr. Regression Modeling Strategies. New York: Springer; 2001.

(64) Efron B, Tibshirani R. An Introduction to the Bootstrap. New York: Chapman &

Hall; 1993.

(65) Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD. Substantial effective

sample sizes were required for external validation studies of predictive logistic

regression models. J Clin Epidemiol 2005;58:475-83.

(66) Arkes HR, Dawson NV, Speroff T, et al. The covariance decomposition of the

probability score and its use in evaluating prognostic estimates. Med Decis Making

1995;15:120-31.

(67) Nagelkerke NJ. A note on a general definition of the coefficient of determination.

Biometrika 1991;78:691-2.





154

(68) Kramer AA. Predictive mortality models are not like fine wine. Crit Care

2005;9:636-7.

(69) Stephens D. Critical illness and its impact on the aboriginal people of the top end

of the Northern Territory, Australia. Anaesth Intensive Care 2003;31:294-9.

(70) Ho KM, Lee KY, Williams T, Finn J, Knuiman M, Webb SA. Comparison of

Acute Physiology and Chronic Health Evaluation (APACHE) II score with organ

failure scores to predict hospital mortality. Anaesthesia 2007;62:466-73.

(71) Williams TA, Dobb GJ, Finn JC, et al. Data linkage enables evaluation of long-

term survival after intensive care. Anaesth Intensive Care 2006;34:307-15.

(72) Elixhauser A, Steiner C, Harris DR, et al. Comorbidity measures for use with

administrative data. Medical Care 1998;36:8-27.

(73) Angus DC. Grappling with intensive care quality-does the readmission rate tell

us anything? Crit Care Med 1998;26:1779-80.

(74) Beck DH, McQuillan P, Smith GH. Waiting for the break of dawn? The effects

of discharge time, discharge TISS scores and discharge facility on hospital mortality

after intensive care. Intensive Care Med 2002;28:1287-93.

(75) Ball C, Kirkby M, Williams S. Effect of the critical care outreach team on patient

survival to discharge from hospital and readmission to critical care: non-randomised

population based study. BMJ 2003;327:1014-7.

(76) Reny JL, Vuagnat A, Ract C, et al. Diagnosis and follow-up of infections in

intensive care patients: value of C-reactive protein compared with other clinical and

biological variables. Crit Care Med 2002;30:529-35.

(77) Kaben A, Corrêa F, Reinhart K, et al. Readmission to a surgical intensive care

unit: incidence, outcome and risk factors. Crit Care 2008;12:R123.

(78) Marmot MG. Status syndrome: a challenge to medicine. JAMA 2006;295:1304-7.

/pubmed/16356253?ordinalpos=12&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_DefaultReportPanel.Pubmed_RVDocSum



/pubmed/18838006?ordinalpos=4&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_DefaultReportPanel.Pubmed_RVDocSum

155

(79) Turrell G, Mathers CD. Socioeconomic status and health in Australia. Med J Aust

2000;172:434-8.

(80) Turrell G, Mathers C. Socioeconomic inequalities in all-cause and specific-cause

mortality in Australia: 1985-1987 and 1995-1997. Int J Epidemiol 2001;30:231-9.

(81) Volkers AC, Westert GP, Schellevis FG. Health disparities by occupation,

modified by education: a cross-sectional population study. BMC Public Health

2007;7:196.

(82) Alter DA, Chong A, Austin PC, et al.; SESAMI Study Group. Socioeconomic

status and mortality after acute myocardial infarction. Ann Intern Med 2006;144:82-

93.

(83) Chawda MN, Hildebrand F, Pape HC, Giannoudis PV. Predicting outcome after

multiple trauma: which scoring system? Injury 2004;35:347-58.

(84) Moreno RP, Metnitz PG, Almeida E, et al; SAPS 3 Investigators. SAPS 3--From

evaluation of the patient to evaluation of the intensive care unit. Part 2: Development

of a prognostic model for hospital mortality at ICU admission. Intensive Care Med

2005;31:1345-55.

http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=10870537&query_hl=3&itool=pubmed_DocSum

http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Retrieve&dopt=AbstractPlus&list_uids=11369721&query_hl=3&itool=pubmed_DocSum


http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Search&itool=pubmed_Abstract&term=%22Alter+DA%22%5BAuthor%5D

http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Search&itool=pubmed_Abstract&term=%22Chong+A%22%5BAuthor%5D

http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Search&itool=pubmed_Abstract&term=%22Austin+PC%22%5BAuthor%5D

http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=pubmed&cmd=Search&itool=pubmed_Abstract&term=%22SESAMI+Study+Group%22%5BCorporate+Author%5D

156

Appendices

1. Confidentiality of Health Information Committee (CHIC) approval letter:

(#200321) Outcomes of critically illness in intensive care unit patients: RPHICU

linked data project.

2. Western Australian Aboriginal Health Information and Ethics Committee

(WAAHIEC) approval letter: (Ho 93-02/05) The short term outcome of critically

ill Aboriginal patients in a tertiary intensive care unit in Western Australia.


(WAAHIEC) approval letter: (158-02/07) The impact of socioeconomic status on

outcomes of critically ill patients: a linked data cohort study.


(WAAHIEC) endorsement to publish manuscript letter: (Ho 93-02/05) The impact

of socioeconomic status on outcomes of critically ill patients: a linked data cohort

study.

use of prognostic scoring systems to · section four: the use of inflammatory markers in addition...

Documents