International Journal for Quality in Health Care 2002; Volume 14, Number 6: pp. 483492

Metric properties of the appropriatenessevaluation protocol and predictors ofinappropriate hospital use in Germany:an approach using longitudinal patientdataOLIVER SANGHA1,2,3, SEBASTIAN SCHNEEWEISS4,5, MANFRED WILDNER2,3, E. FRANCIS COOK4,6,TROYEN A. BRENNAN1,6, JENS WITTE7 AND MATTHEW H. LIANG1,6

1Department of Health Policy and Management, 4Department of Epidemiology, Harvard School of Public Health, Boston, MA,2Robert B. Brigham Arthritis & Musculoskeletal Diseases Clinical Research Center, 6Division of General Internal Medicine,5Division of Pharmacoepidemiology and Pharmacoeconomics, Brigham and Womens Hospital, Boston, MA, USA,3Department of Medical Informatics, Biometrics, and Epidemiology, Ludwig-Maximilians-University, School of Medicine,Munich, 7Department of General Surgery, Central Hospital, Augsburg, Germany

Abstract

Background. The German health care system, renowned for its unrestricted access, high quality care, and comprehensivecoverage, is challenged by increasing health care costs. This has been attributed partly to inefficiencies in the in-patientsector, but has been studied little. Attempts at quality improvement need to relate costs to outcomes. Until now, there hasbeen no standardized methodology to evaluate the appropriateness of hospital care.

Objective. To develop and evaluate the metric properties of a method to assess inappropriate hospital care in Germanybased on a widely used measure, the Appropriateness Evaluation Protocol (AEP).

Methods. The original AEP was translated and adapted to reflect differences in the provision of health care in Germany.Psychometric testing was performed in a stratified sample of all patients admitted to the Departments of Medicine andSurgery of a 400-bed teaching hospital during 1 year. Three board-certified physicians participated in each department toevaluate intra-rater reliability, while two additional independent physicians judged inter-rater reliability.

Results. Inter-rater agreement for the evaluation of hospital days among surgical patients was 84% (8087%), with anaverage kappa value of 0.58 (0.480.68). Corresponding figures for patients in medicine were 76% (7380%) with a valueof 0.42 (0.340.42). Inter-rater agreement for hospital admissions and was 74% (6286%) and 0.44 (0.210.67) in surgery,and 92% (85100%) and 0.31 (00.80) in medicine, respectively. Thirty-three per cent of all admissions and 28% ofconsecutive hospital days were judged inappropriate in surgery; among medicine patients, reviewers found 6% of admissionsand 33% of hospital days inappropriate. Time since admission was the strongest predictor of inappropriate hospital useadjusted for length of stay, comorbidity, age, and gender.

Keywords: Appropriateness Evaluation Protocol, appropriateness of care, Germany, health services research, qualityimprovement, utilization review

National health expenditures in developed countries have public expectations, and an aging population. Even the wealth-iest nations are compelled to cut back on costs and re-increased enormously as a result of the explosion of expensive

medical technology and its dissemination, which has increased examine their assumptions about the coverage of health

Address reprint requests to S. Schneeweiss, Brigham and Womens Hospital and Harvard Medical School, Divisionof Pharmacoepidemiology and Pharmacoeconomics, 221 Longwood Ave (BLI-341), Boston, MA 02115, USA.E-mail: schneeweiss@post.harvard.edu Deceased.

2002 International Society for Quality in Health Care and Oxford University Press 483

O. Sangha et al.

services. Since the single largest budget item for medical careuntil recently was hospital care, efforts have been made toreduce costs in this sector, particularly through avoidingunnecessary and prolonged hospitalizations. Eliminating un-necessary hospitalizations may also improve the quality ofcare and increase access to care for patients on waiting listsin some health care systems [1].

Substantial rates of inappropriate hospital care have beenreported in several countries [213], but few centers havebeen able to reduce inappropriate care using the results ofsuch studies [12,14,15]. Avoiding inappropriate hospital careis difficult, in part because its determinants are multifactorialand include to various degrees practice patterns, patientcharacteristics, the organization of in-hospital care, the co-ordination between hospital care and other providers withinthe health care system, and financial incentives.

The German health care system, internationally renownedfor unrestricted access, high quality care and comprehensivecoverage, is particularly challenged, since the number ofhospitalizations per capita and average length of stay (LOS)is substantially higher than in most industrialized countries.Hence, there are increasing efforts to reduce hospital ad-missions and inappropriate in-patient care. Beginning in1995, the sickness funds commissioned statewide revieworganizations (MDK, comparable to the US Physician ReviewOrganizations) to conduct systematic reviews of selected

Figure 1 Patient sampling.hospital departments. However, the review methodology wasdeveloped ad hoc and has not been tested for reliability andvalidity.

The aim of this study was to develop and evaluate theof factors potentially responsible for such medically un-metric properties of a method to assess the appropriatenessnecessary episodes of care using a complementary list ofof hospital care in Germany based on a widely used measure,reasons and alternatives (reasons list).the Appropriateness Evaluation Protocol (AEP).

The AEP also rates the appropriateness of hospital ad-missions using 17 criteria, which pertain to clinical stabilityof the patient, necessity of medical interventions, and plannedsurgical procedures within 24 hours. An admission is deemedMethodsappropriate if one or more of these criteria are satisfied. TheAEP allows an override option, enabling the reviewer toModification of the AEPevaluate an admission or particular hospital day as ap-

The methodology to assess the appropriateness of in-patient propriate, despite the absence of one criterion. Conversely,care in Germany was based on the AEP, developed by it may be classified as inappropriate, even if one or moreGertman and Restuccia in 1981 [2], and later modified [16] of the criteria are met.and validated as a technique to assess unnecessary days of The original instrument was translated into German by ahospital care. The instrument has been implemented in many native English speaker who was fluent in German, and by ahospitals and has proven useful [6,8,12,1618]. Method- German health care professional who was fluent in English.ological studies of the AEP have focused on its reliability All items were reviewed by an expert panel, includingand validity [2,8,1923], adaptation of the original protocol representatives from professional societies of surgery, ge-to different kinds of hospitals, units within hospitals, or riatrics, and internal medicine, from different State MDKsdiagnostic groups [3,19,24], or the denominators used to and from the Department of Epidemiology of the Universityobtain inappropriateness rates [6]. of Munich, who provided methodological expertise. Changes

The AEP uses 27 criteria to assess the appropriateness of were considered whenever criteria were not German healtheach hospital day (11 relate to medical services/procedures, care practices, or health services not provided in Germany.seven to nursing/life support services, and nine to clinical New criteria were added if newer developments in practicingcharacteristics of the patient necessitating close observation). medicine were not considered in the original AEP.Once a day has been identified as medically unnecessary (i.e. The instrument was then converted to a computerizedno information in the medical record corresponding to any version, allowing reviewers online access and providing mul-

tiple pull-down menus to ease chart abstraction and to permitof the 27 explicit criteria), the AEP also allows the description

484

Hospital use in Germany

Figure 2 Distribution of overall and specific inter-rater agreement in 54 surgical patients.

data entry in a database with automated plausibility checks. examine inter-rater-reliability, four additional reviewers (twoboard-certified internists, two surgeons) from the MDK ofThe instrument is available on the internet [25].the State of Bavaria reviewed 30 charts each, previouslyabstracted by physicians from the Hessen MDK.Study design and patient sample

The study was carried out at a 400-bed teaching hospital in Statistical analysesFrankfurt, Germany, which volunteered its data. All patientsadmitted to general internal medicine and surgery were eligible Sample size estimates were based on the ability to detect a rate

of inappropriate hospitalizations per bed days of 15%3%for retrospective evaluation with the AEP. A stratified sample,with 60 strata based on age (less than, and greater than or [proportionstandard error] with a statistical power of

90%.equal to median age), hospital length of stay (less than, andgreater than or equal to median LOS), 14 most prevalent Reliability between reviewers (inter-rater) and within one

reviewer (intra-rater) was calculated for admission and eachdiagnoses, and one stratum including the remaining diagnoses,was drawn from discharge lists. Prior to sampling, patients hospital day in three ways: (1) overall agreement by dividing

the number of patients where both reviewers agreed oncovered by private health insurance or welfare were removedfrom the list since review of medical records by the MDKs appropriateness of a day by all patients on that specific day

(e.g. admission, day 1 and so forth); (2) specific agreementto assess appropriateness was only allowed for patientscovered by national health insurance (>90% of the popu- by dividing the number of patients where both reviewers

agreed on appropriateness by the total number of patients;lation). The sampling is displayed in Figure 1. Completehospital stays of each sampled patient, including admission and (3) overall agreement between pairs of reviewers by

Cohens kappa statistic [26], which adjusts for the amountand all hospital days, were studied.Three board-certified internists (reviewing medical patients) of agreement occurring by chance alone. It should be noted

that the value may be paradoxically low as observed inand three board-certified surgeons (reviewing surgicalpatients) who are employed by the MDK of the State of these data when overall agreement is high, particularly when

the prevalence of appropriate days is low, also as observedHessen conducted the reviews. Reviewers were instructed tobase their admission assessment only on the basis of the in these data [27].

Previous reports on the reliability of instruments measuringmedical information relative to the day of admission and thefollowing 24 hours. After 3 weeks, two internists and two appropriateness of hospital care treated all hospital-days in

the same way, disregarding whether they were the first, secondsurgeons were asked to review 25 medical records each toassess intra-rater reliability. None of the reviewers was aware or the last days a patient stayed in a hospital. No justification

was provided for why inter-rater reliability should be theof this reliability exercise at the time of the initial review. To

485

O. Sangha et al.

Figure 3 Distribution of overall and specific inter-rater agreement in 49 general internal medicine patients.

Table 1 Proportions of inappropriate admissions and hospital days in Departments of General Internal Medicine andSurgery

Patients1 Days Proportion 95% confidence limitsinappropriate ....................................................

(%) Lower (%) Upper (%).............................................................................................................................................................................................................................Surgery

Inappropriate admissions 269 N/A 33 30 36Inappropriate hospital days 243 2277 28 24 32

Internal medicineInappropriate admissions 245 N/A 6 0 12Inappropriate hospital days 222 3200 33 29 34

1The number of patients eligible for hospital day evaluation has been reduced by the number of patients leaving on the same day.

same at the beginning, the middle, and the end of a stay. patients (days 122 in general internal medicine, days 114in surgery). Overall and specific percentage agreement wasTherefore, we calculated reliability for individual days in the

course of a hospitalization (i.e. for the admission day, day 1, also calculated for the sum of all hospital days. Overall, for hospital days is computed according to Fleiss [28], afterday 2, . . .) [2,8,23]. Since there are increasingly fewer patients

with longer hospital stays, we restricted the reliability evalu- testing whether the values are equal among days withCochrans Q-test [29]. Landis and Koch suggested that ation to those days for which we could identify at least eight

486

Hospital use in Germany

Table 2 Frequency of inappropriate hospital days by patient characteristics among surgical patients

Patients, n (%) Total hospital Proportion 95% confidencedays inappropriate days interval1.............................................................................................................................................................................................................................

Total 243 (100) 2277 (100) 27.8 23.731.9Sex

Male 119 (49.0) 1087 (47.7) 30.2 23.936.5Female 124 (51.0) 1190 (52.3) 25.5 20.031.0

Age (years)49 100 (41.2) 773 (33.9) 28.6 21.535.75059 41 (16.9) 320 (14.1) 20.4 10.630.26069 41 (16.9) 394 (17.3) 33.1 23.043.27079 31 (12.8) 459 (20.2) 24.3 13.734.98089 23 (9.5) 245 (10.8) 36.0 25.846.2[90 7 (2.9) 86 (3.8) 18.0 0.235.8

Appropriate admissionYes 170 (70.0) 1643 (72.2) 22.1 18.226.0No 73 (30.0) 634 (27.8) 44.1 34.553.72

Length of stay (days)17 103 (42.4) 397 (17.4) 32.5 25.040.0814 91 (37.4) 775 (34.0) 25.0 19.031.01521 35 (14.4) 586 (25.7) 23.5 14.632.42228 7 (2.9) 158 (6.9) 27.1 7.846.4[29 7 (2.9) 361 (15.9) 19.2 5.932.5

Number of comorbid conditions0 187 (77.0) 1505 (66.1) 29.5 24.634.41 44 (18.1) 600 (26.4) 21.6 13.030.22 10 (4.1) 127 (5.6) 27.3 12.442.2[3 2 (0.8) 45 (2.0) 7.4 4.610.22

195% confidence limits were calculated with the empirical standard deviation of the mean proportion and normal approximation [31].2P < 0.05.

values of >0.75 may be taken to represent excellent agreement, Resultswhile values

O. Sangha et al.

Table 3 Frequency of inappropriate hospital days by patient characteristics among general internal medicine patients

General internal Total hospital Proportion 95% confidencemedicine patients, n (%) days inappropriate days interval1.............................................................................................................................................................................................................................

Total 222 (100) 3200 (100) 33.0 28.937.1Sex

Male 103 (46.4) 1540 (48.1) 25.3 19.930.72

Female 119 (53.6) 1660 (51.9) 39.6 33.945.3Age (years)49 38 (17.1) 360 (11.3) 37.2 26.148.35059 30 (13.5) 475 (14.8) 33.1 21.644.66069 44 (19.8) 608 (19.0) 40.0 30.050.07079 40 (18.0) 543 (16.9) 25.4 18.032.88089 54 (24.3) 857 (26.8) 30.0 22.038.0[90 16 (7.2) 357 (11.2) 33.1 18.947.3

Appropriate admissionYes 210 (94.6) 3075 (96.1) 31.3 27.435.2No 12 (5.4) 125 (3.9) 60.4 35.585.32

Length of stay (days)17 44 (19.8) 125 (3.9) 23.3 12.134.5814 84 (37.8) 845 (26.4) 36.6 30.442.81521 49 (22.1) 836 (26.1) 35.6 27.543.72228 21 (9.5) 550 (17.2) 28.7 17.640.0[29 21 (9.5) 844 (26.4) 36.9 24.149.7

Number of comorbid conditions0 50 (22.5) 542 (16.9) 39.3 29.649.01 49 (22.1) 604 (18.9) 28.7 20.337.12 71 (32.0) 1155 (36.1) 33.5 26.340.7[3 52 (23.4) 899 (28.1) 30.0 22.737.3

195% confidence limits were calculated with the empirical standard deviation of the mean proportion and normal approximation [31].2P < 0.05.

of Surgery. One hundred and ninety-three medicine patients are shown in Figure 2. Day 0 refers to the judgement ofwhether an admission was appropriate or not. Taking alland 290 surgery patients were covered by private insurancehospital days (excluding the admission day) into account,or welfare, and therefore not eligible for review by stateoverall agreement is 84% (8087%), with an average valueMDKs. Sample-size calculations indicated 278 medicineof 0.58 (95% CI 0.480.68). Kappa values did not differpatients and 292 surgery patients as the necessary targetamong days (QK=8.8, P=0.79). The corresponding valuesnumbers to detect a rate of 15% (3%) inappropriatefor the reliability of admission appropriateness are 74%hospital episodes. In internal medicine and surgery, 242(6286%) overall agreement, and 0.44 (0.210.67).and 260 records were available for review, respectively. The

difference between the numbers in the target samples andInter-rater reliability among interniststhose that were reviewed was due to missing records (n=

29), miscoding of insurance status (n=9), or incomplete Inter-rater agreement was calculated among 49 medicinemedical records (n=30). patients. Figure 3 shows overall and specific agreement as

Patients in medicine had an average age of 66.1 (18.1) well as statistics for admissions and consecutive hospitalyears, were 53.7% female, and had an average LOS of 14.5 days. Overall agreement between two reviewers was 76%(11.5) days. Surgical patients were 54.1 (20.9) years old, (7380%), with an average value of 0.42 (0.340.49) for58.9% female, and had an average LOS of 9.9 (10.4) days. all hospital days, and 92% (85100%) with a value of 0.31

(00.80) for admissions. Kappa values did not differ amongInter-rater reliability among surgeons days (QK=15.2, P=0.82).Fifty-four patients were available for the inter-rater agreement

Intra-rater reliability among surgeonssubstudy among surgical patients. Overall and specific agree-ment and statistics between two different reviewers, in- Intra-rater reliability was assessed for 51 surgery and 49

medicine patients. Medical records were reviewed twice, 3cluding 95% confidence limits for individual hospital days,

488

Hospital use in Germany

Table 4 Patient- and day-specific characteristics associated with inappropriateness of hospital days, in 243patients admitted to a Department of Surgery, totaling 2633 days of hospitalization

Odds ratio 95% confidence interval1 P trend...............................................................................................................................................................................................Sex

Male 1 Female 1.2 0.901.67

Age (years)49 1 5059 0.78 0.431.46069 1.3 0.772.27079 1.2 0.632.38089 2.0 1.13.62

[90 0.67 0.162.83Appropriateness of admission

(inappropriate versus appropriate) 1.4 0.922.2Length of stay (days)

17 1

O. Sangha et al.

Table 5 Patient- and day-specific characteristics associated with inappropriateness of hospital days, in 222 patients admittedto a Department of General Internal Medicine, totaling 3509 days of hospitalization

Odds ratio 95% confidence interval1 P trend.............................................................................................................................................................................................................................Sex

Male 1 Female 1.5 1.012.32

Age (years)49 1 5059 0.88 0.441.86069 1.1 0.572.27079 0.95 0.491.98089 0.96 0.491.9[90 1.4 0.483.9

Appropriateness of admission 0.61 0.201.9(inappropriate versus appropriate)Length of stay (days)

17 1

Hospital use in Germany

to evaluate appropriateness of hospital care in Germany. This effect was even more pronounced when we restrictedour analysis to patients who were appropriately admitted,Review of the original instrument and adaptation to thesince patients with short hospitalizations undergoing medicalGerman health care system was done by representatives frominterventions that could be performed as outpatient pro-the Societies of Internal Medicine, Surgery and Geriatrics,cedures had substantially higher proportions of inappropriateand from the regional review organizations, which are formallycare.in charge of standardized utilization reviews on behalf of the

In our approach of evaluating all hospital days fromGerman sickness funds. Although several items in the originaleach sampled patient, it is necessary to adjust for potentialAEP were omitted, in the German version the diagnosis-correlation among day-specific assessments performed inindependence and the explicitness of the definitions wasthe same patient using standard statistical techniques. Thispreserved. Adaptation of the original instrument for otherrequires a sample size that is slightly greater than would becountries (i.e. Italy [6], Portugal [37], the French-speakingnecessary if each day-specific assessment were independentpart of Switzerland [38], and Turkey [39]) has been reported.of all other days of that patient. Other studies randomlyThe German AEP retains good reliability comparable toselected patient-days out of the pool of all hospital days ofthe findings of the original AEP [4,8,24]. In contrast to thethe entire patient population. The latter approach is stat-methods used in previous studies, we calculated reliabilityistically more efficient, but many more patient records mustnot only for hospital admissions and days of hospital stay inbe identified. Our sampling is the most process-efficienttotal, but also for each consecutive day of a hospital course.approach, independent of the type of medical record (paperOverall agreement in judging the appropriateness of ad-or electronic).missions and hospital days was high, particularly in surgical

In conclusion, our study documented a substantial pro-patients, with percentage agreement rates around or exceedingportion of inappropriate hospital use under the current system80%. Agreement was consistently good for appropriate andof per diem reimbursement for hospital services. Our sam-inappropriate days, underscoring the utility of the AEP forpling approach may be the most process-efficient way tostandardized utilization review.assess this proportion. With the forthcoming implementationThe proportion of inappropriate hospital days was high,of a prospective payment system in Germany based onboth in general internal medicine and in surgery. These resultsdiagnosis-related groups (DRG), the AEP will be used in-are similar to those of other studies [2,4,8,40] and notcreasingly by supervisory agencies to assess the ap-surprising for the German health care system, where hospitalpropriateness of admissions in combination with audits ofcare is basically free of charge and hospital reimbursementDRG coding. For the purpose of clinical quality management,is predominantly per diem. German doctors are not requireda slightly modified AEP should be used to evaluate in-to justify the length of an individual hospital stay, and henceappropriately early discharges after the change in paymentthere is an incentive to prolong hospitalization when theresystem.are empty beds. Not surprisingly, occupancy rates in Germany

exceed 79.8% where there are 6.7 acute care hospital bedsper 1000 inhabitants (1997) [41]. Corresponding figures forthe US are 63.0% and 3.3 per 1000, respectively. In Germany, Referencessubstantial efforts are currently under way to optimize thedelivery of hospital care, including the introduction of pro- 1. Brennan TA, Leape LL, Laird NM et al. Incidence of adverse

events and negligence in hospitalized patients. Results of thespective payment in 2003.Harvard Medical Practice Study I. N Engl J Med 1991; 324:On the other hand, our rates of inappropriate hospital370376.care are similar to other countries that have used the AEP

to evaluate the appropriateness of hospital care. Siu found2. Gertman PM, Restuccia JD. The appropriateness evaluation

23% of all adult admissions in six US sites to be unnecessary protocol: a technique for assessing unnecessary days of hospital[4]. Findings from European studies range from 2.7% and care. Med Care 1981; 19: 855871.4.2% in Switzerland [38] and 14.2% in Italy [42], to 19% [9]

3. Winickoff RN, Restuccia JD, Fincke BJ. Concurrent applicationand 27.6% [43] in Spain. Corresponding figures for theof the Appropriateness Evaluation Protocol to acute admissionsproportion of inappropriate hospital days range betweenin Department of Veteran Affairs Medical Centers. Med Care11.2% and 15.3% in Switzerland [38], 35% in the US [4],1991; 29: AS64AS75.37.3% in Italy [42], and 31.7% [9] and 43.9% [43] in Spain.

A study from Israel revealed 38.9% inappropriate hospital 4. Siu AL, Sonnenberg FA, Manning WG et al. Inappropriate usedays in surgery and 54% in medicine [44]. It is difficult to of hospitals in a randomized trial of health insurance plans. Ngeneralize from these findings, because rates of inappropriate Engl J Med 1986; 315: 12591266.care are influenced by multiple factors such as differences in

5. Gloor JE, Kissoon N, Joubert GI. Appropriateness of hos-delivery and financing of health care, physicians practicepitalization in a Canadian pediatric hospital. Pediatrics 1993; 91:styles, patients behavior, and expectations or study method-7074.ology.

We explored several predictors of inappropriate hospital 6. Apolone G, Alfieri V, Braga A et al. A survey of the necessitydays. However, as reported earlier [6,45], only hospital length of the hospitalization day in an Italian teaching hospital. Qual

Assur Health Care 1991; 3: 19.of stay was clearly associated with inappropriate hospital use.

491

O. Sangha et al.

7. Bare ML, Prat A, Lledo L, Asenjo MA, Salleras L. Ap- 25. Bavarian Public Health Research Center, University of Munich.AEPAppropriateness Evaluation Protocol (German version):propriateness of admissions and hospitalization days in an acute-

care teaching hospital (in French). Rev dEpidem de Sante Publ http://mfv.web.med.uni-muenchen.de/aep.html Accessed 7December 2002.1995; 43: 328336.

26. Cohen J. A coefficient of agreement for nominal scales. Educ8. Rishpon S, Lubacsh S, Epstein LM. Reliability of a method ofPschol Measure 1960; 20: 3746.determining the necessity for hospitalization days in Israel. Med

Care 1986; 24: 279282. 27. Feinstein AR, Cichetti DV. High agreement but low kappa: 1.The problem of two paradoxes. J Clin Epidemiol 1990; 43:9. Alonso J, Munoz A, Anto JM. Using length of stay and inactive543549.days in the hospital to assess appropriateness of utilisation in

28. Fleiss JL. Statistical Methods for Rates and Proportions. New York:Barcelona, Spain. J Epidemiol Commun Health 1996; 50: 196201.Wiley, 1981.

10. Henley L, Smit M, Roux P, Zwarenstein M. Bed use in the29. Cochran WG. The comparison of percentages in matchedmedical wards of Red Cross War Memorial Childrens Hospital,

samples. Biometrica 1950: 256266.Cape Town. S Afr Med J 1991; 80: 487490.30. Landis JR, Koch GG. The measurement of observer agreement

11. Hynes M, OHerlihy BP, Laffoy M, Hayes C. Patients 21 days for categorical data. Biometrics 1977; 33: 159174.or more in an acute hospital bed: appropriateness of care. Irish

31. Armitage P, Berry G. Statistical Methods in Medical Research. London:J Med Sci 1991; 160: 389392.Blackwell, 1994.

12. Payne SM, Ash A, Restuccia JD. The role of feedback in32. Hosmer DW, Lemeshow S. Applied Logistic Regression. New York:reducing medically unnecessary hospital use. Med Care 1991; 28:

Wiley, 1989.AS91AS106.33. Liang KY, Zeger SL. Analysis of Longitudinal Data. Oxford:

13. Chopard P, Perneger TV, Gaspoz J-M et al. Predictors of Clarendon Press, 1994.inappropriate hospital days in a department of internal medicine.

34. SAS Institute Inc. SAS/STAT Users Guide. Release 6.10 edition.Int J Epidemiol 1998; 27: 513519.Cary, NC: SAS Institute Inc., 1996.

14. Restuccia JD. The effect of concurrent feedback in reducing 35. Siu AL, Manning WG, Benjamin B. Patient, provider andinappropriate hospital utilization. Med Care 1982; 20: 4662. hospital characteristics associated with inappropriate hos-

pitalization. Am J Publ Health 1990; 80: 12531256.15. Vardi A, Modan B, Blumstein Z, Lusky A, Schiff E, BarzilayZ. A controlled intervention in reduction of redundant hospital 36. Restuccia JD, Gertman PM, Dayno SJ. A comparative analysisdays. Int J Epidemiol 1996; 25: 604608. of appropriateness of hospital use. Health Aff 1984; 3: 130138.

37. Bentes M, Gonsalves ML, Santos M, Pina E. Design and16. Restuccia JD, Kreger BE, Payne SM, Gertman PM, Dayno SJ,development of a utilization review program in Portugal. Int JLenhart GM. Factors affecting appropriateness of hospital useQual Health Care 1995; 7: 201212.in Massachusetts. Health Care Fin Rev 1986; 8: 4754.

38. Santos-Eggimann B, Paccaud F, Blanc T. Medical ap-17. Payne SM, Campbell D, Penzias BG, Socholitzky E. Newpropriateness of hospital utilization: an overview of the Swissmethods for evaluating utilization management programs. QRB.experience. Int J Qual Health Care 1995; 7: 227232.Qual Rev Bull 1992; 18: 340347.

39. Kaya S, Erdem Y, Dogrusoz S, Halici N. Reliability of a hospital18. Booth BM, Ludke RL, Wakefield DS et al. Nonacute days ofutilization review method in Turkey. Int J Qual Health Care 1998;care within Department of Veterans Affairs medical centers.10: 5358.Med Care 1991; 29: AS51AS56.

40. Mozes B, Halkin H, Katz A, Schiff E. Reduction of redundant19. Payne SM. Identifying and managing inappropriate hospital hospital stay through controlled intervention. Lancet 1987; 25:

utilization: a policy synthesis. Health Serv Res 1987; 22: 709769. 968969.

20. Siu AL, Leibowitz A, Brook RH, Goldman NS, Lurie N, 41. OECD. OECD Health Data 1999a Comparative Analysis of 29Newhouse JP. Use of the hospital in a randomized trial of Countries. Paris: Credes, 1999. Http://www.oecd.org. Accessedprepaid care. J Am Med Assoc 1988; 259: 13431346. 7 December 2002.

42. Angelillo IF, Ricciardi G, Nante N, Boccia A, Group AC.21. Davido A, Nicoulet I, Levy A, Lang T. Appropriateness ofAppropriateness of hospital utilization in Italy. Publ Health 2000;admission in an emergency department: reliability of assessment114: 914.and causes of failure. Qual Assur Health Care 1991; 3: 227234.

43. De la Fuente OD, Peir S, Marchan C. Inappropriate hos-22. Kemper KJ, Fink HD, McCarthy PL. The reliability and validitypitalization. Eur J Publ Health 1996; 6: 126132.of the pediatric appropriateness evaluation protocol. QRB. Qual

44. Mozes B, Schiff E, Modan B. Factors affecting inappropriateRev Bull 1989; 15: 7780.hospital stay. Qual Assur Health Care 1991; 3: 211217.

23. Strumwasser I, Paranjpe NV, Ronis DL, Share D, Sell LJ.45. Paldi V, Porath A, Friedman L, Mozes B. Factors associatedReliability and validity of utilization review criteria. Ap-

with inappropriate hospitalization in medical wards: a cross-propriateness Evaluation Protocol, Standardized Medreview In-sectional study in two university hospitals. Int J Qual Healthstrument, and IntensitySeverityDischarge criteria. Med CareCare 1995; 7: 261265.1990; 28: 95111.

24. Kemper KJ. Medically inappropriate hospital use in a pediatricpopulation. N Engl J Med 1988; 318: 10331037. Accepted for publication 29 July 2002

492