diagnostic bias and toxic shock syndrome

CLINICAL STUDIES

Diagnostic Bias and Toxic Shock Syndrome

MARY HARVEY, M.D.

RALPH I. HORWITZ, M.D.

ALVAN R. FEINSTEIN, M.D.

New Haven. Connecticut

From the Clinical Epidemiology Unit, and the De- partments of Medicine and Epidemiology, Yale University School of Medicine. New Haven, Con- necticut. This research was funded by an uncon- ditional gant from lnternatlonal Ptaytex, Inc. Dr. Horwitz is a Henry J. Kaiser Famify Foundation Facutty Scholar in General Internal Medicine. Requests for reprints should be addressed to Dr. Alvan Ft. Feinstein. Yale University School of Medicine, 333 Cedar Street, P.O. Box 3333, New Haven, Connecticut 06510. Manuscript accepted August 31. 1983.

Diagnostic bias could have occurred in the tampon/toxic shock syndrome association if the original diagnostic judgments by physicians or the subsequent diagnostk seiectkxts by investigators were influenced by knowledge of the patient’s gender, menstrual history, or catameniai product. To determine whether such diagnostic bias can occur, a set of descriptive vignettes was prepared for six cases (Series X) having diverse resemblances to toxic shock syndrome or Kawasaki’s disease. For each vignette in Series X, a paired case in a complementary Series Y was described in identical text except for different information about gender, menstrual history, or tampon use. Either Series X or Series Y was sent to a randomly selected sample of internists, family practitioners, and infectious disease subspecialists. The physicians were asked to provide a diagnosis for each case, using their own nomenclature or choosing from a list of 22 diagnostic possibilities. Six weeks later, respondents were asked to provide diagnoses for the complementary series of cases. Regardless of medkai specialty or type of series received first, the 368 responding physicians were more iikeiy to diagnose toxic shock for menstruating tampon-users or menstruating women than for nonmenstruating women or men. In the most striking difference, toxic shock was diagnosed in 85 percent of instances when the case was described in a menstruating tampon-user, but in 23 percent of instances when the same case was described in a man (odds ratio = 18.8). Since the current epidemioiogic data may be seriously distorted by diagnostic bias, the results indicate the need for scientifically valid investigations of the toxic shock syndrome/tampon relationship.

In the controversy about the role of tampons in toxic shock syndrome, we have suggested [ 11 that the epidemiologic results may have become biased during the decisions made when medical practitioners diagnosed toxic shock syndrome, and when investigators chose to retain or reject a reported case as fulfilling diagnostic criteria for toxic shock syndrome. The diagnosis of toxic shock syndrome depends entirely on a physician’s clinical j@ment, since toxic shock syndrome has no distinctive diagnostic test and may clinically resemble other febrile mucocutaneous illnesses, such as scarlet fever, Kawasaki’s disease, Rocky Mountain spotted fever, staphylococcal and streptococcal illnesses, viral gastroenteritis, and other viral and bacterial diseases.

A physician’s decision to diagnose toxic shock syndrome should presumably depend on only the stipulated diagnostic criteria [ 21, which say nothing about menstruation or usage of tampons. The criteria

March 1984 The American Journal of Medicine Volume 76 351

DIAGNOSTIC BIAS AND TOXIC SHOCK SYNDROME-HARVEY ET AL

require a patient to have (1) fever of 102’F or more; (2) systolic blood pressure of 90 mg Hg or less, or an or- thostatic drop of 15 mm Hg or more: (3) an erythematous maculopapular rash; (4) desquamation of the skin seven to 10 days after the illness began; and (5) clinical or laboratory evidence of involvement of three or more organ systems. Diagnostic bias would occur if a physician’s decision to diagnose toxic shock syndrome were influenced by a patient’s gender, menstrual history, or use of a particular catamenial product, rather than by the diagnostic criteria alone. If this bias occurs, the same clinical manifestations diagnosed as toxic shock syndrome in menstruating tampon-users might be diagnosed otherwise in children, men, or nonmenstruating women.

The occurrence of diagnostic bias would have important implications for the epidemiologic data about tampons and toxic shock. If medical practitioners diagnose toxic shock syndrome predominantly in menstruating tampon-users, the epidemiologic pool of diagnosed and reported cases would be dispropor- tionately enriched with such patients. If the epidemiologic investigators reviewing the available reports were influenced by knowledge of menses or tampon usage, menstruating tampon-users would be most likely to be retained as cases for research. As a result, the epidemiologic data would be biased in the direction of finding an association between tampons and toxic shock syndrome.

Since the possibility of these two forms of diagnostic bias was not evaluated in the epidemiologic investigations of toxic shock syndrome, we undertook the current research to determine whether such bias can occur.

METHODS

Using published accounts and medical records of patients with toxic shock syndrome and Kawasaki’s disease, we developed a set of clinical “vignettes” that included the results of history, physical examination, and laboratory tests for six cases. One of these was a description of Kawasaki’s disease. The other five cases were patterned after descriptions of toxic shock syndrome, with one case fulfilling the criteria for “definite” diagnosis as described by Shands et al [2], and the other four cases being “probable” in that they lacked the requirement for desquamation. The original cases were numbered 1 through 6.

For each of these six cases, a paired vignette was devei- oped and randomly numbered 7 through 12. The paired case was identical to the original except for different ancillary data regarding gender, menstrual history, and tampon use. In each pair, the ancillary data were chosen to make the diagnosis of toxic shock syndrome more “favorable” in one case than in the other. in addition, the “favorable” ancillary data were arranged to create a gradient of “bias” in the compared pairs of cases. The most striking bias would be expected in the contrast between a menstruating tampon-user and a man. Next would be the comparisons of a menstruating tampon- user versus a nonmenstruating woman, followed by a men-

struating woman versus a man. The smallest bias would be expected in comparing a nonmenstruating woman and a man. The accounts of the vignettes for these six pairs of cases, identified as Pairs I to Vi, are shown in the Appendix. The distinctions of the individual cases are briefly summarized in Table I.

Our plan was to ask physicians to diagnose Cases 1 to 6, and six weeks later, to diagnose Cases 7 to 12. Since we wanted to have 200 to 250 respondents for both sets of cases available for analysis and since we anticipated a response rate of 30 to 35 percent, we selected an initial sample of 800 physicians, consisting of all 179 physicians listed as infectious disease specialists and 621 internists randomly selected from the 10,719 listed in the 20th edition (1981-82) of the Directory of Medical Specialists as practicing in Pennsyl- vania, New York, and Connecticut.

Because the response rate to our first set of mailings was 10 percent after four weeks, we sent out a second mailing to an additional 1,200 physicians, consisting of 800 more internists chosen at random, and 400 family practitioners also chosen randomly from the 2,507 listed in the same directory. For the mailing to this second group, we reversed the order of cases, sending Cases 7 to 12 initially, and then sending Cases 1 to 6 six weeks later to those physicians who re- sponded. To identify these distinctions in the subsequent analysis, we shall use Sequence X to refer to the pattern in which Cases 1 to 6 were sent first, with Cases 7 to 12 sent six weeks later. In Sequence Y, Cases 7 to 12 were sent first, followed by Cases 1 to 6. All of the infectious disease specialists and the first 621 internists who were solicited received Sequence X. Ail of the family practitioners and the additional 800 internists received Sequence Y. Physician enrollment and data collection was arbitrarily closed 12 weeks after the last mailing.

A copy of the first letter sent to all the physicians is shown in the Appendix. In addition to the six clinical vignettes, the letter contained a list of 22 possible diagnoses. The list of possible diagnoses and the associated text, which encour- aged physicians to use their own diagnostic terms, are shown in the Appendix, as well as the text of the letter enclosed when each responding physician was sent the second set of cases.

Although 2,000 case sets were mailed, 587 (29 percent) were returned as having incorrect addresses. Of the 1,413 case sets that were presumably delivered, 368 (26 percent) physicians completed at least one set of case histories, and 2 18 (15 percent) completed both sets.

For each case, the diagnosis submitted by each physician was designated as unequivocal toxic shock syndrome, equivocal toxic shock syndrome (for statements such as “probable toxic shock syndrome” or “rule out toxic shock syndrome”), or other (for a diagnosis other than toxic shock syndrome). These designations were made, using strict criteria, by a data coder who was kept unaware of the content or number of the case to which the diagnosis pertained. After these designations were completed, the data were prepared for computer processing. in the subsequent analyses, the unequivocal and equivocal diagnoses of toxic shock were first combined as toxic shock syndrome and compared against the other group. In a second analytic arrangement, only the unequivocal toxic shock diagnoses were compared

352 March 1964 The American Journal of Medicine Volume 76

DIAGNOSTIC BIAS AND TOXIC SHOCK SYNDROME--HARVEY ET AL

TABLE I Summary of Salient Features of Six Pairs of Clinical Vignettes

Case Ancillary Features -- Basic Clinical Data Number Aae Sex Menses Tamoons

Pair I: Definite TSS

Pair II: Probable TSS

Pair III: Probable TSS

Pair IV: Probable TSS

Pair V: Probable TSS

Pair VI: Kawasaki’s Disease

Fever (39’C); hypotension (70/50); erythematous macular rash: desquamation on 12th day; myalgias (elevated CPK); nausea and vomiting; elevated SGOT and bilirubin; prolonged prothrombin time

Fever (40.5%); hypotension (90760); skin bright pink; conjunctival injection; erythematous pharynx; 90,000 platelets; hypocalcemia; elevated CPK

Fever (38.5%); hypotension (70150); erythematous, maculopapular rash; lethargy; disorientation; elevated SGOT and bilirubin; abnormal urinalysis results (2-I protein and 30 to 40 white blood cells per high-power field; prolonged prothrombin time; 80,000 platelets

Fever (4O’C); hypotension (70/50); fine red rash on thighs and arms; lethargy; disorientation; watery diarrhea; elevated SGOT and bilirubin; prolonged prothrombin time; alveolar infiltrates on chest roentgenography

Fever (385°C); hypotension (80/60); “diffuse red rash”; nausea and diarrhea; myalgias; erythematous pharynx; elevated serum creatinine; abnormal electrocardiographic results (nonspecific S-T segment and T wave changes)

Fever (40.5”C); erythematous maculopapular rash with reddening of palms and soles; desquamation of palms and soles; cervical adenopathy; elevated SGOT and bilirubin; nonspecific S-T segment and T wave changes on electrocardiogram

1 27 10 27

4 18 9 18

6 38 7 38

3 23 11 23

5 42 12 42

2 17 8 17

F M

F M

F F

M F

M F

M F

Yes Yes

Yes Yes -

No -

Yes Yes

- - Yes No

- - No -

_- Yes

- Yes

against the other group; 16 equivocal toxic shock syndrome diagnoses were excluded. Since the results were similar for both arrangements, only the arrangement using unequivocal diagnoses of toxic shock syndrome versus unequivocal diagnoses of other is presented here.

The data were analyzed with three statistical procedures. In the first two steps, a “parallel analysis” was used to contrast results in Cases 1 to 6 with those in Cases 7 to 12. This analysis was done twice: first, for all respondents, regardless of whether the two sets of six cases had been completed by the same physician; second, for only those physicians who had completed both sets of cases. In these analyses, we calculated the proportion of physicians diagnosing toxic shock syndrome for each member of a pair of cases. In addition, since an odds ratio was used to express the results of all of the earlier case-control studies of toxic shock syndrome, we also calculated an odds ratio to measure the association between a toxic shock diagnosis and the favorable ancillary features of a case.

The odds ratio was calculated as follows:

Reported Diagnosis \ /

TSS Other

Favorable

Characteristicj ;I;_: wi Ancillary

Odds Ratio = s bc

Lastly, a “cross-over” analysis was used for cases in the two series that had each been diagnosed by the same physician. In this analysis, results were expressed in an agreement matrix or “McNemar table”:

Reported Diagnosis When Potentially Favorable

Characteristics Were Absent

Reported Diagnosis When Potentially

teristics Were re:tble Charac- ) K

Answers in the cells labeled a and d represent concordant pairs. For both the case vignettes, with and without the favorable ancillary features, physicians diagnosed toxic shock syndrome in cell a, and other than toxic shock syndrome for cell d. Answers in the cells labelled b and care discordant responses. In b, physicians diagnosed toxic shock syndrome for the case with favorable ancillary features and other than toxic shock syndrome for the case without favorable ancillary features. In c, the situation was reversed. From these data, the McNemar bias ratio, calculated as (b - c)/(b -I- c), indi- cates, among the total number of discordant pairs, the proportion of discordant pairs due to favorable ancillary char- acteristlcs. The value of this bias ratio can range between 0 and 1, with 0 indicating no bias (i.e., equal numbers of discordant pairs in both favorable and unfavorable directions)



TABLE II Proportion of All Respondents Dlagnoslng TOXIC Shock Syndrome by Medlcal Specialty and Type of Sequence (n = 366)

Ancillary Features

of the Paired Cases

Favor- able

Unfavor- able

Infectious Disease Specialists

(Sequence X*) Favor- Unlavor- able able

Internists (Sequence X’ )

Favor- Unlavor- able able

Internists (Sequence Y’)


Family Practitioners

(Sequence Y’) Favor- Unfavor- able able



Pair Ill: Probable TSS




Female menstruating tampon-user



Female menstruating

Female


Male 61166 (92%)

Male 38166 (58%)

Female 15145 3166 (33%) (5%)

Male

Male

Male

20145 (44%) 1 l/45 (24%) 12145 (27%)

9/66 (14%)

6166 (9%) l/66 (2%)

15/45 (33%)

4145 (9%)

1341160 (84%)

781160 (49%)

38199 (38%)

34f 99 (34%)

14199 (14%)

15199 (15%)

18199 (18%)

5/99 (5%)

41160 (3%)

101160 (6%)

81160 (5%)

21160 (1%)

27132 (84%)

21132 (66%)

59188 (67%)

40188 (45%) 16188 (18%) 22188 (25%)

19188 (22%)

10188 (11%)

6132 (19%)

5132 (16%)

6132 (19%)

4132 (13%)

3514: 13154 (83%) (24%)

18142 5154 (43%) (9%)

29154 8142 (54%) (19%)

21154 9142 (39%) (21%) 10154 6J42 (19%) (14%) 1 II54 1142

(20%) (2%)

l Sequence X = Cases 1 to 6 first; Sequence Y = Cases 7 to 12 first.

and 1 indicating maximal bias (i.e., all discordant pairs due to favorable ancillary characteristics).

Statistical significance was tested using the customary chi-square test for unpaired data and the McNernar chisquare test [3] for pairwise comparisons.

RESULTS

Table II, which includes all responses from all 386 physicians who participated, shows the proportion who diagnosed toxic shock syndrome for each case, according to medical specialty and sequential pattern. The pairs of cases, listed in the first column, are followed in the second main column by the ancillary featwes that distinguish cases favorable and unfavorable to the diagnosis of toxic shock syndrome. In the remaining columns, the number and proportion of physicians diagnosing toxic shock syndrome are shown for the cases with favorable and unfavorable characteristics.

Regardless of medical specialty or sequential pattern, the physicians were more likely to diagnose toxic shock syndrome in menstruating tampon-users or in menstruating women than in nonmenstruating women or men. Whenever a menstruating tampon-user or menstruating woman was compared with a nonmenstruating woman or a man, the higher proportion of toxic shock diagnoses was statistically significant. For Pair V (nonmenstruating woman versus a man), internists (Sequence Y) and family practitioners diagnosed toxic shock syndrome about as often for the woman as for the man, whereas internists (Sequence X) and infectious

disease subspecialists diagnosed toxic shock syndrome more often in the woman.

Table Ill, organized in a manner identical to Table II, shows the proportion of toxic shock diagnoses made by the 218 physicians who gave definite answers for both sets of cases. Table Ill shows the same pattern of results as Table II. For every contrast of a menstruating tampon-user or menstruating woman versus a nonmenstruating woman or a man, there is a statistically significant difference in the proportion of physicians diagnosing toxic shock syndrome for the “favorable” case than for the “unfavorable” case. For the contrast of a nonmenstruating woman versus a man, the proportions diagnosing toxic shock syndrome were not significantly different among internists (Sequence Y) and family practitioners.

Since the results were not affected by either medical specialty cr sequential pattern, the data were combined for all respondents who completed both sets of cases. Table IV shows the proportion of toxic shock diagnoses for each case in Pairs I to VI and an associated odds ratio for the diagnosis of toxic shock syndrome in each contrasted pair. This table shows not onfy that very large odds ratios are obtained, particularly for the contrasts of a menstruating tampon-user versus a man, but also that there is a distinct gradient in the magnitude of the odds ratios according to the type of ancillary features contrasted. The “diagnostic-bias” odds ratio is 18.8 for a menstruating tampon-user versus a man, 8.7 for a menstruating tampon-user versus a nonmenstruating woman, 6.5 for a menstruating woman versus a man, and 2.1 for a nonmenstruating woman versus a man.


DIAGNOSTIC BIAS AND TOXIC SHOCK SYNDROME--HARVEY ET AL

TABLE III Proportion Diagnosing Toxic Shock Syndrome by Medical Specialty and Type of Sequence for Physicians Returning Both Sets of Cases

Ancillary Features

of the Paired Cases

Favor- able

Unfavor- able

lnlectious Disease Specialists

(Sequence X*) 45 Respondents

Favor- Unfavor- able able

Internists (Sequence Xi) 99 Respondents


Internists (Sequence Y “) 32 Respondents ___--___


Family Practitioners

(Sequence Y ‘) 42 Respondents


Pair I: Female Male 34 Definite TSS menstruating (9G, (81%) (21;)

tampon-user Pair II: Female Male

Probable TSS menstruating (6& tampon-user

Pair Ill: Female Female 15 38 Probable TSS menstruating (33%) (,‘,I (38%) (2;) (8:;)

tampon-user Pair IV: Female Male

(4::) (11:) (3& (6;) 14

Probable TSS menstruating (44%) Pair V: Female Male 11 3 14 5 6

4 17 13%) (41%) (10;)

6 23 19%) (55%) (2:;)

5 15 16%) (36%) (7;)

5 9 8 Probable TSS (24%) (7%) (14%) (5%) (19%) (16%) (21%) (19%)

Pair VI: Female Male (2:;) (2;)

0 (25;)

4 10 Kawasaki’s menstruating (0%) (13%) (24%) (2& Disease tampon-user

Sixteen cases were deleted because of equivocal answers. l Sequence X = Cases 1 to 6 first; Sequence Y = Cases 7 to 12 first.

TABLE IV Proportion Diagnosing Toxic Shock Syndrome and Diagnostic Odds Ratios for All Physicians Completing Both Sets of Cases

Ancillary Characteristics Number and Percent of Diagnostic of Paired Cases 218 Respondents Diagnosing TSS Odds

Favorable Unfavorable Favorable Unfavorable Ratio



Pair III: Probable TSS







Female menstruating

Female


Male 185 (85%)

Male 112 (51%)

Female 102 (47%)

Male

Male

Male

12.5

(& 8.7

(A\, 6.4

(l& 2.1

(3\, 9.2

Sixteen cases were deleted because of equivocal answers.

The individual contributions of tampon use and menstruation in diagnostic bias can be assessed by comparing the results for various pairs of the “probable” cases with and without these ancillary features. For example, in Pair II, a menstruating tampon-user is compared with a man, and in Pair IV, a menstruating woman (no tampon use) is compared with a man. The odds ratio drops from 12.5 to 6.5 when tampon use is not mentioned. In Pairs IV and V, which differ only with

respect to information on menstruation, the odds ratio decreased from 6.5 to 2.1 when the menstrual data are omitted.

Table V, which displays some of the precise details that have been summarized in the preceding tables, shows a series of “McNemar matrixes” for each pair of cases by medical specialty and sequential pattern for the 218 physicians who completed both sets of cases. Cases with “favorable” ancillary data are rep-



TABLE V McNemar Bias Ratios for Each Pair of Cases by Medical Specialty and Type of Sequence

Infectious Disease Family Specialists Internists Internists Practitioners McNemar

(Sequence X’) Diagnosis TSS

Other /(~~Js T(88qnsncegtYh’d, B. ;;sRespO”dss# (a,;~~;n,s)

Pair I 1 TSSt 15 26 17 66 6 19 7 26 47 137 Othert 0 4 1 15 0 5 1 8 1 32

0.99

Pair II ( TSSt 3 24 2 45 4 17 4 13 13 99 Othert 1 17 3 49 0 11 0 25 4 102

0.92

Pair Ill 1 TSSt 0 15 1 37 4 22 8 15 13 89 Other7 2 28 1 60 2 4 2 17 7 109

0.85

Pair IV ( TSSt 3 17 5 29 4 10 1 14 13 70 Other7 2 23 1 64 1 17 2 25 6 130

0.84

Pair V 1 TSSt 0 11 4 10 3 2 5 4 12 27 Othert 3 31 1 84 3 24 3 30 10 169

0.46

Pair VI TSSt 0 12 0 15 2 6 1 9 3 42 Othert 0 33 1 83 2 22 0 32 3 170

0.87

Sixteen cases were deleted because of equivocal answers. l Sequence X = Cases 1 to 6 first; Sequence Y = Cases 7 to 12 first. t Diagnosis made for “favorable” case of pair. $ Diagnosis made for “unfavorable” case of pair.

resented in the vertical columns; cases with “unfavorable” ancillary data are represented in the horizontal rows. The table is read in the following way: the upper left hand corner shows the responses of infectious disease subspecialists for Pair I. Fifteen respondents diagnosed toxic shock syndrome for both the “favorable” and “unfavorable” case; 26 diagnosed toxic shock syndrome when the ancillary features were “favorable’ but not when the ancillary features were “unfavorable”; none showed the reverse pattern; four made a diagnosis other than toxic shock syndrome regardless of whether the ancillary features were “favorable” or “unfavorable.” The next to last column of the table shows the total of the McNemar results for all respondents, and the last column shows the McNemar bias ratios calculated from the totals. The gradient of bias previously shown in Table IV, using odds ratios, is again shown here for the bias ratios, which are very large, indicating that almost all of the discordant pairs are attributable to the effects of the favorable ancillary characteristics, particularly for comparisons of a menstruating tampon-user or a menstruating woman with a nonmenstruating woman or a man.

COMMENTS

This study was done to test the suspicion that the diagnosis of toxic shock syndrome can be biased by a knowledge of the patient’s catamenial status. The results not only confirm that suspicion, but also show that the magnitude of the bias directly depends on the specific ancillary features in each candidate patient. The bias is most pronounced when the candidate is a menstruating tampon-user, but is also strongly present when the candidate is a menstruating woman for whom

no information about tampon use is given. Furthermore, despite the claim [4] that bias alone cannot produce odds ratios of the size noted in previous case-control studies of toxic shock syndrome, some of the odds ratios found in the current investigation are as large as those reported in the previous studies.

Since the current research was conducted after publication of the epidemiologic investigations of the relation between tampons and toxic shock syndrome, the current results merely demonstrate that diagnostic bias can occur. They do not prove that the bias occurred in the previous studies. An argument can be offered, in fact, that the medical publicity resulting from the previous studies was in part responsible for the biases noted in this study. On the other hand, since “double- blinding” has been used for many years as a precaution against bias when cause-effect studies are conducted to test the action of therapeutic agents, and since no such precautions were used in the epidemiologic studies, abundant opportunities existed for bias to enter the epidemiologic research at two different stages of diagnostic decisions: when physicians originally diagnosed toxic shock syndrome or when investigators reviewed reported cases to confirm the diagnosis.

In making diagnoses for patients seen in clinical practice, the physicians in the states from which toxic shock syndrome was predominantly reported could have been affected by announcements from state health agencies stating that a relation was suspected, first, between menses and toxic shock syndrome, and later (after June 27, 1980), between tampons and toxic shock. The publicity about menses and toxic shock syndrome, which was transmitted first by state health agencies and later by mass media, had already become


extensive by the time of the Wisconsin, Utah, and first CDC studies [2,5,6]. The diagnoses made in patients described for all the subsequent studies by the CDC and by other health agencies [7-9] would have been affected by the subsequent publicity about tampons and toxic shock syndrome.

A particularly important second source of diagnostic bias could have arisen when the investigators, after reviewing the reports that had been submitted to the health agency by physicians or by patients, decided which were “confirmed” cases of toxic shock syndrome. Since the hypothesis that tampons are associated with toxic shock syndrome had already been well developed before any of these reviews began, and since the investigators knew the tampon-use habits in reported cases, a diagnostic selection bias could have affected the toxic shock cases chosen for inclusion in the very first groups of published studies (CDC, Wis- consin, and Utah). At least two items of evidence suggest that the tampon hypothesis had been developed before the reported information was reviewed. First, the etiologic attention of the questionnaire [IO] used to solicit data in the first CDC study was concentrated almost exclusively on gynecologic aspects of catamenia. One brand of tampons (Rely) was specifically (and exclusively) cited by name, with patients reminded that they might have received it at home as a sample; no questions were asked about nutritional status, cigarette smoking, antecedent episodes of illness, concurrent episodes of illness, previous medications, sources of infection, or other “risk factors” that might have been considered if a specific etiologic agent were not strongly suspected.

Second, as noted in our previous critique [ 11, tampon use was found to be associated with 98 to 100 percent of the cases selected in all the epidemiologic case- control studies. This high prevalence of a suspected etiologic agent is extraordinary for the cases of an epidemiologic case-control study. The proportion is sub- stantially higher than what has been found for the prevalence of agents generally regarded as having an etiologic role in other diseases. Thus, the 98 to 100 percent prevalence of tampon use in the selected cases of toxic shock syndrome is higher than the prevalence found for cigarette smoking in cases of lung cancer, or estrogens in endometrial cancer, and for the foods in- criminated in classic epidemiologic examples of “common-source outbreaks” of gastroenteritis. Since toxic shock syndrome can occur in women who did not use tampons, a tampon-use prevalence of almost 100 percent suggests that a knowledge of tampon usage may have affected the diagnostic decisions when cases were selected for the epidemiologic research.

Because the hypothesis about a tampon “cause” had already been developed, the best scientific way to avoid


the potential problem of bias was to “blind” the investigators to information about catamenial status when they reviewed reported cases to be retained or rejected. Unfortunately, this scientific precaution was not ob- served in any of the epidemiologic studies. The investigators have not published the characteristics of the rejected case reports: however, in the few instances [2,9] in which the total numbers of examined reports have been cited, the excluded case reports comprised more than half of the total material under consideration. The potential for bias when the reports were selected is high because the diagnostic criteria for toxic shock syndrome require many subjective decisions about the clinical constituents of the criteria. For example, how does one decide whether a rash described on a case report form as “red blotches” fulfills the diagnostic criteria requirement for an erythematous, maculopapular lesion? In the absence of a “blind” evaluation, the judgments used to interpret the available clinical data could readily be affected by a knowledge of catamenial status.

Of the possible biases that might affect the statistical association between tampons and toxic shock syndrome, the current research was addressed only to the possibility that catamenial information might influence either the diagnostic decisions made by the original physicians or the diagnostic selections performed in a “non-blinded” review of case reports. We could not address the biases that might occur when interviews about previous catamenial habits were conducted by investigators who knew about the tampon hypothesis and who knew the identity of each interviewed person as a case subject or control subject. We also could not address the issue of reporting bias, which was recently carefully evaluated and demonstrated to be present in a study of a different topic investigated by a different CDC group [ll].

One possible reservation about our results is that the response rate to the questionnaires was lower than we hoped, probably because many physicians were re- luctant to spend the considerable time required to evaluate and “diagnose” each set of six cases. This low response rate might be distressing if only one set of challenges had been given to the solicited physicians. In essence, however, our study resembled a clinical trial in which two different challenges are given to the participants. As in most clinical trials, the results of those challenges can be suitably compared and evaluated in the participants, regardless of how many people de- clined to enter the trial itself. The low rate of volunteers in this study may impede the generalization of results, but the data found in the comparison of responses to the paired series of cases clearly demonstrate that sub- stantial bias can occur when toxic shock syndrome is diagnosed. The demonstration that the bias had similar



magnitude among infectious disease specialists, internists, and family practitioners suggests that such bias can be widespread.

Another possible reservation about our methods is that making a diagnosis from a written case vignette is not the same as actually interviewing and examining a patient. In addition, our case vignettes, although rea- sonably detailed, did not necessarily contain all the information a physician might ideally obtain in practice before making a diagnosis. Although the diagnostic decision-making process in research situations may differ from what would occur in actual practice, the effect of the difference is difficult to judge. Since most physicians are accustomed to making diagnostic as- sessments from case vignettes presented in consul- tation or in a conference, it seems likely that the same diagnostic reasoning is used in these situations as when a patient is actually seen. Moreover, various licensure agencies and specialty boards certify physicians according to their ability to make accurate diagnostic decisions from case vignettes presented in written examinations. The procedure of reviewing written case vignettes was, in fact, the technique used when the epidemiologic investigators made their own “diagnostic” selections in the toxic shock research.

Our previous suggestion that bias may have occurred in the epidemiologic studies of toxic shock syndrome was dismissed [4] on the grounds that no evidence had been collected to substantiate any of the potential possibilities for bias when practicing physicians make diagnoses and report candidate cases to health agencies, and when investigators make selections from the candidate case reports and then conduct interviews with the chosen case and control subjects. The evidence now exists to show that a strikingly high amount of diagnostic bias can occur in the toxic shock/tampon relation. The amount is high enough, even without specific tests for the other possible biases, to raise serious doubt about the current epidemiologic evidence of a relationship between tampons and toxic shock syndrome, and to indicate that objective, scientifically valid studies are needed.

APPENDIX

Case Vignettes

Pak I. Case 1: A P7-year&f female pesented to the emergency room with complaints of fever, headache, myalgias, malaise, nausea, and vomiting of 24 hours’ duration. She had treated herself at home with bedrest, fluids, and aspirin but sought medical attention after she fainted while getting up to go to the bathroom. On examination in the emergency room, she had a blood pressure of 70/50 mm Hg supine, pulse of 120 per minute, respirations of 24 per minute, and temperature of 39%. The conjunctiva were injected, and the pharynx was red. The lungs were clear. Cardiac examination revealed a grade ii/Vi systolic ejection murmur at the lower left

sternal border. The abdomen was diffusely tender, bowel sounds were active, and there was no organomegaiy. She had diffuse muscle tenderness and a fine, erythematous macular rash on her trunk and extremities. Pelvic examination showeda tampon in place in the vagina. Laboratory results available in the emergency room included a white blood ceil count of 19,000/mm3 with 62 percent poiymorphonuciear ceils and 20 percent band forms. Urinalysis showed specific gravity of 1.032, pH of 8,30 white blood cells per high-power fiikl, and few bacteria. She was admitted to the hospital. Blood and uine culture samples were obtained, and broadspectnm antibiotic coverage was begun. On the second and third hospital days, she remained febrile and required large volumes of intravenous fluid and vasopressors to maintain a systolic blood pressure of 90 to 100 mm Hg. Results of laboratory tests done on the second hospital day included: creatine phosphokinase of 750 IU/liter (5 to 75 iU/iiter), serum glutamic oxaioacetic transaminase of 200 iU/iiter (7 to 40 IWiiter), total bilirubin of 5.3 mg/di. and prothrombin time of 14.2 seconds (control 11.4 seconds). On the fourth hospital day, she defervesced, and a fine, flaky desquamation of the skin of the trunk and extremities was noted. Over the following week, the patient continued to improve. At discharge on the 12th hospital day, she was afebrile, ail laboratory results were normal. and gross peeling of the palms of the hands and soles of the feet was noted. The patient remained well when seen for a follow-up visit two weeks after discharge.

Case 10: klenticai except that the first italic phrase was repiaced with “A 27-year-old male” and the second italic phrase was deleted.

Pa& II. Case 4: An l&yearahl female begsn vomiting repeatediy on the third day of a menstrual period during which she had been using tampons. The following day, her temperature was 40.5%. pulse was 144 per minute, respirations were 44 per minute, and blood pressure was 90160 mm Hg. She appeared acutely ill and in severe distress. Her skin was bright pink, her conjunctiva were in-

jected, and her pharynx and tongue were deeply erythematous. She had diffuse muscle tenderness. but otherwise the remainder of the physical examination was unremarkable. Her whffe blood cell ccunt was 16,000/mm3 with 48 percent segmented poiymorphonuciear cells, 15 percent band forms, 30 percent lymphocytes, and 7 percent atypical lymphocytes. Hematocrit was 36 percent. The platelet count was 9O,OOO/mn?. Electrolyte and glucose levels were normal. Calcium was 8.4 mg/di and serum phosphorous was 2.3 mg/di. Creatine phosphokinase was 375 IU/liter (normal 5 to 75 iU/liter). Electrocardiography and chest x-ray gave normal results. She was admitted to the hospital and began receiving intravenous cefoxftin and tobramycin after appropriate culture specimens were taken. After 48 hours of treatment, she defervesced and remained afebriie during the remaining seven days of her hospitalization. At the time of discharge on the 10th hospital day, ail laboratory results were normal. Blood and urine cultures showed no growth. She was foi- lowed in the outpatient clinic fcf the next six months and remained well.

Case 9: Identical except that the italic phrase was replaced with “An ibyear-oid male presented to his physician with a complaint of frequent vomiting for 24 hours.”

Palr III. Case 6: A 38-year-old white female presented to her in- ternist wfth cornpiaints of mild headache and sore throat of one day’s dwation. On examination, she appeared well but had a temperature of 38.5% and an erythematous pharynx without exudate. A throat culture specimen was taken, and penicillin was begun pending cultwe results. Two days later, she was brought to the emergency room by her husband because she had become lethargic and dis- orrented at home. On examination, she was somnolent and became


agitated when aroused. Blood pressure was 100770 mm Hg; pulse was 106 per minute; respirations were 24 per minute; and temperature was 40°C. There was an erythematous, maculopapular rash on her extremities. * There were no other abnormal physical findings. The white blood cell count was 17,000/mm3 with 50 percent segmented polymorphonuclear cells, 38 percent band forms, and 12 percent lymphocytes. Hematocrit was 39 percent. Electrolyte levels were normal, and the glucose level was 247 mg/dl. Serum glutamic oxaloacetic transaminase was 220 units/liter (normal 7 to 40 units/liter), alkaline phosphatase was 300 units/d1 (15 to 30 units/dl), and bilirubin was 3.0 mg/dl. Urinalysis showed 2-t protein, 10 to 15 red blood cells per high-power field and 30-40 white blood cells per high-power field. Chest x-ray showed nothing abnormal.

Cerebrospinal fluid obtained by lumbar puncture was clear; opening pressure was 200 mm Hg. The spinal fluid cell count was 20/mm3, mostly mononuclear cells. Because of concern about possible bacterial meningitis, masked by the penicillin therapy, the patient was treated with highdose intravenous penicillin and with chlor- amphenicol. During the first 48 hours, the patient did poorly. Her blood pressure fell to 70/50 mm Hg, and she required vasopressor’ therapy. Her prothrombin time rose to 15.7 seconds (control 10.7 seconds), and her platelet count dropped to 80,000/mm3. On the third hospital day, however, she defervesced, and her condition gradually improved over the next six days. She was completely recovered and was discharged on the 10th hospital day.

Blood, urine, and cerebrospinal fluid cultures all gave negative results.

Case 7: Identical except that the phrase “On pelvic examination, a tampon was removed from the vagina, and the vaginal mucosa was noted to be hyperemic” was inserted at the asterisk.

Pair IV. Case 3: A 2%year-old white male was admitted to the hospital with a blood pressure of 70/50 mm Hg after 36 hours of profuse, watery diarrhea. On admission, he was lethargic and dis- oriented and had a fever of 40%. On physical examination, bibasilar rales were heard. The findings on cardiac examination were normal. The abdomen was diffusely tender, bowel sounds were hyperactive, and the liver edge was palpable. On rectal examination, a small amount of liquid green stool that was guaiac-negative was found. With exception of lethargy and confusion, the results of neurologic examination were normal. There was a fine red rash on the thighs and arms, and the skin had a generalized sandpaper quality. The white cell blood count was 17,500/mm3 with 78 percent polymorphonuclear cells and 15 percent band forms. Hematocrit was 58 percent. Serum creatinine was 3.5 mg/dl and blood urea nitrogen was 87 mg/dl. Serum glutamic oxaloacetic transaminase level was 220 mu/ml (normal 7 to 40 mu/ml), total bilirubin was 2.9, and prothrombin time was 14.8 seconds (control 11.4 seconds). Elec- trocardiography showed diffuse T wave flattening. Chest x-ray showed scattered alveolar infiltrates. Blood, urine, and stool culture specimens were obtained, and the patient began receiving intravenous ampicillin, 1 g every four hours. Over the next 24 hours, he required large amounts of intravenous fluids and his respiratory status deteriorated. On the third hospital day, blood gas values were pH 7.31, PO* 60, and Pco2 48 while receiving 60 percent oxygen by ventimask, and chest x-ray showed diffuse alvelor infiltrates. Despite intubation and aggressive cardiopulmonary support, the patient had a cardiopulmonary arrest on the sixth hospital day and could not be resuscitated.

Case 11: Identical except that the italic phrase was replaced with “A 23-year-old white female was admitted to the hospital on the fourth day of her menstrual period.”

Pair V. Case 5: A 42-year-old male presented to his physician with complaints of headache, sore throat, myalgias, and nausea. He was treated with aspirin, bedrest, and fluids for these “flu-like”


symptoms. The next day, the patient returned to work and collapsed while walking across his office. He was taken to the emergency room where he was noted to be lethargic but arousable. His blood pressure was 100/60 mm Hg supine; pulse was 130 per minute; respirations were 30 per minute; and temperature was 38.5%. On examination, his lips were cracked and dry, and his pharynx was red. He had a grade III/VI systolic ejection murmur heard at the left sternal border, and there was bilateral costovertebral angle tenderness There were no other abnormal findings on physical examination The white blood cell count was 7,800/mm3 with 68 percent segmented polymorphonuclear cells, 10 percent band forms, 20 percent lymphocytes, and 2 percent basophils. Electro- cardiography showed nonspecific S-T segment and T wave changes. He was admitted for observation and more extensive evaluation. On the second hospital day, temperature was 39’C, a diffuse red rash was noted, and the patient began to have frequent watery stools. Blood pressure dropped to 80760 mm Hg, and large amounts of intravenous fluids were required. A new third heart sound was noted on cardiac examination and repeated chest x-ray showed an increase in size of cardiac silhouette. Repeated blood cell count was 21,000/mm3 with 22 percent band forms. Serum creatinine rose to 3.2 mg/dl and blood urea nitrogen to 57 mgldl. Urinalysis showed specific gravity of 1.028, 20 to 30 red blood cells per high-power field, and 10 to 15 white blood cells per high-power field. Despite aggressive fluid therapy and treatment with ampicillin and chlor- amphenicol, the patient’s clinical condition deteriorated. A cardiac arrhythmia developed, and he died on the fourth hospital day.

Case 12: Identical except that the italic phrase was replaced with “A 42-year-old female.”

Pair VI. Case 2: A 17-year-old Japanese male presented with fever ranging between 39’C and 40.5%, headache, and malaise for three days. On the day he sought medical attention, he noted swelling in the left side of his neck. A throat specimen was cultured. The next day, an erythematous maculopapular rash over his entire body and deep red discoloration of the palms and soles developed. Two days later, because of persistent fever, he was admitted to the hospital. Abnormal physical findings at the time of admission included somnolence and lethargy, conjunctival injection and diffuse injection of the oropharyngeal mucosa without ulceration or exu- dates, cracked and fissured lips, and persistent rash. There was firm, minimally tender, left anterior cervical adenopathy.’

The white blood cell count on admission was 22,000/mm3 with 78 percent segmented polymorphonuclear cells and 17 percent band forms. Hematocrit was 48 percent, and sedimentation rate was 58 mm per hour. Urinalysis showed trace protein and 20 to 30 white blood cells per high-power field. The electrolyte levels, platelet count, and prothrombin time were normal. The serum glutamic oxaloacetic transaminase level was 350 mu/ml (normal 7 to 40 mu/ml) and total bilirubin was 2 mg/dl. Electrocardiography showed diffuse nonspecific S-T segment and T wave changes. Results of the following tests were negative: two blood cultures, urine culture, rheumatoid factor, antinuclear antibody, leptospiral agglutinins, typhoid 0 and H antibody titers, paratyphoid A and B antibody titers, Proteus 0X19, 0S2, OXK antibody titers, and Brucella titers.

The patient received broad-spectrum antibiotic coverage for four days until test results were obtained. Fever persisted until the eighth hospital day, when both fever and adenopathy resolved. On the ninth hospital day, peeling of the skin of the palms and soles was noted. The patient remained afebrile and was discharged on the 11th hospital day. Laboratory and electrocardiographic results were normal at the time of discharge.

Case 8: Identical except that the italic phrase was replaced with “A 17-year-old female,” anb the phrase “Pelvic examination revealed menstrual blood and a tampon in the vagina” was inserted at the asterisk.



Letter Sent with First Set of Six Case Histories

Dear Doctor:

We are writing to you and to many other randomly selected physicians listed in the Direcfory of&&&al Specialists as practiping in New York, Pennsylvania, or Connecticut. We want to ask your help in a research study concerned with diagnostic decisions for certain acute, febrile illnesses that may have similar clinical manifestations. We are interested in the way in which a patient’s clinical and demographic features may affect the diagnostic decisions made in situations where several possible diagnoses may apply equally well. Our goal is to determine how often clinicians agree on the decisions and to identify the characteristics of patients that may influence the diagnosis.

We have assembled 12 brief case reports and would like you to indicate the diagnosis you would apply in each case. Each case should take less than five minutes to complete. To facilitate busy schedules, we have enclosed only the first six cases. We shall send you the second six cases in a few weeks. In order to correlate the first and second set of cases, each set is marked with a code number that identifies individual physicians. However, all responses will be kept confidential, and we will not reveal any individual names when the data are presented or published. Since we are interested mainly in patterns of variations among the group of clinicians, there are no “correct” or “official” answers. We shall be happy to send you a tabulation of the results, showing the most common patterns of variations.

List of Possible Diagnoses

Attached are six case reports for your review. Although we would like you to use your own diagnostic terms, you can choose from the following list of possible diagnoses if you wish:

Acute viral syndrome Influenza A or B Viral hepatitis

Infectious mononucleosis Cytomegalic inclusion virus Aseptic meningitis Rubeola Atypical measles Infectious or viral gastroenteritis Streptococcal pharyngitis Streptococcal scarlet fever Staphylococcal scarlet fever Acute rheumatic fever Mycoplasma pneumoniae Toxic shock syndrome Leptospirosis Brucellosis Toxoplasmosis Rocky Mountain spotted fever Bacteremia, source undetermined Kawasaki’s disease Stevens-Johnson syndrome

Letter Sent with Second Set of Six Case Histories

Dear Doctor: Thank you for completing and returning the six diagnostic chal-

lenges we recently sent you. We brave enclosed the last six cases for your review. Because our.atudy will be incomplete if we do not have your decisions for all 12 Cases, we are especially eager for you to return this second set of diagnostic challenges.

We realize that the case histories may not include all the information you might ideally like for diagnosis, but we would like you to make the best diagnosis possible, given the available data. Part of the purpose of our study is to determine how physicians make decisions when there is diagnostic uncertainty.

At the completion of the study, we will send you a tabulation and interpretation of the results.

REFERENCES

1. Harvey M, Horwitz RI, Feinstein AR: Toxic shock and tampons: 7. Schlech WF, Shands KN, Reingold AL, et al: Risk factors for

evaluation of the epidemiologic evkfence. JAMA 1982; 248: development of toxic shock syndrome. JAMA 1982; 248:

840-846. 835-839.

2. Shands KN, Schmid GP, Dan BB! et al: Toxic shock syndrome 8. Osterholm MT, Davis JP, Gibson RW, et al: Tristate toxic shock

in menstruating women. N Engl J Med 1980; 303: 1436- syndrome study. I. J Infect Dis 1982; 145: 431-440. 1442. 9. Kehrbero MW, Lathan RH, Haslam BT, et al: Risk factors for .-.

3. Fleiss JL: Statistical methods for rates and proportions, 2nd staphylococcal toxic shock syndrome. Am J Epidemiol

ed. New York: John Wiley 8 Sons, 1981. 1981; 114: 873-879.

4. Hulka BS: Tampons and toxic shock syndrome (editorial). 10. Files on toxic shock syndrome investigation, Public Information

JAMA 1982; 248: 872-874. Officer, Centers for Disease Control, Atlanta (available

5. Davis JP, Chesney PJ, Want PJ, et al: Toxic shock syndrome. under Freedom of Information Act).

N Engl J Med 1980; 303: 1429-1435. 11. Jason J, Anderek N, Marks J, Tyler C: Child abuse in Georgia:

6. Toxic shock syndrome-United States. Morbid Mortal Weekly a method to evaluate risk factors and reporting bias. Am

Rep 1980; 29: 297-299. J Public Health 1982; 72: 1353-1358.


diagnostic bias and toxic shock syndrome

Documents