19 scientific qualit oyf original research article osn ... · designs article conclusion, article...

Tobacco ConmU997i6il9~26 19

Scientific quality of original research articles onenvironmental tobacco smoke

Deborah E Barnes, Lisa A Bero

Institute for HealthPolicy Studies, Schoolof Medicine,University ofCalifornia, SanFrancisco, California,

RE Barnes

Department of ClinicalPharmacy, School ofPharmacy, Universityof California, SanFrancisco, California,USAlABero

Correspondence to:Or Lisa A Bero, Institute forHaslth PoJicy Studies, 1388gutter SEteet, 11th floor, SanFrancisco, California 94109,

SAj eiuaii;[email protected]

AbstractObjective—To evaluate the scientific qual-ity of original research articles on thehealth effects of environmental tobaccosmoke; to determine whether poor articlequality is associated with publication innon-peer-reviewed symposium proceed™ings or with other article characteristics.Design—Cross sectional study of originalresearch articles on the health effects ofenvironmental tobacco smoke publishedin peer reviewed Journals and non-peer-reviewed symposium proceedings from1980 to 1994. Article quality was assessedby two independent reviewers who used avalid and reliable instrument, wereunaware of study hypotheses, wereblinded to identifying characteristics ofarticles, and had no disclosed conflicts ofinterest.Participants—All symposium articles (n =68) and a random sample of peer reviewedjournal articles (n = 68) that satisfiedinclusion/exclusion criteria.Main outcome measure—Mean qualityscores, which could range from © (lowestquality) to 1 (highest quality).Results—Using multivariate regressionanalysis, symposium articles were of*poorer scientific quality than peerreviewed journal articles when controllingsimultaneously for the effects of studydesigns article conclusion, article topic,and source of funding acknowledged (P =0.027). Article quality was not associatedwith either source of funding acknowl-edged or article conclusion in multivariateanalyses.Conclusions—In published reports onenvironmental tobacco smoke, non-peer-reviewed symposium articles tend to be ofpoor quality. These articles should not beused in scientific, legal, or policy settingsunless their quality has been independ-ently assessed.

{Tobacco Control 1997;6:19-26)

Keywords: environmental tobacco smoke; peer review;quality assessment; symposium proceedings

A symposium is a meeting or conference fordiscussion of a subject, or a collection ofarticles on a particular topic. A written sympo-sium is usually based on a conference proceed-ing, and is typically published as either asupplement or special section of a scientificjournal or as an individual book or monograph.Symposia are often industry sponsored andthey are rarely peer reviewed, giving rise to

concern that they may contain articles that arebiased in content and poor in quality.15

The tobacco industry has sponsored at leastfour symposia related to environmentaltobacco smoke/*7 Two additional symposia onthis subject3 9 did not explicitly acknowledgetobacco industry sponsorship but wereorganised by Individuals with tobacco industryaffiliations.3 We have previously examined thecontent of symposia on environmental tobaccosmoke,3 finding that symposia are not balancedand that they tend to support the tobaccoindustry position that environmental tobaccosmoke is not harmful to health. It is not known,however, whether symposium articles are alsopoor in quality.

It is important to examine the quality ofsymposium articles on environmental tobaccosmoke because the tobacco industry has useddata from symposia in scientific, legal, andpolicy settings. For example, the tobaccoindustry cited findings from symposiumstudies in its criticism of the US Environmen-tal Protection Agency's risk assessment ofenvironmental tobacco smoke.10 In addition,R J Reynolds Tobacco Company has runadvertisements that use quotes from symposiato suggest that exposure to environmentaltobacco smoke is not hazardous." Further-more, several scientists who have publishedarticles in tobacco industry sponsoredsymposia have testified as expert witnesses onthe tobacco industry's behalf.32 An examina-tion of the quality of symposium articles maytherefore prove useful to judges, policy makers,scientists, and journalists, who must decidewhether to incorporate symposium articles onenvironmental tobacco smoke into their delib-erations.

The primary objective of our study was tocompare the overall scientific quality of peerreviewed journal articles and non-peer-reviewed symposium articles on environmentaltobacco smoke. We defined scientific quality asthe quality of design, reporting, analysis, andinterpretation. We evaluated the overallscientific quality of publications, rather thansimply the quality of methodology, because wearc interested in guiding the use of scientificpublications in policy and legal settings. A sci-entific publication should be well designed,accurately reported, properly analysed, andappropriately interpreted if it is used to guidepolicy or legal decisions.

In addition to comparing the quality of peerreviewed and symposium articles, we evaluatedseveral other characteristics of articles that wehypothesised might be associated with quality.

on February 27, 2021 by guest. P

rotected by copyright.http://tobaccocontrol.bm

j.com/

Tob C

ontrol: first published as 10.1136/tc.6.1.19 on 1 March 1997. D

ownloaded from

http://tobaccocontrol.bmj.com/

20 Barnes, Bero

These included research sponsorship (thesources of funding acknowledged),-article con-clusion (that environmental tobacco smoke isharmful or not harmful), article topic, andstudy design.

Previous studies have suggested that bothresearch sponsorship and article conclusionshave influenced the overall content of scientificreports. For example, several studies haveshown that research sponsored by interestedparties tends to produce results that arefavourable to the sponsors.1*"" Similarly, theexistence of publication bias suggests that thereports may contain a preponderance ofpositive studies.18" However, the relationbetween these factors and the quality of thereports has not been studied.

We hypothesised a priori that poor articlequality would be associated with publication insymposium proceedings, with tobacco industrysponsorship, and with a conclusion thatenvironmental tobacco smoke is not harmful tohealth. Article topic andstudy design(experimental versus observational) wereevaluated as potential confounding variables.

MethodsSELECTION O£ PEER REVIEWED AND SYMPOSIUMARTICLESPeer reviewed articles on the health effects ofenvironmental tobacco smoke were identifiedby searching Medline from 1 January 1980 to15 March 1994 using the key words "environ-mental tobacco smoke," "tobacco smokepollution," "passive smok#," "involuntarysmok#," or "sidestream smoke,** retrieving atotal of 2189 citations. The search results werenarrowed electronically by excluding non-English language publications (n = 393); byexcluding studies not related to humansubjects (n = 215); and by excluding articlesclassified by Medline as reviews, news items,editorials, commentaries, or letters to theeditor (n = 504). The remaining 1077 articleswere evaluated for inclusion.

Articles were included if they described thestudy of the health effects of environmentaltobacco smoke in humans, were originalresearch studies, and were published in peerreviewed journals, as defined below. Of the1077 articles considered, 386 (36%) satisfiedour inclusion/exclusion criteria.

Studies of environmental tobacco smokeArticles were included if environmentaltobacco smoke was the focus of the study, or ifit was one of several risk factors examined.Studies of the effects of maternal smoking onthe fetus were excluded because the fetus isexposed to mainstream smoke rather thanenvironmental tobacco smoke.

Studies of health outcomesArticles were included if they studied anyhealth outcome related to environmentaltobacco smoke, ranging from lung cancer toeye irritation. Articles were excluded if theydiscussed policy issues related to environmen-tal tobacco smoke, measured exposure without

assessing the effects, or discussed theoreticalissues related to study design or analysis.

Studies conducted in hitman subjectsOnly studies conducted in human subjectswere included. In vitro, cellular, and animalstudies were excluded.

Studies that presented original research findingsOriginal research articles were defined as arti-cles that presented data based on research con-ducted by the authors. Articles were included ifthey incorporated a short review of publishedreports followed by a presentation of originalresearch findings. Articles were also included ifthey presented reanalyses of another investiga-tor's data, because the reanalysis wasconsidered to be original work. Meta-analyses,which would have required a differentinstrument for quality assessment, wereexcluded but are being assessed in a differentstudy.

Studies published in peer reviewed journalsArticles were included only if they werepublished in peer reviewed journals. A journalwas considered peer reviewed if it explicitlystated that its articles were peer reviewed, if itpublished a list of peer reviewers, or if itrequired submission of multiple copies ofmanuscripts for review before publication.Articles published in symposium proceedingswithin a peer reviewed journal were consideredfor inclusion as symposium articles, asdescribed below.

Symposium articles on environmentaltobacco smoke were identified through threesources. (1) For a previous study,* we identified11 symposia that primarily contain articles onenvironmental tobacco smoke by searchingMedline, Catalog, Dialog, Conference PapersIndex, Toxline, and International Guide toPeriodicals using the key words "environmen-tal tobacco smoke," tobacco smoke pollution,""passive smoking," or "involuntary smoking"and "symposia,** "proceedings," or "conferencepaper." (2) Additional articles were identifiedby searching Medline as described above foepeer reviewed articles, except that only sympo-sium articles were considered for inclusion. (3)Finally, we identified symposia on indoor air bysearching Catalog using the title words "indoorair" and either "symposi#,M "proceedings,""conference," "meeting," or *congressM> an<*we then examined their contents for articles oaenvironmental tobacco smoke; 301 articleswere identified through these three sources andwere considered for inclusion. Articles w#*included if they studied the health effects ofenvironmental tobacco smoke in humans an«were original research studies, as defined abortfor peer reviewed articles, and if they w«*published in a symposium. "Symposium" v-"35

defined as a publication that contains a collettion of articles based on a conference <*proceeding.

Of the 301 symposium articles identified,^(30%) satisfied our inclusion/exclusion crft*|ria. However, 22 (24%) of these were excludebecause they were duplicate publications; *a



j.com/

Tob C


ownloaded from


f article on environmental tobaco 21

is, they discussed the same outcome data fromthe same group of research subjects as anothersymposium article in the study. In one case, thesame article had been published in four differ-ent symposia.30 " We identified duplicate pub-lications within the symposium article databaseby comparing articles that had at least oneauthor with the same last name. The articlethat appeared to present the most completediscussion of the data was selected forinclusion. (A similar technique was used toidentify duplicate publications within the peerreviewed article database; none wereidentified.)

Our final sample consisted of 68 articlesfrom 22 symposia: 31 articles had beenincluded in our previous study of symposia onenvironmental tobacco smoke, while 37articles were from new symposia identifiedthrough Medline or Catalog. The number ofarticles per symposium ranged from 1 to 11,with a median of 2 (interquartile range; 1 to 4).None of the symposia were peer reviewed.

Sixty eight peer reviewed journal articleswere randomly selected for comparison withthe 68 symposium articles. Because the qualityof the scientific literature has improved overtime/4 2(> we wanted to ensure that the publica-tion dates of the peer reviewed and symposiumarticles did not differ significantly. Wetherefore determined the number of sympo-sium articles published from 1980 to 1984 (n =17), 1985 to 1989 (n - 32), and 1990 to 1994(n =19) and used a random number generatorto select the same number of peer reviewedjournal articles within each time block. The 68peer reviewed articles selected had beenpublished in 41 journals: The number ofarticles per Journal ranged from 1 to 6̂ with amedian of 1 (interquartile range: 1 to 2).

A minimum sample size of 38 articles ineach comparison group (symposium v peerreviewed) was necessary to detect a differencein quality scores of 0.10 (on a scale of 0 to 1) ata two tailed a = 0.05, p = 0.05 (95% power),and standard deviation = 0.12.27

QUALITY ASSESSMENTWe hired two reviewers to assess the quality ofthe articles independently, using an instrumentthat has previously been tested for validity andreliability.23 Both reviewers were postdoctoralfellows who had received doctorate degrees insocial psychology as well as MPH (Masters inPublic Health) degrees in epidemiology.Reviewers were trained to use the instrumentand were given a detailed set of instructions forreferral during the study. In order to minimisepotential biases,39 30 reviewers were unaware ofour study hypotheses, did not have extensiveknowledge of the literature on environmentaltobacco smoke, stated that they had never beeninvolved with either pro-tobacco or tobaccocontrol groups, and stated that they did nothave any financial conflicts of interest.

Articles were masked so that identifyingcharacteristics—such as author names andaffiliations, journal and book titles, dates ofpublication, sources of funding^ andacknowledgments—had been completely re-

moved. Articles were sent to reviewers in a ran-dom order using a random number generatoron a computer.

Our quality assessment instrument isdesigned to measure the quality of design,reportingj analysis, and interpretation of origi-nal research articles."8 This instrument wasspecifically designed to be applicable to a widerange of study designs, regardless of articletopic. Compared to other published qualityassessment instruments, ours has highreliability.31 The instrument consists of a 22item questionnaire that asks reviewers to deter-mine whether certain items associated withhigh quality are present in an article (itemslisted in table 2). Reviewers may answer thatthe item is "present", "partially present", "notpresent", or "not applicable" for the article.Reviewers are also asked to classify the studydesign of the article into one of 15 categories.An overall quality score is calculated based onthe study design and the number of itemspresent using a scoring system has been previ-ously described.28 Quaiity scores may rangefrom 0 (lowest quality) to 1 (highest quality).

Our analyses were based on the mean qualityscore for each article (that is, the average of thetwo reviewers' scores). If the reviewers' scoresdiffered by more than one standard deviation,the masked article was discussed by bothreviewers until consensus was achieved, andthe consensus score was used. The interraterreliability between the two reviewers wasassessed using Kendall's coefficient ofconcordance with adjustment for tied ranks(W) and the intraclass correlation coefficient(R). Initial interrater reliability was good (W =0.85, R - 0.64). Reyiewer scores differed bymore than one standard deviation for 22articles (six peer reviewed, 16 symposia). Afterreviewers had reached consensus on thesearticles, interrater reliability was high (W = 0.92,R = 0.87),

CLASSIFICATION OF OTHER VARIABLESSponsorshipResearch sponsorship was determined basedon statements in the articles and was classifiedas either "tobacco industry", "government","other", or "none acknowledged." Articleswere classified as tobacco industry sponsored ifa tobacco company, the Tobacco Institute, theCouncil for Tobacco Research, the Center forIndoor Air Research, the Smokeless TobaccoResearch Council, or a combination of these,were the sole sources of funding acknowl-edged. Articles were classified as governmentsponsored if one or more government agencies,such as the National Institutes of Health, werethe sole source of funding acknowledged. Arti-cles that acknowledged other sources offunding, such as private foundations, or whichacknowledged funding from multiple sources(for example, government and private founda-tions) were classified in the "other" category.

Article conclusionArticle conclusions were coded as "positive"(environmental tobacco smoke harmful) if thearticle concluded that environmental tobacco



j.com/

Tob C


ownloaded from


22 Barnes, Be?/)

smoke had an adverse effect on one or more ofthe outcomes measured; "negative"" (environ-mental tobacco smoke not harmful) if the arti-cle concluded that environmental tobaccosmoke did not have an adverse effect on any ofthe outcomes measured; or "no conclusion" ifthe article controlled for environmentaltobacco smoke as a confounding variable anddid not draw any conclusions about its healtheffects.

Study designOur scoring system weights experimental stud-ies higher than observational studies.'5 To con-trol for study design as a potential confoundingvariable, we classified each article as eitherobservational (for example, the effects ofenvironmental tobacco smoke were deter-mined by asking people about their history ofexposure) or experimental (for example, the:effects of environmental tobacco smoke weredetermined by exposing subjects to it in anexperimental chamber).

Article topicA preliminary examination of the dataindicated that peer reviewed and symposiumarticles tended to study different topics. Toevaluate an article topic as a potentialconfounding variables the topic of each articlewas coded as either "lung cancer", "respiratoryeffects"; "other chronic disease", "biochemicaleffecfs"(for example, urinary rnutagenidty),"sick building syndrome" (for example,headache and eye irritation in the workplace),or "other."

DUPLICATE PUBLICATIONSAs described above in the section on selectionof peer reviewed and symposium articles, weeliminated duplicate publications that oc-curred within the symposium article database(symposium/symposium duplicates) andwithin the peer reviewed article database (peer/peer duplicates). However, we were interestedin comparing the quality of duplicatepublications that were published in both asymposium and a peer reviewed journal (peer/symposium duplicates). We identified theseduplicate publications by searching our entiredatabase of 386 peer reviewed articles usingthe last names of the authors from the 68 sym-posium articles included in the study. Articlesthat had at least one author with the same lastname were compared. Articles were consideredto be duplicates if they discussed the same out-come data from the same subject population.Fifteen pairs of peer/symposium duplicateswere identified. In four cases, there was onlypartial overlap between the articles (for exam-ple, one article discussed lung cancer,respiratory disease, and heart disease in agroup of subjects, whereas the other articleonly discussed the data related to lung cancer).All of the symposium articles and five of thepeer reviewed articles had been included in themain study. The quality of the 10 remainingpeer reviewed articles was assessed by mixingthem in randomly with the other articles.

ANALYSESWe used %2 tests to compare peer reviewed andsymposium articles in terms of qualitativecharacteristics. To examine the relationbetween our outcome variable, mean qualityscore, and the various predictor variables, wefirst conducted ursivariate analyses using oneway analysis of variance (ANOVA). We then con-ducted multivariate regression analyses inwhich two models were developed. In the firstmodel (full model), all of the variables wereincluded regardless of whether they weresignificantly associated with quality scores.Categorical variables were represented usingdummy variables. To develop our second, sim-pler model, we began by including all variablesthat were significantly associated (P < 0.05)with quality scores in the full model. We thenadded additional variables one at a time, focus-ing on those that were associated with poorarticle quality. Variables were retained if theywere associated with poor quality scores, or ifthey changed the association between qualityscores and the other variables in the model,Duplicate publications were compared usingthe paired t test. AH tests were conducted usinga two tailed a - 0.05. Confidence intervalswere calculated based on the t distribution.

Assumption checkingBoth ANOVA and multivariate regression analy-ses use the F test, which is based on theassumption that the outcome variable (qualityscores) is approximately normally distributed]with constant variance at all values of the inde-pendent variables, In our sample, peerreviewed articles had normally distributedscores, but the distribution of symposium arti-cles was slightly skewed (Shapiro-Wilk test, P ~0.03). However, the median symposium score(0.375) did not differ greatly from the meaasymposium score (0.365). In addition; thestandard deviations for peer reviewed articles(0.08) and symposium articles (0.13) did notdiffer greatly. Furthermore, the F test isreasonably robust when sample sizes aregreater than 30 and roughly balanced in eacftgroup. A plot of residual values from the fuflmultivariate regression model did not revealany obvious violations of the assumptions. V?etherefore believe that our analyses providevalid estimates of the relations between articlequality, peer review status, and the otherpredictor variables examined.

ResultsCHARACTERISTICS OF PEER REVIEWED ANt>SYMPOSIUM ARTICLES jTable 1 shows that peer reviewedarticles and non-peer-reviewedarticles differed from each other in terms ^funding sources acknowledged, article comsion, study design, and topic. Symposiumst'cles were more likely than peer reviewarticles to fail to acknowledge their sources:funding, to conclude that environingtobacco smoke is not harmful, toexperimental study designs, and to study £ ?related to sick building syndrome-reviewed articles, on the other hand) ******



j.com/

Tob C


ownloaded from


Qualityof• articles oner,•voironmental tobacco smoke 23

Table Ianides

Characteristics of peer reviewed and symposium.

Source of funding*Tobacco industryGovernmentOtherNone acknowledged

Article conclusion."ETS harmfulETS nof harmfulNo conclusion

Study design*ExperimentalObservational

Topic11

Lung cancerRespiratory diseaseOther chronic diseaseBiochemical effectsSick building syndromeMiscellaneous

Per csm of articles

Peer reviewedarticles

2.927-923.545.6

76.520.1

2.9

8.891.2

10,351.556.210.34.47.4

Symposiumctnicks(h=68)

4.47.4

14.773.5

45.641.23 3.2

25.075.0

13.235.3

1.510.327.911.8

ETSf environmental tobacco smoke.

bXJffi14.4jdf=2jP«0.00]..

likely than symposium articles to acknowledgegovernment sponsorship, to conclude thatenvironmental tobacco smoke is harmful, touse observational study designs, and to studyrespiratory disorders and other chronicdiseases, Given the underlying differencesbetween peer reviewed and symposiumarticles, it is particularly important to conductrnultivariate analyses in order to ensure thatany differences observed in quality scores arenot actually due to these other factors.

UNIVARIATB ANALYSESIn the univariate analyses, article quality washighly associated with, peer review status. Peer

Table 2 hem 6y item analysis of quality $cores

reviewed journal articles had mean qualityscores of 0.45 (95% confidence interval (CI):0.43 to 0.47), whereas non-peer-reviewedsymposium articles had mean quality scores of0.36 (95% CI; 0.33 to 0.40) (F = 21,15, df = I,P < 0.0001), The scores of peer reviewed arti-cles ranged from 0.24 to 0.61, while the scoresof symposium articles ranged from 0.07 to0,71

Table 2 presents an item by item analysis ofthe differences between peer reviewed andsymposium articles. Peer reviewed articlesscored higher than symposium articles on mostof the items in our quality assessmentinstrument. For example, peer reviewedarticles were more likely than symposium arti-cles to have a well described study question, to•use an appropriate study design, to specifyinclusion and exclusion criteria, to studyappropriate subject populations, to reportfindings completely, and ro have conclusionsthat were consistent with their results. The cat-egories in which peer reviewed and symposiumarticles did not differ tended to be those inwhich both groups scored poorly. For example,neither symposium nor peer reviewed articlesroutinely reported blinding of investigators orsubjects. Symposium articles did not receivesignificantly higher scores than peer reviewedarticles for any of the criteria measured.

The univariate analyses also revealed thatmean quality scores were associated with fund-ing source acknowledged (F - 2.87, df = 3, P =0.039) and with article topic (F - 9.33, df =s 5,P < 0.0001). Post hoc testing using the Scheffetest suggested that articles that failed toacknowledge their funding sources had poorerscores than those that acknowledged any fund-ing source, although the differences were notstatistically significant. In addition, articlesrelated to sick building syndrome hadsignificantly lower scores than articles on allother topics., based on the post hoc Scheffe

Item

Percent (number) of anklessmamningitem

Peer retn&wed articles Symposium articles Fischer's exact P value

1 Study question weii described 96% (65/68) 75% (50/67) 0.0006*2 Study design appropriate 84% (57/68) 55% (36/65) 0.0006*3 Inclusion/exclusion criteria specified 56% (37/66) 15% (9/59) <0.0001*4 For case repori only: patient characteristics reported 0% (0/0) 100% (I/I) n/a5 Subject population appropriate 88% (60/&8) 59% (36/61) 0.0002*6 ConrroS population appropriate 64% (1*22) 28% (7/25) 0.027 Subjects randomly selected 14% (9/66) 9% (5/58) 0.418 Method of random selection described 0% (0/9) 0% (0/6) a/a9 Method of random allocation described 0% (0/0) 0% (0/2) n/a

10 Investigators blinded 4% (2/53) 7% (3/46) O.SS13 Subjects blinded 7% (3/45) 3% (1/39) 0.6212 Measurement bias accounted for 46% (3I/6S) 27% (17/62) 0.0413 Confounders accounted for by study design 43% (18/42) 16% (8/50) 0.00614 Confounders accounted for by analysis 70% (42/60) 50% (29/5S) 0.0415 A priori sample size justification 3% (2/67) 0% (0/61) 0.50% 6 Post-hoc power calculations for non-significant

results 36% (22/61) 18% (9/51) 0.0417 Statistical analyses appropriate 83% (54/65) 6S% (36/53) 0,0818 Statistical tests stated 95% (62/65) 80% (40/50) 0.0219 Exact P values or confidence intervals reported 47% (30/64) 32% (16/50) 0.1320 Attrition of subjects discussed 67% (6/9) 23% (3/13) 0.0821 Results completely reported 97% (66/68) 75% (46/61) 0.0004*22 Findings support conclusions 96% (64/67) 73% (48/66) 0.0003*

The data are presented as the percentage of articles for which one or both reviewers rated the criterion as "present". Thedenominator in each ceil varies because the article was excluded if one or both reviewers felt that the criterion, was "notapplicable" (n/a) for the article.*Using a Bonfemmi adjustment for multiple comparisons, only P < 0.0023 sho«!d be considered statistically significant



j.com/

Tob C


ownloaded from


24

Table 3 Muhivariau regression analysis: full model

Peer review statusPeer reviewed v symposium

Source of fundingTobacco industry v otherGovernment v otherNone v other

Article conclusionETS harmful v no conclusionETS not harmful t> no conclusion

Study designExperimental v observational

TopicLung cancer v miscellaneousRespiratory diseases v miscellaneousOther chronic diseases v miscellaneousBiochemical studies v miscellaneousSick building syndrome v miscellaneous

Coefficient

0.046

-0,0630.016

-0.029

0.0220.023

0.088

0.0760.0680.0860.011

-0.073

95% Cl

0.005 to 0.086

-0.168 to 0.042-0.040 to 0.072-0.074 to 0.016

-0.042 to 0.087-0.043 ro 0.090

0.027 to 0.149

-0.0H to 0,162-0.005 to 0.141-0.004 to 0,176-0,069 to 0.090-0.152 to 0.005

t

2.243

-1.1820.581

-1,291

0.6890.688

2.856

1,7311,8531.8970,26?

-1.845

Barnes, Bero

P value

0.027

0.240.5620,199

0.490.49

0.005

0.0860.0660.0600.7880.067

CIj confidence interval.

test. Article conclusion was marginallyassociated with quaiity scores (F = 2.99, df~ 2,P = 0.054), but study design was not (F - 0.75,

MULTJVAKIATE DEGRESSION ANALYSISThe results of our multivariate analysis usingthe full model are presented in table 3. Themost important predictors of article qualitywere peer review status (P = 0.027) and studydesign (P ~ 0.005). The coefficients may beinterpreted as the change in mean quality scoreassociated with a given variable whilecontrolling for all other variables in the model.Thus when controlling for all other factorsmeasured, peer reviewed articles had meanquality scores that were 0,046 points higher(on a scale of 0 to 1) than symposium articles.Similarly, experimental studies had mean quai-ity scores that were 0.088 points higher thanobservational studies when controlling for allother variables. Neither funding source norarticle conclusion was significantly associatedwith article quality in the multivariate model.Roughly 37% of the variability in quality scoreswas explained when all of the variables wereincluded (R2 = 0.372).

The results of the final, simpler model aregiven in table 4. We included both peer reviewstatus and study design, because they were sig-nificant in the full model. In addition, weobtained the most significant effects when thefunding variable was dichotomised as anyfunding acknowledged versus no fundingacknowledged and the topic variable wasdichotomised as sick building syndrome versusall other topics. In the simpler model, the

Table 4 Multivariate regression analyses; final model

primary predictors of article quality were peerreview status (P = 0.005) and sick buildingsyndrome (P < 0.0001). Roughly 33% of thevariability in quaiity scores was explained usingthe simpler model (R2 = 0.33).

DUPLICATE PUBLICATIONSOur analysis of duplicate publications revealedthat the articles published in peer reviewedjournals had a mean quality score of 0.49 (95%CI: 0.44 to 0.54), whereas the same studiespublished in symposia had a mean qualityscore of 0.43 (95% CI: 0.40 to 0.46) (paired ttest, P ~ 0.01). Quality scores were higher forpeer reviewed articles than for symposium arti-cles in 13 of the 15 pairs.

DiscussionOur findings confirm our primary hypothesisthat articles published in symposium proceed-ings are associated with poor quality. We foundthat symposium articles had significantiypoorer mean quality scores than peer reviewedarticles even after controlling for the effects offunding source acknowledged, article conclu-sion regarding the health effects ofenvironmental tobacco smoke, type of studydesign used, and article topic.

The criterion by criterion analysis showedthat peer reviewed articles were superior tosymposium articles in terms of study designreporting, and interpretation. For example]peer reviewed articles were more likely to useappropriate study designs and subjectpopulations; to report their study objectives,inclusion/exclusion process, and results

Coefficient 95% Cl P value

Peer review statusPeer reviewed v symposium

Source of fundingNone acknowledged v any acknowledged

Study designExperimental v observational

TopicSick building syndrome v ail other topics

0.053

0.032

0,042

0.127

0.016 to 0.089

-0.067 to 0.003

-0,002 to 0.087

-0.174 to -0 .081

2.868

-1 .824

1.871

-5.391

0.005

O.O70

0.064

<o.oooi

CI, confidence interval.



j.com/

Tob C


ownloaded from


Quality ofarticles on environmental tobacco smoke 25

adequately; arid to draw conclusions that wereconsistent with their findings.

Our two other a priori hypotheses were notconfirmed. We had originally proposed thattobacco industry sponsorship would be associ-ated with poor quality. However, because onlyfive articles ia our study (two peer reviewed,three symposium) acknowledged tobaccoIndustry sponsorships we did not have enoughstatistical power to evaluate this hypothesis.Interestingly, our findings suggest that the fail-ure to acknowledge funding sources is margin-ally associated with poor quality. It is possiblethat articles that fail to acknowledge fundingsources may tend to have poor reporting ingeneral, and this may explain why their qualityscores tend to be lower. This finding should beinvestigated in future studies.

Our third a priori hypothesis was thatarticles with negative conclusions (environ-mental tobacco smoke is not harmful tohealth) would be associated with poor quality.This hypothesis was not confirmed. Articleconclusion was not associated with articlequality when controlling for factors such aspeer review. We have previously found thatpeer reviewed articles are more likely thansymposium articles to conclude that environ-mental tobacco smoke is harmful.52 Thetobacco industry has suggested that this maybe due to a bias in the peer reviewed journalsagainst publishing negative studies on environ-mental tobacco smoke.35 Our findings suggest,however, that negative studies on environmen-tal tobacco smoke are published in the peerreviewed journals when they are of highquality.

Our findings also suggest that articles relatedto sick building syndrome may be of poorerquality than articles on other topics. We did notestablish this hypothesis a priori;, and this find-ing should be further evaluated by other stud-ies.

Our analysis of duplicate publicationsshowed that peer reviewed articles are higher inquality even when they discuss trie same dataas symposium articles. Because these pairs ofduplicates, by definition, had the same studydesign characteristics, the differences observedbetween peer reviewed and symposium articleswere most likely to have been due to factorssuch as poor reporting and poor interpretationof findings.

The findings from this study complementour previous work on the content of symposiaon environmental tobacco smoke. We havepreviously found that symposium articles onenvironmental tobacco smoke are associatedwith a lack of balance and that they tend tosupport the tobacco industry position thatenvironmental tobacco smoke is not harmful.2

Our findings reported in this article suggestthat symposium articles on environmentaltobacco smoke are also associated with poorquality. Taken together, these findings suggestthat symposium articles are not reliable sourcesof information about environmental tobaccosmoke.

Our findings are also consistent withprevious research on the quality of symposium

articles in the pharmaceutical literature. Forexample, Rochon3 has found that randomisedcontrolled drug trials are of poorer qualitywhen published in journal supplements than inparent journals. On the other hand; Cho andBero'J did not find a difference in the quality ofarticles on drugs published in symposia andpeer reviewed journals; however, they notedthat their power to detect a difference whencontrolling for the effect of study design waslow.

LIMITATIONSThere are several limitations to our findings.First, as mentioned above, we did not haveenough statistical power to assess the relationbetween funding source and article quality.Therefore our finding that article quality wasnot associated with sponsorship should beinterpreted cautiously.

Another potential source of criticism lies inour selection of peer reviewed and symposiumarticles. Our symposium articles were drawnfrom 22 symposia while our peer reviewed arti-cles were drawn from 41 journals. If a singlejournal or symposium contained extremelyhigh or low quality articles, then it could theor-etically have shifted the mean quality score ofthe entire group. We do not believe that thisoccurred because no single journal orsymposium dominated our samples. Themaximum number of articles per symposiumwas 11 (16%), while the maximum number ofarticles per peer reviewed journal was six (9%).The quality of articles from these sources didnot differ from the overall quality of articles inthe groups.

Another potential source of selection bias isthat the peer reviewed articles in our studywere identified exclusively through Medline,whereas the symposium articles were identifiedthrough a variety of electronic databases. Weused Medline to identify peer reviewed journalarticles on environmental tobacco smokebecause it is the database most commonly usedin the United States to gather informationabout health related research. However, severalpeer reviewed journals that publish articlesrelated to environmental tobacco smoke,including Tobacco Control, were not indexed byMedline during the period of our study.34 3S

Future studies on the quality of the scientificliterature should therefore consider drawingtheir samples from multiple, sources; such assearching a variety of electronic databases*hand searching journals, and checkingreferences.

Another potential criticism of our study isthat the difference in mean quality scoresobserved between peer reviewed andsymposium articles, although statisticallysignificant, was not particularly large. We feelthat table 2 provides qualitative insight into thequantitative differences between the groups.This table shows that peer reviewed articles aremore likely than symposium articles to be welldesigned, to report results completely andaccurate, and to draw appropriate conclusions.When these qualitative and quantitativefindings are taken together, they provide strong



j.com/

Tob C


ownloaded from


26 Barnes, Bero

evidence that peer reviewed articles tend to beof higher quality than symposium articles.

Finally, it is not clear whether our findingsmay be generalised to symposia on othertopics. Our study primarily contained articlesfrom symposia on environmental tobaccosmoke and symposia on indoor air, and wefound that article quality scores were similarlypoor in both groups (data not shown). Otherstudies have found that the quality ofsymposium articles in the pharmaceuticalliterature tends to be poor.3 Taken together,these findings suggest that symposia in generalmay be poor in quality. However, this shouldbe confirmed by future studies.

CONCLUSIONSIn summary, our study provides strongevidence that peer reviewed journal articles onthe health effects of environmental tobaccosmoke are superior to symposium articles.These findings support the decision of the USSupreme Court16 that expert scientifictestimony should be "scientifically valid" andthat "[a] pertinent consideration is whether thetheory or technique has been subjected to peerreview and publication."

However, pur findings also suggest that peerreview is not a guarantee of high quality. Thepeer reviewed articles in our sample had amean quality score of 0.45 on a scale of 0 to 1,suggesting that there is a great deal of room forimprovement. Because decisions should bebased on the highest quality evidence available,we propose that researchers, policy makers,and judges should evaluate the quality of allscientific research before using the findings inresearch, policy, or legal settings, A lack of peerreview should be seen as a flag that an articlehas a higher probability of being poor in qual-ity.

On the basis of our findings, we believe thatjournal and book editors should developstandards for labelling scientific material. It isoften difficult to determine whether an articlehas hsen published in a peer reviewed journalor a non-peer-reviewed symposium becausethis information is not printed directly on thearticles. We therefore recommend that the firstpage of all scientific research articles should beclearly labelled to indicate whether the articlewas peer reviewed and whether it is being pub-lished as part of a symposium. This will allowpolicy makers, researchers, the press, and thepublic to identify those articles that are mostlikely to be associated with high quality,

This study was randed by the Robert Wood JohnsonFoundation (award No 024783), and by the Cigarette andTobacco Surtax Fund of the State of California, through thetobacco related disease research program of the University ofCalifornia (award No 4RT0035).

We wish K> thank our quality assessors, Valerie HoffmanPhD, MPH and Judith Moskowiu PhD, MPH; the WritingSeminar at the Institute for Health Policy Studies, whichprovided invaiuabk insight during the preparation of Ehiamanuscript; Phillip LolSar for administrative assistance; andSteve Seivin PhD for statistical advice.

The results of this study were presented in part at the 1995meeting of the American Public Health Association in SanDiegOj California.

1 Bero LAS Galbraith A, Rermie D. The publication of spon-sored symposiums in medical journals. N Engl J Med1992;32?:1135~40.

2 Bero LA, Galbraith A, Rennk E>. Sponsored symposia onenvironmental tobacco smote. JAMA 1994;27l;612--7.

3 Rochon P- Evaluating the quality of articles published injournal supplements compared with the quality of thosepublished in fee parent journal. JAMA 1994;272:IO8-~13,

4 Ecobichon DJ, Wti JM. Bnvirormieniai tobacco smoke:proceed*ings of ihs International Symposium at McGill University,Lexington, Massachusetts: Lexington Books, 1990,

5 Symposium proceedings, Assessing low-risfe agents for Hungcancer; methodological aspects, (International Sympo-sium held in Clearwater, FL> USA, October 10-14,1989.)JmJEpidemiol I990;I9 (suppl I).

6 Workshop proceedings. ETS—Environmental tobaccosmoke. (Workshop on effects and exposure levels, held atthe University of Geneva, Switzerland, March 15-17)1983.) EurJRespirDis 1984;65(suppl 133).

7 Workshop proceedings. Environmental tobacco smokeeffects of the non-smoker. (Workshop, Bermuda, 1974.)Scand J Respir Dis 1974; suppl 91.

8 Perry Rj Kirk P, Indoor and ambum Mr quality. London:Selper, 1988,

9 Symposium proceedings. International Symposium onMedical Perspectives on Passive Smoking. PrevMed 1084;13.

10 Bero LA, GiantK SG. Tobacco industry response to a riskassessment of environmental tobacco smoke. Tobacco Con-

;11 R J Reynolds Tobacco Company advertisement. "Passive

smoking: an active controversy." (Available from authorsupon request.}

12 Bveringhans R, Woodward S. Tobacco Litigation. Sydney:Legal Books, 1991.

13 Cho MKj Bero LA. The quality of drug studies published vxsymposium proceedings. Ann Intern Med 1996;124:485-9.

3 4 Barnes DE, Bero LA. Industry-ftind«d research and conflictof interest: an analysis of research sponsored by thetobacco industry through the Center for Indoor AirResearch, Ji&a&fc Politics Policy LAW 1996;21:515-42.

15 Rochon PA, Gurwitz JH, Simms RW, et at. A study ofrnanufecturer-supporEfid trials of nortsteroidal anti-inflammatory drugs in the treatment of arthritis. Arch

j16 Swaen G, Meijers J, Influence of design characteristics on

the outcome of retrospective cohort studies. Br J InditstM 4 1 9 8 S 4 5 6 2 4 9

17 Davidson R. Source of funding and outcome of clinical vi~&\$.3<3m Intern Med 1986;l:155-8.

38 Didcersm K.. The existence of publication bias and risk fac-tors for its occurrence. JAMA 1990;263:1385-9.

19 Dickersin K, Min Y-I, Mdnen CL. Factors influencingpublication of research results: Follow-up of applicationssubmitted to two institutional review boards, JAMA 1992;267:374-8.

20 Weber A. Acute effects of environmental tobacco smoke.Bur J Respir £>M 1984;6S:9S~-108.

21 Weber A. Annoyance and irrigation by passive smoking, P?£i>Med 1984;13:618~25.

22 Webet A. Irrigating and annoying effects of passive smoking.TskaiJExp Om Med 1985;10:341-5.

23 Weber A. Environmental Tobacco smoke exposure: acutfieffects-—acceptance level-—protective m«3stires. In: Bfif-glund E, Ljndvall T, Sundell J, eds. Indoor air. Volume 2.Radon, passive smoking, particutate and hottsirig epidemiology*Stockholm: Lybar Tryck AB, 1985: 297-301.

24 Hemmmki E. Qmlity of clinical trials—a concern for threedecades. Meth Inform Med l§82;2l:8l~5.

25 Andrew E> Eide H, Fuglerud P, et al. Publications on clini-cal trials with X-ray contrast media: differences in qualitybetween journals and decades. EurJ Radial 1990;!0:52-J-

26 Sonis Js Joines J. The Quality of clinical trials published inThe Journal of Family Practice, 1974-1993. } Fam Pf<&1994^39:225-55.

27 Hulley SB, Cummings SR- Designing clinicaBaltimore, Maryland: W$kms & WSikins, !988.

28 Cho MK, Bero LA.. Instrument for assessing the quaiityofdrug studies published in the medical literature. JAm1094^272; 101-4. _

29 McNutt RA, Evans AT, Fletcher RH, Fletcher SW. T»effects of blinding on the quality of peer review. JAi^§ Q 3 ? S9 ; 2 I S

30 Ernst Ej Resch K~L. Reviewer bias: a blinded experimentstudy. J Lab din Med 19M; 124:178-82. „ „

31 Moher D, Jadad AR; NIchol G, Penman M, i JWalsh S. Assessing the quality of randomisedcrisis: an annotated bibliography of scales andConzroikdClin Trials I995;16:62~73.

32 Bero LA, Glantz SA, Rennie D. j?ublkatiori bias ^health policy on environment^ tobacco smoke. J1994;272U33~6.

33 Flue-Cured Tobacco Cooperativeand The Council for Burley Tobacco, IncLeaf Tobacco Company, Inc and Philip Morris Jrated and R J Reynolds Tobacco Company andVending Company vs United States Environing"Protection Agency and Carol Browner, 1993,

34 Liguori A, Hughes JR. Where is smoking res«»published? Tobacco Control I996;S:37-8.

35 Davis RM, Tobacco C&nirol\am MdTobacco Control 1996;5:99.

36 Supreme Court of die United States. ^ ^ ^ i ^ Z 23MerreH D<w Pharmaceuticals, Inc (No 92-102) Jw1993.



j.com/

Tob C


ownloaded from


19 scientific qualit oyf original research article osn ... · designs article conclusion, article...

Documents