Reporting Guidelines and scientific articles’ impactA(n) (un)breakable

bond

PEREIRA, Ana Catarina || RODRIGUES, Ana Raquel || BRANDÃO, Daniela || ALVES, Filipe || BASTOS, José Mário || SILVA, José Nuno || MORGADO, Margarida || SILVA, Marta ||Class 21

Adviser: Prof. Pedro Pereira Rodrigues

Introdução à Medicina IIAno lectivo 2011/2012

fmup.turma21@gmail.com

2

Abstract

Reporting Guidelines, impact, guidelines compliance, articles’ impact

About 1.350.000 articles are published in a year and scientific articles have a major importance in scientific knowledge.

Vascular system: role of nitric oxide in cardiovascular diseases.Bian K, Doursout MF, Murad F.SourceThe University of Texas Health Science Center at Houston, The Brown Institute of Molecular Medicine, Houston, TX 77030, USA.

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

3

Original empiric and theoretical work are bothdisseminated through Scientific Literature

Scientific Advancement high impact journals

article may not beproperly written!

• Journals may have:▫ their own policies of publishing;▫ no rules at all.

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Background &

Justification

Conclusions

Example 1: Cardiovascular diseases

It is rare to find a clear description of the monitoring phase in guidelines for cardiovascular diseases

What to monitor?

What to do if a change in a risk factor is detected? [4]

A clear guideline would help clinicians to apply the recommendations in clinical practice

Background &

Justification

[4] du Sert NP. Improving the reporting of animal research: when will we ARRIVE? Disease Models & Mechanisms. 2011; 4: 281-282

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

• The use of specific reporting guidelines, such as CONSORT, actually have a positive influence on the impact of reports. [5]

• The introduction of guidelines resulted in significant improvement in the quality of reporting in the Journal of Pediatric Surgery. [6]

Example 2: CONSORT

Background &

Justification

[5] Cobo E, Cortés J, Ribera JM, Cardellach F, Selva-O’Callaghan A, Kostov B, García L, Cirugeda L, Altman DG, González JA, Sànchez JA, Miras F, Urrutia A, Fonollosa V, Rey-Joly C, Vilardell M. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomized trial. BMJ, 2011 [6] Larson EL, Cortazal M. Publication guidelines need widespread adoption. Journal of Clinical Epidemiology. 2011;

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

!Need of widespread adoption

of reporting guidelines

Research increasingly reported in a standardized and consistent

manner [7]

Background &Justification

[7] Moschetti I, Brandt D, Perera R, Clarke M, Heneghan C. Adequacy of reporting monitoring regimens of risk factors for cardiovascular disease in clinical guidelines: systematic review. BMJ. 2011; 342

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

• Guideline developers must endeavor to:▫ maximize the quality of their product;▫ make journal editors more confident in

endorsing reporting guidelines. [10]

But how to develop guidelines?

Background &Justification

[10] Moher D, Weeks L, Ocampo M, Seely D, Sampson M, Altman DG, Schulz KF, Miller D, Simera I, Grimshaw J, Hoey J. Describing reporting guidelines for health research: a systematic review. Journal of Clinical Epidemiology. 2011; 64:718-742

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

Over the years, researchers tried to reach common

and clear:Reporting Guidelines

Background &Justification

• STREGA

• STROBE

• STARD

• SQUIRE

• MOOSE

• PRISMA

• GNOSIS

• TREND

• ORION

• COREQ

• QUOROM

• REMARK

• CONSORT• ARRIVE

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

• “Reporting guidelines are statements that provide advice on how to report research methods and findings. Most widely recognized guidelines are based on the available evidence and reflect consensus opinion of experts in a particular field, including research methodologists and journal editors.”

• “Reporting guidelines complement advice on scientific writing, which concentrates on the basic writing principles and styles of research reports and publications, and journals' instructions to authors.”

From Equator Network

Reporting Guidelines

Background &Justification

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

• “Reporting guidelines are statements that provide advice on how to report research methods and findings. Most widely recognized guidelines are based on the available evidence and reflect consensus opinion of experts in a particular field, including research methodologists and journal editors.”

• “Reporting guidelines complement advice on scientific writing, which concentrates on the basic writing principles and styles of research reports and publications, and journals' instructions to authors.”

From Equator Network

Reporting Guidelines

Background &Justification

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

11

Does the use of Reporting Guidelines improve the scientific articles’ impact of the General and Internal

Medicine (G&IM) journals?

Research question &

AimsAbstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

12

Research question &Aims

Assess the association between articles' impact and the existence of reporting guidelines available at the

journal web site.

• Measure the utilization rate of Reporting Guidelines

• Describe the G&IM articles’ impact

• Acknowledge the type and number of Reporting Guidelines used in G&IM scientific journals

Specific Objectives

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

13

Research question &Aims

Assess the association between articles' impact and the existence of reporting guidelines available at the

journal web site.

• Measure the utilization rate of Reporting Guidelines

• Describe the G&IM articles’ impact

• Acknowledge the type and number of Reporting Guidelines used in G&IM scientific journals

Specific Objectives

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

14

Methodology

ISI Web of Knowledge

Select G&IM

Acknowledge the 153 journals

Collect information

about all journals

Apply Inclusion & Exclusion

criteria

Statistical analysis

Randomly choose 6 journals

Choose 6 articles from 2008 per

journal

Use CONSORT and STROBE

reporting guidelines –

checklist

Calculate Impact per article

Statistical analysis

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

15

Methodology Collect information

about all journals

• Web Site• Language• Impact Factor

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

16

Methodology Apply Inclusion & Exclusion

criteria

• Web Site• Language – Portuguese, English, Spanish• Impact Factor – at least one from 2008, 2009

or 2010

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

17

MethodologyStatistical analysis

• Statistical analysis based on:• IF’s• Website• Language

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

18

MethodologyAbstract

IntroductionResearch

question & Aims

Methodology

Results

ReferencesJournals which

don’t say anything about RG

Journals which Recommend RG

Journals which Impose RG

These are the three groups in which we divided our articles, in order to analyze them.

Conclusions

19

Methodology

Choose 6 articles from 2008 per

journal

&

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

6 Articles per journal3 -

CONSORT 3 - STROBE

2 Journals per group

3 Groups of journals

Randomly choose 6 journals

Conclusions

20

MethodologyAbstract

IntroductionResearch

question & Aims

Methodology

Results

References

Don’t say anything about using RG• Internal Medicine Journal• QJM: An International Journal of Medicine

Recommend the use of RG• Journal of Pain and Symptom Management• Archives of Internal Medicine

Impose the use of RG• Family Medicine• Healthmed

Conclusions

21

Methodology

We randomly selected the articles for our study, using ISI Web of Knowledge and Excel.

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

22

Methodology Use CONSORT & STROBE reporting

guidelines – checklist

CONSORT:• Randomized

trials• 37 items

STROBE:• Observational

studies• 33 items for

case-control studies, 34 for cohort studies

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

23

MethodologyCalculate Impact

per article

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Impact Factor ≠ Impact per article

Conclusions

ARTICLESJOURNALS

24

MethodologyStatistical analysis

• Based on:• Articles’ impact – ONLY articles from 2008;• Checklist used

Abstract

IntroductionResearch

question & Aims

Methodology

Results

References

Conclusions

25

Results

• Reporting Guidelines&

• Scientific articles’ impact

We’re looking for this association!

Abstract

IntroductionResearch

question & Aims

Results

Methodology

References

Conclusions

26

Abstract

IntroductionResearch

question & Aims

Results

Methodology

References

Results

Table 1. Distribution of IF’s from 2008, 2009 and 2010 for included and exlcuded journals and for all of them. 1) For excluded journals, IF’s have got a normal distribution and data shows as mean ± standard deviation (N=absolute frequencies). 2) For included journals and all of the journals, IF’s haven’t got a normal distribution and data shows as median; [quartil 25, quartil 75] (N=absolute frequencies).

Impact Factor (IF)2008 2009 2010

Excluded journals1

(N=19)

0,90 ± 0,57(12)

0,82 ± 0,57(14)

0,80 ± 0,53(15)

Included journals2

(N=134)

1,63[0,93 ; 2,75]

(92)

1,37[0,64 ; 2,38]

(117)

1,30[0,48 ; 2,14]

(134)

Journals(N=153)

1,52[0,80 ; 2,57]

(104)

1,28[0,58 ; 2,21]

(131)

1,15[0.44 ; 2,03]

(149)

Conclusions

27

Abstract

IntroductionResearch

question & Aims

Results

Methodology

References

Results

Conclusions

Impact Factor (IF)2008 2009 2010

Journals which don’t say anything about RG(N=12)

2,03[0,37; 2,48]

(7)

1,05[0,16; 2,23]

(10)

0,67[0,12; 2,02](12)

Journals which Recommend RG(N=44)

1,84[1; 2,6]

(31)

1,38[0,77; 2,44]

(39)

1,39[0,52; 2,53](44)

Journals which Impose RG(N=78)

1,54[0,90 ; 2,88]

(54)

1,33[0,63 ; 2,18](68)

1,31[0,53 ; 1,96](78)

Table 2. Distribution of IF’s from 2008, 2009 and 2010 for three groups by their utilization of Reporting Guidelines. IF’s haven’t got a normal distribution and data shows as median; [quartis 25 and 75] (N=absolute frequencies).

28

Abstract

IntroductionResearch

question & Aims

Results

Methodology

References

Results Reporting Guidelines’

Endorsement

Conclusions

Reporting Guideline’s

endorsement

Article’s Impact

Mean Standard Deviation

Number of articles

Journals which don’t say anything about RG

3,44 3,68 11

Journals which Recommend RG

3,98 2,38 12

Journals which Impose RG

4,74 4,18 12

The results aren’t statistical

significant(p=0,672)

Table 3. Distribution of the Articles’ impact in the three groups of journals.

29

Abstract

IntroductionResearch

question & Aims

Results

Methodology

References

Results

Conclusions

Levels ofAccordan

ce•Low accordance levelC+

•Normal accordance level

C++

•High accordance levelC+++

30

Abstract

IntroductionResearch

question & Aims

Results

Methodology

References

Results

Conclusions

Accordance

C+/C++

C+++ Total of articles

N 72,7%(8)

27,3%(3)

100%(11)

R 50%(6)

50%(6)

100%(12)

I 66,7%(8)

33,3%(4)

100%(12)

Rep

orti

ng

Gui

delin

e’s

endo

rsem

ent

The results aren’t statistical significant(p=0,501)

Table 4. Distribution of the Accordance of the articles with the Reporting Guideline used in the three groups of journals. N: Journals that say nothing about Reporting Guigelines; R: Journals that Recommend the use of Reporting Guidelines; I: Journals that Impose the use of Reporting Guidelines.

31

Abstract

IntroductionResearch

question & Aims

Results

Methodology

References

ResultsCONSORT

Conclusions

Levels of Accordance

Median IC (95%) Number of articles

C+ 2,55 [0,28; 5,68] 6

C++ 2,26 [-0,02; 5,88] 6

C+++ 3,16 [0,62; 6,16] 5

The results aren’t statistical significant(p=0,884)

Table 5. Ainda não sei como legendar esta.

32

Abstract

IntroductionResearch

question & Aims

Results

Methodology

References

ResultsSTROBE

Conclusions

Levels of Accordance

Median IC (95%) Number of articles

C+ 1,62 [0,13; 3,96] 4

C++ 3,77 [2,26; 5,77] 6

C+++ 5,98 [3,05; 11,41] 8

The results are statistical significant(p=0,041)

Table 6. Ainda não sei como legendar esta.

33

Abstract

IntroductionResearch

question & Aims

Conclusions

Methodology

References

Conclusions

Utilization rate of RG

Article’s impact and Recommendations of the journal

Article’s impact and Accordance

Results

We recommend…

34

[1] Bennett C, Khangura S, Brehaut JC, Graham ID, Moher D, Potter BK, Grimshaw JM. Reporting Guidelines for Survey Research: An Analysis of Published Guidance and Reporting Practices. PLoS Med. 2011; 8(8)

[2] Vandenbroucke JP. STREGA, STROBE, STARD, SQUIRE, MOOSE, PRISMA, GNOSIS, TREND, ORION, COREQ, QUOROM, REMARK. and CONSORT: for whom does the guideline toll? Journal of Clinical Epidemiology. 2009; 62: 594-596

[3] Simera I, Altman DG, Moher D, Schulz KF, Hoey J. Guidelines for Reporting Health Research: The EQUATOR Network’s Survey of Guideline Authors. PLoS Med. 2008; 5(6)

[4] du Sert NP. Improving the reporting of animal research: when will we ARRIVE? Disease Models & Mechanisms. 2011; 4: 281-282

[5] Cobo E, Cortés J, Ribera JM, Cardellach F, Selva-O’Callaghan A, Kostov B, García L, Cirugeda L, Altman DG, González JA, Sànchez JA, Miras F, Urrutia A, Fonollosa V, Rey-Joly C, Vilardell M. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomized trial. BMJ, 2011 [6] Larson EL, Cortazal M. Publication guidelines need widespread adoption. Journal of Clinical Epidemiology. 2011;  [7] Moschetti I, Brandt D, Perera R, Clarke M, Heneghan C. Adequacy of reporting monitoring regimens of risk factors for cardiovascular disease in clinical guidelines: systematic review. BMJ. 2011; 342 [8] Wangler M, Fujikawa R, Hestbæk L, Michielsen T, Raven TJ, Thiel HW, Zaugg B. Creating European guidelines for Chiropractic Incident Reporting and Learning Systems (CIRLS): relevance and structure. Chiropractic & Manual Therapies. 2011; 19(9)

ReferencesAbstract

IntroductionResearch

question & Aims

References

Results

Methodology

35

[9] Moher D, Schulz KF, Simera I, Altman DG. Guidance for Developers of Health Research Reporting Guidelines. PLoS Med. 2010; 7(2) [10] Moher D, Weeks L, Ocampo M, Seely D, Sampson M, Altman DG, Schulz KF, Miller D, Simera I, Grimshaw J, Hoey J. Describing reporting guidelines for health research: a systematic review. Journal of Clinical Epidemiology. 2011; 64:718-742 [11] Grol R, Dalhuijsen J, Thomas S, in’t Veld C, Rutten G, Mokkink H. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998; 317: 858–61 [12] Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gaboury I. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. MJA. 2006; 185: 263–267 [13] Wynne KE, Simpson BJ, Berman L, Rangel SJ, Grosfeld JL, Moss RL. Results of a longitudinal study of rigorous manuscript submission guidelines designed to improve the quality of clinical research reporting in a peer-reviewed surgical journal. Journal of Pediatric Surgery. 2011; 46: 131–137

ReferencesAbstract

IntroductionResearch

question & Aims

References

Results

Methodology

Do you have any questions?

36