ASSOCIATION BETWEEN QUALITY OF CLINICAL PRACTICE GUIDELINES AND CITATIONS GIVEN TO THEIR REFERENCES

JENS PETER ANDERSENMEDICAL LIBRARY, AALBORG UNIVERSITY HOSPITALROYAL SCHOOL OF LIBRARY AND INFORMATION SCIENCE, UNIVERSITY OF COPENHAGEN

”In some ways bibliometrics is at the stage of European navigation in the middle ages. The

familiar territory is well, even obsessively, charted but beyond te known world there are only unknown dragons on the map:” Lewison,

2002, p. 191

SETTING SAIL…

Clinical practice guidelines (CPGs) as study object:

- Recommendations about best treatment and diagnosis for specific diseases.

- Varying quality of CPGs. They are supposed to be based on the best research evidence – not all are.

LET’S AGREE

The AGREE instrument assesses six groups of variables pertaining to the quality of CPGs.

The most elaborate group, A3, is labelled ”Rigour of development” and pertains to the ways in which evidence was sought, assessed and included in and from the literature.

RESEARCH QUESTION

Is there a connection between the A3-score of CPGs and the citations given to their references?

METHODS

A3 scores were collected from reviews of CPGs containing AGREE-scores.

CPGs were extracted from Web of Science where possible.

All references from CPGs were extracted.

All citation scores of articles published in the same year and journal as CPG references were extracted as control group.

METHODS II

Citations were normalised:

• PPtop10% indicator; delimited by journal.• Item-oriented z-score

MATERIALS

CPG Reviews14

CPGs

80

References

5,970

Control group

672,819

MATERIALS II

A1 A2 A3 A4 A5 A6 Acum

0.0

0.2

0.4

0.6

0.8

1.0

AGREE categories

AG

RE

E s

core

s

Included Excluded0

.00

.20

.40

.60

.81

.0Guideline category

A3

sco

re

RESULTS – CITATION DISTRIBUTION

1 5 10 50 100 500 5000

11

01

00

10

00

10

00

0

log(rank)

log

(cita

tion

co

un

t)

-5 0 5 10

0.0

0.2

0.4

0.6

0.8

1.0

Standard deviations from mean

De

nsi

ty

RESULTS II – MAIN FINDINGS

0.0 0.2 0.4 0.6 0.8

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

A3

Z

cor = 0.471, r 2̂ = 0.222

0.0 0.2 0.4 0.6 0.8

0.0

0.2

0.4

0.6

0.8

1.0

A3

PP

top

-10

%

cor = 0.457, r 2̂ = 0.209

RESULTS III – CONFOUNDERS?

0 100 200 300 400 500 600

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

Number of references

Z

cor = 0.0305, r 2̂ = 0.000933

0 20 40 60 80

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

citations / year

Zcor = 0.239, r 2̂ = 0.0571

0 20 40 60 80

0.0

0.2

0.4

0.6

0.8

citations / year

A3

cor = 0.371, r 2̂ = 0.00036

0 100 200 300 400 500 600

0.0

0.2

0.4

0.6

0.8

1.0

Number of references

PP

top-

10%

cor = 0.019, r 2̂ = 0.0429

0 20 40 60 80

0.0

0.2

0.4

0.6

0.8

1.0

citations / year

PP

top-

10%

cor = 0.207, r 2̂ = 0.137

0 20 40 60 80

0.2

0.3

0.4

0.5

0.6

0.7

0.8

citations / year

Acu

m

cor = 0.317, r 2̂ = 0.101

RESULTS IV

A medium-strength correlation between A3 score and citedness of references.

No apparent confounding from reference list length or citedness of CPGs.

More data required.

Results are indicative of connections between citations, clinical evidence and health impact – but there is no evidence of causative mechanisms here.

DISCUSSION

Can references from other document types than journal articles broaden the impact concept?

If a study is cited by a CPG, is this a clinical impact, or policy impact – different from academic citation impact?

CPGs as mega-citations in specific contexts?

THANK YOU FOR YOUR ATTENTION