association between quality of clinical practice guidelines and citations given to their references
DESCRIPTION
It has been suggested that bibliometric analysis of different document types may reveal new aspects of research performance. In medical research a number of study types play different roles in the research process and it has been shown, that the evidence-level of study types is associated with varying citation rates. This study focuses on clinical practice guidelines, which are supposed to gather the highest evidence on a given topic to give the best possible recommendation for practitioners. The quality of clinical practice guidelines, measured using the AGREE score, is compared to the citations given to the references used in these guidelines, as it is hypothesised, that better guidelines are based on higher cited references. AGREE scores are gathered from reviews of clinical practice guidelines on a number of diseases and treatments. Their references are collected from Web of Science and citation counts are normalised using the item-oriented z-score and the PPtop-10% indicators. A positive correlation between both citation indicators and the AGREE score of clinical practice guidelines is found. Some potential confounding factors are identified. While confounding cannot be excluded, results indicate low likelihood for the identified confounders. The results provide a new perspective to and application of citation analysis.TRANSCRIPT
ASSOCIATION BETWEEN QUALITY OF CLINICAL PRACTICE GUIDELINES AND CITATIONS GIVEN TO THEIR REFERENCES
JENS PETER ANDERSENMEDICAL LIBRARY, AALBORG UNIVERSITY HOSPITALROYAL SCHOOL OF LIBRARY AND INFORMATION SCIENCE, UNIVERSITY OF COPENHAGEN
”In some ways bibliometrics is at the stage of European navigation in the middle ages. The
familiar territory is well, even obsessively, charted but beyond te known world there are only unknown dragons on the map:” Lewison,
2002, p. 191
SETTING SAIL…
Clinical practice guidelines (CPGs) as study object:
- Recommendations about best treatment and diagnosis for specific diseases.
- Varying quality of CPGs. They are supposed to be based on the best research evidence – not all are.
LET’S AGREE
The AGREE instrument assesses six groups of variables pertaining to the quality of CPGs.
The most elaborate group, A3, is labelled ”Rigour of development” and pertains to the ways in which evidence was sought, assessed and included in and from the literature.
RESEARCH QUESTION
Is there a connection between the A3-score of CPGs and the citations given to their references?
METHODS
A3 scores were collected from reviews of CPGs containing AGREE-scores.
CPGs were extracted from Web of Science where possible.
All references from CPGs were extracted.
All citation scores of articles published in the same year and journal as CPG references were extracted as control group.
METHODS II
Citations were normalised:
• PPtop10% indicator; delimited by journal.• Item-oriented z-score
MATERIALS
CPG Reviews14
CPGs
80
References
5,970
Control group
672,819
MATERIALS II
A1 A2 A3 A4 A5 A6 Acum
0.0
0.2
0.4
0.6
0.8
1.0
AGREE categories
AG
RE
E s
core
s
Included Excluded0
.00
.20
.40
.60
.81
.0Guideline category
A3
sco
re
RESULTS – CITATION DISTRIBUTION
1 5 10 50 100 500 5000
11
01
00
10
00
10
00
0
log(rank)
log
(cita
tion
co
un
t)
-5 0 5 10
0.0
0.2
0.4
0.6
0.8
1.0
Standard deviations from mean
De
nsi
ty
RESULTS II – MAIN FINDINGS
0.0 0.2 0.4 0.6 0.8
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
A3
Z
cor = 0.471, r 2̂ = 0.222
0.0 0.2 0.4 0.6 0.8
0.0
0.2
0.4
0.6
0.8
1.0
A3
PP
top
-10
%
cor = 0.457, r 2̂ = 0.209
RESULTS III – CONFOUNDERS?
0 100 200 300 400 500 600
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Number of references
Z
cor = 0.0305, r 2̂ = 0.000933
0 20 40 60 80
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
citations / year
Zcor = 0.239, r 2̂ = 0.0571
0 20 40 60 80
0.0
0.2
0.4
0.6
0.8
citations / year
A3
cor = 0.371, r 2̂ = 0.00036
0 100 200 300 400 500 600
0.0
0.2
0.4
0.6
0.8
1.0
Number of references
PP
top-
10%
cor = 0.019, r 2̂ = 0.0429
0 20 40 60 80
0.0
0.2
0.4
0.6
0.8
1.0
citations / year
PP
top-
10%
cor = 0.207, r 2̂ = 0.137
0 20 40 60 80
0.2
0.3
0.4
0.5
0.6
0.7
0.8
citations / year
Acu
m
cor = 0.317, r 2̂ = 0.101
RESULTS IV
A medium-strength correlation between A3 score and citedness of references.
No apparent confounding from reference list length or citedness of CPGs.
More data required.
Results are indicative of connections between citations, clinical evidence and health impact – but there is no evidence of causative mechanisms here.
DISCUSSION
Can references from other document types than journal articles broaden the impact concept?
If a study is cited by a CPG, is this a clinical impact, or policy impact – different from academic citation impact?
CPGs as mega-citations in specific contexts?
THANK YOU FOR YOUR ATTENTION