Introduction to Critical appraisals of the medical literaturePartini Pudjiastuti, Sudigdo Sastroasmoro
Department of Child Health Medical School University of Indonesia
Evidence based medicine 5 steps
Formulate question
Efficiently track down bestavailableevidence
Critically review thevalidity and usefulnessof the evidence
Implement changes in clinical practice
Evaluate performance
>25,000 journals worldwide>2 million published articles per yearMany published articles have methodological (including statistical) flaws – even in most respected journalsNot all results can be applied due to many reasons, a.o. dissimilarities of study subjects with our patientsLimited time of physicians: focus on articles relevant to your clinical practice
Rationale
Errors and inappropriateness may occur in any part of medical research:
in choosing appropriate design in choosing population in selecting study subjects in the details of the designin intervention and measurements in use of statistical methods in analysis in interpreting statistical analysisand in writing research report
The effect of dietary habit on calcium level in pediatric patients with nephrotic
syndromeJ Agric Soc Sci 2006;2
• r = - 0.1213, p = 0.015• There was a significant inverse relationship
between the age and the total calcium level ……
IMPORTANT!!!
Statistical significance vs. clinical importance
Negligible clinical difference may be statistically very significant if the number of subjects >>>. e.g., difference in reduction of cholesterol level of 3 mg/dl, n1=n2 = 10,000; p = 0.00002
Large clinical difference may be statistically non- significant if the no of subjects <<<, e.g. 30% difference in cure rate, if n1 = n2 = 10, p = 0.74
R
300 mg/dl
300mg/dl
Standardtreatment
New treatment
Cholesterol level, mg/dl
t = df = 9998 p = 0.00002
200
197
Clinical
Statistical
Clinical importance vs. statistical significance
n=10000
n=10000
Cured Died
Standard Rx 0 10 (100%)
New Rx 3 7 (70%)
Fischer exact test: p = 0.211
Clinical importance vs. statistical significance
Absolute risk reduction = 30% Clinical
Statistical
Critical appraisal (Making Reading More
Worthwhile)
What is Critical Appraisal?1. Critical appraisal = quality assessment2. ….process of weighing up evidence to
see how useful it is in decision making3. .…a process of assessing the validity,
reliability and usefulness of evidence4. …..is about considering, evaluating and
interpreting information in a systematic and objective way
Critically Appraise What You Read.
• Separating the wheat from the chaff.
• Time is limited – you should aim to quickly stop reading the dross.
• Others contain useful information mixed with rubbish.
• Simple checklists enable the useful information to be identified.
Critical Appraisal – Critical Thinking
• Appraising the available evidence to construct clinical reasoning and to make decisions
• Finding strengths and limitations of written ‘evidence’
• Deciding what evidence to pay attention to versus what to ignore
Why critically appraise?
• Supports sound decision making based on best available evidence
• Helps us determine (three R’s):•How rigorous a piece of research is - Validity
•What the results are telling us - Importance
•How relevant it is to our patient -Applicability
Value =
MortalityMorbidity
QoL
PatientSatisfacti
on
Health Status
Resources used
Quality
Cost
What is “Evidence”?
• People disagree on what constitutes “evidence”
• Evidence – generally = scientific fact
• Evidence - a combination of information obtained from 3 sources: research,
clinical experience, and client preferences (Kitson, Harvey, & McCormack, 1998)
Why do we need evidence?
– Resources should be allocated to things that are EFFECTIVE
– The only way of judging effectiveness is EVIDENCE
– “In God we trust – all others need evidence”
Sources of Evidence
• Primary sources– Based on experiments and published
research• Secondary sources
– Systematic reviews– Clinical guidelines– Journals of secondary publication
e.g. Evidence Based Medicine
Levels of evidence
1. Syst reviews of RCTs and high quality RCTs
2. Syst reviews of cohort studies, lower quality RCTs, outcomes research
3. Syst reviews of case controls, case control studies
4. Case Series
5. Expert opinion www.cebm.net
Types of EvidenceQuestion Types
Type of Question Best EvidenceHealth care interventions: treatment, prevention
Quantitative: Systematic Review of RCTs or RCT
Harm or Etiology Quantitative: Observational Study - Cohort or Case Control
Prognosis Quantitative: Observational Study - Cohort, Case Control
Diagnosis or Assessment Quantitative: Comparison to Gold Standard
Economics Quantitative: Cost-effectiveness Study
Meaning Qualitative: case study, ethnography, grounded theory, phenomenologic approach
Key quality parameters
• Validity
• Reliability
• Importance
Validity
• InternalIs the study designed in such a way that I Can trust the findings?
• ExternalIs the study designed in such a way that ICan generalize the findings?
Reliability
If the study was conducted again,would the results be the same?
Usually interpreted as the accuracyof measurement.
Importance
What was the effect sizeor magnitude of effect?
Clinical vs. statistical significance.
Tools for Critical Appraisal
• Are the results valid?
• What are the results?
• Will the results help me in patient care?
EBM “simplified” approach:
V
I
A
Format of research reports
• Title• Authors and Institutions• Abstract & keywords• IMRAD
– Introduction - why did I start?– Methods - what did I do?– Results - what did I find?– Discussion - what does it mean?
• References
Check list for medical literature
1. Title2. Authors & Institutions3. Abstract: Structured? Informative? Abbreviation?4. Introduction: Length? Relevant ref? Objective?5. Methods:
– Design, time and place– Inclusion criteria – Exclusion criteria– Sample size, sampling method– Randomization technique– Intervention: masking?– Outcome measurement: blinding?
• Primary outcome: type of variable • Secondary outcome: type of variable
– Analysis: Clinical, statistical
6. Results– Baseline characteristics– Main outcome– Secondary outcome
7. Discussion– General– Strength and weakness– Conclusions
8. References– Vancouver style– Constant
9. Acknowledgments10. Ethics approval11. Conflict of interest
Check list for medical literature
What to assess?(in study of cause-effect
relationship)
A. General description• Type of design• Target population, source
population, sample• Sampling method• Dependent and independent
variables• Main results?
B. Internal validity, non-causal relationship– Influence of bias– Influence of chance– Influence of confounders
What to assess?(in study of cause-effect
relationship)
Bias
What is a bias? A process that tends to produce results that depart systematically from the true values existing in the study population
Types of bias1. Sample (subject selection) biases, which may
result in the subjects in the sample being unrepresentative of the population which you are interested in
2. Measurement (detection) biases, which include issues related to how the outcome of interest was measured
3. Intervention (performance) biases, which involve how the treatment itself was carried out.
C. Internal validity, causal relationship• Temporality (cause precedes effect)• Strength of association (large difference, RR, OR, etc)
or small p value or narrow confidence interval• Biological gradient (dose dependence)• Consistency among studies (diff. populations or
designs)• Specificity (certain factor results in certain effect)• Coherence (does not conflict with current knowledge)• Biological plausibility: can be explained with current
knowledge (at least in part)
What to assess?(in study of cause-effect
relationship)
D. External validity
– Applicable to study subjects– Applicable to source population– Applicable to target population
What to assess?(in study of cause-effect
relationship)
11 items, each with 3 sections
1. Can you find this information in the paper?
2. Is the way this was done a problem?
3. Does this problem threaten the validity of the study?
11 items1. What is the research question?2. What is the study type?3. What are the outcome factors and how are they measured?4. What are the study factors and how are they measured?5. What important confounders are considered?6. What are the sampling frame and sampling method?7. In an experimental study, how were the subjects assigned
to groups? In a longitudinal study, how many reached final follow-up? In a case control study, are the controls appropriate? (Etc)
8. Are statistical tests considered?9. Are the results clinically/socially significant?10. Is the study ethical? 11. What conclusions did the authors reach about the
question?
1. What is the research question?
• (Is the way this was done a problem?)
– Is it concerned with the impact of an intervention, causality or determining the magnitude of a health problem?
• (Does this problem threaten the validity of the study?)
– Is it a well stated research question/hypothesis?
2. What is the study type?
• (Is the way this was done a problem?)
– Is the study type appropriate to the research question?
• (Does this problem threaten the validity of the study?)
– If not, how useful are the results produced by this type of study?
3. What are the outcome factors and how are they measured?
• (Is the way this was done a problem?)– a) are all relevant outcomes assessed– b) is there measurement error?
• (Does this problem threaten the validity of the study?)– a) how important are omitted
outcomes– b) is measurement error an important
source of bias?
4. What are the study factors and how are the measured?
• (Is the way this was done a problem?)– Is there measurement error?
• (Does this problem threaten the validity of the study?)– Is measurement error an important
source of bias?
5. What important potential confounders are considered?
• (Is the way this was done a problem?)– Are potential confounders examined and
controlled for?
• (Does this problem threaten the validity of the study?)– Is confounding an important source of bias?
6. What are the sampling frame and sampling method?
• (Is the way this was done a problem?)– Is there selection bias?
• (Does this problem threaten the validity of the study?)– Does this threaten the external
validity of the study?
7. Questions of internal validity
• (Is the way this was done a problem?)– In an experimental study, how were the
subjects assigned to groups?– In a longitudinal study, how many
reached follow-up?– In a case control study, are the controls
appropriate?• of the study?
8. Are statistical tests considered?
• (Is the way this was done a problem?)– Were the tests appropriate for the
data?– Are confidence intervals given?– Is the power given if a null result?– In a trial, are results presented as
absolute risk reduction as well as relative risk reduction?
• How useful are the results?
9. Are the results clinically/socially significant?
• (Is the way this was done a problem?)– Was the sample size adequate to detect
a clinically/socially significant result?– Are the results presented in a way to
help in health policy decisions?• (Does this problem threaten the validity of
the study?)– Is the study useful?
10. Are ethical issues considered?
• (Is the way this was done a problem?)– Does the paper indicate ethics approval?– Can you identify potential ethical issues?
• (Does this problem threaten the validity of the study?)– Are the results or their application
compromised?
11. What conclusions did the authors reach about the study
question?
• (Is the way this was done a problem?)– Do the results apply to the population in
which you are interested?
• (Does this problem threaten the validity of the study?)– Will you use the results of the study?
Appraisal Tools
• Tools from the Critical Appraisal Skills Programme (CASP)– Systematic Reviews– Randomised Controlled Trials– Qualitative Research Studies– Cohort Studies– Case-Control Studies– Diagnostic Test Studies– Economic Evaluation StudiesAvailable at
http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm
Study Designs Recap
Effectiveness of Therapy
Risk Factors / Prognosis
Diagnosis
Attitudes & Beliefs
Randomised Controlled Trial
Cohort Study
Survey using Gold Standard
Qualitative (Interviews, Observations, etc)
Critical appraisal
- Valid- Important- Applicable
Methods Results Discussion
50 massa pemuda dan mahasiswa berunjuk rasa memprotes pelecehan seksual yang makin marak di depan Mabes PolriMenyadari pentingnya Panduan Pelayanan Medis (PPM), dibentuklah Panitia Penyusunan SPM di RSCM
Sekuens dan hubungan subyek-predikat
• Dalam pertemuan ilmiah yang diselenggarakan setiap tahun yang merupakan ajang untuk menyajikan perkembangan mutakhir dalam bidang ilmu penyakit dalam di tanah air tersebut menyimpulkan bahwa pertemuan tersebut disamping dilakukan oleh Universitas selayaknya juga dilakukan oleh cabang-cabang PAPDI di setiap propinsi, bahkan kalau mungkin disetiap kabupaten
Plagiarisme adalah tindakan yang dapat diartikan sebagai pencurian ide atau hasil pemikiran dan tulisan orang lain yang digunakan dalam tulisan seolah-olah ide atau tulisan orang lain tersebut adalah ide atau hasil tulisannya sendiri untuk keuntungannya sendiri sehingga merugikan orang lain baik materiil maupun non-materiil, atau plagiarisme dapat berupa pencurian sebuah kata, frase, kalimat, atau alinea, atau bahkan pencurian suatu bab dari sebuah tulisan atau buku seseorang, tanpa menyebut sumber yang dicuri. (Draft SK Rektor UI)
RCT (Pragmatic trials): Validity Were the study participants randomized?Was the randomization technique described?Was the randomization table concealed?Were the characteristics of the subjects similar at the start of the intervention?Were all participants given equal treatment apart from the intervention?Were all relevant outcomes considered?Were the results analyzed correctly?
RCT (Pragmatic trials): Importance
• Calculate: EER, CER, RRR, ARR, NNT
a b
c d
E
C
Success Failure
EER = a/(a+b)CER = c/(c+d)
RRR = (CER-EER)/CERARR = CER-EERNNT = 1/ARR
RCT: Applicability• Were the participations similar to your
patients?• May be intuitively concluded or use f
(factor indicating how much severe your patient compared to the study participation in terms of prognostic factor)
DIAGNOSTIC TEST: Validity
Was independent and blind comparison to gold standard applied?
Was the diagnostic test include spectrum of disease similar to your real practice?
Was the gold standard applied regardless of the diagnostic result?
Diagnostic Test: Importance
• Calculate: Sensitivity, specificity, predicitive values, likelihood ratios
a b
c d
+
-
+ -
Se = a/(a+c)Sp = d/(b+d)PPV = a/(a+b)NPV = d/(c+d)
LR+ = se/(1-sp)LR - = (1-se) /spPosttest odds = Pretest odds x LR+
Test
Diagnostic Test: Applicability
Were the participations similar to your patients?Is the diagnostic test applicable, acceptable, and affordable in your setting?Will the result of the test help your patient?
Cohort Studies: Validity
Was the inception cohort assembled in usual point of course of the disease?Was the follow-up sufficient & complete?Were outcome criteria applied in blind fashion?Was there any validation in other group of patients?Was subgroup analysis performed after adjustment for prognostic factors?
Cohort studies: Importance
Calculate: Relative risk (RR) and 95% CI
a b
c d
+
-
+ -
Exposure
RR = a/(a+b) : c/(c+d)RR > 1: risk factorRR < 1 : protective factorRR = 1 : not a risk factor
Cohort studies: Applicability
Were the patients similar to yours?Wil the evidence make a clinically important impact for the care of your patient?
Case control study: Validity
Was the selection of patients represent the common patients of yours? (Spectrum of the patients)Was the selection of control group sensible and reasonable?Have sufficient attempts been made to minimize recall bias?
Case control studies: Importance
• Calculate: Odds ratio (OR) and 95%CI
a b
c d
+
-
+ -
Exposure
OR = a/b : c/dOR > 1 : risk factorOR < 1 : protective factorOR = 1 : not a risk factor
Case-control studies: Applicability
• Were the patients similar to yours?• Wil the evidence make a clinically
important impact for overall care (diagnosis, treatment, prognosis) of your future patients?b
From Data to Wisdom
• Data are what researchers collect• Information results when data is
analyzed and interpreted (EVIDENCE)• Knowledge results when information
is shared, acquired, and used• Wisdom is the ability to make the
right use of knowledge
THANKS