critical appraisal
DESCRIPTION
Critical AppraisalTRANSCRIPT
CRITICAL APPRAISAL
AP DR JEMAIMA CHE HAMZAHMD (UKM) MS (OFTAL) UKM PHD (UK)
DEPARTMENT OF OPHTHALMOLOGYUKM MEDICAL CENTRE
MINGGU PENYELIDIKAN PERUBATAN & KESIHATAN
Lecture content
Introduction
What is critical appraisal?
Steps of critical appraisal
Introduction
Continuous medical education is essential part in a doctor’s life Acquiring up-to-date knowledge
and skills for clinical practice Providing patients with the best
possible standard of care Revalidation and appraisal of
competency
However………
Information overload Number of new research articles published
each year are continually increasing > than 12,000 new articles per year + 300 RCTs are added to the MEDLINE database
each week
Evidence-based medicine
Findings of scientific research should be apply to individual patients as part of a doctor’s clinical decision-making process
Therefore, doctors must be able to: Select and appraise scientific literature that
is relevant to their field Understand the implications of research
findings for individual patients Elicit patients' own preferences Develop an appropriate management plan
based on the combination of this information
Bad science
In June 2008, the Sunday Express published an article about the link between suicides and phone masts
The increase in deaths among young people in Britain’s suicide capital could be linked to radio waves from dozens of mobile phone transmitter masts near the victims’ homes.
Dr Roger Coghill, who sits on a Government advisory committee on mobile radiation, has discovered that all youngsters who have killed themselves in Bridgend, South Wales, over the past 18 months lived far closer than average to a mast (Johnston 2008)
Ben Goldacre, a medical doctor and author of the weekly Bad Science column in the Guardian, investigated the claim made by the Sunday Express article and found out the following:
I contacted Dr Coghill, since his work is now a matter of great public concern, and it is vital his evidence can be properly assessed. He was unable to give me the data. No paper has been published. He himself would not describe the work as a “study”. There are no statistics presented on it, and I cannot see the raw figures. In fact Dr Coghill tells me he has lost the figures. Despite its potentially massive public health importance, Dr Coghill is sadly unable to make his material assessable. (Goldacre 2008)
Behind the headlines
Some news are genuinely based on valid studies, but jump to wrong conclusions by failing to consider some important aspects, such as the study design and the level of evidence of the original research.
In July 2008, an article was published on the Daily Mail claiming that there is a link between vegetarian diet and infertility (Daily Mail Reporter 2008).
The article was based on a cross-sectional study on soy food intake and semen quality published in the medical journal Human Reproduction (Chavarro et al. 2008)
Behind the Headlines, A study reported that eating tofu can significantly lower your sperm count
Limitation of the study: it was small, and mainly looked at overweight or obese men who had presented to a fertility clinic. It focused only on soy (soya) intake,
Daily Mail’s claim that there is a causal link between eating a ‘vegetarian diet’ and reduced fertility is misleading. (NHS Knowledge Service 2008)
Bias in the location and selection of studies
If a study had not obtained positive results, would it have been published - and quoted in the news?
When reviewing the literature published in scientific/medical journals, we should consider that papers with significant positive results are more likely to submitted and accepted for publication (publication bias); published in a major journal written in English (Tower of Babel bias); published in a journal indexed in a literature database, especially in less
developed countries (database bias); cited by other authors (citation bias); published repeatedly (multiple publication bias); … and quoted by newspapers!
(Egger & Smith 1998; Gregoire, Derderian, & Le Lorier 1995)
Hierarchy of evidence
Lecture content
Introduction
What is critical appraisal?
Steps of critical appraisal
Critical appraisal
Defined as "...application of rules of evidence to a study to assess the validity of the data, completeness of reporting, methods and procedures, conclusions, compliance with ethical standards, etc. The rules of evidence vary with circumstances.“
Last JE (Ed.; 2001) A Dictionary of Epidemiology (4th Edn). New York: Oxford University Press
Con’t
A systematic process used to identify the strengths and weaknesses of a research article whether to use the results of a study in clinical practice
No 'gold-standard' instrument for critical appraisal
Criteria are not static Different study designs are prone to various sources of
systematic bias Increased awareness of the potential influence of other
non methodological factors, such as conflicts of interest
Lecture content
Introduction
What is critical appraisal?
Steps of critical appraisal
Ten steps of critical appraisal
Is the study question relevant? Does the study add anything new? What type of research question is being asked? Was the study design appropriate for the
research question? Did the study methods address the most
important potential sources of bias? Was the study performed according to the
original protocol? Does the study test a stated hypothesis? Were the statistical analyses performed
correctly? Do the data justify the conclusions? Are there any conflicts of interest?
Is the study’s research Q relevant?
Relevant to the researcher’s own field of work Addresses an important topic and adds to what
is already known about that subject
Subjective It might be crucial to some, but irrelevant to
others
Does the study add anything new?
New ideas and knowledge are developed on the basis of previous work ‘Standing on the shoulders of giants’
Two types of work: Study that make a substantive new contribution to
knowledge - rare Study that makes an incremental advance can also be of
value E.g. a study might increase confidence in the validity of
previous research by replicating its findings, or might enhance the ability to generalize a study by extending the original research findings to a new population of patients or clinical context
What type of research Q do the study pose?
The most fundamental task Identifying the specific research question that
an article addresses - this will determine the optimal study design and have a major bearing on the importance and relevance of the findings
Well-developed research question usually identifies 4 components: Population Intervention Comparator Outcomes
Con’t
Clinical research questions fall into 2 categories:1. Questions about the effectiveness of treatment
Relate to whether one treatment is better than another in terms of clinical effectiveness (benefit and harm) or cost-effectiveness
2. Questions about the frequency of eventsRefer to the incidence or prevalence of disease or other clinical phenomena, risk factors, diagnosis, prognosis or prediction of specific clinical outcomes and investigations on the quality of health care
Was the study design appropriate for the research question?
Study design Study design
Qualitative
Document
Passive observati
onParticipan
t observati
on
In depth interview
Focus group
Quantitative
Clinical trial
Randomised
Non randomis
ed
Observational
Cohort
Case-control
Cross sectional
Cross sectional
Measures prevalence of health outcomes/determinants of health/both in a population at a point in time
Strengths Allows comparison of
many different variables at the same time
Limitation Unable to provide definite
information about cause-and-effect relationships
Case control
Strengths Relatively economical and quick. Ideal for studying rare diseases or
diseases that can be defined specifically
Able to examine the association between multiple exposures and the disease of interest
Limitation Unable to examine multiple
diseases that might be associated with an exposure
Selection of appropriate controls can be challenging
Recall bias Cannot be used absolute risk for
disease after an exposure (uses OR)
Cohort
A longitudinal study (prospective or retrospective)
Strengths For examining disease after rare and
highly specific exposures Able to monitor the occurrence
of multiple diseases potentially caused by an exposure
Allows direct measurement of the absolute risk of developing a disease after an exposure
Limitation Selection bias and confounding factor Can only examine a single exposure or
group of related exposures Cannot be used if the population at risk
is not known
Randomised controlled trial (RCT) Gold standard of study design
Most rigorous way of: determining whether a cause-
effect relation exists between treatment and outcome
for assessing the cost effectiveness of a treatment
Limitations: Ethical and practical concerns More costly and time
consuming than other studies
Sequence generation - a rule to allocate
interventions to participants
Allocation concealment – secure implementation of the
schedule of random assignment
Blin
din
g
Did the study methods address the key potential sources of bias?
Bias – deviation from the truth Can be attributed to:
chance (e.g. a random error) affect the precision of the study
does not influence the results in any particular direction
to the study methods (systematic bias) e.g. how the study is conducted, be it how study participants
were selected, how data was collected, or through the
researchers' analysis or interpretation.
results in the overestimation or underestimation of the 'truth’
has a direction
Type of bias
Type of bias Description
Selection bias Systematic differences between baseline characteristics of the groups that are compared.
Performance bias Systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest.
Detection bias Systematic differences between groups in how outcomes are determined.
Attrition bias Systematic differences between groups in withdrawals from a study.
Reporting bias Systematic differences between reported and unreported findings.
**Different study designs are prone to varying sources of systematic bias
Eg. In RCT:
Was the study performed in line with the original protocol?
Deviations from the planned protocol can affect the validity or relevance of a study
Failure to recruit the planned number of participants Sampling bias - those who actually were recruited might
be different from those who weren't for some reason Reduces the power of the study
changes to the inclusion and exclusion criteria Variation in the provided treatments or interventions changes to the employed techniques or technologies Changes to the duration of follow-up
Does the study test has a stated hypothesis?
Study hypotheses must be identified a priori Developed from theory or previous experience
Post-hoc analysis produces high false positive findings 1 in 20 associations tested will be significant
(positive) by chance alone at significance level of 5% (P = 0.05)
Con’t
Sample size depends on hypothesis Whether it is a equivalence or noninferiority
trial vs. a superiority trial
Check all data relevant to the stated study objectives have been reported, and that selected outcomes have not been omitted
Were the statistical analyses performed correctly?
Tools used in the statistical analysis and the rationale for this approach should be included in the 'Method' section Approach to dealing with missing data and
the statistical techniques applied
In the 'Results' section, Defaulted or lost in follow-up and missing
data should be clearly identified
Do the data justify the conclusions?
Whether the conclusions are based on the accumulated data Overemphasis - Statistically significant findings are too small to be of
clinical value
Under emphasis - Dismissing large and potentially important
differences between groups that are not statistically significant,
because of small sample size
Generalisation of findings to broader groups of patients or
contexts than was reasonable given in their study sample
Whether statistically significant associations have been
misinterpreted to imply a cause and effect
Are there any conflicts of interest?
Conflicts of interest occur when personal factors have the potential to influence professional roles or responsibilities Decision such as which studies will be conducted in their unit, which
patients will be invited to participate in a study and whether certain clinical occurrences should be reported as adverse events affects validity of study
These decisions require researchers to act with integrity and not for personal or institutional gain
Check for declaration about the source of funding for the study and, if a potential conflict of interest had been identified for a statement about how this conflict was managed.
Judge whether the declared factors are important and might have influenced the validity of the study's findings
Evidence based medicine
Formulate your questions into a format according to PICO tool (Population, Intervention, Comparison, Outcome) to easily do literature review
Literature review - if you can find a pre-appraised resource (e.g. systematic review), you can miss out the next step
Critical appraisal of your results
Decide what action to take from your findings
Evaluate your new or amended practice
Conclusion
Critical appraisal is a systematic process through which the
strengths and weaknesses of a research study can be identified
This process enables us to assess the study's usefulness and
whether its findings are trustworthy
The most important component of critical appraisal is careful
assessment of the study design (Don’t forget evaluation of the
statistical methods used, interpretation of the findings and
potential conflicts of interest)
Consideration of the importance of the research to our own
patients will help us identify the most relevant, high-quality
studies available to guide in our clinical practice
Any Questions? Contact [email protected]