critical appraisal or making reading more worthwhile
TRANSCRIPT
Critical Appraisal
Or Making Reading More Worthwhile
August 2001 Bruce Davies 2
The Problem
Vast and expanding literature.
Limited time to read.
Different reasons to read – mean different strategies.
Keeping up to date.
Answering specific clinical questions.
Pursuing a research interest.
August 2001 Bruce Davies 3
Stages
Clarify your reasons for reading.
Specify your information need.
Identify relevant literature.
Critically appraise what you read.
August 2001 Bruce Davies 4
Clarify Your Reasons for Reading.
Keeping up to date.Skimming the main journals and summary bulletins.
Answering specific clinical questions.Finding good quality literature on subject.
Pursuing a research interest.Extensive literature searching.
August 2001 Bruce Davies 5
Specify Your Information Need.
What kind of reports do I want?
How much detail do I need?
How comprehensive do I need to be?
How far back should I search?
The answers to these questions should flow from the reasons for reading.
August 2001 Bruce Davies 6
Identify Relevant Literature.
There are many ways of finding literature.
Remember to ask a librarian – they are the experts.
Selectivity is the key to successful critical appraisal.
August 2001 Bruce Davies 7
Critically Appraise What You Read.
Separating the wheat from the chaff.
Time is limited – you should aim to quickly stop reading the dross.
Others contain useful information mixed with rubbish.
Simple checklists enable the useful information to be identified.
August 2001 Bruce Davies 8
Questions to Ask
Is it of interest?
Why was it done?
How was it done?
What has been found?
What are the implications?
What else is of interest?
August 2001 Bruce Davies 9
Questions to Ask
Is it of interest?Title, abstract, source.
Why was it done?Introduction.
Should end with a clear statement of the purpose of the study.
The absence of such a statement can imply that the authors had no clear idea of what they were trying to find out.
Or they didn’t find anything but wanted to publish!
August 2001 Bruce Davies 10
Questions to Ask
How was it done?Methods.
Brief but should include enough detail to enable one to judge quality.
Must include who was studied and how they were recruited.
Basic demographics must be there.
An important guide to the quality of the paper.
August 2001 Bruce Davies 11
Questions to Ask
What has it found?Results.
The data should be there – not just statistics.Are the aims in the introduction addressed in the results?Look for illogical sequences, bland statements of results.? Flaws and inconsistencies.All research has some flaws – this is not nit picking, the impact of the flaws need to assessed.
August 2001 Bruce Davies 12
Questions to Ask
What are the implications?Abstract / discussion.
The whole use of research is how far the results can be generalised.
All authors will tend to think their work is more important than the rest of us!
What is new here?
What does it mean for health care?
Is it relevant to my patients?
August 2001 Bruce Davies 13
Questions to Ask
What else is of interest?Introduction / discussion.
Useful references?
Important or novel ideas?
Even if the results are discounted it doesn’t mean there is nothing of value.
August 2001 Bruce Davies 14
NOT JUST A FAULT FINDING EXERCISE
!
August 2001 Bruce Davies 15
What Is the Method?
The first task – alternative check lists for different methods.
How was the study conducted to confirm the method?
Authors sometimes use the wrong words to describe their work!
August 2001 Bruce Davies 16
Surveys
Describe how things are now.
Samples of populations or special groups.
The samples must be randomly selected.
August 2001 Bruce Davies 17
Surveys
Do not have separate control or comparison groups.
Comparisons may be made between subgroups – but this is not control.
Use of the term survey should identify the method – beware the use in what is really a cohort study.
August 2001 Bruce Davies 18
Surveys
Cross-sectional is seldom used to describe other methods.
There are many ways of selecting a sample: e.g: stratified, cluster and systematic.
August 2001 Bruce Davies 19
Cohort Studies
Used to find out what happens to patients.
A group is identified and then watched to see what events befall them.
May have comparison or control groups – who must be identified from the start.
Not an essential feature tho.’
August 2001 Bruce Davies 20
Cohort Studies
Must have the element of time flowing forward, from the point at which they are identified.
This is sometimes called a retrospective cohort study.
The term cohort should be diagnostic.
August 2001 Bruce Davies 21
Clinical Trials
Testing.Always concerned with effectiveness.The focus should always be on the outcome.
The outcomes may not be beneficial – in other words side effect trials.
Sometimes cohort trials are used to assess effectiveness. This is very poor research and can usually be dismissed.
August 2001 Bruce Davies 22
Clinical Trials
When more than two things are compared it makes the study more complex and harder to get right.The key words to look for are: random allocation, double blind, single blind, placebo-controlled.The term outcome is sometimes used in cohort studies as well.
August 2001 Bruce Davies 23
Case-control Studies
Ask what makes groups of patients different.
Select a set of patients with a characteristic – eg a disease.
The characteristics of this set are then compared with a control group who do not have the characteristic being studied – but all all other respects must be as identical as possible.
August 2001 Bruce Davies 24
Case-control Studies
Case studies look backward – not forward as cohort studies do.
Other terms used are: case-referrent, case-comparator, case-comparison.
May also be called retrospective – as is used for some cohort studies.
August 2001 Bruce Davies 25
The Results
The major mental challenge.What do I think this really means?CAUTION.
Large unexpected results are rare.Flawed studies and misleading findings are common.
August 2001 Bruce Davies 26
Statistics
A subject in itself.
Some general thoughts are worth emphasising.
Size matters.
August 2001 Bruce Davies 27
Probability
Is just that.
It is not proof.
Think of horse racing and the lottery.
Think of the odds.
August 2001 Bruce Davies 28
Pitfalls
All statistical tests make assumptions about the raw data.
If there is no raw data presented you cannot know if the tests are meaningful.
Outliers.
August 2001 Bruce Davies 29
Pitfalls
Skew.
Non-independence.
Serendipity masquerading as hypothesis – or data trawling!
August 2001 Bruce Davies 30
Pitfalls
Black box analyses. Modern computers make statistical testing easy – the authors may not know what they are doing!Bias – play devils advocate.Confounding.
A very common problem in medicine.Colour televisions do not cause increases in hypertension.
August 2001 Bruce Davies 31
Checklists
Checklists for particular types of literature are a quick and easy way of learning critical appraisal.They all have 3 stages:
Basic questions.Essential appraisal.Detailed appraisal.
August 2001 Bruce Davies 32
BUT, BUT, BUT
Checklists do not tell you about the quality or usefulness.
This is still a subjective question.
All the lists do is enable a more structured and thoughtful response to the subjective question.
August 2001 Bruce Davies 33
References
The Pocket guide to critical appraisal.By: Iain Crombie.
Pub BMJ Books, 1996.
How to lie with statistics.By: Darrell Huff.
Pub: Pelican, 1989.