evidence-based practice introduction to methods and searching for librarians ann mckibbon mls phd...

Evidence-Based PracticeIntroduction to methods and searching for Librarians

Ann McKibbon MLS PhDMcMaster University

mckib@mcmaster.ca

Rules for the day

• Have fun• Stretch your minds• Make clinical decisions• Ask questions and make

comments• Make mistakes and say

dumb things• Develop some new skills• Stop me for breaks

Morning Agenda

Introduction to EBMEBM and the questionEBM and searching/retrieval

–Categories of care–How are the similar–How are they different

EBM and the article–Types of articles–What makes a “good’ one

Agenda

How they differ?What makes them strong?How do I find them?

–Therapy–Diagnosis–Prognosis–Etiology/harm–Systematic reviews

Current Working (Pragmatic) Definition

…a set of tools, resources, and procedures

…for finding current best evidence from research

…and applying this evidence

…for decision making with respect tothe care of individual patients (EBM, EBN, EBD, EBHC)the care of groups of people (EBPublic Health)the running of your library (EBL)

raising your child (EBParenting)

…taking into account the situation, culture, resources, and common sense

Historical Roots of EBP

Daniel—first clinical trial with Shadrack, Meshack, and Abednigo

The results? At the end of 10 days, they "appeared better and fatter than all the young men who had been eating the royal rations" (v:15), and …the king found them in learning and wisdom to be "ten times better than all the magicians and enchanters in his whole kingdom" (vs:18-19).

Ibn Sina (981-1037)

Rules of drug evaluation• The drug must have a specific defined mode of

action• It must be tested on a well defined disease• The time of action must be observed• The effect of the drug must been seen to occur

consistently in many cases• The experimentation must be done on the

human body (horses or lions may react differently than a human)

Historical Roots of EBP

Lind—scurvy

Late 1700s12 sailorsDramatic resultsSlow implementation

Osler—InformationIt is astonishing with how little reading a doctor

can practice medicine, but it is not astonishing how badly he may do it.

Historical Roots of EBP

First randomized trial

--not enough drug in UK

--George Orwell not eligible for trial

EB Health Care—First Version

Patient

Evidence Clinical

Expertise

EB Health Care—New Version

Clinical Circumstances

Clinical Experience

Patient preferences Evidence

and actions

5 Steps of EBHC

• Framing the question appropriate to the needs of the patient

• Finding the evidence• Evaluating the evidence (critical

appraisal)• Making and doing the decision • Evaluation of the whole process

Critical appraisalThree-step process

• Find out how good (strong) the evidence is (assessment of the methods)

• Find out what the results are and how strong they are

• Figure out how our patient (or patient group) and settings matches with the study patients and setting

Critical appraisalGo to first page of supplemental

package

Pictorial representation of the EB process from a clinical perspective

Most of the steps are done intuitively and not on paper

Why our searching skills are important

New Awareness of the Importance of Evidence and Difficulties to

Find/AccumulateSystematic reviews and meta-analyses

– Presence of search strategies a measure of quality– Expert searching—have we lost our edge?– Seeing a true collaboration (work as equals)– Searching and data management

Cochrane and Campbell Collaborations, DARE– Librarians full members with own sections on methods

CADTH, AHRQ and Technology Assessment– As above (Jessie McGowan)

Guidelines and Care Maps– Librarians true partners (Ruth Holst)

Professional Societies hire librarians/contract for services– CFPC, CMA, ACOG, AAN

Need for new information tools and techniques

Books and book-like products– Clinical Evidence has much librarian input—

searching etc– PIER from ACP

Summary Journals– ACP Journal Club, EBM, EBN

Web pages and sites– MEDLINEPLUS and Go Local implementations– Provide sites and production

New products– OVID EBM Reviews and other aggregated services– PubMED Clinical Queries– BMJUpDates+

New services

New Respect for our Abilities/Collections

Grants– Preparation, editing of grants– Teaching of grant writing skills (Wessel & Pitt

modules)

Institutional Review Boards– Membership – Standards (Kate Oliver)

New service opportunities– Scherrer and publication process– Code Yellow—Library 911

Teaching– EBM workshops in Hamilton and Denver

Limited only by our – Imagination– Ability to recognize and create opportunities

Clinical Research

Move now to looking at clinical research…

…the kind of research that is strong enough and applicable enough for use in making/changing clinical decisions

Clinical Research

• Question based—once question formed• Methods

– Observation– Manipulation/experimental

• Which is stronger to do?• Which is easier to do?• Do we need both? • Why?

Observational or Experimental?

• What is the process that elderly people go through as they come to terms with living in an assisted living arrangement?

• Does yoga improve fatigue levels in people recovering from cancer?

• How effective is this appetite reducing drug in relation to exercise?

• Do suduku puzzles protect me from developing Alzheimer disease?

• What is my life expectancy now that you have told me I have ALS?

• Is this blood test as effective as stomach biopsy at telling me that I have celiac disease?

Publishing Wedge (therapy)

IdeaIdea developmentLaboratoryAnimalEarly human Phase IMiddle human Phase IILate human Phase III

Categories of clinical literature

Original studies• Therapy• Diagnosis• Prognosis• Etiology• Clinical prediction

guides• Differential diagnosis• Qualitative studies

Syntheses • Systematic reviews/

meta-analyses• Clinical practice

guidelines• Economic studies• Decision analyses

Similarities across categories

Done to answer legitimate and important problems and issues

Meet standards – Ethical– Moral – Record keeping and reporting– Publishing standards and procedures

Similarities across categories

• Question based• Preplanned• Comparative• Patients/participants• Results and statistics• Settings• Cultures/health care systems• Conflict of interest/disclosure statements• Funding sources

Clinical QuestionFor young children, is a smoke

detector/fire alarm that uses a recording of his or her mother’s voice more effective at awakening the child and shortening time to evacuation as compared to a standard tone-based smoke detector?

Settings

• Important for assessment of match between patient and article

• Most often primary care, tertiary care, hospital, office (UK surgery), nursing home, university medical center, chronic care facilities, home care

• Compare the “typical” headache seen in above settings

Cultures/health care systems

• For profit• Not for profit• Managed care• Health maintenance organizations• Medicare/Medicaid• Veterans Affairs Hospitals• Socialized medicine• Two tier vs. three tier

Health care systems—assignment

What health care system does each country have?

What “type” of research comes from each country?

• Canada• Israel• Sweden• United Kingdom• Australia• The Netherlands• United States

Funding Sources

• Complex and costly issue• Competing demands with “for profit”

funders vs. “not for profit” funders• Kjaergard and Als-Nilesen showed that

– “in pharmacological and nonpharmacological randomised controlled trials from 12 specialties financial competing interests were significantly associated with authors’ conclusions”

– “personal, academic and political competing interests were not significantly associated with authors’ conclusions”.

Conflict of Interest

Researchers or authors of the study or its report stand to gain (probably financially) from certain results of the study.

Watch for this. Should be included in the article—usually at the end and in a small font.

Differences

All articles same for a few features– Question based– Preplanned– Comparative– Patients/participants– Settings– Cultures/health care systems– Funding sources

Then different….

Therapy

Therapy/Treatment

• More of these types of studies than any other

• Methodology is pretty well established for conducting trials and presenting results

• Indexing (and retrieval) excellent

Alternate Names• Therapy/Treatment (medicine)• Intervention (nursing)• Prevention and control• Prevention

– Primary prevention– Secondary prevention– Tertiary prevention

• Quality improvement• Management???

How to do a Therapy Trial

•2 or more groups•Each group gets 1 intervention•All groups followed over time•At the end of the trial groups

compared to assess outcomes

Things to Look for in a Therapy Trial

Common senseAllocation concealmentRandom allocationBlindingFollow-upSensible and important outcomes

Allocation Concealment

Done before and during randomizationNo one with any influence on who goes

into which group can have any knowledge of which group is next

Differences among outcomes if this is not done—more favourable outcomes

Random Allocation

You can randomize:

• parts of people (e.g., arms, warts)• whole people• families• hospitals or wards (cluster)• towns

Methods for Random Allocation

Best• Computer methods that do not allow for

manipulation of randomization• An agency that has no involvement in patient

recruitment such as a pharmacy department • An external trials office that entails calling into a

central registration office and providing details of patient before the randomization is done

• Sealed, opaque, consecutively numbered envelopes with external checking

BlindingIndividuals involved in a study

(e.g., patients, investigators, research staff) do not know who is assigned to treatment or control groups.

Why: Individual expectations can influence study outcomes

Types of BlindingSingle, double, triple blinding although

many more could be listedMasked, dummy also used

– Patients– Care providers– Study personnel

• Data collectors• Outcome assessors

– Data analysts– Report writers– Sponsors

PlaceboTo aid in blinding some trial

participants may get something that is not really a treatment or for “real”– Sugar pill in vitamin C trials– Sham ultrasound in treating BPH and

both real and sham kept “equivalent” by heating pads placed over treatment sites

– Can be for assessors in addition to patient as in fake blood for patients in a scope vs full surgery trial

Follow-upConcerns the number of

participants who completed the study

Look for withdrawals, drop outs, or those who were lost

80% is magic number

Common sense

Clinical QuestionPain is a complex challenge at the best of

times and is especially difficult for children. A hospital committee has been formed that wants to look at alternatives to simply increasing the doses of pain meds. Someone has suggested that video games may distract kids with substantial burns from thinking about their pain. Does the following article support this assertion?

MeSH

Research Comparative studyClinical protocols Placebos*Feasibility studies Clinical trials as topicPilot projects Clinical trials as topic, I-IVResearch design* Multicenter studies

Double-blind method* Randomized controlled trials Meta-analysis as topic

Patient selection Treatment outcomeRandom allocation* Single-blind methodSample size

Publication TypesClinical trial

Clinical trial, phase IClinical trial, phase IIClinical trial, phase IIIClinical trial, phase IV

Randomized controlled trialControlled clinical trialMulticenter studyMeta-analysis

SubheadingsTherapy (explodable)SurgeryRadiotherapyDiet therapyPsychologyTherapeutic use (explodable)Administration and dosage

TextwordsRandom: Double blind:Double-dummy Mask:Sham: Placebo:Control: trial: EfficacyEffectiveness

Diagnosis and Screening

Disease/condition Present?

Diagnosis

This patient has signs and symptoms that suggest diabetes. Does she have it?

Signs and symptoms are present that warrant action.

Screening

We are going to check all students in this school to see if the head lice has spread from room 2.

No symptoms are present but because of the population we are going to assess all of them.

Diagnostic Decision

Three choices after assessing patient:

Do nothing for a while—I am not sure at all if the patient has a condition—low probability of a positive diagnosis

Treat right away—I am sure beyond any doubt that the patient has the condition—the diagnostic tests will give me no more information

Start doing diagnostic tests

Diagnostic Decision

Probability of disease

0% testing threshold treatment threshold 100%

Wait Test Treat

Diagnostic Decision

Wait Test TreatProbability of disease

0% testing threshold treatment threshold100%

Positive test results moves Negative test results moves

Diagnosis study—example

Problem: Is this incontinence urge or stress? One treated with drugs, other behaviour

Possible solution: questionnaire vs urologist?

Tested 301 women some incontinence.

Test results after every women got both75% of time positive when have incontinence77% of time negative when no incontinence

Diagnosis/ScreeningDoes this person have or not have a specific

disease or condition?

Can questionnaires in family medicine settings screen for:– eating disorders– Depression– domestic violence– Alzheimer disease – drinking problems

Old Test vs New Test

• Need patients to have spectrum of disease (none to severe)

• Everyone gets both tests• Old test is often invasive, time

consuming, costly, or has risks involved

• Can only do if “gold” test available or can be “rigged”

Things to look for in a diagnosis study

Old test vs new test

Blinding of assessment of results of both tests

Blinding of Test Reading

• Absolutely crucial for evaluation of diagnostic tests

• Exceptions are things like laboratory tests that do not involve personal biases

• Blinding is almost NEVER indexed by NLM nor is it included in the abstract of the articles. Often hard to find in the body of the article (methods section)

A good test….…is positive when it should be

positive AND negative when it should be negative...

• Sensitivity and specificity• Positive and negative predictive values• False positive and negative reactions• Positive and negative likelihood ratio

Calculating diagnosis numbersDisease is present

Disease is absent

Test shows positive (disease may be present)

A B

Test shows negative (probably no disease)

C D

A + C

B + D

Sensitivity and Specificity

Sensitivity------test is positive when it should be positive

Specificity------test is negative when it should be negative

Predictive ValuesPositive---

---proportion of people with positive results who actually have the disease

Negative------proportion of people with negative results who do NOT have the disease

False Positive/Negative Results

False positive------test is positive when the person does not have the disease (labeling)

False negative------test is negative when the person does have the disease (lose time)

Likelihood RatiosThe likelihood ratio for a positive

result (LR+) tells you how likely you are to have a positive test if you HAVE the disease

The likelihood ratio for a negative result (LR-) tells you how likely you are to have a negative test if you do NOT have the disease.

Diagnostic Decision

Wait Test TreatProbability of disease

testing threshold treatment threshold

0%100%

Positive test results moves Negative test results moves

Likelihood ratios are the SLIDERS on this scale!

Concepts

Accuracy Bayes theoremDiagnosisFalse negative False positiveNegative predictive

value

Positive predictive value

ROC curvesLikelihood ratiosSensitivitySpecificityNomogram Screening

MeSHSensitivity a#d specificity*

Predictive values of testsROC curves

Diagnostic errorsFalse negative reactionsFalse positive reactionsObserver variations

Likelihood functionsDiagnosis, differential*Reproducibility of results

Publication Types

Subheadings

Diagnosis (explodable)

Diagnostic use (explodable)

TextwordsSensitivit:Specificit:Predictive value:False positiveFalse negativeLikelihood ratioAccura:

Prognosis

Prognosis/Natural History

• What is going to happen to me over the next period of time now that I have been diagnosed with…Patient preference for this type of information and not therapy.

• Should I treat (or choose to be treated) rather than what is best to treat with. (Scoliosis and prostate cancer)

Prognosis—Example

• Do we know the natural history or prognosis for patients who have have been diagnosed with Parkinson disease?

• 297 patients (181 men) with 1731 visits over 6.4 years (mean) showed a variable course of the disease and its symptoms

Prognosis vs Natural History

Traditional difference…

Natural history is what happens to untreated disease over time….

Prognosis is what happens to treated disease…

Prognosis vs Natural History

Natural History• Biological onset at

cellular level• Early diagnosis

possible (screening)

• Usual diagnosis• Outcomes

Prognosis

• Usual diagnosis• Outcomes

Prognosis Methodology

Inception cohort study

Group of persons assembled early (or at least at a uniform point) in the timing of their disease and followed over time.

Prognosis Methodology

Inception cohort study…

• Follow up 80% or better• The follow-up time should be

consistent with the demands of the disease

• Common sense rules

Prognosis Numbers

Raw rates of disease progression (14% had a subsequent myocardial infarction within 5 years of follow up)

Relative risksHazards ratiosOdds ratioStandardized mortality ratios

Prognostic vs Risk Factors

Prognostic factorsSome aspect that an individual has that can modify how that disease will play out—e.g., age of person with CAD, breast cancer stage

Risk factorsSome aspect that an individual has that may affect whether that person is more or less prone to acquire a disease or condition, e.g., family history of CAD, Al and Alzheimer’s

Prognosis Terms

Adjustment Natural HistoryCohort study PrevalenceInception PrognosisIncidence Prognostic factorLongitudinal studies Prospective studies Morbidity Risk factors Mortality rates, ratios

MeSH Cohort studies* Survival XXXXXX

longitudinal studiesprospective studiesfollow-up studies

Prognosis*Morbidity

incidenceprevalence

MortalitySurvival analysisDisease progression*Time factors*Age factors*Sex factors*

Publication Types

Subheadings

MortalityEpidemiology (sometimes)

TextwordsNatural history Prognos:Inception cohort Clinical coursePredict: Predictive valueOutcome Prognostic factorCourse

Alternative Source

For short-term prognosis information, randomized controlled trials often have information that can be used when desperate. This is especially true for placebo-controlled or usual-care arms of studies.

Etiology/Causation/Harm

Etiology/Causation/HarmEtiology or causation is the study of what

causes, or what increases or decreases the risk for a disease or condition. This can be either positive (protective) or negative (harmful).

e.g., social support at work is associated with fewer short-term psychiatric absences

giving up driving is associated with depressive symptoms in older adults.

Etiology/Causation/HarmGenerally looking at• Exposures to causative/protective agents• Outcomes• Time (can be now, in the future, or looking back)

• Other factors that can affect outcomes

• Need a lot of creativity and common sense• Many long-term etiology studies come

from countries with socialized medicine.

Q: Does the use of cell phones while driving cause accidents?

Exposures?

Outcomes?

What groups?

Q: Does the use of cell phones while driving cause accidents?

Randomized controlled trial

large group of persons who are told whether they will use cell phone during

driving for long time…..

Q: Does the use of cell phones while driving cause accidents?

Cohort study

a group of persons who uses phones while driving is compared with another group who do not use phones while driving…

Q: Does the use of cell phones while driving cause accidents?

Case-control study

people who have had automobile accidents are compared with people who have not had accidents and both groups are studied to see who used phones during driving…

Q: Does the use of cell phones while driving cause accidents?

Statistically adjusted groups (cross sectional study)

Data on some persons with automobile accidents were collected and compared with data from some persons with no accidents and both groups have cell phone use rates determined….

Etiology study types

Type Time Quality Number

RCT future ***** *Cohort now *** ***Case-control past * *****Statistically past -

infinity adjusted groups

(cross sectional studies)

Etiology Issues

• Association is not necessarily causation

• Ethics• Logistics• Blinding• 2 C’s—Common sense and

Creativity

Association vs Causation

Just because two things occur at the same time does not mean that they are causal--

• Higher intake of ice cream and higher rates of drowning occur in the summer—linked? Don’t think so…...

• Poor quality health and low socioeconomic status……

EthicsSometimes you cannot allocate

persons to exposures

– Smoking– Divorce– Genetic disposition– High socioeconomic status– Can do this however for drug

adverse effects, social programs

LogisticsGroups need to be as similar as

possible to account for confounding

• For power line studies…..• Smoking and drinking issues…• Another set of “intertwined” issues?

Need for creative problem solving

BlindingBlinding is crucial especially for the

case-control studies.Blinding must be 2-fold here:

– forming groups without knowing exposures (cell phone use in cars)

– assessing exposure without knowing disease/exposure status (automobile accidents)

Relative Risk (RR)Used for RCTs and cohort studies (prospective)

Comparison of rates of developing the disease/ condition in the 2 groups of people with and without the risk factor (We know exposures)

Weight gain and coronary heart disease in women (400 women in our sample)if gain > 15 lb 106/200if gain < 15 lb 56/200

RR= (106/200)/(56/200) = 1.9

Concepts

AssociationBlinding CausationCase-control studyCohort studyConfoundersCross sectional

studies

Odds ratioProspective studiesRelative riskRetrospective

studiesRiskRisk factors

MeSHCase-control studies*

Retrospective studiesCohort studies*

Longitudinal studiesProspective studiesFollow-up studies

Cross sectional studiesRisk*

Risk assessmentRisk factors

Odds ratio

Publication Types

Subheadings

Etiology (explodable)

Epidemiology (for distributions, causes, and attributes of disease)

TextwordsCohortCase controlRiskOdds ratioCausation or causal:Relative riskEtiol: or Aetiol:

Systematic Reviews and Meta-Analyses

Two types of review articles exist:

• Narrative reviews

• Systematic reviews– Systematic reviews– Meta-analyses of study data– Meta-analyses using individual

patient data

Systematic Reviews • Medicine did not “invent” or develop• Fully developed by psychology,

education, and related disciplines • Early 1900s

– Pearson and enteric fever in the British Army– NIH (Hygiene Laboratory) report

• People:– Eugene Glass was one of the first developers– Tom Chalmers was one of the first medical

developers (On Golden Pond)– Archie Cochrane plus Tom and Iain Chalmers

Systematic vs Narrative

• Team approach• Narrow specific

purpose• Methods drive the

process• Inclusion/

exclusion criteria• Clinically useful

• One major author• Broad purpose

may not always be stated

• No methods on how articles picked

• No inclusion/ exclusion criteria

• Ecuationally useful

Systematic Reviews

Must have

• Purpose why done• Search strategy in detail• Inclusion and exclusion criteria for

study selection

Why do one?• Too much data• Too little data• Resolve discrepancies • “tighten up” estimates of effects of

treatments, exposures, etc• Analyze patient subgroups• Plan for new studies—similar or “next

generation” studies• Provide data for certain types of studies

such as economic studies, decision analyses, or clinical practice guidelines

Steps

• Problem formulation• Identify and select articles

(searching and retrieval)• Data extraction• Analysis and decision if meta-

analysis is appropriate (clinical and statistical test)

• Presentation of results

Statistical Concepts

HomogenietyHeterogeniety

WeightingPooling

Effect sizes

Statistical Concepts

Same as for other study types but often qualified with “combining” phrases

combined odds ratiopooled relative riskweighted hazards ratiotypical mortality ratesummary estimates sensitivity

MEDLINE Difference

Simplistic difference (and hard to differentiate between the two)

Review—does a summary of existing knowledge

Meta-analysis—produces new knowledge

Meta-analysis MeSH

A quantitative method of combining the results of independent studies (usually drawn from the published literature) and synthesizing summaries and conclusions which may be used to evaluate therapeutic effectiveness, plan new studies, etc., with application chiefly in the areas of research and medicine.

Clinical trials overview, data pooling,

Meta-analysis Previous Indexing

Follow up studies

Outcome and process assessment

Research

Research design

Statistics

Meta-analysis Publication Type

Works consisting of studies using a quantitative method of combining the results of independent studies (usually drawn from the published literature) and synthesizing summaries and conclusions which may be used to evaluate therapeutic effectiveness, plan new studies, etc. It is often an overview of clinical trials. It is usually called a meta-analysis by the author or sponsoring body and should be differentiated from reviews of literature.

Review Literature MeSH

Published materials which provide an examination of recent or current literature. Review articles can cover a wide range of subject matter at various levels of completeness and comprehensiveness based on analyses of literature that may include research findings. The review may reflect the state of the art. It also includes reviews as a literary form.

Systematic Reviews Terms

Effect sizesHeterogenietyHomogenietyMeta-analysis of

study dataMeta-analysis of

individual pt dataPooling

Narrative reviewSummary estimatesSystematic reviewsTypical Weighting

MeSH

Randomized controlled trialsClinical trialsControlled trialsMeta-analysis

Publication Types

Meta-analysis

Review iffANDed with MeSH or textwords

Subheadings

Textwords

Meta-analy:Metanal:Metaanal:Systematic review

or overviewOverview (careful)Quantitative review

or overview

Methodologic: review or overview

HeterogenietyHomogenietyMedlinePsychinfoPsycinfoEmbase

Food for Thought

Sometimes a rare condition that is being written up as a case report will include a substantial amount of background material. Don’t necessarily cross out a case-report if you are desperate for review type material.

Alternative SourcesPubMed Clinical Queries—Review

articlesCochrane—note only for RCTS

– Cochrane reviews– Dare– Clinical Trials

DARE (Database of Reviews of Effectiveness—librarian run project)

Campbell CollaborationTRIP database—one stop shopping

Systematic Reviews Quiz

1 Systematic reviews are the same as ordinary reviews only bigger?

Based on: Petticrew M. Systematic reviews from astronomy to zoology: myths and misconceptions. BMJ. 2001;322:98-101.

Systematic Reviews Quiz

2 Systematic reviews contain only randomized controlled trials.

Systematic Reviews Quiz

3 Only medical topics can be studied in systematic reviews.

– More money on schools improved educational outcomes?

– Do men or women make better leaders?– Does the sexual orientation of the parent matter to

the child’s well being?– Are fathers more likely to treat their children

differently than mothers?– Is job absenteeism an indicator of job dissatisfaction?– Are jurors influenced by the race of the defendants?– Is there a relation between poverty and violence?

Systematic Reviews Quiz

4 Systematic reviews can be done well without experienced information personnel or good library support.

Systematic Reviews Quiz

5 Systematic reviews have no relevance to the real world

Systematic Reviews Quiz

6 Systematic review necessarily involve statistical syntheses.

Systematic Reviews Quiz

7 Systematic reviews have to be done by experts with a lot of training in methodologies.

Systematic Reviews Quiz

8 Systematic reviews are a substitute for doing good quality individual studies.

Systematic Reviews Quiz

9 You can be part of a team doing a systematic review.