critical apraisal.ppt

7/21/2019 critical apraisal.ppt

http://slidepdf.com/reader/full/critical-apraisalppt 1/102



RESEARCH QUESTION

This is the essential question the study is setup to answer. Most studies are concernedwith assessing one of four types ofobjectives:

to assess the magnitude of a health problemor health factor;

to assess the efficacy of an intervention; to asses the causal relation between one

factor or set of factors and the disease oroutcome of interest; or to asses the natural history of disease



HYPOTHESIS

A proportion about the relationship between two or

more factors or variables of interest. The purpose of the study is to collect data which

will allow the researcher to test the hypothesis.

In some studies the hypothesis may not be statedbut it is implicit. (However, not all studies testhypotheses. Some times descriptive studies arecalled hypothesis generating).

Hypotheses are often started in a null form, so asto allow them to be refuted.



STUDY FACTOR

The exposure or variable of interest that ishypothesized to be related to the healthproblem, disease or outcome of interest.

As the independent variable. For example, in astudy of salt intake and its relationship toblood pressure, the study factor is salt intake.

Must be quantifiable, but it may be assessed

by a variety of means both direct and indirect,objective and subjective



OUTCOME FACTOR

The event or occurrence that is supposed tohave happened as a result of the influenceof the study factor.

The example: the outcome factor is bloodpressure, it being seen to be influenced bythe study factor, salt intake. The outcomefactor is also know as the dependent

variable May also be measured by a variety of

methods direct and indirect, objective andsubjective



POPULATION

The whole collection of units(individuals, cases, events) fromwhich a sample may be drawn; andwhich must be defined geographically

and temporally.

REFERENCE POPULATION

This is an abstract concept of thepeople to whom the researcher wishesto refer results. Also known as thetarget population.



SOURCE POPULATION

In setting up a study it is frequentlyimpossible or impractical to include all ofthe reference population and the

researcher will have to resort to a subsetknown as the source population .

The population that is included in the study

actually comes from and its specificationswill be more limited than the referencepopulation



SAMPLING FRAME

To access the source population the researcherrequires a sampling frame or study frame which will represent the source population.

The simplest sampling frame would be completelisting of all the population.

From this sampling frame a sample is collectedwhich may consist of all or part of the frame. As

not all of those identified in the sample will beavailable or willing to participate in a study,those that do will comprise the final studysubjects .



STUDY SAMPLE

This is the subject selected byrandom or non-random means fromthe sampling frame to represent the

source population.

In some studies it is possible to studythe entire source population for the

period of the study and there is nonecessity to draw a sample



SAMPLING

A process for selecting a study samplefrom the sampling frame when it isimpractical to use the complete sourcepopulation.

The usual preferred method is randomsampling whereby inclusion in the studysample is decided by chance.

The objective of sampling is to select anon biased sample which can be studied.

The hierarchy of sampling can beillustrated in the sample given below:



Source Population- the broad group of people from

whom the subjects will be obtained, i.e., schoolchildren in the Hunter Region. Sampling Frame – the list of potential subjects

from which a sample will be drawn, i.e., class listsof all Public Schools in the greater Newcastle area

Reference Population – theabstract concept of peopleto whom the results wouldapply-in these case- Australian school children

Study Subjects Sample

Sampling Frame

Source Population

Reference Population

A researcher wants to study the prevalence ofhearing defects in Australian school children.



Sampling Frame – the list of potentialsubjects from whom which a sample will

be drawn, i.e. class lists of all PublicSchools in the greater Newcastle area.

Sample- the subjects who are selected to

take part in the study, e.g. a randomselection of 25% of each class in each ofthe 8 schools which have been randomlyselected from the 40 in the greater

Newcastle area Sample Subjects- these provide the

data, e.g. 90% of children had parentalapproval to take part in the study.



BIAS

Any effect at any stage of investigation (orinference) tending to produce results thatdepart systematically (i.e., not randomly)

from the truth.

Many varieties of bias have beendescribed including measurement bias,

selection bias and confounding bias



CONFOUNDER

A factor that distorts the apparent magnitude ofthe effect of the study factor on the outcomefactor.

A confounder is it self a determinant of theoutcome of interest, and is unequally distributedamong the subjects that are exposed or notexposed to the study factor.

For example, in a study of the influence ofmaternal nutrition on the birth weight ofnewborns (where it is hypothesised that women

with a poor diet will have low birth weightinfants), smoking is a confounding factor ifwomen with poor dietary habits are also morelikely to smoke (and it is known that smoking isof itself a risk factor for low birth weight infants)



RISK

Probability that an event will occur, e.g.,that an individual will die or become ill within a

stated period of time or age

ATTRIBUTABLE RISK The difference between the rate of disease

in the exposed population and the rate in thenon-exposed population.

It is a measure of the amount of diseasethat might be reasonably attributed

to the exposure in question.



RISK FACTOR An attribute or exposure that increases the

probability occurrence of disease or a specified

outcome. This defines risk factor as a determinantof the outcome, though risk factor can be used todenote association with a specified outcome thatis not necessarily causal (i.e., a risk marker or riskindicator, (e.g., smoking and positive parental

history are considered risk factors for coronaryartery disease.

PREVALENCE The total number of all individuals who have the

disease or factor or interest at a particular time orduring a particular period ( the numerator)divided by the population at risk of having thedisease at this time ( the denominator)



INCIDENCE

The number of new cases of a diseaseoccurring in a defined population within aspecified period of time.

Usually expressed as the incidence ratewhich has the number of new cases of adisease in a defined period of time as the

numerator and the number of persons inthe stated population in which the casesoccurred as the denominator.



RELATIVE RISK

It is one measure of the strength of effect of anexposure on the likelihood of a particularoutcome and is the ratio of risk of disease ordeath among those exposed to the study factor,to the risk of disease or death among those notexposed to the study factor.

Note, that to calculate relative risk, it isnecessary to use incidence figures for disease ordeath. As some study designs preclude thispossibility, an approximation to relative risk,called the odds ratio is used instead.



(STATISTICAL) POWER

This is the probability that a study willdetect a statistically significant differencewhen a difference really exists.

COHORT STUDY The study population is classified according

to exposure to the study factor prior to the

development of the outcome of interest.The incidence of the outcome is thenascertained in groups with differentexposure status



CASE CONTROL (OR CASE REFERENT) STUDY The study sample consists of a group with the

outcome of interest (cases) and a sample of

controls representative of the same sourcepopulation from which the cases arose. Theprevalence of exposure to the study factor in eachgroup at a time prior to the development of theoutcome in the cases is compared

RANDOMIZED CONTROLLED TRIAL An epidemiological experiment whereby the

incidence of the study outcome(s) is compared ingroups assigned by chance to different

interventions and usually one group (the controlgroup) who are not exposed. The aim ofrandomization is to achieve an equal or at leastunbiased distribution of confounders, particularlyunmeasured or potential confounders in exposedand non exposed groups



STUDY VALIDITY

“The degree to which the inference drawnfrom a study, especially generalisationsextending beyond the study sample, are

warranted when account is taken of thestudy methods, the representativeness ofthe study sample, and the nature of the

population from which it is drawn. Twovarieties of study validity aredistinguished” (Last):



1. Internal Validity. The groups being compared (treatedvs. non-treated, exposed vs. non-exposed or cases vs.controls) have been selected and measurements

made in such a way that the results can be considereda good approximation to truth.

2. External Validity. The subjects in the study areselected and described in such a way that the results,

given internal validity, can be applied or generalizedoutside of the study sample.

Threats to internal validity come from systematic error orbias. Threats to external validity come from the selection

and sampling of subjects for a study . In order to seewhether a study has external or generalizability it isimportant to have a clear idea of where the subjects havecome from



MEASUREMENT ISSUES

Epidemiology is very much about making measurements

and about being critical of the measurements that aremade by ourselves and other. There are two issues withall measurements which will be defined here. They areValidity and repeatability .

1. Measurement validity. An expression of the degree towhich a measurement measures what it purports tomeasure, e.g., an autopsy has more validity than abiopsy or x-ray diagnosis

2. Repeatability. Critical to the validity of test is itsrepeatability, i.e., the degree of stability exhibitedwhen the measurement is repeated under identicalconditions. Most biological measurements show somevariation when repeated.



STUDY EFFICIENCY

This addresses the issues of the cost of obtaininginformation. “ the effects or end-results achievedin relation to the effort expended in terms ofmoney, resources, and time” (Last). Case control

studies for example, are considered to be moreefficient than longitudinal studies at obtaininginformation about the effect of an exposure or anoutcome. This is because less effort needs to gointo acquiring data by comparing diseased and

non-diseased people for their exposure versuscomparing exposed and non-exposed people forwhether they develop an outcome



Critical Appraisal

using

Epidemiology a Basic Methods. Critical

Appraisal ( Including StudyTerminology).

University of Newcastle; 1994.



CRITICAL APPRAISAL WORKSHEET (CAW) A General approach to critical appraisal of scientific literature

The worksheet is a matrix composed of nine rows of

questions about the paper, and three columns in which you

are:

firstly whether the information to answer the question isavailable in the paper;

secondly, whether they was a problem with the way inwhich it was handled and;

thirdly, whether the problem identified was sufficient toaffect the validity of the study

CRITICAL APPRAISAL WORKSHEET (CAW) (1991)




(1) (2) (3)

Can you find thisinformation in the

paper?

Is the way this was done aproblem?

Does this problemthreaten the

validity of thestudy

1. What is theresearch questionand/or hypothesis?

2. What is the studytype?

3. What is thereferencepopulation? Whatare the samplingframe andsampling method?

Is it concerned with theimpact on anintervention, causality, ordetermining the

magnitude of a healthproblem?

Is the study typeappropriate to theresearch question ?

Is the sampling frameappropriate for thereference population

Is there selection bias?

If not, how usefulare the resultsproduced by thistype of study?

Do these threatenthe externalvalidity of thestudy





(1) (2) (3)

Can you find this

information in thepaper?

Is the way this was

done a problem?

Does this problem

threaten the validity ofthe study

4. What are the study

factors and how arethey measured?

5.What are the outcome

factors and how

are they

measured?

Is there measurement

error?

a. Are all relevant

outcomes assessed?

b.Is there measurement

error?

Is measurement error

an important source ofbias?

a.How important are

omitted outcomes?

b.Is measurement error

an important source

of bias?





(1) (2) (3)

Can you find this

information in thepaper?

Is the way this was

done a problem?

Does this problem

threaten the validity ofthe study

6. Are these sources of

bias relevant to thestudy? Selection

bias, recall bias,

ascertainment

bias, confounding

bias, non-random

assignment,

incomplete follow-

up

Have steps been taken

to avoid thesebiases?

Is the internal validity

of the study threatenedby bias?





(1) (2) (3)

Can you find this

information in the

paper?

Is the way this was done a

problem?

Does this problem

threaten the


7. Are sample size

issues considered?

Is the power of the

study indicated?

8. Are statistical

methods

described?

Is the sample size sufficient

to detect clinically/

socially meaningful

differences

Are the proposed statistical

methods appropriate for

addressing the stated

research question and/orhypothesis? Are the

proposed statistical

methods appropriate for

the data?

Do the conclusions

drawn follow

logically from the

results of theanalyses?





(1) (2) (3)

Can you find this

information in the

paper?

Is the way this was done a

problem?

Does this problem

threaten the


9. What conclusions

did the authors

reach about the

research question?Did they generate

new hypotheses?

Do you agree with

the conclusions?

Do the results apply to the

population in which you

would be interested

Do you accept the

results of this

study?



Critical Appraisal

using

The Pocket Guide to Critical AppraisalIain K. Crombie

BMJ



THE STANDARD APPRAISAL QUESTIONS

ARE THE AIMS CLEARLY STATED? WAS THE SAMPLE SIZE JUSTIFIED?

ARE THE MEASURMENTS LIKELY TO BE VALID ANDRELIABLE?

ARE THE STATISTICAL METHODS DESCRIBED?

DID UNTOWARD EVENTS OCCUR DURING THE STUDY? WERE THE BASIC DATA ADEQUATELY DESCRIBED?

DO THE NUMBERS ADD UP

WAS THE STATISTICAL SIGNIFICANCE ASSESSED?

WHAT DO THE MAIN FINDINGS MEAN?

HOW IS NULL FINDINS INTERPRETED? ARE IMPORTANT EFFECTS OVERLOOKED?

HOW DO THE RESULTS COMPARE WITH PREVIOUS REPORTS?

WHAT IMPLICATIONS DOES THE STUDY HAVE FOR YOURPRACTICE?



THE COMPLETE LIST FOR APPRAISALOF SURVEYS

The essential questionsWho was studied?How was the sample obtained?What was the response rate?.

The detailed questions* Design

Are the aims clearly stated? Is the design appropriate to the stated objectives? Was the sample size justified?

Are the measurements likely to be valid and reliable? Are the statistical methods described? Is there a suggestion of haste?



Conduct Did untoward events occur during the study?

Analysis Where the basic data adequately described?Do the numbers add up?Was the statistical significance assessed? Were the findings serendipitous?

Interpretation What do the main findings mean? How could selection bias arise?

How are null findings interpreted? Are important effects overlooked? Can the results compare with previous reports? How do the results compare with previous reports?

What implications does the study have for your practice?

THE COMPLETE LIST FOR THE



THE COMPLETE LIST FOR THE APPRAISAL OF COHORT STUDIES

The essential questions Who exactly has been studied? Was a control group used? Should one have been used? How adequate was the follow-up?

The detailed questions* Design Are the aims clearly stated? Is the design appropriate to the stated aims? Was the sample size justified? Are the measurements likely to be valid and reliable? Were relevant outcome measures ignored? Are the statistical methods described?



Conduct Did untoward events occur during the study? Analysis Did the analysis allow for the passage of time? Do the numbers add up?

Were the basic data adequately described? Was statistical significance assessed? Interpretation What do the main findings mean? What else might influence the observed outcome?

How are null findings interpreted? Are important effects overlooked? How do the results compare with previous reports? What implications does the study have for your practice?




THE COMPLETE LIST FOR THE APPRAISAL OF CLINICAL TRIALS

The essential questions Were treatment randomly allocated? Were all the patients accounted for? Were outcomes assessed bind? The detailed questions*

Design Are the aims clearly stated? Was the sample size justified? Are the measurements likely to be valid and reliable? Could the choice of subjects influence the size of treatment effect?

Were the ambiguities in the description of the treatment and itsadministration?



Are the statistical methods described? Could lack of blinding introduce bias? Are the outcomes clinically relevant? Conduct How was the randomization carried out?

Did untoward events occur during the study? Analysis Were the treatment groups comparable at baseline? Were results analyzed by intention to treat? Was the statistical significance assessed?

Were the basic data adequately described? Do the numbers add up? Were side-effect reported?



Interpretation

What do the main findings mean?

How are null findings interpreted?

Are important effects overlooked? How do the results compare with previous

reports?

What implications does the study have foryour practice?




THE COMPLETE LIST FOR THE APPRAISAL OF CASE-CONTROL STUDIES

The essential questions How were the cases obtained? Is the control group appropriate? Were data collected the same way for cases and

controls? The detailed questions*Design Are the aims clearly stated? Is the method appropriate to the aims? Was the sample size justified? Are the measurements likely to be valid and reliable? Are the statistical methods described?



Conduct Did untoward events occur during the study? Analysis Were the basic data adequately described? Do the numbers add up? Was there data-dredging? Was the statistical significance assessed?

Interpretation What do the main findings mean? Where are the biases? Could there be confounding?

How are null findings interpreted? Are important effects overlooked? How do the results compare with previous

reports? What implications does the study have for your

practice?

THE COMPLETE LIST FOR



THE COMPLETE LIST FOR APPRAISAL OF REVIEW PAPERS

The essential questions

How were the papers identified?

How was the quality of papers assessed?

How were the results summarized?

The detailed questions*

Design

Is the topic well defined?

Are the statistical methods described?

Conduct

Were the detailed study designs reviewed?

Was missing information sought?



Analysis

Were the basic data adequately described?

Was publication bias taken into account?

Was heterogeneity of effect investigated?

Interpretation

What do the main findings mean? Are there other findings which merit attention?

Are the conclusions justified?

How do the findings compare with previousreports?

What implications does the study have for yourpractice?



Evidence Based Medicine

Evidence-based medicine (EBM) requires the integration of the best

research evidence with our clinical expertise and our patient‟s

unique values and circumstances

By best research evidence we mean valid and clinically relevant

research, often from the basic sciences of medicine, but

especially from patient-centered clinical research into the

accuracy of diagnostic tests (including the clinical examination),

the power of prognostic markers, and the efficacy and safety of

therapeutic, rehabilitative, and preventive regimens. New

evidence from clinical research both invalidates previouslyaccepted diagnostic tests and treatments and replaces them

with new ones that are more accurate, more efficacious, and

safer



Evidence Based Medicine

By clinical expertise we mean the ability to use our clinical skills and

past experience to rapidly identify each patient‟s unique health state

and prognosis, their individual risks and benefits of potential

interventions, and their personal circumstances and expectations

By patient values we mean the unique preferences, concerns and

expectations each patient brings to a clinical encounter and which

must be integrated into clinical decisions if they are to serve the

patient.

By patient circumstances we mean their individual clinical state and

the clinical setting.



Critical Appraisal

using

Evidence-Based Medicine.

How to Practice and Teach EBM.Sharon E. Straus, W. Scott Richardson, Paul

Glasziou, R. Brian Haynes.

Third edition (CD room); Elseiver Churchill Living stone



Critical Appraisal using EBM Version needs worksheets:

1. Diagnosis and screening worksheet

2. Harm / Etiology worksheet

3. Prognosis worksheet

4. Systematic Review worksheet

5. Therapy or Prevention worksheet

6. Etc



1. DIAGNOSIS WORKSHEET

Are the results of this diagnostic study valid

Was there an independent , blind comparison

with a reference (“gold”) Standard of diagnosis

Was the diagnostic test evaluated in an

appropriate spectrum of patients (like those inwhom it would be used in practice)?

Was the reference standard applied regardless

of the diagnostic test result?

Was the test (or cluster of tests) validated in a

second, independent group of patients?

A th lt f thi di ti t d lid?



Are the results of this diagnostic study valid?

Sample Calculation

Target disorder

(iron deficiency anemia)

Totals

Present Absent

Diagnostictest result(serumferritin)

Positive

(<65 mmol/L)

731

a

270

B

1001

a+b

Negative

(>65 mmol/L)

78

c

1500

d

1578

c+d

Totals 809

a+c

1770

b+d

2579

a+b+c+d



Sensitivity = a/(a+c) = 731/809 = 90%

Specificity = d/(b+d) = 1500/1770 = 85%

Likelihood ratio for a positive test result =LR+=sensitivity/(1-specifity)=90%/15%=6

Likehood ratio for a negative test result =LR-=(1-Sensitivity)/specificity=10%/85%=0.12

Positive predictive value = a/(a+b)=731/1001=73%Negative predictive value = d/(c+d)=1500/1578=95%

Pre-test probability (prevalence) = (a+c) / (a+b+c+d)= 809/2579=32%

Pre-test odds=prevalence/(1-prevalence)=31%/69%=0.45

Post-test odds = ( pre-test odds) x LR

Post-test probability=(post-test odds)/post-test odds+1)



YOUR CALCULATIONS

Target disorder Totals

Present Absent

Diagnostictest result

Positive a b a+b

Negative c d c+d

Totals a+c b+d a+b+c+d



Can You apply this valid, important evidence abouta diagnostic test in caring for your patient?

Is the diagnostic test available,affordable, accurate, and precise in yoursetting?

Can you generate a clinically sensible

estimate of your patient‟s pre-testprobability (from personal experience,prevalence statistics, practice database,or primary studies)?

Are the study patients similar to your

own? Is it unlikely that the disease

possibilities or probabilities have

changed since the evidence was

gathered?



Can You apply this valid, important evidence abouta diagnostic test in caring for your patient?

Will the resulting post-test probabilitiesaffect your management and help yourpatient?

Could it move you across a test-

treatment threshold ?

Would your patient be a willing partner

in carrying it out?

Would the consequences of the test helpyour patient?

Additional Notes:

2 HARM WORKSHEET



2. HARM WORKSHEET

Are the results of this harm study valid?

Were there clearly defined groups ofpatients, similar in all important waysother than exposure to the treatment orother cause?

Were treatments/exposures and clinicaloutcomes measured in the same ways inboth groups (was the assessment ofoutcomes either objective or blinded toexposure)?

Was the follow-up of study patientssufficiently long and complete?

Do the results satisfy some “diagnostictests for caution”?





Is it clear that the exposure preceded theonset of the outcome?

Is there a dose-response gradient?Is there positive evidence from a “dechellenge-rechallenge” study?

Is the association make biological sense?

Does the association make biologicalsense?

What is the magnitude of the associationbetween the exposure and outcome?

What is the precision of the estimate ofthe association between exposure andoutcome

Are the valid results from this harm study important?



Adverse outcome Totals

Present(Case)

Absent(control

Exposed tothetreatment

Yes

(cohort)

a b a+b

No

(cohort)

c d c+d

Totals a+c b+d a+b+c+d

In randomized trial or cohort study: relativerisk=RR=[a/(a+b)]/[c/(c+d)]

In case-control study : odds ratio (or relative odds)=OR=ad/bc



Should this valid, potentially importantresults changes the treatment of yourpatient?1. Do the results apply to our patient?

2. Is our patient so different from those in the study that its

results don‟t apply?

3. What are the patient‟s risks of the adverse event?

4. To calculate the NNH ( number of patients we need to

treat to harm one of them) for any odds ratio (OR) and

our patient‟s expected event rate for this adverse eventif they were not exposed to this treatment (PEER):

PEER (OR-1)+1NNH = ------------------------------

PEER(OR-1)x(1-PEER)



5. What are our patient‟s preferences, concerns andexpectations from this treatment?

6. What alternative treatments are available?

Additional notes:

3. PROGNOSIS WORKSHEET



3. PROGNOSIS WORKSHEET

Are the results of this prognosis study valid ?

1. Was a defined, representative sample of patients

assembled at a common ( usually early) point in the

course of their disease?

2. Was patient follow-up sufficiently long and complete?

3. Were objective outcome criteria applied in a “blind”fashion?

4. If subgroups with different prognoses are identified:- Was there adjustment for important prognostic factors?

- Was there validation in an independent group („test

set‟) of patients?



Are the valid results of this prognosis studyimportant?

1. How likely are the outcome over time?

2. How precise are the prognostic estimates?



IF YOU WANT CALCULATE A CONFIDENCE INTERVAL AROUND THE MEASURE OF PROGNOSIS

Clinical measure Standard error (SE) Typical calculation ofCI

Proportion ( as in therate of someprognostic event, etc.)where:

n=the number ofpatients

P=the proportion ofthese patients who

experience the event

__________√[p x (1-p)/n]

Where p is proportionand n is number ofpatients

If p=24/60=0.4(or40%) and n 60:

_____________

SE =√[0.4x(1-0.4)/60]= 0.063 (or 6.3%)

95%CI is

40%±1.96x6.3% or

27.6% to 52.4 %

n from your evidence:

p from your evidence:

Our calculation:

SE: ___________

95% CI:



Can you apply this valid, important evidenceabout prognosis in caring for your patient?

1. Do the results apply to our patient?

2. Is our patient so different from those in the study that its

results cannot apply?

3. Will this evidence make a clinically important impact on

our conclusions about what to offer or tell our patient?

Additional notes:

4 SYSTEMATIC REVIEW (OF THERAPY) WORKSHEET



4. SYSTEMATIC REVIEW (OF THERAPY) WORKSHEET

Are the results of this systematic review valid?

1. Is this a systematic review of randomized trials?

2. Does it describe a comprehensive and detailed search forrelevant trials?

3. Were the individual studies assessed for validity?

4. Were the individual patient data used in the analysis ( oraggregate data)?

Are the valid results of this systematic review



The numbers in the body of the tables are the NNTs for thecorresponding odds ratio (OR) at that particular patient‟sexpected event rate (PEER)

Are the valid results of this systematic reviewimportant?

1. Are the results consistent across studies?

2. What is the magnitude of the treatment effect?

3. How precise is the treatment effect?

Translating odd ratio to NNTs

When the odds ratio (OR)<1 This table applies when a bad outcome is prevented by therapy



This table applies when a bad outcome is prevented by therapy

OR < 1

Patient‟sexpectedevent

rate(PEER)

0.9 0.8 0.7 0.6 0.5

0.05 2.09a 104 69 52 41b

0.10 110 54 36 27 21

0.20 61 30 20 14 11

0.30

0.40

46

40

22

19

14

12

10

9

8

7

0.50 38 18 11 8 6

0.70 44 20 13 9 6

0.90 101c 46 27 18 12d

a The relative risk reduction (RRR) here is 10%b The RRR here is 49%c For any OR, NTT is lowest when PEER=0.50d The RRR here is 9%

When the odds ratio (OR)>1



When the odds ratio (OR)>1This table applies both when a good outcome is increased by therapyand when a side effect is caused by therapy

OR > 1

Patient‟s

expectedeventrate(PEER)

1.1 1.2 1.3 1.4 1.5

0.05 212 106 71 54 43

0.10 112 57 38 29 23

0.20 64 33 22 17 14

0.30 49 25 17 13 11

0.40

0.50

43

42

23

22

16

15

12

12

10

10

0.70 51 27 19 15 13

0.90 121 66 47 38 32



Can you apply this valid, important evidence fromsystematic in caring for your patient ?

1. Do these results apply to our patient ?

2. Is our patient so different from those in the study that itsresults cannot apply?

3. Is the treatment feasible in our setting?

4. What are our patient‟s potential benefits harms from thetherapy?

5. Method I : In the OR tables above, find the intersectionof the closest OR from the systematic review and ourpatient‟s expected event rate (PEER)



6. Method II : To calculate the NNT from any OR and PEER

1-[PEERx(1-OR)]NNT= ----------------------------

(1-PEER)xPEERx(1-OR)

7. Are our patient‟s values and preferences satisfied by theregimen and its consequences ?

8. Do we and our patient's have clear assessment of theirvalues and preferences?

9. Are they met by this regimen and its consequences?

Should you believe apparent qualitative differences



Should you believe apparent qualitative differencesin the efficacy of therapy in some subgroups ofpatients? Only if you can say “yes” to all of the

following ?1. Do they really make biologic and clinical sense?

2. Is the qualitative difference both clinically (beneficial forsome but useless or harmful for others) and statistically

significant?3. Was this difference hypothesized before the study began

(rather than the product of dredging the data)?

4. Was this one of just a few subgroup analyses carried outin this study?

5. Has the result been confirmed in other independentstudies?

Additional Notes:



5. THERAPY WORKSHEET Are the results of this single preventive ortherapeutic trial valid?

1. Was the assignment of patients to treatmentsrandomized?

2. Was the randomization list concealed?

3. Was follow-up of patients sufficiently long and complete?

4. Were all patients analyzed in the groups to which theywere randomized?

5. Were patients, clinicians, and study personal kept" blind”to treatment?

6. Were the groups treated equally, apart from theexperimental treatment?

7. Were the groups similar at the start of the trial apart

from the experimental therapy?



Are the valid results of this randomized trialimportant?

1. What is the magnitude of the treatment effect?

2. How precise is the estimate of the treatment effect?



SAMPLE CALCULATIONS

Occurrence of diabetic

neuropathy at 5 years amonginsulin-dependent diabetics in

the DCCT trial

Relative risk

reduction(RRR)

Absolute risk

reduction(ARR)

Number

needed totreat (NNT)

Usual insulinregimen

control eventrate (CER)

Intensiveinsulin

regimenexperimentalevent rate

(EER)

CER – EER

CER

CER - EER 1/ARR

9.6% 2.8% 9.6% - 2.8%

9.6%=71%

9.6% - 2.8%

= 6.8 %

1/6.8%

= 15 patients

95% CIa 4.4% to 9.2% 11 to 23



Are the valid results of this randomized trialimportant?

a95% confidence interval (CI) on an NNT

= 1/(limits on the CI of its ARR)

CER x ( 1-CER) EER x (1-EER) = ±1.96 √ ------------------------- + -----------------------------

number of control patients number of experimental patients

0.96 x 0.904 0.028 x 0.972

= ±1.96 √ ----------------------- + ---------------------730 711

= ±2.4%



YOUR CALCULATIONS

Relative risk

reduction(RRR)

Absolute risk

reduction(ARR)

Number

needed totreat (NNT)

CER EER CER – EER

CER

CER - EER 1/ARR

95% CI

Can you apply this valid, important evidence aboutth i i f ti t ?



therapy in caring for your patient ?

1. Do these results apply to our patient ?

2. Is our patient so different from those in the study that itsresults cannot apply?

3. Is the treatment feasible in our setting?

4. What are our patient‟s potential benefits harms from thetherapy?

5. Method I : f

Risk of the outcome in our patient, relative to patients inthe trial

Expressed as a decimal: __________

NNT/f = ---------/--------------=

(NNT for patients like ours)



6. Method II : 1/(PEER xRRR)

Our patient‟s expected event rate if they received the

control treatment (PEER)= ___1/(PEER x RRR) = 1/ =

(NNT for patients like ours)

7. Are our patient‟s values and preferences satisfied by theregimen and its consequences ?

8. Do we and our patient‟s have a clear assessment of theirvalues and preferences?

9. Are they met by this regimen and its consequences?

Additional notes:

RINGKASAN



RINGKASAN

Setelah membaca artikel dan belum jelas

desain penelitiannya (study design dasar),maka “worksheet” yang digunakan adalah:

I.1. Critical appraisal worksheet 1991,

Epidemiology A Basic Methods

RINGKASAN



Setelah membaca artikel dan telah jelas desainpenelitiannya (study design dasar), maka menggunakan

The Pocket Guide to Critical Appraisal By IAIN K.Crombiedengan worksheet:II.1. The complete list for the appraisal of surveys (p:35)II.2. The complete list for the appraisal of cohort studies

(p:42)II.3. The complete list for the appraisal of clinical trials(p:49)

II.4. The complete list for the appraisal of case-controlstudies (p:55)

II.5. The complete list for the appraisal of review papers(p:62)

II.6. Baik point II.1 s/d I.5. perlu jawaban pertanyaan&……..dari I.3.1 – I.3.3

RINGKASAN



Setelah membaca artikel dan telah jelas desain dan jenis

penelitiannya maka menggunakan format EBM yangterdiri dari:III.1 Diagnosis and screeningIII.2 Harm / EtiologyIII.3 Prognosis

III.4 Therapy or PreventionIII.5 Etc

The conclusions have to answer these questions:

1. Is the evidence from this study valid?2. If valid, is this evidence important?3. If valid and important, can you apply this evidence

in caring for your patients

DAFTAR PUSTAKA



DAFTAR PUSTAKA

1. Critical Appraisal ( Including Study Terminology). Epidemiology aBasic Methods. Critical Appraisal ( Including Study Terminology).

Newcastle: University of Newcastle; 1994. p.12

2. Crombie IK. The Pocket Guide to Critical Appraisal: A Handbook

for Health Care Professionals. London: BMJ Publising group;

1996. p.23, 35, 42, 49, 62

3. Straus SE, Richardson WS,Glaziou P, Haynes RB. Evidence Based

Medicine: How to Practice and Teach EBM 3rd ed CD room;

Elseiver Churchill Living stone

GLOSSARY



Terms you are likely to encounter in your clinicalreading

Absolute risk reduction (ARR). See treatment effects Allocation concealment. Occurs when the person who is enrolling aparticipant into a clinical trial is unaware whether the nextparticipant to be enrolling will be allocated to the intervention orcontrol group

Case-control study. A study which involves identifying patients

who have the outcome of interest (cases) and control patientswithout the same outcome, and looking back to see if they had theexposure of interest

Case series. A report on a series of patients with an outcome ofinterest. No control group is involved

Clinical practice guideline. A systematically developed statementdesigned to assist clinician and patient decisions about appropriatehealth care for specific clinical circumstances

Cohort study. Involves identification of two groups (cohort) ofpatients, one that received the exposure of interest, and one thatdid not, and following these cohorts forward for they outcome of

interest.

Confidence interval (CI). Quantifies the uncertainly in



Confidence interval (CI). Quantifies the uncertainly inmeasurement. It is usually reported as 95% CI, which isthe range of values within which we can be 95% sure

that the true value for the whole population lies. Forexample, for an NNT of 10 with a 95% CI or 5 to 15, wewould have 95% confidence that the true NNT valueslies between 5 and 15

Control event rate (CER). See treatment effects

Cost- benefit analysis. Assesses whether the cost of an intervention as worththe benefit measuring both in the same units; monetaryunits are usually used

Cost-minimization analysis. If health effects areknown to be equal, only costs are analyzed and the leastcostly alternative is chosen

Cost-utility analysis. Converts health effects into



Cost utility analysis. Converts health effects intopersonal preferences (or sterilities) and describes howmuch it costs for some additional quality gain(e.q. cost

per additional quality gain (e.q. cost per additionalquality-adjusted life-year, or QALY)

Crossover study design. The administration of two ormore experimental therapies one after the other in aspecified or random order to the same group of patients

Cross-sectional study, The observation of definedpopulation at a single point in time or time interval.Exposure and outcome are determined simultaneously

Decision analysis ( or clinical decision analysis). The application of explicit, quantitative methods thatquantity prognoses, treatment effects, and patientvalues in order to analyze a decision under conditions ofuncertainty

Event rate. The proportion of patients in a group inwhom the event is observed Thus if of 100 patients



whom the event is observed. Thus, if of 100 patients,the event is observed in 27, the event rate is 0.27.Control event rate(CER) and experimental event rate(EER) are used to refer to this in control andexperimental groups of patients, respectively. Thepatient expected event rate (PEER) refers to the rate ofevents we'd expect in a patient who received notreatment or conventional treatment. See treatmenteffects.

Evidence-based health care. Extends the applicationof the principles of evidence-based medicine (seebelow) to all professions associated with health care,including purchasing management.

Evidence-based medicine (EBM). The conscientious,

explicit, and judicious use of current best evidence inmaking decisions about the care of individual patients.The practice of evidence-based medicine requires theintegration of individual clinical expertise with the bestavailable external clinical evidence from systematicresearch and our patient‟s unique values andcircumstances.

Experimental event rate (EER) See treatment



Experimental event rate (EER). See treatmenteffects.

Inception cohort. A group of patients who are

assembled near the onset of the target disorder Incidence The proportion of new cases of the target

disorder in the population at risk a specified timeinterval.

Intention-to-treat analysis. A method of analysis for

randomized trials in which all randomly assigned to oneof the treatments are analyzed together, regardless of ornot they completed or received that treatment, in orderto preserve randomization.

Likelihood ratio (LR). The likelihood that a given testresult would be expected in a patient with the targetdisorder compared with the likelihood that the sameresult would be expected in a patient without the targetdisorder. See Table A2.2 for calculations.

Meta-analysis. A systematic review that uses



Meta analysis. A systematic review that usesquantitative methods to synthesize and e the results.

n-of-1-trials. In such trials, the patient undergoes pairsof treatment periods organized so that one periodinvolves the use of the experimental treatment and theother involves the use of an alternate or placebotherapy. The patient and physician are blinded, ifpossible, and outcomes are monitored. Treatmentperiods are replicated until the clinician and patient are

convinced that the treatments are definitely different ordefinitely not different. Negative predictive value. Proportion of people with

a negative test who are free of the target disorder. Seealso likelihood ratio.

needed to treat (NNT). The inverse of the absoluterisk reduction and the number of patients that need tobe treated to prevent one bad outcome. See treatmenteffects

Odds. A ratio of the number of people incurring an

event to the number of people who have an event.

Odds ratio (OR). The ratio of the odds of having the



Odds ratio (OR). The ratio of the odds of having thetarget disorder in the experimental group relative to theodds in favor of having the target disorder in the control

group (in cohort studies or systematic reviews), or theodds in favor of being exposed in participants with targetdisorder divided by the odds in favor of being exposed incontrol participants (without the target disorder). SeeTable A2A for calculations.

Overview. See systematic review. Patient expected event rate (PEER). See treatment

effects.

Positive predictive value. Proportion of people with a

positive test who have the targ disorder. See alsolikelihood ratio.

Post-test odds. The odds that the patient has thetarget disorder after the test is carried out [(pre-testodds) x (likelihood )]

Post-test probability The proportion of patients with



Post-test probability. The proportion of patients withthat particular test result who have the target disorder[(post-test odds) / (1 + post-test odds)].

Pre-test odds. The odds that the patient has the targetdisorder before the test is carried out [(pre-testprobability) / (1 - pre-test probability)].

Pre-test probability/prevalence. The proportion of

people with the target disorder in the population at riskat a specific time (point prevalence) or time interval(period prevalence). See also likelihood ratio.

Randomization (or random allocation). Method

analogous to tossing a coin to assign patients totreatment groups (the experimental treatment isassigned if the coin lands "heads"; a conventional,"control" or "placebo" treatment is given if the coin lands"tails”

Randomized controlled clinical trial (RCT)



Randomized controlled clinical trial (RCT).

Participants are randomly allocated into an- experimental

group or a control group and followed over time for the

variables/outcomes of interest.

Relative risk reduction (RRR). See treatment effects.

Risk ratio (RR). The ratio of the risk in the treated

group (EER) to the risk in the control group (CER); usedin randomized trials and cohort studies. RR=ERR/CER.

Also called relative risk.

Sensitivity. Proportion of people with the target

disorder who have a positive test result. It is used toassist in assessing and selecting a diagnostic

test/sign/symptom. See also likehood ratio.

SnNout When a sign/test/symptom has a high



SnNout. When a sign/test/symptom has a highsensitivity (5n), a negative result (N) can help rule out(out) the diagnosis. For example, the sensitivity of a

history of ankle swelling for diagnosing ascites is 93%;therefore, if person does not have a history of ankleswelling, it is highly unlikely that the person has ascites.

Specificity. Proportion of people without the target

disorder who have a negative test result. It is used toassist in assessing and selecting a diagnostictest/sign/symptom, See also likelihood ratio.

SpPin. When a sign/test/symptom has a high specificity

(Sp), a positive result (P) can help to rule in (in) thediagnosis. For example, the specificity of a fluid wave fordiagnosing ascites is 92%; therefore, if a person doeshave a fluid wave, he/she may have ascites.

Systematic review. A summary of the medical literature that usesexplicit methods to perform a comprehensive literature search and



explicit methods to perform a comprehensive literature search andcritical appraisal of individual studies, and that uses appropriatestatistical techniques to combine these valid studies.

Treatment effects. The evidence-based journals (Evidence BasedMedicine and ACP journal Club ) have achieved consensus on someof the terms they use to describe both the good and the bad effectsof therapy. We will bring them to life with a synthesis of threerandomized trials in diabetes which individually showed that several

years of intensive insulin therapy reduced the proportion of patientswith worsening retinopathy to 13% from 38, raised the proportionof patients with satisfactory hemoglobin A 1c levels to 60% fromabout 30%, and increased the proportion of patients with at leastone episode of symptomatic hypoglycemia to 57% from 23%. Notethat, in each case, the first number constitutes the "experimentalevent rate" (EER), and the second number the "control event rate"(CER). We will use the following terms and calculations to describethese effects treatment.

When the experimental treatment reduces the probability of a badoutcome (worsening - „diabetic retinopathy)

RRR (relative risk reduction). The proportionalreduction in rates of bad outcomes between



reduction in rates of bad outcomes betweenexperimental and control participants in a trial,calculated as [EER – CER]/CER accompanied by a 95%confidence interval (CI). In the case of worseningdiabetic retinopathy, [EER – CER]/CER = [13% -38%]/38% = 66%.

AAR (absolute risk reduction). The absolutearithmetic difference in rates of bad outcomes betweenexperimental and control participants in a trial,

calculated as [EER – CER], and accompanied by a 95%CI. In this case, [EER – CER] = [13% - 38%] = 25°(Thissometimes called the risk difference.)

NNT (number needed to treat). The number ofpatients who need to be treated to achieve one

additional favorable outcome, calculated as l/ARR andaccompanied by a 95% CI. In this case, 1/ARR = 1/25%= 4.

when the experimental treatment increases theprobability of a good outcome (satisfactory hemoglobin

A 1c levels)

RBI (relative benefit increase). The proportionalincrease in rates of good outcomes between experimental



increase in rates of good outcomes between experimentaland control patients in a trial, calculated as [EER – CER]/CER, and accompanied by a 95% confidence interval

(CI). In the case of satisfactory hemoglobin A 1c levels,[EER – CER]/CER = [60% - 30%]/30% = 100%.

ABI (absolute benefit increase). The absolutearithmetic difference in rates of good outcomes betweenexperimental and control patients in a trial, calculated as[EER – CER], and accompanied by a 95% CI. In the caseof satisfactory hemoglobin A 1c levels, [EER – CER]= [60%- 30%]= 30%

NNT (number needed to treat). The number of

patients who need to be treated to achieve one additionalgood outcome, calculated as 1/ARR and accompanied by a95% C is case, 1/ARR = 1/30% = 3.

When the experimental treatment increases theprobability of a bad outcome (episodes of hypoglycemia) )

RRI (relative risk increase). The proportionalincrease in rates of bad outcomes between experimental



increase in rates of bad outcomes between experimentaland control patients in a trial, calculated as [EER – CER]/CER, and accompanied by a 95% confidence interval(CI). In the case of hypoglycemic episodes, [EER-CER]= [57% - 23% ]/ 23% = 148%. (RRI is also used inassessing the impact of "risk factors" for disease.)

ARI (absolute risk increase). The absolute arithmeticdifference in rates of bad outcomes betweenexperimental and control patients in a trial, calculated as

[EER – CER], and accompanied by a 95% C1. In thecase of hypoglycemic episodes, [EER – CER] = [57% -23%] = 34%. (ARI is also used in assessing the impactof "risk factors" for disease.)

NNH (number needed to harm). The number of patients

who, if they received the experimental treatment, wouldresult in one additional patient being harmed, comparedwith patients who received the control treatment;calculated as I/AAR and accompanied by a 95% CI. Inthe case, 1/ARR=1/34% =3



Table A2.1

How to calculate LRs

Table A2.2

We can assume that there are four possible groups ofpatients as indicated ( a to d ) in the table From these



patients, as indicated ( a to d ) in the table. From thesewe can determine the “sensitivity” and specificity” asfollows:

– Sensitivity = a/(a + c) – Specificity = d/ (b + d)

We can now use these to calculate the likelihoodratio for a positive test result (LR+):

LR+ = sensitivity/ (1 – specificity)= [a/(a +c)]/[b/(b+d)] Similarly, we can calculate the likelihood ratio for a

negative test result (LR-):LR - = (1-sensitivity)/specificity

= [c/(a+c)]/[d/(b+d)] Positive predictive value = a/(a+b) Negative predictive value = d/(c+d)

Pre-test probability = (a+c)/(a+b+c+d)

SAMPLE CALCULATION



Suppose you have a patient with anemia and a

serum feritin of 60 mmol/L You come across asystematic review* of serum ferritin as a

diagnostictest for iron deficiency anemia, with the results

summarized in the following table.

Table A2.3

The results indicate that 90% or the patients with iron deficiencyanemia have a positive test result (serum ferritin < 65 mmol/L)



anemia have a positive test result (serum ferritin < 65 mmol/L).This is known as the “sensitivity” and is calculated as:

Sensitivity = a/(a+c) = 731/809 = 90%

The results also show that 85% of patients who do not have irondeficiency anemia have a negative test result. This is referred to asthe “specificity”, calculated as:

Spesitivity = d/(b+d) = 1500/1770 = 85%

From the sensitivity and specificity, the positive (LR+) and negative(LR-) likelhood ratios can be determinated

LR+ = sensitivity/(1-specificity) = 90%/15% = 6LR- = (1 – sensitivity/specificity) = 10%/85% = 0.12

The results indicate that 90% or the patients with iron deficiencyanemia have a positive test result (serum ferritin < 65 mmol/L)



anemia have a positive test result (serum ferritin < 65 mmol/L).This is known as the “sensitivity” and is calculated as:

Sensitivity = a/(a+c) = 731/809 = 90%

The results also show that 85% of patients who do not have irondeficiency anemia have a negative test result. This is referred to asthe “specificity”, calculated as:

Spesitivity = d/(b+d) = 1500/1770 = 85%

From the sensitivity and specificity, the positive (LR+) and negative(LR-) likelhood ratios can be determinated

LR+ = sensitivity/(1-specificity) = 90%/15% = 6LR- = (1 – sensitivity/specificity) = 10%/85% = 0.12

Thus from your calculation of LR+, you determine that your patient‟spositive test result would be about six times more likely to be seenin someone with iron deficiency anemia than in someone withoutthe disorder.

Calculation of odds ratio/relative risk



The table below can be used to calculate the odds

ratio/relative risk for the use of trimethoprim-sulfamethoxazole prophylaxis in cirrhosis:

Table A2.4

critical apraisal.ppt

Documents