NERDCAT-ObsAclinician’sguidetoappraisingobservational

studies(cohort&case-controlstudies)

TableofContentsCHECKLIST Clinicalquestion,eligibilitycriteria,&generalizability 2Generalizability 2Internalvalidity 3Results 4

EXPLANATION&EVIDENCE Bigpicture:Comparison&contrastofRCTs&observationalstudies 5Biases&confounding:Explanation 6-7Minimizingbias&confounding:Strategies&shortcomings 8Strengthofassociation&othercaveats:Interpretingtheresults 9

ThelatestversionofthistoolandotherNERDCATscanbefoundatnerdcat.orgFullreferencestosupportingliteratureandarticlesusedasexamplesinthisdocumentcanbefoundat:http://nerdcat.org/useful-references/Note:Thiscriticalappraisaltoolbuildsfromthefundamentalslearnedfromreadingandappraisingthemorestraightforwardrandomizedcontrolledtrial(RCT).Formoreonfundamentalandadvancedtopicsinclinicaltrials,pleaseseeNERDCAT-RCT.

2

Articletitle:______________________________________________________________________________CLINICALQUESTION&STUDYFLOW Datasource DefinitionD Studycharacteristics:

• Studytype:• Setting:• Yearofstudy:

P Averagepatient:• Age:• Sex:• Race:• Pathology,stage/severityofdisease:• Comorbidities:• Previousinterventions:

I/C Drug,dose,route,duration

Co-interventions:

O

GENERALIZABILITY 1. Doesmypracticesettingdiffersignificantlyfromthatin

thetrials?Somequestionstoconsider:• Era:Same/similardiagnosticcriteriausedfor

disease/outcome?• Setting:1o,2o,or3ocare?• Singlecentreormulticentrestudy?• Country:WerethereCanadiansinthestudy?

2. Weredifferencesbetweenstudyparticipants&mypatients(i.e.SCRAPPcharacteristics)clinicallysignificant?

3. Aretheinterventionsevaluatedinthestudysimilartothoseavailableinmypractice?

4. CanIreadilymeasuretheoutcomesevaluatedinthestudy?

3

INTERNALVALIDITY–Aretheresultsreliable?Answeringslightlydifferentquestionsbasedondesign:Cohort:Asidefromtheexposureofinterest,didtheexposed&non-exposedgroupsstart&finishwithasimilarriskfortheoutcome?Case-control:Didthecases&controlshaveasimilarriskforbeingexposedinthepast?5. Confounding(including‘confoundingby

indication’)Cohort:Doexposed/non-exposedhavethesamelikelihood&severityofprognosticfactorsknowntobeassociatedwiththeoutcome?Case-control:Didthecases&controlshaveasimilarenoughriskforbeingexposedinthepast?

Knownriskfactorsforoutcome(circlethoseaccounted-forinstudy):

6. Reversecausation(a.k.a.protopathicbias) 7. Misclassification&detectionbias(including

recallbias)Cohort:Weresimilarmethodsusedtodetecttheoutcome?Didsimilarcircumstancesleadtoinvestigationfortheoutcome?Case-control:Werethecircumstances/methodsfordeterminingexposuresimilarforcases&controls?

8. AttritionbiasCohort:Wasfollow-upsufficientlycompleteinbothexposedandnon-exposedparticipants?

9. Immortal-timebiasCohort:Wasthereadifferentiallagtimebetweenstartoffollow-upandeligibilityfortheexposureornon-exposuregroup?

10. Weremethodsusedtominimizeaboveissueswithconfounding/biassufficient(determineforeachissueidentified)?

Methodstominimizeconfounding/biasinobservationalstudies:1. Design:Restriction,matching(±propensity

score,instrumentalvariable)2. Analysis:Stratification,multivariableregression

(±propensityscore)

4

RESULTS11. StrengthofassociationHowlargeisthemagnitudeofassociation?• <2:Small(mayberesultofresidualbiasor

confoundingfromstudydesign)• 2-5:Moderate• >5-10:Large(morelikelytorepresentatrue

associationbetweenexposureandoutcome)Isthereadose-responsegradient(i.e.increasingHR/OR/RRwithlargerdosesorlongerdurationofexposure)?

12. Precision

Informationobtainedfromtheconfidenceinterval:• Aretheresultsstatisticallysignificant?• Doesthemagnitudeofassociationdiffer

betweenthelowerandupperboundoftheconfidenceinterval?

• Areresultsclinicallyimportantatbothboundsoftheconfidenceinterval?

5

Randomization

Clin

ician

’s ch

oice

Clinician’s choice

RCT Cohort Case-ControlFlow

Prospective:AswithRCTs,followspatientforwardintimefrombaselinetoendoffollow-up.Retrospective:Usehistoricaldata(e.g.fromaregistry)toreconstructpatientcharacteristics,EXPOSUREstatus&OUTCOME.

PatientsareidentifiedbasedonwhetherornottheyexperiencedanOUTCOMEofinterest.Then,welooktoseeiftheywereexposedtothedrugs/toxins/etc.ofinterestpriortoexperiencingthisoutcome.

Pros • Lowestsusceptibilitytobiaswhendoneright

(highinternalvalidity)

• Canestablish,betweenaninterventionandan

outcome:

o Cause-and-effectrelationship;o Dose-responserelationship;o Temporalrelationship

• Goodgeneralizability

o “Real-world”populationiswellrepresentedo Cangetarealisticandaccurateincidenceof

OUTCOMESofinterest

• MeasureseffectofEXPOSUREon>1OUTCOMES

• Canexploredose-response&temporal

relationshipsbetweenanEXPOSURE&

OUTCOME

• Notsusceptibletorecallbias

• MeasuresassociationofrareOUTCOMES

o WouldotherwiserequiremassiveRCT/cohort

• Canevaluateassociationofmultiple

EXPOSURES/variablesonasingleOUTCOME

• Abletoevaluateassociationbetween

OUTCOMESthattakealongtimetodevelop

(e.g.cancers)andEXPOSURES

Cons • Oftenpoorgeneralizability(lowexternalvalidity)

• Samplesizeusuallylimitsdetectionofrare

outcomes

• Susceptibletoconfounding,selection/allocationbias,performancebias,detectionbias

• Highloss-to-follow-upmakesitdifficulttofind

associationsbetweenanexposurewith

outcomesthattakealongtimetodevelop(e.g.

cancer)

• Retrospective:

o Maynothavemeasuredpatient

characteristicsofinterest(notabletoadjust

forresidualconfounding)

o Mayhavehighincompletionofdata

• Oftentoounreliableenoughtodetermine

causation

• Susceptibletoconfoundingandsamebiasesascohortsplus,insomeinstances,additional

biases(e.g.recallbias)

Exposure

(Treatment)

Outcome

Nooutcome

Noexposure

(Control)

Outcome

Nooutcome

Exposure

(Treatment)

Outcome

Nooutcome

Noexposure

(Control)

Outcome

Nooutcome

Case

(outcome)

Exposure

(treatment)

Noexposure

Control

(nooutcome)

Exposure

(treatment)

Noexposure

6

INTERNALVALIDITY:Althoughobservationalstudiesincludesomeuniquebiases,themostimportantonesarethesameaswithRCTs,butwiththeirowntwistsandways

ofhandlingthem.

Bias/confounder Explanation Example

Confoundingbyindication(checklist

question#5)

• Aformofselectionbias;

• Themostcommonthreattovalidityin

observationalstudies;

• Occurswhenanindividualismorelikely

toreceiveatreatmentbecauseofan

underlyingindicationthatalsohappens

tobeariskfactorfortheoutcomeunder

study.

ConsiderasituationwhereapatientwithrheumatoidarthritisdevelopsanupperGIbleedonnaproxen.

Despitethisunpleasantexperience,sheinsistsoncontinuingsomesortofNSAIDforongoing,disabling

paininbothupperlimbs.HerphysicianprescribescelecoxibandaPPIinordertominimizeherriskof

recurrentupperGIbleed,butsheunfortunatelydiesfromgastricperforation1yearlater.

• If,inpractice,celecoxibismostlyprescribedtopatientsathighestriskofarecurrentbleed,as

above,acohortstudythatdoesnotaccountfor(orcannotacquireinformationabout)previous

upperGIbleedsmayfalselyconcludethatcelecoxibhasasimilarorgreaterriskofupperGIbleed

comparedtonon-selectiveNSAIDs.

Reversecausation(a.k.a.protopathicbias)(checklist

question#6)

Occurswhenaprecursortotheoutcomeof

interestorasubclinicalversionofitleadsto

theexposureofinterest.Thefull-blown

versionoftheoutcomeofinterestpresents

itselfafteroutcome,whichmaythenleadto

thefalsebeliefthattheexposurecausedthe

outcome.

Considerasituationwhereanobesepatientwithundiagnosedcoronaryarterydisease(CAD)is

experiencingintermittentretrosternalchestpain(whichisactuallystableangina).Hegoestothewalk-

inclinic,isdiagnosedwithGERDandgivenaPPI.HetakesthePPIfor1weekandbelieveshe

experiencesrelief,butwakesuponemorningwithintractablechestpain.Hecalls9-1-1,getstakento

thelocalhospitalandisdiagnosedwithNSTEMI;

• Inotherwords,subclinicalmanifestationoftheoutcomeofinterest(stableanginafromCAD)ledto

exposure(PPIuse),whichwasfollowedbyanoutcome(MI);

• Ifthissituationisprevalentenough,acohortstudylookingattheassociationbetweenPPIuseand

MImayfalselyconcludethatPPIscauseMIs.

Misclassification(checklist

question#7)

Canoccurin2ways:

• Random(i.e.non-differential):Subjectsin

bothgroupshaveequalopportunityto

bemisclassified.Thistypeof

misclassificationleadstoimprecisionin

themeasureofassociation,which

reducesthestudy’spowerandabilityto

findadifference(false-negative);

• Biased(i.e.differential):Subjectsin1

grouparemorelikelytobemisclassified;

o Leadstoexaggerationor

attenuationoftheestimateof

association

Randommisclassification:Considerthathospitalphysiciansfrequentlyomitacutekidneyinjury(AKI)

fromtheirdocumentation,whichinturnleadstoomissionfromadministrativedatabasesforupto85%

ofpatients.AmJKidney2011;57:29-43

• AcohortstudyusingadministrativedataexaminingtheassociationofAKIanduseofantiviralssuch

asvalacyclovirwouldnotnecessarilybeatriskofanybiasfromthismisclassification,butwould

requireaverylargesamplesizeinordertooffsettheresultingimprecisionandriskforafalse-

negativeresult.

Biasedmisclassification:ConsiderthesameissuewithAKIasabove,butforanadministrativecohort

studyexaminingtheassociationbetweenNSAIDuseandAKI.SinceNSAIDsareknownnephrotoxins,

physiciansmaybemorepronetodocumentthepresenceofAKIduringhospitalstayinNSAIDusers

comparedtonon-users.TheresultwouldbegreatermagnitudeofunderreportingofAKIinNSAIDnon-

users,creatingbiasedmisclassification.

7

Recallbias(checklist

question#7)

• Formofbiasedmisclassificationthatcan

occurinretrospectivecohortandcase-

controlstudiesthatinvolveparticipant

interviewafter-the-facttoinquireabout

previousexposures;

• Occursbecauseindividualswhodevelop

adiseaseorexperienceanoutcometend

tohavebetterrecallofpotentially

dangerousexposuresthatcouldhaveled

totheoutcome(becauseitishuman

naturetotrytodeterminethecauseof

badfortune).

Considerachildthatdevelopsasyndromeofunclearetiology/pathophysiologyat3yearsofage.The

parentsofthischild,searchingforthecauseofthismysteriousillness,seeksoutandbecomesattuned

toeveryexposure,fromchemicaltoenvironmental,priortodevelopmentofthesyndrome.Onthe

otherhand,theparentofaperfectlyhealthy,average3year-oldwouldhavenoreasontoobsessover

andmemorizethesefacts.

• Anobservationalstudyusingasurveyofparentstoidentifytheassociationbetweenfrequencyof

acetaminophenusebeforeage3anddevelopmentofthissyndromemayfalselyidentifysuchan

association.

Immortal-timebias(checklist

question#9)

• Uniquetocohortstudies;

• Formofbiasedmisclassification;

• Occurswhenthereisa“lagtime”

betweenbeginningoffollow-up&when

individualbecomeseligibleforoneofthe

studygroups(usuallytheEXPOSURE

group);

o Duringthelagtime,patientsin

theEXPOSUREgroupcannotdie

orexperiencethestudy

outcome,givingtheman

apparentsurvivaladvantage;

Considerthecaseoftheimpactofstatinuseondevelopmentoftype2diabetes.Weknowfrommeta-

analysesofRCTsthatstatinsmodestlyincreasebloodglucoseinadose-dependentmanner,leadingto

anincreaseindiagnosisofdiabetes.JInvestigMed2009;57:495-9,

Lancet2010;375:735-42,JAMA2011;305:2556-64

• Priortothisknowledge,however,acohortstudysuggestedanassociationbetweenstatinuseand

decreasedprogressionofdiabetestoinsulindependence.DiabetMed2004;21:962-7

• Asubsequentre-analysisofthisstudy,BMJ2010;340:b5087

however,exemplifiedwhyimmortal-timebias

completelyexplainedthisapparentbeneficialassociation,duetothefollowing:

1. Thedefinitionof“statinuse”required≥1yearoffillingaprescriptionforastatin;2. Anypatientwhofilledprescriptionsforastatinfor≤364dayswasincludedinthegroupof

“NON-users”;3. Anypatientwhostartedinsulinwithinthe1

styearoftakingstatinswasincludedinthe

groupof“NON-users”.o Asaresultoftheabove,manypatientsexposedtostatinsandexperiencingthestudy

outcomeweremisclassifiedas“non-users”.

8

INTERNALVALIDITY:Methodsusedtominimizeconfounding&biasinobservationalstudies Explanation Issuesthatareminimized

Design

High-qualitydatasource

• Standardized,adjudicatednon-differentialrecordingofallexposure,covariablesandoutcome

dataminimizesbias;

• Hierarchyofdataquality:Patientrecords>codesfromclinicaldatabase>codesfrom

administrativedatabase≥patientself-report.

All

Cleardefinitionofexposure&outcome(orcases&controls) • Misclassification

• Immortal-timebias

Restriction Restrictingeligibilityforastudywithinclusion/exclusioncriteriacanminimizeexpectedbaseline

(&subsequenttreatment)differencesbetweengroups.

• Confounding

• Allocation&performancebias

• Attritionbias(restrictedtopatientswith

completefollow-up)

Matching • Investigatorsselectkeycharacteristicslikelytohaveaconfoundingeffectontheassociation

understudy(generallyage,sex,&1-2otherfactorssuchasacomorbidityscore).Theythen

matchapatientintheexposuregroupto≥1non-exposurepatients(orcases&controlsfor

case-controlstudies),minimizingbaselinedifferences;

• Maycause“overmatching”(iffactorisnotaconfounder,decreasesstudypower)&residual

confounding(ifnoothermethodsusedtominimize);

• Canalsomatchbasedonpropensityscoreorinstrumentalvariable

• Confounding

• Allocationbias

Analysis

Stratification Atypeofsubgroupanalysisbasedon1+characteristicstodeterminewhethertheassociation

betweenexposureandoutcomeisconfoundedbythesubgroupcharacteristic.

• Confounding

• Allocationbias

Multivariableregression

(i.e.statisticaladjustment)

Statisticaladjustmentthataccountsformultiplecovariablesthatcouldaffecttheoutcome&

confoundorbiastheassociation.

• Confounding

• Allocation&performancebias

Sensitivityanalyses

• Activecontrol(s):Definecontrolgroupasindividualsreceivingsimilarinterventionfor

indicationunderstudy(e.g.differentantibioticforUTI,H2RAinsteadofPPI,etc)

• Tracer:Repeatanalysisreplacingexposurewithsimilardrugnotexpectedtocauseoutcome

• Confounding

• Allocationbias

Misc.

Propensityscore • Multivariableregressionisusedtocalculateaprobability(orpropensity)score(0-100%)foranyindividualstudyparticipanttobeexposedbasedontheirbaselinecharacteristics

• Thisscorecaneitherbeusedasacharacteristicformatching,orasacovariableinregression

• Confounding(especiallybyindication)

• Allocationbias

9

RESULTS:Additionalcaveats&considerationsChoiceofmeasureof

association

ResultsofcohortstudiescanbepresentedasHR,ORorRR,&NNTs/NNHscanbecalculated;

Resultsofcase-controlstudiesCANNOTbepresentedasHRorRR;onlyORscanbecalculated.

• AbsoluteriskandNNTscannotgenerallybecalculatedfromcase-controlstudiesbecausetheunderlyingpopulationincidenceisunknown;

• Exception:Nestedcase-controlstudies,inwhichcasesandcontrolsareidentifiedfromadefinedpatientcohort(fromacohortstudyora

RCT),suchastheFraminghamHeartStudyortheGUSTO-1RCTsamplepopulation.

Subgroupanalysesin

observationalstudies

CanbehandledinthesamewayasinRCTs(seeNERDCAT-RCT),exceptwithevenmorecautionandskepticism(i.e.they’reoftennotworth

yourtime).

• Exception:Stratification,whichissimilartosubgroupanalysis,isusedtominimizebiasandconfoundinginobservationalstudiesandis

acceptableaslongasitisusedtobolsterthestudy’sprimaryanalysisratherthancreateadditionalconclusions.“Trendstowardsstatistical

significance”

Don’teventhinkabout“trendstostatisticalsignificance”inobservationalstudies!Itisfarmoreprobablethatunknownconfoundersaccount

fora“trendtowardsbenefit/harm”thantheactualintervention.

ApplythesamelogicaswithRCTsintermsoflookingattheconfidenceintervaltoseewhetheritrulesoutaclinicallyimportantdifference.

InterpretingOddsRatiosFirst,let’sstartwitha2x2tablefromaRCT(nerdlmps.files.wordpress.com/2013/06/nerd-2-asa-after-ugib-rct.pdf):

AnnInternMed.2010;152:1-9 Deadat8weeks(outcome)

Aliveat8weeks(nooutcome)

Totalpatientsingroup

ASArestarted(exposure) 1(a) 77(b) 78(a+b)ASAnotrestarted(noexposure) 10(c) 68(d) 78(c+d)

Totaldead/alive 11(a+c) 145(b+d) Risk(i.e.absoluterisk)isthelikelihoodofanoutcomeoccurringgivenanexposureexpressedasthe%of(#participantswiththeoutcome)/(total#inthatgroup)

• TheriskofdyingifASAisrestartedisa/(a+b)=1/78=0.0128• TheriskofdyingifASAisnotrestartedisc/(c+d)=10/78=0.128

Relativerisk(orriskratio)istheratiooftheriskofoutcomewithexposureversusnoexposure• Therelativerisk(RR)ofdeathifrestartingASAversusnotrestartingASAis==0.1

Oddsaretheratiooflikelihood,givenanexposure,ofanoutcomehappeningversusnothappening

• TheoddsofdyingversusnotdyingifrestartingASAarea/b=1/77• TheoddsofdyingversusnotdyingifnotrestartingASAarec/d=10/68

Theoddsratio,asthenamesuggests,isaratiooftheoddsofanoutcomewithexposureversusnoexposure• Theoddsratio(OR)fordeathifrestartingASAversusnotrestartingASAis=,whichcanberearrangedto

ad/bc=(1*68)/(77*10)=0.088Whentheoutcomeisrare(<10-15%)orextremelycommon(>85-90%)suchthata+b(thedenominatorforrisk)≈b(denominatorforodds)andc+d≈d,RRandORareVERYsimilar,thereforeORcanbeusedasanestimateofRR.Theclosertheincidenceoftheoutcomeisto50%,thelesstheRRandORaresimilar.

• Intheexampletotheright,theriskoftheoutcomeintheexposuregroupis0.50• Theriskoftheoutcomeifnotexposedis0.70• Withexposure,theoddsofdeathvsnodeathis1• Withnon-exposure,theoddsofdeathvsnodeathis2.33• TheresultingRR(0.71)andOR(0.42)arequitedifferent,andthereforetheORisapoorestimateofRR.

Whycan’tweuseRRincase-controlstudies?Incase-controlstudies,theoutcomeofinterestisusuallyrare(≤1%),&thusthenumberofcasesavailableinanygivenpopulationisgenerallymuchsmallerthanthenumberofcontrols.Investigatorsidentifyallcasesinacertaincohort,andthenwillpickasampleofthecontrols.Typically,casesarematchedtoanequalorgreaternumberofcontrolpatientsbasedoncertaincharacteristics(age,gender,comorbidity,co-medication,etc.).Thus,theprevalenceoftheoutcomeinthestudypopulation[a+c/(a+b+c+d)]isdependentonhowmanycontrolsarematchedpercase.Risk,butnotodds,issensitivetounderlyingprevalence,asdemonstratedbelow:

1:1Match Case Control Total “Prevalence”instudysample:50/100=50%

RR:0.875/0.25=3.5OR:7/0.33=21

Exposure 35(a) 5(b) 40(a+b)Noexposure 15(c) 45(d) 60(c+d)

Total 50(a+c) 50(b+d) 1:10Match Case Control Total “Prevalence”in

studysample:50/550=9%

RR:0.41/0.03=13.7OR:0.7/0.033=21

Exposure 35(a) 50(b) 85(a+b)Noexposure 15(c) 450(d) 465(c+d)

Total 50(a+c) 500(b+d)

e.g. Outcome Nooutcome TotalExposed 50(a) 50(b) 100(a+b)Notexposed 70(c) 30(d) 100(c+d)

a/bc/d

1/7710/68

a/(a+b)c/(c+d)

1.28%12.8%