systematic review and meta-analysis of diagnostic test studies

Systema(c review and meta-‐analysis of Diagnos(c Test Studies

Kuan-‐Fu Chen CGMH EM

SR/MA workshop January 8, 2013

Objec(ves

•  Differences of diagnos(c SR/MA? •  Steps of diagnos(c SR/MA •  Formula(ng a focused ques(on •  Reviewing guidelines: QUADAS •  Literature searching •  Evalua(on of studies •  Data extrac(on •  Data synthesis

1/8/13 KFC CGMH EM Dx SR/MR talk 2

Difference of Dx SR/MA

•  ID studies, assessing bias •  Method to combine results •  Paired of summary sta(s(cs to pool •  Design of diagnos(c studies


Diagnos(c studies

•  Defini(on – Dia + gnosis = apart/separated + knowledge – Dia: Greek ‘through’ – Gnosis: Greek ‘knowledge’ – To reduce uncertainty

•  Purposes – Screening, triage, add-‐on or replacement tests, predict outcomes or monitor dz process

– Purpose must be considered during evalua(on


Phases of Dx test evalua(on

•  Prelim evalua(on (case-‐control) –  I: whether results different for pts ± dz –  II: whether dz more likely to have + results

•  Accuracy & probability of condi(on (cross-‐sec(onal/cohort study) –  III: how well dis(nguishes btw pts ± dz with suspicion –  IV: how informa(ve as add-‐on

•  Health outcomes (Randomized or before-‐ader study) –  V: whether leads to beeer outcomes –  VI: acceptable costs


Phases of biomarker evalua(on


Objec(ves



Steps of (diagnos(c) SR/MA

•  Formula(ng a focused ques(on •  Reviewing guidelines •  ID databases/sources of studies •  Run searches and save cita(ons •  First screen by two reviewers •  Second screen by two reviewers •  Data extrac(on and quality assessment •  Data analyses


Objec(ves



PICO(S) for of diagnos(c studies •  Popula(on

–  What are the characteris(cs of the pa(ents? –  What is the condi(on that may be present?

•  Interven(on (diagnos(c test) –  Which diagnos(c test am I considering?

•  Comparison –  What is the diagnos(c gold standard?

•  Outcome –  How likely is the test to predict/rule out this condi(on?

•  Study design/sekng –  What study design would provide the best level of evidence? –  What clinical sekng would this study apply to?


Objec(ves



Guidelines for (SR/MA) studies

•  QUA D A S – QUAlity of Diagnos(c Accuracy research Studies in SR

•  STA R D –  STAndard for Repor(ng of Diagnos(c accuracy

•  RE MARK –  REpor(ng recommenda(ons for tumor MARKer prognos(c studies

•  P R I S M –  Preferred Repor(ng Items for SR/MA


QUADAS (2)

•  Four phases to establish tool for each review •  Four key domains to review •  Three sec(ons in each domain for risk of bias


QUADAS – phases

•  Review ques(ons •  Review specific tailoring •  Flow diagram •  Judgments on bias and applicability


QUADAS – phases

•  Review ques(ons – Four-‐part ques(on:

•  (P) Pa(ents, •  (I) Index test(s), •  (C) Reference standard, and •  (O) Target condi(on

– Diagnos(c pathway: •  Sekng, inten(on, pa(ent presenta(on, and/or prior tes(ng


QUADAS – phases

•  Review specific tailoring


QUADAS – phases

•  Flow diagram


QUADAS – phases

•  Judgments on bias and applicability – Three sec(ons in each domain for risk of bias

•  Informa(on to support •  Signaling ques(ons •  Judgment of risk of bias

– Applicability •  Similar but not including signaling ques(ons •  Refer to first phase (review ques(on)


QUADAS – domains

•  Pa(ent selec(on •  Index test •  Reference standard •  Flow and (ming


QUADAS – domains •  Pa(ent selec(on –  Signaling ques(ons

1.  Was a consecu(ve or random sample of pa(ents enrolled? 2.  Was a case-‐control design avoided? 3.  Did the study avoid inappropriate exclusions?

–  Risk of bias •  Spectrum bias (one of selec(on biases) (lab)? •  Excluding difficult to diagnose; liele diagnosis uncertainty:

–  overop(mis(c •  Exclusion of “red flags”: underes(ma(on •  Selec(on towards more severe manifesta(ons

–  Increase prevalence –  Bias in any direc(on


QUADAS – domains

•  Pa(ent selec(on – Applicability concerns

•  Target vs. Study popula(on: generalizability •  Any discrepancy re: severity, demographics, presence of differen(al diagnosis, comorbidity, sekng and previous tes(ng protocol

– Supplemental: prospec(ve vs. Retrospec(ve? •  Retrospec(ve:

–  No verifica(on of Dz status –  Same reference standard for all? –  Same assessors for reference standard for all?

Target'Population

Study&population

Target'Population

Study&population

Target'Population

Study&population


QUADAS – domains

•  Index test – Signaling ques(ons

1.  Were the index test results interpreted without knowledge of the results of the reference standard?

2.  If a threshold was used, was it pre-‐specified?

– Risk of bias •  Ques(on 1: blinding, subjec(vity •  Ques(on 2: overop(mis(c


QUADAS – domains

•  Index test – Applicability concerns

•  Same index tests?

– Supplemental: (QUADAS 1) •  Reproducibility: for all tests, different phases


QUADAS – domains

•  Reference standard – Signaling ques(ons

1.  Is the reference standard likely to correctly classify the target condi(on?

2.  Were the reference standard results interpreted without knowledge of the results of the index test?

– Risk of bias •  Ques(on 1: direc(on? Misclassifica(on bias (textbook?) –  Bias confounding and role of chance slide

•  Ques(on 2: similar to index test, Incorpora(on bias


QUADAS – domains

•  Reference standard – Applicability concerns:

•  target condi(on defined by the reference standard does not match the ques(on? •  Example: U/C for UTI with different cutoffs


QUADAS – domains

•  Flow and (ming –  Signaling ques(ons

1.  Was there an appropriate interval between index test and reference standard?

2.  Did all pa(ents receive the same reference standard? 3.  Were all pa(ents included in the analysis?

–  Risk of bias •  Ques(on 1: misclassifica(on bias (if delay or treatment started). Direc(on? Acute vs. Chronic

•  Ques(on 2: Verifica(on bias: overes(mate •  Ques(on 3: lost to follow-‐up: Direc(on? Could be either direc(on (selec(on bias)


Objec(ves



Literature searching

•  Electronic database – Less produc(ve and more difficult

•  Different strategies – Different indexing terms/text words – MeSH: diagnosis – Textwords: specificity, sensi(vity


Objec(ves



Evalua(on of studies

•  Selec(on of sample. –  Ideal: consecu(ve/randomly selected

•  Reference test •  Blinding •  Quality repor(ng •  Evidence of bias: –  table of rela(ve DOR in different study design

•  Incorpora(ng QA in SR: Checklist


Objec(ves



Data extrac(on •  Accuracy expression

–  Sensi(vity/specificity: Based on cut-‐off –  ROC curve –  PPV → influence of prevalence

•  Choice of parameters: –  May consider DOR instead of Sensi(vity/specificity, LR

•  DOR (Diagnos(c Odds Ra(o) also = LR(+)/LR(-‐) –  DOR 25: LR(+) 5, LR(-‐) 0.2, DOR 100: LR(+) 10, LR(-‐) 0.1

•  However, discarded informa(on •  Predic(ve value: high heterogeneity→prevalence varia(on

–  Post-‐test probability can be used instead •  Examples of abstract/ar(cle review form


Objec(ves



Data synthesis

Meta-regression DOR on

characteristics, investigate source of

heterogeneity

LM methodtest of

asymmetry,SROC or pool ROC curves

Pool sensitivity & specificity

Test of cut-off effect

Test of heterogeneity

heterogenous

homogenous

no

yes sym

Bayesian adaptation

SROC

asym

SROC: summary ROC LM method: Lieenburg – Moses regression method DOR: diagnos(c odds ra(o


WORKSHOP TIME! Systema(c review and meta-‐analysis of Diagnos(c Test Studies


Objec(ves

•  Understand and workout poten(al bias during diagnos(c SR/MA – Spectrum bias – Misclassifica(on bias –  Incorpora(on bias – Lost of follow up – Selec(ng threshold – verifica(on bias

•  Use QUADAS-‐2 form to evaluate one study


Selec(on of pa(ent (spectrum bias)

•  Excluding difficult to diagnose – Overop(mis(c or underes(ma(ng, why?

•  Excluding ‘red flag’ – Overop(mis(c or underes(ma(ng, why?

•  Evaluate and discuss the reasons in examples


Example 1: Centor’s CPR


Example 2: BNP


Misclassifica(on bias

•  Use those tables to discuss the reasons – PCR vs. B/C for sepsis diagnosis – PCT vs. B/C for sepsis diagnosis

•  Use blood culture as example – Discuss interval and reference test issue


Misclassifica(on bias Cases: B/C + Control: B/C -‐ Total

PCR + 100 40 150

PCR -‐ 50 60 100

150 100 250

False Sensi(vity=66% Specificity=60%

Cases: real sepsis Control: real ctrl Total

PCR + 110 40 150

PCR -‐ 40 60 100

150 100 250

True Sensi(vity=73% Specificity=60%

DifferenAal misclassificaAon bias Say Sensi(vity of B/C = 80%, specificity = 90%, and related to PCR results


Misclassifica(on bias Cases: B/C + Control: B/C -‐ Total

PCT + 100 40 150

PCT -‐ 50 60 100

150 100 250

False Sensi(vity=66% Specificity=60%

Cases: real sepsis Control: real ctrl Total

PCT + 125 25 150

PCT -‐ 63 37 100

188 62 250

True Sensi(vity=66% Specificity=60%

Non-‐DifferenAal misclassificaAon bias Say Sensi(vity of B/C = 80%, specificity = 75%, and not related to PCT results


Incorpora(on bias

•  Example: SIRS vs. WBC for sepsis diagnosis – Overop(mis(c or underes(ma(ng, why?


Lost of follow up

•  Example: transferred to other hospital – Overop(mis(c or underes(ma(ng, why?


Selec(ng threshold

•  Example: – Post-‐hoc determina(on of BUN/Cre > 15 as risk factor in acute stroke

– Overop(mis(c or underes(ma(ng, why?


Verifica(on bias

•  Example: PSA – Verifica(on and incorpora(on biases in studies assessing screening tests: prostate-‐specific an(gen as an example (Gupta & Roehrborn)

– Overop(mis(c or underes(ma(ng, why?


Verifica(on bias Disease + Disease -‐ Total % Verified

PSA + 231 32 263 61%

PSA -‐ 27 54 8 37%

Total 258 86 344

Sensi(vity=90% Specificity=63%

Corrected Disease + Disease -‐ Total

PSA +

PSA -‐

Total

Sensi(vity=__% Specificity=__%


Verifica(on bias Disease + Disease -‐ Total % Verified

PSA + 231 32 263 61%

PSA -‐ 27 54 8 37%

Total 258 86 344


Disease + Disease -‐ Total

PSA + 377 52 429

PSA -‐ 74 147 221

Total 451 199 650



Verifica(on bias


systematic review and meta-analysis of diagnostic test studies

Documents

srma studies

diagnosc studies populaon

dierence of dx srma

quadas literature

diagnosc studies denion

diagnosc gold standard

reporng of diagnosc

quadas phases judgments