Applying the GRADE approach to diagnostic technologies

Holger Schünemann

for the GRADE working group

Today’s talk

1) Focus on patient important outcomes

2) Factors influencing the quality of evidence in diagnostic recommendations

GRADE for diagnosis

Shares the fundamental logic of assessment for treatment

However, assessments present unique challenges

Examples and solutions for how to deal with challenges

Focus on importance to patients and consumers!

Summary Quality of evidence reflects our confidence

that estimates of benefits and downsides from a diagnostic strategy generated from research are correct.

Consideration of the directness of evidence is based on how confident we are of the relation between being classified correctly (as a true positive or negative) or incorrectly (as a false positive or negative) and patient-important consequences.

Recommendation depends on the balance between desirable and undesirable effects of the diagnostic test or strategy in terms of patient-important outcomes.

Testing makes a variety of contributions to patient care Clinicians use tests that are usually referred to as

“diagnostic” signs and symptoms, imaging, biochemistry,

pathology, and psychological Some tests naturally report positive and negative

results (pregnancy) Other tests report their results in categories (e.g.

imaging) Today we assume a diagnostic approach that

ultimately categorizes test results as positive or negative

Purpose of a test

Triage to minimize use of an invasive or expensive

test Add-on

to improve diagnosis beyond what is already done

Replacement to replace test that is harmful or costly

Bossuyt et al. BMJ 2006

What is the sensitivity, specificity (accuracy), or likelihood ratios of multislice spiral computed tomography (CT) of coronary arteries compared with conventional invasive angiography?

Test accuracy is a surrogate for patient important outcomes

When clinicians think about diagnostic tests, they focus on their accuracy

Underlying assumption: obtaining a better idea of whether a target condition is present or absent will result in superior patient management and improved outcome.

Test and treatment threshold

Diagnostic tests are used in patients suspected of disease to exclude or confirm a diagnosis.

Purpose of the test

Identify the limitations for which the (new or) alternative test offers a putative remedy eliminating a high proportion of false

positive or negative results, enhancing availability, decreasing invasiveness, or decreasing cost

Identification of sensible clinical questions: patients, diagnostic intervention, comparison, management, and outcomes of interest

Bossuyt et al. BMJ 2006

Sensible clinical questionPopulation: In patients suspected of

coronary artery disease Intervention: does multislice spiral

computed tomography (CT) of coronary arteries

Comparison: compared with conventional invasive coronary

angiography Outcomes: lower complications with

acceptable rates of false negatives (associated with coronary events) and false

positives (leading to unnecessary treatment andcomplications)?

Study designs for diagnosis

If a test fails to improve important outcomes: no reason to use it, whatever its accuracy

Best way to assess diagnostic strategy: randomized controlled trial in which investigators randomize patients to different diagnostic strategies

Study designs

Focus on: mortality, morbidity, symptoms, and quality of life

GRADE approach for treatment or intervention

Study designs II

Patient benefit required Inference from accuracy data that a diagnostic

test or strategy improves patient-important outcome requires availability of effective management strategies

Others: Reduction in test-related adverse effects Exclusion of a disease and reduction in

anxiety Confirming a diagnosis improves patient

well-being from the prognostic information it imparts

Patient benefit required

Genetic testing for Huntington’s chorea reassurance that a patient will not

suffer from the condition ability to plan for future knowing

that patient will sadly fall victim Ability to plan is analogous to an

effective treatment

Almqvist et al Clin Gen 2003

Example of new test and reference test or strategy

Putative benefit of new test

Diagnostic accuracy Patient Outcomes and expected impact on management for the following test outcomes

Sensitivity Specificity

True positives

False positives

True negatives

False negatives

Helical CT for renal calculus compared with intravenous pyeolgram

Detection of more (but smaller) calculi

greater equal Presumed influence on patient important outcomes

Certain benefit for larger stones, for smaller stones the benefit is less clear and unnecessary treatment can result

Likely detriment from unnecessary additional invasive tests

Almost certain benefit from avoiding unnecessary tests

Likely detriment for large stones, less certain for small stones More testing

Directness of the evidence (test results) for patient-important outcomes

Some uncertainty

No uncertainty

No uncertainty

Major uncertainty

Balance between presumed patient outcomes, complications and cost: Less complications and downsides compared to IVP would support the new test’s usefulness, but the balance between desirable and undesirable effect not clear in view of the uncertain consequences of identifying smaller stones.

Today’s talk

1) Focus on patient important outcomes

2) Factors influencing the quality of evidence in diagnostic recommendations

Factors that decrease the quality of evidence (and how they differ from treatment approach)

Study designDifferent quality criteria for accuracy studiesValid accuracy studies: Diagnostic uncertainty Consecutive patients Evaluators should be blinded

Factors that decrease the quality of evidence

IndirectnessPopulation, Test and ComparisonOutcomes Similar quality criteria Usual absence of direct evidence about impact

on patient-important outcomes Accuracy studies typically provide low quality

evidence

Factors that decrease the quality of evidenceInconsistency in study results

Similar quality criteria & judgments but: other measures

Imprecise evidenceSimilar quality criteria & judgments, but: Wide confidence intervals for estimates of test accuracy, true and false positives and negatives

Reporting biasSimilar quality criteria & judgments

Balance between presumed patient outcomes, complications and cost: Avoiding the undesirable consequences of more false positives with CT is preferable to avoiding the higher rate of complications (infarction and death) and higher cost with angiography

Example of new test and reference test or strategy

Putative benefit of new test

Diagnostic accuracy Patient Outcomes and expected impact on management for the following test outcomes

Sensitivity Specificity

True positives

False positives

True negatives

False negatives

CT for coronary artery disease compared with coronary angiography

Less invasive testing

equal less Presumed influence on patient important outcomes

Benefit from treatment and fewer complications

Harm from unnecessary treatment

Benefit from reassurance and fewer complications

Detriment from delayed diagnosis or myocardial insult

Directness of the evidence (test results) for patient-important outcomes

No uncertainty

No uncertainty

No uncertainty

Some uncertainty

Evaluating tests – when can

comparative evidence of test

accuracy and other intermediate

outcomes be used as an alternative to randomized trials

(Lord, Irwig, Bossuyt)

Terminology

Directness = PICO Synonyms: Generalizability, Applicability Directness of outcomes: Linkage assumptions

Explicit assumptions about management of TP, TN, FP, FN, complications, indeterminates Treatment effects on:

Mortality, morbidity, HRQL (including psychological well being)

25

Summary of findings

Key findings for diagnostic accuracy studies. Should multislice spiral computed tomography versus conventional coronary angiography be used to diagnose coronary artery disease in a population with a low (20%) pretest probability?

Test findings&

Pooled sensitivity 0.96 (95% CI: 0.94 - 0.98) LR(+) 5.4 (95% CI: 3.4–8.3) Pooled specificity 0.74 (95% CI: 0.065 – 0.84) LR(–) 0.05 (95% CI: 0.03–0.09) Consequences Number per 1000 Importance+ TP 192 8 FP 208 7 TN 592 8 FN 8 9

28

Outcomes

Illustrative Risks(95% CI) Number of

participants (studies)

Quality of the Evidence1 Comments

Assumed outcome with CT – prevalence of 20%

True positives(Patients correctly classified as having coronary artery disease)

192 per 1000 1570(21)

Moderate2

Benefit from treatment and fewer complications.* Some patients will have to undergo angiography.

True negatives (Patients correctly classified as not having coronary artery disease)

592 per 1000 1570(21)

Moderate2

Benefit from reassurance and fewer complications

False positives (Patients incorrectly classified as having coronary artery disease)

208 per 1000 1570(21)

Moderate2

Harm from unnecessary treatment

False negatives (Patients incorrectly classified as not having coronary artery disease)

8 per 1000 1570(21)

Low2, 3

Detriment from delayed diagnosis or myocardial insult

Complications(MI, allergic reactions, renal failure)

99 per 1000 1570(21)

Low2

There is a higher rate of rare complications (infarction and death) and higher cost with angiography – a full profile would be required.

Resource use*(cost of CT and Angiography)

See comment See comment

See comment

Cost are higher for angiography,

1- Quality rated from 1 (very low quality) to 4 (high quality), 2- Cross sectional studies. Indirectness of outcomes in a wide spectrum of patients and indirect comparison of tests, 3– there is greater uncertainty whether these patients will have negative outcomes.*Assumed efficacy of: 1) aspirin daily = 20% RRR; 2) beta-blockage = 18% RRR.

Summary (1) Quality of evidence reflects our confidence

that estimates of benefits and downsides from a diagnostic strategy generated from research are correct.

Summary (2) Consideration of the directness of evidence

is based on how confident we are of the relation between being classified correctly (as a true positive or negative) or incorrectly (as a false positive or negative) and patient-important consequences.

Summary Recommendation depends on the balance

between desirable and undesirable effects of the diagnostic test or strategy in terms of patient-important outcomes.

Eskerrik asko A USTEDES!

Example of new test and reference test or strategy

Putative benefit of new test

Diagnostic accuracy Patient Outcomes and expected impact on management for the following test outcomes

Sensitivity Specificity

True positives

False positives

True negatives

False negatives

Helical CT for renal calculus compared with intravenous pyeolgram

Detection of more (but smaller) calculi

greater equal Presumed influence on patient important outcomes

Certain benefit for larger stones, for smaller stones the benefit is less clear and unnecessary treatment can result

Likely detriment from unnecessary additional invasive tests

Almost certain benefit from avoiding unnecessary tests

Likely detriment for large stones, less certain for small stones More testing

Directness of the evidence (test results) for patient-important outcomes

Some uncertainty

No uncertainty

No uncertainty

Major uncertainty

Balance between presumed patient outcomes, complications and cost: Less complications and downsides compared to IVP would support the new test’s usefulness, but the balance between desirable and undesirable effect not clear in view of the uncertain consequences of identifying smaller stones.