causal inference or truth in the universe importance of clinical trials importance of clinical...

32
Causal Inference Causal Inference or or Truth in the Universe Truth in the Universe Importance of clinical Importance of clinical trials trials Major pitfalls in Major pitfalls in clinical trials clinical trials Low power Low power Not randomized Not randomized Unblinded Unblinded Incomplete follow-up Incomplete follow-up

Upload: gervase-walsh

Post on 28-Dec-2015

215 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Causal InferenceCausal Inferenceoror

Truth in the UniverseTruth in the Universe

Causal InferenceCausal Inferenceoror

Truth in the UniverseTruth in the Universe Importance of clinical trialsImportance of clinical trials Major pitfalls in clinical trialsMajor pitfalls in clinical trials

• Low powerLow power• Not randomizedNot randomized• UnblindedUnblinded• Incomplete follow-upIncomplete follow-up

Importance of clinical trialsImportance of clinical trials Major pitfalls in clinical trialsMajor pitfalls in clinical trials

• Low powerLow power• Not randomizedNot randomized• UnblindedUnblinded• Incomplete follow-upIncomplete follow-up

Page 2: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

FrameworkFrameworkFrameworkFramework

Untruth - spurious associationsUntruth - spurious associations• chance (small sample size)chance (small sample size)• bias (selection bias and other biases)bias (selection bias and other biases)

Truth - real associations, not always causalTruth - real associations, not always causal• effect - causeeffect - cause• effect - effect (confounding)effect - effect (confounding)• cause - effect (truth in the universe)cause - effect (truth in the universe)

Untruth - spurious associationsUntruth - spurious associations• chance (small sample size)chance (small sample size)• bias (selection bias and other biases)bias (selection bias and other biases)

Truth - real associations, not always causalTruth - real associations, not always causal• effect - causeeffect - cause• effect - effect (confounding)effect - effect (confounding)• cause - effect (truth in the universe)cause - effect (truth in the universe)

Page 3: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Estrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in Women

RQRQ: Does postmenopausal estrogen therapy : Does postmenopausal estrogen therapy reduce CHD risk in women?reduce CHD risk in women?

DesignDesign: Cross-sectional: Cross-sectional

SubjectsSubjects: 20 postmenopausal women - entire : 20 postmenopausal women - entire population of my Tuesday clinicpopulation of my Tuesday clinic

MeasurementsMeasurements: estrogen therapy (ever/never) : estrogen therapy (ever/never) self-report; CHD (yes/no) chart reviewself-report; CHD (yes/no) chart review

RQRQ: Does postmenopausal estrogen therapy : Does postmenopausal estrogen therapy reduce CHD risk in women?reduce CHD risk in women?

DesignDesign: Cross-sectional: Cross-sectional

SubjectsSubjects: 20 postmenopausal women - entire : 20 postmenopausal women - entire population of my Tuesday clinicpopulation of my Tuesday clinic

MeasurementsMeasurements: estrogen therapy (ever/never) : estrogen therapy (ever/never) self-report; CHD (yes/no) chart reviewself-report; CHD (yes/no) chart review

Page 4: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Estrogen and CHD in WomenEstrogen and CHD in WomenCross-Sectional StudyCross-Sectional Study

Estrogen and CHD in WomenEstrogen and CHD in WomenCross-Sectional StudyCross-Sectional Study

CHDCHD No CHD No CHD

EE

No ENo E

CHDCHD No CHD No CHD

EE

No ENo E

11 44

66 99

55

1515

131377 2020

RR = 0.5RR = 0.5

Page 5: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Estrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in Women

RQRQ: Does estrogen therapy reduce CHD risk?: Does estrogen therapy reduce CHD risk?

DesignDesign: Case-control: Case-controlCases:Cases: 1000 women admitted to SFGH over a 5-year 1000 women admitted to SFGH over a 5-year

period with discharge diagnosis of CHD (ICD-9 codes)period with discharge diagnosis of CHD (ICD-9 codes)

Controls:Controls: 1000 women identified by random digit dialing 1000 women identified by random digit dialing in SF who report no CHDin SF who report no CHD

MeasurementsMeasurements: CHD based on discharge : CHD based on discharge diagnosis; estrogen therapy based on self-reportdiagnosis; estrogen therapy based on self-report

RQRQ: Does estrogen therapy reduce CHD risk?: Does estrogen therapy reduce CHD risk?

DesignDesign: Case-control: Case-controlCases:Cases: 1000 women admitted to SFGH over a 5-year 1000 women admitted to SFGH over a 5-year

period with discharge diagnosis of CHD (ICD-9 codes)period with discharge diagnosis of CHD (ICD-9 codes)

Controls:Controls: 1000 women identified by random digit dialing 1000 women identified by random digit dialing in SF who report no CHDin SF who report no CHD

MeasurementsMeasurements: CHD based on discharge : CHD based on discharge diagnosis; estrogen therapy based on self-reportdiagnosis; estrogen therapy based on self-report

Page 6: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Estrogen and CHD in WomenEstrogen and CHD in WomenCase-Control StudyCase-Control Study

Estrogen and CHD in WomenEstrogen and CHD in WomenCase-Control StudyCase-Control Study

CHDCHD No CHD No CHD

EE

No ENo E

CHDCHD No CHD No CHD

EE

No ENo E

200200 300300

800800 700700

500500

15001500

1000100010001000 20002000

OR = .6; p = .01OR = .6; p = .01

Page 7: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Estrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in Women

RQRQ: Does estrogen therapy reduce CHD risk?: Does estrogen therapy reduce CHD risk?

DesignDesign: Case-control: Case-controlCases:Cases: 1000 women admitted to Kaiser over a 5-year 1000 women admitted to Kaiser over a 5-year

period with discharge diagnosis of CHDperiod with discharge diagnosis of CHD

Controls:Controls: 1000 women admitted to Kaiser over the same 1000 women admitted to Kaiser over the same 5-year period with no discharge diagnosis of CHD5-year period with no discharge diagnosis of CHD

MeasurementsMeasurements: CHD based on discharge : CHD based on discharge diagnosis; estrogen therapy based on diagnosis; estrogen therapy based on computerized pharmacy recordscomputerized pharmacy records

RQRQ: Does estrogen therapy reduce CHD risk?: Does estrogen therapy reduce CHD risk?

DesignDesign: Case-control: Case-controlCases:Cases: 1000 women admitted to Kaiser over a 5-year 1000 women admitted to Kaiser over a 5-year

period with discharge diagnosis of CHDperiod with discharge diagnosis of CHD

Controls:Controls: 1000 women admitted to Kaiser over the same 1000 women admitted to Kaiser over the same 5-year period with no discharge diagnosis of CHD5-year period with no discharge diagnosis of CHD

MeasurementsMeasurements: CHD based on discharge : CHD based on discharge diagnosis; estrogen therapy based on diagnosis; estrogen therapy based on computerized pharmacy recordscomputerized pharmacy records

Page 8: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Estrogen and CHD in WomenEstrogen and CHD in WomenCase-Control StudyCase-Control Study

Estrogen and CHD in WomenEstrogen and CHD in WomenCase-Control StudyCase-Control Study

CHDCHD No CHD No CHD

EE

No ENo E

CHDCHD No CHD No CHD

EE

No ENo E

130130 370370

870870 630630

500500

15001500

1000100010001000 20002000

OR = .25; p = .001OR = .25; p = .001

Page 9: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

130130 370370

870870 630630

500500

15001500

1000100010001000 20002000

OR = .25; p = .001OR = .25; p = .001

CHDCHD No CHDNo CHD

EE

No ENo E

4040 360360

6060 540540

400400

600600

900900100100 10001000

OR = 1.0; p = .9OR = 1.0; p = .9

CHDCHD No CHDNo CHD

EE

No ENo E

9090 1010

810810 9090

100100

900900

100100900900 10001000

OR = 1.0; p = .9OR = 1.0; p = .9

CHDCHD No CHDNo CHD

EE

No ENo E

Age 50-64Age 50-64 Age 65-79Age 65-79

ConfoundingConfounding

AllAll

Page 10: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Controlling ConfoundingControlling ConfoundingControlling ConfoundingControlling Confounding

Design stageDesign stage• MatchingMatching• SpecificationSpecification• RandomizationRandomization

Analysis stageAnalysis stage• StratificationStratification• Multivariate modelingMultivariate modeling

Design stageDesign stage• MatchingMatching• SpecificationSpecification• RandomizationRandomization

Analysis stageAnalysis stage• StratificationStratification• Multivariate modelingMultivariate modeling

Page 11: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Estrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in Women

RQRQ: Does estrogen therapy reduce CHD risk?: Does estrogen therapy reduce CHD risk?

DesignDesign: Prospective cohort: Prospective cohort

Subjects:Subjects: 59,337 PM nurses followed for 16 years 59,337 PM nurses followed for 16 years

MeasurementsMeasurements: Self-reported estrogen use; self-: Self-reported estrogen use; self-reported CHD events validated by chart reviewreported CHD events validated by chart review

Analysis:Analysis: Multivariate logistic regression - age, Multivariate logistic regression - age, ethnicity, education, blood pressure, diabetes, ethnicity, education, blood pressure, diabetes, smoking, alcohol, family history of CHD and smoking, alcohol, family history of CHD and hypercholesterolemiahypercholesterolemia

RQRQ: Does estrogen therapy reduce CHD risk?: Does estrogen therapy reduce CHD risk?

DesignDesign: Prospective cohort: Prospective cohort

Subjects:Subjects: 59,337 PM nurses followed for 16 years 59,337 PM nurses followed for 16 years

MeasurementsMeasurements: Self-reported estrogen use; self-: Self-reported estrogen use; self-reported CHD events validated by chart reviewreported CHD events validated by chart review

Analysis:Analysis: Multivariate logistic regression - age, Multivariate logistic regression - age, ethnicity, education, blood pressure, diabetes, ethnicity, education, blood pressure, diabetes, smoking, alcohol, family history of CHD and smoking, alcohol, family history of CHD and hypercholesterolemiahypercholesterolemia

Page 12: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Nurses’ Health StudyNurses’ Health StudyNurses’ Health StudyNurses’ Health Study

HormonesHormones NN PYARPYAR CHDCHD RRRR P-valueP-value

NeverNever 20,03420,034 324,748324,748 452452 1.01.0 referentreferent

PastPast 12,50312,503 150,238150,238 195195 0.80.8 0.06 0.06

CurrentCurrent 14,00014,000 166,371166,371 98 98 0.60.6 0.01 0.01

HormonesHormones NN PYARPYAR CHDCHD RRRR P-valueP-value

NeverNever 20,03420,034 324,748324,748 452452 1.01.0 referentreferent

PastPast 12,50312,503 150,238150,238 195195 0.80.8 0.06 0.06

CurrentCurrent 14,00014,000 166,371166,371 98 98 0.60.6 0.01 0.01

Grodstein, NEJM, 1996Grodstein, NEJM, 1996

Page 13: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

••••

••••••

••••

••••

••••

••••

••••

••

••••

••••

••••

••••

••••

Summary Relative RiskSummary Relative RiskRosenberg, 1976Rosenberg, 1976Talbott, 1977Talbott, 1977Pfeffer, 1978Pfeffer, 1978Rosenberg, 1980Rosenberg, 1980Adam, 1981Adam, 1981Bain, 1981Bain, 1981Ross, 1981Ross, 1981Szklo, 1984Szklo, 1984Beard, 1989Beard, 1989Croft, 1989Croft, 1989Rosenberg, 1993Rosenberg, 1993Mann, 1994Mann, 1994 Gruchow, 1988Gruchow, 1988Sullivan, 1988Sullivan, 1988McFarland, 1989McFarland, 1989 Wilson, 1985Wilson, 1985Bush, 1987Bush, 1987Petitti, 1987Petitti, 1987Criqui, 1988Criqui, 1988Avila, 1990Avila, 1990Sullivan, 1990Sullivan, 1990Henderson, 1991Henderson, 1991Wolf, 1991Wolf, 1991Falkeborn, 1992Falkeborn, 1992Grodstein, 2000Grodstein, 2000

0.010.01 0.10.1 11 1010

Relative RiskRelative Risk

RISK FOR CORONARY HEART DISEASE INRISK FOR CORONARY HEART DISEASE INESTROGEN USERS VS. NONUSERSESTROGEN USERS VS. NONUSERS

CohortCohort StudiesStudies

Angiographic StudiesAngiographic Studies

Case-Control StudiesCase-Control Studies

RR = 0.65

Page 14: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Potential MechanismsPotential MechanismsPotential MechanismsPotential Mechanisms

ESTROGENESTROGEN Improves lipoproteinsImproves lipoproteins

• Reduces LDL 10-15%Reduces LDL 10-15%

• Increases HDL 10-15%Increases HDL 10-15%

Retards atherosclerosisRetards atherosclerosis Prevents coronary vasoconstrictionPrevents coronary vasoconstriction

ESTROGENESTROGEN Improves lipoproteinsImproves lipoproteins

• Reduces LDL 10-15%Reduces LDL 10-15%

• Increases HDL 10-15%Increases HDL 10-15%

Retards atherosclerosisRetards atherosclerosis Prevents coronary vasoconstrictionPrevents coronary vasoconstriction

Page 15: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Estrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in Women

Observational findingsObservational findings Strong associationStrong association Consistent associationConsistent association Plausible biologic mechanismPlausible biologic mechanism

Observational findingsObservational findings Strong associationStrong association Consistent associationConsistent association Plausible biologic mechanismPlausible biologic mechanism

CausalityCausality

Page 16: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Reasons to be CautiousReasons to be CautiousReasons to be CautiousReasons to be Cautious

Observational findings susceptible Observational findings susceptible to bias and confoundingto bias and confounding

Estrogen has known risksEstrogen has known risks (Was a) preventive therapy widely (Was a) preventive therapy widely

used among healthy womenused among healthy women

Observational findings susceptible Observational findings susceptible to bias and confoundingto bias and confounding

Estrogen has known risksEstrogen has known risks (Was a) preventive therapy widely (Was a) preventive therapy widely

used among healthy womenused among healthy women

Page 17: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Estrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in WomenEstrogen and CHD in Women

RQRQ: Does estrogen therapy reduce CHD risk?: Does estrogen therapy reduce CHD risk?

DesignDesign: Randomized trial: Randomized trial

Subjects:Subjects: 2500 PM women with CHD 2500 PM women with CHD

Intervention:Intervention: Estrogen + progestin vs. placebo Estrogen + progestin vs. placebo

MeasurementsMeasurements: Predictor = treatment: Predictor = treatment

outcome = CHD death or nonfatal MIoutcome = CHD death or nonfatal MI

RQRQ: Does estrogen therapy reduce CHD risk?: Does estrogen therapy reduce CHD risk?

DesignDesign: Randomized trial: Randomized trial

Subjects:Subjects: 2500 PM women with CHD 2500 PM women with CHD

Intervention:Intervention: Estrogen + progestin vs. placebo Estrogen + progestin vs. placebo

MeasurementsMeasurements: Predictor = treatment: Predictor = treatment

outcome = CHD death or nonfatal MIoutcome = CHD death or nonfatal MI

Page 18: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Estrogen and CHD in WomenEstrogen and CHD in WomenRandomized TrialRandomized Trial

Estrogen and CHD in WomenEstrogen and CHD in WomenRandomized TrialRandomized Trial

CHDCHD No CHD No CHD

HTHT

No HTNo HT

CHDCHD No CHD No CHD

HTHT

No HTNo HT

5050 450450

250250 10001000

500500

12501250

14501450300300 17501750

RR = .5; p = .001RR = .5; p = .001

Page 19: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Important Features of RCTsImportant Features of RCTsImportant Features of RCTsImportant Features of RCTs

Adequate PowerAdequate Power Rule out chance associationsRule out chance associationsFind clinically significant associationsFind clinically significant associations

RandomizationRandomization Comparability at Comparability at baselinebaseline- - BiasBias

- Confounding- Confounding

BlindingBlinding Comparability Comparability during follow-upduring follow-up- - Placebo effectPlacebo effect

- Differential outcome ascertainment- Differential outcome ascertainment- Co-intervention- Co-intervention

Complete Complete Follow-upFollow-up Comparability Comparability at the end of the trialat the end of the trial

Adequate PowerAdequate Power Rule out chance associationsRule out chance associationsFind clinically significant associationsFind clinically significant associations

RandomizationRandomization Comparability at Comparability at baselinebaseline- - BiasBias

- Confounding- Confounding

BlindingBlinding Comparability Comparability during follow-upduring follow-up- - Placebo effectPlacebo effect

- Differential outcome ascertainment- Differential outcome ascertainment- Co-intervention- Co-intervention

Complete Complete Follow-upFollow-up Comparability Comparability at the end of the trialat the end of the trial

Page 20: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Power of the PlaceboPower of the PlaceboPower of the PlaceboPower of the Placebo

Internal Mammary Artery Ligation for Angina Internal Mammary Artery Ligation for Angina In unblinded trials, IM ligationIn unblinded trials, IM ligation

• reduced angina 60%reduced angina 60% In blinded trials, reduced angina 65% inIn blinded trials, reduced angina 65% in

Internal Mammary Artery Ligation for Angina Internal Mammary Artery Ligation for Angina In unblinded trials, IM ligationIn unblinded trials, IM ligation

• reduced angina 60%reduced angina 60% In blinded trials, reduced angina 65% inIn blinded trials, reduced angina 65% in

• subjects who underwent IM ligation subjects who underwent IM ligation • subjects who underwent IM ligation subjects who underwent IM ligation

• subjects who underwent sham IM ligation subjects who underwent sham IM ligation • subjects who underwent sham IM ligation subjects who underwent sham IM ligation

Page 21: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Differential Outcome AdjudicationDifferential Outcome AdjudicationDifferential Outcome AdjudicationDifferential Outcome Adjudication

Canadian Cooperative MS Trial Canadian Cooperative MS Trial 165 patients with multiple sclerosis165 patients with multiple sclerosis

• plasma exchange + cyclo + predplasma exchange + cyclo + pred• sham plasma exchange + placebo medssham plasma exchange + placebo meds

Outcome = structured neurologic exam by Outcome = structured neurologic exam by blinded and unblinded neurologistsblinded and unblinded neurologists

More improvement with plasma exchange by More improvement with plasma exchange by unblinded, but not blinded assessmentunblinded, but not blinded assessment

Canadian Cooperative MS Trial Canadian Cooperative MS Trial 165 patients with multiple sclerosis165 patients with multiple sclerosis

• plasma exchange + cyclo + predplasma exchange + cyclo + pred• sham plasma exchange + placebo medssham plasma exchange + placebo meds

Outcome = structured neurologic exam by Outcome = structured neurologic exam by blinded and unblinded neurologistsblinded and unblinded neurologists

More improvement with plasma exchange by More improvement with plasma exchange by unblinded, but not blinded assessmentunblinded, but not blinded assessment

Noseworthy, Neurology, 1994Noseworthy, Neurology, 1994Noseworthy, Neurology, 1994Noseworthy, Neurology, 1994

Page 22: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Co-InterventionCo-InterventionCo-InterventionCo-Intervention

Unintended effective interventionsUnintended effective interventions• participantsparticipants use other therapy or change behavior use other therapy or change behavior• study staff, medical providers, family or friendsstudy staff, medical providers, family or friends

treat participants differentlytreat participants differently Nondifferential decreases powerNondifferential decreases power Differential causes biasDifferential causes bias

Unintended effective interventionsUnintended effective interventions• participantsparticipants use other therapy or change behavior use other therapy or change behavior• study staff, medical providers, family or friendsstudy staff, medical providers, family or friends

treat participants differentlytreat participants differently Nondifferential decreases powerNondifferential decreases power Differential causes biasDifferential causes bias

Page 23: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

HHeart and eart and EEstrogen-progestin strogen-progestin RReplacement eplacement SStudy tudy (HERS)(HERS)

HHeart and eart and EEstrogen-progestin strogen-progestin RReplacement eplacement SStudy tudy (HERS)(HERS)

2763 postmenopausal women < 80 years old 2763 postmenopausal women < 80 years old with documented CHD and a uteruswith documented CHD and a uterus

Randomized to CEE 0.625 mg plus MPA 2.5 Randomized to CEE 0.625 mg plus MPA 2.5 mg or identical placebomg or identical placebo

Followed every 4 months for 4.2 yearsFollowed every 4 months for 4.2 years Separate gynecology group managed bleedingSeparate gynecology group managed bleeding Outcome = nonfatal MI and CHD deathOutcome = nonfatal MI and CHD death

2763 postmenopausal women < 80 years old 2763 postmenopausal women < 80 years old with documented CHD and a uteruswith documented CHD and a uterus

Randomized to CEE 0.625 mg plus MPA 2.5 Randomized to CEE 0.625 mg plus MPA 2.5 mg or identical placebomg or identical placebo

Followed every 4 months for 4.2 yearsFollowed every 4 months for 4.2 years Separate gynecology group managed bleedingSeparate gynecology group managed bleeding Outcome = nonfatal MI and CHD deathOutcome = nonfatal MI and CHD death

Page 24: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

HERS HERS Trial ProfileTrial Profile

RandomizedRandomized2,7632,763

PlaceboPlacebo1,3831,383

Estrogen + ProgestinEstrogen + Progestin1,3801,380

Died - 123Died - 123

Dead or completed follow-up - 91%Dead or completed follow-up - 91%

Vital Status Known - 100%Vital Status Known - 100%

Died - 123Died - 123

Dead or completed follow-up - 91%Dead or completed follow-up - 91%

Vital Status Known - 100%Vital Status Known - 100%

Died - 131Died - 131

Dead or completed follow-up - 91%Dead or completed follow-up - 91%

Vital Status Known - 100%Vital Status Known - 100%

Died - 131Died - 131

Dead or completed follow-up - 91%Dead or completed follow-up - 91%

Vital Status Known - 100%Vital Status Known - 100%

Page 25: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

HERS: Baseline CharacteristicsHERS: Baseline CharacteristicsHERS: Baseline CharacteristicsHERS: Baseline Characteristics

HRTHRT PlaceboPlaceboAge (years)Age (years) 6767 6767White (%)White (%) 8888 9090Current Smoker (%)Current Smoker (%) 1313 1313Diabetes (%)Diabetes (%) 1919 1818Blood pressure Blood pressure (mmHg)(mmHg) 135135 135135LDL-C LDL-C (mg/dL)(mg/dL) 145145 145145BMI > 27 (kg/m2)BMI > 27 (kg/m2) 5757 5555Prior estrogen use (%)Prior estrogen use (%) 2424 2323

HRTHRT PlaceboPlaceboAge (years)Age (years) 6767 6767White (%)White (%) 8888 9090Current Smoker (%)Current Smoker (%) 1313 1313Diabetes (%)Diabetes (%) 1919 1818Blood pressure Blood pressure (mmHg)(mmHg) 135135 135135LDL-C LDL-C (mg/dL)(mg/dL) 145145 145145BMI > 27 (kg/m2)BMI > 27 (kg/m2) 5757 5555Prior estrogen use (%)Prior estrogen use (%) 2424 2323

Page 26: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

CHD Events in HERSCHD Events in HERSCHD Events in HERSCHD Events in HERS

0

3

6

9

12

15

0 1 2 3 4 5

HRTPBO

0

3

6

9

12

15

0 1 2 3 4 5

HRTPBO

YearsYears

Cum

ulat

ive

%

Cum

ulat

ive

%

Hulley, JAMA 1998Hulley, JAMA 1998

R.H. = 1.0 (95% CI 0.8 to 1.2)R.H. = 1.0 (95% CI 0.8 to 1.2)

Page 27: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

HERS: Primary OutcomesHERS: Primary Outcomes

E+PE+P PboPbo RRRR p-valuep-value

Total CHD eventsTotal CHD events 172172 176176 1.01.0 0.90.9

CHD deathCHD death 7171 5858 1.21.2 0.20.2

Non-fatal MINon-fatal MI 116116 129129 0.90.9 0.50.5

Page 28: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

HERS: Cardiovascular OutcomesHERS: Cardiovascular OutcomesHERS: Cardiovascular OutcomesHERS: Cardiovascular Outcomes

HRTHRT PlaceboPlacebo RHRH p-valuep-value (N=1380)(N=1380) (N=1383) (N=1383)

CABGCABG 88 88 101101 0.90.9 .4.4PTCAPTCA 164164 175175 0.90.9 .6.6Unstable anginaUnstable angina 103103 117117 0.90.9 .4.4CHFCHF 128128 112112 1.01.0 .6.6PVDPVD 94 94 108108 0.90.9 .3.3Stroke/TIAStroke/TIA 108108 96 96 1.11.1 .4.4

HRTHRT PlaceboPlacebo RHRH p-valuep-value (N=1380)(N=1380) (N=1383) (N=1383)

CABGCABG 88 88 101101 0.90.9 .4.4PTCAPTCA 164164 175175 0.90.9 .6.6Unstable anginaUnstable angina 103103 117117 0.90.9 .4.4CHFCHF 128128 112112 1.01.0 .6.6PVDPVD 94 94 108108 0.90.9 .3.3Stroke/TIAStroke/TIA 108108 96 96 1.11.1 .4.4

Page 29: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

HERS vs. Observational StudiesHERS vs. Observational StudiesHERS vs. Observational StudiesHERS vs. Observational Studies

Why did the findings of HERS differ?Why did the findings of HERS differ?• HERS design differentHERS design different

- adverse effect of added progestinadverse effect of added progestin

- no benefit in women with CHDno benefit in women with CHD

• Observational findings wrongObservational findings wrong- selection bias - comparison groups differselection bias - comparison groups differ

- adherence biasadherence bias

Why did the findings of HERS differ?Why did the findings of HERS differ?• HERS design differentHERS design different

- adverse effect of added progestinadverse effect of added progestin

- no benefit in women with CHDno benefit in women with CHD

• Observational findings wrongObservational findings wrong- selection bias - comparison groups differselection bias - comparison groups differ

- adherence biasadherence bias

Page 30: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Benefit of Adherence Benefit of Adherence with Medicationwith Medication

Benefit of Adherence Benefit of Adherence with Medicationwith Medication

5 Year Mortality (%)5 Year Mortality (%)

AdherenceAdherence ClofibrateClofibrate PlaceboPlacebo

AllAll 2020 2121

5 Year Mortality (%)5 Year Mortality (%)

AdherenceAdherence ClofibrateClofibrate PlaceboPlacebo

AllAll 2020 2121

2626

1616

2626

1616

<80% pills<80% pills 2222

>>80% pills80% pills 1616

Coronary Drug Project, NEJM, 1980Coronary Drug Project, NEJM, 1980

Page 31: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

Are Observational Studies Useless?Are Observational Studies Useless?Are Observational Studies Useless?Are Observational Studies Useless?

NONO• generate important hypothesesgenerate important hypotheses• provide only answer if trial not feasibleprovide only answer if trial not feasible• generally produce correct answergenerally produce correct answer

But bias and confounding always worrisomeBut bias and confounding always worrisome Particularly problematic for interventions Particularly problematic for interventions

that require selection and adherencethat require selection and adherence

NONO• generate important hypothesesgenerate important hypotheses• provide only answer if trial not feasibleprovide only answer if trial not feasible• generally produce correct answergenerally produce correct answer

But bias and confounding always worrisomeBut bias and confounding always worrisome Particularly problematic for interventions Particularly problematic for interventions

that require selection and adherencethat require selection and adherence

Page 32: Causal Inference or Truth in the Universe Importance of clinical trials Importance of clinical trials Major pitfalls in clinical trials Major pitfalls

QQUUEESSTTIIOONNSS??????