Closed Testing and the Partitioning Principle

Jason C. HsuJason C. Hsu

TheThe Ohio State University Ohio State University

MCP 2002MCP 2002August 2002August 2002

Bethesda, MarylandBethesda, Maryland

Principles of Test-Construction Union-Intersection Testing UITUnion-Intersection Testing UIT

S. N. RoyS. N. Roy Intersection-Union Testing IUTIntersection-Union Testing IUT

Roger Berger (1982) Roger Berger (1982) TechnometricsTechnometrics Closed testingClosed testing

Marcus, Peritz, Gabriel (1976) Marcus, Peritz, Gabriel (1976) BiometrikaBiometrika PartitioningPartitioning

Stefansson, Kim, and Hsu (1984) Stefansson, Kim, and Hsu (1984) Statistical Decision Theory and Statistical Decision Theory and Related TopicsRelated Topics, Berger & Gupta eds., Springer-Verlag., Berger & Gupta eds., Springer-Verlag.

Finner and Strassberger (2002)Finner and Strassberger (2002) Annals of StatisticsAnnals of Statistics Equivariant confidence setEquivariant confidence set

Tukey (1953)Tukey (1953)Scheffe (195?)Scheffe (195?)Dunnett (1955)Dunnett (1955)

Partitioning confidence sets

Multiple Comparison with the BestMultiple Comparison with the BestGunnar Stefansson & HsuGunnar Stefansson & Hsu

1-sided stepdown method (sample-determined 1-sided stepdown method (sample-determined steps) = Naik/Marcus-Peritz-Gabriel closed teststeps) = Naik/Marcus-Peritz-Gabriel closed testHsuHsu

Multiple Comparison with the Sample BestMultiple Comparison with the Sample BestWoochul Kim & Hsu & StefanssonWoochul Kim & Hsu & Stefansson

BioequivalenceBioequivalenceRuberg & HsuRuberg & Hsu & G. Hwang & Liu & Casella & Brown & G. Hwang & Liu & Casella & Brown

1-sided stepdown method (pre-determined steps)1-sided stepdown method (pre-determined steps)Roger Berger & HsuRoger Berger & Hsu

Partitioning1.1. Formulate hypotheses HFormulate hypotheses H

00ii: : ii ** for for ii II

iiII ii ** = entire parameter space = entire parameter space

{{ii**: : iiII } partitions the parameter space } partitions the parameter space

2.2. Test each HTest each H00ii

**: : i i **, , iiII, at , at

3.3. Infer Infer ii if H if H00ii

** is rejected is rejected

4.4. Pivot in each Pivot in each ii a confidence set a confidence set CCii for for 5.5. iiII CCii is a 100(1 is a 100(1)% confidence set for )% confidence set for

Partitioning1.1. Formulate hypotheses HFormulate hypotheses H

0i0i: : ii for i for i I I

iiJJ II = entire parameter space = entire parameter space

2.2. For each J For each J I, let I, letJ J

** = = iiJJ ii (( jjJJ jj))cc

3.3. Test each HTest each H0J0J

**: : J J **, J , J I, at I, at

{{JJ**: J : J I} partitions the parameter space I} partitions the parameter space

4.4. Infer Infer JJ if H if H0J0J

** is rejected is rejected

5.5. Pivot in each Pivot in each JJ a confidence set C a confidence set CJJ for for

6.6. J J J J C CJJ is a 100(1 is a 100(1)% confidence set for )% confidence set for

MCB confidence intervals

ii maxmaxjjii jj

[[((YYii maxmaxjjii Y Yjj WW)), , ((YYii maxmaxjjii Y Yjj + + WW))++], ],

ii = 1, 2, … , = 1, 2, … , kk

Upper bounds imply subset selectionUpper bounds imply subset selection Lower bounds imply indifference zone Lower bounds imply indifference zone

selectionselection

Multiple Comparison with the Best

HH0101: Treatment 1 is the best: Treatment 1 is the best

HH0202: Treatment 2 is the best: Treatment 2 is the best

HH0303: Treatment 3 is the best: Treatment 3 is the best

…… Test each at Test each at using 1-sided Dunnett’s using 1-sided Dunnett’s Collate the resultsCollate the results

Union-Intersection Testing UIT

1.1. Form HForm Haa: : H Haiai (an “or” thing)(an “or” thing)

2.2. Test HTest H00: : H H00ii, the complement of H, the complement of Haa

1.1. If reject, infer at least one HIf reject, infer at least one H00ii false false

2.2. Else, infer nothingElse, infer nothing

Closed Testing

1.1. Formulate hypotheses HFormulate hypotheses H00ii: : ii for for ii II

2.2. For each For each JJ II, let , let JJ = = iiJJ ii

3.3. Form closed family of null hypothesesForm closed family of null hypotheses{H{H0J0J: : JJ: : JJ II}}

4.4. Test each Test each HH00JJ at at

5.5. Infer Infer iiJJ ii if all H if all H00J’J’ with with JJ J’J’ rejected rejected

6.6. Infer Infer ii if all H if all H00JJ’’ with with ii J’J’ rejected rejected

Oneway model

YYirir = = ii + + irir, , ii = 0, 1, 2, … , = 0, 1, 2, … , kk, , rr = 1, … , = 1, … , nnii

irir are i.i.d. Normal(0, are i.i.d. Normal(0, 22))

Dose Dose ii “efficacious” if “efficacious” if ii > > 11 + +

ICH E10 (2000)ICH E10 (2000) Superiority if Superiority if 0 0 Non-inferiority if Non-inferiority if < 0 < 0 Equivalence is 2-sidedEquivalence is 2-sided Non-inferiority is 1-sidedNon-inferiority is 1-sided

Closed testing null hypotheses(sample-determined steps)

HH0202: Dose 2 not efficacious: Dose 2 not efficacious

HH0303: Dose 3 not efficacious: Dose 3 not efficacious

HH0101: Doses 2 and 3 not efficacious: Doses 2 and 3 not efficacious

Test each at Test each at Collate the resultsCollate the results

Partitioning null hypotheses(sample-determined steps)

HH0101: Doses 2 and 3 not efficacious: Doses 2 and 3 not efficacious

HH0202: Dose 2 not efficacious: Dose 2 not efficacious

but dose 3 isbut dose 3 is HH0303: Dose 3 not efficacious: Dose 3 not efficacious

but dose 2 isbut dose 2 is Test each at Test each at Collate the resultsCollate the results

Partitioning implies closed testing

PartitioningPartitioning implies closed testing because implies closed testing because A size A size test for H test for H00ii is a size is a size test for test for HH00ii

Reject Reject HH0101: Doses 2 and 3 not efficacious : Doses 2 and 3 not efficacious

implies either dose 2 or dose 3 efficaciousimplies either dose 2 or dose 3 efficacious Reject Reject HH0202: Dose 2 not efficacious: Dose 2 not efficacious

but dose 3 efficaciousbut dose 3 efficacious implies it is not the implies it is not the case case dose 3 is efficacious dose 3 is efficacious but notbut not dose 2dose 2

Reject HReject H0101 and H and H0202 thus implies dose 2 thus implies dose 2

efficaciousefficacious

Intersection-Union Testing IUT

1.1. Form HForm Haa: : H Haiai (an “and” thing) (an “and” thing)

2.2. Test HTest H00: : H H00ii, the complement of H, the complement of Haa

1.1. If reject, infer all HIf reject, infer all H00ii false false

2.2. Else, infer nothingElse, infer nothing

PK concentration in blood plasma curve

Bioequivalence defined

Bioequivalence: clinical equivalence betweenBioequivalence: clinical equivalence between

1.1. Brand name drugBrand name drug

2.2. Generic drugGeneric drug

Bioequivalence parametersBioequivalence parameters AUCAUC = Area Under the Curve = Area Under the Curve CCmaxmax = maximum Concentration = maximum Concentration

TTmaxmax = Time to maximum concentratin = Time to maximum concentratin

Average bioequivalence

NotationNotation

= expected value of brand name drug= expected value of brand name drug

22 = expected value of generic drug = expected value of generic drug

Average bioequivalence meansAverage bioequivalence means

.8 < .8 < //22 < 1.25 for < 1.25 for AUCAUC

andand

.8 < .8 < //22 < 1.25 for < 1.25 for CCmaxmax

Bioequivalence in practice

If If loglog of observations are normal with of observations are normal with means means and and22 and equal variances, and equal variances,

then average bioequivalence becomes then average bioequivalence becomes

loglog(.8) < (.8) < 22 < < loglog(1.25) for (1.25) for AUCAUC

andand

loglog(.8) < (.8) < 22 < < loglog(1.25) for (1.25) for CCmaxmax

Partitioning

Partition the parameter space asPartition the parameter space as

1.1. HH0<0<: : 22 < < loglog(0.8)(0.8)

2.2. HH0>0>: : 22 > > loglog(1.25) (1.25)

3.3. HHaa: : loglog(.8) < (.8) < 22 < < loglog(1.25) (1.25)

Test HTest H0<0< and H and H0>0> each at each at ..

Infer Infer loglog(.8) < (.8) < 22 < < loglog(1.25) if both H(1.25) if both H0<0< and H and H0>0>

rejected.rejected.

Controls Controls PP{incorrect decision} at {incorrect decision} at ..

Dose-Response (Phase II)

Anti-psychotic drug efficacy trial

Dose of Seroquel (mg)Dose of Seroquel (mg)

00 7575 150150 300300 600600 750750

nn 5151 5252 4848 5151 5151 5353

ii 4.784.78 4.224.22 3.743.74 3.563.56 3.583.58 3.933.93

SESE 0.230.23 0.220.22 0.230.23 0.230.23 0.230.23 0.220.22

Arvanitis et al. (1997 Biological Psychiatry)

CGI = Clinical Global Impression

Minimum Effective Dose (MED)Minimu Effective Dose Minimu Effective Dose

== MEDMED

== smallest smallest ii so that so that ii > > 11 + + for all for all jj, , ii jj kk

Want an upper confidence bound MEDWant an upper confidence bound MED++ so that so that

PP{{MEDMED < MED < MED++} } 100(1 100(1)%)%

Closed testing inference

Infer nothing if HInfer nothing if H0101 is accepted is accepted

Infer at least one of doses 2 and 3 effective Infer at least one of doses 2 and 3 effective if Hif H0101 is rejected is rejected

Infer dose 2 effective if, in addition to HInfer dose 2 effective if, in addition to H0101,,

HH0202 is rejected is rejected

Infer dose 3 effective if, in addition to HInfer dose 3 effective if, in addition to H0101,,

HH0303 is rejected is rejected

Closed testing method(sample-determined steps)

Start from HStart from H0101 to H to H0202 and H and H0303

Stepdown from smallest Stepdown from smallest pp-value to largest -value to largest pp-value-value

Stop as soon as one fails to rejectStop as soon as one fails to reject Multiplicity adjustment decreases from Multiplicity adjustment decreases from

kk to to k k 1 to 1 to k k 2 2 to 2 from to 2 from step 1 to 2 to 3 … to step step 1 to 2 to 3 … to step k k 1 1

Tests of equalities(pre-determined steps)

1.1. HH00kk:: 11 = = 22 = = = = kk

HHakak:: 11 = = 22 = = < < kk

2.2. HH0(k0(k1)1):: 11 = = 22 = = = = kk1 1

HHaa((kk1)1):: 11 = = 22 = = < < kk1 1

3.3. HH0(0(kk2)2):: 11 = = 22 = = = = kk2 2

HHaa((kk2)2):: 11 = = 22 = = < < kk22

4.4.

5.5. HH0202:: 11 = = 22 HHaa22:: 11 < < 22

Closed testing of equalities

Null hypotheses are Null hypotheses are nestednested

1.1. Closure of family remains HClosure of family remains H00kk H H0202

2.2. Test each HTest each H00ii at at 3.3. Stepdown from dose Stepdown from dose kk to dose to dose kk1 to 1 to

to dose 2to dose 2

4.4. Stop as soon as one fails to rejectStop as soon as one fails to reject

5.5. Multiplicity adjustment not neededMultiplicity adjustment not needed

Testing equalities is easy

HH00kk: : 11 = = = = kk

HH0202: : 11 = = 22

HH00ii HH00ii

HH00kk: : 11 kk

HH0202:: 11 22

HH00ii HH00ii

Partitioning null hypotheses (for pre-determined steps)

HH0k0k:: Dose Dose kk not efficacious not efficacious

HH0(k-1)0(k-1):: Dose Dose kk efficacious efficacious

butbut dose dose kk1 not efficacious1 not efficacious HH0(k-1)0(k-1):: Doses Doses kk and and kk1 efficacious1 efficacious

butbut dose dose kk2 not efficacious2 not efficacious HH0202:: Doses Doses kk 3 efficacious 3 efficacious

butbut dose 2 not efficacious dose 2 not efficacious Test each at Test each at Collate the resultsCollate the results

Partitioning inference

1.1. Infer nothing if Infer nothing if HH00kk is accepted is accepted

2.2. Infer dose Infer dose kk effective if effective if HH00kk is rejected is rejected

3.3. Infer dose Infer dose kk1 effective if, in addition to 1 effective if, in addition to HH00kk, , HH0(0(kk-1)-1) is rejected is rejected

4.4. Infer dose Infer dose kk2 effective if, in addition to 2 effective if, in addition to HH00kk and and HH0(0(kk-1)-1) , H , H0303 is rejected is rejected

5.5.

Partitioning method (for pre-determined steps) Stepdown from dose Stepdown from dose kk to dose to dose kk1 to 1 to to to

dose 2dose 2 Stop as soon as one fails to rejectStop as soon as one fails to reject Multiplicity adjustment not neededMultiplicity adjustment not needed Any pre-determined sequence of doses can Any pre-determined sequence of doses can

be usedbe used Confidence set given in Hsu and Berger Confidence set given in Hsu and Berger

(1999 (1999 JASAJASA))

Pairwise t tests for partitioning

Size Size tests for tests for HH00kk H H0202 are also size are also size test test for for HH00kk H H0202

HH00kk:: Dose Dose kk not efficacious not efficacious HH0(0(kk-1)-1):: Dose Dose kk1 not efficacious1 not efficacious HH0(0(kk-2)-2):: Dose Dose kk2 not efficacious2 not efficacious HH0202:: Dose 2 not efficaciousDose 2 not efficacious Test each with a size-Test each with a size- 2-sample 1-sided 2-sample 1-sided

tt-test-test

Testing equalities is easy

HH00kk: : 11 = = = = kk

HH0202: : 11 = = 22

HH00ii HH00ii

HH00kk: : 11 kk

HH0202:: 11 22

HH00ii HH00ii

Could reject for the wrong reason

HH00

HHaa

neitherneither