Statistical Statistical considerations for considerations for

grantsgrantsBrian HealyBrian Healy

Comments from previous Comments from previous classclass

Change time of courseChange time of course Available on-line power calculatorsAvailable on-line power calculators

– http://http://www.cs.uiowa.edu/~rlenth/Powerwww.cs.uiowa.edu/~rlenth/Power// Two-sided vs. one-sidedTwo-sided vs. one-sided Comparison of statistical packagesComparison of statistical packages

ReviewReview Type I errorType I error Type II errorType II error Ways to increase powerWays to increase power Power/sample size calculation with Power/sample size calculation with

continuous outcomecontinuous outcome

Type I errorType I error We could plot the distribution of the We could plot the distribution of the

sample means under the null before sample means under the null before collecting datacollecting data

Type I error is the probability that you Type I error is the probability that you reject the null given that the null is truereject the null given that the null is true

P(reject HP(reject H00 | H | H00 is true) is true)

Notice that the shaded area is still part of the null curve, but it is in the tail of the distribution

Type II errorType II error Definition:Definition: when you fail to reject when you fail to reject

the null hypothesis when the the null hypothesis when the alternative is in fact true (alternative is in fact true (type II type II errorerror))

This type of error is based on a This type of error is based on a specific alternativespecific alternative

P(fail to reject the HP(fail to reject the H00 | H | HAA is true) is true)

PowerPower Definition:Definition: the probability that you the probability that you

reject the null hypothesis given that reject the null hypothesis given that the alternative hypothesis is true. the alternative hypothesis is true. This is what we want to happen.This is what we want to happen.

Power = P(reject HPower = P(reject Ho o | H| HAA is true) = 1 - is true) = 1 - Since this is a good thing, we want Since this is a good thing, we want

this to be highthis to be high

OutlineOutline Aspects of statistical considerations Aspects of statistical considerations

section of a grantsection of a grant Example statistical analysis sectionExample statistical analysis section Worked example from dataset from Worked example from dataset from

students in classstudents in class Management of data collection/ Management of data collection/

spreadsheetspreadsheet

Aspects of statistical Aspects of statistical considerationsconsiderations

Overarching statistical issues:Overarching statistical issues:– Data managementData management– Methodological issues esp. related to data Methodological issues esp. related to data

collection (ex. Image processing)collection (ex. Image processing)– Handling missing dataHandling missing data– Clustering/correlation of observationsClustering/correlation of observations

Specific aims:Specific aims:– Identify outcomes/explanatory variablesIdentify outcomes/explanatory variables– Type of analysisType of analysis– Power calculationPower calculation

Research studyResearch studyI.I. Study designStudy design

• Experimental question- What are you trying to Experimental question- What are you trying to learn? How will you prove this?learn? How will you prove this?

• Sample selection- Who are you going to study?Sample selection- Who are you going to study?II.II. Data collectionData collection

• What should be collected?What should be collected?III.III. Analysis of dataAnalysis of data

• Results- Was there any effect?Results- Was there any effect?• Conclusions- What does this all mean? To whom Conclusions- What does this all mean? To whom

do results apply?do results apply?

Experimental question: What? How?

Sample selection: Who? How many?

Collect Data

Analysis: Is there an effect?

Conclusion: To whom?

How is statistics related to each How is statistics related to each stage?stage?

I.I. Study designStudy design• Experimental question- Define Experimental question- Define

outcome, sources of variability, outcome, sources of variability, unit and analysis planunit and analysis plan

• Sample selection- Sample size, Sample selection- Sample size, type of sampletype of sample

Experimental questionExperimental question In a grant, the experimental question is In a grant, the experimental question is

written as the specific aimswritten as the specific aims– Generally, specific aims can be easily translated Generally, specific aims can be easily translated

to a null hypothesisto a null hypothesis– If specific aims are more general, the specific If specific aims are more general, the specific

null hypotheses are listed in the grant after the null hypotheses are listed in the grant after the aimsaims

– This is the critical step in the grant because This is the critical step in the grant because everything else is based on the aimseverything else is based on the aims

– Usually easiest if can set up hypothesis as Y/N Usually easiest if can set up hypothesis as Y/N questionquestion

ExampleExample Dr. Janet Hall kindly provided a grant to Dr. Janet Hall kindly provided a grant to

use as an exampleuse as an example One goal of the grant was to investigate One goal of the grant was to investigate

whether age had an effect on estrogen whether age had an effect on estrogen treatment in post-menopausal womantreatment in post-menopausal woman– Is there an interaction between estrogen and Is there an interaction between estrogen and

age?age?– The treatment is given to increase resting The treatment is given to increase resting

metabolic activity in the brain as measured by metabolic activity in the brain as measured by PET and other neuroimaging modalitiesPET and other neuroimaging modalities

In addition, the effect of age on resting In addition, the effect of age on resting metabolic activity at baseline (untreated) metabolic activity at baseline (untreated) was of interestwas of interest

Specific aim 1Specific aim 1 SPECIFIC AIM #1: To determine the effect of SPECIFIC AIM #1: To determine the effect of

aging on changes in baseline (resting state) aging on changes in baseline (resting state) cortical function and their responses to cortical function and their responses to estrogen using FDG-PET.estrogen using FDG-PET.– Hypotheses: Hypotheses: – Resting state metabolic activity, as measured by Resting state metabolic activity, as measured by

FDG-PET at baseline, is decreased in the FDG-PET at baseline, is decreased in the dorsolateral prefrontal cortex (DLPFC) and dorsolateral prefrontal cortex (DLPFC) and increased in the hippocampus as a function of age. increased in the hippocampus as a function of age.

– Estrogen exposure results in progressive increases Estrogen exposure results in progressive increases in resting metabolic activity in the DLPFC over in resting metabolic activity in the DLPFC over time in young postmenopausal women that is not time in young postmenopausal women that is not seen in their older counterparts. seen in their older counterparts.

Hypothesis 1Hypothesis 1 Resting state metabolic activity, as Resting state metabolic activity, as

measured by FDG-PET at baseline, is measured by FDG-PET at baseline, is decreased in the dorsolateral prefrontal decreased in the dorsolateral prefrontal cortex (DLPFC) as a function of age.cortex (DLPFC) as a function of age.

What is the experimental question?What is the experimental question?– Is the FDG-PET level different in the Is the FDG-PET level different in the

hippocampus or DLPFC for women of different hippocampus or DLPFC for women of different ages?ages?

– What is the outcome?What is the outcome?– What is the explanatory variable?What is the explanatory variable?

Types of variablesTypes of variables The outcome is FDG-PET level and this is a The outcome is FDG-PET level and this is a

continuous variablecontinuous variable The explanatory variable, age, could be The explanatory variable, age, could be

considered continuous, but for this grant it considered continuous, but for this grant it was decided to group patients into young was decided to group patients into young post-menopausal women (age 45-55) vs. post-menopausal women (age 45-55) vs. old post-menopausal women (age 70-80) old post-menopausal women (age 70-80)

What type of analysis would we use in this What type of analysis would we use in this case?case?– Are the data approximately normal?Are the data approximately normal?

Sample selectionSample selection Our sample selection is based on the Our sample selection is based on the

definition of the groupsdefinition of the groups– What is the effect of this definition? Does it What is the effect of this definition? Does it

affect the generalizability of the findings?affect the generalizability of the findings? For this study, we plan to sample small For this study, we plan to sample small

groups from a single sitegroups from a single site– Could another approach have been used?Could another approach have been used?– What is the advantage of a single site? What is the advantage of a single site?

Disadvantage?Disadvantage?

Sample size calculationSample size calculation We have defined our null hypothesis, We have defined our null hypothesis,

outcome and sample selectionoutcome and sample selection What sample size do we need?What sample size do we need? In this case, previous data showed mean In this case, previous data showed mean

(SD) FDG-PET at DLPFC in the young group of (SD) FDG-PET at DLPFC in the young group of 83.0 (7.3) and in the old group of 76.2 (7.3)83.0 (7.3) and in the old group of 76.2 (7.3)– What else do we need for our sample size What else do we need for our sample size

calculation?calculation?– Power=0.8, Power=0.8, =0.05=0.05

Assuming equal groups, we need 20 patients Assuming equal groups, we need 20 patients per groupper group

Additional considerationsAdditional considerations Multiple comparisonsMultiple comparisons

– We have two outcomes so should we We have two outcomes so should we adjust the significance level for the two adjust the significance level for the two comparisons?comparisons?

– Bonferroni correction for significance Bonferroni correction for significance levellevel

Do any specifics regarding the Do any specifics regarding the measurement need to be discussed?measurement need to be discussed?

Confounders/adjustmentConfounders/adjustment

Abbreviated grant sectionAbbreviated grant section Analysis plan: The two groups will be Analysis plan: The two groups will be

compared using a two-sample t-test. compared using a two-sample t-test. Power calculation: Previous data has Power calculation: Previous data has

estimated the mean (SD) FDG-PET in the estimated the mean (SD) FDG-PET in the young group of 83.0 (7.3) and in the old group young group of 83.0 (7.3) and in the old group of 76.2 (7.3). Group sample sizes of 20 and 20 of 76.2 (7.3). Group sample sizes of 20 and 20 achieve 82% power to detect a difference of achieve 82% power to detect a difference of 6.8 between the two assuming a standard 6.8 between the two assuming a standard deviations of 7.3 in each group and a deviations of 7.3 in each group and a significance level of 0.05 using a two-sided significance level of 0.05 using a two-sided two-sample t-test.two-sample t-test.

Hypothesis 2Hypothesis 2 Estrogen exposure results in progressive Estrogen exposure results in progressive

increases in resting metabolic activity in increases in resting metabolic activity in the DLPFC over time in young the DLPFC over time in young postmenopausal women that is not seen postmenopausal women that is not seen in their older counterparts. in their older counterparts.

What is the experimental question?What is the experimental question?– Is the effect of estrogen on the FDG-PET Is the effect of estrogen on the FDG-PET

level in the DLPFC different for women of level in the DLPFC different for women of different ages?different ages?

Types of variablesTypes of variables One potential outcome is change in FDG-One potential outcome is change in FDG-

PET level and this is a continuous PET level and this is a continuous variablevariable

Age group and treatment are the Age group and treatment are the explanatory variablesexplanatory variables

How many FDG-PET levels are measured How many FDG-PET levels are measured and how many observations contribute and how many observations contribute to the analysis?to the analysis?

What type of analysis could we use in What type of analysis could we use in this case?this case?

Data set-upData set-up We measure the change in four types of We measure the change in four types of

patients (young/treated, young/placebo, patients (young/treated, young/placebo, old/treated, old/placebo)old/treated, old/placebo)

We can estimate the mean change in all We can estimate the mean change in all groups, but what is truly of interest for our groups, but what is truly of interest for our hypothesis?hypothesis?– Interaction between the two measuresInteraction between the two measures– Linear regression/two-way ANOVALinear regression/two-way ANOVA

TreatedTreated PlaceboPlaceboYoungYoung MeanMeanYoung,treatedYoung,treated MeanMeanYoung,placeboYoung,placebo

OldOld MeanMeanOld,treatedOld,treated MeanMeanOld,placeboOld,placebo

Mean in treated old patients

Mean in untreated old patients

Mean in untreated young patients

Mean in treated young patients

Sample selectionSample selection Now that our outcome and Now that our outcome and

explanatory variable are clearly explanatory variable are clearly defineddefined

Our sample selection in this case is a Our sample selection in this case is a little more complexlittle more complex– Age group is defined by enrollmentAge group is defined by enrollment– Patients in each group were randomized Patients in each group were randomized

to treatment or placeboto treatment or placebo– What does the randomization get for us?What does the randomization get for us?

Sample size calculationSample size calculation We have defined our null hypothesis, We have defined our null hypothesis,

outcome and sample selectionoutcome and sample selection What sample size do we need?What sample size do we need?

– What preliminary data would we need or What preliminary data would we need or what would we need to hypothesize to what would we need to hypothesize to calculate the sample size?calculate the sample size?

Some resources for this complex Some resources for this complex design on-line, but likely you should design on-line, but likely you should consider speaking to a statistician for consider speaking to a statistician for thisthis

Abbreviated grant sectionAbbreviated grant section Analysis plan: The effect of age on the Analysis plan: The effect of age on the

treatment effect of estrogen in post-treatment effect of estrogen in post-menopausal women will be investigated menopausal women will be investigated using a two-way ANOVA. The outcome for using a two-way ANOVA. The outcome for the analysis will be the change in the FDG-the analysis will be the change in the FDG-PET level before and after the treatment and PET level before and after the treatment and the two factors will be age and treatment the two factors will be age and treatment group. The focus of the analysis will be the group. The focus of the analysis will be the interaction between the two factors. interaction between the two factors.

Power calculation: Given our preliminary Power calculation: Given our preliminary data and available sample size, we will have data and available sample size, we will have 80% power to detect a hypothesized 80% power to detect a hypothesized difference of x using a two-way ANOVA.difference of x using a two-way ANOVA.

Alternative analysis strategyAlternative analysis strategy Rather than focusing on the difference Rather than focusing on the difference

between the before and after treatment between the before and after treatment measurements, we could have included all of measurements, we could have included all of the measurements in a single modelthe measurements in a single model– Each patient contributes a before and after Each patient contributes a before and after

treatment measurement rather than a differencetreatment measurement rather than a difference The analysis of this approach requires The analysis of this approach requires

accounting for the repeated measures within accounting for the repeated measures within a subjecta subject– Repeated measures ANOVA or mixed effects Repeated measures ANOVA or mixed effects

modelmodel

Advantages of this approachAdvantages of this approach Handles missing data more easilyHandles missing data more easily Generalizes to more than two Generalizes to more than two

measurements easilymeasurements easily Power calculations with mixed effects Power calculations with mixed effects

models can be completed as wellmodels can be completed as well

ConclusionsConclusions Each hypothesis needs an analysis Each hypothesis needs an analysis

plan that describes the type of data plan that describes the type of data and statistical approach used to and statistical approach used to analyze the dataanalyze the data

Each hypothesis also requires a Each hypothesis also requires a sample size or power calculationsample size or power calculation

Additional issues (missing data, Additional issues (missing data, confounding) must be included in the confounding) must be included in the statistical analysis sectionstatistical analysis section

Worked exampleWorked example

Kidney transplant researchKidney transplant research Students in the class are investigating the Students in the class are investigating the

effect of genetics of the donor/recipient effect of genetics of the donor/recipient pair on various outcomespair on various outcomes– Creatinine level measured at time of Creatinine level measured at time of

transplant, 3 month, 6 month, 12 month and transplant, 3 month, 6 month, 12 month and 36 months after transplant36 months after transplant

– Time to rejection of the transplantTime to rejection of the transplant– Type of rejection (acute/chronic)Type of rejection (acute/chronic)

Genetic factor of interest is large deletion Genetic factor of interest is large deletion polymorphisms at 20 sitespolymorphisms at 20 sites

Study designStudy design Patients have been followed at 4 different Patients have been followed at 4 different

sites since 1995sites since 1995– KoreaKorea– FinlandFinland– BWHBWH– MGHMGH

Only HLA genetic data is available at the Only HLA genetic data is available at the moment, but would like to genotype moment, but would like to genotype sufficient numbers of patients to determine sufficient numbers of patients to determine if there is an effectif there is an effect

Experimental questionExperimental question Specific aim: To explore the potential Specific aim: To explore the potential

contribution of a new class of large deletion contribution of a new class of large deletion polymorphisms on the development of acute polymorphisms on the development of acute and chronic renal allograft rejection following and chronic renal allograft rejection following renal transplantation. renal transplantation. – Hypotheses:Hypotheses:– Donor/recipient pairs with matching deletion Donor/recipient pairs with matching deletion

polymorphisms will have lower creatinine levels at polymorphisms will have lower creatinine levels at all time points compared to non-matched pairs all time points compared to non-matched pairs

– Donor/recipient pairs with matching deletion Donor/recipient pairs with matching deletion polymorphisms will have fewer acute/chronic polymorphisms will have fewer acute/chronic rejection events compared to non-matched pairs rejection events compared to non-matched pairs

Definition of groupsDefinition of groups Both the donor and recipient for each Both the donor and recipient for each

transplant will be genotyped and classified transplant will be genotyped and classified as either having the deletion or not having as either having the deletion or not having the deletionthe deletion

We decided to treat each group separately We decided to treat each group separately initially. What type of variable is the initially. What type of variable is the explanatory variable?explanatory variable?

Recipient with Recipient with deletiondeletion

Recipient w/o Recipient w/o deletiondeletion

Donor with Donor with deletiondeletion

Group 1Group 1 Group 3Group 3

Donor w/o Donor w/o deletiondeletion

Group 2Group 2 Group 4Group 4

Creatinine levelsCreatinine levels Here are the initial Here are the initial

values for the values for the creatinine for one creatinine for one of the populationsof the populations

Note the outliers at Note the outliers at the end of the the end of the distribution. These distribution. These would be very would be very important to modelimportant to model

Turned out they Turned out they were incorrect data were incorrect data

Analysis planAnalysis plan Initially, we will compare each creatinine Initially, we will compare each creatinine

measurement separatelymeasurement separately Since I have 4 groups (categorical Since I have 4 groups (categorical

outcome) and a continuous outcome, I outcome) and a continuous outcome, I will compare across the groups using will compare across the groups using ANOVAANOVA– The corrected data look sufficiently normal The corrected data look sufficiently normal

to make this analysis plan reasonableto make this analysis plan reasonable– An alternative option would be to use a An alternative option would be to use a

Kruskal-Wallis test, which is a rank-based Kruskal-Wallis test, which is a rank-based test that is not sensitive to the outlierstest that is not sensitive to the outliers

Abbreviated analysis planAbbreviated analysis plan Analysis plan: The four groups of Analysis plan: The four groups of

donor/recipient pairs will be compared using donor/recipient pairs will be compared using ANOVA. If a significant difference between ANOVA. If a significant difference between the groups is observed, the pairwise the groups is observed, the pairwise comparisons will be completed with the comparisons will be completed with the appropriate correction for multiple appropriate correction for multiple comparisons. Although we could investigate comparisons. Although we could investigate the main effect of the donor’s and recipient’s the main effect of the donor’s and recipient’s deletion status in a two-way ANOVA model, deletion status in a two-way ANOVA model, our interest is in the four group comparison our interest is in the four group comparison given the relationships seen in previous given the relationships seen in previous work. work.

Additional considerationsAdditional considerations Rather than modeling each creatinine Rather than modeling each creatinine

separately, should we model them separately, should we model them together?together?– Trend with time?Trend with time?– Multiple comparisons if treat separately?Multiple comparisons if treat separately?

Confounders:Confounders:– AgeAge– GenderGender– HLA statusHLA status

Should we treat all 20 deletion separately?Should we treat all 20 deletion separately?

Power calculationPower calculation Unlike the previous example, we have Unlike the previous example, we have

no preliminary data regarding the no preliminary data regarding the effect of these deletionseffect of these deletions

How can we complete a power How can we complete a power calculation?calculation?– Option 1: Propose a sample size from Option 1: Propose a sample size from

each group and determine the difference each group and determine the difference between groups you could detectbetween groups you could detect

– Option 2: Estimate the effect using an Option 2: Estimate the effect using an available measurement/literature valueavailable measurement/literature value

Available measurementAvailable measurement In the dataset, we have HLA status In the dataset, we have HLA status

and can calculate the mean (SD) in and can calculate the mean (SD) in each of the four groupseach of the four groups

Using this preliminary data, we can Using this preliminary data, we can perform a power calculation and perform a power calculation and assume the effect size for the assume the effect size for the deletions will be similar to HLAdeletions will be similar to HLA– How good of a surrogate is HLA for How good of a surrogate is HLA for

deletion?deletion?

Abbreviated power Abbreviated power calculationcalculation

Power calculation: Our preliminary data have Power calculation: Our preliminary data have shown that the mean (SD) month 12 shown that the mean (SD) month 12 creatinine levels of the recipient was 1.21 creatinine levels of the recipient was 1.21 (0.29) in HLA identical donor/recipient pairs (0.29) in HLA identical donor/recipient pairs and 1.28 (0.33) in HLA non-identical and 1.28 (0.33) in HLA non-identical donor/recipient pairs. We anticipate that donor/recipient pairs. We anticipate that recipients who are deletion matches will recipients who are deletion matches will behave like the HLA identical recipients and behave like the HLA identical recipients and recipients who are not deletion matches will recipients who are not deletion matches will behave like the HLA non-identical recipients. behave like the HLA non-identical recipients. A sample size of 202 per group is required to A sample size of 202 per group is required to have 80% power to detect the proposed have 80% power to detect the proposed difference between the groups at the 0.05 difference between the groups at the 0.05 level using one-way ANOVAlevel using one-way ANOVA

Additional considerationsAdditional considerations Pairwise testsPairwise tests Is there a better approximation for Is there a better approximation for

the group means?the group means? Clustering by countryClustering by country

Proportion with acute Proportion with acute rejectionrejection

Another outcome for the study is the Another outcome for the study is the proportion of patients who experience proportion of patients who experience acute rejectionacute rejection

The table at the end of the study would The table at the end of the study would look like this:look like this:

DDyesyes/R/Ryesyes DDnono/R/Ryesyes DDyesyes/R/Rnono DDnono/R/Rnono

Acute Acute rejectionrejectionNo acute No acute rejectionrejection

Abbreviated analysis planAbbreviated analysis plan Analysis plan: The proportion of Analysis plan: The proportion of

patients who have acute rejection patients who have acute rejection will be compared across the groups will be compared across the groups using a chi-square test for each using a chi-square test for each deletion separately. In order to deletion separately. In order to investigate the combined effect of investigate the combined effect of deletions, multiple logistic regression deletions, multiple logistic regression models will also be fit.models will also be fit.

Power analysisPower analysis As previously, there is no preliminary As previously, there is no preliminary

data, but let’s try the set sample size data, but let’s try the set sample size approach nowapproach now

Assume that we have two groups, Assume that we have two groups, matched and non-matched, and we matched and non-matched, and we have 200 matched patients and 400 have 200 matched patients and 400 non-matched patientsnon-matched patients

What type of power analysis could What type of power analysis could we complete?we complete?

Abbreviated power analysisAbbreviated power analysis Power analysis: Given our sample size Power analysis: Given our sample size

(200 matched patients and 400 non-(200 matched patients and 400 non-matched patients) and the assumption matched patients) and the assumption that matching would decrease the that matching would decrease the proportion with acute rejection, we will proportion with acute rejection, we will have at least 80% power to detect the have at least 80% power to detect the differences presented in Table xx.differences presented in Table xx.Proportion with acute Proportion with acute

rejection among rejection among matchesmatches

Proportion with acute Proportion with acute rejection among rejection among mismatchesmismatches

0.20.2 0.330.330.30.3 0.450.450.40.4 0.550.550.50.5 0.650.65

Additional considerationsAdditional considerations Clustering by regionClustering by region

– Stratified analysisStratified analysis

Management of data Management of data collectioncollection

Example Excel sheetExample Excel sheet All relevant information should be All relevant information should be

included as a column (Loss to follow-up included as a column (Loss to follow-up date)date)

No symbols in column names (ex. #) No symbols in column names (ex. #) and column names should be as short and column names should be as short as possibleas possible

No empty rows or columns (white No empty rows or columns (white space)space)

ConclusionsConclusions Experimental question must be well Experimental question must be well

defined to set up an appropriate defined to set up an appropriate analysis plananalysis plan

Sample size calculation based on Sample size calculation based on analysis plan. If uncertain of power analysis plan. If uncertain of power calculation, consult a statisticiancalculation, consult a statistician

Attempt to address other aspects of Attempt to address other aspects of your datayour data

THANK YOU!!!THANK YOU!!!