fmri multiple comparisons problem
DESCRIPTION
The False Discovery Rate A New Approach to the Multiple Comparisons Problem Thomas Nichols Department of Biostatistics University of Michigan. fMRI Multiple Comparisons Problem. 4-Dimensional Data 1,000 multivariate observations, each with 100,000 elements - PowerPoint PPT PresentationTRANSCRIPT
The False Discovery Rate
A New Approach to the Multiple Comparisons Problem
Thomas NicholsDepartment of Biostatistics
University of Michigan
The False Discovery Rate
A New Approach to the Multiple Comparisons Problem
Thomas NicholsDepartment of Biostatistics
University of Michigan
fMRI Multiple Comparisons ProblemfMRI Multiple Comparisons Problem
• 4-Dimensional Data– 1,000 multivariate observations,
each with 100,000 elements– 100,000 time series, each
with 1,000 observations
• Massively UnivariateApproach– 100,000 hypothesis
tests
• Massive MCP!
• 4-Dimensional Data– 1,000 multivariate observations,
each with 100,000 elements– 100,000 time series, each
with 1,000 observations
• Massively UnivariateApproach– 100,000 hypothesis
tests
• Massive MCP!
1,000
1
2
3
. . .
Solutions forMultiple Comparison Problem
Solutions forMultiple Comparison Problem
• A MCP Solution Must Control False Positives– How to measure multiple false positives?
• Familywise Error Rate (FWER)– Chance of any false positives– Controlled by Bonferroni & Random Field Methods
• False Discovery Rate (FDR)– Proportion of false positives among rejected tests
• A MCP Solution Must Control False Positives– How to measure multiple false positives?
• Familywise Error Rate (FWER)– Chance of any false positives– Controlled by Bonferroni & Random Field Methods
• False Discovery Rate (FDR)– Proportion of false positives among rejected tests
False Discovery RateIllustration:
False Discovery RateIllustration:
Signal
Signal+Noise
Noise
FWE
6.7% 10.4% 14.9% 9.3% 16.2% 13.8% 14.0% 10.5% 12.2% 8.7%
Control of Familywise Error Rate at 10%
11.3% 11.3% 12.5% 10.8% 11.5% 10.0% 10.7% 11.2% 10.2% 9.5%
Control of Per Comparison Rate at 10%
Percentage of Null Pixels that are False Positives
Control of False Discovery Rate at 10%
Occurrence of Familywise Error
Percentage of Activated Pixels that are False Positives
Benjamini & Hochberg ProcedureBenjamini & Hochberg Procedure
• Select desired limit q on E(FDR)• Order p-values, p(1) p(2) ... p(V)
• Let r be largest i such that
• Reject all hypotheses corresponding to p(1), ... , p(r).
• Select desired limit q on E(FDR)• Order p-values, p(1) p(2) ... p(V)
• Let r be largest i such that
• Reject all hypotheses corresponding to p(1), ... , p(r).
p(i) i/V q/c(V)
p(i)
i/V
i/V q/c(V)p-
valu
e0 1
01
JRSS-B (1995) 57:289-300
Benjamini & Hochberg ProcedureBenjamini & Hochberg Procedure
• c(V) = 1– Positive Regression Dependency on Subsets
• Technical condition, special cases include
– Independent data
– Multivariate Normal with all positive correlations• Result by Benjamini & Yekutieli.
• c(V) = i=1,...,V 1/i log(V)+0.5772
– Arbitrary covariance structure
• c(V) = 1– Positive Regression Dependency on Subsets
• Technical condition, special cases include
– Independent data
– Multivariate Normal with all positive correlations• Result by Benjamini & Yekutieli.
• c(V) = i=1,...,V 1/i log(V)+0.5772
– Arbitrary covariance structure
Benjamini & Hochberg:Varying Signal Extent
Benjamini & Hochberg:Varying Signal Extent
Signal Intensity 3.0 Signal Extent 1.0 Noise Smoothness 3.0
p = z =
1
Benjamini & Hochberg:Varying Signal Extent
Benjamini & Hochberg:Varying Signal Extent
Signal Intensity 3.0 Signal Extent 2.0 Noise Smoothness 3.0
p = z =
2
Benjamini & Hochberg:Varying Signal Extent
Benjamini & Hochberg:Varying Signal Extent
Signal Intensity 3.0 Signal Extent 3.0 Noise Smoothness 3.0
p = z =
3
Benjamini & Hochberg:Varying Signal Extent
Benjamini & Hochberg:Varying Signal Extent
Signal Intensity 3.0 Signal Extent 5.0 Noise Smoothness 3.0
p = 0.000252 z = 3.48
4
Benjamini & Hochberg:Varying Signal Extent
Benjamini & Hochberg:Varying Signal Extent
Signal Intensity 3.0 Signal Extent 9.5 Noise Smoothness 3.0
p = 0.001628 z = 2.94
5
Benjamini & Hochberg:Varying Signal Extent
Benjamini & Hochberg:Varying Signal Extent
Signal Intensity 3.0 Signal Extent 16.5 Noise Smoothness 3.0
p = 0.007157 z = 2.45
6
Benjamini & Hochberg:Varying Signal Extent
Benjamini & Hochberg:Varying Signal Extent
Signal Intensity 3.0 Signal Extent 25.0 Noise Smoothness 3.0
p = 0.019274 z = 2.07
7
Benjamini & Hochberg: PropertiesBenjamini & Hochberg: Properties
• Adaptive– Larger the signal, the lower the threshold– Larger the signal, the more false positives
• False positives constant as fraction of rejected tests
• Not a problem with imaging’s sparse signals
• Smoothness OK– Smoothing introduces positive correlations
• Adaptive– Larger the signal, the lower the threshold– Larger the signal, the more false positives
• False positives constant as fraction of rejected tests
• Not a problem with imaging’s sparse signals
• Smoothness OK– Smoothing introduces positive correlations
FDR: ExampleFDR: Example
• Verbal fluency data
• 14 42-second blocks • ABABAB...
• A: Two syllable words presented aurally
• B: Silence
• Imaging parameters– 2Tesla scanner, TR = 7 sec– 84 64x64x64 images of 3 x 3 x 3 mm voxels
• Verbal fluency data
• 14 42-second blocks • ABABAB...
• A: Two syllable words presented aurally
• B: Silence
• Imaging parameters– 2Tesla scanner, TR = 7 sec– 84 64x64x64 images of 3 x 3 x 3 mm voxels
FDR Example:Plot of FDR Inequality
FDR Example:Plot of FDR Inequality
p(i) ( i/V ) ( q/c(V) )
FDR: ExampleFDR: Example
FDR 0.05Indep/PRDSt0 = 3.8119
FWER 0.05Bonferronit0 = 5.485
FDR 0.05Arbitrary Cov.
t0 = 5.0747
FDR Software for SPMFDR Software for SPM
http://www.sph.umich.edu/~nichols/FDR
FDR: ConclusionsFDR: Conclusions
• False Discovery Rate– A new false positive metric
• Benjamini & Hochberg FDR Method– Straightforward solution to fNI MCP– Just one way of controlling FDR
• New methods under developmente.g. C. Genovese or J. Storey
• Limitations– Arbitrary dependence result less sensitive
• False Discovery Rate– A new false positive metric
• Benjamini & Hochberg FDR Method– Straightforward solution to fNI MCP– Just one way of controlling FDR
• New methods under developmente.g. C. Genovese or J. Storey
• Limitations– Arbitrary dependence result less sensitive
http://www.sph.umich.edu/~nichols/FDR Prop
Ill
Start
ReferencesReferences
• Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57:289--300.
• Benjamini, Y, Yekutieli D (2002). The control of the false discovery rate under dependence. Annals of Statistics. To appear.
• Genovese CR, Lazar N, Nichols TE (2002). Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate. NeuroImage, 15:870-878.
• Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57:289--300.
• Benjamini, Y, Yekutieli D (2002). The control of the false discovery rate under dependence. Annals of Statistics. To appear.
• Genovese CR, Lazar N, Nichols TE (2002). Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate. NeuroImage, 15:870-878.
Positive Regression DependencyPositive Regression Dependency
• Does fMRI data exhibit total positive correlation?• Example
– 160 scan experiment
– Spatialautocorrelationof residuals
– Single voxelwith all others
• Negative correlationexists!
• Does fMRI data exhibit total positive correlation?• Example
– 160 scan experiment
– Spatialautocorrelationof residuals
– Single voxelwith all others
• Negative correlationexists!