introduction to resampling techniques for generating confidence measures

Introduction to resampling techniques for generating confidence measures

2

Resampling techniques

1) Randomization– Resampling without replacement (re-ordering, permutations)

2) Jackknife– Leaving one data point out at a time (not good for small sample sizes),

in paleobiology usually used for phylogenetic analyses3) Sampling Standardization

– When comparing samples of different sizes 4) Bootstrap

– Parametric• Generate datasets from a parametrized model and comparing

these with empirical data– Non parametric

• Most common in paleobiology

Empirical Data Randomization

Randomized Sample 1

Randomized Sample 2

Randomized Sample 3

…. Randomized Sample N

Jack knife sample 3

Jack-KnifeEmpirical Data

Jack knife sample 3

Jack knife sample 1

…..Jack knife sample N

Empirical data 1 Empirical data 2 Empirical data 3

Standardized Sample 1

Standardized Sample 2

… Standardized Sample N

Sampling Standardization

Bootstrapped Sample 1

Bootstrapped Sample 3

Bootstrapped Sample 2

….. Bootstrapped Sample N

Non-parametric bootstrapEmpirical Data

Empirical data

Estimate parameters

(model)

Bootstraps samples

Estimate parameters

Empirical data

Estimate parameters

(model)

Simulated samples

Estimate parameters

Non-parametric bootstrap Parametric bootstrap

8

Resampling techniques

1) Randomization– Resampling without replacement (re-ordering, permutations)

2) Jackknife– Leaving one data point out at a time (not good for small sample sizes),

in paleobiology usually used for phylogenetic analyses3) Sampling Standardization

– When comparing samples of different sizes 4) Bootstrap

– Parametric• Generate datasets from a parametrized model and comparing

these with empirical data– Non parametric

• Most common in paleobiology

9

Why resampling (now)

• Underlying distribution of data not well understood and/or complex

• Convenient way to generate uncertainty measures• Computer intensive (possible only with faster computers)

Bootstrapping• construct estimate of frequency distributions expected from a “generative

process”• Equivalent to generating replicate outcomes from an experiment (doing

something many times to see the range of results)• Assumption: data are representative sample of independent observations

derived randomly from the studied statistical population

Bootstrap error estimates• Estimate standard error by resampling from the single sample we have.• This approach uses sampling with replacement from observed sample to

simulate sampling without replacement from the underlying distribution.Procedure• Start with observed sample of size n and observed sample statistic, call it Z.• Randomly pick a sample of size n, with replacement, from the

observedsample.• Calculate the sample statistic of interest on this random sample; call

isZboot.• Repeat many times (generally hundreds to thousands, ideally untilestimate

of SE stabilizes).• Calculate standard deviation of the Zboot.• This is an estimate of the standard error of the observed sample statistic

Z:SD(Zboot) ≈ SE(Z)

Example (sampling standardization)Alroy et al. 2008. Phanerozoic trends in the global diversity of marine invertebrates. Science 321:97-100

Foote, M. 2006. Substrate affinity and diversity dynamics of Paleozoic marine animals Paleobiology 32:345-366.

Example (non parametric bootstrap)

Example (non parametric bootstrap)

Liow et al- 2009. Lower extinction risk in Sleep-or-Hide Mammals. Am Nat 173:264–272.

R demo• Packages (e.g. boot, boostrap)• Write your own: use the function

sampleNice help http://www.ats.ucla.edu/stat/r/library/bootstrap.htm

Links• http://www.paleo.geos.vt.edu/MK/Kowalewski_PNG_2010.pdf• http://www.stat.cmu.edu/~cshalizi/402/lectures/08-bootstrap/lecture-08.pdf