glasgow seminar, april 20, 2012
TRANSCRIPT
ABC Methods for Bayesian Model Choice
ABC Methods for Bayesian Model Choice
Christian P. Robert
Universite Paris-Dauphine, IuF, & CRESThttp://www.ceremade.dauphine.fr/~xian
Glasgow University, April 20, 2012Joint work with J.-M. Cornuet, A. Grelaud,
J.-M. Marin, N. Pillai, & J. Rousseau
ABC Methods for Bayesian Model Choice
Outline
Approximate Bayesian computation
ABC for model choice
Model choice consistency
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Approximate Bayesian computation
Approximate Bayesian computationABC basicsAlphabet soupSemi-automatic ABCsimulation-based methods inEconometrics
ABC for model choice
Model choice consistency
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
Bayesian computation issues
When faced with a non-standard posterior distribution
π(θ|y) ∝ π(θ)L(θ|y)
the standard solution is to use simulation (Monte Carlo) toproduce a sample
θ1, . . . , θT
from π(θ|y) (or approximately by Markov chain Monte Carlomethods)
[Robert & Casella, 2004]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
Untractable likelihoods
Cases when the likelihood function f(y|θ) is unavailable and whenthe completion step
f(y|θ) =
∫Zf(y, z|θ) dz
is impossible or too costly because of the dimension of zc© MCMC cannot be implemented!
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
Illustrations
Example
Stochastic volatility model: fort = 1, . . . , T,
yt = exp(zt)εt , zt = a+bzt−1+σηt ,
T very large makes it difficult toinclude z within the simulatedparameters
0 200 400 600 800 1000
−0.
4−
0.2
0.0
0.2
0.4
t
Highest weight trajectories
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
Illustrations
Example
Potts model: if y takes values on a grid Y of size kn and
f(y|θ) ∝ exp
{θ∑l∼i
Iyl=yi
}
where l∼i denotes a neighbourhood relation, n moderately largeprohibits the computation of the normalising constant
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
Illustrations
Example
Coalescence tree: in populationgenetics, reconstitution of a commonancestor from a sample of genes viaa phylogenetic tree that is close toimpossible to integrate out[100 processor days with 4parameters]
[Cornuet et al., 2009, Bioinformatics]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
The ABC method
Bayesian setting: target is π(θ)f(x|θ)
When likelihood f(x|θ) not in closed form, likelihood-free rejectiontechnique:
ABC algorithm
For an observation y ∼ f(y|θ), under the prior π(θ), keep jointlysimulating
θ′ ∼ π(θ) , z ∼ f(z|θ′) ,
until the auxiliary variable z is equal to the observed value, z = y.
[Tavare et al., 1997]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
The ABC method
Bayesian setting: target is π(θ)f(x|θ)When likelihood f(x|θ) not in closed form, likelihood-free rejectiontechnique:
ABC algorithm
For an observation y ∼ f(y|θ), under the prior π(θ), keep jointlysimulating
θ′ ∼ π(θ) , z ∼ f(z|θ′) ,
until the auxiliary variable z is equal to the observed value, z = y.
[Tavare et al., 1997]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
The ABC method
Bayesian setting: target is π(θ)f(x|θ)When likelihood f(x|θ) not in closed form, likelihood-free rejectiontechnique:
ABC algorithm
For an observation y ∼ f(y|θ), under the prior π(θ), keep jointlysimulating
θ′ ∼ π(θ) , z ∼ f(z|θ′) ,
until the auxiliary variable z is equal to the observed value, z = y.
[Tavare et al., 1997]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
A as approximative
When y is a continuous random variable, equality z = y is replacedwith a tolerance condition,
%(y, z) ≤ ε
where % is a distance
Output distributed from
π(θ)Pθ{%(y, z) < ε} ∝ π(θ|%(y, z) < ε)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
A as approximative
When y is a continuous random variable, equality z = y is replacedwith a tolerance condition,
%(y, z) ≤ ε
where % is a distanceOutput distributed from
π(θ)Pθ{%(y, z) < ε} ∝ π(θ|%(y, z) < ε)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
ABC algorithm (with summary)
Algorithm 1 Likelihood-free rejection sampler
for i = 1 to N dorepeat
generate θ′ from the prior distribution π(·)generate z from the likelihood f(·|θ′)
until ρ{η(z), η(y)} ≤ εset θi = θ′
end for
where η(y) defines a (maybe in-sufficient) statistic
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
Output (with summary)
The likelihood-free algorithm samples from the marginal in z of:
πε(θ, z|y) =π(θ)f(z|θ)IAε,y(z)∫
Aε,y×Θ π(θ)f(z|θ)dzdθ,
where Aε,y = {z ∈ D|ρ(η(z), η(y)) < ε}.
The idea behind ABC is that the summary statistics coupled with asmall tolerance should provide a good approximation of theposterior distribution based on the summary statistic:
πε(θ|y) =
∫πε(θ, z|y)dz ≈ π(θ|η(y)) .
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
Output (with summary)
The likelihood-free algorithm samples from the marginal in z of:
πε(θ, z|y) =π(θ)f(z|θ)IAε,y(z)∫
Aε,y×Θ π(θ)f(z|θ)dzdθ,
where Aε,y = {z ∈ D|ρ(η(z), η(y)) < ε}.
The idea behind ABC is that the summary statistics coupled with asmall tolerance should provide a good approximation of theposterior distribution based on the summary statistic:
πε(θ|y) =
∫πε(θ, z|y)dz ≈ π(θ|η(y)) .
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
MA example
Consider the MA(q) model
xt = εt +
q∑i=1
ϑiεt−i
Simple prior: uniform prior over the identifiability zone, e.g.triangle for MA(2)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
MA example (2)
ABC algorithm thus made of
1. picking a new value (ϑ1, ϑ2) in the triangle
2. generating an iid sequence (εt)−q<t≤T
3. producing a simulated series (x′t)1≤t≤T
Distance: basic distance between the series
ρ((x′t)1≤t≤T , (xt)1≤t≤T ) =T∑t=1
(xt − x′t)2
or between summary statistics like the first 2 autocorrelations
τj =T∑
t=j+1
xtxt−j
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
MA example (2)
ABC algorithm thus made of
1. picking a new value (ϑ1, ϑ2) in the triangle
2. generating an iid sequence (εt)−q<t≤T
3. producing a simulated series (x′t)1≤t≤T
Distance: basic distance between the series
ρ((x′t)1≤t≤T , (xt)1≤t≤T ) =T∑t=1
(xt − x′t)2
or between summary statistics like the first 2 autocorrelations
τj =
T∑t=j+1
xtxt−j
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
Comparison of distance impact
Impact of the tolerance on the ABC sample against both distances(ε = 100%, 10%, 1%, 0.1%) for an MA(2) model
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
Comparison of distance impact
0.0 0.2 0.4 0.6 0.8
01
23
4
θ1
−2.0 −1.0 0.0 0.5 1.0 1.5
0.00.5
1.01.5
θ2
Impact of the tolerance on the ABC sample against raw distance(ε = 100%, 10%, 1%, 0.1%) for an MA(2) model
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
Comparison of distance impact
0.0 0.2 0.4 0.6 0.8
01
23
4
θ1
−2.0 −1.0 0.0 0.5 1.0 1.5
0.00.5
1.01.5
θ2
Impact of the tolerance on the ABC sample againstautocorrelation distance (ε = 100%, 10%, 1%, 0.1%) for an MA(2)model
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
ABC advances
Simulating from the prior is often poor in efficiency
Either modify the proposal distribution on θ to increase the densityof x’s within the vicinity of y...
[Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007]
...or by viewing the problem as a conditional density estimationand by developing techniques to allow for larger ε
[Beaumont et al., 2002]
.....or even by including ε in the inferential framework [ABCµ][Ratmann et al., 2009]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
ABC advances
Simulating from the prior is often poor in efficiencyEither modify the proposal distribution on θ to increase the densityof x’s within the vicinity of y...
[Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007]
...or by viewing the problem as a conditional density estimationand by developing techniques to allow for larger ε
[Beaumont et al., 2002]
.....or even by including ε in the inferential framework [ABCµ][Ratmann et al., 2009]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
ABC advances
Simulating from the prior is often poor in efficiencyEither modify the proposal distribution on θ to increase the densityof x’s within the vicinity of y...
[Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007]
...or by viewing the problem as a conditional density estimationand by developing techniques to allow for larger ε
[Beaumont et al., 2002]
.....or even by including ε in the inferential framework [ABCµ][Ratmann et al., 2009]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
ABC basics
ABC advances
Simulating from the prior is often poor in efficiencyEither modify the proposal distribution on θ to increase the densityof x’s within the vicinity of y...
[Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007]
...or by viewing the problem as a conditional density estimationand by developing techniques to allow for larger ε
[Beaumont et al., 2002]
.....or even by including ε in the inferential framework [ABCµ][Ratmann et al., 2009]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Alphabet soup
ABC-NP
Better usage of [prior] simulations byadjustement: instead of throwing awayθ′ such that ρ(η(z), η(y)) > ε, replaceθ’s with locally regressed transforms
θ∗ = θ − {η(z)− η(y)}Tβ[Csillery et al., TEE, 2010]
where β is obtained by [NP] weighted least square regression on(η(z)− η(y)) with weights
Kδ {ρ(η(z), η(y))}
[Beaumont et al., 2002, Genetics]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Alphabet soup
ABC-NP (density estimation)
Use of the kernel weights
Kδ {ρ(η(z(θ)), η(y))}
leads to the NP estimate of the posterior expectation∑i θiKδ {ρ(η(z(θi)), η(y))}∑iKδ {ρ(η(z(θi)), η(y))}
[Blum, JASA, 2010]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Alphabet soup
ABC-NP (density estimation)
Use of the kernel weights
Kδ {ρ(η(z(θ)), η(y))}
leads to the NP estimate of the posterior conditional density∑i Kb(θi − θ)Kδ {ρ(η(z(θi)), η(y))}∑
iKδ {ρ(η(z(θi)), η(y))}
[Blum, JASA, 2010]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Alphabet soup
ABC-NP (density estimations)
Other versions incorporating regression adjustments∑i Kb(θ
∗i − θ)Kδ {ρ(η(z(θi)), η(y))}∑iKδ {ρ(η(z(θi)), η(y))}
In all cases, error
E[g(θ|y)]− g(θ|y) = cb2 + cδ2 + OP (b2 + δ2) + OP (1/nδd)
var(g(θ|y)) =c
nbδd(1 + oP (1))
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Alphabet soup
ABC-NP (density estimations)
Other versions incorporating regression adjustments∑i Kb(θ
∗i − θ)Kδ {ρ(η(z(θi)), η(y))}∑iKδ {ρ(η(z(θi)), η(y))}
In all cases, error
E[g(θ|y)]− g(θ|y) = cb2 + cδ2 + OP (b2 + δ2) + OP (1/nδd)
var(g(θ|y)) =c
nbδd(1 + oP (1))
[Blum, JASA, 2010]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Alphabet soup
ABC-NP (density estimations)
Other versions incorporating regression adjustments∑i Kb(θ
∗i − θ)Kδ {ρ(η(z(θi)), η(y))}∑iKδ {ρ(η(z(θi)), η(y))}
In all cases, error
E[g(θ|y)]− g(θ|y) = cb2 + cδ2 + OP (b2 + δ2) + OP (1/nδd)
var(g(θ|y)) =c
nbδd(1 + oP (1))
[standard NP calculations]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Alphabet soup
Which summary statistics?
Fundamental difficulty of the choice of the summary statistic whenthere is no non-trivial sufficient statistic [except when done by theexperimenters in the field]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Alphabet soup
Which summary statistics?
Fundamental difficulty of the choice of the summary statistic whenthere is no non-trivial sufficient statistic [except when done by theexperimenters in the field]
Starting from a large collection of summary statistics is available,Joyce and Marjoram (2008) consider the sequential inclusion intothe ABC target, with a stopping rule based on a likelihood ratiotest.
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Semi-automatic ABC
Semi-automatic ABC
Fearnhead and Prangle (2012) study ABC and the selection of thesummary statistic in close proximity to Wilkinson’s (2008) proposalABC then considered from a purely inferential viewpoint andcalibrated for estimation purposes
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Semi-automatic ABC
Semi-automatic ABC
Fearnhead and Prangle (2012) study ABC and the selection of thesummary statistic in close proximity to Wilkinson’s (2008) proposalABC then considered from a purely inferential viewpoint andcalibrated for estimation purposesUse of a randomised (or ‘noisy’) version of the summary statistics
η(y) = η(y) + τε
Derivation of a well-calibrated version of ABC, i.e. an algorithmthat gives proper predictions for the distribution associated withthis randomised summary statistic [calibration constraint: ABCapproximation with same posterior mean as the true randomisedposterior]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Semi-automatic ABC
Semi-automatic ABC
Fearnhead and Prangle (2012) study ABC and the selection of thesummary statistic in close proximity to Wilkinson’s (2008) proposalABC then considered from a purely inferential viewpoint andcalibrated for estimation purposes
I Optimality of the posterior expectation E[θ|y] of theparameter of interest as summary statistics η(y)!
I Use of the standard quadratic loss function
(θ − θ0)TA(θ − θ0) .
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Semi-automatic ABC
Details on Fearnhead and Prangle (F&P) ABC
Use of a summary statistic S(·), an importance proposal g(·), akernel K(·) ≤ 1 and a bandwidth h > 0 such that
(θ,ysim) ∼ g(θ)f(ysim|θ)
is accepted with probability (hence the bound)
K[{S(ysim)− sobs}/h]
and the corresponding importance weight defined by
π(θ)/g(θ)
[Fearnhead & Prangle, 2012]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Semi-automatic ABC
Errors, errors, and errors
Three levels of approximation
I π(θ|yobs) by π(θ|sobs) loss of information[ignored]
I π(θ|sobs) by
πABC(θ|sobs) =
∫π(s)K[{s− sobs}/h]π(θ|s) ds∫
π(s)K[{s− sobs}/h] ds
noisy observations
I πABC(θ|sobs) by importance Monte Carlo based on Nsimulations, represented by var(a(θ)|sobs)/Nacc [expectednumber of acceptances]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Semi-automatic ABC
Average acceptance asymptotics
For the average acceptance probability/approximate likelihood
p(θ|sobs) =
∫f(ysim|θ)K[{S(ysim)− sobs}/h] dysim ,
overall acceptance probability
p(sobs) =
∫p(θ|sobs)π(θ) dθ = π(sobs)h
d + o(hd)
[F&P, Lemma 1]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Semi-automatic ABC
Calibration of h
“This result gives insight into how S(·) and h affect the MonteCarlo error. To minimize Monte Carlo error, we need hd to be nottoo small. Thus ideally we want S(·) to be a low dimensionalsummary of the data that is sufficiently informative about θ thatπ(θ|sobs) is close, in some sense, to π(θ|yobs)” (F&P, p.5)
I turns h into an absolute value while it should becontext-dependent and user-calibrated
I only addresses one term in the approximation error andacceptance probability (“curse of dimensionality”)
I h large prevents πABC(θ|sobs) to be close to π(θ|sobs)
I d small prevents π(θ|sobs) to be close to π(θ|yobs) (“curse of[dis]information”)
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Semi-automatic ABC
Converging ABC
Theorem (F&P)
For noisy ABC, the expected noisy-ABC log-likelihood,
E {log[p(θ|sobs)]} =
∫ ∫log[p(θ|S(yobs) + ε)]π(yobs|θ0)K(ε)dyobsdε,
has its maximum at θ = θ0.
True for any choice of summary statistic? even ancilary statistics?![Imposes at least identifiability...]
Relevant in asymptotia and not for the data
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Semi-automatic ABC
Converging ABC
Corollary (F&P)
For noisy ABC, the ABC posterior converges onto a point mass onthe true parameter value as m→∞.
For standard ABC, not always the case (unless h goes to zero).
Strength of regularity conditions (c1) and (c2) in Bernardo& Smith, 1994?
[out-of-reach constraints on likelihood and posterior]Again, there must be conditions imposed upon summarystatistics...
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Semi-automatic ABC
Caveat
Since E(θ|yobs) is most usually unavailable, F&P suggest
(i) use a pilot run of ABC to determine a region of non-negligibleposterior mass;
(ii) simulate sets of parameter values and data;
(iii) use the simulated sets of parameter values and data toestimate the summary statistic; and
(iv) run ABC with this choice of summary statistic.
where is the assessment of the first stage error?
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
Semi-automatic ABC
Caveat
Since E(θ|yobs) is most usually unavailable, F&P suggest
(i) use a pilot run of ABC to determine a region of non-negligibleposterior mass;
(ii) simulate sets of parameter values and data;
(iii) use the simulated sets of parameter values and data toestimate the summary statistic; and
(iv) run ABC with this choice of summary statistic.
where is the assessment of the first stage error?
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
simulation-based methods in Econometrics
Econ’ections
Model choice
Similar exploration of simulation-based techniques in Econometrics
I Simulated method of moments
I Method of simulated moments
I Simulated pseudo-maximum-likelihood
I Indirect inference
[Gourieroux & Monfort, 1996]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
simulation-based methods in Econometrics
Simulated method of moments
Given observations yo1:n from a model
yt = r(y1:(t−1), εt, θ) , εt ∼ g(·)
simulate ε?1:n, derive
y?t (θ) = r(y1:(t−1), ε?t , θ)
and estimate θ by
arg minθ
n∑t=1
(yot − y?t (θ))2
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
simulation-based methods in Econometrics
Simulated method of moments
Given observations yo1:n from a model
yt = r(y1:(t−1), εt, θ) , εt ∼ g(·)
simulate ε?1:n, derive
y?t (θ) = r(y1:(t−1), ε?t , θ)
and estimate θ by
arg minθ
{n∑t=1
yot −n∑t=1
y?t (θ)
}2
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
simulation-based methods in Econometrics
Method of simulated moments
Given a statistic vector K(y) with
Eθ[K(Yt)|y1:(t−1)] = k(y1:(t−1); θ)
find an unbiased estimator of k(y1:(t−1); θ),
k(εt, y1:(t−1); θ)
Estimate θ by
arg minθ
∣∣∣∣∣∣∣∣∣∣n∑t=1
[K(yt)−
S∑s=1
k(εst , y1:(t−1); θ)/S
]∣∣∣∣∣∣∣∣∣∣
[Pakes & Pollard, 1989]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
simulation-based methods in Econometrics
Indirect inference
Minimise (in θ) the distance between estimators β based onpseudo-models for genuine observations and for observationssimulated under the true model and the parameter θ.
[Gourieroux, Monfort, & Renault, 1993;Smith, 1993; Gallant & Tauchen, 1996]
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
simulation-based methods in Econometrics
Indirect inference (PML vs. PSE)
Example of the pseudo-maximum-likelihood (PML)
β(y) = arg maxβ
∑t
log f?(yt|β, y1:(t−1))
leading to
arg minθ||β(yo)− β(y1(θ), . . . ,yS(θ))||2
whenys(θ) ∼ f(y|θ) s = 1, . . . , S
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
simulation-based methods in Econometrics
Indirect inference (PML vs. PSE)
Example of the pseudo-score-estimator (PSE)
β(y) = arg minβ
{∑t
∂ log f?
∂β(yt|β, y1:(t−1))
}2
leading to
arg minθ||β(yo)− β(y1(θ), . . . ,yS(θ))||2
whenys(θ) ∼ f(y|θ) s = 1, . . . , S
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
simulation-based methods in Econometrics
ABC using indirect inference (1)
We present a novel approach for developing summary statisticsfor use in approximate Bayesian computation (ABC) algorithms byusing indirect inference(...) In the indirect inference approach toABC the parameters of an auxiliary model fitted to the data becomethe summary statistics. Although applicable to any ABC technique,we embed this approach within a sequential Monte Carlo algorithmthat is completely adaptive and requires very little tuning(...)
[Drovandi, Pettitt & Faddy, 2011]
c© Indirect inference provides summary statistics for ABC...
ABC Methods for Bayesian Model Choice
Approximate Bayesian computation
simulation-based methods in Econometrics
ABC using indirect inference (2)
...the above result shows that, in the limit as h→ 0, ABC willbe more accurate than an indirect inference method whose auxiliarystatistics are the same as the summary statistic that is used forABC(...) Initial analysis showed that which method is moreaccurate depends on the true value of θ.
[Fearnhead and Prangle, 2012]
c© Indirect inference provides estimates rather than global inference...
ABC Methods for Bayesian Model Choice
ABC for model choice
ABC for model choice
Approximate Bayesian computation
ABC for model choicePrincipleGibbs random fieldsGeneric ABC model choice
Model choice consistency
ABC Methods for Bayesian Model Choice
ABC for model choice
Principle
Bayesian model choice
Principle
Several modelsM1,M2, . . .
are considered simultaneously for dataset y and model index Mcentral to inference.Use of a prior π(M = m), plus a prior distribution on theparameter conditional on the value m of the model index, πm(θm)Goal is to derive the posterior distribution of M,
π(M = m|data)
a challenging computational target when models are complex.
ABC Methods for Bayesian Model Choice
ABC for model choice
Principle
Generic ABC for model choice
Algorithm 2 Likelihood-free model choice sampler (ABC-MC)
for t = 1 to T dorepeat
Generate m from the prior π(M = m)Generate θm from the prior πm(θm)Generate z from the model fm(z|θm)
until ρ{η(z), η(y)} < εSet m(t) = m and θ(t) = θm
end for
[Grelaud & al., 2009; Toni & al., 2009]
ABC Methods for Bayesian Model Choice
ABC for model choice
Principle
ABC estimates
Posterior probability π(M = m|y) approximated by the frequencyof acceptances from model m
1
T
T∑t=1
Im(t)=m .
ABC Methods for Bayesian Model Choice
ABC for model choice
Principle
ABC estimates
Posterior probability π(M = m|y) approximated by the frequencyof acceptances from model m
1
T
T∑t=1
Im(t)=m .
Extension to a weighted polychotomous logistic regressionestimate of π(M = m|y), with non-parametric kernel weights
[Cornuet et al., DIYABC, 2009]
ABC Methods for Bayesian Model Choice
ABC for model choice
Gibbs random fields
Potts model
Skip MRFs
Potts model
Distribution with an energy function of the form
θS(y) = θ∑l∼i
δyl=yi
where l∼i denotes a neighbourhood structure
In most realistic settings, summation
Zθ =∑x∈X
exp{θS(x)}
involves too many terms to be manageable and numericalapproximations cannot always be trusted
ABC Methods for Bayesian Model Choice
ABC for model choice
Gibbs random fields
Potts model
Skip MRFs
Potts model
Distribution with an energy function of the form
θS(y) = θ∑l∼i
δyl=yi
where l∼i denotes a neighbourhood structure
In most realistic settings, summation
Zθ =∑x∈X
exp{θS(x)}
involves too many terms to be manageable and numericalapproximations cannot always be trusted
ABC Methods for Bayesian Model Choice
ABC for model choice
Gibbs random fields
Neighbourhood relations
SetupChoice to be made between M neighbourhood relations
im∼ i′ (0 ≤ m ≤M − 1)
withSm(x) =
∑im∼i′
I{xi=xi′}
driven by the posterior probabilities of the models.
ABC Methods for Bayesian Model Choice
ABC for model choice
Gibbs random fields
Model index
Computational target:
P(M = m|x) ∝∫
Θm
fm(x|θm)πm(θm) dθm π(M = m)
If S(x) sufficient statistic for the joint parameters(M, θ0, . . . , θM−1),
P(M = m|x) = P(M = m|S(x)) .
ABC Methods for Bayesian Model Choice
ABC for model choice
Gibbs random fields
Model index
Computational target:
P(M = m|x) ∝∫
Θm
fm(x|θm)πm(θm) dθm π(M = m)
If S(x) sufficient statistic for the joint parameters(M, θ0, . . . , θM−1),
P(M = m|x) = P(M = m|S(x)) .
ABC Methods for Bayesian Model Choice
ABC for model choice
Gibbs random fields
Sufficient statistics in Gibbs random fields
Each model m has its own sufficient statistic Sm(·) andS(·) = (S0(·), . . . , SM−1(·)) is also (model-)sufficient.
Explanation: For Gibbs random fields,
x|M = m ∼ fm(x|θm) = f1m(x|S(x))f2
m(S(x)|θm)
=1
n(S(x))f2m(S(x)|θm)
wheren(S(x)) = ] {x ∈ X : S(x) = S(x)}
c© S(x) is sufficient for the joint parameters
ABC Methods for Bayesian Model Choice
ABC for model choice
Gibbs random fields
Sufficient statistics in Gibbs random fields
Each model m has its own sufficient statistic Sm(·) andS(·) = (S0(·), . . . , SM−1(·)) is also (model-)sufficient.Explanation: For Gibbs random fields,
x|M = m ∼ fm(x|θm) = f1m(x|S(x))f2
m(S(x)|θm)
=1
n(S(x))f2m(S(x)|θm)
wheren(S(x)) = ] {x ∈ X : S(x) = S(x)}
c© S(x) is sufficient for the joint parameters
ABC Methods for Bayesian Model Choice
ABC for model choice
Gibbs random fields
Toy example
iid Bernoulli model versus two-state first-order Markov chain, i.e.
f0(x|θ0) = exp
(θ0
n∑i=1
I{xi=1}
)/{1 + exp(θ0)}n ,
versus
f1(x|θ1) =1
2exp
(θ1
n∑i=2
I{xi=xi−1}
)/{1 + exp(θ1)}n−1 ,
with priors θ0 ∼ U(−5, 5) and θ1 ∼ U(0, 6) (inspired by “phasetransition” boundaries).
ABC Methods for Bayesian Model Choice
ABC for model choice
Gibbs random fields
Toy example (2)
−40 −20 0 10
−50
5
BF01
BF01
−40 −20 0 10−10
−50
510
BF01
BF01
(left) Comparison of the true BFm0/m1(x0) with BFm0/m1
(x0)(in logs) over 2, 000 simulations and 4.106 proposals from theprior. (right) Same when using tolerance ε corresponding to the1% quantile on the distances.
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
About sufficiency
If η1(x) sufficient statistic for model m = 1 and parameter θ1 andη2(x) sufficient statistic for model m = 2 and parameter θ2,(η1(x), η2(x)) is not always sufficient for (m, θm)
c© Potential loss of information at the testing level
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
About sufficiency
If η1(x) sufficient statistic for model m = 1 and parameter θ1 andη2(x) sufficient statistic for model m = 2 and parameter θ2,(η1(x), η2(x)) is not always sufficient for (m, θm)
c© Potential loss of information at the testing level
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Poisson/geometric example
Samplex = (x1, . . . , xn)
from either a Poisson P(λ) or from a geometric G(p)Sum
S =
n∑i=1
xi = η(x)
sufficient statistic for either model but not simultaneously
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Limiting behaviour of B12 (T →∞)
ABC approximation
B12(y) =
∑Tt=1 Imt=1 Iρ{η(zt),η(y)}≤ε∑Tt=1 Imt=2 Iρ{η(zt),η(y)}≤ε
,
where the (mt, zt)’s are simulated from the (joint) prior
As T go to infinity, limit
Bε12(y) =
∫Iρ{η(z),η(y)}≤επ1(θ1)f1(z|θ1) dz dθ1∫Iρ{η(z),η(y)}≤επ2(θ2)f2(z|θ2) dz dθ2
=
∫Iρ{η,η(y)}≤επ1(θ1)fη1 (η|θ1) dη dθ1∫Iρ{η,η(y)}≤επ2(θ2)fη2 (η|θ2) dη dθ2
,
where fη1 (η|θ1) and fη2 (η|θ2) distributions of η(z)
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Limiting behaviour of B12 (T →∞)
ABC approximation
B12(y) =
∑Tt=1 Imt=1 Iρ{η(zt),η(y)}≤ε∑Tt=1 Imt=2 Iρ{η(zt),η(y)}≤ε
,
where the (mt, zt)’s are simulated from the (joint) priorAs T go to infinity, limit
Bε12(y) =
∫Iρ{η(z),η(y)}≤επ1(θ1)f1(z|θ1) dz dθ1∫Iρ{η(z),η(y)}≤επ2(θ2)f2(z|θ2) dz dθ2
=
∫Iρ{η,η(y)}≤επ1(θ1)fη1 (η|θ1) dη dθ1∫Iρ{η,η(y)}≤επ2(θ2)fη2 (η|θ2) dη dθ2
,
where fη1 (η|θ1) and fη2 (η|θ2) distributions of η(z)
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Limiting behaviour of B12 (ε→ 0)
When ε goes to zero,
Bη12(y) =
∫π1(θ1)fη1 (η(y)|θ1) dθ1∫π2(θ2)fη2 (η(y)|θ2) dθ2
c© Bayes factor based on the sole observation of η(y)
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Limiting behaviour of B12 (ε→ 0)
When ε goes to zero,
Bη12(y) =
∫π1(θ1)fη1 (η(y)|θ1) dθ1∫π2(θ2)fη2 (η(y)|θ2) dθ2
c© Bayes factor based on the sole observation of η(y)
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Limiting behaviour of B12 (under sufficiency)
If η(y) sufficient statistic in both models,
fi(y|θi) = gi(y)fηi (η(y)|θi)
Thus
B12(y) =
∫Θ1π(θ1)g1(y)fη1 (η(y)|θ1) dθ1∫
Θ2π(θ2)g2(y)fη2 (η(y)|θ2) dθ2
=g1(y)
∫π1(θ1)fη1 (η(y)|θ1) dθ1
g2(y)∫π2(θ2)fη2 (η(y)|θ2) dθ2
=g1(y)
g2(y)Bη
12(y) .
[Didelot, Everitt, Johansen & Lawson, 2011]
c© No discrepancy only when cross-model sufficiency
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Limiting behaviour of B12 (under sufficiency)
If η(y) sufficient statistic in both models,
fi(y|θi) = gi(y)fηi (η(y)|θi)
Thus
B12(y) =
∫Θ1π(θ1)g1(y)fη1 (η(y)|θ1) dθ1∫
Θ2π(θ2)g2(y)fη2 (η(y)|θ2) dθ2
=g1(y)
∫π1(θ1)fη1 (η(y)|θ1) dθ1
g2(y)∫π2(θ2)fη2 (η(y)|θ2) dθ2
=g1(y)
g2(y)Bη
12(y) .
[Didelot, Everitt, Johansen & Lawson, 2011]
c© No discrepancy only when cross-model sufficiency
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Poisson/geometric example (back)
Samplex = (x1, . . . , xn)
from either a Poisson P(λ) or from a geometric G(p)
Discrepancy ratio
g1(x)
g2(x)=S!n−S/
∏i xi!
1/(
n+S−1S
)
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Poisson/geometric discrepancy
Range of B12(x) versus Bη12(x): The values produced have
nothing in common.
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Formal recovery
Creating an encompassing exponential family
f(x|θ1, θ2, α1, α2) ∝ exp{θT1 η1(x) + θT
2 η2(x) +α1t1(x) +α2t2(x)}
leads to a sufficient statistic (η1(x), η2(x), t1(x), t2(x))[Didelot, Everitt, Johansen & Lawson, 2011]
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Formal recovery
Creating an encompassing exponential family
f(x|θ1, θ2, α1, α2) ∝ exp{θT1 η1(x) + θT
2 η2(x) +α1t1(x) +α2t2(x)}
leads to a sufficient statistic (η1(x), η2(x), t1(x), t2(x))[Didelot, Everitt, Johansen & Lawson, 2011]
In the Poisson/geometric case, if∏i xi! is added to S, no
discrepancy
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Formal recovery
Creating an encompassing exponential family
f(x|θ1, θ2, α1, α2) ∝ exp{θT1 η1(x) + θT
2 η2(x) +α1t1(x) +α2t2(x)}
leads to a sufficient statistic (η1(x), η2(x), t1(x), t2(x))[Didelot, Everitt, Johansen & Lawson, 2011]
Only applies in genuine sufficiency settings...
c© Inability to evaluate information loss due to summarystatistics
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Meaning of the ABC-Bayes factor
The ABC approximation to the Bayes Factor is based solely on thesummary statistics....
In the Poisson/geometric case, if E[yi] = θ0 > 0,
limn→∞
Bη12(y) =
(θ0 + 1)2
θ0e−θ0
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Meaning of the ABC-Bayes factor
The ABC approximation to the Bayes Factor is based solely on thesummary statistics....In the Poisson/geometric case, if E[yi] = θ0 > 0,
limn→∞
Bη12(y) =
(θ0 + 1)2
θ0e−θ0
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
MA example
1 2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
1 2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
1 2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
1 2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Evolution [against ε] of ABC Bayes factor, in terms of frequencies ofvisits to models MA(1) (left) and MA(2) (right) when ε equal to10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sampleof 50 points from a MA(2) with θ1 = 0.6, θ2 = 0.2. True Bayes factorequal to 17.71.
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
MA example
1 2
0.0
0.2
0.4
0.6
1 2
0.0
0.2
0.4
0.6
1 2
0.0
0.2
0.4
0.6
1 2
0.0
0.2
0.4
0.6
0.8
Evolution [against ε] of ABC Bayes factor, in terms of frequencies ofvisits to models MA(1) (left) and MA(2) (right) when ε equal to10, 1, .1, .01% quantiles on insufficient autocovariance distances. Sampleof 50 points from a MA(1) model with θ1 = 0.6. True Bayes factor B21
equal to .004.
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
A population genetics evaluation
Gene free
Population genetics example with
I 3 populations
I 2 scenari
I 15 individuals
I 5 loci
I single mutation parameter
I 24 summary statistics
I 2 million ABC proposal
I importance [tree] sampling alternative
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
A population genetics evaluation
Gene free
Population genetics example with
I 3 populations
I 2 scenari
I 15 individuals
I 5 loci
I single mutation parameter
I 24 summary statistics
I 2 million ABC proposal
I importance [tree] sampling alternative
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Stability of importance sampling
●
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
●
0.0
0.2
0.4
0.6
0.8
1.0
●
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Comparison with ABC
Use of 24 summary statistics and DIY-ABC logistic correction
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
importance sampling
AB
C d
irect
and
logi
stic
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Comparison with ABC
Use of 15 summary statistics
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
● ●
●
●
●●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
−4 −2 0 2 4 6
−4
−2
02
46
importance sampling
AB
C d
irect
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
Comparison with ABC
Use of 15 summary statistics and DIY-ABC logistic correction
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
● ●
●
●
●●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
−4 −2 0 2 4 6
−4
−2
02
46
importance sampling
AB
C d
irect
and
logi
stic
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
The only safe cases???
Besides specific models like Gibbs random fields,
using distances over the data itself escapes the discrepancy...[Toni & Stumpf, 2010; Sousa & al., 2009]
...and so does the use of more informal model fitting measures[Ratmann & al., 2009]
ABC Methods for Bayesian Model Choice
ABC for model choice
Generic ABC model choice
The only safe cases???
Besides specific models like Gibbs random fields,
using distances over the data itself escapes the discrepancy...[Toni & Stumpf, 2010; Sousa & al., 2009]
...and so does the use of more informal model fitting measures[Ratmann & al., 2009]
ABC Methods for Bayesian Model Choice
Model choice consistency
ABC model choice consistency
Approximate Bayesian computation
ABC for model choice
Model choice consistencyFormalised frameworkConsistency resultsSummary statisticsConclusions
ABC Methods for Bayesian Model Choice
Model choice consistency
Formalised framework
The starting point
Central question to the validation of ABC for model choice:
When is a Bayes factor based on an insufficient statisticT (y) consistent?
Note: c© drawn on T (y) through BT12(y) necessarily differs from
c© drawn on y through B12(y)
ABC Methods for Bayesian Model Choice
Model choice consistency
Formalised framework
The starting point
Central question to the validation of ABC for model choice:
When is a Bayes factor based on an insufficient statisticT (y) consistent?
Note: c© drawn on T (y) through BT12(y) necessarily differs from
c© drawn on y through B12(y)
ABC Methods for Bayesian Model Choice
Model choice consistency
Formalised framework
A benchmark if toy example
Comparison suggested by referee of PNAS paper [thanks]:[X, Cornuet, Marin, & Pillai, Aug. 2011]
Model M1: y ∼ N (θ1, 1) opposedto model M2: y ∼ L(θ2, 1/
√2), Laplace distribution with mean θ2
and scale parameter 1/√
2 (variance one).
ABC Methods for Bayesian Model Choice
Model choice consistency
Formalised framework
A benchmark if toy example
Comparison suggested by referee of PNAS paper [thanks]:[X, Cornuet, Marin, & Pillai, Aug. 2011]
Model M1: y ∼ N (θ1, 1) opposedto model M2: y ∼ L(θ2, 1/
√2), Laplace distribution with mean θ2
and scale parameter 1/√
2 (variance one).Four possible statistics
1. sample mean y (sufficient for M1 if not M2);
2. sample median med(y) (insufficient);
3. sample variance var(y) (ancillary);
4. median absolute deviation mad(y) = med(|y −med(y)|);
ABC Methods for Bayesian Model Choice
Model choice consistency
Formalised framework
A benchmark if toy example
Comparison suggested by referee of PNAS paper [thanks]:[X, Cornuet, Marin, & Pillai, Aug. 2011]
Model M1: y ∼ N (θ1, 1) opposedto model M2: y ∼ L(θ2, 1/
√2), Laplace distribution with mean θ2
and scale parameter 1/√
2 (variance one).
●
●
●
●
●
●
●●
●
●
●
Gauss Laplace
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
n=100
ABC Methods for Bayesian Model Choice
Model choice consistency
Formalised framework
A benchmark if toy example
Comparison suggested by referee of PNAS paper [thanks]:[X, Cornuet, Marin, & Pillai, Aug. 2011]
Model M1: y ∼ N (θ1, 1) opposedto model M2: y ∼ L(θ2, 1/
√2), Laplace distribution with mean θ2
and scale parameter 1/√
2 (variance one).
●
●
●
●
●
●
●●
●
●
●
Gauss Laplace
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
n=100
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
Gauss Laplace
0.0
0.2
0.4
0.6
0.8
1.0
n=100
ABC Methods for Bayesian Model Choice
Model choice consistency
Formalised framework
Framework
Starting from sample
y = (y1, . . . , yn)
the observed sample, not necessarily iid with true distribution
y ∼ Pn
Summary statistics
T (y) = T n = (T1(y), T2(y), · · · , Td(y)) ∈ Rd
with true distribution T n ∼ Gn.
ABC Methods for Bayesian Model Choice
Model choice consistency
Formalised framework
Framework
c© Comparison of
– under M1, y ∼ F1,n(·|θ1) where θ1 ∈ Θ1 ⊂ Rp1
– under M2, y ∼ F2,n(·|θ2) where θ2 ∈ Θ2 ⊂ Rp2
turned into
– under M1, T (y) ∼ G1,n(·|θ1), and θ1|T (y) ∼ π1(·|T n)
– under M2, T (y) ∼ G2,n(·|θ2), and θ2|T (y) ∼ π2(·|T n)
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Assumptions
A collection of asymptotic “standard” assumptions:
[A1] is a standard central limit theorem under the true model[A2] controls the large deviations of the estimator T n from the estimandµ(θ)[A3] is the standard prior mass condition found in Bayesian asymptotics(di effective dimension of the parameter)[A4] restricts the behaviour of the model density against the true density
[Think CLT!]
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Assumptions
A collection of asymptotic “standard” assumptions:
[A1] There exist a sequence {vn} converging to +∞,a distribution Q,a symmetric, d× d positive definite matrix V0and a vector µ0 ∈ Rd such that
vnV−1/20 (T n − µ0)
n→∞ Q, under Gn
[Think CLT!]
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Assumptions
A collection of asymptotic “standard” assumptions:
[Think CLT!][A2] For i = 1, 2, there exist sets Fn,i ⊂ Θi,
functions µi(θi) and constants εi, τi, αi > 0 such that for all τ > 0,
supθi∈Fn,i
Gi,n
[|T n − µi(θi)| > τ |µi(θi)− µ0| ∧ εi |θi
](|τµi(θi)− µ0| ∧ εi)−αi
. v−αin
withπi(Fcn,i) = o(v−τin ).
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Assumptions
A collection of asymptotic “standard” assumptions:
[Think CLT!]
[A3] For u > 0
Sn,i(u) ={θi ∈ Fn,i; |µi(θi)− µ0| ≤ u v−1n
}if inf{|µi(θi)− µ0|; θi ∈ Θi} = 0, there exist constants di < τi ∧ αi − 1such that
πi(Sn,i(u)) ∼ udiv−din , ∀u . vn
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Assumptions
A collection of asymptotic “standard” assumptions:
[Think CLT!]
[A4] If inf{|µi(θi)− µ0|; θi ∈ Θi} = 0, for any ε > 0, there existU, δ > 0 and (En)n such that, if θi ∈ Sn,i(U)
En ⊂ {t; gi(t|θi) < δgn(t)} and Gn (Ecn) < ε.
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Assumptions
A collection of asymptotic “standard” assumptions:
[Think CLT!]
Again[A1] is a standard central limit theorem under the true model[A2] controls the large deviations of the estimator T n from the estimandµ(θ)[A3] is the standard prior mass condition found in Bayesian asymptotics(di effective dimension of the parameter)
[A4] restricts the behaviour of the model density against the true density
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Effective dimension
Understanding di in [A3]: defined only whenµ0 ∈ {µi(θi), θi ∈ Θi},
πi(θi : |µi(θi)− µ0| < n−1/2) = O(n−di/2)
is the effective dimension of the model Θi around µ0
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Effective dimension
Understanding di in [A3]:when inf{|µi(θi)− µ0|; θi ∈ Θi} = 0,
mi(Tn)
gn(T n)∼ v−din ,
thuslog(mi(T
n)/gn(T n)) ∼ −di log vn
and v−din penalization factor resulting from integrating θi out asthe effective number of parameters appears in the DIC
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Effective dimension
Understanding di in [A3]:In regular models, di dimension of T (Θi), leading to BIC. Innon-regular models, di can be smaller.
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Asymptotic marginals
Asymptotically, under [A1]–[A4]
mi(t) =
∫Θi
gi(t|θi)πi(θi) dθi
is such that(i) if inf{|µi(θi)− µ0|; θi ∈ Θi} = 0,
Clvd−din ≤ mi(T
n) ≤ Cuvd−din
and(ii) if inf{|µi(θi)− µ0|; θi ∈ Θi} > 0
mi(Tn) = oPn [vd−τin + vd−αin ].
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Within-model consistency
Under same assumptions,if inf{|µi(θi)− µ0|; θi ∈ Θi} = 0,
the posterior distribution of µi(θi) given T n is consistent at rate1/vn provided αi ∧ τi > di.
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Between-model consistency
Consequence of above is that asymptotic behaviour of the Bayesfactor is driven by the asymptotic mean value of T n under bothmodels. And only by this mean value!
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Between-model consistency
Consequence of above is that asymptotic behaviour of the Bayesfactor is driven by the asymptotic mean value of T n under bothmodels. And only by this mean value!
Indeed, if
inf{|µ0 − µ2(θ2)|; θ2 ∈ Θ2} = inf{|µ0 − µ1(θ1)|; θ1 ∈ Θ1} = 0
then
Clv−(d1−d2)n ≤ m1(T n)
/m2(T n) ≤ Cuv−(d1−d2)
n ,
where Cl, Cu = OPn(1), irrespective of the true model.c© Only depends on the difference d1 − d2: no consistency
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Between-model consistency
Consequence of above is that asymptotic behaviour of the Bayesfactor is driven by the asymptotic mean value of T n under bothmodels. And only by this mean value!
Else, if
inf{|µ0 − µ2(θ2)|; θ2 ∈ Θ2} > inf{|µ0 − µ1(θ1)|; θ1 ∈ Θ1} = 0
thenm1(T n)
m2(T n)≥ Cu min
(v−(d1−α2)n , v−(d1−τ2)
n
)
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Consistency theorem
If
inf{|µ0 − µ2(θ2)|; θ2 ∈ Θ2} = inf{|µ0 − µ1(θ1)|; θ1 ∈ Θ1} = 0,
Bayes factorBT
12 = O(v−(d1−d2)n )
irrespective of the true model. It is inconsistent since it alwayspicks the model with the smallest dimension
ABC Methods for Bayesian Model Choice
Model choice consistency
Consistency results
Consistency theorem
If Pn belongs to one of the two models and if µ0 cannot beattained by the other one :
0 = min (inf{|µ0 − µi(θi)|; θi ∈ Θi}, i = 1, 2)
< max (inf{|µ0 − µi(θi)|; θi ∈ Θi}, i = 1, 2) ,
then the Bayes factor BT12 is consistent
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Consequences on summary statistics
Bayes factor driven by the means µi(θi) and the relative position ofµ0 wrt both sets {µi(θi); θi ∈ Θi}, i = 1, 2.
For ABC, this implies the most likely statistics T n are ancillarystatistics with different mean values under both models
Else, if T n asymptotically depends on some of the parameters ofthe models, it is quite likely that there exists θi ∈ Θi such thatµi(θi) = µ0 even though model M1 is misspecified
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Consequences on summary statistics
Bayes factor driven by the means µi(θi) and the relative position ofµ0 wrt both sets {µi(θi); θi ∈ Θi}, i = 1, 2.
For ABC, this implies the most likely statistics T n are ancillarystatistics with different mean values under both models
Else, if T n asymptotically depends on some of the parameters ofthe models, it is quite likely that there exists θi ∈ Θi such thatµi(θi) = µ0 even though model M1 is misspecified
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Toy example: Laplace versus Gauss [1]
If
T n = n−1n∑i=1
X4i , µ1(θ) = 3 + θ4 + 6θ2, µ2(θ) = 6 + · · ·
and the true distribution is Laplace with mean θ0 = 1, under theGaussian model the value θ∗ = 2
√3− 3 leads to µ0 = µ(θ∗)
[here d1 = d2 = d = 1]
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Toy example: Laplace versus Gauss [1]
If
T n = n−1n∑i=1
X4i , µ1(θ) = 3 + θ4 + 6θ2, µ2(θ) = 6 + · · ·
and the true distribution is Laplace with mean θ0 = 1, under theGaussian model the value θ∗ = 2
√3− 3 leads to µ0 = µ(θ∗)
[here d1 = d2 = d = 1]c© A Bayes factor associated with such a statistic is inconsistent
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Toy example: Laplace versus Gauss [1]
If
T n = n−1n∑i=1
X4i , µ1(θ) = 3 + θ4 + 6θ2, µ2(θ) = 6 + · · ·
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Toy example: Laplace versus Gauss [1]
If
T n = n−1n∑i=1
X4i , µ1(θ) = 3 + θ4 + 6θ2, µ2(θ) = 6 + · · ·
●
●●
●
●
●●
M1 M2
0.30
0.35
0.40
0.45
n=1000
Fourth moment
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Toy example: Laplace versus Gauss [0]
WhenT (y) =
{y(4)n , y(6)
n
}and the true distribution is Laplace with mean θ0 = 0, thenµ0 = 6, µ1(θ∗1) = 6 with θ∗1 = 2
√3− 3
[d1 = 1 and d2 = 1/2]thus
B12 ∼ n−1/4 → 0 : consistent
Under the Gaussian model µ0 = 3, µ2(θ2) ≥ 6 > 3 ∀θ2
B12 → +∞ : consistent
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Toy example: Laplace versus Gauss [0]
WhenT (y) =
{y(4)n , y(6)
n
}and the true distribution is Laplace with mean θ0 = 0, thenµ0 = 6, µ1(θ∗1) = 6 with θ∗1 = 2
√3− 3
[d1 = 1 and d2 = 1/2]thus
B12 ∼ n−1/4 → 0 : consistent
Under the Gaussian model µ0 = 3, µ2(θ2) ≥ 6 > 3 ∀θ2
B12 → +∞ : consistent
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Toy example: Laplace versus Gauss [0]
WhenT (y) =
{y(4)n , y(6)
n
}
0.0 0.2 0.4 0.6 0.8 1.0
01
23
45
posterior probabilities
Den
sity
Fourth and sixth moments
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Checking for adequate statistics
After running ABC, i.e. creating reference tables of (θi, xi) fromboth joints, straightforward derivation of ABC estimates θ1 and θ2.
Evaluation of E1θ1
[T (X)] and E2θ2
[T (X)] allows for detection of
different means under both models via Monte Carlo simulations
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Toy example: Laplace versus Gauss
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●
Gauss Laplace Gauss Laplace
010
2030
40
Normalised χ2 without and with mad
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
A population genetic illustration
Two populations (1 and 2) having diverged at a fixed known timein the past and third population (3) which diverged from one ofthose two populations (models 1 and 2, respectively).
Observation of 50 diploid individuals/population genotyped at 5,50 or 100 independent microsatellite loci.
Model 2
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
A population genetic illustration
Two populations (1 and 2) having diverged at a fixed known timein the past and third population (3) which diverged from one ofthose two populations (models 1 and 2, respectively).
Observation of 50 diploid individuals/population genotyped at 5,50 or 100 independent microsatellite loci.
Stepwise mutation model: the number of repeats of the mutatedgene increases or decreases by one. Mutation rate µ common to allloci set to 0.005 (single parameter) with uniform prior distribution
µ ∼ U [0.0001, 0.01]
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
A population genetic illustration
Summary statistics associated to the (δµ)2 distance
xl,i,j repeated number of allele in locus l = 1, . . . , L for individuali = 1, . . . , 100 within the population j = 1, 2, 3. Then
(δµ)2j1,j2 =
1
L
L∑l=1
(1
100
100∑i1=1
xl,i1,j1 −1
100
100∑i2=1
xl,i2,j2
)2
.
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
A population genetic illustration
For two copies of locus l with allele sizes xl,i,j1 and xl,i′,j2 , mostrecent common ancestor at coalescence time τj1,j2 , gene genealogydistance of 2τj1,j2 , hence number of mutations Poisson withparameter 2µτj1,j2 . Therefore,
E{(xl,i,j1 − xl,i′,j2
)2 |τj1,j2} = 2µτj1,j2
andModel 1 Model 2
E{
(δµ)21,2
}2µ1t
′ 2µ2t′
E{
(δµ)21,3
}2µ1t 2µ2t
′
E{
(δµ)22,3
}2µ1t
′ 2µ2t
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
A population genetic illustration
Thus,
I Bayes factor based only on distance (δµ)21,2 not convergent: if
µ1 = µ2, same expectations.
I Bayes factor based only on distance (δµ)21,3 or (δµ)2
2,3 notconvergent: if µ1 = 2µ2 or 2µ1 = µ2 same expectation.
I if two of the three distances are used, Bayes factor converges:there is no (µ1, µ2) for which all expectations are equal.
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
A population genetic illustration
●
● ●
5 50 100
0.0
0.4
0.8
DM2(12)
●
●
●
●
●
●
●
●
●
●●
● ●
●
5 50 100
0.0
0.4
0.8
DM2(13)
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●●●●●●●
●
●
●
●
●
●
5 50 100
0.0
0.4
0.8
DM2(13) & DM2(23)
Posterior probabilities that the data is from model 1 for 5, 50and 100 loci
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Embedded models
When M1 submodel of M2, and if the true distribution belongs tothe smaller model M1, Bayes factor is of order
v−(d1−d2)n
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Embedded models
When M1 submodel of M2, and if the true distribution belongs tothe smaller model M1, Bayes factor is of order
v−(d1−d2)n
If summary statistic only informative on a parameter that is thesame under both models, i.e if d1 = d2, then
c© the Bayes factor is not consistent
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Embedded models
When M1 submodel of M2, and if the true distribution belongs tothe smaller model M1, Bayes factor is of order
v−(d1−d2)n
Else, d1 < d2 and Bayes factor is going to ∞ under M1. If truedistribution not in M1, then
c© Bayes factor is consistent only if µ1 6= µ2 = µ0
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Another toy example: Quantile distribution
Q(p;A,B, g, k) = A+B
[1 +
1− exp{−gz(p)}1 + exp{−gz(p)}
] [1 + z(p)2
]kz(p)
A,B, g and k, location, scale, skewness and kurtosis parametersEmbedded models:
I M1 : g = 0 and k ∼ U [−1/2, 5]
I M2 : g ∼ U [0, 4] and k ∼ U [−1/2, 5].
ABC Methods for Bayesian Model Choice
Model choice consistency
Summary statistics
Consistency [or not]
●
M1 M2
0.3
0.4
0.5
0.6
0.7
●●
●
●
M1 M2
0.3
0.4
0.5
0.6
0.7
M1 M2
0.3
0.4
0.5
0.6
0.7
●
●
●
●
●
●●
●
M1 M2
0.0
0.2
0.4
0.6
0.8
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
M1 M2
0.0
0.2
0.4
0.6
0.8
1.0
●
●
●
●
●
●
●
●
M1 M2
0.0
0.2
0.4
0.6
0.8
1.0
●
●
●
●
●
●
●
M1 M2
0.0
0.2
0.4
0.6
0.8
●
●●●
●
●
●
●
●●●
●
●
●
●
●
●
M1 M2
0.0
0.2
0.4
0.6
0.8
1.0
●
●
●●
●
●●
M1 M2
0.0
0.2
0.4
0.6
0.8
1.0
ABC Methods for Bayesian Model Choice
Model choice consistency
Conclusions
Conclusions
• Model selection feasible with ABC• Choice of summary statistics is paramount• At best, ABC → π(. | T (y)) which concentrates around µ0
• For estimation : {θ;µ(θ) = µ0} = θ0
• For testing {µ1(θ1), θ1 ∈ Θ1} ∩ {µ2(θ2), θ2 ∈ Θ2} = ∅
ABC Methods for Bayesian Model Choice
Model choice consistency
Conclusions
Conclusions
• Model selection feasible with ABC• Choice of summary statistics is paramount• At best, ABC → π(. | T (y)) which concentrates around µ0
• For estimation : {θ;µ(θ) = µ0} = θ0
• For testing {µ1(θ1), θ1 ∈ Θ1} ∩ {µ2(θ2), θ2 ∈ Θ2} = ∅