abc: how bayesian can it be?
DESCRIPTION
tutorial talk in ISBA 2012, Kyoto, June 25, 2012TRANSCRIPT
![Page 1: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/1.jpg)
Approximate Bayesian Computation:how Bayesian can it be?
Christian P. RobertISBA LECTURES ON BAYESIAN FOUNDATIONS
ISBA 2012, Kyoto, Japan
Universite Paris-Dauphine, IuF, & CRESThttp://www.ceremade.dauphine.fr/~xian
June 24, 2012
![Page 2: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/2.jpg)
This talk is dedicated to the memory of our dear friendand fellow Bayesian, George Casella, 1951–2012
![Page 3: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/3.jpg)
Outline
Introduction
Approximate Bayesian computation
ABC as an inference machine
![Page 4: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/4.jpg)
Approximate Bayesian computation
IntroductionMonte Carlo basicssimulation-based methods in Econometrics
Approximate Bayesian computation
ABC as an inference machine
![Page 5: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/5.jpg)
General issue
Given a density π known up to a normalizing constant, e.g. aposterior density, and an integrable function h, how can one use
Ih =
∫h(x)π(x)µ(dx) =
∫h(x)π(x)µ(dx)∫π(x)µ(dx)
when∫
h(x)π(x)µ(dx) is intractable?
![Page 6: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/6.jpg)
Monte Carlo basics
Generate an iid sample x1, . . . , xN from π and estimate Ih by
ImcN (h) = N−1
N∑i=1
h(xi ).
since [LLN] IMCN (h)
as−→ Ih
Furthermore, if Ih2 =∫
h2(x)π(x)µ(dx) <∞,
[CLT]√
N(IMCN (h)− Ih
)L N
(0, I
{[h − Ih]2
}).
Caveat
Often impossible or inefficient to simulate directly from π
![Page 7: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/7.jpg)
Monte Carlo basics
Generate an iid sample x1, . . . , xN from π and estimate Ih by
ImcN (h) = N−1
N∑i=1
h(xi ).
since [LLN] IMCN (h)
as−→ Ih
Furthermore, if Ih2 =∫
h2(x)π(x)µ(dx) <∞,
[CLT]√
N(IMCN (h)− Ih
)L N
(0, I
{[h − Ih]2
}).
Caveat
Often impossible or inefficient to simulate directly from π
![Page 8: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/8.jpg)
Importance Sampling
For Q proposal distribution with density q
Ih =
∫h(x){π/q}(x)q(x)µ(dx).
Principle
Generate an iid sample x1, . . . , xN ∼ Q and estimate Ih by
IISQ,N(h) = N−1N∑i=1
h(xi ){π/q}(xi ).
![Page 9: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/9.jpg)
Importance Sampling
For Q proposal distribution with density q
Ih =
∫h(x){π/q}(x)q(x)µ(dx).
Principle
Generate an iid sample x1, . . . , xN ∼ Q and estimate Ih by
IISQ,N(h) = N−1N∑i=1
h(xi ){π/q}(xi ).
![Page 10: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/10.jpg)
Importance Sampling (convergence)
Then[LLN] IISQ,N(h)
as−→ Ih and if Q((hπ/q)2) <∞,
[CLT]√
N(IISQ,N(h)− Ih)L N
(0,Q{(hπ/q − Ih)2}
).
Caveat
If normalizing constant unknown, impossible to use IISQ,N(h)
Generic problem in Bayesian Statistics: π(θ|x) ∝ f (x |θ)π(θ).
![Page 11: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/11.jpg)
Importance Sampling (convergence)
Then[LLN] IISQ,N(h)
as−→ Ih and if Q((hπ/q)2) <∞,
[CLT]√
N(IISQ,N(h)− Ih)L N
(0,Q{(hπ/q − Ih)2}
).
Caveat
If normalizing constant unknown, impossible to use IISQ,N(h)
Generic problem in Bayesian Statistics: π(θ|x) ∝ f (x |θ)π(θ).
![Page 12: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/12.jpg)
Self-normalised importance Sampling
Self normalized version
ISNISQ,N (h) =
(N∑i=1
{π/q}(xi )
)−1 N∑i=1
h(xi ){π/q}(xi ).
[LLN] ISNISQ,N (h)
as−→ Ihand if I((1+h2)(π/q)) <∞,
[CLT]√
N(ISNISQ,N (h)− Ih)
L N
(0, π {(π/q)(h − Ih}2)
).
Caveat
If π cannot be computed, impossible to use ISNISQ,N (h)
![Page 13: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/13.jpg)
Self-normalised importance Sampling
Self normalized version
ISNISQ,N (h) =
(N∑i=1
{π/q}(xi )
)−1 N∑i=1
h(xi ){π/q}(xi ).
[LLN] ISNISQ,N (h)
as−→ Ihand if I((1+h2)(π/q)) <∞,
[CLT]√
N(ISNISQ,N (h)− Ih)
L N
(0, π {(π/q)(h − Ih}2)
).
Caveat
If π cannot be computed, impossible to use ISNISQ,N (h)
![Page 14: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/14.jpg)
Self-normalised importance Sampling
Self normalized version
ISNISQ,N (h) =
(N∑i=1
{π/q}(xi )
)−1 N∑i=1
h(xi ){π/q}(xi ).
[LLN] ISNISQ,N (h)
as−→ Ihand if I((1+h2)(π/q)) <∞,
[CLT]√
N(ISNISQ,N (h)− Ih)
L N
(0, π {(π/q)(h − Ih}2)
).
Caveat
If π cannot be computed, impossible to use ISNISQ,N (h)
![Page 15: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/15.jpg)
Perspectives
What is the fundamental issue?I a mere computational issue (optimism: can be solved /
pessimism: too costly in the short term)
I more of a inferencial issue (optimism: gathering legit fromclassical B approach / pessimism: lacking the coherence ofclassical B approach)
I calling for a new methodology (optimism: equivalent toclassical B approach / pessimism: not always convergent)
![Page 16: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/16.jpg)
Perspectives
What is the fundamental issue?I a mere computational issue (optimism: can be solved /
pessimism: too costly in the short term)
I more of a inferencial issue (optimism: gathering legit fromclassical B approach / pessimism: lacking the coherence ofclassical B approach)
I calling for a new methodology (optimism: equivalent toclassical B approach / pessimism: not always convergent)
![Page 17: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/17.jpg)
Perspectives
What is the fundamental issue?I a mere computational issue (optimism: can be solved /
pessimism: too costly in the short term)
I more of a inferencial issue (optimism: gathering legit fromclassical B approach / pessimism: lacking the coherence ofclassical B approach)
I calling for a new methodology (optimism: equivalent toclassical B approach / pessimism: not always convergent)
![Page 18: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/18.jpg)
Econom’ections
Model choice
Similar exploration of simulation-based and approximationtechniques in Econometrics
I Simulated method of moments
I Method of simulated moments
I Simulated pseudo-maximum-likelihood
I Indirect inference
[Gourieroux & Monfort, 1996]
![Page 19: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/19.jpg)
Simulated method of moments
Given observations yo1:n from a model
yt = r(y1:(t−1), εt , θ) , εt ∼ g(·)
1. simulate ε?1:n, derive
y?t (θ) = r(y1:(t−1), ε?t , θ)
2. and estimate θ by
arg minθ
n∑t=1
(yot − y?t (θ))2
![Page 20: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/20.jpg)
Simulated method of moments
Given observations yo1:n from a model
yt = r(y1:(t−1), εt , θ) , εt ∼ g(·)
1. simulate ε?1:n, derive
y?t (θ) = r(y1:(t−1), ε?t , θ)
2. and estimate θ by
arg minθ
{n∑
t=1
yot −
n∑t=1
y?t (θ)
}2
![Page 21: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/21.jpg)
Method of simulated moments
Given a statistic vector K (y) with
Eθ[K (Yt)|y1:(t−1)] = k(y1:(t−1); θ)
find an unbiased estimator of k(y1:(t−1); θ),
k(εt , y1:(t−1); θ)
Estimate θ by
arg minθ
∣∣∣∣∣∣∣∣∣∣
n∑t=1
[K (yt)−
S∑s=1
k(εst , y1:(t−1); θ)/S
]∣∣∣∣∣∣∣∣∣∣
[Pakes & Pollard, 1989]
![Page 22: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/22.jpg)
Indirect inference
Minimise [in θ] a distance between estimators β based on apseudo-model for genuine observations and for observationssimulated under the true model and the parameter θ.
[Gourieroux, Monfort, & Renault, 1993;Smith, 1993; Gallant & Tauchen, 1996]
![Page 23: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/23.jpg)
Indirect inference (PML vs. PSE)
Example of the pseudo-maximum-likelihood (PML)
β(y) = arg maxβ
∑t
log f ?(yt |β, y1:(t−1))
leading to
arg minθ||β(yo)− β(y1(θ), . . . , yS(θ))||2
whenys(θ) ∼ f (y|θ) s = 1, . . . ,S
![Page 24: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/24.jpg)
Indirect inference (PML vs. PSE)
Example of the pseudo-score-estimator (PSE)
β(y) = arg minβ
{∑t
∂ log f ?
∂β(yt |β, y1:(t−1))
}2
leading to
arg minθ||β(yo)− β(y1(θ), . . . , yS(θ))||2
whenys(θ) ∼ f (y|θ) s = 1, . . . ,S
![Page 25: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/25.jpg)
Consistent indirect inference
...in order to get a unique solution the dimension ofthe auxiliary parameter β must be larger than or equal tothe dimension of the initial parameter θ. If the problem isjust identified the different methods become easier...
Consistency depending on the criterion and on the asymptoticidentifiability of θ
[Gourieroux & Monfort, 1996, p. 66]
![Page 26: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/26.jpg)
Consistent indirect inference
...in order to get a unique solution the dimension ofthe auxiliary parameter β must be larger than or equal tothe dimension of the initial parameter θ. If the problem isjust identified the different methods become easier...
Consistency depending on the criterion and on the asymptoticidentifiability of θ
[Gourieroux & Monfort, 1996, p. 66]
![Page 27: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/27.jpg)
Choice of pseudo-model
Arbitrariness of pseudo-modelPick model such that
1. β(θ) not flat (i.e. sensitive to changes in θ)
2. β(θ) not dispersed (i.e. robust agains changes in ys(θ))
[Frigessi & Heggland, 2004]
![Page 28: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/28.jpg)
Empirical likelihood
Another approximation method (not yet related with simulation)
Definition
For dataset y = (y1, . . . , yn), and parameter of interest θ, pickconstraints
E[h(Y , θ)] = 0
uniquely identifying θ and define the empirical likelihood as
Lel(θ|y) = maxp
n∏i=1
pi
for p in the set {p ∈ [0; 1]n,∑
pi = 1,∑
i pih(yi ,θ) = 0}.
[Owen, 1988]
![Page 29: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/29.jpg)
Empirical likelihood
Another approximation method (not yet related with simulation)
Example
When θ = Ef [Y ], empirical likelihood is the maximum of
p1 · · · pn
under constraintp1y1 + . . .+ pnyn = θ
![Page 30: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/30.jpg)
ABCel
Another approximation method (now related with simulation!)Importance sampling implementation
Algorithm 1: Raw ABCel sampler
Given observation yfor i = 1 to M do
Generate θi from the prior distribution π(·)Set the weight ωi = Lel(θi |y)
end forProceed with pairs (θi , ωi ) as in regular importance sampling
[Mengersen, Pudlo & Robert, 2012]
![Page 31: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/31.jpg)
A?B?C?
I A stands for approximate[wrong likelihood]
I B stands for Bayesian
I C stands for computation[producing a parametersample]
![Page 32: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/32.jpg)
A?B?C?
I A stands for approximate[wrong likelihood]
I B stands for Bayesian
I C stands for computation[producing a parametersample]
![Page 33: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/33.jpg)
A?B?C?
I A stands for approximate[wrong likelihood]
I B stands for Bayesian
I C stands for computation[producing a parametersample]
![Page 34: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/34.jpg)
How much Bayesian?
I asymptotically so (meaningfull?)
I approximation error unknown (w/o simulation)
I pragmatic Bayes (there is no other solution!)
![Page 35: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/35.jpg)
Approximate Bayesian computation
Introduction
Approximate Bayesian computationGenetics of ABCABC basicsAdvances and interpretationsAlphabet (compu-)soup
ABC as an inference machine
![Page 36: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/36.jpg)
Genetic background of ABC
ABC is a recent computational technique that only requires beingable to sample from the likelihood f (·|θ)
This technique stemmed from population genetics models, about15 years ago, and population geneticists still contributesignificantly to methodological developments of ABC.
[Griffith & al., 1997; Tavare & al., 1999]
![Page 37: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/37.jpg)
Demo-genetic inference
Each model is characterized by a set of parameters θ that coverhistorical (time divergence, admixture time ...), demographics(population sizes, admixture rates, migration rates, ...) and genetic(mutation rate, ...) factors
The goal is to estimate these parameters from a dataset ofpolymorphism (DNA sample) y observed at the present time
Problem:
most of the time, we can not calculate the likelihood of thepolymorphism data f (y|θ).
![Page 38: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/38.jpg)
Demo-genetic inference
Each model is characterized by a set of parameters θ that coverhistorical (time divergence, admixture time ...), demographics(population sizes, admixture rates, migration rates, ...) and genetic(mutation rate, ...) factors
The goal is to estimate these parameters from a dataset ofpolymorphism (DNA sample) y observed at the present time
Problem:
most of the time, we can not calculate the likelihood of thepolymorphism data f (y|θ).
![Page 39: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/39.jpg)
A genuine example of application
94
!""#$%&'()*+,(-*.&(/+0$'"1)()&$/+2!,03 !1/+*%*'"4*+56(""4&7()&$/.+.1#+4*.+8-9':*.+
Pygmies populations: do they have a common origin? Was there alot of exchanges between pygmies and non-pygmies populations?
![Page 40: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/40.jpg)
Alternative scenarios
96
!""#$%&'()*+,(-*.&(/+0$'"1)()&$/+2!,03 !1/+*%*'"4*+56(""4&7()&$/.+.1#+4*.+8-9':*.+
Différents scénarios possibles, choix de scenario par ABC
Verdu et al. 2009
![Page 41: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/41.jpg)
Untractable likelihood
Missing (too missing!) data structure:
f (y|θ) =
∫G
f (y|G ,θ)f (G |θ)dG
cannot be computed in a manageable way...
The genealogies are considered as nuisance parameters
This modelling clearly differs from the phylogenetic perspective
where the tree is the parameter of interest.
![Page 42: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/42.jpg)
Untractable likelihood
Missing (too missing!) data structure:
f (y|θ) =
∫G
f (y|G ,θ)f (G |θ)dG
cannot be computed in a manageable way...
The genealogies are considered as nuisance parameters
This modelling clearly differs from the phylogenetic perspective
where the tree is the parameter of interest.
![Page 43: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/43.jpg)
Untractable likelihood
So, what can we do when thelikelihood function f (y|θ) iswell-defined but impossible / toocostly to compute...?
I MCMC cannot be implemented!
I shall we give up Bayesianinference altogether?!
I or settle for an almost Bayesianinference...?
![Page 44: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/44.jpg)
Untractable likelihood
So, what can we do when thelikelihood function f (y|θ) iswell-defined but impossible / toocostly to compute...?
I MCMC cannot be implemented!
I shall we give up Bayesianinference altogether?!
I or settle for an almost Bayesianinference...?
![Page 45: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/45.jpg)
ABC methodology
Bayesian setting: target is π(θ)f (x |θ)When likelihood f (x |θ) not in closed form, likelihood-free rejectiontechnique:
Foundation
For an observation y ∼ f (y|θ), under the prior π(θ), if one keepsjointly simulating
θ′ ∼ π(θ) , z ∼ f (z|θ′) ,
until the auxiliary variable z is equal to the observed value, z = y,then the selected
θ′ ∼ π(θ|y)
[Rubin, 1984; Diggle & Gratton, 2984; Tavare et al., 1997]
![Page 46: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/46.jpg)
ABC methodology
Bayesian setting: target is π(θ)f (x |θ)When likelihood f (x |θ) not in closed form, likelihood-free rejectiontechnique:
Foundation
For an observation y ∼ f (y|θ), under the prior π(θ), if one keepsjointly simulating
θ′ ∼ π(θ) , z ∼ f (z|θ′) ,
until the auxiliary variable z is equal to the observed value, z = y,then the selected
θ′ ∼ π(θ|y)
[Rubin, 1984; Diggle & Gratton, 2984; Tavare et al., 1997]
![Page 47: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/47.jpg)
ABC methodology
Bayesian setting: target is π(θ)f (x |θ)When likelihood f (x |θ) not in closed form, likelihood-free rejectiontechnique:
Foundation
For an observation y ∼ f (y|θ), under the prior π(θ), if one keepsjointly simulating
θ′ ∼ π(θ) , z ∼ f (z|θ′) ,
until the auxiliary variable z is equal to the observed value, z = y,then the selected
θ′ ∼ π(θ|y)
[Rubin, 1984; Diggle & Gratton, 2984; Tavare et al., 1997]
![Page 48: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/48.jpg)
Why does it work?!
The proof is trivial:
f (θi ) ∝∑z∈D
π(θi )f (z|θi )Iy(z)
∝ π(θi )f (y|θi )= π(θi |y) .
[Accept–Reject 101]
![Page 49: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/49.jpg)
A as A...pproximative
When y is a continuous random variable, strict equality z = y isreplaced with a tolerance zone
%(y, z) ≤ ε
where % is a distanceOutput distributed from
π(θ) Pθ{%(y, z) < ε} def∝ π(θ|%(y, z) < ε)
[Pritchard et al., 1999]
![Page 50: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/50.jpg)
A as A...pproximative
When y is a continuous random variable, strict equality z = y isreplaced with a tolerance zone
%(y, z) ≤ ε
where % is a distanceOutput distributed from
π(θ) Pθ{%(y, z) < ε} def∝ π(θ|%(y, z) < ε)
[Pritchard et al., 1999]
![Page 51: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/51.jpg)
ABC algorithm
In most implementations, further degree of A...pproximation:
Algorithm 1 Likelihood-free rejection sampler
for i = 1 to N dorepeat
generate θ′ from the prior distribution π(·)generate z from the likelihood f (·|θ′)
until ρ{η(z), η(y)} ≤ εset θi = θ′
end for
where η(y) defines a (not necessarily sufficient) statistic
![Page 52: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/52.jpg)
Output
The likelihood-free algorithm samples from the marginal in z of:
πε(θ, z|y) =π(θ)f (z|θ)IAε,y(z)∫
Aε,y×Θ π(θ)f (z|θ)dzdθ,
where Aε,y = {z ∈ D|ρ(η(z), η(y)) < ε}.
The idea behind ABC is that the summary statistics coupled with asmall tolerance should provide a good approximation of theposterior distribution:
πε(θ|y) =
∫πε(θ, z|y)dz ≈ π(θ|y) .
...does it?!
![Page 53: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/53.jpg)
Output
The likelihood-free algorithm samples from the marginal in z of:
πε(θ, z|y) =π(θ)f (z|θ)IAε,y(z)∫
Aε,y×Θ π(θ)f (z|θ)dzdθ,
where Aε,y = {z ∈ D|ρ(η(z), η(y)) < ε}.
The idea behind ABC is that the summary statistics coupled with asmall tolerance should provide a good approximation of theposterior distribution:
πε(θ|y) =
∫πε(θ, z|y)dz ≈ π(θ|y) .
...does it?!
![Page 54: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/54.jpg)
Output
The likelihood-free algorithm samples from the marginal in z of:
πε(θ, z|y) =π(θ)f (z|θ)IAε,y(z)∫
Aε,y×Θ π(θ)f (z|θ)dzdθ,
where Aε,y = {z ∈ D|ρ(η(z), η(y)) < ε}.
The idea behind ABC is that the summary statistics coupled with asmall tolerance should provide a good approximation of theposterior distribution:
πε(θ|y) =
∫πε(θ, z|y)dz ≈ π(θ|y) .
...does it?!
![Page 55: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/55.jpg)
Convergence of ABC (first attempt)
What happens when ε→ 0?
If f (·|θ) is continuous in y , uniformly in θ [!], given an arbitraryδ > 0, there exists ε0 such that ε < ε0 implies
![Page 56: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/56.jpg)
Convergence of ABC (first attempt)
What happens when ε→ 0?
If f (·|θ) is continuous in y , uniformly in θ [!], given an arbitraryδ > 0, there exists ε0 such that ε < ε0 implies
π(θ)∫
f (z|θ)IAε,y(z) dz∫Aε,y×Θ π(θ)f (z|θ)dzdθ
∈ π(θ)f (y|θ)(1∓ δ)µ(Bε)∫Θ π(θ)f (y|θ)dθ(1± δ)µ(Bε)
![Page 57: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/57.jpg)
Convergence of ABC (first attempt)
What happens when ε→ 0?
If f (·|θ) is continuous in y , uniformly in θ [!], given an arbitraryδ > 0, there exists ε0 such that ε < ε0 implies
π(θ)∫
f (z|θ)IAε,y(z) dz∫Aε,y×Θ π(θ)f (z|θ)dzdθ
∈ π(θ)f (y|θ)(1∓ δ)����XXXXµ(Bε)∫Θ π(θ)f (y|θ)dθ(1± δ)����XXXXµ(Bε)
![Page 58: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/58.jpg)
Convergence of ABC (first attempt)
What happens when ε→ 0?
If f (·|θ) is continuous in y , uniformly in θ [!], given an arbitraryδ > 0, there exists ε0 such that ε < ε0 implies
π(θ)∫
f (z|θ)IAε,y(z) dz∫Aε,y×Θ π(θ)f (z|θ)dzdθ
∈ π(θ)f (y|θ)(1∓ δ)����XXXXµ(Bε)∫Θ π(θ)f (y|θ)dθ(1± δ)����XXXXµ(Bε)
[Proof extends to other continuous-in-0 kernels Kε]
![Page 59: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/59.jpg)
Convergence of ABC (second attempt)
What happens when ε→ 0?
For B ⊂ Θ, we have∫B
∫Aε,y
f (z|θ)dz∫Aε,y×Θ π(θ)f (z|θ)dzdθ
π(θ)dθ =
∫Aε,y
∫B f (z|θ)π(θ)dθ∫
Aε,y×Θ π(θ)f (z|θ)dzdθdz
=
∫Aε,y
∫B f (z|θ)π(θ)dθ
m(z)
m(z)∫Aε,y×Θ π(θ)f (z|θ)dzdθ
dz
=
∫Aε,y
π(B|z)m(z)∫
Aε,y×Θ π(θ)f (z|θ)dzdθdz
which indicates convergence for a continuous π(B|z).
![Page 60: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/60.jpg)
Convergence of ABC (second attempt)
What happens when ε→ 0?
For B ⊂ Θ, we have∫B
∫Aε,y
f (z|θ)dz∫Aε,y×Θ π(θ)f (z|θ)dzdθ
π(θ)dθ =
∫Aε,y
∫B f (z|θ)π(θ)dθ∫
Aε,y×Θ π(θ)f (z|θ)dzdθdz
=
∫Aε,y
∫B f (z|θ)π(θ)dθ
m(z)
m(z)∫Aε,y×Θ π(θ)f (z|θ)dzdθ
dz
=
∫Aε,y
π(B|z)m(z)∫
Aε,y×Θ π(θ)f (z|θ)dzdθdz
which indicates convergence for a continuous π(B|z).
![Page 61: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/61.jpg)
Convergence (do not attempt!)
...and the above does not apply to insufficient statistics:
If η(y) is not a sufficient statistics, the best one can hope for is
π(θ|η(y) , not π(θ|y)
If η(y) is an ancillary statistic, the whole information contained iny is lost!, the “best” one can hope for is
π(θ|η(y) = π(η)
Bummer!!!
![Page 62: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/62.jpg)
Convergence (do not attempt!)
...and the above does not apply to insufficient statistics:
If η(y) is not a sufficient statistics, the best one can hope for is
π(θ|η(y) , not π(θ|y)
If η(y) is an ancillary statistic, the whole information contained iny is lost!, the “best” one can hope for is
π(θ|η(y) = π(η)
Bummer!!!
![Page 63: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/63.jpg)
Convergence (do not attempt!)
...and the above does not apply to insufficient statistics:
If η(y) is not a sufficient statistics, the best one can hope for is
π(θ|η(y) , not π(θ|y)
If η(y) is an ancillary statistic, the whole information contained iny is lost!, the “best” one can hope for is
π(θ|η(y) = π(η)
Bummer!!!
![Page 64: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/64.jpg)
Convergence (do not attempt!)
...and the above does not apply to insufficient statistics:
If η(y) is not a sufficient statistics, the best one can hope for is
π(θ|η(y) , not π(θ|y)
If η(y) is an ancillary statistic, the whole information contained iny is lost!, the “best” one can hope for is
π(θ|η(y) = π(η)
Bummer!!!
![Page 65: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/65.jpg)
Probit modelling on Pima Indian women
Example (R benchmark)200 Pima Indian women with observed variables
I plasma glucose concentration in oral glucose tolerance test
I diastolic blood pressure
I diabetes pedigree function
I presence/absence of diabetes
Probability of diabetes as function of variables
P(y = 1|x) = Φ(x1β1 + x2β2 + x3β3) ,
200 observations of Pima.tr and g -prior modelling:
β ∼ N3(0, n(XTX)−1
)importance function inspired from MLE estimator distribution
β ∼ N (β, Σ)
![Page 66: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/66.jpg)
Probit modelling on Pima Indian women
Example (R benchmark)200 Pima Indian women with observed variables
I plasma glucose concentration in oral glucose tolerance test
I diastolic blood pressure
I diabetes pedigree function
I presence/absence of diabetes
Probability of diabetes as function of variables
P(y = 1|x) = Φ(x1β1 + x2β2 + x3β3) ,
200 observations of Pima.tr and g -prior modelling:
β ∼ N3(0, n(XTX)−1
)importance function inspired from MLE estimator distribution
β ∼ N (β, Σ)
![Page 67: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/67.jpg)
Probit modelling on Pima Indian women
Example (R benchmark)200 Pima Indian women with observed variables
I plasma glucose concentration in oral glucose tolerance test
I diastolic blood pressure
I diabetes pedigree function
I presence/absence of diabetes
Probability of diabetes as function of variables
P(y = 1|x) = Φ(x1β1 + x2β2 + x3β3) ,
200 observations of Pima.tr and g -prior modelling:
β ∼ N3(0, n(XTX)−1
)importance function inspired from MLE estimator distribution
β ∼ N (β, Σ)
![Page 68: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/68.jpg)
Probit modelling on Pima Indian women
Example (R benchmark)200 Pima Indian women with observed variables
I plasma glucose concentration in oral glucose tolerance test
I diastolic blood pressure
I diabetes pedigree function
I presence/absence of diabetes
Probability of diabetes as function of variables
P(y = 1|x) = Φ(x1β1 + x2β2 + x3β3) ,
200 observations of Pima.tr and g -prior modelling:
β ∼ N3(0, n(XTX)−1
)importance function inspired from MLE estimator distribution
β ∼ N (β, Σ)
![Page 69: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/69.jpg)
Pima Indian benchmark
−0.005 0.010 0.020 0.030
020
4060
8010
0
Dens
ity
−0.05 −0.03 −0.01
020
4060
80
Dens
ity
−1.0 0.0 1.0 2.0
0.00.2
0.40.6
0.81.0
Dens
ity
Figure: Comparison between density estimates of the marginals on β1
(left), β2 (center) and β3 (right) from ABC rejection samples (red) andMCMC samples (black)
.
![Page 70: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/70.jpg)
MA example
MA(q) model
xt = εt +
q∑i=1
ϑiεt−i
Simple prior: uniform over the inverse [real and complex] roots in
Q(u) = 1−q∑
i=1
ϑiui
under the identifiability conditions
![Page 71: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/71.jpg)
MA example
MA(q) model
xt = εt +
q∑i=1
ϑiεt−i
Simple prior: uniform prior over the identifiability zone, i.e. trianglefor MA(2)
![Page 72: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/72.jpg)
MA example (2)
ABC algorithm thus made of
1. picking a new value (ϑ1, ϑ2) in the triangle
2. generating an iid sequence (εt)−q<t≤T
3. producing a simulated series (x ′t)1≤t≤T
Distance: basic distance between the series
ρ((x ′t)1≤t≤T , (xt)1≤t≤T ) =T∑t=1
(xt − x ′t)2
or distance between summary statistics like the q = 2autocorrelations
τj =T∑
t=j+1
xtxt−j
![Page 73: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/73.jpg)
MA example (2)
ABC algorithm thus made of
1. picking a new value (ϑ1, ϑ2) in the triangle
2. generating an iid sequence (εt)−q<t≤T
3. producing a simulated series (x ′t)1≤t≤T
Distance: basic distance between the series
ρ((x ′t)1≤t≤T , (xt)1≤t≤T ) =T∑t=1
(xt − x ′t)2
or distance between summary statistics like the q = 2autocorrelations
τj =T∑
t=j+1
xtxt−j
![Page 74: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/74.jpg)
Comparison of distance impact
Evaluation of the tolerance on the ABC sample against bothdistances (ε = 100%, 10%, 1%, 0.1%) for an MA(2) model
![Page 75: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/75.jpg)
Comparison of distance impact
0.0 0.2 0.4 0.6 0.8
01
23
4
θ1
−2.0 −1.0 0.0 0.5 1.0 1.50.0
0.51.0
1.5
θ2
Evaluation of the tolerance on the ABC sample against bothdistances (ε = 100%, 10%, 1%, 0.1%) for an MA(2) model
![Page 76: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/76.jpg)
Comparison of distance impact
0.0 0.2 0.4 0.6 0.8
01
23
4
θ1
−2.0 −1.0 0.0 0.5 1.0 1.50.0
0.51.0
1.5
θ2
Evaluation of the tolerance on the ABC sample against bothdistances (ε = 100%, 10%, 1%, 0.1%) for an MA(2) model
![Page 77: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/77.jpg)
Comments
I Role of distance paramount (because ε 6= 0)
I ε matters little if “small enough”
I representative of “curse of dimensionality”
I the data as a whole may be paradoxically weakly informativefor ABC
![Page 78: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/78.jpg)
ABC (simul’) advances
how approximative is ABC?
Simulating from the prior is often poor in efficiencyEither modify the proposal distribution on θ to increase the densityof x ’s within the vicinity of y ...
[Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007]
...or by viewing the problem as a conditional density estimationand by developing techniques to allow for larger ε
[Beaumont et al., 2002]
.....or even by including ε in the inferential framework [ABCµ][Ratmann et al., 2009]
![Page 79: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/79.jpg)
ABC (simul’) advances
how approximative is ABC?
Simulating from the prior is often poor in efficiencyEither modify the proposal distribution on θ to increase the densityof x ’s within the vicinity of y ...
[Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007]
...or by viewing the problem as a conditional density estimationand by developing techniques to allow for larger ε
[Beaumont et al., 2002]
.....or even by including ε in the inferential framework [ABCµ][Ratmann et al., 2009]
![Page 80: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/80.jpg)
ABC (simul’) advances
how approximative is ABC?
Simulating from the prior is often poor in efficiencyEither modify the proposal distribution on θ to increase the densityof x ’s within the vicinity of y ...
[Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007]
...or by viewing the problem as a conditional density estimationand by developing techniques to allow for larger ε
[Beaumont et al., 2002]
.....or even by including ε in the inferential framework [ABCµ][Ratmann et al., 2009]
![Page 81: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/81.jpg)
ABC (simul’) advances
how approximative is ABC?
Simulating from the prior is often poor in efficiencyEither modify the proposal distribution on θ to increase the densityof x ’s within the vicinity of y ...
[Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007]
...or by viewing the problem as a conditional density estimationand by developing techniques to allow for larger ε
[Beaumont et al., 2002]
.....or even by including ε in the inferential framework [ABCµ][Ratmann et al., 2009]
![Page 82: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/82.jpg)
ABC-NP
Better usage of [prior] simulations byadjustement: instead of throwing awayθ′ such that ρ(η(z), η(y)) > ε, replaceθ’s with locally regressed transforms
(use with BIC)
θ∗ = θ − {η(z)− η(y)}Tβ [Csillery et al., TEE, 2010]
where β is obtained by [NP] weighted least square regression on(η(z)− η(y)) with weights
Kδ {ρ(η(z), η(y))}
[Beaumont et al., 2002, Genetics]
![Page 83: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/83.jpg)
ABC-NP (regression)
Also found in the subsequent literature, e.g. in Fearnhead-Prangle (2012) :weight directly simulation by
Kδ {ρ(η(z(θ)), η(y))}
or
1
S
S∑s=1
Kδ {ρ(η(zs(θ)), η(y))}
[consistent estimate of f (η|θ)]Curse of dimensionality: poor estimate when d = dim(η) is large...
![Page 84: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/84.jpg)
ABC-NP (regression)
Also found in the subsequent literature, e.g. in Fearnhead-Prangle (2012) :weight directly simulation by
Kδ {ρ(η(z(θ)), η(y))}
or
1
S
S∑s=1
Kδ {ρ(η(zs(θ)), η(y))}
[consistent estimate of f (η|θ)]Curse of dimensionality: poor estimate when d = dim(η) is large...
![Page 85: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/85.jpg)
ABC-NP (density estimation)
Use of the kernel weights
Kδ {ρ(η(z(θ)), η(y))}
leads to the NP estimate of the posterior expectation∑i θiKδ {ρ(η(z(θi )), η(y))}∑i Kδ {ρ(η(z(θi )), η(y))}
[Blum, JASA, 2010]
![Page 86: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/86.jpg)
ABC-NP (density estimation)
Use of the kernel weights
Kδ {ρ(η(z(θ)), η(y))}
leads to the NP estimate of the posterior conditional density∑i Kb(θi − θ)Kδ {ρ(η(z(θi )), η(y))}∑
i Kδ {ρ(η(z(θi )), η(y))}
[Blum, JASA, 2010]
![Page 87: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/87.jpg)
ABC-NP (density estimations)
Other versions incorporating regression adjustments∑i Kb(θ∗i − θ)Kδ {ρ(η(z(θi )), η(y))}∑
i Kδ {ρ(η(z(θi )), η(y))}
In all cases, error
E[g(θ|y)]− g(θ|y) = cb2 + cδ2 + OP(b2 + δ2) + OP(1/nδd)
var(g(θ|y)) =c
nbδd(1 + oP(1))
![Page 88: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/88.jpg)
ABC-NP (density estimations)
Other versions incorporating regression adjustments∑i Kb(θ∗i − θ)Kδ {ρ(η(z(θi )), η(y))}∑
i Kδ {ρ(η(z(θi )), η(y))}
In all cases, error
E[g(θ|y)]− g(θ|y) = cb2 + cδ2 + OP(b2 + δ2) + OP(1/nδd)
var(g(θ|y)) =c
nbδd(1 + oP(1))
[Blum, JASA, 2010]
![Page 89: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/89.jpg)
ABC-NP (density estimations)
Other versions incorporating regression adjustments∑i Kb(θ∗i − θ)Kδ {ρ(η(z(θi )), η(y))}∑
i Kδ {ρ(η(z(θi )), η(y))}
In all cases, error
E[g(θ|y)]− g(θ|y) = cb2 + cδ2 + OP(b2 + δ2) + OP(1/nδd)
var(g(θ|y)) =c
nbδd(1 + oP(1))
[standard NP calculations]
![Page 90: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/90.jpg)
ABC-NCH
Incorporating non-linearities and heterocedasticities:
θ∗ = m(η(y)) + [θ − m(η(z))]σ(η(y))
σ(η(z))
where
I m(η) estimated by non-linear regression (e.g., neural network)
I σ(η) estimated by non-linear regression on residuals
log{θi − m(ηi )}2 = log σ2(ηi ) + ξi
[Blum & Francois, 2009]
![Page 91: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/91.jpg)
ABC-NCH
Incorporating non-linearities and heterocedasticities:
θ∗ = m(η(y)) + [θ − m(η(z))]σ(η(y))
σ(η(z))
where
I m(η) estimated by non-linear regression (e.g., neural network)
I σ(η) estimated by non-linear regression on residuals
log{θi − m(ηi )}2 = log σ2(ηi ) + ξi
[Blum & Francois, 2009]
![Page 92: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/92.jpg)
ABC-NCH (2)
Why neural network?
I fights curse of dimensionality
I selects relevant summary statistics
I provides automated dimension reduction
I offers a model choice capability
I improves upon multinomial logistic
[Blum & Francois, 2009]
![Page 93: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/93.jpg)
ABC-NCH (2)
Why neural network?
I fights curse of dimensionality
I selects relevant summary statistics
I provides automated dimension reduction
I offers a model choice capability
I improves upon multinomial logistic
[Blum & Francois, 2009]
![Page 94: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/94.jpg)
ABC-MCMC
how approximative is ABC?
Markov chain (θ(t)) created via the transition function
θ(t+1) =
θ′ ∼ Kω(θ′|θ(t)) if x ∼ f (x |θ′) is such that x = y
and u ∼ U(0, 1) ≤ π(θ′)Kω(θ(t)|θ′)π(θ(t))Kω(θ′|θ(t))
,
θ(t) otherwise,
has the posterior π(θ|y) as stationary distribution[Marjoram et al, 2003]
![Page 95: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/95.jpg)
ABC-MCMC
how approximative is ABC?
Markov chain (θ(t)) created via the transition function
θ(t+1) =
θ′ ∼ Kω(θ′|θ(t)) if x ∼ f (x |θ′) is such that x = y
and u ∼ U(0, 1) ≤ π(θ′)Kω(θ(t)|θ′)π(θ(t))Kω(θ′|θ(t))
,
θ(t) otherwise,
has the posterior π(θ|y) as stationary distribution[Marjoram et al, 2003]
![Page 96: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/96.jpg)
ABC-MCMC (2)
Algorithm 2 Likelihood-free MCMC sampler
Use Algorithm 1 to get (θ(0), z(0))for t = 1 to N do
Generate θ′ from Kω
(·|θ(t−1)
),
Generate z′ from the likelihood f (·|θ′),Generate u from U[0,1],
if u ≤ π(θ′)Kω(θ(t−1)|θ′)π(θ(t−1)Kω(θ′|θ(t−1))
IAε,y(z′) then
set (θ(t), z(t)) = (θ′, z′)else
(θ(t), z(t))) = (θ(t−1), z(t−1)),end if
end for
![Page 97: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/97.jpg)
Why does it work?
Acceptance probability does not involve calculating the likelihoodand
πε(θ′, z′|y)
πε(θ(t−1), z(t−1)|y)
× q(θ(t−1)|θ′)f (z(t−1)|θ(t−1))
q(θ′|θ(t−1))f (z′|θ′)
=π(θ′)���
�XXXXf (z′|θ′) IAε,y(z′)
π(θ(t−1)) f (z(t−1)|θ(t−1)) IAε,y(z(t−1))
× q(θ(t−1)|θ′) f (z(t−1)|θ(t−1))
q(θ′|θ(t−1))����XXXXf (z′|θ′)
![Page 98: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/98.jpg)
Why does it work?
Acceptance probability does not involve calculating the likelihoodand
πε(θ′, z′|y)
πε(θ(t−1), z(t−1)|y)
× q(θ(t−1)|θ′)f (z(t−1)|θ(t−1))
q(θ′|θ(t−1))f (z′|θ′)
=π(θ′)���
�XXXXf (z′|θ′) IAε,y(z′)
π(θ(t−1))((((((((hhhhhhhhf (z(t−1)|θ(t−1)) IAε,y(z(t−1))
× q(θ(t−1)|θ′)((((((((hhhhhhhhf (z(t−1)|θ(t−1))
q(θ′|θ(t−1))����XXXXf (z′|θ′)
![Page 99: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/99.jpg)
Why does it work?
Acceptance probability does not involve calculating the likelihoodand
πε(θ′, z′|y)
πε(θ(t−1), z(t−1)|y)
× q(θ(t−1)|θ′)f (z(t−1)|θ(t−1))
q(θ′|θ(t−1))f (z′|θ′)
=π(θ′)���
�XXXXf (z′|θ′) IAε,y(z′)
π(θ(t−1))((((((((hhhhhhhhf (z(t−1)|θ(t−1))���
���XXXXXXIAε,y(z(t−1))
× q(θ(t−1)|θ′)((((((((hhhhhhhhf (z(t−1)|θ(t−1))
q(θ′|θ(t−1))����XXXXf (z′|θ′)
=π(θ′)q(θ(t−1)|θ′)
π(θ(t−1)q(θ′|θ(t−1))IAε,y(z′)
![Page 100: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/100.jpg)
A toy example
Case of
x ∼ 1
2N (θ, 1) +
1
2N (−θ, 1)
under prior θ ∼ N (0, 10)
ABC samplerthetas=rnorm(N,sd=10)
zed=sample(c(1,-1),N,rep=TRUE)*thetas+rnorm(N,sd=1)
eps=quantile(abs(zed-x),.01)
abc=thetas[abs(zed-x)<eps]
![Page 101: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/101.jpg)
A toy example
Case of
x ∼ 1
2N (θ, 1) +
1
2N (−θ, 1)
under prior θ ∼ N (0, 10)
ABC samplerthetas=rnorm(N,sd=10)
zed=sample(c(1,-1),N,rep=TRUE)*thetas+rnorm(N,sd=1)
eps=quantile(abs(zed-x),.01)
abc=thetas[abs(zed-x)<eps]
![Page 102: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/102.jpg)
A toy example
Case of
x ∼ 1
2N (θ, 1) +
1
2N (−θ, 1)
under prior θ ∼ N (0, 10)
ABC-MCMC samplermetas=rep(0,N)
metas[1]=rnorm(1,sd=10)
zed[1]=x
for (t in 2:N){
metas[t]=rnorm(1,mean=metas[t-1],sd=5)
zed[t]=rnorm(1,mean=(1-2*(runif(1)<.5))*metas[t],sd=1)
if ((abs(zed[t]-x)>eps)||(runif(1)>dnorm(metas[t],sd=10)/dnorm(metas[t-1],sd=10))){
metas[t]=metas[t-1]
zed[t]=zed[t-1]}
}
![Page 103: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/103.jpg)
A toy example
x = 2
![Page 104: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/104.jpg)
A toy example
x = 2
θ
−4 −2 0 2 4
0.00
0.05
0.10
0.15
0.20
0.25
0.30
![Page 105: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/105.jpg)
A toy example
x = 10
![Page 106: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/106.jpg)
A toy example
x = 10
θ
−10 −5 0 5 10
0.00
0.05
0.10
0.15
0.20
0.25
![Page 107: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/107.jpg)
A toy example
x = 50
![Page 108: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/108.jpg)
A toy example
x = 50
θ
−40 −20 0 20 40
0.00
0.02
0.04
0.06
0.08
0.10
![Page 109: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/109.jpg)
ABC-PMC
Use of a transition kernel as in population Monte Carlo withmanageable IS correctionGenerate a sample at iteration t by
πt(θ(t)) ∝
N∑j=1
ω(t−1)j Kt(θ
(t)|θ(t−1)j )
modulo acceptance of the associated xt , and use an importance
weight associated with an accepted simulation θ(t)i
ω(t)i ∝ π(θ
(t)i )/πt(θ
(t)i ) .
c© Still likelihood free[Beaumont et al., 2009]
![Page 110: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/110.jpg)
Our ABC-PMC algorithm
Given a decreasing sequence of approximation levels ε1 ≥ . . . ≥ εT ,
1. At iteration t = 1,
For i = 1, ...,N
Simulate θ(1)i ∼ π(θ) and x ∼ f (x |θ(1)
i ) until %(x , y) < ε1
Set ω(1)i = 1/N
Take τ 2 as twice the empirical variance of the θ(1)i ’s
2. At iteration 2 ≤ t ≤ T ,
For i = 1, ...,N, repeat
Pick θ?i from the θ(t−1)j ’s with probabilities ω
(t−1)j
generate θ(t)i |θ
?i ∼ N (θ?i , σ
2t ) and x ∼ f (x |θ(t)
i )
until %(x , y) < εt
Set ω(t)i ∝ π(θ
(t)i )/
∑Nj=1 ω
(t−1)j ϕ
(σ−1t
{θ
(t)i − θ
(t−1)j )
})Take τ 2
t+1 as twice the weighted empirical variance of the θ(t)i ’s
![Page 111: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/111.jpg)
Sequential Monte Carlo
SMC is a simulation technique to approximate a sequence ofrelated probability distributions πn with π0 “easy” and πT astarget.Iterated IS as PMC: particles moved from time n to time n viakernel Kn and use of a sequence of extended targets πn
πn(z0:n) = πn(zn)n∏
j=0
Lj(zj+1, zj)
where the Lj ’s are backward Markov kernels [check that πn(zn) is amarginal]
[Del Moral, Doucet & Jasra, Series B, 2006]
![Page 112: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/112.jpg)
Sequential Monte Carlo (2)
Algorithm 3 SMC sampler
sample z(0)i ∼ γ0(x) (i = 1, . . . ,N)
compute weights w(0)i = π0(z
(0)i )/γ0(z
(0)i )
for t = 1 to N doif ESS(w (t−1)) < NT then
resample N particles z(t−1) and set weights to 1end ifgenerate z
(t−1)i ∼ Kt(z
(t−1)i , ·) and set weights to
w(t)i = w
(t−1)i−1
πt(z(t)i ))Lt−1(z
(t)i ), z
(t−1)i ))
πt−1(z(t−1)i ))Kt(z
(t−1)i ), z
(t)i ))
end for
[Del Moral, Doucet & Jasra, Series B, 2006]
![Page 113: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/113.jpg)
ABC-SMC
[Del Moral, Doucet & Jasra, 2009]
True derivation of an SMC-ABC algorithmUse of a kernel Kn associated with target πεn and derivation of thebackward kernel
Ln−1(z , z ′) =πεn(z ′)Kn(z ′, z)
πn(z)
Update of the weights
win ∝ wi(n−1)
∑Mm=1 IAεn (xm
in )∑Mm=1 IAεn−1
(xmi(n−1))
when xmin ∼ K (xi(n−1), ·)
![Page 114: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/114.jpg)
ABC-SMCM
Modification: Makes M repeated simulations of the pseudo-data zgiven the parameter, rather than using a single [M = 1] simulation,leading to weight that is proportional to the number of acceptedzi s
ω(θ) =1
M
M∑i=1
Iρ(η(y),η(zi ))<ε
[limit in M means exact simulation from (tempered) target]
![Page 115: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/115.jpg)
Properties of ABC-SMC
The ABC-SMC method properly uses a backward kernel L(z , z ′) tosimplify the importance weight and to remove the dependence onthe unknown likelihood from this weight. Update of importanceweights is reduced to the ratio of the proportions of survivingparticlesMajor assumption: the forward kernel K is supposed to be invariantagainst the true target [tempered version of the true posterior]Adaptivity in ABC-SMC algorithm only found in on-lineconstruction of the thresholds εt , slowly enough to keep a largenumber of accepted transitions
![Page 116: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/116.jpg)
Properties of ABC-SMC
The ABC-SMC method properly uses a backward kernel L(z , z ′) tosimplify the importance weight and to remove the dependence onthe unknown likelihood from this weight. Update of importanceweights is reduced to the ratio of the proportions of survivingparticlesMajor assumption: the forward kernel K is supposed to be invariantagainst the true target [tempered version of the true posterior]Adaptivity in ABC-SMC algorithm only found in on-lineconstruction of the thresholds εt , slowly enough to keep a largenumber of accepted transitions
![Page 117: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/117.jpg)
A mixture example (1)
Toy model of Sisson et al. (2007): if
θ ∼ U(−10, 10) , x |θ ∼ 0.5N (θ, 1) + 0.5N (θ, 1/100) ,
then the posterior distribution associated with y = 0 is the normalmixture
θ|y = 0 ∼ 0.5N (0, 1) + 0.5N (0, 1/100)
restricted to [−10, 10].Furthermore, true target available as
π(θ||x | < ε) ∝ Φ(ε−θ)−Φ(−ε−θ)+Φ(10(ε−θ))−Φ(−10(ε+θ)) .
![Page 118: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/118.jpg)
A mixture example (2)
Recovery of the target, whether using a fixed standard deviation ofτ = 0.15 or τ = 1/0.15, or a sequence of adaptive τt ’s.
θθ
−3 −2 −1 0 1 2 3
0.0
0.2
0.4
0.6
0.8
1.0
θθ
−3 −2 −1 0 1 2 3
0.0
0.2
0.4
0.6
0.8
1.0
θθ
−3 −2 −1 0 1 2 3
0.0
0.2
0.4
0.6
0.8
1.0
θθ
−3 −2 −1 0 1 2 3
0.0
0.2
0.4
0.6
0.8
1.0
θθ
−3 −2 −1 0 1 2 3
0.0
0.2
0.4
0.6
0.8
1.0
![Page 119: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/119.jpg)
ABC inference machine
Introduction
Approximate Bayesian computation
ABC as an inference machineError inc.Exact BC and approximate targetssummary statisticSeries B discussion2
![Page 120: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/120.jpg)
How much Bayesian?
I maybe a convergent methodof inference (meaningful?sufficient? foreign?)
I approximation errorunknown (w/o simulation)
I pragmatic Bayes (there is noother solution!)
I many calibration issues(tolerance, distance,statistics)
...should Bayesians care?!
![Page 121: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/121.jpg)
How much Bayesian?
I maybe a convergent methodof inference (meaningful?sufficient? foreign?)
I approximation errorunknown (w/o simulation)
I pragmatic Bayes (there is noother solution!)
I many calibration issues(tolerance, distance,statistics)
...should Bayesians care?!
![Page 122: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/122.jpg)
ABCµ
Idea Infer about the error as well:Use of a joint density
f (θ, ε|y) ∝ ξ(ε|y, θ)× πθ(θ)× πε(ε)
where y is the data, and ξ(ε|y, θ) is the prior predictive density ofρ(η(z), η(y)) given θ and y when z ∼ f (z|θ)Warning! Replacement of ξ(ε|y, θ) with a non-parametric kernelapproximation.
[Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
![Page 123: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/123.jpg)
ABCµ
Idea Infer about the error as well:Use of a joint density
f (θ, ε|y) ∝ ξ(ε|y, θ)× πθ(θ)× πε(ε)
where y is the data, and ξ(ε|y, θ) is the prior predictive density ofρ(η(z), η(y)) given θ and y when z ∼ f (z|θ)Warning! Replacement of ξ(ε|y, θ) with a non-parametric kernelapproximation.
[Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
![Page 124: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/124.jpg)
ABCµ
Idea Infer about the error as well:Use of a joint density
f (θ, ε|y) ∝ ξ(ε|y, θ)× πθ(θ)× πε(ε)
where y is the data, and ξ(ε|y, θ) is the prior predictive density ofρ(η(z), η(y)) given θ and y when z ∼ f (z|θ)Warning! Replacement of ξ(ε|y, θ) with a non-parametric kernelapproximation.
[Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]
![Page 125: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/125.jpg)
ABCµ details
Multidimensional distances ρk (k = 1, . . . ,K ) and errorsεk = ρk(ηk(z), ηk(y)), with
εk ∼ ξk(ε|y, θ) ≈ ξk(ε|y, θ) =1
Bhk
∑b
K [{εk−ρk(ηk(zb), ηk(y))}/hk ]
then used in replacing ξ(ε|y, θ) with mink ξk(ε|y, θ)ABCµ involves acceptance probability
π(θ′, ε′)
π(θ, ε)
q(θ′, θ)q(ε′, ε)
q(θ, θ′)q(ε, ε′)
mink ξk(ε′|y, θ′)mink ξk(ε|y, θ)
![Page 126: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/126.jpg)
ABCµ details
Multidimensional distances ρk (k = 1, . . . ,K ) and errorsεk = ρk(ηk(z), ηk(y)), with
εk ∼ ξk(ε|y, θ) ≈ ξk(ε|y, θ) =1
Bhk
∑b
K [{εk−ρk(ηk(zb), ηk(y))}/hk ]
then used in replacing ξ(ε|y, θ) with mink ξk(ε|y, θ)ABCµ involves acceptance probability
π(θ′, ε′)
π(θ, ε)
q(θ′, θ)q(ε′, ε)
q(θ, θ′)q(ε, ε′)
mink ξk(ε′|y, θ′)mink ξk(ε|y, θ)
![Page 127: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/127.jpg)
ABCµ multiple errors
[ c© Ratmann et al., PNAS, 2009]
![Page 128: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/128.jpg)
ABCµ for model choice
[ c© Ratmann et al., PNAS, 2009]
![Page 129: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/129.jpg)
Questions about ABCµ [and model choice]
For each model under comparison, marginal posterior on ε used toassess the fit of the model (HPD includes 0 or not).
I Is the data informative about ε? [Identifiability]
I How much does the prior π(ε) impact the comparison?
I How is using both ξ(ε|x0, θ) and πε(ε) compatible with astandard probability model? [remindful of Wilkinson’s eABC ]
I Where is the penalisation for complexity in the modelcomparison?
[X, Mengersen & Chen, 2010, PNAS]
![Page 130: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/130.jpg)
Questions about ABCµ [and model choice]
For each model under comparison, marginal posterior on ε used toassess the fit of the model (HPD includes 0 or not).
I Is the data informative about ε? [Identifiability]
I How much does the prior π(ε) impact the comparison?
I How is using both ξ(ε|x0, θ) and πε(ε) compatible with astandard probability model? [remindful of Wilkinson’s eABC ]
I Where is the penalisation for complexity in the modelcomparison?
[X, Mengersen & Chen, 2010, PNAS]
![Page 131: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/131.jpg)
Wilkinson’s exact BC (not exactly!)
ABC approximation error (i.e. non-zero tolerance) replaced withexact simulation from a controlled approximation to the target,convolution of true posterior with kernel function
πε(θ, z|y) =π(θ)f (z|θ)Kε(y − z)∫π(θ)f (z|θ)Kε(y − z)dzdθ
,
with Kε kernel parameterised by bandwidth ε.[Wilkinson, 2008]
Theorem
The ABC algorithm based on the assumption of a randomisedobservation y = y + ξ, ξ ∼ Kε, and an acceptance probability of
Kε(y − z)/M
gives draws from the posterior distribution π(θ|y).
![Page 132: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/132.jpg)
Wilkinson’s exact BC (not exactly!)
ABC approximation error (i.e. non-zero tolerance) replaced withexact simulation from a controlled approximation to the target,convolution of true posterior with kernel function
πε(θ, z|y) =π(θ)f (z|θ)Kε(y − z)∫π(θ)f (z|θ)Kε(y − z)dzdθ
,
with Kε kernel parameterised by bandwidth ε.[Wilkinson, 2008]
Theorem
The ABC algorithm based on the assumption of a randomisedobservation y = y + ξ, ξ ∼ Kε, and an acceptance probability of
Kε(y − z)/M
gives draws from the posterior distribution π(θ|y).
![Page 133: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/133.jpg)
How exact a BC?
“Using ε to represent measurement error isstraightforward, whereas using ε to model the modeldiscrepancy is harder to conceptualize and not ascommonly used”
[Richard Wilkinson, 2008]
![Page 134: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/134.jpg)
How exact a BC?
Pros
I Pseudo-data from true model and observed data from noisymodel
I Interesting perspective in that outcome is completelycontrolled
I Link with ABCµ and assuming y is observed with ameasurement error with density Kε
I Relates to the theory of model approximation[Kennedy & O’Hagan, 2001]
Cons
I Requires Kε to be bounded by M
I True approximation error never assessed
I Requires a modification of the standard ABC algorithm
![Page 135: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/135.jpg)
ABC for HMMs
Specific case of a hidden Markov model
Xt+1 ∼ Qθ(Xt , ·)Yt+1 ∼ gθ(·|xt)
where only y01:n is observed.
[Dean, Singh, Jasra, & Peters, 2011]
Use of specific constraints, adapted to the Markov structure:{y1 ∈ B(y 0
1 , ε)}× · · · ×
{yn ∈ B(y 0
n , ε)}
![Page 136: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/136.jpg)
ABC for HMMs
Specific case of a hidden Markov model
Xt+1 ∼ Qθ(Xt , ·)Yt+1 ∼ gθ(·|xt)
where only y01:n is observed.
[Dean, Singh, Jasra, & Peters, 2011]
Use of specific constraints, adapted to the Markov structure:{y1 ∈ B(y 0
1 , ε)}× · · · ×
{yn ∈ B(y 0
n , ε)}
![Page 137: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/137.jpg)
ABC-MLE for HMMs
ABC-MLE defined by
θεn = arg maxθ
Pθ(Y1 ∈ B(y 0
1 , ε), . . . ,Yn ∈ B(y 0n , ε)
)Exact MLE for the likelihood same basis as Wilkinson!
pεθ(y 01 , . . . , yn)
corresponding to the perturbed process
(xt , yt + εzt)1≤t≤n zt ∼ U(B(0, 1)
[Dean, Singh, Jasra, & Peters, 2011]
![Page 138: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/138.jpg)
ABC-MLE for HMMs
ABC-MLE defined by
θεn = arg maxθ
Pθ(Y1 ∈ B(y 0
1 , ε), . . . ,Yn ∈ B(y 0n , ε)
)Exact MLE for the likelihood same basis as Wilkinson!
pεθ(y 01 , . . . , yn)
corresponding to the perturbed process
(xt , yt + εzt)1≤t≤n zt ∼ U(B(0, 1)
[Dean, Singh, Jasra, & Peters, 2011]
![Page 139: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/139.jpg)
ABC-MLE is biased
I ABC-MLE is asymptotically (in n) biased with target
l ε(θ) = Eθ∗ [log pεθ(Y1|Y−∞:0)]
I but ABC-MLE converges to true value in the sense
l εn(θn)→ l ε(θ)
for all sequences (θn) converging to θ and εn ↘ ε
![Page 140: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/140.jpg)
ABC-MLE is biased
I ABC-MLE is asymptotically (in n) biased with target
l ε(θ) = Eθ∗ [log pεθ(Y1|Y−∞:0)]
I but ABC-MLE converges to true value in the sense
l εn(θn)→ l ε(θ)
for all sequences (θn) converging to θ and εn ↘ ε
![Page 141: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/141.jpg)
Noisy ABC-MLE
Idea: Modify instead the data from the start
(y 01 + εζ1, . . . , yn + εζn)
[ see Fearnhead-Prangle ] andnoisy ABC-MLE estimate
arg maxθ
Pθ(Y1 ∈ B(y 0
1 + εζ1, ε), . . . ,Yn ∈ B(y 0n + εζn, ε)
)[Dean, Singh, Jasra, & Peters, 2011]
![Page 142: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/142.jpg)
Consistent noisy ABC-MLE
I Degrading the data improves the estimation performances:I Noisy ABC-MLE is asymptotically (in n) consistentI under further assumptions, the noisy ABC-MLE is
asymptotically normalI increase in variance of order ε−2
I likely degradation in precision or computing time due to thelack of summary statistic [curse of dimensionality]
![Page 143: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/143.jpg)
SMC for ABC likelihood
Algorithm 4 SMC ABC for HMMs
Given θfor k = 1, . . . , n do
generate proposals (x1k , y
1k ), . . . , (xN
k , yNk ) from the model
weigh each proposal with ωlk = IB(y0
k +εζk ,ε)(y l
k)
renormalise the weights and sample the x lk ’s accordingly
end forapproximate the likelihood by
n∏k=1
(N∑l=1
ωlk
/N
)
[Jasra, Singh, Martin, & McCoy, 2010]
![Page 144: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/144.jpg)
Which summary?
Fundamental difficulty of the choice of the summary statistic whenthere is no non-trivial sufficient statisticsStarting from a large collection of summary statistics is available,Joyce and Marjoram (2008) consider the sequential inclusion intothe ABC target, with a stopping rule based on a likelihood ratiotest
I Not taking into account the sequential nature of the tests
I Depends on parameterisation
I Order of inclusion matters
I likelihood ratio test?!
![Page 145: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/145.jpg)
Which summary?
Fundamental difficulty of the choice of the summary statistic whenthere is no non-trivial sufficient statisticsStarting from a large collection of summary statistics is available,Joyce and Marjoram (2008) consider the sequential inclusion intothe ABC target, with a stopping rule based on a likelihood ratiotest
I Not taking into account the sequential nature of the tests
I Depends on parameterisation
I Order of inclusion matters
I likelihood ratio test?!
![Page 146: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/146.jpg)
Which summary?
Fundamental difficulty of the choice of the summary statistic whenthere is no non-trivial sufficient statisticsStarting from a large collection of summary statistics is available,Joyce and Marjoram (2008) consider the sequential inclusion intothe ABC target, with a stopping rule based on a likelihood ratiotest
I Not taking into account the sequential nature of the tests
I Depends on parameterisation
I Order of inclusion matters
I likelihood ratio test?!
![Page 147: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/147.jpg)
Which summary for model choice?
Depending on the choice of η(·), the Bayes factor based on thisinsufficient statistic,
Bη12(y) =
∫π1(θ1)f η1 (η(y)|θ1) dθ1∫π2(θ2)f η2 (η(y)|θ2) dθ2
,
is consistent or not.[X, Cornuet, Marin, & Pillai, 2012]
Consistency only depends on the range of Ei [η(y)] under bothmodels.
[Marin, Pillai, X, & Rousseau, 2012]
![Page 148: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/148.jpg)
Which summary for model choice?
Depending on the choice of η(·), the Bayes factor based on thisinsufficient statistic,
Bη12(y) =
∫π1(θ1)f η1 (η(y)|θ1) dθ1∫π2(θ2)f η2 (η(y)|θ2) dθ2
,
is consistent or not.[X, Cornuet, Marin, & Pillai, 2012]
Consistency only depends on the range of Ei [η(y)] under bothmodels.
[Marin, Pillai, X, & Rousseau, 2012]
![Page 149: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/149.jpg)
Semi-automatic ABC
Fearnhead and Prangle (2010) study ABC and the selection of thesummary statistic in close proximity to Wilkinson’s proposal
ABC then considered from a purely inferential viewpoint andcalibrated for estimation purposesUse of a randomised (or ‘noisy’) version of the summary statistics
η(y) = η(y) + τε
Derivation of a well-calibrated version of ABC, i.e. an algorithmthat gives proper predictions for the distribution associated withthis randomised summary statistic [calibration constraint: ABCapproximation with same posterior mean as the true randomisedposterior]
![Page 150: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/150.jpg)
Semi-automatic ABC
Fearnhead and Prangle (2010) study ABC and the selection of thesummary statistic in close proximity to Wilkinson’s proposal
ABC then considered from a purely inferential viewpoint andcalibrated for estimation purposesUse of a randomised (or ‘noisy’) version of the summary statistics
η(y) = η(y) + τε
Derivation of a well-calibrated version of ABC, i.e. an algorithmthat gives proper predictions for the distribution associated withthis randomised summary statistic [calibration constraint: ABCapproximation with same posterior mean as the true randomisedposterior]
![Page 151: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/151.jpg)
Summary [of F&P/statistics)
I Optimality of the posterior expectation E[θ|y] of theparameter of interest as summary statistics η(y)!
I Use of the standard quadratic loss function
(θ − θ0)TA(θ − θ0) .
![Page 152: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/152.jpg)
Summary [of F&P/statistics)
I Optimality of the posterior expectation E[θ|y] of theparameter of interest as summary statistics η(y)!
I Use of the standard quadratic loss function
(θ − θ0)TA(θ − θ0) .
![Page 153: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/153.jpg)
Details on Fearnhead and Prangle (F&P) ABC
Use of a summary statistic S(·), an importance proposal g(·), akernel K (·) ≤ 1 and a bandwidth h > 0 such that
(θ, ysim) ∼ g(θ)f (ysim|θ)
is accepted with probability (hence the bound)
K [{S(ysim)− sobs}/h]
and the corresponding importance weight defined by
π(θ)/
g(θ)
[Fearnhead & Prangle, 2012]
![Page 154: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/154.jpg)
Errors, errors, and errors
Three levels of approximation
I π(θ|yobs) by π(θ|sobs) loss of information[ignored]
I π(θ|sobs) by
πABC(θ|sobs) =
∫π(s)K [{s− sobs}/h]π(θ|s) ds∫π(s)K [{s− sobs}/h] ds
noisy observations
I πABC(θ|sobs) by importance Monte Carlo based on Nsimulations, represented by var(a(θ)|sobs)/Nacc [expectednumber of acceptances]
![Page 155: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/155.jpg)
Average acceptance asymptotics
For the average acceptance probability/approximate likelihood
p(θ|sobs) =
∫f (ysim|θ) K [{S(ysim)− sobs}/h] dysim ,
overall acceptance probability
p(sobs) =
∫p(θ|sobs)π(θ) dθ = π(sobs)hd + o(hd)
[F&P, Lemma 1]
![Page 156: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/156.jpg)
Optimal importance proposal
Best choice of importance proposal in terms of effective sample size
g?(θ|sobs) ∝ π(θ)p(θ|sobs)1/2
[Not particularly useful in practice]
I note that p(θ|sobs) is an approximate likelihood
I reminiscent of parallel tempering
I could be approximately achieved by attrition of half of thedata
![Page 157: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/157.jpg)
Optimal importance proposal
Best choice of importance proposal in terms of effective sample size
g?(θ|sobs) ∝ π(θ)p(θ|sobs)1/2
[Not particularly useful in practice]
I note that p(θ|sobs) is an approximate likelihood
I reminiscent of parallel tempering
I could be approximately achieved by attrition of half of thedata
![Page 158: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/158.jpg)
Calibration of h
“This result gives insight into how S(·) and h affect the MonteCarlo error. To minimize Monte Carlo error, we need hd to be nottoo small. Thus ideally we want S(·) to be a low dimensionalsummary of the data that is sufficiently informative about θ thatπ(θ|sobs) is close, in some sense, to π(θ|yobs)” (F&P, p.5)
I turns h into an absolute value while it should becontext-dependent and user-calibrated
I only addresses one term in the approximation error andacceptance probability (“curse of dimensionality”)
I h large prevents πABC(θ|sobs) to be close to π(θ|sobs)
I d small prevents π(θ|sobs) to be close to π(θ|yobs) (“curse of[dis]information”)
![Page 159: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/159.jpg)
Calibrating ABC
“If πABC is calibrated, then this means that probability statementsthat are derived from it are appropriate, and in particular that wecan use πABC to quantify uncertainty in estimates” (F&P, p.5)
Definition
For 0 < q < 1 and subset A, event Eq(A) made of sobs such thatPrABC(θ ∈ A|sobs) = q. Then ABC is calibrated if
Pr(θ ∈ A|Eq(A)) = q
I unclear meaning of conditioning on Eq(A)
![Page 160: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/160.jpg)
Calibrating ABC
“If πABC is calibrated, then this means that probability statementsthat are derived from it are appropriate, and in particular that wecan use πABC to quantify uncertainty in estimates” (F&P, p.5)
Definition
For 0 < q < 1 and subset A, event Eq(A) made of sobs such thatPrABC(θ ∈ A|sobs) = q. Then ABC is calibrated if
Pr(θ ∈ A|Eq(A)) = q
I unclear meaning of conditioning on Eq(A)
![Page 161: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/161.jpg)
Calibrated ABC
Theorem (F&P)
Noisy ABC, where
sobs = S(yobs) + hε , ε ∼ K (·)
is calibrated
[Wilkinson, 2008]no condition on h!!
![Page 162: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/162.jpg)
Calibrated ABC
Consequence: when h =∞
Theorem (F&P)
The prior distribution is always calibrated
is this a relevant property then?
![Page 163: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/163.jpg)
More questions about calibrated ABC
“Calibration is not universally accepted by Bayesians. It is even morequestionable here as we care how statements we make relate to thereal world, not to a mathematically defined posterior.” R. Wilkinson
I Same reluctance about the prior being calibrated
I Property depending on prior, likelihood, and summary
I Calibration is a frequentist property (almost a p-value!)
I More sensible to account for the simulator’s imperfectionsthan using noisy-ABC against a meaningless based measure
[Wilkinson, 2012]
![Page 164: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/164.jpg)
Converging ABC
Theorem (F&P)
For noisy ABC, the expected noisy-ABC log-likelihood,
E {log[p(θ|sobs)]} =
∫ ∫log[p(θ|S(yobs) + ε)]π(yobs|θ0)K (ε)dyobsdε,
has its maximum at θ = θ0.
True for any choice of summary statistic? even ancilary statistics?![Imposes at least identifiability...]
Relevant in asymptotia and not for the data
![Page 165: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/165.jpg)
Converging ABC
Corollary
For noisy ABC, the ABC posterior converges onto a point mass onthe true parameter value as m→∞.
For standard ABC, not always the case (unless h goes to zero).
Strength of regularity conditions (c1) and (c2) in Bernardo& Smith, 1994?
[out-of-reach constraints on likelihood and posterior]Again, there must be conditions imposed upon summarystatistics...
![Page 166: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/166.jpg)
Loss motivated statistic
Under quadratic loss function,
Theorem (F&P)
(i) The minimal posterior error E[L(θ, θ)|yobs] occurs whenθ = E(θ|yobs) (!)
(ii) When h→ 0, EABC(θ|sobs) converges to E(θ|yobs)
(iii) If S(yobs) = E[θ|yobs] then for θ = EABC[θ|sobs]
E[L(θ, θ)|yobs] = trace(AΣ) + h2
∫xTAxK (x)dx + o(h2).
measure-theoretic difficulties?dependence of sobs on h makes me uncomfortable inherent to noisyABCRelevant for choice of K ?
![Page 167: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/167.jpg)
Optimal summary statistic
“We take a different approach, and weaken the requirement forπABC to be a good approximation to π(θ|yobs). We argue for πABC
to be a good approximation solely in terms of the accuracy ofcertain estimates of the parameters.” (F&P, p.5)
From this result, F&P
I derive their choice of summary statistic,
S(y) = E(θ|y)
[almost sufficient]
I suggest
h = O(N−1/(2+d)) and h = O(N−1/(4+d))
as optimal bandwidths for noisy and standard ABC.
![Page 168: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/168.jpg)
Optimal summary statistic
“We take a different approach, and weaken the requirement forπABC to be a good approximation to π(θ|yobs). We argue for πABC
to be a good approximation solely in terms of the accuracy ofcertain estimates of the parameters.” (F&P, p.5)
From this result, F&P
I derive their choice of summary statistic,
S(y) = E(θ|y)
[wow! EABC[θ|S(yobs)] = E[θ|yobs]]
I suggest
h = O(N−1/(2+d)) and h = O(N−1/(4+d))
as optimal bandwidths for noisy and standard ABC.
![Page 169: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/169.jpg)
Caveat
Since E(θ|yobs) is most usually unavailable, F&P suggest
(i) use a pilot run of ABC to determine a region of non-negligibleposterior mass;
(ii) simulate sets of parameter values and data;
(iii) use the simulated sets of parameter values and data toestimate the summary statistic; and
(iv) run ABC with this choice of summary statistic.
where is the assessment of the first stage error?
![Page 170: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/170.jpg)
Caveat
Since E(θ|yobs) is most usually unavailable, F&P suggest
(i) use a pilot run of ABC to determine a region of non-negligibleposterior mass;
(ii) simulate sets of parameter values and data;
(iii) use the simulated sets of parameter values and data toestimate the summary statistic; and
(iv) run ABC with this choice of summary statistic.
where is the assessment of the first stage error?
![Page 171: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/171.jpg)
Approximating the summary statistic
As Beaumont et al. (2002) and Blum and Francois (2010), F&Puse a linear regression to approximate E(θ|yobs):
θi = β(i)0 + β(i)f (yobs) + εi
with f made of standard transforms[Further justifications?]
![Page 172: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/172.jpg)
[my]questions about semi-automatic ABC
I dependence on h and S(·) in the early stage
I reduction of Bayesian inference to point estimation
I approximation error in step (i) not accounted for
I not parameterisation invariant
I practice shows that proper approximation to genuine posteriordistributions stems from using a (much) larger number ofsummary statistics than the dimension of the parameter
I the validity of the approximation to the optimal summarystatistic depends on the quality of the pilot run
I important inferential issues like model choice are not coveredby this approach.
[X, 2012]
![Page 173: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/173.jpg)
More about semi-automatic ABC
[End of section derived from comments on Read Paper, just appeared]
“The apparently arbitrary nature of the choice of summary statisticshas always been perceived as the Achilles heel of ABC.” M.Beaumont
I “Curse of dimensionality” linked with the increase of thedimension of the summary statistic
I Connection with principal component analysis[Itan et al., 2010]
I Connection with partial least squares[Wegman et al., 2009]
I Beaumont et al. (2002) postprocessed output is used as inputby F&P to run a second ABC
![Page 174: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/174.jpg)
More about semi-automatic ABC
[End of section derived from comments on Read Paper, just appeared]
“The apparently arbitrary nature of the choice of summary statisticshas always been perceived as the Achilles heel of ABC.” M.Beaumont
I “Curse of dimensionality” linked with the increase of thedimension of the summary statistic
I Connection with principal component analysis[Itan et al., 2010]
I Connection with partial least squares[Wegman et al., 2009]
I Beaumont et al. (2002) postprocessed output is used as inputby F&P to run a second ABC
![Page 175: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/175.jpg)
Wood’s alternative
Instead of a non-parametric kernel approximation to the likelihood
1
R
∑r
Kε{η(yr )− η(yobs)}
Wood (2010) suggests a normal approximation
η(y(θ)) ∼ Nd(µθ,Σθ)
whose parameters can be approximated based on the R simulations(for each value of θ).
I Parametric versus non-parametric rate [Uh?!]
I Automatic weighting of components of η(·) through Σθ
I Dependence on normality assumption (pseudo-likelihood?)
[Cornebise, Girolami & Kosmidis, 2012]
![Page 176: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/176.jpg)
Wood’s alternative
Instead of a non-parametric kernel approximation to the likelihood
1
R
∑r
Kε{η(yr )− η(yobs)}
Wood (2010) suggests a normal approximation
η(y(θ)) ∼ Nd(µθ,Σθ)
whose parameters can be approximated based on the R simulations(for each value of θ).
I Parametric versus non-parametric rate [Uh?!]
I Automatic weighting of components of η(·) through Σθ
I Dependence on normality assumption (pseudo-likelihood?)
[Cornebise, Girolami & Kosmidis, 2012]
![Page 177: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/177.jpg)
Reinterpretation and extensions
Reinterpretation of ABC output as joint simulation from
π(x , y |θ) = f (x |θ)πY |X (y |x)
whereπY |X (y |x) = Kε(y − x)
Reinterpretation of noisy ABC
if y |y obs ∼ πY |X (·|y obs), then marginally
y ∼ πY |θ(·|θ0)
c© Explain for the consistency of Bayesian inference based on y and π[Lee, Andrieu & Doucet, 2012]
![Page 178: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/178.jpg)
Reinterpretation and extensions
Reinterpretation of ABC output as joint simulation from
π(x , y |θ) = f (x |θ)πY |X (y |x)
whereπY |X (y |x) = Kε(y − x)
Reinterpretation of noisy ABC
if y |y obs ∼ πY |X (·|y obs), then marginally
y ∼ πY |θ(·|θ0)
c© Explain for the consistency of Bayesian inference based on y and π[Lee, Andrieu & Doucet, 2012]
![Page 179: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/179.jpg)
ABC for Markov chains
Rewriting the posterior as
π(θ)1−nπ(θ|x1)∏
π(θ|xt−1, xt)
where π(θ|xt−1, xt) ∝ f (xt |xt−1, θ)π(θ)
I Allows for a stepwise ABC, replacing each π(θ|xt−1, xt) by anABC approximation
I Similarity with F&P’s multiple sources of data (and also withDean et al., 2011 )
[White et al., 2010, 2012]
![Page 180: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/180.jpg)
ABC for Markov chains
Rewriting the posterior as
π(θ)1−nπ(θ|x1)∏
π(θ|xt−1, xt)
where π(θ|xt−1, xt) ∝ f (xt |xt−1, θ)π(θ)
I Allows for a stepwise ABC, replacing each π(θ|xt−1, xt) by anABC approximation
I Similarity with F&P’s multiple sources of data (and also withDean et al., 2011 )
[White et al., 2010, 2012]
![Page 181: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/181.jpg)
Back to sufficiency
Difference between regular sufficiency, equivalent to
π(θ|y) = π(θ|η(y))
for all θ’s and all priors π, andmarginal sufficiency, stated as
π(µ(θ)|y) = π(µ(θ)|η(y))
for all θ’s, the given prior π and a subvector µ(θ)[Basu, 1977]
Relates to F & P’s main result, but could event be reduced toconditional sufficiency
π(µ(θ)|yobs) = π(µ(θ)|η(yobs))
(if feasible at all...)[Dawson, 2012]
![Page 182: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/182.jpg)
Back to sufficiency
Difference between regular sufficiency, equivalent to
π(θ|y) = π(θ|η(y))
for all θ’s and all priors π, andmarginal sufficiency, stated as
π(µ(θ)|y) = π(µ(θ)|η(y))
for all θ’s, the given prior π and a subvector µ(θ)[Basu, 1977]
Relates to F & P’s main result, but could event be reduced toconditional sufficiency
π(µ(θ)|yobs) = π(µ(θ)|η(yobs))
(if feasible at all...)[Dawson, 2012]
![Page 183: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/183.jpg)
Back to sufficiency
Difference between regular sufficiency, equivalent to
π(θ|y) = π(θ|η(y))
for all θ’s and all priors π, andmarginal sufficiency, stated as
π(µ(θ)|y) = π(µ(θ)|η(y))
for all θ’s, the given prior π and a subvector µ(θ)[Basu, 1977]
Relates to F & P’s main result, but could event be reduced toconditional sufficiency
π(µ(θ)|yobs) = π(µ(θ)|η(yobs))
(if feasible at all...)[Dawson, 2012]
![Page 184: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/184.jpg)
Predictive performances
Instead of posterior means, other aspects of posterior to explore.E.g., look at minimising loss of information∫
p(θ, y) logp(θ, y)
p(θ)p(y)dθdy−
∫p(θ, η(y)) log
p(θ, η(y))
p(θ)p(η(y))dθdη(y)
for selection of summary statistics.[Filippi, Barnes, & Stumpf, 2012]
![Page 185: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/185.jpg)
Auxiliary variables
Auxiliary variable method avoids computations of untractableconstant in likelihood
f (y|θ) = Zθ f (y|θ)
Introduce pseudo-data z with artificial target g(z|θ, y)Generate θ′ ∼ K (θ, θ′) and z′ ∼ f (z|θ′)
[Møller, Pettitt, Berthelsen, & Reeves, 2006]
![Page 186: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/186.jpg)
Auxiliary variables
Auxiliary variable method avoids computations of untractableconstant in likelihood
f (y|θ) = Zθ f (y|θ)
Introduce pseudo-data z with artificial target g(z|θ, y)Generate θ′ ∼ K (θ, θ′) and z′ ∼ f (z|θ′)Accept with probability
π(θ′)f (y|θ′)g(z′|θ′, y)
π(θ)f (y|θ)g(z|θ, y)
K (θ′, θ)f (z|θ)
K (θ, θ′)f (z′|θ′)∧ 1
[Møller, Pettitt, Berthelsen, & Reeves, 2006]
![Page 187: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/187.jpg)
Auxiliary variables
Auxiliary variable method avoids computations of untractableconstant in likelihood
f (y|θ) = Zθ f (y|θ)
Introduce pseudo-data z with artificial target g(z|θ, y)Generate θ′ ∼ K (θ, θ′) and z′ ∼ f (z|θ′)Accept with probability
π(θ′)f (y|θ′)g(z′|θ′, y)
π(θ)f (y|θ)g(z|θ, y)
K (θ′, θ)f (z|θ)
K (θ, θ′)f (z′|θ′)∧ 1
[Møller, Pettitt, Berthelsen, & Reeves, 2006]
![Page 188: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/188.jpg)
Auxiliary variables
Auxiliary variable method avoids computations of untractableconstant in likelihood
f (y|θ) = Zθ f (y|θ)
Introduce pseudo-data z with artificial target g(z|θ, y)Generate θ′ ∼ K (θ, θ′) and z′ ∼ f (z|θ′)
For Gibbs random fields, existence of a genuine sufficient statisticη(y).
[Møller, Pettitt, Berthelsen, & Reeves, 2006]
![Page 189: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/189.jpg)
Auxiliary variables and ABC
Special case of ABC when
I g(z|θ, y) = Kε(η(z)− η(y))
I f (y|θ′)f (z|θ)/f (y|θ)f (z′|θ′) replaced by one [or not?!]
Consequences
I likelihood-free (ABC) versus constant-free (AVM)
I in ABC, Kε(·) should be allowed to depend on θ
I for Gibbs random fields, the auxiliary approach should beprefered to ABC
[Møller, 2012]
![Page 190: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/190.jpg)
Auxiliary variables and ABC
Special case of ABC when
I g(z|θ, y) = Kε(η(z)− η(y))
I f (y|θ′)f (z|θ)/f (y|θ)f (z′|θ′) replaced by one [or not?!]
Consequences
I likelihood-free (ABC) versus constant-free (AVM)
I in ABC, Kε(·) should be allowed to depend on θ
I for Gibbs random fields, the auxiliary approach should beprefered to ABC
[Møller, 2012]
![Page 191: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/191.jpg)
ABC and BIC
Idea of applying BIC during the local regression :
I Run regular ABC
I Select summary statistics during local regression
I Recycle the prior simulation sample (reference table) withthose summary statistics
I Rerun the corresponding local regression (low cost)
[Pudlo & Sedki, 2012]
![Page 192: ABC: How Bayesian can it be?](https://reader033.vdocument.in/reader033/viewer/2022060108/554e7ec0b4c905f66a8b536d/html5/thumbnails/192.jpg)
A Brave New World?!