Pittsburgh and Toronto "Halloween US trip" seminars

Relevant statistics for (approximate) Bayesian
model choice
Christian P. Robert
Universit´ Paris-Dauphine, Paris & University of Warwick, Coventry
e

bayesianstatistics@gmail.com

a meeting of potential interest?!

MCMSki IV to be held in Chamonix Mt Blanc, France, from
Monday, Jan. 6 to Wed., Jan. 8 (and BNP workshop on
Jan. 9), 2014
All aspects of MCMC++ theory and methodology and applications
and more! All posters welcomed!

Website http://www.pages.drexel.edu/∼mwl25/mcmski/

Outline

Intractable likelihoods
ABC methods
ABC as an inference machine
ABC for model choice
Model choice consistency
Conclusion and perspectives

Intractable likelihood

Case of a well-deﬁned statistical model where the likelihood
function
(θ|y) = f (y1 , . . . , yn |θ)
is (really!) not available in closed form
cannot (easily!) be either completed or demarginalised
cannot be (at all!) estimated by an unbiased estimator
c Prohibits direct implementation of a generic MCMC algorithm
like Metropolis–Hastings

Getting approximative

Case of a well-deﬁned statistical model where the likelihood
function
(θ|y) = f (y1 , . . . , yn |θ)
is out of reach

Empirical approximations to the original B problem
Degrading the data precision down to a tolerance ε
Replacing the likelihood with a non-parametric approximation
Summarising/replacing the data with insuﬃcient statistics

Diﬀerent [levels of] worries about abc

Impact on B inference
a mere computational issue (that will eventually end up being
solved by more powerful computers, &tc, even if too costly in
the short term, as for regular Monte Carlo methods) Not!
an inferential issue (opening opportunities for new inference
machine on complex/Big Data models, with legitimity
diﬀering from classical B approach)
a Bayesian conundrum (how closely related to the/a B
approach? more than recycling B tools? true B with raw
data? a new type of empirical B inference?)

Approximate Bayesian computation
ABC methods
Genesis of ABC
ABC basics
Advances and interpretations
ABC as knn

Genetic background of ABC

skip genetics

ABC is a recent computational technique that only requires being
able to sample from the likelihood f (·|θ)
This technique stemmed from population genetics models, about
15 years ago, and population geneticists still contribute
signiﬁcantly to methodological developments of ABC.
[Griﬃth & al., 1997; Tavar´ & al., 1999]
e

Demo-genetic inference

Each model is characterized by a set of parameters θ that cover
historical (time divergence, admixture time ...), demographics
(population sizes, admixture rates, migration rates, ...) and genetic
(mutation rate, ...) factors
The goal is to estimate these parameters from a dataset of
polymorphism (DNA sample) y observed at the present time

Problem:
most of the time, we cannot calculate the likelihood of the
polymorphism data f (y|θ)...

Neutral model at a given microsatellite locus, in a closed
panmictic population at equilibrium

Sample of 8 genes

Mutations according to
the Simple stepwise
Mutation Model
(SMM)
• date of the mutations ∼
Poisson process with
intensity θ/2 over the
branches
• MRCA = 100
• independent mutations:
±1 with pr. 1/2

Kingman’s genealogy
When time axis is
normalized,
T (k) ∼ Exp(k(k − 1)/2)
the Simple stepwise
Mutation Model
(SMM)
branches
• MRCA = 100
±1 with pr. 1/2

Kingman’s genealogy
When time axis is
normalized,
T (k) ∼ Exp(k(k − 1)/2)

Observations: leafs of the tree
^
θ=?

the Simple stepwise
Mutation Model
(SMM)
branches
• MRCA = 100
±1 with pr. 1/2

Much more interesting models. . .
several independent locus
Independent gene genealogies and mutations
diﬀerent populations
linked by an evolutionary scenario made of divergences,
admixtures, migrations between populations, selection
pressure, etc.
larger sample size
usually between 50 and 100 genes
MRCA
τ2
τ1

A typical evolutionary scenario:

POP 0

POP 1

POP 2

Instance of ecological questions

How the Asian ladybird
beetle arrived in Europe?
Why does they swarm right
now?
What are the routes of
invasion?
How to get rid of them?
Why did the chicken cross
the road?
[Lombaert & al., 2010, PLoS ONE]

Worldwide invasion routes of Harmonia axyridis

Worldwide routes of invasion of Harmonia axyridis
For each outbreak, the arrow indicates the most likely invasion
pathway and the associated posterior probability, with 95% credible
intervals in brackets
[Estoup et al., 2012, Molecular Ecology Res.]

c Intractable likelihood

Missing (too much missing!) data structure:
f (y|θ) =
G

f (y|G , θ)f (G |θ)dG

cannot be computed in a manageable way...
[Stephens & Donnelly, 2000]
The genealogies are considered as nuisance parameters
This modelling clearly diﬀers from the phylogenetic perspective
where the tree is the parameter of interest.

A?B?C?

A stands for approximate
[wrong likelihood / pic?]
B stands for Bayesian
C stands for computation
[producing a parameter
sample]

A?B?C?

ESS=75.93

ESS=76.87

0.2

0.5

−0.2

−0.2

0.2

0.6

0.0

0.2

0.0

0.2

0.4

0.6

3

4

θ
ESS=83.77

Density

0
0.0

0.2

0.4

0.6

−0.6

−0.4

θ
ESS=92.42

−0.2

0.0

0.2

0.4

0.4

0.6

3.0

θ
ESS=95.01
Density

−0.4

0.0

0.2

0.4

0.6

−0.4

θ
ESS=99.33

0.0

0.2

Density

3

θ
ESS=87.28

0

Density

2.0
1.0
−0.6

−0.2

θ
ESS=85.13

0.0

Density

2.0
1.0

0.0
θ
ESS=139.2

3

0.6

2.0

Density

−0.4

0.0

−0.5

2

Density

0
0.4

0.0
1.0
0.0 1.0 2.0 3.0

0.5

0.0 1.0 2.0 3.0

0.0

θ
ESS=155.7
Density

0.2

1.0

2.0
1.0

Density

0.0
−0.5

Density

0.0

θ
ESS=96.31

0.0

C stands for computation
[producing a parameter
sample]

−0.4

Density
−0.4

θ
ESS=149.1

B stands for Bayesian

0.4

2

0.0

0.2

1

−0.2

0.0

θ
ESS=108.4

1.5

Density
−0.4

−0.2

0.0 1.0 2.0 3.0

−0.4

2

4
3
2
1
−0.6

1

2.0

1.0

0

Density

0.5

1

0.0

θ
ESS=91.54

0.0 1.0 2.0 3.0

−0.5

A stands for approximate
[wrong likelihood / pic?]

1.0

Density

0.0

1.0
0.0

Density

4

ESS=155.6

−0.4

−0.2

0.0

0.2

0.4

−0.2

0.0

0.2

0.4

0.6

How Bayesian is aBc?

Could we turn the resolution into a Bayesian answer?
ideally so (not meaningfull: requires ∞-ly powerful computer
approximation error unknown (w/o costly simulation)
true B for wrong model (formal and artiﬁcial)
true B for noisy model (much more convincing)
true B for estimated likelihood (back to econometrics?)
illuminating the tension between information and precision

ABC methodology
Bayesian setting: target is π(θ)f (x|θ)
When likelihood f (x|θ) not in closed form, likelihood-free rejection
technique:

Foundation
For an observation y ∼ f (y|θ), under the prior π(θ), if one keeps
jointly simulating
θ ∼ π(θ) , z ∼ f (z|θ ) ,
until the auxiliary variable z is equal to the observed value, z = y,
then the selected
θ ∼ π(θ|y)
[Rubin, 1984; Diggle & Gratton, 1984; Tavar´ et al., 1997]
e

A as A...pproximative

When y is a continuous random variable, strict equality z = y is
replaced with a tolerance zone
ρ(y, z)
where ρ is a distance
Output distributed from
def

π(θ) Pθ {ρ(y, z) < } ∝ π(θ|ρ(y, z) < )
[Pritchard et al., 1999]

ABC algorithm

In most implementations, further degree of A...pproximation:
Algorithm 1 Likelihood-free rejection sampler
for i = 1 to N do
repeat
generate θ from the prior distribution π(·)
generate z from the likelihood f (·|θ )
until ρ{η(z), η(y)}
set θi = θ
end for
where η(y) deﬁnes a (not necessarily suﬃcient) statistic

Output
The likelihood-free algorithm samples from the marginal in z of:
π (θ, z|y) =
where A

,y

π(θ)f (z|θ)IA ,y (z)
,
A ,y ×Θ π(θ)f (z|θ)dzdθ

= {z ∈ D|ρ(η(z), η(y)) < }.

The idea behind ABC is that the summary statistics coupled with a
small tolerance should provide a good approximation of the
posterior distribution:
π (θ|y) = π (θ, z|y)dz ≈ π(θ|y) .
...does it?!

Output
The likelihood-free algorithm samples from the marginal in z of:
π (θ, z|y) =
where A

,y

π(θ)f (z|θ)IA ,y (z)
,
A ,y ×Θ π(θ)f (z|θ)dzdθ

= {z ∈ D|ρ(η(z), η(y)) < }.

The idea behind ABC is that the summary statistics coupled with a
small tolerance should provide a good approximation of the
restricted posterior distribution:
π (θ|y) = π (θ, z|y)dz ≈ π(θ|η(y)) .
Not so good..!

Comments

Role of distance paramount (because

= 0)

Scaling of components of η(y) is also determinant
matters little if “small enough”
representative of “curse of dimensionality”
small is beautiful!
the data as a whole may be paradoxically weakly informative
for ABC

curse of dimensionality in summaries

Consider that
the simulated summary statistics η(y1 ), . . . , η(yN )
the observed summary statistics η(yobs )
are iid with uniform distribution on [0, 1]d
Take δ∞ (d, N) = E
δ∞ (1, N)
δ∞ (2, N)
δ∞ (10, N)
δ∞ (200, N)

min

i=1,...,N

N = 100
0.0025
0.033
0.28
0.48

η(yobs ) − η(yi )

N = 1, 000
0.00025
0.01
0.22
0.48

∞

N = 10, 000
0.000025
0.0033
0.18
0.47

N = 100, 000
0.0000025
0.001
0.14
0.46

ABC (simul’) advances

how approximative is ABC?

ABC as knn

Simulating from the prior is often poor in eﬃciency
Either modify the proposal distribution on θ to increase the density
of x’s within the vicinity of y ...
[Marjoram et al, 2003; Bortot et al., 2007, Sisson et al., 2007]
...or by viewing the problem as a conditional density estimation
and by developing techniques to allow for larger
[Beaumont et al., 2002]
.....or even by including

in the inferential framework [ABCµ ]
[Ratmann et al., 2009]

ABC as knn
[Biau et al., 2013, Annales de l’IHP]
Practice of ABC: determine tolerance
distances, say 10% or 1% quantile,
=

N

as a quantile on observed

= qα (d1 , . . . , dN )

Interpretation of ε as nonparametric bandwidth only
approximation of the actual practice
[Blum & Fran¸ois, 2010]
c
ABC is a k-nearest neighbour (knn) method with kN = N N
[Loftsgaarden & Quesenberry, 1965]

ABC consistency
Provided
kN / log log N −→ ∞ and kN /N −→ 0
as N → ∞, for almost all s0 (with respect to the distribution of
S), with probability 1,
1
kN

kN

ϕ(θj ) −→ E[ϕ(θj )|S = s0 ]
j=1

[Devroye, 1982]
Biau et al. (2013) also recall pointwise and integrated mean square error
consistency results on the corresponding kernel estimate of the
conditional posterior distribution, under constraints
kN → ∞,

kN /N → 0,

hN → 0

p
and hN kN → ∞,

Rates of convergence

Further assumptions (on target and kernel) allow for precise
(integrated mean square) convergence rates (as a power of the
sample size N), derived from classical k-nearest neighbour
regression, like
4

when m = 1, 2, 3, kN ≈ N (p+4)/(p+8) and rate N − p+8
4

when m = 4, kN ≈ N (p+4)/(p+8) and rate N − p+8 log N
4

when m > 4, kN ≈ N (p+4)/(m+p+4) and rate N − m+p+4
[Biau et al., 2013]

Drag: Only applies to suﬃcient summary statistics

ABC inference machine
ABC methods
Error inc.
Exact BC and approximate
targets
summary statistic

How Bayesian is aBc..?

may be a convergent method of inference (meaningful?
suﬃcient? foreign?)
approximation error unknown (w/o massive simulation)
pragmatic/empirical B (there is no other solution!)
many calibration issues (tolerance, distance, statistics)
the NP side should be incorporated into the whole B picture
the approximation error should also be part of the B inference

ABCµ

Idea Infer about the error as well as about the parameter:
Use of a joint density
f (θ, |y) ∝ ξ( |y, θ) × πθ (θ) × π ( )
where y is the data, and ξ( |y, θ) is the prior predictive density of
ρ(η(z), η(y)) given θ and y when z ∼ f (z|θ)
Warning! Replacement of ξ( |y, θ) with a non-parametric kernel
approximation.
[Ratmann, Andrieu, Wiuf and Richardson, 2009, PNAS]

Wilkinson’s exact BC (not exactly!)
ABC approximation error (i.e. non-zero tolerance) replaced with
exact simulation from a controlled approximation to the target,
convolution of true posterior with kernel function
π (θ, z|y) =

π(θ)f (z|θ)K (y − z)
,
π(θ)f (z|θ)K (y − z)dzdθ

with K kernel parameterised by bandwidth .
[Wilkinson, 2013]

Theorem
The ABC algorithm based on the production of a randomised
˜
observation y = y + ξ, ξ ∼ K , and an acceptance probability of
K (y − z)/M
gives draws from the posterior distribution π(θ|y).

How exact a BC?
Pros
Pseudo-data from true model vs. observed data from noisy
model
Outcome is completely controlled
Link with ABCµ when assuming y observed with
measurement error with density K
Relates to the theory of model approximation
[Kennedy & O’Hagan, 2001]
leads to “noisy ABC”: just
do it!, i.e. perturbates the data down to precision ε and proceed
[Fearnhead & Prangle, 2012]
Cons
Requires K to be bounded by M
True approximation error never assessed

Which summary?

Fundamental difficulty of the choice of the summary statistic when
there is no non-trivial sufficient statistics
Loss of statistical information balanced against gain in data
roughening
Approximation error and information loss remain unknown
Choice of statistics induces choice of distance function
towards standardisation
may be imposed for external/practical reasons (e.g., DIYABC)
may gather several non-B point estimates
can learn about efficient combination

Semi-automatic ABC

Fearnhead and Prangle (2012) study ABC and the selection of the
summary statistic in close proximity to Wilkinson’s proposal
ABC considered as inferential method and calibrated as such
randomised (or ‘noisy’) version of the summary statistics
˜
η(y) = η(y) + τ
derivation of a well-calibrated version of ABC, i.e. an
algorithm that gives proper predictions for the distribution
associated with this randomised summary statistic
[weak property]

Summary [of F&P/statistics)
optimality of the posterior expectation
E[θ|y]
of the parameter of interest as summary statistics η(y)!
[requires iterative process]
use of the standard quadratic loss function
(θ − θ0 )T A(θ − θ0 )
recent extension to model choice, optimality of Bayes factor
B12 (y)
[F&P, ISBA 2012, Kyoto]

LDA summaries for model choice

In parallel to F& P semi-automatic ABC, selection of most
discriminant subvector out of a collection of summary statistics,
based on Linear Discriminant Analysis (LDA)
[Estoup & al., 2012, Mol. Ecol. Res.]
Solution now implemented in DIYABC.2
[Cornuet & al., 2008, Bioinf.; Estoup & al., 2013]

Implementation
Step 1: Rake a subset of the α% (e.g., 1%) best simulations
from an ABC reference table usually including 106 –109
simulations for each of the M compared scenarios/models
Selection based on normalized Euclidian distance computed
between observed and simulated raw summaries
Step 2: run LDA on this subset to transform summaries into
(M − 1) discriminant variables
When computing LDA functions, weight simulated data with an
Epanechnikov kernel
Step 3: Estimation of the posterior probabilities of each
competing scenario/model by polychotomous local logistic
regression against the M − 1 most discriminant variables
[Cornuet & al., 2008, Bioinformatics]

LDA advantages

much faster computation of scenario probabilities via
polychotomous regression
a (much) lower number of explanatory variables improves the
accuracy of the ABC approximation, reduces the tolerance
and avoids extra costs in constructing the reference table
allows for a large collection of initial summaries
ability to evaluate Type I and Type II errors on more complex
models
LDA reduces correlation among explanatory variables
When available, using both simulated and real data sets, posterior
probabilities of scenarios computed from LDA-transformed and raw
summaries are strongly correlated


ABC methods
BMC Principle
Gibbs random ﬁelds (counterexample)
Generic ABC model choice

Bayesian model choice

BMC Principle
Several models
M1 , M2 , . . .
are considered simultaneously for dataset y and model index M
central to inference.
Use of
prior π(M = m), plus
prior distribution on the parameter conditional on the value m
of the model index, πm (θm )

Bayesian model choice

BMC Principle
Several models
M1 , M2 , . . .
are considered simultaneously for dataset y and model index M
central to inference.
Goal is to derive the posterior distribution of M,
π(M = m|data)
a challenging computational target when models are complex.

Generic ABC for model choice

Algorithm 2 Likelihood-free model choice sampler (ABC-MC)
for t = 1 to T do
repeat
Generate m from the prior π(M = m)
Generate θm from the prior πm (θm )
Generate z from the model fm (z|θm )
until ρ{η(z), η(y)} <
Set m(t) = m and θ(t) = θm
end for
[Grelaud & al., 2009; Toni & al., 2009]

ABC estimates

Posterior probability π(M = m|y) approximated by the frequency
of acceptances from model m
1
T

T

Im(t) =m .
t=1

ABC estimates

Posterior probability π(M = m|y) approximated by the frequency
of acceptances from model m
1
T

T

Im(t) =m .
t=1

Extension to a weighted polychotomous logistic regression
estimate of π(M = m|y), with non-parametric kernel weights
[Cornuet et al., DIYABC, 2009]

Potts model
Skip MRFs

Potts model
Distribution with an energy function of the form
θS(y) = θ

δyl =yi
l∼i

where l∼i denotes a neighbourhood structure
In most realistic settings, summation
Zθ =

exp{θS(x)}
x∈X

involves too many terms to be manageable and numerical
approximations cannot always be trusted

Neighbourhood relations

Setup
Choice to be made between M neighbourhood relations
m

i ∼i

m

(0

M − 1)

with
Sm (x) =

I{xi =xi

}

m

i ∼i

driven by the posterior probabilities of the models.

Model index

Computational target:
P(M = m|x) ∝

fm (x|θm )πm (θm ) dθm π(M = m)
Θm

If S(x) suﬃcient statistic for the joint parameters
(M, θ0 , . . . , θM−1 ),
P(M = m|x) = P(M = m|S(x)) .

Sufficient statistics in Gibbs random fields

Each model m has its own sufficient statistic Sm (·) and
S(·) = (S0 (·), . . . , SM−1 (·)) is also (model-)sufficient.
Explanation: For Gibbs random fields,
1
2
x|M = m ∼ fm (x|θm ) = fm (x|S(x))fm (S(x)|θm )
1
f 2 (S(x)|θm )
=
n(S(x)) m

where
˜
˜
n(S(x)) = {x ∈ X : S(x) = S(x)}
c S(x) is sufficient for the joint parameters

Sufficient statistics in Gibbs random fields

Each model m has its own sufficient statistic Sm (·) and
S(·) = (S0 (·), . . . , SM−1 (·)) is also (model-)sufficient.
Explanation: For Gibbs random fields,
1
2
x|M = m ∼ fm (x|θm ) = fm (x|S(x))fm (S(x)|θm )
1
=
f 2 (S(x)|θm )
n(S(x)) m

where
˜
˜
n(S(x)) = {x ∈ X : S(x) = S(x)}
c S(x) is sufficient for the joint parameters

Toy example

iid Bernoulli model versus two-state ﬁrst-order Markov chain, i.e.
n

f0 (x|θ0 ) = exp θ0

I{xi =1}

{1 + exp(θ0 )}n ,

i=1

versus
f1 (x|θ1 ) =

1
exp θ1
2

n

I{xi =xi−1 }

{1 + exp(θ1 )}n−1 ,

i=2

with priors θ0 ∼ U(−5, 5) and θ1 ∼ U(0, 6) (inspired by “phase
transition” boundaries).

About sufficiency

If η1 (x) sufficient statistic for model m = 1 and parameter θ1 and
η2 (x) sufficient statistic for model m = 2 and parameter θ2 ,
(η1 (x), η2 (x)) is not always sufficient for (m, θm )
c Potential loss of information at the testing level

Poisson/geometric example

Sample
x = (x1 , . . . , xn )
from either a Poisson P(λ) or from a geometric G(p)
Sum
n

S=

xi = η(x)
i=1

suﬃcient statistic for either model but not simultaneously

Limiting behaviour of B12 (T → ∞)
ABC approximation
B12 (y) =

T
t=1 Imt =1 Iρ{η(zt ),η(y)}
T
t=1 Imt =2 Iρ{η(zt ),η(y)}

,

where the (mt , z t )’s are simulated from the (joint) prior
As T goes to inﬁnity, limit
B12 (y) =
=

Iρ{η(z),η(y)}
Iρ{η(z),η(y)}
Iρ{η,η(y)}
Iρ{η,η(y)}

π1 (θ1 )f1 (z|θ1 ) dz dθ1
π2 (θ2 )f2 (z|θ2 ) dz dθ2
π1 (θ1 )f1η (η|θ1 ) dη dθ1
,
π2 (θ2 )f2η (η|θ2 ) dη dθ2

where f1η (η|θ1 ) and f2η (η|θ2 ) distributions of η(z)

Limiting behaviour of B12 ( → 0)

When

goes to zero,
η
B12 (y)

=

π1 (θ1 )f1η (η(y)|θ1 ) dθ1
π2 (θ2 )f2η (η(y)|θ2 ) dθ2

c Bayes factor based on the sole observation of η(y)

Limiting behaviour of B12 (under sufficiency)

If η(y) sufficient statistic in both models,
fi (y|θi ) = gi (y)fi η (η(y)|θi )
Thus

=

Θ1

π(θ1 )g1 (y)f1η (η(y)|θ1 ) dθ1

Θ2

B12 (y) =

π(θ2 )g2 (y)f2η (η(y)|θ2 ) dθ2

g1 (y)
g2 (y)

π1 (θ1 )f1η (η(y)|θ1 ) dθ1
g1 (y) η
=
B (y) .
g2 (y) 12
π2 (θ2 )f2η (η(y)|θ2 ) dθ2
[Didelot, Everitt, Johansen & Lawson, 2011]

c No discrepancy only when cross-model sufficiency

Poisson/geometric example (back)

Sample
x = (x1 , . . . , xn )
from either a Poisson P(λ) or from a geometric G(p)
Discrepancy ratio
S!n−S / i xi !
g1 (x)
=
g2 (x)
1 n+S−1
S

Poisson/geometric discrepancy
η
Range of B12 (x) versus B12 (x): The values produced have nothing
in common.

Formal recovery

Creating an encompassing exponential family
f (x|θ1 , θ2 , α1 , α2 ) ∝ exp{θT η1 (x) + θT η2 (x) + α1 t1 (x) + α2 t2 (x)}
1
2
leads to a suﬃcient statistic (η1 (x), η2 (x), t1 (x), t2 (x))

Formal recovery

1
2
In the Poisson/geometric case, if
discrepancy

i

xi ! is added to S, no

Formal recovery

1
2
Only applies in genuine suﬃciency settings...
c Inability to evaluate information loss due to summary
statistics

Meaning of the ABC-Bayes factor

The ABC approximation to the Bayes Factor is based solely on the
summary statistics....
In the Poisson/geometric case, if E[yi ] = θ0 > 0,
η
lim B12 (y) =

n→∞

(θ0 + 1)2 −θ0
e
θ0

ABC model choice consistency

ABC methods
Formalised framework
Consistency results
Summary statistics

The starting point

Central question to the validation of ABC for model choice:
When is a Bayes factor based on an insuﬃcient statistic T(y)
consistent?
T
Note/warnin: c drawn on T(y) through B12 (y) necessarily diﬀers
from c drawn on y through B12 (y)
[Marin, Pillai, X, & Rousseau, JRSS B, 2013]

A benchmark if toy example

Comparison suggested by referee of PNAS paper [thanks!]:
[X, Cornuet, Marin, & Pillai, Aug. 2011]
Model M1 : y ∼ N(θ1 , 1) opposed
√
to model M2 : y ∼ L(θ2 , 1/ 2), Laplace distribution with mean θ2
√
and scale parameter 1/ 2 (variance one).


√
√
Four possible statistics
1. sample mean y (suﬃcient for M1 if not M2 );
2. sample median med(y) (insuﬃcient);
3. sample variance var(y) (ancillary);
4. median absolute deviation mad(y) = med(|y − med(y)|);

√
√

0.3

0.4

0.5

0.6

0.7

n=100

q
q
q

0.1

0.2

q

q
q
q
q

0.0

q
q
q

Gauss

Laplace

√
√
n=100

0.7

1.0

n=100

q

0.6

q

0.8

q
q

0.5

q
q
q

q
q
q
q

0.4

q
q
q

q

0.1

0.2

q

q

0.2

0.3

0.4

0.6

q

q
q
q

q
q

0.0

q
q
q

Gauss

Laplace

0.0

q

Gauss

Laplace

Framework

Starting from sample
y = (y1 , . . . , yn )
the observed sample, not necessarily iid with true distribution
y ∼ Pn
Summary statistics
T(y) = Tn = (T1 (y), T2 (y), · · · , Td (y)) ∈ Rd
with true distribution Tn ∼ Gn .

Assumptions

A collection of asymptotic “standard” assumptions:
[A1] is a standard central limit theorem under the true model
[A2] controls the large deviations of the estimator Tn from the
estimand µ(θ)
[A3] is the standard prior mass condition found in Bayesian
asymptotics (di eﬀective dimension of the parameter)
[A4] restricts the behaviour of the model density against the true
density
[Think CLT!]

Assumptions

[Think CLT!]
[A1] There exist
a sequence {vn } converging to +∞,
a distribution Q,
a symmetric, d × d positive deﬁnite matrix V0 and
a vector µ0 ∈ Rd
such that
−1/2

vn V0

(Tn − µ0 )

n→∞

Q, under Gn

Assumptions

[Think CLT!]
[A3] If inf{|µi (θi ) − µ0 |; θi ∈ Θi } = 0, deﬁning (u > 0)
Sn,i (u) = θi ∈ Fn,i ; |µi (θi ) − µ0 |

−1
u vn
,

there exist constants di < τi ∧ αi − 1 such that
−d
πi (Sn,i (u)) ∼ u di vn i ,

∀u

vn

Assumptions

[Think CLT!]
[A4] If inf{|µi (θi ) − µ0 |; θi ∈ Θi } = 0, for any
U, δ > 0 and (En )n such that, if θi ∈ Sn,i (U)
En ⊂ {t; gi (t|θi ) < δgn (t)}

> 0, there exist

c
and Gn (En ) < .

Assumptions

[Think CLT!]
Again (sumup)
[A1] is a standard central limit theorem under the true model
[A2] controls the large deviations of the estimator Tn from the
estimand µ(θ)
[A3] is the standard prior mass condition found in Bayesian
asymptotics (di eﬀective dimension of the parameter)
[A4] restricts the behaviour of the model density against the true
density

Effective dimension

Understanding di in [A3]:
defined only when µ0 ∈ {µi (θi ), θi ∈ Θi },
πi (θi : |µi (θi ) − µ0 | < n−1/2 ) = O(n−di /2 )
is the effective dimension of the model Θi around µ0


when inf{|µi (θi ) − µ0 |; θi ∈ Θi } = 0,
mi (Tn )
−d
∼ vn i ,
gn (Tn )
thus
log(mi (Tn )/gn (Tn )) ∼ −di log vn
−d
and vn i penalization factor resulting from integrating θi out (see
eﬀective number of parameters in DIC)


In regular models, di dimension of T(Θi ), leading to BIC
In non-regular models, di can be smaller

Asymptotic marginals
Asymptotically, under [A1]–[A4]
mi (t) =

gi (t|θi ) πi (θi ) dθi
Θi

is such that
(i) if inf{|µi (θi ) − µ0 |; θi ∈ Θi } = 0,
d−d
Cl v n i

mi (Tn )

d−d
Cu vn i

and
(ii) if inf{|µi (θi ) − µ0 |; θi ∈ Θi } > 0
d−τ
d−α
mi (Tn ) = oPn [vn i + vn i ].

Within-model consistency

Under same assumptions,
if inf{|µi (θi ) − µ0 |; θi ∈ Θi } = 0,
the posterior distribution of µi (θi ) given Tn is consistent at rate
1/vn provided αi ∧ τi > di .

Between-model consistency

Consequence of above is that asymptotic behaviour of the Bayes
factor is driven by the asymptotic mean value of Tn under both
models. And only by this mean value!


Indeed, if
inf{|µ0 − µ2 (θ2 )|; θ2 ∈ Θ2 } = inf{|µ0 − µ1 (θ1 )|; θ1 ∈ Θ1 } = 0
then
−(d1 −d2 )

Cl vn

m1 (Tn ) m2 (Tn )

−(d1 −d2 )

Cu vn

,

where Cl , Cu = OPn (1), irrespective of the true model.
c Only depends on the diﬀerence d1 − d2 : no consistency


Else, if
inf{|µ0 − µ2 (θ2 )|; θ2 ∈ Θ2 } > inf{|µ0 − µ1 (θ1 )|; θ1 ∈ Θ1 } = 0
then

m1 (Tn )
m2 (Tn )

−(d1 −α2 )

Cu min vn

−(d1 −τ2 )

, vn

Consistency theorem

If
inf{|µ0 − µ2 (θ2 )|; θ2 ∈ Θ2 } = inf{|µ0 − µ1 (θ1 )|; θ1 ∈ Θ1 } = 0,
Bayes factor
−(d1 −d2 )

T
B12 = O(vn

)

irrespective of the true model. It is inconsistent since it always
picks the model with the smallest dimension

Consistency theorem

If Pn belongs to one of the two models and if µ0 cannot be
attained by the other one :
0 = min (inf{|µ0 − µi (θi )|; θi ∈ Θi }, i = 1, 2)
< max (inf{|µ0 − µi (θi )|; θi ∈ Θi }, i = 1, 2) ,
T
then the Bayes factor B12 is consistent

Consequences on summary statistics

Bayes factor driven by the means µi (θi ) and the relative position
of µ0 wrt both sets {µi (θi ); θi ∈ Θi }, i = 1, 2.
For ABC, this implies the simplest summary statistics Tn are
ancillary statistics with diﬀerent mean values under both models
Else, if Tn asymptotically depends on some of the parameters of
the models, it is possible that there exists θi ∈ Θi such that
µi (θi ) = µ0 even though model M1 is misspeciﬁed

Toy example: Laplace versus Gauss [1]

If
n

Tn = n−1

Xi4 ,

µ1 (θ) = 3 + θ4 + 6θ2 ,

µ2 (θ) = 6 + · · ·

i=1

and the true distribution is Laplace with mean θ0 = 1, under the
√
Gaussian model the value θ∗ = 2 3 − 3 also leads to µ0 = µ(θ∗ )
[here d1 = d2 = d = 1]


If
n
n

T =n

Xi4 ,

−1

µ1 (θ) = 3 + θ4 + 6θ2 ,

µ2 (θ) = 6 + · · ·

i=1

and the true distribution is Laplace with mean θ0 = 1, under the
√
Gaussian model the value θ∗ = 2 3 − 3 also leads to µ0 = µ(θ∗ )
[here d1 = d2 = d = 1]
c A Bayes factor associated with such a statistic is inconsistent


If
n

Tn = n−1

Xi4 ,
i=1

µ1 (θ) = 3 + θ4 + 6θ2 ,

µ2 (θ) = 6 + · · ·

If
n

Tn = n−1

Xi4 ,

µ1 (θ) = 3 + θ4 + 6θ2 ,

i=1

q

0.35

0.40

0.45

n=1000

q
q

0.30

q

q

M1

M2

Fourth moment

µ2 (θ) = 6 + · · ·

When
(4)

(6)

¯
¯
T(y) = yn , yn

and the true distribution is Laplace with mean θ0 = 0, then
√
µ0 = 6, µ1 (θ∗ ) = 6 with θ∗ = 2 3 − 3
1
1
[d1 = 1 and d2 = 1/2]
thus
B12 ∼ n−1/4 → 0 : consistent
Under the Gaussian model µ0 = 3, µ2 (θ2 )
B12 → +∞ :

6 > 3 ∀θ2

consistent

When
(4)

(6)

¯
¯
T(y) = yn , yn

0.8

q
q
q
q
q

0.6

q

0.6

0.6

0.8

q

n=10000
q
q
q
q
q
q

1.0

q
q

0.8

1.0

n=1000

1.0

n=100

q
q

Gauss

Laplace

0.4
0.0

0.2

0.4
0.2
0.0

0.0

0.2

0.4

q
q

Gauss

Laplace

Fourth and sixth moments

Gauss

Laplace

Checking for adequate statistics

Run a practical check of the relevance (or non-relevance) of Tn
null hypothesis that both models are compatible with the statistic
Tn
H0 : inf{|µ2 (θ2 ) − µ0 |; θ2 ∈ Θ2 } = 0
against
H1 : inf{|µ2 (θ2 ) − µ0 |; θ2 ∈ Θ2 } > 0
testing procedure provides estimates of mean of Tn under each
model and checks for equality

Checking in practice
Under each model Mi , generate ABC sample θi,l , l = 1, · · · , L
For each θi,l , generate yi,l ∼ Fi,n (·|ψi,l ), derive Tn (yi,l ) and
compute
L
1
Tn (yi,l ), i = 1, 2 .
µi =
^
L
l=1

n

Conditionally on T (y),
√
L(µi − Eπ [µi (θi )|Tn (y)])
^

N(0, Vi ),

Test for a common mean
H0 : µ1 ∼ N(µ0 , V1 ) , µ2 ∼ N(µ0 , V2 )
^
^
against the alternative of diﬀerent means
H1 : µi ∼ N(µi , Vi ),
^

with µ1 = µ2 .

Toy example: Laplace versus Gauss

q

q

0

10

20

30

40

q

q

Gauss

q
q
q

Laplace

Gauss

Laplace

Normalised χ2 without and with mad

A population genetic illustration
Two populations (1 and 2) having diverged at a ﬁxed known time
in the past and third population (3) which diverged from one of
those two populations (models 1 and 2, respectively).
Observation of 50 diploid individuals/population genotyped at 5,
50 or 100 independent microsatellite loci.

Model 2

Two populations (1 and 2) having diverged at a ﬁxed known time
in the past and third population (3) which diverged from one of
those two populations (models 1 and 2, respectively).
Observation of 50 diploid individuals/population genotyped at 5,
50 or 100 independent microsatellite loci.
Stepwise mutation model: the number of repeats of the mutated
gene increases or decreases by one. Mutation rate µ common to all
loci set to 0.005 (single parameter) with uniform prior distribution
µ ∼ U[0.0001, 0.01]


Summary statistics associated to the (δµ )2 distance
xl,i,j repeated number of allele in locus l = 1, . . . , L for individual
i = 1, . . . , 100 within the population j = 1, 2, 3. Then
(δµ )21 ,j2 =
j

1
L

L
l=1


 1
100

100

xl,i1 ,j1 −
i1 =1

1
100

100

2
xl,i2 ,j2  .

i2 =1


For two copies of locus l with allele sizes xl,i,j1 and xl,i ,j2 , most
recent common ancestor at coalescence time τj1 ,j2 , gene genealogy
distance of 2τj1 ,j2 , hence number of mutations Poisson with
parameter 2µτj1 ,j2 . Therefore,
xl,i,j1 − xl,i

E

2
,j2

|τj1 ,j2

= 2µτj1 ,j2

and
Model 1 Model 2
(δµ )2
1,2

2µ1 t

2µ2 t

E (δµ )2
1,3

2µ1 t

2µ2 t

E (δµ )2
2,3

2µ1 t

2µ2 t

E


Thus,
Bayes factor based only on distance (δµ )2 not convergent: if
1,2
µ1 = µ2 , same expectation
Bayes factor based only on distance (δµ )2 or (δµ )2 not
1,3
2,3
convergent: if µ1 = 2µ2 or 2µ1 = µ2 same expectation
if two of the three distances are used, Bayes factor converges:
there is no (µ1 , µ2 ) for which all expectations are equal


0.8

DM2(12)

q

50

100

0.0

0.4

q

q

5

0.8

DM2(13)

0.4

q
q
q
q

q

q
q
q
q
q

q

0.0

q

5

50

100

DM2(13) & DM2(23)
0.8

q
q
q
q
q
q

q
q
q
q
q
q

q
q

0.4

q
q
q
q

0.0

q

q
q

q

q

5

50

100

Posterior probabilities that the data is from model 1 for 5, 50
and 100 loci


20

0.2

40

0.4

60

80

0.6

100

0.8

120

1.0

140

p−values

0

0.0

q
q
q

DM2(12)

DM2(13) & DM2(23)

q

DM2(12)

DM2(13) & DM2(23)

Embedded models

When M1 submodel of M2 , and if the true distribution belongs to
the smaller model M1 , Bayes factor is of order
−(d1 −d2 )

vn

Embedded models

−(d1 −d2 )

vn

If summary statistic only informative on a parameter that is the
same under both models, i.e if d1 = d2 , then
c the Bayes factor is not consistent

Embedded models

−(d1 −d2 )

vn

Else, d1 < d2 and Bayes factor is going to ∞ under M1 . If true
distribution not in M1 , then
c Bayes factor is consistent only if µ1 = µ2 = µ0

Summary

Model selection feasible with ABC
Choice of summary statistics is paramount
At best, ABC → π(. | T(y)) which concentrates around µ0
For consistent estimation : {θ; µ(θ) = µ0 } = θ0
For consistent testing
{µ1 (θ1 ), θ1 ∈ Θ1 } ∩ {µ2 (θ2 ), θ2 ∈ Θ2 } = ∅

Conclusion/perspectives

abc part of a wider picture
to handle complex/Big Data
models, able to start from
rudimentary machine
learning summaries
many formats of empirical
[likelihood] Bayes methods
available
lack of comparative tools
and of an assessment for
information loss

Pittsburgh and Toronto "Halloween US trip" seminars

Recommandé

Recommandé

Contenu connexe

Similaire à Pittsburgh and Toronto "Halloween US trip" seminars

Similaire à Pittsburgh and Toronto "Halloween US trip" seminars (20)

Plus de Christian Robert

Plus de Christian Robert (20)

Dernier

Dernier (20)

Pittsburgh and Toronto "Halloween US trip" seminars