There are many approaches to Bayesian computation with intractable likelihoods, including the exchange algorithm, approximate Bayesian computation (ABC), thermodynamic integration, and composite likelihood. These approaches vary in accuracy as well as scalability for datasets of significant size. The Potts model is an example where such methods are required, due to its intractable normalising constant. This model is a type of Markov random field, which is commonly used for image segmentation. The dimension of its parameter space increases linearly with the number of pixels in the image, making this a challenging application for scalable Bayesian computation. My talk will introduce various algorithms in the context of the Potts model and describe their implementation in C++, using OpenMP for parallelism. I will also discuss the process of releasing this software as an open source R package on the CRAN repository.
R package 'bayesImageS': a case study in Bayesian computation using Rcpp and OpenMP
1. R packages Potts model Bayesian computation Conclusion
bayesImageS: a case study in Bayesian
computation using Rcpp and OpenMP
Matt Moores
OxWaSP mini-symposium
December 2, 2016
2. R packages Potts model Bayesian computation Conclusion
Outline
1 R packages
Performance of BLAS
2 Potts model
3 Bayesian computation
Chequerboard Gibbs sampler
Pseudolikelihood
Thermodynamic integration
Exchange algorithm
Approximate Bayesian Computation (ABC)
3. R packages Potts model Bayesian computation Conclusion
Why write an R package?
Portability
Test bed for new statistical methods
Build on existing code
Research impact
Kudos
Hadley Wickham (2015) R packages
4. R packages Potts model Bayesian computation Conclusion
Why C++?
Most statistical algorithms are iterative
Markov chain Monte Carlo
Scalability for large datasets
Rcpp
OpenMP
Eigen or Armadillo
Dirk Eddelbuettel (2013) Seamless R and C++ integration with Rcpp
5. R packages Potts model Bayesian computation Conclusion
Inline
One function at a time:
§
library ( i n l i n e )
sum_logs ← cxxfunction ( signature ( log_vec = "numeric") , plugin = "RcppArmadillo" , body=’
arma::vec log_prob = Rcpp::as<arma::vec>(log_vec);
double suml = 0.0;
double maxl = log_prob.max();
for (unsigned i=0; i < log_prob.n_elem; i++)
{
if (arma::is_finite(log_prob(i)))
suml += exp(log_prob(i) - maxl);
}
return Rcpp::wrap(log(suml) + maxl);
’)
6. R packages Potts model Bayesian computation Conclusion
Package Skeleton
Create a new R package:
package.skeleton("myPackage", path=".")
Specific skeletons for each C++ library:
Rcpp.package.skeleton("myRcppPackage")
RcppArmadillo.package.skeleton("MyArmadilloPackage")
RcppEigen.package.skeleton("MyEigenPackage")
7. R packages Potts model Bayesian computation Conclusion
Annotations
Rcpp wrappers generated automatically:
compileAttributes("myRcppPackage")
R package documentation generated automatically:
roxygenize("myRcppPackage")
§
/ / ’ Compute the effective sample size (ESS) of the particles.
/ / ’
/ / ’ The ESS is a ‘‘rule of thumb’’ for assessing the degeneracy of
/ / ’’ the importance distribution:
/ / ’ deqn {ESS = frac { ( sum_ { q=1}^Q w_q ) ^ 2 } { sum_ { q=1}^Q w_q ^2}}
/ / ’
/ / ’’ @param log_weights logarithms of the importance weights of each particle.
/ / ’’ @return the effective sample size, a scalar between 0 and Q
/ / ’’ @references
/ / ’’ Liu, JS (2001) "Monte Carlo Strategies in Scientific Computing." Springer’
/ / [ [ Rcpp : : export ] ]
double effectiveSampleSize ( NumericVector log_weights )
{
double sum_wt = sum_logs ( log_weights ) ;
double sum_sq = sum_logs ( log_weights + log_weights ) ;
double res = exp (sum_wt + sum_wt − sum_sq ) ;
i f ( std : : i s f i n i t e ( res ) ) return res ;
else return 0;
}
8. R packages Potts model Bayesian computation Conclusion
Common Problems
Rcpp parameters are passed by reference (not copied):
Can rely on R for garbage collection
Memory allocation is slower
Can crash R (and Rstudio (and your OS))
R is not thread safe
Cannot call any R functions (even indirectly)
within parallel code!
Drew Schmidt (@wrathematics, 2015) Parallelism, R, and OpenMP
9. R packages Potts model Bayesian computation Conclusion
Performance
A simple performance comparison:
§
library ( RcppEigen )
library ( RcppArmadillo )
library (RcppGSL)
library ( rbenchmark )
source ( system . f i l e ("examples" , "lmBenchmark.R" ,
package ="RcppEigen" ) )
Fitting a linear regression model
(by default, n=100000 rows × p=40 columns,
with 20 repeat measurements)
10. R packages Potts model Bayesian computation Conclusion
Results with default R BLAS
Table: lm benchmark for Mac OS X with reference R BLAS
test relative elapsed user.self sys.self
3 LDLt 1.000 3.611 3.448 0.164
7 QR 1.356 4.896 4.801 0.077
8 LLt 1.421 5.130 5.007 0.123
11 gpuLm.fit 2.634 9.510 8.332 0.395
1 lm.fit 3.760 13.576 13.550 0.025
6 SymmEig 5.475 19.770 19.634 0.126
2 PivQR 5.517 19.923 19.760 0.162
9 arma 20.416 73.723 73.459 0.255
4 GESDD 24.815 89.607 89.429 0.175
10 GSL 198.975 718.500 718.157 0.272
5 SVD 201.717 728.401 727.958 0.423
matrix dimensions:1650 × 875
11. R packages Potts model Bayesian computation Conclusion
Accelerate Umbrella Framework
Table: lm benchmark for Mac OS X with vecLib
test relative elapsed user.self sys.self
3 LDLt 1.000 3.682 3.517 0.165
7 QR 1.330 4.898 4.809 0.088
8 LLt 1.413 5.204 5.081 0.121
1 lm.fit 1.832 6.745 6.702 0.024
9 arma 2.396 8.821 20.572 1.229
11 gpuLm.fit 2.655 9.776 8.757 0.344
4 GESDD 3.313 12.199 22.120 0.884
6 SymmEig 5.405 19.902 19.775 0.127
2 PivQR 5.506 20.273 20.095 0.159
10 GSL 195.244 718.889 718.565 0.273
5 SVD 198.514 730.930 730.403 0.495
Iacus, Urbanek, Goedman & Ripley (2016) R for Mac OS X FAQ, §10.5
12. R packages Potts model Bayesian computation Conclusion
Nested Models
β ← 0
Hidden MRF
θ : scale
D > 1 : dimension
∂i : neighbourhood
ψ : noise param.
p (yi | zi, ψ)
p zi | zi, θ
p (θ | z)
p(ψ) p(θ)
y, z, θ, ψ
Hidden Ising/Potts
z ∈ {1 . . . k}
θ = β : inverse
temperature
p zi | zi, β
p (β | z) p(β)
Mixture of Gaussians
λ = {λ1, . . . , λk} :
k
j=1 λj = 1
ψ = µ, σ2
: noise param.
p yi | zi, µj, σ2
j
p (λ | z) p(λ)
13. R packages Potts model Bayesian computation Conclusion
Hidden Markov Random Field
Joint distribution of observed pixel intensities y = {yi}n
i=1
and latent labels z = {zi}n
i=1:
p(y, z|µ, σ2
, β) = p(y|µ, σ2
, z)p(z|β) (1)
Additive Gaussian noise:
yi|zi =j
iid
∼ N µj, σ2
j (2)
Potts model:
π(zi|zi, β) =
exp {β i∼ δ(zi, z )}
k
j=1 exp {β i∼ δ(j, z )}
(3)
Potts (1952) Proceedings of the Cambridge Philosophical Society 48(1)
14. R packages Potts model Bayesian computation Conclusion
Inverse Temperature
15. R packages Potts model Bayesian computation Conclusion
Doubly-intractable likelihood
p(β|z) ∝ C(β)−1
π(β) exp {β S(z)} (4)
The normalising constant has computational complexity O(nkn):
C(β) =
z∈Z
exp {β S(z)} (5)
S(z) is the sufficient statistic of the Potts model:
S(z) =
i∼ ∈L
δ(zi, z ) (6)
where L is the set of all unique neighbour pairs.
17. R packages Potts model Bayesian computation Conclusion
Chequerboard Gibbs II
Algorithm 1 Chequerboard sampling for z
1: for all blocks b do
2: for all pixels i ∈ b do
3: for all labels j ∈ 1 . . . k do
4: Compute λj ← p(yi | zi = j)π(zi = j | zi∼ , β)
5: end for
6: Draw zi ∼ Multinomial(λ1, . . . , λk)
7: end for
8: end for
18. R packages Potts model Bayesian computation Conclusion
Gibbs sampler in C++
§
void gibbsLabels ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,
arma : : umat & z , arma : : umat & alloc , const double beta ,
const arma : : mat & log_ x f i e l d )
{
const Rcpp : : NumericVector randU = Rcpp : : r u n i f ( neigh . n_rows ) ;
for ( unsigned b=0; b < blocks . size ( ) ; b++)
{
const arma : : uvec block = blocks [ b ] ;
arma : : vec log_prob ( z . n_cols ) ;
#pragma omp p a r a l l e l for private ( log_prob )
for ( unsigned i =0; i < block . size ( ) ; i ++)
{
for ( unsigned j =0; j < z . n_cols ; j ++)
{
unsigned sum_neigh = 0;
for ( unsigned k=0; k < neigh . n_cols ; k++)
{
sum_neigh += z ( neigh ( block [ i ] , k ) , j ) ;
}
log_prob [ j ] = log_ x f i e l d ( block [ i ] , j ) + beta∗sum_neigh ;
}
double t o t a l _ l l i k e = sum_logs ( log_prob ) ;
double cumProb = 0.0;
z . row ( block [ i ] ) . zeros ( ) ;
for ( unsigned j =0; j < log_prob . n_elem ; j ++)
{
cumProb += exp ( log_prob [ j ] − t o t a l _ l l i k e ) ;
i f ( randU [ block [ i ] ] < cumProb )
{
z ( block [ i ] , j ) = 1;
a l l o c ( block [ i ] , j ) += 1;
break ;
19. R packages Potts model Bayesian computation Conclusion
Pseudolikelihood (PL)
Algorithm 2 Metropolis-Hastings with PL
1: Draw proposal β ∼ q(β |β◦)
2: Approximate p(β |z) and p(β◦|z) using equation (7):
ˆpPL(β|z) ≈
n
i=1
exp{β i∼ δ(zi, z )}
k
j=1 exp{β i∼ δ(j, z )}
(7)
3: Calculate the M-H ratio ρ = ˆpPL(β |z)π(β )q(β◦|β )
ˆpPL(β◦|z)π(β◦)q(β |β◦)
4: Draw u ∼ Uniform[0, 1]
5: if u < min(1, ρ) then
6: β ← β
7: else
8: β ← β◦
9: end if
Rydén & Titterington (1998) JCGS 7(2): 194–211
20. R packages Potts model Bayesian computation Conclusion
Pseudolikelihood in C++
§
double pseudolike ( const arma : : mat & ne , const arma : : uvec & e , const double b ,
const unsigned n , const unsigned k )
{
double num = 0.0;
double denom = 0.0;
#pragma omp p a r a l l e l for reduction ( + :num, denom)
for ( unsigned i =0; i < n ; i ++)
{
num=num+ne ( e [ i ] , i ) ;
double tdenom =0.0;
for ( unsigned j =0; j < k ; j ++)
{
tdenom=tdenom+exp ( b∗ne ( j , i ) ) ;
}
denom=denom+log ( tdenom ) ;
}
return b∗num−denom ;
}
21. R packages Potts model Bayesian computation Conclusion
Approximation Error
PL for n = 12, k = 3 in comparison to the exact likelihood
calculated using a brute force method:
0 1 2 3 4
6810121416
β
µ
exact
pseudolikelihood
(a) Expectation
0 1 2 3 4
0.00.51.01.52.02.5
β
σ
exact
pseudolikelihood
(b) Standard deviation
23. R packages Potts model Bayesian computation Conclusion
TI algorithm
Algorithm 3 Random walk Metropolis with TI
1: Draw random walk proposal β ∼ q(β |β◦)
2: Estimate S(z|β◦) and S(z|β ) by interpolation
3: Evaluate the definite integral in equation (8)
4: Calculate the log M-H acceptance ratio:
log{ρ} = log
C(β◦)
C(β )
+ (β − β◦
)S(z) (9)
5: Draw u ∼ Uniform[0, 1]
6: if u < min(1, ρ) then
7: β ← β
8: else
9: β ← β◦
10: end if
24. R packages Potts model Bayesian computation Conclusion
TI in C++
§
unsigned pathBeta ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,
const arma : : mat & path , const arma : : umat & z , double & beta ,
const double p r i o r _beta [ 2 ] , const double bw)
{
double bprime = rwmh( beta , bw, p r i o r _beta ) ; / / truncated Gaussian
/ / approximate log (Z( bprime ) / Z( beta ) )
double log_ r a t i o = quadrature ( bprime , beta , path )
+ ( bprime−beta ) ∗ sum_ ident ( z , neigh , blocks ) ;
/ / accept / r e j e c t
i f ( u n i f _rand ( ) < exp ( log_ r a t i o ) )
{
beta = bprime ;
return 1;
}
return 0;
}
25. R packages Potts model Bayesian computation Conclusion
Approximate Exchange Algorithm (AEA)
Algorithm 4 AEA
1: Draw random walk proposal β ∼ q(β |β◦)
2: Generate w|β by sampling from eq. (3)
3: Calculate the M-H acceptance ratio according to eq. (4):
ρ =
π(β ) exp {β S(z)} C(β◦)
π(β◦) exp {β◦S(z)} C(β )
exp {β◦S(w)} C(β )
exp {β S(w)} C(β◦)
(10)
4: Draw u ∼ Uniform[0, 1]
5: if u < min(1, ρ) then
6: β ← β
7: else
8: β ← β◦
9: end if
Murray, Ghahramani & MacKay (2006) Proc. 22nd
Conf. UAI, 359–366
26. R packages Potts model Bayesian computation Conclusion
AEA in C++
§
unsigned exchangeBeta ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,
const arma : : uvec & slice , const arma : : umat & z , double & beta ,
const double p r i o r _beta [ 2 ] , const unsigned aux , const bool useSW,
const bool swapAux , const double bw)
{
double bprime = rwmh( beta , bw, p r i o r _beta ) ;
arma : : umat a l l o c = arma : : zeros <arma : : umat >(z . n_rows−1, z . n_cols ) ;
arma : : umat w;
i f ( swapAux ) w = z ;
else w = randomIndices ( z . n_rows−1, z . n_cols ) ;
for ( unsigned i =0; i <aux ; i ++)
{
i f (useSW)
{
swLabelsNoData ( neigh , blocks , bprime , w. n_cols , w, a l l o c ) ;
}
else
{
gibbsLabelsNoData ( neigh , blocks , w, alloc , bprime ) ;
}
}
double sum_z = sum_ ident ( z , neigh , blocks ) ;
double sum_w = sum_ ident (w, neigh , blocks ) ;
double log_ r a t i o = ( bprime−beta )∗sum_z + ( beta−bprime )∗sum_w;
/ / accept / r e j e c t
i f ( u n i f _rand ( ) < exp ( log_ r a t i o ) )
{
beta = bprime ;
return 1;
}
return 0;
}
27. R packages Potts model Bayesian computation Conclusion
Approximate Bayesian Computation
Algorithm 5 ABC rejection sampler
1: Draw independent proposal β ∼ π(β)
2: Generate w|β by sampling from eq. (3)
3: if S(w) − S(z) < then
4: β ← β
5: else
6: β ← β◦
7: end if
Grelaud, Robert, Marin, Rodolphe & Taly (2009) Bayesian Analysis 4(2)
Marin & Robert (2014) Bayesian Essentials with R §8.3
28. R packages Potts model Bayesian computation Conclusion
ABC with Metropolis-Hastings
Algorithm 6 ABC-MCMC
1: Draw proposal β ∼ q(β |β◦)
2: Generate w|β by sampling from eq. (3)
3: Draw u ∼ Uniform[0, 1]
4: if u < π(β )q(β◦|β )
π(β◦)q(β |β◦) and S(w) − S(z) < then
5: β ← β
6: else
7: β ← β◦
8: end if
Marjoram, Molitor, Plagnol & Tavaré (2003) PNAS 100(26): 15324–28
29. R packages Potts model Bayesian computation Conclusion
ABC-MCMC in C++
§
unsigned abcBeta ( const arma : : umat & neigh , const std : : vector <arma : : uvec> & blocks ,
const arma : : umat & z , double & beta , const double p r i o r _beta [ 2 ] ,
const unsigned aux , const bool useSW, const bool swapAux ,
const double bw, const double epsilon )
{
double bprime = rwmh( beta , bw, p r i o r _beta ) ;
arma : : umat a l l o c = arma : : zeros <arma : : umat >(z . n_rows−1, z . n_cols ) ;
arma : : umat w;
i f ( swapAux ) w = z ;
else w = randomIndices ( z . n_rows−1, z . n_cols ) ;
for ( unsigned i =0; i <aux ; i ++)
{
i f (useSW)
{
swLabelsNoData ( neigh , blocks , bprime , w. n_cols , w, a l l o c ) ;
}
else
{
gibbsLabelsNoData ( neigh , blocks , w, alloc , bprime ) ;
}
}
double sum_z = sum_ ident ( z , neigh , blocks ) ;
double sum_w = sum_ ident (w, neigh , blocks ) ;
double delta = fabs (sum_w − sum_z ) ;
i f ( delta < epsilon )
{
beta = bprime ;
return 1;
}
return 0;
}
30. R packages Potts model Bayesian computation Conclusion
Summary
bayesImageS supports methods for updating the latent labels:
Chequerboard Gibbs sampling (Winkler 2003)
Swendsen-Wang (1987)
and also methods for updating the inverse temperature β:
Pseudolikelihood (Rydén & Titterington 1998)
Thermodynamic integration (Gelman & Meng 1998)
Exchange algorithm (Murray, Ghahramani & MacKay 2006)
Approximate Bayesian computation (Grelaud et al. 2009)
Sequential Monte Carlo (ABC-SMC) with pre-computation
(Del Moral, Doucet & Jasra 2012; Moores et al. 2015)
31. Appendix
Acknowledgements
Principal supervisor: Kerrie Mengersen
Associate supervisor: Fiona Harden
Radiation Oncology Mater Centre (ROMC), Queensland Health:
Cathy Hargrave
Mike Poulsen
Tim Deegan
Other co-authors:
Christopher Drovandi (QUT)
Anthony N. Pettitt (QUT)
Christian Robert
(University of Warwick & Université Paris Dauphine)
PyMCMC team at QUT:
Clair Alston
Christopher Strickland
32. Appendix
For Further Reading I
D. Eddelbuettel
Seamless R and C++ integration with Rcpp
Springer-Verlag, 2013.
H. Wickham
R packages
O’Reilly, 2015.
D. Bates & D. Eddelbuettel
Fast and elegant numerical linear algebra using the RcppEigen
package.
J. Stat. Soft. 52(5): 1–24, 2013.
D. Eddelbuettel & C. Sanderson
RcppArmadillo: Accelerating R with high-performance C++ linear
algebra.
Comput. Stat. Data Anal. 71: 1054–63, 2014.
33. Appendix
For Further Reading II
M. Moores & K. Mengersen
bayesImageS: Bayesian methods for image segmentation using a
hidden Potts model.
R package version 0.3-3
https://CRAN.R-project.org/package=bayesImageS
M. Moores, A. N. Pettitt & K. Mengersen
Scalable Bayesian inference for the inverse temperature of a hidden
Potts model.
arXiv:1503.08066 [stat.CO], 2015.
M. Moores, C. C. Drovandi, K. Mengersen & C. P. Robert
Pre-processing for approximate Bayesian computation in image
analysis.
Statistics & Computing 25(1): 23–33, 2015.
M. Moores & K. Mengersen
Bayesian approaches to spatial inference: modelling and computational
challenges and solutions.
In Proc. 33rd Int. Wkshp MaxEnt, AIP Conf. Proc. 1636: 112–117, 2014.
34. Appendix
For Further Reading III
G. Winkler
Image analysis, random fields and Markov chain Monte Carlo methods
2nd
ed., Springer-Verlag, 2003.
J.-M. Marin & C. P. Robert
Bayesian Essentials with R
Springer-Verlag, 2014.
G. O. Roberts & S. K. Sahu
Updating Schemes, Correlation Structure, Blocking and
Parameterization for the Gibbs Sampler
J. R. Stat. Soc. Ser. B 59(2): 291–317, 1997.
T. Rydén & D. M. Titterington
Computational Bayesian Analysis of Hidden Markov Models.
J. Comput. Graph. Stat., 7(2): 194–211, 1998.
35. Appendix
For Further Reading IV
A. Gelman & X.-L. Meng
Simulating normalizing constants: from importance sampling to bridge
sampling to path sampling.
Statist. Sci., 13(2): 163–185, 1998.
I. Murray, Z. Ghahramani & D. J. C. MacKay
MCMC for Doubly-intractable Distributions.
Proc. 22nd
Conf. UAI, 359–366, 2006.
A. Grelaud, C. P. Robert, J.-M. Marin, F. Rodolphe & J.-F. Taly
ABC likelihood-free methods for model choice in Gibbs random fields.
Bayesian Analysis 4(2): 317–336, 2009.
R. H. Swendsen & J.-S. Wang
Nonuniversal critical dynamics in Monte Carlo simulations.
Physical Review Letters, 58: 86–8, 1987.