MCMC for mixtures of Gaussians, and model 
selection 
Aaron McDaid, aaronmcdaid@gmail.com 
October 30, 2014 
1 / 36
Six models 
l 
l 
l 
l 
ll 
l l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l ...
Six models 
l 
l l 
l l 
l 
l 
l l 
l 
l 
l 
l 
ll 
l l 
l l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
...
Six models 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
ll 
l 
l 
l l 
l 
l 
l 
l 
l 
l 
l 
l 
ll 
l 
l 
l 
l 
l...
Old Faithful N=272 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
...
Old Faithful N=272 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
l 
...
Overview 
Goals 
De
ne the mclust model 
Bayes Factor and BIC - connection between mclust and 
MCMC 
Priors 
Integration (analytical and numer...
Goals 
Not a `shootout' with mclust 
See what MCMC can do 
Calculate the Bayes Factor more precisely - is it better than 
...
Basic model 
N data points in a p-dimensional space. 
m 2 (fvvv; eee; vvi; eei; vii; eiig) 
K number of clusters 
k covari...
mclust 
MLE (Maximum Likelihood Estimate) 
R package mclust2 
Given (K;m), use Expectation-Maximization (EM) algorithm 
to...
cation 16.2 (1999), pp. 297{306. 
10 / 36
mclust 
Why do we need model selection? 
vvv vvi vii 
eee eei eii 
De
ne  = (;;). 
P(Xj=^ eee;K;m=vvv;K) = P(Xj=^ eee;K;m=eee;K) 
Cannot maximize P(Xj;m;K) 
Count the degrees-of-freedom f , in...
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
MCMC for clustering of multivariate-Normal data
Prochain SlideShare
Chargement dans…5
×

MCMC for clustering of multivariate-Normal data

463 vues

Publié le

Using modern, fast, MCMC methods to cluster a dataset. Data are modelled as clusters of multivariate Gaussians. Each cluster can have a different covariance matrix.

Selecting the number of clusters, and covariance model, is a challenge. Often, the BIC is used as an approximation to the Bayes Factor. With MCMC, we can compute the Bayes Factor more accurately.

This methods leads to improved performance in the case of datasets with a large number of small clusters.

Publié dans : Sciences
0 commentaire
0 j’aime
Statistiques
Remarques
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Aucun téléchargement
Vues
Nombre de vues
463
Sur SlideShare
0
Issues des intégrations
0
Intégrations
2
Actions
Partages
0
Téléchargements
2
Commentaires
0
J’aime
0
Intégrations 0
Aucune incorporation

Aucune remarque pour cette diapositive

MCMC for clustering of multivariate-Normal data

  1. 1. MCMC for mixtures of Gaussians, and model selection Aaron McDaid, aaronmcdaid@gmail.com October 30, 2014 1 / 36
  2. 2. Six models l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l ll l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l ll l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l ll l ll l l l l l l l l l lll l l l l l l l l l l l l l ll l l l l l l ll l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l ll l l l ll l l l l l l l l l l l l ll l l l l l ll ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l −20 0 20 40 60 0 20 40 60 80 100 V1 V2 (a) 1. vvv l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l −10 −5 0 5 10 −30 −20 −10 0 10 20 30 V1 V2 (b) 2. eee 2 / 36
  3. 3. Six models l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l ll l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l −10 0 10 20 30 −20 −10 0 10 V1 V2 (a) 3. vvi l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l ll l l llll l l l l l ll l l l l l l l l l l l l l l l l l l l l l ll l ll ll l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l ll l l l l ll l l l l l l l l l l l ll l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l ll l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l −30 −20 −10 0 10 20 30 −30 −20 −10 0 10 20 30 V1 V2 (b) 4. eei 3 / 36
  4. 4. Six models l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l ll l l l l l l l l l l ll l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l ll l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l ll l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll ll l l ll l l l l l l l l l l l l ll l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l lll l l l l l l l l l l l l l l l l l l l l l l ll l ll l l l l l l l l l l l l l l l l l l l ll l l l l l l l lll l l l l l ll l l l ll l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l ll l l ll l l l l l l l l l l l l l l ll ll l l −20 −10 0 10 20 −20 −10 0 10 20 V1 V2 (a) 5. vii l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l ll l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l ll l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l −100 0 100 200 −200 −100 0 100 200 V1 V2 (b) 6. eii 4 / 36
  5. 5. Old Faithful N=272 l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 50 60 70 80 90 eruptions waiting Old Faithful - Yellowstone National Park 5 / 36
  6. 6. Old Faithful N=272 l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 50 60 70 80 90 V1 V2 Old Faithful - Yellowstone National Park 6 / 36
  7. 7. Overview Goals De
  8. 8. ne the mclust model Bayes Factor and BIC - connection between mclust and MCMC Priors Integration (analytical and numerical) MCMC algorithm1 Selecting from the six models via MCMC Evaluation (on synthetic data) One application 1Mahlet G. Tadesse, Naijun Sha, and Marina Vannucci. Bayesian Variable Selection in Clustering High-Dimensional Data". In: Journal of the American Statistical Association 100.470 (June 2005), pp. 602{617. issn: 0162-1459. doi: 10.1198/016214504000001565. url: http://www.stat.rice.edu/~{}marina/papers/jasa05.pdf. 7 / 36
  9. 9. Goals Not a `shootout' with mclust See what MCMC can do Calculate the Bayes Factor more precisely - is it better than BIC? Push to larger numbers of clusters 8 / 36
  10. 10. Basic model N data points in a p-dimensional space. m 2 (fvvv; eee; vvi; eei; vii; eiig) K number of clusters k covariance of clusterk k mean Pof cluster k K k=1 k = 1 zi P(zi = k) = k xi jzi=k Normal(k ;k ): Mixture models P(xi jzi=k) = N(xi jk ;k ) P(xi ) = XK k=1 kN(xi jk ;k ) 9 / 36
  11. 11. mclust MLE (Maximum Likelihood Estimate) R package mclust2 Given (K;m), use Expectation-Maximization (EM) algorithm to estimate (;;). P(Xjk ;k ;;m;K) Requires running EM for each possible combination of (K;m). Hundreds of runs may be required. f(K = 2;m = VVI); (K = 3;m = EEI ); (K = 50;m = EEI ); : : : g Then use BIC to select among the models. 2Chris Fraley and Adrian E. Raftery. MCLUST: Software for model-based cluster analysis. In: Journal of Classi
  12. 12. cation 16.2 (1999), pp. 297{306. 10 / 36
  13. 13. mclust Why do we need model selection? vvv vvi vii eee eei eii De
  14. 14. ne = (;;). P(Xj=^ eee;K;m=vvv;K) = P(Xj=^ eee;K;m=eee;K) Cannot maximize P(Xj;m;K) Count the degrees-of-freedom f , in order to penalize the more complex model. AIC = 2 log P(XjMLE m;K ;m;K)

×