An introduction to the Gaussian Process Latent Variable Model (GPLVM)

1. Gaussian Process Latent Variable Model
(GPLVM)
James McMurray
PhD Student
Department of Empirical Inference
11/02/2014

2. Outline of talk
Introduction
Why are latent variable models useful?
De

3. nition of a latent variable model
Graphical Model representation
PCA recap
Principal Components Analysis (PCA)
Probabilistic PCA based methods
Probabilistic PCA (PPCA)
Dual PPCA
GPLVM
Examples
Practical points
Variants
Conclusion
Conclusion
References

4. Why are latent variable models useful?
I Data has structure.

5. Why are latent variable models useful?
I Observed high-dimensional data often lies on a
lower-dimensional manifold.
Example Swiss Roll" dataset

6. Why are latent variable models useful?
I The structure in the data means that we don't need such high
dimensionality to describe it.
I The lower dimensional space is often easier to work with.
I Allows for interpolation between observed data points.

7. De

8. nition of a latent variable model
I Assumptions:
I Assume that the observed variables actually result from a
smaller set of latent variables.
I Assume that the observed variables are independent given the
latent variables.
I Diers slightly from dimensionality reduction paradigm which
wishes to

9. nd a lower-dimensional embedding in the
high-dimensional space.
I With the latent variable model we specify the functional form
of the mapping:
y = g(x) +
where x are the latent variables, y are the observed variables
and is noise.
I Obtain dierent latent variable models for dierent
assumptions on g(x) and

10. Graphical model representation
Graphical Model example of Latent Variable Model
Taken from Neil Lawrence:
http: // ml. dcs. shef. ac. uk/ gpss/ gpws14/ gp_ gpws14_ session3. pdf

11. Plan
Introduction
Why are latent variable models useful?
De

12. nition of a latent variable model
Graphical Model representation
PCA recap
Principal Components Analysis (PCA)
Probabilistic PCA based methods
Probabilistic PCA (PPCA)
Dual PPCA
GPLVM
Examples
Practical points
Variants
Conclusion
Conclusion
References

13. Principal Components Analysis (PCA)
I Returns orthogonal dimensions of maximum variance.
I Works well if data lies on a plane in the higher dimensional
space.
I Linear method (although variants allow non-linear application,
e.g. kernel PCA).
Example application of PCA. Taken from
http: // www. nlpca. org/ pca_ principal_ component_ analysis. html

14. Plan
Introduction
Why are latent variable models useful?
De

15. nition of a latent variable model
Graphical Model representation
PCA recap
Principal Components Analysis (PCA)
Probabilistic PCA based methods
Probabilistic PCA (PPCA)
Dual PPCA
GPLVM
Examples
Practical points
Variants
Conclusion
Conclusion
References

16. Probabilistic PCA (PPCA)
I A probabilistic version of PCA.
I Probabilistic formulation is useful for many reasons:
I Allows comparison with other techniques via likelihood
measure.
I Facilitates statistical testing.
I Allows application of Bayesian methods.
I Provides a principled way of handling missing values - via
Expectation Maximization.

17. PPCA De

18. nition
I Consider a set of centered data of n observations and d
dimensions: Y = [y1; : : : ; yn]T .
I We assume this data has a linear relationship with some
embedded latent space data xn. Where Y 2 RND and
x 2 RNq.
I yn = Wxn + n, where xn is the q-dimensional latent variable
associated with each observation, and W 2 RDq is the
transformation matrix relating the observed and latent space.
I We assume a spherical Gaussian distribution for the noise with
a mean of zero and a covariance of

19. 1I
I Likelihood for an observation yn is:
p (ynjxn;W;

20. ) = N
ynjWxn;

21. 1I

22. PPCA Derivation
I Marginalise latent variables xn, put a Gaussian prior on W and
solve using maximum likelihood.
I The prior used for xn in the integration is a zero mean, unit
covariance Gaussian distribution:
p(xn) = N(xnj0; I)
p(ynjW;

23. ) =
Z
p(ynjxn;W;

24. )p(xn)dxn
p(ynjW;

25. ) =
Z
N
ynjWxn;

26. 1I
N(xnj0; I)dxn
p(ynjW;

27. ) = N(ynj0;WWT +

28. 1I)
I Assuming i.i.d. data, the likelihood of the full set is the
product of the individual probabilities:
p(Y jW;

29. ) =
YN
n=1
p(ynjW;

30. )

31. PPCA Derivation
I To calculate that marginalisation step we use the summation
and scaling properties of Gaussians.
I Sum of Gaussian variables is Gaussian.
Xn
i=1
N(i ; 2
i ) N
Xn
i=1
i ;
Xn
i=1
2
i
!
I Scaling a Gaussian leads to a Gaussian:
wN(; 2) N(w;w22)
I So:
y = Wx + ; x N(0; I) ; N(0; 2I)
Wx N(0;WWT)
Wx + N(0;WWT + 2I)

32. PPCA Derivation
I Can

33. nd a solution for W by maximising the likelihood.
I Results in an eigenvalue problem.
I Turns out that the closed-form solution for W is achieved
when W spans the principal sub-space of the data1.
I Same solution as PCA: Probabilistic PCA
I Can it be extended to capture non-linear features?
1Michael E. Tipping and Christopher M. Bishop. Probabilistic principal
component analysis. (1997).

34. Dual PPCA
I Similar to previous derivation of PPCA.
I But marginalise W and optimise xn.
I Same linear-Gaussian relationship between latent variables and
data:
p(YjX;W;

35. ) =
YD
d=1
N(yd;:jWxd;:;

36. 1I)
I Place a conjugate prior on W:
P(W) =
YD
d=1
N(wd;:j0; I)
I Resulting marginal likelihood is:
P(Y jX;

37. ) =
YD
d=1
N(y:;d j0;XXT +

38. 1I)

39. Dual PPCA
I Results in equivalent eigenvalue problem to PPCA.
I So what is the bene

40. t?
I The eigendecomposition is now done on an N q instead of a
d q matrix.
I Recall marginal likelihood:
P(Y jX;

41. ) =
YD
d=1
N(y:;d j0;XXT +

42. 1I)
I The covariance matrix is a covariance function:
K = XXT +

43. 1I
I This linear kernel can be replaced by other covariance
functions for non-linearity.
I This is the GPLVM.

44. GPLVM
I Each dimension of the marginal distribution can be interpreted
as an independent Gaussian Process2.
I Dual PPCA is the special case where the output dimensions
are assumed to be linear, independent and identically
distributed.
I GPLVM removes assumption of linearity.
I Gaussian prior over the function space.
I Choice of covariance function changes family of functions
considered.
I Popular kernels:
I Exponentiated Quadratic (RBF) kernel
I Matern kernels
I Periodic kernels
I Many more...
2Neil Lawrence: Probabilistic non-linear principal component analysis with
Gaussian process latent variable models. JMLR (2005)

45. Plan
Introduction
Why are latent variable models useful?
De

46. nition of a latent variable model
Graphical Model representation
PCA recap
Principal Components Analysis (PCA)
Probabilistic PCA based methods
Probabilistic PCA (PPCA)
Dual PPCA
GPLVM
Examples
Practical points
Variants
Conclusion
Conclusion
References

47. Example: Frey Face data
Example in GPMat

48. Example: Motion Capture data
Taken from Keith Grochow, et al. Style-based inverse kinematics. ACM
Transactions on Graphics (TOG). Vol. 23. No. 3. ACM, 2004.

49. Plan
Introduction
Why are latent variable models useful?
De

50. nition of a latent variable model
Graphical Model representation
PCA recap
Principal Components Analysis (PCA)
Probabilistic PCA based methods
Probabilistic PCA (PPCA)
Dual PPCA
GPLVM
Examples
Practical points
Variants
Conclusion
Conclusion
References

51. Practical points
I Need to optimise over non-convex objective function.
I Achieved using gradient-based methods (scaled conjugate
gradients).
I Several restarts to attempt to avoid local optima.
I Cannot guarantee global optimum.
I High computational cost for large datasets.
I May need to optimise over most-informative subset of data,
the active set for sparsi

52. cation.

53. Practical points
Initialisation can have a large eect on the

54. nal results.
Eect of poor initialisation on Swiss Roll dataset. PCA left, Isomap right. Taken from
Probabilistic non-linear principal component analysis with Gaussian process latent
variable models., Neil Lawrence, JMLR (2005).

55. Plan
Introduction
Why are latent variable models useful?
De

56. nition of a latent variable model
Graphical Model representation
PCA recap
Principal Components Analysis (PCA)
Probabilistic PCA based methods
Probabilistic PCA (PPCA)
Dual PPCA
GPLVM
Examples
Practical points
Variants
Conclusion
Conclusion
References

57. Variants
I There are a number of variants of the GPLVM.
I For example, the GPLVM uses the same covariance function
for each output dimension.
I This can be changed, for example the Scaled GPLVM which
introduces a scaling parameter for each output dimension3.
I The Gaussian Process Dynamic Model (GPDM) adds another
Gaussian process for dynamical mappings4.
I The Bayesian GPLVM approximates integrating over both the
latent variables and the mapping function5.
3Keith Grochow, et al. Style-based inverse kinematics. ACM Transactions
on Graphics (TOG). Vol. 23. No. 3. ACM, 2004.
4Wang, Jack, Aaron Hertzmann, and David M. Blei. Gaussian process
dynamical models. Advances in neural information processing systems. 2005.
5Titsias, Michalis, and Neil Lawrence. Bayesian Gaussian process latent
variable model. (2010).

58. Variants
I The Shared GPLVM learns mappings from a shared latent
space to two separate observational spaces.
I Used by Disney Research in their paper Animating
Non-Humanoid Characters with Human Motion Data for
generating animations for non-human characters from human
motion capture data.
Shared GPLVM mappings as used by Disney Research
I Video

59. Variants
I Can also put a Gaussian Process prior on X to produce Deep
Gaussian Processes6.
I Zhang et al. developed Invariant GPLVM7 - permits
interpretation of causal relations between observed variables,
by allowing arbitrary noise correlations between the latent
variables.
I Currently attempting to implement IGPLVM in GPy.
6Damianou, Andreas C., and Neil D. Lawrence. Deep Gaussian Processes.
arXiv preprint arXiv:1211.0358 (2012).
7Zhang, K., Scholkopf, B., and Janzing, D. (2010). Invariant Gaussian
Process Latent Variable Models and Application in Causal Discovery. UAI
2010.

60. Plan
Introduction
Why are latent variable models useful?
De

61. nition of a latent variable model
Graphical Model representation
PCA recap
Principal Components Analysis (PCA)
Probabilistic PCA based methods
Probabilistic PCA (PPCA)
Dual PPCA
GPLVM
Examples
Practical points
Variants
Conclusion
Conclusion
References

62. Conclusion
I Implemented in GPy (Python) and GPMat (MATLAB).
I Many practical applications - pose modelling, tweening
I Especially if smooth interpolation is desireable.
I Modeling of confounders.
Thanks for your time
Questions?

63. References
I Neil Lawrence, Gaussian process latent variable models for
visualisation of high dimensional data. Advances in neural
information processing systems 16.329-336 (2004): 3.
I Neil Lawrence, Probabilistic non-linear principal component
analysis wit Gaussian process latent variable models. JMLR
(2005)
I Gaussian Process Winter School, Sheeld 2013:
http://ml.dcs.shef.ac.uk/gpss/gpws14/
I WikiCourseNote: http://wikicoursenote.com/wiki/
Probabilistic_PCA_with_GPLVM