Christopher Fonnesbeck Associate Professor Vanderbilt University Medical Center

1. Bayesian Non-parametric Models
for DataScience using PyMC3
Christopher Fonnesbeck
Associate Professor
Vanderbilt University Medical Center

2. Motivation

5. StatisticalAssumptions
» parametric distributions for data
» linear relationships among variables

6. sns.regplot('spawners', 'log_recruits', data=salmon_data)

7. Bayesian Inference

10. Building Modelswith Gaussian
distributions

11. Gaussian (normal) distribution

12. Conditioning property
The conditional distribution of some elements of a
multivariate normal is also normal

13. Marginalization property
The marginal distribution of some elements of a
multivariate normal is also normal

14. Gaussian processes
Gaussian distribution:
Gaussian process:

15. Sowhatisa
Gaussian process?
“an infinite collection of random variables, any
finite subset of which have a Gaussian distribution”

16. Did I sayno parameters?
I meant
infinityparameters.

17. Gaussian processes

18. Covariance Functions
Functions that generate covariance matrices:

19. Quadratic

20. Matern(3/2)

21. Cosine

22. Mean Functions
Functions that generate mean vectors:

23. Mean Functions
Functions that generate mean vectors:
»
»
»

24. Prior over functions

25. Sampling fromaGaussian prior process
x = [1.]
y = [np.random.normal(scale=σ_0)]
y
[0.4967141530112327]

26. Sampling fromaGaussian prior process
x = -0.7
m, s = conditional([-0.7], x, y, θ)
y.append(np.random.normal(m, s))
y
[0.4967141530112327, -0.1382640378102619]

27. Sampling fromaGaussian prior process
x = [-2.1, -1.5, 0.3, 1.8, 2.5]
mu, s = conditional(x_more, x, y, θ)
y += np.random.multivariate_normal(mu, s).tolist()
y
[0.4967141530112327, -0.1382640378102619, -1.5128756 , 0.52371713,
-0.13952425, -0.93665367, -1.29343995

29. Gaussian Processes in Python
» GPy
» GPflow
» scikit-learn
» PyStan
» Edward
» George
» PyMC3

30. PyMC3
» started in 2003
» PP framework for fitting
arbitrary probability models
» based on Theano
» implements "next generation"
Bayesian inference methods
» NumFOCUS sponsored project
github.com/pymc-devs/
pymc3

31. Salmon recruitment

32. Salmon recruitment

33. Bayes Formula

34. Marginal Gaussian Process

35. Salmon recruitmentmodel
Hyperparameter priors
with Model() as salmon_model:
ρ = HalfCauchy('ρ', 3)
η = HalfCauchy('η', 3)

36. Salmon recruitmentmodel
Covariance functionand marginalGP
with salmon_model:
M = gp.mean.Linear(coeffs=(y/X).mean())
K = η * gp.cov.ExpQuad(1, ρ)
gp = gp.Marginal(mean_func=M, cov_func=K)

37. Salmon recruitmentmodel
Marginallikelihood
with salmon_model:
σ = pm.HalfNormal('σ', 1)
recruits = gp.marginal_likelihood('recruits',
X=X, y=y,
noise=σ)

38. Salmon recruitmentmodel
Modelfitting
with salmon_model:
fit = pm.find_MAP()

39. Making predictionswith Gaussian processes
Posterior predictive distribution:

40. Making predictionswith Gaussian processes

41. Salmon recruitmentmodel
Posterior prediction
spawner_range = np.linspace(0, 500, 100).reshape(-1, 1)
with salmon_model:
salmon_pred = gp.conditional("salmon_pred", spawner_range)
salmon_samples = pm.sample_ppc([fit], vars=[salmon_pred],
samples=3)

42. Salmon recruitmentmodel

43. Salmon recruitmentmodel
Posterior prediction
with salmon_model:
salmon_pred_noisy = gp.conditional("salmon_pred_noisy",
spawner_range,
pred_noise=True)
salmon_samples = pm.sample_ppc([fit],
vars=[salmon_pred_noisy],
samples=1000)
100%|██████████| 1000/1000 [00:04<00:00, 207.03it/s]

44. Salmon recruitmentmodel

45. CoalMining
Disasters

46. # Time series of recorded coal mining disasters in the UK from 1851 to 1962
disasters_data = np.array([4, 5, 4, 0, 1, 4, 3, 4, 0, 6, 3, 3, 4, 0, 2, 6,
3, 3, 5, 4, 5, 3, 1, 4, 4, 1, 5, 5, 3, 4, 2, 5,
2, 2, 3, 4, 2, 1, 3, 2, 2, 1, 1, 1, 1, 3, 0, 0,
1, 0, 1, 1, 0, 0, 3, 1, 0, 3, 2, 2, 0, 1, 1, 1,
0, 1, 0, 1, 0, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2,
3, 3, 1, 1, 2, 1, 1, 1, 1, 2, 4, 2, 0, 0, 1, 4,
0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1])
year = np.arange(1851, 1962)

47. Latent Gaussian Process

48. LatentGPmodel
with Model() as disasters_model:
# Hyperpriors on covariance function parameters
ρ = Exponential('ρ', 1)
η = Exponential('η', 1)
# Covariance function
K = η**2 * cov.ExpQuad(1, ρ)
# Latent GP
gp = Latent(cov_func=K)
f = gp.prior('f', X=year[:, None])
# Transform into rate parameter
λ = Deterministic('λ', exp(f))
confirmation = Poisson('confirmation', λ,
observed=disasters_data)

49. LatentGPmodel
with Model() as disasters_model:
# Hyperpriors on covariance function parameters
ρ = Exponential('ρ', 1)
η = Exponential('η', 1)
# Covariance function
K = η**2 * cov.ExpQuad(1, ρ)
# Latent GP
gp = Latent(cov_func=K)
f = gp.prior('f', X=year[:, None])
# Transform into rate parameter
λ = Deterministic('λ', exp(f))
confirmation = Poisson('confirmation', λ,
observed=disasters_data)

50. LatentGPmodel
with Model() as disasters_model:
# Hyperpriors on covariance function parameters
ρ = Exponential('ρ', 1)
η = Exponential('η', 1)
# Covariance function
K = η**2 * cov.ExpQuad(1, ρ)
# Latent GP
gp = Latent(cov_func=K)
f = gp.prior('f', X=year[:, None])
# Transform into rate parameter
λ = Deterministic('λ', exp(f))
confirmation = Poisson('confirmation', λ,
observed=disasters_data)

51. MCMC Sampling
with disasters_model:
trace = sample(1000, tune=2000)
Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (2 chains in 2 jobs)
NUTS: [f_rotated_, η_log__, ρ_log__]
100%|██████████| 3000/3000 [04:04<00:00, 12.27it/s]
There were 1 divergences after tuning. Increase `target_accept` or reparameterize.

53. Posterior hyperparameter
estimates
plot_posterior(trace, varnames=['ρ', 'η'])

54. Measles OutbreakSão Paulo 1997

56. with Model() as confirmation_model:
ρ = Exponential('ρ', 1)
η = Exponential('η', 1)
K = η**2 * cov.ExpQuad(1, ρ)
gp = Latent(cov_func=K)
f = gp.prior('f', X=age[:, None])
π = Deterministic('π', invlogit(f))
confirmation = Binomial('confirmation', p=π, n=n.astype(int),
observed=confirmed.astype(int))

58. Scaling Gaussian Processes

59. Posterior Covariance

60. Posterior Covariance
!
compute time, memory

61. Posterior Covariance
!
compute time, memory
» Sparse approximation methods approximate the
posterior using a set of inducing inputs
(pseudo-inputs)

62. MarathonTimes
2015 Boston Marathon (26,598 finishers)

64. with Model() as marathon_model:
ρ = Exponential('ρ', 1)
η = Exponential('η', 1)
K = η**2 * cov.ExpQuad(1, ρ)
gp = pm.gp.MarginalSparse(cov_func=K, approx="FITC")
# initialize 20 inducing points with K-means
Xu = pm.gp.util.kmeans_inducing_points(10, X)
σ = pm.HalfCauchy("σ", beta=1)
obs = gp.marginal_likelihood("obs", X=X, Xu=Xu, y=y, noise=σ)

66. MultidimensionalGP

67. Walker Lake Dataset
Isaaks and Srivastava, 1989

69. Spatialsamples

70. with pm.Model() as model:
ℓ = pm.Gamma("ℓ", alpha=2, beta=1)
η = pm.HalfCauchy("η_f", beta=5)
cov = η**2 * pm.gp.cov.ExpQuad(2, ℓ)
gp = pm.gp.MarginalSparse(cov_func=cov, approx="FITC")
σ = pm.HalfCauchy("σ", beta=5)
y = gp.marginal_likelihood("y", X=X_obs, Xu=Xu, y=y_obs, noise=σ)
mp = pm.find_MAP()

71. with pm.Model() as model:
ℓ = pm.Gamma("ℓ", alpha=2, beta=1)
η = pm.HalfCauchy("η_f", beta=5)
cov = η**2 * pm.gp.cov.ExpQuad(2, ℓ)
gp = pm.gp.MarginalSparse(cov_func=cov, approx="FITC")
σ = pm.HalfCauchy("σ", beta=5)
y = gp.marginal_likelihood("y", X=X_obs, Xu=Xu, y=y_obs, noise=σ)
mp = pm.find_MAP()

72. Conditionalsample overagrid
nd = 30
z1, z2 = np.meshgrid(np.linspace(0, 300, nd), np.linspace(0, 300, nd))
Z = np.concatenate([z1.reshape(nd*nd, 1), z2.reshape(nd*nd, 1)], 1)
with model:
f_pred = gp.conditional("f_pred", Z)
samples = pm.sample_ppc([mp], vars=[f_pred], samples=15)

74. Gaussian Processes for Data
Science
There's more!
- GP for classification
- automatic relevance determination
- neural networks
- reinforcement learning

75. Gaussian Processes for Data
Science
Advantages:
- flexible modeling of complex, non-linear processes
- fully probabilistic predictions
- interpretable hyperparameters
- easy to automate
Limitations:
- scales poorly with large data (without
approximation)

77. The PyMC3Team
» Colin Carroll
» Peadar Coyle
» Bill Engels
» Maxim Kochurov
» Junpeng Lao
» Eric Ma
» Osvaldo Martin
» Kyle Meyer
» Michael Osthege
» Austin Rochford
» Adrian Seyboldt
» Hannes Vasyura-Bathke
» Thomas Wiecki
» Taku Yoshioka