Chapter 14 Computing: McCulloch, CE, Searle SR and Neuhaus, JM 2008. Generalized, Linear and Mixed Models. John Wiley & Sons, New York. Second Edition.

Review of LM, GLM, LMM, GLMM
Numerical Integration for Solving GLMM
GLMM in R

Chapter 14: Computing
Generalized, Linear, and Mixed Models
Charles E. McCulloch, Shayle R. Searle, John M. Neuhaus

Gota Morota

May 4, 2010

Gota Morota


GLMM in R

Outline

1


2


3

GLMM in R

Gota Morota


GLMM in R

Linear Model and Linear Mixed Model
LM: Solving the MLE with a least squares for fixed effects and
simple formula for estimating residual variance.

ˆ
B = (X X)−1 X y
RSS
σ2 =
ê
N−p
LMM Solving the MLE with a least squares for the fixed effects
and random effects. Solving the MLE with a (RE)ML for the
variance components.

ˆ
B = (X V−1 X)X V−1 y
ˆ
u = DZ V−1 (y − XB)
Variance Components = (RE)ML coupled with iterative methods
where D is Var(u), V is Var(y) = ZDZ + R
Gota Morota


GLMM in R

Generalized Linear Model and Generalized Linear Mixed
Model
GLM: Fixed effects are estimated by solving the MLE
(nonlinear) with an iterative reweighted least squares such as
Fisher Scoring.
Bm+1 = Bm + (X WX)−1 X W∆(y − u)
where ui = g −1 (xi B ), g (ui ) = xi B, ∆ = {gu (ui )}, W = {wi },
2
wi = [υ(ui )gu (ui )]−1
GLMM Requires high dimensional integration to evaluate and
maximizing the likelihood cannot be computed explicitly
(hence not able to solve iteratively like GLM)
L=

fYi |u (yi |u)fU (u)du
i

Gota Morota


GLMM in R

Numerical Integration

Method for numerically approximating the value of a deﬁnite
integral
b

f (x )dx
a

Numerical integration for one-dimensional integrals
Rectangle rule
Trapezoidal rule
Simpson’s rule

Gota Morota


GLMM in R

Trapezoidal Rule
b

f (x )dx =

a

b −a
(f (x0 ) + 2f (x1 ) + 2f (x2 ) · · · + 2f (xn−1 ) + f (xn ))
2n

where
xk = a + k

b −a
n

for k = 0, 1, · · · , n

Figure 1: From Wikipedia http://en.wikipedia.org/wiki/Trapezoidal rule

Gota Morota


GLMM in R

Evaluation of the Integrals

There are various methods to do this:

Approximating the integral
Gauss-Hermite Quadrature
Laplace Approximation
Adaptive Gauss-Hermite Quadrature

Approximating the data
Penalized Quasi-Likelihood

Gota Morota


GLMM in R

Gauss-Hermite Quadrature I
yij |u ∼ indep. fYij |U (yij |u)
fYij |U (yij |u) = exp([yij γij − b (γij )]/γ2 − c (yij , γ))
E [yij |u] = µij
g (µij ) = xij B + ui , ui ∼ i.i.d. N (0, σ2 )
u
L=

fYij |Ui (yij |ui )fUi (ui )dui
i ,j

∞

=

e
i

j [yij γij −b (γij )]γ

−

j

c (yij ,γ) e

=

−ui2 /(2σ2 )
u

2πσ2
u

−∞

e −ui /(2σu )
2

∞

hi (ui )
i

2

−∞
Gota Morota

2

2πσ2
u

dui


dui

GLMM in R

Gauss-Hermite Quadrature II
It be can seen that the likelihood is the product of one-dimensional
integrals of the form:
∞

h (u )

√

2

/(2σ2 )
u

du

2πσ2
u

−∞

changing u to

e −u

2σu υ gives:

∞

2

e −υ

∞

π

√

−∞

h ( 2σu υ) √ dυ ≡
−∞

√

√

where h ∗ (·) ≡ h ( 2σu ·)/ π
Gota Morota

2

h ∗ (υ)e −υ dυ


GLMM in R

Gauss-Hermite Quadrature III

Gauss-Hermite quadrature approximates the integral as a
weighted sum:
d

∞
∗

h (υ)e

−υ2

−∞

h ∗ (xk )wk

dυ
k =1

where wk is the weights, and the evaluation points, xk , are
designed to provide an accurate approximation in the case where
h ∗ (·) is a polynomial.

Gota Morota


GLMM in R

Constants for Gauss-Hermite Quadrature
Table 1: xk and wk for d = 3

d=3

xk
-1.22474487
0
1.22474487

wk
0.29540898
1.18163590
0.29540898

Formula for xk and wk
xk = ith zero of Hn (x )

√

wk =

2n−1 n! π
n2 [Hn−1 (xk )]2

Gota polynomial of degree n.
where Hn (x ) is the Hermite Morota

GLMM in R

Gauss-Hermite Quadrature IV

Example
Approximation of integral using 3-point quadrature:
∞

2

(1 + x 2 )e −x dx

(1 + [−1.22474]2 )(0.29541)

−∞

+ (1 + 02 )(1.18164) + (1 + 1.224742 )(0.29541)
= 2.65868

Gota Morota


GLMM in R

Other Approximation Methods

Laplace Approximation
1

2
3
4

ﬁnd a peak xpeak of the given integrand f (x ) by taking a
derivative
apply a second-order Taylor series expansion around this peak
calculate the variance σ = 1/f (xpeak )
approximate the f (x ) ∼ N (xk , σ2 )

Adaptive Gauss-Hermite Quadrature
1
apply centralization of the f (x ) about zero or standardization
Penalize-Quasi Likelihood
1

approximate the likelihood itself

Gota Morota


GLMM in R

GLMM in R

Table 2: GLMM in R

glmmPQL (1)
glmmML (2)
glmer (3)
MCMCglmm (4)
1
2

Package
MASS
glmmML
lme4
MCMCglmm

Random Effect
intercept/coef
intercept
intercept/coef
intercept/coef

Computing Method
PQL 1
Laplace/AGQ 2
Laplace/AGQ 2
MCMC

Approximation to the likelihood

Numerical Integration & approximation to the likelihood
Can be used in Likelihood-based methods (1-3) or bayesian
approach (4) for obtaining the unknown parameters.
Gota Morota


GLMM in R

Simulation

PQL

AGQ

0.2

0.5

1.0

1.5

0.4

0.6

0.8

1.0

2.0

Laplace

pi =

1
1 + exp (−(4 + xi + γi ))

γ ∼ N (0, 32 )
0.5

1.0

1.5

2.0

Gota Morota


GLMM in R

Summary

AGQ: produces greater accuracy in the evaluation of the
log-likelihood
Laplace: special case of AGQ
PQL: typically ends up in biased estimates
Difﬁculty dealing with more complicated models

⇓
MCMC?

Gota Morota


Chapter 14 Computing: McCulloch, CE, Searle SR and Neuhaus, JM 2008. Generalized, Linear and Mixed Models. John Wiley & Sons, New York. Second Edition.

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Viewers also liked

Viewers also liked (6)

Similar to Chapter 14 Computing: McCulloch, CE, Searle SR and Neuhaus, JM 2008. Generalized, Linear and Mixed Models. John Wiley & Sons, New York. Second Edition.

Similar to Chapter 14 Computing: McCulloch, CE, Searle SR and Neuhaus, JM 2008. Generalized, Linear and Mixed Models. John Wiley & Sons, New York. Second Edition. (20)

Recently uploaded

Recently uploaded (20)

Chapter 14 Computing: McCulloch, CE, Searle SR and Neuhaus, JM 2008. Generalized, Linear and Mixed Models. John Wiley & Sons, New York. Second Edition.