Why Teams call analytics are critical to your entire business
Spectral methods for linear systems with random inputs
1. Spectral methods for linear
systems with random inputs
A parameterized matrix view
David F. Gleich
Sandia National Laboratories
with Paul Constantine @ Sandia
and Gianluca Iaccarino @ Stanford
2. Spectral methods for linear
systems with random inputs
A parameterized matrix view
First linear systems
Second random inputs
Third parameterized
matrices
Fourth spectral methods
3. David F. Gleich (Sandia) Parameterized Matrices 3 / 38
Computational Science
Discretizing Reality
Start with physical model
Discretize space and time
Arrive at linear system or
eigenvalue problem
4. David F. Gleich (Sandia) Parameterized Matrices 4 / 38
Computational Science
Discretizing Reality
5. David F. Gleich (Sandia) Parameterized Matrices 5 / 38
Computational Science
Discretizing Reality
Ax = b
6. David F. Gleich (Sandia) Parameterized Matrices 6 / 38
Matrices at this workshop
A
Random Gaussian
Random sums of
independent matrices
Random adjacency
matrices
7. David F. Gleich (Sandia) Parameterized Matrices 7 / 38
Fireflies and Jellybeans, Creative Commons
∇·∇ =ƒ
8. David F. Gleich (Sandia) Parameterized Matrices 8 / 38
Fireflies and Jellybeans, Creative Commons
∇ · (α(s, )∇ ) = ƒ
K + s1 K1 + s2 K2 + . . . = f
9. David F. Gleich (Sandia) Parameterized Matrices 9 / 38
My favorite model PAG E R A N K
3
1. follow out-edges uniformly with
probability α, and
2 5
4
2. randomly jump according to v
with probability 1 − α, we’ll as-
1 6
sume = 1/ n.
1/ 6 ↓ Induces a Markov chain model
1/ 2 0 0 0 0
1/ 6 0 0 1/ 3 0 0 αP + (1 − α)veT x(α) = x(α)
1/ 6 1/ 2 0 1/ 3 0 0
1/ 6 0 1/ 2 0 0 0
1/ 6 0 1/ 2 1/ 3 0 1 or the linear system
1/ 6 0 0 0 1 0
( − αP)x(α) = (1 − α)v
P
10. David F. Gleich (Sandia) Parameterized Matrices 10 / 38
The PageRank Random Variable
3.0 InfBeta( 3.2 , 2.0 , 1.9e−05 , 0.0019 )
2.5
2.0
density
1.5
1.0
0.5
0.0
0.0 0.2 0.4 0.6 0.8 1.0
Raw α
11. David F. Gleich (Sandia) Parameterized Matrices 11 / 38
Parameterized Matrices
Better Discretized Reality
A(s)x(s) = b(s)
12. David F. Gleich (Sandia) Parameterized Matrices 12 / 38
Parameterized Matrices
Better Discretized Reality
A(s)x(s) = b(s)
s - independent random variables/parameters
bounded, analytic, non-singular
13. David F. Gleich (Sandia) Parameterized Matrices 13 / 38
A Parameterized Matrix View of
Uncertainty Quantification
Setup
A(s)x(s) = b(s)
s∈D
ƒ = ƒ ds
D
Wi l l m y c o o ki e s b u rn ?
Questions
E[x(s)] = 〈x(s)〉
Std[x(s)]
P { (s) ≥ γ}
x(s) ≈ faster y(s) Fireflies and Jellybeans, Creative Commons
14. David F. Gleich (Sandia) Parameterized Matrices 14 / 38
Uncertainty Quantification
At this workshop
Richmond
Unknown sensor array locations.
Schehr
Where are the viscious walkers?
Antonsen
Uncertain component structure.
Assumed "totally" random
15. David F. Gleich (Sandia) Parameterized Matrices 15 / 38
A new type of sensitivity analysis
Ulam Networks on the Chirikov Map
Chirikov map Ulam network
yt+1 = ηyt +k sin( t +θt ) 1. divide phase space into uniform cells
t+1 = t + yt+1 2. form P based on trajectories.
log(E [x(A)]) log(Std [x(A)]))/ log(E [x(A)])
A ∼ Bet (2, 16)
Note White is larger, black is smaller
Google matrix, dynamical attractors, and Ulam networks, Shepelyansky and Zhirov, arXiv
David F. Gleich (UBC) Random sensitivity Sandia 23 / 37
16. David F. Gleich (Sandia) Parameterized Matrices 16 / 38
Improved web-spam classification
Webspam application
Hosts of uk-2006 are labeled as spam, not-spam, other
P R f FP FN
Baseline 0.694 0.558 0.618 0.034 0.442
Beta(0.5,1.5) 0.695 0.561 0.621 0.034 0.439
Beta(1,1) 0.698 0.562 0.622 0.033 0.438
Beta(2,16) 0.699 0.562 0.623 0.033 0.438
Note Bagged (10) J48 decision tree classifier in Weka, mean of 50 repetitions from
10-fold cross-validation of 4948 non-spam and 674 spam hosts (5622 total).
Becchetti et al. Link analysis for Web spam detection, 2008.
David F. Gleich (UBC) Random sensitivity Sandia 29 / 37
17. David F. Gleich (Sandia) Parameterized Matrices 17 / 38
Solutions are rational or analytic
A(s)x(s) = b(s)
det(A (s))
(s) =
det(A(s))
A = A(s) with ith column
replaced by b(s)
19. Spectral methods for linear
systems with random inputs
A parameterized matrix view
First linear systems
Second random inputs
Third parameterized
matrices
Fourth spectral methods
20. David F. Gleich (Sandia) Parameterized Matrices 20 / 38
Spectral Methods
Approximate a function in a polynomial basis!
In UQ, known as
polynomial chaos
generalized polynomial chaos
stochastic Galerkin
stochastic collation
21. David F. Gleich (Sandia) Parameterized Matrices 21 / 38
Spectral Fourier Coefficients
{π , ∈ N} : an orthonormal polynomial basis.
∞
ƒ (s) = ƒ π π (s)
=0
Fourier coefficients
Truncating this representation yields best
approximation in a mean sense.
But how do we compute them?
22. David F. Gleich (Sandia) Parameterized Matrices 22 / 38
Computable Polynomial Approx.
{π , ∈ N} : an orthonormal polynomial basis.
∞
ƒ (s) = ƒ π π (s)
=0
Approx. 〈ƒ π 〉 with m-point Gauss quadrature
pseudo-spectral
23. David F. Gleich (Sandia) Parameterized Matrices 23 / 38
Gaussian Quadrature
b m
ƒ ( ) dω( ) = ƒ (λ )ω
=1
An m point quadrature rule will exactly inte-
grate all polynomials of degree 2m − 1
All ω > 0, all < λ < b.
24. David F. Gleich (Sandia) Parameterized Matrices 24 / 38
Pseudospectral Methods for PMEs
A(s)x(s) = b(s)
N−1
x(s) ≈ x π (s) = Xπ(s)
=0
m
x = x(λj )π (λj )ωj
j=0
“X = x(Λ)DQ”
25. David F. Gleich (Sandia) Parameterized Matrices 25 / 38
Galerkin Approximations for PMEs
A(s)x(s) = b(s)
N−1
x(s) ≈ x π (s) = Xπ(s)
=0
(A(s)Xπ(s) − b(s))π(s)T = 0
A(s)Xπ(s)π(s)T = b(s)π T
π(s)π(s)T ⊗ A(s) vec(X) = π(s) ⊗ b(s)
But how do we compute them?
26. David F. Gleich (Sandia) Parameterized Matrices 26 / 38
Comparison results
ρ1
ρ2
-1 1
Let ρ be the sum of semi-axes of the ellipse
(hyperellipse) of analyticity.
Both methods converge:
Cp ρ−N vs. Cg ρ−N
Is it even worth it?
27. David F. Gleich (Sandia) Parameterized Matrices 27 / 38
Convergence of approximation
SPECTRAL METHODS F
0
1+ s 0 (s) 10
s 1 1 (s)
−2
2 10
=
1
L2 Error −4
10
2−s
0 (s) =
1 + − s2 −6
10
1 + − 2s ε=0.8
1 (s) = −8 ε=0.6
1 + − s2 10
ε=0.4
ε=0.2
Convergence rate −10
10
ρ<1+ . 0 5 10 15 20 25 30
Order
28. David F. Gleich (Sandia) Parameterized Matrices 28 / 38
A Gautschi-Golub comparison
Quadrature
b m
ƒ ( )dω( ) ≈ ƒ (λj )ωj = eT ƒ (Jm )e1
1
j=1
where
Jm is the m × m Jacobi matrix for ω
J is tridiagonal, and encodes three-term
recurrence
29. David F. Gleich (Sandia) Parameterized Matrices 29 / 38
A Gautschi-Golub comparison
Pseudo-spectral Galerkin
A(Jm ) vec(X) = b(Jm )e1 [A(J∞ )]m vec(X) = [b(J∞ )]m
the notation [·]m means take
the leading m × m block of ·.
This solution is truncating This solution is truncating the
the expansion operator
Computational Implication
Given 〈(π(s)π(s)T ⊗ A(s))〉 vec(X) = 〈π(s) ⊗ b(s)〉
Approximate 〈(π(s)π(s)T ⊗ A(s))〉, and 〈π(s) ⊗ b(s)〉 with GQ?
NO! Equivalent to [A(Jm )]m =⇒ same answer.
NOTE! Both equal for linear A(s), and “low-degree” polys b(s)
30. David F. Gleich (Sandia) Parameterized Matrices 30 / 38
Computing the Galerkin solution
,j block of π(s)π(s)T ⊗A(s) = A(s)π (s)πj (s)
IDEA use M > m point quadrature.
If A(s) is a polynomial of degree d, then if
m+m+d
“M > ” not precise
2
the solution will be exact.
If A(s) is an analytic function with a rapidly
converging expansion, large M will be close.
31. David F. Gleich (Sandia) Parameterized Matrices 31 / 38
Numerically integrated Galerkin
,j block of π(s)π(s)T ⊗A(s) = A(s)π (s)πj (s)
Integrate each block with M point quadrature
After much munging with quadrature rules
π(s)π(s)T ⊗ A(s) M = (Q ⊗ )A(Λ)(Q ⊗ )T
where Q:m×M
A(λ1 )
... orthogonal rows
A(Λ) = , weighted rows
A(λM ) of J ’s eigenvecs
All we need is a function for A(·)
M
32. David F. Gleich (Sandia) Parameterized Matrices 32 / 38
Numerical Gakerkin factorization
π(s)π(s)T ⊗ A(s) M = (Q ⊗ )A(Λ)(Q ⊗ )T
Provides
computable matrix-vector product!
eigenvalue bounds on A( s)
preconditioning insights
a computable residual
33. David F. Gleich (Sandia) Parameterized Matrices 33 / 38
Parameterized Matrix Package
PMPACK
A Matlab package for
Parameterized Matrix Problems
https: //github. com/paulcon/pmpack
Implements univariate and multivariate
Galerkin and pseudo-spectral methods
Many demos Residual error estimates
Uncertainty quanti- Arbitrary polynomial
fication helpers bases (anisotropic)
Simple interface Many parameter types
35. David F. Gleich (Sandia) Parameterized Matrices 35 / 38
Where is this going?
Beyond spectral methods!
MapReduce and Surrogate Models
A surrogate model
is a function that
reproduces the
f1 Surrogate
output of a simul-
Sample
ation and predicts
its output at new f2
parameter values.
f5
The Database New Samples
The Surrogate
s1 -> f1 Extraction Interpolation sa -> fa
s2 -> f2 sb -> fb
sk -> fk Just one machine
sc -> fc
On the MapReduce cluster On the MapReduce cluster
David Gleich (Sandia) 5/5/2011 13/18
36. David F. Gleich (Sandia) Parameterized Matrices 36 / 38
Where is this going?
Parameterized Lanczos! constant!
A(s)Vk (s) = Vk+1 (s)Tk,k+1
The matrix Tk,k is the first k terms of the Ja-
cobi matrix for the weight
b(s)T A(s)b(s)
where b(s) is the first Lanczos vector.
uses Chebfun for one-parameter
multivarite methods using Monte Carlo
37. David F. Gleich (Sandia) Parameterized Matrices 37 / 38
Summary
Look at problems in uncertainty quantification
as parameterized matrices
Extended the theory of spectral methods to
the parameterized matrix case.
Devleoped software for spectral methods for
parameterized matrices.
38. Papers
Constantine, Gleich, Iaccarino. Spectral Methods
for Parametrized Matrix Problems. SIMAX, 2010.
Constantine, Gleich, Iaccarino. A Factorization of
the Spectal Galerkin System for Parameterized
Matrix Equations: Derivation and Applications.
SISC, to appear.
Constantine, Gleich. Random Alpha PageRank.
Internet Mathematics, 2010.
Code
https: //github. com/paulcon/pmpack