SVD method

1. Singular Value Decomposition



















































































T
r
V
U
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x



1
n
m
A 
is
A = (orthogonal) (diagonal) (orthogonal)
T
V
U
A 

8)
T
V
U
A 
 T
r
r
r
T
T
v
u
v
u
v
u
A 

 


 
2
2
2
1
1
1

2. Applications of the SVD
/. Image processing
G. Strang, Linear Algebra and its Applications p444












x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
picture matrix
Suppose a satellite takes a picture,
and wants to send it to earth. The
picture may contain 1000 by 1000
"pixels"—little squares each with a
definite color. We can code the colors,
in a range between black and white,
and send back 1,000,000 numbers

3. Applications of the SVD
/. Image processing
It is better to find the essential information in the 1000 by 1000
matrix, and send only that.
T
r
r
r
T
T
v
u
v
u
v
u
A 

 


 
2
2
2
1
1
1
G. Strang, Linear Algebra and its Applications p444
If only 60 terms are kept, we send 60 times 2000 numbers instead of a million.
Suppose we know the SVD.
The key is in the singular values.
Typically, some are significant and others are extremely small.
If we keep 60 and throw away 940, then we send only the
corresponding 60 columns of U, and V.
The other 940 columns are multiplied by small singular values
that are being ignored. In fact, we can do the matrix
multiplication as columns times rows:

4. Applications of the SVD
T
v
u 1
1
1

T
v
u 2
2
2
 T
v
u 3
3
3
 T
v
u 4
4
4

T
v
u 5
5
5
 T
v
u 6
6
6
 T
v
u 7
7
7

T
v
u 8
8
8

the SVD of a 32-times-32 digital image A is computed
the activities are lead by Prof. Per Christian Hansen.

5. Applications of the SVD
1
A 2
A 3
A 4
A
5
A 6
A 7
A 8
A
T
s
s
s
T
T
s v
u
v
u
v
u
A 

 


 
2
2
2
1
1
1
r
s 
G. Strang “ at first you see nothing, and suddenly you recognize everything.”

6. Applications of the SVD
T
s
s
s
T
T
s v
u
v
u
v
u
A 

 


 
2
2
2
1
1
1
r
s 

7. If A is symmetric
How to compute SVD (by hand)
Eigenvalue Decomposition
rs
eigenvecto
]
,
,
,
[ 2
1 m
S
S
S
S 

If A is real n-by-n matrix, then 1


 S
S
A
)
,
,
,
( 2
1 n
diag 

 


T
Q
Q
A 
 I
Q
QT

Example: A































2
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
5
0
0
3
4
1
1
4
Q  T
Q

8. How to compute SVD (by hand)
T
V
U
A 

T
T
T
U
V
A 

)
)(
( T
T
T
T
U
V
V
U
AA 


T
T
T
U
U
AA 






)
(
)
( T
AA
A 
 
)
)(
( T
T
T
T
V
U
U
V
A
A 


T
T
T
V
V
A
A 


)
(
)
( A
A
A T

 
symmetric
is
T
AA
symmetric
is
A
AT

9. How to compute SVD (by hand)
T
V
U
A 

T
T
T
U
V
A 

)
)(
( T
T
T
T
U
V
V
U
AA 

 T
T
T
U
U
AA 

Example:











0
0
1
1
1
1
A 







2
2
2
2
A
A
W T 0
2
2
2
2
)
det( 






I
W
0
4
2
1



















1
1
,
1
1
2
1
2
2
1
1 v
v
0
2
2
1




1
1
1
1
Av
u

 






1
1
2
2
1
A











0
2
1
2
1
1
u )
Null(AA
for
basis
l
orthonorma
, T
3
2 u
u











0
0
0
0
2
2
0
2
2
T
AA










 
0
2
1
2
1
2
u











1
0
0
3
u








































2
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
0
0
0
0
0
2
1
0
0
0
0
0
0
1
1
1
1
Range(A) Rank(A)
Null(A)
)
)(
( T
T
T
T
V
U
U
V
A
A 

 T
T
T
V
V
A
A 







)
(
)
( T
AA
A 
 
)
(
)
( A
A
A T

 






10. 















x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Matrix
Bidiagonal
Diagonal
How to compute SVD (Algorithm)
















x
x
x
x
x
















x
x
x
x
x
x
x
x
x
>>[U,S,V] = svd(A)

11. Example:
Singular Value Decomposition
9) If A is a square matrix then
0
2
1 


 n


 
1
2


A
10) If A is a square matrix then
0
2
1 


 n


 
2
2
2
2
1
2
r
F
A 

 


 
3
2

A 10

F
A



















































































T
r
V
U
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x



1

12. Singular Value Decomposition
11) If A is a square symmetric matrix then the singular values of A are
the absolute values of the eigenvalues of A.



















































































T
r
V
U
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x



1
T
T
Q
Q
Q
Q
A )
sign( 




12) If A is a square matrix then
i
n
i
A 
1
)
det(




13. SVD and Eigenvalue Decomposition
SVD Eigen Decomp
Uses two different
bases U, V
Uses just one
(eigenvectors)
Uses orthonormal bases
Generally is not
orthogonal
All matrices
(even rectangular)
Not all matrices
(even square)
(only diagonalizable)
1


 S
S
A
T
V
U
A 


14. Reduced SVD
A ̂ T
V
Û

Reduced SVD

15. Singular Value Decomposition
Example : Luke Olson.illinois
svd_test.m
iguana.jpg

16. Singular Value Decomposition
Theorem: (Singular Value Decomposition) SVD
m
m
m R
u
u
u
U 

 ]
,
,
,
[ 2
1 
If A is real m-by-n matrix, then there exist orthogonal matrices
n
n
n R
v
v
v
V 

 ]
,
,
,
[ 2
1 
such that
T
V
U
A 

where 0
2
1 


 p


  (m,n)
p min

)
,
,
( 1 p
diag 
 


A 
U T
V

Proof:

17. Singular Value Decomposition
First approximation to A is
T
v
u
A 1
1
1
1 

T
T
v
u
v
u
A 2
2
2
1
1
1
2 
 

second approximation to A is






T
T
T
v
u
v
u
v
u
A 


 

 


 
2
2
2
1
1
1






T
r
r
r
T
T
v
u
v
u
v
u
A 

 


 
2
2
2
1
1
1
2
2
1
)
(
inf
satisfies
also
matrix
the
,
0
with
any
For
:
Theroem
r
F
B
rank
R
B
F
B
A
A
A
A
r
n
m


















 


18. Singular Value Decomposition
In later years he drove a car
with the license plate:
Trefethen (Textbook author):
The SVD was discovered independently by
Beltrami(1873) and Jordan(1874) and again by
Sylvester(1889).
The SVD did not become widely known in
applied mathematics until the late 1960s, when
Golub and others showed that it could be
computed effectively.
Cleve Moler (invented MATLAB, co-founded MathWorks)
Gene Golub has done more than anyone to make the singular value
decomposition one of the most powerful and widely used tools in modern
matrix computation.