DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
tensor-decomposition
1. .
Introduction to Tensors
PFI Seminar
Kenta OONO
2012/07/11
Kenta OONO
Introduction to Tensors
2. Self Introduction
• Kenta OONO (Twitter:@delta2323 )
• PSS(Professional and Support Service) Division
• Jubatus Project
• msgpack-idl, generation of clients
• Survey and Prototyping Team
• 2010 PFI Summer Intern → 2011 Engineer
Kenta OONO
Introduction to Tensors
3. Agenda
• What is Tensor?
• Decomposition of Matrices and Tensors
• Symmetry Parametrized by Young Diagram
Kenta OONO
Introduction to Tensors
4. Agenda
• What is Tensor?
• Decomposition of Matrices and Tensors
• Symmetry Parametrized by Young Diagram
Kenta OONO
Introduction to Tensors
5. Vector as a Special Case of Tensor
Consider a n dimensional vector (Below is the case n = 3)
7
v = 4 (1)
9
When we write this vector as v = (vi ), we have v1 = 7, v2 = 4,
and v3 = 9. As v has One index, this vector is regarded as Rank 1
Tensor.
Note
• We use 1-index notation, not 0-index.
• We somtimes use rank in a different meaning, (also in the
context of tensors).
Kenta OONO
Introduction to Tensors
6. Matrix as a Special Case of Tensor
Consider a n × n Matrix (Below is the case n = 3).
7 10 3
A = 4 −1 −2 (2)
9 4 5
We write this matrix as A = (aij )1≤i,j≤3 .
• e.g. a11 = 7, a23 = −2, a32 = 4.
A has Two indices (namely i and j) and these indeices run through
1 to 3. A can be regarded as Rank 2 Tensor.
→ What happens if the number of indices are not just 1 or 2 ?
Kenta OONO
Introduction to Tensors
7. An Example of Rank 3 Tensor
Consider the set of real numbers X = (xijk )1≤i,j,k≤3 . As i, j, k run
throuth 1 to 3 independently, this set contains 3 × 3 × 3 = 27
elements.
If we fix the value of i, we can write this set in a matrix form.
7 10 3 10 2 1 5 −10 18
x1∗∗ = 4 −1 −2 , x2∗∗ = 3 −1 0 , x3∗∗ = −4 1 −2
9 4 5 8 1 4 9 −4 −10
(3)
Note
• In some field, we concatenate these matrix and name it X(1)
• This is called Flattening or Matricing of X .
Kenta OONO
Introduction to Tensors
8. Is This a Multidimensional Array?
We can realize a tensor as a multiple dimensional array in most
programming languages.
PseudoCode(C like):
int[3][3][3] x;
x[1][1][2] = -1;
Kenta OONO
Introduction to Tensors
9. Inner Product of Tensors
Let X = (xijk ), Y = (yijk ) be two rank 3 tensors and G = (g ij ) be
a symmetric (i.e. g ij = g ji ) and positive definite matrix. We can
introduce an inner product of X and Y by:
∑
n
< X , Y >= g ai g bj g ck xabc yijk (4)
a,b,c,i,j,k=1
Note:
• We can similarly define an inner product of two arbitrary rank
tensor
• X and Y must have same rank.
Kenta OONO
Introduction to Tensors
10. An Example of Inner Product (Vector)
Let’s consider the case of rank 1 tensor ( = vector). We can write
X = (xi ), Y = (yi ). And we set G = (g ij ) an identity matrix:
1 ··· 0
. .
G = . 1
. .
. (5)
0 ··· 1
In this case, this definition of inner product reduce to ordinal inner
product of two vectors, namely,
∑ ∑
< X , Y >= g ai xa yi = xi yi (6)
a,i i
Kenta OONO
Introduction to Tensors
11. An Example of Tensor
Suppose we have a (smooth) function f : R3 → R. We can derive
tensors of arbitrary rank from this function.
Taking Gradient, we obtain rank 1 tensor. (Please replace (1, 2,
3) with (x , y , z) and vice versa.
∂f
∂x
∇f = (∇i f ) = ∂y
∂f
(7)
∂f
∂z
Similarly, Hessian is an example of rank 2 tensor.
∂2f ∂2f ∂2f
∂x2∂x ∂x ∂y ∂x ∂z
H(f ) = (Hij (f )) = ∂ f ∂2f ∂2f
(8)
∂y ∂x ∂y ∂y ∂y ∂z
∂2f ∂2f ∂2f
∂z∂x ∂z∂y ∂z∂z
Kenta OONO
Introduction to Tensors
12. Supersymmetry Property
By taking derivative, we can create an arbitrary rank of tensors.
These tensors are (Super)Symmetric in the sense that
interchange of any two indices remains the tensor identical. For
example,
fx y z = fx zy = fy x z = fy zx = fzx y = fzy x . (9)
Note:
• Afterward, we write this symmetry as .
Kenta OONO
Introduction to Tensors
13. Agenda
• What is Tensor?
• Decomposition of Matrices and Tensors
• Symmetry Parametrized by Young Diagram
Kenta OONO
Introduction to Tensors
14. Decompose!
It is fundamental in mathematics to decompose some object into
simpler, easier-to-handle objects.
∑ √
• Fourier Expansion : f = an exp( −1nx )
n∈Z
∑
• Legendre Polynomial : P = ak Pk
k
1 dk 2
• where Pk (x ) = (x − 1)k
k!2k dx k
• Jordan-H¨lder : e = G0 ◁ G1 ◁ G2 ◁ · · · ◁ Gn−1 ◁ Gn = G
o
Kenta OONO
Introduction to Tensors
15. Decomposition of Matrix
There are many theories regarding the decomposition (or,
factorization) of matrices.
• SVD
• LU
• QR
• Cholesky
• Jordan
• Diagonalization ...
Although we have theories of factorization of Tensors (e.g. Tucker
Decomposition, higher-order SVD etc.) We do NOT go this
direction. Instead, we consider decomposition of matrix into
Summation of matrix.
Kenta OONO
Introduction to Tensors
20. Notation Using Young Diagram
We can symbolize symmetry and antisymmetry with Young
Diagram
7 10 3 7 7 6 0 3 −3
4 −1 −2 = 7 −1 1 + −3 0 −3
9 4 5 6 1 5 3 3 0 (13)
A Asym Aanti
The number of boxes indicates rank of the tensor (one box for
vector and two boxes for matrix).
Kenta OONO
Introduction to Tensors
21. Agenda
• What is Tensor?
• Decomposition of Matrices and Tensors
• Symmetry Parametrized by Young Diagram
Note:
• From now on, we concentrate on Rank 3 Tensors (i.e.
k = 3).
• And we assume that n = 3, that is, indices run from 1 to 3.
Kenta OONO
Introduction to Tensors
22. Symmetrizing Operator
We consider the transformation of tensor:
T : X = (xijk ) → X ′ = (xijk ),
′
(14)
′
where xijk = 1 (xijk + xikj + xjik + xjki + xkij + xkji )
6 (15)
= x(ijk) .
Kenta OONO
Introduction to Tensors
23. Young Diagram and Symmetry of Tensor (Sym. Case)
Let X = (x123 ) be a tensor of rank 3, we call X Has a Symmetry
of , if interchange of any of two indices doesn’t change each
entry of X .
Example:
• If X has a symmetry of , then
x112 = x121 = x211
(16)
x123 = x132 = x213 = x231 = x312 = x321 etc...
• Symmetric matrices have a symmetry of
Kenta OONO
Introduction to Tensors
24. Property of T
• For all X , T (X ) has symmetry of .
• If X has symmetry of , then T (X ) = X .
• For all X , T (T (X )) = T (X ).
Kenta OONO
Introduction to Tensors
25. Antisymmetrizing Operator
Next, we consider Antisymmetrization of tensors:
T : X = (xijk ) → X ′ = (xijk ),
′
(17)
′
where xijk = 1 (xijk − xikj − xjik + xjki + xkij − xkji )
6 (18)
= x[ijk] .
Kenta OONO
Introduction to Tensors
26. Young Diagram and Symmetry of Tensor (Antisym. Case)
Let X = (x123 ) be a tensor of rank k, we call X has a symmetry of
if interchange of any of two indices change only the sign of
each entry of X .
Example.
• If X has a symmetry of , then
• x123 = −x132 = −x213 = x231 = x312 = −x321
• x112 = 0, because x112 must be equal to −x112 .
• Antisymmetric matrix has a symmetry of
Kenta OONO
Introduction to Tensors
27. Orthogonality of Projection
T and T are orthogonal in the sense
T ◦ T (X ) = T ◦ T (X ) = 0. (19)
Kenta OONO
Introduction to Tensors
28. Counting Dimension
How many dimension do the vector space consisting of tensors ...
• with symmetry have ?
• with symmetry have ?
Kenta OONO
Introduction to Tensors
29. Dimensions of each subvector spaces.
If n = 3 (that is, i, j and k runs through 1 to 3), then each
dimension is as follows.
But dimension of total space is 3 × 3 × 3 = 27. This is larger than
1 + 10 = 11.
Kenta OONO
Introduction to Tensors
30. Young Diagram
Young Diagram is an arrangent of boxes of the same size so that:
• two adjacent boxes share one of their side.
• the number of boxes in each row is non-increasing.
• the number of boxes in each column is non-increasing.
Here is an example of Young Diagram with 10 boxes.
(20)
English Notation French Notation
Kenta OONO
Introduction to Tensors
31. Decomposition of Tensor (of Rank 3)
We have three types of Young Diagram which have three boxes,
namely,
(21)
, , and
Symmetric Antisymmetric ???
→ What symmetry does represent?
Kenta OONO
Introduction to Tensors
32. Coefficients of Symmetrizers
We list coneffients of each operator in a tabular form.
Coefficient of xijk
ijk ikj jik jki kij kji
1/6 1/6 1/6 1/6 1/6 1/6
1/6 -1/6 -1/6 1/6 1/6 -1/6
Sum 1
Kenta OONO
Introduction to Tensors
33. Coefficients of Symmetrizers
Coefficient of xijk
ijk ikj jik jki kij kji
1/6 1/6 1/6 1/6 1/6 1/6
1/6 -1/6 -1/6 1/6 1/6 -1/6
4/6 -2/6 -2/6
Sum 1
Kenta OONO
Introduction to Tensors
34. Symmetrizer T
We consider the transformation of tensor:
T : X = (xijk ) → X ′ = (xijk ),
′
(22)
′
where xijk = 1
(2xijk − xjki − xkij ).
3
Kenta OONO
Introduction to Tensors
35. Property of T
• For all X , T (X ) has symmetry of .
• If X has symmetry of , then T (X ) = X .
( )
• For all X , T T (X ) = T (X ).
Kenta OONO
Introduction to Tensors
36. Tensors with Symmetry of
Let X = (xijk ) be a tensor of rank 3. We say X Has a Symmetry
of , if
1
xijk = (2xijk − xjki − xkij ) (23)
3
for all 1 ≤ i, j, k ≤ n.
Kenta OONO
Introduction to Tensors
37. Where This Equation ”Symmetric” ?
Remember that X has a symmetry of , if
1
xijk = (2xijk − xjki − xkij ) (24)
3
for all 1 ≤ i, j, k ≤ n.
This equation can be transormed as follows :
1
(2xijk − xjki − xkij )
xijk =
3 (25)
⇔ 3xijk = 2xijk − xjki − xkij
⇔ xijk + xjki + xkij = 0
Kenta OONO
Introduction to Tensors
38. Projection to Each Symmetrized Tensors
Kenta OONO
Introduction to Tensors
39. Decomposition of Higher Rank Tensors
Higher rank tensors are also decompose into symmetric tensors
parametrized by Young Diagram.
⊕ ⊕
Rn×n×n×n×n =
⊕ ⊕ ⊕ ⊕
(26)
Kenta OONO
Introduction to Tensors
40. Summary
• Tensors as a Generalization of Vectors and Matrices.
• Decomposition of Matrices and Tensors
• Symmetry of Tensors Parametrized by Young Diagrams.
Kenta OONO
Introduction to Tensors
41. References
• T G. Kolda and B W. Bader, Tensor Decompositions and
Applications
• 2009, SIAM Review Vol. 51, No 3, pp 455-500
• D S. Watkins, Fundamentals of Matrix Computations 3rd. ed.
• 2010, A Wiley Series of Texts, Monographs and Tracts
• W. Fulton and J Harris, Representation Theory
• 1991, Graduate Texts in Mathematics Readins in Mathematics.
• And Google ”Representation Theory”, ”Symmetric Group”,
”Young Diagram” and so on.
Kenta OONO
Introduction to Tensors
