One important property of orthogonal sets is that they are linearly independent. To see why, suppose that we have a linear combination of the vectors in the set that equals zero: c_1 v_1 + c_2 v_2 + ... + c_n v_n = 0. Taking the dot product of both sides with v_i, we get c_i ||v_i||^2 = 0, where ||v_i|| represents the length of the vector v_i. Since the vectors are not zero, ||v_i|| ≠ 0, so we must have c_i = 0. Thus, the only linear combination that equals zero is the trivial one, which means that the vectors in the set are linearly independent. Another important property of orthogonal sets is that they can be used to form an orthonormal set, which is a set of vectors that are orthogonal and have unit length. To obtain an orthonormal set from an orthogonal set, we simply divide each vector by its length: {v_1/||v_1||, v_2/||v_2||, ..., v_n/||v_n||}. This set is still orthogonal, since the dot product of any two distinct vectors is zero, and it is orthonormal, since the length of each vector is one. An orthogonal basis is a basis of a vector space that is also an orthogonal set. That is, the basis vectors are mutually perpendicular to each other. One important property of orthogonal bases is that they are automatically orthonormal, since we can simply divide each vector by its length to obtain a unit vector. Orthogonal bases have many applications in linear algebra. For example, they can be used to find the coordinates of a vector with respect to a given basis. Suppose we have an orthogonal basis {v_1, v_2, ..., v_n} for a vector space V. Then, any vector v ∈ V can be written as a linear combination of the basis vectors: v = c_1 v_1 + c_2 v_2 + ... + c_n v_n. To find the coefficients c_i, we can take the dot product of both sides with v_i: v · v_i = (c_1 v_1 + c_2 v_2 + ... + c_n v_n) · v_i = c_i ||v_i||^2, since the dot product of any two distinct basis vectors is zero. Therefore, c_i = (v · v_i)/||v_i||^2. This formula allows us to compute the coordinates of v with respect to the basis {v_1, v_2, ..., v_n}. Orthogonal bases also have applications in diagonalization. Suppose we have a linear transformation T : V → V that is represented by a matrix A with respect to an orthogonal basis {v_1, v_2, ..., v_n}. Then, the matrix A is also orthogonal, since the columns of A are the coordinates of the basis vectors with respect to the standard basis. Since A is orthogonal, its inverse is its transpose: A^T. Therefore, we can diagonalize A by finding an orthogonal basis of

1. Orthogonality and least
squares
1. Kristina Tamang
2. Lakpa Nuru Sherpa
3. Milan Baral
4. Mukunda Rijal
5. Munesh Poudyal

2. Inner product, length and orthogonality

11. Orthogoal Sets
Orthogonal Sets
A set of vectors{u1,u2,...,up} in Rn is called anorthogonal set
if ui · uj =0 wheneveriƒ= j.
Example
1 1 0
Is −1 , 1 , 0 an orthogonalset?
0 0 1
Solution:Label the vectorsu1, u2, and u3 respectively.
u1 · u2 =
u1 · u3 =
u2 · u3=
Therefore,{u1, u2,u3} is an orthogonalset.
Then
1 /12
,

12. Orthogonal Sets: Theorem
Theorem (4)
Suppose S = {u1, u2, . . . , up} is an orthogonal set of nonzero
vectors in Rn and W =span{u1, u2,. . . , up}. Then S is a linearly
independentsetand isthereforea basisforW.
PartialProof: Suppose
c1u1 +c2u2 +· · · +cpup =0
(c1u1 +c2u2 +· · · +cpup)· = 0·
(c1u1)· u1 +(c2u2)· u1 +· · · +(cpup)· u1 = 0 c1(u1 ·
u1)+c2 (u2 · u1)+· · · +cp (up · u1) =0 c1 (u1 · u1) =
0
Since u1 ƒ=0,u1 · u1 > 0 which means c1 = .
In a similarmanner, c2,. ..,cp canbe shown toby all 0.SoS is a
linearly independentset.□
4 /12

13. Orthogonal Sets PrjectionOrthonormal Matrix
Orthogonal Basis
Orthogonal Basis: Example
An orthogonal basisforasubspace Wof Rn is a basis forWthat is
alsoan orthogonal set.
Example
Suppose S ={u1,u2,...,up} is an orthogonalbasis forasubspace
W of Rn and suppose yis in W . Find c1,. . .,cp sothat
y=c1u1 +c2u2 +· · · +cpup.
Solution:
y· =(c1u1 +c2u2 +· · · +cpup)·
y· u1=(c1u1 +c2u2 +· · · +cpup)· u1
y· u1=c1(u1 · u1)+c2(u2 · u1)+· · · +cp (up · u1)
c1 = y·u1
Similarly, c2 =
y· u1=c1 (u1 · u1) =⇒
,..., cp =
u1·u1
3 /12

14. Orthogonal Basis: Theorem
Theorem(5)
Let {u1,u2,...,up} be an orthogonalbasis fora subspaceW of
Rn. Then each y in W has a uniquerepresentationasa linear
combination of u1,u2,...,up. In fact, if
y=c1u1 +c2u2 +· · · +cpup
then y· uj
cj =u · u
j j
(j = 1, ...,p)
4 /12

15. QR Factorization

16. Proof :-
Let ‘A’ be m×n matrix with linearly independent columns,
then ‘A’ can be written product of two matrix Q and R
i.e A = QR
where Q is m×n matrix with orthonormal column vectors
and R is n×n Upper triangular matrix with positive diagonal.
i.e R = Q A
T
Theorem
If A is an m×n matrix with linearly independent column vectors, then A
can be factored as
A = QR
where Q is m×n an matrix with orthonormal column vectors, and R is
n×n an upper triangular matrix.

17. Steps to find Q
Let A=[ ]
a31
a21
a11 a12 a31
a22
a32 a33
a32
Step 1 :- Let v1 = (a11,a21,a31) , v2 = (a12,a22,a23) ,
v3 = (a31,a32,a33)
Step 2 :- Find orthogonal vectors u1 , u2 , u3
corresponding to v1, v2, v3 and orthonormal q1 , q2,
q3 corresponding to u1 , u2 , u3 .

18. Let Q=[ ]
Then, the matrix with orthonormal columns represented by
Q is
b11
b22
b31
b21
b21 b32
b31
b33
b23
Where q1 = (b11,b21,b31) , q2 = (b21,b22,b23) ,
q3=(b31,b32,b33) are orthonormal columns .
To check Q
First find Q
Then, should satisfy Q × Q = I
And for R , R = Q A
T
T
T

19. QR factorization Example :-
Let A=[ ]
0
1
1 1 0
0
1 1
1
Let v1 = (1,1,0) , v2 = (1,0,1) , v3 = (0,1,1)
For orthogonal vector u1 , u2, u3 corresponding to v1
, v2 , v3
Let u1 = v1 = (1,1,0)
A = QR (Say)

20. u2 = v2 - proj.of v2 on u1
= v2 – v2•u1 u1
ǁu1ǁ²
= (1,0,1)- (1,0,1)•(1,1,0) (1,1,0)
(1,1,0)
u2= (½ ,-½,1)
Similarly, For u3,
u3 = v3 - proj.of v3 on u1 - proj.of v3 on u2
We get ,
u3 = (-⅔ , ⅔ , ⅔ )

21. We get ,
u1 = (1,1,0) , u2= (½ ,-½,1) & u3 = (-⅔ , ⅔ , ⅔ )
To check u1 , u2 , u3 as orthogonal vectors :-
u1 • u2 = 0
u2 • u3 = 0
u3 • u1 = 0
Since they satisfy condition they are orthogonal vectors .
For orthonormal vector q1 , q2 , q3 corresponding to
u1 , u2 , u3
q1 = u1 = ( 1, 1,0) = (1 , 1 , 0 )
ǁu1ǁ √2 √2 √2

22. q2 = u2 = ( 1/2, ½,1) = (1 , 1 , √2/3 )
ǁu2ǁ √3/2 √6 √6
q3 = u3 = (-⅔ ,⅔ ,⅔ ) = (-1 , 1 , 1)
ǁu3ǁ √4/3 √3 √3 √3
To check q1 , q2 , q3 as orthogonal vectors :-
ǁq1ǁ = 1
ǁq2ǁ = 1
ǁq3ǁ = 1
Since they satisfy condition they are orthonormal vectors .

23. Q=[ ]
So orthonormal matrix Q is :-
1/√2
1/√2
0
1/√6
-1/√6
√2/3
1/√3
-1/√3
1/√3
Q=[ ]
Now, transpose of Q ,
T
1/√2
1/√6
-1/√3
1/√2 0
-1/√6
1/√3
√2/3
1/√3

24. Now,
for R, In A = QR
R = Q A
T
R=[ ] × [ ]
= [ ]
1/√2
1/√2 0
1/√6 -1/√6 √2/3
-1/√3 1/√3
1/√3
1
1
1
1 0
1
1
0
0
2/√2 1/√2 1/√2
0
0
0 3/√6 1/√6
2/√3

25. So, R = [ ]
2/√2 1/√2
1/√2
1/√6
3/√6
0
0
0 2/√3
To check If QR factorization is correct,
A = Q× R
[ ]=[ ]×[ ]
0
1
1
1
1
0 1
1
0 1/√2
1/√2
0
1/√6
-1/√6
√2/3 1/√3
1/√3
-1/√3 2/√2 1/√2 1/√2
3/√6 1/√6
2/√3
0
0
0

26. THE GRAM- SCHMIDT
PROCESS

27. The Gram-shmidt process is a simple algorithm for producing an orthogonal or
orthonormal basis for any subspace of ℝ𝑛 .
Let {𝑣1, 𝑣2} be a set of two non-orthogonal vectors. Then we can find orthogonal
vectors {𝑢1, 𝑢2} as follows:
𝑣2
𝑣1
𝑣2 − 𝑐𝑣1
Here, 𝑐 is the scalar projection of 𝑣2 on 𝑣1 i.e. c =
𝑣2.𝑣1
| 𝑣1 |
Let 𝑢1 = 𝑣1 and 𝑢2= 𝑣2 − 𝑐𝑣1
Then, clearly we can see that vectors 𝑢1 and 𝑢2 are orthogonal.

28. Generalizing this concept for 𝑛 vectors,
Let 𝑣1, 𝑣2, 𝑣3, … … … . , 𝑣𝑛 be 𝑛 vectors which are not orthogonal.
Then we construct orthogonal vectors 𝑢1, 𝑢2, 𝑢3, … … … . , 𝑢𝑛 as follows:
Let 𝑢1 = 𝑣1
𝑢2 = 𝑣2 − 𝑝𝑟𝑜𝑗𝑢1
𝑣2
𝑢3 = 𝑣3 − 𝑝𝑟𝑜𝑗𝑢1
𝑣3 − 𝑝𝑟𝑜𝑗𝑢2
𝑣3
.
.
.
𝑢𝑛 = 𝑣𝑛 − 𝑝𝑟𝑜𝑗𝑢1
𝑣𝑛 − 𝑝𝑟𝑜𝑗𝑢2
𝑣𝑛 − … … … . . 𝑝𝑟𝑜𝑗𝑢𝑛
𝑣𝑛
𝑝𝑟𝑜𝑗𝑢𝑣=
𝑣.𝑢
𝑢
2 𝑢

29. Example:
Let W = 𝑆𝑝𝑎𝑛 1, −4,0,1 , 7, −7, −4,1 , produce an orthogonal basis of W
Solution:
Let 𝑣1 = 1, −4,0,1 and 𝑣2 = 7, −7, −4,1
Using Gram-Schmidt orthogonalization process,
Let 𝑢1 = 𝑣1= 1, −4,0,1
𝑢2 = 𝑣2 − 𝑝𝑟𝑜𝑗𝑢1
𝑣2
= 𝑣2 −
𝑣2.𝑢1
𝑢1
2 𝑢1
= 7, −7, −4,1 −
1,−4,0,1 . 7,−7,−4,1
1,−4,0,1 . 1,−4,0,1
1, −4,0,1
= 7, −7, −4,1 − 2 1, −4,0,1
= (5,1, −4, −1)
Therefore, 5,1, −4, −1 , 1, −4,0,1 is an orthogonal basis of W.

30. Least squares methods
DEFINATION
If A is m×n and b is in 𝑹𝒏,a least-squares solutions of Ax=b is an x* in 𝑹𝒏 such that
||b-Ax*||<=||b-Ax||. For all x in 𝑹𝒏.
Consider system of equations
x+y=2……………….(1)
x-y=0………………..(2)
3x+y=2……………..(3)
The above system has no exact solution.
Here, Ax=b means Ax is approximately equation to b.
The least square solution is to find about those values of x must that the approaches
||b-ax*|| is minimum and satisfied ||b-Ax*||<=||b-Ax||

31. Rules to find least square solution
To find least square solution of the equation,
Ax=b………..(1)
Step 1: Let x* be the least square solution of (1)
then the normal equation of (1) is
AT Ax*=AT b.
If AT A is invertible the
x*=(AT A)-1(AT b)…………….(2)
where,
(𝐴𝑇
𝐴)−1
=
𝐴𝑑𝑔(𝐴𝑇𝐴)
|𝐴𝑇𝐴|
Find (𝐴𝑇𝐴)−1 and put on (2) and simplify to get x*.

32. Note:
 The list square solution is unique
 The difference ||b-Ax|| is called error of the system of equation Ax=b.
Find the least square solution of:
3 − 6
4 − 8
0 1
𝑥1
𝑥2
=
−1
7
2
Solution:
Let A=
3 − 6
4 − 8
0 1
and b=
−1
7
2
Ax= b…………………….(1)
Let x* be the least square solution of (1) then the normal equation is
AT Ax*=AT b
Now,
AT A=
3 4 0
−6 − 8 1
3 − 6
4 − 8
0 1
=
25 −50
−50 101

33. .
AT b=
3 4 0
−6 − 8 1
−1
7
2
=
25
−48
Now,
Let, x*=
𝑥1
𝑥2
So,
25 −50
−50 101
𝑥1
𝑥2
=
25
−48
AT Ax*= AT b=> x*= (AT A)−1
AT b
Now,
(AT A)−1
=
1
25
25
−48
Therefore,
𝑥1
𝑥2
=
1
−1.92