Orthonormal Bases and the Gram-Schmidt Process - Revision history

Khanh at 17:04, 4 November 2021

2021-11-04T17:04:32Z

Lila at 19:56, 8 October 2021

2021-10-08T19:56:37Z

Lila: Created page with "The prior subsection suggests that projecting onto the line spanned by $\vec{s}$ decomposes a vector $\vec{v}$ into two parts
2021-10-08T19:40:27Z
Created page with "The prior subsection suggests that projecting onto the line spanned by <math> \vec{s} </math> decomposes a vector <math>\vec{v}</math> into two parts <center> <TABLE border=0p..."
New page
The prior subsection suggests
that projecting onto the line spanned by <math> \vec{s} </math>
decomposes a vector <math>\vec{v}</math> into two parts
<center>
<TABLE border=0px cellspacing=20px>
<TR>
<TD>[[Image:Linalg_projection_and_orthog.png|x200px]]
<TD><math>\displaystyle \vec{v}=\mbox{proj}_{[\vec{s}\,]}({\vec{v}})
\,+\,\left(\vec{v}-\mbox{proj}_{[\vec{s}\,]}({\vec{v}})\right)</math>
</TABLE>
</center>
that are orthogonal and so are "not interacting".
We will now develop that suggestion.

{{TextBox|1=
;Definition 2.1{{anchor|def:mutually orthogonal}}:
Vectors <math> \vec{v}_1,\dots,\vec{v}_k\in\mathbb{R}^n </math> are '''mutually orthogonal''' when any two are orthogonal: if <math> i\neq j </math> then the dot product <math> \vec{v}_i\cdot\vec{v}_j </math> is zero.
}}

{{TextBox|1=
;Theorem 2.2{{anchor|th:OrthoIsInd}}: 
If the vectors in a set <math> \{\vec{v}_1,\dots,\vec{v}_k\}\subset\mathbb{R}^n </math> are mutually orthogonal and nonzero then that set is linearly independent.
}}

{{TextBox|1=
;Proof:
Consider a linear relationship
<math> c_1\vec{v}_1+c_2\vec{v}_2+\dots+c_k\vec{v}_k=\vec{0} </math>.
If <math>i\in [1..k]</math> then taking the dot product of <math> \vec{v}_i </math>
with both sides of the equation

:<math>\begin{array}{rl}
\vec{v}_i\cdot (c_1\vec{v}_1+c_2\vec{v}_2+\dots+c_k\vec{v}_k)
&=\vec{v}_i\cdot\vec{0} \\
c_i\cdot(\vec{v}_i\cdot\vec{v}_i)
&=0
\end{array}</math>

shows, since <math> \vec{v}_i </math> is nonzero, that <math> c_i </math> is zero.
}}

{{TextBox|1=
;Corollary 2.3{{anchor|cor:OrthAndBigEnoughIsBasis}}:
If the vectors in a size <math> k </math> subset of a <math>k</math> dimensional space are mutually orthogonal and nonzero then that set is a basis for the space.
}}

{{TextBox|1=
;Proof:
Any linearly independent size <math> k </math> subset of a <math>k</math> dimensional space is a basis.
}}

Of course, the converse of [[#cor:OrthAndBigEnoughIsBasis|Corollary 2.3]]
does not hold— not every basis of every subspace
of <math>\mathbb{R}^n</math> is made of mutually orthogonal vectors.
However, we can get the partial converse
that for every subspace of <math>\mathbb{R}^n</math> there is at least one basis
consisting of mutually orthogonal vectors.

{{TextBox|1=
;Example 2.4:
The members <math>\vec{\beta}_1</math> and <math>\vec{\beta}_2</math> of this basis for <math>\mathbb{R}^2</math>
are not orthogonal.
<center>
<TABLE border=0px cellspacing=20px>
<TR>
<TD><math>\displaystyle
B=\left\langle
\begin{pmatrix} 4 \\ 2 \end{pmatrix},
\begin{pmatrix} 1 \\ 3 \end{pmatrix} \right\rangle </math>
<TD>[[Image:Linalg_non_orthog_basis_R2.png|x200px]]
</TABLE>
</center>
However, we can derive from <math>B</math>
a new basis for the same space that does have mutually orthogonal members.
For the first member of the new basis we simply use <math>\vec{\beta}_1</math>.

:<math>
\vec{\kappa}_1=\begin{pmatrix} 4 \\ 2 \end{pmatrix}
</math>

For the second member of the new basis,
we take away from <math>\vec{\beta}_2</math> its part in the direction of
<math>\vec{\kappa}_1</math>,
<center>
<TABLE border=0px cellspacing=20px>
<TR>
<TD><math>\displaystyle \vec{\kappa}_2=\begin{pmatrix} 1 \\ 3 \end{pmatrix}
-\mbox{proj}_{\scriptstyle[\vec{\kappa}_1]}({\begin{pmatrix} 1 \\ 3 \end{pmatrix}})
=\begin{pmatrix} 1 \\ 3 \end{pmatrix}-\begin{pmatrix} 2 \\ 1 \end{pmatrix}=\begin{pmatrix} -1 \\ 2 \end{pmatrix}</math>
<TD>[[Image:Linalg_non_orthog_basis_R2_2.png|x200px]]
</TABLE>
</center>
which leaves the part, <math>\vec{\kappa}_2</math> pictured above, of <math>\vec{\beta}_2</math> that is orthogonal to <math>\vec{\kappa}_1</math> (it is orthogonal by the definition of the projection onto the span of <math>\vec{\kappa}_1</math>). Note that, by the corollary, <math>\{\vec{\kappa}_1,\vec{\kappa}_2\}</math> is a basis for <math>\mathbb{R}^2</math>.
}}

{{TextBox|1=
;Definition 2.5{{anchor|def:orthogonal basis}}:
An '''orthogonal basis''' for a vector space is a basis of mutually orthogonal vectors.
}}

{{TextBox|1=
;Example 2.6{{anchor|ex:OrthoBasisForReThree}}: 
To turn this basis for <math> \mathbb{R}^3 </math>

:<math>
\left\langle
\begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix},
\begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix},
\begin{pmatrix} 1 \\ 0 \\ 3 \end{pmatrix}
\right\rangle
</math>

into an orthogonal basis, we take the first vector as it is given.

:<math>
\vec{\kappa}_1=\begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix}
</math>

We get <math> \vec{\kappa}_2 </math> by starting with the given second vector
<math>\vec{\beta}_2</math> and
subtracting away the part of it in the direction of <math> \vec{\kappa}_1 </math>.

:<math>
\vec{\kappa}_2=\begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix}
-\mbox{proj}_{[\vec{\kappa}_1]}({\begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix}})
=\begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix}-\begin{pmatrix} 2/3 \\ 2/3 \\ 2/3 \end{pmatrix}
=\begin{pmatrix} -2/3 \\ 4/3 \\ -2/3 \end{pmatrix}
</math>

Finally, we get <math>\vec{\kappa}_3</math>
by taking the third given vector and subtracting the part of it
in the direction of <math> \vec{\kappa}_1 </math>, and also the part
of it in the direction of <math> \vec{\kappa}_2 </math>.

:<math>
\vec{\kappa}_3=\begin{pmatrix} 1 \\ 0 \\ 3 \end{pmatrix}
-\mbox{proj}_{[\vec{\kappa}_1]}({\begin{pmatrix} 1 \\ 0 \\ 3 \end{pmatrix}})
-\mbox{proj}_{[\vec{\kappa}_2]}({\begin{pmatrix} 1 \\ 0 \\ 3 \end{pmatrix}})
=\begin{pmatrix} -1 \\ 0 \\ 1 \end{pmatrix}
</math>

Again the corollary gives that

:<math>
\left\langle
\begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix},
\begin{pmatrix} -2/3 \\ 4/3 \\ -2/3 \end{pmatrix},
\begin{pmatrix} -1 \\ 0 \\ 1 \end{pmatrix}
\right\rangle
</math>

is a basis for the space.
}}

The next result verifies that
the process used in those examples works with any basis for any
subspace of an <math>\mathbb{R}^n</math>
(we are restricted to <math>\mathbb{R}^n</math> only because we have not given a
definition of orthogonality for other vector spaces).

{{TextBox|1=
;Theorem 2.7 (Gram-Schmidt orthogonalization){{anchor|th:GramSchmidt}}:
If <math> \left\langle \vec{\beta}_1,\ldots\vec{\beta}_k \right\rangle </math>
is a basis for a subspace of <math> \mathbb{R}^n </math> then, where

:<math>\begin{array}{rl}
\vec{\kappa}_1
&=\vec{\beta}_1 \\
\vec{\kappa}_2
&=\vec{\beta}_2-\mbox{proj}_{[\vec{\kappa}_1]}({\vec{\beta}_2}) \\
\vec{\kappa}_3
&=\vec{\beta}_3
-\mbox{proj}_{[\vec{\kappa}_1]}({\vec{\beta}_3})
-\mbox{proj}_{[\vec{\kappa}_2]}({\vec{\beta}_3}) \\
&\vdots \\
\vec{\kappa}_k
&=\vec{\beta}_k
-\mbox{proj}_{[\vec{\kappa}_1]}({\vec{\beta}_k})-\cdots
-\mbox{proj}_{[\vec{\kappa}_{k-1}]}({\vec{\beta}_k})
\end{array}</math>

the <math> \vec{\kappa}\, </math>'s form an orthogonal basis for the same subspace.
}}

{{TextBox|1=
;Proof:
We will use induction to check that each <math> \vec{\kappa}_i </math> is nonzero,
is in the span of <math>\left\langle \vec{\beta}_1,\ldots\vec{\beta}_i \right\rangle </math>
and is orthogonal to all preceding vectors:
<math> \vec{\kappa}_1\cdot\vec{\kappa}_i= \cdots
=\vec{\kappa}_{i-1}\cdot\vec{\kappa}_{i}=0 </math>.
With those, and with
[[#cor:OrthAndBigEnoughIsBasis|Corollary 2.3]], we will have that
<math> \left\langle \vec{\kappa}_1,\ldots\vec{\kappa}_k \right\rangle </math>
is a basis for the same space as
<math> \left\langle \vec{\beta}_1,\ldots\vec{\beta}_k \right\rangle </math>.

We shall cover the cases up to <math> i=3 </math>, which give the
sense of the argument.
Completing the details is [[#exer:GramSchmidt|Problem 15]].

The <math> i=1 </math> case is trivial— setting <math> \vec{\kappa}_1 </math> equal to
<math> \vec{\beta}_1 </math>
makes it a nonzero vector since <math>\vec{\beta}_1</math> is a member of a basis,
it is obviously in the desired span,
and the "orthogonal to all preceding vectors" condition is vacuously met.

For the <math> i=2 </math> case, expand the definition of <math>\vec{\kappa}_2</math>.

:<math>
\vec{\kappa}_2=\vec{\beta}_2
-\mbox{proj}_{[\vec{\kappa}_1]}({\vec{\beta}_2})=
\vec{\beta}_2
-\frac{\vec{\beta}_2\cdot\vec{\kappa}_1}{
\vec{\kappa}_1\cdot\vec{\kappa}_1}
\cdot\vec{\kappa}_1
=
\vec{\beta}_2
-\frac{\vec{\beta}_2\cdot\vec{\kappa}_1}{
\vec{\kappa}_1\cdot\vec{\kappa}_1}
\cdot\vec{\beta}_1
</math>

This expansion shows that <math>\vec{\kappa}_2</math> is nonzero or else this
would be a non-trivial linear dependence among the <math> \vec{\beta} </math>'s
(it is nontrivial because the coefficient of <math>\vec{\beta}_2</math> is <math>1</math>)
and also shows that
<math>\vec{\kappa}_2</math> is in the desired span.
Finally, <math>\vec{\kappa}_2</math> is orthogonal to the only preceding vector

:<math>
\vec{\kappa}_1\cdot\vec{\kappa}_2=
\vec{\kappa}_1\cdot(\vec{\beta}_2
-\mbox{proj}_{[\vec{\kappa}_1]}({\vec{\beta}_2}))=0
</math>

because this projection is orthogonal.

The <math> i=3 </math> case is the same as the <math>i=2</math> case except for one detail.
As in the <math>i=2</math> case, expanding the definition

:<math>\begin{array}{rl}
\vec{\kappa}_3
&=\vec{\beta}_3
-\frac{\vec{\beta}_3\cdot\vec{\kappa}_1}{
\vec{\kappa}_1\cdot\vec{\kappa}_1}
\cdot\vec{\kappa}_1
-\frac{\vec{\beta}_3\cdot\vec{\kappa}_2}{
\vec{\kappa}_2\cdot\vec{\kappa}_2}
\cdot\vec{\kappa}_2 \\
&=\vec{\beta}_3
-\frac{\vec{\beta}_3\cdot\vec{\kappa}_1}{
\vec{\kappa}_1\cdot\vec{\kappa}_1}
\cdot\vec{\beta}_1
-\frac{\vec{\beta}_3\cdot\vec{\kappa}_2}{
\vec{\kappa}_2\cdot\vec{\kappa}_2}
\cdot\bigl(\vec{\beta}_2
-\frac{\vec{\beta}_2\cdot\vec{\kappa}_1}{
\vec{\kappa}_1\cdot\vec{\kappa}_1}
\cdot\vec{\beta}_1\bigr)
\end{array}</math>

shows that <math>\vec{\kappa}_3</math> is nonzero and is in the span.
A calculation shows that
<math>\vec{\kappa}_3</math> is orthogonal to the preceding vector <math>\vec{\kappa}_1</math>.

:<math>\begin{array}{rl}
\vec{\kappa}_1\cdot\vec{\kappa}_3
&=\vec{\kappa}_1\cdot\bigl(\vec{\beta}_3
-\mbox{proj}_{[\vec{\kappa}_1]}({\vec{\beta}_3})
-\mbox{proj}_{[\vec{\kappa}_2]}({\vec{\beta}_3})\bigr) \\
&=\vec{\kappa}_1\cdot\bigl(\vec{\beta}_3
-\mbox{proj}_{[\vec{\kappa}_1]}({\vec{\beta}_3})\bigr)
-\vec{\kappa}_1\cdot\mbox{proj}_{[\vec{\kappa}_2]}({\vec{\beta}_3}) \\
&=0
\end{array}</math>

(Here's the difference from the <math>i=2</math> case— the second line has
two kinds of terms.
The first term is zero because this projection is orthogonal, as in the
<math>i=2</math> case.
The second term is zero because <math> \vec{\kappa}_1 </math> is orthogonal
to <math> \vec{\kappa}_2 </math> and so is orthogonal to any vector
in the line spanned by <math> \vec{\kappa}_2 </math>.)
The check that <math>\vec{\kappa}_3</math> is also
orthogonal to the other preceding vector <math>\vec{\kappa}_2</math> is similar.
}}

{{anchor|normalize}}Beyond having the vectors in the basis be orthogonal, we can do more; we can arrange for each vector to have length one by dividing each by its own length (we can '''normalize''' the lengths).

{{TextBox|1=
;Example 2.8{{anchor|orthonormal}}:
Normalizing the length of each vector in the orthogonal basis of
[[#ex:OrthoBasisForReThree|Example 2.6]]
produces this '''orthonormal basis'''.

:<math>
\left\langle
\begin{pmatrix} 1/\sqrt{3} \\ 1/\sqrt{3} \\ 1/\sqrt{3} \end{pmatrix},
\begin{pmatrix} -1/\sqrt{6} \\ 2/\sqrt{6} \\ -1/\sqrt{6} \end{pmatrix},
\begin{pmatrix} -1/\sqrt{2} \\ 0 \\ 1/\sqrt{2} \end{pmatrix}
\right\rangle
</math>
}}

Besides its intuitive appeal, and its analogy with the
standard basis <math>\mathcal{E}_n</math> for <math>\mathbb{R}^n</math>, an orthonormal basis also simplifies
some computations.

==Resources==
* [https://en.wikibooks.org/wiki/Linear_Algebra/Gram-Schmidt_Orthogonalization Gram-Schmidt Orthogonalization], Wikibooks: Linear Algebra

@@ Line 300: / Line 300: @@
 some computations.
-==Resources==
+== Licensing ==
-* [https://en.wikibooks.org/wiki/Linear_Algebra/Gram-Schmidt_Orthogonalization Gram-Schmidt Orthogonalization], Wikibooks: Linear Algebra
+Content obtained and/or adapted from:

@@ Line 13: / Line 13: @@
 We will now develop that suggestion.
-{{TextBox|1=
+<blockquote style="background: white; border: 1px solid black; padding: 1em;">
-;Definition 2.1{{anchor|def:mutually orthogonal}}:
+'''Definition 2.1'''
-Vectors <math> \vec{v}_1,\dots,\vec{v}_k\in\mathbb{R}^n </math> are '''mutually orthogonal''' when any two are orthogonal: if <math> i\neq j </math> then the dot product <math> \vec{v}_i\cdot\vec{v}_j </math> is zero.
+:Vectors <math> \vec{v}_1,\dots,\vec{v}_k\in\mathbb{R}^n </math> are '''mutually orthogonal''' when any two are orthogonal: if <math> i\neq j </math> then the dot product <math> \vec{v}_i\cdot\vec{v}_j </math> is zero.
-}}
+</blockquote>
-{{TextBox|1=
+<blockquote style="background: white; border: 1px solid black; padding: 1em;">
-;Theorem 2.2{{anchor|th:OrthoIsInd}}: <!--\label{th:OrthoIsInd}-->
+'''Theorem 2.2'''
-If the vectors in a set <math> \{\vec{v}_1,\dots,\vec{v}_k\}\subset\mathbb{R}^n </math> are mutually orthogonal and nonzero then that set is linearly independent.
+: If the vectors in a set <math> \{\vec{v}_1,\dots,\vec{v}_k\}\subset\mathbb{R}^n </math> are mutually orthogonal and nonzero then that set is linearly independent.
-}}
+</blockquote>
-{{TextBox|1=
+Proof:
 Consider a linear relationship
 <math> c_1\vec{v}_1+c_2\vec{v}_2+\dots+c_k\vec{v}_k=\vec{0} </math>.
@@ Line 38: / Line 37: @@
 shows, since <math> \vec{v}_i </math> is nonzero, that <math> c_i </math> is zero.
-{{TextBox|1=
+<blockquote style="background: white; border: 1px solid black; padding: 1em;">
-;Corollary 2.3{{anchor|cor:OrthAndBigEnoughIsBasis}}:<!--\label{cor:OrthAndBigEnoughIsBasis}-->
+'''Corollary 2.3'''
-If the vectors in a size <math> k </math> subset of a <math>k</math> dimensional space are mutually orthogonal and nonzero then that set is a basis for the space.
+: If the vectors in a size <math> k </math> subset of a <math>k</math> dimensional space are mutually orthogonal and nonzero then that set is a basis for the space.
-}}
+</blockquote>
-{{TextBox|1=
+Proof:
 Any linearly independent size <math> k </math> subset of a <math>k</math> dimensional space is a basis.
-Of course, the converse of [[#cor:OrthAndBigEnoughIsBasis|Corollary 2.3]]<!--\ref{cor:OrthAndBigEnoughIsBasis}-->
+Of course, the converse of Corollary 2.3
 does not hold&mdash; not every basis of every subspace
 of <math>\mathbb{R}^n</math> is made of mutually orthogonal vectors.
@@ Line 57: / Line 53: @@
 consisting of mutually orthogonal vectors.
-{{TextBox|1=
+<blockquote style="background: white; border: 1px solid black; padding: 1em;">
-;Example 2.4:
+'''Example 2.4''':
 The members <math>\vec{\beta}_1</math> and <math>\vec{\beta}_2</math> of this basis for <math>\mathbb{R}^2</math>
 are not orthogonal.
@@ Line 92: / Line 88: @@
 </center>
 which leaves the part, <math>\vec{\kappa}_2</math> pictured above, of <math>\vec{\beta}_2</math> that is orthogonal to <math>\vec{\kappa}_1</math> (it is orthogonal by the definition of the projection onto the span of <math>\vec{\kappa}_1</math>). Note that, by the corollary, <math>\{\vec{\kappa}_1,\vec{\kappa}_2\}</math> is a basis for <math>\mathbb{R}^2</math>.
-}}
+</blockquote>
-{{TextBox|1=
+<blockquote style="background: white; border: 1px solid black; padding: 1em;">
-;Definition 2.5{{anchor|def:orthogonal basis}}:
+'''Definition 2.5''':
-An '''orthogonal basis''' for a vector space is a basis of mutually orthogonal vectors.
+:An '''orthogonal basis''' for a vector space is a basis of mutually orthogonal vectors.
-}}
+</blockquote>
-{{TextBox|1=
+<blockquote style="background: white; border: 1px solid black; padding: 1em;">
-;Example 2.6{{anchor|ex:OrthoBasisForReThree}}: <!--\label{ex:OrthoBasisForReThree}-->
+'''Example 2.6''':
 To turn this basis for <math> \mathbb{R}^3 </math>
@@ Line 151: / Line 147: @@
 is a basis for the space.
-}}
+</blockquote>
 The next result verifies that
@@ Line 159: / Line 155: @@
 definition of orthogonality for other vector spaces).
-{{TextBox|1=
+<blockquote style="background: white; border: 1px solid black; padding: 1em;">
-;Theorem 2.7 (Gram-Schmidt orthogonalization){{anchor|th:GramSchmidt}}:<!--\label{th:GramSchmidt}-->
+'''Theorem 2.7 (Gram-Schmidt orthogonalization)''':
-If <math> \left\langle \vec{\beta}_1,\ldots\vec{\beta}_k \right\rangle  </math>
+:If <math> \left\langle \vec{\beta}_1,\ldots\vec{\beta}_k \right\rangle  </math>
 is a basis for a subspace of <math> \mathbb{R}^n </math> then, where
@@ Line 181: / Line 177: @@
 the <math> \vec{\kappa}\, </math>'s form an orthogonal basis for the same subspace.
-}}
+</blockquote>
-{{TextBox|1=
+Proof:
 We will use induction to check that each <math> \vec{\kappa}_i </math> is nonzero,
 is in the span of <math>\left\langle \vec{\beta}_1,\ldots\vec{\beta}_i \right\rangle </math>
@@ Line 285: / Line 280: @@
 The check that <math>\vec{\kappa}_3</math> is also
 orthogonal to the other preceding vector <math>\vec{\kappa}_2</math> is similar.
-{{anchor|normalize}}Beyond having the vectors in the basis be orthogonal, we can do more; we can arrange for each vector to have length one by dividing each by its own length (we can '''normalize''' the lengths).
+Beyond having the vectors in the basis be orthogonal, we can do more; we can arrange for each vector to have length one by dividing each by its own length (we can '''normalize''' the lengths).
-{{TextBox|1=
+<blockquote style="background: white; border: 1px solid black; padding: 1em;">
-;Example 2.8{{anchor|orthonormal}}:
+'''Example 2.8''':
-Normalizing the length of each vector in the orthogonal basis of
+Normalizing the length of each vector in the orthogonal basis of Example 2.6 produces this '''orthonormal basis'''.
 :<math>
@@ Line 302: / Line 294: @@
 \right\rangle
 </math>
-}}
+</blockquote>
 Besides its intuitive appeal, and its analogy with the