Matrix Algebra and Matrix Multiplication

Matrix Operations

Adding and subtracting matrices

In order to add or subtract two matrices, they must be of the same dimension; that is, the two matrices must have the same number of rows and the same number of columns. To add two matrices together, we simply need to add every entry in one matrix to the entry in the same row and same column in the other matrix. For example:

${\begin{bmatrix}a_{11}&a_{12}&a_{13}\\a_{21}&a_{22}&a_{23}\\a_{31}&a_{32}&a_{33}\end{bmatrix}}+{\begin{bmatrix}b_{11}&b_{12}&b_{13}\\b_{21}&b_{22}&b_{23}\\b_{31}&b_{32}&b_{33}\end{bmatrix}}={\begin{bmatrix}a_{11}+b_{11}&a_{12}+b_{12}&a_{13}+b_{13}\\a_{21}+b_{21}&a_{22}+b_{22}&a_{23}+b_{23}\\a_{31}+b_{31}&a_{32}+b_{32}&a_{33}+b_{33}\end{bmatrix}}$

${\begin{bmatrix}1&2&3\\0&1&2\\0&0&1\end{bmatrix}}+{\begin{bmatrix}5&2&1\\3&3&1\\4&2&0\end{bmatrix}}={\begin{bmatrix}1+5&2+2&3+1\\0+3&1+3&2+1\\0+4&0+2&1+0\end{bmatrix}}={\begin{bmatrix}6&4&4\\3&4&3\\4&2&1\end{bmatrix}}$

Multiplying matrices by scalars

When multiplying a matrix by a scalar (or number), all we need to do is multiply each entry of the matrix by the scalar. For example:

$3{\begin{bmatrix}1&2&3\\0&1&2\\0&0&1\end{bmatrix}}={\begin{bmatrix}3&6&9\\0&3&6\\0&0&3\end{bmatrix}}$

$-{\frac {1}{2}}{\begin{bmatrix}2&-4&8\\-2&1&-6\\0&0&7\end{bmatrix}}={\begin{bmatrix}-1&2&-4\\1&-{\frac {1}{2}}&3\\0&0&-{\frac {7}{2}}\end{bmatrix}}$

Multiplying matrices

Matrix multiplication is not commutative; that is, for most matrices $A$ and $B$, $AB\neq BA$. Two matrices can only be multiplied together if the number of columns in the first matrix equals the number of rows in the second matrix. For example, we can take the product of a 3-by-2 matrix times a 2-by-5 matrix, but not a 3-by-2 matrix times a 3-by-2 matrix. The product of two matrices has the same number of rows as the first matrix and the same number of columns as the second matrix. For example, a 2-by-3 matrix times a 3-by-2 matrix will result in a 2-by-2 matrix.

The product of two 3-by-3 matrices is ${\begin{bmatrix}a_{11}&a_{12}&a_{13}\\a_{21}&a_{22}&a_{23}\\a_{31}&a_{32}&a_{33}\end{bmatrix}}{\begin{bmatrix}b_{11}&b_{12}&b_{13}\\b_{21}&b_{22}&b_{23}\\b_{31}&b_{32}&b_{33}\end{bmatrix}}={\begin{bmatrix}c_{11}&c_{12}&c_{13}\\c_{21}&c_{22}&c_{23}\\c_{31}&c_{32}&c_{33}\end{bmatrix}}$

where $c_{ij}$ is the dot product of the i-th row of the first matrix and the j-th column of the second matrix; that is, $c_{ij}=a_{i1}b_{1j}+a_{i2}b_{2j}+a_{i3}b_{3j}$.

Here is a 2-by-2 example of matrix multiplication:

${\begin{bmatrix}1&2\\3&4\end{bmatrix}}{\begin{bmatrix}5&6\\7&8\end{bmatrix}}={\begin{bmatrix}1(5)+2(7)&1(6)+2(8)\\3(5)+4(7)&3(6)+4(8)\end{bmatrix}}={\begin{bmatrix}5+14&6+16\\15+28&18+32\end{bmatrix}}{\begin{bmatrix}19&22\\43&50\end{bmatrix}}$

Note that there is no "matrix division". Rather, we multiply one matrix by the inverse of another (for example, $AB^{-1}$, NOT $A/B$) to obtain a "quotient" from two matrices.

Inverse of an n-by-n matrix

An n-by-n matrix A is the inverse of n-by-n matrix B (and B the inverse of A) if BA = AB = I, where I is an identity matrix.

The inverse of an n-by-n matrix can be calculated by creating an n-by-2n matrix which has the original matrix on the left and the identity matrix on the right. Row reduce this matrix and the right half will be the inverse. If the matrix does not row reduce completely (i.e., a row is formed with all zeroes as its entries), it does not have an inverse.

Example 1

Let $\mathrm {A} ={\begin{bmatrix}1&4&4\\2&5&8\\3&6&9\end{bmatrix}}$

We begin by expanding and partitioning A to include the identity matrix, and then proceed to row reduce A until we reach the identity matrix on the left-hand side.

${\begin{bmatrix}1&4&4&{\big |}&1&0&0\\2&5&8&{\big |}&0&1&0\\3&6&9&{\big |}&0&0&1\end{bmatrix}}\rightsquigarrow {\begin{bmatrix}1&4&4&{\big |}&1&0&0\\0&-3&0&{\big |}&-2&1&0\\0&-6&-3&{\big |}&-3&0&1\end{bmatrix}}\rightsquigarrow {\begin{bmatrix}1&4&4&{\big |}&1&0&0\\0&1&0&{\big |}&2/3&-1/3&0\\0&-6&-3&{\big |}&-3&0&1\end{bmatrix}}\rightsquigarrow$

${\begin{bmatrix}1&0&4&{\big |}&-5/3&4/3&0\\0&1&0&{\big |}&2/3&-1/3&0\\0&0&-3&{\big |}&1&-2&1\end{bmatrix}}\rightsquigarrow {\begin{bmatrix}1&0&4&{\big |}&-5/3&4/3&0\\0&1&0&{\big |}&2/3&-1/3&0\\0&0&1&{\big |}&-1/3&2/3&-1/3\end{bmatrix}}\rightsquigarrow {\begin{bmatrix}1&0&0&{\big |}&-1/3&-4/3&4/3\\0&1&0&{\big |}&2/3&-1/3&0\\0&0&1&{\big |}&-1/3&2/3&-1/3\end{bmatrix}}$

The matrix $\mathrm {B} ={\begin{bmatrix}-1/3&-4/3&4/3\\2/3&-1/3&0\\-1/3&2/3&-1/3\end{bmatrix}}$ is then the inverse of the original matrix A.

Inverse of a Linear Transformation

We now consider how to represent the inverse of a linear map. We start by recalling some facts about function inverses. Some functions have no inverse, or have an inverse on the left side or right side only.

Definitions

Where $\pi :\mathbb {R} ^{3}\to \mathbb {R} ^{2}$ is the projection map

${\begin{pmatrix}x\\y\\z\end{pmatrix}}\mapsto {\begin{pmatrix}x\\y\end{pmatrix}}$

and $\eta :\mathbb {R} ^{2}\to \mathbb {R} ^{3}$ is the embedding

${\begin{pmatrix}x\\y\end{pmatrix}}\mapsto {\begin{pmatrix}x\\y\\0\end{pmatrix}}$

the composition $\pi \circ \eta$ is the identity map on $\mathbb {R} ^{2}$.

${\begin{pmatrix}x\\y\end{pmatrix}}{\stackrel {\eta }{\longmapsto }}{\begin{pmatrix}x\\y\\0\end{pmatrix}}{\stackrel {\pi }{\longmapsto }}{\begin{pmatrix}x\\y\end{pmatrix}}$

We say $\pi$ is a left inverse map of $\eta$ or, what is the same thing, that $\eta$ is a right inverse map of $\pi$. However, composition in the other order $\eta \circ \pi$ doesn't give the identity map— here is a vector that is not sent to itself under $\eta \circ \pi$.

${\begin{pmatrix}0\\0\\1\end{pmatrix}}{\stackrel {\pi }{\longmapsto }}{\begin{pmatrix}0\\0\end{pmatrix}}{\stackrel {\eta }{\longmapsto }}{\begin{pmatrix}0\\0\\0\end{pmatrix}}$

In fact, the projection $\pi$ has no left inverse at all. For, if $f$ were to be a left inverse of $\pi$ then we would have

${\begin{pmatrix}x\\y\\z\end{pmatrix}}{\stackrel {\pi }{\longmapsto }}{\begin{pmatrix}x\\y\end{pmatrix}}{\stackrel {f}{\longmapsto }}{\begin{pmatrix}x\\y\\z\end{pmatrix}}$

for all of the infinitely many $z$'s. But no function $f$ can send a single argument to more than one value.

(An example of a function with no inverse on either side is the zero transformation on $\mathbb {R} ^{2}$.)

Some functions have a two-sided inverse map, another function that is the inverse of the first, both from the left and from the right. For instance, the map given by ${\vec {v}}\mapsto 2\cdot {\vec {v}}$ has the two-sided inverse ${\vec {v}}\mapsto (1/2)\cdot {\vec {v}}$. In this subsection we will focus on two-sided inverses. The appendix shows that a function has a two-sided inverse if and only if it is both one-to-one and onto. The appendix also shows that if a function $f$ has a two-sided inverse then it is unique, and so it is called "the" inverse, and is denoted $f^{-1}$. So our purpose in this subsection is, where a linear map $h$ has an inverse, to find the relationship between ${\rm {Rep}}_{B,D}(h)$ and ${\rm {Rep}}_{D,B}(h^{-1})$.

A matrix $G$ is a left inverse matrix of the matrix $H$ if $GH$ is the identity matrix. It is a right inverse matrix if $HG$ is the identity. A matrix $H$ with a two-sided inverse is an invertible matrix. That two-sided inverse is called the inverse matrix and is denoted $H^{-1}$.

Because of the correspondence between linear maps and matrices, statements about map inverses translate into statements about matrix inverses.

Lemmas and Theorems

• If a matrix has both a left inverse and a right inverse then the two are equal.

• A matrix is invertible if and only if it is nonsingular.
  • Proof: (For both results.) Given a matrix $H$, fix spaces of appropriate dimension for the domain and codomain. Fix bases for these spaces. With respect to these bases, $H$ represents a map $h$. The statements are true about the map and therefore they are true about the matrix.

• A product of invertible matrices is invertible— if $G$ and $H$ are invertible and if $GH$ is defined then $GH$ is invertible and $(GH)^{-1}=H^{-1}G^{-1}$.
  • Proof: (This is just like the prior proof except that it requires two maps.) Fix appropriate spaces and bases and consider the represented maps $h$ and $g$. Note that $h^{-1}g^{-1}$ is a two-sided map inverse of $gh$ since $(h^{-1}g^{-1})(gh)=h^{-1}({\mbox{id}})h=h^{-1}h={\mbox{id}}$ and $(gh)(h^{-1}g^{-1})=g({\mbox{id}})g^{-1}=gg^{-1}={\mbox{id}}$. This equality is reflected in the matrices representing the maps, as required.

Here is the arrow diagram giving the relationship between map inverses and matrix inverses. It is a special case of the diagram for function composition and matrix multiplication.

Beyond its place in our general program of seeing how to represent map operations, another reason for our interest in inverses comes from solving linear systems. A linear system is equivalent to a matrix equation, as here.

${\begin{array}{*{2}{rc}r}x_{1}&+&x_{2}&=&3\\2x_{1}&-&x_{2}&=&2\end{array}}\quad \Longleftrightarrow \quad {\begin{pmatrix}1&1\\2&-1\end{pmatrix}}{\begin{pmatrix}x_{1}\\x_{2}\end{pmatrix}}={\begin{pmatrix}3\\2\end{pmatrix}}\qquad \qquad (*)$

By fixing spaces and bases (e.g., $\mathbb {R} ^{2},\mathbb {R} ^{2}$ and ${\mathcal {E}}_{2},{\mathcal {E}}_{2}$), we take the matrix $H$ to represent some map $h$. Then solving the system is the same as asking: what domain vector ${\vec {x}}$ is mapped by $h$ to the result ${\vec {d}}\,$? If we could invert $h$ then we could solve the system by multiplying ${\rm {Rep}}_{D,B}(h^{-1})\cdot {\rm {Rep}}_{D}({\vec {d}})$ to get ${\rm {Rep}}_{B}({\vec {x}})$.

Example 2

We can find a left inverse for the matrix just given

${\begin{pmatrix}m&n\\p&q\end{pmatrix}}{\begin{pmatrix}1&1\\2&-1\end{pmatrix}}={\begin{pmatrix}1&0\\0&1\end{pmatrix}}$

by using Gauss' method to solve the resulting linear system.

${\begin{array}{*{4}{rc}r}m&+&2n&&&&&=&1\\m&-&n&&&&&=&0\\&&&&p&+&2q&=&0\\&&&&p&-&q&=&1\end{array}}$

Answer: $m=1/3$, $n=1/3$, $p=2/3$, and $q=-1/3$. This matrix is actually the two-sided inverse of $H$, as can easily be checked. With it we can solve the system ($*$) above by applying the inverse.

${\begin{pmatrix}x\\y\end{pmatrix}}={\begin{pmatrix}1/3&1/3\\2/3&-1/3\end{pmatrix}}{\begin{pmatrix}3\\2\end{pmatrix}}={\begin{pmatrix}5/3\\4/3\end{pmatrix}}$

Remark

Why solve systems this way, when Gauss' method takes less arithmetic (this assertion can be made precise by counting the number of arithmetic operations, as computer algorithm designers do)? Beyond its conceptual appeal of fitting into our program of discovering how to represent the various map operations, solving linear systems by using the matrix inverse has at least two advantages.

First, once the work of finding an inverse has been done, solving a system with the same coefficients but different constants is easy and fast: if we change the entries on the right of the system ($*$) then we get a related problem

${\begin{pmatrix}1&1\\2&-1\end{pmatrix}}{\begin{pmatrix}x\\y\end{pmatrix}}={\begin{pmatrix}5\\1\end{pmatrix}}$

with a related solution method.

${\begin{pmatrix}x\\y\end{pmatrix}}={\begin{pmatrix}1/3&1/3\\2/3&-1/3\end{pmatrix}}{\begin{pmatrix}5\\1\end{pmatrix}}={\begin{pmatrix}2\\3\end{pmatrix}}$

In applications, solving many systems having the same matrix of coefficients is common.

Another advantage of inverses is that we can explore a system's sensitivity to changes in the constants. For example, tweaking the $3$ on the right of the system ($*$) to

${\begin{pmatrix}1&1\\2&-1\end{pmatrix}}{\begin{pmatrix}x_{1}\\x_{2}\end{pmatrix}}={\begin{pmatrix}3.01\\2\end{pmatrix}}$

can be solved with the inverse.

${\begin{pmatrix}1/3&1/3\\2/3&-1/3\end{pmatrix}}{\begin{pmatrix}3.01\\2\end{pmatrix}}={\begin{pmatrix}(1/3)(3.01)+(1/3)(2)\\(2/3)(3.01)-(1/3)(2)\end{pmatrix}}$

to show that $x_{1}$ changes by $1/3$ of the tweak while $x_{2}$ moves by $2/3$ of that tweak. This sort of analysis is used, for example, to decide how accurately data must be specified in a linear model to ensure that the solution has a desired accuracy. }}

We finish by describing the computational procedure usually used to find the inverse matrix.

• A matrix is invertible if and only if it can be written as the product of elementary reduction matrices. The inverse can be computed by applying to the identity matrix the same row steps, in the same order, as are used to Gauss-Jordan reduce the invertible matrix.
  • Proof: A matrix $H$ is invertible if and only if it is nonsingular and thus Gauss-Jordan reduces to the identity. This reduction can be done with elementary matrices

$R_{r}\cdot R_{r-1}\cdot \dots R_{1}\cdot H=I$.

This equation gives the two halves of the result.

First, elementary matrices are invertible and their inverses are also elementary. Applying $R_{r}^{-1}$ to the left of both sides of that equation, then $R_{r-1}^{-1}$, etc., gives $H$ as the product of elementary matrices $H=R_{1}^{-1}\cdots R_{r}^{-1}\cdot I$ (the $I$ is here to cover the trivial $r=0$ case).

Second, matrix inverses are unique and so comparison of the above equation with $H^{-1}H=I$ shows that $H^{-1}=R_{r}\cdot R_{r-1}\dots R_{1}\cdot I$. Therefore, applying $R_{1}$ to the identity, followed by $R_{2}$, etc., yields the inverse of $H$.

Example 3

To find the inverse of

${\begin{pmatrix}1&1\\2&-1\end{pmatrix}}$

we do Gauss-Jordan reduction, meanwhile performing the same operations on the identity. For clerical convenience we write the matrix and the identity side-by-side, and do the reduction steps together.

${\begin{array}{rcl}\left({\begin{array}{cc|cc}1&1&1&0\\2&-1&0&1\end{array}}\right)&{\xrightarrow[{}]{-2\rho _{1}+\rho _{2}}}&\left({\begin{array}{cc|cc}1&1&1&0\\0&-3&-2&1\end{array}}\right)\\&{\xrightarrow[{}]{-1/3\rho _{2}}}&\left({\begin{array}{cc|cc}1&1&1&0\\0&1&2/3&-1/3\end{array}}\right)\\&{\xrightarrow[{}]{-\rho _{2}+\rho _{1}}}&\left({\begin{array}{cc|cc}1&0&1/3&1/3\\0&1&2/3&-1/3\end{array}}\right)\end{array}}$

This calculation has found the inverse.

${\begin{pmatrix}1&1\\2&-1\end{pmatrix}}^{-1}={\begin{pmatrix}1/3&1/3\\2/3&-1/3\end{pmatrix}}$

Example 4

This one happens to start with a row swap.

${\begin{array}{rcl}\left({\begin{array}{ccc|ccc}0&3&-1&1&0&0\\1&0&1&0&1&0\\1&-1&0&0&0&1\end{array}}\right)&{\xrightarrow[{}]{\rho _{1}\leftrightarrow \rho _{2}}}&\left({\begin{array}{ccc|ccc}1&0&1&0&1&0\\0&3&-1&1&0&0\\1&-1&0&0&0&1\end{array}}\right)\\&{\xrightarrow[{}]{-\rho _{1}+\rho _{3}}}&\left({\begin{array}{ccc|ccc}1&0&1&0&1&0\\0&3&-1&1&0&0\\0&-1&-1&0&-1&1\end{array}}\right)\\&\vdots \\&{\xrightarrow[{}]{}}&\left({\begin{array}{ccc|ccc}1&0&0&1/4&1/4&3/4\\0&1&0&1/4&1/4&-1/4\\0&0&1&-1/4&3/4&-3/4\end{array}}\right)\end{array}}$

Example 5

A non-invertible matrix is detected by the fact that the left half won't reduce to the identity.

$\left({\begin{array}{cc|cc}1&1&1&0\\2&2&0&1\end{array}}\right){\xrightarrow[{}]{-2\rho _{1}+\rho _{2}}}\left({\begin{array}{cc|cc}1&1&1&0\\0&0&-2&1\end{array}}\right)$

This procedure will find the inverse of a general $n\!\times \!n$ matrix. The $2\!\times \!2$ case is handy.

Resources

• Matrix Addition and Subtraction, Khan Academy
• Multiplying Matrices by Scalars, Khan Academy
• Matrix Multiplication, Khan Academy
• Matrices and Matrix Operations, OpenStax
• Multiplying a 2-by-3 Matrix and 3-by-2 Matrix, patrickJMT

Licensing

Content obtained and/or adapted from:

• Inverses, Wikibooks: Linear Algebra under a CC BY-SA license
• The Inverse of a Matrix, Wikibooks: Linear Algebra under a CC BY-SA license