This LaTeX document is available as postscript or asAdobe PDF.

Matrix Algebra
L. R. Schaeffer, March 1999

Definitions

1.

Scalars. A scalar is a single quantity usually designated by an italicized lower case letter, for example a, r, and k.

2.

Matrix. A matrix is a group of elements arranged in rows and columns. Elements can be scalars, mathematical expressions, or other matrices. Upper case letters in bold face type denote matrices. Examples of matrices are

$$\bf {A} \mbox{ = } \left( \begin{array}{rrrr} 7 & 18 & -2 & 22 \\ -16 & 3 & 55 & 1 \\ 9 & -4 & 0 & 31 \end{array} \right),$$

$$\bf {B} \mbox{ = } \left( \begin{array}{ccc} x & y+1 & x+y+z... ... c \log d & e \\ \sqrt{x-y} & (m+n)/n & p \end{array} \right),$$

and

$$\bf {C} \mbox{ = } \left( \begin{array}{ll} \bf {C}_{11} & \bf {C}_{12} \\ \bf {C}_{21} & \bf {C}_{22} \end{array} \right).$$

The order or dimension of a matrix is the number of rows and columns.

3.

Vectors. A column vector is a matrix with one column, and a row vector is a matrix with one row. Usually if the word vector is used, it is implied that the vector is a column vector. Vectors are commonly denoted by lower case letters in bold face type, but this is not a universal practice.

4.

Transposition. Transposition is the formation of a new matrix by making rows of the original matrix into columns of the new matrix in corresponding order. The transpose of a column vector is a row vector, and the transpose of a matrix of order results in a new matrix of order . If is the original matrix, then the transpose is denoted as in these notes, but in some places the transpose is denoted as .

5.

Diagonals. The diagonal elements of a matrix are those elements whose row and column numbers are the same. For example, if

$$\bf {D} \mbox{ = } \left( \begin{array}{rrr} 8 & 3 & -2 \\ 5 & 7 & -1 \end{array} \right)$$

then the diagonal elements are 8 and 7.

Off-diagonal elements of a matrix are all of the other elements excluding the diagonals.

Types of Matrices

1.

Rectangular. A matrix is rectangular if the number of rows does not equal the number of columns. All vectors are rectangular matrices.

2.

Square. Square matrices have the same number of rows as columns.

3.

Symmetric. Symmetric matrices are also square matrices. The element in the i^th row and j^th column is equal to the element in the j^th row and i^th column, for all values of i and j. The transpose of a symmetric matrix is that same matrix, i.e. . Take any matrix, say and form the product, , then that product is symmetric and square.

4.

Diagonal. A diagonal matrix is a square, symmetric matrix in which all of the elements are zero except the diagonal elements. A diagonal matrix can be written as

$$\bf {D} \mbox{ = } {\bf diag}( d_{1} \mbox{ } d_{2} \cdots d_{n})$$

where d_i are the diagonal elements.

5.

Identity. An identity matrix is a special diagonal matrix in which all of the diagonal elements are equal to one. An identity matrix is usually represented by . Because is it a diagonal matrix, an identity matrix is also square and symmetric.

6.

Null. A null matrix is a matrix of any dimension in which all elements are equal to zero, and is denoted by .

7.

Matrices With Only 1's. A column vector of length n with every element equal to 1, is usually denoted as . A matrix with r rows and c columns and all elements equal to 1 is denoted as .

8.

Triangular. Triangular matrices are square matrices with either all off-diagonal elements above or below the diagonals equal to zero. An upper triangular matrix has all elements below the diagonal equal to zero, and a lower triangular matrix has all elements above the diagonals equal to zero.

9.

Tridiagonal. A tridiagonal matrix is a square matrix with all elements equal to zero except the diagonals and the elements immediately to the left and right of the diagonal. An example is shown below.

$$\bf {B} \mbox{ = } \left( \begin{array}{rrrrrr} 10 & 3 & 0 & ... ... 0 & 3 & 10 & 3 \\ 0 & 0 & 0 & 0 & 3 & 10 \end{array} \right)$$

Addition of Matrices

Matrices are conformable for addition if they have the same order. The resulting sum is a matrix having the same number of rows and columns as the two matrices to be added. Matrices that are not of the same order cannot be added together. If and , then

$$\bf {A}+\bf {B} \mbox{ = } \{ a_{ij} + b_{ij} \}.$$

For example, let

$${\bf A} \mbox{ = } \left( \begin{array}{rrr} 4 & 5 & 3 \\ 6 ... ...\begin{array}{rrr} 1 & 0 & 2 \\ 3 & 4 & 1 \end{array} \right)$$

then

$${\bf A}+{\bf B} \mbox{ = } \left( \begin{array}{rrr} 4+1 & 5+0 & 3+2 \\ 6+3 & 0+4 & 2+1 \end{array} \right)$$

$$\mbox{ = } \left( \begin{array}{rrr} 5 & 5 & 5 \\ 9 & 4 & 3 \end{array} \right) \mbox{ = } {\bf B} + {\bf A} .$$

Subtraction is the addition of two matrices, one of which has all elements multiplied by a minus one (-1). That is,

$${\bf A}+(-1){\bf B} \mbox{ = } \left( \begin{array}{rrr} 3 & 5 & 1 \\ 3 & -4 & 1 \end{array} \right).$$

Multiplication of Matrices

Two matrices are conformable for multiplication if the number of columns in the first matrix equals the number of rows in the second matrix. If $\bf {C}$ has order $p \times q$ and $\bf {D}$ has order $m \times n$, then the product $\bf {CD}$ exists only if q=m. The product matrix has order $p \times n$. In general, does not equal , and most often the product may not even exist because $\bf {D}$ may not be conformable for multiplication with $\bf {C}$. Thus, the ordering of matrices in a product must be carefully and precisely written.

The computation of a product is defined as follows: let

$${\bf C}_{p \times q} = \{ c_{ij} \}$$

and

$${\bf D}_{m \times n} = \{ d_{ij} \}$$

and q=m, then

$${\bf CD}_{p \times n} = \{ \sum_{k=1}^{m} c_{ik}d_{kj} \}.$$

As an example, let

$${\bf C} \mbox{ = } \left( \begin{array}{rrr} 6 & 4 & -3 \\ 3 ... ...begin{array}{rr} 1 & 1 \\ 2 & 0 \\ 3 & -1 \end{array} \right),$$

then

$${\bf CD} \mbox{ = } \left( \begin{array}{rr} 6(1)+4(2)-3(3) &... ...in{array}{rr} 5 & 9 \\ 0 & 10 \\ 12 & 10 \end{array} \right).$$

Transpose of a Product

The transpose of the product of two or more matrices is the product of the transposes of each matrix in reverse order. That is, the transpose of $\bf {CDE}$, for example, is $\bf {E'D'C'}$. The reader should verify that the result is conformable for multiplication, and that the result has the correct dimensions.

Special Products

A matrix, say $\bf {A}$, is idempotent if the product of the matrix with itself equals itself, i.e. $\bf {AA} = \bf {A}$. This implies that the matrix must be square, but not necessarily symmetric.

A matrix is nilpotent if the product of the matrix with itself equals a null matrix, i.e. $\bf {BB} = \bf {0}$, then $\bf {B}$ is nilpotent.

A matrix is orthogonal if the product of the matrix with its transpose equals an identity matrix, i.e. $\bf {UU'} = \bf {I}$, which also implies that $\bf {U'U} = \bf {I}$.

Some Rules on Multiplication

1.

Multiplication of a matrix by a scalar results in multiplying every element of the matrix by that scalar.

2.

.

3.

.

4.

.

Traces of Square Matrices

The trace is an operation that is applied to a square matrix and is denoted by the letters tr. The trace operation is the sum of the diagonal elements of a matrix. The sum is a scalar quantity. Let

$${\bf A} \mbox{ = } \left( \begin{array}{rrr} .51 & -.32 & -.19 \\ -.28 & .46 & -.14 \\ -.21 & -.16 & .33 \end{array} \right),$$

then the trace is

$$tr({\bf A}) \mbox{ = } .51 + .46 + .33 \mbox{ = } 1.30.$$

The trace of the product of conformable matrices has a special property known as the rotation rule of traces. That is,

$$tr({\bf ABC}) \mbox{ = } tr({\bf BCA}) \mbox{ = } tr({\bf CAB}).$$

The traces are equal, because they are scalars, even though the dimensions of the three products might be greatly different.

Euclidean Norm

The Euclidean Norm is a matrix operation usually used to determine the degree of difference between two matrices of the same order. The norm is a scalar and is denoted as $\Vert {\bf A} \Vert$ .

$$\Vert {\bf A} \Vert \mbox{ = } [ tr({\bf AA'}) ]^{.5} \mbox{ = } ( \sum_{i=1}^{n} \sum_{j=1}^{n} a_{ij}^{2} )^{.5}.$$

For example, let

$${\bf A} \mbox{ = } \left( \begin{array}{rrr} 7 & -5 & 2 \\ 4 & 6 & 3 \\ 1 & -1 & 8 \end{array} \right),$$

then

$$\Vert {\bf A} \Vert \mbox{ = } (49 + 25 + 4 + \cdots + 1 + 64 )^{.5}$$

$$\mbox{ = } (205)^{.5} = 14.317821.$$

Other types of norms also exist.

Other Useful Matrix Operations

1.

Direct Sum. For matrices of any dimension, say ${\bf H}_{1}$, ${\bf H}_{2}$, ... ${\bf H}_{n}$, the direct sum is

$$\mbox{ $ \sum_{i}^{+} $ } {\bf H}_{i} \mbox{ = } {\bf H}_{1} ... ... {\bf0} & {\bf0} & \cdots & {\bf H}_{n} \end{array} \right).$$

2.

Kronecker Product. The Kronecker product, also known as the direct product, is where every element of the first matrix is multiplied, as a scalar, times the second matrix. Suppose that $\bf {B}$ is a matrix of order $m \times n$ and that $\bf {A}$ is of order $2 \times 2$, then the direct product of $\bf {A}$ times $\bf {B}$ is

$${\bf A} \star {\bf B} \mbox{ = } {\bf A} \otimes {\bf B} \mbo... ...{\bf B} \\ a_{21}{\bf B} & a_{22}{\bf B} \end{array} \right).$$

Notice that the dimension of the example product is $2m \times 2n$.

3.

Hadamard Product. The Hadamard product exists for two matrices that are conformable for addition. The corresponding elements of the two matrices are multiplied together. The order of the resulting product is the same as the matrices in the product. For two matrices, $\bf {A}$ and $\bf {B}$ of order $2 \times 2$, then the Hadamard product is

$${\bf A} \odot {\bf B} \mbox{ = } \left( \begin{array}{cc} a_{... ...12}b_{12} \\ a_{21}b_{21} & a_{22}b_{22} \end{array} \right).$$

Elementary Operators

Elementary operator matrices are identity matrices that have been modified by one of three methods. The product of an elementary operator matrix with any other matrix does not alter the rank of that matrix. Elementary operator matrices reduce a matrix to either a triangular or a diagonal form. The reduction process may involve many elementary operator matrices multiplied together. To illustrate the three types of elementary operator matrices and the process of reducing a matrix, begin with an example matrix, $\bf {A}$, where

$${\bf A} \mbox{ = } \left( \begin{array}{rrr} 4 & 16 & 24 \\ 3 & -1 & -2 \\ 1 & 5 & 8 \end{array} \right).$$

The first type of elementary operator matrix has one of the diagonal elements of an identity matrix replaced with a constant other than 1. Let ${\bf E}_{11}(.25)$ be an elementary operator where the first diagonal element has been changed to .25. Pre-multiplication of $\bf {A}$ by this matrix gives ${\bf A}_{1}$, where

$${\bf A}_{1} \mbox{ = } \left( \begin{array}{ccc} .25 & 0 & 0 ... ...r} 4 & 16 & 24 \\ 3 & -1 & -2 \\ 1 & 5 & 8 \end{array} \right)$$

$$\mbox{ = } \left( \begin{array}{rrr} 1 & 4 & 6 \\ 3 & -1 & -2 \\ 1 & 5 & 8 \end{array} \right).$$

The second type of elementary operator matrix interchanges rows or columns of a matrix. Let ${\bf E}_{ij}$ be an elementary operator where rows i and j are interchanged. Pre-multiplication of ${\bf A}_{1}$ by this matrix gives ${\bf A}_{2}$ where

$${\bf A}_{2} \mbox{ = } \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{array} \right) {\bf A}_{1}$$

$$\mbox{ = } \left( \begin{array}{rrr} 1 & 4 & 6 \\ 1 & 5 & 8 \\ 3 & -1 & -2 \end{array} \right).$$

The third type of elementary operator matrix combines the elements of two rows or columns of a matrix by addition or subtraction. Let ${\bf E}_{ij}(c)$ be an elementary operator where c times row i is added to row j. Pre-multiplication of ${\bf A}_{2}$ by this matrix gives ${\bf A}_{3}$ where

$${\bf A}_{3} \mbox{ = } \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 0 \\ -3 & 0 & 1 \end{array} \right) {\bf A}_{2}$$

$$\mbox{ = } \left( \begin{array}{rrr} 1 & 4 & 6 \\ 1 & 5 & 8 \\ 0 & -13 & -20 \end{array} \right).$$

By continued pre-multiplication of ${\bf A}_{3}$ by other elementary operators, $\bf {A}$ may be reduced to an upper triangular matrix. Further post-multiplication by another set of elementary operators can reduce $\bf {A}$ to a diagonal matrix.

Matrix Inversion

By definition, an inverse of a square matrix $\bf {A}$, denoted by ${\bf A}^{-1}$, is a matrix which when pre- or post- multiplied times the original matrix yields an identity matrix. That is,

$${\bf AA^{-1} } \mbox{ = } {\bf I}, \mbox{ and } {\bf A^{-1}A } \mbox{ = } {\bf I} .$$

Any square matrix with a nonzero determinant can be inverted. Thus, the first step to matrix inversion is the calculation of the determinant of the matrix. The determinant is a scalar quantity. For a $2 \times 2$ matrix, say

$${\bf A} \mbox{ = } \left( \begin{array}{cc} a_{11} & a_{12} \\ a_{21} & a_{22} \end{array} \right),$$

then the determinant is

$$\vert {\bf A} \vert \mbox{ = } a_{11}a_{22}-a_{21}a_{12} .$$

For a $3 \times 3$ matrix, the determinant can be reduced to a series of determinants of $2 \times 2$ matrices. For example, let

$${\bf B} \mbox{ = } \left( \begin{array}{rrr} 6 & -1 & 2 \\ 3 & 4 & -5 \\ 1 & 0 & -2 \end{array} \right),$$

then

$$\vert {\bf B} \vert \mbox{ = } 6 \left\vert \begin{array}{rr}... ...\vert \begin{array}{rr} 3 & 4 \\ 1 & 0 \end{array} \right\vert$$

$$\mbox{ = } 6(-8) \mbox{ } +1(-1) \mbox{ } +2(-4)$$

$$\mbox{ = } -57.$$

The general expression for the determinant of a matrix is

$$\vert {\bf B} \vert \mbox{ = } \sum_{j=1}^{n} (-1)^{i+j}b_{ij} \vert {\bf M}_{ij} \vert$$

where $\bf {B}$ is of order n, and ${\bf M}_{ij}$ is a minor submatrix of $\bf {B}$ resulting from the deletion of the i^th row and j^th column of $\bf {B}$. Any row of $\bf {B}$ may be used to compute the determinant because the result should be the same for each row. Columns may also be used instead of rows. The reader may verify this assertion.

If the determinant is non-zero, then proceed to the next step which is to find the matrix of signed minors, known as the adjoint matrix. Using the same matrix, $\bf {B}$ above, the minors and their determinants are as follows:

$${\bf M}_{11} \mbox{ = } +1 \left( \begin{array}{rr} 4 & -5 \... ...\right), \mbox{ and } \vert {\bf M}_{11} \vert \mbox{ = } -8,$$

$${\bf M}_{12} \mbox{ = } -1 \left( \begin{array}{rr} 3 & -5 \... ... \right), \mbox{ and } \vert {\bf M}_{12} \vert \mbox{ = } +1,$$

$${\bf M}_{13} \mbox{ = } +1 \left( \begin{array}{rr} 3 & 4 \\ ... ... \right), \mbox{ and } \vert {\bf M}_{13} \vert \mbox{ = } -4,$$

$${\bf M}_{21} \mbox{ = } -1 \left( \begin{array}{rr} -1 & 2 \\... ... \right), \mbox{ and } \vert {\bf M}_{21} \vert \mbox{ = } -2,$$

$${\bf M}_{22} \mbox{ = } +1 \left( \begin{array}{rr} 6 & 2 \\ ... ...\right), \mbox{ and } \vert {\bf M}_{22} \vert \mbox{ = } -14,$$

$${\bf M}_{23} \mbox{ = } -1 \left( \begin{array}{rr} 6 & -1 \\... ... \right), \mbox{ and } \vert {\bf M}_{23} \vert \mbox{ = } -1,$$

$${\bf M}_{31} \mbox{ = } +1 \left( \begin{array}{rr} -1 & 2 \\... ... \right), \mbox{ and } \vert {\bf M}_{31} \vert \mbox{ = } -3,$$

$${\bf M}_{32} \mbox{ = } -1 \left( \begin{array}{rr} 6 & 2 \\ ... ...ox{ and } \vert {\bf M}_{32} \vert \mbox{ = } 36, \mbox{ and }$$

$${\bf M}_{33} \mbox{ = } +1 \left( \begin{array}{rr} 6 & -1 \\... ... \right), \mbox{ and } \vert {\bf M}_{33} \vert \mbox{ = } 27.$$

The adjoint matrix of signed minors, ${\bf M}_{B}$ is then

$${\bf M}_{B} \mbox{ = } \left( \begin{array}{rrr} -8 & 1 & -4 \\ -2 & -14 & -1 \\ -3 & 36 & 27 \end{array} \right).$$

The inverse of $\bf {B}$ is then

$${\bf B}^{-1} \mbox{ = } \vert {\bf B} \vert^{-1} {\bf M'}_{B}$$

$$\mbox{ = } \frac{1}{-57} \left( \begin{array}{rrr} -8 & -2 & -3 \\ 1 & -14 & 36 \\ -4 & -1 & 27 \end{array} \right).$$

If the determinant is zero, then the inverse is not defined or does not exist. A square matrix with a non-zero determinant is said to be nonsingular. Only nonsingular matrices have inverses. Matrices with zero determinants are called singular and do not have an inverse.

An important result is that the inverse of an inverse matrix is equal to the original matrix. That is,

$$({\bf A^{-1}})^{-1} \mbox{ = } {\bf A}.$$

The inverse of the product of two or more nonsingular matrices follows a rule similar to that for the transpose of a product of matrices. Let be three nonsingular matrices, then

$$( {\bf ABC})^{-1} \mbox{ = } {\bf C^{-1}B^{-1}A^{-1} },$$

and

$${\bf C^{-1}B^{-1}A^{-1}ABC} \mbox{ = } {\bf I}.$$

Generalized Inverses

Suppose we have a set of equations, , then a solution for ${\bf x}$ is obtained by pre-multiplying both sides of the equation by ${\bf A}^{-1}$ provided that $\bf {A}$ is a nonsingular matrix. Then . Often, however, $\bf {A}$ is a singular matrix, i.e. the determinant is zero and the inverse does not exist. Solutions to the set of equations can still be calculated using a generalized inverse of $\bf {A}$. A generalized inverse, denoted as ${\bf A^{-}}$, is a matrix that satisfies the following expression

$${\bf AA^{-}A} \mbox{ = } {\bf A}.$$

In general, neither are equal to ${\bf I}$ except in the case when $\bf {A}$ is nonsingular.

There are an infinite number of generalized inverses for any one singular matrix. A unique generalized inverse is the Moore-Penrose inverse which satisfies the following conditions:

1.

2.

3.

and

4.

Usually, a generalized inverse that satisfies only the first condition is sufficient for practical purposes.

• Linear Independence. A square matrix with a nonzero determinant is a matrix in which the rows and columns of the matrix are linearly independent. If $\bf {A}$ is the matrix, then linear independence means that no vector, say ${\bf k}$, exists such that except for ${\bf k} = {\bf0}$.

  A square matrix with a zero determinant implies that at least one non-null vector, ${\bf k}$, exists such that To compute a generalized inverse of a matrix that has a zero determinant requires that we know the number of linearly independent rows and columns in that matrix. The number of linearly independent rows or columns is called the rank of a matrix and is a term that applies to rectangular as well as square matrices.

• Rank. Elementary operators are used to determine the rank of a matrix. Reduction of the matrix to a diagonal matrix is called reduction to canonical form, and the reduced matrix is called the canonical form under equivalence. If the matrix is square and symmetric, then the reduction of the matrix to a diagonal form with either ones or minus ones on the diagonals is called congruent reduction and the resulting matrix is the canonical form under congruence. Usually, only a reduction to an upper triangular form is necessary to determine the rank of the matrix.

  Let
  

  $${\bf A} \mbox{ = } \left( \begin{array}{rrrrr} 2 & 1 & -1 & 3... ...4 & 7 & 5 & -2 & -1 \\ 6 & 8 & 4 & 1 & 7 \end{array} \right).$$
  
  
  The rank of a matrix can not be greater than the minimum of the number of rows or columns, whichever is smaller. In the example above, $\bf {A}$ has 3 rows and 5 columns, and therefore the rank of $\bf {A}$ can not be greater than 3. To find the exact rank, elementary operator matrices are used. Let ${\bf P_{1}}$ be the first elementary operator matrix that will zero out the first elements in the second and third rows. Premultiplication of any matrix by elementary operator matrices does not change the rank of that matrix. Let
  
  $${\bf P_{1}} \mbox{ = } \left( \begin{array}{rrr} 1 & 0 & 0 \\ -2 & 1 & 0 \\ -3 & 0 & 1 \end{array} \right),$$
  
  
  then
  
  $${\bf P_{1}A} \mbox{ = } \left( \begin{array}{rrrrr} 2 & 1 & -... ...5 & 7 & -8 & -17 \\ 0 & 5 & 7 & -8 & -17 \end{array} \right).$$
  
  
  Now use ${\bf P_{2}}$ to subtract the third row from the second row, so that
  
  $${\bf P_{2}} \mbox{ = } \left( \begin{array}{rrr} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & -1 & 1 \end{array} \right),$$
  
  
  and
  
  $${\bf P_{2}P_{1}A} \mbox{ = } \left( \begin{array}{rrrrr} 2 & ... ...0 & 5 & 7 & -8 & -17 \\ 0 & 0 & 0 & 0 & 0 \end{array} \right).$$
  
  
  The rank is the number of non-zero diagonal elements of the reduced matrix, i.e. .

  Elementary operators of order $5 \times 5$ would be postmultiplied times $\bf {A}$ in order to reduce the matrix to a diagonal form, and the rank would still be the number of non-zero diagonal elements.

• Definitions.

  Full-row rank

  If $\bf {A}$ has order $m \times n$ with rank equal to m, then $\bf {A}$ has full row rank.

  Full-column rank

  A matrix with rank equal to the number of columns has full-column rank.

  Full rank

  A square matrix with rank equal to the number of rows or columns has full rank. A full rank matrix is nonsingular, has a non-zero determinant, and has an inverse.

  1.

  A null matrix has a rank of zero.

  2.

  A J-matrix has a rank of one.

  3.

  The rank of an idempotent matrix is equal to its trace.

  4.

  If $\bf {A}$ has rank r and $\bf {B}$ has rank q, and the two matrices are conformable for multiplication, then the product, ${\bf AB}$, has a maximum possible rank equal to the lesser of r or q.

• Consequences. The fact that a matrix has rank that is less than the number of rows or columns in the matrix means that we can partition the matrix into a square matrix of order equal to the rank and that has rank equal to that matrix, and into other matrices of the remaining rows and columns. The other rows and columns are linearly dependent upon the rows and columns in that square matrix. That means that they can be formed from the rows and columns that are linearly independent.

  Let $\bf {A}$ be a matrix of order $p \times q$ with rank r, and r is less than either p or q, then there are p-r rows of $\bf {A}$ and q-r columns which are not linearly independent. Partition $\bf {A}$ as follows:
  

  $${\bf A_{p \times q}} \mbox{ = } \left( \begin{array}{cc} {\bf... ...bf A_{12}} \\ {\bf A_{21}} & {\bf A_{22}} \end{array} \right)$$
  
  
  such that ${\bf A_{11}}$ has order $r \times r$ and rank of r. Re-arrangement of rows and columns of $\bf {A}$ may be needed to find an appropriate ${\bf A_{11}}$. ${\bf A_{12}}$ has order $r \times (q-r)$, ${\bf A_{21}}$ has order $(p-r) \times r$, and ${\bf A_{22}}$ has order $(p-r) \times (q-r)$.

  There exist matrices, ${\bf K_{1}}$ and ${\bf K_{2}}$ such that
  

  $$\left( \begin{array}{cc} {\bf A_{21}} & {\bf A_{22}} \end{arr... ...in{array}{cc} {\bf A_{11}} & {\bf A_{12}} \end{array} \right),$$
  
  
  and
  
  $$\left( \begin{array}{c} {\bf A_{12}} \\ {\bf A_{22}} \end{arr... ...} {\bf A_{11}} \\ {\bf A_{21}} \end{array} \right){\bf K_{1}},$$
  
  
  and
  
  $${\bf A} \mbox{ = } \left( \begin{array}{cc} {\bf A_{11}} & {\... ...\bf K_{2}A_{11}} & {\bf K_{2}A_{11}K_{1}} \end{array} \right).$$
  
  
  To illustrate, let
  
  $${\bf A} \mbox{ = } \left( \begin{array}{rrr} 3 & -1 & 4 \\ 1 & 2 & -1 \\ -5 & 4 & -9 \\ 4 & 1 & 3 \end{array}\right),$$
  
  
  and the rank of this matrix is 2. A $2 \times 2$ full rank submatrix within $\bf {A}$ is the upper left $2 \times 2$ matrix. Let
  
  $${\bf A_{11}} \mbox{ = } \left( \begin{array}{rr} 3 & -1 \\ 1 & 2 \end{array} \right),$$
  
  
  then
  
  $${\bf A_{12}} \mbox{ = } \left( \begin{array}{r} 4 \\ -1 \end{array} \right) \mbox{ = } {\bf A_{11}K_{1}},$$
  
  
  where
  
  $${\bf K_{1}} \mbox{ = } \left( \begin{array}{r} 1 \\ -1 \end{array} \right),$$
  
  
  
  
  $${\bf A_{21}} \mbox{ = } \left( \begin{array}{rr} -5 & 4 \\ 4 & 1 \end{array} \right) \mbox{ = } {\bf K_{2}A_{11}},$$
  
  
  where
  
  $${\bf K_{2}} \mbox{ = } \left( \begin{array}{rr} -2 & 1 \\ 1 & 1 \end{array} \right),$$
  
  
  and
  
  $${\bf A_{22}} \mbox{ = } {\bf K_{2}A_{11}K_{1}}$$
  
  
  
  
  $$\mbox{ = } \left( \begin{array}{rr} -2 & 1 \\ 1 & 1 \end{arra... ...mbox{ = } \left( \begin{array}{r} -9 \\ 3 \end{array} \right).$$
  
  

  This kind of partitioning of matrices with less than full rank is always possible. In practice, we need only know that we can perform this kind of partitioning, but it is usually not necessary to derive ${\bf K_{1}}$ and ${\bf K_{2}}$ explicitly.

• Calculations. For a matrix, $\bf {A}$, of less than full rank, there is an infinite number of possible generalized inverses, all of which would satisfy . However, only one generalized inverse needs to be computed in practice. A method to derive one particular type of generalized inverse has the following steps:

  1.

  Determine the rank of $\bf {A}$.

  2.

  Obtain ${\bf A_{11}}$, a square, full rank submatrix of $\bf {A}$ with rank equal to the rank of $\bf {A}$.

  3.

  Partition $\bf {A}$ as
  

  $${\bf A} \mbox{ = } \left( \begin{array}{cc} {\bf A_{11}} & {\bf A_{12}} \\ {\bf A_{21}} & {\bf A_{22}} \end{array}\right).$$

  
  

  4.

  Compute the generalized inverse as
  

  $${\bf A^{-}} \mbox{ = } \left( \begin{array}{cc} {\bf A_{11}^{-1}} & {\bf0} \\ {\bf0} & {\bf0} \end{array} \right).$$

  
  

  If $\bf {A}$ has order $p \times q$, then ${\bf A^{-}}$ must have order $q \times p$. To prove that ${\bf A^{-}}$ is a generalized inverse of $\bf {A}$, then multiply out the expression
  

  $${\bf AA^{-}A} \mbox{ = } \left( \begin{array}{cc} {\bf A_{11}... ...{\bf A_{12}} \\ {\bf A_{21}} & {\bf A_{22}} \end{array}\right)$$
  
  
  
  
  $$\mbox{ = } \left( \begin{array}{cc} {\bf A_{11}} & {\bf A_{12... ... A_{21}} & {\bf A_{21}A_{11}^{-1} A_{12}} \end{array} \right).$$
  
  
  From the previous section, so that
  
  $${\bf A_{21}A_{11}^{-1}A_{12}} \mbox{ = } {\bf K_{2}A_{11}A_{1... ...A_{12}} \mbox{ = } {\bf K_{2}A_{12}} \mbox{ = } {\bf A_{22}} .$$
  
  

• Solutions to Equations. Because there are an infinite number of generalized inverses to a matrix that has less than full rank, then it logically follows that for a system of consistent equations, , where the solutions are computed as , then there would also be an infinite number of solution vectors for ${\bf x}$. Having computed only one generalized inverse, however, it is possible to compute many different solution vectors. From Searle(1971 page 9, Theorem 2), if $\bf {A}$ has q columns and if ${\bf G}$ is one generalized inverse of $\bf {A}$, then the consistent equations have solution
  
  $$\tilde{{\bf x}} \mbox{ = } {\bf Gr}+ ({\bf GA}-{\bf I}){\bf z},$$
  
  
  where ${\bf z}$ is any arbitrary vector of length q. The number of linearly independent solution vectors, however, is (q-r+1).

  It is also possible to generate other generalized inverses of the same matrix. If ${\bf G}$ is a generalized inverse of $\bf {A}$ then so is
  

  $${\bf F} \mbox{ = } {\bf GAG}+({\bf I}-{\bf GA}){\bf X} + {\bf Y}({\bf I} - {\bf AG})$$
  
  
  for any ${\bf X}$ and ${\bf Y}$. Pre- and post- multiplication of ${\bf F}$ by $\bf {A}$ shows that this is so.

• Special Generalized Inverses. In linear models analyses, the product ${\bf X'R^{-1}X}$ occurs frequently where ${\bf X}$ is a matrix that usually has more rows than it has columns and has rank less than the number of columns, and ${\bf R}$ is a square matrix that is usually diagonal. Generalized inverses of this product matrix have special features that are used in the derivation and proof of various statistical procedures. Let ${\bf X}$ be a matrix of order $N \times p$with rank r. The product, ${\bf X'R^{-1}X}$ has order $p \times p$ and is symmetric with rank r. Let ${\bf G}$ represent any generalized inverse of the product matrix, then the following results are true.

  1.

  ${\bf G}$ is not necessarily a symmetric matrix, in which case ${\bf G'}$ is also a generalized inverse of the product matrix.

  2.

  or

  3.

  is a generalized inverse of ${\bf X}$.

  4.

  is always symmetric and unique for all generalized inverses of the product matrix, ${\bf X'R^{-1}X}$.

  5.

  If ${\bf 1'}$ = ${\bf k'X}$ for some ${\bf k'}$, then .

  The proofs of the above results can be found in Searle (1971). These results are very important.

Eigenvalues and Eigenvectors

There are a number of square, and sometimes symmetric, matrices involved in statistical procedures that must be positive definite. Suppose that ${\bf Q}$ is any square matrix then

• ${\bf Q}$ is positive definite if ${\bf y'Qy}$ is always greater than zero for all vectors, ${\bf y}$.
• ${\bf Q}$ is positive semi-definite if ${\bf y'Qy}$ is greater than or equal to zero for all vectors ${\bf y}$, and for at least one vector ${\bf y}$, then ${\bf y'Qy} = 0$.
• ${\bf Q}$ is non-negative definite if ${\bf Q}$ is either positive definite or positive semi-definite.

To determine the appropriate condition of the matrix ${\bf Q}$, then one must calculate the eigenvalues (or latent roots or characteristic roots) of the matrix. The eigenvalues are useful in that

• The product of the eigenvalues equals the determinant of the matrix.
• The number of non-zero eigenvalues equals the rank of the matrix.
• If all the eigenvalues are greater than zero, then the matrix is positive definite.
• If all the eigenvalues are greater than or equal to zero and one or more are equal to zero, then the matrix is positive semi-definite.

If ${\bf Q}$ is a square, symmetric matrix, then it can be represented as

$${\bf Q} \mbox{ = } {\bf UDU'}$$

where $\bf {D}$ is a diagonal matrix, the canonical form of ${\bf Q}$, containing the eigenvalues of ${\bf Q}$, and ${\bf U}$ is an orthogonal matrix of the corresponding eigenvectors. Recall that for a matrix to be orthogonal then and .

The eigenvalues and eigenvectors are found by solving

$$\mid {\bf Q}-d{\bf I} \mid \mbox{ = } 0,$$

and

$${\bf Qu}-d{\bf u} \mbox{ = } {\bf0},$$

where d is one of the eigenvalues of ${\bf Q}$ and ${\bf u}$ is the corresponding eigenvector. There are numerous computer routines for calculating $\bf {D}$ and ${\bf U}$.

Differentiation

The differentiation of mathematical expressions involving matrices follows similar rules as for those involving scalars. Some of the basic results are shown below.

Let

$$c = 3x_{1} + 5x_{2} + 9x_{3} = {\bf b'x}$$

$$\mbox{ = } \left( \begin{array}{rrr} 3 & 5 & 9 \end{array} \r... ... \begin{array}{c} x_{1} \\ x_{2} \\ x_{3} \end{array} \right).$$

With scalars we have

$$\frac{\partial c}{\partial x_{1}} \mbox{ = } 3$$

$$\frac{\partial c}{\partial x_{2}} \mbox{ = } 5$$

$$\frac{\partial c}{\partial x_{3}} \mbox{ = } 9,$$

but with vectors we have

$$\frac{\partial c}{\partial {\bf x}} \mbox{ = } {\bf b}.$$

The general rule is

$$\frac{\partial {\bf A'x}}{\partial {\bf x}} \mbox{ = } {\bf A}.$$

Another function might be

c = 9x₁² + 6x₁x₂ + 4x₂²

or

$$c = \left( \begin{array}{cc} x_{1} & x_{2} \end{array} \right... ...}{c} x_{1} \\ x_{2} \end{array} \right) \mbox{ = } {\bf x'Ax}.$$

With scalars we would have

$$\frac{\partial c}{\partial x_{1}} \mbox{ = } 2(9)x_{1} + 6x_{2}$$

$$\frac{\partial c}{\partial x_{2}} \mbox{ = } 6x_{1} + 2(4)x_{2},$$

then in matrix form,

$$\frac{\partial c}{\partial {\bf x}} \mbox{ = } 2{\bf Ax}.$$

If $\bf {A}$ was not a symmetric matrix, then

$$\frac{\partial {\bf x'Ax}}{\partial {\bf x}} \mbox{ = } {\bf Ax} + {\bf A'x}.$$

Cholesky Decomposition

In simulation studies or applications of Gibb's sampling there is frequently a need to factor a symmetric positive definite matrix into the product of a matrix times its transpose. The Cholesky decomposition of a matrix, say ${\bf V}$, is a lower triangular matrix such that

$${\bf V} = {\bf T}{\bf T}',$$

and ${\bf T}$ is lower triangular. Suppose that

$${\bf V} = \left( \begin{array}{rrr} 9 & 3 & -6 \\ 3 & 5 & 0 \\ -6 & 0 & 21 \end{array} \right).$$

The problem is to derive

$${\bf T} = \left( \begin{array}{rrr} t_{11} & 0 & 0 \\ t_{21} & t_{22} & 0 \\ t_{31} & t_{32} & t_{33} \end{array} \right),$$

such that

$$\begin{eqnarray*}t_{11}^{2} & = & 9 \\ t_{11}t_{21} & = & 3 \\ t_{11}t_{31} ... ...\ \ {\rm and} \\ t_{31}^{2} + t_{32}^{2} + t_{33}^{2} & = & 21 \end{eqnarray*}$$

These equations give t₁₁ = 3, then t₂₁ = 3/t₁₁ = 1, and t₃₁ = -6/t₁₁ = -2. From the fourth equation, t₂₂ is the square root of (5 - t₂₁²) or t₂₂ = 2. The fifth equation says that (1)(-2) + (2)t₃₂ = 0 or t₃₂ = 1. The last equation says that t₃₃² is 21 - (-2)² - (1)² = 16. The end result is

$${\bf T} = \left( \begin{array}{rrr} 3 & 0 & 0 \\ 1 & 2 & 0 \\ -2 & 1 & 4 \end{array} \right).$$

The derivation of the Cholesky decomposition is easily programmed for a computer. Note that in calculating the diagonals of ${\bf T}$ the square root of a number is needed, and consequently this number must always be positive. Hence, if the matrix is positive definite, then all necessary square roots will be of positive numbers. However, the opposite is not true. That is, if all of the square roots are of positive numbers, the matrix is not necessarily guaranteed to be positive definite. The only way to guarantee positive definiteness is to calculate the eigenvalues of a matrix and to see that they are all positive.

Inverse of a Lower Triangular Matrix

The inverse of a lower triangular matrix is also a lower triangular matrix, and can be easily derived. The diagonals of the inverse are the inverse of the diagonals of the original matrix. Using the matrix ${\bf T}$ from the previous section, then

$${\bf T}^{-1} = \left( \begin{array}{rrr} t^{11} & 0 & 0 \\ ... ... & t^{22} & 0 \\ t^{31} & t^{32} & t^{33} \end{array} \right),$$

where

$$\begin{eqnarray*}t^{ii} & = & 1 / t_{ii} \\ t_{21}t^{11} + t_{22}t^{21} & = & ... ...+ t_{33}t^{31} & = & 0 \\ t_{32}t^{22} + t_{33}t^{32} & = & 0. \end{eqnarray*}$$

These equations give

$$\begin{eqnarray*}t^{11} & = & \frac{1}{3} \\ t^{21} & = & - \frac{1}{6} \\ t... ... & = & - \frac{1}{8} \ \ {\rm and} \\ t^{33} & = & \frac{1}{4} \end{eqnarray*}$$

Likewise the determinant of a triangular matrix is the product of all of the diagonal elements. Hence all diagonal elements need to be non-zero for the inverse to exist.

The natural logarithm of the determinant of a triangular matrix is the summation of the natural logarithms of the individual diagonal elements. This property is useful in derivative free restricted maximum likelihood.

This LaTeX document is available as postscript or asAdobe PDF.