In linear algebra, a projection matrix is a matrix associated to a linear operator that maps vectors into their projections onto a subspace.
Let us start by reviewing some notions that are essential for understanding projections.
Let be a linear space. Let and be subspaces of .
Remember that the sum is the set
When and have only the zero vector in common (i.e., ), then the sum is called a direct sum and it is denoted by .
Moreover, when the direct sum is equal to the whole space, that is,we say that the two spaces are complementary.
As we have proved, when and are complementary, any vector belonging to can be uniquely written aswhere and .
After revising all these notions, we are ready to define projections.
Definition Let be a linear space. Let and be complementary subspaces (i.e., ). Let with its unique decompositionin which and . Then, the vector is called the projection of onto along , and the vector is called the projection of onto along .
We note that the locutions "along " and "along " are needed because the complement of a given subspace is not necessarily unique. For example, there may be another subspace that is complementary to . As a consequence, when we project a vector onto , we need to specify whether we are considering or as a complement of .
Example Let be the space of all real vectors. Let be the space spanned by which contains all the scalar multiples of . Let be the space spanned by the two vectors which contains all the linear combinations of and . We have that because no non-zero vector of can be written as a linear combination of and . Therefore, . Now, consider the vector We have thatThus, the unique decomposition of iswhere and . The projection of onto along is and the projection of onto along is
Projections as defined above are also sometimes called oblique projections in order to distinguish them from orthogonal projections, which are a particular kind of projection in which the two complementary subspaces and are orthogonal complements.
We now define projection operators.
Definition Let be a linear space and and two subspaces such that . The function that associates to each its projection onto along is called the projection operator onto along .
The first important property of the projection operator is that it is a linear operator, that is, it preserves addition and multiplication by scalars.
Proposition The projection operator onto along is a linear operator.
Arbitrarily choose two vectors . They have the unique decompositionswhere and . Denote by the projection operator onto along . Then,Take any two scalars and and consider the linear combinationThen,Therefore,Since , , and were arbitrary, the latter equality implies that the projection operator is linear.
A couple of observations are in order:
the range of (i.e., the subset of the codomain formed by all the values actually taken by ) is ;
the kernel of (i.e., the subset of the domain formed by all the vectors that are mapped to the zero vector) is .
Let be a basis for . Any vector can be represented by its coordinate vector with respect to , denoted by . If can be written as a linear combination of the basis asthen
Moreover, any linear operator can be represented by a square matrix, called matrix of the operator with respect to and denoted by , such that
In the case of a projection operator , this implies that there is a square matrix that, once post-multiplied by the coordinates of a vector , gives the coordinates of the projection of onto along . Such a matrix is called a projection matrix (or a projector).
Definition The matrix of a projection operator with respect to a given basis is called a projection matrix.
Now that we know what a projection matrix is, we can learn how to derive it.
In the lecture on complementary subspaces we have shown that, if is a basis for , is a basis for , and then is a basis for .
For the sake of legibility, denote the projection simply by in what follows.
Note that projects:
vectors of into themselves (because their component from is equal to zero);
vectors of into the zero vector (because their component from is zero).
By applying the general rule for deriving the matrix of a linear operator, we obtain thatwhere is the identity matrix and the other blocks are zero matrices (in particular, the diagonal one is ).
In step we have used the fact that the coordinate vector of with respect to the basis (to which itself belongs, occupying the -th position) is a vector that has a single entry equal to (the -th) and all the other entries equal to .
Thus, the projection has an extremely simple structure: when we use it to project a vector on , we leave the coordinates corresponding to the basis of unchanged and we set all the other coordinates to zero.
However, in most cases we are not so lucky as to have coordinates already expressed with respect to . In such cases, we need to perform a change of basis (please revise how it works here).
Suppose that the basis used to express coordinates is . Then, the change-of-basis from to is
The matrix of the projection operator with respect to the basis is
Example As in the previous example, we consider the space of all real vectors. Coordinates are naturally expressed with respect to the canonical basis whereThe basis of is whereand the basis of is whereWe have already argued that . As before, denote (the projection operator onto along ) simply by . First of all, we have that because the projection operator preserves the first coordinate and annihilates the other two (when coordinates are expressed with respect to ). The change-of-basis matrix that fits our purposes isIts inverse isThe projection matrix under the canonical basis isLet us compute the projection onto of the vectorWe have done it already in the previous exercise, but this time we can use the projection matrix:which is the same result we have derived previously.
Once we have derived the projection matrix that allows us to project vectors onto , it is very easy to derive the matrix that allows us to project vectors onto the complementary subspace .
If a vector is decomposed as then we can write the projection onto asand its coordinates as
Thus, the matrix of the projection operator onto , sometimes called complementary projector, is
In the derivation above we have also seen that
Thus, we have that
A square matrix is said to be idempotent if and only if it is equal to its square:
It turns out that idempotent matrices and projection matrices are the same thing!
Proposition A matrix is idempotent if and only if it is a projection matrix.
Let us prove the "if part". We start from the hypothesis that is a projection matrix. As such, it is the matrix of a projection operator with respect to some basis , that is, . As proved above, we have thatTherefore,which proves that is idempotent. Let us now prove the "only if" part, starting from the hypothesis that is idempotent. Suppose that is . Let be the space of all vectors. Define the two subspacesandIn other words, and are the range and kernel of the operator defined by the matrix . We have thatsince any vector can be written aswhere: and becauseSuppose that a vector belongs to both and . Since , there exists such thatWe can pre-multiply both sides by and obtainSince , we have that . As a consequence, by equation (3), and by equation (2). Therefore,Hence,From equation (1) we know that projects into its component . Therefore, it is the matrix of the projection operator that projects vectors of into along . Hence, is a projection matrix.
Below you can find some exercises with explained solutions.
Consider the projection problem analyzed in the previous two examples, where we have already derived the projection matrix of the projection operator onto . Derive the complementary projection matrix (onto ) and use it to find the projection onto of the vector
Define . We have thatThe projection of onto is
Please cite as:
Taboga, Marco (2021). "Projection matrix", Lectures on matrix algebra. https://www.statlect.com/matrix-algebra/projection-matrix.
Most of the learning materials found on this website are now available in a traditional textbook format.