Linear Algebra/Projection Onto a Subspace

Linear Algebra Projection Onto a Subspace Gram-Schmidt Orthogonalization Topic: Line of Best Fit This subsection, like the others in this section, is optional. It also requires material from the optional earlier subsection on Combining Subspaces. The prior subsections project a vector onto a line by decomposing it into two parts: the part in the line and the rest . To generalize projection to arbitrary subspaces, we follow this idea. Definition 3.1def:ProjOntoMAlongN For any direct sum and any , the projection of onto along is where with . This definition doesn't involve a sense of "orthogonal" so we can apply it to spaces other than subspaces of an . (Definitions of orthogonality for other spaces are perfectly possible, but we haven't seen any in this book.) Example 3.2ex:OrthProjOne The space of matrices is the direct sum of these two. To project onto along , we first fix bases for the two subspaces. The concatenation of these is a basis for the entire space, because the space is the direct sum, so we can use it to represent . Now the projection of onto along is found by keeping the part of this sum and dropping the part. Example 3.3ex:ProjOntoMAlongNGener Both subscripts on are significant. The first subscript matters because the result of the projection is an , and changing this subspace would change the possible results. For an example showing that the second subscript matters, fix this plane subspace of and its basis and compare the projections along two different subspaces. (Verification that and is routine.) We will check that these projections are different by checking that they have different effects on this vector. For the first one we find a basis for and represent with respect to the concatenation . The projection of onto along is found by keeping the part and dropping the part. For the other subspace , this basis is natural. Representing with respect to the concatenation and then keeping only the part gives this. Therefore projection along different subspaces may yield different results. These pictures compare the two maps. Both show that the projection is indeed "onto" the plane and "along" the line. <TR> <TD>x200px <TD>x200px Notice that the projection along is not orthogonal— there are members of the plane that are not orthogonal to the dotted line. But the projection along is orthogonal. A natural question is: what is the relationship between the projection operation defined above, and the operation of orthogonal projection onto a line? The second picture above suggests the answer— orthogonal projection onto a line is a special case of the projection defined above; it is just projection along a subspace perpendicular to the line. x200px In addition to pointing out that projection along a subspace is a generalization, this scheme shows how to define orthogonal projection onto any subspace of , of any dimension. Definition 3.4def:OrthComp The orthogonal complementof a subspace of is (read " perp"). The orthogonal projection of a vector is its projection onto along . Example 3.5ex:OrthoCompOne In , to find the orthogonal complement of the plane we start with a basis for . Any perpendicular to every vector in is perpendicular to every vector in the span of (the proof of this assertion is Problem 10). Therefore, the subspace consists of the vectors that satisfy these two conditions. We can express those conditions more compactly as a linear system. We are thus left with finding the nullspace of the map represented by the matrix, that is, with calculating the solution set of a homogeneous linear system. Example 3.6ex:OrthoCompTwo Where is the -plane subspace of , what is ? A common first reaction is that is the -plane, but that's not right. Some vectors from the -plane are not perpendicular to every vector in the -plane. <TR> <TD> <TD>x200px <TD> Instead is the -axis, since proceeding as in the prior example and taking the natural basis for the -plane gives this. The two examples that we've seen since Definition 3.4 illustrate the first sentence in that definition. The next result justifies the second sentence. Lemma 3.7le:OrthoProjWellDefd Let be a subspace of . The orthogonal complement of is also a subspace. The space is the direct sum of the two . And, for any , the vector is perpendicular to every vector in . Proof First, the orthogonal complement is a subspace of because, as noted in the prior two examples, it is a nullspace. Next, we can start with any basis for and expand it to a basis for the entire space. Apply the Gram-Schmidt process to get an orthogonal basis for . This is the concatenation of two bases (with the same number of members as ) and . The first is a basis for , so if we show that the second is a basis for then we will have that the entire space is the direct sum of the two subspaces. Problem 9 from the prior subsection proves this about any orthogonal basis: each vector in the space is the sum of its orthogonal projections onto the lines spanned by the basis vectors. To check this, represent the vector , apply to both sides , and solve to get , as desired. Since obviously any member of the span of is orthogonal to any vector in , to show that this is a basis for we need only show the other containment— that any is in the span of this basis. The prior paragraph does this. On projections onto basis vectors from , any gives and therefore () gives that is a linear combination of . Thus this is a basis for and is the direct sum of the two. The final sentence is proved in much the same way. Write . Then is gotten by keeping only the part and dropping the part . Therefore consists of a linear combination of elements of and so is perpendicular to every vector in . We can find the orthogonal projection onto a subspace by following the steps of the proof, but the next result gives a convienent formula. Theorem 3.8th:OrthProjMat Let be a vector in and let be a subspace of with basis . If is the matrix whose columns are the 's then where the coefficients are the entries of the vector . That is, . Proof The vector is a member of and so it is a linear combination of basis vectors . Since 's columns are the 's, that can be expressed as: there is a such that (this is expressed compactly with matrix multiplication as in Example 3.5 and 3.6). Because is perpendicular to each member of the basis, we have this (again, expressed compactly). Solving for (showing that is invertible is an exercise) gives the formula for the projection matrix as . Example 3.9 To orthogonally project this vector onto this subspace first make a matrix whose columns are a basis for the subspace and then compute. With the matrix, calculating the orthogonal projection of any vector onto is easy. Exercises Problem 1 Project the vectors onto along . <li> <li> <li> Problem 2 Find . <li> <li> <li> <li> <li> <li> <li> Problem 3 This subsection shows how to project orthogonally in two ways, the method of Example 3.2 and 3.3, and the method of Theorem 3.8. To compare them, consider the plane specified by in . <li> Find a basis for . <li> Find and a basis for . <li> Represent this vector with respect to the concatenation of the two bases from the prior item. <li> Find the orthogonal projection of onto by keeping only the part from the prior item. <li> Check that against the result from applying Theorem 3.8. Problem 4 We have three ways to find the orthogonal projection of a vector onto a line, the Definition 1.1 way from the first subsection of this section, the Example 3.2 and 3.3 way of representing the vector with respect to a basis for the space and then keeping the part, and the way of Theorem 3.8. For these cases, do all three ways. <li> <li> Problem 5 Check that the operation of Definition 3.1 is well-defined. That is, in Example 3.2 and 3.3, doesn't the answer depend on the choice of bases? Problem 6 What is the orthogonal projection onto the trivial subspace? Problem 7 What is the projection of onto along if ? Problem 8 Show that if is a subspace with orthonormal basis then the orthogonal projection of onto is this. Problem 9 Prove that the map is the projection onto along if and only if the map is the projection onto along . (Recall the definition of the difference of two maps: .) Problem 10exer:PerpOnBasisImplPerp Show that if a vector is perpendicular to every vector in a set then it is perpendicular to every vector in the span of that set. Problem 11 True or false: the intersection of a subspace and its orthogonal complement is trivial. Problem 12 Show that the dimensions of orthogonal complements add to the dimension of the entire space. Problem 13 Suppose that are such that for all complements , the projections of and onto along are equal. Must equal ? (If so, what if we relax the condition to: all orthogonal projections of the two are equal?) Problem 14exer:AlgOfPerps Let be subspaces of . The perp operator acts on subspaces; we can ask how it interacts with other such operations. <li> Show that two perps cancel: . <li> Prove that implies that . <li> Show that . Problem 15Fund Thm of Linear Algebra The material in this subsection allows us to express a geometric relationship that we have not yet seen between the rangespace and the nullspace of a linear map. <li> Represent given by with respect to the standard bases and show that is a member of the perp of the nullspace. Prove that is equal to the span of this vector. <li> Generalize that to apply to any . <li> Represent with respect to the standard bases and show that are both members of the perp of the nullspace. Prove that is the span of these two. (Hint. See the third item of Problem 14.) <li> Generalize that to apply to any . This, and related results, is called the Fundamental Theorem of Linear Algebra in Strang 1993. Problem 16projection Define a projection to be a linear transformation with the property that repeating the projection does nothing more than does the projection alone: for all . <li> Show that orthogonal projection onto a line has that property. <li> Show that projection along a subspace has that property. <li> Show that for any such there is a basis for such that where is the rank of . <li> Conclude that every projection is a projection along a subspace. <li> Also conclude that every projection has a representation in block partial-identity form. Problem 17symmetric A square matrix is symmetric if each entry equals the entry (i.e., if the matrix equals its transpose). Show that the projection matrix is symmetric. Strang 1980 Hint. Find properties of transposes by looking in the index under "transpose". /Solutions/ References Gilbert Strang The Fundamental Theorem of Linear Algebra American Mathematical Monthly American Mathematical Society Nov. 1993 848-855. Gilbert Strang Linear Algebra and its Applications 2nd Hartcourt Brace Javanovich 1980 Linear Algebra Projection Onto a Subspace Gram-Schmidt Orthogonalization Topic: Line of Best Fit