Recapitulation

The optimal (or least squares) approximation to a square integrable function is uniquely determined by specifying a subspace in the Hilbert space. If one wishes to improve this optimal approximation, then the only way one can do this, in general, is to have a higher dimensional subspace, obtained by increasing the number of orthogonal spanning vectors. The quality of the approximation is measured by the squared magnitude of its error away from the given function.

Instead of judging the quality of an optimal approximation by its mean squared error, one can judge it by its length. The longer it is, the better. The length of an approximation in a subspace is not arbitrary. It is bounded from above, as we shall see presently. The closer an optimal approximation comes to this upper bound, the better the approximation.

What is happening geometrically is that our attention is shifting from the height of the right triangle $(f,h^\ast ,w^\ast )$ to its hypotenuse and its base.

Indeed, from the right triangle, whose sides satisfy the vector eqation

$$f=w^\ast_N +h^\ast_N~~,$$

one obtains the theorem of Pythagoras in Hilbert space:

$$\Vert f\Vert^2=\Vert w^\ast_N\Vert^2+\Vert h^\ast_N\Vert^2\,.$$

See Figure 1.6 on page . This evidently yields

$$\Vert f\Vert^2\ge\Vert w^\ast_N\Vert^2$$

which is Bessel's inequality.

Using the optimal ($=$ ``least square'') approximation

$$w^\ast_N = \sum^N_{i=1} c_i u_i\,,~~\qquad~~c_i=\langle u_i,f\rangle$$

one obtains

$$\Vert f\Vert^2\ge \sum^N_{i=1}\overline{c_i}c_i~;\qquad~c_i~\textrm{determined~ by~the~least~squares~approximation,}$$

which for our square integrable functions is

$$\int^b_a \vert f(x)\vert^2 \rho(x)dx\ge \sum^N_{i=1}\vert c_i\vert^2\,.$$

Geometrically this inequality says

$$(\textrm{length~of~vector})^2\ge\left(\begin{array}{c} \textrm{length~of~its~projection}\\ \textrm{onto~the~subspace~}W_N \end{array}\right)^2\,.$$

Consider now a sequence of subspaces

$$W_1\subset W_2\subset\cdots\subset W_N\subset W_{N+1}\subseteq\cdots\,,$$

the respective optimal approximations to the given function $f$ , and the corresponding sequence of least square errors

$$\Vert h^\ast_N\Vert^2 = \Vert f-\sum^N_{i=1} c_i u_i\Vert^2 ~, \qquad~ N=1,2,\dots\,\quad.$$

If $\Vert h^\ast_N\Vert^2$ approaches zero as $N$ tends to infinity, one obtains the following

Definition: The set of O.N. functions $\{ u_i\}$ is said to be complete if

$$\lim_{N\to\infty} \Vert h^\ast_N\Vert^2=0~~\qquad~~\forall\, f\in{\cal H}\,.$$

Thus the least squares approximation becomes exact in the limit of an infinite (generalized) Fourier series. From the Pythagorean theorem we therefore see that the Bessel's inequality becomes an equality

$$\Vert f \Vert^2 =\sum^\infty_{i=1}\vert c_i\vert^2$$

or

$$\langle f,f\rangle =\sum^\infty_{i=1} \langle f,u_i\rangle\langle u_i,f\rangle~~\quad~~\forall\, f\in{\cal H}\,.$$ (117)

This alternative completeness criterion is known as the completeness relation, or Parseval's relation.

Example (Spectral Decomposition of the Identity in $L^2(a,b)$ ):

Let us apply the completeness relation to the Hilbert space of square integrable functions on $[a,b]$ , $L^2(a,b)$ . Consider the fact that this relation implies (see homework set 1)

$$\langle f,g\rangle = \sum^\infty_{k=1}\langle f,u_k\rangle\langle u_k,g \rangle\,,~~f~\textrm{and}~g\in{\cal H}~~(=L^2(a,b))\,.$$

Explicitly, one has

$$\int^b_a \overline{f}(x)g(x)\rho (x)dx = \sum^\infty_{k=1}\int^b_a\overline{f} (x)u_k(x)\rho (x)dx\int^b_a\overline{u}_k (x')g(x')\rho (x') dx'\,.$$

This can be rewritten in terms of the Dirac delta function (which is developed in Section 2.2 starting on page ) as

$$\int^b_a\int^b_a \overline{f}(x)\delta (x-x')g(x')\rho (x)dxdx'$$

$$\qquad~\qquad= \int^b_a \int^b_a\overline{f}(x)\sum^\infty_{k=1}u_k(x)\overline{u}_k(x')\rho (x)\rho (x')g(x')dxdx'\,.$$

This holds for all $f,g\in{\cal H}=L^2(a,b)$ . Consequently, we have the following alternate form for the completeness of the set of orthonormal functions

$$\frac{\delta (x-x')}{\rho (x')} = \sum^\infty_{k=1}u_k(x)\overline{u}_k (x')$$

or

$$\frac{\delta (x-x')}{\rho (x')} = \sum^\infty_{k=1}\vert u_k(x)\rangle \langle u_k(x')\vert$$

in quantum mechanical notation.

Usually the orthonormal functions $u_k$ are the eigenfunctions of some operator (for example, the Sturm-Liouville operator $+$ boundary conditions, which we shall meet later). The Dirac delta function

$$\frac{\delta (x-x')}{\rho (x')}=\frac{\delta (x-x')}{\rho (x)}$$

can be viewed as the identity operator on the Hilbert space ${\cal H}$ . Consequently, the alternate form of the completeness relation

$$\frac{\delta (x-x')}{\rho (x)} = \sum^\infty_{k=1}u_k(x)\overline{u}_k (x')$$ (118)

can be viewed as a spectral representation of the identity operator in ${\cal H}$ .