Worksheet 8-2: Orthogonal Projection

Worksheet 8-2: Orthogonal Projection#

Download: CMSE382-WS8_2.pdf

Warning

This is an AI-generated transcript of the worksheet and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.

Worksheet 8-2: Q1#

For the following sets and each point drawn (\(v\), \(w\), \(x\), \(y\), and \(z\)), mark the point that minimizes the projection operator (\(P_C(v)\), \(P_C(w)\), \(P_C(x)\), \(P_C(y)\), and \(P_C(z)\)).

Q1 figure

Worksheet 8-2: Q2#

Q2 figure

For each of the shown vectors \(\mathbf{x}\), answer the following

  1. Find an expression for the non-negative real part \([\mathbf{x}]_+\) in terms of \(\mathbf{x} = (x_1,x_2)\).

  2. Sketch \([\mathbf{x}]_+\) for each vector.

Worksheet 8-2: Q3#

  1. Consider the set \(C = \{(y_1,y_2,y_3) \in \mathbb{R}^3 \mid y_1\geq 0, y_2\geq 0, y_3 = 0 \}\). Write the orthogonal projection operator \(P_C(\mathbf{x})\) for any \(\mathbf{x} = (x_1,x_2,x_3) \in \mathbb{R}^3\). What point in \(C\) minimizes \(P_c(\mathbf{x})\)? Write it in terms of \([\cdot]_+\) if possible.

  2. For each of the following points, write the expression for the projected point in terms of just \(x_1, x_2, x_3\). Sketch the point.

Q3 figure

Worksheet 8-2: Q4#

A box is a subset of \(\mathbb{R}^n\) of the form

\[\begin{split} \begin{align*} B &= [\ell_1,u_1]\times [\ell_2, u_2]\times \ldots \times [\ell_n,u_n]\\ & =\{\mathbf{x} \in \mathbb{R}^n: \ell_i \leq x_i \leq u_i\}, \end{align*} \end{split}\]

where \(\ell_i \leq u_i\) for all \(i=1,2,\ldots,n\).

Q4 figure

We will assume that some of the \(u_i\)s can be \(\infty\) and some of the \(\ell_i\)s can be \(-\infty\); but in these cases we will assume that \(-\infty\) or \(\infty\) are not contained in the intervals. The figure above shows some two examples in boxes in \(\mathbb{R}^2\) and \(\mathbb{R}^3\). The orthognal projection on the box is the minimizer of the convex optimization problem

\[\begin{split} \begin{aligned} & \text{min} & & \|\mathbf{y}-\mathbf{x}\|^2 \\ & \text{s.t.} & & \mathbf{y} \in B \end{aligned} \end{split}\]

for a given box \(B\).

  1. Write the minimization problem above in terms of only \(x_i\)’s and \(y_i\)’s.

  2. Is the resulting functional equation separable? Justify your answer.

  3. Write down and solve the optimization problem for each \(y_i\). Use this to determine \(\mathbf{y} = P_C(\mathbf{x})\).

Worksheet 8-2: Q5#

Consider the normal ball in \( \mathbb{R}^2\), \(C = B[0,r] = \{\mathbf{y} =(y_1,y_2) \mid \|\mathbf{y}\|_2 \leq r\}\). We will find the projection \(P_C(\mathbf{x})\) for some point \(\mathbf{x} \in \mathbb{R}^2\), which is the \(\mathbf{y}\) that minimizes

\[\begin{split} \begin{aligned} & \text{min} & & \|\mathbf{y}-\mathbf{x}\|^2 \\ & \text{s.t.} & & \|\mathbf{y}\|^2 \leq r^2. \end{aligned} \end{split}\]

  1. Assume \(\mathbf{x} \in B[0,r]\). What is \(P_C(\mathbf{x})\) and why?

  2. What is \(\nabla f(\mathbf{z})\) for \(f(\mathbf{z}) = \|\mathbf{z}\|^2\)?

  3. Now we can deal with the case where \(\mathbf{x} \not\in B[0,r]\), equivalently \(\|\mathbf{x}\|^2 \geq r^2\). We know (from the first order optimality condition for local optima, Thm 2.6 in the book) that if \(\mathbf{x}^* \in \text{int}(C)\) is a local optimum and all partial derivatives exist, then \(\nabla f(\mathbf{x}^*) = 0\). If \(\mathbf{x} \not\in B[0,r]\) and somehow \(\mathbf{x}^* = P_C(\mathbf{x}) \in \text{int}(B[0,r])\), use your calculated gradient above to conclude that the result is impossible so \(P_C(\mathbf{x})\) must be on the boundary of \(B[0,r]\).

  4. By the previous, we know that if \(\|\mathbf{x}\|\geq r\), the solution must be on the boundary, so we can replace the problem with

\[\begin{split} \begin{aligned} & \text{min} & & \|\mathbf{y}-\mathbf{x}\|^2 \\ & \text{s.t.} & & \|\mathbf{y}\|^2 = r^2. \end{aligned} \end{split}\]

Then I can expand \(\|\mathbf{y}-\mathbf{x}\|^2 = \|\mathbf{y}\|^2 -2 \mathbf{y}^\top \mathbf{x} + \|\mathbf{x}\|^2\) and replace this problem with

\[\begin{split} \begin{aligned} & \text{min} & & \|\mathbf{y}\|^2 -2 \mathbf{y}^\top \mathbf{x} + \|\mathbf{x}\|^2 \\ & \text{s.t.} & & \|\mathbf{y}\|^2= r^2. \end{aligned} \end{split}\]

Why, then, can I replace this problem with the following problem?

\[\begin{split} \begin{aligned} & \text{min} & & -2 \mathbf{y}^\top \mathbf{x} \\ & \text{s.t.} & & \|\mathbf{y}\|^2 = r^2. \end{aligned} \end{split}\]

  1. The Cauchy-Schwartz inequality (\(|\mathbf{u}^\top \mathbf{v}| \leq \|\mathbf{u}\| \cdot \|\mathbf{v}\|\)) gives us a bound

\[ \mathbf{y}^\top \mathbf{x} \leq |\mathbf{y}^\top \mathbf{x}| \leq \|\mathbf{y}\| \cdot \|\mathbf{x}\| \]

Use the above to justify each inequality below.

\[ -2 \mathbf{y}^\top \mathbf{x} \geq -2\|\mathbf{y}\| \|\mathbf{x}\| = -2r\|\mathbf{x}\|. \]

  1. Check that equality in Eqn. (bound) (meaning \(-2 \mathbf{y}^\top \mathbf{x} = -2r\|\mathbf{x}\|\)) occurs when \(\mathbf{y}^* = r\frac{\mathbf{x}}{\|\mathbf{x}\|}\).

  2. Check that \(\mathbf{y}^* = r\frac{\mathbf{x}}{\|\mathbf{x}\|}\) is in \(B[0,r]\).

  3. Putting the above together, fill in the orthogonal projection for the ball:

\[\begin{split} P_{B[0,r]} = \begin{cases} \boxed{\phantom{\prod} \phantom{\prod}} & \|\mathbf{x}\|\leq r \\ \\ \boxed{\phantom{\prod} \phantom{\prod}} & \|\mathbf{x}\| > r \end{cases} \end{split}\]
Contents