Lecture 10-3: Optimality Conditions for Linearly Constrained Problems

Lecture 10-3: Optimality Conditions for Linearly Constrained Problems#

Download the original slides: CMSE382-Lec10_3.pdf

Warning

This is an AI-generated transcript of the lecture slides and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.

This Lecture#

Topics:

  • Orthogonal projection onto an affine space

  • Orthogonal projection onto hyperplanes

Announcements:

  • Quiz today!

Orthogonal projection using KKT conditions#

Recall: Orthogonal projection#

Definition (Recall: Orthogonal projection operator)

Given a nonempty closed convex set \(C\), the orthogonal projection operator \(P_C:\mathbb{R}^n \to C\) is defined by

\[ P_C(\mathbf{y}) = \arg\min{\|\mathbf{x} - \mathbf{y}\|^2: \mathbf{x} \in C}. \]

  • Returns the vector \(\mathbf{x}\) in \(C\) that is closest to input vector \(\mathbf{y}\).

  • Is a convex optimization problem:

\[\begin{split} \begin{aligned} & \text{min} & & \|\mathbf{x}-\mathbf{y}\|^2 \\ & \text{s.t.} & & \mathbf{x} \in C. \end{aligned} \end{split}\]

Orthogonal Projection onto an Affine Space with KKT conditions#

Let \(C\) be an affine space

\[\{\mathbf{x} \in \mathbb{R}^n: A\mathbf{x}=\mathbf{b} \},\]

where \(A\in \mathbb{R}^{m\times n}\) and \(\mathbf{b} \in \mathbb{R}^m\). Assume that the rows of \(A\) are linearly independent.

Given \(\mathbf{y} \in \mathbb{R}^n\), find \(P_C(\mathbf{y})\) which is the solution to the optimization problem

\[\begin{split} \begin{aligned} & \min_{\mathbf{x}} & & \|\mathbf{x} - \mathbf{y}\|^{2} \\ & \text{s.t.} & & A\mathbf{x} = \mathbf{b}. \end{aligned} \end{split}\]

  • The Langrangian simplifies to

\[L(\mathbf{x},\boldsymbol{\mu}) = \|\mathbf{x} - \mathbf{y}\|^{2} + \boldsymbol{\mu}^{\top} (A \mathbf{x} - \mathbf{b}),\]

for \(\boldsymbol{\mu} \in \mathbb{R}^m\)

  • The KKT stationarity conditions are:

\[ \nabla_{\mathbf{x}} L(\mathbf{x},\boldsymbol{\mu}) = 2\mathbf{x} -2 \mathbf{y} + A^{\top} \boldsymbol{\mu}=\mathbf{0} \]

  • Solving with \({A}\mathbf{x} = \mathbf{b}\) gives

\[ \begin{aligned} P_C(\mathbf{y}) &= \mathbf{y}-A^{\top}(AA^{\top})^{-1}(A\mathbf{y}-\mathbf{b}) \end{aligned} \]

Orthogonal Projection onto a hyperplane with KKT conditions#

Given a hyperplane

\[H=\{\mathbf{x} \in \mathbb{R}^n: \mathbf{a}^{\top} \mathbf{x} =b\}.\]

Given \(\mathbf{y} \in \mathbb{R}^n\), find \(P_H(\mathbf{y})\) which is the solution to the optimization problem

\[\begin{split} \begin{aligned} & \min_{\mathbf{x}} & & \|\mathbf{x} - \mathbf{y}\|^{2} \\ & \text{s.t.} & & \mathbf{x} \in H \end{aligned} \end{split}\]

  • Special case of orthogonal projection onto an affine space with \(A=\mathbf{a}^{\top}\) and \(\mathbf{b}=b\).

  • Replacing in the previous solution gives

\[\begin{split} \begin{aligned} P_H(\mathbf{y})& =\mathbf{y}-\mathbf{a}(\mathbf{a}^{\top} \mathbf{a})^{-1}(\mathbf{a}^{\top} \mathbf{y} - b)\\ & =\mathbf{y} - \frac{\mathbf{a}^{\top}\mathbf{y}-b}{\|\mathbf{a}\|^2} \mathbf{a} \end{aligned} \end{split}\]
Contents