# Lecture 10-3: Optimality Conditions for Linearly Constrained Problems

Download the original slides: [CMSE382-Lec10_3.pdf](CMSE382-Lec10_3.pdf)

```{warning}
This is an AI-generated transcript of the lecture slides and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.
```

---

## This Lecture

**Topics:**

- Orthogonal projection onto an affine space
- Orthogonal projection onto hyperplanes

**Announcements:**

- Quiz today!

---

## Orthogonal projection using KKT conditions

### Recall: Orthogonal projection

**Definition (Recall: Orthogonal projection operator)**

Given a nonempty closed convex set $C$, the orthogonal projection operator $P_C:\mathbb{R}^n \to C$ is defined by

$$
P_C(\mathbf{y}) = \arg\min{\|\mathbf{x} - \mathbf{y}\|^2: \mathbf{x} \in C}.
$$

![](../../../figures/orthogonal_projection_from_y.png)

- Returns the vector $\mathbf{x}$ in $C$ that is closest to input vector $\mathbf{y}$.
- Is a convex optimization problem:

$$
\begin{aligned}
& \text{min}  &  & \|\mathbf{x}-\mathbf{y}\|^2 \\
& \text{s.t.} & & \mathbf{x} \in C.
\end{aligned}
$$

---

### Orthogonal Projection onto an Affine Space with KKT conditions

Let $C$ be an affine space

$$\{\mathbf{x} \in \mathbb{R}^n: A\mathbf{x}=\mathbf{b} \},$$

where $A\in \mathbb{R}^{m\times n}$ and $\mathbf{b} \in \mathbb{R}^m$.
Assume that the rows of $A$ are linearly independent.

Given $\mathbf{y} \in \mathbb{R}^n$, find $P_C(\mathbf{y})$ which is the solution to the optimization problem

$$
\begin{aligned}
& \min_{\mathbf{x}} & & \|\mathbf{x} - \mathbf{y}\|^{2} \\
& \text{s.t.} & & A\mathbf{x} = \mathbf{b}.
\end{aligned}
$$

- The Langrangian simplifies to

$$L(\mathbf{x},\boldsymbol{\mu}) = \|\mathbf{x} - \mathbf{y}\|^{2} + \boldsymbol{\mu}^{\top} (A \mathbf{x} - \mathbf{b}),$$

for $\boldsymbol{\mu} \in \mathbb{R}^m$

- The KKT stationarity conditions are:

$$
\nabla_{\mathbf{x}} L(\mathbf{x},\boldsymbol{\mu}) = 2\mathbf{x} -2 \mathbf{y} + A^{\top} \boldsymbol{\mu}=\mathbf{0}
$$

- Solving with ${A}\mathbf{x} = \mathbf{b}$ gives

$$
\begin{aligned}
P_C(\mathbf{y}) &= \mathbf{y}-A^{\top}(AA^{\top})^{-1}(A\mathbf{y}-\mathbf{b})
\end{aligned}
$$

---

### Orthogonal Projection onto a hyperplane with KKT conditions

Given a hyperplane

$$H=\{\mathbf{x} \in \mathbb{R}^n: \mathbf{a}^{\top} \mathbf{x} =b\}.$$

Given $\mathbf{y} \in \mathbb{R}^n$, find $P_H(\mathbf{y})$ which is the solution to the optimization problem

$$
\begin{aligned}
& \min_{\mathbf{x}} & & \|\mathbf{x} - \mathbf{y}\|^{2} \\
& \text{s.t.} & & \mathbf{x} \in H
\end{aligned}
$$

- Special case of orthogonal projection onto an affine space with $A=\mathbf{a}^{\top}$ and $\mathbf{b}=b$.
- Replacing in the previous solution gives

$$
\begin{aligned}
P_H(\mathbf{y})& =\mathbf{y}-\mathbf{a}(\mathbf{a}^{\top} \mathbf{a})^{-1}(\mathbf{a}^{\top} \mathbf{y} - b)\\
& =\mathbf{y} - \frac{\mathbf{a}^{\top}\mathbf{y}-b}{\|\mathbf{a}\|^2} \mathbf{a}
\end{aligned}
$$