# Lecture 10-2: Optimality Conditions for Linearly Constrained Problems

Download the original slides: [CMSE382-Lec10_2.pdf](CMSE382-Lec10_2.pdf)

```{warning}
This is an AI-generated transcript of the lecture slides and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.
```

---

## This Lecture

**Topics (Review):**

- KKT conditions
- Lagrangian function

**Added topic:**

- Active Constraints

**Announcements:**

- Quiz Weds April 1

---

## Optimality Conditions

### Optimality conditions

![](../../../figures/material_outline_linearly_constrained.png)

---

### KKT for linearly constrained problems

**Theorem (Necessary optimality conditions)**

Consider the minimization problem

$$
\begin{aligned}
(\text{P}) \quad &\min_{\mathbf{x}} f (\mathbf{x}) \\
&\text{s.t.} \quad \mathbf{a}_i^T \mathbf{x} \leq b_i, \quad i = 1, 2, \dots, m,
\end{aligned}
$$

where $f$ is continuously differentiable over $\mathbb{R}^n$, $a_1, a_2, \dots, a_m \in \mathbb{R}^n$, $b_1, b_2, \dots, b_m \in \mathbb{R}$, and let $\mathbf{x}^*$ be a local minimum point of (P). Then there exist $\lambda_1, \lambda_2, \dots, \lambda_m \geq 0$ such that

$$
\nabla f (\mathbf{x}^*) + \sum_{i=1}^{m}\lambda_i a_i = 0, \quad \text{ and } \quad \lambda_i(\mathbf{a}_i^T \mathbf{x}^* - b_i) = 0, \quad i = 1, 2, \dots, m.
$$

- $\lambda_1,\ldots,\lambda_m$ are Lagrange multipliers. Non-negative for minimization with inequality constraints.

---

### KKT for *convex* linearly constrained problems

**Theorem (Necessary and sufficient optimality conditions)**

Consider the minimization problem

$$
\begin{aligned}
(\text{P}) \quad &\min_{\mathbf{x}} f (\mathbf{x}) \\
&\text{s.t.} \quad \mathbf{a}_i^T \mathbf{x} \leq b_i, \quad i = 1, 2, \dots, m,
\end{aligned}
$$

where $f$ is a *convex* continuously differentiable over $\mathbb{R}^n$, $a_1, a_2, \dots, a_m \in \mathbb{R}^n$, $b_1, b_2, \dots, b_m \in \mathbb{R}$, and let $\mathbf{x}^*$ be a feasible solution of (P).
Then $\mathbf{x}^*$ is an optimal solution of (P) *if and only if* there exist $\lambda_1, \lambda_2, \dots, \lambda_m \geq 0$ such that

$$
\nabla f (\mathbf{x}^*) + \sum_{i=1}^{m}\lambda_i a_i = 0, \quad \text{ and } \quad \lambda_i(\mathbf{a}_i^T \mathbf{x}^* - b_i) = 0, \quad i = 1, 2, \dots, m.
$$

- The condition $\lambda_i(\mathbf{a}_i^T \mathbf{x}^* - b_i) = 0, \quad i = 1, 2, \dots, m$ is called the complementary slackness condition.

---

### The Lagrangian function

**Definition (The Lagrangian function)**

Consider the Nonlinear Programming Problem (NLP)

$$
(\text{NLP}) \quad \min_{\mathbf{x}} f (\mathbf{x}) \quad
\text{s.t.} \quad \{g_i(\mathbf{x})  \leq 0\}_{i=1}^m, \quad
\{h_j(\mathbf{x}) = 0\}_{j=1}^p,
$$

where $f$, and all the $g_i$ and $h_j$ are continuously differentiable over $\mathbb{R}^n$.

The associated *Lagrangian function* is of the form

$$
\begin{aligned}
L(&\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu}) =  \\
& f(\mathbf{x}) + \sum\limits_{i=1}^m{\lambda_i g_i(\mathbf{x})} + \sum\limits_{j=1}^p{\mu_j h_j(\mathbf{x})}.
\end{aligned}
$$

The *necessary KKT condition (stationarity condition)* is

$$
\begin{aligned}
\nabla_{\mathbf{x}}&L(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu}) = \\
&\nabla f(\mathbf{x}) + \sum\limits_{i=1}^m{\lambda_i \nabla g_i(\mathbf{x})} + \sum\limits_{j=1}^p{\mu_j \nabla h_j(\mathbf{x})}=\mathbf{0}.
\end{aligned}
$$

---

### The Lagrangian function for linearly constrained optimization

Recall the minimization problem *with linear constraints*

$$
(\text{Q}) \quad  \min_{\mathbf{x}} f (\mathbf{x}) \quad \text{s.t.} \quad A \mathbf{x} \leq \mathbf{b}, \quad C \mathbf{x} = \mathbf{d}.
$$

The associated *Lagrangian function* is of the form

$$
L(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu}) = f(\mathbf{x}) + \boldsymbol{\lambda}^{\top}(A \mathbf{x}-\mathbf{b}) + \boldsymbol{\mu}^{\top}(C \mathbf{x}-\mathbf{d}).
$$

The *necessary KKT condition* $\nabla f(\mathbf{x}^*) + \sum_{i=1}^{m} \lambda_i \mathbf{a}_i + \sum_{j=1}^{p} \mu_j \mathbf{c}_j = \mathbf{0}$ can be written in terms of the Lagrangian as

$$
\nabla_{\mathbf{x}} L(\mathbf{x},\boldsymbol{\lambda}, \boldsymbol{\mu}) = \nabla f(\mathbf{x}) + A^{\top} \boldsymbol{\lambda} + C^T \boldsymbol{\mu}=\mathbf{0}.
$$

---

### Steps for finding the stationary points for a linearly constrained problem

- Write the problem in the standard form

$$
\min_{\mathbf{x}} f (\mathbf{x}) \quad \text{s.t.} \quad A \mathbf{x} \leq \mathbf{b}, \quad C \mathbf{x} = \mathbf{d}.
$$

- Write down the Lagrangian function

$$
L(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu}) = f(\mathbf{x}) + \boldsymbol{\lambda}^{\top}(A \mathbf{x}-\mathbf{b}) + \boldsymbol{\mu}^{\top}(C \mathbf{x}-\mathbf{d}).
$$

- Write down the KKT conditions

$$
\nabla_{\mathbf{x}} L(\mathbf{x},\boldsymbol{\lambda}, \boldsymbol{\mu}) = \nabla f(\mathbf{x}) + A^{\top} \boldsymbol{\lambda} + C^T \boldsymbol{\mu}=\mathbf{0},\, \text{and}\, \lambda_i(\mathbf{a}_i^\top \mathbf{x}^* - b_i) = 0.
$$

- Write down the feasibility constraints

$$
(A \mathbf{x}-\mathbf{b})\quad \leq 0 \quad \text{and}\quad (C \mathbf{x}-\mathbf{d})=0
$$

- If inequality constraints are present, include $\boldsymbol{\lambda} \geq \mathbf{0}$ as a constraint.
- Solve the stationarity and feasibility constraints for the stationary points of the problem.
- If the problem is convex, then stationarity implies optimality.

---

## Active Constraints

### Optimality conditions

**Active constraints**

![](../../../figures/active_constraints_geometric.png)

The constraint is *active* or *binding* when at the optimal solution the constraint holds with equality `'='`.

---

### Video Example

$$
\begin{aligned}
\min \quad & x+y \\
  \text{s.t.} \quad & -6x - 10 y \le -100 \\
& x-y        \le 0 \\
& 0.5 x + y  \le 13 \\
& -x,-y         \le 0,
\end{aligned}
$$

**Variables**

- $x$, $y$
- $\lambda_1, \lambda_2, \lambda_3, \lambda_4, \lambda_5$

**Stationarity**

$$
\begin{aligned}
   1 - 6\lambda_1 + \lambda_2 +0.5 \lambda_3 - \lambda_4 &= 0 \\
   1 - 10 \lambda_1-\lambda_2+\lambda_3 -\lambda_5 &= 0
\end{aligned}
$$

**Slackness**

$$
\begin{aligned}
   \lambda_1(-6x-10y+100) &= 0 \\
   \lambda_2(x-y) &= 0 \\
   \lambda_3(0.5x+y-13) &= 0 \\
   \lambda_4(-x) &= 0 \\
   \lambda_5(-y) &= 0
\end{aligned}
$$

**Feasibility**

$$
\begin{aligned}
\lambda_i &\geq 0\\
(i = 1,&\ldots,5)\\
-6x-10y &\leq -100 \\
x-y &\leq 0 \\
0.5x + y &\leq 13 \\
-x, -y &\leq 0.
\end{aligned}
$$

---

### Optimality conditions

**Slackness**

$$
\begin{gathered}
   \lambda_1(-6x-10y+100) = 0 \\
   \lambda_2(x-y) = 0 \\
   \lambda_3(0.5x+y-13) = 0 \\
   \lambda_4(-x) = 0 \\
   \lambda_5(-y) = 0
\end{gathered}
$$

Optimal solution at $(x,y)=(0,10)$.

![](../../../figures/linear_optimization_muffins_example.png)

---

### Active constraints for example

Which of the constraints is active or binding at the optimal solution?

$$
\begin{aligned}
-6x-10y+100 &= 0 \\
x &= 0
\end{aligned}
$$

$\lambda_i=0$ if $i$ is not active.

![](../../../figures/linear_optimization_muffins_example_lambdas.png)

---

## Big picture

### Big picture view

![](../../../figures/useful_solution_charts/KKT_conditions_summary.png)