# Lecture 11-1: The KKT Conditions

Download the original slides: [CMSE382-Lec11_1.pdf](CMSE382-Lec11_1.pdf)

```{warning}
This is an AI-generated transcript of the lecture slides and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.
```

---

## This Lecture

**Topics:**

- Feasible descent direction
- Inequality and equality constrained problems
- Example: Equality constrained
- Example KKT not satisfied

**Announcements:**

- None

---

## Inequality and equality constrained problems

### Outlook

![](../../../figures/material_outline_KKT.png)

---

### Idea: Feasible descent direction

![](../../../figures/feasible_descent_direction.png)

---

### Feasible descent direction

**Theorem (Feasible descent direction)**

Consider the optimization problem

$$
\begin{aligned}
\llap{(P)} \hspace{0.5in}
& \text{minimize}  & & f(\mathbf{x}) \\
& \text{such that} & & \mathbf{x} \in C
\end{aligned}
$$

where $f$ is continuously differentiable function over the set $C \subseteq \mathbb{R}^n$.

Then a vector $\mathbf{d} \neq 0$ is called a *feasible descent direction at $\mathbf{x} \in C$* if

- $\nabla f(\mathbf{x})^{\top} \mathbf{d} < 0$, and
- there exists $\e > 0$ such that $\mathbf{x} + t \mathbf{d} \in C$ for all $t \in [0,\e]$.

**Lemma**

If $\mathbf{x}^*$ is a local optimal solution, then there are no feasible descent directions at $\mathbf{x}^*$.

Idea: there is no direction you can move that will both decrease the function's value and stay within the problem's constraints.

---

### Recall: Active constraints

![](../../../figures/active_constraints_abstract.png)

**Definition (Recall: Active constraints)**

Given a set of inequalities

$$
g_i(\mathbf{x}) \leq 0, \quad i=1,2,\ldots, m,
$$

where $g_i:\mathbb{R}^n \to \mathbb{R}$ are functions, and a vector $\tilde{\mathbf{x}} \in \mathbb{R}^n$, the *active constraints at $\tilde{\mathbf{x}}$* are the constraints satisfied as equalities at $\tilde{\mathbf{x}}$.
The set of active constraints is denoted by

$$
I(\tilde{\mathbf{x}}) = \{i:g_i(\tilde{\mathbf{x}})=0\}.
$$

---

### Regular Points

Consider the minimization problem

$$
\begin{aligned}
& \text{min}  & & f(\mathbf{x}) \\
& \text{such that} & & g_i(\mathbf{x}) \leq 0, i=1,2,\ldots m, \\
&  & & h_j(\mathbf{x}) = 0, j=1,2,\ldots p.
\end{aligned}
$$

where $f,g_1,\ldots,g_m,h_1,h_2,\ldots,h_p$ are continuously differentiable functions over $\mathbb{R}^n$.

**Definition**

A feasible point $\mathbf{x}^*$ is called *regular* if the gradients of the active constraints among the inequality constraints and of the equality constraints

$$
\{\nabla g_i(\mathbf{x}^*)\mid i \in I(\mathbf{x}^*)\}
\cup \{\nabla h_j(\mathbf{x}^*) \mid j=1,\ldots, p\}
$$

are linearly independent.

- Feasible points that are not regular are called *irregular* points.

---

### Linear Independence

**Definition**

A collection of vectors

$$
\{\mathbf{v}_1,\mathbf{v}_2,\ldots,\mathbf{v}_k\}
$$

is *linearly independent* if the only solution to the equation

$$
\alpha_1 \mathbf{v}_1 + \alpha_2 \mathbf{v}_2 + \ldots + \alpha_k \mathbf{v}_k = \mathbf{0}
$$

is $\alpha_1 = \alpha_2 = \ldots = \alpha_k = 0$.

**Some methods for checking:**

- Check using the definition directly.
- If there is one vector, this is linearly independent if it is nonzero.
- If there are two vectors, they are linearly independent if they are not scalar multiples of each other.
- Put the $k$ vectors as columns in a matrix $A$.
- If $k\leq n$ and $\mathrm{rank}(A) = k$, then the vectors are linearly independent.
- If $k=n$, then the vectors are linearly independent if $\det(A) \neq 0$.
- If $k>n$, then the vectors are linearly dependent.

---

### KKT Points

Consider the minimization problem

$$
\begin{aligned}
& \text{min}  & & f(\mathbf{x}) \\
& \text{such that} & & g_i(\mathbf{x}) \leq 0, i=1,2,\ldots m, \\
&  & & h_j(\mathbf{x}) = 0, j=1,2,\ldots p.
\end{aligned}
$$

where $f,g_1,\ldots,g_m,h_1,h_2,\ldots,h_p$ are continuously differentiable functions over $\mathbb{R}^n$.

**Definition**

A feasible point $\mathbf{x}^*$ is called a *KKT point*
if there exist $\lambda_1,\lambda_2,\ldots,\lambda_m \geq 0$ and $\mu_1,\mu_2,\ldots,\mu_p \in \mathbb{R}$ such that

$$
\nabla f(\mathbf{x}^*) + \sum\limits_{i=1}^m{\lambda_i \nabla g_i(\mathbf{x}^*)} + \sum\limits_{j=1}^p{\mu_j \nabla h_j(\mathbf{x}^*)} = \mathbf{0},
$$

$$
\lambda_i g_i(\mathbf{x}^*) = 0, i=1,2,\ldots, m.
$$

---

### KKT conditions for Inequality and equality constrained problems

**Theorem (Inequality and equality constrained problems)**

Let $\mathbf{x}^*$ be a local minimum of the problem

$$
\begin{aligned}
& \text{min}  & & f(\mathbf{x}) \\
& \text{such that} & & g_i(\mathbf{x}) \leq 0, i=1,2,\ldots m, \\
&  & & h_j(\mathbf{x}) = 0, j=1,2,\ldots p.
\end{aligned}
$$

where $f,g_1,\ldots,g_m,h_1,h_2,\ldots,h_p$ are continuously differentiable functions over $\mathbb{R}^n$.
Suppose that $\mathbf{x}^*$ is **regular**, then $\mathbf{x}^*$ is a **KKT point**.

- A necessary condition for local optimality of a regular point is that it is a KKT point.
- Regularity is not required in the linearly constrained case.