Lecture 11-2: The KKT Conditions

Lecture 11-2: The KKT Conditions#

Download the original slides: CMSE382-Lec11_2.pdf

Warning

This is an AI-generated transcript of the lecture slides and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.

This Lecture#

Topics:

  • The convex case: KKT sufficiency

  • Slater conditions

  • The convex case: KKT necessity

Announcements:

  • HW…

Big picture#

Big picture view#

Sufficiency and Necessity#

Sufficient

  • All KKT points are optima.

  • Optima might not be KKT points.

Necessary

  • All optima are KKT points.

  • KKT points might not be optima.

Convex case: KKT sufficiency#

Recall: KKT conditions for Inequality and equality constrained problems#

Theorem (Recall: Inequality and equality constrained problems)

Let \(\mathbf{x}^*\) be a local minimum of the problem

\[\begin{split} \begin{aligned} & \text{min} & & f(\mathbf{x}) \\ & \text{such that} & & g_i(\mathbf{x}) \leq 0, i=1,2,\ldots m, \\ & & & h_j(\mathbf{x}) = 0, j=1,2,\ldots p. \end{aligned} \end{split}\]

where \(f,g_1,\ldots,g_m,h_1,h_2,\ldots,h_p\) are continuously differentiable functions over \(\mathbb{R}^n\). Suppose that \(\mathbf{x}^*\) is regular, then \(\mathbf{x}^*\) is a KKT point.

  • A necessary condition for local optimality of a regular point is that it is a KKT point.

  • Regularity is not required in the linearly constrained case.

Sufficiency of KKT conditions for convex problems#

Theorem

Let \(\mathbf{x}^*\) be a feasible solution of

\[\begin{split} \begin{aligned} & \text{min} & & f(\mathbf{x}) \\ & \text{such that} & & g_i(\mathbf{x}) \leq 0, i=1,\ldots m, \\ & & & h_j(\mathbf{x}) = 0, j=1,\ldots p. \end{aligned} \end{split}\]

where \(f,g_1,\ldots,g_m\) are continuously differentiable convex functions over \(\mathbb{R}^n\), and \(h_1,h_2,\ldots,h_p\) are affine functions.

Suppose that there exist \(\lambda_1, \ldots, \lambda_m \geq 0\) and \(\mu_1, \ldots, \mu_p \in \mathbb{R}\) such that

\[ \nabla f(\mathbf{x}^*) + \sum\limits_{i=1}^m{\lambda_i \nabla g_i(\mathbf{x}^*)} + \sum\limits_{j=1}^p{\mu_j \nabla h_j(\mathbf{x}^*)} = \mathbf{0}, \]
\[ \lambda_i g_i(\mathbf{x}^*) = 0, i=1,\ldots, m. \]

Then \(\mathbf{x}^*\) is an optimal solution.

In convex problems, the KKT conditions are always sufficient for optimality. No further conditions (such as regularity) are required.

Convex case: KKT necessity#

Slater’s condition#

Definition (Slater’s condition)

Given a set of convex inequalities

\[ g_i(\mathbf{x}) \leq 0, \quad i=1,2,\ldots, m, \]

where \(g_1,g_2,\ldots, g_m\) are given convex functions, we say that Slater’s condition is satisfied if there exists \(\hat{\mathbf{x}} \in \mathbb{R}^n\) such that

\[ g_i(\hat{\mathbf{x}}) < 0, \quad i=1,2,\ldots, m. \]

  • Requires a point that strictly satisfies the constraints.

  • Does not require knowledge on candidates for the optimal solution.

  • Usually easier to check than regularity.

Necessity of KKT conditions under Slater’s condition#

Theorem

Let \(\mathbf{x}^*\) be an optimal solution of the problem

\[\begin{split} \begin{aligned} \min\ & f(\mathbf{x}) \\ \text{s.t.}\ & g_i(\mathbf{x}) \leq 0,\quad i=1,2,\dots, m, \end{aligned} \end{split}\]

where \(f, g_1, \dots, g_m\) are continuously differentiable functions over \(\mathbb{R}^n\). In addition, assume \(g_1, g_2, \dots, g_m\) are convex functions and there exists \(\hat{\mathbf{x}}\) which satisfies Slater’s condition. Then \(\mathbf{x}^*\) is a KKT point.

Note: The point \(\hat{\mathbf{x}}\) from Slater’s condition does not need to be the same as \(\mathbf{x}^*\) (the candidate for the optimal solution).

Generalized Slater’s condition#

Definition (Generalized Slater’s condition)

Consider the system

\[\begin{split} \begin{aligned} g_i(\mathbf{x}) &\leq 0, \quad i=1,2,\ldots, m, \\ h_j(\mathbf{x}) &\le 0, \quad j=1,2,\ldots, p, \\ s_k(\mathbf{x}) &= 0, \quad k=1,2,\ldots, q, \end{aligned} \end{split}\]

where \(g_1,g_2,\ldots, g_m\) are convex functions, and \(h_1,h_2,\ldots, h_p, s_1,s_2,\ldots,s_k\) are affine functions. We say that generalized Slater’s condition is satisfied if there exists \(\hat{\mathbf{x}} \in \mathbb{R}^n\) such that

\[\begin{split} \begin{aligned} g_i(\hat{\mathbf{x}}) &< 0, \quad i=1,2,\ldots, m, \\ h_j(\hat{\mathbf{x}}) &\leq 0, \quad j=1,2,\ldots, p, \\ s_k(\hat{\mathbf{x}}) &= 0, \quad k=1,2,\ldots, q. \end{aligned} \end{split}\]

Necessity of KKT conditions under the generalized Slater’s condition#

Theorem

Let \(\mathbf{x}^*\) be an optimal solution of

\[\begin{split} \begin{aligned} \min\ & f(\mathbf{x}) \\ \text{s.t.}\ & g_i(\mathbf{x}) \le 0,\quad i = 1, \ldots, m \\ & h_j(\mathbf{x}) \le 0,\quad j = 1, \ldots, p \\ & s_k(\mathbf{x}) = 0,\quad k = 1, \ldots, q, \end{aligned} \end{split}\]

where \(f\) and all \(g_i\) are continuously differentiable convex functions over \(\mathbb{R}^n\), and all \(h_j\), \(s_k\) are affine. Suppose that there exists \(\hat{\mathbf{x}} \in \mathbb{R}^n\) satisfying the generalized Slater’s condition.

Then \(\mathbf{x}^*\) is a KKT point, i.e. there exist multipliers \(\lambda_1, \ldots, \lambda_m,\eta_1,\ldots,\eta_p \ge 0\), \(\mu_1,\ldots,\mu_q \in \mathbb{R}\) such that

\[ \nabla f (\mathbf{x}^*) + \sum_{i=1}^{m} \lambda_i\nabla g_i(\mathbf{x}^*) + \sum_{j=1}^{p} \eta_j\nabla h_j(\mathbf{x}^*) + \sum_{k=1}^{q} \mu_k\nabla s_k(\mathbf{x}^*) = 0, \]
\[ \lambda_i g_i(\mathbf{x}^*) = 0, \quad i = 1,\ldots, m, \]
\[ \eta_j h_j(\mathbf{x}^*) = 0, \quad j = 1,\ldots, p. \]

Big picture#

Big picture view#

Contents