Lecture 10-2: Optimality Conditions for Linearly Constrained Problems

Lecture 10-2: Optimality Conditions for Linearly Constrained Problems#

Download the original slides: CMSE382-Lec10_2.pdf

Warning

This is an AI-generated transcript of the lecture slides and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.

This Lecture#

Topics (Review):

  • KKT conditions

  • Lagrangian function

Added topic:

  • Active Constraints

Announcements:

  • Quiz Weds April 1

Optimality Conditions#

Optimality conditions#

KKT for linearly constrained problems#

Theorem (Necessary optimality conditions)

Consider the minimization problem

\[\begin{split} \begin{aligned} (\text{P}) \quad &\min_{\mathbf{x}} f (\mathbf{x}) \\ &\text{s.t.} \quad \mathbf{a}_i^T \mathbf{x} \leq b_i, \quad i = 1, 2, \dots, m, \end{aligned} \end{split}\]

where \(f\) is continuously differentiable over \(\mathbb{R}^n\), \(a_1, a_2, \dots, a_m \in \mathbb{R}^n\), \(b_1, b_2, \dots, b_m \in \mathbb{R}\), and let \(\mathbf{x}^*\) be a local minimum point of (P). Then there exist \(\lambda_1, \lambda_2, \dots, \lambda_m \geq 0\) such that

\[ \nabla f (\mathbf{x}^*) + \sum_{i=1}^{m}\lambda_i a_i = 0, \quad \text{ and } \quad \lambda_i(\mathbf{a}_i^T \mathbf{x}^* - b_i) = 0, \quad i = 1, 2, \dots, m. \]

  • \(\lambda_1,\ldots,\lambda_m\) are Lagrange multipliers. Non-negative for minimization with inequality constraints.

KKT for convex linearly constrained problems#

Theorem (Necessary and sufficient optimality conditions)

Consider the minimization problem

\[\begin{split} \begin{aligned} (\text{P}) \quad &\min_{\mathbf{x}} f (\mathbf{x}) \\ &\text{s.t.} \quad \mathbf{a}_i^T \mathbf{x} \leq b_i, \quad i = 1, 2, \dots, m, \end{aligned} \end{split}\]

where \(f\) is a convex continuously differentiable over \(\mathbb{R}^n\), \(a_1, a_2, \dots, a_m \in \mathbb{R}^n\), \(b_1, b_2, \dots, b_m \in \mathbb{R}\), and let \(\mathbf{x}^*\) be a feasible solution of (P). Then \(\mathbf{x}^*\) is an optimal solution of (P) if and only if there exist \(\lambda_1, \lambda_2, \dots, \lambda_m \geq 0\) such that

\[ \nabla f (\mathbf{x}^*) + \sum_{i=1}^{m}\lambda_i a_i = 0, \quad \text{ and } \quad \lambda_i(\mathbf{a}_i^T \mathbf{x}^* - b_i) = 0, \quad i = 1, 2, \dots, m. \]

  • The condition \(\lambda_i(\mathbf{a}_i^T \mathbf{x}^* - b_i) = 0, \quad i = 1, 2, \dots, m\) is called the complementary slackness condition.

The Lagrangian function#

Definition (The Lagrangian function)

Consider the Nonlinear Programming Problem (NLP)

\[ (\text{NLP}) \quad \min_{\mathbf{x}} f (\mathbf{x}) \quad \text{s.t.} \quad \{g_i(\mathbf{x}) \leq 0\}_{i=1}^m, \quad \{h_j(\mathbf{x}) = 0\}_{j=1}^p, \]

where \(f\), and all the \(g_i\) and \(h_j\) are continuously differentiable over \(\mathbb{R}^n\).

The associated Lagrangian function is of the form

\[\begin{split} \begin{aligned} L(&\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu}) = \\ & f(\mathbf{x}) + \sum\limits_{i=1}^m{\lambda_i g_i(\mathbf{x})} + \sum\limits_{j=1}^p{\mu_j h_j(\mathbf{x})}. \end{aligned} \end{split}\]

The necessary KKT condition (stationarity condition) is

\[\begin{split} \begin{aligned} \nabla_{\mathbf{x}}&L(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu}) = \\ &\nabla f(\mathbf{x}) + \sum\limits_{i=1}^m{\lambda_i \nabla g_i(\mathbf{x})} + \sum\limits_{j=1}^p{\mu_j \nabla h_j(\mathbf{x})}=\mathbf{0}. \end{aligned} \end{split}\]

The Lagrangian function for linearly constrained optimization#

Recall the minimization problem with linear constraints

\[ (\text{Q}) \quad \min_{\mathbf{x}} f (\mathbf{x}) \quad \text{s.t.} \quad A \mathbf{x} \leq \mathbf{b}, \quad C \mathbf{x} = \mathbf{d}. \]

The associated Lagrangian function is of the form

\[ L(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu}) = f(\mathbf{x}) + \boldsymbol{\lambda}^{\top}(A \mathbf{x}-\mathbf{b}) + \boldsymbol{\mu}^{\top}(C \mathbf{x}-\mathbf{d}). \]

The necessary KKT condition \(\nabla f(\mathbf{x}^*) + \sum_{i=1}^{m} \lambda_i \mathbf{a}_i + \sum_{j=1}^{p} \mu_j \mathbf{c}_j = \mathbf{0}\) can be written in terms of the Lagrangian as

\[ \nabla_{\mathbf{x}} L(\mathbf{x},\boldsymbol{\lambda}, \boldsymbol{\mu}) = \nabla f(\mathbf{x}) + A^{\top} \boldsymbol{\lambda} + C^T \boldsymbol{\mu}=\mathbf{0}. \]

Steps for finding the stationary points for a linearly constrained problem#

  • Write the problem in the standard form

\[ \min_{\mathbf{x}} f (\mathbf{x}) \quad \text{s.t.} \quad A \mathbf{x} \leq \mathbf{b}, \quad C \mathbf{x} = \mathbf{d}. \]

  • Write down the Lagrangian function

\[ L(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu}) = f(\mathbf{x}) + \boldsymbol{\lambda}^{\top}(A \mathbf{x}-\mathbf{b}) + \boldsymbol{\mu}^{\top}(C \mathbf{x}-\mathbf{d}). \]

  • Write down the KKT conditions

\[ \nabla_{\mathbf{x}} L(\mathbf{x},\boldsymbol{\lambda}, \boldsymbol{\mu}) = \nabla f(\mathbf{x}) + A^{\top} \boldsymbol{\lambda} + C^T \boldsymbol{\mu}=\mathbf{0},\, \text{and}\, \lambda_i(\mathbf{a}_i^\top \mathbf{x}^* - b_i) = 0. \]

  • Write down the feasibility constraints

\[ (A \mathbf{x}-\mathbf{b})\quad \leq 0 \quad \text{and}\quad (C \mathbf{x}-\mathbf{d})=0 \]

  • If inequality constraints are present, include \(\boldsymbol{\lambda} \geq \mathbf{0}\) as a constraint.

  • Solve the stationarity and feasibility constraints for the stationary points of the problem.

  • If the problem is convex, then stationarity implies optimality.

Active Constraints#

Optimality conditions#

Active constraints

The constraint is active or binding when at the optimal solution the constraint holds with equality '='.

Video Example#

\[\begin{split} \begin{aligned} \min \quad & x+y \\ \text{s.t.} \quad & -6x - 10 y \le -100 \\ & x-y \le 0 \\ & 0.5 x + y \le 13 \\ & -x,-y \le 0, \end{aligned} \end{split}\]

Variables

  • \(x\), \(y\)

  • \(\lambda_1, \lambda_2, \lambda_3, \lambda_4, \lambda_5\)

Stationarity

\[\begin{split} \begin{aligned} 1 - 6\lambda_1 + \lambda_2 +0.5 \lambda_3 - \lambda_4 &= 0 \\ 1 - 10 \lambda_1-\lambda_2+\lambda_3 -\lambda_5 &= 0 \end{aligned} \end{split}\]

Slackness

\[\begin{split} \begin{aligned} \lambda_1(-6x-10y+100) &= 0 \\ \lambda_2(x-y) &= 0 \\ \lambda_3(0.5x+y-13) &= 0 \\ \lambda_4(-x) &= 0 \\ \lambda_5(-y) &= 0 \end{aligned} \end{split}\]

Feasibility

\[\begin{split} \begin{aligned} \lambda_i &\geq 0\\ (i = 1,&\ldots,5)\\ -6x-10y &\leq -100 \\ x-y &\leq 0 \\ 0.5x + y &\leq 13 \\ -x, -y &\leq 0. \end{aligned} \end{split}\]

Optimality conditions#

Slackness

\[\begin{split} \begin{gathered} \lambda_1(-6x-10y+100) = 0 \\ \lambda_2(x-y) = 0 \\ \lambda_3(0.5x+y-13) = 0 \\ \lambda_4(-x) = 0 \\ \lambda_5(-y) = 0 \end{gathered} \end{split}\]

Optimal solution at \((x,y)=(0,10)\).

Active constraints for example#

Which of the constraints is active or binding at the optimal solution?

\[\begin{split} \begin{aligned} -6x-10y+100 &= 0 \\ x &= 0 \end{aligned} \end{split}\]

\(\lambda_i=0\) if \(i\) is not active.

Big picture#

Big picture view#

Contents