Lecture 2-2: Optimality Conditions - Part 2

Lecture 2-2: Optimality Conditions - Part 2#

Download the original slides: CMSE382-Lec2_2.pdf

Warning

This is an AI-generated transcript of the lecture slides and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.

Definite Matrices and Their Eigenvalues#

Positive (and Negative) Definite Matrices#

Definition: A symmetric matrix \(A \in \mathbb{R}^{n\times n}\) is:

Term

Notation

Condition

positive semidefinite

\(A \succeq 0\)

\(x^T A x \geq 0\) for every \(x \in \mathbb{R}^n\)

positive definite

\(A \succ 0\)

\(x^T A x > 0\) for every \(x \neq 0\) in \(\mathbb{R}^n\)

negative semidefinite

\(A \preceq 0\)

\(x^T A x \leq 0\) for every \(x \in \mathbb{R}^n\)

negative definite

\(A \prec 0\)

\(x^T A x < 0\) for every \(x \neq 0\) in \(\mathbb{R}^n\)

indefinite

there is an \(x \in \mathbb{R}^n\) such that \(x^T A x > 0\) and a \(y \in \mathbb{R}^n\) such that \(y^T A y < 0\)

Note: The book always assumes that a positive/negative definite/semidefinite matrix is symmetric.

Eigenvalue Characterization Theorem#

Theorem: For a symmetric matrix \(A \in \mathbb{R}^{n\times n}\), it is:

Term

Condition on eigenvalues

positive semidefinite

if and only if all eigenvalues are nonnegative (\(\geq 0\))

positive definite

if and only if all eigenvalues are positive (\(> 0\))

negative semidefinite

if and only if all eigenvalues are nonpositive (\(\leq 0\))

negative definite

if and only if all eigenvalues are negative (\(< 0\))

indefinite

if and only if it has at least one positive and one negative eigenvalue

Diagonal Entries#

Lemma: For a symmetric matrix \(A \in \mathbb{R}^{n\times n}\):

If…

Then…

positive semidefinite

all diagonal entries are nonnegative (\(\geq 0\))

positive definite

all diagonal entries are positive (\(> 0\))

negative semidefinite

all diagonal entries are nonpositive (\(\leq 0\))

negative definite

all diagonal entries are negative (\(< 0\))

Warning: You cannot conclude positive/negative definiteness by checking the diagonal entries!

Second Order Optimality Condition#

Recall: First Order Optimality Condition#

First order optimality condition: Let \(f:U\to \mathbb{R}\) be a function defined on \(U\subseteq \mathbb{R}^n\). Suppose \(x^* \in \text{int}(U)\) is a local optimum (minimum or maximum) point and that all partial derivatives of \(f\) exist at \(x^*\).

Then \(\nabla f(x^*)=0\).

Second Order Optimality Condition#

Theorem (Second order optimality condition): Let \(f \colon U \to \mathbb{R}\) be twice continuously differentiable on an open set \(U \subseteq \mathbb{R}^n\). Let \(\mathbf{x}^*\) be a stationary point of \(f\) (i.e., \(\nabla f(\mathbf{x}^*)=\mathbf{0}\)). Then:

  • Necessary optimality conditions:

    • If \(\mathbf{x}^*\) is a local minimum, then \(\nabla^2 f(\mathbf{x}^*) \succeq 0\) (positive semi-definite)

    • If \(\mathbf{x}^*\) is a local maximum, then \(\nabla^2 f(\mathbf{x}^*) \preceq 0\) (negative semi-definite)

  • Sufficient optimality conditions:

    • If \(\nabla^2 f(\mathbf{x}^*) \succ 0\) (positive definite), then \(\mathbf{x}^*\) is a strict local minimum

    • If \(\nabla^2 f(\mathbf{x}^*) \prec 0\) (negative definite), then \(\mathbf{x}^*\) is a strict local maximum

  • Saddle point:

    • If \(\nabla^2 f(\mathbf{x}^*)\) is an indefinite matrix, then \(\mathbf{x}^*\) is a saddle point.

Contents