# Lecture 12-1: Duality

Download the original slides: [CMSE382-Lec12_1.pdf](CMSE382-Lec12_1.pdf)

```{warning}
This is an AI-generated transcript of the lecture slides and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.
```

---

## This Lecture

**Topics:**

- Motivation for Duality
- Definition and weak duality

**Announcements:**

- Homework 5 is due today, Friday April 10, at 11:59pm.

---

## Motivation

### Duality Motivation

![](../../../figures/duality_motivation_pic.png)

- The optimal value (val) of unconstrained problem is lower bound for the constrained one.
  - $\text{val}(unconstrained) \leq \text{val}(constrained)$
- Interested in finding lower bounds for constrained optimization by solving unconstrained problems.

---

### Obtaining lower bounds

**Approach#1: Drop constraints**

$$
\begin{aligned}
& \text{min}  & & f(\mathbf{x}) = x^2+y^2+2x \\
& \text{such that} & & x+y=0.
\end{aligned}
$$

$$
\begin{aligned}
& \text{min}  & & f(\mathbf{x}) = x^2+y^2+2x
\end{aligned}
$$

- $\text{val}(unconstrained) \leq \text{val}(constrained)$
- May not be the best lower bound.
- We want the largest lower bound.

[Desmos example](https://www.desmos.com/3d/mnrsngmy2s)

---

### Obtaining lower bounds

**Approach#2: Optimize for best lower bounds**

$$
\begin{aligned}
\llap{(P)} \hspace{0.1in}
& \text{min}  & & f(\mathbf{x}) + \mu h(\mathbf{x}) \\
&&&= x^2+y^2+2x+\mu(x+y)\\
& \text{s.t.} & & h(\mathbf{x})=x+y=0.
\end{aligned}
$$

$$
\begin{aligned}
\llap{(P_\mu)} \hspace{0.1in}
& \text{min}  & & f(\mathbf{x}) + \mu h(\mathbf{x})\\
&&&= x^2+y^2+2x+\mu(x+y)\\
\end{aligned}
$$

- $\text{val}(\text{P}_{\mu}) \leq \text{val}(\text{P})$ for all $\mu \in \mathbb{R}$.
- Best lower bound is the solution to

$$\llap{($\text{D}$)} \hspace{0.2in} \max_{\mu}{\{\text{val}(P_{\mu})\}}.$$

[Desmos example](https://www.desmos.com/3d/mnrsngmy2s)

---

### Primal and Dual Problems

**Primal Problem**

$$
\begin{aligned}
\llap{(P)} \hspace{0.2in}
& \text{min}  & & f(\mathbf{x}) \\
& \text{such that} & & h(\mathbf{x})=0.
\end{aligned}
$$

**Dual Problem**

$$
\begin{aligned}
\llap{(D)} \hspace{0.2in}
& \text{max}  & & \text{val}(P_{\mu})
\end{aligned}
$$

---

## Duality Definition

### Dual objective function

Consider the general model referred to as the primal model

$$
\begin{aligned}
& f^* = \text{min}  & & f(\mathbf{x}) \\
& \text{such that} & & g_i(\mathbf{x}) \leq 0, i=1,2,\ldots m, \\
&  & & h_j(\mathbf{x}) = 0, j=1,2,\ldots p, \\
&  & & \mathbf{x} \in X, \text{ where } X \subseteq \mathbb{R}^n,
\end{aligned}
$$

and $f, g_i,h_j$ are functions defined on $X$.

The Lagrangian of the problem is

$$L(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu}) = f(\mathbf{x}) + \sum\limits_{i=1}^m{\lambda_i g_i(\mathbf{x})} + \sum\limits_{j=1}^p{\mu_j h_j(\mathbf{x})},$$

The dual objective function $q: \mathbb{R}_+^m \times \mathbb{R}^p \to \mathbb{R} \cup \{-\infty\}$ is

$$q(\boldsymbol{\lambda},\boldsymbol{\mu})=\min_{x\in X}{L(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu})},$$

---

### Dual problem

**Definition (Dual Problem)**

The dual problem is given by

$$
\begin{aligned}
& q^* = \max\limits_{\boldsymbol{\lambda},\boldsymbol{\mu}}  & & q(\boldsymbol{\lambda},\boldsymbol{\mu}) \\
& \text{such that} & &  (\boldsymbol{\lambda},\boldsymbol{\mu}) \in \text{dom}(q),
\end{aligned}
$$

where the domain of the dual objective function is

$$\text{dom}(q)=\{(\boldsymbol{\lambda},\boldsymbol{\mu}) \in \mathbb{R}_{+}^m \times \mathbb{R}^p: q(\boldsymbol{\lambda},\boldsymbol{\mu}) > -\infty\}.$$

---

### Convexity of the dual problem

**Theorem (Convexity of the dual problem)**

Let the dual problem be given by

$$
\begin{aligned}
& q^* = \max\limits_{\boldsymbol{\lambda},\boldsymbol{\mu}}  & & q(\boldsymbol{\lambda},\boldsymbol{\mu}) \\
& \text{such that} & & (\boldsymbol{\lambda},\boldsymbol{\mu}) \in \text{dom}(q),
\end{aligned}
$$

where $f,g_1, \ldots, g_m, h_1,\ldots, h_p$ are functions defined on $X \subseteq \mathbb{R}^n$, and $q(\boldsymbol{\lambda},\boldsymbol{\mu})=\min_{\mathbf{x}\in X}{L(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu})}$. Then

(a) $\text{dom}(q)$ is a convex set.

(b) $q$ is a concave function over $\text{dom}(q)$.

Maximizing a concave function over a convex set defines a convex problem.

---

### Weak duality theorem

**Primal Problem**

$$
\begin{aligned}
& f^* = \text{min}  & & f(\mathbf{x}) \\
& \text{such that} & & g_i(\mathbf{x}) \leq 0, i=1,2,\ldots m, \\
&  & & h_j(\mathbf{x}) = 0, j=1,2,\ldots p, \\
&  & & \mathbf{x} \in X, \text{ where } X \subseteq \mathbb{R}^n,
\end{aligned}
$$

and $f, g_i,h_j$ are functions defined on $X$.

**Dual Problem**

$$
\begin{aligned}
& q^* = \text{max}  & & q(\boldsymbol{\lambda},\boldsymbol{\mu}) \\
& \text{such that} & &  (\boldsymbol{\lambda},\boldsymbol{\mu}) \in \text{dom}(q),
\end{aligned}
$$

where $\text{dom}(q)=\{(\boldsymbol{\lambda},\boldsymbol{\mu}) \in \mathbb{R}_{+}^m \times \mathbb{R}^p: q(\boldsymbol{\lambda},\boldsymbol{\mu}) > -\infty\}$, and $q(\boldsymbol{\lambda},\boldsymbol{\mu})=\min_{\mathbf{x}\in X}{L(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\mu})}$.

**Theorem (Weak duality theorem)**

Consider the primal problem and its dual. Then $q^* \leq f^*$,
where $q^*, f^*$ are the optimal dual and primal values, respectively.