# Worksheet 12-1: Duality (with Solutions)

Download: [CMSE382-WS12_1.pdf](CMSE382-WS12_1.pdf), [CMSE382-WS12_1-Soln.pdf](CMSE382-WS12_1-Soln.pdf)

```{warning}
This is an AI-generated transcript of the worksheet and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.
```

---

## Worksheet 12-1: Q1

Consider the problem

$$
\begin{aligned}
f^* = \min &\quad x_1^2 - x_2\\
\text{s.t.} &\quad x_1-x_2^2 = 0,
\end{aligned}
$$

1. Find the optimal solution by replacing $x_1$ with $x_2^2$ and making the objective a single variable function. What is the optimal value $f^*$? (Note $f^*$ is the minimimal value of the objection function $f(x_1,x_2) = x_1^2-x_2$, not the point $x_1,x_2$ that achieves the minimum.)

   ```{dropdown} Solution
   After replacement, the objective function becomes $f(x_2) = x_2^4-x_2$.
   Take a derivative, we have $f'(x_2) = 4x_2^3-1$.
   Set $f'(x_2)=0$, we have $x_2 = \sqrt[3]{\frac{1}{4}}$.
   So, the optimal value is

   $$f^* = -\frac{3}{4\sqrt[3]{4}} \approx -0.472.$$

   ```

2. Now solve the problem using duality:

   See [this desmos plot](https://www.desmos.com/3d/wwuya1epxa) for the visualization of the problem we discussed in class.

   a. Write down the Lagrangian.

      ```{dropdown} Solution
      The Lagrangian is

      $$L(x_1,x_2,\mu) = x_1^2-x_2 + \mu(x_1-x_2^2).$$

      ```

   b. What is $\nabla_{x_1,x_2} L(x_1,x_2,\mu)$? When is $\nabla_{x_1,x_2} L(x_1,x_2,\mu) = 0$?

      ```{dropdown} Solution
      The gradient is

      $$\nabla_{x_1,x_2} L(x_1,x_2,\mu) = \begin{bmatrix} 2x_1 + \mu\\ -1 - 2\mu x_2 \end{bmatrix}.$$

      The gradient is zero when $x_1 = -\frac{\mu}{2}$ and $x_2 = -\frac{1}{2\mu}$.
      ```

   c. Write a formula for $q(\mu) = \min_{x_1,x_2} L(x_1,x_2,\mu)$ in terms of $\mu$.

      ```{dropdown} Solution
      From above, we have that the Lagrangian is minimized at $x_1 = -\frac{\mu}{2}$ and $x_2 = -\frac{1}{2\mu}$.
      This means the minimum value of the Lagrangian is

      $$\begin{aligned}
      L\left(-\frac{\mu}{2}, -\frac{1}{2\mu}, \mu\right) &=
      \left(-\frac{\mu}{2}\right)^2 - \left(-\frac{1}{2\mu}\right) + \mu\left(-\frac{\mu}{2} - \left(-\frac{1}{2\mu}\right)^2\right)\\
      &= -\frac{\mu^2}{4} + \frac{1}{4\mu}.
      \end{aligned}$$

      ```

   d. What value of $\mu$ maximizes $q(\mu)$?

      ```{dropdown} Solution
      Take a derivative, we have $q'(\mu) = -\frac{\mu}{2} - \frac{1}{4\mu^2}$.
      Set $q'(\mu)=0$, we have

      $$\mu = -\sqrt[3]{\frac{1}{2}} = -2^{-1/3}.$$

      ```

   e. What is the optimal dual value $q^* = \max_\mu q(\mu)$?

      ```{dropdown} Solution
      The optimal dual value is

      $$\begin{aligned}
      q^* = q(-2^{-1/3}) &= \frac{(-2^{-1/3})^2}{4} + \frac{1}{4(-2^{-1/3})} \approx -0.472
      \end{aligned}$$

      ```

3. We know from the weak duality theorem that $q^* \leq f^*$. Is the bound tight for this problem? That is, do we have $q^* = f^*$ in this problem?

   ```{dropdown} Solution
   Yes, we have $q^* = f^*$, so the bound is tight.
   ```

---

## Worksheet 12-1: Q2

Consider the problem

$$
\begin{aligned}
\min &\quad x_1^2 - x_2\\
\text{s.t.} &\quad x_2^2 \leq 0,
\end{aligned}
$$

1. Find the optimal solution by reducing the problem to a single variable unconstrained optimization problem without using duality.

   ```{dropdown} Solution
   The constraint $x_2^2 \leq 0$ implies that $x_2 = 0$.
   So, the problem reduces to $\min_{x_1} x_1^2$, which has optimal solution $x_1 = 0$ and optimal value $f^* = 0$.
   ```

2. Write down the Lagrangian.

   ```{dropdown} Solution
   The Lagrangian is

   $$L(x_1,x_2,\lambda) = x_1^2 - x_2 + \lambda x_2^2.$$

   ```

3. Obtain the dual objective function $q(\lambda)=\min_{x_1,x_2} L(x_1,x_2,\lambda)$.
   Note that you will need to consider the two cases: $\lambda=0$ and $\lambda>0$. This means your objective function will be a piece-wise function.

   ```{dropdown} Solution
   See [this desmos plot](https://www.desmos.com/3d/bdd0zni1m5) for a visual of everything.

   When $\lambda=0$, the Lagrangian is just the objective function,

   $$L(x_1,x_2,0) = x_1^2 -x_2.$$

   The gradient is

   $$\nabla_{x_1,x_2} L(x_1,x_2,0) = \begin{bmatrix} 2x_1\\ -1 \end{bmatrix}$$

   which is never zero.
   We also notice that the function goes to $-\infty$ as $x_2$ goes to $\infty$, so there is no finite minimum.
   So, we have $q(0) = -\infty$.

   For $\lambda > 0$, the gradient of the Lagrangian is

   $$\nabla_{x_1,x_2} L(x_1,x_2,\lambda) = \begin{bmatrix} 2x_1\\ -1 + 2\lambda x_2 \end{bmatrix}$$

   which is zero when $x_1=0$ and $x_2 = \frac{1}{2\lambda}$.
   The value of the Lagrangian at this point is

   $$L\left(0, \frac{1}{2\lambda}, \lambda\right) = -\frac{1}{2\lambda} + \lambda \left(\frac{1}{2\lambda}\right)^2 = -\frac{1}{4\lambda}.$$

   So, we have

   $$q(\lambda) = \begin{cases} -\infty & \text{if } \lambda = 0\\ -\frac{1}{4\lambda} & \text{if } \lambda > 0 \end{cases}.$$

   ```

4. What is $q^* = \max_{\lambda \geq 0} q(\lambda)$?

   ```{dropdown} Solution
   Since $q(\lambda) = -\frac{1}{4\lambda}$ for $\lambda > 0$, we have $q(\lambda) \to 0$ as $\lambda \to \infty$.
   So, we have $q^* = \sup_{\lambda \geq 0} q(\lambda) = 0$.
   ```

5. Is the bound from the weak duality theorem tight for this problem? That is, do we have $q^* = f^*$ in this problem?

   ```{dropdown} Solution
    Yes, we have $q^* = 0  = f^*$, so the bound is tight. 

   ```

---

## Worksheet 12-1: Q3

$$
\begin{aligned}
\text{(P)}\quad \min_{(x,y)} \; & x^2+y^2 \\
\text{s.t.} \; & 1 - x^4 \le 0
\end{aligned}
$$

1. Using the [desmos plot](https://www.desmos.com/3d/bbh7v1ykwn), what do you think the optimal solution $f^*$ is?

   ```{dropdown} Solution
   The minimum occurs at $x=\pm 1$ and $y=0$, and the optimal value is $f^* = 1$.

   You can also get this by noting that the constraint is $x^4 \ge 1$, so $|x|\ge 1$.
   ```

2. Now solve the problem using duality:

   a. Write down the Lagrangian.

      ```{dropdown} Solution
      The Lagrangian is

      $$L(x,y,\lambda) = x^2+y^2 + \lambda(1-x^4).$$

      ```

   c. In a different Desmos plot from above, plot the Lagrangian as a function of $x$ and $y$ using a slider bar for different values of $\lambda \geq 0$. What do you observe? What does this suggest about the dual function $q(\lambda) = \min_{x,y} L(x,y,\lambda)$? What is $q^* = \max_\lambda q(\lambda)$?

      *For an additional challenge, can you justify your observation without using Desmos?*

      ```{dropdown} Solution
      When $\lambda=0$, the Lagrangian is just the objective function, which is a bowl-shaped function with minimum at $x=0$ and $y=0$.

      When $\lambda > 0$, we can see that the function goes to $-\infty$ as $|x|$ goes to $\infty$, so there is no finite minimum.

      Since $q(\lambda) = \min_{x,y} L(x,y,\lambda)$, we have $q(0) = 0$ and $q(\lambda) = -\infty$ for $\lambda > 0$.
      This means that $q^* = \max_\lambda q(\lambda) = 0$.
      ```

3. Based on what you found above, is the bound from the weak duality theorem tight for this problem? That is, do we have $q^* = f^*$ in this problem?

   ```{dropdown} Solution
   No, we have $q^* = 0 < 1 = f^*$, so the bound is not tight.
   ```