# Worksheet 11-1: The KKT Conditions (with Solutions)

Download: [CMSE382-WS11_1.pdf](CMSE382-WS11_1.pdf), [CMSE382-WS11_1-Soln.pdf](CMSE382-WS11_1-Soln.pdf)

```{warning}
This is an AI-generated transcript of the worksheet and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.
```

---

## Worksheet 11-1: Q1

Consider the problem

$$
\begin{aligned}
\min &\quad x_1^2 - 2x_2\\
\text{s.t.} &\quad x_1^2 + x_2^2  =4
\end{aligned}
$$

1. On [this Desmos plot](https://www.desmos.com/3d/mwt36nkney), what are the purple surface, and the orange and green curves? Visually on the plot, what should the answer to the optimization problem be?

```{dropdown} Solution
The purple surface is the graph of the objective function $f(x_1,x_2)=x_1^2-2x_2$. The orange circle is the constraint set where $x_1^2+x_2^2=4$.
The green curve is the set of possible function values the function can take on the constraint set.

We are trying to minimize the possible function values, so we are looking for the $(x_1,x_2)$ on the orange circle that gives the lowest point on the green curve. Visually, this looks to be around $x_1=0$ and $x_2=2$.
```

2. If you couldn't see the plot, how would you justify that the problem has an optimal solution?

```{dropdown} Solution
The objective function $f(x_1,x_2)=x_1^2-2x_2$ is continuous. The set defined by the constraint is a circle of radius $2$, so not empty, closed, and bounded (so compact). By the GEVT, a minimizer exists.
```

3. What are the regular points of the problem? Use this to conclude that the optimal solution will be one of the KKT points.

```{dropdown} Solution
The constraint function is $g(x_1,x_2)=x_1^2+x_2^2-4$. The gradient of the constraint function is $\nabla g(x_1,x_2)=(2x_1,2x_2)$.

With only one vector, linear independence is equivalent to the vector being nonzero.
So, the regular points are the points where $\nabla g(x_1,x_2)\neq 0$, which is everywhere except at the origin $(0,0)$.
Since the origin isn't part of the set $x_1^2+x_2^2 = 4$ anyway, there are no irregular points of the problem.
So, the regular points are all of the points on the circle defined by $x_1^2+x_2^2=4$.
```

4. Write down the KKT constraint equations.

```{dropdown} Solution
The Lagrangian is

$$
L(x_1,x_2,\mu)=x_1^2-2x_2 + \mu(x_1^2+x_2^2-4)
$$

so the stationarity condition gives

$$
\begin{aligned}
2x_1 + 2\mu x_1 &=0\\
-2 + 2\mu x_2 &=0
\end{aligned}
$$

The additional constraint is directly from the problem statement:

$$
x_1^2+x_2^2=4
$$

So, there are 3 equations in total for the KKT system.
```

5. Find the KKT points. *Hint: There are four of them*.

```{dropdown} Solution
- Note that $\mu \neq 0$ since the second stationarity condition gives $\mu=\frac{1}{x_2}$, and $x_2$ cannot be zero since the first constraint would not be satisfied. So, we can assume $\mu \neq 0$.
- Then the second stationarity condition gives $-2 + 2\mu x_2=0$, which can be rearranged to $x_2 = \frac{1}{\mu}$ (allowed since $\mu \neq 0$).
- From the first stationarity condition, we have $2x_1(1+\mu)=0$, which can be rearranged to $x_1=0$ or $\mu=-1$.
- If $x_1 = 0$, the requirement $x_1^2+x_2^2=4$ gives $x_2^2=4$, so $x_2=\pm 2$. So, we have two KKT points from this case: $(0,2)$ and $(0,-2)$.
  We can actually also figure out $\mu$ although it doesn't help, but just for completeness, substituting into the second stationarity condition gives $\mu=\frac{1}{x_2}$, so $\mu=\frac{1}{2}$ or $\mu=-\frac{1}{2}$.
- If $\mu = -1$, the second stationarity condition gives $-2 + 2(-1)x_2=0$, which can be rearranged to $x_2=-1$.
  Substituting into the circle constraint gives $x_1^2+(-1)^2=4$, which can be rearranged to $x_1^2=3$. So, we have two KKT points from this case: $(\sqrt{3},-1)$ and $(-\sqrt{3},-1)$.
```

6. What are the function values at the KKT points? Which one is the optimal solution? Plot the two KKT points on the Desmos plot to verify your answer.

```{dropdown} Solution
- At $(0,2)$, the function value is $f(0,2)=0^2-2(2)=-4$.
- At $(0,-2)$, the function value is $f(0,-2)=0^2-2(-2)=4$.
- At $(\sqrt{3},-1)$, the function value is $f(\sqrt{3},-1)=(\sqrt{3})^2-2(-1)=3+2=5$.
- At $(-\sqrt{3},-1)$, the function value is $f(-\sqrt{3},-1)=(-\sqrt{3})^2-2(-1)=3+2=5$.
- So, the optimal solution is $(0,2)$ with function value $-4$.
- See [this Desmos plot with the KKT points added](https://www.desmos.com/3d/qamotztqk1).
```

---

## Worksheet 11-1: Q2

Consider the problem

$$
\begin{aligned}
\min &\quad 2 x_1 + 3x_2 - x_3\\
\text{s.t.} &\quad x_1^2 + x_2^2 +x_3^2 = 1\\
            & \quad x_1^2 + 2 x_2^2 + 2x_3^2 = 2.
\end{aligned}
$$

1. Do you expect the problem to have an optimal solution? Explain.

```{dropdown} Solution
$f(x)=2x_1+3x_2-x_3$ is continuous. The set defined by the constraint is a circle of radius $1$, so not empty, closed, and bounded (so compact). By the GEVT, a minimizer exists.
```

2. Write down all the KKT constraint equations.

```{dropdown} Solution
The Lagrangian is

$$
L(x_1,x_2,x_3,\mu_1,\mu_2)= x_1 + 3x_2 - x_3 + \mu_1(x_1^2 + x_2^2 +x_3^2-1) + \mu_2(x_1^2 + 2 x_2^2 + 2x_3^2-2)
$$

so the stationarity condition gives

$$
\begin{aligned}
2+2(\mu_1+\mu_2)x_1 &=0\\
3+2(\mu_1+2\mu_2)x_2 &= 0\\
-1 +2(\mu_1+2\mu_2)x_3 &=0
\end{aligned}
$$

The additional constraints are directly from the problem statement:

$$
\begin{aligned}
x_1^2+x_2^2+x_3^2 &=1\\
x_1^2+2x_2^2+2x_3^2 &=2
\end{aligned}
$$

So, there are 5 equations in total for the KKT system.
```

3. Use your stationarity conditions to check that $\mu_1 + \mu_2 \neq 0$ and $\mu_1 + 2\mu_2\neq 0$.

```{dropdown} Solution
If $\mu_1 + \mu_2 = 0$, then the first stationarity condition gives $2=0$, which is a contradiction. So, $\mu_1 + \mu_2 \neq 0$.
Similarly, if $\mu_1 + 2\mu_2 = 0$, then the second stationarity condition gives $3=0$, which is a contradiction. So, $\mu_1 + 2\mu_2 \neq 0$.
```

4. Write $a = \frac{1}{\mu_1 + \mu_2}$ and $b = \frac{1}{\mu_1 + 2\mu_2}$.
Use the stationarity conditions to express any KKT point $\mathbf{x} = (x_1,x_2,x_3)$ in terms of $a$ and $b$.

```{dropdown} Solution
- From the first stationarity condition, we have $2+2(\mu_1+\mu_2)x_1=0$, which can be rearranged to $x_1=-\frac{1}{\mu_1 + \mu_2}=-a$.
- From the second stationarity condition, we have $3+2(\mu_1+2\mu_2)x_2=0$, which can be rearranged to $x_2=-\frac{3}{2(\mu_1 + 2\mu_2)}=-\frac{3}{2}b$.
- From the third stationarity condition, we have $-1 +2(\mu_1+2\mu_2)x_3=0$, which can be rearranged to $x_3=\frac{1}{2(\mu_1 + 2\mu_2)}=\frac{1}{2}b$.
- The resulting point is $\mathbf{x}=(-a,-\frac{3}{2}b,\frac{1}{2}b)$.
```

5. Plug this point into the two constraint equations. Can any KKT point satisfy the constraints?

```{dropdown} Solution
- The first constraint says $a^2+10b^2 =1$.
- The second says $a^2+20b^2=2$.
- Subtracting the twice the first from the second gives $-a^2 = 0$, which can only be satisfied if $a=0$.
- However, we defined $a = \frac{1}{\mu_1 + \mu_2}$, so $a=0$ is not possible since $\mu_1 + \mu_2 \neq 0$ from part (c).
- So, there are no KKT points that satisfy the constraints.
```

From part (a), you hopefully came to the conclusion that an optimal solution must exist. From part (e), we know there are no KKT points. This means the optimal solution must be an irregular point.

Here we have two constraints $h_1$ and $h_2$, so we have two gradient vectors.
Two vectors are linearly dependent if one is a scalar multiple of the other, that is, if

$$
\nabla h_1(x_1,x_2,x_3) = \alpha \nabla h_2(x_1,x_2,x_3).
$$

for some $\alpha \neq 0$.

There are two cases in this problem where this can happen.

6. If $x_1 = 0$, check that the two gradients are linearly dependent. What possible values of $x_2$ and $x_3$ can we have in this case?

```{dropdown} Solution
- Since $x_1=0$, the first gradient is $\nabla h_1(0,x_2,x_3)=(0,2x_2,2x_3)$ and the second gradient is $\nabla h_2(0,x_2,x_3)=(0,4x_2,4x_3)$.
- We can see that $\nabla h_2(0,x_2,x_3)=2\nabla h_1(0,x_2,x_3)$, so the two gradients are linearly dependent for all $x_2$ and $x_3$.
- This means that any point $(0,x_2,x_3)$ is an irregular point of the problem.
```

7. Rewrite the original problem replacing $x_1=0$ which should now give you a problem in two variables with one constraint.

```{dropdown} Solution
The original problem becomes

$$
\begin{aligned}
(Q) \qquad \min &\quad 3x_2 - x_3\\
\text{s.t.} &\quad x_2^2 +x_3^2 = 1
\end{aligned}
$$

The second constraint is redundant since it is just twice the first constraint, so we didn't write it.
```

8. Find the solution for the optimization problem you set up in the last question where we assumed $x_1 = 0$, and use this to give an irregular point to be checked for the solution of the original problem.

```{dropdown} Solution
- The Lagrangian for $(Q)$ is $L(x_2,x_3,\mu)=3x_2 - x_3 + \mu(x_2^2 + x_3^2 -1)$, so the stationarity conditions give

$$
\begin{aligned}
3 + 2\mu x_2 &=0\\
-1 + 2\mu x_3 &=0
\end{aligned}
$$

- From the first stationarity condition, we have $3 + 2\mu x_2=0$, which can be rearranged to $x_2=-\frac{3}{2\mu}$.
- From the second stationarity condition, we have $-1 + 2\mu x_3=0$, which can be rearranged to $x_3=\frac{1}{2\mu}$.
- Substituting into the first constraint gives $\left(-\frac{3}{2\mu}\right)^2 + \left(\frac{1}{2\mu}\right)^2=1$, which can be rearranged to $\frac{10}{4\mu^2}=1$, so $\mu = \pm \frac{\sqrt{10}}{2}$.
- This means we have two KKT points for $(Q)$: $\left(-\frac{3}{\sqrt{10}},\frac{1}{\sqrt{10}}\right)$ and $\left(\frac{3}{\sqrt{10}},-\frac{1}{\sqrt{10}}\right)$.
- Evaluating the objective function at these two points gives $3\left(-\frac{3}{\sqrt{10}}\right) - \left(\frac{1}{\sqrt{10}}\right)=-\frac{10}{\sqrt{10}}=-\sqrt{10}$ and $3\left(\frac{3}{\sqrt{10}}\right) - \left(-\frac{1}{\sqrt{10}}\right)=\frac{10}{\sqrt{10}}=\sqrt{10}$, so the optimal solution to $(Q)$ is $\left(-\frac{3}{\sqrt{10}},\frac{1}{\sqrt{10}}\right)$ with optimal value $-\sqrt{10}$.
- Going back to the original problem, we got here with $x_1= 0 $, so the irregular point we want to check is $\left(0,-\frac{3}{\sqrt{10}},\frac{1}{\sqrt{10}}\right)$.
```

9. The other case where there can be linear dependence is if $x_2 = x_3 = 0$. Check that the two gradients of the two constraint functions are linearly dependent in this case and give all possible irregular points of this form.

```{dropdown} Solution
- Since $x_2=x_3=0$, the first gradient is $\nabla h_1(x_1,0,0)=(2x_1,0,0)$ and the second gradient is $\nabla h_2(x_1,0,0)=(2x_1,0,0)$.
- We can see that $\nabla h_2(x_1,0,0)=\nabla h_1(x_1,0,0)$, so the two gradients are linearly dependent for all $x_1$.
- This means that any point $(x_1,0,0)$ is an irregular point of the problem.
```

10. Plug these possible points into the original constraints. Are there any irregular points of this form that satisfy the constraints?

```{dropdown} Solution
- The first constraint says $x_1^2=1$, so $x_1=\pm 1$.
- The second constraint says $x_1^2=2$, which cannot be satisfied for $x_1 = \pm 1$.
- So, there are no irregular points of the form $(x_1,0,0)$ that satisfy the constraints.
```

11. Putting all of this together, what is the solution to the original problem?

```{dropdown} Solution
The only candidate solution we have is the irregular point $\left(0,-\frac{3}{\sqrt{10}},\frac{1}{\sqrt{10}}\right)$, so this must be the optimal solution to the original problem. It has value $2(0)+3\left(-\frac{3}{\sqrt{10}}\right)-\left(\frac{1}{\sqrt{10}}\right)=-\sqrt{10}$.
```