# Worksheet 10-1: Optimality Conditions for Linearly Constrained Problems (with Solutions)

Download: [CMSE382-WS10_1.pdf](CMSE382-WS10_1.pdf), [CMSE382-WS10_1-Soln.pdf](CMSE382-WS10_1-Soln.pdf)

```{warning}
This is an AI-generated transcript of the worksheet and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.
```

---

## Worksheet 10-1: Q1

Let $m, n = 2$, and the matrices

$$

\mathbf{A} =
\begin{bmatrix}
6 & 3 \\
4 & 0
\end{bmatrix},
\qquad
\mathbf{c} =
\begin{bmatrix}
3 \\
1
\end{bmatrix}.

$$

1. The first possibility from Farkas' lemma is that there exists $\mathbf{x} \in \mathbb{R}^2$ such that $\mathbf{A} \mathbf{x} \leq \mathbf{0}$ and $\mathbf{c}^T \mathbf{x} > 0$.

   a. Write down the system of inequalities implied by $\mathbf{A} \mathbf{x} \leq \mathbf{0}$. Use Desmos to sketch the region defined by these inequalities.

   ```{dropdown} Solution
   The assumption $\mathbf{A} \mathbf{x} \leq \mathbf{0}$ means
   

   $$

   \begin{align*}
   6x_1 + 3x_2 &\leq 0, \\
   4x_1 &\leq 0.
   \end{align*}

   $$

   ```

   b. Write down the inequality implied by $\mathbf{c}^T \mathbf{x} > 0$. Draw this restriction on the same Desmos plot.

   ```{dropdown} Solution
   The assumption $\mathbf{c}^T \mathbf{x} > 0$ means $3x_1 + x_2 > 0$.
   ```

   c. Does there exist $\mathbf{x} \in \mathbb{R}^2$ such that $\mathbf{A} \mathbf{x} \leq \mathbf{0}$ and $\mathbf{c}^T \mathbf{x} > 0$? Justify your answer using the Desmos plot.

   ```{dropdown} Solution
   No. See my Desmos plot [here](https://www.desmos.com/calculator/dckaytsowz).
   
   - The region defined by $\mathbf{A} \mathbf{x} \leq \mathbf{0}$ is the intersection of the blue and red regions in the plot.
   - The region defined by $\mathbf{c}^T \mathbf{x} > 0$ is the green region in the plot.
   - Since there is no overlap between the blue/red region and the green region, there is no $\mathbf{x}$ that satisfies both conditions.
   ```

2. The second possibility from Farkas' lemma is that there exists $\mathbf{y} \in \mathbb{R}^2$ such that $\mathbf{A}^\top\mathbf{y} =\mathbf{c}$ and $\mathbf{y}\geq 0$.

   a. Write down the system of equations implied by $\mathbf{A}^\top\mathbf{y} =\mathbf{c}$. Plot the solution of these equations in a new Desmos plot.

   ```{dropdown} Solution
   The assumption $\mathbf{A}^\top\mathbf{y} =\mathbf{c}$ means
   

   $$

   \begin{align*}
   6y_1 + 4y_2 &= 3, \\
   3y_1 + 0y_2 &= 1.
   \end{align*}

   $$

   ```

   b. Is there a solution $\mathbf{y} \geq 0$ to the equations above? Use your Desmos plot to justify.

   ```{dropdown} Solution
   The lines intersect at $\mathbf{y} = (1/3, 1/4)$, see my Desmos plot [here](https://www.desmos.com/calculator/0kscc3s4o8). Since both entries are positive, $\mathbf{y} \geq 0$. So there is a solution $\mathbf{y} \geq 0$ to the equations above.
   ```

3. Based on the previous parts, which of the two possibilities from Farkas' lemma holds for the matrices $\mathbf{A}$ and $\mathbf{c}$ above? Justify your answer.

   ```{dropdown} Solution
   The second possibility from Farkas' lemma holds for the matrices $\mathbf{A}$ and $\mathbf{c}$ above.
   
   - We showed in part (a) that there is no $\mathbf{x}$ such that $\mathbf{A} \mathbf{x} \leq \mathbf{0}$ and $\mathbf{c}^T \mathbf{x} > 0$. So the first possibility does not hold.
   - We showed in part (b) that there is a $\mathbf{y}$ such that $\mathbf{A}^\top\mathbf{y} =\mathbf{c}$ and $\mathbf{y}\geq 0$. So the second possibility does hold.
   ```

4. Repeat the previous part for the matrices

$$

\mathbf{A} =
\begin{bmatrix}
6 & 3 \\
4 & 0
\end{bmatrix},
\qquad
\mathbf{d} =
\begin{bmatrix}
1 \\
-1
\end{bmatrix}.

$$

Which of the two possibilities from Farkas' lemma holds for the matrices $\mathbf{A}$ and $\mathbf{d}$ above? Justify your answer.

```{dropdown} Solution
For the first option of Farkas' lemma:

- The inequalities from $\mathbf{A} \mathbf{x} \leq 0$ are the same:  

  $$

  \begin{align*}
  6x_1 + 3x_2 &\leq 0, \\
  4x_1 &\leq 0.
  \end{align*}

  $$

  They are the red and green regions of [this Desmos plot](https://www.desmos.com/calculator/4kursizf6c).
- The inequality from $\mathbf{d}^T \mathbf{x} > 0$ is $x_1 - x_2 > 0$, which is the green region in the plot above.
- There are lots of options for $\mathbf{x}$ that satisfy both conditions. For example, $\mathbf{x} = (-2, -4)$ satisfies both conditions. So the first option of Farkas' lemma holds for the matrices $\mathbf{A}$ and $\mathbf{d}$ above.

The first option of Farkas' lemma holds so we know the second doesn't. But just for the sake of practice, we can check.
- $\mathbf{A}^\top \mathbf{y} = \mathbf{d}$ means

  $$

  \begin{align*}
  6y_1 + 4y_2 &= 1, \\
  3y_1 + 0y_2 &= -1.
  \end{align*}

  $$

- See [this Desmos plot](https://www.desmos.com/calculator/gp49bdhbph).
- The lines intersect at $(-1/3, 3/4)$, which is not non-negative. So there is no $\mathbf{y} \geq 0$ such that $\mathbf{A}^\top \mathbf{y} = \mathbf{d}$.
```

---

## Worksheet 10-1: Q2

Find the stationary point(s) for

$$

\min \frac{1}{2}\left(x_1^2 + x_2^2 +x_3^2\right) \quad \text{s.t.} \quad x_1 + x_2 + x_3=3

$$

by following the steps below.

(a) Determine $f$, $A$, $\mathbf{b}$, $C$, and $\mathbf{d}$ to write the problem in standard form:

$$

\min_{\mathbf{x}} f(\mathbf{x}) \quad \text{s.t.} \quad A \mathbf{x} \leq \mathbf{b},\; C\mathbf{x} = \mathbf{d}.

$$

```{dropdown} Solution
- $f(\mathbf{x}) = \frac{1}{2} \left(x_1^2 + x_2^2 +x_3^2\right)$
- $A = \mathbf{0}$, $\mathbf{b} = \mathbf{0}$ (since there are no inequality constraints)
- $C = \begin{bmatrix}1 & 1 & 1 \end{bmatrix}$, $\mathbf{d} = 3$ (since the equality constraint is $x_1 + x_2 + x_3 = 3$)
```

(b) Write down the Lagrangian function.

```{dropdown} Solution
The Lagrangian function is

$$

L(\mathbf{x},\mu) = f(\mathbf{x}) + \boldsymbol{\mu}^{\top} (C\mathbf{x} - \mathbf{d}) = \frac{1}{2} \left(x_1^2 + x_2^2 +x_3^2\right) + \mu (x_1 + x_2 + x_3 - 3).

$$

(Since $A = \mathbf{0}$ and $\mathbf{b} = \mathbf{0}$, we can drop the inequality constraint terms.)
```

(c) Write down the KKT condition (also called the stationarity condition).

```{dropdown} Solution
The KKT condition is

$$

\nabla_\mathbf{x} L(\mathbf{x}, \mu) = \nabla f(\mathbf{x}) + C^\top \boldsymbol{\mu} = \mathbf{0}.

$$

We can compute $\nabla f(\mathbf{x}) = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}$ and $C^\top \boldsymbol{\mu} = \begin{bmatrix} 1 \\  1 \\ 1 \end{bmatrix} \mu$. So the stationarity condition is

$$

\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} + \begin{bmatrix} 1 \\  1 \\ 1 \end{bmatrix} \mu 
= \begin{bmatrix}
x_1 + \mu \\ x_2 + \mu \\ x_3 + \mu
\end{bmatrix}
= \mathbf{0}.

$$

This can be solved as $x_1 = -\mu$, $x_2 = -\mu$, and $x_3 = -\mu$.
```

(d) Write down the feasibility constraints.

```{dropdown} Solution
The feasibility condition is $C\mathbf{x} = \mathbf{d}$, which here is

$$

x_1 + x_2 + x_3 = 3.

$$

(We don't need $A \mathbf{x} \leq \mathbf{b}$ or $\boldsymbol{\lambda} \geq \mathbf{0}$ since there are no inequality constraints.)
```

(e) Solve for the stationary point(s) by solving the stationarity and feasibility constraints together.

```{dropdown} Solution
- From the stationarity condition, we have $x_1 = -\mu$, $x_2 = -\mu$, and $x_3 = -\mu$.
- Plugging into $x_1+x_2+x_3 = 3$, this means $-3\mu = 3$, which implies $\mu = -1$.
- So the stationary point is $\mathbf{x} = (1, 1, 1)$.
```

(f) Is the stationary point(s) optimal? Justify your answer.

```{dropdown} Solution
The problem is convex since $f$ is a convex function and the constraints are linear. So the stationary point is a global optimal solution of the problem.
```

---

## Worksheet 10-1: Q3

Consider the problem

$$

\begin{aligned}
& \text{minimize} & & x_1^2 + 2x_2^2 + 4x_1x_2 \\
& \text{subject to} & & x_1 + x_2 = 1, \\
& & & x_1, x_2 \geq 0.
\end{aligned}

$$

(a) Is this problem convex? Justify your answer.

```{dropdown} Solution
The problem is not convex because the objective function is not convex. To see this, we can compute the Hessian of the objective function:

$$

H = \begin{bmatrix}2 & 4 \\ 
4 & 4
\end{bmatrix}.

$$

The eigenvalues of $H$ are $-2$ and $8$, so $H$ is not positive semidefinite. Therefore, the objective function is not convex, and the problem is not convex.
```

(b) Recall the Generalized Extreme Value Theorem (GEVT): If $f:U \to \mathbb{R}$ is a continuous function and $U \subseteq \mathbb{R}^n$ is compact, then $f$ is bounded and there exist $\mathbf{x}^*, \mathbf{x}_* \in U$ such that

$$

f(\mathbf{x}^*) = \sup_{\mathbf{x} \in U} f(\mathbf{x})
\quad \text{and} \quad
f(\mathbf{x}_*) = \inf_{\mathbf{x} \in U} f(\mathbf{x}).

$$

Use the GEVT to argue that there exists an optimal solution to the problem above.

```{dropdown} Solution
- The objective function is continuous since it is a polynomial.
- The feasible set is compact since it is closed and bounded. The feasible set is closed since it is the intersection of the closed sets $\{(x_1, x_2) : x_1 + x_2 = 1\}$, $\{(x_1, x_2) : x_1 \geq 0\}$, and $\{(x_1, x_2) : x_2 \geq 0\}$. The feasible set is bounded since $x_1 + x_2 = 1$ implies $x_1 \leq 1$ and $x_2 \leq 1$.
- Since the objective function is continuous and the feasible set is compact, the GEVT implies that there exists an optimal solution to the problem above.
```

(c) Find the Lagrangian.

```{dropdown} Solution
Following the standard notation:

- $f(x_1, x_2) = x_1^2 + 2x_2^2 + 4x_1x_2$
- $A = \begin{bmatrix} -1 & 0 \\ 0 & -1 \end{bmatrix}$, $\mathbf{b} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}$ (since the inequality constraints are $-x_1 \leq 0$ and $-x_2 \leq 0$)
- $C = \begin{bmatrix} 1 & 1 \end{bmatrix}$, $\mathbf{d} = 1$ (since the equality constraint is $x_1 + x_2 = 1$)

So the Lagrangian is

$$

L(x_1, x_2, \lambda_1, \lambda_2, \mu) = x_1^2 + 2x_2^2 + 4x_1x_2 + \lambda_1 (-x_1) + \lambda_2 (-x_2) + \mu (x_1 + x_2 - 1).

$$

```

(d) Write down the stationarity KKT condition.

```{dropdown} Solution
The stationarity condition is

$$

\nabla_\mathbf{x} L(\mathbf{x}, \boldsymbol{\lambda}, \mu) = \nabla f(\mathbf{x}) + A^\top \boldsymbol{\lambda} + C^\top \mu
= 
\begin{bmatrix}
2x_1 + 4x_2 \\
4x_1 + 4x_2
\end{bmatrix} +
\begin{bmatrix}-1 & 0 \\
0 & -1
\end{bmatrix}^\top
\begin{bmatrix}\lambda_1 \\
\lambda_2
\end{bmatrix} +
\begin{bmatrix} 1 \\
1
\end{bmatrix} \mu
=
\mathbf{0}.

$$

With matrix multiplication, this is equivalent to the system of equations

$$

\begin{align*}
2x_1 + 4x_2 - \lambda_1 + \mu &= 0, \\
4x_1 + 4x_2 - \lambda_2 + \mu &= 0.
\end{align*}

$$

```

(e) Write down the complementary slackness conditions.

```{dropdown} Solution
The complementary slackness conditions are $\lambda_i (\mathbf{a}_i^\top \mathbf{x}^* - b_i) = 0$ for each inequality constraint. Here, they are

$$

\begin{align*}
\lambda_1 (-x_1) &= 0, \\
\lambda_2 (-x_2) &= 0.
\end{align*}

$$

This can be simplified to $\lambda_1 x_1 = 0$ and $\lambda_2 x_2 = 0$.
```

(f) Write down the feasibility conditions from the problem constraints.

```{dropdown} Solution
- $x_1+x_2 = 1$
- $x_1 \geq 0$
- $x_2 \geq 0$
```

(g) Write down the feasibility conditions for each of the $\lambda$s.

```{dropdown} Solution
- $\lambda_1 \geq 0$
- $\lambda_2 \geq 0$
```

(h) From everything above, copy down the 9 total equations/inequalities to be satisfied by an optimal solution.

```{dropdown} Solution

$$

\begin{align*}
2x_1 + 4x_2 - \lambda_1 + \mu &= 0, \\
4x_1 + 4x_2 - \lambda_2 + \mu &= 0, \\
\lambda_1 x_1 &= 0, \\
\lambda_2 x_2 &= 0, \\
x_1 + x_2 &= 1, \\
x_1 &\geq 0, \\
x_2 &\geq 0, \\
\lambda_1 &\geq 0, \\
\lambda_2 &\geq 0.
\end{align*}

$$

```

(i) This problem involves multiple cases for $\lambda_1$ and $\lambda_2$. We will address each separately.

**Case 1:** $\lambda_1=\lambda_2= 0$. Use the complementary slackness conditions to solve for $x_1$ and $x_2$. Is the solution consistent with the constraints?

```{dropdown} Solution
- If $\lambda_1 = \lambda_2 = 0$, then the equations simplify to

  $$

  \begin{align*}
  2x_1 + 4x_2 + \mu &= 0, \\
  4x_1 + 4x_2 + \mu &= 0, \\
  x_1 + x_2 &= 1.
  \end{align*}

  $$

- Solving this system gives $x_1 = 0$, $x_2 = 1$, $\mu = -4$.
- All constraints are satisfied: $x_1 \geq 0$, $x_2 \geq 0$, $\lambda_1 \geq 0$, $\lambda_2 \geq 0$.
- So $(x_1,x_2) = (0, 1)$ is a KKT point.
```

**Case 2:** $\lambda_1, \lambda_2 > 0$. Use the complementary slackness conditions to solve for $x_1$ and $x_2$. Is the solution consistent with the constraints?

```{dropdown} Solution
- If $\lambda_1, \lambda_2 > 0$, then the slackness conditions $\lambda_1 x_1 = 0$ and $\lambda_2 x_2 = 0$ imply $x_1 = 0$ and $x_2 = 0$.
- But this contradicts the equality constraint $x_1 + x_2 = 1$.
- So there is no KKT point arising from $\lambda_1, \lambda_2 > 0$.
```

**Case 3:** $\lambda_1>0, \lambda_2 = 0$. Use the complementary slackness conditions to solve for $x_1$ and $x_2$. Is the solution consistent with the constraints?

```{dropdown} Solution
- If $\lambda_1 > 0$ and $\lambda_2 = 0$, then the slackness conditions imply $x_1 = 0$.
- From the equality constraint $x_1 + x_2 = 1$, we get $x_2 = 1$.
- All constraints are satisfied.
- So $(x_1,x_2) = (0, 1)$ is a KKT point. However, this is the same point found in Case 1, not a new point.
```

**Case 4:** $\lambda_1=0, \lambda_2 > 0$. Use the complementary slackness conditions to solve for $x_1$ and $x_2$. Is the solution consistent with the constraints?

```{dropdown} Solution
- If $\lambda_1 = 0$ and $\lambda_2 > 0$, then the slackness conditions imply $x_2 = 0$.
- From the equality constraint $x_1 + x_2 = 1$, we get $x_1 = 1$.
- All constraints are satisfied.
- So $(x_1,x_2) = (1, 0)$ is a KKT point.
```

(j) Write down the KKT points you found above (there should be two of them).

```{dropdown} Solution
The KKT points are $(x_1,x_2) = (0, 1)$ and $(x_1,x_2) = (1, 0)$.
```

(k) Can you use the KKT theorem to determine which of the points you found above is a local optimal solution of the problem? Justify your answer.

```{dropdown} Solution
No, since the problem is not convex, the KKT conditions are not sufficient for optimality. So we cannot use the KKT theorem to determine which of the points is a local optimal solution.
```

(l) Which of the two points is the global optimal solution of the problem? Justify your answer.

```{dropdown} Solution
- Even though we can't use the KKT condition theorem, the GEVT guarantees that there is a global optimal solution.
- From the KKT conditions, any local optimal solution must be a KKT point. So the global optimal must be one of our KKT points.
- Evaluate the objective function at each KKT point:
  - At $(x_1,x_2) = (0, 1)$: $f = 0^2 + 2(1)^2 + 4(0)(1) = 2$
  - At $(x_1,x_2) = (1, 0)$: $f = 1^2 + 2(0)^2 + 4(1)(0) = 1$
- Since we are minimizing, the global optimal solution is $(x_1,x_2) = (1, 0)$ with objective value $1$.
```