Worksheet 5-1: Newton’s Method (with Solutions)

Worksheet 5-1: Newton’s Method (with Solutions)#

Download: CMSE382-WS5_1.pdf · Solution PDF

Warning

This is an AI-generated transcript of the worksheet and may contain errors or inaccuracies. Please refer to the original course materials for authoritative content.

Worksheet 5-1: Q1#

Consider the function \(f(x,y) = 100x^4 + 0.01y^4\) and starting point \(\mathbf{x}_0 = (1, 1)^\top\).

  1. Compute the gradient \(\nabla f(\mathbf{x}_0)\).

    Solution
    \[\begin{split}\nabla f(x,y) = \begin{bmatrix} 400x^3 \\ 0.04y^3 \end{bmatrix}, \qquad \nabla f(\mathbf{x}_0) = \begin{bmatrix} 400 \\ 0.04 \end{bmatrix}\end{split}\]

  2. Compute the Hessian \(\nabla^2 f(\mathbf{x}_0)\).

    Solution
    \[\begin{split}\nabla^2 f(x,y) = \begin{bmatrix} 1200x^2 & 0 \\ 0 & 0.12y^2 \end{bmatrix}, \qquad \nabla^2 f(\mathbf{x}_0) = \begin{bmatrix} 1200 & 0 \\ 0 & 0.12 \end{bmatrix}\end{split}\]

  3. Find the Newton direction \(\mathbf{d}_0 = -[\nabla^2 f(\mathbf{x}_0)]^{-1} \nabla f(\mathbf{x}_0)\).

    Solution
    \[\begin{split}[\nabla^2 f(\mathbf{x}_0)]^{-1} = \begin{bmatrix} 1/1200 & 0 \\ 0 & 1/0.12 \end{bmatrix}\end{split}\]
    \[\begin{split}\mathbf{d}_0 = -\begin{bmatrix} 1/1200 & 0 \\ 0 & 1/0.12 \end{bmatrix}\begin{bmatrix} 400 \\ 0.04 \end{bmatrix} = -\begin{bmatrix} 400/1200 \\ 0.04/0.12 \end{bmatrix} = -\begin{bmatrix} 1/3 \\ 1/3 \end{bmatrix}\end{split}\]

  4. Compute one step of Newton’s method: \(\mathbf{x}_1 = \mathbf{x}_0 + \mathbf{d}_0\).

    Solution
    \[\begin{split}\mathbf{x}_1 = \begin{bmatrix} 1 \\ 1 \end{bmatrix} - \begin{bmatrix} 1/3 \\ 1/3 \end{bmatrix} = \begin{bmatrix} 2/3 \\ 2/3 \end{bmatrix}\end{split}\]

    Note: Both components are reduced by the same fraction, but the function is highly imbalanced (\(100x^4\) vs \(0.01y^4\)), so Newton’s method automatically compensates via the Hessian scaling.

Worksheet 5-1: Q2#

Consider the scalar function \(f(x) = \sqrt{1 + x^2}\).

Its derivatives are: $\(f'(x) = \frac{x}{\sqrt{1+x^2}}, \qquad f''(x) = \frac{1}{(1+x^2)^{3/2}}\)$

  1. Write the Newton update formula: \(x_{k+1} = x_k - \dfrac{f'(x_k)}{f''(x_k)}\).

    Solution
    \[x_{k+1} = x_k - \frac{f'(x_k)}{f''(x_k)} = x_k - \frac{\dfrac{x_k}{\sqrt{1+x_k^2}}}{\dfrac{1}{(1+x_k^2)^{3/2}}}\]

  2. Simplify to show that \(x_{k+1} = -x_k^3\).

    Solution
    \[x_{k+1} = x_k - x_k \cdot \frac{(1+x_k^2)^{3/2}}{\sqrt{1+x_k^2}} = x_k - x_k(1+x_k^2) = x_k - x_k - x_k^3 = -x_k^3\]

  3. For which starting values \(x_0\) does Newton’s method converge to \(x^* = 0\)?

    Solution

    The update is \(x_{k+1} = -x_k^3\). For convergence to \(0\), we need \(|x_{k+1}| < |x_k|\): $\(|-x_k^3| < |x_k| \iff |x_k|^3 < |x_k| \iff |x_k|^2 < 1 \iff |x_0| < 1\)$

    Newton’s method converges to \(x^* = 0\) for \(x_0 \in (-1, 1)\), \(x_0 \ne 0\).

  4. For which starting values does it diverge?

    Solution

    For \(|x_0| > 1\): \(|x_{k+1}| = |x_k|^3 > |x_k|\), so the iterates grow without bound → diverges.

    For \(|x_0| = 1\): \(x_1 = -1\) or \(x_1 = 1\), oscillating between \(\pm 1\) → does not converge.

Contents