Ch 5.1.1-2: Leave One Out Cross-validation

Lecture 12 - CMSE 381

Michigan State University
::
Dept of Computational Mathematics, Science /span> Engineering

Fri, 2/13, 2026

Announcements

Last time:

Exam

Announcements:

Exam 1 grades.... hopefully soon

Screenshot of the course schedule fromlectures 11 to 20.

Covered in this lecture

Validation Set
LOO CV

Section 1

Validation set

What’s the problem?

How well is my ML method doing? Model Assessment
Which method is best for our data?
How many features should I use? Which ones? Model selection
What is the uncertainty in the learned parameters?

Training Error vs Testing Error

Training Error

Testing Error

Throw-back

Plot on left is scatter plot for simulated close to linear data with three different fits of varying
flexibility. Plot on right shows the training and testing MSE as a function of flexibility

Model tradeoffs

Prediction error versus model complexity. One curve shows the training sample error, while another
curve shows the test sample error.

Validation set approach

A horizontal bar representing data points with random chunks colored as either red (training set) or
yellow (test set).

Divide randomly into two parts:
- Training set
- Validation/Hold-out/Testing set
Fit model on training set
Use fitted model to predict response for observations in the test set
Evaluate quality (e.g. MSE)

Example with the auto data

Predicting $𝚖𝚙𝚐$ using $𝚑𝚘𝚛𝚜𝚎𝚙𝚘𝚠𝚎𝚛$ :

𝚖𝚙𝚐 = β_{0} + β_{1} 𝚑𝚙 + β_{2} {𝚑𝚙}^{2} + \dots + β_{p} {𝚑𝚙}^{p}

Rinse and repeat

A horizontal bar representing data points with random chunks colored as either red (training set) or
yellow (test set).

Again example with auto data

Mean squared error versus degree of
polynomial. Multiple curves corresponding
to different validation sets are shown.

Coding example in jupyter notebook

Section 2

Leave-One-Out Cross-Validation (LOOCV)

The idea

A horizontal bar representing data points with a yellow square (test set) at the zeroth position of the
bar with the rest of the bar red (training set)

A horizontal bar representing data points
with a yellow square (test set) at the first position of the bar with the rest of the bar red
(training set)

A horizontal bar representing data points with a yellow square (test set) at
the second position of the bar with the rest of the bar red (training set)

A horizontal bar
representing data points with a yellow square (test set) at the third position of the bar with the
rest of the bar red (training set)

A horizontal bar representing data points with a yellow
square (test set) at the fourth position of the bar with the rest of the bar red (training set)

⋮

A
horizontal bar representing data points with a yellow square (test set) at last, rightmost position of the
bar with the rest of the bar red (training set)

The idea in mathy words

Remove $(x_{1}, y_{1})$ for testing.
Train the model on $n - 1$ points: ${(x_{2}, y_{2}), \dots, (x_{n}, y_{n})}$
Calculate ${MSE}_{1} = {(y_{1} - ŷ_{1})}^{2}$
Remove $(x_{2}, y_{2})$ for testing.
Train the model on $n - 1$ points: ${(x_{1}, y_{1}), (x_{3}, y_{3}), \dots, (x_{n}, y_{n})}$
Calculate ${MSE}_{2} = {(y_{2} - ŷ_{2})}^{2}$
Rinse and repeat

Return the score:

C V_{(n)} = \frac{1}{n} \sum_{i = 1}^{n} {MSE}_{i}

Again example with auto data

Mean squared error versus degree of
polynomial for LOOCV.

Do the LOOCV coding section

LOOCV Pros and Cons

Advantages:

Disadvantages:

TL;DR

Validation set A horizontal
bar representing data points
with random chunks colored
as either red (training set)
or yellow (test set).

LOO-CV

⋮

LOO-CV Score

C V_{(n)} = \frac{1}{n} \sum_{i = 1}^{n} {MSE}_{i}

Next time

Screenshot of the course schedule from lectures 11 to 20.