Ch 5.1.1-3: Leave one out and $k$ -Fold Cross-Validation

Lecture 13 - CMSE 381

Michigan State University
::
Dept of Computational Mathematics, Science /span> Engineering

Mon, Feb 16, 2026

Announcements

Last time:

Exam

Announcements:

Exam 1 grades ....hopefully soon

Screenshot of the course schedule forlectures 11 to 20.

Covered in this lecture

Validation Set
LOO CV
K-fold CV

Section 1

Validation set

What’s the problem?

How well is my ML method doing? Model Assessment
Which method is best for our data?
How many features should I use? Which ones? Model selection
What is the uncertainty in the learned parameters?

Training Error vs Testing Error

Training Error

Testing Error

Throw-back

Throwback slide showing a figure labeled Fig2-10 from earlier lecture material.

Model tradeoffs

Diagram showing tradeoffs among different statistical models.

Validation set approach

Diagram showing how data are split in the validation set approach.

Divide randomly into two parts:
- Training set
- Validation/Hold-out/Testing set
Fit model on training set
Use fitted model to predict response for observations in the test set
Evaluate quality (e.g. MSE)

Example with the auto data

Diagram showing one split of the data
into a training set and a validation set.

Predicting $𝚖𝚙𝚐$ using $𝚑𝚘𝚛𝚜𝚎𝚙𝚘𝚠𝚎𝚛$ :

𝚖𝚙𝚐 = β_{0} + β_{1} 𝚑𝚙 + β_{2} {𝚑𝚙}^{2} + \dots + β_{p} {𝚑𝚙}^{p}

Rinse and repeat

Diagram showing the first data split in the validation set approach. Diagram showing the second data
split in the validation set approach. Diagram showing the third data split in the validation set
approach. Diagram showing the fourth data split in the validation set approach. Diagram showing the
fifth data split in the validation set approach.

Again example with auto data

Diagram showing a different training and
validation split, illustrating that the
validation set approach can produce
different results across splits.

Coding example in jupyter notebook

Section 2

Leave-One-Out Cross-Validation (LOOCV)

The idea

Diagram showing the first leave-one-out cross-validation split, where one observation is
held out for validation and the rest are used for training.

Diagram showing the second
leave-one-out cross-validation split, where a different observation is held out for validation.

Diagram showing the third leave-one-out cross-validation split, where a different observation is
held out for validation.

Diagram showing the fourth leave-one-out cross-validation split,
where a different observation is held out for validation.

Diagram showing the fifth
leave-one-out cross-validation split, where a different observation is held out for validation.

⋮

Diagram showing the final leave-one-out cross-validation split, where the last observation is held out
for validation.

The idea in mathy words

Remove $(x_{1}, y_{1})$ for testing.
Train the model on $n - 1$ points: ${(x_{2}, y_{2}), \dots, (x_{n}, y_{n})}$
Calculate ${MSE}_{1} = {(y_{1} - ŷ_{1})}^{2}$
Remove $(x_{2}, y_{2})$ for testing.
Train the model on $n - 1$ points: ${(x_{1}, y_{1}), (x_{3}, y_{3}), \dots, (x_{n}, y_{n})}$
Calculate ${MSE}_{2} = {(y_{2} - ŷ_{2})}^{2}$
Rinse and repeat

Return the score:

C V_{(n)} = \frac{1}{n} \sum_{i = 1}^{n} {MSE}_{i}

Again example with auto data

Figure using the auto data example to
show a fixed result with no randomness
across repeated runs.

Do the LOOCV coding section

LOOCV Pros and Cons

Advantages:

Disadvantages:

TL;DR

Validation set Diagram
showing the first validation
set split.

Diagram showing
the second validation set
split.

Diagram showing the
third validation set split.

Diagram showing the fourth
validation set split.

Diagram
showing the fifth validation
set split.

LOO-CV

Diagram showing
the first leave-one-out
cross-validation split.

Diagram showing the
second leave-one-out
cross-validation split.

Diagram showing the third
leave-one-out
cross-validation split.

⋮

Diagram showing the final
leave-one-out
cross-validation split.

LOO-CV Score

C V_{(n)} = \frac{1}{n} \sum_{i = 1}^{n} {MSE}_{i}

Section 3

k

-Fold CV

The idea

Mathy version

Randomly split data into $k$ -groups (folds)
Approximately equal sized. For the sake of notation, say each set has $ℓ$ points
Remove $i$ th fold $U_{i}$ and reserve for testing.
Train the model on remaining points
Calculate ${MSE}_{i} = \frac{1}{ℓ} \sum_{(x_{j}, y_{j}) \in U_{i}} {(y_{j} - ŷ_{j})}^{2}$
Rinse and repeat

Return

C V_{(k)} = \frac{1}{k} \sum_{i = 1}^{k} {MSE}_{i}

By hand first!

There are 10 students in the class, and we have data points for each. They have already been randomly permuted below. Write down the training/testing sets for a

3

-fold CV

Damien
Alice
Greta
Jasmin
Benji
Inigo
Frank
Carina
Enrique
Hubert

Fold 1

Fold 2

Fold 3

Coding - Building

k

-fold CV

Pros and Cons

Pros:

Cons:

Comparison

Figure from the auto data
example.

Validation set
Figure labeled validation set
for comparison.

Next time

Screenshot of the course schedule for lectures 11 to 20.

Ch 5.1.1-3: Leave one out and k-Fold Cross-Validation

Ch 5.1.1-3: Leave one out and $k$ -Fold Cross-Validation