black black

toc

Ch 5.1.4-5: More Cross-Validation

Lecture 14 - CMSE 381

Michigan State University
::
Dept of Computational Mathematics, Science /span> Engineering

Wed, Feb 18, 2026

Announcements

Last time:

k-fold CV

This lecture:

More $k$ -fold CV
Bias-Variance Tradeoff
CV for classification

Screenshot of the course schedule forlectures 11 to 20.

Section 1

k

-fold CV

Approximations of Test Error

Validation Set Diagram
showing the first validation
set split.

Diagram
showing the first validation
set split.

Diagram showing
the second validation set
split.

Diagram showing the
third validation set split.

Diagram showing the fourth
validation set split.

Diagram
showing the fifth validation
set split.

Figure labeled
validation set for
comparison.

LOOCV

Diagram showing
the first leave-one-out
cross-validation split.

Diagram showing the
second leave-one-out
cross-validation split.

Diagram showing the third
leave-one-out
cross-validation split.

⋮

Diagram showing the final
leave-one-out
cross-validation split.

Figure
using the auto data example
to show a fixed result with
no randomness across
repeated runs.

K-fold CV

Diagram
showing the first split in
five-fold cross-validation.

Diagram showing the
second split in five-fold
cross-validation.

Diagram
showing the third split in
five-fold cross-validation.

Diagram showing the fourth
split in five-fold
cross-validation.

Diagram
showing the fifth split in
five-fold cross-validation.

Second figure from the auto
data example.

Definition of

k

-fold CV

Randomly split data into $k$ -groups (folds)
Approximately equal sized. For the sake of notation, say each set has $ℓ$ points
Remove $i$ th fold $U_{i}$ and reserve for testing.
Train the model on remaining points
Calculate ${MSE}_{i} = \frac{1}{ℓ} \sum_{(x_{j}, y_{j}) \in U_{i}} {(y_{j} - ŷ_{j})}^{2}$
Rinse and repeat

Diagram showing the first split in five-fold
cross-validation.

Diagram showing the
second split in five-fold cross-validation.

Diagram showing the third split in five-fold
cross-validation.

Diagram showing the
fourth split in five-fold cross-validation.

Diagram showing the fifth split in five-fold
cross-validation.

Return

C V_{(k)} = \frac{1}{k} \sum_{i = 1}^{k} {MSE}_{i}

Comparison with simulated data: Ex 3

Plot of simulated nonlinear data
with fitted curves of different
flexibility and corresponding
error behavior.

Plot comparing true test
MSE, LOOCV estimate, and
10-fold cross-validation
estimate, showing very
similar results.

Comparison with simulated data: Ex 1

Plot of simulated nonlinear data
with fitted curves of different
flexibility and corresponding
training and test error behavior.

Plot comparing true test
MSE, LOOCV estimate, and
10-fold cross-validation
estimate, showing that the
cross-validation estimates
underestimate the true test
MSE.

Comparison with simulated data: Ex 2

Plot of simulated close-to-linear
data with fitted curves of
different flexibility and
corresponding training and test
error behavior.

Plot comparing true test
MSE, LOOCV estimate,
and 10-fold cross-validation
estimate, showing close
agreement at low flexibility
and overestimation at
higher flexibility.

Takeaways from the examples

Bias-Variance Tradeoff: Bias

E {(y_{0} - \hat{f} (x_{0}))}^{2} = Var (\hat{f} (x_{0})) + {[Bias (\hat{f} (x_{0}))]}^{2} + Var (𝜀)

Bias-Variance Tradeoff: Variance

E {(y_{0} - \hat{f} (x_{0}))}^{2} = Var (\hat{f} (x_{0})) + {[Bias (\hat{f} (x_{0}))]}^{2} + Var (𝜀)

In short: Vadidation vs Test

all the time, we are pretending the validation set etc is the test set...
when it is not.

— Next Frame —

Real-world example: Chekroud et al., Science 383, 164–167 (2024)

Figure from a real-world example in Chekroud et al., Science 383, 164–167 (2024).

Real-world example: Chekroud et al., Science 383, 164–167 (2024)

Second figure from a real-world example in Chekroud et al., Science 383, 164–167 (2024).

Section 2

Using K-Fold CV on Polynomial Linear Regression

Polynomial regression

Replace linear model

y_{i} = β_{0} + β_{1} x_{1} + 𝜀_{i}

with

y_{i} = β_{0} + β_{1} x_{1} + β_{2} x_{1}^{2} + \dots + β_{d} x_{1}^{d} + 𝜀_{i}

Faking linear regression into doing our work for us

Coding - Build a plot for train/test scores vs flexibility

Next time

Screenshot of the course schedule for lectures 11 to 20.