black black

toc

Ch 5.1.5: $k$ -fold Cross-Validation for Classification

Lecture 15 - CMSE 381

Michigan State University
::
Dept of Computational Mathematics, Science /span> Engineering

Fri, Feb 20, 2026

Announcements

Last time:

k-fold CV

This lecture:

CV for classification

Announcements:

Homework #3 is due Sunday 3/8

Section 1

Last time

Approximations of Test Error

Validation Set Diagram
showing the first validation
set split.

Diagram
showing the first validation
set split.

Diagram showing
the second validation set
split.

Diagram showing the
third validation set split.

Diagram showing the fourth
validation set split.

Diagram
showing the fifth validation
set split.

Figure labeled
validation set for
comparison.

LOOCV

Diagram showing
the first leave-one-out
cross-validation split.

Diagram showing the
second leave-one-out
cross-validation split.

Diagram showing the third
leave-one-out
cross-validation split.

⋮

Diagram showing the final
leave-one-out
cross-validation split.

Figure
using the auto data example
to show a fixed result with
no randomness across
repeated runs.

K-fold CV

Diagram
showing the first split in
five-fold cross-validation.

Diagram showing the
second split in five-fold
cross-validation.

Diagram
showing the third split in
five-fold cross-validation.

Diagram showing the fourth
split in five-fold
cross-validation.

Diagram
showing the fifth split in
five-fold cross-validation.

Second figure from the auto
data example.

Definition of

k

-fold CV

Randomly split data into $k$ -groups (folds)
Approximately equal sized. For the sake of notation, say each set has $ℓ$ points
Remove $i$ th fold $U_{i}$ and reserve for testing.
Train the model on remaining points
Calculate ${MSE}_{i} = \frac{1}{ℓ} \sum_{(x_{j}, y_{j}) \in U_{i}} {(y_{j} - ŷ_{j})}^{2}$
Rinse and repeat

Diagram showing the first split in five-fold
cross-validation.

Diagram showing the
second split in five-fold cross-validation.

Diagram showing the third split in five-fold
cross-validation.

Diagram showing the
fourth split in five-fold cross-validation.

Diagram showing the fifth split in five-fold
cross-validation.

Return

C V_{(k)} = \frac{1}{k} \sum_{i = 1}^{k} {MSE}_{i}

Section 2

CV for Classification

Setup: LOOCV

Remove $i$ th point $(x_{i}, y_{i})$ and reserve for testing.
Train the model on remaining points
Calculate ${Err}_{i} = I (y_{j} \neq ŷ_{j})$
Rinse and repeat

Diagram showing the first leave-one-out
cross-validation split.

Diagram showing the
second leave-one-out cross-validation split.

Diagram showing the third leave-one-out
cross-validation split.

Diagram showing the fourth
leave-one-out cross-validation split.

Diagram
showing the fifth leave-one-out cross-validation
split.

⋮

Diagram showing the final leave-one-out
cross-validation split.

Return

C V_{(n)} = \frac{1}{n} \sum_{i = 1}^{n} {Err}_{i}

Setup:

k

-fold

Randomly split data into $k$ -groups (folds)
Approximately equal sized. For the sake of notation, say each set has $ℓ$ points
Remove $i$ th fold $U_{i}$ and reserve for testing.
Train the model on remaining points
Calculate ${Err}_{i} = \frac{1}{ℓ} \sum_{(x_{j}, y_{j}) \in U_{i}} I (y_{j} \neq ŷ_{j})$
Rinse and repeat

Diagram showing the first split in five-fold
cross-validation.

Diagram showing the
second split in five-fold cross-validation.

Diagram showing the third split in
five-fold cross-validation.

Diagram
showing the fourth split in five-fold
cross-validation.

Diagram showing the
fifth split in five-fold cross-validation.

Return

C V_{(k)} = \frac{1}{k} \sum_{i = 1}^{k} {Err}_{i}

Example on simulated data: Linear

Classification plot illustrating data
points and the Bayes decision
boundary.

Purple: Bayes decision boundary.
- Error rate: 0.133
Black: Logistic regression
- $log (p ∕ (1 - p)) = β_{0} + β_{1} X_{1} + β_{2} X_{2}$
- Error rate: 0.201

Example on simulated data: Quadratic logistic regression

Classification plot showing data
points and the Bayes decision
boundary for a second example.

Purple: Bayes decision boundary.
- Error rate: 0.133
Black: Logistic regression
- $log (p ∕ (1 - p)) = β_{0} + β_{1} X_{1} + β_{2} X_{1}^{2} + β_{3} X_{2} + β_{4} X_{2}^{2}$
- Error rate: 0.197

Example on simulated data: all the polynomials!

Classification plot comparing the Bayes
decision boundary and logistic regression
fits with different polynomial degrees.

Purple: Bayes decision boundary.
- Error rate: 0.133
Black: Logistic regression
- Deg 1 Error rate: 0.201
- Deg 2 Error rate: 0.197
- Deg 3 Error rate: 0.160
- Deg 4 Error rate: 0.162

Decide degree based on CV

Plot of training error, test error, and
10-fold cross-validation error versus model
degree, used to choose the degree.

Test error (brown)
Training error (blue)
10-fold CV error (black)

Similar game for KNN

Plot of training error, test error, and
10-fold cross-validation error used to
choose the value of K in classification.

Test error (brown)
Training error (blue)
10-fold CV error (black)

Coding - k-fold for penguin classification section

TL;DR

k

-fold CV

Diagram showing the first split in five-fold
cross-validation.

Diagram showing the
second split in five-fold cross-validation.

Diagram showing the third split in five-fold
cross-validation.

Diagram showing the
fourth split in five-fold cross-validation.

Diagram showing the fifth split in five-fold
cross-validation.

C V_{(k)} = \frac{1}{k} \sum_{i = 1}^{k} {MSE}_{i}

Use $k = 5$ or 10 usually

k

-fold CV for classification

{Err}_{i} = I (y_{j} \neq ŷ_{j})

C V_{(k)} = \frac{1}{k} \sum_{i = 1}^{k} {Err}_{i}

Next time

Screenshot of the course schedule for lectures 11 to 20.