Ch 6.1: Subset Selection

Lecture 16 - CMSE 381

Michigan State University
::
Dept of Computational Mathematics, Science /span> Engineering

Mon, Feb 23, 2026

Announcements

Last time

$k$ -fold CV for Classification

Covered in this lecture

Subset selection
Forward and Backward Selection

Announcements:

HW #4 due Sunday 3/8
HW #5 due Sunday 3/15

Screenshot of the course schedule forlectures 11 to 20.

What should you learn from this lecture?

Why shouldn’t you have as many predictors in your model as possible?
What are the three basic methods could be used to select the appropriate predictors (feature selection)?
What are the steps/procedures in the algorithm for each of these methods?
How to implement these procedures by hand given the training and CV test error for each combo of predictors?
What are the pros and cons for choosing each of these feature selection methods? When you can or cannot use them?
How do you implement the feature selection algorithm in Python?

Section 1

Previously on linear regression ...

The problem of many features (p) relative to samples (n)

Up to now, we’ve focused on standard linear model:

Y = β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p} + 𝜀

and done least squares estimation.

Prediction accuracy

The problem of many features (p) relative to samples (n)

Up to now, we’ve focused on standard linear model:

Y = β_{0} + β_{1} X_{1} + \dots + β_{p} X_{p} + 𝜀

and done least squares estimation.

Model Interpretability

Section 2

Best Subset Selection

Go through each combo of variables exhaustively (exhausting?)

All subsets of 4 variables ( $2^{4} = 16$ )

$\emptyset$

$X_{1}$
$X_{2}$
$X_{3}$
$X_{4}$

$X_{1}$ $X_{2}$
$X_{1}$ $X_{3}$
$X_{1}$ $X_{4}$
$X_{2}$ $X_{3}$
$X_{2}$ $X_{4}$
$X_{3}$ $X_{4}$

$X_{1}$ $X_{2}$ $X_{3}$
$X_{1}$ $X_{2}$ $X_{4}$
$X_{1}$ $X_{3}$ $X_{4}$
$X_{2}$ $X_{3}$ $X_{4}$

$X_{1}$ $X_{2}$ $X_{3}$ $X_{4}$

One way of breaking this up

Algorithm diagram showing one way to break up the subset
selection procedure into steps for choosing models based on
training and testing scores.

Calculate by hand

We train a model using four variables,

X_{1}, X_{2}, X_{3}, X_{4}

. We’re interested in getting a subset of the variables to use. The following table shows the mean squared error and the MSE value computed for the model learned using each possible subset of variables.

Table showing mean squared error values
for models fit using all possible subsets of
four variables in a subset selection
example.

: What subset of variables is found for each of the sets $M_{0}, M_{1}, M_{2}, M_{3}, M_{4}$ when using best subset selection?
: What subset of variables is returned using best subset selection?

Extra work space if it helps

Table
showing
mean
squared
error
values
for
models
fit
using
all
possible
subsets
of
four
variables
in
a
subset
selection
example.

$\emptyset$

$X_{1}$
$X_{2}$
$X_{3}$
$X_{4}$

$X_{1}$ $X_{2}$
$X_{1}$ $X_{3}$
$X_{1}$ $X_{4}$
$X_{2}$ $X_{3}$
$X_{2}$ $X_{4}$
$X_{3}$ $X_{4}$

$X_{1}$ $X_{2}$ $X_{3}$
$X_{1}$ $X_{2}$ $X_{4}$
$X_{1}$ $X_{3}$ $X_{4}$
$X_{2}$ $X_{3}$ $X_{4}$

$X_{1}$ $X_{2}$ $X_{3}$ $X_{4}$

Code to do this

Section 3

Forward Selection

What’s the problem with best subset selection?

Forward Stepwise Selection

An example for Forward Stepwise Selection

$\emptyset$

$X_{1}$
$X_{2}$
$X_{3}$
$X_{4}$

$X_{1}$ $X_{2}$
$X_{1}$ $X_{3}$
$X_{1}$ $X_{4}$
$X_{2}$ $X_{3}$
$X_{2}$ $X_{4}$
$X_{3}$ $X_{4}$

$X_{1}$ $X_{2}$ $X_{3}$
$X_{1}$ $X_{2}$ $X_{4}$
$X_{1}$ $X_{3}$ $X_{4}$
$X_{2}$ $X_{3}$ $X_{4}$

$X_{1}$ $X_{2}$ $X_{3}$ $X_{4}$

Group work: by hand same example with forward example

We train a model using four variables,

X_{1}, X_{2}, X_{3}, X_{4}

. We’re interested in getting a subset of the variables to use. The following table shows the mean squared error and the

R^{2}

value computed for the model learned using each possible subset of variables.

Table showing mean squared error values
for models fit using all possible subsets of
four variables in a subset selection
example.

: What subset of variables is found for each of the sets $M_{0}, M_{1}, M_{2}, M_{3}, M_{4}$ when using forward selection?
: What subset of variables is returned using forward subset selection?