Introduction to Machine Learning

# Practice Problems Day 13

### Author:

University of Redlands - DATA 301

Prof: Joanna Bieri joanna_bieri@redlands.edu

Class Website

---

These are practice problems. They are not graded and you do not turn them in. They are here so you can practice today’s code before the weekly homework. Weekly Homework 7 assumes you have done them. Plan on about 45 minutes.

Solutions are in HW-day13-solutions.ipynb in this folder. Try every problem yourself first, then check your work.

Reading: Geron chapter 11, Avoiding Overfitting Through Regularization (Dropout, and ℓ₁ and ℓ₂ Regularization), and chapter 4, Early Stopping.

GOALS:

1. Spot overfitting in the training and validation RMSE.
2. Add dropout to a network.
3. Add early stopping to a training loop.

How much of this should you write yourself? The starting cell is given. Problem 1 adds a few lines to it. Problem 2 lists the steps. Problem 3 is words only. The Optional exploration at the end is extra, only if you are curious. Each training run takes less than a minute on a laptop.

Save your work. Commit and push when you are done, so you have your code for the weekly homework and the exam:

git add .
git commit -m "Day 13 practice"
git push

---

# RUN THIS. The California housing data from the notes, with only the first 2000 training rows.
import copy
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

housing = pd.read_csv("data/california_housing.csv")
X = housing.drop(columns=["MedHouseVal"])
y = housing["MedHouseVal"]
X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.25, random_state=42)
X_train = X_train[:2000]                                       # only the first 2000 training rows, like the notes
y_train = y_train[:2000]

preprocessing = Pipeline([("scale", StandardScaler())])      # the preprocessing pipeline, fit on the training rows only
X_train_scaled = preprocessing.fit_transform(X_train)
X_valid_scaled = preprocessing.transform(X_valid)

X_train_t = torch.tensor(X_train_scaled, dtype=torch.float32)
X_valid_t = torch.tensor(X_valid_scaled, dtype=torch.float32)
y_train_t = torch.tensor(y_train.values, dtype=torch.float32).reshape(-1, 1)
y_valid_t = torch.tensor(y_valid.values, dtype=torch.float32).reshape(-1, 1)
train_data = TensorDataset(X_train_t, y_train_t)               # pairs the features with the labels for the DataLoader

print("training rows:", X_train_t.shape[0], "   validation rows:", X_valid_t.shape[0])

# RUN THIS. A network with 3 hidden layers of 100 neurons and no regularization, trained for 100 epochs.
torch.manual_seed(42)

model = nn.Sequential(
    nn.Linear(8, 100),                        # 8 features in, 100 neurons
    nn.ReLU(),
    nn.Linear(100, 100),
    nn.ReLU(),
    nn.Linear(100, 100),
    nn.ReLU(),
    nn.Linear(100, 1))                        # 1 prediction out

# the starting weights
for layer in model:
    if isinstance(layer, nn.Linear):          # only the Linear layers have weights
        nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")
        nn.init.zeros_(layer.bias)

# the batches, the loss and the optimizer
torch.manual_seed(42)
train_loader = DataLoader(train_data, batch_size=32, shuffle=True)
loss_fn = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

# the training loop
for epoch in range(100):
    model.train()                             # training mode
    for X_batch, y_batch in train_loader:
        optimizer.zero_grad()
        y_pred = model(X_batch)
        loss = loss_fn(y_pred, y_batch)
        loss.backward()
        optimizer.step()

    # every 20 epochs, print the training and validation RMSE
    if (epoch + 1) % 20 == 0:
        model.eval()                          # prediction mode
        with torch.no_grad():
            train_rmse = torch.sqrt(loss_fn(model(X_train_t), y_train_t)).item()
            valid_rmse = torch.sqrt(loss_fn(model(X_valid_t), y_valid_t)).item()
        print("epoch", epoch + 1, "  train RMSE:", round(train_rmse, 4), "  valid RMSE:", round(valid_rmse, 4))

Problem 1: Spot the overfitting, then add dropout

What to copy, what to write: The cell above is given. Copy it into the cell below and add the dropout lines yourself.

1a. Run the starting cell above. Is this network overfitting? Point at the numbers that tell you that there is overfitting.

1b. Copy the cell and add nn.Dropout(p=0.1) after each nn.ReLU(). Run it. Compare the training RMSE and the validation RMSE after 100 epochs to 1a. What changed?

# 1b: your copy, with dropout

Your answers here.

---

Problem 2: Add early stopping (steps outlined)

What to copy, what to write: Start from your dropout cell from 1b. Write the early stopping lines yourself, following the “real life” cell at the end of the notes.

2a. Add early stopping with a patience of 15, and let it train for at most 200 epochs. Print the epoch where it stopped, the best epoch, and the best validation RMSE.

2b. Is the best validation RMSE better than what you got after 100 epochs in 1b? How many epochs did the network train this time?

# Step 1: copy your dropout cell from 1b, and change range(100) to range(200)

# Step 2: before the loop, make patience = 15, best_rmse = float("inf"), best_weights = None,
#         and epochs_without_improving = 0

# Step 3: at the end of EVERY epoch (not every 20), compute the validation RMSE in prediction mode

# Step 4: if it is a new best, save it, save the epoch, save copy.deepcopy(model.state_dict()),
#         and set epochs_without_improving back to 0. Otherwise add 1 to epochs_without_improving

# Step 5: if epochs_without_improving == patience, print the epoch and break

# Step 6: after the loop, model.load_state_dict(best_weights) and print the best epoch and RMSE

Your answers here.

---

Problem 3: In words

3a. Explain early stopping to a friend who knows what overfitting is but has never trained a neural network. Three or four sentences, no code.

3b. A classmate’s network has a training RMSE of 0.05 and a validation RMSE of 0.9. Name two things from today they could try, and say why each one could help.

Your answers here.

---

Optional exploration

Only if you are curious. Nothing here is needed for the weekly homework.

A. In your early stopping cell, take out the dropout layers and add weight_decay=0.0001 to the optimizer instead. How does it compare?

B. Change the dropout rate to 0.5. Is more dropout better here?

# your code here

Your answers here.