Introduction to Machine Learning

# Weekly Homework 5: Life Expectancy

### Author:

University of Redlands - DATA 301

Prof: Joanna Bieri joanna_bieri@redlands.edu

Class Website

Due Sunday 10/18 at 11:59pm. This covers Day 9 and Day 10.

Can we predict how long people live in a country from its health and economic numbers? This week you answer that with a linear regression and with a neural network you train yourself in PyTorch, and you find out whether the network is worth it.

GOALS:

Split the data in a way that is honest for this problem.
Prepare the data with a ColumnTransformer from Day 8.
Build and train an MLP with a mini-batch training loop, and read its learning curve.
Open the test set once and explain what the model can and can not do.

What you can copy. Copy freely from the Day 8, 9 and 10 notes and the practice problems, and change the names. Every sentence you write is yours.

The data. data/LifeExpectancyData.csv. One row per country per year, 2000 to 2015, 193 countries. This is the “Life Expectancy (WHO)” dataset from Kaggle. It was put together from World Health Organization and United Nations data and first posted by KumarRajarshi (Kaggle page). This is real data, with a lot of missing values and some mistakes in it.

How long does it take? Plan on about three hours. All of the code runs in a few seconds.

How to turn this in. Your repository on GitHub is your submission. There is nothing to hand in on Canvas.

The finished work is on your main branch.
It is pushed to GitHub before the deadline.
Your name is on the Author line at the top of this notebook.

git add .
git commit -m "Weekly homework 5"
git push

I grade from whatever is on GitHub at the deadline. If it is not pushed, I cannot see it.

Part 1: Load it and look

The first cell is written for you. The column names in this file have extra spaces in them (for example "Life expectancy "), so the cell removes them with .str.strip().

1a. How many rows, countries and years are there? Some rows are missing the label, Life expectancy. How many? Drop those rows.

1b. Which three columns have the most missing values? We will fill them in with the median. Name one thing that could go wrong with that for a column like Population.

1c. Make a scatter plot of Life expectancy against Schooling (average years of school). Describe what you see in one or two sentences.

# RUN THIS. Load the data and clean up the column names.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

life = pd.read_csv("data/LifeExpectancyData.csv")
life.columns = life.columns.str.strip()          # remove the extra spaces around the column names

print("rows and columns:", life.shape)
print(life.columns.tolist())

# your code here

Your answers here.

Part 2: Split by country

Every country shows up in the data up to 16 times, once per year. That changes how we have to split.

2a. (The old way - split randomly) Run the first cell below. It does an ordinary random split, like we have done all semester, and checks how many test rows come from a country that is also in the training set. Explain why that makes the test score too optimistic. (Think about how different a country’s life expectancy is from one year to the next. Does it make sense that we use the same country in both testing and training? Is there leakage here?)

2b. (The new way - split by country) The second cell splits by country instead, with GroupShuffleSplit: every row for a country goes into the same set. It makes a test set (20 percent of the countries), then a validation set out of what is left. Run it and report the size of each set and how many countries are in each.

# RUN THIS. An ordinary random split, just to check it.
from sklearn.model_selection import train_test_split

y = life["Life expectancy"]
X = life.drop(columns=["Life expectancy"])

X_r_train, X_r_test, y_r_train, y_r_test = train_test_split(X, y, test_size=0.2, random_state=42)
share = X_r_test["Country"].isin(X_r_train["Country"]).mean()
print("share of random-split test rows whose country is also in training:", round(share, 3))

# RUN THIS. Split by country.
from sklearn.model_selection import GroupShuffleSplit

# test set: 20 percent of the countries
splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_rows, test_rows = next(splitter.split(X, y, groups=X["Country"]))     # groups= says which rows belong together
X_train_full, X_test = X.iloc[train_rows], X.iloc[test_rows]
y_train_full, y_test = y.iloc[train_rows], y.iloc[test_rows]

# validation set: 20 percent of the countries that are left
splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_rows, valid_rows = next(splitter.split(X_train_full, y_train_full, groups=X_train_full["Country"]))
X_train, X_valid = X_train_full.iloc[train_rows], X_train_full.iloc[valid_rows]
y_train, y_valid = y_train_full.iloc[train_rows], y_train_full.iloc[valid_rows]

print("train rows:", len(X_train), "  countries:", X_train["Country"].nunique())
print("valid rows:", len(X_valid), "  countries:", X_valid["Country"].nunique())
print("test rows: ", len(X_test), "  countries:", X_test["Country"].nunique(), "   (put away until Part 5)")
print("countries in both train and test:", len(set(X_train["Country"]) & set(X_test["Country"])))

NOTE - we did not do GroupShuffleSplit() in the lecture notes because we have not yet run into this type of data. The important thing here is to get used to thinking deeply about how to do a test train split.

The rest of the code should be like normal, just the split is different.

Your answers here.

Part 3: Prepare the data and fit a baseline

3a. Preprocessing pipeline Build a ColumnTransformer like Day 8. The number columns (everything except Country and Status) get their missing values filled with the median and then get scaled. Status gets filled with the most frequent value and one-hot encoded. Leave Country out, since we want the model to work for countries it has never seen. Just build it here, do not fit it yet. The full pipeline in 3b fits it on the training rows for you.

3b. Full pipeline = preprocessing + model Put your preprocessing and a LinearRegression together in one Pipeline, with the steps named "prep" and "model", like Day 8. Fit it on the raw training rows X_train (the pipeline does the preprocessing for you) and print the training and validation RMSE. Life expectancy is in years, so what does the validation RMSE mean in plain words?

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LinearRegression
from sklearn.metrics import root_mean_squared_error

number_columns = [c for c in X.columns if c not in ["Country", "Status"]]      # every column except these two
text_columns = ["Status"]

# Step 1 (3a): a ColumnTransformer called preprocessing, 
# with a number path and a text path


# Step 2 (3b): a Pipeline called lin_pipeline with ("prep", preprocessing) and ("model", LinearRegression()).
#              Fit it on X_train and y_train, the raw rows. Print the training and validation RMSE.

Your answers here.

Part 4: Train a neural network

NOTE - A PyTorch model does not have .fit() and .predict(), so it can not go inside a scikit-learn Pipeline.

Instead we use the preprocessing pipeline from Part 3, which learned from the training rows only, and hand its output to PyTorch.

So the first thing we do is send the training and validation rows through preprocessing ourselves: fit_transform on the training rows, and transform only on the validation rows. This is a very common way to work in practice: scikit-learn prepares the data, and PyTorch trains the model.

4a. Make X_train_prep with preprocessing.fit_transform(X_train) and X_valid_prep with preprocessing.transform(X_valid), and print how many columns come out.

Then turn the prepared data and the labels into float32 tensors (remember, the labels need .reshape(-1, 1)).

Build an MLP with one hidden layer of 50 neurons and ReLU, like Day 10. Set torch.manual_seed(42) right before you build it.

Train it with mini-batches of 32, nn.MSELoss and torch.optim.SGD with a learning rate of 0.01, for 5 epochs, and print the training RMSE after each epoch. What happens?

The same learning rate worked fine for the house prices on Day 10. What is different about these labels? (Compare the size of a life expectancy to the size of a house value.)

4b. Build a fresh model (seed 42 again) and train it with a learning rate of 0.001 for 100 epochs. Save the training and validation RMSE after every epoch, and plot the learning curve with the linear regression validation RMSE as a dashed line. Print the final training and validation RMSE.

4c. Compare the MLP to linear regression. Which is better on the validation set, and by how much in years? Look at the gap between the training and validation curves. What does it tell you?

import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader

# Step 1 (4a): X_train_prep (fit_transform on X_train) and X_valid_prep (transform only on X_valid).
#              Print the shape of X_train_prep.


# Step 2 (4a): float32 tensors for the prepared training and validation data, and the labels as columns


# Step 3 (4a): a DataLoader for the training data with batch_size=32 and shuffle=True


# Step 4 (4a): seed 42, the MLP, the loss and the optimizer with lr=0.01. Train 5 epochs, print the training RMSE after each one

# Step 5 (4b): a fresh model with seed 42, lr=0.001, 100 epochs, save the training and validation RMSE every epoch


# Step 6 (4b): the learning curve, with plt.axhline for the linear regression validation RMSE


# Step 7 (4b): print the final training and validation RMSE

Your answers here.

Part 5: The test set, once

BEWARE! If you choose the MLP, the test rows have to go through the same preprocessing first:

X_test_prep = preprocessing.transform(X_test)     # transform ONLY, never fit_transform

Use transform and never fit_transform, because the test set must not change anything the preprocessing learned from the training rows. Then turn X_test_prep into a tensor like the others.

If you choose linear regression, the pipeline does this for you: lin_pipeline.predict(X_test).

5a. Choose one model, the linear regression or the MLP, and say why in one or two sentences. Then compute its RMSE on the test set, one time.

5b. Find the five test rows where your model is the most wrong (the biggest absolute error). Print the country, the year, the real life expectancy and the prediction. What do you notice about them? Is it one country, or several?

5c. Write three or four sentences to a public health official who wants to use your model for a country that is not in the data. Say what the model does, how far off it typically is, and one reason to be careful with it. No jargon: no “RMSE”, no “MLP”.

# your code here

Your answers here.