Introduction to Machine Learning

# Weekly Homework 3: Bank Marketing

### Author:

University of Redlands - DATA 301

Prof: Joanna Bieri joanna_bieri@redlands.edu

Class Website

Due Sunday 9/20 at 11:59pm. This covers Day 5 and Day 6.

A Portuguese bank phoned 45,211 customers between 2008 and 2010 to sell them a term deposit (a savings account you lock up for a fixed time). Your job is to build a model that tells the call center who to call, so they spend their time on the people most likely to say yes.

GOALS:

Load and inspect a real file whose documentation is not quite honest with you.
Find a leak before it finds you.
Compare logistic regression and a random forest the Day 5 way, on validation, with a score made for rare positives.
Pick a threshold that fits a real constraint: how many calls the bank can actually make.
Open the test set once, and say what the result means for real people.

What you can copy. You may copy code from the Day 5 and Day 6 notes and change the names. Every sentence you write is yours.

Some questions ask you to commit to a guess before you run any code. Write your guess first and do not change it afterward. A wrong guess costs you nothing. A guess you went back and fixed costs you the whole point of the question.

The data. data/bank-full.csv in this folder. It comes from S. Moro, R. Laureano and P. Cortez, Using Data Mining for Bank Direct Marketing: An Application of the CRISP-DM Methodology, Proceedings of the European Simulation and Modelling Conference, 2011, and is shared through the UCI Machine Learning Repository (doi:10.24432/C5K306) under a CC BY 4.0 license.

How long will this take to run? The random forests in Part 5 take a few seconds each on a lab computer, and maybe 15 to 30 seconds on an older laptop. Nothing here should take minutes. If it does, check that you did not put n_estimators=2000 by accident.

How to turn this in. Your repository on GitHub is your submission. There is no Pull Request to open any more.

Manage your git however you like. Use branches if you want them, or commit straight to main if you do not. What I need is only this:

The finished work is on your main branch.
It is pushed to GitHub before the deadline.
Your name is on the Author line at the top of this notebook.

git add .
git commit -m "Weekly homework 3"
git push

If you did the work on a branch, merge it into main and push before the deadline:

git checkout main
git merge my-branch-name
git push

I grade from whatever is on GitHub at the deadline. If it is not pushed, I cannot see it.

Part 1: Load it and look at it

1a. Load data/bank-full.csv. (Hint: it uses the same separator as the wine file.) How many rows and columns? What are the column names?

1b. The documentation that comes with this data says, word for word: “Missing Attribute Values: None.” Check it with .isna().sum(). Then count how many times the text "unknown" appears in each column. Which columns have it, and how many times? Is the documentation telling the truth?

1c. The documentation also says that pdays is -1 when “client was not previously contacted”. How many rows have pdays == -1? Compare that to the number of "unknown" values in poutcome (the outcome of the previous campaign). What do you think is going on?

1d. Decide what to do about the "unknown" values, and defend it in two or three sentences. Your options include dropping rows, filling them in with something, or keeping "unknown" as its own category. (Before you drop anything, look at how many rows that would throw away.)

1e. What share of customers subscribed (y == "yes")? Is this a rare positive problem? Based on Day 5, which score should you use to compare models, and which one should you not trust on its own?

1f. balance (average yearly balance, in euros) has negative values. How many? Is that a data entry mistake or something real? Say what you think and why.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# your code here

Your answers here.

Part 2: The leak

The documentation for the companion version of this data, from the same authors, says this about the duration column (the length of the phone call, in seconds):

“Important note: this attribute highly affects the output target (e.g., if duration=0 then y=”no”). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.”

2a. In your own words, why can the bank not use duration to decide who to call?

2b. Commit to a guess before you run anything. In Part 4 you will fit logistic regression with and without duration. How much do you think the validation average precision will drop when you take it out? A number is fine.

Your answers here.

Part 3: Encode, then split first

Every model we have used does math on its features, so the features have to be numbers. Nine of the bank columns are text (job, marital, education, default, housing, loan, contact, month, poutcome). The standard fix is one-hot encoding, and in pandas the function is pd.get_dummies. For every text column it makes one new column per category, with a 1 where the row has that category and a 0 where it does not. Number columns are left alone, and the original text column disappears.

Here is a tiny example you can run. The first customer’s contact is "unknown", and notice that "unknown" becomes its own column, contact_unknown. That is exactly what “keep unknown as its own category” means in 1d.

customers = pd.DataFrame({
    "age": [58, 44, 33, 47],
    "housing": ["yes", "yes", "no", "yes"],
    "contact": ["unknown", "cellular", "telephone", "cellular"]})

pd.get_dummies(customers, dtype=int)   # dtype=int gives 0 and 1 instead of True and False

gives the columns age, housing_no, housing_yes, contact_cellular, contact_telephone, contact_unknown, and the first row is 58, 0, 1, 0, 0, 1.

3a. Here is the code for the bank data. Run it (after anything you did to the unknowns in 1d).

# the label: 1 if the customer said yes, 0 if they said no
y = (bank["y"] == "yes").astype(int)

# the features: every column except the label, with the text columns turned into 0/1 columns
X = pd.get_dummies(
    bank.drop(columns=["y"]),     # drop the label so it cannot sneak into the features
    dtype=int)                    # 0 and 1 instead of True and False

How many columns does X have now? Where did the extra ones come from? (Hint: count the columns whose names start with job_, and compare that to the number of different jobs.)

It is fine to run get_dummies before splitting here, because it only turns categories into 0/1 columns. It does not learn a mean, a scale, or anything about y, so nothing leaks. Anything with a .fit() still has to wait until after the split.

3b. Split the test set off first (25 percent, stratify=y, random_state=42) and put it away. Then split a validation set off the training data the same way (25 percent, stratified, random_state=42). Print the size of each set and the share of yeses in each.

3c. Why does stratifying matter for this data in particular?

# your code here

Your answers here.

Part 4: Logistic regression, with and without the leak

4a. Scale the features (fit the scaler on the training set only) and fit LogisticRegression(max_iter=5000) with all the columns, duration included. Print the validation average precision.

4b. Do it again without duration. Print the validation average precision. How far did it drop? Compare to your guess in 2b.

4c. From here on, duration is gone for good. For the model in 4b, at the default threshold of 0.5, print the validation accuracy, the always-no baseline, the confusion matrix, precision, and recall. Then say in one sentence, in plain words for the bank, what that recall means.

# your code here

Your answers here.

Part 5: Random forest

5a. Commit to a guess before you run anything. On wine, the forest beat logistic regression by a lot. Here, will the forest (without duration) beat 4b’s validation average precision? By how much?

5b. Fit RandomForestClassifier(n_estimators=200, random_state=42) on the training data without duration. No scaling needed. Print the training accuracy and the validation average precision.

5c. Compare 5b to 4b and to your guess. Why might a forest not help much here? Think about Day 6: what problem does a forest fix, and does logistic regression have that problem on 25,000 rows?

5d. Try one change to the forest, min_samples_leaf=5 (no leaf may hold fewer than 5 customers). Does validation average precision go up or down? Which forest do you keep, and why is it fine to make this choice using the validation set?

5e. Print the 10 most important features of the forest you kept. The day of the month (day) is near the top. Do you believe the day of the month matters that much to whether someone opens a savings account? (Hint: think about how many different values day, age, and balance can take, compared to a 0/1 column like housing_yes, and what that gives a tree more chances to do.)

# your code here

Your answers here.

Part 6: A threshold that fits the call center

The call center has room to call at most 20 percent of the customers on its list.

6a. For both logistic regression (4b) and the forest you kept (5d), try the thresholds 0.5, 0.4, 0.3, 0.25, 0.2, 0.15, and 0.1 on the validation set. For each one print how many customers would be called, the precision, and the recall. (The validation set has 8477 customers, so 20 percent is about 1695 calls.)

6b. For each model, what is the lowest threshold that stays within the 20 percent budget? At that threshold, which model finds more of the customers who would say yes? Pick one model and one threshold to take to the test set.

6c. In one or two sentences, explain why comparing the two models at the same number of calls is fairer than comparing them at the same threshold.

# your code here

Your answers here.

Part 7: The test set, once

7a. Apply the model and threshold you chose in 6b to the test set, once. Print the confusion matrix, precision, recall, average precision, and the share of test customers who would be called.

7b. Write the two or three sentences you would send to the bank manager. Say how many calls, how many of those calls reach someone who says yes, and what share of all the likely yeses you find.

7c. Who is hurt when this model is wrong? Say what a false positive costs and who pays it, and what a false negative costs and who pays it.

7d. The model uses age and job. Is it fair for a bank to decide who gets offered a product based on age? Two or three sentences. There is no single right answer, and I care about your reasoning.

# your code here

Your answers here.