Introduction to Machine Learning

# Weekly Homework 4: The Bank, Done Properly

### Author:

University of Redlands - DATA 301

Prof: Joanna Bieri joanna_bieri@redlands.edu

Class Website

Due Friday 10/2 at 11:59pm. This covers Day 7 and Day 8.

Same bank, same 45,211 phone calls, same question: who should the call center call? In Weekly Homework 3 you did all the preparation by hand. This time you build one pipeline that does it for you, boost the model, let a search pick the settings, and open the test set once.

GOALS:

Turn last week’s by-hand preparation into a ColumnTransformer and a Pipeline.
Find out whether “unknown” is missing data or an answer, and let the data settle it.
Put boosting inside the pipeline and search for settings on a budget you decide first.
Choose a threshold that fits the call center, open the test set once, and say what it means.

What you can copy. Copy freely from the Day 7 and Day 8 notes and change the names. Every sentence you write is yours.

Some questions ask you to commit to a guess before you run anything. Write the guess first and leave it alone. A wrong guess costs nothing. A guess you went back and fixed costs the whole point of the question.

The data. data/bank-full.csv, the same file as Weekly Homework 3. From S. Moro, R. Laureano and P. Cortez, Using Data Mining for Bank Direct Marketing, 2011, in the UCI Machine Learning Repository (doi:10.24432/C5K306), CC BY 4.0.

How long does it run? The search in Part 5 is the slow part, around fifteen to thirty seconds on a newer laptop and up to a minute or two on an older one. Everything else is seconds. If something runs for many minutes, your search budget is too big: lower n_iter or the number of folds.

How to turn this in. Your repository on GitHub is your submission. There is nothing to hand in on Canvas.

Manage your git however you like. Use branches if you want them, or commit straight to main if you do not. What I need is only this:

The finished work is on your main branch.
It is pushed to GitHub before the deadline.
Your name is on the Author line at the top of this notebook.

git add .
git commit -m "Weekly homework 4"
git push

I grade from whatever is on GitHub at the deadline. If it is not pushed, I cannot see it.

Part 1: Load it, and deal with the leak

1a. Load data/bank-full.csv. Drop the duration column immediately and say in one sentence why it cannot be used. (This is the Weekly Homework 3 leak. If you need to, re-read the note from the data’s own documentation.)

1b. Make your label y: 1 if the customer subscribed, 0 if not. What share said yes?

1c. Last week you found "unknown" hiding in four columns. Count them again, and list which columns and how many.

1d. Split off a test set (25 percent, stratified, random_state=42) and put it away. You do not need a separate validation set this week, because every choice from here on is made with cross validation inside a search. Explain in one sentence why that is enough.

# your code here

Your answers here.

Part 2: Is “unknown” missing, or is it an answer?

Last week you left "unknown" in place, so get_dummies gave it its own column. A pipeline invites you to treat it as missing instead, and fill it in with the most common value. Those are different claims about the world, and only one of them can be better here.

2a. Commit to a guess before you run anything. Which will score better: treating "unknown" as its own category, or replacing it with the most frequent value?

2b. Build two preprocessing versions.

Version A, “unknown is a category”: leave the text as it is, so OneHotEncoder makes an unknown column. Numbers get an imputer and a scaler.
Version B, “unknown is missing”: replace "unknown" with np.nan first, then use SimpleImputer(strategy="most_frequent") on the text columns before encoding.

Put LogisticRegression(max_iter=5000) on the end of each and score both with cross_val_score on the training data, cv=3, scoring="average_precision". Report both means.

2c. Which won, and by how much? Compare that gap to the fold-to-fold spread you get from cross_val_score.

2d. Explain the result. What does "unknown" mean in the poutcome column, and is that the same kind of thing as a missing job title? (Look back at what you found in Weekly Homework 3 Part 1c.)

2e. Use the winner for the rest of this assignment. Say which one you are using.

# your code here

Your answers here.

Part 3: One pipeline, end to end

3a. Build your chosen preprocessing as a single ColumnTransformer, with a number path and a text path. Print how many columns go in and how many come out, and explain where the new ones came from.

3b. Put LogisticRegression(max_iter=5000) on the end in a Pipeline and score it with cross_val_score on the training data, cv=3, scoring="average_precision". How does it compare to the 0.423 you got by hand last week?

3c. In two or three sentences: name two things this pipeline protects you from that your by-hand version last week did not.

# your code here

Your answers here.

Part 4: Boosting, from Day 7

4a. Commit to a guess. On this data, how much will boosting beat logistic regression?

4b. Put XGBClassifier in the pipeline instead, with n_estimators=300, learning_rate=0.1, max_depth=4, random_state=42, and eval_metric="logloss". Score it the same way. Compare to 4a and to Part 3.

4c. Day 7 said boosting can overfit as trees pile up. You have n_estimators=300 here with no early stopping. Why is that safer inside this search than it would be if you were fitting once by hand? (Think about what cross validation is measuring.)

# your code here

Your answers here.

Part 5: Let a search pick the settings

5a. Before you run anything, decide your budget. You will use RandomizedSearchCV with n_iter=6 and cv=3. How many fits is that? If one fit takes about a second, how long should the search take?

5b. Run the search over the boosted pipeline. Search these four settings:

{"model__n_estimators": randint(100, 400),
 "model__max_depth": randint(2, 6),
 "model__learning_rate": uniform(0.02, 0.2),
 "model__subsample": uniform(0.6, 0.4)}

Report the best settings and the best cross validated average precision. Was your time estimate right?

5c. Compare the search’s best score to Part 4b, which used settings I picked. Did the search find something better? Is the difference bigger than the fold-to-fold spread?

5d. One of the settings is subsample, which you have not seen. Use the Day 5 AI prompt, or the XGBoost documentation, and explain in one sentence what it does and why it might help.

# your code here

Your answers here.

Part 6: A threshold the call center can live with

The call center can call at most 20 percent of the customers on a list.

6a. Using your searched model, get predicted probabilities for the training data with cross_val_predict (cv=3, method="predict_proba"). Why does this have to be cross_val_predict rather than just fitting and predicting on the training data?

6b. Find the threshold that flags exactly the top 20 percent of those customers. (Hint: np.quantile on the probabilities.) At that threshold, what are precision and recall?

6c. In plain words, what do those two numbers mean for the bank’s afternoon?

# your code here

Your answers here.

Part 7: The test set, once

7a. Fit your chosen pipeline on all the training data, then score the test set once. Use the same 20 percent rule: flag the top 20 percent of test customers. Print the confusion matrix, precision, recall, and average precision.

7b. Write the three sentences you would send the head of the call center: how many calls, how many of them reach someone who says yes, what share of all the likely yeses you find, and how that compares to calling at random.

7c. Compare your test numbers to Weekly Homework 3, where you did this by hand. Did the pipeline and the search buy you much? Answer honestly, and say what you think would help more than a better model.

7d. Who is hurt when this model is wrong, and who is hurt when it is right? Two or three sentences. Think about the customer who gets called, the customer who never does, and the bank.

# your code here

Your answers here.