Introduction to Machine Learning

# Practice Problems Day 15

### Author:

University of Redlands - DATA 301

Prof: Joanna Bieri joanna_bieri@redlands.edu

Class Website


These are practice problems. They are not graded and you do not turn them in. They are here so you can practice today’s code: an embedding, an RNN, and writing your own network as a class. Plan on about 45 minutes. Each training run takes less than a minute on a laptop.

Solutions are in HW-day15-solutions.ipynb in this folder. Try every problem yourself first, then check your work.

Reading: Geron chapter 13, the sections LSTMs and GRUs, and chapter 14, the section Embeddings (inside “Generating Shakespearean Text Using a Character RNN”).

GOALS:

  1. Write a network as a class with __init__ and forward, using nn.Embedding and nn.RNN.
  2. Compare an RNN and a GRU on long reviews.
  3. Explain in words why an RNN can not learn from the start of a long review.

How much should you write yourself? The first code cell is given: the words, the review maker and the training function from the notes. In Problem 1 most of the class is there and you write the missing lines. Problem 2 lists the steps and you write the code. Problem 3 is words only. The Optional exploration at the end is extra, only if you are curious.

Save your work. Commit and push when you are done, so you have your code for later:

git add .
git commit -m "Day 15 practice"
git push

# RUN THIS. The words, the review maker and the training function from the notes.
import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader

feeling_words = ["great", "good", "fun", "loved", "wonderful",       # positive, ids 0 to 4
                 "bad", "boring", "awful", "hated", "terrible"]      # negative, ids 5 to 9
filler_words = ["the", "movie", "was", "and", "a", "plot", "actors", "it", "story", "long",
                "music", "ending", "i", "saw", "with", "my", "friends", "on", "friday", "popcorn"]
vocab = feeling_words + filler_words      # every word gets an id: its spot in this list
vocab_size = len(vocab)

def make_reviews(n_reviews, n_words, seed):
    generator = torch.Generator().manual_seed(seed)       # so we get the same reviews every time
    # Start with all filler words (ids 10 to 29)
    X = torch.randint(10, vocab_size, (n_reviews, n_words), generator=generator)
    # Pick one feeling word (ids 0 to 9) and one spot for it, for every review
    feeling = torch.randint(0, 10, (n_reviews,), generator=generator)
    spot = torch.randint(0, n_words, (n_reviews,), generator=generator)
    X[torch.arange(n_reviews), spot] = feeling           # drop the feeling word into its spot
    # Positive (ids 0 to 4) is label 1, negative (ids 5 to 9) is label 0
    y = (feeling < 5).long()
    return X, y

# The mini-batch training loop from the notes. It returns the validation accuracy after every epoch.
def train_model(model, X_train, y_train, X_valid, y_valid, n_epochs):
    torch.manual_seed(42)
    train_loader = DataLoader(TensorDataset(X_train, y_train), batch_size=32, shuffle=True)
    loss_fn = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

    valid_accuracy = []
    for epoch in range(n_epochs):
        model.train()
        for X_batch, y_batch in train_loader:
            optimizer.zero_grad()
            scores = model(X_batch)
            loss = loss_fn(scores, y_batch)
            loss.backward()
            optimizer.step()

        model.eval()
        with torch.no_grad():
            y_pred = model(X_valid).argmax(dim=1)
            accuracy = (y_pred == y_valid).float().mean().item()
        valid_accuracy.append(accuracy)
        print("epoch", epoch + 1, "  valid accuracy:", round(accuracy, 4))

    return valid_accuracy

# 20 word reviews for Problem 1
X_train, y_train = make_reviews(4000, n_words=20, seed=0)
X_valid, y_valid = make_reviews(1000, n_words=20, seed=1)
print("X_train:", X_train.shape)

Problem 1: Write the RNN class

In the notes we wrote our first network as a class, ReviewNet. Now you get to write it yourself! Most of the class is below, and you fill in the four lines the comments ask for. Try it without looking at the notes first, then check.

This time the reviews are 20 words long. That is in between the 10 word reviews (where the RNN got 1.0) and the 100 word reviews (where it stayed near 0.5) from the notes.

1a. Fill in the class and train it on the 20 word reviews for 5 epochs.

1b. Count how many numbers your network has to learn. Would that number change if the reviews were 100 words long? Why or why not?

1c. How does your accuracy compare to the notes? There the RNN got 1.0 on the 10 word reviews and stayed near 0.5 on the 100 word reviews.

class ReviewNet(nn.Module):

    def __init__(self):
        super().__init__()                       # always the first line
        # Write the nn.Embedding layer: one row for every word, 8 numbers per word

        # Write the nn.RNN layer: 8 numbers in, a memory of 32 numbers, batch_first=True

        self.out = nn.Linear(32, 2)                       # go from 32 to 2 scores: negative, positive

    def forward(self, X):
        # Send X through the embedding

        # Send the vectors through the RNN. It gives back TWO things!

        scores = self.out(last_hidden[0])                 # [0] takes (1, reviews, 32) to (reviews, 32)
        return scores

torch.manual_seed(42)
model_1 = ReviewNet()
accuracy_1 = train_model(model_1, X_train, y_train, X_valid, y_valid, n_epochs=5)
# 1b: count the numbers to learn

Your answers here.


Problem 2: RNN or GRU on 50 word reviews (steps outlined)

Now let’s make the reviews longer, 50 words, and see which network can still learn them. Copy your class from Problem 1 and write the steps below yourself.

2a. Make 50 word reviews (4,000 to train on with seed=0, 1,000 to validate on with seed=1). Train a new ReviewNet on them for 5 epochs. Then write a ReviewGRU class, the same as ReviewNet with nn.GRU in place of nn.RNN, and train it the same way.

2b. Which network learned the 50 word reviews? Use the words gradient and gate to explain why.

# Step 1: make the 50 word reviews with make_reviews (train with seed=0, validate with seed=1)

# Step 2: train a new ReviewNet on them for 5 epochs (set torch.manual_seed(42) first)

# Step 3: write a class ReviewGRU. Copy ReviewNet and change ONE line: nn.RNN becomes nn.GRU

# Step 4: train a ReviewGRU on the 50 word reviews for 5 epochs
# Each training run prints 5 lines, one validation accuracy per epoch.

Your answers here.


Problem 3: In words

No code here, just explaining. Writing it out is the best way to check that you really understand it!

3a. Explain to a friend in DATA 101 why an RNN has trouble learning from the start of a long review. Use the words hidden state and gradient. Three or four sentences, no code.

3b. Here is a real sentence: “The movie that my friends from high school, who I had not seen in years, picked for Friday night was great.” To know what was great, which two words does a network need to connect? How many words are between them? Why is that hard for an RNN?

Your answers here.


Optional exploration

Only if you are curious! Nothing here is needed later.

A. Where does the RNN break? Train ReviewNet on 30 and 40 word reviews. Where does the accuracy drop?

B. Try an LSTM, the older version of the GRU idea from the reading. Use nn.LSTM(8, 32, batch_first=True). BEWARE: for the last memory it gives back a pair, so the forward line becomes all_hidden, (last_hidden, last_cell) = self.memory(vectors). Train it on the 50 word reviews.

C. Play with the sizes! In your ReviewGRU class the embedding size is the 8 in nn.Embedding(vocab_size, 8) and in nn.GRU(8, 32, batch_first=True). Change BOTH of them. Try an embedding size of 2, 8 and 32 on the 50 word reviews. For each one write down the validation accuracy after every epoch and how many numbers there are to learn (the p.numel() loop from Problem 1). Does a bigger embedding always help? You can also try the memory size (the 32, in two places), the learning rate (lr=0.01 inside train_model) or the number of epochs.

NOTE: With an embedding size of 2 every word is just 2 numbers, so you could plot the words directly, no PCA needed!

# your code here

Your answers here.