78. Evaluating our model's predictions (but could be anywhere we discuss Training vs Testing)

[tljthree]

I'd appreciate help in my assumptions I'm growing from the course, and a question at the end. Let's use the code at following location in the course: https://www.learnpytorch.io/02_pytorch_classification/#5-improving-a-model-from-a-model-perspective

for epoch in range(epochs):
### Training
# 1. Forward pass
y_logits = model_1(X_train).squeeze()
y_pred = torch.round(torch.sigmoid(y_logits)) # logits -> prediction probabilities -> prediction labels

# 2. Calculate loss/accuracy
loss = loss_fn(y_logits, y_train)
acc = accuracy_fn(y_true=y_train, 
                  y_pred=y_pred)

# 3. Optimizer zero grad
optimizer.zero_grad()

# 4. Loss backwards
loss.backward()

# 5. Optimizer step
optimizer.step()

### Testing
model_1.eval()
with torch.inference_mode():
    # 1. Forward pass
    test_logits = model_1(X_test).squeeze() 
    test_pred = torch.round(torch.sigmoid(test_logits))
    # 2. Caculate loss/accuracy
    test_loss = loss_fn(test_logits,
                        y_test)
    test_acc = accuracy_fn(y_true=y_test,
                           y_pred=test_pred)

# Print out what's happening every 10 epochs
if epoch % 100 == 0:
    print(f"Epoch: {epoch} | Loss: {loss:.5f}, Accuracy: {acc:.2f}% | Test loss: {test_loss:.5f}, Test acc: {test_acc:.2f}%")

My assumptions:

A1: The TRAINING portion in the loop is the only part of the loop that influences/improves the model.
A2: The TESTING portion of the loop uses the model, but doesn't change the model.

My questions:

Q1: Why is the testing portion placed inside the loop? Is this just to let us see if the iterations of training are also improving the accuracy of the model with the test feature values?
Q2: If we want to evaluate the model using the test data, couldn't we just do that after (but outside) the loop?

My observations:

O1: Placing the Testing code inside the loop that does training, then separately placing code to predict (e.g. y_preds) outside of the loop, confuses things for me.

Can someone please confirm/correct my assumptions, and respond to my questions/observations.

Thanks,
Tom

btw, thank you so much, Daniel, for your amazing course. You are a great teacher.

[37Lime]

Greetings @tljthree ,

Q1: Yes, I think you have understood it correct. This is useful because it allows you to track the model's performance during the training phase. For example, it allows you to detect overfitting (training continues to improve while testing plateaus/decline). Also, by tracking your model's performance you can evaluate the effect of using different hyperparameter values (i.e., hyperparameter tuning). These are just two examples.

Q2: Yes, you can do that. That is usually what you do when you want to use your trained model on new data. Perhaps you can think of this as the phase where you put your model into work. For example, if you trained a model to classify cats, you start to use it as a cat classifier.

Best wishes.

[tljthree]

Thank you so much. I didn't think about detecting overfitting. Makes sense!