Same outputs after model.eval() in EfficientNet 3d model.

[HyeonhoonLee]

When using EfficientNetBN model with 3 spatial dimension, model.train() works well, but model.eval() makes the same output.
It might be the same issue as the other repository for EfficientNet3d model as follows:
shijianjian/EfficientNet-PyTorch-3D#6

model.eval() should change the layers including BatchNorm, dropout, and so on, for evaludation mode (not training mode), but it might not work.
Does anyone solve this problem?

pytoch ==1.7.0.
cuda==11.0.
monai==0.6.0

[Nic-Ma]

Hi @yiheng-wang-nv ,

Could you please help take a look this ticket?

Thanks in advance.

[yiheng-wang-nv]

Hi @HyeonhoonLee , I did not detect this issue when using monai 0.6.0:

but model.eval() makes the same output

can you provided a detailed example for the issue you met? Thanks.

[HyeonhoonLee]

@yiheng-wang-nv Thank you for your consideration,

When using the other networks for classification such as Densenet or ViT, model.eval() works well,
but when using Efficientnet, model.eval() makes the same output as follows:

'''

model = monai.networks.nets.EfficientNetBN('efficientnet-b2', spatial_dims=3, in_channels=4, num_classes=1)
model.to(device)
dataset.set_fold(k, True)
optimizer = torch.optim.AdamW(model.parameters(), lr=LEARNING_RATE)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', factor=0.5, patience=3, verbose=True)
best_auroc = 0.0
for epoch in range(EPOCHS):
     dataset.set_fold(k, train=True)
     train_prob = []
     train_label = []
     train_loss = []
     model.train()
     for x in dataloader:
         volume = x['volume'].to(device)
         label = x['label'].to(device)

         # forward
         logit = model(volume).squeeze(1)
         loss = criterion(logit, label)
         optimizer.zero_grad()
         loss.backward()
         optimizer.step()

         train_loss.append(loss.detach().cpu().numpy())
         train_prob.append(logit.sigmoid().detach().cpu().numpy())
         train_label.append(label.detach().cpu().numpy())
         torch.cuda.empty_cache()

     train_loss = np.mean(train_loss)
     train_auroc = roc_auc_score(np.concatenate(train_label), np.concatenate(train_prob))

     dataset.set_fold(k, train=False)
     val_prob = []
     val_label = []
     val_loss = []
     model.eval()
     with torch.no_grad():
         for x in dataloader:
             # print(x['volume'][0].sum())
             # print(x['volume'][1].sum())
             # print(x['volume'][2].sum())

             # print(x[1])
             volume = x['volume'].to(device)
             label = x['label'].to(device)
                
             # print(volume[0].sum())
             # print(volume[1].sum())
             # print(volume[2].sum())

             # forward
             logit = model(volume).squeeze(1)

             loss = criterion(logit, label)

             val_loss.append(loss.detach().cpu().numpy())
             val_prob.append(logit.sigmoid().detach().cpu().numpy())
             val_label.append(label.detach().cpu().numpy())
             torch.cuda.empty_cache()
     print(train_prob)
     print(val_prob)

'''
In the last line of this case above, you might see
train_probabilities are (naturally) different between input samples,
but valid_probablilites are all the same in input samples.

Thanks.

[yiheng-wang-nv]

Hi @HyeonhoonLee , since I did not have your dataset, I used a MedMNIST 3D dataset (https://github.com/MedMNIST/MedMNIST) to do the test.
I found that the convergence speed for efficientnet-b1 is very slow compared with resnet10, it may be the reason that you detected same valid probabilties for different input samples.
I've uploaded the script for my test, can you please take a look at it?
If setting total_epochs = 1 in the script, you will see same predictions via print(val_outputs), else if setting total_epochs=100, you will see the predictions are different.

test_3d.ipynb.zip

[HyeonhoonLee]

Hi @yiheng-wang-nv
I check it via your attached file, and identified that efficientnet has slow convergence speed.
Thank you for your kind work!!