About the sliding_window_inference

[ZDYsszz]

if the model has two or more outputs, the sliding_window_inference doesn't work,emm...

[Nic-Ma]

Hi @ZDYsszz ,

Thanks for your feedback.
Yes, the sliding window module doesn't support multiple outputs so far, it can support multiple inputs:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/inferers/inferer.py#L165

Thanks.

[a-parida12]

@Nic-Ma @wyli I would like to use the sliding window inference (https://docs.monai.io/en/stable/inferers.html#monai.inferers.sliding_window_inference) for a case with multiple inputs and single outputs. How can I do it? Do you have an example

[wyli]

we'll have this feature in v0.9, please see #4363

[sulaimanvesal]

hi @wyli

I have modified DynUNet to have two heads for segmentation and classification. Basically for a patch of image it should generate the segmentation and classify the patch as 0 (no lesion) or 1 (contain lesion). The training works fine, however during validation using sliding_window_inference, I get the following error:

Traceback (most recent call last):
  File "/home/svesal/TRUSDetection/src_ucil/scripts/train_decison_normalized.py", line 487, in <module>
    val_outputs = sliding_window_inference(
  File "/home/svesal/anaconda3/lib/python3.9/site-packages/monai/inferers/utils.py", line 241, in sliding_window_inference
    zoomed_start = original_idx[axis].start * zoom_scale[axis - 2]
IndexError: list index out of range

Here is a snippet of the evaluation part. Based on the MONAI documentation for multi-output the sliding window_inference should give a dictionary of outputs. Doesn't it mean that val_outputs[0] contain segmentation and val_outputs[1] the classification score for all patches?

roi_size = args.image_shape
sw_batch_size = 1
val_outputs = sliding_window_inference(val_inputs, roi_size, sw_batch_size, model)
val_outputs = [post_pred(i) for i in decollate_batch(val_outputs[0])]
val_labels = [post_label(i) for i in decollate_batch(val_labels)]

[wyli]

please make sure the classification head outputs spatial dimensions, hope this gives you some ideas:

import torch
from monai.inferers import sliding_window_inference


class MyModel:
    def __init__(self):
        self.m = 1  # model segmentation param
        self.c = 20  # model classification param

    def __call__(self, x):
        seg = torch.tensor(x + self.m)
        batch = x.shape[0]
        cls = torch.ones((batch, 5, 1, 1)) * self.c  # classification output batch, 5 classes, 2D
        return (seg, cls)

input_volume = torch.ones((1, 2, 200, 200))  # batch size 1, 2 channels, 2D
output = sliding_window_inference(input_volume, roi_size=12, sw_batch_size=4, predictor=MyModel())
print(type(output))
print(output[0].shape)
print(output[1].shape)

output:

<class 'tuple'>
torch.Size([1, 2, 200, 200])
torch.Size([1, 5, 16, 16])

[YazdanSalimi]

Thank you for your help.
I want to use the sliding_window_inference for the classification by the below code.

import monai, torch
model = monai.networks.nets.resnet18(num_classes=2, spatial_dims=2)
input_volume = torch.rand((1, 2, 200, 200)) # batch size 1, 2 channels, 2D
output = monai.inferers.sliding_window_inference(input_volume, roi_size=12, sw_batch_size=4, predictor=model)
print(type(output))
print(output[0].shape)
print(output[1].shape)

I get this error:
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[4, 2, 12, 12] to have 3 channels, but got 2 channels instead

Could you please help to solve the problem?
Thanks

[wyli]

sure, the key idea is to make sure the model can output spatial dimensions such as (batch, channel, 1, 1) for 2D, so this works:

import monai, torch

num_classes = 2
spatial_dims = 2
model = monai.networks.nets.resnet18(num_classes=num_classes, spatial_dims=spatial_dims)

class Wrapper:
    def __init__(self, model):
        self.model = model

    def __call__(self, x):
        return self.model(x).view(-1, num_classes, *([1] * spatial_dims))

input_volume = torch.rand((1, 3, *([200] * spatial_dims)))  # batch size 1, 2 channels, 2D
output = monai.inferers.sliding_window_inference(input_volume, roi_size=12, sw_batch_size=4, predictor=Wrapper(model))
print(type(output))
print(output.shape)

[YazdanSalimi]

Thank you

[jwutsetro]

@sulaimanvesal, we are considering a similar approach to enhance lesion segmentation of very small lesions. What was you experience with this approach? Is there any publication of this work ?

[valecorbetta]

Hi, I'm using monai version 1.1.0 and I'm still getting this error when using the sliding window inferer for multiple outputs. Are there any updates on this feature?

Thanks

[wyli]

this is available in 1.1.0, the example I posted above works fine, could you please elaborate the error? I can have a look.

[valecorbetta]

Hi, sorry, I realized that the error happens not with sliding_window_inference (which works perfectly for multiple outputs), but with SliceInferer when doing inference on a 3D image using a 2D model. Is this feature not implemented for this inference?

[wyli]

yes, with SliceInferer:

import torch

from monai.inferers import SliceInferer


class MyModel:
    def __init__(self):
        self.m = 1  # model segmentation param
        self.c = 20  # model classification param

    def __call__(self, x):
        print(f"input shape {x.shape}")
        seg = torch.as_tensor(x + self.m)
        batch = x.shape[0]
        cls = torch.ones((batch, 5, 1, 1)) * self.c  # classification output batch, 5 classes, 2D
        return (seg, cls)


input_volume = torch.ones((1, 2, 200, 200, 15))  # batch size 1, 2 channels, 3D
output = SliceInferer(roi_size=(200, 200), spatial_dim=2)(input_volume, network=MyModel())
print(type(output))
print(output[0].shape)
print(output[1].shape)

output:

torch.Size([1, 2, 200, 200, 15])
torch.Size([1, 5, 1, 1, 15])