Should there be a permute transformation?

[mattwarkentin]

Hi @fepegar,

I may be mis-understanding something but here is the possible issue I'm facing...

When loading a CT scan from disk as a image = torchio.ScalarImage(...) object, the image dimensions are Channels x Height x Width x Depth, where the term depth is interchangeable with axial slices in my case. So the image.shape is (1, H, W, D).

However, when doing 3D convolution with torch.nn.Conv3D the expectation is that the tensor has dimensions (ignoring the batch dimension) (Channels, Depth, Height, Width). So, since I consider the Z dimension (axial) as the Depth, I would need to permute my tensor before running convolution. Do you agree?

I suppose which axes correspond to "height", "width", and "depth" is context dependent.

If so, should there be a Permute() spatial transform implemented with torch.Tensor.permute? It would be relatively straightforward to implement and I would again be happy to contribute.

[fepegar]

I would say the concept of permutation is typically replaced by reorientation in medical imaging. Have you taken a look at ToCanonical?

In [1]: import torchio as tio

In [2]: t1 = tio.datasets.FPG().t1

In [3]: t1.shape, t1.orientation
Out[3]: ((1, 256, 256, 176), ('P', 'I', 'R'))

Thankfully, we can think anatomically to describe better-defined axes names than depth, height, etc. It is typical to normalize all images to have the same orientation during training, normally RAS (I guess sometimes LPS). Instead of figuring out permutations, flipping, etc., you can use ToCanonical to transform all images to RAS, so all dimensions mean the same across the dataset.

In [4]: to_ras = tio.ToCanonical()

In [5]: reoriented = to_ras(t1)  # data is permuted

In [6]: reoriented.shape, reoriented.orientation
Out[6]: ((1, 176, 256, 256), ('R', 'A', 'S'))

What do you think?

More info on NiBabel docs: Image voxel orientation

[fepegar]

Or does this distinction not matter from a practical sense. I may just be overthinking things...

I think this is the case :)

I suppose all that really matters is that all of the arrays are in the same orientation so that they mean the same thing - but I am just wondering if the order of the spatial dimensions matters from a training perspective.

Exactly. torch.nn.Conv3d doesn't care about the meaning of each dimension, the weights in the convolution kernel are going to be (carefully) random anyway, and there's no conception of up, down, left, right, etc. It only matters for pretraining models for example, but then again all the images were (probably, hopefully) oriented the same way during training and preprocessing instructions are provided for inference as well.

[fepegar]

I think these height, weight, depth names are mostly to help computer vision (2D) people. They are less meaningful to us.

[fepegar]

By the way, do you think orientation is confusing? So do the FSL people, who know a bit about image processing:

[Orientation] is intrinsically a confusing issue

[mattwarkentin]

Glad to see it's not a me issue.

[fepegar]

No. Welcome to the club.