ADN behaviour in ResidualUnit

[danieltudosiu]

Hey :D

I was using ResidualUnit in my network and observed that its outputs are not 0-1 bounded even with norm="layer".

Upon further inspection, I observed that the Convolution on the skip connection does not have the ADN component, nor do we have an ADN component at the end of the residual unit.

From my point of view, this creates ambiguity and can lead to a time sink of debugging since it is not specified in the docs that no ADN is found on the skip connection or at the end of the block and people will assume the output will be well behaved if we have a normalization.

Was this done due to a design choice or is it a mistake?

Thanks in advance!

[Nic-Ma]

Hi @ericspod ,

Could you please help share some comments here?

Thanks.

[ericspod]

With the ResidualUnit class one wouldn't necessarily get unit outputs even with regularization since the residual skip connection is added after the ADN layers. If the stride values are all 1 and the number of channels for the output is the same as the input the skip connection will used nn.Identity as the layer it passes through rather than a convolution, which is used only to resize the connection data to make the actual output so they can be summed. If this is the case then the lack of ADN on the skip connection isn't a problem as this matches the original definition. However I do agree all this should be noted in the documentation for a class.