Noise & Smoothness

[alexjc]

I've noticed that adding smoothness is not required when starting from the original content image, the optimization finds some great images that are smooth where necessary and crisp otherwise. However, when starting from random seed, the result tends to look dithered: pixels of different colors often combine to give the correct look zoomed out, but when zoomed in there's a lot of noise. Adding the smoothing parameter helps, but it effectively blurs the whole output and makes it less crisp than with a content image as seed.

http://i.imgur.com/TKCAdvt.png (smoothing, from random, looks blurred)
http://i.imgur.com/tuIMLOh.png (no smoothing, from random, looks dithered)
http://i.imgur.com/IH5CEcs.png (no smoothing, from image, clean/crisp)

The original paper doesn't seem to mention post-processing (except that they use average pooling), so I'm wondering what's so different that creates these dithered results. Any ideas?

NOTE: When I start with pure noise I changed the current code to use the mean pixel color and usual color range, rather than the current additive approach. Starting with pure noise makes it more reliable to get good solutions that integrate the style well, similarly to L-BFGS.

EDIT: Added images, some details.

[alexjc]

Pondering more about this... If you think of the optimization process as "de-convolution" going backwards through the neural network, is the code currently considering only the middle pixel for each 3x3 filter? This could explain the noise levels... Would it make the results smoother to consider all 3x3 pixels for each deconvolution? This would act like the current smoothing parameter (could be the same command line argument), but instead operating in a context-sensitive way that depends on the specific convolution weights for each pixel.

I don't know the code quite well enough yet, but if you have some pointers I can look into it.

[andersbll]

Hey, I'm occupied at the moment. I'll get back to you in a few days.

[alexjc]

Thanks Anders, there's no rush.

I've made a few more (and better) renderings to try to understand the problem:
http://i.imgur.com/pfMpKqq.png (smoothing, from random)
http://i.imgur.com/s1ZLcex.png (no smoothing, from random)
http://i.imgur.com/Lw1xMuK.png (no smoothing, from image)

With some tuning of the parameters the difference is not as pronounced as before, but I'm curious about how the paper deals with smoothing. Also, with the new crisp versions, I noticed a pixel line on the right and bottom now, despite style and content being 32 pixel multiples. (It doesn't happen consistently.)

[andersbll]

Hey, thank you for bringing up this issue. There is definitely something fishy going on. I played around with the scale of the noise in the initial image and it seems that smaller initial noise leads to smoother images. The reason might be that high noise levels makes it easier for the optimization to end up in a bad local optimum.
I have committed the changes and I think it looks a lot better. Also, the default smoothing level is decreased as this is not as necessary anymore. Let me know what you think.

I think the problem with the border pixels may be a result of the conv filters overlapping with the zero padding. You might want to discard the outermost pixels. :)

[alexjc]

Thanks, Anders. I also tuned down the weight to something similar but it feels like a workaround :-)

I believe the images generated starting from pure noise reflect the fact that there's no penalty in the loss function for discontinuities between neighboring pixels. That why a smoothing pass was necessary, I presume?

When back-propagating the first convolution layer with to_imgs=True would it be possible to adjust all pixels in the 3x3 range not just the center one? Then weight the adjustment to the middle pixel a bit higher, if possible... Ideally this would also happen at all layers. Is this somehow supported?