We follow @zhang2016salient in pretraining our model with synthetic images and later fine-tuning on the SOS dataset. However, some small chances to their synthetic image pretraining setup are proposed. First, the synthetic dataset is extened with natural images from the SOS dataset such that the amount of examples per class per epoch is always equal (hopefully reducing problems encountered with class imbalance, see [section X.X](#imbalance)). Another reason for constructing a hybrid dataset was the fact that the generation process of synthetic images was noted to produes 1. fairly unrealistic looking examples and 2. considerably less suitable than natural data for supporting subitizing performance [@zhang2016salient]. A further intuition behind this dataset is thus that the representation of the VAE must always be at least a little supportive of natural images, instead of settling on some optimum for synthetic images. A final reason for including natural images is that any tested growth in dataset size during pretraining resulted into lower losses. The ratio of natural to synthethic images is increased over time, defined by a bezier curve with parameters $u_0=0, u_1=-0.01, u_2=0.02, u_3=1.0$ shown in figure X.X. We grow the original dataset into a hybrid dataset by roughly 8 times, pretraining with a total of 80000 hybrid samples per epoch. Testing many different parameters for the hybrid dataset was not given much priority as the total loss seemed to shrink with dataset expansion and training and testing a full model was time expansive.
0 commit comments