How to fine-tune the Unidiffusier?

[6Roy]

I currently have a picture and text data set (the picture is a normal picture, while the text length is long, which is more similar to a composition or a long description（about 400 words in Chinese）). I hope to complete the task of generating pictures and texts from each other as you mentioned in the paper, but I don't know whether to directly use your pretrainedmodel or I need to fine-tune it? If I need to fine-tune, how do I do it, because I'm not familiar with the code.I was very lost for a few weeks。。。
Thank you for work and time！ Looking forward to your reply！

[NieShenRuc]

Thank you for your interest.
Our model was trained on large-scale image-text data pairs, but all these texts are in English. Based on your description, you have new data, and all your texts are in Chinese, so you need to finetune our model. For finetuning, you can refer to U-ViT. You only need to modify the model configuration, loss calculation method, model loading, etc. (The training and sampling code of Unidiffuser are simply modified from U-ViT).
Since our model highly depends on the CLIP text encoder, and CLIP has only been trained on English datasets, I must honestly express my doubts about whether our model can perform well in Chinese.
Wish you success.