Finetuning Base Model on chat data should change "eos_token" to "<|im_end|>

[mertunsall]

Reminder

• I have read the above rules and searched the existing issues.

Description

I ran a finetuning job on a pretrained base model such as https://huggingface.co/Qwen/Qwen2.5-Math-7B with chat style data. Then, when I try to do generation with vllm with default settings, the model doesn't stop generating as the "eos_token" parameter of the base model is <|end_of_text|> instead of <|im_end|> (which is the case for the instruct model).

I suggest this should be automatically changed for the finetuned model's config, but I don't know if there is a general way to do this. I believe one can simply change the "eos_token" to whatever was used in the chat template as the end token.

Pull Request

No response