Eagerly Awaiting the Megatron Code Release

[mactavish91]

I really appreciate the amazing work you’ve done on the Moonshot AI open source MoE model. I’m excited about the upcoming release of the Megatron code and can't wait to explore it.
Thank you for your efforts!

[toothacher17]

Thanks for the interest! We will publish it ASAP!

[toothacher17]

hi, @mactavish91 Please see the example proof of concept Megatron-LM PR in here:

NVIDIA/Megatron-LM#1428

[huyiwen]

Are there any plab to open source the training code for MoE mentioned in the paper?

[toothacher17]

@huyiwen MoE training code is not related with the Muon optimizer itself, so we do not plan to release it.

[huyiwen]

are the released intermediate checkpoints for the MoE model or the dense model? If they are for the MoE model, how should we load and train them?

[toothacher17]

are the released intermediate checkpoints for the MoE model or the dense model? If they are for the MoE model, how should we load and train them?

They are MoE models:

• for loading, using huggingface transfomrers library: https://huggingface.co/moonshotai
• for training, it is discussed in this issue: The remote code[modeling_deepseek.py] in Moonlight-16B-A3B-Instruct do not support training #11