Deepspeed version for PyTorch: 2.4.1?

[ewebgh33]

Hi
I can see an option for
DeepSpeed v14.0 for CUDA 12.1 and Python-3.11.x and PyTorch 2.2.x

But dignostics tells me I have
Python: 3.11.10 PyTorch: 2.4.1+cu121 CUDA: 12.1

So OK except my PyTorch is higher.
This is weird because I did a complete fresh, new install of TextgenWebUI (twgui) yesterday, same with Alltalk v2 Beta. So the Oobabooga (tgwui) should be, I don't know, standard that AllTalk_v2 is made for?
Installed Alltalk v2 as part of tgwui not as standalone.

In this thread,
#199
it's said
"I only build DeepSpeed versions for whatever Text-generation-webui is currently installing as a default installation."
But from a brand new install of tgwui yesterday, this doesn't appear to be the case if tgwui is running with pytorch 2.4.1

Is the recommendation to downgrade pytorch?
Tgwui must have installed with 2.4.1 yesterday for reasons?

Or, would I be better off NOT installing AllTalk as part of tgwui, and instead installing it as standalone?
And does that mean manually starting AllTalk when I start tgwui, or if alltalk_tts is checked in the tgwui session it will somehow auto-start?

Thanks

[erew123]

Hi @ewebgh33 There are 3x complications here:

1. TGWUI has jumped up to the Pytorch 2.4.1 and I have not yet been able to match all version within AllTalk as a integrated install (due to point 3 below).
2. DeepSpeed has to be rebuilt for each Python/Torch version, which is a complicated process, however Microsoft are starting to provide pre-built wheels for DeepSpeed on Windows now. Please see my comments here AllTalk v2 Download Details & Discussion #245 (reply in thread) and further to this, the current wheel MS are releasing have a couple of liitle issues, which I have been in communication with Logan at Microsoft re this Windows wheel build error deepspeedai/DeepSpeed#6465 (comment) so I am confident they will get these resolved at some point.
3. Finally, I am travelling to care for a critically unwell family member, so away from my home machine that I can code, test etc, so limited at the moment as to what changes/updates I can make.

AllTalk v2 is currently aimed at Pytorch 2.2.1 and so the DeepSpeed wheel file is built for that. As such you have a couple of options here:

1. Try the Microsoft wheel, and see it if would work with Pytorch 2.4.1 but it may have issues as mentioend. https://github.com/microsoft/DeepSpeed/blob/master/blogs/windows/08-2024/README.md
2. Build your own DeepSpeed wheel file for Pytorch 2.4.1. I worked/wrote the instruction with S95Sedan on this and their Github contains instructions for DeepSpeed 14 https://github.com/S95Sedan/Deepspeed-Windows
3. Downgrade your Pytorch in TGWUI, BUT this may cause you other issues. Please see here AllTalk v2 Download Details & Discussion #245 (reply in thread)
4. Run AllTalk V2 as a Standalone (not installed as part of TGWUI) see here AllTalk v2 Download Details & Discussion #245 (reply in thread) this would be the simplest solution of all the ones, but needs more disk space. You can use AllTalk with the TGWUI Remote Extension, which will give you all the features, as if it was running as part of TGWUI's Python environement.
5. Dont use DeepSpeed for now/wait until I can get an updated version built OR microsoft fix their build. I cannot say when I will be able to build a new version/test etc, for hopefully obvious reasons, and of course, I dont have any control over when MS will update their release.

Hopefully that gives you a few suggestions and as I say, 4 would be the easiest of all the options.

Thanks

[ewebgh33]

Got it, thanks for the explanation, and best wishes for your family.

Thanks also for being so precise even while travelling. I was expecting a reply from someone in the community around the discussions here actually, as I had seen your note already about travelling.

Those solutions are great for now. If standalone doesn't work for me, I will also try going back to a version of TGWUI from a few months back when they were still using the older pytorch.

Many regards to you and yours