Maximum Model Size for M2 Ultra with 196 GB RAM

[gileneusz]

I'm currently exploring the capabilities of the M2 Ultra and its 192 GB RAM configuration. I've read that it's possible to fit the Llama 2 70B model. However, I'm curious if this is the upper limit or if it's feasible to fit even larger models within this memory capacity.

Any insights or experiences regarding the maximum model size (in terms of parameters) that can comfortably fit within the 192 GB RAM would be greatly appreciated.

Thank you in advance!

edit: 192 GB 😅

[psugihara]

That rig will be awesome. If you look at TheBloke's GGUF conversion descriptions (like this one for Llama 2 70B Chat), they say how much RAM is required for different quantization levels. You should be able to run the big models with that RAM with space left over (perhaps for a draft model for speculative sampling when that gets integrated more widely).

[ggerganov]

I've recently demonstrated Q8 LLaMAv2 70B running on M2 Ultra 192GB at about ~8 t/s with Metal inference.
It takes about 80GB of your unified memory

Note that Metal can access only ~155GB of the total 192GB (more info).
So any model that is smaller than ~140GB should work OK for most use cases.

You can access all 192GB with the CPU (i.e. without Metal), but this is significantly slower.

These are some objective numbers, valid only about llama.cpp.
Let me know if you need more specific information about something.

---

In case you are looking for a more subjective and potentially biased opinion, here is mine:

The M2 Ultra is the absolute best personal LLM inference node that you can buy today.
Just max out out the spec if you can afford it.

But again - this is the opinion of an Apple "fanboy", so take it with a grain of salt 😄

[digicali]

Have you heard about macweb.com? they have M2 Ultra configurations for testing

[crasm]

I have a 128 GB M2 Ultra (60 core). I can run Falcon 180B at Q3_K_M quantization (80GB), without needing to resort to patching the VRAM split. With 192 GB, you could probably run up to Falcon 180B @ Q6_K (148 GB).

[EthanLipnik]

How fast did it run for you? If the 70B LLaMA 2 model was 8t/s I can't imagine the 180B Falcon model being very usable.

[crasm]

Around 5 tok/s. Definitely fast enough for interactive use.

[EthanLipnik]

That is extremely promising, thanks!

[aifirstd3v]

In my opinion, the Apple M-series is definitely the best, most stable, quiet, and low-electricity LLM machine for testing LLM locally. Of course, it cannot be expanded as freely as nvidia, but at least it is very stable for developers who use Mac as their main weapon.

[YuanBoXie]

Will the inference speed of Mac M2 Ultra be slow after running LLM?