Anybody running an Intel arc 750 or 770?

[RahulVivekNair]

If anyone is running these cards for their vram capacity , what is your experience like? How many iterations/ms are you getting through opencl offloading? Does it work with UI's like oobabooga and is it worth getting one?

[KerfuffleV2]

This is gaming oriented, but it seems like it's about half the speed of a 3090: https://gpu.userbenchmark.com/Compare/Nvidia-RTX-3090-vs-Intel-Arc-A770/4081vsm1850973

It also has 33% less memory. However, it costs about half as much. Assuming the OpenCL performance is in line with the gaming performance, it could possibly make sense to get two of them and use stuff like GGML GPU splitting feature. However, the cards have 250 watt TDP so that's a huge amount of power.

The cheapest one I found was $339, used 3090s are $700-800.

I didn't find anything much about OpenCL usage, but it seems like the card is pretty well supported on Linux.

[Green-Sky]

looks like Intel MKL (BLAS) is supported. So at least prompt ingestion should be good.

[JohannesGaessler]

get two of them and use stuff like GGML GPU splitting feature.

As far as I know OpenCL currently does not support multi GPU.

[ianscrivener]

I checked out the the specs on these recently. Yes, price for performance is very good. Though am I right to assume they are using 50-100% more electricity than an equaivalent nvidia card? I'm SUPER impressed with the performance versus energy of my MacOS M2. The Amazon Inferentia etc ARM chips are also super energy efficient.

[taka-tuos]

I have an A380 (ASRock Challenger) and tried llama.cpp. 7B (vicuna-1.0, Q4_0) produced a terrible result of 240ms/tok. (ArchLinux, E5-2670 v3 with DDR4-2133 32GB)
P104-100 hits 30ms/tok in the same environment. (Slightly unfair comparison: A380 cost 22000 Yen brand new, while P104-100 costs only 3000 Yen because I co-bought a mining rig and took it apart lol)

[taka-tuos]

Thanks for the useful information, I'll try tuning CLBlast on my A380 this weekend.

[DarwinAnim8or]

@taka-tuos did you ever re-try running llama on the a380? curious if it's doable

[taka-tuos]

I'll be trying out SYCL in the next few weeks.
As far as I know, the only 75W, 6GB (that can run 8b Q3_K_M) is the A380, so if it works, I think it's a pretty attractive option.

[DarwinAnim8or]

I'll be trying out SYCL in the next few weeks. As far as I know, the only 75W, 6GB (that can run 8b Q3_K_M) is the A380, so if it works, I think it's a pretty attractive option.

I have an A380 now, and tried to run SYCL on Ubuntu but it spat an error at the end :(

[DarwinAnim8or]

Update-- seems that it's largely dependent on what model you're loading and at what Quant, Q4_K_M works, but Q5_K_S didn't play nice, for example.

[tarunmcom]

Run LLama-2 13B, very fast, Locally on Low Cost Intel's ARC GPU , iGPU and on CPU:- https://youtu.be/FRWy7rzOsRs

[delock]

Hi @tarunmcom from your video I saw you are using A770M and the speed for 13B is quite decent.. I have tuned for A770M in CLBlast but the result runs extermly slow. Also when I try to copy A770 tuning result, the speed to inference llama2 7b model with q5_M is not very high (around 5 tokens/s), which is even slower than using 6 Intel 12gen CPU P cores.

Are you using A770 tunning result in CLBlast or do you tune for A770M by yourself? would like to know whether the tunning result can be shared.

[NeoZhangJianyu]

@delock
You could try with llama.cpp for SYCL backend, refer to https://github.com/ggerganov/llama.cpp/blob/master/docs/backend/SYCL.md
It's faster than CLBlast on Intel GPU.

[HumerousGorgon]

Using the sycl backend, whenever it tries to allocate >4GB of VRAM to my A770, the model fails loading. Trying to load anything higher than Q4 means a failure, even though I have the available VRAM for it. Any idea on how to solve this?

[NeoZhangJianyu]

Please share the whole log, including the cmd line.
Please report an issue, title with [SYCL].
I will monitor the issue in general.

[HumerousGorgon]

@NeoZhangJianyu
Just posted my error with relevant logs. thank you.

[tarunmcom]

here is video how to run using sycl its father than CLBLAST :- https://www.youtube.com/watch?v=Q7t4CmziaqA

…

On Mon, 19 Aug 2024 at 07:25, Neo Zhang Jianyu ***@***.***> wrote: @delock <https://github.com/delock> You could try with llama.cpp for SYCL backend, refer to https://github.com/ggerganov/llama.cpp/blob/master/docs/backend/SYCL.md It's faster than CLBlast on Intel GPU. — Reply to this email directly, view it on GitHub <#1923 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFKFRVK4HWROL7SE6OXRLQLZSFGAXAVCNFSM6AAAAABL64FVKCVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZXG4ZDEOA> . You are receiving this because you were mentioned.Message ID: ***@***.***>