Compile for Android armeabi-v7a

[idekterev]

New to the space - apologies in advance for the possibly noob question.
Wondering at the possibility of running a quantized 7B model on a 32-bit arm (armeabi-v7a).
Is is possible to compile this for armeabi-v7a?

[BarfingLemurs]

@idekterev most of the time, you need 6-8gb of ram to run 7b on android, there is a way to run 3b and 1b models using the ggml repo, that is pending, which may reduce ram requirements to 3-4gb ram for 4bit models.

https://huggingface.co/replit/replit-code-v1-3b
https://huggingface.co/togethercomputer/RedPajama-INCITE-Chat-3B-v1
https://huggingface.co/mosaicml/mpt-1b-redpajama-200b-dolly

these are some models that could work in the near future

[idekterev]

Thanks - I was not sure of the actual memory requirements, so thanks for clarifying. I will aim for smaller 1B and 3B models, as you suggest.
In terms of compiling llama.cpp for armeabi-v7a, I tried using these parameters (not sure if correct) but the make step failed:

cmake -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=armeabi-v7a -DANDROID_PLATFORM=android-33 -DCMAKE_C_FLAGS=-march=armv7-a+dotprod ..

error:
/llama.cpp/ggml.c:734:11: error: redefinition of 'vcvtnq_s32_f32'
int32x4_t vcvtnq_s32_f32(float32x4_t v) {

[ghost]

Do you need to specify your architecture like that? I'd cmake like this:

cd llama.cpp

cmake -B build

cd build

cmake --build . --config Release

[dermotfix]

check out my comment here #9318 (comment)