[Build] Possibly unintentional or misconfigured dependencies for QNN EP in onnxruntime_python.cmake

[deivard]

Describe the issue

Hi,

ORT cannot be built with QNN EP if onnxruntime_BUILD_UNIT_TESTS=OFF and enable onnxruntime_ENABLE_PYTHON=ON for releases with version >= v.1.20.0. I believe this is unintentional behavior caused by a dependency to the target onnxruntime_qnn_ctx_gen.

The error occurs in the add_custom_command below:

onnxruntime/cmake/onnxruntime_python.cmake

Lines 1037 to 1042 in 5c1b7cc

	add_custom_command(
	TARGET onnxruntime_pybind11_state POST_BUILD
	COMMAND ${CMAKE_COMMAND} -E copy
	$<TARGET_FILE:onnxruntime_qnn_ctx_gen>
	$<TARGET_FILE_DIR:${build_output_target}>/onnxruntime/capi/
	)

I believe the reason why this target doesn't exist when building with unit tests disabled, is because it is only created in the onnxruntime_unittests.cmake here:

onnxruntime/cmake/onnxruntime_unittests.cmake

Line 1321 in 5c1b7cc

onnxruntime_add_executable(onnxruntime_qnn_ctx_gen ${onnxruntime_qnn_ctx_gen_src})

I am using CMake instead of the build scripts since I am cross compiling from amd64 Linux to Aarch64 Linux with BitBake.

Urgency

Not urgent from my side, I will just build with unit tests enabled. But projects that are upgrading to version 1.20+ might notice breaking changes.

Target platform

AArch64

Build script

CMake build script with the following options:

-Donnxruntime_CROSS_COMPILING=ON \
-Donnxruntime_BUILD_SHARED_LIB=ON \
-Donnxruntime_USE_QNN=ON \
-Donnxruntime_BUILD_UNIT_TESTS=OFF \
-Donnxruntime_ENABLE_PYTHON=ON \
-DCMAKE_BUILD_TYPE=RelWithDebInfo

Error / output

CMake Error at onnxruntime_python.cmake:1037 (add_custom_command):                                                                            Error evaluating generator 
     $<TARGET_FILE:onnxruntime_qnn_ctx_gen
  
     No target "onnxruntime_qnn_ctx_gen"

Visual Studio Version

No response

GCC / Compiler Version

No response

[snnn]

If I remember correctly, the "onnxruntime_test_all" has a POST_BUILD step that copies test data to the build directory, and to build onnxruntime you may need some of the test data. Therefore onnxruntime_BUILD_UNIT_TESTS=ON is required for building python. I am not sure if it is still the case.

[deivard]

Interesting. Do you know if it is possible to build the tests without them automatically running when not using the build.py script? I am not sure but it feels that onnxruntime_RUN_ONNX_TESTS=OFF does not make any difference when set. It is hard for me to see if the tests running is causing my BitBake recipe to fail. Is there an easy way to see in the console output if the tests are running?

[snnn]

pass --update --build to build.py

[deivard]

@snnn, I am unable to use build.py since I am compiling for a Qualcomm Aarch64 platform with Yocto. Do you know which CMake options can replace the --update --build arguments?

[edgchen1]

Just calling cmake to generate the build system (equivalent to build.py --update) or cmake --build ... to build the code (equivalent to build.py --build) will not run the tests.

[deivard]

Thank you for your replies, they have been helpful to me!

I have now figured out that my current build error is caused by the cmake -E copy command that is created here (see line 1047):

onnxruntime/cmake/onnxruntime_unittests.cmake

Lines 1043 to 1049 in cff0ec5

	if (onnxruntime_USE_QNN)
	if (MSVC OR ${CMAKE_SYSTEM_NAME} STREQUAL "Linux")
	add_custom_command(
	TARGET ${test_data_target} POST_BUILD
	COMMAND ${CMAKE_COMMAND} -E copy ${QNN_LIB_FILES} $<TARGET_FILE_DIR:${test_data_target}>
	)
	endif()

If ${QNN_LIB_FILES} is empty, the build will result in an obscure and confusing error due to cmake -E copy missing one of the arguments. Since I am using QNN EP, I must pass the -Donnxruntime_QNN_HOME variable. I was at fault here since the documentation for building with QNN EP support states that --qnn_home=[QNN_SDK path] must be set in the build script. Of course, this also applies when using CMake to build.

However, perhaps it would be sensible to make the CMake script fail if the QNN_HOME variable is empty and USE_QNN is ON.

Let's not digress too far from the topic of this issue: Configuration dependency. If you think it is OK that this new dependency I mentioned is introduced, perhaps we should close this issue, if not, shall we leave it open to allow for more discussions?

[deivard]

A suggestion would be to add a check here to see if the onnxruntime_QNN_HOME is empty or if the path doesn't exist:

onnxruntime/cmake/CMakeLists.txt

Lines 895 to 907 in cff0ec5

	if (MSVC OR ${CMAKE_SYSTEM_NAME} STREQUAL "Linux")
	file(GLOB QNN_LIB_FILES LIST_DIRECTORIES false "${onnxruntime_QNN_HOME}/lib/${QNN_ARCH_ABI}/libQnn*.so"
	"${onnxruntime_QNN_HOME}/lib/${QNN_ARCH_ABI}/Qnn*.dll"
	"${onnxruntime_QNN_HOME}/lib/${QNN_ARCH_ABI}/libHtpPrepare.so"
	"${onnxruntime_QNN_HOME}/lib/${QNN_ARCH_ABI}/HtpPrepare.dll")
	if (${QNN_ARCH_ABI} STREQUAL "aarch64-windows-msvc" OR ${QNN_ARCH_ABI} STREQUAL "arm64x-windows-msvc")
	file(GLOB EXTRA_HTP_LIB LIST_DIRECTORIES false "${onnxruntime_QNN_HOME}/lib/hexagon-v68/unsigned/libQnnHtpV68Skel.so"
	"${onnxruntime_QNN_HOME}/lib/hexagon-v73/unsigned/libQnnHtpV73Skel.so"
	"${onnxruntime_QNN_HOME}/lib/hexagon-v73/unsigned/libqnnhtpv73.cat")
	list(APPEND QNN_LIB_FILES ${EXTRA_HTP_LIB})
	endif()
	message(STATUS "QNN lib files: " ${QNN_LIB_FILES})
	endif()