Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Android / Java build fails on CentOS 7 #2231

Open
stephan57160 opened this issue Sep 6, 2022 · 2 comments
Open

Android / Java build fails on CentOS 7 #2231

stephan57160 opened this issue Sep 6, 2022 · 2 comments

Comments

@stephan57160
Copy link
Contributor

stephan57160 commented Sep 6, 2022

Was trying to setup something to use BUILD_PREFIX and have zyre/czmq/libzmq dependencies installed somewhere else than under
/usr/local/lib

Finally, I come to this:

export BUILD_PREFIX=${PWD}/build-native

Next:

export PKG_CONFIG_PATH=${BUILD_PREFIX}/lib/pkgconfig
export LIBZMQ_ROOT=${PWD}/libzmq
export CZMQ_ROOT=${PWD}/czmq
export ZYRE_ROOT=${PWD}/zyre
...

Then, for libzmq, czmq, zyre and all other dependencies:

./autogen.sh
./configure \
	--prefix=${BUILD_PREFIX} \
	PKG_CONFIG_PATH=${PKG_CONFIG_PATH}
make
make install

make install will install all of them under ${BUILD_PREFIX}, instead of
/usr/local.

With this, Android build fails:

prompt> cd zyre/bindings/jni/zyre-jni/android
prompt> ./build.sh arm
...
:czmq-jni-native:copyLibs (Thread[Execution worker for ':',5,main]) completed. Took 0.054 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':czmq-jni-native:copyLibs'.
> Entry libcurl.so is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/7.4.2/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --debug option to get more log output.
> Run with --scan to get full insights.

prompt>

I googled a bit and found this happens frequently since Gradle 7:
gradle/gradle#17236

A workaround comes with adding something like duplicatesStrategy = DuplicatesStrategy.WARN,
in a build.gradle.

I had a deeper look and it seems that on my system, java.library.path is set to /usr/lib64 and /lib64 ...
Unfortunatelly, on recent RH-like systems:

prompt> ls -ald /lib64
lrwxrwxrwx. 1 root root 9 12 janv.  2020 /lib64 -> usr/lib64
prompt> 

Now, I have the duplicate search path...
libcurl.so is visible from /usr/lib64 and from /lib64...

Question:
The workaround works for me, and I could raise a PR, only for this.

But when I read this:
https://github.com/zeromq/czmq/blob/master/bindings/jni/czmq-jni-native/build.gradle#L15-L62
I think it would be a better idea to search for dependencies in BUILD_PREFIX first, then in system paths (in case one wants his
own dependency version, rather than the system one).

With previous versions of Gradle, I don't know what was happening:

  • override existing libs (then the dependencies installed in BUILD_PREFIX were overriding system ones).
  • silently skip existing libs (like with this workaround).

Your opinion on that ?
(changes to be made in ZProject too, I suppose)

@stephan57160
Copy link
Contributor Author

Regarding only the duplicatesStrategy, I come with this modification, in https://github.com/zeromq/czmq/blob/master/bindings/jni/czmq-jni-native/build.gradle#L31-L61:

    def oldStrategy = duplicatesStrategy                     /* Backup */
    duplicatesStrategy = DuplicatesStrategy.WARN             /* Workaround, for duplicate library entries */
    libraryPaths.each { path ->
        from path
            include 'libczmqjni.so'
            include 'libczmqjni.dylib'
...
            include 'libmicrohttpd.dylib'
            include '*microhttpd*.dll'
        into 'build/natives'
    }
    duplicatesStrategy = oldStrategy                         /* Restore */

The Backup + Restore operation is there to keep existing behaviour on the rest ...
... in case ...

@stephan57160
Copy link
Contributor Author

Regarding the local dependencies, I come with this, in the same file:

    def libraryPaths = []
    if (project.hasProperty('buildPrefix')) {
        if (osdetector.os == 'windows') {
            // DLLs are installed to the bin directory by cmake
            libraryPaths.add("${project.buildPrefix}/bin")
        }
        libraryPaths.add("${project.buildPrefix}/lib")
    }

    def javaLibraryPaths = System.getProperty('java.library.path').split(File.pathSeparator).toList()
    libraryPaths.addAll(javaLibraryPaths)

    libraryPaths.add('/usr/local/lib')
    if (osdetector.os == 'windows') {
        libraryPaths.add("${rootDir}/czmq-jni/build/Release")
    } else {
        libraryPaths.add("${rootDir}/czmq-jni/build")
    }

A couple of system.out.println() were added to validate this. No other way, for me, so far.

stephan57160 added a commit to stephan57160/zproject that referenced this issue Sep 19, 2022
ZYRE compilation started for Android, with a similar
script to zyre/bingings/jni/ci_build.sh.

Compilation ends with:
```
> Task :czmq-jni-native:classes UP-TO-DATE
Skipping task ':czmq-jni-native:classes' as it has no actions.
:czmq-jni-native:classes (Thread[Execution worker for ':',5,main]) completed. Took 0.0 secs.
:czmq-jni-native:copyLibs (Thread[Execution worker for ':',5,main]) started.

> Task :czmq-jni-native:copyLibs FAILED
Caching disabled for task ':czmq-jni-native:copyLibs' because:
  Build cache is disabled
Task ':czmq-jni-native:copyLibs' is not up-to-date because:
  No history is available.
file or directory '/usr/java/packages/lib', not found
:czmq-jni-native:copyLibs (Thread[Execution worker for ':',5,main]) completed. Took 0.062 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':czmq-jni-native:copyLibs'.
> Entry libcurl.so is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/7.4.2/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org
```

Investigation have shown 2 things:
- On CentOS, /lib and /lib64 point to /usr/lib and /usr/lib64:
    ```
    prompt> ls -ald /lib /lib64
    lrwxrwxrwx. 1 root root 7 12 janv.  2020 /lib -> usr/lib
    lrwxrwxrwx. 1 root root 9 12 janv.  2020 /lib64 -> usr/lib64
    prompt>
    ```
- Since Gradle 7, one has to set a duplicate strategy, when using copyLibs:
    gradle/gradle#17236

Solution: Set duplicate strategy to warn, not to stop the build.

Additionnally, add libraries found in $BUILD_PREFIX first,
then, the system ones, in case one wants to have his own library
version in the JAR.

This fixes CZMQ issue zeromq/czmq#2231
stephan57160 added a commit to stephan57160/zproject that referenced this issue Oct 6, 2022
ZYRE compilation started for Android, with a similar
script to zyre/bingings/jni/ci_build.sh.

Compilation ends with:
```
> Task :czmq-jni-native:classes UP-TO-DATE
Skipping task ':czmq-jni-native:classes' as it has no actions.
:czmq-jni-native:classes (Thread[Execution worker for ':',5,main]) completed. Took 0.0 secs.
:czmq-jni-native:copyLibs (Thread[Execution worker for ':',5,main]) started.

> Task :czmq-jni-native:copyLibs FAILED
Caching disabled for task ':czmq-jni-native:copyLibs' because:
  Build cache is disabled
Task ':czmq-jni-native:copyLibs' is not up-to-date because:
  No history is available.
file or directory '/usr/java/packages/lib', not found
:czmq-jni-native:copyLibs (Thread[Execution worker for ':',5,main]) completed. Took 0.062 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':czmq-jni-native:copyLibs'.
> Entry libcurl.so is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/7.4.2/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org
```

Investigation have shown 2 things:
- On CentOS, /lib and /lib64 point to /usr/lib and /usr/lib64:
    ```
    prompt> ls -ald /lib /lib64
    lrwxrwxrwx. 1 root root 7 12 janv.  2020 /lib -> usr/lib
    lrwxrwxrwx. 1 root root 9 12 janv.  2020 /lib64 -> usr/lib64
    prompt>
    ```
- Since Gradle 7, one has to set a duplicate strategy, when using copyLibs:
    gradle/gradle#17236

Solution: Set duplicate strategy to warn, not to stop the build.

Additionnally, add libraries found in $BUILD_PREFIX first,
then, the system ones, in case one wants to have his own library
version in the JAR.

This fixes CZMQ issue zeromq/czmq#2231
stephan57160 added a commit to stephan57160/zproject that referenced this issue Oct 11, 2022
ZYRE compilation started for Android, with a similar
script to zyre/bingings/jni/ci_build.sh.

Compilation ends with:
```
> Task :czmq-jni-native:classes UP-TO-DATE
Skipping task ':czmq-jni-native:classes' as it has no actions.
:czmq-jni-native:classes (Thread[Execution worker for ':',5,main]) completed. Took 0.0 secs.
:czmq-jni-native:copyLibs (Thread[Execution worker for ':',5,main]) started.

> Task :czmq-jni-native:copyLibs FAILED
Caching disabled for task ':czmq-jni-native:copyLibs' because:
  Build cache is disabled
Task ':czmq-jni-native:copyLibs' is not up-to-date because:
  No history is available.
file or directory '/usr/java/packages/lib', not found
:czmq-jni-native:copyLibs (Thread[Execution worker for ':',5,main]) completed. Took 0.062 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':czmq-jni-native:copyLibs'.
> Entry libcurl.so is a duplicate but no duplicate handling strategy has been set. Please refer to https://docs.gradle.org/7.4.2/dsl/org.gradle.api.tasks.Copy.html#org.gradle.api.tasks.Copy:duplicatesStrategy for details.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org
```

Investigation have shown 2 things:
- On CentOS, /lib and /lib64 point to /usr/lib and /usr/lib64:
    ```
    prompt> ls -ald /lib /lib64
    lrwxrwxrwx. 1 root root 7 12 janv.  2020 /lib -> usr/lib
    lrwxrwxrwx. 1 root root 9 12 janv.  2020 /lib64 -> usr/lib64
    prompt>
    ```
- Since Gradle 7, one has to set a duplicate strategy, when using copyLibs:
    gradle/gradle#17236

Solution: Set duplicate strategy to warn, not to stop the build.

Additionnally, add libraries found in $BUILD_PREFIX first,
then, the system ones, in case one wants to have his own library
version in the JAR.

This fixes CZMQ issue zeromq/czmq#2231
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant