Linux releases changing system requirements between releases (1.13.14)

[sshanks-kx]

Describe the bug

The pre-build linux releases (e.g. https://github.com/eclipse-paho/paho.mqtt.c/releases/download/v1.3.14/Eclipse-Paho-MQTT-C-1.3.14-Linux.tar.gz) are altering what systems they support between releases (by changing the requirements of which libc they need to run)

The latest has jumped in its requirement for its version of libc to a very recent version, so it cant be run on many default setups of different linux operating systems (prev versions could).

Why this matters:

Various production environments run on particular deploys (e.g. Redhat supported RedHat Enterprise 8, etc).
They cant update their versions of libc just because of an mqtt lib (effects other deployed software on same machine, cant be updated promptly without going through various checks, etc).
People can find the new versions will suddenly fail at runtime. Even when they might be able to upgrade, it may not be immediately clear of why/what they need to do.

To Reproduce

example dependancy check on version 1.3.13 (no dependancy on libc2.38)
[root@5c1b9a566c03 src]# nm -C ./Eclipse-Paho-MQTT-C-1.3.13-Linux/lib/libpaho-mqtt3c.so.1 | grep 'GLIBC_2.38'
example dependancy check on version 1.3.14 (dependancy on libc2.38)
[root@5c1b9a566c03 src]# nm -C ./Eclipse-Paho-MQTT-C-1.3.14-Linux/lib/libpaho-mqtt3c.so.1 | grep 'GLIBC_2.38'
U __isoc23_strtol@GLIBC_2.38

Some example RedHat/Centos libc versions
redhat/centos 7.9
glibc-2.17-326.el7_9.3.x86_64
redhat/centos 9.3
glibc-2.34-83.el9.7.x86_64

Therefore can now get this sort of error when running on rh7.9 with 1.13.14, which didn't occur with 1.13.13
'/lib64/libc.so.6: version `GLIBC_2.38' not found (required by /source/paho.mqtt.c/lib/libpaho-mqtt3c.so.1)

Expected behavior
To support form redhat7 onwards (e.g. systems supported by versions prior to 1.3.14) to have more system support on which the software can run (unless theres a real requirement/fix that needs a newer version).

Screenshots
N/A

Log files
N/A

** Environment (please complete the following information):**
See above

Additional context

I have not checked how the binaries are built deployed.
If this is not an intended action, it could be due to using a setting of a latest box/os on a ci/cd setup, on which the ci/cd environment could be randomly changing the os/etc on which they are build between releases, and hence changing the run requirements between releases (rather than be hardcoded to a certain build box).

If the previous build boxes/etc cant be reinstated, it would be helpful if the release notes of how a mqtt release has changed its requirement for the pre-build binaries.

Thanks

[sshanks-kx]

this might be the related change that caused the issue
1b43176
would be helpful change was reverted (or a potentially older version to improve deployment support) & if it wasnt on a potentially build version between releases, where its currently only supporting the latest version of ubuntu

[fpagliughi]

Standard support for Ubuntu 18.04 ended nearly two years ago and, I believe, is no longer available as a target for GitHub actions.

So I assume that was the reason for the build platform being changed.

The glibc version issue is notorious, and holding it back would probably create similar issues for people on different platforms. I don't think the pre-built binaries on a repository are meant to substitute for universal package management.

I agree that this is unfortunate, but whatever simple solution works for someone would probably not work for someone else.

[jumoog]

Yeah, I don't understand why we even ship a pre-built Linux release. In most cases you have to build them yourself anyway.

[sshanks-kx]

Pre-built binaries are extremely useful (e.g. wanting to test an mqtt installation with paho). Not everyone has the ability to install development tools, or just want to quickly try it , or they might not have the knowledge to build it successfully (though it is documented). Not using them as a substitute for package management (may be a user at home, or a user in a big organisation with different resources/needs/etc)

I dont understand how having increased compatibility would create issues for others (more likely, the opposite would be true). Its only notorious when using builds created on newer versions but trying to run on older versions, which can give a cryptic error at runtime. Even more so when not documented or suddenly changes between releases. Its been changed now to create more issues for users (rather than the opposite).

There are tools such as https://github.com/wheybags/glibc_version_header/ to enable better compatibility when you must build on newer setups while retaining increased compatibility. Or you could just build on something that isnt the very latest (e.g. a useful 'meet in the middle' is to use the oldest currently supported, rather than the very latest).

[icraggs]

I only expected the pre-built packages to be helpful in some cases on Linux and MacOS and that people would generally build the libraries themselves. A little different for Windows where it can be more tricky to build, so I thought those would be more useful.

The main reason for changing the build version in Github Actions is as @fpagliughi suggested, that Github removes support for older OSes from time to time. Ubuntu 18.04 was removed - the current list is here: https://github.com/actions/runner-images. I change to "latest" to avoid having to migrate the builds in the future.

The primary reason for these builds is to run the CI tests, the artifacts are just an effect of that. I agree that it would be good to document the build environment of the binaries.

Providing pre-built binaries for older environments would be nice, but I'd have to have access to stable environments to build on. I can think of a variety of options, but seems like quite a bit of work whatever.

[sshanks-kx]

Would prefer if it built with ubuntu-20.04 (GLIBC 2.31) to give the wider range of support rather than ubuntu-latest.
Also given that latest could change at any point. If using ubuntu-20.04 , can then detect when it goes away (builds break) and document on the release notes, listing the change in dependancy.
Currently the dependancies have the potential to change with each release (if GitHub changes what ubuntu-latest points to) .. potentially unknowingly unless monitoring the changes to that 'latest' placeholder, and an unsuspecting user will run into this same issue (as opposed to reading about the change in release notes).