Use BinaryBuilder & BinaryProvider

[giordano]

After #15 is fixed, would it make sense to build ROOT with BinaryBuilder.jl and install the dependency here with BinaryProvider.jl? I'll probably be able to have a look at this if you're interested

[oschulz]

I think so, yes. Having ROOT installed automatically (only if not present on PATH, so people can choose a specific build) was always kind of in the back of my mind. Would also help with CI testing. Would it make sense to use the official CERN builds where applicable?

[giordano]

The way BinaryBuilder + BinaryProvider work is that one builds from source the library for all platforms (or at least those for which they succeed to build the library), currently:

julia> using BinaryBuilder

julia> supported_platforms()
13-element Array{Platform,1}:
 Linux(:i686, libc=:glibc)                    
 Linux(:x86_64, libc=:glibc)                  
 Linux(:aarch64, libc=:glibc)                 
 Linux(:armv7l, libc=:glibc, call_abi=:eabihf)
 Linux(:powerpc64le, libc=:glibc)             
 Linux(:i686, libc=:musl)                     
 Linux(:x86_64, libc=:musl)                   
 Linux(:aarch64, libc=:musl)                  
 Linux(:armv7l, libc=:musl, call_abi=:eabihf) 
 MacOS(:x86_64)                               
 FreeBSD(:x86_64)                             
 Windows(:i686)                               
 Windows(:x86_64)

Then they upload the resulting artifacts somewhere (usually a release page in a GitHub repository) and BinaryProvider downloads and unpacks the tarball for the desired platform.

only if not present on PATH

The idea of BinaryProvider is to always install the libraries in this way, not "only if not already available". I can see that this is not really ideal, but it has its advantages in terms of reproducibility (a given version of the Julia package is linked to a specific version of the binary library it uses) and ease of installation (you're almost sure that the library will be installed correctly)

Would it make sense to use the official CERN builds where applicable?

Well, theoretically we could ignore not use BinaryBuilder to build the package and just reuse the prebuilt tarballs on CERN's website. The only problem is that BinaryProvider expects the archive to have a specific structure (e.g., no top directory), so we couldn't directly use the official tarballs as they are because they have a "Root" top directory.

---

I just realised that actually you build a ROOT-compatible Julia, rather than a Julia-compatible ROOT. Perhaps @staticfloat might be interested in this use case.

[staticfloat]

The only problem is that BinaryProvider expects the archive to have a specific structure (e.g., no top directory)

Note that this can be worked around by setting up the products in your build.jl properly, but by default, yes this is true. :) (these little tricks abound!)

I just realised that actually you build a ROOT-compatible Julia

What needs to be special about Julia in order to work with ROOT?

[oschulz]

What needs to be special about Julia in order to work with ROOT?

ROOT also uses LLVM, and it uses a specially modified version.

ROOT is unhappy if it finds an active LLVM instance when it starts - it will print a warning, and dependent on the current LLVM version in ROOT and in Julia, you get segfaults and crashes (or not, for some combination of LLVM versions). The workaround ROOT.jl uses (@jpata came up with it) is to build a special julia executable that loads ROOT before it starts Julia. I tried other approaches (like LD_PRELOAD), but that's the only way that has worked consistently so far.

I'm pretty sure this is ROOT's fault, not Julia's.

[staticfloat]

Would it be helpful to try building ROOT against the exact version of LLVM that Julia uses? Julia is also quite opinionated about the LLVM it uses, as I'm sure you're aware. ;)

If you think that might help, you can get prebuilt tarballs of the versions of LLVM that Julia uses from here; where each of those tarballs is compiled for a different platform, linked against different compiler support libraries (e.g. if you wanted to ROOT on an x86_64-linux-gnu machine that is using GCC 4, you'd use this tarball).

[oschulz]

I just realised that actually you build a ROOT-compatible Julia, rather than a Julia-compatible ROOT.

That would be interesting - if we could find a way to do that in a clean fashion, I think we could even get the required changes merged into ROOT.

However, I don't know how tricky this is - I never looks into what exactly goes wrong there, deep down, and I'm not an LLVM expert, so I can't judge, really.

[oschulz]

Would it be helpful to try building ROOT against the exact version of LLVM that Julia uses? Julia is also quite opinionated about the LLVM it uses, as I'm sure you're aware. ;)

Yes, I know. :-) There are, however, some problems here:

• I'm not sure if in the instances where things did work (despite ROOT's warning), the LLVM versions were very similar or very different. We can try though.

• ROOT's LLVM is modified for ROOT's own C++ REPL and some other things. So I don't think we could build ROOT against the Julia's LLVM binary. It may be enough if we use the same LLVM base version, though.

• AFAIK specific ROOT versions only work with specific LLVM versions. This means that we'd have to find a ROOT version that brings the right LLVM version with it (and there might not be one, if the match has to be exact). Also, physicists often need to (or are required to) work with a specific ROOT version that is kept fixed for a long time, esp. in larger collaborations. Depending on the environment, that ROOT version may also need to be built with specific options.

Still, it may be workable, but it might get tricky.

[oschulz]

@staticfloat, If you're interested to take a look at this, I can put together a short Julia script that will load Cxx and ROOT and demonstrate the warning and the crash (without ROOT.jl, even).

[oschulz]

Here's a script that will demonstrate the problem:

const ROOT_INCDIR = String(strip(read(`root-config --incdir`, String)))
const ROOT_LIBDIR = String(strip(read(`root-config --libdir`, String)))
const SHEXT = "so"

using Cxx
using Libdl: Libdl

function load_rootlib(libname)
    libpath = joinpath(ROOT_LIBDIR, "lib$libname.$SHEXT")
    isfile(libpath) || error("could not find ROOT library $libname at \"$libpath\"")
    Libdl.dlopen(libpath, Libdl.RTLD_GLOBAL)
end


addHeaderDir(ROOT_INCDIR, kind=C_System)

load_rootlib.([
    "Core", "Gpad", "Graf", "Graf3d", "Hist", "MathCore", "Matrix", "MultiProc", "Net",
    "Physics", "Postscript", "Rint", "RIO", "Thread", "Tree"
])


cxxinclude("TFile.h")

f = icxx"""new TFile("out.root", "recreate");"""

This results in

Error in <UnknownClass::InitInterpreter()>: LLVM SYMBOLS ARE EXPOSED TO CLING! This will cause problems; please hide them or dlopen() them after the call to TROOT::InitInterpreter()!

 *** Break *** segmentation violation
 [...]

(tested just now on CentOS-7 with Julia v1.2.0-rc1.0 and ROOT 6.16/00, but observed with many combinations of Julia and ROOT versions in the past).

To run the script about, ROOT's "bin" directory, containing root-config, but be on $PATH.

Official binary builds of ROOT for Ubuntu, CentOS, Fedora, OS-X and Windows are available here: https://root.cern.ch/downloading-root

[StefanKarpinski]

I wonder if it's possible to build an LLVM that is compatible with both Julia and ROOT and then arrange for that to be the LLVM that Julia uses and then load ROOT via binary builder. The tricky part would be arranging for Julia to load the right LLVM, but it might be possible by installing LLVM_ROOT_jll or whatever it's called and then setting the DL_LIBRARY_PATH to point to that and restarting Julia, which wouldn't be a completely horrible user experience.

[oschulz]

That would be a nice clean solution. I'm not sure how much it would limit Julia and ROOT versions, since both are kinda opinionated about that (ROOT likely a lot more than JULIA). I'm definitely not an LLVM expert though. @StefanKarpinski, would someone on the Julia core team have time and interest to talk about this with the ROOT team? That might yield clarity very quickly - but I realize you guys are quite busy.

[oschulz]

If so, I could definitely get you in contact with the right people at CERN.

[StefanKarpinski]

I think the thing to make progress would be to get a description of what version of LLVM ROOT needs and how it modifies LLVM.

[oschulz]

@lmoneta, may I pull you into this discussion, as a member of the ROOT team?

[vgvassilev]

Hi all, thanks for looking into this!

ROOT's fork of LLVM and Clang are based on version 9. The LLVM fork is essentially patch free, however the Clang fork has a set of patches which can be seen here.

I believe the requirement how LLVM is loaded (per #17 (comment)) was relaxed. Could you try again?

[oschulz]

Well, it's kinda hard to try at the moment, we'll have to wait until Cxx.jl supports current Julia versions again. I think there is some work going on in that direction.

[grasph]

Hello,

I've investigated the issue. Here are my findings.

1. It works now on my laptop after recompilation of ROOT with both 6.24.00 and 6.24.02 and Julia official binaries! I can call ROOT via it's python binding, make, fill and draw an histogram. See the example below.

2. It works with the root image from Docker hub. I've tried with success with two tags: latest (→ 6.24.02-ubuntu20.04) and 6.22.08-centos7. This can be reproduced with the following commands:

$ docker run --rm --net=host -e DISPLAY -v $HOME/.Xauthority:/root/.Xauthority -it rootproject/root /bin/bash
$ curl https://julialang-s3.julialang.org/bin/linux/x64/1.6/julia-1.6.1-linux-x86_64.tar.gz | gunzip -c | tar xf -
$ julia-1.6.1/bin/julia
julia> ]
pkg> add PyCall
pkg> [backspace]
julia> using PyCall; R = pyimport("ROOT")

If it does not segment fault here, it means it's a success.

To proceed with the example execute the following commands. the root_loop task is required to use ROOT GUI.

julia> root_loop = @task begin; while true; R.gSystem.ProcessEvents(); yield(); end; end
julia> schedule(root_loop)
julia> h = R.TH1D("h", "h", 100, -5, 5)
julia> h.FillRandom("gaus")
julia> h.Draw()

If everything goes as expected a plot with a normal distribution will pop up!

3. Last week, all my attempts with a recompilation failed expect once, even when using Julia recompiled with LLVM 9.0.0 or 9.0.1. From Friday night, all my recompilation works independently of the LLVM version used by Julia. I haven't manage to identify the difference that was making fail last week and work since last weekend.

4. It does not work with the LCG100 installation from CERN lxplus (restricted to CERN users):

lxplus> source /cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc10-dbg/setup.sh
lxplus> gdb julia
(gdb) r
julia> ]
(@v1.6) pkg> add PyCall

(be patient. It can take a while before showing any message)

(@v1.6) pkg> [backspace]
julia> using PyCall; R = pyimport("ROOT")

output:

Thread 1 "julia" received signal SIGSEGV, Segmentation fault.
0x00007f23852ffc04 in llvm::detail::UniqueFunctionBase<llvm::Expected<unsigned long>>::CallbacksHolder<llvm::orc::LegacyRTDyldObjectLinkingLayer::ConcreteLinkedObject<std::shared_ptr<llvm::RuntimeDyld::MemoryManager> >::getSymbolMaterializer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda()#1}, l--Type <RET> for more, q to quit, c to continue without paging--
lvm::orc::LegacyRTDyldObjectLinkingLayer::ConcreteLinkedObject<std::shared_ptr<llvm::RuntimeDyld::MemoryManager> >::getSymbolMaterializer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda()#1}, void>::Callbacks ()
   from /cvmfs/sft.cern.ch/lcg/releases/julia/1.6.0-82998/x86_64-centos7-gcc10-dbg/bin/../lib/julia/libLLVM-11jl.so

And to display the call stack:

(gdb) bt

output:
#0  0x00007f23852ffc04 in llvm::detail::UniqueFunctionBase<llvm::Expected<unsigned long>>::CallbacksHolder<llvm::orc::LegacyRTDyldObjectLinkingLayer::ConcreteLinkedObject<std::shared_ptr<llvm::RuntimeDyld::MemoryManager> >::getSymbolMaterializer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda()#1}, llvm::orc::LegacyRTDyldObjectLinkingLayer::ConcreteLinkedObject<std::shared_ptr<llvm::RuntimeDyld::MemoryManager> >::getSymbolMaterializer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda()#1}, void>::Callbacks ()
   from /cvmfs/sft.cern.ch/lcg/releases/julia/1.6.0-82998/x86_64-centos7-gcc10-dbg/bin/../lib/julia/libLLVM-11jl.so
#1  0x00007f230f9d9a12 in llvm::JITSymbol::getAddress() ()
   from /cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc10-dbg/lib/libCling.so
[...]
#7  0x00007f230f7016b5 in cling::Interpreter::Interpreter (this=0x3362e70, argc=31, argv=0x3362d60, 
    llvmdir=0x359a4f0 "/cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.24.00-a725e/x86_64-centos7-gcc10-dbg/etc//cling", moduleExtensions=..., noRuntime=false)
    at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-v6.24.00/src/ROOT/v6.24.00/interpreter/cling/include/cling/Interpreter/Interpreter.h:342  ← First line with debugger information
[...]

Note: we are missing the debugging information of the llvm code included in cling. I believe this is not intended and there is a problem in the CMAKE configuration of ROOT: the code from the interpreter directory is built with -O3 option and without -g even in the Debug build type (CMAKE_BUILD_TYPE=Debug).

6. I believe the issue is the multiple definitions of the function (in Julia and in ROOT libs) (see the error above).

7. I would expect that using -Bsymbolic (https://linux.die.net/man/1/ld) when building ROOT would solve the problem. I have no experience with this flag, but it looks from the ld documentation it will do what we need if my understanding of the issue is correct.

8. I'm not able to reproduce the problem with a freshly compiled ROOT anymore. This would be needed to validate that -Bsymbolic would solve the issue.

[oschulz]

It works now on my laptop after recompilation of ROOT with both 6.24.00 and 6.24.02 and Julia official binaries!

Oh, that's very encouraging. I'm not absolutely sure that loading ROOT via Python will result in the same situation (in regard to how the two LLVM's coexist) as when loaded directly from Julia, but I guess it should be kinda the same.

the root_loop task is required to use ROOT GUI.

Yes, that's very similar to how we did it in the past:

https://github.com/JuliaHEP/ROOTFramework.jl/blob/master/src/gui.jl#L19

ROOTFramework.jl was (and maybe could be again) and add-on package to ROOT.jl, proving Julia wrappers for TTree access, the GUI task, etc. It was never registered, but me and a few other people did actively use it for a while, reading data that has custom streamers from ROOT files and so on.

[oschulz]

I just tried the current ROOT Docker Container with Julia v1.3 (last version currently supported by CXX.jl). This is as far as I got so far: [...]

• See below for update*

[giordano]

Fatal in <TROOT::InitInterpreter>: cannot load library /opt/julia-1.3.1/bin/../lib/julia/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /opt/root/lib/libCling.so)

GLIBCXX 3.4.26 corresponds to GCC 9.1, according to libstdc++ documentation. You should either use a much newer julia version (1.6+) or replace julia's libstdc++ (/opt/julia-1.3.1/lib/julia/libstdc++.so.6) with the system one (provided that it corresponds to GCC 9.1+)

[oschulz]

You should either use a much newer julia version (1.6+) or replace julia's libstdc++

Yes, that's the rub, can't use Julia v1.6 with Cxx currently. :-) But you're right, brute force replacement does work (had to do this in the past on occasion as well).

So here we go:

docker run -it --rm rootproject/root /bin/bash

In the container:

curl https://julialang-s3.julialang.org/bin/linux/x64/1.3/julia-1.3.1-linux-x86_64.tar.gz | gunzip -c | tar xf -

# Replace the outdated libstdc++.so.6 of Julia v1.3 with the system one:
rm julia-1.3.1/lib/julia/libstdc++.so.6
ln -s /usr/lib/x86_64-linux-gnu/libstdc++.so.6 julia-1.3.1/lib/julia/libstdc++.so.6

# Important, Cxx.jl doesn't enable RTTI by default, but ROOT needs RTTI:
export JULIA_CXX_RTTI=1

julia-1.3.1/bin/julia 

In Julia:

import Pkg; Pkg.add("Cxx")

using Cxx

const ROOT_INCDIR = String(strip(read(`root-config --incdir`, String)))
const ROOT_LIBDIR = String(strip(read(`root-config --libdir`, String)))
const SHEXT = "so"

using Cxx
using Libdl: Libdl

function load_rootlib(libname)
    libpath = joinpath(ROOT_LIBDIR, "lib$libname.$SHEXT")
    isfile(libpath) || error("could not find ROOT library $libname at \"$libpath\"")
    Libdl.dlopen(libpath, Libdl.RTLD_GLOBAL)
end


addHeaderDir(ROOT_INCDIR, kind=C_System)

load_rootlib.([
    "Core", "Gpad", "Graf", "Graf3d", "Hist", "MathCore", "Matrix", "MultiProc", "Net",
    "Physics", "Postscript", "Rint", "RIO", "Thread", "Tree"
])


cxxinclude("TFile.h")

f = icxx"""new TFile("out.root", "recreate");"""

results in

(class TFile *) @0x00000000047d7ca0

Yay!

[oschulz]

According to @Axel-Naumann, the LLVM conflict problem should indeed not occur anymore with ROOT >= v6.24.02. Thanks, Axel!

[oschulz]

Quick update:

I just updated ROOT.jl (master branch) to work with Julia v1.3. I suggest we don't re-register ROOT.jl yet, but wait until Cxx.jl is compatible with current Julia versions again. Also, we should add ROOT installation via BinaryBuilder & BinaryProvider first (with an option for the user to provide their own ROOT, similar to how CUDA.jl does it), so tests can run.

I also updated ROOTFramework.jl (not registered), it's examples (high- and low-level file I/O and some histogram stuff and GUI) work again - currently limited to Julia v1.3, of course.

I moved the basic ROOT GUI support from ROOTFramework.jl to ROOT.jl (see README).

[StefanKarpinski]

but wait until Cxx.jl is compatible with current Julia versions again

Fair warning: I'm not aware of any plans to make that happen. However, someone could take up the job.

[oschulz]

Fair warning: I'm not aware of any plans to make that happen. However, someone could take up the job.

I think there is some activity on that again: JuliaInterop/Cxx.jl#490

[StefanKarpinski]

Excellent!

[grasph]

Some update on this issue: providing a _jll for the ROOT libraries has been attempted (see JuliaPackaging/Yggdrasil#7666). It is currently on hold because the ROOT library source does not support cross-compiling, while it is used by the BinaryBuilder infrastructure. Adding cross-compilation support is not straightforward due to interplay between ROOT and c++ compiler

As a mid-term solution, a mechanism that downloads the ROOT libraries from conda-forge if not already available on the system has been added. The wrapping code is compiled on the fly at package installation, ensuring it is compiled with same compiler and stdlib versions as the used ROOT libraries.

[Moelf]

The wrapping code is compiled on the fly at package installation, ensuring it is compiled with same compiler and stdlib versions as the used ROOT libraries.

this probably needs to be a bit more flexible because real users would change LCG release or ATLAS/CMS` environment all the time.

[grasph]

Maybe I was not clear in my explanations.

The compilation occurs when adding the package. If, during this operation a ROOT installation with the expected version number is found in the PATH, then it will assume you have a consistent environment and use the compiler set in you environment. Otherwise, it will install ROOT and its dependencies, that includes the compiler, from conda-forge (in ~/julia/conda/3).

The installation records the location of the ROOT library. So if you start a new Julia session with a different LCG environment, it will still use the version found at installation time.

The current version asks for a specific ROOT release, so even if you are in an LCG environment when adding the package, the chance that it installs it own version via conda is high.

[Moelf]

records the location of the ROOT library. So if you start a new Julia session with a different LCG environment, it will still use the version found at installation time.

That's precisely why it's confusing to users. It's not too urgent or anything, we could just have some mechanism similar to how PythonCall.jl allows user to select Python environment.

[oschulz]

Could/should we use the artifacts system to download the ROOT binaries and dependencies from conda-forge directly, instead of downloading them via conda?

[grasph]

ROOT has many dependencies. For the _jll, we should compile it using dependencies distributed as _jll. So what needs to be done is to fix the cross-compiling issue.

In the meantime the conda solution is perfectly fine.

In case the automatic selection of the existing ROOT installation is confusing, we can either drop this option and limit to conda install or make the selection explicit.

Nevertheless, we should not invest time in this intermediate solution. Effort should be focused the proper _jll solution.

Philippe.

[StefanKarpinski]

For what it's worth, JLLs are a fantastic way of making complex binaries available across platforms even if you're not using Julia. Might be a good way to let people use ROOT locally otherwise as well. But yes, getting cross-compilation to work is essential (since that's how the whole system works).

[StefanKarpinski]

I should add that there is a lot of expertise and willingness to help on the JuliaLang Slack #binarybuilder channel. Modifying builds to support cross-compilation is usually not too bad and makes the build more correct at no real cost, one just has to be willing to understand cross compilation a bit.

[oschulz]

Nevertheless, we should not invest time in this intermediate solution. Effort should be focused the proper _jll solution.

Yes, you're right.

[oschulz]

@StefanKarpinski can BinaryBuilder cross-compile Julia itself and it's dependencies? Just curious because the challenge may have similarities with doing the same for ROOT.

[giordano]

can BinaryBuilder cross-compile Julia itself and it's dependencies?

Julia dependencies are already available in Yggdrasil: those are the binaries shipped in the official distribution of Julia. BinaryBuilder can compile the Julia executable, but not build the sysimage, because Julia itself doesn't support cross-compilation