cross compilation example using flakes

[joprice]

I'm trying to figure out an approach to using cross compilation in a flake, where there would be a client and a server, the client being run on a raspberry pi, and the server on x86 linux. I'm unsure about the project structure and workflow for that, whether the cross overlay is meant to enable dune to support cross compilation, a system flag is meant to be passed to nix build, or a separate packages should be defined for each system, where each package calls buildDunePackage with the appropriate cross or host ocamlPackages. I'm also not sure about where tools like dune should be provided to enable the cross compilation. Is there an example that does something like along these lines?

[anmonteiro]

Does this example have what you’re looking for? mseri/doi2bib#19

[joprice]

Thank you! That did get me much further along. I was getting the below error as I had not overridden the installPhase:

error: attribute 'ocaml_cross_test' missing

       at /nix/store/v6h8a5ks4la4xlj3cm2xv78x42i3qsny-source/cross/ocaml.nix:176:17:

          175|                 # sqlite3
          176|                 natocamlPackages."ocaml_${args.pname}";
             |                 ^
          177|           in
(use '--show-trace' to show detailed location information) 

[joprice]

I was able to get mus64 building, but for arm I get

error: builder for '/nix/store/4alzk78ck5cxl241p0ac2324wn8vyi12-ocaml4.14.0-cross_test-0.1.0-aarch64-unknown-linux-musl.drv' failed with exit code 1;
       last 10 log lines:
       > source root is source
       > patching sources
       > updateAutotoolsGnuConfigScriptsPhase
       > configuring
       > no configure script, doing nothing
       > building
       > ocamlfind: [WARNING] Undefined toolchain: aarch64
       > Error: ocamlfind toolchain aarch64 isn't defined in
       > /nix/store/8m0xm0xyskx9p92y2llxk3svmdp4mkka-ocaml4.14.0-findlib-1.9.6/etc/findlib.conf.d
       > (context: default)

[joprice]

Looks like I can use qemu or this idea https://discourse.nixos.org/t/how-to-cross-compile-kernel-for-aarch64-on-x86-host-without-qemu/16970 to allow my local nixos to support the arch. That brings me to the next issue, which is that I actually need 32-bit arm, which would be armv7l-hf, which isn't available in the overlays. I tried adding it to the existing overlay, but I assume it won't be that easy and might require other changes to support.

[joprice]

I was able to get aarch64 working by wrapping the buildDunePackage with fixOCamlPackage: https://github.com/nix-ocaml/nix-overlays/compare/master...joprice:nix-overlays:addArmv7l?expand=1. Without this change, the findlib.conf.d was the default location, which does not have the aarch64 override. I also tried adding armv7 on that branch but haven't gotten a static version working.

[joprice]

I was able to get a minimum static arm7l binary built, but it now fails on a ppx that uess dlopen.

       > (.text+0x394): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
       > File "bin/dune", line 5, characters 13-38:
       > 5 |  (preprocess (pps ppx_deriving_yojson)))
       >                  ^^^^^^^^^^^^^^^^^^^^^^^^^
       > qemu: uncaught target signal 11 (Segmentation fault) - core dumped

I can put up a repo with my experiment and maybe open a separate issue for armv7l support and this other issue, if you'd prefer?

[joprice]

I was able to get a minimum static arm7l binary built, but it now fails on a ppx that uses dlopen.

       > (.text+0x394): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
       > File "bin/dune", line 5, characters 13-38:
       > 5 |  (preprocess (pps ppx_deriving_yojson)))
       >                  ^^^^^^^^^^^^^^^^^^^^^^^^^
       > qemu: uncaught target signal 11 (Segmentation fault) - core dumped

I can put up a repo with my experiment and maybe open a separate issue for armv7l support and this other issue, if you'd prefer?

[anmonteiro]

I'd recommend revisiting cross compilation now. there have been some nice improvements in dune / overlays that should make your experience a bit better.

[Kakadu]

Could you also add cross-compilation instructions for total nix newcomers? Looking at CI I discovered ./ci.sh x86_64-linux arm64_4_14 but it's not obvious what to do next with all of this. Compiling hello-world for target architecture and running it via qemu would be helpful...