bin folder not supported in latest NextFlow

[VasLem]

It is mentioned in the workflow structure that a user can add their binaries to a bin folder, which is automatically added to PATH. Latest Nextflow does not have this functionality, as you can see from an example .command.run (I had to change the name to be able to upload it here):
command.run.txt

Could you please update the page with a deprecation flag or something to let the users now about the obsoletion of this feature? Thank you in advance.

[robsyme]

This behaviour hasn't changed in any recent version of Nextflow. Which version of Nextflow did you see the bin dir not being added to PATH?

[VasLem]

Thank you for the ultra fast response. I am using version 24.03.0-edge . There is no reference to this structure in the edge or latest process documentation either.

[VasLem]

I see there is a reference for binaries inclusion in the modules page , however there a different directory structure is mentioned, <module-dir>/resources/usr/bin .

[mribeirodantas]

What you're saying doesn't seem to be right, @VasLem. I just tried with 24.03.0-edge and it works as expected.

tree .
.
├── bin
│   └── test.sh
└── main.nf

2 directories, 2 files

main.nf

process FOO {
  debug true

  input:
  val x

  output:
  stdout

  script:
  """
  test.sh $x
  """
}

workflow {
  Channel
    .of(1..5)
    | FOO

./bin/test.sh

#!/bin/bash

echo hello $1

Output:

[VasLem]

Hi Robert, thanks for remaining active on this topic. Sure, here is a minimum reproducible example.

binary.zip

The command I use is ./nextflow run binary .
I am getting the error:

/mnt/disks/mybucket/nextflow/4f/933deace222a28a15d9e4c267bdef1/.command.sh: line 2: ./hello.sh: No such file or directory

Here are the files existing in the directory:
933deace222a28a15d9e4c267bdef1.zip

The tmp folder mentioned in .command.run also exists, that is the hello.sh is uploaded, however the copy does not seem to work.

[robsyme]

Is your hello.sh executable? In the example zip provided it is not:

ls -lh bin
total 8
-rw-rw-r--@ 1 robsyme  staff    39B 22 Apr  2024 hello.sh

Can you try after making the shell script executable:

chmod +x bin/hello.sh

... and then committing and pushing your changes before running the workflow again?

[VasLem]

This has already been done. The reason why you don't see it as an executable is that I used Windows to zip it (from wsl), where the permissions are not consistent.

[VasLem]

Okay, I was curious so let me provide some more input. I spawned a machine replacing the hello with a sleep, ssh-ed to it, and then entered in the docker container spawned by nextflow. The file is clearly there, however the PATH environment variable is not how it should be, it does not contain any reference to .nextflow-bin, it is just the reset one (directly printed from within process):

/usr/local/bin:/usr/bin:/bin:/usr/games

Long story short, the sequence of the commands within the .command.run must be problematic. The command block:

cp -r /mnt/disks/uza-genomics-workdirs/nextflow/tutorial/tmp/4d/87f44822f6a065520cd4aa2be9ab4d $HOME/.nextflow-bin
chmod +x $HOME/.nextflow-bin/*
export PATH=$HOME/.nextflow-bin:$PATH

instead of the beginning of the file, it needs to be inside the staged working directory, that is:

nxf_launch() {
>>>>>>>>>> HERE
     export PATH=.nextflow-bin:$PATH
<<<<<<<<<<
    /bin/bash -ue /mnt/disks/uza-genomics-workdirs/nextflow/tutorial/4f/933deace222a28a15d9e4c267bdef1/.command.sh
}


.
.
.

nxf_main() {
    trap on_exit EXIT
    trap on_term TERM INT USR2
    trap '' USR1

    [[ "${NXF_CHDIR:-}" ]] && cd "$NXF_CHDIR"
    NXF_SCRATCH="$(set +u; nxf_mktemp $TMPDIR)"

>>>>>>>>>>>>>>>>>>>>>>>>AND HERE

cp -r /mnt/disks/uza-genomics-workdirs/nextflow/tutorial/tmp/4d/87f44822f6a065520cd4aa2be9ab4d $NXF_SCRATCH/.nextflow-bin
chmod +x $NXF_SCRATCH/.nextflow-bin/*

<<<<<<<<<<<<<<<<<<<<<<<

    [[ $NXF_DEBUG > 0 ]] && nxf_env
    touch /mnt/disks/uza-genomics-workdirs/nextflow/tutorial/4f/933deace222a28a15d9e4c267bdef1/.command.begin
    set +u
    set -u
    [[ $NXF_SCRATCH ]] && cd $NXF_SCRATCH
    export NXF_TASK_WORKDIR="$PWD"
    nxf_stage

    set +e
    (set -o pipefail; (nxf_launch | tee .command.out) 3>&1 1>&2 2>&3 | tee .command.err) &
    pid=$!
    wait $pid || nxf_main_ret=$?
    nxf_unstage
}

I don't have much of knowledge with Groovy, but I feel this fix is just a bit of code translocation....

[VasLem]

Should I make a new issue regarding that?