Please create a new release for Julia 1.x support

[wkharold]

The SlurmManager fails under Julia 1.x because of a change to the replace() function. The master branch incorporates the necessary modification but has not been incorporated in a release.

[vchuravy]

Yes =, we do need a new release. I have been hesitant to do so because we currently lack tests (#105) and without them it is very hard to gauge which managers are currently working an which patches fixed things.

I am currently traveling, but if you need a release maybe you could help with transitioning from REQUIRE to Project.toml

[jamitzky]

I still get the replace bug in the 1.x Versions for SLURM. Any plans to fix that?

[kescobo]

Given julia 1.0 has been out for nearly 2 years, seems like a release supporting it would be worth doing. Yes, it's unfortunate that it lacks tests, but that doesn't seem worse than the current situation...

[vchuravy]

Yes and no... Not having a release is a clear documentation that this package currently does not have a maintainer and I am kinda hoping that someone would like to help out with this. People relying on software that might not work is worse than them being aware of that it might not work... So yes if you are using ClusterManagers.jl it would be great if you could help out with maintenance and maybe setting up a test system...

[kescobo]

That's fair - I can definitely see the point. But I'd argue that something that might not work in the release is better than something definitely doesn't work. Especially if it's clearly stated in the README that it's poorly tested.

That said, I'd potentially be willing to take on a little bit of maintenance work if that would be helpful. In general, my only experience with distributed computing is using SLURM on Harvard's and MIT's clusters.

[bjarthur]

how would a test system work given that this pkg supports several different schedulers? perhaps it's time to break it up into smaller packages?

[vchuravy]

@bjarthur yes that might be possible. I would also be more than happy to give people the commit bit if they want to contribute to ClusterManagers.jl

There is also https://github.com/JuliaParallel/MPIClusterManagers.jl

[DrChainsaw]

Hi,

I'm willing to put some effort into helping maintaining the LSF parts, preferably as a separate package. I don't own an LSF cluster but I don't know if thats a showstopper or if it is just a matter of needing an extra pair of hands.

I was thinking that one could install the trial version in a build script (maybe ask IBM first if this is ok). I have no idea how hard this will be though. Perhaps it has already been tried?

[vchuravy]

@DrChainsaw I gave you access/

[DrChainsaw]

@vchuravy Thanks alot.

What do people feel about mocking as a way to support testing? I think it has some merits given that one disadvantage of spinning up a real (but local) cluster is that one probably will see more issues with that than the code it is supposed to test.

I started drafting up a tiny PoC a few days ago but then I got some other things on my plate.

In a linux environment it could be as simple as just piping a few echo job 23432 submitted to a file called bsub (or whatever is needed for the other managers) and similar for other commands while spinning up local worker processes.

It seems to be slightly more annoying in windows as it does not execute files without extensions, and julia shell won't run e.g. bsub.bat if you type ;bsub. It does work for exe-files though and my 5 minute search for how to realiably create an exe in any windows environment lead to .NET and csc.exe which might be workable despite being non-ideal.

All this goes away ofc if it is possible to modify which command is being executed.

Worth pursuing?