Re: Slurm with containers (i.e., orchestration)

[Pjotr Prins <pjotr.public12@thebird.nl>]

Ricardo added slurm-drmaa in the past (I can't believe it
almost 4 years ago we packaged slurm!) which may also help in
addressing some points

  http://www.drmaa.org/

Pj.


On Mon, May 18, 2020 at 07:49:00AM -0500, Pjotr Prins wrote:
> I am looking into some light-weight style orchestration. One
> possibility is to use Slurm with Guix containers - on a cluster with
> Guix that is almost trivial (we use Guix containers a lot! They are
> great) and would also allow non-container jobs.
> 
> Once we have containers and Slurm it should also be possible to deploy
> in some cloud infrastructure, provided there are no dependencies on
> the cluster itself. I think it would make a terrific BLOG story if we
> put something like that together. 
> 
> Bcbio describes an architecture that uses the common workflow language
> (CWL) to run pipelines with containers
> 
>   https://bcbio-nextgen.readthedocs.io/en/latest/contents/cwl.html#running-with-cromwell-local-hpc
> 
> I am not promoting the use of this, but it shows that infrastructure
> exists that can deploy workflows on containers in different setups
> (Bcbio supports Slurm). I know the Guix infrastructure uses Guix
> deploy to achieve similar roll-outs. What that lacks is the
> orchestration mechanism itself which should handle dependencies
> between jobs (i.e. a workflow). The GNU Workflow Language goes some
> way, but it does not handle orchestration itself.
> 
> In other words, we almost have the pieces, but one thing is missing
> :). Thoughts? I know I have brought this up before in different
> guises, but we start to really need something here.
> 
> What makes orchestration? I guess it concerns a dynamic database of
> machines that can execute jobs and some type of software registry
> (Guix).  Next it should be able to schedule and execute jobs using
> some constraint specifiers (like network/CPU/RAM). It could be a
> 'dynamic' Slurm that makes use of real machines and VMs. Or hook into
> an existing cloud service. A slurm job could monitor sending a
> container into a cloud service. 
> 
> I think we can build this up a step at a time. 
> 
> Thoughts?
> 
> Pj.
>