Re: Getting started with GWL 0.3.0

[Ricardo Wurmus <rekado@elephly.net>]

Roel Janssen <roel@gnu.org> writes:

> On Tue, 2021-03-23 at 18:34 +0100, Ricardo Wurmus wrote:
>> 
>> Before you get too enthusiastic about the GWL, though, I’d like to
>> note
>> that 0.3.0 has a few known bugs that are already fixed in the
>> repository.  I’ve been putting off making a new release until either
>> Guile-AWS or Guile-DRMAA are ready and usable with the GWL.
>
> Is there a feature-branch to try out GWL with Guile-DRMAA? :)

Unfortunately not yet.

I haven’t been 100% successful with the only DRMAA-enabled cluster that
I have access to, and it turns out that it’s not as simple as SGE’s
“hold_jid”.

It’s no longer “fire and forget”, which is a bit sad, but that’s how
DRMAA works.  We need a run-time component that keeps track of submitted
jobs and their status and actively starts held jobs when the
prerequisites have finished.

It’s not clear to me if and how we should persist workflow state.  The
GWL will submit all jobs to the scheduler in a held state and then
change their status when its their turn.  I wonder if and how we should
handle the case where the GWL runtime monitor dies and is restarted.
The easiest way is to simply kill all queued up jobs, but I don’t know
if there’s a better approach.

Ideas?

-- 
Ricardo