Getting started with GWL 0.3.0

Getting started with GWL 0.3.0

[Konrad Hinsen]

Hi everyone,

After lurking here for a long time, I am finally starting to explore GWL
for real, meaning that I intend to use it for a (small) real project.

Unfortunately, my first attempt didn't get very far:

   $ guix workflow run hello.w
   guix: workflow: command not found
   Try `guix --help' for more information.

I do have gwl@0.3.0 installed. That made me curious about how GWL
actually adds subcommands to "guix", but I quickly discovered
that it uses an apparently undocumented extension mechanism
by placing scripts under guix/extensions. So... I am lost.
How do I actually run a workflow?

Cheers,
  Konrad.

Re: Getting started with GWL 0.3.0

[zimoun]

Hi Konrad,

On Mon, 22 Mar 2021 at 11:32, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> Unfortunately, my first attempt didn't get very far:
>
>    $ guix workflow run hello.w
>    guix: workflow: command not found
>    Try `guix --help' for more information.

I do not know.  I had similar issues though.

> I do have gwl@0.3.0 installed. That made me curious about how GWL
> actually adds subcommands to "guix", but I quickly discovered
> that it uses an apparently undocumented extension mechanism
> by placing scripts under guix/extensions. So... I am lost.
> How do I actually run a workflow?

The plan of the documentation is discussed here:

   <https://yhetil.org/guix/86k0sekkj8.fsf@gmail.com>

This is really cool because “guix repl -- foo.scm arg1 arg2” can now be
really handy with “guix foo arg1 arg2”.  However, it is still
undocumented because we have not reached consensus with Ricardo. :-)
See the 3 thread emails starting there:

  <https://yhetil.org/guix/86k0rsdylk.fsf@gmail.com>


Well, about this GUIX_EXTENSIONS_PATHS, as the author of “guix repl --”
your feedback is very welcome. :-)


Cheers,
simon

Re: Getting started with GWL 0.3.0

[Konrad Hinsen]

Hi Simon,

> The plan of the documentation is discussed here:
>
>    <https://yhetil.org/guix/86k0sekkj8.fsf@gmail.com>

Looks like I missed a discussion on guix-devel. My excuse is that I
can't keep up with guix-devel any more, it's getting too much!

> This is really cool because “guix repl -- foo.scm arg1 arg2” can now be
> really handy with “guix foo arg1 arg2”. 

Handy, yes. But is it a good idea from a security point of view? As a
Guix user, I trust "guix" with all its subcommands because I know that
all the code is carefully inspected by several competent developers.  I
don't have the same level of trust in software packaged within Guix.

I'd rather see packages building on "guix" but provide their own
top-level scripts with distinct names. And support for writing such
packages in making it easier to access the user's default Guix profile.

> Well, about this GUIX_EXTENSIONS_PATHS, as the author of “guix repl --”
> your feedback is very welcome. :-)

See above :-)

Cheers,
  Konrad.

Re: Getting started with GWL 0.3.0

[zimoun]

Hi,

On Mon, 22 Mar 2021 at 14:04, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> Looks like I missed a discussion on guix-devel. My excuse is that I
> can't keep up with guix-devel any more, it's getting too much!

Do not worry!  And I do not know if these days someone is actually
able to grasp all the discussions happening on guix-devel. :-)

> > This is really cool because “guix repl -- foo.scm arg1 arg2” can now be
> > really handy with “guix foo arg1 arg2”.
>
> Handy, yes. But is it a good idea from a security point of view? As a
> Guix user, I trust "guix" with all its subcommands because I know that
> all the code is carefully inspected by several competent developers.  I
> don't have the same level of trust in software packaged within Guix.

What do you mean?

The user has to explicitly set GUIX_EXTENSIONS_PATH or explicitly
install a package (or a channel, as "guix home").  I do not see where
there is a security flaw, I mean it is the same vulnerability as for
"guix repl -- foo.scm" or as for "guix install foo && foo".

And if you worry, I guess you can run GWL in a container, something like;

  guix environment -C --ad-hoc gwl -- guix workflow


> I'd rather see packages building on "guix" but provide their own
> top-level scripts with distinct names. And support for writing such
> packages in making it easier to access the user's default Guix profile.

Personally, I like the idea of extensions.  Similarly as "git foo"
works if "git-foo" is an executable on the PATH.

I imagine couple of extensions.  For instance, testing idea on UI is
hard because Guix itself is really conservative about the backward
compatibility---for a good! :-)
And we can imagine extensions as a way to test other flavours, either
before introducing a new subcommand or either as a replacement of
current subcommand.


Cheers,
simon

Re: Getting started with GWL 0.3.0

[Konrad Hinsen]

zimoun <zimon.toutoune@gmail.com> writes:

> The user has to explicitly set GUIX_EXTENSIONS_PATH or explicitly
> install a package (or a channel, as "guix home").  I do not see where
> there is a security flaw, I mean it is the same vulnerability as for
> "guix repl -- foo.scm" or as for "guix install foo && foo".

The vulnerability level is the same, but a typical user's expectations
are not. When I run a script via "guix repl", it is perfectly obvious to
me that that script is not part of Guix. When I run the command "foo",
it is also perfectly obvious to me that "foo" is not part of Guix. But
"guix workflow" looks as if it were part of Guix.

Guix users need a significant level of trust in Guix and its developers
in order to use Guix. Attackers could exploit this trust by tricking
users into adding malicious code (via a channel, for example) that takes
the form of a Guix extension. People do run downloaded bash scripts
with root permissions, after all (e.g. for installing Guix ;-).

> And if you worry, I guess you can run GWL in a container, something like;
>
>   guix environment -C --ad-hoc gwl -- guix workflow

Sure, but that's not the issue. The issue is being aware that I have no
particular reason to trust "guix workflow" as much as I trust "guix
package".

Of course GWL is a bad example because its developers are a subset of
Guix developers. But the extension mechanism looks open to everyone.

Imagine a package "bitcoin-utils" that installs, among lots of other
stuff, an extension "guix pul" that starts a bitcoin-mining background
process whenever an unsuspecting user mistypes a frequent Guix command.

Cheers,
  Konrad.

Re: Getting started with GWL 0.3.0

[zimoun]

Hi,

On Mon, 22 Mar 2021 at 16:07, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:

> The vulnerability level is the same, but a typical user's expectations
> are not. When I run a script via "guix repl", it is perfectly obvious to
> me that that script is not part of Guix. When I run the command "foo",
> it is also perfectly obvious to me that "foo" is not part of Guix. But
> "guix workflow" looks as if it were part of Guix.
>
> Guix users need a significant level of trust in Guix and its developers
> in order to use Guix. Attackers could exploit this trust by tricking
> users into adding malicious code (via a channel, for example) that takes
> the form of a Guix extension. People do run downloaded bash scripts
> with root permissions, after all (e.g. for installing Guix ;-).
>
> > And if you worry, I guess you can run GWL in a container, something like;
> >
> >   guix environment -C --ad-hoc gwl -- guix workflow
>
> Sure, but that's not the issue. The issue is being aware that I have no
> particular reason to trust "guix workflow" as much as I trust "guix
> package".

[...]

> Imagine a package "bitcoin-utils" that installs, among lots of other
> stuff, an extension "guix pul" that starts a bitcoin-mining background
> process whenever an unsuspecting user mistypes a frequent Guix command.

Well, I understand your concerns but I am not convinced to share them.

IIUC, you are saying that "git annex" or "git lfs" which are
extensions to Git are a security issue because if any malware-package
providing a "git-pul" malware, then a user typing "git pul" with a
typo can have bad surprise.  But at first, you need to trust a channel
providing this malware-package, then second  you need to install this
malware-package and third make the typo.

And what about "guxi pull" where the typo is not on the subcommand but
on the command itself?

To me, being handy beats the concern. :-)  It is the responsability of
the user to know what they is installing and running on their own
machine.


Cheers,
simon

Re: Getting started with GWL 0.3.0

[Konrad Hinsen]

Hi Simon,

> Well, I understand your concerns but I am not convinced to share them.

We can certainly agree to disagree!

> IIUC, you are saying that "git annex" or "git lfs" which are
> extensions to Git are a security issue because if any malware-package
> providing a "git-pul" malware, then a user typing "git pul" with a

Yes, exactly. Like what happened to npm:

  https://threatpost.com/attackers-use-typo-squatting-to-steal-npm-credentials/127235/

Apparently this is now called typo-squatting.

> typo can have bad surprise.  But at first, you need to trust a channel
> providing this malware-package, then second  you need to install this
> malware-package and third make the typo.

The last part comes with zero effort :-)

As for trusting channels and packages, this is not much of an issue
today, but if Guix ever becomes as popular as Debian is today, then we
will have plenty of users with no clue about who or what they can trust.

In the long run, maybe a command spell-checker would be a nice way out.
Some piece of software that decides, based on my command history,
whether a command I type is more likely a typo or the intention to run
some exotic software.

Cheers,
  Konrad.

Re: Getting started with GWL 0.3.0

[Ricardo Wurmus]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> In the long run, maybe a command spell-checker would be a nice way out.
> Some piece of software that decides, based on my command history,
> whether a command I type is more likely a typo or the intention to run
> some exotic software.

This would need more work to allow replacement of Guix commands with an
extension.  For example, I may want to use an extension that replaces
“guix deploy” with a variant that can deploy to AWS, falling back to the
default “guix deploy” for all other deployments.

We should not prevent this use case, so at the very least this needs to
be configurable (though I’d be happier if we could come up with a
configuration-free solution).

A configuration-free alternative could be “guix ext pull” to explicitly
select the extension named “pull”.  There can be no conflicts with this
scheme, but it’s more work to type.

-- 
Ricardo

Re: Getting started with GWL 0.3.0

[Roel Janssen]

On 3/23/21 2:16 PM, Ricardo Wurmus wrote:
> Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
>
>> In the long run, maybe a command spell-checker would be a nice way out.
>> Some piece of software that decides, based on my command history,
>> whether a command I type is more likely a typo or the intention to run
>> some exotic software.
> This would need more work to allow replacement of Guix commands with an
> extension.  For example, I may want to use an extension that replaces
> “guix deploy” with a variant that can deploy to AWS, falling back to the
> default “guix deploy” for all other deployments.
>
> We should not prevent this use case, so at the very least this needs to
> be configurable (though I’d be happier if we could come up with a
> configuration-free solution).
>
> A configuration-free alternative could be “guix ext pull” to explicitly
> select the extension named “pull”.  There can be no conflicts with this
> scheme, but it’s more work to type.
>
Or perhaps it could work with a warning message when invoking
an extension command for the first time (similar to what Emacs
does when hitting a key combination that may be unintended).

For example:
$ guix workflow ...
You are about to run an extension to GNU Guix.
Enable this extension for future use? [y/N]

Perhaps this message could include the package from which
the extension originates.

Kind regards,
Roel Janssen

Re: Getting started with GWL 0.3.0

[Konrad Hinsen]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Unfortunately, my first attempt didn't get very far:
>
>    $ guix workflow run hello.w
>    guix: workflow: command not found
>    Try `guix --help' for more information.

That was with Guix installed under Ubuntu. Today I tried on a VM
running Guix System. Result: the same.

So... how do you all actually use GWL? Should I try to build/run
from source? I don't see any instructions for that either, nor
any pointer to the source repo.

Cheers,
  Konrad.

Re: Getting started with GWL 0.3.0

[Ricardo Wurmus]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
>
>> Unfortunately, my first attempt didn't get very far:
>>
>>    $ guix workflow run hello.w
>>    guix: workflow: command not found
>>    Try `guix --help' for more information.
>
> That was with Guix installed under Ubuntu. Today I tried on a VM
> running Guix System. Result: the same.
>
> So... how do you all actually use GWL? Should I try to build/run
> from source? I don't see any instructions for that either, nor
> any pointer to the source repo.

I should add this to the manual.  Here’s what I do:

--8<---------------cut here---------------start------------->8---
$ guix install gwl
$ export GUIX_EXTENSIONS_PATH=$HOME/.guix-profile/share/guix/extensions/
$ guix workflow --help
Usage: guix workflow 

Subcommands:
  graph     Visualize a workflow
  run       Run a workflow
  web       Start the web interface

The Guix Workflow Language (GWL) is a scientific computing extension to the Guix
package manager.  It combines the specification of work units and their
relationship to one another with the reproducible software deployment facilities
of the functional package manager GNU Guix.  A GWL workflow will always run in a
reproducible environment that GNU Guix automatically prepares.  The GWL extends
your Guix installation with a single new sub-command: guix workflow.
$ 
--8<---------------cut here---------------end--------------->8---

Before you get too enthusiastic about the GWL, though, I’d like to note
that 0.3.0 has a few known bugs that are already fixed in the
repository.  I’ve been putting off making a new release until either
Guile-AWS or Guile-DRMAA are ready and usable with the GWL.

-- 
Ricardo

Re: Getting started with GWL 0.3.0

[Roel Janssen]

On Tue, 2021-03-23 at 18:34 +0100, Ricardo Wurmus wrote:
> 
> Before you get too enthusiastic about the GWL, though, I’d like to
> note
> that 0.3.0 has a few known bugs that are already fixed in the
> repository.  I’ve been putting off making a new release until either
> Guile-AWS or Guile-DRMAA are ready and usable with the GWL.

Is there a feature-branch to try out GWL with Guile-DRMAA? :)

Kind regards,
Roel Janssen

Re: Getting started with GWL 0.3.0

[Ricardo Wurmus]

Roel Janssen <roel@gnu.org> writes:

> On Tue, 2021-03-23 at 18:34 +0100, Ricardo Wurmus wrote:
>> 
>> Before you get too enthusiastic about the GWL, though, I’d like to
>> note
>> that 0.3.0 has a few known bugs that are already fixed in the
>> repository.  I’ve been putting off making a new release until either
>> Guile-AWS or Guile-DRMAA are ready and usable with the GWL.
>
> Is there a feature-branch to try out GWL with Guile-DRMAA? :)

Unfortunately not yet.

I haven’t been 100% successful with the only DRMAA-enabled cluster that
I have access to, and it turns out that it’s not as simple as SGE’s
“hold_jid”.

It’s no longer “fire and forget”, which is a bit sad, but that’s how
DRMAA works.  We need a run-time component that keeps track of submitted
jobs and their status and actively starts held jobs when the
prerequisites have finished.

It’s not clear to me if and how we should persist workflow state.  The
GWL will submit all jobs to the scheduler in a held state and then
change their status when its their turn.  I wonder if and how we should
handle the case where the GWL runtime monitor dies and is restarted.
The easiest way is to simply kill all queued up jobs, but I don’t know
if there’s a better approach.

Ideas?

-- 
Ricardo

Re: Getting started with GWL 0.3.0

[zimoun]

Hi Konrad,

On Tue, 23 Mar 2021 at 13:57, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:

> Yes, exactly. Like what happened to npm:
>
>   https://threatpost.com/attackers-use-typo-squatting-to-steal-npm-credentials/127235/
>
> Apparently this is now called typo-squatting.

And PyPI and.. whatever "store" with no clear policy...

> > typo can have bad surprise.  But at first, you need to trust a channel
> > providing this malware-package, then second  you need to install this
> > malware-package and third make the typo.
>
> The last part comes with zero effort :-)
>
> As for trusting channels and packages, this is not much of an issue
> today, but if Guix ever becomes as popular as Debian is today, then we
> will have plenty of users with no clue about who or what they can trust.

...and you can do the same with any package manager.  For instance,
add an evil channel to your /etc/apt/sources.list, then "apt-get
update" and then you have no guarantee on what you are installing, for
example, "apt-get install git" will install regular Git with a git-pul
typo malware, or even the Git version you get is a malware.

The issue at first is the channel.  There is official channels that
you are trusting and other channels that you cannot trust.  Well, your
concern is about any a suspicious channel, for instance, if I blindly
install the python-paper-foo package then I have no guarantee that
this package also install the malicious 'emasc' binary which is a typo
I often make.

> In the long run, maybe a command spell-checker would be a nice way out.
> Some piece of software that decides, based on my command history,
> whether a command I type is more likely a typo or the intention to run
> some exotic software.

Well, checking at each command invocation could slow Guix, since it is
already not the fastest CLI of West. :-)


Cheers,
simon

Re: Getting started with GWL 0.3.0

[Roel Janssen]

On Tue, 2021-03-23 at 21:14 +0100, Ricardo Wurmus wrote:
> 
> Roel Janssen <roel@gnu.org> writes:
> 
> > On Tue, 2021-03-23 at 18:34 +0100, Ricardo Wurmus wrote:
> > > 
> > > Before you get too enthusiastic about the GWL, though, I’d like to
> > > note
> > > that 0.3.0 has a few known bugs that are already fixed in the
> > > repository.  I’ve been putting off making a new release until
> > > either
> > > Guile-AWS or Guile-DRMAA are ready and usable with the GWL.
> > 
> > Is there a feature-branch to try out GWL with Guile-DRMAA? :)
> 
> Unfortunately not yet.
> 
> I haven’t been 100% successful with the only DRMAA-enabled cluster that
> I have access to, and it turns out that it’s not as simple as SGE’s
> “hold_jid”.
> 
> It’s no longer “fire and forget”, which is a bit sad, but that’s how
> DRMAA works.  We need a run-time component that keeps track of
> submitted
> jobs and their status and actively starts held jobs when the
> prerequisites have finished.

That's unfortunate, but I believe having a daemon that keeps track of
the workflow opens possibilities for "cloud" "orchestration".

> It’s not clear to me if and how we should persist workflow state.  The
> GWL will submit all jobs to the scheduler in a held state and then
> change their status when its their turn.  I wonder if and how we should
> handle the case where the GWL runtime monitor dies and is restarted.
> The easiest way is to simply kill all queued up jobs, but I don’t know
> if there’s a better approach.
> 
> Ideas?

I find killing/removing queued jobs upon exiting the runtime monitor a
good idea!

Maybe not suitable anymore, but I wrote a "qsub" command that
translates to "squeue" here:
https://github.com/roelj/qsub-slurm

Could we use the same approach?  It works because jobs are submitted in
order.  The look-up mechanism can be found here:
https://github.com/roelj/qsub-slurm/blob/master/qsub.in#L233-L253


I have access to a SLURM cluster (I don't know which version of DRMAA
it supports), but I can test it.

Kind regards,
Roel Janssen

Re: Getting started with GWL 0.3.0

[Konrad Hinsen]

Hi Ricardo,

> I should add this to the manual.  Here’s what I do:
>
> --8<---------------cut here---------------start------------->8---
> $ guix install gwl
> $ export GUIX_EXTENSIONS_PATH=$HOME/.guix-profile/share/guix/extensions/
> $ guix workflow --help

Thanks, that works for me as well. I'll go back to my playground now :-)

> Before you get too enthusiastic about the GWL, though, I’d like to note
> that 0.3.0 has a few known bugs that are already fixed in the
> repository.  I’ve been putting off making a new release until either
> Guile-AWS or Guile-DRMAA are ready and usable with the GWL.

OK, if I hit anything strange I will rebuild from source before
complaining. I actually found the source repository by following
the link to the source code for the Web site.

Cheers,
  Konrad.

Re: Getting started with GWL 0.3.0

[Konrad Hinsen]

Hi Simon,

>> As for trusting channels and packages, this is not much of an issue
>> today, but if Guix ever becomes as popular as Debian is today, then we
>> will have plenty of users with no clue about who or what they can trust.
>
> ...and you can do the same with any package manager.  For instance,

Yes, exactly. Trusting software sources is becoming an ever more
important issue everywhere, as people rely on ever more complex software
assemblies whose components they can no longer verify individually.
Which is also why package managers now become targets of attacks.

> The issue at first is the channel.  There is official channels that
> you are trusting and other channels that you cannot trust.  Well, your

The channel is only the top level. Do I trust the "Guix" channel? More
than other channels, but I don't really know how much the current
maintainers check each individual package submission. They certainly
look at the package definition itself, but do they also check that the
packaged software itself is free from malware? If so, how thorough are
those checks? There are so many possible levels of attack today.

> Well, checking at each command invocation could slow Guix, since it is
> already not the fastest CLI of West. :-)

Such checks could happen at a higher level, e.g. shell or terminal, to
cover not only Guix but also everything else. As Ricardo pointed out,
such checks cannot be perfect, but that's true for spell checkers as
well, which nevertheless turn out to be useful. The goal is not provably
absolute security, but noticeably increased security.

BTW, I consider IT security and reproducibility in research as almost
the same problem. The former's enemy is malice, the latter's enemy
is mistakes, but the common aspect of both is users not fully knowing
which exact software they are running. In reproducibility, typos are a
well-known issue and one reason why we recommend scripting everything,
to turn the random typo into a reproducible typo.

Cheers,
  Konrad.

Re: Getting started with GWL 0.3.0

[zimoun]

Hi Konrad,

On Wed, 24 Mar 2021 at 11:08, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote:
>>> As for trusting channels and packages, this is not much of an issue
>>> today, but if Guix ever becomes as popular as Debian is today, then we
>>> will have plenty of users with no clue about who or what they can trust.
>>
>> ...and you can do the same with any package manager.  For instance,
>
> Yes, exactly. Trusting software sources is becoming an ever more
> important issue everywhere, as people rely on ever more complex software
> assemblies whose components they can no longer verify individually.
> Which is also why package managers now become targets of attacks.

I totally agree.  And currently it is hard to introduce a malware to the
official Guix channel, in the meaning the commits are pushed by a small
set of vouched people.  Which is not the case for these npm and other
PyPI repositories.  Other said, “malware” could only be “mistake” and
not “malice”.

>> The issue at first is the channel.  There is official channels that
>> you are trusting and other channels that you cannot trust.  Well, your
>
> The channel is only the top level. Do I trust the "Guix" channel? More
> than other channels, but I don't really know how much the current
> maintainers check each individual package submission. They certainly
> look at the package definition itself, but do they also check that the
> packaged software itself is free from malware? If so, how thorough are
> those checks? There are so many possible levels of attack today.

I agree, again. :-) However, I miss your security flaw about extensions
because these extensions should come from this “trusted” channel, as any
other packages.

I miss why you trust enough the official channel to install the package
<foo> but not the extension <bar>.  I do not see any difference.  To me,
extensions are Guile programs which are allowed to run with the
command-line call “guix <bar>” and I do not see why the call using
“guix” is more important than other calls; as “git annex”.

Anyway!  We agree that we disagree. :-)

>> Well, checking at each command invocation could slow Guix, since it is
>> already not the fastest CLI of West. :-)
>
> Such checks could happen at a higher level, e.g. shell or terminal, to
> cover not only Guix but also everything else. As Ricardo pointed out,
> such checks cannot be perfect, but that's true for spell checkers as
> well, which nevertheless turn out to be useful. The goal is not provably
> absolute security, but noticeably increased security.

I agree, again again. :-)

BTW, Guix has now a subcommand and option-name dumb recommender for
typo:

--8<---------------cut here---------------start------------->8---
$ guix environement --ad-foc hello
guix: environement: command not found
hint: Did you mean `environment'?

Try `guix --help' for more information.

$ guix environment --ad-foc hello
guix environment: error: ad-foc: unrecognized option
hint: Did you mean `ad-hoc'?
--8<---------------cut here---------------end--------------->8---


> BTW, I consider IT security and reproducibility in research as almost
> the same problem. The former's enemy is malice, the latter's enemy
> is mistakes, but the common aspect of both is users not fully knowing
> which exact software they are running. In reproducibility, typos are a
> well-known issue and one reason why we recommend scripting everything,
> to turn the random typo into a reproducible typo.

I agree, again again again.

Finally, I do not know on what exactly we agree to disagree. ;-)


Cheers,
simon

Re: Getting started with GWL 0.3.0

[Ricardo Wurmus]

Hi Roel,

>> > Is there a feature-branch to try out GWL with Guile-DRMAA? :)
>> 
>> Unfortunately not yet.
>> 
>> I haven’t been 100% successful with the only DRMAA-enabled cluster that
>> I have access to, and it turns out that it’s not as simple as SGE’s
>> “hold_jid”.
>> 
>> It’s no longer “fire and forget”, which is a bit sad, but that’s how
>> DRMAA works.  We need a run-time component that keeps track of
>> submitted
>> jobs and their status and actively starts held jobs when the
>> prerequisites have finished.
>
> That's unfortunate, but I believe having a daemon that keeps track of
> the workflow opens possibilities for "cloud" "orchestration".

Yes, it’s pretty much the same mechanism, except that for the “cloud” we
generally don’t have a ready-made “select” or “wait” equivalent.  There
we would either need to write code that lets the instances contact a
coordination service or let the GWL process poll their status.

With DRMAA it’s pretty simple: we submit all jobs in hold state, then
start the first layer, and then we use the “wait” call to be notified of
any completed job.  The docstring in Guile DRMAA says:

--8<---------------cut here---------------start------------->8---
   "Wait for the completion of a job with identifier JOB-ID.  If the
JOB-ID is the special symbol '*, wait for the completion of any job that
has been submitted during this session.

TIMEOUT (an integer) specifies the number of seconds to block.  If it
is not provided or is #FALSE this procedure will block forever.

This procedure returns three values: the identifier of the job that
has completed, the status code of the job (an opaque value), and an
alist of resource usage statistics."
--8<---------------cut here---------------end--------------->8---

The GWL already knows the graph of processes and each process
corresponds to a submitted job, so with the return values of this
procedure it should really not be complicated to implement.

>> It’s not clear to me if and how we should persist workflow state.  The
>> GWL will submit all jobs to the scheduler in a held state and then
>> change their status when its their turn.  I wonder if and how we should
>> handle the case where the GWL runtime monitor dies and is restarted.
>> The easiest way is to simply kill all queued up jobs, but I don’t know
>> if there’s a better approach.
>> 
>> Ideas?
>
> I find killing/removing queued jobs upon exiting the runtime monitor a
> good idea!

With DRMAA this is very easy.  The “control” procedure allows us to kill
all jobs that were enqueued in the current session.  In Guile DRMAA
that’s

   (control '* 'terminate)

> I have access to a SLURM cluster (I don't know which version of DRMAA
> it supports), but I can test it.

SLURM has an external DRMAA 1.0 implementation; it is not included by
default.  In Guix that’s provided by the slurm-drmaa package.

-- 
Ricardo

Re: Getting started with GWL 0.3.0

[Ricardo Wurmus]

Roel Janssen <roel@gnu.org> writes:

> On Tue, 2021-03-23 at 18:34 +0100, Ricardo Wurmus wrote:
>> 
>> Before you get too enthusiastic about the GWL, though, I’d like 
>> to
>> note
>> that 0.3.0 has a few known bugs that are already fixed in the
>> repository.  I’ve been putting off making a new release until 
>> either
>> Guile-AWS or Guile-DRMAA are ready and usable with the GWL.
>
> Is there a feature-branch to try out GWL with Guile-DRMAA? :)

There is wip-drmaa.

Set the GWL_USE_DRMAA variable and use the default execution 
engine.

I only tested this with a local Slurm cluster on my laptop, but it 
seems to work fine.  I’d be happy to hear test reports!

Before this can make its way into the main branch we have to 
figure out if the concept of an execution engine still makes sense 
the way it is implemented.

-- 
Ricardo