* Getting started with GWL 0.3.0 @ 2021-03-22 10:32 Konrad Hinsen 2021-03-22 11:03 ` zimoun 2021-03-23 15:51 ` Konrad Hinsen 0 siblings, 2 replies; 20+ messages in thread From: Konrad Hinsen @ 2021-03-22 10:32 UTC (permalink / raw) To: gwl-devel Hi everyone, After lurking here for a long time, I am finally starting to explore GWL for real, meaning that I intend to use it for a (small) real project. Unfortunately, my first attempt didn't get very far: $ guix workflow run hello.w guix: workflow: command not found Try `guix --help' for more information. I do have gwl@0.3.0 installed. That made me curious about how GWL actually adds subcommands to "guix", but I quickly discovered that it uses an apparently undocumented extension mechanism by placing scripts under guix/extensions. So... I am lost. How do I actually run a workflow? Cheers, Konrad. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-22 10:32 Getting started with GWL 0.3.0 Konrad Hinsen @ 2021-03-22 11:03 ` zimoun 2021-03-22 13:04 ` Konrad Hinsen 2021-03-23 15:51 ` Konrad Hinsen 1 sibling, 1 reply; 20+ messages in thread From: zimoun @ 2021-03-22 11:03 UTC (permalink / raw) To: Konrad Hinsen, gwl-devel Hi Konrad, On Mon, 22 Mar 2021 at 11:32, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote: > Unfortunately, my first attempt didn't get very far: > > $ guix workflow run hello.w > guix: workflow: command not found > Try `guix --help' for more information. I do not know. I had similar issues though. > I do have gwl@0.3.0 installed. That made me curious about how GWL > actually adds subcommands to "guix", but I quickly discovered > that it uses an apparently undocumented extension mechanism > by placing scripts under guix/extensions. So... I am lost. > How do I actually run a workflow? The plan of the documentation is discussed here: <https://yhetil.org/guix/86k0sekkj8.fsf@gmail.com> This is really cool because “guix repl -- foo.scm arg1 arg2” can now be really handy with “guix foo arg1 arg2”. However, it is still undocumented because we have not reached consensus with Ricardo. :-) See the 3 thread emails starting there: <https://yhetil.org/guix/86k0rsdylk.fsf@gmail.com> Well, about this GUIX_EXTENSIONS_PATHS, as the author of “guix repl --” your feedback is very welcome. :-) Cheers, simon ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-22 11:03 ` zimoun @ 2021-03-22 13:04 ` Konrad Hinsen 2021-03-22 13:51 ` zimoun 0 siblings, 1 reply; 20+ messages in thread From: Konrad Hinsen @ 2021-03-22 13:04 UTC (permalink / raw) To: zimoun, gwl-devel Hi Simon, > The plan of the documentation is discussed here: > > <https://yhetil.org/guix/86k0sekkj8.fsf@gmail.com> Looks like I missed a discussion on guix-devel. My excuse is that I can't keep up with guix-devel any more, it's getting too much! > This is really cool because “guix repl -- foo.scm arg1 arg2” can now be > really handy with “guix foo arg1 arg2”. Handy, yes. But is it a good idea from a security point of view? As a Guix user, I trust "guix" with all its subcommands because I know that all the code is carefully inspected by several competent developers. I don't have the same level of trust in software packaged within Guix. I'd rather see packages building on "guix" but provide their own top-level scripts with distinct names. And support for writing such packages in making it easier to access the user's default Guix profile. > Well, about this GUIX_EXTENSIONS_PATHS, as the author of “guix repl --” > your feedback is very welcome. :-) See above :-) Cheers, Konrad. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-22 13:04 ` Konrad Hinsen @ 2021-03-22 13:51 ` zimoun 2021-03-22 15:07 ` Konrad Hinsen 0 siblings, 1 reply; 20+ messages in thread From: zimoun @ 2021-03-22 13:51 UTC (permalink / raw) To: Konrad Hinsen; +Cc: gwl-devel Hi, On Mon, 22 Mar 2021 at 14:04, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote: > Looks like I missed a discussion on guix-devel. My excuse is that I > can't keep up with guix-devel any more, it's getting too much! Do not worry! And I do not know if these days someone is actually able to grasp all the discussions happening on guix-devel. :-) > > This is really cool because “guix repl -- foo.scm arg1 arg2” can now be > > really handy with “guix foo arg1 arg2”. > > Handy, yes. But is it a good idea from a security point of view? As a > Guix user, I trust "guix" with all its subcommands because I know that > all the code is carefully inspected by several competent developers. I > don't have the same level of trust in software packaged within Guix. What do you mean? The user has to explicitly set GUIX_EXTENSIONS_PATH or explicitly install a package (or a channel, as "guix home"). I do not see where there is a security flaw, I mean it is the same vulnerability as for "guix repl -- foo.scm" or as for "guix install foo && foo". And if you worry, I guess you can run GWL in a container, something like; guix environment -C --ad-hoc gwl -- guix workflow > I'd rather see packages building on "guix" but provide their own > top-level scripts with distinct names. And support for writing such > packages in making it easier to access the user's default Guix profile. Personally, I like the idea of extensions. Similarly as "git foo" works if "git-foo" is an executable on the PATH. I imagine couple of extensions. For instance, testing idea on UI is hard because Guix itself is really conservative about the backward compatibility---for a good! :-) And we can imagine extensions as a way to test other flavours, either before introducing a new subcommand or either as a replacement of current subcommand. Cheers, simon ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-22 13:51 ` zimoun @ 2021-03-22 15:07 ` Konrad Hinsen 2021-03-22 18:16 ` zimoun 0 siblings, 1 reply; 20+ messages in thread From: Konrad Hinsen @ 2021-03-22 15:07 UTC (permalink / raw) To: zimoun; +Cc: gwl-devel zimoun <zimon.toutoune@gmail.com> writes: > The user has to explicitly set GUIX_EXTENSIONS_PATH or explicitly > install a package (or a channel, as "guix home"). I do not see where > there is a security flaw, I mean it is the same vulnerability as for > "guix repl -- foo.scm" or as for "guix install foo && foo". The vulnerability level is the same, but a typical user's expectations are not. When I run a script via "guix repl", it is perfectly obvious to me that that script is not part of Guix. When I run the command "foo", it is also perfectly obvious to me that "foo" is not part of Guix. But "guix workflow" looks as if it were part of Guix. Guix users need a significant level of trust in Guix and its developers in order to use Guix. Attackers could exploit this trust by tricking users into adding malicious code (via a channel, for example) that takes the form of a Guix extension. People do run downloaded bash scripts with root permissions, after all (e.g. for installing Guix ;-). > And if you worry, I guess you can run GWL in a container, something like; > > guix environment -C --ad-hoc gwl -- guix workflow Sure, but that's not the issue. The issue is being aware that I have no particular reason to trust "guix workflow" as much as I trust "guix package". Of course GWL is a bad example because its developers are a subset of Guix developers. But the extension mechanism looks open to everyone. Imagine a package "bitcoin-utils" that installs, among lots of other stuff, an extension "guix pul" that starts a bitcoin-mining background process whenever an unsuspecting user mistypes a frequent Guix command. Cheers, Konrad. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-22 15:07 ` Konrad Hinsen @ 2021-03-22 18:16 ` zimoun 2021-03-23 12:57 ` Konrad Hinsen 0 siblings, 1 reply; 20+ messages in thread From: zimoun @ 2021-03-22 18:16 UTC (permalink / raw) To: Konrad Hinsen; +Cc: gwl-devel Hi, On Mon, 22 Mar 2021 at 16:07, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote: > zimoun <zimon.toutoune@gmail.com> writes: > The vulnerability level is the same, but a typical user's expectations > are not. When I run a script via "guix repl", it is perfectly obvious to > me that that script is not part of Guix. When I run the command "foo", > it is also perfectly obvious to me that "foo" is not part of Guix. But > "guix workflow" looks as if it were part of Guix. > > Guix users need a significant level of trust in Guix and its developers > in order to use Guix. Attackers could exploit this trust by tricking > users into adding malicious code (via a channel, for example) that takes > the form of a Guix extension. People do run downloaded bash scripts > with root permissions, after all (e.g. for installing Guix ;-). > > > And if you worry, I guess you can run GWL in a container, something like; > > > > guix environment -C --ad-hoc gwl -- guix workflow > > Sure, but that's not the issue. The issue is being aware that I have no > particular reason to trust "guix workflow" as much as I trust "guix > package". [...] > Imagine a package "bitcoin-utils" that installs, among lots of other > stuff, an extension "guix pul" that starts a bitcoin-mining background > process whenever an unsuspecting user mistypes a frequent Guix command. Well, I understand your concerns but I am not convinced to share them. IIUC, you are saying that "git annex" or "git lfs" which are extensions to Git are a security issue because if any malware-package providing a "git-pul" malware, then a user typing "git pul" with a typo can have bad surprise. But at first, you need to trust a channel providing this malware-package, then second you need to install this malware-package and third make the typo. And what about "guxi pull" where the typo is not on the subcommand but on the command itself? To me, being handy beats the concern. :-) It is the responsability of the user to know what they is installing and running on their own machine. Cheers, simon ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-22 18:16 ` zimoun @ 2021-03-23 12:57 ` Konrad Hinsen 2021-03-23 13:16 ` Ricardo Wurmus 2021-03-23 20:16 ` zimoun 0 siblings, 2 replies; 20+ messages in thread From: Konrad Hinsen @ 2021-03-23 12:57 UTC (permalink / raw) To: zimoun; +Cc: gwl-devel Hi Simon, > Well, I understand your concerns but I am not convinced to share them. We can certainly agree to disagree! > IIUC, you are saying that "git annex" or "git lfs" which are > extensions to Git are a security issue because if any malware-package > providing a "git-pul" malware, then a user typing "git pul" with a Yes, exactly. Like what happened to npm: https://threatpost.com/attackers-use-typo-squatting-to-steal-npm-credentials/127235/ Apparently this is now called typo-squatting. > typo can have bad surprise. But at first, you need to trust a channel > providing this malware-package, then second you need to install this > malware-package and third make the typo. The last part comes with zero effort :-) As for trusting channels and packages, this is not much of an issue today, but if Guix ever becomes as popular as Debian is today, then we will have plenty of users with no clue about who or what they can trust. In the long run, maybe a command spell-checker would be a nice way out. Some piece of software that decides, based on my command history, whether a command I type is more likely a typo or the intention to run some exotic software. Cheers, Konrad. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-23 12:57 ` Konrad Hinsen @ 2021-03-23 13:16 ` Ricardo Wurmus 2021-03-23 13:24 ` Roel Janssen 2021-03-23 20:16 ` zimoun 1 sibling, 1 reply; 20+ messages in thread From: Ricardo Wurmus @ 2021-03-23 13:16 UTC (permalink / raw) To: Konrad Hinsen; +Cc: gwl-devel Konrad Hinsen <konrad.hinsen@fastmail.net> writes: > In the long run, maybe a command spell-checker would be a nice way out. > Some piece of software that decides, based on my command history, > whether a command I type is more likely a typo or the intention to run > some exotic software. This would need more work to allow replacement of Guix commands with an extension. For example, I may want to use an extension that replaces “guix deploy” with a variant that can deploy to AWS, falling back to the default “guix deploy” for all other deployments. We should not prevent this use case, so at the very least this needs to be configurable (though I’d be happier if we could come up with a configuration-free solution). A configuration-free alternative could be “guix ext pull” to explicitly select the extension named “pull”. There can be no conflicts with this scheme, but it’s more work to type. -- Ricardo ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-23 13:16 ` Ricardo Wurmus @ 2021-03-23 13:24 ` Roel Janssen 0 siblings, 0 replies; 20+ messages in thread From: Roel Janssen @ 2021-03-23 13:24 UTC (permalink / raw) To: gwl-devel On 3/23/21 2:16 PM, Ricardo Wurmus wrote: > Konrad Hinsen <konrad.hinsen@fastmail.net> writes: > >> In the long run, maybe a command spell-checker would be a nice way out. >> Some piece of software that decides, based on my command history, >> whether a command I type is more likely a typo or the intention to run >> some exotic software. > This would need more work to allow replacement of Guix commands with an > extension. For example, I may want to use an extension that replaces > “guix deploy” with a variant that can deploy to AWS, falling back to the > default “guix deploy” for all other deployments. > > We should not prevent this use case, so at the very least this needs to > be configurable (though I’d be happier if we could come up with a > configuration-free solution). > > A configuration-free alternative could be “guix ext pull” to explicitly > select the extension named “pull”. There can be no conflicts with this > scheme, but it’s more work to type. > Or perhaps it could work with a warning message when invoking an extension command for the first time (similar to what Emacs does when hitting a key combination that may be unintended). For example: $ guix workflow ... You are about to run an extension to GNU Guix. Enable this extension for future use? [y/N] Perhaps this message could include the package from which the extension originates. Kind regards, Roel Janssen ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-23 12:57 ` Konrad Hinsen 2021-03-23 13:16 ` Ricardo Wurmus @ 2021-03-23 20:16 ` zimoun 2021-03-24 10:08 ` Konrad Hinsen 1 sibling, 1 reply; 20+ messages in thread From: zimoun @ 2021-03-23 20:16 UTC (permalink / raw) To: Konrad Hinsen; +Cc: gwl-devel Hi Konrad, On Tue, 23 Mar 2021 at 13:57, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote: > Yes, exactly. Like what happened to npm: > > https://threatpost.com/attackers-use-typo-squatting-to-steal-npm-credentials/127235/ > > Apparently this is now called typo-squatting. And PyPI and.. whatever "store" with no clear policy... > > typo can have bad surprise. But at first, you need to trust a channel > > providing this malware-package, then second you need to install this > > malware-package and third make the typo. > > The last part comes with zero effort :-) > > As for trusting channels and packages, this is not much of an issue > today, but if Guix ever becomes as popular as Debian is today, then we > will have plenty of users with no clue about who or what they can trust. ...and you can do the same with any package manager. For instance, add an evil channel to your /etc/apt/sources.list, then "apt-get update" and then you have no guarantee on what you are installing, for example, "apt-get install git" will install regular Git with a git-pul typo malware, or even the Git version you get is a malware. The issue at first is the channel. There is official channels that you are trusting and other channels that you cannot trust. Well, your concern is about any a suspicious channel, for instance, if I blindly install the python-paper-foo package then I have no guarantee that this package also install the malicious 'emasc' binary which is a typo I often make. > In the long run, maybe a command spell-checker would be a nice way out. > Some piece of software that decides, based on my command history, > whether a command I type is more likely a typo or the intention to run > some exotic software. Well, checking at each command invocation could slow Guix, since it is already not the fastest CLI of West. :-) Cheers, simon ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-23 20:16 ` zimoun @ 2021-03-24 10:08 ` Konrad Hinsen 2021-03-24 10:44 ` zimoun 0 siblings, 1 reply; 20+ messages in thread From: Konrad Hinsen @ 2021-03-24 10:08 UTC (permalink / raw) To: zimoun; +Cc: gwl-devel Hi Simon, >> As for trusting channels and packages, this is not much of an issue >> today, but if Guix ever becomes as popular as Debian is today, then we >> will have plenty of users with no clue about who or what they can trust. > > ...and you can do the same with any package manager. For instance, Yes, exactly. Trusting software sources is becoming an ever more important issue everywhere, as people rely on ever more complex software assemblies whose components they can no longer verify individually. Which is also why package managers now become targets of attacks. > The issue at first is the channel. There is official channels that > you are trusting and other channels that you cannot trust. Well, your The channel is only the top level. Do I trust the "Guix" channel? More than other channels, but I don't really know how much the current maintainers check each individual package submission. They certainly look at the package definition itself, but do they also check that the packaged software itself is free from malware? If so, how thorough are those checks? There are so many possible levels of attack today. > Well, checking at each command invocation could slow Guix, since it is > already not the fastest CLI of West. :-) Such checks could happen at a higher level, e.g. shell or terminal, to cover not only Guix but also everything else. As Ricardo pointed out, such checks cannot be perfect, but that's true for spell checkers as well, which nevertheless turn out to be useful. The goal is not provably absolute security, but noticeably increased security. BTW, I consider IT security and reproducibility in research as almost the same problem. The former's enemy is malice, the latter's enemy is mistakes, but the common aspect of both is users not fully knowing which exact software they are running. In reproducibility, typos are a well-known issue and one reason why we recommend scripting everything, to turn the random typo into a reproducible typo. Cheers, Konrad. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-24 10:08 ` Konrad Hinsen @ 2021-03-24 10:44 ` zimoun 0 siblings, 0 replies; 20+ messages in thread From: zimoun @ 2021-03-24 10:44 UTC (permalink / raw) To: Konrad Hinsen; +Cc: gwl-devel Hi Konrad, On Wed, 24 Mar 2021 at 11:08, Konrad Hinsen <konrad.hinsen@fastmail.net> wrote: >>> As for trusting channels and packages, this is not much of an issue >>> today, but if Guix ever becomes as popular as Debian is today, then we >>> will have plenty of users with no clue about who or what they can trust. >> >> ...and you can do the same with any package manager. For instance, > > Yes, exactly. Trusting software sources is becoming an ever more > important issue everywhere, as people rely on ever more complex software > assemblies whose components they can no longer verify individually. > Which is also why package managers now become targets of attacks. I totally agree. And currently it is hard to introduce a malware to the official Guix channel, in the meaning the commits are pushed by a small set of vouched people. Which is not the case for these npm and other PyPI repositories. Other said, “malware” could only be “mistake” and not “malice”. >> The issue at first is the channel. There is official channels that >> you are trusting and other channels that you cannot trust. Well, your > > The channel is only the top level. Do I trust the "Guix" channel? More > than other channels, but I don't really know how much the current > maintainers check each individual package submission. They certainly > look at the package definition itself, but do they also check that the > packaged software itself is free from malware? If so, how thorough are > those checks? There are so many possible levels of attack today. I agree, again. :-) However, I miss your security flaw about extensions because these extensions should come from this “trusted” channel, as any other packages. I miss why you trust enough the official channel to install the package <foo> but not the extension <bar>. I do not see any difference. To me, extensions are Guile programs which are allowed to run with the command-line call “guix <bar>” and I do not see why the call using “guix” is more important than other calls; as “git annex”. Anyway! We agree that we disagree. :-) >> Well, checking at each command invocation could slow Guix, since it is >> already not the fastest CLI of West. :-) > > Such checks could happen at a higher level, e.g. shell or terminal, to > cover not only Guix but also everything else. As Ricardo pointed out, > such checks cannot be perfect, but that's true for spell checkers as > well, which nevertheless turn out to be useful. The goal is not provably > absolute security, but noticeably increased security. I agree, again again. :-) BTW, Guix has now a subcommand and option-name dumb recommender for typo: --8<---------------cut here---------------start------------->8--- $ guix environement --ad-foc hello guix: environement: command not found hint: Did you mean `environment'? Try `guix --help' for more information. $ guix environment --ad-foc hello guix environment: error: ad-foc: unrecognized option hint: Did you mean `ad-hoc'? --8<---------------cut here---------------end--------------->8--- > BTW, I consider IT security and reproducibility in research as almost > the same problem. The former's enemy is malice, the latter's enemy > is mistakes, but the common aspect of both is users not fully knowing > which exact software they are running. In reproducibility, typos are a > well-known issue and one reason why we recommend scripting everything, > to turn the random typo into a reproducible typo. I agree, again again again. Finally, I do not know on what exactly we agree to disagree. ;-) Cheers, simon ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-22 10:32 Getting started with GWL 0.3.0 Konrad Hinsen 2021-03-22 11:03 ` zimoun @ 2021-03-23 15:51 ` Konrad Hinsen 2021-03-23 17:34 ` Ricardo Wurmus 1 sibling, 1 reply; 20+ messages in thread From: Konrad Hinsen @ 2021-03-23 15:51 UTC (permalink / raw) To: gwl-devel Konrad Hinsen <konrad.hinsen@fastmail.net> writes: > Unfortunately, my first attempt didn't get very far: > > $ guix workflow run hello.w > guix: workflow: command not found > Try `guix --help' for more information. That was with Guix installed under Ubuntu. Today I tried on a VM running Guix System. Result: the same. So... how do you all actually use GWL? Should I try to build/run from source? I don't see any instructions for that either, nor any pointer to the source repo. Cheers, Konrad. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-23 15:51 ` Konrad Hinsen @ 2021-03-23 17:34 ` Ricardo Wurmus 2021-03-23 19:30 ` Roel Janssen 2021-03-24 9:52 ` Konrad Hinsen 0 siblings, 2 replies; 20+ messages in thread From: Ricardo Wurmus @ 2021-03-23 17:34 UTC (permalink / raw) To: Konrad Hinsen; +Cc: gwl-devel Konrad Hinsen <konrad.hinsen@fastmail.net> writes: > Konrad Hinsen <konrad.hinsen@fastmail.net> writes: > >> Unfortunately, my first attempt didn't get very far: >> >> $ guix workflow run hello.w >> guix: workflow: command not found >> Try `guix --help' for more information. > > That was with Guix installed under Ubuntu. Today I tried on a VM > running Guix System. Result: the same. > > So... how do you all actually use GWL? Should I try to build/run > from source? I don't see any instructions for that either, nor > any pointer to the source repo. I should add this to the manual. Here’s what I do: --8<---------------cut here---------------start------------->8--- $ guix install gwl $ export GUIX_EXTENSIONS_PATH=$HOME/.guix-profile/share/guix/extensions/ $ guix workflow --help Usage: guix workflow Subcommands: graph Visualize a workflow run Run a workflow web Start the web interface The Guix Workflow Language (GWL) is a scientific computing extension to the Guix package manager. It combines the specification of work units and their relationship to one another with the reproducible software deployment facilities of the functional package manager GNU Guix. A GWL workflow will always run in a reproducible environment that GNU Guix automatically prepares. The GWL extends your Guix installation with a single new sub-command: guix workflow. $ --8<---------------cut here---------------end--------------->8--- Before you get too enthusiastic about the GWL, though, I’d like to note that 0.3.0 has a few known bugs that are already fixed in the repository. I’ve been putting off making a new release until either Guile-AWS or Guile-DRMAA are ready and usable with the GWL. -- Ricardo ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-23 17:34 ` Ricardo Wurmus @ 2021-03-23 19:30 ` Roel Janssen 2021-03-23 20:14 ` Ricardo Wurmus 2021-04-30 21:50 ` Ricardo Wurmus 2021-03-24 9:52 ` Konrad Hinsen 1 sibling, 2 replies; 20+ messages in thread From: Roel Janssen @ 2021-03-23 19:30 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: gwl-devel On Tue, 2021-03-23 at 18:34 +0100, Ricardo Wurmus wrote: > > Before you get too enthusiastic about the GWL, though, I’d like to > note > that 0.3.0 has a few known bugs that are already fixed in the > repository. I’ve been putting off making a new release until either > Guile-AWS or Guile-DRMAA are ready and usable with the GWL. Is there a feature-branch to try out GWL with Guile-DRMAA? :) Kind regards, Roel Janssen ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-23 19:30 ` Roel Janssen @ 2021-03-23 20:14 ` Ricardo Wurmus 2021-03-23 20:30 ` Roel Janssen 2021-04-30 21:50 ` Ricardo Wurmus 1 sibling, 1 reply; 20+ messages in thread From: Ricardo Wurmus @ 2021-03-23 20:14 UTC (permalink / raw) To: Roel Janssen; +Cc: gwl-devel Roel Janssen <roel@gnu.org> writes: > On Tue, 2021-03-23 at 18:34 +0100, Ricardo Wurmus wrote: >> >> Before you get too enthusiastic about the GWL, though, I’d like to >> note >> that 0.3.0 has a few known bugs that are already fixed in the >> repository. I’ve been putting off making a new release until either >> Guile-AWS or Guile-DRMAA are ready and usable with the GWL. > > Is there a feature-branch to try out GWL with Guile-DRMAA? :) Unfortunately not yet. I haven’t been 100% successful with the only DRMAA-enabled cluster that I have access to, and it turns out that it’s not as simple as SGE’s “hold_jid”. It’s no longer “fire and forget”, which is a bit sad, but that’s how DRMAA works. We need a run-time component that keeps track of submitted jobs and their status and actively starts held jobs when the prerequisites have finished. It’s not clear to me if and how we should persist workflow state. The GWL will submit all jobs to the scheduler in a held state and then change their status when its their turn. I wonder if and how we should handle the case where the GWL runtime monitor dies and is restarted. The easiest way is to simply kill all queued up jobs, but I don’t know if there’s a better approach. Ideas? -- Ricardo ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-23 20:14 ` Ricardo Wurmus @ 2021-03-23 20:30 ` Roel Janssen 2021-03-26 21:01 ` Ricardo Wurmus 0 siblings, 1 reply; 20+ messages in thread From: Roel Janssen @ 2021-03-23 20:30 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: gwl-devel On Tue, 2021-03-23 at 21:14 +0100, Ricardo Wurmus wrote: > > Roel Janssen <roel@gnu.org> writes: > > > On Tue, 2021-03-23 at 18:34 +0100, Ricardo Wurmus wrote: > > > > > > Before you get too enthusiastic about the GWL, though, I’d like to > > > note > > > that 0.3.0 has a few known bugs that are already fixed in the > > > repository. I’ve been putting off making a new release until > > > either > > > Guile-AWS or Guile-DRMAA are ready and usable with the GWL. > > > > Is there a feature-branch to try out GWL with Guile-DRMAA? :) > > Unfortunately not yet. > > I haven’t been 100% successful with the only DRMAA-enabled cluster that > I have access to, and it turns out that it’s not as simple as SGE’s > “hold_jid”. > > It’s no longer “fire and forget”, which is a bit sad, but that’s how > DRMAA works. We need a run-time component that keeps track of > submitted > jobs and their status and actively starts held jobs when the > prerequisites have finished. That's unfortunate, but I believe having a daemon that keeps track of the workflow opens possibilities for "cloud" "orchestration". > It’s not clear to me if and how we should persist workflow state. The > GWL will submit all jobs to the scheduler in a held state and then > change their status when its their turn. I wonder if and how we should > handle the case where the GWL runtime monitor dies and is restarted. > The easiest way is to simply kill all queued up jobs, but I don’t know > if there’s a better approach. > > Ideas? I find killing/removing queued jobs upon exiting the runtime monitor a good idea! Maybe not suitable anymore, but I wrote a "qsub" command that translates to "squeue" here: https://github.com/roelj/qsub-slurm Could we use the same approach? It works because jobs are submitted in order. The look-up mechanism can be found here: https://github.com/roelj/qsub-slurm/blob/master/qsub.in#L233-L253 I have access to a SLURM cluster (I don't know which version of DRMAA it supports), but I can test it. Kind regards, Roel Janssen ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-23 20:30 ` Roel Janssen @ 2021-03-26 21:01 ` Ricardo Wurmus 0 siblings, 0 replies; 20+ messages in thread From: Ricardo Wurmus @ 2021-03-26 21:01 UTC (permalink / raw) To: Roel Janssen; +Cc: gwl-devel Hi Roel, >> > Is there a feature-branch to try out GWL with Guile-DRMAA? :) >> >> Unfortunately not yet. >> >> I haven’t been 100% successful with the only DRMAA-enabled cluster that >> I have access to, and it turns out that it’s not as simple as SGE’s >> “hold_jid”. >> >> It’s no longer “fire and forget”, which is a bit sad, but that’s how >> DRMAA works. We need a run-time component that keeps track of >> submitted >> jobs and their status and actively starts held jobs when the >> prerequisites have finished. > > That's unfortunate, but I believe having a daemon that keeps track of > the workflow opens possibilities for "cloud" "orchestration". Yes, it’s pretty much the same mechanism, except that for the “cloud” we generally don’t have a ready-made “select” or “wait” equivalent. There we would either need to write code that lets the instances contact a coordination service or let the GWL process poll their status. With DRMAA it’s pretty simple: we submit all jobs in hold state, then start the first layer, and then we use the “wait” call to be notified of any completed job. The docstring in Guile DRMAA says: --8<---------------cut here---------------start------------->8--- "Wait for the completion of a job with identifier JOB-ID. If the JOB-ID is the special symbol '*, wait for the completion of any job that has been submitted during this session. TIMEOUT (an integer) specifies the number of seconds to block. If it is not provided or is #FALSE this procedure will block forever. This procedure returns three values: the identifier of the job that has completed, the status code of the job (an opaque value), and an alist of resource usage statistics." --8<---------------cut here---------------end--------------->8--- The GWL already knows the graph of processes and each process corresponds to a submitted job, so with the return values of this procedure it should really not be complicated to implement. >> It’s not clear to me if and how we should persist workflow state. The >> GWL will submit all jobs to the scheduler in a held state and then >> change their status when its their turn. I wonder if and how we should >> handle the case where the GWL runtime monitor dies and is restarted. >> The easiest way is to simply kill all queued up jobs, but I don’t know >> if there’s a better approach. >> >> Ideas? > > I find killing/removing queued jobs upon exiting the runtime monitor a > good idea! With DRMAA this is very easy. The “control” procedure allows us to kill all jobs that were enqueued in the current session. In Guile DRMAA that’s (control '* 'terminate) > I have access to a SLURM cluster (I don't know which version of DRMAA > it supports), but I can test it. SLURM has an external DRMAA 1.0 implementation; it is not included by default. In Guix that’s provided by the slurm-drmaa package. -- Ricardo ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-23 19:30 ` Roel Janssen 2021-03-23 20:14 ` Ricardo Wurmus @ 2021-04-30 21:50 ` Ricardo Wurmus 1 sibling, 0 replies; 20+ messages in thread From: Ricardo Wurmus @ 2021-04-30 21:50 UTC (permalink / raw) To: Roel Janssen; +Cc: gwl-devel Roel Janssen <roel@gnu.org> writes: > On Tue, 2021-03-23 at 18:34 +0100, Ricardo Wurmus wrote: >> >> Before you get too enthusiastic about the GWL, though, I’d like >> to >> note >> that 0.3.0 has a few known bugs that are already fixed in the >> repository. I’ve been putting off making a new release until >> either >> Guile-AWS or Guile-DRMAA are ready and usable with the GWL. > > Is there a feature-branch to try out GWL with Guile-DRMAA? :) There is wip-drmaa. Set the GWL_USE_DRMAA variable and use the default execution engine. I only tested this with a local Slurm cluster on my laptop, but it seems to work fine. I’d be happy to hear test reports! Before this can make its way into the main branch we have to figure out if the concept of an execution engine still makes sense the way it is implemented. -- Ricardo ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: Getting started with GWL 0.3.0 2021-03-23 17:34 ` Ricardo Wurmus 2021-03-23 19:30 ` Roel Janssen @ 2021-03-24 9:52 ` Konrad Hinsen 1 sibling, 0 replies; 20+ messages in thread From: Konrad Hinsen @ 2021-03-24 9:52 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: gwl-devel Hi Ricardo, > I should add this to the manual. Here’s what I do: > > --8<---------------cut here---------------start------------->8--- > $ guix install gwl > $ export GUIX_EXTENSIONS_PATH=$HOME/.guix-profile/share/guix/extensions/ > $ guix workflow --help Thanks, that works for me as well. I'll go back to my playground now :-) > Before you get too enthusiastic about the GWL, though, I’d like to note > that 0.3.0 has a few known bugs that are already fixed in the > repository. I’ve been putting off making a new release until either > Guile-AWS or Guile-DRMAA are ready and usable with the GWL. OK, if I hit anything strange I will rebuild from source before complaining. I actually found the source repository by following the link to the source code for the Web site. Cheers, Konrad. ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2021-04-30 21:52 UTC | newest] Thread overview: 20+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-03-22 10:32 Getting started with GWL 0.3.0 Konrad Hinsen 2021-03-22 11:03 ` zimoun 2021-03-22 13:04 ` Konrad Hinsen 2021-03-22 13:51 ` zimoun 2021-03-22 15:07 ` Konrad Hinsen 2021-03-22 18:16 ` zimoun 2021-03-23 12:57 ` Konrad Hinsen 2021-03-23 13:16 ` Ricardo Wurmus 2021-03-23 13:24 ` Roel Janssen 2021-03-23 20:16 ` zimoun 2021-03-24 10:08 ` Konrad Hinsen 2021-03-24 10:44 ` zimoun 2021-03-23 15:51 ` Konrad Hinsen 2021-03-23 17:34 ` Ricardo Wurmus 2021-03-23 19:30 ` Roel Janssen 2021-03-23 20:14 ` Ricardo Wurmus 2021-03-23 20:30 ` Roel Janssen 2021-03-26 21:01 ` Ricardo Wurmus 2021-04-30 21:50 ` Ricardo Wurmus 2021-03-24 9:52 ` Konrad Hinsen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).