Hi Simon, > The use-case “Containerized workflow in containerized processes” appears > to me interesting. :-) > > It is almost done by design with GWL, no? That's an interesting observation. But I don't see anything in the GWL manual that explains how GWL manages processes. The chapter "Process engines" says: "The simplest way is to turn the workflow into a Guile script that sets up the desired environment and then executes the workflow processes on the current machine. Fine, but is that script run in a container? If not, the programs and code snippets from that process could run arbitrary binaries from the file system. In the examples shown, the workflow itself is launched from the command line, so it is not running in a container either. In principle, the Guile script defining the workflow could access arbitrary files, and thus not be reproducible. I suspect that the risk is low in practice, because I see no good reason for doing this. Cheers, Konrad > > -------------------- Start of forwarded message -------------------- > From: Konrad Hinsen <konrad.hinsen@fastmail.net> > To: Simon Tournier <zimon.toutoune@gmail.com>, Guix Devel <guix-devel@gnu.org> > Subject: Re: Using Guix inside a Guix container > Date: Sat, 18 Feb 2023 10:21:52 +0100 > > Hi Simon, > >> Which part of Guix do you need inside the containerized shell that you >> cannot do outside? > > That's not the right question. There's always a way to do what I want to > do outside. But that may be very inconvenient. > >> Considering your use-case with Snakemake, what I am doing is to wrap >> each rule with one containerized Guix shell which controls the >> permissions, rule by rule; or a big containerized shell: >> >> guix shell -C -m manifest.scm --expose=… > > Nice example. I do the same: "guix shell" in every rule. Then I add > stuff to my Snakefile, which is a Python script after all. For example, > I import pandas to read a data frame from which I construct my workflow. > Now I am at the point where I'd like to run snakemake itself in a > container, to manage the dependencies of my Snakefile. In fact, given > that I have workflows that depend on specific Snakemake versions, I'd > really like to run Snakemake in a container all the time, even without > additional dependencies. > > Without nested containers, I have to go through all the rules, collect > the packages from their manifest files (or command line), and add them > to the container in which I run the whole workflow. Possible, but not > convenient. > > Another example: I run command-line programs from my Pharo image, and I > have developed the habit of doing this always through Guix. The > advantage is that my Pharo code becomes portable: it depends on Guix, > but not on my profile. > > But if I want, one day, to move on to a full Guix system, I have to run > Pharo in a container with LFS simulation. And then all my command line > shell-outs will break. > > Both examples are about composing tools freely, without worrying if they > use Guix internally or now. > > Cheers, > Konrad > > -------------------- End of forwarded message -------------------- > > Cheers, > simon
Hi, From thread in guix-devel: Using Guix inside a Guix container Sat, 18 Feb 2023 11:01:50 +0100 id:m1mt5uk7y1.fsf@fastmail.net https://yhetil.org/guix/m1mt5uk7y1.fsf@fastmail.net/#r The use-case “Containerized workflow in containerized processes” appears to me interesting. :-) It is almost done by design with GWL, no? -------------------- Start of forwarded message -------------------- From: Konrad Hinsen <konrad.hinsen@fastmail.net> To: Simon Tournier <zimon.toutoune@gmail.com>, Guix Devel <guix-devel@gnu.org> Subject: Re: Using Guix inside a Guix container Date: Sat, 18 Feb 2023 10:21:52 +0100 Hi Simon, > Which part of Guix do you need inside the containerized shell that you > cannot do outside? That's not the right question. There's always a way to do what I want to do outside. But that may be very inconvenient. > Considering your use-case with Snakemake, what I am doing is to wrap > each rule with one containerized Guix shell which controls the > permissions, rule by rule; or a big containerized shell: > > guix shell -C -m manifest.scm --expose=… Nice example. I do the same: "guix shell" in every rule. Then I add stuff to my Snakefile, which is a Python script after all. For example, I import pandas to read a data frame from which I construct my workflow. Now I am at the point where I'd like to run snakemake itself in a container, to manage the dependencies of my Snakefile. In fact, given that I have workflows that depend on specific Snakemake versions, I'd really like to run Snakemake in a container all the time, even without additional dependencies. Without nested containers, I have to go through all the rules, collect the packages from their manifest files (or command line), and add them to the container in which I run the whole workflow. Possible, but not convenient. Another example: I run command-line programs from my Pharo image, and I have developed the habit of doing this always through Guix. The advantage is that my Pharo code becomes portable: it depends on Guix, but not on my profile. But if I want, one day, to move on to a full Guix system, I have to run Pharo in a container with LFS simulation. And then all my command line shell-outs will break. Both examples are about composing tools freely, without worrying if they use Guix internally or now. Cheers, Konrad -------------------- End of forwarded message -------------------- Cheers, simon
From: Jelle Licht <jlicht@fsfe.org> * doc/gwl.texi (Options for guix workflow run): Add missing name to example workflow to demonstrate input file mapping. --- doc/gwl.texi | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/gwl.texi b/doc/gwl.texi index 10fe896..6a44953 100644 --- a/doc/gwl.texi +++ b/doc/gwl.texi @@ -1473,7 +1473,7 @@ process state-the-obvious echo "This is a genome: @{@{inputs@}@}" > @{@{outputs@}@} @} -workflow +workflow captain processes list state-the-obvious @end lisp base-commit: 84f7d654278aae191c890e301dc0c05380df9421 -- 2.38.1
[-- Attachment #1: Type: text/plain, Size: 2680 bytes --] We are pleased to announce the release of the GNU Guix Workflow Language version 0.5.1, representing 17 commits by two people, incorporating the results of productive discussions among a number of helpful people on the #guix and #guix-hpc IRC channels on libera.chat and on the gwl-devel@gnu.org mailing list. This is a maintenance release with a number of bug fixes. For details see the NEWS excerpt below. See also the manual: https://workflows.guix.info/manual/ • About The Guix Workflow Language (GWL) provides an extension to GNU Guix's declarative language for package management to automate the execution of programs in scientific workflows. The GWL can use process engines to integrate with various computing environments. • Download Here are the compressed sources and a GPG detached signature[*]: https://ftpmirror.gnu.org/gwl/gwl-0.5.1.tar.gz https://ftpmirror.gnu.org/gwl/gwl-0.5.1.tar.gz.sig Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html [*] Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this: gpg --verify gwl-0.5.1.tar.gz.sig If that command fails because you don't have the required public key, then run this command to import it: gpg --keyserver keys.gnupg.net --recv-keys BCA689B636553801C3C62150197A5888235FACAC and rerun the 'gpg --verify' command. This release was bootstrapped with the following tools: Autoconf 2.69 Automake 1.16.3 Gnulib v0.1-3269-g03d7a6b1f NEWS * Changes in 0.5.1 (since 0.5.0) ** Package handling - Packages are now handled as Guix manifests instead of plain lists. This makes handling of package outputs nondestructive. It also fixes an unrelated issue in the ordering of process packages – the implicit bash-minimal is now ordered last. ** Bug fixes - Fix argument parsing handler for =max-file-size= option. - Fix argument paring validation for =workflow-directory= option. - Fix secondary argument parsing for input mapping. This was likely broken by commit 2a5a34bc2062ccd04d33d61530a1e9ed03140d82 that causes “--” to be passed as an argument to the GWL entry point. - Use an identity comparison for all lookups of processes in hash tables. This is a continuation of work done in 0.5.0. - Remove over-zealous caching of processes based on identical procedures. This would ignore differences in the execution environment. ** Miscellaneous - The contents of input files are now hashed instead of just their metadata. -- Ricardo [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 511 bytes --]
On Thu, 25 Aug 2022, Ricardo Wurmus <rekado@elephly.net> wrote:
> Hi Liliana,
>
>> This makes handling of package outputs nondestructive. It also fixes
>> an unrelated issue in the ordering of process packages – the implicit
>> bash-minimal is now ordered *last*.
>
> Thank you for the patch, and my apologies for the long delay in
> responding!
>
> I haven’t looked over it yet; since this essentially replaces Olivier’s
> previous implementation I’d like to ask:
>
> @Olivier, does this patch address your use-case satisfactorily?
Tested-by: Olivier Dion <olivier-dion@proton.me>
--
Olivier Dion
oldiob.dev
Hi Liliana,
> This makes handling of package outputs nondestructive. It also fixes
> an unrelated issue in the ordering of process packages – the implicit
> bash-minimal is now ordered *last*.
Thank you for the patch, and my apologies for the long delay in
responding!
I haven’t looked over it yet; since this essentially replaces Olivier’s
previous implementation I’d like to ask:
@Olivier, does this patch address your use-case satisfactorily?
--
Ricardo
Hi, I am late to the party. :-) On Mon, 06 Jun 2022 at 10:50, Ricardo Wurmus <rekado@elephly.net> wrote: > It is not entirely surprising to me that the GWL can express this, > because it has really simple abstractions: that of a process and that of > a workflow consisting of processes. [...] > Perhaps there is space for a different tool that takes lessons from the > GWL and Scsh alike, with a focus on command composition and shell > abstractions. Perhaps that tool already exists and is called Metabash: > > https://github.com/artyom-poptsov/metabash From my understanding, metabash allows to remotely run processes, i.e., distribute the pipeline. Somehow, it could be see as an extension of Scsh. However, a pipeline is a linear sequence of processes. When a workflow is a DAG of processes. Therefore, it would appear difficult to me to be able to express a build-system using only pipelines. Last, it appears to me expected that GWL could be considered as a build-system. A scientific workflow system [1] (as GWL) is just a specialized implementation to deal with a graph of dependencies. Software folks speak about the venerable Make as build automation workflow, while bioinfo folks speak about a specific Python implementation SnakeMake as data analysis workflow. Just the same concepts but viewed by different communities. :-) If I might, an interesting analysis of different strategies for dealing with the graph of dependencies is done in the paper «Build systems à la carte» [2]. It presents the various abstractions using Haskell notations. 1: <https://en.wikipedia.org/wiki/Scientific_workflow_system> 2: <https://doi.org/10.1017/S0956796820000088> Cheers, simon
This makes handling of package outputs nondestructive. It also fixes an unrelated issue in the ordering of process packages – the implicit bash-minimal is now ordered *last*. * gwl/packages.scm (lookup-package): Return multiple values. (package-output): Deleted variable. * gwl/workflows/utils.scm (activate-workflow-environment!): Adjust accordingly. * gwl/processes.scm (<process>)[packages]: Use manifest as init-form. Use manifest? as validator. Return a manifest in transformer. * gwl/processes.scm (process->script): Adjust accordingly. * gwl/workflows/graph.scm (workflow-dot-prettify-node): Likewise. --- gwl/packages.scm | 6 +--- gwl/processes.scm | 64 ++++++++++++++++++++++------------------- gwl/workflows/graph.scm | 3 +- gwl/workflows/utils.scm | 2 +- 4 files changed, 39 insertions(+), 36 deletions(-) diff --git a/gwl/packages.scm b/gwl/packages.scm index 6a598ba..37107f6 100644 --- a/gwl/packages.scm +++ b/gwl/packages.scm @@ -46,7 +46,6 @@ lookup-package valid-package? package-name - package-output bash-minimal build-time-guix @@ -86,8 +85,7 @@ (_ (raise (condition (&gwl-package-error (package-spec specification)))))))) - (set! (package-output package) output) - package)) + (values package output))) (define (valid-package? val) (or (package? val) @@ -110,8 +108,6 @@ the version. By default, DELIMITER is \"@\"." ((? inferior-package? pkg) (inferior-package-full-name pkg)))) -(define package-output (make-object-property)) - (define bash-minimal (mlambda () (lookup-package "bash-minimal"))) diff --git a/gwl/processes.scm b/gwl/processes.scm index 2452d1f..07b376a 100644 --- a/gwl/processes.scm +++ b/gwl/processes.scm @@ -24,8 +24,10 @@ #:use-module ((guix profiles) #:select (profile + manifest manifest? manifest-search-paths - packages->manifest)) + packages->manifest + concatenate-manifests)) #:use-module ((guix search-paths) #:select (search-path-specification->sexp)) @@ -38,6 +40,7 @@ #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) + #:use-module (srfi srfi-71) #:use-module (ice-9 threads) #:use-module (ice-9 rdelim) #:export (make-process @@ -184,27 +187,33 @@ (packages #:accessor process-packages #:init-keyword #:packages - #:init-value '() + #:init-form (manifest '()) #:implicit-list? #t - #:validator (lambda (value) - (every valid-package? value)) + #:validator manifest? #:transformer ;; TODO: the instance name is not be available at this point, so we ;; can't report the process name here. We should move the ;; transformers and validators to a point after initialization. - (lambda (instance value) - (map (match-lambda - ((and (? string?) spec) - (lookup-package spec)) - ((and (? valid-package?) pkg) - pkg) - (x - (raise - (condition - (&gwl-type-error - (expected-type (list "<package>" "<inferior-package>" "<string>")) - (actual-value x)))))) - value))) + (match-lambda* + ((_ (? manifest? value)) value) + ((_ packages) + (packages->manifest + (map + (match-lambda + ((? string? spec) + (let ((pkg out (lookup-package spec))) + (list pkg out))) + ((? valid-package? pkg) + pkg) + (((? valid-package? pkg) (? string? out)) + (list pkg out)) + (x + (raise + (condition + (&gwl-type-error + (expected-type (list "<package>" "<inferior-package>" "<string>")) + (actual-value x)))))) + packages))))) (inputs #:accessor process-raw-inputs #:init-keyword #:inputs @@ -686,12 +695,11 @@ tags if WITH-TAGS? is #FALSE or missing." "Return a lowerable object for the script that will execute the PROCESS." (let* ((name (process-full-name process)) - (packages (cons (bash-minimal) - (process-packages process))) - (manifest (packages->manifest (map - (lambda (pkg) - (list pkg (package-output pkg))) - packages))) + (manifest (concatenate-manifests + ;; Put process packages before bash-minimal, so that + ;; they're not shadowed. + (list (process-packages process) + (packages->manifest (list (bash-minimal)))))) (profile (profile (content manifest))) (search-paths (delete-duplicates (map search-path-specification->sexp @@ -700,12 +708,10 @@ PROCESS." (exp (with-imported-modules (source-module-closure (script-modules)) #~(begin - #$@(if (null? packages) '() - `((use-modules (guix search-paths)) - (set-search-paths (map sexp->search-path-specification - ',search-paths) - (cons ,profile - ',packages)))) + (use-modules (guix search-paths)) + (set-search-paths (map sexp->search-path-specification + '#$search-paths) + (list #$profile)) #$(if out `(setenv "out" ,out) "") (setenv "_GWL_PROFILE" #$profile) (use-modules (ice-9 match)) diff --git a/gwl/workflows/graph.scm b/gwl/workflows/graph.scm index ea3fec9..7ea2fca 100644 --- a/gwl/workflows/graph.scm +++ b/gwl/workflows/graph.scm @@ -17,6 +17,7 @@ (define-module (gwl workflows graph) #:use-module (ice-9 format) #:use-module (ice-9 match) + #:use-module (guix profiles) #:use-module (gwl packages) #:use-module (gwl processes) #:use-module (gwl workflows) @@ -46,7 +47,7 @@ label=<<FONT POINT-SIZE=\"14\">~a</FONT><BR/>\ (match (process-packages process) (() "") (inputs (format #f "<BR/>Uses: ~{~a~^, ~}." - (map package-name inputs))))))) + (map manifest-entry-name inputs))))))) (define (workflow-restriction->dot process . restrictions) "Write the dependency relationships of a restriction in dot format." diff --git a/gwl/workflows/utils.scm b/gwl/workflows/utils.scm index 666d5f0..22e6ced 100644 --- a/gwl/workflows/utils.scm +++ b/gwl/workflows/utils.scm @@ -180,7 +180,7 @@ modify the load path of the current process." ((package-names (required-packages file-name)) (_assert (not (null? package-names))) (manifest (packages->manifest - (map lookup-package package-names))) + (map (compose list lookup-package) package-names))) (profile (profile (content manifest))) (profile-directory (parameterize ((%guile-for-build (default-guile-derivation))) -- 2.37.1
Am Sonntag, dem 31.07.2022 um 17:17 -0400 schrieb Olivier Dion: > This patch should now have all outputs in _GWL_PROFILE. There's > still a problem with the derivation though. ... at the cost of propagating all the additional outputs to processes that don't need them. Am Samstag, dem 30.07.2022 um 14:47 -0400 schrieb Olivier Dion: > I invite you to read this thread > <https://lists.gnu.org/archive/html/gwl-devel/2022-04/msg00009.html> > and the one in the following month for all the details. > Changing all the matching patterns used by GWL to handle new cases > was deemed too cumbersome. By whom? In any case, Guix itself already has a few wrapper types that bundle packages and outputs. At the lowest level (as far as I know), there's <gexp-input>. Assuming you ignore native?, you can convert a package and output easily to a <gexp-input> using (gexp-input package output), and convert it back using the gexp-input-thing and gexp-input-output accessors. (Of course, you'd have to restrict your inputs so that only gexp-inputs whose thing is a package are valid). On a somewhat higher level there's manifest entries, which you could create via package->manifest-entry. Given that GWL is built on top of manifests it might make sense to allow raw manifest entries and to transform instances of package+output to manifest entries. > [T]he short answer is that outputs are not needed by GWL except at a > single place (gwl/processes.scm:702). First, I don't think that's true. Second, if it were, you shouldn't have that many matching patterns, should you? Looking at the code around this line, if you were to go with the second option, you could SRFI-1 partition process-packages into actual packages and manifest entries, construct one manifest per group and then merge them. Alternatively, process packages could already be lowered to manifests at process construction time, which would also work for `guix workflow graph'. Cheers
This patch should now have all outputs in _GWL_PROFILE. There's still a problem with the derivation though. --8<---------------cut here---------------start------------->8--- diff --git a/gwl/packages.scm b/gwl/packages.scm index 6a598ba..2ad3929 100644 --- a/gwl/packages.scm +++ b/gwl/packages.scm @@ -43,10 +43,11 @@ #:export (current-guix inferior-store + add-package-output! lookup-package valid-package? package-name - package-output + package-outputs bash-minimal build-time-guix @@ -86,7 +87,7 @@ (_ (raise (condition (&gwl-package-error (package-spec specification)))))))) - (set! (package-output package) output) + (add-package-output! package output) package)) (define (valid-package? val) @@ -110,7 +111,14 @@ the version. By default, DELIMITER is \"@\"." ((? inferior-package? pkg) (inferior-package-full-name pkg)))) -(define package-output (make-object-property)) +(define package-outputs (make-object-property)) + +(define (add-package-output! pkg out) + (let ((outs (package-outputs pkg))) + (set! (package-outputs pkg) + (if outs + (cons out outs) + (list out))))) (define bash-minimal (mlambda () diff --git a/gwl/processes.scm b/gwl/processes.scm index 2452d1f..3616ad2 100644 --- a/gwl/processes.scm +++ b/gwl/processes.scm @@ -45,6 +45,7 @@ process-name process-full-name process-version + process-raw-packages process-packages process-raw-inputs process-inputs @@ -182,7 +183,7 @@ #:init-keyword #:description #:init-value "") (packages - #:accessor process-packages + #:accessor process-raw-packages #:init-keyword #:packages #:init-value '() #:implicit-list? #t @@ -197,12 +198,21 @@ ((and (? string?) spec) (lookup-package spec)) ((and (? valid-package?) pkg) + (add-package-output! pkg "out") + pkg) + (((? valid-package? pkg) (? string? output)) + (add-package-output! pkg output) pkg) (x (raise (condition (&gwl-type-error - (expected-type (list "<package>" "<inferior-package>" "<string>")) + (expected-type + (list "<package>" + "<inferior-package>" + "<string>" + "(<package> <string>)" + "(inferior-package> <string>)")) (actual-value x)))))) value))) (inputs @@ -622,6 +632,9 @@ the container." ;;; ADDITIONAL FUNCTIONS ;;; --------------------------------------------------------------------------- +(define (process-packages process) + (delete-duplicates (process-raw-packages process))) + (define* (process-inputs process #:optional with-tags?) "Return the plain values of all inputs of PROCESS, without any keyword tags if WITH-TAGS? is #FALSE or missing." @@ -688,10 +701,13 @@ PROCESS." (let* ((name (process-full-name process)) (packages (cons (bash-minimal) (process-packages process))) - (manifest (packages->manifest (map - (lambda (pkg) - (list pkg (package-output pkg))) - packages))) + (manifest (packages->manifest + (append-map + (lambda (pkg) + (map (lambda (output) + (list pkg output)) + (package-outputs pkg))) + packages))) (profile (profile (content manifest))) (search-paths (delete-duplicates (map search-path-specification->sexp --8<---------------cut here---------------end--------------->8--- -- Olivier Dion oldiob.dev
On Sun, 31 Jul 2022, Olivier Dion via <gwl-devel@gnu.org> wrote:
> On Sun, 31 Jul 2022, Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> wrote:
>> Am Samstag, dem 30.07.2022 um 14:47 -0400 schrieb Olivier Dion:
>
>> That example looks broken to me:
>> ,[("/gnu/store/0db6w2wd7sj8b62w9fx18krxc4pnmwql-gcc-
>> 12.1.0.drv",["out"])
>
Never mind. It seems that the property gets overwrite each time. So
you end up with only one output. The solution would be to clone the
record or perhaps accumulate the outputs and then prune duplicated
packages.
--
Olivier Dion
oldiob.dev
On Sun, 31 Jul 2022, Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> wrote: > Am Samstag, dem 30.07.2022 um 14:47 -0400 schrieb Olivier Dion: > That example looks broken to me: > ,[("/gnu/store/0db6w2wd7sj8b62w9fx18krxc4pnmwql-gcc- > 12.1.0.drv",["out"]) Seems like debug and lib outputs are missing. However, they do appear in the Guix profile (_GWL_PROFILE in the process). It looks like some path in GWL is missing the usage of the package's outputs for building the derivation. -- Olivier Dion oldiob.dev
Am Samstag, dem 30.07.2022 um 14:47 -0400 schrieb Olivier Dion:
> I don't think so. I have a workflow that requires `out' and `debug'
> outputs to instrument packages. Works flawlessly.
>
> Example:
> --8<---------------cut here---------------start------------->8---
> use-modules : gnu packages base
> use-modules : gnu packages gcc
>
> process greet
> packages
> list
> list hello "out"
> list gcc "out"
> list gcc "lib"
> list gcc "debug"
> # { hello }
>
> workflow simple-wisp
> processes
> list greet
> --8<---------------cut here---------------end--------------->8---
That example looks broken to me:
--8<---------------cut here---------------start------------->8---
Derive
([("out","/gnu/store/hxwdg9pfka1f6z44mvj8g5z1r90pphc6-gwl-
greet.scm","","")]
,[("/gnu/store/0db6w2wd7sj8b62w9fx18krxc4pnmwql-gcc-
12.1.0.drv",["out"])
,("/gnu/store/1mzvmkh3a81xm4xz6g2l0b0vgzpl52c5-hello-
2.12.1.drv",["out"])
,("/gnu/store/bw8sn6a328ngr6c5a42wsvqnxfb3qvpr-module-import-
compiled.drv",["out"])
,("/gnu/store/mlqw9mzmb4p1mrc2z0pvfj0n409lvazf-profile.drv",["out"])
,("/gnu/store/na0z5xp80avbixaaslcrizpdnwjvn7r6-guile-
3.0.8.drv",["out"])
,("/gnu/store/xlldmyxlrfzr5b8nnnf6k5yp2vranawz-bash-minimal-
5.1.8.drv",["out"])]
,["/gnu/store/cq44z87hcbm2943nglazz8wh923ldpgg-module-
import","/gnu/store/l7cyfcxh1ivbpb7qnz3fhnk6mbbiwp26-gwl-greet.scm-
builder"]
,"x86_64-linux","/gnu/store/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-
3.0.8/bin/guile",["--no-auto-compile","-
L","/gnu/store/cq44z87hcbm2943nglazz8wh923ldpgg-module-import","-
C","/gnu/store/nhpi5gqypm7clln2665qlv8ki2xpv0ik-module-import-
compiled","/gnu/store/l7cyfcxh1ivbpb7qnz3fhnk6mbbiwp26-gwl-greet.scm-
builder"]
,[("allowSubstitutes","0")
,("out","/gnu/store/hxwdg9pfka1f6z44mvj8g5z1r90pphc6-gwl-greet.scm")
,("preferLocalBuild","1")])
--8<---------------cut here---------------end--------------->8---
Notice anything missing? I do.
On Sat, 30 Jul 2022, Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> wrote: > but it raises more questions. Like "why is package-ouput a object- > property?" I invite you to read this thread <https://lists.gnu.org/archive/html/gwl-devel/2022-04/msg00009.html> and the one in the following month for all the details. But the short answer is that outputs are not needed by GWL except at a single place (gwl/processes.scm:702). Changing all the matching patterns used by GWL to handle new cases was deemed too cumbersome. I found that object properties -- or data layering -- was an elegant and none invasive solution for adding outputs support to GWL. I just forgot to add this particular case that you had :-). Maybe Ricardo has more to say on that. > and "wouldn't this break if someone needed two different > outputs of the same package in a process or even anywhere in the > workflow?" I don't think so. I have a workflow that requires `out' and `debug' outputs to instrument packages. Works flawlessly. Example: --8<---------------cut here---------------start------------->8--- use-modules : gnu packages base use-modules : gnu packages gcc process greet packages list list hello "out" list gcc "out" list gcc "lib" list gcc "debug" # { hello } workflow simple-wisp processes list greet --8<---------------cut here---------------end--------------->8--- -- Olivier Dion oldiob.dev
Am Samstag, dem 30.07.2022 um 19:03 +0200 schrieb Liliana Marie
Prikler:
> Am Freitag, dem 29.07.2022 um 11:40 -0400 schrieb Olivier Dion:
> > Please try the following patch:
> > --8<---------------cut here---------------start------------->8---
> > diff --git a/gwl/processes.scm b/gwl/processes.scm
> > index 2452d1f..4207f51 100644
> > --- a/gwl/processes.scm
> > +++ b/gwl/processes.scm
> > @@ -197,12 +197,21 @@
> > ((and (? string?) spec)
> > (lookup-package spec))
> > ((and (? valid-package?) pkg)
> > + (set! (package-output pkg) "out")
> > + pkg)
> > + (((? valid-package? pkg) (? string? output))
> > + (set! (package-output pkg) output)
> Looking at the code for handling plain packages
s/plain packages/package specs/ of course
Am Freitag, dem 29.07.2022 um 11:40 -0400 schrieb Olivier Dion:
> Please try the following patch:
> --8<---------------cut here---------------start------------->8---
> diff --git a/gwl/processes.scm b/gwl/processes.scm
> index 2452d1f..4207f51 100644
> --- a/gwl/processes.scm
> +++ b/gwl/processes.scm
> @@ -197,12 +197,21 @@
> ((and (? string?) spec)
> (lookup-package spec))
> ((and (? valid-package?) pkg)
> + (set! (package-output pkg) "out")
> + pkg)
> + (((? valid-package? pkg) (? string? output))
> + (set! (package-output pkg) output)
Looking at the code for handling plain packages, that'd probably work,
but it raises more questions. Like "why is package-ouput a object-
property?" and "wouldn't this break if someone needed two different
outputs of the same package in a process or even anywhere in the
workflow?"
Please try the following patch: --8<---------------cut here---------------start------------->8--- diff --git a/gwl/processes.scm b/gwl/processes.scm index 2452d1f..4207f51 100644 --- a/gwl/processes.scm +++ b/gwl/processes.scm @@ -197,12 +197,21 @@ ((and (? string?) spec) (lookup-package spec)) ((and (? valid-package?) pkg) + (set! (package-output pkg) "out") + pkg) + (((? valid-package? pkg) (? string? output)) + (set! (package-output pkg) output) pkg) (x (raise (condition (&gwl-type-error - (expected-type (list "<package>" "<inferior-package>" "<string>")) + (expected-type + (list "<package>" + "<inferior-package>" + "<string>" + "(<package> <string>)" + "(inferior-package> <string>)")) (actual-value x)))))) value))) (inputs --8<---------------cut here---------------end--------------->8--- -- Olivier Dion oldiob.dev
Hi Guix, using this rather simple workflow --8<---------------cut here---------------start------------->8--- use-modules : gnu packages base process greet packages hello # { hello } workflow simple-wisp processes list greet --8<---------------cut here---------------end--------------->8--- I get the following error: --8<---------------cut here---------------start------------->8--- Backtrace: In guix/gexp.scm: 893:4 19 (_ _) In guix/store.scm: 2053:12 18 (_ #<store-connection 256.99 7f0d3e5eef00>) 1380:11 17 (map/accumulate-builds #<store-connection 256.99 7f0d3…> …) 1298:8 16 (call-with-build-handler #<procedure 7f0d3d783990 at g…> …) 2168:25 15 (run-with-store #<store-connection 256.99 7f0d3e5eef00> …) In guix/gexp.scm: 898:13 14 (_ _) In guix/store.scm: 1996:8 13 (_ _) In guix/gexp.scm: 300:22 12 (_ _) In guix/profiles.scm: 1935:2 11 (_ _) In guix/store.scm: 2053:12 10 (_ #<store-connection 256.99 7f0d3e5eef00>) 1380:11 9 (map/accumulate-builds #<store-connection 256.99 7f0d3…> …) 1298:8 8 (call-with-build-handler #<procedure 7f0d3d7fb900 at g…> …) 2168:25 7 (run-with-store #<store-connection 256.99 7f0d3e5eef00> …) In guix/gexp.scm: 1181:2 6 (_ _) 1047:2 5 (_ _) 1388:2 4 (_ _) In guix/monads.scm: 471:9 3 (_ _) 471:9 2 (_ _) In guix/gexp.scm: 332:25 1 (_ _) In ice-9/boot-9.scm: 1685:16 0 (raise-exception _ #:continuable? _) ice-9/boot-9.scm:1685:16: In procedure raise-exception: ERROR: 1. &derivation-missing-output-error: derivation: #<derivation /gnu/store/1mzvmkh3a81xm4xz6g2l0b0vgzpl52c5-hello-2.12.1.drv => /gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1 7f0d3ba40af0> output: #f --8<---------------cut here---------------end--------------->8--- I'd hazard a guess this has to do with the recently added support for outputs. Note that attempting to specify an output --8<---------------cut here---------------start------------->8--- process greet packages list : list hello "out" # { hello } --8<---------------cut here---------------end--------------->8--- results in a different error --8<---------------cut here---------------start------------->8--- ice-9/boot-9.scm:1752:10: error: type error: expected one of `<package>' `<inferior-package>' `<string>', but got `<pair>' in `(#<package hello@2.12.1 gnu/packages/base.scm:86 7fd084ba76e0> out)' --8<---------------cut here---------------end--------------->8--- Thus, there is no way to use plain packages with GWL 0.5. Cheers
[-- Attachment #1: Type: text/plain, Size: 2745 bytes --] We are pleased to announce the release of the GNU Guix Workflow Language version 0.5.0, representing 46 commits by two people, incorporating the results of productive discussions among a number of helpful people on the #guix and #guix-hpc IRC channels on libera.chat and on the gwl-devel@gnu.org mailing list. This is a maintenance release with a number of bug fixes. For details see the NEWS excerpt below. • About The Guix Workflow Language (GWL) provides an extension to GNU Guix's declarative language for package management to automate the execution of programs in scientific workflows. The GWL can use process engines to integrate with various computing environments. See the manual for details: https://workflows.guix.info/manual/ • Download Here are the compressed sources and a GPG detached signature[*]: https://ftpmirror.gnu.org/gwl/gwl-0.5.0.tar.gz https://ftpmirror.gnu.org/gwl/gwl-0.5.0.tar.gz.sig Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html [*] Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this: gpg --verify gwl-0.5.0.tar.gz.sig If that command fails because you don't have the required public key, then run this command to import it: gpg --keyserver keys.gnupg.net --recv-keys BCA689B636553801C3C62150197A5888235FACAC and rerun the 'gpg --verify' command. This release was bootstrapped with the following tools: Autoconf 2.69 Automake 1.16.3 Gnulib v0.1-3269-g03d7a6b1f NEWS * Changes in 0.5.0 (since 0.4.0) ** Package handling - Package specifications may also contain outputs. ** Testing - Integration tests verify that all examples from the documentation can be executed correctly. - Integration tests include submission of jobs to a local Slurm cluster via DRMAA. ** Bug fixes - Processes comparisons are now performed with an identity comparison. This was always intended in cache table lookups, but the wrong comparison procedure was used. - Custom interpreters in code snippets would be miscompiled due to a regression. =compile-procedure= was modified to address this regression. - With containerized execution, the contained script was not passed its arguments for inputs, outputs, and values. Now the arguments are passed through. - The container scripts were compiled with fixed values for process inputs and outputs; now they read these values from their arguments as in other execution modes. - Set the job’s working directory when launching a DRMAA job. -- Ricardo [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 511 bytes --]
Am Dienstag, dem 12.07.2022 um 21:05 +0200 schrieb Ricardo Wurmus: > > Hi Liliana, > > > I have a scientific workflow that depends on a bash script using an > > extension. (I wish I did not.) Said script is the least common > > denominator working on all systems that have some basic utilities, > > and needs therefore not consider the reproducibility of its inputs > > (it is assumed, that those are up to date). However, at least on > > my personal machine I want to ensure the reproducibility of the > > entire workflow, including the invocation of said script. > > > > I wrote a GWL workflow that calls the script, but due to the > > extension the script actually fails in the GWL context. This is > > because GWL's process->script unconditionally conses bash-minimal > > to the process inputs. It should probably check whether a bash > > input already exists or alternatively just append (list (bash- > > minimal)) instead. > > Would appending bash-minimal really help? Does it really give > preference to your bash when resolving conflicts as it builds the > profile? Why should it not? > Not sure how we should check for the presence of a package providing > a shell. We can check for certain packages by name, of course, but > it doesn’t feel right to me. I'm fairly certain this lookup uses PATH or an equivalent implementation in Guile (search-input-file springs to mind, which also uses inputs in the order given). > Any other ideas how to accommodate the need to swap out the package > providing a shell? You could check for the presence of bash in inputs before adding it, but that'd be more error-prone. Alternatively the implicit vs. explicit problem also exists in gnu-build-system, so that solution could be adapted as well. Cheers
Hi Liliana, > I have a scientific workflow that depends on a bash script using an > extension. (I wish I did not.) Said script is the least common > denominator working on all systems that have some basic utilities, and > needs therefore not consider the reproducibility of its inputs (it is > assumed, that those are up to date). However, at least on my personal > machine I want to ensure the reproducibility of the entire workflow, > including the invocation of said script. > > I wrote a GWL workflow that calls the script, but due to the extension > the script actually fails in the GWL context. This is because GWL's > process->script unconditionally conses bash-minimal to the process > inputs. It should probably check whether a bash input already exists > or alternatively just append (list (bash-minimal)) instead. Would appending bash-minimal really help? Does it really give preference to your bash when resolving conflicts as it builds the profile? Not sure how we should check for the presence of a package providing a shell. We can check for certain packages by name, of course, but it doesn’t feel right to me. Any other ideas how to accommodate the need to swap out the package providing a shell? > For those in a similar situation as I am needing a fix, the following > works for me: instead of calling "bash" inside the workflow's inline > code, I specify > > values > run-with-store > open-connection > package-file bash "bin/bash" > > and use {{values}} instead of "bash" to call my script. (Don't forget > to add bash to your inputs, because package-file does not guarantee > existence of the file). That’s an interesting solution. Thanks for sharing it. -- Ricardo
>> I am currently working on the following issue.
>> https://issues.guix.gnu.org/55612
>
> This commit makes the GWL ready for guile-config 0.5.x:
>
> https://git.savannah.gnu.org/cgit/gwl.git/commit/?id=c0f281d2ef927686ed42f9c44828c501011a60dc
>
> We don’t have a new release yet, but I don’t want your upgrade of
> guile-hall to be blocked any longer.
>
> Thanks for notifying me!
Thank you for your commit!
So far I am not having trouble, but if I apply this patch to v0.4.0,
would it be possible to increase the guile-hall version of Guix?
If other changes are needed, I am willing to wait for a new release of
GWL.
Regards,
--
Taiju
Taiju HIGASHI <higashi@taiju.info> writes: > I am currently working on the following issue. > https://issues.guix.gnu.org/55612 This commit makes the GWL ready for guile-config 0.5.x: https://git.savannah.gnu.org/cgit/gwl.git/commit/?id=c0f281d2ef927686ed42f9c44828c501011a60dc We don’t have a new release yet, but I don’t want your upgrade of guile-hall to be blocked any longer. Thanks for notifying me! -- Ricardo
On Thu, 16 Jun 2022, Ricardo Wurmus <rekado@elephly.net> wrote:
> Olivier Dion <olivier.dion@polymtl.ca> writes:
>
>> Processes in the same list can be executed in parallel with `par-map'. The
>> results are then appended to the accumulator.
>
> Thanks for the patch. I applied it.
I just realize that we can remove the `(reverse item)' in the patch.
Although we can't predict the order of `par-map', the reversal of the
list is unnecessary with the removal of `fold'.
--
Olivier Dion
oldiob.dev
Olivier Dion <olivier.dion@polymtl.ca> writes:
> When executing processes in parallel, outputs from threads can be mangled
> together if the access is not exclusive.
Thanks, applied!
--
Ricardo