GWL 0.5 fails with plain packages

GWL 0.5 fails with plain packages

[Liliana Marie Prikler]

Hi Guix,

using this rather simple workflow

--8<---------------cut here---------------start------------->8---
use-modules : gnu packages base

process greet
  packages hello
  # { hello }

workflow simple-wisp
  processes
    list greet
--8<---------------cut here---------------end--------------->8---

I get the following error:

--8<---------------cut here---------------start------------->8---
Backtrace:
In guix/gexp.scm:
    893:4 19 (_ _)
In guix/store.scm:
  2053:12 18 (_ #<store-connection 256.99 7f0d3e5eef00>)
  1380:11 17 (map/accumulate-builds #<store-connection 256.99 7f0d3…>
…)
   1298:8 16 (call-with-build-handler #<procedure 7f0d3d783990 at g…>
…)
  2168:25 15 (run-with-store #<store-connection 256.99 7f0d3e5eef00> …)
In guix/gexp.scm:
   898:13 14 (_ _)
In guix/store.scm:
   1996:8 13 (_ _)
In guix/gexp.scm:
   300:22 12 (_ _)
In guix/profiles.scm:
   1935:2 11 (_ _)
In guix/store.scm:
  2053:12 10 (_ #<store-connection 256.99 7f0d3e5eef00>)
  1380:11  9 (map/accumulate-builds #<store-connection 256.99 7f0d3…>
…)
   1298:8  8 (call-with-build-handler #<procedure 7f0d3d7fb900 at g…>
…)
  2168:25  7 (run-with-store #<store-connection 256.99 7f0d3e5eef00> …)
In guix/gexp.scm:
   1181:2  6 (_ _)
   1047:2  5 (_ _)
   1388:2  4 (_ _)
In guix/monads.scm:
    471:9  3 (_ _)
    471:9  2 (_ _)
In guix/gexp.scm:
   332:25  1 (_ _)
In ice-9/boot-9.scm:
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
ERROR:
  1. &derivation-missing-output-error:
      derivation: #<derivation
/gnu/store/1mzvmkh3a81xm4xz6g2l0b0vgzpl52c5-hello-2.12.1.drv =>
/gnu/store/s5pd3rnzymliafb4la5sca63j86xs0y0-hello-2.12.1 7f0d3ba40af0>
      output: #f
--8<---------------cut here---------------end--------------->8---

I'd hazard a guess this has to do with the recently added support for
outputs.  Note that attempting to specify an output

--8<---------------cut here---------------start------------->8---
process greet
  packages
    list : list hello "out"
  # { hello }
--8<---------------cut here---------------end--------------->8---

results in a different error

--8<---------------cut here---------------start------------->8---
ice-9/boot-9.scm:1752:10: error: type error: expected one of
`<package>' `<inferior-package>' `<string>', but got `<pair>' in
`(#<package hello@2.12.1 gnu/packages/base.scm:86 7fd084ba76e0> out)'
--8<---------------cut here---------------end--------------->8---

Thus, there is no way to use plain packages with GWL 0.5.

Cheers

Re: GWL 0.5 fails with plain packages

[Olivier Dion via]

Please try the following patch:
--8<---------------cut here---------------start------------->8---
diff --git a/gwl/processes.scm b/gwl/processes.scm
index 2452d1f..4207f51 100644
--- a/gwl/processes.scm
+++ b/gwl/processes.scm
@@ -197,12 +197,21 @@
             ((and (? string?) spec)
              (lookup-package spec))
             ((and (? valid-package?) pkg)
+             (set! (package-output pkg) "out")
+             pkg)
+            (((? valid-package? pkg) (? string? output))
+             (set! (package-output pkg) output)
              pkg)
             (x
              (raise
               (condition
                (&gwl-type-error
-                (expected-type (list "<package>" "<inferior-package>" "<string>"))
+                (expected-type
+                 (list "<package>"
+                       "<inferior-package>"
+                       "<string>"
+                       "(<package> <string>)"
+                       "(inferior-package> <string>)"))
                 (actual-value x))))))
           value)))
   (inputs
--8<---------------cut here---------------end--------------->8---
-- 
Olivier Dion
oldiob.dev

Re: GWL 0.5 fails with plain packages

[Liliana Marie Prikler]

Am Freitag, dem 29.07.2022 um 11:40 -0400 schrieb Olivier Dion:
> Please try the following patch:
> --8<---------------cut here---------------start------------->8---
> diff --git a/gwl/processes.scm b/gwl/processes.scm
> index 2452d1f..4207f51 100644
> --- a/gwl/processes.scm
> +++ b/gwl/processes.scm
> @@ -197,12 +197,21 @@
>              ((and (? string?) spec)
>               (lookup-package spec))
>              ((and (? valid-package?) pkg)
> +             (set! (package-output pkg) "out")
> +             pkg)
> +            (((? valid-package? pkg) (? string? output))
> +             (set! (package-output pkg) output)
Looking at the code for handling plain packages, that'd probably work,
but it raises more questions.  Like "why is package-ouput a object-
property?" and "wouldn't this break if someone needed two different
outputs of the same package in a process or even anywhere in the
workflow?"

Re: GWL 0.5 fails with plain packages

[Liliana Marie Prikler]

Am Samstag, dem 30.07.2022 um 19:03 +0200 schrieb Liliana Marie
Prikler:
> Am Freitag, dem 29.07.2022 um 11:40 -0400 schrieb Olivier Dion:
> > Please try the following patch:
> > --8<---------------cut here---------------start------------->8---
> > diff --git a/gwl/processes.scm b/gwl/processes.scm
> > index 2452d1f..4207f51 100644
> > --- a/gwl/processes.scm
> > +++ b/gwl/processes.scm
> > @@ -197,12 +197,21 @@
> >              ((and (? string?) spec)
> >               (lookup-package spec))
> >              ((and (? valid-package?) pkg)
> > +             (set! (package-output pkg) "out")
> > +             pkg)
> > +            (((? valid-package? pkg) (? string? output))
> > +             (set! (package-output pkg) output)
> Looking at the code for handling plain packages
s/plain packages/package specs/ of course

Re: GWL 0.5 fails with plain packages

[Olivier Dion via]

On Sat, 30 Jul 2022, Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> wrote:

> but it raises more questions.  Like "why is package-ouput a object-
> property?"

I invite you to read this thread
<https://lists.gnu.org/archive/html/gwl-devel/2022-04/msg00009.html> and
the one in the following month for all the details.

But the short answer is that outputs are not needed by GWL except at a
single place (gwl/processes.scm:702).  Changing all the matching
patterns used by GWL to handle new cases was deemed too cumbersome.  I
found that object properties -- or data layering -- was an elegant and
none invasive solution for adding outputs support to GWL.  I just forgot
to add this particular case that you had :-).

Maybe Ricardo has more to say on that.

> and "wouldn't this break if someone needed two different
> outputs of the same package in a process or even anywhere in the
> workflow?"

I don't think so.  I have a workflow that requires `out' and `debug'
outputs to instrument packages.  Works flawlessly.

Example:
--8<---------------cut here---------------start------------->8---
use-modules : gnu packages base
use-modules : gnu packages gcc

process greet
  packages
    list
      list hello "out"
      list gcc "out"
      list gcc "lib"
      list gcc "debug"
  # { hello }

workflow simple-wisp
  processes
    list greet
--8<---------------cut here---------------end--------------->8---

-- 
Olivier Dion
oldiob.dev

Re: GWL 0.5 fails with plain packages

[Liliana Marie Prikler]

Am Samstag, dem 30.07.2022 um 14:47 -0400 schrieb Olivier Dion:
> I don't think so.  I have a workflow that requires `out' and `debug'
> outputs to instrument packages.  Works flawlessly.
> 
> Example:
> --8<---------------cut here---------------start------------->8---
> use-modules : gnu packages base
> use-modules : gnu packages gcc
> 
> process greet
>   packages
>     list
>       list hello "out"
>       list gcc "out"
>       list gcc "lib"
>       list gcc "debug"
>   # { hello }
> 
> workflow simple-wisp
>   processes
>     list greet
> --8<---------------cut here---------------end--------------->8---
That example looks broken to me:
--8<---------------cut here---------------start------------->8---
Derive
([("out","/gnu/store/hxwdg9pfka1f6z44mvj8g5z1r90pphc6-gwl-
greet.scm","","")]
 ,[("/gnu/store/0db6w2wd7sj8b62w9fx18krxc4pnmwql-gcc-
12.1.0.drv",["out"])
   ,("/gnu/store/1mzvmkh3a81xm4xz6g2l0b0vgzpl52c5-hello-
2.12.1.drv",["out"])
   ,("/gnu/store/bw8sn6a328ngr6c5a42wsvqnxfb3qvpr-module-import-
compiled.drv",["out"])
   ,("/gnu/store/mlqw9mzmb4p1mrc2z0pvfj0n409lvazf-profile.drv",["out"])
   ,("/gnu/store/na0z5xp80avbixaaslcrizpdnwjvn7r6-guile-
3.0.8.drv",["out"])
   ,("/gnu/store/xlldmyxlrfzr5b8nnnf6k5yp2vranawz-bash-minimal-
5.1.8.drv",["out"])]
 ,["/gnu/store/cq44z87hcbm2943nglazz8wh923ldpgg-module-
import","/gnu/store/l7cyfcxh1ivbpb7qnz3fhnk6mbbiwp26-gwl-greet.scm-
builder"]
 ,"x86_64-linux","/gnu/store/1jgcbdzx2ss6xv59w55g3kr3x4935dfb-guile-
3.0.8/bin/guile",["--no-auto-compile","-
L","/gnu/store/cq44z87hcbm2943nglazz8wh923ldpgg-module-import","-
C","/gnu/store/nhpi5gqypm7clln2665qlv8ki2xpv0ik-module-import-
compiled","/gnu/store/l7cyfcxh1ivbpb7qnz3fhnk6mbbiwp26-gwl-greet.scm-
builder"]
 ,[("allowSubstitutes","0")
   ,("out","/gnu/store/hxwdg9pfka1f6z44mvj8g5z1r90pphc6-gwl-greet.scm")
   ,("preferLocalBuild","1")])
--8<---------------cut here---------------end--------------->8---
Notice anything missing?  I do.

Re: GWL 0.5 fails with plain packages

[Olivier Dion via]

On Sun, 31 Jul 2022, Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> wrote:
> Am Samstag, dem 30.07.2022 um 14:47 -0400 schrieb Olivier Dion:

> That example looks broken to me:
>  ,[("/gnu/store/0db6w2wd7sj8b62w9fx18krxc4pnmwql-gcc-
> 12.1.0.drv",["out"])

Seems like debug and lib outputs are missing.  However, they do appear
in the Guix profile (_GWL_PROFILE in the process).  It looks like some
path in GWL is missing the usage of the package's outputs for building
the derivation.

-- 
Olivier Dion
oldiob.dev

Re: GWL 0.5 fails with plain packages

[Olivier Dion via]

On Sun, 31 Jul 2022, Olivier Dion via <gwl-devel@gnu.org> wrote:
> On Sun, 31 Jul 2022, Liliana Marie Prikler <liliana.prikler@ist.tugraz.at> wrote:
>> Am Samstag, dem 30.07.2022 um 14:47 -0400 schrieb Olivier Dion:
>
>> That example looks broken to me:
>>  ,[("/gnu/store/0db6w2wd7sj8b62w9fx18krxc4pnmwql-gcc-
>> 12.1.0.drv",["out"])
>
Never mind.  It seems that the property gets overwrite each time.  So
you end up with only one output.  The solution would be to clone the
record or perhaps accumulate the outputs and then prune duplicated
packages.
-- 
Olivier Dion
oldiob.dev

Re: GWL 0.5 fails with plain packages

[Olivier Dion via]

This patch should now have all outputs in _GWL_PROFILE.  There's still a
problem with the derivation though.
--8<---------------cut here---------------start------------->8---
diff --git a/gwl/packages.scm b/gwl/packages.scm
index 6a598ba..2ad3929 100644
--- a/gwl/packages.scm
+++ b/gwl/packages.scm
@@ -43,10 +43,11 @@
   #:export (current-guix
             inferior-store
 
+            add-package-output!
             lookup-package
             valid-package?
             package-name
-            package-output
+            package-outputs
 
             bash-minimal
             build-time-guix
@@ -86,7 +87,7 @@
                     (_ (raise (condition
                                (&gwl-package-error
                                 (package-spec specification))))))))
-    (set! (package-output package) output)
+    (add-package-output! package output)
     package))
 
 (define (valid-package? val)
@@ -110,7 +111,14 @@ the version.  By default, DELIMITER is \"@\"."
     ((? inferior-package? pkg)
      (inferior-package-full-name pkg))))
 
-(define package-output (make-object-property))
+(define package-outputs (make-object-property))
+
+(define (add-package-output! pkg out)
+  (let ((outs (package-outputs pkg)))
+    (set! (package-outputs pkg)
+          (if outs
+              (cons out outs)
+              (list out)))))
 
 (define bash-minimal
   (mlambda ()
diff --git a/gwl/processes.scm b/gwl/processes.scm
index 2452d1f..3616ad2 100644
--- a/gwl/processes.scm
+++ b/gwl/processes.scm
@@ -45,6 +45,7 @@
             process-name
             process-full-name
             process-version
+            process-raw-packages
             process-packages
             process-raw-inputs
             process-inputs
@@ -182,7 +183,7 @@
    #:init-keyword #:description
    #:init-value "")
   (packages
-   #:accessor process-packages
+   #:accessor process-raw-packages
    #:init-keyword #:packages
    #:init-value '()
    #:implicit-list? #t
@@ -197,12 +198,21 @@
             ((and (? string?) spec)
              (lookup-package spec))
             ((and (? valid-package?) pkg)
+             (add-package-output! pkg "out")
+             pkg)
+            (((? valid-package? pkg) (? string? output))
+             (add-package-output! pkg output)
              pkg)
             (x
              (raise
               (condition
                (&gwl-type-error
-                (expected-type (list "<package>" "<inferior-package>" "<string>"))
+                (expected-type
+                 (list "<package>"
+                       "<inferior-package>"
+                       "<string>"
+                       "(<package> <string>)"
+                       "(inferior-package> <string>)"))
                 (actual-value x))))))
           value)))
   (inputs
@@ -622,6 +632,9 @@ the container."
 ;;; ADDITIONAL FUNCTIONS
 ;;; ---------------------------------------------------------------------------
 
+(define (process-packages process)
+  (delete-duplicates (process-raw-packages process)))
+
 (define* (process-inputs process #:optional with-tags?)
   "Return the plain values of all inputs of PROCESS, without any
 keyword tags if WITH-TAGS? is #FALSE or missing."
@@ -688,10 +701,13 @@ PROCESS."
   (let* ((name         (process-full-name process))
          (packages     (cons (bash-minimal)
                              (process-packages process)))
-         (manifest     (packages->manifest (map
-                                            (lambda (pkg)
-                                              (list pkg (package-output pkg)))
-                                            packages)))
+         (manifest     (packages->manifest
+                        (append-map
+                         (lambda (pkg)
+                           (map (lambda (output)
+                                  (list pkg output))
+                                (package-outputs pkg)))
+                         packages)))
          (profile      (profile (content manifest)))
          (search-paths (delete-duplicates
                         (map search-path-specification->sexp
--8<---------------cut here---------------end--------------->8---

-- 
Olivier Dion
oldiob.dev

Re: GWL 0.5 fails with plain packages

[Liliana Marie Prikler]

Am Sonntag, dem 31.07.2022 um 17:17 -0400 schrieb Olivier Dion:
> This patch should now have all outputs in _GWL_PROFILE.  There's
> still a problem with the derivation though.
... at the cost of propagating all the additional outputs to processes
that don't need them.

Am Samstag, dem 30.07.2022 um 14:47 -0400 schrieb Olivier Dion:
> I invite you to read this thread
> <https://lists.gnu.org/archive/html/gwl-devel/2022-04/msg00009.html>
> and the one in the following month for all the details.

> Changing all the matching patterns used by GWL to handle new cases
> was deemed too cumbersome.
By whom?

In any case, Guix itself already has a few wrapper types that bundle
packages and outputs.  At the lowest level (as far as I know), there's
<gexp-input>.  Assuming you ignore native?, you can convert a package
and output easily to a <gexp-input> using (gexp-input package output),
and convert it back using the gexp-input-thing and gexp-input-output
accessors.  (Of course, you'd have to restrict your inputs so that only
gexp-inputs whose thing is a package are valid).

On a somewhat higher level there's manifest entries, which you could
create via package->manifest-entry.  Given that GWL is built on top of
manifests it might make sense to allow raw manifest entries and to
transform instances of package+output to manifest entries.

> [T]he short answer is that outputs are not needed by GWL except at a
> single place (gwl/processes.scm:702).
First, I don't think that's true.  Second, if it were, you shouldn't
have that many matching patterns, should you?

Looking at the code around this line, if you were to go with the second
option, you could SRFI-1 partition process-packages into actual
packages and manifest entries, construct one manifest per group and
then merge them.  Alternatively, process packages could already be
lowered to manifests at process construction time, which would also
work for `guix workflow graph'.

Cheers