running a shell script in a snippet

running a shell script in a snippet

[Ricardo Wurmus]

[-- Attachment #1: Type: text/plain, Size: 3929 bytes --]

Hi Guix,

Shogun comes with a script to remove all non-free source files and
sections.  Previously we removed the non-free code in a snippet with
some Guile code.  It would be better to use the upstream script, because
it’s hard to keep the Guile code in sync with upstream changes.

So I thought I could just run the script in the snippet, but there is no
shell at the time when the snippet is executed.

Patching and snippet application happens in an environment where the
result of “%standard-patch-inputs” (a procedure in “(guix packages)”)
is available.  If a package defines “patch-inputs” those inputs would be
available instead.

I tried to specify “patch-inputs” for shogun, but I cannot get this to
work.  This is the naive way:

--8<---------------cut here---------------start------------->8---
       (patch-inputs
        `(("bash" ,bash)
          ("tar" ,tar)
          ("bzip2" ,bzip2)
          ("patch" ,patch)))
       (snippet
        `(zero? (system* "bash" "scripts/light-scrubber.sh")))
--8<---------------cut here---------------end--------------->8---

It fails when trying to run “guix build -S shogun”:

--8<---------------cut here---------------start------------->8---
guix build: warning: failed to load '(gnu packages abiword)':
ERROR: Unbound variable: bash
guix build: warning: failed to load '(gnu packages avr)':
ERROR: In procedure module-lookup: Unbound variable: gcc
guix build: warning: failed to load '(gnu packages bioinformatics)':
ERROR: In procedure module-lookup: Unbound variable: perl-libwww
guix build: warning: failed to load '(gnu packages commencement)':
ERROR: In procedure module-lookup: Unbound variable: gnu-make
guix build: warning: failed to load '(gnu packages debug)':
ERROR: In procedure module-lookup: Unbound variable: gnu-make
guix build: warning: failed to load '(gnu packages embedded)':
ERROR: In procedure #<procedure 2e83160 ()>: Unbound variable: cross-gcc
guix build: warning: failed to load '(gnu packages make-bootstrap)':
ERROR: In procedure module-lookup: Unbound variable: coreutils
guix build: warning: failed to load '(gnu packages mate)':
ERROR: In procedure module-lookup: Unbound variable: gtk+
guix build: warning: failed to load '(gnu packages u-boot)':
ERROR: no binding `bc' in module (gnu packages algebra)
guix build: warning: failed to load '(gnu packages unrtf)':
ERROR: In procedure module-lookup: Unbound variable: coreutils
guix build: error: shogun: unknown package
--8<---------------cut here---------------end--------------->8---

This looks like a dependency cycle, so lets try using “module-ref”.

--8<---------------cut here---------------start------------->8---
       (patch-inputs
        `(("bash" ,(module-ref (resolve-interface '(gnu packages bash)) 'bash))
          ("tar" ...)
          ("bzip2" ...)
          ("patch" ...)))
--8<---------------cut here---------------end--------------->8---

No luck:

--8<---------------cut here---------------start------------->8---
guix build: warning: failed to load '(gnu packages abiword)':
ERROR: No variable named bash in #<interface (gnu packages bash) 3e756c0>
...
--8<---------------cut here---------------end--------------->8---

Maybe the patch-inputs field must be thunked like “inputs”,
“native-inputs”, and “propagated-inputs” in the package record?

I changed the field to be thunked and changed “origin->derivation” to
unthunk the inputs before passing them to “patch-and-repack”, but here’s
the next problem: “patch-and-repack” doesn’t set the PATH to include all
patch-inputs.  This means that overriding “patch-inputs” gets me
nowhere.

So here’s a patch that triggers a rebuild of everything, which – I guess
— does the right thing.  Since it causes a rebuild of the world I
couldn’t actually confirm that it helps.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-WIP-thunk-patch-inputs.patch --]
[-- Type: text/x-patch, Size: 2253 bytes --]

From b2647c307165a995aa68dc9b83c533c6c05fabf2 Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
Date: Thu, 19 Jan 2017 14:11:05 +0100
Subject: [PATCH] WIP: thunk patch-inputs.

* guix/packages.scm (<origin>): Mark "patch-inputs" as thunked.
(patch-and-repack): Add "bin" directories of all inputs to the PATH.
(origin->derivation): Unthunk patch inputs when necessary.
---
 guix/packages.scm | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/guix/packages.scm b/guix/packages.scm
index beb958f15..860b02849 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -166,7 +166,7 @@
   ;; Patching requires Guile, GNU Patch, and a few more.  These two fields are
   ;; used to specify these dependencies when needed.
   (patch-inputs origin-patch-inputs               ; input list or #f
-                (default #f))
+                (default #f) (thunked))
   (modules      origin-modules                    ; list of module names
                 (default '()))
 
@@ -518,8 +518,11 @@ specifies modules in scope when evaluating SNIPPET."
                                             (package-version locales))))
               (setlocale LC_ALL "en_US.utf8"))
 
-            (setenv "PATH" (string-append #+xz "/bin" ":"
-                                          #+decomp "/bin"))
+            (let ((PATH (string-join (map (lambda (input)
+                                            (string-append input "/bin"))
+                                          #+inputs)
+                                     ":")))
+              (setenv "PATH" PATH))
 
             ;; SOURCE may be either a directory or a tarball.
             (and (if (file-is-directory? #+source)
@@ -1208,7 +1211,8 @@ cross-compilation target triplet."
                                                       system
                                                       #:graft? #f)))
        (patch-and-repack source patches
-                         #:inputs inputs
+                         #:inputs (if (procedure? inputs)
+                                      (inputs) inputs)
                          #:snippet snippet
                          #:flags flags
                          #:system system
-- 
2.11.0


[-- Attachment #3: Type: text/plain, Size: 89 bytes --]


Does this seem like a sane way of achieving my goal or am I way off
target?

~~ Ricardo

Re: running a shell script in a snippet

[Ludovic Courtès]

Hello!

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> skribis:

> Shogun comes with a script to remove all non-free source files and
> sections.  Previously we removed the non-free code in a snippet with
> some Guile code.  It would be better to use the upstream script, because
> it’s hard to keep the Guile code in sync with upstream changes.
>
> So I thought I could just run the script in the snippet, but there is no
> shell at the time when the snippet is executed.

That makes sense, assuming the script really does what we want.

> Patching and snippet application happens in an environment where the
> result of “%standard-patch-inputs” (a procedure in “(guix packages)”)
> is available.  If a package defines “patch-inputs” those inputs would be
> available instead.
>
> I tried to specify “patch-inputs” for shogun, but I cannot get this to
> work.  This is the naive way:
>
>        (patch-inputs
>         `(("bash" ,bash)
>           ("tar" ,tar)
>           ("bzip2" ,bzip2)
>           ("patch" ,patch)))
>        (snippet
>         `(zero? (system* "bash" "scripts/light-scrubber.sh")))

In fact, I think you can leave ‘patch-inputs’ as is and write:

  (snippet
    #~(zero? (system* #+(file-append bash "/bin/sh") …)))

A little-known feature.  ;-)

(That doesn’t solve the circular dependency problem though.)

> I changed the field to be thunked and changed “origin->derivation” to
> unthunk the inputs before passing them to “patch-and-repack”, but here’s
> the next problem: “patch-and-repack” doesn’t set the PATH to include all
> patch-inputs.  This means that overriding “patch-inputs” gets me
> nowhere.
>
> So here’s a patch that triggers a rebuild of everything, which – I guess
> — does the right thing.  Since it causes a rebuild of the world I
> couldn’t actually confirm that it helps.
>
> From b2647c307165a995aa68dc9b83c533c6c05fabf2 Mon Sep 17 00:00:00 2001
> From: Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>
> Date: Thu, 19 Jan 2017 14:11:05 +0100
> Subject: [PATCH] WIP: thunk patch-inputs.
>
> * guix/packages.scm (<origin>): Mark "patch-inputs" as thunked.
> (patch-and-repack): Add "bin" directories of all inputs to the PATH.
> (origin->derivation): Unthunk patch inputs when necessary.

Could you separate the PATH thing and commit it to ‘core-updates’?

The rest of the patch LGTM, and is OK for master.

Another approach would be to make ‘snippet’ thunked, so you could use
the gexp shown above.

Yet another one (probably better) would be to introduce a <gexp-delay>
type with its own “gexp compiler”, so that one could write

  #~(system* #+(file-append (gexp-delay bash) …) …)

and the promise would be forced only when the gexp is compiled.

Ludo’.