guile2.2-gdbm-ffi issue

guile2.2-gdbm-ffi issue

[Christopher Allan Webber]

[-- Attachment #1.1: Type: text/plain, Size: 804 bytes --]

(Originally accidentally sent this to guile-devel@gnu.org ... sorry!)

Hiya,

I pushed guile2.2-gdbm because it was just a couple of lines and it
seemed to work and didn't affect anything.  I made a mistake though that
I didn't realize until I ran
"guix environment --ad-hoc guile-next guile2.2-gdbm-ffi --pure".
The trivial-build-system in guile-gdbm-ffi dumps the built module into
(string-append out "/share/guile/site/2.0") ... oops!  Before I ran
--pure, the 2.0 directory was on my load path, and thus I didn't bump
into the error.  (Maybe I should have asked for review anyway...)

I'm trying to think of how to fix this.  I have a very kludgy solution
attached.  I don't feel great about it but I don't know how to signal to
the builder whether it's using guile 2.2 or 2.0 otherwise.  Thoughts?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-guile-gdbm-ffi-Write-to-correct-guile-output-directo.patch --]
[-- Type: text/x-patch, Size: 2606 bytes --]

From 711a23036417807d444729eaa778c9cadffa8646 Mon Sep 17 00:00:00 2001
From: Christopher Allan Webber <cwebber@dustycloud.org>
Date: Fri, 10 Feb 2017 19:24:57 -0600
Subject: [PATCH 3/3] guile-gdbm-ffi: Write to correct guile output directory.

* gnu/packages/guile.scm (make-guile-gdbm-ffi): New variable.
(guile-gdbm-ffi, guile2.2-gdbm-ffi): Use make-guile-gdbm-ffi.
---
 gnu/packages/guile.scm | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
index 3e8ab007b..3e62949f2 100644
--- a/gnu/packages/guile.scm
+++ b/gnu/packages/guile.scm
@@ -825,9 +825,11 @@ inspired by the SCSH regular expression system.")
 ;; There are two guile-gdbm packages, one using the FFI and one with
 ;; direct C bindings, hence the verbose name.
 
-(define-public guile-gdbm-ffi
+(define (make-guile-gdbm-ffi guile-2.2?)
   (package
-    (name "guile-gdbm-ffi")
+    (name (if guile-2.2?
+              "guile2.2-gdbm-ffi"
+              "guile-gdbm-ffi"))
     (version "20120209.fa1d5b6")
     (source (origin
               (method git-fetch)
@@ -848,7 +850,9 @@ inspired by the SCSH regular expression system.")
                       (system base compile))
 
          (let* ((out (assoc-ref %outputs "out"))
-                (module-dir (string-append out "/share/guile/site/2.0"))
+                (module-dir (string-append out "/share/guile/site/"
+                                           ,(if guile-2.2?
+                                                "2.2" "2.0")))
                 (source (assoc-ref %build-inputs "source"))
                 (doc (string-append out "/share/doc"))
                 (guild (string-append (assoc-ref %build-inputs "guile")
@@ -877,7 +881,8 @@ inspired by the SCSH regular expression system.")
            (compile-file gdbm.scm-dest
                          #:output-file gdbm.go-dest)))))
     (inputs
-     `(("guile" ,guile-2.0)))
+     `(("guile" ,(if guile-2.2?
+                     guile-next guile-2.0))))
     (propagated-inputs
      `(("gdbm" ,gdbm)))
     (home-page "https://github.com/ijp/guile-gdbm")
@@ -887,8 +892,11 @@ inspired by the SCSH regular expression system.")
 Guile's foreign function interface.")
     (license gpl3+)))
 
+(define-public guile-gdbm-ffi
+  (make-guile-gdbm-ffi #f))
+
 (define-public guile2.2-gdbm-ffi
-  (package-for-guile-2.2 guile-gdbm-ffi))
+  (make-guile-gdbm-ffi #t))
 
 (define-public guile-sqlite3
   (let ((commit "607721fe1174a299e45d457acacf94eefb964071"))
-- 
2.11.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Fixing non-reproducibility in some guile packages (was: guile2.2-gdbm-ffi issue)

[Christopher Allan Webber]

[-- Attachment #1.1: Type: text/plain, Size: 1542 bytes --]

Christopher Allan Webber writes:

> (Originally accidentally sent this to guile-devel@gnu.org ... sorry!)
>
> Hiya,
>
> I pushed guile2.2-gdbm because it was just a couple of lines and it
> seemed to work and didn't affect anything.  I made a mistake though that
> I didn't realize until I ran
> "guix environment --ad-hoc guile-next guile2.2-gdbm-ffi --pure".
> The trivial-build-system in guile-gdbm-ffi dumps the built module into
> (string-append out "/share/guile/site/2.0") ... oops!  Before I ran
> --pure, the 2.0 directory was on my load path, and thus I didn't bump
> into the error.  (Maybe I should have asked for review anyway...)
>
> I'm trying to think of how to fix this.  I have a very kludgy solution
> attached.  I don't feel great about it but I don't know how to signal to
> the builder whether it's using guile 2.2 or 2.0 otherwise.  Thoughts?

It turns out there was a more serious issue involved here.  The guile
input was never used... instead, we were compiling from whatever guile
process guix was using to run by using the compile-file procedure from
(system base compile).

I think this is a pretty serious bug.  It means that guile-gdbm-ffi was
never properly reproducible by our standard methods of determining
inputs.  This is entirely my fault, since I'm the one who put together
these packages.

The guile-wisp package also has this issue.  I'm going to work on a fix
for it.  In the meanwhile, here's a patch that both fixes
guile2.2-gdbm-ffi and fixes the forementioned reproducibility problem.

 - Chris


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-guile-gdbm-ffi-Write-to-correct-guile-output-directo.patch --]
[-- Type: text/x-patch, Size: 3824 bytes --]

From b718720d72f1da7655db49ec6ec25b658b0ca27f Mon Sep 17 00:00:00 2001
From: Christopher Allan Webber <cwebber@dustycloud.org>
Date: Fri, 10 Feb 2017 19:24:57 -0600
Subject: [PATCH] guile-gdbm-ffi: Write to correct guile output directory and
 use guild.

* gnu/packages/guile.scm (make-guile-gdbm-ffi): New variable.
Adapts from the previous guile-gdbm-ffi definition.  Also fixes
a bug where the guild command was not getting called, and instead
was calling the internal guile compile-file procedure.  This meant
that the package produced was dependent on whatever version of
guile was powering Guix at the time.
(guile-gdbm-ffi, guile2.2-gdbm-ffi): Use make-guile-gdbm-ffi.
---
 gnu/packages/guile.scm | 29 +++++++++++++++++++----------
 1 file changed, 19 insertions(+), 10 deletions(-)

diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
index 3e8ab007b..a02e4887a 100644
--- a/gnu/packages/guile.scm
+++ b/gnu/packages/guile.scm
@@ -825,9 +825,11 @@ inspired by the SCSH regular expression system.")
 ;; There are two guile-gdbm packages, one using the FFI and one with
 ;; direct C bindings, hence the verbose name.
 
-(define-public guile-gdbm-ffi
+(define (make-guile-gdbm-ffi guile-2.2?)
   (package
-    (name "guile-gdbm-ffi")
+    (name (if guile-2.2?
+              "guile2.2-gdbm-ffi"
+              "guile-gdbm-ffi"))
     (version "20120209.fa1d5b6")
     (source (origin
               (method git-fetch)
@@ -844,11 +846,12 @@ inspired by the SCSH regular expression system.")
        ((guix build utils))
        #:builder
        (begin
-         (use-modules (guix build utils)
-                      (system base compile))
+         (use-modules (guix build utils))
 
          (let* ((out (assoc-ref %outputs "out"))
-                (module-dir (string-append out "/share/guile/site/2.0"))
+                (module-dir (string-append out "/share/guile/site/"
+                                           ,(if guile-2.2?
+                                                "2.2" "2.0")))
                 (source (assoc-ref %build-inputs "source"))
                 (doc (string-append out "/share/doc"))
                 (guild (string-append (assoc-ref %build-inputs "guile")
@@ -856,7 +859,10 @@ inspired by the SCSH regular expression system.")
                 (gdbm.scm-dest
                  (string-append module-dir "/gdbm.scm"))
                 (gdbm.go-dest
-                 (string-append module-dir "/gdbm.go")))
+                 (string-append module-dir "/gdbm.go"))
+                (compile-file
+                 (lambda (in-file out-file)
+                   (system* guild "compile" "-o" out-file in-file))))
            ;; Make installation directories.
            (mkdir-p module-dir)
            (mkdir-p doc)
@@ -874,10 +880,10 @@ inspired by the SCSH regular expression system.")
                       (assoc-ref %build-inputs "gdbm"))))
 
            ;; compile to the destination
-           (compile-file gdbm.scm-dest
-                         #:output-file gdbm.go-dest)))))
+           (compile-file gdbm.scm-dest gdbm.go-dest)))))
     (inputs
-     `(("guile" ,guile-2.0)))
+     `(("guile" ,(if guile-2.2?
+                     guile-next guile-2.0))))
     (propagated-inputs
      `(("gdbm" ,gdbm)))
     (home-page "https://github.com/ijp/guile-gdbm")
@@ -887,8 +893,11 @@ inspired by the SCSH regular expression system.")
 Guile's foreign function interface.")
     (license gpl3+)))
 
+(define-public guile-gdbm-ffi
+  (make-guile-gdbm-ffi #f))
+
 (define-public guile2.2-gdbm-ffi
-  (package-for-guile-2.2 guile-gdbm-ffi))
+  (make-guile-gdbm-ffi #t))
 
 (define-public guile-sqlite3
   (let ((commit "607721fe1174a299e45d457acacf94eefb964071"))
-- 
2.11.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Fixing non-reproducibility in some guile packages

[Jan Nieuwenhuizen]

Christopher Allan Webber writes:

Hi!

> I think this is a pretty serious bug.  It means that guile-gdbm-ffi was
> never properly reproducible by our standard methods of determining
> inputs.  This is entirely my fault, since I'm the one who put together
> these packages.
>
> The guile-wisp package also has this issue.  I'm going to work on a fix
> for it.  In the meanwhile, here's a patch that both fixes
> guile2.2-gdbm-ffi and fixes the forementioned reproducibility problem.

While building guile2.2-gdbm-ffi an error is printed that does not
prevent the package from being built

     @ build-started /gnu/store/z9m20fz1ayyl0g9b4ad6wgmq3fv2h7gi-guile2.2-gdbm-ffi-20120209.fa1d5b6.drv - x86_64-linux /var/log/guix/drvs/z9//m20fz1ayyl0g9b4ad6wgmq3fv2h7gi-guile2.2-gdbm-ffi-20120209.fa1d5b6.drv.bz2
     ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
     ;;;       or pass the --no-auto-compile argument to disable.
     ;;; compiling /gnu/store/d3gli8g5bv6yhd3qwk5rfzqpsfvnj4lv-guile-next-2.1.5/bin/guild
     ;;; WARNING: compilation of /gnu/store/d3gli8g5bv6yhd3qwk5rfzqpsfvnj4lv-guile-next-2.1.5/bin/guild failed:
     ;;; ERROR: failed to create path for auto-compiled file "/gnu/store/d3gli8g5bv6yhd3qwk5rfzqpsfvnj4lv-guile-next-2.1.5/bin/guild"
     wrote `/gnu/store/xskgkfsxz936nifjs8vxqwk95kf62ia8-guile2.2-gdbm-ffi-20120209.fa1d5b6/share/guile/site/2.2/gdbm.go'
     @ build-succeeded /gnu/store/z9m20fz1ayyl0g9b4ad6wgmq3fv2h7gi-guile2.2-gdbm-ffi-20120209.fa1d5b6.drv -
     /gnu/store/xskgkfsxz936nifjs8vxqwk95kf62ia8-guile2.2-gdbm-ffi-20120209.fa1d5b6
     22:00:06 janneke@dundal:~/src/guix

The builds are not reproducible, but I guess that's expected because
guile-2.0.12 is used and guile-2.1.x is not patched to be reproducible
yet.

Greetings, janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar®  http://AvatarAcademy.nl  

Re: Fixing non-reproducibility in some guile packages

[Christopher Allan Webber]

[-- Attachment #1.1: Type: text/plain, Size: 1316 bytes --]

Jan Nieuwenhuizen writes:

> While building guile2.2-gdbm-ffi an error is printed that does not
> prevent the package from being built
>
>      @ build-started /gnu/store/z9m20fz1ayyl0g9b4ad6wgmq3fv2h7gi-guile2.2-gdbm-ffi-20120209.fa1d5b6.drv - x86_64-linux /var/log/guix/drvs/z9//m20fz1ayyl0g9b4ad6wgmq3fv2h7gi-guile2.2-gdbm-ffi-20120209.fa1d5b6.drv.bz2
>      ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
>      ;;;       or pass the --no-auto-compile argument to disable.
>      ;;; compiling /gnu/store/d3gli8g5bv6yhd3qwk5rfzqpsfvnj4lv-guile-next-2.1.5/bin/guild
>      ;;; WARNING: compilation of /gnu/store/d3gli8g5bv6yhd3qwk5rfzqpsfvnj4lv-guile-next-2.1.5/bin/guild failed:
>      ;;; ERROR: failed to create path for auto-compiled file "/gnu/store/d3gli8g5bv6yhd3qwk5rfzqpsfvnj4lv-guile-next-2.1.5/bin/guild"
>      wrote `/gnu/store/xskgkfsxz936nifjs8vxqwk95kf62ia8-guile2.2-gdbm-ffi-20120209.fa1d5b6/share/guile/site/2.2/gdbm.go'
>      @ build-succeeded /gnu/store/z9m20fz1ayyl0g9b4ad6wgmq3fv2h7gi-guile2.2-gdbm-ffi-20120209.fa1d5b6.drv -
>      /gnu/store/xskgkfsxz936nifjs8vxqwk95kf62ia8-guile2.2-gdbm-ffi-20120209.fa1d5b6
>      22:00:06 janneke@dundal:~/src/guix

Ah yeah, okay!  I fixed this by adding GUILE_AUTO_COMPILE=0 to the
environment while compiling.  New patch attached!


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-guile-gdbm-ffi-Write-to-correct-guile-output-directo.patch --]
[-- Type: text/x-patch, Size: 3956 bytes --]

From 447cf04e0fc5947e8384851ecfa9fffd61638c39 Mon Sep 17 00:00:00 2001
From: Christopher Allan Webber <cwebber@dustycloud.org>
Date: Fri, 10 Feb 2017 19:24:57 -0600
Subject: [PATCH] guile-gdbm-ffi: Write to correct guile output directory and
 use guild.

* gnu/packages/guile.scm (make-guile-gdbm-ffi): New variable.
Adapts from the previous guile-gdbm-ffi definition.  Also fixes
a bug where the guild command was not getting called, and instead
was calling the internal guile compile-file procedure.  This meant
that the package produced was dependent on whatever version of
guile was powering Guix at the time.  Also set GUILE_AUTO_COMPILE
to 0 to avoid gnarly looking warnings during build.
(guile-gdbm-ffi, guile2.2-gdbm-ffi): Use make-guile-gdbm-ffi.
---
 gnu/packages/guile.scm | 31 +++++++++++++++++++++----------
 1 file changed, 21 insertions(+), 10 deletions(-)

diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
index 3e8ab007b..6e1831fb4 100644
--- a/gnu/packages/guile.scm
+++ b/gnu/packages/guile.scm
@@ -825,9 +825,11 @@ inspired by the SCSH regular expression system.")
 ;; There are two guile-gdbm packages, one using the FFI and one with
 ;; direct C bindings, hence the verbose name.
 
-(define-public guile-gdbm-ffi
+(define (make-guile-gdbm-ffi guile-2.2?)
   (package
-    (name "guile-gdbm-ffi")
+    (name (if guile-2.2?
+              "guile2.2-gdbm-ffi"
+              "guile-gdbm-ffi"))
     (version "20120209.fa1d5b6")
     (source (origin
               (method git-fetch)
@@ -844,11 +846,14 @@ inspired by the SCSH regular expression system.")
        ((guix build utils))
        #:builder
        (begin
-         (use-modules (guix build utils)
-                      (system base compile))
+         (use-modules (guix build utils))
+
+         (setenv "GUILE_AUTO_COMPILE" "0")
 
          (let* ((out (assoc-ref %outputs "out"))
-                (module-dir (string-append out "/share/guile/site/2.0"))
+                (module-dir (string-append out "/share/guile/site/"
+                                           ,(if guile-2.2?
+                                                "2.2" "2.0")))
                 (source (assoc-ref %build-inputs "source"))
                 (doc (string-append out "/share/doc"))
                 (guild (string-append (assoc-ref %build-inputs "guile")
@@ -856,7 +861,10 @@ inspired by the SCSH regular expression system.")
                 (gdbm.scm-dest
                  (string-append module-dir "/gdbm.scm"))
                 (gdbm.go-dest
-                 (string-append module-dir "/gdbm.go")))
+                 (string-append module-dir "/gdbm.go"))
+                (compile-file
+                 (lambda (in-file out-file)
+                   (system* guild "compile" "-o" out-file in-file))))
            ;; Make installation directories.
            (mkdir-p module-dir)
            (mkdir-p doc)
@@ -874,10 +882,10 @@ inspired by the SCSH regular expression system.")
                       (assoc-ref %build-inputs "gdbm"))))
 
            ;; compile to the destination
-           (compile-file gdbm.scm-dest
-                         #:output-file gdbm.go-dest)))))
+           (compile-file gdbm.scm-dest gdbm.go-dest)))))
     (inputs
-     `(("guile" ,guile-2.0)))
+     `(("guile" ,(if guile-2.2?
+                     guile-next guile-2.0))))
     (propagated-inputs
      `(("gdbm" ,gdbm)))
     (home-page "https://github.com/ijp/guile-gdbm")
@@ -887,8 +895,11 @@ inspired by the SCSH regular expression system.")
 Guile's foreign function interface.")
     (license gpl3+)))
 
+(define-public guile-gdbm-ffi
+  (make-guile-gdbm-ffi #f))
+
 (define-public guile2.2-gdbm-ffi
-  (package-for-guile-2.2 guile-gdbm-ffi))
+  (make-guile-gdbm-ffi #t))
 
 (define-public guile-sqlite3
   (let ((commit "607721fe1174a299e45d457acacf94eefb964071"))
-- 
2.11.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Fixing non-reproducibility in some guile packages

[Maxim Cournoyer]

[-- Attachment #1: Type: text/plain, Size: 1450 bytes --]

Hi!

Christopher Allan Webber <cwebber@dustycloud.org> writes:

> Jan Nieuwenhuizen writes:
>
>> While building guile2.2-gdbm-ffi an error is printed that does not
>> prevent the package from being built
>>
>>      @ build-started
>> /gnu/store/z9m20fz1ayyl0g9b4ad6wgmq3fv2h7gi-guile2.2-gdbm-ffi-20120209.fa1d5b6.drv
>> - x86_64-linux
>> /var/log/guix/drvs/z9//m20fz1ayyl0g9b4ad6wgmq3fv2h7gi-guile2.2-gdbm-ffi-20120209.fa1d5b6.drv.bz2
>>      ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
>>      ;;;       or pass the --no-auto-compile argument to disable.
>>      ;;; compiling /gnu/store/d3gli8g5bv6yhd3qwk5rfzqpsfvnj4lv-guile-next-2.1.5/bin/guild
>>      ;;; WARNING: compilation of /gnu/store/d3gli8g5bv6yhd3qwk5rfzqpsfvnj4lv-guile-next-2.1.5/bin/guild failed:
>>      ;;; ERROR: failed to create path for auto-compiled file

I've notice this same problem recently while looking at the build-log of
our GnuCash package.. It seems to be the problem is caused by the Guile
wanting to auto-compile the guild script, and not being able to as it
wants to store the compiled version somewhere under $HOME.

Could a generally applicable solution be to distribute a pre-compiled
guild.go in the store, so that it would use that and never attempt to
'auto-compile' it?  This would effectively fix this problem for good
without having to create workaround such as using environment variables
or setting HOME to a temp dir.

WDYT?

Maxim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Fixing non-reproducibility in some guile packages

[Andy Wingo]

Hi :)

[+guile-devel]

On Mon 13 Feb 2017 07:18, Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

>>>      ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
>>>      ;;;       or pass the --no-auto-compile argument to disable.
>>>      ;;; compiling /gnu/store/d3gli8g5bv6yhd3qwk5rfzqpsfvnj4lv-guile-next-2.1.5/bin/guild
>
> I've notice this same problem recently while looking at the build-log of
> our GnuCash package.. It seems to be the problem is caused by the Guile
> wanting to auto-compile the guild script, and not being able to as it
> wants to store the compiled version somewhere under $HOME.
>
> Could a generally applicable solution be to distribute a pre-compiled
> guild.go in the store, so that it would use that and never attempt to
> 'auto-compile' it?  This would effectively fix this problem for good
> without having to create workaround such as using environment variables
> or setting HOME to a temp dir.

The compiled .go files that get installed are only for files found in
the Guile load path.  So like when you load (foo bar), it will look for
foo/bar.go in the %load-compiled-path (and foo/bar.scm in the
%load-path).  This lets you compile foo/bar.scm in your build directory
and have the resulting foo/bar.go in the builddir directly usable from
the build-time %load-path / %load-compiled-path, and to install you just
copy into the final %load-path / %load-compiled-path and that works too.

For scripts that are loaded by absolute file name (guild is a prominent
example) that don't live in the %load-path, this technique isn't
directly applicable.  The existing %load-compiled-path is effectively a
function from name-suffix (like "foo/bar.scm") to absolute path of
compiled .go file.  We would need an additional mechanism to be a
function from absolute source file name to absolute path of .go file.
Perhaps that's doable with an additional path.  Perhaps the fallback
compilation path (~/.cache/guile/...) could be generalized to include
something in $prefix also so that packages can install compiled files.
Incidentally probably we should SHA256 the path to prevent so much
directory traversal...

In some future (is it near or far?), the source -> compiled function
needs additional inputs: checksums or timestamps of "build inputs" or
so, so that when for-syntax definitions (like macros) change, users of
those definitions will recompile.  That is a harder problem though.

Andy

Re: Fixing non-reproducibility in some guile packages

[Christopher Allan Webber]

Andy Wingo writes:

> In some future (is it near or far?), the source -> compiled function
> needs additional inputs: checksums or timestamps of "build inputs" or
> so, so that when for-syntax definitions (like macros) change, users of
> those definitions will recompile.  That is a harder problem though.

Ah yeah, I've been bit by this before.  Though, I imagine we'd run into
a problem where we'd never know how to "garbage collect" anything in
~/.cache/guile/ .  "Maybe not a problem", except if you're hacking some files
constantly... :)

Maybe every guile hacker needs to get in the habit of
`rm -rf ~/.cache/guile/' though? :)

Re: Fixing non-reproducibility in some guile packages

[Maxim Cournoyer]

[-- Attachment #1.1: Type: text/plain, Size: 2668 bytes --]

Hi Andy, and thank you for your detailed answer!

Andy Wingo <wingo@igalia.com> writes:

> Hi :)
>
> [+guile-devel]
>
> On Mon 13 Feb 2017 07:18, Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>>>>      ;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
>>>>      ;;;       or pass the --no-auto-compile argument to disable.
>>>>      ;;; compiling /gnu/store/d3gli8g5bv6yhd3qwk5rfzqpsfvnj4lv-guile-next-2.1.5/bin/guild
>>
>> I've notice this same problem recently while looking at the build-log of
>> our GnuCash package.. It seems to be the problem is caused by the Guile
>> wanting to auto-compile the guild script, and not being able to as it
>> wants to store the compiled version somewhere under $HOME.
>>
>> Could a generally applicable solution be to distribute a pre-compiled
>> guild.go in the store, so that it would use that and never attempt to
>> 'auto-compile' it?  This would effectively fix this problem for good
>> without having to create workaround such as using environment variables
>> or setting HOME to a temp dir.
>
> The compiled .go files that get installed are only for files found in
> the Guile load path.  So like when you load (foo bar), it will look for
> foo/bar.go in the %load-compiled-path (and foo/bar.scm in the
> %load-path).  This lets you compile foo/bar.scm in your build directory
> and have the resulting foo/bar.go in the builddir directly usable from
> the build-time %load-path / %load-compiled-path, and to install you just
> copy into the final %load-path / %load-compiled-path and that works too.
>
> For scripts that are loaded by absolute file name (guild is a prominent
> example) that don't live in the %load-path, this technique isn't
> directly applicable.  The existing %load-compiled-path is effectively a
> function from name-suffix (like "foo/bar.scm") to absolute path of
> compiled .go file.

OK, I see how %load-path and %load-compiled-path currently fail to be
useful in the case of loading a guile script from its absolute
path. Thanks for explaining. 

> function from absolute source file name to absolute path of .go file.
> Perhaps that's doable with an additional path.  Perhaps the fallback
> compilation path (~/.cache/guile/...) could be generalized to include
> something in $prefix also so that packages can install compiled files.

This seem like it would be the simplest solution in the short term, but
maybe not the most elegant. The fallback compilation path seems to refer
to the source file by their exact location though, so it would mean that
if someone was to change the location of the store it would break:


[-- Attachment #1.2: Type: text/plain, Size: 334 bytes --]

guild
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling /gnu/store/jz5pb07j8s9vv07f27p52mmsiap3cwps-profile/bin/guild
;;; compiled
/home/maxim/.cache/guile/ccache/2.0-LE-8-2.0/gnu/store/060piiiz4nmb51jc3wk01bgikajrnfjd-guile-2.0.13/bin/guild.go

[-- Attachment #1.3: Type: text/plain, Size: 429 bytes --]


> Incidentally probably we should SHA256 the path to prevent so much
> directory traversal...
>
> In some future (is it near or far?), the source -> compiled function
> needs additional inputs: checksums or timestamps of "build inputs" or
> so, so that when for-syntax definitions (like macros) change, users of
> those definitions will recompile.  That is a harder problem though.
>

Interesting problem!

Thanks again,

Maxim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Fixing non-reproducibility in some guile packages

[Ludovic Courtès]

Hi Chris,

Sorry for the late reply!

Christopher Allan Webber <cwebber@dustycloud.org> skribis:

>           (let* ((out (assoc-ref %outputs "out"))
> -                (module-dir (string-append out "/share/guile/site/2.0"))
> +                (module-dir (string-append out "/share/guile/site/"
> +                                           ,(if guile-2.2?
> +                                                "2.2" "2.0")))
>                  (source (assoc-ref %build-inputs "source"))

Another approach, which is more future-proof but also more verbose, is
to evaluate (effective-version) for the Guile that’s being used, on the
“build side” (thus, no need to do the unquote thing above).

The ‘guile-minikanren’ package does exactly that:

         (let* ((out (assoc-ref %outputs "out"))
                (guile (assoc-ref %build-inputs "guile"))
                (effective (read-line
                            (open-pipe* OPEN_READ
                                        (string-append guile "/bin/guile")
                                        "-c" "(display (effective-version))")))
                (module-dir (string-append out "/share/guile/site/"
                                           effective)) …)
            …)

We should probably factorize this somewhere (a new (guix build guile)
module?), but for now that’s what we have.

How does that sound?

Thanks,
Ludo’.

Re: Fixing non-reproducibility in some guile packages

[Christopher Allan Webber]

[-- Attachment #1.1: Type: text/plain, Size: 2021 bytes --]

Ludovic Courtès writes:

> Christopher Allan Webber <cwebber@dustycloud.org> skribis:
>
>>           (let* ((out (assoc-ref %outputs "out"))
>> -                (module-dir (string-append out "/share/guile/site/2.0"))
>> +                (module-dir (string-append out "/share/guile/site/"
>> +                                           ,(if guile-2.2?
>> +                                                "2.2" "2.0")))
>>                  (source (assoc-ref %build-inputs "source"))
>
> Another approach, which is more future-proof but also more verbose, is
> to evaluate (effective-version) for the Guile that’s being used, on the
> “build side” (thus, no need to do the unquote thing above).
>
> The ‘guile-minikanren’ package does exactly that:
>
>          (let* ((out (assoc-ref %outputs "out"))
>                 (guile (assoc-ref %build-inputs "guile"))
>                 (effective (read-line
>                             (open-pipe* OPEN_READ
>                                         (string-append guile "/bin/guile")
>                                         "-c" "(display (effective-version))")))
>                 (module-dir (string-append out "/share/guile/site/"
>                                            effective)) …)
>             …)
>
> We should probably factorize this somewhere (a new (guix build guile)
> module?), but for now that’s what we have.
>
> How does that sound?

It looks much better!  I've updated my patch to use this method.

Re: the new module, I agree it's a good idea.  We had some talks at
FOSDEM about maybe supporting more declarative guild'y type
configuration again, and maybe that's the right approach, I don't really
know.  It feels like "what should we do about easy to package guile
packages that don't necessarily use autotools" is still a conversation
to be had.  Not sure if that's a prerequisite for the module.

Anyway, okay to push?  I'd love to have the buggy package not be buggy
in master. :)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-guile-gdbm-ffi-Write-to-correct-guile-output-directo.patch --]
[-- Type: text/x-patch, Size: 3082 bytes --]

From 369582d7c8b5ce1249761337319e79cc117f161e Mon Sep 17 00:00:00 2001
From: Christopher Allan Webber <cwebber@dustycloud.org>
Date: Fri, 10 Feb 2017 19:24:57 -0600
Subject: [PATCH] guile-gdbm-ffi: Write to correct guile output directory and
 use guild.

* gnu/packages/guile.scm (guile-gdbm-ffi): Check guile for effective version
before writing to output path.  Also fixes a bug where the guild command was
not getting called, and instead was calling the internal guile compile-file
procedure.  This meant that the package produced was dependent on whatever
version of guile was powering Guix at the time.  Also set GUILE_AUTO_COMPILE
to 0 to avoid gnarly looking warnings during build.
---
 gnu/packages/guile.scm | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)

diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm
index 3e8ab007b..75f561c03 100644
--- a/gnu/packages/guile.scm
+++ b/gnu/packages/guile.scm
@@ -844,11 +844,20 @@ inspired by the SCSH regular expression system.")
        ((guix build utils))
        #:builder
        (begin
-         (use-modules (guix build utils)
-                      (system base compile))
+         (use-modules (guix build utils))
+
+         (setenv "GUILE_AUTO_COMPILE" "0")
 
          (let* ((out (assoc-ref %outputs "out"))
-                (module-dir (string-append out "/share/guile/site/2.0"))
+                (effective-version
+                 (read-line
+                  (open-pipe* OPEN_READ
+                              (string-append
+                               (assoc-ref %build-inputs "guile")
+                               "/bin/guile")
+                              "-c" "(display (effective-version))")))
+                (module-dir (string-append out "/share/guile/site/"
+                                           effective-version))
                 (source (assoc-ref %build-inputs "source"))
                 (doc (string-append out "/share/doc"))
                 (guild (string-append (assoc-ref %build-inputs "guile")
@@ -856,7 +865,10 @@ inspired by the SCSH regular expression system.")
                 (gdbm.scm-dest
                  (string-append module-dir "/gdbm.scm"))
                 (gdbm.go-dest
-                 (string-append module-dir "/gdbm.go")))
+                 (string-append module-dir "/gdbm.go"))
+                (compile-file
+                 (lambda (in-file out-file)
+                   (system* guild "compile" "-o" out-file in-file))))
            ;; Make installation directories.
            (mkdir-p module-dir)
            (mkdir-p doc)
@@ -874,8 +886,7 @@ inspired by the SCSH regular expression system.")
                       (assoc-ref %build-inputs "gdbm"))))
 
            ;; compile to the destination
-           (compile-file gdbm.scm-dest
-                         #:output-file gdbm.go-dest)))))
+           (compile-file gdbm.scm-dest gdbm.go-dest)))))
     (inputs
      `(("guile" ,guile-2.0)))
     (propagated-inputs
-- 
2.11.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]