bug#26987: guix pull without guile-ssh.

bug#26987: guix pull without guile-ssh.

[Mathieu Othacehe]

Hi,

If Guile-SSH is not detected, depends-on-guile-ssh? will be called here:

--8<---------------cut here---------------start------------->8---
(guix build pull)

(let* ((files (remove (if (false-if-exception
                           (resolve-interface '(ssh session)))
                          (const #f)
                        depends-on-guile-ssh?)
                      (all-scheme-files out)))
--8<---------------cut here---------------end--------------->8---

Thus, source-module-closure is called with a file. However,
source-module-closure is expecting a module list like '((guix ...)).

This result in the following error while running guix pull:

--8<---------------cut here---------------start------------->8---
In ./guix/modules.scm:
    118:4  0 (module-closure _ #:select? _ #:dependencies _)

./guix/modules.scm:118:4: In procedure module-closure:
./guix/modules.scm:118:4: Throw to key `match-error' with args `("match" "no matching pattern" "/gnu/store/0slxknjvjwhp31qryv42419vqwjrjifb-guix-latest/guix.scm")'.
builder for `/gnu/store/lq6ga13k75ljrdnqam9jm1q92cv4g0pm-guix-latest.drv' failed with exit code 1
--8<---------------cut here---------------end--------------->8---

I'm not sure how to fix it.

Thanks,

Mathieu

bug#26987: [PATCH 0/2] Fix compilation list construction.

[Mathieu Othacehe]

Hi,

Here are two patches to fix compilation list construction.
It seems to work but I'm higly unsure this is the right
thing to do.

Plus, while testing I could not add #:use-modules (guix discovery)
and had to write that instead :

((@@ (guix discovery) scheme-modules) directory)

Thanks,

Mathieu

Mathieu Othacehe (2):
  build: pull: Fix compilation list construction.
  guix: modules: Export module-name->file-name.

 guix/build/pull.scm | 31 ++++++++-----------------------
 guix/modules.scm    |  1 +
 2 files changed, 9 insertions(+), 23 deletions(-)

-- 
2.13.0

bug#26987: [PATCH 1/2] guix: modules: Export module-name->file-name.

[Mathieu Othacehe]

* guix/modules.scm (module-name->file-name): Export it.
---
 guix/modules.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/guix/modules.scm b/guix/modules.scm
index 8c63f21a9..26c38e8d2 100644
--- a/guix/modules.scm
+++ b/guix/modules.scm
@@ -23,6 +23,7 @@
   #:use-module (ice-9 match)
   #:export (source-module-closure
             live-module-closure
+            module-name->file-name
             guix-module-name?))
 
 ;;; Commentary:
-- 
2.13.0

bug#26987: [PATCH 2/2] build: pull: Fix compilation list construction.

[Mathieu Othacehe]

* guix/build/pull.scm (depends-on-guile-ssh?): Remove.
(all-scheme-files): Use "scheme-modules" to detect all loadable
modules in given directory and rebuild a file list from those modules.
---
 guix/build/pull.scm | 31 ++++++++-----------------------
 1 file changed, 8 insertions(+), 23 deletions(-)

diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index d2e0404b1..e6ec71a54 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2013, 2014, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2015 Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
+;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -36,26 +37,14 @@
 ;;;
 ;;; Code:
 
-(define (depends-on-guile-ssh? file)
-  "Return true if FILE is a Scheme source file that depends, directly or
-indirectly, on Guile-SSH."
-  (find (match-lambda
-          (('ssh _ ...) #t)
-          (_ #f))
-        (source-module-closure file #:select? (const #t))))
-
 (define (all-scheme-files directory)
   "Return a sorted list of Scheme files found in DIRECTORY."
-  ;; Load guix/ modules before gnu/ modules to get somewhat steadier
-  ;; progress reporting.
-  (sort (filter (cut string-suffix? ".scm" <>)
-                (find-files directory "\\.scm"))
-        (let ((guix (string-append directory "/guix"))
-              (gnu  (string-append directory "/gnu")))
-          (lambda (a b)
-            (or (and (string-prefix? guix a)
-                     (string-prefix? gnu  b))
-                (string<? a b))))))
+  (let ((modules (map module-name
+		      ((@@ (guix discovery) scheme-modules) directory))))
+    (map (lambda (module)
+           (string-append directory "/"
+                          (module-name->file-name module)))
+        modules)))
 
 (cond-expand
   (guile-2.2 (use-modules (language tree-il optimize)
@@ -145,11 +134,7 @@ containing the source code.  Write any debugging output to DEBUG-PORT."
     ;; Compile the .scm files.  Load all the files before compiling them to
     ;; work around <http://bugs.gnu.org/15602> (FIXME).
     ;; Filter out files depending on Guile-SSH when Guile-SSH is missing.
-    (let* ((files (remove (if (false-if-exception
-                               (resolve-interface '(ssh session)))
-                              (const #f)
-                              depends-on-guile-ssh?)
-                          (all-scheme-files out)))
+    (let* ((files (all-scheme-files out))
            (total (length files)))
       (let loop ((files files)
                  (completed 0))
-- 
2.13.0

bug#26987: [PATCH 1/2] guix: modules: Export module-name->file-name.

[Ludovic Courtès]

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> * guix/modules.scm (module-name->file-name): Export it.

OK!

bug#26987: [PATCH 2/2] build: pull: Fix compilation list construction.

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 2993 bytes --]

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

> * guix/build/pull.scm (depends-on-guile-ssh?): Remove.
> (all-scheme-files): Use "scheme-modules" to detect all loadable
> modules in given directory and rebuild a file list from those modules.
> ---
>  guix/build/pull.scm | 31 ++++++++-----------------------
>  1 file changed, 8 insertions(+), 23 deletions(-)
>
> diff --git a/guix/build/pull.scm b/guix/build/pull.scm
> index d2e0404b1..e6ec71a54 100644
> --- a/guix/build/pull.scm
> +++ b/guix/build/pull.scm
> @@ -1,6 +1,7 @@
>  ;;; GNU Guix --- Functional package management for GNU
>  ;;; Copyright © 2013, 2014, 2016, 2017 Ludovic Courtès <ludo@gnu.org>
>  ;;; Copyright © 2015 Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
> +;;; Copyright © 2017 Mathieu Othacehe <m.othacehe@gmail.com>
>  ;;;
>  ;;; This file is part of GNU Guix.
>  ;;;
> @@ -36,26 +37,14 @@
>  ;;;
>  ;;; Code:
>  
> -(define (depends-on-guile-ssh? file)
> -  "Return true if FILE is a Scheme source file that depends, directly or
> -indirectly, on Guile-SSH."
> -  (find (match-lambda
> -          (('ssh _ ...) #t)
> -          (_ #f))
> -        (source-module-closure file #:select? (const #t))))
> -
>  (define (all-scheme-files directory)
>    "Return a sorted list of Scheme files found in DIRECTORY."
> -  ;; Load guix/ modules before gnu/ modules to get somewhat steadier
> -  ;; progress reporting.
> -  (sort (filter (cut string-suffix? ".scm" <>)
> -                (find-files directory "\\.scm"))
> -        (let ((guix (string-append directory "/guix"))
> -              (gnu  (string-append directory "/gnu")))
> -          (lambda (a b)
> -            (or (and (string-prefix? guix a)
> -                     (string-prefix? gnu  b))
> -                (string<? a b))))))
> +  (let ((modules (map module-name
> +		      ((@@ (guix discovery) scheme-modules) directory))))
> +    (map (lambda (module)
> +           (string-append directory "/"
> +                          (module-name->file-name module)))
> +        modules)))
>  
>  (cond-expand
>    (guile-2.2 (use-modules (language tree-il optimize)
> @@ -145,11 +134,7 @@ containing the source code.  Write any debugging output to DEBUG-PORT."
>      ;; Compile the .scm files.  Load all the files before compiling them to
>      ;; work around <http://bugs.gnu.org/15602> (FIXME).
>      ;; Filter out files depending on Guile-SSH when Guile-SSH is missing.
> -    (let* ((files (remove (if (false-if-exception
> -                               (resolve-interface '(ssh session)))
> -                              (const #f)
> -                              depends-on-guile-ssh?)
> -                          (all-scheme-files out)))
> +    (let* ((files (all-scheme-files out))
>             (total (length files)))

I think this patch does to thing: (1) fixing the bug at hand, and (2)
using (guix discovery).

For (1), I think this should work:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 812 bytes --]

diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index d2e0404b1..84629f6b0 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -39,10 +39,15 @@
 (define (depends-on-guile-ssh? file)
   "Return true if FILE is a Scheme source file that depends, directly or
 indirectly, on Guile-SSH."
+  (let ((module (call-with-input-file file
+                  (lambda (port)
+                    (match (read port)
+                      (('define-module name _ ...)
+                       name))))))
     (find (match-lambda
             (('ssh _ ...) #t)
             (_ #f))
-        (source-module-closure file #:select? (const #t))))
+          (source-module-closure module #:select? (const #t)))))
 
 (define (all-scheme-files directory)
   "Return a sorted list of Scheme files found in DIRECTORY."

[-- Attachment #3: Type: text/plain, Size: 21 bytes --]


WDYT?

Ludo’.

bug#26987: [PATCH 2/2] build: pull: Fix compilation list construction.

[Mathieu Othacehe]

Hi Ludo,

It seems better but I can't get it to work neither :

>      (find (match-lambda
>              (('ssh _ ...) #t)
>              (_ #f))
> -        (source-module-closure file #:select? (const #t))))
> +          (source-module-closure module #:select? (const #t)))))

                                       ^
                                       we need a list here.
>  
>  (define (all-scheme-files directory)
>    "Return a sorted list of Scheme files found in DIRECTORY."
>
> WDYT?

So with

--8<---------------cut here---------------start------------->8---
(define (depends-on-guile-ssh? file)
  "Return true if FILE is a Scheme source file that depends, directly or
indirectly, on Guile-SSH."
  (let ((module (call-with-input-file file
                  (lambda (port)
                    (match (read port)
                      (('define-module name _ ...)
                       name))))))
    (find (match-lambda
            (('ssh _ ...) #t)
            (_ #f))
          (source-module-closure (list module) #:select? (const #t)))))
--8<---------------cut here---------------end--------------->8---

I get,

--8<---------------cut here---------------start------------->8---
((@@ (guix build pull) depends-on-guile-ssh?) "/home/mathieu/guix/guix/ssh.scm")

ERROR: In procedure open-file:
ERROR: Wrong type (expecting string): #f
--8<---------------cut here---------------end--------------->8---

I'm having a hard time tring to understand ,trace output.

Mathieu

bug#26987: [PATCH 2/2] build: pull: Fix compilation list construction.

[Mathieu Othacehe]

> I'm having a hard time tring to understand ,trace output.

I get why, it's because guile-ssh is not in my %load-path. So loading
"ssh/session.scm" fails in "source-module-dependencies".

It's a bit of a vicious circle here : if resolve-interface '(ssh
session) fails, depends-on-guile-ssh? will fail too.

I'm note sure we can depend on source-module-closure to do the job here.

Mathieu

bug#26987: [PATCH 2/2] build: pull: Fix compilation list construction.

[Ludovic Courtès]

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>> I'm having a hard time tring to understand ,trace output.
>
> I get why, it's because guile-ssh is not in my %load-path. So loading
> "ssh/session.scm" fails in "source-module-dependencies".
>
> It's a bit of a vicious circle here : if resolve-interface '(ssh
> session) fails, depends-on-guile-ssh? will fail too.
>
> I'm note sure we can depend on source-module-closure to do the job here.

Oh, got it.  Hmm, needs more thought.

Ludo’.

bug#26987: [PATCH 2/2] build: pull: Fix compilation list construction.

[Ludovic Courtès]

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>> I'm having a hard time tring to understand ,trace output.
>
> I get why, it's because guile-ssh is not in my %load-path. So loading
> "ssh/session.scm" fails in "source-module-dependencies".
>
> It's a bit of a vicious circle here : if resolve-interface '(ssh
> session) fails, depends-on-guile-ssh? will fail too.
>
> I'm note sure we can depend on source-module-closure to do the job here.

I believe this is fixed by commit
20ed093977cc80ba1729c38e05ae7955a38069a6, which follows a modification
to the ‘source-module-closure’ so that callers can catch
missing-dependency errors.

Please let me know what you think!

After that, you should be able to reinstate the (guix git) commit,
provided build-aux/build-self.scm does:

  (define guile-git
    (package-for-current-guile "guile-git"
                               "guile2.2-git"
                               "guile2.0-git"))


and passes that down to ‘build-guix’.

Thanks,
Ludo’.

bug#26987: [PATCH 2/2] build: pull: Fix compilation list construction.

[Mathieu Othacehe]

Hi Ludo,

> I believe this is fixed by commit
> 20ed093977cc80ba1729c38e05ae7955a38069a6, which follows a modification
> to the ‘source-module-closure’ so that callers can catch
> missing-dependency errors.
>
> Please let me know what you think!

It seems ok, thanks for fixing it !

>
> After that, you should be able to reinstate the (guix git) commit,
> provided build-aux/build-self.scm does:
>
>   (define guile-git
>     (package-for-current-guile "guile-git"
>                                "guile2.2-git"
>                                "guile2.0-git"))
                                        ^
                           for guile2.0-git,
we need to define guile2.0-bytestructures, so I tried to package it,
and I get the following error :

--8<---------------cut here---------------start------------->8---
ice-9/boot-9.scm:109:20: In procedure #<procedure 9dbe40 at ice-9/boot-9.scm:100:6 (thrown-k . 
ice-9/boot-9.scm:109:20: re-exporting local variable: cstring-pointer
--8<---------------cut here---------------end--------------->8---

It is something introduced in bytestructures commit 91d042e, that works
on 2.2 but not on 2.0, I have no idea why !

Amirouche any idea ?

Thanks,

Mathieu

bug#26987: [PATCH 2/2] build: pull: Fix compilation list construction.

[Ludovic Courtès]

Hi Mathieu,

Mathieu Othacehe <m.othacehe@gmail.com> skribis:

>> I believe this is fixed by commit
>> 20ed093977cc80ba1729c38e05ae7955a38069a6, which follows a modification
>> to the ‘source-module-closure’ so that callers can catch
>> missing-dependency errors.
>>
>> Please let me know what you think!
>
> It seems ok, thanks for fixing it !
>
>>
>> After that, you should be able to reinstate the (guix git) commit,
>> provided build-aux/build-self.scm does:
>>
>>   (define guile-git
>>     (package-for-current-guile "guile-git"
>>                                "guile2.2-git"
>>                                "guile2.0-git"))
>                                         ^
>                            for guile2.0-git,
> we need to define guile2.0-bytestructures, so I tried to package it,
> and I get the following error :
>
> ice-9/boot-9.scm:109:20: In procedure #<procedure 9dbe40 at ice-9/boot-9.scm:100:6 (thrown-k . 
> ice-9/boot-9.scm:109:20: re-exporting local variable: cstring-pointer
>
> It is something introduced in bytestructures commit 91d042e, that works
> on 2.2 but not on 2.0, I have no idea why !

It would be nice to have guile2.0-git, but note that it does not *have*
to exist: ‘package-for-current-guile’ returns #f if none of the packages
exists, and (guix build pull) skips modules that lack their
dependencies.

So I would first suggest to re-add (guix git), and later see with
Amirouche et al. what could be done to get Guile-Git on 2.0.

Does that make sense?

Thanks,
Ludo’.