Question about sbcl-package->ecl-package

Question about sbcl-package->ecl-package

[Guillaume Le Vaillant]

Hi,

I'm trying to package a Common Lisp library and I have a strange
problem.

In 'gnu/packages/lisp.scm', there are packages called 'sbcl-chanl' and
'ecl-chanl' whose definitions are:

--8<---------------cut here---------------start------------->8---
(define-public sbcl-chanl
  (let ((commit "2362b57550c2c9238cc882d03553aaa1040b7340")
        (revision "0"))
    (package
      (name "sbcl-chanl")
      ...)))

(define-public ecl-chanl
  (let ((base (sbcl-package->ecl-package sbcl-chanl)))
    (package
      (inherit base)
      (arguments
       (substitute-keyword-arguments (package-arguments base)
         ;; Some phases are modified here so that compiling
         ;; with ECL works.
         ...)))))
--8<---------------cut here---------------end--------------->8---

Compiling these two packages works fine.

Now, I define the following new packages:

--8<---------------cut here---------------start------------->8---
(define-public sbcl-simple-parallel-tasks
  (let ((commit "db460f7a3f7bbfe2d3a2223ed21e162068d04dda")
        (revision "0"))
    (package
      (name "sbcl-simple-parallel-tasks")
      ...
      (inputs
       `(("chanl" ,sbcl-chanl)))
      ...)))

(define-public ecl-simple-parallel-tasks
  (sbcl-package->ecl-package sbcl-simple-parallel-tasks))
--8<---------------cut here---------------end--------------->8---

Compiling 'sbcl-simple-parallel-tasks' works fine.

However, when I try to compile 'ecl-simple-parallel-tasks', guix first
tries to build a different derivation of 'ecl-chanl', which fails
because it apparently doesn't have the modified phases declared in the
definition of 'ecl-chanl'.

More precisely:
- If I do 'guix build ecl-chanl', guix builds
  'y60p0wn2hwp5jr0hy2qb09yazkp29m7i-ecl-chanl-0.4.1-0.2362b57'
  successfully.
- If I do 'guix build ecl-simple-parallel-tasks', guix tries to build
  'qqzlyknj5wgrm0f0nm4wwafv3ldvhrgi-ecl-chanl-0.4.1-0.2362b57'
  and fails.

When doing 'guix build ecl-simple-parallel-tasks' I was expecting guix
to use the 'ecl-chanl' I had already compiled as input, but instead it
looks as if it tries to build
'(sbcl-package->ecl-package sbcl-chanl)' instead.

Does anyone know why in this case guix tries to compile a different
derivation of 'ecl-chanl' that I didn't define anywhere?

Re: Question about sbcl-package->ecl-package

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

I've encountered the same problem a couple of times.
If you try to compile ecl-dexador, you'll see it fails because it does
not re-use the arguments of sbcl-dexador which patches out a failing
test.

Something is wrong in sbcl-package->ecl-package.
Andy?

That said, it's not a very big deal, since the cl- package works for all compilers.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Question about sbcl-package->ecl-package

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 926 bytes --]

On Wed, Oct 16, 2019 at 01:59:01PM +0200, Pierre Neidhardt wrote:
> I've encountered the same problem a couple of times.
> If you try to compile ecl-dexador, you'll see it fails because it does
> not re-use the arguments of sbcl-dexador which patches out a failing
> test.
> 
> Something is wrong in sbcl-package->ecl-package.
> Andy?
> 
> That said, it's not a very big deal, since the cl- package works for all compilers.
> 

Sounds like the conversion isn't "recursive enough". On a per-package
basis you can replace the created ecl package with the real one. The
better option would be to look at how python defines python2- variants,
which sounds a lot like the problem you're having here.


-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Question about sbcl-package->ecl-package

[Guillaume Le Vaillant]

Efraim Flashner skribis:

> On Wed, Oct 16, 2019 at 01:59:01PM +0200, Pierre Neidhardt wrote:
>> I've encountered the same problem a couple of times.
>> If you try to compile ecl-dexador, you'll see it fails because it does
>> not re-use the arguments of sbcl-dexador which patches out a failing
>> test.
>> 
>> Something is wrong in sbcl-package->ecl-package.
>> Andy?
>> 
>> That said, it's not a very big deal, since the cl- package works for all compilers.
>> 
>
> Sounds like the conversion isn't "recursive enough". On a per-package
> basis you can replace the created ecl package with the real one. The
> better option would be to look at how python defines python2- variants,
> which sounds a lot like the problem you're having here.

I suspect the problem comes from the 'rewrite' function used to change
the package inputs. In 'guix/build-system/asdf.scm':

--8<---------------cut here---------------start------------->8---
(define rewrite
  (match-lambda
    ((name content . rest)
     (let* ((is-package? (package? content))
            (new-content (if is-package? (transform content) content)))
       `(,name ,new-content ,@rest)))))
--8<---------------cut here---------------end--------------->8---

Won't '(transform content)' create a new ecl-package for each input
package instead of trying to find the already defined ecl-package?

Maybe it could be replaced by something like:

--8<---------------cut here---------------start------------->8---
(let* ((sbcl-input-name (package-name content))
       (ecl-input-name (transform-package-name sbcl-input-name))
       (ecl-input-package (find-package ecl-name)))
  (if (package? ecl-input-package)
      ecl-input-package
      (transform content)))
--8<---------------cut here---------------end--------------->8---

However, it's the first time I look at the internals of Guix, so I'm
not sure if this would work or not...
What do you think?

Re: Question about sbcl-package->ecl-package

[Guillaume Le Vaillant]

Guillaume Le Vaillant skribis:

> However, when I try to compile 'ecl-simple-parallel-tasks', guix first
> tries to build a different derivation of 'ecl-chanl', which fails
> because it apparently doesn't have the modified phases declared in the
> definition of 'ecl-chanl'.

I was able to get guix to fetch the right package as input using the
following patch:

--8<---------------cut here---------------start------------->8---
From 4213b8b9d64e2536df7ede308772a38cc71e2bf4 Mon Sep 17 00:00:00 2001
From: Guillaume Le Vaillant <glv@posteo.net>
Date: Thu, 17 Oct 2019 12:07:38 +0200
Subject: [PATCH] build-system/asdf: Fix package transform.

* guix/build-system/asdf.scm (package-with-build-system):
  [find-input-package]: New function.
  [rewrite]: Use it.
---
 guix/build-system/asdf.scm | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/guix/build-system/asdf.scm b/guix/build-system/asdf.scm
index af04084c86..f794bf006b 100644
--- a/guix/build-system/asdf.scm
+++ b/guix/build-system/asdf.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016, 2017 Andy Patterson <ajpatter@uwaterloo.ca>
+;;; Copyright © 2019 Guillaume Le Vaillant <glv@posteo.net>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -32,6 +33,7 @@
   #:use-module (ice-9 regex)
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26)
+  #:use-module (gnu packages)
   #:export (%asdf-build-system-modules
             %asdf-build-modules
             asdf-build
@@ -160,13 +162,22 @@ set up using CL source package conventions."
   (define (has-from-build-system? pkg)
     (eq? from-build-system (package-build-system pkg)))
 
+  (define (find-input-package pkg)
+    (let* ((name (package-name pkg))
+           (new-name (transform-package-name name))
+           (pkgs (find-packages-by-name new-name)))
+      (if (null? pkgs) #f (list-ref pkgs 0))))
+
   (define transform
     (mlambda (pkg)
       (define rewrite
         (match-lambda
           ((name content . rest)
            (let* ((is-package? (package? content))
-                  (new-content (if is-package? (transform content) content)))
+                  (new-content (if is-package?
+                                   (or (find-input-package content)
+                                       (transform content))
+                                   content)))
              `(,name ,new-content ,@rest)))))
 
       ;; Special considerations for source packages: CL inputs become
-- 
2.23.0
--8<---------------cut here---------------end--------------->8---

I was not sure if using the '(gnu packages)' module here (for the
'find-packages-by-name' function) would cause a circular dependency or
not, but apparently it works.

What do you think about this patch? Should I submit it to guix-patches?

Re: Question about sbcl-package->ecl-package

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 116 bytes --]

Cool!  Thanks for working on this! :)

Does it work for dexador?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Question about sbcl-package->ecl-package

[Guillaume Le Vaillant]

Pierre Neidhardt skribis:

> Cool!  Thanks for working on this! :)
>
> Does it work for dexador?

I just tried compiling ecl-dexador, and it failed. However I think it
fails for a different reason.

The error is:

--8<---------------cut here---------------start------------->8---
An error occurred during initialization:
No MIME.TYPES file found anywhere!
--8<---------------cut here---------------end--------------->8---

I think it comes from the trivial-mimes library having a hard
reference to the 'sbcl-source' directory in a string:

--8<---------------cut here---------------start------------->8---
(arguments
 '(#:phases
   (modify-phases %standard-phases
     (add-after
         'unpack 'fix-paths
       (lambda* (#:key inputs #:allow-other-keys)
         (let ((anchor "#p\"/etc/mime.types\""))
           (substitute* "mime-types.lisp"
             ((anchor all)
              (string-append
               anchor "\n"
               "(asdf:system-relative-pathname :trivial-mimes \"../../share/common-lisp/sbcl-source/trivial-mimes/mime.types\")")))))))))
--8<---------------cut here---------------end--------------->8---

I guess in this case writing a 'fix-paths' phase with the path specific
to ECL in the ecl-trivial-mimes package will be necessary...

Re: Question about sbcl-package->ecl-package

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1536 bytes --]

Guillaume Le Vaillant <guillaume.le.vaillant@gmail.com> writes:

> I just tried compiling ecl-dexador, and it failed. However I think it
> fails for a different reason.
>
> The error is:
>
> --8<---------------cut here---------------start------------->8---
> An error occurred during initialization:
> No MIME.TYPES file found anywhere!
> --8<---------------cut here---------------end--------------->8---

I didn't have this error, mine was about a failing test, so I guess your
patch fixes it!

> I think it comes from the trivial-mimes library having a hard
> reference to the 'sbcl-source' directory in a string:
>
> --8<---------------cut here---------------start------------->8---
> (arguments
>  '(#:phases
>    (modify-phases %standard-phases
>      (add-after
>          'unpack 'fix-paths
>        (lambda* (#:key inputs #:allow-other-keys)
>          (let ((anchor "#p\"/etc/mime.types\""))
>            (substitute* "mime-types.lisp"
>              ((anchor all)
>               (string-append
>                anchor "\n"
>                "(asdf:system-relative-pathname :trivial-mimes \"../../share/common-lisp/sbcl-source/trivial-mimes/mime.types\")")))))))))
> --8<---------------cut here---------------end--------------->8---
>
> I guess in this case writing a 'fix-paths' phase with the path specific
> to ECL in the ecl-trivial-mimes package will be necessary...

Maybe an easier fix: replace "sbcl" with (%lisp-type).  Should work.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Question about sbcl-package->ecl-package

[Guillaume Le Vaillant]

Pierre Neidhardt skribis:

> Maybe an easier fix: replace "sbcl" with (%lisp-type).  Should work.

Indeed, with the following changes, building ecl-dexador works.

--8<---------------cut here---------------start------------->8---
diff --git a/gnu/packages/lisp.scm b/gnu/packages/lisp.scm
index 2bdebed04e..5f08acf7ef 100644
--- a/gnu/packages/lisp.scm
+++ b/gnu/packages/lisp.scm
@@ -5130,7 +5130,9 @@ performance and simplicity in mind.")
                    ((anchor all)
                     (string-append
                      anchor "\n"
-                     "(asdf:system-relative-pathname :trivial-mimes \"../../share/common-lisp/sbcl-source/trivial-mimes/mime.types\")")))))))))
+                     "(asdf:system-relative-pathname :trivial-mimes "
+                     "\"../../share/common-lisp/"
+                     (%lisp-type) "-source/trivial-mimes/mime.types\")")))))))))
       (native-inputs
        `(("stefil" ,sbcl-hu.dwim.stefil)))
       (inputs
@@ -5145,6 +5147,9 @@ mime-type of a file.")
 (define-public cl-trivial-mimes
   (sbcl-package->cl-source-package sbcl-trivial-mimes))
 
+(define-public ecl-trivial-mimes
+  (sbcl-package->ecl-package sbcl-trivial-mimes))
+
 (define-public sbcl-lack-middleware-static
   (let ((commit "abff8efeb0c3a848e6bb0022f2b8b7fa3a1bc88b")
         (revision "1"))
@@ -6251,6 +6256,9 @@ cookie headers, cookie creation, cookie jar creation and more.")
 neat APIs and connection-pooling.  It is meant to supersede Drakma.")
       (license license:expat))))
 
+(define-public ecl-dexador
+  (sbcl-package->ecl-package sbcl-dexador))
+
 (define-public sbcl-lisp-namespace
   (let ((commit "28107cafe34e4c1c67490fde60c7f92dc610b2e0")
         (revision "1"))
--8<---------------cut here---------------end--------------->8---

Re: Question about sbcl-package->ecl-package

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 125 bytes --]

Great! :)
Can you send a patch for all this?  I'll merge as soon as I can.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Question about sbcl-package->ecl-package

[Guillaume Le Vaillant]

Pierre Neidhardt skribis:

> Great! :)
> Can you send a patch for all this?  I'll merge as soon as I can.

I sent the patches (bug#37791).

Re: Question about sbcl-package->ecl-package

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 57 bytes --]

Thanks!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Question about sbcl-package->ecl-package

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 177 bytes --]

I've merged your last 3 patches, thank you so much for your continuous
contribution to the best Common Lisp package manager ;)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Question about sbcl-package->ecl-package

[Guillaume Le Vaillant]

Pierre Neidhardt skribis:

> I've merged your last 3 patches, thank you so much for your continuous
> contribution to the best Common Lisp package manager ;)

Thanks!

Re: Question about sbcl-package->ecl-package

[Guillaume Le Vaillant]

Pierre Neidhardt skribis:

> Cool!  Thanks for working on this! :)
>
> Does it work for dexador?

I just tried compiling ecl-dexador, and it failed. However I think it
fails for a different reason.

The error is:

--8<---------------cut here---------------start------------->8---
An error occurred during initialization:
No MIME.TYPES file found anywhere!
--8<---------------cut here---------------end--------------->8---

I think it comes from the trivial-mimes library having a hard
reference to the 'sbcl-source' directory in a string:

--8<---------------cut here---------------start------------->8---
(arguments
 '(#:phases
   (modify-phases %standard-phases
     (add-after
         'unpack 'fix-paths
       (lambda* (#:key inputs #:allow-other-keys)
         (let ((anchor "#p\"/etc/mime.types\""))
           (substitute* "mime-types.lisp"
             ((anchor all)
              (string-append
               anchor "\n"
               "(asdf:system-relative-pathname :trivial-mimes \"../../share/common-lisp/sbcl-source/trivial-mimes/mime.types\")")))))))))
--8<---------------cut here---------------end--------------->8---

I guess in this case writing a 'fix-paths' phase with the path specific
to ECL in the ecl-trivial-mimes package will be necessary...