Looking for help with packaging a Common Lisp library

Looking for help with packaging a Common Lisp library

[Konrad Hinsen]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

Hi Guix,

I hope that some Common Lisp expert here can help me with a strange
error message.

I am trying to package the library hdf5-cffi, my current package
definition is attached. It builds without any error message, but
when I try to load the library with

  (asdf:load-system "hdf5-cffi")

I get the error message

 The name "CFFI-GROVEL" does not designate any package.

when loading

  /gnu/store/3lcw9qvx76770wpsbgdvma154y84v54i-sbcl-hdf5-cffi-1.8.18/lib/sbcl/hdf5-cffi--system.fasl

It doesn't make a difference if I put sbcl-cffi-grovel under inputs or
propagated-inputs.

Does anyone have an idea what could be going wrong here?

Cheers,
  Konrad.


[-- Attachment #2: cl-hdf5-cffi.scm --]
[-- Type: application/octet-stream, Size: 2293 bytes --]

(use-modules (gnu packages lisp-xyz)
             (gnu packages maths)
	     ((guix licenses) #:prefix license:)
	     (guix packages)
	     (guix download)
             (guix git-download)
	     (guix build-system asdf)
             (guix utils))

(define-public sbcl-hdf5-cffi
  (let ((commit "5b5c88f191e470e4fe96b462334e3ce0806eed5c")
        (revision "1"))
    (package
      (name "sbcl-hdf5-cffi")
      (version "1.8.18")
      (source
       (origin
         (method git-fetch)
         (uri (git-reference
               (url "https://github.com/hdfgroup/hdf5-cffi.git")
               (commit commit)))
         (file-name (git-file-name name version))
         (sha256
          (base32
           "0vda3075423xz83qky998lpac5b04dwfv7bwgh9jq8cs5v0zrxjf"))))
      (build-system asdf-build-system/sbcl)
      (synopsis "Common Lisp bindings for the HDF5 library")
      (description
       "hdf5-cffi is a CFFI wrapper for the HDF5 library.")
      (home-page "https://github.com/hdfgroup/hdf5-cffi")
      (license (license:non-copyleft
                (string-append "https://github.com/HDFGroup/hdf5-cffi/raw/"
                               commit
                               "/LICENSE")))
      (inputs
       `(("cffi" ,sbcl-cffi)
         ("hdf5" ,hdf5-1.10)))
      ;; This shouldn't need to be a propagated input.
      (propagated-inputs
       `(("cffi-grovel" ,sbcl-cffi-grovel)))
      (native-inputs
       `(("fiveam" ,sbcl-fiveam)))
      (arguments
       `(#:asd-system-name "hdf5-cffi"
         #:asd-file "hdf5-cffi.asd"
         #:test-asd-file "hdf5-cffi.test.asd"
         ;; Tests require loading hdf5-cffi.examples.asd as well,
         ;; I don't know if that is possible with asdf-build-system.
         #:tests? #f
         #:phases
         (modify-phases %standard-phases
           (add-after 'unpack 'fix-paths
             (lambda* (#:key inputs #:allow-other-keys)
               (substitute* "src/library.lisp"
                 (("libhdf5.so")
                  (string-append
                   (assoc-ref inputs "hdf5")
                   "/lib/libhdf5.so")))))))))))


(define-public cl-hdf5-cffi
  (sbcl-package->cl-source-package sbcl-hdf5-cffi))

(define-public ecl-hdf5-cffi
  (sbcl-package->ecl-package sbcl-hdf5-cffi))

sbcl-hdf5-cffi

Re: Looking for help with packaging a Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]

It seems unrelated, but first there seems to be an issue during the
build regarding the missing pkg-config package.
If you add it to the native-inputs, then you'll get the following error:

--8<---------------cut here---------------start------------->8---
; ERROR: Subprocess #<UIOP/LAUNCH-PROGRAM::PROCESS-INFO {1004FACA93}>
 with command ("pkg-config" "hdf5" "--cflags")
--8<---------------cut here---------------end--------------->8---

This is because our hdf5 package has no pkg-config file.  We should
probably fix this first.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Looking for help with packaging a Common Lisp library

[Guillaume Le Vaillant]

[-- Attachment #1: Type: text/plain, Size: 600 bytes --]


I think the generated '...-sbcl-hdf5-cffi-1.8.18/lib/sbcl/hdf5-cffi.asd'
file is missing a dependency. It contains ':depends-on ("cffi")', but
I think it should be ':depends-on ("cffi" "cffi-grovel")', because the
original asd file has:

--8<---------------cut here---------------start------------->8---
:defsystem-depends-on (:cffi-grovel)
:depends-on (:cffi)
--8<---------------cut here---------------end--------------->8---

Maybe the 'defsystem-depends-on' part is currently ignored by
asdf-build-system...

This would explain why cffi-grovel is not loaded automatically when
loading hdf5-cffi.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Looking for help with packaging a Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 116 bytes --]

You can look at static-vectors for reference, it is quite similar.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Looking for help with packaging a Common Lisp library

[Guillaume Le Vaillant]

[-- Attachment #1: Type: text/plain, Size: 1895 bytes --]


We could add all the dependencies declared in the original asd files (by
'defsystem-depends-on' and 'depends-on') to the generated asd files with
the following patch:

--8<---------------cut here---------------start------------->8---
From 7e2009177137b3ad5831b5c42d9a4387d408bc57 Mon Sep 17 00:00:00 2001
From: Guillaume Le Vaillant <glv@posteo.net>
Date: Sat, 25 Jan 2020 18:07:37 +0100
Subject: [PATCH] build: lisp-utils: Take  defsystem dependencies into
 consideration.

* guix/build/lisp-utils.scm (system-dependencies): Add the dependencies
  declared with ':defsystem-depends-on' in asd-file to the list of
  dependencies.
---
 guix/build/lisp-utils.scm | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/guix/build/lisp-utils.scm b/guix/build/lisp-utils.scm
index c7a589c902..cce8047d61 100644
--- a/guix/build/lisp-utils.scm
+++ b/guix/build/lisp-utils.scm
@@ -165,10 +165,11 @@ asdf:system-depends-on.  First load the system's ASD-FILE."
       (asdf:load-asd (truename ,asd-file) :name ,(normalize-string system))
       (with-open-file
        (stream ,deps-file :direction :output)
-       (format stream
-               "~s~%"
-               (asdf:system-depends-on
-                (asdf:find-system ,system))))))
+       (let* ((sys (asdf:find-system ,system))
+              (deps (union (asdf:system-defsystem-depends-on sys)
+                           (asdf:system-depends-on sys)
+                           :test (function equal))))
+         (format stream "~s~%" deps)))))
 
   (dynamic-wind
     (lambda _
-- 
2.25.0
--8<---------------cut here---------------end--------------->8---

With this patch, we might need to add some inputs to a few Lisp
packages, but probably not a lot of them.

Do you think this patch would be a good idea, or do you see some cases
where it could break something?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Looking for help with packaging a Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 191 bytes --]

I'm not sure: isn't defsystem-depends-on specific to the .asd?
If the Common Lisp package depends on it, it should be in the depends, right?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Looking for help with packaging a Common Lisp library

[Konrad Hinsen]

Hi Pierre and Guillaume,

thanks for your quick help!

Guillaume Le Vaillant <glv@posteo.net> writes:

> I think the generated '...-sbcl-hdf5-cffi-1.8.18/lib/sbcl/hdf5-cffi.asd'
> file is missing a dependency. It contains ':depends-on ("cffi")', but
> I think it should be ':depends-on ("cffi" "cffi-grovel")', because the

That was the problem indeed. I patched hdf5-cffi.asd to add :cffi-grovel
to the dependencies, and with that patch everything works fine.

> We could add all the dependencies declared in the original asd files (by
> 'defsystem-depends-on' and 'depends-on') to the generated asd files with
> the following patch:

I don't know ASDF and the ASDF build system well enough to comment on
that. All I can say is that in a "standard" SBCL + QuickLisp
environment, it is not necessary to patch  hdf5-cffi.asd, so it seems
to be correct as far as I can tell, meaning that it would be nice if it
worked out of the box as well in Guix.


Pierre Neidhardt <mail@ambrevar.xyz> writes:

> It seems unrelated, but first there seems to be an issue during the
> build regarding the missing pkg-config package.

I am so used to pkg-config related errors that I hardly see them any
more. So yes, this ought to be fixed, but it's not related to the
problem I had.


Thanks again,
  Konrad.

Re: Looking for help with packaging a Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1417 bytes --]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> Hi Pierre and Guillaume,
>
> thanks for your quick help!
>
> Guillaume Le Vaillant <glv@posteo.net> writes:
>
>> I think the generated '...-sbcl-hdf5-cffi-1.8.18/lib/sbcl/hdf5-cffi.asd'
>> file is missing a dependency. It contains ':depends-on ("cffi")', but
>> I think it should be ':depends-on ("cffi" "cffi-grovel")', because the
>
> That was the problem indeed. I patched hdf5-cffi.asd to add :cffi-grovel
> to the dependencies, and with that patch everything works fine.
>
>> We could add all the dependencies declared in the original asd files (by
>> 'defsystem-depends-on' and 'depends-on') to the generated asd files with
>> the following patch:
>
> I don't know ASDF and the ASDF build system well enough to comment on
> that. All I can say is that in a "standard" SBCL + QuickLisp
> environment, it is not necessary to patch  hdf5-cffi.asd, so it seems
> to be correct as far as I can tell, meaning that it would be nice if it
> worked out of the box as well in Guix.

The Guix package recreates the .asd file (without the
defsystem-depends-on), which is why it worked in Quicklisp but not in
Guix.

We could merge Guillaume's patch, but first I'd like to make sure that
this is not an upstream bug.  Isn't cffi-grovel supposed to be added to
the :depends-on list instead?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Looking for help with packaging a Common Lisp library

[Konrad Hinsen]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> The Guix package recreates the .asd file (without the
> defsystem-depends-on), which is why it worked in Quicklisp but not in
> Guix.

OK, that makes sense.

> We could merge Guillaume's patch, but first I'd like to make sure that
> this is not an upstream bug.  Isn't cffi-grovel supposed to be added to
> the :depends-on list instead?

I am certainly not the right person to answer that question. But here's
what the ASDF manual says:

   The :defsystem-depends-on option to defsystem allows the programmer
   to specify another ASDF-defined system or set of systems that must be
   loaded before the system definition is processed. Typically this is
   used to load an ASDF extension that is used in the system definition.

This looks like what the cffi-grovel manual suggests, with an example:

   https://common-lisp.net/project/cffi/manual/html_node/Groveller-ASDF-Integration.html#Groveller-ASDF-Integration

In hdf5-cffi.asd, some components are labelled :cffi-grovel-file,
which looks like an ASDF extension provided by cffi-grovel. If that is
true, then the :defsystem-depends-on is necessary. A quick test supports
that hypothesis: if you remove the line, the system fails to load.

Cheers,
  Konrad.

Re: Looking for help with packaging a Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 600 bytes --]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> In hdf5-cffi.asd, some components are labelled :cffi-grovel-file,
> which looks like an ASDF extension provided by cffi-grovel. If that is
> true, then the :defsystem-depends-on is necessary. A quick test supports
> that hypothesis: if you remove the line, the system fails to load.

Indeed.  What I meant is that if the Common Lisp package (the code inside the src
folder) depends on cffi-grovel, then I believe it should be added to
:depends-on as well.  We can ask ASDF and hdf5-cffi.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Looking for help with packaging a Common Lisp library

[Konrad Hinsen]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Indeed.  What I meant is that if the Common Lisp package (the code
> inside the src folder) depends on cffi-grovel, then I believe it
> should be added to :depends-on as well.  We can ask ASDF and
> hdf5-cffi.

I'd be happy to ask hdf5-cffi, but I excpect their answer to be
"we just copied the example from the cffi-grovel manual", which
indeed they did. So maybe cffi-grovel is the better place to ask.
Although ultimately it's an ASDF question in my opinion.
And ASDF looks seriously underdocumented compared to the complexity of
all it does.

Cheers,
  Konrad.

Re: Looking for help with packaging a Common Lisp library

[Guillaume Le Vaillant]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]


Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

> Pierre Neidhardt <mail@ambrevar.xyz> writes:
>
>> Indeed.  What I meant is that if the Common Lisp package (the code
>> inside the src folder) depends on cffi-grovel, then I believe it
>> should be added to :depends-on as well.  We can ask ASDF and
>> hdf5-cffi.
>
> I'd be happy to ask hdf5-cffi, but I excpect their answer to be
> "we just copied the example from the cffi-grovel manual", which
> indeed they did. So maybe cffi-grovel is the better place to ask.
> Although ultimately it's an ASDF question in my opinion.
> And ASDF looks seriously underdocumented compared to the complexity of
> all it does.
>
> Cheers,
>   Konrad.

I opened an issue about ASDF bundles at
https://gitlab.common-lisp.net/asdf/asdf/issues/23 where I also asked
whether the dependencies declared in 'defsystem-depends-on' are expected
to be loaded along the other dependencies even after compiling a bundle.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Looking for help with packaging a Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 74 bytes --]

Excellent, thanks a lot!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Looking for help with packaging a Common Lisp library

[Guillaume Le Vaillant]

[-- Attachment #1: Type: text/plain, Size: 292 bytes --]


A maintainer of ASDF answered that only the dependencies declared in
'depends-on' should be put in the generated asd file for the bundle.

Therefore my patch is not necessary, and concerning hdf5-cffi, the
cffi-grovel system should be listed in both 'defsystem-depends-on'
and 'depends-on'.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Looking for help with packaging a Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 129 bytes --]

Great, I've let upstream know:
https://github.com/HDFGroup/hdf5-cffi/issues/70

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Looking for help with packaging a Common Lisp library

[Konrad Hinsen]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Great, I've let upstream know:
> https://github.com/HDFGroup/hdf5-cffi/issues/70

And I just submitted a fix as a pull request:

  https://github.com/HDFGroup/hdf5-cffi/issues/71

Thanks to both of you for getting this sorted out quickly! If upstream
doesn't merge my patch in the next days, I'll submit my current Guix package
which does the patch itself.

Cheers,
  Konrad.