bug#32770: Packaging SLIME/SWANK as Common Lisp library

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

On the long road to packaging Next browser (bug #32729), I've hit a big
obstacle: packaging the SLIME/SWANK dependency.

We currently have an emacs-slime package, but that cannot be used as a
Common Lisp dependency.

See those conversations:

	https://lists.gnu.org/archive/html/guix-devel/2018-08/msg00138.html

and

	https://github.com/slime/slime/issues/457

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Andy Patterson]

Hey Pierre,

Sorry for taking so long to get back to you.

On Wed, 19 Sep 2018 09:19:46 +0200
Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> On the long road to packaging Next browser (bug #32729), I've hit a
> big obstacle: packaging the SLIME/SWANK dependency.
> 
> We currently have an emacs-slime package, but that cannot be used as a
> Common Lisp dependency.
> 

I've managed to figure out why that is.  With the branch you're using
for the package, the asd file defines the package and variables that
get used - but in guix that asd file gets discarded.  I think the right
way is to have those definitions defined in a separate file, and
actually they are defined in one as well.  The following patch adds
that file to the asd and makes sure it compiles successfully.

I've noticed that the pull request that branch was made for is closed
though - I'll try to see if I can get things building with the other
one that's open and maybe co-ordinate with upstream to get that in.

For now though, maybe this will unblock you.  The patch was made after
rebasing your branch unto master.  I was also able to drop the commit
named "build-system/asdf: Fix normalize-string to support versioned
dependencies", since there's a more general version of it in master
now.

--
Andy

From a7b945bb58f22ac6c0bcee1bee62e40ffa9873a6 Mon Sep 17 00:00:00 2001
From: Andy Patterson <ajpatter@uwaterloo.ca>
Date: Sat, 29 Sep 2018 15:04:40 -0400
Subject: [PATCH] gnu: slime-swank: Make it loadable.

* gnu/packages/lisp.scm (sbcl-slime-swank)[origin]<modules>: Add
(guix build utils).
<snippet>: Patch the asd file to load swank-loader-asdf.
---
 gnu/packages/lisp.scm | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/gnu/packages/lisp.scm b/gnu/packages/lisp.scm
index aa146c9a5..a05d95f60 100644
--- a/gnu/packages/lisp.scm
+++ b/gnu/packages/lisp.scm
@@ -2223,12 +2223,18 @@ writing code that contains string literals that
contain code themselves.") (base32
          ;; "065bc4y6iskazdfwlhgcjlzg9bi2hyjbhmyjw3461506pgkj08vi"
          "0pkmg94wn4ii1zhlrncn44mdc5i6c5v0i9gbldx4dwl2yy7ibz5c"))
+       (modules '((guix build utils)))
        (snippet
         '(begin
            (substitute* "contrib/swank-listener-hooks.lisp"
              ((":compile-toplevel :load-toplevel ") ""))
            (substitute* "contrib/swank-presentations.lisp"
              ((":compile-toplevel :load-toplevel ") ""))
+           (substitute* "swank.asd"
+             ((":file \"packages\".*" all)
+              (string-append all "(:file \"swank-loader-asdf\")\n")))
+           (substitute* "swank-loader-asdf.lisp"
+             ((":common-lisp" all) (string-append all " #:asdf")))
            #t))))
     (build-system asdf-build-system/sbcl)
     (arguments
-- 
2.19.0

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 395 bytes --]

Andy, you are the hero of the day!  Thank you so much!

Thinking about it, that explains why Quicklisp does not have a problem with it:
it always loads the .asd file I suppose.  (I haven't checked the code.)

I'll keep on packaging Next, hopefully there won't be such an obstacle again and
we will see it land in Guix very soon!

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

Also see

	https://github.com/quicklisp/quicklisp-projects/issues/1561

There are some interesting links, mostly about the Nix system.  See the

	https://github.com/NixOS/nixpkgs/blob/5048b0d1f9e908665a01c35f49b2d9816128e089/pkgs/development/lisp-modules/quicklisp-to-nix-output/swank.nix

file: I think it would have given us a clear hint with ~asdFilesToKeep =
["swank.asd"];~, had we looked at Nix before.

I haven't looked to closely into what else Nix is doing to package Common Lisp,
but it seems rather complete.  Any good idea we could borrow, Andy?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1474 bytes --]

Another issue, Andy, is with the new `normalize-dependency' function that adds
support for the dependency syntax as specified by the ASDF grammar.  I think
your approach is good but it fails for `jpl-queues` because it produces the
following .asd file in the store:

--8<---------------cut here---------------start------------->8---
(asdf:defsystem
  "jpl-queues"
  :class
  asdf/bundle:prebuilt-system
  :version
  "0.1"
  :depends-on
  ("bordeaux-threads" (:version "jpl-util" "0.2"))
  :components
  ((:compiled-file "jpl-queues--system")))

(progn (asdf/source-registry:ensure-source-registry)
       (setf (gethash
               "bordeaux-threads"
               asdf/source-registry:*source-registry*)
             #p
             "/gnu/store/67dhvxva7345r2xijz9yqp94s0fslrpn-sbcl-bordeaux-threads-0.8.5-1.354abb0/share/common-lisp/sbcl-bundle-systems/bordeaux-threads.asd"))
--8<---------------cut here---------------end--------------->8---

Note that jpl-util is missing from the registry link list.

It turns out that it fails because the match-lambda you've used to implement the
standard are case-sensitive, while the standard is case-insensitive.

Adn... jpl-queues uses ":VERSION" and not ":version" as you expected :(

I think the simple fix would be to make the two match-lambda case-insensitive
(both in generate-dependency-links and in make-asd-file).

What do you think?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Andy Patterson]

Hi,

On Mon, 01 Oct 2018 15:53:50 +0200
Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> Another issue, Andy, is with the new `normalize-dependency' function
> that adds support for the dependency syntax as specified by the ASDF
> grammar.  I think your approach is good but it fails for `jpl-queues`
> because it produces the following .asd file in the store:
>
> [...]
> 
> Note that jpl-util is missing from the registry link list.
> 
> It turns out that it fails because the match-lambda you've used to
> implement the standard are case-sensitive, while the standard is
> case-insensitive.

Right, good catch.

> 
> Adn... jpl-queues uses ":VERSION" and not ":version" as you
> expected :(
> 
> I think the simple fix would be to make the two match-lambda
> case-insensitive (both in generate-dependency-links and in
> make-asd-file).

We're always retrieving that list from a lisp implementation, so I
think it's safe to assume that those symbols will be up-cased in the
output that it sends us.  I'll send a patch with that change.

Thanks,

--
Andy

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Andy Patterson]

Hi,

On Sun, 30 Sep 2018 23:46:08 +0200
Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> Also see
> 
> 	https://github.com/quicklisp/quicklisp-projects/issues/1561
> 
> There are some interesting links, mostly about the Nix system.  See
> the
> 
> 	https://github.com/NixOS/nixpkgs/blob/5048b0d1f9e908665a01c35f49b2d9816128e089/pkgs/development/lisp-modules/quicklisp-to-nix-output/swank.nix
> 

Having taken just a quick look at what's there, it seems that the
approach is fairly different from what we're doing.  They're using the
default asdf build target, and I think they must be relying on output
translations having been set up right to avoid re-compilation.

I'm trying to get a Nix OS VM set up to play around with what's there.

> file: I think it would have given us a clear hint with
> ~asdFilesToKeep = ["swank.asd"];~, had we looked at Nix before.
> 

It looks like they're doing that in all of the packages.  In general,
we shouldn't expect to have to bring definitions from upstream's asd
file into our own.  Asdf's best practices documentation warns against
it.  Also, they're using upstream slime, unlike your package, so I
don't think anything is being compiled at all.

The difference here between Nix's system and ours
is that they're using the upstream asd file to load the system, whereas
we generate our own just for the build's output.  This allows us to
load the compiled file (or files) directly, without having to rely on
mapping source files to compiled files.

> I haven't looked to closely into what else Nix is doing to package
> Common Lisp, but it seems rather complete.  Any good idea we could
> borrow, Andy?

It'd be nice to have an importer.  Maybe we can use Nix's work to
reduce the amount of investigation needed to get there.

--
Andy

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]


> It'd be nice to have an importer.  Maybe we can use Nix's work to
> reduce the amount of investigation needed to get there.

Absolutely.  ASDF is good enough that the vast majority of CL libraries can be
packaged without an itch.

If we could use Quicklisp or

  http://quickdocs.org/
	
to write a recursive importer, we could get programs like Next browser packaged
in one go but for a couple of tweaks for the 2-3 odd packages.

Let me know if you want to work on this, otherwise I'll give it a shot.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Andy Patterson]

Hi Pierre,

On Tue, 02 Oct 2018 12:24:29 +0200
Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> > It'd be nice to have an importer.  Maybe we can use Nix's work to
> > reduce the amount of investigation needed to get there.  
> 
> Absolutely.  ASDF is good enough that the vast majority of CL
> libraries can be packaged without an itch.
> 
> If we could use Quicklisp or
> 
>   http://quickdocs.org/
> 	
> to write a recursive importer, we could get programs like Next
> browser packaged in one go but for a couple of tweaks for the 2-3 odd
> packages.
> 
> Let me know if you want to work on this, otherwise I'll give it a
> shot.
> 

I'd like to be involved as I have some ideas on things we could do
differently from Nix that I'd begun investigating.  I'll elaborate more
once I get the chance to organize my thoughts.

Thanks,

--
Andy

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1.1: Type: text/plain, Size: 951 bytes --]

Last (hopefully?) step before getting Next up and running: add support for CFFI.

I've re-used your CFFI package and they seem to work.  Now if I want to package
cl-sqlite, it fails because sqlite.so cannot be found.  Indeed, the patch to
/gnu/store/...-sqlite.../lib must be added to the
cffi:*foreign-library-directories* global variable before compiling.

If I do that from quicklisp, it works.  Now I guess we need to tweak
asdf-build-system so that packages can specify which paths to add to that global
variable.

Makes sense?

I've given it a quick shot (patch attached) but then every ASDF build fails with

--8<---------------cut here---------------start------------->8---
ice-9/boot-9.scm:222:17: In procedure map1:
Syntax error:
unknown location: unexpected syntax in form ()
--8<---------------cut here---------------end--------------->8---

I'm still lacking good knowledge of the asdf build system.

I'll give it a closer look tomorrow.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-cffi-support.patch --]
[-- Type: text/x-patch, Size: 3212 bytes --]

From 1c4b9b915086b0fdfb4229f5bbfa5aecff784f42 Mon Sep 17 00:00:00 2001
From: Pierre Neidhardt <mail@ambrevar.xyz>
Date: Wed, 3 Oct 2018 20:27:00 +0200
Subject: [PATCH] cffi support

---
 guix/build-system/asdf.scm       | 2 ++
 guix/build/asdf-build-system.scm | 4 ++--
 guix/build/lisp-utils.scm        | 5 ++++-
 3 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/guix/build-system/asdf.scm b/guix/build-system/asdf.scm
index 57e294d74..37c219460 100644
--- a/guix/build-system/asdf.scm
+++ b/guix/build-system/asdf.scm
@@ -266,6 +266,7 @@ set up using CL source package conventions."
                   (tests? #t)
                   (asd-file #f)
                   (asd-system-name #f)
+                  (cffi-foreign-library-directories '())
                   (test-asd-file #f)
                   (phases '(@ (guix build asdf-build-system)
                               %standard-phases))
@@ -298,6 +299,7 @@ set up using CL source package conventions."
                        #:asd-file ,(or asd-file (string-append system-name ".asd"))
                        #:asd-system-name ,system-name
                        #:test-asd-file ,test-asd-file
+                       #:cffi-foreign-library-directories ,cffi-foreign-library-directories
                        #:system ,system
                        #:tests? ,tests?
                        #:phases ,phases
diff --git a/guix/build/asdf-build-system.scm b/guix/build/asdf-build-system.scm
index dd6373b33..7d879ddb9 100644
--- a/guix/build/asdf-build-system.scm
+++ b/guix/build/asdf-build-system.scm
@@ -106,7 +106,7 @@ valid."
       (delete-file-recursively "systems")))
   #t)
 
-(define* (build #:key outputs inputs asd-file asd-system-name
+(define* (build #:key outputs inputs asd-file asd-system-name cffi-foreign-library-directories
                 #:allow-other-keys)
   "Compile the system."
   (let* ((out (library-output outputs))
@@ -121,7 +121,7 @@ valid."
 
     (setenv "HOME" out) ; ecl's asdf sometimes wants to create $HOME/.cache
 
-    (compile-system asd-system-name asd-file)
+    (compile-system asd-system-name asd-file cffi-foreign-library-directories)
 
     ;; As above, ecl will sometimes create this even though it doesn't use it
 
diff --git a/guix/build/lisp-utils.scm b/guix/build/lisp-utils.scm
index 5dea30486..5d7b5afd0 100644
--- a/guix/build/lisp-utils.scm
+++ b/guix/build/lisp-utils.scm
@@ -149,13 +149,16 @@ with PROGRAM."
          `(asdf:load-system ,system))
        systems))
 
-(define (compile-system system asd-file)
+(define* (compile-system system asd-file #:optional cffi-foreign-library-directories)
   "Use a lisp implementation to compile SYSTEM using asdf.  Load ASD-FILE
 first."
   (lisp-eval-program
    `((require :asdf)
      (let ((*package* (find-package :asdf)))
        (load ,asd-file))
+     ,(unless (null? cffi-foreign-library-directories)
+        `(progn (require :cffi)
+                (setq cffi:*foreign-library-directories* ,cffi-foreign-library-directories)))
      (asdf:operate 'asdf:compile-bundle-op ,system))))
 
 (define (system-dependencies system asd-file)
-- 
2.19.0


[-- Attachment #1.3: Type: text/plain, Size: 48 bytes --]


-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 117 bytes --]

Hmmm, I should have posted this on Next's bug, not this one, sorry.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Andy Patterson]

Hi,

On Wed, 03 Oct 2018 20:34:24 +0200
Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> Last (hopefully?) step before getting Next up and running: add
> support for CFFI.
> 
> I've re-used your CFFI package and they seem to work.  Now if I want
> to package cl-sqlite, it fails because sqlite.so cannot be found.
> Indeed, the patch to /gnu/store/...-sqlite.../lib must be added to the
> cffi:*foreign-library-directories* global variable before compiling.
> 
> If I do that from quicklisp, it works.  Now I guess we need to tweak
> asdf-build-system so that packages can specify which paths to add to
> that global variable.

I've been using a cffi consumer for a while now, and my approach for
that package was to update references to library and header files with
the full paths to their respective store items.

Adding to *foreign-library-directories* at compile time may allow a
package to compile, but I get the feeling that it may not have the
desired effect when you try to load the package later on.

I guess another approach we could take if replacement is undesirable
would be to modify the cffi package to honour some environment variable
for its default, and then add that variable to the search path for
cffi. Maybe such a variable already exists, even. I think this is the
approach that Nix takes.

I personally tend to favour replacement, but others could chime in here
as well since this problem isn't specific to lisp packaging.

> 
> Makes sense?
> 
> I've given it a quick shot (patch attached) but then every ASDF build
> fails with
> 
> --8<---------------cut here---------------start------------->8---
> ice-9/boot-9.scm:222:17: In procedure map1:
> Syntax error:
> unknown location: unexpected syntax in form ()
> --8<---------------cut here---------------end--------------->8---
> 

It's confusing, but you either need to double quote the default
argument (e.g. (cffi-foreign-library-directories ''())), or re-quote
the unquoted variable when you pass it along, depending on how you want
users to specify the argument in the package (quoted for the former,
unquoted for the latter).

Hoping that helps,

--
Andy

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1574 bytes --]

CC-ing Ludovic and Ricardo if they want to chime in.

Thanks for the feedback, Andy, so invaluable as always.

> I've been using a cffi consumer for a while now, and my approach for
> that package was to update references to library and header files with
> the full paths to their respective store items.

Cool!  Can you share this package?  Or provide an example?
Can't wait to get this up and running!

> Adding to *foreign-library-directories* at compile time may allow a
> package to compile, but I get the feeling that it may not have the
> desired effect when you try to load the package later on.

Why?  In my experience it works for a compiled Next executable.

> I guess another approach we could take if replacement is undesirable
> would be to modify the cffi package to honour some environment variable
> for its default, and then add that variable to the search path for
> cffi.

The cffi:*foreign-library-directories* approach got recommended to me on
the CFFI mailing list.  I find it similar to what you want, a sort of
environment variable (a search-path global variable).  I haven't looked
into Nix.

> I personally tend to favour replacement, but others could chime in here
> as well since this problem isn't specific to lisp packaging.

Could this be automated?  What I like with cffi:*foreign-library-directories* is
that we could automatically push

    (string-append (assoc-ref inputs "foo") "/lib)

to it so that packages only have to write something like

    #:cffi-packages (list ,cl-sqlite ,cl-foo...)

--
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Ricardo Wurmus]

Hi,

>> Adding to *foreign-library-directories* at compile time may allow a
>> package to compile, but I get the feeling that it may not have the
>> desired effect when you try to load the package later on.
>
> Why?  In my experience it works for a compiled Next executable.
>
>> I guess another approach we could take if replacement is undesirable
>> would be to modify the cffi package to honour some environment variable
>> for its default, and then add that variable to the search path for
>> cffi.
>
> The cffi:*foreign-library-directories* approach got recommended to me on
> the CFFI mailing list.  I find it similar to what you want, a sort of
> environment variable (a search-path global variable).  I haven't looked
> into Nix.
>
>> I personally tend to favour replacement, but others could chime in here
>> as well since this problem isn't specific to lisp packaging.

We usually replace plain references to libraries with exact store paths.

> Could this be automated?  What I like with cffi:*foreign-library-directories* is
> that we could automatically push
>
>     (string-append (assoc-ref inputs "foo") "/lib)
>
> to it so that packages only have to write something like
>
>     #:cffi-packages (list ,cl-sqlite ,cl-foo...)

This sounds useful, but to me it feels like wrapping executables in
LD_LIBRARY_PATH, which we try hard to avoid.

When using cffi:*foreign-library-directories* how would we deal with
cases where we have more than one package specifying cffi-packages?
Would this option need to be considered for every package in the package
closure?  Would it need to be handled by a profile hook in case these
packages are installed?

--
Ricardo

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]


> This sounds useful, but to me it feels like wrapping executables in
> LD_LIBRARY_PATH, which we try hard to avoid.
> 
> When using cffi:*foreign-library-directories* how would we deal with
> cases where we have more than one package specifying cffi-packages?
> Would this option need to be considered for every package in the package
> closure?  Would it need to be handled by a profile hook in case these
> packages are installed?

Every new path is appended to the list, and CFFI looks for the first matching
library.  This means that there could be conflicts if two packages feature the
same library name (which could be the case when a library is built with
different flags for instance).

I think you are right, we should go with Andy's approach.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 102 bytes --]

Fixed with 2026204a13515aaa9b7814f9a6f0a60b65493bd1.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Andy Patterson]

Hi Pierre,

I've seen your later e-mail but I'd still like to answer some of your
questions here.

On Thu, 04 Oct 2018 10:37:08 +0200
Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> CC-ing Ludovic and Ricardo if they want to chime in.
> 
> Thanks for the feedback, Andy, so invaluable as always.
> 
> > I've been using a cffi consumer for a while now, and my approach for
> > that package was to update references to library and header files
> > with the full paths to their respective store items.  
> 
> Cool!  Can you share this package?  Or provide an example?
> Can't wait to get this up and running!

Sure, I'll append it.

> 
> > Adding to *foreign-library-directories* at compile time may allow a
> > package to compile, but I get the feeling that it may not have the
> > desired effect when you try to load the package later on.  
> 
> Why?  In my experience it works for a compiled Next executable.

I assume that you created this executable by saving the lisp image?  If
that's the case, then it probably saved your
*foreign-library-directories* with the values you added to it.  What
I'm considering is consumers of the libraries which should still work
if you load them from a fresh lisp session with no configuration.

I haven't looked into it yet but I don't think loading the library
would work in that case.

> 
> > I guess another approach we could take if replacement is undesirable
> > would be to modify the cffi package to honour some environment
> > variable for its default, and then add that variable to the search
> > path for cffi.  
> 
> The cffi:*foreign-library-directories* approach got recommended to me
> on the CFFI mailing list.  I find it similar to what you want, a sort
> of environment variable (a search-path global variable).  I haven't
> looked into Nix.
> 
> > I personally tend to favour replacement, but others could chime in
> > here as well since this problem isn't specific to lisp packaging.  
> 
> Could this be automated?  What I like with
> cffi:*foreign-library-directories* is that we could automatically push
> 
>     (string-append (assoc-ref inputs "foo") "/lib)
> 
> to it so that packages only have to write something like
> 
>     #:cffi-packages (list ,cl-sqlite ,cl-foo...)
> 

Based on what you've written here, I guess that you're thinking of
adding to the *foreign-library-directories* for the consumers of
cffi-based packages.  I thought you meant to do it for those packages
themselves.  I think that might require some packages to propagate the
c libraries that they depend on.

In terms of automating things, it might be possible to do something
with *foreign-library-directories* in the asd files that we generate.

To be honest though, I think the fact that replacement can't be
automated is more of a bug in the libraries' build tooling.  What I'm
thinking of is something like a configure script to set the library
paths before building is done.  I think there are some examples of
guile projects which do just that.  It might be something that we could
discuss with the lisp and asdf community.

--
Andy

---
I think this package might include some dependencies that aren't
upstreamed yet but it should provide the gist of it.

(define sbcl-cl-freetype2
  (let ((revision "1")
        (commit "5e4c9ff898f84c888f7e4bfb3402ee35abbf8ad9"))
    (package
      (name "sbcl-cl-freetype2")
      (version (string-append "1.1-" revision "." (string-take commit
7))) (source
       (origin
         (method git-fetch)
         (uri
          (git-reference
           (url "https://github.com/rpav/cl-freetype2.git")
           (commit commit)))
         (sha256
          (base32
"0xmpswm65g5qq85n9nl5blqij2hw4nd4nc143nxhc77w49nf613d")) (file-name
(string-append "cl-freetype2-next-" version "-checkout"))))
      (build-system asdf-build-system/sbcl)
      (inputs
       `(("freetype" ,freetype)
         ("sbcl-alexandria" ,sbcl-alexandria)
         ("sbcl-cffi" ,sbcl-cffi)
         ("sbcl-trivial-garbage" ,sbcl-trivial-garbage)))
      (native-inputs
       `(("tests:cl-fiveam" ,sbcl-fiveam)
         ("asdf:cl-cffi-grovel" ,sbcl-cffi-grovel)))
      (arguments
       '(#:phases
         (modify-phases %standard-phases
           (add-after 'unpack 'fix-paths
             (lambda* (#:key inputs #:allow-other-keys)
               (define freetype (assoc-ref inputs "freetype"))
               (substitute* "src/ffi/ft2-lib.lisp"
                 (("libfreetype" all) (string-append
                                       freetype "/lib/" all)))
               (substitute* "src/ffi/grovel/grovel-freetype2.lisp"
                 (("/usr/include") (string-append freetype
"/include")))))))) (home-page "https://github.com/rpav/cl-freetype2")
      (synopsis "Common Lisp bindings for Freetype 2")
      (description "This is a general Freetype 2 wrapper for Common Lisp
using CFFI.  It's geared toward both using Freetype directly by
providing a simplified API, as well as providing access to the
underlying C structures and functions for use with other libraries
which may also use Freetype.")
      (license license:bsd-3))))

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1492 bytes --]


> Sure, I'll append it.

Great, thanks!

> What
> I'm considering is consumers of the libraries which should still work
> if you load them from a fresh lisp session with no configuration.
>
> I haven't looked into it yet but I don't think loading the library
> would work in that case.

Are you saying that 'compile-bundle' would discard the global variable?

I'm still rather new to Common Lisp so let me know if I get it right:

- Compiling a bundle only compiles the target files, it discards the other
  global values.

- Saving an image keeps everything, including global variables.

> >     #:cffi-packages (list ,cl-sqlite ,cl-foo...)
> >
>
> Based on what you've written here, I guess that you're thinking of
> adding to the *foreign-library-directories* for the consumers of
> cffi-based packages.  I thought you meant to do it for those packages
> themselves.

Sorry, you understood right at first, I wanted to write

--8<---------------cut here---------------start------------->8---
  (define-public sbcl-cl-sqlite
    #:cffi-packages (list ,sqlite ,foo...)
--8<---------------cut here---------------end--------------->8---

> What I'm
> thinking of is something like a configure script to set the library
> paths before building is done.  I think there are some examples of
> guile projects which do just that.

Hmm, I'm not sure I understand what you mean.  Could you provide an example?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Ricardo Wurmus]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

>> What I'm
>> thinking of is something like a configure script to set the library
>> paths before building is done.  I think there are some examples of
>> guile projects which do just that.
>
> Hmm, I'm not sure I understand what you mean.  Could you provide an example?

In “guile-sly”, for example, the configure script accepts
“--with-libfreeimage-prefix=” and “--with-libgslcblas-prefix=”, which
are then used to construct a full path to the libraries.  In other cases
where the build system does not provide a mechanism for this we need to
patch the sources such as in “guile-dbi”.

-- 
Ricardo

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 507 bytes --]


> In “guile-sly”, for example, the configure script accepts
> “--with-libfreeimage-prefix=” and “--with-libgslcblas-prefix=”, which
> are then used to construct a full path to the libraries.  In other cases
> where the build system does not provide a mechanism for this we need to
> patch the sources such as in “guile-dbi”.

But as far I know of, most Common Lisp packages don't provide a configure
script.  So can this apply here?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

bug#32770: Packaging SLIME/SWANK as Common Lisp library

[Andy Patterson]

Hi Pierre,

On Fri, 05 Oct 2018 15:30:03 +0200
Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> > In “guile-sly”, for example, the configure script accepts
> > “--with-libfreeimage-prefix=” and “--with-libgslcblas-prefix=”,
> > which are then used to construct a full path to the libraries.  In
> > other cases where the build system does not provide a mechanism for
> > this we need to patch the sources such as in “guile-dbi”.  
> 
> But as far I know of, most Common Lisp packages don't provide a
> configure script.  So can this apply here?
> 

I was suggesting that lisp packages should adopt some configuration
mechanism, either through scripts or by something that asdf could take
care of.  It's something that I think we could discuss with the
upstream communities once we have a good grasp on our use case.

--
Andy