Lisp libraries: Other variants? (CCL, CLisp, etc.)

Lisp libraries: Other variants? (CCL, CLisp, etc.)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

Hi guix,

Right now we have the following Lisp systems packaged in Guix:

- gcl
- ecl
- clisp
- sbcl
- ccl (Clozure)
- femtolisp
- lush2

(Am I missing any?)

For Common Lisp libraries, we have 3 ASDF-based build systems:

- asdf-build-system/sbcl: Build against SBCL (de facto choice?)
- asdf-build-system/ecl: Build against ECL
- asdf-build-system/source: Raw source.

Should we support more variants?  For instance, CCL is fairly popular,
maybe it would be a good idea.  If so, should we keep going with the
current method?  That is to say, for Alexandria:

--8<---------------cut here---------------start------------->8---
(define-public cl-alexandria
  (sbcl-package->cl-source-package sbcl-alexandria))

(define-public ecl-alexandria
  (sbcl-package->ecl-package sbcl-alexandria))

; etc.
--8<---------------cut here---------------end--------------->8---

It can be a little verbose, maybe a function could help here, e.g.

--8<---------------cut here---------------start------------->8---
(define-lisp-package sbcl-alexandria cl ecl ccl)
--8<---------------cut here---------------end--------------->8---

But does it matter at all if we have the source build system?  Can all
other implementations use cl-alexandria for instance?

--
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Lisp libraries: Other variants? (CCL, CLisp, etc.)

[Programmer]

After having emailed Pierre Neidhardt, he kindly directed me to express my Common Lisp Guix package
proposal here.

I have for a short while been mulling over a proposal for an alternative and purely declarative
method of defining Common Lisp packages, with the intention of being far simpler in implementation
and use than the combination of ASDF and Quicklisp, along with being more secure than the latter.

The idea is simpler packaging, through a system called PACKAGER, which would provide a very
high-level and exhaustive declaration of an entire Common Lisp package or collection of packages
that could be compiled down to ASDF or Guix declarations, as two examples.  However, the particular
form of this has not yet been shaped by sufficiently varied packages and is not particularly
intended to be used by the user, so I'll continue to the second stage.

The simpler replacement for Quicklisp is distribution of the Common Lisp packages by the system
package manager itself.  I'm aware that Guix already hosts several Common Lisp packages, but tends
to group them with implementations.  While I can see that the common prefix is ``cl-'', I'm not yet
clear on how a package that itself starts with ``cl-'' would be prefixed.  Moving on, the idea is
simply, to easily permit installing just the Common Lisp itself in a standard location to then be
loaded easily from any Common Lisp implementation.

This location could default to "~/.common-lisp/" and be controlled through an environmental
variable, with a name such as GUIX_COMMON_LISP_HOME.  The intent is that Common Lisp libraries could
be exposed under /run/current-system/profile/lib/ and you could then have Common Lisp programs such
as the following in the standard location to load conveniently.  The following example is how the
file for ACUTE-TERMINAL-CONTROL, a library I've written, looks under this system, at
acute-terminal-control.lisp:

(unless (find-package "ACUTE-TERMINAL-CONTROL")
  (load (make-pathname :name "cl-ecma-48" :type "lisp" :defaults *load-pathname*))
  (load (make-pathname :directory '(:absolute "run" "current-system" "profile" "lib")
                       :name "acute-terminal-control" :type "lisp")))

This file checks for itself, loads its sole dependency, and then loads itself.  Note the use of
*LOAD-PATHNAME* to permit customization of the Common Lisp directory.  The dependency is another
library I've written, a leaf, and looks like this, at cl-ecma-48.lisp:

(unless (find-package "CL-ECMA-48")
  (load (make-pathname :directory '(:absolute "run" "current-system" "profile" "lib")
                       :name "cl-ecma-48" :type "lisp")))

It is entirely feasible to store FASLs under a subdirectory of this Common Lisp directory and to
have subdirectories in that by implementation with names decided by LISP-IMPLEMENTATION-TYPE.

One main way I've seen ASDF fail under Guix is in loading libraries written in other languages, as
it assumes a typical POSIX file system structure.  As the files actually loaded by this system
proposed would be generated as needed and for the system, that eliminates this.  I desire to spread
this system so that it's very easy to install and use Common Lisp libraries from the beginning of a
new distribution, using the system package manager and standard Common Lisp functions.  Note that
there's no need to install any software from the user side, unlike ASDF and Quicklisp.

I'm, of course, seeking thoughts about this proposed system and am open to any suggestions any of
you may have.  I'd be very glad to get such a pleasant system started under Guix and then slowly
moving the convention to increasingly more systems.

Re: Lisp libraries: Other variants? (CCL, CLisp, etc.)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1243 bytes --]

Hi again!

Thanks for your proposal, it brings up some interesting ideas.

Have you looked at lisp-utils.scm?  I think it incorporates some of your ideas
already.  It could perhaps be generalized following your ideas so that it works
for all Lisps out of the box.

Question: Are .fasl files compatible across Common Lisp implementations?  If
not, then what's your suggestion?

I have a few more questions:

> The idea is simpler packaging, through a system called PACKAGER, which would provide a very
> high-level and exhaustive declaration of an entire Common Lisp package or collection of packages

By "system", do you mean a _build system_ or something else?

> This location could default to "~/.common-lisp/" and be controlled through an environmental
> variable, with a name such as GUIX_COMMON_LISP_HOME.  

We want to install libraries in the store.  Why do you suggest we put it in the
user's home?

> As the files actually loaded by this system
> proposed would be generated as needed and for the system, that eliminates this.

Sorry, I don't understand how your proposal would eliminate the POSIX-assumption
issue.  Can you provide an example?

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Lisp libraries: Other variants? (CCL, CLisp, etc.)

[Programmer]

>Have you looked at lisp-utils.scm?  I think it incorporates some of your ideas
>already.  It could perhaps be generalized following your ideas so that it works
>for all Lisps out of the box.
I've not.  I did a cursory search for this as well, but didn't find anything.  Would you be so kind
as to link me to it so I can give a good idea of whether it's similar or not?

I was aiming for something specifically suited to Common Lisp, since many Schemes have their own way
of doing things, but if a general approach works best then that would be great.  It is important to
note, however, that I find it a very important quality that there be no need to load code into the
Common Lisp implementation for doing this, as that's inconvenient.  Ideally, the Common Lisp
implementation is just told what file to LOAD and everything else is handled nicely.

I'd also want to make my proposed method work across many different systems, ideally, with little
friction.

>Question: Are .fasl files compatible across Common Lisp implementations?  If
>not, then what's your suggestion?
FASLs aren't standardized in form, so even the same implementation across different versions may be
unable to use earlier or later FASLs.

My suggestion, if this becomes an issue, is to have FASLs deposited in a fasl subdirectory of the
Common Lisp directory with further subdirectories denoted by Lisp implementation name and it may be
a good idea to think ahead and also use the particular version in the subdirectory name.  This
provides a simple, perfectly portable, and rather foolproof way to get the proper behavior for this.

>By "system", do you mean a _build system_ or something else?
I meant something like ASDF, but more abstract and able to be compiled to representations including
ASDF and Guix declarations and whatnot, with the reasoning that ASDF is rather grotesquely large and
unwieldy, among other things.

>We want to install libraries in the store.  Why do you suggest we put it in the
>user's home?
You misunderstand.  The library would be in the store, but there would be a file placed in the
user's home directory, because Common Lisp has a USER-HOMEDIR-PATHNAME function, that would handle
loading dependencies and the actual library.

So, you'd LOAD the same file whether you're using Guix, Debian, etc. and so whether the actual
library is the store, /usr/lib/, or anywhere else, varying on how the system package manager does
it.

>Sorry, I don't understand how your proposal would eliminate the POSIX-assumption
>issue.  Can you provide an example?
Sure I can.  It's been a while since I've tried this, but common Common Lisp tools to load code
don't actually know anything about the structure of the system, so under POSIX they simply check the
usual locations, instead.  This blows up miserably under Guix, since it does things differently.
With the idea of generating the code used to load things from a higher-level description, you can
adapt it to the particular system, rather than simply checking many directories like it's a
configure script or something.

Just let me know if anything else is still unclear.  I'd be very glad to help Guix have a pleasant
Common Lisp packaging method that works across systems.

Re: Lisp libraries: Other variants? (CCL, CLisp, etc.)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 4671 bytes --]


> : Have you looked at lisp-utils.scm?  I think it incorporates some of your ideas
> : already.  It could perhaps be generalized following your ideas so that it works
> : for all Lisps out of the box.
> I've not.  I did a cursory search for this as well, but didn't find anything.  Would you be so kind
> as to link me to it so I can give a good idea of whether it's similar or not?

Sure: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/lisp-utils.scm
More generally, you'd be better off checking out the whole repository.  Then I
recommend you look at those files:

- guix/build-system/asdf.scm
- guix/build/lisp-utils.scm
- guix/build/asdf-build-system.scm

> I was aiming for something specifically suited to Common Lisp, since many Schemes have their own way
> of doing things, but if a general approach works best then that would be great.

I did not mean Scheme either.  I don't know if something general enough could be
devised for Scheme as well.  It's probably too ambitious ;)

> : Question: Are .fasl files compatible across Common Lisp implementations?  If
> : not, then what's your suggestion?
> FASLs aren't standardized in form, so even the same implementation across different versions may be
> unable to use earlier or later FASLs.
>
> My suggestion, if this becomes an issue, is to have FASLs deposited in a fasl subdirectory of the
> Common Lisp directory with further subdirectories denoted by Lisp implementation name and it may be
> a good idea to think ahead and also use the particular version in the subdirectory name.  This
> provides a simple, perfectly portable, and rather foolproof way to get the proper behavior for this.

But how do you build this for a single Guix package?  If you want FASLs for,
say, SBCL and CCL, then you need to invoke both compilers when building the Guix
package.  It would be much to heavy weight that every single Lisp package
depended on all Lisp compilers, plus it would result in disk consuming packages
that would benefit practically no one (no one uses all Lisp compilers at the
same time).

So we need different packages.  Which is what we are doing at the moment.
Does that make sense?

I could be missing something.  Can you see a way around this?

> : By "system", do you mean a _build system_ or something else?
> I meant something like ASDF, but more abstract and able to be compiled to representations including
> ASDF and Guix declarations and whatnot, with the reasoning that ASDF is rather grotesquely large and
> unwieldy, among other things.

If this is more general than Guix, does it still belong to the Guix project?
Shouldn't it belong to the Common Lisp community as a replacement for ASDF?  I
know the Common Lisp community has been talking about replacing ASDF for a
while.

Fare (one of the maintainers) has talked about it a few times:
https://fare.livejournal.com/190738.html and other articles.

> : We want to install libraries in the store.  Why do you suggest we put it in the
> : user's home?
> You misunderstand.  The library would be in the store, but there would be a file placed in the
> user's home directory, because Common Lisp has a USER-HOMEDIR-PATHNAME function, that would handle
> loading dependencies and the actual library.

But who places those files?  Guix cannot.
Did you mean the user guix-profile instead?

> : Sorry, I don't understand how your proposal would eliminate the POSIX-assumption
> : issue.  Can you provide an example?
> Sure I can.  It's been a while since I've tried this, but common Common Lisp tools to load code
> don't actually know anything about the structure of the system, so under POSIX they simply check the
> usual locations, instead.  This blows up miserably under Guix, since it does things differently.
> With the idea of generating the code used to load things from a higher-level description, you can
> adapt it to the particular system, rather than simply checking many directories like it's a
> configure script or something.

My understanding is that CL-CFFI makes those assumptions.  A few months back I
had brought up the issue upstream and they told me they were not eager to change
it.

https://mailman.common-lisp.net/pipermail/cffi-devel/2018-May/003048.html

The way we do it now is probably optimal.  See cl-sqlite for instance.

I'm realizing my answers might come off as discouraging.  I hope it's not, we
still need to work on the build system on the Guix side, while the Common Lisp
side would certainly benefit from improvements to ASDF (or its successor).

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Lisp libraries: Other variants? (CCL, CLisp, etc.)

[Programmer]

>Sure: https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/lisp-utils.scm
I've only taken a look at lisp-utils.scm so far.  I can see it explicitly supports sbcl and ecl.
It's possible to get most or all of this in purely portable Common Lisp, which would make adding
more implementations require no effort.

>But how do you build this for a single Guix package?  If you want FASLs for,
>say, SBCL and CCL, then you need to invoke both compilers when building the Guix
>package.  It would be much to heavy weight that every single Lisp package
>depended on all Lisp compilers, plus it would result in disk consuming packages
>that would benefit practically no one (no one uses all Lisp compilers at the
>same time).

>I could be missing something.  Can you see a way around this?
I do see value in offering compiled forms of standalone Common Lisp programs, such as StumpWM, but
don't see the point in doing this with Common Lisp libraries.

The common way to do this in Common Lisp is to distribute the source and simply compile it as
needed.  It can be advantageous to have this choice easily.

The basic idea is simply that the system package manager installs libraries wherever it will and
places a file that will load this library and its own dependencies properly in the user's home
directory.  It would be simple to have it check for and load compiled versions if present or compile
them if not after loading for faster reloading.

So, this would enable you landing into a new system to, say, reuse your Common Lisp initialization
files that reference the directory and seamlessly pick up from where you left off, once the system
package manager installs things.

Does this idea make sense?

>If this is more general than Guix, does it still belong to the Guix project?
>Shouldn't it belong to the Common Lisp community as a replacement for ASDF?  I
>know the Common Lisp community has been talking about replacing ASDF for a
>while.
I thought Guix would be a good first place to propose this idea.  If it would be more appropriate, I
now have time to build a first approximation of this idea, although I lack the Guix knowledge for a
robust implementation.  I did provide a simple example of this in my initial message, which shows
the file a library would have that loads another library that has no dependencies.

>Fare (one of the maintainers) has talked about it a few times:
>https://fare.livejournal.com/190738.html and other articles.
I'll make certain to read this.  I've read some of his articles before.

>But who places those files?  Guix cannot.
>Did you mean the user guix-profile instead?
I'm not certain by what you mean.  I did ask around earlier and learned it's not uncommon for Guix
to place files in the user's home directory when needed.

I'm not particularly familiar with Guix terminology yet.

>My understanding is that CL-CFFI makes those assumptions.  A few months back I
>had brought up the issue upstream and they told me they were not eager to change
>it.
>The way we do it now is probably optimal.  See cl-sqlite for instance.
I'll need to take a look at those later.

>I'm realizing my answers might come off as discouraging.  I hope it's not, we
>still need to work on the build system on the Guix side, while the Common Lisp
>side would certainly benefit from improvements to ASDF (or its successor).
You need not worry about discouraging me.  I'd be delighted to see ASDF and Quicklisp replaced with
more intelligent, simpler, and more secure solutions for this simple problem.

Re: Lisp libraries: Other variants? (CCL, CLisp, etc.)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 765 bytes --]

The point of pre-compiling for SBCL, ECL, etc. is that is saves time to the
user.

If you want the source-only Common Lisp library which would work across all
implementations (as long as they are supported), you can install the "cl-"
prefixed package.
Then the Common Lisp compiler invoked by the user will compile the sources into
some FASL somewhere in ~/.cache/common-lisp/... or similar.

Doesn't it suit your needs?

> I'm not certain by what you mean.  I did ask around earlier and learned it's
> not uncommon for Guix to place files in the user's home directory when needed.

I think there was a misunderstanding, Guix cannot modify anything in the user's
home but the Guix profile.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]