Problems with McCLIM (Common Lisp)

Problems with McCLIM (Common Lisp)

[Ricardo Wurmus]

Hi Guix,

I’d like to play with McCLIM, but I don’t know enough about Common Lisp
to understand how to launch the examples.

Here’s what I’ve done:

    guix install sbcl sbcl-mcclim sbcl-slime-swank

This is the first question, actually: if I don’t manually install
sbcl-slime-swank I get errors complaining about swank being absent.
Should sbcl-mcclim propagate sbcl-slime-swank?

Then I started the sbcl REPL and input this:

    (require :asdf)   ; I don’t know why I need this
    (require :mcclim)

This seems to compile Swank, prints a whole bunch of warnings about
invalid version strings in McCLIM and returns me to the REPL.

Then I tried

    (require :clim-examples)

but it fails with “Don't know how to REQUIRE CLIM-EXAMPLES.”.  So I
loaded the asd file manually:

    (load "/home/rekado/.guix-profile/share/common-lisp/sbcl-source/mcclim/Examples/clim-examples.asd")

Now at least clim-examples appears to be found:

   (asdf:load-system "clim-examples")

This fails with this error:

--8<---------------cut here---------------start------------->8---
debugger invoked on a ASDF/FIND-COMPONENT:MISSING-DEPENDENCY in thread
#<THREAD "main thread" RUNNING {10009C8083}>:
  Component #:MCCLIM-LAYOUTS/TAB not found, required by
  #<SYSTEM "clim-examples">
--8<---------------cut here---------------end--------------->8---

I see that “lib/sbcl/mcclim-layouts-tab.asd” exists, but loading it
doesn’t help.

At this point it should be clear that I don’t know what I’m doing.  Is
it expected that Common Lisp packages installed with Guix behave this
way…?

-- 
Ricardo

Re: Problems with McCLIM (Common Lisp)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 3836 bytes --]

Hi Ricardo,

I haven't packaged McCLIM nor looked at the package, so I'll try to help
with what I know :)

> I’d like to play with McCLIM, but I don’t know enough about Common Lisp
> to understand how to launch the examples.
>
> Here’s what I’ve done:
>
>     guix install sbcl sbcl-mcclim sbcl-slime-swank
>
> This is the first question, actually: if I don’t manually install
> sbcl-slime-swank I get errors complaining about swank being absent.
> Should sbcl-mcclim propagate sbcl-slime-swank?

When SBCL- packages are built, they include references to all the
dependencies in the .asd so there should be no need to propagate it.

This won't work if SLIME-Swank is loaded differently at load time.
This is probably what's happening here.

> Then I started the sbcl REPL and input this:
>
>     (require :asdf)   ; I don’t know why I need this

ASDF is the Common Lisp build system.  Among others, it allows you to
load Common Lisp systems.

ASDF is not standard, it's actually distributed as a package, often
together with compilers.

Some compilers enable it by default, some, like SBCL, don't, so you need
to require it yourself.

I recommend you put this in your ~/.sbclrc:

--8<---------------cut here---------------start------------->8---
(require "asdf")
--8<---------------cut here---------------end--------------->8---

>     (require :mcclim)
>
> This seems to compile Swank, prints a whole bunch of warnings about
> invalid version strings in McCLIM and returns me to the REPL.

The invalid version strings are due to the Guix sbcl-* packages using
git versions.  There are harmless.

> Then I tried
>
>     (require :clim-examples)
>
> but it fails with “Don't know how to REQUIRE CLIM-EXAMPLES.”.  So I
> loaded the asd file manually:
>
>     (load "/home/rekado/.guix-profile/share/common-lisp/sbcl-source/mcclim/Examples/clim-examples.asd")

Only the .asd files in the well known locations are recognized by ASDF
if I'm not mistaken.

Note that an .asd can forward to another.  So here the problem seems to
be with the main McCLIM .asd file.  This could be a Guix problem, see below.

> Now at least clim-examples appears to be found:
>
>    (asdf:load-system "clim-examples")
>
> This fails with this error:
>
> --8<---------------cut here---------------start------------->8---
> debugger invoked on a ASDF/FIND-COMPONENT:MISSING-DEPENDENCY in thread
> #<THREAD "main thread" RUNNING {10009C8083}>:
>   Component #:MCCLIM-LAYOUTS/TAB not found, required by
>   #<SYSTEM "clim-examples">
> --8<---------------cut here---------------end--------------->8---
>
> I see that “lib/sbcl/mcclim-layouts-tab.asd” exists, but loading it
> doesn’t help.

What's the error then?

Try

--8<---------------cut here---------------start------------->8---
(asdf:locate-system :mcclim-layouts/tab)
--8<---------------cut here---------------end--------------->8---

It will tell you where the system is.
If it does not find it, it means the .asd file did not do  the job.



With all that said, here comes the most important piece of information:
our SBCL build system has some serious flaws, most importantly because
it _rewrites_ the .asd files.  This could be why the examples cannot be
found or why mcclim-layouts/tab causes problem.

Personally I've stopped using sbcl- packages altogether.  I'm not
exclusively using cl-* packages.  The drawback is that the .fasl must be
built the first time, but then you get the smoothest experience just
like with Quicklisp.

The only exception to this, in my experience, is Osicat (the POSIX
library): the cl-* version does not work but that's an upstream issue.
For this one I use sbcl-osicat.

Hope that helps! :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Problems with McCLIM (Common Lisp)

[Guillaume Le Vaillant]

[-- Attachment #1: Type: text/plain, Size: 4824 bytes --]


Hi,

Ricardo Wurmus <rekado@elephly.net> skribis:

> I’d like to play with McCLIM, but I don’t know enough about Common Lisp
> to understand how to launch the examples.

When I packaged McCLIM, I only made the packages for the main McCLIM
library. IIRC the examples also use some extra extensions that I have
not packaged, so they might not work out of the box (even if the error
with mcclim-layouts/tab had not happened).

> Here’s what I’ve done:
>
>     guix install sbcl sbcl-mcclim sbcl-slime-swank
>
> This is the first question, actually: if I don’t manually install
> sbcl-slime-swank I get errors complaining about swank being absent.
> Should sbcl-mcclim propagate sbcl-slime-swank?

sbcl-slime-swank is actually an alias for cl-slime-swank because,
according to a comment in lisp-xyz.scm, "SLIME does not have a ASDF
system definition to build all of Swank". So in fact there is no
pre-compiled package for Swank, and this is why it needs to be 
available to packages that depend on it (directly or indirectly) so that
they can compile it before using it.


Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Hi Ricardo,
> [...]
>> Then I tried
>>
>>     (require :clim-examples)
>>
>> but it fails with “Don't know how to REQUIRE CLIM-EXAMPLES.”.  So I
>> loaded the asd file manually:
>>
>>     (load "/home/rekado/.guix-profile/share/common-lisp/sbcl-source/mcclim/Examples/clim-examples.asd")
>
> Only the .asd files in the well known locations are recognized by ASDF
> if I'm not mistaken.
>
> Note that an .asd can forward to another.  So here the problem seems to
> be with the main McCLIM .asd file.  This could be a Guix problem, see below.
>
>> Now at least clim-examples appears to be found:
>>
>>    (asdf:load-system "clim-examples")
>>
>> This fails with this error:
>>
>> --8<---------------cut here---------------start------------->8---
>> debugger invoked on a ASDF/FIND-COMPONENT:MISSING-DEPENDENCY in thread
>> #<THREAD "main thread" RUNNING {10009C8083}>:
>>   Component #:MCCLIM-LAYOUTS/TAB not found, required by
>>   #<SYSTEM "clim-examples">
>> --8<---------------cut here---------------end--------------->8---
>>
>> I see that “lib/sbcl/mcclim-layouts-tab.asd” exists, but loading it
>> doesn’t help.
>
> What's the error then?
>
> Try
>
> --8<---------------cut here---------------start------------->8---
> (asdf:locate-system :mcclim-layouts/tab)
> --8<---------------cut here---------------end--------------->8---
>
> It will tell you where the system is.
> If it does not find it, it means the .asd file did not do  the job.

The SBCL build system currently doesn't accept pre-compiled sbcl-*
packages having a slash in their name. Slashes are replaced with
hyphens, and the ASDF system name for the pre-compiled package for
"mcclim-layouts/tab" is in fact "mcclim-layouts-tab". Some package
definitions have a phase changing slashes to hyphens in asd files when
necessary (see for example the package definition for
sbcl-mcclim-extensions), however the asd files in
"share/common-lisp/sbcl-source/mcclim/" still use the system names with
slashes, this is why when loading the "clim-examples.asd" file by hand
you get an error saying that "mcclim-layouts/tab" would not be found.

> With all that said, here comes the most important piece of information:
> our SBCL build system has some serious flaws, most importantly because
> it _rewrites_ the .asd files.  This could be why the examples cannot be
> found or why mcclim-layouts/tab causes problem.
>
> Personally I've stopped using sbcl- packages altogether.  I'm not
> exclusively using cl-* packages.  The drawback is that the .fasl must be
> built the first time, but then you get the smoothest experience just
> like with Quicklisp.
>
> The only exception to this, in my experience, is Osicat (the POSIX
> library): the cl-* version does not work but that's an upstream issue.
> For this one I use sbcl-osicat.
>
> Hope that helps! :)

I agree that the asdf-build-system could need an overhaul.

Currently the sbcl-xxx package is the base definition and the cl-xxx and
ecl-xxx packages are derived from it, I think it would make more sense
make cl-xxx the base definition and derive sbcl-xxx and ecl-xxx from it.

The "deliver-asd" operation has been fixed in recent versions of ASDF,
so maybe it would be possible to use it instead of our home-made
functions to create asd files for the bundles.

Another approach could be not to use ASDF bundles at all, and just use
the regular compilation operation of ASDF, except the fasl files would
be put it "/gnu/store/..." instead of "$HOME/.cache/common-lisp/...",
and our asdf-build-system would indicate to ASDF where to search for the
files.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Problems with McCLIM (Common Lisp)

[Ricardo Wurmus]

Hi Guillaume,

thank you for your response!  (And to Pierre, whose comments you
commented on.)

> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> I’d like to play with McCLIM, but I don’t know enough about Common Lisp
>> to understand how to launch the examples.
>
> When I packaged McCLIM, I only made the packages for the main McCLIM
> library. IIRC the examples also use some extra extensions that I have
> not packaged, so they might not work out of the box (even if the error
> with mcclim-layouts/tab had not happened).
>
>> Here’s what I’ve done:
>>
>>     guix install sbcl sbcl-mcclim sbcl-slime-swank
>>
>> This is the first question, actually: if I don’t manually install
>> sbcl-slime-swank I get errors complaining about swank being absent.
>> Should sbcl-mcclim propagate sbcl-slime-swank?
>
> sbcl-slime-swank is actually an alias for cl-slime-swank because,
> according to a comment in lisp-xyz.scm, "SLIME does not have a ASDF
> system definition to build all of Swank". So in fact there is no
> pre-compiled package for Swank, and this is why it needs to be 
> available to packages that depend on it (directly or indirectly) so that
> they can compile it before using it.

But shouldn’t it then be propagated?  I had to guess that I might need
to install it.  I think it should just be installed when installing
McClim.

> The SBCL build system currently doesn't accept pre-compiled sbcl-*
> packages having a slash in their name. Slashes are replaced with
> hyphens, and the ASDF system name for the pre-compiled package for
> "mcclim-layouts/tab" is in fact "mcclim-layouts-tab". Some package
> definitions have a phase changing slashes to hyphens in asd files when
> necessary (see for example the package definition for
> sbcl-mcclim-extensions), however the asd files in
> "share/common-lisp/sbcl-source/mcclim/" still use the system names with
> slashes, this is why when loading the "clim-examples.asd" file by hand
> you get an error saying that "mcclim-layouts/tab" would not be found.

So, is this something we need to patch in all the asd files for McClim?
Or is this something we should change in the build system?

I’d like to play with McClim and all its applications to see if it would
be worth doing something like this for Guile :)

> I agree that the asdf-build-system could need an overhaul.
>
> Currently the sbcl-xxx package is the base definition and the cl-xxx and
> ecl-xxx packages are derived from it, I think it would make more sense
> make cl-xxx the base definition and derive sbcl-xxx and ecl-xxx from it.
>
> The "deliver-asd" operation has been fixed in recent versions of ASDF,
> so maybe it would be possible to use it instead of our home-made
> functions to create asd files for the bundles.
>
> Another approach could be not to use ASDF bundles at all, and just use
> the regular compilation operation of ASDF, except the fasl files would
> be put it "/gnu/store/..." instead of "$HOME/.cache/common-lisp/...",
> and our asdf-build-system would indicate to ASDF where to search for the
> files.

I don’t know enough about Common Lisp to give a valuable comment here,
but I’d very much like to be able to install Common Lisp things without
having to do additional work post installation.  Using more “default”
ways to install ASDF bundles perhaps would get us closer to a default
experience.

-- 
Ricardo

Re: Problems with McCLIM (Common Lisp)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2102 bytes --]

Hi Ricardo,

Ricardo Wurmus <rekado@elephly.net> writes:

> But shouldn’t it then be propagated?  I had to guess that I might need
> to install it.  I think it should just be installed when installing
> McClim.

I guess so, but I've never used McClim myself.

> So, is this something we need to patch in all the asd files for McClim?
> Or is this something we should change in the build system?

Something to change in our build system.

> I’d like to play with McClim and all its applications to see if it would
> be worth doing something like this for Guile :)

Maybe.  Have you considered using GObject Introspection to build
full-fledged GTK apps in Guile?  I've heard of 2 Guile libraries for GI:
guile-gi and g-golf.

>> The "deliver-asd" operation has been fixed in recent versions of ASDF,
>> so maybe it would be possible to use it instead of our home-made
>> functions to create asd files for the bundles.
>>
>> Another approach could be not to use ASDF bundles at all, and just use
>> the regular compilation operation of ASDF, except the fasl files would
>> be put it "/gnu/store/..." instead of "$HOME/.cache/common-lisp/...",
>> and our asdf-build-system would indicate to ASDF where to search for the
>> files.

Good ideas.

In my opinion, the fasls (pre-built binaries) should go to their
respective package outputs.

Every Common Lisp package would then have its sources in "out", the SBCL
fasls in "sbcl", the ECL libs in "ecl", etc.

After all, the main benefit of compiling the fasls is to make sure our
package loads properly.

> I don’t know enough about Common Lisp to give a valuable comment here,
> but I’d very much like to be able to install Common Lisp things without
> having to do additional work post installation.  Using more “default”
> ways to install ASDF bundles perhaps would get us closer to a default
> experience.

We already have this if you install the cl- packages instead of the
sbcl- ones.  It should provide a smooth user experience.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Problems with McCLIM (Common Lisp)

[Ricardo Wurmus]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

>> I’d like to play with McClim and all its applications to see if it would
>> be worth doing something like this for Guile :)
>
> Maybe.  Have you considered using GObject Introspection to build
> full-fledged GTK apps in Guile?  I've heard of 2 Guile libraries for GI:
> guile-gi and g-golf.

That’s not the same.  I really do want the Lisp Machine experience, not
just GTK apps.

-- 
Ricardo

Re: Problems with McCLIM (Common Lisp)

[Guillaume Le Vaillant]

Ricardo Wurmus <rekado@elephly.net> skribis:

> Pierre Neidhardt <mail@ambrevar.xyz> writes:
>
>>> I’d like to play with McClim and all its applications to see if it would
>>> be worth doing something like this for Guile :)
>>
>> Maybe.  Have you considered using GObject Introspection to build
>> full-fledged GTK apps in Guile?  I've heard of 2 Guile libraries for GI:
>> guile-gi and g-golf.
>
> That’s not the same.  I really do want the Lisp Machine experience, not
> just GTK apps.

I added a few more McCLIM packages, which should allow you to test the
examples with something like:

--8<---------------cut here---------------start------------->8---
guix environment --ad-hoc sbcl sbcl-clim-examples cl-slime-swank -- \
  sbcl --eval '(require :asdf)' \
       --eval '(asdf:load-system "clim-examples")' \
       --eval '(clim-demo:demodemo)'
--8<---------------cut here---------------end--------------->8---

Re: Problems with McCLIM (Common Lisp)

[Konrad Hinsen]

Hi Pierre,

>>> Another approach could be not to use ASDF bundles at all, and just use
>>> the regular compilation operation of ASDF, except the fasl files would
>>> be put it "/gnu/store/..." instead of "$HOME/.cache/common-lisp/...",
>>> and our asdf-build-system would indicate to ASDF where to search for the
>>> files.
> Good ideas.
>
> In my opinion, the fasls (pre-built binaries) should go to their
> respective package outputs.

That sounds good, as does getting rid of ADSF bundles. I have more or 
less given up on numcl, for example, which fails to compile to a bundle 
in recent versions but seems to work find via fasls (at least it works 
fine with quicklisp).

Konrad.

Re: Problems with McCLIM (Common Lisp)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

> That sounds good, as does getting rid of ADSF bundles. I have more or 
> less given up on numcl, for example, which fails to compile to a bundle 
> in recent versions but seems to work find via fasls (at least it works 
> fine with quicklisp).

It seems that the Common Lisp community is mostly relying on Quicklisp
and thus rarely, if ever, uses asdf:compile-bundle-op.
The latter has been a source of oddities to me: several times a piece of
code would behave differently between compile-bundle-op and compile-op.
Upstream is almost always relying on compile-op and thus not aware of
the compile-bundle-op issues.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Problems with McCLIM (Common Lisp)

[Katherine Cox-Buday]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
>
>> That sounds good, as does getting rid of ADSF bundles. I have more or 
>> less given up on numcl, for example, which fails to compile to a bundle 
>> in recent versions but seems to work find via fasls (at least it works 
>> fine with quicklisp).
>
> It seems that the Common Lisp community is mostly relying on Quicklisp
> and thus rarely, if ever, uses asdf:compile-bundle-op.
> The latter has been a source of oddities to me: several times a piece of
> code would behave differently between compile-bundle-op and compile-op.
> Upstream is almost always relying on compile-op and thus not aware of
> the compile-bundle-op issues.

It's been awhile since I packaged any new Common Lisp systems into Guix,
but I most often experienced this with the new package-inferred style of
systems (you can probably find my messages in the guix-devel archives).

To try and side-step the aforementioned issues with Guix's bundling
code, I tried using the compile-bundle-op operation and a few others,
and none of them worked with package-inferred systems. I believe I did
some research and found that there was some esoteric reason these
operations didn't work with package-inferred systems. I would be
delighted if they did.

We do need to overhaul how we package Common Lisp software in general.
Regardless of whether the functionality we're discussing works, I think
we have many more "standard" options for laying out packages and fasls
on disk. We might want to consider looking closely at what Quicklisp
does and see if we can't map that onto the store.

I'm glad people like Pierre are looking into this! I wish I had more
time to help at the moment, but: 2020.

-- 
Katherine

Re: Problems with McCLIM (Common Lisp)

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]

Katherine Cox-Buday <cox.katherine.e@gmail.com> writes:

> To try and side-step the aforementioned issues with Guix's bundling
> code, I tried using the compile-bundle-op operation and a few others,
> and none of them worked with package-inferred systems. I believe I did
> some research and found that there was some esoteric reason these
> operations didn't work with package-inferred systems. I would be
> delighted if they did.

I had a discussion with upstream ASDF: if I recall correctly,
compile-bundle-op is not (easily) compatible with package-inferred
style.
Anyways, we should not use compile-bundle-op: this would solve so many
issues, including this one.

> We do need to overhaul how we package Common Lisp software in general.
> Regardless of whether the functionality we're discussing works, I think
> we have many more "standard" options for laying out packages and fasls
> on disk. We might want to consider looking closely at what Quicklisp
> does and see if we can't map that onto the store.

Quicklisp does not compile the Lisp files: it's roughly equivalent to
our cl-* packages.  So there is not much to learn from Quicklisp because
what we want here is provide pre-compiled packages.

And thanks for your numerous contributions! :)

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]