Using CLISP instead of CCL to bootstrap SBCL

Using CLISP instead of CCL to bootstrap SBCL

[Mark H Weaver]

Hi Pierre,

I just noticed that last November, you changed our SBCL package to use
CCL for bootstrapping.  Previously, it used CLISP.

  commit 4bddcae94bb9d19112354f8f0b93f6e381e67768
  Author: Pierre Neidhardt <mail@ambrevar.xyz>
  Date:   Sat Nov 24 18:33:55 2018 +0100
  
      gnu: sbcl: Update to 1.4.13.
      
      * gnu/packages/lisp.scm (sbcl): Update to 1.4.13.
      [native-inputs]: Use minimal texlive-union instead of full texlive.
      [native-inputs]: Use CCL instead of buggy CLISP.
      [arguments]: Replace all (zero? (system* ...)) by invoke.

Since our CCL package only supports i686 and x86_64, a few days later
Efraim changed it back to use CLISP on all non-x86 systems:

  commit 76d520facb54f4f86977683fd21bf1d4ac5ba45d
  Author: Efraim Flashner <efraim@flashner.co.il>
  Date:   Thu Nov 29 11:54:30 2018 +0200
  
      gnu: sbcl: bootstrap with clisp on non-Intel machines.
      
      * gnu/packages/lisp.scm (sbcl)[native-inputs]: If the current system is
      not x86_64-linux or i686-linux, use clisp in place of ccl.
      [arguments]: In the custom build phase, use the correct bootstrap lisp.

The most severe problem with CCL, from my perspective, is that
apparently it cannot be built from source code, or at least our package
in Guix doesn't.  As documented in our CCL package definition:

    ;; CCL consists of a "lisp kernel" and "heap image", both of which are
    ;; shipped in precompiled form in source tarballs.  The former is a C
    ;; program which we can rebuild from scratch, but the latter cannot be
    ;; generated without an already working copy of CCL, and is platform
    ;; dependent, so we need to fetch the correct tarball for the platform.

I consider this issue to be sufficiently serious that I'd like to
propose switching back to CLISP for all systems.

In your commit log, you wrote "Use CCL instead of buggy CLISP".  What
bugs are you referring to?  Is there a bug in CLISP that prevents it
from successfully bootstrapping current SBCL releases?  If so, have the
CLISP developers been informed?  Or did you make this change
preemptively, based on lack of confidence in CLISP to do the job?

My immediate interest in this is that I'd like to try "Next" browser,
but I don't want to trust a precompiled CCL heap image.  Is there a
reasonable way forward to address this concern?

       Thanks,
         Mark

Re: Using CLISP instead of CCL to bootstrap SBCL

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1276 bytes --]

Hi Mark,

First of all, thanks for your interest in Next! :)

If you look at the sbcl package, you'll see this comment which I
copy-pasted from SBCL "INSTALL" file:

--8<---------------cut here---------------start------------->8---
     ;; From INSTALL:
     ;;     Supported build hosts are:
     ;;       SBCL
     ;;       CMUCL
     ;;       CCL (formerly known as OpenMCL)
     ;;       ABCL (recent versions only)
     ;;       CLISP (only some versions: 2.44.1 is OK, 2.47 is not)
     ;;       XCL
--8<---------------cut here---------------end--------------->8---

The point was 2-fold:

- CLISP seems unreliable.
- SBCL takes ages to compile with it :p  (I know, this is mostly practical.)

We can ask the SBCL developers to let us know if they think CLISP can be
re-approved, but as far as I understand, it's mostly untested.

Another solution would be to bootstrap SBCL or CCL
differently.  I haven't looked into the details, but there may be some
older version of CCL or SBCL that could be build from C or CLISP
reliably, then use those versions to build the latest CCL and SBCL.

I can look into maybe later next week (no promise, September is going to
be tight for me).

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Using CLISP instead of CCL to bootstrap SBCL

[Guillaume Le Vaillant]

Hi,

According to the NEWS file, SBCL 1.5.0 and later can also be
bootstrapped using ECL:

---
changes in sbcl-1.5.0 relative to sbcl-1.4.16:
  [...]
  * build enhancement: new host quirks mechanism, support for building under
    ABCL and ECL (as well as CCL, CMUCL, CLISP and SBCL itself)
  [...]
---

Maybe it could be used to have the same bootstrap procedure for all
systems.

Re: Using CLISP instead of CCL to bootstrap SBCL

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 218 bytes --]

I had a look into this, and it seems that CCL cannot currently be built
without itself :(

I've opened an issue on GitHub:
https://github.com/Clozure/ccl/issues/222

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Using CLISP instead of CCL to bootstrap SBCL

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 261 bytes --]

So CCL might be a lost cause.
I've asked SBCL if it's OK to use CLISP 2.49 or ECL:

https://bugs.launchpad.net/sbcl/+bug/1842319

Note that we could also try with CMUCL, which we need to package.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Using CLISP instead of CCL to bootstrap SBCL

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 240 bytes --]

Hmmm, after some reading it seems that CMUCL suffers from the same issue
as CCL:

https://gitlab.common-lisp.net/cmucl/cmucl/wikis/InstallingCmucl

I'd need to test to make sure though.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Using CLISP instead of CCL to bootstrap SBCL

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> I had a look into this, and it seems that CCL cannot currently be built
> without itself :(

That’s what Mark was hinting at, and that’s something people here and at
<https://bootstrappable.org> have been trying hard to fix, so let’s not
spoil it!  ;-)

Ludo’.

Re: Using CLISP instead of CCL to bootstrap SBCL

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]

>> I had a look into this, and it seems that CCL cannot currently be built
>> without itself :(
>
> That’s what Mark was hinting at,

Yes, what I meant is that upstream confirmed it.

> and that’s something people here and at
> <https://bootstrappable.org> have been trying hard to fix,

Do you mean that the people at bootstrappable have worked on CCL?

> so let’s not spoil it!  ;-)

What do you mean?

From the discussion with SBCL people, it seems that the best course of
action for now is to build against ECL or CLISP.
I'll test both, pick the one that's faster and merge.

What shall we do about CCL?  Keep it as "unbootstrappable" package?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Using CLISP instead of CCL to bootstrap SBCL

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 145 bytes --]

I've updated SBCL to build against CLISP.

So now Next is back on the bootstrappability road!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Using CLISP instead of CCL to bootstrap SBCL

[Mark H Weaver]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> I've updated SBCL to build against CLISP.
>
> So now Next is back on the bootstrappability road!

Thanks very much for taking care of this!

      Mark