bug#48682: recursive import between (gnu packages chez) and (gnu packages racket)

bug#48682: recursive import between (gnu packages chez) and (gnu packages racket)

[raingloom]

[-- Attachment #1: Type: text/plain, Size: 1140 bytes --]

bisected it to this commit:
ae88e30a0f8403e781f8b01262766cdc46b1018a

* bisect.sh: the script used for git bisect run
* BISECT_LOG: obvious
* test.scm: more detailed test, with backtrace. guile dies while
  printing the backtrace.

Issue doesn't occur when i comment out the use-module line in
racket.scm that imports (@ (gnu packages chez) chez-scheme).

IRC discussion:
roptat suggests it might be triggered by something inheriting from
chez-scheme, cbaines says recursive modules usually don't cause issues,
only recursive inputs do. i haven't found any relevant inherits with a
quick grep, but maybe it's transitive or something.
I also haven't found any other mention of chez-scheme in racket.scm
other than the line that imports it, which is weird, given that Racket
is now built on Chez, so I'd expect it to use it as an input. I guess
it's a bundled version? In any case I don't think I should just remove
the import, because it will be needed eventually, so this issue needs
to be fixed by then.

That's as far as I've gotten today, will hopefully have time to look
into it tomorrow.

ps.: should I forward this to the guile list?

[-- Attachment #2: test.scm --]
[-- Type: text/x-scheme, Size: 61 bytes --]

(set! %load-verbosely #t)
(load "gnu/packages/chez.scm")
,bt

[-- Attachment #3: bisect.sh --]
[-- Type: application/x-shellscript, Size: 102 bytes --]

[-- Attachment #4: BISECT_LOG --]
[-- Type: application/octet-stream, Size: 1522 bytes --]

git bisect start
# bad: [f921981536d6e99f5807f721977e83c13827c091] gnu: Add r-decipher.
git bisect bad f921981536d6e99f5807f721977e83c13827c091
# good: [6a54e8138bda39722c9418d4cb674f53ecc4e594] gnu: Add jumpapp.
git bisect good 6a54e8138bda39722c9418d4cb674f53ecc4e594
# bad: [f921981536d6e99f5807f721977e83c13827c091] gnu: Add r-decipher.
git bisect bad f921981536d6e99f5807f721977e83c13827c091
# good: [9b4f9039e8b2ac4087b1d714e7987c2a1d4e8c4c] gnu: HYPRE: Update source and home page URLs.
git bisect good 9b4f9039e8b2ac4087b1d714e7987c2a1d4e8c4c
# bad: [a154048d980ada9b7dcc35e35cbe75056a0cd756] gnu: emacs-shell-command+: Update to 2.2.0.
git bisect bad a154048d980ada9b7dcc35e35cbe75056a0cd756
# good: [77bd1b5be0cfebdd3c6fbc55658c473cf43f2832] gnu: Add cl-parser-combinators.
git bisect good 77bd1b5be0cfebdd3c6fbc55658c473cf43f2832
# good: [3c52c96ca2828c580986a72944c60ae10b6024c3] gnu: Remove unnecessary patches for emacs-geiser.
git bisect good 3c52c96ca2828c580986a72944c60ae10b6024c3
# bad: [f820315ddd92c4c8516b0ee168993b95de076c11] gnu: emacs-xyz.scm: Add missing copyright line.
git bisect bad f820315ddd92c4c8516b0ee168993b95de076c11
# bad: [ae88e30a0f8403e781f8b01262766cdc46b1018a] gnu: Add emacs-geiser-racket.
git bisect bad ae88e30a0f8403e781f8b01262766cdc46b1018a
# good: [7c1065030d5632ee5d83086df4014b80c3b6f8c2] gnu: lagrange: Update to 1.4.2.
git bisect good 7c1065030d5632ee5d83086df4014b80c3b6f8c2
# first bad commit: [ae88e30a0f8403e781f8b01262766cdc46b1018a] gnu: Add emacs-geiser-racket.

bug#48682: recursive import between (gnu packages chez) and (gnu packages racket)

[Philip McGrath]

Hi, I've been working on Racket packaging. I haven't looked into this 
much yet---hopefully I can tomorrow---but here are a few quick thoughts.

On 5/26/21 2:54 PM, raingloom wrote:
 > I also haven't found any other mention of chez-scheme in racket.scm
 > other than the line that imports it, which is weird, given that Racket

This was my doing. When I added `(gnu packages racket)` in [1], I 
adapted the patch from a racket.scm I'd been using to experiment ([2], 
also some branches of [3]): the changes that actually use (gnu packages 
chez) aren't ready to go yet, but apparently I left the import line in.


On 5/26/21 2:54 PM, raingloom wrote:
 > other than the line that imports it, which is weird, given that Racket
 > is now built on Chez, so I'd expect it to use it as an input. I guess
 > it's a bundled version? In any case I don't think I should just remove
 > the import, because it will be needed eventually, so this issue needs
 > to be fixed by then.

Racket uses a fork of Chez Scheme: I described the situation in [4]. 
Racket won't have upstream Chez as an input in the foreseeable, but, 
from a packaging perspective, building Racket's Chez fork uses many of 
the same pieces as building upstream Chez. I think it may make sense to 
refactor out some of the common infrastructure, so both `(gnu packages 
racket)` and `(gnu packages chez)` could depend on some low-level 
module, but I'm not sure what the Guix-preferred way to organize things 
would be. (I'm mostly a Racketeer using Racket packaging to learn more 
about Guix.)

Here are some ways `(gnu packages chez)` and `(gnu packages racket)` are 
intertwined, in no particular order:

   - `racket-minimal` should use the `nanopass` and `stex` origins
      as `chez-scheme` does. (But it would be easiest to do this
      once I've changed `racket-minimal` to build from the Git source,
      rather than the bootstrapped tarball: I hope that will be soon.)
      Some more of my not-quite-current thoughts on that at [5].

   - The `chez-scheme` phases `unpack-nanopass+stex`, `configure`,
     `prepare-stex`, and `install-doc` should be shared with Racket.
     I think it would be better to put them in a build-side module and
     actually share them, rather than to do tricky things to extract
     their s-expression representation from
     `(package-arguments chez-scheme)`. On the other hand, I think a
     build system would be overkill: it would only build vanilla Chez
     and Racket-flavored Chez.

   - Racket-flavored Chez has added some backends that vanilla Chez
     doesn't support (and can't be readily ported, unless upstream Chez
     eventually adopts Racket's changes to generating backends).
     In particular, Racket-flavored Chez adds support for threading on
     ARMv6 and support for aarch64 (which vanilla Chez doesn't support at
     all). I haven't thought at all deeply about this, but it might make
     sense for the default `chez-scheme` package on those architectures
     to be Racket-flavored Chez.

   - We may in fact want to use Racket to bootstrap vanilla Chez.

     Chez has the usual problem with self-hosting compilers that you
     need the old version to compile the new version. Specifically, you
     need "bootfiles", which encapsulate the Scheme-implemented portion
     of Chez, compiled to machine code. Thus, (a) they are platform-
     specific and (b) they need to agree precisely with the C-implemented
     part of Chez on the layout of datatypes and such.

     The vanilla Chez Git repository [6] has bootfiles for i686, x86_64,
     and (non-threaded) ARMv6. Once you've done a native build for one of
     those platforms, you can use it to cross-compile for any platform
     Chez supports (ppc32, various BSDs, etc.). But these are binary
     blobs. From a "trusting trust" perspective, it's especially striking
     if you consider that Chez Scheme was non-free software from 1984 to
     2016, and the first libre release likewise needed the bootfiles of
     its ancestors. (However, building bootfiles is reproducible:
     Chez in fact builds them twice and errors if they aren't
     byte-for-byte identical.)

     But Racket is able to simulate enough of Chez to (slowly)
     compile the Chez compiler and generate bootfiles, providing a path
     to Chez from just a C compiler. Racket does its whole bootstrapping
     process regularly in CI, and I'm working on getting the Guix package
     to do likewise.

     Unfortunately, the Racket fork has diverged enough from vanilla Chez
     that the current Racket "cs-bootstrap" package [7] can't build
     vanilla Chez, but I hope that the solution is just a matter of
     walking through the Git history to find the right older version of
     the bootstrapping package, before the Racket fork's `#!base-rtd`
     gained a vector of ancestors rather than a parent and a few such
     things. But I'd like to be able to build Racket packages with Guix
     before I try that :)

Ok, that got a bit long.

I don't know where the cycle came from with `emacs-geiser-racket`, but I 
think it would be reasonable either to do some refactoring to `(gnu 
packages chez)` and `(gnu packages racket)`, or to just remove the 
import for now to un-break things and figure out the rest later.

-Philip

[1]: https://issues.guix.gnu.org/47829
[2]: https://gitlab.com/philip1/guix-racket-experiment
[3]: https://gitlab.com/philip1/guix-patches/
[4]: https://issues.guix.gnu.org/46865#3
[5]: https://issues.guix.gnu.org/47153#2
[6]: https://github.com/cisco/ChezScheme
[7]: 
https://github.com/racket/racket/tree/master/racket/src/ChezScheme/rktboot

bug#48682: recursive import between (gnu packages chez) and (gnu packages racket)

[raingloom]

[-- Attachment #1: Type: text/plain, Size: 5903 bytes --]

On Thu, 27 May 2021 00:26:56 -0400
Philip McGrath <philip@philipmcgrath.com> wrote:

> Hi, I've been working on Racket packaging. I haven't looked into this 
> much yet---hopefully I can tomorrow---but here are a few quick
> thoughts.
> 
> On 5/26/21 2:54 PM, raingloom wrote:
>  > I also haven't found any other mention of chez-scheme in racket.scm
>  > other than the line that imports it, which is weird, given that
>  > Racket  
> 
> This was my doing. When I added `(gnu packages racket)` in [1], I 
> adapted the patch from a racket.scm I'd been using to experiment
> ([2], also some branches of [3]): the changes that actually use (gnu
> packages chez) aren't ready to go yet, but apparently I left the
> import line in.
> 
> 
> On 5/26/21 2:54 PM, raingloom wrote:
>  > other than the line that imports it, which is weird, given that
>  > Racket is now built on Chez, so I'd expect it to use it as an
>  > input. I guess it's a bundled version? In any case I don't think I
>  > should just remove the import, because it will be needed
>  > eventually, so this issue needs to be fixed by then.  
> 
> Racket uses a fork of Chez Scheme: I described the situation in [4]. 
> Racket won't have upstream Chez as an input in the foreseeable, but, 
> from a packaging perspective, building Racket's Chez fork uses many
> of the same pieces as building upstream Chez. I think it may make
> sense to refactor out some of the common infrastructure, so both
> `(gnu packages racket)` and `(gnu packages chez)` could depend on
> some low-level module, but I'm not sure what the Guix-preferred way
> to organize things would be. (I'm mostly a Racketeer using Racket
> packaging to learn more about Guix.)
> 
> Here are some ways `(gnu packages chez)` and `(gnu packages racket)`
> are intertwined, in no particular order:
> 
>    - `racket-minimal` should use the `nanopass` and `stex` origins
>       as `chez-scheme` does. (But it would be easiest to do this
>       once I've changed `racket-minimal` to build from the Git source,
>       rather than the bootstrapped tarball: I hope that will be soon.)
>       Some more of my not-quite-current thoughts on that at [5].
> 
>    - The `chez-scheme` phases `unpack-nanopass+stex`, `configure`,
>      `prepare-stex`, and `install-doc` should be shared with Racket.
>      I think it would be better to put them in a build-side module and
>      actually share them, rather than to do tricky things to extract
>      their s-expression representation from
>      `(package-arguments chez-scheme)`. On the other hand, I think a
>      build system would be overkill: it would only build vanilla Chez
>      and Racket-flavored Chez.
> 
>    - Racket-flavored Chez has added some backends that vanilla Chez
>      doesn't support (and can't be readily ported, unless upstream
> Chez eventually adopts Racket's changes to generating backends).
>      In particular, Racket-flavored Chez adds support for threading on
>      ARMv6 and support for aarch64 (which vanilla Chez doesn't
> support at all). I haven't thought at all deeply about this, but it
> might make sense for the default `chez-scheme` package on those
> architectures to be Racket-flavored Chez.
> 
>    - We may in fact want to use Racket to bootstrap vanilla Chez.
> 
>      Chez has the usual problem with self-hosting compilers that you
>      need the old version to compile the new version. Specifically,
> you need "bootfiles", which encapsulate the Scheme-implemented portion
>      of Chez, compiled to machine code. Thus, (a) they are platform-
>      specific and (b) they need to agree precisely with the
> C-implemented part of Chez on the layout of datatypes and such.
> 
>      The vanilla Chez Git repository [6] has bootfiles for i686,
> x86_64, and (non-threaded) ARMv6. Once you've done a native build for
> one of those platforms, you can use it to cross-compile for any
> platform Chez supports (ppc32, various BSDs, etc.). But these are
> binary blobs. From a "trusting trust" perspective, it's especially
> striking if you consider that Chez Scheme was non-free software from
> 1984 to 2016, and the first libre release likewise needed the
> bootfiles of its ancestors. (However, building bootfiles is
> reproducible: Chez in fact builds them twice and errors if they aren't
>      byte-for-byte identical.)
> 
>      But Racket is able to simulate enough of Chez to (slowly)
>      compile the Chez compiler and generate bootfiles, providing a
> path to Chez from just a C compiler. Racket does its whole
> bootstrapping process regularly in CI, and I'm working on getting the
> Guix package to do likewise.
> 
>      Unfortunately, the Racket fork has diverged enough from vanilla
> Chez that the current Racket "cs-bootstrap" package [7] can't build
>      vanilla Chez, but I hope that the solution is just a matter of
>      walking through the Git history to find the right older version
> of the bootstrapping package, before the Racket fork's `#!base-rtd`
>      gained a vector of ancestors rather than a parent and a few such
>      things. But I'd like to be able to build Racket packages with
> Guix before I try that :)
> 
> Ok, that got a bit long.
> 
> I don't know where the cycle came from with `emacs-geiser-racket`,
> but I think it would be reasonable either to do some refactoring to
> `(gnu packages chez)` and `(gnu packages racket)`, or to just remove
> the import for now to un-break things and figure out the rest later.
> 
> -Philip
> 
> [1]: https://issues.guix.gnu.org/47829
> [2]: https://gitlab.com/philip1/guix-racket-experiment
> [3]: https://gitlab.com/philip1/guix-patches/
> [4]: https://issues.guix.gnu.org/46865#3
> [5]: https://issues.guix.gnu.org/47153#2
> [6]: https://github.com/cisco/ChezScheme
> [7]: 
> https://github.com/racket/racket/tree/master/racket/src/ChezScheme/rktboot

Well, here is a quick and obvious fix for now.

[-- Attachment #2: 0001-gnu-Break-up-import-loop-between-gnu-packages-racket.patch --]
[-- Type: text/x-patch, Size: 868 bytes --]

From cef344639ccbf45b5397368d643a36fa48dac098 Mon Sep 17 00:00:00 2001
From: raingloom <raingloom@riseup.net>
Date: Fri, 28 May 2021 03:59:16 +0200
Subject: [PATCH] gnu: Break up import loop between (gnu packages racket) and
 (gnu packages chez).

* gnu/packages/racket.scm: Remove (gnu packages chez) import.
---
 gnu/packages/racket.scm | 2 --
 1 file changed, 2 deletions(-)

diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
index be33270b30..2d606071fe 100644
--- a/gnu/packages/racket.scm
+++ b/gnu/packages/racket.scm
@@ -32,8 +32,6 @@
   #:use-module (ice-9 match)
   #:use-module (gnu packages)
   #:use-module (gnu packages bash)
-  #:use-module ((gnu packages chez)
-                #:select (chez-scheme))
   #:use-module (gnu packages compression)
   #:use-module (gnu packages databases)
   #:use-module (gnu packages fontutils)
-- 
2.31.1

bug#48682: recursive import between (gnu packages chez) and (gnu packages racket)

[Philip McGrath]

On 5/27/21 10:02 PM, raingloom wrote:
> Well, here is a quick and obvious fix for now.

This looks good to me.

-Philip

bug#48682: recursive import between (gnu packages chez) and (gnu packages racket)

[Ludovic Courtès]

Hi,

raingloom <raingloom@riseup.net> skribis:

>>From cef344639ccbf45b5397368d643a36fa48dac098 Mon Sep 17 00:00:00 2001
> From: raingloom <raingloom@riseup.net>
> Date: Fri, 28 May 2021 03:59:16 +0200
> Subject: [PATCH] gnu: Break up import loop between (gnu packages racket) and
>  (gnu packages chez).
>
> * gnu/packages/racket.scm: Remove (gnu packages chez) import.
> ---
>  gnu/packages/racket.scm | 2 --
>  1 file changed, 2 deletions(-)

Applied, thanks!

> diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
> index be33270b30..2d606071fe 100644
> --- a/gnu/packages/racket.scm
> +++ b/gnu/packages/racket.scm
> @@ -32,8 +32,6 @@
>    #:use-module (ice-9 match)
>    #:use-module (gnu packages)
>    #:use-module (gnu packages bash)
> -  #:use-module ((gnu packages chez)
> -                #:select (chez-scheme))
>    #:use-module (gnu packages compression)

In general we cannot use #:select for (gnu packages …) modules because
that doesn’t play well with circular module dependencies.

That said… what problem is this fixing raingloom?

Usually, circular top-level references lead things like ‘guix show
chez-scheme’ (this amounts to loading (gnu packages chez)), but it
worked fine for me.

Thanks,
Ludo’.

bug#48682: recursive import between (gnu packages chez) and (gnu packages racket)

[Philip McGrath]

Hi, thanks!

> That said… what problem is this fixing raingloom?
> 
> Usually, circular top-level references lead things like ‘guix show
> chez-scheme’ (this amounts to loading (gnu packages chez)), but it
> worked fine for me.

It seems like some other commit may have broken the cycle in the mean 
time. Here's what I had been seeing:

```
philip@avalon:~$ guix time-machine 
--commit=ae88e30a0f8403e781f8b01262766cdc46b1018a -- show chez-scheme
Backtrace:
           19 (primitive-load-path "gnu/packages/racket" #<procedure …>)
In gnu/packages/racket.scm:
      22:0 18 (_)
In ice-9/boot-9.scm:
    3409:4 17 (define-module* _ #:filename _ #:pure _ #:version _ # _ …)
   3422:24 16 (_)
    222:29 15 (map1 (((guix licenses) #:select (asl2.0 expat #)) (#) …))
    222:29 14 (map1 (((guix packages)) ((guix download)) ((guix #)) …))
    222:29 13 (map1 (((guix download)) ((guix git-download)) ((# …)) …))
    222:29 12 (map1 (((guix git-download)) ((guix utils)) ((guix …)) …))
    222:29 11 (map1 (((guix utils)) ((guix gexp)) ((guix # gnu)) (#) …))
    222:29 10 (map1 (((guix gexp)) ((guix build-system gnu)) ((# …)) …))
    222:29  9 (map1 (((guix build-system gnu)) ((srfi srfi-1)) ((…)) …))
    222:29  8 (map1 (((srfi srfi-1)) ((ice-9 match)) ((gnu #)) ((…)) …))
    222:29  7 (map1 (((ice-9 match)) ((gnu packages)) ((gnu # #)) # …))
    222:29  6 (map1 (((gnu packages)) ((gnu packages bash)) ((…) …) …))
    222:29  5 (map1 (((gnu packages bash)) ((gnu packages chez) # …) …))
    222:17  4 (map1 (((gnu packages chez) #:select (chez-scheme)) # …))
   3353:12  3 (resolve-interface (gnu packages chez) #:select _ #:hide …)
    260:13  2 (for-each #<procedure 7fe3d526c500 at ice-9/boot-9.scm…> …)
   3360:19  1 (_ _)
   1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Throw to key `unbound-variable' with args `("resolve-interface" "no 
binding `~A' in module ~A" (chez-scheme (gnu packages chez)) #f)'.
```

I don't think anyone ever figured out where the cycle was. The tl;dr of 
my long comment is that there wasn't a logical cycle yet, and I was 
intending to have (gnu packages racket) import (gnu packages chez), but 
not vice versa, for now.

raingloom had discussed more on IRC---I only happened to see a little of 
it, but I could reproduce.

-Philip

bug#48682: recursive import between (gnu packages chez) and (gnu packages racket)

[Philip McGrath]

On 5/29/21 4:15 PM, Ludovic Courtès wrote:
 > In general we cannot use #:select for (gnu packages …) modules because
 > that doesn’t play well with circular module dependencies.

Ah, interesting, I'll keep that in mind. I'm used to Racket, where all 
cyclic module dependencies cause errors at compile time.

Do you have any advice on what would be good practice?

In the near future, I'll want to get the `nanopass` and `stex` origins 
for Racket, potentially via `(package-inputs chez-scheme)`---especially 
because those are not exported variables. And also this part:

 >    - The `chez-scheme` phases `unpack-nanopass+stex`, `configure`,
 >      `prepare-stex`, and `install-doc` should be shared with Racket.
 >      I think it would be better to put them in a build-side module and
 >      actually share them, rather than to do tricky things to extract
 >      their s-expression representation from
 >      `(package-arguments chez-scheme)`. On the other hand, I think a
 >      build system would be overkill: it would only build vanilla Chez
 >      and Racket-flavored Chez.

I'm getting very close to being able to make `racket` accept 
`racket-minimal` as an input, rather than duplicate most of it. This is 
exercising some features Racket has theoretically had for a while, but 
which apparently didn't quite work until ... well, this afternoon it 
worked for me outside of Guix, thanks to a bunch of fixes in the last 
few weeks by Matthew Flatt.

Would it make sense for me, when some useful amount of this works, to 
send a patch series adding a `racket-next` package? I think the changes 
are too much to backport to Racket 8.1.

-Philip

bug#48682: recursive import between (gnu packages chez) and (gnu packages racket)

[Ludovic Courtès]

Hi,

Philip McGrath <philip@philipmcgrath.com> skribis:

> On 5/29/21 4:15 PM, Ludovic Courtès wrote:
>> In general we cannot use #:select for (gnu packages …) modules because
>> that doesn’t play well with circular module dependencies.
>
> Ah, interesting, I'll keep that in mind. I'm used to Racket, where all
> cyclic module dependencies cause errors at compile time.

Yeah, in hindsight, that’s probably safer…

> Do you have any advice on what would be good practice?

For package modules, the main things are:

  1. Don’t use #:select or #:autoload for (gnu packages …) modules in a
     (gnu packages …) module.

  2. At the top level of a module, only refer to variables within that
     module.  For instance, the following would be wrong:

       (define-module (gnu packages racket)
         #:use-module (gnu packages chez)
         …)

       (define whatever
         ;; Wrong because ‘chez-scheme’ is defined in another module,
         ;; which might be part of a cycle with this one.
         (package (inherit chez-scheme) …))

       (define something
         (package
           ;; …
           (license (package-license chez-scheme))))  ;likewise

Note that references from ‘inputs’ and ‘arguments’ fields are perfectly
fine (fortunately!) because those fields are “thunked” (their value is
wrapped in a thunk).

> In the near future, I'll want to get the `nanopass` and `stex` origins
> for Racket, potentially via `(package-inputs
> chez-scheme)`---especially because those are not exported
> variables. And also this part:
>
>>    - The `chez-scheme` phases `unpack-nanopass+stex`, `configure`,
>>      `prepare-stex`, and `install-doc` should be shared with Racket.
>>      I think it would be better to put them in a build-side module and
>>      actually share them, rather than to do tricky things to extract
>>      their s-expression representation from
>>      `(package-arguments chez-scheme)`. On the other hand, I think a
>>      build system would be overkill: it would only build vanilla Chez
>>      and Racket-flavored Chez.
>
> I'm getting very close to being able to make `racket` accept
> `racket-minimal` as an input, rather than duplicate most of it. This
> is exercising some features Racket has theoretically had for a while,
> but which apparently didn't quite work until ... well, this afternoon
> it worked for me outside of Guix, thanks to a bunch of fixes in the
> last few weeks by Matthew Flatt.

Neat.

> Would it make sense for me, when some useful amount of this works, to
> send a patch series adding a `racket-next` package? I think the
> changes are too much to backport to Racket 8.1.

Possibly, your call!  :-)

Thanks,
Ludo’.