[bug#49280] [PATCH 0/4] gnu: racket: Add racket-next. Bootstrap from C.

[bug#49280] [PATCH 0/4] gnu: racket: Add racket-next. Bootstrap from C.

[Philip McGrath]

This patch series packages adds `racket-next` and `racket-next-minimal`
packages for Racket 8.1.900 (the first release candidate for Racket 8.2,
scheduled for late July). Beyond the release, having a development
package will be useful for Guix users who contribute to Racket or help
test the latest features.

In particular, these patches make use of improvements since the 8.1
release in Racket's support for layered and tethered instalations
to make some improvements to Guix's Racket packaging.

We can now unbundle `racket-next-minimal` from `racket-next`
by installing the main distribution as a layer on top of minimal Racket,
rather than duplicating the core runtime system in both packages.

This improvement facilitates changing the source of `racket-next-minimal`
to use the upstream Git repository, further unbundling main-distribution
packages developed in that repository, and bootstrapping Racket from C.
(One remaining bootstrapping limitation is discussed in comments added
in the patch.) The same features seem like the most promising path toward
adding support for Racket packages to Guix more generally.

-Philip

Philip McGrath (4):
  gnu: racket: Fix lib-search-dirs configuration.
  gnu: racket: Add racket-next and racket-next-minimal.
  gnu: racket-next: Unbundle racket-next-minimal.
  gnu: racket-next-minimal: Bootstrap from C.

 gnu/local.mk                                  |   1 +
 .../racket-next-minimal-sh-via-rktio.patch    |  87 +++
 gnu/packages/racket.scm                       | 525 +++++++++++++++++-
 3 files changed, 611 insertions(+), 2 deletions(-)
 create mode 100644 gnu/packages/patches/racket-next-minimal-sh-via-rktio.patch

-- 
2.30.2

[bug#49280] [PATCH 1/4] gnu: racket: Fix lib-search-dirs configuration.

[Philip McGrath]

When a config.rktd file doesn't contain an entry for `lib-search-dirs`,
the default is equivalent to `'(#f)`, not `'()`.

* gnu/packages/racket.scm (racket)[#:arguments]: Fix
patch-config.rktd-lib-search-dirs phase.
---
 gnu/packages/racket.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
index 2d606071fe..d0cfed7292 100644
--- a/gnu/packages/racket.scm
+++ b/gnu/packages/racket.scm
@@ -125,7 +125,7 @@
                                   'lib-search-dirs
                                   (lambda (dirs)
                                     (append dirs extra-lib-search-dirs))
-                                  null)
+                                  '(#f))
                                  #:exists 'truncate/replace
                                  file)))
                     "--"
-- 
2.30.2

[bug#49280] [PATCH 2/4] gnu: racket: Add racket-next and racket-next-minimal.

[Philip McGrath]

* gnu/packages/racket.scm (racket-next-minimal,racket-next): New variables.
---
 gnu/packages/racket.scm | 62 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
index d0cfed7292..363f19825b 100644
--- a/gnu/packages/racket.scm
+++ b/gnu/packages/racket.scm
@@ -23,6 +23,7 @@
   #:use-module ((guix licenses)
                 #:select (asl2.0 expat lgpl3+))
   #:use-module (guix packages)
+  #:use-module (guix base16)
   #:use-module (guix download)
   #:use-module (guix git-download)
   #:use-module (guix utils)
@@ -45,6 +46,36 @@
   #:use-module (gnu packages tls)
   #:use-module (gnu packages xorg))
 
+;; Commentary:
+;;
+;; Preliminary guidelines on naming things:
+;;   - `racket` is the main package. It corresponds to `racket-minimal`
+;;     with the Racket-level package "main-distribution" installed.
+;;   - `racket-minimal` is Racket runtime system and core libraries:
+;;     just enough to implement the package system and install the rest.
+;;     Upstream refers to this as "minimal Racket".
+;;   - `racket-pkg-` should probably be the prefix for Racket packages
+;;     available as Guix packages, once we're able to build those.
+;;     More specifically, it should correspond
+;;     to packages registered in the catalog at https://pkgs.rackat-lang.org.
+;;     This is a social convention to manage the namespace, not a technical
+;;     limitation: Racket can use other catalogs (e.g. for pre-built packages
+;;     or packages pinned to specific versions), unregistered package source
+;;     urls, or purely local packages. But we also need a convention to
+;;     manage the namespace, so we should use this one. In practice,
+;;     all generally useful libre Racket packages are registered there.
+;;     We probably will need a clever encoding scheme to deal with the fact
+;;     that Racket package names can contain [A-Za-z_-], i.e. including "_",
+;;     which is not allowed in Guix package names.
+;;   - `racket-next` is a development version of `racket`, following either
+;;     the upstrean Git HEAD or the release candidate, when one exists.
+;;   - `racket-next-` is the prefix for other development packages,
+;;     including `racket-next-minimal`. When we can build Racket packages
+;;     individually, we will need `racket-next-pkg-` for the packages
+;;     that make up `racket-next`.
+;;
+;; Code:
+
 
 (define %installer-mirrors
   ;; Source:
@@ -211,3 +242,34 @@ languages to complete language implementations.
 The main Racket distribution comes with many bundled packages, including the
 DrRacket IDE, libraries for GUI and web programming, and implementations of
 languages such as Typed Racket, R5RS and R6RS Scheme, Algol 60, and Datalog.")))
+
+(define %pre-release-installers
+  "https://pre-release.racket-lang.org/installers/")
+
+(define-public racket-next-minimal
+  (package
+    (inherit racket-minimal)
+    (name "racket-next-minimal")
+    (version "8.1.900")
+    (source
+     (origin
+       (inherit (package-source racket-minimal))
+       (sha256
+        (base32
+         "0dm849wvlaxpfgz2qmgy2kwdslyi515rxn1m1yff38lagbn21vxq"))
+       (uri (string-append %pre-release-installers
+                           "racket-minimal-src.tgz"))))))
+
+(define-public racket-next
+  (package
+    (inherit racket)
+    (name "racket-next")
+    (version (package-version racket-next-minimal))
+    (source
+     (origin
+       (inherit (package-source racket))
+       (sha256
+        (base32
+         "0ysvzgm0lx4b1p4k9balvcbvh2kapbfx91c9ls80ba062cd8y5qv"))
+       (uri (string-append %pre-release-installers
+                           "racket-src.tgz"))))))
-- 
2.30.2

[bug#49280] [PATCH 0/4] gnu: racket: Add racket-next. Bootstrap from C.

[Ludovic Courtès]

Hi!

Philip McGrath <philip@philipmcgrath.com> skribis:

> * gnu/packages/racket.scm (racket-next-minimal,racket-next): New variables.

[...]

> +++ b/gnu/packages/racket.scm
> @@ -23,6 +23,7 @@
>    #:use-module ((guix licenses)
>                  #:select (asl2.0 expat lgpl3+))
>    #:use-module (guix packages)
> +  #:use-module (guix base16)

Leftover?

> +;; Preliminary guidelines on naming things:
> +;;   - `racket` is the main package. It corresponds to `racket-minimal`
> +;;     with the Racket-level package "main-distribution" installed.
> +;;   - `racket-minimal` is Racket runtime system and core libraries:
> +;;     just enough to implement the package system and install the rest.
> +;;     Upstream refers to this as "minimal Racket".

Note that these two names match existing conventions in Guix.

I’d suggest moving the bits about the package contents/features next to
the definition of ‘racket’ and ‘racket-minimal’.

> +;;   - `racket-pkg-` should probably be the prefix for Racket packages
> +;;     available as Guix packages, once we're able to build those.
> +;;     More specifically, it should correspond
> +;;     to packages registered in the catalog at https://pkgs.rackat-lang.org.
> +;;     This is a social convention to manage the namespace, not a technical
> +;;     limitation: Racket can use other catalogs (e.g. for pre-built packages
> +;;     or packages pinned to specific versions), unregistered package source
> +;;     urls, or purely local packages. But we also need a convention to
> +;;     manage the namespace, so we should use this one. In practice,
> +;;     all generally useful libre Racket packages are registered there.
> +;;     We probably will need a clever encoding scheme to deal with the fact
> +;;     that Racket package names can contain [A-Za-z_-], i.e. including "_",
> +;;     which is not allowed in Guix package names.

For this there’s already a documented convention (info "(guix) Package
Naming"), although part of it is undocumented.  The prefix would rather
be “racket-” to match what we do with other packages–“ghc-”, “ocaml-”,
“guile-”, and so forth.

> +;;   - `racket-next` is a development version of `racket`, following either
> +;;     the upstrean Git HEAD or the release candidate, when one exists.
> +;;   - `racket-next-` is the prefix for other development packages,
> +;;     including `racket-next-minimal`. When we can build Racket packages
> +;;     individually, we will need `racket-next-pkg-` for the packages
> +;;     that make up `racket-next`.

These two are also conventional and don’t need to be documented here
IMO.

> +(define %pre-release-installers
> +  "https://pre-release.racket-lang.org/installers/")
> +
> +(define-public racket-next-minimal
> +  (package
> +    (inherit racket-minimal)
> +    (name "racket-next-minimal")
> +    (version "8.1.900")
> +    (source
> +     (origin
> +       (inherit (package-source racket-minimal))
> +       (sha256
> +        (base32
> +         "0dm849wvlaxpfgz2qmgy2kwdslyi515rxn1m1yff38lagbn21vxq"))
> +       (uri (string-append %pre-release-installers
> +                           "racket-minimal-src.tgz"))))))
> +
> +(define-public racket-next
> +  (package
> +    (inherit racket)
> +    (name "racket-next")
> +    (version (package-version racket-next-minimal))
> +    (source
> +     (origin
> +       (inherit (package-source racket))
> +       (sha256
> +        (base32
> +         "0ysvzgm0lx4b1p4k9balvcbvh2kapbfx91c9ls80ba062cd8y5qv"))
> +       (uri (string-append %pre-release-installers
> +                           "racket-src.tgz"))))))

Do I get it right that *-src.tgz are not versioned?  That they’re
updated in place regularly?

In that case, we cannot refer to them in a package definition since the
hash is bound to become stale.

What we could do is refer to, say,
<https://pre-release.racket-lang.org/installers/racket-8.1.900.1-src.tgz>.
However, I suspect this file would vanish fairly quickly from the web
site, which is not okay either.

I’m not sure what a good solution would be.  WDYT?

It may be that
‘--with-source=https://pre-release.racket-lang.org/installers/racket-src.tgz’
would do the job for those who’re into that.

Thanks,
Ludo’.

[bug#49280] [PATCH 0/4] gnu: racket: Add racket-next. Bootstrap from C.

[Philip McGrath]

Hi!

I've been mostly offline for a bit, and Racket 8.2 was released today (a 
little ahead of schedule), so I will rework this patch series to just 
update to 8.2 and not deal with adding "-next" variants for now. I'll 
respond to here, though, to keep the discussion together.

On 7/8/21 5:25 PM, Ludovic Courtès wrote:
> Philip McGrath <philip@philipmcgrath.com> skribis:
> 
>> * gnu/packages/racket.scm (racket-next-minimal,racket-next): New variables.
> 
> [...]
> 
>> +++ b/gnu/packages/racket.scm
>> @@ -23,6 +23,7 @@
>>     #:use-module ((guix licenses)
>>                   #:select (asl2.0 expat lgpl3+))
>>     #:use-module (guix packages)
>> +  #:use-module (guix base16)
> 
> Leftover?

Yes, thanks!

>> +;;   - `racket-pkg-` should probably be the prefix for Racket packages
>> +;;     available as Guix packages, once we're able to build those.
>> +;;     More specifically, it should correspond
>> +;;     to packages registered in the catalog at https://pkgs.rackat-lang.org.
>> +;;     This is a social convention to manage the namespace, not a technical
>> +;;     limitation: Racket can use other catalogs (e.g. for pre-built packages
>> +;;     or packages pinned to specific versions), unregistered package source
>> +;;     urls, or purely local packages. But we also need a convention to
>> +;;     manage the namespace, so we should use this one. In practice,
>> +;;     all generally useful libre Racket packages are registered there.
>> +;;     We probably will need a clever encoding scheme to deal with the fact
>> +;;     that Racket package names can contain [A-Za-z_-], i.e. including "_",
>> +;;     which is not allowed in Guix package names.
> 
> For this there’s already a documented convention (info "(guix) Package
> Naming"), although part of it is undocumented.  The prefix would rather
> be “racket-” to match what we do with other packages–“ghc-”, “ocaml-”,
> “guile-”, and so forth.

I wrote these as statements in the hope of eliciting any disagreement :)

The problem I see with using just “racket-” as the prefix is the 
potential for collisions, especially because Racket uses a lot of the 
namespace: for example, "_" is a useful example package for testing 
package issues, and I maintain the "_-exp" package. There don't seem to 
be Racket packages named "minimal" or "next" right now, but they seem 
reasonably likely to be used in the future, and Guix likewise may want 
to add packages that don't correspond directly to a single Racket-level 
package. (In fact, I think this may be necessary to build Racket 
packages with mutually recursive dependencies.) Other Racket package 
names that I think might be less confusing if prefixed with 
“racket-pkg-” include "base", "racket-lib", "unstable", "profile", 
"make", "data", "images", "compiler", "compatibility", "pkg-build", and 
"main-distribution".

But we don't need to resolve this now, and maybe actually implementing 
that support will clarify what issues really do or don't exist. I will 
just remove this whole comment for now, since I don't need to make a 
choice between "racket-next-minimal" and "racket-minimal-next".


>> +(define %pre-release-installers
>> +  "https://pre-release.racket-lang.org/installers/")
>> +
>> +(define-public racket-next-minimal
>> +  (package
>> +    (inherit racket-minimal)
>> +    (name "racket-next-minimal")
>> +    (version "8.1.900")
>> +    (source
>> +     (origin
>> +       (inherit (package-source racket-minimal))
>> +       (sha256
>> +        (base32
>> +         "0dm849wvlaxpfgz2qmgy2kwdslyi515rxn1m1yff38lagbn21vxq"))
>> +       (uri (string-append %pre-release-installers
>> +                           "racket-minimal-src.tgz"))))))
>> +
>> +(define-public racket-next
>> +  (package
>> +    (inherit racket)
>> +    (name "racket-next")
>> +    (version (package-version racket-next-minimal))
>> +    (source
>> +     (origin
>> +       (inherit (package-source racket))
>> +       (sha256
>> +        (base32
>> +         "0ysvzgm0lx4b1p4k9balvcbvh2kapbfx91c9ls80ba062cd8y5qv"))
>> +       (uri (string-append %pre-release-installers
>> +                           "racket-src.tgz"))))))
> 
> Do I get it right that *-src.tgz are not versioned?  That they’re
> updated in place regularly?
> 
> In that case, we cannot refer to them in a package definition since the
> hash is bound to become stale.
> 
> What we could do is refer to, say,
> <https://pre-release.racket-lang.org/installers/racket-8.1.900.1-src.tgz>.
> However, I suspect this file would vanish fairly quickly from the web
> site, which is not okay either.
> 
> I’m not sure what a good solution would be.  WDYT?
> 
> It may be that
> ‘--with-source=https://pre-release.racket-lang.org/installers/racket-src.tgz’
> would do the job for those who’re into that.

This is also a good catch!

For now, I will avoid the problem by just not dealing with "-next" variants.

For posterity: while working on this patch series before the release, I 
faced a similar issue, because the "snapshot" builds explicitly are not 
retained indefinitely. As a work-around, I based my work on snapshots 
from Northwestern University (as opposed to the University of Utah), 
because they retain one snapshot per week for a few months. For the 
longer term, rather than using the tarballs directly, I used them to 
produce patch files, which I checked into Guix. Since minimal Racket 
could be build from Git, I could restrict the patch to main-distribution 
Racket package sources, which kept the size manageable.

Something analogous would probably work for release candidates, but the 
right long-term solution is for Guix to be able to build Racket packages 
directly, so we don't have to rely on particular snapshot bundles.


On 7/8/21 5:43 PM, Ludovic Courtès wrote:
 > I’d find it clearer like this:
 >
 >    (add-before 'configure 'change-directory
 >      (lambda _
 >        (chdir "racket/src")))

Ah, that's nice.

 >
 >> +         (add-after 'install 'remove-pkgs-directory
 >> +           ;; otherwise, e.g., `raco pkg show` will try and fail to
 >> +           ;; create a lock file
 >> +           (lambda* (#:key outputs #:allow-other-keys)
 >> +             ;; rmdir because we want an error if it isn't empty
 >> +             (rmdir (string-append (assoc-ref outputs "out")
 >> +                                   "/share/racket/pkgs"))
 >> +             #t)))))
 >
 > Please write full sentences with a bit more context (“Remove package
 > directory, otherwise ‘raco pkg show’ …”).

Will do.

 >> +(define-public racket-next-minimal-bc-3m
 >> +  (hidden-package
 >> +   (package/inherit racket-next-minimal
 >> +     (name "racket-next-minimal-bc-3m")
 >
 > This is “-next” because it’s targeting 8.1, which is not released yet,
 > right?

Correct, but 8.2 (8.1 was released in May). Now that it's been released, 
the name would be "racket-minimal-bc-3m".

 > Since it’s only used for bootstrapping, perhaps use ‘define’ instead of
 > ‘define-public’ and remove the call to ‘hidden-package’.

In addition to bootstrapping, there are three reasons I know of to want 
Racket BC:

  1. The BC and CS implementations have different C APIs, so some
     low-level code may support BC but not CS. But this isn't usually a
     good reason. Racket packages should support both implementations.
     Embedding applications ideally would also be portable: if it's
     only feasible to support one implementation, it should be CS.

  2. Comparing the BC and CS implementations can be useful for testing
     and debugging, both for packages that use the FFI and when hacking
     on the Racket runtime system itself.

  3. Most importantly, BC supports some architectures that CS does not.

In particular, Racket CS does not (yet) support ppc64le, which Racket BC 
does support. The recommendation to packagers, and what Debian does, is
to explicitly use BC on platforms without CS support: 
https://github.com/racket/racket/issues/3773#issuecomment-832935403

I'm not sure what the most idiomatic way to do this is in Guix.

(Just for the record, Racket CS also supports platforms which Racket BC 
supports only partially---without the JIT, places, or futures---or does 
not support at all. One motivation of Racket CS was to make porting 
easier in general.)

 >
 > It should also be (package (inherit …) …) rather than (package/inherit
 > …).  The latter is only useful when defining variants of a package (same
 > version, same code) where the same security updates would apply.

I don't think I understand this very well. Setting aside “-next”-related 
issues, a given commit in the Racket source repository will be used to 
build CGC, 3M, and CS (the default) variants with the same version---at 
least in the Racket senses of “version” and “variant”. It's possible 
that there could be a VM-specific security issue, but usually a bug in 
Racket, security-related or otherwise, will affect all three variants.

 >> +     (inputs
 >> +      `(("libffi" ,libffi) ;; <- only for BC variants
 >> +        ,@(filter (match-lambda
 >> +                    ((label . _)
 >> +                     (not (member label
 >> +                                  '("zlib" "zlib:static"
 >> +                                    "lz4" "lz4:static")))))
 >> +                  (package-inputs racket-next-minimal))))
 >
 > Please use this more common idiom:
 >
 >    ,@(fold alist-delete (package-inputs racket-next-minimal) '("zlib" …))

Thanks, I was looking for something like `alist-delete` but didn't find it.

 >> +This packackage is the normal implementation of Racket BC with a 
precise garbage collector, 3M (``Moving Memory Mana
 >          ^
 > Typo here, and lines too long (here and in other places).  :-)

Thanks, usually I have Emacs set up to catch that.

 >> +     (license (package-license chez-scheme)))))
 >
 > You cannot do that since here since potentially we could end up with
 > circular top-level references from these two modules.
 >
 > Instead, restate what the license is.

Ok, I'd been lulled into complacency by the implicitly thunked fields.

- Philip

[bug#49280] [PATCH v2 1/3] gnu: racket: Update to 8.2.

[Philip McGrath]

* gnu/packages/racket.scm (racket-minimal, racket): Update to 8.2.
* gnu/packages/racket.scm (racket-minimal)[#:arguments]: Fix
patch-config.rktd-lib-search-dirs phase. When a config.rktd file
doesn't contain an entry for `lib-search-dirs`, the default is
equivalent to `'(#f)`, not `'()`.
---
 gnu/packages/racket.scm | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
index 2d606071fe..6b2a011d51 100644
--- a/gnu/packages/racket.scm
+++ b/gnu/packages/racket.scm
@@ -64,14 +64,14 @@
 (define-public racket-minimal
   (package
     (name "racket-minimal")
-    (version "8.1")            ; note: remember to also update racket!
+    (version "8.2")            ; note: remember to also update racket!
     (source
      (origin
        (method url-fetch)
        (uri (map (lambda (base)
                    (string-append base version "/racket-minimal-src.tgz"))
                  %installer-mirrors))
-       (sha256 "04zzqybpxss50n1jrwwq98539gw0y0ygpw9civl2sq3s4ww7m8l3")
+       (sha256 "13qfg56w554vdj5iwa8lpacy83s7bzhhyr44pjns68mkhj69ring")
        (patches (search-patches
                  "racket-sh-via-rktio.patch"))))
     (home-page "https://racket-lang.org")
@@ -125,7 +125,7 @@
                                   'lib-search-dirs
                                   (lambda (dirs)
                                     (append dirs extra-lib-search-dirs))
-                                  null)
+                                  '(#f))
                                  #:exists 'truncate/replace
                                  file)))
                     "--"
@@ -183,7 +183,7 @@ DrRacket IDE, are not included.")
                  %installer-mirrors))
        (sha256
         (base32
-         "0xdqwrwm604bbnr97h75dps2ixxz2svlw0fn0f674bn04dcfd60f"))))
+         "10sgzsraxzxp1k2y2wvz8rcjwvhbcd6k72l9lyqr34yazlwfdz26"))))
     (inputs
      `(;; sqlite and libraries for `racket/draw' are needed to build the doc.
        ("cairo" ,cairo)
-- 
2.30.2

[bug#49280] [PATCH v2 2/3] gnu: racket: Unbundle racket-minimal.

[Philip McGrath]

This change takes advantage of improved support for layered
and tethered installations in Racket 8.2.

* gnu/packages/racket.scm (extend-layer): New private variable.
This is a script for configuring a new config-tethered layer
chaining to an existing Racket installation.
* gnu/packages/racket.scm (racket)[source](snippet): Unbundle
`racket-minimal`.
[inputs]: Remove inputs that properly belong to `racket-minimal`.
[native-inputs]: Add `racket-minimal` and `extend-layer`.
[arguments]: Stop inheriting from `racket-minimal`. Add phase
'unpack-packages to move the sources and links file into place.
Replace 'configure phase using `extend-layer`.
Replace 'build phase using `raco setup`.
Delete 'install phase.
* gnu/packages/patches/racket-sh-via-rktio.patch: Rename to ...
* gnu/packages/patches/racket-minimal-sh-via-rktio.patch: ... this
file to placate `guix lint`.
* gnu/local.mk (dist_patch_DATA): Update accordingly.
* gnu/packages/racket.scm (racket-minimal)[source]: Likewise.
---
 gnu/local.mk                                  |   2 +-
 ...atch => racket-minimal-sh-via-rktio.patch} |   0
 gnu/packages/racket.scm                       | 196 +++++++++++++++++-
 3 files changed, 191 insertions(+), 7 deletions(-)
 rename gnu/packages/patches/{racket-sh-via-rktio.patch => racket-minimal-sh-via-rktio.patch} (100%)

diff --git a/gnu/local.mk b/gnu/local.mk
index 62a5e41a46..93c022d1b8 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1673,7 +1673,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/ripperx-missing-file.patch		\
   %D%/packages/patches/rpcbind-CVE-2017-8779.patch		\
   %D%/packages/patches/rtags-separate-rct.patch			\
-  %D%/packages/patches/racket-sh-via-rktio.patch		\
+  %D%/packages/patches/racket-minimal-sh-via-rktio.patch	\
   %D%/packages/patches/remake-impure-dirs.patch			\
   %D%/packages/patches/retroarch-LIBRETRO_DIRECTORY.patch	\
   %D%/packages/patches/rnp-add-version.cmake.patch		\
diff --git a/gnu/packages/patches/racket-sh-via-rktio.patch b/gnu/packages/patches/racket-minimal-sh-via-rktio.patch
similarity index 100%
rename from gnu/packages/patches/racket-sh-via-rktio.patch
rename to gnu/packages/patches/racket-minimal-sh-via-rktio.patch
diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
index 6b2a011d51..c095de42b3 100644
--- a/gnu/packages/racket.scm
+++ b/gnu/packages/racket.scm
@@ -73,7 +73,7 @@
                  %installer-mirrors))
        (sha256 "13qfg56w554vdj5iwa8lpacy83s7bzhhyr44pjns68mkhj69ring")
        (patches (search-patches
-                 "racket-sh-via-rktio.patch"))))
+                 "racket-minimal-sh-via-rktio.patch"))))
     (home-page "https://racket-lang.org")
     (synopsis "Racket without bundled packages such as DrRacket")
     (inputs
@@ -183,10 +183,55 @@ DrRacket IDE, are not included.")
                  %installer-mirrors))
        (sha256
         (base32
-         "10sgzsraxzxp1k2y2wvz8rcjwvhbcd6k72l9lyqr34yazlwfdz26"))))
+         "10sgzsraxzxp1k2y2wvz8rcjwvhbcd6k72l9lyqr34yazlwfdz26"))
+       (snippet
+        (with-imported-modules '((guix build utils)
+                                 (ice-9 match)
+                                 (ice-9 regex))
+          #~(begin
+              (use-modules (guix build utils)
+                           (ice-9 match)
+                           (ice-9 regex))
+              ;; unbundle minimal Racket
+              (for-each delete-file-recursively
+                        '("collects"
+                          "doc"
+                          "etc"
+                          "README"
+                          "src"))
+              ;; unbundle package sources included elsewhere
+              (define (substitute/delete file pattern)
+                (substitute
+                 file
+                 (list (cons pattern
+                             (lambda (line matches)
+                               ;; must match exactly once
+                               (match matches
+                                 ((m)
+                                  (string-append (match:prefix m)
+                                                 (match:suffix m)))))))))
+              (define (unbundle-pkg pkg)
+                (define quoted-pkg (regexp-quote pkg))
+                (with-directory-excursion "share"
+                  (substitute/delete
+                   "links.rktd"
+                   (string-append
+                    "[(][^()]+[(]#\"pkgs\" #\""
+                    quoted-pkg
+                    "\"[)][)]"))
+                  (with-directory-excursion "pkgs"
+                    (substitute/delete
+                     "pkgs.rktd"
+                     (string-append
+                      "[(]\""
+                      quoted-pkg
+                      "\" \\. #s[(]"
+                      "(pkg-info|[(]sc-pkg-info pkg-info 3[)])"
+                      " [(][^()]+[)] [^()]+[)][)]"))
+                    (delete-file-recursively pkg))))
+              (unbundle-pkg "racket-lib"))))))
     (inputs
-     `(;; sqlite and libraries for `racket/draw' are needed to build the doc.
-       ("cairo" ,cairo)
+     `(("cairo" ,cairo)
        ("fontconfig" ,fontconfig)
        ("glib" ,glib)
        ("glu" ,glu)
@@ -199,8 +244,67 @@ DrRacket IDE, are not included.")
        ("mpfr" ,mpfr)
        ("pango" ,pango)
        ("unixodbc" ,unixodbc)
-       ("libedit" ,libedit)
-       ,@(package-inputs racket-minimal)))
+       ("libedit" ,libedit)))
+    (native-inputs
+     `(("racket" ,racket-minimal)
+       ("extend-layer" ,extend-layer)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-before 'configure 'unpack-packages
+           (lambda* (#:key native-inputs inputs outputs #:allow-other-keys)
+             (let ((racket (assoc-ref (or native-inputs inputs) "racket"))
+                   (prefix (assoc-ref outputs "out")))
+               (mkdir-p (string-append prefix "/share/racket/pkgs"))
+               (copy-recursively
+                "share/links.rktd"
+                (string-append prefix "/share/racket/links.rktd"))
+               (copy-recursively
+                "share/pkgs"
+                (string-append prefix "/share/racket/pkgs"))
+               #t)))
+         (replace 'configure
+           (lambda* (#:key native-inputs inputs outputs #:allow-other-keys)
+             (let ((racket (assoc-ref (or native-inputs inputs) "racket"))
+                   (prefix (assoc-ref outputs "out")))
+               (apply invoke
+                      (string-append racket "/bin/racket")
+                      (assoc-ref inputs "extend-layer")
+                      racket
+                      prefix
+                      (map
+                       (lambda (lib)
+                         (string-append (assoc-ref inputs lib) "/lib"))
+                       '("cairo"
+                         "fontconfig"
+                         "glib"
+                         "glu"
+                         "gmp"
+                         "gtk+"
+                         "libjpeg"
+                         "libpng"
+                         "libx11"
+                         "mesa"
+                         "mpfr"
+                         "pango"
+                         "unixodbc"
+                         "libedit")))
+               #t)))
+         (replace 'build
+           (lambda* (#:key native-inputs inputs outputs #:allow-other-keys)
+             (invoke (string-append (assoc-ref (or native-inputs inputs)
+                                               "racket")
+                                    "/bin/racket")
+                     "--config"
+                     (string-append (assoc-ref outputs "out")
+                                    "/etc/racket")
+                     "-l"
+                     "raco"
+                     "setup")
+             #t))
+         (delete 'install))
+       ;; we still don't have these:
+       #:tests? #f))
     (synopsis "A programmable programming language in the Scheme family")
     (description
      "Racket is a general-purpose programming language in the Scheme family,
@@ -211,3 +315,83 @@ languages to complete language implementations.
 The main Racket distribution comes with many bundled packages, including the
 DrRacket IDE, libraries for GUI and web programming, and implementations of
 languages such as Typed Racket, R5RS and R6RS Scheme, Algol 60, and Datalog.")))
+
+
+(define extend-layer
+  (scheme-file
+   "extend-layer.rkt"
+   `(module
+     extend-layer racket/base
+     (require racket/cmdline
+              racket/match
+              racket/file
+              racket/list
+              racket/pretty)
+     (define config-file-pth
+       "etc/racket/config.rktd")
+     (define (build-path-string . args)
+       (path->string (apply build-path args)))
+     (define rx:racket
+       ;; Guile's reader doesn't support #rx"racket"
+       (regexp "racket"))
+     (command-line
+      #:args (parent-layer prefix . lib-dir*)
+      (let* ([config
+              (for/fold
+               ([config (file->value (build-path parent-layer
+                                                 config-file-pth))])
+               ([spec (in-list
+                       '((lib-dir lib-search-dirs "lib/racket")
+                         (share-dir share-search-dirs "share/racket")
+                         (links-file
+                          links-search-files
+                          "share/racket/links.rktd")
+                         (pkgs-dir pkgs-search-dirs "share/racket/pkgs")
+                         (bin-dir bin-search-dirs "bin")
+                         (man-dir man-search-dirs "share/man")
+                         (doc-dir doc-search-dirs "share/doc/racket")
+                         (include-dir
+                          include-search-dirs
+                          "include/racket")))])
+               (match-define (list main-key search-key pth) spec)
+               (hash-set*
+                config
+                main-key
+                (build-path-string prefix pth)
+                search-key
+                (list* #f
+                       (hash-ref config
+                                 main-key
+                                 (build-path-string parent-layer pth))
+                       (filter values (hash-ref config search-key null)))))]
+             [config
+              (hash-set config
+                        'apps-dir
+                        (build-path-string prefix "share/applications"))]
+             [config
+              ;; place new foreign lib-search-dirs before old
+              ;; foreign dirs, but after Racket layers
+              (let-values
+                  ([(rkt extra)
+                    (partition (lambda (pth)
+                                 (or (not pth)
+                                     (regexp-match? rx:racket pth)))
+                               (hash-ref config 'lib-search-dirs))])
+                (hash-set config
+                          'lib-search-dirs
+                          (append rkt
+                                  lib-dir*
+                                  extra)))]
+             [bin-dir
+              (hash-ref config 'bin-dir)]
+             [config
+              (hash-set* config
+                         'config-tethered-console-bin-dir bin-dir
+                         'config-tethered-gui-bin-dir bin-dir)]
+             [new-config-pth
+              (build-path prefix config-file-pth)])
+        (make-parent-directory* new-config-pth)
+        (call-with-output-file*
+         new-config-pth
+         (lambda (out)
+           (pretty-write config out))))))))
-- 
2.30.2

[bug#49280] [PATCH v2 0/3] gnu: racket: Update to 8.2. Bootstrap from C.

[Ludovic Courtès]

Philip McGrath <philip@philipmcgrath.com> skribis:

> +       (snippet
> +        (with-imported-modules '((guix build utils)
> +                                 (ice-9 match)
> +                                 (ice-9 regex))
> +          #~(begin
> +              (use-modules (guix build utils)
> +                           (ice-9 match)
> +                           (ice-9 regex))

This leads to this warning:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env  guix build racket -n
gnu/packages/racket.scm:429:10: warning: importing modules (ice-9 match) (ice-9 regex) from the host
--8<---------------cut here---------------end--------------->8---

In fact, we can omit ‘with-imported-modules’ altogether because (guix
build utils) happens to already be available.

I’ll try with this change.

Ludo’.

[bug#49280] [PATCH v2 3/3] gnu: racket-minimal: Bootstrap from C.

[Philip McGrath]

This commit bootstraps the Racket compiler and runtime system from source,
including Racket CS as well as both variants of Racket BC. (One remaining
limitation is discussed in comments added to gnu/packages/racket.scm.)

In the process, it moves to building minimal Racket from the Git repository,
rather than the packaged source tarballs. The Git repository is slightly
better as the ``corresponding source'':

 1. A few packages especially closely tied to the Racket core implementation
    (like "compiler-lib", "base", and "racket-doc") are developed in the
    same Git repository. Having them use the same Guix origin, too, will
    help to keep them in sync.

 2. The top-level Makefile in the Git repository is an important
    ``script[] used to control compilation and installation.''
    In particular, it cooperates with the "distro-build" package to
    create the source tarballs and installers for a Racket distribution.
    (Racket supports a notion of custom distributions.)

 3. It is ``the preferred form ... for making modifications'' to the core
    Racket implementation.

Racket releases are tagged in the Git repository (e.g. "v8.1"). At the
beginning of each release cycle, a branch is created to stabilizer a version
for extra testing. Active development happens on the "master" branch.

* gnu/packages/racket-minimal-sh-via-rktio.patch: Adjust for extra directory
layer.
* gnu/local/racket.scm (cfg-flag:sh-for-rktio, cfg-flag:enable-lt,
cfg-flag:enable-racket, unpack-nanopass+stex,
%main-repo-main-distribution-pkgs): New private variables.
* gnu/local/racket.scm (racket-minimal)[source]: Use Git.
[source](snippet): Unbundle nanopass, stex, and libffi.
[native-inputs]: Use racket-bootstrap-chez-bootfiles, plus its
dependencies (for Chez, plus a Racket for bootstrap pig).
[arguments]: Revise extensively.
* gnu/local/racket.scm (racket-minimal-bc-3m, racket-minimal-bc-cgc): New
packages, hidden at least for now.
(racket-bootstrap-chez-bootfiles): Another new package, but this one is
especially likely to stay hidden.
* gnu/local/racket.scm (racket)[origin](snippet): Unbundle packages
developed in the main Git repository, but leave their links.rktd and
pkgs.rktd entries in place.
[native-inputs]: Add the main Racket Git repository.
[arguments](#:phases): Adjust 'unpack-packages to also unpack package
sources from the main Racket Git repository.
---
 .../patches/racket-minimal-sh-via-rktio.patch |   8 +-
 gnu/packages/racket.scm                       | 488 +++++++++++++-----
 2 files changed, 359 insertions(+), 137 deletions(-)

diff --git a/gnu/packages/patches/racket-minimal-sh-via-rktio.patch b/gnu/packages/patches/racket-minimal-sh-via-rktio.patch
index b4fefd1514..6bc2ee8331 100644
--- a/gnu/packages/patches/racket-minimal-sh-via-rktio.patch
+++ b/gnu/packages/patches/racket-minimal-sh-via-rktio.patch
@@ -35,13 +35,13 @@ making this change at the C level is both:
       conditional and a runtime check that the file in the store exists,
       we make it much less likely that it will "leak" out of Guix.
 ---
- src/rktio/rktio_process.c | 21 ++++++++++++++++++++-
+ racket/src/rktio/rktio_process.c | 21 ++++++++++++++++++++-
  1 file changed, 20 insertions(+), 1 deletion(-)
 
-diff --git a/src/rktio/rktio_process.c b/src/rktio/rktio_process.c
+diff --git a/racket/src/rktio/rktio_process.c b/racket/src/rktio/rktio_process.c
 index 89202436c0..465ebdd5c5 100644
---- a/src/rktio/rktio_process.c
-+++ b/src/rktio/rktio_process.c
+--- a/racket/src/rktio/rktio_process.c
++++ b/racket/src/rktio/rktio_process.c
 @@ -1224,12 +1224,14 @@ int rktio_process_allowed_flags(rktio_t *rktio)
  /*========================================================================*/
  
diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
index c095de42b3..0349b77704 100644
--- a/gnu/packages/racket.scm
+++ b/gnu/packages/racket.scm
@@ -31,7 +31,9 @@
   #:use-module (srfi srfi-1)
   #:use-module (ice-9 match)
   #:use-module (gnu packages)
+  #:use-module (gnu packages autotools)
   #:use-module (gnu packages bash)
+  #:use-module (gnu packages chez)
   #:use-module (gnu packages compression)
   #:use-module (gnu packages databases)
   #:use-module (gnu packages fontutils)
@@ -40,25 +42,87 @@
   #:use-module (gnu packages gtk)
   #:use-module (gnu packages image)
   #:use-module (gnu packages libedit)
+  #:use-module (gnu packages libffi)
   #:use-module (gnu packages multiprecision)
   #:use-module (gnu packages sqlite)
   #:use-module (gnu packages tls)
   #:use-module (gnu packages xorg))
 
+;; Commentary:
+;;
+;; Here's how bootstrapping minimal Racket works:
+;;
+;;   - Racket BC [CGC] can be built with only a C compiler (except for
+;;     one caveat discussed below).
+;;   - Racket BC [3M] needs an existing Racket to run "xform",
+;;     which transforms its own C source code to add additional annotations
+;;     for the precise garbage collector.
+;;   - Racket CS needs (bootfiles for) Racket's fork of Chez Scheme.
+;;     It also needs an existing Racket to compile Racket-implemented
+;;     parts of the runtime system to R6RS libraries.
+;;   - Chez Scheme also needs bootfiles for itself, but Racket can simulate
+;;     enough of Chez Scheme to load Racket's fork of the Chez Scheme compiler
+;;     purely from source into Racket and apply the compiler to itself,
+;;     producing the needed bootfiles (albeit very slowly).
+;;     Any variant of Racket since version 7.1 can run the simulation.
+;;
+;; So, we build CGC to build 3M to build bootfiles and CS.
+;;
+;; One remaining bootstrapping limitation is that Racket's reader, module
+;; system, and macro expander are implemented in Racket. For Racket CS,
+;; they are compiled to R6RS libraries as discussed above. This note from the
+;; README file applies to all such subsystems:
+;;
+;;     The Racket version must be practically the same as the current Racket
+;;     verson, although it can be the Racket BC implementation (instead of
+;;     the Racket CS implementation).
+;;
+;;     Unlike Chez Scheme boot files, the files generated in "schemified"
+;;     are human-readable and -editable Scheme code. That provides a way
+;;     out of bootstrapping black holes, even without BC.
+;;
+;; However, other Racket subsystems implemented in Racket for Racket CS
+;; use older C implementations for Racket BC, whereas the reader, expander,
+;; and module system were completely replaced with the Racket implementation
+;; as of Racket 7.0.
+;;
+;; For Racket BC, the compiled "linklet" s-expressions (primitive modules)
+;; are embeded in C as a static string constant. Eventually, they are further
+;; compiled by the C-implemented Racket BC bytecode and JIT compilers.
+;; (On platforms where Racket BC's JIT is not supported, yet another compiler
+;; instead compiles the linklets to C code, but this is not a bootstrapping
+;; issue.)
+;;
+;; Code:
 
-(define %installer-mirrors
-  ;; Source:
-  ;; https://github.com/racket/racket-lang-org/blob/master/download/data.rkt#L58
-  ;; Matthew Flatt says: "note that many are commented out"
-  ;; INVARIANT: End with a trailing "/"!
-  '("https://mirror.racket-lang.org/installers/"
-    "https://www.cs.utah.edu/plt/installers/"
-    "https://plt.cs.northwestern.edu/racket-mirror/"
-    "https://mirror.csclub.uwaterloo.ca/racket/racket-installers/"
-    ;; Universität Tübingen is using a self-signed HTTPS certificate:
-    "http://mirror.informatik.uni-tuebingen.de/mirror/racket/"
-    "https://racket.infogroep.be/"
-    ))
+(define cfg-flag:sh-for-rktio
+  `(string-append "CPPFLAGS=-DGUIX_RKTIO_PATCH_BIN_SH="
+                  (assoc-ref %build-inputs "sh")
+                  "/bin/sh"))
+(define cfg-flag:enable-lt
+  `(string-append "--enable-lt="
+                  (assoc-ref %build-inputs "libtool")
+                  "/bin/libtool"))
+(define cfg-flag:enable-racket
+  `(let ((racket (assoc-ref %build-inputs "racket")))
+     (string-append "--enable-racket="
+                    racket
+                    "/bin/racket")))
+
+(define unpack-nanopass+stex
+  ;; Copied from chez-scheme.
+  ;; TODO: Eventually, we should refactor Chez Scheme
+  ;; enough to share more directly, so that we can make
+  ;; Racket's version of Chez avalable as a Guix package,
+  ;; e.g. for architectures not supported upstream.
+  ;; For now, we let Racket drive the Chez build process
+  ;; other than this step.
+  `(for-each (lambda (dep)
+               (define src
+                 (assoc-ref (or native-inputs inputs) dep))
+               (copy-recursively src dep
+                                 #:keep-mtime? #t))
+             '("nanopass" "stex")))
 
 
 (define-public racket-minimal
@@ -67,96 +131,113 @@
     (version "8.2")            ; note: remember to also update racket!
     (source
      (origin
-       (method url-fetch)
-       (uri (map (lambda (base)
-                   (string-append base version "/racket-minimal-src.tgz"))
-                 %installer-mirrors))
-       (sha256 "13qfg56w554vdj5iwa8lpacy83s7bzhhyr44pjns68mkhj69ring")
-       (patches (search-patches
-                 "racket-minimal-sh-via-rktio.patch"))))
-    (home-page "https://racket-lang.org")
-    (synopsis "Racket without bundled packages such as DrRacket")
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/racket/racket")
+             (commit (string-append "v" version))))
+       (sha256
+        "061bhiyjlvazph0dj9i3i3x2q5z53rp8h5cjwg3frjimkr45lncn")
+       (file-name (git-file-name name version))
+       (patches (search-patches "racket-minimal-sh-via-rktio.patch"))
+       (snippet
+        (with-imported-modules '((guix build utils))
+          #~(begin
+              (use-modules (guix build utils))
+              ;; unbundle Chez submodules
+              (with-directory-excursion "racket/src/ChezScheme"
+                #$(origin-snippet (package-source chez-scheme)))
+              ;; unbundle libffi
+              (for-each
+               delete-file-recursively
+               '("racket/src/bc/foreign/libffi")))))))
     (inputs
-     `(("openssl" ,openssl)
+     `(;; common to all racket-minimal variants:
+       ("openssl" ,openssl)
        ("sqlite" ,sqlite)
        ("sh" ,bash-minimal)
+       ;; only for CS
        ("zlib" ,zlib)
        ("zlib:static" ,zlib "static")
        ("lz4" ,lz4)
        ("lz4:static" ,lz4 "static")))
+    (native-inputs
+     `(("bootfiles" ,racket-bootstrap-chez-bootfiles)
+       ,@(package-native-inputs racket-bootstrap-chez-bootfiles)))
     (build-system gnu-build-system)
     (arguments
      `(#:configure-flags
-       `(,(string-append "CPPFLAGS=-DGUIX_RKTIO_PATCH_BIN_SH="
-                         (assoc-ref %build-inputs "sh")
-                         "/bin/sh")
-         "--enable-libz"
-         "--enable-liblz4")
-       #:modules
-       ((guix build gnu-build-system)
-        (guix build utils)
-        (srfi srfi-1))
+       (list "--enable-csonly"
+             "--enable-libz"
+             "--enable-liblz4"
+             ,cfg-flag:enable-racket
+             ,cfg-flag:sh-for-rktio)
+       #:out-of-source? #true
+       ;; Tests are in packages like racket-test-core and
+       ;; main-distribution-test that aren't part of the main distribution.
+       #:tests? #f
+       #:modules ((ice-9 match)
+                  (guix build gnu-build-system)
+                  (guix build utils))
        #:phases
        (modify-phases %standard-phases
-         (add-before 'configure 'pre-configure-minimal
+         (add-after 'unpack 'unpack-nanopass+stex
+           (lambda* (#:key inputs native-inputs #:allow-other-keys)
+             (with-directory-excursion "racket/src/ChezScheme"
+               ,unpack-nanopass+stex)
+             #t))
+         (add-after 'unpack-nanopass+stex 'unpack-bootfiles
+           (lambda* (#:key inputs #:allow-other-keys)
+             (with-directory-excursion "racket/src/ChezScheme"
+               (copy-recursively
+                (string-append (assoc-ref inputs "bootfiles") "/boot")
+                "boot"))
+             #t))
+         (add-before 'configure 'initialize-config.rktd
            (lambda* (#:key inputs #:allow-other-keys)
-             (chdir "src")
+             (define (write-racket-hash alist)
+               ;; inside must use dotted pair notation
+               (display "#hash(")
+               (for-each (match-lambda
+                           ((k . v)
+                            (format #t "(~s . ~s)" k v)))
+                         alist)
+               (display ")\n"))
+             (mkdir-p "racket/etc")
+             (with-output-to-file "racket/etc/config.rktd"
+               (lambda ()
+                 (write-racket-hash
+                  `((lib-search-dirs
+                     . (#f ,@(map (lambda (lib)
+                                    (string-append (assoc-ref inputs lib)
+                                                   "/lib"))
+                                  '("openssl"
+                                    "sqlite"))))
+                    (catalogs
+                     . (,(string-append
+                          "https://download.racket-lang.org/releases/"
+                          ,version
+                          "/catalog/")
+                        #f))))))
              #t))
-         (add-after 'build 'patch-config.rktd-lib-search-dirs
-           (lambda* (#:key inputs outputs #:allow-other-keys)
-             ;; We do this between the `build` and `install` phases
-             ;; so that we have racket to read and write the hash table,
-             ;; but it comes before `raco setup`, when foreign libraries
-             ;; are needed to build the documentation.
-             (define out (assoc-ref outputs "out"))
-             (apply invoke
-                    "./cs/c/racketcs"
-                    "-e"
-                    ,(format #f
-                             "~s"
-                             '(let* ((args
-                                      (vector->list
-                                       (current-command-line-arguments)))
-                                     (file (car args))
-                                     (extra-lib-search-dirs (cdr args)))
-                                (write-to-file
-                                 (hash-update
-                                  (file->value file)
-                                  'lib-search-dirs
-                                  (lambda (dirs)
-                                    (append dirs extra-lib-search-dirs))
-                                  '(#f))
-                                 #:exists 'truncate/replace
-                                 file)))
-                    "--"
-                    "../etc/config.rktd"
-                    (filter-map (lambda (lib)
-                                  (cond
-                                   ((assoc-ref inputs lib)
-                                    => (lambda (pth)
-                                         (string-append pth "/lib")))
-                                   (else
-                                    #f)))
-                                '("cairo"
-                                  "fontconfig"
-                                  "glib"
-                                  "glu"
-                                  "gmp"
-                                  "gtk+"
-                                  "libjpeg"
-                                  "libpng"
-                                  "libx11"
-                                  "mesa"
-                                  "mpfr"
-                                  "openssl"
-                                  "pango"
-                                  "sqlite"
-                                  "unixodbc"
-                                  "libedit")))
-             #t)))
-       ;; Tests are in packages like racket-test-core and
-       ;; main-distribution-test that aren't part of the main distribution.
-       #:tests? #f))
+         (add-before 'configure 'change-directory
+           (lambda _
+             (chdir "racket/src")
+             #t))
+         (add-after 'install 'remove-pkgs-directory
+           ;; If the configured pkgs-dir exists, "pkgs.rktd" does not
+           ;; exist, and a lock file does not exist, commands like
+           ;; `raco pkg show` will try to create a lock file and fail
+           ;; due to the read-only store.
+           ;; Arguably this may be a bug in `pkg/private/lock`:
+           ;; see <https://github.com/racket/racket/issues/3851>.
+           ;; As a workaround, remove the directory.
+           (lambda* (#:key outputs #:allow-other-keys)
+             ;; rmdir because we want an error if it isn't empty
+             (rmdir (string-append (assoc-ref outputs "out")
+                                   "/share/racket/pkgs"))
+             #t)))))
+    (home-page "https://racket-lang.org")
+    (synopsis "Racket without bundled packages such as DrRacket")
     (description
      "Racket is a general-purpose programming language in the Scheme family,
 with a large set of libraries and a compiler based on Chez Scheme.  Racket is
@@ -167,7 +248,164 @@ The ``minimal Racket'' distribution includes just enough of Racket for you to
 use @command{raco pkg} to install more.  Bundled packages, such as the
 DrRacket IDE, are not included.")
     ;; https://download.racket-lang.org/license.html
-    (license (list lgpl3+ asl2.0 expat))))
+    ;; The LGPL components are only used by Racket BC.
+    (license (list asl2.0 expat))))
+
+
+(define-public racket-minimal-bc-3m
+  (hidden-package
+   (package
+     (inherit racket-minimal)
+     (name "racket-minimal-bc-3m")
+     (inputs
+      `(("libffi" ,libffi) ;; <- only for BC variants
+        ,@(fold alist-delete
+                (package-inputs racket-minimal)
+                '("zlib" "zlib:static" "lz4" "lz4:static"))))
+     (native-inputs
+      `(("libtool" ,libtool)
+        ("racket" ,(if (%current-target-system)
+                       racket-minimal
+                       racket-minimal-bc-cgc))))
+     (arguments
+      (substitute-keyword-arguments (package-arguments racket-minimal)
+        ((#:configure-flags _ '())
+         `(list "--enable-bconly"
+                ,cfg-flag:enable-racket
+                ,cfg-flag:enable-lt
+                ,cfg-flag:sh-for-rktio))
+        ((#:phases usual-phases)
+         `(modify-phases ,usual-phases
+            (delete 'unpack-nanopass+stex)
+            (delete 'unpack-bootfiles)))))
+     (synopsis "Minimal Racket with the BC [3M] runtime system")
+     (description "The Racket BC (``before Chez'' or ``bytecode'')
+implementation was the default before Racket 8.0.  It uses a compiler written
+in C targeting architecture-independent bytecode, plus a JIT compiler on most
+platforms.  Racket BC has a different C API and supports a slightly different
+set of architectures than the current default runtime system, Racket CS (based
+on ``Chez Scheme'').
+
+This package is the normal implementation of Racket BC with a precise garbage
+collector, 3M (``Moving Memory Manager'').")
+     ;; https://download.racket-lang.org/license.html
+     ;; The LGPL components are only used by Racket BC.
+     (license (list lgpl3+ asl2.0 expat)))))
+
+
+(define-public racket-minimal-bc-cgc
+  (package
+    (inherit racket-minimal-bc-3m)
+    (name "racket-minimal-bc-cgc")
+    (native-inputs
+     (alist-delete "racket" (package-native-inputs racket-minimal-bc-3m)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments racket-minimal-bc-3m)
+       ((#:configure-flags _ '())
+        `(list "--enable-cgcdefault"
+               ,cfg-flag:enable-lt
+               ,cfg-flag:sh-for-rktio))))
+    (synopsis "Old Racket implementation used for bootstrapping")
+    (description "This variant of the Racket BC (``before Chez'' or
+``bytecode'') implementation is not recommended for general use.  It uses
+CGC (a ``Conservative Garbage Collector''), which was succeeded as default in
+PLT Scheme version 370 (which translates to 3.7 in the current versioning
+scheme) by the 3M variant, which in turn was succeeded in version 8.0 by the
+Racket CS implementation.
+
+Racket BC [CGC] is primarily used for bootstrapping Racket BC [3M].  It may
+also be used for embedding applications without the annotations needed in C
+code to use the 3M garbage collector.")))
+
+
+(define-public racket-bootstrap-chez-bootfiles
+  (hidden-package
+   (package
+     (inherit racket-minimal)
+     (name "racket-bootstrap-chez-bootfiles")
+     (inputs `())
+     (native-inputs
+      `(("racket" ,(if (%current-target-system)
+                       racket-minimal
+                       racket-minimal-bc-3m))
+        ("stex" ,@(assoc-ref (package-native-inputs chez-scheme) "stex"))
+        ("nanopass" ,@(assoc-ref (package-native-inputs chez-scheme)
+                                 "nanopass"))))
+     (arguments
+      `(#:phases
+        (modify-phases %standard-phases
+          (add-after 'unpack 'unpack-nanopass+stex
+            (lambda* (#:key inputs native-inputs #:allow-other-keys)
+              (with-directory-excursion "racket/src/ChezScheme"
+                ,unpack-nanopass+stex)
+              #t))
+          (delete 'configure)
+          (delete 'patch-generated-file-shebangs)
+          (replace 'build
+            (lambda* (#:key inputs outputs #:allow-other-keys)
+              (with-directory-excursion "racket/src/ChezScheme"
+                (invoke (string-append (assoc-ref inputs "racket")
+                                       "/bin/racket")
+                        "rktboot/main.rkt"
+                        "--dest" (assoc-ref outputs "out")))
+              #t))
+          (delete 'check)
+          (delete 'install))))
+     (synopsis "Chez Scheme bootfiles bootstrapped by Racket")
+     (description "Chez Scheme is a self-hosting compiler: building it
+requires ``bootfiles'' containing the Scheme-implemented portions compiled for
+the current platform.  (Chez can then cross-compile bootfiles for all other
+supported platforms.)
+
+The Racket package @code{cs-bootstrap} (part of the main Racket Git
+repository) implements enough of a Chez Scheme simulation to load the Chez
+Scheme compiler purely from source into Racket and apply the compiler to
+itself, thus bootstrapping Chez Scheme.  Bootstrapping takes about 10 times as
+long as using an existing Chez Scheme, but @code{cs-bootstrap} supports Racket
+7.1 and later, including the Racket BC variant.
+
+Note that the generated bootfiles are specific to Racket's fork of Chez
+Scheme, and @code{cs-bootstrap} does not currently support building upstream
+Chez Scheme.")
+     (license (list asl2.0)))))
+
+
+(define %installer-mirrors
+  ;; Source:
+  ;; https://github.com/racket/racket-lang-org/blob/master/download/data.rkt#L58
+  ;; Matthew Flatt says: "note that many are commented out"
+  ;; INVARIANT: End with a trailing "/"!
+  '("https://mirror.racket-lang.org/installers/"
+    "https://www.cs.utah.edu/plt/installers/"
+    "https://plt.cs.northwestern.edu/racket-mirror/"
+    "https://mirror.csclub.uwaterloo.ca/racket/racket-installers/"
+    ;; Universität Tübingen is using a self-signed HTTPS certificate:
+    "http://mirror.informatik.uni-tuebingen.de/mirror/racket/"
+    "https://racket.infogroep.be/"
+    ))
+
+(define %main-repo-main-distribution-pkgs
+  ;; These are the packages developed in the main Racket Git repository
+  ;; that are part of the main distribution.
+  '("at-exp-lib"
+    "base"
+    "compiler-lib"
+    ;; NOT "compiler-test"
+    "compiler"
+    "net-doc"
+    "net-lib"
+    ;; NOT "net-test"
+    "net"
+    ;; NOT "plt-services"
+    ;; NOT "racket-benchmarks"
+    ;; NOT "racket-build-guide"
+    "racket-doc"
+    "racket-index"
+    "racket-lib"
+    ;; NOT "racket-test-core"
+    ;; NOT "racket-test-extra"
+    ;; NOT "racket-test"
+    "zo-lib"))
 
 
 (define-public racket
@@ -177,7 +415,7 @@ DrRacket IDE, are not included.")
     (version (package-version racket-minimal)) ; needed for origin uri to work
     (source
      (origin
-       (inherit (package-source racket-minimal))
+       (method url-fetch)
        (uri (map (lambda (base)
                    (string-append base version "/racket-src.tgz"))
                  %installer-mirrors))
@@ -200,36 +438,10 @@ DrRacket IDE, are not included.")
                           "README"
                           "src"))
               ;; unbundle package sources included elsewhere
-              (define (substitute/delete file pattern)
-                (substitute
-                 file
-                 (list (cons pattern
-                             (lambda (line matches)
-                               ;; must match exactly once
-                               (match matches
-                                 ((m)
-                                  (string-append (match:prefix m)
-                                                 (match:suffix m)))))))))
-              (define (unbundle-pkg pkg)
-                (define quoted-pkg (regexp-quote pkg))
-                (with-directory-excursion "share"
-                  (substitute/delete
-                   "links.rktd"
-                   (string-append
-                    "[(][^()]+[(]#\"pkgs\" #\""
-                    quoted-pkg
-                    "\"[)][)]"))
-                  (with-directory-excursion "pkgs"
-                    (substitute/delete
-                     "pkgs.rktd"
-                     (string-append
-                      "[(]\""
-                      quoted-pkg
-                      "\" \\. #s[(]"
-                      "(pkg-info|[(]sc-pkg-info pkg-info 3[)])"
-                      " [(][^()]+[)] [^()]+[)][)]"))
-                    (delete-file-recursively pkg))))
-              (unbundle-pkg "racket-lib"))))))
+              (with-directory-excursion "share/pkgs"
+                (for-each delete-file-recursively
+                          '#+%main-repo-main-distribution-pkgs))
+              #t)))))
     (inputs
      `(("cairo" ,cairo)
        ("fontconfig" ,fontconfig)
@@ -247,22 +459,32 @@ DrRacket IDE, are not included.")
        ("libedit" ,libedit)))
     (native-inputs
      `(("racket" ,racket-minimal)
-       ("extend-layer" ,extend-layer)))
+       ("extend-layer" ,extend-layer)
+       ("main-repo" ,(package-source racket-minimal))))
     (arguments
      `(#:phases
        (modify-phases %standard-phases
          (add-before 'configure 'unpack-packages
-           (lambda* (#:key native-inputs inputs outputs #:allow-other-keys)
-             (let ((racket (assoc-ref (or native-inputs inputs) "racket"))
-                   (prefix (assoc-ref outputs "out")))
-               (mkdir-p (string-append prefix "/share/racket/pkgs"))
-               (copy-recursively
-                "share/links.rktd"
-                (string-append prefix "/share/racket/links.rktd"))
-               (copy-recursively
-                "share/pkgs"
-                (string-append prefix "/share/racket/pkgs"))
-               #t)))
+           (let ((unpack (assoc-ref %standard-phases 'unpack)))
+             (lambda* (#:key  native-inputs inputs outputs #:allow-other-keys)
+               (let* ((racket (assoc-ref (or native-inputs inputs) "racket"))
+                      (prefix (assoc-ref outputs "out"))
+                      (pkgs-dir (string-append prefix "/share/racket/pkgs")))
+                 (mkdir-p pkgs-dir)
+                 (copy-recursively
+                  "share/links.rktd"
+                  (string-append prefix "/share/racket/links.rktd"))
+                 (copy-recursively "share/pkgs" pkgs-dir)
+                 ;; NOTE: unpack changes the working directory
+                 (unpack #:source (assoc-ref (or native-inputs inputs)
+                                             "main-repo"))
+                 (for-each (lambda (pkg)
+                             (define dest (string-append pkgs-dir "/" pkg))
+                             (mkdir-p dest)
+                             (copy-recursively (string-append "pkgs/" pkg)
+                                               dest))
+                           ',%main-repo-main-distribution-pkgs)
+                 #t))))
          (replace 'configure
            (lambda* (#:key native-inputs inputs outputs #:allow-other-keys)
              (let ((racket (assoc-ref (or native-inputs inputs) "racket"))
-- 
2.30.2

[bug#49280] [PATCH v2 3/3] gnu: racket-minimal: Bootstrap from C.

[Philip McGrath]

The Racket maintainers have just disclosed that Racket 8.2 fixes a 
security vulnerability, CVE-2021-32773.

Official advisory: 
https://github.com/racket/racket/security/advisories/GHSA-cgrw-p7p7-937c

Mailing-list thread: 
https://groups.google.com/g/racket-users/c/BNVybdBZ8wk/m/xJK5_nkDAgAJ

-Philip

[bug#49280] [PATCH v2 1/3] gnu: racket: Update to 8.2.

[Leo Prikler]

Am Montag, den 19.07.2021, 02:31 -0400 schrieb Philip McGrath:
> * gnu/packages/racket.scm (racket-minimal, racket): Update to 8.2.
> * gnu/packages/racket.scm (racket-minimal)[#:arguments]: Fix
> patch-config.rktd-lib-search-dirs phase. When a config.rktd file
> doesn't contain an entry for `lib-search-dirs`, the default is
> equivalent to `'(#f)`, not `'()`.
What is the point of this value?  Can we use a (sequence of) string(s)
in its stead?
> ---
>  gnu/packages/racket.scm | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
> index 2d606071fe..6b2a011d51 100644
> --- a/gnu/packages/racket.scm
> +++ b/gnu/packages/racket.scm
> @@ -64,14 +64,14 @@
>  (define-public racket-minimal
>    (package
>      (name "racket-minimal")
> -    (version "8.1")            ; note: remember to also update
> racket!
> +    (version "8.2")            ; note: remember to also update
> racket!
>      (source
>       (origin
>         (method url-fetch)
>         (uri (map (lambda (base)
>                     (string-append base version "/racket-minimal-
> src.tgz"))
>                   %installer-mirrors))
> -       (sha256
> "04zzqybpxss50n1jrwwq98539gw0y0ygpw9civl2sq3s4ww7m8l3")
> +       (sha256
> "13qfg56w554vdj5iwa8lpacy83s7bzhhyr44pjns68mkhj69ring")
>         (patches (search-patches
>                   "racket-sh-via-rktio.patch"))))
>      (home-page "https://racket-lang.org")
> @@ -125,7 +125,7 @@
>                                    'lib-search-dirs
>                                    (lambda (dirs)
>                                      (append dirs extra-lib-search-
> dirs))
> -                                  null)
> +                                  '(#f))
>                                   #:exists 'truncate/replace
>                                   file)))
>                      "--"
> @@ -183,7 +183,7 @@ DrRacket IDE, are not included.")
>                   %installer-mirrors))
>         (sha256
>          (base32
> -         "0xdqwrwm604bbnr97h75dps2ixxz2svlw0fn0f674bn04dcfd60f"))))
> +         "10sgzsraxzxp1k2y2wvz8rcjwvhbcd6k72l9lyqr34yazlwfdz26"))))
>      (inputs
>       `(;; sqlite and libraries for `racket/draw' are needed to build
> the doc.
>         ("cairo" ,cairo)
Otherwise LGTM.

[bug#49280] [PATCH v2 1/3] gnu: racket: Update to 8.2.

[Philip McGrath]

Hi,

On 7/19/21 3:46 PM, Leo Prikler wrote:
> Am Montag, den 19.07.2021, 02:31 -0400 schrieb Philip McGrath:
>> * gnu/packages/racket.scm (racket-minimal, racket): Update to 8.2.
>> * gnu/packages/racket.scm (racket-minimal)[#:arguments]: Fix
>> patch-config.rktd-lib-search-dirs phase. When a config.rktd file
>> doesn't contain an entry for `lib-search-dirs`, the default is
>> equivalent to `'(#f)`, not `'()`.
> What is the point of this value?  Can we use a (sequence of) string(s)
> in its stead?

As you'd probably guess, `lib-search-dirs` and other `-search-dirs` 
"config.rktd" entries specify search paths. The `#f` value is used to 
specify the point at which the default search path should be spliced 
into the list: a configuration file can ignore the default altogether or 
exercise fine-grained control over the search order. Using `#f` also 
helps to maintain something closer to a single point of control, rather 
than hard-code the same string constants in several places.

Most importantly, the default value is not always a constant: for 
example, command-line flags and Racket parameters control whether 
user-specific paths are included.

(For `lib-search-dirs` in particular, it's also worth noting that these 
are Racket-specific search directories: it does not control the use of 
OS-level defaults for e.g. `dlopen`.)

The problems with omitting `#f` from `lib-search-dirs` were not very 
noticeable until the patch to start using layered and tethered 
installations. For a more obvious example, if `catalogs` is `'()`, `raco 
pkg` won't consult any package catalogs, whereas `'(#f)` will cause it 
to use the default catalogs.

The documentation for these configuration options is here: 
https://docs.racket-lang.org/raco/config-file.html#(idx._(gentag._70._(lib._scribblings%2Fraco%2Fraco..scrbl)))

(As of this writing, the docs on the website are still at 8.1---the 
whole package catalog is rebuilt after a new Racket release, which takes 
a little while to finish. You could alternatively apply this patch and 
run `lynx "file://"$(./pre-inst-env guix build 
racket)"/share/doc/racket/raco/config-file.html#(idx._(gentag._70._(lib._scribblings%2Fraco%2Fraco..scrbl)))"`.)


-Philip

[bug#49280] [PATCH v2 1/3] gnu: racket: Update to 8.2.

[Leo Prikler]

Am Montag, den 19.07.2021, 17:46 -0400 schrieb Philip McGrath:
> Hi,
> 
> On 7/19/21 3:46 PM, Leo Prikler wrote:
> > Am Montag, den 19.07.2021, 02:31 -0400 schrieb Philip McGrath:
> > > * gnu/packages/racket.scm (racket-minimal, racket): Update to
> > > 8.2.
> > > * gnu/packages/racket.scm (racket-minimal)[#:arguments]: Fix
> > > patch-config.rktd-lib-search-dirs phase. When a config.rktd file
> > > doesn't contain an entry for `lib-search-dirs`, the default is
> > > equivalent to `'(#f)`, not `'()`.
> > What is the point of this value?  Can we use a (sequence of)
> > string(s)
> > in its stead?
> 
> As you'd probably guess, `lib-search-dirs` and other `-search-dirs` 
> "config.rktd" entries specify search paths. The `#f` value is used
> to 
> specify the point at which the default search path should be spliced 
> into the list: a configuration file can ignore the default altogether
> or 
> exercise fine-grained control over the search order. Using `#f` also 
> helps to maintain something closer to a single point of control,
> rather 
> than hard-code the same string constants in several places.
Okay, but for this specific config, we could still splice #f ourselves
(particularly to also get full store paths), or can we not thanks to
the non-constant nature of #f?

> Most importantly, the default value is not always a constant: for 
> example, command-line flags and Racket parameters control whether 
> user-specific paths are included.
How exactly would this play out?  Would for example one version of #f
contain all of the user-installed packages in ~/.guix-profile whereas
the other would only contain racket's own path?

> (For `lib-search-dirs` in particular, it's also worth noting that
> these are Racket-specific search directories: it does not control the
> use of OS-level defaults for e.g. `dlopen`.)
Perhaps a confusing naming scheme, but okay.

> The problems with omitting `#f` from `lib-search-dirs` were not very 
> noticeable until the patch to start using layered and tethered 
> installations. For a more obvious example, if `catalogs` is `'()`,
> `raco pkg` won't consult any package catalogs, whereas `'(#f)` will
> cause it to use the default catalogs.
Okay, but `catalogs` is not `lib-search-dirs`, is it?  I'd assume
`'(#f)` is roughly equivalent to `'("default")` or `'("@spam@" "@ham@"
"@eggs@")` for some configure-time constant spam, ham and eggs.  Or
does the command-line flag variability apply to catalogs as well?

> The documentation for these configuration options is here: 
> https://docs.racket-lang.org/raco/config-file.html#(idx._(gentag._70._(lib._scribblings%2Fraco%2Fraco..scrbl)))
> 
> (As of this writing, the docs on the website are still at 8.1---the 
> whole package catalog is rebuilt after a new Racket release, which
> takes a little while to finish. You could alternatively apply this
> patch and run `lynx "file://"$(./pre-inst-env guix build 
> racket)"/share/doc/racket/raco/config-
> file.html#(idx._(gentag._70._(lib._scribblings%2Fraco%2Fraco..scrbl))
> )"`.)
I think more important than the semantics of the configuration file is
the purpose of this particular thing.  Is it a template from which
other stuff is derived?  In that case, we might want to use #f as you
did.  Is it instead used to build stuff in the racket package?  Then
we'd need to substitute it imo.

Regards

[bug#49280] [PATCH v2 1/3] gnu: racket: Update to 8.2.

[Philip McGrath]

On 7/20/21 5:40 AM, Leo Prikler wrote:
> Am Montag, den 19.07.2021, 17:46 -0400 schrieb Philip McGrath:
>> As you'd probably guess, `lib-search-dirs` and other `-search-dirs`
>> "config.rktd" entries specify search paths. The `#f` value is used
>> to
>> specify the point at which the default search path should be spliced
>> into the list: a configuration file can ignore the default altogether
>> or
>> exercise fine-grained control over the search order. Using `#f` also
>> helps to maintain something closer to a single point of control,
>> rather
>> than hard-code the same string constants in several places.
> Okay, but for this specific config, we could still splice #f ourselves
> (particularly to also get full store paths), or can we not thanks to
> the non-constant nature of #f?
> 
>> Most importantly, the default value is not always a constant: for
>> example, command-line flags and Racket parameters control whether
>> user-specific paths are included.
> How exactly would this play out?  Would for example one version of #f
> contain all of the user-installed packages in ~/.guix-profile whereas
> the other would only contain racket's own path?
> 
>> (For `lib-search-dirs` in particular, it's also worth noting that
>> these are Racket-specific search directories: it does not control the
>> use of OS-level defaults for e.g. `dlopen`.)
> Perhaps a confusing naming scheme, but okay.

The short answer is that I don't think including #f is causing any 
problems, whereas trying not to include it seems likely to cause a 
variety of problems.

I'll try to explain more clearly.

It might be more useful to look at the second patch in the series, which 
uses the "extend-layer.rkt" script to generate a "config.rkt" file for 
the `racket` package, and especially the third patch, which replaces 
this code completely for the `racket-minimal` package:

On 7/19/21 2:31 AM, Philip McGrath wrote:
> +         (add-before 'configure 'initialize-config.rktd
>              (lambda* (#:key inputs #:allow-other-keys)
> -             (chdir "src")
> +             (define (write-racket-hash alist)
> +               ;; inside must use dotted pair notation
> +               (display "#hash(")
> +               (for-each (match-lambda
> +                           ((k . v)
> +                            (format #t "(~s . ~s)" k v)))
> +                         alist)
> +               (display ")\n"))
> +             (mkdir-p "racket/etc")
> +             (with-output-to-file "racket/etc/config.rktd"
> +               (lambda ()
> +                 (write-racket-hash
> +                  `((lib-search-dirs
> +                     . (#f ,@(map (lambda (lib)
> +                                    (string-append (assoc-ref inputs 
lib)
> +                                                   "/lib"))
> +                                  '("openssl"
> +                                    "sqlite"))))
> +                    (catalogs
> +                     . (,(string-append
> +                          "https://download.racket-lang.org/releases/"
> +                          ,version
> +                          "/catalog/")
> +                        #f))))))
>                #t))

This code creates a template "config.rktd" file used in the build 
process: the distributed source tarballs contain such a template 
already, which is why we didn't need explicitly configure `catalogs` to 
add the release-pinned package catalog until this change. It is added 
before the `#f` so that the release catalog is checked before the 
default catalogs (which point to the latest sources). For 
`lib-search-dirs`, on the other hand, we want Racket-specific library 
paths to be tried first, and indeed for layers of a Racket installation 
to be searched in order, so `#f` is at the head of the list.

The Racket build process extends the template "config.rktd" file based 
on build options like the `--prefix` passed to `configure`. For example, 
it configures `lib-dir` to "lib/racket" within the store output 
directory. (It would be incorrect to set those values in the template 
"config.rktd" file because it is used in the build process before 
installation.)

The `#f` entry in `lib-search-dirs` is usually replaced by a 
user-specific path like "/home/philip/.local/share/racket/8.1/lib" and 
the installation-wide path specified by the `lib-dir` key, unless one or 
both are changed. Omitting the `#f` entry means that neither of paths 
are ever included. I don't know of any real-life circumstance in which 
one would want such a "config.rktd" file. In particular, missing `#f` 
entries creates problems for layered installations, which use these 
search paths to find earlier layers.

There are some other configuration possibilities we may want to explore 
as Guix's support for Racket packages improves, such as "addon" 
tethering and customizing the "installation name" or "build stamp". 
However, this patch series does not attempt to change how Guix's Racket 
packages work, other than correcting the error I introduced in 
<https://issues.guix.gnu.org/47180>. Racket installed via Guix has the 
same behavior in this respect as Racket installed via Debian or other 
package managers, and that's a way of using Racket I think Guix will 
want to continue to support.

-Philip

[bug#49280] [PATCH v2 1/3] gnu: racket: Update to 8.2.

[Leo Prikler]

Hi Philip,

Am Sonntag, den 25.07.2021, 04:22 -0400 schrieb Philip McGrath:
> The short answer is that I don't think including #f is causing any 
> problems, whereas trying not to include it seems likely to cause a 
> variety of problems.

That short explanation imo doesn't adequately summarize the long one. 
Rest assured, the long explanation gives us a good reason to use #f as
you did, it's just that the way to reach this point of understanding
appears a bit of a long one.

> I'll try to explain more clearly.
> 
> It might be more useful to look at the second patch in the series,
> which 
> uses the "extend-layer.rkt" script to generate a "config.rkt" file
> for 
> the `racket` package, and especially the third patch, which replaces 
> this code completely for the `racket-minimal` package:
> 
> On 7/19/21 2:31 AM, Philip McGrath wrote:
> > +         (add-before 'configure 'initialize-config.rktd
> >              (lambda* (#:key inputs #:allow-other-keys)
> > -             (chdir "src")
> > +             (define (write-racket-hash alist)
> > +               ;; inside must use dotted pair notation
> > +               (display "#hash(")
> > +               (for-each (match-lambda
> > +                           ((k . v)
> > +                            (format #t "(~s . ~s)" k v)))
> > +                         alist)
> > +               (display ")\n"))
> > +             (mkdir-p "racket/etc")
> > +             (with-output-to-file "racket/etc/config.rktd"
> > +               (lambda ()
> > +                 (write-racket-hash
> > +                  `((lib-search-dirs
> > +                     . (#f ,@(map (lambda (lib)
> > +                                    (string-append (assoc-ref
> > inputs 
> lib)
> > +                                                   "/lib"))
> > +                                  '("openssl"
> > +                                    "sqlite"))))
> > +                    (catalogs
> > +                     . (,(string-append
> > +                          "
> > https://download.racket-lang.org/releases/"
> > +                          ,version
> > +                          "/catalog/")
> > +                        #f))))))
> >                #t))
This is perhaps a somewhat noobish question, but why must we use dotted
pair notation here?  To me personally, reading '(a . (b c)) is
confusing as it could more clearly be written as '(a b c).  Is this a
Racket convention?

> This code creates a template "config.rktd" file used in the build 
> process: the distributed source tarballs contain such a template 
> already, which is why we didn't need explicitly configure `catalogs`
> to add the release-pinned package catalog until this change. It is
> added before the `#f` so that the release catalog is checked before
> the default catalogs (which point to the latest sources). For 
> `lib-search-dirs`, on the other hand, we want Racket-specific
> library paths to be tried first, and indeed for layers of a Racket
> installation to be searched in order, so `#f` is at the head of the
> list.
> 
> The Racket build process extends the template "config.rktd" file
> based on build options like the `--prefix` passed to `configure`. For
> example, it configures `lib-dir` to "lib/racket" within the store
> output directory. (It would be incorrect to set those values in the
> template "config.rktd" file because it is used in the build process
> before installation.)
> 
> The `#f` entry in `lib-search-dirs` is usually replaced by a 
> user-specific path like "/home/philip/.local/share/racket/8.1/lib"
> and the installation-wide path specified by the `lib-dir` key, unless
> one or both are changed. Omitting the `#f` entry means that neither
> of paths are ever included. I don't know of any real-life
> circumstance in which one would want such a "config.rktd" file. In
> particular, missing `#f` entries creates problems for layered
> installations, which use these search paths to find earlier layers.
> 
> There are some other configuration possibilities we may want to
> explore as Guix's support for Racket packages improves, such as
> "addon" tethering and customizing the "installation name" or "build
> stamp". However, this patch series does not attempt to change how
> Guix's Racket packages work, other than correcting the error I
> introduced in <https://issues.guix.gnu.org/47180>;. Racket installed
> via Guix has the same behavior in this respect as Racket installed
> via Debian or other package managers, and that's a way of using
> Racket I think Guix will want to continue to support.
To attempt a better summary: Specifying `#f' will allow Racket to
search for user-specific libraries etc. (in
$XDG_DATA_HOME/racket/$RACKET_VERSION) in addition to "system-specific" 
libraries stored in $HOME/.guix-profile, am I correct?

If so, then yes, doing that is absolutely fine (you could compare it to
how Emacs users can still install stuff via ELPA).

Regards,

[bug#49280] [PATCH v2 1/3] gnu: racket: Update to 8.2.

[Philip McGrath]

Hi,

On 7/25/21 9:03 AM, Leo Prikler wrote:
>> On 7/19/21 2:31 AM, Philip McGrath wrote:
>>> +         (add-before 'configure 'initialize-config.rktd
>>>               (lambda* (#:key inputs #:allow-other-keys)
>>> -             (chdir "src")
>>> +             (define (write-racket-hash alist)
>>> +               ;; inside must use dotted pair notation
>>> +               (display "#hash(")
>>> +               (for-each (match-lambda
>>> +                           ((k . v)
>>> +                            (format #t "(~s . ~s)" k v)))
>>> +                         alist)
>>> +               (display ")\n"))
>>> +             (mkdir-p "racket/etc")
>>> +             (with-output-to-file "racket/etc/config.rktd"
>>> +               (lambda ()
>>> +                 (write-racket-hash
>>> +                  `((lib-search-dirs
>>> +                     . (#f ,@(map (lambda (lib)
>>> +                                    (string-append (assoc-ref
>>> inputs
>> lib)
>>> +                                                   "/lib"))
>>> +                                  '("openssl"
>>> +                                    "sqlite"))))
>>> +                    (catalogs
>>> +                     . (,(string-append
>>> +                          "
>>> https://download.racket-lang.org/releases/"
>>> +                          ,version
>>> +                          "/catalog/")
>>> +                        #f))))))
>>>                 #t))
> This is perhaps a somewhat noobish question, but why must we use dotted
> pair notation here?  To me personally, reading '(a . (b c)) is
> confusing as it could more clearly be written as '(a b c).  Is this a
> Racket convention?

Yes, the Racket reader requires that hash table literals use dotted pair 
notation, e.g. #hash((a . (b c))(d . 2)). (Additionally, hand-written 
Racket would often use square brackets for the key–value pairs.) I think 
it's ultimately for historical reasons, but there are various 
subtleties: under `quasiquote`, for example, the value position can 
contain `unquote`, but the key position cannot.

Of course, that doesn't necessarily mean the Guile code must use dotted 
pairs, but, at least as a Racketeer, I found writing these s-expressions 
with a bit of a "Racket accent" helped me to remember that this code is 
generating Racket.

> To attempt a better summary: Specifying `#f' will allow Racket to
> search for user-specific libraries etc. (in
> $XDG_DATA_HOME/racket/$RACKET_VERSION) in addition to "system-specific"
> libraries stored in $HOME/.guix-profile, am I correct?
> 
> If so, then yes, doing that is absolutely fine (you could compare it to
> how Emacs users can still install stuff via ELPA).

Yes, that's a much better summary.

(Pedantically, the user-specific path could be different, depending on 
the configured installation name and other things.)

The other issue is that I believe *not* specifying `#f` for 
`racket-minimal` would break the build of `racket` starting in the 
second patch, when the main distribution becomes a layer chaining to 
`racket-minimal`. At least, it broke some examples of layered 
installation I tried, which is how I figured out I'd done the wrong 
thing in <https://issues.guix.gnu.org/47180>. If that were the only 
issue, there might be some other workaround, but it's worth bearing in 
mind, particularly because the problems caused by buggy "config.rktd" 
files can be subtle and easy to miss.

-Philip

bug#49280: [PATCH v2 0/3] gnu: racket: Update to 8.2. Bootstrap from C.

[Ludovic Courtès]

Hi,

Finally pushed this second series, minus ‘with-imported-modules’.

  db2c7e70ad gnu: racket-minimal: Bootstrap from C.
  65bad4d036 gnu: racket: Unbundle racket-minimal.
  1ae95ebcdd gnu: racket: Update to 8.2.

Thank you!

Ludo’.

[bug#49280] [PATCH v2 0/3] gnu: racket: Update to 8.2. Bootstrap from C.

[Ludovic Courtès]

Hi Philip,

Sorry for the delay and thanks for the explanations!  Comments/answers
follow.

Philip McGrath <philip@philipmcgrath.com> skribis:

> On 7/8/21 5:25 PM, Ludovic Courtès wrote:
>> Philip McGrath <philip@philipmcgrath.com> skribis:
>> 
>>> * gnu/packages/racket.scm (racket-next-minimal,racket-next): New variables.

[...]

>> For this there’s already a documented convention (info "(guix)
>> Package
>> Naming"), although part of it is undocumented.  The prefix would rather
>> be “racket-” to match what we do with other packages–“ghc-”, “ocaml-”,
>> “guile-”, and so forth.
>
> I wrote these as statements in the hope of eliciting any disagreement :)
>
> The problem I see with using just “racket-” as the prefix is the
> potential for collisions, especially because Racket uses a lot of the 
> namespace: for example, "_" is a useful example package for testing
> package issues, and I maintain the "_-exp" package. There don't seem
> to be Racket packages named "minimal" or "next" right now, but they
> seem reasonably likely to be used in the future, and Guix likewise may
> want to add packages that don't correspond directly to a single
> Racket-level package. (In fact, I think this may be necessary to build
> Racket packages with mutually recursive dependencies.) Other Racket
> package names that I think might be less confusing if prefixed with 
> “racket-pkg-” include "base", "racket-lib", "unstable", "profile",
> "make", "data", "images", "compiler", "compatibility", "pkg-build",
> and "main-distribution".

I would not worry too much about name collisions.  After all, we have
18K packages and a great potential for collisions already.  :-)
We can deal with a hypothetical “next” Racket package when it comes into
existence.

> But we don't need to resolve this now, and maybe actually implementing
> that support will clarify what issues really do or don't exist. I will 
> just remove this whole comment for now, since I don't need to make a
> choice between "racket-next-minimal" and "racket-minimal-next".

Either way is fine with me.  :-)

> In addition to bootstrapping, there are three reasons I know of to
> want Racket BC:
>
>  1. The BC and CS implementations have different C APIs, so some
>     low-level code may support BC but not CS. But this isn't usually a
>     good reason. Racket packages should support both implementations.
>     Embedding applications ideally would also be portable: if it's
>     only feasible to support one implementation, it should be CS.
>
>  2. Comparing the BC and CS implementations can be useful for testing
>     and debugging, both for packages that use the FFI and when hacking
>     on the Racket runtime system itself.
>
>  3. Most importantly, BC supports some architectures that CS does not.
>
> In particular, Racket CS does not (yet) support ppc64le, which Racket
> BC does support. The recommendation to packagers, and what Debian
> does, is
> to explicitly use BC on platforms without CS support:
> https://github.com/racket/racket/issues/3773#issuecomment-832935403
>
> I'm not sure what the most idiomatic way to do this is in Guix.

Once we have a ‘racket-build-system’, it could pick the right Racket as
a function of the target system.

Otherwise we could do a trick of the sort we have for ‘pkg-config’, but
I’d rather avoid that.

Thanks,
Ludo’.

[bug#49280] References to unversioned source tarballs

[Ludovic Courtès]

Philip McGrath <philip@philipmcgrath.com> skribis:

>>> +       (uri (string-append %pre-release-installers
>>> +                           "racket-src.tgz"))))))
>> Do I get it right that *-src.tgz are not versioned?  That they’re
>> updated in place regularly?
>> In that case, we cannot refer to them in a package definition since
>> the
>> hash is bound to become stale.
>> What we could do is refer to, say,
>> <https://pre-release.racket-lang.org/installers/racket-8.1.900.1-src.tgz>.
>> However, I suspect this file would vanish fairly quickly from the web
>> site, which is not okay either.
>> I’m not sure what a good solution would be.  WDYT?

[...]

> For now, I will avoid the problem by just not dealing with "-next" variants.

I just realized that the problem already exists right now, with ‘racket’
and ‘racket-minimal’ referring to ‘racket-src.tgz’.

I think we should fix it after this series, possibly by using
‘git-fetch’ instead.

Ludo’.

[bug#49280] References to unversioned source tarballs

[Philip McGrath]

On 7/30/21 5:31 PM, Ludovic Courtès wrote:
> Philip McGrath <philip@philipmcgrath.com> skribis:
> 
>>>> +       (uri (string-append %pre-release-installers
>>>> +                           "racket-src.tgz"))))))
>>> Do I get it right that *-src.tgz are not versioned?  That they’re
>>> updated in place regularly?
>>> In that case, we cannot refer to them in a package definition since
>>> the
>>> hash is bound to become stale.
>>> What we could do is refer to, say,
>>> <https://pre-release.racket-lang.org/installers/racket-8.1.900.1-src.tgz>.
>>> However, I suspect this file would vanish fairly quickly from the web
>>> site, which is not okay either.
>>> I’m not sure what a good solution would be.  WDYT?
> 
> [...]
> 
>> For now, I will avoid the problem by just not dealing with "-next" variants.
> 
> I just realized that the problem already exists right now, with ‘racket’
> and ‘racket-minimal’ referring to ‘racket-src.tgz’.
> 
> I think we should fix it after this series, possibly by using
> ‘git-fetch’ instead.

I think it is actually ok, because the the URIs are formed by:
```
        (uri (map (lambda (base)
                    (string-append base version "/racket-src.tgz"))
                  %installer-mirrors))
```
so the version is present as a path element, just not in the file name, 
e.g. <https://mirror.racket-lang.org/installers/8.2/racket-src.tgz>. 
These are definitely stable URLs relied on by a variety of tools, e.g. 
for CI.

An equivalent tarball is also available at 
<https://mirror.racket-lang.org/installers/8.2/racket-8.2-src.tgz> (and 
likewise for the other mirrors), but most tools prefer the "versionless 
path" URLs so that code manipulating the downloaded artifacts doesn't 
have to deal with version numbers.

(Download sites like these are created by the "distro-build" Racket 
package and have a well-known structure, though the documentation is not 
all in one place. A good place to start would be 
<https://docs.racket-lang.org/racket-build-guide/distribute.html>, which 
is generated from the same Scribble source that generates "build.md" in 
the main Racket Git repository.)

-Philip

[bug#49280] [PATCH 3/4] gnu: racket-next: Unbundle racket-next-minimal.

[Philip McGrath]

This takes advantage of improvements since the Racket 8.1 release in support
for layered and tethered installation.

* gnu/packages/racket.scm (extend-layer): New private variable. This is a
script for configuring a new config-tethered layer chaining to an existing
Racket installation.
* gnu/packages/racket.scm (racket-next)[source](snippet): Unbundle
`racket-next-minimal`.
[inputs]: Stop inheriting from `racket`. Remove inputs that properly
beling to `racket-next-minimal`.
[native-inputs]: Add `racket-next-minimal` and `extend-layer`.
[arguments]: Stop inheriting from `racket`.
Add phase 'unpack-packages to move the sources and links file into place.
Replace 'configure phase using `extend-layer`.
Replace 'build phase using `raco setup`.
Delete 'install phase.
---
 gnu/packages/racket.scm | 202 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 201 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
index 363f19825b..cf0240be5c 100644
--- a/gnu/packages/racket.scm
+++ b/gnu/packages/racket.scm
@@ -272,4 +272,204 @@ languages such as Typed Racket, R5RS and R6RS Scheme, Algol 60, and Datalog.")))
         (base32
          "0ysvzgm0lx4b1p4k9balvcbvh2kapbfx91c9ls80ba062cd8y5qv"))
        (uri (string-append %pre-release-installers
-                           "racket-src.tgz"))))))
+                           "racket-src.tgz"))
+       (snippet
+        (with-imported-modules '((guix build utils)
+                                 (ice-9 match)
+                                 (ice-9 regex))
+          #~(begin
+              (use-modules (guix build utils)
+                           (ice-9 match)
+                           (ice-9 regex))
+              ;; unbundle minimal Racket
+              (for-each delete-file-recursively
+                        '("collects"
+                          "doc"
+                          "etc"
+                          "README"
+                          "src"))
+              ;; unbundle package sources included elsewhere
+              (define (substitute/delete file pattern)
+                (substitute
+                 file
+                 (list (cons pattern
+                             (lambda (line matches)
+                               ;; must match exactly once
+                               (match matches
+                                 ((m)
+                                  (string-append (match:prefix m)
+                                                 (match:suffix m)))))))))
+              (define (unbundle-pkg pkg)
+                (define quoted-pkg (regexp-quote pkg))
+                (with-directory-excursion "share"
+                  (substitute/delete
+                   "links.rktd"
+                   (string-append
+                    "[(][^()]+[(]#\"pkgs\" #\""
+                    quoted-pkg
+                    "\"[)][)]"))
+                  (with-directory-excursion "pkgs"
+                    (substitute/delete
+                     "pkgs.rktd"
+                     (string-append
+                      "[(]\""
+                      quoted-pkg
+                      "\" \\. #s[(]"
+                      "(pkg-info|[(]sc-pkg-info pkg-info 3[)])"
+                      " [(][^()]+[)] [^()]+[)][)]"))
+                    (delete-file-recursively pkg))))
+              (unbundle-pkg "racket-lib"))))))
+    (inputs
+     `(("cairo" ,cairo)
+       ("fontconfig" ,fontconfig)
+       ("glib" ,glib)
+       ("glu" ,glu)
+       ("gmp" ,gmp)
+       ("gtk+" ,gtk+)                   ; propagates gdk-pixbuf+svg
+       ("libjpeg" ,libjpeg-turbo)
+       ("libpng" ,libpng)
+       ("libx11" ,libx11)
+       ("mesa" ,mesa)
+       ("mpfr" ,mpfr)
+       ("pango" ,pango)
+       ("unixodbc" ,unixodbc)
+       ("libedit" ,libedit)))
+    (native-inputs
+     `(("racket" ,racket-next-minimal)
+       ("extend-layer" ,extend-layer)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-before 'configure 'unpack-packages
+           (lambda* (#:key native-inputs inputs outputs #:allow-other-keys)
+             (let ((racket (assoc-ref (or native-inputs inputs) "racket"))
+                   (prefix (assoc-ref outputs "out")))
+               (mkdir-p (string-append prefix "/share/racket/pkgs"))
+               (copy-recursively
+                "share/links.rktd"
+                (string-append prefix "/share/racket/links.rktd"))
+               (copy-recursively
+                "share/pkgs"
+                (string-append prefix "/share/racket/pkgs"))
+               #t)))
+         (replace 'configure
+           (lambda* (#:key native-inputs inputs outputs #:allow-other-keys)
+             (let ((racket (assoc-ref (or native-inputs inputs) "racket"))
+                   (prefix (assoc-ref outputs "out")))
+               (apply invoke
+                      (string-append racket "/bin/racket")
+                      (assoc-ref inputs "extend-layer")
+                      racket
+                      prefix
+                      (map
+                       (lambda (lib)
+                         (string-append (assoc-ref inputs lib) "/lib"))
+                       '("cairo"
+                         "fontconfig"
+                         "glib"
+                         "glu"
+                         "gmp"
+                         "gtk+"
+                         "libjpeg"
+                         "libpng"
+                         "libx11"
+                         "mesa"
+                         "mpfr"
+                         "pango"
+                         "unixodbc"
+                         "libedit")))
+               #t)))
+         (replace 'build
+           (lambda* (#:key native-inputs inputs outputs #:allow-other-keys)
+             (invoke (string-append (assoc-ref (or native-inputs inputs)
+                                               "racket")
+                                    "/bin/racket")
+                     "--config"
+                     (string-append (assoc-ref outputs "out")
+                                    "/etc/racket")
+                     "-l"
+                     "raco"
+                     "setup")
+             #t))
+         ;; we still don't have these:
+         (delete 'install))
+       #:tests? #f))))
+
+(define extend-layer
+  (scheme-file
+   "extend-layer.rkt"
+   `(module
+     extend-layer racket/base
+     (require racket/cmdline
+              racket/match
+              racket/file
+              racket/list
+              racket/pretty)
+     (define config-file-pth
+       "etc/racket/config.rktd")
+     (define (build-path-string . args)
+       (path->string (apply build-path args)))
+     (define rx:racket
+       ;; Guile's reader doesn't support #rx"racket"
+       (regexp "racket"))
+     (command-line
+      #:args (parent-layer prefix . lib-dir*)
+      (let* ([config
+              (for/fold
+               ([config (file->value (build-path parent-layer
+                                                 config-file-pth))])
+               ([spec (in-list
+                       '((lib-dir lib-search-dirs "lib/racket")
+                         (share-dir share-search-dirs "share/racket")
+                         (links-file
+                          links-search-files
+                          "share/racket/links.rktd")
+                         (pkgs-dir pkgs-search-dirs "share/racket/pkgs")
+                         (bin-dir bin-search-dirs "bin")
+                         (man-dir man-search-dirs "share/man")
+                         (doc-dir doc-search-dirs "share/doc/racket")
+                         (include-dir
+                          include-search-dirs
+                          "include/racket")))])
+               (match-define (list main-key search-key pth) spec)
+               (hash-set*
+                config
+                main-key
+                (build-path-string prefix pth)
+                search-key
+                (list* #f
+                       (hash-ref config
+                                 main-key
+                                 (build-path-string parent-layer pth))
+                       (filter values (hash-ref config search-key null)))))]
+             [config
+              (hash-set config
+                        'apps-dir
+                        (build-path-string prefix "share/applications"))]
+             [config
+              ;; place new foreign lib-search-dirs before old
+              ;; foreign dirs, but after Racket layers
+              (let-values
+                  ([(rkt extra)
+                    (partition (lambda (pth)
+                                 (or (not pth)
+                                     (regexp-match? rx:racket pth)))
+                               (hash-ref config 'lib-search-dirs))])
+                (hash-set config
+                          'lib-search-dirs
+                          (append rkt
+                                  lib-dir*
+                                  extra)))]
+             [bin-dir
+              (hash-ref config 'bin-dir)]
+             [config
+              (hash-set* config
+                         'config-tethered-console-bin-dir bin-dir
+                         'config-tethered-gui-bin-dir bin-dir)]
+             [new-config-pth
+              (build-path prefix config-file-pth)])
+        (make-parent-directory* new-config-pth)
+        (call-with-output-file*
+         new-config-pth
+         (lambda (out)
+           (pretty-write config out))))))))
-- 
2.30.2

[bug#49280] [PATCH 4/4] gnu: racket-next-minimal: Bootstrap from C.

[Philip McGrath]

This commit bootstraps the Racket compiler and runtime system from source,
including Racket CS as well as both variants of Racket BC. (One remaining
limitation is discussed in comments added to gnu/packages/racket.scm.)

In the process, it moves to building minimal Racket from the Git repository,
rather than the packaged source tarballs. The Git repository is slightly
better as the ``corresponding source'':

 1. A few packages especially closely tied to the Racket core implementation
    (like "compiler-lib", "base", and "racket-doc") are developed in the
    same Git repository. Having them use the same Guix origin, too, will
    help to keep them in sync.

 2. The top-level Makefile in the Git repository is an important
    ``script[] used to control compilation and installation.''
    In particular, it cooperates with the "distro-build" package to
    create the source tarballs and installers for a Racket distribution.
    (Racket supports a notion of custom distributions.)

 3. It is ``the preferred form ... for making modifications'' to the core
    Racket implementation.

Racket releases are tagged in the Git repository (e.g. "v8.1"). At the
beginning of each release cycle, a branch is created to stabalize a version
for extra testing. Active development happens on the "master" branch.

* gnu/packages/racket-next-minimal-sh-via-rktio.patch: New file, coppied
from racket-sh-via-rktio.patch to accomodate an extra directory layer.
When racket-next-minimal becomes racket-minimal, this version will be
the only one needed.
* gnu/local.mk (dist_patch_DATA): Add it.
* gnu/local/racket.scm (cfg-flag:sh-for-rktio, cfg-flag:enable-lt,
cfg-flag:enable-racket, unpack-nanopass+stex,
%main-repo-main-distribution-pkgs): New private variables.
* gnu/local/racket.scm (racket-next-minimal)[source]: Use Git.
[source](snippet): Unbundle nanopass, stex, and libffi.
[inputs]: List explicitly.
[native-inputs]: Use racket-next-bootstrap-bootfiles, plus its
dependencies (for Chez, plus a Racket for bootstrappig).
[arguments]: Revise extensively.
* gnu/local/racket.scm (racket-next-minimal-bc-3m,
racket-next-minimal-bc-cgc): New packages, hidden at least for now.
(racket-next-bootstrap-chez-bootfiles): Another new package, but this one
is especially likely to stay hidden.
* gnu/local/racket.scm (racket-next)[origin](snippet): Unbundle packages
developed in the main Git repository, but leave their links.rktd and
pkgs.rktd entries in place.
[native-inputs]: Add the main Racket Git repository.
[arguments](#:phases): Adjust 'unpack-packages to also unpack package
sources from the main Racket Git repository.
---
 gnu/local.mk                                  |   1 +
 .../racket-next-minimal-sh-via-rktio.patch    |  87 +++++
 gnu/packages/racket.scm                       | 357 +++++++++++++++---
 3 files changed, 396 insertions(+), 49 deletions(-)
 create mode 100644 gnu/packages/patches/racket-next-minimal-sh-via-rktio.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index 6b9202cba1..4ca35cf56a 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1671,6 +1671,7 @@ dist_patch_DATA =						\
   %D%/packages/patches/ripperx-missing-file.patch		\
   %D%/packages/patches/rpcbind-CVE-2017-8779.patch		\
   %D%/packages/patches/rtags-separate-rct.patch			\
+ %D%/packages/patches/racket-next-minimal-sh-via-rktio.patch    \
   %D%/packages/patches/racket-sh-via-rktio.patch		\
   %D%/packages/patches/remake-impure-dirs.patch			\
   %D%/packages/patches/retroarch-LIBRETRO_DIRECTORY.patch	\
diff --git a/gnu/packages/patches/racket-next-minimal-sh-via-rktio.patch b/gnu/packages/patches/racket-next-minimal-sh-via-rktio.patch
new file mode 100644
index 0000000000..6bc2ee8331
--- /dev/null
+++ b/gnu/packages/patches/racket-next-minimal-sh-via-rktio.patch
@@ -0,0 +1,87 @@
+From 3574b567c486d264d680a37586436c3b5a8cb978 Mon Sep 17 00:00:00 2001
+From: Philip McGrath <philip@philipmcgrath.com>
+Date: Thu, 4 Mar 2021 04:11:50 -0500
+Subject: [PATCH] patch rktio_process for "/bin/sh" on Guix
+
+Racket provides the functions `system` and `process`,
+which execute shell commands using `sh` (or `cmd` on Windows).
+Racket assumes that `sh` can be found at "/bin/sh",
+which is not necessarily true on Guix.
+
+This patch adds a special case for "/bin/sh" to `rktio_process`,
+the C function that implements the core of `system`, `process`,
+and related Racket functions.
+
+Guix should enable the special case by defining the C preprocessor
+macro `GUIX_RKTIO_PATCH_BIN_SH` with the path to `sh` in the store.
+If:
+
+    1. The `GUIX_RKTIO_PATCH_BIN_SH` macro is defined; and
+
+    2. `rktio_process` is called with the exact path "/bin/sh"; and
+
+    3. The path specified by `GUIX_RKTIO_PATCH_BIN_SH` does exists;
+
+then `rktio_process` will execute the file specified
+by `GUIX_RKTIO_PATCH_BIN_SH` instead of "/bin/sh".
+
+Compared to previous attempts to patch the Racket sources,
+making this change at the C level is both:
+
+    - More comprehensive: it catches all attempts to execute "/bin/sh",
+      without having to track down the source of every occurance; and
+
+    - Less intrusive: by guarding the special case with a C preprocessor
+      conditional and a runtime check that the file in the store exists,
+      we make it much less likely that it will "leak" out of Guix.
+---
+ racket/src/rktio/rktio_process.c | 21 ++++++++++++++++++++-
+ 1 file changed, 20 insertions(+), 1 deletion(-)
+
+diff --git a/racket/src/rktio/rktio_process.c b/racket/src/rktio/rktio_process.c
+index 89202436c0..465ebdd5c5 100644
+--- a/racket/src/rktio/rktio_process.c
++++ b/racket/src/rktio/rktio_process.c
+@@ -1224,12 +1224,14 @@ int rktio_process_allowed_flags(rktio_t *rktio)
+ /*========================================================================*/
+ 
+ rktio_process_result_t *rktio_process(rktio_t *rktio,
+-                                      const char *command, int argc, rktio_const_string_t *argv,
++                                      /* PATCHED for Guix (next line) */
++                                      const char *_guix_orig_command, int argc, rktio_const_string_t *argv,
+                                       rktio_fd_t *stdout_fd, rktio_fd_t *stdin_fd, rktio_fd_t *stderr_fd,
+                                       rktio_process_t *group_proc,
+                                       const char *current_directory, rktio_envvars_t *envvars,
+                                       int flags)
+ {
++  const char *command; /* PATCHED for Guix */
+   rktio_process_result_t *result;
+   intptr_t to_subprocess[2], from_subprocess[2], err_subprocess[2];
+   int pid;
+@@ -1255,6 +1257,23 @@ rktio_process_result_t *rktio_process(rktio_t *rktio,
+   int i;
+ #endif
+ 
++/* BEGIN PATCH for Guix */
++#if defined(GUIX_RKTIO_PATCH_BIN_SH)
++# define GUIX_AS_a_STR_HELPER(x) #x
++# define GUIX_AS_a_STR(x) GUIX_AS_a_STR_HELPER(x)
++  /* A level of indirection makes `#` work as needed: */
++  command =
++      ((0 == strcmp(_guix_orig_command, "/bin/sh"))
++       && rktio_file_exists(rktio, GUIX_AS_a_STR(GUIX_RKTIO_PATCH_BIN_SH)))
++      ? GUIX_AS_a_STR(GUIX_RKTIO_PATCH_BIN_SH)
++      : _guix_orig_command;
++# undef GUIX_AS_a_STR
++# undef GUIX_AS_a_STR_HELPER
++#else
++  command = _guix_orig_command;
++#endif
++/* END PATCH for Guix */
++
+   /* avoid compiler warnings: */
+   to_subprocess[0] = -1;
+   to_subprocess[1] = -1;
+-- 
+2.21.1 (Apple Git-122.3)
+
diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
index cf0240be5c..4e2d498364 100644
--- a/gnu/packages/racket.scm
+++ b/gnu/packages/racket.scm
@@ -32,7 +32,9 @@
   #:use-module (srfi srfi-1)
   #:use-module (ice-9 match)
   #:use-module (gnu packages)
+  #:use-module (gnu packages autotools)
   #:use-module (gnu packages bash)
+  #:use-module (gnu packages chez)
   #:use-module (gnu packages compression)
   #:use-module (gnu packages databases)
   #:use-module (gnu packages fontutils)
@@ -41,6 +43,7 @@
   #:use-module (gnu packages gtk)
   #:use-module (gnu packages image)
   #:use-module (gnu packages libedit)
+  #:use-module (gnu packages libffi)
   #:use-module (gnu packages multiprecision)
   #:use-module (gnu packages sqlite)
   #:use-module (gnu packages tls)
@@ -49,6 +52,7 @@
 ;; Commentary:
 ;;
 ;; Preliminary guidelines on naming things:
+;;
 ;;   - `racket` is the main package. It corresponds to `racket-minimal`
 ;;     with the Racket-level package "main-distribution" installed.
 ;;   - `racket-minimal` is Racket runtime system and core libraries:
@@ -74,6 +78,49 @@
 ;;     individually, we will need `racket-next-pkg-` for the packages
 ;;     that make up `racket-next`.
 ;;
+;; Here's how bootstrapping minimal Racket works:
+;;
+;;   - Racket BC [CGC] can be built with only a C compiler (except for
+;;     one caveat discussed below).
+;;   - Racket BC [3M] needs an existing Racket to run "xform",
+;;     which transforms its own C source code to add additional annotations
+;;     for the precise garbage collector.
+;;   - Racket CS needs (bootfiles for) Racket's fork of Chez Scheme.
+;;     It also needs an existing Racket to compile Racket-implemented
+;;     parts of the runtime system to R6RS libraries.
+;;   - Chez Scheme also needs bootfiles for itself, but Racket can simulate
+;;     enough of Chez Scheme to load Racket's fork of the Chez Scheme compiler
+;;     purely from source into Racket and apply the compiler to itself,
+;;     producing the needed bootfiles (albeit very slowly).
+;;     Any variant of Racket since version 7.1 can run the simulation.
+;;
+;; So, we build CGC to build 3M to build bootfiles and CS.
+;;
+;; One remaining bootstrapping limitation is that Racket's reader, module
+;; system, and macro expander are implemented in Racket. For Racket CS,
+;; they are compiled to R6RS libraries as discussed above. This note from the
+;; README file applies to all such subsystems:
+;;
+;;     The Racket version must be practically the same as the current Racket
+;;     verson, although it can be the Racket BC implementation (instead of
+;;     the Racket CS implementation).
+;;
+;;     Unlike Chez Scheme boot files, the files generated in "schemified"
+;;     are human-readable and -editable Scheme code. That provides a way
+;;     out of bootstrapping black holes, even without BC.
+;;
+;; However, other Racket subsystems implemented in Racket for Racket CS
+;; use older C implementations for Racket BC, whereas the reader, expander,
+;; and module system were completely replaced with the Racket implementation
+;; as of Racket 7.0.
+;;
+;; For Racket BC, the compiled "linklet" s-expressions (primitive modules)
+;; are embeded in C as a static string constant. Eventually, they are further
+;; compiled by the C-implemented Racket BC bytecode and JIT compilers.
+;; (On platforms where Racket BC's JIT is not supported, yet another compiler
+;; instead compiles the linklets to C code, but this is not a bootstrapping
+;; issue.)
+;;
 ;; Code:
 
 
@@ -200,7 +247,6 @@ DrRacket IDE, are not included.")
     ;; https://download.racket-lang.org/license.html
     (license (list lgpl3+ asl2.0 expat))))
 
-
 (define-public racket
   (package
     (inherit racket-minimal)
@@ -243,8 +289,35 @@ The main Racket distribution comes with many bundled packages, including the
 DrRacket IDE, libraries for GUI and web programming, and implementations of
 languages such as Typed Racket, R5RS and R6RS Scheme, Algol 60, and Datalog.")))
 
-(define %pre-release-installers
-  "https://pre-release.racket-lang.org/installers/")
+
+(define cfg-flag:sh-for-rktio
+  `(string-append "CPPFLAGS=-DGUIX_RKTIO_PATCH_BIN_SH="
+                  (assoc-ref %build-inputs "sh")
+                  "/bin/sh"))
+(define cfg-flag:enable-lt
+  `(string-append "--enable-lt="
+                  (assoc-ref %build-inputs "libtool")
+                  "/bin/libtool"))
+(define cfg-flag:enable-racket
+  `(let ((racket (assoc-ref %build-inputs "racket")))
+     (string-append "--enable-racket="
+                    racket
+                    "/bin/racket")))
+
+(define unpack-nanopass+stex
+  ;; Copied from chez-scheme.
+  ;; TODO: Eventually, we should refactor Chez Scheme
+  ;; enough to share more directly, so that we can make
+  ;; Racket's version of Chez avalable as a Guix package,
+  ;; e.g. for architectures not supported upstream.
+  ;; For now, we let Racket drive the Chez build process
+  ;; other than this step.
+  `(for-each (lambda (dep)
+               (define src
+                 (assoc-ref (or native-inputs inputs) dep))
+               (copy-recursively src dep
+                                 #:keep-mtime? #t))
+             '("nanopass" "stex")))
 
 (define-public racket-next-minimal
   (package
@@ -253,12 +326,214 @@ languages such as Typed Racket, R5RS and R6RS Scheme, Algol 60, and Datalog.")))
     (version "8.1.900")
     (source
      (origin
-       (inherit (package-source racket-minimal))
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://github.com/racket/racket")
+             (commit "0874b76de4f147ada46607857d8acf8445a1073d")))
        (sha256
         (base32
-         "0dm849wvlaxpfgz2qmgy2kwdslyi515rxn1m1yff38lagbn21vxq"))
-       (uri (string-append %pre-release-installers
-                           "racket-minimal-src.tgz"))))))
+         "0gy6rwyrpaij5k5pcyiif821b4vffqiaxg1vpg4iykw2c5ypfp43"))
+       (file-name
+        (git-file-name name version))
+       (patches
+        (search-patches
+         "racket-next-minimal-sh-via-rktio.patch"))
+       (snippet
+        (with-imported-modules '((guix build utils))
+          #~(begin
+              (use-modules (guix build utils))
+              ;; unbundle Chez submodules
+              (with-directory-excursion "racket/src/ChezScheme"
+                #$(origin-snippet (package-source chez-scheme)))
+              ;; unbundle libffi
+              (for-each
+               delete-file-recursively
+               '("racket/src/bc/foreign/libffi")))))))
+    (inputs
+     `(;; common to all racket-minimal variants:
+       ("openssl" ,openssl)
+       ("sqlite" ,sqlite)
+       ("sh" ,bash-minimal)
+       ;; only for CS
+       ("zlib" ,zlib)
+       ("zlib:static" ,zlib "static")
+       ("lz4" ,lz4)
+       ("lz4:static" ,lz4 "static")))
+    (native-inputs
+     `(("bootfiles" ,racket-next-bootstrap-chez-bootfiles)
+       ,@(package-native-inputs racket-next-bootstrap-chez-bootfiles)))
+    (build-system gnu-build-system)
+    (arguments
+     `(#:configure-flags
+       (list "--enable-csonly"
+             "--enable-libz"
+             "--enable-liblz4"
+             ,cfg-flag:enable-racket
+             ,cfg-flag:sh-for-rktio)
+       #:out-of-source? #true
+       #:tests? #f ;; not yet
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'unpack-nanopass+stex
+           (lambda* (#:key inputs native-inputs #:allow-other-keys)
+             (with-directory-excursion "racket/src/ChezScheme"
+               ,unpack-nanopass+stex)))
+         (add-after 'unpack-nanopass+stex 'unpack-bootfiles
+           (lambda* (#:key inputs #:allow-other-keys)
+             (with-directory-excursion "racket/src/ChezScheme"
+               (copy-recursively
+                (string-append (assoc-ref inputs "bootfiles") "/boot")
+                "boot"))))
+         (add-before 'configure 'initialize-config.rktd
+           (lambda* (#:key inputs #:allow-other-keys)
+             (mkdir-p "racket/etc")
+             (with-output-to-file "racket/etc/config.rktd"
+               (lambda ()
+                 (format #t
+                         "#hash((lib-search-dirs . ~s))\n"
+                         (cons #f
+                               (map (lambda (lib)
+                                      (string-append (assoc-ref inputs lib)
+                                                     "/lib"))
+                                    '("openssl"
+                                      "sqlite"))))))
+             #t))
+         (replace 'configure
+           (let ((inner (assq-ref %standard-phases 'configure)))
+             (lambda args
+               (chdir "racket/src")
+               (apply inner args))))
+         (add-after 'install 'remove-pkgs-directory
+           ;; otherwise, e.g., `raco pkg show` will try and fail to
+           ;; create a lock file
+           (lambda* (#:key outputs #:allow-other-keys)
+             ;; rmdir because we want an error if it isn't empty
+             (rmdir (string-append (assoc-ref outputs "out")
+                                   "/share/racket/pkgs"))
+             #t)))))
+    ;; https://download.racket-lang.org/license.html
+    ;; The LGPL components are only used by Racket BC.
+    (license (list asl2.0 expat))))
+
+(define-public racket-next-minimal-bc-3m
+  (hidden-package
+   (package/inherit racket-next-minimal
+     (name "racket-next-minimal-bc-3m")
+     (inputs
+      `(("libffi" ,libffi) ;; <- only for BC variants
+        ,@(filter (match-lambda
+                    ((label . _)
+                     (not (member label
+                                  '("zlib" "zlib:static"
+                                    "lz4" "lz4:static")))))
+                  (package-inputs racket-next-minimal))))
+     (native-inputs
+      `(("libtool" ,libtool)
+        ("racket" ,(if (%current-target-system)
+                       racket-next-minimal
+                       racket-next-minimal-bc-cgc))))
+     (arguments
+      (substitute-keyword-arguments (package-arguments racket-next-minimal)
+        ((#:configure-flags _ '())
+         `(list "--enable-bconly"
+                ,cfg-flag:enable-racket
+                ,cfg-flag:enable-lt
+                ,cfg-flag:sh-for-rktio))
+        ((#:phases usual-phases)
+         `(modify-phases ,usual-phases
+            (delete 'unpack-nanopass+stex)
+            (delete 'unpack-bootfiles)))))
+     (synopsis "Minimal Racket with the BC [3M] runtime system")
+     (description "The Racket BC (``before Chez'' or ``bytecode'') implementation was the default before Racket 8.0. It uses a compiler written in C targeting architecture-independent bytecode, plus a JIT compiler on most platforms. Racket BC has a different C API than the newer runtune system (Racket CS) supports a slightly different set of architectures than the current runtime system, Racket CS (based on ``Chez Scheme'').
+
+This packackage is the normal implementation of Racket BC with a precise garbage collector, 3M (``Moving Memory Manager'')."))))
+
+(define-public racket-next-minimal-bc-cgc
+  (package/inherit racket-next-minimal-bc-3m
+    (name "racket-next-minimal-bc-cgc")
+    (native-inputs
+     (filter (match-lambda
+               (("racket" . _)
+                #f)
+               (_
+                #t))
+             (package-native-inputs racket-next-minimal-bc-3m)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments racket-next-minimal-bc-3m)
+       ((#:configure-flags _ '())
+        `(list "--enable-cgcdefault"
+               ,cfg-flag:enable-lt
+               ,cfg-flag:sh-for-rktio))))
+    (synopsis "Old Racket implemetation used for bootstapping")
+    (description "This variant of the Racket BC (``before Chez'' or ``bytecode'') implementation is not recommended for general use. It uses CGC (a ``Conservative Garbage Collector''), which was succeeded as default in PLT Scheme version 370 (which translates to 3.7 in the current versioning scheme) by the 3M variant, which in turn was succeeded in version 8.0 by the Racket CS implementation.
+
+Racket BC [CGC] is primarily used for bootstrapping Racket BC [3M]. It may also be used for embedding applications without the annotations needed in C code to use the 3M garbage collector.")))
+
+(define-public racket-next-bootstrap-chez-bootfiles
+  (hidden-package
+   (package/inherit racket-next-minimal
+     (name "racket-next-bootstrap-chez-bootfiles")
+     (inputs `())
+     (native-inputs
+      `(("racket" ,(if (%current-target-system)
+                       racket-next-minimal
+                       racket-next-minimal-bc-3m))
+        ("stex" ,@(assoc-ref (package-native-inputs chez-scheme) "stex"))
+        ("nanopass" ,@(assoc-ref (package-native-inputs chez-scheme)
+                                 "nanopass"))))
+     (arguments
+      `(#:phases
+        (modify-phases %standard-phases
+          (add-after 'unpack 'unpack-nanopass+stex
+            (lambda* (#:key inputs native-inputs #:allow-other-keys)
+              (with-directory-excursion "racket/src/ChezScheme"
+                ,unpack-nanopass+stex)))
+          (delete 'configure)
+          (delete 'patch-generated-file-shebangs)
+          (replace 'build
+            (lambda* (#:key inputs outputs #:allow-other-keys)
+              (with-directory-excursion "racket/src/ChezScheme"
+                (invoke (string-append (assoc-ref inputs "racket")
+                                       "/bin/racket")
+                        "rktboot/main.rkt"
+                        "--dest" (assoc-ref outputs "out")))
+              #t))
+          (delete 'check)
+          (delete 'install))))
+     (synopsis "Chez Scheme bootfiles bootstrapped by Racket")
+     (description "Chez Scheme is a self-hosting compiler: building it requires ``bootfiles'' containing the Scheme-implemented portions compiled for the current platform. (Chez can then cross-compile bootfiles for all other supported platforms.)
+
+The Racket package ``cs-bootstrap'' (part of the main Racket Git repository) implements enough of a Chez Scheme simulation to load the Chez Scheme compiler purely from source into Racket and apply the compiler to itself, thus bootstrapping Chez Scheme. Bootstrapping takes about 10 times as long as using an existing Chez Scheme, but ``cs-bootstap'' supports Racket 7.1 and later, including the Racket BC variant.
+
+Note that the generated bootfiles are specific to Racket's fork of Chez Scheme, and ``cs-bootstrap'' does not currently support building upstream Chez Scheme.")
+     (license (package-license chez-scheme)))))
+
+
+(define %pre-release-installers
+  "https://pre-release.racket-lang.org/installers/")
+
+(define %main-repo-main-distribution-pkgs
+  ;; These are the packages developed in the main Racket Git repository
+  ;; that are part of the main distribution.
+  '("at-exp-lib"
+    "base"
+    "compiler-lib"
+    ;; NOT "compiler-test"
+    "compiler"
+    "net-doc"
+    "net-lib"
+    ;; NOT "net-test"
+    "net"
+    ;; NOT "plt-services"
+    ;; NOT "racket-benchmarks"
+    ;; NOT "racket-build-guide"
+    "racket-doc"
+    "racket-index"
+    "racket-lib"
+    ;; NOT "racket-test-core"
+    ;; NOT "racket-test-extra"
+    ;; NOT "racket-test"
+    "zo-lib"))
 
 (define-public racket-next
   (package
@@ -289,36 +564,10 @@ languages such as Typed Racket, R5RS and R6RS Scheme, Algol 60, and Datalog.")))
                           "README"
                           "src"))
               ;; unbundle package sources included elsewhere
-              (define (substitute/delete file pattern)
-                (substitute
-                 file
-                 (list (cons pattern
-                             (lambda (line matches)
-                               ;; must match exactly once
-                               (match matches
-                                 ((m)
-                                  (string-append (match:prefix m)
-                                                 (match:suffix m)))))))))
-              (define (unbundle-pkg pkg)
-                (define quoted-pkg (regexp-quote pkg))
-                (with-directory-excursion "share"
-                  (substitute/delete
-                   "links.rktd"
-                   (string-append
-                    "[(][^()]+[(]#\"pkgs\" #\""
-                    quoted-pkg
-                    "\"[)][)]"))
-                  (with-directory-excursion "pkgs"
-                    (substitute/delete
-                     "pkgs.rktd"
-                     (string-append
-                      "[(]\""
-                      quoted-pkg
-                      "\" \\. #s[(]"
-                      "(pkg-info|[(]sc-pkg-info pkg-info 3[)])"
-                      " [(][^()]+[)] [^()]+[)][)]"))
-                    (delete-file-recursively pkg))))
-              (unbundle-pkg "racket-lib"))))))
+              (with-directory-excursion "share/pkgs"
+                (for-each delete-file-recursively
+                          '#+%main-repo-main-distribution-pkgs))
+              #t)))))
     (inputs
      `(("cairo" ,cairo)
        ("fontconfig" ,fontconfig)
@@ -336,22 +585,32 @@ languages such as Typed Racket, R5RS and R6RS Scheme, Algol 60, and Datalog.")))
        ("libedit" ,libedit)))
     (native-inputs
      `(("racket" ,racket-next-minimal)
-       ("extend-layer" ,extend-layer)))
+       ("extend-layer" ,extend-layer)
+       ("main-repo" ,(package-source racket-next-minimal))))
     (arguments
      `(#:phases
        (modify-phases %standard-phases
          (add-before 'configure 'unpack-packages
-           (lambda* (#:key native-inputs inputs outputs #:allow-other-keys)
-             (let ((racket (assoc-ref (or native-inputs inputs) "racket"))
-                   (prefix (assoc-ref outputs "out")))
-               (mkdir-p (string-append prefix "/share/racket/pkgs"))
-               (copy-recursively
-                "share/links.rktd"
-                (string-append prefix "/share/racket/links.rktd"))
-               (copy-recursively
-                "share/pkgs"
-                (string-append prefix "/share/racket/pkgs"))
-               #t)))
+           (let ((unpack (assoc-ref %standard-phases 'unpack)))
+             (lambda* (#:key  native-inputs inputs outputs #:allow-other-keys)
+               (let* ((racket (assoc-ref (or native-inputs inputs) "racket"))
+                      (prefix (assoc-ref outputs "out"))
+                      (pkgs-dir (string-append prefix "/share/racket/pkgs")))
+                 (mkdir-p pkgs-dir)
+                 (copy-recursively
+                  "share/links.rktd"
+                  (string-append prefix "/share/racket/links.rktd"))
+                 (copy-recursively "share/pkgs" pkgs-dir)
+                 ;; NOTE: unpack changes the working directory
+                 (unpack #:source (assoc-ref (or native-inputs inputs)
+                                             "main-repo"))
+                 (for-each (lambda (pkg)
+                             (define dest (string-append pkgs-dir "/" pkg))
+                             (mkdir-p dest)
+                             (copy-recursively (string-append "pkgs/" pkg)
+                                               dest))
+                           ',%main-repo-main-distribution-pkgs)
+                 #t))))
          (replace 'configure
            (lambda* (#:key native-inputs inputs outputs #:allow-other-keys)
              (let ((racket (assoc-ref (or native-inputs inputs) "racket"))
-- 
2.30.2

[bug#49280] [PATCH 0/4] gnu: racket: Add racket-next. Bootstrap from C.

[Ludovic Courtès]

Hi,

Philip McGrath <philip@philipmcgrath.com> skribis:

> This commit bootstraps the Racket compiler and runtime system from source,
> including Racket CS as well as both variants of Racket BC. (One remaining
> limitation is discussed in comments added to gnu/packages/racket.scm.)
>
> In the process, it moves to building minimal Racket from the Git repository,
> rather than the packaged source tarballs. The Git repository is slightly
> better as the ``corresponding source'':
>
>  1. A few packages especially closely tied to the Racket core implementation
>     (like "compiler-lib", "base", and "racket-doc") are developed in the
>     same Git repository. Having them use the same Guix origin, too, will
>     help to keep them in sync.
>
>  2. The top-level Makefile in the Git repository is an important
>     ``script[] used to control compilation and installation.''
>     In particular, it cooperates with the "distro-build" package to
>     create the source tarballs and installers for a Racket distribution.
>     (Racket supports a notion of custom distributions.)
>
>  3. It is ``the preferred form ... for making modifications'' to the core
>     Racket implementation.
>
> Racket releases are tagged in the Git repository (e.g. "v8.1"). At the
> beginning of each release cycle, a branch is created to stabalize a version
> for extra testing. Active development happens on the "master" branch.
>
> * gnu/packages/racket-next-minimal-sh-via-rktio.patch: New file, coppied
> from racket-sh-via-rktio.patch to accomodate an extra directory layer.
> When racket-next-minimal becomes racket-minimal, this version will be
> the only one needed.
> * gnu/local.mk (dist_patch_DATA): Add it.
> * gnu/local/racket.scm (cfg-flag:sh-for-rktio, cfg-flag:enable-lt,
> cfg-flag:enable-racket, unpack-nanopass+stex,
> %main-repo-main-distribution-pkgs): New private variables.
> * gnu/local/racket.scm (racket-next-minimal)[source]: Use Git.
> [source](snippet): Unbundle nanopass, stex, and libffi.
> [inputs]: List explicitly.
> [native-inputs]: Use racket-next-bootstrap-bootfiles, plus its
> dependencies (for Chez, plus a Racket for bootstrappig).
> [arguments]: Revise extensively.
> * gnu/local/racket.scm (racket-next-minimal-bc-3m,
> racket-next-minimal-bc-cgc): New packages, hidden at least for now.
> (racket-next-bootstrap-chez-bootfiles): Another new package, but this one
> is especially likely to stay hidden.
> * gnu/local/racket.scm (racket-next)[origin](snippet): Unbundle packages
> developed in the main Git repository, but leave their links.rktd and
> pkgs.rktd entries in place.
> [native-inputs]: Add the main Racket Git repository.
> [arguments](#:phases): Adjust 'unpack-packages to also unpack package
> sources from the main Racket Git repository.

Exciting!

[...]

> +       (method git-fetch)
> +       (uri (git-reference
> +             (url "https://github.com/racket/racket")
> +             (commit "0874b76de4f147ada46607857d8acf8445a1073d")))
>         (sha256
>          (base32
> -         "0dm849wvlaxpfgz2qmgy2kwdslyi515rxn1m1yff38lagbn21vxq"))
> -       (uri (string-append %pre-release-installers
> -                           "racket-minimal-src.tgz"))))))
> +         "0gy6rwyrpaij5k5pcyiif821b4vffqiaxg1vpg4iykw2c5ypfp43"))
> +       (file-name
> +        (git-file-name name version))
> +       (patches
> +        (search-patches
> +         "racket-next-minimal-sh-via-rktio.patch"))

Please keep these on a single line, as in:

  (file-name (git-file-name name version))

> +         (replace 'configure
> +           (let ((inner (assq-ref %standard-phases 'configure)))
> +             (lambda args
> +               (chdir "racket/src")
> +               (apply inner args))))

I’d find it clearer like this:

  (add-before 'configure 'change-directory
    (lambda _
      (chdir "racket/src")))

> +         (add-after 'install 'remove-pkgs-directory
> +           ;; otherwise, e.g., `raco pkg show` will try and fail to
> +           ;; create a lock file
> +           (lambda* (#:key outputs #:allow-other-keys)
> +             ;; rmdir because we want an error if it isn't empty
> +             (rmdir (string-append (assoc-ref outputs "out")
> +                                   "/share/racket/pkgs"))
> +             #t)))))

Please write full sentences with a bit more context (“Remove package
directory, otherwise ‘raco pkg show’ …”).

> +(define-public racket-next-minimal-bc-3m
> +  (hidden-package
> +   (package/inherit racket-next-minimal
> +     (name "racket-next-minimal-bc-3m")

This is “-next” because it’s targeting 8.1, which is not released yet,
right?

Since it’s only used for bootstrapping, perhaps use ‘define’ instead of
‘define-public’ and remove the call to ‘hidden-package’.

It should also be (package (inherit …) …) rather than (package/inherit
…).  The latter is only useful when defining variants of a package (same
version, same code) where the same security updates would apply.

> +     (inputs
> +      `(("libffi" ,libffi) ;; <- only for BC variants
> +        ,@(filter (match-lambda
> +                    ((label . _)
> +                     (not (member label
> +                                  '("zlib" "zlib:static"
> +                                    "lz4" "lz4:static")))))
> +                  (package-inputs racket-next-minimal))))

Please use this more common idiom:

  ,@(fold alist-delete (package-inputs racket-next-minimal) '("zlib" …))

(It matters notably because ‘guix style’ recognizes it:
<https://issues.guix.gnu.org/49169>.)

> +     (synopsis "Minimal Racket with the BC [3M] runtime system")
> +     (description "The Racket BC (``before Chez'' or ``bytecode'') implementation was the default before Racket 8.0. It uses a compiler written in C targeting architecture-independent bytecode, plus a JIT compiler on most platforms. Racket BC has a different C API than the newer runtune system (Racket CS) supports a slightly different set of architectures than the current runtime system, Racket CS (based on ``Chez Scheme'').
> +
> +This packackage is the normal implementation of Racket BC with a precise garbage collector, 3M (``Moving Memory Mana
        ^
Typo here, and lines too long (here and in other places).  :-)

Please also check what ‘guix lint’ thinks!

> +(define-public racket-next-minimal-bc-cgc
> +  (package/inherit racket-next-minimal-bc-3m
> +    (name "racket-next-minimal-bc-cgc")
> +    (native-inputs
> +     (filter (match-lambda
> +               (("racket" . _)
> +                #f)
> +               (_
> +                #t))
> +             (package-native-inputs racket-next-minimal-bc-3m)))

Rather: (alist-delete "racket" (package-native-inputs racket-next-minimal-bc-3m))

> +     (license (package-license chez-scheme)))))

You cannot do that since here since potentially we could end up with
circular top-level references from these two modules.

Instead, restate what the license is.

Thanks for all this!

Ludo’.