Hi,

On Friday, February 18, 2022 2:38:52 AM EST Liliana Marie Prikler wrote:
> Hi,
> 
> Am Donnerstag, dem 17.02.2022 um 15:50 -0500 schrieb Philip McGrath:
> > -;; Commentary:
> > -;;
> > -;; Here's how bootstrapping minimal Racket works:
> > -;;
> > -;;   - Racket BC [CGC] can be built with only a C compiler (except
> > for
> > -;;     one caveat discussed below).
> > -;;   - Racket BC [3M] needs an existing Racket to run "xform",
> > -;;     which transforms its own C source code to add additional
> > annotations
> > -;;     for the precise garbage collector.
> > -;;   - Racket CS needs (bootfiles for) Racket's fork of Chez Scheme.
> > -;;     It also needs an existing Racket to compile Racket-
> > implemented
> > -;;     parts of the runtime system to R6RS libraries.
> > -;;   - Chez Scheme also needs bootfiles for itself, but Racket can
> > simulate
> > -;;     enough of Chez Scheme to load Racket's fork of the Chez
> > Scheme compiler
> > -;;     purely from source into Racket and apply the compiler to
> > itself,
> > -;;     producing the needed bootfiles (albeit very slowly).
> > -;;     Any variant of Racket since version 7.1 can run the
> > simulation.
> > -;;
> > -;; So, we build CGC to build 3M to build bootfiles and CS.
> > -;;
> > -;; One remaining bootstrapping limitation is that Racket's reader,
> > module
> > -;; system, and macro expander are implemented in Racket. For Racket
> > CS,
> > -;; they are compiled to R6RS libraries as discussed above. This note
> > from the
> > -;; README file applies to all such subsystems:
> > -;;
> > -;;     The Racket version must be practically the same as the
> > current Racket
> > -;;     verson, although it can be the Racket BC implementation
> > (instead of
> > -;;     the Racket CS implementation).
> > -;;
> > -;;     Unlike Chez Scheme boot files, the files generated in
> > "schemified"
> > -;;     are human-readable and -editable Scheme code. That provides a
> > way
> > -;;     out of bootstrapping black holes, even without BC.
> > -;;
> > -;; However, other Racket subsystems implemented in Racket for Racket
> > CS
> > -;; use older C implementations for Racket BC, whereas the reader,
> > expander,
> > -;; and module system were completely replaced with the Racket
> > implementation
> > -;; as of Racket 7.0.
> > -;;
> > -;; For Racket BC, the compiled "linklet" s-expressions (primitive
> > modules)
> > -;; are embeded in C as a static string constant. Eventually, they
> > are further
> > -;; compiled by the C-implemented Racket BC bytecode and JIT
> > compilers.
> > -;; (On platforms where Racket BC's JIT is not supported, yet another
> > compiler
> > -;; instead compiles the linklets to C code, but this is not a
> > bootstrapping
> > -;; issue.)
> 
> I think it'd be clearer if this commentary was moved along with the
> bootstrapping code.

I did add it to "chez-and-racket-bootstrap.scm" at the same time as I added 
the Racket bootstrapping code there. But I didn't delete the bootstrapping 
code from this file until this commit, so I deleted the associated comment at 
the same time.

> Is there a reason why we can't use (racket-vm-for-
> system) before updating Racket to 8.4?  This looks like another of
> those "two things at once" patches.
>  

The whole series based on `racket-vm-*` would not work with Racket 8.3 without 
backporting at least some things: off the top of my head, at a minimum, we 
would need b53090140596cc8522037f4c812325c71648df7a and 
2b282d9c48df811cd4678cdbaed8258cdef23946 to be able to build "chez-scheme-for-
racket:doc".

Maybe it will seem less like "two things at once" if I explain more explicitly 
that "racket-minimal@8.3" is actually the same content that is now in "racket-
vm-cs@8.4" (just installed into different directories), and does *not* contain 
anything that is now in "racket-minimal@8.4". 

Shortly before the release of Racket 8.3, it came to light from the 
intersection of a few conversations (scattered across several places, but 
summarised in <https://github.com/racket/racket/issues/
3851#issuecomment-932641908> and the following two comments) that the contents 
of a "minimal Racket" were inconsistent. On Windows, Mac OS, and "x86_64-
linux-natipkg" (a special configuration that avoids relying on a system package 
manager, e.g. for CI), minimal Racket had "racket-lib", "base", and packages 
providing native libraries (e.g. OpenSSL and SQLite). On other systems, 
whether using pre-built binaries or building from source using the released 
tarballs, "minimal Racket" would end up with only the "racket-lib" package, 
because "base" was pulled in only as a dependency of the native library 
packages. However, when building minimal Racket from the Git sources in the 
way Guix was, "minimal Racket" ended up with no packages installed at all.

Matthew Flatt's conclusion was that, starting after the 8.3 release (to allow 
more time for testing), "racket-lib" should directly depend on "base", and 
"minimal Racket" should always explicitly install "racket-lib". (That fits the 
semantic roles of those packages, which represent the current always-available 
native libraries and the current "built in" collections in the Racket package 
system's model of dependencies and compatibility.)

That works out especially nicely for Guix, as it gives us a clean boundary 
between the core Racket VM and compiler, with all of the bootstrapping 
involved, and building Racket packages and installation layers, which can be 
handled in a nice, uniform way and eventually turned into a `racket-build-
system`.

> >  (define-public racket-minimal
> >    (package
> >      (name "racket-minimal")
> > -    (version "8.3")            ; note: remember to also update
> > racket!
> > -    (source
> > -     (origin
> > -       (method git-fetch)
> > -       (uri (git-reference
> > -             (url "https://github.com/racket/racket")
> > -             (commit (string-append "v" version))))
> > -       (sha256
> > -        "1i1jnv1wb0kanfg47hniafx2vhwjc33qqx66lq7wkf5hbmgsyws3")
> > -       (file-name (git-file-name name version))
> > -       (patches (search-patches "racket-minimal-sh-via-
> > rktio.patch"))
> > -       (modules '((guix build utils)))
> > -       (snippet
> > -        (with-imported-modules '((guix build utils))
> > -          #~(begin
> > -              ;; Unbundle Chez submodules.
> > -              (with-directory-excursion "racket/src/ChezScheme"
> > -                ;; Remove bundled libraries (copied from 'chez-
> > scheme').
> > -                (for-each delete-file-recursively
> > -                          '("stex"
> > -                            "nanopass"
> > -                            "lz4"
> > -                            "zlib")))
> > -              ;; Unbundle libffi.
> > -              (delete-file-recursively
> > "racket/src/bc/foreign/libffi"))))))
> > -    (inputs
> > -     `(;; common to all racket-minimal variants:
> > -       ("openssl" ,openssl)
> > -       ("sqlite" ,sqlite)
> > -       ("sh" ,bash-minimal)
> > -       ;; only for CS
> > -       ("zlib" ,zlib)
> > -       ("zlib:static" ,zlib "static")
> > -       ("lz4" ,lz4)
> > -       ("lz4:static" ,lz4 "static")))
> > -    (native-inputs
> > -     `(("bootfiles" ,racket-bootstrap-chez-bootfiles)
> > -       ,@(package-native-inputs racket-bootstrap-chez-bootfiles)))
> > +    (version (package-version (racket-vm-for-system)))
> > +    (source (package-source (racket-vm-for-system)))
> > +    ;; For cross-compilation, Matthew Flatt recommends reusing
> > +    ;; as much of `raco cross` as possible. So, put that off until
> > +    ;; we have a build system for Racket packages.
> > +    (inputs (list openssl sqlite (racket-vm-for-system)))
> 
> As outlined earlier, I believe Racket should define its version, not
> racket-vm-for-system.
> 

As I said, I'll send a v3 with %racket-version.

But the reason I think the `racket` packages would be a particularly bad place 
to define this for reasons related to what I was just describing. Once we have 
a `racket-build-system`—and we are getting ever closer—`racket-minimal` will 
simply be a tethered installation layer with two packages (ignoring 
"natipkg"), assembled with something somewhat like the `texlive-udpmap.cfg` 
function. The `racket` package will likewise be a tethered installation layer 
with 203 packages, two of which will be shared by `racket-minimal`. We will 
want to have others, both larger (e.g. "main-distribution-test" and all of its 
dependencies) and smaller (e.g. some people like just "drracket" without some 
of the more niche dependencies of "main-distribution", like the support 
libraries for the textbook, "Schreibe Dein Programm!"). A major motivation for 
the whole design of the Racket package system (actually, its second package 
system) is that the "main-distribution" package and the Racket distribution 
based on it should not be in any way special or built in: it happens to be 
released at download.racket-lang.org, but there can be many Racket 
distributions. Some might go so far as to argue that any special status of 
"main-distribution" falls under the category of weaknesses and restrictions 
that should be removed.

> >  (define-public racket
> > 
> >    (package
> >      (inherit racket-minimal)
> >      (name "racket")
> > -    (version (package-version racket-minimal)) ; needed for origin
> > uri to work
> > -    (source
> > -     (origin
> > -       (method url-fetch)
> > -       (uri (map (lambda (base)
> > -                   (string-append base version "/racket-src.tgz"))
> > -                 %installer-mirrors))
> > -       (sha256
> > -        (base32
> > -         "0jdr0y7scvv2a3sq456ifrgq0yfsbiwavdf2m86zmrapp481mby4"))
> > -       (snippet
> > -        #~(begin
> > -            (use-modules (guix build utils)
> > -                         (ice-9 match)
> > -                         (ice-9 regex))
> > -            ;; unbundle minimal Racket
> > -            (for-each delete-file-recursively
> > -                      '("collects"
> > -                        "doc"
> > -                        "etc"
> > -                        "README"
> > -                        "src"))
> > -            ;; unbundle package sources included elsewhere
> > -            (with-directory-excursion "share/pkgs"
> > -              (for-each delete-file-recursively
> > -                        '#+%main-repo-main-distribution-pkgs))
> > -            #t))))
> > +    (source #f)
> 
> Why?
> 

The vast majority of package in `racket` are not developed in the <https://
github.com/racket/racket> repository. For that matter, the source of the "main 
distribution" package itself is <https://github.com/racket/main-distribution> 
(and, under the Racket package system's notion of versions, it is at version 
"0.0").

> >      (inputs
> > -     `(("cairo" ,cairo)
> > -       ("fontconfig" ,fontconfig)
> > -       ("glib" ,glib)
> > -       ("glu" ,glu)
> > -       ("gmp" ,gmp)
> > -       ("gtk+" ,gtk+)                   ; propagates gdk-pixbuf+svg
> > -       ("libjpeg" ,libjpeg-turbo)
> > -       ("libpng" ,libpng)
> > -       ("libx11" ,libx11)
> > -       ("mesa" ,mesa)
> > -       ("mpfr" ,mpfr)
> > -       ("pango" ,pango)
> > -       ("unixodbc" ,unixodbc)
> > -       ("libedit" ,libedit)))
> > -    (native-inputs
> > -     `(("racket" ,racket-minimal)
> > -       ("extend-layer" ,extend-layer)
> > -       ("main-repo" ,(package-source racket-minimal))))
> > +     (list cairo
> > +           fontconfig
> > +           glib
> > +           glu
> > +           gmp
> > +           gtk+ ;; propagates gdk-pixbuf+svg
> > +           libjpeg-turbo
> > +           libpng
> > +           libx11 ;; ?? wayland ??
> > +           mesa
> > +           mpfr
> > +           pango
> > +           unixodbc
> > +           libedit ;; TODO reconsider in light of expeditor and
> > readline-gpl
> > +           racket-minimal ;; <-- TODO non-tethered layer
> > +           (racket-vm-for-system)))
> >      (arguments
> > -     `(#:phases
> > -       (modify-phases %standard-phases
> > -         (add-before 'configure 'unpack-packages
> > -           (let ((unpack (assoc-ref %standard-phases 'unpack)))
> > -             (lambda* (#:key  native-inputs inputs outputs #:allow-
> > other-keys)
> > -               (let* ((racket (assoc-ref (or native-inputs inputs)
> > "racket"))
> > -                      (prefix (assoc-ref outputs "out"))
> > -                      (pkgs-dir (string-append prefix
> > "/share/racket/pkgs")))
> > -                 (mkdir-p pkgs-dir)
> > -                 (copy-recursively
> > -                  "share/links.rktd"
> > -                  (string-append prefix "/share/racket/links.rktd"))
> > -                 (copy-recursively "share/pkgs" pkgs-dir)
> > -                 ;; NOTE: unpack changes the working directory
> > -                 (unpack #:source (assoc-ref (or native-inputs
> > inputs)
> > -                                             "main-repo"))
> > -                 (for-each (lambda (pkg)
> > -                             (define dest (string-append pkgs-dir
> > "/" pkg))
> > -                             (mkdir-p dest)
> > -                             (copy-recursively (string-append
> > "pkgs/" pkg)
> > -                                               dest))
> > -                           ',%main-repo-main-distribution-pkgs)
> > -                 #t))))
> > -         (replace 'configure
> > -           (lambda* (#:key native-inputs inputs outputs #:allow-
> > other-keys)
> > -             (let ((racket (assoc-ref (or native-inputs inputs)
> > "racket"))
> > -                   (prefix (assoc-ref outputs "out")))
> > -               (apply invoke
> > -                      (string-append racket "/bin/racket")
> > -                      (assoc-ref inputs "extend-layer")
> > -                      racket
> > -                      prefix
> > -                      (map
> > -                       (lambda (lib)
> > -                         (string-append (assoc-ref inputs lib)
> > "/lib"))
> > -                       '("cairo"
> > -                         "fontconfig"
> > -                         "glib"
> > -                         "glu"
> > -                         "gmp"
> > -                         "gtk+"
> > -                         "libjpeg"
> > -                         "libpng"
> > -                         "libx11"
> > -                         "mesa"
> > -                         "mpfr"
> > -                         "pango"
> > -                         "unixodbc"
> > -                         "libedit")))
> > -               #t)))
> > -         (replace 'build
> > -           (lambda* (#:key native-inputs inputs outputs #:allow-
> > other-keys)
> > -             (invoke (string-append (assoc-ref (or native-inputs
> > inputs)
> > -                                               "racket")
> > -                                    "/bin/racket")
> > -                     "--config"
> > -                     (string-append (assoc-ref outputs "out")
> > -                                    "/etc/racket")
> > -                     "-l"
> > -                     "raco"
> > -                     "setup")
> > -             #t))
> > -         (delete 'install))
> > -       ;; we still don't have these:
> > -       #:tests? #f))
> > +     (substitute-keyword-arguments (package-arguments racket-
> > minimal)
> > +       ((#:configure-flags _ '())
> > +        #~`("--tethered"
> > +            "--extra-foreign-lib-search-dirs"
> > +            ,(format #f "~s"
> > +                     '(#$@(map (lambda (name)
> > +                                 (cond
> > +                                  ((this-package-input name)
> > +                                   => (cut file-append <> "/lib"))
> > +                                  (else
> > +                                   (raise-exception
> > +                                    (make-exception
> > +                                     (make-assertion-failure)
> > +                                     (make-exception-with-message
> > +                                      "missing input to the 'racket'
> > package")
> > +                                     (make-exception-with-irritants
> > +                                      (list name)))))))
> > +                               '("cairo"
> > +                                 "fontconfig-minimal" ;; aka
> > fontconfig
> > +                                 "glib"
> > +                                 "glu"
> > +                                 "gmp"
> > +                                 "gtk+"
> > +                                 "libjpeg-turbo"
> > +                                 "libpng"
> > +                                 "libx11"
> > +                                 "mesa"
> > +                                 "mpfr"
> > +                                 "pango"
> > +                                 "unixodbc"
> > +                                 "libedit"))))))
> > +       ((#:make-flags _ '())
> > +        #~`("main-distribution"))
> > +       ((#:phases parent-phases #~%standard-phases)
> > +        #~(modify-phases #$parent-phases
> > +            (delete 'unpack)
> > +            (replace 'build
> > +              (lambda args
> > +                (mkdir-p (string-append #$output
> > "/lib/racket/pkgs"))
> > +                (for-each
> > +                 (match-lambda
> > +                   ((name src)
> > +                    (copy-recursively
> > +                     src
> > +                     (string-append #$output "/lib/racket/pkgs/"
> > name))))
> > +                 '(#$@main-distribution-packages))))))))
> >      (synopsis "Programmable programming language in the Scheme
> > family")
> >      (description
> >       "Racket is a general-purpose programming language in the Scheme
> > family,
> > @@ -539,82 +224,899 @@ (define dest (string-append pkgs-dir "/" pkg))
> >  DrRacket IDE, libraries for GUI and web programming, and
> > implementations of
> >  languages such as Typed Racket, R5RS and R6RS Scheme, Algol 60, and
> > Datalog.")))
> 
> This looks like a very weird way of phrasing union-build.  Is there a
> reason to do this rather than union-build?
> 

IIUC, packages aren't supposed to install symbolic links into their sources, 
as `union-build` would do—wouldn't that prevent all of the extraneous files in 
those origins from being GCed? There's also the fact that `name` may be 
different from the name of the last element of `src`: in particular, in a 
significant number of cases, `src` will be something like "/gnu/store/
x22awqf0rbcyyk88kj82zihmdgkfhgza-racket-main-distribution-8.4-checkout" when 
`name` is "main-distribution".

-Philip