[bug#47180] [PATCH] gnu: racket: Don't inject store paths into Racket files.

[Philip McGrath <philip@philipmcgrath.com>]

Hi!

On 4/10/21 4:59 PM, Ludovic Courtès wrote:
> Hi Philip,
> 
> Philip McGrath <philip@philipmcgrath.com> skribis:
> 
>> Apparently, during grafting, Guix can somehow mangle compiled
>> Racket CS files (.zo) such that Racket will refuse to load them.
>> (Maybe it has something to do with compression?)
> 
> If those files are compressed, and if a store file name survives despite
> compression, then grafting can patch it, which could lead to checksum
> mismatches or similar.

Yes, that's what seems to be happening.

> 
> What error message does Racket produce?

The first error I heard of (and reproduced) was reported by Jack in 
<https://issues.guix.gnu.org/47064>:

```
$bytevector-uncompress: internal error uncompressing 
#"\0\0\0\0chez\310\224\206:\r()#\201\256R-d\205\233\24\363\5\20\201P\6A\v\300\0\16\f\6\31\2\f\6\f&H\275\0\1\0\362\bA\377e\0\1\0C\6A\21\3\v\300\0\201\265!\f\6\n\0\a\1\35\0\1+\0\360\27\201\375\300\0\0\0\17\205\210Z\0\0M\215\245\b\4\0\0M9fH\17\206fZ\0\0I\2...
    context...:
     body of 
"/gnu/store/mmrax3f1vx3c8ih9hhgffpvka6chk96w-racket-8.0/share/racket/pkgs/gui-lib/mred/private/wx/gtk/utils.rkt"
     body of 
"/gnu/store/mmrax3f1vx3c8ih9hhgffpvka6chk96w-racket-8.0/share/racket/pkgs/gui-lib/mred/private/wx/platform.rkt"
```

The error message is referring to an internal Chez Scheme primitive.

The report at <https://issues.guix.gnu.org/47614> seems to me to be an 
alternative manifestation of the same problem: the hexdump there is 
useful and explains why I did see some of the store path when I had 
attempted to investigate using `strings`.

So there seem to be at least three bad cases:

  1. The grafter can mangle .zo files so that Racket can't
     read them at all.
  2. The grafter can miss store references in .zo files, so Racket could
     end up using the ungrafted versions.
  3. With a garbage collection, Racket could try to use the ungrafted
     versions but fail to find them at runtime.

>> So, we stop patching Racket sources with absolute paths to store
>> files (i.e. for foreign libraries to dlopen).
>> Instead, we put them in a data file that doesn't get compiled or,
>> in one case, embed it in C.
> 
> That solves the problem for Racket itself, but wouln’t Racket libraries
> have the same issue?

Because the problems are triggered by grafting, they only affect 
packages that have been patched by Guix. For now, Guix doesn't have the 
ability to install more Racket packages. In the longer term, the 
one-sentence answer is that it's always possible we might find and 
switch to a better approach, but this seems most promising. (A bit more 
of my current thinking on that toward the end.)

> Would it be an option to instead turn off compression and keep doing
> things as usual?

In theory, this should be possible. I see two significant downsides:

  1. Compiled code would be much larger—maybe twice as big—and, if I
     recall correctly, load times would be worse, too. With the move to
     Racket CS, existing Racket code moved from a world of small and
     cheap bytecode to a world of machine code: the default compression
     settings have been tuned to avoid an unacceptable worsening of
     binary size and load time.
  2. Racket very intentionally does not specify the format of zo files,
     and indeed the details routinely change: I think there have now 
been
     14 such changes on Git since the 8.0 release in February. Continuing
     to patch zo files sets us up for future breakage, and it seems like
     it would be especially easy for maintainers of the Guix package to
     miss the implications of such changes to low-level implementation
     details (as I did!). For example, Chez Scheme seems to make
     compression options programmer-configurable even within a single
     object file: if Racket exposed such options, we could well end up
     here again.

More broadly, I think the best strategy for Racket packaging will be, as 
much as possible, to use Racket's supported configuration features 
rather than Guix-specific hacks. This seems especially viable since 
Racket has been willing to accept unobtrusive patches upstream that help 
things go smoothly for Guix, e.g. with 8.1 we should no longer need any 
patches to the build scripts: we're all friendly, parentheses-loving folks.

For another example, it looks like existing 
"racket-store-checksum-override.patch" fixes a previous issue discussed 
in <https://issues.guix.gnu.org/30680> caused by grafting zo files: I 
hope this change will also let us remove that patch, though I'd want to 
test more before proposing we drop it. So these problems aren't 
fundamentally new; they just have an additional symptom since the change 
to Racket CS by default in Racket 8.0. If we can fix the root problem of 
violating Racket's assumptions by patching zo files, we should be able 
to stop hunting down symptoms.

Rather than using "config.rktd", an alternative approach would be to set 
things up so that `dlopen` would find the foreign libraries, perhaps via 
`LD_LIBRARY_PATH`. This has some intriguing possibilities: I could 
imagine Guix providing an alternate `dlopen` implementation that might 
be useful beyond Racket.

Nix apparently configures some things via `LD_LIBRARY_PATH`, but their 
approach (as I understand it) relies on generating Racket scripts around 
all Racket-generated executable, which causes other problems. There 
should be workarounds, but it seems better to avoid going down that road 
if we can.

Finally, here's a sketch of how `guix import racket` and such might 
work. Racket's package system has a concept of "package scope", so that 
"installation" scope can coexist with narrower scopes (mostly per-user 
scope, though there are more complex possibilities). Right now, Guix 
puts installation scope in the read-only store, which basically 
corresponds to how other package managers put it in root-owned places, 
except Guix can't write to the store to install additional packages. I'm 
still at the information-gathering stage, but my current thinking is 
that the hypothetical `racket-build-system` should basically take the 
source package and turn it into what Racket calls a "built" package 
ready to be installed in `static-link` mode, which includes compiling 
the code and building the docs (which can involve quite a lot, e.g. ray 
tracing icons). Then a profile hook could knit together all of the 
Racket packages into an installation package scope. For packages that 
depend on foreign libraries, this would be a chance for Guix to add the 
necessary paths to the "config.rktd" for the installation.

-Philip