From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id 3FWRF5LBc2BaUwEAgWs5BA (envelope-from ) for ; Mon, 12 Apr 2021 05:42:10 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id aGJrEJLBc2A7KAAAbx9fmQ (envelope-from ) for ; Mon, 12 Apr 2021 03:42:10 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id B597211A13 for ; Mon, 12 Apr 2021 05:42:09 +0200 (CEST) Received: from localhost ([::1]:38782 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lVnSi-0004Yn-DK for larch@yhetil.org; Sun, 11 Apr 2021 23:42:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35518) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lVnSc-0004Yg-AL for guix-patches@gnu.org; Sun, 11 Apr 2021 23:42:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44601) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lVnSc-0005EA-3L for guix-patches@gnu.org; Sun, 11 Apr 2021 23:42:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1lVnSc-0002Ro-0z for guix-patches@gnu.org; Sun, 11 Apr 2021 23:42:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#47180] [PATCH] gnu: racket: Don't inject store paths into Racket files. Resent-From: Philip McGrath Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 12 Apr 2021 03:42:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 47180 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: jackhill@trilug.org, 47180@debbugs.gnu.org Received: via spool by 47180-submit@debbugs.gnu.org id=B47180.16181988699345 (code B ref 47180); Mon, 12 Apr 2021 03:42:01 +0000 Received: (at 47180) by debbugs.gnu.org; 12 Apr 2021 03:41:09 +0000 Received: from localhost ([127.0.0.1]:56147 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lVnRl-0002Qf-2Q for submit@debbugs.gnu.org; Sun, 11 Apr 2021 23:41:09 -0400 Received: from mail-qt1-f177.google.com ([209.85.160.177]:38414) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lVnRj-0002QT-CQ for 47180@debbugs.gnu.org; Sun, 11 Apr 2021 23:41:07 -0400 Received: by mail-qt1-f177.google.com with SMTP id j7so8938304qtx.5 for <47180@debbugs.gnu.org>; Sun, 11 Apr 2021 20:41:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=philipmcgrath.com; s=google; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=HDFDY3FmSQl/uLFZNSpJJzBH7fwGtbjjAmhaNNkZtzY=; b=j36APpVEpXbghDphT19yWRQ91ttFlw277sRGMrN5MfZX5GBMacwUJg4s58IYwMjy67 Bl/g/wr93cqTC+3frqfh8sGXlTCLcGv+jsyKXiSz6bhlhOEeHEQq1J/EeUmOi57TqnXn 7vckicV1RjA+iof0c+LrxjgJvNnS6QaPwEiseGC9V0ZAE4sIZKQ8i5hhrOC8Ur948sV/ GST/mZEO3EN81EZpmcwkTNNe07/i0nl2GCiM9I7011cGJYAwRdsBGH220TOGnFICXFmZ JK+JvQX19DAL39IfvWpQAnY9LdoWpXRdECztXavWLiu45nQJp0f0nf+hk2Lb/l7QkgSY e2vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=HDFDY3FmSQl/uLFZNSpJJzBH7fwGtbjjAmhaNNkZtzY=; b=mo7oavwxIkRWM/hWwnqGx5RjUB66zgDbQp95va1w+F7U85W9LLCTSjEIQOfwv9fUHx +/bUoSx+kA60bZu2yDBCqovWSB/YVwMR6ODxAM6Ou8x//7+MGRE7b1GCMROwaV8HSCw6 8IJh9a3Xho2yxjtRHpcG+ScUeWZWYHSjLUwQQtmRK3DVk3VtH337IdbkHVHITLCRYfZ4 CztZvAH7t31vQwRWGVi54+kHtp1ULtsb+PUyFnB68ftn6hB6JPyebEjPxQPnCIRR3gD6 mygPtAQPaV3uT+bOFDjbFeBc0XOaR4xLjeFvjnC2B33ugGCqhfGkfCDCsNQXS6CPL4pe /16Q== X-Gm-Message-State: AOAM532+zxdo2HOFFwiNO4B4JC/qxNmLPeRurcaRTUaZ1ihzLn/0FAQt Pr7o00IVQ1+MOqQWTKSbVM5hMS4RWY6Cb73JyyA= X-Google-Smtp-Source: ABdhPJzaX/AYaw0lA1szgwUMOth9qXKvNsmOkS30RJlYSWryMPJKVKeKanBbYzFQBrZj0LtfiQwCvA== X-Received: by 2002:a05:622a:253:: with SMTP id c19mr23594462qtx.355.1618198861676; Sun, 11 Apr 2021 20:41:01 -0700 (PDT) Received: from Sapientia.local (c-73-125-89-242.hsd1.fl.comcast.net. [73.125.89.242]) by smtp.gmail.com with ESMTPSA id y9sm6994653qtf.58.2021.04.11.20.41.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 11 Apr 2021 20:41:01 -0700 (PDT) References: <20210316025632.9767-1-philip@philipmcgrath.com> <87wnt9zwix.fsf@gnu.org> From: Philip McGrath Message-ID: Date: Sun, 11 Apr 2021 23:40:59 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 In-Reply-To: <87wnt9zwix.fsf@gnu.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1618198929; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=HDFDY3FmSQl/uLFZNSpJJzBH7fwGtbjjAmhaNNkZtzY=; b=Bi52HLj0AOZ2Zb0J6jobOe7SjOCGUQrU9giVBRUI2pkBAJ+tzx0Dcyxe5A5RZBHEwqF9Fc TAcw9MWiR/6RkmtsYpjITuQMX7kSfmT8xsI8QZQP4MktvidNHHLtagK7zvGJ4huZZUTvrU qcwh9MmgL4c0NzjhfMkuCdqNq/H6XxybhrMkHRJSd+OQWRIZVtrrVfkseUzNRtVyuiJzVP v3fUMwWWNQ6r5K+3ODp0Qkwbahl/H/ZFtsIxQohxxmr44zqtohpHvHmRZghPnV9p54a46E DBEoUSlOPFSfTA6ip3d8eBrMbltN7rZ0Zm37dYrSf4eenYbzpjhrw/+sYRAldQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1618198929; a=rsa-sha256; cv=none; b=QbjuJEd0bn5j+exS5H8Bq4lbi6KdwR7eZm4kRGr4K6jylKDmgGpgHQiC7wgsAShk0Sqolo +0v6k98iKS6vCD4ftkjuqvCqSNHGscfEIrRrHscOkx6hOlHUM9L0H0L82HJrCjrUs5kGlN ix+tangeuU9/zbSZY2hOpPa1HCNRjgjp/2cAcVsXxO9cuVsPZdfil2LI8QqXlwXfu5l7bN 8uRCrK+AsatKUiIem43hXTTbagkLHkfFGU52LWTDuLfoNIRTOEV21v+5AMoKS0HhxHwBUn YVsR/oo2j+nhOU/QmYUeAaE9/mjzGPxBpwjVC4NyL3W583St2SRQrF5EDFSYfw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=philipmcgrath.com header.s=google header.b=j36APpVE; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.44 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=philipmcgrath.com header.s=google header.b=j36APpVE; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: B597211A13 X-Spam-Score: -1.44 X-Migadu-Scanner: scn0.migadu.com X-TUID: u3SZDk/QRdAu Hi! On 4/10/21 4:59 PM, Ludovic Courtès wrote: > Hi Philip, > > Philip McGrath skribis: > >> Apparently, during grafting, Guix can somehow mangle compiled >> Racket CS files (.zo) such that Racket will refuse to load them. >> (Maybe it has something to do with compression?) > > If those files are compressed, and if a store file name survives despite > compression, then grafting can patch it, which could lead to checksum > mismatches or similar. Yes, that's what seems to be happening. > > What error message does Racket produce? The first error I heard of (and reproduced) was reported by Jack in : ``` $bytevector-uncompress: internal error uncompressing #"\0\0\0\0chez\310\224\206:\r()#\201\256R-d\205\233\24\363\5\20\201P\6A\v\300\0\16\f\6\31\2\f\6\f&H\275\0\1\0\362\bA\377e\0\1\0C\6A\21\3\v\300\0\201\265!\f\6\n\0\a\1\35\0\1+\0\360\27\201\375\300\0\0\0\17\205\210Z\0\0M\215\245\b\4\0\0M9fH\17\206fZ\0\0I\2... context...: body of "/gnu/store/mmrax3f1vx3c8ih9hhgffpvka6chk96w-racket-8.0/share/racket/pkgs/gui-lib/mred/private/wx/gtk/utils.rkt" body of "/gnu/store/mmrax3f1vx3c8ih9hhgffpvka6chk96w-racket-8.0/share/racket/pkgs/gui-lib/mred/private/wx/platform.rkt" ``` The error message is referring to an internal Chez Scheme primitive. The report at seems to me to be an alternative manifestation of the same problem: the hexdump there is useful and explains why I did see some of the store path when I had attempted to investigate using `strings`. So there seem to be at least three bad cases: 1. The grafter can mangle .zo files so that Racket can't read them at all. 2. The grafter can miss store references in .zo files, so Racket could end up using the ungrafted versions. 3. With a garbage collection, Racket could try to use the ungrafted versions but fail to find them at runtime. >> So, we stop patching Racket sources with absolute paths to store >> files (i.e. for foreign libraries to dlopen). >> Instead, we put them in a data file that doesn't get compiled or, >> in one case, embed it in C. > > That solves the problem for Racket itself, but wouln’t Racket libraries > have the same issue? Because the problems are triggered by grafting, they only affect packages that have been patched by Guix. For now, Guix doesn't have the ability to install more Racket packages. In the longer term, the one-sentence answer is that it's always possible we might find and switch to a better approach, but this seems most promising. (A bit more of my current thinking on that toward the end.) > Would it be an option to instead turn off compression and keep doing > things as usual? In theory, this should be possible. I see two significant downsides: 1. Compiled code would be much larger—maybe twice as big—and, if I recall correctly, load times would be worse, too. With the move to Racket CS, existing Racket code moved from a world of small and cheap bytecode to a world of machine code: the default compression settings have been tuned to avoid an unacceptable worsening of binary size and load time. 2. Racket very intentionally does not specify the format of zo files, and indeed the details routinely change: I think there have now been 14 such changes on Git since the 8.0 release in February. Continuing to patch zo files sets us up for future breakage, and it seems like it would be especially easy for maintainers of the Guix package to miss the implications of such changes to low-level implementation details (as I did!). For example, Chez Scheme seems to make compression options programmer-configurable even within a single object file: if Racket exposed such options, we could well end up here again. More broadly, I think the best strategy for Racket packaging will be, as much as possible, to use Racket's supported configuration features rather than Guix-specific hacks. This seems especially viable since Racket has been willing to accept unobtrusive patches upstream that help things go smoothly for Guix, e.g. with 8.1 we should no longer need any patches to the build scripts: we're all friendly, parentheses-loving folks. For another example, it looks like existing "racket-store-checksum-override.patch" fixes a previous issue discussed in caused by grafting zo files: I hope this change will also let us remove that patch, though I'd want to test more before proposing we drop it. So these problems aren't fundamentally new; they just have an additional symptom since the change to Racket CS by default in Racket 8.0. If we can fix the root problem of violating Racket's assumptions by patching zo files, we should be able to stop hunting down symptoms. Rather than using "config.rktd", an alternative approach would be to set things up so that `dlopen` would find the foreign libraries, perhaps via `LD_LIBRARY_PATH`. This has some intriguing possibilities: I could imagine Guix providing an alternate `dlopen` implementation that might be useful beyond Racket. Nix apparently configures some things via `LD_LIBRARY_PATH`, but their approach (as I understand it) relies on generating Racket scripts around all Racket-generated executable, which causes other problems. There should be workarounds, but it seems better to avoid going down that road if we can. Finally, here's a sketch of how `guix import racket` and such might work. Racket's package system has a concept of "package scope", so that "installation" scope can coexist with narrower scopes (mostly per-user scope, though there are more complex possibilities). Right now, Guix puts installation scope in the read-only store, which basically corresponds to how other package managers put it in root-owned places, except Guix can't write to the store to install additional packages. I'm still at the information-gathering stage, but my current thinking is that the hypothetical `racket-build-system` should basically take the source package and turn it into what Racket calls a "built" package ready to be installed in `static-link` mode, which includes compiling the code and building the docs (which can involve quite a lot, e.g. ray tracing icons). Then a profile hook could knit together all of the Racket packages into an installation package scope. For packages that depend on foreign libraries, this would be a chance for Guix to add the necessary paths to the "config.rktd" for the installation. -Philip