Racket 8 and store references (was [security] Chunked store references in .zo files in Racket 8 #47614)

Racket 8 and store references (was [security] Chunked store references in .zo files in Racket 8 #47614)

[Jack Hill]

On Tue, 6 Apr 2021, Mark H Weaver wrote:

> Anyway, I doubt that imposing such a limitation would adequately solve
> the problem here of chunked references in Racket 8, because I suspect
> that Racket 8 could split store references at arbitrary points in the
> string.  I doubt that we can safely assume that the hash component of
> store references will be stored contiguously in *.zo files.

Mark and everyone,

I wanted to spin off a subthread on guix-devel, to make you aware of 
another problem that we've run into with reference in .zo getting mangled: 
https://issues.guix.gnu.org/47180

Best,
Jack

Re: Racket 8 and store references (was [security] Chunked store references in .zo files in Racket 8 #47614)

[Philip McGrath]

Indeed, I expect this is a more precise diagnosis of the same problem. 
My patch in https://issues.guix.gnu.org/47180 solves it by putting the 
store references (search paths for foreign libraries) in a configuration 
data file that isn't compiled, so they don't end up in .zo files in the 
first place.

The .zo format is intentionally undocumented and subject to breaking 
change, including from different compilation options. At a minimum, a 
change to the Racket version number signals a breaking change to 
compiled code (e.g. Git is now at 8.0.0.13, so 13 breaking changes since 
the release). Internally, I don't know all the details, but the normal 
8.0 .zo format has a Racket layer around the Chez Scheme object format, 
which seems to be very complex: it looks like it supports 
user-configurable compression at the granularity of the individual 
object within an object file. So it seems much better to avoid rewriting 
.zo files altogether.

-Philip

On 4/6/21 9:20 PM, Jack Hill wrote:
> On Tue, 6 Apr 2021, Mark H Weaver wrote:
> 
>> Anyway, I doubt that imposing such a limitation would adequately solve
>> the problem here of chunked references in Racket 8, because I suspect
>> that Racket 8 could split store references at arbitrary points in the
>> string.  I doubt that we can safely assume that the hash component of
>> store references will be stored contiguously in *.zo files.
> 
> Mark and everyone,
> 
> I wanted to spin off a subthread on guix-devel, to make you aware of 
> another problem that we've run into with reference in .zo getting 
> mangled: https://issues.guix.gnu.org/47180
> 
> Best,
> Jack
> 

bug#47614: [security] Chunked store references in .zo files in Racket 8 #47614

[Philip McGrath]

Ah, I see the thread for https://issues.guix.gnu.org/47614 wasn't cc'ed 
here:


-------- Forwarded Message --------
Subject: Re: Racket 8 and store references (was [security] Chunked store 
references in .zo files in Racket 8 #47614)
Date: Tue, 6 Apr 2021 21:38:57 -0400
From: Philip McGrath <philip@philipmcgrath.com>
To: Jack Hill <jackhill@jackhill.us>, Mark H Weaver <mhw@netris.org>
CC: guix-devel@gnu.org

Indeed, I expect this is a more precise diagnosis of the same problem. 
My patch in https://issues.guix.gnu.org/47180 solves it by putting the 
store references (search paths for foreign libraries) in a configuration 
data file that isn't compiled, so they don't end up in .zo files in the 
first place.

The .zo format is intentionally undocumented and subject to breaking 
change, including from different compilation options. At a minimum, a 
change to the Racket version number signals a breaking change to 
compiled code (e.g. Git is now at 8.0.0.13, so 13 breaking changes since 
the release). Internally, I don't know all the details, but the normal 
8.0 .zo format has a Racket layer around the Chez Scheme object format, 
which seems to be very complex: it looks like it supports 
user-configurable compression at the granularity of the individual 
object within an object file. So it seems much better to avoid rewriting 
.zo files altogether.

-Philip

On 4/6/21 9:20 PM, Jack Hill wrote:
> On Tue, 6 Apr 2021, Mark H Weaver wrote:
> 
>> Anyway, I doubt that imposing such a limitation would adequately solve
>> the problem here of chunked references in Racket 8, because I suspect
>> that Racket 8 could split store references at arbitrary points in the
>> string.  I doubt that we can safely assume that the hash component of
>> store references will be stored contiguously in *.zo files.
> 
> Mark and everyone,
> 
> I wanted to spin off a subthread on guix-devel, to make you aware of 
> another problem that we've run into with reference in .zo getting 
> mangled: https://issues.guix.gnu.org/47180
> 
> Best,
> Jack
> 

bug#47614: [security] Chunked store references in .zo files in Racket 8

[Ludovic Courtès]

Hi Philip and all,

Philip McGrath <philip@philipmcgrath.com> skribis:

> Indeed, I expect this is a more precise diagnosis of the same
> problem. My patch in https://issues.guix.gnu.org/47180 solves it by
> putting the store references (search paths for foreign libraries) in a
> configuration data file that isn't compiled, so they don't end up in
> .zo files in the first place.

IIUC, now that <https://issues.guix.gnu.org/47180> has been closed, this
bug is fixed.  Am I right?

Thanks,
Ludo’.

bug#47614: [security] Chunked store references in .zo files in Racket 8

[Philip McGrath]

Hi Ludo’,

On 4/16/21 11:46 AM, Ludovic Courtès wrote:
> IIUC, now that <https://issues.guix.gnu.org/47180> has been closed, this
> bug is fixed.  Am I right?

Yes, it seems to be fixed with respect to Racket, though Mark mentioned 
here in <https://issues.guix.gnu.org/47064#9> (also now fixed) some 
broader implications for the grafting code.

-Philip

bug#47614: [security] Chunked store references in .zo files in Racket 8

[Mark H Weaver]

Ludovic Courtès <ludo@gnu.org> writes:
> IIUC, now that <https://issues.guix.gnu.org/47180> has been closed,
> this bug is fixed.  Am I right?

Yes, I believe so.  All store items referenced by Racket now seem to be
properly grafted, so I'm closing this bug now.

The more general issue with the grafting code--namely that since commit
57bdd79e48, it no longer has the desirable property of checking every
byte against an expected value before rewriting it, which can lead to
silent corruption of files such as Racket .zo files if any store items
references sneak in--can be addressed in another bug report.

     Thanks,
       Mark