bug#30820: Chunked store references in compiled code break grafting (again)

[ludo@gnu.org (Ludovic Courtès)]

Mark H Weaver <mhw@netris.org> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> ludo@gnu.org (Ludovic Courtès) writes:
>>>
>>>> The recently added glibc grafts triggered issues that, in the end, show
>>>> the return of <http://bugs.gnu.org/24703> (“Store references in 8-byte
>>>> chunks in compiled code”).
>>>
>>> I think that we should generalize our reference scanning and grafting
>>> code to support store references broken into pieces, as long as each
>>> piece containing part of the hash is at least 8 bytes long.
>>>
>>> Here's my preliminary proposal:
>>>
>>> (1) The reference scanner should recognize any 8-byte substring of a
>>>     hash as a valid reference to that hash.
>>>
>>> (2) To enable reliable grafting of chunked references, we should impose
>>>     the following new restrictions: (a) the store prefix must be at
>>>     least 6 bytes, (b) grafting can change only the hash, not the
>>>     readable part of the store name, and (c) the readable part of the
>>>     store name must be at least 6 bytes.
>>>
>>> (3) The grafter should recognize and replace any 8-byte subsequence of
>>>     the absolute store file name.
>>
>> I’m quite reluctant because it would add complexity, it will probably
>> slow things down, and yet it may not handle all the cases, as Danny
>> suggests.
>>
>> Mind you, the GCC patches are not perfect either, but they’re relatively
>> easy to deal with (well, so far at least).  In theory we would need
>> similar patches for LLVM and maybe a couple other native compilers,
>> though, which is obviously a downside, although we haven’t had any
>> problems so far.
>
> We would also need to find a solution to the problem described in the
> thread "broken references in jar manifests" on guix-devel started by
> Ricardo, which still has not found a satifactory solution.
>
>   https://lists.gnu.org/archive/html/guix-devel/2018-03/msg00006.html
>
> My opinion is that I consider Guix's current expectations for how
> software must store its data on disk to be far too onerous, in cases
> where that data might include a store reference.  I don't see sufficient
> justification for imposing such an onerous requirement on the software
> in Guix.

In practice Guix and Nix have been living fine under these constraints,
and with almost no modifications to upstream software, so it’s not that
bad.  Nix doesn’t have grafts though, which is why this problem was less
visible there.

> Ultimately, I would prefer to see the scanning and grafting operations
> completely generalized, so that in general each package can specify how
> to scan and graft that particular package, making use of libraries in
> (guix build ...) to cover the usual cases.  In most cases, that code
> would be within build-systems.

That would be precise GC instead of conservative GC in a way, right?
So in essence we’d have, say, a scanner for ELF files (like ‘dh_shdep’
in Debian or whatever it’s called), a scanner for jars, and so on?
Still, how would we deal with strings embedded in the middle of
binaries, as in this case?  It seems to remain an open issue, no?

I’m interested in experiments in that direction.  I think that’s a
longer-term goal, though, and there are open questions: we have no idea
how well that would work in practice.

Thanks,
Ludo’.