bug#30820: Chunked store references in compiled code break grafting (again)

[Mark H Weaver <mhw@netris.org>]

ludo@gnu.org (Ludovic Courtès) writes:

> Mark H Weaver <mhw@netris.org> skribis:
>
>> ludo@gnu.org (Ludovic Courtès) writes:
>>
>>> The recently added glibc grafts triggered issues that, in the end, show
>>> the return of <http://bugs.gnu.org/24703> (“Store references in 8-byte
>>> chunks in compiled code”).
>>
>> I think that we should generalize our reference scanning and grafting
>> code to support store references broken into pieces, as long as each
>> piece containing part of the hash is at least 8 bytes long.
>>
>> Here's my preliminary proposal:
>>
>> (1) The reference scanner should recognize any 8-byte substring of a
>>     hash as a valid reference to that hash.
>>
>> (2) To enable reliable grafting of chunked references, we should impose
>>     the following new restrictions: (a) the store prefix must be at
>>     least 6 bytes, (b) grafting can change only the hash, not the
>>     readable part of the store name, and (c) the readable part of the
>>     store name must be at least 6 bytes.
>>
>> (3) The grafter should recognize and replace any 8-byte subsequence of
>>     the absolute store file name.
>
> I’m quite reluctant because it would add complexity, it will probably
> slow things down, and yet it may not handle all the cases, as Danny
> suggests.
>
> Mind you, the GCC patches are not perfect either, but they’re relatively
> easy to deal with (well, so far at least).  In theory we would need
> similar patches for LLVM and maybe a couple other native compilers,
> though, which is obviously a downside, although we haven’t had any
> problems so far.

We would also need to find a solution to the problem described in the
thread "broken references in jar manifests" on guix-devel started by
Ricardo, which still has not found a satifactory solution.

  https://lists.gnu.org/archive/html/guix-devel/2018-03/msg00006.html

My opinion is that I consider Guix's current expectations for how
software must store its data on disk to be far too onerous, in cases
where that data might include a store reference.  I don't see sufficient
justification for imposing such an onerous requirement on the software
in Guix.

Ultimately, I would prefer to see the scanning and grafting operations
completely generalized, so that in general each package can specify how
to scan and graft that particular package, making use of libraries in
(guix build ...) to cover the usual cases.  In most cases, that code
would be within build-systems.

       Mark