On 19-09-2022 18:26, Josselin Poiret wrote:
> Hi everyone,
> 
> Maxime Devos <maximedevos@telenet.be> writes:
>> Fallbacks might be necessary (not every store item is constructed from a
>> package), but it all sounds doable and efficient.  Also the union could
>> needs to be modified to ignore the .graft-offsets of the union'ed things.
> 
> If I understand the whole thing properly, the daemon already scans for
> references and could register their location in the database, avoiding
> the need for a potentially brittle in-store format as you highlighted,
> although that would require a non-trivial change to both the daemon and
> the database format.

If you do this, fallbacks are still required for old daemons.

Daemon database changes sound much more brittle to me than the simple
.graft-offsets -- the only brittleness I see is the unioning code, but 
that's easy to address.

OTOH, in the ".graft-offsets" proposal, the build process has to do 
reference scanning (even though the daemon will do so as ell later), 
which adds some inefficiency -- so perhaps performance-wise, the 
additional testing for the fallbacks and brittleness might be worth it.

> In any case, I don't think it's necessary to treat packages in a special
> way: any derivation can hold references and thus be grafted.

(Corrected a likely typo derivations->references.)

They indeed don't need to be treated specially -- in my proposal, the 
grafting code wouldn't care whether a store item is of a package or 
something else, the only thing it cares about, is the presence of a 
.graft-offsets file.

However (unless we go for daemon changes), this .graft-offsets still 
needs to be made somewhere -- my proposal was to do so this, in case of 
packages, in an additional phase (which would call a 
'make-graft-offsets' procedure '(make-graft-offsets #$output ...)'.

Then, I suppose this could be extended to non-packages as well (e.g. 
modify gexp->derivation and sexp->derivation to do the 
make-graft-offsets automatically).  However, packages should cover 
almost everything (profiles are mostly a bunch of symlinks and thus 
don't benefit from grafting optimisations much IIUC), and then 
gexp->derivation would insert an additional file without it being asked 
for, which would sound rather surprising to me, so I think that covering 
packages would be sufficient.

Greetings,
Maxime.