Re: store reference detection (was Re: JARs and reference scanning)

[ludo@gnu.org (Ludovic Courtès)]

Hello,

Thomas Danckaert <post@thomasdanckaert.be> skribis:

> From: Chris Marusich <cmmarusich@gmail.com>
> Subject: Re: JARs and reference scanning
> Date: Tue, 25 Apr 2017 22:34:02 -0700
>
>>> I have to admit that I do not know at all how the reference
>>> scanning and
>>> dependency-tracking in the store works.
>>
>> As I understand it, the mechanism used by the Guix daemon (and the
>> Nix
>> daemon) for scanning references doesn't work when the output of a
>> derivation is scrambled in some way (e.g. compressed).  Therefore,
>> if we
>> use JAR files, they should not be compressed.
>
> The code scanning for reference is in nix/libstore/references.cc .  It
> looks for base32 hashes encoded as character strings in the binaries.
>
> Could/should this be generalized somehow?  Apart from compression,
> store filenames encoded with 16-bit character encodings also cause
> problems (can happen with Qt or WxWidgets).  And the are probably more
> cases where it fails.

Really? Qt/WxWidgets “hide” store references by default?

> Does it make sense to expand the reference detecting code (perhaps
> this would lead to too many different special cases?), or maybe
> provide a mechanism to force references when the daemon cannot detect
> them.  I suppose you can always add a text file with a list of store
> items to the output, but maybe there's a more elegant way?

‘propagated-inputs’ is one way to manually specify run-time references.
It works at the package level and not at the store level—that is, the
store item’s references are unaffected by what ‘propagated-inputs’
contains.  It’s usually enough for our purposes though.

In that sense, I think jars are comparable to Python/Perl libraries,
which do not carry reference information by themselves and thus need a
manually-provided ‘propagated-inputs’.

WDYT?

Ludo’.