[bug#50384] [PATCH v4] Optimise search-patch (reducing I/O)

["Ludovic Courtès" <ludo@gnu.org>]

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

> Ludovic Courtès schreef op di 21-09-2021 om 18:55 [+0200]:

[...]

>> Looking at the big picture, what I’d like to have is a package
>> derivation cache designed in such a way that “guix install foo” wouldn’t
>> even need to load any package module on a cache hit.  That’d make a
>> noticeable difference performance-wise, that’s another level of
>> complexity…  (I have a rough design in mind that we could discuss.)
>
> This ‘package derivation cache’ seems an interesting idea to pursue,
> though I wonder what could be used as ‘keys’ in the derivation cache.

The existing package (used by ‘find-packages-by-name’ & co.) already has
keys: the module and variable name of a top-level package definition.

More precisely, there’d be two keys: (1) a hash of the channel instance
commits being used (as you wrote), and (2) the module/variable name.
That means we’d only keep packages with a public top-level definition in
the cache.


Rough design sketch:

The idea I have in mind would be to have <package> be a sub-record-type
of <proto-package>, where <proto-package> would contain only the
metadata currently provided by ‘fold-available-packages’ (mainly: name,
version, module/variable).

<proto-package> would have a gexp compiler that looks up the
package/derivation mapping in ~/.cache/guix and either return it
directly (cache hit) or look up the corresponding <package>, do all the
work, and add an entry in the cache (cache miss).

Challenges include: arranging so (guix records) can deal with record
type inheritance, deciding what to do with grafts, and probably more.

>> Ludovic Courtès <ludo@gnu.org> skribis:
>> 
>> > > +;; repeated 'stat' calls.  Allow computing the hash of the file in advance,
>> > > +;; to avoid having to send the file to the daemon when it is already interned
>> > > +;; in the store.
>> > >  (define-record-type <local-file>
>> > > -  (%%local-file file absolute name recursive? select?)
>> > > +  (%%local-file file absolute name sha256 recursive? select?)
>> > >    local-file?
>> > >    (file       local-file-file)                    ;string
>> > >    (absolute   %local-file-absolute-file-name)     ;promise string
>> > >    (name       local-file-name)                    ;string
>> > > +  (sha256     local-file-sha256)                  ;sha256 bytevector | #f
>> > 
>> > Could we store the result of ‘fixed-output-path’ rather than the SHA256,
>> > while we’re at it?
>
> Embedding the result of ‘fixed-output-path’ in the .go might be problematic
> from a closure size perspective, as that would create additional references in the
> store items of guix.

Yes, I wonder, not sure about that.

> As it is only <1%, I would prefer trying the ‘package derivation cache’
> first, as it seems to have more potential.

Alright.

Thanks,
Ludo’.