Re: Package file indexing

["Ludovic Courtès" <ludo@gnu.org>]

Hello!

Christopher Baines <mail@cbaines.net> skribis:

> Pierre Neidhardt <mail@ambrevar.xyz> writes:
>
>> Hello again!
>>
>> I'm resurrecting this since I've just started working on this as part of
>> the NGI application! :)
>>
>>>>> Internally it’d call ‘guix substitute’ to fetch the file index from
>>>>> the substitute server, check its signature, cache it locally, and then
>>>>> look up the file.
>>
>> What about storing the file listing in the narinfo instead?
>> Is this doable?  If so, then it should be quite simple to implement, it
>> would basically mimic "guix size."
>
> I haven't followed this thread particularly well, but at least from my
> recent experience messing with nar and narinfo stuff in the Guix Data
> Service, I'd be cautious about trying to adapt narinfo files for this
> purpose.
>
> It seems to me that the narinfo file is a good at capturing the
> information about the hash, size, location and signature of the
> nar. They're small, and human readable.
>
> I think making information about the contents of Guix store items more
> available is great, but even in the average case, it seems like that's
> too much information to pack in to a narinfo file. Imagining a manifest
> in abstract, having a list of the files and directories as well as the
> hashes and sizes of the files could be really useful, but that for most
> store items, all that information is much larger than the narinfo
> files. A separate file might be more flexible.

I concur!  Actually, there’s a separate file already: the nar itself.

  wget -q -O - https://ci.guix.gnu.org/nar/lzip/1gyi4i5lbpr7apm74p08dwy11fhzh4j7-sed-4.7 \
     | lzip -d | guix archive -t

But…

> Additionally, now that I'm thinking about this, having information about
> each store item is great, but if you want to know which store items in a
> particular revision of Guix contain files called foo, then it might take
> a while to download and search them all. Having something that's focused
> around the packages in a channel, and acts as an index for all of the
> files in all of the available outputs might be faster to search, by
> doing the combining of the data upfront.

… I agree.  I think file search has to be a service providing access to
a fast database.

I think the Guix Data Service is a good fit since it knows about
packages, derivations, commits, and how they map to each other.  :-)  It
could download nars and do the equivalent of ‘guix archive -t’ to get
the list of file names.

There’s an argument that it would be nice if file search were
implemented as part of ‘guix publish’ because that would immediately
benefit everyone without going through complex setups.  However, ‘guix
publish’ wouldn’t really know what to index upfront, or maybe it could
index lazily like it does with “baking”.

Food for thought!

Ludo’.