From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: Re: Package file indexing Date: Fri, 03 Jan 2020 12:26:58 +0100 Message-ID: <87tv5cpypp.fsf@gnu.org> References: <20190314204941.GA21065@jasmine.lan> <87mulx9kuv.fsf@nckx> <87zhpx846u.fsf@ambrevar.xyz> <87bm21y2s2.fsf_-_@gnu.org> <87imw7cpe7.fsf@bababa.i-did-not-set--mail-host-address--so-tickle-me> <87pnqdhkpf.fsf@gnu.org> <87imlt3hr2.fsf@ambrevar.xyz> <87png11xgi.fsf@cbaines.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:36988) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1inL6b-00047T-Gl for guix-devel@gnu.org; Fri, 03 Jan 2020 06:27:02 -0500 In-Reply-To: <87png11xgi.fsf@cbaines.net> (Christopher Baines's message of "Thu, 02 Jan 2020 19:15:41 +0000") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Christopher Baines Cc: guix-devel@gnu.org Hello! Christopher Baines skribis: > Pierre Neidhardt writes: > >> Hello again! >> >> I'm resurrecting this since I've just started working on this as part of >> the NGI application! :) >> >>>>> Internally it=E2=80=99d call =E2=80=98guix substitute=E2=80=99 to fet= ch the file index from >>>>> the substitute server, check its signature, cache it locally, and then >>>>> look up the file. >> >> What about storing the file listing in the narinfo instead? >> Is this doable? If so, then it should be quite simple to implement, it >> would basically mimic "guix size." > > I haven't followed this thread particularly well, but at least from my > recent experience messing with nar and narinfo stuff in the Guix Data > Service, I'd be cautious about trying to adapt narinfo files for this > purpose. > > It seems to me that the narinfo file is a good at capturing the > information about the hash, size, location and signature of the > nar. They're small, and human readable. > > I think making information about the contents of Guix store items more > available is great, but even in the average case, it seems like that's > too much information to pack in to a narinfo file. Imagining a manifest > in abstract, having a list of the files and directories as well as the > hashes and sizes of the files could be really useful, but that for most > store items, all that information is much larger than the narinfo > files. A separate file might be more flexible. I concur! Actually, there=E2=80=99s a separate file already: the nar itsel= f. wget -q -O - https://ci.guix.gnu.org/nar/lzip/1gyi4i5lbpr7apm74p08dwy11fh= zh4j7-sed-4.7 \ | lzip -d | guix archive -t But=E2=80=A6 > Additionally, now that I'm thinking about this, having information about > each store item is great, but if you want to know which store items in a > particular revision of Guix contain files called foo, then it might take > a while to download and search them all. Having something that's focused > around the packages in a channel, and acts as an index for all of the > files in all of the available outputs might be faster to search, by > doing the combining of the data upfront. =E2=80=A6 I agree. I think file search has to be a service providing acces= s to a fast database. I think the Guix Data Service is a good fit since it knows about packages, derivations, commits, and how they map to each other. :-) It could download nars and do the equivalent of =E2=80=98guix archive -t=E2=80= =99 to get the list of file names. There=E2=80=99s an argument that it would be nice if file search were implemented as part of =E2=80=98guix publish=E2=80=99 because that would im= mediately benefit everyone without going through complex setups. However, =E2=80=98g= uix publish=E2=80=99 wouldn=E2=80=99t really know what to index upfront, or may= be it could index lazily like it does with =E2=80=9Cbaking=E2=80=9D. Food for thought! Ludo=E2=80=99.