From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher Baines Subject: Re: Package file indexing Date: Fri, 10 Jan 2020 13:30:48 +0000 Message-ID: <878smf1lrr.fsf@cbaines.net> References: <20190314204941.GA21065@jasmine.lan> <87mulx9kuv.fsf@nckx> <87zhpx846u.fsf@ambrevar.xyz> <87bm21y2s2.fsf_-_@gnu.org> <87imw7cpe7.fsf@bababa.i-did-not-set--mail-host-address--so-tickle-me> <87pnqdhkpf.fsf@gnu.org> <87imlt3hr2.fsf@ambrevar.xyz> <87png11xgi.fsf@cbaines.net> <87tv5cpypp.fsf@gnu.org> <874kx4anch.fsf@ambrevar.xyz> <87blrc1sos.fsf@cbaines.net> <87h813wktw.fsf@ambrevar.xyz> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:46917) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ipuNQ-0006Ch-1X for guix-devel@gnu.org; Fri, 10 Jan 2020 08:31:01 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ipuNN-000059-Pz for guix-devel@gnu.org; Fri, 10 Jan 2020 08:30:59 -0500 In-reply-to: <87h813wktw.fsf@ambrevar.xyz> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane-mx.org@gnu.org Sender: "Guix-devel" To: Pierre Neidhardt Cc: guix-devel@gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Pierre Neidhardt writes: > Christopher Baines writes: > >> So, to elaborate a bit more on the architecture I've had in mind for >> dealing with the actual nars=E2=80=A6 >> >> I see the scope of the Guix Data Service extending as far as what nars >> are available for outputs, and what outputs are associated with each >> revision, but I don't think it should store the actual nar files. >> >> What you could have is another service, which subscribes to the Guix >> Data Service to find out about new revisions and nars (from build >> servers). When this new service finds out about Guix revisions, it would >> ask this Guix Data Service for all the outputs, and store this away in a >> database. When it finds out about nars, it would download them, and >> maybe extract out the list of files. >> >> I think this setup would allow this new service to construct a file >> containing information about all files in all the outputs for a >> revision, which it has nars available for. This file could then be >> downloaded, and searched through when you want to find which output >> contains a file. > > Tell me if I understood you correctly: in this scenario we would modify > the Guix derivation process to store the file list in the nars. Is this = correct? Not quite. As Ludo mentioned, you can trivially extract out the file list from nar files already (like guix archive -t). So this new service I'm thinking about which stores the nar files, would be able to read the list of files from the nar. > Question about the Guix Data Service: I suppose that the information abou= t the outputs > of a given revision is built incrementally, i.e. as they get published > by the build farm. Is this correct? So the information about what outputs the derivations produce is completely available once the revision has been loaded. However, nar files for those outputs become available as build farms build them, and the Guix Data Service hopefully finds out about these soon after. Also, since the build could be non-deterministic, it's possible for multiple different nar files to be generated for the same output (maybe even containing different files in the extreme case!). > If so, then the file index service needs to update the database > incrementally as well. So we need some entry point to fetch the > information delta between now and the last time we fetch the information. Yeah, you might fetch the file list database, and then another derivation is built, revealing which files are in the associated outputs. A new database can then be constructed containing that additional information. In the trivial case, the new file could be downloaded to replace the old one. This could maybe be optimised by just downloading the changes, maybe by using something like xdelta. > Please correct me if I got it all wrong! :D All great questions, hopefully I've managed to clear things up! --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEEPonu50WOcg2XVOCyXiijOwuE9XcFAl4YfIhfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcACgkQXiijOwuE 9XexbxAAuiB1RBLPGsxc28jmt/Xing5D0lvP6JYD4kxWUSJG2JyrxZZfFLBcbn5x p4sJYBTlNoH7C29kb9OP9hiq8RRpyHtUpXKo71qwd/B4hksGL956+qYG2CXK4pj4 1zrkQnjWoF1ue5q8YS/Kk/nKPNcJ1IFjKw1v+6vdABXhu+XCuASxyTlxT/WhVvi8 O9CT4hrJyiiCd9pxo09vbNRGuHk+qho15jFA6RSIPCgzQ7eHqrtFXHIfYb21vfYj 6FDfq8JgMJShAo712klO/v+ifSM9YaR6LCkwF43pnJz/jMJKZtnMWh0uqGJydPEQ 6oX4IWQ1XjAkz9U+NVIwTw3IUP5JgrlSrKTVJGcJW5byevd7hBK/P/HIJYJEXkMS k/+C/Bpx32ktAqQv82IYUDB8aIuvo/7bkVLqxrlvqyAT2mQhe0+zWCPB6NvoZEF8 KYkR3Syz9vO+g7ztTy4eDIx/cmlaRizs+Tls3ask6ku0OicIIDJ91F1sGNVuCzgH 5o/qqtE57eo7FqKx/ViAdB40hDxioIp+4frfe4V237fFUfRnAY1NxDWv9t9inX8n 2A2p8/z6qwH7FsH94Wz/t1woOrXo5PvIXWz2rr62T5K6wGsfJiJRgDNJrc6w/8ia S/wMncataQ3MBXswUiUUOWUZd7B6zYJALwcbcJ5Lqm9/sZ2ZSQc= =8aUJ -----END PGP SIGNATURE----- --=-=-=--