From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id 6N5lMniF9V/gEQAA0tVLHw (envelope-from ) for ; Wed, 06 Jan 2021 09:40:08 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id mPRELniF9V/QMAAAbx9fmQ (envelope-from ) for ; Wed, 06 Jan 2021 09:40:08 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8D623940414 for ; Wed, 6 Jan 2021 09:40:08 +0000 (UTC) Received: from localhost ([::1]:49748 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kx5IV-0003s7-Hc for larch@yhetil.org; Wed, 06 Jan 2021 04:40:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:44788) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kx5IM-0003rx-NU for guix-devel@gnu.org; Wed, 06 Jan 2021 04:39:58 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:57044) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kx5IM-0000dq-GK; Wed, 06 Jan 2021 04:39:58 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=41404 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kx5IL-00013y-VX; Wed, 06 Jan 2021 04:39:58 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Miguel =?utf-8?Q?=C3=81ngel?= Arruga Vivas Subject: Re: Identical files across subsequent package revisions References: <87wnx9wlea.fsf@gnu.org> <878s9oy8f7.fsf@gmail.com> <87y2hirih4.fsf@gnu.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 17 =?utf-8?Q?Niv=C3=B4se?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 06 Jan 2021 10:39:56 +0100 In-Reply-To: <87y2hirih4.fsf@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s=22'?= =?utf-8?Q?s?= message of "Sun, 27 Dec 2020 23:29:11 +0100") Message-ID: <8735zepfo3.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guix Devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -2.84 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 8D623940414 X-Spam-Score: -2.84 X-Migadu-Scanner: scn0.migadu.com X-TUID: fayKH1Aswk2v Hi, Ludovic Court=C3=A8s skribis: > It could go along these lines: > > 1. GET /digest/xyz-emacs; the digest contains a list of file/hash > pairs essentially; > > 2. traverse digest and hardlink the files already in /gnu/store/.links > to the target directory; > > 3. pipeline-GET the remaining files and install them. > > Like you write, since we already have deduplication, part of it comes > for free. I prototyped that during the holidays and pushed the result as =E2=80=98wip-digests=E2=80=99 (rough on the edges!). It works essentially as written above. =E2=80=98guix substitute=E2=80=99 f= etches digests at the same time as it fetches narinfos so it can keep them in cache, which in turn means that when you install things it already has info locally to determine which files it already has and which ones it needs to fetch (in the best case, the substitute phase simply hard-links files locally without connecting to the server!). One of the open issues is compression. =E2=80=98guix publish=E2=80=99 now = serves individual files, but obviously they should be compressed somehow. For nars, we have /lzip, /gzip, etc. URLs, which gives clients more freedom on the choice of compression but is also quite heavyweight. For individual files, I=E2=80=99m thinking we could use =E2=80=98Accept-Encodin= g=E2=80=99 to let the client declare which compression formats it supports and then let =E2=80=98= guix publish=E2=80=99 choose the one it prefers. In the worst case, when none of the files are available locally, we=E2=80= =99d be downloading all the individual files, which is probably more data than a single compressed nar. I guess =E2=80=98guix substitute=E2=80=99 could fal= l back to nar download when too little is available locally. The downside compared to using ERIS or IPFS is that it=E2=80=99s really a h= ack. The upside is that it=E2=80=99s little work and it=E2=80=99s efficient sinc= e we already have that content-addressed file store in /gnu/store/.links. Thoughts? Ludo=E2=80=99.