From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id kNRtCkdxuWImOAEAbAwnHQ (envelope-from ) for ; Mon, 27 Jun 2022 10:58:47 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id SOktCkdxuWI6UwAAauVa8A (envelope-from ) for ; Mon, 27 Jun 2022 10:58:47 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C9D4F3FC57 for ; Mon, 27 Jun 2022 10:58:46 +0200 (CEST) Received: from localhost ([::1]:54440 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o5kZy-0001ti-0g for larch@yhetil.org; Mon, 27 Jun 2022 04:58:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47464) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o5kUC-0003kT-OK for guix-devel@gnu.org; Mon, 27 Jun 2022 04:52:48 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:34048) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o5kUC-0005Do-AJ; Mon, 27 Jun 2022 04:52:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=XMpwqeMykbzww11PaPgL22GiOIX4ioppR2jYWNWYZQ8=; b=qS7b55VeOpB6peua/1le UeMvEhR5oessnJK3gyya2pFM9wwDik1nOfSvSDfuy1aU9OeZwlAuK+1GYXcthA6aRPXAUsnjOLNA4 5wqpRLu3MNXsndx29FNK/nzRCNUnoRtegFKaprMyiObTGbyJxcgCXCLqfwTzwkWV07bUvGzyA6RLP rI1qaIfgs2OkYcN9WKkvgYCnzv/a/Yq0KOsDxzc/F2wlQGIRKtkfHd3faKMpVrsWNPyQr8U4rwwR/ rKC5O9acsZi9KwATmY7BV2q9i+aTjvhcMtfBitCtHzDnfNmgdvfcswLGrrydVK9UlR+NdQMIG1Bfg KfMPeoTGVEvGRA==; Received: from [193.50.110.235] (port=32888 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o5kUB-00064Q-Pw; Mon, 27 Jun 2022 04:52:48 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Cc: guix-devel@gnu.org Subject: Re: Experimental nar-herder support for serving fixed output files by hash References: <87o7yilbnd.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Nonidi 9 Messidor an 230 de la =?utf-8?Q?R=C3=A9volu?= =?utf-8?Q?tion=2C?= jour de l'Absinthe X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 27 Jun 2022 10:52:45 +0200 In-Reply-To: <87o7yilbnd.fsf@cbaines.net> (Christopher Baines's message of "Fri, 24 Jun 2022 09:10:00 +0100") Message-ID: <87mtdyjxnm.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1656320326; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=XMpwqeMykbzww11PaPgL22GiOIX4ioppR2jYWNWYZQ8=; b=RFinpDr3hp4ACNHVgpZA1krGIO7H/zifx2+DkGEQ9VR2R6CPFffy0bqMYEkQUMQiOzPe86 sSZC9LAJXEzRIhqhl+Qew8lBxv9nZDIbGGa8iXQ01WEZ8HTWCgYlIbyI0kqq0C0h0+Hh2L Os9zRCzAG/sM8AnGiypW6FDrR6JJP5zd5WHgEhSh74vzMsjMPcVZqIk9wj18c1dZCm3mX8 3jTa/vul7xgvpDmSczvFJi8Kvty4hRMzTuYjrT9MxsQfmt204FzEt1tgUxoZ7uFkkU4PEE ICdUJBdwIJ85iL5bEF4ChOgUyUm9iGpaffgsHPQkhFRrFWAyMXpaM4aRvwBmeA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1656320326; a=rsa-sha256; cv=none; b=ZRH2eBTQbKPN7ItEzq/PVbdmqzAOMN60xFnJD094evR6wCVRAFWuN/YpomdHwjk2jX593j xwgkBGQ9OIOPLP4InPV2pImjCeRsI8vt0+pvzwYet9gzjd4VMEhAzYRcy2lx4DEtKUEcN0 SgzpXPia0+Qy9TBXfe51KHyhG3CW/jYfhjuesCqvNMr3c5LQc/DYQjjehjUgg0SAfBKQg3 hiAF2Nip/2Vz+LEdIa+aMU5kkT/ThhhDmGhtOcepcnJL696T1HrtCO6W3NQ5zIKtHC4Q56 pOp3ETDuCoo6TDRrrpKGRpeasF99muQEzc3pFPYOcrtGgOVPJzxxbP23KWTUBg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=qS7b55Ve; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -5.15 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=qS7b55Ve; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: C9D4F3FC57 X-Spam-Score: -5.15 X-Migadu-Scanner: scn1.migadu.com X-TUID: 8mdsazYm6w9a Hello, Christopher Baines skribis: > With the nar-herder, the implementation is a little trickier. Since the > nar-herder manages a collection of nars, rather than serving things from > the store, it might have the file being requested but it's inside a > probably compressed nar file. So, to respond to these requests, the > nar-herder has to take the relevant nar file and then read the file out > of it. I've now got an initial implementation of this: > > https://git.cbaines.net/guix/nar-herder/commit/?id=3D042f49e5fb52ea844e= d5d29c17b26fbc8ad49f0e Interesting. > The code isn't great, there's some difficulty in extracting the single > file from the nar, but the biggest problem is a limitation in the guile > fibers web server. Currently, responses have to be read in to memory, > which is fine for we pages, but not great if you're trying to serve > files which can be multiple gigabytes in size. This also means that the > first byte of the response is available when all the bytes are > available, so the download is slow to start. That, and in practice a cache (with some eviction mechanism) would be necessary so nars don=E2=80=99t need to be extracted every time and so we c= an use sendfile(2). > With all of that said though, it does seem to work. For testing, I've > enabled it on bishan, which serves the bordeaux.guix.gnu.org collection > of nars. It only has IPv6 connectivity, so you'll only be able to try > this out if you've got an IPv6 support locally: > > https://bishan.guix.gnu.org/file/0ad-0.0.25b-alpha.tar.xz/sha256/1p9fa8= f7sjb9c5wl3mawzyfqvgr614kdkhrj2k4db9vkyisws3fp Nice! > In terms of next steps, there's some things to do with improving the > implementation, but it would be good to hear if this is actually > worthwile? IWBN to share as much code as possible with =E2=80=98guix publish=E2=80=99,= which has great test suite coverage and is being hammered every day. Clearly the bit about extracting nars is specific to the nar-herder though, so that may prove difficult. Thoughts? Ludo=E2=80=99.