From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id 6Bt9KfJytWImGwAAbAwnHQ (envelope-from ) for ; Fri, 24 Jun 2022 10:16:50 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id wGBCKfJytWISngAAauVa8A (envelope-from ) for ; Fri, 24 Jun 2022 10:16:50 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2EE244145D for ; Fri, 24 Jun 2022 10:16:50 +0200 (CEST) Received: from localhost ([::1]:47100 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o4eUj-0001Pj-AN for larch@yhetil.org; Fri, 24 Jun 2022 04:16:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37088) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o4eUA-0001PQ-Gx for guix-devel@gnu.org; Fri, 24 Jun 2022 04:16:14 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:41749) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o4eU7-0003hx-Vo for guix-devel@gnu.org; Fri, 24 Jun 2022 04:16:13 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:54d1:d5d4:280e:f699]) by mira.cbaines.net (Postfix) with ESMTPSA id 3AB3427BBE9 for ; Fri, 24 Jun 2022 09:16:10 +0100 (BST) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 2b43a679 for ; Fri, 24 Jun 2022 08:16:08 +0000 (UTC) User-agent: mu4e 1.6.10; emacs 28.1 From: Christopher Baines To: guix-devel@gnu.org Subject: Experimental nar-herder support for serving fixed output files by hash Date: Fri, 24 Jun 2022 09:10:00 +0100 Message-ID: <87o7yilbnd.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1656058610; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=ISDx8yoHo+3zDgun+Vz5B3MiHB2zCDKBs+H2oTVQzGo=; b=fyokqPxUjfHsE0T6az1nLPTLT4nRwi0XLBe22g28cvKcpj4F4rJCd4/QBQC74vBarcud6P Yz5J+IKN2whY9M8fLcmKCAFrlWRzEjkKe4n2wJpVRRn2sOH2kq5eQorF7k82Gj5fLrP7w9 cD32XyFjmXOvNs76cvt7EDoXYJpt/OMs0R5dQbmhMfy4LQwHKmf8ImxRlPQnf+7HknGLdY 5buRvRgP1JdEbwl6jq2LAqIJGQi87/oW3P96Npn0tFWTEe91Dz/8n7EG2vx/3dE0nHnwXE RK3/4X1Cq3UynemILJHbwO661inmVwcdJMV99G/xkD3LiWHE6vNuKkdaEgRaOw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1656058610; a=rsa-sha256; cv=none; b=e5QlrJSIFz8QSR6VP+Z5dZoHF1I1ZyBc+crIL1y4X2eT/hQqE7tudpvPRgFeWsUQcQQhwh K19xNFKs+grqd62G25AanXoiiYfEmRbFXFUyDcQ+wYYUxv0Ic5xRstAMixJBWGflYD2ZLc qedp+9XywhH844hgzZ6/E/lOrtXa4pjjR+3Ct/HaDIofnaPihxPCP9NqH/j161+Fl4lGGn 4U54W8pgTucKHvCLml1DxR9WYndt0cwDVkQ+E40hku/NEARf5uFj8qQ8tg6cZoyWMPknx1 vsh+bVaOkRNiaM/RMDhidbAou+XxwhIzFmlEoCY4TdPXKCIKOyTLGy2DGxsuZQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.86 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 2EE244145D X-Spam-Score: -4.86 X-Migadu-Scanner: scn1.migadu.com X-TUID: ZF+jmRfqpIqG --=-=-= Content-Type: text/plain Hey! The nar-herder helps with managing a collection of nars. There's some overlap with the functionality of guix publish in that both tools can serve narinfo files which is a key part of providing substitutes. One thing that guix publish does aside from serving narinfo files is providing access to files in the store produced by fixed output derivations. Package sources are fixed output derivations, so this basically means single file package sources, like tar files. Using ci.guix.gnu.org as an example, this looks like: https://ci.guix.gnu.org/file/0ad-0.0.25b-alpha.tar.xz/sha256/1p9fa8f7sjb9c5wl3mawzyfqvgr614kdkhrj2k4db9vkyisws3fp You can request a file from the store, if you know it's name and hash. In guix publish, this works by computing the /gnu/store/... filename for a file with this name and hash, and then serving it if it exists. Additionally, on ci.guix.gnu.org, there's some NGinx caching in front so some files may be still available, even if they've been removed from the store. With the nar-herder, the implementation is a little trickier. Since the nar-herder manages a collection of nars, rather than serving things from the store, it might have the file being requested but it's inside a probably compressed nar file. So, to respond to these requests, the nar-herder has to take the relevant nar file and then read the file out of it. I've now got an initial implementation of this: https://git.cbaines.net/guix/nar-herder/commit/?id=042f49e5fb52ea844ed5d29c17b26fbc8ad49f0e The code isn't great, there's some difficulty in extracting the single file from the nar, but the biggest problem is a limitation in the guile fibers web server. Currently, responses have to be read in to memory, which is fine for we pages, but not great if you're trying to serve files which can be multiple gigabytes in size. This also means that the first byte of the response is available when all the bytes are available, so the download is slow to start. With all of that said though, it does seem to work. For testing, I've enabled it on bishan, which serves the bordeaux.guix.gnu.org collection of nars. It only has IPv6 connectivity, so you'll only be able to try this out if you've got an IPv6 support locally: https://bishan.guix.gnu.org/file/0ad-0.0.25b-alpha.tar.xz/sha256/1p9fa8f7sjb9c5wl3mawzyfqvgr614kdkhrj2k4db9vkyisws3fp In terms of next steps, there's some things to do with improving the implementation, but it would be good to hear if this is actually worthwile? ci.guix.gnu.org is already used as a content addressed mirror, although given that there's a push to keep the store on berlin small, I'm not sure how many files are actually available, or will be available in the future. There's a 50G NGinx cache, of which I think 7G is used, so this feature is probably being used a bit at least. In terms of what enabling this for the bordeaux.guix.gnu.org collection of nars would look like, I think there's roughly 50,000 tarballs taking up at least a tebibyte of space which would be downloadable. These are available as substitutes, but maybe there's value in making them available this way as well? Let me know what you think? Thanks, Chris 1: sqlite> SELECT SUM(size) FROM narinfo_files WHERE url LIKE '%.tar.%'; 1102376493623 sqlite> SELECT COUNT(*) FROM narinfo_files WHERE url LIKE '%.tar.%'; 48326 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmK1csZfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XcPQA//V8nXdqxU5vhgt/WVrQ4Db79w9nWxPdP4 O9SsnVfKifRmRHLPSMoogR0fufTV7Ol6xppj6dfQzX53NaRLEeyEam+QAQ5NnFe3 yUtAvm36S1rZ2xzxNIRl3aEG5WKWNbJ6RIGo61/NcddJ7Q7hJr7hxglCJHV7sJjc k6Nh/BVtNS0KfB9o9uxC/B5EPdz7v6PlQLWM7m9OMDzvMoHzxPkV+joO2W0Vs55H xzoRzhmpcwLgUBs9hsw4NhN9Ke63tlu+eWTU2H+YZYQLcQf4Hj/lk3WvSY6mrb9V cuUAtOLmu3m9bdlNt4OSg20iKzxfGxWlHYXCtChDa67RHzfVcQysRjCkH4zBY62J +5dAoHt/Y2stXi8Igd1DGLaFn2iHlXoZFml6oO9M8uyNCTJhhvJVV1xwu/Z2uDLb UUbQvs1hlpqSzjyzL+Xy/9UTd6VM2Bpb17QSur0VfHDN3ZekWLRRPguDAG+H//hz ZZtUrTrD+tOUxNNKcwH0uEd5nlF1hNwDtdwjaqDWmC3UZQww0mut51b82AEFZuEi n22ftaTa2uTdmQcH0l10/knOOu+pFG3ygHrkGa/QRtZFSY4KpJlAubM7ACiw24/1 YhDefrI+RmMRwtBmWU0stQrO4wNLfi5+WgRiLXV1qWzfL0KFhRWT4I0ZgwYOf7LP uNfCW6ERa5E= =ZmUE -----END PGP SIGNATURE----- --=-=-=--