From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id eGpCMNn0vWKyRAAAbAwnHQ (envelope-from ) for ; Thu, 30 Jun 2022 21:09:13 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id 6HAwMNn0vWJGmQAA9RJhRA (envelope-from ) for ; Thu, 30 Jun 2022 21:09:13 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id C7E0C37B49 for ; Thu, 30 Jun 2022 21:09:12 +0200 (CEST) Received: from localhost ([::1]:55618 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o6zXM-00070w-1g for larch@yhetil.org; Thu, 30 Jun 2022 15:09:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:47816) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o6zX6-00070m-DN for guix-devel@gnu.org; Thu, 30 Jun 2022 15:08:56 -0400 Received: from mira.cbaines.net ([2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27]:50497) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o6zX3-00077R-Gq; Thu, 30 Jun 2022 15:08:55 -0400 Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:54d1:d5d4:280e:f699]) by mira.cbaines.net (Postfix) with ESMTPSA id CC6A027BBE9; Thu, 30 Jun 2022 20:08:51 +0100 (BST) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 51ce5a2d; Thu, 30 Jun 2022 19:08:51 +0000 (UTC) References: <87o7yilbnd.fsf@cbaines.net> <87mtdyjxnm.fsf@gnu.org> <87bkuel349.fsf@cbaines.net> <878rpe8jmr.fsf@gnu.org> User-agent: mu4e 1.6.10; emacs 28.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org Subject: Re: Experimental nar-herder support for serving fixed output files by hash Date: Thu, 30 Jun 2022 19:28:12 +0100 In-reply-to: <878rpe8jmr.fsf@gnu.org> Message-ID: <87h7420y0u.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=2a01:7e00:e000:2f8:fd4d:b5c7:13fb:3d27; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1656616152; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=JiI8qHMFHGYNnwobuoGUtdwPYkH0H5vRHH/qUIq4b7c=; b=AN2paPbT01cp5aft0HlUQ4Wrxt6VpFZfYWTc6vJXFX2BCPnvzyKFZruLyZVE4kMdQmu8Cf agTsVJ6Q9zSbIV7uyitb/7CfSG4N7/VFjyvMIwdI+UY0mKD2Rl8pzb8BHS7uCKC+7Uqmqd dZ2Rvh68t9KiqFr+BipWBQ4M7sFkooRpPIgiGEUEso5J2PyDQoNLVd6ZY2c3nbslpcYRrn lU7n5VwbFDR8RZpdpheISMcJwvHhR8MADwlo4a/Yyq/r1lQIA3deufhjgKPCDkF6oKyUGG iyoS+k3pTaYqET8xppbXyI0zX96s1ReOJAZgCsxv2BX6ATAl+BujXUAsSgYmMw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1656616152; a=rsa-sha256; cv=none; b=RraGsNZh4kx1S2NsdjnF9R2AtaZ1JPRBFmoUqUmcocC9i5bxg3imbhfy3U57b9OB6W5Scy I7K+d+EwM4oCqZya58M916D9pdG+oiolmQA/z6Pne//V3muI0qz0NTZGfrUsYL//Bj7imN MXwIoN46AjTHI4oVK/JkE8nsN7MZGw414roX039ijm+rGuWqv7hGaoQPC+9orhl++erZFC muV5bFWnJiJpwo5kU8HaH7noHLLLtdGQNelfcNvqQLOIKIXLD5HQQ2RtRCvGWmp3znbN5b rhIDeXtiYr0Yo6QyW36zqeH716m9tlYu3bfuLerjsx/dA/pDnQ2p/8OKRblx2A== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.85 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: C7E0C37B49 X-Spam-Score: -4.85 X-Migadu-Scanner: scn1.migadu.com X-TUID: 6DPfQo47fnve --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: >>> IWBN to share as much code as possible with =E2=80=98guix publish=E2=80= =99, which has >>> great test suite coverage and is being hammered every day. Clearly the >>> bit about extracting nars is specific to the nar-herder though, so that >>> may prove difficult. >> >> I'm going to look at the Guile Fibers web server, hopefully that can be >> improved to support streaming responses, which would allow removing a >> lot of custom code from guix publish. > > By =E2=80=9Cstreaming responses=E2=80=9D, do you mean pipelining? Streaming might not be the best word, but I'm referring to not having to have the entire response body in memory before sending the first byte. I've now done an initial implementation of this: https://github.com/wingo/fibers/pull/63 > How would that affect =E2=80=98guix publish=E2=80=99? My reading of the concurrent-http-server in the publish script suggests it could be replaced by the fibers web server, if it supports "streaming" response bodies, as I've described above. That might introduce some fibers related issues as I'm guessing there might be some native code used for compression, but it would be a way to remove the workarounds related to the web server part. >> There isn't all that much code to the nar-herder though, and most of >> waht is there is doing different things to guix publish, so I'm not sure >> there's all that much to share. >> >> What I was getting at here though, ignoring the implementation, was >> whether this is worthwhile to do? As in, is there benefit to having this >> and being able to extend the content addressed mirrors that Guix uses? > > Having more content-addressed mirrors is worthwhile IMO, yes. Cool :) > Having two different implementations of the same interfaces may not be > ideal, though, in terms of long-term maintenance cost. Indeed, and I'm not set on keeping the nar-herder separate from guix publish, but the nar-herder does seem to be doing a good job, and I haven't seen a way of unifying the tooling yet. Thanks, Chris --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmK99MFfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XdWNBAArILKZ8leWhkBLAPwyzGy9JTdldYEJakR tPNGDZZh1R0V8ihTVX+Ys+qUz2pbD2c5QY9qSsIX/5jMdfiQgtwGd2astrW46IRp Cs3yJkDQJ9YKpHc+m/bND9WyD/UUQ/3mFBE/pukCsILstgMsTpYClwCtPUEXVXdD oWPYvCo5Wj+7Srq1qBJqOWB2DB03Sp10kZJNkBsHdLOqMdVfcYaRdTEAW5GeXuzo PXUD6xx8TcakPi1Ja6Ygjl0AeSrsFhihi3Jz9ISJN4UjC64+pxN5gG9/utoIbAg6 3uOqbvq4HEYQW+nnACQe3WxWAXrTggO+aXB5GJZP53PuYSLsl3feS0YP10ghkB1q 9dTZP7TigUhdbnBmTxu6XWjUdgUsaidCwHloii+T0Te/y3NFfXuhRluen+k9HvpG Wd6Ts02a7Atp+VnjOjabf14FXBJbq87KVUDD2c/DHp7AOnQ5omVJ3SjkwVpfXfwH xwy2OSoUHISGygeiK3QL8xAq/oN455PVHZgdvveUU9GsVe9FvT170GZzGy5bqdpE BHOq11DTvrFvPq6zOMbtQ3l/5/CUoP+qWO5ytcrYzbLSOtwVRf6/fVnB6Zw4lZwJ v9cOFKNIn4SCOqQwoUlNNr1p3SjhYALx02FaayFFvP9OwTPQ6CQq52pwAnKmAtCa 3rBqMnkSvW8= =24CZ -----END PGP SIGNATURE----- --=-=-=--