From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id mNcQNGobYF/sAwAA0tVLHw (envelope-from ) for ; Tue, 15 Sep 2020 01:39:54 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id qF23LmobYF98DAAAbx9fmQ (envelope-from ) for ; Tue, 15 Sep 2020 01:39:54 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 534199404D7 for ; Tue, 15 Sep 2020 01:39:54 +0000 (UTC) Received: from localhost ([::1]:59506 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kHzwn-0008Bd-60 for larch@yhetil.org; Mon, 14 Sep 2020 21:39:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56048) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kHzwe-0008BR-OK for guix-devel@gnu.org; Mon, 14 Sep 2020 21:39:44 -0400 Received: from mail-qt1-x82b.google.com ([2607:f8b0:4864:20::82b]:43927) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kHzwd-0007PQ-4X for guix-devel@gnu.org; Mon, 14 Sep 2020 21:39:44 -0400 Received: by mail-qt1-x82b.google.com with SMTP id g3so1994169qtq.10 for ; Mon, 14 Sep 2020 18:39:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=8mQL/REg5bWFgd1JXmAqwCcIsc0X6z/pbtUME7yMSrE=; b=MaUi/RRt/+wZvqwh58KvMc2KEJapBCOubxWOboba5AEjcRKl51mBmkuTrvg6CSOsB8 /ecNS6p1rr5PqulzCuiPupYaZ55+Um5/pEAkyUot5lY4G/sUi0l/Iua/5i+xa7VaKyjV JM/Hm0ZEWHSmTYhXSwoUDI2w2tv5zeCq9v9V05mWHcpfZNODuW6KkNPzB9cyRhFh4D3d 8K1njZshDqSeM0fZrTuW+f8IFHLjAuxTAb0AKxKkR2M/BOf3VkfQfkH+syzNtauzjD4f CWKMmoUcrYlh7vN1Mr/03TgadK7X14VrlzW3wyxJQlCBpSrPssAg8xMREimV7sfH1MvZ eFEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=8mQL/REg5bWFgd1JXmAqwCcIsc0X6z/pbtUME7yMSrE=; b=fFyZVy+hscYftfqSah9IYL8BO6i9HdEQQETeleZ7mEKHymeLW2f32+KDG+8LJHj0V9 jBtVbl5Nz5thTgPsl73gKvwAtkC51WSHhOzEdPEE8vfWJ0Tqh5TvijF/sPRpEe6+8AkG gOFDcZb80pAIrmx4fvnDithvo63HNDFk0vOt4PXr0EMg9hI+PVjoHqD+RwDWiDTzioEQ usdzgeE8vk1E5YETCS5elbE1K7wuig0db4BProtAraR/fSkiFKiRbfIeZ31pIQyZXUcq VwXUUzeveC7CkjQU/O2ZxFT/zSxSuRZnQLhNM7AtOzExrRCUv4ymhgeuqllvdGTXhoOt PcgA== X-Gm-Message-State: AOAM530sU/IlDJk5WSGOzPXpcppd6s+dmEnNBwiqxmOdb7TlZgH3SCnD WXjfH9qbH8dd7Ak5nSxgPGVhwXLDmEvipg== X-Google-Smtp-Source: ABdhPJxLNH0PPHpFyi3Lef4pSvoej0aLl8sjSBLA2h8jJI52s++WGu6rdzzADpqcJocvXO5l+h8eQw== X-Received: by 2002:ac8:c47:: with SMTP id l7mr16597934qti.112.1600133981482; Mon, 14 Sep 2020 18:39:41 -0700 (PDT) Received: from hurd (dsl-205-233-124-4.b2b2c.ca. [205.233.124.4]) by smtp.gmail.com with ESMTPSA id x124sm15915493qkd.72.2020.09.14.18.39.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2020 18:39:40 -0700 (PDT) From: Maxim Cournoyer To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: Speeding up archive export References: <87a6xyhujp.fsf@inria.fr> Date: Mon, 14 Sep 2020 21:40:30 -0400 In-Reply-To: <87a6xyhujp.fsf@inria.fr> ("Ludovic =?utf-8?Q?Court=C3=A8s=22?= =?utf-8?Q?'s?= message of "Thu, 10 Sep 2020 11:04:26 +0200") Message-ID: <87d02n4y29.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2607:f8b0:4864:20::82b; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x82b.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Guix-devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=MaUi/RRt; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Spam-Score: -1.71 X-TUID: vaP23L4klUVl Hi Ludo! Ludovic Court=C3=A8s writes: > Hello Guix! > > If you=E2=80=99ve ever used offloading (or =E2=80=98guix copy=E2=80=99), = you=E2=80=99ve probably noticed > that the time to send store items is proportional to the number of store > items to send rather than their total size. Namely: > > guix archive --export coreutils > > is fast, but: > > guix archive --export $(guix build -d coreutils) > > is slow (there are lots of small files). > > Running =E2=80=98perf timechart record guix archive --export =E2=80=A6=E2= =80=99 confirms the > problem: guix-daemon is mostly idle, waiting for all the tiny =E2=80=98gu= ix > authenticate=E2=80=99 programs it spawns to sign each every store item. = Here=E2=80=99s > the Gantt diagram (grey =3D idle, blue =3D busy): Very cool! The timechart suggests the guix-authenticate programs are run sequentially? Perhaps running them in parallel would be a cheap, first step to improve performance? > How can we improve on that? > > Here are several solutions that come to mind: > > 1. Sign the whole bundle instead of each individual item. > > That solves the problem, but that would prevent the receiver from > storing individual store item signatures in the future (a few years > ago Nix added signatures as part of the =E2=80=98ValidPathInfo=E2=80= =99 table of > the store database, and I think that=E2=80=99s something we might wa= nt to > have too). Why? Couldn't the receiver do the book keeping no matter if it received a signed bundle or a single file? It could assign the bundle signature to individual store files in the database, for example. This seems the obvious, easy solution. We need good arguments to not implement it. > 2. Sign fewer items: we can do that by signing only store items that > are not content-addressed=E2=80=94i.e., resulting from a fixed-output > derivation or being a =E2=80=9Csource=E2=80=9D coming from =E2=80=98= add-to-store=E2=80=99 or > similar. > > That means we wouldn=E2=80=99t have to sign .drv and *-guile-builder= , which > would make a big difference and is generally advisable. > Unfortunately, there=E2=80=99s no easy way to determine whether a st= ore > item is content-addressable. Again Nix added > =E2=80=9Ccertificate-addressability claims=E2=80=9D to =E2=80=98Vali= dPathInfo=E2=80=99, which might > help, though it=E2=80=99s not entirely clear. > > 3. Reimplement =E2=80=98guix authenticate=E2=80=99 and a subset of (gui= x pki) in C++ (!). > We could load the keys and the ACL only once, and we wouldn=E2=80=99= t have > to fork and all, I=E2=80=99m sure it=E2=80=99d be very fast=E2=80=A6= and very distracting > too: I=E2=80=99d rather investigate in the daemon rewrite in Scheme. > > 4. Spawn =E2=80=98guix authenticate=E2=80=99 once and talk to it over a= pipe (similar > to =E2=80=98guix offload=E2=80=99). That might be the easiest short= -term solution. Failing 1., 4. is my second favorite, because it seems the most Guixy of the remaining options, and should provide acceptable performance. Maxim