From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id YMYXEuC8TGHFMgEAgWs5BA (envelope-from ) for ; Thu, 23 Sep 2021 19:44:00 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id 4C3BDeC8TGE3DwAA1q6Kng (envelope-from ) for ; Thu, 23 Sep 2021 17:44:00 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 81DCF75C9 for ; Thu, 23 Sep 2021 19:43:59 +0200 (CEST) Received: from localhost ([::1]:49330 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mTSlK-0000UK-0q for larch@yhetil.org; Thu, 23 Sep 2021 13:43:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46232) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mTSVu-0000N7-FW for guix-patches@gnu.org; Thu, 23 Sep 2021 13:28:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44225) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mTSVu-0002l3-7H for guix-patches@gnu.org; Thu, 23 Sep 2021 13:28:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mTSVt-0008Ip-Vy for guix-patches@gnu.org; Thu, 23 Sep 2021 13:28:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50384] [PATCH v4] Optimise search-patch (reducing I/O) Resent-From: Maxime Devos Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 23 Sep 2021 17:28:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50384 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: 50384@debbugs.gnu.org Received: via spool by 50384-submit@debbugs.gnu.org id=B50384.163241802331832 (code B ref 50384); Thu, 23 Sep 2021 17:28:01 +0000 Received: (at 50384) by debbugs.gnu.org; 23 Sep 2021 17:27:03 +0000 Received: from localhost ([127.0.0.1]:55771 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mTSUw-0008HM-Pm for submit@debbugs.gnu.org; Thu, 23 Sep 2021 13:27:03 -0400 Received: from laurent.telenet-ops.be ([195.130.137.89]:38974) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mTSUu-0008Gr-DR for 50384@debbugs.gnu.org; Thu, 23 Sep 2021 13:27:01 -0400 Received: from ptr-bvsjgyjmffd7q9timvx.18120a2.ip6.access.telenet.be ([IPv6:2a02:1811:8c09:9d00:aaf1:9810:a0b8:a55d]) by laurent.telenet-ops.be with bizsmtp id xVSy250040mfAB401VSyEa; Thu, 23 Sep 2021 19:26:58 +0200 Message-ID: <3563a99b37c8ef3b226b7fef25a5afa273c08b6e.camel@telenet.be> From: Maxime Devos Date: Thu, 23 Sep 2021 19:26:51 +0200 In-Reply-To: <87r1dhj2bk.fsf_-_@gnu.org> References: <8900fa8c8eef7f72fc97adc2408be26c88de7803.camel@telenet.be> <87tuj0xaja.fsf@gnu.org> <0ec7f0270fcccec730808f9210f074cd5339961f.camel@telenet.be> <87ee9xerac.fsf_-_@gnu.org> <87r1dhj2bk.fsf_-_@gnu.org> Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-STReeQE9TXFr506Wlmil" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r21; t=1632418018; bh=R+Shn0SMHkGBfThsKyNabrVQpWZuQE4iWaxtQkzwny0=; h=Subject:From:To:Cc:Date:In-Reply-To:References; b=iRqYRboQW1s5928GcGx6dStkwFnDPiOUemXzCwJMvCQrN0QsJ2khqvZvk16eRkqW8 pZ1Ntnm7B6YHsUzLBdk0kBG+RWuBoQsuYn6TnKybn3hX3RjssgUCh06MsH7pYzZdz3 FP/WB3QnZo72MyRRFW+pLpDEsk5TaUiWHuZtLjHcM1PFbGwJPKTH4MwaMM/7DQIwC1 Ab1UTe1xMmq27sOWjk8dlCUEb4jRny8c3Ra9P6PfVEaYNziYDbE2eztgGnjDn3sfnK uv5jUbKQclbSmD8NsGvbxx7GehATu62nLby7tQDUhFGbNwFYfc6I4oMYoNDsCJSjs4 zwWJfaRmWLJZQ== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1632419039; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=R+Shn0SMHkGBfThsKyNabrVQpWZuQE4iWaxtQkzwny0=; b=e6PdpraDkfw1izDKc3Fek3tnxxWiypGg7XvncAidazUuLbgLJbU0yA7++wjhL/wBLA3h7T Mjb45UhPLO+cD/odj7vfuDWz3vAv8FrMl9W6IwpbxFbW2Q9gWhG3yjHvjsiPjiA+sPPQgw Zy9cciboFrFEwC+CIGcXKy3OT64uXnm0ZM+UOmIJrIdCZAhLmNRh+pS7H7KzyBu6+6jMoR W6wH8g7DG9GH+fRlvYcAm46pk+UZQWWZuiAc0NKJfvbjunHhu1ruLIBSNbVRwL8STyZkhO fsxlyjmPvJQCN8KatyTsFEaXeNJCdk09pbTIkWPmxApJ6MBbSZX26zjMtkJuLA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1632419039; a=rsa-sha256; cv=none; b=JHvLcSBnRsHQa4M7E8PgNl3sPfz4W5eH8wKpIzpduyWncxhObybF2b2SlRVYrENUeYVVq0 M9anqsKDHMKOg0vXJH+RuE4t9yAuha4NvVEOVx+HF2Xfk6w86xSqGyCkK2zaliLdDlERLY lc/XGb+3DISxpGN8SEMKmImT/qaeVItp/tuoDJyq/7+3ld2Nu/5NegwbNsjqbheKS0XqMc BOTlCRyGymSglsnCwPvbfARmljmaUoGYR34Rn4P1gn3GUZnTzoyFSQ64jX7NEJ7zo3jIYg cZydIIBj9UomrTEcjJSV/49+eh19LtXmRwDVuK5qSyN3rzs7qOO25pYyzY8fcQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r21 header.b=iRqYRboQ; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.89 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=telenet.be header.s=r21 header.b=iRqYRboQ; dmarc=fail reason="SPF not aligned (relaxed)" header.from=telenet.be (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 81DCF75C9 X-Spam-Score: -1.89 X-Migadu-Scanner: scn0.migadu.com X-TUID: VXadBVuVwxqy --=-STReeQE9TXFr506Wlmil Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s schreef op di 21-09-2021 om 18:55 [+0200]: > Hi! >=20 > I took the liberty to reopen this patch because there were good ideas > IMO. I=E2=80=99m sorry if my many questions and lack of responsiveness c= ame out > as a suggestion that this approach wasn=E2=80=99t good. I reordered your mail a little. > Looking at the big picture, what I=E2=80=99d like to have is a package > derivation cache designed in such a way that =E2=80=9Cguix install foo=E2= =80=9D wouldn=E2=80=99t > even need to load any package module on a cache hit. That=E2=80=99d make= a > noticeable difference performance-wise, that=E2=80=99s another level of > complexity=E2=80=A6 (I have a rough design in mind that we could discuss= .) This =E2=80=98package derivation cache=E2=80=99 seems an interesting idea t= o pursue, though I wonder what could be used as =E2=80=98keys=E2=80=99 in the derivat= ion cache. Package names aren't sufficient because packages can have multiple versions= , package names + package versions aren't sufficient because packages can hav= e multiple variants. Grafts might need some care. Having multiple versions = of guix can be addressed by including the commits of every channel in the key. Even if =E2=80=98foo=E2=80=99 isn't in the cache, the cache can still be us= eful if the inputs =E2=80=98bar=E2=80=99 and =E2=80=98baz=E2=80=99 of foo are in the ca= che. > Ludovic Court=C3=A8s skribis: >=20 > > > +;; repeated 'stat' calls. Allow computing the hash of the file in a= dvance, > > > +;; to avoid having to send the file to the daemon when it is already= interned > > > +;; in the store. > > > (define-record-type > > > - (%%local-file file absolute name recursive? select?) > > > + (%%local-file file absolute name sha256 recursive? select?) > > > local-file? > > > (file local-file-file) ;string > > > (absolute %local-file-absolute-file-name) ;promise string > > > (name local-file-name) ;string > > > + (sha256 local-file-sha256) ;sha256 bytevector= | #f > >=20 > > Could we store the result of =E2=80=98fixed-output-path=E2=80=99 rather= than the SHA256, > > while we=E2=80=99re at it? Embedding the result of =E2=80=98fixed-output-path=E2=80=99 in the .go migh= t be problematic from a closure size perspective, as that would create additional references= in the store items of guix. > I tried that with the patch below, roughly taking the same approach as > your patch series, but somewhat simplified, mostly so I could > experiment. [...] >=20 > We can estimate the performance of that strategy by commenting out the > =E2=80=98add-temp-root*=E2=80=99 call (thus getting a single RPC) in > =E2=80=98local-file-compiler=E2=80=99: this time it=E2=80=99s slightly fa= ster, but we=E2=80=99re in the > 1% range on the wall-clock time of =E2=80=98guix build pigx -d --no-graft= s=E2=80=99: > [...] > Perhaps the gains would be a bit higher if we change all the package > files to use =E2=80=98local-patches=E2=80=99, but we probably can=E2=80= =99t expect a lot more > anyway since that process is CPU-bound. >=20 > So I don=E2=80=99t know. It feels like a worthy optimization, and one th= at=E2=80=99s > manageable from a maintenance viewpoint, but it buys us very little. >=20 > Thoughts? As it is only <1%, I would prefer trying the =E2=80=98package derivation ca= che=E2=80=99 first, as it seems to have more potential. > Ludo=E2=80=99. >=20 --=-STReeQE9TXFr506Wlmil Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYUy42xccbWF4aW1lZGV2 b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7o0YAP92vmojOgaiI7vtCjLc3x/akVtC Gx8kiPOKLtr/2bgDIQEAt7YyiFMiVPXJwFRjL4D80kJ4sR/P7yISSYev/7IUjwA= =26Wc -----END PGP SIGNATURE----- --=-STReeQE9TXFr506Wlmil--