From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:303:5f26::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id AKIHJKHxm2XUFAAAkFu2QA (envelope-from ) for ; Mon, 08 Jan 2024 13:59:13 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id gC29HKHxm2UvogAAqHPOHw (envelope-from ) for ; Mon, 08 Jan 2024 13:59:13 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b="bdOWE8/K"; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1704718753; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=AhJevY/UC4P1WGapLORaR3p3cNiwQt8E8f9Tv8ZE7DQ=; b=LMFj+NOJF8x6i4FeCna6H833ZEl9YhRDeISrluxX/xyGDJ6cW87tKS7q42Ah0JwtPkw/gU H0oSmWHEKfoe7i/Se4fkX9/rxJqMi/3PeH2kCrLx5/HWw93YxuXL7nW/SKBFO/2Wd9+eXm SA7JkRUKwhl2q//PwP64gHargOS8pgSXyQVOBA20qH4qeFK/8xwtb8VGhcLXfqvgRfxerg KlWoI+95WlIAHI5Ek893yloUM5miz2qS6PWrH9JmsMWVZ09Ukkc5oTDZu768M088N5AfJn iYfhDXa8Yg7w68GCy4/Ii6Km0Lt46HtE0haN1hWqP4FCu50E3Ick8BiASEfGJw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b="bdOWE8/K"; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Seal: i=1; s=key1; d=yhetil.org; t=1704718753; a=rsa-sha256; cv=none; b=ZUpkASlGeLRZ8sbpw77KPW9L+3N7S+R9SACugLNaMRrT+iImFhjN9YXtzKuaTu9e8RGEBw IMbOJ0Vg8Qfz44mDCGv4ZaTZ2zdpsouPExys1Il/0x2K85lIYk52dCyx/3H2yjWyRo5EH3 Ck2DTjgeQWOEMxJAR8RPHSisggvS0gntraI6ZqMVt10LAPH2ah5duQH0TS7KMRMD4VXCVD w6fKxkz9bH2N4T1eEaEMv7zTOq+3RPNHXWB+IAF2r/+sP9oy4trPW9DGXBlWPMr/CE3L3O TUFTMhLwE70zw1IreIbBEgs8j2EV2R6dlK4yJm0gX+RMpxJdG3bpczU3f7lwlg== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DFB64538D1 for ; Mon, 8 Jan 2024 13:59:12 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rMpD3-0006Jg-IN; Mon, 08 Jan 2024 07:58:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rMpD0-0006JK-Ox for guix-devel@gnu.org; Mon, 08 Jan 2024 07:58:27 -0500 Received: from mail-wr1-x42e.google.com ([2a00:1450:4864:20::42e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rMpCy-0007ZS-1n; Mon, 08 Jan 2024 07:58:25 -0500 Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-3376555b756so818126f8f.0; Mon, 08 Jan 2024 04:58:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704718701; x=1705323501; darn=gnu.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:sender:from:to :cc:subject:date:message-id:reply-to; bh=AhJevY/UC4P1WGapLORaR3p3cNiwQt8E8f9Tv8ZE7DQ=; b=bdOWE8/Kf2UtM0zwp+5R8cwUHxX2iJ/6YMDAesV0/jo6O5Tc6OHZ2WJlRfWSDWemKM UFHBe9qIuUrb/PmXcdz3G80Z7vmdO/4+SAcRTQ5JN12L0i+OYS2uSkRZ49mqnWt2bZJY Wsf7hY//ogNyFvP+LNsWN3XwbSlbwoM2rEEafsdcA4HZR1gzvEJWn069Mq9eRf54J8Wq awYUus/KpgL8T01cXrXb+c+9fHYVgqolN2aDq29SdXK/YochUFTdY4DiOvmw/sYDgmTK MEClJRHANcrUb0Pa6aqpXXF4FvwU1XIhDbg5fF3xuSXieBye0kWXSsQsXBX3TFukE38a klJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704718701; x=1705323501; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AhJevY/UC4P1WGapLORaR3p3cNiwQt8E8f9Tv8ZE7DQ=; b=JDJH2XzC6B/fCque0v7WAKpn66bL2ZbXqiaiC5njFWkgnJ1kBjDO6WERT80C2jqQp1 9CTKt2cd185ROaJ68eDSTXrhsFO+mdb08Wg+7EPKVaC85GuMn0uayLS14gJF3QrxuXvZ AwGaFGfxZjbrIzAL4IPHQj3Sp/sA2COLTWlNnvXfS64vFeb6dpU3eAkeaXzgne9cIYdM WAUwjk7r6ORVU1SBt9wCEDyRW0ObjDGDaZqBHDEk7ZzoF7ugqDg2Hx5Q8gTWWN/FNCHa yoIwLJ+rdvs//SLALSUd9IwoqqsFzWkcZPSKdFzOhcjQ40qntywLAreklGrsy6OOjSbB ueqQ== X-Gm-Message-State: AOJu0YysmPemeJgdQCAPI/FfqoBGf3l7hSdY2HvTdoPAKiMA3bXHUS17 ZCJrSiVLRFoysdmdTGf9es0= X-Google-Smtp-Source: AGHT+IG3kjbrbGlpCDmG6dDSUodRld4AxnAogCOs/AqqYZxea4/Y2WF4kBMSva0RyiCsrZSeLHJNSQ== X-Received: by 2002:adf:f10e:0:b0:337:39c7:e386 with SMTP id r14-20020adff10e000000b0033739c7e386mr1559816wro.34.1704718700453; Mon, 08 Jan 2024 04:58:20 -0800 (PST) Received: from localhost ([141.226.15.142]) by smtp.gmail.com with ESMTPSA id u10-20020adfa18a000000b00336ca349bdesm7815475wru.47.2024.01.08.04.58.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Jan 2024 04:58:19 -0800 (PST) Date: Mon, 8 Jan 2024 14:58:18 +0200 From: Efraim Flashner To: Christopher Baines Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , guix-devel@gnu.org Subject: Re: Performance of computing cross derivations Message-ID: Mail-Followup-To: Christopher Baines , Ludovic =?utf-8?Q?Court=C3=A8s?= , guix-devel@gnu.org References: <87zg00xvuv.fsf@cbaines.net> <87y1exvj2n.fsf@gnu.org> <87le93r9v3.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="77Jjco0VhpslzydL" Content-Disposition: inline In-Reply-To: <87le93r9v3.fsf@cbaines.net> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Received-SPF: pass client-ip=2a00:1450:4864:20::42e; envelope-from=efraim.flashner@gmail.com; helo=mail-wr1-x42e.google.com X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.248, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Scanner: mx13.migadu.com X-Migadu-Spam-Score: -6.89 X-Spam-Score: -6.89 X-Migadu-Queue-Id: DFB64538D1 X-TUID: snf0W8Sr+xig --77Jjco0VhpslzydL Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jan 05, 2024 at 04:41:14PM +0000, Christopher Baines wrote: >=20 > Ludovic Court=C3=A8s writes: >=20 > > Hi, > > > > Christopher Baines skribis: > > > >> When asked by the data service, it seems to take Guix around 3 minutes > >> to compute cross derivations for all packages (to a single > >> target). Here's a simple script that replicates this: >=20 > ... >=20 > > One idiom that defeats caching is: > > > > (define (make-me-a-package x y z) > > (package > > =E2=80=A6)) > > > > Such a procedure returns a fresh package every time it=E2=80=99s called, > > preventing caching from happening (because cache entries are compared > > with =E2=80=98eq?=E2=80=99). That typically leads to lower hit rates. > > > > Anyway, lots of words to say that I don=E2=80=99t see anything immediat= ely > > obvious with cross-compilation, yet I wouldn=E2=80=99t be surprised if = some of > > these cache-defeating idioms were used because we=E2=80=99ve payed less > > attention to this. >=20 > I've got a feeling that performance has got worse since I looked at this > originally, I've finally got around to having a further look. >=20 > I spent some time looking at various metrics, but it was most useful to > just write the cache keys of various types to files and have a read. >=20 > The cross-base module was causing many issues, as all but one of the > procedures there produced new package records each time. There is also > make-rust-sysroot which showed up. >=20 > I've sent some patches as #68266 to add memoization to avoid this, and > that seems to speed things up. >=20 > Looking at other things in the cache, I think there are some issues with > file-append and local-file. The use of file-append in svn-fetch and > local-file in the lower procedure in the python build system both bloat > the cache for example, although I'm less sure about how to address these > cases. >=20 > One thing I am sure about though, is that these problems will come > back. Maybe we could add some reporting in to Guix to look through the > cache at the keys, lower them all and check for equivalence. That way it > should be possible to automate saying that having [1] in the cache > several thousand times is unhelpful. The data service could then run > this reporting and store it. >=20 > 1: # "/bin/svn"> I grabbed the patch for make-rust-sysroot to try it out: Native builds: time GUIX_PROFILING=3D"object-cache" ./pre-inst-env guix build --no-grafts = $(./pre-inst-env ~/list-all-cargo-build-system-packages | grep rust- | head= -n 100) -d Object Cache: fresh caches: 21 lookups: 133146 hits: 130101 (97.7%) cache size: 3044 entries real 0m7.539s user 0m10.239s sys 0m0.327s Before: time GUIX_PROFILING=3D"object-cache" ./pre-inst-env guix build --no-grafts = $(./pre-inst-env ~/list-all-cargo-build-system-packages | grep rust- | head= -n 100) --target=3Daarch64-linux-gnu -d Object Cache: fresh caches: 20 lookups: 221189 hits: 211390 (95.6%) cache size: 9798 entries real 0m18.215s user 0m14.492s sys 0m0.469s After: time GUIX_PROFILING=3D"object-cache" ./pre-inst-env guix build --no-grafts = $(./pre-inst-env ~/list-all-cargo-build-system-packages | grep rust- | head= -n 100) --target=3Daarch64-linux-gnu -d Object Cache: fresh caches: 20 lookups: 138654 hits: 135291 (97.6%) cache size: 3362 entries real 0m7.753s user 0m10.248s sys 0m0.328s That's a massive drop in the size of the cache and a big decrease in the amount of time it took to calculate those 100 items. --=20 Efraim Flashner =D7=A8=D7=A0=D7=A9=D7=9C=D7=A4 = =D7=9D=D7=99=D7=A8=D7=A4=D7=90 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --77Jjco0VhpslzydL Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmWb8WYACgkQQarn3Mo9 g1Em1RAAmZ290o1R/Es23Pmx+Gyfc2EWfEthU6UILQbxlx8J6S6GBP60kyhQPSKl VIPQrm4jzoNwvAITzjLqsbegBscyx0+X0h/Aqa1GCzTqhZmvUY1mxD8FYU68SoIc SPwjTV1SHNwYB2VAbvV5D+kMIWPurf4NCrGlYxyWQscg7CK157VudDiwenV+Pq5r F2CUc/YaoiAu6+jKJfRw+8m6Q863ogMhDAH/uXW78U2KvNid8ldFpyLZpzQZrPQy ZI7WxmZ6k9olFt35hhC8h8gcPF0ogMijFfpHim1q6pcm07UJGXVGtf6NLiaSA4X+ nSVENUo/JDpTjbnwlzmqufz7O735fjiKEJcLFlmUnuvydn9nJqMlrMa0IAObGp6c tj15B2B2Gqt9MW/PUlK9nsb0VYPGFQXrgJNEf3r47hopwfCQACuxkZGQaO7XSUI6 bCiU2mAbv3b4R49LTE/4Sz3l/S4BTjbDdXHbW7GrqcMQZhm+PXcSh22SpfinukfC cf4HSwKs4rdVj9KqsSOQzzfmn7lpOPzTBMGqQba9HiMjzbfHZlM5vNWtLha0e1di P4+kkJYyP8LP3Q8L21FtxkiRf3m6jkzl25nr4btN9GDtxiEDAc4e3WNT+zV98QfJ dUK0Xxzjxr2v7cLwxN82Vdn/kmMvTI1c+uUZ21bWA42nzjglZe4= =OzgW -----END PGP SIGNATURE----- --77Jjco0VhpslzydL--