From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <guix-devel-bounces+larch=yhetil.org@gnu.org>
Received: from mp0.migadu.com ([2001:41d0:303:5f26::])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	by ms8.migadu.com with LMTPS
	id AKIHJKHxm2XUFAAAkFu2QA
	(envelope-from <guix-devel-bounces+larch=yhetil.org@gnu.org>)
	for <larch@yhetil.org>; Mon, 08 Jan 2024 13:59:13 +0100
Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
	by mp0.migadu.com with LMTPS
	id gC29HKHxm2UvogAAqHPOHw
	(envelope-from <guix-devel-bounces+larch=yhetil.org@gnu.org>)
	for <larch@yhetil.org>; Mon, 08 Jan 2024 13:59:13 +0100
X-Envelope-To: larch@yhetil.org
Authentication-Results: aspmx1.migadu.com;
	dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b="bdOWE8/K";
	spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org";
	dmarc=none
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org;
	s=key1; t=1704718753;
	h=from:from:sender:sender:reply-to:subject:subject:date:date:
	 message-id:message-id:to:to:cc:cc:mime-version:mime-version:
	 content-type:content-type:in-reply-to:in-reply-to:
	 references:references:list-id:list-help:list-unsubscribe:
	 list-subscribe:list-post:dkim-signature;
	bh=AhJevY/UC4P1WGapLORaR3p3cNiwQt8E8f9Tv8ZE7DQ=;
	b=LMFj+NOJF8x6i4FeCna6H833ZEl9YhRDeISrluxX/xyGDJ6cW87tKS7q42Ah0JwtPkw/gU
	H0oSmWHEKfoe7i/Se4fkX9/rxJqMi/3PeH2kCrLx5/HWw93YxuXL7nW/SKBFO/2Wd9+eXm
	SA7JkRUKwhl2q//PwP64gHargOS8pgSXyQVOBA20qH4qeFK/8xwtb8VGhcLXfqvgRfxerg
	KlWoI+95WlIAHI5Ek893yloUM5miz2qS6PWrH9JmsMWVZ09Ukkc5oTDZu768M088N5AfJn
	iYfhDXa8Yg7w68GCy4/Ii6Km0Lt46HtE0haN1hWqP4FCu50E3Ick8BiASEfGJw==
ARC-Authentication-Results: i=1;
	aspmx1.migadu.com;
	dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b="bdOWE8/K";
	spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org";
	dmarc=none
ARC-Seal: i=1; s=key1; d=yhetil.org; t=1704718753; a=rsa-sha256; cv=none;
	b=ZUpkASlGeLRZ8sbpw77KPW9L+3N7S+R9SACugLNaMRrT+iImFhjN9YXtzKuaTu9e8RGEBw
	IMbOJ0Vg8Qfz44mDCGv4ZaTZ2zdpsouPExys1Il/0x2K85lIYk52dCyx/3H2yjWyRo5EH3
	Ck2DTjgeQWOEMxJAR8RPHSisggvS0gntraI6ZqMVt10LAPH2ah5duQH0TS7KMRMD4VXCVD
	w6fKxkz9bH2N4T1eEaEMv7zTOq+3RPNHXWB+IAF2r/+sP9oy4trPW9DGXBlWPMr/CE3L3O
	TUFTMhLwE70zw1IreIbBEgs8j2EV2R6dlK4yJm0gX+RMpxJdG3bpczU3f7lwlg==
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by aspmx1.migadu.com (Postfix) with ESMTPS id DFB64538D1
	for <larch@yhetil.org>; Mon,  8 Jan 2024 13:59:12 +0100 (CET)
Received: from localhost ([::1] helo=lists1p.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.90_1)
	(envelope-from <guix-devel-bounces@gnu.org>)
	id 1rMpD3-0006Jg-IN; Mon, 08 Jan 2024 07:58:29 -0500
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <efraim.flashner@gmail.com>)
 id 1rMpD0-0006JK-Ox
 for guix-devel@gnu.org; Mon, 08 Jan 2024 07:58:27 -0500
Received: from mail-wr1-x42e.google.com ([2a00:1450:4864:20::42e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <efraim.flashner@gmail.com>)
 id 1rMpCy-0007ZS-1n; Mon, 08 Jan 2024 07:58:25 -0500
Received: by mail-wr1-x42e.google.com with SMTP id
 ffacd0b85a97d-3376555b756so818126f8f.0; 
 Mon, 08 Jan 2024 04:58:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20230601; t=1704718701; x=1705323501; darn=gnu.org;
 h=in-reply-to:content-disposition:mime-version:references
 :mail-followup-to:message-id:subject:cc:to:from:date:sender:from:to
 :cc:subject:date:message-id:reply-to;
 bh=AhJevY/UC4P1WGapLORaR3p3cNiwQt8E8f9Tv8ZE7DQ=;
 b=bdOWE8/Kf2UtM0zwp+5R8cwUHxX2iJ/6YMDAesV0/jo6O5Tc6OHZ2WJlRfWSDWemKM
 UFHBe9qIuUrb/PmXcdz3G80Z7vmdO/4+SAcRTQ5JN12L0i+OYS2uSkRZ49mqnWt2bZJY
 Wsf7hY//ogNyFvP+LNsWN3XwbSlbwoM2rEEafsdcA4HZR1gzvEJWn069Mq9eRf54J8Wq
 awYUus/KpgL8T01cXrXb+c+9fHYVgqolN2aDq29SdXK/YochUFTdY4DiOvmw/sYDgmTK
 MEClJRHANcrUb0Pa6aqpXXF4FvwU1XIhDbg5fF3xuSXieBye0kWXSsQsXBX3TFukE38a
 klJQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1704718701; x=1705323501;
 h=in-reply-to:content-disposition:mime-version:references
 :mail-followup-to:message-id:subject:cc:to:from:date:sender
 :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
 bh=AhJevY/UC4P1WGapLORaR3p3cNiwQt8E8f9Tv8ZE7DQ=;
 b=JDJH2XzC6B/fCque0v7WAKpn66bL2ZbXqiaiC5njFWkgnJ1kBjDO6WERT80C2jqQp1
 9CTKt2cd185ROaJ68eDSTXrhsFO+mdb08Wg+7EPKVaC85GuMn0uayLS14gJF3QrxuXvZ
 AwGaFGfxZjbrIzAL4IPHQj3Sp/sA2COLTWlNnvXfS64vFeb6dpU3eAkeaXzgne9cIYdM
 WAUwjk7r6ORVU1SBt9wCEDyRW0ObjDGDaZqBHDEk7ZzoF7ugqDg2Hx5Q8gTWWN/FNCHa
 yoIwLJ+rdvs//SLALSUd9IwoqqsFzWkcZPSKdFzOhcjQ40qntywLAreklGrsy6OOjSbB
 ueqQ==
X-Gm-Message-State: AOJu0YysmPemeJgdQCAPI/FfqoBGf3l7hSdY2HvTdoPAKiMA3bXHUS17
 ZCJrSiVLRFoysdmdTGf9es0=
X-Google-Smtp-Source: AGHT+IG3kjbrbGlpCDmG6dDSUodRld4AxnAogCOs/AqqYZxea4/Y2WF4kBMSva0RyiCsrZSeLHJNSQ==
X-Received: by 2002:adf:f10e:0:b0:337:39c7:e386 with SMTP id
 r14-20020adff10e000000b0033739c7e386mr1559816wro.34.1704718700453; 
 Mon, 08 Jan 2024 04:58:20 -0800 (PST)
Received: from localhost ([141.226.15.142]) by smtp.gmail.com with ESMTPSA id
 u10-20020adfa18a000000b00336ca349bdesm7815475wru.47.2024.01.08.04.58.19
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Mon, 08 Jan 2024 04:58:19 -0800 (PST)
Date: Mon, 8 Jan 2024 14:58:18 +0200
From: Efraim Flashner <efraim@flashner.co.il>
To: Christopher Baines <mail@cbaines.net>
Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@gnu.org>, guix-devel@gnu.org
Subject: Re: Performance of computing cross derivations
Message-ID: <ZZvxakbu-sX4tVZ9@3900XT>
Mail-Followup-To: Christopher Baines <mail@cbaines.net>,
 Ludovic =?utf-8?Q?Court=C3=A8s?= <ludo@gnu.org>, guix-devel@gnu.org
References: <87zg00xvuv.fsf@cbaines.net> <87y1exvj2n.fsf@gnu.org>
 <87le93r9v3.fsf@cbaines.net>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
 protocol="application/pgp-signature"; boundary="77Jjco0VhpslzydL"
Content-Disposition: inline
In-Reply-To: <87le93r9v3.fsf@cbaines.net>
X-PGP-Key-ID: 0x41AAE7DCCA3D8351
X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc
X-PGP-Fingerprint: A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Received-SPF: pass client-ip=2a00:1450:4864:20::42e;
 envelope-from=efraim.flashner@gmail.com; helo=mail-wr1-x42e.google.com
X-Spam_score_int: -14
X-Spam_score: -1.5
X-Spam_bar: -
X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.249,
 FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.248,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,
 T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no
X-Spam_action: no action
X-BeenThere: guix-devel@gnu.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "Development of GNU Guix and the GNU System distribution."
 <guix-devel.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/guix-devel>,
 <mailto:guix-devel-request@gnu.org?subject=unsubscribe>
List-Archive: <https://lists.gnu.org/archive/html/guix-devel>
List-Post: <mailto:guix-devel@gnu.org>
List-Help: <mailto:guix-devel-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/guix-devel>,
 <mailto:guix-devel-request@gnu.org?subject=subscribe>
Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org
Sender: guix-devel-bounces+larch=yhetil.org@gnu.org
X-Migadu-Country: US
X-Migadu-Flow: FLOW_IN
X-Migadu-Scanner: mx13.migadu.com
X-Migadu-Spam-Score: -6.89
X-Spam-Score: -6.89
X-Migadu-Queue-Id: DFB64538D1
X-TUID: snf0W8Sr+xig


--77Jjco0VhpslzydL
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Jan 05, 2024 at 04:41:14PM +0000, Christopher Baines wrote:
>=20
> Ludovic Court=C3=A8s <ludo@gnu.org> writes:
>=20
> > Hi,
> >
> > Christopher Baines <mail@cbaines.net> skribis:
> >
> >> When asked by the data service, it seems to take Guix around 3 minutes
> >> to compute cross derivations for all packages (to a single
> >> target). Here's a simple script that replicates this:
>=20
> ...
>=20
> > One idiom that defeats caching is:
> >
> >   (define (make-me-a-package x y z)
> >     (package
> >       =E2=80=A6))
> >
> > Such a procedure returns a fresh package every time it=E2=80=99s called,
> > preventing caching from happening (because cache entries are compared
> > with =E2=80=98eq?=E2=80=99).  That typically leads to lower hit rates.
> >
> > Anyway, lots of words to say that I don=E2=80=99t see anything immediat=
ely
> > obvious with cross-compilation, yet I wouldn=E2=80=99t be surprised if =
some of
> > these cache-defeating idioms were used because we=E2=80=99ve payed less
> > attention to this.
>=20
> I've got a feeling that performance has got worse since I looked at this
> originally, I've finally got around to having a further look.
>=20
> I spent some time looking at various metrics, but it was most useful to
> just write the cache keys of various types to files and have a read.
>=20
> The cross-base module was causing many issues, as all but one of the
> procedures there produced new package records each time. There is also
> make-rust-sysroot which showed up.
>=20
> I've sent some patches as #68266 to add memoization to avoid this, and
> that seems to speed things up.
>=20
> Looking at other things in the cache, I think there are some issues with
> file-append and local-file. The use of file-append in svn-fetch and
> local-file in the lower procedure in the python build system both bloat
> the cache for example, although I'm less sure about how to address these
> cases.
>=20
> One thing I am sure about though, is that these problems will come
> back. Maybe we could add some reporting in to Guix to look through the
> cache at the keys, lower them all and check for equivalence. That way it
> should be possible to automate saying that having [1] in the cache
> several thousand times is unhelpful. The data service could then run
> this reporting and store it.
>=20
> 1: #<file-append #<package subversion@1.14.2 gnu/packages/version-control=
=2Escm:2267 7f294d908840> "/bin/svn">

I grabbed the patch for make-rust-sysroot to try it out:
Native builds:
time GUIX_PROFILING=3D"object-cache" ./pre-inst-env guix build --no-grafts =
$(./pre-inst-env ~/list-all-cargo-build-system-packages | grep rust- | head=
 -n 100) -d

Object Cache:
  fresh caches:    21
  lookups:      133146
  hits:         130101 (97.7%)
  cache size:    3044 entries

real    0m7.539s
user    0m10.239s
sys     0m0.327s

Before:
time GUIX_PROFILING=3D"object-cache" ./pre-inst-env guix build --no-grafts =
$(./pre-inst-env ~/list-all-cargo-build-system-packages | grep rust- | head=
 -n 100) --target=3Daarch64-linux-gnu -d

Object Cache:
  fresh caches:    20
  lookups:      221189
  hits:         211390 (95.6%)
  cache size:    9798 entries

real    0m18.215s
user    0m14.492s
sys     0m0.469s

After:
time GUIX_PROFILING=3D"object-cache" ./pre-inst-env guix build --no-grafts =
$(./pre-inst-env ~/list-all-cargo-build-system-packages | grep rust- | head=
 -n 100) --target=3Daarch64-linux-gnu -d

Object Cache:
  fresh caches:    20
  lookups:      138654
  hits:         135291 (97.6%)
  cache size:    3362 entries

real    0m7.753s
user    0m10.248s
sys     0m0.328s

That's a massive drop in the size of the cache and a big decrease in the
amount of time it took to calculate those 100 items.

--=20
Efraim Flashner   <efraim@flashner.co.il>   =D7=A8=D7=A0=D7=A9=D7=9C=D7=A4 =
=D7=9D=D7=99=D7=A8=D7=A4=D7=90
GPG key =3D A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

--77Jjco0VhpslzydL
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmWb8WYACgkQQarn3Mo9
g1Em1RAAmZ290o1R/Es23Pmx+Gyfc2EWfEthU6UILQbxlx8J6S6GBP60kyhQPSKl
VIPQrm4jzoNwvAITzjLqsbegBscyx0+X0h/Aqa1GCzTqhZmvUY1mxD8FYU68SoIc
SPwjTV1SHNwYB2VAbvV5D+kMIWPurf4NCrGlYxyWQscg7CK157VudDiwenV+Pq5r
F2CUc/YaoiAu6+jKJfRw+8m6Q863ogMhDAH/uXW78U2KvNid8ldFpyLZpzQZrPQy
ZI7WxmZ6k9olFt35hhC8h8gcPF0ogMijFfpHim1q6pcm07UJGXVGtf6NLiaSA4X+
nSVENUo/JDpTjbnwlzmqufz7O735fjiKEJcLFlmUnuvydn9nJqMlrMa0IAObGp6c
tj15B2B2Gqt9MW/PUlK9nsb0VYPGFQXrgJNEf3r47hopwfCQACuxkZGQaO7XSUI6
bCiU2mAbv3b4R49LTE/4Sz3l/S4BTjbDdXHbW7GrqcMQZhm+PXcSh22SpfinukfC
cf4HSwKs4rdVj9KqsSOQzzfmn7lpOPzTBMGqQba9HiMjzbfHZlM5vNWtLha0e1di
P4+kkJYyP8LP3Q8L21FtxkiRf3m6jkzl25nr4btN9GDtxiEDAc4e3WNT+zV98QfJ
dUK0Xxzjxr2v7cLwxN82Vdn/kmMvTI1c+uUZ21bWA42nzjglZe4=
=OzgW
-----END PGP SIGNATURE-----

--77Jjco0VhpslzydL--