From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:1008:1e59::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id +HxWIKzxn2UDUQEAkFu2QA (envelope-from ) for ; Thu, 11 Jan 2024 14:48:28 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id 8F+AG6zxn2UMxgAA62LTzQ (envelope-from ) for ; Thu, 11 Jan 2024 14:48:28 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1704980908; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=jAa/tjlmI0s4Et0DobwePzvlRfv0YjfZBpqJZaD+wPA=; b=Ss7xx0DXIq9RA2Om1Y8/5vY7AguF1qTcPHVICECLmsQKFXMwHuWViWDLBKxcstDfCeZB1r NH7eC6bx7nDJ0rPEP4HtIWsJ5ghYvvKujL3cYk3Ih4Om4qp5jEK8KBLzTLYgCGy0k8upD2 aUwbzzV950YW3ojLS849NkLodM/pxvc6J4nAFTRe3UC/MCoAISr6NI3Fj98cJnbU8H/WY4 0lZ4+aK9fkU+aCPUlmX2sFnr4C8UiNXxaLOndV8wf+Qvcm8T7qVICQEPpvvzj9zNi/ZuiQ E/9gvkXCAE/A1GyPtYIskwgkSEXzlNCa2mK+9/YMJC5XGiU3gx3e2jbEHhkqAA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=none ARC-Seal: i=1; s=key1; d=yhetil.org; t=1704980908; a=rsa-sha256; cv=none; b=FQTmWU6EG5xWlzm6Ler1Jh3PFmSMbRWWiiw99sdr5ariC/KDb3SQiU12rBYe5zFDAAn5x4 CFlYLiuWe9MEoe7XxUgMQF2NDbPO4uichytX6N2FgzUIXl+7hM4Y/4t+6oyqKuzKTlSrGn B9MtoVAVqsUhQ0pL4ckfsrr4CiRoB+Kc7DAEvoi4SC/TXoi7O4RTJnksGNcB7CDrt9c8v6 LXMd+TlmEpK0DgnFv4h3rlSdtTaPL1oPXf7p8qC/qmZkUBOR1j6xU1eQaJ3EdCNBhGAKZe ANcQK6Ukxad+q+x+Xci07zunU80WVhQZHfYQqZSYOy2a9gU4gkcGIAqysdLszg== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4BD0E72026 for ; Thu, 11 Jan 2024 14:48:28 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rNvPY-0002nC-BR; Thu, 11 Jan 2024 08:47:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rNvPS-0002mr-GD for guix-devel@gnu.org; Thu, 11 Jan 2024 08:47:50 -0500 Received: from mira.cbaines.net ([212.71.252.8]) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rNvPQ-0001MR-8x; Thu, 11 Jan 2024 08:47:49 -0500 Received: from localhost (unknown [217.155.61.229]) by mira.cbaines.net (Postfix) with ESMTPSA id 1FC5527BBE2; Thu, 11 Jan 2024 13:47:43 +0000 (GMT) Received: from felis (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 86464ee0; Thu, 11 Jan 2024 13:47:42 +0000 (UTC) References: <87zg00xvuv.fsf@cbaines.net> <87y1exvj2n.fsf@gnu.org> <87le93r9v3.fsf@cbaines.net> <878r4xgxr6.fsf@cbaines.net> <875y00oxo4.fsf@gnu.org> User-agent: mu4e 1.10.7; emacs 29.1 From: Christopher Baines To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: Efraim Flashner , guix-devel@gnu.org Subject: Re: Performance of computing cross derivations Date: Thu, 11 Jan 2024 13:26:35 +0000 In-reply-to: <875y00oxo4.fsf@gnu.org> Message-ID: <874jfkgewz.fsf@cbaines.net> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx10.migadu.com X-Migadu-Spam-Score: -8.41 X-Spam-Score: -8.41 X-Migadu-Queue-Id: 4BD0E72026 X-TUID: N/bvfe81Ig/+ --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s writes: > Christopher Baines skribis: > >> I think you're right, while I send some other changes in #68266, I think >> it's this change around make-rust-sysroot that has pretty much all the >> effects on performance. >> >> I think the tens of thousands of duplicated packages from cross-base >> that I was looking at are almost entirely coming from >> make-rust-sysroot. As Ludo mentions in [1], maybe this has something to >> do with use of cross- procedures in native-inputs, although I'm not sure >> that moving those calls out of native-inputs is a correct thing to do. >> >> I don't know what the correct approach here is, but I think something >> needs doing here to address the performance regression. > > I probably missed it in the thread: what commit caused the regression, > and how can I test any changes? I=E2=80=99m willing to help but I missed= some > of the context. It's not a pure performance regression, more that in it's current form, rust cross derivations are very expensive to compute. It's been this way since cross-compiling was enabled in [1]. 1: https://git.savannah.gnu.org/cgit/guix.git/patch/?id=3De604972d9c6973026= 91aeb22e9c50c933a1a3c72 I've been looking at data service slowness in processing revisions over the last few weeks, and I think it's mostly down to this. Looking at the revision prior to the change [2], computing all the derivations took around 3 hours, which is ages, but still quick compared to the nearly 9 hours it took after this change [3]. 2: https://data.guix.gnu.org/revision/58bbb38c5bd2e42aab9e9408d8c9d8da3409f= 178 3: https://data.guix.gnu.org/revision/c9e1a72cc27925484635ae01bc4de28bf2326= 89d Obviously having more derivations is good and that usually means more work for the data service, but in this case it seems like things can be sped up quite a bit. For testing locally, I've been computing all the derivations for i586-pc-gnu, but Efraim also posted a concise command to look at computing some cross derivations for a subset of rust packages [4]. 4: https://lists.gnu.org/archive/html/guix-devel/2024-01/msg00053.html --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmWf8XxfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9Xf7/w/8DtEtC8KFKa45WMA2sKNiKkz7Nlhhd1A4 lXulvZMuXz7foEnjV9qGEkA4O/pHMGqOAueLfmCSlcDqKVh8Mt3q9xmhaWpz9AG5 Mxl6XSD5QS2DEs0KLxO5wDQsHioeJWA+mfgkTsHo5vDD722Rk23Ks0nYhr7PX7Lh yvgyRWdvyGzfqYk4SgaCa3IbLikkxOwc0qxd3bFxLqkKszcf1lrzk2p/g/xelo0a k9Oir2UPFeN71fR53LRaAaemaszPalKy0DcDWCv609z3bewo6I7LlaIDtJPSF5W+ ZThg7JT9fDdCtBhavgb40gdvsbrVDdL3ZO8YpmnT9YgPIPNHtZu3HRrvMRpQrjIz KW+atnWL08bOCWwv8MSi7yTH4b1w7AabBVsZTXpdeJVft+pKb8k9cJoAWpGWxGMm Hl2cfLl4bRkq6tWpR0Nxpv+uRESG66Pnr4XGhlsn/oDDtEHazJhGkQ9ODtzYFcwn t21EeEb9HeblZm8FqI7hLaT8WS1dqj25VR9NuKLrBbm4juI+dp+1QtsYvGrqFSfu ps+iLX1KA0DiVc3Dlow4Ild+DN/NfePNTMtlcyvRCTMOSpgNFJNNQ0RdE7R0pezn /H58qEHDEc4R7CENE4nRaxR9i4QVl+s/bMw5KKKnajdUDL/7IyiD8ihx8ZEl8p6V w0HWuKB3yWY= =0R07 -----END PGP SIGNATURE----- --=-=-=--