From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id 6MFKHwRJUWfjBQEA62LTzQ:P1 (envelope-from ) for ; Thu, 05 Dec 2024 06:32:36 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id 6MFKHwRJUWfjBQEA62LTzQ (envelope-from ) for ; Thu, 05 Dec 2024 07:32:36 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=k3r5+Zmm; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1733380356; a=rsa-sha256; cv=none; b=GJXr70BjWttAw+3SvVml5YaUX8hQc9Y1KWBPowMVwo/SapWktLxBpPSO+AZQJMuLC7/Spw 059XHXvVmecqLuFjfQIV5AsqNO1REoH9HxRn/k8L8eune16Uyxhg8XBV8swlWYZgrhUbR8 Lsfv1iqG+5rrImVU756rlD2O07nDmA3tmTGzk7qZFwCgnGgL+kaHCQoKzEkH06Jkv8GfkQ PCEjyzQGkeTV2uqxkJI0aPUJeqwkyH+O0e2Kk5VQs03G8U5LFh1d3Dx1UeWz0LKA7uBjVS w8huHvFh4s9EVJKCWHALAVvFW1NV/QjrPT6PNFrMyO9pr7EuaLg7qe5UB7QluA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20230601 header.b=k3r5+Zmm; dmarc=none; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1733380356; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=6kBzAMwNaDrJk2YAE0Vk2LuTAL5aRIhfgKj1Utk+ACI=; b=lWpKJ0Jz0kAsjv54lJkcvEm94fM+GRvDfXhxvFjRE9WRCQBffXhiK3z/PnFdfMoHBreeUM zc22xs4JVtD5hiQDStWr8EyeX7LCksJ7rnUID9jbiKrjjDimfXLKw1mm+kXPg+ZjTiPD/P eiarVHGmkohFhQDMAqCudrZZOQkCF4k+T2dUVwMMIVVQ1T+eSKVK9S+jH+OwrXmxidlRrK Zcy0cqy5mOag53LUhZ+NkTX179WusVjFrKDpsrvypkupEq+H0pY00/YLCKmiORuyasMDB6 9Rrw3gGvPHlaTk7SRPsFBVtFKtVsywe0/JNu4TIjr5p8vPoD9Sn0t8M6WXvFLg== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2E82136D1 for ; Thu, 05 Dec 2024 07:32:36 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tJ5PG-0004C7-6P; Thu, 05 Dec 2024 01:32:10 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tJ5PE-0004Bf-3H for guix-devel@gnu.org; Thu, 05 Dec 2024 01:32:08 -0500 Received: from mail-wr1-x429.google.com ([2a00:1450:4864:20::429]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1tJ5PC-0001dq-6y for guix-devel@gnu.org; Thu, 05 Dec 2024 01:32:07 -0500 Received: by mail-wr1-x429.google.com with SMTP id ffacd0b85a97d-385e1f3f2a6so265723f8f.3 for ; Wed, 04 Dec 2024 22:32:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733380323; x=1733985123; darn=gnu.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:sender:from:to :cc:subject:date:message-id:reply-to; bh=6kBzAMwNaDrJk2YAE0Vk2LuTAL5aRIhfgKj1Utk+ACI=; b=k3r5+ZmmSpjCx3Orr8SrcRF02fe8XY17+9uxH0RsVzMPnzywFAG4dFV3PE0N+ESPkx FFYh973330l2C7S9p7o1xxOjrGwb/6f4t/w52UbZiK/T80e5yrbFPMl3DsJ6lkec7eUv XWQhX3/T4fPSTJcw9VzHKCXadig+YHnm3LoqTvjtQvHioDdb7Ovs6MNCpeyvH81hIOxR 7jU8DwwdI2q3mDNoiLuM/77pHgfX2ndH8zpsWaq8E+jYc6gNeachxiSluJIc+DxvM3od Zgy2edRtL3UdMpTt8IsupqK5j/O4L5YrhDdcXPfasZvgvzPG10SzpbaGrglWtGm8nv8H RZGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733380323; x=1733985123; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:sender :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6kBzAMwNaDrJk2YAE0Vk2LuTAL5aRIhfgKj1Utk+ACI=; b=H4Wli6w75lrZO9EEv8qq/YsPGt6AcwlmVHFJKBy9SJ/JkYmvQOvxz0Z98xsfeIqyDj QgNCLnIZdtLp0/e5N7CWLRrvLYAwPEJdCQoguuOX+moqLraGOx9HfZPoVlG9iP2ES4qL 2pvw0RoCh7Gcoq/3tgWfvb6hAEQ83MpgzBpFI1OPdc1mzSPs+VtYXbT/H4x1quVvgiL8 O6Yola/48LBx9Q1FAZ5PtWhXd4CtMYxL+ASNJIB7emya/ak6ZZEOtP53k7VRYIkG2K8H 5UXpjbM2xnLIDB2W4lcSpcFuZ6ZIiAdrEEJyBGHhE1bnJqiqo6kVmiXM8u5h0fAIh90T /zcg== X-Gm-Message-State: AOJu0Yx11gZ/ADzHI8QJ13P783XAuyCxDZDH94cDWHAXW+1sDXz38ZyE U67xG0JxPX3cw+WNOLM2aw68R7eTTisoj1nw2/bpg21OxaUK4zmP X-Gm-Gg: ASbGncuG0bNif5SApAGj1IgadqeMrKo74J4kFOl/h+OzXhyKAnv4zSkildJs2jjfNK7 7IIaw1WB4g0bdFk4wdgDm/5oAVDBe4+PVvYEzwj0BKW0ArxFzLvi4CWBfs/x5IdH7aNiehJHnv0 UhnB+K2KTsAdKXJePhNwDuQKqkEho+UqFcN4+6aoZhz1JIjhLrCnDjzZxp0gdsRp4O+dsLC9Qqh 5+CtMYSDBAAGHlstRnVqosGuJQYsshyorv0fk3X68z+gfg= X-Google-Smtp-Source: AGHT+IHlr207IAKnPtbNBLMQedlE4oNhX+MevOor2UGH587PJTPX3JVcN04BxQftce8IkWJHPLeTZw== X-Received: by 2002:a05:6000:70d:b0:385:e17a:ce6f with SMTP id ffacd0b85a97d-385fd3e9d2fmr8430596f8f.24.1733380323244; Wed, 04 Dec 2024 22:32:03 -0800 (PST) Received: from localhost ([141.226.9.46]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3861fc51654sm1045015f8f.49.2024.12.04.22.32.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Dec 2024 22:32:01 -0800 (PST) Date: Thu, 5 Dec 2024 08:32:00 +0200 From: Efraim Flashner To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel Subject: Re: =?utf-8?Q?=E2=80=98cargo-build-system?= =?utf-8?B?4oCZ?= makes everything slow Message-ID: Mail-Followup-To: Ludovic =?utf-8?Q?Court=C3=A8s?= , guix-devel References: <87ldwy3uhr.fsf@inria.fr> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="ozPvvoGfMw7jLsch" Content-Disposition: inline In-Reply-To: <87ldwy3uhr.fsf@inria.fr> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Received-SPF: pass client-ip=2a00:1450:4864:20::429; envelope-from=efraim.flashner@gmail.com; helo=mail-wr1-x429.google.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.001, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Queue-Id: 2E82136D1 X-Migadu-Scanner: mx13.migadu.com X-Migadu-Spam-Score: -6.53 X-Spam-Score: -6.53 X-TUID: N8H1QLTOiHmW --ozPvvoGfMw7jLsch Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Dec 02, 2024 at 05:44:16PM +0100, Ludovic Court=C3=A8s wrote: > Hi, >=20 > I was profiling the =E2=80=98dependents=E2=80=99 procedure use by the new= =E2=80=98guix build > --dependents=E2=80=99 option in (guix scripts build). Like =E2=80=98guix= refresh -l=E2=80=99, > it lowers all the packages to a bag, constructs the graph, and traverses > all the nodes backward. And it=E2=80=99s slow (~25s). >=20 > The profile is clear (test program below): >=20 > --8<---------------cut here---------------start------------->8--- > 100.0% with-exception-handler at ice-9/boot-9.scm:1689:4 > 100.0% thunk at guix/store.scm:686:4 > 100.0% run-with-store at guix/store.scm:2196:0 > 100.0% #x7f1cb0ccbeb0 at guix/scripts/build.scm:596:6 > 100.0% #x7f1c9c353ef0 at guix/graph.scm:98:7 > 99.6% #x7f1cb1b61e00 at guix/store.scm:1925:0 > 99.6% loop at guix/store.scm:1925:0 > 99.6% add-edge at guix/graph.scm:92:7 > 99.6% #x7f1c9c3340bc at guix/scripts/graph.scm:187:10 > 96.3% #x15d3b3bc > 95.8% bag-node-edges at guix/scripts/graph.scm:159:0 > 93.1% package->bag at guix/packages.scm:1755:0 > 92.5% thunk at guix/packages.scm:1763:4 > 89.3% lower at guix/build-system/cargo.scm:305:0 > 33.5% vhash-assq at ice-9/vlist.scm:539:0 > 9.4% #x15d3cf98 > 21.0% #x15d3de90 > 3.6% vhash-cons at ice-9/vlist.scm:449:0 > 0.9% #x15d3cf98 > 0.1% #x15d3b430 > 0.1% #x15d3daf4 > 0.1% reap-pipes at ice-9/popen.scm:183:0 > 2.5% filter-map at srfi/srfi-1.scm:681:0 > 2.0% #x7f1cb0a997f4 at guix/build-system/ca= rgo.scm:298:4 > --8<---------------cut here---------------end--------------->8--- >=20 > 89% of the time is spent lowering =E2=80=98cargo-build-system=E2=80=99 pa= ckages, > specifically in =E2=80=98expand-crate-sources=E2=80=99, which computes mo= re or less the > same thing over and over again. (It also returns more than just > transitive sources of Crates, AFAICS.) >=20 > Problem is little can be done about it, at least not without triggering > a world rebuild. >=20 > But even then, it would help a lot to just not do that. >=20 > Thoughts? >=20 > Ludo=E2=80=99. If its a world rebuild through the cargo-build-system its not so bad, CI and QA build it out pretty quickly. If it's everything, then yeah, that's going to be slow. Can we cache it after the first time like with guix shell? Or a hash-table of some sort? (I'm just using big words, I don't remember the specifics of how the different hashing and caching bits work). Chris Baines sped up the time to compute derivations for cross-compiling with something with the make-rust-sysroot. > (use-modules (statprof) > (ice-9 time) > (guix) > ((gnu packages) #:select (all-packages)) > ((guix scripts build) #:select (dependents)) > (gnu packages version-control)) >=20 > (time (->bool (all-packages))) ;warm up >=20 > (with-store store > (statprof > (lambda () > (dependents store (list libgit2) 1)) > #:full-stacks? #t > #:display-style 'tree)) --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --ozPvvoGfMw7jLsch Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmdRSNwACgkQQarn3Mo9 g1G3QA/+NfRstTh7cDXyNgOQLFzl9Nid1XRrmwvba9SjvrrBhwSvJxiNCeN4eKpv AxStKvuI8TugP3FIYveg8OSXHKKHRjJJamMIvF63Qud5wLWupgGuS/8NsRB7CJ0d Avetxzrbkfkzkk9RCarw84wGU5biLp4vxCtOzTaYrncHT8/v1H7mi19P19RDPJbD 5Mqo8vPLQQN98fAMXgtiyrXCyoHTJMuIfnWEz3gHw8BuIfHyJF80KLZUf0KlYbdV ZiZQwVqSstmo/WtjH02lhgst1Lt/MUZRNOKCh4rFSww68Wt36JZbw3XFGQJff21e Ufi5+qn4Vv4+q4Mzt0/N39rKMYvOKr+PbEoyyvmpc4cLgsNPisZeiA3q4B2Xo0uU tzJOgUeiw3Y0RMt5rKP1TQPNthxALsnc514WlguXeK0qA3WcFQSZjavws++XtDfi zr9GFAAIl3HbTQQWVBtwuhx50AZ3NZxbSmt4T6g4jgdh7dIzlxupvq/lNkD+2dO8 /snN6LNbEdHe3PC5ET3KtNmADAjII+6J5wrdXNLqxhG66JxDo3gXVWoQWyCbLI9P CPyfp/IEuAdVd2hsUDlbp7BkJ9npY6i2YazsesMfcr+AloxanuDGY/1IzkCZCNGl es9HBobNyJzretabyWEPgy724Cu+xsQTMO8Q0Xg344bZc4Awb2U= =KUst -----END PGP SIGNATURE----- --ozPvvoGfMw7jLsch--