From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id cKIPC1U8nV9QWAAA0tVLHw (envelope-from ) for ; Sat, 31 Oct 2020 10:28:37 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id kHfTBlU8nV/oGAAAbx9fmQ (envelope-from ) for ; Sat, 31 Oct 2020 10:28:37 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 938FF9403A0 for ; Sat, 31 Oct 2020 10:28:36 +0000 (UTC) Received: from localhost ([::1]:34070 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kYo7f-0008LB-BH for larch@yhetil.org; Sat, 31 Oct 2020 06:28:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:51052) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kYo78-0007uH-Pn for bug-guix@gnu.org; Sat, 31 Oct 2020 06:28:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:49043) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kYo78-0005aX-FT for bug-guix@gnu.org; Sat, 31 Oct 2020 06:28:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kYo78-0006sb-CJ for bug-guix@gnu.org; Sat, 31 Oct 2020 06:28:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#44254: Performance of package input rewriting Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 31 Oct 2020 10:28:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 44254 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Lars-Dominik Braun Received: via spool by 44254-submit@debbugs.gnu.org id=B44254.160414003926383 (code B ref 44254); Sat, 31 Oct 2020 10:28:02 +0000 Received: (at 44254) by debbugs.gnu.org; 31 Oct 2020 10:27:19 +0000 Received: from localhost ([127.0.0.1]:60589 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kYo6R-0006rT-9K for submit@debbugs.gnu.org; Sat, 31 Oct 2020 06:27:19 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50826) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kYo6Q-0006rD-6n for 44254@debbugs.gnu.org; Sat, 31 Oct 2020 06:27:18 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:36287) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kYo6I-0005Jw-T3; Sat, 31 Oct 2020 06:27:12 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=56806 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1kYo6G-0007xd-2E; Sat, 31 Oct 2020 06:27:08 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20201030084245.GB3128@zpidnp36> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 10 Brumaire an 229 de la =?UTF-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sat, 31 Oct 2020 11:27:06 +0100 In-Reply-To: <20201030084245.GB3128@zpidnp36> (Lars-Dominik Braun's message of "Fri, 30 Oct 2020 09:42:45 +0100") Message-ID: <87tuuavgg5.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -3.3 (---) X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 44254@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Scanner: ns3122888.ip-94-23-21.eu Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Spam-Score: -1.51 X-TUID: 5w96YAz1W9EV Hi Lars, Lars-Dominik Braun skribis: >> Another thing to look at is the object graph (as show by =E2= =80=98guix >> graph=E2=80=99). Input rewriting can duplicate parts of the graph, whic= h in >> turn defeats package->derivation memoization. Just looking at the >> number of nodes in the graph can give hints. > Aha, it=E2=80=99s 913 nodes without rewriting, 13916 with rewriting (#:de= ep? #t) and > 4286 with rewriting (#:deep? #f) as determined by a rather ad-hoc `guix g= raph > -L . -t package python-jupyterlab | grep 'shape =3D box' | wc -l`. That s= eems way > too much. Does that mean I=E2=80=99m using package rewriting in the wrong= way or is > that a bug? It could be a mixture thereof. :-) I guess it=E2=80=99s easy to end up creating huge object graphs. Here=E2= =80=99s an example of an anti-pattern: (define a ((package-input-rewriting x) ((package-input-rewriting y) p1)))=20 (define b ((package-input-rewriting x) ((package-input-rewriting y) p2))) The correct use is: (define transform (package-input-rewriting (append x y))) (define a (transform p1)) (define b (transform p2)) That guarantees that =E2=80=98a=E2=80=99 and =E2=80=98b=E2=80=99 share most= of the nodes of their object graph. >From a quick look, the code in Guix-Science seemed to be following the pattern above. For example, there=E2=80=99s: --8<---------------cut here---------------start------------->8--- (define python-ipykernel-5.3-bootstrap (let ((rewritten ((package-input-rewriting `((,python-jupyter-client . ,python-jupyter-client-6.1-bootstrap) ;; Indirect through IPython. (,python-testpath . ,python-testpath-0.4) (,python-nbformat . ,python-nbformat-5.0))) python-ipykernel-5.3-proper))) (package (inherit rewritten) (name "python-ipykernel-bootstrap")))) (define-public python-jupyter-client-6.1 ((package-input-rewriting `((,python-ipykernel . ,python-ipykernel-5.3-bootstrap) (,python-jupyter-core . ,python-jupyter-core-4.6) ;; Indirect through IPython. (,python-testpath . ,python-testpath-0.4) (,python-nbformat . ,python-nbformat-5.0))) python-jupyter-client-6.1-proper)) (define-public python-ipykernel-5.3 ((package-input-rewriting `((,python-jupyter-client . ,python-jupyter-client-6.1) ;; Indirect through IPython. (,python-testpath . ,python-testpath-0.4) (,python-nbformat . ,python-nbformat-5.0))) python-ipykernel-5.3-proper)) --8<---------------cut here---------------end--------------->8--- It seems to me that you=E2=80=99re redefining a dependency graph, node by n= ode. Thus, you probably don=E2=80=99t need =E2=80=98package-input-rewriting=E2= =80=99 here. What you did in Guix-Science commit 972795a23cc9eb5a0bb1a2ffb5681d151fc4d4b0 looks more appropriate to me, in terms of style and semantics. Does that make sense? Thanks, Ludo=E2=80=99.