From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id OG/fALfY214WZQAA0tVLHw (envelope-from ) for ; Sat, 06 Jun 2020 17:56:07 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id VXNBOLbY215VJwAAbx9fmQ (envelope-from ) for ; Sat, 06 Jun 2020 17:56:06 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6B44A9401AD for ; Sat, 6 Jun 2020 17:56:06 +0000 (UTC) Received: from localhost ([::1]:57162 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jhd37-0005dd-DS for larch@yhetil.org; Sat, 06 Jun 2020 13:56:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52916) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jhd2g-0004vo-NS for bug-guix@gnu.org; Sat, 06 Jun 2020 13:55:38 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:40854) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jhd2g-0003sN-Dn for bug-guix@gnu.org; Sat, 06 Jun 2020 13:55:38 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jhczC-0006L6-54 for bug-guix@gnu.org; Sat, 06 Jun 2020 13:52:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#41702: `guix environment` performance issues Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 06 Jun 2020 17:52:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 41702 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Lars-Dominik Braun Received: via spool by 41702-submit@debbugs.gnu.org id=B41702.159146590024319 (code B ref 41702); Sat, 06 Jun 2020 17:52:02 +0000 Received: (at 41702) by debbugs.gnu.org; 6 Jun 2020 17:51:40 +0000 Received: from localhost ([127.0.0.1]:52388 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jhcyp-0006KB-UR for submit@debbugs.gnu.org; Sat, 06 Jun 2020 13:51:40 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43682) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jhcyn-0006Jr-F0 for 41702@debbugs.gnu.org; Sat, 06 Jun 2020 13:51:37 -0400 Received: from [2001:470:142:3::e] (port=34123 helo=fencepost.gnu.org) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jhbN9-0000lM-Mq; Sat, 06 Jun 2020 12:08:39 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=52514 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jhbN3-0006Ob-AW; Sat, 06 Jun 2020 12:08:33 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <20200604082316.GA3146@zpidnp36> Date: Sat, 06 Jun 2020 18:08:31 +0200 In-Reply-To: <20200604082316.GA3146@zpidnp36> (Lars-Dominik Braun's message of "Thu, 4 Jun 2020 10:23:16 +0200") Message-ID: <87mu5gtbwg.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: -3.3 (---) X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: 41702@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Spam-Score: -1.01 X-TUID: PjQT1/XsCsU2 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Lars-Dominik Braun skribis: > Total time: 24.672604202 seconds (19.431122691 seconds in GC) > ./pre-inst-env guix environment --ad-hoc r-learnr -- true 25,18s user 0,= 24s system 308% cpu 8,248 total > > More specifically in an anonymous function and reap-pipes, which is a gc = hook, > I believe: > > % cumulative self > time seconds seconds calls procedure > 33.41 14.49 8.24 anon #xbb8480 > 27.95 6.90 6.90 ice-9/popen.scm:145= :0:reap-pipes > 4.37 1.08 1.08 anon #xbbdcd8 > 3.28 0.86 0.81 ice-9/vlist.scm:539= :0:vhash-assq > 2.40 2.37 0.59 guix/grafts.scm:202= :22 Guile master has a fix for statprof that yields more useful info: --8<---------------cut here---------------start------------->8--- scheme@(guile-user)> ,use(guix scripts environment) scheme@(guile-user)> ,pr (guix-environment "--ad-hoc" "r-learnr" "--" "true= ") % cumulative self=20=20=20=20=20=20=20=20=20=20=20=20=20 time seconds seconds procedure 29.84 9.87 6.16 append 19.56 4.04 4.04 %after-gc-thunk 6.85 1.87 1.42 ice-9/vlist.scm:539:0:vhash-assq 5.44 1.17 1.12 write 3.23 0.67 0.67 guix/derivations.scm:665:0:derivation->output-p= aths 2.82 0.58 0.58 string=3D? 2.42 2.37 0.50 guix/grafts.scm:202:22 2.42 0.50 0.50 list? 2.22 0.46 0.46 hashq 2.02 0.42 0.42 display 1.61 15.82 0.33 guix/grafts.scm:186:0:reference-origin 1.61 0.87 0.33 guix/grafts.scm:204:31 1.21 0.33 0.25 guix/derivations.scm:667:7 1.21 0.29 0.25 srfi/srfi-1.scm:817:0:any 1.01 1232.14 0.21 srfi/srfi-1.scm:584:5:map1 0.81 0.83 0.17 guix/derivations.scm:697:0:derivation/masked-in= puts 0.81 0.75 0.17 srfi/srfi-1.scm:580:2:map 0.81 0.17 0.17 guix/derivations.scm:158:0:%derivation-input-de= rivation-procedure 0.60 0.17 0.12 reverse 0.60 0.12 0.12 hashq-ref 0.60 0.12 0.12 get-bytevector-n 0.60 0.12 0.12 procedure? 0.40 0.67 0.08 guix/packages.scm:1232:0:fold-bag-dependencies 0.40 0.12 0.08 string->utf8 0.40 0.12 0.08 ice-9/vlist.scm:534:0:vhash-assoc 0.40 0.12 0.08 ice-9/vlist.scm:449:0:vhash-cons 0.40 0.12 0.08 delete-duplicates 0.40 0.08 0.08 ice-9/boot-9.scm:1389:0:->bool 0.40 0.08 0.08 ice-9/boot-9.scm:2201:0:%load-announce 0.40 0.08 0.08 hash 0.40 0.08 0.08 guix/derivations.scm:665:0:derivation->output-p= aths 0.20 20.73 0.04 guix/gexp.scm:1061:2 --8<---------------cut here---------------end--------------->8--- Notice that the same command with =E2=80=98--no-grafts=E2=80=99 takes 2s in= stead of 11s. The patch below arranges so that =E2=80=98cumulative-grafts=E2=80=99 proces= ses dependencies in a batch, such that the derivation=E2=80=99s dependency grap= h is traversed once for all, which makes a difference for derivations with lots of inputs. Here=E2=80=99s the before/after comparison: --8<---------------cut here---------------start------------->8--- $ time guix environment --ad-hoc r-learnr --search-paths export PATH=3D"/gnu/store/n4wxbmqpafjfyawrla8xymzzdm5hxwph-profile/bin${PAT= H:+:}$PATH" real 0m11.328s user 0m20.155s sys 0m0.172s $ time ./pre-inst-env guix environment --ad-hoc r-learnr --search-paths export PATH=3D"/gnu/store/if6z77la3mx0qdzvcyl4qv9i5cyp48i0-profile/bin${PAT= H:+:}$PATH" real 0m4.602s user 0m6.189s sys 0m0.136s --8<---------------cut here---------------end--------------->8--- There=E2=80=99s still room for improvement, but it=E2=80=99s much better. Ludo=E2=80=99. --=-=-= Content-Type: text/x-patch Content-Disposition: inline diff --git a/guix/grafts.scm b/guix/grafts.scm index 69d6fe4469..910dcadc8a 100644 --- a/guix/grafts.scm +++ b/guix/grafts.scm @@ -20,10 +20,12 @@ #:use-module (guix store) #:use-module (guix monads) #:use-module (guix records) + #:use-module (guix combinators) #:use-module (guix derivations) #:use-module ((guix utils) #:select (%current-system)) #:use-module (guix sets) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-9 gnu) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) @@ -183,32 +185,47 @@ references." (set-current-state (vhash-cons key result cache)) (return result))))))) -(define (reference-origin drv item) - "Return the derivation/output pair among the inputs of DRV, recursively, -that produces ITEM. Return #f if ITEM is not produced by a derivation (i.e., -it's a content-addressed \"source\"), or if it's not produced by a dependency -of DRV." +(define (reference-origins drv items) + "Return the derivation/output pairs among the inputs of DRV, recursively, +that produce ITEMS. Elements of ITEMS not produced by a derivation (i.e., +it's a content-addressed \"source\"), or not produced by a dependency of DRV, +have no corresponding element in the resulting list." + (define (lookup-derivers drv result items) + ;; Return RESULT augmented by all the drv/output pairs producing one of + ;; ITEMS, and ITEMS stripped of matching items. + (fold2 (match-lambda* + (((output . file) result items) + (if (member file items) + (values (alist-cons drv output result) + (delete file items)) + (values result items)))) + result items + (derivation->output-paths drv))) + ;; Perform a breadth-first traversal of the dependency graph of DRV in - ;; search of the derivation that produces ITEM. + ;; search of the derivations that produce ITEMS. (let loop ((drv (list drv)) + (items items) + (result '()) (visited (setq))) (match drv (() - #f) + result) ((drv . rest) - (if (set-contains? visited drv) - (loop rest visited) - (let ((inputs (derivation-inputs drv))) - (or (any (lambda (input) - (let ((drv (derivation-input-derivation input))) - (any (match-lambda - ((output . file) - (and (string=? file item) - (cons drv output)))) - (derivation->output-paths drv)))) - inputs) - (loop (append rest (map derivation-input-derivation inputs)) - (set-insert drv visited))))))))) + (cond ((null? items) + result) + ((set-contains? visited drv) + (loop rest items result visited)) + (else + (let*-values (((inputs) + (map derivation-input-derivation + (derivation-inputs drv))) + ((result items) + (fold2 lookup-derivers + result items inputs))) + (loop (append rest inputs) + items result + (set-insert drv visited))))))))) (define* (cumulative-grafts store drv grafts #:key @@ -233,25 +250,27 @@ derivations to the corresponding set of grafts." (_ #f))) - (define (dependency-grafts item) - (match (reference-origin drv item) - ((drv . output) - ;; If GRAFTS already contains a graft from DRV, do not override it. - (if (find (cut graft-origin? drv <>) grafts) - (state-return grafts) - (cumulative-grafts store drv grafts - #:outputs (list output) - #:guile guile - #:system system))) - (#f - (state-return grafts)))) + (define (dependency-grafts items) + (mapm %store-monad + (lambda (drv+output) + (match drv+output + ((drv . output) + ;; If GRAFTS already contains a graft from DRV, do not + ;; override it. + (if (find (cut graft-origin? drv <>) grafts) + (state-return grafts) + (cumulative-grafts store drv grafts + #:outputs (list output) + #:guile guile + #:system system))))) + (reference-origins drv items))) (with-cache (cons (derivation-file-name drv) outputs) (match (non-self-references store drv outputs) (() ;no dependencies (return grafts)) (deps ;one or more dependencies - (mlet %state-monad ((grafts (mapm %state-monad dependency-grafts deps))) + (mlet %state-monad ((grafts (dependency-grafts deps))) (let ((grafts (delete-duplicates (concatenate grafts) equal?))) (match (filter (lambda (graft) (member (graft-origin-file-name graft) deps)) --=-=-=--