From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id cABlFhUeAGFaaAAAgWs5BA (envelope-from ) for ; Tue, 27 Jul 2021 16:54:13 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id sAYZEhUeAGH4GwAAB5/wlQ (envelope-from ) for ; Tue, 27 Jul 2021 14:54:13 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1E3FAF515 for ; Tue, 27 Jul 2021 16:54:13 +0200 (CEST) Received: from localhost ([::1]:49046 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m8OTE-0001Kc-6G for larch@yhetil.org; Tue, 27 Jul 2021 10:54:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42914) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m8OT3-0001KB-Rc for bug-guix@gnu.org; Tue, 27 Jul 2021 10:54:01 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:41990) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m8OT3-0005m8-K1 for bug-guix@gnu.org; Tue, 27 Jul 2021 10:54:01 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1m8OT3-0007Um-JN for bug-guix@gnu.org; Tue, 27 Jul 2021 10:54:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#49439: grafts cause =?UTF-8?Q?=E2=80=9Cguix_?= =?UTF-8?Q?environment=E2=80=9D?= to get killed with OOM Resent-From: Julien Lepiller Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Tue, 27 Jul 2021 14:54:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49439 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 49439@debbugs.gnu.org, iskarian@mgsn.dev, rekado@elephly.net X-Debbugs-Original-To: bug-guix@gnu.org, Sarah Morgensen , Ricardo Wurmus X-Debbugs-Original-Cc: 49439@debbugs.gnu.org Received: via spool by submit@debbugs.gnu.org id=B.162739759228741 (code B ref -1); Tue, 27 Jul 2021 14:54:01 +0000 Received: (at submit) by debbugs.gnu.org; 27 Jul 2021 14:53:12 +0000 Received: from localhost ([127.0.0.1]:53534 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m8OSF-0007TU-OQ for submit@debbugs.gnu.org; Tue, 27 Jul 2021 10:53:12 -0400 Received: from lists.gnu.org ([209.51.188.17]:35998) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m8OSE-0007TN-Bw for submit@debbugs.gnu.org; Tue, 27 Jul 2021 10:53:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:42554) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m8OSE-0000vA-3X for bug-guix@gnu.org; Tue, 27 Jul 2021 10:53:10 -0400 Received: from lepiller.eu ([2a00:5884:8208::1]:42562) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m8OS9-00054B-KU for bug-guix@gnu.org; Tue, 27 Jul 2021 10:53:09 -0400 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id d20c2537; Tue, 27 Jul 2021 14:52:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date :in-reply-to:references:mime-version:content-type :content-transfer-encoding:subject:to:cc:from:message-id; s= dkim; bh=AqQtugjQphSi3WixQ/xCqEtGTXBRTyG1qSnUwcWps0o=; b=I6u3cvT 5Knx2v+0o84lDdNPmuku5/8QZVqtJ1pdTuO+ENBB1dMrWieiWxeo2apcbyulBM/R m0E/hVBwi4EN1Attjf+LJy9yln0vsB9WBSmDu1EHHMzMNZJ328ebH4Fn47/+2QDE 8fgGKOb5ixIHNy5tNnu7JX9ZjNIDhX4icZF+uyc0YoP1BMCk1YqIC3c2IKNLzhFD Vpnnk+Qdj35K+lrwzv6FYx7y4qgod2T0glPU3qBTpXOvk9gViIhZrBClyCIDTM3a aBWU2qL5IHghMoLV7p6qLVjVgaaciwQF7lFKeJGATmNks0svxcb4Gp3xpThpUwoN QHFgOxeQBJIN/Pw== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id 0325c994 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Tue, 27 Jul 2021 14:52:56 +0000 (UTC) Date: Tue, 27 Jul 2021 10:52:44 -0400 User-Agent: K-9 Mail for Android In-Reply-To: <86tuklr5g6.fsf@mgsn.dev> References: <87sg0rse1n.fsf@elephly.net> <86tuklr5g6.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----WOJJJVY4L3END7ODIMFCSXM2YM5MTH" Content-Transfer-Encoding: 7bit From: Julien Lepiller Message-ID: <464151E3-FBF5-4B80-947E-0F1291FD879D@lepiller.eu> Received-SPF: pass client-ip=2a00:5884:8208::1; envelope-from=julien@lepiller.eu; helo=lepiller.eu X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -4.00 Authentication-Results: aspmx1.migadu.com; none X-Migadu-Queue-Id: 1E3FAF515 X-Spam-Score: -4.00 X-Migadu-Scanner: scn1.migadu.com X-TUID: CLI0Fa4+LggP ------WOJJJVY4L3END7ODIMFCSXM2YM5MTH Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I have a similar issue with an ocaml package I use at work=2E It's not free= software, but all its dependencies are=2E The dependencies are not all yee= t in guix, so to reproduce you might have to import them first with "guix i= mport opam -r foo" for ocaml-foo=2E The package depends on ocaml-ounit, ocaml-lp, ocaml-apronext, menhir, ocam= l-async, ocaml-core, ocaml-graph, ocaml-libsvm, ocaml-minisat, ocaml-ppx-de= riving-yojson, ocaml-yojson, ocaml-z3, ocaml-zarith and z3=2E In total, that's 118 transitive inputs=2E Building the profile takes 30 mi= nutes for me, on an SSD=2E The builder takes 1=2E5g resident=2E Other than that, I measured time and memory for creating the environment w= hen the profile was already built (no more derivation to build): `which time` -v ~/guix/pre-inst-env guix environment ocaml-dummy-package -= - echo done User time: 121=2E43s System time: 2=2E28s Maximum resident: 1803028kB (1=2E8 GB) With a warning from GC: Repeated allocation of very large block (approx=2E size 35606528) Note that I get the same numbers with --no-grafts, so it might be a differ= ent issue=2E "guix build" terminates quickly=2E Le 23 juillet 2021 00:59:21 GMT-04:00, Sarah Morgensen a =C3=A9crit : >Hello, > >Ricardo Wurmus writes: > >> With a recent version of Guix, =E2=80=9Cguix environment=E2=80=9D will = not=20 >> terminate on its own, keeps the CPU busy, and gets killed when the=20 >> system eventually runs out of memory=2E >> >> $ guix describe -f channels >> >> (list (channel >> (name 'guix) >> (url "/home/rekado/dev/gx/branches/master") >> (commit >> "685cfdec94e5e48c4ad28de53466a28dfc258edb"))) >> >> >> $ guix environment pigx-scrnaseq >> [wait until it gets killed] > >I can reproduce this with pigx-scrnaseq as well a number of other >packages (listed below)=2E > >$ =2E/pre-inst-env guix describe -f channels >(list (channel > (name 'guix) > (url "/home/sarah/guix") > (commit > "3217a04b0352c2dd13323257b369604eeabfccc3"))) > >Does not complete within 5 minutes: >package # inputs # transitive inputs > (from package-transitive-inputs) >pigx-chipseq 48 338 >pigx-scrnaseq 41 321 >r-cellchat 34 110 >pigx-rnaseq 34 343 >pigx-bsseq 32 358 >pigx-sars-cov2-ww 25 261 >r-circus 16 134 > >Does complete: >r-chipseq 6 37 completes in >2m >r-shortread 17 36 completes in >1m >python-scanpy 25 113 completes in <15s > >I suspect it has something to do with the number of transitive inputs, >because it is so prevalent with these R packages, which all use >propagated inputs=2E However=2E=2E=2E python-scanpy succeeds in under 15 >seconds, and it has more transitive inputs than r-chipseq=2E > >Can we reproduce this with a large number of low-transitivity packages >directly on the command line? > >> >> The problem disappears when grafts are disabled: >> >> $ guix environment --no-grafts pigx-scrnaseq >> $ [env] yay! > >-- >Sarah ------WOJJJVY4L3END7ODIMFCSXM2YM5MTH Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable I have a similar issue with an ocaml package I use= at work=2E It's not free software, but all its dependencies are=2E The dep= endencies are not all yeet in guix, so to reproduce you might have to impor= t them first with "guix import opam -r foo" for ocaml-foo=2E

The pac= kage depends on ocaml-ounit, ocaml-lp, ocaml-apronext, menhir, ocaml-async,= ocaml-core, ocaml-graph, ocaml-libsvm, ocaml-minisat, ocaml-ppx-deriving-y= ojson, ocaml-yojson, ocaml-z3, ocaml-zarith and z3=2E

In total, that= 's 118 transitive inputs=2E Building the profile takes 30 minutes for me, o= n an SSD=2E The builder takes 1=2E5g resident=2E

Other than that, I = measured time and memory for creating the environment when the profile was = already built (no more derivation to build):

`which time` -v ~/guix/= pre-inst-env guix environment ocaml-dummy-package -- echo done

User = time: 121=2E43s
System time: 2=2E28s
Maximum resident: 1803028kB (1= =2E8 GB)

With a warning from GC:

Repeated allocation of very = large block (approx=2E size 35606528)

Note that I get the same numbe= rs with --no-grafts, so it might be a different issue=2E

"guix build= " terminates quickly=2E

Le 23 juillet 202= 1 00:59:21 GMT-04:00, Sarah Morgensen <iskarian@mgsn=2Edev> a =C3=A9c= rit :
Hello,

Ricardo Wurmus <rekado@elephly=2Ene=
t> writes:

 With=
 a recent version of Guix, =E2=80=9Cguix environment=E2=80=9D will not 
=
 terminate on its own, keeps the CPU busy, and gets killed when the 
 sy=
stem eventually runs out of memory=2E

 $ guix describe -f channels
 (list (channel
         (name 'guix)
         (url "/home/reka=
do/dev/gx/branches/master")
         (commit
           "685cfdec94e5=
e48c4ad28de53466a28dfc258edb")))


 $ guix environment pigx-scrnas=
eq
 [wait until it gets killed]

I can reproduce this=
 with pigx-scrnaseq as well a number of other
packages (listed below)=2E=


$ =2E/pre-inst-env guix describe -f channels
(list (channel
 =
       (name 'guix)
        (url "/home/sarah/guix")
        (commit<=
br>          "3217a04b0352c2dd13323257b369604eeabfccc3")))

Does not =
complete within 5 minutes:
package            # inputs  # transitive inp=
uts
                                 (from package-transitive-inputs)pigx-chipseq       48        338
pigx-scrnaseq      41        321
r-=
cellchat         34        110
pigx-rnaseq        34        343
pigx-=
bsseq         32        358
pigx-sars-cov2-ww  25        261
r-circus=
           16        134

Does complete:
r-chipseq          6     =
    37     completes in >2m
r-shortread        17        36     compl=
etes in >1m
python-scanpy      25        113    completes in <15s<=
br>
I suspect it has something to do with the number of transitive input=
s,
because it is so prevalent with these R packages, which all use
pr=
opagated inputs=2E However=2E=2E=2E python-scanpy succeeds in under 15
s=
econds, and it has more transitive inputs than r-chipseq=2E

Can we r=
eproduce this with a large number of low-transitivity packages
directly =
on the command line?