From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id GLcqGonUAWGphgAAgWs5BA (envelope-from ) for ; Thu, 29 Jul 2021 00:04:57 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 0KChFYnUAWHGZQAAbx9fmQ (envelope-from ) for ; Wed, 28 Jul 2021 22:04:57 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D2AF7C421 for ; Thu, 29 Jul 2021 00:04:56 +0200 (CEST) Received: from localhost ([::1]:53672 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m8rfb-0000EH-AL for larch@yhetil.org; Wed, 28 Jul 2021 18:04:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34864) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m8rek-0000At-AY for bug-guix@gnu.org; Wed, 28 Jul 2021 18:04:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44451) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m8rek-000861-1N for bug-guix@gnu.org; Wed, 28 Jul 2021 18:04:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1m8rej-00040e-St for bug-guix@gnu.org; Wed, 28 Jul 2021 18:04:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#49439: grafts cause =?UTF-8?Q?=E2=80=9Cguix_?= =?UTF-8?Q?environment=E2=80=9D?= to get killed with OOM Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Wed, 28 Jul 2021 22:04:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49439 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Sarah Morgensen Received: via spool by 49439-submit@debbugs.gnu.org id=B49439.162750983715401 (code B ref 49439); Wed, 28 Jul 2021 22:04:01 +0000 Received: (at 49439) by debbugs.gnu.org; 28 Jul 2021 22:03:57 +0000 Received: from localhost ([127.0.0.1]:55997 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m8ref-00040L-24 for submit@debbugs.gnu.org; Wed, 28 Jul 2021 18:03:57 -0400 Received: from eggs.gnu.org ([209.51.188.92]:37074) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m8red-000400-V2 for 49439@debbugs.gnu.org; Wed, 28 Jul 2021 18:03:56 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42200) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m8reX-0007y4-B1; Wed, 28 Jul 2021 18:03:49 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=45320 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m8reX-0001g7-3L; Wed, 28 Jul 2021 18:03:49 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87sg0rse1n.fsf@elephly.net> <86tuklr5g6.fsf@mgsn.dev> <87y29r3enb.fsf@gnu.org> <867dhbpbyt.fsf@mgsn.dev> <87k0la3ghk.fsf@gnu.org> Date: Thu, 29 Jul 2021 00:03:46 +0200 In-Reply-To: <87k0la3ghk.fsf@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Wed, 28 Jul 2021 12:00:39 +0200") Message-ID: <87wnpa14fx.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Wurmus , 49439@debbugs.gnu.org Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1627509897; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post; bh=/ZJxJbF6ZRus7dfZE5gWGm2jKfgvuomv8QgcVI0+k+w=; b=qsXK36MlO9fpdbKujWqCd2cWlU2grs1n0qoVXY5sP7qhG/g97U3YVpTNbkpDTpRH1t+Q2J WIEvuedpntp0b0Z5SCp5Okjkbn3EVujsVtmajMS1FGWT1iQ9uiqgKs0fvgXrLKZLEC81ot BX7DOeYAhriu8dTl+dOqJUDhEHrMyXVh7/5c1JY+ybHLl45c7nmjTaWu0EIyde+UNrNuPd wcXwluqHyJWyzcnXsuAbQ9ULs8GAvMRB4a/xs6QnXQV2lEGC+GAhTbwnDNd2Znpo8UAxks atNqr5aaOAvWSZrjvuznykc/RLJpfL4zTuOfhOna2czNdvx4W74I7Zg4u5Pq2g== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1627509897; a=rsa-sha256; cv=none; b=ndJRPuCteh4fLHinxBR5eE7i+Dp2X37/Clp5ZG+vM1LtX9aY78hFTA3HLL9ah3zA3xqRcU LX2ywkIKdkZto7zhk2lCvfab7a4GlVab94SO4BOm7sazs473RluhJspH6oxXntnLaWI+RB Qs7+/1JeF7a0kQtBVCF6ioj0fdbREAZoRCQopx8jX8UKTpjUlA/N07y2njp4sqfnX0eCXB f4ynkpNE0nrexOFpQXqWFdKK5iKx90JjEBsE+228fKy/F+XssByCu7Fb/Dj3ZIZowdXMFY k2pakgGS9rbOsQVT2lYE3houp7SJoOyQBw90iY60BKhgfLNao/SpDU6fJkypaQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Spam-Score: -2.92 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: D2AF7C421 X-Spam-Score: -2.92 X-Migadu-Scanner: scn0.migadu.com X-TUID: SsBDOIEFSGLK Hi, Ludovic Court=C3=A8s skribis: > Thinking about it, the grafts code depends on what=E2=80=99s in the store= : when > nothing is in the store, it bounces to the =E2=80=9Cbuild handler=E2=80= =9D, which > accumulates the list of missing store items, until it starts building > them. So I can reproduce the problem Ricardo and you initially reported by running: ./pre-inst-env guix environment pigx-scrnaseq --search-paths after removing some of the ungrafted store items with: guix gc -D $(guix build r-rlang --no-grafts) \ $(guix gc --references $(guix build pigx-scrnaseq --no-grafts)) The seemingly endless CPU usage and unbound memory use comes from the =E2=80=98build-accumulator=E2=80=99 build handler. Here, the graph of =E2= =80=98pigx-scrnaseq=E2=80=99 has many nodes, and many of them depend on, say, =E2=80=98r-rlang=E2=80=99.= Thus, when =E2=80=98r-rlang=E2=80=99 is not in the store, the grafting code keeps aski= ng for it by calling =E2=80=98build-derivations=E2=80=99, which aborts to the build hand= ler; the build handler saves the .drv file name and the continuation and keeps going. But since the next package also depends on =E2=80=98r-langr=E2=80= =99, we abort again to the build handler, and so on. The end result is a very long list of nodes, probably of this order in this case: $ guix graph -t reverse-package r-rlang |grep 'label =3D "'|wc -l 594 Presumably, the captured continuations occupy quite a bit of memory, hence the quick growth. I suppose one solution is to fire suspended builds when the build handler sees a build request for a given derivation for the second time. It needs more thought and testing=E2=80=A6 Ludo=E2=80=99. PS: Did you know =E2=80=98pigx-scrnaseq=E2=80=99 has twice as many nodes as =E2=80=98libreoffice=E2=80=99? $ guix graph -t bag pigx-scrnaseq |grep 'label =3D "'|wc -l 1359 $ guix graph -t bag libreoffice |grep 'label =3D "'|wc -l 699 That makes it a great example to study and fix scalability issues!