From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id AEeWJ95Ie2Qi6QAASxT56A (envelope-from ) for ; Sat, 03 Jun 2023 16:06:22 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id YBFJJ95Ie2T8XgAA9RJhRA (envelope-from ) for ; Sat, 03 Jun 2023 16:06:22 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 390EE2CE57 for ; Sat, 3 Jun 2023 16:06:22 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q5RtO-0003v3-1s; Sat, 03 Jun 2023 10:06:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q5RtL-0003uj-UR for bug-guix@gnu.org; Sat, 03 Jun 2023 10:06:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1q5RtK-00050L-8Z for bug-guix@gnu.org; Sat, 03 Jun 2023 10:06:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1q5RtK-00045N-3Q for bug-guix@gnu.org; Sat, 03 Jun 2023 10:06:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#63852: =?UTF-8?Q?=E2=80=98guix-package-cache.drv=E2=80=99?= eats all your memory with =?UTF-8?Q?=E2=80=98guix-cran=E2=80=99?= Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 03 Jun 2023 14:06:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 63852 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 63852@debbugs.gnu.org Cc: Lars-Dominik Braun Received: via spool by 63852-submit@debbugs.gnu.org id=B63852.168580113515661 (code B ref 63852); Sat, 03 Jun 2023 14:06:02 +0000 Received: (at 63852) by debbugs.gnu.org; 3 Jun 2023 14:05:35 +0000 Received: from localhost ([127.0.0.1]:44017 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q5Rst-00044X-9o for submit@debbugs.gnu.org; Sat, 03 Jun 2023 10:05:35 -0400 Received: from eggs.gnu.org ([209.51.188.92]:48290) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1q5Rso-000449-Gm for 63852@debbugs.gnu.org; Sat, 03 Jun 2023 10:05:33 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q5Rsi-0004rz-V4; Sat, 03 Jun 2023 10:05:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=7xuf+T9Wcjp2S7mUtylyERRPOZ5SxelTlHkSSL7xzpA=; b=I31onZnZz6zEYGk7jE+2 hWHGmwoczI40jpH3kjEd0J0Zp1OBCDDvCGHp0sBQqTxvuoZsnpWp6iuEd60OvStH5OooE70VSUs1D StVOEnrR5V1kHnngh0/5qDkl06aOFT9ri0JTpz3WFjGK9Wu0U3Tycl4TvPBGGj8CpM38iu1IRzWZR 0muxNXnGngE2zG+jrOc3bZXOWOleR653QrVZ/P4lU8Uld/gNLMD0ZPH8YO+q6A3Ir5kEgWJ/Itm0p dADTcSkmNCP4ppREtWL2vAOvT/DyVt7V4kudrf33zZZQ/11jzBLiPgJ5E3RVyZPz/cmSKlUcdTij/ bSQ8P8gYD3A6ug==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q5RsV-0003xI-Be; Sat, 03 Jun 2023 10:05:24 -0400 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= References: <87sfb93kca.fsf@inria.fr> Date: Sat, 03 Jun 2023 16:05:08 +0200 In-Reply-To: <87sfb93kca.fsf@inria.fr> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Fri, 02 Jun 2023 18:22:29 +0200") Message-ID: <875y843aln.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1685801182; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=7xuf+T9Wcjp2S7mUtylyERRPOZ5SxelTlHkSSL7xzpA=; b=O63rm/g/lklyNY2U1v+KBMNMJPH0Ou/zI+e7fN/iO4+AuTmXktDhLw5FkZ+KvW2MkbIQBo A2e2ljAZVfmqabVWpt8FbQeQGpkCdpxMcyC+u5mf2fM3X96e/k24miZdB5X4XQQrpmh4Of iKonv5EN8jgQmdfr3l1vPBERK+xOaTi6V3hldnAIzBDc1MpnoLvTqJuJFaSOq8qQ0RjUES 6FS82mesW+mq+LEid46O74pMByBagAYzUIe5IIkGxk/zM6OmWl21yyYlYzLgiP7Vmf7Vl3 wzIowyNvF0hTnYAOXaB9vXEBPo95rtTUvIXFaCUjUBPqdCf9yVLLsKwdHN+Jqg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=I31onZnZ; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1685801182; a=rsa-sha256; cv=none; b=bTZcm0EoXqKmBo2CRvu7QU5ImfC3ZJNndN/ebp6S+dTnC1Rfd99c5s/p+XKRljnC8cvW1Q 9aBHhl9n9BZnv9yFPy98DJjrB+ye/e97f+KyHaUhfAT50+Vc+1BB99GAcs7a8VHuKFKE22 D/DzJ88hbGMYKxWLfAkuhVxJ/2wj1Uozkf6RPSyDM96+x2wN5Yh+c+SHQY9r8juZ3vmenO GfmHMjEMSpiyUSLPn/o4/U2YBtwc3BK4yCidZTdUOz6DZu2mmvlHl9jnsUfY9Lgw+tXIDh IO7m1NizceRmPXnab5JX2kFNgO+3iIzevfIAn3pn65k6ClHO8xq7ZtthqXShag== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -4.08 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=I31onZnZ; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 390EE2CE57 X-Spam-Score: -4.08 X-TUID: WwLtuJ/bKMGy --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s skribis: > It goes OK until building /gnu/store/6rfaqfq693vda59a55asc4wjjg52ilns-gui= x-package-cache.drv, which > consumes memory until it gets OOM-killed. The culprit is easily found (here on just the 22K packages of =E2=80=98guix= =E2=80=99, so less than half of what you get with =E2=80=98guix-cran=E2=80=99): --8<---------------cut here---------------start------------->8--- $ guix repl GNU Guile 3.0.9 Copyright (C) 1995-2023 Free Software Foundation, Inc. Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'. This program is free software, and you are welcome to redistribute it under certain conditions; type `,show c' for details. Enter `,help' for help. scheme@(guix-user)> ,m(gnu packages) scheme@(gnu packages)> ,use(statprof) scheme@(gnu packages)> (gcprof (lambda () (generate-package-cache "/tmp/t")= )) % cumulative self=20=20=20=20=20=20=20=20=20=20=20=20=20 time seconds seconds procedure 50.00 13.77 13.77 ice-9/boot-9.scm:2498:4 6.94 1.91 1.91 display 5.56 1.53 1.53 object->string 2.78 0.76 0.76 reverse 2.78 0.76 0.76 gensym 2.78 0.76 0.76 ice-9/boot-9.scm:2217:0:%load-announce 1.39 6.88 0.38 gnu/packages.scm:388:4:expand-cache 1.39 0.76 0.38 srfi/srfi-1.scm:1028:0:lset-intersection 1.39 0.38 0.38 number->string 1.39 0.38 0.38 gnu/packages/crates-io.scm:17783:2:arguments 1.39 0.38 0.38 open-output-string 1.39 0.38 0.38 gnu/packages/crates-io.scm:18450:2:arguments 1.39 0.38 0.38 gnu/packages/crates-io.scm:36208:2:arguments 1.39 0.38 0.38 guix/packages.scm:1320:8:mproc 1.39 0.38 0.38 gnu/packages/crates-io.scm:25101:2:arguments 1.39 0.38 0.38 gnu/packages/crates-io.scm:29972:2:arguments 1.39 0.38 0.38 gnu/packages/crates-io.scm:883:2:arguments 1.39 0.38 0.38 ice-9/vlist.scm:449:0:vhash-cons 1.39 0.38 0.38 append 1.39 0.38 0.38 string-append 1.39 0.38 0.38 system/vm/assembler.scm:2175:8 1.39 0.38 0.38 guix/build/syscalls.scm:1168:0:read-dirent-head= er/linux 1.39 0.38 0.38 srfi/srfi-1.scm:1033:17 1.39 0.38 0.38 gnu/packages/crates-io.scm:9336:2:arguments 1.39 0.38 0.38 gnu/packages/crates-io.scm:64009:2:arguments 1.39 0.38 0.38 guix/build-system/cargo.scm:229:4 1.39 0.38 0.38 ice-9/boot-9.scm:2759:0:module-make-local-var! 0.00 7328.64 0.00 system/vm/assembler.scm:1258:0:intern-constant 0.00 1423.97 0.00 ice-9/boot-9.scm:220:5:map1 0.00 201.90 0.00 ice-9/threads.scm:388:4 [...] --- Sample count: 72 Total time: 27.531145665 seconds (14.056589389 seconds in GC) --8<---------------cut here---------------end--------------->8--- The assembler allocates a huge constant table in =E2=80=98intern-constant= =E2=80=99 (=E2=80=98generate-package-cache=E2=80=99 produces one literal list contain= ing one small vector per package, each of which contains strings and other constants). The attached file emulates what =E2=80=98generate-package-cache=E2=80=99 do= es. Each vector in the list looks like this: #(0 "xyzxyz-0" "1.0.0" (gnu packages xyz0) xyzxyz-0 ("out") #t #f "gnu/pa= ckages/xyz0" 0 1) The program terminates with roughly a 250=C2=A0MiB heap for 22K entries=E2= =80=94about 10=C2=A0MiB per vector. The heap profile on completion looks like this: --8<---------------cut here---------------start------------->8--- % type self avg obj size 57.5 bytevector 8,205,632 455868.4 23.9 vector 3,413,857 549.3 7.7 struct 1,093,840 40.8 5.3 pair 753,376 16.0 2.7 stringbuf 385,856 51.1 1.1 symbol 151,424 32.0 0.4 program 59,424 47.3 0.3 string 49,152 32.0 0.2 heap-number 23,520 32.0 0.2 variable 21,696 30.3 0.1 smob 21,056 32.0 0.1 vm-continuation 20,864 32.0 0.1 atomic-box 20,736 32.0 0.1 weak-table 19,584 31.8 0.1 unknown 19,040 32.0 0.0 pointer 2,000 16.0 0.0 hash-table 896 32.0 0.0 primitive 144 16.0 0.0 weak-vector 96 16.0 0.0 frame 96 32.0 0.0 primitive-generic 64 32.0 0.0 keyword 16 16.0 sampled heap: 13.60166 MiB (heap size: 233.16016 MiB) --8<---------------cut here---------------end--------------->8--- It=E2=80=99s likely that the =E2=80=98buf=E2=80=99 field of is what= =E2=80=99s taking space On that topic, see also: https://lists.gnu.org/archive/html/guile-devel/2023-01/msg00013.html Ludo=E2=80=99. --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=assembler-memory-consumption.scm Content-Description: the reproducer ;; https://issues.guix.gnu.org/63852 (use-modules (system base compile) (language tree-il) (srfi srfi-1) (srfi srfi-26) (ice-9 match) (ice-9 time) (statprof)) (define lst (unfold (cut > <> 22000) (lambda (i) ;; Mimic 'generate-package-cache'. (define name (string-append "xyzxyz-" (number->string i 8))) (vector i name (string-append "1.0." (number->string (modulo i 300))) (list 'gnu 'packages (string->symbol (string-append "xyz" (number->string i)))) (string->symbol name) '("out") #t #f (string-append "gnu/packages/xyz" (number->string (modulo i 300))) (modulo i 1000) 1)) 1+ 0)) (pk 'go! (gc-stats)) (let ((exp (match (or (getenv "VARIANT") "0") ("0" `',lst) ("1" `(list ,@lst)) ("2" (make-const #f lst)) ;avoid psyntax overhead ("3" (make-primcall #f 'list (map (lambda (vector) (make-const #f vector)) lst)))))) (compile exp #:to 'bytecode #:from (if (pair? exp) 'scheme 'tree-il) #:optimization-level 0 #:opts '(#:to-file? #t))) (pk (gc-stats)) --=-=-=--