From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id gA8VJBDe8WRXSgAA9RJhRA:P1 (envelope-from ) for ; Fri, 01 Sep 2023 14:50:24 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id gA8VJBDe8WRXSgAA9RJhRA (envelope-from ) for ; Fri, 01 Sep 2023 14:50:24 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id ED46F60601 for ; Fri, 1 Sep 2023 14:50:23 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=pwuX5n4d; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1693572624; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=kiJ9+wIc58mvfXQZTng7YzBPInHkqy0QH+XPMFT7f9o=; b=dU/CGh0b7fvgbCsqrbw+ixXjqsUR3zkHsHf0RmBDRK+RVm9X//WYAoZO5GoR3kFyWPT4MA RzOEM0lvUqlBow6iEoWZP1QJe7a6V7PVuUW7o62gjao/WOeChFcTK/ZpsxTgiSj3pV6CuB +t5B/xY24ymzoni03OpgmSS2HGYEz+QiomabCJ0Fhpklc+6fVIuBEi9adhWCyr93Qt9oKF lgOOKY+WDGHVsfbVLVsJLkhYXgvmXFgrKSisXY7kpUPh1h0V928wqOUNJAwP3jWzVi5pkM y56NQ/JQqB41AI62SEQZoj48S5kVUZYvnXMT8tHCvZDkUHgiuLQmPoTCeGcYsg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=pwuX5n4d; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=1; s=key1; d=yhetil.org; t=1693572624; a=rsa-sha256; cv=none; b=YU+I6YCPorPkv76InCIXWXuPKTOkb1PedqUeg8Lcz2pIMNrj+QM/zcXQgvrhpJPAJzCSzi gCKbu9D815oE/9BuTdm3iL1tIj2CNJ+zkdSTB3oj0H4AF7BZqKr6GemiOTqEK3CuuF44pg 4hEyy+8QAm3MYihoH2NtVQyg7AL1cNuON2fnnXOIXbvO0jrxvQBM9y++dkz0uHNzLvboL1 OHnxRGyNn+mBhbiekimpFQfyk2Ke4Z00AL/9B6IlEociqJq0FeQnZqSeQ8NdhNgWSMotcN BIMiu6PCYst0tO5A3gAsPJQyFfUVh8r2yAPnrH2jqNpbhsRTpqF1wIxAelsx3A== Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qc3b2-0006lo-Fv; Fri, 01 Sep 2023 08:49:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qc3b0-0006kg-5k for bug-guix@gnu.org; Fri, 01 Sep 2023 08:49:54 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qc3az-0005Dr-SL for bug-guix@gnu.org; Fri, 01 Sep 2023 08:49:53 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qc3b8-000232-L7 for bug-guix@gnu.org; Fri, 01 Sep 2023 08:50:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Fri, 01 Sep 2023 12:50:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Josselin Poiret , 65456@debbugs.gnu.org, Simon Tournier , Mathieu Othacehe , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines , Janneke Nieuwenhuizen Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.16935725427766 (code B ref 65456); Fri, 01 Sep 2023 12:50:02 +0000 Received: (at 65456) by debbugs.gnu.org; 1 Sep 2023 12:49:02 +0000 Received: from localhost ([127.0.0.1]:60033 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qc3a9-00020z-Vs for submit@debbugs.gnu.org; Fri, 01 Sep 2023 08:49:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:38356) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qc3a7-00020h-KZ for 65456@debbugs.gnu.org; Fri, 01 Sep 2023 08:49:00 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qc3Zr-0004wp-Mj; Fri, 01 Sep 2023 08:48:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=kiJ9+wIc58mvfXQZTng7YzBPInHkqy0QH+XPMFT7f9o=; b=pwuX5n4dXAg+0H6ti7N/ uM1GEx4AIEg4jn+oRTXIkLcqX1sUUVITSNAB7yeks4Z8eCI9AejV/jVMROKO0ct6biq+6epkfs1R6 hPwZ2NFHtJXGdHtm7PNcGs9OTP7SZuKF6AH7c1P+WXFr8CTm5eqR+VAcoZbSRiNJpumprFNLBCunc 2je0yoWD/Dnt0iI6r6JrKPlaOaTzlRbku/yNB9UU+mx6NbiBRiQEbCclVVTK7xy5K4pY/A0yWeT92 2vJxtw38jKqLcUTAC4o2EsQ1moFPX09HpRBXh3XoFvBr3NgE7dr9kAxGpvZZwI0B5hhwbWV7PLFTc X6R9udkelxfa6A==; From: Janneke Nieuwenhuizen Organization: AvatarAcademy.nl References: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> <87pm3e3fuu.fsf_-_@gnu.org> <871qfu8erx.fsf_-_@gnu.org> <87h6oq6qpg.fsf_-_@gnu.org> <87zg2gy013.fsf_-_@gnu.org> X-Url: http://AvatarAcademy.nl Date: Fri, 01 Sep 2023 14:48:38 +0200 In-Reply-To: <87zg2gy013.fsf_-_@gnu.org> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Thu, 24 Aug 2023 16:42:32 +0200") Message-ID: <87h6oeys7t.fsf@verum.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -2.43 X-Spam-Score: -2.43 X-Migadu-Scanner: mx1.migadu.com X-Migadu-Queue-Id: ED46F60601 X-TUID: okUXr/kzBMug Ludovic Court=C3=A8s writes: Hello! > Janneke Nieuwenhuizen skribis: > >>>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001 >> Message-ID: >> From: Janneke Nieuwenhuizen >> Date: Thu, 22 Jun 2023 08:30:25 +0200 >> Subject: [PATCH v4] self: Build directories in chunks of max 25 files at= a >> time. >> >> Similar to split build of make-go in Makefile.am, this breaks-up building >> directories into chunks of max 25 files. Also force garbage collection. > > The big difference with =E2=80=98make-go=E2=80=99 is that =E2=80=98make-g= o=E2=80=99 spawns a new process > for each chunk of files: each process starts with an empty heap, which > is not the case here as we reuse the same process. Right. > However, (guix self) is already splitting gnu/packages/*.scm in two > pieces: =E2=80=98guix-packages-base=E2=80=99 and =E2=80=98guix-packages= =E2=80=99. The former is the > closure of (gnu packages base), and the latter contains the remaining > files. Unfortunately this is uneven: Okay... > $ readlink -f $(type -P guix) > /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command > $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|gre= p packages-base > /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base > /gnu/store/05pjlcfcfa0k9y833nnxxxjcn5mqr8zj-guix-packages-base-source > /gnu/store/gnxjbyfwfmb216krz2x0cf1z5k1lla9x-guix-packages-base-modules > $ find /gnu/store/ivprgy9b2lv8wmkm10wkypf7k24cdifb-guix-packages-base -t= ype f |wc -l > 361 > $ guix gc -R /gnu/store/12p5axbr4gjrghlrqa4ikmhsxwq2wgw3-guix-command|gre= p packages$ > /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9f1avx-guix-locale-guix-packages > ludo@ribbon ~/src/guix [env]$ find /gnu/store/8cda50hsayydrlw0qrhcy8q4dr9= f1avx-guix-locale-guix-packages | wc -l > 64 > $ guix describe > Generation 271 Aug 20 2023 23:48:59 (current) > guix a0f5885 > repository URL: https://git.savannah.gnu.org/git/guix.git > branch: master > commit: a0f5885fefd93a3859b6e4b82b18a6db9faeee05 > > Maxime Devos looked into this a while back: > > https://issues.guix.gnu.org/54539 Oh my.... >> * guix/self.scm (compiled-modules)[process-directory]: Split building of >> directories into chunks of max 25 files. >> + (for-each >> + (lambda (chunck) > > s/chunck/chunk/ Oops, fixed. > Can you confirm that this reduces memory usage observably? One way to > check that would be to print (gc-stats) from =E2=80=98process-directory= =E2=80=99, with > and without the change. Could you give it a try? What a good and seemingly simple question. After a week of instrumentation and testing, my answer can only be: I tried, and maybe. (see below). > Intuitively, I don=E2=80=99t see why it would eat less memory; maybe peak= memory > usage is lower because we do less at once? Okay... > Also, I think we should remove the explicit (gc) call: it should not be > necessary, and if we depend on that, something=E2=80=99s wrong. > Anyhow, thanks for tackling this issue! Hehe. You've probably seen Josselin's recent GraphML backend effort that might really help to address this? I'm afraid this patch can maybe only postpone what really needs to be done... There is gc-stats output from a successful `guix pull' or `make as-derivation' on Guix/Hurd, that I can show you, and I've tried more than 20 times; it always fails (OOM, hang, spontaneous reset, ...). Below is a typical output of gc-stats on the Hurd for building self.scm, when heap-size peaks (using the the max 25 files patch): --8<---------------cut here---------------start------------->8--- ((gc-time-taken . 1530) (heap-size . 2,625,474,560) (heap-free-size . 1127989248) (heap-total-allocated . 1337029496) (heap-allocated-since-gc . 28728) (protected-objects . 28) (gc-times . 324)) --8<---------------cut here---------------end--------------->8--- notice that it's *much* bigger (more than twice) than my findings on linux-64 below. I have no idea why this is of what it might mean... So I turned to Guix GNU/Linux to get some gc-stat measurements. What you see below is the maximum head-size at any point (I also have heap-total-allocated but I think that's irrelevant? and initially didn't use a script that measured the time). --8<---------------cut here---------------start------------->8--- * guix/self.scm: Vanilla, not chunked; print gc-stats. ((gc-time-taken . 27319485051) (heap-size . 1,360,330,752) (heap-free-size . 285,696,000) (heap-total-allocated . 74,067,590,944) (heap-allocated-since-gc . 186,250,144) (protected-objects . 28) (gc-times . 464)) real 24m36.643s * guix/self.scm: Split building of directories into 26 chunks; print gc-sta= ts. (heap-size . 1,131,298,816) * guix/self.scm: Split building of directories into 26 chunks; no gc; print= gc-stats. (heap-size . 1,121,116,160) * guix/self.scm: Chunks of 25 files; run gc; print gc-stats. (heap-size . 1,066,725,376) * guix/self.scm: Chunks of 50 files; no gc; print gc-stats. (heap-size . 1,299,230,720) real 26m40.708s * guix/self.scm: Chunks of 25 files; no gc; print gc-stats. (heap-size . 1,024,045,056) ; 1st run real 28m4.451s * guix/self.scm: Chunks of 10 files; no gc; print gc-stats. (heap-size . 1,077,895,168) real 30m14.049s --8<---------------cut here---------------end--------------->8--- ...strangely enough, if we assume that these statistics translate to the Hurd, using chunks of max 25 files seems to be a sort of sweet spot? 25% less peak memory (~300MB), "only" 12% (3"45') slower... though not great for GNU/Linux users... I have produced a handful of successful `guix pull's (from a local checked-out worktree) using the 26-way split and chunks of max-25 files patches, but sadly also many more attempts failed. Initially, when creating this patch series, I was convinced this fixed building on the Hurd, but I'm much less enthusiastic now. So I still have a slight preference for using the latest max-25-files patch, but I'm sorry to say that I cannot back it up with tangible data. All in all a rather discouraging week with much effort spent for little gain. Hopefully Josselin can do some of his magic here :) Greetings, Janneke --=20 Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar=C2=AE https://AvatarAcade= my.com