From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id gEK9EQDHBWULgwEAG6o9tA:P1 (envelope-from ) for ; Sat, 16 Sep 2023 17:17:20 +0200 Received: from aspmx1.migadu.com ([2001:41d0:306:2d92::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id gEK9EQDHBWULgwEAG6o9tA (envelope-from ) for ; Sat, 16 Sep 2023 17:17:20 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A5CD040C93 for ; Sat, 16 Sep 2023 17:17:19 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=EBja5e0T; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org ARC-Seal: i=1; s=key1; d=yhetil.org; t=1694877440; a=rsa-sha256; cv=none; b=uukkkCOfcclbR0A/n0AwUQSxhZa0TVJZ0y9PXq1thMPTVMTA6W2/Oc8XY8/jfc/qS6ExYP L/cFWJN42vBRXhZCuiSB09s6iARyXEVVGGCQvwHHSinuf4Q024t8c+wsJT06ix6KA46C4X x8Hd20dy6ZA3e7yqGCekPxTyce+LE+0cMsGQPweP7KKev51+b1iGnPtm29SmKELxF32mcP u0PUwIMSdTWVrTKees/3BLkwn2CQRxXHEuGtEc6yi+mF3MoP6vq7hWSOEyX5Pda6R91leh UCXuJMuahRVJutJSwKyptL0N48tM1V1OGJUtV3umRMpCFGipxPA98MyjQHFgZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1694877440; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=66Fm+6ivg6wgunEtF2D7GBV5QewktvOHAv2DVmSKan4=; b=FmUVU8yCgaK2q/tdDE8aHunk845nzuFPyVMXARsjJAo0c+wSRrP1AdFQPJqwPEgUpeuopM Vf2nxtlOXRCQSUxpTb0dgJAmtNeAu0JhiCYjQtuPTgav+SKNl2GSUo8eyuATGbf30FlsfY CAdCUE1x6PTd8pv4SL3sU3ptjmihYF7UWzI6w4t0lWBzcG6FALkBG6s3n1l+kqVqJ4fO+j Sy2InL0iG55iijadP4mmNCPFDwU16GUugmsVQ6ehkMD8GOcykrIN1aVgJSKwic7h50ZVy0 OJkPRNhcdT7ywNNoKlP/FlGxyVl+2QuQlDMd/exxm5ZuGkBjRXuVLwykuEQt7g== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gnu.org header.s=fencepost-gnu-org header.b=EBja5e0T; spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gnu.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qhX2X-00078h-Jt; Sat, 16 Sep 2023 11:16:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qhX2V-00078F-4e for bug-guix@gnu.org; Sat, 16 Sep 2023 11:16:55 -0400 Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qhX2U-0005Kd-Se for bug-guix@gnu.org; Sat, 16 Sep 2023 11:16:54 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qhX2b-0000v8-SF for bug-guix@gnu.org; Sat, 16 Sep 2023 11:17:01 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#65456: [PATCH 0/2] Split guix build into more steps for 32bit hosts. Resent-From: Janneke Nieuwenhuizen Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 16 Sep 2023 15:17:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 65456 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Josselin Poiret , 65456@debbugs.gnu.org, Simon Tournier , Mathieu Othacehe , Tobias Geerinckx-Rice , Ricardo Wurmus , Christopher Baines Received: via spool by 65456-submit@debbugs.gnu.org id=B65456.16948774113518 (code B ref 65456); Sat, 16 Sep 2023 15:17:01 +0000 Received: (at 65456) by debbugs.gnu.org; 16 Sep 2023 15:16:51 +0000 Received: from localhost ([127.0.0.1]:48278 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qhX2R-0000uf-9s for submit@debbugs.gnu.org; Sat, 16 Sep 2023 11:16:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50042) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qhX2O-0000uP-Ea for 65456@debbugs.gnu.org; Sat, 16 Sep 2023 11:16:50 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qhX29-0005AV-9w; Sat, 16 Sep 2023 11:16:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=66Fm+6ivg6wgunEtF2D7GBV5QewktvOHAv2DVmSKan4=; b=EBja5e0TtBzWdF2e2Bfp K0iGcr2AB9q1Wyp+wC4h+G3iu39eI60zk/sT8OQrXiUP8N+GzXSHhwMaDbDsrzOY9qW13w+bcTVjV ddqBpe6sNusNLpEI912OBagdsddHug63apaki301QfJsi78mHaJarEOhz5bUQxK/Aesefl/HWF10F skW4UKYdXymrcgYR5BG2SHuZLiWowiDb5LTopEv8B0+k9ePxy30q8oGMJtbNxcPZTiBRipE8k8WSH vAagyFt+Xv9KyIWiAebZSCk6LtgC7M/LkIyyrJRHPm7rgKNn7RkXpMG1T1bx2sevKaWRFdLA0PfRS bOJXg4ncLPs94Q==; From: Janneke Nieuwenhuizen Organization: AvatarAcademy.nl References: <887d53ad5bee76b33d765e744c50e94e063b1ab8.1692723764.git.janneke@gnu.org> <87pm3e3fuu.fsf_-_@gnu.org> <871qfu8erx.fsf_-_@gnu.org> <87h6oq6qpg.fsf_-_@gnu.org> <87zg2gy013.fsf_-_@gnu.org> <87h6oeys7t.fsf@verum.com> X-Url: http://AvatarAcademy.nl Date: Sat, 16 Sep 2023 17:16:28 +0200 In-Reply-To: <87h6oeys7t.fsf@verum.com> (Janneke Nieuwenhuizen's message of "Fri, 01 Sep 2023 14:48:38 +0200") Message-ID: <87led6ywqb.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: bug-guix-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -5.41 X-Spam-Score: -5.41 X-Migadu-Queue-Id: A5CD040C93 X-Migadu-Scanner: mx2.migadu.com X-TUID: iVnQEBQNmnWZ Janneke Nieuwenhuizen writes: Hi! > Ludovic Court=C3=A8s writes: > > Hello! > >> Janneke Nieuwenhuizen skribis: >> >>>>>From ad94f06620e53fcc1495a2e2479dfc627177047c Mon Sep 17 00:00:00 2001 [..] >> Anyhow, thanks for tackling this issue! > > Hehe. You've probably seen Josselin's recent GraphML backend effort > that might really help to address this? I'm afraid this patch can maybe > only postpone what really needs to be done... > > There is gc-stats output from a successful `guix pull' or `make > as-derivation' on Guix/Hurd, that I can show you, and I've tried more > than 20 times; it always fails (OOM, hang, spontaneous reset, ...). > > Below is a typical output of gc-stats on the Hurd for building self.scm, > when heap-size peaks (using the the max 25 files patch): > > ((gc-time-taken . 1530) > (heap-size . 2,625,474,560) > (heap-free-size . 1127989248) > (heap-total-allocated . 1337029496) > (heap-allocated-since-gc . 28728) > (protected-objects . 28) > (gc-times . 324)) > > > notice that it's *much* bigger (more than twice) than my findings on > linux-64 below. I have no idea why this is of what it might mean... > > So I turned to Guix GNU/Linux to get some gc-stat measurements. What > you see below is the maximum head-size at any point (I also have > heap-total-allocated but I think that's irrelevant? and initially didn't > use a script that measured the time). > > * guix/self.scm: Vanilla, not chunked; print gc-stats. > ((gc-time-taken . 27319485051) > (heap-size . 1,360,330,752) > (heap-free-size . 285,696,000) > (heap-total-allocated . 74,067,590,944) > (heap-allocated-since-gc . 186,250,144) > (protected-objects . 28) > (gc-times . 464)) > real 24m36.643s > > * guix/self.scm: Split building of directories into 26 chunks; print gc-s= tats. > (heap-size . 1,131,298,816) > > * guix/self.scm: Split building of directories into 26 chunks; no gc; pri= nt gc-stats. > (heap-size . 1,121,116,160) > > * guix/self.scm: Chunks of 25 files; run gc; print gc-stats. > (heap-size . 1,066,725,376) > > * guix/self.scm: Chunks of 50 files; no gc; print gc-stats. > (heap-size . 1,299,230,720) > real 26m40.708s > > * guix/self.scm: Chunks of 25 files; no gc; print gc-stats. > (heap-size . 1,024,045,056) ; 1st run > real 28m4.451s > > * guix/self.scm: Chunks of 10 files; no gc; print gc-stats. > (heap-size . 1,077,895,168) > real 30m14.049s > > ...strangely enough, if we assume that these statistics translate to the > Hurd, using chunks of max 25 files seems to be a sort of sweet spot? > 25% less peak memory (~300MB), "only" 12% (3"45') slower... though not > great for GNU/Linux users... > > I have produced a handful of successful `guix pull's (from a local > checked-out worktree) using the 26-way split and chunks of max-25 files > patches, but sadly also many more attempts failed. Initially, when > creating this patch series, I was convinced this fixed building on the > Hurd, but I'm much less enthusiastic now. > > So I still have a slight preference for using the latest max-25-files > patch, but I'm sorry to say that I cannot back it up with tangible data. > All in all a rather discouraging week with much effort spent for little > gain. Hopefully Josselin can do some of his magic here :) Anyway, I've finally --8<---------------cut here---------------start------------->8--- --8<---------------cut here---------------end--------------->8--- managed to get all remaining necessary dependencies built to run `guix system reconfigure' on the Hurd*. The only patch that we need that's not on master yet is this one. I'm still rooting for Josselin's graphml cycle analysis and/or lechner Felix's bespoke make rewrite will render this hack obsolete real soon now... If there are no objections I'll go forward and install this patch by the end of the weekend. Greetings, Janneke *) using the hurd-team branch, grub install still fails, apparently because "someone" imagines /dev/wd0 to be /dev/hd0, but the new system derivation is being enabled. --=20 Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar=C2=AE https://AvatarAcade= my.com