From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id OAcNGDA1S2eLRwAAqHPOHw:P1 (envelope-from ) for ; Sat, 30 Nov 2024 15:54:24 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id OAcNGDA1S2eLRwAAqHPOHw (envelope-from ) for ; Sat, 30 Nov 2024 16:54:24 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gnu.org header.s=fencepost-gnu-org header.b=mBopCycc; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org" Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4A09B65E9C for ; Sat, 30 Nov 2024 16:54:19 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tHPmx-0003sY-E6; Sat, 30 Nov 2024 10:53:43 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tHPmv-0003sQ-Nr for guix-devel@gnu.org; Sat, 30 Nov 2024 10:53:41 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tHPmu-00058C-KK; Sat, 30 Nov 2024 10:53:40 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=xc9ycA6FaxXdPCQ7q+CK9cj+S5e05JQWrdk1a8rqW9A=; b=mBopCycc0oCrjaz78CHp wzY3l+fW6oQqEmo5krBB8K8gLW6YWnFloAKwrajZ2ovrJOzVGQo5S/wUMHY+32z4o3kfQnimIjoJX VzIgjr6nzq3Gzd/DrzrR7yFVQ7XDzRqD0DVO+lzWkTbAbArFWsNKVO6fxdc7rQDcJMjlb6xTsFV+7 TynyeBkPYUzj30zFoCYJkImubBKPInxdxZDj85TI7OPX714riHHAhvg5R992m5QbUhwXAd8mxC+MU hS21Xxc+pob5FjwUPqjkJ68yzeaubteY4oXxqCOkm+8Mlp0gTCbAoFcrZe8Wv7aozb9a7KKVLjbpW sCm2u6mEAJjWHA==; From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Christopher Baines Cc: Guix Devel Subject: Re: Persistent heap usage when computing derivations In-Reply-To: <87a5e8tc9i.fsf@cbaines.net> (Christopher Baines's message of "Sat, 09 Nov 2024 19:53:13 +0000") References: <87a5e8tc9i.fsf@cbaines.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: =?utf-8?Q?D=C3=A9cadi?= 10 Frimaire an 233 de la =?utf-8?Q?R=C3=A9volution=2C?= jour de la Pioche X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sat, 30 Nov 2024 16:53:35 +0100 Message-ID: <87jzckyayo.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -5.00 X-Spam-Score: -5.00 X-Migadu-Queue-Id: 4A09B65E9C X-Migadu-Scanner: mx12.migadu.com X-TUID: eoqbJB0Z1h+n Hi Christopher, Apologies for not answering earlier. Christopher Baines skribis: > One of the memory problems I'm having relates to the Guix inferior > processes that the data service uses when computing derivations. The > data serivce goes through the list of systems (x86_64-linux, > aarch64-linux, ...) and because the data cached for x86_64-linux > probably doesn't relate to aarch64-linux, there's some code that > attempts to clear the caches [1]. > > 1: https://git.savannah.gnu.org/cgit/guix/data-service.git/tree/guix-data= -service/jobs/load-new-guix-revision.scm#n1970 > > Unfortunately this code has to reach in to Guix internals to try and do > this, and it does reduce the heap usage significantly, but this doesn't > result in stable memory usage. Each system processed seems to add about > 250MiB of data to the Guile heap that isn't cleared out. To me that > sounds like a lot of memory, but there's also a lot of systems/targets, > so overall this leads to the inferior process using with around 6GiB of > data in the heap after processing all the systems/targets. This peak > memory usage really limits how much the machine can do. Did you consider running one inferior per system type, as a way to sidestep the problem? This is what =E2=80=98cuirass evaluate=E2=80=99 does and it manages to run = typically 4 of them in parallel (x86_64-linux, i686-linux, aarch64-linux, and powerpc64le-linux). Memory usage may be high but at least it=E2=80=99s not= a blocker. https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/src/cuirass/= scripts/evaluate.scm#n137 Ludo=E2=80=99.