From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id 4HMBMu4S0GJcBAEAbAwnHQ (envelope-from ) for ; Thu, 14 Jul 2022 14:58:22 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id 2DTpMe4S0GIsIAAA9RJhRA (envelope-from ) for ; Thu, 14 Jul 2022 14:58:22 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8909E8323 for ; Thu, 14 Jul 2022 14:58:22 +0200 (CEST) Received: from localhost ([::1]:59448 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oByQ9-0005KS-II for larch@yhetil.org; Thu, 14 Jul 2022 08:58:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39402) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oByPr-0005K3-7v for bug-guix@gnu.org; Thu, 14 Jul 2022 08:58:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:39204) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oByPq-0002rQ-H6 for bug-guix@gnu.org; Thu, 14 Jul 2022 08:58:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1oByPq-0000kA-DZ for bug-guix@gnu.org; Thu, 14 Jul 2022 08:58:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#39813: Running Guix in a Virtual Machine - says 1 GB RAM is enough, but it isn't for guix pull Resent-From: Csepp Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Thu, 14 Jul 2022 12:58:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 39813 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: Liliana Marie Prikler Cc: Julien Lepiller , Maxim Cournoyer , dannym@scratchpost.org, 39813@debbugs.gnu.org, 39813-done@debbugs.gnu.org, Csepp Received: via spool by 39813-done@debbugs.gnu.org id=D39813.16578034442794 (code D ref 39813); Thu, 14 Jul 2022 12:58:02 +0000 Received: (at 39813-done) by debbugs.gnu.org; 14 Jul 2022 12:57:24 +0000 Received: from localhost ([127.0.0.1]:36960 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oByPD-0000iz-Sa for submit@debbugs.gnu.org; Thu, 14 Jul 2022 08:57:24 -0400 Received: from mx0.riseup.net ([198.252.153.6]:47514) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oByPB-0000ii-HR; Thu, 14 Jul 2022 08:57:22 -0400 Received: from fews1.riseup.net (fews1-pn.riseup.net [10.0.1.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.riseup.net", Issuer "R3" (not verified)) by mx0.riseup.net (Postfix) with ESMTPS id 4LkDzH3J8hz9tCY; Thu, 14 Jul 2022 12:57:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak; t=1657803435; bh=zdNSQbJndSBF4uU9TqKasNA8c/jI0VY4Vb0xtI6vPmg=; h=References:From:To:Cc:Subject:Date:In-reply-to:From; b=FmHN0cn+T5+tEUDnZZlPOQlzKvaNYzZKaoA/N+4iWh9OGf8zrV1nDuD7WdzPO2h2u fPhC2VJSSgZSEtm5AeIwZfH+NyjRWSAujTp+L93lWaLMCrbsdL1l+Os+/qKUTkQLMT eHNEDfT2bqsLfGdLvr4NzCkP5y6HC0Wnbe6FRwZ4= X-Riseup-User-ID: C6A33FD8A39A1F86C0C109A35042991C821A95B25C4DAB923BF58C1AC72486F9 Received: from [127.0.0.1] (localhost [127.0.0.1]) by fews1.riseup.net (Postfix) with ESMTPSA id 4LkDzG2JY0z5w5Z; Thu, 14 Jul 2022 12:57:14 +0000 (UTC) References: <20200227210601.20011001@scratchpost.org> <877d4h159j.fsf@gmail.com> <87lesx7xsx.fsf@riseup.net> <9eb8491e8353fb2ddfeb9d508a058fca15b19d40.camel@gmail.com> From: Csepp Date: Thu, 14 Jul 2022 14:50:34 +0200 In-reply-to: <9eb8491e8353fb2ddfeb9d508a058fca15b19d40.camel@gmail.com> Message-ID: <8735f3c0ns.fsf@riseup.net> MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1657803502; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=FeXBjicrKOGh+ReoAna2hl2VQKT2VnityacasgdmT1Y=; b=DgHF7m2OlrI2K/6Wtnf8Id5XX9O+afD8r4mReOee4giZK9ME/HXCHR0MolwrA/ZxYLYlpC oAPK9Hv6GPFbZMPdNB8rXqZuT0Uf+DLGRaSgilgjOMbWKMNOaaLuT/kzyfcJb8sga/YlS/ PqTH0cLdQuR/bIKtltMFkp0I2KQqQQ9th+faCxtxm2LTsR8qMMBFJGMKVvAhhtmWc+AS/a c8qa6bD+HQW5LEeTMkGyNKZAnaa65GSPXayK0KR8SJxkpmj7we5NEI0hvbD/iPy5LQ56LM UxTGRmm+Kl56+/7ZOTMx//i/SWAlxllZMZkvwj12KE21kpvnOwhCjr6ZpuWbFw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1657803502; a=rsa-sha256; cv=none; b=Ir2A2Q5HEuIQ4Ref7jeo9eyYk/18amh2kk7T2dMGu+/5LbOxfCUAYXINDwy4JOrHASsNaX WQPFLtbxQ4UM7w/Kt/DpkKyp+s7NFtOHF3Hk3f0s1lpspdyCi66Mx3D2oQM9g/RES3WiIc 769SJdG7+Sm8bnqpFrCQQUeA22AUEo4fFEV9fZgoHQv/DChB8GWoYhD3KrM/kwaUjx5n2Z mEvWL3WQl0EdKPRJ8nshDC7x2atijhAGPntDi6e4bn46PBvacBymDQd4mj8zsMvHGaJtsM TawO+wSxS2aGnxre45EtDniHBi4RUoWsYaKC71pV/svVU1K3t46cZiiU3Zrxiw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=riseup.net header.s=squak header.b=FmHN0cn+; dmarc=fail reason="SPF not aligned (relaxed)" header.from=riseup.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 7.45 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=riseup.net header.s=squak header.b=FmHN0cn+; dmarc=fail reason="SPF not aligned (relaxed)" header.from=riseup.net (policy=none); spf=pass (aspmx1.migadu.com: domain of "bug-guix-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="bug-guix-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 8909E8323 X-Spam-Score: 7.45 X-Migadu-Scanner: scn0.migadu.com X-TUID: kkQtXMoxN8g3 Liliana Marie Prikler writes: > Am Mittwoch, dem 13.07.2022 um 19:21 +0200 schrieb Julien Lepiller: >> I've heard that theory before. From observation on my late armhf >> server >> (two cores): >> >> - it takes just below 2GB to build one of the derivations >> - It doesn't swap a single byte >> - whether with two or a single core, it takes roughly the same amount >> of memory >> - substitution is nice, it doesn't require lots of memory (and then - >> - >> cores is useless) >> >> I think it's because we load all the files in a batch before we build >> them. The biggest amount of memory required is not for running the >> compiler on a thread, but for loading files and keeping them in >> memory for the whole duration of the build. With more threads, we >> still don't load each file more than once (twice to build it), so >> there's no reason it should take more memory. >> >> Or maybe the process of loading and building is inherently single- >> threaded? I don't think so, but maybe? > Loading and building is implemented in build-aux/compile-all.scm, which > does use multiple parallel workers. However, since all compilation > appears to be done in the same Guile process, I don't think multi- > threading makes too big of an impact (it'll be the same garbage > collector regardless of ordering). > > Cheers Hmm, for some reason it finishes much faster on my i686 netbook with --cores=1. I'll try to look into it further. I enable swap on it but it doesn't use a lot of it. Maybe using an HDD matters too, since swapping is much more expensive on it. Disabling parallelism tends to help because it can halve the worst case memory usage. If foo.c and bar.c both require at most 1M during build, building them in parallel is 2M in the worst case, but only 1M when serialized. Oh and this is with channel-with-substitutes-available. So either that's broken or something still needs building. Anyways, I should take a looksie at build-aux/compile-all.scm, maybe I can decrease the memory usage.