From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!) Date: Mon, 09 Oct 2017 09:42:06 +0200 Message-ID: <87bmlgahtd.fsf@gnu.org> References: <877f16z9eo.fsf@gnu.org> <87infv54m3.fsf@gnu.org> <87efqgnn7x.fsf@elephly.net> <87fuavzjms.fsf_-_@netris.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:35828) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e1ShW-0002Ba-Ib for guix-devel@gnu.org; Mon, 09 Oct 2017 03:42:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e1ShV-0004T0-DC for guix-devel@gnu.org; Mon, 09 Oct 2017 03:42:10 -0400 In-Reply-To: <87fuavzjms.fsf_-_@netris.org> (Mark H. Weaver's message of "Sat, 07 Oct 2017 00:06:51 -0400") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Mark H Weaver Cc: guix-devel@gnu.org Hi, Mark H Weaver skribis: > Ricardo Wurmus writes: > >> Hi Ludo, >> >>>> Here are some important items I can think of: >> [=E2=80=A6] >>>> =E2=80=A2 Guile 2.2 compiler terrible issue: >>>> . >> >> One way to side-step this issue for the upcoming release is to use one >> Guile process per file when running =E2=80=9Cguix pull=E2=80=9D. This w= ill make it run >> a lot slower, but that would be better than the current situation. > > I've attached a workaround that I've been using for the last 6 weeks on > my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and > otherwise it would not be able to successfully build the 'guix' package. > > Note that I never use 'guix pull', so I'm not sure off-hand whether this > solves the problem there, but it certainly greatly reduces the memory > needed to run 'make' and thus to build the 'guix' package. > > This patch modifies build-aux/compile-all.scm to work as follows: after > loading all modules in the parent process, it then forks off a child and > compiles 20 modules in parallel within that child while the parent > waits. When the child is finished compiling those 20 modules, the child > exits (thus freeing the memory leaked during compilation), and then the > parent spawns a new child to compile the next 20 modules, and so on, > until all the modules are compiled. > > We should probably consider applying this to master. Thoughts? That sounds like a good idea. If it works, I=E2=80=99m all for it. We=E2=80=99ll need to do something similar in (guix scripts pull). I=E2=80=99ve been working on having =E2=80=98guix pull=E2=80=99 build modul= es as several derivations as part of . I=E2=80=99ll try to p= ost a prototype hopefully later today. Thanks, Ludo=E2=80=99.