From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark H Weaver Subject: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!) Date: Sat, 07 Oct 2017 00:06:51 -0400 Message-ID: <87fuavzjms.fsf_-_@netris.org> References: <877f16z9eo.fsf@gnu.org> <87infv54m3.fsf@gnu.org> <87efqgnn7x.fsf@elephly.net> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:34994) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e0gOS-0001rn-Tc for guix-devel@gnu.org; Sat, 07 Oct 2017 00:07:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e0gOP-0004hi-MB for guix-devel@gnu.org; Sat, 07 Oct 2017 00:07:16 -0400 In-Reply-To: <87efqgnn7x.fsf@elephly.net> (Ricardo Wurmus's message of "Fri, 06 Oct 2017 20:30:10 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Ricardo Wurmus Cc: guix-devel@gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ricardo Wurmus writes: > Hi Ludo, > >>> Here are some important items I can think of: > [=E2=80=A6] >>> =E2=80=A2 Guile 2.2 compiler terrible issue: >>> . > > One way to side-step this issue for the upcoming release is to use one > Guile process per file when running =E2=80=9Cguix pull=E2=80=9D. This wi= ll make it run > a lot slower, but that would be better than the current situation. I've attached a workaround that I've been using for the last 6 weeks on my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and otherwise it would not be able to successfully build the 'guix' package. Note that I never use 'guix pull', so I'm not sure off-hand whether this solves the problem there, but it certainly greatly reduces the memory needed to run 'make' and thus to build the 'guix' package. This patch modifies build-aux/compile-all.scm to work as follows: after loading all modules in the parent process, it then forks off a child and compiles 20 modules in parallel within that child while the parent waits. When the child is finished compiling those 20 modules, the child exits (thus freeing the memory leaked during compilation), and then the parent spawns a new child to compile the next 20 modules, and so on, until all the modules are compiled. We should probably consider applying this to master. Thoughts? Mark --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline; filename=0001-DRAFT-build-Compile-scheme-modules-in-batches.patch Content-Transfer-Encoding: quoted-printable Content-Description: [PATCH] DRAFT: build: Compile scheme modules in batches >From 05d5581ff71eb3b48773a5d46b612202de0492fb Mon Sep 17 00:00:00 2001 From: Mark H Weaver Date: Tue, 22 Aug 2017 03:26:10 -0400 Subject: [PATCH] DRAFT: build: Compile scheme modules in batches. --- build-aux/compile-all.scm | 48 ++++++++++++++++++++++++++++++++++++++++---= ---- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm index 147bb8019..96658e069 100644 --- a/build-aux/compile-all.scm +++ b/build-aux/compile-all.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2016 Taylan Ulrich Bay=C4=B1rl=C4=B1/Kammer ;;; Copyright =C2=A9 2016, 2017 Ludovic Court=C3=A8s +;;; Copyright =C2=A9 2017 Mark H Weaver ;;; ;;; This file is part of GNU Guix. ;;; @@ -19,6 +20,7 @@ =20 (use-modules (system base target) (system base message) + (srfi srfi-1) (ice-9 match) (ice-9 threads) (guix build utils)) @@ -118,13 +120,45 @@ ((_ . files) (let ((files (filter file-needs-compilation? files))) (for-each load-module-file files) - (let ((mutex (make-mutex))) - ;; Make sure compilation related modules are loaded before starting= to - ;; compile files in parallel. - (compile #f) - (par-for-each (lambda (file) - (compile-file* file mutex)) - files))))) + ;; Make sure compilation related modules are loaded before starting to + ;; compile files in parallel. + (compile #f) + ;; Flush all ports before entering the fork loop, to avoid flushing t= hem + ;; more than once within the child processes created below. + (flush-all-ports) + + ;; FIXME The following loop works around the apparent memory leak in = the + ;; compiler of guile-2.2.2, where compiling scheme modules requires + ;; increasing amounts of memory, up to nearly 2 gigabytes when all gu= ix + ;; sources are compiled within a single process. + ;; + ;; Ideally, we would simply apply 'par-for-each' to the entire set of + ;; files. For now, to work around the memory leak, we spawn subproce= sses + ;; to compile the files in batches of up to 20 files each. + (let fork-loop ((files files)) + (unless (null? files) + (call-with-values (lambda () + (split-at files (min 20 (length files)))) + (lambda (current-batch remaining-files) + ;; IMPORTANT: as noted in the Guile manual, it is unsafe to f= ork a + ;; process that has multiple threads running. Here we avoid = this + ;; difficulty by spawning threads only within the child proce= sses, + ;; which never call fork. + (match (primitive-fork) + (0 + ;; This is the child. It spawns threads but never forks. + (let ((mutex (make-mutex))) + (par-for-each (lambda (file) + (compile-file* file mutex)) + current-batch)) + (primitive-exit)) + (child-pid + ;; This is the parent. It forks but never spawns threads. + (match (waitpid child-pid) + ((_ . 0) + (fork-loop remaining-files)) + ((_ . status) + (primitive-exit (or (status:exit-val status) 1)))))))))= )))) =20 ;;; Local Variables: ;;; eval: (put 'with-target 'scheme-indent-function 1) --=20 2.14.1 --=-=-=--