unofficial mirror of guix-devel@gnu.org 
 help / color / mirror / code / Atom feed
* [PATCH] build: pull: Compile .scm files in one process.
@ 2015-11-05 17:10 Taylan Ulrich Bayırlı/Kammer
  2015-11-05 21:06 ` Ludovic Courtès
  2015-11-06 16:12 ` Andy Wingo
  0 siblings, 2 replies; 41+ messages in thread
From: Taylan Ulrich Bayırlı/Kammer @ 2015-11-05 17:10 UTC (permalink / raw)
  To: guix-devel

[-- Attachment #1: Type: text/plain, Size: 710 bytes --]

This reduces memory and CPU usage (unless you only have one CPU core),
and reduces the time needed to compile, during 'guix pull'.


Memory:

It used to use as many guile processes as you have cores, each consuming
about 100-150 MB memory.  Now it uses a single guile process whose
memory consumption grows linearly until it reaches almost 300 MB.


CPU:

It used to max out every CPU core, now just one. :-)


Time:

It used to take ~18 minutes on my machine, now less than 3.


The code is vastly simplified as well, as you see.  I'm not sure if
there's a good way to parallelize it though, and it's somewhat hacky to
keep all the modules in memory instead of spawning a clean guile to
compile each file...


[-- Attachment #2: 0001-build-pull-Compile-.scm-files-in-one-process.patch --]
[-- Type: text/x-diff, Size: 6717 bytes --]

From fd5d9bf77fd38fad354d66c31e57bc9bbc86481f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 <taylanbayirli@gmail.com>
Date: Thu, 5 Nov 2015 16:21:48 +0100
Subject: [PATCH] build: pull: Compile .scm files in one process.

* guix/build/pull.scm (call-with-process): Removed procedure.
(report-build-progress): Removed procedure.
(p-for-each): Removed procedure.
(build-guix): Don't create subprocesses to compile the .scm files.
---
 guix/build/pull.scm | 119 +++++++++++-----------------------------------------
 1 file changed, 25 insertions(+), 94 deletions(-)

diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index 281be23..0aa14eb 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -33,75 +33,10 @@
 ;;;
 ;;; Code:
 
-(define (call-with-process thunk)
-  "Run THUNK in a separate process that will return 0 if THUNK terminates
-normally, and 1 if an exception is raised."
-  (match (primitive-fork)
-    (0
-     (catch #t
-       (lambda ()
-         (thunk)
-         (primitive-exit 0))
-       (lambda (key . args)
-         (print-exception (current-error-port) #f key args)
-         (primitive-exit 1))))
-    (pid
-     #t)))
-
-(define* (report-build-progress total completed cont
-                                #:optional (log-port (current-error-port)))
-  "Report that COMPLETED out of TOTAL files have been completed, and call
-CONT."
-  (display #\cr log-port)
-  (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
-          (* 100. (/ completed total)) total)
-  (force-output log-port)
-  (cont))
-
-(define* (p-for-each proc lst
-                     #:optional (max-processes (current-processor-count))
-                     #:key (progress report-build-progress))
-  "Invoke PROC for each element of LST in a separate process, using up to
-MAX-PROCESSES processes in parallel.  Call PROGRESS at each step, passing it
-the continuation.  Raise an error if one of the processes exit with non-zero."
-  (define total
-    (length lst))
-
-  (define (wait-for-one-process)
-    (match (waitpid WAIT_ANY)
-      ((_ . status)
-       (unless (zero? (status:exit-val status))
-         (error "process failed" proc status)))))
-
-  (let loop ((lst       lst)
-             (running   0)
-             (completed 0))
-    (match lst
-      (()
-       (or (zero? running)
-           (let ((running   (- running 1))
-                 (completed (+ completed 1)))
-             (wait-for-one-process)
-             (progress total completed
-                       (lambda ()
-                         (loop lst running completed))))))
-      ((head . tail)
-       (if (< running max-processes)
-           (let ((running (+ 1 running)))
-             (call-with-process (cut proc head))
-             (progress total completed
-                       (lambda ()
-                         (loop tail running completed))))
-           (let ((running   (- running 1))
-                 (completed (+ completed 1)))
-             (wait-for-one-process)
-             (progress total completed
-                       (lambda ()
-                         (loop lst running completed)))))))))
-
 (define* (build-guix out source
                      #:key gcrypt
-                     (debug-port (%make-void-port "w")))
+                     (debug-port (%make-void-port "w"))
+                     (log-port (current-error-port)))
   "Build and install Guix in directory OUT using SOURCE, a directory
 containing the source code.  Write any debugging output to DEBUG-PORT."
   (setvbuf (current-output-port) _IOLBF)
@@ -130,33 +65,29 @@ containing the source code.  Write any debugging output to DEBUG-PORT."
     (set! %load-path (cons out %load-path))
     (set! %load-compiled-path (cons out %load-compiled-path))
 
-    ;; Compile the .scm files.  Do that in independent processes, à la
-    ;; 'make -j', to work around <http://bugs.gnu.org/15602> (FIXME).
-    ;; This ensures correctness, but is overly conservative and slow.
-    ;; The solution initially implemented (and described in the bug
-    ;; above) was slightly faster but consumed memory proportional to the
-    ;; number of modules, which quickly became unacceptable.
-    (p-for-each (lambda (file)
-                  (let ((go (string-append (string-drop-right file 4)
-                                           ".go")))
-                    (format debug-port "~%compiling '~a'...~%" file)
-                    (parameterize ((current-warning-port debug-port))
-                      (compile-file file
-                                    #:output-file go
-                                    #:opts
-                                    %auto-compilation-options))))
-
-                (filter (cut string-suffix? ".scm" <>)
-
-                        ;; Build guix/*.scm before gnu/*.scm to speed
-                        ;; things up.
-                        (sort (find-files out "\\.scm")
-                              (let ((guix (string-append out "/guix"))
-                                    (gnu  (string-append out "/gnu")))
-                                (lambda (a b)
-                                  (or (and (string-prefix? guix a)
-                                           (string-prefix? gnu b))
-                                      (string<? a b))))))))
+    ;; Compile the .scm files.  Also load every compiled file after writing it
+    ;; to work around <http://bugs.gnu.org/15602> (FIXME).
+    (let* ((files (filter (cut string-suffix? ".scm" <>)
+                          (find-files out "\\.scm")))
+           (total (length files)))
+      (let loop ((file (car files))
+                 (files (cdr files))
+                 (completed 0))
+        (display #\cr log-port)
+        (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
+                (* 100. (/ completed total)) total)
+        (force-output log-port)
+        (let ((go (string-append (string-drop-right file 4)
+                                 ".go")))
+          (format debug-port "~%compiling '~a'...~%" file)
+          (parameterize ((current-warning-port debug-port))
+            (compile-file file
+                          #:output-file go
+                          #:opts
+                          %auto-compilation-options)
+            (load-compiled go)))
+        (unless (null? files)
+          (loop (car files) (cdr files) (+ 1 completed))))))
 
   ;; Remove the "fake" (guix config).
   (delete-file (string-append out "/guix/config.scm"))
-- 
2.5.0


^ permalink raw reply related	[flat|nested] 41+ messages in thread
end of thread, other threads:[~2016-01-07 18:10 UTC | newest]

Thread overview: 41+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-05 17:10 [PATCH] build: pull: Compile .scm files in one process Taylan Ulrich Bayırlı/Kammer
2015-11-05 21:06 ` Ludovic Courtès
2015-11-06  8:56   ` Taylan Ulrich Bayırlı/Kammer
2015-11-06 20:50     ` Ludovic Courtès
2015-11-07  9:54       ` Taylan Ulrich Bayırlı/Kammer
2015-11-10 18:00         ` Ludovic Courtès
2015-11-11 12:12           ` Taylan Ulrich Bayırlı/Kammer
2015-11-11 13:26             ` Ludovic Courtès
2015-11-12 10:23               ` Taylan Ulrich Bayırlı/Kammer
2015-11-12 12:07                 ` Andy Wingo
2015-11-12 12:27                   ` Taylan Ulrich Bayırlı/Kammer
2015-11-12 12:36                     ` Andy Wingo
2015-11-12 13:52                       ` Taylan Ulrich Bayırlı/Kammer
2015-11-12 20:44                         ` Ludovic Courtès
2015-11-13 14:28                           ` Taylan Ulrich Bayırlı/Kammer
2015-11-14 14:54                             ` Ludovic Courtès
2015-11-26 22:20                             ` Ludovic Courtès
2015-11-27  8:53                               ` Taylan Ulrich Bayırlı/Kammer
2015-11-27 10:07                                 ` Ludovic Courtès
2015-11-27 15:16                                   ` Taylan Ulrich Bayırlı/Kammer
2015-11-30 13:04                                     ` Ludovic Courtès
2015-12-02  8:09                                       ` Taylan Ulrich Bayırlı/Kammer
2015-12-03 10:44                                         ` Ludovic Courtès
2015-12-21  9:49                                           ` Taylan Ulrich Bayırlı/Kammer
2016-01-05 17:56                                             ` Taylan Ulrich Bayırlı/Kammer
2016-01-05 21:39                                             ` Ludovic Courtès
2016-01-05 22:32                                               ` Taylan Ulrich Bayırlı/Kammer
2016-01-07 17:01                                                 ` Leo Famulari
2016-01-07 17:54                                                   ` Alex Sassmannshausen
2016-01-07 18:10                                                   ` Ludovic Courtès
2015-12-03 15:27                                     ` Mark H Weaver
2015-12-03 15:54                                       ` Taylan Ulrich Bayırlı/Kammer
2015-12-04 14:11                                       ` Ludovic Courtès
2015-11-06 16:12 ` Andy Wingo
2015-11-06 16:41   ` Taylan Ulrich Bayırlı/Kammer
2015-11-06 20:48     ` Ludovic Courtès
2015-11-09  7:41     ` Andy Wingo
2015-11-09  8:51       ` Taylan Ulrich Bayırlı/Kammer
2015-11-09  9:07         ` Andy Wingo
2015-11-09  9:50           ` Taylan Ulrich Bayırlı/Kammer
2015-11-09 10:49             ` Andy Wingo

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/guix.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).