From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
To: guix-devel@gnu.org
Subject: [PATCH] build: pull: Compile .scm files in one process.
Date: Thu, 05 Nov 2015 18:10:57 +0100 [thread overview]
Message-ID: <87si4kxtge.fsf@T420.taylan> (raw)
[-- Attachment #1: Type: text/plain, Size: 710 bytes --]
This reduces memory and CPU usage (unless you only have one CPU core),
and reduces the time needed to compile, during 'guix pull'.
Memory:
It used to use as many guile processes as you have cores, each consuming
about 100-150 MB memory. Now it uses a single guile process whose
memory consumption grows linearly until it reaches almost 300 MB.
CPU:
It used to max out every CPU core, now just one. :-)
Time:
It used to take ~18 minutes on my machine, now less than 3.
The code is vastly simplified as well, as you see. I'm not sure if
there's a good way to parallelize it though, and it's somewhat hacky to
keep all the modules in memory instead of spawning a clean guile to
compile each file...
[-- Attachment #2: 0001-build-pull-Compile-.scm-files-in-one-process.patch --]
[-- Type: text/x-diff, Size: 6717 bytes --]
From fd5d9bf77fd38fad354d66c31e57bc9bbc86481f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
<taylanbayirli@gmail.com>
Date: Thu, 5 Nov 2015 16:21:48 +0100
Subject: [PATCH] build: pull: Compile .scm files in one process.
* guix/build/pull.scm (call-with-process): Removed procedure.
(report-build-progress): Removed procedure.
(p-for-each): Removed procedure.
(build-guix): Don't create subprocesses to compile the .scm files.
---
guix/build/pull.scm | 119 +++++++++++-----------------------------------------
1 file changed, 25 insertions(+), 94 deletions(-)
diff --git a/guix/build/pull.scm b/guix/build/pull.scm
index 281be23..0aa14eb 100644
--- a/guix/build/pull.scm
+++ b/guix/build/pull.scm
@@ -33,75 +33,10 @@
;;;
;;; Code:
-(define (call-with-process thunk)
- "Run THUNK in a separate process that will return 0 if THUNK terminates
-normally, and 1 if an exception is raised."
- (match (primitive-fork)
- (0
- (catch #t
- (lambda ()
- (thunk)
- (primitive-exit 0))
- (lambda (key . args)
- (print-exception (current-error-port) #f key args)
- (primitive-exit 1))))
- (pid
- #t)))
-
-(define* (report-build-progress total completed cont
- #:optional (log-port (current-error-port)))
- "Report that COMPLETED out of TOTAL files have been completed, and call
-CONT."
- (display #\cr log-port)
- (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
- (* 100. (/ completed total)) total)
- (force-output log-port)
- (cont))
-
-(define* (p-for-each proc lst
- #:optional (max-processes (current-processor-count))
- #:key (progress report-build-progress))
- "Invoke PROC for each element of LST in a separate process, using up to
-MAX-PROCESSES processes in parallel. Call PROGRESS at each step, passing it
-the continuation. Raise an error if one of the processes exit with non-zero."
- (define total
- (length lst))
-
- (define (wait-for-one-process)
- (match (waitpid WAIT_ANY)
- ((_ . status)
- (unless (zero? (status:exit-val status))
- (error "process failed" proc status)))))
-
- (let loop ((lst lst)
- (running 0)
- (completed 0))
- (match lst
- (()
- (or (zero? running)
- (let ((running (- running 1))
- (completed (+ completed 1)))
- (wait-for-one-process)
- (progress total completed
- (lambda ()
- (loop lst running completed))))))
- ((head . tail)
- (if (< running max-processes)
- (let ((running (+ 1 running)))
- (call-with-process (cut proc head))
- (progress total completed
- (lambda ()
- (loop tail running completed))))
- (let ((running (- running 1))
- (completed (+ completed 1)))
- (wait-for-one-process)
- (progress total completed
- (lambda ()
- (loop lst running completed)))))))))
-
(define* (build-guix out source
#:key gcrypt
- (debug-port (%make-void-port "w")))
+ (debug-port (%make-void-port "w"))
+ (log-port (current-error-port)))
"Build and install Guix in directory OUT using SOURCE, a directory
containing the source code. Write any debugging output to DEBUG-PORT."
(setvbuf (current-output-port) _IOLBF)
@@ -130,33 +65,29 @@ containing the source code. Write any debugging output to DEBUG-PORT."
(set! %load-path (cons out %load-path))
(set! %load-compiled-path (cons out %load-compiled-path))
- ;; Compile the .scm files. Do that in independent processes, à la
- ;; 'make -j', to work around <http://bugs.gnu.org/15602> (FIXME).
- ;; This ensures correctness, but is overly conservative and slow.
- ;; The solution initially implemented (and described in the bug
- ;; above) was slightly faster but consumed memory proportional to the
- ;; number of modules, which quickly became unacceptable.
- (p-for-each (lambda (file)
- (let ((go (string-append (string-drop-right file 4)
- ".go")))
- (format debug-port "~%compiling '~a'...~%" file)
- (parameterize ((current-warning-port debug-port))
- (compile-file file
- #:output-file go
- #:opts
- %auto-compilation-options))))
-
- (filter (cut string-suffix? ".scm" <>)
-
- ;; Build guix/*.scm before gnu/*.scm to speed
- ;; things up.
- (sort (find-files out "\\.scm")
- (let ((guix (string-append out "/guix"))
- (gnu (string-append out "/gnu")))
- (lambda (a b)
- (or (and (string-prefix? guix a)
- (string-prefix? gnu b))
- (string<? a b))))))))
+ ;; Compile the .scm files. Also load every compiled file after writing it
+ ;; to work around <http://bugs.gnu.org/15602> (FIXME).
+ (let* ((files (filter (cut string-suffix? ".scm" <>)
+ (find-files out "\\.scm")))
+ (total (length files)))
+ (let loop ((file (car files))
+ (files (cdr files))
+ (completed 0))
+ (display #\cr log-port)
+ (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n
+ (* 100. (/ completed total)) total)
+ (force-output log-port)
+ (let ((go (string-append (string-drop-right file 4)
+ ".go")))
+ (format debug-port "~%compiling '~a'...~%" file)
+ (parameterize ((current-warning-port debug-port))
+ (compile-file file
+ #:output-file go
+ #:opts
+ %auto-compilation-options)
+ (load-compiled go)))
+ (unless (null? files)
+ (loop (car files) (cdr files) (+ 1 completed))))))
;; Remove the "fake" (guix config).
(delete-file (string-append out "/guix/config.scm"))
--
2.5.0
next reply other threads:[~2015-11-05 17:11 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-11-05 17:10 Taylan Ulrich Bayırlı/Kammer [this message]
2015-11-05 21:06 ` [PATCH] build: pull: Compile .scm files in one process Ludovic Courtès
2015-11-06 8:56 ` Taylan Ulrich Bayırlı/Kammer
2015-11-06 20:50 ` Ludovic Courtès
2015-11-07 9:54 ` Taylan Ulrich Bayırlı/Kammer
2015-11-10 18:00 ` Ludovic Courtès
2015-11-11 12:12 ` Taylan Ulrich Bayırlı/Kammer
2015-11-11 13:26 ` Ludovic Courtès
2015-11-12 10:23 ` Taylan Ulrich Bayırlı/Kammer
2015-11-12 12:07 ` Andy Wingo
2015-11-12 12:27 ` Taylan Ulrich Bayırlı/Kammer
2015-11-12 12:36 ` Andy Wingo
2015-11-12 13:52 ` Taylan Ulrich Bayırlı/Kammer
2015-11-12 20:44 ` Ludovic Courtès
2015-11-13 14:28 ` Taylan Ulrich Bayırlı/Kammer
2015-11-14 14:54 ` Ludovic Courtès
2015-11-26 22:20 ` Ludovic Courtès
2015-11-27 8:53 ` Taylan Ulrich Bayırlı/Kammer
2015-11-27 10:07 ` Ludovic Courtès
2015-11-27 15:16 ` Taylan Ulrich Bayırlı/Kammer
2015-11-30 13:04 ` Ludovic Courtès
2015-12-02 8:09 ` Taylan Ulrich Bayırlı/Kammer
2015-12-03 10:44 ` Ludovic Courtès
2015-12-21 9:49 ` Taylan Ulrich Bayırlı/Kammer
2016-01-05 17:56 ` Taylan Ulrich Bayırlı/Kammer
2016-01-05 21:39 ` Ludovic Courtès
2016-01-05 22:32 ` Taylan Ulrich Bayırlı/Kammer
2016-01-07 17:01 ` Leo Famulari
2016-01-07 17:54 ` Alex Sassmannshausen
2016-01-07 18:10 ` Ludovic Courtès
2015-12-03 15:27 ` Mark H Weaver
2015-12-03 15:54 ` Taylan Ulrich Bayırlı/Kammer
2015-12-04 14:11 ` Ludovic Courtès
2015-11-06 16:12 ` Andy Wingo
2015-11-06 16:41 ` Taylan Ulrich Bayırlı/Kammer
2015-11-06 20:48 ` Ludovic Courtès
2015-11-09 7:41 ` Andy Wingo
2015-11-09 8:51 ` Taylan Ulrich Bayırlı/Kammer
2015-11-09 9:07 ` Andy Wingo
2015-11-09 9:50 ` Taylan Ulrich Bayırlı/Kammer
2015-11-09 10:49 ` Andy Wingo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://guix.gnu.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87si4kxtge.fsf@T420.taylan \
--to=taylanbayirli@gmail.com \
--cc=guix-devel@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).