From mboxrd@z Thu Jan 1 00:00:00 1970 From: taylanbayirli@gmail.com (Taylan Ulrich =?utf-8?Q?Bay=C4=B1rl=C4=B1?= =?utf-8?Q?=2FKammer?=) Subject: Re: [PATCH] build: pull: Compile .scm files in one process. Date: Mon, 21 Dec 2015 10:49:59 +0100 Message-ID: <8737uw5dx4.fsf@T420.taylan> References: <87si4kxtge.fsf@T420.taylan> <87h9kzy09b.fsf@T420.taylan> <87bnb6c0nh.fsf@gnu.org> <874mgyxhgy.fsf@T420.taylan> <877flpohu6.fsf@gnu.org> <87mvuku444.fsf@T420.taylan> <87pozgfyzt.fsf@gnu.org> <87io57tt2s.fsf@T420.taylan> <876117mnef.fsf@igalia.com> <87egfvtnbw.fsf@T420.taylan> <87y4e3l7hm.fsf@igalia.com> <87a8qjtje8.fsf@T420.taylan> <876117t0ax.fsf@gnu.org> <877flmrn2m.fsf@T420.taylan> <87a8q0ies5.fsf@gnu.org> <87fuzrlt6f.fsf@T420.taylan> <87bnafbvrs.fsf@gnu.org> <87bnaflbg2.fsf@T420.taylan> <87h9k3ab9k.fsf@gnu.org> <87zixt9sr6.fsf@T420.taylan> <877fkvvmk9.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:41243) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aAx6S-00011J-Ta for guix-devel@gnu.org; Mon, 21 Dec 2015 04:50:06 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aAx6R-0004zP-8e for guix-devel@gnu.org; Mon, 21 Dec 2015 04:50:04 -0500 In-Reply-To: <877fkvvmk9.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Thu, 03 Dec 2015 12:44:22 +0200") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org --=-=-= Content-Type: text/plain >>>> After some tinkering around I realized that the problem is that our >>>> workaround of loading files explicitly causes the package record >>>> type to be redefined after some packages have already been defined. >>>> More generally, we force the top-level of many files to be >>>> re-executed after they've already been executed as a result of a >>>> module import... Returning to this issue after a long pause, I suddenly had a new idea, which is to use something akin to 'use-modules' instead of 'load' on the files. (I need a procedure and not a macro, and don't actually need to 'use' the modules, so I use 'resolve-interface' in the patch below. Don't know if there's a better suited procedure for this purpose.) The tricky part is turning file names into module names, but so far all seems unproblematic, so a straightforward transformation from the string "$out/foo/bar.scm" to the list (foo bar) does the job. (Possible issues would be when a file name doesn't match a module name, or a file doesn't contain a module at all.) This entails the least amount of duplicated effort from all the variants we had so far, and finishes in below 2 minutes on my machine. (I think the only remaining duplication is the compilation of the modules during 'resolve-interface' and their compilation via 'compile-file', and the former cannot be done in parallel, so we could again cut the time to below half of what it is now, if we solved that.) Tell me if you see any problems with this variant. --=-=-= Content-Type: text/x-diff; charset=utf-8 Content-Disposition: inline; filename=0001-build-pull-Compile-.scm-files-in-one-process.patch Content-Transfer-Encoding: quoted-printable >From 77ac65593a94673872c477bbbd18eb2465c76030 Mon Sep 17 00:00:00 2001 From: =3D?UTF-8?q?Taylan=3D20Ulrich=3D20Bay=3DC4=3DB1rl=3DC4=3DB1/Kammer?=3D Date: Fri, 27 Nov 2015 09:27:55 +0100 Subject: [PATCH] build: pull: Compile .scm files in one process. * guix/build/pull.scm (call-with-process, report-build-progress) (p-for-each): Remove. (build-guix): Load and compile files in one process. --- guix/build/pull.scm | 149 +++++++++++++++++++-----------------------------= ---- 1 file changed, 55 insertions(+), 94 deletions(-) diff --git a/guix/build/pull.scm b/guix/build/pull.scm index 281be23..4ddb12a 100644 --- a/guix/build/pull.scm +++ b/guix/build/pull.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2013, 2014 Ludovic Court=C3=A8s +;;; Copyright =C2=A9 2015 Taylan Ulrich Bay=C4=B1rl=C4=B1/Kammer ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,6 +23,7 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 format) + #:use-module (ice-9 threads) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) @@ -33,75 +35,10 @@ ;;; ;;; Code: =20 -(define (call-with-process thunk) - "Run THUNK in a separate process that will return 0 if THUNK terminates -normally, and 1 if an exception is raised." - (match (primitive-fork) - (0 - (catch #t - (lambda () - (thunk) - (primitive-exit 0)) - (lambda (key . args) - (print-exception (current-error-port) #f key args) - (primitive-exit 1)))) - (pid - #t))) - -(define* (report-build-progress total completed cont - #:optional (log-port (current-error-port))) - "Report that COMPLETED out of TOTAL files have been completed, and call -CONT." - (display #\cr log-port) - (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n - (* 100. (/ completed total)) total) - (force-output log-port) - (cont)) - -(define* (p-for-each proc lst - #:optional (max-processes (current-processor-count)) - #:key (progress report-build-progress)) - "Invoke PROC for each element of LST in a separate process, using up to -MAX-PROCESSES processes in parallel. Call PROGRESS at each step, passing = it -the continuation. Raise an error if one of the processes exit with non-ze= ro." - (define total - (length lst)) - - (define (wait-for-one-process) - (match (waitpid WAIT_ANY) - ((_ . status) - (unless (zero? (status:exit-val status)) - (error "process failed" proc status))))) - - (let loop ((lst lst) - (running 0) - (completed 0)) - (match lst - (() - (or (zero? running) - (let ((running (- running 1)) - (completed (+ completed 1))) - (wait-for-one-process) - (progress total completed - (lambda () - (loop lst running completed)))))) - ((head . tail) - (if (< running max-processes) - (let ((running (+ 1 running))) - (call-with-process (cut proc head)) - (progress total completed - (lambda () - (loop tail running completed)))) - (let ((running (- running 1)) - (completed (+ completed 1))) - (wait-for-one-process) - (progress total completed - (lambda () - (loop lst running completed))))))))) - (define* (build-guix out source #:key gcrypt - (debug-port (%make-void-port "w"))) + (debug-port (%make-void-port "w")) + (log-port (current-error-port))) "Build and install Guix in directory OUT using SOURCE, a directory containing the source code. Write any debugging output to DEBUG-PORT." (setvbuf (current-output-port) _IOLBF) @@ -130,33 +67,57 @@ containing the source code. Write any debugging outpu= t to DEBUG-PORT." (set! %load-path (cons out %load-path)) (set! %load-compiled-path (cons out %load-compiled-path)) =20 - ;; Compile the .scm files. Do that in independent processes, =C3=A0 la - ;; 'make -j', to work around (FIXME). - ;; This ensures correctness, but is overly conservative and slow. - ;; The solution initially implemented (and described in the bug - ;; above) was slightly faster but consumed memory proportional to the - ;; number of modules, which quickly became unacceptable. - (p-for-each (lambda (file) - (let ((go (string-append (string-drop-right file 4) - ".go"))) - (format debug-port "~%compiling '~a'...~%" file) - (parameterize ((current-warning-port debug-port)) - (compile-file file - #:output-file go - #:opts - %auto-compilation-options)))) - - (filter (cut string-suffix? ".scm" <>) - - ;; Build guix/*.scm before gnu/*.scm to speed - ;; things up. - (sort (find-files out "\\.scm") - (let ((guix (string-append out "/guix")) - (gnu (string-append out "/gnu"))) - (lambda (a b) - (or (and (string-prefix? guix a) - (string-prefix? gnu b)) - (string (FIXME). + (let* ((files + ;; Load guix/ modules before gnu/ modules to get somewhat stea= dier + ;; progress reporting. + (sort (filter (cut string-suffix? ".scm" <>) + (find-files out "\\.scm")) + (let ((guix (string-append out "/guix")) + (gnu (string-append out "/gnu"))) + (lambda (a b) + (or (and (string-prefix? guix a) + (string-prefix? gnu b)) + (string/foo/bar.scm" into (foo bar). + (let* ((relative-file (string-drop file (+ (string-length out) = 1))) + (module-path (string-drop-right relative-file 4)) + (module-name (map string->symbol + (string-split module-path #\/)))) + (parameterize ((current-warning-port debug-port)) + (resolve-interface module-name))) + (loop files (+ 1 completed))))) + (newline) + (let ((mutex (make-mutex)) + (completed 0)) + (par-for-each + (lambda (file) + (with-mutex mutex + (display #\cr log-port) + (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i= 18n + (* 100. (/ completed total)) total) + (force-output log-port) + (format debug-port "~%compiling '~a'...~%" file)) + (let ((go (string-append (string-drop-right file 4) ".go"))) + (parameterize ((current-warning-port (%make-void-port "w"))) + (compile-file file + #:output-file go + #:opts %auto-compilation-options))) + (with-mutex mutex + (set! completed (+ 1 completed)))) + files)))) =20 ;; Remove the "fake" (guix config). (delete-file (string-append out "/guix/config.scm")) --=20 2.6.3 --=-=-=--