From mboxrd@z Thu Jan 1 00:00:00 1970 From: taylanbayirli@gmail.com (Taylan Ulrich =?utf-8?Q?Bay=C4=B1rl=C4=B1?= =?utf-8?Q?=2FKammer?=) Subject: Re: [PATCH] build: pull: Compile .scm files in one process. Date: Fri, 27 Nov 2015 09:53:28 +0100 Message-ID: <87fuzrlt6f.fsf@T420.taylan> References: <87si4kxtge.fsf@T420.taylan> <87611gdul8.fsf@gnu.org> <87h9kzy09b.fsf@T420.taylan> <87bnb6c0nh.fsf@gnu.org> <874mgyxhgy.fsf@T420.taylan> <877flpohu6.fsf@gnu.org> <87mvuku444.fsf@T420.taylan> <87pozgfyzt.fsf@gnu.org> <87io57tt2s.fsf@T420.taylan> <876117mnef.fsf@igalia.com> <87egfvtnbw.fsf@T420.taylan> <87y4e3l7hm.fsf@igalia.com> <87a8qjtje8.fsf@T420.taylan> <876117t0ax.fsf@gnu.org> <877flmrn2m.fsf@T420.taylan> <87a8q0ies5.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:40649) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a2Emb-0005Mc-Io for guix-devel@gnu.org; Fri, 27 Nov 2015 03:53:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a2EmZ-0008G7-FK for guix-devel@gnu.org; Fri, 27 Nov 2015 03:53:33 -0500 In-Reply-To: <87a8q0ies5.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Thu, 26 Nov 2015 23:20:26 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable ludo@gnu.org (Ludovic Court=C3=A8s) writes: > taylanbayirli@gmail.com (Taylan Ulrich "Bay=C4=B1rl=C4=B1/Kammer") skribi= s: > >> From 78be6d09d2d4c0a563be14c66ac2a1a345ff9b1d Mon Sep 17 00:00:00 2001 >> From: =3D?UTF-8?q?Taylan=3D20Ulrich=3D20Bay=3DC4=3DB1rl=3DC4=3DB1/Kammer= ?=3D >> >> Date: Thu, 5 Nov 2015 23:43:20 +0100 >> Subject: [PATCH] build: pull: Compile .scm files in one process. >> >> * guix/build/pull.scm (call-with-process, report-build-progress) >> (p-for-each): Remove. >> (thread-safe-port): New procedure. >> (build-guix): Load and compile files in one process. > > Just tried this patch without the =E2=80=98thread-safe-port=E2=80=99 proc= edure, but I > got this (current master): > > loading... 95.4% of 474 filesBacktrace: > In ice-9/boot-9.scm: > 157: 14 [catch #t # ...] > In unknown file: > ?: 13 [apply-smob/1 #] > In ice-9/boot-9.scm: > 63: 12 [call-with-prompt prompt0 ...] > In ice-9/eval.scm: > 432: 11 [eval # #] > In ice-9/boot-9.scm: > 2401: 10 [save-module-excursion #] > 4050: 9 [#] > 1724: 8 [%start-stack load-stack #] > 1729: 7 [#] > In unknown file: > ?: 6 [primitive-load "/gnu/store/hx0jk73cx50f3vpi0yyrbn0pd8ws8m0v-guix= -latest-builder"] > In ./guix/build/pull.scm: > 47: 5 [build-guix "/gnu/store/d51z2xkwp1vh0dh6gqadyyzv21m0b772-guix-lat= est" ...] > 91: 4 [#] > In ice-9/boot-9.scm: > 2401: 3 [save-module-excursion #] > In unknown file: > ?: 2 [primitive-load "/gnu/store/d51z2xkwp1vh0dh6gqadyyzv21m0b772-guix= -latest/guix/scripts/import/hackage.scm"] > In ice-9/eval.scm: > 453: 1 Exception thrown while printing backtrace: > ERROR: In procedure package-location: Wrong type argument: Error while pr= inting exception. > > ice-9/eval.scm:387:11: In procedure eval: > ice-9/eval.scm:387:11: In procedure package-version: Wrong type argument:= Error while printing exception. > builder for `/gnu/store/pc1i5s6vx9yx97prhskx178gj5swxw4k-guix-latest.drv'= failed with exit code 1 > guix pull: error: build failed: build of `/gnu/store/pc1i5s6vx9yx97prhskx= 178gj5swxw4k-guix-latest.drv' failed > > Any idea? > > To me it sounds like there are two record type descriptors in > the wild, which is why =E2=80=98package-location=E2=80=99 in the package = record printer > bails out. That's one of the errors that result from a "bad" order of compiling the files and when the 'load' hack isn't used to work around it, which isn't the case in that patch... Indeed I can't seem to reproduce the issue. The attached patch below also builds on the quoted patch, removes the thread-safe-port procedure, and just sets the warning port to a void port. Applied on top of the current master, it works for me. Maybe you applied a different patch by accident? Thanks for picking this up and sorry that I couldn't finish it. :-) --=-=-= Content-Type: text/x-diff; charset=utf-8 Content-Disposition: inline; filename=0001-build-pull-Compile-.scm-files-in-one-process.patch Content-Transfer-Encoding: quoted-printable >From 4cb8ad8006ba359e984f4b6e765be082b7d5f9c0 Mon Sep 17 00:00:00 2001 From: =3D?UTF-8?q?Taylan=3D20Ulrich=3D20Bay=3DC4=3DB1rl=3DC4=3DB1/Kammer?=3D Date: Fri, 27 Nov 2015 09:27:55 +0100 Subject: [PATCH] build: pull: Compile .scm files in one process. * guix/build/pull.scm (call-with-process, report-build-progress) (p-for-each): Remove. (build-guix): Load and compile files in one process. --- guix/build/pull.scm | 142 ++++++++++++++++++------------------------------= ---- 1 file changed, 48 insertions(+), 94 deletions(-) diff --git a/guix/build/pull.scm b/guix/build/pull.scm index 281be23..e77a582 100644 --- a/guix/build/pull.scm +++ b/guix/build/pull.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2013, 2014 Ludovic Court=C3=A8s +;;; Copyright =C2=A9 2015 Taylan Ulrich Bay=C4=B1rl=C4=B1/Kammer ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,6 +23,7 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 format) + #:use-module (ice-9 threads) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) @@ -33,75 +35,10 @@ ;;; ;;; Code: =20 -(define (call-with-process thunk) - "Run THUNK in a separate process that will return 0 if THUNK terminates -normally, and 1 if an exception is raised." - (match (primitive-fork) - (0 - (catch #t - (lambda () - (thunk) - (primitive-exit 0)) - (lambda (key . args) - (print-exception (current-error-port) #f key args) - (primitive-exit 1)))) - (pid - #t))) - -(define* (report-build-progress total completed cont - #:optional (log-port (current-error-port))) - "Report that COMPLETED out of TOTAL files have been completed, and call -CONT." - (display #\cr log-port) - (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: i18n - (* 100. (/ completed total)) total) - (force-output log-port) - (cont)) - -(define* (p-for-each proc lst - #:optional (max-processes (current-processor-count)) - #:key (progress report-build-progress)) - "Invoke PROC for each element of LST in a separate process, using up to -MAX-PROCESSES processes in parallel. Call PROGRESS at each step, passing = it -the continuation. Raise an error if one of the processes exit with non-ze= ro." - (define total - (length lst)) - - (define (wait-for-one-process) - (match (waitpid WAIT_ANY) - ((_ . status) - (unless (zero? (status:exit-val status)) - (error "process failed" proc status))))) - - (let loop ((lst lst) - (running 0) - (completed 0)) - (match lst - (() - (or (zero? running) - (let ((running (- running 1)) - (completed (+ completed 1))) - (wait-for-one-process) - (progress total completed - (lambda () - (loop lst running completed)))))) - ((head . tail) - (if (< running max-processes) - (let ((running (+ 1 running))) - (call-with-process (cut proc head)) - (progress total completed - (lambda () - (loop tail running completed)))) - (let ((running (- running 1)) - (completed (+ completed 1))) - (wait-for-one-process) - (progress total completed - (lambda () - (loop lst running completed))))))))) - (define* (build-guix out source #:key gcrypt - (debug-port (%make-void-port "w"))) + (debug-port (%make-void-port "w")) + (log-port (current-error-port))) "Build and install Guix in directory OUT using SOURCE, a directory containing the source code. Write any debugging output to DEBUG-PORT." (setvbuf (current-output-port) _IOLBF) @@ -130,33 +67,50 @@ containing the source code. Write any debugging outpu= t to DEBUG-PORT." (set! %load-path (cons out %load-path)) (set! %load-compiled-path (cons out %load-compiled-path)) =20 - ;; Compile the .scm files. Do that in independent processes, =C3=A0 la - ;; 'make -j', to work around (FIXME). - ;; This ensures correctness, but is overly conservative and slow. - ;; The solution initially implemented (and described in the bug - ;; above) was slightly faster but consumed memory proportional to the - ;; number of modules, which quickly became unacceptable. - (p-for-each (lambda (file) - (let ((go (string-append (string-drop-right file 4) - ".go"))) - (format debug-port "~%compiling '~a'...~%" file) - (parameterize ((current-warning-port debug-port)) - (compile-file file - #:output-file go - #:opts - %auto-compilation-options)))) - - (filter (cut string-suffix? ".scm" <>) - - ;; Build guix/*.scm before gnu/*.scm to speed - ;; things up. - (sort (find-files out "\\.scm") - (let ((guix (string-append out "/guix")) - (gnu (string-append out "/gnu"))) - (lambda (a b) - (or (and (string-prefix? guix a) - (string-prefix? gnu b)) - (string (FIXME). + (let* ((files + ;; gnu.scm depends on many other modules, so to avoid an early + ;; progress report stall, we move it to the end. + (sort (filter (cut string-suffix? ".scm" <>) + (find-files out "\\.scm")) + (lambda (a b) + (or (string-prefix? (string-append out "/gnu.scm") b) + (string