From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: [PATCH] build: pull: Compile .scm files in one process. Date: Thu, 12 Nov 2015 21:44:38 +0100 Message-ID: <876117t0ax.fsf@gnu.org> References: <87si4kxtge.fsf@T420.taylan> <87611gdul8.fsf@gnu.org> <87h9kzy09b.fsf@T420.taylan> <87bnb6c0nh.fsf@gnu.org> <874mgyxhgy.fsf@T420.taylan> <877flpohu6.fsf@gnu.org> <87mvuku444.fsf@T420.taylan> <87pozgfyzt.fsf@gnu.org> <87io57tt2s.fsf@T420.taylan> <876117mnef.fsf@igalia.com> <87egfvtnbw.fsf@T420.taylan> <87y4e3l7hm.fsf@igalia.com> <87a8qjtje8.fsf@T420.taylan> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:57523) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zwyjc-0005q5-Vx for guix-devel@gnu.org; Thu, 12 Nov 2015 15:44:45 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZwyjZ-0007Ve-Mv for guix-devel@gnu.org; Thu, 12 Nov 2015 15:44:44 -0500 In-Reply-To: <87a8qjtje8.fsf@T420.taylan> ("Taylan Ulrich \=\?utf-8\?Q\?\=5C\=22Bay\=C4\=B1rl\=C4\=B1\=2FKammer\=5C\=22\=22's\?\= message of "Thu, 12 Nov 2015 14:52:15 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Taylan Ulrich =?utf-8?Q?=22Bay=C4=B1rl=C4=B1?= =?utf-8?Q?=2FKammer=22?= Cc: guix-devel@gnu.org taylanbayirli@gmail.com (Taylan Ulrich "Bay=C4=B1rl=C4=B1/Kammer") skribis: > With that and your further help on IRC, I ended up with the following > patch. > > This takes about 2 minutes on my system. The CPU gets fried during the > parallel compilation, as expected. ;-) Nice. :-) > From 0eaf7d4c264dc531718a4c0b933323f48ea91930 Mon Sep 17 00:00:00 2001 > From: =3D?UTF-8?q?Taylan=3D20Ulrich=3D20Bay=3DC4=3DB1rl=3DC4=3DB1/Kammer?= =3D > > Date: Thu, 5 Nov 2015 23:43:20 +0100 > Subject: [PATCH] build: pull: Compile .scm files in one process. > > * guix/build/pull.scm (call-with-process): Removed procedure. > (report-build-progress): Removed procedure. > (p-for-each): Removed procedure. Rather: =E2=80=A6 (call-with-process, report-build-progress, p-for-each): Remove. > + (let ((mutex (make-mutex)) > + (completed 0)) > + (par-for-each > + (lambda (file) > + (with-mutex mutex > + (display #\cr log-port) > + (format log-port "compiling...\t~5,1f% of ~d files" ;FIXM= E: i18n > + (* 100. (/ completed total)) total) > + (force-output log-port) > + (format debug-port "~%compiling '~a'...~%" file)) > + (let ((go (string-append (string-drop-right file 4) ".go"))) > + (compile-file file > + #:output-file go > + #:opts %auto-compilation-options)) > + (with-mutex mutex > + (set! completed (+ 1 completed)))) > + files))))) Does it actually work reliably? :-) I think we=E2=80=99re in trouble if DEBUG-PORT is a real port because ports aren=E2=80=99t thread-safe in 2.0, and =E2=80=98compile-file=E2=80=99 might= write to it. Void ports seem to be OK because their =E2=80=98write=E2=80=99 method doesn=E2= =80=99t do anything. So I think we have to do things sequentially when DEBUG-PORT is non-void. I=E2=80=99m also concerned about modules accessed by concurrent calls to =E2=80=98compile-file=E2=80=99 because modules and hash tables aren=E2=80= =99t thread-safe either. WDYT? Thanks to the two of you! Sounds like we=E2=80=99ll soon have a less slow = or even a fast =E2=80=98guix pull=E2=80=99. ;-) Ludo=E2=80=99.