From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mark H Weaver Subject: Re: [PATCH] build: Speed up .go compilation. Date: Sun, 10 Jan 2016 11:47:11 -0500 Message-ID: <877fjhjsa8.fsf@netris.org> References: <87lha3rx04.fsf@T420.taylan> <87mvsgxpef.fsf@T420.taylan> <87ziwgf1b4.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:45160) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aIJ9X-0006z6-6G for guix-devel@gnu.org; Sun, 10 Jan 2016 11:47:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aIJ9U-00046D-1J for guix-devel@gnu.org; Sun, 10 Jan 2016 11:47:39 -0500 In-Reply-To: <87ziwgf1b4.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Fri, 08 Jan 2016 18:06:39 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org ludo@gnu.org (Ludovic Court=C3=A8s) writes: > taylanbayirli@gmail.com (Taylan Ulrich "Bay=C4=B1rl=C4=B1/Kammer") skribi= s: > >> + (with-target target >> + (lambda () >> + (let ((mutex (make-mutex))) >> + (par-for-each >> + (lambda (file) >> + (let ((go (scm->go file))) >> + (unless (and (file-exists? go) >> + (file-mtime> + (with-mutex mutex >> + (format #t " GUILEC ~s~%" file) >> + (force-output)) >> + (compile-file file #:output-file go #:opts compile-optio= ns) >> + (with-mutex mutex >> + (format #t " WROTE ~s~%" go) >> + (force-output))))) >> + files))))) I'm sorry I haven't been following this discussion closely, but let me first say that the performance gains you've been able to achieve are very exciting. Thanks for working on this! Unfortunately, I have concerns: A few people mentioned on IRC that when doing these concurrent compilations within a single process, they sometimes see warnings like this: ;;; WARNING (module # not in submodules table) I haven't yet investigated, but my strong suspicion is that this is due to the fact that Guile's module system is not thread safe. More specifically, when a new module is created, it mutates the global directory of modules in a way that is not thread safe. New modules are created by 'compile-file', both for the module being compiled and for any imported modules that haven't been previously loaded. Unfortunately, this means that this approach of compiling files in multiple threads within a single guile process is not safe. There are likely to be random crashes and corruptions. I suggest that for now, the best way can do safely is to adopt an approach of running multiple Guile processes, where each one compiles multiple files within a single thread. Regards, Mark