From mboxrd@z Thu Jan 1 00:00:00 1970 From: taylanbayirli@gmail.com (Taylan Ulrich =?utf-8?Q?Bay=C4=B1rl=C4=B1?= =?utf-8?Q?=2FKammer?=) Subject: Re: [PATCH] build: Speed up .go compilation. Date: Sun, 10 Jan 2016 21:33:29 +0100 Message-ID: <87lh7xw4x2.fsf@T420.taylan> References: <87lha3rx04.fsf@T420.taylan> <87mvsgxpef.fsf@T420.taylan> <87ziwgf1b4.fsf@gnu.org> <877fjhjsa8.fsf@netris.org> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:45165) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aIMg9-0000K8-90 for guix-devel@gnu.org; Sun, 10 Jan 2016 15:33:34 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aIMg8-0001Zy-DG for guix-devel@gnu.org; Sun, 10 Jan 2016 15:33:33 -0500 In-Reply-To: <877fjhjsa8.fsf@netris.org> (Mark H. Weaver's message of "Sun, 10 Jan 2016 11:47:11 -0500") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Mark H Weaver Cc: guix-devel@gnu.org Mark H Weaver writes: > I'm sorry I haven't been following this discussion closely, but let me > first say that the performance gains you've been able to achieve are > very exciting. Thanks for working on this! Hi Mark, thanks for the kind words. :-) > Unfortunately, I have concerns: > > A few people mentioned on IRC that when doing these concurrent > compilations within a single process, they sometimes see warnings like > this: > > ;;; WARNING (module # not in submodules table) > > I haven't yet investigated, but my strong suspicion is that this is due > to the fact that Guile's module system is not thread safe. More > specifically, when a new module is created, it mutates the global > directory of modules in a way that is not thread safe. > > New modules are created by 'compile-file', both for the module being > compiled and for any imported modules that haven't been previously > loaded. Unfortunately, this means that this approach of compiling files > in multiple threads within a single guile process is not safe. There > are likely to be random crashes and corruptions. > > I suggest that for now, the best way can do safely is to adopt an > approach of running multiple Guile processes, where each one compiles > multiple files within a single thread. The latest version of the patch loads all needed modules first, in the main thread, and then compiles the files corresponding to the modules in parallel. Does that still lead to mutation in the directory of modules in the parallelized segment of the code? > Regards, > Mark Taylan