From mboxrd@z Thu Jan 1 00:00:00 1970 From: taylanbayirli@gmail.com (Taylan Ulrich =?utf-8?Q?Bay=C4=B1rl=C4=B1?= =?utf-8?Q?=2FKammer?=) Subject: Re: [PATCH] build: Speed up .go compilation. Date: Mon, 11 Jan 2016 22:56:33 +0100 Message-ID: <87ziwbvkz2.fsf@T420.taylan> References: <87lha3rx04.fsf@T420.taylan> <87mvsgxpef.fsf@T420.taylan> <87ziwgf1b4.fsf@gnu.org> <877fjhjsa8.fsf@netris.org> <87r3hnes40.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:57080) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aIkS5-0006mz-HU for guix-devel@gnu.org; Mon, 11 Jan 2016 16:56:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aIkS4-0001vn-EB for guix-devel@gnu.org; Mon, 11 Jan 2016 16:56:37 -0500 In-Reply-To: <87r3hnes40.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 11 Jan 2016 22:14:23 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Ludovic =?utf-8?Q?Court=C3=A8s?= Cc: guix-devel@gnu.org ludo@gnu.org (Ludovic Court=C3=A8s) writes: > Mark H Weaver skribis: > >> I haven't yet investigated, but my strong suspicion is that this is due >> to the fact that Guile's module system is not thread safe. More >> specifically, when a new module is created, it mutates the global >> directory of modules in a way that is not thread safe. >> >> New modules are created by 'compile-file', both for the module being >> compiled and for any imported modules that haven't been previously >> loaded. Unfortunately, this means that this approach of compiling files >> in multiple threads within a single guile process is not safe. There >> are likely to be random crashes and corruptions. > > Right. This is one of the concerns I raised before I forgot again. ;-) > > https://lists.gnu.org/archive/html/guix-devel/2015-11/msg00359.html > > Taylan writes: > >> The latest version of the patch loads all needed modules first, in the >> main thread, and then compiles the files corresponding to the modules in >> parallel. Does that still lead to mutation in the directory of modules >> in the parallelized segment of the code? > > It seems to be safe, but we=E2=80=99re treading in a risky zone. > > We=E2=80=99re pushing =E2=80=98compile-file=E2=80=99 and related code to = its limits. It would > be great to address these issues in Guile itself. If we don't call compile-file in parallel but still do everything in one process, we will still be cutting down the time significantly. For instance, currently the load phase and compile phase take about one minute each on my machine with 4 cores. Making it non-parallel would mean the compile time takes 4x longer, i.e. 4 min., which brings us to 5 minutes total. The original was around 20. On devices with fewer cores, the parallel compile is even less important. So in the worst case we can just turn that par-for-each into a for-each and still get benefits. Taylan