From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: [PATCH] build: Speed up .go compilation. Date: Thu, 14 Jan 2016 15:02:35 +0100 Message-ID: <87a8o8qmx0.fsf@gnu.org> References: <87lha3rx04.fsf@T420.taylan> <87mvsgxpef.fsf@T420.taylan> <87ziwgf1b4.fsf@gnu.org> <877fjhjsa8.fsf@netris.org> <87r3hnes40.fsf@gnu.org> <87ziwbvkz2.fsf@T420.taylan> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:38573) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aJiU5-0001yR-W0 for guix-devel@gnu.org; Thu, 14 Jan 2016 09:02:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aJiU1-0006Bd-Re for guix-devel@gnu.org; Thu, 14 Jan 2016 09:02:41 -0500 In-Reply-To: <87ziwbvkz2.fsf@T420.taylan> ("Taylan Ulrich \=\?utf-8\?Q\?\=5C\=22Bay\=C4\=B1rl\=C4\=B1\=2FKammer\=5C\=22\=22's\?\= message of "Mon, 11 Jan 2016 22:56:33 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Taylan Ulrich =?utf-8?Q?=22Bay=C4=B1rl=C4=B1?= =?utf-8?Q?=2FKammer=22?= Cc: guix-devel@gnu.org taylanbayirli@gmail.com (Taylan Ulrich "Bay=C4=B1rl=C4=B1/Kammer") skribis: > ludo@gnu.org (Ludovic Court=C3=A8s) writes: > >> Mark H Weaver skribis: >> >>> I haven't yet investigated, but my strong suspicion is that this is due >>> to the fact that Guile's module system is not thread safe. More >>> specifically, when a new module is created, it mutates the global >>> directory of modules in a way that is not thread safe. >>> >>> New modules are created by 'compile-file', both for the module being >>> compiled and for any imported modules that haven't been previously >>> loaded. Unfortunately, this means that this approach of compiling files >>> in multiple threads within a single guile process is not safe. There >>> are likely to be random crashes and corruptions. >> >> Right. This is one of the concerns I raised before I forgot again. ;-) >> >> https://lists.gnu.org/archive/html/guix-devel/2015-11/msg00359.html >> >> Taylan writes: >> >>> The latest version of the patch loads all needed modules first, in the >>> main thread, and then compiles the files corresponding to the modules in >>> parallel. Does that still lead to mutation in the directory of modules >>> in the parallelized segment of the code? >> >> It seems to be safe, but we=E2=80=99re treading in a risky zone. >> >> We=E2=80=99re pushing =E2=80=98compile-file=E2=80=99 and related code to= its limits. It would >> be great to address these issues in Guile itself. > > If we don't call compile-file in parallel but still do everything in one > process, we will still be cutting down the time significantly. > > For instance, currently the load phase and compile phase take about one > minute each on my machine with 4 cores. Making it non-parallel would > mean the compile time takes 4x longer, i.e. 4 min., which brings us to 5 > minutes total. The original was around 20. On devices with fewer > cores, the parallel compile is even less important. > > So in the worst case we can just turn that par-for-each into a for-each > and still get benefits. Right. Mark, WDYT? I would say: go for it, and let=E2=80=99s switch back to =E2=80=98for-each= =E2=80=99 if/when we have evidence of things going wrong. I think that if things go wrong, that=E2=80=99ll be a hard failure (segfaul= t or some random run-time error) and not a silent hard-to-detect corruption. Ludo=E2=80=99.