From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: taylanbayirli@gmail.com (Taylan Ulrich =?utf-8?Q?Bay=C4=B1rl=C4=B1?= =?utf-8?Q?=2FKammer?=) Newsgroups: gmane.lisp.guile.devel,gmane.comp.gnu.guix.bugs Subject: Module system thread unsafety and .go compilation Date: Tue, 09 Feb 2016 21:02:27 +0100 Message-ID: <8737t1k5yk.fsf@T420.taylan> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: ger.gmane.org 1455048156 13756 80.91.229.3 (9 Feb 2016 20:02:36 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 9 Feb 2016 20:02:36 +0000 (UTC) Cc: guile-devel@gnu.org To: bug-guix@gnu.org Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Tue Feb 09 21:02:36 2016 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1aTEUd-0008S9-Va for guile-devel@m.gmane.org; Tue, 09 Feb 2016 21:02:36 +0100 Original-Received: from localhost ([::1]:60172 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aTEUd-0001rd-9F for guile-devel@m.gmane.org; Tue, 09 Feb 2016 15:02:35 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:53791) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aTEUZ-0001qy-I9 for guile-devel@gnu.org; Tue, 09 Feb 2016 15:02:32 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aTEUY-00063c-IL for guile-devel@gnu.org; Tue, 09 Feb 2016 15:02:31 -0500 Original-Received: from mail-wm0-x22f.google.com ([2a00:1450:400c:c09::22f]:33640) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aTEUY-00063T-BE; Tue, 09 Feb 2016 15:02:30 -0500 Original-Received: by mail-wm0-x22f.google.com with SMTP id g62so189072111wme.0; Tue, 09 Feb 2016 12:02:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:user-agent:mime-version :content-type; bh=vahbhIvWVj41OaVJM9OS+1iR9q+B+w1Da14IMPLAj4s=; b=ISRUc9M5+uAi1iMiEW0EewNz75z1mJh11oy7JiLNBFEkfRIbAUdNJ9kGlBgFDNjIO/ OmOPeg2BbEuTJdkMO2WF0Qo0hckzc6CfApggGw6o5PBxp5B1khlZbilS7WS2eapY7ay9 zW5rA21RZzmgI4NaVrhbDC9+zeBM32lS0c9P0YWqXjR/gSw7IiVqXI+mY5Tq57po4jgk KInB7Z8aiWWgDU2Y0Ls5iRIqi79FqyKnvAz75d+yaxEifLSWGYJEMOEd10crWtZMJkR6 mNromyxL2OFSYT73AKNUDA4nGKl1V94kFnpSyMcNeEjSxjY3IGCUSIG4fdswr0sZ31MF xkXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:user-agent :mime-version:content-type; bh=vahbhIvWVj41OaVJM9OS+1iR9q+B+w1Da14IMPLAj4s=; b=OM5p1VU1oYDlz8JzcCm9vh0xXGWPZz/JpianWuMQEfaJdSTqzRbYGOHnNrDacpgEuS s3m7BWyZ10E33pI/3lqXklFtfMQhEUmJKvjqMc18un2E9Sffn7+vagiBOw/Eb1ciSU09 VqmVGqAxxU1QnHyyOlWNg4eFbMQscQ39gvZIcddwUeryQksoWRGUrrp6SHFtvOX68kFf DAUzXJsMT6TCQTQ3Hq//bvDGIFVAEWJO+IWU55uhpHFBJMjslY0CMFuN61dwn065AO6r KrS1Ra5j/ZdlzPcpk8UybBpBhvzS1+1qTQC16ozS6Lg9wjhpRUfR2E09cogdXuLTc45G QJ5w== X-Gm-Message-State: AG10YOTNHBwF8CyJ4mBIIzKC1mYNNLlqmox6MXBxmgGzWZ5hUq8n+6MzLdriUC/Ij9AuPA== X-Received: by 10.28.103.5 with SMTP id b5mr7257922wmc.5.1455048149355; Tue, 09 Feb 2016 12:02:29 -0800 (PST) Original-Received: from T420.taylan ([2a02:908:c30:3ec0:221:ccff:fe66:68f0]) by smtp.gmail.com with ESMTPSA id y188sm164723wmy.11.2016.02.09.12.02.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Feb 2016 12:02:28 -0800 (PST) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a00:1450:400c:c09::22f X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.lisp.guile.devel:18165 gmane.comp.gnu.guix.bugs:3543 Archived-At: To speed up the compilation of the many Scheme files in Guix, we use a script that first loads all modules to be compiled into the Guile process (by calling 'resolve-interface' on the module names), and then the corresponding Scheme files are compiled in a par-for-each. While Guile's module system is known to be thread unsafe, the idea was that all mutation should happen in the serial loading phase, and the parallel compile-file calls should then be thread safe. Sadly that assumption isn't met when autoloads are involved. Minimal-ish test-case: - Check out 0889321. - Build it. - Edit gnu/build/activation.scm and gnu/build/linux-boot.scm to contain merely the following expressions, respectively: (define-module (gnu build activation) #:use-module (gnu build linux-boot)) (define-module (gnu build linux-boot) #:autoload (system base compile) (compile-file)) - Run make again. If you're on a multi-core system, you will probably get an error saying something weird like "no such language scheme". Note: when you then run make *again* it succeeds. Solution proposals: 1. s/par-for-each/for-each/. Will make compilation slower on multi-core machines. We would do the same for guix pull, which is a bit sad because it's so fast right now. Very simple solution though. 2. We find out some partitioning of the Scheme modules such that there is minimal overlap in total loaded modules when the modules in one subset are each loaded by one Guile process. Then each Guile process loads & compiles the modules in its given subset serially, but these Guile processes run in parallel. This could speed things up even more than now because the module-loading phases of the processes would be parallel too. It also has the side-effect that less memory is consumed the fewer cores you have (because less Scheme modules loaded into memory at once). If someone (Ludo?) has a good general overview of Guix's module graph then maybe they can come up with a sensible partitioning of the modules, say into 4 subsets (maxing out benefits at quad-core), such that loading all modules in one subset loads a minimal amount of modules that are outside that subset. That should be the only challenging part of this solution. 3. We do nothing for now since this bug triggers rarely, and can be worked around by simply re-running make. (We just have to hope that it doesn't trigger on guix pull or on clean builds after some commit; there's no "just rerun make" in guix pull or an automated build of Guix.) AFAIU Wingo expressed motivation to make Guile's module system thread safe, so this problem would then truly disappear. I think #2 is a pretty good solution. The only thing worrying me is that we might not be able to sensibly partition the Scheme modules according to any simple logic that can be automated (like guix/ is one subset, gnu/packages/ is another, etc.). Maintaining the subsets manually in the Makefile would be pretty ugly. But maybe some simple logic, possibly combined with few special-cases in the code, would be good enough. Thoughts? Taylan