From mboxrd@z Thu Jan 1 00:00:00 1970 From: taylanbayirli@gmail.com (Taylan Ulrich =?utf-8?Q?Bay=C4=B1rl=C4=B1?= =?utf-8?Q?=2FKammer?=) Subject: Re: [PATCH] build: pull: Compile .scm files in one process. Date: Mon, 09 Nov 2015 09:51:40 +0100 Message-ID: <87611bwo6b.fsf@T420.taylan> References: <87si4kxtge.fsf@T420.taylan> <87611frtsx.fsf@igalia.com> <87d1vnxepw.fsf@T420.taylan> <87lha7r557.fsf@igalia.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:35801) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZviAy-0002gJ-Qu for guix-devel@gnu.org; Mon, 09 Nov 2015 03:51:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZviAx-0004XZ-MF for guix-devel@gnu.org; Mon, 09 Nov 2015 03:51:44 -0500 Received: from mail-wi0-x22c.google.com ([2a00:1450:400c:c05::22c]:34565) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZviAx-0004X6-G5 for guix-devel@gnu.org; Mon, 09 Nov 2015 03:51:43 -0500 Received: by wikq8 with SMTP id q8so63466748wik.1 for ; Mon, 09 Nov 2015 00:51:42 -0800 (PST) In-Reply-To: <87lha7r557.fsf@igalia.com> (Andy Wingo's message of "Mon, 09 Nov 2015 07:41:40 +0000") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Andy Wingo Cc: guix-devel@gnu.org Andy Wingo writes: > On Fri 06 Nov 2015 16:41, taylanbayirli@gmail.com (Taylan Ulrich "Bay=C4= =B1rl=C4=B1/Kammer") writes: > >> Andy Wingo writes: >> >>> On Thu 05 Nov 2015 17:10, taylanbayirli@gmail.com (Taylan Ulrich "Bay= =C4=B1rl=C4=B1/Kammer") writes: >>> >>>> It used to max out every CPU core, now just one. :-) >>>> It used to take ~18 minutes on my machine, now less than 3. >>> >>> If you compile within a par-for-each you should be able to peg your CPU >>> core again, but actually reduce the time :) >> >> From what I understand, that would probably ignite the bug again. >> >> We need to ensure that as soon as a module file is compiled, it's also >> explicitly loaded before anything else is compiled (which might import >> it), otherwise that compilation will import the "degenerate" version of >> the module that results from compiling but not loading it. >> >> But that's really just my shallow high-level understanding of the bug, >> and could be way off. If you have any insights on what's really going >> on, that would be greatly appreciated. :-) > > AFAIU the problem that makes compilation slow is that *expansion* is > slow. When all your Scheme files are fresh, compiling 1 module has to > expand all N modules. Using one process to compile avoids this N^2 > penalty, just paying the O(N) cost up-front and then the marginal > compilation cost is O(1). Right, I was conflating expansion and compilation. > There is benefit to compiling support modules before compiling (gnu > packages) so that Guix's macros run compiled and not interpreted, but if > you already have all of the modules expanded and loaded I don't think > there is any advantage to loading the freshly compiled .go files. > > I do not understand what you mean by "degenerate" modules :) An > interpreted module should act the same as a compiled module. I am > interested to hear what difference you can perceive between the two. The relevant bug report is . According to Ludo's explanation, compiling a module file leads to the module being created in the runtime, but with syntax bindings only, and runtime bindings missing. That's what I mean with "degenerate" module for lack of a better term. Loading the same file explicitly afterwards (or using load-compiled on the generated .go) seems to solve the issue. Taylan