From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andy Wingo Subject: Re: [PATCH] build: pull: Compile .scm files in one process. Date: Mon, 09 Nov 2015 07:41:40 +0000 Message-ID: <87lha7r557.fsf@igalia.com> References: <87si4kxtge.fsf@T420.taylan> <87611frtsx.fsf@igalia.com> <87d1vnxepw.fsf@T420.taylan> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:47523) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zvh5r-0004Q7-Px for guix-devel@gnu.org; Mon, 09 Nov 2015 02:42:24 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zvh5o-00062E-Ji for guix-devel@gnu.org; Mon, 09 Nov 2015 02:42:23 -0500 Received: from pb-sasl0.int.icgroup.com ([208.72.237.25]:55292 helo=sasl.smtp.pobox.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zvh5o-000605-E3 for guix-devel@gnu.org; Mon, 09 Nov 2015 02:42:20 -0500 In-Reply-To: <87d1vnxepw.fsf@T420.taylan> ("Taylan Ulrich =?utf-8?Q?=5C=22Bay=C4=B1rl=C4=B1=2FKammer=5C=22=22's?= message of "Fri, 06 Nov 2015 17:41:31 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Taylan Ulrich =?utf-8?Q?=22Bay=C4=B1rl=C4=B1?= =?utf-8?Q?=2FKammer=22?= Cc: guix-devel@gnu.org On Fri 06 Nov 2015 16:41, taylanbayirli@gmail.com (Taylan Ulrich "Bay=C4=B1= rl=C4=B1/Kammer") writes: > Andy Wingo writes: > >> On Thu 05 Nov 2015 17:10, taylanbayirli@gmail.com (Taylan Ulrich "Bay=C4= =B1rl=C4=B1/Kammer") writes: >> >>> It used to max out every CPU core, now just one. :-) >>> It used to take ~18 minutes on my machine, now less than 3. >> >> If you compile within a par-for-each you should be able to peg your CPU >> core again, but actually reduce the time :) > > From what I understand, that would probably ignite the bug again. > > We need to ensure that as soon as a module file is compiled, it's also > explicitly loaded before anything else is compiled (which might import > it), otherwise that compilation will import the "degenerate" version of > the module that results from compiling but not loading it. > > But that's really just my shallow high-level understanding of the bug, > and could be way off. If you have any insights on what's really going > on, that would be greatly appreciated. :-) AFAIU the problem that makes compilation slow is that *expansion* is slow. When all your Scheme files are fresh, compiling 1 module has to expand all N modules. Using one process to compile avoids this N^2 penalty, just paying the O(N) cost up-front and then the marginal compilation cost is O(1). There is benefit to compiling support modules before compiling (gnu packages) so that Guix's macros run compiled and not interpreted, but if you already have all of the modules expanded and loaded I don't think there is any advantage to loading the freshly compiled .go files. I do not understand what you mean by "degenerate" modules :) An interpreted module should act the same as a compiled module. I am interested to hear what difference you can perceive between the two. Andy