From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: Re: [PATCH] build: Speed up .go compilation. Date: Sat, 09 Jan 2016 22:59:18 +0100 Message-ID: <87bn8u76tl.fsf@gnu.org> References: <87lha3rx04.fsf@T420.taylan> <87mvsgxpef.fsf@T420.taylan> <87ziwgf1b4.fsf@gnu.org> <87egdqy24m.fsf@T420.taylan> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:40749) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aI1Xh-000465-AQ for guix-devel@gnu.org; Sat, 09 Jan 2016 16:59:26 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aI1Xd-0003RK-HX for guix-devel@gnu.org; Sat, 09 Jan 2016 16:59:25 -0500 In-Reply-To: <87egdqy24m.fsf@T420.taylan> ("Taylan Ulrich \=\?utf-8\?Q\?\=5C\=22Bay\=C4\=B1rl\=C4\=B1\=2FKammer\=5C\=22\=22's\?\= message of "Sat, 09 Jan 2016 20:38:33 +0100") List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org To: Taylan Ulrich =?utf-8?Q?=22Bay=C4=B1rl=C4=B1?= =?utf-8?Q?=2FKammer=22?= Cc: guix-devel@gnu.org taylanbayirli@gmail.com (Taylan Ulrich "Bay=C4=B1rl=C4=B1/Kammer") skribis: > ludo@gnu.org (Ludovic Court=C3=A8s) writes: > >> So, what speedup to you get compared to =E2=80=98make -jN=E2=80=99? > > It should have the same improvement as 'guix pull' since it does > basically the same thing. I measured to make things concrete, and on my > machine the run time of 'make -j4' decreases from ~26m to ~6m. On my 4-core machine, it takes a bit less than 2mn. > Here the lambda cannot entirely be factored out since the 'mutex' from > the lexical scope is needed. I still factored out the actual code and > pass the mutex as an argument to the procedure in a minimal lambda, as > seen in the new patch below; I hope it's readable this way. It is. >> It would be awesome if you could check that =E2=80=98make distcheck=E2= =80=99 still >> passes, and also make sure things behave correctly when modifying just >> one file and running =E2=80=98make=E2=80=99, things like that. > > I'm having headaches with distcheck. Currently it bails out because I'm > missing tex. Debian's version is apparently too old, and Guix's version > is huge and has been downloading for many hours now. I'll report back > again when I'm done with that. OK. > Other notable changes in this version of the patch: > > - I noticed guix/config.scm and guix/tests.scm are not in MODULES in > Makefile.am (intentionally?), so I added them to the make-go rule. Good catch. This is indeed intentional, because $(MODULES) goes to the distribution, but we wouldn=E2=80=99t want to distribute config.scm, which is derived from config.scm.in at configure-time; tests.scm is not in $(MODULES) because it is not installed. > - I removed the MKDIR_P loop in the make-go rule, and do the equivalent > in the Scheme code now. > > - The target host and top source directory are now passed to the script > via lowercase environment variables, which makes the code a little > simpler. I hope this is stylistically fine. Good! > From 697950b82ea86f7b7438e586bbf4efae3e87d8f8 Mon Sep 17 00:00:00 2001 > From: =3D?UTF-8?q?Taylan=3D20Ulrich=3D20Bay=3DC4=3DB1rl=3DC4=3DB1/Kammer?= =3D > > Date: Thu, 5 Nov 2015 23:42:45 +0100 > Subject: [PATCH] build: Speed up .go compilation. > > * build-aux/compile-all.scm: New file. > * Makefile.am (EXTRA_DIST): Add it. > (%.go, make-go): New rules. The code looks good to me. I=E2=80=99ve tested it, it works fine indeed, a= nd it=E2=80=99s much faster than =E2=80=98make -j4=E2=80=99! I have one concern: while running it peaked at 300=C2=A0MiB resident in =E2=80=98top=E2=80=99, which is OK on many machines but still quite a lot. = I realize we have the same problem with =E2=80=98guix pull=E2=80=99 now, but that=E2=80= =99s not great. Part of the problem may be that modules are not GC=E2=80=99d. But anyway, that= =E2=80=99s a scalability problem. What do people think? A lesser concern is the long command-line passed to compile-all.scm, notably because Emacs=E2=80=99 compilation-mode runs regexps on each line, = and that takes time proportional to the length of the line, so that leads Emacs to hang for a second when it sees that line. Silly. ;-) Thanks! Ludo=E2=80=99.