From mboxrd@z Thu Jan 1 00:00:00 1970 From: Efraim Flashner Subject: Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!) Date: Sat, 7 Oct 2017 22:35:08 +0300 Message-ID: <20171007193507.GB2547@macbook42.flashner.co.il> References: <877f16z9eo.fsf@gnu.org> <87infv54m3.fsf@gnu.org> <87efqgnn7x.fsf@elephly.net> <87fuavzjms.fsf_-_@netris.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="bCsyhTFzCvuiizWE" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:42645) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e0usc-0000UL-RO for guix-devel@gnu.org; Sat, 07 Oct 2017 15:35:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1e0usY-00019a-2N for guix-devel@gnu.org; Sat, 07 Oct 2017 15:35:22 -0400 Received: from flashner.co.il ([178.62.234.194]:44025) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1e0usX-00018r-Gp for guix-devel@gnu.org; Sat, 07 Oct 2017 15:35:17 -0400 Content-Disposition: inline In-Reply-To: <87fuavzjms.fsf_-_@netris.org> List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+gcggd-guix-devel=m.gmane.org@gnu.org Sender: "Guix-devel" To: Mark H Weaver Cc: guix-devel@gnu.org --bCsyhTFzCvuiizWE Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Oct 07, 2017 at 12:06:51AM -0400, Mark H Weaver wrote: > Ricardo Wurmus writes: >=20 > > Hi Ludo, > > > >>> Here are some important items I can think of: > > [=E2=80=A6] > >>> =E2=80=A2 Guile 2.2 compiler terrible issue: > >>> . > > > > One way to side-step this issue for the upcoming release is to use one > > Guile process per file when running =E2=80=9Cguix pull=E2=80=9D. This = will make it run > > a lot slower, but that would be better than the current situation. >=20 > I've attached a workaround that I've been using for the last 6 weeks on > my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and > otherwise it would not be able to successfully build the 'guix' package. >=20 > Note that I never use 'guix pull', so I'm not sure off-hand whether this > solves the problem there, but it certainly greatly reduces the memory > needed to run 'make' and thus to build the 'guix' package. >=20 > This patch modifies build-aux/compile-all.scm to work as follows: after > loading all modules in the parent process, it then forks off a child and > compiles 20 modules in parallel within that child while the parent > waits. When the child is finished compiling those 20 modules, the child > exits (thus freeing the memory leaked during compilation), and then the > parent spawns a new child to compile the next 20 modules, and so on, > until all the modules are compiled. >=20 > We should probably consider applying this to master. Thoughts? >=20 > Mark >=20 >=20 Can we give it a set to compile in order? For example building guix/build[-system] and then gnu/build, and then only at the end build gnu/services {guix,gnu}/tests? It might be slightly faster to build the modules that don't pull in gnu/packages/* first and then end with those that would use those modules. --=20 Efraim Flashner =D7=90=D7=A4=D7=A8=D7=99=D7=9D = =D7=A4=D7=9C=D7=A9=D7=A0=D7=A8 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --bCsyhTFzCvuiizWE Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAlnZLFkACgkQQarn3Mo9 g1FibA/9GG59oucnwaFqmZsfU8AIF/2SS2fFvntJE1Uh+Y5d3PajmeFeCHLZnShU QYjyjxk43Nj/0bxIWJLZGht6AyZGHu9LM96WHtqjc9yebLkoDi8qd5DL5raV+iy9 4KqnHnkcr795HaiVlFO1E/KZwg+qFzIzhwVDLU6XpMhWuy1FI8o0aRBz9/8280W0 GObqN/EqAPTRiqk2daFuEqWFaSqcyYGjAb/tthw1I2hoqNy1vslC5KSFOY1eTKKi MIqcOUranBB9oIfbQzvVzSUgr9XjdXECO9kIwYrmUYAAbC9SIsXddfA2bBpsaKpP PQ9k0cR1+JtTECgw4iFYxGztoF37s4hYklw434Y5y60hIkw7GqGNwfBtwQni8IE/ LopKWgyOBRaBy5mj3CEW89TStDHLyXc9CFTvaaxHlNFuGExu/2fYRFHC++EymLYe T1i6RQcFhgp7zklj8AIPDLEuP0qqZZ0sDar5Gfwl1XuJLg8Am9/6I1iQ8oE21sR1 C/KrTCQfnpdQC2cL8jSd0sSFeMYAWz2HSzdGSXDDc3GonHfuYNi8TKZBWMgAwUf8 TS1N0dHTx8dBfYSB1SbT+WEQuFnHzenchgjvkyNMfWLfH0856DKcpqJXyu+OcO0I e9QqbLW7G+uze1ShlvCHruDA9eVwWtSqcRSphOMpfasdadRyFJE= =6ttV -----END PGP SIGNATURE----- --bCsyhTFzCvuiizWE--