From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#27284: Memory leak in 'guix pull' or 'make' in guix source Date: Sun, 12 Nov 2017 22:33:57 +0100 Message-ID: <87375jchbu.fsf@gnu.org> References: <20170608083935.izw747zaetkaxv4o@abyayala> <87poamv2i7.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:54451) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eDzuF-0007Bn-Kp for bug-guix@gnu.org; Sun, 12 Nov 2017 16:35:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eDzuA-0005Ec-SB for bug-guix@gnu.org; Sun, 12 Nov 2017 16:35:07 -0500 Received: from debbugs.gnu.org ([208.118.235.43]:56843) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eDzuA-0005EV-P4 for bug-guix@gnu.org; Sun, 12 Nov 2017 16:35:02 -0500 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87poamv2i7.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Tue, 19 Sep 2017 22:48:16 +0200") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: 27284@debbugs.gnu.org Heya, ludo@gnu.org (Ludovic Court=C3=A8s) skribis: > So, we have two problems: compilation time, and memory consumption. I > *think* I=E2=80=99ve identified one of the major causes for both in Guile, > though it=E2=80=99s too early to say exactly how much this will impact re= source > consumption for a full Guix compilation. See > > for details. The latest news is that (1) the weak-table fix in Guile=C2=B9 helps in general but does not significantly reduce memory consumption when compiling gnu/packages/python.scm (which requires ~1.5G on x86_64), and (2) we=E2=80=99ve identified the compiler=E2=80=99s slot allocator as a maj= or cause of memory consumption and CPU cost when compiling large files like python.scm: https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00035.html Andy may work on a simpler slot allocator for -O0 and/or large functions that doesn=E2=80=99t exhibit this pathological behavior. In the meantime, our best workaround to reduce memory consumption is=E2=80= =A6 to split large files into smaller ones. Per M-x guix-locations, the candidates are: gnu/packages/python.scm 986 gnu/packages/perl.scm 401 gnu/packages/haskell.scm 348 gnu/packages/bioinformatics.scm 274 gnu/packages/statistics.scm 222 gnu/packages/emacs.scm 213 gnu/packages/xorg.scm 196 gnu/packages/web.scm 188 gnu/packages/gnome.scm 162 gnu/packages/ruby.scm 159 gnu/packages/java.scm 149 gnu/packages/ocaml.scm 126 gnu/packages/tex.scm 107 gnu/packages/linux.scm 106 I think we could focus on the first two or three files. FTR, compiling bioinformatics.scm peaks at ~500 MiB resident on x86_64. Ricardo, WDYT? If we do this, do we split arbitrarily? Like the second half of python.scm goes to python-cont.scm (provided there are no cross top-level references)? Or do you have a better idea? Ludo=E2=80=99. =C2=B9 https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00051.html