From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#27284: Memory leak in 'guix pull' or 'make' in guix source Date: Tue, 19 Sep 2017 22:48:16 +0200 Message-ID: <87poamv2i7.fsf@gnu.org> References: <20170608083935.izw747zaetkaxv4o@abyayala> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:33597) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1duPS3-00050y-Ex for bug-guix@gnu.org; Tue, 19 Sep 2017 16:49:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1duPS2-0000pJ-9u for bug-guix@gnu.org; Tue, 19 Sep 2017 16:49:03 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:40011) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1duPS2-0000ol-5A for bug-guix@gnu.org; Tue, 19 Sep 2017 16:49:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <20170608083935.izw747zaetkaxv4o@abyayala> (ng0@pragmatique.xyz's message of "Thu, 8 Jun 2017 08:39:35 +0000") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: 27284@debbugs.gnu.org Hello Guix! A heads-up to share the cognitive burden related to this topic. :-) So, we have two problems: compilation time, and memory consumption. I *think* I=E2=80=99ve identified one of the major causes for both in Guile, though it=E2=80=99s too early to say exactly how much this will impact reso= urce consumption for a full Guix compilation. See for details. When that is fixed, we=E2=80=99ll still have a performance problem: buildin= g all of Guix will still take more time than we=E2=80=99d like, and it won=E2=80= =99t get better as we add new files. So we need to address this. This has been discussed informally many times, and here=E2=80=99s a summary= of the ideas I=E2=80=99m aware of: 1. Build Guix as separate derivations: the first derivation builds the closure of (guix packages), the second one builds the closure of (guix scripts *), and finally we build (gnu *). We could also have derivations for the non-Scheme parts: the daemon, the manual, etc. (That amounts to compartmentalizing Guix in sub-packages, which in a way solves the bootstrapping issue that Pjotr complained about at some point=E2=80=94the fact that to build Guix one needs Guile, Guile-= JSON, GnuTLS, Autotools, etc.) The advantage is that when running =E2=80=98guix pull=E2=80=99 frequen= tly, you won=E2=80=99t have to recompile all of these. However, you=E2=80=99ll= almost always have to rebuild (gnu packages *), which is the longest part. 2. Build all of Guix like the =E2=80=98guix=E2=80=99 package does, and ho= pe that we can get a substitute. Bootstrapping issue: to do that, we first need compute the derivation of this new =E2=80=98guix=E2=80=99 package. Thus, we at le= ast need to build the closure of (guix packages), which should take a minute or so, after which we can compute the derivation, which could take a couple of minutes maybe. The problem is that building all of Guix (including running the test suite) takes some time, potentially more than the interval between two subsequent pushes to the repo. Thus, it=E2=80=99s quite l= ikely that the build farm would always be lagging behind. We could work around that by having the build farm automatically tag commits for which it has successfully built Guix. That would introduce delays in deploying the latest Guix to users, but maybe that can be short enough to be acceptable. 2b. To address the bootstrapping issue above, we also discussed the possibility of setting up a =E2=80=9Cmeta-derivation=E2=80=9D service:= you=E2=80=99d give it a Git commit, and it=E2=80=99d return the derivation of =E2=80=98gu= ix=E2=80=99 for that commit. Less computation would take place on the user side, but that doesn=E2=80=99t reduce the delay mentioned above. It also has the downside of introducing another service without which using Guix is more painful. I think that=E2=80=99s about it. Thoughts? Hacks? :-) Ludo=E2=80=99.