From mboxrd@z Thu Jan 1 00:00:00 1970 From: ludo@gnu.org (Ludovic =?UTF-8?Q?Court=C3=A8s?=) Subject: bug#27284: Memory leak in 'guix pull' or 'make' in guix source Date: Wed, 20 Sep 2017 13:42:39 +0200 Message-ID: <878th9tx3k.fsf@gnu.org> References: <20170608083935.izw747zaetkaxv4o@abyayala> <87poamv2i7.fsf@gnu.org> <87y3paccu5.fsf@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:59282) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1duhJE-0001gI-0u for bug-guix@gnu.org; Wed, 20 Sep 2017 11:53:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1duhJ9-0001Ln-8Q for bug-guix@gnu.org; Wed, 20 Sep 2017 11:53:07 -0400 Received: from debbugs.gnu.org ([208.118.235.43]:41698) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1duhJ8-0001KJ-De for bug-guix@gnu.org; Wed, 20 Sep 2017 11:53:02 -0400 Sender: "Debbugs-submit" Resent-Message-ID: In-Reply-To: <87y3paccu5.fsf@gmail.com> (Maxim Cournoyer's message of "Tue, 19 Sep 2017 22:40:02 -0400") List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+gcggb-bug-guix=m.gmane.org@gnu.org Sender: "bug-Guix" To: Maxim Cournoyer Cc: 27284@debbugs.gnu.org Howdy, Maxim Cournoyer skribis: > ludo@gnu.org (Ludovic Court=C3=A8s) writes: [...] >> This has been discussed informally many times, and here=E2=80=99s a summ= ary of >> the ideas I=E2=80=99m aware of: >> >> 1. Build Guix as separate derivations: the first derivation builds the > > [...] > >> >> 2. Build all of Guix like the =E2=80=98guix=E2=80=99 package does, and= hope that we >> can get a substitute. > > This seems like the best option to me. It strikes me as odd that Guix > isn't distributed as its own substitutable package that can be > efficiently downloaded and unpacked to the store, rather than the > current intensive build-it-all process. Yeah. >> Bootstrapping issue: to do that, we first need compute the >> derivation of this new =E2=80=98guix=E2=80=99 package. Thus, we at= least need to >> build the closure of (guix packages), which should take a minute or >> so, after which we can compute the derivation, which could take a >> couple of minutes maybe. >> The problem is that building all of Guix (including running the >> test suite) takes some time, potentially more than the interval >> between two subsequent pushes to the repo. Thus, it=E2=80=99s quit= e likely >> that the build farm would always be lagging behind. > > Do we really have to build it *all*, every time? There are two > optimizations which I can think of: > > 1. Only rebuild the files that changed. That=E2=80=99s more or less what #1 would give us, though not necessarily a= t the finest granularity. > 2. Remove the --no-auto-compile arg from (guix scripts guix). > > I'm not sure why this is desirable. Is it because otherwise the cached > compiled copies would be susceptible to ABI breakages? Yes, that=E2=80=99s one thing. Another thing is that on the first run of =E2=80=98guix package=E2=80=99 fo= llowing =E2=80=98guix pull=E2=80=99, you=E2=80=99d end up compiling everything, which is not much= better than compiling them upfront. There=E2=80=99s also the option of not compiling (gnu packages *) and inste= ad evaluating them, but currently this is too costly in terms of memory and CPU. > Even with such optimizations it could still take more time to build Guix > than the rate at which commits hit master. Maybe a cron job that could > build the latest Guix every, say, 5 or 10 minutes (only when new commits > appeared since the last time it was built)? We would then distribute the > latest built Guix package when users would issue a 'guix pull'. There > could be a 'guix pull' option to force building the package from the > latest sources, for those who prefer the current behavior. Yes, what I was suggesting was to build Guix at each push, and automatically tag successful builds, such that =E2=80=98guix pull=E2=80=99 = can pull up to the latest successfully-built tag. (Though we=E2=80=99d need to be care= ful: if the auto-tagging machinery stops working, we still want users to get the latest Guix so they have security updates.) Thanks for chiming in! Ludo=E2=80=99.