all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
From: ludo@gnu.org (Ludovic Courtès)
To: Maxim Cournoyer <maxim.cournoyer@gmail.com>
Cc: 27284@debbugs.gnu.org
Subject: bug#27284: Memory leak in 'guix pull' or 'make' in guix source
Date: Wed, 20 Sep 2017 13:42:39 +0200	[thread overview]
Message-ID: <878th9tx3k.fsf@gnu.org> (raw)
In-Reply-To: <87y3paccu5.fsf@gmail.com> (Maxim Cournoyer's message of "Tue, 19 Sep 2017 22:40:02 -0400")

Howdy,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:

[...]

>> This has been discussed informally many times, and here’s a summary of
>> the ideas I’m aware of:
>>
>>   1. Build Guix as separate derivations: the first derivation builds the
>
> [...]
>
>>
>>   2. Build all of Guix like the ‘guix’ package does, and hope that we
>>      can get a substitute.
>
> This seems like the best option to me. It strikes me as odd that Guix
> isn't distributed as its own substitutable package that can be
> efficiently downloaded and unpacked to the store, rather than the
> current intensive build-it-all process.

Yeah.

>>      Bootstrapping issue: to do that, we first need compute the
>>      derivation of this new ‘guix’ package.  Thus, we at least need to
>>      build the closure of (guix packages), which should take a minute or
>>      so, after which we can compute the derivation, which could take a
>>      couple of minutes maybe.
>>      The problem is that building all of Guix (including running the
>>      test suite) takes some time, potentially more than the interval
>>      between two subsequent pushes to the repo.  Thus, it’s quite likely
>>      that the build farm would always be lagging behind.
>
> Do we really have to build it *all*, every time? There are two
> optimizations which I can think of:
>
> 1. Only rebuild the files that changed.

That’s more or less what #1 would give us, though not necessarily at the
finest granularity.

> 2. Remove the --no-auto-compile arg from (guix scripts guix).
>
> I'm not sure why this is desirable. Is it because otherwise the cached
> compiled copies would be susceptible to ABI breakages?

Yes, that’s one thing.

Another thing is that on the first run of ‘guix package’ following ‘guix
pull’, you’d end up compiling everything, which is not much better than
compiling them upfront.

There’s also the option of not compiling (gnu packages *) and instead
evaluating them, but currently this is too costly in terms of memory and
CPU.

> Even with such optimizations it could still take more time to build Guix
> than the rate at which commits hit master. Maybe a cron job that could
> build the latest Guix every, say, 5 or 10 minutes (only when new commits
> appeared since the last time it was built)? We would then distribute the
> latest built Guix package when users would issue a 'guix pull'. There
> could be a 'guix pull' option to force building the package from the
> latest sources, for those who prefer the current behavior.

Yes, what I was suggesting was to build Guix at each push, and
automatically tag successful builds, such that ‘guix pull’ can pull up
to the latest successfully-built tag.  (Though we’d need to be careful:
if the auto-tagging machinery stops working, we still want users to get
the latest Guix so they have security updates.)

Thanks for chiming in!

Ludo’.

  reply	other threads:[~2017-09-20 15:53 UTC|newest]

Thread overview: 51+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-06-08  8:39 bug#27284: Memory leak in 'guix pull' or 'make' in guix source ng0
2017-06-08 15:02 ` ng0
2017-09-19 20:48 ` Ludovic Courtès
2017-09-20  2:40   ` Maxim Cournoyer
2017-09-20 11:42     ` Ludovic Courtès [this message]
2017-09-20 18:29       ` Maxim Cournoyer
2017-09-20 20:12         ` Ludovic Courtès
2017-09-21 14:43           ` Maxim Cournoyer
2017-09-23 18:14       ` Taylan Ulrich Bayırlı/Kammer
2017-09-24 19:44         ` Ludovic Courtès
2017-09-25 21:00           ` Maxim Cournoyer
2017-10-20 16:05   ` bug#27284: [PATCH 0/8] 'guix pull' creates several derivations Ludovic Courtès
2017-10-20 16:05     ` bug#27284: [PATCH 1/8] build: Factorize module compilation in (guix build compile) Ludovic Courtès
2017-10-22 21:22       ` Maxim Cournoyer
2017-10-23  1:50         ` Ludovic Courtès
2017-10-22 21:42           ` Eric Bavier
2017-10-23  2:51             ` Ludovic Courtès
2017-10-22 22:52               ` Eric Bavier
2017-10-23  5:10                 ` Ludovic Courtès
2017-10-20 16:05     ` bug#27284: [PATCH 2/8] build: Honor make's '-j' flag Ludovic Courtès
2017-10-20 16:05     ` bug#27284: [PATCH 3/8] discovery: Move 'file-name->module-name' to (guix modules) Ludovic Courtès
2017-10-20 16:05     ` bug#27284: [PATCH 4/8] gexp: Add 'file-union' Ludovic Courtès
2017-10-20 16:05     ` bug#27284: [PATCH 5/8] gexp: Add 'directory-union' Ludovic Courtès
2017-10-20 16:05     ` bug#27284: [PATCH 6/8] union: Parametrize the symlink procedure Ludovic Courtès
2017-10-20 16:05     ` bug#27284: [PATCH 7/8] gexp: 'directory-union' has a #:quiet? parameter Ludovic Courtès
2017-10-20 16:05     ` bug#27284: [PATCH 8/8] DRAFT Add (guix self) and use it when pulling Ludovic Courtès
2017-10-22 20:05       ` Maxim Cournoyer
2017-10-27 23:49         ` Ludovic Courtès
2017-11-21 22:26     ` bug#27284: [PATCH 0/8] 'guix pull' creates several derivations Ludovic Courtès
2017-11-21 22:56       ` Ludovic Courtès
2017-12-11 10:52         ` bug#27284: [PATCH 0/4] 'guix pull' reloads modules, second try Ludovic Courtès
2017-12-11 10:52           ` bug#27284: [PATCH 1/4] gnu: Fix ambiguous 'zip' reference Ludovic Courtès
2017-12-11 10:52           ` bug#27284: [PATCH 2/4] gexp: 'computed-file' has a new #:guile parameter Ludovic Courtès
2017-12-11 10:52           ` bug#27284: [PATCH 3/4] Add (guix self) and use it when pulling Ludovic Courtès
2017-12-18 14:57             ` Ludovic Courtès
2018-03-27  9:14               ` bug#27284: ‘guix pull’ builds using multiple derivations Ludovic Courtès
2018-03-27 14:33                 ` Ludovic Courtès
2018-03-27 19:25                 ` Nils Gillmann
2018-03-27 20:51                   ` Ludovic Courtès
2018-04-08 16:37                 ` Ludovic Courtès
2018-04-09 19:53                   ` Ricardo Wurmus
2018-04-10 21:53                     ` bug#27284: ‘guix pull’ broken on Guile 2.0 Ludovic Courtès
2018-04-10 23:18                       ` bug#31117: " Ludovic Courtès
2018-04-14 17:39                         ` Ricardo Wurmus
2017-12-11 10:52           ` bug#27284: [PATCH 4/4] pull: Reload modules before doing anything else Ludovic Courtès
2017-11-12 21:33   ` bug#27284: Memory leak in 'guix pull' or 'make' in guix source Ludovic Courtès
2017-11-13  8:59     ` Ricardo Wurmus
2017-11-13  9:28       ` Ludovic Courtès
2017-11-13 14:09         ` Ricardo Wurmus
2017-11-13 17:48           ` Ricardo Wurmus
2017-11-14  7:54           ` Ludovic Courtès

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=878th9tx3k.fsf@gnu.org \
    --to=ludo@gnu.org \
    --cc=27284@debbugs.gnu.org \
    --cc=maxim.cournoyer@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.