Re: [PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[ludo@gnu.org (Ludovic Courtès)]

Hi,

Mark H Weaver <mhw@netris.org> skribis:

> Ricardo Wurmus <rekado@elephly.net> writes:
>
>> Hi Ludo,
>>
>>>> Here are some important items I can think of:
>> […]
>>>>   • Guile 2.2 compiler terrible issue:
>>>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.
>>
>> One way to side-step this issue for the upcoming release is to use one
>> Guile process per file when running “guix pull”.  This will make it run
>> a lot slower, but that would be better than the current situation.
>
> I've attached a workaround that I've been using for the last 6 weeks on
> my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
> otherwise it would not be able to successfully build the 'guix' package.
>
> Note that I never use 'guix pull', so I'm not sure off-hand whether this
> solves the problem there, but it certainly greatly reduces the memory
> needed to run 'make' and thus to build the 'guix' package.
>
> This patch modifies build-aux/compile-all.scm to work as follows: after
> loading all modules in the parent process, it then forks off a child and
> compiles 20 modules in parallel within that child while the parent
> waits.  When the child is finished compiling those 20 modules, the child
> exits (thus freeing the memory leaked during compilation), and then the
> parent spawns a new child to compile the next 20 modules, and so on,
> until all the modules are compiled.
>
> We should probably consider applying this to master.  Thoughts?

That sounds like a good idea.  If it works, I’m all for it.

We’ll need to do something similar in (guix scripts pull).

I’ve been working on having ‘guix pull’ build modules as several
derivations as part of <https://bugs.gnu.org/27284>.  I’ll try to post a
prototype hopefully later today.

Thanks,
Ludo’.