Re: Faster "guix pull" by incremental compilation and non-circular modules?

["Gábor Boskovits" <gboskovits@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3405 bytes --]

Hello Maxime,

Maxime Devos <maximedevos@telenet.be> ezt írta (időpont: 2022. febr. 28.,
H, 19:51):

> Ludovic Courtès schreef op ma 28-02-2022 om 14:17 [+0100]:
> > Hi,
> >
> > Maxime Devos <maximedevos@telenet.be> skribis:
> >
> > >   2. Instead of building all of Guix as a single derivation,
> > >      create a DAG of derivations.  More concretely:
> > >
> > >      First read the *.scm files to determine which module imports
> > >      which modules. Then to compile, say, (gnu packages acl),
> > >      a derivation taking gnu/packages/acl.scm and its dependencies
> > >      gnu/packages/attr.go, gnu/packages/base.go, ... is made
> > >      compiling gnu/packages/acl.scm to a gnu/packages/acl.go.
> > >
> > >      Then to build all of Guix, 'union-build' or 'file-union' is used.
> >
> > This is what (guix self), used by ‘guix pull’, is already doing.
> >
> > However, currently, package modules are split in just two groups: the
> > “base” group is the closure of (guix packages base), and the second
> > group has all the rest:
> >
> > [...]
>
> Looking at (guix self), it also has a few groups for non-package
> modules (system tests, scripts, ...).
>
> > At its core though, the situation pretty much reflects the free software
> > situation: there are low-level packages (glibc, GCC, GTK, etc.) that
> > might depend on high-level packages (Python, Pandoc, Rust, etc.).
> >
> > It’s not easy to split this spaghetti ball in smaller groups.
>
> It's not easy to manually split the spaghetti, but we don't have
> to, we could let the computer split the spaghetti for us (at least
> partially, because of the circular imports), by computing the graph of
> strongly-connected components and considering each SCC to be a ‘group’,
> some of which depend on other groups, forming a DAG.
>

I was thinking about a bit of a different structure that can also be
automated. My original idea was to use the already existing tree structure
of the derivations, and split it based on depth. I think that gives a bit
more structure, but might require splitting things that now are together
(for example iirc sometimes we are defining bootstrap packages inheriting
from the fully fledged ones, which introduces a syntactic dependency on
something that is  higher up the tree). Wdyt?

Regards,
g_bor


> I believe Ricardo Wurmus has some script for computing the SCC?
>
> Splitting large SCC in smaller parts can be left as an exercise
> for later, it's a somewhat orthogonal concern.
>
> > Thoughts?
>
> I think it would be nice to let (guix self) automatically determine the
> DAG of groups.  It would reduce the ad-hocness of the *...-modules*
> variables (some care required for patches, guix/man-db.scm, .js ...).
>
> It would also make the node tree wide, which could reduce memory usage
> (which might help with the ‘guix pull segfaults on i686-linux’
> reports).  In case of a crash (*), "guix pull" does not have to start
> over from scratch, which would also help with those reports.
>
> (*) This does not help with "failed to compute the derivation of Guix".
>
> Some work would be required, but I think it will be worth it, and it
> only has to be done once.
>
> TBC, this was just an idea I wanted to share, I won't be working on it
> in the forseeable future.
>
> Greetings,
> Maxime.
>

[-- Attachment #2: Type: text/html, Size: 4320 bytes --]