defining core modules

defining core modules

[Ricardo Wurmus]

Hi Guix,

for the past few days I’ve been trying to reduce the module closure of
“coreutils” by inspecting the output of

    guix graph -t module coreutils

This has shown a number of modules that are much too large and pull in
almost all other modules.

I’d like to propose a reduction of the modules to a core set.  To ensure
that they stay small we would move them to the directory
gnu/packages/core/.  Adding new module references to any of the modules
in that directory would only be permitted for very good reasons.

What do you think about separating these modules?

One place to start with is (gnu packages linux), which is huge.  (gnu
packages base) only needs libcap and linux-libre-headers, however.
These could be moved to gnu/packages/core/linux.scm.

Or we could give all the packages in the core set their own module.

(Let’s please not discuss moving all packages to their own modules.
This has been discussed elsewhere.)

-- 
Ricardo

Re: defining core modules

[Ludovic Courtès]

Hello,

Ricardo Wurmus <rekado@elephly.net> skribis:

> for the past few days I’ve been trying to reduce the module closure of
> “coreutils” by inspecting the output of
>
>     guix graph -t module coreutils

Much appreciated!

For the record, this is important for several reasons: it makes it
easier for (guix self) and thus ‘guix pull’ to build modules in smaller
chunks, and it means we can reduce I/O and memory usage, especially now
that the new cache is in place (see
<https://issues.guix.info/issue/34060>).

> This has shown a number of modules that are much too large and pull in
> almost all other modules.
>
> I’d like to propose a reduction of the modules to a core set.  To ensure
> that they stay small we would move them to the directory
> gnu/packages/core/.  Adding new module references to any of the modules
> in that directory would only be permitted for very good reasons.
>
> What do you think about separating these modules?

I support the idea; I’m not entirely sure about the core/ name space but
that’s a secondary issue.

It may prove to be tricky though.  For example, the set of dependencies
of GCC has been steadily growing over the last few years.  Those, in
turn, may pull in all sorts of C, C++, and Python packages.  So the
notion of “core” is really a moving target.

> One place to start with is (gnu packages linux), which is huge.  (gnu
> packages base) only needs libcap and linux-libre-headers, however.
> These could be moved to gnu/packages/core/linux.scm.

Right.  Initially linux.scm was for “kernel + Linux-specific packages”.
I think we should change it to have:

  • linux.scm for the kernel, header, ‘perf’, and little more.

  • linux-specific.scm (or similar) for Linux-specific packages (ALSA,
    PAM, libnl, btrfs, FUSE, etc.)

Thoughts?

> Or we could give all the packages in the core set their own module.

I’m not a big fan of it, and this would have to be applied recursively…

Alternately, it’d be interesting to visualize clusters in whole package
graph and guide module layout based on graph metrics, though I don’t
know exactly how.

Ludo’.

Re: defining core modules

[Ricardo Wurmus]

Ludovic Courtès <ludo@gnu.org> writes:

>> I’d like to propose a reduction of the modules to a core set.  To ensure
>> that they stay small we would move them to the directory
>> gnu/packages/core/.  Adding new module references to any of the modules
>> in that directory would only be permitted for very good reasons.
>>
>> What do you think about separating these modules?
>
> I support the idea; I’m not entirely sure about the core/ name space but
> that’s a secondary issue.
>
> It may prove to be tricky though.  For example, the set of dependencies
> of GCC has been steadily growing over the last few years.  Those, in
> turn, may pull in all sorts of C, C++, and Python packages.  So the
> notion of “core” is really a moving target.

Yeah, a minute after sending out this email I noticed the same problem :)

>> One place to start with is (gnu packages linux), which is huge.  (gnu
>> packages base) only needs libcap and linux-libre-headers, however.
>> These could be moved to gnu/packages/core/linux.scm.
>
> Right.  Initially linux.scm was for “kernel + Linux-specific packages”.
> I think we should change it to have:
>
>   • linux.scm for the kernel, header, ‘perf’, and little more.
>
>   • linux-specific.scm (or similar) for Linux-specific packages (ALSA,
>     PAM, libnl, btrfs, FUSE, etc.)
>
> Thoughts?

I agree.  I’m working on linux.scm already.  I wasted some time on
splitting guile.scm, which is difficult as package-for-guile2.0 cannot
be used.  This needs to be done as all these other Guile bindings pull
in many modules.  I wonder if we can get rid of the package-for-guile2.0
variants.

> Alternately, it’d be interesting to visualize clusters in whole package
> graph and guide module layout based on graph metrics, though I don’t
> know exactly how.

I’d love to have better visualisations as well.

One thing that helped me identify the next steps to change was to look
for modules that a) have been added for just a handful of packages they
provide and b) that depend on many other modules.

--
Ricardo

Re: defining core modules

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 3195 bytes --]

Hi,

On Mon, 28 Jan 2019 11:51:53 +0100
Ludovic Courtès <ludo@gnu.org> wrote:

> I support the idea; I’m not entirely sure about the core/ name space but
> that’s a secondary issue.

I support the idea, too.

> It may prove to be tricky though.  For example, the set of dependencies
> of GCC has been steadily growing over the last few years.

Yeah, it's worrying.  Minimalism is truly rare nowadays :(

> > One place to start with is (gnu packages linux), which is huge.  (gnu
> > packages base) only needs libcap and linux-libre-headers, however.
> > These could be moved to gnu/packages/core/linux.scm.  
> 
> Right.  Initially linux.scm was for “kernel + Linux-specific packages”.
> I think we should change it to have:
> 
>   • linux.scm for the kernel, header, ‘perf’, and little more.
> 
>   • linux-specific.scm (or similar) for Linux-specific packages (ALSA,
>     PAM, libnl, btrfs, FUSE, etc.)

In general I would always prefer that (I thought the python-xyz was
something like that for python).

There's a difference between "A" and "written using A" (even if it's an
extension to A), and it doesn't make much sense to conflate them.

For example, CPython is a C program and it doesn't make much sense to
put it with the Python programs (pypy, on the other hand, is a Python
program).  CPython won't use any of them.

So CPython itself is in gnu/packages/python.scm and the Python
modules are NOT in gnu/packages/python.scm, but for example in
gnu/packages/python-xyz.scm or more specific modules.

In your case, gnu/packages/linux.scm would be the Linux kernel and
gnu/packages/linux-xyz.scm would be packages that only work on the
Linux kernel (i.e. *clients* of the Linux kernel).

Otherwise, there are all kinds of circular dependencies (for example
for the build systems) which are just there because the build system
wants to use the compiler, but the packages using the build system are
in the same module as the compiler for no reason.  Worse, among the
packages using the build system are bindings for half the universe,
whose guile modules now also get pulled in.

Once a compiler starts to self-host, that's more difficult of course.

In the end, we move packages only when there's a good reason - but right
now, avoiding unresolvable dependency cycles is a serious drain on
everyone's time (I still feel kinda bad in the stomach whenever I add a
#:use-module -- even after I wrote a silly tool to find the cycles, most
of which are harmless and thus false positives).

That's in addition to the problem that packages pull in half the universe
if you aren't careful, which is much worse.

If there was a "core" namespace, at least one could be reasonably sure
that these don't pull in non-core modules (if we enforced that when
reviewing), making the module closure smaller and problems more
manageable.

I think it's worth a try to talk to upstream if there are outrageously
large or numerous dependencies to something and try to convince them
to reduce them.  After all, it only increases bug surface, including
security bug surface, to have many large dependencies (it's a cost).

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: defining core modules

[Ludovic Courtès]

Hi,

Danny Milosavljevic <dannym@scratchpost.org> skribis:

> On Mon, 28 Jan 2019 11:51:53 +0100
> Ludovic Courtès <ludo@gnu.org> wrote:

[...]

>> Right.  Initially linux.scm was for “kernel + Linux-specific packages”.
>> I think we should change it to have:
>> 
>>   • linux.scm for the kernel, header, ‘perf’, and little more.
>> 
>>   • linux-specific.scm (or similar) for Linux-specific packages (ALSA,
>>     PAM, libnl, btrfs, FUSE, etc.)
>
> In general I would always prefer that (I thought the python-xyz was
> something like that for python).
>
> There's a difference between "A" and "written using A" (even if it's an
> extension to A), and it doesn't make much sense to conflate them.
>
> For example, CPython is a C program and it doesn't make much sense to
> put it with the Python programs (pypy, on the other hand, is a Python
> program).  CPython won't use any of them.
>
> So CPython itself is in gnu/packages/python.scm and the Python
> modules are NOT in gnu/packages/python.scm, but for example in
> gnu/packages/python-xyz.scm or more specific modules.
>
> In your case, gnu/packages/linux.scm would be the Linux kernel and
> gnu/packages/linux-xyz.scm would be packages that only work on the
> Linux kernel (i.e. *clients* of the Linux kernel).

Agreed.

More generally, I think we should have LANGUAGE-CATEGORY.scm modules,
like {python,haskell}-{web,crypto}.scm, rather than catchall web.scm or
crypto.scm.

That way, at least for “non-scripting languages” (Haskell, OCaml, Rust,
etc.), the language packages would all leave in their own set of modules
that you won’t load unless you actually use the language.

(I’m drawing a distinction between “scripting languages”, namely Perl
and Python, and the rest, because those other languages tend to grow
package graphs that are very much independent from the typical GNU/Linux
C code base.)

Thanks,
Ludo’.