* defining core modules @ 2019-01-27 21:20 Ricardo Wurmus 2019-01-28 10:51 ` Ludovic Courtès 0 siblings, 1 reply; 5+ messages in thread From: Ricardo Wurmus @ 2019-01-27 21:20 UTC (permalink / raw) To: guix-devel Hi Guix, for the past few days I’ve been trying to reduce the module closure of “coreutils” by inspecting the output of guix graph -t module coreutils This has shown a number of modules that are much too large and pull in almost all other modules. I’d like to propose a reduction of the modules to a core set. To ensure that they stay small we would move them to the directory gnu/packages/core/. Adding new module references to any of the modules in that directory would only be permitted for very good reasons. What do you think about separating these modules? One place to start with is (gnu packages linux), which is huge. (gnu packages base) only needs libcap and linux-libre-headers, however. These could be moved to gnu/packages/core/linux.scm. Or we could give all the packages in the core set their own module. (Let’s please not discuss moving all packages to their own modules. This has been discussed elsewhere.) -- Ricardo ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: defining core modules 2019-01-27 21:20 defining core modules Ricardo Wurmus @ 2019-01-28 10:51 ` Ludovic Courtès 2019-01-28 11:30 ` Ricardo Wurmus 2019-01-28 12:19 ` Danny Milosavljevic 0 siblings, 2 replies; 5+ messages in thread From: Ludovic Courtès @ 2019-01-28 10:51 UTC (permalink / raw) To: Ricardo Wurmus; +Cc: guix-devel Hello, Ricardo Wurmus <rekado@elephly.net> skribis: > for the past few days I’ve been trying to reduce the module closure of > “coreutils” by inspecting the output of > > guix graph -t module coreutils Much appreciated! For the record, this is important for several reasons: it makes it easier for (guix self) and thus ‘guix pull’ to build modules in smaller chunks, and it means we can reduce I/O and memory usage, especially now that the new cache is in place (see <https://issues.guix.info/issue/34060>). > This has shown a number of modules that are much too large and pull in > almost all other modules. > > I’d like to propose a reduction of the modules to a core set. To ensure > that they stay small we would move them to the directory > gnu/packages/core/. Adding new module references to any of the modules > in that directory would only be permitted for very good reasons. > > What do you think about separating these modules? I support the idea; I’m not entirely sure about the core/ name space but that’s a secondary issue. It may prove to be tricky though. For example, the set of dependencies of GCC has been steadily growing over the last few years. Those, in turn, may pull in all sorts of C, C++, and Python packages. So the notion of “core” is really a moving target. > One place to start with is (gnu packages linux), which is huge. (gnu > packages base) only needs libcap and linux-libre-headers, however. > These could be moved to gnu/packages/core/linux.scm. Right. Initially linux.scm was for “kernel + Linux-specific packages”. I think we should change it to have: • linux.scm for the kernel, header, ‘perf’, and little more. • linux-specific.scm (or similar) for Linux-specific packages (ALSA, PAM, libnl, btrfs, FUSE, etc.) Thoughts? > Or we could give all the packages in the core set their own module. I’m not a big fan of it, and this would have to be applied recursively… Alternately, it’d be interesting to visualize clusters in whole package graph and guide module layout based on graph metrics, though I don’t know exactly how. Ludo’. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: defining core modules 2019-01-28 10:51 ` Ludovic Courtès @ 2019-01-28 11:30 ` Ricardo Wurmus 2019-01-28 12:19 ` Danny Milosavljevic 1 sibling, 0 replies; 5+ messages in thread From: Ricardo Wurmus @ 2019-01-28 11:30 UTC (permalink / raw) To: Ludovic Courtès; +Cc: guix-devel Ludovic Courtès <ludo@gnu.org> writes: >> I’d like to propose a reduction of the modules to a core set. To ensure >> that they stay small we would move them to the directory >> gnu/packages/core/. Adding new module references to any of the modules >> in that directory would only be permitted for very good reasons. >> >> What do you think about separating these modules? > > I support the idea; I’m not entirely sure about the core/ name space but > that’s a secondary issue. > > It may prove to be tricky though. For example, the set of dependencies > of GCC has been steadily growing over the last few years. Those, in > turn, may pull in all sorts of C, C++, and Python packages. So the > notion of “core” is really a moving target. Yeah, a minute after sending out this email I noticed the same problem :) >> One place to start with is (gnu packages linux), which is huge. (gnu >> packages base) only needs libcap and linux-libre-headers, however. >> These could be moved to gnu/packages/core/linux.scm. > > Right. Initially linux.scm was for “kernel + Linux-specific packages”. > I think we should change it to have: > > • linux.scm for the kernel, header, ‘perf’, and little more. > > • linux-specific.scm (or similar) for Linux-specific packages (ALSA, > PAM, libnl, btrfs, FUSE, etc.) > > Thoughts? I agree. I’m working on linux.scm already. I wasted some time on splitting guile.scm, which is difficult as package-for-guile2.0 cannot be used. This needs to be done as all these other Guile bindings pull in many modules. I wonder if we can get rid of the package-for-guile2.0 variants. > Alternately, it’d be interesting to visualize clusters in whole package > graph and guide module layout based on graph metrics, though I don’t > know exactly how. I’d love to have better visualisations as well. One thing that helped me identify the next steps to change was to look for modules that a) have been added for just a handful of packages they provide and b) that depend on many other modules. -- Ricardo ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: defining core modules 2019-01-28 10:51 ` Ludovic Courtès 2019-01-28 11:30 ` Ricardo Wurmus @ 2019-01-28 12:19 ` Danny Milosavljevic 2019-01-28 14:25 ` Ludovic Courtès 1 sibling, 1 reply; 5+ messages in thread From: Danny Milosavljevic @ 2019-01-28 12:19 UTC (permalink / raw) To: Ludovic Courtès, Ricardo Wurmus; +Cc: guix-devel [-- Attachment #1: Type: text/plain, Size: 3195 bytes --] Hi, On Mon, 28 Jan 2019 11:51:53 +0100 Ludovic Courtès <ludo@gnu.org> wrote: > I support the idea; I’m not entirely sure about the core/ name space but > that’s a secondary issue. I support the idea, too. > It may prove to be tricky though. For example, the set of dependencies > of GCC has been steadily growing over the last few years. Yeah, it's worrying. Minimalism is truly rare nowadays :( > > One place to start with is (gnu packages linux), which is huge. (gnu > > packages base) only needs libcap and linux-libre-headers, however. > > These could be moved to gnu/packages/core/linux.scm. > > Right. Initially linux.scm was for “kernel + Linux-specific packages”. > I think we should change it to have: > > • linux.scm for the kernel, header, ‘perf’, and little more. > > • linux-specific.scm (or similar) for Linux-specific packages (ALSA, > PAM, libnl, btrfs, FUSE, etc.) In general I would always prefer that (I thought the python-xyz was something like that for python). There's a difference between "A" and "written using A" (even if it's an extension to A), and it doesn't make much sense to conflate them. For example, CPython is a C program and it doesn't make much sense to put it with the Python programs (pypy, on the other hand, is a Python program). CPython won't use any of them. So CPython itself is in gnu/packages/python.scm and the Python modules are NOT in gnu/packages/python.scm, but for example in gnu/packages/python-xyz.scm or more specific modules. In your case, gnu/packages/linux.scm would be the Linux kernel and gnu/packages/linux-xyz.scm would be packages that only work on the Linux kernel (i.e. *clients* of the Linux kernel). Otherwise, there are all kinds of circular dependencies (for example for the build systems) which are just there because the build system wants to use the compiler, but the packages using the build system are in the same module as the compiler for no reason. Worse, among the packages using the build system are bindings for half the universe, whose guile modules now also get pulled in. Once a compiler starts to self-host, that's more difficult of course. In the end, we move packages only when there's a good reason - but right now, avoiding unresolvable dependency cycles is a serious drain on everyone's time (I still feel kinda bad in the stomach whenever I add a #:use-module -- even after I wrote a silly tool to find the cycles, most of which are harmless and thus false positives). That's in addition to the problem that packages pull in half the universe if you aren't careful, which is much worse. If there was a "core" namespace, at least one could be reasonably sure that these don't pull in non-core modules (if we enforced that when reviewing), making the module closure smaller and problems more manageable. I think it's worth a try to talk to upstream if there are outrageously large or numerous dependencies to something and try to convince them to reduce them. After all, it only increases bug surface, including security bug surface, to have many large dependencies (it's a cost). [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: defining core modules 2019-01-28 12:19 ` Danny Milosavljevic @ 2019-01-28 14:25 ` Ludovic Courtès 0 siblings, 0 replies; 5+ messages in thread From: Ludovic Courtès @ 2019-01-28 14:25 UTC (permalink / raw) To: Danny Milosavljevic; +Cc: guix-devel Hi, Danny Milosavljevic <dannym@scratchpost.org> skribis: > On Mon, 28 Jan 2019 11:51:53 +0100 > Ludovic Courtès <ludo@gnu.org> wrote: [...] >> Right. Initially linux.scm was for “kernel + Linux-specific packages”. >> I think we should change it to have: >> >> • linux.scm for the kernel, header, ‘perf’, and little more. >> >> • linux-specific.scm (or similar) for Linux-specific packages (ALSA, >> PAM, libnl, btrfs, FUSE, etc.) > > In general I would always prefer that (I thought the python-xyz was > something like that for python). > > There's a difference between "A" and "written using A" (even if it's an > extension to A), and it doesn't make much sense to conflate them. > > For example, CPython is a C program and it doesn't make much sense to > put it with the Python programs (pypy, on the other hand, is a Python > program). CPython won't use any of them. > > So CPython itself is in gnu/packages/python.scm and the Python > modules are NOT in gnu/packages/python.scm, but for example in > gnu/packages/python-xyz.scm or more specific modules. > > In your case, gnu/packages/linux.scm would be the Linux kernel and > gnu/packages/linux-xyz.scm would be packages that only work on the > Linux kernel (i.e. *clients* of the Linux kernel). Agreed. More generally, I think we should have LANGUAGE-CATEGORY.scm modules, like {python,haskell}-{web,crypto}.scm, rather than catchall web.scm or crypto.scm. That way, at least for “non-scripting languages” (Haskell, OCaml, Rust, etc.), the language packages would all leave in their own set of modules that you won’t load unless you actually use the language. (I’m drawing a distinction between “scripting languages”, namely Perl and Python, and the rest, because those other languages tend to grow package graphs that are very much independent from the typical GNU/Linux C code base.) Thanks, Ludo’. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-01-28 14:26 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-01-27 21:20 defining core modules Ricardo Wurmus 2019-01-28 10:51 ` Ludovic Courtès 2019-01-28 11:30 ` Ricardo Wurmus 2019-01-28 12:19 ` Danny Milosavljevic 2019-01-28 14:25 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).