Stratification of GNU Guix into Independent Channels

Stratification of GNU Guix into Independent Channels

[jgart]

Hi Guixers,

I wanted to ask you what are your thoughts on this idea. This is a thought experiment at this stage.

Should GNU Guix be a small core of packages (and services?)?

For example, GNU Guix would be a core channel containing the reduced binary seed bootstrap and a few other core packages to bootstrap the system. That's it. Users subscribe to this channel to get started.

Users could then decide what channels they'd like to subscribe to/opt in to by adding any of the following channels as they please:

python-channel
rust-channel
lisp-channel
node-channel
ocaml-channel
clojure-channel
chicken-channel
go-channel
haskell-channel
erlang-channel
elixir channel (maybe elixir and erlang are together in one channel)
elm-channel
gaming-channel
texlive-channel
ruby-channel
linux-channel (kernel)
java-channel
emacs-channel
vim-channel
perl-channel
php-channel
julia-channel
r-channel
dlang-channel
purescript-channel
mozilla-channel
chromium-channel
browsers-channel
coq-channel
agda-channel
idris-channel
kde-channel
gnome-channel
xfce-channel
enlightenment-channel
pantheon-channel
scheme-channel
etc...

The above channels would still be maintained under the auspices of GNU.

What do you think would be the pros and cons of the stratified approach versus the monorepo approach that we currently have?

This is mostly a thought experiment as implementing something like this, I imagine, would be quite ambitious and radically different from the approach we are currently following. 

I'm curious to know what the community thinks about the stratified approach. I'm also curious to learn more about the history of why we chose the monorepo approach. Are there any references that I can read for the latter?

Here are some pros I can foresee with the stratified approach.

Pros
=========

GNU Guix proper would be a solid core of packages that is very easy to maintain. This would greatly reduce the maintenance burden since maintaining a world of rust, golang, or python packages is opt in by those who want to do that particular work.

Users can opt out of certain channels. For example, there might be some users that want absolutely no rust or some other language or ecosystem of packages from being downloaded on their machine for whatever reason. Same goes for any other languages.

Would being subscribed to just the hypothetical small core channel in this proposal increase download/installation speeds given the availability of substitutes?

WDYT

Re: Stratification of GNU Guix into Independent Channels

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 2513 bytes --]

On 2022-12-24, jgart wrote:
> I wanted to ask you what are your thoughts on this idea. This is a
> thought experiment at this stage.
>
> Should GNU Guix be a small core of packages (and services?)?
>
> For example, GNU Guix would be a core channel containing the reduced
> binary seed bootstrap and a few other core packages to bootstrap the
> system. That's it. Users subscribe to this channel to get started.

I definitely am excited and hopeful to think this *could* be possible!


> Users could then decide what channels they'd like to subscribe to/opt
> in to by adding any of the following channels as they please:
>
> python-channel
> rust-channel
...
> scheme-channel
> etc...
>
> The above channels would still be maintained under the auspices of GNU.
>
> What do you think would be the pros and cons of the stratified
> approach versus the monorepo approach that we currently have?

The pros would be only having to pull the small bits you actually want,
and that could be faster to run "guix pull".

This would certainly make it much easier to package a "core" subset of
guix for other distros (e.g. I work on packaging guix in Debian) that
having to package the whole entire guix universe.

I suspect the "small bootstrap core set" to eventually pull in quite a
bit of other things, and continually be at risk of needing to pull in
more things...

You would probably need to have inter-channel dependencies... which
makes me wonder how many cyclic dependencies might result...

git alone pulls in dependencies for perl, python, git, subversion,
openssl ... and what is the dependency cycle for each of those? Guix
apparently can generate pretty charts for this stuff, though I have yet
to try. :)

I have the (perhaps mininformed) impression nix has a split more along
these lines with the core of nix being one thing, and then collections
of packages being other repositories.


> GNU Guix proper would be a solid core of packages that is very easy to
> maintain. This would greatly reduce the maintenance burden since
> maintaining a world of rust, golang, or python packages is opt in by
> those who want to do that particular work.

If it is possible, it seems likely to make releasing more regularly and
less stressfully...


> Would being subscribed to just the hypothetical small core channel in
> this proposal increase download/installation speeds given the
> availability of substitutes?

I suspect it would significantly increase guix pull speeds, at the very
least.


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Stratification of GNU Guix into Independent Channels

[Joshua Branson]

Vagrant Cascadian <vagrant@debian.org> writes:

> On 2022-12-24, jgart wrote:
>> I wanted to ask you what are your thoughts on this idea. This is a
>> thought experiment at this stage.
>>
>> Should GNU Guix be a small core of packages (and services?)?
>>
>> For example, GNU Guix would be a core channel containing the reduced
>> binary seed bootstrap and a few other core packages to bootstrap the
>> system. That's it. Users subscribe to this channel to get started.
>
> I definitely am excited and hopeful to think this *could* be possible!
>
>
>> Users could then decide what channels they'd like to subscribe to/opt
>> in to by adding any of the following channels as they please:
>>
>> python-channel
>> rust-channel
> ...
>> scheme-channel
>> etc...
>>
>> The above channels would still be maintained under the auspices of GNU.
>>
>> What do you think would be the pros and cons of the stratified
>> approach versus the monorepo approach that we currently have?
>
> The pros would be only having to pull the small bits you actually want,
> and that could be faster to run "guix pull".
>
> This would certainly make it much easier to package a "core" subset of
> guix for other distros (e.g. I work on packaging guix in Debian) that
> having to package the whole entire guix universe.
>
> I suspect the "small bootstrap core set" to eventually pull in quite a
> bit of other things, and continually be at risk of needing to pull in
> more things...
>
> You would probably need to have inter-channel dependencies... which
> makes me wonder how many cyclic dependencies might result...
>
> git alone pulls in dependencies for perl, python, git, subversion,
> openssl ... and what is the dependency cycle for each of those? Guix
> apparently can generate pretty charts for this stuff, though I have yet
> to try. :)
>

Fun fact of the day, the OpenBSD crowd is creating got, which is a
rewrite of git, with less features, in C.  :)

>
> I have the (perhaps mininformed) impression nix has a split more along
> these lines with the core of nix being one thing, and then collections
> of packages being other repositories.
>
>
>> GNU Guix proper would be a solid core of packages that is very easy to
>> maintain. This would greatly reduce the maintenance burden since
>> maintaining a world of rust, golang, or python packages is opt in by
>> those who want to do that particular work.
>
> If it is possible, it seems likely to make releasing more regularly and
> less stressfully...
>
>
>> Would being subscribed to just the hypothetical small core channel in
>> this proposal increase download/installation speeds given the
>> availability of substitutes?
>
> I suspect it would significantly increase guix pull speeds, at the very
> least.
>
>
> live well,
>   vagrant

Re: Stratification of GNU Guix into Independent Channels

[jgart]

I think we also need a channel manager subcommand:

guix channel add python
guix channel add rust
guix channel add gaming
guix channel add rde
guix channel add guixrus
guix channel add non****
etc.

The above commands would subscribe you to a given trusted Guix channel.

The last three are channels not under the auspices of GNU but I think that this `guix channel` subcommand should allow you to extend the Guix "channel webring" as you like.

Just another (un)related idea that I'd like to see someday.

all best,

jgart

Stratification of GNU Guix into Independent Channels, X-mas and Guix days!

[Pjotr Prins]

tis like packages under a X-mas tree

guix xmas add games
guix xmas add tex
guix xmas add paint
guix xmas add payamas

unlike debian these packages contain packages that can overlap too!

that would be interesting.

happy holidays Guix and thank you community! Hope to see you all at
Guix days in Brussels 2+3 Feb. Do sign up if you plan to come. We will
be there!

https://libreplanet.org/wiki/Group:Guix/FOSDEM2023

Pj.

On Sat, Dec 24, 2022 at 05:24:08AM +0000, jgart wrote:
> I think we also need a channel manager subcommand:
> 
> guix channel add python
> guix channel add rust
> guix channel add gaming
> guix channel add rde
> guix channel add guixrus
> guix channel add non****
> etc.
> 
> The above commands would subscribe you to a given trusted Guix channel.
> 
> The last three are channels not under the auspices of GNU but I think that this `guix channel` subcommand should allow you to extend the Guix "channel webring" as you like.
> 
> Just another (un)related idea that I'd like to see someday.
> 
> all best,
> 
> jgart
> 

Re: Stratification of GNU Guix into Independent Channels, X-mas and Guix days!

[indieterminacy]

On 24-12-2022 09:37, Pjotr Prins wrote:
> tis like packages under a X-mas tree
> 
> guix xmas add games
> guix xmas add tex
> guix xmas add paint
> guix xmas add payamas

I just guix pulled version 1.4 out of a cracker.
That will make for a happy new year

Thanks all!

Happy holidays everybody!


Jonathan

Re: Stratification of GNU Guix into Independent Channels, X-mas and Guix days!

[jgart]

Goldfish Crackers are on Rollback at Walmart for $5.98!

Merry Guixmas and a Happy GNU Year!!

Re: Stratification of GNU Guix into Independent Channels

[Liliana Marie Prikler]

Am Samstag, dem 24.12.2022 um 03:49 +0000 schrieb jgart:
> Hi Guixers,
> 
> I wanted to ask you what are your thoughts on this idea. This is a
> thought experiment at this stage.
> 
> Should GNU Guix be a small core of packages (and services?)?
No, it should be a functional operating system.

> What do you think would be the pros and cons of the stratified
> approach versus the monorepo approach that we currently have?
I think monorepo is a weird term to apply to Guix.  Yes, it is just a
single repository, but notably, it doesn't collect the sources of every
other package out there to put it into a single tree.  Instead, it
contains a recipe per package.  This makes it no different from Debian,
Arch, Gentoo, or any other package manager out there.

> GNU Guix proper would be a solid core of packages that is very easy
> to maintain. This would greatly reduce the maintenance burden since
> maintaining a world of rust, golang, or python packages is opt in by
> those who want to do that particular work.
No, it wouldn't.  Suppose this bootstrap Guix consisted only of
packages up to GCC.  Then any change to it would result in non-local
changes (and potential breakages) in every other channel in existence
and the local CI would never catch that.  As it currently stands, GCC
breaking some other package would be caught on core-updates.

> Users can opt out of certain channels. For example, there might be
> some users that want absolutely no rust or some other language or
> ecosystem of packages from being downloaded on their machine for
> whatever reason. Same goes for any other languages.
You can not opt out of Rust if you depend on librsvg.  In future, you
might not even be able to opt out of Rust when using the Linux kernel.
Yes, this is possibly the worst timeline to live in, but we need
solutions to actually deal with it, not burying our heads in sand.

> Would being subscribed to just the hypothetical small core channel in
> this proposal increase download/installation speeds given the
> availability of substitutes?
Not really, because for a functional system you would have to pull in
other channels.  With a stricter module layout we could possibly
modularize guix pull further, thus decreasing individual derivation
time, but that alone is a daunting task and not guaranteed to bring big
improvements.

What does work is convincing upstreams to pull in less dependencies and
drop the outdated ones, because that makes it so that eventually Guix
has to ship less packages.

Cheers

Re: Stratification of GNU Guix into Independent Channels

[Josselin Poiret]

Hi jgart,

IMO, having everything in one repo, while going against the "modularity"
philosophy, really helps a lot when updating packages.  In my experience
with the Other Channel That We Should Not Talk About, there often are
breaking changes in the main repository that lead to the other channel
being broken for a couple of days.  It would take way longer for changes
to propagate things to the "leaf" channels.

Also, you can't reasonably expect programs to be categorized that well
(although I'm pretty sure there could be consensus on non-PITA
vs. PITA-packages with the usual suspects :) ).

I don't think there is a way to reconcile the "complete system
distribution" and "completely modular distribution" viewpoints, at least
at that level.  The channels approach that exists for now is a nice
pragmatic solution that still allows quite a lot of freedom.

Best,
-- 
Josselin Poiret

Re: Stratification of GNU Guix into Independent Channels

[Hartmut Goebel]

Am 24.12.22 um 04:49 schrieb jgart:
> Should GNU Guix be a small core of packages (and services?)?

No. As others already stated, this would complicate things for users. 
Having all available software at on place is a big plus. (For many 
GNU/Linux distros you need to add another ppa for many „standard“ cases. 
IMHO this is disgusting)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Stratification of GNU Guix into Independent Channels

[Denis 'GNUtoo' Carikli]

[-- Attachment #1: Type: text/plain, Size: 5081 bytes --]

On Sat, 24 Dec 2022 03:49:30 +0000
"jgart" <jgart@dismail.de> wrote:

> Hi Guixers,
Hi,

> Should GNU Guix be a small core of packages (and services?)?
I think it would also make sense to consider first find the features we
want, and then look at how to get them, to see if that works better for
us.

For instance if the goal is to get faster updates, there might be
other things that could be done to improve the speed way more than
reducing the number of packages in the repository. 

For instance making sure that there are substitutes might improve the
speed a lot more. It might also be possible to optimize Guile more for
some tasks. For instance:
- Copying files on filesystems that have deduplication could be
  improved.
- Maybe it is possible to substitute more things (like the things it
  builds when we do guix pull) or at least rebuilding only some of them
  (the ones that changed between the last git revision and the new
  one). 
- It might also be possible to combine information from Guix weather
  with the current system or packages to see on which revision the
  packages ca be upgraded without having to recompile anything (unless
  we use non-substitutable packages) and update to that revision
  automatically.
- etc.

For some of these ideas, we would first need to benchmark somehow what
is taking time during updates though, to make sure that the
optimization really yields some gains.

In my case I've already a clear use case (basically a non-substitutable
package that consist of a huge data file) where optimizing Guile will
yield a lot of performance improvements.

If the idea is (also) to avoid using certain packages, it might make
more sense to have a system to do that inside the current Guix source
code. 

For instance a user might want to make sure not to ship compiled zfs
compiled kernel modules, though right now it's not a big issue as you
need to manually select either zfs-auto-snapshot or zfs to get it, no
other packages depend on it. So users usually won't accidentally install
it.

Another idea would be to have some flag to have only free culture
licensed packages (like to avoid cc-nd non-functional data).

Though I think that if we do something like that, it would complicate
things a lot. Since Guix is meant to be reproducible, we would need to
find a way to handle that kind of configuration in a reproducible way.

So far the configurability has been integrated in ways that are not
hidden for users. For instance if you use a package transformation, you
are aware of it. Packages specific to CPU Microarchitectures[1][2] also
need to be used specifically, and if I understand users often use
package transformation for that.

An easier way here would probably be to teach tools like guix lint to
enable to check certain things, for instance with 'guix lint
--extra-check=./my-checks.scm', or just to write a new tool or external
tools for that. We can already use the Guix repl in guile scripts.

Though here it would still be up to users to run the checks and try to
fix things (for instance by using package transformations), but at
least it would not complicate things in Guix.

As for weather or not switching to a layered approach would improve
development, some projects like openembedded/yocto did that. I've no
idea why exactly they did it and what turned out to work well or not for
their use cases, but to me the result looks really complex[3],
especially if we want to do something like that with Guix.

For instance we'd need to handle this layered approach in all the
infrastructure, tools, and so on. Users would also need to know where
to send patches, and if we try to autodetect that in the tools that
receive the patches, it might becomes error prone. And we'd also need
to make sure that layers stay compatible with each other. If everything
is split, that's harder to do.

I think that one of the big strength of Guix is precisely the fact that
everything is well integrated together. And as far as I understand,
here having a single repository makes it much easier to do that.

As I understand Guix maintainers do care about giving some time
to third party packages or channels to adjust for changes[4], though
with a single repository we can more easily do changes and test them
than if the changes would be scattered around multiple repositories.

A single repository also ensures that everything stay consistent
without much effort.

References:
-----------
[1]https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/maths.scm?id=fddf1dc3aba3176b6efc9e0be0918245665a6ebf#n2762
[2]https://hpc.guix.info/blog/2018/01/pre-built-binaries-vs-performance/
[3]https://layers.openembedded.org/layerindex/branch/master/layers/
[4]"We recommend waiting until the next Guix release is out though,
   which could be a month from now, so that your channel remains usable
   by those who pinned an older revision of the guix channel."[5]
[5]https://guix.gnu.org/en/blog/2021/the-big-change/

Denis.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Stratification of GNU Guix into Independent Channels

[zimoun]

Hi,

On Sat, 24 Dec 2022 at 03:49, "jgart" <jgart@dismail.de> wrote:

> Users could then decide what channels they'd like to subscribe to/opt
> in to by adding any of the following channels as they please: 
>
> python-channel
> rust-channel
[...]
> etc...
>
> The above channels would still be maintained under the auspices of
> GNU. 
>
> What do you think would be the pros and cons of the stratified
> approach versus the monorepo approach that we currently have?

The Big cons is about maintenance.

It is possible to have a look at the complete current state [1] but it
would be hard, if not impossible, when using many many many channels.
When using other channels [2], time to time it breaks.  Somehow, you
have a combinatorial problem.

Consider that some Git tools are part of one channel and for instance
Julia packages are part of another channel.  The update of Git from its
channel could have an impact on the packages defined in some other
channels; for concrete case: Git update from 2.38.0 to 2.38.1 breaks
julia-documenter [3].  Using few channels, it makes doable to tackle
such issues (guix refresh, guix graph, etc.).  Using many many channels,
it makes it hard to detect; although CI and QA are improving a lot and
for sure they can help, the situation is not good enough yet, IMHO.

Moreover, many channels would be dependant from one to the other.  Give
a look at (gnu packages python-xyz).  If the packages defined here would
become the channel ’python-channel’ then this ’python-channel’ would
depend on many others – this (gnu packages python-xyz) module requires
103 other (gnu packages …) modules.

Concretely, the split would be a painful, boring and tedious work where
I am doubtful about the practical outcome.

From my point of view, solutions for improving the situation of “guix
pull” are:

 + more modular; maybe split *package-modules* in (guix self)
 
 + transform subcommands as extensions; here we would have a better
   control for the Guix feature dependencies via the Scheme API


1: <http://ci.guix.gnu.org/eval/78350/dashboard>
2: <https://hpc.guix.info/channels/>
3: <https://yhetil.org/guix/20221205153710.10684-1-zimon.toutoune@gmail.com>

Cheers,
simon

Re: Stratification of GNU Guix into Independent Channels

[Giovanni Biscuolo]

[-- Attachment #1: Type: text/plain, Size: 1923 bytes --]

Hi,

just a quick comment

zimoun <zimon.toutoune@gmail.com> writes:

[...]

> Moreover, many channels would be dependant from one to the other.

and this would be **a nightmare** to maintain (as already clearly stated
by others much more competent than me in Guix-things)

to recap: all PROS that you jgart mentioned in his original message does
not apply (or are "reversed" to CONS) and users would not benefit of
this kind of modularity in terms of speed (likely speed would be much
worse with more than 4 channels to pull in, and 4 channels would be too
few for a complete system)

as far as I can understand from my journey in Guix, it has the **great**
advantage to /expose/ the weaknesses of some (some?) upstream packages
not so... diligent with dependency hygiene, leading to the **big**
problem of dependency hell or dependency confusion, that sooner or later
/someone/ (distro mainatienrs often) have to solve *before* including
that package in the distribution (unlike what is happening with PyPI,
npm and alike: good luck users!)

as Liliana said:

--8<---------------cut here---------------start------------->8---

 What does work is convincing upstreams to pull in less dependencies and
 drop the outdated ones, because that makes it so that eventually Guix
 has to ship less packages.

--8<---------------cut here---------------end--------------->8---

Unfortunately I have the impression that not so many upstream developers
are really aware of the problem and when they are they think it's a
problem of someone else, not their.

Minimalism at (Guix) System level, one package at a time.

Happy hacking! Gio'

[...]

P.S.: the elevator pitch of all this thread could be: «yes, software
systems are way too /complicated/... but please don't blame Guix for it
(instead "git blame" and bless it!)» :-D

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]