Contribute or create a channel?

Contribute or create a channel?

[Hartmut Goebel]

[-- Attachment #1: Type: text/plain, Size: 2087 bytes --]

Hi,

I'm currently updating Tryton to version 7.0 and am wondering whether 
it's better to contribute the change to Guix or to set up a channel for 
Tryton.

WDYT? I'm eager to learn about your thoughts.

Here is why I'm wondering:

  * Tryton consists of a client, a server and about 200 module/add-on
    providing business logic.
  *

    Tryton publishes a LTS version every 2.5 years. Two LTS versions are
    supported (currently 6.0 and 7.0) and bugfixes are backported there
    for 5 years.

  *

    Every 6 month a new release is crafted (x.2, x.4, x.6, x,8) which
    will get bugfixes for1 year. Releases typically provide new modules
    (which is why updating is of interest) , might change inputs and
    might require database updates.

  * Bugfixes happens rather often and per-module, since they are
    published even for smaller fixes. Upstream promises to not contain
    functional changes or change requirements. Each bugfix could be
    implemented as a graft, since .

Given this, it might be interesting to have three versions of Tryton 
available: the two LTS versions and the latest version.

Now the idea is to provide a channel which provides a branch for each 
LTS version and a "main" branch for the latest release. This would allow 
to checkout the respective branch and refresh the packages of the 
respective version semi-automatically.

OTOH in Guix, maintaining several version seems laborious.

Anyhow I'm unsure whether it's worth the effort maintaining three 
versions and whether I'll be able to keep three version up to date - 
esp. given that I don't have much automation for this.

Some more background-info:

  * Within each version, there is guarantee that the database schema
    will not be changed. Anyhow between versions the db schema might
    change, requiring manual migration steps.
  * Debian as of now provides packages for 6.0 only (7.0 was released )


-- 
Regards
Hartmut Goebel

| Hartmut Goebel          |h.goebel@crazy-compilers.com                |
|www.crazy-compilers.com  | compilers which you thought are impossible |

[-- Attachment #2: Type: text/html, Size: 2818 bytes --]

Re: Contribute or create a channel?

[Saku Laesvuori]

[-- Attachment #1: Type: text/plain, Size: 1875 bytes --]

> I'm currently updating Tryton to version 7.0 and am wondering whether 
> it's better to contribute the change to Guix or to set up a channel for 
> Tryton.

As a general rule: it is always better to contribute a change instead of
maintaining a separate channel for it if the change could be accepted in
Guix.

>   * Bugfixes happens rather often and per-module, since they are
>     published even for smaller fixes. Upstream promises to not contain
>     functional changes or change requirements. Each bugfix could be
>     implemented as a graft, since .

I don't think it would make much sense to implement bugfixes as grafts
if the package isn't depended on by a huge number of other packages.

> Given this, it might be interesting to have three versions of Tryton 
> available: the two LTS versions and the latest version.
> 
> Now the idea is to provide a channel which provides a branch for each 
> LTS version and a "main" branch for the latest release. This would allow 
> to checkout the respective branch and refresh the packages of the 
> respective version semi-automatically.
> 
> OTOH in Guix, maintaining several version seems laborious.

It just requires a different updating method. The different versions can
just be defined as separate packages (see postgresql for an example) and
the user the defines which one they want to use. They can either refer
to the package variable directly in scheme (e.g. postgresql-15) or on
the command line with the name@version syntax (e.g. postgresql@15).

> Some more background-info:
> 
>   * Within each version, there is guarantee that the database schema
>     will not be changed. Anyhow between versions the db schema might
>     change, requiring manual migration steps.

This is the case with postgresql, too. (which is why I chose it as the
example before)

- Saku

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Contribute or create a channel?

[Hartmut Goebel]

Am 01.03.24 um 19:16 schrieb Saku Laesvuori:
> It just requires a different updating method. The different versions can
> just be defined as separate packages (see postgresql for an example) and
> the user the defines which one they want to use. They can either refer
> to the package variable directly in scheme (e.g. postgresql-15) or on
> the command line with the name@version syntax (e.g. postgresql@15).

I'm aware of this way. It's fisable when there is a small number of 
packages like for postgres. But it is very laborious for rust packages 
already. I would not want to do the same for more than 200 packages.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Contribute or create a channel?

[Attila Lendvai]

> WDYT? I'm eager to learn about your thoughts.


the patch inflow to the guix repo is currently overwhelming the available capacity for review and pushing.

if you want an agile experience, i.e. where you can quickly fix/update this and that, then i suggest your own channel (unless you have the commit bit for the guix repo... or there's a committed maintainer who is a regular user, and as such will fast-track your patches).

otherwise you'll end up using a channel anyway (i.e. your fork of the guix repo while your patches are waiting in the queue to be reviewed and pushed).

PS: i don't mean to sound cynical here, just matter-of-factly.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Opportunity is missed by most people because it comes dressed in overalls and looks like work.”
	— Thomas A. Edison (1847–1931)

Re: Contribute or create a channel?

[Ricardo Wurmus]

Attila Lendvai <attila@lendvai.name> writes:

>> WDYT? I'm eager to learn about your thoughts.
>
> the patch inflow to the guix repo is currently overwhelming the
> available capacity for review and pushing.

With an email like the one sent by Hartmut we can better arrange for
shepherding this large submission.  (Nothing is to be gained from
repeatedly bemoaning well-known issues in the patch review processes
here and in other threads on the mailing list.)

-- 
Ricardo

Re: Contribute or create a channel?

[Attila Lendvai]

> > the patch inflow to the guix repo is currently overwhelming the
> > available capacity for review and pushing.
> 
> 
> With an email like the one sent by Hartmut we can better arrange for
> shepherding this large submission. (Nothing is to be gained from
> repeatedly bemoaning well-known issues in the patch review processes
> here and in other threads on the mailing list.)


i was reflecting on why i wrote this, and what i wanted to express is that i think guix has reached a point where a monorepo is becoming a net negative, and i don't see this being discussed.

my gut feeling is that new abstractions are needed that would enable splitting the monorepo/community into less tightly coupled subgroups where they can have their own coding standards, repos, channels, etc, and a more federated way to maintain/integrate all the software that exists out there into a guix system.

in this hypothetical setup commit rights could be issued much more liberally to non-core sub-repos, and more rigorous code reviews would only need to be done when a new version of the split-out part is being incorporated back into a new revision of the core/bootstrap chain (if e.g. assuming python is needed for the bootstrap of the core, then the python subgroup's stuff would only need core review when a new version of that is pointed to by the core).

or alternatively, simply try to split guix into a minimal core that is essential for the bootstrap, and everything else into multiple subchannels (gnome, gui stuff in general, random apps, etc). i have no impression how much that alone could shrink the monorepo part, though.

channels are a step towards this, but they are not enough in their current form to successfully accommodate for such a setup. an obvious thing that is missing is a way to formally express inter-channel dependencies, including some form of versioning.

sadly, i don't have any proposals beyond discussing the observable issue (i.e. the insufficient patch throughput).

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Values in a free society are accepted voluntarily, not through coercion, and certainly not by law… every time we write a law to control private behavior, we imply that somebody has to arrive with a gun [to enforce it].”
	— Ron Paul

Re: Contribute or create a channel?

[Troy Figiel]

[-- Attachment #1.1.1: Type: text/plain, Size: 2724 bytes --]

Hi Attila,

I know this wasn't directly sent to me, but I wanted to share a couple
of thoughts anyway :-)

On 2024-03-12 14:18, Attila Lendvai wrote:
> my gut feeling is that new abstractions are needed that would enable splitting the monorepo/community into less tightly coupled subgroups where they can have their own coding standards, repos, channels, etc, and a more federated way to maintain/integrate all the software that exists out there into a guix system.

As an end user, it is incredibly nice to have a single Guix channel to
pull from.  I just have to remember (or write down...) a single commit
hash and that's all that's required to reproduce my setup.

> channels are a step towards this, but they are not enough in their current form to successfully accommodate for such a setup. an obvious thing that is missing is a way to formally express inter-channel dependencies, including some form of versioning.

Although I understand where you are coming from, I am not convinced that
this solves more problems than it causes.  For example, a naive approach
to inter-channel dependencies with versioning could reintroduce the
diamond dependency problem.  However, solving this by locking every
single transitive dependent channel would not allow security updates
across all of them simultaneously anymore.

> in this hypothetical setup commit rights could be issued much more liberally to non-core sub-repos, and more rigorous code reviews would only need to be done when a new version of the split-out part is being incorporated back into a new revision of the core/bootstrap chain (if e.g. assuming python is needed for the bootstrap of the core, then the python subgroup's stuff would only need core review when a new version of that is pointed to by the core).

The same topic has come up in Nix as well [1, 2, 3], but they are using
Github, so they are bound to a centralized workflow.  Using Github (or
any such git forge for that matter) reintroduces some of the same
problems Git was designed to solve in the first place, which become most
painful at scale [4].

From a technical point of view, I don't see why we couldn't split into
less tightly coupled subgroups with their own repos, standards and CI
merging into a single Guix "monorepo".  I think there is still a lot to
learn from how the kernel does its development and we could extrapolate
to something that makes sense for Guix.

Best wishes,

Troy

[1]
https://discourse.nixos.org/t/wild-idea-how-about-splitting-nixpkgs-and-nixos/11487/25
[2] https://github.com/NixOS/rfcs/pull/70
[3] https://github.com/NixOS/nixpkgs/issues/50105
[4] https://blog.ffwll.ch/2017/08/github-why-cant-host-the-kernel.html

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 6367 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Contribute or create a channel?

[Carlo Zancanaro]

On Tue, Mar 12 2024, Attila Lendvai wrote:
> channels are a step towards this, but they are not enough in their
> current form to successfully accommodate for such a setup. an obvious
> thing that is missing is a way to formally express inter-channel
> dependencies, including some form of versioning.

Do we not have this? The manual documents a mechanism for channel
dependencies in "(guix) Declaring Channel Dependencies".

I haven't used it, but it looks like the dependencies are declared as
channels, which can have the usual branch/commit specifications to tie
them to specific versions.

What are we missing?

Carlo

Re: Contribute or create a channel?

[Attila Lendvai]

> > channels are a step towards this, but they are not enough in their
> > current form to successfully accommodate for such a setup. an obvious
> > thing that is missing is a way to formally express inter-channel
> > dependencies, including some form of versioning.
> 
> 
> Do we not have this? The manual documents a mechanism for channel
> dependencies in "(guix) Declaring Channel Dependencies".
> 
> I haven't used it, but it looks like the dependencies are declared as
> channels, which can have the usual branch/commit specifications to tie
> them to specific versions.

good point, thanks!

i looked briefly at the code just now. it's not trivial, and it seems to treat the guix channel specially (because i don't need to specify it as a dependency in my channel's .guix-channel file), and i'm not sure how it behaves when e.g. two channels depend on the same channel, but pick two different commits... or all the other convoluted situations.

the reason i assumed it doesn't exist is that i've never seen it used by any channels that i looked at.


> What are we missing?


i guess it's time to experiment to be able to answer your question.

FTR, it's READ-CHANNEL-METADATA and friends in guix/channels.scm

note that it's not the same thing as /etc/guix/channels.scm, even though they appear similar (https://issues.guix.gnu.org/53657).

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“People who have never gone to school have never developed negative attitudes toward exploring their world.”
	— Grace Llewellyn

Re: Contribute or create a channel?

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]

Hi Attila,

Attila Lendvai <attila@lendvai.name> writes:

>> > channels are a step towards this, but they are not enough in their
>> > current form to successfully accommodate for such a setup. an obvious
>> > thing that is missing is a way to formally express inter-channel
>> > dependencies, including some form of versioning.
>> 
>> 
>> Do we not have this? The manual documents a mechanism for channel
>> dependencies in "(guix) Declaring Channel Dependencies".
>> 
>> I haven't used it, but it looks like the dependencies are declared as
>> channels, which can have the usual branch/commit specifications to tie
>> them to specific versions.
>
> good point, thanks!
>
> i looked briefly at the code just now. it's not trivial, and it seems to treat the guix channel specially (because i don't need to specify it as a dependency in my channel's .guix-channel file), and i'm not sure how it behaves when e.g. two channels depend on the same channel, but pick two different commits... or all the other convoluted situations.

The situation you describe where two channels pick different commits for
a channel dependency *cannot* be made to work, at least using Guile.  I
think what would currently happen is that either:

1) Guix refuses to build (unlikely);
2) Guix just chooses one commit that it then uses to build all dependent
channels, contrary to the requirements;
3) Guix does build each channel with the proper dependencies, but the
final Guix will have profile collisions.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Contribute or create a channel?

[Ricardo Wurmus]

Hi Hartmut,

> I'm currently updating Tryton to version 7.0 and am wondering whether it's better to contribute the change to Guix or to set up a
> channel for Tryton.
>
> WDYT? I'm eager to learn about your thoughts.
>
> Here is why I'm wondering:
>
> * Tryton consists of a client, a server and about 200 module/add-on providing business logic.
> * Tryton publishes a LTS version every 2.5 years. Two LTS versions are supported (currently 6.0 and 7.0) and bugfixes are backported
>  there for 5 years.

I think it would be preferable to have one LTS version in Guix.  If you
want to commit to maintaining the packages I think it would be worth
discussing whether to grant you commit rights to handle these upgrades
by yourself.

> OTOH in Guix, maintaining several version seems laborious.

What makes you say this?  What exactly is the obstacle here?

> Anyhow I'm unsure whether it's worth the effort maintaining three versions and whether I'll be able to keep three version up to date -
> esp. given that I don't have much automation for this.

Personally, I don't think it makes sense to commit to maintaining there
separate versions.  It seems like a really big committment.

-- 
Ricardo

Re: Contribute or create a channel?

[Hartmut Goebel]

Hi Ricardo,
> I think it would be preferable to have one LTS version in Guix.
Okay, that's a clear statement.

> whether to grant you commit rights to handle these upgrades
> by yourself.

Well, I already have commit rights :-) Anyhow the review period is a bit 
hindering, esp. for bug-fixes.


>> OTOH in Guix, maintaining several version seems laborious.
> What makes you say this?  What exactly is the obstacle here?

The only way I know for handling several versions of the same package is 
what Saku Laesvuori described: Adding a new variable for each of these 
versions, possibly making the older version inherit from the new one. 
This means moving quite some code on every new version. Done this for 
quite some rust-packages and it i quite laborious and hard to automate. 
And for Tryton we have about 200 packages and growing.

When using branches (in a channel) one would just checkout that branch, 
automatically apply the patches and commit. If the version is no longer 
supported, simply stop applying updates on that branch.

Maybe using one file per release (and accept duplicate code) would be a 
suitable workaround.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Contribute or create a channel?

[Andreas Enge]

Am Sat, Mar 02, 2024 at 11:32:37AM +0100 schrieb Hartmut Goebel:
> Maybe using one file per release (and accept duplicate code) would be a
> suitable workaround.

I think that would be okay if you think it will be easier to maintain
(not needing to "roll over" code from an old package at the inheritance
root when it is deleted), assuming that the old packages are removed
from time to time.

Andreas

Re: Contribute or create a channel?

[Hartmut Goebel]

Am 04.03.24 um 09:32 schrieb Andreas Enge:
> I think that would be okay if you think it will be easier to maintain
> (not needing to "roll over" code from an old package at the inheritance
> root when it is deleted), assuming that the old packages are removed
> from time to time.

This sounds like maintaining multiple LTS versions is desired. Anyhow 
Ricardo wrote:

> I think it would be preferable to have one LTS version in Guix.

Thus the discussion about maintaining several versions is only relevant 
if this is what Guix wants.

In both cases I need some tooling to fetch the current bug-fix version 
of the series in question. This can not be done using "guix refresh" 
only AFAIU, as this would use the next release-series if this is already 
released while our packages are not yet updated. Thus maintaining two 
LTS versions should not be too much work (except if many dependencies 
change incompatible).

WDYT?

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: Contribute or create a channel?

[Ricardo Wurmus]

Hartmut Goebel <h.goebel@crazy-compilers.com> writes:

> In both cases I need some tooling to fetch the current bug-fix version
> of the series in question. This can not be done using "guix refresh"
> only AFAIU, as this would use the next release-series if this is
> already released while our packages are not yet updated.

"guix refresh" can be modified, and more specific importers / updaters
can be added to better suit the needs of groups of packages.

-- 
Ricardo