Core-updates coordination and plans

Core-updates coordination and plans

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

Hi everyone,

Since we're a couple people working on core-updates at the same time, it
might be a good idea to coordinate a bit (of course, other volunteers
welcome :) ).

One conundrum we have for now: glibc 2.38 has a couple of new CVEs, and
we have three options:
1) change glibc to track the 2.38 release branch → world rebuild.
2) graft glibc → bad user experience (and we're not supposed to graft
outside of master).
3) switch to 2.39 → world rebuild + possibly more work fixing new build
failures.

glibc 2.39 should hopefully release tomorrow (01/02/2024)

What is everyone's opinion regarding those?

IMO, option 2 is the one I'd like to avoid, and between 1 and 3 I'd
ideally prefer 3 but we don't know yet if there is going to be a lot of
breakage because of that (feels like usually it's toolchain updates, not
glibc updates that cause them the most).

Also, I see that most of the patches that were requested to be merged
into c-u (like the big pages for jemalloc) actually got pushed, are
there any other (well-tested) ones we can go for at the same time as the
glibc rebuild?  I will stress that this is about *core* packages now,
I feel that it's too late to introduce more complications and delay the
update any longer.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Core-updates coordination and plans

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Josselin,

On Wed, Jan 31 2024, Josselin Poiret wrote:

> One conundrum we have for now: glibc 2.38 has a couple of new CVEs

The authors describe CVE-2023-6246, which is probably the most serious
of the four vulnerabilities, as "significant" [1] and Red Bat ranks it
as "8.4 (HIGH)" under the new CVD scale. [2]

Most important, "This flaw allows local privilege escalation, enabling
an unprivileged user to gain full root access, as demonstrated in Fedora
38." [again, 1]

> we have three options:
> 1) change glibc to track the 2.38 release branch → world rebuild.
> 2) graft glibc → bad user experience (and we're not supposed to graft
> outside of master).
> 3) switch to 2.39 → world rebuild + possibly more work fixing new build
> failures.

Personally, I would go straight to Glibc 2.39.

I am not sure it's helpful to obsess about "world rebuilds." The fear
and cost of rebuilding in Guix is real, but it is a necessary
consequence of the way Guix works. The fear is also a dominant and
persistent mental obstacle to making Guix better. [3]

> I'd ideally prefer 3 but we don't know yet if there is going to be a
> lot of breakage

Your argument that more work may be needed is well-placed, however.. We
won't know until we get there. Perhaps we can revert to version 2.38
prior to your merge if the problems are severe.

> glibc 2.39 should hopefully release tomorrow (01/02/2024)

Fortunately, your merge schedule conincides more or less with Glib's
release cycle. There should be plenty of data from around the world
about the new Glibc version's performance in two or three weeks.

> most of the patches ... got pushed, are there any other ... ?

I don't know whether eudev is a core-package but I personally find it
irresponsible that my patch to fix the use of MAC-based addresses for
network interfaces [4][5] has not been accepted in some form.

In a functional OS like Guix, no administrator should rely on device
enumerations provided by the BIOS, by UEFI or by the Linux kernel. The
fix is a one-liner. [6]

I used 'regexp-quote' to avoid Guile's quoting madness. [7] As an
alternative, we could quote the literals with the old Perl::Critic trick
that avoids escapes for metacharacters: [8]

  (substitute* "src/udev/Makefile.am"
    (("[$][(]udevrulesdir[)]") "/etc/udev/rules.d"))

Either way, I'd appreciate if Eudev could please be fixed in this
core-updates cycle. The fix has been available for more than half a
year.

The fix was furthermore deployed on all my systems, including the one
running the test deployment of Debbugs in Guix, which is available at
debbugs.juix.org. The fix works. Thanks!

Kind regards
Felix

[1] https://blog.qualys.com/vulnerabilities-threat-research/2024/01/30/qualys-tru-discovers-important-vulnerabilities-in-gnu-c-librarys-syslog
[2] https://nvd.nist.gov/vuln/detail/CVE-2023-6246
[3] https://lists.gnu.org/archive/html/guix-devel/2024-01/msg00243.html
[4] https://issues.guix.gnu.org/63508
[5] https://issues.guix.gnu.org/63787
[6] https://issues.guix.gnu.org/63508#12-lineno51
[7] https://www.gnu.org/software/guile/manual/html_node/Backslash-Escapes.html
[8] https://metacpan.org/pod/Perl::Critic::Policy::RegularExpressions::ProhibitEscapedMetacharacters

Re: Core-updates coordination and plans

[Vivien Kraus]

Dear guix,

Le mercredi 31 janvier 2024 à 10:19 -0800, Felix Lechner via
Development of GNU Guix and the GNU System distribution. a écrit :
> Either way, I'd appreciate if Eudev could please be fixed in this
> core-updates cycle. The fix has been available for more than half a
> year.

The eudev package has been touched on gnome-team (unmerged yet, sorry
this is taking time) a few months ago with the following commits:

d6462be6a8..498db4de1f

I submitted these changes against gnome-team because we needed to
update libgudev, but in retrospect, it might belong to core-updates. I
don’t know if these changes will work with a non-updated libgudev.

If we start changing eudev in divergent branches, then it might be a
headache to merge. Could you try and rebase your work on gnome-team, to
see whether it works?

Best regards,

Vivien

Eudev in gnome-team [Was: Core-updates coordination and plans]

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Vivien,

> The eudev package has been touched on gnome-team

I reviewed the patches but do not believe they solve Bug#63508 or
Bug#63787.

That issue, stated succinctly, arises because configure.ac sets the
Makefile variable $(udevrulesdir) to the store output [1] while custom
rules like mine [2] live in the global folder /etc/udev/rules.d. After
some analysis, the upstream sources are best patched here. [3]

A explanation for why the flags to ./configure are ineffective can be
found here. [4]

> Could you try and rebase your work on gnome-team, to see whether it
> works?

Unfortunately, that rebuild takes two days on my system. Absent an
insight that your patches will fix the bug, it is too much effort.

Kind regards
Felix

[1] https://github.com/eudev-project/eudev/blob/611b6fbae2cca9ceeacb820befb3a0e8caec88b8/configure.ac#L180
[2] https://codeberg.org/lechner/system-config/src/commit/a6962d1414d11040072f0e79ffde354fa523137a/service/udev-rules-net-name-mac.scm#L4-L9
[3] https://github.com/eudev-project/eudev/blob/611b6fbae2cca9ceeacb820befb3a0e8caec88b8/src/udev/Makefile.am#L10
[4] https://issues.guix.gnu.org/63508#26

Re: Core-updates coordination and plans

[Andreas Enge]

Hello,

Am Wed, Jan 31, 2024 at 05:44:21PM +0100 schrieb Josselin Poiret:
> One conundrum we have for now: glibc 2.38 has a couple of new CVEs, and
> we have three options:
> 1) change glibc to track the 2.38 release branch → world rebuild.
> 2) graft glibc → bad user experience (and we're not supposed to graft
> outside of master).
> 3) switch to 2.39 → world rebuild + possibly more work fixing new build
> failures.

I would go for whatever fixes the security problems (obviously) and leads
to a faster merge. Probably 1), which, if I understand correctly, means
using a newer point release or git commit of glibc-2.38 that fixes the CVEs
and has a low chance of fundamentally breaking things. Then in the spirit
of feature branches, we could have a feature branch "core-team" (not
"core-updates"!; well, I do not care about the names, but about the mental
idea we attach to them) updating glibc.

2.39 might delay the merge for more months if things do not go well, but
you are probably better placed to judge how big the risks are of a lot
of breakage.

I am also not that worried about world-rebuilds: We should be able to do
a world-rebuild not once or twice a year, but at least every month, say,
or maybe every week. If this is not the case currently, it is an infra-
structure problem that we should try to address. (Relatedly, we should
ungraft more often; there are now packages with over 100 grafts, and in
updating the system behind a fast internet line, grafting ends up taking
a non-negligible proportion of the total time, even on an SSD.)

In any case, thanks for your work on getting things in shape!

Andreas

Re: Core-updates coordination and plans

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Andreas,

On Wed, Jan 31 2024, Andreas Enge wrote:

> We should be able to do a world-rebuild not once or twice a year, but
> at least every month

How about a 48-hour period every month in which commits are permitted
even if they cause "world rebuilds"?

We could pause the substitute builders during that period. It would get
rid of core-updates forever.

Kind regards
Felix

Re: Core-updates coordination and plans

[Andreas Enge]

Hello Felix,

Am Mon, Feb 26, 2024 at 07:26:57AM -0800 schrieb Felix Lechner:
> How about a 48-hour period every month in which commits are permitted
> even if they cause "world rebuilds"?
> We could pause the substitute builders during that period. It would get
> rid of core-updates forever.

a time-based approach sounds like a good idea indeed; if just for things
like ungrafting, which are considered extremely low risk. It might still
be good to do it in a separate branch instead of master, and to merge it
after substitutes are available. Since "guix pull" takes the latest commit
from the master branch, users could otherwise end up with a world-rebuild
commit without substitutes.

So maybe we could have a time window, but also discuss and prepare before-
hand which big changes we would like to push?

Having a team or a dedicated individual (in both senses of the terms,
a designated person with a lot of dedication) to shepherd this through
would also be good.

Andreas

Re: Core-updates coordination and plans

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

On 2024-02-27, Andreas Enge wrote:
> Am Mon, Feb 26, 2024 at 07:26:57AM -0800 schrieb Felix Lechner:
>> How about a 48-hour period every month in which commits are permitted
>> even if they cause "world rebuilds"?
>> We could pause the substitute builders during that period. It would get
>> rid of core-updates forever.
>
> a time-based approach sounds like a good idea indeed; if just for things
> like ungrafting, which are considered extremely low risk. It might still
> be good to do it in a separate branch instead of master, and to merge it
> after substitutes are available. Since "guix pull" takes the latest commit
> from the master branch, users could otherwise end up with a world-rebuild
> commit without substitutes.
>
> So maybe we could have a time window, but also discuss and prepare before-
> hand which big changes we would like to push?

Or a similarly "short" time period where commits to master are blocked,
so that substitutes being built for branches-to-be-merged are not
chasing a moving target?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Core-updates coordination and plans

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Andreas,

On Tue, Feb 27 2024, Andreas Enge wrote:

> a time-based approach sounds like a good idea

How about the second Monday and Tuesday of every month?  That is a slow
time for contributors who have more time on weekends.

> It might still be good to do it in a separate branch instead of
> master, and to merge it after substitutes are available.

While it is good practice to build and test changes, I have a better
proposal for a better availability of substitutes, below.

> Since "guix pull" takes the latest commit from the master branch,
> users could otherwise end up with a world-rebuild commit without
> substitutes.

In a nod to Vagrant's remark, I would accept the "moving target" and
instead publish the substitute coverage for each commit.

A good algorithm could figure out from a user's system configuration
which most recent commit provides all the required packages.

For more general use, a "prebuilt" branch could track commits that
provide 100% universal coverage.  That should be our goal, even if it
sounds elusive today.

> So maybe we could have a time window, but also discuss and prepare before-
> hand which big changes we would like to push?

Please have hope humankind.  Why not a free for all?  Most people will
feel uncomfortable with the freedom, anyway.  The fear of messing up
will drive them to the mailing lists.

> Having a team or a dedicated individual (in both senses of the terms,
> a designated person with a lot of dedication) to shepherd this through
> would also be good.

I like the idea of you taking charge! You would bring experience,
credibility and an agreeable personality---all in nearly perfect
harmony. Are you available?

Kind regards
Felix

Re: Core-updates coordination and plans

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 1285 bytes --]

On 2024-01-31, Josselin Poiret wrote:
> One conundrum we have for now: glibc 2.38 has a couple of new CVEs, and
> we have three options:
> 1) change glibc to track the 2.38 release branch → world rebuild.
> 2) graft glibc → bad user experience (and we're not supposed to graft
> outside of master).
> 3) switch to 2.39 → world rebuild + possibly more work fixing new build
> failures.
>
> glibc 2.39 should hopefully release tomorrow (01/02/2024)
>
> What is everyone's opinion regarding those?
>
> IMO, option 2 is the one I'd like to avoid, and between 1 and 3 I'd
> ideally prefer 3 but we don't know yet if there is going to be a lot of
> breakage because of that (feels like usually it's toolchain updates, not
> glibc updates that cause them the most).

How about doing *both* 1 and 3 ... more world rebuilding, sure, but it
means we will have the information to make the correct decision about
which to merge into master. It is *possible* that 2.39 is no worse or
even less broken than the 2.38 release, even if it is likely that 2.38
is more well tested... and even if 2.38 ends up being the glibc to merge
to master, things are better positioned to fairly quickly switch to 2.39
once it becomes a more known quantity...

live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Core-updates coordination and plans

[Vivien Kraus]

Hello,

Le mercredi 31 janvier 2024 à 17:44 +0100, Josselin Poiret a écrit :
> Also, I see that most of the patches that were requested to be merged
> into c-u (like the big pages for jemalloc) actually got pushed, are
> there any other (well-tested) ones we can go for at the same time as
> the
> glibc rebuild?  I will stress that this is about *core* packages now,
> I feel that it's too late to introduce more complications and delay
> the
> update any longer.

Is it too late to ask if you could cherry-pick the commits that show up
in the d6462be6a8..b701a7018d range (from gnu: eudev: Update to 3.2.14.
to gnu: upower: Update to 1.90.2. inclusive)? We have them on gnome-
team, but maybe it would rather belong in core-updates.

What do you think?

Best regards,

Vivien

Re: Core-updates coordination and plans

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 1232 bytes --]

Hi again everyone,

Thanks for the feedback!  So in the meantime I chose to go ahead and try
with 2.39 (how hard could it be?).

The main visible change for us in 2.39 is the removal of the crypt
library, with a replacement being the external libxcrypt.  This wasn't
too bad to fix in most places, you can find that work on
core-updates-glibc-2.39.  I am able to build up to Gnome, but trying to
build the desktop.tmpl vm image, I stumbled upon a quite annoying issue:
Guix recommends using Guile's (crypt ...) function, which is included
only if libcrypt is available during the build.  However, Guile appears
very early in the dependency graph, and you can't just add libxcrypt to
its dependencies without causing some cycle.

Should commencement's guile-final be a minimal version of Guile without
libcrypt, while guile-3.0 includes it?  Do we want to be able to use
(crypt ...) in g-exps?  I don't really know what the answer to those
questions should be, but it probably involves a world rebuild :(.
Anyone have an idea/opinion about this conundrum?

Other than that, I think that branch is in a pretty ok shape, I've also
included a couple of patches that were requested.

WDYT?
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Core-updates coordination and plans

[Ludovic Courtès]

Hello!

Josselin Poiret <dev@jpoiret.xyz> skribis:

> Thanks for the feedback!  So in the meantime I chose to go ahead and try
> with 2.39 (how hard could it be?).

I’m glad you did that; at the same time, I feel like it’s delayed the
merge.  Maybe in the future we need a calendar-based schedule for
‘core-updates’ to help make decisions like this?

> Should commencement's guile-final be a minimal version of Guile without
> libcrypt, while guile-3.0 includes it?  Do we want to be able to use
> (crypt ...) in g-exps?  I don't really know what the answer to those
> questions should be, but it probably involves a world rebuild :(.
> Anyone have an idea/opinion about this conundrum?

As discussed on IRC, the ‘crypt’ procedure is documented in the Guix
manual as a way to provide an initial user account password, so it “has
to” be available in ‘guile-final’.

> Other than that, I think that branch is in a pretty ok shape, I've also
> included a couple of patches that were requested.

Yay!

Ludo’.