Status of ‘core-updates’

Status of ‘core-updates’

[Ludovic Courtès]

Hello!

What’s the status of ‘core-updates’?  What are the areas where help is
needed?

I know a lot has happened since the last update¹, which is roughly when
I dropped the ball due to other commitments, but I’m not sure where we
are now.

Thanks,
Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2024-01/msg00096.html

Re: Status of ‘core-updates’

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> What’s the status of ‘core-updates’?  What are the areas where help is
> needed?

Disclaimer: I've been quite busy with work recently and haven't been
able to work on core-updates that much (having to build the world
locally doesn't help).

Currently, the desktop system image doesn't build anymore because we've
also integrated patches to use pkgconf instead of pkg-config.  I think
the effort to actually make that work with all of our packages is going
to be a bit too much to handle right now.  IMO, we should probably
revert/branch out before then so that we can push core-updates past the
finish line.  A feature branch to integrate pkgconf properly could then
be worked on later.

Other than that, I was able to get a working desktop before the
aforementioned patches but with some locale errors that I fixed with
ae07bc2dd0124b625acf70e594ccc90d6d128562, so I'm not expecting too much
trouble.  We'll probably have to add libxcrypt as an input to some other
packages that I haven't tested yet, but that's quite trivial.

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Status of ‘core-updates’

[Ludovic Courtès]

Hello!

Josselin Poiret <dev@jpoiret.xyz> skribis:

> Disclaimer: I've been quite busy with work recently and haven't been
> able to work on core-updates that much (having to build the world
> locally doesn't help).

No problem.  We should find someone willing to pick up the coordination
work for the coming month or so.  Any volunteer?  :-)

To be clear (but I guess it’s crystal clear to anyone who’s been around
long enough :-)), what we need most is someone to keep track of changes,
coordinate efforts, decide what goes in the branch and what’s postponed
or moved to a separate branch, and send periodic (weekly) status updates
over the course of a couple of months.  This can (and probably should)
be done without doing any actual hacking on the branch.

> Currently, the desktop system image doesn't build anymore because we've
> also integrated patches to use pkgconf instead of pkg-config.  I think
> the effort to actually make that work with all of our packages is going
> to be a bit too much to handle right now.  IMO, we should probably
> revert/branch out before then so that we can push core-updates past the
> finish line.  A feature branch to integrate pkgconf properly could then
> be worked on later.

Last time we discussed it on IRC, you were in the process of putting up
a branch without those pkgconf changes, IIRC.  Were you able to complete
that work?

If not, I guess that’ll be the first thing to do, unless someone has a
better idea.

> Other than that, I was able to get a working desktop before the
> aforementioned patches but with some locale errors that I fixed with
> ae07bc2dd0124b625acf70e594ccc90d6d128562, so I'm not expecting too much
> trouble.  We'll probably have to add libxcrypt as an input to some other
> packages that I haven't tested yet, but that's quite trivial.

Alright.

Thanks for all the work so far!

Ludo’.

Re: Status of ‘core-updates’

[Steve George]

On 10 Apr, Ludovic Courtès wrote:
> Hello!
> 
> Josselin Poiret <dev@jpoiret.xyz> skribis:
> 
> > Disclaimer: I've been quite busy with work recently and haven't been
> > able to work on core-updates that much (having to build the world
> > locally doesn't help).
> 
> No problem.  We should find someone willing to pick up the coordination
> work for the coming month or so.  Any volunteer?  :-)
> 
> To be clear (but I guess it’s crystal clear to anyone who’s been around
> long enough :-)), what we need most is someone to keep track of changes,
> coordinate efforts, decide what goes in the branch and what’s postponed
> or moved to a separate branch, and send periodic (weekly) status updates
> over the course of a couple of months.  This can (and probably should)
> be done without doing any actual hacking on the branch.
(...)

This sounds like something I can do:

- track changes (in branch)
- co-ordinate blocking issues (in bug system)
- co-ordinate with people doing the actual work :-P
- send (periodic) weekly emails
- encourage shipping by minimising scope creep ;-)


Sounds like there's already agreement to revert the 'pkgconf' change and push a new branch without them which becomes 'core-updates'. Josselin on IRC I had the impression you were working on that?

Thanks,

Re: Status of ‘core-updates’

[Ludovic Courtès]

Hi Steve,

Steve George <steve@futurile.net> skribis:

> On 10 Apr, Ludovic Courtès wrote:

[...]

>> To be clear (but I guess it’s crystal clear to anyone who’s been around
>> long enough :-)), what we need most is someone to keep track of changes,
>> coordinate efforts, decide what goes in the branch and what’s postponed
>> or moved to a separate branch, and send periodic (weekly) status updates
>> over the course of a couple of months.  This can (and probably should)
>> be done without doing any actual hacking on the branch.
> (...)
>
> This sounds like something I can do:
>
> - track changes (in branch)
> - co-ordinate blocking issues (in bug system)
> - co-ordinate with people doing the actual work :-P
> - send (periodic) weekly emails
> - encourage shipping by minimising scope creep ;-)

Awesome, thanks for volunteering!

> Sounds like there's already agreement to revert the 'pkgconf' change and push a new branch without them which becomes 'core-updates'. Josselin on IRC I had the impression you were working on that?

I’m not sure what the situation is (I see Maxim just pushed changes on
top of current ‘core-updates’, so maybe it’s OK?).

Josselin, Maxim: could you explain what problems there are around
pkgconf and what you would recommend?

Thanks,
Ludo’.

Re: Status of ‘core-updates’

[Maxim Cournoyer]

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Steve,
>
> Steve George <steve@futurile.net> skribis:
>
>> On 10 Apr, Ludovic Courtès wrote:
>
> [...]
>
>>> To be clear (but I guess it’s crystal clear to anyone who’s been around
>>> long enough :-)), what we need most is someone to keep track of changes,
>>> coordinate efforts, decide what goes in the branch and what’s postponed
>>> or moved to a separate branch, and send periodic (weekly) status updates
>>> over the course of a couple of months.  This can (and probably should)
>>> be done without doing any actual hacking on the branch.
>> (...)
>>
>> This sounds like something I can do:
>>
>> - track changes (in branch)
>> - co-ordinate blocking issues (in bug system)
>> - co-ordinate with people doing the actual work :-P
>> - send (periodic) weekly emails
>> - encourage shipping by minimising scope creep ;-)
>
> Awesome, thanks for volunteering!
>
>> Sounds like there's already agreement to revert the 'pkgconf' change
>> and push a new branch without them which becomes
>> 'core-updates'. Josselin on IRC I had the impression you were
>> working on that?
>
> I’m not sure what the situation is (I see Maxim just pushed changes on
> top of current ‘core-updates’, so maybe it’s OK?).

Since branches were merged in, I believe the problem we are facing at
the moment is librsvg failing its test suite with a segfault (!).  Could
be the glibc upgrade, or rust itself, I'm not sure.  I was trying to
upgrade librsvg, which needs an update anyway, but it pulls many rust
crates updates.  I'll get there eventually, if nobody beats me to it.

> Josselin, Maxim: could you explain what problems there are around
> pkgconf and what you would recommend?

Nothing in particular to point at the moment, but remaining problems
would manifest in the form of missing inputs, due to transitive libtool
dependencies causing overlinking and the new pkgconf being smart enough
to optimize away some previously captured link directives that would be
baked in the RUNPATH and sastify libtool overlinking needs.

The solution is to hunt the libtool .la files from the transitive
dependencies causing the problem and removing them. See commit
b6540bd285cbe5920ad379ddfc6256359ee7204c for an example.

-- 
Thanks,
Maxim

Re: Status of ‘core-updates’

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Maxim,

On Wed, Apr 17 2024, Maxim Cournoyer wrote:

> The solution is to hunt the libtool .la files

Would my old web-based file finder, which I could resurrect, help with
locating such files?

Kind regards
Felix

Re: Status of ‘core-updates’

[Ludovic Courtès]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Since branches were merged in, I believe the problem we are facing at
> the moment is librsvg failing its test suite with a segfault (!).  Could
> be the glibc upgrade, or rust itself, I'm not sure.  I was trying to
> upgrade librsvg, which needs an update anyway, but it pulls many rust
> crates updates.  I'll get there eventually, if nobody beats me to it.

Ouch, OK.  I guess it doesn’t hurt if several of us take a look and we
share our findings here.

>> Josselin, Maxim: could you explain what problems there are around
>> pkgconf and what you would recommend?
>
> Nothing in particular to point at the moment, but remaining problems
> would manifest in the form of missing inputs, due to transitive libtool
> dependencies causing overlinking and the new pkgconf being smart enough
> to optimize away some previously captured link directives that would be
> baked in the RUNPATH and sastify libtool overlinking needs.
>
> The solution is to hunt the libtool .la files from the transitive
> dependencies causing the problem and removing them. See commit
> b6540bd285cbe5920ad379ddfc6256359ee7204c for an example.

Good.  So it seems like we can move forward after all and just do the
“normal” job and finding and fixing build failures along these lines.

Do we need ci.guix to build more packages to facilitate testing and
debugging?  That’s something I can help with (though I’ll be away for a
week).

Thank you for explaining,
Ludo’.

Re: Status of ‘core-updates’

[Maxim Cournoyer]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Since branches were merged in, I believe the problem we are facing at
>> the moment is librsvg failing its test suite with a segfault (!).  Could
>> be the glibc upgrade, or rust itself, I'm not sure.  I was trying to
>> upgrade librsvg, which needs an update anyway, but it pulls many rust
>> crates updates.  I'll get there eventually, if nobody beats me to it.
>
> Ouch, OK.  I guess it doesn’t hurt if several of us take a look and we
> share our findings here.
>
>>> Josselin, Maxim: could you explain what problems there are around
>>> pkgconf and what you would recommend?
>>
>> Nothing in particular to point at the moment, but remaining problems
>> would manifest in the form of missing inputs, due to transitive libtool
>> dependencies causing overlinking and the new pkgconf being smart enough
>> to optimize away some previously captured link directives that would be
>> baked in the RUNPATH and sastify libtool overlinking needs.
>>
>> The solution is to hunt the libtool .la files from the transitive
>> dependencies causing the problem and removing them. See commit
>> b6540bd285cbe5920ad379ddfc6256359ee7204c for an example.
>
> Good.  So it seems like we can move forward after all and just do the
> “normal” job and finding and fixing build failures along these lines.
>
> Do we need ci.guix to build more packages to facilitate testing and
> debugging?  That’s something I can help with (though I’ll be away for a
> week).

If we do this, and I believe it'd be useful, I think it'd help to fork a
'core-updates-frozen' and have the CI rebuild this one fully, leaving
'core-updates' open to business as usual.

-- 
Thanks,
Maxim

Re: Status of ‘core-updates’

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2017 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> What’s the status of ‘core-updates’?  What are the areas where help is
> needed?
>
> I know a lot has happened since the last update¹, which is roughly when
> I dropped the ball due to other commitments, but I’m not sure where we
> are now.

I haven't really been following core-updates, but I have had a look
since there's a request to merge it now [1].

I'm really concerned by the commits on the branch though, assuming I'm
using Git right, there are 6351 commits on the branch:

  git log --pretty=oneline core-updates ^master | wc -l

Somehow, I think there's been a couple of pushes of commits to
core-updates that have partially duplicated lots of commits from master,
I put some more details in:

  https://issues.guix.gnu.org/70456#3

I think keeping the Git commit history clean and representative is
really important, so to me at least this means core-updates can't be
merged to master in it's current form, even if the changes overall from
these 6351 commits are reasonable.

I'm really not sure how to move forward though, I had a go at trying to
rebuild the branch without introducing the thousands of duplicate
commits and that produced a branch with 765 commits over master, which
still seems a lot, but a big improvement over 6351:

  https://git.cbaines.net/guix/log/?h=chris-core-updates-no-duplicates-attempt

That was really hard going though, as there's plenty of merge conflicts
along the way, and I'm pretty sure I solved some of them
incorrectly. The resulting branch also differs from core-updates.

Maybe someone with more time, care and attention could do a better job,
but it might be more worthwhile just starting fresh and rather than
trying to produce a like for like branch just without the thousands of
duplicate commits, effectively manually rebase the branch (without the
duplicate commits) on master and try to get the commits in to a usable
state.

Any ideas?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Status of ‘core-updates’

[Maxim Cournoyer]

Hi Christopher,

Christopher Baines <mail@cbaines.net> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> What’s the status of ‘core-updates’?  What are the areas where help is
>> needed?
>>
>> I know a lot has happened since the last update¹, which is roughly when
>> I dropped the ball due to other commitments, but I’m not sure where we
>> are now.
>
> I haven't really been following core-updates, but I have had a look
> since there's a request to merge it now [1].
>
> I'm really concerned by the commits on the branch though, assuming I'm
> using Git right, there are 6351 commits on the branch:
>
>   git log --pretty=oneline core-updates ^master | wc -l
>
> Somehow, I think there's been a couple of pushes of commits to
> core-updates that have partially duplicated lots of commits from master,
> I put some more details in:
>
>   https://issues.guix.gnu.org/70456#3
>
> I think keeping the Git commit history clean and representative is
> really important, so to me at least this means core-updates can't be
> merged to master in it's current form, even if the changes overall from
> these 6351 commits are reasonable.
>
> I'm really not sure how to move forward though, I had a go at trying to
> rebuild the branch without introducing the thousands of duplicate
> commits and that produced a branch with 765 commits over master, which
> still seems a lot, but a big improvement over 6351:
>
>   https://git.cbaines.net/guix/log/?h=chris-core-updates-no-duplicates-attempt
>
> That was really hard going though, as there's plenty of merge conflicts
> along the way, and I'm pretty sure I solved some of them
> incorrectly. The resulting branch also differs from core-updates.

I also think Git commit history is important, but in this case I weigh
the value of removing ~5000 duplicated rust commits against the risks of
resolving merge conflicts wrong or forgetting commits upon attempting to
recreate the branch from scratch lower than the benefit.

> Maybe someone with more time, care and attention could do a better job,
> but it might be more worthwhile just starting fresh and rather than
> trying to produce a like for like branch just without the thousands of
> duplicate commits, effectively manually rebase the branch (without the
> duplicate commits) on master and try to get the commits in to a usable
> state.

Given the little attention core-updates is currently receiving, I doubt
someone is willing to put the effort to recreate the branch from scratch
to clean its git history; at least speaking for myself I'd rather spend
the little hack time I have to work on it toward getting it finalized.

I believe how these duplicates came to exist was probably two separate
master -> core-updates merge commits, with one of them ending up being
rebased on top of the other, probably so that it could be pushed.
Perhaps we could capture in our contribution guidelines that rebasing a
merge commit should never be done to keep the history clean, and that in
a situation where:

1. a merge has been prepared locally (with conflicts resolved and all)
2. a new commit has appeared on the remote branch

the solution should be to merge the remote branch into the local one
instead of rebasing the local one on the remote one (as is usually
done).  Disclaimer: I haven't actually tried this suggested approach,
which should be done before documenting it, if there's a consensus to do
so.

In other words, I suggest we document what *not* to do to avoid
repeating the same mistake in the future, and move on.

-- 
Thanks,
Maxim

Re: Status of ‘core-updates’

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3194 bytes --]

On Wed, Apr 17, 2024 at 01:47:49PM -0400, Maxim Cournoyer wrote:
> Hello,
> 
> Ludovic Courtès <ludo@gnu.org> writes:
> 
> > Hi Steve,
> >
> > Steve George <steve@futurile.net> skribis:
> >
> >> On 10 Apr, Ludovic Courtès wrote:
> >
> > [...]
> >
> >>> To be clear (but I guess it’s crystal clear to anyone who’s been around
> >>> long enough :-)), what we need most is someone to keep track of changes,
> >>> coordinate efforts, decide what goes in the branch and what’s postponed
> >>> or moved to a separate branch, and send periodic (weekly) status updates
> >>> over the course of a couple of months.  This can (and probably should)
> >>> be done without doing any actual hacking on the branch.
> >> (...)
> >>
> >> This sounds like something I can do:
> >>
> >> - track changes (in branch)
> >> - co-ordinate blocking issues (in bug system)
> >> - co-ordinate with people doing the actual work :-P
> >> - send (periodic) weekly emails
> >> - encourage shipping by minimising scope creep ;-)
> >
> > Awesome, thanks for volunteering!
> >
> >> Sounds like there's already agreement to revert the 'pkgconf' change
> >> and push a new branch without them which becomes
> >> 'core-updates'. Josselin on IRC I had the impression you were
> >> working on that?
> >
> > I’m not sure what the situation is (I see Maxim just pushed changes on
> > top of current ‘core-updates’, so maybe it’s OK?).
> 
> Since branches were merged in, I believe the problem we are facing at
> the moment is librsvg failing its test suite with a segfault (!).  Could
> be the glibc upgrade, or rust itself, I'm not sure.  I was trying to
> upgrade librsvg, which needs an update anyway, but it pulls many rust
> crates updates.  I'll get there eventually, if nobody beats me to it.

I personally think getting librsvg updated will be easier on the
rust-team branch than on the core-updates branch, since there have been
many commits to the rust-team branch that haven't yet been merged into
master.

Assuming that now that the cairo upgrade to 1.18 was what was holding
back the librsvg upgrade, I can get librsvg up to a newer version and
then submit it for a merge.  Overall the branch should be in fairly good
shape.

> > Josselin, Maxim: could you explain what problems there are around
> > pkgconf and what you would recommend?
> 
> Nothing in particular to point at the moment, but remaining problems
> would manifest in the form of missing inputs, due to transitive libtool
> dependencies causing overlinking and the new pkgconf being smart enough
> to optimize away some previously captured link directives that would be
> baked in the RUNPATH and sastify libtool overlinking needs.
> 
> The solution is to hunt the libtool .la files from the transitive
> dependencies causing the problem and removing them. See commit
> b6540bd285cbe5920ad379ddfc6256359ee7204c for an example.
> 
> -- 
> Thanks,
> Maxim
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: bug#70456: Request for merging "core-updates" branch

[Ludovic Courtès]

Hi Chris and all,

Christopher Baines <mail@cbaines.net> skribis:

> I think keeping the Git commit history clean and representative is
> really important, so to me at least this means core-updates can't be
> merged to master in it's current form, even if the changes overall from
> these 6351 commits are reasonable.
>
> I'm really not sure how to move forward though, I had a go at trying to
> rebuild the branch without introducing the thousands of duplicate
> commits and that produced a branch with 765 commits over master, which
> still seems a lot, but a big improvement over 6351:
>
>   https://git.cbaines.net/guix/log/?h=chris-core-updates-no-duplicates-attempt
>
> That was really hard going though, as there's plenty of merge conflicts
> along the way, and I'm pretty sure I solved some of them
> incorrectly. The resulting branch also differs from core-updates.

Woow, impressive.  How did you go about finding which commits were
duplicates/cherry-picked from master?  Which commit did you start from?

Given everything you’ve explained, it seems to me it’s worth trying to
start from a clean branch like this.

I checked it out (commit da77ea23daa0bfa4a73290dff99b22d6825ff80b) to
get an idea of where we are and got this:

--8<---------------cut here---------------start------------->8---
make[2]: *** No rule to make target 'gnu/packages/patches/glib-networking-gnutls-binding.patch', needed by 'all-am'.
make[2]: *** No rule to make target 'gnu/packages/patches/librecad-support-for-boost-1.76.patch', needed by 'all-am'.
--8<---------------cut here---------------end--------------->8---

It stopped at:

--8<---------------cut here---------------start------------->8---
gnu/packages/sdl.scm:72:2: error: (package (name "sdl2") (version "2.30.1") (source (origin (method url-fetch) (uri (string-append "https://libsdl.org/release/SDL2-" version ".tar.gz")) (sha256 (base32 "0fj7gxc7rlzzrafnx9nmf7ws3paxy583fmx7bcbavi6gr3xmy881")))) (arguments (list #:tests? #f #:configure-flags (gexp (append (quote ("--disable-wayland-shared" "--enable-video-kmsdrm" "--disable-kmsdrm-shared")) (quote ("--disable-alsa-shared" "--disable-pulseaudio-shared" "--disable-x11-shared" "LDFLAGS=-lGL")))) #:make-flags (gexp (cons* (string-append "LDFLAGS=-Wl,-rpath," (ungexp (this-package-input "eudev")) "/lib" ",-rpath," (ungexp (this-package-input "vulkan-loader")) "/lib") (quote ("V=1")))))) (propagated-inputs (list libx11 libcap mesa)) (native-inputs (list pkg-config)) (inputs (list libxrandr glu alsa-lib pulseaudio dbus eudev glib ibus-minimal libxkbcommon libxcursor vulkan-loader wayland wayland-protocols)) (outputs (quote ("out" "debug"))) (synopsis "Cross platform game development library") (description "Simple DirectMedia Layer is a cross-platform development library designed to\nprovide low level access to audio, keyboard, mouse, joystick, and graphics\nhardware.") (home-page "https://libsdl.org/") (license license:bsd-3)): missing field initializers (build-system)
--8<---------------cut here---------------end--------------->8---

I guess these are merge conflicts that weren’t correctly resolved.

This branch rewrites the entire ‘core-updates’ history.  What about
rewriting starting from the first series of “duplicate” commits?  That
should solve the immediate issue while keeping the “known good” history?

Thanks,
Ludo’.