Core-updates merge

Core-updates merge

[Andreas Enge]

Hello all,

I have just merged core-updates into master and deleted the branch!
This has been a long adventure, which became particularly intensive
after the last Guix Days in February. First and foremost many thanks to
everyone who contributed to the branch, be it by commits, discussions or
by working on the infrastructure.

Each and every package is not yet in shape; please feel free to submit
patches for your favourite packages that fail to build. In particular:
- python-yubikey-manager does not build currently; work to correct this
  is underway.
- R on powerpc does not build; this will also be corrected soon;
- aarch64 has very few substitutes; I think this is mainly due to the build
  farm catching up and not so much to packages not building, but this is
  difficult to know.
If any of these are essential to you, you may wish to wait a little bit
longer before a "guix pull", or for the time being pull to a commit just
before the merge by issuing
   guix pull --commit=472706ae2f9160833951a4e4bcc4c206e03097b0

Every end of a story is the beginning of many new ones. Several areas of
work have already been identified; I will summarise what came up on the
mailing list and who expressed interest to work on it - please feel free
to join if a topic interests you, or launch another initiative if you
want to work on a different subject!
- rust-team already has a branch that is almost ready to be built and
  merged, as a precursor to the team based workflow that we need to invent
  (Efraim Flashner).
- R on powerpc64le needs to be built by changes to valgrind and lz4
  (Simon Tournier, I).
- Many Python packages need updates, in particular with the aim of building
  python-yubikey-manager (John Kehayias, Lars-Dominik Braun, Brian Cully).
- There is still work to do to bootstrap GHC until the latest version on
  i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).
- OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
- After the mesa update is before the mesa update, and it looks like more
  features can be enabled (Kaelyn Takata, John Kehayias).
- Too much in Guix depends on too much else, which makes building things
  needlessly entangled; in particular time zone data should not be referred
  to by packages, but be loaded at runtime (Leo Famulari).

All the best in your Guix endeavours,

Andreas

Re: Core-updates merge

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Andreas,

On Tue, Apr 25, 2023 at 7:09 AM Andreas Enge <andreas@enge.fr> wrote:
>
> many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.

I'd like to please also acknowledge your pivotal role in shepherding
that monumental merge.

With grace and patience you took on an unrewarding legacy task that
had hardly any potential for glory. For your dedication in advancing
the codebase, you earned my sincere admiration and gratitude. Thank
you!

Kind regards
Felix

Re: Core-updates merge

[Katherine Cox-Buday]

On 4/25/23 8:40 AM, Felix Lechner via Development of GNU Guix and the 
GNU System distribution. wrote:
> Hi Andreas,
> 
> On Tue, Apr 25, 2023 at 7:09 AM Andreas Enge <andreas@enge.fr> wrote:
>>
>> many thanks to
>> everyone who contributed to the branch, be it by commits, discussions or
>> by working on the infrastructure.
> 
> I'd like to please also acknowledge your pivotal role in shepherding
> that monumental merge.
> 
> With grace and patience you took on an unrewarding legacy task that
> had hardly any potential for glory. For your dedication in advancing
> the codebase, you earned my sincere admiration and gratitude. Thank
> you!

An emphatic +1.

Re: Core-updates merge

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 2222 bytes --]

Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

> Hello all,
>
> I have just merged core-updates into master and deleted the branch!
> This has been a long adventure, which became particularly intensive
> after the last Guix Days in February. First and foremost many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.

Great work and congrats!  Thank you for taking care of this, especially
with the time it took.  Hopefully we'll never have to do this again :)

> Each and every package is not yet in shape; please feel free to submit
> patches for your favourite packages that fail to build. In particular:
> - python-yubikey-manager does not build currently; work to correct this
>   is underway.
> - R on powerpc does not build; this will also be corrected soon;
> - aarch64 has very few substitutes; I think this is mainly due to the build
>   farm catching up and not so much to packages not building, but this is
>   difficult to know.

The graphical installer is also broken, but that should be adressed by
my recent patchset.

> Every end of a story is the beginning of many new ones. Several areas of
> work have already been identified; I will summarise what came up on the
> mailing list and who expressed interest to work on it - please feel free
> to join if a topic interests you, or launch another initiative if you
> want to work on a different subject!
> [...]

I also want to add something to the list: working out how we can best
implement the team-driven workflow.  We're currently lacking a lot of
documentation for it, and one thing I would really love to see is what
each team's current goals are.  Following the MLs/IRC is one thing, but
not everyone is able to read up on everything.  Should intra-team
communication be on guix-devel for everyone to see?  Should we display
the teams on the website?  How should we approach merging feature
branches to master?

Also, we have the new release on the horizon (I don't really remember
how far in the future it is), so we could maybe try to create a calendar
for it, and set team goals accordingly?

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Core-updates merge

[Leo Famulari]

Thank you for leading this effort!

The value of leadership and project management is sometimes underappreciated in software projects, but they are essential for our success.

On Tue, Apr 25, 2023, at 10:09, Andreas Enge wrote:
> Hello all,
>
> I have just merged core-updates into master and deleted the branch!
> This has been a long adventure, which became particularly intensive
> after the last Guix Days in February. First and foremost many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.
>
> Each and every package is not yet in shape; please feel free to submit
> patches for your favourite packages that fail to build. In particular:
> - python-yubikey-manager does not build currently; work to correct this
>   is underway.
> - R on powerpc does not build; this will also be corrected soon;
> - aarch64 has very few substitutes; I think this is mainly due to the build
>   farm catching up and not so much to packages not building, but this is
>   difficult to know.
> If any of these are essential to you, you may wish to wait a little bit
> longer before a "guix pull", or for the time being pull to a commit just
> before the merge by issuing
>    guix pull --commit=472706ae2f9160833951a4e4bcc4c206e03097b0
>
> Every end of a story is the beginning of many new ones. Several areas of
> work have already been identified; I will summarise what came up on the
> mailing list and who expressed interest to work on it - please feel free
> to join if a topic interests you, or launch another initiative if you
> want to work on a different subject!
> - rust-team already has a branch that is almost ready to be built and
>   merged, as a precursor to the team based workflow that we need to invent
>   (Efraim Flashner).
> - R on powerpc64le needs to be built by changes to valgrind and lz4
>   (Simon Tournier, I).
> - Many Python packages need updates, in particular with the aim of building
>   python-yubikey-manager (John Kehayias, Lars-Dominik Braun, Brian Cully).
> - There is still work to do to bootstrap GHC until the latest version on
>   i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).
> - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
> - After the mesa update is before the mesa update, and it looks like more
>   features can be enabled (Kaelyn Takata, John Kehayias).
> - Too much in Guix depends on too much else, which makes building things
>   needlessly entangled; in particular time zone data should not be referred
>   to by packages, but be loaded at runtime (Leo Famulari).
>
> All the best in your Guix endeavours,
>
> Andreas

Re: Core-updates merge

[Maxim Cournoyer]

Hello,

My gratitude for your patience and perseverance on merging the
core-updates branch :-).

Congratulations everyone!

-- 
Thanks,
Maxim

Re: Core-updates merge

[John Kehayias]

Dear Andreas and fellow Guix-ers,

On Tue, Apr 25, 2023 at 04:09 PM, Andreas Enge wrote:

> Hello all,
>
> I have just merged core-updates into master and deleted the branch!
> This has been a long adventure, which became particularly intensive
> after the last Guix Days in February. First and foremost many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.
>

As others have said, a big thank you to you once again for leading the
charge! Much appreciated!

> Each and every package is not yet in shape; please feel free to submit
> patches for your favourite packages that fail to build. In particular:
> - python-yubikey-manager does not build currently; work to correct this
>   is underway.

I've just submitted <https://issues.guix.gnu.org/63139> which does a bit
more than just fix that package and would be good for a Python feature
branch. I'll send the cover letter to this list for wider visibility,
though the Python team was cc'ed on the series. I suppose I should add
myself to the team after this.

> - R on powerpc does not build; this will also be corrected soon;
> - aarch64 has very few substitutes; I think this is mainly due to the build
>   farm catching up and not so much to packages not building, but this is
>   difficult to know.
> If any of these are essential to you, you may wish to wait a little bit
> longer before a "guix pull", or for the time being pull to a commit just
> before the merge by issuing
>    guix pull --commit=472706ae2f9160833951a4e4bcc4c206e03097b0
>
> Every end of a story is the beginning of many new ones. Several areas of
> work have already been identified; I will summarise what came up on the
> mailing list and who expressed interest to work on it - please feel free
> to join if a topic interests you, or launch another initiative if you
> want to work on a different subject!
> - rust-team already has a branch that is almost ready to be built and
>   merged, as a precursor to the team based workflow that we need to invent
>   (Efraim Flashner).
> - R on powerpc64le needs to be built by changes to valgrind and lz4
>   (Simon Tournier, I).
> - Many Python packages need updates, in particular with the aim of building
>   python-yubikey-manager (John Kehayias, Lars-Dominik Braun, Brian Cully).

I'll have to see what other patches we want to add to my series for
the branch and creating/building that as soon as possible. There is
some feedback needed on the deeper changes in my patch series, but
hopefully won't take much to finalize.

> - There is still work to do to bootstrap GHC until the latest version on
>   i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).
> - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
> - After the mesa update is before the mesa update, and it looks like more
>   features can be enabled (Kaelyn Takata, John Kehayias).

When I get a chance next I'll be pulling the relevant patches locally
and building as a first check. And then we'll want this feature branch
to be created and built for wider testing.

> - Too much in Guix depends on too much else, which makes building things
>   needlessly entangled; in particular time zone data should not be referred
>   to by packages, but be loaded at runtime (Leo Famulari).
>
> All the best in your Guix endeavours,
>
> Andreas

Thanks everyone!
John

Re: Core-updates merge

[Simon Tournier]

Hi Andreas,

On mar., 25 avril 2023 at 16:09, Andreas Enge <andreas@enge.fr> wrote:


> I have just merged core-updates into master and deleted the branch!

Awesome!  Thank you for your patient leadership over the past months. :-)


> - R on powerpc64le needs to be built by changes to valgrind and lz4
>   (Simon Tournier, I).

Now, we are good, right?


> - There is still work to do to bootstrap GHC until the latest version on
>   i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).

Argh, I will try to finish my patches soon.  I have:

 + turned off the test suite for ’ghc’ – it allows to not be blocked,
 + introduced ’ghc-testsuite’ to run the test suite,
 + introduced ’ghc-toolchain’ which depends on ’ghc’ and ’ghc-testsuite’
   – well, ’ghc’ being hidden.  Something similar as GCC.
 + shorten the chain as discussed elsewhere [1].

1: https://yhetil.org/guix/87r0siem5c.fsf@gmail.com


> - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).

Well, 4.07 is the version that is de-bootstrapped, i.e. bootstrapped
using ’camlboot’ via Guile – for details see [2].

However, higher versions (4.09, 4.14, 5) does not use this seed and thus
they are not de-bootstrapped.  Well, I do not know the status upstream;
from my point of view, we have two options:

 a) Agree with other distros and OCaml folks to rely on a common OCaml
 4.07 bootstrapped using camlboot and then use this OCaml 4.07 as the
 seed for the subsequent versions.  Somehow having a way to verify the
 current OCaml compiler without running again and again via camlboot.

 b) Build ourselves a chain from 4.07 bootstrapped with camlboot to
 modern OCaml compilers.  However, each time we modify one dependency of
 camlboot, it means rebuild the complete chain.  Well, bootstrapping via
 camlboot can be very slow and I do not know if we have the resources
 for non-x86_64 architecture.  Here, the list of the emerged
 dependencies:

    $ guix graph camlboot -t bag-emerged | grep label | cut -d'=' -f2
     "camlboot@0.0.0-1.45045d0", shape 
     "guile@3.0.9", shape 
     "pkg-config@0.29.2", shape 
     "tar@1.34", shape 
     "gzip@1.12", shape 
     "bzip2@1.0.8", shape 
     "file@5.44", shape 
     "diffutils@3.8", shape 
     "patch@2.7.6", shape 
     "findutils@4.9.0", shape 
     "gawk@5.2.1", shape 
     "sed@4.8", shape 
     "grep@3.8", shape 
     "xz@5.2.8", shape 
     "coreutils@9.1", shape 
     "make@4.3", shape 
     "bash-minimal@5.1.16", shape 
     "ld-wrapper@0", shape 
     "binutils@2.38", shape 
     "gcc@11.3.0", shape 
     "glibc@2.35", shape 
     "glibc-utf8-locales@2.35", shape 
     "libffi@3.4.4", shape 
     "bash-minimal@5.1.16", shape 
     "libunistring@1.0", shape 
     "libgc@8.2.2", shape

 c) Fix the dependencies of camlboot.


Well, it seems a separated discussion but it echoes the recent blog post [3]
about “The Full-Source Bootstrap”. :-)

2: https://10years.guix.gnu.org/video/camlboot-debootstrapping-the-ocaml-compiler
3: https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/


Cheers,
simon

Re: Core-updates merge

[Ludovic Courtès]

Hello!

"Leo Famulari" <leo@famulari.name> skribis:

> Thank you for leading this effort!
>
> The value of leadership and project management is sometimes underappreciated in software projects, but they are essential for our success.

Agreed!  You did a great job Andreas at coordinating all the work *and*
actually doing a lot of it—thank you, and thanks to everyone who
tirelessly worked on fixing things!  Really happy to be enjoying the
fruits of this hard labor now.  :-)

Ludo’.

Re: Core-updates merge

[John Kehayias]

Bringing back up an old thread, but

On Fri, Apr 28, 2023 at 05:55 AM, John Kehayias wrote:

> Dear Andreas and fellow Guix-ers,
>
> On Tue, Apr 25, 2023 at 04:09 PM, Andreas Enge wrote:
>
[...]
>> Each and every package is not yet in shape; please feel free to submit
>> patches for your favourite packages that fail to build. In particular:
>> - python-yubikey-manager does not build currently; work to correct this
>>   is underway.
>
> I've just submitted <https://issues.guix.gnu.org/63139> which does a bit
> more than just fix that package and would be good for a Python feature
> branch. I'll send the cover letter to this list for wider visibility,
> though the Python team was cc'ed on the series. I suppose I should add
> myself to the team after this.
>

Thanks to Pierre Langlois python-yubikey should now finally be fixed
on master, without all the big python build changes. (Poetry is still
broken.) This was via <https://issues.guix.gnu.org/63354>. Unfortunately
I managed to clobber the author line in the git log after editing and
testing locally, sorry about that! (Anything that can be done to fix
that?)

Now to return to the big python series, minus a few patches.

John

Re: Core-updates merge

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi John,

On Fri, Jul 14, 2023 at 9:37 AM John Kehayias
<john.kehayias@protonmail.com> wrote:
>
> Unfortunately
> I managed to clobber the author line in the git log after editing and
> testing locally, sorry about that! (Anything that can be done to fix
> that?)

Giving proper attribution can be essential in group projects. You
could revert your commit and then re-commit the change with the
correct author.

How to handle it best depends on your dynamics with the injured party.
I personally never cared enough—especially not in the case of an
innocent mistake—but I know that some folks feel strongly about being
made whole.

Kind regards
Felix

Re: Core-updates merge

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]

Hi John!

John Kehayias <john.kehayias@protonmail.com> writes:

> Bringing back up an old thread, but
>
> On Fri, Apr 28, 2023 at 05:55 AM, John Kehayias wrote:
>
>> Dear Andreas and fellow Guix-ers,
>>
>> On Tue, Apr 25, 2023 at 04:09 PM, Andreas Enge wrote:
>>
> [...]
>>> Each and every package is not yet in shape; please feel free to submit
>>> patches for your favourite packages that fail to build. In particular:
>>> - python-yubikey-manager does not build currently; work to correct this
>>>   is underway.
>>
>> I've just submitted <https://issues.guix.gnu.org/63139> which does a bit
>> more than just fix that package and would be good for a Python feature
>> branch. I'll send the cover letter to this list for wider visibility,
>> though the Python team was cc'ed on the series. I suppose I should add
>> myself to the team after this.
>>
>
> Thanks to Pierre Langlois python-yubikey should now finally be fixed
> on master, without all the big python build changes. (Poetry is still
> broken.) This was via <https://issues.guix.gnu.org/63354>. Unfortunately
> I managed to clobber the author line in the git log after editing and
> testing locally, sorry about that! (Anything that can be done to fix
> that?)

Thank you for picking up the series!  I admit I had forgotten I had sent
it, I don't have too much time for Guix these days I'm afraid.
Attribution is always nice of course, but mistakes happen, especially
with git, so I wouln't worry about it in this instance! At least
personnally I don't mind :-), just happy that the patches went in.

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

Re: Core-updates merge

[John Kehayias]

Hi Felix,

On Fri, Jul 14, 2023 at 09:52 AM, Felix Lechner wrote:

> Hi John,
>
> On Fri, Jul 14, 2023 at 9:37 AM John Kehayias
> <john.kehayias@protonmail.com> wrote:
>>
>> Unfortunately
>> I managed to clobber the author line in the git log after editing and
>> testing locally, sorry about that! (Anything that can be done to fix
>> that?)
>
> Giving proper attribution can be essential in group projects. You
> could revert your commit and then re-commit the change with the
> correct author.
>

I agree, which is why I had a minor panic upon seeing the wrong name
in the git log on savannah after pushing (after having checked locally
so much to make sure I got it right, I must have glossed over it in
the final formation).

Revert/re-commit seems how we have to handle it, thanks for that note.

> How to handle it best depends on your dynamics with the injured party.
> I personally never cared enough—especially not in the case of an
> innocent mistake—but I know that some folks feel strongly about being
> made whole.
>

I made sure Pierre was aware in the original patch thread and here,
and see a response now...

> Kind regards
> Felix

Thanks Felix!
John

Re: Core-updates merge

[John Kehayias]

Hi Pierre!

On Fri, Jul 14, 2023 at 07:12 PM, Pierre Langlois wrote:

> Hi John!
>
> John Kehayias <john.kehayias@protonmail.com> writes:
>
>> Bringing back up an old thread, but
>>
>> On Fri, Apr 28, 2023 at 05:55 AM, John Kehayias wrote:
>>
>>> Dear Andreas and fellow Guix-ers,
>>>
>>> On Tue, Apr 25, 2023 at 04:09 PM, Andreas Enge wrote:
>>>
>> [...]
>>>> Each and every package is not yet in shape; please feel free to submit
>>>> patches for your favourite packages that fail to build. In particular:
>>>> - python-yubikey-manager does not build currently; work to correct this
>>>>   is underway.
>>>
>>> I've just submitted <https://issues.guix.gnu.org/63139> which does a bit
>>> more than just fix that package and would be good for a Python feature
>>> branch. I'll send the cover letter to this list for wider visibility,
>>> though the Python team was cc'ed on the series. I suppose I should add
>>> myself to the team after this.
>>>
>>
>> Thanks to Pierre Langlois python-yubikey should now finally be fixed
>> on master, without all the big python build changes. (Poetry is still
>> broken.) This was via <https://issues.guix.gnu.org/63354>. Unfortunately
>> I managed to clobber the author line in the git log after editing and
>> testing locally, sorry about that! (Anything that can be done to fix
>> that?)
>
> Thank you for picking up the series!  I admit I had forgotten I had sent
> it, I don't have too much time for Guix these days I'm afraid.
> Attribution is always nice of course, but mistakes happen, especially
> with git, so I wouln't worry about it in this instance! At least
> personnally I don't mind :-), just happy that the patches went in.
>

Thanks for the patches! I was so caught up in all the other python
stuff I ran into at the time that I never looked to pull out the fixes
for python-yubikey-manager (which I use and had been using from my
local branch).

I appreciate the understanding! We do have your copyright lines in
these additions. Given the overlap and complementary nature of both
patch series (I hadn't done your cleanup but did other things) I
should have just had us both as co-authors clearly from the start.

I'll give myself a slight pass since I think this was the first
non-trivial thing I pushed for someone else (working on a separate
branch locally, rebasing several times to make fixes, etc.). Hopefully
that means the next ones will be smoother and I can get rolling on
helping out with reviews and pushing our many great patches we have
waiting.

Thanks again for the work and understanding and Guix is always happy
to have your contributions!

John

Re: Core-updates merge

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Pierre,

On Fri, Jul 14, 2023 at 11:18 AM Pierre Langlois
<pierre.langlois@gmx.com> wrote:
>
> Attribution is always nice of course, but mistakes happen

Thank you for your kind and generous reaction to the innocent oversight.

Please have a good weekend!

Kind regards
Felix