Starting 'core-updates'?

Starting 'core-updates'?

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1444 bytes --]

Hello Guix,

The 'core-updates' branch is starting to look pretty good, and I am
happy to report that it "works for me".  :-)

Some of the big changes include:

* Large parts of Guix can now be cross-compiled, allowing building Guix
  System for foreign architectures without emulation.

* Python was updated to 3.8.1, and most of the popular packages such as
  Pytest are at the latest available versions.

* The 'libjpeg' library has been deprecated in favor of 'libjpeg-turbo'.

* 'util-linux' gained a "lib" output, decreasing the closure size of
  packages that only need the libraries by ~11.5 MiB.

* 'boost' now uses Python 3 by default.

* 'sqlite-with-column-metadata' has been merged with 'sqlite'.

* The quest to remove static libraries from core packages is ongoing.
  Many packages are a tiny bit smaller for that reason.  On the flip
  side, the closure size of some packages increased, because they are
  forced to use the shared library instead of embedding a static copy.

* cmake-build-system's support for cross-compilation is significantly
  improved.  CMake itself also no longer bundles any of its dependencies
  (even if they were mostly unused).

* Of course we have the latest versions of core packages such as glibc,
  make, sed, binutils, etc.

I suggest that we set a "freeze" date shortly after FOSDEM to start
integrating it.  Are there other branches that should be included?
Maybe wip-bootstrap or GNOME 3.34?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Starting 'core-updates'?

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 95 bytes --]

Fantastic news!  Thanks to everyone involved!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Starting 'core-updates'?

[zimoun]

Hi Marius,

Great !
Thanks!

On Mon, 27 Jan 2020 at 21:00, Marius Bakke <mbakke@fastmail.com> wrote:

> I suggest that we set a "freeze" date shortly after FOSDEM to start
> integrating it.  Are there other branches that should be included?
> Maybe wip-bootstrap or GNOME 3.34?

Does this look like a v1.1? :-)


All the best,
simon

Re: Starting 'core-updates'?

[Mathieu Othacehe]

Hello Marius,

Thanks for your great work. Concerning the first point about
cross-compilation, we are almost there, but I still have a few patches
to apply:

* Remove canonical-packages calls[1].

* Use a default target in most (guix gexp) procedures[2].

* Fix Guix itself cross-compilation. This means make sure that all
guile-* libraries that Guix depends on are cross-compilable. This is
almost done, it's just a matter of updating some package definitions.

As the second point is a bit tricky, I'd like to discuss it during Guix
Days.

Thanks,

Mathieu

[1]: https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00044.html
[2]: https://lists.gnu.org/archive/html/guix-patches/2019-12/msg00751.html

Marius Bakke writes:

> Hello Guix,
>
> The 'core-updates' branch is starting to look pretty good, and I am
> happy to report that it "works for me".  :-)
>
> Some of the big changes include:
>
> * Large parts of Guix can now be cross-compiled, allowing building Guix
>   System for foreign architectures without emulation.
>
> * Python was updated to 3.8.1, and most of the popular packages such as
>   Pytest are at the latest available versions.
>
> * The 'libjpeg' library has been deprecated in favor of 'libjpeg-turbo'.
>
> * 'util-linux' gained a "lib" output, decreasing the closure size of
>   packages that only need the libraries by ~11.5 MiB.
>
> * 'boost' now uses Python 3 by default.
>
> * 'sqlite-with-column-metadata' has been merged with 'sqlite'.
>
> * The quest to remove static libraries from core packages is ongoing.
>   Many packages are a tiny bit smaller for that reason.  On the flip
>   side, the closure size of some packages increased, because they are
>   forced to use the shared library instead of embedding a static copy.
>
> * cmake-build-system's support for cross-compilation is significantly
>   improved.  CMake itself also no longer bundles any of its dependencies
>   (even if they were mostly unused).
>
> * Of course we have the latest versions of core packages such as glibc,
>   make, sed, binutils, etc.
>
> I suggest that we set a "freeze" date shortly after FOSDEM to start
> integrating it.  Are there other branches that should be included?
> Maybe wip-bootstrap or GNOME 3.34?

Re: Starting 'core-updates'?

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]


Mathieu Othacehe <m.othacehe@gmail.com> writes:

> Hello Marius,
>
> Thanks for your great work. Concerning the first point about
> cross-compilation, we are almost there, but I still have a few patches
> to apply:
>
> * Remove canonical-packages calls[1].
>
> * Use a default target in most (guix gexp) procedures[2].
>
> * Fix Guix itself cross-compilation. This means make sure that all
> guile-* libraries that Guix depends on are cross-compilable. This is
> almost done, it's just a matter of updating some package definitions.
>
> As the second point is a bit tricky, I'd like to discuss it during Guix
> Days.

Hey,

Something I'd be interested in learning more about is
cross-compilation. The Guix Data Service computes some derivations for
cross-compilation (see the table on the right of [1] for example), but
I'm not sure if these are the right one, and because there aren't any
automated builds for these derivations as far as I'm aware, it's
difficult to know if they can actually be built.

1: http://data.guix.gnu.org/revision/ac841750a52e44d68d7f1b02e9507421f3e3824f

I'm coming to the Guix Days, so maybe it's something that can be talked
about then.

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Starting 'core-updates'?

[Ludovic Courtès]

Hi!

Christopher Baines <mail@cbaines.net> skribis:

> Something I'd be interested in learning more about is
> cross-compilation. The Guix Data Service computes some derivations for
> cross-compilation (see the table on the right of [1] for example), but
> I'm not sure if these are the right one, and because there aren't any
> automated builds for these derivations as far as I'm aware, it's
> difficult to know if they can actually be built.
>
> 1: http://data.guix.gnu.org/revision/ac841750a52e44d68d7f1b02e9507421f3e3824f
>
> I'm coming to the Guix Days, so maybe it's something that can be talked
> about then.

Definitely!

The short story though is that there’s confusion between “system types”
(like “x86-64_linux”) and GNU triplets (like “x86_64-linux-gnu”,
“arm-linux-gnueabihf”, etc.).

The former is used by the daemon as an “input”, to distinguish
derivations that target different CPUs or different OS kernels.  It’s
the option you pass to ‘guix build -s’.

The latter is used by the GNU toolchain to identify a target system:
CPU, kernel, userland, and optionally hardware vendor.  It’s what you
pass to ‘guix build --target’.

(gnu ci) contains the list of cross-compilation targets (triplets) that
we intend to support.  I believe these are those you should be
targeting.  Also, we only even build cross toolchain from “x86_64-linux”
because in practice that’s the only well-tested (and relevant) platform
for that.

Ludo’.

Re: Starting 'core-updates'?

[Ludovic Courtès]

Hi!

Marius Bakke <mbakke@fastmail.com> skribis:

> I suggest that we set a "freeze" date shortly after FOSDEM to start
> integrating it.  Are there other branches that should be included?
> Maybe wip-bootstrap or GNOME 3.34?

Is ‘default-guile’ pointing to Guile 3.0 there?  We should do that,
though perhaps a 3.0.1 would be welcome.

Besides I’d like to merge ’wip-build-systems-gexp’ one day, but I
haven’t touched it in a long time and there’s potential for breakage.
Dunno if this should wait some more…

Anyhow, thanks for getting the ball rolling!

Ludo’.

Re: Starting 'core-updates'?

[Maxim Cournoyer]

Hello Marius!

Marius Bakke <mbakke@fastmail.com> writes:

> Hello Guix,
>
> The 'core-updates' branch is starting to look pretty good, and I am
> happy to report that it "works for me".  :-)
>
> Some of the big changes include:
>
> * Large parts of Guix can now be cross-compiled, allowing building Guix
>   System for foreign architectures without emulation.
>
> * Python was updated to 3.8.1, and most of the popular packages such as
>   Pytest are at the latest available versions.
>
> * The 'libjpeg' library has been deprecated in favor of 'libjpeg-turbo'.
>
> * 'util-linux' gained a "lib" output, decreasing the closure size of
>   packages that only need the libraries by ~11.5 MiB.
>
> * 'boost' now uses Python 3 by default.
>
> * 'sqlite-with-column-metadata' has been merged with 'sqlite'.
>
> * The quest to remove static libraries from core packages is ongoing.
>   Many packages are a tiny bit smaller for that reason.  On the flip
>   side, the closure size of some packages increased, because they are
>   forced to use the shared library instead of embedding a static copy.
>
> * cmake-build-system's support for cross-compilation is significantly
>   improved.  CMake itself also no longer bundles any of its dependencies
>   (even if they were mostly unused).
>
> * Of course we have the latest versions of core packages such as glibc,
>   make, sed, binutils, etc.
>
> I suggest that we set a "freeze" date shortly after FOSDEM to start
> integrating it.  Are there other branches that should be included?
> Maybe wip-bootstrap or GNOME 3.34?

Congratulations!  I'm using core-updates as well, it seems to be working
pretty well so far as well (except some issue with CPATH causing
compilation warnings from system libraries to be emitted when only the
project's headers should be considered (https://bugs.gnu.org/30756).

I'll be looking at ways to resolve this (going back to
CPLUS_INCLUDE_PATH and C_INCLUDE_PATH without triggering include_next
errors from glibc's headers).

Maxim

Re: Starting 'core-updates'?

[Jan Nieuwenhuizen]

Marius Bakke writes:

Hello Marius,

> The 'core-updates' branch is starting to look pretty good, and I am
> happy to report that it "works for me".  :-)
>
> Some of the big changes include:

> I suggest that we set a "freeze" date shortly after FOSDEM to start
> integrating it.  Are there other branches that should be included?

> Maybe wip-bootstrap

Definately maybe!  I think that Timothy, Ludo and I have finally done
what set out to do (see https://bugs.gnu.org/38390).

I have squashed the remaining squash-commits and freshly rebased
`wip-bootstrap' on core-updates, so I think that from our point of view
we are ready for merge.  WYDT?

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Starting 'core-updates'?

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Marius Bakke writes:
>
> Hello Marius,
>
>> The 'core-updates' branch is starting to look pretty good, and I am
>> happy to report that it "works for me".  :-)
>>
>> Some of the big changes include:
>
>> I suggest that we set a "freeze" date shortly after FOSDEM to start
>> integrating it.  Are there other branches that should be included?
>
>> Maybe wip-bootstrap
>
> Definately maybe!  I think that Timothy, Ludo and I have finally done
> what set out to do (see https://bugs.gnu.org/38390).
>
> I have squashed the remaining squash-commits and freshly rebased
> `wip-bootstrap' on core-updates, so I think that from our point of view
> we are ready for merge.  WYDT?

Excellent, really looking forward to seeing this merged!  Cutting the
bootstrap seed down to ~60 MiB is nothing short of amazing.

I think that with this branch merged as well as the glibc&binutils
update from <https://issues.guix.gnu.org/issue/39456> we are ready to
start the 'core-updates' branch.  \o/

We still need to change the default Guile to 3.0.  Any takers?  :-)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [bug#38390] [core-updates] Scheme-only bootstrap: merge wip-bootstrap / Starting `core-updates'

[Jan Nieuwenhuizen]

Marius Bakke writes:

> Jan Nieuwenhuizen <janneke@gnu.org> writes:
>
>> Marius Bakke writes:
>>
>> Hello Marius,
>>
>>> The 'core-updates' branch is starting to look pretty good, and I am
>>> happy to report that it "works for me".  :-)
>>>
>>> Some of the big changes include:
>>
>>> I suggest that we set a "freeze" date shortly after FOSDEM to start
>>> integrating it.  Are there other branches that should be included?
>>
>>> Maybe wip-bootstrap
>>
>> Definately maybe!  I think that Timothy, Ludo and I have finally done
>> what set out to do (see https://bugs.gnu.org/38390).
>>
>> I have squashed the remaining squash-commits and freshly rebased
>> `wip-bootstrap' on core-updates, so I think that from our point of view
>> we are ready for merge.  WYDT?
>
> Excellent, really looking forward to seeing this merged!  Cutting the
> bootstrap seed down to ~60 MiB is nothing short of amazing.

Thank you!!  Pushed as e157ed72ec7b673b4204c84a7bf74f13afb44dc7
to core-updates.

And a very big thank you to Ludo' and Timothy: let's keep pushing this
bootstrapping thing forward!

> I think that with this branch merged as well as the glibc&binutils
> update from <https://issues.guix.gnu.org/issue/39456> we are ready to
> start the 'core-updates' branch.  \o/

\o/

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Successfully running GNOME on core-updates + staging

[Mark H Weaver]

Hello Guix,

I've successfully updated my x86_64 GuixSD system to my private branch
based on 'core-updates' with recent 'master' and 'staging' merged into
it.  This system includes a full GNOME desktop environment plus a few
programs based on Qt.  It all works quite well.

My branch includes a few draft fixes and workarounds that I haven't yet
pushed, but nothing that would require many rebuilds to update later.

So, I think it might be time to ask Hydra to build all of core-updates,
after staging is merged into it.

What do you think?

      Mark

Re: Successfully running GNOME on core-updates + staging

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> I've successfully updated my x86_64 GuixSD system to my private branch
> based on 'core-updates' with recent 'master' and 'staging' merged into
> it.  This system includes a full GNOME desktop environment plus a few
> programs based on Qt.  It all works quite well.
>
> My branch includes a few draft fixes and workarounds that I haven't yet
> pushed, but nothing that would require many rebuilds to update later.
>
> So, I think it might be time to ask Hydra to build all of core-updates,
> after staging is merged into it.

I agree.  There was an issue with cross-compiling ‘bootstrap-tarballs’
that Marius reported a few days ago, which I’m looking into right now.
I don’t expect the fix(es) to trigger a full rebuild.

If Marius and others don’t object, I’d say go for it!

Thanks,
Ludo’.

Re: Successfully running GNOME on core-updates + staging

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 1266 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Hi Mark,
>
> Mark H Weaver <mhw@netris.org> skribis:
>
>> I've successfully updated my x86_64 GuixSD system to my private branch
>> based on 'core-updates' with recent 'master' and 'staging' merged into
>> it.  This system includes a full GNOME desktop environment plus a few
>> programs based on Qt.  It all works quite well.
>>
>> My branch includes a few draft fixes and workarounds that I haven't yet
>> pushed, but nothing that would require many rebuilds to update later.
>>
>> So, I think it might be time to ask Hydra to build all of core-updates,
>> after staging is merged into it.
>
> I agree.  There was an issue with cross-compiling ‘bootstrap-tarballs’
> that Marius reported a few days ago, which I’m looking into right now.
> I don’t expect the fix(es) to trigger a full rebuild.
>
> If Marius and others don’t object, I’d say go for it!

No objections from me.  However I do have a bunch of fairly innocent
updates in my queue, such as SQLite, Glib and CMake.  It's also tempting
to get rid of that Perl graft.  Is it too late for such changes?

Hydra will be busy for a couple of days with 'master' and 'staging', so
there's little use in starting it immediately.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Successfully running GNOME on core-updates + staging

[Ludovic Courtès]

Hello,

Marius Bakke <mbakke@fastmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi Mark,
>>
>> Mark H Weaver <mhw@netris.org> skribis:
>>
>>> I've successfully updated my x86_64 GuixSD system to my private branch
>>> based on 'core-updates' with recent 'master' and 'staging' merged into
>>> it.  This system includes a full GNOME desktop environment plus a few
>>> programs based on Qt.  It all works quite well.
>>>
>>> My branch includes a few draft fixes and workarounds that I haven't yet
>>> pushed, but nothing that would require many rebuilds to update later.
>>>
>>> So, I think it might be time to ask Hydra to build all of core-updates,
>>> after staging is merged into it.
>>
>> I agree.  There was an issue with cross-compiling ‘bootstrap-tarballs’
>> that Marius reported a few days ago, which I’m looking into right now.
>> I don’t expect the fix(es) to trigger a full rebuild.
>>
>> If Marius and others don’t object, I’d say go for it!
>
> No objections from me.  However I do have a bunch of fairly innocent
> updates in my queue, such as SQLite, Glib and CMake.  It's also tempting
> to get rid of that Perl graft.  Is it too late for such changes?

I think it’s OK for sqlite/glib/cmake, but changing Perl would further
delay things, which perhaps is not desirable.

> Hydra will be busy for a couple of days with 'master' and 'staging', so
> there's little use in starting it immediately.

It took me a couple of days to reply :-), so maybe we can start the
evaluation now?

We can also get berlin to build all of ‘core-updates’ if we want.

Thanks,
Ludo’.

Starting 'core-updates'

[Marius Bakke]

[-- Attachment #1: Type: text/plain, Size: 2367 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Hello,
>
> Marius Bakke <mbakke@fastmail.com> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>>
>>> Hi Mark,
>>>
>>> Mark H Weaver <mhw@netris.org> skribis:
>>>
>>>> I've successfully updated my x86_64 GuixSD system to my private branch
>>>> based on 'core-updates' with recent 'master' and 'staging' merged into
>>>> it.  This system includes a full GNOME desktop environment plus a few
>>>> programs based on Qt.  It all works quite well.
>>>>
>>>> My branch includes a few draft fixes and workarounds that I haven't yet
>>>> pushed, but nothing that would require many rebuilds to update later.
>>>>
>>>> So, I think it might be time to ask Hydra to build all of core-updates,
>>>> after staging is merged into it.
>>>
>>> I agree.  There was an issue with cross-compiling ‘bootstrap-tarballs’
>>> that Marius reported a few days ago, which I’m looking into right now.
>>> I don’t expect the fix(es) to trigger a full rebuild.
>>>
>>> If Marius and others don’t object, I’d say go for it!
>>
>> No objections from me.  However I do have a bunch of fairly innocent
>> updates in my queue, such as SQLite, Glib and CMake.  It's also tempting
>> to get rid of that Perl graft.  Is it too late for such changes?
>
> I think it’s OK for sqlite/glib/cmake, but changing Perl would further
> delay things, which perhaps is not desirable.

I was running a bit late with my patches and pushed them to a separate
branch before noticing the 'rhash' update on 'master'.  Now there have
been a couple of world-rebuilding commits on the 'core-updates-next'
branch since, so I wonder how to move forward.

* Start 'core-updates' as-is.
* Pick all updates from the -next branch that won't rebuild the world
  (that is everything apart from "xz" and "file").
* Take all the -next commits, remove the Perl graft, and do a new 'core'
  evaluation.

Any preferences?  Due to the "rhash" update, I suppose we can take
anything from -next that depends on CMake also with option #1.

>> Hydra will be busy for a couple of days with 'master' and 'staging', so
>> there's little use in starting it immediately.
>
> It took me a couple of days to reply :-), so maybe we can start the
> evaluation now?

Let's get this rolling as soon as the current Hydra queue clears!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Starting 'core-updates'

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

On Tue, May 01, 2018 at 04:12:42PM +0200, Marius Bakke wrote:
> I was running a bit late with my patches and pushed them to a separate
> branch before noticing the 'rhash' update on 'master'.  Now there have
> been a couple of world-rebuilding commits on the 'core-updates-next'
> branch since, so I wonder how to move forward.
> 
> * Start 'core-updates' as-is.
> * Pick all updates from the -next branch that won't rebuild the world
>   (that is everything apart from "xz" and "file").
> * Take all the -next commits, remove the Perl graft, and do a new 'core'
>   evaluation.
> 
> Any preferences?  Due to the "rhash" update, I suppose we can take
> anything from -next that depends on CMake also with option #1.

I haven't been paying attention this cycle. But if anyone has, then I
think it's best to do option 1 along with the rhash, since most of the
bug-fixing work will still be valid.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Starting 'core-updates'

[Mark H Weaver]

Leo Famulari <leo@famulari.name> writes:

> On Tue, May 01, 2018 at 04:12:42PM +0200, Marius Bakke wrote:
>> I was running a bit late with my patches and pushed them to a separate
>> branch before noticing the 'rhash' update on 'master'.  Now there have
>> been a couple of world-rebuilding commits on the 'core-updates-next'
>> branch since, so I wonder how to move forward.
>> 
>> * Start 'core-updates' as-is.
>> * Pick all updates from the -next branch that won't rebuild the world
>>   (that is everything apart from "xz" and "file").
>> * Take all the -next commits, remove the Perl graft, and do a new 'core'
>>   evaluation.
>> 
>> Any preferences?  Due to the "rhash" update, I suppose we can take
>> anything from -next that depends on CMake also with option #1.
>
> I haven't been paying attention this cycle. But if anyone has, then I
> think it's best to do option 1 along with the rhash, since most of the
> bug-fixing work will still be valid.

I agree.  I've been running core-updates for a long time now.  It works.

The 'rhash' update has forced a great deal of rebuilding on it (my X200
has been rebuilding all night, and is still going), but I do not expect
that it will cause any new problems.  The further updates on -next might
very well cause more bugs that need to be investigated and fixed.

The reason that I moved my own systems so agressively to core-updates
this cycle is because I no longer trust that grafting works properly on
'master', and so security flaws might not be fully addressed there.  I'm
disappointed that there have been so many delays since then, and I'd
prefer not to add any more.

What do you think?

      Mark

Re: Starting 'core-updates'

[Ludovic Courtès]

Mark H Weaver <mhw@netris.org> skribis:

> The reason that I moved my own systems so agressively to core-updates
> this cycle is because I no longer trust that grafting works properly on
> 'master', and so security flaws might not be fully addressed there.  I'm
> disappointed that there have been so many delays since then, and I'd
> prefer not to add any more.

I agree, let’s not delay things further.

So option #1 + rhash, so be it!  When you’ve done that Marius, I’m happy
to get ‘core-updates’ built on berlin.

Besides, what makes you think grafting doesn’t work properly on master?

Thanks,
Ludo’.

Re: Starting 'core-updates'

[Mark H Weaver]

Hi Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:
> Besides, what makes you think grafting doesn’t work properly on master?

Because of bug 30820: the default GCC on our master branch sometimes
incorporates string literals containing store references directly into
the generated x86 code, broken up into 8-byte chunks within immediate
load instructions, which cannot be rewritten by our grafting code.  This
issue was fixed on core-updates only.  The fix fundamentally requires a
full rebuild, so it cannot be done on master.

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30820#51

      Mark