Core-updates merge

Core-updates merge

[Andreas Enge]

Hello all,

I have just merged core-updates into master and deleted the branch!
This has been a long adventure, which became particularly intensive
after the last Guix Days in February. First and foremost many thanks to
everyone who contributed to the branch, be it by commits, discussions or
by working on the infrastructure.

Each and every package is not yet in shape; please feel free to submit
patches for your favourite packages that fail to build. In particular:
- python-yubikey-manager does not build currently; work to correct this
  is underway.
- R on powerpc does not build; this will also be corrected soon;
- aarch64 has very few substitutes; I think this is mainly due to the build
  farm catching up and not so much to packages not building, but this is
  difficult to know.
If any of these are essential to you, you may wish to wait a little bit
longer before a "guix pull", or for the time being pull to a commit just
before the merge by issuing
   guix pull --commit=472706ae2f9160833951a4e4bcc4c206e03097b0

Every end of a story is the beginning of many new ones. Several areas of
work have already been identified; I will summarise what came up on the
mailing list and who expressed interest to work on it - please feel free
to join if a topic interests you, or launch another initiative if you
want to work on a different subject!
- rust-team already has a branch that is almost ready to be built and
  merged, as a precursor to the team based workflow that we need to invent
  (Efraim Flashner).
- R on powerpc64le needs to be built by changes to valgrind and lz4
  (Simon Tournier, I).
- Many Python packages need updates, in particular with the aim of building
  python-yubikey-manager (John Kehayias, Lars-Dominik Braun, Brian Cully).
- There is still work to do to bootstrap GHC until the latest version on
  i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).
- OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
- After the mesa update is before the mesa update, and it looks like more
  features can be enabled (Kaelyn Takata, John Kehayias).
- Too much in Guix depends on too much else, which makes building things
  needlessly entangled; in particular time zone data should not be referred
  to by packages, but be loaded at runtime (Leo Famulari).

All the best in your Guix endeavours,

Andreas

Re: Core-updates merge

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Andreas,

On Tue, Apr 25, 2023 at 7:09 AM Andreas Enge <andreas@enge.fr> wrote:
>
> many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.

I'd like to please also acknowledge your pivotal role in shepherding
that monumental merge.

With grace and patience you took on an unrewarding legacy task that
had hardly any potential for glory. For your dedication in advancing
the codebase, you earned my sincere admiration and gratitude. Thank
you!

Kind regards
Felix

Re: Core-updates merge

[Katherine Cox-Buday]

On 4/25/23 8:40 AM, Felix Lechner via Development of GNU Guix and the 
GNU System distribution. wrote:
> Hi Andreas,
> 
> On Tue, Apr 25, 2023 at 7:09 AM Andreas Enge <andreas@enge.fr> wrote:
>>
>> many thanks to
>> everyone who contributed to the branch, be it by commits, discussions or
>> by working on the infrastructure.
> 
> I'd like to please also acknowledge your pivotal role in shepherding
> that monumental merge.
> 
> With grace and patience you took on an unrewarding legacy task that
> had hardly any potential for glory. For your dedication in advancing
> the codebase, you earned my sincere admiration and gratitude. Thank
> you!

An emphatic +1.

Re: Core-updates merge

[Maxim Cournoyer]

Hello,

My gratitude for your patience and perseverance on merging the
core-updates branch :-).

Congratulations everyone!

-- 
Thanks,
Maxim

Re: Core-updates merge

[Josselin Poiret]

[-- Attachment #1: Type: text/plain, Size: 2222 bytes --]

Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

> Hello all,
>
> I have just merged core-updates into master and deleted the branch!
> This has been a long adventure, which became particularly intensive
> after the last Guix Days in February. First and foremost many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.

Great work and congrats!  Thank you for taking care of this, especially
with the time it took.  Hopefully we'll never have to do this again :)

> Each and every package is not yet in shape; please feel free to submit
> patches for your favourite packages that fail to build. In particular:
> - python-yubikey-manager does not build currently; work to correct this
>   is underway.
> - R on powerpc does not build; this will also be corrected soon;
> - aarch64 has very few substitutes; I think this is mainly due to the build
>   farm catching up and not so much to packages not building, but this is
>   difficult to know.

The graphical installer is also broken, but that should be adressed by
my recent patchset.

> Every end of a story is the beginning of many new ones. Several areas of
> work have already been identified; I will summarise what came up on the
> mailing list and who expressed interest to work on it - please feel free
> to join if a topic interests you, or launch another initiative if you
> want to work on a different subject!
> [...]

I also want to add something to the list: working out how we can best
implement the team-driven workflow.  We're currently lacking a lot of
documentation for it, and one thing I would really love to see is what
each team's current goals are.  Following the MLs/IRC is one thing, but
not everyone is able to read up on everything.  Should intra-team
communication be on guix-devel for everyone to see?  Should we display
the teams on the website?  How should we approach merging feature
branches to master?

Also, we have the new release on the horizon (I don't really remember
how far in the future it is), so we could maybe try to create a calendar
for it, and set team goals accordingly?

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

Re: Core-updates merge

[Leo Famulari]

Thank you for leading this effort!

The value of leadership and project management is sometimes underappreciated in software projects, but they are essential for our success.

On Tue, Apr 25, 2023, at 10:09, Andreas Enge wrote:
> Hello all,
>
> I have just merged core-updates into master and deleted the branch!
> This has been a long adventure, which became particularly intensive
> after the last Guix Days in February. First and foremost many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.
>
> Each and every package is not yet in shape; please feel free to submit
> patches for your favourite packages that fail to build. In particular:
> - python-yubikey-manager does not build currently; work to correct this
>   is underway.
> - R on powerpc does not build; this will also be corrected soon;
> - aarch64 has very few substitutes; I think this is mainly due to the build
>   farm catching up and not so much to packages not building, but this is
>   difficult to know.
> If any of these are essential to you, you may wish to wait a little bit
> longer before a "guix pull", or for the time being pull to a commit just
> before the merge by issuing
>    guix pull --commit=472706ae2f9160833951a4e4bcc4c206e03097b0
>
> Every end of a story is the beginning of many new ones. Several areas of
> work have already been identified; I will summarise what came up on the
> mailing list and who expressed interest to work on it - please feel free
> to join if a topic interests you, or launch another initiative if you
> want to work on a different subject!
> - rust-team already has a branch that is almost ready to be built and
>   merged, as a precursor to the team based workflow that we need to invent
>   (Efraim Flashner).
> - R on powerpc64le needs to be built by changes to valgrind and lz4
>   (Simon Tournier, I).
> - Many Python packages need updates, in particular with the aim of building
>   python-yubikey-manager (John Kehayias, Lars-Dominik Braun, Brian Cully).
> - There is still work to do to bootstrap GHC until the latest version on
>   i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).
> - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
> - After the mesa update is before the mesa update, and it looks like more
>   features can be enabled (Kaelyn Takata, John Kehayias).
> - Too much in Guix depends on too much else, which makes building things
>   needlessly entangled; in particular time zone data should not be referred
>   to by packages, but be loaded at runtime (Leo Famulari).
>
> All the best in your Guix endeavours,
>
> Andreas

Re: Core-updates merge

[Ludovic Courtès]

Hello!

"Leo Famulari" <leo@famulari.name> skribis:

> Thank you for leading this effort!
>
> The value of leadership and project management is sometimes underappreciated in software projects, but they are essential for our success.

Agreed!  You did a great job Andreas at coordinating all the work *and*
actually doing a lot of it—thank you, and thanks to everyone who
tirelessly worked on fixing things!  Really happy to be enjoying the
fruits of this hard labor now.  :-)

Ludo’.

Build dependency inflation (was: Re: Core-updates merge)

[Simon Josefsson via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 1921 bytes --]

Andreas Enge <andreas@enge.fr> writes:

> - Too much in Guix depends on too much else, which makes building things
>   needlessly entangled; in particular time zone data should not be referred
>   to by packages, but be loaded at runtime (Leo Famulari).

This is an important open problem -- is there any way to attack this
problem in a systematic way?  I guess it is hard to understand which
packages ends up depending on what since it is a large graph with long
cycles, and also to understand which build depencies are essential and
which are superficial, and thus consequently challenging to know where
to start working to reduce build dependencies?

I wonder if it is possible to graph all the build dependencies, and do
something like a monte-carlo what-if simulation: randomly pick one
build-dependency from the entire build-dependency graph and remove it,
and recompute the graph.  If the difference between these two graphs
leads to a significantly lower total build computational cost than
before, we may be on to something.  My theory is that "true" build
dependencies will show up in so many places that removing just one
instance of it will not affect total build time.  But "needless" build
dependencies may occur just once or few times, and this approach would
catch low-hanging fruit to work on.  Maybe the simulation needs to be
done on more than just removing one build-dependency, it could play
what-if games removing two build-dependencies etc, or three random
build-dependencies, and so on.  Maybe my idea is flawed, and this will
only lead to a list of build-dependencies that are impossible to get rid
off anyway.

Is there some other method to understand what build dependencies would
be important to remove, to speed up total rebuild time?

Maybe we could analyze how much of a particular build-dependency
actually is used by a particular build?  By looking into file-access
patterns etc.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Build dependency inflation (was: Re: Core-updates merge)

[bokr]

Hi,
TL;DR:

If we have 22k sources, how about expressing dependency as a
22k x 22k matrix of weights Wij and dependency by
multiplication of 22k x 1 matrix os sources variables
ordered Sj by immediate dependency first. I guess the Sj
column matrix is transposed to do the multiplication, so the
Wij j's and Sj j's match.

Start with weights of only 1 0r 0. Later maybe measures of
compile times?

The diagonal Wjj would be all ones, and counts of ones in
vertical columns Wkj would be maximal if that Sj was
directly depended on for a lot of k's

Reordering the Wjj and Sj sort of like solving simultaneous
equations and colorizing somehow to show the clusters of Wij
dependency bits vs a backgrond of zero bits might make an
interesting display. WDYT?

Any GnuPlot whizzes out there? :-)

Regards,
Bengt Richter



On +2023-04-27 19:56:38 +0200, Simon Josefsson via Development of GNU Guix and the GNU System distribution. wrote:
> Andreas Enge <andreas@enge.fr> writes:
> 
> > - Too much in Guix depends on too much else, which makes building things
> >   needlessly entangled; in particular time zone data should not be referred
> >   to by packages, but be loaded at runtime (Leo Famulari).
> 
> This is an important open problem -- is there any way to attack this
> problem in a systematic way?  I guess it is hard to understand which
> packages ends up depending on what since it is a large graph with long
> cycles, and also to understand which build depencies are essential and
> which are superficial, and thus consequently challenging to know where
> to start working to reduce build dependencies?
> 
> I wonder if it is possible to graph all the build dependencies, and do
> something like a monte-carlo what-if simulation: randomly pick one
> build-dependency from the entire build-dependency graph and remove it,
> and recompute the graph.  If the difference between these two graphs
> leads to a significantly lower total build computational cost than
> before, we may be on to something.  My theory is that "true" build
> dependencies will show up in so many places that removing just one
> instance of it will not affect total build time.  But "needless" build
> dependencies may occur just once or few times, and this approach would
> catch low-hanging fruit to work on.  Maybe the simulation needs to be
> done on more than just removing one build-dependency, it could play
> what-if games removing two build-dependencies etc, or three random
> build-dependencies, and so on.  Maybe my idea is flawed, and this will
> only lead to a list of build-dependencies that are impossible to get rid
> off anyway.
> 
> Is there some other method to understand what build dependencies would
> be important to remove, to speed up total rebuild time?
> 
> Maybe we could analyze how much of a particular build-dependency
> actually is used by a particular build?  By looking into file-access
> patterns etc.
> 
> /Simon

Re: Build dependency inflation (was: Re: Core-updates merge)

[Simon Tournier]

Hi,

On jeu., 27 avril 2023 at 19:56, Simon Josefsson via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org> wrote:
> Andreas Enge <andreas@enge.fr> writes:
>
>> - Too much in Guix depends on too much else, which makes building things
>>   needlessly entangled; in particular time zone data should not be referred
>>   to by packages, but be loaded at runtime (Leo Famulari).
>
> This is an important open problem -- is there any way to attack this
> problem in a systematic way?  I guess it is hard to understand which
> packages ends up depending on what since it is a large graph with long
> cycles, and also to understand which build depencies are essential and
> which are superficial, and thus consequently challenging to know where
> to start working to reduce build dependencies?

Hum, I am not to get the “large graph with long cycles” part.  Since the
graph for packages is a DAG.

Well, “guix graph” and “guix graph --path” help to spot the chain of
dependencies between two packages that seem unrelated.

From my point of view, we should increase the tooling to inspect these
graphs.  Other said, add feature to “guix graph”.

At some point, it could be nice to have “guile-igraph” a Guile binding
of igraph [1].

Last, back on December [2], I started to apply some well-known network
algorithm (link analysis, centrality measure, pagerank, etc.) to the
graph of packages.  From my point of view, these kind of tools could be
very helpful to spot out the packages that need some specific care.

Maybe it would fit a project for a student. ;-)


1: https://igraph.org/
2: https://yhetil.org/guix/874ju4qyd4.fsf@gmail.com
   Some stats about the graph of dependencies
   Fri, 09 Dec 2022 18:29:43 +0100
   id:874ju4qyd4.fsf@gmail.com


Cheers,
simon

Re: Build dependency inflation

[Ludovic Courtès]

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Last, back on December [2], I started to apply some well-known network
> algorithm (link analysis, centrality measure, pagerank, etc.) to the
> graph of packages.  From my point of view, these kind of tools could be
> very helpful to spot out the packages that need some specific care.
>
> Maybe it would fit a project for a student. ;-)

Yes, I think we need more work along the lines of what you did!

We could also couple the actual graph with data such as individual
output size.

Ludo’.

Re: Build dependency inflation

[Simon Tournier]

Hi,

On Wed, 03 May 2023 at 23:10, Ludovic Courtès <ludo@gnu.org> wrote:

> We could also couple the actual graph with data such as individual
> output size.

Oh, that’s a cool idea: apply network algorithms to a weighted graph.

It looks like a nice student project. :-)

Cheers,
simon

Re: Build dependency inflation (was: Re: Core-updates merge)

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 2347 bytes --]


Simon Josefsson via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org> writes:

> [[PGP Signed Part:Undecided]]
> Andreas Enge <andreas@enge.fr> writes:
>
>> - Too much in Guix depends on too much else, which makes building things
>>   needlessly entangled; in particular time zone data should not be referred
>>   to by packages, but be loaded at runtime (Leo Famulari).
>
> This is an important open problem -- is there any way to attack this
> problem in a systematic way?  I guess it is hard to understand which
> packages ends up depending on what since it is a large graph with long
> cycles, and also to understand which build depencies are essential and
> which are superficial, and thus consequently challenging to know where
> to start working to reduce build dependencies?
>
> I wonder if it is possible to graph all the build dependencies, and do
> something like a monte-carlo what-if simulation: randomly pick one
> build-dependency from the entire build-dependency graph and remove it,
> and recompute the graph.  If the difference between these two graphs
> leads to a significantly lower total build computational cost than
> before, we may be on to something.  My theory is that "true" build
> dependencies will show up in so many places that removing just one
> instance of it will not affect total build time.  But "needless" build
> dependencies may occur just once or few times, and this approach would
> catch low-hanging fruit to work on.  Maybe the simulation needs to be
> done on more than just removing one build-dependency, it could play
> what-if games removing two build-dependencies etc, or three random
> build-dependencies, and so on.  Maybe my idea is flawed, and this will
> only lead to a list of build-dependencies that are impossible to get rid
> off anyway.
>
> Is there some other method to understand what build dependencies would
> be important to remove, to speed up total rebuild time?
>
> Maybe we could analyze how much of a particular build-dependency
> actually is used by a particular build?  By looking into file-access
> patterns etc.

This is something I'd like to see incorporated in to qa.guix.gnu.org for
patch (and branch) review. While one off analysis is good, I think it's
most important to be able to look at changes and see how they change the
situation.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

Re: Core-updates merge

[John Kehayias]

Dear Andreas and fellow Guix-ers,

On Tue, Apr 25, 2023 at 04:09 PM, Andreas Enge wrote:

> Hello all,
>
> I have just merged core-updates into master and deleted the branch!
> This has been a long adventure, which became particularly intensive
> after the last Guix Days in February. First and foremost many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.
>

As others have said, a big thank you to you once again for leading the
charge! Much appreciated!

> Each and every package is not yet in shape; please feel free to submit
> patches for your favourite packages that fail to build. In particular:
> - python-yubikey-manager does not build currently; work to correct this
>   is underway.

I've just submitted <https://issues.guix.gnu.org/63139> which does a bit
more than just fix that package and would be good for a Python feature
branch. I'll send the cover letter to this list for wider visibility,
though the Python team was cc'ed on the series. I suppose I should add
myself to the team after this.

> - R on powerpc does not build; this will also be corrected soon;
> - aarch64 has very few substitutes; I think this is mainly due to the build
>   farm catching up and not so much to packages not building, but this is
>   difficult to know.
> If any of these are essential to you, you may wish to wait a little bit
> longer before a "guix pull", or for the time being pull to a commit just
> before the merge by issuing
>    guix pull --commit=472706ae2f9160833951a4e4bcc4c206e03097b0
>
> Every end of a story is the beginning of many new ones. Several areas of
> work have already been identified; I will summarise what came up on the
> mailing list and who expressed interest to work on it - please feel free
> to join if a topic interests you, or launch another initiative if you
> want to work on a different subject!
> - rust-team already has a branch that is almost ready to be built and
>   merged, as a precursor to the team based workflow that we need to invent
>   (Efraim Flashner).
> - R on powerpc64le needs to be built by changes to valgrind and lz4
>   (Simon Tournier, I).
> - Many Python packages need updates, in particular with the aim of building
>   python-yubikey-manager (John Kehayias, Lars-Dominik Braun, Brian Cully).

I'll have to see what other patches we want to add to my series for
the branch and creating/building that as soon as possible. There is
some feedback needed on the deeper changes in my patch series, but
hopefully won't take much to finalize.

> - There is still work to do to bootstrap GHC until the latest version on
>   i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).
> - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
> - After the mesa update is before the mesa update, and it looks like more
>   features can be enabled (Kaelyn Takata, John Kehayias).

When I get a chance next I'll be pulling the relevant patches locally
and building as a first check. And then we'll want this feature branch
to be created and built for wider testing.

> - Too much in Guix depends on too much else, which makes building things
>   needlessly entangled; in particular time zone data should not be referred
>   to by packages, but be loaded at runtime (Leo Famulari).
>
> All the best in your Guix endeavours,
>
> Andreas

Thanks everyone!
John

Re: Core-updates merge

[Simon Tournier]

Hi Andreas,

On mar., 25 avril 2023 at 16:09, Andreas Enge <andreas@enge.fr> wrote:


> I have just merged core-updates into master and deleted the branch!

Awesome!  Thank you for your patient leadership over the past months. :-)


> - R on powerpc64le needs to be built by changes to valgrind and lz4
>   (Simon Tournier, I).

Now, we are good, right?


> - There is still work to do to bootstrap GHC until the latest version on
>   i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).

Argh, I will try to finish my patches soon.  I have:

 + turned off the test suite for ’ghc’ – it allows to not be blocked,
 + introduced ’ghc-testsuite’ to run the test suite,
 + introduced ’ghc-toolchain’ which depends on ’ghc’ and ’ghc-testsuite’
   – well, ’ghc’ being hidden.  Something similar as GCC.
 + shorten the chain as discussed elsewhere [1].

1: https://yhetil.org/guix/87r0siem5c.fsf@gmail.com


> - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).

Well, 4.07 is the version that is de-bootstrapped, i.e. bootstrapped
using ’camlboot’ via Guile – for details see [2].

However, higher versions (4.09, 4.14, 5) does not use this seed and thus
they are not de-bootstrapped.  Well, I do not know the status upstream;
from my point of view, we have two options:

 a) Agree with other distros and OCaml folks to rely on a common OCaml
 4.07 bootstrapped using camlboot and then use this OCaml 4.07 as the
 seed for the subsequent versions.  Somehow having a way to verify the
 current OCaml compiler without running again and again via camlboot.

 b) Build ourselves a chain from 4.07 bootstrapped with camlboot to
 modern OCaml compilers.  However, each time we modify one dependency of
 camlboot, it means rebuild the complete chain.  Well, bootstrapping via
 camlboot can be very slow and I do not know if we have the resources
 for non-x86_64 architecture.  Here, the list of the emerged
 dependencies:

    $ guix graph camlboot -t bag-emerged | grep label | cut -d'=' -f2
     "camlboot@0.0.0-1.45045d0", shape 
     "guile@3.0.9", shape 
     "pkg-config@0.29.2", shape 
     "tar@1.34", shape 
     "gzip@1.12", shape 
     "bzip2@1.0.8", shape 
     "file@5.44", shape 
     "diffutils@3.8", shape 
     "patch@2.7.6", shape 
     "findutils@4.9.0", shape 
     "gawk@5.2.1", shape 
     "sed@4.8", shape 
     "grep@3.8", shape 
     "xz@5.2.8", shape 
     "coreutils@9.1", shape 
     "make@4.3", shape 
     "bash-minimal@5.1.16", shape 
     "ld-wrapper@0", shape 
     "binutils@2.38", shape 
     "gcc@11.3.0", shape 
     "glibc@2.35", shape 
     "glibc-utf8-locales@2.35", shape 
     "libffi@3.4.4", shape 
     "bash-minimal@5.1.16", shape 
     "libunistring@1.0", shape 
     "libgc@8.2.2", shape

 c) Fix the dependencies of camlboot.


Well, it seems a separated discussion but it echoes the recent blog post [3]
about “The Full-Source Bootstrap”. :-)

2: https://10years.guix.gnu.org/video/camlboot-debootstrapping-the-ocaml-compiler
3: https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/


Cheers,
simon

ocaml bootstrap (was Re: Core-updates merge)

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 3422 bytes --]

On Fri, Apr 28, 2023 at 04:17:40PM +0200, Simon Tournier wrote:
> Hi Andreas,
> 
> On mar., 25 avril 2023 at 16:09, Andreas Enge <andreas@enge.fr> wrote:
> 
> > - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
> 
> Well, 4.07 is the version that is de-bootstrapped, i.e. bootstrapped
> using ’camlboot’ via Guile – for details see [2].
> 
> However, higher versions (4.09, 4.14, 5) does not use this seed and thus
> they are not de-bootstrapped.  Well, I do not know the status upstream;
> from my point of view, we have two options:
> 
>  a) Agree with other distros and OCaml folks to rely on a common OCaml
>  4.07 bootstrapped using camlboot and then use this OCaml 4.07 as the
>  seed for the subsequent versions.  Somehow having a way to verify the
>  current OCaml compiler without running again and again via camlboot.
> 
>  b) Build ourselves a chain from 4.07 bootstrapped with camlboot to
>  modern OCaml compilers.  However, each time we modify one dependency of
>  camlboot, it means rebuild the complete chain.  Well, bootstrapping via
>  camlboot can be very slow and I do not know if we have the resources
>  for non-x86_64 architecture.  Here, the list of the emerged
>  dependencies:

I think similarly to the very slow *-mes part of the bootstrap chain, we
have to realize that some parts are just very slow. (A vote for b)

I built camlboot once on aarch64, probably a year ago. IIRC it took more
than 24 hours. I will say that it is doable without needing 8GB of ram,
which is also a limiting factor for the slow architectures.

>     $ guix graph camlboot -t bag-emerged | grep label | cut -d'=' -f2
>      "camlboot@0.0.0-1.45045d0", shape 
>      "guile@3.0.9", shape 
>      "pkg-config@0.29.2", shape 
>      "tar@1.34", shape 
>      "gzip@1.12", shape 
>      "bzip2@1.0.8", shape 
>      "file@5.44", shape 
>      "diffutils@3.8", shape 
>      "patch@2.7.6", shape 
>      "findutils@4.9.0", shape 
>      "gawk@5.2.1", shape 
>      "sed@4.8", shape 
>      "grep@3.8", shape 
>      "xz@5.2.8", shape 
>      "coreutils@9.1", shape 
>      "make@4.3", shape 
>      "bash-minimal@5.1.16", shape 
>      "ld-wrapper@0", shape 
>      "binutils@2.38", shape 
>      "gcc@11.3.0", shape 
>      "glibc@2.35", shape 
>      "glibc-utf8-locales@2.35", shape 
>      "libffi@3.4.4", shape 
>      "bash-minimal@5.1.16", shape 
>      "libunistring@1.0", shape 
>      "libgc@8.2.2", shape
>
>  c) Fix the dependencies of camlboot.

Looking at camlboot, there's a native input of guile-3.0. Unless we want
to bootstrap using some of the boot0 packages or actually find the
minimum set of packages needed and remove the rest that are brought in
by implicit-inputs (recursively) I think it's already as small as
possible.

> 
> Well, it seems a separated discussion but it echoes the recent blog post [3]
> about “The Full-Source Bootstrap”. :-)
> 
> 2: https://10years.guix.gnu.org/video/camlboot-debootstrapping-the-ocaml-compiler
> 3: https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/
> 
> 
> Cheers,
> simon
> 
> 
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

OCaml bootstrap

[Ludovic Courtès]

Hello,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Well, 4.07 is the version that is de-bootstrapped, i.e. bootstrapped
> using ’camlboot’ via Guile – for details see [2].
>
> However, higher versions (4.09, 4.14, 5) does not use this seed and thus
> they are not de-bootstrapped.  Well, I do not know the status upstream;
> from my point of view, we have two options:
>
>  a) Agree with other distros and OCaml folks to rely on a common OCaml
>  4.07 bootstrapped using camlboot and then use this OCaml 4.07 as the
>  seed for the subsequent versions.  Somehow having a way to verify the
>  current OCaml compiler without running again and again via camlboot.
>
>  b) Build ourselves a chain from 4.07 bootstrapped with camlboot to
>  modern OCaml compilers.  However, each time we modify one dependency of
>  camlboot, it means rebuild the complete chain.  Well, bootstrapping via
>  camlboot can be very slow and I do not know if we have the resources
>  for non-x86_64 architecture.  Here, the list of the emerged
>  dependencies:

Julien, do you happen to know if there are plans to make camlboot more
capable so it can be used to build newer versions of OCaml?  Maybe
something to discuss with Nathanaëlle Courant and Gabriel Scherer?

Ludo’.

Re: OCaml bootstrap

[Julien Lepiller]

We had some discussion here, but there's still some work to do: https://github.com/Ekdohibs/camlboot/issues/59

Le 3 mai 2023 23:15:49 GMT+02:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
>Hello,
>
>Simon Tournier <zimon.toutoune@gmail.com> skribis:
>
>> Well, 4.07 is the version that is de-bootstrapped, i.e. bootstrapped
>> using ’camlboot’ via Guile – for details see [2].
>>
>> However, higher versions (4.09, 4.14, 5) does not use this seed and thus
>> they are not de-bootstrapped.  Well, I do not know the status upstream;
>> from my point of view, we have two options:
>>
>>  a) Agree with other distros and OCaml folks to rely on a common OCaml
>>  4.07 bootstrapped using camlboot and then use this OCaml 4.07 as the
>>  seed for the subsequent versions.  Somehow having a way to verify the
>>  current OCaml compiler without running again and again via camlboot.
>>
>>  b) Build ourselves a chain from 4.07 bootstrapped with camlboot to
>>  modern OCaml compilers.  However, each time we modify one dependency of
>>  camlboot, it means rebuild the complete chain.  Well, bootstrapping via
>>  camlboot can be very slow and I do not know if we have the resources
>>  for non-x86_64 architecture.  Here, the list of the emerged
>>  dependencies:
>
>Julien, do you happen to know if there are plans to make camlboot more
>capable so it can be used to build newer versions of OCaml?  Maybe
>something to discuss with Nathanaëlle Courant and Gabriel Scherer?
>
>Ludo’.

Re: OCaml bootstrap

[Simon Tournier]

Hi,

On Wed, 03 May 2023 at 23:25, Julien Lepiller <julien@lepiller.eu> wrote:

>>Julien, do you happen to know if there are plans to make camlboot more
>>capable so it can be used to build newer versions of OCaml?  Maybe
>>something to discuss with Nathanaëlle Courant and Gabriel Scherer?

> We had some discussion here, but there's still some work to do:
> https://github.com/Ekdohibs/camlboot/issues/59

Cool!

Just to be sure, the discussion is from one year ago, right?  Any
update?


Well, rehashing, I think we could consider a full-bootstrap from source
for OCaml.  Whatever the seed (4.07 or 4.09), we could consider it as
done and fixed.  Here all the “inputs”:

--8<---------------cut here---------------start------------->8---
     "bash-minimal@5.1.16" 
     "bash-minimal@5.1.16" 
     "binutils@2.38" 
     "bzip2@1.0.8" 
     "coreutils@9.1" 
     "diffutils@3.8" 
     "file@5.44" 
     "findutils@4.9.0" 
     "gawk@5.2.1" 
     "gcc@11.3.0" 
     "glibc-utf8-locales@2.35" 
     "glibc@2.35" 
     "grep@3.8" 
     "guile@3.0.9" 
     "gzip@1.12" 
     "ld-wrapper@0" 
     "libffi@3.4.4" 
     "libgc@8.2.2"
     "libunistring@1.0" 
     "make@4.3" 
     "patch@2.7.6" 
     "pkg-config@0.29.2" 
     "sed@4.8" 
     "tar@1.34" 
     "xz@5.2.8" 
--8<---------------cut here---------------end--------------->8---

Only guile-3.0 is not a package deep in the graph.  All the others are.
My question is: do we want to rebuild camlboot and then all the OCaml
world each time we update one of these “inputs“?

For example, what does it bring on the table to rebuild camlboot because
grep had been updated?


Cheers,
simon

Re: OCaml bootstrap

[Julien Lepiller]

Le 4 mai 2023 10:41:18 GMT+02:00, Simon Tournier <zimon.toutoune@gmail.com> a écrit :
>Hi,
>
>On Wed, 03 May 2023 at 23:25, Julien Lepiller <julien@lepiller.eu> wrote:
>
>>>Julien, do you happen to know if there are plans to make camlboot more
>>>capable so it can be used to build newer versions of OCaml?  Maybe
>>>something to discuss with Nathanaëlle Courant and Gabriel Scherer?
>
>> We had some discussion here, but there's still some work to do:
>> https://github.com/Ekdohibs/camlboot/issues/59
>
>Cool!
>
>Just to be sure, the discussion is from one year ago, right?  Any
>update?

No updates. I don't think any of us worked more on that than you can read.

>
>Well, rehashing, I think we could consider a full-bootstrap from source
>for OCaml.  Whatever the seed (4.07 or 4.09), we could consider it as
>done and fixed.  Here all the “inputs”:
>
>--8<---------------cut here---------------start------------->8---
>     "bash-minimal@5.1.16" 
>     "bash-minimal@5.1.16" 
>     "binutils@2.38" 
>     "bzip2@1.0.8" 
>     "coreutils@9.1" 
>     "diffutils@3.8" 
>     "file@5.44" 
>     "findutils@4.9.0" 
>     "gawk@5.2.1" 
>     "gcc@11.3.0" 
>     "glibc-utf8-locales@2.35" 
>     "glibc@2.35" 
>     "grep@3.8" 
>     "guile@3.0.9" 
>     "gzip@1.12" 
>     "ld-wrapper@0" 
>     "libffi@3.4.4" 
>     "libgc@8.2.2"
>     "libunistring@1.0" 
>     "make@4.3" 
>     "patch@2.7.6" 
>     "pkg-config@0.29.2" 
>     "sed@4.8" 
>     "tar@1.34" 
>     "xz@5.2.8" 
>--8<---------------cut here---------------end--------------->8---
>
>Only guile-3.0 is not a package deep in the graph.  All the others are.
>My question is: do we want to rebuild camlboot and then all the OCaml
>world each time we update one of these “inputs“?
>
>For example, what does it bring on the table to rebuild camlboot because
>grep had been updated?

I don't understand what you suggest here. I think grep is used in the Makefile, so we can't just remove it from the inputs. What would not building camlboot when grep is updated look like?

Have a grep-for-build that is never updated? Build camlboot once and repackage the binary (making it a bootstrap seed)?

>
>
>Cheers,
>simon

Re: OCaml bootstrap

[Simon Tournier]

Hi Julien,

On jeu., 04 mai 2023 at 12:01, Julien Lepiller <julien@lepiller.eu> wrote:

> Have a grep-for-build that is never updated? Build camlboot once and
> repackage the binary (making it a bootstrap seed)? 

Yes, something like that.

Well, the packages that camlboot depends on barely change because most
of them are “core-updates” packages.  However, still.

Some packages (autotools) use guile-3.0/pinned instead of guile-3.0.
Other gdb/pinned (rust).  Etc.

Therefore we could use something like /pinned (or -boot or else) for
building camlboot once.  This allows transparency and being able to
rebuild if the worst is necessary.

But then, we consider the binary as the bootstrap seed of OCaml world.

Maybe, it could be discussed with the OCaml community.

Well, I do not know if it is worth but somehow I think that a similar
strategy as MES and bootstrapping C could be applied for camlboot and
bootstrapping OCaml.  Something like gnu/packages/ocaml-commencement.scm
that bootstraps the OCaml world and this would depend on packages that
are tweaked with a lot of care.  Similarly as the MES bootstrap chain.


Cheers,
simon

Re: Core-updates merge

[John Kehayias]

Bringing back up an old thread, but

On Fri, Apr 28, 2023 at 05:55 AM, John Kehayias wrote:

> Dear Andreas and fellow Guix-ers,
>
> On Tue, Apr 25, 2023 at 04:09 PM, Andreas Enge wrote:
>
[...]
>> Each and every package is not yet in shape; please feel free to submit
>> patches for your favourite packages that fail to build. In particular:
>> - python-yubikey-manager does not build currently; work to correct this
>>   is underway.
>
> I've just submitted <https://issues.guix.gnu.org/63139> which does a bit
> more than just fix that package and would be good for a Python feature
> branch. I'll send the cover letter to this list for wider visibility,
> though the Python team was cc'ed on the series. I suppose I should add
> myself to the team after this.
>

Thanks to Pierre Langlois python-yubikey should now finally be fixed
on master, without all the big python build changes. (Poetry is still
broken.) This was via <https://issues.guix.gnu.org/63354>. Unfortunately
I managed to clobber the author line in the git log after editing and
testing locally, sorry about that! (Anything that can be done to fix
that?)

Now to return to the big python series, minus a few patches.

John

Re: Core-updates merge

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi John,

On Fri, Jul 14, 2023 at 9:37 AM John Kehayias
<john.kehayias@protonmail.com> wrote:
>
> Unfortunately
> I managed to clobber the author line in the git log after editing and
> testing locally, sorry about that! (Anything that can be done to fix
> that?)

Giving proper attribution can be essential in group projects. You
could revert your commit and then re-commit the change with the
correct author.

How to handle it best depends on your dynamics with the injured party.
I personally never cared enough—especially not in the case of an
innocent mistake—but I know that some folks feel strongly about being
made whole.

Kind regards
Felix

Re: Core-updates merge

[John Kehayias]

Hi Felix,

On Fri, Jul 14, 2023 at 09:52 AM, Felix Lechner wrote:

> Hi John,
>
> On Fri, Jul 14, 2023 at 9:37 AM John Kehayias
> <john.kehayias@protonmail.com> wrote:
>>
>> Unfortunately
>> I managed to clobber the author line in the git log after editing and
>> testing locally, sorry about that! (Anything that can be done to fix
>> that?)
>
> Giving proper attribution can be essential in group projects. You
> could revert your commit and then re-commit the change with the
> correct author.
>

I agree, which is why I had a minor panic upon seeing the wrong name
in the git log on savannah after pushing (after having checked locally
so much to make sure I got it right, I must have glossed over it in
the final formation).

Revert/re-commit seems how we have to handle it, thanks for that note.

> How to handle it best depends on your dynamics with the injured party.
> I personally never cared enough—especially not in the case of an
> innocent mistake—but I know that some folks feel strongly about being
> made whole.
>

I made sure Pierre was aware in the original patch thread and here,
and see a response now...

> Kind regards
> Felix

Thanks Felix!
John

Re: Core-updates merge

[Pierre Langlois]

[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]

Hi John!

John Kehayias <john.kehayias@protonmail.com> writes:

> Bringing back up an old thread, but
>
> On Fri, Apr 28, 2023 at 05:55 AM, John Kehayias wrote:
>
>> Dear Andreas and fellow Guix-ers,
>>
>> On Tue, Apr 25, 2023 at 04:09 PM, Andreas Enge wrote:
>>
> [...]
>>> Each and every package is not yet in shape; please feel free to submit
>>> patches for your favourite packages that fail to build. In particular:
>>> - python-yubikey-manager does not build currently; work to correct this
>>>   is underway.
>>
>> I've just submitted <https://issues.guix.gnu.org/63139> which does a bit
>> more than just fix that package and would be good for a Python feature
>> branch. I'll send the cover letter to this list for wider visibility,
>> though the Python team was cc'ed on the series. I suppose I should add
>> myself to the team after this.
>>
>
> Thanks to Pierre Langlois python-yubikey should now finally be fixed
> on master, without all the big python build changes. (Poetry is still
> broken.) This was via <https://issues.guix.gnu.org/63354>. Unfortunately
> I managed to clobber the author line in the git log after editing and
> testing locally, sorry about that! (Anything that can be done to fix
> that?)

Thank you for picking up the series!  I admit I had forgotten I had sent
it, I don't have too much time for Guix these days I'm afraid.
Attribution is always nice of course, but mistakes happen, especially
with git, so I wouln't worry about it in this instance! At least
personnally I don't mind :-), just happy that the patches went in.

Thanks,
Pierre

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 519 bytes --]

Re: Core-updates merge

[John Kehayias]

Hi Pierre!

On Fri, Jul 14, 2023 at 07:12 PM, Pierre Langlois wrote:

> Hi John!
>
> John Kehayias <john.kehayias@protonmail.com> writes:
>
>> Bringing back up an old thread, but
>>
>> On Fri, Apr 28, 2023 at 05:55 AM, John Kehayias wrote:
>>
>>> Dear Andreas and fellow Guix-ers,
>>>
>>> On Tue, Apr 25, 2023 at 04:09 PM, Andreas Enge wrote:
>>>
>> [...]
>>>> Each and every package is not yet in shape; please feel free to submit
>>>> patches for your favourite packages that fail to build. In particular:
>>>> - python-yubikey-manager does not build currently; work to correct this
>>>>   is underway.
>>>
>>> I've just submitted <https://issues.guix.gnu.org/63139> which does a bit
>>> more than just fix that package and would be good for a Python feature
>>> branch. I'll send the cover letter to this list for wider visibility,
>>> though the Python team was cc'ed on the series. I suppose I should add
>>> myself to the team after this.
>>>
>>
>> Thanks to Pierre Langlois python-yubikey should now finally be fixed
>> on master, without all the big python build changes. (Poetry is still
>> broken.) This was via <https://issues.guix.gnu.org/63354>. Unfortunately
>> I managed to clobber the author line in the git log after editing and
>> testing locally, sorry about that! (Anything that can be done to fix
>> that?)
>
> Thank you for picking up the series!  I admit I had forgotten I had sent
> it, I don't have too much time for Guix these days I'm afraid.
> Attribution is always nice of course, but mistakes happen, especially
> with git, so I wouln't worry about it in this instance! At least
> personnally I don't mind :-), just happy that the patches went in.
>

Thanks for the patches! I was so caught up in all the other python
stuff I ran into at the time that I never looked to pull out the fixes
for python-yubikey-manager (which I use and had been using from my
local branch).

I appreciate the understanding! We do have your copyright lines in
these additions. Given the overlap and complementary nature of both
patch series (I hadn't done your cleanup but did other things) I
should have just had us both as co-authors clearly from the start.

I'll give myself a slight pass since I think this was the first
non-trivial thing I pushed for someone else (working on a separate
branch locally, rebasing several times to make fixes, etc.). Hopefully
that means the next ones will be smoother and I can get rolling on
helping out with reviews and pushing our many great patches we have
waiting.

Thanks again for the work and understanding and Guix is always happy
to have your contributions!

John

Re: Core-updates merge

[Felix Lechner via Development of GNU Guix and the GNU System distribution.]

Hi Pierre,

On Fri, Jul 14, 2023 at 11:18 AM Pierre Langlois
<pierre.langlois@gmx.com> wrote:
>
> Attribution is always nice of course, but mistakes happen

Thank you for your kind and generous reaction to the innocent oversight.

Please have a good weekend!

Kind regards
Felix