all messages for Guix-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* Core-updates merge
@ 2023-04-25 14:09 Andreas Enge
  2023-04-25 14:40 ` Felix Lechner via Development of GNU Guix and the GNU System distribution.
                   ` (5 more replies)
  0 siblings, 6 replies; 21+ messages in thread
From: Andreas Enge @ 2023-04-25 14:09 UTC (permalink / raw)
  To: guix-devel

Hello all,

I have just merged core-updates into master and deleted the branch!
This has been a long adventure, which became particularly intensive
after the last Guix Days in February. First and foremost many thanks to
everyone who contributed to the branch, be it by commits, discussions or
by working on the infrastructure.

Each and every package is not yet in shape; please feel free to submit
patches for your favourite packages that fail to build. In particular:
- python-yubikey-manager does not build currently; work to correct this
  is underway.
- R on powerpc does not build; this will also be corrected soon;
- aarch64 has very few substitutes; I think this is mainly due to the build
  farm catching up and not so much to packages not building, but this is
  difficult to know.
If any of these are essential to you, you may wish to wait a little bit
longer before a "guix pull", or for the time being pull to a commit just
before the merge by issuing
   guix pull --commit=472706ae2f9160833951a4e4bcc4c206e03097b0

Every end of a story is the beginning of many new ones. Several areas of
work have already been identified; I will summarise what came up on the
mailing list and who expressed interest to work on it - please feel free
to join if a topic interests you, or launch another initiative if you
want to work on a different subject!
- rust-team already has a branch that is almost ready to be built and
  merged, as a precursor to the team based workflow that we need to invent
  (Efraim Flashner).
- R on powerpc64le needs to be built by changes to valgrind and lz4
  (Simon Tournier, I).
- Many Python packages need updates, in particular with the aim of building
  python-yubikey-manager (John Kehayias, Lars-Dominik Braun, Brian Cully).
- There is still work to do to bootstrap GHC until the latest version on
  i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).
- OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
- After the mesa update is before the mesa update, and it looks like more
  features can be enabled (Kaelyn Takata, John Kehayias).
- Too much in Guix depends on too much else, which makes building things
  needlessly entangled; in particular time zone data should not be referred
  to by packages, but be loaded at runtime (Leo Famulari).

All the best in your Guix endeavours,

Andreas



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Core-updates merge
  2023-04-25 14:09 Core-updates merge Andreas Enge
@ 2023-04-25 14:40 ` Felix Lechner via Development of GNU Guix and the GNU System distribution.
  2023-04-25 14:57   ` Katherine Cox-Buday
  2023-04-25 15:31 ` Josselin Poiret
                   ` (4 subsequent siblings)
  5 siblings, 1 reply; 21+ messages in thread
From: Felix Lechner via Development of GNU Guix and the GNU System distribution. @ 2023-04-25 14:40 UTC (permalink / raw)
  To: Andreas Enge; +Cc: guix-devel

Hi Andreas,

On Tue, Apr 25, 2023 at 7:09 AM Andreas Enge <andreas@enge.fr> wrote:
>
> many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.

I'd like to please also acknowledge your pivotal role in shepherding
that monumental merge.

With grace and patience you took on an unrewarding legacy task that
had hardly any potential for glory. For your dedication in advancing
the codebase, you earned my sincere admiration and gratitude. Thank
you!

Kind regards
Felix


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Core-updates merge
  2023-04-25 14:40 ` Felix Lechner via Development of GNU Guix and the GNU System distribution.
@ 2023-04-25 14:57   ` Katherine Cox-Buday
  2023-04-25 19:27     ` Maxim Cournoyer
  0 siblings, 1 reply; 21+ messages in thread
From: Katherine Cox-Buday @ 2023-04-25 14:57 UTC (permalink / raw)
  To: Felix Lechner, Andreas Enge; +Cc: guix-devel

On 4/25/23 8:40 AM, Felix Lechner via Development of GNU Guix and the 
GNU System distribution. wrote:
> Hi Andreas,
> 
> On Tue, Apr 25, 2023 at 7:09 AM Andreas Enge <andreas@enge.fr> wrote:
>>
>> many thanks to
>> everyone who contributed to the branch, be it by commits, discussions or
>> by working on the infrastructure.
> 
> I'd like to please also acknowledge your pivotal role in shepherding
> that monumental merge.
> 
> With grace and patience you took on an unrewarding legacy task that
> had hardly any potential for glory. For your dedication in advancing
> the codebase, you earned my sincere admiration and gratitude. Thank
> you!

An emphatic +1.



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Core-updates merge
  2023-04-25 14:09 Core-updates merge Andreas Enge
  2023-04-25 14:40 ` Felix Lechner via Development of GNU Guix and the GNU System distribution.
@ 2023-04-25 15:31 ` Josselin Poiret
  2023-04-25 16:09 ` Leo Famulari
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 21+ messages in thread
From: Josselin Poiret @ 2023-04-25 15:31 UTC (permalink / raw)
  To: Andreas Enge, guix-devel

[-- Attachment #1: Type: text/plain, Size: 2222 bytes --]

Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

> Hello all,
>
> I have just merged core-updates into master and deleted the branch!
> This has been a long adventure, which became particularly intensive
> after the last Guix Days in February. First and foremost many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.

Great work and congrats!  Thank you for taking care of this, especially
with the time it took.  Hopefully we'll never have to do this again :)

> Each and every package is not yet in shape; please feel free to submit
> patches for your favourite packages that fail to build. In particular:
> - python-yubikey-manager does not build currently; work to correct this
>   is underway.
> - R on powerpc does not build; this will also be corrected soon;
> - aarch64 has very few substitutes; I think this is mainly due to the build
>   farm catching up and not so much to packages not building, but this is
>   difficult to know.

The graphical installer is also broken, but that should be adressed by
my recent patchset.

> Every end of a story is the beginning of many new ones. Several areas of
> work have already been identified; I will summarise what came up on the
> mailing list and who expressed interest to work on it - please feel free
> to join if a topic interests you, or launch another initiative if you
> want to work on a different subject!
> [...]

I also want to add something to the list: working out how we can best
implement the team-driven workflow.  We're currently lacking a lot of
documentation for it, and one thing I would really love to see is what
each team's current goals are.  Following the MLs/IRC is one thing, but
not everyone is able to read up on everything.  Should intra-team
communication be on guix-devel for everyone to see?  Should we display
the teams on the website?  How should we approach merging feature
branches to master?

Also, we have the new release on the horizon (I don't really remember
how far in the future it is), so we could maybe try to create a calendar
for it, and set team goals accordingly?

Best,
-- 
Josselin Poiret

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 682 bytes --]

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Core-updates merge
  2023-04-25 14:09 Core-updates merge Andreas Enge
  2023-04-25 14:40 ` Felix Lechner via Development of GNU Guix and the GNU System distribution.
  2023-04-25 15:31 ` Josselin Poiret
@ 2023-04-25 16:09 ` Leo Famulari
  2023-05-03 21:07   ` Ludovic Courtès
  2023-04-27 17:56 ` Build dependency inflation (was: Re: Core-updates merge) Simon Josefsson via Development of GNU Guix and the GNU System distribution.
                   ` (2 subsequent siblings)
  5 siblings, 1 reply; 21+ messages in thread
From: Leo Famulari @ 2023-04-25 16:09 UTC (permalink / raw)
  To: Andreas Enge, guix-devel

Thank you for leading this effort!

The value of leadership and project management is sometimes underappreciated in software projects, but they are essential for our success.

On Tue, Apr 25, 2023, at 10:09, Andreas Enge wrote:
> Hello all,
>
> I have just merged core-updates into master and deleted the branch!
> This has been a long adventure, which became particularly intensive
> after the last Guix Days in February. First and foremost many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.
>
> Each and every package is not yet in shape; please feel free to submit
> patches for your favourite packages that fail to build. In particular:
> - python-yubikey-manager does not build currently; work to correct this
>   is underway.
> - R on powerpc does not build; this will also be corrected soon;
> - aarch64 has very few substitutes; I think this is mainly due to the build
>   farm catching up and not so much to packages not building, but this is
>   difficult to know.
> If any of these are essential to you, you may wish to wait a little bit
> longer before a "guix pull", or for the time being pull to a commit just
> before the merge by issuing
>    guix pull --commit=472706ae2f9160833951a4e4bcc4c206e03097b0
>
> Every end of a story is the beginning of many new ones. Several areas of
> work have already been identified; I will summarise what came up on the
> mailing list and who expressed interest to work on it - please feel free
> to join if a topic interests you, or launch another initiative if you
> want to work on a different subject!
> - rust-team already has a branch that is almost ready to be built and
>   merged, as a precursor to the team based workflow that we need to invent
>   (Efraim Flashner).
> - R on powerpc64le needs to be built by changes to valgrind and lz4
>   (Simon Tournier, I).
> - Many Python packages need updates, in particular with the aim of building
>   python-yubikey-manager (John Kehayias, Lars-Dominik Braun, Brian Cully).
> - There is still work to do to bootstrap GHC until the latest version on
>   i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).
> - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
> - After the mesa update is before the mesa update, and it looks like more
>   features can be enabled (Kaelyn Takata, John Kehayias).
> - Too much in Guix depends on too much else, which makes building things
>   needlessly entangled; in particular time zone data should not be referred
>   to by packages, but be loaded at runtime (Leo Famulari).
>
> All the best in your Guix endeavours,
>
> Andreas


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Core-updates merge
  2023-04-25 14:57   ` Katherine Cox-Buday
@ 2023-04-25 19:27     ` Maxim Cournoyer
  0 siblings, 0 replies; 21+ messages in thread
From: Maxim Cournoyer @ 2023-04-25 19:27 UTC (permalink / raw)
  To: Andreas Enge; +Cc: Felix Lechner, Katherine Cox-Buday, guix-devel

Hello,

My gratitude for your patience and perseverance on merging the
core-updates branch :-).

Congratulations everyone!

-- 
Thanks,
Maxim


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Build dependency inflation (was: Re: Core-updates merge)
  2023-04-25 14:09 Core-updates merge Andreas Enge
                   ` (2 preceding siblings ...)
  2023-04-25 16:09 ` Leo Famulari
@ 2023-04-27 17:56 ` Simon Josefsson via Development of GNU Guix and the GNU System distribution.
  2023-04-27 23:28   ` bokr
                     ` (2 more replies)
  2023-04-28  5:55 ` Core-updates merge John Kehayias
  2023-04-28 14:17 ` Simon Tournier
  5 siblings, 3 replies; 21+ messages in thread
From: Simon Josefsson via Development of GNU Guix and the GNU System distribution. @ 2023-04-27 17:56 UTC (permalink / raw)
  To: Andreas Enge; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 1921 bytes --]

Andreas Enge <andreas@enge.fr> writes:

> - Too much in Guix depends on too much else, which makes building things
>   needlessly entangled; in particular time zone data should not be referred
>   to by packages, but be loaded at runtime (Leo Famulari).

This is an important open problem -- is there any way to attack this
problem in a systematic way?  I guess it is hard to understand which
packages ends up depending on what since it is a large graph with long
cycles, and also to understand which build depencies are essential and
which are superficial, and thus consequently challenging to know where
to start working to reduce build dependencies?

I wonder if it is possible to graph all the build dependencies, and do
something like a monte-carlo what-if simulation: randomly pick one
build-dependency from the entire build-dependency graph and remove it,
and recompute the graph.  If the difference between these two graphs
leads to a significantly lower total build computational cost than
before, we may be on to something.  My theory is that "true" build
dependencies will show up in so many places that removing just one
instance of it will not affect total build time.  But "needless" build
dependencies may occur just once or few times, and this approach would
catch low-hanging fruit to work on.  Maybe the simulation needs to be
done on more than just removing one build-dependency, it could play
what-if games removing two build-dependencies etc, or three random
build-dependencies, and so on.  Maybe my idea is flawed, and this will
only lead to a list of build-dependencies that are impossible to get rid
off anyway.

Is there some other method to understand what build dependencies would
be important to remove, to speed up total rebuild time?

Maybe we could analyze how much of a particular build-dependency
actually is used by a particular build?  By looking into file-access
patterns etc.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Build dependency inflation (was: Re: Core-updates merge)
  2023-04-27 17:56 ` Build dependency inflation (was: Re: Core-updates merge) Simon Josefsson via Development of GNU Guix and the GNU System distribution.
@ 2023-04-27 23:28   ` bokr
  2023-04-28 13:54   ` Simon Tournier
  2023-04-29  8:32   ` Build dependency inflation (was: Re: Core-updates merge) Christopher Baines
  2 siblings, 0 replies; 21+ messages in thread
From: bokr @ 2023-04-27 23:28 UTC (permalink / raw)
  To: Simon Josefsson; +Cc: Andreas Enge, guix-devel

Hi,
TL;DR:

If we have 22k sources, how about expressing dependency as a
22k x 22k matrix of weights Wij and dependency by
multiplication of 22k x 1 matrix os sources variables
ordered Sj by immediate dependency first. I guess the Sj
column matrix is transposed to do the multiplication, so the
Wij j's and Sj j's match.

Start with weights of only 1 0r 0. Later maybe measures of
compile times?

The diagonal Wjj would be all ones, and counts of ones in
vertical columns Wkj would be maximal if that Sj was
directly depended on for a lot of k's

Reordering the Wjj and Sj sort of like solving simultaneous
equations and colorizing somehow to show the clusters of Wij
dependency bits vs a backgrond of zero bits might make an
interesting display. WDYT?

Any GnuPlot whizzes out there? :-)

Regards,
Bengt Richter



On +2023-04-27 19:56:38 +0200, Simon Josefsson via Development of GNU Guix and the GNU System distribution. wrote:
> Andreas Enge <andreas@enge.fr> writes:
> 
> > - Too much in Guix depends on too much else, which makes building things
> >   needlessly entangled; in particular time zone data should not be referred
> >   to by packages, but be loaded at runtime (Leo Famulari).
> 
> This is an important open problem -- is there any way to attack this
> problem in a systematic way?  I guess it is hard to understand which
> packages ends up depending on what since it is a large graph with long
> cycles, and also to understand which build depencies are essential and
> which are superficial, and thus consequently challenging to know where
> to start working to reduce build dependencies?
> 
> I wonder if it is possible to graph all the build dependencies, and do
> something like a monte-carlo what-if simulation: randomly pick one
> build-dependency from the entire build-dependency graph and remove it,
> and recompute the graph.  If the difference between these two graphs
> leads to a significantly lower total build computational cost than
> before, we may be on to something.  My theory is that "true" build
> dependencies will show up in so many places that removing just one
> instance of it will not affect total build time.  But "needless" build
> dependencies may occur just once or few times, and this approach would
> catch low-hanging fruit to work on.  Maybe the simulation needs to be
> done on more than just removing one build-dependency, it could play
> what-if games removing two build-dependencies etc, or three random
> build-dependencies, and so on.  Maybe my idea is flawed, and this will
> only lead to a list of build-dependencies that are impossible to get rid
> off anyway.
> 
> Is there some other method to understand what build dependencies would
> be important to remove, to speed up total rebuild time?
> 
> Maybe we could analyze how much of a particular build-dependency
> actually is used by a particular build?  By looking into file-access
> patterns etc.
> 
> /Simon




^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Core-updates merge
  2023-04-25 14:09 Core-updates merge Andreas Enge
                   ` (3 preceding siblings ...)
  2023-04-27 17:56 ` Build dependency inflation (was: Re: Core-updates merge) Simon Josefsson via Development of GNU Guix and the GNU System distribution.
@ 2023-04-28  5:55 ` John Kehayias
  2023-04-28 14:17 ` Simon Tournier
  5 siblings, 0 replies; 21+ messages in thread
From: John Kehayias @ 2023-04-28  5:55 UTC (permalink / raw)
  To: Andreas Enge; +Cc: guix-devel

Dear Andreas and fellow Guix-ers,

On Tue, Apr 25, 2023 at 04:09 PM, Andreas Enge wrote:

> Hello all,
>
> I have just merged core-updates into master and deleted the branch!
> This has been a long adventure, which became particularly intensive
> after the last Guix Days in February. First and foremost many thanks to
> everyone who contributed to the branch, be it by commits, discussions or
> by working on the infrastructure.
>

As others have said, a big thank you to you once again for leading the
charge! Much appreciated!

> Each and every package is not yet in shape; please feel free to submit
> patches for your favourite packages that fail to build. In particular:
> - python-yubikey-manager does not build currently; work to correct this
>   is underway.

I've just submitted <https://issues.guix.gnu.org/63139> which does a bit
more than just fix that package and would be good for a Python feature
branch. I'll send the cover letter to this list for wider visibility,
though the Python team was cc'ed on the series. I suppose I should add
myself to the team after this.

> - R on powerpc does not build; this will also be corrected soon;
> - aarch64 has very few substitutes; I think this is mainly due to the build
>   farm catching up and not so much to packages not building, but this is
>   difficult to know.
> If any of these are essential to you, you may wish to wait a little bit
> longer before a "guix pull", or for the time being pull to a commit just
> before the merge by issuing
>    guix pull --commit=472706ae2f9160833951a4e4bcc4c206e03097b0
>
> Every end of a story is the beginning of many new ones. Several areas of
> work have already been identified; I will summarise what came up on the
> mailing list and who expressed interest to work on it - please feel free
> to join if a topic interests you, or launch another initiative if you
> want to work on a different subject!
> - rust-team already has a branch that is almost ready to be built and
>   merged, as a precursor to the team based workflow that we need to invent
>   (Efraim Flashner).
> - R on powerpc64le needs to be built by changes to valgrind and lz4
>   (Simon Tournier, I).
> - Many Python packages need updates, in particular with the aim of building
>   python-yubikey-manager (John Kehayias, Lars-Dominik Braun, Brian Cully).

I'll have to see what other patches we want to add to my series for
the branch and creating/building that as soon as possible. There is
some feedback needed on the deeper changes in my patch series, but
hopefully won't take much to finalize.

> - There is still work to do to bootstrap GHC until the latest version on
>   i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).
> - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
> - After the mesa update is before the mesa update, and it looks like more
>   features can be enabled (Kaelyn Takata, John Kehayias).

When I get a chance next I'll be pulling the relevant patches locally
and building as a first check. And then we'll want this feature branch
to be created and built for wider testing.

> - Too much in Guix depends on too much else, which makes building things
>   needlessly entangled; in particular time zone data should not be referred
>   to by packages, but be loaded at runtime (Leo Famulari).
>
> All the best in your Guix endeavours,
>
> Andreas

Thanks everyone!
John



^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Build dependency inflation (was: Re: Core-updates merge)
  2023-04-27 17:56 ` Build dependency inflation (was: Re: Core-updates merge) Simon Josefsson via Development of GNU Guix and the GNU System distribution.
  2023-04-27 23:28   ` bokr
@ 2023-04-28 13:54   ` Simon Tournier
  2023-05-03 21:10     ` Build dependency inflation Ludovic Courtès
  2023-04-29  8:32   ` Build dependency inflation (was: Re: Core-updates merge) Christopher Baines
  2 siblings, 1 reply; 21+ messages in thread
From: Simon Tournier @ 2023-04-28 13:54 UTC (permalink / raw)
  To: Simon Josefsson, Andreas Enge; +Cc: guix-devel

Hi,

On jeu., 27 avril 2023 at 19:56, Simon Josefsson via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org> wrote:
> Andreas Enge <andreas@enge.fr> writes:
>
>> - Too much in Guix depends on too much else, which makes building things
>>   needlessly entangled; in particular time zone data should not be referred
>>   to by packages, but be loaded at runtime (Leo Famulari).
>
> This is an important open problem -- is there any way to attack this
> problem in a systematic way?  I guess it is hard to understand which
> packages ends up depending on what since it is a large graph with long
> cycles, and also to understand which build depencies are essential and
> which are superficial, and thus consequently challenging to know where
> to start working to reduce build dependencies?

Hum, I am not to get the “large graph with long cycles” part.  Since the
graph for packages is a DAG.

Well, “guix graph” and “guix graph --path” help to spot the chain of
dependencies between two packages that seem unrelated.

From my point of view, we should increase the tooling to inspect these
graphs.  Other said, add feature to “guix graph”.

At some point, it could be nice to have “guile-igraph” a Guile binding
of igraph [1].

Last, back on December [2], I started to apply some well-known network
algorithm (link analysis, centrality measure, pagerank, etc.) to the
graph of packages.  From my point of view, these kind of tools could be
very helpful to spot out the packages that need some specific care.

Maybe it would fit a project for a student. ;-)


1: https://igraph.org/
2: https://yhetil.org/guix/874ju4qyd4.fsf@gmail.com
   Some stats about the graph of dependencies
   Fri, 09 Dec 2022 18:29:43 +0100
   id:874ju4qyd4.fsf@gmail.com


Cheers,
simon


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Core-updates merge
  2023-04-25 14:09 Core-updates merge Andreas Enge
                   ` (4 preceding siblings ...)
  2023-04-28  5:55 ` Core-updates merge John Kehayias
@ 2023-04-28 14:17 ` Simon Tournier
  2023-04-30  6:05   ` ocaml bootstrap (was Re: Core-updates merge) Efraim Flashner
  2023-05-03 21:15   ` OCaml bootstrap Ludovic Courtès
  5 siblings, 2 replies; 21+ messages in thread
From: Simon Tournier @ 2023-04-28 14:17 UTC (permalink / raw)
  To: Andreas Enge, guix-devel; +Cc: Lars-Dominik Braun, Julien Lepiller

Hi Andreas,

On mar., 25 avril 2023 at 16:09, Andreas Enge <andreas@enge.fr> wrote:


> I have just merged core-updates into master and deleted the branch!

Awesome!  Thank you for your patient leadership over the past months. :-)


> - R on powerpc64le needs to be built by changes to valgrind and lz4
>   (Simon Tournier, I).

Now, we are good, right?


> - There is still work to do to bootstrap GHC until the latest version on
>   i686, and to potentially shorten the bootstrap chain (Lars-Dominik Braun).

Argh, I will try to finish my patches soon.  I have:

 + turned off the test suite for ’ghc’ – it allows to not be blocked,
 + introduced ’ghc-testsuite’ to run the test suite,
 + introduced ’ghc-toolchain’ which depends on ’ghc’ and ’ghc-testsuite’
   – well, ’ghc’ being hidden.  Something similar as GCC.
 + shorten the chain as discussed elsewhere [1].

1: https://yhetil.org/guix/87r0siem5c.fsf@gmail.com


> - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).

Well, 4.07 is the version that is de-bootstrapped, i.e. bootstrapped
using ’camlboot’ via Guile – for details see [2].

However, higher versions (4.09, 4.14, 5) does not use this seed and thus
they are not de-bootstrapped.  Well, I do not know the status upstream;
from my point of view, we have two options:

 a) Agree with other distros and OCaml folks to rely on a common OCaml
 4.07 bootstrapped using camlboot and then use this OCaml 4.07 as the
 seed for the subsequent versions.  Somehow having a way to verify the
 current OCaml compiler without running again and again via camlboot.

 b) Build ourselves a chain from 4.07 bootstrapped with camlboot to
 modern OCaml compilers.  However, each time we modify one dependency of
 camlboot, it means rebuild the complete chain.  Well, bootstrapping via
 camlboot can be very slow and I do not know if we have the resources
 for non-x86_64 architecture.  Here, the list of the emerged
 dependencies:

    $ guix graph camlboot -t bag-emerged | grep label | cut -d'=' -f2
     "camlboot@0.0.0-1.45045d0", shape 
     "guile@3.0.9", shape 
     "pkg-config@0.29.2", shape 
     "tar@1.34", shape 
     "gzip@1.12", shape 
     "bzip2@1.0.8", shape 
     "file@5.44", shape 
     "diffutils@3.8", shape 
     "patch@2.7.6", shape 
     "findutils@4.9.0", shape 
     "gawk@5.2.1", shape 
     "sed@4.8", shape 
     "grep@3.8", shape 
     "xz@5.2.8", shape 
     "coreutils@9.1", shape 
     "make@4.3", shape 
     "bash-minimal@5.1.16", shape 
     "ld-wrapper@0", shape 
     "binutils@2.38", shape 
     "gcc@11.3.0", shape 
     "glibc@2.35", shape 
     "glibc-utf8-locales@2.35", shape 
     "libffi@3.4.4", shape 
     "bash-minimal@5.1.16", shape 
     "libunistring@1.0", shape 
     "libgc@8.2.2", shape

 c) Fix the dependencies of camlboot.


Well, it seems a separated discussion but it echoes the recent blog post [3]
about “The Full-Source Bootstrap”. :-)

2: https://10years.guix.gnu.org/video/camlboot-debootstrapping-the-ocaml-compiler
3: https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/


Cheers,
simon




^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Build dependency inflation (was: Re: Core-updates merge)
  2023-04-27 17:56 ` Build dependency inflation (was: Re: Core-updates merge) Simon Josefsson via Development of GNU Guix and the GNU System distribution.
  2023-04-27 23:28   ` bokr
  2023-04-28 13:54   ` Simon Tournier
@ 2023-04-29  8:32   ` Christopher Baines
  2 siblings, 0 replies; 21+ messages in thread
From: Christopher Baines @ 2023-04-29  8:32 UTC (permalink / raw)
  To: Simon Josefsson; +Cc: guix-devel

[-- Attachment #1: Type: text/plain, Size: 2347 bytes --]


Simon Josefsson via "Development of GNU Guix and the GNU System distribution." <guix-devel@gnu.org> writes:

> [[PGP Signed Part:Undecided]]
> Andreas Enge <andreas@enge.fr> writes:
>
>> - Too much in Guix depends on too much else, which makes building things
>>   needlessly entangled; in particular time zone data should not be referred
>>   to by packages, but be loaded at runtime (Leo Famulari).
>
> This is an important open problem -- is there any way to attack this
> problem in a systematic way?  I guess it is hard to understand which
> packages ends up depending on what since it is a large graph with long
> cycles, and also to understand which build depencies are essential and
> which are superficial, and thus consequently challenging to know where
> to start working to reduce build dependencies?
>
> I wonder if it is possible to graph all the build dependencies, and do
> something like a monte-carlo what-if simulation: randomly pick one
> build-dependency from the entire build-dependency graph and remove it,
> and recompute the graph.  If the difference between these two graphs
> leads to a significantly lower total build computational cost than
> before, we may be on to something.  My theory is that "true" build
> dependencies will show up in so many places that removing just one
> instance of it will not affect total build time.  But "needless" build
> dependencies may occur just once or few times, and this approach would
> catch low-hanging fruit to work on.  Maybe the simulation needs to be
> done on more than just removing one build-dependency, it could play
> what-if games removing two build-dependencies etc, or three random
> build-dependencies, and so on.  Maybe my idea is flawed, and this will
> only lead to a list of build-dependencies that are impossible to get rid
> off anyway.
>
> Is there some other method to understand what build dependencies would
> be important to remove, to speed up total rebuild time?
>
> Maybe we could analyze how much of a particular build-dependency
> actually is used by a particular build?  By looking into file-access
> patterns etc.

This is something I'd like to see incorporated in to qa.guix.gnu.org for
patch (and branch) review. While one off analysis is good, I think it's
most important to be able to look at changes and see how they change the
situation.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 987 bytes --]

^ permalink raw reply	[flat|nested] 21+ messages in thread

* ocaml bootstrap (was Re: Core-updates merge)
  2023-04-28 14:17 ` Simon Tournier
@ 2023-04-30  6:05   ` Efraim Flashner
  2023-05-03 21:15   ` OCaml bootstrap Ludovic Courtès
  1 sibling, 0 replies; 21+ messages in thread
From: Efraim Flashner @ 2023-04-30  6:05 UTC (permalink / raw)
  To: Simon Tournier
  Cc: Andreas Enge, guix-devel, Lars-Dominik Braun, Julien Lepiller

[-- Attachment #1: Type: text/plain, Size: 3422 bytes --]

On Fri, Apr 28, 2023 at 04:17:40PM +0200, Simon Tournier wrote:
> Hi Andreas,
> 
> On mar., 25 avril 2023 at 16:09, Andreas Enge <andreas@enge.fr> wrote:
> 
> > - OCaml could be simplified by dropping version 4.07 (Julien Lepiller).
> 
> Well, 4.07 is the version that is de-bootstrapped, i.e. bootstrapped
> using ’camlboot’ via Guile – for details see [2].
> 
> However, higher versions (4.09, 4.14, 5) does not use this seed and thus
> they are not de-bootstrapped.  Well, I do not know the status upstream;
> from my point of view, we have two options:
> 
>  a) Agree with other distros and OCaml folks to rely on a common OCaml
>  4.07 bootstrapped using camlboot and then use this OCaml 4.07 as the
>  seed for the subsequent versions.  Somehow having a way to verify the
>  current OCaml compiler without running again and again via camlboot.
> 
>  b) Build ourselves a chain from 4.07 bootstrapped with camlboot to
>  modern OCaml compilers.  However, each time we modify one dependency of
>  camlboot, it means rebuild the complete chain.  Well, bootstrapping via
>  camlboot can be very slow and I do not know if we have the resources
>  for non-x86_64 architecture.  Here, the list of the emerged
>  dependencies:

I think similarly to the very slow *-mes part of the bootstrap chain, we
have to realize that some parts are just very slow. (A vote for b)

I built camlboot once on aarch64, probably a year ago. IIRC it took more
than 24 hours. I will say that it is doable without needing 8GB of ram,
which is also a limiting factor for the slow architectures.

>     $ guix graph camlboot -t bag-emerged | grep label | cut -d'=' -f2
>      "camlboot@0.0.0-1.45045d0", shape 
>      "guile@3.0.9", shape 
>      "pkg-config@0.29.2", shape 
>      "tar@1.34", shape 
>      "gzip@1.12", shape 
>      "bzip2@1.0.8", shape 
>      "file@5.44", shape 
>      "diffutils@3.8", shape 
>      "patch@2.7.6", shape 
>      "findutils@4.9.0", shape 
>      "gawk@5.2.1", shape 
>      "sed@4.8", shape 
>      "grep@3.8", shape 
>      "xz@5.2.8", shape 
>      "coreutils@9.1", shape 
>      "make@4.3", shape 
>      "bash-minimal@5.1.16", shape 
>      "ld-wrapper@0", shape 
>      "binutils@2.38", shape 
>      "gcc@11.3.0", shape 
>      "glibc@2.35", shape 
>      "glibc-utf8-locales@2.35", shape 
>      "libffi@3.4.4", shape 
>      "bash-minimal@5.1.16", shape 
>      "libunistring@1.0", shape 
>      "libgc@8.2.2", shape
>
>  c) Fix the dependencies of camlboot.

Looking at camlboot, there's a native input of guile-3.0. Unless we want
to bootstrap using some of the boot0 packages or actually find the
minimum set of packages needed and remove the rest that are brought in
by implicit-inputs (recursively) I think it's already as small as
possible.

> 
> Well, it seems a separated discussion but it echoes the recent blog post [3]
> about “The Full-Source Bootstrap”. :-)
> 
> 2: https://10years.guix.gnu.org/video/camlboot-debootstrapping-the-ocaml-compiler
> 3: https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/
> 
> 
> Cheers,
> simon
> 
> 
> 

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Core-updates merge
  2023-04-25 16:09 ` Leo Famulari
@ 2023-05-03 21:07   ` Ludovic Courtès
  0 siblings, 0 replies; 21+ messages in thread
From: Ludovic Courtès @ 2023-05-03 21:07 UTC (permalink / raw)
  To: Leo Famulari; +Cc: Andreas Enge, guix-devel

Hello!

"Leo Famulari" <leo@famulari.name> skribis:

> Thank you for leading this effort!
>
> The value of leadership and project management is sometimes underappreciated in software projects, but they are essential for our success.

Agreed!  You did a great job Andreas at coordinating all the work *and*
actually doing a lot of it—thank you, and thanks to everyone who
tirelessly worked on fixing things!  Really happy to be enjoying the
fruits of this hard labor now.  :-)

Ludo’.


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Build dependency inflation
  2023-04-28 13:54   ` Simon Tournier
@ 2023-05-03 21:10     ` Ludovic Courtès
  2023-05-04  8:20       ` Simon Tournier
  0 siblings, 1 reply; 21+ messages in thread
From: Ludovic Courtès @ 2023-05-03 21:10 UTC (permalink / raw)
  To: Simon Tournier; +Cc: Simon Josefsson, Andreas Enge, guix-devel

Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Last, back on December [2], I started to apply some well-known network
> algorithm (link analysis, centrality measure, pagerank, etc.) to the
> graph of packages.  From my point of view, these kind of tools could be
> very helpful to spot out the packages that need some specific care.
>
> Maybe it would fit a project for a student. ;-)

Yes, I think we need more work along the lines of what you did!

We could also couple the actual graph with data such as individual
output size.

Ludo’.


^ permalink raw reply	[flat|nested] 21+ messages in thread

* OCaml bootstrap
  2023-04-28 14:17 ` Simon Tournier
  2023-04-30  6:05   ` ocaml bootstrap (was Re: Core-updates merge) Efraim Flashner
@ 2023-05-03 21:15   ` Ludovic Courtès
  2023-05-03 21:25     ` Julien Lepiller
  1 sibling, 1 reply; 21+ messages in thread
From: Ludovic Courtès @ 2023-05-03 21:15 UTC (permalink / raw)
  To: Simon Tournier
  Cc: Andreas Enge, guix-devel, Lars-Dominik Braun, Julien Lepiller

Hello,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Well, 4.07 is the version that is de-bootstrapped, i.e. bootstrapped
> using ’camlboot’ via Guile – for details see [2].
>
> However, higher versions (4.09, 4.14, 5) does not use this seed and thus
> they are not de-bootstrapped.  Well, I do not know the status upstream;
> from my point of view, we have two options:
>
>  a) Agree with other distros and OCaml folks to rely on a common OCaml
>  4.07 bootstrapped using camlboot and then use this OCaml 4.07 as the
>  seed for the subsequent versions.  Somehow having a way to verify the
>  current OCaml compiler without running again and again via camlboot.
>
>  b) Build ourselves a chain from 4.07 bootstrapped with camlboot to
>  modern OCaml compilers.  However, each time we modify one dependency of
>  camlboot, it means rebuild the complete chain.  Well, bootstrapping via
>  camlboot can be very slow and I do not know if we have the resources
>  for non-x86_64 architecture.  Here, the list of the emerged
>  dependencies:

Julien, do you happen to know if there are plans to make camlboot more
capable so it can be used to build newer versions of OCaml?  Maybe
something to discuss with Nathanaëlle Courant and Gabriel Scherer?

Ludo’.


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: OCaml bootstrap
  2023-05-03 21:15   ` OCaml bootstrap Ludovic Courtès
@ 2023-05-03 21:25     ` Julien Lepiller
  2023-05-04  8:41       ` Simon Tournier
  0 siblings, 1 reply; 21+ messages in thread
From: Julien Lepiller @ 2023-05-03 21:25 UTC (permalink / raw)
  To: Ludovic Courtès, Simon Tournier
  Cc: Andreas Enge, guix-devel, Lars-Dominik Braun

We had some discussion here, but there's still some work to do: https://github.com/Ekdohibs/camlboot/issues/59

Le 3 mai 2023 23:15:49 GMT+02:00, "Ludovic Courtès" <ludo@gnu.org> a écrit :
>Hello,
>
>Simon Tournier <zimon.toutoune@gmail.com> skribis:
>
>> Well, 4.07 is the version that is de-bootstrapped, i.e. bootstrapped
>> using ’camlboot’ via Guile – for details see [2].
>>
>> However, higher versions (4.09, 4.14, 5) does not use this seed and thus
>> they are not de-bootstrapped.  Well, I do not know the status upstream;
>> from my point of view, we have two options:
>>
>>  a) Agree with other distros and OCaml folks to rely on a common OCaml
>>  4.07 bootstrapped using camlboot and then use this OCaml 4.07 as the
>>  seed for the subsequent versions.  Somehow having a way to verify the
>>  current OCaml compiler without running again and again via camlboot.
>>
>>  b) Build ourselves a chain from 4.07 bootstrapped with camlboot to
>>  modern OCaml compilers.  However, each time we modify one dependency of
>>  camlboot, it means rebuild the complete chain.  Well, bootstrapping via
>>  camlboot can be very slow and I do not know if we have the resources
>>  for non-x86_64 architecture.  Here, the list of the emerged
>>  dependencies:
>
>Julien, do you happen to know if there are plans to make camlboot more
>capable so it can be used to build newer versions of OCaml?  Maybe
>something to discuss with Nathanaëlle Courant and Gabriel Scherer?
>
>Ludo’.


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: Build dependency inflation
  2023-05-03 21:10     ` Build dependency inflation Ludovic Courtès
@ 2023-05-04  8:20       ` Simon Tournier
  0 siblings, 0 replies; 21+ messages in thread
From: Simon Tournier @ 2023-05-04  8:20 UTC (permalink / raw)
  To: Ludovic Courtès; +Cc: Simon Josefsson, Andreas Enge, guix-devel

Hi,

On Wed, 03 May 2023 at 23:10, Ludovic Courtès <ludo@gnu.org> wrote:

> We could also couple the actual graph with data such as individual
> output size.

Oh, that’s a cool idea: apply network algorithms to a weighted graph.

It looks like a nice student project. :-)

Cheers,
simon


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: OCaml bootstrap
  2023-05-03 21:25     ` Julien Lepiller
@ 2023-05-04  8:41       ` Simon Tournier
  2023-05-04 10:01         ` Julien Lepiller
  0 siblings, 1 reply; 21+ messages in thread
From: Simon Tournier @ 2023-05-04  8:41 UTC (permalink / raw)
  To: Julien Lepiller, Ludovic Courtès
  Cc: Andreas Enge, guix-devel, Lars-Dominik Braun

Hi,

On Wed, 03 May 2023 at 23:25, Julien Lepiller <julien@lepiller.eu> wrote:

>>Julien, do you happen to know if there are plans to make camlboot more
>>capable so it can be used to build newer versions of OCaml?  Maybe
>>something to discuss with Nathanaëlle Courant and Gabriel Scherer?

> We had some discussion here, but there's still some work to do:
> https://github.com/Ekdohibs/camlboot/issues/59

Cool!

Just to be sure, the discussion is from one year ago, right?  Any
update?


Well, rehashing, I think we could consider a full-bootstrap from source
for OCaml.  Whatever the seed (4.07 or 4.09), we could consider it as
done and fixed.  Here all the “inputs”:

--8<---------------cut here---------------start------------->8---
     "bash-minimal@5.1.16" 
     "bash-minimal@5.1.16" 
     "binutils@2.38" 
     "bzip2@1.0.8" 
     "coreutils@9.1" 
     "diffutils@3.8" 
     "file@5.44" 
     "findutils@4.9.0" 
     "gawk@5.2.1" 
     "gcc@11.3.0" 
     "glibc-utf8-locales@2.35" 
     "glibc@2.35" 
     "grep@3.8" 
     "guile@3.0.9" 
     "gzip@1.12" 
     "ld-wrapper@0" 
     "libffi@3.4.4" 
     "libgc@8.2.2"
     "libunistring@1.0" 
     "make@4.3" 
     "patch@2.7.6" 
     "pkg-config@0.29.2" 
     "sed@4.8" 
     "tar@1.34" 
     "xz@5.2.8" 
--8<---------------cut here---------------end--------------->8---

Only guile-3.0 is not a package deep in the graph.  All the others are.
My question is: do we want to rebuild camlboot and then all the OCaml
world each time we update one of these “inputs“?

For example, what does it bring on the table to rebuild camlboot because
grep had been updated?


Cheers,
simon


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: OCaml bootstrap
  2023-05-04  8:41       ` Simon Tournier
@ 2023-05-04 10:01         ` Julien Lepiller
  2023-05-04 18:26           ` Simon Tournier
  0 siblings, 1 reply; 21+ messages in thread
From: Julien Lepiller @ 2023-05-04 10:01 UTC (permalink / raw)
  To: Simon Tournier, Ludovic Courtès
  Cc: Andreas Enge, guix-devel, Lars-Dominik Braun



Le 4 mai 2023 10:41:18 GMT+02:00, Simon Tournier <zimon.toutoune@gmail.com> a écrit :
>Hi,
>
>On Wed, 03 May 2023 at 23:25, Julien Lepiller <julien@lepiller.eu> wrote:
>
>>>Julien, do you happen to know if there are plans to make camlboot more
>>>capable so it can be used to build newer versions of OCaml?  Maybe
>>>something to discuss with Nathanaëlle Courant and Gabriel Scherer?
>
>> We had some discussion here, but there's still some work to do:
>> https://github.com/Ekdohibs/camlboot/issues/59
>
>Cool!
>
>Just to be sure, the discussion is from one year ago, right?  Any
>update?

No updates. I don't think any of us worked more on that than you can read.

>
>Well, rehashing, I think we could consider a full-bootstrap from source
>for OCaml.  Whatever the seed (4.07 or 4.09), we could consider it as
>done and fixed.  Here all the “inputs”:
>
>--8<---------------cut here---------------start------------->8---
>     "bash-minimal@5.1.16" 
>     "bash-minimal@5.1.16" 
>     "binutils@2.38" 
>     "bzip2@1.0.8" 
>     "coreutils@9.1" 
>     "diffutils@3.8" 
>     "file@5.44" 
>     "findutils@4.9.0" 
>     "gawk@5.2.1" 
>     "gcc@11.3.0" 
>     "glibc-utf8-locales@2.35" 
>     "glibc@2.35" 
>     "grep@3.8" 
>     "guile@3.0.9" 
>     "gzip@1.12" 
>     "ld-wrapper@0" 
>     "libffi@3.4.4" 
>     "libgc@8.2.2"
>     "libunistring@1.0" 
>     "make@4.3" 
>     "patch@2.7.6" 
>     "pkg-config@0.29.2" 
>     "sed@4.8" 
>     "tar@1.34" 
>     "xz@5.2.8" 
>--8<---------------cut here---------------end--------------->8---
>
>Only guile-3.0 is not a package deep in the graph.  All the others are.
>My question is: do we want to rebuild camlboot and then all the OCaml
>world each time we update one of these “inputs“?
>
>For example, what does it bring on the table to rebuild camlboot because
>grep had been updated?

I don't understand what you suggest here. I think grep is used in the Makefile, so we can't just remove it from the inputs. What would not building camlboot when grep is updated look like?

Have a grep-for-build that is never updated? Build camlboot once and repackage the binary (making it a bootstrap seed)?

>
>
>Cheers,
>simon


^ permalink raw reply	[flat|nested] 21+ messages in thread

* Re: OCaml bootstrap
  2023-05-04 10:01         ` Julien Lepiller
@ 2023-05-04 18:26           ` Simon Tournier
  0 siblings, 0 replies; 21+ messages in thread
From: Simon Tournier @ 2023-05-04 18:26 UTC (permalink / raw)
  To: Julien Lepiller, Ludovic Courtès
  Cc: Andreas Enge, guix-devel, Lars-Dominik Braun

Hi Julien,

On jeu., 04 mai 2023 at 12:01, Julien Lepiller <julien@lepiller.eu> wrote:

> Have a grep-for-build that is never updated? Build camlboot once and
> repackage the binary (making it a bootstrap seed)? 

Yes, something like that.

Well, the packages that camlboot depends on barely change because most
of them are “core-updates” packages.  However, still.

Some packages (autotools) use guile-3.0/pinned instead of guile-3.0.
Other gdb/pinned (rust).  Etc.

Therefore we could use something like /pinned (or -boot or else) for
building camlboot once.  This allows transparency and being able to
rebuild if the worst is necessary.

But then, we consider the binary as the bootstrap seed of OCaml world.

Maybe, it could be discussed with the OCaml community.

Well, I do not know if it is worth but somehow I think that a similar
strategy as MES and bootstrapping C could be applied for camlboot and
bootstrapping OCaml.  Something like gnu/packages/ocaml-commencement.scm
that bootstraps the OCaml world and this would depend on packages that
are tweaked with a lot of care.  Similarly as the MES bootstrap chain.


Cheers,
simon


^ permalink raw reply	[flat|nested] 21+ messages in thread

end of thread, other threads:[~2023-05-04 18:27 UTC | newest]

Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-04-25 14:09 Core-updates merge Andreas Enge
2023-04-25 14:40 ` Felix Lechner via Development of GNU Guix and the GNU System distribution.
2023-04-25 14:57   ` Katherine Cox-Buday
2023-04-25 19:27     ` Maxim Cournoyer
2023-04-25 15:31 ` Josselin Poiret
2023-04-25 16:09 ` Leo Famulari
2023-05-03 21:07   ` Ludovic Courtès
2023-04-27 17:56 ` Build dependency inflation (was: Re: Core-updates merge) Simon Josefsson via Development of GNU Guix and the GNU System distribution.
2023-04-27 23:28   ` bokr
2023-04-28 13:54   ` Simon Tournier
2023-05-03 21:10     ` Build dependency inflation Ludovic Courtès
2023-05-04  8:20       ` Simon Tournier
2023-04-29  8:32   ` Build dependency inflation (was: Re: Core-updates merge) Christopher Baines
2023-04-28  5:55 ` Core-updates merge John Kehayias
2023-04-28 14:17 ` Simon Tournier
2023-04-30  6:05   ` ocaml bootstrap (was Re: Core-updates merge) Efraim Flashner
2023-05-03 21:15   ` OCaml bootstrap Ludovic Courtès
2023-05-03 21:25     ` Julien Lepiller
2023-05-04  8:41       ` Simon Tournier
2023-05-04 10:01         ` Julien Lepiller
2023-05-04 18:26           ` Simon Tournier

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/guix.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.