bootstrap integration strategies

bootstrap integration strategies

[Jan Nieuwenhuizen]

Hi!

With Mes 0.16 released I felt that after two years of straight hacking
I was pretty much done.  Hmm...

On the wip-bootstrap branch we have these packages

    binutils-2.20.1, gcc-2.95.3, and glibc-2.2.5

built without using any of these 3 main tools from the bootstrap
binaries.

Using these we have built gcc-4.1.0 and glibc-2.3.6.

We do however, use all these bootstrap binaries/binary seeds

    bash, bison, coreutils, diffutils, file, flex, gawk, grep, gzip,
    make, sed, tar

...and guile.

Also: this Mes bootstrap is currently x86-linux only.

And the above is an over-simplification, here's the list of seeds and
packages in order:

    %mescc-tools-seed
    %mes-seed
    %tinycc-seed
    mescc-tools-boot
    mes-boot
    nyacc-boot
    tcc-boot-0@0.9.26
    tcc-boot@0.9.27
    binutils-mesboot@2.20.1a
    gcc-core-mesboot@2.95.3
    glibc-mesboot@2.2.5
    gcc-mesboot0@2.95.3
    binutils-mesboot@2.20.1a
    gcc-mesboot@4.1.0
    [glibc-mesboot@2.3.6]

I hoped to get gcc-4.7.4 packaged easily but haven't succeeded in two
weeks.  That means I'm stuck[0].  Much has happened: I'm helping to get
mescc-tools, Nyacc and Mes packaged in Debian, am writing some
documentation[1], got a preliminary hello-world on Hurd, worked on Gash[2]
to get it on par with bournish, applied for a GNU evaluation of Mes...

If we get gcc-4.7.4 built for x86 that's nice, but can we bootstrap
other architectures from that?  Do we want that?  We probably do not
want different bootstrap paths for different architectures.  What
about the Hurd?

Let's say we have gcc-4.7.4, then it would be nice to eliminate other
bootstrap binaries.  Obvious places to start are bash and make.  As
Guix we depend on the Guile bootstrap binary most.

That is why I've also been working on Gash--Guile As SHell--by
Rutger[cc] lately.  If we make Gash powerful enough to replace bash to
build make and some of the other bootstrap binaries, that would be
nice.  After a promising re-start it now proved that the heart of
Gash, the SH grammar in PEG and its subsequent transformations needs
an overhaul.  That's being worked on but there is not much visible
progress.

So, where to go?  Integrate the Mes x86 bootstrap and build other
architectures on top of that (is that even possible)?  Add the Mes
bootstrap as an additional, weird `x86-linux' architecture to mature
first?  Add other architectures to Mes and keep one bootstrap path in
Guix?  Finish the x86 bootstrap/gcc-4.7.4 (I need help!)?

Lots I haven't mentioned, hoping to get a discussion going and to create
something of a plan.  Thoughts?

Greetings,
janneke

[0] https://gcc.gnu.org/ml/gcc-help/2018-07/msg00060.html
[1] http://lilypond.org/janneke/mes
[2] http://gitlab.com/rutger.van.beusekom/gash
    friendly fork with bournish commands: http://gitlab.com/janneke/gash

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrap integration strategies

[Ludovic Courtès]

Hello Jan!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> With Mes 0.16 released I felt that after two years of straight hacking
> I was pretty much done.  Hmm...
>
> On the wip-bootstrap branch we have these packages
>
>     binutils-2.20.1, gcc-2.95.3, and glibc-2.2.5
>
> built without using any of these 3 main tools from the bootstrap
> binaries.
>
> Using these we have built gcc-4.1.0 and glibc-2.3.6.

I’ve said it before, but really: thumbs up to you and everyone at
#bootstrapable for getting this far!  It’s something that seemed like a
pipe dream not so long ago, so it’s really amazing that you made it a
reality.

> And the above is an over-simplification, here's the list of seeds and
> packages in order:
>
>     %mescc-tools-seed
>     %mes-seed
>     %tinycc-seed
>     mescc-tools-boot
>     mes-boot
>     nyacc-boot
>     tcc-boot-0@0.9.26
>     tcc-boot@0.9.27
>     binutils-mesboot@2.20.1a
>     gcc-core-mesboot@2.95.3
>     glibc-mesboot@2.2.5
>     gcc-mesboot0@2.95.3
>     binutils-mesboot@2.20.1a
>     gcc-mesboot@4.1.0
>     [glibc-mesboot@2.3.6]

Is tcc-boot a seed, or is it built from source using MesCC?

> I hoped to get gcc-4.7.4 packaged easily but haven't succeeded in two
> weeks.  That means I'm stuck[0].  Much has happened: I'm helping to get
> mescc-tools, Nyacc and Mes packaged in Debian, am writing some
> documentation[1], got a preliminary hello-world on Hurd, worked on Gash[2]
> to get it on par with bournish, applied for a GNU evaluation of Mes...

Woow.

> If we get gcc-4.7.4 built for x86 that's nice, but can we bootstrap
> other architectures from that?  Do we want that?  We probably do not
> want different bootstrap paths for different architectures.  What
> about the Hurd?

I think that’s the main difficulty.  I think we’d rather not have
separate bootstrap paths for Intel GNU/Linux on one hand, and everything
else on the other hand.

Yet, we know that porting what you already did on x86-linux-gnu to
GNU/Hurd and ARMv7 and AArch64 etc. is going to be a lot of non-trivial
work (especially since historical versions of the GNU toolchain did not
support AArch64, for instance.)

Waiting for this to be “solved” (and we don’t even know how) would
equate to a status quo.  But obviously, it’d be sad to have all this
work already done on Intel and not be able to benefit from it.

So perhaps we’ll have to get over it and have a different bootstrap path
on x86-linux-gnu.

(BTW, I suspect we can get away with using 32-bit bootstrap binaries on
both i686/x86_64 and armv7/aarch64, no?)

> That is why I've also been working on Gash--Guile As SHell--by
> Rutger[cc] lately.  If we make Gash powerful enough to replace bash to
> build make and some of the other bootstrap binaries, that would be
> nice.  After a promising re-start it now proved that the heart of
> Gash, the SH grammar in PEG and its subsequent transformations needs
> an overhaul.  That's being worked on but there is not much visible
> progress.

Gash seems to be a low-hanging fruit and a relatively easy thing,
because it’s architecture-independent.  How
far is it from being able to run typical ‘configure’ scripts?

I think the day it’s able to run ‘configure’ scripts, we can switch to
it right away without further ado, and then incrementally improve it as
we stumble upon limitations and bugs.

WDYT?

> So, where to go?  Integrate the Mes x86 bootstrap and build other
> architectures on top of that (is that even possible)?  Add the Mes
> bootstrap as an additional, weird `x86-linux' architecture to mature
> first?  Add other architectures to Mes and keep one bootstrap path in
> Guix?  Finish the x86 bootstrap/gcc-4.7.4 (I need help!)?

If you’d like to take a break from the hardcode GCC bootstrap path ;-),
I think Gash for bootstrap is a good quickly-rewarding task.

There’s also another option you didn’t mention: ditching the 2.0
bootstrap Guile in favor of Mes.  That can be done in several steps:

  1. Replace the guile-2.0.*.xz binary tarballs with Mes, and add a step
     that builds Guile 2.x using our big bootstrap GCC binary.

  2. Same, but build Guile 2.x, libgc, etc. using MesCC.

This could allow us to remove quite a lot of KiBs from our binary seeds.

Thoughts?

Thank you!

Ludo’.

Re: bootstrap integration strategies

[Orians, Jeremiah (DTMB)]

> I think that's the main difficulty.  I think we'd rather not have
> separate bootstrap paths for Intel GNU/Linux on one hand, and everything
> else on the other hand.

Well, due to the design of mescc-tools; the bootstrap paths only have to be divergent up to the M1-macro level.
After that, we could simply use flags make the source work on different platforms

> Yet, we know that porting what you already did on x86-linux-gnu to
> GNU/Hurd and ARMv7 and AArch64 etc. is going to be a lot of non-trivial
> work (especially since historical versions of the GNU toolchain did not
> support AArch64, for instance.)
Nor RISC-V but that is likely to be a much bigger issue in terms of bootstrapping

> Waiting for this to be "solved" (and we don't even know how) would
> equate to a status quo.  But obviously, it'd be sad to have all this
> work already done on Intel and not be able to benefit from it.
Actually the work for the stage0 bootstrap steps have already been done on non-x86 hardware (Knight platform to be precise)
And the engineering decisions involved where explicitly selected to minimize porting and cross-platform bootstrapping effort.
M1-macro and hex2-linker only need flags to be set to build for all of the different supported platforms

> So perhaps we'll have to get over it and have a different bootstrap path
> on x86-linux-gnu.
A multiway bootstrap path that exceeds the requirements of DDC actually

> (BTW, I suspect we can get away with using 32-bit bootstrap binaries on
> both i686/x86_64 and armv7/aarch64, no?)
For AMD64, absolutely, ARM however I am not familiar enough to say

> Gash seems to be a low-hanging fruit and a relatively easy thing,
> because it's architecture-independent.  How
> far is it from being able to run typical 'configure' scripts?
Well we would have to replace the parser at a bare minimum

> I think the day it's able to run 'configure' scripts, we can switch to
> it right away without further ado, and then incrementally improve it as
> we stumble upon limitations and bugs.
Well we only need Gash to get to the build make and bash level, after that its scope can be limited.
In theory, someone could hand replace the make build script with a custom version that gash can use right now instead of us enhancing gash

> There's also another option you didn't mention: ditching the 2.0
> bootstrap Guile in favor of Mes.  That can be done in several steps:

>  1. Replace the guile-2.0.*.xz binary tarballs with Mes, and add a step
>      that builds Guile 2.x using our big bootstrap GCC binary.
Slow but possible

>   2. Same, but build Guile 2.x, libgc, etc. using MesCC.
MesCC can't directly build Guile yet but I do enjoy that ambition ;-)

> This could allow us to remove quite a lot of MiBs from our binary seeds.
FTFY

At this point, we effectively have a rope bridge to full bootstrappability
But we still have a lot of details to hammer out, like getting basic ARM support and having the ARM and x86 binaries verify each other's bootstrap;
Finding 6502, z80, 8051, 68K, VAX, pdp11, Alpha, MIPS, SPARC and PowerPC/Power Developer(s) to do stage0 work for their platforms and perform the cross verify steps.
Hammer out cross-platform build details for MesCC and M2-Planet

Jeremiah Orians
Cell phone: (517) 896-2948

Re: bootstrap integration strategies

[Ludovic Courtès]

Hello OriansJ,

"Orians, Jeremiah (DTMB)" <OriansJ@michigan.gov> skribis:

>> I think that's the main difficulty.  I think we'd rather not have
>> separate bootstrap paths for Intel GNU/Linux on one hand, and everything
>> else on the other hand.
>
> Well, due to the design of mescc-tools; the bootstrap paths only have to be divergent up to the M1-macro level.
> After that, we could simply use flags make the source work on different platforms

Sounds nice.  I wonder if Jan was referring to something else then?

There’s still the question of GNU/Hurd, though, which requires a vastly
different libc.

>> Yet, we know that porting what you already did on x86-linux-gnu to
>> GNU/Hurd and ARMv7 and AArch64 etc. is going to be a lot of non-trivial
>> work (especially since historical versions of the GNU toolchain did not
>> support AArch64, for instance.)
> Nor RISC-V but that is likely to be a much bigger issue in terms of bootstrapping

So far the initial ports of Guix to non-x86 were done through
cross-compilation (info "(guix) Porting").  So in a way, the binary
seeds for these platforms were built from source; we just “cut” the
source-to-binary connection by making those binaries the root of the
dependency graph on these platforms.

Maybe that’s something we’ll have to live with on new architectures.

>> Waiting for this to be "solved" (and we don't even know how) would
>> equate to a status quo.  But obviously, it'd be sad to have all this
>> work already done on Intel and not be able to benefit from it.
> Actually the work for the stage0 bootstrap steps have already been done on non-x86 hardware (Knight platform to be precise)
> And the engineering decisions involved where explicitly selected to minimize porting and cross-platform bootstrapping effort.
> M1-macro and hex2-linker only need flags to be set to build for all of the different supported platforms

So, problem solved?  Or am I missing something? :-)

I think the ‘wip-bootstrap’ branch does not use M1 at this point, does
it?

[...]

>> There's also another option you didn't mention: ditching the 2.0
>> bootstrap Guile in favor of Mes.  That can be done in several steps:
>
>>  1. Replace the guile-2.0.*.xz binary tarballs with Mes, and add a step
>>      that builds Guile 2.x using our big bootstrap GCC binary.
> Slow but possible
>
>>   2. Same, but build Guile 2.x, libgc, etc. using MesCC.
> MesCC can't directly build Guile yet but I do enjoy that ambition ;-)

I wonder what it would take to fix that.  After all, compiling libguile
must not be much harder than compiling tcc, no?

> At this point, we effectively have a rope bridge to full bootstrappability
> But we still have a lot of details to hammer out, like getting basic ARM support and having the ARM and x86 binaries verify each other's bootstrap;
> Finding 6502, z80, 8051, 68K, VAX, pdp11, Alpha, MIPS, SPARC and PowerPC/Power Developer(s) to do stage0 work for their platforms and perform the cross verify steps.
> Hammer out cross-platform build details for MesCC and M2-Planet

One thing at a time.  :-)

IMO what matters most at this point is to come up with a plan that allow
us to incrementally reduce the size of our binary seeds.  A port of
M1/stage0 to Z80 can wait.  ;-)  So we really need a list of actionable
items in the short term to start taking advantage of all the work that’s
been done on Mes, MesCC, M1, stage0, Gash, etc.

Thanks for your feedback!

Ludo’.

RE: bootstrap integration strategies

[Orians, Jeremiah (DTMB)]

> Sounds nice.  I wonder if Jan was referring to something else then?
Probably alternate operating systems like Hurd is my guess but I'm probably wrong.

> There’s still the question of GNU/Hurd, though, which requires a vastly different libc.
Fortunately Janneke has done a good job making that selectable

> So far the initial ports of Guix to non-x86 were done through
> cross-compilation (info "(guix) Porting").  So in a way, the binary
> seeds for these platforms were built from source; we just “cut” the
> source-to-binary connection by making those binaries the root of the
> dependency graph on these platforms.
Thank you for clarifying

> Maybe that’s something we’ll have to live with on new architectures.
Unless you want to make qemu a root dependency

> So, problem solved?  Or am I missing something? :-)
Mescc-tools can build valid binaries for all instruction sets with sane immediate representations (RISC-V is the only exception here; hopefully they fix that)
But Definition files need to be written and tests generated

> I think the ‘wip-bootstrap’ branch does not use M1 at this point, does it?
M1 and Hex2 are core pieces of mescc-tools and are required for MesCC to produce binaries.
As MesCC outputs M1-macro files (the .S files)

> I wonder what it would take to fix that.  After all, compiling libguile
> must not be much harder than compiling tcc, no?
Janneke know far better than me on this one

> One thing at a time.  :-)
But this is lots of fun :D

> IMO what matters most at this point is to come up with a plan that allow
> us to incrementally reduce the size of our binary seeds.  A port of
> M1/stage0 to Z80 can wait.  ;-)  So we really need a list of actionable
> items in the short term to start taking advantage of all the work that’s
Ok, how does this sound:
We walk the bootstrap binaries towards MesCC (This will add more bootstrap binaries in the short term)
Then we eliminate them one at a time until guile and mescc-tools are the only binaries that remain
Mescc is a scheme program which is interpreted by either guile or mes.c and thus could leverage guile until mes.c is in a state that it can replace guile for : guix, mescc and GASH
Thus the making of the bootstrap binaries can follow 2 possible paths:
1) Fast via standard C compilers: just build mes.c and mescc-tools and be done
2) Platform specific Stage0, which starts with hex0 (`200B) -> hex1 (~500B) -> hex2 (`1KB) -> M0 (~2KB)  -> [Everything here on is platform neutral] ->M2-Planet (~16KB) -> mes.c + mescc-tools (M1 and Hex2)

The best part is all of the binary seeds of all of the platforms will be able to build the binary seeds for all of the other platforms with bit for bit identical results (Which eliminates hardware based Trusting Trust attacks avoiding detection)

Jeremiah Orians
Cell phone: (517) 896-2948

Re: bootstrap integration strategies

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

>>> I think that's the main difficulty.  I think we'd rather not have
>>> separate bootstrap paths for Intel GNU/Linux on one hand, and everything
>>> else on the other hand.
>>
>> Well, due to the design of mescc-tools; the bootstrap paths only have to be divergent up to the M1-macro level.
>> After that, we could simply use flags make the source work on different platforms
>
> Sounds nice.  I wonder if Jan was referring to something else then?

I was looking at a narrower, or more short-term perspective.  M1 is
solving *a lot* of problems for us.  The only platform-dependency that
Mes then has, is:

    lib/x86-mes/x86.M1: About 200 M1 macros such as

        DEFINE add____$i32,%eax 05
        DEFINE add____$i32,%ecx 81c1

So, technically it's solved.  In practice, MesCC doesn't do proper
register mapping.  So, we could hard-wire a mapping for AArch64 but we'd
probably want to introduce abstraction.  That's a significant effort.
Also, we'll have to solve all tiny details/bugs that we encounter in
MesCC.  There's no real limitation, just some hard choices and work.

Then there is some assembly in Mes lib/linux-*.c as well, much less of a
problem, could still take weeks of work.

Then we have TCC; i'm not sure at their support for AArch64, and of
course we currently use gcc-2.95 which also doesn't do AArch64.  GCC 3
depends on a richer C library.  Of course we can make that (and this is
one of the things we will most probably do anyway) but still: work.

So what I was saying is probably: we have x86 NOW, can we use it and do
we want that somehow?  OR do we plan some of the work above, and go that
route?

> There’s still the question of GNU/Hurd, though, which requires a vastly
> different libc.

Yes.  I "almost" have a hello world there.  TBH, after spending about
half a week on it, I think x86/Hurd may be the easiest second
architecture...maybe amd64.

> So, problem solved?  Or am I missing something? :-)
>
> I think the ‘wip-bootstrap’ branch does not use M1 at this point, does
> it?

Sure, wip-bootstrap uses M1.  Currently the %mes-seed consists of
mes.M1; M1 is then used to produce a binary.  We are working to remove
this seed by translating mes.c into mes.M2.

Also, MesCC produces M1 output; so tcc is compiled to M1, and then to
binary.

>>> there's also another option you didn't mention: ditching the 2.0
>>> bootstrap Guile in favor of Mes.  That can be done in several steps:
>>
>>>  1. Replace the guile-2.0.*.xz binary tarballs with Mes, and add a step
>>>      that builds Guile 2.x using our big bootstrap GCC binary.
>> Slow but possible

Yes, performance is really the thing here.  Currently, mes is about 30x
slower than Guile.  It will definately not work if mes has to interpret
all of gnu/packages/*.scm, it may work if we can do something smart.

Also, Mes resembles Guile just enough to run Nyacc, but it's not on par
with Guile by far.

>>>   2. Same, but build Guile 2.x, libgc, etc. using MesCC.
>> MesCC can't directly build Guile yet but I do enjoy that ambition ;-)
>
> I wonder what it would take to fix that.  After all, compiling libguile
> must not be much harder than compiling tcc, no?

tcc is ~20,000LOC (1.5h to build), libguile is ~100,000LOC, libgc is
~30,000LOC.  Unless we find a way to compile a fraction of libguile #if
!BOOTSTRAP ..., compilation could take a day.  The good news is that
MesCC also runs on Guile, so development would suffer less from this.

I *really* like taking the early Mes route, or early Guile route (1 or
2).  I don't see how it these would work out--it just feels right.

> IMO what matters most at this point is to come up with a plan that allow
> us to incrementally reduce the size of our binary seeds.  A port of
> M1/stage0 to Z80 can wait.  ;-)  So we really need a list of actionable
> items in the short term to start taking advantage of all the work that’s
> been done on Mes, MesCC, M1, stage0, Gash, etc.

Hmm, yes!

My idea was to remove gcc, glibc and binutils from the x86 bootstrap
binaries (possibly one at a time) and replace them by M1+Mes+Tcc-built
ones.  Possibly by adding an alternative %bootstrap-binaries package
without them.  Then build the x86 system on top of that.  Then, or
meanwhile, start thinking about x86_64 or AArch64.

I'm not saying this because I think it's better than an early Guile
route, but it's only because I think I know this can be done.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrap integration strategies

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

Jan Nieuwenhuizen writes:

> My idea was to remove gcc, glibc and binutils from the x86 bootstrap
> binaries (possibly one at a time) and replace them by M1+Mes+Tcc-built
> ones.  Possibly by adding an alternative %bootstrap-binaries package
> without them.  Then build the x86 system on top of that.  Then, or
> meanwhile, start thinking about x86_64 or AArch64.

I've been trying to cleanup the dependencies in wip-bootstrap.  I found
that using `package-with-bootstrap-guile' confuses guix graph.  Or so it
seems, many duplicates of those packages show up.

Here's an image created with this hack: replace
`package-with-bootstrap-guile' with `identity' in commencement.scm and
mes.scm

Where it says `bootstrap-binaries0', not that binutils, gcc, glibc are
not used.


[-- Attachment #2: gcc-mesboot.png --]
[-- Type: image/png, Size: 847666 bytes --]

[-- Attachment #3: Type: text/plain, Size: 175 bytes --]


Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrap integration strategies

[Ludovic Courtès]

Hello!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Ludovic Courtès writes:
>
>>>> I think that's the main difficulty.  I think we'd rather not have
>>>> separate bootstrap paths for Intel GNU/Linux on one hand, and everything
>>>> else on the other hand.
>>>
>>> Well, due to the design of mescc-tools; the bootstrap paths only have to be divergent up to the M1-macro level.
>>> After that, we could simply use flags make the source work on different platforms
>>
>> Sounds nice.  I wonder if Jan was referring to something else then?
>
> I was looking at a narrower, or more short-term perspective.  M1 is
> solving *a lot* of problems for us.  The only platform-dependency that
> Mes then has, is:
>
>     lib/x86-mes/x86.M1: About 200 M1 macros such as
>
>         DEFINE add____$i32,%eax 05
>         DEFINE add____$i32,%ecx 81c1
>
> So, technically it's solved.  In practice, MesCC doesn't do proper
> register mapping.  So, we could hard-wire a mapping for AArch64 but we'd
> probably want to introduce abstraction.  That's a significant effort.
> Also, we'll have to solve all tiny details/bugs that we encounter in
> MesCC.  There's no real limitation, just some hard choices and work.
>
> Then there is some assembly in Mes lib/linux-*.c as well, much less of a
> problem, could still take weeks of work.
>
> Then we have TCC; i'm not sure at their support for AArch64, and of
> course we currently use gcc-2.95 which also doesn't do AArch64.  GCC 3
> depends on a richer C library.  Of course we can make that (and this is
> one of the things we will most probably do anyway) but still: work.

OK, I see.  Thanks for explaining.

> So what I was saying is probably: we have x86 NOW, can we use it and do
> we want that somehow?  OR do we plan some of the work above, and go that
> route?

I think we should try and use what we have now in ‘wip-bootstrap’, and
keeps things unchanged for ARM and GNU/Hurd.  Ricardo?

>>>> there's also another option you didn't mention: ditching the 2.0
>>>> bootstrap Guile in favor of Mes.  That can be done in several steps:
>>>
>>>>  1. Replace the guile-2.0.*.xz binary tarballs with Mes, and add a step
>>>>      that builds Guile 2.x using our big bootstrap GCC binary.
>>> Slow but possible
>
> Yes, performance is really the thing here.  Currently, mes is about 30x
> slower than Guile.  It will definately not work if mes has to interpret
> all of gnu/packages/*.scm, it may work if we can do something smart.

No no, in my view we’d use Mes simply as the guile-for-build in the
early derivations (the interpreter that runs the build phases from (guix
build build-system)).

It’s a job where we don’t need much performance, but we need the POSIX
layer—‘system*’, (ice-9 ftw), and so on.

>>>>   2. Same, but build Guile 2.x, libgc, etc. using MesCC.
>>> MesCC can't directly build Guile yet but I do enjoy that ambition ;-)
>>
>> I wonder what it would take to fix that.  After all, compiling libguile
>> must not be much harder than compiling tcc, no?
>
> tcc is ~20,000LOC (1.5h to build), libguile is ~100,000LOC, libgc is
> ~30,000LOC.  Unless we find a way to compile a fraction of libguile #if
> !BOOTSTRAP ..., compilation could take a day.  The good news is that
> MesCC also runs on Guile, so development would suffer less from this.
>
> I *really* like taking the early Mes route, or early Guile route (1 or
> 2).  I don't see how it these would work out--it just feels right.

OK.

> My idea was to remove gcc, glibc and binutils from the x86 bootstrap
> binaries (possibly one at a time) and replace them by M1+Mes+Tcc-built
> ones.  Possibly by adding an alternative %bootstrap-binaries package
> without them.  Then build the x86 system on top of that.  Then, or
> meanwhile, start thinking about x86_64 or AArch64.
>
> I'm not saying this because I think it's better than an early Guile
> route, but it's only because I think I know this can be done.

Sure, understood.

My hesitation comes from the fact that this will increase maintenance
cost on the Guix side.  At the same time, this is clearly the direction
we want to take, and I such I think we have to get our act together and
go forth.


What’s the exact status of ‘wip-bootstrap’ on non Intel arches?  Is it
still like ‘master’?  If it is, that’s fine.

Does it use the Mes/MesCC/tcc path for i686 only, or is it i686 +
x86_64?  (I would expect the latter.)

If there are no regressions, I’d be willing to simply merge it in
core-updates.  I’d like some of us to take another look at it—Ricardo,
Mark, and anyone with an interest in this.  And then I guess we could
go.

How does that sound?

Thank you for your patience!
v
Ludo’.

Re: bootstrap integration strategies

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

>> So what I was saying is probably: we have x86 NOW, can we use it and do
>> we want that somehow?  OR do we plan some of the work above, and go that
>> route?
>
> I think we should try and use what we have now in ‘wip-bootstrap’, and
> keeps things unchanged for ARM and GNU/Hurd.  Ricardo?

Yes, let's also ask Ricardo.

>> Yes, performance is really the thing here.  Currently, mes is about 30x
>> slower than Guile.  It will definately not work if mes has to interpret
>> all of gnu/packages/*.scm, it may work if we can do something smart.
>
> No no, in my view we’d use Mes simply as the guile-for-build in the
> early derivations (the interpreter that runs the build phases from (guix
> build build-system)).

OK...

> It’s a job where we don’t need much performance, but we need the POSIX
> layer—‘system*’, (ice-9 ftw), and so on.

Mes has system*, no ice-9 ftw yet, not sure about the so on; adding
things like these should be fun though.

> My hesitation comes from the fact that this will increase maintenance
> cost on the Guix side.  At the same time, this is clearly the direction
> we want to take, and I such I think we have to get our act together and
> go forth.

Agreed.  That's important to keep an eye on.  A requirement for a
bootstrap process is also its transparency.

> What’s the exact status of ‘wip-bootstrap’ on non Intel arches?  Is it
> still like ‘master’?  If it is, that’s fine.

> Does it use the Mes/MesCC/tcc path for i686 only, or is it i686 +
> x86_64?  (I would expect the latter.)

I haven't started integrating the bootprocess at all, not even for
i686/x86_64; there's only an alternative path to build gcc-4.7.4 atm.
That i686x86 gcc can be built on i686 and x86_64:

   ./pre-inst-env guix build gcc-mesboot

it's advisable to set

    (define %fake-bootstrap? #t)  ; cheat using Guile instead of Mes for speed-up?

in gnu/packages/mes.scm at first; set it to #f later and let it run
overnight :-)

> If there are no regressions, I’d be willing to simply merge it in
> core-updates.  I’d like some of us to take another look at it—Ricardo,
> Mark, and anyone with an interest in this.  And then I guess we could
> go.
>
> How does that sound?

That's really great...but we need integration work into the x86
bootstrap first too.  Do you/Ricardo want to help with that too?

> Thank you for your patience!

Thanks for your help and support!
janneke.

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: bootstrap integration strategies

[Ricardo Wurmus]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

>> So what I was saying is probably: we have x86 NOW, can we use it and do
>> we want that somehow?  OR do we plan some of the work above, and go that
>> route?
>
> I think we should try and use what we have now in ‘wip-bootstrap’, and
> keeps things unchanged for ARM and GNU/Hurd.  Ricardo?

I agree.  We need to make sure, though, that the Guix build
infrastructure doesn’t add more complicated packages to the environment
that are not needed.


>>>>> there's also another option you didn't mention: ditching the 2.0
>>>>> bootstrap Guile in favor of Mes.  That can be done in several steps:
>>>>
>>>>>  1. Replace the guile-2.0.*.xz binary tarballs with Mes, and add a step
>>>>>      that builds Guile 2.x using our big bootstrap GCC binary.
>>>> Slow but possible
>>
>> Yes, performance is really the thing here.  Currently, mes is about 30x
>> slower than Guile.  It will definately not work if mes has to interpret
>> all of gnu/packages/*.scm, it may work if we can do something smart.
>
> No no, in my view we’d use Mes simply as the guile-for-build in the
> early derivations (the interpreter that runs the build phases from (guix
> build build-system)).
>
> It’s a job where we don’t need much performance, but we need the POSIX
> layer—‘system*’, (ice-9 ftw), and so on.

Right.  We would need to cut out Guile on the build side.

> What’s the exact status of ‘wip-bootstrap’ on non Intel arches?  Is it
> still like ‘master’?  If it is, that’s fine.
>
> Does it use the Mes/MesCC/tcc path for i686 only, or is it i686 +
> x86_64?  (I would expect the latter.)
>
> If there are no regressions, I’d be willing to simply merge it in
> core-updates.  I’d like some of us to take another look at it—Ricardo,
> Mark, and anyone with an interest in this.  And then I guess we could
> go.

I would love to take a closer look again before merging it.
Unfortunately, these days I’m a bit short on time as I’m on “vacation”
with other plans imposed on my schedule.

--
Ricardo

RE: bootstrap integration strategies

[Orians, Jeremiah (DTMB)]

> I agree.  We need to make sure, though, that the Guix build infrastructure doesn’t add more complicated packages to the environment that are not needed.
Especially since those are the varieties that no one wants to be responsible for maintaining.

> Right.  We would need to cut out Guile on the build side.
Which would significantly increase system resource requirements, unless we make this a 1 time only cost sort of thing.

> I would love to take a closer look again before merging it.
> Unfortunately, these days I’m a bit short on time as I’m on “vacation”
> with other plans imposed on my schedule.
Ricardo, we love you dearly but please for the love of all that is holy;
Get back to that vacation! *cracks whip* Burnout is a real thing and believe me when I say bootstrapping is a marathon


Jeremiah Orians
Cell phone: (517) 896-2948

Re: bootstrap integration strategies

[Brett Gilio]

Orians, Jeremiah (DTMB) writes:
> Ricardo, we love you dearly but please for the love of all that 
> is holy;
> Get back to that vacation! *cracks whip* Burnout is a real thing 
> and believe me when I say bootstrapping is a marathon

In spite of making noise, burn out is such a terrible killer of
ambition. Please take care of yourselves, all of you.

-- 
Brett M. Gilio
Free Software Foundation, Member
https://parabola.nu | https://emacs.org

Re: bootstrap integration strategies

[Jan Nieuwenhuizen]

Orians, Jeremiah (DTMB) writes:

>> Gash seems to be a low-hanging fruit and a relatively easy thing,
>> because it's architecture-independent.  How
>> far is it from being able to run typical 'configure' scripts?
> Well we would have to replace the parser at a bare minimum

Yes, the parser is being refactored.  It "parses" a lot of SH code, but
it proved that handling variables properly was getting really tricky.

>> This could allow us to remove quite a lot of MiBs from our binary seeds.
> FTFY
>
> At this point, we effectively have a rope bridge to full bootstrappability

Hear, hear!

> But we still have a lot of details to hammer out, like getting basic
> ARM support and having the ARM and x86 binaries verify each other's
> bootstrap;

> Hammer out cross-platform build details for MesCC and M2-Planet

These I certainly agree with!  Some other archs can come later?

janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com