wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 2233 bytes --]

Hi!

I have reset Guix' wip-full-source-bootstrap branch with a first working
implementation of the, well, "Full Source Bootstrap" for x86-linux (and
x86_64-linux).  This bootstrap is rooted in the 357-byte hex0-seed from
the Stage0 project (https://savannah.gnu.org/projects/stage0):

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build hello --verbosity=1
[..]
/gnu/store/w61gf93yn2bxwyc6d1xp4y9lavvw1l3d-hello-2.10
17:58:54 janneke@dundal:~/src/guix/wip-fsb [env]
--8<---------------cut here---------------end--------------->8---

(it runs too!).

When you look at the bottom of the graph (see attached), you will notice
"%bootstrap-guile": the driver that we use for the Guix build and also
for "bootar", "gash", and "gash-utils".  This "%bootstrap-guile" is not
used as a seed in anything that is built, "%bootstrap-guile", "bootar",
"gash", and "gash-utils" could be replaced with any other driver.

Two new packages are added: "bootstrap-seeds", which contains the
hex0-seed binary
(https://github.com/oriansj/bootstrap-seeds/blob/master/POSIX/x86/hex0-seed)
with ASCII-equivalent
(https://github.com/oriansj/bootstrap-seeds/blob/master/POSIX/x86/hex0_x86.hex0),
and "m2-planet-boot" which, starting from hex0, via hex1, M0, hex2 and
M1, bootstraps the M2-Planet transpiler.  M2 is a language that closely
resembles a subset of C.

The breakthrough is that this M2-Planet can now compile a version of GNU
Mes, as yet unreleased: the wip-m2 branch.  This removes the remaining
binary seeds: %bootstrap-mescc-tools and %bootstrap-mes, together with
the "%bootstrap-mes-rewired" hack.

Apart from a review there is still some work before it can be
integrated, in short (from the top commit message):

XXX TODO:
   * wip-full-source-bootstrap
     - release mes-0.24, update
     - possibly release m2-planet-1.8.0, update
     - rebase wip-full-source-bootstrap onto core-updates
     - integrate
   * wip-arm-bootstrap
     - finish; currently stuck on gawk-mesboot0
     - release mes-0.23
     - devise strategy for integrating wip-full-source-bootstrap and
       wip-arm-bootstrap

Greetings,
Janneke

*) https://git.savannah.gnu.org/cgit/guix.git/log/?h=wip-full-source-bootstrap


[-- Attachment #2: gcc-core-mesboot0-graph.dot --]
[-- Type: application/octet-stream, Size: 7284 bytes --]

digraph "Guix package" {
  "139965588787520" [label = "gcc-core-mesboot0@2.95.3", shape = box, fontname = sans];
  "139965588787520" -> "139965588787680" [color = darkviolet];
  "139965588787520" -> "139965588788480" [color = darkviolet];
  "139965588787520" -> "139965588788640" [color = darkviolet];
  "139965588787520" -> "139965588788160" [color = darkviolet];
  "139965588787520" -> "139965588788960" [color = darkviolet];
  "139965588787520" -> "139965588788000" [color = darkviolet];
  "139965588787520" -> "139965588787840" [color = darkviolet];
  "139965588787520" -> "139965588788320" [color = darkviolet];
  "139965588787520" -> "139965588788800" [color = darkviolet];
  "139965588787520" -> "139965588789920" [color = darkviolet];
  "139965588787520" -> "139965588789760" [color = darkviolet];
  "139965588787520" -> "139965588790080" [color = darkviolet];
  "139965588787520" -> "139965749563136" [color = darkviolet];
  "139965588787680" [label = "binutils-mesboot0@2.14", shape = box, fontname = sans];
  "139965588787680" -> "139965588788480" [color = dimgrey];
  "139965588787680" -> "139965588788640" [color = dimgrey];
  "139965588787680" -> "139965588788160" [color = dimgrey];
  "139965588787680" -> "139965588788960" [color = dimgrey];
  "139965588787680" -> "139965588788000" [color = dimgrey];
  "139965588787680" -> "139965588787840" [color = dimgrey];
  "139965588787680" -> "139965588788320" [color = dimgrey];
  "139965588787680" -> "139965588788800" [color = dimgrey];
  "139965588787680" -> "139965588789920" [color = dimgrey];
  "139965588787680" -> "139965588789760" [color = dimgrey];
  "139965588787680" -> "139965588790080" [color = dimgrey];
  "139965588787680" -> "139965749563136" [color = dimgrey];
  "139965588788480" [label = "bash-mesboot0@2.05b", shape = box, fontname = sans];
  "139965588788480" -> "139965588788800" [color = dimgrey];
  "139965588788480" -> "139965588789120" [color = dimgrey];
  "139965588788480" -> "139965588789920" [color = dimgrey];
  "139965588788480" -> "139965588789760" [color = dimgrey];
  "139965588788480" -> "139965588790080" [color = dimgrey];
  "139965588788480" -> "139965749563136" [color = dimgrey];
  "139965588788800" [label = "make-mesboot0@3.80", shape = box, fontname = sans];
  "139965588788800" -> "139965588789120" [color = peachpuff4];
  "139965588788800" -> "139965588789920" [color = peachpuff4];
  "139965588788800" -> "139965588789760" [color = peachpuff4];
  "139965588788800" -> "139965588790080" [color = peachpuff4];
  "139965588788800" -> "139965749563136" [color = peachpuff4];
  "139965588789120" [label = "tcc-boot0@0.9.26-1136-g5bba73cc", shape = box, fontname = sans];
  "139965588789120" -> "139965588789280" [color = dimgrey];
  "139965588789120" -> "139965588789440" [color = dimgrey];
  "139965588789120" -> "139965588789920" [color = dimgrey];
  "139965588789120" -> "139965588789760" [color = dimgrey];
  "139965588789120" -> "139965588790080" [color = dimgrey];
  "139965588789120" -> "139965749563136" [color = dimgrey];
  "139965588789280" [label = "mes-boot@0.22-305-g2ab4c5c67", shape = box, fontname = sans];
  "139965588789280" -> "139965588789440" [color = red];
  "139965588789280" -> "139965588789920" [color = red];
  "139965588789280" -> "139965588789760" [color = red];
  "139965588789280" -> "139965588790080" [color = red];
  "139965588789280" -> "139965749563136" [color = red];
  "139965588789440" [label = "m2-planet-boot@1.7.0-31-g358b6cf", shape = box, fontname = sans];
  "139965588789440" -> "139965588789600" [color = cyan3];
  "139965588789440" -> "139965588789920" [color = cyan3];
  "139965588789440" -> "139965588789760" [color = cyan3];
  "139965588789440" -> "139965588790080" [color = cyan3];
  "139965588789440" -> "139965749563136" [color = cyan3];
  "139965588789600" [label = "bootstrap-seeds@1.0.0", shape = ellipse, fontname = sans];
  "139965588789600" -> "139965588790080" [color = peachpuff4];
  "139965588790080" [label = "bootar@1a", shape = box, fontname = sans];
  "139965588790080" -> "139965749563136" [color = darkseagreen];
  "139965749563136" [label = "guile-bootstrap@2.0", shape = ellipse, fontname = sans];
  "139965588789920" [label = "gash-boot@0.2.0", shape = box, fontname = sans];
  "139965588789920" -> "139965588790080" [color = magenta];
  "139965588789920" -> "139965749563136" [color = magenta];
  "139965588789760" [label = "gash-utils-boot@0.1.0", shape = box, fontname = sans];
  "139965588789760" -> "139965588790080" [color = magenta];
  "139965588789760" -> "139965588789920" [color = magenta];
  "139965588789760" -> "139965749563136" [color = magenta];
  "139965588788640" [label = "bzip2-mesboot@1.0.8", shape = box, fontname = sans];
  "139965588788640" -> "139965588788800" [color = dimgrey];
  "139965588788640" -> "139965588789120" [color = dimgrey];
  "139965588788640" -> "139965588789920" [color = dimgrey];
  "139965588788640" -> "139965588789760" [color = dimgrey];
  "139965588788640" -> "139965588790080" [color = dimgrey];
  "139965588788640" -> "139965749563136" [color = dimgrey];
  "139965588788160" [label = "diffutils-mesboot@2.7", shape = box, fontname = sans];
  "139965588788160" -> "139965588788800" [color = cyan3];
  "139965588788160" -> "139965588789120" [color = cyan3];
  "139965588788160" -> "139965588789920" [color = cyan3];
  "139965588788160" -> "139965588789760" [color = cyan3];
  "139965588788160" -> "139965588790080" [color = cyan3];
  "139965588788160" -> "139965749563136" [color = cyan3];
  "139965588788960" [label = "gzip-mesboot@1.2.4", shape = box, fontname = sans];
  "139965588788960" -> "139965588789120" [color = cyan3];
  "139965588788960" -> "139965588789920" [color = cyan3];
  "139965588788960" -> "139965588789760" [color = cyan3];
  "139965588788960" -> "139965588790080" [color = cyan3];
  "139965588788960" -> "139965749563136" [color = cyan3];
  "139965588788000" [label = "patch-mesboot@2.5.9", shape = box, fontname = sans];
  "139965588788000" -> "139965588788800" [color = dimgrey];
  "139965588788000" -> "139965588789120" [color = dimgrey];
  "139965588788000" -> "139965588789920" [color = dimgrey];
  "139965588788000" -> "139965588789760" [color = dimgrey];
  "139965588788000" -> "139965588790080" [color = dimgrey];
  "139965588788000" -> "139965749563136" [color = dimgrey];
  "139965588787840" [label = "sed-mesboot0@1.18", shape = box, fontname = sans];
  "139965588787840" -> "139965588788800" [color = peachpuff4];
  "139965588787840" -> "139965588789120" [color = peachpuff4];
  "139965588787840" -> "139965588789920" [color = peachpuff4];
  "139965588787840" -> "139965588789760" [color = peachpuff4];
  "139965588787840" -> "139965588790080" [color = peachpuff4];
  "139965588787840" -> "139965749563136" [color = peachpuff4];
  "139965588788320" [label = "tcc-boot@0.9.27", shape = box, fontname = sans];
  "139965588788320" -> "139965588788640" [color = dimgrey];
  "139965588788320" -> "139965588788800" [color = dimgrey];
  "139965588788320" -> "139965588789120" [color = dimgrey];
  "139965588788320" -> "139965588789920" [color = dimgrey];
  "139965588788320" -> "139965588789760" [color = dimgrey];
  "139965588788320" -> "139965588790080" [color = dimgrey];
  "139965588788320" -> "139965749563136" [color = dimgrey];

}

[-- Attachment #3: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: [bootstrappable] wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[jeremiah]

Amazing work as always janneke.

We will just have to do some kaem work to make it work all on POSIX systems.

-Jeremiah

Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 182 bytes --]

Hi Jan!

Woohoo, this look super exciting!  Congrats on all the progress, and
good luck with the remaining milestones!

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: [bootstrappable] wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Ludovic Courtès]

Hi!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> I have reset Guix' wip-full-source-bootstrap branch with a first working
> implementation of the, well, "Full Source Bootstrap" for x86-linux (and
> x86_64-linux).  This bootstrap is rooted in the 357-byte hex0-seed from
> the Stage0 project (https://savannah.gnu.org/projects/stage0):
>
> $ ./pre-inst-env guix build hello --verbosity=1
> [..]
> /gnu/store/w61gf93yn2bxwyc6d1xp4y9lavvw1l3d-hello-2.10
> 17:58:54 janneke@dundal:~/src/guix/wip-fsb [env]

This is amazing!  Incredible.  Thumbs up!

(BTW, you recently worked on the secret service¹, and now the
FSB—coincidence?)

¹ https://git.savannah.gnu.org/cgit/guix.git/commit?id=ec32d4f291b3cc039a99f8090b6c2b2444be5a83

> When you look at the bottom of the graph (see attached), you will notice
> "%bootstrap-guile": the driver that we use for the Guix build and also
> for "bootar", "gash", and "gash-utils".  This "%bootstrap-guile" is not
> used as a seed in anything that is built, "%bootstrap-guile", "bootar",
> "gash", and "gash-utils" could be replaced with any other driver.

Longer-term, could bootar, Gash, etc. run on Mes?  Would that help?

> Two new packages are added: "bootstrap-seeds", which contains the
> hex0-seed binary
> (https://github.com/oriansj/bootstrap-seeds/blob/master/POSIX/x86/hex0-seed)
> with ASCII-equivalent
> (https://github.com/oriansj/bootstrap-seeds/blob/master/POSIX/x86/hex0_x86.hex0),
> and "m2-planet-boot" which, starting from hex0, via hex1, M0, hex2 and
> M1, bootstraps the M2-Planet transpiler.  M2 is a language that closely
> resembles a subset of C.
>
> The breakthrough is that this M2-Planet can now compile a version of GNU
> Mes, as yet unreleased: the wip-m2 branch.  This removes the remaining
> binary seeds: %bootstrap-mescc-tools and %bootstrap-mes, together with
> the "%bootstrap-mes-rewired" hack.

Woow.  I’m willing to take a closer look at all this, this is
impressive!

> Apart from a review there is still some work before it can be
> integrated, in short (from the top commit message):
>
> XXX TODO:
>    * wip-full-source-bootstrap
>      - release mes-0.24, update
>      - possibly release m2-planet-1.8.0, update
>      - rebase wip-full-source-bootstrap onto core-updates
>      - integrate

All this should be piece of cake compared to what you’ve gone through.
;-)  But it does mean long rebuild cycles, which I guess can be tiring.

>    * wip-arm-bootstrap
>      - finish; currently stuck on gawk-mesboot0
>      - release mes-0.23
>      - devise strategy for integrating wip-full-source-bootstrap and
>        wip-arm-bootstrap

Ah, that’s the second big challenge!

Congratulations to you and everyone involved for going this far!
You made it!

Ludo’.

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Andrius Štikonas via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 3468 bytes --]

2021 m. sausio 6 d., trečiadienis 11:32:48 GMT Ludovic Courtès rašė:
> Hi!
> 
> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
> 
> > I have reset Guix' wip-full-source-bootstrap branch with a first working
> > implementation of the, well, "Full Source Bootstrap" for x86-linux (and
> > x86_64-linux).  This bootstrap is rooted in the 357-byte hex0-seed from
> > the Stage0 project (https://savannah.gnu.org/projects/stage0):
> >
> > $ ./pre-inst-env guix build hello --verbosity=1
> > [..]
> > /gnu/store/w61gf93yn2bxwyc6d1xp4y9lavvw1l3d-hello-2.10
> > 17:58:54 janneke@dundal:~/src/guix/wip-fsb [env]
> 
> This is amazing!  Incredible.  Thumbs up!
> 
> (BTW, you recently worked on the secret service¹, and now the
> FSB—coincidence?)
> 
> ¹ https://git.savannah.gnu.org/cgit/guix.git/commit?id=ec32d4f291b3cc039a99f8090b6c2b2444be5a83
> 
> > When you look at the bottom of the graph (see attached), you will notice
> > "%bootstrap-guile": the driver that we use for the Guix build and also
> > for "bootar", "gash", and "gash-utils".  This "%bootstrap-guile" is not
> > used as a seed in anything that is built, "%bootstrap-guile", "bootar",
> > "gash", and "gash-utils" could be replaced with any other driver.
> 
> Longer-term, could bootar, Gash, etc. run on Mes?  Would that help?

I think that's what mes-m2 rewrite [1] (not to be confused with mes wip-m2 branch)
is trying to achieve.

Outside of Guix we are working on bootstrap that does not depend on guile driver
and is driven only by hex-0 seed (357 bytes) kaem-optional-seed (737 bytes) and any POSIX
kernel.

At the moment it goes all the way up to Mes (tcc is now in progress).

Andrius

[1] https://github.com/oriansj/mes-m2

> 
> > Two new packages are added: "bootstrap-seeds", which contains the
> > hex0-seed binary
> > (https://github.com/oriansj/bootstrap-seeds/blob/master/POSIX/x86/hex0-seed)
> > with ASCII-equivalent
> > (https://github.com/oriansj/bootstrap-seeds/blob/master/POSIX/x86/hex0_x86.hex0),
> > and "m2-planet-boot" which, starting from hex0, via hex1, M0, hex2 and
> > M1, bootstraps the M2-Planet transpiler.  M2 is a language that closely
> > resembles a subset of C.
> >
> > The breakthrough is that this M2-Planet can now compile a version of GNU
> > Mes, as yet unreleased: the wip-m2 branch.  This removes the remaining
> > binary seeds: %bootstrap-mescc-tools and %bootstrap-mes, together with
> > the "%bootstrap-mes-rewired" hack.
> 
> Woow.  I’m willing to take a closer look at all this, this is
> impressive!
> 
> > Apart from a review there is still some work before it can be
> > integrated, in short (from the top commit message):
> >
> > XXX TODO:
> >    * wip-full-source-bootstrap
> >      - release mes-0.24, update
> >      - possibly release m2-planet-1.8.0, update
> >      - rebase wip-full-source-bootstrap onto core-updates
> >      - integrate
> 
> All this should be piece of cake compared to what you’ve gone through.
> ;-)  But it does mean long rebuild cycles, which I guess can be tiring.
> 
> >    * wip-arm-bootstrap
> >      - finish; currently stuck on gawk-mesboot0
> >      - release mes-0.23
> >      - devise strategy for integrating wip-full-source-bootstrap and
> >        wip-arm-bootstrap
> 
> Ah, that’s the second big challenge!
> 
> Congratulations to you and everyone involved for going this far!
> You made it!
> 
> Ludo’.
> 
> 

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

RE: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Orians, Jeremiah (DTMB)]

> I think that's what mes-m2 rewrite [1] (not to be confused with mes wip-m2 branch) is trying to achieve.
My fault for that confusion. Wish I was faster at implementing syntax-case in C -_-

> Outside of Guix we are working on bootstrap that does not depend on guile driver and is driven only by hex-0 seed (357 bytes) kaem-optional-seed (737 bytes) and any POSIX kernel.
We love it ^_^

> At the moment it goes all the way up to Mes (tcc is now in progress).
Eternal progress

Oh and we are currently joking about replacing mes.c with a scheme written in Haskell because we bootstrapped a minimal Haskell too.
https://github.com/oriansj/blynn-compiler/

Then the loop would be:
a scheme interpreter written in Haskell 
running 
a C compiler written in scheme 
that can build 
the Haskell compiler able to build the original scheme interpreter.

If we get it to enough guile compatibility; then it becomes:
once you have Gnu Mes, you are already bootstrapped. 

^_^

- Jeremiah

Re: [bootstrappable] wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Paul Sherwood]

On 2021-01-04 17:01, Jan Nieuwenhuizen wrote:
> I have reset Guix' wip-full-source-bootstrap branch with a first 
> working
> implementation of the, well, "Full Source Bootstrap" for x86-linux (and
> x86_64-linux).  This bootstrap is rooted in the 357-byte hex0-seed from
> the Stage0 project (https://savannah.gnu.org/projects/stage0):

This is really exciting, great job Jan! Do you need any help, e.g. on 
the ARM side, or with optimising the integration?

Re: [bootstrappable] wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1794 bytes --]

Paul Sherwood writes:

Hello Paul,

> On 2021-01-04 17:01, Jan Nieuwenhuizen wrote:
>> I have reset Guix' wip-full-source-bootstrap branch with a first
>> working
>> implementation of the, well, "Full Source Bootstrap" for x86-linux (and
>> x86_64-linux).  This bootstrap is rooted in the 357-byte hex0-seed from
>> the Stage0 project (https://savannah.gnu.org/projects/stage0):
>
> This is really exciting, great job Jan! Do you need any help, e.g. on
> the ARM side, or with optimising the integration?

Thanks!  We really could use help with this (Danny?).  To paint the
picture for people listening in: Next to ARM it may need some Guix
skills, or even more work to reproduce our experimental ARM bootstrap
outside of Guix.

I'm a bit hesitant about the timing, because I cannot make much time the
coming week.

A very short summary of where we are, on wip-arm-bootstrap, building
gawk-mesboot0

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix build -e '(@@ (gnu packages commencement) gawk-mesboot0)'
--8<---------------cut here---------------end--------------->8---

produces a gawk binary that fails to increment a variable

--8<---------------cut here---------------start------------->8---
11:35:59 janneke@novena:~/src/wip-arm-bootstrap [env]
$ /gnu/store/f756xxxqj3mabaax5r4ldrxh01a9q54r-gawk-mesboot0-3.0.0/bin/gawk -f add.awk add.awk 
i=1
i=2
11:36:14 janneke@novena:~/src/wip-arm-bootstrap [env]
$ /gnu/store/f756xxxqj3mabaax5r4ldrxh01a9q54r-gawk-mesboot0-3.0.0/bin/gawk -f inc.awk inc.awk 
i=     0
i=     0
11:36:27 janneke@novena:~/src/wip-arm-bootstrap [env]
--8<---------------cut here---------------end--------------->8---

So could be anything, could bin in tcc-boot or in gawk-mesboot0...

Thanks for reaching out!
Greetings,
Janneke


[-- Attachment #2: add.awk --]
[-- Type: application/octet-stream, Size: 95 bytes --]

BEGIN {
    i = 0;
    i = i + 1
    printf "i=%d\n", i
    i = i + 1
    printf "i=%d\n", i
}

[-- Attachment #3: inc.awk --]
[-- Type: application/octet-stream, Size: 71 bytes --]

BEGIN {
    i = 0;
    printf "i=%d\n", ++i
    printf "i=%d\n", ++i
}

[-- Attachment #4: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Danny Milosavljevic]

[-- Attachment #1.1: Type: text/plain, Size: 13894 bytes --]

Hi Paul,
Hi Janneke,

On Thu, 07 Jan 2021 11:43:39 +0100
Jan Nieuwenhuizen <janneke@gnu.org> wrote:

> > This is really exciting, great job Jan! Do you need any help, e.g. on
> > the ARM side, or with optimising the integration?

> Thanks!  We really could use help with this (Danny?).  

@Paul:

Yeah, help would be nice!

We are now pretty far along in bootstrapping--which means you'd ideally set up
an armhf machine, install guix on it and then debug gawk-mesboot0 using that in
order to help.  Can you do that?

A writeup of how to debug a current problem we are facing follows.  If you can
help with fixing that problem, please do :)

We currently are at mescc -> tinycc -> tinycc -> tinycc -> gawk-mesboot0 or so.

The next steps for us are:

* Debug why gawk-mesboot0 doesn't work correctly (see below--and Janneke's
E-Mail)

  * That will entail enabling gdb to work on tcc (TinyCC) executables and/or
    reading ARM assembly.  Since the error does not cause a failure
    immediately, it will be difficult to pinpoint exactly where the error was
    introduced.

To reproduce the bug:

  ./gawk 'BEGIN { i = 5; ++i; print(i) }'
  5

(yes, it prints 5.  Same happens with "--".  But i += 1 works fine.
So does i *= 2).

You will need Guix on armhf.  I do NOT recommend using an aarch64 machine
with an aarch64 kernel for the time being (I'll fix it eventually--but right
now that's a really bad idea with Guix).

> To paint the
> picture for people listening in: Next to ARM it may need some Guix
> skills, or even more work to reproduce our experimental ARM bootstrap
> outside of Guix.

> A very short summary of where we are, on wip-arm-bootstrap, building
> gawk-mesboot0
> 
> --8<---------------cut here---------------start------------->8---
> ./pre-inst-env guix build -e '(@@ (gnu packages commencement) gawk-mesboot0)'
> --8<---------------cut here---------------end--------------->8---
> 
> produces a gawk binary that fails to increment a variable
> 
> --8<---------------cut here---------------start------------->8---
> 11:35:59 janneke@novena:~/src/wip-arm-bootstrap [env]
> $ /gnu/store/f756xxxqj3mabaax5r4ldrxh01a9q54r-gawk-mesboot0-3.0.0/bin/gawk -f add.awk add.awk 
> i=1
> i=2
> 11:36:14 janneke@novena:~/src/wip-arm-bootstrap [env]
> $ /gnu/store/f756xxxqj3mabaax5r4ldrxh01a9q54r-gawk-mesboot0-3.0.0/bin/gawk -f inc.awk inc.awk 
> i=     0
> i=     0
> 11:36:27 janneke@novena:~/src/wip-arm-bootstrap [env]
> --8<---------------cut here---------------end--------------->8---
> 
> So could be anything, could bin in tcc-boot or in gawk-mesboot0...

Next steps:

* find function in gawk 3.0.0 that interprets "++" (yacc grammar),
  * find out why it doesn't work
    * find out why tcc miscompiles it

In order to read the source of the gawk used, invoke

  guix build -S -e '(@@ (gnu packages commencement) gawk-mesboot0)'

.

In there, the yacc grammar is in awk.y (and the generated parser is in
awktab.c).

The AST node types generated for "++" and "--" are Node_preincrement and
Node_predecrement, respectively.

The evaluator is in eval.c (@janneke: and it uses setjmp.  At least
longjmp seems not to be entered for our gawk problem here... phiew).

The place where Node_preincrement is finally handled is in op_assign in the
latter file, which does:

        case Node_preincrement:
        case Node_predecrement:
                unref(*lhs);
                *lhs = make_number(lval +
                               (tree->type == Node_preincrement ? 1.0 : -1.0));
                tree->lnode->flags |= SCALAR;
                if (after_assign)
                        (*after_assign)();
                return *lhs;

The case that does work (+= 1) is handled in the same function:

        lval = force_number(*lhs);
        rval = force_number(tmp);
[...]
        case Node_assign_plus:
                *lhs = make_number(lval + rval);
                break;

Debugging gawk on armhf via

   gdb --args 'BEGIN { ++i; print(i) }'

I get:

eval.c 

│  >857                         *lhs = make_number(lval +                                                                                                     │
│   858                                        (tree->type == Node_preincrement ? 1.0 : -1.0));                                                               │

And in asm, after it ascertained that tree->type == Node_preincrement, it does:

│   0x15598 <op_assign+320> ldr     lr, [pc]        ; 0x155a0 <op_assign+328>                                                                                 │
│   0x1559c <op_assign+324> b       0x155a4 <op_assign+332>                                                                                                   │
│   0x155a0 <op_assign+328>                CONSTANT 0
│   0x155a4 <op_assign+332> vldr    d0, [lr]                                                                                                                  │
│   0x155a8 <op_assign+336> vldr    d1, [r11, #-16]                                                                                                           │
│   >0x155ac <op_assign+340> vadd.f64        d0, d1, d0                                                                                                       │
│   0x155b0 <op_assign+344> vpush   {d0}                                                                                                                      │
│   0x155b4 <op_assign+348> mov     r2, #97 ; 0x61                                                                                                            │
│   0x155b8 <op_assign+352> pop     {r0, r1}                                                                                                                  │
│   0x155bc <op_assign+356> bl      0x23208 <mk_number>                                                                                                       │

Registers before the vadd are:

d0             {u8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, u16 = {0x0, 0x0, 0x0, 0x0}, u32 = {0x0, 0x0}, u64 = 0x0, f32 = {0x0, 0x0}, f64 = 0x0}
d1             {u8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, u16 = {0x0, 0x0, 0x0, 0x0}, u32 = {0x0, 0x0}, u64 = 0x0, f32 = {0x0, 0x0}, f64 = 0x0}

And [r11, #-16] seems to be used more often in the assembly, so is probably lval.  Unsurprisingly, it's 0.

But so is the other constant added to it.  WTF!

And maybe more useful:

 gdb --args ../gawk 'BEGIN { i = 5; ++i ; print(i) }':

right before it does the ++:

│   0x1555c <op_assign+260> ldr     r1, [r11, #12]                                                                                                            │
│   0x15560 <op_assign+264> add     r1, r1, #24                                                                                                               │
│   0x15564 <op_assign+268> ldr     r2, [r1]                                                                                                                  │
│   0x15568 <op_assign+272> cmp     r2, #10                                                                                                                   │
│   0x1556c <op_assign+276> moveq   r1, #1                                                                                                                    │
│   0x15570 <op_assign+280> movne   r1, #0                                                                                                                    │
│   0x15574 <op_assign+284> str     r0, [r11, #-60] ; 0xffffffc4                                                                                              │
│   0x15578 <op_assign+288> cmp     r1, #0                                                                                                                    │
│   0x1557c <op_assign+292> beq     0x15584 <op_assign+300>                                                                                                   │
│   0x15580 <op_assign+296> b       0x15598 <op_assign+320>                                                                                                   │
│   0x15584 <op_assign+300> ldr     lr, [pc]        ; 0x1558c <op_assign+308>                                                                                 │
│   0x15588 <op_assign+304> b       0x15590 <op_assign+312>                                                                                                   │
│   0x1558c <op_assign+308> CONSTANT 0x000508ec (little endian)
│   0x15590 <op_assign+312> vldr    d0, [lr]                                                                                                                  │
│   0x15594 <op_assign+316> b       0x155a8 <op_assign+336>                                                                                                   │
│   0x15598 <op_assign+320> ldr     lr, [pc]        ; 0x155a0 <op_assign+328>                                                                                 │
│   0x1559c <op_assign+324> b       0x155a4 <op_assign+332>                                                                                                   │
│   0x155a0 <op_assign+328> CONSTANT 0x000508f4 (little endian)
│   0x155a4 <op_assign+332> vldr    d0, [lr]          ; instruction: 0xed9e0b00; so sign=1; note: double
│   0x155a8 <op_assign+336> vldr    d1, [r11, #-16]   ; instruction: 0xed1b1b04; so sign=0; note: double
│   >0x155ac <op_assign+340> vadd.f64        d0, d1, d0                                                                                                       │
│   0x155b0 <op_assign+344> vpush   {d0}                                                                                                                      │
│   0x155b4 <op_assign+348> mov     r2, #97 ; 0x61                                                                                                            │
│   0x155b8 <op_assign+352> pop     {r0, r1}                                                                                                                  │
│   0x155bc <op_assign+356> bl      0x23208 <mk_number>                                                                                                       │

At address 0x000508ec there is: 0 (64 bit).  That is incorrect.
At address 0x000508f4 there is: 0 (64 bit).  That is incorrect.

Registers before the vadd are:

d0             {u8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, u16 = {0x0, 0x0, 0x0, 0x0}, u32 = {0x0, 0x0}, u64 = 0x0, f32 = {0x0, 0x0}, f64 = 0x0}
d1             {u8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x14, 0x40}, u16 = {0x0, 0x0, 0x0, 0x4014}, u32 = {0x0, 0x40140000}, u64 = 0x4014000000000000, f32 = {0x0,
0x2}, f64 = 0x5}

So now we have to find out why tcc put those 0s in those constant literal slots.

My wild guess is because it's trying to use 64 bit ints and we only gave it a
32 bit int at some point.  But that should have been fixed by compiling tcc
using tcc.  Weird...

tcc does the following (in arm-gen.c):

  int v, ft, fc, fr, sign;
  uint32_t op;
  SValue v1;

  fr = sv->r;
  ft = sv->type.t;
  fc = sv->c.i;

      if(is_float(ft)) {
        calcaddr(&base,&fc,&sign,1020,2);
#ifdef TCC_ARM_VFP
        op=0xED100A00; /* flds */
        if(!sign)
          op|=0x800000;
        if ((ft & VT_BTYPE) != VT_FLOAT)
          op|=0x100;   /* flds -> fldd */
        o(op|(vfpr(r)<<12)|(fc>>2)|(base<<16));

Uhhh.... how does a 64 bit double fit into an ARM (32 bit) int (variable "fc")?

calcaddr seems to take the second argument, stuff it into a const
pool (if too big) and return the address to the item in the second argument
again.  If it's not too big, then it's just used directly without pool entry.

So it might make sense to add printfs to these locations in tcc and
see what it does with the constants.

@Paul: Could you decode what calcaddr is supposed to do, exactly (see
git@gitlab.com:janneke/tinycc.git arm-gen.c)?  That would help a lot.

Something maybe unrelated: trying to run gawk using

  qemu-arm -singlestep -d nochain,in_asm,cpu ./gawk

I get

  0x0004e760:  e3511000  cmp      r1, #0

  R00=00000000 R01=00000020 R02=407ffc58 R03=00000000
  R04=00000000 R05=00000000 R06=00000000 R07=00000000
  R08=00000000 R09=00000000 R10=0004fb78 R11=407ffc40
  R12=407ffc58 R13=407ffc30 R14=0004e9b8 R15=0004e760
  PSR=20000010 --C- A usr32
  qemu: uncaught target signal 4 (Illegal instruction) - core dumped
  Illegal instruction

The correct encoding of CMP is e3510000.
The "illegal" encoding of CMP has "set condition codes" (bit 20) set.

@Paul:

Is setting the "set condition codes" flag on CMP only deprecated or
actually forbidden?  (I.e. am I chasing a qemu misemulation (again...) or
is this actually a problem?)

Attached the buggy gawk executable.

I've also searched for other double literals (/= 0) in gawk's source code,
and there's one in io.c (in do_getline), and one in builtin.c (rlength = -1.0
if a regexp does not match in do_match).
That's it O_o.
So there are very few double literals in there apparently.

And indeed:

$ echo | ./gawk 'BEGIN { print(getline); }'
0    <--- wrong

No idea how to test the rlength thing, though.

@Janneke: 

I'm not sure we have automated it far enough for Paul to reproduce this
bug from scratch yet, right?

Maybe we should already do a Mes for ARM release after all ?
(provided the x86 bootstrap still works with that version, of course)

I mean release early, release often and all.  Otherwise it's really
difficult for new people to get started.

[-- Attachment #1.2: gawk --]
[-- Type: application/octet-stream, Size: 629828 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 66 bytes --]

Oops, git@gitlab.com:janneke/tinycc.git in branch "mes-0.23"



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 3911 bytes --]

Hi Janneke,

I just found the bug in tinycc that caused failed our ARM bootstrap to fail.

I use the following reproducer:

int main() {
        double f = 1.0;
        return 0;
}

and then invoke

  tcc a.c

on ARM (32) using your patched tcc.  (but it's also broken in the unpatched tcc)

(tcc cross compiler is not enough.  tcc has to actually itself be an ARM
EABI executable)

I get a bus error here:

│   0x24698 <init_putv+1688>        vstr    d0, [r0]                                                                                                          │

Debugging some more, I find:

tccgen.c:

/* store a value or an expression directly in global data or in local array */
static void init_putv(CType *type, Section *sec, unsigned long c)
{
[...]
        size = type_size(type, &align);
        section_reserve(sec, c + size); // c == 0, size == 8
        ptr = sec->data + c; // sec->data == 0x24b01e, c == 0
            switch(bt) {
                /* XXX: when cross-compiling we assume that each type has the
                   same representation on host and target, which is likely to
                   be wrong in the case of long double */
            case VT_BOOL:
                vtop->c.i = vtop->c.i != 0;
            case VT_BYTE:
                *(char *)ptr = vtop->c.i;
                break;
            case VT_SHORT:
                *(short *)ptr = vtop->c.i;
                break;
            case VT_FLOAT:
                *(float*)ptr = vtop->c.f;
                break;
            case VT_DOUBLE:
                *(double *)ptr = vtop->c.d;
                break;
[... and so on]

tccelf.c:

/* reserve at least 'size' bytes from section start */
ST_FUNC void section_reserve(Section *sec, unsigned long size)
{
    if (size > sec->data_allocated)  // both 8
        section_realloc(sec, size);
    if (size > sec->data_offset) // both 8
        sec->data_offset = size;
}

Nothing here make sure that the VFP double is aligned to 8 Byte.

And indeed, (0x24b01e % 8) == 6, not 0.

Alignment could be disabled on the CPU

  https://developer.arm.com/documentation/ddi0464/f/system-control/register-descriptions/system-control-register

but I don't think EABI wants that.

tinycc does have:

/* reserve at least 'size' bytes aligned per 'align' in section
   'sec' from current offset, and return the aligned offset */
ST_FUNC size_t section_add(Section *sec, addr_t size, int align)
{
    size_t offset, offset1;

    offset = (sec->data_offset + align - 1) & -align;
    offset1 = offset + size;
    if (sec->sh_type != SHT_NOBITS && offset1 > sec->data_allocated)
        section_realloc(sec, offset1);
    sec->data_offset = offset1;
    if (align > sec->sh_addralign)
        sec->sh_addralign = align;
    return offset;
}

But that's not used for init_putv.

And section_reserve, which is used, doesn't care about alignment at all.

(it seems there's a reserved part and a data part in each section, and
it holds that the data part elements are aligned--but the reserved part
elements are NOT aligned.  I don't see how sec->data would be aligned
by the dynamic memory allocator either)

Other notes:

tccgen.c even has this:

>                /* XXX: when cross-compiling we assume that each type has the
>                   same representation on host and target, which is likely to
>                   be wrong in the case of long double */

Yeah, and even when NOT cross-compiling, the alignment is wrong--which means
it sometimes won't work at all on ARM, depending on luck.

As a workaround, we can patch tcc to instead do the assignments on elements
on the stack and then copy those over, instead of doing

                *(double *)ptr = vtop->c.d

(the latter of which emits VFP instructions that expect double-aligned
pointers).

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Jan Wielkiewicz]

Dnia 2021-01-04, o godz. 18:01:21
Jan Nieuwenhuizen <janneke@gnu.org> napisał(a):

> Hi!
> 
> I have reset Guix' wip-full-source-bootstrap branch with a first
> working implementation of the, well, "Full Source Bootstrap" for
> x86-linux (and x86_64-linux).  This bootstrap is rooted in the
> 357-byte hex0-seed from the Stage0 project

Great job! Looks like dark magic to me.
It would be interesting to compare binaries made using the bootstrapped
and the unbootstrapped system and look for *interesting stuff* like
some decades old self-replicating code.
 
> Greetings,
> Janneke
> 
> *)
> https://git.savannah.gnu.org/cgit/guix.git/log/?h=wip-full-source-bootstrap
> 

Jan Wielkiewicz

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Jan Nieuwenhuizen]

Danny Milosavljevic writes:

Hello Danny,

> I just found the bug in tinycc that caused failed our ARM bootstrap to fail.
>
> I use the following reproducer:
>
> int main() {
>         double f = 1.0;
>         return 0;
> }

Beautiful!  Well done!  I can confirm that adding this patch

--8<---------------cut here---------------start------------->8---
diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index a50a238ddd..5a3fa694b3 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -1417,8 +1417,8 @@ ac_cv_c_float_format='IEEE (little-endian)'
                (substitute* "test/Makefile.in"
                  (("^bigtest:.*") "bigtest: basic\n")
                  (("( |\t)(childin|convfmt|fflush|longwrds|math|negexp)" all sep) sep))
-               (substitute* "io.c"
-                 (("char rs;") "int rs;"))
+               (substitute* '("builtin.c" "eval.c" "io.c")
+                 (("1.0") "1"))
                #t))
            (add-before 'configure 'setenv
              (lambda _
--8<---------------cut here---------------end--------------->8---

to the gawk-mesbot0 recipe also fixes "inc.awk".  The pre
increment/decrement code looks like this:

--8<---------------cut here---------------start------------->8---
		*lhs = make_number(lval +
			       (tree->type == Node_preincrement ? 1.0 : -1.0));

--8<---------------cut here---------------end--------------->8---

> on ARM (32) using your patched tcc.  (but it's also broken in the unpatched tcc)
>
> (tcc cross compiler is not enough.  tcc has to actually itself be an ARM
> EABI executable)
>
> I get a bus error here:
>
> │   0x24698 <init_putv+1688>        vstr    d0, [r0]                                                                                                          │

Ah, so it may result in a wrong assingment, or bus error even.  Great!

> Debugging some more, I find:
>
> tccgen.c:
>
> /* store a value or an expression directly in global data or in local array */
> static void init_putv(CType *type, Section *sec, unsigned long c)
> {
> [...]
>         size = type_size(type, &align);
>         section_reserve(sec, c + size); // c == 0, size == 8
>         ptr = sec->data + c; // sec->data == 0x24b01e, c == 0
>             switch(bt) {
>                 /* XXX: when cross-compiling we assume that each type has the
>                    same representation on host and target, which is likely to
>                    be wrong in the case of long double */
>             case VT_BOOL:
>                 vtop->c.i = vtop->c.i != 0;
>             case VT_BYTE:
>                 *(char *)ptr = vtop->c.i;
>                 break;
>             case VT_SHORT:
>                 *(short *)ptr = vtop->c.i;
>                 break;
>             case VT_FLOAT:
>                 *(float*)ptr = vtop->c.f;
>                 break;
>             case VT_DOUBLE:
>                 *(double *)ptr = vtop->c.d;
>                 break;
> [... and so on]

Ah yes, this code has been problematic in the sense that I found bus errors
here and tried workarounds several times (look at the
wip-arm-bootstrap14 branch).

> tccelf.c:
>
> /* reserve at least 'size' bytes from section start */
> ST_FUNC void section_reserve(Section *sec, unsigned long size)
> {
>     if (size > sec->data_allocated)  // both 8
>         section_realloc(sec, size);
>     if (size > sec->data_offset) // both 8
>         sec->data_offset = size;
> }
>
> Nothing here make sure that the VFP double is aligned to 8 Byte.
>
> And indeed, (0x24b01e % 8) == 6, not 0.

Ah...I wasn't aware of this requirement...

> Alignment could be disabled on the CPU
>
>   https://developer.arm.com/documentation/ddi0464/f/system-control/register-descriptions/system-control-register
>
> but I don't think EABI wants that.

Hmm, what does this mean?  We are not really using EABI, or are we?  We
seem to need this terrible hack

--8<---------------cut here---------------start------------->8---
diff --git a/tccelf.c b/tccelf.c
index 2ac7466c..42546f57 100644
--- a/tccelf.c
+++ b/tccelf.c
@@ -1867,7 +1867,7 @@ static void tcc_output_elf(TCCState *s1, FILE *f, int phnum, ElfW(Phdr) *phdr,
     ehdr.e_ident[EI_OSABI] = ELFOSABI_FREEBSD;
 #endif
 #ifdef TCC_TARGET_ARM
-#ifdef TCC_ARM_EABI
+#if defined (TCC_ARM_EABI) || BOOTSTRAP
     ehdr.e_ident[EI_OSABI] = 0;
     ehdr.e_flags = EF_ARM_EABI_VER4;
     if (file_type == TCC_OUTPUT_EXE || file_type == TCC_OUTPUT_DLL)
--8<---------------cut here---------------end--------------->8---

at least to get tcc's ARM binaries to run on Aarch64-Linux.  Is this
related; iow, could it be that this "fix" for Aarch64 break ARM?

> tinycc does have:
>
> /* reserve at least 'size' bytes aligned per 'align' in section
>    'sec' from current offset, and return the aligned offset */
> ST_FUNC size_t section_add(Section *sec, addr_t size, int align)
> {
>     size_t offset, offset1;
>
>     offset = (sec->data_offset + align - 1) & -align;
>     offset1 = offset + size;
>     if (sec->sh_type != SHT_NOBITS && offset1 > sec->data_allocated)
>         section_realloc(sec, offset1);
>     sec->data_offset = offset1;
>     if (align > sec->sh_addralign)
>         sec->sh_addralign = align;
>     return offset;
> }
>
> But that's not used for init_putv.

OK.

> And section_reserve, which is used, doesn't care about alignment at all.
>
> (it seems there's a reserved part and a data part in each section, and
> it holds that the data part elements are aligned--but the reserved part
> elements are NOT aligned.  I don't see how sec->data would be aligned
> by the dynamic memory allocator either)
>
> Other notes:
>
> tccgen.c even has this:
>
>>                /* XXX: when cross-compiling we assume that each type has the
>>                   same representation on host and target, which is likely to
>>                   be wrong in the case of long double */
>
> Yeah, and even when NOT cross-compiling, the alignment is wrong--which means
> it sometimes won't work at all on ARM, depending on luck.
>
> As a workaround, we can patch tcc to instead do the assignments on elements
> on the stack and then copy those over, instead of doing
>
>                 *(double *)ptr = vtop->c.d
>
> (the latter of which emits VFP instructions that expect double-aligned
> pointers).

So alignment should be fixed, but that's more work and you propose a
workaround, right?  I'm struggling to understand the implications of
this last bit...guessing you will be preparing a patch for the mes-0.23
branch of our "bootstrappable tinycc"?  Oh, and we need that same patch
for plain tcc-0.9.27, for "tcc-boot" of course!

Greetings,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: [Tinycc-devel] [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[grischka]

Danny Milosavljevic wrote:
> int main() {
>         double f = 1.0;
>         return 0;
> }
...
> I get a bus error here:
>
> │   0x24698 <init_putv+1688>  vstr  d0, [r0]                                                                                                          │
...
> And indeed, (0x24b01e % 8) == 6, not 0.
...
>                 *(double *)ptr = vtop->c.d
>
> (the latter of which emits VFP instructions that expect double-aligned
> pointers).

It seems that in fact, on certain systems, initializing intentionally
misaligned (packed) structure members could crash tcc already during
compilation.

But no such thing happens in this case.  The 'ptr' in init_putv()
comes from

         ptr = sec->data + c;

and it seems that if tcc is doing the right thing then 'c' cannot
be misaligned, and if malloc/realloc on that system is doing the
right thing,  then sec->data cannot be misaligned either.  So...?

--- grischka

Re: [Tinycc-devel] [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[arnold]

Jan Nieuwenhuizen <janneke@gnu.org> wrote:

> to the gawk-mesbot0 recipe also fixes "inc.awk".  The pre
> increment/decrement code looks like this:
>
> --8<---------------cut here---------------start------------->8---
> 		*lhs = make_number(lval +
> 			       (tree->type == Node_preincrement ? 1.0 : -1.0));
>
> --8<---------------cut here---------------end--------------->8---

What in the world?  That looks like gawk 3.x code, which is
terribly ancient.  What project is still using a version that old?

Arnold
(The gawk maintainer)

Re: [Tinycc-devel] [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Jan Nieuwenhuizen]

> Jan Nieuwenhuizen <janneke@gnu.org> wrote:

Hello Arnold!

>> to the gawk-mesbot0 recipe also fixes "inc.awk".  The pre
>> increment/decrement code looks like this:
>>
>> --8<---------------cut here---------------start------------->8---
>> 		*lhs = make_number(lval +
>> 			       (tree->type == Node_preincrement ? 1.0 : -1.0));
>>
>> --8<---------------cut here---------------end--------------->8---
>
> What in the world?  That looks like gawk 3.x code, which is
> terribly ancient.  What project is still using a version that old?

We are removing binary seeds from the GNU Guix package graph.  The
binary packages in Guix form an acyclic graph and at the bottom of the
graph we originally had binutils, glibc, gcc, bash, coreutils&co (gawk,
gzip, sed, tar, ...).

Since 2016 we have been working to eliminate those binary seeds.  For a
complete overview and more background see

    https://guix.gnu.org/en/blog/2020/guix-further-reduces-bootstrap-seed-to-25/
    https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/

but what we did is remove all those, replacing them by Stage0, GNU Mes,
tinycc...and multiple versions of ancient GNU tools.

Using ancient tools is less than great, we are using those because "it
works" or rather, we didn't succeed as yet using newer versions.  Often,
newer versions of a software are more demanding in their requirements
and are less bootstrappale.  In other cases, ancient software does not
build with newer tools, because they are more strict.

> Arnold
> (The gawk maintainer)

Thanks for reaching out!

Sadly I do not have more concrete information (let alone a bug report or
feature request) for you yet, other than that we are using gawk-3.0.0,
lateron v3.1.8, and only finally v5.0.1.  Simalarly for other tools.

The biggest hudle was bootstrapping glibc and gcc, as you can imagine.
Currently, we start with gcc-2.95.3 and I would very much like to target
gcc-4.6.4 directly instead.  For a tool as gawk, it would be great to
be able to the latest greatest!

Greetings,
Janneke (GNU Mes author)

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: [Tinycc-devel] [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

Hi grischka,
Hi Janneke,

On Fri, 08 Jan 2021 08:16:29 +0100
grischka <grishka@gmx.de> wrote:

> But no such thing happens in this case.  The 'ptr' in init_putv()
> comes from
> 
>          ptr = sec->data + c;
> 
> and it seems that if tcc is doing the right thing then 'c' cannot
> be misaligned, and if malloc/realloc on that system is doing the
> right thing,  then sec->data cannot be misaligned either.  So...?

How does tcc allocate dynamic memory?  I've tried to find out, but
tcc_malloc is defined to be "use_tcc_malloc", which I don't find
anywhere.  Does it use libc malloc for that ?

(With MEM_DEBUG defined, tcc_malloc_debug seems to use malloc.  But
what about without MEM_DEBUG defined ?)

If so, is libc malloc supposed to ensure alignment of allocated memory?

According to https://man7.org/linux/man-pages/man3/malloc.3.html yes.

@Janneke: So our mes libc malloc should be aligning the stuff--but it's not
doing it.  So it's a bug in our libc.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

RE: [bootstrappable] Re: [Tinycc-devel] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Orians, Jeremiah (DTMB)]

> If so, is libc malloc supposed to ensure alignment of allocated memory?
> According to https://man7.org/linux/man-pages/man3/malloc.3.html yes.
> @Janneke: So our mes libc malloc should be aligning the stuff--but it's not doing it.  So it's a bug in our libc.

Looks like you'll have to waste 3.7bytes on average per malloc to always pad to the 8byte boundary. 

-Jeremiah

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

Hi Janneke,

On Fri, 08 Jan 2021 07:25:52 +0100
Jan Nieuwenhuizen <janneke@gnu.org> wrote:

> > Alignment could be disabled on the CPU
> >
> >   https://developer.arm.com/documentation/ddi0464/f/system-control/register-descriptions/system-control-register
> >
> > but I don't think EABI wants that.  
> 
> Hmm, what does this mean?  We are not really using EABI, or are we?

VFP is a floating point unit on ARM CPUs.  It has been designed to either
require aligned members and be fast, or not, depending on a CPU flag that
for example the kernel can set.

See also https://www.keil.com/support/man/docs/armcc/armcc_chr1359124231926.htm
for much more detail.

ARM is a famously mix-and-match CPU, so the client can choose whatever
they want.

If you choose aligned-only, you will get a bus error on misaligned access.
A single program really shouldn't be choosing--it's more the entire platform
doing the choosing.

EABI has standardized on particular settings, among which is required alignment.

<https://www.keil.com/support/man/docs/armcc/armcc_chr1359124228744.htm>

> So alignment should be fixed, but that's more work and you propose a
> workaround, right?  

No, I wanted to find out what's going on first.

Now that grischka commented and I looked up the malloc docs, I suggest
to fix mes libc's malloc to align what it gives back.  That's all--that
should fix everything.

We should provide maxalign_t to make it known to the user what the
malloc alignment we are using is.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1179 bytes --]

Hi Janneke,

I propose to, instead, change mes libc to align stuff malloc returns like this:

That should fix it.

diff --git a/include/stddef.h b/include/stddef.h
index a597c9bb..a682d726 100644
--- a/include/stddef.h
+++ b/include/stddef.h
@@ -37,6 +37,10 @@
 #endif // !__MESC__
 #endif // offsetof
 
+/* TODO: On armhf gcc, max_align_t is 16 Byte big instead.  Use that? */
+
+typedef double max_align_t;
+
 #endif // ! SYSTEM_LIBC
 
 #endif // __MES_STDDEF_H
diff --git a/lib/stdlib/malloc.c b/lib/stdlib/malloc.c
index f4be4de1..aaf99886 100644
--- a/lib/stdlib/malloc.c
+++ b/lib/stdlib/malloc.c
@@ -20,6 +20,8 @@
 
 #include <mes/lib.h>
 #include <string.h>
+#include <stddef.h>
+#include <stdint.h>
 
 /* FIXME: We want bin/mes-mescc's x86-linux sha256sum to stay the same.
    Therfore we cannot remove stdlib/malloc from libc_SOURCES, which is
@@ -37,6 +39,8 @@ malloc (size_t size)
 {
   if (!__brk)
     __brk = (char *) brk (0);
+  /* align what we give back. */
+  __brk = (char*) (((uintptr_t) __brk + sizeof(max_align_t) - 1) & -sizeof(max_align_t));
   if (brk (__brk + size) == -1)
     return 0;
   char *p = __brk;

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Tinycc-devel] [bootstrappable] Re: Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Vincent Lefevre]

On 2021-01-08 13:36:26 +0000, Orians, Jeremiah (DTMB) wrote:
> > If so, is libc malloc supposed to ensure alignment of allocated memory?
> > According to https://man7.org/linux/man-pages/man3/malloc.3.html yes.
> > @Janneke: So our mes libc malloc should be aligning the stuff--but it's not doing it.  So it's a bug in our libc.
> 
> Looks like you'll have to waste 3.7bytes on average per malloc to
> always pad to the 8byte boundary.

Note that it's an 8-byte boundary for 32-bit systems, but a 16-byte
boundary for 64-bit systems:

  https://www.gnu.org/software/libc/manual/html_node/Aligned-Memory-Blocks.html

And about the average waste, this depends on other factors (the
main one may be that the block size is often a multiple of some
power of two).

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: [Tinycc-devel] [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Jan Nieuwenhuizen]

Danny Milosavljevic writes:

Hi Danny, Grishka!

> On Fri, 08 Jan 2021 08:16:29 +0100
> grischka <grishka@gmx.de> wrote:
>
>> But no such thing happens in this case.  The 'ptr' in init_putv()
>> comes from
>> 
>>          ptr = sec->data + c;
>> 
>> and it seems that if tcc is doing the right thing then 'c' cannot
>> be misaligned, and if malloc/realloc on that system is doing the
>> right thing,  then sec->data cannot be misaligned either.  So...?
>
> How does tcc allocate dynamic memory?  I've tried to find out, but
> tcc_malloc is defined to be "use_tcc_malloc", which I don't find
> anywhere.  Does it use libc malloc for that ?
>
> (With MEM_DEBUG defined, tcc_malloc_debug seems to use malloc.  But
> what about without MEM_DEBUG defined ?)
>
> If so, is libc malloc supposed to ensure alignment of allocated memory?
>
> According to https://man7.org/linux/man-pages/man3/malloc.3.html yes.
>
> @Janneke: So our mes libc malloc should be aligning the stuff--but it's not
> doing it.  So it's a bug in our libc.

Beautiful!  Maybe this explains other differences we saw between
aarch64-linux and arm-linux?

Greetings,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Jan Nieuwenhuizen]

Danny Milosavljevic writes:

Hi Danny,

> I propose to, instead, change mes libc to align stuff malloc returns like this:
>
> That should fix it.

That's great; I'd like to go test this.  Would you like to push to "wip"
when you're ready?

> diff --git a/include/stddef.h b/include/stddef.h
> index a597c9bb..a682d726 100644
> --- a/include/stddef.h
> +++ b/include/stddef.h
> @@ -37,6 +37,10 @@
>  #endif // !__MESC__
>  #endif // offsetof
>  
> +/* TODO: On armhf gcc, max_align_t is 16 Byte big instead.  Use that? */
> +
> +typedef double max_align_t;
> +
>  #endif // ! SYSTEM_LIBC

Is this something you can get more info on, or do we just try it like
this?

>  #endif // __MES_STDDEF_H
> diff --git a/lib/stdlib/malloc.c b/lib/stdlib/malloc.c
> index f4be4de1..aaf99886 100644
> --- a/lib/stdlib/malloc.c
> +++ b/lib/stdlib/malloc.c
> @@ -20,6 +20,8 @@
>  
>  #include <mes/lib.h>
>  #include <string.h>
> +#include <stddef.h>
> +#include <stdint.h>
>  
>  /* FIXME: We want bin/mes-mescc's x86-linux sha256sum to stay the same.
>     Therfore we cannot remove stdlib/malloc from libc_SOURCES, which is
> @@ -37,6 +39,8 @@ malloc (size_t size)
>  {
>    if (!__brk)
>      __brk = (char *) brk (0);
> +  /* align what we give back. */
> +  __brk = (char*) (((uintptr_t) __brk + sizeof(max_align_t) - 1) & -sizeof(max_align_t));
>    if (brk (__brk + size) == -1)
>      return 0;
>    char *p = __brk;

Very nice, thanks!!
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

Hi Janneke,

On Fri, 08 Jan 2021 17:15:24 +0100
Jan Nieuwenhuizen <janneke@gnu.org> wrote:

> > +/* TODO: On armhf gcc, max_align_t is 16 Byte big instead.  Use that? */
> > +
> > +typedef double max_align_t;
> > +
> >  #endif // ! SYSTEM_LIBC  
> 
> Is this something you can get more info on, or do we just try it like
> this?

I would just try like this.  I mean I'm sure we could find out what the
16 Byte thing is (long double ?  Nope, not according to both gcc and tcc
on armhf)--but seriously, mescc can't represent that object anyhow--so
there's really no upside to making this bigger (if mescc and tinycc
don't emit it, it doesn't need to be aligned either :) ).

The CI on nanana is currently building and running the tests.

I'm curious what it will say.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 324 bytes --]

Hi Janneke,

On Fri, 8 Jan 2021 19:56:19 +0100
Danny Milosavljevic <dannym@scratchpost.org> wrote:

> The CI on nanana is currently building and running the tests.
> 
> I'm curious what it will say.

Tests succeeded.

Pushed to mes on savannah as commit 10c38e112f177bc0b01ecf107ffffd193e4c6b13
("wip" branch).

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Jan Nieuwenhuizen]

Danny Milosavljevic writes:

Hi Danny,

> On Fri, 8 Jan 2021 19:56:19 +0100
> Danny Milosavljevic <dannym@scratchpost.org> wrote:
>
>> The CI on nanana is currently building and running the tests.
>> 
>> I'm curious what it will say.
>
> Tests succeeded.
>
> Pushed to mes on savannah as commit 10c38e112f177bc0b01ecf107ffffd193e4c6b13
> ("wip" branch).

Lovely, thanks!

I'll inject it into wip-arm-bootstrap and see what happens.  Especially
I hope we'll be able to remove these hacks

--8<---------------cut here---------------start------------->8---
ebd1a594 HACK bootstrappable: ARM: "tccgen_ok".
8d475711 HACK bootstrappable: ARM: "tccpp_ok".
--8<---------------cut here---------------end--------------->8---

from bootstrappable tinycc.

Greetings,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Ludovic Courtès]

Hi!

Andrius Štikonas <andrius@stikonas.eu> skribis:

> I think that's what mes-m2 rewrite [1] (not to be confused with mes wip-m2 branch)
> is trying to achieve.

Oh I see.  It’s still kinda confusing to have two Mes.  Wouldn’t it be
nice to have just one?  I understand the goals are not exactly the same,
but is there some way to converge?  (This is a naive question, you’ve
probably already thought about it, but anyways.  :-))

> Outside of Guix we are working on bootstrap that does not depend on guile driver
> and is driven only by hex-0 seed (357 bytes) kaem-optional-seed (737 bytes) and any POSIX
> kernel.

Nice!

Ludo’.

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[jeremiah]

>> I think that's what mes-m2 rewrite [1] (not to be confused with mes wip-m2 branch)
>> is trying to achieve.
> Oh I see.  It’s still kinda confusing to have two Mes.  Wouldn’t it be
> nice to have just one?
yes it would.

Which is why a third scheme is being written in the Haskell subset we
are bootstrapping.

Ironically time from M2-Planet to Haskell was just a couple of weeks of work.

but M2-Planet to scheme is a bit of a pain point, as janneke and I seem
to have very different styles for scheme in C.

But that might simply because I spent the last 4 years dealing with hex
and assembly and my scheme code is now crap.

> I understand the goals are not exactly the same,
> but is there some way to converge?  (This is a naive question, you’ve
> probably already thought about it, but anyways.  :-))
There are 3 easy ways to converge the two.

1) transplant janneke's wip-m2 eval into mes-m2 to solve the mes-m2
macro problem enough to run MesCC (but might make guile compatibility harder).

2) transplant my mes-m2's garbage collector into wip-m2 to solve the wip-m2
pointer arithmetic problem (but is even farther from guile compatibilty).

3) not actually converge the code and simply throw one or both of them
away. Say write the whole thing in a better language than C (haskell
perhaps but ultimately requires abandoning previous work).

I wish I had better answers but we still have guile's psyntax.pp
bootstrapping problem and figuring out how to do syntax-case in C is a
b**** of a problem; not even having to deal with the do this with a
minimal C compiler restrictions involved.

If we pulled the scheme macro requirement out, then the number of
minimal schemes which could run MesCC would explode.

But it seems unlikely such a change would occur as macro-less scheme is
no more productive than standard C coding.

- Jeremiah

Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Timothy Sample]

Hi janneke,

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> I have reset Guix' wip-full-source-bootstrap branch with a first working
> implementation of the, well, "Full Source Bootstrap" for x86-linux (and
> x86_64-linux).  This bootstrap is rooted in the 357-byte hex0-seed from
> the Stage0 project (https://savannah.gnu.org/projects/stage0).

The dream is alive!  Congratulations on this big leap forward!

> When you look at the bottom of the graph (see attached), you will notice
> "%bootstrap-guile": the driver that we use for the Guix build and also
> for "bootar", "gash", and "gash-utils".  This "%bootstrap-guile" is not
> used as a seed in anything that is built, "%bootstrap-guile", "bootar",
> "gash", and "gash-utils" could be replaced with any other driver.

I never mentioned it, but a few months ago I took a little look at
porting Gash & friends to Mes.  The big issue that I ran into is that
Mes doesn’t really have a module system.  My plan was to build up Mes
modules and strip down Gash requirements until they met in the middle.
Sometime (probably not worth derailing this thread right now) we should
discuss what needs to be done for Mes modules.  (It looked like
something I could do with a little guidance on the design.)

> XXX TODO:
>    * wip-full-source-bootstrap
>      [...]
>    * wip-arm-bootstrap
>      - finish; currently stuck on gawk-mesboot0
>      [...]

It looks like you’ve made a lot of progress on this already (judging by
the rest of this thread).  However, if it helps, the current Gash-Utils
awk could _probably_ be used to skip most (all?) of the old versions of
Gawk.

Sorry I can’t be more helpful ATM.  I appreciate the work you do to keep
this project rolling!


-- Tim

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Ludovic Courtès]

Hi,

jeremiah@pdp10.guru skribis:

> 3) not actually converge the code and simply throw one or both of them
> away. Say write the whole thing in a better language than C (haskell
> perhaps but ultimately requires abandoning previous work).

I see a fourth option, which is to keep both.  :-)

In effect, it seems there are now two diverging projects.  I think
that’s fine: more bootstrapping work and more diversity is better!

For Guix, the Scheme-based approach Janneke et al. have been pursing
remains the most attractive.  At any rate, work on Haskell will probably
benefit Guix (and other distros I guess!) to have a fully
built-from-source Haskell platform.

Thanks for explaining,
Ludo’.

RE: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Orians, Jeremiah (DTMB)]

> I see a fourth option, which is to keep both.  :-)
Might want to fix up the confusing naming in that case

> In effect, it seems there are now two diverging projects.  I think that’s fine: more bootstrapping work and more diversity is better!
Converging actually as they share the exact same goal of bootstrap from nothing and run MesCC

> For Guix, the Scheme-based approach Janneke et al. have been pursing remains the most attractive. 
And would be faster if MesCC running on guile was used as the lone bootstrap seed.

> At any rate, work on Haskell will probably benefit Guix (and other distros I guess!) to have a fully built-from-source Haskell platform.
Indeed

-Jeremiah

Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Efraim Flashner]

[-- Attachment #1: Type: text/plain, Size: 2385 bytes --]

On Wed, Jan 20, 2021 at 03:19:49PM -0500, Timothy Sample wrote:
> Hi janneke,
> 
> Jan Nieuwenhuizen <janneke@gnu.org> writes:
> 
> > I have reset Guix' wip-full-source-bootstrap branch with a first working
> > implementation of the, well, "Full Source Bootstrap" for x86-linux (and
> > x86_64-linux).  This bootstrap is rooted in the 357-byte hex0-seed from
> > the Stage0 project (https://savannah.gnu.org/projects/stage0).
> 
> The dream is alive!  Congratulations on this big leap forward!
> 
> > When you look at the bottom of the graph (see attached), you will notice
> > "%bootstrap-guile": the driver that we use for the Guix build and also
> > for "bootar", "gash", and "gash-utils".  This "%bootstrap-guile" is not
> > used as a seed in anything that is built, "%bootstrap-guile", "bootar",
> > "gash", and "gash-utils" could be replaced with any other driver.
> 
> I never mentioned it, but a few months ago I took a little look at
> porting Gash & friends to Mes.  The big issue that I ran into is that
> Mes doesn’t really have a module system.  My plan was to build up Mes
> modules and strip down Gash requirements until they met in the middle.
> Sometime (probably not worth derailing this thread right now) we should
> discuss what needs to be done for Mes modules.  (It looked like
> something I could do with a little guidance on the design.)
> 
> > XXX TODO:
> >    * wip-full-source-bootstrap
> >      [...]
> >    * wip-arm-bootstrap
> >      - finish; currently stuck on gawk-mesboot0
> >      [...]
> 
> It looks like you’ve made a lot of progress on this already (judging by
> the rest of this thread).  However, if it helps, the current Gash-Utils
> awk could _probably_ be used to skip most (all?) of the old versions of
> Gawk.
> 
> Sorry I can’t be more helpful ATM.  I appreciate the work you do to keep
> this project rolling!
> 

Using this post as inspiration I replaced diffutils-mesboot with
gash-utils-boot. diffutils-mesboot provided cmp and diff, both of which
are available in gash-utils. Unfortunately sed from gash-utils-boot
didn't seem to work so I wasn't able to remove that.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

ARM Unified Assembly Language - GNU as does some weird stuff

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 2929 bytes --]

Hello Paul,

we are now implementing ARM inline assembly in TinyCC.

The traditional ARM inline assembler is finished now.  Now we started
implementing the Unified Assembly Language dialect.

I'd like to have some help which constructs are valid and which are not.

(GNU as 2.34 seems to have some bugs with Unified Assembly Language--so I
guess we shouldn't use "do what GNU as does" as a yardstick here)

The questionable--maybe invalid--constructs follow (the "#" are for clarity):

================================================================================

(1) b #60

It seems that GNU as ignores the immediate entirely and just always encodes
#0 (to test, do ".syntax unified" and then "b #60" in GNU as).  WTF?

Likewise with bl, blx.

It seems that the debug info still has the user-specified immediate value--but
the executable object file does not.  So in order to test you should strip the
resulting file--otherwise you are gonna be very confused.

No error or warning messages are printed by GNU as here.

================================================================================

(2) push #4

It works in GNU as--but is it specified by ARM to push the register r2 ?
I think exposing ISA implementation details like that is a leaky abstraction--and
no good can come from it.

Likewise with pop, stm*, ldm*.

No error or warning messages are printed by GNU as here.

================================================================================

(3) and r3, r4, LSL #5

This doesn't work in unified mode--but does work in non-unified mode.
Note that "and r3, r3, r4, LSL #5" always works--both in unified mode and in
non-unified mode.

I would have thought the LSL token would be a dead giveaway for disambiguating
what to do here--but apparently GNU as thinks otherwise.

Likewise with ands, eor, eors, sub, subs, sbc, sbcs, rsc, rscs, rsb, rsbs,
orr, orrs, bic, bics, cmp, cmns, mov, movs, mvn, mvns (all arithmetic
instructions), and with other modifiers (LSR, ASR, ROR, RRX).

GNU as fails to assemble these.

================================================================================

(4) lsl r1, #4, #2

GNU as encodes exactly the same as "lsl r1, #4"--drops the "#2" silently.

Likewise with lsr, lsrs, asr, asrs, orr, rors.

No error or warning messages are printed by GNU as here.

================================================================================

(5) vmov.f32 r2, r3, d1

f32 and two 32 bit ARM registers, and one 64 bit VFP register?  What does this
do?!

Likewise with "vmov.f32 d1, r2, r3".

No error or warning messages are printed by GNU as here.

================================================================================

Which of those are good (and should thus be implemented in the same way)
and which of those are bad (and should thus not be implemented the
same way)?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

RE: [bootstrappable] ARM Unified Assembly Language - GNU as does some weird stuff

[Orians, Jeremiah (DTMB)]

> (1) b #60
> It seems that GNU as ignores the immediate entirely and just always encodes
> #0 (to test, do ".syntax unified" and then "b #60" in GNU as).  WTF?
> Likewise with bl, blx.

All assemblers (except M1) do that because the linker is to populate that value at link time using the symbol table to allow code segment relocation.

> (2) push #4
> It works in GNU as--but is it specified by ARM to push the register r2 ?
> I think exposing ISA implementation details like that is a leaky abstraction--and no good can come from it.
> Likewise with pop, stm*, ldm*.

Well arm doesn't actually have a push/pop instructions, only load and store instructions

> GNU as fails to assemble these.
And no one cared enough to fix it for 30 years? Is it possible something is missed?

> (4) lsl r1, #4, #2
> GNU as encodes exactly the same as "lsl r1, #4"--drops the "#2" silently.
4 << 2 is 16. Log2(16) == 4; sounds about right

-Jeremiah

Re: ARM Unified Assembly Language - GNU as does some weird stuff

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 616 bytes --]

> (1) b #60
> 
> It seems that GNU as ignores the immediate entirely and just always encodes
> #0 (to test, do ".syntax unified" and then "b #60" in GNU as).  WTF?
> 
> Likewise with bl, blx.
> 
> It seems that the debug info still has the user-specified immediate value--but
> the executable object file does not.  So in order to test you should strip the
> resulting file--otherwise you are gonna be very confused.
> 
> No error or warning messages are printed by GNU as here.

With GNU as 2.34 on ARM the same happens for:

  b r3

It encodes as

     eafffffe        b       0x0

Spooky...

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Ludovic Courtès]

Hi,

"Orians, Jeremiah (DTMB)" <OriansJ@michigan.gov> skribis:

>> I see a fourth option, which is to keep both.  :-)
> Might want to fix up the confusing naming in that case

Yes, that’s probably a good idea.

>> In effect, it seems there are now two diverging projects.  I think that’s fine: more bootstrapping work and more diversity is better!
> Converging actually as they share the exact same goal of bootstrap from nothing and run MesCC

My understanding is that they are nevertheless two different development
branches; is that right?

>> For Guix, the Scheme-based approach Janneke et al. have been pursing remains the most attractive. 
> And would be faster if MesCC running on guile was used as the lone bootstrap seed.

What do you mean?  Faster in what sense?

Again I’m commenting as an enthusiastic outsider, so I may miss pieces
of the puzzle.  My impression is that we have now two different things
called “Mes” and two different approaches to the whole issue, so it
might make sense to label them differently.

At any rate, thanks to all the bootstrappers for the amazing work!

Ludo’.

RE: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Orians, Jeremiah (DTMB)]

>>> In effect, it seems there are now two diverging projects.  I think that’s fine: more bootstrapping work and more diversity is better!
>> Converging actually as they share the exact same goal of bootstrap 
>> from nothing and run MesCC
> My understanding is that they are nevertheless two different development branches; is that right?
There was a lot of cross-pollination between them during the original M2-Planet build attempt on Mes.c

>>> For Guix, the Scheme-based approach Janneke et al. have been pursing remains the most attractive.
>> And would be faster if MesCC running on guile was used as the lone bootstrap seed.
> What do you mean?  Faster in what sense?
Guile is a faster scheme than mes.c

-Jeremiah

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Jan Nieuwenhuizen]

Efraim Flashner writes:

Hi!

> On Wed, Jan 20, 2021 at 03:19:49PM -0500, Timothy Sample wrote:
>> Hi janneke,
>> 
>> Jan Nieuwenhuizen <janneke@gnu.org> writes:
>> 
>> It looks like you’ve made a lot of progress on this already (judging by
>> the rest of this thread).  However, if it helps, the current Gash-Utils
>> awk could _probably_ be used to skip most (all?) of the old versions of
>> Gawk.
>> 
>> Sorry I can’t be more helpful ATM.  I appreciate the work you do to keep
>> this project rolling!
>> 
>
> Using this post as inspiration I replaced diffutils-mesboot with
> gash-utils-boot. diffutils-mesboot provided cmp and diff, both of which
> are available in gash-utils.

Oh, that's nice.  It would be good to get rid of as much unspported
software releases as we can.

> Unfortunately sed from gash-utils-boot
> didn't seem to work so I wasn't able to remove that.

Yeah, distilling a nice bug report for that can take quite some time.

Greetigns,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[andrius--- via Development of GNU Guix and the GNU System distribution.]

[-- Attachment #1: Type: text/plain, Size: 1702 bytes --]

2021 m. sausio 29 d., penktadienis 07:23:24 GMT Jan Nieuwenhuizen rašė:
> Efraim Flashner writes:
> 
> Hi!
> 
> > On Wed, Jan 20, 2021 at 03:19:49PM -0500, Timothy Sample wrote:
> >> Hi janneke,
> >> 
> >> Jan Nieuwenhuizen <janneke@gnu.org> writes:
> >> 
> >> It looks like you’ve made a lot of progress on this already (judging by
> >> the rest of this thread).  However, if it helps, the current Gash-Utils
> >> awk could _probably_ be used to skip most (all?) of the old versions of
> >> Gawk.
> >> 
> >> Sorry I can’t be more helpful ATM.  I appreciate the work you do to keep
> >> this project rolling!
> >> 
> >
> > Using this post as inspiration I replaced diffutils-mesboot with
> > gash-utils-boot. diffutils-mesboot provided cmp and diff, both of which
> > are available in gash-utils.

It would be better if gash-utils worked with mes though. Now they all run in guile.
In live-bootstrap project (bootstrapping from hex0 with just kaem driver)
we had to skip gash and all gash-utils. Fortunately we now mostly got relevant GNU utils.

By the way we managed to build reasonably new sed 4.0.7 instead of sed 1.18 which is used in guix bootstrap

Andrius

> 
> Oh, that's nice.  It would be good to get rid of as much unsupported
> software releases as we can.
> 
> > Unfortunately sed from gash-utils-boot
> > didn't seem to work so I wasn't able to remove that.
> 
> Yeah, distilling a nice bug report for that can take quite some time.
> 
> Greetigns,
> Janneke
> 
> 


-- 
I encourage the use of end to end email encryption

GPG key:   https://stikonas.eu/andrius.asc
Fingerprint:  1EE5 A320 5904 BAA2 B88C 0A9D 24FD 3194 0095 C0E1

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

RE: [bootstrappable] Re: wip-full-source-bootstrap: from a 357-byte `hex0' to 'hello'

[Orians, Jeremiah (DTMB)]

>>> Using this post as inspiration I replaced diffutils-mesboot with 
>>> gash-utils-boot. diffutils-mesboot provided cmp and diff, both of 
>>> which are available in gash-utils.
> It would be better if gash-utils worked with mes though. Now they all run in guile.
> In live-bootstrap project (bootstrapping from hex0 with just kaem driver) we had to skip gash and all gash-utils. Fortunately we now mostly got relevant GNU utils.

Hence the overly ambitious goal of mes-m2 (Which might end up being a dead end) and a proper scheme written in the Haskell Subset supported by blynn-compiler which has been bootstrapped.

We have to think long term here, because we are going to have to support the bootstrap forever.
And porting to new architectures and Operating Systems is going to be something we will have to deal with.

- Jeremiah