Update on wip-arm-bootstrap

Update on wip-arm-bootstrap

[Jan Nieuwenhuizen]

Hi,

Last month, we found that

--8<---------------cut here---------------start------------->8---
// prereq.c
#if defined __GNUC__ && defined __GNUC_MINOR__
# define __GNUC_PREREQ(maj, min) \
	((__GNUC__ << 16) + __GNUC_MINOR__ >= ((maj) << 16) + (min))
#else
# define __GNUC_PREREQ(maj, min) 0
#endif

#if __GNUC_PREREQ (3,1) && !defined __GNUG__
#error gcc too new
#endif
--8<---------------cut here---------------end--------------->8---

compiled wrong with gcc-core-mesboot0:

--8<---------------cut here---------------start------------->8---
16:34:37 janneke@novena:~/src/guix [env]
$ /gnu/store/h0v5by5fgvcqi5gkhq0yi7ma6nhyp2ql-gcc-core-mesboot0-2.95.3/bin/gcc -S prereq.c 
prereq.c:8: warning: integer constant out of range
prereq.c:8: warning: integer constant out of range
prereq.c:8: warning: integer constant out of range
prereq.c:9: #error gcc too new
[1]16:35:27 janneke@novena:~/src/guix [env]
--8<---------------cut here---------------end--------------->8---

Well, Danny identified the problem in gcc-core-mesboot0:

--8<---------------cut here---------------start------------->8---
//noadd.c
#if (1 >= 10)
#endif
--8<---------------cut here---------------end--------------->8---

=>

--8<---------------cut here---------------start------------->8---
$ /gnu/store/h0v5by5fgvcqi5gkhq0yi7ma6nhyp2ql-gcc-core-mesboot0-2.95.3/bin/gcc -E noadd.c
noadd.c:1: warning: integer constant out of range
# 1 "noadd.c"
--8<---------------cut here---------------end--------------->8---

which lead to

--8<---------------cut here---------------start------------->8---
<dannym> "#if (5)" works
<dannym> "#if (10)" is broken  [13:10]
<janneke> any idea where that code "lives"
<janneke> ?
<dannym> gcc/cppexp.c  [13:13]
<dannym> Maybe "case CPP_NUMBER"
<dannym> -> parse_number
--8<---------------cut here---------------end--------------->8---

That turned out to be caused by a problems in Mes lib c

--8<---------------cut here---------------start------------->8---
<dannym> janneke: I fixed the __*div* functions in mes a little--commit
https://git.savannah.gnu.org/cgit/mes.git/commit/?id=33cf5ea5e820e21a8f46de7df08a8b49bb8f62ee
--8<---------------cut here---------------end--------------->8---

functions that used to be stubs on x86, need a working implementation on
ARM.  Okay, that makes sense.  Danny pushes an update for Mes to
wip-arm-bootstrap.

[13:05:53] <dannym> And then ../sysdeps/unix/sysv/linux/arm/sigaction.c:139: `__NR_sigaction' undeclared (first use in this function)
[13:07:32] <janneke> dannym: that's great!
[13:08:31] <janneke> dannym: seems familiar
[13:09:04] <janneke> dannym: when trying to get glibc-mesboot0 built, i was collecting a number of patches
[13:10:10] <janneke> i didn't share or push these yet (or maybe reverted), as they became more uninformed and hacky as i went along

Okay, Janneke sorts-out glibc patches for ARM and pushes an update to
wip-arm-bootstrap.  And now, we can build glibc-mesboot0!

--8<---------------cut here---------------start------------->8---
[21:39:23] <janneke> dannym: amazing: /gnu/store/jgkf60r29blzhrg0w3dar3rz06xwkx5q-glibc-mesboot0-2.2.5
[21:39:37] <dannym> \o/
[21:40:34] <janneke> yeah \o/
--8<---------------cut here---------------end--------------->8---

As we were stuck on building the first glibc for quite some time now,
this was something to celebrate!

Sadly, the gcc-core-mesboot0 + glibc-mesboot0 combo produces executables
that "Illegal instruction" upon syscalls.

--8<---------------cut here---------------start------------->8---
Program received signal SIGILL, Illegal instruction.
0x000276bc in __writev (fd=2, vector=0xbeffd010, count=10) at ../sysdeps/unix/sysv/linux/writev.c:51
51	  bytes_written = INLINE_SYSCALL (writev, 3, fd, CHECK_N (vector, count), count);
(gdb) disas /r
Dump of assembler code for function __writev:
   0x00027698 <+0>:	0d c0 a0 e1	mov	r12, sp
   0x0002769c <+4>:	f0 dd 2d e9	push	{r4, r5, r6, r7, r8, r10, r11, r12, lr, pc}
   0x000276a0 <+8>:	04 b0 4c e2	sub	r11, r12, #4
   0x000276a4 <+12>:	02 50 a0 e1	mov	r5, r2
   0x000276a8 <+16>:	01 70 a0 e1	mov	r7, r1
   0x000276ac <+20>:	00 60 a0 e1	mov	r6, r0
   0x000276b0 <+24>:	6c a0 9f e5	ldr	r10, [pc, #108]	; 0x27724 <__writev+140>
   0x000276b4 <+28>:	00 80 9a e5	ldr	r8, [r10]
   0x000276b8 <+32>:	14 00 00 ef	svc	0x00000014
=> 0x000276bc <+36>:	00 40 a0 e1	mov	r4, r0
--8<---------------cut here---------------end--------------->8---

Let's try to bisect where the problem is; we now have tree first
candidates: gcc-core-mesboot0, glibc-mesboot0 and binutils-mesboot0.
Luckily, Debian "woody" carries an almost compatible set.  Doing
someting like

--8<---------------cut here---------------start------------->8---
guix environment --ad-hoc binutils patchelf wget
wget http://archive.debian.org/debian/pool/main/g/glibc/libc6_2.2.5-11.8_arm.deb
ar x libc6_2.2.5-11.8_arm.deb 
tar xf data.tar.gz 
wget http://archive.debian.org/debian/pool/main/g/glibc/libc6-dev_2.2.5-11.8_arm.deb
ar x libc6-dev_2.2.5-11.8_arm.deb 
tar xf data.tar.gz 
wget http://archive.debian.org/debian/pool/main/b/binutils/binutils_2.12.90.0.1-4_arm.deb
ar x binutils_2.12.90.0.1-4_arm.deb 
tar xf data.tar.gz 
wget http://archive.debian.org/debian/pool/main/g/gcc-2.95/gcc-2.95_2.95.4-27_arm.deb
ar x gcc-2.95_2.95.4-27_arm.deb
tar xf data.tar.gz 
patchelf --print-interpreter usr/bin/as
/lib/ld-linux.so.2
patchelf --set-interpreter $PWD/lib/ld-linux.so.2 usr/bin/as
usr/bin/as
Illegal instruction
--8<---------------cut here---------------end--------------->8---

Hmm...does it?  Using gdb, the problem looks...

--8<---------------cut here---------------start------------->8---
Program received signal SIGILL, Illegal instruction.
0xb6ff3b6c in writev () from /home/janneke/src/debian/lib/ld-linux.so.2
(gdb) disas /r
Dump of assembler code for function writev:
[..]
   0xb6ff3b58 <+28>:	05 20 a0 e1	mov	r2, r5
   0xb6ff3b5c <+32>:	07 10 a0 e1	mov	r1, r7
   0xb6ff3b60 <+36>:	00 80 90 e5	ldr	r8, [r0]
   0xb6ff3b64 <+40>:	06 00 a0 e1	mov	r0, r6
   0xb6ff3b68 <+44>:	92 00 90 ef	svc	0x00900092
=> 0xb6ff3b6c <+48>:	00 40 a0 e1	mov	r4, r0
--8<---------------cut here---------------end--------------->8---

...pretty familiar.  So, what's going on here?  Do the "woody"
binaries not run on novena?

--8<---------------cut here---------------start------------->8---
$ uname -a
Linux novena 5.9.11-gnu #1 SMP 1 armv7l GNU/Linux
10:11:54 janneke@novena:~/src/debian [env]
$ readelf -h usr/bin/as
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 61 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            ARM
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x9c34
  Start of program headers:          52 (bytes into file)
  Start of section headers:          267716 (bytes into file)
  Flags:                             0x2, GNU EABI, <unknown>
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         7
  Size of section headers:           40 (bytes)
  Number of section headers:         25
  Section header string table index: 24
--8<---------------cut here---------------end--------------->8---

Overdrive1 seems to think so, and respects "fail early"

--8<---------------cut here---------------start------------->8---
$ uname -a
Linux overdrive1 5.8.13-gnu #1 SMP 1 aarch64 GNU/Linux
10:18:39 janneke@overdrive1:~/src/debian [env]
$ usr/bin/as
bash: usr/bin/as: cannot execute binary file: Exec format error
--8<---------------cut here---------------end--------------->8---

Hmm?

Greetings,
Janneke & Danny

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Update on wip-arm-bootstrap

[Ludovic Courtès]

Hi!

I read the story, which I found rather fun and full of suspense, but I
admit I was disappointed by the ending.  :-)

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> ...pretty familiar.  So, what's going on here?  Do the "woody"
> binaries not run on novena?

Does that mean there are no old reference binaries known to work on
Novena?

Ludo’.

Re: Update on wip-arm-bootstrap

[Vagrant Cascadian]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]

On 2021-02-13, Jan Nieuwenhuizen wrote:
> Let's try to bisect where the problem is; we now have tree first
> candidates: gcc-core-mesboot0, glibc-mesboot0 and binutils-mesboot0.
> Luckily, Debian "woody" carries an almost compatible set.  Doing
> someting like
>
> --8<---------------cut here---------------start------------->8---
> guix environment --ad-hoc binutils patchelf wget
> wget http://archive.debian.org/debian/pool/main/g/glibc/libc6_2.2.5-11.8_arm.deb
> ar x libc6_2.2.5-11.8_arm.deb 
> tar xf data.tar.gz 
> wget http://archive.debian.org/debian/pool/main/g/glibc/libc6-dev_2.2.5-11.8_arm.deb
> ar x libc6-dev_2.2.5-11.8_arm.deb 
> tar xf data.tar.gz 
> wget http://archive.debian.org/debian/pool/main/b/binutils/binutils_2.12.90.0.1-4_arm.deb
> ar x binutils_2.12.90.0.1-4_arm.deb 
> tar xf data.tar.gz 
> wget http://archive.debian.org/debian/pool/main/g/gcc-2.95/gcc-2.95_2.95.4-27_arm.deb
> ar x gcc-2.95_2.95.4-27_arm.deb
> tar xf data.tar.gz 
> patchelf --print-interpreter usr/bin/as
> /lib/ld-linux.so.2
> patchelf --set-interpreter $PWD/lib/ld-linux.so.2 usr/bin/as
> usr/bin/as
> Illegal instruction
> --8<---------------cut here---------------end--------------->8---
>
> Hmm...does it?  Using gdb, the problem looks...
>
> --8<---------------cut here---------------start------------->8---
> Program received signal SIGILL, Illegal instruction.
> 0xb6ff3b6c in writev () from /home/janneke/src/debian/lib/ld-linux.so.2
> (gdb) disas /r
> Dump of assembler code for function writev:
> [..]
>    0xb6ff3b58 <+28>:	05 20 a0 e1	mov	r2, r5
>    0xb6ff3b5c <+32>:	07 10 a0 e1	mov	r1, r7
>    0xb6ff3b60 <+36>:	00 80 90 e5	ldr	r8, [r0]
>    0xb6ff3b64 <+40>:	06 00 a0 e1	mov	r0, r6
>    0xb6ff3b68 <+44>:	92 00 90 ef	svc	0x00900092
> => 0xb6ff3b6c <+48>:	00 40 a0 e1	mov	r4, r0
> --8<---------------cut here---------------end--------------->8---
>
> ...pretty familiar.  So, what's going on here?  Do the "woody"
> binaries not run on novena?

My guess would be OABI (debian "arm" architecture) vs. EABI (debian
"armel" or "armhf" architectures). The hardware may likly support OABI,
but the kernel may need a compatibility layer enabled.

  https://wiki.debian.org/ArmPorts


live well,
  vagrant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: Update on wip-arm-bootstrap

[Jan Nieuwenhuizen]

Vagrant Cascadian writes:

Hi!

> On 2021-02-13, Jan Nieuwenhuizen wrote:
[..]
>> ...pretty familiar.  So, what's going on here?  Do the "woody"
>> binaries not run on novena?
>
> My guess would be OABI (debian "arm" architecture) vs. EABI (debian
> "armel" or "armhf" architectures). The hardware may likly support OABI,
> but the kernel may need a compatibility layer enabled.
>
>   https://wiki.debian.org/ArmPorts

Ah, interesting.  The kernel config has

# CONFIG_OABI_COMPAT is not set

...certainly a lot easier to find when you know what you're looking
for.

@Danny: I'm wondering if we could (should?) try a kernel with OABI
compatibility?  I suppose it would be better to somehow target
EABI...but I'm not sure that's possible with glibc-2.2.5 / gcc-2.95.3.
Interesting choices here!

Greetings,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Update on wip-arm-bootstrap

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1138 bytes --]

Hi Janneke,

On Thu, 18 Feb 2021 22:52:57 +0100
Jan Nieuwenhuizen <janneke@gnu.org> wrote:

> # CONFIG_OABI_COMPAT is not set
> 
> ...certainly a lot easier to find when you know what you're looking
> for.
> 
> @Danny: I'm wondering if we could (should?) try a kernel with OABI
> compatibility?  I suppose it would be better to somehow target
> EABI...but I'm not sure that's possible with glibc-2.2.5 / gcc-2.95.3.
> Interesting choices here!

OABI is older than year 2000, and the kernel docs say not to enable it.
It also breaks seccomp.
I doubt that people have it enabled, and thus would have trouble reproducing
our stuff.  Since this only affects the syscall interface and since also our
ELF headers specify EABI, I would just change the syscalls to EABI:
Just put the syscall number into r7 and use svc 0.

I'd do it myself but I don't see what libc the gcc 2.95 we built has been
using.
Is it ours?
If so, how come it then uses svc 90000... in the first place? 
We don't do that.

Or is it using glibc ?

How do I build that gcc on novena?  Where would the syscall headers that
I could change be?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Update on wip-arm-bootstrap

[Jan Nieuwenhuizen]

Danny Milosavljevic writes:

Hey Danny,

> On Thu, 18 Feb 2021 22:52:57 +0100
> Jan Nieuwenhuizen <janneke@gnu.org> wrote:
>
>> # CONFIG_OABI_COMPAT is not set
>> 
>> ...certainly a lot easier to find when you know what you're looking
>> for.
>> 
>> @Danny: I'm wondering if we could (should?) try a kernel with OABI
>> compatibility?  I suppose it would be better to somehow target
>> EABI...but I'm not sure that's possible with glibc-2.2.5 / gcc-2.95.3.
>> Interesting choices here!
>
> OABI is older than year 2000, and the kernel docs say not to enable it.
> It also breaks seccomp.
> I doubt that people have it enabled, and thus would have trouble reproducing
> our stuff.

Okay, I agree; better to not go this route if we can avoid it.

> Since this only affects the syscall interface and since also our
> ELF headers specify EABI, I would just change the syscalls to EABI:
> Just put the syscall number into r7 and use svc 0.

Oh, if that's all we should be able to find and fix it?

> I'd do it myself but I don't see what libc the gcc 2.95 we built has been
> using.
> Is it ours?
> If so, how come it then uses svc 90000... in the first place? 
> We don't do that.
>
> Or is it using glibc ?

Not it's not ours, it's gcc-core-mesboot0-2.95.3 binaries built against
glibc-mesboot0-2.2.5.  So then it's probably glibc-2.2.5 that is
using/built for OABI.  I didn't know of OABI before, but did look for
EABI options and didn't find any in those early gcc/glibc's...

We'll have to see if that glibc can be built for OABI, possibly by
patching it.  If we need to upgrade glibc then that may be an
"interesting" undertaking...

From Vagrant's link, I found that Lenny (5.0) is the first armel (EABI)
release, it uses glibc-2.7, gcc-4.1, binutils-2.18.

> How do I build that gcc on novena?  Where would the syscall headers that
> I could change be?

On latest wip-arm-bootstrap

    3c78496c32 DRAFT gnu: mes: Update to 0.22-124-33cf5ea5e8.

then building

    ./pre-inst-env guix build -e '(@@ (gnu packages commencement) gcc-mesboot0)'

will be using the gcc,glibc,binutils that's broken and configure will
fail on running a conftest.  You can get the PATH and other environment
settings from that build.  I have those settings also in a script:

    novena:~janneke/src/guix/gcc-mesboot.sh

is a reproducer.

Greetings,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Update on wip-arm-bootstrap

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hello,

> I read the story, which I found rather fun and full of suspense, but I
> admit I was disappointed by the ending.  :-)
>
> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
>> ...pretty familiar.  So, what's going on here?  Do the "woody"
>> binaries not run on novena?
>
> Does that mean there are no old reference binaries known to work on
> Novena?

Not if they are binaries from before Lenny (2009), unless we compile the
kernel with OABI support, which as I understand from Danny, is a bad
idea.  Weird.

Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Update on wip-arm-bootstrap

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

Jan Nieuwenhuizen writes:

Hi Danny!

>> On Thu, 18 Feb 2021 22:52:57 +0100
>> Jan Nieuwenhuizen <janneke@gnu.org> wrote:
>> Since this only affects the syscall interface and since also our
>> ELF headers specify EABI, I would just change the syscalls to EABI:
>> Just put the syscall number into r7 and use svc 0.
>
> Oh, if that's all we should be able to find and fix it?

I took a stab at it and created the attached patch, also on
wip-arm-bootstrap now.  This allows creating simple binaries using
gcc-core-mesboot0 and glibc-mesboot0 that run, like "return 42" and even
"exit (42)".  But something like puts or printf segfaults "Illegal
instruction" in strlen:

--8<---------------cut here---------------start------------->8---
(gdb) disas /r
Dump of assembler code for function strlen:
   0x0000d160 <+0>:	03 10 c0 e3	bic	r1, r0, #3
=> 0x0000d164 <+4>:	04 20 91 e4	ldr	r2, [r1], #4
--8<---------------cut here---------------end--------------->8---

makes no sens to me?  So, trying to build the first glibc-based gcc

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix build -e '(@@ (gnu packages commencement) gcc-mesboot0)'
--8<---------------cut here---------------end--------------->8---

still fails during configure.  Seems like I'm stuck here again,
hopefully you can spot some silly mistake in my patch or have another
helpful insight :-)

Greetings,
Janneke


[-- Attachment #2: 0008-bootstrappable-arm-Support-EABI-system-calls.patch --]
[-- Type: text/x-patch, Size: 10090 bytes --]

From dda9f6d081d68848a1d602c375e0d7a8871e4ae4 Mon Sep 17 00:00:00 2001
From: "Jan (janneke) Nieuwenhuizen" <janneke@gnu.org>
Date: Sat, 20 Feb 2021 15:32:53 +0100
Subject: [PATCH 8/8] bootstrappable: arm: Support EABI system calls.
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8

* sysdeps/unix/sysv/linux/arm/sysdep.h (DO_CALL,INLINE_SYSCALL)
[__ARM_EABI__]: Use eabi calling convention for syscalls.
* sysdeps/arm/dl-machine.h (CLEAR_CACHE): Likewise.
* sysdeps/unix/sysv/linux/arm/brk.c (__brk): Likewise.Likewise.
* sysdeps/unix/sysv/linux/arm/clone.S: Likewise.
* sysdeps/unix/sysv/linux/arm/mmap.S: Likewise.
* sysdeps/unix/sysv/linux/arm/mmap64.S: Likewise.
* sysdeps/unix/sysv/linux/arm/sigrestorer.S: Likewise.
* sysdeps/unix/sysv/linux/arm/socket.S: Likewise.
* sysdeps/unix/sysv/linux/arm/vfork.S: Likewise.
---
 sysdeps/arm/dl-machine.h                  | 14 ++++++
 sysdeps/unix/sysv/linux/arm/brk.c         | 10 +++++
 sysdeps/unix/sysv/linux/arm/clone.S       |  7 +++
 sysdeps/unix/sysv/linux/arm/mmap.S        |  7 +++
 sysdeps/unix/sysv/linux/arm/mmap64.S      |  7 +++
 sysdeps/unix/sysv/linux/arm/sigrestorer.S | 14 ++++++
 sysdeps/unix/sysv/linux/arm/socket.S      |  7 +++
 sysdeps/unix/sysv/linux/arm/sysdep.h      | 54 +++++++++++++++++++++++
 sysdeps/unix/sysv/linux/arm/vfork.S       | 14 ++++++
 9 files changed, 134 insertions(+)

diff --git a/sysdeps/arm/dl-machine.h b/sysdeps/arm/dl-machine.h
index 2d802b7..02c48bd 100644
--- a/sysdeps/arm/dl-machine.h
+++ b/sysdeps/arm/dl-machine.h
@@ -32,6 +32,19 @@
   && VALID_ELF_OSABI (hdr[EI_OSABI]) \
   && VALID_ELF_ABIVERSION (hdr[EI_ABIVERSION])
 
+#if __ARM_EABI__
+#define CLEAR_CACHE(BEG,END)						\
+{									\
+  register unsigned long _beg __asm ("a1") = (unsigned long)(BEG);	\
+  register unsigned long _end __asm ("a2") = (unsigned long)(END);	\
+  register unsigned long _flg __asm ("a3") = 0;				\
+  __asm __volatile ("mov r7, 0x9f0002\n\t"				\
+		    "swi 0x0		@ sys_cacheflush"		\
+		    : /* no outputs */					\
+		    : /* no inputs */					\
+		    : "a1","r7");					\
+}
+#else // !__ARM_EABI__
 #define CLEAR_CACHE(BEG,END)						\
 {									\
   register unsigned long _beg __asm ("a1") = (unsigned long)(BEG);	\
@@ -42,6 +55,7 @@
 		    : /* no inputs */					\
 		    : "a1");						\
 }
+#endif // !__ARM_EABI__
 
 /* Return nonzero iff ELF header is compatible with the running host.  */
 static inline int __attribute__ ((unused))
diff --git a/sysdeps/unix/sysv/linux/arm/brk.c b/sysdeps/unix/sysv/linux/arm/brk.c
index 153d893..d5d0904 100644
--- a/sysdeps/unix/sysv/linux/arm/brk.c
+++ b/sysdeps/unix/sysv/linux/arm/brk.c
@@ -29,12 +29,22 @@ __brk (void *addr)
 {
   void *newbrk;
 
+#if __ARM_EABI__
+  asm ("mov a1, %1\n"	/* save the argment in r0 */
+       "mov r7, %2\n"	/* system call nr in r7 */
+       "swi 0x0\n"	/* do the system call */
+       "mov %0, a1;"	/* keep the return value */
+       : "=r"(newbrk)
+       : "r"(addr), "i" (SYS_ify (brk))
+       : "a1","r7");
+#else // !__ARM_EABI__
   asm ("mov a1, %1\n"	/* save the argment in r0 */
        "swi %2\n"	/* do the system call */
        "mov %0, a1;"	/* keep the return value */
        : "=r"(newbrk) 
        : "r"(addr), "i" (SYS_ify (brk))
        : "a1");
+#endif // !__ARM_EABI__
 
   __curbrk = newbrk;
 
diff --git a/sysdeps/unix/sysv/linux/arm/clone.S b/sysdeps/unix/sysv/linux/arm/clone.S
index c9a1ec2..bd088b4 100644
--- a/sysdeps/unix/sysv/linux/arm/clone.S
+++ b/sysdeps/unix/sysv/linux/arm/clone.S
@@ -44,7 +44,14 @@ ENTRY(__clone)
 	@ get flags
 	mov	r0, r2
 	@ new sp is already in r1
+#if __ARM_EABI__
+	str	r7, [sp, #-4]!
+	mov     r7, #SYS_ify(clone)
+	swi	0x0
+	ldr	r7, [sp], #4
+#else // !__ARM_EABI__
 	swi	SYS_ify(clone)
+#endif // !__ARM_EABI__
 	movs	a1, a1
 	blt	PLTJMP(C_SYMBOL_NAME(__syscall_error))
 	RETINSTR(movne, pc, lr)
diff --git a/sysdeps/unix/sysv/linux/arm/mmap.S b/sysdeps/unix/sysv/linux/arm/mmap.S
index af93c7b..97b4a55 100644
--- a/sysdeps/unix/sysv/linux/arm/mmap.S
+++ b/sysdeps/unix/sysv/linux/arm/mmap.S
@@ -41,7 +41,14 @@ ENTRY (__mmap)
 
 	/* do the syscall */
 	mov	a1, sp
+#if __ARM_EABI__
+	str	r7, [sp, #-4]!
+	mov	r7, #SYS_ify (mmap)
+	swi	0x0
+	ldr	r7, [sp], #4
+#else // !__ARM_EABI__
 	swi	SYS_ify (mmap)
+#endif // !__ARM_EABI__
 
 	/* pop args off the stack. */
 	add	sp, sp, #16
diff --git a/sysdeps/unix/sysv/linux/arm/mmap64.S b/sysdeps/unix/sysv/linux/arm/mmap64.S
index 1f19bf0..8cb40f5 100644
--- a/sysdeps/unix/sysv/linux/arm/mmap64.S
+++ b/sysdeps/unix/sysv/linux/arm/mmap64.S
@@ -36,7 +36,14 @@ ENTRY (__mmap64)
 	movs	ip, ip, lsr $12
 	bne	.Linval			@ check for overflow
 	mov	ip, r0
+#if __ARM_EABI__
+	str	r7, [sp, #-4]!
+	mov	r7, #SYS_ify (mmap2)
+	swi	0x0
+	ldr	r7, [sp], #4
+#else // !__ARM_EABI__
 	swi	SYS_ify (mmap2)
+#endif // !__ARM_EABI__
 	cmn	r0, $4096
 	LOADREGS(ccfd, sp!, {r4, r5, pc})
 	cmn	r0, $ENOSYS
diff --git a/sysdeps/unix/sysv/linux/arm/sigrestorer.S b/sysdeps/unix/sysv/linux/arm/sigrestorer.S
index a4769ca..54aa196 100644
--- a/sysdeps/unix/sysv/linux/arm/sigrestorer.S
+++ b/sysdeps/unix/sysv/linux/arm/sigrestorer.S
@@ -24,12 +24,26 @@
 
 #ifdef __NR_sigreturn
 ENTRY(__default_sa_restorer)
+#if __ARM_EABI__
+	str	r7, [sp, #-4]!
+	mov	r7, #SYS_ify(sigreturn)
+	swi	0x0
+	ldr	r7, [sp], #4
+#else // !__ARM_EABI__
 	swi	SYS_ify(sigreturn)
+#endif // !__ARM_EABI__
 #endif
 
 #ifdef __NR_rt_sigreturn
 
 ENTRY(__default_rt_sa_restorer)
+#if __ARM_EABI__
+	str	r7, [sp, #-4]!
+	mov	r7, #SYS_ify(rt_sigreturn)
+	swi	0x0
+	ldr	r7, [sp], #4
+#else // !__ARM_EABI__
 	swi	SYS_ify(rt_sigreturn)
+#endif // !__ARM_EABI__
 
 #endif
diff --git a/sysdeps/unix/sysv/linux/arm/socket.S b/sysdeps/unix/sysv/linux/arm/socket.S
index c8143a2..f0d6e08 100644
--- a/sysdeps/unix/sysv/linux/arm/socket.S
+++ b/sysdeps/unix/sysv/linux/arm/socket.S
@@ -72,7 +72,14 @@ ENTRY (__socket)
 	mov a1, $P(SOCKOP_,socket)
 	mov a2, sp
 #ifdef __NR_socketcall
+#if __ARM_EABI__
+	str	r7, [sp, #-4]!
+	mov r7, #SYS_ify(socketcall)
+	swi 0x0
+	ldr	r7, [sp], #4
+#else // !__ARM_EABI__
 	swi SYS_ify(socketcall)
+#endif // !__ARM_EABI__
 #endif
 
 	/* Pop args off the stack */
diff --git a/sysdeps/unix/sysv/linux/arm/sysdep.h b/sysdeps/unix/sysv/linux/arm/sysdep.h
index 89ad194..f8e88e4 100644
--- a/sysdeps/unix/sysv/linux/arm/sysdep.h
+++ b/sysdeps/unix/sysv/linux/arm/sysdep.h
@@ -95,10 +95,40 @@
 */
 
 #undef	DO_CALL
+#if __ARM_EABI__
+#define DO_CALL(args, syscall_name)		\
+    DOARGS_##args				\
+    ldr r7, =SYS_ify (syscall_name);		\
+    swi 0x0;					\
+    UNDOARGS_##args
+#else //!__ARM_EABI__
 #define DO_CALL(args, syscall_name)		\
     DOARGS_##args				\
     swi SYS_ify (syscall_name); 		\
     UNDOARGS_##args
+#endif //!__ARM_EABI__
+
+#if __ARM_EABI__
+
+#define DOARGS_0 str r7, [sp, $-4]!;
+#define DOARGS_1 str r7, [sp, $-4]!;
+#define DOARGS_2 str r7, [sp, $-4]!;
+#define DOARGS_3 str r7, [sp, $-4]!;
+#define DOARGS_4 str r7, [sp, $-4]!;
+#define DOARGS_5 mov ip, sp; stmfd sp!, {r4, r7}; ldmia ip, {r4, r7};
+#define DOARGS_6 mov ip, sp; stmfd sp!, {r4, r5, r7}; ldmia ip, {r4, r5, r7};
+#define DOARGS_7 mov ip, sp; stmfd sp!, {r4, r5, r6, r7}; ldmia ip, {r4, r5, r6, r7};
+
+#define UNDOARGS_0 ldr r7, [sp], $4;
+#define UNDOARGS_1 ldr r7, [sp], $4;
+#define UNDOARGS_2 ldr r7, [sp], $4;
+#define UNDOARGS_3 ldr r7, [sp], $4;
+#define UNDOARGS_4 ldr r7, [sp], $4;
+#define UNDOARGS_5 ldmfd sp!, {r4, r7};
+#define UNDOARGS_6 ldmfd sp!, {r4, r5, r7};
+#define UNDOARGS_7 ldmfd sp!, {r4, r5, r6, r7};
+
+#else //!__ARM_EABI__
 
 #define DOARGS_0 /* nothing */
 #define DOARGS_1 /* nothing */
@@ -118,11 +148,34 @@
 #define UNDOARGS_6 ldmfd sp!, {r4, r5};
 #define UNDOARGS_7 ldmfd sp!, {r4, r5, r6};
 
+#endif //!__ARM_EABI__
+
 #else /* not __ASSEMBLER__ */
 
 /* Define a macro which expands into the inline wrapper code for a system
    call.  */
 #undef INLINE_SYSCALL
+
+#if __ARM_EABI__
+#define INLINE_SYSCALL(name, nr, args...)			\
+  ({ unsigned int _sys_result;					\
+     {								\
+       register int _a1 asm ("a1");				\
+       LOAD_ARGS_##nr (args)					\
+       asm volatile ("mov 	r7, %1\n\t"			\
+		     "swi	$0	@ syscall " #name	\
+		     : "=r" (_a1)				\
+		     : "i" (SYS_ify(name)) ASM_ARGS_##nr	\
+		     : "a1", "r7", "memory");                   \
+       _sys_result = _a1;					\
+     }								\
+     if (_sys_result >= (unsigned int) -4095)			\
+       {							\
+	 __set_errno (-_sys_result);				\
+	 _sys_result = (unsigned int) -1;			\
+       }							\
+     (int) _sys_result; })
+#else //!__ARM_EABI__
 #define INLINE_SYSCALL(name, nr, args...)			\
   ({ unsigned int _sys_result;					\
      {								\
@@ -140,6 +193,7 @@
 	 _sys_result = (unsigned int) -1;			\
        }							\
      (int) _sys_result; })
+#endif //!__ARM_EABI__
 
 #define LOAD_ARGS_0()
 #define ASM_ARGS_0
diff --git a/sysdeps/unix/sysv/linux/arm/vfork.S b/sysdeps/unix/sysv/linux/arm/vfork.S
index b10117e..ba2259d 100644
--- a/sysdeps/unix/sysv/linux/arm/vfork.S
+++ b/sysdeps/unix/sysv/linux/arm/vfork.S
@@ -29,7 +29,14 @@
 ENTRY (__vfork)
 
 #ifdef __NR_vfork
+#if __ARM_EABI__
+	str	r7, [sp, #-4]!
+	mov	r7, #__NR_vfork
+	swi	0x0
+	ldr	r7, [sp], #4
+#else // !__ARM_EABI__
 	swi	__NR_vfork
+#endif // !__ARM_EABI__
 	cmn	a1, #4096
 	RETINSTR(movcc, pc, lr)
 
@@ -40,7 +47,14 @@ ENTRY (__vfork)
 #endif
 
 	/* If we don't have vfork, fork is close enough.  */
+#if __ARM_EABI__
+	str	r7, [sp, #-4]!
+	mov	r7, #__NR_fork
+	swi	0x0
+	ldr	r7, [sp], #4
+#else // !__ARM_EABI__
 	swi	__NR_fork
+#endif // !__ARM_EABI__
 	cmn	a1, #4096
 	RETINSTR(movcc, pc, lr)
     	b	PLTJMP(C_SYMBOL_NAME(__syscall_error))
-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com


[-- Attachment #3: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Update on wip-arm-bootstrap

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1208 bytes --]

> > Does that mean there are no old reference binaries known to work on
> > Novena?  
> 
> Not if they are binaries from before Lenny (2009), unless we compile the
> kernel with OABI support, which as I understand from Danny, is a bad
> idea.  Weird.

It backdoors https://en.wikipedia.org/wiki/Seccomp if you enable the kernel
option CONFIG_OABI_COMPAT (it also breaks syscall auditing).

And it's very easy to just change the syscall facility in glibc to emit EABI
syscalls (load one new register with constant, and the swi number is different).

Also, binaries compiled with gcc already [say they] are EABI (while using
OABI syscalls :P).

So I think just using EABI syscalls is easier than figuring out how to get
gcc to emit OABI executables--which otherwise are an obsolete format.

GNU Mes for ARM does not use OABI for its own system calls either.

I mean as a test we could totally enable that kernel option and see
whether the bootstrap works.  But it's of limited long-term use.  If we want
regular people to be able to build our bootstrap binaries, we probably
shouldn't rely on them enabling obscure kernel options where the kernel docs
say you shouldn't enable those.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Update on wip-arm-bootstrap

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]

Hi Janneke,

Nice!

Patch review:

>+#if __ARM_EABI__
>+#define CLEAR_CACHE(BEG,END)						\
>+{									\
>+  register unsigned long _beg __asm ("a1") = (unsigned long)(BEG);	\
>+  register unsigned long _end __asm ("a2") = (unsigned long)(END);	\
>+  register unsigned long _flg __asm ("a3") = 0;				\
>+  __asm __volatile ("mov r7, 0x9f0002\n\t"				\

should be mov r7, 0xf0002 in EABI.
(SWI #(0x900000 + syscallnr) is for OABI syscalls only)

>* sysdeps/unix/sysv/linux/arm/brk.c (__brk): Likewise.Likewise.

Duplicate "Likewise".

Otherwise LGTM!

>But something like puts or printf segfaults "Illegal
>instruction" in strlen:

>--8<---------------cut here---------------start------------->8---
>(gdb) disas /r
>Dump of assembler code for function strlen:
>   0x0000d160 <+0>:	03 10 c0 e3	bic	r1, r0, #3
>=> 0x0000d164 <+4>:	04 20 91 e4	ldr	r2, [r1], #4  
>--8<---------------cut here---------------end--------------->8---

Hmm... looks good to me?  Weird.

I'm building it right now and will have a look.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Update on wip-arm-bootstrap

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

Hi Janneke,

I get

gcc-mesboot.sh: line 20:  9438 Illegal instruction     (core dumped) ./a.out

│   0x276b8 <__writev+32>   svc     0x00000014                                 │
│  >0x276bc <__writev+36>   mov     r4, r0                                     │

The actual problem is at the first line above--the svc with argument != 0.

#0  0x000276bc in __writev (fd=2, vector=0xbebc54c0, count=10)
    at ../sysdeps/unix/sysv/linux/writev.c:51

glibc 2.2.5:sysdeps/unix/sysv/linux/writev.c:51

>  bytes_written = INLINE_SYSCALL (writev, 3, fd, CHECK_N (vector, count), count);

Err.... hmm?!

You *are* patching INLINE_SYSCALL body--so I'd guess that __ARM_EABI__ for
some reason is not being defined.

I'd suggest to #error (or at least #warning) if !__ARM_EABI__.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Update on wip-arm-bootstrap

[Jan Nieuwenhuizen]

Danny Milosavljevic writes:

> Hi Janneke,
>
> I get
>
> gcc-mesboot.sh: line 20:  9438 Illegal instruction     (core dumped) ./a.out
>
> │   0x276b8 <__writev+32>   svc     0x00000014                                 │
> │  >0x276bc <__writev+36>   mov     r4, r0                                     │
>
> The actual problem is at the first line above--the svc with argument != 0.
>
> #0  0x000276bc in __writev (fd=2, vector=0xbebc54c0, count=10)
>     at ../sysdeps/unix/sysv/linux/writev.c:51
>
> glibc 2.2.5:sysdeps/unix/sysv/linux/writev.c:51
>
>>  bytes_written = INLINE_SYSCALL (writev, 3, fd, CHECK_N (vector, count), count);
>
> Err.... hmm?!

Hmm, I'm not seeing this (I did see something like this before).  You
*are* on

    e44b6b7eed squash! commencement: binutils-mesboot0: Support ARM.

right?

> You *are* patching INLINE_SYSCALL body--

Yes

> so I'd guess that __ARM_EABI__ for some reason is not being defined.
>
> I'd suggest to #error (or at least #warning) if !__ARM_EABI__.

That's a good suggestion; I'm setting it in the glibc-mesboot0 recipe

    ,(if (equal? triplet "arm-unknown-linux-gnu")
         " -D __ARM_EABI__=1"
         "")))

and it "looks like" that's being used in every compilation...

Weird,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Update on wip-arm-bootstrap

[Jan Nieuwenhuizen]

Danny Milosavljevic writes:

Hello Danny,

> Nice!
>
> Patch review:
>
>>+#if __ARM_EABI__
>>+#define CLEAR_CACHE(BEG,END)						\
>>+{									\
>>+  register unsigned long _beg __asm ("a1") = (unsigned long)(BEG);	\
>>+  register unsigned long _end __asm ("a2") = (unsigned long)(END);	\
>>+  register unsigned long _flg __asm ("a3") = 0;				\
>>+  __asm __volatile ("mov r7, 0x9f0002\n\t"				\
>
> should be mov r7, 0xf0002 in EABI.
> (SWI #(0x900000 + syscallnr) is for OABI syscalls only)

Ok, great; that could help!

>>* sysdeps/unix/sysv/linux/arm/brk.c (__brk): Likewise.Likewise.
>
> Duplicate "Likewise".
>
> Otherwise LGTM!

Thanks.  Testing a fixup now (currently only on my branch @gitlab).

>>But something like puts or printf segfaults "Illegal
>>instruction" in strlen:
>
>>--8<---------------cut here---------------start------------->8---
>>(gdb) disas /r
>>Dump of assembler code for function strlen:
>>   0x0000d160 <+0>:	03 10 c0 e3	bic	r1, r0, #3
>>=> 0x0000d164 <+4>:	04 20 91 e4	ldr	r2, [r1], #4  
>>--8<---------------cut here---------------end--------------->8---
>
> Hmm... looks good to me?  Weird.

Yes...

> I'm building it right now and will have a look.

Great, thanks!
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: Update on wip-arm-bootstrap

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 3016 bytes --]

> Hmm, I'm not seeing this (I did see something like this before).  You
> *are* on
> 
>     e44b6b7eed squash! commencement: binutils-mesboot0: Support ARM.
> 
> right?

Yes, but your gcc-mesboot.sh hardcoded a specific (older) path.  Sorry.

With updated gcc-mesboot.sh I get:

Program terminated with signal SIGSEGV, Segmentation fault.
(gdb) bt
#0  _IO_setb (f=0x4, b=0x4b <error: Cannot access memory at address 0x4b>, eb=0x4c <error: Cannot access memory at address 0x4c>, a=0) at genops.c:387
#1  0x00008f1c in _IO_doallocbuf (fp=0x4) at genops.c:406
#2  0x0002409c in _IO_new_file_overflow (f=0x6cde0 <_IO_2_1_stdout_>, ch=0) at fileops.c:576
#3  0x000249b0 in _IO_new_file_xsputn (f=0x6cde0 <_IO_2_1_stdout_>, data=0xbecfd313, n=1) at fileops.c:962
#4  0x0001a09c in _IO_vfprintf (s=0x6cde0 <_IO_2_1_stdout_>, format=0xbecfd313 "2\224\323Ͼ\001", 
    ap=<error reading variable: Cannot access memory at address 0xbecfce5c>) at vfprintf.c:1523
#5  0x00008828 in printf (format=0x525f0 "%d\n") at printf.c:33
#6  0x00008214 in main (argc=<error reading variable: Cannot access memory at address 0xbecfd364>, 
    argv=<error reading variable: Cannot access memory at address 0xbecfd360>) at test.c:6

Looks like a NULL pointer for a FILE* ?  Or is gdb just weird?

glibc-2.2.5/libio/genops.c:

void
_IO_setb (f, b, eb, a)
     _IO_FILE *f;
      char *b;
     char *eb;
     int a;
{
  if (f->_IO_buf_base && !(f->_flags & _IO_USER_BUF)) <---------------- here
    FREE_BUF (f->_IO_buf_base, _IO_blen (f));
  f->_IO_buf_base = b;
  f->_IO_buf_end = eb;
  if (a)
    f->_flags &= ~_IO_USER_BUF;
  else
    f->_flags |= _IO_USER_BUF;
}

int
_IO_new_file_overflow (f, ch)
      _IO_FILE *f;
      int ch;
{
  if (f->_flags & _IO_NO_WRITES) /* SET ERROR */
    {
      f->_flags |= _IO_ERR_SEEN;
      __set_errno (EBADF);
      return EOF;
    }
  /* If currently reading or no buffer allocated. */
  if ((f->_flags & _IO_CURRENTLY_PUTTING) == 0 || f->_IO_write_base == 0)
    {
      /* Allocate a buffer if needed. */
      if (f->_IO_write_base == 0)
        {
          _IO_doallocbuf (f); <------------- here
          _IO_setg (f, f->_IO_buf_base, f->_IO_buf_base, f->_IO_buf_base);
        }

Maybe add some debugging statements there?  But printf is broken, so maybe
use write(2, "foo\n", 4) and similar.

(I'm not sure about just defining -D__ARM_EABI__=1 as you are doing--after all,
it entails further stack alignment requirements.  Maybe try


+++ glibc-2.2.5/sysdeps/unix/sysv/linux/arm/sysdep.h    2021-02-22 00:44:43.300906460 +0100
@@ -29,8 +29,8 @@
    of the kernel.  But these symbols do not follow the SYS_* syntax
    so we have to redefine the `SYS_ify' macro here.  */
 #undef SYS_ify
-#define SWI_BASE  (0x900000)
-#define SYS_ify(syscall_name)  (__NR_##syscall_name)
+#define SWI_BASE  (0x000000)
+#define SYS_ify(syscall_name)  (__NR_##syscall_name & 0x8fffff)


as a more minimal patch?  Not sure...)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Update on wip-arm-bootstrap

[Danny Milosavljevic]

[-- Attachment #1: Type: text/plain, Size: 86 bytes --]


Oops, I meant

> +#define SYS_ify(syscall_name)  (__NR_##syscall_name & 0xfffff)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]