Re: chroot/aarch64 issues

[Efraim Flashner <efraim@flashner.co.il>]

[-- Attachment #1.1: Type: text/plain, Size: 3861 bytes --]

On Tue, Jul 05, 2016 at 10:45:51AM +0200, Ludovic Courtès wrote:
> Efraim Flashner <efraim@flashner.co.il> skribis:
> 
> > On Mon, Jul 04, 2016 at 03:37:12PM +0200, Ludovic Courtès wrote:
> >> Hi!
> >> 
> >> Efraim Flashner <efraim@flashner.co.il> skribis:
> >> 
> >> > On the side I'm working on the aarch64 port, and I've come across a
> >> 
> >> Woohoo!
> >> 
> >> > snag. I've been unable to actually get anything to build. When I start
> >> > the daemon as:
> >> > `sudo ./pre-inst-env guix-daemon --build-users-group=guixbuild',
> >> > when I attempt to build hello I get back the error:
> >> > guix build: error: build failed: cloning builder process: Invalid argument
> >> 
> >> Could you strace the daemon itself?
> >> 
> >>   sudo ./pre-inst-env strace -f -o log guix-daemon \
> >>            --build-users-group=guixbuild
> >> 
> >> I presume EINVAL is returned by clone(2), but it would be good to see
> >> that syscall precisely.
> >
> > attached
> 
> The culprit is:
> 
> 4871  clone(child_stack=0x7fd850d338, flags=CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|CLONE_NEWPID|CLONE_NEWNET|SIGCHLD) = -1 EINVAL (Invalid argument)
> 
> >> What Linux version is this?
> >
> > $ uname -a
> > Linux odroid-jessie64 3.14.65+ #2 SMP PREEMPT Sun Jul 3 18:29:45
> > UTC 2016 aarch64 GNU/Linux
> 
> According to clone(2), this version supports all of the above flags.
> 
> Also, the kernel config you posted has all the namespace options
> enabled:
> 
> --8<---------------cut here---------------start------------->8---
> CONFIG_UTS_NS=y
> CONFIG_IPC_NS=y
> CONFIG_USER_NS=y
> CONFIG_PID_NS=y
> CONFIG_NET_NS=y
> --8<---------------cut here---------------end--------------->8---
> 
> If the kernel config corresponds to this kernel, I don’t see which of
> the other EINVAL reasons given in clone(2) would apply.  Does ‘dmesg’
> show something?
> 
> Could you maybe try a C program that invokes clone(2) and progressively
> remove CLONE_ flags until you find the one that’s causing EINVAL?
> 
> Thanks,
> Ludo’.

I ran the program from here
http://man7.org/tlpi/code/online/dist/procexec/demo_clone.c.html and
from what I could tell it worked. I've attached the log anyway.

grepping for "cloning builder process" led me to nix/libstore/build.cc .
pid = clone(childEntry, stack + sizeof(stack) - 8, flags, this);

the failing clone call:
4871  clone(child_stack=0x7fd850d338, flags=CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|CLONE_NEWPID|CLONE_NEWNET|SIGCHLD) = -1 EINVAL (Invalid argument)

the other clone calls:
4866  clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb24220d0) = 4871
4871  clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb24220d0) = 4872
4872  clone(child_stack=0x7facb0bae0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7facb0c2b0, tls=0x7facb0c8d0, child_tidptr=0x7facb0c2b0) = 4873
4871  clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb24220d0) = 4875

without chroot there's no error since `if (useChroot)' is false.
tidptr is apparently thread id pointer.
I'm going to look more at
http://man7.org/linux/man-pages/man2/set_tid_address.2.html and
http://man7.org/linux/man-pages/man2/clone.2.html in case there's
something specific about aarch64 causing problems.


also, is this something I should look into?
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

[-- Attachment #1.2: clone-log --]
[-- Type: text/plain, Size: 4639 bytes --]

10223 execve("./procexec/demo_clone", ["./procexec/demo_clone"], [/* 19 vars */]) = 0
10223 brk(0)                            = 0x413000
10223 faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
10223 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82e76000
10223 faccessat(AT_FDCWD, "/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
10223 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
10223 fstat(3, {st_mode=S_IFREG|0644, st_size=22634, ...}) = 0
10223 mmap(NULL, 22634, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f82e70000
10223 close(3)                          = 0
10223 faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
10223 openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
10223 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\340R\0\0\0\0\0\0"..., 832) = 832
10223 fstat(3, {st_mode=S_IFREG|0644, st_size=588696, ...}) = 0
10223 mmap(NULL, 651944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f82daf000
10223 mprotect(0x7f82e3e000, 61440, PROT_NONE) = 0
10223 mmap(0x7f82e4d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8e000) = 0x7f82e4d000
10223 close(3)                          = 0
10223 faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
10223 openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
10223 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\250\16\2\0\0\0\0\0"..., 832) = 832
10223 fstat(3, {st_mode=S_IFREG|0755, st_size=1287872, ...}) = 0
10223 mmap(NULL, 1360760, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f82c62000
10223 mprotect(0x7f82d95000, 65536, PROT_NONE) = 0
10223 mmap(0x7f82da5000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x133000) = 0x7f82da5000
10223 mmap(0x7f82dab000, 13176, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f82dab000
10223 close(3)                          = 0
10223 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82e6f000
10223 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82e6e000
10223 mprotect(0x7f82da5000, 16384, PROT_READ) = 0
10223 mprotect(0x7f82e4d000, 4096, PROT_READ) = 0
10223 mprotect(0x7f82e7a000, 4096, PROT_READ) = 0
10223 munmap(0x7f82e70000, 22634)       = 0
10223 getpid()                          = 10223
10223 getppid()                         = 10218
10223 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 4), ...}) = 0
10223 mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f82c52000
10223 write(1, "Parent: PID=10223 PPID=10218\n", 29) = 29
10223 umask(02)                         = 022
10223 openat(AT_FDCWD, "/dev/null", O_RDWR) = 3
10223 rt_sigaction(SIGTERM, {SIG_IGN, [TERM], SA_RESTART}, {SIG_DFL, [], 0}, 8) = 0
10223 brk(0)                            = 0x413000
10223 brk(0x444000)                     = 0x444000
10223 rt_sigaction(SIGUSR1, {0x400f90, [], SA_RESTART}, NULL, 8) = 0
10223 clone( <unfinished ...>
10224 getpid( <unfinished ...>
10223 <... clone resumed> child_stack=0x423000, flags=SIGUSR1) = 10224
10224 <... getpid resumed> )            = 10224
10224 getppid( <unfinished ...>
10223 wait4(-1,  <unfinished ...>
10224 <... getppid resumed> )           = 10223
10224 write(1, "Child:  PID=10224 PPID=10223\n", 29) = 29
10224 umask(020)                        = 02
10224 close(3)                          = 0
10224 rt_sigaction(SIGTERM, {SIG_DFL, [TERM], SA_RESTART}, {SIG_IGN, [TERM], SA_RESTART}, 8) = 0
10224 exit_group(22)                    = ?
10224 +++ exited with 22 +++
10223 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 22}], __WCLONE, NULL) = 10224
10223 --- SIGUSR1 {si_signo=SIGUSR1, si_code=0x1, si_pid=10224, si_uid=1000, si_value={int=22, ptr=0x16}} ---
10223 write(1, "Caught signal 10 (User defined s"..., 41) = 41
10223 rt_sigreturn()                    = 10224
10223 write(1, "    Child PID=10224\n", 20) = 20
10223 write(1, "    Status: child exited, status"..., 36) = 36
10223 write(1, "Parent - checking process attrib"..., 38) = 38
10223 umask(0)                          = 02
10223 write(1, "    umask has not changed\n", 26) = 26
10223 write(3, "Hello world\n", 12)     = 12
10223 write(1, "    write() on file descriptor 3"..., 43) = 43
10223 rt_sigaction(SIGTERM, NULL, {SIG_IGN, [TERM], SA_RESTART}, 8) = 0
10223 write(1, "    signal disposition has not c"..., 39) = 39
10223 exit_group(0)                     = ?
10223 +++ exited with 0 +++

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]