Everything segfaults

Everything segfaults

[Paul van der Walt]

Hello Guix,

It's been a while, so i apologise for asking what are probably dumb
questions.

Quite a while ago i ran into this issue where (it might have been after
a system upgrade including libc and such, i'm using git Guix on Arch
Linux) all binaries segfault.

I didn't really have time to look at it then, and now i'm trying to get
back on the Guix-horse.  I've done `guix package -i ..` for a few
packages, some of which have and some of which haven't been updated in
the meantime, but the problem persists.  Should i blow away the entire
/gnu/store and reinstall Guix from scratch, or can something be
debugged/salvaged?  My debugging-fu is weak, so i might need pointers.

Hope this isn't too much PEBKAC for one afternoon!

Cheers,
p.

Re: Everything segfaults

[Roel Janssen]

Paul van der Walt writes:

> Hello Guix,
>
> It's been a while, so i apologise for asking what are probably dumb
> questions.
>
> Quite a while ago i ran into this issue where (it might have been after
> a system upgrade including libc and such, i'm using git Guix on Arch
> Linux) all binaries segfault.

From what kernel version, to what kernel version have you moved?

> I didn't really have time to look at it then, and now i'm trying to get
> back on the Guix-horse.  I've done `guix package -i ..` for a few
> packages, some of which have and some of which haven't been updated in
> the meantime, but the problem persists.  Should i blow away the entire
> /gnu/store and reinstall Guix from scratch, or can something be
> debugged/salvaged?  My debugging-fu is weak, so i might need pointers.

Could you try running a program from your Guix profile using strace?
So, running it as normal, but then prepend "strace", like:
  strace git log

It may help figuring out what goes wrong.

Maybe this question belongs to help-guix@gnu.org instead.

Kind regards,
Roel Janssen

Re: Everything segfaults

[Paul van der Walt]

On 2016-06-09 at 17:11, quoth Roel Janssen:
> Maybe this question belongs to help-guix@gnu.org instead.

Ah!  I'll use that forum instead, thanks for the pointer.  Sorry for the
spam.

p.

Everything segfaults

[Paul van der Walt]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

Hello Guix (sorry for the cross-post, and thanks Roel for the tip),

It's been a while, so i apologise for asking what are probably dumb
questions.

Quite a while ago i ran into this issue where (it might have been after
a system upgrade including libc and such, i'm using git Guix on Arch
Linux) all binaries segfault.  This is quite a while ago, so it would've
definitely been an older kernel version then, but i can't say which
one.  We're talking like 4 months ago.

I didn't really have time to look at it then, and now i'm trying to get
back on the Guix-horse.  I've done `guix package -i ..` for a few
packages, some of which have and some of which haven't been updated in
the meantime, but the problem persists.  Should i blow away the entire
/gnu/store and reinstall Guix from scratch, or can something be
debugged/salvaged?  My debugging-fu is weak, so i might need pointers.

I've taken Roel's suggestion and attached the output of `strace less`
which is one of the binaries that segfaults.

Hope this isn't too much PEBKAC for one afternoon!

Cheers,
p.

[-- Attachment #2: strace-less.log --]
[-- Type: text/plain, Size: 3975 bytes --]

execve("/home/paul/.guix-profile/bin/less", ["less"], [/* 55 vars */]) = 0
brk(NULL)                               = 0x1319000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f99bd00c000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/home/paul/GNUstep/Library/Libraries/tls/x86_64/libncursesw.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/paul/GNUstep/Library/Libraries/tls/x86_64", 0x7ffd3128db80) = -1 ENOENT (No such file or directory)
open("/home/paul/GNUstep/Library/Libraries/tls/libncursesw.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/paul/GNUstep/Library/Libraries/tls", 0x7ffd3128db80) = -1 ENOENT (No such file or directory)
open("/home/paul/GNUstep/Library/Libraries/x86_64/libncursesw.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/paul/GNUstep/Library/Libraries/x86_64", 0x7ffd3128db80) = -1 ENOENT (No such file or directory)
open("/home/paul/GNUstep/Library/Libraries/libncursesw.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/home/paul/GNUstep/Library/Libraries", 0x7ffd3128db80) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/x86_64/libncursesw.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls/x86_64", 0x7ffd3128db80) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/libncursesw.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls", 0x7ffd3128db80)    = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64/libncursesw.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64", 0x7ffd3128db80) = -1 ENOENT (No such file or directory)
open("/usr/lib/libncursesw.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260u\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=444680, ...}) = 0
mmap(NULL, 2542224, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f99bcb81000
mprotect(0x7f99bcbe8000, 2097152, PROT_NONE) = 0
mmap(0x7f99bcde8000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x67000) = 0x7f99bcde8000
close(3)                                = 0
open("/usr/lib/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p*\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=721704, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f99bd00b000
mmap(NULL, 2185552, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f99bc96b000
mprotect(0x7f99bc981000, 2093056, PROT_NONE) = 0
mmap(0x7f99bcb80000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f99bcb80000
close(3)                                = 0
open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\10\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1960968, ...}) = 0
mmap(NULL, 3803440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f99bc5ca000
mprotect(0x7f99bc761000, 2097152, PROT_NONE) = 0
mmap(0x7f99bc961000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x197000) = 0x7f99bc961000
mmap(0x7f99bc967000, 14640, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f99bc967000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f99bd00a000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f99bd009000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f99bd008000
arch_prctl(ARCH_SET_FS, 0x7f99bd009700) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
+++ killed by SIGSEGV (core dumped) +++
zsh: segmentation fault (core dumped)  strace less &> strace-less.log

Re: Everything segfaults

[Andreas Enge]

Hello,

the following looks very strange:

On Thu, Jun 09, 2016 at 05:20:38PM +0200, Paul van der Walt wrote:
> execve("/home/paul/.guix-profile/bin/less", ["less"], [/* 55 vars */]) = 0
> ...
> open("/usr/lib/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
> read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p*\0\0\0\0\0\0"..., 832) = 832
> ...
> open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
> read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\10\2\0\0\0\0\0"..., 832) = 832
> fstat(3, {st_mode=S_IFREG|0755, st_size=1960968, ...}) = 0

Your Guix less seems to open libraries from your Arch system, which are
incompatible. Did you set LD_LIBRARY_PATH?
Maybe you could try "ldd /home/paul/.guix-profile/bin/less".

Andreas

Re: Everything segfaults

[Paul van der Walt]

Hey Andreas!

Sorry for the delay!  Busy busy.

On 2016-06-09 at 19:28, quoth Andreas Enge:
> the following looks very strange:
>
>> [very strange output]
>
> Your Guix less seems to open libraries from your Arch system, which are
> incompatible. Did you set LD_LIBRARY_PATH?
> Maybe you could try "ldd /home/paul/.guix-profile/bin/less".

If i open a terminal, then

$ echo $LD_LIBRARY_PATH 
/home/paul/GNUstep/Library/Libraries:/usr/lib

Here's some output:

Case A:
$ ldd .guix-profile/bin/less                                                                 
	linux-vdso.so.1 (0x00007ffe775d7000)
	libncursesw.so.6 => /usr/lib/libncursesw.so.6 (0x00007f8401bc3000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f84019ad000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f840160c000)
	/gnu/store/8m00x5x8ykmar27s9248cmhnkdb2n54a-glibc-2.22/lib/ld-linux-x86-64.so.2 (0x00007f8401e30000)

Case B:
$ LD_LIBRARY_PATH="~/.guix-profile/lib" ldd .guix-profile/bin/less
	linux-vdso.so.1 (0x00007ffe7cedb000)
	libncursesw.so.6 => /gnu/store/xadbq6k36aphlx0haxxzym3xmd5r1rp8-ncurses-6.0/lib/libncursesw.so.6 (0x00007fbc6dbdb000)
	libgcc_s.so.1 => /gnu/store/v39bh3ln3ncnzhyw0kd12d46kww9747v-gcc-4.9.3-lib/lib/libgcc_s.so.1 (0x00007fbc6d9c5000)
	libc.so.6 => /gnu/store/8m00x5x8ykmar27s9248cmhnkdb2n54a-glibc-2.22/lib/libc.so.6 (0x00007fbc6d620000)
	/gnu/store/8m00x5x8ykmar27s9248cmhnkdb2n54a-glibc-2.22/lib/ld-linux-x86-64.so.2 (0x00007fbc6de4b000)

(when i try setting LD_..="~/.guix-profile/lib:$LD_..", that is, i only
prepend the Guix lib path, then i get the same output from ldd as in
case A -- is that what i expect?)

So there's definitely something funny going on with LD_LIBRARY_PATH
indeed.  But here's another stupid question: surely if i want arbitrary
Guix binaries to work on my system i'd apparently (?) have to put
something like

    export LD_LIBRARY_PATH="$HOME/.guix-profile/lib"

in my .xinitrc, but that would surely break other binaries (those
installed by my usual package manager)?  And if i put

    export LD_LIBRARY_PATH="$HOME/.guix-profile/lib:$LD_LIBRARY_PATH"

in my .xinitrc, then as in case A, surely i won't have fixed my issue?

Sorry to be so thick about this!

Kind regards,
p.

Re: Everything segfaults

[Ludovic Courtès]

Hello!

Paul van der Walt <paul@denknerd.org> skribis:

> On 2016-06-09 at 19:28, quoth Andreas Enge:
>> the following looks very strange:
>>
>>> [very strange output]
>>
>> Your Guix less seems to open libraries from your Arch system, which are
>> incompatible. Did you set LD_LIBRARY_PATH?
>> Maybe you could try "ldd /home/paul/.guix-profile/bin/less".
>
> If i open a terminal, then
>
> $ echo $LD_LIBRARY_PATH 
> /home/paul/GNUstep/Library/Libraries:/usr/lib

Just unset it, it shouldn’t be needed, and it’s evil!

Binaries of the host distro will undoubtedly look for libraries in
/usr/lib even when LD_LIBRARY_PATH is unset.  As for GNUstep, I’m not
sure, but most likely everything will be fine.

HTH!  :-)

Ludo’.

Re: Everything segfaults

[Paul van der Walt]

On 2016-06-14 at 23:46 CEST+0200, quoth Ludovic Courtès:
>> $ echo $LD_LIBRARY_PATH 
>> /home/paul/GNUstep/Library/Libraries:/usr/lib
>
> Just unset it, it shouldn’t be needed, and it’s evil!
>
> Binaries of the host distro will undoubtedly look for libraries in
> /usr/lib even when LD_LIBRARY_PATH is unset.  As for GNUstep, I’m not
> sure, but most likely everything will be fine.

Okay, that did it.  Eventually it turned out that a package
`gnustep-make` had created a file in my distro's /etc/profile.d/ folder
which was setting this variable.  Removing the package (a dependency of
some software i don't use any more) solved the issue.  Thanks!

Now i just need to figure out why fonts in Emacs (installed from Guix)
seem whack... :/  But that's for tomorrow.

Bye now, and thanks!
p.

Re: Everything segfaults

[Ludovic Courtès]

Howdy!

Paul van der Walt <paul@denknerd.org> skribis:

> Now i just need to figure out why fonts in Emacs (installed from Guix)
> seem whack... :/  But that's for tomorrow.

Hopefully this will help:

  https://www.gnu.org/software/guix/manual/html_node/Application-Setup.html#X11-Fonts

Cheers,
Ludo’.