bug#53506: guix-packages-base.drv leads to segfault on i686-linux

bug#53506: guix-packages-base.drv leads to segfault on i686-linux

[Ludovic Courtès]

Hello,

This command fails:

  guix pull -s i686-linux \
    --commit=13b905bf28ec6309043bd61c5a92744b13352021 \
    -p /tmp/test

‘guix-packages-base.drv’ fails to build due to a Guile segfault (!):

--8<---------------cut here---------------start------------->8---
[653/656] compiling...	 99.1% of 328 files[654/656] compiling...	 99.4% of 328 files[655/656] compiling...	 99.7% of 328 filesGC Warning: Failed to expand heap by 8388608 bytes
GC Warning: Failed to expand heap by 8388608 bytes
GC Warning: Failed to expand heap by 8388608 bytes

[...]

GC Warning: Failed to expand heap by 8388608 bytes
GC Warning: Failed to expand heap by 8388608 bytes
builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)
@ build-failed /gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv - 1 builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)
--8<---------------cut here---------------end--------------->8---

I wonder when it started failing, but it may have been pre-core-updates
merge.

Ricardo, do you have a known-good commit?

<https://ci.guix.gnu.org/jobset/guix?border-high=53879> suggests it
passes 25% of the time.

Thanks,
Ludo’.

bug#53506: guix-packages-base.drv leads to segfault on i686-linux

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> ‘guix-packages-base.drv’ fails to build due to a Guile segfault (!):
>
> [653/656] compiling...	 99.1% of 328 files[654/656] compiling...	 99.4% of 328 files[655/656] compiling...	 99.7% of 328 filesGC Warning: Failed to expand heap by 8388608 bytes
> GC Warning: Failed to expand heap by 8388608 bytes
> GC Warning: Failed to expand heap by 8388608 bytes
>
> [...]
>
> GC Warning: Failed to expand heap by 8388608 bytes
> GC Warning: Failed to expand heap by 8388608 bytes
> builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)
> @ build-failed /gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv - 1 builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)

On closer inspection, this is caused by OOM, with Guile peaking at 2.8G
resident (!) at that point, more than on x86_64.

I’m quite sure this is because the compiler resorts to bignums more than
on x86_64 (fixnums are smaller), thereby consuming more heap.

Splitting creates-io.scm into two files might work as a temporary
workaround since the compiler creates a number of labels (integers)
roughly proportional to the number of lines in the file:

--8<---------------cut here---------------start------------->8---
$ wc -l gnu/packages/*.scm|sort -k1 -n |tail
   13977 gnu/packages/java.scm
   15275 gnu/packages/bioconductor.scm
   15929 gnu/packages/bioinformatics.scm
   16086 gnu/packages/haskell-xyz.scm
   20378 gnu/packages/lisp-xyz.scm
   28770 gnu/packages/python-xyz.scm
   29960 gnu/packages/emacs-xyz.scm
   32071 gnu/packages/cran.scm
   70442 gnu/packages/crates-io.scm
  690662 totalo
--8<---------------cut here---------------end--------------->8---

Background:

  https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00035.html
  https://lists.gnu.org/archive/html/guile-devel/2020-02/msg00023.html
  https://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile

Ludo’.

bug#53506: guix-packages-base.drv leads to segfault on i686-linux

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> ‘guix-packages-base.drv’ fails to build due to a Guile segfault (!):
>>
>> [653/656] compiling...	 99.1% of 328 files[654/656] compiling...	 99.4% of 328 files[655/656] compiling...	 99.7% of 328 filesGC Warning: Failed to expand heap by 8388608 bytes
>> GC Warning: Failed to expand heap by 8388608 bytes
>> GC Warning: Failed to expand heap by 8388608 bytes
>>
>> [...]
>>
>> GC Warning: Failed to expand heap by 8388608 bytes
>> GC Warning: Failed to expand heap by 8388608 bytes
>> builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)
>> @ build-failed /gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv - 1 builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)
>
> On closer inspection, this is caused by OOM, with Guile peaking at 2.8G
> resident (!) at that point, more than on x86_64.

An update: with changes made in Guile “main” over the last couple of
weeks, memory consumption is 20% lower and compilation is 20% faster
compared to 3.0.7 (on x86_64):

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env time -f '%U seconds\n%M KiB' guile -c '(use-modules (system base compile)) (compile-file "gnu/packages/crates-io.scm" #:optimization-level 1)'
53.84 seconds
795972 KiB
$ ./pre-inst-env time -f '%U seconds\n%M KiB' /data/src/guile-3.0/meta/guile -c '(use-modules (system base compile)) (compile-file "gnu/packages/crates-io.scm" #:optimization-level 1 #:opts (list #:inlinable-exports? #f #:resolve-free-vars? #f))'
43.00 seconds
618724 KiB
$
$ guile --version
guile (GNU Guile) 3.0.7
Copyright (C) 2021 Free Software Foundation, Inc.

License LGPLv3+: GNU LGPL 3 or later <http://gnu.org/licenses/lgpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ (cd /data/src/guile-3.0; git log | head -1)
commit 2aed3c117c2d667ecca1e38a016f2cb4b524ab50
--8<---------------cut here---------------end--------------->8---

To be continued…

Ludo’.

bug#53506: guix-packages-base.drv leads to segfault on i686-linux

[Maxim Cournoyer]

Hi Ludovic!

Ludovic Courtès <ludo@gnu.org> writes:

> Hi!
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> skribis:
>>
>>> ‘guix-packages-base.drv’ fails to build due to a Guile segfault (!):
>>>
>>> [653/656] compiling...  99.1% of 328 files[654/656] compiling...
>>> 99.4% of 328 files[655/656] compiling...  99.7% of 328 filesGC
>>> Warning: Failed to expand heap by 8388608 bytes
>>> GC Warning: Failed to expand heap by 8388608 bytes
>>> GC Warning: Failed to expand heap by 8388608 bytes
>>>
>>> [...]
>>>
>>> GC Warning: Failed to expand heap by 8388608 bytes
>>> GC Warning: Failed to expand heap by 8388608 bytes
>>> builder for
>>> `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv'
>>> failed due to signal 11 (Segmentation fault)
>>> @ build-failed
>>> /gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv
>>> - 1 builder for
>>> `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv'
>>> failed due to signal 11 (Segmentation fault)
>>
>> On closer inspection, this is caused by OOM, with Guile peaking at 2.8G
>> resident (!) at that point, more than on x86_64.
>
> An update: with changes made in Guile “main” over the last couple of
> weeks, memory consumption is 20% lower and compilation is 20% faster
> compared to 3.0.7 (on x86_64):
>
> $ ./pre-inst-env time -f '%U seconds\n%M KiB' guile -c '(use-modules (system base compile)) (compile-file "gnu/packages/crates-io.scm" #:optimization-level 1)'
> 53.84 seconds
> 795972 KiB
> $ ./pre-inst-env time -f '%U seconds\n%M KiB' /data/src/guile-3.0/meta/guile -c '(use-modules (system base compile)) (compile-file "gnu/packages/crates-io.scm" #:optimization-level 1 #:opts (list #:inlinable-exports? #f #:resolve-free-vars? #f))'
> 43.00 seconds
> 618724 KiB
> $
> $ guile --version
> guile (GNU Guile) 3.0.7
> Copyright (C) 2021 Free Software Foundation, Inc.
>
> License LGPLv3+: GNU LGPL 3 or later <http://gnu.org/licenses/lgpl.html>.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> $ (cd /data/src/guile-3.0; git log | head -1)
> commit 2aed3c117c2d667ecca1e38a016f2cb4b524ab50

Impressive!  Keep up the good work!

Thanks,

Maxim

bug#53506: guix-packages-base.drv leads to segfault on i686-linux

[André Batista]

Hi Ludo!

seg 07 fev 2022 às 23:00:27 (1644285627), ludo@gnu.org enviou:
> Hi!
> 
> An update: with changes made in Guile “main” over the last couple of
> weeks, memory consumption is 20% lower and compilation is 20% faster
> compared to 3.0.7 (on x86_64):
> 
> To be continued…

I don't know if it is of any help, but after
'076e825dc5d585943ce820a279fffe4af09757fb' I could pull again after
a couple of weeks hitting this bug. Thanks a lot!

Previously I had tried to move rust-tokio* package definitions to
another file but it wasn't enough to work around it.

While parsing crates-io.scm, it also occured to me to try to
create a 'crates-crypto.scm' and move all crypto related definitions
there. Would you think this to be useful even after you get to the
solution you've been chasing? Are there any guix currently
working on chopping crates-io.scm down?

Cheers!

bug#53506: guix-packages-base.drv leads to segfault on i686-linux

[Ludovic Courtès]

Hi André,

André Batista <nandre@riseup.net> skribis:

> I don't know if it is of any help, but after
> '076e825dc5d585943ce820a279fffe4af09757fb' I could pull again after
> a couple of weeks hitting this bug. Thanks a lot!

Yes.  I just tried ‘guix pull -s i686-linux’ for commit
e641d707e1ec8de2bfc658dcd1757360300aa509 and it passed!

This is certainly due to the reduced heap usage in Guile 3.0.8.
However, while building
/gnu/store/87mqnqwxqbcidbx5bpyrq9xpxmhw1035-guix-packages-base.drv we’re
still peaking at 2.6G resident—only 7% less than before (the packages
files have probably grown in the meantime), so we cannot claim victory
yet.

> Previously I had tried to move rust-tokio* package definitions to
> another file but it wasn't enough to work around it.
>
> While parsing crates-io.scm, it also occured to me to try to
> create a 'crates-crypto.scm' and move all crypto related definitions
> there. Would you think this to be useful even after you get to the
> solution you've been chasing? Are there any guix currently
> working on chopping crates-io.scm down?

I think splitting the file would still be useful, yes; I don’t think
anyone is working on it.

Another thing to consider would be to balance things a bit better, by
arranging so that fewer modules are in ‘guix-packages-base’:

--8<---------------cut here---------------start------------->8---
$ find /gnu/store/ry7fcdq7nwqaca6vanzc5d6z22njr92p-guix-packages-base |wc -l
331
$ find /gnu/store/45izww13rx5lll4pl0vj8xl0633bkzh7-guix-packages |wc -l
212
$ find /gnu/store/ry7fcdq7nwqaca6vanzc5d6z22njr92p-guix-packages-base -name crates\*go
/gnu/store/ry7fcdq7nwqaca6vanzc5d6z22njr92p-guix-packages-base/gnu/packages/crates-graphics.go
/gnu/store/ry7fcdq7nwqaca6vanzc5d6z22njr92p-guix-packages-base/gnu/packages/crates-io.go
/gnu/store/ry7fcdq7nwqaca6vanzc5d6z22njr92p-guix-packages-base/gnu/packages/crates-gtk.go
--8<---------------cut here---------------end--------------->8---

For the record, ‘guix-packages-base’ is computed in (guix self) as the
closure of (gnu packages base).

Thanks for your message!

Ludo’.

bug#53506: guix-packages-base.drv leads to segfault on i686-linux

[Ludovic Courtès]

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

> Yes.  I just tried ‘guix pull -s i686-linux’ for commit
> e641d707e1ec8de2bfc658dcd1757360300aa509 and it passed!
>
> This is certainly due to the reduced heap usage in Guile 3.0.8.
> However, while building
> /gnu/store/87mqnqwxqbcidbx5bpyrq9xpxmhw1035-guix-packages-base.drv we’re
> still peaking at 2.6G resident—only 7% less than before (the packages
> files have probably grown in the meantime), so we cannot claim victory
> yet.

I’ve tried again and it went fine:

--8<---------------cut here---------------start------------->8---
$ guix pull -s i686-linux -p /tmp/i686-guix
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Authenticating channel 'guix', commits 9edb3f6 to 8f0d45c (7 new commits)...
Building from this channel:
  guix      https://git.savannah.gnu.org/git/guix.git   8f0d45c
 ⋮
building /gnu/store/zd1i483p23hdz73hrqs63lgcdclk4aaq-guix-packages.drv...
 ⋮
hint: After setting `PATH', run `hash guix' to make sure your shell refers to
`/tmp/i686-guix/bin/guix'.

$ /tmp/i686-guix/bin/guix describe
Generation 1    Jul 18 2022 22:06:47    (current)
  guix 8f0d45c
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 8f0d45ccac3f6cee69eba8de5e4ae5e5555f1a3d
$ /tmp/i686-guix/bin/guix repl -q
GNU Guile 3.0.8
Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guix-user)> %host-type
$1 = "i686-unknown-linux-gnu"
--8<---------------cut here---------------end--------------->8---

<https://ci.guix.gnu.org/jobset/guix> and
<https://ci.guix.gnu.org/search?query=spec%3Aguix+system%3Ai686-linux>
seem to agree.

So, closing this issue!

Ludo’.