* bug#53506: guix-packages-base.drv leads to segfault on i686-linux
@ 2022-01-24 16:56 Ludovic Courtès
2022-01-24 17:15 ` Ludovic Courtès
0 siblings, 1 reply; 7+ messages in thread
From: Ludovic Courtès @ 2022-01-24 16:56 UTC (permalink / raw)
To: 53506
Hello,
This command fails:
guix pull -s i686-linux \
--commit=13b905bf28ec6309043bd61c5a92744b13352021 \
-p /tmp/test
‘guix-packages-base.drv’ fails to build due to a Guile segfault (!):
--8<---------------cut here---------------start------------->8---
[653/656] compiling... 99.1% of 328 files[654/656] compiling... 99.4% of 328 files[655/656] compiling... 99.7% of 328 filesGC Warning: Failed to expand heap by 8388608 bytes
GC Warning: Failed to expand heap by 8388608 bytes
GC Warning: Failed to expand heap by 8388608 bytes
[...]
GC Warning: Failed to expand heap by 8388608 bytes
GC Warning: Failed to expand heap by 8388608 bytes
builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)
@ build-failed /gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv - 1 builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)
--8<---------------cut here---------------end--------------->8---
I wonder when it started failing, but it may have been pre-core-updates
merge.
Ricardo, do you have a known-good commit?
<https://ci.guix.gnu.org/jobset/guix?border-high=53879> suggests it
passes 25% of the time.
Thanks,
Ludo’.
^ permalink raw reply [flat|nested] 7+ messages in thread* bug#53506: guix-packages-base.drv leads to segfault on i686-linux
2022-01-24 16:56 bug#53506: guix-packages-base.drv leads to segfault on i686-linux Ludovic Courtès
@ 2022-01-24 17:15 ` Ludovic Courtès
2022-02-07 22:00 ` Ludovic Courtès
0 siblings, 1 reply; 7+ messages in thread
From: Ludovic Courtès @ 2022-01-24 17:15 UTC (permalink / raw)
To: 53506
Ludovic Courtès <ludo@gnu.org> skribis:
> ‘guix-packages-base.drv’ fails to build due to a Guile segfault (!):
>
> [653/656] compiling... 99.1% of 328 files[654/656] compiling... 99.4% of 328 files[655/656] compiling... 99.7% of 328 filesGC Warning: Failed to expand heap by 8388608 bytes
> GC Warning: Failed to expand heap by 8388608 bytes
> GC Warning: Failed to expand heap by 8388608 bytes
>
> [...]
>
> GC Warning: Failed to expand heap by 8388608 bytes
> GC Warning: Failed to expand heap by 8388608 bytes
> builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)
> @ build-failed /gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv - 1 builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)
On closer inspection, this is caused by OOM, with Guile peaking at 2.8G
resident (!) at that point, more than on x86_64.
I’m quite sure this is because the compiler resorts to bignums more than
on x86_64 (fixnums are smaller), thereby consuming more heap.
Splitting creates-io.scm into two files might work as a temporary
workaround since the compiler creates a number of labels (integers)
roughly proportional to the number of lines in the file:
--8<---------------cut here---------------start------------->8---
$ wc -l gnu/packages/*.scm|sort -k1 -n |tail
13977 gnu/packages/java.scm
15275 gnu/packages/bioconductor.scm
15929 gnu/packages/bioinformatics.scm
16086 gnu/packages/haskell-xyz.scm
20378 gnu/packages/lisp-xyz.scm
28770 gnu/packages/python-xyz.scm
29960 gnu/packages/emacs-xyz.scm
32071 gnu/packages/cran.scm
70442 gnu/packages/crates-io.scm
690662 totalo
--8<---------------cut here---------------end--------------->8---
Background:
https://lists.gnu.org/archive/html/guile-devel/2017-10/msg00035.html
https://lists.gnu.org/archive/html/guile-devel/2020-02/msg00023.html
https://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile
Ludo’.
^ permalink raw reply [flat|nested] 7+ messages in thread
* bug#53506: guix-packages-base.drv leads to segfault on i686-linux
2022-01-24 17:15 ` Ludovic Courtès
@ 2022-02-07 22:00 ` Ludovic Courtès
2022-02-08 14:03 ` Maxim Cournoyer
2022-02-12 0:28 ` André Batista
0 siblings, 2 replies; 7+ messages in thread
From: Ludovic Courtès @ 2022-02-07 22:00 UTC (permalink / raw)
To: 53506
Hi!
Ludovic Courtès <ludo@gnu.org> skribis:
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> ‘guix-packages-base.drv’ fails to build due to a Guile segfault (!):
>>
>> [653/656] compiling... 99.1% of 328 files[654/656] compiling... 99.4% of 328 files[655/656] compiling... 99.7% of 328 filesGC Warning: Failed to expand heap by 8388608 bytes
>> GC Warning: Failed to expand heap by 8388608 bytes
>> GC Warning: Failed to expand heap by 8388608 bytes
>>
>> [...]
>>
>> GC Warning: Failed to expand heap by 8388608 bytes
>> GC Warning: Failed to expand heap by 8388608 bytes
>> builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)
>> @ build-failed /gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv - 1 builder for `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv' failed due to signal 11 (Segmentation fault)
>
> On closer inspection, this is caused by OOM, with Guile peaking at 2.8G
> resident (!) at that point, more than on x86_64.
An update: with changes made in Guile “main” over the last couple of
weeks, memory consumption is 20% lower and compilation is 20% faster
compared to 3.0.7 (on x86_64):
--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env time -f '%U seconds\n%M KiB' guile -c '(use-modules (system base compile)) (compile-file "gnu/packages/crates-io.scm" #:optimization-level 1)'
53.84 seconds
795972 KiB
$ ./pre-inst-env time -f '%U seconds\n%M KiB' /data/src/guile-3.0/meta/guile -c '(use-modules (system base compile)) (compile-file "gnu/packages/crates-io.scm" #:optimization-level 1 #:opts (list #:inlinable-exports? #f #:resolve-free-vars? #f))'
43.00 seconds
618724 KiB
$
$ guile --version
guile (GNU Guile) 3.0.7
Copyright (C) 2021 Free Software Foundation, Inc.
License LGPLv3+: GNU LGPL 3 or later <http://gnu.org/licenses/lgpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ (cd /data/src/guile-3.0; git log | head -1)
commit 2aed3c117c2d667ecca1e38a016f2cb4b524ab50
--8<---------------cut here---------------end--------------->8---
To be continued…
Ludo’.
^ permalink raw reply [flat|nested] 7+ messages in thread
* bug#53506: guix-packages-base.drv leads to segfault on i686-linux
2022-02-07 22:00 ` Ludovic Courtès
@ 2022-02-08 14:03 ` Maxim Cournoyer
2022-02-12 0:28 ` André Batista
1 sibling, 0 replies; 7+ messages in thread
From: Maxim Cournoyer @ 2022-02-08 14:03 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 53506
Hi Ludovic!
Ludovic Courtès <ludo@gnu.org> writes:
> Hi!
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> skribis:
>>
>>> ‘guix-packages-base.drv’ fails to build due to a Guile segfault (!):
>>>
>>> [653/656] compiling... 99.1% of 328 files[654/656] compiling...
>>> 99.4% of 328 files[655/656] compiling... 99.7% of 328 filesGC
>>> Warning: Failed to expand heap by 8388608 bytes
>>> GC Warning: Failed to expand heap by 8388608 bytes
>>> GC Warning: Failed to expand heap by 8388608 bytes
>>>
>>> [...]
>>>
>>> GC Warning: Failed to expand heap by 8388608 bytes
>>> GC Warning: Failed to expand heap by 8388608 bytes
>>> builder for
>>> `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv'
>>> failed due to signal 11 (Segmentation fault)
>>> @ build-failed
>>> /gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv
>>> - 1 builder for
>>> `/gnu/store/cnrmdbcyy8r9bs2gl2kgpnyplivrzf5c-guix-packages-base.drv'
>>> failed due to signal 11 (Segmentation fault)
>>
>> On closer inspection, this is caused by OOM, with Guile peaking at 2.8G
>> resident (!) at that point, more than on x86_64.
>
> An update: with changes made in Guile “main” over the last couple of
> weeks, memory consumption is 20% lower and compilation is 20% faster
> compared to 3.0.7 (on x86_64):
>
> $ ./pre-inst-env time -f '%U seconds\n%M KiB' guile -c '(use-modules (system base compile)) (compile-file "gnu/packages/crates-io.scm" #:optimization-level 1)'
> 53.84 seconds
> 795972 KiB
> $ ./pre-inst-env time -f '%U seconds\n%M KiB' /data/src/guile-3.0/meta/guile -c '(use-modules (system base compile)) (compile-file "gnu/packages/crates-io.scm" #:optimization-level 1 #:opts (list #:inlinable-exports? #f #:resolve-free-vars? #f))'
> 43.00 seconds
> 618724 KiB
> $
> $ guile --version
> guile (GNU Guile) 3.0.7
> Copyright (C) 2021 Free Software Foundation, Inc.
>
> License LGPLv3+: GNU LGPL 3 or later <http://gnu.org/licenses/lgpl.html>.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> $ (cd /data/src/guile-3.0; git log | head -1)
> commit 2aed3c117c2d667ecca1e38a016f2cb4b524ab50
Impressive! Keep up the good work!
Thanks,
Maxim
^ permalink raw reply [flat|nested] 7+ messages in thread
* bug#53506: guix-packages-base.drv leads to segfault on i686-linux
2022-02-07 22:00 ` Ludovic Courtès
2022-02-08 14:03 ` Maxim Cournoyer
@ 2022-02-12 0:28 ` André Batista
2022-02-12 14:24 ` Ludovic Courtès
1 sibling, 1 reply; 7+ messages in thread
From: André Batista @ 2022-02-12 0:28 UTC (permalink / raw)
To: Ludovic Courtès; +Cc: 53506
Hi Ludo!
seg 07 fev 2022 às 23:00:27 (1644285627), ludo@gnu.org enviou:
> Hi!
>
> An update: with changes made in Guile “main” over the last couple of
> weeks, memory consumption is 20% lower and compilation is 20% faster
> compared to 3.0.7 (on x86_64):
>
> To be continued…
I don't know if it is of any help, but after
'076e825dc5d585943ce820a279fffe4af09757fb' I could pull again after
a couple of weeks hitting this bug. Thanks a lot!
Previously I had tried to move rust-tokio* package definitions to
another file but it wasn't enough to work around it.
While parsing crates-io.scm, it also occured to me to try to
create a 'crates-crypto.scm' and move all crypto related definitions
there. Would you think this to be useful even after you get to the
solution you've been chasing? Are there any guix currently
working on chopping crates-io.scm down?
Cheers!
^ permalink raw reply [flat|nested] 7+ messages in thread
* bug#53506: guix-packages-base.drv leads to segfault on i686-linux
2022-02-12 0:28 ` André Batista
@ 2022-02-12 14:24 ` Ludovic Courtès
2022-07-18 20:12 ` Ludovic Courtès
0 siblings, 1 reply; 7+ messages in thread
From: Ludovic Courtès @ 2022-02-12 14:24 UTC (permalink / raw)
To: André Batista; +Cc: 53506
Hi André,
André Batista <nandre@riseup.net> skribis:
> I don't know if it is of any help, but after
> '076e825dc5d585943ce820a279fffe4af09757fb' I could pull again after
> a couple of weeks hitting this bug. Thanks a lot!
Yes. I just tried ‘guix pull -s i686-linux’ for commit
e641d707e1ec8de2bfc658dcd1757360300aa509 and it passed!
This is certainly due to the reduced heap usage in Guile 3.0.8.
However, while building
/gnu/store/87mqnqwxqbcidbx5bpyrq9xpxmhw1035-guix-packages-base.drv we’re
still peaking at 2.6G resident—only 7% less than before (the packages
files have probably grown in the meantime), so we cannot claim victory
yet.
> Previously I had tried to move rust-tokio* package definitions to
> another file but it wasn't enough to work around it.
>
> While parsing crates-io.scm, it also occured to me to try to
> create a 'crates-crypto.scm' and move all crypto related definitions
> there. Would you think this to be useful even after you get to the
> solution you've been chasing? Are there any guix currently
> working on chopping crates-io.scm down?
I think splitting the file would still be useful, yes; I don’t think
anyone is working on it.
Another thing to consider would be to balance things a bit better, by
arranging so that fewer modules are in ‘guix-packages-base’:
--8<---------------cut here---------------start------------->8---
$ find /gnu/store/ry7fcdq7nwqaca6vanzc5d6z22njr92p-guix-packages-base |wc -l
331
$ find /gnu/store/45izww13rx5lll4pl0vj8xl0633bkzh7-guix-packages |wc -l
212
$ find /gnu/store/ry7fcdq7nwqaca6vanzc5d6z22njr92p-guix-packages-base -name crates\*go
/gnu/store/ry7fcdq7nwqaca6vanzc5d6z22njr92p-guix-packages-base/gnu/packages/crates-graphics.go
/gnu/store/ry7fcdq7nwqaca6vanzc5d6z22njr92p-guix-packages-base/gnu/packages/crates-io.go
/gnu/store/ry7fcdq7nwqaca6vanzc5d6z22njr92p-guix-packages-base/gnu/packages/crates-gtk.go
--8<---------------cut here---------------end--------------->8---
For the record, ‘guix-packages-base’ is computed in (guix self) as the
closure of (gnu packages base).
Thanks for your message!
Ludo’.
^ permalink raw reply [flat|nested] 7+ messages in thread
* bug#53506: guix-packages-base.drv leads to segfault on i686-linux
2022-02-12 14:24 ` Ludovic Courtès
@ 2022-07-18 20:12 ` Ludovic Courtès
0 siblings, 0 replies; 7+ messages in thread
From: Ludovic Courtès @ 2022-07-18 20:12 UTC (permalink / raw)
To: 53506-done
Hi,
Ludovic Courtès <ludo@gnu.org> skribis:
> Yes. I just tried ‘guix pull -s i686-linux’ for commit
> e641d707e1ec8de2bfc658dcd1757360300aa509 and it passed!
>
> This is certainly due to the reduced heap usage in Guile 3.0.8.
> However, while building
> /gnu/store/87mqnqwxqbcidbx5bpyrq9xpxmhw1035-guix-packages-base.drv we’re
> still peaking at 2.6G resident—only 7% less than before (the packages
> files have probably grown in the meantime), so we cannot claim victory
> yet.
I’ve tried again and it went fine:
--8<---------------cut here---------------start------------->8---
$ guix pull -s i686-linux -p /tmp/i686-guix
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Authenticating channel 'guix', commits 9edb3f6 to 8f0d45c (7 new commits)...
Building from this channel:
guix https://git.savannah.gnu.org/git/guix.git 8f0d45c
⋮
building /gnu/store/zd1i483p23hdz73hrqs63lgcdclk4aaq-guix-packages.drv...
⋮
hint: After setting `PATH', run `hash guix' to make sure your shell refers to
`/tmp/i686-guix/bin/guix'.
$ /tmp/i686-guix/bin/guix describe
Generation 1 Jul 18 2022 22:06:47 (current)
guix 8f0d45c
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 8f0d45ccac3f6cee69eba8de5e4ae5e5555f1a3d
$ /tmp/i686-guix/bin/guix repl -q
GNU Guile 3.0.8
Copyright (C) 1995-2021 Free Software Foundation, Inc.
Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.
Enter `,help' for help.
scheme@(guix-user)> %host-type
$1 = "i686-unknown-linux-gnu"
--8<---------------cut here---------------end--------------->8---
<https://ci.guix.gnu.org/jobset/guix> and
<https://ci.guix.gnu.org/search?query=spec%3Aguix+system%3Ai686-linux>
seem to agree.
So, closing this issue!
Ludo’.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2022-07-18 20:13 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-24 16:56 bug#53506: guix-packages-base.drv leads to segfault on i686-linux Ludovic Courtès
2022-01-24 17:15 ` Ludovic Courtès
2022-02-07 22:00 ` Ludovic Courtès
2022-02-08 14:03 ` Maxim Cournoyer
2022-02-12 0:28 ` André Batista
2022-02-12 14:24 ` Ludovic Courtès
2022-07-18 20:12 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/guix.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).