bug#69596: ‘guix publish’ memory leak

bug#69596: ‘guix publish’ memory leak

[Ludovic Courtès]

It seems that ‘guix publish’ has been leaking memory noticeably I’d say
since the beginning of the year on berlin.  After roughly 10 days, it
has several GiB resident and needs to be restarted or it becomes too
unresponsive.

I wonder what could be causing this because we used to let it run for
months without problems I believe.

This is what’s currently running on berlin:

  /gnu/store/cpnshv80n3mar5lwp4qqa2dxxxv4zb03-guix-1.4.0-16.aeb4943/libexec/guix/guile \
  /gnu/store/cpnshv80n3mar5lwp4qqa2dxxxv4zb03-guix-1.4.0-16.aeb4943/bin/guix \
  publish -u guix-publish -p 3000 -C lzip:9 -C zstd:19 --nar-path=nar \
  --listen=localhost --workers=8 --ttl=15552000s \
  --cache=/var/cache/guix/publish --cache-bypass-threshold=157286400

… coming from commit 21e4d6cd6913eca131f2c0fd0cd509fc843c7eb8.

I don’t see anything that changed recently, be it in maintenance.git
(for the parameters), ‘guix publish’, guile-{lzlib,zstd}, or Guile.
Maybe we had just been lucky?  Anyone else seeing this?

Ludo’.

bug#69596: ‘guix publish’ memory leak

[Ludovic Courtès]

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

> It seems that ‘guix publish’ has been leaking memory noticeably I’d say
> since the beginning of the year on berlin.  After roughly 10 days, it
> has several GiB resident and needs to be restarted or it becomes too
> unresponsive.
>
> I wonder what could be causing this because we used to let it run for
> months without problems I believe.
>
> This is what’s currently running on berlin:
>
>   /gnu/store/cpnshv80n3mar5lwp4qqa2dxxxv4zb03-guix-1.4.0-16.aeb4943/libexec/guix/guile \
>   /gnu/store/cpnshv80n3mar5lwp4qqa2dxxxv4zb03-guix-1.4.0-16.aeb4943/bin/guix \
>   publish -u guix-publish -p 3000 -C lzip:9 -C zstd:19 --nar-path=nar \
>   --listen=localhost --workers=8 --ttl=15552000s \
>   --cache=/var/cache/guix/publish --cache-bypass-threshold=157286400
>
> … coming from commit 21e4d6cd6913eca131f2c0fd0cd509fc843c7eb8.

It turned out to be a guile-lzlib leak that had always been present:

  https://notabug.org/guile-lzlib/guile-lzlib/commit/74bd35b690801a10ed775d486fffc7372b1b341c

The reason we were seeing it more on berlin is probably because we
increased the cache-bypass-threshold, which goes through the
‘make-lzip-output-port’ code path (as opposed to
‘call-with-lzip-output-port’).

The bug could be reproduced with:

  guix publish -p 8124 … &
  while true ; do wget -q -O/dev/full http://localhost:8124/nar/lzip/…-coreutils-9.1 ; done

(Replace the ellipses with the actual store file name of coreutils.)

Fixed by commit 7cef6b7ba555a9dfaf6d09cb7e112b0df77d5141, which updates
guile-lzlib.

Ludo’.

bug#69596: Possibly breaks Guix pull on foreign distros

[Jonathan Roberts via Bug reports for GNU Guix]

Ran in to the error that a lot of people are reporting - too many heap sections on Guix pull. 

Worked through commits to see where Guix pull stopped working for me, and the answer seems to be that it broke with 7cef6b7ba - gnu: guile-lzlib: Update to 0.3.0.

Could this be confirmed and reverted if it is breaking pull? 

Sent from my iPhone