[bug#59078] [PATCH] lint: Split the derivation lint checker by system.

["Ludovic Courtès" <ludo@gnu.org>]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> I've put some numbers below, with no changes the last batch to finish
> processing leaves the heap at 7755MiB [1], then Guile crashes after
> that.
>
> With the patch you sent, the heap size seems to stabilise at 4042MiB
> [2]. It also crashes at the end due to the match block not matching '(),
> but that's not important.
>
> I also hacked the lint script to run the checkers in the same manor as
> the Guix Data Service, so one checker at a time rather than one package
> at a time. That led to a max heap size of 3505MiB [3].

Thanks for testing and reporting back!

> Reducing the memory usage helps reduce the cost/improve the throughput
> of loading data in to the Guix Data Service which is my primary
> motivation here. I'm also not only concerned with reducing the peak
> memory usage, but trying to have an implementation that'll gracefully
> handle more systems being supported in the future.

Understood.  From the viewpoint of core Guix, I’d like to understand
where all this memory is going; this is unreasonable.  The heap profiler
I posted recently¹, coupled with ‘gcprof’, might help us understand
what’s going on.

¹ https://lists.gnu.org/archive/html/guile-user/2022-11/msg00012.html

> 1: no batching, one derivation checker

[...]

> 7755.0 MiB
>
> 2: batches of 1000 with fresh store connection, one derivation checker
> 1057.0 MiB
> 1137.0 MiB
> 1481.0 MiB
> 1481.0 MiB
> 1481.0 MiB
> 1481.0 MiB
> 1697.0 MiB
> 1697.0 MiB
> 1697.0 MiB
> 1761.0 MiB
> 1761.0 MiB
> 1761.0 MiB
> 1937.0 MiB
> 1977.0 MiB
> 1985.0 MiB
> 3065.0 MiB
> 3633.0 MiB
> 4041.0 MiB
> 4042.0 MiB
> 4042.0 MiB
> 4042.0 MiB

This seems to indicate a memory leak outside <store-connection>, such as
a static cache.  “GUIX_PROFILING=memoization” may give a hint.

I’d really like to get it solved before we come up with workarounds like
those we’re talking about.  (However I’ll postpone this one because I’d
like to focus on the release for now.)

Thanks,
Ludo’.