bug#49439: grafts cause “guix environment” to get killed with OOM

[Sarah Morgensen <iskarian@mgsn.dev>]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> Thinking about it, the grafts code depends on what’s in the store: when
>> nothing is in the store, it bounces to the “build handler”, which
>> accumulates the list of missing store items, until it starts building
>> them.
>
> So I can reproduce the problem Ricardo and you initially reported by
> running:
>
>   ./pre-inst-env guix environment pigx-scrnaseq --search-paths
>
> after removing some of the ungrafted store items with:
>
>   guix gc -D $(guix build r-rlang --no-grafts) \
>     $(guix gc --references $(guix build pigx-scrnaseq --no-grafts))

Same here. I'm glad we were able to pinpoint this!

>
> The seemingly endless CPU usage and unbound memory use comes from the
> ‘build-accumulator’ build handler.  Here, the graph of ‘pigx-scrnaseq’
> has many nodes, and many of them depend on, say, ‘r-rlang’.  Thus, when
> ‘r-rlang’ is not in the store, the grafting code keeps asking for it by
> calling ‘build-derivations’, which aborts to the build handler; the
> build handler saves the .drv file name and the continuation and keeps
> going.  But since the next package also depends on ‘r-langr’, we abort
> again to the build handler, and so on.
>
> The end result is a very long list of <unresolved> nodes, probably of
> this order in this case:
>
>   $ guix graph -t reverse-package r-rlang |grep 'label = "'|wc -l
>   594
>
> Presumably, the captured continuations occupy quite a bit of memory,
> hence the quick growth.
>
> I suppose one solution is to fire suspended builds when the build
> handler sees a build request for a given derivation for the second time.
> It needs more thought and testing…

I wonder if instead it's possible to reduce the memory taken by the
continuations? As someone who has absolutely no experience with the
build/derivation system, it seems like all we *should* need to save is
the derivation file name.

>
> Ludo’.
>
> PS: Did you know ‘pigx-scrnaseq’ has twice as many nodes as
>     ‘libreoffice’?
>
>       $ guix graph -t bag pigx-scrnaseq |grep 'label = "'|wc -l
>       1359
>       $ guix graph -t bag libreoffice |grep 'label = "'|wc -l
>       699
>
>     That makes it a great example to study and fix scalability issues!

For those interested, I've compiled a list of the top 60
highest-dependency packages, as reported by package-closure:

pigx             	1630
nextcloud-client 	1539
akregator        	1486
kmail            	1484
korganizer       	1481
kdepim-runtime   	1480
kincidenceeditor 	1478
keventviews      	1477
kmailcommon      	1476
kcalendarsupport 	1475
kmessagelib      	1474
knotes           	1472
kaddressbook     	1469
libksieve        	1467
kdepim-apps-libs 	1465
libgravatar      	1463
kpimcommon       	1462
akonadi-calendar 	1453
pigx-bsseq       	1448
elisa            	1446
kaffeine         	1432
kdenlive         	1431
kmailtransport   	1431
dolphin-plugins  	1426
k3b              	1424
libkgapi         	1422
dolphin          	1421
kopete           	1403
pigx-sars-cov2-ww	1401
krdc             	1398
baloo-widgets    	1397
baloo            	1396
pigx-chipseq     	1396
krfb             	1389
ffmpegthumbs     	1388
kget             	1382
kmplayer         	1381
kdelibs4support  	1375
pigx-scrnaseq    	1342
kdevelop         	1340
kmailimporter    	1326
libkdepim        	1325
pigx-rnaseq      	1324
labplot          	1316
smb4k            	1315
kleopatra        	1311
kalarmcal        	1311
choqok           	1311
okular           	1310
ktnef            	1310
ktorrent         	1310
kate             	1308
akonadi-search   	1308
kcalutils        	1307
yakuake          	1306
khelpcenter      	1305
libksysguard     	1305
kdeconnect       	1304
konsole          	1304
libkleo          	1304

There seem to be a lot of kde packages in there, so perhaps this issue
isn't as rare as we might otherwise expect?

--
Sarah