Avoiding the garbage collector for the Guix Data Service

Avoiding the garbage collector for the Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 774 bytes --]

Hey,

The Guix Data Service has Guix compute lots of derivations, and then it
loads the data in to a database. I'm having some issues with the garbage
collector though, as sometimes it removes data from the disk before I've
had a chance to read it (at least I think this is what's happening, I
think this job failed for this reason [1]).

1: http://data.guix.gnu.org/job/10638

I'm not quite sure what the best strategy is to combat this? I had a
look, and I couldn't see how to acquire a lock to prevent the garbage
collector from running. I thought about registering thousands of gc
roots, but that seems to just move the race a bit, as there's still a
period between generating the derivation and registering the root where
it can be deleted.

Any ideas?

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Avoiding the garbage collector for the Guix Data Service

[Ludovic Courtès]

Hi Christopher,

Christopher Baines <mail@cbaines.net> skribis:

> The Guix Data Service has Guix compute lots of derivations, and then it
> loads the data in to a database. I'm having some issues with the garbage
> collector though, as sometimes it removes data from the disk before I've
> had a chance to read it (at least I think this is what's happening, I
> think this job failed for this reason [1]).
>
> 1: http://data.guix.gnu.org/job/10638
>
> I'm not quite sure what the best strategy is to combat this? I had a
> look, and I couldn't see how to acquire a lock to prevent the garbage
> collector from running. I thought about registering thousands of gc
> roots, but that seems to just move the race a bit, as there's still a
> period between generating the derivation and registering the root where
> it can be deleted.
>
> Any ideas?

At the Guix Days we discussed using ‘add-temp-root’ and similar to
ensure that .drv files are at least GC-protected for the duration of the
session (connection to the daemon).

Does that address your problem?

Thanks,
Ludo’.

Re: Avoiding the garbage collector for the Guix Data Service

[Christopher Baines]

[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]


Ludovic Courtès <ludo@gnu.org> writes:

> Hi Christopher,
>
> Christopher Baines <mail@cbaines.net> skribis:
>
>> The Guix Data Service has Guix compute lots of derivations, and then it
>> loads the data in to a database. I'm having some issues with the garbage
>> collector though, as sometimes it removes data from the disk before I've
>> had a chance to read it (at least I think this is what's happening, I
>> think this job failed for this reason [1]).
>>
>> 1: http://data.guix.gnu.org/job/10638
>>
>> I'm not quite sure what the best strategy is to combat this? I had a
>> look, and I couldn't see how to acquire a lock to prevent the garbage
>> collector from running. I thought about registering thousands of gc
>> roots, but that seems to just move the race a bit, as there's still a
>> period between generating the derivation and registering the root where
>> it can be deleted.
>>
>> Any ideas?
>
> At the Guix Days we discussed using ‘add-temp-root’ and similar to
> ensure that .drv files are at least GC-protected for the duration of the
> session (connection to the daemon).
>
> Does that address your problem?

I think so, it at least seems to greatly reduce the chance the
derivations get removed. I made the Guix Data Service change here [1],
and I haven't seen the issue since.

1: https://git.savannah.gnu.org/cgit/guix/data-service.git/commit/?id=f59354ed23a9c50e43b7aaaeaddb5365feda4e29

Thanks,

Chris

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]

Re: Avoiding the garbage collector for the Guix Data Service

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> I think so, it at least seems to greatly reduce the chance the
> derivations get removed. I made the Guix Data Service change here [1],
> and I haven't seen the issue since.
>
> 1: https://git.savannah.gnu.org/cgit/guix/data-service.git/commit/?id=f59354ed23a9c50e43b7aaaeaddb5365feda4e29

Good!  It looks like the right thing to do, to me.

Ludo’.