bug#33378: offload: Avoid hosts with little or no free disk space

bug#33378: offload: Avoid hosts with little or no free disk space

[Ludovic Courtès]

‘choose-build-machine’ in (guix scripts offload) could/should exclude
build machines that don’t have at least, say, 100 MiB available in their
store.  That would avoid ENOSPC build failures such as
<https://berlin.guixsd.org/log/0hvhrfdjwnmdm2rdf58knshbbikm4ryv-r-bsgenome-hsapiens-ucsc-hg19-masked-1.3.99>.

Thanks,
Ludo’.

bug#33378: offload: Avoid hosts with little or no free disk space

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> ‘choose-build-machine’ in (guix scripts offload) could/should exclude
> build machines that don’t have at least, say, 100 MiB available in their
> store.  That would avoid ENOSPC build failures such as
> <https://berlin.guixsd.org/log/0hvhrfdjwnmdm2rdf58knshbbikm4ryv-r-bsgenome-hsapiens-ucsc-hg19-masked-1.3.99>.

Done in commit 63b0c3eaccdf1816b419632cd7fe721934d2eb27.

Ludo'.

bug#33378: offload: Avoid hosts with little or no free disk space

[Mark H Weaver]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> ludo@gnu.org (Ludovic Courtès) skribis:
>
>> ‘choose-build-machine’ in (guix scripts offload) could/should exclude
>> build machines that don’t have at least, say, 100 MiB available in their
>> store.  That would avoid ENOSPC build failures such as
>> <https://berlin.guixsd.org/log/0hvhrfdjwnmdm2rdf58knshbbikm4ryv-r-bsgenome-hsapiens-ucsc-hg19-masked-1.3.99>.
>
> Done in commit 63b0c3eaccdf1816b419632cd7fe721934d2eb27.

Many builds require more free space than 100 MB, so this is only a
partial solution.  Also, if multiple builds are happening on the same
machine, the amount of free space available for each build will be less
than 100 MB.

Finally, the other related commit that attempts to detect build failures
caused by ENOSPC by checking if there's at least 10 MB free after the
failure won't often work in practice, I suspect.  The problem is, by the
time you perform that check, the build directory has already been
deleted.

       Mark

bug#33378: offload: Avoid hosts with little or no free disk space

[Ludovic Courtès]

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> ludo@gnu.org (Ludovic Courtès) skribis:
>>
>>> ‘choose-build-machine’ in (guix scripts offload) could/should exclude
>>> build machines that don’t have at least, say, 100 MiB available in their
>>> store.  That would avoid ENOSPC build failures such as
>>> <https://berlin.guixsd.org/log/0hvhrfdjwnmdm2rdf58knshbbikm4ryv-r-bsgenome-hsapiens-ucsc-hg19-masked-1.3.99>.
>>
>> Done in commit 63b0c3eaccdf1816b419632cd7fe721934d2eb27.
>
> Many builds require more free space than 100 MB, so this is only a
> partial solution.  Also, if multiple builds are happening on the same
> machine, the amount of free space available for each build will be less
> than 100 MB.

You’re right, but this at least avoids the most obvious problems.

> Finally, the other related commit that attempts to detect build failures
> caused by ENOSPC by checking if there's at least 10 MB free after the
> failure won't often work in practice, I suspect.  The problem is, by the
> time you perform that check, the build directory has already been
> deleted.

Correct.  (I wasn’t sure the build directory would already be deleted at
that point but rereading build.cc confirms this.)

I suppose the only way to solve this correctly would be to keep the
build tree and then delete it from (guix scripts offload).  WDYT?

Thanks for your feedback!

Ludo’.

bug#33378: offload: Avoid hosts with little or no free disk space

[Mark H Weaver]

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:
> I suppose the only way to solve this correctly would be to keep the
> build tree and then delete it from (guix scripts offload).  WDYT?

Generally, this sounds like a reasonable approach.  However, one issue
is that it will likely lead to build directories being left around
occasionally when transient network outages occur.  Those would need to
be cleaned up manually, or by cron jobs on the build slaves.

Also, does the daemon API provide a way to find the name of the build
directory?  What if there's more than one build directory with the same
name and version number, differing only by the numeric suffix?

Ideally, there would be a way for the build-side code to return a result
code indicating that the build "aborted" rather than "failed", for this
to be recognized by the daemon as a transient failure (i.e. not cached),
and to propagate this information back to the client who requested the
build.  Do the relevant daemon APIs and offloading protocols allow for
this information to be propagated to the client?

       Thanks,
         Mark