bug#35181: Hydra offloads often get stuck while exporting build requisites

[Mark H Weaver <mhw@netris.org>]

Hi Efraim,

Efraim Flashner <efraim@flashner.co.il> writes:
> For these two specifically it's possible that they are just really
> really big.

The source checkout currently being transferred for build 3432472
(/gnu/store/…-font-google-material-design-icons-3.0.1-checkout) is 176
megabytes uncompressed, as measured by "du -s --si", which is not
precisely same as NAR size, but hopefully close enough for a rough
estimate.  As I write this, build 3432472 been stuck here for 24 hours
15 minutes.  Even if the average transfer rate were 4 kilobytes per
second, it should have been done in half that time.

I should also note that this is the *10th* attempt for build 3432472.
I didn't realize until now, but I've manually aborted this same build
9 times before now, before filing this bug report.

On its fourth attempt, it ran for just over *48* hours before I aborted
it.  Here's a full list of how long each of the build attempts have run:

  Nr      Duration          Machine Status
  --------------------------------------------------
  1       20h 23m 27s       Aborted (log, raw, tail)
  2       5m 28s            Aborted (log, raw, tail)
  3       1d 9h 54m 42s     Aborted (log, raw, tail)
  4       2d 0h 1m 17s      Aborted (log, raw, tail)
  5       8h 12m 4s         Aborted (log, raw, tail)
  6       1d 21h 37m 24s    Aborted (log, raw, tail)
  7       8h 26m 1s         Aborted (log, raw, tail)
  8       11h 21m 6s        Aborted (log, raw, tail)
  9       4h 38m 10s        Aborted (log, raw, tail)
  10      1d 0h 22m 19s     Building (log, raw, tail)
  --------------------------------------------------
  Source: <https://hydra.gnu.org/build/3432472#tabs-buildsteps>

The other build (3432052) has a very similar story.  Its source checkout
is slightly larger at 287 megabytes, and it's currently on its 8th
attempt, which has been running for 23 hours 27 minutes as I write this.

  <https://hydra.gnu.org/build/3432052#tabs-buildsteps>

  Nr      Duration          Machine Status
  --------------------------------------------------
  1       1d 4h 32m 24s     Aborted (log, raw, tail)
  2       2d 0h 1m 17s      Aborted (log, raw, tail)
  3       8h 12m 5s         Aborted (log, raw, tail)
  4       1d 10h 11m 48s    Aborted (log, raw, tail)
  5       8h 26m 11s        Aborted (log, raw, tail)
  6       10h 49m 44s       Aborted (log, raw, tail)
  7       4h 38m 20s        Aborted (log, raw, tail)
  8       22h 43m 9s        Building (log, raw, tail)
  --------------------------------------------------

So far, these two builds alone have consumed a total of 15 days of
Hydra's build slot time.  One of the builds is on x86_64-linux and the
other on i686-linux.

As I recall, a similar thing happened with the mozjs-60 builds, although
I didn't look closely.

       Mark