Re: Suggestion: disable offloading for texlive builds on hydra?

[ludo@gnu.org (Ludovic Courtès)]

[-- Attachment #1: Type: text/plain, Size: 1776 bytes --]

John Darrington <john@darrington.wattle.id.au> skribis:

> On Sun, Oct 26, 2014 at 03:36:03AM -0400, Mark H Weaver wrote:
>      When texlive is built on hydra, the build slave that built it is tied up
>      for 12 hours or more waiting for the build outputs (over 3 gigabytes!)
>      to be transferred back to hydra.
>      
>      By design, only one transfer can happen at a time from a given build
>      slave, so during those 12 hours, the build slave's CPU is left idle, and
>      typically another 3 built-but-not-yet-transferred packages must wait
>      until the texlive transfer finishes.
>
> Why is it designed like that?  It seems like a poor design to me.

The rationale was that, in general, you just slow everything down by
sending several things at once.  TeX Live is a pathological case in that
respect.

As for disabling offloading, see my reply to Federico: we could expose
#:local-build? to gnu-build-system, and use that here, but at the moment
that also disables substitutes.  WDYT?

>      I suggest that we arrange for hydra.gnu.org to build texlive locally for
>      x86_64 and i686, to avoid this problem.
>
> Would it help if texlive was split into more outputs?  For example, the docs 
> take up a lot of space, and not everyone needs them.

I think it may help a bit, at least by leaving a small window during
which other builds could get started.  And it would also be more
convenient for users, who could choose whether to install the whole
thing or not.

When you mentioned it some time ago on IRC, I gave it a try, but then
failed to actually test the patch due to...  ENOSPC.  :-)

Anyway, here’s the patch I had.  I’d be happy if you or someone else
could just confirm it works as expected:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 439 bytes --]

diff --git a/gnu/packages/texlive.scm b/gnu/packages/texlive.scm
index e562b02..bc0ece7 100644
--- a/gnu/packages/texlive.scm
+++ b/gnu/packages/texlive.scm
@@ -88,7 +88,7 @@
       ("pkg-config" ,pkg-config)
       ("python" ,python-2) ; incompatible with Python 3 (print syntax)
       ("tcsh" ,tcsh)))
-   (outputs '("out" "data"))
+   (outputs '("out" "data" "doc"))
    (arguments
     `(#:out-of-source? #t
       #:configure-flags


[-- Attachment #3: Type: text/plain, Size: 438 bytes --]


Data point: there’s 1.6 GiB in texmf-dist/doc (which the patch above
splits out), and 1.4 GiB in texmf-dist/fonts.

Another option Andreas and I discussed a while back would be to use a
fixed-output derivation for the data, since it’s really what it is.
That’s a bit hacky though: we’d have to install it, compute the hash of
the installed files, and then use that as the derivation’s output hash.

Thanks,
Ludo’.