bug#19976: Cannot install texlive

bug#19976: Cannot install texlive

[Ricardo Wurmus]

After a network interruption while downloading the binary substitute for
texlive I can no longer install it.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~/dev/guix $ ./pre-inst-env guix package -i texlive
The following package will be installed:
   texlive	2014	/gnu/store/za7kilci43in8dfdrrhq90rwlxzrxwf9-texlive-2014

substitute-binary: updating list of substitutes from 'http://hydra.gnu.org'...
The following derivations will be built:
   /gnu/store/kj71r2vy4dncgnm403rb675wbgnzylvh-profile.drv
   /gnu/store/8q0680g1pfifi4h6dbzx9651g44rzj8l-texlive-2014.drv
   /gnu/store/mslmw7h8yhs84qvjvn1l0dhyi0bcz8fi-info-dir.drv
The following files will be downloaded:
   /gnu/store/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data
   /gnu/store/dar8cp7sb52iwjvrq3pmansx4ic7amwh-texlive-2014
found valid signature for '/gnu/store/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data', from 'http://hydra.gnu.org/nar/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data'
downloading `/gnu/store/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data' (3392.6 MiB installed)...
GC Warning: Failed to expand heap by 1986744320 bytes
GC Warning: Failed to expand heap by 1986220032 bytes
GC Warning: Out of Memory! Heap size: 6 MiB. Returning NULL!

bzip2: Compressed file ends unexpectedly;
	perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
	Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

guix substitute-binary: error: corrupt input while restoring '/gnu/store/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data' from #{read pipe}#
guix package: error: build failed: some substitutes for the outputs of derivation `/gnu/store/627fwfqdpv85q6j7dvcjwnmi6gfwy2ll-texlive-2014.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


The error leading up to this situation looks like this:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
downloading `/gnu/store/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data' (3392.6 MiB installed)...
http://hydra.gnu.org/nar/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data	1612808.0 KiB transferredBacktrace:
In ice-9/eval.scm:
 432: 19 [eval # #]
In ice-9/boot-9.scm:
2401: 18 [save-module-excursion #<procedure 9ff7ca0 at ice-9/boot-9.scm:4045:3 ()>]
4050: 17 [#<procedure 9ff7ca0 at ice-9/boot-9.scm:4045:3 ()>]
1724: 16 [%start-stack load-stack ...]
1729: 15 [#<procedure 9ffd708 ()>]
In unknown file:
   ?: 14 [primitive-load "/usr/local/bin/guix"]
In guix/ui.scm:
 818: 13 [run-guix-command substitute-binary "--substitute" ...]
In ice-9/boot-9.scm:
 157: 12 [catch getaddrinfo-error ...]
 157: 11 [catch srfi-34 #<procedure a564530 at guix/ui.scm:220:2 ()> ...]
 157: 10 [catch system-error ...]
In guix/scripts/substitute-binary.scm:
 762: 9 [#<procedure a564540 at guix/scripts/substitute-binary.scm:677:3 ()>]
In guix/utils.scm:
 214: 8 [filtered-port # #<input: r6rs-custom-binary-input-port a5728e8>]
In guix/build/utils.scm:
 575: 7 [#<procedure ae7ab20 at guix/build/utils.scm:573:20 ()>]
In unknown file:
   ?: 6 [get-bytevector-n! #<input: r6rs-custom-binary-input-port a5728e8> ...]
In guix/scripts/substitute-binary.scm:
 517: 5 [read! #vu8(153 67 181 183 33 154 71 141 116 19 119 200 170 61 233 ... 225 243 29 228 124 174 111 40 123 94 54 250 82 79 143 24 163 120 30) ...]
In unknown file:
   ?: 4 [get-bytevector-n! #<input: r6rs-custom-binary-input-port a572960> ...]
In web/http.scm:
1936: 3 [read! #vu8(119 208 62 20 255 50 96 203 50 203 70 164 183 47 241 170 205 90 0 134 114 191 241 17 23 13 252 ...  161 54 192 107 98 33 231 144 100 90 118 35 176 54 79 162 225 243 29 228 124 174 111 40 123 94 54 250 82 79 143 24 163 120 30) ...]
1920: 2 [read-chunk #<input-output: socket 10>]
1910: 1 [read-chunk-header #<input-output: socket 10>]
In unknown file:
   ?: 0 [string-index #<eof> #<procedure a52d440 at web/http.scm:1911:44 (c)> ...]

ERROR: In procedure string-index:
ERROR: In procedure string-index: Wrong type argument in position 1 (expecting string): #<eof>

bzip2: Compressed file ends unexpectedly;
	perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
	Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

guix substitute-binary: error: corrupt input while restoring '/gnu/store/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data/texmf-dist/fonts/type1/public/fetamont/ffmho9.pfb' from #{read pipe}#
@ substituter-failed /gnu/store/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data 256 fetching path `/gnu/store/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data' failed with exit code 1
killing process 25165
guix build: error: build failed: some substitutes for the outputs of derivation `/gnu/store/627fwfqdpv85q6j7dvcjwnmi6gfwy2ll-texlive-2014.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

bug#19976: Cannot install texlive

[Ludovic Courtès]

merge 19939 19976
thanks

Ricardo Wurmus <rekado@elephly.net> skribis:

> found valid signature for '/gnu/store/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data', from 'http://hydra.gnu.org/nar/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data'
> downloading `/gnu/store/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data' (3392.6 MiB installed)...
> GC Warning: Failed to expand heap by 1986744320 bytes
> GC Warning: Failed to expand heap by 1986220032 bytes
> GC Warning: Out of Memory! Heap size: 6 MiB. Returning NULL!

When running:

  strace -s 234 -o t wget -O /dev/null \
     http://hydra.gnu.org/nar/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data

we see:

  write(5, "GET /nar/6rz8gpy7ax6334ihw87zyaxyibz96hc3-texlive-2014-data HTTP/1.1\r\nUser-Agent: Wget/1.16.1 (linux-gnu)\r\nAccept: */*\r\nAccept-Encoding: identity\r\nHost: hydra.gnu.org\r\nConnection: Keep-Alive\r\n\r\n", 194) = 194

  [...]

  recvfrom(5, "HTTP/1.1 200 OK\r\nServer: nginx/1.6.2\r\nDate: Sun, 01 Mar 2015 20:41:09 GMT\r\nContent-Type: application/x-nix-archive\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\n\r\n76634478\r\nBZh91AY&SYDH\330\5\0S\n\177\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377! \0\2\345\10\0\20\10\0\210t\204\260\320'c\264="..., 511, MSG_PEEK, NULL, NULL) = 511

  [...]

  read(5, "76634478\r\n", 10)             = 10

This is telling us that this is chunked encoding, and that the first
chunk’s size is #x76634478 B = 1986217080 B = 1.9 GiB.  Ouch!

You could buy more RAM.  ;-)  We’ll fix <http://bugs.gnu.org/19939>, see
whether we can backport it in Guix, and also see if we can ask nginx to
provide reasonable chunks.

Thanks,
Ludo’.

bug#19976: Cannot install texlive

[Mark H Weaver]

ludo@gnu.org (Ludovic Courtès) writes:

> This is telling us that this is chunked encoding, and that the first
> chunk’s size is #x76634478 B = 1986217080 B = 1.9 GiB.  Ouch!
>
> You could buy more RAM.  ;-)

Or add more swap.  In this case, although the memory needs to be
allocated, but it is not under active use so swap would work well.

On the other hand, on a 32-bit system, depending on the policy for
expanding/copying the receive buffer, we might be hitting the 3GB(?)
virtual memory limit.

> We’ll fix <http://bugs.gnu.org/19939>, see whether we can backport it
> in Guix, and also see if we can ask nginx to provide reasonable
> chunks.

'sendfile_max_chunk' might be the right knob for this.

http://nginx.org/en/docs/http/ngx_http_core_module.html#sendfile_max_chunk

     Mark

bug#19976: Cannot install texlive

[Mark H Weaver]

Mark H Weaver <mhw@netris.org> writes:

> ludo@gnu.org (Ludovic Courtès) writes:
>
>> We’ll fix <http://bugs.gnu.org/19939>, see whether we can backport it
>> in Guix, and also see if we can ask nginx to provide reasonable
>> chunks.
>
> 'sendfile_max_chunk' might be the right knob for this.
>
> http://nginx.org/en/docs/http/ngx_http_core_module.html#sendfile_max_chunk

Alas, this didn't seem to have any effect, and I haven't been able to
find another relevant knob in the docs.  Maybe it's time to look into
what we'd have to fix in Guile...

       Mark

bug#19976: Cannot install texlive

[Ludovic Courtès]

Commits 776463b and c28606b backport the Guile fix to (guix
http-client), which is what ‘guix substitute-binary’ uses.

You can check that it solves the initial problem you reported by running
the daemon with the new substitute code:

  ./pre-inst-env guix-daemon --build-users-group=guixbuild

Thanks,
Ludo’.