Guix publish pipe error from #{read pipe}#

Guix publish pipe error from #{read pipe}#

[Pjotr Prins]

On a recent guix-publish server I get many of these

  gzip: stdin: invalid compressed data--format violated
    r-annotationdbi-1.36.0
    1.7MiB/s 00:02 | 3.0MiB transferredguix substitute: error: corrupt
  input while restoring
    '/gnu/store/g33ns71m61zs5sn758smmc4lnv6h4c4q-r-annotationdbi-1.36.0/site-library/AnnotationDbi/extdata/HG-U95Av2_probe_tab.gz' from #{read pipe}#

I don't think this is a guix problem per se. I have a hunch the people
changed the firewall which may cause this.  Anyone any other ideas?

Pj.

Re: Guix publish pipe error from #{read pipe}#

[Andy Wingo]

On Tue 16 May 2017 08:29, Pjotr Prins <pjotr.public12@thebird.nl> writes:

> On a recent guix-publish server I get many of these
>
>   gzip: stdin: invalid compressed data--format violated
>     r-annotationdbi-1.36.0
>     1.7MiB/s 00:02 | 3.0MiB transferredguix substitute: error: corrupt
>   input while restoring
>     '/gnu/store/g33ns71m61zs5sn758smmc4lnv6h4c4q-r-annotationdbi-1.36.0/site-library/AnnotationDbi/extdata/HG-U95Av2_probe_tab.gz' from #{read pipe}#
>
> I don't think this is a guix problem per se. I have a hunch the people
> changed the firewall which may cause this.  Anyone any other ideas?

IME this means that the .nar or whatever was truncated by some
intermediate cache.  Like if you wget the file and try to gunzip it, it
will give the same error AFAIU.

Andy

Re: Guix publish pipe error from #{read pipe}# - what nginx.conf to use?

[Pjotr Prins]

On Tue, May 16, 2017 at 08:29:17AM +0200, Pjotr Prins wrote:
> On a recent guix-publish server I get many of these
> 
>   gzip: stdin: invalid compressed data--format violated
>     r-annotationdbi-1.36.0
>     1.7MiB/s 00:02 | 3.0MiB transferredguix substitute: error: corrupt
>   input while restoring
>     '/gnu/store/g33ns71m61zs5sn758smmc4lnv6h4c4q-r-annotationdbi-1.36.0/site-library/AnnotationDbi/extdata/HG-U95Av2_probe_tab.gz' from #{read pipe}#
> 
> I don't think this is a guix problem per se. I have a hunch the people
> changed the firewall which may cause this.  Anyone any other ideas?

It is an nginx thing. When I go around nginx guix publish works as
expected. Anyone have an example nginx.conf file for guix publish?
Funny thing is that it used to be fine. I think parallel downloads
is messing it up now.

Now we are doing parallel downloads, would it be an idea to do a round
robin on substitute-urls too? Make it optional. May speed downloads up
nicely.

Pj.
-- 

Re: Guix publish pipe error from #{read pipe}# - what nginx.conf to use?

[Ludovic Courtès]

Pjotr Prins <pjotr.public12@thebird.nl> skribis:

> On Tue, May 16, 2017 at 08:29:17AM +0200, Pjotr Prins wrote:
>> On a recent guix-publish server I get many of these
>> 
>>   gzip: stdin: invalid compressed data--format violated
>>     r-annotationdbi-1.36.0
>>     1.7MiB/s 00:02 | 3.0MiB transferredguix substitute: error: corrupt
>>   input while restoring
>>     '/gnu/store/g33ns71m61zs5sn758smmc4lnv6h4c4q-r-annotationdbi-1.36.0/site-library/AnnotationDbi/extdata/HG-U95Av2_probe_tab.gz' from #{read pipe}#
>> 
>> I don't think this is a guix problem per se. I have a hunch the people
>> changed the firewall which may cause this.  Anyone any other ideas?
>
> It is an nginx thing. When I go around nginx guix publish works as
> expected. Anyone have an example nginx.conf file for guix publish?

My guess is that you’re running ‘guix publish’ without --cache.  As a
result, ‘guix publish’ compresses archives on the fly, and thus it is
not able to provide a Content-Length HTTP header.

Consequently, your nginx cache cannot detect truncation.  So if for some
reason (e.g., overloaded machine, ‘guix publish’ restart, etc.), the
connection between nginx and ‘guix publish’ is terminated prematurely,
nginx may end up caching a truncated store.

Also, quoth the manual:

     By default, ‘guix publish’ compresses archives on the fly as it
  serves them.  This “on-the-fly” mode is convenient in that it requires
  no setup and is immediately available.  However, when serving lots of
  clients, we recommend using the ‘--cache’ option, which enables caching
  of the archives before they are sent to clients—see below for details.

HTH!

Ludo’.