bug#26811: guix does not resume download on FTP timeouts

bug#26811: guix does not resume download on FTP timeouts

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 6117 bytes --]

I have slow internet :)

It looks like ftp://tug.org has a FTP session timeout of 2 hours 11 minutes, or something like that.
Over night I tried to build sbcl and failed. Note how download silently stopped at ~80% [1].

Would be nice if guix had a mechanism to reconnect to FTP and continue download after timeout.

Thanks!

[1]: $ while :; do ./pre-inst-env guix build sbcl --no-substitutes; done

@ build-started /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - x86_64-linux /var/log/guix/drvs/yw//pxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv.bz2

Starting download of /gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz
From ftp://tug.org/historic/systems/texlive/2016/texlive-20160523b-texmf.tar.xz...
 …3b-texmf.tar.xz  1.93GiB       213KiB/s 02:11:32 [################    ]  83.0%
output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `062qbbhlin1ys9lihax34kq503ig1fl55br84j94vwawbk04p6j0'
@ build-failed /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - 1 output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `062qbbhlin1ys9lihax34kq503ig1fl55br84j94vwawbk04p6j0'
@ build-started /gnu/store/in6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv - x86_64-linux /var/log/guix/drvs/in//6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv.bz2
cannot build derivation `/gnu/store/q9s2wdsaqvrllkcsf8w1a8424zd9ybc1-texlive-20160523b-texmf.tar.xz.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/b7abk0aarqv0ylxp2g4s2sf9j75xdy04-texlive-texmf-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/kpjy0c1pl8vzxbkys6p45ij3za6a92gj-texlive-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv': 1 dependencies couldn't be built
guix build: error: build failed: build of `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv' failed
@ build-started /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - x86_64-linux /var/log/guix/drvs/yw//pxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv.bz2

Starting download of /gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz
From ftp://tug.org/historic/systems/texlive/2016/texlive-20160523b-texmf.tar.xz...
 …3b-texmf.tar.xz  1.93GiB       213KiB/s 02:11:46 [################    ]  83.3%
output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `1l296b4ypphffv1zpamp01g6a633kqlmfbf9l4x9411rlblb1hfd'
@ build-failed /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - 1 output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `1l296b4ypphffv1zpamp01g6a633kqlmfbf9l4x9411rlblb1hfd'
@ build-started /gnu/store/in6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv - x86_64-linux /var/log/guix/drvs/in//6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv.bz2
cannot build derivation `/gnu/store/q9s2wdsaqvrllkcsf8w1a8424zd9ybc1-texlive-20160523b-texmf.tar.xz.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/b7abk0aarqv0ylxp2g4s2sf9j75xdy04-texlive-texmf-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/kpjy0c1pl8vzxbkys6p45ij3za6a92gj-texlive-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv': 1 dependencies couldn't be built
guix build: error: build failed: build of `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv' failed
@ build-started /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - x86_64-linux /var/log/guix/drvs/yw//pxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv.bz2

Starting download of /gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz
From ftp://tug.org/historic/systems/texlive/2016/texlive-20160523b-texmf.tar.xz...
 …3b-texmf.tar.xz  1.93GiB       213KiB/s 02:11:31 [################    ]  83.1%
output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `143zan8sh0lw2jjhdpvc7w9yjh7cw75w1ry60qq4g1n3hjmfrwi3'
@ build-failed /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - 1 output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `143zan8sh0lw2jjhdpvc7w9yjh7cw75w1ry60qq4g1n3hjmfrwi3'
@ build-started /gnu/store/in6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv - x86_64-linux /var/log/guix/drvs/in//6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv.bz2
cannot build derivation `/gnu/store/q9s2wdsaqvrllkcsf8w1a8424zd9ybc1-texlive-20160523b-texmf.tar.xz.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/b7abk0aarqv0ylxp2g4s2sf9j75xdy04-texlive-texmf-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/kpjy0c1pl8vzxbkys6p45ij3za6a92gj-texlive-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv': 1 dependencies couldn't be built
guix build: error: build failed: build of `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv' failed
@ build-started /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - x86_64-linux /var/log/guix/drvs/yw//pxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv.bz2

-- 

  Sergei

[-- Attachment #2: Цифровая подпись OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#26811: guix does not resume download on FTP timeouts

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 665 bytes --]

On Sun, May 07, 2017 at 09:31:59AM +0100, Sergei Trofimovich wrote:
> I have slow internet :)
> 
> It looks like ftp://tug.org has a FTP session timeout of 2 hours 11 minutes, or something like that.
> Over night I tried to build sbcl and failed. Note how download silently stopped at ~80% [1].
> 
> Would be nice if guix had a mechanism to reconnect to FTP and continue download after timeout.

In the meantime, you can download the files with a resumable tool like
wget, and then use `guix download file:///path/to/file` to work around
this missing feature. Note that both the client *and* the server must
support resuming downloads for this to work.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#26811: guix does not resume download on FTP timeouts

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 966 bytes --]

On Sun, 7 May 2017 14:25:43 -0400
Leo Famulari <leo@famulari.name> wrote:

> On Sun, May 07, 2017 at 09:31:59AM +0100, Sergei Trofimovich wrote:
> > I have slow internet :)
> > 
> > It looks like ftp://tug.org has a FTP session timeout of 2 hours 11 minutes, or something like that.
> > Over night I tried to build sbcl and failed. Note how download silently stopped at ~80% [1].
> > 
> > Would be nice if guix had a mechanism to reconnect to FTP and continue download after timeout.  
> 
> In the meantime, you can download the files with a resumable tool like
> wget, and then use `guix download file:///path/to/file` to work around
> this missing feature. Note that both the client *and* the server must
> support resuming downloads for this to work.

The workaround worked.

I've noticed it took guix-daemon 2GB of RAM to process this file locally.
Does guix keep all the file in RAM until download finishes?

Thank you!

-- 

  Sergei

[-- Attachment #2: Цифровая подпись OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

bug#26811: guix does not resume download on FTP timeouts

[Leo Famulari]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

On Sun, May 07, 2017 at 07:49:32PM +0100, Sergei Trofimovich wrote:
> On Sun, 7 May 2017 14:25:43 -0400
> Leo Famulari <leo@famulari.name> wrote:
> 
> > On Sun, May 07, 2017 at 09:31:59AM +0100, Sergei Trofimovich wrote:
> > > I have slow internet :)
> > > 
> > > It looks like ftp://tug.org has a FTP session timeout of 2 hours 11 minutes, or something like that.
> > > Over night I tried to build sbcl and failed. Note how download silently stopped at ~80% [1].
> > > 
> > > Would be nice if guix had a mechanism to reconnect to FTP and continue download after timeout.  
> > 
> > In the meantime, you can download the files with a resumable tool like
> > wget, and then use `guix download file:///path/to/file` to work around
> > this missing feature. Note that both the client *and* the server must
> > support resuming downloads for this to work.
> 
> The workaround worked.
> 
> I've noticed it took guix-daemon 2GB of RAM to process this file locally.
> Does guix keep all the file in RAM until download finishes?

I haven't looked, but that's how I'd expect it be coded. The OS can
handle paging things in and out as needed, right?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

bug#26811: guix does not resume download on FTP timeouts

[Ludovic Courtès]

Sergei Trofimovich <slyfox@inbox.ru> skribis:

> Starting download of /gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz
> From ftp://tug.org/historic/systems/texlive/2016/texlive-20160523b-texmf.tar.xz...

Fundamentally, I think we should just fix our ‘texlive’ package such
that it doesn’t download 4G at once…

Ludo’.