bug#46877: Guix assumes ideal network exists, does not

bug#46877: Guix assumes ideal network exists, does not

[Tobias Geerinckx-Rice via Bug reports for GNU Guix]

[-- Attachment #1: Type: text/plain, Size: 2177 bytes --]

Guixfriends,

Errors like the one below are omnipresent.  It's a problem.

Expecting an ever-increasing (16k+?) number of requests to finish 
flawlessly or aborting a very long (20+ minutes here) process is 
doomed.

We don't seem to have a plan forward (I include myself!).  Looking 
at other package managers, a blunt ‘retry a finite number of 
times’ seems common.  Guix's ‘pretend that "transient" is a magic 
powerword’ approach less so.

What do people think?  Does the heavy pipelining complicate 
matters?

--8<---------------cut here---------------start------------->8---
~ $ guix weather --substitute-urls=https://guix.tobias.gr
computing 16,745 package derivations for x86_64-linux...
looking for 18,127 store items on https://guix.tobias.gr...
updating substitutes from 'https://guix.tobias.gr'... 
41.6%Backtrace:
          11 (primitive-load 
          "/home/nckx/.config/guix/current/bin/guix")
In guix/ui.scm:
  2164:12 10 (run-guix-command _ . _)
In ice-9/boot-9.scm:
  1736:10  9 (with-exception-handler _ _ #:unwind? _ 
  #:unwind-for-type _)
  1731:15  8 (with-exception-handler #<procedure 7fe8a173d7e0 at 
  ice-9/boot-9.scm:1815:7 (exn)> _ #:unwind? _ …)
In guix/scripts/weather.scm:
    546:9  7 (_)
In guix/build/utils.scm:
   569:23  6 (every* #<procedure 7fe875fdd4e0 at 
   guix/scripts/weather.scm:546:17 (server)> _)
In guix/scripts/weather.scm:
   547:19  5 (_ "https://guix.tobias.gr")
   116:17  4 (report-server-coverage _ _ #:display-missing? _)
In unknown file:
           3 (_ #<procedure 7fe86e3cbbe0 at 
           guix/scripts/weather.scm:184:2 ()> #<procedure list _> 
           . #w())
In guix/scripts/substitute.scm:
   411:23  2 (lookup-narinfos _ _ #:open-connection _)
   371:26  1 (fetch-narinfos _ _ #:open-connection _)
In ice-9/boot-9.scm:
  1669:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1669:16: In procedure raise-exception:
Throw to key `gnutls-error' with args `(#<gnutls-error-enum Error 
in the pull function.> read_from_session_record_port)'.
--8<---------------cut here---------------end--------------->8---

Kind regards,

T G-R

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 247 bytes --]

bug#46877: Guix assumes ideal network exists, does not

[Leo Famulari]

On Tue, Mar 02, 2021 at 05:32:50PM +0100, Tobias Geerinckx-Rice via Bug reports for GNU Guix wrote:
> Guixfriends,
> 
> Errors like the one below are omnipresent.  It's a problem.
> 
> Expecting an ever-increasing (16k+?) number of requests to finish flawlessly
> or aborting a very long (20+ minutes here) process is doomed.
> 
> We don't seem to have a plan forward (I include myself!).  Looking at other
> package managers, a blunt ‘retry a finite number of times’ seems common.
> Guix's ‘pretend that "transient" is a magic powerword’ approach less so.
> 
> What do people think?  Does the heavy pipelining complicate matters?
> 
> --8<---------------cut here---------------start------------->8---
> ~ $ guix weather --substitute-urls=https://guix.tobias.gr

I sometimes see similar failures with `guix weather`.

To clarify, do you also see them when substituting? Or just when
checking for substitutes?

bug#46877: Guix assumes ideal network exists, does not

[zimoun]

Hi Tobias,

On Tue, 2 Mar 2021 at 17:33, Tobias Geerinckx-Rice via Bug reports for
GNU Guix <bug-guix@gnu.org> wrote:

> We don't seem to have a plan forward (I include myself!).  Looking
> at other package managers, a blunt ‘retry a finite number of
> times’ seems common.  Guix's ‘pretend that "transient" is a magic
> powerword’ approach less so.

If I read correctly, the issue comes from (guix scripts substitutes)
in 'fetch-narinfos':

--8<---------------cut here---------------start------------->8---
                          (call-with-connection-error-handling
                           uri
                           (lambda ()
                             (http-multiple-get uri
                                                handle-narinfo-response '()
                                                requests
                                                #:open-connection
open-connection
                                                #:verify-certificate? #f))))))
--8<---------------cut here---------------end--------------->8---

where 'requests' can be really large (all the missing information from
local cache).

Instead of processing the list in one go, we could imagine split it
into several chunks, say X, and so process only X requests by X
requests.

WDYT?


Cheers,
simon

bug#46877: Guix assumes ideal network exists, does not

[zimoun]

On Tue, 2 Mar 2021 at 21:00, zimoun <zimon.toutoune@gmail.com> wrote:

> Instead of processing the list in one go, we could imagine split it
> into several chunks, say X, and so process only X requests by X
> requests.

My bad, it is already the case where X=1000 in (guix http-client).
Well, the error / retry seems poorly handled in 'http-multiple-get'.

Cheers,
simon

bug#46877: Guix assumes ideal network exists, does not

[Ludovic Courtès]

Hi,

Tobias Geerinckx-Rice via Bug reports for GNU Guix <bug-guix@gnu.org>
skribis:

> ~ $ guix weather --substitute-urls=https://guix.tobias.gr
> computing 16,745 package derivations for x86_64-linux...
> looking for 18,127 store items on https://guix.tobias.gr...
> updating substitutes from 'https://guix.tobias.gr'... 
> 41.6%Backtrace:
>           11 (primitive-load 
>           "/home/nckx/.config/guix/current/bin/guix")
> In guix/ui.scm:
>   2164:12 10 (run-guix-command _ . _)
> In ice-9/boot-9.scm:
>   1736:10  9 (with-exception-handler _ _ #:unwind? _ 
>   #:unwind-for-type _)
>   1731:15  8 (with-exception-handler #<procedure 7fe8a173d7e0 at 
>   ice-9/boot-9.scm:1815:7 (exn)> _ #:unwind? _ …)
> In guix/scripts/weather.scm:
>     546:9  7 (_)
> In guix/build/utils.scm:
>    569:23  6 (every* #<procedure 7fe875fdd4e0 at 
>    guix/scripts/weather.scm:546:17 (server)> _)
> In guix/scripts/weather.scm:
>    547:19  5 (_ "https://guix.tobias.gr")
>    116:17  4 (report-server-coverage _ _ #:display-missing? _)
> In unknown file:
>            3 (_ #<procedure 7fe86e3cbbe0 at 
>            guix/scripts/weather.scm:184:2 ()> #<procedure list _> 
>            . #w())
> In guix/scripts/substitute.scm:
>    411:23  2 (lookup-narinfos _ _ #:open-connection _)
>    371:26  1 (fetch-narinfos _ _ #:open-connection _)
> In ice-9/boot-9.scm:
>   1669:16  0 (raise-exception _ #:continuable? _)
>
> ice-9/boot-9.scm:1669:16: In procedure raise-exception:
> Throw to key `gnutls-error' with args `(#<gnutls-error-enum Error 
> in the pull function.> read_from_session_record_port)'.

Hmm isn’t it a regression?

First, in (guix scripts substitute) there’s a ‘with-networking’ macro to
nicely present networking errors; perhaps we’re missing it here?

Second, is it a genuine bug?  Do you see the same thing from (guix
scripts substitute)?

Thanks,
Ludo’.

bug#46877: Guix assumes ideal network exists, does not

[zimoun]

Hi,

On Wed, 10 Mar 2021 at 12:17, Ludovic Courtès <ludo@gnu.org> wrote:


>> In guix/scripts/substitute.scm:
>>    411:23  2 (lookup-narinfos _ _ #:open-connection _)
>>    371:26  1 (fetch-narinfos _ _ #:open-connection _)
>> In ice-9/boot-9.scm:
>>   1669:16  0 (raise-exception _ #:continuable? _)
>>
>> ice-9/boot-9.scm:1669:16: In procedure raise-exception:
>> Throw to key `gnutls-error' with args `(#<gnutls-error-enum Error 
>> in the pull function.> read_from_session_record_port)'.
>
> Hmm isn’t it a regression?
>
> First, in (guix scripts substitute) there’s a ‘with-networking’ macro to
> nicely present networking errors; perhaps we’re missing it here?
>
> Second, is it a genuine bug?  Do you see the same thing from (guix
> scripts substitute)?

I do not know if it is related, the bug#47055 seems “similar”:

<http://issues.guix.gnu.org/issue/47055>


Cheers,
simon