Guile 2.0.5, HTTP downloads, and progress reports

Guile 2.0.5, HTTP downloads, and progress reports

[Ludovic Courtès]

Hello,

I just realized that users of Guile 2.0.5 are observing weird behavior
when substituting binaries: the substituter would remain silent, write
“the server is unresponsive”, then remain silent some more, and finally
write tens of lines of “progress report” in a row when it’s done
downloading.

This is because 2.0.5’s HTTP client would return directly all of the
downloaded data, instead of returning an input port, making it
impossible to implement a progress report.  (guix http-client) hides
that, hence the behavior above.

I really recommend upgrading Guile anyway, since 2.0.5 is 1.5 year old.

We still want to support it though (because some distros still have
that), but I’m not sure we can do much better.  Well, we could do
better by shipping the (web ...) modules from Guile 2.0.9 when we detect
an old Guile, but that sounds overkill to me.

Thoughts?

Ludo’.

Re: Guile 2.0.5, HTTP downloads, and progress reports

[Ludovic Courtès]

ludo@gnu.org (Ludovic Courtès) skribis:

> I just realized that users of Guile 2.0.5 are observing weird behavior
> when substituting binaries: the substituter would remain silent, write
> “the server is unresponsive”, then remain silent some more, and finally
> write tens of lines of “progress report” in a row when it’s done
> downloading.
>
> This is because 2.0.5’s HTTP client would return directly all of the
> downloaded data, instead of returning an input port, making it
> impossible to implement a progress report.  (guix http-client) hides
> that, hence the behavior above.

Commit db90b40 partially addresses this by removing the fake progress
report on Guile 2.0.5, and by printing “please wait” (and “please
upgrade”).

Feedback from 2.0.5 users is welcome!

Ludo’.

Re: Guile 2.0.5, HTTP downloads, and progress reports

[Andreas Enge]

Am Mittwoch, 21. August 2013 schrieb Ludovic Courtès:
> Commit db90b40 partially addresses this by removing the fake progress
> report on Guile 2.0.5, and by printing “please wait” (and “please
> upgrade”).
> 
> Feedback from 2.0.5 users is welcome!

As far as I can tell, nothing changed, I still get the progress report.

Andreas

Re: Guile 2.0.5, HTTP downloads, and progress reports

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

Andreas Enge <andreas@enge.fr> skribis:

> Am Mittwoch, 21. August 2013 schrieb Ludovic Courtès:
>> Commit db90b40 partially addresses this by removing the fake progress
>> report on Guile 2.0.5, and by printing “please wait” (and “please
>> upgrade”).
>> 
>> Feedback from 2.0.5 users is welcome!
>
> As far as I can tell, nothing changed, I still get the progress report.

What do the following print?

  guile -c '(use-modules (guix utils)) (pk (version>? (version) "2.0.5"))'
  guile -c '(pk (version))'

My guess is that (version) return "2.0.5-foobar-deb", which defeats the
test above.

If that is the case, that patch should solve the problem:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 820 bytes --]

diff --git a/guix/scripts/substitute-binary.scm b/guix/scripts/substitute-binary.scm
index 0fdec45..bfaebee 100755
--- a/guix/scripts/substitute-binary.scm
+++ b/guix/scripts/substitute-binary.scm
@@ -418,7 +418,10 @@ PORT.  REPORT-PROGRESS is a two-argument procedure such as that returned by
 
   ;; Since `http-fetch' in Guile 2.0.5 returns all the data once it's done,
   ;; don't pretend to report any progress in that case.
-  (if (version>? (version) "2.0.5")
+  (if (version>? (string-append (major-version) "."
+                                (minor-version) "."
+                                (micro-version))
+                 "2.0.5")
       (make-custom-binary-input-port "progress-port-proc"
                                      read! #f #f
                                      (cut close-port port))

[-- Attachment #3: Type: text/plain, Size: 52 bytes --]


Could you try and report back?

TIA,
Ludo’.

Re: Guile 2.0.5, HTTP downloads, and progress reports

[Andreas Enge]

Am Mittwoch, 21. August 2013 schrieb Ludovic Courtès:
> What do the following print?
> guile -c '(pk (version))'
> My guess is that (version) return "2.0.5-foobar-deb", which defeats the
> test above.

Indeed, this prints
;;; ("2.0.5-deb+1-3")

My report may have been wrong, as it appears that I did not correctly 
install the newest guix version. Anyway, with 
git-911b1b9cb693b59c4001899f144d77c3437c12fe I now cannot download any 
more:

$ git build intltool

The following file will be downloaded:
   /nix/store/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-intltool-0.50.2
@ substituter-started /nix/store/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2 /usr/local/guix-git/libexec/guix/substitute-binary
downloading `/nix/store/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-intltool-0.50.2' 
from `http://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2'...
http://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-intltool-0.50.2          
4.0 KiBhttp://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2          8.0 KiB transferredBacktrace:
In ice-9/boot-9.scm:
http://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-intltool-0.50.2         
12.0 KiBhttp://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2         16.0 
KiBhttp://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2         20.0 
KiBhttp://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2         24.0 
KiBhttp://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2         28.0 
KiBhttp://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2         32.0 
KiBhttp://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2         34.4 
KiBhttp://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2         34.4 
KiBhttp://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2         34.4 
KiBhttp://hydra.gnu.org/nar/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2         34.4 KiB transferred 149: 12 [catch #t #<catch-
closure 92d1a0> ...]
 157: 11 Exception thrown while printing backtrace:
ERROR: In procedure set-current-output-port: Wrong type argument in 
position 1 (expecting open output port): #<closed: file 0>

ERROR: In procedure execl:
ERROR: In procedure execl: Datei oder Verzeichnis nicht gefunden
Backtrace:
In ice-9/boot-9.scm:
 149: 14 [catch #t #<catch-closure 92d1a0> ...]
 157: 13 [#<procedure 8cb0f0 ()>]
In unknown file:
   ?: 12 [catch-closure]
In ice-9/boot-9.scm:
  63: 11 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
 407: 10 [eval # #]
In ice-9/boot-9.scm:
2111: 9 [save-module-excursion #<procedure 8d0100 at 
ice-9/boot-9.scm:3646:3 ()>]
3651: 8 [#<procedure 8d0100 at ice-9/boot-9.scm:3646:3 ()>]
In unknown file:
   ?: 7 [load-compiled/vm "/root/.cache/guile/ccache/2.0-
LE-8-2.0/usr/local/guix-git/bin/guix.go"]
In guix/ui.scm:
 475: 6 [run-guix-command substitute-binary "--substitute" ...]
In ice-9/boot-9.scm:
 149: 5 [catch getaddrinfo-error ...]
 157: 4 [#<procedure c62370 ()>]
In guix/scripts/substitute-binary.scm:
 532: 3 [#<procedure d81400 at guix/scripts/substitute-binary.scm:457:2 
()>]
In guix/nar.scm:
 177: 2 [restore-file #<input: #{read pipe}# 10> ...]
In guix/serialization.scm:
  77: 1 [read-string #<input: #{read pipe}# 10>]
  49: 0 [read-int #<input: #{read pipe}# 10>]

guix/serialization.scm:49:4: In procedure read-int:
guix/serialization.scm:49:4: In procedure bv-u32-ref: Wrong type argument 
in position 1 (expecting bytevector): #<eof>
@ substituter-failed /nix/store/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
intltool-0.50.2 256 fetching path 
`/nix/store/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-intltool-0.50.2' failed with 
exit code 1
guix build: error: build failed: some substitutes for the outputs of 
derivation `/nix/store/ag99pv3rjk6bjf3fblnx2jrvmg6q7a96-
intltool-0.50.2.drv' failed; try `--fallback'


The file itself can be downloaded without problem with wget.

After applying your patch, the progress report is indeed suppressed, but 
the download problem persists.

Andreas

Re: Guile 2.0.5, HTTP downloads, and progress reports

[Ludovic Courtès]

Andreas Enge <andreas@enge.fr> skribis:

> Am Mittwoch, 21. August 2013 schrieb Ludovic Courtès:
>> What do the following print?
>> guile -c '(pk (version))'
>> My guess is that (version) return "2.0.5-foobar-deb", which defeats the
>> test above.
>
> Indeed, this prints
> ;;; ("2.0.5-deb+1-3")

OK, I committed something similar to what I posted.

> $ git build intltool
>
> The following file will be downloaded:
>    /nix/store/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-intltool-0.50.2
> @ substituter-started /nix/store/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-

[...]

> In guix/scripts/substitute-binary.scm:
>  532: 3 [#<procedure d81400 at guix/scripts/substitute-binary.scm:457:2 
> ()>]
> In guix/nar.scm:
>  177: 2 [restore-file #<input: #{read pipe}# 10> ...]
> In guix/serialization.scm:
>   77: 1 [read-string #<input: #{read pipe}# 10>]
>   49: 0 [read-int #<input: #{read pipe}# 10>]
>
> guix/serialization.scm:49:4: In procedure read-int:
> guix/serialization.scm:49:4: In procedure bv-u32-ref: Wrong type argument 
> in position 1 (expecting bytevector): #<eof>
> @ substituter-failed /nix/store/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-
> intltool-0.50.2 256 fetching path 
> `/nix/store/khmpbvi9ivsfhhxz3a7k7cy7vkadczdp-intltool-0.50.2' failed with 

So commit b6952ca fixes what I thought was the source of the bug, but
according to our last discussion IRL, it seems not.  We’ll keep
investigating...

Thanks,
Ludo’, live from the GHM.  :-)