* bug#22937: guix package fails when --substitute-urls specifies an HTTPS endpoint @ 2016-03-08 6:16 Chris Marusich 2016-03-09 22:51 ` Ludovic Courtès 0 siblings, 1 reply; 5+ messages in thread From: Chris Marusich @ 2016-03-08 6:16 UTC (permalink / raw) To: 22937 Hi, I've noticed that "guix package" fails when I specify an HTTPS endpoint for --substitute-urls. Is that expected behavior? I recently set up a caching proxy for hydra.gnu.org. The endpoint's DNS name is hydra-mirror.marusich.info. It's configured to accept both HTTP and HTTPS requests. When it receives an HTTPS request, it proxies the request (assuming it's a cache miss) to hydra.gnu.org over HTTP. I've configured it this way because my understanding is that hydra.gnu.org is currently only available via HTTP. Commands like wget, curl, and even "guix download" work fine with the mirror, even over HTTPS. For example, the following command succeeds: guix download https://hydra-mirror.marusich.info/nar/8kvb2k0n1jqjd5sa7g6qj5qllq0ckcya-linux-libre-4.4 However, when I try to install a package with "guix package" using the endpoint, it fails with a backtrace like the following (this command was invoked using the "guix package" from commit 7b3f2682de38a8e39f052705795ec85fcdfc8a96): --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix package --substitute-urls="https://hydra-mirror.marusich.info" -i graphviz substitute: Backtrace: substitute: In ice-9/boot-9.scm: substitute: 63: 19 [call-with-prompt prompt0 ...] substitute: In ice-9/eval.scm: substitute: 432: 18 [eval # #] substitute: In ice-9/boot-9.scm: substitute: 2401: 17 [save-module-excursion #<procedure 16ac940 at ice-9/boot-9.scm:4045:3 ()>] substitute: 4050: 16 [#<procedure 16ac940 at ice-9/boot-9.scm:4045:3 ()>] substitute: 1724: 15 [%start-stack load-stack ...] substitute: 1729: 14 [#<procedure 16c4ea0 ()>] substitute: In unknown file: substitute: ?: 13 [primitive-load "/gnu/store/3lg5c1nidbj0kjdz5b63hn3vp29kzf0s-guix-0.9.0.c3f29bc/bin/.guix-real"] substitute: In guix/ui.scm: substitute: 1175: 12 [run-guix-command substitute "--query"] substitute: In ice-9/boot-9.scm: substitute: 157: 11 [catch getaddrinfo-error ...] substitute: 157: 10 [catch srfi-34 #<procedure 25850a0 at guix/ui.scm:411:2 ()> ...] substitute: 157: 9 [catch system-error ...] substitute: In guix/scripts/substitute.scm: substitute: 946: 8 [#<procedure 25850c0 at guix/scripts/substitute.scm:939:3 ()>] substitute: 804: 7 [process-query "info /gnu/store/x29dbk9inqjym79q0907sx4arp1bfp28-graphviz-2.38.0-doc " ...] substitute: 633: 6 [lookup-narinfos/diverse # #] substitute: 617: 5 [lookup-narinfos "https://hydra-mirror.marusich.info" #] substitute: 589: 4 [fetch-narinfos "https://hydra-mirror.marusich.info" #] substitute: 222: 3 [download-cache-info "https://hydra-mirror.marusich.info"] substitute: In guix/records.scm: substitute: 331: 2 [recutils->alist #<unspecified>] substitute: In ice-9/rdelim.scm: substitute: 184: 1 [read-line #<unspecified> trim] substitute: In unknown file: substitute: ?: 0 [%read-line #<unspecified>] substitute: substitute: ERROR: In procedure %read-line: substitute: ERROR: In procedure %read-line: Wrong type argument in position 1 (expecting open input port): #<unspecified> guix package: error: build failed: substituter `substitute' died unexpectedly --8<---------------cut here---------------end--------------->8--- When I replaced --substitute-urls="https://hydra-mirror.marusich.info" with --substitute-urls="http://hydra-mirror.marusich.info", the same command worked fine. Chris ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#22937: guix package fails when --substitute-urls specifies an HTTPS endpoint 2016-03-08 6:16 bug#22937: guix package fails when --substitute-urls specifies an HTTPS endpoint Chris Marusich @ 2016-03-09 22:51 ` Ludovic Courtès 2016-03-10 13:08 ` Ludovic Courtès 0 siblings, 1 reply; 5+ messages in thread From: Ludovic Courtès @ 2016-03-09 22:51 UTC (permalink / raw) To: Chris Marusich; +Cc: 22937 [-- Attachment #1: Type: text/plain, Size: 1647 bytes --] First, the graceless error handling is fixed in 204d34ff961d6dabf18b255decc29712e03afef0. Second, here’s a preliminary patch that almost works with <https://hydra-mirror.marusich.info>. The problem is that sometimes the server closes the connection unexpectedly, leading to an obscure backtrace like this: --8<---------------cut here---------------start------------->8--- substitute: 629: 6 [lookup-narinfos "https://hydra-mirror.marusich.info" #] substitute: 585: 5 [fetch-narinfos "https://hydra-mirror.marusich.info" #] substitute: 510: 4 [http-multiple-get # ...] substitute: In web/response.scm: substitute: 197: 3 [read-response #<input-output: gnutls-session-port 38f59c0>] substitute: In web/http.scm: substitute: 1157: 2 [read-response-line #<input-output: gnutls-session-port 38f59c0>] substitute: 151: 1 [read-header-line #<input-output: gnutls-session-port 38f59c0>] substitute: In unknown file: substitute: ?: 0 [%read-line #<input-output: gnutls-session-port 38f59c0>] substitute: substitute: ERROR: In procedure %read-line: substitute: ERROR: Throw to key `gnutls-error' with args `(#<gnutls-error-enum Error in the pull function.> fill_session_record_port_input)'. --8<---------------cut here---------------end--------------->8--- The “error in the pull function” is because ‘gnutls_record_recv’ got ECONNRESET while reading. I wonder whether this could be due to the particular configuration of nginx at Cloudfront, so I’ll try with another server (I’ve set up Let’s Encrypt on that server but it’s not accessible yet via port 443.) To be continued! Ludo’. [-- Attachment #2: Type: text/x-patch, Size: 4338 bytes --] diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm index b82fc17..df95de0 100755 --- a/guix/scripts/substitute.scm +++ b/guix/scripts/substitute.scm @@ -32,6 +32,7 @@ #:use-module ((guix build utils) #:select (mkdir-p dump-port)) #:use-module ((guix build download) #:select (progress-proc uri-abbreviation + open-connection-for-uri store-path-abbreviation byte-count->string)) #:use-module (ice-9 rdelim) #:use-module (ice-9 regex) @@ -171,7 +172,7 @@ to the caller without emitting an error message." (let ((port (open-file (uri-path uri) (if buffered? "rb" "r0b")))) (values port (stat:size (stat port))))) - ((http) + ((http https) (guard (c ((http-get-error? c) (let ((code (http-get-error-code c))) (if (and (= code 404) quiet-404?) @@ -201,10 +202,10 @@ to the caller without emitting an error message." (close-port port)))) (begin (when (or (not port) (port-closed? port)) - (set! port (open-socket-for-uri uri)) + (set! port (open-connection-for-uri uri)) (unless buffered? (setvbuf port _IONBF))) - (http-fetch uri #:text? #f #:port port)))))))) + (http-fetch uri #:text? #f #:port port)))))) (else (leave (_ "unsupported substitute URI scheme: ~a~%") (uri->string uri))))) @@ -478,20 +479,26 @@ may be #f, in which case it indicates that PATH is unavailable at CACHE-URL." ".narinfo"))) (build-request (string->uri url) #:method 'GET))) -(define (http-multiple-get base-url proc seed requests) - "Send all of REQUESTS to the server at BASE-URL. Call PROC for each +(define (http-multiple-get base-uri proc seed requests) + "Send all of REQUESTS to the server at BASE-URI. Call PROC for each response, passing it the request object, the response, a port from which to read the response body, and the previous result, starting with SEED, à la 'fold'. Return the final result." (let connect ((requests requests) (result seed)) - ;; (format (current-error-port) "connecting (~a requests left)..." - ;; (length requests)) - (let ((p (open-socket-for-uri base-url))) + (format (current-error-port) "connecting (~a requests left)..." + (length requests)) + (let ((p (open-connection-for-uri base-uri))) + ;; For HTTPS, P is not a file port and does not support 'setvbuf'. + (when (file-port? p) + (setvbuf p _IOFBF (expt 2 16))) + ;; Send all of REQUESTS in a row. - (setvbuf p _IOFBF (expt 2 16)) - (for-each (cut write-request <> p) requests) - (force-output p) + ;; XXX: Do our own caching to work around <http://bugs.gnu.org/22966>. + (let-values (((buffer get) (open-bytevector-output-port))) + (for-each (cut write-request <> buffer) requests) + (put-bytevector p (get)) + (force-output p)) ;; Now start processing responses. (let loop ((requests requests) @@ -501,6 +508,8 @@ read the response body, and the previous result, starting with SEED, à la (reverse result)) ((head tail ...) (let* ((resp (read-response p)) + ;; (xxx (format (current-error-port) + ;; "http response: ~s~%" resp)) (body (response-body-port resp)) (result (proc head resp body result))) ;; The server can choose to stop responding at any time, in which @@ -570,10 +579,10 @@ if file doesn't exist, and the narinfo otherwise." (define (do-fetch uri) (case (and=> uri uri-scheme) - ((http) + ((http https) (let ((requests (map (cut narinfo-request url <>) paths))) (update-progress!) - (let ((result (http-multiple-get url + (let ((result (http-multiple-get uri handle-narinfo-response '() requests))) (newline (current-error-port)) ^ permalink raw reply related [flat|nested] 5+ messages in thread
* bug#22937: guix package fails when --substitute-urls specifies an HTTPS endpoint 2016-03-09 22:51 ` Ludovic Courtès @ 2016-03-10 13:08 ` Ludovic Courtès 2016-03-10 21:21 ` Leo Famulari 0 siblings, 1 reply; 5+ messages in thread From: Ludovic Courtès @ 2016-03-10 13:08 UTC (permalink / raw) To: Chris Marusich; +Cc: 22937-done After some more testing and tweaks, notably with <https://hydra.gnunet.org> and its brand new Let’s Encrypt certificate, I pushed the patch as 9b7bd1b160be7c740a798c09e3b8986368b92036. I can no longer reproduce the ECONNRESET issue I mentioned for <https://hydra-mirror.marusich.info>, but feel free to open a new bug if that comes up again! For the record, the main test I used for that was: $ sudo rm -rf /var/guix/substitute/cache $ guix build libreoffice --no-grafts -n \ --substitute-urls=https://hydra.gnunet.org That forces a redownload of all .narinfo files from hydra.gnunet.org, which involves HTTP pipelining. Anyway, thanks for giving me an incentive to address this. This was long overdue! Ludo’. ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#22937: guix package fails when --substitute-urls specifies an HTTPS endpoint 2016-03-10 13:08 ` Ludovic Courtès @ 2016-03-10 21:21 ` Leo Famulari 2016-03-11 8:39 ` Ludovic Courtès 0 siblings, 1 reply; 5+ messages in thread From: Leo Famulari @ 2016-03-10 21:21 UTC (permalink / raw) To: 22937, ludo, cmmarusich On Thu, Mar 10, 2016 at 02:08:17PM +0100, Ludovic Courtès wrote: > After some more testing and tweaks, notably with > <https://hydra.gnunet.org> and its brand new Let’s Encrypt certificate, > I pushed the patch as 9b7bd1b160be7c740a798c09e3b8986368b92036. With this commit, I still get the backtrace in the original report of this bug. I ran `guix pull` and confirmed the changes were in ~/.config/guix/latest. I cleared the cache at /vat/guix/substitute/cache. Then, I updated --substitute-urls in the daemon invocation and reloaded the daemon, and then did `guix build hello`. Did I miss a step? > > I can no longer reproduce the ECONNRESET issue I mentioned for > <https://hydra-mirror.marusich.info>, but feel free to open a new bug if > that comes up again! > > For the record, the main test I used for that was: > > $ sudo rm -rf /var/guix/substitute/cache > $ guix build libreoffice --no-grafts -n \ > --substitute-urls=https://hydra.gnunet.org > > That forces a redownload of all .narinfo files from hydra.gnunet.org, > which involves HTTP pipelining. > > Anyway, thanks for giving me an incentive to address this. This was > long overdue! > > Ludo’. > > > ^ permalink raw reply [flat|nested] 5+ messages in thread
* bug#22937: guix package fails when --substitute-urls specifies an HTTPS endpoint 2016-03-10 21:21 ` Leo Famulari @ 2016-03-11 8:39 ` Ludovic Courtès 0 siblings, 0 replies; 5+ messages in thread From: Ludovic Courtès @ 2016-03-11 8:39 UTC (permalink / raw) To: Leo Famulari; +Cc: 22937 Leo Famulari <leo@famulari.name> skribis: > On Thu, Mar 10, 2016 at 02:08:17PM +0100, Ludovic Courtès wrote: >> After some more testing and tweaks, notably with >> <https://hydra.gnunet.org> and its brand new Let’s Encrypt certificate, >> I pushed the patch as 9b7bd1b160be7c740a798c09e3b8986368b92036. > > With this commit, I still get the backtrace in the original report of > this bug. You need to make sure that ‘guix-daemon’ launches the right ‘guix substitute’. A simple way to do that from your Git checkout is: sudo ./pre-inst-env guix-daemon --build-users-group=guixbuild \ --substitute-urls=https://mirror.guixsd.org Make sure to stop any running guix-daemon before; on GuixSD, this goes like this: sudo herd stop guix-daemon HTH! Ludo’. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-03-11 8:40 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-03-08 6:16 bug#22937: guix package fails when --substitute-urls specifies an HTTPS endpoint Chris Marusich 2016-03-09 22:51 ` Ludovic Courtès 2016-03-10 13:08 ` Ludovic Courtès 2016-03-10 21:21 ` Leo Famulari 2016-03-11 8:39 ` Ludovic Courtès
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/guix.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).