From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Christopher Baines Newsgroups: gmane.lisp.guile.user Subject: Making HTTP requests over TLS from multiple threads Date: Thu, 18 Mar 2021 09:27:35 +0000 Message-ID: <87r1kcdd6g.fsf@cbaines.net> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="3291"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: mu4e 1.4.15; emacs 27.1 To: guile-user@gnu.org Original-X-From: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Thu Mar 18 10:28:35 2021 Return-path: Envelope-to: guile-user@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lMoxH-0000kK-8Y for guile-user@m.gmane-mx.org; Thu, 18 Mar 2021 10:28:35 +0100 Original-Received: from localhost ([::1]:34006 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMoxG-0003nf-6G for guile-user@m.gmane-mx.org; Thu, 18 Mar 2021 05:28:34 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:37658) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lMowa-0003mR-G8 for guile-user@gnu.org; Thu, 18 Mar 2021 05:27:52 -0400 Original-Received: from mira.cbaines.net ([212.71.252.8]:56486) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMowN-0001Xi-Ic for guile-user@gnu.org; Thu, 18 Mar 2021 05:27:52 -0400 Original-Received: from localhost (unknown [IPv6:2a02:8010:68c1:0:8ac0:b4c7:f5c8:7caa]) by mira.cbaines.net (Postfix) with ESMTPSA id B975D27BC55 for ; Thu, 18 Mar 2021 09:27:37 +0000 (GMT) Original-Received: from capella (localhost [127.0.0.1]) by localhost (OpenSMTPD) with ESMTP id 540d9f75 for ; Thu, 18 Mar 2021 09:27:37 +0000 (UTC) Received-SPF: pass client-ip=212.71.252.8; envelope-from=mail@cbaines.net; helo=mira.cbaines.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane-mx.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.io gmane.lisp.guile.user:17368 Archived-At: --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain Hey, I think I'm having some issues with making HTTP requests over TLS with Guile, I'm using the open-socket-for-uri procedure from (web client) for the port. I've attached a small test program that reliably reproduces the issue for me, and I've included the output I typically get below [1]. At the start, a few requests are made from a single thread, and that works fine. As soon as there are two threads though, some uses of the port raise an error/again exception, at which point the program waits a second and retries (I guess that's sensible). Other errors that don't look more serious then happen. I've read the GnuTLS and GnuTLS Guile documentation, but the only things I've found that look relevant are suggestions in the GnuTLS documentation that using multiple threads should be OK. I have tried copying large amounts of the relevant Guile procedures so that I can access and tweak the GnuTLS related code, but that didn't reveal anything obvious to me at least. One other interesting thing is that if the different threads connect to different sites, that doesn't seem to break. Could there be some shared state somewhere for the connections that's leading to things going wrong? Thanks, Chris --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=threaded-https-connections-test.scm (use-modules (web uri) (web request) (web response) (web client) (web http) (srfi srfi-1) (ice-9 threads) (ice-9 match) (rnrs bytevectors) (srfi srfi-11) (srfi srfi-9) (srfi srfi-9 gnu) (srfi srfi-26) (gnutls) (ice-9 binary-ports) ((ice-9 ftw) #:select (scandir)) ((rnrs io ports) #:prefix rnrs-ports:)) (define* (call-with-streaming-http-request uri callback #:key (headers '())) (let* ((port (open-socket-for-uri uri)) (request (build-request uri #:method 'PUT #:version '(1 . 1) #:headers `((connection close) (Transfer-Encoding . "chunked") (Content-Type . "application/octet-stream") ,@headers) #:port port))) (set-port-encoding! port "ISO-8859-1") (let ((request (write-request request port))) (let ((chunked-output-port (make-chunked-output-port port #:buffering 128 #:keep-alive? #t))) ;; A SIGPIPE will kill Guile, so ignore it (sigaction SIGPIPE (lambda (arg) (simple-format (current-error-port) "warning: SIGPIPE\n"))) (set-port-encoding! chunked-output-port "ISO-8859-1") (callback chunked-output-port) (retry-gnutls-resource-temporarily-unavailable (lambda () (close-port chunked-output-port))) (display "\r\n" port) (force-output port)) (let ((response (read-response port))) (let ((body (read-response-body response))) (close-port port) (values response body)))))) (define (retry-gnutls-resource-temporarily-unavailable thunk) (catch 'gnutls-error thunk (lambda (key err proc . rest) (if (eq? error/again err) (begin (simple-format (current-error-port) "error/again\n") (sleep 1) (thunk)) (throw key (cons* err proc rest)))))) (define (start-thread thread-index) (call-with-new-thread (lambda () (for-each (lambda (request-index) (with-throw-handler #t (lambda () (call-with-streaming-http-request ;; The URL doesn't realy matter as the response to the ;; request doesn't matter. (peek (string->uri (if (= thread-index 1) "https://guix.cbaines.net/test" "https://www.cbaines.net/test"))) (lambda (port) (simple-format (current-error-port) "thread ~A making request\n" thread-index) (let* ((buffer-size 1024) (buffer (make-bytevector buffer-size))) (for-each (lambda (index) ;; (usleep 10) (retry-gnutls-resource-temporarily-unavailable (lambda () (put-bytevector port buffer 0 buffer-size)))) (iota 512)))))) (lambda (key . args) (simple-format #t "thread ~A: exception: ~A ~A\n" thread-index key args) (backtrace)))) (iota 2 1))))) ;; (define threads ;; (list (start-thread 1))) ;; (for-each join-thread threads) ;; (define threads ;; (list (start-thread 1))) ;; (for-each join-thread threads) ;; (define threads ;; (list (start-thread 1))) ;; (for-each join-thread threads) ;; (simple-format (current-error-port) ;; "\ntrying concurrent threads\n\n") (define threads (map start-thread (iota 2 1))) (for-each join-thread threads) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable 1: thread 1 making request thread 1 making request thread 1 making request thread 1 making request thread 1 making request thread 1 making request trying concurrent threads thread 1 making request thread 2 making request error/again error/again error/again thread 1: exception: gnutls-error (# write_to_session_record_port) Backtrace: In srfi/srfi-1.scm: 634:9 12 (for-each # =E2=80=A6) In ice-9/boot-9.scm: 1736:10 11 (with-exception-handler _ _ #:unwind? _ # _) In ice-9/eval.scm: 619:8 10 (_ #(#(#(#(#(#) =E2=80= =A6) =E2=80=A6) =E2=80=A6) =E2=80=A6)) 619:8 9 (_ #(#(#(#(#(#(#) =E2=80=A6) = =E2=80=A6) =E2=80=A6) =E2=80=A6) =E2=80=A6)) In srfi/srfi-1.scm: 634:9 8 (for-each # =E2=80=A6) In unknown file: 7 (put-bytevector # #vu8(0 =E2=80= =A6) =E2=80=A6) In web/http.scm: 2029:35 6 (flush) In unknown file: 5 (put-char # #\nul) In web/client.scm: 267:8 4 (write! #vu8(0 53 55 97 13 10 0 0 0 0 0 0 0 0 0 0 0 0 =E2=80= =A6) =E2=80=A6) In unknown file: 3 (put-bytevector # # 0 =E2= =80=A6) In ice-9/boot-9.scm: 1669:16 2 (raise-exception _ #:continuable? _) 1764:13 1 (_ #<&compound-exception components: (#<&error> #<&irri=E2=80= =A6>) In unknown file: 0 (backtrace #) In srfi/srfi-1.scm: 634:9 11 (for-each # =E2=80=A6) In ice-9/boot-9.scm: 1736:10 10 (with-exception-handler _ _ #:unwind? _ # _) In ice-9/eval.scm: 619:8 9 (_ #(#(#(#(#(#) =E2=80= =A6) =E2=80=A6) =E2=80=A6) =E2=80=A6)) 619:8 8 (_ #(#(#(#(#(#(#) =E2=80=A6) = =E2=80=A6) =E2=80=A6) =E2=80=A6) =E2=80=A6)) In srfi/srfi-1.scm: 634:9 7 (for-each # =E2=80=A6) In unknown file: 6 (put-bytevector # #vu8(0 =E2=80= =A6) =E2=80=A6) In web/http.scm: 2029:35 5 (flush) In unknown file: 4 (put-char # #\nul) In web/client.scm: 267:8 3 (write! _ _ 1024) In unknown file: 2 (put-bytevector # # 0 =E2= =80=A6) In ice-9/boot-9.scm: 1669:16 1 (raise-exception _ #:continuable? _) 1669:16 0 (raise-exception _ #:continuable? _) ice-9/boot-9.scm:1669:16: In procedure raise-exception: Throw to key `gnutls-error' with args `(# write_to_session_record_port)'. warning: SIGPIPE thread 2 making request In procedure write_to_session_record_port: Wrong type argument in position = 1: # --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmBTHQdfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh aW5lcy5uZXQACgkQXiijOwuE9XfB8Q/9FN/m4ZBsQnBC5P7cp1C5Pnuo0I+GS1x8 Gt6kWF83c5+5jzs4A49pWCGG/hcJwZVA8ZaA00/HdFYTB1bPwijNvX4UdAMAyzZX GcHlu5Da0gpAATuBsLQxGx/zuCH7dqB9Lq9AYAuLyzx8CD/efoVv/CINwluM64l2 vp4zHcts11TJdcj4ppC+A+LiERaGuiDjuZIWWtwT33E6bpjIMB00D84jl2cTQ4xT BJEk2Zs9dra5PkRc3MK3FOEsJCi0xJr6RZqsdsVkmt27W42Ej00MOROmqBVMbImn c5S3bMxFjI+hLi5sidEU6HxrypNwlwV1ra2YrrwYN1g3O6oDbyeyKnrOhwTzA7iP AZFaKNcvM241pxrMiVzeQid3P3me/OZ8dBhynTQMVkiwz3CQsN/Kf1hqxQFwVUnl yLaLuRxPm8CB7GXOIJZ4h3rQdlHKORzOQWh0mr3Mpd/jRfhcBJmrZq9nguOVlBOK nnUr/7+Xn8aj3exhVYJnf3GBgXFUjGu2U10zecXQIUtXq4z7CQ9ob/Mu8vWg4Lwt TuNfzFBCaeOasqZMcV+dmktUfFy88F5ZUPBZDrUnWoX9kZaLhqoJj11KDXNNVVa6 EqtfjcoDOJzZcVs4QnOPEXcdPYJGrNId0Cpm05Eytw82edNyZsPmDU5cD7KMwrrS NjDlKqWyTYY= =2dxX -----END PGP SIGNATURE----- --==-=-=--