From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: thomas@m3y3r.de Newsgroups: gmane.emacs.bugs Subject: bug#32658: gnutls + non-blocking url-retrieve Date: Mon, 01 Oct 2018 22:48:12 +0200 Message-ID: <86h8i5pf5f.fsf@DESKTOP-DQBDJ0U.i-did-not-set--mail-host-address--so-tickle-me> References: <861sa5zmpt.fsf@DESKTOP-DQBDJ0U.i-did-not-set--mail-host-address--so-tickle-me> <86tvm6smax.fsf@DESKTOP-DQBDJ0U.i-did-not-set--mail-host-address--so-tickle-me> <83efda431j.fsf@gnu.org> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: blaine.gmane.org 1538427150 8461 195.159.176.226 (1 Oct 2018 20:52:30 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 1 Oct 2018 20:52:30 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (windows-nt) Cc: thomas@m3y3r.de, 32658@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Mon Oct 01 22:52:26 2018 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g75B3-00026x-EX for geb-bug-gnu-emacs@m.gmane.org; Mon, 01 Oct 2018 22:52:25 +0200 Original-Received: from localhost ([::1]:40615 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g75D9-0006Gf-Vh for geb-bug-gnu-emacs@m.gmane.org; Mon, 01 Oct 2018 16:54:36 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:44348) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1g75Cu-00064T-Lq for bug-gnu-emacs@gnu.org; Mon, 01 Oct 2018 16:54:27 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1g757m-0003g2-MB for bug-gnu-emacs@gnu.org; Mon, 01 Oct 2018 16:49:07 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:57513) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1g757m-0003ft-IU for bug-gnu-emacs@gnu.org; Mon, 01 Oct 2018 16:49:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1g757m-0006nc-DD for bug-gnu-emacs@gnu.org; Mon, 01 Oct 2018 16:49:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: thomas@m3y3r.de Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 01 Oct 2018 20:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 32658 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 32658-submit@debbugs.gnu.org id=B32658.153842690026089 (code B ref 32658); Mon, 01 Oct 2018 20:49:02 +0000 Original-Received: (at 32658) by debbugs.gnu.org; 1 Oct 2018 20:48:20 +0000 Original-Received: from localhost ([127.0.0.1]:33538 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g7575-0006mi-Dd for submit@debbugs.gnu.org; Mon, 01 Oct 2018 16:48:19 -0400 Original-Received: from www17.your-server.de ([213.133.104.17]:57716) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1g7573-0006mV-Sk for 32658@debbugs.gnu.org; Mon, 01 Oct 2018 16:48:18 -0400 Original-Received: from [78.46.172.2] (helo=sslproxy05.your-server.de) by www17.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89_1) (envelope-from ) id 1g756w-00070U-Je; Mon, 01 Oct 2018 22:48:10 +0200 Original-Received: from [2a02:908:4c22:d900:61b1:a3b5:71b6:2d59] (helo=DESKTOP-DQBDJ0U) by sslproxy05.your-server.de with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1g756w-000Vwy-Di; Mon, 01 Oct 2018 22:48:10 +0200 In-Reply-To: <83efda431j.fsf@gnu.org> (Eli Zaretskii's message of "Mon, 01 Oct 2018 09:03:04 +0300") X-Authenticated-Sender: thomas@m3y3r.de X-Virus-Scanned: Clear (ClamAV 0.100.1/24998/Mon Oct 1 10:58:25 2018) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:150888 Archived-At: Eli Zaretskii writes: >> From: thomas@m3y3r.de >> Date: Sun, 30 Sep 2018 23:33:10 +0200 >> >> 1.) I needed to revert to gnutls 3.5.19, the mingw64 build from the >> gitlab ci build seems to have a working gnutls-cli tools on windows 10. >> the gitlab builds for 3.6.3 and 3.6.4 seems to have another bug >> (error code -53) in the gnutls-cli command. >> >> so only gnutls 3.5.19 have a working gnutls-cli. i installed this version in emacs 26.1 >> >> 2.) testing gnutls stream >> using open-gnutls-stream directly gives me a correct tls connection but >> eww still fails to load the site. >> >> when I change url-open-stream in url/url-gw.el to: >> (open-network-stream >> name buffer host service >> :type gw-method >> ;; Use non-blocking socket if we can. >> :nowait nil)) >> >> I finally can open lwn.net in eww. >> >> so something seems to be wrong possible with blocking/non-blocking >> network access. >> >> any ideas? > > Thanks for the info. Hi, thanks for looking into this. > > First, I don't understand what does gnutls-cli have to do with this. okay, thats an easy one to explain. I did download emacs 26.1 from here: http://mirror.netcologne.de/gnu/emacs/windows/emacs-26/emacs-26.1-x86_64.zip in the bin directory there is the gnutls packaged. version is 3.6.0. I wasn't sure where the bug in the TLS problems I see was, so I first tried to use gnutls-cli to check if a connection can be made: C:\Users\thomas\Downloads\emacs-26.1-x86_64\bin>gnutls-cli lwn.net |<1>| There was a non-CA certificate in the trusted list: OU=Copyright (c) 1997 Microsoft Corp.,OU=Microsoft Corporation,CN=Microsoft Root Authority. |<1>| There was a non-CA certificate in the trusted list: C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority. |<1>| There was a non-CA certificate in the trusted list: CN=Root Agency. Processed 59 CA certificate(s). Resolving 'lwn.net:443'... Connecting to '2600:3c03::f03c:91ff:fe61:5c5b:443'... *** Fatal error: Error in the push function. Connecting to '45.33.94.129:443'... *** Fatal error: Error in the push function. Could not connect to 45.33.94.129:443: Bad file descriptor This doesn't work, because of some error -53 (ERROR_BAD_NETPATH). so this is why I did first try to upgrade to gnutls 3.6.3 from the gnutls homepage which is a gitlab ci build, but with no success. then i tried to downgrade the gnutls version to 3.5.19 and there the gnutls-cli tool did work. so gnutls is now able to create an TLS connection. now the question why it doesn't work in emacs. > Emacs on Windows doesn't support TLS connections that use gnutls-cli, > because the way that works, it requires working support for signals, > which cannot happen on Windows. Are you saying that these problems > happen when you use gnutls-cli? If so, please move to the built-in > GnuTLS support, because connections using gnutls-cli are deprecated, > and I see no point in trying to support them on Windows. see above explanation. hopefully this makes it clear. > > Second, I cannot reproduce the problem you are reporting. Using stock > Emacs 26.1 I built myself, with GnuTLS 3.4.15, I have no problems > connecting to lwn.net via eww. I see EAGAIN errors like you do, but > they are non-fatal, so don't prevent the connection from continuing. > It is strange that you are having these problems, but maybe these > problems are specific to GnuTLS 3.6.x? 3.6.x is not a stable branch > of GnuTLS, it could have bugs, in particular bugs specific to Windows. > It is also possible that there are incompatibilities between GnuTLS > 3.6.x and whatever version the Emacs binary you are using was built > against. I do use the emacs 26.1 zip file that is pre-build and linked to from the emacs homepage, see above. > > In this message you say that you downgraded to GnuTLS 3.5.19, but you > didn't show the gnutls.c log for that version -- does it mean you see > an identical problem with EAGAIN there? > > Is it possible for you to downgrade GnuTLS to some version of the > 3.4.x branch, and see if the problem persists? will try out and report. > > Also, does this happen in "emacs -Q"? yes, same error. > > Or maybe this is specific to your network connection? Does any HTTPS > connection cause these problems? no, only some I think. > > Finally, what about other machines and/or Windows versions other than > 10 -- do you have the same problem there with this Emacs version > (assuming you can test that)? I use emacs 26.1 on an linux x86 system and on android arm with termux, both on the same network as the windows laptop, and everything works as expected on these systems. > > Bottom line: I'm surprised that you have these problems, because I see > none of that on my machines -- TLS connections "just work" for me, > without any need to tinker with url-gw.el or elsewhere. And judging > by lack of similar bug reports, this also works for others. So I > wonder what causes this in your case. I do also wonder! here some more details, with vanilla emacs 26.1 + gnutls 3.5.19 and gnutls-log-level 1: 1.) eww lwn.net Contacting host: lwn.net:80 gnutls.c: [1] (Emacs) connecting to host: lwn.net gnutls.c: [1] (Emacs) allocating credentials gnutls.c: [audit] There was a non-CA certificate in the trusted list: OU=Copyright (c) 1997 Microsoft Corp.,OU=Microsoft Corporation,CN=Microsoft Root Authority. gnutls.c: [audit] There was a non-CA certificate in the trusted list: C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority. gnutls.c: [audit] There was a non-CA certificate in the trusted list: CN=Root Agency. gnutls.c: [1] (Emacs) setting the trustfile: C:\Users\thomas\emacs-26.1\etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem gnutls.c: [1] (Emacs) setting the trustfile: C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.crt gnutls.c: [1] (Emacs) setting the trustfile: C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.trust.crt gnutls.c: [1] (Emacs) gnutls callbacks gnutls.c: [1] (Emacs) gnutls_init gnutls.c: [1] (Emacs) got non-default priority string: NORMAL:%DUMBFW gnutls.c: [1] (Emacs) setting the priority string gnutls.c: [audit] Note that the security level of the Diffie-Hellman key exchange has been lowered to 128 bits and this may allow decryption of the session data gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily unavailable, try again. [3 times] After that eww shows: Bad Request Your browser sent a request that this server could not understand. Reason: You're speaking plain HTTP to an SSL-enabled server port. Instead use the HTTPS scheme to access this URL, please. 2.) open-gnutls-stream lwn.net (open-gnutls-stream "test" (current-buffer) "lwn.net" "https") gnutls.c: [1] (Emacs) connecting to host: lwn.net gnutls.c: [1] (Emacs) allocating credentials gnutls.c: [audit] There was a non-CA certificate in the trusted list: OU=Copyright (c) 1997 Microsoft Corp.,OU=Microsoft Corporation,CN=Microsoft Root Authority. gnutls.c: [audit] There was a non-CA certificate in the trusted list: C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority. gnutls.c: [audit] There was a non-CA certificate in the trusted list: CN=Root Agency. gnutls.c: [1] (Emacs) setting the trustfile: C:\Users\thomas\emacs-26.1\etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem gnutls.c: [1] (Emacs) setting the trustfile: C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.crt gnutls.c: [1] (Emacs) setting the trustfile: C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.trust.crt gnutls.c: [1] (Emacs) gnutls callbacks gnutls.c: [1] (Emacs) gnutls_init gnutls.c: [1] (Emacs) got non-default priority string: NORMAL:%DUMBFW gnutls.c: [1] (Emacs) setting the priority string gnutls.c: [audit] Note that the security level of the Diffie-Hellman key exchange has been lowered to 128 bits and this may allow decryption of the session data gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily unavailable, try again. [4905 times] gnutls.c: [1] (Emacs) verification: the certificate was signed by an unknown and therefore untrusted authority gnutls.c: [1] (Emacs) verification: certificate could not be verified gnutls.c: [1] (Emacs) certificate validation failed: lwn.net I think after that a TLS connection was successfully established and the buffer prints: # what springs into the eye is the difference of number of re-tries that are necessary to establish an TLS connection 4905 vs. 3. Why does url-retrieve give up after 3 re-tries? Here an debug-on-entry of open-network-stream of eww lwn.net: * open-network-stream("lwn.net" # "lwn.net" 80 :type plain :nowait (:nowait t)) url-open-stream("lwn.net" # "lwn.net" 80 nil) url-http-find-free-connection("lwn.net" 80 nil) url-http(#s(url :type "http" :user nil :password nil :host "lwn.net" :portspec nil :filename "/" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t) eww-render (nil "http://lwn.net/" nil #)) url-retrieve-internal("http://lwn.net/" eww-render (nil "http://lwn.net/" nil #) nil nil) url-retrieve("http://lwn.net/" eww-render ("http://lwn.net/" nil #)) eww("lwn.net") funcall-interactively(eww "lwn.net") call-interactively(eww record nil) command-execute(eww record) execute-extended-command(nil "eww" "eww") funcall-interactively(execute-extended-command nil "eww" "eww") call-interactively(execute-extended-command nil nil) command-execute(execute-extended-command) any ideas what going on here? btw. emacs 25.3 with gnutls 3.5.19 works correctly on the same machine when trying to connect to lwn.net with eww.