bug#32658: 26.1; Cannot connect to TLS websites

bug#32658: 26.1; Cannot connect to TLS websites

[thomas]

when I try to connect to an TLS enabled website on this Windows 10
machine, I get an error message.

I tried with the packages gnutls version 3.6.0 on emacs 26.1 and I did
upgrade the gnutls version to the 3.6.3 but still no success.

I did set gnutls-log-level to 5 here is the log with emacs 26.1 and
upgrade gnutls library to 3.6.3:

Contacting host: lwn.net:80
5 (#o5, #x5, ?\C-e)
Contacting host: lwn.net:443
gnutls.c: [1] (Emacs) connecting to host: lwn.net
gnutls.c: [1] (Emacs) allocating credentials
gnutls.c: [2] (Emacs) allocating x509 credentials
gnutls.c: [2] (Emacs) using default verification flags
gnutls.c: [3] ASSERT: verify-high.c[gnutls_x509_trust_list_add_cas]:321

gnutls.c: [audit] There was a non-CA certificate in the trusted list: OU=Copyright (c) 1997 Microsoft Corp.,OU=Microsoft Corporation,CN=Microsoft Root Authority.

gnutls.c: [3] ASSERT: verify-high.c[gnutls_x509_trust_list_add_cas]:321

gnutls.c: [audit] There was a non-CA certificate in the trusted list: C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority.

gnutls.c: [3] ASSERT: common.c[_gnutls_x509_get_raw_field2]:1566

gnutls.c: [3] ASSERT: x509.c[gnutls_x509_crt_get_subject_unique_id]:3895

gnutls.c: [3] ASSERT: x509.c[gnutls_x509_crt_get_issuer_unique_id]:3945

gnutls.c: [3] ASSERT: common.c[_gnutls_x509_get_raw_field2]:1566

gnutls.c: [3] ASSERT: x509.c[gnutls_x509_crt_get_subject_unique_id]:3895

gnutls.c: [3] ASSERT: x509.c[gnutls_x509_crt_get_issuer_unique_id]:3945

gnutls.c: [3] ASSERT: common.c[_gnutls_x509_get_raw_field2]:1566

gnutls.c: [3] ASSERT: x509.c[gnutls_x509_crt_get_subject_unique_id]:3895

gnutls.c: [3] ASSERT: x509.c[gnutls_x509_crt_get_issuer_unique_id]:3945

gnutls.c: [3] ASSERT: dn.c[_gnutls_x509_compare_raw_dn]:988

gnutls.c: [3] ASSERT: common.c[_gnutls_x509_get_raw_field2]:1566

gnutls.c: [3] ASSERT: x509.c[gnutls_x509_crt_get_subject_unique_id]:3895

gnutls.c: [3] ASSERT: x509.c[gnutls_x509_crt_get_issuer_unique_id]:3945

gnutls.c: [3] ASSERT: common.c[_gnutls_x509_get_raw_field2]:1566

gnutls.c: [3] ASSERT: x509.c[gnutls_x509_crt_get_subject_unique_id]:3895

gnutls.c: [3] ASSERT: x509.c[gnutls_x509_crt_get_issuer_unique_id]:3945

gnutls.c: [3] ASSERT: verify-high.c[gnutls_x509_trust_list_add_cas]:321

gnutls.c: [audit] There was a non-CA certificate in the trusted list: CN=Root Agency.

gnutls.c: [1] (Emacs) gnutls callbacks
gnutls.c: [1] (Emacs) gnutls_init
gnutls.c: [5] REC[0000000005a734f0]: Allocating epoch #0

gnutls.c: [1] (Emacs) got non-default priority string: NORMAL:%DUMBFW
gnutls.c: [1] (Emacs) setting the priority string
gnutls.c: [2] added 5 protocols, 29 ciphersuites, 15 sig algos and 8 groups into priority list

gnutls.c: [audit] Note that the security level of the Diffie-Hellman key exchange has been lowered to 256 bits and this may allow decryption of the session data

gnutls.c: [5] REC[0000000005a734f0]: Allocating epoch #1

gnutls.c: [4] HSK[0000000005a734f0]: Adv. version: 3.3

gnutls.c: [2] Keeping ciphersuite c0.2c (GNUTLS_ECDHE_ECDSA_AES_256_GCM_SHA384)

gnutls.c: [2] Keeping ciphersuite cc.a9 (GNUTLS_ECDHE_ECDSA_CHACHA20_POLY1305)

gnutls.c: [2] Keeping ciphersuite c0.ad (GNUTLS_ECDHE_ECDSA_AES_256_CCM)

gnutls.c: [2] Keeping ciphersuite c0.0a (GNUTLS_ECDHE_ECDSA_AES_256_CBC_SHA1)

gnutls.c: [2] Keeping ciphersuite c0.2b (GNUTLS_ECDHE_ECDSA_AES_128_GCM_SHA256)

gnutls.c: [2] Keeping ciphersuite c0.ac (GNUTLS_ECDHE_ECDSA_AES_128_CCM)

gnutls.c: [2] Keeping ciphersuite c0.09 (GNUTLS_ECDHE_ECDSA_AES_128_CBC_SHA1)

gnutls.c: [2] Keeping ciphersuite c0.30 (GNUTLS_ECDHE_RSA_AES_256_GCM_SHA384)

gnutls.c: [2] Keeping ciphersuite cc.a8 (GNUTLS_ECDHE_RSA_CHACHA20_POLY1305)

gnutls.c: [2] Keeping ciphersuite c0.14 (GNUTLS_ECDHE_RSA_AES_256_CBC_SHA1)

gnutls.c: [2] Keeping ciphersuite c0.2f (GNUTLS_ECDHE_RSA_AES_128_GCM_SHA256)

gnutls.c: [2] Keeping ciphersuite c0.13 (GNUTLS_ECDHE_RSA_AES_128_CBC_SHA1)

gnutls.c: [2] Keeping ciphersuite 00.9d (GNUTLS_RSA_AES_256_GCM_SHA384)

gnutls.c: [2] Keeping ciphersuite c0.9d (GNUTLS_RSA_AES_256_CCM)

gnutls.c: [2] Keeping ciphersuite 00.35 (GNUTLS_RSA_AES_256_CBC_SHA1)

gnutls.c: [2] Keeping ciphersuite 00.9c (GNUTLS_RSA_AES_128_GCM_SHA256)

gnutls.c: [2] Keeping ciphersuite c0.9c (GNUTLS_RSA_AES_128_CCM)

gnutls.c: [2] Keeping ciphersuite 00.2f (GNUTLS_RSA_AES_128_CBC_SHA1)

gnutls.c: [2] Keeping ciphersuite 00.9f (GNUTLS_DHE_RSA_AES_256_GCM_SHA384)

gnutls.c: [2] Keeping ciphersuite cc.aa (GNUTLS_DHE_RSA_CHACHA20_POLY1305)

gnutls.c: [2] Keeping ciphersuite c0.9f (GNUTLS_DHE_RSA_AES_256_CCM)

gnutls.c: [2] Keeping ciphersuite 00.39 (GNUTLS_DHE_RSA_AES_256_CBC_SHA1)

gnutls.c: [2] Keeping ciphersuite 00.9e (GNUTLS_DHE_RSA_AES_128_GCM_SHA256)

gnutls.c: [2] Keeping ciphersuite c0.9e (GNUTLS_DHE_RSA_AES_128_CCM)

gnutls.c: [2] Keeping ciphersuite 00.33 (GNUTLS_DHE_RSA_AES_128_CBC_SHA1)

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Maximum Record Size/1) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (OCSP Status Request/5) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Sending extension OCSP Status Request/5 (5 bytes)

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Supported Groups/10) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Sent group SECP256R1 (0x17)

gnutls.c: [4] EXT[0000000005a734f0]: Sent group SECP384R1 (0x18)

gnutls.c: [4] EXT[0000000005a734f0]: Sent group SECP521R1 (0x19)

gnutls.c: [4] EXT[0000000005a734f0]: Sent group X25519 (0x1d)

gnutls.c: [4] EXT[0000000005a734f0]: Sent group FFDHE2048 (0x100)

gnutls.c: [4] EXT[0000000005a734f0]: Sent group FFDHE3072 (0x101)

gnutls.c: [4] EXT[0000000005a734f0]: Sent group FFDHE4096 (0x102)

gnutls.c: [4] EXT[0000000005a734f0]: Sent group FFDHE8192 (0x104)

gnutls.c: [4] EXT[0000000005a734f0]: Sending extension Supported Groups/10 (18 bytes)

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Supported EC Point Formats/11) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Sending extension Supported EC Point Formats/11 (2 bytes)

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (SRP/12) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Signature Algorithms/13) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (4.1) RSA-SHA256

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (8.9) RSA-PSS-SHA256

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (8.4) RSA-PSS-RSAE-SHA256

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (4.3) ECDSA-SHA256

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (8.7) EdDSA-Ed25519

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (5.1) RSA-SHA384

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (8.10) RSA-PSS-SHA384

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (8.5) RSA-PSS-RSAE-SHA384

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (5.3) ECDSA-SHA384

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (6.1) RSA-SHA512

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (8.11) RSA-PSS-SHA512

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (8.6) RSA-PSS-RSAE-SHA512

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (6.3) ECDSA-SHA512

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (2.1) RSA-SHA1

gnutls.c: [4] EXT[0000000005a734f0]: sent signature algo (2.3) ECDSA-SHA1

gnutls.c: [4] EXT[0000000005a734f0]: Sending extension Signature Algorithms/13 (32 bytes)

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (SRTP/14) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Heartbeat/15) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (ALPN/16) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Encrypt-then-MAC/22) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Sending extension Encrypt-then-MAC/22 (0 bytes)

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Extended Master Secret/23) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Sending extension Extended Master Secret/23 (0 bytes)

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Session Ticket/35) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Sending extension Session Ticket/35 (0 bytes)

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Key Share/51) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Supported Versions/43) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Post Handshake Auth/49) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Safe Renegotiation/65281) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Sending extension Safe Renegotiation/65281 (1 bytes)

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Server Name Indication/0) for 'client hello'

gnutls.c: [2] HSK[0000000005a734f0]: sent server name: 'lwn.net'

gnutls.c: [4] EXT[0000000005a734f0]: Sending extension Server Name Indication/0 (12 bytes)

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Cookie/44) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (PSK Key Exchange Modes/45) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (ClientHello Padding/21) for 'client hello'

gnutls.c: [4] EXT[0000000005a734f0]: Preparing extension (Pre Shared Key/41) for 'client hello'

gnutls.c: [4] HSK[0000000005a734f0]: CLIENT HELLO was queued [201 bytes]

gnutls.c: [5] REC[0000000005a734f0]: Preparing Packet Handshake(22) with length: 201 and min pad: 0

gnutls.c: [5] REC[0000000005a734f0]: Sent Packet[1] Handshake(22) in epoch 0 and length: 206

gnutls.c: [3] ASSERT: buffers.c[_gnutls_writev_emu]:464

gnutls.c: [2] WRITE: -1 returned from 0000000005f1ae30, errno: 11

gnutls.c: [3] (Emacs) retry: Resource temporarily unavailable, try again.
gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily unavailable, try again.
gnutls.c: [3] ASSERT: buffers.c[_gnutls_writev_emu]:464

gnutls.c: [2] WRITE: -1 returned from 0000000005f1ae30, errno: 11

gnutls.c: [3] (Emacs) retry: Resource temporarily unavailable, try again.
gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily unavailable, try again.
gnutls.c: [2] (Emacs) Deallocating x509 credentials
gnutls.c: [5] REC[0000000005a734f0]: Start of epoch cleanup

gnutls.c: [5] REC[0000000005a734f0]: End of epoch cleanup

gnutls.c: [5] REC[0000000005a734f0]: Epoch #0 freed

gnutls.c: [5] REC[0000000005a734f0]: Epoch #1 freed

I'm not sure why the write get's an EAGAIN.
Another setup special is that I login into this Windows 10 machine as a
normal user without admin permissions. Most people doesn't do that and
only have one account with admin permission. maybe this is somehow
related, maybe not.


In GNU Emacs 26.1 (build 1, x86_64-w64-mingw32)
 of 2018-05-30 built on CIRROCUMULUS
Repository revision: 07f8f9bc5a51f5aa94eb099f3e15fbe0c20ea1ea
Windowing system distributor 'Microsoft Corp.', version 10.0.17134
Recent messages:
gnutls.c: [5] REC[0000000005a734f0]: Start of epoch cleanup

gnutls.c: [5] REC[0000000005a734f0]: End of epoch cleanup

gnutls.c: [5] REC[0000000005a734f0]: Epoch #0 freed

gnutls.c: [5] REC[0000000005a734f0]: Epoch #1 freed

scroll-down-command: Beginning of buffer [7 times]
Making completion list...

Configured using:
 'configure --without-dbus --host=x86_64-w64-mingw32
 --without-compress-install 'CFLAGS=-O2 -static -g3''

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS THREADS LCMS2

Important settings:
  value of $LANG: DE
  locale-coding-system: cp1252

Major mode: Messages

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired dired-loaddefs rfc822 mml
mml-sec epa derived epg epg-config mm-decode mm-bodies mm-encode
mailabbrev gmm-utils mailheader sendmail misearch multi-isearch
network-stream starttls url-http tls gnutls mail-parse rfc2231 url-gw
nsm rmc url-cache url-auth eww easymenu puny mm-url gnus nnheader
gnus-util rmail rmail-loaddefs rfc2047 rfc2045 ietf-drums mail-utils
wid-edit mm-util mail-prsvr url-queue url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
url-parse auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache url-vars mailcap shr svg xml seq byte-opt gv bytecomp
byte-compile cconv dom browse-url format-spec cl-loaddefs cl-lib
elec-pair time-date mule-util tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel dos-w32 ls-lisp disp-table term/w32-win
w32-win w32-vars term/common-win tool-bar dnd fontset image regexp-opt
fringe tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote w32notify w32 lcms2 multi-tty make-network-process emacs)

Memory information:
((conses 16 127886 9148)
 (symbols 56 23472 1)
 (miscs 48 88 141)
 (strings 32 39033 926)
 (string-bytes 1 1042801)
 (vectors 16 17742)
 (vector-slots 8 533924 11674)
 (floats 8 78 408)
 (intervals 56 446 0)
 (buffers 992 14))

bug#32658: gnutls + non-blocking url-retrieve

[thomas]

Hi,

some more details.

1.) I needed to revert to gnutls 3.5.19, the mingw64 build from the
gitlab ci build seems to have a working gnutls-cli tools on windows 10.
the gitlab builds for 3.6.3 and 3.6.4 seems to have another bug
(error code -53) in the gnutls-cli command.

so only gnutls 3.5.19 have a working gnutls-cli. i installed this version in emacs 26.1

2.) testing gnutls stream
using open-gnutls-stream directly gives me a correct tls connection but
eww still fails to load the site.

when I change url-open-stream in url/url-gw.el to:
			  (open-network-stream
			   name buffer host service
			   :type gw-method
			   ;; Use non-blocking socket if we can.
                          :nowait nil))

I finally can open lwn.net in eww.

so something seems to be wrong possible with blocking/non-blocking
network access.

any ideas?

bug#32658: gnutls + non-blocking url-retrieve

[Eli Zaretskii]

> From: thomas@m3y3r.de
> Date: Sun, 30 Sep 2018 23:33:10 +0200
> 
> 1.) I needed to revert to gnutls 3.5.19, the mingw64 build from the
> gitlab ci build seems to have a working gnutls-cli tools on windows 10.
> the gitlab builds for 3.6.3 and 3.6.4 seems to have another bug
> (error code -53) in the gnutls-cli command.
> 
> so only gnutls 3.5.19 have a working gnutls-cli. i installed this version in emacs 26.1
> 
> 2.) testing gnutls stream
> using open-gnutls-stream directly gives me a correct tls connection but
> eww still fails to load the site.
> 
> when I change url-open-stream in url/url-gw.el to:
> 			  (open-network-stream
> 			   name buffer host service
> 			   :type gw-method
> 			   ;; Use non-blocking socket if we can.
>                           :nowait nil))
> 
> I finally can open lwn.net in eww.
> 
> so something seems to be wrong possible with blocking/non-blocking
> network access.
> 
> any ideas?

Thanks for the info.

First, I don't understand what does gnutls-cli have to do with this.
Emacs on Windows doesn't support TLS connections that use gnutls-cli,
because the way that works, it requires working support for signals,
which cannot happen on Windows.  Are you saying that these problems
happen when you use gnutls-cli?  If so, please move to the built-in
GnuTLS support, because connections using gnutls-cli are deprecated,
and I see no point in trying to support them on Windows.

Second, I cannot reproduce the problem you are reporting.  Using stock
Emacs 26.1 I built myself, with GnuTLS 3.4.15, I have no problems
connecting to lwn.net via eww.  I see EAGAIN errors like you do, but
they are non-fatal, so don't prevent the connection from continuing.
It is strange that you are having these problems, but maybe these
problems are specific to GnuTLS 3.6.x?  3.6.x is not a stable branch
of GnuTLS, it could have bugs, in particular bugs specific to Windows.
It is also possible that there are incompatibilities between GnuTLS
3.6.x and whatever version the Emacs binary you are using was built
against.

In this message you say that you downgraded to GnuTLS 3.5.19, but you
didn't show the gnutls.c log for that version -- does it mean you see
an identical problem with EAGAIN there?

Is it possible for you to downgrade GnuTLS to some version of the
3.4.x branch, and see if the problem persists?

Also, does this happen in "emacs -Q"?

Or maybe this is specific to your network connection?  Does any HTTPS
connection cause these problems?

Finally, what about other machines and/or Windows versions other than
10 -- do you have the same problem there with this Emacs version
(assuming you can test that)?

Bottom line: I'm surprised that you have these problems, because I see
none of that on my machines -- TLS connections "just work" for me,
without any need to tinker with url-gw.el or elsewhere.  And judging
by lack of similar bug reports, this also works for others.  So I
wonder what causes this in your case.

bug#32658: gnutls + non-blocking url-retrieve

[thomas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: thomas@m3y3r.de
>> Date: Sun, 30 Sep 2018 23:33:10 +0200
>> 
>> 1.) I needed to revert to gnutls 3.5.19, the mingw64 build from the
>> gitlab ci build seems to have a working gnutls-cli tools on windows 10.
>> the gitlab builds for 3.6.3 and 3.6.4 seems to have another bug
>> (error code -53) in the gnutls-cli command.
>> 
>> so only gnutls 3.5.19 have a working gnutls-cli. i installed this version in emacs 26.1
>> 
>> 2.) testing gnutls stream
>> using open-gnutls-stream directly gives me a correct tls connection but
>> eww still fails to load the site.
>> 
>> when I change url-open-stream in url/url-gw.el to:
>> 			  (open-network-stream
>> 			   name buffer host service
>> 			   :type gw-method
>> 			   ;; Use non-blocking socket if we can.
>>                           :nowait nil))
>> 
>> I finally can open lwn.net in eww.
>> 
>> so something seems to be wrong possible with blocking/non-blocking
>> network access.
>> 
>> any ideas?
>

> Thanks for the info.

Hi, thanks for looking into this.

>
> First, I don't understand what does gnutls-cli have to do with this.

okay, thats an easy one to explain.
I did download emacs 26.1 from here:
http://mirror.netcologne.de/gnu/emacs/windows/emacs-26/emacs-26.1-x86_64.zip

in the bin directory there is the gnutls packaged. version is 3.6.0.

I wasn't sure where the bug in the TLS problems I see was, so I first
tried to use gnutls-cli to check if a connection can be made:

C:\Users\thomas\Downloads\emacs-26.1-x86_64\bin>gnutls-cli lwn.net
|<1>| There was a non-CA certificate in the trusted list: OU=Copyright (c) 1997 Microsoft Corp.,OU=Microsoft Corporation,CN=Microsoft Root Authority.
|<1>| There was a non-CA certificate in the trusted list: C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority.
|<1>| There was a non-CA certificate in the trusted list: CN=Root Agency.
Processed 59 CA certificate(s).
Resolving 'lwn.net:443'...
Connecting to '2600:3c03::f03c:91ff:fe61:5c5b:443'...
*** Fatal error: Error in the push function.
Connecting to '45.33.94.129:443'...
*** Fatal error: Error in the push function.
Could not connect to 45.33.94.129:443: Bad file descriptor

This doesn't work, because of some error -53 (ERROR_BAD_NETPATH).

so this is why I did first try to upgrade to gnutls 3.6.3 from the
gnutls homepage which is a gitlab ci build, but with no success.

then i tried to downgrade the gnutls version to 3.5.19 and there the
gnutls-cli tool did work.

so gnutls is now able to create an TLS connection. now the question why
it doesn't work in emacs.

> Emacs on Windows doesn't support TLS connections that use gnutls-cli,
> because the way that works, it requires working support for signals,
> which cannot happen on Windows.  Are you saying that these problems
> happen when you use gnutls-cli?  If so, please move to the built-in
> GnuTLS support, because connections using gnutls-cli are deprecated,
> and I see no point in trying to support them on Windows.

see above explanation. hopefully this makes it clear.

>
> Second, I cannot reproduce the problem you are reporting.  Using stock
> Emacs 26.1 I built myself, with GnuTLS 3.4.15, I have no problems
> connecting to lwn.net via eww.  I see EAGAIN errors like you do, but
> they are non-fatal, so don't prevent the connection from continuing.
> It is strange that you are having these problems, but maybe these
> problems are specific to GnuTLS 3.6.x?  3.6.x is not a stable branch
> of GnuTLS, it could have bugs, in particular bugs specific to Windows.
> It is also possible that there are incompatibilities between GnuTLS
> 3.6.x and whatever version the Emacs binary you are using was built
> against.

I do use the emacs 26.1 zip file that is pre-build and linked to from
the emacs homepage, see above.

>
> In this message you say that you downgraded to GnuTLS 3.5.19, but you
> didn't show the gnutls.c log for that version -- does it mean you see
> an identical problem with EAGAIN there?
>
> Is it possible for you to downgrade GnuTLS to some version of the
> 3.4.x branch, and see if the problem persists?

will try out and report.

>
> Also, does this happen in "emacs -Q"?

yes, same error.

>
> Or maybe this is specific to your network connection?  Does any HTTPS
> connection cause these problems?

no, only some I think.

>
> Finally, what about other machines and/or Windows versions other than
> 10 -- do you have the same problem there with this Emacs version
> (assuming you can test that)?

I use emacs 26.1 on an linux x86 system and on android arm with termux,
both on the same network as the windows laptop, and everything works as
expected on these systems.

>
> Bottom line: I'm surprised that you have these problems, because I see
> none of that on my machines -- TLS connections "just work" for me,
> without any need to tinker with url-gw.el or elsewhere.  And judging
> by lack of similar bug reports, this also works for others.  So I
> wonder what causes this in your case.

I do also wonder!

here some more details, with vanilla emacs 26.1 + gnutls
3.5.19 and gnutls-log-level 1:

1.) eww lwn.net
Contacting host: lwn.net:80
gnutls.c: [1] (Emacs) connecting to host: lwn.net
gnutls.c: [1] (Emacs) allocating credentials
gnutls.c: [audit] There was a non-CA certificate in the trusted list:
OU=Copyright (c) 1997 Microsoft Corp.,OU=Microsoft
Corporation,CN=Microsoft Root Authority.

gnutls.c: [audit] There was a non-CA certificate in the trusted list: C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority.

gnutls.c: [audit] There was a non-CA certificate in the trusted list: CN=Root Agency.

gnutls.c: [1] (Emacs) setting the trustfile:  C:\Users\thomas\emacs-26.1\etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
gnutls.c: [1] (Emacs) setting the trustfile:  C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.crt
gnutls.c: [1] (Emacs) setting the trustfile:  C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.trust.crt
gnutls.c: [1] (Emacs) gnutls callbacks
gnutls.c: [1] (Emacs) gnutls_init
gnutls.c: [1] (Emacs) got non-default priority string: NORMAL:%DUMBFW
gnutls.c: [1] (Emacs) setting the priority string
gnutls.c: [audit] Note that the security level of the Diffie-Hellman
key exchange has been lowered to 128 bits and this may allow
decryption of the session data
gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily unavailable, try again. [3 times]

After that eww shows:

Bad Request

Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.

2.) open-gnutls-stream lwn.net

(open-gnutls-stream "test" (current-buffer) "lwn.net" "https")

gnutls.c: [1] (Emacs) connecting to host: lwn.net
gnutls.c: [1] (Emacs) allocating credentials
gnutls.c: [audit] There was a non-CA certificate in the trusted list:
OU=Copyright (c) 1997 Microsoft Corp.,OU=Microsoft
Corporation,CN=Microsoft Root Authority.

gnutls.c: [audit] There was a non-CA certificate in the trusted list: C=US,O=MSFT,CN=Microsoft Authenticode(tm) Root Authority.

gnutls.c: [audit] There was a non-CA certificate in the trusted list: CN=Root Agency.

gnutls.c: [1] (Emacs) setting the trustfile:  C:\Users\thomas\emacs-26.1\etc\pki\ca-trust\extracted\pem\tls-ca-bundle.pem
gnutls.c: [1] (Emacs) setting the trustfile:  C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.crt
gnutls.c: [1] (Emacs) setting the trustfile:  C:\Users\thomas\emacs-26.1\ssl\certs\ca-bundle.trust.crt
gnutls.c: [1] (Emacs) gnutls callbacks
gnutls.c: [1] (Emacs) gnutls_init
gnutls.c: [1] (Emacs) got non-default priority string: NORMAL:%DUMBFW
gnutls.c: [1] (Emacs) setting the priority string
gnutls.c: [audit] Note that the security level of the Diffie-Hellman
key exchange has been lowered to 128 bits and this may allow
decryption of the session data

gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily unavailable, try again. [4905 times]
gnutls.c: [1] (Emacs) verification: the certificate was signed by an unknown and therefore untrusted authority
gnutls.c: [1] (Emacs) verification: certificate could not be verified
gnutls.c: [1] (Emacs) certificate validation failed: lwn.net

I think after that a TLS connection was successfully established and the
buffer prints:
#<process test>

what springs into the eye is the difference of number of re-tries that
are necessary to establish an TLS connection 4905 vs. 3.

Why does url-retrieve give up after 3 re-tries?

Here an debug-on-entry of open-network-stream of eww lwn.net:

* open-network-stream("lwn.net" #<buffer  *url-http-temp*> "lwn.net" 80 :type plain :nowait (:nowait t))
  url-open-stream("lwn.net" #<buffer  *url-http-temp*> "lwn.net" 80 nil)
  url-http-find-free-connection("lwn.net" 80 nil)
  url-http(#s(url :type "http" :user nil :password nil :host "lwn.net"
:portspec nil :filename "/" :target nil :attributes nil :fullness t
:silent nil :use-cookies t :asynchronous t) eww-render (nil
"http://lwn.net/" nil #<buffer *eww*>))
  url-retrieve-internal("http://lwn.net/" eww-render (nil "http://lwn.net/" nil #<buffer *eww*>) nil nil)
  url-retrieve("http://lwn.net/" eww-render ("http://lwn.net/" nil #<buffer *eww*>))
  eww("lwn.net")
  funcall-interactively(eww "lwn.net")
  call-interactively(eww record nil)
  command-execute(eww record)
  execute-extended-command(nil "eww" "eww")
  funcall-interactively(execute-extended-command nil "eww" "eww")
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

any ideas what going on here?

btw. emacs 25.3 with gnutls 3.5.19 works correctly on the same machine
when trying to connect to lwn.net with eww.

bug#32658: gnutls + non-blocking url-retrieve

[Noam Postavsky]

On Mon, 1 Oct 2018 at 16:54, <thomas@m3y3r.de> wrote:

> okay, thats an easy one to explain.
> I did download emacs 26.1 from here:
> http://mirror.netcologne.de/gnu/emacs/windows/emacs-26/emacs-26.1-x86_64.zip
>
> in the bin directory there is the gnutls packaged. version is 3.6.0.

> *** Fatal error: Error in the push function.
> Could not connect to 45.33.94.129:443: Bad file descriptor
>
> This doesn't work, because of some error -53 (ERROR_BAD_NETPATH).

For the record, I tried with the same Emacs (albeit from my local
mirror), and I get the same error with gnutls-cli, but M-x eww works
fine.

> > Or maybe this is specific to your network connection?  Does any HTTPS
> > connection cause these problems?
>
> no, only some I think.

I think it could be interesting to characterize the difference (e.g.,
is it always with certain servers?)

> what springs into the eye is the difference of number of re-tries that
> are necessary to establish an TLS connection 4905 vs. 3.
>
> Why does url-retrieve give up after 3 re-tries?

When I try with eww here, I see

gnutls.c: [1] (Emacs) non-fatal error: Resource temporarily
unavailable, try again. [6 times]

(and it does open the page successfully, as I said above)

bug#32658: gnutls + non-blocking url-retrieve

[thomas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: thomas@m3y3r.de
>> Date: Sun, 30 Sep 2018 23:33:10 +0200
>> 
>> 1.) I needed to revert to gnutls 3.5.19, the mingw64 build from the
>> gitlab ci build seems to have a working gnutls-cli tools on windows 10.
>> the gitlab builds for 3.6.3 and 3.6.4 seems to have another bug
>> (error code -53) in the gnutls-cli command.
>> 
>> so only gnutls 3.5.19 have a working gnutls-cli. i installed this version in emacs 26.1
>> 
>> 2.) testing gnutls stream
>> using open-gnutls-stream directly gives me a correct tls connection but
>> eww still fails to load the site.
>> 
>> when I change url-open-stream in url/url-gw.el to:
>> 			  (open-network-stream
>> 			   name buffer host service
>> 			   :type gw-method
>> 			   ;; Use non-blocking socket if we can.
>>                           :nowait nil))
>> 
>> I finally can open lwn.net in eww.
>> 
>> so something seems to be wrong possible with blocking/non-blocking
>> network access.
>> 
>> any ideas?
>
> Thanks for the info.

so what happens in process.c:3669 in function connect_network_socket when gnutls_boot
returns with GNUTLS_STAGE_HANDSHAKE_TRIED and boot(error code) will
error GNUTLS_E_AGAIN (and
not even considered, as far as I understand the code).

I think this is what happens in may case.
gnutls_boot calls gnutls_try_handshake (gnutls.c:595) and the do/while loops returns after 3 times (what
I don't understand is: why is this happening, can maybe_quit() somewho break the loop?)

  do
    {
      ret = gnutls_handshake (state);
      emacs_gnutls_handle_error (state, ret);
      maybe_quit ();
    }
  while (ret < 0
	 && gnutls_error_is_fatal (ret) == 0
	 && ! non_blocking);

//HINT: maybe save emacs_gnutls_handle_error return value and check this
instead of calling gnutls_error_is_fatal again?

  proc->gnutls_initstage = GNUTLS_STAGE_HANDSHAKE_TRIED;

  if (ret == GNUTLS_E_SUCCESS)
    {
      /* Here we're finally done.  */
      proc->gnutls_initstage = GNUTLS_STAGE_READY;
    }
  else
    {
      /* check_memory_full (gnutls_alert_send_appropriate (state, ret));  */
    }
  return ret;

so what do you think?

bug#32658: gnutls + non-blocking url-retrieve

[thomas]

thomas@m3y3r.de writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: thomas@m3y3r.de
>>> Date: Sun, 30 Sep 2018 23:33:10 +0200
>>> 
>>> 1.) I needed to revert to gnutls 3.5.19, the mingw64 build from the
>>> gitlab ci build seems to have a working gnutls-cli tools on windows 10.
>>> the gitlab builds for 3.6.3 and 3.6.4 seems to have another bug
>>> (error code -53) in the gnutls-cli command.
>>> 
>>> so only gnutls 3.5.19 have a working gnutls-cli. i installed this version in emacs 26.1
>>> 
>>> 2.) testing gnutls stream
>>> using open-gnutls-stream directly gives me a correct tls connection but
>>> eww still fails to load the site.
>>> 
>>> when I change url-open-stream in url/url-gw.el to:
>>> 			  (open-network-stream
>>> 			   name buffer host service
>>> 			   :type gw-method
>>> 			   ;; Use non-blocking socket if we can.
>>>                           :nowait nil))
>>> 
>>> I finally can open lwn.net in eww.
>>> 
>>> so something seems to be wrong possible with blocking/non-blocking
>>> network access.
>>> 
>>> any ideas?
>>
>> Thanks for the info.
>

okay, some more infos.

I was able to bootstrap the emacs compile with mingw64.

while I was trying to debug this problem with fprintf, eww suddenly
started to work!!

It started to work after I did insert an fprintf after the
gnutls_try_handshake call in process.c !!

what is going on here?

diff --git a/src/gnutls.c b/src/gnutls.c
index 9e105b948f..374dfeb6e5 100644
--- a/src/gnutls.c
+++ b/src/gnutls.c
@@ -583,6 +583,7 @@ gnutls_try_handshake (struct Lisp_Process *proc)
 {
   gnutls_session_t state = proc->gnutls_state;
   int ret;
+  bool non_fatal;
   bool non_blocking = proc->is_non_blocking_client;
 
   if (proc->gnutls_complete_negotiation_p)
@@ -594,11 +595,11 @@ gnutls_try_handshake (struct Lisp_Process *proc)
   do
     {
       ret = gnutls_handshake (state);
-      emacs_gnutls_handle_error (state, ret);
+      non_fatal = emacs_gnutls_handle_error (state, ret);
       maybe_quit ();
     }
   while (ret < 0
-	 && gnutls_error_is_fatal (ret) == 0
+	 &&   non_fatal
 	 && ! non_blocking);
 
   proc->gnutls_initstage = GNUTLS_STAGE_HANDSHAKE_TRIED;
@@ -779,7 +780,7 @@ emacs_gnutls_handle_error (gnutls_session_t session, int err)
 
   /* TODO: use a Lisp_Object generated by gnutls_make_error?  */
   if (err >= 0)
-    return 1;
+    return true;
 
   check_memory_full (err);
 
diff --git a/src/process.c b/src/process.c
index b0a327229c..5bdb74868c 100644
--- a/src/process.c
+++ b/src/process.c
@@ -899,6 +899,10 @@ static void
 remove_process (register Lisp_Object proc)
 {
   register Lisp_Object pair;
+  struct Lisp_Process *lp = XPROCESS(proc);
+
+  fprintf (stderr, "DEBUG: remove process called for proc %s ", SDATA(lp->name) /*, status_message(&lp->status) */);
+  fprintf (stderr, "gnutls_initstage: %u\n", lp->gnutls_initstage);
 
   pair = Frassq (proc, Vprocess_alist);
   Vprocess_alist = Fdelq (pair, Vprocess_alist);
@@ -3283,6 +3287,8 @@ finish_after_tls_connection (Lisp_Object proc)
   Lisp_Object contact = p->childp;
   Lisp_Object result = Qt;
 
+  add_to_log("DEBUG: finish_after_tls_connection");
+
   if (!NILP (Ffboundp (Qnsm_verify_connection)))
     result = call3 (Qnsm_verify_connection,
 		    proc,
@@ -5097,9 +5103,12 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 		if (p->gnutls_initstage == GNUTLS_STAGE_HANDSHAKE_TRIED
 		    && p->is_non_blocking_client)
 		  {
-		    gnutls_try_handshake (p);
+		    fprintf (stderr, "DEBUG: trying_gnutls_handshake: %s, %lld, %d, %d, %d\n", SDATA(p->name), time_limit, nsecs, read_kbd, do_display);
+		    int rcgt = gnutls_try_handshake (p);
 		    p->gnutls_handshakes_tried++;
 
+		    fprintf (stderr, "DEBUG: after gnutls_handshake: %d, %d, %u\n", rcgt, p->gnutls_handshakes_tried, p->gnutls_initstage);
+
 		    if (p->gnutls_initstage == GNUTLS_STAGE_READY)
 		      {
 			gnutls_verify_boot (aproc, Qnil);

Here the output of stderr:

DEBUG: trying_gnutls_handshake: lwn.net<1>, 1, 843955000, -1, 1
DEBUG: after gnutls_handshake: -28, 1, 8
DEBUG: trying_gnutls_handshake: lwn.net<1>, 1, 650501000, -1, 1
DEBUG: after gnutls_handshake: -28, 2, 8
DEBUG: trying_gnutls_handshake: lwn.net<1>, 1, 650501000, -1, 1
DEBUG: after gnutls_handshake: -28, 3, 8
DEBUG: trying_gnutls_handshake: lwn.net<1>, 1, 650501000, -1, 1
DEBUG: after gnutls_handshake: -28, 4, 8
DEBUG: trying_gnutls_handshake: lwn.net<1>, 1, 650501000, -1, 1
DEBUG: after gnutls_handshake: -28, 5, 8
DEBUG: trying_gnutls_handshake: lwn.net<1>, 1, 650501000, -1, 1
DEBUG: after gnutls_handshake: -28, 6, 8
DEBUG: trying_gnutls_handshake: lwn.net<1>, 1, 650501000, -1, 1
DEBUG: after gnutls_handshake: -28, 7, 8
DEBUG: trying_gnutls_handshake: lwn.net<1>, 1, 650501000, -1, 1
DEBUG: after gnutls_handshake: 0, 8, 9
DEBUG: remove process called for proc lwn.net gnutls_initstage: 0
DEBUG: remove process called for proc lwn.net<1> gnutls_initstage: 3
DEBUG: trying_gnutls_handshake: lwn.net, 30, 0, -1, 1
DEBUG: after gnutls_handshake: -28, 1, 8
DEBUG: trying_gnutls_handshake: lwn.net, 30, 0, -1, 1
DEBUG: after gnutls_handshake: -28, 2, 8
DEBUG: trying_gnutls_handshake: lwn.net, 30, 0, -1, 1
DEBUG: after gnutls_handshake: -28, 3, 8
DEBUG: trying_gnutls_handshake: lwn.net, 30, 0, -1, 1
DEBUG: after gnutls_handshake: -28, 4, 8
DEBUG: trying_gnutls_handshake: lwn.net, 30, 0, -1, 1
DEBUG: after gnutls_handshake: -28, 5, 8
DEBUG: trying_gnutls_handshake: lwn.net, 30, 0, -1, 1
DEBUG: after gnutls_handshake: -28, 6, 8
DEBUG: trying_gnutls_handshake: lwn.net, 30, 0, -1, 1
DEBUG: after gnutls_handshake: -28, 7, 8
DEBUG: trying_gnutls_handshake: lwn.net, 30, 0, -1, 1
DEBUG: after gnutls_handshake: -28, 8, 8
DEBUG: trying_gnutls_handshake: lwn.net, 30, 0, -1, 1
DEBUG: after gnutls_handshake: 0, 9, 9
DEBUG: remove process called for proc lwn.net gnutls_initstage: 3

maybe this problem seems to be an timing and/or cpu issue? my laptop uses an
relativley new intel core i7-7500u cpu.

bug#32658: gnutls + non-blocking url-retrieve

[Noam Postavsky]

thomas@m3y3r.de writes:

>>>> From: thomas@m3y3r.de
>>>> 
>>>> 1.) I needed to revert to gnutls 3.5.19, the mingw64 build from the
>>>> gitlab ci build seems to have a working gnutls-cli tools on windows 10.
>>>> the gitlab builds for 3.6.3 and 3.6.4 seems to have another bug
>>>> (error code -53) in the gnutls-cli command.
>>>> 
>>>> so only gnutls 3.5.19 have a working gnutls-cli. i installed this version in emacs 26.1

> okay, some more infos.
>
> I was able to bootstrap the emacs compile with mingw64.
>
> while I was trying to debug this problem with fprintf, eww suddenly
> started to work!!
>
> It started to work after I did insert an fprintf after the
> gnutls_try_handshake call in process.c !!
>
> what is going on here?

It sounds like this might be Bug#34341, the problem was
emacs_gnutls_read didn't handle GNUTLS_E_AGAIN properly.  The symptoms
only seem to occur with servers supporting TLS1.3 and more recent gnutls
versions (I thought it was 3.6+, but I haven't pinned the precise
version numbers, so maybe 3.5.19 has it too, or maybe it's a bit
different on Windows).  And some people reported that increasing
gnutls-log-level made it go away, so it's certainly timing dependent.

So can you check if the problem is fixed in the latest emacs-26 or
master branch?

https://debbugs.gnu.org/34341

bug#32658: gnutls + non-blocking url-retrieve

[Lars Ingebrigtsen]

Noam Postavsky <npostavs@gmail.com> writes:

> It sounds like this might be Bug#34341, the problem was
> emacs_gnutls_read didn't handle GNUTLS_E_AGAIN properly.  The symptoms
> only seem to occur with servers supporting TLS1.3 and more recent gnutls
> versions (I thought it was 3.6+, but I haven't pinned the precise
> version numbers, so maybe 3.5.19 has it too, or maybe it's a bit
> different on Windows).  And some people reported that increasing
> gnutls-log-level made it go away, so it's certainly timing dependent.
>
> So can you check if the problem is fixed in the latest emacs-26 or
> master branch?

More information was requested, but no response was given within a few
months, so I'm closing this bug report.  If the problem still exists,
please reopen this bug report.

(And it seems likely that this problem has been fixed.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no