bug#27658: 26.0.50; response parsing hangs when using tls.el and gnutls-cli.exe on Windows

bug#27658: 26.0.50; response parsing hangs when using tls.el and gnutls-cli.exe on Windows

[Richard Copley]

Install GNUTLS. Obtain certificates and set `tls-program' as described
in Step 3 of Glyph's blog post at
"https://glyph.twistedmatrix.com/2015/11/editor-malware.html".

Ensure Emacs is using "gnutls-cli.exe" instead of "libgnutls-30.dll"
by (a) removing the DLL from your path or (b) configuring Emacs with
the "--without-gnutls" configure option.

In emacs -Q:

(pop-to-buffer
 (url-retrieve
  "https://elpa.gnu.org/packages/archive-contents"
  #'ignore))

Expected result: a buffer is displayed and, soon afterwards, filled with
the HTTP response (with the network CRLF line-endings in the raw
response translated to ordinary newlines, without CR, in the buffer).

Actual result: "Opening TLS connection [...]...done" is shown in
the minibuffer and the function never returns. (Type C-g to quit).

(This is probably the cause of #22929.)

----
Observing the process filter in the debugger, there is an extra carriage
return before every newline in the process buffer of the gnutls-cli.exe
process.

By "extra", I mean the informational text from "gnutls-cli.exe" has
one carriage return at the end of each line (where there should be none)
and the HTTP headers have two carriage returns at the end of each line
(where there should be one, because this is the raw payload and has
network line endings).

The extra newlines in the informational output cause `open-tls-stream'
in "tls.el" to hang indefinitely because the regexp `tls-end-of-info'
never matches. Fixing that regexp is not a solution. When I try that,
the HTTP header parsing in "url-http.el" fails in a similar way.

Possible fix:

The other obvious experiment I tried was to add
"(set-process-coding-system process 'raw-text-dos)" to
`open-tls-stream', just after the call to `start-process'. (Line
238.) Having done that, I do get a reasonable HTML document (and no
error) from `url-retrieve'.

That's a data point for information. I'm not sure if it's a good fix.
In the blog post I referred to above, Glyph says:

"[...] for some reason, Emacs could not parse gnutls-cli.exe   s output no
matter what I did."

Indeed, with the change(s) above, "package.el" still doesn't
succeed in parsing package lists retrieved over HTTPS; but that
might be a separate bug.
----

In GNU Emacs 26.0.50 (build 1, x86_64-w64-mingw32)
 of 2017-07-11 built on MACHINE
Repository revision: 7168bd2957a567c235d1b8b57acb4325187bd542
Windowing system distributor 'Microsoft Corp.', version 10.0.15063
Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Configured using:
 'configure --prefix=/mingw64 --config-cache --with-modules
 --without-gnutls --without-pop --enable-locallisppath=/c/emacs-lisp
 CFLAGS=-O3'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS MODULES

Important settings:
  value of $EMACSLOADPATH: c:\emacs-lisp;
  value of $LANG: ENG
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message subr-x puny seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win w32-vars
term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote w32notify w32 multi-tty make-network-process emacs)

Memory information:
((conses 16 96670 10234)
 (symbols 56 20064 1)
 (miscs 48 39 87)
 (strings 32 29950 1608)
 (string-bytes 1 773498)
 (vectors 16 13875)
 (vector-slots 8 480519 8489)
 (floats 8 50 151)
 (intervals 56 240 3)
 (buffers 976 11))

bug#27658: 26.0.50; response parsing hangs when using tls.el and gnutls-cli.exe on Windows

[npostavs]

Richard Copley <rcopley@gmail.com> writes:

> Ensure Emacs is using "gnutls-cli.exe" instead of "libgnutls-30.dll"
> by (a) removing the DLL from your path or (b) configuring Emacs with
> the "--without-gnutls" configure option.

In https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23759#8 Eli said:

    TLS connections on MS-Windows are supported via the GnuTLS library.
    External TLS programs will never work correctly on Windows, since
    they use signals to communicate with Emacs.  So there's little sense
    in fixing this issue, because the result will not work anyway.

bug#27658: 26.0.50; response parsing hangs when using tls.el and gnutls-cli.exe on Windows

[Richard Copley]

tags 27658 wontfix
close 27658
thank you

On 12 July 2017 at 01:17,  <npostavs@users.sourceforge.net> wrote:
> Richard Copley <rcopley@gmail.com> writes:
>
>> Ensure Emacs is using "gnutls-cli.exe" instead of "libgnutls-30.dll"
>> by (a) removing the DLL from your path or (b) configuring Emacs with
>> the "--without-gnutls" configure option.
>
> In https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23759#8 Eli said:
>
>     TLS connections on MS-Windows are supported via the GnuTLS library.
>     External TLS programs will never work correctly on Windows, since
>     they use signals to communicate with Emacs.  So there's little sense
>     in fixing this issue, because the result will not work anyway.
>

I see, thank you.
That's a lot of hours I'll never get back. No big deal! But a little comment
near the top of "tls.el", or somewhere, to let people know not to bother,
would be nice.

bug#27658: 26.0.50; response parsing hangs when using tls.el and gnutls-cli.exe on Windows

[npostavs]

Richard Copley <rcopley@gmail.com> writes:

>>
>> In https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23759#8 Eli said:
>>
>>     TLS connections on MS-Windows are supported via the GnuTLS library.
>>     External TLS programs will never work correctly on Windows, since
>>     they use signals to communicate with Emacs.  So there's little sense
>>     in fixing this issue, because the result will not work anyway.
>>
>
> I see, thank you.
> That's a lot of hours I'll never get back. No big deal! But a little comment
> near the top of "tls.el", or somewhere, to let people know not to bother,
> would be nice.

Hmm, and I see that #15905 was closed as wontfix too, so it's basically
unsupported at this point even on GNU/Linux platforms.  We should mark
it as obsolete at least, if not remove it entirely.

bug#27658: 26.0.50; response parsing hangs when using tls.el and gnutls-cli.exe on Windows

[Richard Copley]

On 12 July 2017 at 13:09,  <npostavs@users.sourceforge.net> wrote:
> Richard Copley <rcopley@gmail.com> writes:
>
>>>
>>> In https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23759#8 Eli said:
>>>
>>>     TLS connections on MS-Windows are supported via the GnuTLS library.
>>>     External TLS programs will never work correctly on Windows, since
>>>     they use signals to communicate with Emacs.  So there's little sense
>>>     in fixing this issue, because the result will not work anyway.
>>>
>>
>> I see, thank you.
>> That's a lot of hours I'll never get back. No big deal! But a little comment
>> near the top of "tls.el", or somewhere, to let people know not to bother,
>> would be nice.
>
> Hmm, and I see that #15905 was closed as wontfix too, so it's basically
> unsupported at this point even on GNU/Linux platforms.  We should mark
> it as obsolete at least, if not remove it entirely.

That's a shame, because the library-based implementation has some flaws
too. I've been trying to debug where that goes wrong, when it accepts certs
for the wrong host and self-signed certs. (Test case in Glyph's blog post.)
I didn't learn much. Never mind :)

bug#27658: 26.0.50; response parsing hangs when using tls.el and gnutls-cli.exe on Windows

[Eli Zaretskii]

> From: Richard Copley <rcopley@gmail.com>
> Date: Thu, 13 Jul 2017 03:42:56 +0100
> Cc: 27658@debbugs.gnu.org
> 
> > Hmm, and I see that #15905 was closed as wontfix too, so it's basically
> > unsupported at this point even on GNU/Linux platforms.  We should mark
> > it as obsolete at least, if not remove it entirely.
> 
> That's a shame, because the library-based implementation has some flaws
> too. I've been trying to debug where that goes wrong, when it accepts certs
> for the wrong host and self-signed certs. (Test case in Glyph's blog post.)
> I didn't learn much. Never mind :)

Are you sure that blog is still accurate?  It's quite old, and newer
versions of the GnuTLS library became meanwhile available.

bug#27658: 26.0.50; response parsing hangs when using tls.el and gnutls-cli.exe on Windows

[Richard Copley]

[-- Attachment #1: Type: text/plain, Size: 1764 bytes --]

On 13 Jul 2017 16:49, "Eli Zaretskii" <eliz@gnu.org> wrote:

> > From: Richard Copley <rcopley@gmail.com>
> > Date: Thu, 13 Jul 2017 03:42:56 +0100
> > Cc: 27658@debbugs.gnu.org
> >
> > > Hmm, and I see that #15905 was closed as wontfix too, so it's basically
> > > unsupported at this point even on GNU/Linux platforms.  We should mark
> > > it as obsolete at least, if not remove it entirely.
> >
> > That's a shame, because the library-based implementation has some flaws
> > too. I've been trying to debug where that goes wrong, when it accepts
> certs
> > for the wrong host and self-signed certs. (Test case in Glyph's blog
> post.)
> > I didn't learn much. Never mind :)
>
> Are you sure that blog is still accurate?  It's quite old, and newer
> versions of the GnuTLS library became meanwhile available.
>

No doubt some stuff there is no longer valid, but the test case should
succeed.

I have the latest release of GnuTLS and I did my own testing and
debugging using gnutls-cli.exe before writing this bug report.

I mentioned the library in my last message. I find (on my own system
today) that the Emacs TLS implementation using the library (in
gnutls.{c,el}) works except that it accepts bad certificates. I don't
think that's stated in the blog at all -- the blog is also mostly
about the implementation based on an external program (in tls.el).

The possible bug in gnutls.{c,el} or the library itself, the one I was
talking about in my last message, appears to be what is spoken about
here(1) in November 2015 and here(2) in February 2016. As far as I
know it doesn't have an Emacs bug report.

(1)
https://emacs.stackexchange.com/questions/18079/emacs-tls-check-is-still-ill-configured
(2) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816063

[-- Attachment #2: Type: text/html, Size: 2565 bytes --]

bug#27658: 26.0.50; response parsing hangs when using tls.el and gnutls-cli.exe on Windows

[Robert Pluim]

Richard Copley <rcopley@gmail.com> writes:

> On 13 Jul 2017 16:49, "Eli Zaretskii" <eliz@gnu.org> wrote:
>> Are you sure that blog is still accurate?  It's quite old, and newer
>> versions of the GnuTLS library became meanwhile available.
>>
>
> No doubt some stuff there is no longer valid, but the test case should
> succeed.
>
> I have the latest release of GnuTLS and I did my own testing and
> debugging using gnutls-cli.exe before writing this bug report.
>
> I mentioned the library in my last message. I find (on my own system
> today) that the Emacs TLS implementation using the library (in
> gnutls.{c,el}) works except that it accepts bad certificates. I don't
> think that's stated in the blog at all -- the blog is also mostly
> about the implementation based on an external program (in tls.el).

I've not been able to locate this blog entry. Could you point me at
it? Or provide details of hosts providing bad certificates that emacs
accepts when it shouldn't? Perhaps the default settings of GnuTLS in
emacs need tweaking.

> The possible bug in gnutls.{c,el} or the library itself, the one I was
> talking about in my last message, appears to be what is spoken about
> here(1) in November 2015 and here(2) in February 2016. As far as I
> know it doesn't have an Emacs bug report.
>

(2) leads to an Emacs bug report that is marked fixed. (1) is unclear
to me, I'll have to keep reading.

> (1)
> https://emacs.stackexchange.com/questions/18079/emacs-tls-check-is-still-ill-configured
> (2) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816063