bug#47321: 27.1; gnutls.c does not handle TLS rehandshake from server

bug#47321: 27.1; gnutls.c does not handle TLS rehandshake from server

[Jan Ypma]

-------------------------------------------------------------------

I'm using gnutls with a client x509 certificate to talk to an 
https server. The server is a Microsoft "Azure API Management" 
instance, which has a particular way of handling client 
certificates. It accepts an initial handshake without requesting a 
certificate. It then awaits the http "Host: " header. Only after 
that, it requests a rehandshake on the TLS layer, expecting the 
client to send a certificate.

Gnutls-cli handles this fine, and so does curl. But within emacs, 
it seems that the "rehandshake" feature isn't implemented. From 
what I could deduce, gnutls expects one to either:

- Handle GNUTLS_E_REHANDSHAKE as a return value to gnutls function 
  calls, OR
- Pass GNUTLS_AUTO_REAUTH as a flag into gnutls_init

It seems emacs isn't doing either. Probably because rehandshake in 
this way is a little exotic.

I can share log output of the gnutls-cli call if requested; I've 
included log output of the failed gnutls call from within emacs 
below.

Kind regards,
Jan Ypma

------------------------------------------------------------------

In GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 
3.24.22, cairo version 1.17.3)
 of 2020-08-28 built on juergen
Windowing system distributor 'The X.Org Foundation', version 
11.0.12010000
System Description: Arch Linux

Recent messages:


gnutls.c: [3] ASSERT: 
x509.c[gnutls_x509_crt_get_issuer_unique_id]:3986

gnutls.c: [3] ASSERT: 
x509.c[gnutls_x509_crt_get_subject_unique_id]:3936

gnutls.c: [3] ASSERT: x509.c[_gnutls_get_key_id]:3028

gnutls.c: [3] ASSERT: mpi.c[wrap_nettle_mpi_print]:60

gnutls.c: [3] ASSERT: mpi.c[wrap_nettle_mpi_print]:60

gnutls.c: [3] ASSERT: common.c[_gnutls_copy_string]:1582

gnutls.c: [3] ASSERT: x509.c[gnutls_x509_crt_export]:2955

gnutls.c: [5] REC[0x56136629bac0]: Preparing Packet Application 
Data(23) with length: 219 and min pad: 0

gnutls.c: [5] REC[0x56136629bac0]: Sent Packet[2] Application 
Data(23) in epoch 1 and length: 248

gnutls.c: [5] REC[0x56136629bac0]: SSL 3.3 Handshake packet 
received. Epoch 1, length: 28

gnutls.c: [5] REC[0x56136629bac0]: Expected Packet Application 
Data(23)

gnutls.c: [5] REC[0x56136629bac0]: Received Packet Handshake(22) 
with length: 28

gnutls.c: [5] REC[0x56136629bac0]: Decrypted Packet[1] 
Handshake(22) with length: 4

gnutls.c: [3] ASSERT: record.c[_gnutls_recv_in_buffers]:1578

gnutls.c: [3] ASSERT: record.c[_gnutls_recv_int]:1776

gnutls.c: [1] (Emacs) non-fatal error: Rehandshake was requested 
by the peer.
gnutls.c: [2] (Emacs) Deallocating x509 credentials
gnutls.c: [5] REC[0x56136629bac0]: Start of epoch cleanup

gnutls.c: [5] REC[0x56136629bac0]: End of epoch cleanup

gnutls.c: [5] REC[0x56136629bac0]: Epoch #1 freed

error in process sentinel: restclient-http-handle-response: 
peculiar error
error in process sentinel: peculiar error
progn: Restclient encountered an error


Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft 
 --with-wide-int
 --with-modules --with-cairo --with-harfbuzz 'CFLAGS=-march=x86-64
 -mtune=generic -O2 -pipe -fno-plt' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'
Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB 
NOTIFY
INOTIFY ACL GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD 
JSON
PDUMPER LCMS2 GMP
Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  dap-tooltip-mode: t
  dap-ui-many-windows-mode: t
  dap-ui-controls-mode: t
  dap-ui-mode: t
  treemacs-filewatch-mode: t
  treemacs-follow-mode: t
  treemacs-git-mode: deferred
  treemacs-fringe-indicator-mode: t
  dap-auto-configure-mode: t
  dap-mode: t
  global-company-mode: t
  company-mode: t
  magit-todos-mode: t
  global-magit-file-mode: t
  magit-auto-revert-mode: t
  rainbow-mode: t
  rainbow-delimiters-mode: t
  ws-butler-mode: t
  highlight-indent-guides-mode: t
  doom-modeline-mode: t
  eyebrowse-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  global-git-commit-mode: t
  shell-dirtrack-mode: t
  electric-pair-mode: t
  global-auto-revert-mode: t
  flx-ido-mode: t
  recentf-mode: t
  global-git-gutter-mode: t
  flyspell-mode: t
  global-hl-line-mode: t
  goto-address-prog-mode: t
  ido-ubiquitous-mode: t
  ido-vertical-mode: t
  ido-everywhere: t
  which-key-mode: t
  projectile-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  save-place-mode: t
  show-paren-mode: t
  global-subword-mode: t
  subword-mode: t
  override-global-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  global-prettify-symbols-mode: t
  prettify-symbols-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  hs-minor-mode: t

Load-path shadows:
/home/jan/.emacs.d/git/hide-lines/hide-lines hides 
~/.emacs.d/lisp/hide-lines
~/.emacs.d/lisp/dired-git-info hides 
/home/jan/.emacs.d/elpa/dired-git-info-0.2/dired-git-info

Features:
(shadow emacsbug cl-print flycheck-rust lsp-diagnostics 
lsp-modeline
dap-mouse lsp-ui lsp-ui-flycheck lsp-ui-doc lsp-ui-imenu 
lsp-ui-peek
lsp-ui-sideline flycheck lsp-ui-util lsp-steep lsp-svelte lsp-sqls
lsp-yaml lsp-xml lsp-vimscript lsp-vhdl lsp-vetur lsp-html 
lsp-verilog
lsp-vala lsp-terraform lsp-tex lsp-sorbet lsp-solargraph lsp-rust 
lsp-rf
lsp-r lsp-purescript lsp-pyls lsp-pwsh lsp-php lsp-perl lsp-ocaml
lsp-nix lsp-nim lsp-lua lsp-kotlin lsp-json lsp-javascript 
lsp-haxe
lsp-groovy lsp-hack lsp-go lsp-completion lsp-gdscript lsp-fsharp
lsp-fortran lsp-eslint lsp-erlang lsp-elixir lsp-elm 
lsp-dockerfile
lsp-dhall lsp-css lsp-csharp lsp-crystal lsp-cmake lsp-clojure
lsp-clangd lsp-bash lsp-angular lsp-ada lsp-actionscript dap-ui 
gdb-mi
gud bui bui-list bui-info bui-entry bui-core bui-history 
bui-button
bui-utils dap-java lsp-java request lsp-metals lsp-metals-treeview
lsp-treemacs lsp-treemacs-themes treemacs treemacs-header-line
treemacs-compatibility treemacs-mode treemacs-bookmarks
treemacs-interface treemacs-extensions treemacs-mouse-interface
treemacs-tags treemacs-persistence treemacs-filewatch-mode
treemacs-follow-mode treemacs-rendering treemacs-async
treemacs-workspaces treemacs-dom treemacs-visuals
treemacs-fringe-indicator treemacs-scope pulse treemacs-faces
treemacs-icons treemacs-themes treemacs-core-utils pfuture
treemacs-logging treemacs-customization treemacs-macros
lsp-metals-protocol lsp-lens dap-mode dap-launch posframe 
dap-overlays
lsp-mode lsp-protocol spinner lv inline bindat scala-mode
scala-mode-prettify-symbols scala-mode-imenu scala-mode-map
scala-mode-fontlock scala-mode-indent scala-mode-paragraph
scala-mode-syntax scala-mode-lib cal-move reveal help-fns
company-oddmuse company-keywords company-etags etags fileloop xref
project company-gtags company-dabbrev-code company-dabbrev 
company-files
company-clang company-capf company-cmake company-semantic
company-template company-bbdb company-emoji company-emoji-list 
company
magit-extras magit-todos pcre2el rxt re-builder async forge-list
forge-commands forge-semi forge-bitbucket buck forge-gogs gogs
forge-gitea gtea forge-gitlab glab forge-github ghub-graphql 
treepy
gsexp ghub let-alist forge-notify forge-revnote forge-pullreq
forge-issue forge-topic bug-reference forge-post forge-repo forge
forge-core forge-db closql emacsql-sqlite emacsql emacsql-compiler
magit-bookmark magit-submodule magit-obsolete magit-blame 
magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch 
magit-clone
magit-remote magit-commit magit-sequence magit-notes 
magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files 
magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
which-func magit-diff smerge-mode magit-core magit-autorevert
magit-margin magit-transient magit-process magit-mode 
expand-region
subword-mode-expansions text-mode-expansions cc-mode-expansions
the-org-mode-expansions nxml-mode-expansions js-mode-expansions
html-mode-expansions er-basic-expansions expand-region-core
expand-region-custom ace-window avy url-http url-gw url-cache 
url-auth
org-superstar ol-eww ol-rmail ol-mhe ol-irc ol-info ol-gnus nnir
gnus-sum gnus-group gnus-undo gnus-start gnus-cloud nnimap nnmail
mail-source utf7 netrc nnoo gnus-spec gnus-int gnus-range gnus-win 
gnus
nnheader ol-docview ol-bibtex bibtex ol-bbdb ol-w3m dired-aux 
rng-xsd
xsd-regexp rng-cmpct kubectl smex vc-mtn vc-hg vc-bzr vc-src 
vc-sccs
vc-svn vc-cvs vc-rcs adaptive-wrap yaml-mode misearch 
multi-isearch
dired-collapse diff-hl-dired diff-hl log-view vc-dir ewoc vc
vc-dispatcher elfeed-show elfeed-search elfeed-csv elfeed 
elfeed-curl
elfeed-log elfeed-db elfeed-lib url-queue xml-query gnutls
network-stream nsm mailalias qp mail-extr sort shr svg mu4e-alert 
ht
alert log4e notifications gntp mu4e desktop frameset mu4e-org 
face-remap
rainbow-mode rainbow-delimiters ws-butler hideshow
highlight-indent-guides doom-modeline doom-modeline-segments
doom-modeline-env doom-modeline-core shrink-path f 
page-break-lines
display-line-numbers linum tramp-gvfs zeroconf dbus xml tramp-sh
org-tree-slide org-timer org-clock daemons ob-restclient 
restclient url
url-proxy url-privacy url-expand url-methods url-history 
url-cookie
url-domsuf url-util mailcap ob-plantuml visual-regexp noccur 
ob-http
ob-http-mode s eyebrowse yasnippet hide-lines docker-tramp 
tramp-cache
tramp tramp-loaddefs trampver tramp-integration files-x 
tramp-compat
parse-time iso8601 ls-lisp git-auto-commit-mode pcase 
markdown-mode
edit-indirect color ox-md ox-odt ox-latex ox-icalendar ox-html 
table
ox-ascii ox-publish ox org-element avl-tree generator org-expiry
ob-python ob-shell ob-sql org ob ob-tangle ob-ref ob-lob ob-table 
ob-exp
org-macro org-footnote org-src ob-comint org-pcomplete org-list
org-faces org-entities noutline outline org-version ob-emacs-lisp
ob-core ob-eval org-table ol org-keys org-compat org-macs 
org-loaddefs
find-func dired-x disp-table rng-nxml rng-valid rng-loc rng-uri
rng-parse nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns
nxml-mode nxml-outln nxml-rap sgml-mode dom nxml-util nxml-enc 
xmltok
git-commit with-editor shell pcomplete server rx log-edit 
pcvs-util
add-log magit-git magit-section magit-utils crm elec-pair 
protobuf-mode
cc-langs c-block-info-inline-mode autorevert filenotify flx-ido 
flx
dashboard dashboard-widgets time recentf tree-widget git-gutter
flyspell-popup popup flyspell ispell js imenu cc-mode cc-fonts 
cc-guess
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
mu4e-contrib eshell esh-cmd esh-ext esh-opt esh-proc esh-io 
esh-arg
esh-module esh-groups esh-util bookmark pp mu4e-view cal-menu 
calendar
cal-loaddefs mu4e-headers mu4e-compose mu4e-draft mu4e-actions 
rfc2368
mu4e-mark mu4e-message flow-fill mu4e-proc mule-util hl-line 
mu4e-main
mu4e-context mu4e-utils doc-view jka-compr image-mode exif 
mu4e-lists
mu4e-vars message rmc puny rfc822 mml mml-sec epa epg epg-config
gnus-util rmail rmail-loaddefs text-property-search time-date 
mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils 
mailheader
mu4e-meta smtpmail sendmail rfc2047 rfc2045 ietf-drums mm-util
mail-prsvr mail-utils dired-rainbow dired-hacks-utils dash 
dired-du
find-dired dired dired-loaddefs goto-addr vc-git diff-mode 
autoload
radix-tree lisp-mnt ido-completing-read+ memoize cus-edit wid-edit
minibuf-eldef ido-vertical-mode ido which-key advice hl-todo 
transient
format-spec ibuffer-projectile projectile grep compile comint 
ansi-color
ring ibuf-ext ibuffer ibuffer-loaddefs thingatpt ordbog uuidgen
calc-misc calc-ext calc calc-loaddefs calc-macs colorpicker 
derived view
backup-walker cl undo-tree diff cl-extra help-mode all-the-icons
all-the-icons-faces data-material data-weathericons data-octicons
data-fileicons data-faicons data-alltheicons saveplace 
misterioso-theme
paren cap-words superword subword cus-start cus-load thread-dump
whitespace use-package use-package-ensure use-package-delight
use-package-diminish use-package-bind-key bind-key easy-mmode
use-package-core finder-inf edmacro kmacro info package easymenu
browse-url url-handlers url-parse auth-source cl-seq eieio 
eieio-core
cl-macs eieio-loaddefs password-cache json subr-x map url-vars seq
byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip 
eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel 
term/x-win
x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt 
fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu 
font-core
term/tty-colors frame minibuffer cl-generic cham georgian 
utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese 
eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic 
indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice 
loaddefs
button faces cus-face macroexp files text-properties overlay sha1 
md5
base64 format env code-pages mule custom widget 
hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting cairo move-toolbar gtk 
x-toolkit
x multi-tty make-network-process emacs)

Memory information:
((conses 16 1964416 173703)
 (symbols 48 70107 1)
 (strings 32 335397 40695)
 (string-bytes 1 10519056)
 (vectors 16 235483)
 (vector-slots 8 6383589 348424)
 (floats 8 3220 4352)
 (intervals 56 82158 17356)
 (buffers 1000 108))

bug#47321: 27.1; gnutls.c does not handle TLS rehandshake from server

[Lars Ingebrigtsen]

Jan Ypma <jan@ypmania.nl> writes:

> I'm using gnutls with a client x509 certificate to talk to an https
> server. The server is a Microsoft "Azure API Management" instance,
> which has a particular way of handling client certificates. It accepts
> an initial handshake without requesting a certificate. It then awaits
> the http "Host: " header. Only after that, it requests a rehandshake
> on the TLS layer, expecting the client to send a certificate.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

Would it be possible to create a complete test case for this, by any
chance?  Without a test case, it's difficult to start fixing this issue,
but I realise that it may be difficult to set one up, since it requires
certificates etc...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47321: 27.1; gnutls.c does not handle TLS rehandshake from server

[Jan Ypma]

Unfortunately, I don't have access to the related Azure 
infrastructure directly. And, with this being a closed-source 3rd 
party cloud service, they could have upgraded/changed their server 
behavior and nobody would know :).

I suggest we leave the discussion open, perhaps others on Azure 
that /are/ able to use GnuTLS could chime in. 

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Jan Ypma <jan@ypmania.nl> writes:
>
>> I'm using gnutls with a client x509 certificate to talk to an 
>> https
>> server. The server is a Microsoft "Azure API Management" 
>> instance,
>> which has a particular way of handling client certificates. It 
>> accepts
>> an initial handshake without requesting a certificate. It then 
>> awaits
>> the http "Host: " header. Only after that, it requests a 
>> rehandshake
>> on the TLS layer, expecting the client to send a certificate.
>
> (I'm going through old bug reports that unfortunately weren't 
> resolved
> at the time.)
>
> Would it be possible to create a complete test case for this, by 
> any
> chance?  Without a test case, it's difficult to start fixing 
> this issue,
> but I realise that it may be difficult to set one up, since it 
> requires
> certificates etc...

bug#47321: 27.1; gnutls.c does not handle TLS rehandshake from server

[Robert Pluim]

>>>>> On Thu, 07 Jul 2022 10:48:15 +0200, Jan Ypma <jan@ypmania.net> said:

    Jan> Unfortunately, I don't have access to the related Azure infrastructure
    Jan> directly. And, with this being a closed-source 3rd party cloud
    Jan> service, they could have upgraded/changed their server behavior and
    Jan> nobody would know :).

    Jan> I suggest we leave the discussion open, perhaps others on Azure that
    Jan> /are/ able to use GnuTLS could chime in.

You can actually test this with ʼopenssl s_serverʼ

The following patch works for me, but only with TLS1.2. For some
reason GnuTLS and OpenSSL donʼt want to enable secure renegotiation
when talking TLS1.3 to each other. Also, youʼre supposed to loop checking
for GNUTLS_E_AGAIN when calling gnutls_record_recv, but if I do that
Emacs ends up spinning at 100% CPU sometimes.

If you want to force TLS1.2, youʼll have to set
gnutls-algorithm-priority to something like
"NORMAL:%DUMBFW:!VERS-ALL:+VERS-TLS1.2"

TL;DR Probably not ready for primetime

You can run the server like so:

    cd emacs/test/lisp/net/network-stream-resources
    openssl s_server -port 6666 -cert cert.pem -key key.pem -tls1_2

For the client, something like this:

(defun rtest ()
  (require 'gnutls)
  (require 'network-stream)
  (let ((proc (open-gnutls-stream "rtest" "rtest" "localhost" 6666)))
    (process-send-string proc "Host:\n")
    (switch-to-buffer "rtest")))

then in the terminal where the openssl server is running, you can use
the 'r' key to request renegotiation (or just type random stuff to
have it be echoed back to Emacs).

diff --git a/src/gnutls.c b/src/gnutls.c
index a0de0238c4..846721ba86 100644
--- a/src/gnutls.c
+++ b/src/gnutls.c
@@ -74,6 +74,14 @@
 #  define HAVE_GNUTLS_AEAD
 # endif
 
+# if GNUTLS_VERSION_NUMBER >= 0x030605
+#  define HAVE_GNUTLS_AUTO_REAUTH
+# endif
+
+# if GNUTLS_VERSION_NUMBER >= 0x030610
+#  define HAVE_GNUTLS_POST_HANDSHAKE_AUTH
+# endif
+
 # ifdef WINDOWSNT
 #  include <windows.h>
 #  include "w32common.h"
@@ -771,11 +779,16 @@ emacs_gnutls_read (struct Lisp_Process *proc, char *buf, ptrdiff_t nbyte)
       return -1;
     }
 
+  /* We have to handle GNUTLS_E_AGAIN here for the case where Secure
+     Renegotiation happens.  The renegotiation itself is handled
+     inside GnuTLS, but we need to retry the gnutls_record_recv for
+     things to actually work.  */
   ssize_t rtnval;
   do
     rtnval = gnutls_record_recv (state, buf, nbyte);
   while (rtnval == GNUTLS_E_INTERRUPTED);
-
+  if (rtnval == GNUTLS_E_AGAIN)
+    rtnval = gnutls_record_recv (state, buf, nbyte);
   if (rtnval >= 0)
     return rtnval;
   else if (rtnval == GNUTLS_E_UNEXPECTED_PACKET_LENGTH)
@@ -2066,6 +2079,12 @@ DEFUN ("gnutls-boot", Fgnutls_boot, Sgnutls_boot, 3, 3, 0,
   if (XPROCESS (proc)->is_non_blocking_client)
     gnutls_flags |= GNUTLS_NONBLOCK;
 # endif
+# ifdef HAVE_GNUTLS_AUTO_REAUTH
+  gnutls_flags |= GNUTLS_AUTO_REAUTH;
+# endif
+# ifdef HAVE_GNUTLS_POST_HANDSHAKE_AUTH
+  gnutls_flags |= GNUTLS_POST_HANDSHAKE_AUTH;
+#endif
   ret = gnutls_init (&state, gnutls_flags);
   XPROCESS (proc)->gnutls_state = state;
   if (ret < GNUTLS_E_SUCCESS)

Robert
--