bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

I think that's a bug...  I haven't looked into whether the gnutls
library allows very async processes, but I think it does.



In GNU Emacs 25.1.50.186 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.7)
 of 2016-01-30 built on mouse
Repository revision: e5b2f25f3560eb6b590de8391b6a71c2178ff591
Windowing system distributor 'The X.Org Foundation', version 11.0.11702000
System Description:	Ubuntu 15.10

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LC_MONETARY: nb_NO.UTF-8
  value of $LC_NUMERIC: nb_NO.UTF-8
  value of $LC_TIME: nb_NO.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  edebug-mode: t
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t

Recent messages:

Result: 443 (#o673, #x1bb, ?ƻ)

Result: (:type tls :nowait t)
network-stream-open-tls
open-network-stream
Mark saved where search started
Type C-x 1 to delete the help window, C-M-v to scroll help. [2 times]
user-error: No cross-reference here
Mark saved where search started

Load-path shadows:
/home/larsi/mgnus/lisp/md4 hides /home/larsi/src/emacs/async-dns/lisp/md4
/home/larsi/mgnus/lisp/color hides /home/larsi/src/emacs/async-dns/lisp/color
/home/larsi/mgnus/lisp/format-spec hides /home/larsi/src/emacs/async-dns/lisp/format-spec
/home/larsi/mgnus/lisp/hex-util hides /home/larsi/src/emacs/async-dns/lisp/hex-util
/home/larsi/mgnus/lisp/password-cache hides /home/larsi/src/emacs/async-dns/lisp/password-cache
/home/larsi/mgnus/lisp/dns-mode hides /home/larsi/src/emacs/async-dns/lisp/textmodes/dns-mode
/home/larsi/mgnus/lisp/dns hides /home/larsi/src/emacs/async-dns/lisp/net/dns
/home/larsi/mgnus/lisp/ntlm hides /home/larsi/src/emacs/async-dns/lisp/net/ntlm
/home/larsi/mgnus/lisp/rfc2104 hides /home/larsi/src/emacs/async-dns/lisp/net/rfc2104
/home/larsi/mgnus/lisp/sasl-scram-rfc hides /home/larsi/src/emacs/async-dns/lisp/net/sasl-scram-rfc
/home/larsi/mgnus/lisp/dig hides /home/larsi/src/emacs/async-dns/lisp/net/dig
/home/larsi/mgnus/lisp/sasl-cram hides /home/larsi/src/emacs/async-dns/lisp/net/sasl-cram
/home/larsi/mgnus/lisp/sasl-ntlm hides /home/larsi/src/emacs/async-dns/lisp/net/sasl-ntlm
/home/larsi/mgnus/lisp/netrc hides /home/larsi/src/emacs/async-dns/lisp/net/netrc
/home/larsi/mgnus/lisp/sasl hides /home/larsi/src/emacs/async-dns/lisp/net/sasl
/home/larsi/mgnus/lisp/sasl-digest hides /home/larsi/src/emacs/async-dns/lisp/net/sasl-digest
/home/larsi/mgnus/lisp/hmac-def hides /home/larsi/src/emacs/async-dns/lisp/net/hmac-def
/home/larsi/mgnus/lisp/tls hides /home/larsi/src/emacs/async-dns/lisp/net/tls
/home/larsi/mgnus/lisp/hmac-md5 hides /home/larsi/src/emacs/async-dns/lisp/net/hmac-md5
/home/larsi/mgnus/lisp/binhex hides /home/larsi/src/emacs/async-dns/lisp/mail/binhex
/home/larsi/mgnus/lisp/hashcash hides /home/larsi/src/emacs/async-dns/lisp/mail/hashcash
/home/larsi/mgnus/lisp/uudecode hides /home/larsi/src/emacs/async-dns/lisp/mail/uudecode
/home/larsi/mgnus/lisp/mail-source hides /home/larsi/src/emacs/async-dns/lisp/gnus/mail-source
/home/larsi/mgnus/lisp/gnus-ml hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-ml
/home/larsi/mgnus/lisp/gmm-utils hides /home/larsi/src/emacs/async-dns/lisp/gnus/gmm-utils
/home/larsi/mgnus/lisp/gnus-sum hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-sum
/home/larsi/mgnus/lisp/gnus-fun hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-fun
/home/larsi/mgnus/lisp/rfc2231 hides /home/larsi/src/emacs/async-dns/lisp/gnus/rfc2231
/home/larsi/mgnus/lisp/nnmbox hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnmbox
/home/larsi/mgnus/lisp/nnimap hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnimap
/home/larsi/mgnus/lisp/gnus-score hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-score
/home/larsi/mgnus/lisp/rfc2047 hides /home/larsi/src/emacs/async-dns/lisp/gnus/rfc2047
/home/larsi/mgnus/lisp/ietf-drums hides /home/larsi/src/emacs/async-dns/lisp/gnus/ietf-drums
/home/larsi/mgnus/lisp/gnus-mh hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-mh
/home/larsi/mgnus/lisp/gnus-vm hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-vm
/home/larsi/mgnus/lisp/ecomplete hides /home/larsi/src/emacs/async-dns/lisp/gnus/ecomplete
/home/larsi/mgnus/lisp/nndoc hides /home/larsi/src/emacs/async-dns/lisp/gnus/nndoc
/home/larsi/mgnus/lisp/plstore hides /home/larsi/src/emacs/async-dns/lisp/gnus/plstore
/home/larsi/mgnus/lisp/nndiary hides /home/larsi/src/emacs/async-dns/lisp/gnus/nndiary
/home/larsi/mgnus/lisp/spam hides /home/larsi/src/emacs/async-dns/lisp/gnus/spam
/home/larsi/mgnus/lisp/spam-stat hides /home/larsi/src/emacs/async-dns/lisp/gnus/spam-stat
/home/larsi/mgnus/lisp/spam-report hides /home/larsi/src/emacs/async-dns/lisp/gnus/spam-report
/home/larsi/mgnus/lisp/sieve-mode hides /home/larsi/src/emacs/async-dns/lisp/gnus/sieve-mode
/home/larsi/mgnus/lisp/gnus-undo hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-undo
/home/larsi/mgnus/lisp/gnus-agent hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-agent
/home/larsi/mgnus/lisp/mm-uu hides /home/larsi/src/emacs/async-dns/lisp/gnus/mm-uu
/home/larsi/mgnus/lisp/utf7 hides /home/larsi/src/emacs/async-dns/lisp/gnus/utf7
/home/larsi/mgnus/lisp/nnmail hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnmail
/home/larsi/mgnus/lisp/gnus-async hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-async
/home/larsi/mgnus/lisp/mm-decode hides /home/larsi/src/emacs/async-dns/lisp/gnus/mm-decode
/home/larsi/mgnus/lisp/gnus-topic hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-topic
/home/larsi/mgnus/lisp/gnus-dired hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-dired
/home/larsi/mgnus/lisp/gnus-win hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-win
/home/larsi/mgnus/lisp/yenc hides /home/larsi/src/emacs/async-dns/lisp/gnus/yenc
/home/larsi/mgnus/lisp/rfc1843 hides /home/larsi/src/emacs/async-dns/lisp/gnus/rfc1843
/home/larsi/mgnus/lisp/nnvirtual hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnvirtual
/home/larsi/mgnus/lisp/gnus-util hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-util
/home/larsi/mgnus/lisp/gnus-draft hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-draft
/home/larsi/mgnus/lisp/gnus-registry hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-registry
/home/larsi/mgnus/lisp/nndraft hides /home/larsi/src/emacs/async-dns/lisp/gnus/nndraft
/home/larsi/mgnus/lisp/nnnil hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnnil
/home/larsi/mgnus/lisp/gnus-eform hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-eform
/home/larsi/mgnus/lisp/nnfolder hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnfolder
/home/larsi/mgnus/lisp/nndir hides /home/larsi/src/emacs/async-dns/lisp/gnus/nndir
/home/larsi/mgnus/lisp/gnus-cus hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-cus
/home/larsi/mgnus/lisp/nnir hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnir
/home/larsi/mgnus/lisp/gnus-srvr hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-srvr
/home/larsi/mgnus/lisp/gnus-dup hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-dup
/home/larsi/mgnus/lisp/gssapi hides /home/larsi/src/emacs/async-dns/lisp/gnus/gssapi
/home/larsi/mgnus/lisp/gnus-spec hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-spec
/home/larsi/mgnus/lisp/nnoo hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnoo
/home/larsi/mgnus/lisp/gnus-salt hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-salt
/home/larsi/mgnus/lisp/gnus-sieve hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-sieve
/home/larsi/mgnus/lisp/mm-extern hides /home/larsi/src/emacs/async-dns/lisp/gnus/mm-extern
/home/larsi/mgnus/lisp/nnregistry hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnregistry
/home/larsi/mgnus/lisp/nnspool hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnspool
/home/larsi/mgnus/lisp/sieve-manage hides /home/larsi/src/emacs/async-dns/lisp/gnus/sieve-manage
/home/larsi/mgnus/lisp/mm-encode hides /home/larsi/src/emacs/async-dns/lisp/gnus/mm-encode
/home/larsi/mgnus/lisp/gnus-bookmark hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-bookmark
/home/larsi/mgnus/lisp/mm-util hides /home/larsi/src/emacs/async-dns/lisp/gnus/mm-util
/home/larsi/mgnus/lisp/gnus-uu hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-uu
/home/larsi/mgnus/lisp/gnus-diary hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-diary
/home/larsi/mgnus/lisp/mml-smime hides /home/larsi/src/emacs/async-dns/lisp/gnus/mml-smime
/home/larsi/mgnus/lisp/nnweb hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnweb
/home/larsi/mgnus/lisp/nnmaildir hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnmaildir
/home/larsi/mgnus/lisp/mm-partial hides /home/larsi/src/emacs/async-dns/lisp/gnus/mm-partial
/home/larsi/mgnus/lisp/gnus-html hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-html
/home/larsi/mgnus/lisp/nngateway hides /home/larsi/src/emacs/async-dns/lisp/gnus/nngateway
/home/larsi/mgnus/lisp/compface hides /home/larsi/src/emacs/async-dns/lisp/gnus/compface
/home/larsi/mgnus/lisp/mml2015 hides /home/larsi/src/emacs/async-dns/lisp/gnus/mml2015
/home/larsi/mgnus/lisp/mailcap hides /home/larsi/src/emacs/async-dns/lisp/gnus/mailcap
/home/larsi/mgnus/lisp/mm-view hides /home/larsi/src/emacs/async-dns/lisp/gnus/mm-view
/home/larsi/mgnus/lisp/flow-fill hides /home/larsi/src/emacs/async-dns/lisp/gnus/flow-fill
/home/larsi/mgnus/lisp/mml1991 hides /home/larsi/src/emacs/async-dns/lisp/gnus/mml1991
/home/larsi/mgnus/lisp/nnrss hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnrss
/home/larsi/mgnus/lisp/gnus-cite hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-cite
/home/larsi/mgnus/lisp/gnus-logic hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-logic
/home/larsi/mgnus/lisp/gnus-icalendar hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-icalendar
/home/larsi/mgnus/lisp/mm-bodies hides /home/larsi/src/emacs/async-dns/lisp/gnus/mm-bodies
/home/larsi/mgnus/lisp/gnus-bcklg hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-bcklg
/home/larsi/mgnus/lisp/gnus hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus
/home/larsi/mgnus/lisp/mail-prsvr hides /home/larsi/src/emacs/async-dns/lisp/gnus/mail-prsvr
/home/larsi/mgnus/lisp/nnagent hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnagent
/home/larsi/mgnus/lisp/nneething hides /home/larsi/src/emacs/async-dns/lisp/gnus/nneething
/home/larsi/mgnus/lisp/rtree hides /home/larsi/src/emacs/async-dns/lisp/gnus/rtree
/home/larsi/mgnus/lisp/legacy-gnus-agent hides /home/larsi/src/emacs/async-dns/lisp/gnus/legacy-gnus-agent
/home/larsi/mgnus/lisp/rfc2045 hides /home/larsi/src/emacs/async-dns/lisp/gnus/rfc2045
/home/larsi/mgnus/lisp/gnus-picon hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-picon
/home/larsi/mgnus/lisp/gnus-delay hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-delay
/home/larsi/mgnus/lisp/gnus-ems hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-ems
/home/larsi/mgnus/lisp/gravatar hides /home/larsi/src/emacs/async-dns/lisp/gnus/gravatar
/home/larsi/mgnus/lisp/gnus-gravatar hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-gravatar
/home/larsi/mgnus/lisp/smiley hides /home/larsi/src/emacs/async-dns/lisp/gnus/smiley
/home/larsi/mgnus/lisp/gnus-demon hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-demon
/home/larsi/mgnus/lisp/mail-parse hides /home/larsi/src/emacs/async-dns/lisp/gnus/mail-parse
/home/larsi/mgnus/lisp/nnheader hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnheader
/home/larsi/mgnus/lisp/gnus-cloud hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-cloud
/home/larsi/mgnus/lisp/gnus-sync hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-sync
/home/larsi/mgnus/lisp/gnus-start hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-start
/home/larsi/mgnus/lisp/nntp hides /home/larsi/src/emacs/async-dns/lisp/gnus/nntp
/home/larsi/mgnus/lisp/gnus-mlspl hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-mlspl
/home/larsi/mgnus/lisp/nnbabyl hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnbabyl
/home/larsi/mgnus/lisp/registry hides /home/larsi/src/emacs/async-dns/lisp/gnus/registry
/home/larsi/mgnus/lisp/nnml hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnml
/home/larsi/mgnus/lisp/html2text hides /home/larsi/src/emacs/async-dns/lisp/gnus/html2text
/home/larsi/mgnus/lisp/gnus-range hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-range
/home/larsi/mgnus/lisp/gnus-cache hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-cache
/home/larsi/mgnus/lisp/nnmairix hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnmairix
/home/larsi/mgnus/lisp/mml hides /home/larsi/src/emacs/async-dns/lisp/gnus/mml
/home/larsi/mgnus/lisp/gnus-group hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-group
/home/larsi/mgnus/lisp/mml-sec hides /home/larsi/src/emacs/async-dns/lisp/gnus/mml-sec
/home/larsi/mgnus/lisp/starttls hides /home/larsi/src/emacs/async-dns/lisp/gnus/starttls
/home/larsi/mgnus/lisp/score-mode hides /home/larsi/src/emacs/async-dns/lisp/gnus/score-mode
/home/larsi/mgnus/lisp/gnus-art hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-art
/home/larsi/mgnus/lisp/smime hides /home/larsi/src/emacs/async-dns/lisp/gnus/smime
/home/larsi/mgnus/lisp/spam-wash hides /home/larsi/src/emacs/async-dns/lisp/gnus/spam-wash
/home/larsi/mgnus/lisp/sieve hides /home/larsi/src/emacs/async-dns/lisp/gnus/sieve
/home/larsi/mgnus/lisp/deuglify hides /home/larsi/src/emacs/async-dns/lisp/gnus/deuglify
/home/larsi/mgnus/lisp/canlock hides /home/larsi/src/emacs/async-dns/lisp/gnus/canlock
/home/larsi/mgnus/lisp/message hides /home/larsi/src/emacs/async-dns/lisp/gnus/message
/home/larsi/mgnus/lisp/gnus-int hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-int
/home/larsi/mgnus/lisp/nnmh hides /home/larsi/src/emacs/async-dns/lisp/gnus/nnmh
/home/larsi/mgnus/lisp/messcompat hides /home/larsi/src/emacs/async-dns/lisp/gnus/messcompat
/home/larsi/mgnus/lisp/mm-archive hides /home/larsi/src/emacs/async-dns/lisp/gnus/mm-archive
/home/larsi/mgnus/lisp/gnus-msg hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-msg
/home/larsi/mgnus/lisp/gnus-kill hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-kill
/home/larsi/mgnus/lisp/mm-url hides /home/larsi/src/emacs/async-dns/lisp/gnus/mm-url
/home/larsi/mgnus/lisp/qp hides /home/larsi/src/emacs/async-dns/lisp/gnus/qp
/home/larsi/mgnus/lisp/gnus-notifications hides /home/larsi/src/emacs/async-dns/lisp/gnus/gnus-notifications
/home/larsi/mgnus/lisp/pop3 hides /home/larsi/src/emacs/async-dns/lisp/gnus/pop3
/home/larsi/mgnus/lisp/auth-source hides /home/larsi/src/emacs/async-dns/lisp/gnus/auth-source
/home/larsi/mgnus/lisp/time-date hides /home/larsi/src/emacs/async-dns/lisp/calendar/time-date
/home/larsi/mgnus/lisp/parse-time hides /home/larsi/src/emacs/async-dns/lisp/calendar/parse-time

Features:
(shadow ecomplete emacsbug sendmail eieio-opt speedbar sb-image ezimage
dframe find-func grep compile edebug misearch multi-isearch timezone
shr-color color eww qp gnus-html url-queue url-cache mm-archive server
sort smiley gnus-cite gnus-async gnus-dup gnus-ml gmane spam-gmane dns
mm-url disp-table log-edit vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs
vc-dir ewoc bug-reference whitespace log-view pcvs-util vc vc-dispatcher
vc-git diff-mode easy-mmode map cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs gnus-fun gnus-mdrtn
gnus-topic nndraft nnmh utf-7 nnfolder copyright network-stream nsm
starttls nnir spam-report spam spam-stat gnus-uu yenc gnus-agent
gnus-srvr gnus-score score-mode nnvirtual gnus-msg gnus-art mm-uu
mml2015 mm-view mml-smime smime dig nntp gnus-cache gnus-sum gnus-group
gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source utf7 netrc
nnoo parse-time gnus-spec gnus-int gnus-range message rfc822 mml mml-sec
mailabbrev gmm-utils mailheader gnus-win gnus-load gnus gnus-ems
gnus-compat nnheader mail-utils tramp-cache tramp tramp-compat
tramp-loaddefs trampver ucs-normalize shell pcomplete comint ansi-color
ring format-spec advice movie mkv shr subr-x browse-url imdb dom pvr
debug debbugs-gnu wid-edit debbugs soap-client mm-decode mm-bodies
mm-encode url-http tls gnutls url-auth mail-parse rfc2231 rfc2047
rfc2045 ietf-drums url-gw puny url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util mailcap warnings
rng-xsd rng-dt rng-util xsd-regexp xml ido seq flyspell ispell dired
dired-loaddefs add-log mail-extr jka-compr cl finder-inf info package
epg-config url-handlers url-parse auth-source cl-seq eieio byte-opt
bytecomp byte-compile cl-extra cconv eieio-core cl-macs gv
eieio-loaddefs gnus-util mm-util help-fns help-mode easymenu cl-loaddefs
pcase cl-lib mail-prsvr password-cache url-vars time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt
fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register
page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock
font-lock syntax facemenu font-core frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese charscript case-table epa-hook
jka-cmpr-hook help simple abbrev obarray minibuffer cl-preloaded nadvice
loaddefs button faces cus-face macroexp files text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote dbusbind inotify dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 1550650 148329)
 (symbols 48 105632 4)
 (miscs 40 454 1318)
 (strings 32 173572 15311)
 (string-bytes 1 7554332)
 (vectors 16 66183)
 (vector-slots 8 1913821 206828)
 (floats 8 704 1263)
 (intervals 56 245488 6252)
 (buffers 976 53)
 (heap 1024 256242 70451))

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Looking at the code, and doing some experimentation, it looks like this
should be possible to implement.

First of all, gnutls-open-stream should call open-network-stream with
:nowait, and put a sentinel on the stream to see when it's opened, and
then it should call gnutls-negotiate from the sentinel.

The problem is, though, that the process isn't marked as a gnutls
process until gnutls_boot is called, so this needs to happen much
earlier.  Possibly by adding another keyword to make-network-process
that just sets

  XPROCESS (proc)->gnutls_p = 1;

The other problem is that reading/writing from these things will just
send plain text if the gnutls stuff hasn't been initialised:

#ifdef HAVE_GNUTLS
	      if (p->gnutls_p && p->gnutls_state)
		written = emacs_gnutls_write (p, cur_buf, cur_len);
	      else
#endif
		written = emacs_write_sig (outfd, cur_buf, cur_len);

I think that looks rather nonsensical.  If it's p->gnutls_p, then it
should never write to the process, no matter what the state is.  Rather
is should just skip writing until p->gnutls_state gets set (which
happens during gnutls_boot).

Or is there something subtle here I'm missing?  When would we ever want
to write plain text to something that's p->gnutls_p?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 30 Jan 2016 05:45:21 +0100
> 
> First of all, gnutls-open-stream should call open-network-stream with
> :nowait, and put a sentinel on the stream to see when it's opened, and
> then it should call gnutls-negotiate from the sentinel.
> 
> The problem is, though, that the process isn't marked as a gnutls
> process until gnutls_boot is called, so this needs to happen much
> earlier.  Possibly by adding another keyword to make-network-process
> that just sets
> 
>   XPROCESS (proc)->gnutls_p = 1;

Why do you need to mark a process gnutls_p, when it definitely still
isn't, not until the GnuTLS negotiation is successfully completed?
That'd just add confusion (a.k.a. "bugs waiting to happen") on the C
level.  Can't we use a GnuTLS-specific sentinel instead?

> The other problem is that reading/writing from these things will just
> send plain text if the gnutls stuff hasn't been initialised:
> 
> #ifdef HAVE_GNUTLS
> 	      if (p->gnutls_p && p->gnutls_state)
> 		written = emacs_gnutls_write (p, cur_buf, cur_len);
> 	      else
> #endif
> 		written = emacs_write_sig (outfd, cur_buf, cur_len);
> 
> I think that looks rather nonsensical.

Does this happen today?  If so, in what scenario?

> If it's p->gnutls_p, then it should never write to the process, no
> matter what the state is.  Rather is should just skip writing until
> p->gnutls_state gets set (which happens during gnutls_boot).
> 
> Or is there something subtle here I'm missing?  When would we ever want
> to write plain text to something that's p->gnutls_p?

If you want GnuTLS negotiation to happen from the sentinel, you need
some machinery to block such process objects from communicating.
I'm not sure silently skipping I/O is TRT for that: won't Lisp
programs that happen to start communicating too early become confused
about what's going on?

More generally, what advantages do we expect to gain by making these
changes?  Let DNS resolution proceed in the background, only to have
to wait for GnuTLS negotiations anyway, before we can start using the
stream?  Or are there more significant advantages?

Thanks.

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> The problem is, though, that the process isn't marked as a gnutls
>> process until gnutls_boot is called, so this needs to happen much
>> earlier.  Possibly by adding another keyword to make-network-process
>> that just sets
>> 
>>   XPROCESS (proc)->gnutls_p = 1;
>
> Why do you need to mark a process gnutls_p, when it definitely still
> isn't, not until the GnuTLS negotiation is successfully completed?

It is a socket that will only be used for TLS.  And the current way
we're creating TLS streams is by calling `open-gnutls-stream', which
(simplified) just calls `make-network-stream' and then `gnutls-boot' in
rapid succession.  So any stream you get out of `open-gnutls-stream'
will look just the same -- it'll have gnutls_p set.

It'll just happen slightly earlier.

> That'd just add confusion (a.k.a. "bugs waiting to happen") on the C
> level.  Can't we use a GnuTLS-specific sentinel instead?

I'm not sure I follow...  a sentinel?

>> The other problem is that reading/writing from these things will just
>> send plain text if the gnutls stuff hasn't been initialised:
>> 
>> #ifdef HAVE_GNUTLS
>> 	      if (p->gnutls_p && p->gnutls_state)
>> 		written = emacs_gnutls_write (p, cur_buf, cur_len);
>> 	      else
>> #endif
>> 		written = emacs_write_sig (outfd, cur_buf, cur_len);
>> 
>> I think that looks rather nonsensical.
>
> Does this happen today?  If so, in what scenario?

It does not happen today, but it will happen if we open the TLS stream
:nowait.  For instance, url.el calls `open-network-stream' and then
sends a "GET /" on the socket immediately.  Since setup is synchronous
today, that's fine, but when it gets async, then that p->gnutls_state
will be 0, and url.el will send an unencrypted "GET /" to the server,
which will fail, of course.

> If you want GnuTLS negotiation to happen from the sentinel, you need
> some machinery to block such process objects from communicating.
> I'm not sure silently skipping I/O is TRT for that: won't Lisp
> programs that happen to start communicating too early become confused
> about what's going on?

The "GET /" will block, sort of, as it would on any socket that's backed
up.  I don't think there will be anything user-noticeable from, say, the
perspective of `send-process-string'...

> More generally, what advantages do we expect to gain by making these
> changes?  Let DNS resolution proceed in the background, only to have
> to wait for GnuTLS negotiations anyway, before we can start using the
> stream?  Or are there more significant advantages?

It makes the entire connection, both the DNS resolution and the TLS
negotiation, asynchronous.  So Emacs doesn't hang while eww is
connecting to https://google.com.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

I have now implemented this, but in a slightly different way.  I
introduced a new process slot to say that we shouldn't be writing yet.

However, the Lisp part of the implementation is not very satisfactory.
This is basically it:

  (let ((process (open-network-stream name buffer host service
                                      :nowait nowait)))
    (if nowait
        (progn
          (gnutls-mark-process process t)
          (set-process-sentinel process 'gnutls-async-sentinel)
          process)
      (gnutls-negotiate :process (open-network-stream name buffer host service)
                        :type 'gnutls-x509pki
                        :hostname host))))
...

(defun gnutls-async-sentinel (process change)
  (when (string-match "open" change)
    (gnutls-negotiate :process process
                      :type 'gnutls-x509pki
                      :hostname (car (process-contact process)))
    (gnutls-mark-process process nil)))


The problem here is that this library is now putting a sentinel on the
process.  But any callers that want an asynchronous connection will also
be setting sentinels on the same process, which means that the
connection sentinel will be overwritten.

I've kludged this together in one of the callers (in url-http.el), but
that's too ugly to live.  (It checks for a sentinel and daisy-chains the
previous one.  Eek.)

So that has to be rewritten.  But I'm not sure how...  We, like, have
several layers of possible sentinels here, and...  uhm...

Ideas?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So that has to be rewritten.  But I'm not sure how...  We, like, have
> several layers of possible sentinels here, and...  uhm...

gnutls-negotiate (in Lisp) is basically a shim over gnutls-boot (in C).
It computes some parameters that are passed in as an keyword list to
gnutls-boot.

So: gnutls.el could be refactored to have a function that computes that
stuff, and then gnutls-mark-process could take that keyword list and
stash it in the process object.  connect_network_socket could then (in
:nowait mode) call gnutls-boot directly, keeping it all nice and safe in
the C layer.

That would obviate the sentinel, and make this all transparent for the
caller.

I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sun, 31 Jan 2016 02:10:04 +0100
> 
> So: gnutls.el could be refactored to have a function that computes that
> stuff, and then gnutls-mark-process could take that keyword list and
> stash it in the process object.  connect_network_socket could then (in
> :nowait mode) call gnutls-boot directly, keeping it all nice and safe in
> the C layer.
> 
> That would obviate the sentinel, and make this all transparent for the
> caller.

And how will that help you make the negotiations asynchronous?

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sun, 31 Jan 2016 01:40:51 +0100
> 
> I have now implemented this, but in a slightly different way.  I
> introduced a new process slot to say that we shouldn't be writing yet.
> 
> However, the Lisp part of the implementation is not very satisfactory.
> This is basically it:
> 
>   (let ((process (open-network-stream name buffer host service
>                                       :nowait nowait)))
>     (if nowait
>         (progn
>           (gnutls-mark-process process t)
>           (set-process-sentinel process 'gnutls-async-sentinel)
>           process)
>       (gnutls-negotiate :process (open-network-stream name buffer host service)
>                         :type 'gnutls-x509pki
>                         :hostname host))))
> ...
> 
> (defun gnutls-async-sentinel (process change)
>   (when (string-match "open" change)
>     (gnutls-negotiate :process process
>                       :type 'gnutls-x509pki
>                       :hostname (car (process-contact process)))
>     (gnutls-mark-process process nil)))
> 
> 
> The problem here is that this library is now putting a sentinel on the
> process.  But any callers that want an asynchronous connection will also
> be setting sentinels on the same process, which means that the
> connection sentinel will be overwritten.
> 
> I've kludged this together in one of the callers (in url-http.el), but
> that's too ugly to live.  (It checks for a sentinel and daisy-chains the
> previous one.  Eek.)
> 
> So that has to be rewritten.  But I'm not sure how...  We, like, have
> several layers of possible sentinels here, and...  uhm...

Ahmm... maybe it would have been better to wait until the discussion
completes and reaches some solid conclusions?

> Ideas?

Like I wrote elsewhere, I don't understand what this gives us.  The
sentinel runs in the main thread, so you still wait for the GnuTLS
handshake to complete.  Right?

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Like I wrote elsewhere, I don't understand what this gives us.  The
> sentinel runs in the main thread, so you still wait for the GnuTLS
> handshake to complete.  Right?

It depends on what you mean by "you".  :-)  If you go to a web page that
includes <img>, Emacs will pause while doing DNS lookups (for non-TLS
images) and the entire DNS lookup/connection/TLS negotiation cycle (for
TLS connections).  Having Emacs stop randomly while you're doing
something is rather annoying.

Both the connection and the TLS negotiation happens on the idle thread,
of course, so there will still be hangs, but they are much shorter than
the entire cycle.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

This has now been implemented.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22493@debbugs.gnu.org
> Date: Mon, 01 Feb 2016 00:17:10 +0100
> 
> Both the connection and the TLS negotiation happens on the idle thread,
> of course, so there will still be hangs, but they are much shorter than
> the entire cycle.

There's no such thing in Emacs as an idle thread.

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> There's no such thing in Emacs as an idle thread.

Sorry; I meant the idle loop.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22493@debbugs.gnu.org
> Date: Mon, 01 Feb 2016 05:01:55 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > There's no such thing in Emacs as an idle thread.
> 
> Sorry; I meant the idle loop.

A sentinel indeed starts when Emacs is idle, but once it kicks in, it
runs to completion, and Emacs cannot do anything else while it does.
So if the user presses a key a millisecond after the sentinel is
invoked and starts the GnuTLS negotiation procedure, the user will
still have to wait.

So once again I must ask what exactly do we gain with these changes.
If all we want is to make the GnuTLS negotiation wait until Emacs is
idle, we could run it from an idle timer, which would have been
simpler, and won't require us to have a process object in a strange
limbo state that cannot be communicated through.

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> A sentinel indeed starts when Emacs is idle, but once it kicks in, it
> runs to completion, and Emacs cannot do anything else while it does.
> So if the user presses a key a millisecond after the sentinel is
> invoked and starts the GnuTLS negotiation procedure, the user will
> still have to wait.

Sure.  But the wait is much shorter.

Anyway, the next step is, of course, to make the GnuTLS negotiation
asynchronous, too, so that the entire thing is unnoticeable.
The library itself is async, but it block because emacs_gnutls_handshake
does this:

  do
    {
      ret = gnutls_handshake (state);
      emacs_gnutls_handle_error (state, ret);
      QUIT;
    }
  while (ret < 0 && gnutls_error_is_fatal (ret) == 0);

Loop until the handshake completes.  The function should instead mark
the process' status as 'tls-handshake, return and continue the handshake
from the idle loop.

Which is something I'm not going to do straight away, because the
changes already in are complex enough.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22493@debbugs.gnu.org
> Date: Tue, 02 Feb 2016 02:43:02 +0100
> 
> > A sentinel indeed starts when Emacs is idle, but once it kicks in, it
> > runs to completion, and Emacs cannot do anything else while it does.
> > So if the user presses a key a millisecond after the sentinel is
> > invoked and starts the GnuTLS negotiation procedure, the user will
> > still have to wait.
> 
> Sure.  But the wait is much shorter.

If it really is, then I must be missing something, because I don't see
how it could.  Can you show some timings, comparing the "old" and the
"new" methods for making a GnuTLS connection?

And if somehow it indeed is shorter, the same effect can be achieved
by running the GnuTLS negotiation from an idle timer, something that
would most probably avoid most or all of the complexities on the C
level.  Have you considered this alternative?

> Anyway, the next step is, of course, to make the GnuTLS negotiation
> asynchronous, too, so that the entire thing is unnoticeable.
> The library itself is async, but it block because emacs_gnutls_handshake
> does this:
> 
>   do
>     {
>       ret = gnutls_handshake (state);
>       emacs_gnutls_handle_error (state, ret);
>       QUIT;
>     }
>   while (ret < 0 && gnutls_error_is_fatal (ret) == 0);
> 
> Loop until the handshake completes.  The function should instead mark
> the process' status as 'tls-handshake, return and continue the handshake
> from the idle loop.

This will have to be optional, since Emacs might not become idle for a
prolonged time.  IOW, only certain applications will want to benefit
from such a background handshake, some will need to wait for its
completion.  So if you make this be a background thing by default, you
might break existing users that don't expect the negotiation to return
before it's completed or failed.

> Which is something I'm not going to do straight away, because the
> changes already in are complex enough.

Thanks for working on this.  Please allow me some review comments
based on what's on the branch now.

  +@item :tls-parameters
  +When opening a TLS connection, this should be where the first element
  +is the TLS type,

This should tell what values are acceptable for TYPE, since no other
text around this one hints on that.

                   and the remaining elements should form a keyword list
  +acceptable for @code{gnutls-boot}.  The TLS connection will then be
  +negotiated after completing the connection to the host.

This should tell how to obtain those parameters.

  -@defun open-gnutls-stream name buffer host service
  +@defun open-gnutls-stream name buffer host service &optional nowait
   This function creates a buffer connected to a specific @var{host} and
   @var{service} (port number or service name).  The parameters and their
   syntax are the same as those given to @code{open-network-stream}

The meaning of 'nowait' should be explicitly described, since relying
on "the same as those given to 'open-gnutls-stream'" hides some
important information about the handshake in this case.

  -(defun open-gnutls-stream (name buffer host service)
  +(defun open-gnutls-stream (name buffer host service &optional nowait)
     "Open a SSL/TLS connection for a service to a host.
   Returns a subprocess-object to represent the connection.
   Input and output work as for subprocesses; `delete-process' closes it.
  @@ -109,6 +109,8 @@ BUFFER is the buffer (or `buffer-name') to associate with the process.
   Third arg is name of the host to connect to, or its IP address.
   Fourth arg SERVICE is name of the service desired, or an integer
   specifying a port number to connect to.
  +Fifth arg NOWAIT (which is optional) means that the socket should
  +be opened asynchronously.

This last sentence should say more explicitly what happens in that
case.

   It must be omitted, a number, or nil; if omitted or nil it
  -defaults to GNUTLS_VERIFY_ALLOW_X509_V1_CA_CRT."
  +defaults to GNUTLS_VERIFY_ALLOW_X509_V1_CA_CRT.
  +
  +If RETURN-KEYWORDS, don't connect to anything, but just return
  +the computed parameters that we otherwise would be calling
  +gnutls-boot with.  The return value will be a list where the
  +first element is the TLS type, and the rest of the list consists
  +of the keywords."

It's confusing to have a function named "open-gnutls-stream" that
doesn't really open any streams.  So I'd rather have
open-gnutls-stream refactored into 2 functions, with the part that
computes the parameters made a separate function that can be called
directly.  Then both open-gnutls-stream and the async code could call
that new function.

  --- a/lisp/net/network-stream.el
  +++ b/lisp/net/network-stream.el
  @@ -137,7 +137,12 @@ non-nil, is used warn the user if the connection isn't encrypted.
   a greeting from the server.

   :nowait is a boolean that says the connection should be made
  -asynchronously, if possible."
  +asynchronously, if possible.
  +
  +:tls-parameters is a list that should be supplied if you're
  +opening a TLS connection.  The first element is the TLS type, and
  +the remaining elements should be a keyword list accepted by
  +gnutls-boot."

Please mention here how to obtain that keyword list.

  --- a/src/Makefile.in
  +++ b/src/Makefile.in
  @@ -229,6 +229,8 @@ IMAGEMAGICK_CFLAGS= @IMAGEMAGICK_CFLAGS@
   LIBXML2_LIBS = @LIBXML2_LIBS@
   LIBXML2_CFLAGS = @LIBXML2_CFLAGS@

  +GETADDRINFO_A_LIBS = @GETADDRINFO_A_LIBS@

Please add a comment saying when GETADDRINFO_A_LIBS is non-empty.

  +static void
  +boot_error (struct Lisp_Process *p, const char *m, ...)
  +{
  +  va_list ap;
  +  va_start (ap, m);
  +  if (p->is_non_blocking_client)
  +    pset_status (p, Qfailed);
  +  else
  +    verror (m, ap);

AFAIU, here we are losing the error information, which I think we
shouldn't.  Why not keep it around and let the sentinel use it if it
wants to?

  @@ -5665,6 +5901,13 @@ send_process (Lisp_Object proc, const char *buf, ptrdiff_t len,
     if (p->outfd < 0)
       error ("Output file descriptor of %s is closed", SDATA (p->name));

  +#ifdef HAVE_GNUTLS
  +  /* The TLS connection hasn't been set up yet, so we can't write
  +     anything on the socket. */
  +  if (!NILP (p->gnutls_boot_parameters))
  +    return;
  +#endif

I don't think this is a good idea: you are silently returning here
without doing anything; the callers of send_process won't expect
that.  I think this should signal an error instead.

One final comment: I think this change will need addition of tests to
the test suite, to exercise the affected APIs, so we could make sure
we don't introduce any bugs in the existing functionalities.

Thanks.

======================================================================

> > Of course it does.  It's a major architecture change that we should
> > know about, and probably discuss before doing.
> 
> Ok.  :-)  The last time this was discussed, I mentioned threads and I
> think the response was basically "sure, if you can get it to work"...
> 
> What problems do you see with stopping/starting threads in Emacs (that
> do no Lisp-related stuff)?

We need to know which code can run on a separate thread, because some
things cannot be safely done from any thread but the main
(a.k.a. "Lisp") thread.  Running the Lisp interpreter is one of them,
but it's not the only one.  You cannot QUIT or signal an error or do
anything else that throws to top-level.  You cannot call malloc or
free, or any code that does.  You cannot modify any data structures
visible from Lisp, like buffers, Lisp strings, and the undo list.  You
cannot access or modify global variables or call any non-reentrant
Emacs functions.  And there are other no-no's, these are just a few
that popped in my mind within the first 5 sec.

So starting threads from Emacs application code is not something to
take easily, it should be justified and clearly marked in the sources.

However, it sounds like this is all much ado about nothing: I see no
references to any threads, old or new, in the changes you made on your
feature branch, so is there really anything to discuss here?  (I asked
the question which led to this sub-thread because you mentioned that
you "start a new thread that does getaddrinfo".)

> I'm assuming there are plenty of libraries that Emacs uses that
> already does this behind Emacs' back, and we don't seem to care...

First, we do care: the threads run by GTK, for example, caused us a
lot of grief at the time.  More importantly: those threads can hardly
run any Emacs code, can they?

> Anyway, this reminds me of something that I've been wondering about gdb
> output when running Emacs.  It often says something like this:
> 
> warning: Corrupted shared library list: 0x84582e0 != 0x5104c30
> warning: Corrupted shared library list: 0x2f172a0 != 0x21688a0
> warning: Corrupted shared library list: 0x79f1910 != 0x21688a0
> warning: Corrupted shared library list: 0x79f1910 != 0x21688a0
> warning: Corrupted shared library list: 0x7a07ae0 != 0x21688a0
> 
> Is that something to be worried about, or is it...  normal?

It's a real problem.  Crystal ball says that some of the system
libraries Emacs uses don't have debug info files to match them;
instead, you have debug info files from different versions of the
libraries.  Try "info sharedlibrary" at the GDB prompt, maybe it will
tell you which libraries are the offending ones.

If you cannot figure this out, it is best to ask on the GDB mailing
list.

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Sure.  But the wait is much shorter.
>
> If it really is, then I must be missing something, because I don't see
> how it could.  Can you show some timings, comparing the "old" and the
> "new" methods for making a GnuTLS connection?

You don't see how taking the DNS resolution out of the "stop everything
Emacs is doing" bit makes Emacs stop everything it's doing for a shorter
while?  I think I must be misunderstanding you...

> And if somehow it indeed is shorter, the same effect can be achieved
> by running the GnuTLS negotiation from an idle timer, something that
> would most probably avoid most or all of the complexities on the C
> level.  Have you considered this alternative?

Sure, but we were talking about the DNS resolution...

> This will have to be optional, since Emacs might not become idle for a
> prolonged time.  IOW, only certain applications will want to benefit
> from such a background handshake, some will need to wait for its
> completion.  So if you make this be a background thing by default, you
> might break existing users that don't expect the negotiation to return
> before it's completed or failed.

Sure.  The optional part here is that the user says :nowait when they
don't want to wait, so I'm not sure I'm getting your objection here,
either.

> Thanks for working on this.  Please allow me some review comments
> based on what's on the branch now.
>
>   +@item :tls-parameters
>   +When opening a TLS connection, this should be where the first element
>   +is the TLS type,
>
> This should tell what values are acceptable for TYPE, since no other
> text around this one hints on that.

Ok, fixed.

>                    and the remaining elements should form a keyword list
>   +acceptable for @code{gnutls-boot}.  The TLS connection will then be
>   +negotiated after completing the connection to the host.
>
> This should tell how to obtain those parameters.

It's currently obtained from `gnutls-negotiate' with a :return-keywords
parameter, but I think it would make sense to refactor that so that
there's a separate function that computes the parameter.  I'll fix that.

>   -@defun open-gnutls-stream name buffer host service
>   +@defun open-gnutls-stream name buffer host service &optional nowait
>    This function creates a buffer connected to a specific @var{host} and
>    @var{service} (port number or service name).  The parameters and their
>    syntax are the same as those given to @code{open-network-stream}
>
> The meaning of 'nowait' should be explicitly described, since relying
> on "the same as those given to 'open-gnutls-stream'" hides some
> important information about the handshake in this case.

Ok, fixed.

>   -(defun open-gnutls-stream (name buffer host service)
>   +(defun open-gnutls-stream (name buffer host service &optional nowait)
>      "Open a SSL/TLS connection for a service to a host.
>    Returns a subprocess-object to represent the connection.
>    Input and output work as for subprocesses; `delete-process' closes it.
>   @@ -109,6 +109,8 @@ BUFFER is the buffer (or `buffer-name') to associate with the process.
>    Third arg is name of the host to connect to, or its IP address.
>    Fourth arg SERVICE is name of the service desired, or an integer
>    specifying a port number to connect to.
>   +Fifth arg NOWAIT (which is optional) means that the socket should
>   +be opened asynchronously.
>
> This last sentence should say more explicitly what happens in that
> case.

Done.

>    It must be omitted, a number, or nil; if omitted or nil it
>   -defaults to GNUTLS_VERIFY_ALLOW_X509_V1_CA_CRT."
>   +defaults to GNUTLS_VERIFY_ALLOW_X509_V1_CA_CRT.
>   +
>   +If RETURN-KEYWORDS, don't connect to anything, but just return
>   +the computed parameters that we otherwise would be calling
>   +gnutls-boot with.  The return value will be a list where the
>   +first element is the TLS type, and the rest of the list consists
>   +of the keywords."
>
> It's confusing to have a function named "open-gnutls-stream" that
> doesn't really open any streams.  So I'd rather have
> open-gnutls-stream refactored into 2 functions, with the part that
> computes the parameters made a separate function that can be called
> directly.  Then both open-gnutls-stream and the async code could call
> that new function.

Yup.  :-)

>   +:tls-parameters is a list that should be supplied if you're
>   +opening a TLS connection.  The first element is the TLS type, and
>   +the remaining elements should be a keyword list accepted by
>   +gnutls-boot."
>
> Please mention here how to obtain that keyword list.

Will do.

> Please add a comment saying when GETADDRINFO_A_LIBS is non-empty.
>
>   +static void
>   +boot_error (struct Lisp_Process *p, const char *m, ...)
>   +{
>   +  va_list ap;
>   +  va_start (ap, m);
>   +  if (p->is_non_blocking_client)
>   +    pset_status (p, Qfailed);
>   +  else
>   +    verror (m, ap);
>
> AFAIU, here we are losing the error information, which I think we
> shouldn't.  Why not keep it around and let the sentinel use it if it
> wants to?

Oh yeah, that's true.  When I did that change I didn't know that a
status could be a list that returned more error info.  Fixed now.

>   +#ifdef HAVE_GNUTLS
>   +  /* The TLS connection hasn't been set up yet, so we can't write
>   +     anything on the socket. */
>   +  if (!NILP (p->gnutls_boot_parameters))
>   +    return;
>   +#endif
>
> I don't think this is a good idea: you are silently returning here
> without doing anything; the callers of send_process won't expect
> that.  I think this should signal an error instead.

It's perfectly fine to say

(setq proc (make-network-process ... :nowait t :tls-parameters ...))
(process-send-string proc)

It's not an error, and the idle loop will try resending it until it
succeeds.  (Which will only happen after DNS resolution and TLS
negotiation has failed.)

If the user has requested a TLS socket, then it's nonsensical to try
sending data on it before it's completed the negotiation.

> One final comment: I think this change will need addition of tests to
> the test suite, to exercise the affected APIs, so we could make sure
> we don't introduce any bugs in the existing functionalities.

Unfortunately, there are virtually no network tests in the suite.  At
least I couldn't find any.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>>   +#ifdef HAVE_GNUTLS
>>   +  /* The TLS connection hasn't been set up yet, so we can't write
>>   +     anything on the socket. */
>>   +  if (!NILP (p->gnutls_boot_parameters))
>>   +    return;
>>   +#endif
>>
>> I don't think this is a good idea: you are silently returning here
>> without doing anything; the callers of send_process won't expect
>> that.  I think this should signal an error instead.
>
> It's perfectly fine to say
>
> (setq proc (make-network-process ... :nowait t :tls-parameters ...))
> (process-send-string proc)

I think you're right again.  I thought I had tested this, but I hadn't:

(progn
  (setq proc (make-network-process :name "foo"
				   :buffer (get-buffer-create "*foo*")
				   :host "gmane.org"
				   :service "http"
				   :nowait t))
  (process-send-string proc "GET / HTTP/1.0\n\n"))

With async DNS, this will fail, because the process-send-string happens
before the connection had completed.  (And this isn't a TLS socket.)  So
I think that (like I said on the emacs-devel threads) we may have to
change the :nowait stuff to allow a more fine-grained control.

So the rule would be if you want a fully async connection, you have to
say ":nowait 'dns" or something, and then put a sentinel on the process
to not do anything until it changes status to "connected".  (This is
what url.el does already, which is why I didn't see this before...)

But in the case of TLS connections, then, er, we have to ... do
something.

Ok, here's the scenario.  With a non-TLS socket, this is what's going
on:

make-network-process gives us a process in "open", and then when it
changes to "connected" (after connecting the socket) we can start
talking.  (This is what url.el does.)

With a TLS socket:

make-network-process gives us a process in "open", and then when it
changes to "connected" (after connecting the socket) we can't start
talking.  We have to wait until the TLS has been negotiated.  So perhaps
it should only move to the "connected" state after the negotiation has
finished?  Or introduce more states?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22493@debbugs.gnu.org
> Date: Wed, 03 Feb 2016 12:58:30 +1100
> 
> I think you're right again.  I thought I had tested this, but I hadn't:
> 
> (progn
>   (setq proc (make-network-process :name "foo"
>                                    :buffer (get-buffer-create "*foo*")
>                                    :host "gmane.org"
>                                    :service "http"
>                                    :nowait t))
>   (process-send-string proc "GET / HTTP/1.0\n\n"))
> 
> With async DNS, this will fail, because the process-send-string happens
> before the connection had completed.  (And this isn't a TLS socket.)  So
> I think that (like I said on the emacs-devel threads) we may have to
> change the :nowait stuff to allow a more fine-grained control.

If you make process-send-string wait until DNS completes, the problem
will disappear.  Then, if code such as above wants better async
behavior, its maintainer will have to restructure the code, for
example like this:

> So the rule would be if you want a fully async connection, you have to
> say ":nowait 'dns" or something, and then put a sentinel on the process
> to not do anything until it changes status to "connected".  (This is
> what url.el does already, which is why I didn't see this before...)

But if the code is not changed, it should still "just work", albeit
not as fast as it could.

> With a TLS socket:
> 
> make-network-process gives us a process in "open", and then when it
> changes to "connected" (after connecting the socket) we can't start
> talking.  We have to wait until the TLS has been negotiated.  So perhaps
> it should only move to the "connected" state after the negotiation has
> finished?  Or introduce more states?

Like with process-send-string, we could make gnutls-boot wait until
DNS resolution completes (and error out if DNS fails).

As for making gnutls-boot run in the background, I'm still not
convinced it buys us enough advantages (or any advantages) to justify
the changes.  Let's continue discussing that, OK?

Running gnutls-boot from a separate thread on the C level is also
possible, but AFAIU that requires a total rewrite of the current C
implementation, as it does a lot of stuff that cannot be done from a
non-main thread.

Either way, if we decide that making gnutls-boot run in the background
is a Good Thing, then I feel that doing so requires code restructuring
that must involve application-level considerations.  So these changes
probably shouldn't be done on the level of open-gnutls-stream.
Instead, we should provide the building blocks for applications that
want this, and let them restructure their code as they see fit.
Trying to do that transparently controlled by a single argument is not
going to work well, I think.

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> With async DNS, this will fail, because the process-send-string happens
>> before the connection had completed.  (And this isn't a TLS socket.)  So
>> I think that (like I said on the emacs-devel threads) we may have to
>> change the :nowait stuff to allow a more fine-grained control.
>
> If you make process-send-string wait until DNS completes, the problem
> will disappear.

Yes.  That was basically what the

#ifdef HAVE_GNUTLS
  /* The TLS connection hasn't been set up yet, so we can't write
     anything on the socket. */
  if (!NILP (p->gnutls_boot_parameters))
    return;
#endif

in send_process achieves (in the TLS case).  If we amend that to

  if (!NILP (p->gnutls_boot_parameters) || p->outfd == 0)
    return;

then that will have the same effect on non-TLS sockets.  But as noted in
the previous email, set-process-coding-system would also have to be
changed to remove these outfd guards...

>> make-network-process gives us a process in "open", and then when it
>> changes to "connected" (after connecting the socket) we can't start
>> talking.  We have to wait until the TLS has been negotiated.  So perhaps
>> it should only move to the "connected" state after the negotiation has
>> finished?  Or introduce more states?
>
> Like with process-send-string, we could make gnutls-boot wait until
> DNS resolution completes (and error out if DNS fails).

That's true...  Hm...

> As for making gnutls-boot run in the background, I'm still not
> convinced it buys us enough advantages (or any advantages) to justify
> the changes.  Let's continue discussing that, OK?

I think we've been talking a bit past each other on the TLS issue, so
I'll just write out the scenario as I see it, and why I think it has to
be implemented the way I did it, and you can correct me if you see other
options.

OK: eww has displayed a web page, and now wants to fetch an image over
https.  Our goal is to have Emacs stop as little as possible while this
is happening in the background, so that the user can scroll around or do
whatever the user wants to do while eww is working on getting the image.

So eww issues a url-retrieve command in the background, which ends up
calling make-network-process :nowait t.  This returns immediately, with
no user-noticeable hangs.  getaddrinfo_a has been issued, and after a
while a response is received and is noticed by the idle loop.

Emacs then calls connect_network_process.  This basically is a wrapper
around the connect(2), and it's pretty async on a non-blocking socket,
so this should not be noticeable to the user.

Everything up until now has happened identically for TLS and non-TLS
sockets.

Now, for a TLS socket, we have to do the negotiating.  This is currently
a blocking process, but it doesn't have to be.  The GnuTLS library in
itself is non-blocking, but our interaction with it currently is.

So we call gnutls_boot after the connection has happened.  I originally
did this with a sentinel on a process, but that doesn't really work,
because the callers of make_network_process want their own sentinels on
the process.  So I call gnutls_boot from the C layer instead of from a
sentinel.

No matter how we call gnutls_boot, it will currently hang Emacs while
it's transferring all those certificates back and forth.

After that, the url.el sentinel will fire with a "connect" state, and
it'll say "GET /foo.png" to the server, and the server will output the
image (non-blocking).  And then the image will be parsed and inserted
into the buffer (blocking and noticeable for large images).

*phew*  :-)

> Running gnutls-boot from a separate thread on the C level is also
> possible, but AFAIU that requires a total rewrite of the current C
> implementation, as it does a lot of stuff that cannot be done from a
> non-main thread.

Yup.

> Either way, if we decide that making gnutls-boot run in the background
> is a Good Thing, then I feel that doing so requires code restructuring
> that must involve application-level considerations.

Actually, almost all of the major network usages in Emacs today require
no changes.  The only one I've found is erc.  url.el, Gnus, smtpmail,
imap.el, nntp.el all work without any changes with the current
implementation.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22493@debbugs.gnu.org
> Date: Thu, 04 Feb 2016 13:47:05 +1100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> With async DNS, this will fail, because the process-send-string happens
> >> before the connection had completed.  (And this isn't a TLS socket.)  So
> >> I think that (like I said on the emacs-devel threads) we may have to
> >> change the :nowait stuff to allow a more fine-grained control.
> >
> > If you make process-send-string wait until DNS completes, the problem
> > will disappear.
> 
> Yes.  That was basically what the
> 
> #ifdef HAVE_GNUTLS
>   /* The TLS connection hasn't been set up yet, so we can't write
>      anything on the socket. */
>   if (!NILP (p->gnutls_boot_parameters))
>     return;
> #endif
> 
> in send_process achieves (in the TLS case).

No, I meant to actually wait: loop checking the async DNS status until
that indicates the DNS resolution is complete, or fails.

>   if (!NILP (p->gnutls_boot_parameters) || p->outfd == 0)
>     return;
> 
> then that will have the same effect on non-TLS sockets.  But as noted in
> the previous email, set-process-coding-system would also have to be
> changed to remove these outfd guards...

Every API that expects a fully-capable process object should have such
a wait for DNS added to it.  And if we make GnuTLS negotiation run in
the background, they should also wait for that to complete.  An
application that doesn't want to wait will have to "do other things"
without calling these functions, and from time to time check whether
the background processing is complete (which might require an
additional API).

> OK: eww has displayed a web page, and now wants to fetch an image over
> https.  Our goal is to have Emacs stop as little as possible while this
> is happening in the background, so that the user can scroll around or do
> whatever the user wants to do while eww is working on getting the image.
> 
> So eww issues a url-retrieve command in the background, which ends up
> calling make-network-process :nowait t.  This returns immediately, with
> no user-noticeable hangs.  getaddrinfo_a has been issued, and after a
> while a response is received and is noticed by the idle loop.
> 
> Emacs then calls connect_network_process.  This basically is a wrapper
> around the connect(2), and it's pretty async on a non-blocking socket,
> so this should not be noticeable to the user.
> 
> Everything up until now has happened identically for TLS and non-TLS
> sockets.
> 
> Now, for a TLS socket, we have to do the negotiating.  This is currently
> a blocking process, but it doesn't have to be.  The GnuTLS library in
> itself is non-blocking, but our interaction with it currently is.
> 
> So we call gnutls_boot after the connection has happened.  I originally
> did this with a sentinel on a process, but that doesn't really work,
> because the callers of make_network_process want their own sentinels on
> the process.  So I call gnutls_boot from the C layer instead of from a
> sentinel.
> 
> No matter how we call gnutls_boot, it will currently hang Emacs while
> it's transferring all those certificates back and forth.

That last sentence is exactly the point I was trying to make all
along: we have to wait for this, therefore any time savings from
running gnutls_boot in the background are minor or even non-existent.
So I question the need for complicating the heck out of the underlying
code, for no practical gain.

> > Either way, if we decide that making gnutls-boot run in the background
> > is a Good Thing, then I feel that doing so requires code restructuring
> > that must involve application-level considerations.
> 
> Actually, almost all of the major network usages in Emacs today require
> no changes.  The only one I've found is erc.  url.el, Gnus, smtpmail,
> imap.el, nntp.el all work without any changes with the current
> implementation.

Why shouldn't we assume that the problem you saw in erc is the tip of
an iceberg, and the other places are happy exceptions?  Who knows how
many other packages are out there that are like erc?

And again, I don't see the gains in doing gnutls-boot asynchronously.
DNS resolution, yes, but gnutls-boot is another matter.

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> So we call gnutls_boot after the connection has happened.  I originally
>> did this with a sentinel on a process, but that doesn't really work,
>> because the callers of make_network_process want their own sentinels on
>> the process.  So I call gnutls_boot from the C layer instead of from a
>> sentinel.
>> 
>> No matter how we call gnutls_boot, it will currently hang Emacs while
>> it's transferring all those certificates back and forth.
>
> That last sentence is exactly the point I was trying to make all
> along: we have to wait for this, therefore any time savings from
> running gnutls_boot in the background are minor or even non-existent.
> So I question the need for complicating the heck out of the underlying
> code, for no practical gain.

I describe the call chain, and why it's necessary to have the
gnutls_boot called from the C layer.  If you see anything wrong with the
logic I described, please let me know.

And as I've said repeatedly, gnutls_boot is only synchronous now as a
quirk of our implementation, and making it async is the next step I am
going to take.  But it's a separate issue, and will be dealt with once
the first round of changes are in.

> Why shouldn't we assume that the problem you saw in erc is the tip of
> an iceberg, and the other places are happy exceptions?  Who knows how
> many other packages are out there that are like erc?

That's why :async 'dns.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22493@debbugs.gnu.org
> Date: Fri, 05 Feb 2016 13:36:06 +1100
> 
> >> No matter how we call gnutls_boot, it will currently hang Emacs while
> >> it's transferring all those certificates back and forth.
> >
> > That last sentence is exactly the point I was trying to make all
> > along: we have to wait for this, therefore any time savings from
> > running gnutls_boot in the background are minor or even non-existent.
> > So I question the need for complicating the heck out of the underlying
> > code, for no practical gain.
> 
> I describe the call chain, and why it's necessary to have the
> gnutls_boot called from the C layer.  If you see anything wrong with the
> logic I described, please let me know.
> 
> And as I've said repeatedly, gnutls_boot is only synchronous now as a
> quirk of our implementation, and making it async is the next step I am
> going to take.  But it's a separate issue, and will be dealt with once
> the first round of changes are in.

If by "making it async" you mean to test for it in the idle loop, that
is not really async, and _that_ is _my_ point, which I've expressed
repeatedly already.  Waiting in the idle loop is still waiting, as far
as the main thread is concerned.

> > Why shouldn't we assume that the problem you saw in erc is the tip of
> > an iceberg, and the other places are happy exceptions?  Who knows how
> > many other packages are out there that are like erc?
> 
> That's why :async 'dns.

As I wrote elsewhere, this is not enough.

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> And as I've said repeatedly, gnutls_boot is only synchronous now as a
>> quirk of our implementation, and making it async is the next step I am
>> going to take.  But it's a separate issue, and will be dealt with once
>> the first round of changes are in.
>
> If by "making it async" you mean to test for it in the idle loop, that
> is not really async, and _that_ is _my_ point, which I've expressed
> repeatedly already.  Waiting in the idle loop is still waiting, as far
> as the main thread is concerned.

There is no waiting in the GnuTLS library during negotiation.  It is
fully asynchronous.  If any action blocks, it will return, and we can
proceed later.  It works, seen from the application side, as any other
network connection.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22493@debbugs.gnu.org
> Date: Fri, 05 Feb 2016 18:34:44 +1100
> 
> There is no waiting in the GnuTLS library during negotiation.  It is
> fully asynchronous.  If any action blocks, it will return, and we can
> proceed later.  It works, seen from the application side, as any other
> network connection.

The _waiting_ is asynchronous.  The processing cannot be, because
GnuTLS doesn't start any threads.

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: 22493@debbugs.gnu.org
>> Date: Fri, 05 Feb 2016 18:34:44 +1100
>> 
>> There is no waiting in the GnuTLS library during negotiation.  It is
>> fully asynchronous.  If any action blocks, it will return, and we can
>> proceed later.  It works, seen from the application side, as any other
>> network connection.
>
> The _waiting_ is asynchronous.  The processing cannot be, because
> GnuTLS doesn't start any threads.

I think you're nitpicking semantics now.  There will be no user-visible
waiting while a TLS connection is negotiated, OK?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22493@debbugs.gnu.org
> Date: Sat, 06 Feb 2016 14:03:40 +1100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Lars Ingebrigtsen <larsi@gnus.org>
> >> Cc: 22493@debbugs.gnu.org
> >> Date: Fri, 05 Feb 2016 18:34:44 +1100
> >> 
> >> There is no waiting in the GnuTLS library during negotiation.  It is
> >> fully asynchronous.  If any action blocks, it will return, and we can
> >> proceed later.  It works, seen from the application side, as any other
> >> network connection.
> >
> > The _waiting_ is asynchronous.  The processing cannot be, because
> > GnuTLS doesn't start any threads.
> 
> I think you're nitpicking semantics now.  There will be no user-visible
> waiting while a TLS connection is negotiated, OK?

That is only true if the foreground program has something useful to do
while TLS negotiation goes on.  You have a very particular application
in mind (a browser) where you can just let user scroll through the
rest of the buffer, but (1) what if the user doesn't want to scroll,
but wants to see that image that wasn't yet downloaded; and (b) not
every application is a browser where there's something to do while
these negotiations go on.

IOW, your changes might make sense for some small set of applications,
but for the others it's just a major complication in the code base
with no real gains.

We shouldn't make such major changes in low-level code when they
benefit only a small class of applications, IMO.

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> That is only true if the foreground program has something useful to do
> while TLS negotiation goes on.  You have a very particular application
> in mind (a browser) where you can just let user scroll through the
> rest of the buffer

Yes, that's the use case these changes are for.

> IOW, your changes might make sense for some small set of applications,
> but for the others it's just a major complication in the code base
> with no real gains.
>
> We shouldn't make such major changes in low-level code when they
> benefit only a small class of applications, IMO.

Well, web browsers and news and mail readers aren't a small class of
applications when it comes to the network layer.  They are the majority
of the applications.  :-)  Having rmail pause Emacs while it's fetching
HTML assets, while you just want to go to the next message, is not good
user interface design.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22493@debbugs.gnu.org
> Date: Wed, 03 Feb 2016 12:10:21 +1100
> 
> >> Sure.  But the wait is much shorter.
> >
> > If it really is, then I must be missing something, because I don't see
> > how it could.  Can you show some timings, comparing the "old" and the
> > "new" methods for making a GnuTLS connection?
> 
> You don't see how taking the DNS resolution out of the "stop everything
> Emacs is doing" bit makes Emacs stop everything it's doing for a shorter
> while?

Of course I do.  But this thread is not about async DNS, it's about
async GnuTLS initialization.  I'm asking how attempting to do that in
the background, _after_ DNS resolution ended, makes the wait shorter.

IOW, I have nothing against using getaddrinfo_a where it's available
(or emulate it where it isn't).  But I don't see any visible
advantages to backgrounding the GnuTLS negotiation, because the way
you did it, it runs in the main thread, and so doesn't save us any
waiting time, AFAIU.  All it does is tremendously increase complexity
of starting a TLS connection.

> > And if somehow it indeed is shorter, the same effect can be achieved
> > by running the GnuTLS negotiation from an idle timer, something that
> > would most probably avoid most or all of the complexities on the C
> > level.  Have you considered this alternative?
> 
> Sure, but we were talking about the DNS resolution...

No, we weren't.  This is about the parts of open-gnutls-stream, which
runs _after_ the DNS resolution.  The changes which I reviewed and
commented on, and which bother me, are those in gnutls.el and
elsewhere that are related to the initial GnuTLS negotiation.

> > This will have to be optional, since Emacs might not become idle for a
> > prolonged time.  IOW, only certain applications will want to benefit
> > from such a background handshake, some will need to wait for its
> > completion.  So if you make this be a background thing by default, you
> > might break existing users that don't expect the negotiation to return
> > before it's completed or failed.
> 
> Sure.  The optional part here is that the user says :nowait when they
> don't want to wait, so I'm not sure I'm getting your objection here,
> either.

An application may wish to make DNS resolution asynchronous, but might
not rely on Emacs becoming idle for GnuTLS initialization.  IOW, we
need a separate option for that.

> If the user has requested a TLS socket, then it's nonsensical to try
> sending data on it before it's completed the negotiation.

So make the sending function wait for the end of the negotiation.

> > One final comment: I think this change will need addition of tests to
> > the test suite, to exercise the affected APIs, so we could make sure
> > we don't introduce any bugs in the existing functionalities.
> 
> Unfortunately, there are virtually no network tests in the suite.  At
> least I couldn't find any.

I know.  We need to add them.  Such a radical change in the core
infrastructure used all over the place cannot be trusted without some
minimal testing of the popular paradigms of its usage.  E.g., does
synchronous URL retrieval still work? what about asynchronous one?
what about SMTP with TLS? etc. etc.  There's a potential here to break
a lot of functionality out there.

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> An application may wish to make DNS resolution asynchronous, but might
> not rely on Emacs becoming idle for GnuTLS initialization.  IOW, we
> need a separate option for that.

Well, if you're saying :nowait, it means that you're setting up to talk
to the process asynchronously.  So you'll be using process filters and
sentinels to talk to it.  Otherwise you'll be using
accept-process-output I think.  In either case, you're not involved with
the TLS negotiation -- it's not something you're ... relying on, it's
just there.

I mean, if you're opening a :nowait connection, and Emacs never becomes
idle ever, you're never getting your process-filtered output.  But
that's not something that happens.

>> Unfortunately, there are virtually no network tests in the suite.  At
>> least I couldn't find any.
>
> I know.  We need to add them.  Such a radical change in the core
> infrastructure used all over the place cannot be trusted without some
> minimal testing of the popular paradigms of its usage.  E.g., does
> synchronous URL retrieval still work? what about asynchronous one?
> what about SMTP with TLS? 

I've tested all of that, but without a test suite.

> etc. etc. There's a potential here to break a lot of functionality out
> there.

Yes.  Somebody should write a test suite anyway.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 22493@debbugs.gnu.org
> Date: Sat, 30 Jan 2016 23:55:57 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> The problem is, though, that the process isn't marked as a gnutls
> >> process until gnutls_boot is called, so this needs to happen much
> >> earlier.  Possibly by adding another keyword to make-network-process
> >> that just sets
> >> 
> >>   XPROCESS (proc)->gnutls_p = 1;
> >
> > Why do you need to mark a process gnutls_p, when it definitely still
> > isn't, not until the GnuTLS negotiation is successfully completed?
> 
> It is a socket that will only be used for TLS.  And the current way
> we're creating TLS streams is by calling `open-gnutls-stream', which
> (simplified) just calls `make-network-stream' and then `gnutls-boot' in
> rapid succession.  So any stream you get out of `open-gnutls-stream'
> will look just the same -- it'll have gnutls_p set.
> 
> It'll just happen slightly earlier.

There's a certain semantics to this flag being set, and overloading it
doesn't sound like a good idea to me.

But this is a minor issue -- the larger one is below.

> >> #ifdef HAVE_GNUTLS
> >> 	      if (p->gnutls_p && p->gnutls_state)
> >> 		written = emacs_gnutls_write (p, cur_buf, cur_len);
> >> 	      else
> >> #endif
> >> 		written = emacs_write_sig (outfd, cur_buf, cur_len);
> >> 
> >> I think that looks rather nonsensical.
> >
> > Does this happen today?  If so, in what scenario?
> 
> It does not happen today, but it will happen if we open the TLS stream
> :nowait.

So not doing that will avoid this problem.  Good.

> > If you want GnuTLS negotiation to happen from the sentinel, you need
> > some machinery to block such process objects from communicating.
> > I'm not sure silently skipping I/O is TRT for that: won't Lisp
> > programs that happen to start communicating too early become confused
> > about what's going on?
> 
> The "GET /" will block, sort of, as it would on any socket that's backed
> up.

That depends on what the (as yet unwritten) code will do when it gets
to the above fragment and discovers that GnuTLS did not yet finish
handshaking.  You didn't really describe what you intend to do.  Your
options are (a) silently do nothing, or (b) fail the write.  Either
way sounds problematic to me.

> > More generally, what advantages do we expect to gain by making these
> > changes?  Let DNS resolution proceed in the background, only to have
> > to wait for GnuTLS negotiations anyway, before we can start using the
> > stream?  Or are there more significant advantages?
> 
> It makes the entire connection, both the DNS resolution and the TLS
> negotiation, asynchronous.  So Emacs doesn't hang while eww is
> connecting to https://google.com.

I guess I'm missing something important here, because I don't
understand how can this work.  Async DNS works because getaddrinfo_a
starts threads under the hood to perform the resolution, while the
main (a.k.a. "Lisp") thread is free to do other things, until a call
to 'pselect' tells it the DNS resolution is complete.  By contrast,
the GnuTLS negotiation, whether it runs from a sentinel or from inside
open-network-stream, runs in the main thread, so Emacs must wait for
it to finish before it can do anything else.  Running it from a
sentinel doesn't magically make it asynchronous, since sentinels run
in the main thread, and the main thread is busy as long as the
sentinel runs.  Maybe you thought about running gnutls-boot from a
separate thread, but I see no easy way of doing that, without some
very thorough rewrite, since current code access Lisp data structures
an (at least optionally) displays logging messages in the echo area.

So how will this work asynchronously?  What am I missing?