bug#22493: 25.1.50; open-gnutls-stream doesn't respect :nowait, so the connections are synchronous

[Lars Ingebrigtsen <larsi@gnus.org>]

Eli Zaretskii <eliz@gnu.org> writes:

>> With async DNS, this will fail, because the process-send-string happens
>> before the connection had completed.  (And this isn't a TLS socket.)  So
>> I think that (like I said on the emacs-devel threads) we may have to
>> change the :nowait stuff to allow a more fine-grained control.
>
> If you make process-send-string wait until DNS completes, the problem
> will disappear.

Yes.  That was basically what the

#ifdef HAVE_GNUTLS
  /* The TLS connection hasn't been set up yet, so we can't write
     anything on the socket. */
  if (!NILP (p->gnutls_boot_parameters))
    return;
#endif

in send_process achieves (in the TLS case).  If we amend that to

  if (!NILP (p->gnutls_boot_parameters) || p->outfd == 0)
    return;

then that will have the same effect on non-TLS sockets.  But as noted in
the previous email, set-process-coding-system would also have to be
changed to remove these outfd guards...

>> make-network-process gives us a process in "open", and then when it
>> changes to "connected" (after connecting the socket) we can't start
>> talking.  We have to wait until the TLS has been negotiated.  So perhaps
>> it should only move to the "connected" state after the negotiation has
>> finished?  Or introduce more states?
>
> Like with process-send-string, we could make gnutls-boot wait until
> DNS resolution completes (and error out if DNS fails).

That's true...  Hm...

> As for making gnutls-boot run in the background, I'm still not
> convinced it buys us enough advantages (or any advantages) to justify
> the changes.  Let's continue discussing that, OK?

I think we've been talking a bit past each other on the TLS issue, so
I'll just write out the scenario as I see it, and why I think it has to
be implemented the way I did it, and you can correct me if you see other
options.

OK: eww has displayed a web page, and now wants to fetch an image over
https.  Our goal is to have Emacs stop as little as possible while this
is happening in the background, so that the user can scroll around or do
whatever the user wants to do while eww is working on getting the image.

So eww issues a url-retrieve command in the background, which ends up
calling make-network-process :nowait t.  This returns immediately, with
no user-noticeable hangs.  getaddrinfo_a has been issued, and after a
while a response is received and is noticed by the idle loop.

Emacs then calls connect_network_process.  This basically is a wrapper
around the connect(2), and it's pretty async on a non-blocking socket,
so this should not be noticeable to the user.

Everything up until now has happened identically for TLS and non-TLS
sockets.

Now, for a TLS socket, we have to do the negotiating.  This is currently
a blocking process, but it doesn't have to be.  The GnuTLS library in
itself is non-blocking, but our interaction with it currently is.

So we call gnutls_boot after the connection has happened.  I originally
did this with a sentinel on a process, but that doesn't really work,
because the callers of make_network_process want their own sentinels on
the process.  So I call gnutls_boot from the C layer instead of from a
sentinel.

No matter how we call gnutls_boot, it will currently hang Emacs while
it's transferring all those certificates back and forth.

After that, the url.el sentinel will fire with a "connect" state, and
it'll say "GET /foo.png" to the server, and the server will output the
image (non-blocking).  And then the image will be parsed and inserted
into the buffer (blocking and noticeable for large images).

*phew*  :-)

> Running gnutls-boot from a separate thread on the C level is also
> possible, but AFAIU that requires a total rewrite of the current C
> implementation, as it does a lot of stuff that cannot be done from a
> non-main thread.

Yup.

> Either way, if we decide that making gnutls-boot run in the background
> is a Good Thing, then I feel that doing so requires code restructuring
> that must involve application-level considerations.

Actually, almost all of the major network usages in Emacs today require
no changes.  The only one I've found is erc.  url.el, Gnus, smtpmail,
imap.el, nntp.el all work without any changes with the current
implementation.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no