:nowait t misbehaves when falling back from IPv6 to IPv4

:nowait t misbehaves when falling back from IPv6 to IPv4

[Robert Pluim]

This requires you to be on a dual-stack IPv4/v6 host, with working
IPv6 connectivity.

elpa.gnu.org is currently reachable via IPv4, but not IPv6.

    ./emacs-26 -Q
    M-x package-list-packages

    <time passes>
    error in process filter: Error retrieving:
    https://elpa.gnu.org/packages/archive-contents (error http 400) [2
    times]

Looking at the network traces, Emacs tries to connect using IPv6,
which times out, and it then falls back to the IPv4
address. Unfortunately something goes wrong, and the connection fails.

If I change the 'open-network-stream' call in 'url-open-stream' to not
use :nowait t, then everything works (although you still have to wait
for the IPv6 connect to timeout).

emacs-27 is different, presumably because of
e87e6a24c49542111e669b7d0f1a412024663f8e : on macOS it never sees the
IPv6 connection time out, so never falls back, on GNU/Linux itʼs the
same as emacs-26.

I unfortunately donʼt have time to look into this right now, but
perhaps someone familiar with the network code has some ideas.

Robert

Re: :nowait t misbehaves when falling back from IPv6 to IPv4

[Robert Pluim]

Robert Pluim <rpluim@gmail.com> writes:

> I unfortunately donʼt have time to look into this right now, but
> perhaps someone familiar with the network code has some ideas.

So this is GnuTLS related: setting package-archives to use http rather
than https makes everything work.

Robert

Re: :nowait t misbehaves when falling back from IPv6 to IPv4

[Robert Pluim]

Robert Pluim <rpluim@gmail.com> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> I unfortunately donʼt have time to look into this right now, but
>> perhaps someone familiar with the network code has some ideas.
>
> So this is GnuTLS related: setting package-archives to use http rather
> than https makes everything work.

And after several hours of debugging, the fix turns out to be small,
but one that I definitely want others to look at.

Hereʼs what happens:

1. update-package-archives requests an asynchronous load of an url on
   elpa.gnu.org
2. make-network-process looks up 'elpa.gnu.org', which returns an IPv6
   and an IPv4 address (in that order)
3. it calls 'connect_network_socket', which calls 'connect', starts TLS
   negotiation, and then continues on because this is a non-blocking
   connect
4. some time later, that IPv6 connection has failed, which is noticed
   in 'wait_reading_process_output'. Because we have more addresses to
   try, it calls 'connect_network_socket' again, this time for the
   IPv4 address
5. 'connect_network_socket' again calls 'connect', but doesnʼt start TLS
   negotiation, since the TLS boot parameters were deleted in [3]
6. TCP finishes connecting, we call the user-provided sentinel, which
   ends up sending an HTTP request on what is supposed to be a TLS
   secured channel, and elpa.gnu.org quite rightly sends a '400' error
   and shuts down the connection

The fix below works for both emacs-26 and master. I have not fully
reasoned about its consequences.

This close to a release Iʼm certain it won't go into emacs-26, so
perhaps we should put an entry in PROBLEMS telling people to either
use elpa.gnu.org's IPv4 address in 'package-archives', or to use http
instead of https when they have problems connecting.

Broken IPv6 setups will only make this more likely in the future, so
perhaps we should think about implementing a limited version of 'Happy
Eyeballs 2' <https://tools.ietf.org/html/rfc8305> at some point.

2019-02-20  Robert Pluim  <rpluim@gmail.com>

	* src/process.c (connect_network_socket): Only delete
	gnutls_boot_parameters if TLS negotiation actually succeeded,
	as they may be needed for connection to other addresses for
	the same host.

diff --git a/src/process.c b/src/process.c
index b0a327229c..8784827cd4 100644
--- a/src/process.c
+++ b/src/process.c
@@ -3657,11 +3657,13 @@ connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
       Lisp_Object boot, params = p->gnutls_boot_parameters;
 
       boot = Fgnutls_boot (proc, XCAR (params), XCDR (params));
-      p->gnutls_boot_parameters = Qnil;
 
       if (p->gnutls_initstage == GNUTLS_STAGE_READY)
+        {
 	/* Run sentinels, etc. */
+        p->gnutls_boot_parameters = Qnil;
 	finish_after_tls_connection (proc);
+        }
       else if (p->gnutls_initstage != GNUTLS_STAGE_HANDSHAKE_TRIED)
 	{
 	  deactivate_process (proc);

Re: :nowait t misbehaves when falling back from IPv6 to IPv4

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Wed, 20 Feb 2019 19:44:38 +0100
> 
> The fix below works for both emacs-26 and master. I have not fully
> reasoned about its consequences.

Didn't you say at some point the problem didn't exist on master?  Or
am I confused?

Thanks.

Re: :nowait t misbehaves when falling back from IPv6 to IPv4

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Date: Wed, 20 Feb 2019 19:44:38 +0100
>> 
>> The fix below works for both emacs-26 and master. I have not fully
>> reasoned about its consequences.
>
> Didn't you say at some point the problem didn't exist on master?  Or
> am I confused?

The symptoms were different on master: the initial IPv6 connect never
failed at all. However, that may have been due to transient IPv6
problems my end: master and emacs-26 now fail the same way.

You may be thinking of the TLS1.3 issue with gmail. That works in
master and fails in emacs-26.

Robert

Re: :nowait t misbehaves when falling back from IPv6 to IPv4

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Wed, 20 Feb 2019 20:29:54 +0100
> Cc: emacs-devel@gnu.org
> 
> You may be thinking of the TLS1.3 issue with gmail. That works in
> master and fails in emacs-26.

Could be.  Sorry.