bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Juliusz Chroboczek]

My machine has both IPv6 and IPv4.  I'm accessing a server that has both
IPv6 and IPv4, but whose web server listens on IPv4 only.  The web server
is doing the right thing, sending RST upon IPv6 connection attempts.  Wget
and telnet do the right thing, they fallback to IPv4:

  $ telnet moule.informatique.univ-paris-diderot.fr 8080
  Trying 2001:660:3301:8070::40...
  Trying 194.254.199.40...
  Connected to moule.informatique.univ-paris-diderot.fr.
  Escape character is '^]'.

On the other hand, url-retrieve-synchronously returns an empty string.
Tcpdump shows that it's never even trying IPv4:

  (with-current-buffer
    (url-retrieve-synchronously
     "http://moule.informatique.univ-paris-diderot.fr:8080")
    (buffer-substring (point-min) (point-max)))
  ""

  (with-current-buffer
    (url-retrieve-synchronously "http://194.254.199.40:8080")
    (buffer-substring (point-min) (point-max)))
  "HTTP/1.1 200 OK
  ..."

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Andreas Schwab]

Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:

> My machine has both IPv6 and IPv4.  I'm accessing a server that has both
> IPv6 and IPv4, but whose web server listens on IPv4 only.  The web server
> is doing the right thing, sending RST upon IPv6 connection attempts.  Wget
> and telnet do the right thing, they fallback to IPv4:
>
>   $ telnet moule.informatique.univ-paris-diderot.fr 8080
>   Trying 2001:660:3301:8070::40...
>   Trying 194.254.199.40...
>   Connected to moule.informatique.univ-paris-diderot.fr.
>   Escape character is '^]'.
>
> On the other hand, url-retrieve-synchronously returns an empty string.
> Tcpdump shows that it's never even trying IPv4:
>
>   (with-current-buffer
>     (url-retrieve-synchronously
>      "http://moule.informatique.univ-paris-diderot.fr:8080")
>     (buffer-substring (point-min) (point-max)))
>   ""

I cannot reproduce that here.  Emacs is properly falling back to IPv4
and I get the expected contents from this command.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Juliusz Chroboczek]

Andreas Schwab wrote:

> I cannot reproduce that here.

Strange.  I've just reproduced this with a recent Emacs-24 (dated 5 July 2014).

> Emacs is properly falling back to IPv4

Have you confirmed that you're getting an RST and falling back to IPv4?

-- Juliusz

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Andreas Schwab]

Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:

> Have you confirmed that you're getting an RST and falling back to IPv4?

Yes, sure.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Juliusz Chroboczek]

The following patch works around the issue for me, so it could be
a timing-dependent problem:

--- url-gw.el.orig      2014-07-10 22:26:05.087977204 +0200
+++ url-gw.el   2014-07-10 22:25:32.972959153 +0200
@@ -240,8 +240,7 @@
                           name buffer host service
                           :type gw-method
                           ;; Use non-blocking socket if we can.
-                          :nowait (featurep 'make-network-process
-                                            '(:nowait t))))
+                          :nowait nil))
                         (`socks
                          (socks-open-network-stream name buffer host service))
                         (`telnet

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Juliusz Chroboczek]

Here's a trace (edited to remove irrelevant stuff).  I'm only guessing,
but to my untrained eyes it looks like the code around process.c:4920.

-- Juliusz

socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 13
fcntl(13, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
connect(13, {sa_family=AF_INET6, sin6_port=htons(8080), inet_pton(AF_INET6, "2001:660:3301:8070::40", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EINPROGRESS (Operation now in progress)
getsockname(13, {sa_family=AF_INET6, sin6_port=htons(33826), inet_pton(AF_INET6, "2a01:e34:ec22:84a0::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
fcntl(13, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
pselect6(14, [12], [13], NULL, {1957, 16494001}, {NULL, 8}) = 1 (out [13], left {1957, 10004292})
getsockopt(13, SOL_SOCKET, SO_ERROR, [111], [4]) = 0
close(13)                               = 0

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Andreas Schwab]

Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:

> Andreas Schwab wrote:
>
>> I cannot reproduce that here.
>
> Strange.  I've just reproduced this with a recent Emacs-24 (dated 5 July 2014).

I can as well, after disabling the proxy services.  Sorry.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Lars Ingebrigtsen]

Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:

> My machine has both IPv6 and IPv4.  I'm accessing a server that has both
> IPv6 and IPv4, but whose web server listens on IPv4 only.  The web server
> is doing the right thing, sending RST upon IPv6 connection attempts.  Wget
> and telnet do the right thing, they fallback to IPv4:
>
>   $ telnet moule.informatique.univ-paris-diderot.fr 8080
>   Trying 2001:660:3301:8070::40...
>   Trying 194.254.199.40...
>   Connected to moule.informatique.univ-paris-diderot.fr.
>   Escape character is '^]'.

Perhaps this should be a general network connection thing?

I know nothing about IPv6, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 25 Dec 2015 21:33:56 +0100
> Cc: 17976@debbugs.gnu.org
> 
> Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:
> 
> > My machine has both IPv6 and IPv4.  I'm accessing a server that has both
> > IPv6 and IPv4, but whose web server listens on IPv4 only.  The web server
> > is doing the right thing, sending RST upon IPv6 connection attempts.  Wget
> > and telnet do the right thing, they fallback to IPv4:
> >
> >   $ telnet moule.informatique.univ-paris-diderot.fr 8080
> >   Trying 2001:660:3301:8070::40...
> >   Trying 194.254.199.40...
> >   Connected to moule.informatique.univ-paris-diderot.fr.
> >   Escape character is '^]'.
> 
> Perhaps this should be a general network connection thing?

If by "general network connection" you mean on the C level in
process.c, then no, I don't think so.  Protocol-specific code is not
there, it's above that in Lisp.  IMO, url*.el is where this should be
handled.

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> If by "general network connection" you mean on the C level in
> process.c, then no, I don't think so.  Protocol-specific code is not
> there, it's above that in Lisp.  IMO, url*.el is where this should be
> handled.

I meant in `open-network-stream', so between the C layer and url*.el.

This is probably as relevant for, say, IMAP as it is for HTTP, I would
guess?

But I've never used IPv6.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Andreas Schwab]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:
>
>> My machine has both IPv6 and IPv4.  I'm accessing a server that has both
>> IPv6 and IPv4, but whose web server listens on IPv4 only.  The web server
>> is doing the right thing, sending RST upon IPv6 connection attempts.  Wget
>> and telnet do the right thing, they fallback to IPv4:
>>
>>   $ telnet moule.informatique.univ-paris-diderot.fr 8080
>>   Trying 2001:660:3301:8070::40...
>>   Trying 194.254.199.40...
>>   Connected to moule.informatique.univ-paris-diderot.fr.
>>   Escape character is '^]'.
>
> Perhaps this should be a general network connection thing?

It needs to be fixed in open-network-stream.  Currently it doesn't
provide a way to fall back to a different address of a given host.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Andreas Schwab]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> But I've never used IPv6.

This is independent of IPv6.  A hostname could also resolve to multiple
IPv4 addresses where only one of them is (currently) reachable.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Lars Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

> This is independent of IPv6.  A hostname could also resolve to multiple
> IPv4 addresses where only one of them is (currently) reachable.

Yeah, that's true.  Emacs should reconnect anyway.

But how does this work in practice?  How are we doing name resolution?
Do we have access to all the IP addresses?  Do we have access to the
IPv6 addresses without...  doing stuff?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Andreas Schwab]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Do we have access to all the IP addresses?

getaddrinfo gets them all.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Juliusz Chroboczek]

>> This is independent of IPv6.  A hostname could also resolve to multiple
>> IPv4 addresses where only one of them is (currently) reachable.

Please hear this man.

> But how does this work in practice?  How are we doing name resolution?
> Do we have access to all the IP addresses?  Do we have access to the
> IPv6 addresses without...  doing stuff?  :-)

If we don't, it's a bug that must be fixed.  The multihoming techniques
that are being developed at IETF right now rely on giving multiple
addresses to a single host (and implementing a routing infrastructure to
match), we really need to be trying them all.

-- Juliusz

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 387 bytes --]

I reproduced this bug in a test Fedora 23 environment where attempting to 
connect to one address failed (connection refused). The attached patch to Emacs 
master causes Emacs to try addresses addresses until one succeeds. Does this fix 
your problem? I suspect the code should check for error codes other than 
ECONNREFUSED, but I don't know what the list of such codes should be.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Try-other-addresses-when-connecting-to-multihomed.patch --]
[-- Type: text/x-diff; name="0001-Try-other-addresses-when-connecting-to-multihomed.patch", Size: 6477 bytes --]

From 958a23698c9f1b9b804aa79d410b11b07998b223 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sun, 12 Jun 2016 11:08:41 -0700
Subject: [PATCH] Try other addresses when connecting to multihomed

* src/process.c (decode_status, Fmake_network_process):
Support (connect . ADDRINFOS) status.
(connecting): New function, to support (connect . ADDRINFOS).
(connect_network_socket, check_for_dns, wait_for_socket_fds)
(wait_while_connecting, wait_reading_process_output, status_notify):
Use it.
(connect_network_socket) [!WINDOWSNT]:
If the connection failed with ECONNREFUSED and there are other
addresses to try, do not signal an error; instead, loop around
to try the next address.
(wait_reading_process_output): Advance to the next address
if there are multiple addresses and the first remaining address
failed.
* src/process.h (struct Lisp_Process.status): Adjust comment
to describe (connect . ADDRINFOS).
---
 src/process.c | 45 ++++++++++++++++++++++++++++++++-------------
 src/process.h |  4 +++-
 2 files changed, 35 insertions(+), 14 deletions(-)

diff --git a/src/process.c b/src/process.c
index e669278..a72bf20 100644
--- a/src/process.c
+++ b/src/process.c
@@ -542,6 +542,9 @@ decode_status (Lisp_Object l, Lisp_Object *symbol, Lisp_Object *code,
 {
   Lisp_Object tem;
 
+  if (CONSP (l) && EQ (XCAR (l), Qconnect))
+    l = XCAR (l);
+
   if (SYMBOLP (l))
     {
       *symbol = l;
@@ -3106,6 +3109,12 @@ finish_after_tls_connection (Lisp_Object proc)
 }
 #endif
 
+static bool
+connecting (struct Lisp_Process *p)
+{
+  return CONSP (p->status) && EQ (XCAR (p->status), Qconnect);
+}
+
 static void
 connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
                         Lisp_Object use_external_socket_p)
@@ -3288,9 +3297,10 @@ connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
 	  eassert (FD_ISSET (s, &fdset));
 	  if (getsockopt (s, SOL_SOCKET, SO_ERROR, &xerrno, &len) < 0)
 	    report_file_error ("Failed getsockopt", Qnil);
-	  if (xerrno)
+	  if (xerrno == 0)
+	    break;
+	  if (xerrno != ECONNREFUSED || NILP (addrinfos))
 	    report_file_errno ("Failed connect", Qnil, xerrno);
-	  break;
 	}
 #endif /* !WINDOWSNT */
 
@@ -3399,7 +3409,8 @@ connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
       /* We may get here if connect did succeed immediately.  However,
 	 in that case, we still need to signal this like a non-blocking
 	 connection.  */
-      pset_status (p, Qconnect);
+      if (! (connecting (p) && EQ (XCDR (p->status), addrinfos)))
+	pset_status (p, Fcons (Qconnect, addrinfos));
       if (!FD_ISSET (inch, &connect_wait_mask))
 	{
 	  FD_SET (inch, &connect_wait_mask);
@@ -3960,7 +3971,7 @@ usage: (make-network-process &rest ARGS)  */)
   if (!p->is_server && NILP (addrinfos))
     {
       p->dns_request = dns_request;
-      p->status = Qconnect;
+      p->status = list1 (Qconnect);
       return proc;
     }
 #endif
@@ -4673,7 +4684,7 @@ check_for_dns (Lisp_Object proc)
       addrinfos = Fnreverse (addrinfos);
     }
   /* The DNS lookup failed. */
-  else if (EQ (p->status, Qconnect))
+  else if (connecting (p))
     {
       deactivate_process (proc);
       pset_status (p, (list2
@@ -4686,7 +4697,7 @@ check_for_dns (Lisp_Object proc)
   free_dns_request (proc);
 
   /* This process should not already be connected (or killed). */
-  if (!EQ (p->status, Qconnect))
+  if (! connecting (p))
     return Qnil;
 
   return addrinfos;
@@ -4697,8 +4708,7 @@ check_for_dns (Lisp_Object proc)
 static void
 wait_for_socket_fds (Lisp_Object process, char const *name)
 {
-  while (XPROCESS (process)->infd < 0
-	 && EQ (XPROCESS (process)->status, Qconnect))
+  while (XPROCESS (process)->infd < 0 && connecting (XPROCESS (process)))
     {
       add_to_log ("Waiting for socket from %s...", build_string (name));
       wait_reading_process_output (0, 20 * 1000 * 1000, 0, 0, Qnil, NULL, 0);
@@ -4708,7 +4718,7 @@ wait_for_socket_fds (Lisp_Object process, char const *name)
 static void
 wait_while_connecting (Lisp_Object process)
 {
-  while (EQ (XPROCESS (process)->status, Qconnect))
+  while (connecting (XPROCESS (process)))
     {
       add_to_log ("Waiting for connection...");
       wait_reading_process_output (0, 20 * 1000 * 1000, 0, 0, Qnil, NULL, 0);
@@ -5010,7 +5020,7 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 	update_status (wait_proc);
       if (wait_proc
 	  && ! EQ (wait_proc->status, Qrun)
-	  && ! EQ (wait_proc->status, Qconnect))
+	  && ! connecting (wait_proc))
 	{
 	  bool read_some_bytes = false;
 
@@ -5520,9 +5530,18 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 #endif
 	      if (xerrno)
 		{
-		  p->tick = ++process_tick;
-		  pset_status (p, list2 (Qfailed, make_number (xerrno)));
+		  Lisp_Object addrinfos
+		    = connecting (p) ? XCDR (p->status) : Qnil;
+		  if (!NILP (addrinfos))
+		    XSETCDR (p->status, XCDR (addrinfos));
+		  else
+		    {
+		      p->tick = ++process_tick;
+		      pset_status (p, list2 (Qfailed, make_number (xerrno)));
+		    }
 		  deactivate_process (proc);
+		  if (!NILP (addrinfos))
+		    connect_network_socket (proc, addrinfos, Qnil);
 		}
 	      else
 		{
@@ -6999,7 +7018,7 @@ status_notify (struct Lisp_Process *deleting_process,
 
 	  /* If process is still active, read any output that remains.  */
 	  while (! EQ (p->filter, Qt)
-		 && ! EQ (p->status, Qconnect)
+		 && ! connecting (p)
 		 && ! EQ (p->status, Qlisten)
 		 /* Network or serial process not stopped:  */
 		 && ! EQ (p->command, Qt)
diff --git a/src/process.h b/src/process.h
index 4430377..6c227bc 100644
--- a/src/process.h
+++ b/src/process.h
@@ -83,7 +83,9 @@ struct Lisp_Process
     Lisp_Object mark;
 
     /* Symbol indicating status of process.
-       This may be a symbol: run, open, closed, listen, connect, or failed.
+       This may be a symbol: run, open, closed, listen, or failed.
+       Or it may be a pair (connect . ADDRINFOS) where ADDRINFOS is
+       a list of remaining (PROTOCOL . ADDRINFO) pairs to try.
        Or it may be (failed ERR) where ERR is an integer, string or symbol.
        Or it may be a list, whose car is stop, exit or signal
        and whose cdr is a pair (EXIT_CODE . COREDUMP_FLAG)
-- 
2.5.5

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Noam Postavsky]

close 17976
quit

Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:

> I can confirm that I still see the bug with Emacs 25, and I don't see the
> bug with current trunk (commit g2989ad9).
>
> I'm not sure if it's possible, but I'd appreciate it if this patch could
> be backported to Emacs 25 (which is what I run usually).

I think it's clear enough by now that there's not going to be any more
Emacs 25 releases (unless there's another emergency, in which case this
patch still won't go in).