bug#23620: 25.0.94; url-retrieve fails on localhost

bug#23620: 25.0.94; url-retrieve fails on localhost

[Artur Malabarba]

Assuming you have a webserver serving on localhost:5100, run the
following snippet:

(url-retrieve "http://localhost:5100/"
              (lambda (&rest r)
                (message "%S" r)))

It fails with the following error:
    (:error (error connection-failed "failed with code 111\n"
                   :host "localhost"
                   :service 5100))

Note that if I switch to a terminal and try curl:
    curl "http://localhost:5100/"
it works fine.

Using `url-retrieve' on remote urls also works fine.

bug#23620: 25.0.94; url-retrieve fails on localhost

[Paul Eggert]

I'm not observing this problem on my platform (Fedora 23 x86-64, local web 
server is sthttpd).

I do see the problem if I attempt to connect to a non-webserver attached to a 
local TCP port (in my case, port 943 attached to sshd); perhaps your webserver 
isn't set up properly? But that situation looks like Bug#20159, i.e., user error 
not a bug in url-retrieve. Its callbacks are supposed to check status, not 
merely call 'message' willy-nilly.

bug#23620: 25.0.94; url-retrieve fails on localhost

[Artur Malabarba]

Paul Eggert <eggert@cs.ucla.edu> writes:

> I'm not observing this problem on my platform (Fedora 23 x86-64, local
> web server is sthttpd).
>[...]
> perhaps your webserver isn't set up properly?

Indeed, I just tested with a different server project I have here and
there was no problem, so it seems to be specific to how this server is
configured.

However, I wouldn't go as far as to say the server is MISconfigured.
Like I said, it works fine with curl, wget, webrowsers, and any other
means I tried. Only url-retrieve failed on it.
Does anyone know how we could look further into it?

> I do see the problem if I attempt to connect to a non-webserver
> attached to a local TCP port (in my case, port 943 attached to sshd);

In my case it's just a rails server started with:
    bundle exec rails server -p 5100

> But that situation looks like Bug#20159, i.e., user error not a bug in
> url-retrieve. Its callbacks are supposed to check status, not merely
> call 'message' willy-nilly.

I used that callback simply to point out the 111 error code. The problem
here is that the request the fails, not how the user handles the failure.

bug#23620: 25.0.94; url-retrieve fails on localhost

[Paul Eggert]

On 06/01/2016 07:55 AM, Artur Malabarba wrote:
>> >I do see the problem if I attempt to connect to a non-webserver
>> >attached to a local TCP port (in my case, port 943 attached to sshd);
> In my case it's just a rails server started with:
>      bundle exec rails server -p 5100
>
Possibly url-retrieve is being too picky about what it accepts. I'm 
afraid you'll have to debug it on your end, though. Sorry, I don't have 
much experience with this sort of debugging, but perhaps you can track 
all the bytes going from Emacs to your server and back again and see 
what the problem is.

bug#23620: 25.0.94; url-retrieve fails on localhost

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 98 bytes --]

After some more digging, looks like this is a duplicate of #17976. Could
someone mark it as such?

[-- Attachment #2: Type: text/html, Size: 120 bytes --]

bug#23620: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

No further comment on the proposed patch, and as it should be a win for 
non-WINDOWSNT I decided to install it on the Emacs master branch. I 
looked into how libcurl addresses the issue, and it doesn't care whether 
the errno value reported by getsockopt is ECONNREFUSED so I adjusted the 
Emacs patch accordingly (see attached) before installing. Please give it 
a try if you have the time.

This patch does not affect behavior if WINDOWSNT. If the bug does not 
occur on MS-Windows I hope we can close Bug#17976. Otherwise we should 
leave the bug open for the WINDOWSNT case.

[-- Attachment #2: 0001-Try-other-addresses-when-connecting-to-multihomed.patch --]
[-- Type: text/x-patch, Size: 6589 bytes --]

From 6a90b600323afc42ae3bd644d82ba7ea054a816a Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sun, 26 Jun 2016 23:27:21 +0200
Subject: [PATCH] Try other addresses when connecting to multihomed

Problem reported by Juliusz Chroboczek (Bug#17976)
and by Artur Malabarba (Bug#23620).
Patch from a suggestion by Andreas Schwab in:
http://bugs.gnu.org/17976#39
This patch is for non-MS-Windows platforms.
I don't know the situation on MS-Windows.
* src/process.c (connecting_status):
New function, for (connect . ADDRINFOS).
(connect_network_socket, check_for_dns, wait_for_socket_fds)
(wait_while_connecting, wait_reading_process_output, status_notify):
Use it.
(decode_status, Fmake_network_process):
Support (connect . ADDRINFOS) status.
(connect_network_socket) [!WINDOWSNT]:
If the connection failed and there are other addresses to try, do not
signal an error; instead, loop around to try the next address.
(wait_reading_process_output): Advance to the next address
if there are multiple addresses and the first remaining address
failed.
* src/process.h (struct Lisp_Process.status): Adjust comment
to describe (connect . ADDRINFOS).
---
 src/process.c | 47 +++++++++++++++++++++++++++++++++++------------
 src/process.h |  4 +++-
 2 files changed, 38 insertions(+), 13 deletions(-)

diff --git a/src/process.c b/src/process.c
index e669278..ed0c529 100644
--- a/src/process.c
+++ b/src/process.c
@@ -533,6 +533,14 @@ status_convert (int w)
     return Qrun;
 }
 
+/* True if STATUS is that of a process attempting connection.  */
+
+static bool
+connecting_status (Lisp_Object status)
+{
+  return CONSP (status) && EQ (XCAR (status), Qconnect);
+}
+
 /* Given a status-list, extract the three pieces of information
    and store them individually through the three pointers.  */
 
@@ -542,6 +550,9 @@ decode_status (Lisp_Object l, Lisp_Object *symbol, Lisp_Object *code,
 {
   Lisp_Object tem;
 
+  if (connecting_status (l))
+    l = XCAR (l);
+
   if (SYMBOLP (l))
     {
       *symbol = l;
@@ -3288,9 +3299,10 @@ connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
 	  eassert (FD_ISSET (s, &fdset));
 	  if (getsockopt (s, SOL_SOCKET, SO_ERROR, &xerrno, &len) < 0)
 	    report_file_error ("Failed getsockopt", Qnil);
-	  if (xerrno)
+	  if (xerrno == 0)
+	    break;
+	  if (NILP (addrinfos))
 	    report_file_errno ("Failed connect", Qnil, xerrno);
-	  break;
 	}
 #endif /* !WINDOWSNT */
 
@@ -3399,7 +3411,9 @@ connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
       /* We may get here if connect did succeed immediately.  However,
 	 in that case, we still need to signal this like a non-blocking
 	 connection.  */
-      pset_status (p, Qconnect);
+      if (! (connecting_status (p->status)
+	     && EQ (XCDR (p->status), addrinfos)))
+	pset_status (p, Fcons (Qconnect, addrinfos));
       if (!FD_ISSET (inch, &connect_wait_mask))
 	{
 	  FD_SET (inch, &connect_wait_mask);
@@ -3960,7 +3974,7 @@ usage: (make-network-process &rest ARGS)  */)
   if (!p->is_server && NILP (addrinfos))
     {
       p->dns_request = dns_request;
-      p->status = Qconnect;
+      p->status = list1 (Qconnect);
       return proc;
     }
 #endif
@@ -4673,7 +4687,7 @@ check_for_dns (Lisp_Object proc)
       addrinfos = Fnreverse (addrinfos);
     }
   /* The DNS lookup failed. */
-  else if (EQ (p->status, Qconnect))
+  else if (connecting_status (p->status))
     {
       deactivate_process (proc);
       pset_status (p, (list2
@@ -4686,7 +4700,7 @@ check_for_dns (Lisp_Object proc)
   free_dns_request (proc);
 
   /* This process should not already be connected (or killed). */
-  if (!EQ (p->status, Qconnect))
+  if (! connecting_status (p->status))
     return Qnil;
 
   return addrinfos;
@@ -4698,7 +4712,7 @@ static void
 wait_for_socket_fds (Lisp_Object process, char const *name)
 {
   while (XPROCESS (process)->infd < 0
-	 && EQ (XPROCESS (process)->status, Qconnect))
+	 && connecting_status (XPROCESS (process)->status))
     {
       add_to_log ("Waiting for socket from %s...", build_string (name));
       wait_reading_process_output (0, 20 * 1000 * 1000, 0, 0, Qnil, NULL, 0);
@@ -4708,7 +4722,7 @@ wait_for_socket_fds (Lisp_Object process, char const *name)
 static void
 wait_while_connecting (Lisp_Object process)
 {
-  while (EQ (XPROCESS (process)->status, Qconnect))
+  while (connecting_status (XPROCESS (process)->status))
     {
       add_to_log ("Waiting for connection...");
       wait_reading_process_output (0, 20 * 1000 * 1000, 0, 0, Qnil, NULL, 0);
@@ -5010,7 +5024,7 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 	update_status (wait_proc);
       if (wait_proc
 	  && ! EQ (wait_proc->status, Qrun)
-	  && ! EQ (wait_proc->status, Qconnect))
+	  && ! connecting_status (wait_proc->status))
 	{
 	  bool read_some_bytes = false;
 
@@ -5520,9 +5534,18 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 #endif
 	      if (xerrno)
 		{
-		  p->tick = ++process_tick;
-		  pset_status (p, list2 (Qfailed, make_number (xerrno)));
+		  Lisp_Object addrinfos
+		    = connecting_status (p->status) ? XCDR (p->status) : Qnil;
+		  if (!NILP (addrinfos))
+		    XSETCDR (p->status, XCDR (addrinfos));
+		  else
+		    {
+		      p->tick = ++process_tick;
+		      pset_status (p, list2 (Qfailed, make_number (xerrno)));
+		    }
 		  deactivate_process (proc);
+		  if (!NILP (addrinfos))
+		    connect_network_socket (proc, addrinfos, Qnil);
 		}
 	      else
 		{
@@ -6999,7 +7022,7 @@ status_notify (struct Lisp_Process *deleting_process,
 
 	  /* If process is still active, read any output that remains.  */
 	  while (! EQ (p->filter, Qt)
-		 && ! EQ (p->status, Qconnect)
+		 && ! connecting_status (p->status)
 		 && ! EQ (p->status, Qlisten)
 		 /* Network or serial process not stopped:  */
 		 && ! EQ (p->command, Qt)
diff --git a/src/process.h b/src/process.h
index 4430377..6c227bc 100644
--- a/src/process.h
+++ b/src/process.h
@@ -83,7 +83,9 @@ struct Lisp_Process
     Lisp_Object mark;
 
     /* Symbol indicating status of process.
-       This may be a symbol: run, open, closed, listen, connect, or failed.
+       This may be a symbol: run, open, closed, listen, or failed.
+       Or it may be a pair (connect . ADDRINFOS) where ADDRINFOS is
+       a list of remaining (PROTOCOL . ADDRINFO) pairs to try.
        Or it may be (failed ERR) where ERR is an integer, string or symbol.
        Or it may be a list, whose car is stop, exit or signal
        and whose cdr is a pair (EXIT_CODE . COREDUMP_FLAG)
-- 
2.5.5

bug#23620: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Juliusz Chroboczek]

Hi Paul,

I'm the original submitter of #17976.

I can confirm that I still see the bug with Emacs 25, and I don't see the
bug with current trunk (commit g2989ad9).

I'm not sure if it's possible, but I'd appreciate it if this patch could
be backported to Emacs 25 (which is what I run usually).

Thanks for your help,

-- Juliusz Chroboczek

bug#17976: 24.3; url-retrieve-synchronously doesn't fallback to IPv4

[Noam Postavsky]

close 17976
quit

Juliusz Chroboczek <jch@pps.univ-paris-diderot.fr> writes:

> I can confirm that I still see the bug with Emacs 25, and I don't see the
> bug with current trunk (commit g2989ad9).
>
> I'm not sure if it's possible, but I'd appreciate it if this patch could
> be backported to Emacs 25 (which is what I run usually).

I think it's clear enough by now that there's not going to be any more
Emacs 25 releases (unless there's another emergency, in which case this
patch still won't go in).