Async DNS lookups

Async DNS lookups

[Lars Magne Ingebrigtsen]

I've been trying to chase down reasons for why url-retrieve sometimes
makes Emacs totally unresponsive, and I suspect that it has to do with
DNS lookups.

Gnus is triggering url-retrieve asynchronously from a process filter.
url-retrieve then has to resolve the domain name, of course, and then do
the HTTP stuff.  If the DNS resolving fails, or it's slow, then Emacs
will hang entirely while resolving the domain names.

Which is bad.

I don't really see any easy fixes for this with Emacs' feature set at
the moment.  A process filter will block Emacs' "main thread", and it
needs to resolve the addresses, so...

However, we could side-step the issue by either:

1) implement a C-level resolving functionality that has a callback that
will be called upon resolving the address.  This seems hard.

2) add a callback mechanism to dns.el.  Somebody kah-razy has already
implemented a resolver in Emacs Lisp, so that seems like a possible
option.  But can you have process filters/sentinels on UDP processes?

Before I do anything here, does anybody have any thoughts on this issue?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Async DNS lookups

[Julien Danjou]

On Sat, Oct 30 2010, Lars Magne Ingebrigtsen wrote:

> Before I do anything here, does anybody have any thoughts on this
> issue?

Just thinking out loud, but once upon a time I used udns[1] to do
asynchronous DNS resolving.

I don't know if it's an option for Emacs to depends on such a library,
but at least it can inspire something.

[1]  http://www.corpit.ru/mjt/udns.html

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

Re: Async DNS lookups

[Ted Zlatanov]

On Sat, 30 Oct 2010 13:57:22 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> 1) implement a C-level resolving functionality that has a callback that
LMI> will be called upon resolving the address.  This seems hard.

LMI> 2) add a callback mechanism to dns.el.  Somebody kah-razy has already
LMI> implemented a resolver in Emacs Lisp, so that seems like a possible
LMI> option.  But can you have process filters/sentinels on UDP processes?

How about:

3) use symbols or other opaque structures to represent domain lookups
and asynchronously set a property when the DNS lookup succeeds.

Ted

Re: Async DNS lookups

[Helmut Eller]

* Lars Magne Ingebrigtsen [2010-10-30 11:57] writes:

> Before I do anything here, does anybody have any thoughts on this issue?

Emacs supposedly has non-bocking connect, but I think that even that
uses synchronous DNS lookups.  It seems that nobody uses the
non-blocking connect feature; maybe it should be removed.

The asynchronous DNS code could also be written as an external C program
that Emacs starts when needed; like emacsserver used to work.

Helmut

Re: Async DNS lookups

[Stefan Monnier]

> 2) add a callback mechanism to dns.el.  Somebody kah-razy has already
> implemented a resolver in Emacs Lisp, so that seems like a possible
> option.  But can you have process filters/sentinels on UDP processes?

While I generally like to move Emacs's C code to Elisp, I prefer not to
move code from C libraries to Elisp.


        Stefan

Re: Async DNS lookups

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> While I generally like to move Emacs's C code to Elisp, I prefer not to
> move code from C libraries to Elisp.

I wasn't suggesting using an Emacs Lisp-based resolver generally, only
adding an async resolver to use when doing async stuff.  It looks
doable, so I'll add a callback-based interface to dns.el.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Async DNS lookups

[Tom Tromey]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

Lars> I don't really see any easy fixes for this with Emacs' feature set at
Lars> the moment.  A process filter will block Emacs' "main thread", and it
Lars> needs to resolve the addresses, so...

Lars> 1) implement a C-level resolving functionality that has a callback that
Lars> will be called upon resolving the address.  This seems hard.

There is a GNU library for this:

http://directory.fsf.org/project/c-ares/


Apparently glibc already includes getaddrinfo_a for this.  It seems to
be undocumented, but this has some info & pointers:

http://bitsup.blogspot.com/2008/09/asynchronous-dns-lookups-on-linux-with.html


A couple other approaches come to mind as well.

You could run the synchronous resolver in a new thread, and have the
main thread run the event loop.

You could use a small helper program instead of calling the resolver
in-process.  That is basically the same as the previous idea,
substituting a process for a thread.

Tom

Re: Async DNS lookups

[Ken Raeburn]

On Nov 1, 2010, at 12:09, Tom Tromey wrote:
> A couple other approaches come to mind as well.
> 
> You could run the synchronous resolver in a new thread, and have the
> main thread run the event loop.
> 
> You could use a small helper program instead of calling the resolver
> in-process.  That is basically the same as the previous idea,
> substituting a process for a thread.

These would have the advantage that, rather than interacting with DNS directly, you can call getaddrinfo and use /etc/hosts or anything else configured by nsswitch or equivalent.  Assuming that all name-resolution calls should go straight to DNS (when you're not specifically looking for non-A/AAAA/PTR data) would be incorrect....

I think getaddrinfo_a looks nice, but I'd rather not use it unless we also have a workaround for non-glibc systems.  Presumably that would use your basic "spin off a thread that blocks in getaddrinfo and invokes a callback when done" approach, which (according to the blog Tom pointed to) is all getaddrinfo_a does anyways.  Maybe that means the code could be borrowed; maybe that means we should just implement the latter and not bother with getaddrinfo_a.

Ken

Re: Async DNS lookups

[Lars Magne Ingebrigtsen]

Ken Raeburn <raeburn@raeburn.org> writes:

>  Presumably that would use your basic "spin off a thread that blocks
> in getaddrinfo and invokes a callback when done" approach, which
> (according to the blog Tom pointed to) is all getaddrinfo_a does
> anyways.

Are we Allowed to spin off a thread in Emacs?  If so, this sounds like a
no-brainer to do in C instead of in dns.el.  

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Async DNS lookups

[Stefan Monnier]

>> Presumably that would use your basic "spin off a thread that blocks
>> in getaddrinfo and invokes a callback when done" approach, which
>> (according to the blog Tom pointed to) is all getaddrinfo_a does
>> anyways.
> Are we Allowed to spin off a thread in Emacs?

Of course, with the usual restriction: no access to shared data without
proper synchronization.

> If so, this sounds like a no-brainer to do in C instead of in dns.el.

Patch welcome,


        Stefan

Re: Async DNS lookups

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> If so, this sounds like a no-brainer to do in C instead of in dns.el.
>
> Patch welcome,

I meant for somebody who knows what they're doing.  :-)

I, like, see it happening like this:

1) User does: (resolve-async "gwene.org" #'my-callback my-data)

2) C does pthread_create() (right?), and lets the main thread return
   from the call.  The other thread just does a normal getaddrinfo().

3) When getaddrinfo() returns, we do
   (funcall #'my-callback result my-data)   

But 3) probably needs to be in the same sort of state that timers run
in, so that we don't have two Emacs Lisp threads running at the same
time.  Any pointers where to look for how to "halt" the main Emacs
thread?  Or something?   
   
-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Async DNS lookups

[Lars Magne Ingebrigtsen]

Tom Tromey <tromey@redhat.com> writes:

> I would suggest just keeping all the thread stuff in C.  Don't provide
> any callback.  Instead, have the main thread run the event loop (a la
> sit-for -- I don't know the C functions offhand) and return an answer
> when the worker thread reports back.

This is meant to be used to provide non-blocking domain lookups for
things that are asynchronous.  Having the Lisp code wait for this async
lookup to finish would be, er, not helpful.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Async DNS lookups

[Tom Tromey]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

Lars> 2) C does pthread_create() (right?), and lets the main thread return
Lars>    from the call.  The other thread just does a normal getaddrinfo().

Lars> 3) When getaddrinfo() returns, we do
Lars>    (funcall #'my-callback result my-data)   

Lars> But 3) probably needs to be in the same sort of state that timers run
Lars> in, so that we don't have two Emacs Lisp threads running at the same
Lars> time.  Any pointers where to look for how to "halt" the main Emacs
Lars> thread?  Or something?   
   
I would suggest just keeping all the thread stuff in C.  Don't provide
any callback.  Instead, have the main thread run the event loop (a la
sit-for -- I don't know the C functions offhand) and return an answer
when the worker thread reports back.

You especially want to avoid referencing any lisp stuff from the worker
thread.  If you do that then you will need to make the GC thread-aware.
This is doable (it is on the concurrency branch), but it is overkill for
this application.

Tom

Re: Async DNS lookups

[Tom Tromey]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

Lars> This is meant to be used to provide non-blocking domain lookups for
Lars> things that are asynchronous.  Having the Lisp code wait for this async
Lars> lookup to finish would be, er, not helpful.  :-)

I guess there are 2 cases.

One is make-network-process with :nowait t.
In that case it seems to me that the work could be done entirely in C
and failures reported to lisp via the sentinel function.

The other is a blocking open.  In this case, Emacs is already "blocking"
in the sense that users can't do anything but wait.  However, currently
users also can't C-g out of it, and other I/O is blocked waiting for
name resolution.

I think my proposed approach could solve both these scenarios.

If there's a reason to want async resolution but nothing else, then
yeah, you'd need something more complicated.

Tom

Re: Async DNS lookups

[Stefan Monnier]

> 3) When getaddrinfo() returns, we do
>    (funcall #'my-callback result my-data)

> But 3) probably needs to be in the same sort of state that timers run
> in, so that we don't have two Emacs Lisp threads running at the same
> time.  Any pointers where to look for how to "halt" the main Emacs
> thread?  Or something?

You can't "halt" the main thread, you can only wait for it to take
a break.  E.g. you could stuff your callback into pending_funcalls.
Of course, pending_funcalls would then need to be protected by a mutex.


        Stefan

Re: Async DNS lookups

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> You can't "halt" the main thread, you can only wait for it to take
> a break.  E.g. you could stuff your callback into pending_funcalls.
> Of course, pending_funcalls would then need to be protected by a mutex.

Which mutex would that be?

I've grepped a bit through the code, and the usage pattern seems to be a
la: 

    pending_funcalls
      = Fcons (list3 (Qrun_hook_with_args,
		      Qdelete_terminal_functions, terminal),
	       pending_funcalls);

I don't see any obvious mutexes...

I've also grepped around for pthread_* stuff, and in particular
pthread_create, and I can't really find anything.  Do you have an
example of a code snippet that uses a thread inside Emacs? 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Async DNS lookups

[Davis Herring]

>> Of course, pending_funcalls would then need to be protected by a mutex.
>
> Which mutex would that be?

I think the implication is that you would have to add one (and use it
every time any code whatsoever accessed that variable).

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Async DNS lookups

[Lars Magne Ingebrigtsen]

"Davis Herring" <herring@lanl.gov> writes:

> I think the implication is that you would have to add one (and use it
> every time any code whatsoever accessed that variable).

It's already being used several places without any locking.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Async DNS lookups

[Stefan Monnier]

>> I think the implication is that you would have to add one (and use it
>> every time any code whatsoever accessed that variable).
> It's already being used several places without any locking.

Indeed, because it's only accessed by a single thread right now (it's
not even accessed from a signal handler).  So you'd have to add locking
if it becomes shared between threads.


        Stefan

Re: Async DNS lookups

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> I think the implication is that you would have to add one (and use it
>>> every time any code whatsoever accessed that variable).
>> It's already being used several places without any locking.
>
> Indeed, because it's only accessed by a single thread right now (it's
> not even accessed from a signal handler).  So you'd have to add locking
> if it becomes shared between threads.

Since it is using Lisp values it cannot be used by anything but the main
thread anyway.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Async DNS lookups

[Stefan Monnier]

>>>> I think the implication is that you would have to add one (and use it
>>>> every time any code whatsoever accessed that variable).
>>> It's already being used several places without any locking.
>> Indeed, because it's only accessed by a single thread right now (it's
>> not even accessed from a signal handler).  So you'd have to add locking
>> if it becomes shared between threads.
> Since it is using Lisp values it cannot be used by anything but the main
> thread anyway.

It's true that the other thread can't do things like call Fcons (before
the allocation code is not synchronized either), but the other thread can
still perform an assignment to a Lisp_Object variable, as long as that
variable is properly synchronized.

So you just have to ensure that the allocation of the pending_funcalls
element is done before forking the new thread.
I.e. when the name resolution is over, the other thread will want to do
something like:

   lock(pending_funcalls);
   setcar(result, pending_funcalls);
   pending_funcalls = result;
   unlock(pending_funcalls);

Of course, that leaves open the problem of how to translate the non-Lisp
result into Lisp_Objects.

I.e. you'll probably want to use some new variable, very similar to
pending_funcalls, but for "pending_c_funcalls" so that the pending calls
can first do some C preprocessing to turn the C result data into Lisp data
and then call the Elisp callback.


        Stefan

Re: Async DNS lookups

[Ken Raeburn]

By "threads" I meant "threads or helper subprocesses"; we don't *have* to introduce threads to Emacs right now if it would be too complicated for this one little task.

That said, I do hope we can go that route, as I can imagine other things that might be improved with careful use of threads.  For example, file access on a slow or unresponsive NFS server blocks the thread trying to do the I/O... but if that's not the thread managing the display and checking for ^G, maybe we can make it appear interruptible to the user, or do redisplay as needed while waiting for the file I/O to finish.  (POSIX has an async I/O API that could help too, but I'm not sure if it's portable enough yet, and again the glibc implementation appears to just spin off helper threads.)

Ken

Re: Async DNS lookups

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I.e. you'll probably want to use some new variable, very similar to
> pending_funcalls, but for "pending_c_funcalls" so that the pending calls
> can first do some C preprocessing to turn the C result data into Lisp data
> and then call the Elisp callback.

So -- what I'm getting from this is that:

1) The normal Lisp functions (like Fcons) can only be used in the main
thread, 

2) so we need a new list, pending_c_funcalls, which might perhaps not be
a Lisp thing at all, but would be (for instance) a chained struct
containing pointers to the callback Lisp_Object data, as well as a place
to stash C values and a mechanism to say what these C objects are, so
that they can be converted to Lisp_Objects, and

3) nobody has done anything like this in Emacs before?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Async DNS lookups

[Davis Herring]

>    lock(pending_funcalls);
>    setcar(result, pending_funcalls);
>    pending_funcalls = result;
>    unlock(pending_funcalls);
[snip]
> I.e. you'll probably want to use some new variable, very similar to
> pending_funcalls, but for "pending_c_funcalls" so that the pending calls
> can first do some C preprocessing to turn the C result data into Lisp data
> and then call the Elisp callback.

And then you don't want to lock the pending_funcalls variable after all:
you want to have a lock on the pending_c_funcalls and do something like

/* DNS thread */
x->data=lookup(...); /* slow */
lock(pending_c_funcalls_mutex);
x->next=pending_c_funcalls;
pending_c_funcalls=x;
unlock(pending_c_funcalls_mutex);

/* main thread */
lock(pending_c_funcalls_mutex);
for(;pending_c_funcalls;pending_c_funcalls=pending_c_funcalls->next)
  pending_funcalls=Fcons(convert_async(pending_c_funcalls),
                         pending_funcalls);
unlock(pending_c_funcalls_mutex);
/* ...existing Ffuncall() code... */

Right?

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Async DNS lookups

[Stefan Monnier]

> And then you don't want to lock the pending_funcalls variable after all:
> you want to have a lock on the pending_c_funcalls and do something like

> /* DNS thread */
x-> data=lookup(...); /* slow */
> lock(pending_c_funcalls_mutex);
x-> next=pending_c_funcalls;
> pending_c_funcalls=x;
> unlock(pending_c_funcalls_mutex);

> /* main thread */
> lock(pending_c_funcalls_mutex);
> for(;pending_c_funcalls;pending_c_funcalls=pending_c_funcalls->next)
>   pending_funcalls=Fcons(convert_async(pending_c_funcalls),
>                          pending_funcalls);
> unlock(pending_c_funcalls_mutex);
> /* ...existing Ffuncall() code... */

> Right?

Right, except that instead of

   pending_funcalls=Fcons(convert_async(pending_c_funcalls),
                          pending_funcalls);

I'd just do

   pending_c_funcalls->f (pending_c_funcalls->arg);
   

-- Stefan