Asynchronous DNS resolving

Asynchronous DNS resolving

[Lars Ingebrigtsen]

The major usability problem in eww now seems to be synchronous DNS
resolution.

When you go to a web page, eww will render it, and then insert images
asynchronously.  All the network stuff happens pretty asynchronously --
except for the DNS lookups.  If you're unlucky, a lookup may take
several seconds, and there may be several of these lookups.  Meanwhile,
Emacs is pretty much frozen while waiting for the DNS packets to find
their way back to home base...

This is seriously yucky.

A few years back I looked into making the Emacs domain lookup thing be
asynchronous, but that was way beyond my knowledge of Emacs internals,
and what happens when I fork Emacs, and what Emacs Lisp primitives can
be used in the forked instance.

So perhaps doing what Firefox used to do in the 90s would be a possibly
solution, anyway?  That is, have a teensy multi-threaded executable that
reads commands from STDIN and outputs the replies on STDOUT.  Like
lib-src/dns-helper.c or something.

The command format would be something like

<tag> <command> <data>

I.e.,

45 lookup fsf.org
46 lookup images.fsf.org
...

(Tags for streaming commands and matching up the responses with the
requests, a la IMAP.)

We'd then have something like

(with-domain-name-lookup (ip "fsf.org")
  (message "We got %s and now we're going to connect" ip)
  (with-url ("http://fsf.org" :ip-address ip ...)
    (pop-to-buffer (current-buffer))))

to do a completely asynchronous web request.

Or does anybody have a better idea for solving this problem?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Asynchronous DNS resolving

[Paul Eggert]

Lars Ingebrigtsen wrote:
> A few years back I looked into making the Emacs domain lookup thing be
> asynchronous, but that was way beyond my knowledge of Emacs internals,
> and what happens when I fork Emacs, and what Emacs Lisp primitives can
> be used in the forked instance.

Nowadays there is getaddrinfo_a; will that do?

Re: Asynchronous DNS resolving

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Lars Ingebrigtsen wrote:
>> A few years back I looked into making the Emacs domain lookup thing be
>> asynchronous, but that was way beyond my knowledge of Emacs internals,
>> and what happens when I fork Emacs, and what Emacs Lisp primitives can
>> be used in the forked instance.
>
> Nowadays there is getaddrinfo_a; will that do?

Sure; if you can get Emacs to provide an interface to that function with
an async callback, that would be great.  A process filter on the lookup,
perhaps?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Asynchronous DNS resolving

[Paul Eggert]

Lars Ingebrigtsen wrote:
> Sure; if you can get Emacs to provide an interface to that function with
> an async callback, that would be great.  A process filter on the lookup,
> perhaps?

I suppose so.  It's not something I've done before, alas.

Re: Asynchronous DNS resolving

[Stefan Monnier]

> Sure; if you can get Emacs to provide an interface to that function with
> an async callback, that would be great.  A process filter on the lookup,
> perhaps?

Actually, do we really need to make the lookup Elisp-visible?  I mean,
don't we currently just pass the DNS name to make-network-process which
then does the DNS lookup and the TCP connect?  If so, we don't really
need to make getaddrinfo_a visible at the Elisp level, we just
need to make use of it in make-network-process if :nowait was specified.

Am I missing something?


        Stefan

Re: Asynchronous DNS resolving

[chad]

Not what you asked, but related:

Modern browsers use a technique called DNS pre-fetching to increase
the responsiveness of their UI.  More or less, they pre-resolve the
DNS names of every link on a page when rendering it (usually with
some async technique). This helps with both images and links, which
can be a pretty big deal with a typical Web 2.0 page that has links
and images from many domains (for example: the host site, its CDN,
a comments site, and anywhere between a passel and a gaggle of
social media buttons).

There are some downsides to consider, unfortunately.  Many CDNs
really want a very short TTL on their resolves, and there are even
(I hear) some systems that use a system of ephemeral names in such
links to track users across sites (potentially bypassing cookies,
privacy policies, and anonymizing services). I suspect that eww
would like DNS pre-fetching anyway, but you might want a way to
turn it off.

I mention this now because it conflicts with the suggestion that
you can just make the entire open-network-stream (etc) process
async, and avoid the question at the level of eww.  Put another
way, you probably *can* do that, but youll be giving up a potentially
valuable browser speed-up.

I hope that helps,
~Chad

Re: Asynchronous DNS resolving

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Actually, do we really need to make the lookup Elisp-visible?  I mean,
> don't we currently just pass the DNS name to make-network-process which
> then does the DNS lookup and the TCP connect?  If so, we don't really
> need to make getaddrinfo_a visible at the Elisp level, we just
> need to make use of it in make-network-process if :nowait was specified.
>
> Am I missing something?

No, I think that's a great idea.  Making `make-network-process' totally
asynchronous would be good for general asynchronousness (that should be
a word), too, since the handshake part is synchronous today, I think.

*looks at documentation*  Oh, right.  There's :nowait, so it would make
sense to only use getaddrinfo_a if we've got a :nowait connection.

I can take a look at implementing this (if nobody else feels inspired
to), but I've got one questions to start with:

Is getaddrinfo_a available on all the platforms we care about?  Or is
there a gnulib module for compatibility or something?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Asynchronous DNS resolving

[Paul Eggert]

Lars Ingebrigtsen wrote:
> Is getaddrinfo_a available on all the platforms we care about?  Or is
> there a gnulib module for compatibility or something?

There isn't a gnulib module for it.  It should be reasonably easy to add one, 
though, that falls back on getaddrinfo if getaddrinfo_a doesn't exist.  Or 
perhaps we could just borrow the glibc implementation, though that'd be more work.

As far as I know only glibc supports it natively.

Re: Asynchronous DNS resolving

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Lars Ingebrigtsen wrote:
>> Is getaddrinfo_a available on all the platforms we care about?  Or is
>> there a gnulib module for compatibility or something?
>
> There isn't a gnulib module for it.  It should be reasonably easy to
> add one, though, that falls back on getaddrinfo if getaddrinfo_a
> doesn't exist.  Or perhaps we could just borrow the glibc
> implementation, though that'd be more work.

Writing a version that has an async callback when using getaddrinfo
would be rather difficult, I'd guess?

> As far as I know only glibc supports it natively.

Right, so this would not work on Windows and OS X, probably?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Asynchronous DNS resolving

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 13 Feb 2015 18:10:24 +1100
> 
> So perhaps doing what Firefox used to do in the 90s would be a possibly
> solution, anyway?  That is, have a teensy multi-threaded executable that
> reads commands from STDIN and outputs the replies on STDOUT.  Like
> lib-src/dns-helper.c or something.
> 
> The command format would be something like
> 
> <tag> <command> <data>
> 
> I.e.,
> 
> 45 lookup fsf.org
> 46 lookup images.fsf.org
> ...
> 
> (Tags for streaming commands and matching up the responses with the
> requests, a la IMAP.)
> 
> We'd then have something like
> 
> (with-domain-name-lookup (ip "fsf.org")
>   (message "We got %s and now we're going to connect" ip)
>   (with-url ("http://fsf.org" :ip-address ip ...)
>     (pop-to-buffer (current-buffer))))
> 
> to do a completely asynchronous web request.
> 
> Or does anybody have a better idea for solving this problem?

Can't you simulate async resolution with timers?  Or even by opening a
network-process to the DNS server and talking to it directly (which
can be done asynchronously with filters and sentinels and stuff)?

Re: Asynchronous DNS resolving

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Can't you simulate async resolution with timers?  Or even by opening a
> network-process to the DNS server and talking to it directly (which
> can be done asynchronously with filters and sentinels and stuff)?

That's what dns.el does, but it suffers from not having access to
OS-level funky stuff like possible local caching, and the general
difficulty of getting at the current DNS setup on modern machines.

Using the C-level resolution functions is the safest best, since libc is
really the authority here...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Asynchronous DNS resolving

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Can't you simulate async resolution with timers?  Or even by opening a
>> network-process to the DNS server and talking to it directly (which
>> can be done asynchronously with filters and sentinels and stuff)?
>
> That's what dns.el does, but it suffers from not having access to
> OS-level funky stuff like possible local caching, and the general
> difficulty of getting at the current DNS setup on modern machines.
>
> Using the C-level resolution functions is the safest best, since libc is
> really the authority here...

Although using getaddrinfo_a is best, a thought just occurred to me: The
main problem with dns.el is that it's not very good at getting the IP
address of the domain name server it's supposed to talk to.  (Talking
DNS itself is really easy.)

So if we added a `get-domain-name-server-addresses' C-level function, dns.el
could use that, and it'd work more reliably.  (That function would
basically just call res_init and then return the list of IP addresses.)

dns.el is currently mostly used to do reverse resolving of IP addresses
for display purposes in Gnus, so this would be a win even if we don't
use dns.el for forward resolution.  (But we could on systems without
getaddrinfo_a, if we wanted.)

Would anybody mind if I add the `get-domain-name-server-addresses'
function?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/