ai_flags in calls to getaddrinfo

ai_flags in calls to getaddrinfo

[Eli Zaretskii]

On one of the systems on which I work, getaddrinfo called with
AF_INET6 fails with EAI_NODATA, for some reason.  That causes
network-lookup-address-info to fail when passed 'ipv6' as the last
argument.

Adding the AI_V4MAPPED flag, as in the patch below, seems to solve the
problem.  So I wonder why we don't do this in general.  I'm not an
expert on DNS, so would people who know more than I do about this
please comment on whether the patch below is a good idea?

diff --git a/src/process.c b/src/process.c
index 28ab15c903..f550703c2a 100644
--- a/src/process.c
+++ b/src/process.c
@@ -4559,12 +4559,18 @@ DEFUN ("network-lookup-address-info", Fnetwork_lookup_address_info,
 
   memset (&hints, 0, sizeof hints);
   if (EQ (family, Qnil))
-    hints.ai_family = AF_UNSPEC;
+    {
+      hints.ai_family = AF_UNSPEC;
+      hints.ai_flags = AI_ALL | AI_V4MAPPED;
+    }
   else if (EQ (family, Qipv4))
     hints.ai_family = AF_INET;
 #ifdef AF_INET6
   else if (EQ (family, Qipv6))
-    hints.ai_family = AF_INET6;
+    {
+      hints.ai_family = AF_INET6;
+      hints.ai_flags = AI_V4MAPPED;
+    }
 #endif
   else
     error ("Unsupported lookup type");

Re: ai_flags in calls to getaddrinfo

[Robin Tarsiger]

Eli Zaretskii wrote:
> On one of the systems on which I work, getaddrinfo called with
> AF_INET6 fails with EAI_NODATA, for some reason.  That causes
> network-lookup-address-info to fail when passed 'ipv6' as the last
> argument.
What OS is this on? Does the host actually have an IPv6 network stack
at all? Does it have any IPv6 addresses configured, or no? What is
its resolver configuration like? What names are you looking up in
order to obtain the EAI_NODATA answer---does it happen with all
names, including for instance "he.net" or "ipv6.google.com" (assuming
this system is connected to the Internet in the first place)? What
about "localhost"?

> Adding the AI_V4MAPPED flag, as in the patch below, seems to solve the
> problem.  So I wonder why we don't do this in general.  I'm not an
> expert on DNS, so would people who know more than I do about this
> please comment on whether the patch below is a good idea?

AI_V4MAPPED is meant for the case where an application expects only
IPv6-formatted addresses in a list, but a host with only IPv4 addresses
should have those addresses included in a mangled form. It is an
address format compatibility hack. (The compatibility hack extends
to other parts of the network stack, so that attempting to connect to
such an address actually results in IPv4 packets on the wire, etc.)

> diff --git a/src/process.c b/src/process.c
> index 28ab15c903..f550703c2a 100644
> --- a/src/process.c
> +++ b/src/process.c
> @@ -4559,12 +4559,18 @@ DEFUN ("network-lookup-address-info", Fnetwork_lookup_address_info,
>  
>    memset (&hints, 0, sizeof hints);
>    if (EQ (family, Qnil))
> -    hints.ai_family = AF_UNSPEC;
> +    {
> +      hints.ai_family = AF_UNSPEC;
> +      hints.ai_flags = AI_ALL | AI_V4MAPPED;
> +    }

On a GNU system, I think this will do nothing; it's not clear to me
what AI_V4MAPPED would even mean when ai_family = AF_UNSPEC,
because AF_UNSPEC means IPv4 addresses can be returned directly.

>  #ifdef AF_INET6
>    else if (EQ (family, Qipv6))
> -    hints.ai_family = AF_INET6;
> +    {
> +      hints.ai_family = AF_INET6;
> +      hints.ai_flags = AI_V4MAPPED;
> +    }
>  #endif

And what this would do is allow returning IPv4 addresses crammed
into the IPv6 compatibility format to an elisp caller that has
specifically asked for IPv6, when looking up a host with only
IPv4 addresses. I would consider this unexpected behavior; I
would think the elisp code would pass nil and receive those
addresses in native IPv4 format if it wanted to process them.
There is not nearly as much need for the compatibility hack when
data structures are not as rigid as in C.

So I think the new code looks wrong, but per the more extended
questions above, it's possible I don't understand the problem
it's meant to fix; my experience with this part of networking
code is primarily from the perspective of GNU/Linux systems
in fairly conventional configurations.

-RTT

Re: ai_flags in calls to getaddrinfo

[Eli Zaretskii]

> From: Robin Tarsiger <rtt@dasyatidae.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 31 Dec 2020 16:40:18 -0600
> 
> Eli Zaretskii wrote:
> > On one of the systems on which I work, getaddrinfo called with
> > AF_INET6 fails with EAI_NODATA, for some reason.  That causes
> > network-lookup-address-info to fail when passed 'ipv6' as the last
> > argument.
> What OS is this on?

MS-Windows 10.

> Does the host actually have an IPv6 network stack
> at all?

Yes, I think so.

> Does it have any IPv6 addresses configured, or no? What is
> its resolver configuration like?

No idea.  If you tell me how to find out, I will try.  This system is
behind firewalls up the kazoo, so it could be something essential for
IPv6 DNS is blocked or intentionally disabled.

> What names are you looking up in
> order to obtain the EAI_NODATA answer

google.com and a few others.

> does it happen with all
> names, including for instance "he.net" or "ipv6.google.com" (assuming
> this system is connected to the Internet in the first place)?

Happens with all the names I tried, but I didn't try the above two.
Yes, there is an Internet connection.

> What about "localhost"?

Works as expected, without EAI_NODATA.

> > Adding the AI_V4MAPPED flag, as in the patch below, seems to solve the
> > problem.  So I wonder why we don't do this in general.  I'm not an
> > expert on DNS, so would people who know more than I do about this
> > please comment on whether the patch below is a good idea?
> 
> AI_V4MAPPED is meant for the case where an application expects only
> IPv6-formatted addresses in a list, but a host with only IPv4 addresses
> should have those addresses included in a mangled form. It is an
> address format compatibility hack. (The compatibility hack extends
> to other parts of the network stack, so that attempting to connect to
> such an address actually results in IPv4 packets on the wire, etc.)

Yes, I understand that much.  But it could be better than a total
failure, at least in some use cases, no?  That's why this flag exists,
right?

> > -    hints.ai_family = AF_UNSPEC;
> > +    {
> > +      hints.ai_family = AF_UNSPEC;
> > +      hints.ai_flags = AI_ALL | AI_V4MAPPED;
> > +    }
> 
> On a GNU system, I think this will do nothing; it's not clear to me
> what AI_V4MAPPED would even mean when ai_family = AF_UNSPEC,
> because AF_UNSPEC means IPv4 addresses can be returned directly.

According to the GNU/Linux getaddrinfo man page:

       If hints.ai_flags specifies the AI_V4MAPPED flag, and
       hints.ai_family was specified as AF_INET6, and no matching IPv6
       addresses could be found, then return IPv4-mapped IPv6 addresses
       in the list pointed to by res.  If both AI_V4MAPPED and AI_ALL
       are specified in hints.ai_flags, then return both IPv6 and
       IPv4-mapped IPv6 addresses in the list pointed to by res.  AI_ALL
       is ignored if AI_V4MAPPED is not also specified.

So it does sound like these flags do have effect on GNU/Linux.  On the
system in question, using the patched code and nil as FAMILY in the
network-lookup-address-info yields the expected result, including,
surprisingly, what looks like a valid IPv6 address, even though
specifying 'ipv6 for FAMILY does indeed return a compatibility-hacked
IPv4 address.

> >  #ifdef AF_INET6
> >    else if (EQ (family, Qipv6))
> > -    hints.ai_family = AF_INET6;
> > +    {
> > +      hints.ai_family = AF_INET6;
> > +      hints.ai_flags = AI_V4MAPPED;
> > +    }
> >  #endif
> 
> And what this would do is allow returning IPv4 addresses crammed
> into the IPv6 compatibility format to an elisp caller that has
> specifically asked for IPv6, when looking up a host with only
> IPv4 addresses. I would consider this unexpected behavior; I
> would think the elisp code would pass nil and receive those
> addresses in native IPv4 format if it wanted to process them.
> There is not nearly as much need for the compatibility hack when
> data structures are not as rigid as in C.
> 
> So I think the new code looks wrong, but per the more extended
> questions above, it's possible I don't understand the problem
> it's meant to fix; my experience with this part of networking
> code is primarily from the perspective of GNU/Linux systems
> in fairly conventional configurations.

If this is controversial, maybe having it as opt-in behavior,
controlled by some new optional argument, would be useful?

I found about this issue because 2 tests in test/src/process-tests.el
failed, both of them called network-lookup-address-info with second
arg 'ipv6'.  The patch I propose fixed the failures.  So at least in
this simple case the current code behaves as if IPv6 DNS doesn't work
in Emacs, whereas actually it isn't the fault of Emacs at all, and
using these flags produces a correct result for the test.  Thus, I
wonder whether these flags couldn't be useful in other use cases as
well.

Thanks.

Re: ai_flags in calls to getaddrinfo

[Robin Tarsiger]

So, as I come back to this, I think I tunnel-visioned on the AI_V4MAPPED
behavior at first since that was what you changed, but I now think that's a
red herring, and the potential problem in Emacs is the _last_ part of this:

>>> On one of the systems on which I work, getaddrinfo called with
>>> AF_INET6 fails with EAI_NODATA, for some reason.  That causes
>>> network-lookup-address-info to fail when passed 'ipv6' as the last
>>> argument.

Some non-zero return values from getaddrinfo, including EAI_NODATA,
don't actually indicate resolution failures. From glibc getaddrinfo(3),
there's also EAI_NONAME and EAI_ADDRFAMILY. I think the real point
of difficulty is that network_lookup_address_info_1 always turns these
into messageable warnings. To be clear, is that the behavior you are
seeing on the Windows 10 machine, or is there some different
failure occurring?

I notice that EAI_NODATA isn't mentioned in the Windows docs[1],
but if it's actually getting returned, I'd guess it has the same
semantics as in glibc: "got a valid result indicating no addresses
found". The exact distinction between "answer is: no data" error
codes is more nonportable, but that won't matter if you treat them
identically.

[1] https://docs.microsoft.com/en-us/windows/win32/api/ws2tcpip/nf-ws2tcpip-getaddrinfo#return-value

From an elisp standpoint, what do you want to have actually happen:

 1. If a program requests a name that doesn't exist at all?

 2. If a program requests a name that exists and has addresses of
    some families (such as IPv4 addresses), but the program has
    specifically requested another family (such as IPv6)?

 3. If a program requests a name that exists but has no addresses
    at all? This is entirely possible in the DNS; dasyatidae.com
    is even currently such a name, having only MX records and no
    address records.

Based on the docstring, I would imagine a silent nil for all three.
In particular, the docstring implies that out-of-family addresses
are _not_ returned when FAMILY is not nil. But I don't know how
elisp programs currently use network-lookup-address-info.

From another perspective, when would a program pass a non-nil
FAMILY value in the first place? If a program just wants any
address, that's already handled by passing nil for FAMILY. If
what you want is to treat it more like a preference rather than
a requirement, you'll probably have to do something more complicated.
If FAMILY expresses a requirement, then nil would be correct
if nothing matches the criteria.

(There's a potential question of whether AI_ADDRCONFIG is
relevant if you want to be DWIM-y, but let's leave that for later.)

I think that's the key here in terms of what can be done in Emacs.
More comments below on other things.

>> Does it have any IPv6 addresses configured, or no? What is
>> its resolver configuration like?
> 
> No idea.  If you tell me how to find out, I will try.  This system is
> behind firewalls up the kazoo, so it could be something essential for
> IPv6 DNS is blocked or intentionally disabled.

If you run 'ipconfig/all' from a command line (which, if you are not
familiar, you can launch by running 'cmd' from the 'Run' dialog,
which in turn can be accessed through the Windows+R shortcut), it
should display a bunch of information about active network connections,
including local IP addresses and DNS resolver addresses.

>> AI_V4MAPPED is meant for the case where an application expects only
>> IPv6-formatted addresses in a list, but a host with only IPv4 addresses
>> should have those addresses included in a mangled form. It is an
>> address format compatibility hack. (The compatibility hack extends
>> to other parts of the network stack, so that attempting to connect to
>> such an address actually results in IPv4 packets on the wire, etc.)
>
> Yes, I understand that much.  But it could be better than a total
> failure, at least in some use cases, no?  That's why this flag exists,
> right?

Not really. It's more a convenience for C programs that for whatever
reason don't want to or can't dispatch on the address family all the
time, potentially do multiple getaddrinfo calls, etc. In almost all cases,
handling v4 and v6 addresses as separate is saner, or at least I think so,
and it looks like we're in that straightforward dual-format world already
from elisp's perspective.

To put it differently: from my perspective, the main use for AI_V4MAPPED
is when there is no IPv4-handling code path at all, and all IPv4 addresses
are routed through the "IPv6" path as compatibility-mapped, which can
save some complexity and hassle elsewhere. If you code against "true"
multi-address-family, then it's just a faucet for bugs (say, trying
to filter out some IPv4 netblocks but then letting them through anyway
when they come through mapped into pseudo-IPv6 addresses). I'm not sure
all C network programmers would agree with me.

>> On a GNU system, I think this will do nothing; it's not clear to me
>> what AI_V4MAPPED would even mean when ai_family = AF_UNSPEC,
>> because AF_UNSPEC means IPv4 addresses can be returned directly.
> 
> According to the GNU/Linux getaddrinfo man page:
> 
>        If hints.ai_flags specifies the AI_V4MAPPED flag, and
>        hints.ai_family was specified as AF_INET6, and no matching IPv6
>        addresses could be found, then return IPv4-mapped IPv6 addresses
>        in the list pointed to by res.  If both AI_V4MAPPED and AI_ALL
>        are specified in hints.ai_flags, then return both IPv6 and
>        IPv4-mapped IPv6 addresses in the list pointed to by res.  AI_ALL
>        is ignored if AI_V4MAPPED is not also specified.
> 
> So it does sound like these flags do have effect on GNU/Linux.

If ai_family is AF_INET6, it does. I don't think it has an effect if
ai_family is AF_UNSPEC. I am not 100% sure, but a quick test with
resolving some names from a GNU/Linux machine (v4-only, v6-only,
dual-stack) with AF_UNSPEC didn't return anything different with
AI_V4MAPPED or AI_V4MAPPED | AI_ALL compared to none of those.

>> What about "localhost"?
> Works as expected, without EAI_NODATA.

I interpret "as expected" to mean that when you ask for localhost
in AF_INET6, you get ::1 back. Is that right?

> On the
> system in question, using the patched code and nil as FAMILY in the
> network-lookup-address-info yields the expected result, including,
> surprisingly, what looks like a valid IPv6 address, even though
> specifying 'ipv6 for FAMILY does indeed return a compatibility-hacked
> IPv4 address.

What happens if you pass nil for FAMILY but without the AI_V4MAPPED
patch, on that system? Have you observed AI_V4MAPPED producing better
results in any circumstance when you do not set FAMILY to ipv6?

I conjecture that your upstream resolver is blocking AAAA queries, and
that AF_UNSPEC is going through a slightly different path, maybe even
issuing an ANY query which is actually responded to with records from
both address families, which are then passed through because response
filtering and query filtering are different. If this is close to true,
I'm not sure there's a sane way for an application to handle
inconsistent resolver behavior of that stripe.

> If this is controversial, maybe having it as opt-in behavior,
> controlled by some new optional argument, would be useful?

If it turns out to be something Windows-specific, you could of course
conditionalize on that.

If it's network-specific, and there's an actual need to handle it,
then maybe a global "workaround for this particular bad behavior"
toggle, but that could be starting down a rabbit hole, because there's
a lot of possible network-specific bogus behaviors.

It would separately be sane to add a hint-list argument or something
to network-lookup-address-info and derive ai_flags from that; if it
is determined that AI_V4MAPPED is actually desirable sometimes, then
that would be a good way to go about it, essentially passing the
getaddrinfo behavior up one level.

But let's see what's actually going on first.

> I found about this issue because 2 tests in test/src/process-tests.el
> failed, both of them called network-lookup-address-info with second
> arg 'ipv6'.

You mean lookup-family-specification and lookup-google, I imagine?
Hmm. If I get a chance, I will see what Emacs does with the forms
from those tests on one of my Windows machines and report back.

-RTT

Re: ai_flags in calls to getaddrinfo

[Robin Tarsiger]

Robin Tarsiger wrote:
> What happens if you pass nil for FAMILY but without the AI_V4MAPPED
> patch, on that system? Have you observed AI_V4MAPPED producing better
> results in any circumstance when you do not set FAMILY to ipv6?
> 
> I conjecture that your upstream resolver is blocking AAAA queries, and
> that AF_UNSPEC is going through a slightly different path, maybe even
> issuing an ANY query which is actually responded to with records from
> both address families, which are then passed through because response
> filtering and query filtering are different. If this is close to true,
> I'm not sure there's a sane way for an application to handle
> inconsistent resolver behavior of that stripe.

Aside:

Inconsistent caching resolver behavior often varies depending on the
state of the cache. So it is useful to try things like this two or
three times in a row in different orders, if you are not sure if
your DNS resolution chain is doing something dubious.

-RTT

Re: ai_flags in calls to getaddrinfo

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Robin Tarsiger <rtt@dasyatidae.com>
> Date: Fri, 1 Jan 2021 05:40:48 -0600
> 
> Some non-zero return values from getaddrinfo, including EAI_NODATA,
> don't actually indicate resolution failures. From glibc getaddrinfo(3),
> there's also EAI_NONAME and EAI_ADDRFAMILY. I think the real point
> of difficulty is that network_lookup_address_info_1 always turns these
> into messageable warnings. To be clear, is that the behavior you are
> seeing on the Windows 10 machine, or is there some different
> failure occurring?

That's exactly what I see: the addresses return as nil and a warning
message is displayed.

> I notice that EAI_NODATA isn't mentioned in the Windows docs[1],

Yes, because Windows has its own codes for socket-related errors.
Look for WSANO_DATA.  I said "EAI_NODATA" to avoid using Windows
specific terminology that people might be unfamiliar with.

> but if it's actually getting returned, I'd guess it has the same
> semantics as in glibc: "got a valid result indicating no addresses
> found".

The Windows text is "The requested name is valid, but no data of the
requested type was found."

> From an elisp standpoint, what do you want to have actually happen:
> 
>  1. If a program requests a name that doesn't exist at all?

A failure, of course.

>  2. If a program requests a name that exists and has addresses of
>     some families (such as IPv4 addresses), but the program has
>     specifically requested another family (such as IPv6)?

The address returned when AI_V4MAPPED is used is a valid IPv6 address,
isn't it?  That is, it can be used in IPv6 connections, and will allow
such connections to work as intended, right?  Then I'd expect to have
them if a "real" IPv6 address is not available for some reason.

>  3. If a program requests a name that exists but has no addresses
>     at all? This is entirely possible in the DNS; dasyatidae.com
>     is even currently such a name, having only MX records and no
>     address records.

In that case, wouldn't using AI_V4MAPPED produce an error and/or an
empty list of addresses anyway?  If so, that is the result I'd expect.

> Based on the docstring, I would imagine a silent nil for all three.
> In particular, the docstring implies that out-of-family addresses
> are _not_ returned when FAMILY is not nil. But I don't know how
> elisp programs currently use network-lookup-address-info.

If this is a documentation issue, we can easily make the doc string
more detailed (until yesterday it didn't even mention the warning
message that is displayed in the case of a problem).  the current text
is not sacred, nor does it define a spec or the contract for that API
in any shape or form.

I don't have enough experience with using the results of the address
resolution in Emacs, which is why I asked the questions.  But it seems
to me that if using these additional flags produces usable addresses,
then using these flags is a good idea, because it will allow
connections using the resolution results in more cases.

> >From another perspective, when would a program pass a non-nil
> FAMILY value in the first place? If a program just wants any
> address, that's already handled by passing nil for FAMILY.

Again, isn't the address returns when AF_INET6 and AI_V4MAPPED are
used a valid IPv6 address that can be used in IPv6 connections?  If
so, then the program which specified FAMILY does get what it asked
for.

> >> Does it have any IPv6 addresses configured, or no? What is
> >> its resolver configuration like?
> > 
> > No idea.  If you tell me how to find out, I will try.  This system is
> > behind firewalls up the kazoo, so it could be something essential for
> > IPv6 DNS is blocked or intentionally disabled.
> 
> If you run 'ipconfig/all' from a command line (which, if you are not
> familiar, you can launch by running 'cmd' from the 'Run' dialog,
> which in turn can be accessed through the Windows+R shortcut), it
> should display a bunch of information about active network connections,
> including local IP addresses and DNS resolver addresses.

Will do, thanks.

> To put it differently: from my perspective, the main use for AI_V4MAPPED
> is when there is no IPv4-handling code path at all, and all IPv4 addresses
> are routed through the "IPv6" path as compatibility-mapped, which can
> save some complexity and hassle elsewhere.

Isn't that the common use case?

> If you code against "true" multi-address-family, then it's just a
> faucet for bugs (say, trying to filter out some IPv4 netblocks but
> then letting them through anyway when they come through mapped into
> pseudo-IPv6 addresses).

Lisp programs that want to be so strict will have no problem
identifying "fake" IPv6 addresses and filtering them out, I think.

> >        If hints.ai_flags specifies the AI_V4MAPPED flag, and
> >        hints.ai_family was specified as AF_INET6, and no matching IPv6
> >        addresses could be found, then return IPv4-mapped IPv6 addresses
> >        in the list pointed to by res.  If both AI_V4MAPPED and AI_ALL
> >        are specified in hints.ai_flags, then return both IPv6 and
> >        IPv4-mapped IPv6 addresses in the list pointed to by res.  AI_ALL
> >        is ignored if AI_V4MAPPED is not also specified.
> > 
> > So it does sound like these flags do have effect on GNU/Linux.
> 
> If ai_family is AF_INET6, it does. I don't think it has an effect if
> ai_family is AF_UNSPEC. I am not 100% sure, but a quick test with
> resolving some names from a GNU/Linux machine (v4-only, v6-only,
> dual-stack) with AF_UNSPEC didn't return anything different with
> AI_V4MAPPED or AI_V4MAPPED | AI_ALL compared to none of those.

It did change on Windows.  I don't have a GNU/Linux machine on that
network to try that.

> >> What about "localhost"?
> > Works as expected, without EAI_NODATA.
> 
> I interpret "as expected" to mean that when you ask for localhost
> in AF_INET6, you get ::1 back. Is that right?

Yes.

> > On the
> > system in question, using the patched code and nil as FAMILY in the
> > network-lookup-address-info yields the expected result, including,
> > surprisingly, what looks like a valid IPv6 address, even though
> > specifying 'ipv6 for FAMILY does indeed return a compatibility-hacked
> > IPv4 address.
> 
> What happens if you pass nil for FAMILY but without the AI_V4MAPPED
> patch, on that system?

I get only an IPv4 address.

> Have you observed AI_V4MAPPED producing better
> results in any circumstance when you do not set FAMILY to ipv6?

Yes, as reported in my original message: I get both IPv4 and IPv6
addresses if I use AI_ALL | AI_V4MAPPED.

> > I found about this issue because 2 tests in test/src/process-tests.el
> > failed, both of them called network-lookup-address-info with second
> > arg 'ipv6'.
> 
> You mean lookup-family-specification and lookup-google, I imagine?

Yes.

> Hmm. If I get a chance, I will see what Emacs does with the forms
> from those tests on one of my Windows machines and report back.

Thanks.

Re: ai_flags in calls to getaddrinfo

[Robin Tarsiger]

Eli Zaretskii wrote:
>> I notice that EAI_NODATA isn't mentioned in the Windows docs[1],
> 
> Yes, because Windows has its own codes for socket-related errors.
> Look for WSANO_DATA.  I said "EAI_NODATA" to avoid using Windows
> specific terminology that people might be unfamiliar with.

Ah, I see. In any case it does seem like the same thing.

>>  2. If a program requests a name that exists and has addresses of
>>     some families (such as IPv4 addresses), but the program has
>>     specifically requested another family (such as IPv6)?
> 
> The address returned when AI_V4MAPPED is used is a valid IPv6 address,
> isn't it?  That is, it can be used in IPv6 connections, and will allow
> such connections to work as intended, right?

Basically no; only yes under a very constrained perspective. The socket
mechanism on an OS that supports this compatibility mechanism will do
translation behind the scenes, but that's the extent of it. You should
never see these on the wire, and they are not routable over the IPv6
Internet. If the local host is not running an IPv4 stack, or does not
have an appropriate IPv4 address, they cannot be communicated with from
that host. Any actual IPv6 connectivity on the local host is irrelevant.
They are, in fact, IPv4 addresses in disguise. That's _all_ they are.

Is there a case where an application would ask for the IPv6 family
specifically _without_ caring about the above? Because if it mainly
cares about being able to pass the address into connect(), then it
should just be family-agnostic with a nil FAMILY, and potentially use
locally-available connectivity hints if some mechanism for them is
there, no? (And make-network-process already internalizes the name
lookup.) And if it wants addresses of either family but to specifically
prefer IPv6 addresses if they exist (which is basically what AI_V4MAPPED
does), it can rearrange the list.

>> To put it differently: from my perspective, the main use for AI_V4MAPPED
>> is when there is no IPv4-handling code path at all, and all IPv4 addresses
>> are routed through the "IPv6" path as compatibility-mapped, which can
>> save some complexity and hassle elsewhere.
> > Isn't that the common use case?

Where would you expect to see an IPv6-only code path in an elisp program?
In C, this happens when porting programs out of the IPv4 world, largely
due to type rigidity in C, and then lingers. But in elisp, treating addresses
from families not recognized by the application as opaque is easy, because
of the boxing and dynamic typing; you just don't introspect into them in
the first place. And if the application _does_ want to introspect into them,
then it will want to handle IPv4 addresses by IPv4 standards, at which point
having them represented in native form is better.

(See RFC4038 section 4.2 for a more authoritative description, in the
context of the surrounding section 4 about transition mechanisms.)

Having _both_ v4-mapped addresses and true multi-family address
representation in the same place is the worst of both worlds. Now
you have to handle both families _and_ you have to handle two different
representations for the same IPv4 address. This may not always be
possible to avoid, but we can avoid creating the issue where it didn't
previously exist.

For server sockets it is likely a different trade-off, though even then
I would say that if the Emacs implementation chooses to use a single
"IPv6"-with-dual-stack-underneath socket for C-side simplicity (which
would be reasonable---this in fact would fall into the exact "porting
C code out of the IPv4 world" case), it should then do the detection of
actually-IPv4 addresses and convert them as IPv4 before they hit
the elisp layer. (I haven't checked whether it does this.)

> Lisp programs that want to be so strict will have no problem
> identifying "fake" IPv6 addresses and filtering them out, I think.

This is basically true, but per above it is a needless cost.

>>> On the
>>> system in question, using the patched code and nil as FAMILY in the
>>> network-lookup-address-info yields the expected result, including,
>>> surprisingly, what looks like a valid IPv6 address, even though
>>> specifying 'ipv6 for FAMILY does indeed return a compatibility-hacked
>>> IPv4 address.
>>
>> What happens if you pass nil for FAMILY but without the AI_V4MAPPED
>> patch, on that system?
> 
> I get only an IPv4 address.

Okay, that is interesting. I wonder if the AI_ALL flag on Windows is
overriding some equivalent of AI_ADDRCONFIG, now. I'll see if I can
check this from the C level on my machines too.

Incidentally, distinguishing NXDOMAIN ("name does not exist at all")
from "nothing in any recognized address family" may not be (portably,
reliably) doable with getaddrinfo AFAIK (though the distinction this
would have to the application from the perspective of a "look up
addresses" API is unclear to start with), and distinguishing
"nothing in this address family, maybe-or-maybe-not anything in
others" when an address family _is_ provided may also not be. So
having a warning come from network-lookup-address-info in case of
all successful lookups of nothing is a bit fraught to start with,
but since it is the existing behavior, I think it may be necessary
to look at how existing elisp code uses this function to figure
out what's expected.

The lookup-google test is sort of asking to fail on locked-down
resolver configurations in its current form, too... and I wonder
if lookup-family-specification may want to be separated into
"works for known" and "doesn't work for unknown" and/or use
localhost instead...

-RTT

Re: ai_flags in calls to getaddrinfo

[Eli Zaretskii]

> From: Robin Tarsiger <rtt@dasyatidae.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 31 Dec 2020 16:40:18 -0600
> 
> does it happen with all names, including for instance "he.net" or
> "ipv6.google.com"

It does:

  With AI_ALL and AI_V4MAPPED flags:

    (network-lookup-address-info "he.net")
      => ([216 218 236 2 0] [8193 1136 0 1283 0 0 0 2 0])

    (network-lookup-address-info "he.net" 'ipv6)
      => ([0 0 0 0 0 65535 55514 60418 0])

    (network-lookup-address-info "ipv6.google.com")
      => nil: WSANO_DATA error

    (network-lookup-address-info "ipv6.google.com" 'ipv6)
      => nil: WSAHOST_NOT_FOUND error

  Without the AI_* flags:

    (network-lookup-address-info "he.net")
      => ([216 218 236 2 0])

    (network-lookup-address-info "he.net" 'ipv6)
      => nil: WSANO_DATA error

    (network-lookup-address-info "ipv6.google.com")
      => nil: WSAHOST_NOT_FOUND error

    (network-lookup-ddress-info "ipv6.google.com" 'ipv6)
      => nil: WSANO_DATA error

Note the difference in he.net resolution with and without the 'ipv6
argument, and also the behavior of ipv6.google.com with the ipv6
argument.  Interesting, no?

>  What about "localhost"?

As I said, localhost works the same with and without the AI_* flags.
I get:

  (network-lookup-address-info "localhost")
    => ([0 0 0 0 0 0 0 1 0] [127 0 0 1 0])

  (network-lookup-address-info "localhost" 'ipv6)
    => ([0 0 0 0 0 0 0 1 0])

> >> Does it have any IPv6 addresses configured, or no? What is
> >> its resolver configuration like?
> > 
> > No idea.  If you tell me how to find out, I will try.  This system is
> > behind firewalls up the kazoo, so it could be something essential for
> > IPv6 DNS is blocked or intentionally disabled.
> 
> If you run 'ipconfig/all' from a command line (which, if you are not
> familiar, you can launch by running 'cmd' from the 'Run' dialog,
> which in turn can be accessed through the Windows+R shortcut), it
> should display a bunch of information about active network connections,
> including local IP addresses and DNS resolver addresses.

Here's the result, with some fields redacted:

  Windows IP Configuration

     Host Name . . . . . . . . . . . . : (redacted)
     Primary Dns Suffix  . . . . . . . : (redacted)
     Node Type . . . . . . . . . . . . : Hybrid
     IP Routing Enabled. . . . . . . . : No
     WINS Proxy Enabled. . . . . . . . : No
     DNS Suffix Search List. . . . . . : (redacted)

  Ethernet adapter Ethernet:

     Connection-specific DNS Suffix  . : (redacted)
     Description . . . . . . . . . . . : Realtek PCIe GbE Family Controller
     Physical Address. . . . . . . . . : 6C-4B-90-A3-36-AB
     DHCP Enabled. . . . . . . . . . . : Yes
     Autoconfiguration Enabled . . . . : Yes
     Link-local IPv6 Address . . . . . : fe80::e855:89df:21cc:1554%7(Preferred)
     IPv4 Address. . . . . . . . . . . : 10.99.32.1(Preferred)
     Subnet Mask . . . . . . . . . . . : 255.255.252.0
     Lease Obtained. . . . . . . . . . : Sun 27 Dec 2020 09:20:18
     Lease Expires . . . . . . . . . . : Wed 06 Jan 2021 21:20:19
     Default Gateway . . . . . . . . . : 10.99.35.254
     DHCP Server . . . . . . . . . . . : 10.1.20.36
     DHCPv6 IAID . . . . . . . . . . . : 107760528
     DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-25-C0-F1-1B-6C-4B-90-A3-36-AB

Thanks.

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robin Tarsiger]

I'm sorry I haven't been able to get back to this the way I'd hoped;
too many things to deal with last week, and the machines I was planning
to use are having their own issues which I want to make sure are fixed
first...

Eli Zaretskii wrote:
>   With AI_ALL and AI_V4MAPPED flags:
> 
>     (network-lookup-address-info "he.net")
>       => ([216 218 236 2 0] [8193 1136 0 1283 0 0 0 2 0])
> 
>     (network-lookup-address-info "he.net" 'ipv6)
>       => ([0 0 0 0 0 65535 55514 60418 0])
> 
>     (network-lookup-address-info "ipv6.google.com")
>       => nil: WSANO_DATA error
> 
>     (network-lookup-address-info "ipv6.google.com" 'ipv6)
>       => nil: WSAHOST_NOT_FOUND error
> 
>   Without the AI_* flags:
> 
>     (network-lookup-address-info "he.net")
>       => ([216 218 236 2 0])
> 
>     (network-lookup-address-info "he.net" 'ipv6)
>       => nil: WSANO_DATA error
> 
>     (network-lookup-address-info "ipv6.google.com")
>       => nil: WSAHOST_NOT_FOUND error
> 
>     (network-lookup-ddress-info "ipv6.google.com" 'ipv6)
>       => nil: WSANO_DATA error
> 
> Note the difference in he.net resolution with and without the 'ipv6
> argument, and also the behavior of ipv6.google.com with the ipv6
> argument.  Interesting, no?

Definitely. The behavior of "he.net" with 'ipv6 in the former group
is something I think of as to be avoided, per elsethread, but this
shows the latter truncation for nil more concretely, of course...
and the error-swapping behavior with "ipv6.google.com" is bizarre
and hard to reconcile with earlier observations, especially the
nil-family case. I wonder whether that changes with 'ipv4?

Your ipconfig result showed no "real" IPv6 connectivity, which is
about what I'd have expected but at least sweeps some tangential
possibilities away. Unfortunately there isn't a clear way to get
an idea of what upstream resolvers might be doing from the host
configuration. It occurs to me that there may be something
nsswitch-like going on as well, but I don't know what that would
be on Windows OTTOMH and I haven't had time to research it.

Thanks for showing the details; it helps avoid misunderstandings.

In the meantime, if others on the list would be willing to test these
on Windows and/or with unusual resolver configurations, especially
if you can get or point to a packet dump of how a Windows system
converts different getaddrinfo parameters to DNS requests on the
wire (and the responses, if comfortable with that), that would also
be useful. In particular, if this turns out to be specific to
Windows but not specific to that one host's network, then it may
make sense to set some flags conditioned on OS. I am also curious
whether AI_ALL _without_ AI_V4MAPPED does anything on Windows, per
what I mentioned upthread.

-RTT "Why is wanting things to be correct so messy sometimes"

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Robin Tarsiger <rtt@dasyatidae.com> writes:

> In the meantime, if others on the list would be willing to test these
> on Windows and/or with unusual resolver configurations, especially
> if you can get or point to a packet dump of how a Windows system
> converts different getaddrinfo parameters to DNS requests on the
> wire (and the responses, if comfortable with that), that would also
> be useful. In particular, if this turns out to be specific to
> Windows but not specific to that one host's network, then it may
> make sense to set some flags conditioned on OS. I am also curious
> whether AI_ALL _without_ AI_V4MAPPED does anything on Windows, per
> what I mentioned upthread.
>

Iʼve only been tangentially following this discussion, but my first
reaction is that V4 mapped V6 addresses are best avoided. Either go
for proper IPv6 connectivity or avoid IPv6 entirely..

> -RTT "Why is wanting things to be correct so messy sometimes"

Such is life (and not just with technology).

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Eli Zaretskii]

> From: Robin Tarsiger <rtt@dasyatidae.com>
> Cc: emacs-devel@gnu.org
> Date: Mon, 11 Jan 2021 04:47:11 -0600
> 
> > Note the difference in he.net resolution with and without the 'ipv6
> > argument, and also the behavior of ipv6.google.com with the ipv6
> > argument.  Interesting, no?
> 
> Definitely. The behavior of "he.net" with 'ipv6 in the former group
> is something I think of as to be avoided, per elsethread, but this
> shows the latter truncation for nil more concretely, of course...
> and the error-swapping behavior with "ipv6.google.com" is bizarre
> and hard to reconcile with earlier observations, especially the
> nil-family case. I wonder whether that changes with 'ipv4?

No, it behaves as expected.

> Your ipconfig result showed no "real" IPv6 connectivity

How did you see this?  Would it be possible to deduce that from the
output of, say, network-interface-list?

I'm asking because I'm still bothered by failures in those two tests,
and would like to avoid that.  I think it's bad if a test fails when
there's nothing wrong with Emacs; we should try to skip such tests
when we know their results cannot be trusted.

Thanks.

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robin Tarsiger <rtt@dasyatidae.com>
>> Cc: emacs-devel@gnu.org
>> Date: Mon, 11 Jan 2021 04:47:11 -0600
>> 
>> > Note the difference in he.net resolution with and without the 'ipv6
>> > argument, and also the behavior of ipv6.google.com with the ipv6
>> > argument.  Interesting, no?
>> 
>> Definitely. The behavior of "he.net" with 'ipv6 in the former group
>> is something I think of as to be avoided, per elsethread, but this
>> shows the latter truncation for nil more concretely, of course...
>> and the error-swapping behavior with "ipv6.google.com" is bizarre
>> and hard to reconcile with earlier observations, especially the
>> nil-family case. I wonder whether that changes with 'ipv4?
>
> No, it behaves as expected.
>
>> Your ipconfig result showed no "real" IPv6 connectivity
>
> How did you see this?  Would it be possible to deduce that from the
> output of, say, network-interface-list?
>

You only have fe80 and fec0 addresses listed, which are link-local and
site-local prefixes (and the latter has been deprecated so long itʼs
old enough to drink beer in Belgium)

The only assigned globally routable IPv6 prefix is currently 2000::/3,
so we could check for that via network-interface-list.

> I'm asking because I'm still bothered by failures in those two tests,
> and would like to avoid that.  I think it's bad if a test fails when
> there's nothing wrong with Emacs; we should try to skip such tests
> when we know their results cannot be trusted.

The results can be trusted. However, the assumption of the tests is
that DNS lookups on AAAA records will work, and return results for
google.com, which is not the case for you. If you prefer we can change
the tests to check for "no error" rather than 'returns a result' (is
there an ERT 'should-not-error' clause)?

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Robin Tarsiger <rtt@dasyatidae.com>,  emacs-devel@gnu.org
> Date: Mon, 11 Jan 2021 16:47:29 +0100
> 
> >> Your ipconfig result showed no "real" IPv6 connectivity
> >
> > How did you see this?  Would it be possible to deduce that from the
> > output of, say, network-interface-list?
> >
> 
> You only have fe80 and fec0 addresses listed, which are link-local and
> site-local prefixes (and the latter has been deprecated so long itʼs
> old enough to drink beer in Belgium)
> 
> The only assigned globally routable IPv6 prefix is currently 2000::/3,
> so we could check for that via network-interface-list.

Is it possible to check this via network-interface-list, and avoid the
test with IPv6 if that is bound to fail?

> The results can be trusted. However, the assumption of the tests is
> that DNS lookups on AAAA records will work, and return results for
> google.com, which is not the case for you. If you prefer we can change
> the tests to check for "no error" rather than 'returns a result' (is
> there an ERT 'should-not-error' clause)?

Yes, that'd be good as well.  In short, anything that will avid false
negatives in this test, especially since the process.c area has some
serious development lately.

Thanks.

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robin Tarsiger]

Eli Zaretskii wrote:
> Is it possible to check this via network-interface-list, and avoid the
> test with IPv6 if that is bound to fail?

The relevant part of the tests should, generally, be about DNS _lookups_
of IPv6 addresses. This is not "truly" related to whether your local host
has IPv6 connectivity at the network layer. I have no IPv6 routing at
home currently, but getaddrinfo will still give me all the IPv6 results
I want; I just won't be able to connect there. I expect that case to be
common. The opposite case, where I could have IPv6 networking available
but have restrictive upstream DNS resolvers blocking queries for or results
containing IPv6 addresses, is also theoretically possible, though I would
expect it to be rare.

So basically you'd be testing the wrong thing. You could try to err on the
side of caution, but then you'd be excluding the test in a lot of cases
that really should work.

If your upstream resolvers default to mangling or blocking AAAA queries, as
my current best hypothesis suggests, this is not something an application
can reasonably take responsibility for fixing up, and we'd want to go the
route of "skip network-environment-dependent tests when on hosts that are
in strange/broken network environments". I don't know whether there's an
existing toggle for that; if not, then that would be the thing to put in
(at whatever granularity seems appropriate).

There is no _reliable_ way I know of to determine this programmatically,
certainly not without sending some probe queries with a high level of
specific control over them, and I expect that would be a rather hairy
thing to do just for a few tests, and not reasonably maintainable.

Another perspective on above is that when you mention "behind firewalls up
the kazoo", what my intuition imagines is something like: corporate DNS
filtering which only allows specific request types and either was never
updated to think of AAAA queries as valid, or is operated in a way assuming
that users will never have a legitimate need for them (because they "won't
be able to use" the results), or does this as a workaround for some broken
behavior elsewhere (such as some other device used in the organization
which will prefer an IPv6 address response even when it's not reachable,
and fail to fall back to IPv4?) or... any number of things like that.

The main thing that hypothesis doesn't explain is the discrepancy in the
any-family results, but that could easily be by accident depending on how
the specifics of the filtering interact with the specifics of the pattern
of queries Windows generates under those circumstances, which is why I was
originally thinking of finding out the latter; or it could be some local
filtering that Windows does depending on local connectivity for similar
reasons to the above, which is why I wanted to see whether those flags
made any difference for Windows machines on more "conventional" networks.

-RTT

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Robin Tarsiger <rtt@dasyatidae.com> writes:

> Eli Zaretskii wrote:
>> Is it possible to check this via network-interface-list, and avoid the
>> test with IPv6 if that is bound to fail?
>
> The relevant part of the tests should, generally, be about DNS _lookups_
> of IPv6 addresses. This is not "truly" related to whether your local host
> has IPv6 connectivity at the network layer. I have no IPv6 routing at
> home currently, but getaddrinfo will still give me all the IPv6 results
> I want; I just won't be able to connect there. I expect that case to be
> common. The opposite case, where I could have IPv6 networking available
> but have restrictive upstream DNS resolvers blocking queries for or results
> containing IPv6 addresses, is also theoretically possible, though I would
> expect it to be rare.
>

Yes. DNS lookups is the bit that generally works without intervention.

> So basically you'd be testing the wrong thing. You could try to err on the
> side of caution, but then you'd be excluding the test in a lot of cases
> that really should work.
>
> If your upstream resolvers default to mangling or blocking AAAA queries, as
> my current best hypothesis suggests, this is not something an application
> can reasonably take responsibility for fixing up, and we'd want to go the
> route of "skip network-environment-dependent tests when on hosts that are
> in strange/broken network environments". I don't know whether there's an
> existing toggle for that; if not, then that would be the thing to put in
> (at whatever granularity seems appropriate).
>
> There is no _reliable_ way I know of to determine this programmatically,
> certainly not without sending some probe queries with a high level of
> specific control over them, and I expect that would be a rather hairy
> thing to do just for a few tests, and not reasonably maintainable.
>

(require 'dns)
(skip-unless (dns-query "google.com" 'AAAA))

and then do the network-lookup-address-info tests (although on Windows
this depends on DNS over TCP working, and either /etc/resolv.conf or a
working nslookup). Eli?

> Another perspective on above is that when you mention "behind firewalls up
> the kazoo", what my intuition imagines is something like: corporate DNS
> filtering which only allows specific request types and either was never
> updated to think of AAAA queries as valid, or is operated in a way assuming
> that users will never have a legitimate need for them (because they "won't
> be able to use" the results), or does this as a workaround for some broken
> behavior elsewhere (such as some other device used in the organization
> which will prefer an IPv6 address response even when it's not reachable,
> and fail to fall back to IPv4?) or... any number of things like that.
>

Iʼd love to be able to persuade Eli's network admins to fix such
things, but I only attempt things that have a reasonable chance of
success (and such things are sometimes dependent on vendors rather
than admins).

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robin Tarsiger]

Robert Pluim wrote:
>> There is no _reliable_ way I know of to determine this programmatically,
>> certainly not without sending some probe queries with a high level of
>> specific control over them, and I expect that would be a rather hairy
>> thing to do just for a few tests, and not reasonably maintainable.
> 
> (require 'dns)
> (skip-unless (dns-query "google.com" 'AAAA))

Oh, wow, there's _already_ a dns.el in core? Huh. Well, that obviates
my primary concerns about doing direct DNS probes, provided it works
and is a reasonable thing to load for the test suite otherwise...

> and then do the network-lookup-address-info tests (although on Windows
> this depends on DNS over TCP working,

Ergh. I just checked with `dig +tcp` against my crappy ISP CPE and it
rejects the connection, which seems like anecdata toward that failing
in a lot of configurations.

Do you happen to know OTTOYH _why_ datagram sockets aren't supported
in Emacs on Windows to start with... ?

> and either /etc/resolv.conf or a
> working nslookup).
The latter should at least be stock on Windows AFAIK. I'd forgotten that
it shows the list of resolver addresses too.

-RTT

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 1727 bytes --]

Robin Tarsiger <rtt@dasyatidae.com> writes:

> Robert Pluim wrote:
>>> There is no _reliable_ way I know of to determine this programmatically,
>>> certainly not without sending some probe queries with a high level of
>>> specific control over them, and I expect that would be a rather hairy
>>> thing to do just for a few tests, and not reasonably maintainable.
>> 
>> (require 'dns)
>> (skip-unless (dns-query "google.com" 'AAAA))
>
> Oh, wow, there's _already_ a dns.el in core? Huh. Well, that obviates
> my primary concerns about doing direct DNS probes, provided it works
> and is a reasonable thing to load for the test suite otherwise...
>
>> and then do the network-lookup-address-info tests (although on Windows
>> this depends on DNS over TCP working,
>
> Ergh. I just checked with `dig +tcp` against my crappy ISP CPE and it
> rejects the connection, which seems like anecdata toward that failing
> in a lot of configurations.
>

Yep.

> Do you happen to know OTTOYH _why_ datagram sockets aren't supported
> in Emacs on Windows to start with... ?
>

Because the select implementation on windows does a one-byte readahead
to see if data is available, which breaks UDP. I had a patch at one
point to fix this, but I remember Eli not being very enthusiastic
about it. Iʼve attached what I think is the right version below (my
windows box died, so I can't be sure)

>> and either /etc/resolv.conf or a
>> working nslookup).
> The latter should at least be stock on Windows AFAIK. I'd forgotten that
> it shows the list of resolver addresses too.

Itʼs my least preferred option, it depends on running an external
binary and parsing the output, which is fragile.

Robert


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: sys_select_improvement.patch --]
[-- Type: text/x-diff, Size: 1745 bytes --]

diff --git i/src/w32.c w/src/w32.c
index 698e10e234..d14abc2f8e 100644
--- i/src/w32.c
+++ w/src/w32.c
@@ -8798,6 +8798,37 @@ _sys_wait_accept (int fd)
   return cp->status;
 }
 
+int
+_sys_wait_readable (int fd)
+{
+  HANDLE hEv;
+  child_process * cp;
+  int rc;
+
+  if (fd < 0 || fd >= MAXDESC)
+    return STATUS_READ_ERROR;
+
+  cp = fd_info[fd].cp;
+
+  if (cp == NULL || cp->fd != fd || cp->status != STATUS_READ_READY)
+    return STATUS_READ_ERROR;
+
+  cp->status = STATUS_READ_FAILED;
+
+  hEv = pfn_WSACreateEvent ();
+  rc = pfn_WSAEventSelect (SOCK_HANDLE (fd), hEv, FD_READ);
+  if (rc != SOCKET_ERROR)
+    {
+	rc = WaitForSingleObject (hEv, INFINITE);
+        pfn_WSAEventSelect (SOCK_HANDLE (fd), NULL, 0);
+        if (rc == WAIT_OBJECT_0)
+          cp->status = STATUS_READ_SUCCEEDED;
+    }
+  pfn_WSACloseEvent (hEv);
+
+  return cp->status;
+}
+
 int
 _sys_wait_connect (int fd)
 {
diff --git i/src/w32.h w/src/w32.h
index cf1dadf64c..4a6b7c4b53 100644
--- i/src/w32.h
+++ w/src/w32.h
@@ -175,6 +175,7 @@ #define FILE_SERIAL             0x0800
 
 extern int _sys_read_ahead (int fd);
 extern int _sys_wait_accept (int fd);
+extern int _sys_wait_readable (int fd);
 extern int _sys_wait_connect (int fd);
 
 extern HMODULE w32_delayed_load (Lisp_Object);
diff --git i/src/w32proc.c w/src/w32proc.c
index de33726905..1e109669f7 100644
--- i/src/w32proc.c
+++ w/src/w32proc.c
@@ -1225,7 +1225,7 @@ reader_thread (void *arg)
       else if (cp->fd >= 0 && (fd_info[cp->fd].flags & FILE_LISTEN) != 0)
 	rc = _sys_wait_accept (cp->fd);
       else
-	rc = _sys_read_ahead (cp->fd);
+	rc = _sys_wait_readable (cp->fd);
 
       /* Don't bother waiting for the event if we already have been
 	 told to exit by delete_child.  */

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Stefan Monnier]

> Because the select implementation on windows does a one-byte readahead
> to see if data is available, which breaks UDP. I had a patch at one
> point to fix this, but I remember Eli not being very enthusiastic
> about it. Iʼve attached what I think is the right version below (my
> windows box died, so I can't be sure)

How 'bout installing it but make it conditional on some config var?
And maybe set that config var if/when a UDP socket is requested?


        Stefan

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Because the select implementation on windows does a one-byte readahead
>> to see if data is available, which breaks UDP. I had a patch at one
>> point to fix this, but I remember Eli not being very enthusiastic
>> about it. Iʼve attached what I think is the right version below (my
>> windows box died, so I can't be sure)
>
> How 'bout installing it but make it conditional on some config var?
> And maybe set that config var if/when a UDP socket is requested?

I guess thatʼs possible. Iʼve now found the actual patch. If Eli
thinks itʼs worth persuing, I can make it conditional, unless someone
wants to spare me the trouble of setting up a working Windows dev
environment :-).

As it stands you need to arrange for WORKING_SELECT_EMULATION to be
defined.

diff --git i/nt/inc/ms-w32.h w/nt/inc/ms-w32.h
index 1cce2c3062..ea6ba38dea 100644
--- i/nt/inc/ms-w32.h
+++ w/nt/inc/ms-w32.h
@@ -63,10 +63,11 @@ #define _CALLBACK_ __cdecl
    Look in <sys/time.h> for a timeval structure.  */
 #define HAVE_TIMEVAL 1
 
+#ifndef WORKING_SELECT_EMULATION
 /* Our select emulation does 1-byte read-ahead waiting for received
    packets, so datagrams are broken.  */
 #define BROKEN_DATAGRAM_SOCKETS 1
-
+#endif
 #define MAIL_USE_SYSTEM_LOCK 1
 
 /* Define to 1 if GCC-style __attribute__ ((__aligned__ (expr))) works. */
diff --git i/src/w32.c w/src/w32.c
index 698e10e234..c0457ff00f 100644
--- i/src/w32.c
+++ w/src/w32.c
@@ -8798,6 +8798,45 @@ _sys_wait_accept (int fd)
   return cp->status;
 }
 
+#ifdef WORKING_SELECT_EMULATION
+int
+_sys_wait_readable (int fd)
+{
+  HANDLE hEv;
+  child_process * cp;
+  int rc;
+
+  if (fd < 0 || fd >= MAXDESC)
+    return STATUS_READ_ERROR;
+
+  cp = fd_info[fd].cp;
+
+  if (cp == NULL || cp->fd != fd || cp->status != STATUS_READ_READY)
+    return STATUS_READ_ERROR;
+
+  cp->status = STATUS_READ_FAILED;
+
+  hEv = pfn_WSACreateEvent ();
+  rc = pfn_WSAEventSelect (SOCK_HANDLE (fd), hEv, FD_READ);
+  if (rc != SOCKET_ERROR)
+    {
+      do
+        {
+          rc = WaitForSingleObject (hEv, 500);
+          Sleep (5);
+        } while (rc == WAIT_TIMEOUT
+                 && cp->status != STATUS_READ_ERROR
+                 && cp->char_avail);
+      pfn_WSAEventSelect (SOCK_HANDLE (fd), NULL, 0);
+      if (rc == WAIT_OBJECT_0)
+        cp->status = STATUS_READ_SUCCEEDED;
+    }
+  pfn_WSACloseEvent (hEv);
+
+  return cp->status;
+}
+#endif
+
 int
 _sys_wait_connect (int fd)
 {
@@ -8923,10 +8962,16 @@ sys_read (int fd, char * buffer, unsigned int count)
 	      return -1;
 
 	    case STATUS_READ_SUCCEEDED:
-	      /* consume read-ahead char */
-	      *buffer++ = cp->chr;
-	      count--;
-	      nchars++;
+#ifdef WORKING_SELECT_EMULATION
+              /* select on sockets no longer requires a 1-byte read.  */
+              if (fd_info[fd].flags & FILE_SOCKET == 0)
+#endif
+		{
+		  /* consume read-ahead char */
+		  *buffer++ = cp->chr;
+		  count--;
+		  nchars++;
+		}
 	      cp->status = STATUS_READ_ACKNOWLEDGED;
 	      ResetEvent (cp->char_avail);
 
diff --git i/src/w32.h w/src/w32.h
index cf1dadf64c..cabe39fb6d 100644
--- i/src/w32.h
+++ w/src/w32.h
@@ -175,6 +175,9 @@ #define FILE_SERIAL             0x0800
 
 extern int _sys_read_ahead (int fd);
 extern int _sys_wait_accept (int fd);
+#ifdef WORKING_SELECT_EMULATION
+extern int _sys_wait_readable (int fd);
+#endif
 extern int _sys_wait_connect (int fd);
 
 extern HMODULE w32_delayed_load (Lisp_Object);
diff --git i/src/w32proc.c w/src/w32proc.c
index de33726905..376e49d13d 100644
--- i/src/w32proc.c
+++ w/src/w32proc.c
@@ -1225,7 +1225,12 @@ reader_thread (void *arg)
       else if (cp->fd >= 0 && (fd_info[cp->fd].flags & FILE_LISTEN) != 0)
 	rc = _sys_wait_accept (cp->fd);
       else
-	rc = _sys_read_ahead (cp->fd);
+#ifdef WORKING_SELECT_EMULATION
+        if (fd_info[cp->fd].flags & FILE_SOCKET)
+          rc = _sys_wait_readable (cp->fd);
+        else
+#endif
+          rc = _sys_read_ahead (cp->fd);
 
       /* Don't bother waiting for the event if we already have been
 	 told to exit by delete_child.  */

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Robin Tarsiger <rtt@dasyatidae.com>,  Eli Zaretskii <eliz@gnu.org>,
>   emacs-devel@gnu.org
> Date: Mon, 11 Jan 2021 21:12:44 +0100
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> Because the select implementation on windows does a one-byte readahead
> >> to see if data is available, which breaks UDP. I had a patch at one
> >> point to fix this, but I remember Eli not being very enthusiastic
> >> about it. Iʼve attached what I think is the right version below (my
> >> windows box died, so I can't be sure)
> >
> > How 'bout installing it but make it conditional on some config var?
> > And maybe set that config var if/when a UDP socket is requested?
> 
> I guess thatʼs possible. Iʼve now found the actual patch. If Eli
> thinks itʼs worth persuing, I can make it conditional, unless someone
> wants to spare me the trouble of setting up a working Windows dev
> environment :-).

My main concern is who will investigate the bug reports about this,
debug the problem, find fixes, etc.?  If you or Stefan or someone else
volunteers, then please go ahead.  But if you count on me, implicitly
or otherwise, then let's wait for a volunteer to emerge.  Especially
in this area, where I'm far from being an expert.  Don't forget that
two threads are involved in this game, which provides ample
opportunity for exciting deadlocks and races, apart of
network-specific issues.

(I'm sorry, but too often lately I'm the only one who gets to debug
and fix MS-Windows specific issues; if I don't do that, the build
remains broken for days.  It's too much of a burden on me, and takes a
significant enough fraction of my time that I must be very cautious
with adventures.  Otherwise, I won't be able to do my main job here,
which I already am barely capable of doing.)

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Stefan Monnier]

>> I guess thatʼs possible. Iʼve now found the actual patch. If Eli
>> thinks itʼs worth persuing, I can make it conditional, unless someone
>> wants to spare me the trouble of setting up a working Windows dev
>> environment :-).
>
> My main concern is who will investigate the bug reports about this,
> debug the problem, find fixes, etc.?

Indeed, since Robert (luckily) doesn't use Windows (any more?), we'd
need to find someone else to take responsibility for the patch.

> If you or Stefan or someone else volunteers, then please go ahead.

Don't count on me with Windows code, no.


        Stefan

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Cc: Robin Tarsiger <rtt@dasyatidae.com>,  Eli Zaretskii <eliz@gnu.org>,
>>   emacs-devel@gnu.org
>> Date: Mon, 11 Jan 2021 21:12:44 +0100
>> 
>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> 
>> >> Because the select implementation on windows does a one-byte readahead
>> >> to see if data is available, which breaks UDP. I had a patch at one
>> >> point to fix this, but I remember Eli not being very enthusiastic
>> >> about it. Iʼve attached what I think is the right version below (my
>> >> windows box died, so I can't be sure)
>> >
>> > How 'bout installing it but make it conditional on some config var?
>> > And maybe set that config var if/when a UDP socket is requested?
>> 
>> I guess thatʼs possible. Iʼve now found the actual patch. If Eli
>> thinks itʼs worth persuing, I can make it conditional, unless someone
>> wants to spare me the trouble of setting up a working Windows dev
>> environment :-).
>
> My main concern is who will investigate the bug reports about this,
> debug the problem, find fixes, etc.?  If you or Stefan or someone else
> volunteers, then please go ahead.  But if you count on me, implicitly
> or otherwise, then let's wait for a volunteer to emerge.  Especially
> in this area, where I'm far from being an expert.  Don't forget that
> two threads are involved in this game, which provides ample
> opportunity for exciting deadlocks and races, apart of
> network-specific issues.

I donʼt think thereʼs too much scope for thread-induced breakage, but
this is threads, so that statement might well come back to bite me :-)

> (I'm sorry, but too often lately I'm the only one who gets to debug
> and fix MS-Windows specific issues; if I don't do that, the build
> remains broken for days.  It's too much of a burden on me, and takes a
> significant enough fraction of my time that I must be very cautious
> with adventures.  Otherwise, I won't be able to do my main job here,
> which I already am barely capable of doing.)

I can understand this. I donʼt run windows except in a VM, and the
whole reason for the patch initially was to reduce the differences in
the network implementation between different platforms (you can call
this an obsessive desire for parity on my part if you like).

If the cost of making udp processes work on Windows is not worth it,
then we can stop here.

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> If the cost of making udp processes work on Windows is not worth it,
> then we can stop here.

I'd really like to have UDP on Windows work.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> If the cost of making udp processes work on Windows is not worth it,
>> then we can stop here.
>
> I'd really like to have UDP on Windows work.

So would I, but I can't make the future time commitment that Eli says
is necessary.

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> So would I, but I can't make the future time commitment that Eli says
> is necessary.

I appreciate Eli's worries here, but you said you had a Windows VM, so
you could test this before it's applied?  So do I, so if we test this
thoroughly first, perhaps that'll alleviate some of the concerns.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> So would I, but I can't make the future time commitment that Eli says
>> is necessary.
>
> I appreciate Eli's worries here, but you said you had a Windows VM, so
> you could test this before it's applied?  So do I, so if we test this
> thoroughly first, perhaps that'll alleviate some of the concerns.

I have a Windows VM, but it doesnʼt currently have any dev tools
installed. I think Eli is more worried about a year down the line when
something breaks.

I suppose if we made the change conditional on some variable as Stefan
suggested, default enable, then people could always get back the
previous behaviour by disabling it. I dislike that idea because itʼs
more work, but perhaps itʼs necessary here.

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> I have a Windows VM, but it doesnʼt currently have any dev tools
> installed. I think Eli is more worried about a year down the line when
> something breaks.

Do you mean the MSYS2 stuff?  I wrote up a recipe when I installed my
VM; you can copy/paste from here.  :-)

https://lars.ingebrigtsen.no/2020/12/24/building-the-development-version-of-emacs-on-windows-mingw-edition/

> I suppose if we made the change conditional on some variable as Stefan
> suggested, default enable, then people could always get back the
> previous behaviour by disabling it. I dislike that idea because itʼs
> more work, but perhaps itʼs necessary here.

I tried your patch, and it worked fine for me (using `dns-query' as the
test case).  I don't really see the need for a variable here, either.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> I have a Windows VM, but it doesnʼt currently have any dev tools
>> installed. I think Eli is more worried about a year down the line when
>> something breaks.
>
> Do you mean the MSYS2 stuff?  I wrote up a recipe when I installed my
> VM; you can copy/paste from here.  :-)
>
> https://lars.ingebrigtsen.no/2020/12/24/building-the-development-version-of-emacs-on-windows-mingw-edition/
>

I know how to do it (the instructions shipped with emacs work fine),
Iʼm just lacking any motivation to touch Windows :-)

>> I suppose if we made the change conditional on some variable as Stefan
>> suggested, default enable, then people could always get back the
>> previous behaviour by disabling it. I dislike that idea because itʼs
>> more work, but perhaps itʼs necessary here.
>
> I tried your patch, and it worked fine for me (using `dns-query' as the
> test case).  I don't really see the need for a variable here, either.

You verified it sent UDP dns queries? The patch as is didnʼt enable
that.

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> You verified it sent UDP dns queries? The patch as is didnʼt enable
> that.

Yup, I defined the thingie and verified that it sent UDP DNS queries.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> You verified it sent UDP dns queries? The patch as is didnʼt enable
>> that.
>
> Yup, I defined the thingie and verified that it sent UDP DNS queries.

OK. I leave it up to you and Eli to decide what to do with it.

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> OK. I leave it up to you and Eli to decide what to do with it.

I've filed a bug report for it, so that we don't lose track of the patch
again.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Stefan Monnier]

>> I suppose if we made the change conditional on some variable as Stefan
>> suggested, default enable, then people could always get back the
>> previous behaviour by disabling it. I dislike that idea because itʼs
>> more work, but perhaps itʼs necessary here.
>
> I tried your patch, and it worked fine for me (using `dns-query' as the
> test case).  I don't really see the need for a variable here, either.

Clearly we want this patch to make UDP work, but whether UDP works is
somewhat secondary: The worry is that the patch will introduce
a regression in other places.  That makes it hard to test, and that's
why Eli wants someone to take charge of the potential consequences down
the line, which may only show up in hard-to-reproduce corner cases.


        Stefan

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 12 Jan 2021 10:14:39 -0500
> Cc: emacs-devel@gnu.org
> 
> Clearly we want this patch to make UDP work, but whether UDP works is
> somewhat secondary: The worry is that the patch will introduce
> a regression in other places.

The modified code will affect every network connection made by Emacs,
not only UDP connections.

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: monnier@iro.umontreal.ca,  rtt@dasyatidae.com,  emacs-devel@gnu.org
> Date: Mon, 11 Jan 2021 22:02:24 +0100
> 
> If the cost of making udp processes work on Windows is not worth it,
> then we can stop here.

I don't know if it's worth it.  I know we didn't have any significant
complaints about that (or not at all), but that doesn't necessarily
mean adding that won't be appreciated.

I just cannot be responsible for fixing the breakage which could
follow.  If someone steps forward, I don't mind adding that code
(after we discuss some of the minor details).

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 11 Jan 2021 18:53:51 +0100
> 
> (require 'dns)
> (skip-unless (dns-query "google.com" 'AAAA))
> 
> and then do the network-lookup-address-info tests (although on Windows
> this depends on DNS over TCP working, and either /etc/resolv.conf or a
> working nslookup). Eli?

Tell me what to try, and I will show the results.  TIA.

> > Another perspective on above is that when you mention "behind firewalls up
> > the kazoo", what my intuition imagines is something like: corporate DNS
> > filtering which only allows specific request types and either was never
> > updated to think of AAAA queries as valid, or is operated in a way assuming
> > that users will never have a legitimate need for them (because they "won't
> > be able to use" the results), or does this as a workaround for some broken
> > behavior elsewhere (such as some other device used in the organization
> > which will prefer an IPv6 address response even when it's not reachable,
> > and fail to fall back to IPv4?) or... any number of things like that.
> >
> 
> Iʼd love to be able to persuade Eli's network admins to fix such
> things, but I only attempt things that have a reasonable chance of
> success (and such things are sometimes dependent on vendors rather
> than admins).

No danger of that succeeding in this case, forget it.  I had trouble
convincing them to let me access Git repositories via HTTPS, let alone
SSH.  Every non-standard port under the sun is blocked.  I cannot even
ping any address, have to use tcping instead.

Thanks.

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Mon, 11 Jan 2021 18:53:51 +0100
>> 
>> (require 'dns)
>> (skip-unless (dns-query "google.com" 'AAAA))
>> 
>> and then do the network-lookup-address-info tests (although on Windows
>> this depends on DNS over TCP working, and either /etc/resolv.conf or a
>> working nslookup). Eli?
>
> Tell me what to try, and I will show the results.  TIA.

Iʼve lost track of which tests were failing, but something like
following. And if the 'dns-query' calls succeed but the
network-lookup-address-info ones fail then we have a really messed up
situation.

diff --git a/test/src/process-tests.el b/test/src/process-tests.el
index 921bcd5f85..172294b0ed 100644
--- a/test/src/process-tests.el
+++ b/test/src/process-tests.el
@@ -356,7 +356,9 @@ lookup-family-specification
   (with-timeout (60 (ert-fail "Test timed out"))
   (should-error (network-lookup-address-info "google.com" 'both))
   (should (network-lookup-address-info "google.com" 'ipv4))
-  (when (featurep 'make-network-process '(:family ipv6))
+  (when (and (featurep 'make-network-process '(:family ipv6))
+             (require 'dns)
+             (dns-query "google.com" 'AAAA))
     (should (network-lookup-address-info "google.com" 'ipv6)))))
 
 (ert-deftest lookup-unicode-domains ()
@@ -380,7 +382,9 @@ lookup-google
         (addresses-v4 (network-lookup-address-info "google.com" 'ipv4)))
     (should addresses-both)
     (should addresses-v4))
-  (when (featurep 'make-network-process '(:family ipv6))
+  (when (and (featurep 'make-network-process '(:family ipv6))
+             (require 'dns)
+             (dns-query "google.com" 'AAAA))
     (should (network-lookup-address-info "google.com" 'ipv6)))))
 
 (ert-deftest non-existent-lookup-failure ()

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: rtt@dasyatidae.com,  emacs-devel@gnu.org
> Date: Mon, 11 Jan 2021 21:56:57 +0100
> 
> > Tell me what to try, and I will show the results.  TIA.
> 
> Iʼve lost track of which tests were failing, but something like
> following. And if the 'dns-query' calls succeed but the
> network-lookup-address-info ones fail then we have a really messed up
> situation.

You assume that a TCP connection to the DNS server there will succeed?
It doesn't, at least judging by the error message (dns.el has no debug
facilities, so it's hard to know better).

So I think I will now stop wasting your time, and just give up on
having these tests avoid false negatives.

But before I go!  I guess at least one of the 2 offending tests,
i.e. this one:

  (ert-deftest lookup-family-specification ()
    "`network-lookup-address-info' should only accept valid family symbols."
    (skip-unless (not (getenv "EMACS_HYDRA_CI")))
    (with-timeout (60 (ert-fail "Test timed out"))
    (should-error (network-lookup-address-info "google.com" 'both))
    (should (network-lookup-address-info "google.com" 'ipv4))
    (when (featurep 'make-network-process '(:family ipv6))
      (should (network-lookup-address-info "google.com" 'ipv6)))))

could use "localhost" instead of google.com, since it only tests the
input validity?

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Cc: rtt@dasyatidae.com,  emacs-devel@gnu.org
>> Date: Mon, 11 Jan 2021 21:56:57 +0100
>> 
>> > Tell me what to try, and I will show the results.  TIA.
>> 
>> Iʼve lost track of which tests were failing, but something like
>> following. And if the 'dns-query' calls succeed but the
>> network-lookup-address-info ones fail then we have a really messed up
>> situation.
>
> You assume that a TCP connection to the DNS server there will succeed?
> It doesn't, at least judging by the error message (dns.el has no debug
> facilities, so it's hard to know better).
>

Perhaps UDP would work better... (sorry, couldn't resist)

> So I think I will now stop wasting your time, and just give up on
> having these tests avoid false negatives.
>
> But before I go!  I guess at least one of the 2 offending tests,
> i.e. this one:
>
>   (ert-deftest lookup-family-specification ()
>     "`network-lookup-address-info' should only accept valid family symbols."
>     (skip-unless (not (getenv "EMACS_HYDRA_CI")))
>     (with-timeout (60 (ert-fail "Test timed out"))
>     (should-error (network-lookup-address-info "google.com" 'both))
>     (should (network-lookup-address-info "google.com" 'ipv4))
>     (when (featurep 'make-network-process '(:family ipv6))
>       (should (network-lookup-address-info "google.com" 'ipv6)))))
>
> could use "localhost" instead of google.com, since it only tests the
> input validity?

It could indeed. 'lookup-google' further down repeats the lookups
anyway. Iʼll queue that up.

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Eli Zaretskii]

> Date: Tue, 12 Jan 2021 17:01:53 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: rtt@dasyatidae.com, emacs-devel@gnu.org
> 
> > Iʼve lost track of which tests were failing, but something like
> > following. And if the 'dns-query' calls succeed but the
> > network-lookup-address-info ones fail then we have a really messed up
> > situation.
> 
> You assume that a TCP connection to the DNS server there will succeed?

Btw, this succeeds even on my XP box, which doesn't have IPv6
installed:

  (dns-query "google.com" 'AAAA)
    => "2a00:1450:4009:80b:0:0:0:200e"

So I'm not sure that test is reliable enough for these purposes.

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Tue, 12 Jan 2021 17:01:53 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: rtt@dasyatidae.com, emacs-devel@gnu.org
>> 
>> > Iʼve lost track of which tests were failing, but something like
>> > following. And if the 'dns-query' calls succeed but the
>> > network-lookup-address-info ones fail then we have a really messed up
>> > situation.
>> 
>> You assume that a TCP connection to the DNS server there will succeed?
>
> Btw, this succeeds even on my XP box, which doesn't have IPv6
> installed:
>
>   (dns-query "google.com" 'AAAA)
>     => "2a00:1450:4009:80b:0:0:0:200e"
>
> So I'm not sure that test is reliable enough for these purposes.

Itʼs checking if IPv6 names can be resolved, not if IPv6 works. If you
think thatʼs not a useful test, we can remove it.

I can condition all the IPv6 tests on the following, which should
alleviate your concerns:

;; This will need updating when IANA assign more IPv6 global ranges.
(defun ipv6-is-available ()
  (and (featurep 'make-network-process '(:family ipv6))
       (cl-rassoc-if
        (lambda (elt)
          (and (eq 9 (length elt))
               (= (logand (aref elt 0) #xe000) #x2000)))
        (network-interface-list))))

(although itʼs needed in nsm-tests.el and process-tests.el, which
means duplication <sigh>)

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: rtt@dasyatidae.com,  emacs-devel@gnu.org
> Date: Tue, 12 Jan 2021 17:00:56 +0100
> 
> >   (dns-query "google.com" 'AAAA)
> >     => "2a00:1450:4009:80b:0:0:0:200e"
> >
> > So I'm not sure that test is reliable enough for these purposes.
> 
> Itʼs checking if IPv6 names can be resolved, not if IPv6 works. If you
> think thatʼs not a useful test, we can remove it.

No, feel free to ignore me.  I was just surprised it worked, that's
all.

> I can condition all the IPv6 tests on the following, which should
> alleviate your concerns:
> 
> ;; This will need updating when IANA assign more IPv6 global ranges.
> (defun ipv6-is-available ()
>   (and (featurep 'make-network-process '(:family ipv6))
>        (cl-rassoc-if
>         (lambda (elt)
>           (and (eq 9 (length elt))
>                (= (logand (aref elt 0) #xe000) #x2000)))
>         (network-interface-list))))

Thanks, that would be good, I think.

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Robert Pluim <rpluim@gmail.com>
>> Cc: rtt@dasyatidae.com,  emacs-devel@gnu.org
>> Date: Tue, 12 Jan 2021 17:00:56 +0100
>> 
>> >   (dns-query "google.com" 'AAAA)
>> >     => "2a00:1450:4009:80b:0:0:0:200e"
>> >
>> > So I'm not sure that test is reliable enough for these purposes.
>> 
>> Itʼs checking if IPv6 names can be resolved, not if IPv6 works. If you
>> think thatʼs not a useful test, we can remove it.
>
> No, feel free to ignore me.  I was just surprised it worked, that's
> all.
>
>> I can condition all the IPv6 tests on the following, which should
>> alleviate your concerns:
>> 
>> ;; This will need updating when IANA assign more IPv6 global ranges.
>> (defun ipv6-is-available ()
>>   (and (featurep 'make-network-process '(:family ipv6))
>>        (cl-rassoc-if
>>         (lambda (elt)
>>           (and (eq 9 (length elt))
>>                (= (logand (aref elt 0) #xe000) #x2000)))
>>         (network-interface-list))))
>
> Thanks, that would be good, I think.

Done as 0f6c083251

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Basil L. Contovounesios]

Robert Pluim <rpluim@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> I'm asking because I'm still bothered by failures in those two tests,
>> and would like to avoid that.  I think it's bad if a test fails when
>> there's nothing wrong with Emacs; we should try to skip such tests
>> when we know their results cannot be trusted.
>
> The results can be trusted. However, the assumption of the tests is
> that DNS lookups on AAAA records will work, and return results for
> google.com, which is not the case for you. If you prefer we can change
> the tests to check for "no error" rather than 'returns a result' (is
> there an ERT 'should-not-error' clause)?

Yes, progn (implicit or otherwise) ;).
[ See the penultimate paragraph in (info "(ert) The should Macro"). ]

This is definitely a long shot, but any chance bug#45798 relates to
this?

Thanks,

-- 
Basil

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> Robert Pluim <rpluim@gmail.com> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>> I'm asking because I'm still bothered by failures in those two tests,
>>> and would like to avoid that.  I think it's bad if a test fails when
>>> there's nothing wrong with Emacs; we should try to skip such tests
>>> when we know their results cannot be trusted.
>>
>> The results can be trusted. However, the assumption of the tests is
>> that DNS lookups on AAAA records will work, and return results for
>> google.com, which is not the case for you. If you prefer we can change
>> the tests to check for "no error" rather than 'returns a result' (is
>> there an ERT 'should-not-error' clause)?
>
> Yes, progn (implicit or otherwise) ;).
> [ See the penultimate paragraph in (info "(ert) The should Macro"). ]
>
> This is definitely a long shot, but any chance bug#45798 relates to
> this?

Itʼs possible. Iʼll bombard you with questions over there.

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Andy Moreton]

On Mon 11 Jan 2021, Robert Pluim wrote:

> Eli Zaretskii <eliz@gnu.org> writes:

> The results can be trusted. However, the assumption of the tests is
> that DNS lookups on AAAA records will work, and return results for
> google.com, which is not the case for you. If you prefer we can change
> the tests to check for "no error" rather than 'returns a result' (is
> there an ERT 'should-not-error' clause)?

CAn the tests please be changed to assume that IPv6 is not always
avvailable and AAAA lookup may fail ? There are many people using
networks without IPv6 available. These tests usually fail or hang on my
systems.

It would be better if the tests did not rely on external connectivity,
and did not need to contact google.com gratuitously.

    AndyM

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[Robert Pluim]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> On Mon 11 Jan 2021, Robert Pluim wrote:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>
>> The results can be trusted. However, the assumption of the tests is
>> that DNS lookups on AAAA records will work, and return results for
>> google.com, which is not the case for you. If you prefer we can change
>> the tests to check for "no error" rather than 'returns a result' (is
>> there an ERT 'should-not-error' clause)?
>
> CAn the tests please be changed to assume that IPv6 is not always
> avvailable and AAAA lookup may fail ? There are many people using
> networks without IPv6 available. These tests usually fail or hang on my
> systems.
>

The IPv6 tests are only run when Emacs is compiled with IPv6
support. The question "is IPv6 available" is harder to answer, but we
can take a stab at it.

None of the tests should hang. If you can pinpoint ones that do,
please let us know.

> It would be better if the tests did not rely on external connectivity,
> and did not need to contact google.com gratuitously.
>
>     AndyM

They donʼt contact google.com, they perform DNS lookups on
google.com. The connectivity tests we have connect to localhost.

External connectivity is one of the big features of Emacs, it should
be tested by the test suite.

Robert

Re: ai_flags in calls to getaddrinfo, broader call for reproducibility check

[tomas]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

On Tue, Jan 12, 2021 at 10:44:57AM +0100, Robert Pluim wrote:

[...]

> > It would be better if the tests did not rely on external connectivity,
> > and did not need to contact google.com gratuitously.
> >
> >     AndyM
> 
> They donʼt contact google.com, they perform DNS lookups on
> google.com. The connectivity tests we have connect to localhost.
> 
> External connectivity is one of the big features of Emacs, it should
> be tested by the test suite.

Agreed. Still, it would be nice if there were alternatives to google
more in resonance with Emacs; perhaps, since (arguably) DNS resolution
of such a behemoth as Google would be more reliable than everyone
else, we might need a couple of alternatives. Gnu.org comes to mind,
but perhaps debian.org, wikipedia.org or archive.org might qualify?

I don't have the chops to assess the ramifications of this -- perhaps
something for a discussio (not necessarily in this thread).

Cheers
-- t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: ai_flags in calls to getaddrinfo

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Adding the AI_V4MAPPED flag, as in the patch below, seems to solve the
> problem.  So I wonder why we don't do this in general.  I'm not an
> expert on DNS, so would people who know more than I do about this
> please comment on whether the patch below is a good idea?

I think that when the user asks for ipv6 explicitly, they should get
back ipv6 (or get nothing/a failure).  If the user has specified the
protocol, it's presumably because they really want that protocol.

If they don't specify the protocol version, they get back either/both
ipv6/ipv4 now (depending on the system), don't they?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: ai_flags in calls to getaddrinfo

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 01 Jan 2021 11:59:37 +0100
> 
> I think that when the user asks for ipv6 explicitly, they should get
> back ipv6 (or get nothing/a failure).  If the user has specified the
> protocol, it's presumably because they really want that protocol.

But isn't the return value a valid IPv6 address when using AI_V4MAPPED?

> If they don't specify the protocol version, they get back either/both
> ipv6/ipv4 now (depending on the system), don't they?

On that system, I get only the IPv4 address if I don't use AI_ALL and
AI_V4MAPPED.  If I do use the two flags, I get both IPv4 and IPv6
addresses.

Re: ai_flags in calls to getaddrinfo

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> If they don't specify the protocol version, they get back either/both
>> ipv6/ipv4 now (depending on the system), don't they?
>
> On that system, I get only the IPv4 address if I don't use AI_ALL and
> AI_V4MAPPED.  If I do use the two flags, I get both IPv4 and IPv6
> addresses.

But the IPv6 addresses you get are presumably just the
ipv4-remapped-to-ipv6 addresses?  If so, that's another reason not to
use AI_V4MAPPED -- Robin explained in another message how those aren't
that useful.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no