bug#47431: Process Whois connection broken by remote peer.

bug#47431: Process Whois connection broken by remote peer.

[Hongyi Zhao]

Hi,

On Ubuntu 20.04, I'm using the self compiled git master version of Emacs.
It seems that the whois client in Emacs only works for some specific domain
 names, for example gnu.org and emacs.org, but for others like
facebook.com it just hangs and returns nothing. The following command
will trigger the error:

"M-x RET whois RET facebook.com"

Regards
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

bug#47431: Process Whois connection broken by remote peer.

[Eli Zaretskii]

> From: Hongyi Zhao <hongyi.zhao@gmail.com>
> Date: Sat, 27 Mar 2021 09:39:20 +0800
> 
> On Ubuntu 20.04, I'm using the self compiled git master version of Emacs.
> It seems that the whois client in Emacs only works for some specific domain
>  names, for example gnu.org and emacs.org, but for others like
> facebook.com it just hangs and returns nothing. The following command
> will trigger the error:
> 
> "M-x RET whois RET facebook.com"

It doesn't hang here (I'm not on Ubuntu, though).  It times out:

  open-network-stream: make client process failed: Connection timed out, :name, Whois, :buffer, *Whois*, :host, rs.internic.net, :service, 43, :nowait, nil, :tls-parameters, nil, :coding, nil

If I try a different server, it does work:

  C-u M-x whois RET facebook.com RET whois.crsnic.net RET

This returns immediately with a large *Whois* buffer.

Perhaps some network expert could tell why we use by default a server
that is less useful.

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

>> On Ubuntu 20.04, I'm using the self compiled git master version of 
>> Emacs. It seems that the whois client in Emacs only works for some 
>> specific domain names, for example gnu.org and emacs.org, but for 
>> others like facebook.com it just hangs and returns nothing. The 
>> following command will trigger the error:
>>
>> "M-x RET whois RET facebook.com"
>
> It doesn't hang here (I'm not on Ubuntu, though).  It times out:
>
>  open-network-stream: make client process failed: Connection timed out, 
> :name, Whois, :buffer, *Whois*, :host, rs.internic.net, :service, 43, 
> :nowait, nil, :tls-parameters, nil, :coding, nil
>
> If I try a different server, it does work:
>
>  C-u M-x whois RET facebook.com RET whois.crsnic.net RET
>
> This returns immediately with a large *Whois* buffer.
>
> Perhaps some network expert could tell why we use by default a server 
> that is less useful.
>

(I'm not a network expert, but...)

It is probably better to let the whois client select the appropriate 
server itself.  The current default whois-server-name (rs.internic.net) 
doesn't work at all, and although whois-guess-server is t, it doesn't work 
well.

The manpage of whois says: "This version of the whois client tries to 
guess the right server to ask for the specified object. If no guess can be 
made it will connect to whois.networksolutions.com for NIC handles or 
whois.arin.net for IPv4 addresses and network names."

bug#47431: Process Whois connection broken by remote peer.

[Eli Zaretskii]

> Date: Sat, 27 Mar 2021 07:23:20 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: Hongyi Zhao <hongyi.zhao@gmail.com>, 47431@debbugs.gnu.org
> 
> The manpage of whois says: "This version of the whois client tries to 
> guess the right server to ask for the specified object. If no guess can be 
> made it will connect to whois.networksolutions.com for NIC handles or 
> whois.arin.net for IPv4 addresses and network names."

We don't use the external 'whois' command, so its man page is not
relevant, I think.

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

>> The manpage of whois says: "This version of the whois client tries to 
>> guess the right server to ask for the specified object. If no guess can 
>> be made it will connect to whois.networksolutions.com for NIC handles 
>> or whois.arin.net for IPv4 addresses and network names."
>
> We don't use the external 'whois' command, so its man page is not 
> relevant, I think.
>

Indeed.  Would it not make sense to use the external whois command if it 
is available, and to fall back to the Lisp code when it is not?

In any case, the whois-server-tld alist needs to be updated, it has only 
15 entries [1], the whois client for GNU/Linux has more than 400 (see 
[2]).  And these 400 are only for the domain name lookups, there are two 
others for IP addresses, another for NIC handles, and another one for AS 
numbers...

[1] Its first entry is not a valid server name, all other ones except the 
"org" and "mil" are valid server names but are not the appropriate server 
for the corresponding TLD.

[2] https://raw.githubusercontent.com/rfc1036/whois/next/tld_serv_list

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

>>> The manpage of whois says: "This version of the whois client tries to 
>>> guess the right server to ask for the specified object. If no guess 
>>> can be made it will connect to whois.networksolutions.com for NIC 
>>> handles or whois.arin.net for IPv4 addresses and network names."
>> 
>> We don't use the external 'whois' command, so its man page is not 
>> relevant, I think.
>
> Indeed.  Would it not make sense to use the external whois command if it 
> is available, and to fall back to the Lisp code when it is not?
>
> In any case, the whois-server-tld alist needs to be updated, it has only 
> 15 entries [1], the whois client for GNU/Linux has more than 400 (see 
> [2]).  And these 400 are only for the domain name lookups, there are two 
> others for IP addresses, another for NIC handles, and another one for AS 
> numbers...
>

Two other _lists_, I mean.

>
> [1] Its first entry is not a valid server name, all other ones except 
> the "org" and "mil" are valid server names but are not the appropriate 
> server for the corresponding TLD.
>
> [2] https://raw.githubusercontent.com/rfc1036/whois/next/tld_serv_list
>

bug#47431: Process Whois connection broken by remote peer.

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

> [2] https://raw.githubusercontent.com/rfc1036/whois/next/tld_serv_list

The list originates at whois.iana.org...  which is a server that seems
to respond to whois queries nicely.  So we could just update
whois-server-name to that?  And remove whois-server-tld.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

>> [2] https://raw.githubusercontent.com/rfc1036/whois/next/tld_serv_list
>
> The list originates at whois.iana.org...  which is a server that seems 
> to respond to whois queries nicely.  So we could just update 
> whois-server-name to that?  And remove whois-server-tld.
>

That would be ideal/simpler, but whois.iana.org only contains information 
about the whois server to use for each TLD.  For instance whois -h 
whois.iana.org gnu.org tells you that the whois server to contact is 
whois.pir.org.  IOW, without a built-in list, each whois query would 
create two requests, one to whois.iana.org, and one to the actual whois 
server.  IOW again, a whois request would become:

whois -h $(whois -h whois.iana.org DOMAIN | grep '^whois:' | cut -d: -f2) DOMAIN

Would that be acceptable?

bug#47431: Process Whois connection broken by remote peer.

[Hongyi Zhao]

On Mon, Mar 29, 2021 at 4:23 AM Gregory Heytings <gregory@heytings.org> wrote:
>
>
> >> [2] https://raw.githubusercontent.com/rfc1036/whois/next/tld_serv_list
> >
> > The list originates at whois.iana.org...  which is a server that seems
> > to respond to whois queries nicely.  So we could just update
> > whois-server-name to that?  And remove whois-server-tld.
> >
>
> That would be ideal/simpler, but whois.iana.org only contains information
> about the whois server to use for each TLD.  For instance whois -h
> whois.iana.org gnu.org tells you that the whois server to contact is
> whois.pir.org.  IOW, without a built-in list, each whois query would
> create two requests, one to whois.iana.org, and one to the actual whois
> server.  IOW again, a whois request would become:
>
> whois -h $(whois -h whois.iana.org DOMAIN | grep '^whois:' | cut -d: -f2) DOMAIN
>
> Would that be acceptable?

The suggested method really can do the trick:

;;; begin quote
$ whois -h $(whois -h whois.iana.org facebook.com | grep '^whois:' |
cut -d: -f2) facebook.com
   Domain Name: FACEBOOK.COM
   Registry Domain ID: 2320948_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.registrarsafe.com
   Registrar URL: http://www.registrarsafe.com
   Updated Date: 2020-03-10T18:53:59Z
   Creation Date: 1997-03-29T05:00:00Z
   Registry Expiry Date: 2028-03-30T04:00:00Z
   Registrar: RegistrarSafe, LLC
   Registrar IANA ID: 3237
   Registrar Abuse Contact Email: abusecomplaints@registrarsafe.com
   Registrar Abuse Contact Phone: +1-650-308-7004
   Domain Status: clientDeleteProhibited
https://icann.org/epp#clientDeleteProhibited
   Domain Status: clientTransferProhibited
https://icann.org/epp#clientTransferProhibited
   Domain Status: clientUpdateProhibited
https://icann.org/epp#clientUpdateProhibited
   Domain Status: serverDeleteProhibited
https://icann.org/epp#serverDeleteProhibited
   Domain Status: serverTransferProhibited
https://icann.org/epp#serverTransferProhibited
   Domain Status: serverUpdateProhibited
https://icann.org/epp#serverUpdateProhibited
   Name Server: A.NS.FACEBOOK.COM
   Name Server: B.NS.FACEBOOK.COM
   Name Server: C.NS.FACEBOOK.COM
   Name Server: D.NS.FACEBOOK.COM
   DNSSEC: unsigned
   URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of whois database: 2021-03-29T00:14:06Z <<<

For more information on Whois status codes, please visit https://icann.org/epp

NOTICE: The expiration date displayed in this record is the date the
registrar's sponsorship of the domain name registration in the registry is
currently set to expire. This date does not necessarily reflect the expiration
date of the domain name registrant's agreement with the sponsoring
registrar.  Users may consult the sponsoring registrar's Whois database to
view the registrar's reported date of expiration for this registration.

TERMS OF USE: You are not authorized to access or query our Whois
database through the use of electronic processes that are high-volume and
automated except as reasonably necessary to register domain names or
modify existing registrations; the Data in VeriSign Global Registry
Services' ("VeriSign") Whois database is provided by VeriSign for
information purposes only, and to assist persons in obtaining information
about or related to a domain name registration record. VeriSign does not
guarantee its accuracy. By submitting a Whois query, you agree to abide
by the following terms of use: You agree that you may use this Data only
for lawful purposes and that under no circumstances will you use this Data
to: (1) allow, enable, or otherwise support the transmission of mass
unsolicited, commercial advertising or solicitations via e-mail, telephone,
or facsimile; or (2) enable high volume, automated, electronic processes
that apply to VeriSign (or its computer systems). The compilation,
repackaging, dissemination or other use of this Data is expressly
prohibited without the prior written consent of VeriSign. You agree not to
use electronic processes that are automated and high-volume to access or
query the Whois database except as reasonably necessary to register
domain names or modify existing registrations. VeriSign reserves the right
to restrict your access to the Whois database in its sole discretion to ensure
operational stability.  VeriSign may restrict or terminate your access to the
Whois database for failure to abide by these terms of use. VeriSign
reserves the right to modify these terms at any time.

The Registry database contains ONLY .COM, .NET, .EDU domains and
Registrars.

;;; end quote


But I'm not sure if this is the standard query procedures
suggested/defined by the whois protocol itself.

Regards
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

bug#47431: Process Whois connection broken by remote peer.

[Eli Zaretskii]

> Date: Sun, 28 Mar 2021 20:23:17 +0000
> From: Gregory Heytings <gregory@heytings.org>
> Cc: 47431@debbugs.gnu.org, hongyi.zhao@gmail.com
> 
> IOW, without a built-in list, each whois query would create two
> requests, one to whois.iana.org, and one to the actual whois server.

I don't see any problems with that, do you?

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

>> IOW, without a built-in list, each whois query would create two 
>> requests, one to whois.iana.org, and one to the actual whois server.
>
> I don't see any problems with that, do you?
>

In principle, I don't see any problems.  But I seem to recall that RMS 
dislikes solutions that make unnecessary network connections, or IOW that 
avoidable network connections should be avoided.

If you agree on the general design, I'd be happy to implement it. 
Possibly with a cache to mitigate the above problem.

bug#47431: Process Whois connection broken by remote peer.

[Eli Zaretskii]

> Date: Mon, 29 Mar 2021 07:35:37 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: larsi@gnus.org, 47431@debbugs.gnu.org, hongyi.zhao@gmail.com
> 
> 
> >> IOW, without a built-in list, each whois query would create two 
> >> requests, one to whois.iana.org, and one to the actual whois server.
> >
> > I don't see any problems with that, do you?
> >
> 
> In principle, I don't see any problems.  But I seem to recall that RMS 
> dislikes solutions that make unnecessary network connections, or IOW that 
> avoidable network connections should be avoided.
> 
> If you agree on the general design, I'd be happy to implement it. 

I don't see a problem, no.  We frequently make network connections
when necessary.

> Possibly with a cache to mitigate the above problem.

I'd wait with caching until we see a performance problem.  It isn't
like people are expected to invoke this command many times in a row.

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

>>>> IOW, without a built-in list, each whois query would create two 
>>>> requests, one to whois.iana.org, and one to the actual whois server.
>>>
>>> I don't see any problems with that, do you?
>>
>> In principle, I don't see any problems.  But I seem to recall that RMS 
>> dislikes solutions that make unnecessary network connections, or IOW 
>> that avoidable network connections should be avoided.
>>
>> If you agree on the general design, I'd be happy to implement it.
>
> I don't see a problem, no.  We frequently make network connections when 
> necessary.
>
>> Possibly with a cache to mitigate the above problem.
>
> I'd wait with caching until we see a performance problem.  It isn't like 
> people are expected to invoke this command many times in a row.
>

I think you misunderstood what I meant.  The idea would be to cache the 
replies from the whois.iana.org server, not those of the final whois 
server.  IOW, the idea would be to dynamically build a local database 
instead of relying on a hard-coded list.

bug#47431: Process Whois connection broken by remote peer.

[Eli Zaretskii]

> Date: Mon, 29 Mar 2021 11:31:59 +0000
> From: Gregory Heytings <gregory@heytings.org>
> cc: larsi@gnus.org, 47431@debbugs.gnu.org, hongyi.zhao@gmail.com
> 
> >> Possibly with a cache to mitigate the above problem.
> >
> > I'd wait with caching until we see a performance problem.  It isn't like 
> > people are expected to invoke this command many times in a row.
> >
> 
> I think you misunderstood what I meant.  The idea would be to cache the 
> replies from the whois.iana.org server, not those of the final whois 
> server.  IOW, the idea would be to dynamically build a local database 
> instead of relying on a hard-coded list.

I understand that, and I'm still questioning the need for such a
cache.

If you have a cache, you need to manage it: add items that aren't
there, delete items no longer pertinent, etc.  Emacs sessions can run
for many moons, so the cache will have to be dynamically adjusted.  By
contrast, requesting the list each time a query is invoked is much
easier, so if performance is reasonable, why bother with a cache and
risk subtle issues?

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

>>>> Possibly with a cache to mitigate the above problem.
>>>
>>> I'd wait with caching until we see a performance problem.  It isn't 
>>> like people are expected to invoke this command many times in a row.
>>
>> I think you misunderstood what I meant.  The idea would be to cache the 
>> replies from the whois.iana.org server, not those of the final whois 
>> server.  IOW, the idea would be to dynamically build a local database 
>> instead of relying on a hard-coded list.
>
> I understand that, and I'm still questioning the need for such a cache.
>
> If you have a cache, you need to manage it: add items that aren't there, 
> delete items no longer pertinent, etc.  Emacs sessions can run for many 
> moons, so the cache will have to be dynamically adjusted.
>

Yes, I know this.

>
> By contrast, requesting the list each time a query is invoked is much 
> easier, so if performance is reasonable, why bother with a cache and 
> risk subtle issues?
>

It's not a performance issue, it's a privacy issue.  Emacs users might not 
want to communicate all their queries to two servers when communicating 
them to a single server is possible.

It might also (but that's not yet clear) be an abuse issue, as Lars 
pointed out.  It's not clear whether the whois.iana.org server is intended 
to be repeatedly queried by whois clients.

bug#47431: Process Whois connection broken by remote peer.

[Hongyi Zhao]

On Mon, Mar 29, 2021 at 8:12 PM Gregory Heytings <gregory@heytings.org> wrote:
>
>
> >>>> Possibly with a cache to mitigate the above problem.
> >>>
> >>> I'd wait with caching until we see a performance problem.  It isn't
> >>> like people are expected to invoke this command many times in a row.
> >>
> >> I think you misunderstood what I meant.  The idea would be to cache the
> >> replies from the whois.iana.org server, not those of the final whois
> >> server.  IOW, the idea would be to dynamically build a local database
> >> instead of relying on a hard-coded list.
> >
> > I understand that, and I'm still questioning the need for such a cache.
> >
> > If you have a cache, you need to manage it: add items that aren't there,
> > delete items no longer pertinent, etc.  Emacs sessions can run for many
> > moons, so the cache will have to be dynamically adjusted.
> >
>
> Yes, I know this.
>
> >
> > By contrast, requesting the list each time a query is invoked is much
> > easier, so if performance is reasonable, why bother with a cache and
> > risk subtle issues?
> >
>
> It's not a performance issue, it's a privacy issue.  Emacs users might not
> want to communicate all their queries to two servers when communicating
> them to a single server is possible.
>
> It might also (but that's not yet clear) be an abuse issue, as Lars
> pointed out.  It's not clear whether the whois.iana.org server is intended
> to be repeatedly queried by whois clients.

AFAICS, there should be the following considerations:

1. The whois.iana.org should not be queried by each of the clients on
the world repeated for each query which may overload the server.
2. Every whois query should be sent directly to a server near the
client's geographic location.

Regards
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

bug#47431: Process Whois connection broken by remote peer.

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

> IOW, without a built-in list, each whois query would create two
> requests, one to whois.iana.org, and one to the actual whois server.

That's fine.  The only issue here is whether whois.iana.org is designed
for this kind of use -- if not, it might be abusive of Emacs to do this,
and we should indeed keep the server list updated in Emacs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

>> IOW, without a built-in list, each whois query would create two 
>> requests, one to whois.iana.org, and one to the actual whois server.
>
> That's fine.  The only issue here is whether whois.iana.org is designed 
> for this kind of use -- if not, it might be abusive of Emacs to do this, 
> and we should indeed keep the server list updated in Emacs.
>

As far as I can see, whois.iana.org is indeed designed for this kind of 
use, and the GNU/Linux whois command even has an option to do this (-I). 
But I'm not 100% sure.  Do you think it's better if I first check this 
with IANA?

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

>>> IOW, without a built-in list, each whois query would create two 
>>> requests, one to whois.iana.org, and one to the actual whois server.
>> 
>> That's fine.  The only issue here is whether whois.iana.org is designed 
>> for this kind of use -- if not, it might be abusive of Emacs to do 
>> this, and we should indeed keep the server list updated in Emacs.
>
> As far as I can see, whois.iana.org is indeed designed for this kind of 
> use, and the GNU/Linux whois command even has an option to do this (-I). 
> But I'm not 100% sure.  Do you think it's better if I first check this 
> with IANA?
>

I've just sent a mail to IANA to clarify that point.

bug#47431: Process Whois connection broken by remote peer.

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

> I've just sent a mail to IANA to clarify that point.

Great; thanks.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

>> I've just sent a mail to IANA to clarify that point.
>
> Great; thanks.
>

I got the answer, from which I quote the relevant parts:

>
> We accept that users will query whois.iana.org port 43 for ad-hoc, 
> infrequent, and low volume queries (e.g. queries resulting from a 
> user-driven web form, upon receipt of an email, and given a small list 
> of domains).
>
> It reads like your client will be primarily user-driven and so we're 
> fine with your approach. Note, the Mac OSX whois client first queries 
> whois.iana.org and follows the referral, so there is precedent.
>

So from a technical point of view, the solution is okay.  What remains is 
the potential privacy problem, but I think it's possible to avoid it 
without using a cache.

bug#47431: Process Whois connection broken by remote peer.

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

>> We accept that users will query whois.iana.org port 43 for ad-hoc,
>> infrequent, and low volume queries (e.g. queries resulting from a
>> user-driven web form, upon receipt of an email, and given a small
>> list of domains).
>>
>> It reads like your client will be primarily user-driven and so we're
>> fine with your approach. Note, the Mac OSX whois client first
>> queries whois.iana.org and follows the referral, so there is
>> precedent.
>
> So from a technical point of view, the solution is okay.  What remains
> is the potential privacy problem, but I think it's possible to avoid
> it without using a cache.

I don't think there's an expectation of privacy here -- the user knows
that doing a whois query will result in server(s) being queried.

So I think whois.el should be rewritten to do the two step query (and be
asynchronous, which should be pretty trivial).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

>
> I don't think there's an expectation of privacy here -- the user knows 
> that doing a whois query will result in server(s) being queried.
>

Okay.

>
> So I think whois.el should be rewritten to do the two step query (and be 
> asynchronous, which should be pretty trivial).
>

Okay, I'll do that.

bug#47431: Process Whois connection broken by remote peer.

[Stefan Kangas]

Gregory Heytings <gregory@heytings.org> writes:

>>
>> I don't think there's an expectation of privacy here -- the user knows that
>> doing a whois query will result in server(s) being queried.
>>
>
> Okay.
>
>>
>> So I think whois.el should be rewritten to do the two step query (and be
>> asynchronous, which should be pretty trivial).
>>
>
> Okay, I'll do that.

I was completely unaware of this bug report, but I did some related work
here:

    commit 5d456136169468e78c877ead2a3e279d9ebc7e4c
    Author: Stefan Kangas <stefan@marxist.se>
    Date:   Wed Apr 7 13:35:59 2021 +0200

        Update whois-server-tld

        * lisp/net/net-utils.el (whois-server-tld): Update and add some
        missing entries.

FWIW, I added a comment that argues in favour of the solution chosen
here.  Even better that you have here checked it with IANA.  Thanks for
working on this!

bug#47431: Process Whois connection broken by remote peer.

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

>> I don't think there's an expectation of privacy here -- the user
>> knows that doing a whois query will result in server(s) being
>> queried.
>
> Okay.
>
>> So I think whois.el should be rewritten to do the two step query
>> (and be asynchronous, which should be pretty trivial).
>
> Okay, I'll do that.

Gregory, did you get any further here?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47431: Process Whois connection broken by remote peer.

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>>> I don't think there's an expectation of privacy here -- the user
>>> knows that doing a whois query will result in server(s) being
>>> queried.
>>
>> Okay.
>>
>>> So I think whois.el should be rewritten to do the two step query
>>> (and be asynchronous, which should be pretty trivial).
>>
>> Okay, I'll do that.
>
> Gregory, did you get any further here?

If you didn't get any further here, I can take a whack at implementing
this.

bug#47431: Process Whois connection broken by remote peer.

[Gregory Heytings]

[-- Attachment #1: Type: text/plain, Size: 491 bytes --]


>>>> So I think whois.el should be rewritten to do the two step query (and 
>>>> be asynchronous, which should be pretty trivial).
>>>
>>> Okay, I'll do that.
>>
>> Gregory, did you get any further here?
>
> If you didn't get any further here, I can take a whack at implementing 
> this.
>

Actually, I did, but I took some side roads in the meantime, and it's 
unlikely that I will have enough time to finish this before you, you're 
too fast for me 😉  So please go ahead.

bug#47431: Process Whois connection broken by remote peer.

[Lars Ingebrigtsen]

Gregory Heytings <gregory@heytings.org> writes:

> Actually, I did, but I took some side roads in the meantime, and it's
> unlikely that I will have enough time to finish this before you,
> you're too fast for me 😉  So please go ahead.

OK, I'll probably have a go at this (but it may not happen soonish for
me either).  I'll send a mail before starting so that we don't happen to
both implement this at the same time.

bug#47431: Process Whois connection broken by remote peer.

[Hongyi Zhao]

On Sat, Mar 27, 2021 at 2:30 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Hongyi Zhao <hongyi.zhao@gmail.com>
> > Date: Sat, 27 Mar 2021 09:39:20 +0800
> >
> > On Ubuntu 20.04, I'm using the self compiled git master version of Emacs.
> > It seems that the whois client in Emacs only works for some specific domain
> >  names, for example gnu.org and emacs.org, but for others like
> > facebook.com it just hangs and returns nothing. The following command
> > will trigger the error:
> >
> > "M-x RET whois RET facebook.com"
>
> It doesn't hang here (I'm not on Ubuntu, though).  It times out:
>
>   open-network-stream: make client process failed: Connection timed out, :name, Whois, :buffer, *Whois*, :host, rs.internic.net, :service, 43, :nowait, nil, :tls-parameters, nil, :coding, nil
>
> If I try a different server, it does work:
>
>   C-u M-x whois RET facebook.com RET whois.crsnic.net RET

Yep. I confirmed that you're absolutely right. But how can we change
the default whois server picked by the client, say, in Emacs's
initialization file?

>
> This returns immediately with a large *Whois* buffer.
>
> Perhaps some network expert could tell why we use by default a server
> that is less useful.



-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

bug#47431: Process Whois connection broken by remote peer.

[Hongyi Zhao]

On Sat, Mar 27, 2021 at 2:30 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Hongyi Zhao <hongyi.zhao@gmail.com>
> > Date: Sat, 27 Mar 2021 09:39:20 +0800
> >
> > On Ubuntu 20.04, I'm using the self compiled git master version of Emacs.
> > It seems that the whois client in Emacs only works for some specific domain
> >  names, for example gnu.org and emacs.org, but for others like
> > facebook.com it just hangs and returns nothing. The following command
> > will trigger the error:
> >
> > "M-x RET whois RET facebook.com"
>
> It doesn't hang here (I'm not on Ubuntu, though).  It times out:
>
>   open-network-stream: make client process failed: Connection timed out, :name, Whois, :buffer, *Whois*, :host, rs.internic.net, :service, 43, :nowait, nil, :tls-parameters, nil, :coding, nil

How to obtain the above debug information?

> If I try a different server, it does work:
>
>   C-u M-x whois RET facebook.com RET whois.crsnic.net RET
>
> This returns immediately with a large *Whois* buffer.
>
> Perhaps some network expert could tell why we use by default a server
> that is less useful.



-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

bug#47431: Process Whois connection broken by remote peer.

[Eli Zaretskii]

> From: Hongyi Zhao <hongyi.zhao@gmail.com>
> Date: Sat, 27 Mar 2021 17:57:31 +0800
> Cc: 47431@debbugs.gnu.org
> 
> > It doesn't hang here (I'm not on Ubuntu, though).  It times out:
> >
> >   open-network-stream: make client process failed: Connection timed out, :name, Whois, :buffer, *Whois*, :host, rs.internic.net, :service, 43, :nowait, nil, :tls-parameters, nil, :coding, nil
> 
> How to obtain the above debug information?

It's in *Messages*.

bug#47431: Process Whois connection broken by remote peer.

[Hongyi Zhao]

On Sat, Mar 27, 2021 at 6:33 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Hongyi Zhao <hongyi.zhao@gmail.com>
> > Date: Sat, 27 Mar 2021 17:57:31 +0800
> > Cc: 47431@debbugs.gnu.org
> >
> > > It doesn't hang here (I'm not on Ubuntu, though).  It times out:
> > >
> > >   open-network-stream: make client process failed: Connection timed out, :name, Whois, :buffer, *Whois*, :host, rs.internic.net, :service, 43, :nowait, nil, :tls-parameters, nil, :coding, nil
> >
> > How to obtain the above debug information?
>
> It's in *Messages*.

But for my case, I see the following in *Messages* buffer:

Eager macro-expansion failure: (wrong-type-argument listp [(first-item
. rest-items) (sp-get-list-items)])
For information about GNU Emacs and the GNU system, type C-h C-a.

Regards
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

bug#47431: Process Whois connection broken by remote peer.

[Eli Zaretskii]

> From: Hongyi Zhao <hongyi.zhao@gmail.com>
> Date: Sat, 27 Mar 2021 20:29:24 +0800
> Cc: 47431@debbugs.gnu.org
> 
> > > >   open-network-stream: make client process failed: Connection timed out, :name, Whois, :buffer, *Whois*, :host, rs.internic.net, :service, 43, :nowait, nil, :tls-parameters, nil, :coding, nil
> > >
> > > How to obtain the above debug information?
> >
> > It's in *Messages*.
> 
> But for my case, I see the following in *Messages* buffer:
> 
> Eager macro-expansion failure: (wrong-type-argument listp [(first-item
> . rest-items) (sp-get-list-items)])
> For information about GNU Emacs and the GNU system, type C-h C-a.

I think that's unrelated, so I don't understand why it hangs
indefinitely for you.  Are you sure you waited enough time before
giving up?

bug#47431: Process Whois connection broken by remote peer.

[Hongyi Zhao]

On Sat, Mar 27, 2021 at 9:44 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Hongyi Zhao <hongyi.zhao@gmail.com>
> > Date: Sat, 27 Mar 2021 20:29:24 +0800
> > Cc: 47431@debbugs.gnu.org
> >
> > > > >   open-network-stream: make client process failed: Connection timed out, :name, Whois, :buffer, *Whois*, :host, rs.internic.net, :service, 43, :nowait, nil, :tls-parameters, nil, :coding, nil
> > > >
> > > > How to obtain the above debug information?
> > >
> > > It's in *Messages*.
> >
> > But for my case, I see the following in *Messages* buffer:
> >
> > Eager macro-expansion failure: (wrong-type-argument listp [(first-item
> > . rest-items) (sp-get-list-items)])
> > For information about GNU Emacs and the GNU system, type C-h C-a.
>
> I think that's unrelated, so I don't understand why it hangs
> indefinitely for you.  Are you sure you waited enough time before
> giving up?

It will hang up there forever.

Regards
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

bug#47431: Process Whois connection broken by remote peer.

[Eli Zaretskii]

> From: Hongyi Zhao <hongyi.zhao@gmail.com>
> Date: Sun, 28 Mar 2021 09:41:20 +0800
> Cc: 47431@debbugs.gnu.org
> 
> > I think that's unrelated, so I don't understand why it hangs
> > indefinitely for you.  Are you sure you waited enough time before
> > giving up?
> 
> It will hang up there forever.

Maybe your network connection blocks some addresses or something?
Otherwise, how to explain that it doesn't block for me?

bug#47431: Process Whois connection broken by remote peer.

[Hongyi Zhao]

On Sun, Mar 28, 2021 at 2:38 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Hongyi Zhao <hongyi.zhao@gmail.com>
> > Date: Sun, 28 Mar 2021 09:41:20 +0800
> > Cc: 47431@debbugs.gnu.org
> >
> > > I think that's unrelated, so I don't understand why it hangs
> > > indefinitely for you.  Are you sure you waited enough time before
> > > giving up?
> >
> > It will hang up there forever.
>
> Maybe your network connection blocks some addresses or something?
> Otherwise, how to explain that it doesn't block for me?

I further debug this problem with tcpdump for a runing `emcas -Q`
process, as described below.

For `M-x whois RET gnu.org RET`, when the query succeeds, the
following info will be captured:

$ sudo tcpdump -i any port 43
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size
262144 bytes
15:00:29.061914 IP6 X10DAi.56918 >
whois.publicinterestregistry.net.whois: Flags [S], seq 1929893031, win
64440, options [mss 1432,sackOK,TS val 417951633 ecr 0,nop,wscale 7],
length 0
15:00:29.398867 IP6 whois.publicinterestregistry.net.whois >
X10DAi.56918: Flags [S.], seq 1816069595, ack 1929893032, win 1440,
options [mss 1379,nop,nop,TS val 2007125977 ecr 417951633,sackOK,eol],
length 0
15:00:29.398923 IP6 X10DAi.56918 >
whois.publicinterestregistry.net.whois: Flags [.], ack 1, win 64440,
options [nop,nop,TS val 417951970 ecr 2007125977], length 0
15:00:29.399038 IP6 X10DAi.56918 >
whois.publicinterestregistry.net.whois: Flags [P.], seq 1:10, ack 1,
win 64440, options [nop,nop,TS val 417951970 ecr 2007125977], length 9
15:00:30.297592 IP6 X10DAi.56918 >
whois.publicinterestregistry.net.whois: Flags [P.], seq 1:10, ack 1,
win 64440, options [nop,nop,TS val 417952869 ecr 2007125977], length 9
15:00:31.321603 IP6 X10DAi.56918 >
whois.publicinterestregistry.net.whois: Flags [P.], seq 1:10, ack 1,
win 64440, options [nop,nop,TS val 417953893 ecr 2007125977], length 9
15:00:31.339068 IP6 whois.publicinterestregistry.net.whois >
X10DAi.56918: Flags [P.], seq 1:1368, ack 10, win 2880, options
[nop,nop,TS val 2007127334 ecr 417952869], length 1367
15:00:31.339120 IP6 X10DAi.56918 >
whois.publicinterestregistry.net.whois: Flags [.], ack 1368, win
63073, options [nop,nop,TS val 417953910 ecr 2007127334], length 0

For `M-x whois RET baidu.com RET`, when the query fails, the following
info will be captured:

$ sudo tcpdump -i any port 43
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size
262144 bytes
15:05:24.433913 IP X10DAi.47186 > 198.41.3.79.whois: Flags [S], seq
1963456950, win 64240, options [mss 1460,sackOK,TS val 2174254150 ecr
0,nop,wscale 7], length 0
15:05:25.465604 IP X10DAi.47186 > 198.41.3.79.whois: Flags [S], seq
1963456950, win 64240, options [mss 1460,sackOK,TS val 2174255182 ecr
0,nop,wscale 7], length 0
15:05:27.481605 IP X10DAi.47186 > 198.41.3.79.whois: Flags [S], seq
1963456950, win 64240, options [mss 1460,sackOK,TS val 2174257198 ecr
0,nop,wscale 7], length 0

I also will see the following in *Messages* buffer:

For information about GNU Emacs and the GNU system, type C-h C-a.
open-network-stream: Failed connect: No route to host

Regards
-- 
Assoc. Prof. Hongyi Zhao <hongyi.zhao@gmail.com>
Theory and Simulation of Materials
Hebei Polytechnic University of Science and Technology engineering
NO. 552 North Gangtie Road, Xingtai, China

bug#47431: Process Whois connection broken by remote peer.

[Lars Ingebrigtsen]

Hongyi Zhao <hongyi.zhao@gmail.com> writes:

> open-network-stream: Failed connect: No route to host

I can reproduce this in Emacs 28...  but only intermittently.  Sometimes
it will hang completely, and sometimes I get the timeout.

This is the backtrace (with debug-on-quit) when it just hangs:

Debugger entered--Lisp error: (quit)
  make-network-process(:name "Whois" :buffer #<buffer *Whois*> :host "rs.internic.net" :service 43 :nowait nil :tls-parameters nil :coding nil)
  open-network-stream("Whois" #<buffer *Whois*> "rs.internic.net" 43)
  run-network-program("Whois" "rs.internic.net" 43 "facebook.com")
  whois(nil "facebook.com")

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47431: Process Whois connection broken by remote peer.

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  47431@debbugs.gnu.org
> Date: Sun, 28 Mar 2021 16:11:14 +0200
> 
> Hongyi Zhao <hongyi.zhao@gmail.com> writes:
> 
> > open-network-stream: Failed connect: No route to host
> 
> I can reproduce this in Emacs 28...  but only intermittently.  Sometimes
> it will hang completely, and sometimes I get the timeout.
> 
> This is the backtrace (with debug-on-quit) when it just hangs:
> 
> Debugger entered--Lisp error: (quit)
>   make-network-process(:name "Whois" :buffer #<buffer *Whois*> :host "rs.internic.net" :service 43 :nowait nil :tls-parameters nil :coding nil)
>   open-network-stream("Whois" #<buffer *Whois*> "rs.internic.net" 43)
>   run-network-program("Whois" "rs.internic.net" 43 "facebook.com")
>   whois(nil "facebook.com")

I think we should simply update the list of servers we use, it sounds
like what we have is outdated.  The default should be useful and
easily reachable.  But I'm not an expert on this stuff.

bug#47431: Process Whois connection broken by remote peer.

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I think we should simply update the list of servers we use, it sounds
> like what we have is outdated.  The default should be useful and
> easily reachable.  But I'm not an expert on this stuff.

Oh, sure; we should fix whois.el.  I was just wondering whether there
was something odd going on in our basic networking layer here.

I let the connect run for longer, and I do seem to reliably get timeouts
here.  It's just that they take a long time -- this one ran for about
five minutes before I got:

Debugger entered--Lisp error: (file-error "Failed connect" "Connection timed out")
  make-network-process(:name "Whois" :buffer #<buffer *scratch*> :host "rs.internic.net" :service 43 :nowait nil :tls-parameters nil :coding nil)

That is an excessive timeout...  but I guess this is up to the OS?  We
don't specify a timeout in the make-network-process function, I think?
(It's been a while since I've read it...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#47431: Process Whois connection broken by remote peer.

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> That is an excessive timeout...  but I guess this is up to the OS?  We
> don't specify a timeout in the make-network-process function, I think?
> (It's been a while since I've read it...)

(So whois.el should also probably be rewritten to use async network
connections and do a reasonable timeout.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no