Emacs Git make bootstrap failure - idna related

Emacs Git make bootstrap failure - idna related

[CHENG Gao]

make bootstrap error:

,----
| Eager macro-expansion failure: (void-variable idna-program)
| 
| In toplevel form:
| gnus/deuglify.el:221:1:Error: Symbol’s value as variable is void: idna-program
| make[3]: *** [gnus/deuglify.elc] Error 1
| make[2]: *** [compile-main] Error 2
| make[1]: *** [lisp] Error 2
| make: *** [bootstrap] Error 2
`----

I know Lars the BugSweeper is overhauling things. So this mainly serves
as FYI only. 

Re: Emacs Git make bootstrap failure - idna related

[Lars Ingebrigtsen]

CHENG Gao <chenggao@royau.me> writes:

> make bootstrap error:
>
> ,----
> | Eager macro-expansion failure: (void-variable idna-program)
> | 
> | In toplevel form:
> | gnus/deuglify.el:221:1:Error: Symbol’s value as variable is void: idna-program

Ah, this is probably because there's an old idna.el that Gnus is trying
to access, and the new one I wrote tonight.  That's probably not a good
thing, so I should rename the new one.

Anybody have an idea what the new library should be called?  IDNA is
probably short for something like Internationalized Domain Name...
algorithm?

Also, does anybody have an opinion on where Emacs should do IDNA
decoding?  Since any service can theoretically be located on an IDNA
name, I think the most likely place to do it is in
`open-network-stream'.  Then both

(open-network-stream ... "méxico.icom.museum" "imap")

and

`M-x eww http://méxico.icom.museum'

would work without doing much work in all the different libraries...

(Hm, url.el seems do be doing URL-encoding on the host name at present, so
that'll have to be fixed.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Emacs Git make bootstrap failure - idna related

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 28 Dec 2015 17:56:25 +0100
> Cc: emacs-devel@gnu.org
> 
> Anybody have an idea what the new library should be called?

eidna.el?

Re: Emacs Git make bootstrap failure - idna related

[Paul Eggert]

Lars Ingebrigtsen wrote:
> Anybody have an idea what the new library should be called?  IDNA is
> probably short for something like Internationalized Domain Name...
> algorithm?

How about ‘puny’?  That way, you’ll have names like puny-encode-string and 
puny-decode-string, which fits in with the common term “punycode” for this sort 
of thing.

Re: Emacs Git make bootstrap failure - idna related

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Lars Ingebrigtsen wrote:
>> Anybody have an idea what the new library should be called?  IDNA is
>> probably short for something like Internationalized Domain Name...
>> algorithm?
>
> How about ‘puny’?  That way, you’ll have names like puny-encode-string
> and puny-decode-string, which fits in with the common term “punycode”
> for this sort of thing.

Good idea.  I'll rename.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Emacs Git make bootstrap failure - idna related

[CHENG Gao]

Seems it's not directly related to your new addition idna.el.

Error message is
,----
| Eager macro-expansion failure: (void-variable idna-program)
| 
| In toplevel form:
| gnus/deuglify.el:221:1:Error: Symbol’s value as variable is void: idna-program
| make[3]: *** [gnus/deuglify.elc] Error 1
| make[2]: *** [compile-main] Error 2
| make[1]: *** [lisp] Error 2
| make: *** [bootstrap] Error 2
`----

In gnus-art.el line 1628:
,----
| (defvar idna-program)
`----

In gnus-art.el line 1632:
,----
| 				  (require 'idna)
`----
But I did not see Lars added this into gnus-art.el after new idna.el
added. So what's THIS idna?

I have no idea, but guess the problem should be in gnus-art.el or gnus-sum.el.

Re: Emacs Git make bootstrap failure - idna related

[Lars Ingebrigtsen]

CHENG Gao <chenggao@royau.me> writes:

> Seems it's not directly related to your new addition idna.el.

I think it probably is?  Gnus requires idna.el, but then it turns out
that idna-program isn't defined, and that makes the build fail.  I
think.

It should be fixed on the trunk now, anyway.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Emacs Git make bootstrap failure - idna related

[CHENG Gao]

*On Mon, 28 Dec 2015 18:43:00 +0100
* Also sprach Lars Ingebrigtsen <larsi@gnus.org>:

> CHENG Gao <chenggao@royau.me> writes:
>
>> Seems it's not directly related to your new addition idna.el.
>
> I think it probably is? Gnus requires idna.el, but then it turns out
> that idna-program isn't defined, and that makes the build fail. I
> think.
>
> It should be fixed on the trunk now, anyway.

Thank you Lars. I confirm it builds now.

Oops I figured out why.

Included in GNU Libidn there is a file idna.el written by Simon
Josefsson. Seems gnus-art.el and gnus-sum.el expects it for idna.

Re: Emacs Git make bootstrap failure - idna related

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Also, does anybody have an opinion on where Emacs should do IDNA
> decoding?  Since any service can theoretically be located on an IDNA
> name, I think the most likely place to do it is in
> `open-network-stream'.  Then both
>
> (open-network-stream ... "méxico.icom.museum" "imap")
>
> and
>
> `M-x eww http://méxico.icom.museum'
>
> would work without doing much work in all the different libraries...
>
> (Hm, url.el seems do be doing URL-encoding on the host name at present, so
> that'll have to be fixed.)

I've now made these changes, but haven't pushed them yet.  I'm now able
to visit that Mexican museum site with eww.  :-)

The changes are basically

 	(make-network-process :name name :buffer buffer
-			      :host host :service service
+			      :host (puny-encode-domain host) :service service

in the calls to `make-network-process' in network-stream.el.
`puny-encode-domain' is a NOP for all normal, non-IDNA domain names, so
the potential for breakage should be pretty small...

Does anybody object to this change, or have a suggestion as to how to do
this in a different way?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Use IDNA on all network connections

[Lars Ingebrigtsen]

I should probably have changed the subject

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I've now made these changes, but haven't pushed them yet.  I'm now able
> to visit that Mexican museum site with eww.  :-)
>
> The changes are basically
>
>  	(make-network-process :name name :buffer buffer
> -			      :host host :service service
> +			      :host (puny-encode-domain host) :service service
>
> in the calls to `make-network-process' in network-stream.el.
> `puny-encode-domain' is a NOP for all normal, non-IDNA domain names, so
> the potential for breakage should be pretty small...
>
> Does anybody object to this change, or have a suggestion as to how to do
> this in a different way?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Use IDNA on all network connections

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I should probably have changed the subject

I went ahead and committed these IDNA changes to the trunk, anyway.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Use IDNA on all network connections

[Lars Ingebrigtsen]

Hm.  Should we do something about the Unicode homographic situation in
relation to URLs now that we have IDNA?

Note, for instance, that this URL doesn't lead to Paypal:

http://www.pаypal.com/

It's no longer registered, though.  The registrars are "supposed" to
weed them out, but it was registered for a while (by somebody who wanted
to demonstrate the attack).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Use IDNA on all network connections

[Paul Eggert]

Lars Ingebrigtsen wrote:
> Should we do something about the Unicode homographic situation in
> relation to URLs now that we have IDNA?

Yes, that would be good. How about if we display a URL component in punycode if 
it doesn’t satisfy the Unicode Restriction Level 3 (Highly Restrictive)? The 
Unicode folks say this restriction level “will satisfy the vast majority of users”.

http://www.unicode.org/reports/tr39/#Restriction_Level_Detection

Here's a summary of what some popular browsers do:

http://www.chromium.org/developers/design-documents/idn-in-google-chrome

Re: Use IDNA on all network connections

[Lars Ingebrigtsen]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Lars Ingebrigtsen wrote:
>> Should we do something about the Unicode homographic situation in
>> relation to URLs now that we have IDNA?
>
> Yes, that would be good. How about if we display a URL component in
> punycode if it doesn’t satisfy the Unicode Restriction Level 3 (Highly
> Restrictive)? The Unicode folks say this restriction level “will
> satisfy the vast majority of users”.
>
> http://www.unicode.org/reports/tr39/#Restriction_Level_Detection

-----

3 Highly Restrictive 

 * All characters in each identifier must be from a single script, or from the
 combinations: 

 * Latin + Han + Hiragana + Katakana; 
 * Latin + Han + Bopomofo; or 
 * Latin + Han + Hangul 

 * No characters in the identifier can be outside of the Identifier Profile 

 Note that this level will satisfy the vast majority of users. 

-----

That seems like a sensible restriction.  But when Emacs detects an URL,
should it punify the URL (for display purposes) (if it doesn't satisfy
"Highly Restrictive"), or should it just ... do something when the user
tries to act on the URL?

> Here's a summary of what some popular browsers do:
>
> http://www.chromium.org/developers/design-documents/idn-in-google-chrome

For eww, I think Firefox' solution sounds nice: Display the punycode
version if the domain name doesn't satisfy "Highly Restrictive"...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Use IDNA on all network connections

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> 3 Highly Restrictive 
>
>  * All characters in each identifier must be from a single script, or from the
>  combinations: 
>
>  * Latin + Han + Hiragana + Katakana; 
>  * Latin + Han + Bopomofo; or 
>  * Latin + Han + Hangul 

I've now implemented this in puny.el, but...

>  * No characters in the identifier can be outside of the Identifier Profile 

I'm not sure I quite follow what they're saying here:

http://www.unicode.org/reports/tr31/#Table_Candidate_Characters_for_Inclusion_in_Identifiers

But do we have this table somewhere in Emacs already?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Use IDNA on all network connections

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 29 Dec 2015 12:43:25 +0100
> Cc: emacs-devel@gnu.org
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
> > 3 Highly Restrictive 
> >
> >  * All characters in each identifier must be from a single script, or from the
> >  combinations: 
> >
> >  * Latin + Han + Hiragana + Katakana; 
> >  * Latin + Han + Bopomofo; or 
> >  * Latin + Han + Hangul 
> 
> I've now implemented this in puny.el

Thanks.

> but...
> 
> >  * No characters in the identifier can be outside of the Identifier Profile 
> 
> I'm not sure I quite follow what they're saying here:
> 
> http://www.unicode.org/reports/tr31/#Table_Candidate_Characters_for_Inclusion_in_Identifiers

They are saying we should allow these characters in identifiers,
unless we have a very good reason not to.

> But do we have this table somewhere in Emacs already?  :-)

I don't think so.  However, the list of those characters is very
short, so having it as a list or a regexp is not a problem.

Re: Use IDNA on all network connections

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> http://www.unicode.org/reports/tr31/#Table_Candidate_Characters_for_Inclusion_in_Identifiers
>
> They are saying we should allow these characters in identifiers,
> unless we have a very good reason not to.

So they're saying that even if it's otherwise latin, a, say,
KATAKANA-HIRAGANA DOUBLE HYPHEN still doesn't take it out of the Highly
Restrictive class?  Makes sense, but zero width characters in domain
names?  

 Code Point  Glyph  Name  
 0027  '  APOSTROPHE  
 002D  -  HYPHEN-MINUS  
 002E  .  FULL STOP  
 003A  :  COLON  
 00B7  ·  MIDDLE DOT  
 058A  ֊  ARMENIAN HYPHEN  
 05F3  ׳  HEBREW PUNCTUATION GERESH  
 05F4  ״  HEBREW PUNCTUATION GERSHAYIM  
 0F0B  ་  TIBETAN MARK INTERSYLLABIC TSHEG  
 200C    ZERO WIDTH NON-JOINER*  
 200D    ZERO WIDTH JOINER*  
 2010  ‐  HYPHEN  
 2019  ’  RIGHT SINGLE QUOTATION MARK  
 2027  ‧  HYPHENATION POINT  
 30A0  ゠  KATAKANA-HIRAGANA DOUBLE HYPHEN  
 30FB  ・  KATAKANA MIDDLE DOT  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Use IDNA on all network connections

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: eggert@cs.ucla.edu,  emacs-devel@gnu.org
> Date: Tue, 29 Dec 2015 17:28:55 +0100
> 
> So they're saying that even if it's otherwise latin, a, say,
> KATAKANA-HIRAGANA DOUBLE HYPHEN still doesn't take it out of the Highly
> Restrictive class?

I think so, yes.

> but zero width characters in domain names?

They are needed to force or prevent character composition, I guess
that's the reason.  Emacs displays them as a thin space, so someone
with sharp enough eyes will be able to notice, I think.

Re: Use IDNA on all network connections

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Cc: eggert@cs.ucla.edu,  emacs-devel@gnu.org
>> Date: Tue, 29 Dec 2015 17:28:55 +0100
>> 
>> So they're saying that even if it's otherwise latin, a, say,
>> KATAKANA-HIRAGANA DOUBLE HYPHEN still doesn't take it out of the Highly
>> Restrictive class?
>
> I think so, yes.
>
>> but zero width characters in domain names?
>
> They are needed to force or prevent character composition, I guess
> that's the reason.  Emacs displays them as a thin space, so someone
> with sharp enough eyes will be able to notice, I think.

Okidoke.  I've now added that table to the predicate.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Use IDNA on all network connections

[Lars Magne Ingebrigtsen]

I've now made eww display non-restrictive domain names as punycode.

So when you `M-x eww http://www.pаypal.com/' you get a page saying

Loading http://www.xn--pypal-4ve.com/...

This is the same that most (all?) external browsers do these days, so
I'm not sure whether we need to adapt `browse-url' here in any way?

Anybody think of anything further we should be doing to protect the
users?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no