admin/unidata updates

admin/unidata updates

[Glenn Morris]

I updated the admin/unidata data files to the latest versions.
Emacs builds fine for me afterwards; but I have no idea if more changes
are needed in light of the data changes...

Re: admin/unidata updates

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Sat, 21 Jun 2014 18:10:34 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>,
> 	YAMAMOTO Mitsuharu <mituharu@math.s.chiba-u.ac.jp>,
> 	Kenichi Handa <handa@m17n.org>
> 
> 
> I updated the admin/unidata data files to the latest versions.

But why on the trunk? why not on the release branch?  It makes little
sense to postpone Unicode 7.0 compatibility for another year or two.

Re: admin/unidata updates

[Glenn Morris]

Eli Zaretskii wrote:

>> I updated the admin/unidata data files to the latest versions.
>
> But why on the trunk? why not on the release branch?  It makes little
> sense to postpone Unicode 7.0 compatibility for another year or two.

I have no idea if it is a safe change or not.
If you think it is, then by all means it seems like it could be
backported.

Re: admin/unidata updates

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: emacs-devel@gnu.org,  mituharu@math.s.chiba-u.ac.jp,  handa@m17n.org
> Date: Sun, 22 Jun 2014 19:59:09 -0400
> 
> Eli Zaretskii wrote:
> 
> >> I updated the admin/unidata data files to the latest versions.
> >
> > But why on the trunk? why not on the release branch?  It makes little
> > sense to postpone Unicode 7.0 compatibility for another year or two.
> 
> I have no idea if it is a safe change or not.

I see no reason why it would be unsafe.

> If you think it is, then by all means it seems like it could be
> backported.

Yes, I think we should.

Re: admin/unidata updates

[Glenn Morris]

Eli Zaretskii wrote:

>> I have no idea if it is a safe change or not.
>
> I see no reason why it would be unsafe.

All I know is, the last time I updated BidiMirroring.txt, it caused some
issues that were not immediately obvious. (I know you put some
safeguards into effect for that.)

Re: admin/unidata updates

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: emacs-devel@gnu.org,  mituharu@math.s.chiba-u.ac.jp,  handa@m17n.org
> Date: Mon, 23 Jun 2014 02:08:38 -0400
> 
> Eli Zaretskii wrote:
> 
> >> I have no idea if it is a safe change or not.
> >
> > I see no reason why it would be unsafe.
> 
> All I know is, the last time I updated BidiMirroring.txt, it caused some
> issues that were not immediately obvious. (I know you put some
> safeguards into effect for that.)

(It was UnicodeData.txt whose changes caused trouble, AFAIR.)

I said "I see no reason" because I've reviewed the diffs and saw
nothing that could be a problem, just addition of characters.

But I agree that we should be better safe than sorry.  So I suggest to
wait for a couple of weeks, and if there are no complaints from trunk
users, backport this to the branch.

Re: admin/unidata updates

[Eli Zaretskii]

> Date: Mon, 23 Jun 2014 17:55:00 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: handa@m17n.org, mituharu@math.s.chiba-u.ac.jp, emacs-devel@gnu.org
> 
> But I agree that we should be better safe than sorry.  So I suggest to
> wait for a couple of weeks, and if there are no complaints from trunk
> users, backport this to the branch.

In addition, I updated a few data bases we maintain that are based on
Unicode, but are not generated automatically (perhaps they should be).
These should also be backported.

(Is there any good place to record what needs to be done when a new
Unicode standard is released?  I thought admin/notes/unicode, but it
talks about very different issues.)

And I have a couple of questions for Handa-san:

  . In otf-script-alist, some scripts have the dashes in their names
    replaced with an underscore, but others don't.  Is there some
    subtle issue here?  Do we need to use underscores instead of
    dashes exclusively (e.g., because XLFD specs disallow dashes in
    font specs)?

  . Should every script we support have a representative character in
    script-representative-chars?  What happens with a script that
    doesn't?

  . setup-default-fontset has a section labeled "for simple scripts".
    What is a "simple script" in this context?  If additional scripts
    are added to the database in characters.el, should they also be
    added to the list in setup-default-fontset?  What happens with a
    script that is not mentioned there?

TIA

Re: admin/unidata updates

[Glenn Morris]

Eli Zaretskii wrote:

>> But I agree that we should be better safe than sorry.  So I suggest to
>> wait for a couple of weeks, and if there are no complaints from trunk
>> users, backport this to the branch.

If we are considering doing it, we should do it right now, so it gets as
much testing as possible. (Then revert it in emacs-24 if it goes wrong.)

> In addition, I updated a few data bases we maintain that are based on
> Unicode, but are not generated automatically (perhaps they should be).

Everything that can be auto-generated should be! :)

Re: admin/unidata updates

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: handa@gnu.org,  mituharu@math.s.chiba-u.ac.jp,  emacs-devel@gnu.org
> Date: Mon, 23 Jun 2014 12:04:25 -0400
> 
> Eli Zaretskii wrote:
> 
> >> But I agree that we should be better safe than sorry.  So I suggest to
> >> wait for a couple of weeks, and if there are no complaints from trunk
> >> users, backport this to the branch.
> 
> If we are considering doing it, we should do it right now, so it gets as
> much testing as possible. (Then revert it in emacs-24 if it goes wrong.)

OK, then let's.

Re: admin/unidata updates

[Stefan Monnier]

>> >> But I agree that we should be better safe than sorry.  So I suggest to
>> >> wait for a couple of weeks, and if there are no complaints from trunk
>> >> users, backport this to the branch.
>> If we are considering doing it, we should do it right now, so it gets as
>> much testing as possible. (Then revert it in emacs-24 if it goes wrong.)
> OK, then let's.

Agreed.


        Stefan

Re: admin/unidata updates

[Paul Eggert]

On 06/23/2014 08:37 AM, Eli Zaretskii wrote:
> (Is there any good place to record what needs to be done when a new
> Unicode standard is released?  I thought admin/notes/unicode, but it
> talks about very different issues.)

admin/notes/unicode would be a reasonable place to record it. That file 
already talks about two quite-different issues -- namely, (1) a todo 
list for unicode-related issues, and (2) a list of source files that are 
not encoded in UTF-8.  It'd be OK if that file talked about this third 
issue as well.

Re: admin/unidata updates

[Eli Zaretskii]

> Date: Mon, 23 Jun 2014 12:54:22 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: emacs-devel@gnu.org
> 
> On 06/23/2014 08:37 AM, Eli Zaretskii wrote:
> > (Is there any good place to record what needs to be done when a new
> > Unicode standard is released?  I thought admin/notes/unicode, but it
> > talks about very different issues.)
> 
> admin/notes/unicode would be a reasonable place to record it. That file 
> already talks about two quite-different issues -- namely, (1) a todo 
> list for unicode-related issues, and (2) a list of source files that are 
> not encoded in UTF-8.  It'd be OK if that file talked about this third 
> issue as well.

OK, I did that on the emacs-24 branch.  (Thanks, Glenn, for
backporting the changes to Unicode tables.)

Re: admin/unidata updates

[K. Handa]

In article <83egyfsklj.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> And I have a couple of questions for Handa-san:

>   . In otf-script-alist, some scripts have the dashes in their names
>     replaced with an underscore, but others don't.  Is there some
>     subtle issue here?  Do we need to use underscores instead of
>     dashes exclusively (e.g., because XLFD specs disallow dashes in
>     font specs)?

I don't remember now why I used both "-" and "_" in script
names.  :-(
But, we don't have to avoid "-" here because XLFD anyway
can't contain a script name.

>   . Should every script we support have a representative character in
>     script-representative-chars?

That's better because that information may lead to a better
font selection.

>     What happens with a script that doesn't?

Then, even if a font-spec contains :script attribute, that
information can't be used to find a font for that font-spec.

>   . setup-default-fontset has a section labeled "for simple scripts".
>     What is a "simple script" in this context?

Such scripts that don't need CTL (Complex Text Layout)
processing, and thus don't require OTF fonts.

>     If additional scripts are added to the database in
>     characters.el, should they also be added to the list
>     in setup-default-fontset?

If we assign representative characters to those scripts, it
is better to added them to that list.

>     What happens with a script that is not mentioned
>     there?

Then a font for a character of that script is selected just
because that font has a glyph for that character.

---
Kenichi Handa
handa@gnu.org

Re: admin/unidata updates

[Eli Zaretskii]

> From: handa@gnu.org (K. Handa)
> Cc: rgm@gnu.org, mituharu@math.s.chiba-u.ac.jp, emacs-devel@gnu.org
> Date: Thu, 26 Jun 2014 00:41:54 +0900
> 
> In article <83egyfsklj.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And I have a couple of questions for Handa-san:
> 
> >   . In otf-script-alist, some scripts have the dashes in their names
> >     replaced with an underscore, but others don't.  Is there some
> >     subtle issue here?  Do we need to use underscores instead of
> >     dashes exclusively (e.g., because XLFD specs disallow dashes in
> >     font specs)?
> 
> I don't remember now why I used both "-" and "_" in script
> names.  :-(
> But, we don't have to avoid "-" here because XLFD anyway
> can't contain a script name.

OK, so I guess we can keep using the same names as in characters.el,
and see if something breaks one day.

> >   . setup-default-fontset has a section labeled "for simple scripts".
> >     What is a "simple script" in this context?
> 
> Such scripts that don't need CTL (Complex Text Layout)
> processing, and thus don't require OTF fonts.

How does one know whether a given script needs CTL?

The practical issue here is whether scripts added to characters.el
(because the Unicode Standard added them) should also be added to the
list of "simple scripts" in setup-default-fontset.

Thanks.

Re: admin/unidata updates

[K. Handa]

In article <834mz8rkix.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> How does one know whether a given script needs CTL?

If a script requires CTL, I think that the section of that
script in The Unicode Standard should contain some
information.

> The practical issue here is whether scripts added to characters.el
> (because the Unicode Standard added them) should also be added to the
> list of "simple scripts" in setup-default-fontset.

As I wrote, adding them without specifying representative
characters is useless.  But, as I have no knowledge about
recently added scripts, I can't provide such information.
So, let's leave the code of setup-default-fontset untouched.

---
Kenichi Handa
handa@gnu.org

Re: admin/unidata updates

[Eli Zaretskii]

> From: handa@gnu.org (K. Handa)
> Cc: rgm@gnu.org, mituharu@math.s.chiba-u.ac.jp, emacs-devel@gnu.org
> Date: Thu, 26 Jun 2014 23:03:48 +0900
> 
> In article <834mz8rkix.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How does one know whether a given script needs CTL?
> 
> If a script requires CTL, I think that the section of that
> script in The Unicode Standard should contain some
> information.
> 
> > The practical issue here is whether scripts added to characters.el
> > (because the Unicode Standard added them) should also be added to the
> > list of "simple scripts" in setup-default-fontset.
> 
> As I wrote, adding them without specifying representative
> characters is useless.  But, as I have no knowledge about
> recently added scripts, I can't provide such information.
> So, let's leave the code of setup-default-fontset untouched.

FWIW, I already added a representative character for each script I
added based on Unicode 7.0.

Re: admin/unidata updates

[K. Handa]

In article <83a98zptyc.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> FWIW, I already added a representative character for each script I
> added based on Unicode 7.0.

Ah, thank you.  But, how did you select them?

---
Kenichi Handa
handa@gnu.org

Re: admin/unidata updates

[Eli Zaretskii]

> From: handa@gnu.org (K. Handa)
> Cc: rgm@gnu.org, mituharu@math.s.chiba-u.ac.jp, emacs-devel@gnu.org
> Date: Sat, 28 Jun 2014 23:35:51 +0900
> 
> In article <83a98zptyc.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > FWIW, I already added a representative character for each script I
> > added based on Unicode 7.0.
> 
> Ah, thank you.  But, how did you select them?

By looking at the code charts on the Unicode Consortium site.  In
general, whenever one of the first codepoints of the script looked
something like "letter A", I used that.  Examples:

  10350;OLD PERMIC LETTER AN;Lo;0;L;;;;;N;;;;;
  10530;CAUCASIAN ALBANIAN LETTER ALT;Lo;0;L;;;;;N;;;;;
  11305;GRANTHA LETTER A;Lo;0;L;;;;;N;;;;;

Otherwise, I used the first codepoint that looked similar to most
others.

If there are some special requirements or guidelines for selecting the
representative characters, I'd surely want to know and document them.

Thanks.