Generating charset map files during bootstrap

Generating charset map files during bootstrap

[Glenn Morris]

I'm interested in removing (at least some of) the etc/charset/*.map
files from the repository, and generating them during bootstrap (the
first time only, as is done eg for the unidata files). To that end:

1) Please could Someone(s) check that this works on MS Windows:

cd admin/charsets
make extraclean
make local

Ie, regenerates files in etc/charsets.
If it doesn't, please figure out what needs fixing, TIA very much. ;)

Note that the following charsets are regenerated with slight differences:
CNS-6.map, GB180302.map, GB180304.map

This is true in emacs-24 as well. I have no idea what the differences mean,
expect that presumably these files are out-of-sync with their sources?


2) We don't keep all the sources for the charset mapfiles in Emacs.
(Isn't this the kind of thing we have gotten in trouble for before?)
Some are in admin/charsets/mapfiles, but the rest are taken from
/usr/share/i18n/charmaps. Adding the relevant files to the Emacs repo
would add 103 (compressed) files to admin/charsets, totalling 2.2MB in
size (which is 10% less than the existing mapfiles/ directory). 2/103
account for 1MB of the size.

Shouldn't those sources be added?

Then all etc/charsets/*.map files could be removed (131 files, 3.1MB).
So a net win for a git snapshot (a loss for tarballs, which would
have both versions).

Re: Generating charset map files during bootstrap

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Fri, 22 May 2015 03:10:29 -0400
> 
> 
> I'm interested in removing (at least some of) the etc/charset/*.map
> files from the repository, and generating them during bootstrap (the
> first time only, as is done eg for the unidata files). To that end:
> 
> 1) Please could Someone(s) check that this works on MS Windows:
> 
> cd admin/charsets
> make extraclean
> make local

It's no-go here:

    GEN      ../../etc/charsets/ALTERNATIVNYJ.map
  make: *** No rule to make target `/usr/share/i18n/charmaps/GB18030.gz', needed by `../../etc/charsets/GB180302.map'.  Stop.

If I remove GB180304.map from the value of LOCAL, it goes on, and then
fails again:

    GEN      ../../lisp/international/cp51932.el
  make: *** No rule to make target `/usr/share/i18n/charmaps/EUC-JP-MS.gz', needed by `../../lisp/international/eucjp-ms.el'.  Stop.

at which point I stopped trying.

I assume the same failure will happen on any non-glibc Posix host as
well.

Maybe we should simply import that glibc directory, and use our own
copy?  It makes little sense to me to be able to generate only a part
of the maps, but not the others.

Re: Generating charset map files during bootstrap

[Glenn Morris]

Eli Zaretskii wrote:

>     GEN      ../../etc/charsets/ALTERNATIVNYJ.map
>   make: *** No rule to make target `/usr/share/i18n/charmaps/GB18030.gz', needed by `../../etc/charsets/GB180302.map'.  Stop.
>
> If I remove GB180304.map from the value of LOCAL, it goes on, and then
> fails again:
>
>     GEN      ../../lisp/international/cp51932.el
>   make: *** No rule to make target `/usr/share/i18n/charmaps/EUC-JP-MS.gz', needed by `../../lisp/international/eucjp-ms.el'.  Stop.
>
> at which point I stopped trying.

Blargh, I messed up the definition of LOCAL, which was supposed to be
the easy bit. Please could you update and try again?

> Maybe we should simply import that glibc directory, and use our own
> copy?

Well, that was my point 2).
I think so.

Re: Generating charset map files during bootstrap

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Fri, 22 May 2015 18:49:13 -0400
> 
> Blargh, I messed up the definition of LOCAL, which was supposed to be
> the easy bit. Please could you update and try again?

This works, but please note that "make extraclean" removes
lisp/international/eucjp-ms.el, which "make local" doesn't produce.

Also, is this supposed to work in an out-of-tree build?  (I never
build that way and don't care about it, but some do.)

> > Maybe we should simply import that glibc directory, and use our own
> > copy?
> 
> Well, that was my point 2).
> I think so.

I'd say go for it.

Thanks.

Re: Generating charset map files during bootstrap

[Glenn Morris]

Eli Zaretskii wrote:

> This works, but please note that "make extraclean" removes
> lisp/international/eucjp-ms.el, which "make local" doesn't produce.

Another goof, sigh. But now irrelevant, since I added the glibc sources.
Please could you do one last check: make extraclean, make all.

If that works I will add the remaining bootstrap glue and delete
etc/charsets/*.map and cp51932.el, eucjp-ms.el.

Note that there are changes in BIG5-HKSCS.map, GB180302.map, GB180304.map.
I assume these are due to changes in the source, but I have no way to tell.
There was a similar change that turned out to be due to a bug in the
mapconv script (missing LC_ALL). That gave a similar looking diff,
but caused a segfault (fortunately).

> Also, is this supposed to work in an out-of-tree build? 

Yes.

Re: Generating charset map files during bootstrap

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 23 May 2015 14:34:40 -0400
> 
> Please could you do one last check: make extraclean, make all.

Works.

> Note that there are changes in BIG5-HKSCS.map, GB180302.map, GB180304.map.
> I assume these are due to changes in the source, but I have no way to tell.

Looks like that, yes.

Thanks.