bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

[Stefan Kangas]

I noticed that codespell was having trouble with one of our files
(ethio-util.el), refusing to read it as UTF-8.  This lead to some
unexpected and unwanted behavior.

So I took a closer look and found that several files in lisp/ appear to
be use some other encoding than UTF-8:

$ cd lisp ; for f in $(git ls-files|egrep '.el$'); \
    do file $f | grep -v UTF-8 | grep -v " ASCII" ; \
  done ; cd -
international/titdic-cnv.el: Lisp/Scheme program, Non-ISO
extended-ASCII text, with LF, NEL line terminators
language/ethio-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators
language/ethiopic.el: Lisp/Scheme program, Non-ISO extended-ASCII text
language/ind-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators
language/tibet-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators
language/tibetan.el: Non-ISO extended-ASCII text, with LF, NEL line terminators
leim/quail/ethiopic.el: Non-ISO extended-ASCII text, with LF, NEL line
terminators
leim/quail/tibetan.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators

Should these files be converted to UTF-8?

bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 17 Nov 2022 11:38:46 -0800
> 
> I noticed that codespell was having trouble with one of our files
> (ethio-util.el), refusing to read it as UTF-8.  This lead to some
> unexpected and unwanted behavior.
> 
> So I took a closer look and found that several files in lisp/ appear to
> be use some other encoding than UTF-8:
> 
> $ cd lisp ; for f in $(git ls-files|egrep '.el$'); \
>     do file $f | grep -v UTF-8 | grep -v " ASCII" ; \
>   done ; cd -
> international/titdic-cnv.el: Lisp/Scheme program, Non-ISO
> extended-ASCII text, with LF, NEL line terminators
> language/ethio-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
> language/ethiopic.el: Lisp/Scheme program, Non-ISO extended-ASCII text
> language/ind-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
> language/tibet-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
> language/tibetan.el: Non-ISO extended-ASCII text, with LF, NEL line terminators
> leim/quail/ethiopic.el: Non-ISO extended-ASCII text, with LF, NEL line
> terminators
> leim/quail/tibetan.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
> 
> Should these files be converted to UTF-8?

No.  AFAIR, they are in utf-8-emacs because they include characters
beyond the Unicode range, which UTF-8 cannot encode.  See, for
example, the codepoints that start around line 645 in ind-util.el,
which are used for converting between IS 13194 (ISCII) and Unicode.

bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> No.  AFAIR, they are in utf-8-emacs because they include characters
> beyond the Unicode range, which UTF-8 cannot encode.  See, for
> example, the codepoints that start around line 645 in ind-util.el,
> which are used for converting between IS 13194 (ISCII) and Unicode.

I see, thanks.

Do we need these characters to be raw bytes in the source code though?
I was thinking of a change similar to the below, which would
incidentally make it a bit easier to read the code.

diff --git a/lisp/language/ind-util.el b/lisp/language/ind-util.el
index e2a21820f4..16161319ef 100644
--- a/lisp/language/ind-util.el
+++ b/lisp/language/ind-util.el
@@ -644,9 +644,9 @@ indian-dev-aiba-decode-region
     ;;Unicode vs IS13194  ;; only Devanagari is supported now.
     ((ucs-devanagari-to-is13194-alist
       '((?\x0900 . "[U+0900]")
-	(?\x0901 . " ")
-	(?\x0902 . " ")
-	(?\x0903 . " ")
+        (?\x0901 . "?\x180000")
+        (?\x0902 . "?\x180001")
+        (?\x0903 . "?\x180002")
 	(?\x0904 . "[U+0904]")

[and so on]

This change would also avoid confusing external tools.  For example, the
code is completely unreadable in many external viewers, such as:

https://github.com/emacs-mirror/emacs/blob/master/lisp/language/ind-util.el#L647

bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 17 Nov 2022 20:14:09 -0800
> Cc: 59341@debbugs.gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > No.  AFAIR, they are in utf-8-emacs because they include characters
> > beyond the Unicode range, which UTF-8 cannot encode.  See, for
> > example, the codepoints that start around line 645 in ind-util.el,
> > which are used for converting between IS 13194 (ISCII) and Unicode.
> 
> I see, thanks.
> 
> Do we need these characters to be raw bytes in the source code though?

(They are not raw bytes, they are UTF-8 encoded sequences, except that
the encoding uses more bytes than the "official" UTF-8 allows.  IOW,
we have there encoded codepoints beyond u+0010FFFF, not raw bytes.)

Yes we need them to appear as characters, because then they will be
displayed as glyphs if you have a suitable font, and in the context
such as this one it is very important to _see_ the character, if only
understand the mapping and to detect mistakes in it.

However, I think we should add a note in the Commentary section about
these subtleties, so that whoever next is bothered about this will be
able to have their questions answered.

> This change would also avoid confusing external tools.  For example, the
> code is completely unreadable in many external viewers, such as:
> 
> https://github.com/emacs-mirror/emacs/blob/master/lisp/language/ind-util.el#L647

I'm not too bothered about this.  Emacs sources are best viewed with
Emacs, and there's no way around this.

bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> However, I think we should add a note in the Commentary section about
> these subtleties, so that whoever next is bothered about this will be
> able to have their questions answered.

How about adding something like this to the Commentary section of the
relevant files (i.e. the list I posted originally):

;; Note that this file contains non-Unicode characters, which are
;; needed for certain conversions.  See the discussion in Bug#59341.

bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Fri, 18 Nov 2022 03:11:15 -0800
> Cc: 59341@debbugs.gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > However, I think we should add a note in the Commentary section about
> > these subtleties, so that whoever next is bothered about this will be
> > able to have their questions answered.
> 
> How about adding something like this to the Commentary section of the
> relevant files (i.e. the list I posted originally):
> 
> ;; Note that this file contains non-Unicode characters, which are
> ;; needed for certain conversions.  See the discussion in Bug#59341.

I'd rather describe the reason in more detail there.

I can do this myself if you prefer.

bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> I'd rather describe the reason in more detail there.
>
> I can do this myself if you prefer.

Yes, that would be much appreciated.  Thank you.

bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Fri, 18 Nov 2022 18:14:50 +0100
> Cc: 59341@debbugs.gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'd rather describe the reason in more detail there.
> >
> > I can do this myself if you prefer.
> 
> Yes, that would be much appreciated.  Thank you.

Done, and closing the bug.

bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

[Drew Adams]

> How about adding something like this to the Commentary section of the
> relevant files (i.e. the list I posted originally):
> 
> ;; Note that this file contains non-Unicode characters, which are
> ;; needed for certain conversions.  See the discussion in Bug#59341.

Why send readers off to a bug thread?  Wouldn't it
be better to summarize the important info here?

bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

[Stefan Kangas]

Drew Adams <drew.adams@oracle.com> writes:

> > ;; Note that this file contains non-Unicode characters, which are
> > ;; needed for certain conversions.  See the discussion in Bug#59341.
>
> Why send readers off to a bug thread?  Wouldn't it
> be better to summarize the important info here?

Yes, but unlike Eli, I don't have many expert things to say here.
Luckily, Eli offered to write up something better.

bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?

[Drew Adams]

> However, I think we should add a note in the Commentary section about
> these subtleties, so that whoever next is bothered about this will be
> able to have their questions answered.

+1