* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
@ 2022-11-17 19:38 Stefan Kangas
2022-11-17 19:54 ` Eli Zaretskii
0 siblings, 1 reply; 11+ messages in thread
From: Stefan Kangas @ 2022-11-17 19:38 UTC (permalink / raw)
To: 59341
I noticed that codespell was having trouble with one of our files
(ethio-util.el), refusing to read it as UTF-8. This lead to some
unexpected and unwanted behavior.
So I took a closer look and found that several files in lisp/ appear to
be use some other encoding than UTF-8:
$ cd lisp ; for f in $(git ls-files|egrep '.el$'); \
do file $f | grep -v UTF-8 | grep -v " ASCII" ; \
done ; cd -
international/titdic-cnv.el: Lisp/Scheme program, Non-ISO
extended-ASCII text, with LF, NEL line terminators
language/ethio-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators
language/ethiopic.el: Lisp/Scheme program, Non-ISO extended-ASCII text
language/ind-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators
language/tibet-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators
language/tibetan.el: Non-ISO extended-ASCII text, with LF, NEL line terminators
leim/quail/ethiopic.el: Non-ISO extended-ASCII text, with LF, NEL line
terminators
leim/quail/tibetan.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators
Should these files be converted to UTF-8?
^ permalink raw reply [flat|nested] 11+ messages in thread* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
2022-11-17 19:38 bug#59341: 29.0.50; Lisp files with other encoding than UTF-8? Stefan Kangas
@ 2022-11-17 19:54 ` Eli Zaretskii
2022-11-18 4:14 ` Stefan Kangas
0 siblings, 1 reply; 11+ messages in thread
From: Eli Zaretskii @ 2022-11-17 19:54 UTC (permalink / raw)
To: Stefan Kangas; +Cc: 59341
> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 17 Nov 2022 11:38:46 -0800
>
> I noticed that codespell was having trouble with one of our files
> (ethio-util.el), refusing to read it as UTF-8. This lead to some
> unexpected and unwanted behavior.
>
> So I took a closer look and found that several files in lisp/ appear to
> be use some other encoding than UTF-8:
>
> $ cd lisp ; for f in $(git ls-files|egrep '.el$'); \
> do file $f | grep -v UTF-8 | grep -v " ASCII" ; \
> done ; cd -
> international/titdic-cnv.el: Lisp/Scheme program, Non-ISO
> extended-ASCII text, with LF, NEL line terminators
> language/ethio-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
> language/ethiopic.el: Lisp/Scheme program, Non-ISO extended-ASCII text
> language/ind-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
> language/tibet-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
> language/tibetan.el: Non-ISO extended-ASCII text, with LF, NEL line terminators
> leim/quail/ethiopic.el: Non-ISO extended-ASCII text, with LF, NEL line
> terminators
> leim/quail/tibetan.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
>
> Should these files be converted to UTF-8?
No. AFAIR, they are in utf-8-emacs because they include characters
beyond the Unicode range, which UTF-8 cannot encode. See, for
example, the codepoints that start around line 645 in ind-util.el,
which are used for converting between IS 13194 (ISCII) and Unicode.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
2022-11-17 19:54 ` Eli Zaretskii
@ 2022-11-18 4:14 ` Stefan Kangas
2022-11-18 7:57 ` Eli Zaretskii
0 siblings, 1 reply; 11+ messages in thread
From: Stefan Kangas @ 2022-11-18 4:14 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: 59341
Eli Zaretskii <eliz@gnu.org> writes:
> No. AFAIR, they are in utf-8-emacs because they include characters
> beyond the Unicode range, which UTF-8 cannot encode. See, for
> example, the codepoints that start around line 645 in ind-util.el,
> which are used for converting between IS 13194 (ISCII) and Unicode.
I see, thanks.
Do we need these characters to be raw bytes in the source code though?
I was thinking of a change similar to the below, which would
incidentally make it a bit easier to read the code.
diff --git a/lisp/language/ind-util.el b/lisp/language/ind-util.el
index e2a21820f4..16161319ef 100644
--- a/lisp/language/ind-util.el
+++ b/lisp/language/ind-util.el
@@ -644,9 +644,9 @@ indian-dev-aiba-decode-region
;;Unicode vs IS13194 ;; only Devanagari is supported now.
((ucs-devanagari-to-is13194-alist
'((?\x0900 . "[U+0900]")
- (?\x0901 . " ")
- (?\x0902 . " ")
- (?\x0903 . " ")
+ (?\x0901 . "?\x180000")
+ (?\x0902 . "?\x180001")
+ (?\x0903 . "?\x180002")
(?\x0904 . "[U+0904]")
[and so on]
This change would also avoid confusing external tools. For example, the
code is completely unreadable in many external viewers, such as:
https://github.com/emacs-mirror/emacs/blob/master/lisp/language/ind-util.el#L647
^ permalink raw reply related [flat|nested] 11+ messages in thread* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
2022-11-18 4:14 ` Stefan Kangas
@ 2022-11-18 7:57 ` Eli Zaretskii
2022-11-18 11:11 ` Stefan Kangas
2022-11-18 16:49 ` Drew Adams
0 siblings, 2 replies; 11+ messages in thread
From: Eli Zaretskii @ 2022-11-18 7:57 UTC (permalink / raw)
To: Stefan Kangas; +Cc: 59341
> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 17 Nov 2022 20:14:09 -0800
> Cc: 59341@debbugs.gnu.org
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> > No. AFAIR, they are in utf-8-emacs because they include characters
> > beyond the Unicode range, which UTF-8 cannot encode. See, for
> > example, the codepoints that start around line 645 in ind-util.el,
> > which are used for converting between IS 13194 (ISCII) and Unicode.
>
> I see, thanks.
>
> Do we need these characters to be raw bytes in the source code though?
(They are not raw bytes, they are UTF-8 encoded sequences, except that
the encoding uses more bytes than the "official" UTF-8 allows. IOW,
we have there encoded codepoints beyond u+0010FFFF, not raw bytes.)
Yes we need them to appear as characters, because then they will be
displayed as glyphs if you have a suitable font, and in the context
such as this one it is very important to _see_ the character, if only
understand the mapping and to detect mistakes in it.
However, I think we should add a note in the Commentary section about
these subtleties, so that whoever next is bothered about this will be
able to have their questions answered.
> This change would also avoid confusing external tools. For example, the
> code is completely unreadable in many external viewers, such as:
>
> https://github.com/emacs-mirror/emacs/blob/master/lisp/language/ind-util.el#L647
I'm not too bothered about this. Emacs sources are best viewed with
Emacs, and there's no way around this.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
2022-11-18 7:57 ` Eli Zaretskii
@ 2022-11-18 11:11 ` Stefan Kangas
2022-11-18 11:59 ` Eli Zaretskii
2022-11-18 16:53 ` Drew Adams
2022-11-18 16:49 ` Drew Adams
1 sibling, 2 replies; 11+ messages in thread
From: Stefan Kangas @ 2022-11-18 11:11 UTC (permalink / raw)
To: Eli Zaretskii; +Cc: 59341
Eli Zaretskii <eliz@gnu.org> writes:
> However, I think we should add a note in the Commentary section about
> these subtleties, so that whoever next is bothered about this will be
> able to have their questions answered.
How about adding something like this to the Commentary section of the
relevant files (i.e. the list I posted originally):
;; Note that this file contains non-Unicode characters, which are
;; needed for certain conversions. See the discussion in Bug#59341.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
2022-11-18 11:11 ` Stefan Kangas
@ 2022-11-18 11:59 ` Eli Zaretskii
2022-11-18 17:14 ` Stefan Kangas
2022-11-18 16:53 ` Drew Adams
1 sibling, 1 reply; 11+ messages in thread
From: Eli Zaretskii @ 2022-11-18 11:59 UTC (permalink / raw)
To: Stefan Kangas; +Cc: 59341
> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Fri, 18 Nov 2022 03:11:15 -0800
> Cc: 59341@debbugs.gnu.org
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> > However, I think we should add a note in the Commentary section about
> > these subtleties, so that whoever next is bothered about this will be
> > able to have their questions answered.
>
> How about adding something like this to the Commentary section of the
> relevant files (i.e. the list I posted originally):
>
> ;; Note that this file contains non-Unicode characters, which are
> ;; needed for certain conversions. See the discussion in Bug#59341.
I'd rather describe the reason in more detail there.
I can do this myself if you prefer.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
2022-11-18 11:11 ` Stefan Kangas
2022-11-18 11:59 ` Eli Zaretskii
@ 2022-11-18 16:53 ` Drew Adams
2022-11-18 17:16 ` Stefan Kangas
1 sibling, 1 reply; 11+ messages in thread
From: Drew Adams @ 2022-11-18 16:53 UTC (permalink / raw)
To: Stefan Kangas, Eli Zaretskii; +Cc: 59341@debbugs.gnu.org
> How about adding something like this to the Commentary section of the
> relevant files (i.e. the list I posted originally):
>
> ;; Note that this file contains non-Unicode characters, which are
> ;; needed for certain conversions. See the discussion in Bug#59341.
Why send readers off to a bug thread? Wouldn't it
be better to summarize the important info here?
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
2022-11-18 16:53 ` Drew Adams
@ 2022-11-18 17:16 ` Stefan Kangas
0 siblings, 0 replies; 11+ messages in thread
From: Stefan Kangas @ 2022-11-18 17:16 UTC (permalink / raw)
To: Drew Adams; +Cc: Eli Zaretskii, 59341@debbugs.gnu.org
Drew Adams <drew.adams@oracle.com> writes:
> > ;; Note that this file contains non-Unicode characters, which are
> > ;; needed for certain conversions. See the discussion in Bug#59341.
>
> Why send readers off to a bug thread? Wouldn't it
> be better to summarize the important info here?
Yes, but unlike Eli, I don't have many expert things to say here.
Luckily, Eli offered to write up something better.
^ permalink raw reply [flat|nested] 11+ messages in thread
* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
2022-11-18 7:57 ` Eli Zaretskii
2022-11-18 11:11 ` Stefan Kangas
@ 2022-11-18 16:49 ` Drew Adams
1 sibling, 0 replies; 11+ messages in thread
From: Drew Adams @ 2022-11-18 16:49 UTC (permalink / raw)
To: Eli Zaretskii, Stefan Kangas; +Cc: 59341@debbugs.gnu.org
> However, I think we should add a note in the Commentary section about
> these subtleties, so that whoever next is bothered about this will be
> able to have their questions answered.
+1
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-11-19 9:26 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-17 19:38 bug#59341: 29.0.50; Lisp files with other encoding than UTF-8? Stefan Kangas
2022-11-17 19:54 ` Eli Zaretskii
2022-11-18 4:14 ` Stefan Kangas
2022-11-18 7:57 ` Eli Zaretskii
2022-11-18 11:11 ` Stefan Kangas
2022-11-18 11:59 ` Eli Zaretskii
2022-11-18 17:14 ` Stefan Kangas
2022-11-19 9:26 ` Eli Zaretskii
2022-11-18 16:53 ` Drew Adams
2022-11-18 17:16 ` Stefan Kangas
2022-11-18 16:49 ` Drew Adams
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).