unofficial mirror of bug-gnu-emacs@gnu.org 
 help / color / mirror / code / Atom feed
* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
@ 2022-11-17 19:38 Stefan Kangas
  2022-11-17 19:54 ` Eli Zaretskii
  0 siblings, 1 reply; 11+ messages in thread
From: Stefan Kangas @ 2022-11-17 19:38 UTC (permalink / raw)
  To: 59341

I noticed that codespell was having trouble with one of our files
(ethio-util.el), refusing to read it as UTF-8.  This lead to some
unexpected and unwanted behavior.

So I took a closer look and found that several files in lisp/ appear to
be use some other encoding than UTF-8:

$ cd lisp ; for f in $(git ls-files|egrep '.el$'); \
    do file $f | grep -v UTF-8 | grep -v " ASCII" ; \
  done ; cd -
international/titdic-cnv.el: Lisp/Scheme program, Non-ISO
extended-ASCII text, with LF, NEL line terminators
language/ethio-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators
language/ethiopic.el: Lisp/Scheme program, Non-ISO extended-ASCII text
language/ind-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators
language/tibet-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators
language/tibetan.el: Non-ISO extended-ASCII text, with LF, NEL line terminators
leim/quail/ethiopic.el: Non-ISO extended-ASCII text, with LF, NEL line
terminators
leim/quail/tibetan.el: Lisp/Scheme program, Non-ISO extended-ASCII
text, with LF, NEL line terminators

Should these files be converted to UTF-8?





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
  2022-11-17 19:38 bug#59341: 29.0.50; Lisp files with other encoding than UTF-8? Stefan Kangas
@ 2022-11-17 19:54 ` Eli Zaretskii
  2022-11-18  4:14   ` Stefan Kangas
  0 siblings, 1 reply; 11+ messages in thread
From: Eli Zaretskii @ 2022-11-17 19:54 UTC (permalink / raw)
  To: Stefan Kangas; +Cc: 59341

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 17 Nov 2022 11:38:46 -0800
> 
> I noticed that codespell was having trouble with one of our files
> (ethio-util.el), refusing to read it as UTF-8.  This lead to some
> unexpected and unwanted behavior.
> 
> So I took a closer look and found that several files in lisp/ appear to
> be use some other encoding than UTF-8:
> 
> $ cd lisp ; for f in $(git ls-files|egrep '.el$'); \
>     do file $f | grep -v UTF-8 | grep -v " ASCII" ; \
>   done ; cd -
> international/titdic-cnv.el: Lisp/Scheme program, Non-ISO
> extended-ASCII text, with LF, NEL line terminators
> language/ethio-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
> language/ethiopic.el: Lisp/Scheme program, Non-ISO extended-ASCII text
> language/ind-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
> language/tibet-util.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
> language/tibetan.el: Non-ISO extended-ASCII text, with LF, NEL line terminators
> leim/quail/ethiopic.el: Non-ISO extended-ASCII text, with LF, NEL line
> terminators
> leim/quail/tibetan.el: Lisp/Scheme program, Non-ISO extended-ASCII
> text, with LF, NEL line terminators
> 
> Should these files be converted to UTF-8?

No.  AFAIR, they are in utf-8-emacs because they include characters
beyond the Unicode range, which UTF-8 cannot encode.  See, for
example, the codepoints that start around line 645 in ind-util.el,
which are used for converting between IS 13194 (ISCII) and Unicode.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
  2022-11-17 19:54 ` Eli Zaretskii
@ 2022-11-18  4:14   ` Stefan Kangas
  2022-11-18  7:57     ` Eli Zaretskii
  0 siblings, 1 reply; 11+ messages in thread
From: Stefan Kangas @ 2022-11-18  4:14 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 59341

Eli Zaretskii <eliz@gnu.org> writes:

> No.  AFAIR, they are in utf-8-emacs because they include characters
> beyond the Unicode range, which UTF-8 cannot encode.  See, for
> example, the codepoints that start around line 645 in ind-util.el,
> which are used for converting between IS 13194 (ISCII) and Unicode.

I see, thanks.

Do we need these characters to be raw bytes in the source code though?
I was thinking of a change similar to the below, which would
incidentally make it a bit easier to read the code.

diff --git a/lisp/language/ind-util.el b/lisp/language/ind-util.el
index e2a21820f4..16161319ef 100644
--- a/lisp/language/ind-util.el
+++ b/lisp/language/ind-util.el
@@ -644,9 +644,9 @@ indian-dev-aiba-decode-region
     ;;Unicode vs IS13194  ;; only Devanagari is supported now.
     ((ucs-devanagari-to-is13194-alist
       '((?\x0900 . "[U+0900]")
-	(?\x0901 . " ")
-	(?\x0902 . " ")
-	(?\x0903 . " ")
+        (?\x0901 . "?\x180000")
+        (?\x0902 . "?\x180001")
+        (?\x0903 . "?\x180002")
 	(?\x0904 . "[U+0904]")

[and so on]

This change would also avoid confusing external tools.  For example, the
code is completely unreadable in many external viewers, such as:

https://github.com/emacs-mirror/emacs/blob/master/lisp/language/ind-util.el#L647





^ permalink raw reply related	[flat|nested] 11+ messages in thread

* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
  2022-11-18  4:14   ` Stefan Kangas
@ 2022-11-18  7:57     ` Eli Zaretskii
  2022-11-18 11:11       ` Stefan Kangas
  2022-11-18 16:49       ` Drew Adams
  0 siblings, 2 replies; 11+ messages in thread
From: Eli Zaretskii @ 2022-11-18  7:57 UTC (permalink / raw)
  To: Stefan Kangas; +Cc: 59341

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 17 Nov 2022 20:14:09 -0800
> Cc: 59341@debbugs.gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > No.  AFAIR, they are in utf-8-emacs because they include characters
> > beyond the Unicode range, which UTF-8 cannot encode.  See, for
> > example, the codepoints that start around line 645 in ind-util.el,
> > which are used for converting between IS 13194 (ISCII) and Unicode.
> 
> I see, thanks.
> 
> Do we need these characters to be raw bytes in the source code though?

(They are not raw bytes, they are UTF-8 encoded sequences, except that
the encoding uses more bytes than the "official" UTF-8 allows.  IOW,
we have there encoded codepoints beyond u+0010FFFF, not raw bytes.)

Yes we need them to appear as characters, because then they will be
displayed as glyphs if you have a suitable font, and in the context
such as this one it is very important to _see_ the character, if only
understand the mapping and to detect mistakes in it.

However, I think we should add a note in the Commentary section about
these subtleties, so that whoever next is bothered about this will be
able to have their questions answered.

> This change would also avoid confusing external tools.  For example, the
> code is completely unreadable in many external viewers, such as:
> 
> https://github.com/emacs-mirror/emacs/blob/master/lisp/language/ind-util.el#L647

I'm not too bothered about this.  Emacs sources are best viewed with
Emacs, and there's no way around this.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
  2022-11-18  7:57     ` Eli Zaretskii
@ 2022-11-18 11:11       ` Stefan Kangas
  2022-11-18 11:59         ` Eli Zaretskii
  2022-11-18 16:53         ` Drew Adams
  2022-11-18 16:49       ` Drew Adams
  1 sibling, 2 replies; 11+ messages in thread
From: Stefan Kangas @ 2022-11-18 11:11 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 59341

Eli Zaretskii <eliz@gnu.org> writes:

> However, I think we should add a note in the Commentary section about
> these subtleties, so that whoever next is bothered about this will be
> able to have their questions answered.

How about adding something like this to the Commentary section of the
relevant files (i.e. the list I posted originally):

;; Note that this file contains non-Unicode characters, which are
;; needed for certain conversions.  See the discussion in Bug#59341.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
  2022-11-18 11:11       ` Stefan Kangas
@ 2022-11-18 11:59         ` Eli Zaretskii
  2022-11-18 17:14           ` Stefan Kangas
  2022-11-18 16:53         ` Drew Adams
  1 sibling, 1 reply; 11+ messages in thread
From: Eli Zaretskii @ 2022-11-18 11:59 UTC (permalink / raw)
  To: Stefan Kangas; +Cc: 59341

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Fri, 18 Nov 2022 03:11:15 -0800
> Cc: 59341@debbugs.gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > However, I think we should add a note in the Commentary section about
> > these subtleties, so that whoever next is bothered about this will be
> > able to have their questions answered.
> 
> How about adding something like this to the Commentary section of the
> relevant files (i.e. the list I posted originally):
> 
> ;; Note that this file contains non-Unicode characters, which are
> ;; needed for certain conversions.  See the discussion in Bug#59341.

I'd rather describe the reason in more detail there.

I can do this myself if you prefer.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
  2022-11-18  7:57     ` Eli Zaretskii
  2022-11-18 11:11       ` Stefan Kangas
@ 2022-11-18 16:49       ` Drew Adams
  1 sibling, 0 replies; 11+ messages in thread
From: Drew Adams @ 2022-11-18 16:49 UTC (permalink / raw)
  To: Eli Zaretskii, Stefan Kangas; +Cc: 59341@debbugs.gnu.org

> However, I think we should add a note in the Commentary section about
> these subtleties, so that whoever next is bothered about this will be
> able to have their questions answered.

+1





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
  2022-11-18 11:11       ` Stefan Kangas
  2022-11-18 11:59         ` Eli Zaretskii
@ 2022-11-18 16:53         ` Drew Adams
  2022-11-18 17:16           ` Stefan Kangas
  1 sibling, 1 reply; 11+ messages in thread
From: Drew Adams @ 2022-11-18 16:53 UTC (permalink / raw)
  To: Stefan Kangas, Eli Zaretskii; +Cc: 59341@debbugs.gnu.org

> How about adding something like this to the Commentary section of the
> relevant files (i.e. the list I posted originally):
> 
> ;; Note that this file contains non-Unicode characters, which are
> ;; needed for certain conversions.  See the discussion in Bug#59341.

Why send readers off to a bug thread?  Wouldn't it
be better to summarize the important info here?

^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
  2022-11-18 11:59         ` Eli Zaretskii
@ 2022-11-18 17:14           ` Stefan Kangas
  2022-11-19  9:26             ` Eli Zaretskii
  0 siblings, 1 reply; 11+ messages in thread
From: Stefan Kangas @ 2022-11-18 17:14 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: 59341

Eli Zaretskii <eliz@gnu.org> writes:

> I'd rather describe the reason in more detail there.
>
> I can do this myself if you prefer.

Yes, that would be much appreciated.  Thank you.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
  2022-11-18 16:53         ` Drew Adams
@ 2022-11-18 17:16           ` Stefan Kangas
  0 siblings, 0 replies; 11+ messages in thread
From: Stefan Kangas @ 2022-11-18 17:16 UTC (permalink / raw)
  To: Drew Adams; +Cc: Eli Zaretskii, 59341@debbugs.gnu.org

Drew Adams <drew.adams@oracle.com> writes:

> > ;; Note that this file contains non-Unicode characters, which are
> > ;; needed for certain conversions.  See the discussion in Bug#59341.
>
> Why send readers off to a bug thread?  Wouldn't it
> be better to summarize the important info here?

Yes, but unlike Eli, I don't have many expert things to say here.
Luckily, Eli offered to write up something better.





^ permalink raw reply	[flat|nested] 11+ messages in thread

* bug#59341: 29.0.50; Lisp files with other encoding than UTF-8?
  2022-11-18 17:14           ` Stefan Kangas
@ 2022-11-19  9:26             ` Eli Zaretskii
  0 siblings, 0 replies; 11+ messages in thread
From: Eli Zaretskii @ 2022-11-19  9:26 UTC (permalink / raw)
  To: Stefan Kangas; +Cc: 59341-done

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Fri, 18 Nov 2022 18:14:50 +0100
> Cc: 59341@debbugs.gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'd rather describe the reason in more detail there.
> >
> > I can do this myself if you prefer.
> 
> Yes, that would be much appreciated.  Thank you.

Done, and closing the bug.





^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2022-11-19  9:26 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-11-17 19:38 bug#59341: 29.0.50; Lisp files with other encoding than UTF-8? Stefan Kangas
2022-11-17 19:54 ` Eli Zaretskii
2022-11-18  4:14   ` Stefan Kangas
2022-11-18  7:57     ` Eli Zaretskii
2022-11-18 11:11       ` Stefan Kangas
2022-11-18 11:59         ` Eli Zaretskii
2022-11-18 17:14           ` Stefan Kangas
2022-11-19  9:26             ` Eli Zaretskii
2022-11-18 16:53         ` Drew Adams
2022-11-18 17:16           ` Stefan Kangas
2022-11-18 16:49       ` Drew Adams

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).