Garbled non-Ascii characters in ChangeLog files

Garbled non-Ascii characters in ChangeLog files

[Sven Joachim]

It seems that non-Ascii characters have been messed up in ChangeLog
files for a while.  For instance, the latest entry in lisp/ChangeLog in
EMACS_22_BASE looks like this:

,----
| 2008-03-25  Johan Bockg^[$(Q)[^[(Brd  <bojohan@gnu.org>
| 
| 	* info.el (Info-isearch-search): Always return point.
| 
`----

The trunk show similar garbage for Jan Djärv's changes in several files,
apparently starting after the merge of the unicode-2 branch.

Sven

Re: Garbled non-Ascii characters in ChangeLog files

[Andreas Schwab]

Sven Joachim <svenjoac@gmx.de> writes:

> It seems that non-Ascii characters have been messed up in ChangeLog
> files for a while.  For instance, the latest entry in lisp/ChangeLog in
> EMACS_22_BASE looks like this:
>
> ,----
> | 2008-03-25  Johan Bockg^[$(Q)[^[(Brd  <bojohan@gnu.org>
> | 
> | 	* info.el (Info-isearch-search): Always return point.
> | 
> `----
>
> The trunk show similar garbage for Jan Djärv's changes in several files,
> apparently starting after the merge of the unicode-2 branch.

That appears to be a bug in the iso-2022 decoder in Emacs 22.  Emacs 23
can decode this properly.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

iso-2022-7bit coding changed (Was: Garbled non-Ascii characters in ChangeLog files)

[Jan Djärv]

Sven Joachim skrev:
> It seems that non-Ascii characters have been messed up in ChangeLog
> files for a while.  For instance, the latest entry in lisp/ChangeLog in
> EMACS_22_BASE looks like this:
> 
> ,----
> | 2008-03-25  Johan Bockg^[$(Q)[^[(Brd  <bojohan@gnu.org>
> | 
> | 	* info.el (Info-isearch-search): Always return point.
> | 
> `----
> 
> The trunk show similar garbage for Jan Djärv's changes in several files,
> apparently starting after the merge of the unicode-2 branch.
> 

It seems to be some change in Emacs itself.  I use C-x 4 a to add a ChangeLog
entry, and in Emacs it looks OK.

The old coding för Djärv prior to the unicode merge is (\E => escape)

  Dj\E,Ad\E(Brv

but now it is

  Dj\E$(Q)Z\E(Brv

Both are displayed correctly in Emacs from HEAD, but not in Emacs 22 (or
earlier).  I don't know if this is a bug or an intentional change.

	Jan D.

Re: iso-2022-7bit coding changed (Was: Garbled non-Ascii characters in ChangeLog files)

[Juanma Barranquero]

2008/3/26 Jan Djärv <jan.h.d@swipnet.se>:

>  The old coding f�r Dj�rv prior to the unicode merge is (\E => escape)
>
>   Dj\E,Ad\E(Brv

  charset = latin-iso8859-1

>  but now it is
>
>   Dj\E$(Q)Z\E(Brv

  charset = japanese-jisx0213\.2004-1

according to Emacs 23's describe-char.

 Juanma

Re: Garbled non-Ascii characters in ChangeLog files

[Kenichi Handa]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=ISO-2022-JP-2, Size: 1010 bytes --]

In article <87abklyh19.fsf@gmx.de>, Sven Joachim <svenjoac@gmx.de> writes:

> It seems that non-Ascii characters have been messed up in ChangeLog
> files for a while.  For instance, the latest entry in lisp/ChangeLog in
> EMACS_22_BASE looks like this:

> ,----
> | 2008-03-25  Johan Bockg^[$(Q)[^[(Brd  <bojohan@gnu.org>
> | 
> | 	* info.el (Info-isearch-search): Always return point.
> | 
> `----

I've just fixed that line for EMACS_22_BASE.

> The trunk show similar garbage for Jan Dj^[$(D+#^[(Brv's changes in several files,
> apparently starting after the merge of the unicode-2 branch.

I see no garbage for Jan's names in the ChangeLog files of
the trunk.

By the way, ChangeLog files use iso-2022-7bit that uses
ESC-sequence for encoding legacy charsets, and to which
legacy charset to map characters depends on the priorities
of charsets (and thus depends on one's locale).  Perhaps we
should now use utf-8 for ChangeLog files to keep constant
encoding result.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Garbled non-Ascii characters in ChangeLog files

[Juanma Barranquero]

2008/3/26 Kenichi Handa <handa@m17n.org>:

>  I see no garbage for Jan's names in the ChangeLog files of
>  the trunk.

Not even editing the trunk's lips/ChangeLog with the EMACS_22_BASE binary?

>  Perhaps we
>  should now use utf-8 for ChangeLog files to keep constant
>  encoding result.

Yes, please.

 Juanma

Re: Garbled non-Ascii characters in ChangeLog files

[Stefan Monnier]

> Perhaps we should now use utf-8 for ChangeLog files to keep constant
> encoding result.

Yes, please,


        Stefan

Re: Garbled non-Ascii characters in ChangeLog files

[Kenichi Handa]

In article <jezlslkdbs.fsf@sykes.suse.de>, Andreas Schwab <schwab@suse.de> writes:

> Sven Joachim <svenjoac@gmx.de> writes:
> > It seems that non-Ascii characters have been messed up in ChangeLog
> > files for a while.  For instance, the latest entry in lisp/ChangeLog in
> > EMACS_22_BASE looks like this:
> >
> > ,----
> > | 2008-03-25  Johan Bockg^[$(Q)[^[(Brd  <bojohan@gnu.org>
> > | 
> > | 	* info.el (Info-isearch-search): Always return point.
> > | 
> > `----
> >
> > The trunk show similar garbage for Jan Djärv's changes in several files,
> > apparently starting after the merge of the unicode-2 branch.

> That appears to be a bug in the iso-2022 decoder in Emacs 22.  Emacs 23
> can decode this properly.

"ESC $ ( Q" is for a charset newly introduced in Emacs 23.
So, Emacs 22 doesn't know how to decode it, and as Emacs 22
has a limitation on the number of charsets, there's no way
to support all of such escape sequence.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Garbled non-Ascii characters in ChangeLog files

[Eli Zaretskii]

> From: Kenichi Handa <handa@ni.aist.go.jp>
> Date: Thu, 27 Mar 2008 08:10:48 +0900
> Cc: svenjoac@gmx.de, emacs-devel@gnu.org
> 
> "ESC $ ( Q" is for a charset newly introduced in Emacs 23.
> So, Emacs 22 doesn't know how to decode it, and as Emacs 22
> has a limitation on the number of charsets, there's no way
> to support all of such escape sequence.

This means people should not edit Emacs 22.x ChangeLog files with
Emacs 23.  I don't know if we can expect people to remember that.

Re: Garbled non-Ascii characters in ChangeLog files

[Kenichi Handa]

In article <uabkkg5r5.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > From: Kenichi Handa <handa@ni.aist.go.jp>
> > Date: Thu, 27 Mar 2008 08:10:48 +0900
> > Cc: svenjoac@gmx.de, emacs-devel@gnu.org
> > 
> > "ESC $ ( Q" is for a charset newly introduced in Emacs 23.
> > So, Emacs 22 doesn't know how to decode it, and as Emacs 22
> > has a limitation on the number of charsets, there's no way
> > to support all of such escape sequence.

> This means people should not edit Emacs 22.x ChangeLog files with
> Emacs 23.  I don't know if we can expect people to remember that.

I'm going to modify the code to have the more reasonable
charset priority.  After that change, in most cases,
iso-2022-7bit maps characters to the same charsets, or at
least, to the charsets supported in Emacs 22.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Garbled non-Ascii characters in ChangeLog files

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Date: Thu, 27 Mar 2008 16:26:09 +0900
> Cc: schwab@suse.de, svenjoac@gmx.de, emacs-devel@gnu.org
> 
> I'm going to modify the code to have the more reasonable
> charset priority.  After that change, in most cases,
> iso-2022-7bit maps characters to the same charsets, or at
> least, to the charsets supported in Emacs 22.

Thank you.

Re: Garbled non-Ascii characters in ChangeLog files

[Kenichi Handa]

In article <jwvbq51wn5c.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > Perhaps we should now use utf-8 for ChangeLog files to keep constant
> > encoding result.

> Yes, please,

Ok, done.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Garbled non-Ascii characters in ChangeLog files

[Kenichi Handa]

In article <u8x04ey9s.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > From: Kenichi Handa <handa@m17n.org>
> > Date: Thu, 27 Mar 2008 16:26:09 +0900
> > Cc: schwab@suse.de, svenjoac@gmx.de, emacs-devel@gnu.org
> > 
> > I'm going to modify the code to have the more reasonable
> > charset priority.  After that change, in most cases,
> > iso-2022-7bit maps characters to the same charsets, or at
> > least, to the charsets supported in Emacs 22.

> Thank you.

I committed that change.

---
Kenichi Handa
handa@ni.aist.go.jp