Coding system detection: Emacs 23 vs. 22

Coding system detection: Emacs 23 vs. 22

[Reiner Steib]

[-- Attachment #1: Type: text/plain, Size: 1471 bytes --]

Hi,

when opening the attached file (see attachment `ae.el.gz'), thee
coding system detection of Emacs 23 and 22 give different results:

- Emacs 22 (both, Emacs 21.1 and the current Emacs 22 branch) says
  emacs-mule-unix.  (The file was created with Emacs 22)

- Emacs 23, says iso-latin-1-unix

The real problem with this is that my Gnus score files of Emacs 22 no
longer work with Emacs 23 if the score entry (subject, author, ...)
contains a non-ascii character.


Expected behavior:

Emacs 23 should detect that the attached file is not iso-latin-1 (even
`file' detects this) but emacs-mule.


To reproduce:

Save attachment to /tmp/ae.el.gz

$ gunzip /tmp/ae.el.gz

$ file /tmp/ae.el 
/tmp/ae.el: Non-ISO extended-ASCII text
$ /usr/bin/emacs --batch -Q /tmp/ae.el --eval \
  '(message "buffer-file-coding-system=%s, %s" buffer-file-coding-system \
  emacs-version)' -kill
buffer-file-coding-system=emacs-mule-unix, 22.1.1
$ cvs-EMACS_22_BASE/i686/src/emacs --batch -Q /tmp/ae.el --eval \
  '(message "buffer-file-coding-system=%s, %s" buffer-file-coding-system \
  emacs-version)' -kill
buffer-file-coding-system=emacs-mule-unix, 22.1.92.2
$ cvs-HEAD/i686/src/emacs --batch -Q /tmp/ae.el --eval \
  '(message "buffer-file-coding-system=%s, %s" buffer-file-coding-system \
  emacs-version)' -kill
buffer-file-coding-system=iso-latin-1-unix, 23.0.60.2

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

[-- Attachment #2: ae.el.gz --]
[-- Type: application/x-gzip, Size: 27 bytes --]

Re: Coding system detection: Emacs 23 vs. 22

[Stefan Monnier]

> when opening the attached file (see attachment `ae.el.gz'), thee
> coding system detection of Emacs 23 and 22 give different results:

> - Emacs 22 (both, Emacs 21.1 and the current Emacs 22 branch) says
>   emacs-mule-unix.  (The file was created with Emacs 22)

> - Emacs 23, says iso-latin-1-unix

> The real problem with this is that my Gnus score files of Emacs 22 no
> longer work with Emacs 23 if the score entry (subject, author, ...)
> contains a non-ascii character.


> Expected behavior:

> Emacs 23 should detect that the attached file is not iso-latin-1 (even
> `file' detects this) but emacs-mule.

I suspect this is related to bug #16
http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=16

Handa?


        Stefan

Re: Coding system detection: Emacs 23 vs. 22

[Kenichi Handa]

In article <jwvejahgdsv.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > when opening the attached file (see attachment `ae.el.gz'), thee
> > coding system detection of Emacs 23 and 22 give different results:

> > - Emacs 22 (both, Emacs 21.1 and the current Emacs 22 branch) says
> >   emacs-mule-unix.  (The file was created with Emacs 22)

> > - Emacs 23, says iso-latin-1-unix

> > The real problem with this is that my Gnus score files of Emacs 22 no
> > longer work with Emacs 23 if the score entry (subject, author, ...)
> > contains a non-ascii character.

In Emacs 23, the charset definition for iso-8859-1 is taken
from glibc's locale data
(/usr/share/i18n/charmaps/ISO-8859-1.gz), and it contains
0x80..0x9F.  So, in the environment that prefer iso-latin-1,
most files are detected as iso-latin-1.

> > Expected behavior:

> > Emacs 23 should detect that the attached file is not iso-latin-1 (even
> > `file' detects this) but emacs-mule.

> I suspect this is related to bug #16
> http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=16

If gnus writes score files in emacs-mule, it must read that
file by explicitly specifying emacs-mule.  Or, it must write
score files with coding: tag.

The detection of emacs-mule is not 100% correct even in
Emacs 22.  If the problem didn't appear in Emacs 22, it is
just because of good luck.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Coding system detection: Emacs 23 vs. 22

[Stefan Monnier]

>> > Emacs 23 should detect that the attached file is not iso-latin-1 (even
>> > `file' detects this) but emacs-mule.

>> I suspect this is related to bug #16
>> http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=16

Any comment on this bug report?

> If gnus writes score files in emacs-mule, it must read that
> file by explicitly specifying emacs-mule.  Or, it must write
> score files with coding: tag.

I guess that's fair.  But the inability to detect "obviously binary"
files (as in bug#16) still seems like a bug to me.


        Stefan

Re: Coding system detection: Emacs 23 vs. 22

[Kenichi Handa]

In article <jwvk5k8wuz5.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> > Emacs 23 should detect that the attached file is not iso-latin-1 (even
>>> > `file' detects this) but emacs-mule.

>>> I suspect this is related to bug #16
>>> http://emacsbugs.donarmstrong.com/cgi-bin/bugreport.cgi?bug=16

> Any comment on this bug report?

> > If gnus writes score files in emacs-mule, it must read that
> > file by explicitly specifying emacs-mule.  Or, it must write
> > score files with coding: tag.

> I guess that's fair.  But the inability to detect "obviously binary"
> files (as in bug#16) still seems like a bug to me.

Ouch!  I installed the null-byte detection in the trunk
before unicode-merge, but it seems that the change was lost
by the merge.  I'll re-install the change soon.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Coding system detection: Emacs 23 vs. 22

[Kenichi Handa]

In article <E1JZHuO-0006cF-2m@etlken.m17n.org>, Kenichi Handa <handa@m17n.org> writes:

> Ouch!  I installed the null-byte detection in the trunk
> before unicode-merge, but it seems that the change was lost
> by the merge.  I'll re-install the change soon.

I've just installed it.

---
Kenichi Handa
handa@ni.aist.go.jp

Re: Coding system detection: Emacs 23 vs. 22

[Angelo Graziosi]

Kenichi Handa wrote:

 > Ouch!  I installed the null-byte detection in the trunk
 > before unicode-merge, but it seems that the change was lost
 > by the merge.

Could this thread be related to the warnings I flagged in [1]?


Cheers,
    Angelo.

---
[1] http://lists.gnu.org/archive/html/emacs-devel/2008-02/msg00689.html

Facesti come quei che va di notte,
che porta il lume dietro e se' non giova,
ma dopo se' fa le persone dotte.
-
                   DANTE, Purgatorio, xxii 67-69

Re: Coding system detection: Emacs 23 vs. 22

[Kenichi Handa]

In article <47D79921.4060102@alice.it>, Angelo Graziosi <angelo.graziosi@alice.it> writes:

> Ouch!  I installed the null-byte detection in the trunk
> before unicode-merge, but it seems that the change was lost
> by the merge.

> Could this thread be related to the warnings I flagged in [1]?

No, I don't think so.

---
Kenichi Handa
handa@ni.aist.go.jp