bug#3607: 23.0.94; odd character in fringe.el

bug#3607: 23.0.94; odd character in fringe.el

[Teemu Likonen]

On 2009-06-18 11:00 (-0700), Drew Adams wrote:

> I don't claim this is a bug, but perhaps someone could take a look to
> be sure.
>  
> In fringe.el, I see this text: Pavel Jan=c3=adk. The next-to-last
> character shows in my Emacs with face `escape-glyph'. This is what
> `C-u C-x =' shows:

Perhaps you know most of this already but here's some information
anyway. The name is "Pavel Janík" and it displays just fine in my
system. File fringe.el is UTF-8-encoded.

When encoded in UTF-8 the character í (U+00CD) consists of two bytes,
0xC3 and 0xAD. When some system interpretes those bytes as separete
ISO-8859-1-encoded characters they are à (0xC3) and a soft hyphen
(0xAD). This is what your system seems to have done:

> preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
>        code point: 0xAD

>   name: SOFT HYPHEN


So it sounds like some kind of singlebyte-multibyte encoding problem.

bug#3607: 23.0.94; odd character in fringe.el

[Drew Adams]

> Perhaps you know most of this already but here's some information
> anyway. The name is "Pavel Janík" and it displays just fine in my
> system. File fringe.el is UTF-8-encoded.
> 
> When encoded in UTF-8 the character í (U+00CD) consists of two bytes,
> 0xC3 and 0xAD. When some system interpretes those bytes as separete
> ISO-8859-1-encoded characters they are à (0xC3) and a soft hyphen
> (0xAD). This is what your system seems to have done:
> 
> > preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
> >        code point: 0xAD
> >   name: SOFT HYPHEN
> 
> So it sounds like some kind of singlebyte-multibyte encoding problem.

I see. Thanks for the info. I'm pretty ignorant about this stuff.

I see this in emacs -Q (on MS Windows), however, so I wonder if it isn't a bug.

And I wonder how you can see it as being UTF-8 encoded - are you using emacs -Q?

I don't see any local-variable thingy that would specify that the file is to be
UTF-8 encoded.

bug#3607: 23.0.94; odd character in fringe.el

[Teemu Likonen]

On 2009-06-18 11:54 (-0700), Drew Adams wrote:

> I see this in emacs -Q (on MS Windows), however, so I wonder if it
> isn't a bug.
>
> And I wonder how you can see it as being UTF-8 encoded - are you using
> emacs -Q?
>
> I don't see any local-variable thingy that would specify that the file
> is to be UTF-8 encoded.

Yes, the file shows correctly with "emacs -Q lisp/fringe.el" too. And it
is UTF-8 encoded file.

    $ file lisp/fringe.el
    lisp/fringe.el: UTF-8 Unicode English text

My Debian GNU/Linux system uses UTF-8 locale (as all GNU/Linux systems
do these days). Emacs probably detects my environment and uses correct
encoding settings. But I don't know Emacs works - except everything just
works. :-)

It really seems that your default environment is something other than
UTF-8, something single-byte.

bug#3607: 23.0.94; odd character in fringe.el

[Drew Adams]

> Yes, the file shows correctly with "emacs -Q lisp/fringe.el" 
> too. And it is UTF-8 encoded file.
> 
>     $ file lisp/fringe.el
>     lisp/fringe.el: UTF-8 Unicode English text
> 
> My Debian GNU/Linux system uses UTF-8 locale (as all GNU/Linux systems
> do these days). Emacs probably detects my environment and uses correct
> encoding settings. But I don't know Emacs works - except 
> everything just
> works. :-)
> 
> It really seems that your default environment is something other than
> UTF-8, something single-byte.

OK, thanks for checking.

IMO, if the file should be encoded in UTF-8, then the file itself should control
that - as buff-menu.el does, for instance. The user's locale shouldn't enter
into it at this level. Seems like a bug, to me. (But I'm no expert on this.)

bug#3607: 23.0.94; odd character in fringe.el

[Lennart Borgman]

On Thu, Jun 18, 2009 at 11:08 PM, Drew Adams<drew.adams@oracle.com> wrote:
>> Yes, the file shows correctly with "emacs -Q lisp/fringe.el"
>> too. And it is UTF-8 encoded file.
>>
>>     $ file lisp/fringe.el
>>     lisp/fringe.el: UTF-8 Unicode English text
>>
>> My Debian GNU/Linux system uses UTF-8 locale (as all GNU/Linux systems
>> do these days). Emacs probably detects my environment and uses correct
>> encoding settings. But I don't know Emacs works - except
>> everything just
>> works. :-)
>>
>> It really seems that your default environment is something other than
>> UTF-8, something single-byte.
>
> OK, thanks for checking.
>
> IMO, if the file should be encoded in UTF-8, then the file itself should control
> that - as buff-menu.el does, for instance. The user's locale shouldn't enter
> into it at this level. Seems like a bug, to me. (But I'm no expert on this.)


Yes, that must be a bug.

bug#3607: 23.0.94; odd character in fringe.el

[Kenichi Handa]

I've just added "coding: utf-8" cookie to fringe.el.

---
Kenichi Handa
handa@m17n.org


In article <e01d8a50906181434t6e6c296ega704dc11fbb77b31@mail.gmail.com>, Lennart Borgman <lennart.borgman@gmail.com> writes:

> On Thu, Jun 18, 2009 at 11:08 PM, Drew Adams<drew.adams@oracle.com> wrote:
>>> Yes, the file shows correctly with "emacs -Q lisp/fringe.el"
>>> too. And it is UTF-8 encoded file.
>>> 
>>>     $ file lisp/fringe.el
>>>     lisp/fringe.el: UTF-8 Unicode English text
>>> 
>>> My Debian GNU/Linux system uses UTF-8 locale (as all GNU/Linux systems
>>> do these days). Emacs probably detects my environment and uses correct
>>> encoding settings. But I don't know Emacs works - except
>>> everything just
>>> works. :-)
>>> 
>>> It really seems that your default environment is something other than
>>> UTF-8, something single-byte.
> >
> > OK, thanks for checking.
> >
> > IMO, if the file should be encoded in UTF-8, then the file itself should control
> > that - as buff-menu.el does, for instance. The user's locale shouldn't enter
> > into it at this level. Seems like a bug, to me. (But I'm no expert on this.)


> Yes, that must be a bug.

bug#3607: 23.0.94; odd character in fringe.el

[Stefan Monnier]

> I've just added "coding: utf-8" cookie to fringe.el.

Thanks.  But there is another bug here: a utf-8 file should be
recognized as such even in a latin-1 locale (i.e. utf-8 should always
have (one of) the highest priority).  IIUC this is done right in
GNU/Linux but not under Windows.


        Stefan

bug#3607: 23.0.94; odd character in fringe.el

[Kenichi Handa]

In article <jwveit6iizt.fsf-monnier+emacsbugreports@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > I've just added "coding: utf-8" cookie to fringe.el.

> Thanks.  But there is another bug here: a utf-8 file should be
> recognized as such even in a latin-1 locale (i.e. utf-8 should always
> have (one of) the highest priority).

Current Emacs doesn't give utf-8 the higher priority than
iso-8859-1 in Latin-X language environment.  Are you
proposing such a change?  I can't decide that is good or not
because I'm not that familiar with such locales.

> IIUC this is done right in GNU/Linux but not under
> Windows.

?? Even on GNU/Linux (ubuntu), when I start emacs as this:

% LANG=de_DE emacs

iso-8859-1 has higher priority than utf-8.

Or, do you mean the other applications on GNU/Linux?

---
Kenichi Handa
handa@m17n.org

bug#3607: 23.0.94; odd character in fringe.el

[Stefan Monnier]

> Current Emacs doesn't give utf-8 the higher priority than
> iso-8859-1 in Latin-X language environment.  Are you
> proposing such a change?  I can't decide that is good or not
> because I'm not that familiar with such locales.

It is a good change, because the likelyhood of a valid utf-8 file being
a proper latin-1 file is extremely low.

> ?? Even on GNU/Linux (ubuntu), when I start emacs as this:
> % LANG=de_DE emacs
> iso-8859-1 has higher priority than utf-8.

Duh, you're right.  Even Emacs-22 still does that.  It's wrong.

> Or, do you mean the other applications on GNU/Linux?

No, I meant Emacs, I was convinced we'd fixed it in Emacs-22 for
GNU/Linux, but it appears I was confused.


        Stefan

bug#3607: 23.0.94; odd character in fringe.el

[Kenichi Handa]

In article <jwvbpo9gxsu.fsf-monnier+emacsbugreports@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > Current Emacs doesn't give utf-8 the higher priority than
> > iso-8859-1 in Latin-X language environment.  Are you
> > proposing such a change?  I can't decide that is good or not
> > because I'm not that familiar with such locales.

> It is a good change, because the likelyhood of a valid utf-8 file being
> a proper latin-1 file is extremely low.

Ok.  For that, we must do:

  (set-coding-system-priority 'utf-8) 

somewhere.  I at first thought it could be done by
`setup-function' of Latin-1 language environment.  Actually,
when a user does C-x C-m L Latin-1 RET, it works.

But, when emacs starts up, it calls set-locale-environment,
and it at first calls set-language-environment then
overrides coding-system setups.  So, at the moment, I don't
have a good idea other than this very ad-hoc change for 23.1.

--- mule-cmds.el.~1.360.~	2009-04-09 03:03:17.000000000 +0900
+++ mule-cmds.el	2009-06-29 16:45:08.000000000 +0900
@@ -2643,6 +2643,10 @@
 		   (not (coding-system-equal coding-system
 					     locale-coding-system)))
 	  (prefer-coding-system coding-system)
+	  ;; Even if we prefer "iso-latin-1", it is better to detect
+	  ;; UTF-8.
+	  (if (eq (coding-system-base coding-system) 'iso-latin-1)
+	      (set-coding-system-priority 'utf-8))
 	  ;; Fixme: perhaps prefer-coding-system should set this too.
 	  ;; But it's not the time to do such a fundamental change.
 	  (setq default-sendmail-coding-system coding-system)

For 23.2, I think we should re-design language-info-alist.

---
Kenichi Handa
handa@m17n.org

bug#3607: 23.0.94; odd character in fringe.el

[Stefan Monnier]

>> It is a good change, because the likelyhood of a valid utf-8 file being
>> a proper latin-1 file is extremely low.

> Ok.  For that, we must do:
>   (set-coding-system-priority 'utf-8) 
> somewhere.  I at first thought it could be done by
> `setup-function' of Latin-1 language environment.  Actually,
> when a user does C-x C-m L Latin-1 RET, it works.

> But, when emacs starts up, it calls set-locale-environment,
> and it at first calls set-language-environment then
> overrides coding-system setups.  So, at the moment, I don't
> have a good idea other than this very ad-hoc change for 23.1.

Actually, AFAIK the "unlikely false positives" property of the utf-8
encoding is not only true when applied to latin-1 files but also to most
other encodings.  So really utf-8 should probably always be first (not
only for latin-1 environments), except maybe for some envs where there's
a knows non-negligible risk of false positives.


        Stefan

bug#3607: 23.0.94; odd character in fringe.el

[Kenichi Handa]

In article <jwv63ef4e6w.fsf-monnier+emacsbugreports@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Actually, AFAIK the "unlikely false positives" property of the utf-8
> encoding is not only true when applied to latin-1 files but also to most
> other encodings.  So really utf-8 should probably always be first (not
> only for latin-1 environments), except maybe for some envs where there's
> a knows non-negligible risk of false positives.

I think it's only Latin-X (and perhaps Vietnamese too) that
are mostly safe to give utf-8 the higher priority on code
detection, because only they use Latin script in which 8-bit
characters rarely appear succeedingly.

---
Kenichi Handa
handa@m17n.org

bug#3607: 23.0.94; odd character in fringe.el

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Date: Mon, 29 Jun 2009 16:49:32 +0900
> Cc: tlikonen@iki.fi, 3607@emacsbugs.donarmstrong.com
> 
> But, when emacs starts up, it calls set-locale-environment,
> and it at first calls set-language-environment then
> overrides coding-system setups.  So, at the moment, I don't
> have a good idea other than this very ad-hoc change for 23.1.

PLEEEEAAAAASE do _not_ make such ad-hoc changes on the branch at this time.
Experience shows that there be dragons, and we _do_ want to release
Emacs 23.1 some time this year...

> --- mule-cmds.el.~1.360.~	2009-04-09 03:03:17.000000000 +0900
> +++ mule-cmds.el	2009-06-29 16:45:08.000000000 +0900
> @@ -2643,6 +2643,10 @@
>  		   (not (coding-system-equal coding-system
>  					     locale-coding-system)))
>  	  (prefer-coding-system coding-system)
> +	  ;; Even if we prefer "iso-latin-1", it is better to detect
> +	  ;; UTF-8.
> +	  (if (eq (coding-system-base coding-system) 'iso-latin-1)
> +	      (set-coding-system-priority 'utf-8))
>  	  ;; Fixme: perhaps prefer-coding-system should set this too.
>  	  ;; But it's not the time to do such a fundamental change.
>  	  (setq default-sendmail-coding-system coding-system)
> 
> For 23.2, I think we should re-design language-info-alist.

Then let's defer the whole thing to Emacs 23.2.  It's not a grave
problem, IMO, certainly not worth taking a risk of unintended
consequences.

bug#3607: 23.0.94; odd character in fringe.el

[Stefan Monnier]

>> But, when emacs starts up, it calls set-locale-environment,
>> and it at first calls set-language-environment then
>> overrides coding-system setups.  So, at the moment, I don't
>> have a good idea other than this very ad-hoc change for 23.1.

> PLEEEEAAAAASE do _not_ make such ad-hoc changes on the branch at this time.
> Experience shows that there be dragons, and we _do_ want to release
> Emacs 23.1 some time this year...

Indeed, those things should go on the trunk only,


        Stefan

bug#3607: 23.0.94; odd character in fringe.el

[Drew Adams]

I don't claim this is a bug, but perhaps someone could take a look to
be sure.
 
In fringe.el, I see this text: Pavel Jan=c3=adk. The next-to-last
character shows in my Emacs with face `escape-glyph'. This is what
`C-u C-x =' shows:
 
        character: =ad (173, #o255, #xad)
preferred charset: iso-8859-1 (Latin-1 (ISO/IEC 8859-1))
       code point: 0xAD
           syntax: _  which means: symbol
         category: b:Arabic, h:Korean, j:Japanese, l:Latin
      buffer code: #xC2 #xAD
        file code: #xAD (encoded by coding system iso-latin-1-unix)
          display: by this font (glyph code)
    uniscribe:-outline-Courier
New-normal-normal-normal-mono-13-*-*-*-c-*-iso8859-1 (#x10)
   hardcoded face: escape-glyph
 
Character code properties: customize what to show
  name: SOFT HYPHEN
  general-category: Cf (Other, Format)
 
There are text properties here:
  charset              iso-8859-1
  face                 font-lock-comment-face
  fontified            t
 
Is this something weird, or is it OK? Since I have customized face
`escape-glyph', I notice this easily, but it is not really noticeable
in emacs -Q. Why would a soft hyphen character be displayed using face
`escape-glyph'?
 
BTW, when trying to send this, Emacs asks if I want to convert
non-ASCII chars to hexadecimal. Dunno which would be more helpful, so
I'll guess yes. The *Help* text quoted should give enough info,
anyway.
 
 
 
In GNU Emacs 23.0.94.1 (i386-mingw-nt5.1.2600)
 of 2009-05-24 on SOFT-MJASON
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (3.4)'