bug#10383: 24.0.91; (prefer-coding-system 'utf-8-with-signature) breaks the world

bug#10383: 24.0.91; (prefer-coding-system 'utf-8-with-signature) breaks the world

[Eric Hanchrow]

This bug report will be sent to the Bug-GNU-Emacs mailing list
and the GNU bug tracker at debbugs.gnu.org.  Please check that
the From: line contains a valid email address.  After a delay of up
to one day, you should receive an acknowledgement at that address.

Please write in English if possible, as the Emacs maintainers
usually do not have translators for other languages.

Please describe exactly what actions triggered the bug, and
the precise symptoms of the bug.  If you can, give a recipe
starting from `emacs -Q':

This is

    commit 0bd650c
    Author: Chong Yidong <cyd@gnu.org>
    Date:   Sun Dec 25 06:16:00 2011

        More updates to Maintaining chapter of Emacs manual.

Type the following in *scratch*, then C-x C-e:

     (prefer-coding-system 'utf-8-with-signature)

Now type C-x d.

You'll see an error:

        Cannot open load file: dired

You will soon find that your emacs is entirely broken, since no files
can be loaded.

If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/home/ehanchrow/local/emacs-master-gui/share/emacs/24.0.91/etc/DEBUG.

In GNU Emacs 24.0.91.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.20.1)
 of 2011-11-16 on rampage.sea
Windowing system distributor `The X.Org Foundation', version 11.0.10402000
configured using `configure  '--prefix=/home/ehanchrow/local/emacs-master-gui''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: C
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Shell

Minor modes in effect:
  shell-dirtrack-mode: t
  diff-auto-refine-mode: t
  global-auto-revert-mode: t
  recentf-mode: t
  ido-everywhere: t
  display-time-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
f o r SPC o SPC i n SPC M-d M-d C-d M-f C-f C-k SPC
; SPC d o SPC C-y M-b M-b C-b C-b C-b C-b C-k $ o ;
SPC d o n e C-x o C-f C-u C-x = C-x b <return> M-b
M-b M-b M-b M-b C-b C-b ' \ 0 2 7 3 ' SPC C-x b <return>
C-f C-u C-x = C-x b <return> ' \ 0 2 7 7 ' C-e <return>
M-p M-b M-b M-b M-b <backspace> <backspace> <return>
M-p S-SPC | S-SPC o d SPC - c <return> M-p <M-backspace>
<backspace> <backspace> <backspace> <backspace> <backspace>
<backspace> > SPC / t m p / g a a a h <return> C-x
o C-x b <return> C-n C-n C-f C-f C-f C-f C-SPC M-f
M-f M-f M-f M-f M-f M-f M-f M-w C-x o e c h o SPC C-y
SPC > > SPC / t m p / g a <tab> <return> f i l e SPC
/ t m p / g a <tab> <return> C-x o C-x C-f / t m p
/ g s <backspace> a <return> C-f C-b C-b C-u C-x =
M-x h e x l - f i n d <tab> <return> g a <return> y
M-x f u n d <tab> <return> y M-x n o r <tab> m <tab>
<return> C-x o C-x b s h e <return> ~ / l o c <tab>
s r <tab> e m <tab> s r <tab> e m <tab> SPC - Q <return>
C-p C-p C-p C-p C-p C-SPC C-e M-w C-n C-SPC C-M-b C-M-b
C-M-b C-M-b C-M-b C-M-b C-M-b C-M-b C-M-b M-w M-> M-p
<return> C-c C-c M-x r e p o r t - e m a c s - b u
g <return>

Recent messages:
The file gaaah is already visited normally.
You have asked to visit it literally,
meaning no coding system decoding, format conversion, or local variables.
But Emacs can only visit a file in one way at a time.

Do you want to revisit the file literally now? (y or n)  y
Convert contents back to binary format? (y or n)  y
Auto-saving...done
Mark set [3 times]
History item: 1

Load-path shadows:
~/.emacs.d/custom hides
/home/ehanchrow/local/emacs-master-gui/share/emacs/24.0.91/lisp/custom
/usr/local/src/magit/.dir-locals hides
/home/ehanchrow/local/emacs-master-gui/share/emacs/24.0.91/lisp/gnus/.dir-locals

Features:
(shadow sort message format-spec rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader emacsbug
ruler-mode mule-util hl-line hexl mail-extr pp descr-text newcomment
multi-isearch cperl-mode ffap help-mode pcmpl-unix shell pcomplete woman
man package tabulated-list midnight uniquify warnings protbuf
marchex-erich column-marker magit-bisect magit-key-mode magit diff-mode
log-edit pcvs-util add-log vc-git autorevert paste-to-dropbox ansi-color
geiser-racket geiser-guile geiser-repl geiser-doc geiser-compile
geiser-debug geiser-menu geiser-company geiser-autodoc advice
advice-preload eldoc info-look info geiser-edit etags geiser-completion
easy-mmode geiser-eval geiser-connection tq geiser-syntax scheme
geiser-log derived geiser-popup view geiser-impl help-fns geiser-custom
geiser-base geiser path server starttls smtpmail auth-source eieio
byte-opt bytecomp byte-compile cconv macroexp assoc gnus-util
password-cache sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils grep compile comint ring cl find-func ibuf-ext ibuf-macs
ibuffer recentf tree-widget wid-edit ido dired-x easymenu dired
regexp-opt edmacro kmacro rx autoinsert time time-date tooltip
ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar dnd
fontset image fringe lisp-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer loaddefs button faces cus-face files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dbusbind dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty emacs)

bug#10383: 24.0.91; (prefer-coding-system 'utf-8-with-signature) breaks the world

[Kenichi Handa]

In article <CAHZoxq8wxViADRk+rgq5eWEwNcB8FV-F5sxbZLjqjfMtDwgVOg@mail.gmail.com>, Eric Hanchrow <eric.hanchrow@gmail.com> writes:

> Type the following in *scratch*, then C-x C-e:
>      (prefer-coding-system 'utf-8-with-signature)
> Now type C-x d.
> You'll see an error:
>         Cannot open load file: dired
> You will soon find that your emacs is entirely broken, since no files
> can be loaded.

This is because utf-8-with-signature is recognized as ASCII
compatible and thus default-file-coding-system is also set
to it.  I've just committed a fix to make
utf-8-with-signature not ASCII compatible.  Please try
again with the latest trunk code.

By the way, as a result, now

    (prefer-coding-system 'utf-8-with-signature)

does not change default-file-coding-system.  I'm not sure
what is the intention of preferring utf-8-with-signature,
one may want default-file-coding-system be set to utf-8 in
such a case.  But, doing that automatically is too
heuristic.  Another possibility is that what actually wanted
is such a coding as utf-8-with-signature-on-decoding-only
(i.e. on decoding, BOM is recognized, but on encoding, don't
produce BOM).

---
Kenichi Handa
handa@m17n.org

bug#10383: 24.0.91; (prefer-coding-system 'utf-8-with-signature) breaks the world

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Date: Wed, 28 Dec 2011 17:07:43 +0900
> Cc: 10383@debbugs.gnu.org
> 
> By the way, as a result, now
> 
>     (prefer-coding-system 'utf-8-with-signature)
> 
> does not change default-file-coding-system.  I'm not sure
> what is the intention of preferring utf-8-with-signature,
> one may want default-file-coding-system be set to utf-8 in
> such a case.  But, doing that automatically is too
> heuristic.

How about making prefer-coding-system display a message suggesting
that the user considers setting default-file-coding-system to utf-8?

> Another possibility is that what actually wanted is such a coding as
> utf-8-with-signature-on-decoding-only (i.e. on decoding, BOM is
> recognized, but on encoding, don't produce BOM).

If we can express this with current facilities, I'd say include that
information in the above-mentioned message.

bug#10383: 24.0.91; (prefer-coding-system 'utf-8-with-signature) breaks the world

[Stefan Monnier]

> This is because utf-8-with-signature is recognized as ASCII
> compatible and thus default-file-coding-system is also set
> to it.  I've just committed a fix to make
> utf-8-with-signature not ASCII compatible.  Please try
> again with the latest trunk code.

That looks like a good fix.

> By the way, as a result, now
>     (prefer-coding-system 'utf-8-with-signature)
> does not change default-file-coding-system.

That's fine.  A call to `message' might be a good addition to help the
user understand that his config might not be doing what she expects.

> I'm not sure what is the intention of preferring utf-8-with-signature,
> one may want default-file-coding-system be set to utf-8 in such
> a case.

I wouldn't waste too much time on it.

> But, doing that automatically is too heuristic.  Another possibility
> is that what actually wanted is such a coding as
> utf-8-with-signature-on-decoding-only (i.e. on decoding, BOM is
> recognized, but on encoding, don't produce BOM).

Not necessarily.  My guess is that the user doesn't realize that
coding-systems are not only used to load and save whole files but also
on chunks of texts in lots of other cases, where a BOM doesn't make
any sense.


        Stefan

bug#10383: 24.0.91; (prefer-coding-system 'utf-8-with-signature) breaks the world

[Eric Hanchrow]

Thanks, that indeed fixes it.

On Wed, Dec 28, 2011 at 12:07 AM, Kenichi Handa <handa@m17n.org> wrote:
> In article <CAHZoxq8wxViADRk+rgq5eWEwNcB8FV-F5sxbZLjqjfMtDwgVOg@mail.gmail.com>, Eric Hanchrow <eric.hanchrow@gmail.com> writes:
>
>> Type the following in *scratch*, then C-x C-e:
>>      (prefer-coding-system 'utf-8-with-signature)
>> Now type C-x d.
>> You'll see an error:
>>         Cannot open load file: dired
>> You will soon find that your emacs is entirely broken, since no files
>> can be loaded.
>
> This is because utf-8-with-signature is recognized as ASCII
> compatible and thus default-file-coding-system is also set
> to it.  I've just committed a fix to make
> utf-8-with-signature not ASCII compatible.  Please try
> again with the latest trunk code.
>
> By the way, as a result, now
>
>    (prefer-coding-system 'utf-8-with-signature)
>
> does not change default-file-coding-system.  I'm not sure
> what is the intention of preferring utf-8-with-signature,
> one may want default-file-coding-system be set to utf-8 in
> such a case.  But, doing that automatically is too
> heuristic.  Another possibility is that what actually wanted
> is such a coding as utf-8-with-signature-on-decoding-only
> (i.e. on decoding, BOM is recognized, but on encoding, don't
> produce BOM).
>
> ---
> Kenichi Handa
> handa@m17n.org