bug#8122: 23.2.94; (prefer-coding-system 'utf-8) changes indentation behavior for emacsclient

bug#8122: 23.2.94; (prefer-coding-system 'utf-8) changes indentation behavior for emacsclient

[Steve Revilak]

[-- Attachment #1: Type: text/plain, Size: 6093 bytes --]

This bug report will be sent to the Free Software Foundation,
not to your local site managers!
Please write in English if possible, because the Emacs maintainers
usually do not have translators to read other languages for them.

Your report will be posted to the bug-gnu-emacs@gnu.org mailing list
and the gnu.emacs.bug news group, and at http://debbugs.gnu.org.

Please describe exactly what actions triggered the bug
and the precise symptoms of the bug.  If you can, give
a recipe starting from `emacs -Q':

In the 23.2.94 emacs pretest, the use of (prefer-coding-system 'utf-8)
changes text-mode indentation behavior with emacsclient/emacs --daemon.

Steps to reproduce

(1) Start emacs in daemon mode, as follows:
     env TMPDIR=$HOME/tmp emacs  -Q --eval "(prefer-coding-system 'utf-8)" --daemon

(2) Use emacsclient to connect to the server started in step (1),
     editing a new text file.
     rm -f a.txt
     env TMPDIR=$HOME/tmp emacsclient a.txt

(3) In the buffer for a.txt, type the lines that appear below.  Do not
     press return after typing "ccc"

aaa
bbb

   ccc

(4) With point at the end of line "ccc", press RETURN.  If "ccc"
     appeared in line 5 of the buffer, then the cursor will appear at
     line 6, column 2.  

Expected behavior: after pressing RETURN, the cursor should have
appeared line line 6, column 0.

(5) With the cursor at line 6, column 2.  Press ESC-\ (aka
     "delete-horizonal-space").   Now, the cursor appears in line 6,
     column 0

(6) With the cursor in line 6, column 0, press RETURN.  The cursor now
     appears in line 7, column 2.

Expected behavior: after pressing RETURN, the cursor should have
appeared in line 7, column 0.

Regression Notes:

* The behavior I've described appears to be specific to emacsclient.
   For example, 
       emacs  -Q --eval "(prefer-coding-system 'utf-8)"
   does not exhibit the behavior I've described.

* The last emacs pretest that I used on this system was 23.2.91.
   23.2.91 did not exhibit this behavior.
   Later today, I will try building 23.2.92 and 23.2.93, to get see if
   the behavior appeared in those versions.

* I've observed this behavior on Mac OS X 
   Darwin oatmeal 10.4.0 Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 i386

   and on OpenSUSE 11.3.
   Linux sunny 2.6.34.7-0.7-desktop #1 SMP PREEMPT 2010-12-13 11:13:53 +0100 x86_64 x86_64 x86_64 GNU/Linux


If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
     `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/Applications/Emacs.app/Contents/Resources/etc/DEBUG.


In GNU Emacs 23.2.94.1 (x86_64-apple-darwin10.4.0, NS apple-appkit-1038.32)
  of 2011-02-15 on oatmeal.ma.runwaynine.com
Windowing system distributor `Apple', version 10.3.1038
configured using `configure  '--with-ns''

Important settings:
   value of $LC_ALL: nil
   value of $LC_COLLATE: C
   value of $LC_CTYPE: nil
   value of $LC_MESSAGES: nil
   value of $LC_MONETARY: nil
   value of $LC_NUMERIC: nil
   value of $LC_TIME: nil
   value of $LANG: nil
   value of $XMODIFIERS: nil
   locale-coding-system: nil
   default enable-multibyte-characters: t

Major mode: Emacs-Lisp

Minor modes in effect:
   diff-auto-refine-mode: t
   tooltip-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t

Recent input:
A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC 
[ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A 
ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC 
[ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A 
ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC 
[ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A 
ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC 
[ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A 
ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC 
[ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A 
ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC [ A ESC 
[ A ESC [ A ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B 
ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC [ B ESC 
[ B ESC [ B ESC [ B ESC [ B ESC [ B DEL C-x C-s C-@ 
C-e ESC w C-x v u y e s RET C-x C-s ESC x r e p o r 
t - e m a c s TAB RET

Recent messages:
Saving file /Users/srevilak/.emacs...
Wrote /Users/srevilak/.emacs
Saving file /Users/srevilak/.emacs...
Wrote /Users/srevilak/.emacs
Saving file /Users/srevilak/.emacs...
Wrote /Users/srevilak/.emacs
Mark activated
Finding changes in /Users/srevilak/.emacs...
Reverting /Users/srevilak/.emacs...done
(No changes need to be saved)

Load-path shadows:
None found.

Features:
(shadow sort mail-extr message sendmail regexp-opt ecomplete rfc822 mml
mml-sec password-cache mm-decode mm-bodies mm-encode mailcap mail-parse
rfc2231 rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader gnus-util
netrc time-date mm-util mail-prsvr gmm-utils wid-edit mailheader canlock
sha1 hex-util hashcash mail-utils emacsbug log-edit ring pcvs-util
add-log diff-mode easy-mmode vc vc-dispatcher rect pp find-func help-fns
help-mode view debug vc-git tooltip ediff-hook vc-hooks lisp-float-type
mwheel ns-win easymenu tool-bar dnd fontset image fringe lisp-mode
register page menu-bar rfn-eshadow timer select scroll-bar mldrag mouse
jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
loaddefs button minibuffer faces cus-face files text-properties overlay
md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process ns multi-tty
emacs)

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

bug#8122: More regression notes for bug#8122

[Steve Revilak]

[-- Attachment #1: Type: text/plain, Size: 236 bytes --]

I've tried to reproduce bug 8122 on pretest versions 23.2.92 and
23.2.93.  Here are my findings:

Bug 8122 occurs in 23.2.94.

Bug 8122 DOES NOT occur with 23.2.91, 23.2.92, or 23.2.93.

So, this appears to be new behavior in 23.2.94.


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

bug#8122: 23.2.94; (prefer-coding-system 'utf-8) changes indentation behavior for emacsclient

[Eli Zaretskii]

> Date: Sat, 26 Feb 2011 09:31:16 -0500
> From: Steve Revilak <steve@srevilak.net>
> Cc: 
> 
> (3) In the buffer for a.txt, type the lines that appear below.  Do not
>      press return after typing "ccc"
> 
> aaa
> bbb
> 
>    ccc
> 
> (4) With point at the end of line "ccc", press RETURN.  If "ccc"
>      appeared in line 5 of the buffer, then the cursor will appear at
>      line 6, column 2.  
> 
> Expected behavior: after pressing RETURN, the cursor should have
> appeared line line 6, column 0.

This happens because RET is somehow perceived as C-j, which invokes
newline-and-indent, and not as RET that runs newline.  Try "C-h c RET"
and see what it says.

I have no idea how prefer-coding-system is able to modify the way RET
is interpreted by Emacs.

bug#8122: More regression notes for bug#8122

[Eli Zaretskii]

> Date: Sat, 26 Feb 2011 10:05:24 -0500
> From: Steve Revilak <steve@srevilak.net>
> Cc: 
> 
> I've tried to reproduce bug 8122 on pretest versions 23.2.92 and
> 23.2.93.  Here are my findings:
> 
> Bug 8122 occurs in 23.2.94.
> 
> Bug 8122 DOES NOT occur with 23.2.91, 23.2.92, or 23.2.93.
> 
> So, this appears to be new behavior in 23.2.94.

The culprit seems to be this change:

  2011-02-12  Eli Zaretskii  <eliz@gnu.org>

	  * terminal.c (create_terminal): Use default-keyboard-coding-system
	  and default-terminal-coding-system to initialize coding systems of
	  the new terminal.  (Bug#7840)

but I'd be damned if I understand how could it affect the character
produced by the RET key.  Hmm...

bug#8122: 23.2.94; (prefer-coding-system 'utf-8) changes indentation behavior for emacsclient

[Eli Zaretskii]

> Date: Sat, 26 Feb 2011 09:31:16 -0500
> From: Steve Revilak <steve@srevilak.net>
> Cc: 
> 
> (1) Start emacs in daemon mode, as follows:
>      env TMPDIR=$HOME/tmp emacs  -Q --eval "(prefer-coding-system 'utf-8)" --daemon
> 
> (2) Use emacsclient to connect to the server started in step (1),
>      editing a new text file.
>      rm -f a.txt
>      env TMPDIR=$HOME/tmp emacsclient a.txt
> 
> (3) In the buffer for a.txt, type the lines that appear below.  Do not
>      press return after typing "ccc"
> 
> aaa
> bbb
> 
>    ccc
> 
> (4) With point at the end of line "ccc", press RETURN.  If "ccc"
>      appeared in line 5 of the buffer, then the cursor will appear at
>      line 6, column 2.  
> 
> Expected behavior: after pressing RETURN, the cursor should have
> appeared line line 6, column 0.

I found the reason for this: when keyboard-coding-system is utf-8,
typing RET is decoded as C-j.  If you start the Emacs daemon like
this:

 emacs  -Q --eval "(prefer-coding-system 'utf-8-unix)" --daemon
                                          ^^^^^^^^^^
the problem goes away.

bug#8122: 23.2.94; (prefer-coding-system 'utf-8) changes indentation behavior for emacsclient

[Eli Zaretskii]

> Date: Sat, 26 Feb 2011 09:31:16 -0500
> From: Steve Revilak <steve@srevilak.net>
> Cc: 
> 
> (1) Start emacs in daemon mode, as follows:
>      env TMPDIR=$HOME/tmp emacs  -Q --eval "(prefer-coding-system 'utf-8)" --daemon
> 
> (2) Use emacsclient to connect to the server started in step (1),
>      editing a new text file.
>      rm -f a.txt
>      env TMPDIR=$HOME/tmp emacsclient a.txt
> 
> (3) In the buffer for a.txt, type the lines that appear below.  Do not
>      press return after typing "ccc"
> 
> aaa
> bbb
> 
>    ccc
> 
> (4) With point at the end of line "ccc", press RETURN.  If "ccc"
>      appeared in line 5 of the buffer, then the cursor will appear at
>      line 6, column 2.  
> 
> Expected behavior: after pressing RETURN, the cursor should have
> appeared line line 6, column 0.

I found the reason for this: when keyboard-coding-system is utf-8,
typing RET is decoded as C-j.  If you start the Emacs daemon like
this:

 emacs  -Q --eval "(prefer-coding-system 'utf-8-unix)" --daemon
                                          ^^^^^^^^^^
the problem goes away.

This happens because, unlike set-keyboard-coding-system, which adds
"-unix" to its argument, we don't do the same when we set up
default-keyboard-coding-system.

(This problem is quite old; I guess no one used
default-keyboard-coding-system until the change I made in terminal.c
exposed this bug.)

I propose the following change to fix this:

--- lisp/international/mule-cmds.el~0	2011-01-08 12:45:14.000000000 -0500
+++ lisp/international/mule-cmds.el	2011-02-26 11:32:22.483284753 -0500
@@ -368,7 +368,8 @@
 		 (coding-system-get coding-system 'ascii-compatible-p)))
 	(setq default-file-name-coding-system coding-system)))
   (setq default-terminal-coding-system coding-system)
-  (setq default-keyboard-coding-system coding-system)
+  (setq default-keyboard-coding-system
+	(coding-system-change-eol-conversion coding-system 'unix))
   ;; Preserve eol-type from existing default-process-coding-systems.
   ;; On non-unix-like systems in particular, these may have been set
   ;; carefully by the user, or by the startup code, to deal with the

bug#8122: 23.2.94; (prefer-coding-system 'utf-8) changes indentation behavior for emacsclient

[Stefan Monnier]

> I propose the following change to fix this:

Sounds right, thank you,


        Stefan


> --- lisp/international/mule-cmds.el~0	2011-01-08 12:45:14.000000000 -0500
> +++ lisp/international/mule-cmds.el	2011-02-26 11:32:22.483284753 -0500
> @@ -368,7 +368,8 @@
>  		 (coding-system-get coding-system 'ascii-compatible-p)))
>  	(setq default-file-name-coding-system coding-system)))
>    (setq default-terminal-coding-system coding-system)
> -  (setq default-keyboard-coding-system coding-system)
> +  (setq default-keyboard-coding-system
> +	(coding-system-change-eol-conversion coding-system 'unix))
>    ;; Preserve eol-type from existing default-process-coding-systems.
>    ;; On non-unix-like systems in particular, these may have been set
>    ;; carefully by the user, or by the startup code, to deal with the

bug#8122: 23.2.94; (prefer-coding-system 'utf-8) changes indentation behavior for emacsclient

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Steve Revilak <steve@srevilak.net>,  Kenichi Handa <handa@m17n.org>,  8122@debbugs.gnu.org
> Date: Sat, 26 Feb 2011 14:56:37 -0500
> 
> > I propose the following change to fix this:
> 
> Sounds right, thank you,

Installed on the emacs-23 branch.

bug#8122: 23.2.94; (prefer-coding-system 'utf-8) changes indentation behavior for emacsclient

[Steve Revilak]

[-- Attachment #1: Type: text/plain, Size: 1225 bytes --]

>If you start the Emacs daemon like this:
>
> emacs  -Q --eval "(prefer-coding-system 'utf-8-unix)" --daemon
>                                          ^^^^^^^^^^
>the problem goes away.

Eli,

As you suggested, I tried changing (prefer-coding-system 'utf-8) to
(prefer-coding-system 'utf-8-unix).  Indeed, that makes the problem go
away.


>I propose the following change to fix this:
>
>--- lisp/international/mule-cmds.el~0	2011-01-08 12:45:14.000000000 -0500
>+++ lisp/international/mule-cmds.el	2011-02-26 11:32:22.483284753 -0500
>@@ -368,7 +368,8 @@
> 		 (coding-system-get coding-system 'ascii-compatible-p)))
> 	(setq default-file-name-coding-system coding-system)))
>   (setq default-terminal-coding-system coding-system)
>-  (setq default-keyboard-coding-system coding-system)
>+  (setq default-keyboard-coding-system
>+	(coding-system-change-eol-conversion coding-system 'unix))
>   ;; Preserve eol-type from existing default-process-coding-systems.
>   ;; On non-unix-like systems in particular, these may have been set
>   ;; carefully by the user, or by the startup code, to deal with the


I tested this patch on my system (with "(prefer-coding-system
'utf-8)").  It works.

Thanks for the speedy fix!

Steve

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

bug#8122: 23.2.94; (prefer-coding-system 'utf-8) changes indentation behavior for emacsclient

[Eli Zaretskii]

> Date: Sat, 26 Feb 2011 17:57:38 -0500
> From: Steve Revilak <steve@srevilak.net>
> Cc: Kenichi Handa <handa@m17n.org>, 8122@debbugs.gnu.org
> 
> >If you start the Emacs daemon like this:
> >
> > emacs  -Q --eval "(prefer-coding-system 'utf-8-unix)" --daemon
> >                                          ^^^^^^^^^^
> >the problem goes away.
> 
> Eli,
> 
> As you suggested, I tried changing (prefer-coding-system 'utf-8) to
> (prefer-coding-system 'utf-8-unix).  Indeed, that makes the problem go
> away.
> 
> 
> >I propose the following change to fix this:
> >
> >--- lisp/international/mule-cmds.el~0	2011-01-08 12:45:14.000000000 -0500
> >+++ lisp/international/mule-cmds.el	2011-02-26 11:32:22.483284753 -0500
> >@@ -368,7 +368,8 @@
> > 		 (coding-system-get coding-system 'ascii-compatible-p)))
> > 	(setq default-file-name-coding-system coding-system)))
> >   (setq default-terminal-coding-system coding-system)
> >-  (setq default-keyboard-coding-system coding-system)
> >+  (setq default-keyboard-coding-system
> >+	(coding-system-change-eol-conversion coding-system 'unix))
> >   ;; Preserve eol-type from existing default-process-coding-systems.
> >   ;; On non-unix-like systems in particular, these may have been set
> >   ;; carefully by the user, or by the startup code, to deal with the
> 
> 
> I tested this patch on my system (with "(prefer-coding-system
> 'utf-8)").  It works.

Thanks for testing.  The fix is now on the release branch.