bug#18022: 24.3; Emacs hard locks when attempting to do a replacement with ^M in it.

bug#18022: 24.3; Emacs hard locks when attempting to do a replacement with ^M in it.

[Marty Rosenberg]

[-- Attachment #1: Type: text/plain, Size: 7413 bytes --]

--text follows this line--
This bug report will be sent to the Bug-GNU-Emacs mailing list
and the GNU bug tracker at debbugs.gnu.org.  Please check that
the From: line contains a valid email address.  After a delay of up
to one day, you should receive an acknowledgment at that address.

Please write in English if possible, as the Emacs maintainers
usually do not have translators for other languages.

Please describe exactly what actions triggered the bug, and
the precise symptoms of the bug.  If you can, give a recipe
starting from `emacs -Q':
This does in fact reproduce with emacs -Q.
1) type Simulator::foobar()
2) run M-%
   query string is "Simulator::" RET
   replace string is "template <bool b>" C-q RET "SimulatorBase<b>::"
3) hit 'y' to accept the first change (I feel like something is wrong at
   this point)
3a) emacs enters a soft-locked state, I hit C-b 4 times while it was locked
4) hit C-g several times, to get the replacement to be shown in the
buffer
5) emacs is now totally locked

here's a backtrace of running emacs under gdb.  I had to send it a
SIGILL in order give control back to gdb.
(gdb) bt
#0  0x00000000004f6378 in Fassq ()
#1  0x000000000053a1e4 in lookup_char_property ()
#2  0x0000000000517e8e in update_syntax_table ()
#3  0x000000000051d514 in back_comment ()
#4  0x000000000051f0b5 in Fforward_comment ()
#5  0x00000000004f247a in Ffuncall ()
#6  0x000000000052603e in exec_byte_code ()
#7  0x00000000004f1fdf in funcall_lambda ()
#8  0x00000000004f22eb in Ffuncall ()
#9  0x000000000052603e in exec_byte_code ()
#10 0x00000000004f1fdf in funcall_lambda ()
#11 0x00000000004f22eb in Ffuncall ()
#12 0x000000000052603e in exec_byte_code ()
#13 0x00000000004f1fdf in funcall_lambda ()
#14 0x00000000004f22eb in Ffuncall ()
#15 0x000000000052603e in exec_byte_code ()
#16 0x00000000004f1fdf in funcall_lambda ()
#17 0x00000000004f22eb in Ffuncall ()
#18 0x000000000052603e in exec_byte_code ()
#19 0x00000000004f1fdf in funcall_lambda ()
#20 0x00000000004f22eb in Ffuncall ()
#21 0x000000000052603e in exec_byte_code ()
#22 0x00000000004f1fdf in funcall_lambda ()
#23 0x00000000004f22eb in Ffuncall ()
#24 0x00000000004f2579 in funcall_nil ()
#25 0x00000000004f0e8d in run_hook_with_args ()
#26 0x00000000004f24df in Ffuncall ()
#27 0x000000000052603e in exec_byte_code ()
#28 0x00000000004f1995 in eval_sub ()
#29 0x00000000004f4ad1 in internal_lisp_condition_case ()
#30 0x0000000000526c1b in exec_byte_code ()
#31 0x00000000004f1fdf in funcall_lambda ()
#32 0x00000000004f22eb in Ffuncall ()
#33 0x000000000052603e in exec_byte_code ()
#34 0x00000000004f1fdf in funcall_lambda ()
#35 0x00000000004f22eb in Ffuncall ()
#36 0x00000000004f0dbe in internal_condition_case_n ()
#37 0x000000000041c725 in safe_call ()
#38 0x000000000041c86f in handle_fontified_prop ()
#39 0x000000000041f9be in handle_stop ()
#40 0x0000000000427f40 in next_element_from_buffer ()
#41 0x0000000000423462 in get_next_display_element ()
#42 0x0000000000421601 in move_it_in_display_line_to ()
#43 0x00000000004283f2 in move_it_to ()
#44 0x0000000000434035 in redisplay_internal ()
#45 0x000000000048aba4 in read_char ()
#46 0x000000000048cc88 in read_key_sequence.constprop ()
#47 0x000000000048ef76 in command_loop_1 ()
#48 0x00000000004f09b3 in internal_condition_case ()
#49 0x0000000000481dde in command_loop_2 ()
#50 0x00000000004f088e in internal_catch ()
#51 0x0000000000485827 in recursive_edit_1 ()
#52 0x0000000000485b24 in Frecursive_edit ()
#53 0x0000000000405d0f in main ()



If Emacs crashed, and you have the Emacs process in the gdb debugger,
please include the output from the following gdb commands:
    `bt full' and `xbacktrace'.
For information about debugging Emacs, please read the file
/usr/share/emacs/24.3/etc/DEBUG.


In GNU Emacs 24.3.1 (x86_64-pc-linux-gnu)
 of 2014-06-23 on planetaryAC
Configured using:
 `configure '--prefix=/usr' '--build=x86_64-pc-linux-gnu'
 '--host=x86_64-pc-linux-gnu' '--mandir=/usr/share/man'
 '--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc'
 '--localstatedir=/var/lib' '--libdir=/usr/lib64'
 '--disable-silent-rules' '--disable-dependency-tracking'
 '--program-suffix=-emacs-24' '--infodir=/usr/share/info/emacs-24'
 '--localstatedir=/var'
 '--enable-locallisppath=/etc/emacs:/usr/share/emacs/site-lisp'
 '--with-crt-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/../../../../lib64'
 '--with-gameuser=games' '--without-compress-info' '--without-hesiod'
 '--without-kerberos' '--without-kerberos5' '--without-gpm'
 '--without-dbus' '--without-gnutls' '--without-xml2'
 '--without-selinux' '--without-wide-int' '--without-sound'
 '--without-x' '--without-ns' 'GENTOO_PACKAGE=app-editors/emacs-24.3-r6'
 'build_alias=x86_64-pc-linux-gnu' 'host_alias=x86_64-pc-linux-gnu'
 'CFLAGS=-O2 -pipe -march=native' 'LDFLAGS=-Wl,-O1 -Wl,--as-needed'
 'CPPFLAGS=''

Important settings:
  value of $LANG: en_US.utf8
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: C++/l

Minor modes in effect:
  ggtags-mode: t
  global-whitespace-mode: t
  show-paren-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Recent input:
ESC x r e p o r t - e m a c s - b u g RET E m a c s
SPC h a r d SPC l o c k s SPC w h e n SPC a t t e m
p t i n g SPC t o SPC d o SPC a SPC r p DEL e C-g C-x
k C-x o C-n C-n C-x o C-x o C-n C-n ESC x r e C-p ESC
[ A RET

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Simulator-arm.cpp has auto save data; consider M-x recover-this-file
Package lucid is obsolete!
loading in custom C hooks.
Loading cc-langs...done
Loading vc-git...done
Loading vc-hg...done
Quit
byte-code: Command attempted to use minibuffer while in minibuffer
byte-code: Beginning of buffer

Load-path shadows:
~/.emacs.d/rust-mode hides /usr/share/emacs/site-lisp/rust-mode/rust-mode

Features:
(shadow sort mail-extr emacsbug message idna format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mail-utils
vc-hg vc-git ggtags pcase etags ewoc url-parse auth-source eieio
byte-opt bytecomp byte-compile cconv gnus-util mm-util mail-prsvr
password-cache url-vars cc-langs cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs ggtags-autoloads
finder-inf package monky easy-mmode help-fns bindat spice-mode compile
comint ansi-color tool-bar speedbar sb-image ezimage image dframe
skeleton tempo lucid whitespace time-date paren xcscope ring darcsum
easymenu derived cl-macs gv add-log cl cl-lib site-gentoo rust-mode
regexp-opt edmacro kmacro imenu ediff-hook vc-hooks lisp-float-type
tabulated-list newcomment lisp-mode register page menu-bar rfn-eshadow
timer jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer loaddefs button faces cus-face macroexp files text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote make-network-process multi-tty emacs)

[-- Attachment #2: Type: text/html, Size: 8491 bytes --]

bug#18022: 24.3; Emacs hard locks when attempting to do a replacement with ^M in it.

[Eli Zaretskii]

> Date: Mon, 14 Jul 2014 23:13:23 -0400
> From: Marty Rosenberg <marty.rosenberg@gmail.com>
> 
> This does in fact reproduce with emacs -Q.
> 1) type Simulator::foobar()
> 2) run M-%
>    query string is "Simulator::" RET
>    replace string is "template <bool b>" C-q RET "SimulatorBase<b>::"
> 3) hit 'y' to accept the first change (I feel like something is wrong at
>    this point)
> 3a) emacs enters a soft-locked state, I hit C-b 4 times while it was locked
> 4) hit C-g several times, to get the replacement to be shown in the
> buffer
> 5) emacs is now totally locked

I cannot reproduce this here, FWIW.

bug#18022: 24.3; Emacs hard locks when attempting to do a replacement with ^M in it.

[Eli Zaretskii]

> Date: Tue, 15 Jul 2014 12:28:10 -0400
> From: Marty Rosenberg <marty.rosenberg@gmail.com>
> 
> Oh. I also opened a new file called "test.cc". I think that cc-mode is part
> of the issue.

Yes, that makes the difference.

Alan, can you take a look?

bug#18022: 24.3; Emacs hard locks when attempting to do a replacement with ^M in it.

[Alan Mackenzie]

Hello, Marty and Eli.

On Tue, Jul 15, 2014 at 07:46:08PM +0300, Eli Zaretskii wrote:
> > Date: Tue, 15 Jul 2014 12:28:10 -0400
> > From: Marty Rosenberg <marty.rosenberg@gmail.com>

> > Oh. I also opened a new file called "test.cc". I think that cc-mode is part
> > of the issue.

> Yes, that makes the difference.

> Alan, can you take a look?

OK, I've made some progress.  Just to be specific what the problem is:
When in 64-bit Gnu/Linux, a C++ file is exactly this (where ^M is a
carriage return)

template <bool b>^MSimulatorBase<b>::foobar()

(there being no newline at EOB), font-locking hangs, though hitting a
frame changing command followed by C-g often enough will switch frames.

With the help of elp-instrument-package RET c- RET, it is apparent that
c-beginning-of-macro is being called in a loop.  By putting this at the
start of c-beginning-of-macro:

  (message "1. %s" (backtrace-frame 1))
  (message "2. %s" (backtrace-frame 2))
  (message "3. %s" (backtrace-frame 3))
  (message "4. %s" (backtrace-frame 4))
  (message "5. %s" (backtrace-frame 5))
  (message "6. %s" (backtrace-frame 6))
  (message "7. %s" (backtrace-frame 7))
  (message "8. %s" (backtrace-frame 8))
  (message "9. %s" (backtrace-frame 9))
  (message "a. %s" (backtrace-frame 10))
  (message "b. %s" (backtrace-frame 11))
  (message "c. %s" (backtrace-frame 12))

, it becomes clear that the looping is in c-backward-sws, the function
generated by macro c-backward-syntactic-ws, which is being called from
c-font-lock-<>-arglists.

I don't know c-backward-sws very well, but it looks like that's about to
change.  ;-)

Marty, thanks for such a high quality bug report.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#18022: 24.3; Emacs hard locks when attempting to do a replacement with ^M in it.

[Alan Mackenzie]

Hello, Stefan, Marty and Eli.

On Wed, Jul 16, 2014 at 09:41:44PM +0000, Alan Mackenzie wrote:
> On Tue, Jul 15, 2014 at 07:46:08PM +0300, Eli Zaretskii wrote:
> > Alan, can you take a look?

> OK, I've made some progress.  Just to be specific what the problem is:
> When in 64-bit Gnu/Linux, a C++ file is exactly this (where ^M is a
> carriage return)

> template <bool b>^MSimulatorBase<b>::foobar()

> (there being no newline at EOB), font-locking hangs, though hitting a
> frame changing command followed by C-g often enough will switch frames.

OK.  The problem is in forward-comment.  If you put point on the S after
the ^M and do M-: (forward-comment -1), point moves one character
forward.  This leads to an infinite loop in CC Mode code.  (Note: for
this to manifest, the buffer must, at some stage, have had font lock
enabled, setting up some text properties).

The immediate cause of this is in Fforward_comment at ~L+168:

          else if (code == Sendcomment)
            {
------->      found = back_comment (from, from_byte, stop, comnested, comstyle,
                                    &out_charpos, &out_bytepos);
              if (!found)

Before the call to back_comment, "print SYNTAX(13)" in gdb returns
(correctly) Sendcomment.  After back_comment returns, it returns Sclose.
(In fact SYNTAX(x), for any x, returns Sclose).  So back_comment plays
with the current syntax table structure, and fails to restore it
properly - it appears that rather than restoring it for ^M, it is setting
up the (syntax table property) value of >.  13 lines lower down, we have:

                      if (SYNTAX (c) != code)
                        /* It was a two-char Sendcomment.  */
------------>           INC_BOTH (from, from_byte);
                      goto leave;

This spurious call to INC_BOTH is what causes the buggy character
movement forwards.

Looking at back_comment, just after the main loop at L+205, we have the
code which restores point and the current syntax table:

  if (comstart_pos == 0)
    {
      from = comment_end;
      from_byte = comment_end_byte;
=>    UPDATE_SYNTAX_TABLE_FORWARD (comment_end - 1);
    }

It seems to me that that " - 1" is what is causing the syntax table to be
restored to the value for ">", and shouldn't be there.  Stefan, am I
right?

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#18022: 24.3; Emacs hard locks when attempting to do a replacement with ^M in it.

[Alan Mackenzie]

Hi, everybody!

On Sat, Jul 19, 2014 at 08:33:10PM +0000, Alan Mackenzie wrote:
> Hello, Stefan, Marty and Eli.

> Looking at back_comment, just after the main loop at L+205, we have the
> code which restores point and the current syntax table:

>   if (comstart_pos == 0)
>     {
>       from = comment_end;
>       from_byte = comment_end_byte;
> =>    UPDATE_SYNTAX_TABLE_FORWARD (comment_end - 1);
>     }

> It seems to me that that " - 1" is what is causing the syntax table to be
> restored to the value for ">", and shouldn't be there.  Stefan, am I
> right?

I should have included a patch.  Sorry!  Here it is, in case anybody
didn't feel like working through the textual description.  I think this
fixes the bug, if it's correct.


=== modified file 'src/syntax.c'
*** src/syntax.c	2014-07-09 23:39:58 +0000
--- src/syntax.c	2014-07-23 18:26:45 +0000
***************
*** 825,831 ****
      {
        from = comment_end;
        from_byte = comment_end_byte;
!       UPDATE_SYNTAX_TABLE_FORWARD (comment_end - 1);
      }
    /* If comstart_pos is set and we get here (ie. didn't jump to `lossage'
       or `done'), then we've found the beginning of the non-nested comment.  */
--- 825,831 ----
      {
        from = comment_end;
        from_byte = comment_end_byte;
!       UPDATE_SYNTAX_TABLE_FORWARD (comment_end);
      }
    /* If comstart_pos is set and we get here (ie. didn't jump to `lossage'
       or `done'), then we've found the beginning of the non-nested comment.  */


-- 
Alan Mackenzie (Nuremberg, Germany).

bug#18022: 24.3; Emacs hard locks when attempting to do a replacement with ^M in it.

[Stefan Monnier]

> I should have included a patch.  Sorry!  Here it is, in case anybody
> didn't feel like working through the textual description.  I think this
> fixes the bug, if it's correct.

I don't have time to look into this now, but the symptoms, your
analysis, and our patch look mutually consistent to me.  I can't
guarantee that this is the right UPDATE_SYNTAX_TABLE_FORWARD to fix (or
whether another is needed elsewhere, maybe), but it sounds credible.


        Stefan

bug#18022: 24.3; Emacs hard locks when attempting to do a replacement with ^M in it.

[Alan Mackenzie]

Hello, Stefan.

Ping?

On Wed, Jul 23, 2014 at 06:41:46PM +0000, Alan Mackenzie wrote:
> Hi, everybody!

> On Sat, Jul 19, 2014 at 08:33:10PM +0000, Alan Mackenzie wrote:
> > Hello, Stefan, Marty and Eli.

> > Looking at back_comment, just after the main loop at L+205, we have the
> > code which restores point and the current syntax table:

> >   if (comstart_pos == 0)
> >     {
> >       from = comment_end;
> >       from_byte = comment_end_byte;
> > =>    UPDATE_SYNTAX_TABLE_FORWARD (comment_end - 1);
> >     }

> > It seems to me that that " - 1" is what is causing the syntax table to be
> > restored to the value for ">", and shouldn't be there.  Stefan, am I
> > right?

> I should have included a patch.  Sorry!  Here it is, in case anybody
> didn't feel like working through the textual description.  I think this
> fixes the bug, if it's correct.


> === modified file 'src/syntax.c'
> *** src/syntax.c	2014-07-09 23:39:58 +0000
> --- src/syntax.c	2014-07-23 18:26:45 +0000
> ***************
> *** 825,831 ****
>       {
>         from = comment_end;
>         from_byte = comment_end_byte;
> !       UPDATE_SYNTAX_TABLE_FORWARD (comment_end - 1);
>       }
>     /* If comstart_pos is set and we get here (ie. didn't jump to `lossage'
>        or `done'), then we've found the beginning of the non-nested comment.  */
> --- 825,831 ----
>       {
>         from = comment_end;
>         from_byte = comment_end_byte;
> !       UPDATE_SYNTAX_TABLE_FORWARD (comment_end);
>       }
>     /* If comstart_pos is set and we get here (ie. didn't jump to `lossage'
>        or `done'), then we've found the beginning of the non-nested comment.  */


-- 
Alan Mackenzie (Nuremberg, Germany).

bug#18022: 24.3; Emacs hard locks when attempting to do a replacement with ^M in it.

[Stefan Monnier]

>> === modified file 'src/syntax.c'
>> *** src/syntax.c	2014-07-09 23:39:58 +0000
>> --- src/syntax.c	2014-07-23 18:26:45 +0000
>> ***************
>> *** 825,831 ****
>> {
>> from = comment_end;
>> from_byte = comment_end_byte;
>> !       UPDATE_SYNTAX_TABLE_FORWARD (comment_end - 1);
>> }
>> /* If comstart_pos is set and we get here (ie. didn't jump to `lossage'
>> or `done'), then we've found the beginning of the non-nested comment.  */
>> --- 825,831 ----
>> {
>> from = comment_end;
>> from_byte = comment_end_byte;
>> !       UPDATE_SYNTAX_TABLE_FORWARD (comment_end);
>> }
>> /* If comstart_pos is set and we get here (ie. didn't jump to `lossage'
>> or `done'), then we've found the beginning of the non-nested comment.  */

This looks right, sorry for the delay,


        Stefan