bug#19511: 25.0.50; Failed assertions in redisplay() code. (dispnew.c:1405; xdisp.c:17524)

bug#19511: 25.0.50; Failed assertions in redisplay() code. (dispnew.c:1405; xdisp.c:17524)

[Andreas Matthias]

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]

With the attached example code I can trigger two different assertions
somewhere down the line of redisplay():

1) dispnew.c:1405: Emacs fatal error: assertion failed: row >= 0 && row < matrix->nrows

2) xdisp.c:17524: Emacs fatal error: assertion failed: row->enabled_p


Unfortunately I could not further isolate the elisp code which eventually
causes these assertions to fail. The following packages from MELPA are
involved: lua-mode, polymode. (version numbers see example code)

This is the example code triggering the mentioned assertions.


[-- Attachment #2: pm-01.el --]
[-- Type: application/emacs-lisp, Size: 1057 bytes --]

[-- Attachment #3: pm-01.nw --]
[-- Type: test/plain, Size: 74 bytes --]

[-- Attachment #4: Type: text/plain, Size: 7108 bytes --]


Note: With a recently checked out Emacs 25.0.50 I can trigger both assertions
as described below. Concerning Emacs 24.3.1 I can trigger only the first
assertion but not the second one.

Kind regards,
Andreas


1)  The first assertion is triggered by:

      * run emacs -Q -l pm-01.el
      * press "delete" twice

    And this is the backtrace:

#0  0x00007ffff37d120b in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
#1  0x000000000057f286 in terminate_due_to_signal (sig=6, backtrace_limit=2147483647)
    at emacs.c:386
#2  0x000000000060ca9d in die (msg=0x6f2768 "row >= 0 && row < matrix->nrows", 
    file=0x6f2540 "dispnew.c", line=1405) at alloc.c:7108
#3  0x0000000000418d3e in matrix_row (matrix=0xf84b40, row=-1) at dispnew.c:1405
#4  0x0000000000479d85 in try_window_id (w=0x137c530) at xdisp.c:18436
#5  0x000000000046ee38 in redisplay_window (window=..., just_this_one_p=true)
    at xdisp.c:16396
#6  0x00000000004660e2 in redisplay_window_1 (window=...) at xdisp.c:14310
#7  0x000000000062cade in internal_condition_case_1 (bfun=0x4660a0 <redisplay_window_1>, 
    arg=..., handlers=..., hfun=0x466024 <redisplay_window_error>) at eval.c:1369
#8  0x000000000046529f in redisplay_internal () at xdisp.c:13953
#9  0x000000000046287a in redisplay () at xdisp.c:13158
#10 0x0000000000586ef8 in read_char (commandflag=1, map=..., prev_event=..., 
    used_mouse_menu=0x7fffffffd865, end_time=0x0) at keyboard.c:2643
#11 0x0000000000596ced in read_key_sequence (keybuf=0x7fffffffda90, bufsize=30, prompt=..., 
    dont_downcase_last=false, can_return_switch_frame=true, fix_current_buffer=true, 
    prevent_redisplay=false) at keyboard.c:9257
#12 0x0000000000583e16 in command_loop_1 () at keyboard.c:1510
#13 0x000000000062c96d in internal_condition_case (bfun=0x583a3f <command_loop_1>, 
    handlers=..., hfun=0x5831af <cmd_error>) at eval.c:1345
#14 0x00000000005836dd in command_loop_2 (ignore=...) at keyboard.c:1245
#15 0x000000000062bddb in internal_catch (tag=..., func=0x5836ba <command_loop_2>, arg=...)
    at eval.c:1106
#16 0x0000000000583691 in command_loop () at keyboard.c:1224
#17 0x0000000000582cdb in recursive_edit_1 () at keyboard.c:834
#18 0x0000000000582eab in Frecursive_edit () at keyboard.c:905
#19 0x0000000000580c10 in main (argc=4, argv=0x7fffffffde58) at emacs.c:1619


2) The second assertion is triggered by:

      * run emacs -Q -l pm-01.el
      * press "a" twice

    And this is the backtrace:

#0  0x00007ffff37d120b in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
#1  0x000000000057f286 in terminate_due_to_signal (sig=6, backtrace_limit=2147483647)
    at emacs.c:386
#2  0x000000000060ca9d in die (msg=0x6f92ee "row->enabled_p", file=0x6f6640 "xdisp.c", 
    line=17524) at alloc.c:7108
#3  0x00000000004772c9 in find_last_row_displaying_text (matrix=0x15f57d0, 
    it=0x7fffffff8690, start=0x1d8c720) at xdisp.c:17524
#4  0x0000000000479fce in try_window_id (w=0x137c530) at xdisp.c:18485
#5  0x000000000046ee38 in redisplay_window (window=..., just_this_one_p=true)
    at xdisp.c:16396
#6  0x00000000004660e2 in redisplay_window_1 (window=...) at xdisp.c:14310
#7  0x000000000062cade in internal_condition_case_1 (bfun=0x4660a0 <redisplay_window_1>, 
    arg=..., handlers=..., hfun=0x466024 <redisplay_window_error>) at eval.c:1369
#8  0x000000000046529f in redisplay_internal () at xdisp.c:13953
#9  0x000000000046287a in redisplay () at xdisp.c:13158
#10 0x0000000000586ef8 in read_char (commandflag=1, map=..., prev_event=..., 
    used_mouse_menu=0x7fffffffd865, end_time=0x0) at keyboard.c:2643
#11 0x0000000000596ced in read_key_sequence (keybuf=0x7fffffffda90, bufsize=30, prompt=..., 
    dont_downcase_last=false, can_return_switch_frame=true, fix_current_buffer=true, 
    prevent_redisplay=false) at keyboard.c:9257
#12 0x0000000000583e16 in command_loop_1 () at keyboard.c:1510
#13 0x000000000062c96d in internal_condition_case (bfun=0x583a3f <command_loop_1>, 
    handlers=..., hfun=0x5831af <cmd_error>) at eval.c:1345
#14 0x00000000005836dd in command_loop_2 (ignore=...) at keyboard.c:1245
#15 0x000000000062bddb in internal_catch (tag=..., func=0x5836ba <command_loop_2>, arg=...)
    at eval.c:1106
#16 0x0000000000583691 in command_loop () at keyboard.c:1224
#17 0x0000000000582cdb in recursive_edit_1 () at keyboard.c:834
#18 0x0000000000582eab in Frecursive_edit () at keyboard.c:905
#19 0x0000000000580c10 in main (argc=4, argv=0x7fffffffde58) at emacs.c:1619



In GNU Emacs 25.0.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8)
 of 2015-01-04 on winky
Repository revision: d7e858bcc6f353ea3e955ca2a91d7b5c33bb6611
Windowing system distributor `The X.Org Foundation', version 11.0.11501000
System Description:	Ubuntu 14.04.1 LTS

Configured using:
 `configure --prefix=/home/andreas/local/emacs --enable-checking=all
 --enable-check-lisp-object-type 'CFLAGS=-g3 -O0''

Configured features:
XPM JPEG TIFF GIF PNG SOUND DBUS GSETTINGS NOTIFY GNUTLS LIBXML2
FREETYPE XFT ZLIB

Important settings:
  value of $LC_COLLATE: en_US.UTF-8
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_MESSAGES: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message dired format-spec
rfc822 mml easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mm-util help-fns mail-prsvr mail-utils time-date tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win
x-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list
newcomment elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax
facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak
czech european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev minibuffer nadvice loaddefs button
faces cus-face macroexp files text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind gfilenotify dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty emacs)

Memory information:
((conses 16 76943 4320)
 (symbols 48 18291 1)
 (miscs 40 45 125)
 (strings 32 11760 3895)
 (string-bytes 1 323298)
 (vectors 16 10065)
 (vector-slots 8 395354 9010)
 (floats 8 72 61)
 (intervals 56 192 0)
 (buffers 976 11)
 (heap 1024 44733 1424))

bug#19511: 25.0.50; Failed assertions in redisplay() code. (dispnew.c:1405; xdisp.c:17524)

[Eli Zaretskii]

> From: Andreas Matthias <andreas.matthias@gmail.com>
> Date: Sun, 04 Jan 2015 23:27:33 +0100
> 
> With the attached example code I can trigger two different assertions
> somewhere down the line of redisplay():
> 
> 1) dispnew.c:1405: Emacs fatal error: assertion failed: row >= 0 && row < matrix->nrows
> 
> 2) xdisp.c:17524: Emacs fatal error: assertion failed: row->enabled_p
> 
> 
> Unfortunately I could not further isolate the elisp code which eventually
> causes these assertions to fail. The following packages from MELPA are
> involved: lua-mode, polymode. (version numbers see example code)

The reason for this is that Polymode does something unthinkable: it
calls bury-buffer in the function it installs as
font-lock-fontify-region-function, which buries the buffer being
fontified, and as side effect switches the buffer displayed in the
window.  When this is called by redisplay, the effect is that the
information about the window end point gets invalidated right from
under the feet of the display engine, in the middle of the code that
tries to use that information for one of redisplay optimizations.

I have now disabled that optimization for packages which commit such
atrocities, so Polymode is now merely a performance killer, not a
crasher.

I hope Polymode will be changed to not call bury-buffer in that
situation (I always thought bury-buffer is strictly for interactive
use, FWIW).

> Note: With a recently checked out Emacs 25.0.50 I can trigger both assertions
> as described below. Concerning Emacs 24.3.1 I can trigger only the first
> assertion but not the second one.

Emacs 24.3 escapes the 2nd assertion by sheer luck, AFAICS, and 24.4
is affected by both.  So I installed the fix for this in the emacs-24
branch (commit d279e66).

Thanks.

bug#19511: 25.0.50; Failed assertions in redisplay() code. (dispnew.c:1405; xdisp.c:17524)

[Vitalie Spinu]

 >>> Eli Zaretskii on Mon, 05 Jan 2015 17:58:05 +0200 wrote:

[...]

> I hope Polymode will be changed to not call bury-buffer in that
> situation (I always thought bury-buffer is strictly for interactive
> use, FWIW).

Bury-buffer is used to "hide" from the indirect buffer from the
user. That was the easiest way to implement that and should have been
rewritten anyways.

Removing it doesn't change the fact that buffers are switched
(with-current-buffer ...) inside font-lock-fontify-region-function. But
I guess that's not an issue (right?).

Would it be enough to remove `bury-buffer` call to get back the
optimization? What are other elisp functions that can potentially
invalidate window_end?

Thanks,

  Vitalie

bug#19511: 25.0.50; Failed assertions in redisplay() code. (dispnew.c:1405; xdisp.c:17524)

[Stefan Monnier]

> Removing it doesn't change the fact that buffers are switched
> (with-current-buffer ...) inside font-lock-fontify-region-function.

with-current-buffer is basically equivalent to something like

  (let ((current-buffer ..)) ...)

[ tho implemented differently for technical reasons.  ]
So it doesn't have any undesirable interactions with the redisplay code.

> Would it be enough to remove `bury-buffer` call to get back the
> optimization? What are other elisp functions that can potentially
> invalidate window_end?

Any function which changes which buffer is being displayed in
a particular window.


        Stefan

bug#19511: 25.0.50; Failed assertions in redisplay() code. (dispnew.c:1405; xdisp.c:17524)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Andreas Matthias <andreas.matthias@gmail.com>,  19511-done@debbugs.gnu.org
> Date: Mon, 05 Jan 2015 12:53:54 -0500
> 
> Any function which changes which buffer is being displayed in
> a particular window.

And also any function that changes the window display itself, like
split-window, for example.

bug#19511: 25.0.50; Failed assertions in redisplay() code. (dispnew.c:1405; xdisp.c:17524)

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Cc: Andreas Matthias <andreas.matthias@gmail.com>,  19511-done@debbugs.gnu.org
> Date: Mon, 05 Jan 2015 08:59:02 -0800
> 
> > I hope Polymode will be changed to not call bury-buffer in that
> > situation (I always thought bury-buffer is strictly for interactive
> > use, FWIW).
> 
> Bury-buffer is used to "hide" from the indirect buffer from the
> user. That was the easiest way to implement that and should have been
> rewritten anyways.

bury-buffer is a command.  A command typically does a lot of dwim-ish
stuff that is not directly related to its main job.  If you needed to
merely put the buffer at the end of buffer-list, you could have called
bury-buffer-internal instead, which does precisely that and nothing
else.  Assuming you really need to manipulate buffer-list at all.

> Removing it doesn't change the fact that buffers are switched
> (with-current-buffer ...) inside font-lock-fontify-region-function. But
> I guess that's not an issue (right?).

Right, that's not the issue.  The problem is the call to
switch-to-prev-buffer that bury-buffer does.  That results in a call
to set-window-buffer, which is the one that invalidates window_end_pos
etc.

> Would it be enough to remove `bury-buffer` call to get back the
> optimization?

Probably, but if you show me the changes, I can see if they achieve
that.

> What are other elisp functions that can potentially invalidate
> window_end?

Any function that calls apply_window_adjustment, for example.  Which
is called in just a handful of places, see window.c.  Also
set-window-start and functions that split or delete windows.
Basically, those that really change the window's display (and
therefore shouldn't be called in a fontification function).

bug#19511: 25.0.50; Failed assertions in redisplay() code. (dispnew.c:1405; xdisp.c:17524)

[Andreas Matthias]

Eli Zaretskii wrote:

> I have now disabled that optimization for packages which commit such
> atrocities, so Polymode is now merely a performance killer, not a
> crasher.

The decrease in performance is noticeable, but this is definitely better
than continuous crashes.

> So I installed the fix for this in the emacs-24 branch (commit d279e66).

Thank you.

Andreas

bug#19511: 25.0.50; Failed assertions in redisplay() code. (dispnew.c:1405; xdisp.c:17524)

[Eli Zaretskii]

> From: Andreas Matthias <andreas.matthias@gmail.com>
> Cc: 19511-done@debbugs.gnu.org,  Vitalie Spinu <spinuvit@gmail.com>
> Date: Mon, 05 Jan 2015 19:10:59 +0100
> 
> Eli Zaretskii wrote:
> 
> > I have now disabled that optimization for packages which commit such
> > atrocities, so Polymode is now merely a performance killer, not a
> > crasher.
> 
> The decrease in performance is noticeable

There really is nothing else I can do under these circumstances.  I
initially tried not to reject so many optimizations, but that left
strange artifacts on the screen, so I was forced to do it more
thoroughly.

Hopefully, the underlying problem will be solved in Polymode, and
performance will be back where it was before.

bug#19511: 25.0.50; Failed assertions in redisplay() code. (dispnew.c:1405; xdisp.c:17524)

[Vitalie Spinu]

 >>> Eli Zaretskii on Mon, 05 Jan 2015 20:36:48 +0200 wrote:

 > Hopefully, the underlying problem will be solved in Polymode, and
 > performance will be back where it was before.

I have removed bury-buffer from polymode and I will make sure not to get
into this area again. Your explanation was very helpful.

Thanks,

  Vitalie