bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[Stefan-W. Hahn]

Hello,

running following emacs version

origin/emacs-25 emacs-25 cf1f9852d0e8d571dfe74486c26417828faa945a
Author:     Noam Postavsky <npostavs@gmail.com>
AuthorDate: Tue Dec 20 21:43:46 2016 -0500

I have the following problem:

For showing some information in compile buffer I have a function which
uses the hooks compilation-start-hook and compilation-filter-hook to
scan the compile buffer while compiling and adding an overlay together
with information of the compilation at the end of the compile buffer.

The compile buffer has
  truncate-lines t

And the function to add the overlay is like:

(defun xx ()
  (interactive)
  (let ((text (propertize
                (concat
                (format "Already done %d test (%d positive, %d negative, %d undecided).\n"
                        0 0 0 0))
               'face 'highlight))
        (ov (make-overlay (point-min) (point-min) nil 'front-advance)))
    (overlay-put ov 'after-string text)
    (goto-char (point-max))
    (move-overlay ov (point-at-bol) (point-at-eol))))

Together with an content of a buffer like:

--- snipp
    xxxx xxxxxx xxxxxx `xxxxxxxx_xxxxx_xxxx.xxx'.
./../../xxxxx/xxxxxx xxxxxxxxxxxxx.xxx xxxxxxx.xxx xxxxxxx.xxx xxxxxxxxx.xxx xxxxx.xxx xxxxxxxxxxxx.xxx xxxxxxxxx.xxx xxxxxxxxxx.xxx xxx_xxxx.xxx xxxxxxxxx.xxx xxxxxxxxxx.xxx xxxxxxxxxx.xxxxxxxxxxxxx.xxxxxxxxxxxxx.xxxxxxxxxxxxx.xxxxxxxxxxxx
--- snipp

The last line is longer then the screen width, so truncated.

If running the above defined funtion xx in this buffer (with emacs -Q), the
buffer is not responsible any more afterwards.

If truncate-lines is nil, it works.

The problem happens on Windows 7 and Linux both in 64-bit compiled emacs version.
On Windows I tested this back till 

commit d35d398bdbeb393f3ebf17918d82c7573562f01e
Author: John Wiegley <johnw@newartisans.com>
Date:   Tue Aug 2 16:55:16 2016 -0700


Running the above on Linux 64 Bit with commit a3487a8 from Dec 2 and emacs -Q,
attaching gdb to the not responding process gets following backtrace:

--- snipp
0x00000000005caedc in Fassq (key=key@entry=12192, list=<optimised out>, list@entry=45444787) at fns.c:1437
1437	      QUIT;
(gdb) backtrace
#0  0x00000000005caedc in Fassq (key=key@entry=12192, list=<optimised out>, list@entry=45444787)
    at fns.c:1437
#1  0x0000000000424717 in store_in_alist (val=0, prop=12192, alistptr=<synthetic pointer>) at frame.c:2342
#2  Fframe_parameters (frame=frame@entry=19550117) at frame.c:2573
#3  0x0000000000425287 in Fframe_parameter (frame=<optimised out>, parameter=parameter@entry=46176)
    at frame.c:2641
#4  0x0000000000459e56 in x_consider_frame_title (frame=19550117) at xdisp.c:11697
#5  0x0000000000478de5 in redisplay_window (window=19557973, just_this_one_p=just_this_one_p@entry=true)
    at xdisp.c:17142
#6  0x000000000047c90e in redisplay_window_1 (window=window@entry=19557973) at xdisp.c:14495
#7  0x00000000005c1769 in internal_condition_case_1 (bfun=bfun@entry=0x47c8e0 <redisplay_window_1>, 
    arg=19557973, handlers=<optimised out>, hfun=hfun@entry=0x4317c0 <redisplay_window_error>) at eval.c:1338
#8  0x000000000046709e in redisplay_internal () at xdisp.c:14120
#9  0x0000000000469ff5 in redisplay () at xdisp.c:13255
#10 0x0000000000536021 in read_char (commandflag=1, map=map@entry=18151331, prev_event=0, 
    used_mouse_menu=used_mouse_menu@entry=0x7fff2925f61b, end_time=end_time@entry=0x0) at keyboard.c:2482
#11 0x0000000000538b16 in read_key_sequence (keybuf=keybuf@entry=0x7fff2925f770, prompt=prompt@entry=0, 
    dont_downcase_last=dont_downcase_last@entry=false, 
    can_return_switch_frame=can_return_switch_frame@entry=true, 
    fix_current_buffer=fix_current_buffer@entry=true, prevent_redisplay=false, bufsize=30) at keyboard.c:9068
#12 0x000000000053a876 in command_loop_1 () at keyboard.c:1370
#13 0x00000000005c1601 in internal_condition_case (bfun=bfun@entry=0x53a670 <command_loop_1>, 
    handlers=handlers@entry=19104, hfun=hfun@entry=0x52d150 <cmd_error>) at eval.c:1314
#14 0x0000000000529a9c in command_loop_2 (ignore=ignore@entry=0) at keyboard.c:1112
#15 0x00000000005c14ec in internal_catch (tag=tag@entry=46320, func=func@entry=0x529a80 <command_loop_2>, 
    arg=arg@entry=0) at eval.c:1079
#16 0x0000000000529a59 in command_loop () at keyboard.c:1091
#17 0x000000000052ec04 in recursive_edit_1 () at keyboard.c:697
#18 Frecursive_edit () at keyboard.c:768
#19 0x0000000000414b60 in main (argc=2, argv=0x7fff2925fb38) at emacs.c:1629
--- snipp

Following information is from emacs running on Windows 7:

In GNU Emacs 25.1.90.2 (x86_64-w64-mingw32)
 of 2016-12-22 built on MD12GA7C
Repository revision: cf1f9852d0e8d571dfe74486c26417828faa945a
Windowing system distributor 'Microsoft Corp.', version 6.1.7601
Configured using:
 'configure --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32
 --build=x86_64-w64-mingw32 --with-wide-int --with-jpeg --with-xpm
 --with-png --with-tiff --with-rsvg --with-xml2 --with-gnutls --with-xft
 --without-imagemagick 'CFLAGS=-static -g3 -O3'
 GIT_VERSION=emacs-24.5-rc3-fixed-8567-gcf1f985'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS

Important settings:
  value of $LANG: en_GB.iso-8859-1
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Making completion list...

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message dired format-spec rfc822 mml
mml-sec password-cache epg epg-config gnus-util mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util help-fns help-mode easymenu
cl-loaddefs pcase cl-lib mail-prsvr mail-utils time-date mule-util
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel dos-w32 ls-lisp disp-table w32-win w32-vars term/common-win
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote w32notify w32 multi-tty
make-network-process emacs)

Memory information:
((conses 16 91337 6635)
 (symbols 56 20392 0)
 (miscs 48 47 148)
 (strings 32 17043 4347)
 (string-bytes 1 446534)
 (vectors 16 11864)
 (vector-slots 8 431279 5382)
 (floats 8 159 77)
 (intervals 56 255 13)
 (buffers 976 22))

If more information is needed please tell me.

If my code to have an overlay at the end of compile buffer is wrong please
tell me too.

With kind regards,
Stefan

-- 
Stefan-W. Hahn                          It is easy to make things.
                                        It is hard to make things simple.

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[npostavs]

tags 25246 confirmed
quit

"Stefan-W. Hahn" <stefan.hahn@s-hahn.de> writes:

>
> I have the following problem:
>
> For showing some information in compile buffer I have a function which
> uses the hooks compilation-start-hook and compilation-filter-hook to
> scan the compile buffer while compiling and adding an overlay together
> with information of the compilation at the end of the compile buffer.
>
> The compile buffer has
>   truncate-lines t
>
> And the function to add the overlay is like:
>
> (defun xx ()
>   (interactive)
>   (let ((text (propertize
>                 (concat
>                 (format "Already done %d test (%d positive, %d negative, %d undecided).\n"
>                         0 0 0 0))
>                'face 'highlight))
>         (ov (make-overlay (point-min) (point-min) nil 'front-advance)))
>     (overlay-put ov 'after-string text)
>     (goto-char (point-max))
>     (move-overlay ov (point-at-bol) (point-at-eol))))
>
> Together with an content of a buffer like:
>
> --- snipp
>     xxxx xxxxxx xxxxxx `xxxxxxxx_xxxxx_xxxx.xxx'.
> ./../../xxxxx/xxxxxx xxxxxxxxxxxxx.xxx xxxxxxx.xxx xxxxxxx.xxx xxxxxxxxx.xxx xxxxx.xxx xxxxxxxxxxxx.xxx xxxxxxxxx.xxx xxxxxxxxxx.xxx xxx_xxxx.xxx xxxxxxxxx.xxx xxxxxxxxxx.xxx xxxxxxxxxx.xxxxxxxxxxxxx.xxxxxxxxxxxxx.xxxxxxxxxxxxx.xxxxxxxxxxxx
> --- snipp
>
> The last line is longer then the screen width, so truncated.
>
> If running the above defined funtion xx in this buffer (with emacs -Q), the
> buffer is not responsible any more afterwards.
>
> If truncate-lines is nil, it works.

I think this is a similar case to #24633 "highlight-region func using
(window-hscroll) in :align-to spec can cause inf loop".  If you try your
recipe in the master branch, Emacs won't freeze, although the overlay
you put blinks in and out of visibility on every redisplay (trigged by
cursor blinking).

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[Eli Zaretskii]

> Date: Thu, 22 Dec 2016 10:13:05 +0100
> From: "Stefan-W. Hahn" <stefan.hahn@s-hahn.de>
> 
> For showing some information in compile buffer I have a function which
> uses the hooks compilation-start-hook and compilation-filter-hook to
> scan the compile buffer while compiling and adding an overlay together
> with information of the compilation at the end of the compile buffer.
> 
> The compile buffer has
>   truncate-lines t
> 
> And the function to add the overlay is like:
> 
> (defun xx ()
>   (interactive)
>   (let ((text (propertize
>                 (concat
>                 (format "Already done %d test (%d positive, %d negative, %d undecided).\n"
>                         0 0 0 0))
>                'face 'highlight))
>         (ov (make-overlay (point-min) (point-min) nil 'front-advance)))
>     (overlay-put ov 'after-string text)
>     (goto-char (point-max))
>     (move-overlay ov (point-at-bol) (point-at-eol))))
> 
> Together with an content of a buffer like:
> 
> --- snipp
>     xxxx xxxxxx xxxxxx `xxxxxxxx_xxxxx_xxxx.xxx'.
> ./../../xxxxx/xxxxxx xxxxxxxxxxxxx.xxx xxxxxxx.xxx xxxxxxx.xxx xxxxxxxxx.xxx xxxxx.xxx xxxxxxxxxxxx.xxx xxxxxxxxx.xxx xxxxxxxxxx.xxx xxx_xxxx.xxx xxxxxxxxx.xxx xxxxxxxxxx.xxx xxxxxxxxxx.xxxxxxxxxxxxx.xxxxxxxxxxxxx.xxxxxxxxxxxxx.xxxxxxxxxxxx
> --- snipp
> 
> The last line is longer then the screen width, so truncated.
> 
> If running the above defined funtion xx in this buffer (with emacs -Q), the
> buffer is not responsible any more afterwards.
> 
> If truncate-lines is nil, it works.

Could you please show a complete recipe, starting from "emacs -Q"?
I'm afraid I don't understand from your description how to reproduce
the problem, in order to debug it.

It should be enough to provide some text to serve as buffer contents,
and a Lisp function to put the overlay on that text, such that Emacs
stops responding.

Thanks.

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 377 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:
>
> It should be enough to provide some text to serve as buffer contents,
> and a Lisp function to put the overlay on that text, such that Emacs
> stops responding.

I think the OP already provided those.  Here's some elisp code to
reproduce the problem based on the description (some adjustment may be
needed depending on display width):


[-- Attachment #2: bug demo --]
[-- Type: application/emacs-lisp, Size: 945 bytes --]

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[Stefan-W. Hahn]

Mail von npostavs@users.sourceforge.net, Thu, 22 Dec 2016 at 19:01:56 -0500:

Hello,

> > It should be enough to provide some text to serve as buffer contents,
> > and a Lisp function to put the overlay on that text, such that Emacs
> > stops responding.
> 
> I think the OP already provided those.  Here's some elisp code to
> reproduce the problem based on the description (some adjustment may be
> needed depending on display width):

thanks for the programatically recipe. I will do it better next time.
And yes this shows the bad behaviour.

Also I can confirm that with version from master branch, namely

@ origin/master origin/HEAD a978d300a3faf58ee6e94ba57f764ca99a9ec308
Author:     Mark Oteiza <mvoteiza@udel.edu>
AuthorDate: Thu Dec 22 19:09:01 2016 -0500

the problem has gone on both systems, Linux and Windows.

(Little odd ting is the blinking of the overlay, but it's ok.)

With kind regards,
Stefan

-- 
Stefan-W. Hahn                          It is easy to make things.
                                        It is hard to make things simple.

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[Eli Zaretskii]

> Date: Fri, 23 Dec 2016 08:33:15 +0100
> From: "Stefan-W. Hahn" <stefan.hahn@s-hahn.de>
> Cc: Eli Zaretskii <eliz@gnu.org>, 25246@debbugs.gnu.org
> 
> > > It should be enough to provide some text to serve as buffer contents,
> > > and a Lisp function to put the overlay on that text, such that Emacs
> > > stops responding.
> > 
> > I think the OP already provided those.  Here's some elisp code to
> > reproduce the problem based on the description (some adjustment may be
> > needed depending on display width):
> 
> thanks for the programatically recipe. I will do it better next time.

It sounds like you provided that in your original report, but I didn't
understand it.  Sorry about that.

> Also I can confirm that with version from master branch, namely
> 
> @ origin/master origin/HEAD a978d300a3faf58ee6e94ba57f764ca99a9ec308
> Author:     Mark Oteiza <mvoteiza@udel.edu>
> AuthorDate: Thu Dec 22 19:09:01 2016 -0500
> 
> the problem has gone on both systems, Linux and Windows.
> 
> (Little odd ting is the blinking of the overlay, but it's ok.)

No, the problem is not gone in the master branch.  It's just that
master has a safety device, whereby we stop the infinite loop after a
few iterations.  Those iterations is that "blinking" you see.

I will look into this soon.  One thing I already saw: this is a very
old problem, I see it in Emacs 22.1.

Thanks.

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[Eli Zaretskii]

> Date: Fri, 23 Dec 2016 11:21:00 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 25246@debbugs.gnu.org, npostavs@users.sourceforge.net
> 
> I will look into this soon.  One thing I already saw: this is a very
> old problem, I see it in Emacs 22.1.

If the change below, when applied to the master branch, gives good
results, I will install it.

The results as I see after this change are not ideal (you cannot see
the entire overlay string, unless you manually hscroll the window with
"C-x >", and the cursor at EOB is not visible).  But at least the loop
is avoided, AFAICT, and there's only one "blink" of incorrect display.
Is this satisfactory enough to install this simple change?

Thanks.

diff --git a/src/xdisp.c b/src/xdisp.c
index ad0b968..37ca81d 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -13049,6 +13049,17 @@ hscroll_window_tree (Lisp_Object window)
 	      init_to_row_start (&it, w, cursor_row);
 	      it.last_visible_x = INFINITY;
 	      move_it_in_display_line_to (&it, pt, -1, MOVE_TO_POS);
+	      /* If the line ends in an overlay string with a newline,
+		 we might infloop, because displaying the window will
+		 want to put the cursor after the overlay, i.e. at X
+		 coordinate of zero on the next screen line.  So we
+		 use the buffer position prior to the overlay string
+		 instead.  */
+	      if (it.method == GET_FROM_STRING && pt > 1)
+		{
+		  init_to_row_start (&it, w, cursor_row);
+		  move_it_in_display_line_to (&it, pt - 1, -1, MOVE_TO_POS);
+		}
 	      current_buffer = saved_current_buffer;
 
 	      /* Position cursor in window.  */

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[Stefan-W. Hahn]

Mail von Eli Zaretskii, Fri, 23 Dec 2016 at 13:08:57 +0200:

Hello,

> If the change below, when applied to the master branch, gives good
> results, I will install it.

tried this on Linux and Windows. Works really good.

> The results as I see after this change are not ideal (you cannot see
> the entire overlay string, unless you manually hscroll the window with
> "C-x >", and the cursor at EOB is not visible).  But at least the loop

That is no problem, because the overlay moves if compilation proceeds and
the compilation buffer gets more content.

The problem only arises if a single line of output in compilation buffer was
bigger then the screen width (and I have truncates-lines set to t) and had
no LF. In all other cases there was no problem. I saw this just now,,
because our production environment is producing such long lines since not so
long time.

> is avoided, AFAICT, and there's only one "blink" of incorrect display.
> Is this satisfactory enough to install this simple change?

As said. It works very well.

Thank you.

With kind regards,
Stefan

> 
> Thanks.
> 
> diff --git a/src/xdisp.c b/src/xdisp.c
> index ad0b968..37ca81d 100644
> --- a/src/xdisp.c
> +++ b/src/xdisp.c
> @@ -13049,6 +13049,17 @@ hscroll_window_tree (Lisp_Object window)
>  	      init_to_row_start (&it, w, cursor_row);
>  	      it.last_visible_x = INFINITY;
>  	      move_it_in_display_line_to (&it, pt, -1, MOVE_TO_POS);
> +	      /* If the line ends in an overlay string with a newline,
> +		 we might infloop, because displaying the window will
> +		 want to put the cursor after the overlay, i.e. at X
> +		 coordinate of zero on the next screen line.  So we
> +		 use the buffer position prior to the overlay string
> +		 instead.  */
> +	      if (it.method == GET_FROM_STRING && pt > 1)
> +		{
> +		  init_to_row_start (&it, w, cursor_row);
> +		  move_it_in_display_line_to (&it, pt - 1, -1, MOVE_TO_POS);
> +		}
>  	      current_buffer = saved_current_buffer;
>  
>  	      /* Position cursor in window.  */
> 

-- 
Stefan-W. Hahn                          It is easy to make things.
                                        It is hard to make things simple.

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[npostavs]

"Stefan-W. Hahn" <stefan.hahn@s-hahn.de> writes:
>
> That is no problem, because the overlay moves if compilation proceeds and
> the compilation buffer gets more content.

By the way, I wonder if your use case might be better served by setting
`header-line-format'.

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[Eli Zaretskii]

> Date: Fri, 23 Dec 2016 14:36:47 +0100
> From: "Stefan-W. Hahn" <stefan.hahn@s-hahn.de>
> Cc: 25246@debbugs.gnu.org, npostavs@users.sourceforge.net
> 
> > If the change below, when applied to the master branch, gives good
> > results, I will install it.
> 
> tried this on Linux and Windows. Works really good.

Thanks for testing.

> The problem only arises if a single line of output in compilation buffer was
> bigger then the screen width (and I have truncates-lines set to t) and had
> no LF.

Yes, this is because the after-string ends in a newline, so it causes
the cursor to be displayed in a different screen line.

> > Is this satisfactory enough to install this simple change?
> 
> As said. It works very well.

OK, thanks.  Noam, do you agree?

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 23 Dec 2016 14:36:47 +0100
>> From: "Stefan-W. Hahn" <stefan.hahn@s-hahn.de>
>> Cc: 25246@debbugs.gnu.org, npostavs@users.sourceforge.net
>> 
>> > If the change below, when applied to the master branch, gives good
>> > results, I will install it.
>> 
>> tried this on Linux and Windows. Works really good.
>
> Thanks for testing.
>
>> The problem only arises if a single line of output in compilation buffer was
>> bigger then the screen width (and I have truncates-lines set to t) and had
>> no LF.
>
> Yes, this is because the after-string ends in a newline, so it causes
> the cursor to be displayed in a different screen line.
>
>> > Is this satisfactory enough to install this simple change?
>> 
>> As said. It works very well.
>
> OK, thanks.  Noam, do you agree?

Yes, this looks like a clear improvement.

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[Stefan-W. Hahn]

Mail von npostavs@users.sourceforge.net, Fri, 23 Dec 2016 at 08:53:53 -0500:

Hello,

> By the way, I wonder if your use case might be better served by setting
> `header-line-format'.

I use this too as summary of runtime and summary. The moving overlay at the
end contains normaly more then one line.

Stefan

-- 
Stefan-W. Hahn                          It is easy to make things.
                                        It is hard to make things simple.

bug#25246: 25.1.90; Buffer not responsible with overlay at buffer end.

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: "Stefan-W. Hahn" <stefan.hahn@s-hahn.de>,  25246@debbugs.gnu.org
> Date: Fri, 23 Dec 2016 09:14:02 -0500
> 
> >> > Is this satisfactory enough to install this simple change?
> >> 
> >> As said. It works very well.
> >
> > OK, thanks.  Noam, do you agree?
> 
> Yes, this looks like a clear improvement.

Thanks, pushed to master.

I'm marking this bug done; feel free to reopen if it causes problems.