bug#17244: 24.3.90; `line-move-visual' errors when moving across wrapped lines with an overlay property of 'display

bug#17244: 24.3.90; `line-move-visual' errors when moving across wrapped lines with an overlay property of 'display

[Ivan Andrus]

[-- Attachment #1: Type: text/plain, Size: 4045 bytes --]

Starting from Emacs -Q, create a buffer with the following contents.
Notice that the first two (2) lines are blank, and the next is one long
line (in case it gets broken by email)

#### buffer start:


;; I think this needs to be really long here.  Why?  I don't really know,
but a guess.  I think this needs to be really long here.  Why?  I don't
really know, but a guess.  I think this needs to be really long here.  Why?
 I don't really know, but a guess.  I think this needs to be really long
here.  Why?  I don't really know, but a guess.  I think this needs to be
really long here.  Why?  I don't really know, but a guess.  I think this
needs to be really long here.  Why?  I don't really know, but a guess.
(let ((ov (make-overlay 3 515)))
 (overlay-put ov 'display "hi\n"))
(line-move-visual -5)

1. Make sure the window is skinny enough that the comment line wraps.
2. Evaluate the first form, creating an overlay.
3. Evaluate the second form.  It is important that point be at the end
   of the "line-move-visual" line.
4. You get a warning about beginning-of-buffer, when it clearly did not
   reach the beginning-of-buffer.

As near as I can tell, the incorrect error message is because it doesn't
move as far at it thinks it should (the real bug), and it assumes the
reason it didn't is because it made it to the beginning (or end) of the
buffer.

-Ivan

In GNU Emacs 24.3.90.1 (x86_64-apple-darwin13.1.0, NS apple-appkit-1265.19)
of 2014-04-11 on iandrus-osx
Windowing system distributor `Apple', version 10.3.1265
Configured using:
`configure --with-ns'

Important settings:
 value of $LANG: en_US.UTF-8
 locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
 tooltip-mode: t
 electric-indent-mode: t
 mouse-wheel-mode: t
 tool-bar-mode: t
 menu-bar-mode: t
 file-name-shadow-mode: t
 global-font-lock-mode: t
 font-lock-mode: t
 blink-cursor-mode: t
 auto-composition-mode: t
 auto-encryption-mode: t
 auto-compression-mode: t
 line-number-mode: t
 transient-mark-mode: t

Recent input:
<down-mouse-1> <mouse-1> C-x C-e <down> C-x C-e <down-mouse-1>
<mouse-1> <up> <up> <up> <up> <up> <down> <down> <up>
<up> <down> <down> <up> <up> <down> <down> <down> <down>
<down> <up> <up> <up> <up> <up> <down> <down> <down>
<down> <down> <up> <up> <up> <up> <up> <down> <down>
<down> <down> <down> <down> C-x C-e <down> <down> <down>
<down> C-e C-x C-e <down-mouse-1> <mouse-1> <return>
SPC SPC SPC SPC SPC C-x C-e C-/ C-/ <help-echo> C-c
C-c s-x C-/ M-x m a i <tab> <C-backspace> r e p o <tab>
r t <tab> <return>

Recent messages:
line-move-visual: Beginning of buffer
Beginning of buffer [5 times]
t
line-move-visual: Beginning of buffer
t
Undo! [2 times]
C-c C-c is undefined
Undo!
Making completion list...
C-c C-c is undefined

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev
gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util
help-fns mail-prsvr mail-utils help-mode easymenu vc-hg time-date
tooltip electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
ns-win tool-bar dnd fontset image regexp-opt fringe tabulated-list
newcomment lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer nadvice loaddefs button faces cus-face macroexp
files text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
cocoa ns multi-tty emacs)

Memory information:
((conses 16 74077 4946)
(symbols 48 17641 0)
(miscs 40 49 163)
(strings 32 10552 3708)
(string-bytes 1 281155)
(vectors 16 9174)
(vector-slots 8 367902 7774)
(floats 8 54 455)
(intervals 56 212 7)
(buffers 960 13))

[-- Attachment #2: Type: text/html, Size: 5225 bytes --]

bug#17244: 24.3.90; `line-move-visual' errors when moving across wrapped lines with an overlay property of 'display

[Eli Zaretskii]

> Date: Fri, 11 Apr 2014 15:28:55 -0600
> From: Ivan Andrus <darthandrus@gmail.com>
> 
> Starting from Emacs -Q, create a buffer with the following contents.
> Notice that the first two (2) lines are blank, and the next is one long
> line (in case it gets broken by email)
> 
> #### buffer start:
> 
> 
> ;; I think this needs to be really long here.  Why?  I don't really know,
> but a guess.  I think this needs to be really long here.  Why?  I don't
> really know, but a guess.  I think this needs to be really long here.  Why?
>  I don't really know, but a guess.  I think this needs to be really long
> here.  Why?  I don't really know, but a guess.  I think this needs to be
> really long here.  Why?  I don't really know, but a guess.  I think this
> needs to be really long here.  Why?  I don't really know, but a guess.
> (let ((ov (make-overlay 3 515)))
>  (overlay-put ov 'display "hi\n"))
> (line-move-visual -5)
> 
> 1. Make sure the window is skinny enough that the comment line wraps.
> 2. Evaluate the first form, creating an overlay.
> 3. Evaluate the second form.  It is important that point be at the end
>    of the "line-move-visual" line.
> 4. You get a warning about beginning-of-buffer, when it clearly did not
>    reach the beginning-of-buffer.
> 
> As near as I can tell, the incorrect error message is because it doesn't
> move as far at it thinks it should (the real bug), and it assumes the
> reason it didn't is because it made it to the beginning (or end) of the
> buffer.

Thanks for the report.  I fixed this in revision 116947 on the
emacs-24 branch.

<rant>
To tell the truth, Lisp code which covers large portions of buffer
text with much shorter display strings that include newlines deserves
to be broken.  The current Emacs display engine was never designed to
handle situations where the displayed text is so starkly different
from buffer text, so the result of trying to fix "bugs" such as this
one is a never-ending series of band-aids, one upon the other, which
make the code utterly incomprehensible and unmaintainable.

So I'm this close to refusing to fix such "bugs", and instead asking
the authors of such Lisp to either find more benign ways of expressing
what they need, or work around the limitations of the display engine
in their own Lisp.
</rant>

bug#17244: 24.3.90; `line-move-visual' errors when moving across wrapped lines with an overlay property of 'display

[Ivan Andrus]

On Apr 12, 2014, at 5:31 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> Thanks for the report.  I fixed this in revision 116947 on the
> emacs-24 branch.

Thanks for fixing this Eli.

> <rant>
> To tell the truth, Lisp code which covers large portions of buffer
> text with much shorter display strings that include newlines deserves
> to be broken.  The current Emacs display engine was never designed to
> handle situations where the displayed text is so starkly different
> from buffer text, so the result of trying to fix "bugs" such as this
> one is a never-ending series of band-aids, one upon the other, which
> make the code utterly incomprehensible and unmaintainable.
> 
> So I'm this close to refusing to fix such "bugs", and instead asking
> the authors of such Lisp to either find more benign ways of expressing
> what they need, or work around the limitations of the display engine
> in their own Lisp.
> </rant>	

So is the problem having newlines in the 'display property, or hiding large portions of the buffer?  Because, in my original use case (my modified version of http://www.emacswiki.org/emacs/fold.el) I don’t think I have newlines in the display property, though I definitely hide large portions of the buffer.  The overlays are also nested, so it might it be related to that?

That said, your change seems to have fixed it.  

Thanks again,
Ivan

bug#17244: 24.3.90; `line-move-visual' errors when moving across wrapped lines with an overlay property of 'display

[Eli Zaretskii]

> From: Ivan Andrus <darthandrus@gmail.com>
> Date: Sun, 13 Apr 2014 19:43:08 -0600
> Cc: 17244-done@debbugs.gnu.org
> 
> > <rant>
> > To tell the truth, Lisp code which covers large portions of buffer
> > text with much shorter display strings that include newlines deserves
> > to be broken.  The current Emacs display engine was never designed to
> > handle situations where the displayed text is so starkly different
> > from buffer text, so the result of trying to fix "bugs" such as this
> > one is a never-ending series of band-aids, one upon the other, which
> > make the code utterly incomprehensible and unmaintainable.
> > 
> > So I'm this close to refusing to fix such "bugs", and instead asking
> > the authors of such Lisp to either find more benign ways of expressing
> > what they need, or work around the limitations of the display engine
> > in their own Lisp.
> > </rant>	
> 
> So is the problem having newlines in the 'display property, or hiding large portions of the buffer?

Each one of these is a problem.  When they are present together in the
same display string, the problems grow exponentially.

> Because, in my original use case (my modified version of http://www.emacswiki.org/emacs/fold.el) I don’t think I have newlines in the display property, though I definitely hide large portions of the buffer.

This is enough to cause problems.  Consider what Emacs needs to do
when you invoke "C-u N C-p" or "(line-move-visual -N)".  It needs to
find out what is the buffer position N screen lines before point.  But
where in the buffer is that?  When those N lines display what is
mostly buffer text, we can go back the appropriate amount of physical
lines, and then start looking from there.  But if what's on display in
those N lines has little if any resemblance to what's in the buffer,
the only safe algorithm is to go to the beginning of the buffer and
start from there, which is prohibitively expensive.  So Emacs
implements several optimizations to make the operations reasonably
fast.  These optimizations only succeed when the display strings and
the text they cover are approximately similar in length, or if the
display strings are very short.  The specific optimization that caused
this bug was an attempt to be faster when text lines in a buffer are
very long.

> The overlays are also nested, so it might it be related to that?

In what way are they nested?  The problems happen only with overlays
that have the 'display' property.

> That said, your change seems to have fixed it.  

Thanks for testing.  I must ask, though: why do you need to use
display strings (rather than, say, invisible property) to do folding?

bug#17244: 24.3.90; `line-move-visual' errors when moving across wrapped lines with an overlay property of 'display

[Ivan Andrus]

[-- Attachment #1: Type: text/plain, Size: 2948 bytes --]

On Apr 14, 2014, at 2:11 AM, Eli Zaretskii <eliz@gnu.org> wrote:

From: Ivan Andrus <darthandrus@gmail.com>
Date: Sun, 13 Apr 2014 19:43:08 -0600
Cc: 17244-done@debbugs.gnu.org

<rant>
To tell the truth, Lisp code which covers large portions of buffer
text with much shorter display strings that include newlines deserves
to be broken.  The current Emacs display engine was never designed to
handle situations where the displayed text is so starkly different
from buffer text, so the result of trying to fix "bugs" such as this
one is a never-ending series of band-aids, one upon the other, which
make the code utterly incomprehensible and unmaintainable.

So I'm this close to refusing to fix such "bugs", and instead asking
the authors of such Lisp to either find more benign ways of expressing
what they need, or work around the limitations of the display engine
in their own Lisp.
</rant>


So is the problem having newlines in the 'display property, or hiding large
portions of the buffer?


Each one of these is a problem.  When they are present together in the
same display string, the problems grow exponentially.

Because, in my original use case (my modified version of
http://www.emacswiki.org/emacs/fold.el) I don’t think I have newlines in
the display property, though I definitely hide large portions of the buffer.


This is enough to cause problems.  Consider what Emacs needs to do
when you invoke "C-u N C-p" or "(line-move-visual -N)".  It needs to
find out what is the buffer position N screen lines before point.  But
where in the buffer is that?  When those N lines display what is
mostly buffer text, we can go back the appropriate amount of physical
lines, and then start looking from there.  But if what's on display in
those N lines has little if any resemblance to what's in the buffer,
the only safe algorithm is to go to the beginning of the buffer and
start from there, which is prohibitively expensive.  So Emacs
implements several optimizations to make the operations reasonably
fast.  These optimizations only succeed when the display strings and
the text they cover are approximately similar in length, or if the
display strings are very short.  The specific optimization that caused
this bug was an attempt to be faster when text lines in a buffer are
very long.


I see.  Thanks for explaining.

The overlays are also nested, so it might it be related to that?


In what way are they nested?  The problems happen only with overlays
that have the 'display' property.


Well, only the outermost have a display property, so I guess it doesn’t
apply.

That said, your change seems to have fixed it.


Thanks for testing.  I must ask, though: why do you need to use
display strings (rather than, say, invisible property) to do folding?


Because I like to have it show how many lines are hidden.  So it’s not
necessary, but nice.

-Ivan

[-- Attachment #2: Type: text/html, Size: 3833 bytes --]