bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Vitalie Spinu]

Hi,

 (defun insert-with-fringe ()
   (insert
    "      "
    (concat
     (propertize "fringe" 'display (list 'left-fringe 'filled-square font-lock-doc-face))
     (propertize "dummy" 'display "AAA\n" 'font-lock-face font-lock-doc-face))
    "some more text")
   (previous-line))

Execute (insert-with-fringe). It should position the cursor just in front of the
"AAA" string.

Now (backward-char 1) or (goto-char (1- (point))) don't have any effect, and
(previous-line) goes to bol instead of the previous line.

It has to do with the trailing "\n" in "AAA\n". If that new line is removed,
everything works as expected.


  Vitalie


In GNU Emacs 25.0.50.4 (x86_64-unknown-linux-gnu, GTK+ Version 3.12.2)
 of 2015-08-19 on galago
Repository revision: 481859ba71253725f4aed4877b89123e11aaef0c
Windowing system distributor `The X.Org Foundation', version 11.0.11600000
System Description:	Ubuntu 14.10

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Date: Fri, 28 Aug 2015 17:46:54 +0200
> 
>  (defun insert-with-fringe ()
>    (insert
>     "      "
>     (concat
>      (propertize "fringe" 'display (list 'left-fringe 'filled-square font-lock-doc-face))
>      (propertize "dummy" 'display "AAA\n" 'font-lock-face font-lock-doc-face))
>     "some more text")
>    (previous-line))
> 
> Execute (insert-with-fringe). It should position the cursor just in front of the
> "AAA" string.
> 
> Now (backward-char 1) or (goto-char (1- (point))) don't have any effect, and
> (previous-line) goes to bol instead of the previous line.

The first two do work, you just need to invoke backward-char twice to
see the cursor move.  But "C-x =" will show you that backward-char did
move even after the first time.  The cursor doesn't move the first
time because you have buffer positions covered by a display property
that is displayed on the fringe, and the first backward-char moves
into those buffer positions.  This is nothing new: you can see it
whenever you use the fringe property without all the rest in this
scenario.  So this part of the behavior is not a bug.

The behavior of previous-line is indeed a bug, and it is hard to fix.

> It has to do with the trailing "\n" in "AAA\n". If that new line is removed,
> everything works as expected.

No, it's because you have 2 consecutive display properties, _and_ the
second one ends in a newline.  There's code in vertical-motion that
attempts to avoid the lossage when a display property includes
newlines, but the fringe display property before that defeats that
code.

Is there some important real-life use case that needs this to work?
Otherwise, I'm inclined to leave this alone.  After all, it's not a
catastrophe: the next call to previous-line will go to the line you
want.

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Vitalie Spinu]

>> On Fri, Aug 28 2015 22:45, Eli Zaretskii wrote:

> The first two do work, you just need to invoke backward-char twice to
> see the cursor move.  

This is not happening with `M-x (backward-char 1)`. It does indeed work when
backward-char is invoked interactively with C-b.

This is a relatively recent change in behavior (past 2-3 months or so).

> Is there some important real-life use case that needs this to work?
> Otherwise, I'm inclined to leave this alone.  After all, it's not a
> catastrophe: the next call to previous-line will go to the line you
> want.

I have encountered this issue in 3 different projects. One is the implementation
of visual breakpoints in ESS which look like this:

   https://ess-tracebug.googlecode.com/svn/trunk/img/breakpoint_types.PNG


For years I used to have a defadvice workaround along the following lines:

     ...
     (when (and (eq major-mode 'ess-mode)
                (get-text-property (point) 'intangible))
      (backward-char 1))
     add-do-it 
     ...

With the new emacs dev the backward-char stopped working. That's why I am
finally reporting this bug.


Another real life use case is the yet unfinished image-display package where I
attempted to build a grid of images with intangible segments and simply rely on
emacs motion commands for navigation between cells. This didn't work properly
because the vertical motion was jumping columns.

Another example is a spredsheet application for data manipulation where I have
read-only cells which I want to navigate with standard emacs motion
commands. Same design as above and the same problem.


   Vitalie

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Cc: 21368@debbugs.gnu.org
> Date: Sat, 29 Aug 2015 00:02:22 +0200
> 
> >> On Fri, Aug 28 2015 22:45, Eli Zaretskii wrote:
> 
> > The first two do work, you just need to invoke backward-char twice to
> > see the cursor move.  
> 
> This is not happening with `M-x (backward-char 1)`. It does indeed work when
> backward-char is invoked interactively with C-b.
> 
> This is a relatively recent change in behavior (past 2-3 months or so).

What is a relatively recent change?  The C-b part or the M-x part?

> > Is there some important real-life use case that needs this to work?
> > Otherwise, I'm inclined to leave this alone.  After all, it's not a
> > catastrophe: the next call to previous-line will go to the line you
> > want.
> 
> I have encountered this issue in 3 different projects. One is the implementation
> of visual breakpoints in ESS which look like this:
> 
>    https://ess-tracebug.googlecode.com/svn/trunk/img/breakpoint_types.PNG
> 
> For years I used to have a defadvice workaround along the following lines:
>      ...
>      (when (and (eq major-mode 'ess-mode)
>                 (get-text-property (point) 'intangible))
>       (backward-char 1))
>      add-do-it 
>      ...
> 
> With the new emacs dev the backward-char stopped working. That's why I am
> finally reporting this bug.

Can't you use an overlay to show the bitmap on the fringe?  IOW,
instead of the first display property, define an empty overlay
(i.e. overlay whose start and end positions are equal) with a
before-string that is propertized with a 'fringe' display property.
Here's an example, based on your original scenario:

  (defun insert-with-fringe ()
    (let ((pt (point))
	  ov)
      (insert
       "      "
       (propertize "dummy" 'display "AAA\n" 'font-lock-face font-lock-doc-face)
       "some more text")
      (setq ov (make-overlay (+ pt 5) (+ pt 5)))
      (overlay-put ov 'before-string
		   (propertize "fringe"
			       'display
			       (list 'left-fringe
				     'filled-square font-lock-doc-face)))
      (previous-line)))

This produces the same display as in your example, but has none of its
problems, because the situation with 2 consecutive display strings on
adjacent buffer positions is eliminated.

> Another real life use case is the yet unfinished image-display package where I
> attempted to build a grid of images with intangible segments and simply rely on
> emacs motion commands for navigation between cells. This didn't work properly
> because the vertical motion was jumping columns.
> 
> Another example is a spredsheet application for data manipulation where I have
> read-only cells which I want to navigate with standard emacs motion
> commands. Same design as above and the same problem.

OK, but if using overlays as shown above doesn't fill your needs,
please don't hold your breath for a quick solution.  It's a hard
problem, and the code that handles it is already too convoluted.
Visual line movement is a tough nut when there's a lot of invisible
text around.  (Btw, yet another solution is to disable
line-move-visual in these cases -- would that be a good idea in those
real-life examples?)

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Wolfgang Jenkner]

On Sat, Aug 29 2015, Vitalie Spinu wrote:

>>> On Fri, Aug 28 2015 22:45, Eli Zaretskii wrote:
>
>> The first two do work, you just need to invoke backward-char twice to
>> see the cursor move.  
>
> This is not happening with `M-x (backward-char 1)`. It does indeed work when
> backward-char is invoked interactively with C-b.
>
> This is a relatively recent change in behavior (past 2-3 months or so).


A recent change in behaviour would likely be due to my 68529c8 from Jun
3, which is a fix for vertical-motion.

However, I tried your test case with emacs trunk fd93edb built on
2015-03-27 (since that happens to be the the oldest version which I've
kept around) but it seems to behave in the same way as a recent version
of emacs (84a9787).

In particular, the first backward-char (however invoked) brings point
(which does not visually move) 6 positions back (from the "d" of "dummy"
to the "f" of "fringe"); the next backward-char moves point by one
position back to the last space character before "fringe".

By the way, (vertical-motion 0) instead of (previous-line) brings point
to the next (visual) line; this is perhaps a more direct test case for
the same thing.

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Vitalie Spinu]

>> On Sat, Aug 29 2015 10:25, Eli Zaretskii wrote:

> What is a relatively recent change?  The C-b part or the M-x part?

The M-x part. Aka, invoking `backward-char` from programs.


> Can't you use an overlay to show the bitmap on the fringe?  

Indeed. Thanks. It will surely work for breakpoints use case.

> text around.  (Btw, yet another solution is to disable line-move-visual in
> these cases -- would that be a good idea in those real-life examples?)

So the core of the issue is `line-move-visual`? Disabling `line-move-visual` in
all those rectangular grid scenarios won't be a problem.


   Vitalie

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Vitalie Spinu]

> However, I tried your test case with emacs trunk fd93edb built on 2015-03-27

> In particular, the first backward-char (however invoked) brings point (which
> does not visually move) 6 positions back (from the "d" of "dummy" to the "f"
> of "fringe"); the next backward-char moves point by one position back to the
> last space character before "fringe".

This doesn't happen in current trunk, M-: (backward-char 1) doesn't move the
pointer.

  Vitalie

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Vitalie Spinu]

>> On Sat, Aug 29 2015 21:08, Vitalie Spinu wrote:


>> Can't you use an overlay to show the bitmap on the fringe?  

> Indeed. Thanks. It will surely work for breakpoints use case.

Well, I just recalled why I didn't use overlays when I wrote that visual
breakpoints code about 5 years ago. I wanted the breakpoints to be moved with
the text on copy/paste. With overlays that's not possible.


  Vitalie

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Cc: 21368@debbugs.gnu.org
> Date: Sat, 29 Aug 2015 21:08:10 +0200
> 
> > What is a relatively recent change?  The C-b part or the M-x part?
> 
> The M-x part. Aka, invoking `backward-char` from programs.

I wonder what change that could be.

> > text around.  (Btw, yet another solution is to disable line-move-visual in
> > these cases -- would that be a good idea in those real-life examples?)
> 
> So the core of the issue is `line-move-visual`?

Yes.  Without it, previous-line just goes to the previous physical
line.  With it, Emacs needs to decide where to put point, and that's
not easy when invisible text is involved, because we can only put
point on some buffer position.

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Eli Zaretskii]

> From: Vitalie Spinu <spinuvit@gmail.com>
> Cc: 21368@debbugs.gnu.org
> Date: Sat, 29 Aug 2015 21:17:43 +0200
> 
> Well, I just recalled why I didn't use overlays when I wrote that visual
> breakpoints code about 5 years ago. I wanted the breakpoints to be moved with
> the text on copy/paste. With overlays that's not possible.

You can use overlays only for displaying the breakpoint, and use text
properties to indicate a breakpoint.

Or override the copy/paste commands to reinstate the overlay at the
new place.

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Wolfgang Jenkner]

On Sat, Aug 29 2015, Vitalie Spinu wrote:

>> However, I tried your test case with emacs trunk fd93edb built on 2015-03-27
>
>> In particular, the first backward-char (however invoked) brings point (which
>> does not visually move) 6 positions back (from the "d" of "dummy" to the "f"
>> of "fringe"); the next backward-char moves point by one position back to the
>> last space character before "fringe".
>
> This doesn't happen in current trunk, M-: (backward-char 1) doesn't move the
> pointer.

I guess you tested this in the *scratch* buffer, whereas I tested it in
a fundamental-mode buffer.

In *scratch* point seems indeed to be stuck on the "d" of "dummy" while
in fundamental-mode the behaviour is like I described it above.

However, fd93edb from 2015-03-27 behaves in the same way as trunk does,
in *scratch*, too.

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Vitalie Spinu]

>> On Sat, Aug 29 2015 23:53, Wolfgang Jenkner wrote:

> However, fd93edb from 2015-03-27 behaves in the same way as trunk does,
> in *scratch*, too.

It turned out to be more nuanced.

I can reproduce M-x (backward-char 1) behavior on emacs 24.3.1. So that's not
new. The new part shows only when the 'intangible property is set:

     
     (defun insert-with-fringe ()
       (interactive)
       (insert
        "      "
        (concat
         (propertize "fringe" 'display (list 'left-fringe 'filled-square font-lock-doc-face)
     		'intangible 'test)
         (propertize "dummy\n" 'display "AAA\n" 'font-lock-face font-lock-doc-face
     		'intangible 'test))
        "some more text")
       (previous-line))


Prior to 1st of April 2015 the (backward-char 1) used to work with the above
example. Something changed between 1st of April and 1st of June and as a
consequence the point is now stuck even with the above example. I wasn't able to
build emacs on commits between April 1st and June 1st.


  Vitalie

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Wolfgang Jenkner]

On Sun, Aug 30 2015, Vitalie Spinu wrote:

> Prior to 1st of April 2015 the (backward-char 1) used to work with the above
> example. Something changed between 1st of April and 1st of June and as a
> consequence the point is now stuck even with the above example. I

That's perhaps

  commit 84e0b7dad6f1a8e53261f9b96f5a9080fea681a4
  Author: Stefan Monnier <monnier@iro.umontreal.ca>
  Date:   Mon Apr 13 15:51:15 2015 -0400

  Deprecate `intangible' and `point-entered' properties

The change was documented in a NEWS entry:

** Obsolete text properties `intangible', `point-entered', and `point-left'.
Replaced by properties `cursor-intangible' and `cursor-sensor-functions',
implemented by the new `cursor-intangible-mode' and
`cursor-sensor-mode' minor modes.

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Eli Zaretskii]

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Date: Sun, 30 Aug 2015 14:55:45 +0200
> Cc: 21368@debbugs.gnu.org
> 
> On Sun, Aug 30 2015, Vitalie Spinu wrote:
> 
> > Prior to 1st of April 2015 the (backward-char 1) used to work with the above
> > example. Something changed between 1st of April and 1st of June and as a
> > consequence the point is now stuck even with the above example. I
> 
> That's perhaps
> 
>   commit 84e0b7dad6f1a8e53261f9b96f5a9080fea681a4
>   Author: Stefan Monnier <monnier@iro.umontreal.ca>
>   Date:   Mon Apr 13 15:51:15 2015 -0400
> 
>   Deprecate `intangible' and `point-entered' properties

I don't think so, because I have a build from May 1, and the problem
with backward-char is not there.

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Wolfgang Jenkner]

On Sun, Aug 30 2015, Eli Zaretskii wrote:

[I wrote]
>> That's perhaps
>> 
>>   commit 84e0b7dad6f1a8e53261f9b96f5a9080fea681a4
>>   Author: Stefan Monnier <monnier@iro.umontreal.ca>
>>   Date:   Mon Apr 13 15:51:15 2015 -0400
>> 
>>   Deprecate `intangible' and `point-entered' properties
>
> I don't think so, because I have a build from May 1, and the problem
> with backward-char is not there.

Ah, sorry then.  In any case, using the stuff introduced by this change
works for the OP's example.

bug#21368: 25.0.50; Core navigation commands fail in a multi-line intangible text with fringe

[Wolfgang Jenkner]

On Sun, Aug 30 2015, Wolfgang Jenkner wrote:

> On Sun, Aug 30 2015, Eli Zaretskii wrote:
>
> [I wrote]
>>> That's perhaps
>>> 
>>>   commit 84e0b7dad6f1a8e53261f9b96f5a9080fea681a4
>>>   Author: Stefan Monnier <monnier@iro.umontreal.ca>
>>>   Date:   Mon Apr 13 15:51:15 2015 -0400
>>> 
>>>   Deprecate `intangible' and `point-entered' properties
>>
>> I don't think so, because I have a build from May 1, and the problem
>> with backward-char is not there.
>
> Ah, sorry then.  In any case, using the stuff introduced by this change
> works for the OP's example.

I properly bisected commits for the month of May and it gives

  commit d090be146176e9acee89fdaadc86e2eb26209ef5
  Author: Stefan Monnier <monnier@iro.umontreal.ca>
  Date:   Wed May 27 11:52:28 2015 -0400

  Change inhibit-point-motion-hooks to t

  * src/textprop.c (syms_of_textprop): Default Vinhibit_point_motion_hooks
  to t and document it as obsolete.


Indeed, with the following definition (where both the `intangible' text
property and the use of `inhibit-point-motion-hooks' are deprecated now)
point is placed on the "f" of "fringe" (not on the "d" of "dummy"), and
so M-: (backward-char) works.

(defun insert-with-fringe ()
  (interactive)
  (let (inhibit-point-motion-hooks)
    (insert
     "      "
     (concat
      (propertize "fringe" 'display (list 'left-fringe 'filled-square font-lock-doc-face)
		  'intangible 'test)
      (propertize "dummy\n" 'display "AAA\n" 'font-lock-face font-lock-doc-face
		  'intangible 'test))
     "some more text")
    (previous-line)))


Of course, this is not really the original subject of this bug report,
I think...