bug#8627: 24.0.50: cursor property behaves irregularly in before-strings

bug#8627: 24.0.50: cursor property behaves irregularly in before-strings

[Alp Aker]

The documentation for the cursor property (info node "(elisp) Special 
Properties") states that:

"Normally, the cursor is displayed at the end of any overlay and text 
property strings present at the current buffer position.  You can place 
the cursor on any desired character of these strings by giving that 
character a non-`nil' `cursor' text property."

And then follows the description of using integer values for the cursor 
property to make it applicable to a range of buffer positions other than 
the range between the overlay's start and end.

However, when used with overlay before-strings (or after-strings), the 
cursor property appears to behave in ways that aren't consonant with the 
docs and that don't follow a consistent pattern.

Here's a recipe for producing the variety of observed behaviors.  (I use 
before-strings, but using after-strings gives similar results.)

(1) Insert some boilerplate text to interact with.

   (save-excursion
     (goto-char (point-min))
     (insert "Lorem ipsum dolor sit amet.\n"))

Make an overlay and give it a before-string with a non-nil cursor property 
on one character:

   (setq olay (make-overlay 5 6)
         str1 (concat "XX"
                      (propertize "Y" 'cursor t)
                      "ZZ"))
   (overlay-put olay 'before-string str1)

If one now does:

   (goto-char (overlay-start olay))

the cursor property is ignored; the cursor appears after the 
before-string.  However, if the before-string appears before a newline, 
the cursor property is respected:

   (save-excursion
     (goto-char (point-min))
     (search-forward "\n")
     (move-overlay olay (1- (point)) (point)))
   (goto-char (overlay-start olay))

The cursor now appears at the propertized character in the before-string.

(2)  Now give the overlay a before-string one of whose characters has a 
numeric cursor property:

   (setq str2 (concat "XX"
                      (propertize "Y" 'cursor 1)
                      "ZZ"))
   (overlay-put olay 'before-string str2)

In this case, the property is respected regardless of location.  With the 
overlay still at the newline:

   (goto-char (overlay-start olay))

and in the middle of a row:

   (move-overlay olay 5 6)
   (goto-char (overlay-start olay))

the cursor appears at the propertized character in the before-string.

(3) But if the before-string contains a newline, the cursor property 
appears to be ignored regardless of location and regardless of whether the 
value of the cursor property is numeric or merely non-nil.  With the 
overlay still in the middle of the line:

   (setq str3 (concat str1 "\nWWW")
         str4 (concat str2 "\nWWW"))
   (overlay-put olay 'before-string str3)
   (goto-char (overlay-start olay))
   (overlay-put olay 'before-string str4)
   (goto-char (overlay-start olay))

the cursor appears after the before string, for both types of property 
value.  Now moving the overlay back to the newline:

   (save-excursion
     (goto-char (point-min))
     (search-forward "\n")
     (move-overlay olay (1- (point)) (point)))
   (goto-char (overlay-start olay))
   (overlay-put olay 'before-string str3)
   (goto-char (overlay-start olay))

the cursor property is again ignored, in both cases.

I started to read through xdisp.c to make some sense of this, but I'm too 
new to Emacs's internals to be able to follow much of the display routine. 
I did observe, though, that cursor_row_p only checks display properties, 
and doesn't appear to check before-strings and after-strings.

In any case, it would appear that either the treatment of before-strings 
and after-strings should be changed so that the cursor property works with 
them as it does with strings that come from display properties, or the 
documentation should be amended to make clear that that it can't be 
expected to do so.  (In the latter case, it would still seem desirable 
that cursor properties in before- and after-strings behave systematically, 
which isn't currently the case.)

bug#8627: 24.0.50: cursor property behaves irregularly in before-strings

[Eli Zaretskii]

> Date: Thu, 05 May 2011 20:01:00 -0400 (EDT)
> From: Alp Aker <aker@pitt.edu>
> 
> The documentation for the cursor property (info node "(elisp) Special 
> Properties") states that:
> 
> "Normally, the cursor is displayed at the end of any overlay and text 
> property strings present at the current buffer position.  You can place 
> the cursor on any desired character of these strings by giving that 
> character a non-`nil' `cursor' text property."
> 
> And then follows the description of using integer values for the cursor 
> property to make it applicable to a range of buffer positions other than 
> the range between the overlay's start and end.

Note that, crucially, the before-string and after-string properties
are not described in this section.  They are described elsewhere,
where the `cursor' property is not mentioned.  So I wonder if the
`cursor' property was really supposed to work with before-strings and
after-strings...

> However, when used with overlay before-strings (or after-strings), the 
> cursor property appears to behave in ways that aren't consonant with the 
> docs and that don't follow a consistent pattern.

It looks like it was never supported consistently, ever since the
`cursor' property was introduced (in Emacs 22.1).

> (1) Insert some boilerplate text to interact with.
> 
>    (save-excursion
>      (goto-char (point-min))
>      (insert "Lorem ipsum dolor sit amet.\n"))
> 
> Make an overlay and give it a before-string with a non-nil cursor property 
> on one character:
> 
>    (setq olay (make-overlay 5 6)
>          str1 (concat "XX"
>                       (propertize "Y" 'cursor t)
>                       "ZZ"))
>    (overlay-put olay 'before-string str1)
> 
> If one now does:
> 
>    (goto-char (overlay-start olay))
> 
> the cursor property is ignored; the cursor appears after the 
> before-string.

This works as expected in Emacs 23, so it must be some problem with
the bidi-aware redisplay in Emacs 24.  I will take a look when I have
time; thanks for a clear test case.

> (3) But if the before-string contains a newline, the cursor property 
> appears to be ignored regardless of location and regardless of whether the 
> value of the cursor property is numeric or merely non-nil.

This never worked, I tested it in Emacs 22.1, and it behaves like this
as well.

I will take a look at fixing this, bat can you show a real-life use
case where this is needed?

> I started to read through xdisp.c to make some sense of this, but I'm too 
> new to Emacs's internals to be able to follow much of the display routine. 
> I did observe, though, that cursor_row_p only checks display properties, 
> and doesn't appear to check before-strings and after-strings.

It may come as a surprise, but the implementation of display
properties is very different from that of before-string and
after-string properties.  It's a small wonder they don't behave
similarly.

> In any case, it would appear that either the treatment of before-strings 
> and after-strings should be changed so that the cursor property works with 
> them as it does with strings that come from display properties, or the 
> documentation should be amended to make clear that that it can't be 
> expected to do so.  (In the latter case, it would still seem desirable 
> that cursor properties in before- and after-strings behave systematically, 
> which isn't currently the case.)

It sounds like it works in most use cases, so fixing the
implementation is probably a better alternative.

bug#8627: 24.0.50: cursor property behaves irregularly in before-strings

[Stefan Monnier]

> Note that, crucially, the before-string and after-string properties
> are not described in this section.  They are described elsewhere,
> where the `cursor' property is not mentioned.  So I wonder if the
> `cursor' property was really supposed to work with before-strings and
> after-strings...

The only use of `cursor' of which I remember is in minibuffer-message,
on an after-string.  It's used when we display " [Sole completion]" and
such messages.
So, yes, it's supposed to work on before/after-strings ;-)


        Stefan

bug#8627: 24.0.50: cursor property behaves irregularly in before-strings

[Alp Aker]

>> But if the before-string contains a newline, the cursor property
>> appears to be ignored.
>
> I will take a look at fixing this, but can you show a real-life use
> case where this is needed?

I ran into the need while working on this package:

   emacswiki.org/emacs/FillColumnIndicator

It shades the area past the fill column by putting overlays on newlines 
(if you look at the screenshot it'll be obvious how it works).  In a few 
edge cases it would be useful to be able to display a newline that's not 
part of the buffer content.  For example, if the last line of the buffer 
doesn't end in a newline, then no fill-column shading appears on that line 
(since there's no newline to propertize).  I'd like to put an overlay at 
point-max with an after-string containing a propertized newline character, 
so that the fill-column shading can appear on the last line of the buffer. 
But I can't do so, because the newline in the after-string causes any 
cursor property in it to be ignored, and so cursor motion at the end of 
the buffer becomes deranged.  (Display strings won't do here, because I 
don't want to replace the display of any characters in the buffer.)

Does that qualify as a use case?

I might as well also mention, at least for the record, another (minor) 
issue with cursor properties in before-strings that I came across:  If the 
position with the cursor property is out of sight because of horizontal 
scrolling, then the property is ignored, even when auto-hscroll-mode is 
enabled.  Perhaps it would make more sense to respect the cursor property 
when auto-hscroll-mode is non-nil, and adjust hscrolling to bring the 
requested cursor position into view?

Here's a test case:

(progn
   (setq auto-hscroll-mode t)
   ;; A line of text, moving to the end of which will trigger hscrolling.
   (insert (make-string (+ hscroll-margin (window-width) 5) ?X))
   (setq m (point-marker))
   (insert "\n\n")
   (setq o (make-overlay (1- (point)) (point)))
   ;; Request that cursor be at bol, but make the before-string long
   ;; enough that if the cursor property is ignored we won't return from
   ;; hscrolling.
   (overlay-put o
                'before-string
                (concat
                 (propertize " " 'cursor 1)
                 (make-string (window-width) 32)))
   (goto-char m)
   (redisplay)
   (forward-line 1))

After evaluation the cursor will be at the end of the before-string, 
rather than at the requested position.  (This happens with 22 and 23 as 
well.)

bug#8627: 24.0.50: cursor property behaves irregularly in before-strings

[Eli Zaretskii]

> Date: Fri, 06 May 2011 17:17:16 -0400 (EDT)
> From: Alp Aker <aker@pitt.edu>
> Cc: Eli Zaretskii <eliz@gnu.org>, 8627@debbugs.gnu.org
> 
> Does that qualify as a use case?

Yes, of course.

Thanks for the other info.  I will take a look.

bug#8627: 24.0.50: cursor property behaves irregularly in before-strings

[Eli Zaretskii]

> Date: Thu, 05 May 2011 20:01:00 -0400 (EDT)
> From: Alp Aker <aker@pitt.edu>
> 
> However, when used with overlay before-strings (or after-strings), the 
> cursor property appears to behave in ways that aren't consonant with the 
> docs and that don't follow a consistent pattern.

I took a quick look at this bug.  I must say I wish this feature were
never introduced into Emacs: it is hackyish, its implementation is
kludgey, and supporting it in the bidirectional display engine will be
a PITA.  It also makes no sense at all in some use cases.  E.g., if
you put a `cursor' property with an integer value of 1 on either a
before-string or an after-string, the cursor will never be positioned
on the character at point, although it is clearly visible.  And if the
value is greater than 1, the cursor will appear to be "stuck" for that
many C-f keypresses.  I'm sure some users will think it's a bug.

Anyway, I will eventually fix the code to do what it did in Emacs 23,
with one exception:

> (3) But if the before-string contains a newline, the cursor property 
> appears to be ignored regardless of location and regardless of whether the 
> value of the cursor property is numeric or merely non-nil.  With the 
> overlay still in the middle of the line:
> 
>    (setq str3 (concat str1 "\nWWW")
>          str4 (concat str2 "\nWWW"))
>    (overlay-put olay 'before-string str3)
>    (goto-char (overlay-start olay))
>    (overlay-put olay 'before-string str4)
>    (goto-char (overlay-start olay))
> 
> the cursor appears after the before string, for both types of property 
> value.

This cannot possibly work, not without rewriting the Emacs display
engine in ways I don't intend to.  Quite simply, you cannot put the
`cursor' property on a newline that belongs to a string, because a
newline, obviously, doesn't have a graphic representation (a glyph) on
the screen, it just causes Emacs to continue drawing on the next
screen line.

The function that decides where to position the cursor scans the
glyphs on the line that contains point, looking for a glyph that came
from some string position that has a non-nil `cursor' property.  If it
finds such a glyph, it then does what you expect.  But it will never
find a glyph that corresponds to a newline, and so the `cursor'
property on a newline that comes from a before- or after-string will
never be noticed.  It is effectively lost in the data structure
(called a `glyph matrix') which Emacs uses to draw and redraw the
screen.

In fact, with a newline from a string, Emacs doesn't even consider the
screen line where you wanted it to put the cursor be a candidate for
drawing the cursor, because the newline moves the buffer position
"covered" by the overlay to the next screen line.

I will eventually update the documentation with this caveat.

It's too bad that this is exactly the feature that you needed, but you
will have to rethink how to get the same effect.  I suggest putting
the overlay on some different character in the string, and giving the
`cursor' property a value > 1, maybe that will do what you want
(although I admit that I don't quite understand your exact situation).

> In any case, it would appear that either the treatment of before-strings 
> and after-strings should be changed so that the cursor property works with 
> them as it does with strings that come from display properties

That's not possible, because there's a fundamental difference between
the `display' properties whose values are strings and before- and
after-string properties: the former are _replacing_ properties,
i.e. these strings are displayed _instead_ of the buffer text covered
by the property/overlay.  By contrast, before- and after-strings do
not replace the text that is covered by the overlay.  The code that
treats these two use cases is thus different in pretty much
fundamental ways.

bug#8627: 24.0.50: cursor property behaves irregularly in before-strings

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> This cannot possibly work, not without rewriting the Emacs display
> engine in ways I don't intend to.  Quite simply, you cannot put the
> `cursor' property on a newline that belongs to a string, because a
> newline, obviously, doesn't have a graphic representation (a glyph) on
> the screen, it just causes Emacs to continue drawing on the next
> screen line.

[...]

> I will eventually update the documentation with this caveat.

This was ten years ago -- I didn't check whether the documentation has
been updated, but there doesn't seem to be anything more to do here than
that?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#8627: 24.0.50: cursor property behaves irregularly in before-strings

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Alp Aker <aker@pitt.edu>,  8627@debbugs.gnu.org
> Date: Sat, 04 Dec 2021 21:22:44 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > This cannot possibly work, not without rewriting the Emacs display
> > engine in ways I don't intend to.  Quite simply, you cannot put the
> > `cursor' property on a newline that belongs to a string, because a
> > newline, obviously, doesn't have a graphic representation (a glyph) on
> > the screen, it just causes Emacs to continue drawing on the next
> > screen line.
> 
> [...]
> 
> > I will eventually update the documentation with this caveat.
> 
> This was ten years ago -- I didn't check whether the documentation has
> been updated, but there doesn't seem to be anything more to do here than
> that?

I promised to fix at least some of the behavior, but never did.  I
will take a look soon.

bug#8627: 24.0.50: cursor property behaves irregularly in before-strings

[Eli Zaretskii]

> Date: Sat, 04 Dec 2021 22:44:50 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 8627@debbugs.gnu.org, aker@pitt.edu
> 
> > From: Lars Ingebrigtsen <larsi@gnus.org>
> > Cc: Alp Aker <aker@pitt.edu>,  8627@debbugs.gnu.org
> > Date: Sat, 04 Dec 2021 21:22:44 +0100
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > > This cannot possibly work, not without rewriting the Emacs display
> > > engine in ways I don't intend to.  Quite simply, you cannot put the
> > > `cursor' property on a newline that belongs to a string, because a
> > > newline, obviously, doesn't have a graphic representation (a glyph) on
> > > the screen, it just causes Emacs to continue drawing on the next
> > > screen line.
> > 
> > [...]
> > 
> > > I will eventually update the documentation with this caveat.
> > 
> > This was ten years ago -- I didn't check whether the documentation has
> > been updated, but there doesn't seem to be anything more to do here than
> > that?
> 
> I promised to fix at least some of the behavior, but never did.  I
> will take a look soon.

I took a look.  The code works correctly: since the before-string
leaves the buffer position visible, Emacs by default places the cursor
there, and a non-nil value of the 'cursor' property cannot override
this basic behavior.  If Emacs 23 behaved differently, it was a bug in
Emacs 23.  By contrast, when the value of 'cursor' is an integer, then
the cursor is placed on that character even if the corresponding
buffer position is visible.  Which is what Emacs now does.

So I've now documented these caveats, including the one wrt newline
characters, and I'm finally marking this bug done.