bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Artur Malabarba]

1. Type `single line' into an empty buffer, and place the cursor at the space.
2. Hit M-: and evaluate the following snippet
    (add-text-properties (point) (1+ (point)) '(display "\n"))
3. When the space in the middle gets "turned" into a newline, the
cursor is left where it was (at the fake end of the first line, right
after the word `single').
4. Now follow the same steps on a new buffer, but evalate this instead:
(add-text-properties (point) (1+ (point)) '(display "\n------"))
5. This time, the cursor will be moved down to the start of the second
"fake" line (right before the `-----'). Furthermore, it is impossible
to place it at the position we had in item 3, it just jumps straight
from the first `e' to the first `-', skipping the actual newline char.

If I can place the cursor on the newline in the first scenario, why
can't I do so in the second?
Is there a way around this?
I tried adding a '(cursor t) property to the "\n" substring, but
nothing changed.


Cheers,
Artur

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Eli Zaretskii]

> Date: Fri, 7 Aug 2015 16:07:27 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> 
> 1. Type `single line' into an empty buffer, and place the cursor at the space.
> 2. Hit M-: and evaluate the following snippet
>     (add-text-properties (point) (1+ (point)) '(display "\n"))
> 3. When the space in the middle gets "turned" into a newline, the
> cursor is left where it was (at the fake end of the first line, right
> after the word `single').
> 4. Now follow the same steps on a new buffer, but evalate this instead:
> (add-text-properties (point) (1+ (point)) '(display "\n------"))
> 5. This time, the cursor will be moved down to the start of the second
> "fake" line (right before the `-----'). Furthermore, it is impossible
> to place it at the position we had in item 3, it just jumps straight
> from the first `e' to the first `-', skipping the actual newline char.
> 
> If I can place the cursor on the newline in the first scenario, why
> can't I do so in the second?

Because we have special support for cursor positioning on display
strings that end in a newline (AFAIR, since some packages that change
how a newline is displayed needed that).

> Is there a way around this?

Why do you need it?  What is your real-life use case?

> I tried adding a '(cursor t) property to the "\n" substring, but
> nothing changed.

You cannot put a 'cursor' property on a newline, because a newline
leaves no glyphs on display.  To be effective, the 'cursor' property
needs to be placed on a character that produces some glyph.

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Eli Zaretskii]

> Date: Fri, 07 Aug 2015 18:29:10 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 21207@debbugs.gnu.org
> 
> > If I can place the cursor on the newline in the first scenario, why
> > can't I do so in the second?
> 
> Because we have special support for cursor positioning on display
> strings that end in a newline

More importantly: in the first scenario the newline ends the display
string, whereas in the second it is in its middle.  We never put the
cursor in the middle of a display string, always after it (unless
there's a 'cursor' property on some of its glyphs).

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Artur Malabarba]

2015-08-07 19:23 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
>> Date: Fri, 07 Aug 2015 18:29:10 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 21207@debbugs.gnu.org
>>
>> > If I can place the cursor on the newline in the first scenario, why
>> > can't I do so in the second?
>>
>> Because we have special support for cursor positioning on display
>> strings that end in a newline
>
> More importantly: in the first scenario the newline ends the display
> string, whereas in the second it is in its middle.

Isn't it at the start of the display string?

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Eli Zaretskii]

> Date: Fri, 7 Aug 2015 20:43:41 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: 21207@debbugs.gnu.org
> 
> 2015-08-07 19:23 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
> >> Date: Fri, 07 Aug 2015 18:29:10 +0300
> >> From: Eli Zaretskii <eliz@gnu.org>
> >> Cc: 21207@debbugs.gnu.org
> >>
> >> > If I can place the cursor on the newline in the first scenario, why
> >> > can't I do so in the second?
> >>
> >> Because we have special support for cursor positioning on display
> >> strings that end in a newline
> >
> > More importantly: in the first scenario the newline ends the display
> > string, whereas in the second it is in its middle.
> 
> Isn't it at the start of the display string?

Yes, sorry.  The important part is that it's not at the end of the
string.

In any case, due to the way cursor positioning in these cases is
implemented, in order to decide that the cursor could be placed on a
line that is part of the display string, the display engine needs to
see at least one glyph from that display string on that line.  And a
newline leaves no glyph.  So if you change your second scenario to
this:

  (add-text-properties (point) (1+ (point)) '(display "-\n------"))

you will see that the cursor is placed on that line, before the
newline.

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Artur Malabarba]

> Yes, sorry.  The important part is that it's not at the end of the
> string.
>
> In any case, due to the way cursor positioning in these cases is
> implemented, in order to decide that the cursor could be placed on a
> line that is part of the display string, the display engine needs to
> see at least one glyph from that display string on that line.  And a
> newline leaves no glyph.  So if you change your second scenario to
> this:
>
>   (add-text-properties (point) (1+ (point)) '(display "-\n------"))
>
> you will see that the cursor is placed on that line, before the
> newline.

Yes, I see what you mean.

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1173 bytes --]

> > If I can place the cursor on the newline in the first scenario, why
> > can't I do so in the second?
>
> Because we have special support for cursor positioning on display
> strings that end in a newline (AFAIR, since some packages that change
> how a newline is displayed needed that).
>
> > I tried adding a '(cursor t) property to the "\n" substring, but
> > nothing changed.
>
> You cannot put a 'cursor' property on a newline, because a newline
> leaves no glyphs on display.  To be effective, the 'cursor' property
> needs to be placed on a character that produces some glyph.

That makes sense. Thanks for the explanations, Eli.

> > Is there a way around this?
>
> Why do you need it?  What is your real-life use case?

I'm using this to visually wrap some lines, but not others (as part of
rewriting longlines-mode to work without editing the buffer).
I can wrap only the lines I want by doing the procedure in item 2
(displaying a space as a newline). When the line I'm wrapping happens to be
indented, I have to use the procedure in item 4 (so the second line gets
indented too), and it's mildly annoying that I can't place the cursor at
the end of the line.

[-- Attachment #2: Type: text/html, Size: 1439 bytes --]

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Eli Zaretskii]

> Date: Fri, 7 Aug 2015 19:31:29 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: 21207@debbugs.gnu.org
> 
> I'm using this to visually wrap some lines, but not others (as part of
> rewriting longlines-mode to work without editing the buffer). 
> I can wrap only the lines I want by doing the procedure in item 2 (displaying a
> space as a newline). When the line I'm wrapping happens to be indented, I have
> to use the procedure in item 4 (so the second line gets indented too), and it's
> mildly annoying that I can't place the cursor at the end of the line.

Can you use line-prefix and/or wrap-prefix for that?

It should be possible to support the 'cursor' property on a newline
inside a display string, but not inside an overlay string.  Not sure
if this disparity will not cause confusion and resistance, as we
generally try to support the same features in both cases.

In general, I hoped that line wrap at some constant column other than
the window edge (which is what I understand you are trying to do) will
be implemented on the display engine level, where IMO it belongs, not
by using display properties and such likes.

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Artur Malabarba]

2015-08-07 20:04 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
>> Date: Fri, 7 Aug 2015 19:31:29 +0100
>> From: Artur Malabarba <bruce.connor.am@gmail.com>
>> Cc: 21207@debbugs.gnu.org
>>
>> I'm using this to visually wrap some lines, but not others (as part of
>> rewriting longlines-mode to work without editing the buffer).
>> I can wrap only the lines I want by doing the procedure in item 2 (displaying a
>> space as a newline). When the line I'm wrapping happens to be indented, I have
>> to use the procedure in item 4 (so the second line gets indented too), and it's
>> mildly annoying that I can't place the cursor at the end of the line.
>
> Can you use line-prefix and/or wrap-prefix for that?

Thanks. That works if I place the line-prefix at the char after the
space, so it looks like it's enough.

> It should be possible to support the 'cursor' property on a newline
> inside a display string, but not inside an overlay string.  Not sure
> if this disparity will not cause confusion and resistance, as we
> generally try to support the same features in both cases.
>
> In general, I hoped that line wrap at some constant column other than
> the window edge (which is what I understand you are trying to do) will
> be implemented on the display engine level, where IMO it belongs, not
> by using display properties and such likes.

That makes sense. Would it still be possible to make that depend on a
predicate defined in a lisp variable? Or would the display engine run
too deep for that?

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Eli Zaretskii]

> Date: Fri, 7 Aug 2015 20:56:38 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: 21207@debbugs.gnu.org
> 
> > In general, I hoped that line wrap at some constant column other than
> > the window edge (which is what I understand you are trying to do) will
> > be implemented on the display engine level, where IMO it belongs, not
> > by using display properties and such likes.
> 
> That makes sense. Would it still be possible to make that depend on a
> predicate defined in a lisp variable? Or would the display engine run
> too deep for that?

Not sure I understand what kind of predicate you have in mind.  Please
elaborate.

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 1001 bytes --]

On Aug 7, 2015 9:13 PM, "Eli Zaretskii" <eliz@gnu.org> wrote:
>
> > Date: Fri, 7 Aug 2015 20:56:38 +0100
> > From: Artur Malabarba <bruce.connor.am@gmail.com>
> > Cc: 21207@debbugs.gnu.org
> >
> > > In general, I hoped that line wrap at some constant column other than
> > > the window edge (which is what I understand you are trying to do) will
> > > be implemented on the display engine level, where IMO it belongs, not
> > > by using display properties and such likes.
> >
> > That makes sense. Would it still be possible to make that depend on a
> > predicate defined in a lisp variable? Or would the display engine run
> > too deep for that?
>
> Not sure I understand what kind of predicate you have in mind.  Please
> elaborate.

Something like *fill-nobreak-predicate*. You provide a nullary predicate
function, and wrapping is not done on lines where that predicate evaluates
to t.

Longlines-mode does that and it's the reason I use it. It lets me wrap
latex text without wrapping equations.

[-- Attachment #2: Type: text/html, Size: 1399 bytes --]

bug#21207: 25.0.50; Cursor position behaves unpredictably on the 'display text property

[Eli Zaretskii]

> Date: Sat, 8 Aug 2015 04:39:49 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: 21207@debbugs.gnu.org
> 
> > > > In general, I hoped that line wrap at some constant column other than
> > > > the window edge (which is what I understand you are trying to do) will
> > > > be implemented on the display engine level, where IMO it belongs, not
> > > > by using display properties and such likes.
> > >
> > > That makes sense. Would it still be possible to make that depend on a
> > > predicate defined in a lisp variable? Or would the display engine run
> > > too deep for that?
> >
> > Not sure I understand what kind of predicate you have in mind. Please
> > elaborate.
> 
> Something like fill-nobreak-predicate. You provide a nullary predicate
> function, and wrapping is not done on lines where that predicate evaluates to
> t. 

I see no problem to support that, once the code which wraps is
written.  The Emacs display engine already supports Lisp evaluation,
e.g., :eval in mode-line format.