bug#61361: cursor cannot be at the start of overlay that starts with a newline

bug#61361: cursor cannot be at the start of overlay that starts with a newline

[Xinyang Chen]

[-- Attachment #1: Type: text/plain, Size: 329 bytes --]

steps to reproduce:
;; test line
(if (and (boundp 'overlay) (overlayp overlay)) (delete-overlay overlay))
(setq overlay (make-overlay 1 2))
(overlay-put overlay 'display "\nhello")
;; same result if you do this intead
;; (overlay-put overlay 'display (propertize "\nhello" 'cursor 0))

cursor does not display in the first line.

[-- Attachment #2: Type: text/html, Size: 415 bytes --]

bug#61361: cursor cannot be at the start of overlay that starts with a newline

[Eli Zaretskii]

tags 61361 wontfix
thanks

> From: Xinyang Chen <chenxy@mit.edu>
> Date: Tue, 7 Feb 2023 23:29:31 -0500
> 
> steps to reproduce:
> ;; test line
> (if (and (boundp 'overlay) (overlayp overlay)) (delete-overlay overlay))
> (setq overlay (make-overlay 1 2))
> (overlay-put overlay 'display "\nhello")
> ;; same result if you do this intead
> ;; (overlay-put overlay 'display (propertize "\nhello" 'cursor 0))
> 
> cursor does not display in the first line.

Emacs cannot do what you are asking because the newline leaves no
glyph on display.  So the display engine cannot place the cursor on
the newline as it does on other characters, which do have glyphs.

This is a limitation of the 'cursor' property feature.  Sorry.

bug#61361: cursor cannot be at the start of overlay that starts with a newline

[Dmitry Gutov]

Hi Eli,

On 08/02/2023 15:08, Eli Zaretskii wrote:
> tags 61361 wontfix
> thanks
> 
>> From: Xinyang Chen<chenxy@mit.edu>
>> Date: Tue, 7 Feb 2023 23:29:31 -0500
>>
>> steps to reproduce:
>> ;; test line
>> (if (and (boundp 'overlay) (overlayp overlay)) (delete-overlay overlay))
>> (setq overlay (make-overlay 1 2))
>> (overlay-put overlay 'display "\nhello")
>> ;; same result if you do this intead
>> ;; (overlay-put overlay 'display (propertize "\nhello" 'cursor 0))
>>
>> cursor does not display in the first line.
> Emacs cannot do what you are asking because the newline leaves no
> glyph on display.  So the display engine cannot place the cursor on
> the newline as it does on other characters, which do have glyphs.
> 
> This is a limitation of the 'cursor' property feature.  Sorry.

Could you explain that a little further?

When I have a line with text and move point to the end of it, the cursor 
is rendered, right? Does that use some other method than "glyphs", 
making it unavailable to the handling of 'cursor' in a display spec?

bug#61361: cursor cannot be at the start of overlay that starts with a newline

[Eli Zaretskii]

> Date: Wed, 8 Feb 2023 16:41:28 +0200
> Cc: 61361@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> Hi Eli,
> 
> On 08/02/2023 15:08, Eli Zaretskii wrote:
> > tags 61361 wontfix
> > thanks
> > 
> >> From: Xinyang Chen<chenxy@mit.edu>
> >> Date: Tue, 7 Feb 2023 23:29:31 -0500
> >>
> >> steps to reproduce:
> >> ;; test line
> >> (if (and (boundp 'overlay) (overlayp overlay)) (delete-overlay overlay))
> >> (setq overlay (make-overlay 1 2))
> >> (overlay-put overlay 'display "\nhello")
> >> ;; same result if you do this intead
> >> ;; (overlay-put overlay 'display (propertize "\nhello" 'cursor 0))
> >>
> >> cursor does not display in the first line.
> > Emacs cannot do what you are asking because the newline leaves no
> > glyph on display.  So the display engine cannot place the cursor on
> > the newline as it does on other characters, which do have glyphs.
> > 
> > This is a limitation of the 'cursor' property feature.  Sorry.
> 
> Could you explain that a little further?
> 
> When I have a line with text and move point to the end of it, the cursor 
> is rendered, right? Does that use some other method than "glyphs", 
> making it unavailable to the handling of 'cursor' in a display spec?

Yes, it does use a different method, because in that case the cursor
is displayed at point's position, not at some arbitrary index of a
string.

I just tried to explain the difference, in a previous message.  If
something is still unclear, please ask more specific questions.

bug#61361: cursor cannot be at the start of overlay that starts with a newline

[Eli Zaretskii]

> From: Xinyang Chen <chenxy@mit.edu>
> Date: Wed, 8 Feb 2023 09:25:07 -0500
> 
> I'm not seeing the point here though; what happens if you have a line with a single newline? What allows
> placing a cursor there in that case?

In that case, Emacs _knows_ that point is positioned at the newline,
so it does an extra-special trick of placing the cursor after the last
character of the line.

But the 'cursor' property just supplies the index in the display
string where your Lisp program wants to place the cursor.  Emacs
cannot place point inside a display string.  So the display engine
needs to find where that index is on display.  And it cannot find that
place because there's no glyph that corresponds to the newline.

bug#61361: cursor cannot be at the start of overlay that starts with a newline

[Dmitry Gutov]

On 08/02/2023 17:09, Eli Zaretskii wrote:
>> From: Xinyang Chen<chenxy@mit.edu>
>> Date: Wed, 8 Feb 2023 09:25:07 -0500
>>
>> I'm not seeing the point here though; what happens if you have a line with a single newline? What allows
>> placing a cursor there in that case?
> In that case, Emacs_knows_  that point is positioned at the newline,
> so it does an extra-special trick of placing the cursor after the last
> character of the line.
> 
> But the 'cursor' property just supplies the index in the display
> string where your Lisp program wants to place the cursor.  Emacs
> cannot place point inside a display string.  So the display engine
> needs to find where that index is on display.  And it cannot find that
> place because there's no glyph that corresponds to the newline.

So, no chance of the display engine detecting that the same display 
string with the 'cursor' property has a newline at that position?

That's unfortunate: that means we'll need to create some wonky 
workaround for displaying a completion preview (in Company) when a 
completion starts with a newline.

bug#61361: cursor cannot be at the start of overlay that starts with a newline

[Eli Zaretskii]

> Date: Wed, 8 Feb 2023 17:16:40 +0200
> Cc: 61361@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 08/02/2023 17:09, Eli Zaretskii wrote:
> >> From: Xinyang Chen<chenxy@mit.edu>
> >> Date: Wed, 8 Feb 2023 09:25:07 -0500
> >>
> > But the 'cursor' property just supplies the index in the display
> > string where your Lisp program wants to place the cursor.  Emacs
> > cannot place point inside a display string.  So the display engine
> > needs to find where that index is on display.  And it cannot find that
> > place because there's no glyph that corresponds to the newline.
> 
> So, no chance of the display engine detecting that the same display 
> string with the 'cursor' property has a newline at that position?

"No chance" is too strong: this is software, after all.

But it's not easy to do that.  It involves several places with tricky
code, that currently all work in unison, and make the same
assumptions:

  . the decision where to place the cursor works on a screen-line
    basis: as we perform layout of each screen line, we decide whether
    the cursor should be in that line, and if so, on what glyph of
    that line
  . the decision whether a screen line is a candidate for showing the
    cursor when that screen line ends in a newline from a display or
    an overlay string, and/or when there's a gap in buffer positions
    shown on the screen due to buffer text hidden by some feature
  . the tricky (to say the least) code which finds the glyph where to
    put the cursor on a given screen line when buffer positions change
    non-monotonically with screen positions (due to bidirectional
    text) and/or have gaps due to overlay and display strings (it is
    here that the 'cursor' property is implemented)

Historically, the 'cursor' property is a relatively late addition to
the display engine, it was added in Emacs 22.1.  It complicated the
display code a little, but then along came bidirectional display and
complicated that _a_lot_.  So we are now at a place where the original
design of the Emacs 21 display never meant to be.

The entire design of overlay string display is problematic for adding
significant features, because when an overlay string is rendered, we
lose too much information about the original overlay (e.g., we don't
even record where in the buffer the overlay was positioned, nor the
overlay from which the string came).  So any feature that needs to
support properties of overlay strings must actually go back to buffer
text and look up the overlays there to find the one we want!

> That's unfortunate: that means we'll need to create some wonky 
> workaround for displaying a completion preview (in Company) when a 
> completion starts with a newline.

Yes, I know.  If someone wants to work on lifting this limitation, I
can offer help, but I don't think I'll have time (nor motivation, to
say the truth) to work on this myself.

bug#61361: cursor cannot be at the start of overlay that starts with a newline

[Dmitry Gutov]

On 08/02/2023 17:50, Eli Zaretskii wrote:
>> Date: Wed, 8 Feb 2023 17:16:40 +0200
>> Cc: 61361@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 08/02/2023 17:09, Eli Zaretskii wrote:
>>>> From: Xinyang Chen<chenxy@mit.edu>
>>>> Date: Wed, 8 Feb 2023 09:25:07 -0500
>>>>
>>> But the 'cursor' property just supplies the index in the display
>>> string where your Lisp program wants to place the cursor.  Emacs
>>> cannot place point inside a display string.  So the display engine
>>> needs to find where that index is on display.  And it cannot find that
>>> place because there's no glyph that corresponds to the newline.
>>
>> So, no chance of the display engine detecting that the same display
>> string with the 'cursor' property has a newline at that position?
> 
> "No chance" is too strong: this is software, after all.
> 
> But it's not easy to do that.  It involves several places with tricky
> code, that currently all work in unison, and make the same
> assumptions:
> 
>    . the decision where to place the cursor works on a screen-line
>      basis: as we perform layout of each screen line, we decide whether
>      the cursor should be in that line, and if so, on what glyph of
>      that line
>    . the decision whether a screen line is a candidate for showing the
>      cursor when that screen line ends in a newline from a display or
>      an overlay string, and/or when there's a gap in buffer positions
>      shown on the screen due to buffer text hidden by some feature
>    . the tricky (to say the least) code which finds the glyph where to
>      put the cursor on a given screen line when buffer positions change
>      non-monotonically with screen positions (due to bidirectional
>      text) and/or have gaps due to overlay and display strings (it is
>      here that the 'cursor' property is implemented)
> 
> Historically, the 'cursor' property is a relatively late addition to
> the display engine, it was added in Emacs 22.1.  It complicated the
> display code a little, but then along came bidirectional display and
> complicated that _a_lot_.  So we are now at a place where the original
> design of the Emacs 21 display never meant to be.
> 
> The entire design of overlay string display is problematic for adding
> significant features, because when an overlay string is rendered, we
> lose too much information about the original overlay (e.g., we don't
> even record where in the buffer the overlay was positioned, nor the
> overlay from which the string came).  So any feature that needs to
> support properties of overlay strings must actually go back to buffer
> text and look up the overlays there to find the one we want!
> 
>> That's unfortunate: that means we'll need to create some wonky
>> workaround for displaying a completion preview (in Company) when a
>> completion starts with a newline.
> 
> Yes, I know.  If someone wants to work on lifting this limitation, I
> can offer help, but I don't think I'll have time (nor motivation, to
> say the truth) to work on this myself.

Thank you for the description. I might try that someday, but definitely 
not in the near term. Maybe someone beats me to it. Hopefully.

bug#61361: cursor cannot be at the start of overlay that starts with a newline

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> tags 61361 wontfix
> thanks
>
>> From: Xinyang Chen <chenxy@mit.edu>
>> Date: Tue, 7 Feb 2023 23:29:31 -0500
>>
>> steps to reproduce:
>> ;; test line
>> (if (and (boundp 'overlay) (overlayp overlay)) (delete-overlay overlay))
>> (setq overlay (make-overlay 1 2))
>> (overlay-put overlay 'display "\nhello")
>> ;; same result if you do this intead
>> ;; (overlay-put overlay 'display (propertize "\nhello" 'cursor 0))
>>
>> cursor does not display in the first line.
>
> Emacs cannot do what you are asking because the newline leaves no
> glyph on display.  So the display engine cannot place the cursor on
> the newline as it does on other characters, which do have glyphs.
>
> This is a limitation of the 'cursor' property feature.  Sorry.

Since this is a wontfix, I'm closing this bug report.