bug#61361: cursor cannot be at the start of overlay that starts with a newline

[Dmitry Gutov <dgutov@yandex.ru>]

On 08/02/2023 17:50, Eli Zaretskii wrote:
>> Date: Wed, 8 Feb 2023 17:16:40 +0200
>> Cc: 61361@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 08/02/2023 17:09, Eli Zaretskii wrote:
>>>> From: Xinyang Chen<chenxy@mit.edu>
>>>> Date: Wed, 8 Feb 2023 09:25:07 -0500
>>>>
>>> But the 'cursor' property just supplies the index in the display
>>> string where your Lisp program wants to place the cursor.  Emacs
>>> cannot place point inside a display string.  So the display engine
>>> needs to find where that index is on display.  And it cannot find that
>>> place because there's no glyph that corresponds to the newline.
>>
>> So, no chance of the display engine detecting that the same display
>> string with the 'cursor' property has a newline at that position?
> 
> "No chance" is too strong: this is software, after all.
> 
> But it's not easy to do that.  It involves several places with tricky
> code, that currently all work in unison, and make the same
> assumptions:
> 
>    . the decision where to place the cursor works on a screen-line
>      basis: as we perform layout of each screen line, we decide whether
>      the cursor should be in that line, and if so, on what glyph of
>      that line
>    . the decision whether a screen line is a candidate for showing the
>      cursor when that screen line ends in a newline from a display or
>      an overlay string, and/or when there's a gap in buffer positions
>      shown on the screen due to buffer text hidden by some feature
>    . the tricky (to say the least) code which finds the glyph where to
>      put the cursor on a given screen line when buffer positions change
>      non-monotonically with screen positions (due to bidirectional
>      text) and/or have gaps due to overlay and display strings (it is
>      here that the 'cursor' property is implemented)
> 
> Historically, the 'cursor' property is a relatively late addition to
> the display engine, it was added in Emacs 22.1.  It complicated the
> display code a little, but then along came bidirectional display and
> complicated that _a_lot_.  So we are now at a place where the original
> design of the Emacs 21 display never meant to be.
> 
> The entire design of overlay string display is problematic for adding
> significant features, because when an overlay string is rendered, we
> lose too much information about the original overlay (e.g., we don't
> even record where in the buffer the overlay was positioned, nor the
> overlay from which the string came).  So any feature that needs to
> support properties of overlay strings must actually go back to buffer
> text and look up the overlays there to find the one we want!
> 
>> That's unfortunate: that means we'll need to create some wonky
>> workaround for displaying a completion preview (in Company) when a
>> completion starts with a newline.
> 
> Yes, I know.  If someone wants to work on lifting this limitation, I
> can offer help, but I don't think I'll have time (nor motivation, to
> say the truth) to work on this myself.

Thank you for the description. I might try that someday, but definitely 
not in the near term. Maybe someone beats me to it. Hopefully.