Moving point around empty overlays with 'after-text

Moving point around empty overlays with 'after-text

[Ash]

https://github.com/emacs-lsp/lsp-mode/issues/3263 is a bug in lsp-mode (emacs's
own eglot has the same bug as far as I can tell) that appears to boil down to
the behavior of emacs overlays and after-string. That is, if your buffer looks
like

let my_value{: Vec<i32>} = vec![0, 1, 2];

(where the curly braces indicate the after-string property of an
overlay), you need to put your cursor *after* the overlay to
insert text at the end of the variable name, which comes *before*
it, and it's impossible to put your cursor immediately between
the overlay and the preceding text. I assume the behavior the
user desires is that you can put your cursor either immediately
before or immediately after the overlay and insert text, and that
pressing the left/right arrow would move you over the overlay but
leave the actual position of point unchahnged.

My suspicion is that this isn't fixable just by setting the right text/overlay
properties, since both the cursor locations immediately before and after the
overlay actually correspond to the same location in the underlying string. But
I'm not good at text property arcana. Any advice?

Re: Moving point around empty overlays with 'after-text

[Eli Zaretskii]

> Date: Fri, 07 Apr 2023 22:46:19 -0700
> From: Ash <ext0l@catgirl.ai>
> 
> https://github.com/emacs-lsp/lsp-mode/issues/3263 is a bug in lsp-mode (emacs's
> own eglot has the same bug as far as I can tell) that appears to boil down to
> the behavior of emacs overlays and after-string. That is, if your buffer looks
> like
> 
> let my_value{: Vec<i32>} = vec![0, 1, 2];
> 
> (where the curly braces indicate the after-string property of an
> overlay), you need to put your cursor *after* the overlay to
> insert text at the end of the variable name, which comes *before*
> it, and it's impossible to put your cursor immediately between
> the overlay and the preceding text. I assume the behavior the
> user desires is that you can put your cursor either immediately
> before or immediately after the overlay and insert text, and that
> pressing the left/right arrow would move you over the overlay but
> leave the actual position of point unchahnged.
> 
> My suspicion is that this isn't fixable just by setting the right text/overlay
> properties, since both the cursor locations immediately before and after the
> overlay actually correspond to the same location in the underlying string. But
> I'm not good at text property arcana. Any advice?

Did you try to use on the overlay string the 'cursor' text property
whose value is zero?

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

On 2023-04-08 13:46, Ash wrote:
> https://github.com/emacs-lsp/lsp-mode/issues/3263 is a bug in lsp-mode (emacs's
> own eglot has the same bug as far as I can tell) that appears to boil down to
> the behavior of emacs overlays and after-string. That is, if your buffer looks
> like
> 
> let my_value{: Vec<i32>} = vec![0, 1, 2];
> 
> (where the curly braces indicate the after-string property of an
> overlay), you need to put your cursor *after* the overlay to
> insert text at the end of the variable name, which comes *before*
> it, and it's impossible to put your cursor immediately between
> the overlay and the preceding text. I assume the behavior the
> user desires is that you can put your cursor either immediately
> before or immediately after the overlay and insert text, and that
> pressing the left/right arrow would move you over the overlay but
> leave the actual position of point unchahnged.
> 
> My suspicion is that this isn't fixable just by setting the right text/overlay
> properties, since both the cursor locations immediately before and after the
> overlay actually correspond to the same location in the underlying string. But
> I'm not good at text property arcana. Any advice?

Github issue has some suggestion about how it could possibly be done, and the poster
rightfully notes that the solution is nasty (essentially catching the event of cursor
moving 1 char forward, tweaking overlay properties and resetting the cursor back).

Does it actually make any sense to put the cursor right after the overlay?

In my opinion the easier solution would be to always put it before the overlay - this way
when the text is typed it is inserted right before the cursor, not somewhere else.

However there are problems with that as well, because right now there is no correct way to
set cursor position when near the zero-width overlay
(see https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62540).

-- 
Best regards,
Platon Pronko
PGP 2A62D77A7A2CB94E

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

On 2023-04-08 18:06, Eli Zaretskii wrote:
>> Date: Fri, 07 Apr 2023 22:46:19 -0700
>> From: Ash <ext0l@catgirl.ai>
>>
>> https://github.com/emacs-lsp/lsp-mode/issues/3263 is a bug in lsp-mode (emacs's
>> own eglot has the same bug as far as I can tell) that appears to boil down to
>> the behavior of emacs overlays and after-string. That is, if your buffer looks
>> like
>>
>> let my_value{: Vec<i32>} = vec![0, 1, 2];
>>
>> (where the curly braces indicate the after-string property of an
>> overlay), you need to put your cursor *after* the overlay to
>> insert text at the end of the variable name, which comes *before*
>> it, and it's impossible to put your cursor immediately between
>> the overlay and the preceding text. I assume the behavior the
>> user desires is that you can put your cursor either immediately
>> before or immediately after the overlay and insert text, and that
>> pressing the left/right arrow would move you over the overlay but
>> leave the actual position of point unchahnged.
>>
>> My suspicion is that this isn't fixable just by setting the right text/overlay
>> properties, since both the cursor locations immediately before and after the
>> overlay actually correspond to the same location in the underlying string. But
>> I'm not good at text property arcana. Any advice?
> 
> Did you try to use on the overlay string the 'cursor' text property
> whose value is zero?
> 

As I understand that it won't help, because they have to use zero-width overlays here,
and the behavior of 'cursor property in such cases is not to be relied upon (as you previously
suggested in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62540).

-- 
Best regards,
Platon Pronko
PGP 2A62D77A7A2CB94E

Re: Moving point around empty overlays with 'after-text

[Ash]

On Sat, Apr 8, 2023, at 3:10 AM, Platon Pronko wrote:
> On 2023-04-08 13:46, Ash wrote:
> > https://github.com/emacs-lsp/lsp-mode/issues/3263 is a bug in lsp-mode (emacs's
> > own eglot has the same bug as far as I can tell) that appears to boil down to
> > the behavior of emacs overlays and after-string. That is, if your buffer looks
> > like
> > 
> > let my_value{: Vec<i32>} = vec![0, 1, 2];
> > 
> > (where the curly braces indicate the after-string property of an
> > overlay), you need to put your cursor *after* the overlay to
> > insert text at the end of the variable name, which comes *before*
> > it, and it's impossible to put your cursor immediately between
> > the overlay and the preceding text. I assume the behavior the
> > user desires is that you can put your cursor either immediately
> > before or immediately after the overlay and insert text, and that
> > pressing the left/right arrow would move you over the overlay but
> > leave the actual position of point unchahnged.
> > 
> > My suspicion is that this isn't fixable just by setting the right text/overlay
> > properties, since both the cursor locations immediately before and after the
> > overlay actually correspond to the same location in the underlying string. But
> > I'm not good at text property arcana. Any advice?
> 
> Github issue has some suggestion about how it could possibly be done, and the poster
> rightfully notes that the solution is nasty (essentially catching the event of cursor
> moving 1 char forward, tweaking overlay properties and resetting the cursor back).
> 
> Does it actually make any sense to put the cursor right after the overlay?
> 
> In my opinion the easier solution would be to always put it before the overlay - this way
> when the text is typed it is inserted right before the cursor, not somewhere else.
> 
> However there are problems with that as well, because right now there is no correct way to
> set cursor position when near the zero-width overlay
> (see https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62540).
> 
> -- 
> Best regards,
> Platon Pronko
> PGP 2A62D77A7A2CB94E

Yeah, that's my message, hence why I was asking if I was missing something :)

In some cases it makes sense to put the cursor after the overlay; for example, when invoking a function, the overlay can look like

some_function({argument_name: }some_value)

in which case you'd expect to be able to put the cursor after the overlay (to edit the value) and before (to add another argument to the list). Both cursor positions would correspond to the same location, but you're doing something different semantically. So I would expect to be able to type 'another_value, ' before the overlay and 'foo' after and get

some_function(another_value, {argument_name: }foosome_value)

and not

some_function({argument_name: }another_value, foosome_value)

In practice, I think for the existing overlays in rust-analyzer's inlay hints there's a 'preferred' cursor position (start for type annotations, end for param name annotations) where the user will want it 80% of the time, so it could set things up based on that. Might also investigate the nasty solution and see how clean I can get it and see if it's expensive CPU-wise.

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

On 2023-04-09 07:06, Ash wrote:
> Yeah, that's my message, hence why I was asking if I was missing something :)
> 
> In some cases it makes sense to put the cursor after the overlay; for example, when invoking a function, the overlay can look like
> 
> some_function({argument_name: }some_value)
> 
> in which case you'd expect to be able to put the cursor after the overlay (to edit the value) and before (to add another argument to the list). Both cursor positions would correspond to the same location, but you're doing something different semantically. So I would expect to be able to type 'another_value, ' before the overlay and 'foo' after and get
> 
> some_function(another_value, {argument_name: }foosome_value)
> 
> and not
> 
> some_function({argument_name: }another_value, foosome_value)
> 
> In practice, I think for the existing overlays in rust-analyzer's inlay hints there's a 'preferred' cursor position (start for type annotations, end for param name annotations) where the user will want it 80% of the time, so it could set things up based on that. Might also investigate the nasty solution and see how clean I can get it and see if it's expensive CPU-wise.
> 

I looked at what other IDE is doing (IDEA in my case). They indeed allow the cursor to be positioned both before and after the inlay, with added text appearing exactly as you described.

I still think that doing this hack with "detect if the cursor moved exactly one position forward, tweak overlay, move it back" is horrible. It feels like it might be much better if this was handled at much lower level, not at the level of lsp-mode.

-- 
Best regards,
Platon Pronko
PGP 2A62D77A7A2CB94E

Re: Moving point around empty overlays with 'after-text

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1926 bytes --]

On Sun, Apr 09, 2023 at 08:15:31PM +0800, Platon Pronko wrote:
> On 2023-04-09 07:06, Ash wrote:
> > Yeah, that's my message, hence why I was asking if I was missing something :)
> > 
> > In some cases it makes sense to put the cursor after the overlay; for example, when invoking a function, the overlay can look like
> > 
> > some_function({argument_name: }some_value)
> > 
> > in which case you'd expect to be able to put the cursor after the overlay (to edit the value) and before (to add another argument to the list). Both cursor positions would correspond to the same location, but you're doing something different semantically. So I would expect to be able to type 'another_value, ' before the overlay and 'foo' after and get
> > 
> > some_function(another_value, {argument_name: }foosome_value)
> > 
> > and not
> > 
> > some_function({argument_name: }another_value, foosome_value)
> > 
> > In practice, I think for the existing overlays in rust-analyzer's inlay hints there's a 'preferred' cursor position (start for type annotations, end for param name annotations) where the user will want it 80% of the time, so it could set things up based on that. Might also investigate the nasty solution and see how clean I can get it and see if it's expensive CPU-wise.
> > 
> 
> I looked at what other IDE is doing (IDEA in my case). They indeed allow the cursor to be positioned both before and after the inlay, with added text appearing exactly as you described.
> 
> I still think that doing this hack with "detect if the cursor moved exactly one position forward, tweak overlay, move it back" is horrible. It feels like it might be much better if this was handled at much lower level, not at the level of lsp-mode.

What I don't understand is why the overlay doesn't cover the
whole argument value instead of sticking before it. Perhaps
this would ease some of the problems?

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Moving point around empty overlays with 'after-text

[Ash]

Yeah, I think doing this "right" might require adding a new property to overlays/strings (or giving an existing property a new value) to enable this behavior and modifying C code. Not sure how viable that is or if it's something the devs would want.

On Sun, Apr 9, 2023, at 5:15 AM, Platon Pronko wrote:
> On 2023-04-09 07:06, Ash wrote:
> > Yeah, that's my message, hence why I was asking if I was missing something :)
> > 
> > In some cases it makes sense to put the cursor after the overlay; for example, when invoking a function, the overlay can look like
> > 
> > some_function({argument_name: }some_value)
> > 
> > in which case you'd expect to be able to put the cursor after the overlay (to edit the value) and before (to add another argument to the list). Both cursor positions would correspond to the same location, but you're doing something different semantically. So I would expect to be able to type 'another_value, ' before the overlay and 'foo' after and get
> > 
> > some_function(another_value, {argument_name: }foosome_value)
> > 
> > and not
> > 
> > some_function({argument_name: }another_value, foosome_value)
> > 
> > In practice, I think for the existing overlays in rust-analyzer's inlay hints there's a 'preferred' cursor position (start for type annotations, end for param name annotations) where the user will want it 80% of the time, so it could set things up based on that. Might also investigate the nasty solution and see how clean I can get it and see if it's expensive CPU-wise.
> > 
> 
> I looked at what other IDE is doing (IDEA in my case). They indeed allow the cursor to be positioned both before and after the inlay, with added text appearing exactly as you described.
> 
> I still think that doing this hack with "detect if the cursor moved exactly one position forward, tweak overlay, move it back" is horrible. It feels like it might be much better if this was handled at much lower level, not at the level of lsp-mode.
> 
> -- 
> Best regards,
> Platon Pronko
> PGP 2A62D77A7A2CB94E
> 
> 

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

On 2023-04-09 22:49, tomas@tuxteam.de wrote:
> On Sun, Apr 09, 2023 at 08:15:31PM +0800, Platon Pronko wrote:
>> On 2023-04-09 07:06, Ash wrote:
>>> Yeah, that's my message, hence why I was asking if I was missing something :)
>>>
>>> In some cases it makes sense to put the cursor after the overlay; for example, when invoking a function, the overlay can look like
>>>
>>> some_function({argument_name: }some_value)
>>>
>>> in which case you'd expect to be able to put the cursor after the overlay (to edit the value) and before (to add another argument to the list). Both cursor positions would correspond to the same location, but you're doing something different semantically. So I would expect to be able to type 'another_value, ' before the overlay and 'foo' after and get
>>>
>>> some_function(another_value, {argument_name: }foosome_value)
>>>
>>> and not
>>>
>>> some_function({argument_name: }another_value, foosome_value)
>>>
>>> In practice, I think for the existing overlays in rust-analyzer's inlay hints there's a 'preferred' cursor position (start for type annotations, end for param name annotations) where the user will want it 80% of the time, so it could set things up based on that. Might also investigate the nasty solution and see how clean I can get it and see if it's expensive CPU-wise.
>>>
>>
>> I looked at what other IDE is doing (IDEA in my case). They indeed allow the cursor to be positioned both before and after the inlay, with added text appearing exactly as you described.
>>
>> I still think that doing this hack with "detect if the cursor moved exactly one position forward, tweak overlay, move it back" is horrible. It feels like it might be much better if this was handled at much lower level, not at the level of lsp-mode.
> 
> What I don't understand is why the overlay doesn't cover the
> whole argument value instead of sticking before it. Perhaps
> this would ease some of the problems?
> 
> Cheers

Why should it?

(maybe there's a confusion because of the words - what is being used here is technically a zero-width overlay, so it's not "overlaying" anything, it's probably better to call it "inlay")

Anyway, even if you make this overlay to cover something it won't solve the problem - it still won't be possible to navigate the cursor to be before or after the hint.

-- 
Best regards,
Platon Pronko
PGP 2A62D77A7A2CB94E

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

On 2023-04-10 04:44, Ash wrote:
> Yeah, I think doing this "right" might require adding a new property to overlays/strings (or giving an existing property a new value) to enable this behavior and modifying C code. Not sure how viable that is or if it's something the devs would want.

I think it's even worse than adding a property to the overlay. You need common point manipulation functions to account for possibility of inlays, i.e. (point) for position before and after inlay will be returning different values, (forward-char) will correctly advance the point from the left side to the right side of the inlay, etc.

(on second thought, making (point) return different values for positions around overlays sounds horrifying, because this will break about half of all Elisp code written)

But inlay hints seem to be a common functionality for any modern IDE nowdays, so it might make sense to support them natively, without making major-mode developers resort to horrible hacks like described before.

-- 
Best regards,
Platon Pronko
PGP 2A62D77A7A2CB94E

Re: Moving point around empty overlays with 'after-text

[Ash]

On Sun, Apr 9, 2023, at 7:00 PM, Platon Pronko wrote:
> On 2023-04-10 04:44, Ash wrote:
> > Yeah, I think doing this "right" might require adding a new property to overlays/strings (or giving an existing property a new value) to enable this behavior and modifying C code. Not sure how viable that is or if it's something the devs would want.
> 
> I think it's even worse than adding a property to the overlay. You need common point manipulation functions to account for possibility of inlays, i.e. (point) for position before and after inlay will be returning different values, (forward-char) will correctly advance the point from the left side to the right side of the inlay, etc.
> 
> (on second thought, making (point) return different values for positions around overlays sounds horrifying, because this will break about half of all Elisp code written)
> 
> But inlay hints seem to be a common functionality for any modern IDE nowdays, so it might make sense to support them natively, without making major-mode developers resort to horrible hacks like described before.
> 
> -- 
> Best regards,
> Platon Pronko
> PGP 2A62D77A7A2CB94E
> 

You could make it so only these special overlays (I'm going to call them inlay-type overlays or just inlays) have weird behavior with (point), but that'd still make things very complicated and I wouldn't do it. A sketch for what I would do is something like this:

When point is on a position with an inlay, the new variable point-is-after-inlay (name subject to bikeshedding) controls where point renders (before if nil, after if t). When inserting text, the inlay moves forward if point-is-after-inlay is nil, and doesn't move if it's t; this means characters appear where you'd expect.

We add a new (forward-char-respecting-inlay) function that sets point-is-after-inlay to t if point is at the start of an inlay *and* point-is-after-inlay is nil, or increments point if not (and the same for backwards). Possibly add an 'always-respect-inlay' mode that makes forward-char act like this, with the caveat that things might break in strange ways.

Each inlay has a 'bias' of 'before or 'after that indicates what point-is-before-inlay should be set to when navigating to it 'from afar' or other cases where there's no good heuristic for where to put it; in general, this should correspond to where the text is the inlay is semantically annotating.

There's probably all kinds of edge cases I haven't thought about, of course. Conversely, an even more general approach that would support multiple inlays in a row would be to have point-is-after-inlay be an *index* (and rename it 'point-inlay-index' or some such). Not sure what a concrete use case for that would be.

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

On 2023-04-10 11:21, Ash wrote:
> 
> 
> On Sun, Apr 9, 2023, at 7:00 PM, Platon Pronko wrote:
>> On 2023-04-10 04:44, Ash wrote:
>>> Yeah, I think doing this "right" might require adding a new property to overlays/strings (or giving an existing property a new value) to enable this behavior and modifying C code. Not sure how viable that is or if it's something the devs would want.
>>
>> I think it's even worse than adding a property to the overlay. You need common point manipulation functions to account for possibility of inlays, i.e. (point) for position before and after inlay will be returning different values, (forward-char) will correctly advance the point from the left side to the right side of the inlay, etc.
>>
>> (on second thought, making (point) return different values for positions around overlays sounds horrifying, because this will break about half of all Elisp code written)
>>
>> But inlay hints seem to be a common functionality for any modern IDE nowdays, so it might make sense to support them natively, without making major-mode developers resort to horrible hacks like described before.
>>
>> -- 
>> Best regards,
>> Platon Pronko
>> PGP 2A62D77A7A2CB94E
>>
> 
> You could make it so only these special overlays (I'm going to call them inlay-type overlays or just inlays) have weird behavior with (point), but that'd still make things very complicated and I wouldn't do it. A sketch for what I would do is something like this:
> 
> When point is on a position with an inlay, the new variable point-is-after-inlay (name subject to bikeshedding) controls where point renders (before if nil, after if t). When inserting text, the inlay moves forward if point-is-after-inlay is nil, and doesn't move if it's t; this means characters appear where you'd expect.
> 
> We add a new (forward-char-respecting-inlay) function that sets point-is-after-inlay to t if point is at the start of an inlay *and* point-is-after-inlay is nil, or increments point if not (and the same for backwards). Possibly add an 'always-respect-inlay' mode that makes forward-char act like this, with the caveat that things might break in strange ways.
> 
> Each inlay has a 'bias' of 'before or 'after that indicates what point-is-before-inlay should be set to when navigating to it 'from afar' or other cases where there's no good heuristic for where to put it; in general, this should correspond to where the text is the inlay is semantically annotating.
> 
> There's probably all kinds of edge cases I haven't thought about, of course. Conversely, an even more general approach that would support multiple inlays in a row would be to have point-is-after-inlay be an *index* (and rename it 'point-inlay-index' or some such). Not sure what a concrete use case for that would be.

I like this approach. Seems to be mostly backward-compatible.

-- 
Best regards,
Platon Pronko
PGP 2A62D77A7A2CB94E

Re: Moving point around empty overlays with 'after-text

[Eli Zaretskii]

> Date: Mon, 10 Apr 2023 09:52:54 +0800
> From: Platon Pronko <platon7pronko@gmail.com>
> 
> Anyway, even if you make this overlay to cover something it won't solve the problem - it still won't be possible to navigate the cursor to be before or after the hint.

I don't understand from this discussion why would you want to move the
cursor to be before or after the hint.  I also don't understand why,
if you do need to move the cursor, adding or removing the 'cursor'
property to the overlay string would not solve that.

More generally, what is the requested functionality here and/or the
problem you are trying to solve?  Can you explain that clearly in
terms of user-facing features and behaviors?

Re: Moving point around empty overlays with 'after-text

[Eli Zaretskii]

> Date: Mon, 10 Apr 2023 10:00:52 +0800
> From: Platon Pronko <platon7pronko@gmail.com>
> 
> I think it's even worse than adding a property to the overlay. You need common point manipulation functions to account for possibility of inlays, i.e. (point) for position before and after inlay will be returning different values, (forward-char) will correctly advance the point from the left side to the right side of the inlay, etc.

Please keep in mind that point is a _buffer_ position.  So asking for
point to return different values when the buffer position didn't
change is like, to borrow from Albert Einstein, "doing the same thing
over and over and expecting different results".

> (on second thought, making (point) return different values for positions around overlays sounds horrifying, because this will break about half of all Elisp code written)

Exactly.

> But inlay hints seem to be a common functionality for any modern IDE nowdays, so it might make sense to support them natively, without making major-mode developers resort to horrible hacks like described before.

Well, maybe "someone" should reconsider whether using overlays for
that is a good idea, then?  Why not help-echo, for example?

And I don't think you've tried all of the overlay-related features
yet, see my other message.

Re: Moving point around empty overlays with 'after-text

[Eli Zaretskii]

> Date: Sun, 09 Apr 2023 20:21:09 -0700
> From: Ash <ext0l@catgirl.ai>
> 
> When point is on a position with an inlay, the new variable point-is-after-inlay (name subject to bikeshedding) controls where point renders (before if nil, after if t). When inserting text, the inlay moves forward if point-is-after-inlay is nil, and doesn't move if it's t; this means characters appear where you'd expect.

How is this different from the front-advance and rear-advance
arguments you can specify when creating the overlay.

> We add a new (forward-char-respecting-inlay) function that sets point-is-after-inlay to t if point is at the start of an inlay *and* point-is-after-inlay is nil, or increments point if not (and the same for backwards). Possibly add an 'always-respect-inlay' mode that makes forward-char act like this, with the caveat that things might break in strange ways.
> 
> Each inlay has a 'bias' of 'before or 'after that indicates what point-is-before-inlay should be set to when navigating to it 'from afar' or other cases where there's no good heuristic for where to put it; in general, this should correspond to where the text is the inlay is semantically annotating.
> 
> There's probably all kinds of edge cases I haven't thought about, of course. Conversely, an even more general approach that would support multiple inlays in a row would be to have point-is-after-inlay be an *index* (and rename it 'point-inlay-index' or some such). Not sure what a concrete use case for that would be.

I think before we invent some new complicated feature we should
explore the ones already there.

Did you look at how set-minibuffer-message succeeds in setting the
cursor? did you try using the same technique?

Did you try both before-string and after-string overlays?

Did you try non-"empty" overlays, i.e. those whose start and end are
not the same buffer position?  They can have before-string and
after-string properties as well.

And that is even before considering whether overlays are the best
infrastructure for implementing this stuff in Emacs.

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

[-- Attachment #1: Type: text/plain, Size: 1630 bytes --]

On 2023-04-10 12:56, Eli Zaretskii wrote:
>> Date: Mon, 10 Apr 2023 09:52:54 +0800
>> From: Platon Pronko <platon7pronko@gmail.com>
>>
>> Anyway, even if you make this overlay to cover something it won't solve the problem - it still won't be possible to navigate the cursor to be before or after the hint.
> 
> I don't understand from this discussion why would you want to move the
> cursor to be before or after the hint.  I also don't understand why,
> if you do need to move the cursor, adding or removing the 'cursor'
> property to the overlay string would not solve that.
> 
> More generally, what is the requested functionality here and/or the
> problem you are trying to solve?  Can you explain that clearly in
> terms of user-facing features and behaviors?
> 

I'll try to do it with bunch of attached images, since it feels that it will be easier to explain. If it's better to use text examples, please tell me, I'll try to reformat them.

The base example is in inlay-base.png image - you can see the inline hint on the second line ("a1 =").

I can position the cursor right before the inline hint (inlay-cursor-before.png), type "41, ", thus adding a new argument before existing one (inlay-cursor-before-result.png).

But I can also position the cursor right after the inline hint (inlay-cursor-after.png), type "12", which will be prepended before the old argument value (inlay-cursor-after-result.png).

The difference is of course only visual, the characters are still insterted into the buffer in the same spot - but if we don't allow the user to select the side of the inlay it would look confusing half the time.

[-- Attachment #2: inlay-base.png --]
[-- Type: image/png, Size: 4655 bytes --]

[-- Attachment #3: inlay-cursor-before.png --]
[-- Type: image/png, Size: 5133 bytes --]

[-- Attachment #4: inlay-cursor-before-result.png --]
[-- Type: image/png, Size: 5441 bytes --]

[-- Attachment #5: inlay-cursor-after.png --]
[-- Type: image/png, Size: 5102 bytes --]

[-- Attachment #6: inlay-cursor-after-result.png --]
[-- Type: image/png, Size: 5289 bytes --]

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

On 2023-04-10 13:01, Eli Zaretskii wrote:

>> But inlay hints seem to be a common functionality for any modern IDE nowdays, so it might make sense to support them natively, without making major-mode developers resort >> to horrible hacks like described before.
> Well, maybe "someone" should reconsider whether using overlays for
> that is a good idea, then?

I agree, using overlays as they are currently implemented is definitely not optimal. Are there some better alternatives?

> Why not help-echo, for example?

This will show the help only for the current symbol, while the way inline hints are used is to show annotations for many symbols on the screen (for example, you have 20 different variable definitions, and IDE shows you inline type hints for each of them).

Re: Moving point around empty overlays with 'after-text

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1537 bytes --]

On Mon, Apr 10, 2023 at 09:52:54AM +0800, Platon Pronko wrote:
> On 2023-04-09 22:49, tomas@tuxteam.de wrote:

[...]

> > What I don't understand is why the overlay doesn't cover the
> > whole argument value instead of sticking before it. Perhaps
> > this would ease some of the problems?
> > 
> > Cheers
> 
> Why should it?
> 
> (maybe there's a confusion because of the words - what is being used here is technically a zero-width overlay, so it's not "overlaying" anything, it's probably better to call it "inlay")

I did understand that it is a zero-width overlay, so the
description wasn't confusing to me.

> Anyway, even if you make this overlay to cover something it won't solve the problem - it still won't be possible to navigate the cursor to be before or after the hint.

For non-zero width overlays, it is quite easy to have
point to each side of the overlay's "border": it depends
on which side you are coming from. For me, at least, it
is intuitive.

Quite possibly I'm not understanding what you are trying
to achieve. The mental model I got from your description
is that the overlay describes some property of the arg
value in question (the name given to the corresponding
formal argument elsewhere in the function declaration
viz. definition). That's why I'd expect the overlay to
cover /all/ of the arg value (and let it grow and shrink
automatically as the latter is edited by the user).

But, as said, perhaps I'm lost in the woods.If yes, then
sorry for that :)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

On 2023-04-10 13:09, Eli Zaretskii wrote:
> How is this different from the front-advance and rear-advance
> arguments you can specify when creating the overlay.

front-advance and rear-advance control if the typed text should be included in overlay. In case of inline type hints, typed text never needs to be included.

As I understand it, no combination of front-advance and rear-advance will allow overlay to stay in-place if something is typed at start point. Example:

<cursor>{hint} => type A => {hint}A<cursor>

> Did you look at how set-minibuffer-message succeeds in setting the
> cursor? did you try using the same technique?
> 
> Did you try both before-string and after-string overlays?
> 
> Did you try non-"empty" overlays, i.e. those whose start and end are
> not the same buffer position?  They can have before-string and
> after-string properties as well.

I did. All these examples are about consitently positioning cursor before or after overlay at all times, and they of course work. But none of them allow cursor position to move from beginning to the end of overlay without nasty hacks with detecting cursor position and manually changing overlay properties.

> And that is even before considering whether overlays are the best
> infrastructure for implementing this stuff in Emacs.

It definitely feels they are not :) Even the name says "overlay", while they are overlaying nothing in this case. What can be used instead?

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

On 2023-04-10 13:35, tomas@tuxteam.de wrote:
> For non-zero width overlays, it is quite easy to have
> point to each side of the overlay's "border": it depends
> on which side you are coming from. For me, at least, it
> is intuitive.

Does it work that way for you? For me the default non-propertized overlay always shows the cursor at the end, regardless of which direction I arrived from.

Here's the test file I used:

```
ABEF

(defun test-overlay-cursor (completion pt)
   (remove-overlays)
   (save-excursion
     (let* ((p-completion (propertize completion 'face 'error))
            (ov (make-overlay pt pt nil t t)))
       (overlay-put ov 'display "")
       (overlay-put ov 'after-string p-completion)))
   (goto-char 0))
(test-overlay-cursor "CD" 3)
```

I can navigate so I see "AB{CD}<cursor>EF" or "A<cusor>B{CD}EF", but I can't get the cursor to be before the overlay: "AB<cursor>{CD}EF".

Of course one can add properties to the overlay to get the cursor be positioned always in front, but then you can't get to show it at end.

The main point of the discussion is that it would be nice if it would be possible use usual navigation commands to get from "AB<cusror>{CD}EF" to "AB{CD}<cusor>EF".

-- 
Best regards,
Platon Pronko
PGP 2A62D77A7A2CB94E

Re: Moving point around empty overlays with 'after-text

[Eli Zaretskii]

> Date: Mon, 10 Apr 2023 13:37:37 +0800
> From: Platon Pronko <platon7pronko@gmail.com>
> 
> On 2023-04-10 13:09, Eli Zaretskii wrote:
> > How is this different from the front-advance and rear-advance
> > arguments you can specify when creating the overlay.
> 
> front-advance and rear-advance control if the typed text should be included in overlay. In case of inline type hints, typed text never needs to be included.

Is that the only effect of front-advance and rear-advance in this
case?

> > Did you look at how set-minibuffer-message succeeds in setting the
> > cursor? did you try using the same technique?
> > 
> > Did you try both before-string and after-string overlays?
> > 
> > Did you try non-"empty" overlays, i.e. those whose start and end are
> > not the same buffer position?  They can have before-string and
> > after-string properties as well.
> 
> I did. All these examples are about consitently positioning cursor before or after overlay at all times, and they of course work. But none of them allow cursor position to move from beginning to the end of overlay without nasty hacks with detecting cursor position and manually changing overlay properties.

I'm not sure which part(s) are considered "nasty hacks" in your eyes,
given that you are already okay with using "empty" overlays for
displaying the hints.  In any case, an overlay can have a 'keymap'
property, where you could perhaps redefine what C-f, C-b, and perhaps
some other relevant movement commands do.  Did you try that?

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

On 2023-04-10 16:03, Eli Zaretskii wrote:
>> front-advance and rear-advance control if the typed text should be included in overlay. In case of inline type hints, typed text never needs to be included.
> 
> Is that the only effect of front-advance and rear-advance in this
> case?

I get your point, you are talking about moving the overlay forward if characters are typed before the overlay :)
But for zero-width overlays it never moves the overlay - I tested different variations, and right now no combination of these properties allows one to insert characters before the inlay:

nil nil => characters are inserted after the overlay
nil t => characters are not visible (because they are inside overlay)
t nil => characters are inserted after the overlay
t t => characters are inserted after the overlay

>> I did. All these examples are about consitently positioning cursor before or after overlay at all times, and they of course work. But none of them allow cursor position to move from beginning to the end of overlay without nasty hacks with detecting cursor position and manually changing overlay properties.
> 
> I'm not sure which part(s) are considered "nasty hacks" in your eyes,
> given that you are already okay with using "empty" overlays for
> displaying the hints.

I'm definitely not okay with using empty overlays. My whole point is that it would be nice to have some alternative well-defined way to have inline text, and not resort to trying to force overlays into doing what they are not supposed to. Currently lsp-mode, eglot, company-mode, auto-complete (and bunch of lesser-known packages that copied code from them) all try to use overlays for this, and that's probably not the best thing.

We have narrowing and invisible text properties - these provide a way to hide a portion of the buffer.
We have overlays - these provide a way to replace a portion of the buffer with something else.
But we seem to miss the capability to reliably show stuff that isn't actually in the buffer.

> In any case, an overlay can have a 'keymap'
> property, where you could perhaps redefine what C-f, C-b, and perhaps
> some other relevant movement commands do.  Did you try that?

Yes. Keymap is only used when front-advance is nil and rear-advance is t, which coincidentally is the only combination that causes typed characters to be "hidden" by the overlay. (perhaps I am doing something incorrectly, however)

But even if I could override C-f and some other keys, that would still be only a half-solution - people might bind forward-char and backward-char to different keys, or they might be using entirely different methods of moving around the buffer (for example I am using avy-goto-char often). I don't think there is a way to handle all such cases by simply overriding the keymap.

-- 
Best regards,
Platon Pronko
PGP 2A62D77A7A2CB94E

Re: Moving point around empty overlays with 'after-text

[Yuri Khan]

On Mon, 10 Apr 2023 at 12:23, Platon Pronko <platon7pronko@gmail.com> wrote:

> I'll try to do it with bunch of attached images, since it feels that it will be easier to explain. If it's better to use text examples, please tell me, I'll try to reformat them.
>
> The base example is in inlay-base.png image - you can see the inline hint on the second line ("a1 =").
>
> I can position the cursor right before the inline hint (inlay-cursor-before.png), type "41, ", thus adding a new argument before existing one (inlay-cursor-before-result.png).

You’re talking about typing “41, ” as if it were an atomic action. In
some cases (e.g. pasting from clipboard) it could be, but normally
you’ll be inserting characters one by one, and it might be instructive
to look at the buffer contents and expected inlay behavior through the
intermediate stages.

0. Base state:

    def test(a1: Int = 1, a2: Int = 2, a3: Int = 3) {}
    test({a1=}42)

1. You type “4”. Depending on how the editor interprets the insertion
around the inlay:

1a. The editor inserts the digit 4 at the position where the inlay is.
The text is now “test(442)”. Syntactically, you just changed the
argument value, so the inlay should stay before the newly inserted
character.

    def test(a1: Int = 1, a2: Int = 2, a3: Int = 3) {}
    test({a1=}42)

1b. The editor lets you insert the digit 4 before the inlay and
somehow knows it is separate from the “42” that used to be the first
argument. The text is now “test(4█42)” where the block represents some
kind of separator. Likely, this text is invalid syntax, so the inlay
is now technically incorrect.

    def test(a1: Int = 1, a2: Int = 2, a3: Int = 3) {}
    test(4█a1=█42)

2. You type “1”. Same thing as “4” before, only in case 2a the
insertion is no longer adjacent to the inlay so the issue does not
even arise. In case 2b, you now have “test(41█42)” in the buffer and
“test(41█a1=█42)” in the view.

3. You type “,”.

3a. The text is now “test(41,42)” and syntactically it’s a call with
two arguments. The inlay for the first argument stays right after the
opening paren; a new inlay appears after the comma and before the 42.

3b. The text is now syntactically well-formed, so the IDE can update the inlays.

    def test(a1: Int = 1, a2: Int = 2, a3: Int = 3) {}
    test({a1=}41,{a2=}42)

4. You type a space.

For typographical reasons, a well-designed grammar will infer the
space to belong to the separator, not the argument expression. The
inlay will shift after the space.

    def test(a1: Int = 1, a2: Int = 2, a3: Int = 3) {}
    test({a1=}41, {a2=}42)


Basically, after working out the above, my question is: Why is it
important to let the user choose whether to insert newly typed
characters before or after the inlay, if the inlay is going to be
updated and repositioned based on the resulting buffer text?
Alternatively, why is it important to let the user bring the buffer
into an invalid state that may even be unrepresentable in text?

Re: Moving point around empty overlays with 'after-text

[Ash]

On Sun, Apr 9, 2023, at 8:31 PM, Platon Pronko wrote:
> On 2023-04-10 11:21, Ash wrote:
> > 
> > 
> > On Sun, Apr 9, 2023, at 7:00 PM, Platon Pronko wrote:
> >> On 2023-04-10 04:44, Ash wrote:
> >>> Yeah, I think doing this "right" might require adding a new property to overlays/strings (or giving an existing property a new value) to enable this behavior and modifying C code. Not sure how viable that is or if it's something the devs would want.
> >>
> >> I think it's even worse than adding a property to the overlay. You need common point manipulation functions to account for possibility of inlays, i.e. (point) for position before and after inlay will be returning different values, (forward-char) will correctly advance the point from the left side to the right side of the inlay, etc.
> >>
> >> (on second thought, making (point) return different values for positions around overlays sounds horrifying, because this will break about half of all Elisp code written)
> >>
> >> But inlay hints seem to be a common functionality for any modern IDE nowdays, so it might make sense to support them natively, without making major-mode developers resort to horrible hacks like described before.
> >>
> >> -- 
> >> Best regards,
> >> Platon Pronko
> >> PGP 2A62D77A7A2CB94E
> >>
> > 
> > You could make it so only these special overlays (I'm going to call them inlay-type overlays or just inlays) have weird behavior with (point), but that'd still make things very complicated and I wouldn't do it. A sketch for what I would do is something like this:
> > 
> > When point is on a position with an inlay, the new variable point-is-after-inlay (name subject to bikeshedding) controls where point renders (before if nil, after if t). When inserting text, the inlay moves forward if point-is-after-inlay is nil, and doesn't move if it's t; this means characters appear where you'd expect.
> > 
> > We add a new (forward-char-respecting-inlay) function that sets point-is-after-inlay to t if point is at the start of an inlay *and* point-is-after-inlay is nil, or increments point if not (and the same for backwards). Possibly add an 'always-respect-inlay' mode that makes forward-char act like this, with the caveat that things might break in strange ways.
> > 
> > Each inlay has a 'bias' of 'before or 'after that indicates what point-is-before-inlay should be set to when navigating to it 'from afar' or other cases where there's no good heuristic for where to put it; in general, this should correspond to where the text is the inlay is semantically annotating.
> > 
> > There's probably all kinds of edge cases I haven't thought about, of course. Conversely, an even more general approach that would support multiple inlays in a row would be to have point-is-after-inlay be an *index* (and rename it 'point-inlay-index' or some such). Not sure what a concrete use case for that would be.
> 
> I like this approach. Seems to be mostly backward-compatible.
> 
> -- 
> Best regards,
> Platon Pronko
> PGP 2A62D77A7A2CB94E
> 

Thank you :) One problem I realized later is that it's unclear how this interacts with the front-advance and rear-advance properties of a nonempty text overlay; if you know what region of text an inlay 'describes', then you may want to use that region as the start/end of the overlay so it can evaporate if you delete that region. This gets especially hairy if there are *multiple* overlays, or one that ends at the same position another starts... I think for now I'll just use the hack I wrote in the github issue and leave this yak mostly unshorn. Thanks for the feedback :)

Re: Moving point around empty overlays with 'after-text

[Platon Pronko]

On 2023-04-10 17:56, Yuri Khan wrote:
> You’re talking about typing “41, ” as if it were an atomic action. In
> some cases (e.g. pasting from clipboard) it could be, but normally
> you’ll be inserting characters one by one, and it might be instructive
> to look at the buffer contents and expected inlay behavior through the
> intermediate stages.
> 
> 0. Base state:
> 
>      def test(a1: Int = 1, a2: Int = 2, a3: Int = 3) {}
>      test({a1=}42)
> 
> 1. You type “4”. Depending on how the editor interprets the insertion
> around the inlay:
> 
> 1a. The editor inserts the digit 4 at the position where the inlay is.
> The text is now “test(442)”. Syntactically, you just changed the
> argument value, so the inlay should stay before the newly inserted
> character.
> 
>      def test(a1: Int = 1, a2: Int = 2, a3: Int = 3) {}
>      test({a1=}42)
> 
> 1b. The editor lets you insert the digit 4 before the inlay and
> somehow knows it is separate from the “42” that used to be the first
> argument. The text is now “test(4█42)” where the block represents some
> kind of separator. Likely, this text is invalid syntax, so the inlay
> is now technically incorrect.
> 
>      def test(a1: Int = 1, a2: Int = 2, a3: Int = 3) {}
>      test(4█a1=█42)
> 
> 2. You type “1”. Same thing as “4” before, only in case 2a the
> insertion is no longer adjacent to the inlay so the issue does not
> even arise. In case 2b, you now have “test(41█42)” in the buffer and
> “test(41█a1=█42)” in the view.
> 
> 3. You type “,”.
> 
> 3a. The text is now “test(41,42)” and syntactically it’s a call with
> two arguments. The inlay for the first argument stays right after the
> opening paren; a new inlay appears after the comma and before the 42.
> 
> 3b. The text is now syntactically well-formed, so the IDE can update the inlays.
> 
>      def test(a1: Int = 1, a2: Int = 2, a3: Int = 3) {}
>      test({a1=}41,{a2=}42)
> 
> 4. You type a space.
> 
> For typographical reasons, a well-designed grammar will infer the
> space to belong to the separator, not the argument expression. The
> inlay will shift after the space.
> 
>      def test(a1: Int = 1, a2: Int = 2, a3: Int = 3) {}
>      test({a1=}41, {a2=}42)
> 
> 
> Basically, after working out the above, my question is: Why is it
> important to let the user choose whether to insert newly typed
> characters before or after the inlay, if the inlay is going to be
> updated and repositioned based on the resulting buffer text?
> Alternatively, why is it important to let the user bring the buffer
> into an invalid state that may even be unrepresentable in text?

You are discussing the technical part, and there you are correct on all points. Of course, from the buffer standpoint all my examples are equivalent and it makes absolutely no difference on what side of the overlay the cursor is. And after the user types something the well-implemented mode will remove the overlay and everything will be fine.

I'm talking more about the user intention, before he actually types anything, and consequently the confusion he will temporarily experience when he will try to execute this intention. We here in Emacs mailing list understand the internals quite well and thus can understand that the overlay is zero-width and cursor position is actually the same on both sides. Less-enlightened user won't - for him the inlay is something tangible, present in the buffer the same way as the characters are present in the buffer.


And in the discussed example the user can have two different intentions:

Intention 1: prepending something to argument value. In this case he will want to position the cursor after the overlay:

test(a1=<cursor>42)
test(a1=1<cursor>42)

(here and after - I'm intentionally omitting the braces around the {a1=} part, to illustrate my point better - usually the inlays look very close to syntactically correct code, so when the developer is working with the code he is likely to treat that "nonexistent" code very similarly to the real one, as if it was actually present)

It will not make sense for him to want to position the cursor before the overlay. Look how confusing his intuition would be if overlay was setup to show cursor before:

test(<cusor>a1=42)
// User is confused.
// He might want to move cursor to the right,
// but then it jumps into the middle of the next argument:
test(a1=4<cursor>)
// He returns back:
test(<cursor>a1=42)
// User types "1". Here's how his brain predicts the next state of the buffer:
test(1a1=42)

Of course once the inlay is hidden, everything is back to normal. But user is already slightly frustrated.

Intention 2: prepending another argument. In this case he will want to position the cursor before the overlay:

test(<cursor>a1=42)
// User tries to type "a0=1," (let's imagine there was such an argument)
test(a<cursor>a1=42)
// More typing
test(a0=1,<cursor>a1=42)

Here, if the overlay was set up such that the cursor is shown after, the user's intuition would be screwed again:

test(a1=<cursor>42)
// Why my cursor is between the argument name and argument value?
// Move left:
test<cursor>(a1=42)
// Something is broken, let's move back:
test(a1=<cursor>42)
// Let's try typing "a0=1". Here's the intuitively expected state of the buffer, completely garbled:
test(a1=a0=142)


In all these examples nothing horrible happened - after a short struggle the user will be able to fix the issue and continue on his way. But such issues contribute to the overall feeling of the editor, and I would be sad if Emacs was leaving people with the feeling of "ever so slightly broken". We can be better than that :)

(and in "we" I include myself - I'm not just idly discussing, I will be willing to go into the code and create a patch if we can agree on how it should be patched, if it should be)


Oh, and by the way, even if we disregard all of the above - we still don't have any well-defined way of positioning the cursor around the zero-width overlay. The way it works currently is it's either broken and cursor is shown on the wrong side, or the code relies on some side effect of overlay implementation (c.f. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=62540).

Re: Moving point around empty overlays with 'after-text

[Yuri Khan]

On Tue, 11 Apr 2023 at 15:49, Platon Pronko <platon7pronko@gmail.com> wrote:

> I'm talking more about the user intention, before he actually types anything, and consequently the confusion he will temporarily experience when he will try to execute this intention. We here in Emacs mailing list understand the internals quite well and thus can understand that the overlay is zero-width and cursor position is actually the same on both sides. Less-enlightened user won't - for him the inlay is something tangible, present in the buffer the same way as the characters are present in the buffer.

Then the less-enlightened user is screwed, because treating the inlay
text as ordinary code may teach them wrong syntax.

Example 1: C++ does not have named arguments, but inlays will still
show them. And, presumably, not in /* comment markers */.

Example 2: In Python, named arguments are allowed only after all the
positional arguments. If a user sees test(a1=42) and does not realize
“a1=” is an inlay, they may try to prepend an explicitly named a0=3.14
argument and be stumped at a syntax error message for “test(a0=3.14,
42)”.


> And in the discussed example the user can have two different intentions:
>
> […]
>
> Intention 2: prepending another argument. In this case he will want to position the cursor before the overlay:
>
> test(<cursor>a1=42)

Yes, but in order to consistently act on this intention, the editor
must insert the argument separator as soon as a character is inserted
before the inlay, and also be prepared to deal with the user typing it
in manually after the argument value.

> // User tries to type "a0=1," (let's imagine there was such an argument)
> test(a<cursor>a1=42)
> // More typing
> test(a0=1,<cursor>a1=42)


All that said, I think it may be considered an argument in favor of
suppressing the inlay when the point is adjacent to it, the same way
prettify-symbols-unprettify-at-point does for prettify-symbols-mode.