Cursor positioning with `after-string' overlays

Cursor positioning with `after-string' overlays

[Eli Zaretskii]

Evaluate the following form in some buffer:

    (let ((pos (goto-char (point-max))))
      (insert "foobar")
      (overlay-put 
	(make-overlay (+ pos 2) (+ pos 3))
	'after-string (propertize "-" 'cursor t)))

This displays "foo-bar", with the "-" coming from the `after-string'
property of the overlay.  Now go to `f' at the beginning of "foo-bar",
type C-f repeatedly, and watch where the cursor is placed when you
type C-f on the second `o' in "foo".  Emacs-23 branch (and Emacs 22)
places the cursor on `-', and jumps over `b' to `a' on the next C-f.
The trunk version, by contrast, jumps from `o' directly to `b', and
then continues to `a' on the next C-f.

The trunk version of the cursor positioning, which was rewritten for
support of bidirectional editing, behaves like that because it always
tries to find an exact match for point, and if found, puts the cursor
on the corresponding glyph.  It only considers positioning the cursor
on glyphs whose position is different if an exact match could not be
found.  IOW, exact match wins over all other considerations.  Since
C-f from the second `o' moves point to buffer position to which `b'
corresponds exactly, that is where the trunk version puts the cursor.

The old version puts the cursor on `-' because it examines the glyphs
from left to right, and finds the glyph with a `cursor' property
_before_ it has a chance to see that `b' has the required character
position.

Does anyone think that the version on the trunk is wrong and the old
one is right?

More generally, what are the use-cases for putting the `cursor'
property on a `before-string' or `after-string' overlay, and what is
expected from cursor positioning in those use-cases?

Re: Cursor positioning with `after-string' overlays

[Kim F. Storm]

Eli Zaretskii <eliz@gnu.org> writes:

> The old version puts the cursor on `-' because it examines the glyphs
> from left to right, and finds the glyph with a `cursor' property
> _before_ it has a chance to see that `b' has the required character
> position.
>
> Does anyone think that the version on the trunk is wrong and the old
> one is right?
>
> More generally, what are the use-cases for putting the `cursor'
> property on a `before-string' or `after-string' overlay, and what is
> expected from cursor positioning in those use-cases?

CUA rectangle mode may use the old behaviour to place cursor midway
through a tab character - I cannot judge from your example if that
is still working with the trunk code, and I don't have time to check
one out (I got lost in the Bazaar)...

But you can easily try it out.
Just enter:

M-x cua-mode RET
a C-q C-i b
C-a C-RET C-f C-f C-f ..

cursor should move successively from a through the tab until it reaches b

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Cursor positioning with `after-string' overlays

[Eli Zaretskii]

> From: storm@cua.dk (Kim F. Storm)
> Cc: emacs-devel@gnu.org
> Date: Thu, 01 Apr 2010 23:54:16 +0200
> 
> CUA rectangle mode may use the old behaviour to place cursor midway
> through a tab character - I cannot judge from your example if that
> is still working with the trunk code, and I don't have time to check
> one out (I got lost in the Bazaar)...
> 
> But you can easily try it out.
> Just enter:
> 
> M-x cua-mode RET
> a C-q C-i b
> C-a C-RET C-f C-f C-f ..
> 
> cursor should move successively from a through the tab until it reaches b

You are right, it doesn't work.  But it doesn't work in Emacs 23.1 and
in Emacs 22.3, either, at least not on MS-Windows and on GNU/Linux.

Are you sure the recipe is correct?  If so, in what version of Emacs
did it work for you?

Re: Cursor positioning with `after-string' overlays

[Eli Zaretskii]

> Date: Fri, 02 Apr 2010 10:53:11 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: storm@cua.dk (Kim F. Storm)
> > Cc: emacs-devel@gnu.org
> > Date: Thu, 01 Apr 2010 23:54:16 +0200
> > 
> > CUA rectangle mode may use the old behaviour to place cursor midway
> > through a tab character - I cannot judge from your example if that
> > is still working with the trunk code, and I don't have time to check
> > one out (I got lost in the Bazaar)...
> > 
> > But you can easily try it out.
> > Just enter:
> > 
> > M-x cua-mode RET
> > a C-q C-i b
> > C-a C-RET C-f C-f C-f ..
> > 
> > cursor should move successively from a through the tab until it reaches b
> 
> You are right, it doesn't work.  But it doesn't work in Emacs 23.1 and
> in Emacs 22.3, either, at least not on MS-Windows and on GNU/Linux.
> 
> Are you sure the recipe is correct?  If so, in what version of Emacs
> did it work for you?

I think I see the answer.  The recipe should be modified like this:

 M-x cua-mode RET
 a C-q C-i b RET
 C-p C-RET C-f C-f C-f ..

That is, there should be a newline after `b'.  When modified like
this, I see the expected behavior both with the current trunk and with
older versions.

Is this satisfactory?

Note that what cua-rect.el does differs from the example I posted in
one crucial aspect: cua-rect.el uses an integer number, rather than
just `t', for the value of the `cursor' property.  This number
actually tells the cursor-positioning code for which buffer positions
we should display the cursor on the character that has this property.
When the `cursor' property has an integer value, it really does
override the ``exact match for point always wins'' strategy, for
buffer positions that are ``covered'' by the value of `cursor'.

Re: Cursor positioning with `after-string' overlays

[Kim F. Storm]

Eli Zaretskii <eliz@gnu.org> writes:

> I think I see the answer.  The recipe should be modified like this:
>
>  M-x cua-mode RET
>  a C-q C-i b RET
>  C-p C-RET C-f C-f C-f ..
>
> That is, there should be a newline after `b'.  

Thanks - I just made the example in a non-empty buffer, so there was
a newline there already.

Surely looks like a bug in cua-rect.

>                                                When modified like
> this, I see the expected behavior both with the current trunk and with
> older versions.
>
> Is this satisfactory?

Yes.

> Note that what cua-rect.el does differs from the example I posted in
> one crucial aspect: cua-rect.el uses an integer number, rather than
> just `t', for the value of the `cursor' property.  This number
> actually tells the cursor-positioning code for which buffer positions
> we should display the cursor on the character that has this property.
> When the `cursor' property has an integer value, it really does
> override the ``exact match for point always wins'' strategy, for
> buffer positions that are ``covered'' by the value of `cursor'.

Right. I no longer remebered the details - thanks for digging into it.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: Cursor positioning with `after-string' overlays

[Stefan Monnier]

> found.  IOW, exact match wins over all other considerations.  Since
> C-f from the second `o' moves point to buffer position to which `b'
> corresponds exactly, that is where the trunk version puts the cursor.

But depending on the insertion-type of the end marker of your overlay,
text inserted "at point" will be inserted (visually) between the o and
the - rather than between the - and the b, so while this choice would
sometimes be correct, it's sometimes incorrect.  That's why we have the
`cursor' property (although admittedly, for this particular use, the
insertion-type of the marker should already provide the needed info).

> More generally, what are the use-cases for putting the `cursor'
> property on a `before-string' or `after-string' overlay, and what is
> expected from cursor positioning in those use-cases?

AFAIK, there are a few different use cases for the `cursor' property.

One use case is when you simply want to control where the cursor is
displayed on a piece of text that's not in the buffer (typically an
after-string).  In such a case, without any extra information, it would
not be incorrect to place the cursor before the after-string, or after
the after-string or anywhere in between.  So the `cursor' property
allows to specify the intended behavior (e.g. the after-string has the
form "()" and you want the cursor to appear in between the two parens).

The other use case is when the cursor positioning code gets it wrong
because it works at too low a level (typically the after-string or
similar thingy is on an overlay with carefully chosen stickiness which
should make it clear whether the cursor should come before or after the
string, but the cursor positioning code only gets to see a "flattened"
representation of the text, so it can't know the stickiness property).


        Stefan

Re: Cursor positioning with `after-string' overlays

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 01 Apr 2010 18:06:26 -0400
> Cc: emacs-devel@gnu.org
> 
> > found.  IOW, exact match wins over all other considerations.  Since
> > C-f from the second `o' moves point to buffer position to which `b'
> > corresponds exactly, that is where the trunk version puts the cursor.
> 
> But depending on the insertion-type of the end marker of your overlay,
> text inserted "at point" will be inserted (visually) between the o and
> the - rather than between the - and the b, so while this choice would
> sometimes be correct, it's sometimes incorrect.

Sorry, I don't understand the specific situation.  In my example, when
the cursor is on `b', insertion happens between `-' and `b', which is
visually correct.  Can you modify my example to create the situation
you are describing?  Then I could try to reason about it and perhaps
modify the code if necessary.

In any case, do you really think that being unable to put the cursor
on `b' (with the old code) is correct behavior?  The `after-string'
does not replace `b' on the screen and doesn't cover its position, so
this behavior looks like a subtle bug, since `b' comes from the
buffer.

> That's why we have the
> `cursor' property (although admittedly, for this particular use, the
> insertion-type of the marker should already provide the needed info).

What do you mean by ``the insertion-type of the marker''?  I've seen
your comments in minibuffer.el about something similar, so perhaps now
is the time to add whatever features you thought were absent.

> One use case is when you simply want to control where the cursor is
> displayed on a piece of text that's not in the buffer (typically an
> after-string).  In such a case, without any extra information, it would
> not be incorrect to place the cursor before the after-string, or after
> the after-string or anywhere in between.  So the `cursor' property
> allows to specify the intended behavior (e.g. the after-string has the
> form "()" and you want the cursor to appear in between the two parens).

This may make sense when the string is displayed _instead_ of some
portion of the buffer, or perhaps when we have a single buffer
position that uses up several columns on display, like in Kim's
example with a TAB in Cua Mode.

But if you type C-f, which moves point to the next buffer position,
and the glyph produced from that buffer position is displayed on the
screen, how can it be TRT not to place the cursor on that glyph?

> The other use case is when the cursor positioning code gets it wrong
> because it works at too low a level (typically the after-string or
> similar thingy is on an overlay with carefully chosen stickiness which
> should make it clear whether the cursor should come before or after the
> string, but the cursor positioning code only gets to see a "flattened"
> representation of the text, so it can't know the stickiness property).

I think this kind of problems should be fixed, not worked around with
the `cursor' property.  The cursor-positioning code does not work only
on glyphs, it does search the buffer for `display' properties and
overlays, so whatever information is available in the buffer, the
cursor-positioning code can get at it.

Re: Cursor positioning with `after-string' overlays

[Stefan Monnier]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Thu, 01 Apr 2010 18:06:26 -0400
>> Cc: emacs-devel@gnu.org
>> 
>> > found.  IOW, exact match wins over all other considerations.  Since
>> > C-f from the second `o' moves point to buffer position to which `b'
>> > corresponds exactly, that is where the trunk version puts the cursor.
>> 
>> But depending on the insertion-type of the end marker of your overlay,
>> text inserted "at point" will be inserted (visually) between the o and
>> the - rather than between the - and the b, so while this choice would
>> sometimes be correct, it's sometimes incorrect.

> Sorry, I don't understand the specific situation.  In my example, when
> the cursor is on `b', insertion happens between `-' and `b', which is
> visually correct.  Can you modify my example to create the situation
> you are describing?  Then I could try to reason about it and perhaps
> modify the code if necessary.

Try:

    (let ((pos (goto-char (point-max))))
      (insert "foobar")
      (overlay-put
	(make-overlay (+ pos 2) (+ pos 3) nil nil t)
	'after-string (propertize "-" 'cursor t)))

(i.e. I added "nil nil t" to the call to make-overlay).

> In any case, do you really think that being unable to put the cursor
> on `b' (with the old code) is correct behavior?

No.  The old behavior had its share of problems as well.

>> That's why we have the `cursor' property (although admittedly, for
>> this particular use, the insertion-type of the marker should already
>> provide the needed info).
> What do you mean by ``the insertion-type of the marker''?

I hope the sample code above makes it more clear (I'm talking about the
marker-insertion-type of the underlying/implicit markers used as end
points of the overlay).  For text-properties it's called stickiness.

>> One use case is when you simply want to control where the cursor is
>> displayed on a piece of text that's not in the buffer (typically an
>> after-string).  In such a case, without any extra information, it would
>> not be incorrect to place the cursor before the after-string, or after
>> the after-string or anywhere in between.  So the `cursor' property
>> allows to specify the intended behavior (e.g. the after-string has the
>> form "()" and you want the cursor to appear in between the two parens).
> This may make sense when the string is displayed _instead_ of some
> portion of the buffer, or perhaps when we have a single buffer
> position that uses up several columns on display, like in Kim's
> example with a TAB in Cua Mode.

Exactly, yes.

> But if you type C-f, which moves point to the next buffer position,
> and the glyph produced from that buffer position is displayed on the
> screen, how can it be TRT not to place the cursor on that glyph?

See my sample code above again or think of the minibuffer messages
during completion: depending on the particular case, the intention of
the text inserted via an after-string may be to appear "before" or
"after" the cursor.

>> The other use case is when the cursor positioning code gets it wrong
>> because it works at too low a level (typically the after-string or
>> similar thingy is on an overlay with carefully chosen stickiness which
>> should make it clear whether the cursor should come before or after the
>> string, but the cursor positioning code only gets to see a "flattened"
>> representation of the text, so it can't know the stickiness property).
> I think this kind of problems should be fixed, not worked around with
> the `cursor' property.  The cursor-positioning code does not work only
> on glyphs, it does search the buffer for `display' properties and
> overlays, so whatever information is available in the buffer, the
> cursor-positioning code can get at it.

IIRC last time I looked at it, it would require a significant
restructuring to get the needed information.  I can't remember the
details, but I seem to remember that one of the problem is that at the
moment we handle the cursor position all we know is "we're displaying
string S", so we know the text displayed comes from a string rather than
from a buffer, but we don't know where that string comes from: we've
forgotten all about whether it's from a display property, or an overlay,
so we can't check the corresponding stickiness/insertion-type.


        Stefan

Re: Cursor positioning with `after-string' overlays

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 02 Apr 2010 14:17:31 -0400
> 
> > Sorry, I don't understand the specific situation.  In my example, when
> > the cursor is on `b', insertion happens between `-' and `b', which is
> > visually correct.  Can you modify my example to create the situation
> > you are describing?  Then I could try to reason about it and perhaps
> > modify the code if necessary.
> 
> Try:
> 
>     (let ((pos (goto-char (point-max))))
>       (insert "foobar")
>       (overlay-put
> 	(make-overlay (+ pos 2) (+ pos 3) nil nil t)
> 	'after-string (propertize "-" 'cursor t)))
> 
> (i.e. I added "nil nil t" to the call to make-overlay).

Thanks, I will study this.

> >> One use case is when you simply want to control where the cursor is
> >> displayed on a piece of text that's not in the buffer (typically an
> >> after-string).  In such a case, without any extra information, it would
> >> not be incorrect to place the cursor before the after-string, or after
> >> the after-string or anywhere in between.  So the `cursor' property
> >> allows to specify the intended behavior (e.g. the after-string has the
> >> form "()" and you want the cursor to appear in between the two parens).
> > This may make sense when the string is displayed _instead_ of some
> > portion of the buffer, or perhaps when we have a single buffer
> > position that uses up several columns on display, like in Kim's
> > example with a TAB in Cua Mode.
> 
> Exactly, yes.

Okay, this kind of use-cases already works with the new code as it
did with the old (as far as my testing goes).

> > But if you type C-f, which moves point to the next buffer position,
> > and the glyph produced from that buffer position is displayed on the
> > screen, how can it be TRT not to place the cursor on that glyph?
> 
> See my sample code above again or think of the minibuffer messages
> during completion: depending on the particular case, the intention of
> the text inserted via an after-string may be to appear "before" or
> "after" the cursor.

The minibuffer completion use-case is different: there, we have no
text that comes from a buffer beyond the place where the cursor is
displayed.  IOW, there're no glyphs to the right of the cursor that
came from the minibuffer.  So there are no buffer positions that
"compete" for the cursor position, only the overlay with the
completion message.

> I seem to remember that one of the problem is that at the
> moment we handle the cursor position all we know is "we're displaying
> string S", so we know the text displayed comes from a string rather than
> from a buffer, but we don't know where that string comes from: we've
> forgotten all about whether it's from a display property, or an overlay,
> so we can't check the corresponding stickiness/insertion-type.

There's no "memory" in the glyphs of where the string came from,
that's true.  But that doesn't mean we cannot resurrect that
knowledge.  We already do something like that, see the call to the
function string_buffer_position_lim inside set_cursor_from_row.  If
required, we could extend that function, or call it for other similar
jobs when we position the cursor.  Or we could record the buffer
position where we found the string, and remove the need to look it up
again during cursor positioning.

Re: Cursor positioning with `after-string' overlays

[Stefan Monnier]

>> See my sample code above again or think of the minibuffer messages
>> during completion: depending on the particular case, the intention of
>> the text inserted via an after-string may be to appear "before" or
>> "after" the cursor.

> The minibuffer completion use-case is different: there, we have no
> text that comes from a buffer beyond the place where the cursor is
> displayed.  IOW, there're no glyphs to the right of the cursor that
> came from the minibuffer.  So there are no buffer positions that
> "compete" for the cursor position, only the overlay with the
> completion message.

I can imagine using the same kind of message for in-buffer completion.
So in my view the fact that there's no buffer text after the overlay in
the minibuffer completion case is just a lucky accident.

>> I seem to remember that one of the problem is that at the
>> moment we handle the cursor position all we know is "we're displaying
>> string S", so we know the text displayed comes from a string rather than
>> from a buffer, but we don't know where that string comes from: we've
>> forgotten all about whether it's from a display property, or an overlay,
>> so we can't check the corresponding stickiness/insertion-type.
> There's no "memory" in the glyphs of where the string came from,
> that's true.  But that doesn't mean we cannot resurrect that
> knowledge.  We already do something like that, see the call to the
> function string_buffer_position_lim inside set_cursor_from_row.  If
> required, we could extend that function, or call it for other similar
> jobs when we position the cursor.  Or we could record the buffer
> position where we found the string, and remove the need to look it up
> again during cursor positioning.

The position is not enough because we need to find the actual overlay
where it came from and there can be several overlays at that buffer
position.

But I do not object at all to your general argument "I think this kind
of problems should be fixed, not worked around with the `cursor'
property".  I seem to remember making the same observation, then jumping
to the C code, and finally deciding "too hard for me, I'll live with the
workaround for now".  So if you can fix it, that would be great.


        Stefan

Re: Cursor positioning with `after-string' overlays

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 02 Apr 2010 16:35:50 -0400
> 
> > The minibuffer completion use-case is different: there, we have no
> > text that comes from a buffer beyond the place where the cursor is
> > displayed.  IOW, there're no glyphs to the right of the cursor that
> > came from the minibuffer.  So there are no buffer positions that
> > "compete" for the cursor position, only the overlay with the
> > completion message.
> 
> I can imagine using the same kind of message for in-buffer completion.

Then you need to use the "integer as `cursor' property value"
feature.  I.e., don't just set the property's value non-nil, set it to
the integer number that specifies how many buffer positions are
``covered'' by that cursor positions.  That's what CUA Mode does in
cua-rect.el.  Integer property values do override the "exact match for
point always wins" strategy.

> > There's no "memory" in the glyphs of where the string came from,
> > that's true.  But that doesn't mean we cannot resurrect that
> > knowledge.  We already do something like that, see the call to the
> > function string_buffer_position_lim inside set_cursor_from_row.  If
> > required, we could extend that function, or call it for other similar
> > jobs when we position the cursor.  Or we could record the buffer
> > position where we found the string, and remove the need to look it up
> > again during cursor positioning.
> 
> The position is not enough because we need to find the actual overlay
> where it came from and there can be several overlays at that buffer
> position.

Right, but we have the string, so we could look for the overlay that
specifies the same string.

> But I do not object at all to your general argument "I think this kind
> of problems should be fixed, not worked around with the `cursor'
> property".  I seem to remember making the same observation, then jumping
> to the C code, and finally deciding "too hard for me, I'll live with the
> workaround for now".  So if you can fix it, that would be great.

I'd need to see the requirements first.

Re: Cursor positioning with `after-string' overlays

[Stefan Monnier]

>> > The minibuffer completion use-case is different: there, we have no
>> > text that comes from a buffer beyond the place where the cursor is
>> > displayed.  IOW, there're no glyphs to the right of the cursor that
>> > came from the minibuffer.  So there are no buffer positions that
>> > "compete" for the cursor position, only the overlay with the
>> > completion message.
>> I can imagine using the same kind of message for in-buffer completion.
> Then you need to use the "integer as `cursor' property value"
> feature.  I.e., don't just set the property's value non-nil, set it to
> the integer number that specifies how many buffer positions are
> ``covered'' by that cursor positions.  That's what CUA Mode does in
> cua-rect.el.  Integer property values do override the "exact match for
> point always wins" strategy.

Yes, that sounds OK.  But note that (except for when the after-string is
at EOB) there's basically always a "exact match for point", so that
basically means that the use of a value t for the `cursor' property will
simply not work any more and might just be dropped.

>> The position is not enough because we need to find the actual overlay
>> where it came from and there can be several overlays at that buffer
>> position.
> Right, but we have the string, so we could look for the overlay that
> specifies the same string.

Yes, we could, tho of course the same string might be placed on several
overlays at the same buffer position, and we have to check whether it's
on a before-string or an after-string or ...
I.e. it's workable but it'd probably be ugly.

>> But I do not object at all to your general argument "I think this kind
>> of problems should be fixed, not worked around with the `cursor'
>> property".  I seem to remember making the same observation, then jumping
>> to the C code, and finally deciding "too hard for me, I'll live with the
>> workaround for now".  So if you can fix it, that would be great.
> I'd need to see the requirements first.

Just that it places the cursor on the "before" or "after" position of
(after|before|display)-strings depending on the
stickiness/insertion-type of the corresponding overlay/text-property.


        Stefan

Re: Cursor positioning with `after-string' overlays

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 02 Apr 2010 21:21:09 -0400
> Cc: emacs-devel@gnu.org
> 
> >> I can imagine using the same kind of message for in-buffer completion.
> > Then you need to use the "integer as `cursor' property value"
> > feature.  I.e., don't just set the property's value non-nil, set it to
> > the integer number that specifies how many buffer positions are
> > ``covered'' by that cursor positions.  That's what CUA Mode does in
> > cua-rect.el.  Integer property values do override the "exact match for
> > point always wins" strategy.
> 
> Yes, that sounds OK.  But note that (except for when the after-string is
> at EOB) there's basically always a "exact match for point", so that
> basically means that the use of a value t for the `cursor' property will
> simply not work any more and might just be dropped.

Not necessarily: when the overlay (or text property) has a `display'
property, the value of the property is displayed _instead_ of the
text.  In that case, there might be no exact match for point, and the
value `t' still has its merit.

> Just that it places the cursor on the "before" or "after" position of
> (after|before|display)-strings depending on the
> stickiness/insertion-type of the corresponding overlay/text-property.

Okay, I will try to look into this.

Re: redisplay code is ugly (was: Cursor positioning with `after-string' overlays)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 02 Apr 2010 21:21:09 -0400
> Cc: emacs-devel@gnu.org
> 
> I.e. it's workable but it'd probably be ugly.

Ugly is xdisp.c's middle name.  I hate to say it, but almost every
important function of the display engine has become so convoluted and
under-commented that they all simply cry for refactoring.  Look at
move_it_in_display_line_to or at display_line -- with all the
different temporary `struct it' and flag variables, the amount of
cruft there is so large that it is next to impossible to even grasp
the overall control and data flow there, let alone make a non-trivial
modification.  (set_cursor_from_row was in a similar state; I hope
that after the rewrite on the trunk, it is a bit more manageable, but
I'd certainly love some critical peer review there.)

I frequently need to look at the Emacs 21 sources, just to understand
the original design and implementation.  At least there, they are
clearly visible and adequately commented.

My conclusion from almost a year of hacking the display engine is that
we are currently stretching it way past its original design
limitations.  Somebody(TM) should take a fresh look at the code and
refactor it.  I'm quite sure that quite a few of ugly places could be
rectified by adding some data to the core data structures, for
example.

Emacs 24 sounds like a good opportunity for such a refactoring.

And no, I'm not volunteering.  One reason is that I'm not good at this
job -- I try to stick to the original design on some deeply
subconscious level.

Re: Cursor positioning with `after-string' overlays

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 02 Apr 2010 21:21:09 -0400
> Cc: emacs-devel@gnu.org
> 
> >> But I do not object at all to your general argument "I think this kind
> >> of problems should be fixed, not worked around with the `cursor'
> >> property".  I seem to remember making the same observation, then jumping
> >> to the C code, and finally deciding "too hard for me, I'll live with the
> >> workaround for now".  So if you can fix it, that would be great.
> > I'd need to see the requirements first.
> 
> Just that it places the cursor on the "before" or "after" position of
> (after|before|display)-strings depending on the
> stickiness/insertion-type of the corresponding overlay/text-property.

Hmm... this could be possible.  But can you first specify, for each
combination of (after|before|display)-strings and the insertion-type
of the overlay's start and end, where would you want the cursor?  I
don't have enough experience using these features to be sure I can
decide this myself.

Re: Cursor positioning with `after-string' overlays

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Fri, 02 Apr 2010 14:17:31 -0400
> 
> >> But depending on the insertion-type of the end marker of your overlay,
> >> text inserted "at point" will be inserted (visually) between the o and
> >> the - rather than between the - and the b, so while this choice would
> >> sometimes be correct, it's sometimes incorrect.
> 
> > Sorry, I don't understand the specific situation.  In my example, when
> > the cursor is on `b', insertion happens between `-' and `b', which is
> > visually correct.  Can you modify my example to create the situation
> > you are describing?  Then I could try to reason about it and perhaps
> > modify the code if necessary.
> 
> Try:
> 
>     (let ((pos (goto-char (point-max))))
>       (insert "foobar")
>       (overlay-put
> 	(make-overlay (+ pos 2) (+ pos 3) nil nil t)
> 	'after-string (propertize "-" 'cursor t)))
> 
> (i.e. I added "nil nil t" to the call to make-overlay).

Yes, I see the issue now.  However, the old redisplay has a similar
problem when the additional arguments to make-overlay are "nil t nil":
the cursor is placed on `-', but the first insertion appears _after_
`-'.  So I'm not sure which one is better.  Maybe using the marker
insertion type (see my other email) will allow us to DTRT in each
case.