bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

1. Install company (from GNU ELPA, for instance).

2. Paste the contents of
https://gist.githubusercontent.com/sooheon/97a62f433897b52da3d1/raw/a42af658ec3ccd11a4faa7e2581f7413687b1811/gistfile1.txt
into the scratch buffer.

3. Look for the line with "hello halleo helo" at the end of the first
paragraph, and delete or add a few characters before it, to make sure
that the last "h" is two columns away from the right window border. So
as when you type "el" after it, the cursor is displayed in the margin.

4. M-x company-mode

5. (setq company-backends '(company-dabbrev))

6. Go to after the last "h", type "el", wait 0.3 seconds, see the
completion popup displayed, while the cursor is displayed in the margin.

Problem 1: even though the overlay's `cursor' property places it at the
same line as where "hel" ends, the cursor is rendered in the margin two
lines below.

Problem 2:

7. Type "l", see it wrapped to the next line.

8. Type backspace. See the cursor move to the second paragraph.

9. Continue backspacing. See the completion popup disappear, and the text
being deleted in the second paragraph.

Originally reported at
https://github.com/company-mode/company-mode/issues/362.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 19 Jun 2015 04:04:45 +0300
> 
> 1. Install company (from GNU ELPA, for instance).
> 
> 2. Paste the contents of
> https://gist.githubusercontent.com/sooheon/97a62f433897b52da3d1/raw/a42af658ec3ccd11a4faa7e2581f7413687b1811/gistfile1.txt
> into the scratch buffer.
> 
> 3. Look for the line with "hello halleo helo" at the end of the first
> paragraph, and delete or add a few characters before it, to make sure
> that the last "h" is two columns away from the right window border. So
> as when you type "el" after it, the cursor is displayed in the margin.
> 
> 4. M-x company-mode
> 
> 5. (setq company-backends '(company-dabbrev))
> 
> 6. Go to after the last "h", type "el", wait 0.3 seconds, see the
> completion popup displayed, while the cursor is displayed in the margin.
> 
> Problem 1: even though the overlay's `cursor' property places it at the
> same line as where "hel" ends, the cursor is rendered in the margin two
> lines below.

I'm halfway through investigating this, and this is what I saw till
now:

 . There's no 'cursor' property on the overlay string that Company
   creates for its "tooltip" of completions.  Or at least I couldn't
   find that property: I tried both "M-x describe-text-properties" and
   looking at the overlay string in GDB -- I see no 'cursor' property,
   certainly not on the newline that starts the string.  If indeed
   there is such a property there, please tell how to see it.

 . In any case, you cannot put the 'cursor' property on a newline and
   hope it to work: the newline doesn't leave any glyphs on display,
   certainly not when the cursor is displayed on the fringe.  So the
   display engine doesn't know you've put the 'cursor' property there.

 . The overlay string generated by Company in this case is
   problematic: it puts a newline at the end of each screen line, and
   that removes the last character of each screen line from display.
   I don't understand why you need to insert newlines when the
   original text lines were one long continued line.  This is a bug in
   Company.

> Problem 2:
> 
> 7. Type "l", see it wrapped to the next line.
> 
> 8. Type backspace. See the cursor move to the second paragraph.
> 
> 9. Continue backspacing. See the completion popup disappear, and the text
> being deleted in the second paragraph.

I didn't yet finish debugging this part, but I clearly see that some
code actually _moves_ point to that place in the second paragraph, I'm
not yet sure why.  For starters, if you turn off font-lock in the
buffer, this second problem doesn't happen at all.  I see some weird
interaction between JIT Font Lock and the post-command-hook installed
by Company, they seem to somehow conspire to force point to move to
that place.  I'll try to debug more to see why this happens.  (Any
idea why Company's post-command-hook calls sit-for, thus forcing
redisplay?)

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Date: Fri, 19 Jun 2015 22:07:47 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 20847@debbugs.gnu.org
> 
> > Problem 2:
> > 
> > 7. Type "l", see it wrapped to the next line.
> > 
> > 8. Type backspace. See the cursor move to the second paragraph.
> > 
> > 9. Continue backspacing. See the completion popup disappear, and the text
> > being deleted in the second paragraph.
> 
> I didn't yet finish debugging this part, but I clearly see that some
> code actually _moves_ point to that place in the second paragraph, I'm
> not yet sure why.  For starters, if you turn off font-lock in the
> buffer, this second problem doesn't happen at all.  I see some weird
> interaction between JIT Font Lock and the post-command-hook installed
> by Company, they seem to somehow conspire to force point to move to
> that place.  I'll try to debug more to see why this happens.  (Any
> idea why Company's post-command-hook calls sit-for, thus forcing
> redisplay?)

I think I understand why this is happening, and I find nothing wrong
with what the display engine does under these circumstances.

In a nutshell, when a screen line ends in a newline that comes from an
overlay string, we don't want to display the cursor on that line.  The
reasons are heuristic, but they give good results, and we used this
heuristic for a very long time, so changing it now is out of question.

Due to this, when you type Backspace to delete "l" in "hell", and the
Company post-command-hook runs and puts an overlay string on the same
line that begins with a newline, the display engine decides, after
redrawing the window, that it doesn't want to put the cursor there,
and looks for an alternative place.  The first such place is after the
overlay string, so point is moved there.

The font-lock part of this riddle is that when font-lock-mode is
active in the buffer, making any changes to buffer text cause JIT Lock
to spring to action, which doesn't really do anything, but disables a
certain redisplay optimization, which bypasses the above test.

My suggestion would be to use the 'cursor' property on the overlay
string in some place where it could be picked up by the display engine
(i.e. not on a newline), to countermand this problem.  E.g., perhaps
begin the overlay string a few characters earlier, so that it replaces
part of buffer text in "hel", and have the 'cursor' property on that
part of the string.

I still think the overlay string is constructed incorrectly in this
case, something that should be fixed in Company.  The above special
setting of 'cursor' could be part of that fix.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/20/2015 02:51 PM, Eli Zaretskii wrote:

> In a nutshell, when a screen line ends in a newline that comes from an
> overlay string, we don't want to display the cursor on that line.  The
> reasons are heuristic, but they give good results, and we used this
> heuristic for a very long time, so changing it now is out of question.

Do you have a scenario in mind that performs better under the current 
behavior?

> Due to this, when you type Backspace to delete "l" in "hell", and the
> Company post-command-hook runs and puts an overlay string on the same
> line that begins with a newline, the display engine decides, after
> redrawing the window, that it doesn't want to put the cursor there,
> and looks for an alternative place.  The first such place is after the
> overlay string, so point is moved there.

I can understand the scenario until now, but why move point?

Displaying cursor in a different place is relatively fine, but moving 
the point is destructive.

> The font-lock part of this riddle is that when font-lock-mode is
> active in the buffer, making any changes to buffer text cause JIT Lock
> to spring to action, which doesn't really do anything, but disables a
> certain redisplay optimization, which bypasses the above test.

Sounds messy.

> My suggestion would be to use the 'cursor' property on the overlay
> string in some place where it could be picked up by the display engine
> (i.e. not on a newline), to countermand this problem.

See the bottom of `company--replacement-string'. If `cursor' is applied 
unconditionally, and if I change the arguments 0 and 1 to 1 and 2, on 
step 6 the cursor is displayed at the beginning of the next line (so we 
know the change has effect), but the second problem (after step 9) is 
still present.

> E.g., perhaps
> begin the overlay string a few characters earlier, so that it replaces
> part of buffer text in "hel", and have the 'cursor' property on that
> part of the string.

That's doable, even if I don't like the extra complexity. Are you sure 
about this?

> I still think the overlay string is constructed incorrectly in this
> case, something that should be fixed in Company.  The above special
> setting of 'cursor' could be part of that fix.

Do you also have explanations for the following?

- The bug only manifests after the step 9 (backspacing), whereas the 
whole explanation seems to apply to the step 6 as well. Yet, point stays 
in place there.

- With bidi-display-reordering set to nil, there's no bug.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 21 Jun 2015 16:56:56 +0300
> 
> On 06/20/2015 02:51 PM, Eli Zaretskii wrote:
> 
> > In a nutshell, when a screen line ends in a newline that comes from an
> > overlay string, we don't want to display the cursor on that line.  The
> > reasons are heuristic, but they give good results, and we used this
> > heuristic for a very long time, so changing it now is out of question.
> 
> Do you have a scenario in mind that performs better under the current 
> behavior?

Any scenario where a screen line ends in a newline that comes from an
overlay string.  Try several such scenarios, and then tell me whether
the place we display the cursor looks better than the alternative.

> I can understand the scenario until now, but why move point?
> 
> Displaying cursor in a different place is relatively fine, but moving 
> the point is destructive.

Emacs cannot move cursor except by moving point, I'm sure you know
that.  The only exception is when we show the cursor on a display or
overlay string, guided by the 'cursor' property.  There are no other
exceptions.

> > The font-lock part of this riddle is that when font-lock-mode is
> > active in the buffer, making any changes to buffer text cause JIT Lock
> > to spring to action, which doesn't really do anything, but disables a
> > certain redisplay optimization, which bypasses the above test.
> 
> Sounds messy.

You can say that again.

> > My suggestion would be to use the 'cursor' property on the overlay
> > string in some place where it could be picked up by the display engine
> > (i.e. not on a newline), to countermand this problem.
> 
> See the bottom of `company--replacement-string'. If `cursor' is applied 
> unconditionally, and if I change the arguments 0 and 1 to 1 and 2, on 
> step 6 the cursor is displayed at the beginning of the next line (so we 
> know the change has effect), but the second problem (after step 9) is 
> still present.

AFAICT, this will put the 'cursor' property on a character that is
after the leading newline of the overlay string, yes?  If so, that's
not going to work: you need th 'cursor' property on some glyph that is
displayed on the same line where the newline is.  That is, you need to
make at least one character of "hel" part of the overlay string, and
put the 'cursor' property on it, making its value large enough to
"cover" the position of the newline.

> > E.g., perhaps
> > begin the overlay string a few characters earlier, so that it replaces
> > part of buffer text in "hel", and have the 'cursor' property on that
> > part of the string.
> 
> That's doable, even if I don't like the extra complexity. Are you sure 
> about this?

I didn't have time to actually try that, I just looked at the code.
So it might not work as things are, but if so, I'm quite sure I can
fix that.  As long as some glyph that came from the overlay string is
visible on the same line, and the corresponding string character has a
'cursor' property on it, the display engine has enough information to
decide that the cursor can be displayed there (on the fringe).

> Do you also have explanations for the following?
> 
> - The bug only manifests after the step 9 (backspacing), whereas the 
> whole explanation seems to apply to the step 6 as well. Yet, point stays 
> in place there.

Like I said, I didn't investigate that.  I think some redisplay
optimization is responsible.  If it's important to have the same
(mis)behavior in both cases, I can look into that.

> - With bidi-display-reordering set to nil, there's no bug.

Because the unidirectional display could assume that buffer positions
increase monotonically with the screen's vertical coordinate, and so
once it saw a single screen line whose first and last characters have
buffer positions on both sides of point, it could decide to put the
cursor on that line right there and then; it didn't need to test other
conditions or consider following screen lines.  So this situation
worked "by sheer luck".  The bidirectional display doesn't have that
luxury, so it needs additional support information, and that
information is simply absent in this case.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/21/2015 07:43 PM, Eli Zaretskii wrote:

> Any scenario where a screen line ends in a newline that comes from an
> overlay string.  Try several such scenarios, and then tell me whether
> the place we display the cursor looks better than the alternative.

Hmm, yeah, after sounds better that before. But we do display point in 
the margin on step 6. So it must be possible.

> Emacs cannot move cursor except by moving point, I'm sure you know
> that.  The only exception is when we show the cursor on a display or
> overlay string, guided by the 'cursor' property.  There are no other
> exceptions.

Not really. I only know that *I* can't move cursor by any other means.

I also vaguely recall someone (yourself?) stating that this limitation 
could be lifted without too much work.

>> See the bottom of `company--replacement-string'. If `cursor' is applied
>> unconditionally, and if I change the arguments 0 and 1 to 1 and 2, on
>> step 6 the cursor is displayed at the beginning of the next line (so we
>> know the change has effect), but the second problem (after step 9) is
>> still present.
>
> AFAICT, this will put the 'cursor' property on a character that is
> after the leading newline of the overlay string, yes?

True.

> If so, that's
> not going to work: you need th 'cursor' property on some glyph that is
> displayed on the same line where the newline is.

Augh.

Like mentioned in the previous message, I don't see a good, 
non-user-confusing place for it on the same line. The beginning of the 
next line would've worked reasonably well, though.

I'd also accept the cursor not being displayed at all.

 > That is, you need to
> make at least one character of "hel" part of the overlay string, and
> put the 'cursor' property on it, making its value large enough to
> "cover" the position of the newline.

I was kinda hoping that "overlay with display string starting with 
newline" was the only issue.

>> - The bug only manifests after the step 9 (backspacing), whereas the
>> whole explanation seems to apply to the step 6 as well. Yet, point stays
>> in place there.
>
> Like I said, I didn't investigate that.  I think some redisplay
> optimization is responsible.  If it's important to have the same
> (mis)behavior in both cases, I can look into that.

Couldn't the same optimization have a reason to be enabled in both 
cases? It might be worth investigating, at least.

Consistent behavior would also be good; so that the users don't have to 
try too hard to catch problematic cases like this one.

>The bidirectional display doesn't have that
> luxury, so it needs additional support information, and that
> information is simply absent in this case.

I see.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 21 Jun 2015 21:06:07 +0300
> 
> On 06/21/2015 07:43 PM, Eli Zaretskii wrote:
> 
> > Any scenario where a screen line ends in a newline that comes from an
> > overlay string.  Try several such scenarios, and then tell me whether
> > the place we display the cursor looks better than the alternative.
> 
> Hmm, yeah, after sounds better that before. But we do display point in 
> the margin on step 6. So it must be possible.

I'm quite sure the code which is responsible simply doesn't check.

> > Emacs cannot move cursor except by moving point, I'm sure you know
> > that.  The only exception is when we show the cursor on a display or
> > overlay string, guided by the 'cursor' property.  There are no other
> > exceptions.
> 
> Not really. I only know that *I* can't move cursor by any other means.
> 
> I also vaguely recall someone (yourself?) stating that this limitation 
> could be lifted without too much work.

It should be possible, yes.  I was describing how the code works now.

> >> See the bottom of `company--replacement-string'. If `cursor' is applied
> >> unconditionally, and if I change the arguments 0 and 1 to 1 and 2, on
> >> step 6 the cursor is displayed at the beginning of the next line (so we
> >> know the change has effect), but the second problem (after step 9) is
> >> still present.
> >
> > AFAICT, this will put the 'cursor' property on a character that is
> > after the leading newline of the overlay string, yes?
> 
> True.
> 
> > If so, that's
> > not going to work: you need th 'cursor' property on some glyph that is
> > displayed on the same line where the newline is.
> 
> Augh.
> 
> Like mentioned in the previous message, I don't see a good, 
> non-user-confusing place for it on the same line. The beginning of the 
> next line would've worked reasonably well, though.

Then what you had in mind should do the trick, I think.

> I'd also accept the cursor not being displayed at all.

That's tricky.  We only do that due to hscroll, I think.

>  > That is, you need to
> > make at least one character of "hel" part of the overlay string, and
> > put the 'cursor' property on it, making its value large enough to
> > "cover" the position of the newline.
> 
> I was kinda hoping that "overlay with display string starting with 
> newline" was the only issue.

That's not an issue at all.

What we need is some way of telling the display engine that this is
where we want the cursor, so it could include that clue in its logic.

Hmm... would it be possible to put a 'cursor' property on the newline
in the buffer that follows the "hel" text?  That might be all that's
needed to DTRT.

> >> - The bug only manifests after the step 9 (backspacing), whereas the
> >> whole explanation seems to apply to the step 6 as well. Yet, point stays
> >> in place there.
> >
> > Like I said, I didn't investigate that.  I think some redisplay
> > optimization is responsible.  If it's important to have the same
> > (mis)behavior in both cases, I can look into that.
> 
> Couldn't the same optimization have a reason to be enabled in both 
> cases? It might be worth investigating, at least.

I will see what I can do.

> Consistent behavior would also be good; so that the users don't have to 
> try too hard to catch problematic cases like this one.

I think we should try keeping point at the locus of
insertion/deletion.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Date: Sun, 21 Jun 2015 21:24:39 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 20847@debbugs.gnu.org
> 
> > >> - The bug only manifests after the step 9 (backspacing), whereas the
> > >> whole explanation seems to apply to the step 6 as well. Yet, point stays
> > >> in place there.
> > >
> > > Like I said, I didn't investigate that.  I think some redisplay
> > > optimization is responsible.  If it's important to have the same
> > > (mis)behavior in both cases, I can look into that.
> > 
> > Couldn't the same optimization have a reason to be enabled in both 
> > cases? It might be worth investigating, at least.
> 
> I will see what I can do.

The reason is that there's an invisible property on buffer text.  When
redisplay fails to place the cursor where it belongs, i.e. on the line
that ends with "hel", it looks for alternative strategies.  One of
those is for a use case where point is at the beginning of invisible
text that is before the 1st character displayed in the screen line.
In that case, we put cursor where the invisible text ends.

Of course, the code which handles that was not written for when the
same text also has an overlay string, so it misbehaves.

I can make the code bypass this in the case in point, but the result
will be that cursor will be displayed at the end of the first screen
line, which is hardly better.

So I think we should try telling the display engine where to put the
cursor via the 'cursor' property on buffer text, as proposed earlier.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/21/2015 10:23 PM, Eli Zaretskii wrote:

> The reason is that there's an invisible property on buffer text.  When
> redisplay fails to place the cursor where it belongs, i.e. on the line
> that ends with "hel", it looks for alternative strategies.

I guess the next question would be why redisplay fails to place the 
cursor after 9, but not after 6. The value of point, and the buffer 
contents, are the same in both situations, aren't they?

> I can make the code bypass this in the case in point, but the result
> will be that cursor will be displayed at the end of the first screen
> line, which is hardly better.

If cursor was displayed wherever but point didn't move, that would be 
better. But you probably mean to move point as well.

> So I think we should try telling the display engine where to put the
> cursor via the 'cursor' property on buffer text, as proposed earlier.

Okay, thanks.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/21/2015 09:24 PM, Eli Zaretskii wrote:

> Hmm... would it be possible to put a 'cursor' property on the newline
> in the buffer that follows the "hel" text?  That might be all that's
> needed to DTRT.

Possible, yes, but I'd rather not track an additional buffer 
modification I'd have to undo later.

>> Consistent behavior would also be good; so that the users don't have to
>> try too hard to catch problematic cases like this one.
>
> I think we should try keeping point at the locus of
> insertion/deletion.

Not sure what you mean here.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 21 Jun 2015 23:02:34 +0300
> 
> On 06/21/2015 09:24 PM, Eli Zaretskii wrote:
> 
> > Hmm... would it be possible to put a 'cursor' property on the newline
> > in the buffer that follows the "hel" text?  That might be all that's
> > needed to DTRT.
> 
> Possible, yes, but I'd rather not track an additional buffer 
> modification I'd have to undo later.

I understand, but doing that will allow an efficient and clean (from
the user POV) solution to this tricky situation.

> >> Consistent behavior would also be good; so that the users don't have to
> >> try too hard to catch problematic cases like this one.
> >
> > I think we should try keeping point at the locus of
> > insertion/deletion.
> 
> Not sure what you mean here.

I mean it's confusing to have point jump under your feet to a far-off
position in the middle of editing text.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/22/2015 05:45 AM, Eli Zaretskii wrote:

>>>> Consistent behavior would also be good; so that the users don't have to
>>>> try too hard to catch problematic cases like this one.
>>>
>>> I think we should try keeping point at the locus of
>>> insertion/deletion.
>>
>> Not sure what you mean here.
>
> I mean it's confusing to have point jump under your feet to a far-off
> position in the middle of editing text.

Yes, we do want to fix this bug (in Company).

But I meant something different: if the display engine worked 
consistently between step 6 and step 9, the bug would've been easier to 
reproduce, and might have already been fixed at this point.

Maybe I would've already noticed it myself (instead of waiting for a 
user to report it), and it could have influenced the choice of the popup 
rendering strategy.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/22/2015 05:45 AM, Eli Zaretskii wrote:

>>> Hmm... would it be possible to put a 'cursor' property on the newline
>>> in the buffer that follows the "hel" text?  That might be all that's
>>> needed to DTRT.
>>
>> Possible, yes, but I'd rather not track an additional buffer
>> modification I'd have to undo later.
>
> I understand, but doing that will allow an efficient and clean (from
> the user POV) solution to this tricky situation.

I think any reasonable solution should look efficient and clean to the user.

This doesn't seem to have any effect, though. Adding

       (when (eq ?\n (char-after (overlay-start ov)))
         (let ((inhibit-modification-hooks t))
           (put-text-property (overlay-start ov) (1+ (overlay-start ov)) 
'cursor t)))

to the end of company-pseudo-tooltip-unhide creates no change in the 
behavior.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 22 Jun 2015 16:40:35 +0300
> 
> This doesn't seem to have any effect, though. Adding
> 
>        (when (eq ?\n (char-after (overlay-start ov)))
>          (let ((inhibit-modification-hooks t))
>            (put-text-property (overlay-start ov) (1+ (overlay-start ov)) 
> 'cursor t)))
> 
> to the end of company-pseudo-tooltip-unhide creates no change in the 
> behavior.

That's because I have yet to write the code to support that ;-)

Will do next; meanwhile, could you perhaps send the above as diffs
relatively to whatever version of company.el you want me to test this
with?  TIA.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/22/2015 07:26 PM, Eli Zaretskii wrote:

>> This doesn't seem to have any effect, though. Adding
>>
>>         (when (eq ?\n (char-after (overlay-start ov)))
>>           (let ((inhibit-modification-hooks t))
>>             (put-text-property (overlay-start ov) (1+ (overlay-start ov))
>> 'cursor t)))
>>
>> to the end of company-pseudo-tooltip-unhide creates no change in the
>> behavior.
>
> That's because I have yet to write the code to support that ;-)

I'm not sure I like that idea. If I have to write a workaround (this 
definitely feels like it), I'd rather write a workaround that will work 
in the current Emacs releases too. (If I manage to, of course).

And supporting `cursor' on buffer text newline but not on overlay 
display string newline feels like adding a yet-another edge case.

> Will do next; meanwhile, could you perhaps send the above as diffs
> relatively to whatever version of company.el you want me to test this
> with?  TIA.

Maybe after I exhaust the other approaches.

Going back to your earlier message, I'm actually not sure I agree with this:

 > Any scenario where a screen line ends in a newline that comes from an
 > overlay string [performs better under the current behavior].  Try
 > several such scenarios, and then tell me whether
 > the place we display the cursor looks better than the alternative.

The current example shows that it's better to display the cursor on the 
margin, rather than after the overlay. What are the examples where this 
is not true? Are those cases where the said newline is not at the 
beginning of the overlay?

Even if cursor would look weird in some case, at least point is not 
forcibly moved to a different position.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 23 Jun 2015 00:06:06 +0300
> 
> On 06/22/2015 07:26 PM, Eli Zaretskii wrote:
> 
> >> This doesn't seem to have any effect, though. Adding
> >>
> >>         (when (eq ?\n (char-after (overlay-start ov)))
> >>           (let ((inhibit-modification-hooks t))
> >>             (put-text-property (overlay-start ov) (1+ (overlay-start ov))
> >> 'cursor t)))
> >>
> >> to the end of company-pseudo-tooltip-unhide creates no change in the
> >> behavior.
> >
> > That's because I have yet to write the code to support that ;-)
> 
> I'm not sure I like that idea. If I have to write a workaround (this 
> definitely feels like it), I'd rather write a workaround that will work 
> in the current Emacs releases too. (If I manage to, of course).

Would it be possible in this specific case to start the overlay from
the next line, i.e. leave the newline alone?  I think if you do that,
things ought to work without any changes.  Since the "tooltip" pops up
below this line, it seems to me that the newline which begins the
overlay now is not needed, is it?

> And supporting `cursor' on buffer text newline but not on overlay 
> display string newline feels like adding a yet-another edge case.

It's indeed a kludge.  I just didn't have any other ideas.

> Going back to your earlier message, I'm actually not sure I agree with this:
> 
>  > Any scenario where a screen line ends in a newline that comes from an
>  > overlay string [performs better under the current behavior].  Try
>  > several such scenarios, and then tell me whether
>  > the place we display the cursor looks better than the alternative.
> 
> The current example shows that it's better to display the cursor on the 
> margin, rather than after the overlay. What are the examples where this 
> is not true?

All the others ;-)

> Are those cases where the said newline is not at the beginning of
> the overlay?

No, I think what makes this case special is the fact that (the visible
part of) the overlay begins on the line below the one where the user
types.  So from the user POV, the current line is still occupied only
by buffer text, and so users will expect the cursor to be displayed as
if there were no overlay at all.

By contrast, the "usual" case with overlays that span multiple lines
is that the situation with positioning the cursor arises when some of
the overlay string is visible on the current line, and then the user
expectation is to see the cursor after the string.  That's what the
code which handles this situation tries to make happen.

> Even if cursor would look weird in some case, at least point is not 
> forcibly moved to a different position.

We are in agreement regarding the best place of showing the cursor in
this case.  The only problem is how to do that.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/23/2015 07:39 PM, Eli Zaretskii wrote:

> Would it be possible in this specific case to start the overlay from
> the next line, i.e. leave the newline alone?  I think if you do that,
> things ought to work without any changes.

I suppose.

> Since the "tooltip" pops up
> below this line, it seems to me that the newline which begins the
> overlay now is not needed, is it?

"now"?

If the line after the current has a `display' property, it could still 
lead to the problem like bug#18285. However, `M-x report-emacs-bug' 
buffer (which was the main example for that bug) doesn't have 
particularly long lines, so we could hope the workaround won't be 
triggered there. But anyway, wrong rendering is still better than moving 
point to a different place.

>> The current example shows that it's better to display the cursor on the
>> margin, rather than after the overlay. What are the examples where this
>> is not true?
>
> All the others ;-)

A concrete example would help. The fact that you'd *try to* display the 
cursor at the newline belonging to an overlay display string indicates 
that the overlay must start at that position, doesn't it? Or end.

If it starts earlier, then the cursor might be displayed before or after 
(if wouldn't be displayed in the middle of the overlay string, right?).

> No, I think what makes this case special is the fact that (the visible
> part of) the overlay begins on the line below the one where the user
> types.  So from the user POV, the current line is still occupied only
> by buffer text, and so users will expect the cursor to be displayed as
> if there were no overlay at all.

If I had to pick, I'd probably always display the cursor before such 
overlays, not after. That seems consistent with the logic of expecting 
the cursor "to be displayed as if...".

> By contrast, the "usual" case with overlays that span multiple lines
> is that the situation with positioning the cursor arises when some of
> the overlay string is visible on the current line, and then the user
> expectation is to see the cursor after the string.  That's what the
> code which handles this situation tries to make happen.

If the overlay display string ends at that newline, and point is at the 
end of the overlay, then the display engine exception under the 
discussion will be a no-op. If it ends later, why would we even try to 
display the cursor at that newline in the first place?

>> Even if cursor would look weird in some case, at least point is not
>> forcibly moved to a different position.
>
> We are in agreement regarding the best place of showing the cursor in
> this case.  The only problem is how to do that.

Removing that exception might do the trick. But then, maybe I don't 
really understand what it does.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 23 Jun 2015 21:44:58 +0300
> 
> On 06/23/2015 07:39 PM, Eli Zaretskii wrote:
> 
> > Would it be possible in this specific case to start the overlay from
> > the next line, i.e. leave the newline alone?  I think if you do that,
> > things ought to work without any changes.
> 
> I suppose.
> 
> > Since the "tooltip" pops up
> > below this line, it seems to me that the newline which begins the
> > overlay now is not needed, is it?
> 
> "now"?

"Now" as in "what Company does now".

> >> The current example shows that it's better to display the cursor on the
> >> margin, rather than after the overlay. What are the examples where this
> >> is not true?
> >
> > All the others ;-)
> 
> A concrete example would help.

emacs -Q
M-: (overlay-put (make-overlay 65 66) 'before-string "how\ndo\nyou\ndo?") RET

> The fact that you'd *try to* display the cursor at the newline
> belonging to an overlay display string indicates that the overlay
> must start at that position, doesn't it? Or end.

Sorry, I don't follow: what do you mean by "you'd try"?

> If it starts earlier, then the cursor might be displayed before or after 

We always display it after, when point is at buffer position that has
an overlay string property.

> (if wouldn't be displayed in the middle of the overlay string, right?).

Never (unless there's a character with 'cursor' property).

> > No, I think what makes this case special is the fact that (the visible
> > part of) the overlay begins on the line below the one where the user
> > types.  So from the user POV, the current line is still occupied only
> > by buffer text, and so users will expect the cursor to be displayed as
> > if there were no overlay at all.
> 
> If I had to pick, I'd probably always display the cursor before such 
> overlays, not after.

That goes against what Emacs did since we began supporting overlay
strings.  Among other problems, it makes strange effects when
inserting characters: they appear not where the cursor is.

> That seems consistent with the logic of expecting the cursor "to be
> displayed as if...".

Again, this case is IMO singular, in that none of the overlay string
characters are visible on that line.  Thus "as if...".

> > By contrast, the "usual" case with overlays that span multiple lines
> > is that the situation with positioning the cursor arises when some of
> > the overlay string is visible on the current line, and then the user
> > expectation is to see the cursor after the string.  That's what the
> > code which handles this situation tries to make happen.
> 
> If the overlay display string ends at that newline, and point is at the 
> end of the overlay, then the display engine exception under the 
> discussion will be a no-op.

What exception?  Sorry, I lost track: we are discussing many different
things simultaneously.

> If it ends later, why would we even try to display the cursor at
> that newline in the first place?

Because its corresponding buffer position matches point.  That's how
cursor positioning works in Emacs: it always starts by looking for the
glyph on the screen that corresponds to the buffer position where
point is.  If such a glyph is not found, for one of the many reasons
(invisible text, overlay or display string, hscroll, you name it),
then the fun begins: we try very hard to find some alternative
position that fits.

> >> Even if cursor would look weird in some case, at least point is not
> >> forcibly moved to a different position.
> >
> > We are in agreement regarding the best place of showing the cursor in
> > this case.  The only problem is how to do that.
> 
> Removing that exception might do the trick.

It will bring other problems.

> But then, maybe I don't really understand what it does.

I can tell you which parts of code to read, if you want.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/23/2015 10:07 PM, Eli Zaretskii wrote:

> "Now" as in "what Company does now".

It very much uses that newline "now". The most recent major change, 
IIRC, was related to working around the previously mentioned 
invisible<->display bug.

> emacs -Q
> M-: (overlay-put (make-overlay 65 66) 'before-string "how\ndo\nyou\ndo?") RET

But there's nothing special about newlines here. The display engine 
probably never "tried" to display the cursor at any of those that are 
inside the display string.

>> The fact that you'd *try to* display the cursor at the newline
>> belonging to an overlay display string indicates that the overlay
>> must start at that position, doesn't it? Or end.
>
> Sorry, I don't follow: what do you mean by "you'd try"?

Consider for displaying the cursor at?

Allow me to quote yourself: "In a nutshell, when a screen line ends in a 
newline that comes from an overlay string, we don't want to display the 
cursor on that line."

For that heuristic to apply, I'd say first you have to "try" to display 
it at that position.

>> If it starts earlier, then the cursor might be displayed before or after
>
> We always display it after, when point is at buffer position that has
> an overlay string property.

So maybe I misunderstood, and there's no heuristic about newline at the 
end of visual line coming from an overlay. And that other circumstances 
conspire to make it seem that way, namely:

- Cursor is always displayed after an overlay when it's "inside" it.
- We somehow always consider point to be "inside" overlay when it's at 
the window edge, and an overlay begins there too.

Maybe the latter could still be changed.

>> (if wouldn't be displayed in the middle of the overlay string, right?).
>
> Never (unless there's a character with 'cursor' property).

Yes, we're not considering this case, for now.

>> If I had to pick, I'd probably always display the cursor before such
>> overlays, not after.
>
> That goes against what Emacs did since we began supporting overlay
> strings.  Among other problems, it makes strange effects when
> inserting characters: they appear not where the cursor is.

Inserting characters with overlays applied already? If you're talking 
about one existing overlay, maybe the behavior should depend on the 
stickiness of the overlay's bounds.

> Again, this case is IMO singular, in that none of the overlay string
> characters are visible on that line.  Thus "as if...".

Maybe it is singular.

>> If the overlay display string ends at that newline, and point is at the
>> end of the overlay, then the display engine exception under the
>> discussion will be a no-op.
>
> What exception?  Sorry, I lost track: we are discussing many different
> things simultaneously.

Again, refer to the quote above. And I probably have misunderstood.

>> If it ends later, why would we even try to display the cursor at
>> that newline in the first place?
>
> Because its corresponding buffer position matches point.  That's how
> cursor positioning works in Emacs: it always starts by looking for the
> glyph on the screen that corresponds to the buffer position where
> point is.  If such a glyph is not found, for one of the many reasons
> (invisible text, overlay or display string, hscroll, you name it),
> then the fun begins: we try very hard to find some alternative
> position that fits.

Then the overlay must begin at the edge of the window, and all the cases 
where your quote (above) applies look like this case.

>> But then, maybe I don't really understand what it does.
>
> I can tell you which parts of code to read, if you want.

If you think that's wise.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 24 Jun 2015 00:15:57 +0300
> 
> On 06/23/2015 10:07 PM, Eli Zaretskii wrote:
> 
> > "Now" as in "what Company does now".
> 
> It very much uses that newline "now".

Yes, and I was asking whether in this specific case it could refrain
from doing that.  (And yes, I understand that doing so would be
additional complexity for you.)

> > emacs -Q
> > M-: (overlay-put (make-overlay 65 66) 'before-string "how\ndo\nyou\ndo?") RET
> 
> But there's nothing special about newlines here. The display engine 
> probably never "tried" to display the cursor at any of those that are 
> inside the display string.

Yes, it did.  When the display engine needs to decide where to put the
cursor, it examines all the screen lines whose buffer positions are
around point.  So in this case, it did examine all those lines that
came from an overlay string, and rejected them all.

> >> The fact that you'd *try to* display the cursor at the newline
> >> belonging to an overlay display string indicates that the overlay
> >> must start at that position, doesn't it? Or end.
> >
> > Sorry, I don't follow: what do you mean by "you'd try"?
> 
> Consider for displaying the cursor at?
> 
> Allow me to quote yourself: "In a nutshell, when a screen line ends in a 
> newline that comes from an overlay string, we don't want to display the 
> cursor on that line."
> 
> For that heuristic to apply, I'd say first you have to "try" to display 
> it at that position.

We do, see above.  We actually examine all those lines one by one.

> >> If it starts earlier, then the cursor might be displayed before or after
> >
> > We always display it after, when point is at buffer position that has
> > an overlay string property.
> 
> So maybe I misunderstood, and there's no heuristic about newline at the 
> end of visual line coming from an overlay.

No, you didn't misunderstand.  There is such a heuristic.

> And that other circumstances conspire to make it seem that way,
> namely:
> 
> - Cursor is always displayed after an overlay when it's "inside" it.
> - We somehow always consider point to be "inside" overlay when it's at 
> the window edge, and an overlay begins there too.

That's correct, but the heuristic applies to the line where the
overlay string ends, when it ends with a newline that comes from the
overlay string.  When this happens, we could place the cursor either
at the end of the line with that newline, or at the beginning of the
next line.  We prefer the latter.

> Maybe the latter could still be changed.

I tried to do that, but unfortunately, when the display engine gets to
the point where it needs to place the cursor, it lacks information for
treating this situation specially.  That's because the information
about the overlay is long gone, and the newline didn't leave any
glyphs on the screen.  So it's impossible to discern between this
situation and any other where we have a a line ending in a newline
that comes from an overlay string, without once again searching for
the overlays in the buffer, soring them, finding the ones that should
have been displayed, etc.

> >> If it ends later, why would we even try to display the cursor at
> >> that newline in the first place?
> >
> > Because its corresponding buffer position matches point.  That's how
> > cursor positioning works in Emacs: it always starts by looking for the
> > glyph on the screen that corresponds to the buffer position where
> > point is.  If such a glyph is not found, for one of the many reasons
> > (invisible text, overlay or display string, hscroll, you name it),
> > then the fun begins: we try very hard to find some alternative
> > position that fits.
> 
> Then the overlay must begin at the edge of the window, and all the cases 
> where your quote (above) applies look like this case.

Yes.  And that's exactly the problem: we would like to treat this case
differently, but don't have the necessary information to do so.

What I need to do so is (1) the starting and ending point of the
overlay which begot the string, and (2) the fact that the string
begins with a newline, or some handle for the string itself.  None of
that is available at the point where we decide whether this line is
"appropriate" for the cursor.

> >> But then, maybe I don't really understand what it does.
> >
> > I can tell you which parts of code to read, if you want.
> 
> If you think that's wise.

It's not up to me, it's up to you.  If you want to understand the gory
details without myself standing in between (and maybe describing some
things inaccurately or not clearly enough), then I will gladly tell
you where to look.  I'm also okay with describing it for you as best I
can, like I did until now.  It's your call.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/24/2015 07:18 PM, Eli Zaretskii wrote:

> Yes, and I was asking whether in this specific case it could refrain
> from doing that.  (And yes, I understand that doing so would be
> additional complexity for you.)

Probably yes. But the problem is not only in complexity, but also in the 
lack of testability (it's not easy to write automatic checks to see if 
each case behaves as expected; or rather, virtually impossible).

Chiefly, I need a reliable predicate. Say I have computed the current 
visual column of point. What do I compare it to?

We have this function to determine the "usable" part of the window's width:

(defun company--window-width ()
   (let ((ww (window-body-width)))
     ;; Account for the line continuation column.
     (when (zerop (cadr (window-fringes)))
       (cl-decf ww))
     (unless (or (display-graphic-p)
                 (version< "24.3.1" emacs-version))
       ;; Emacs 24.3 and earlier included margins
       ;; in window-width when in TTY.
       (cl-decf ww
                (let ((margins (window-margins)))
                  (+ (or (car margins) 0)
                     (or (cdr margins) 0)))))
     (when (and word-wrap
                (version< emacs-version "24.4.51.5"))
       ;; http://debbugs.gnu.org/19300
       (cl-decf ww))
     ;; whitespace-mode with newline-mark
     (when (and buffer-display-table
                (aref buffer-display-table ?\n))
       (cl-decf ww (1- (length (aref buffer-display-table ?\n)))))
     ww))

Do I compare the visual column against the value it returns, or against 
straight (window-body-width), or against something in between?

> Yes, it did.  When the display engine needs to decide where to put the
> cursor, it examines all the screen lines whose buffer positions are
> around point.  So in this case, it did examine all those lines that
> came from an overlay string, and rejected them all.

We could say that if the positions were rejected for other reasons, 
then, for the purposes of this discussion, they weren't considered for 
displaying the cursor at.

> That's correct, but the heuristic applies to the line where the
> overlay string ends, when it ends with a newline that comes from the
> overlay string.  When this happens, we could place the cursor either
> at the end of the line with that newline, or at the beginning of the
> next line.  We prefer the latter.

Why not prefer the former? And use the margin if the cursor goes behind 
the edge as a result.

Unless there are cases which would lead to accidental point movements 
with this choice as well, this would seem like a better one.

> I tried to do that, but unfortunately, when the display engine gets to
> the point where it needs to place the cursor, it lacks information for
> treating this situation specially.  That's because the information
> about the overlay is long gone, and the newline didn't leave any
> glyphs on the screen.

How is the heuristic applied, then? It somehow needs to know, IIUC, that 
a string coming from a overlay is there.

> Yes.  And that's exactly the problem: we would like to treat this case
> differently, but don't have the necessary information to do so.
>
> What I need to do so is (1) the starting and ending point of the
> overlay which begot the string, and (2) the fact that the string
> begins with a newline, or some handle for the string itself.  None of
> that is available at the point where we decide whether this line is
> "appropriate" for the cursor.

Maybe the decision where to put the cursor should be made earlier, then.

> It's not up to me, it's up to you.  If you want to understand the gory
> details without myself standing in between (and maybe describing some
> things inaccurately or not clearly enough), then I will gladly tell
> you where to look.  I'm also okay with describing it for you as best I
> can, like I did until now.  It's your call.

I wouldn't mind taking a glance (just out of curiosity, or for a distant 
future reference), but you'll most likely have to continue translating 
anyway.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 29 Jun 2015 18:48:09 +0300
> 
> Chiefly, I need a reliable predicate. Say I have computed the current 
> visual column of point. What do I compare it to?

Are we talking about detecting the situation where the last
(rightmost) character displayed on a visual line leaves no space for
the newline, therefore forcing the newline be displayed on the fringe?

If so, does "point" here mean that buffer position of that rightmost
character?

> (defun company--window-width ()
>    (let ((ww (window-body-width)))
>      ;; Account for the line continuation column.
>      (when (zerop (cadr (window-fringes)))
>        (cl-decf ww))
>      (unless (or (display-graphic-p)
>                  (version< "24.3.1" emacs-version))
>        ;; Emacs 24.3 and earlier included margins
>        ;; in window-width when in TTY.
>        (cl-decf ww
>                 (let ((margins (window-margins)))
>                   (+ (or (car margins) 0)
>                      (or (cdr margins) 0)))))
>      (when (and word-wrap
>                 (version< emacs-version "24.4.51.5"))
>        ;; http://debbugs.gnu.org/19300
>        (cl-decf ww))
>      ;; whitespace-mode with newline-mark
>      (when (and buffer-display-table
>                 (aref buffer-display-table ?\n))
>        (cl-decf ww (1- (length (aref buffer-display-table ?\n)))))
>      ww))
> 
> Do I compare the visual column against the value it returns, or against 
> straight (window-body-width), or against something in between?

In general, you compare with the value it returns, I think.  But I
will need to think about this some more.  (The issue that bothers me
is the width of that last character -- it could be wider than one
column.  Also, I think the width of the next "display element", the
one that will be displayed on the next visual line, might affect the
issue.)

> > Yes, it did.  When the display engine needs to decide where to put the
> > cursor, it examines all the screen lines whose buffer positions are
> > around point.  So in this case, it did examine all those lines that
> > came from an overlay string, and rejected them all.
> 
> We could say that if the positions were rejected for other reasons, 
> then, for the purposes of this discussion, they weren't considered for 
> displaying the cursor at.

They were rejected for the reason we are discussing, not for other
reasons.

> > That's correct, but the heuristic applies to the line where the
> > overlay string ends, when it ends with a newline that comes from the
> > overlay string.  When this happens, we could place the cursor either
> > at the end of the line with that newline, or at the beginning of the
> > next line.  We prefer the latter.
> 
> Why not prefer the former?

Insertion point, I think.  If you type characters, they will appear on
the next line, which is confusing.

> > I tried to do that, but unfortunately, when the display engine gets to
> > the point where it needs to place the cursor, it lacks information for
> > treating this situation specially.  That's because the information
> > about the overlay is long gone, and the newline didn't leave any
> > glyphs on the screen.
> 
> How is the heuristic applied, then? It somehow needs to know, IIUC, that 
> a string coming from a overlay is there.

It does know that, but that's all it knows.  Specifically, the last
examined string position is recorded in the "glyph row" structure;
when that is non-negative, we know that there's some string there.  If
a glyph produced from the string is available, then we can also access
the string itself, but in the case we are discussing there are no
glyphs.

> > Yes.  And that's exactly the problem: we would like to treat this case
> > differently, but don't have the necessary information to do so.
> >
> > What I need to do so is (1) the starting and ending point of the
> > overlay which begot the string, and (2) the fact that the string
> > begins with a newline, or some handle for the string itself.  None of
> > that is available at the point where we decide whether this line is
> > "appropriate" for the cursor.
> 
> Maybe the decision where to put the cursor should be made earlier, then.

I don't see how this is possible: you cannot place the cursor on a
visual line before you have fully laid out that line.  And that's what
the code does: it invokes the function that finds where to place the
cursor immediately after layout of the line is completed.

It might be possible to record more information, though, and use that
to make the right decisions.

But once again, you wanted a solution that will work with currently
released Emacs, so such changes are out of scope for now.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/30/2015 08:46 PM, Eli Zaretskii wrote:

> If so, does "point" here mean that buffer position of that rightmost
> character?

Let's say yes.

> In general, you compare with the value it returns, I think.  But I
> will need to think about this some more.  (The issue that bothers me
> is the width of that last character -- it could be wider than one
> column.  Also, I think the width of the next "display element", the
> one that will be displayed on the next visual line, might affect the
> issue.)

That's why I asked. Creating a yet another variation of this function 
isn't going be very fun, especially when it's hard to test automatically.

>> Why not prefer the former?
>
> Insertion point, I think.  If you type characters, they will appear on
> the next line, which is confusing.

Isn't it the same in the absence of any overlays, if point is under the 
edge of the window (and so the cursor is displayed on the margin)?

Nothing particularly confusing about that.

> It does know that, but that's all it knows.  Specifically, the last
> examined string position is recorded in the "glyph row" structure;
> when that is non-negative, we know that there's some string there.  If
> a glyph produced from the string is available, then we can also access
> the string itself, but in the case we are discussing there are no
> glyphs.

I see.

>> Maybe the decision where to put the cursor should be made earlier, then.
>
> I don't see how this is possible: you cannot place the cursor on a
> visual line before you have fully laid out that line.  And that's what
> the code does: it invokes the function that finds where to place the
> cursor immediately after layout of the line is completed.

Maybe it could be performed in two stages: on some higher level, you 
maybe adjust the value of point if it has ventured beyond the currently 
visible area.

On the lower level, when drawing glyphs, you can adjust the placement of 
cursor, but without affecting point. Admittedly, that sounds like a 
non-trivial change.

> It might be possible to record more information, though, and use that
> to make the right decisions.
>
> But once again, you wanted a solution that will work with currently
> released Emacs, so such changes are out of scope for now.

What I said is, if it's a hack, then I want a hack that will work today.

But if a "proper" solution is introduced (one that won't make the popup 
code in Company more complex overall), then I'm not above telling all 
users to either disable bidi, or upgrade to the latest Emacs.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 30 Jun 2015 22:41:57 +0300
> 
> >> Why not prefer the former?
> >
> > Insertion point, I think.  If you type characters, they will appear on
> > the next line, which is confusing.
> 
> Isn't it the same in the absence of any overlays, if point is under the 
> edge of the window (and so the cursor is displayed on the margin)?

No, in that case the characters are inserted before the newline,
i.e. on the same line.

> > But once again, you wanted a solution that will work with currently
> > released Emacs, so such changes are out of scope for now.
> 
> What I said is, if it's a hack, then I want a hack that will work today.

If you can avoid the leading newline in the overlay string, there's no
hack here.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/30/2015 11:11 PM, Eli Zaretskii wrote:

> No, in that case the characters are inserted before the newline,
> i.e. on the same line.

That is also true if point is before an overlay-rendered newline, isn't it?

>> What I said is, if it's a hack, then I want a hack that will work today.
>
> If you can avoid the leading newline in the overlay string, there's no
> hack here.

I don't think I can. Like mentioned, it's the solution for 
http://debbugs.gnu.org/18285, which commonly appeared in 
report-emacs-bug buffers (so it's not like a rare case I could sweep 
under a rug).

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 30 Jun 2015 23:20:03 +0300
> 
> On 06/30/2015 11:11 PM, Eli Zaretskii wrote:
> 
> > No, in that case the characters are inserted before the newline,
> > i.e. on the same line.
> 
> That is also true if point is before an overlay-rendered newline, isn't it?

Yes, but that's not the situation we are talking about.  We are
talking about point being _covered" by an overlay string that ends in
a newline.

> >> What I said is, if it's a hack, then I want a hack that will work today.
> >
> > If you can avoid the leading newline in the overlay string, there's no
> > hack here.
> 
> I don't think I can. Like mentioned, it's the solution for 
> http://debbugs.gnu.org/18285, which commonly appeared in 
> report-emacs-bug buffers (so it's not like a rare case I could sweep 
> under a rug).

I meant only in this special situation.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 07/01/2015 05:42 AM, Eli Zaretskii wrote:

> Yes, but that's not the situation we are talking about.  We are
> talking about point being _covered" by an overlay string that ends in
> a newline.

Fair point.

> I meant only in this special situation.

It will be a big hack: just look at the huge predicate we'll need to 
detect it. This situation is not special from the standpoint of the 
underlying logic.

Anyway, I'm inclining toward revisiting the fix for #18285, and doing a 
smaller hack: using `display' normally, and only resorting to 
`invisible' + `after-string' when the overlay is empty.

I'll report how it goes.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 1 Jul 2015 13:21:12 +0300
> 
> > I meant only in this special situation.
> 
> It will be a big hack: just look at the huge predicate we'll need to 
> detect it.

It might be easier than you think: it turns out that when point is on
a newline that is displayed on the fringe, you can detect that like
this:

   (posn-area (posn-at-point)) => right-fringe

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 07/01/2015 06:16 PM, Eli Zaretskii wrote:

>     (posn-area (posn-at-point)) => right-fringe

Thanks, I'll revisit this if the current solution springs a new leak.

For now, it seems to function well enough, across different Emacs 
versions, in report-emacs-bug buffer, at eol and eob.

I'm sorry I haven't gone this route sooner and spared you some of this 
long conversation. It's not easy to let go of one's hard-word workarounds.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847-done@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 1 Jul 2015 19:36:16 +0300
> 
> I'm sorry I haven't gone this route sooner and spared you some of this 
> long conversation.

No harm done.

> It's not easy to let go of one's hard-word workarounds.

I've had my share of those.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/19/2015 10:07 PM, Eli Zaretskii wrote:

Hi Eli,

sorry for the late reply.

>   . There's no 'cursor' property on the overlay string that Company
>     creates for its "tooltip" of completions.  Or at least I couldn't
>     find that property: I tried both "M-x describe-text-properties" and
>     looking at the overlay string in GDB -- I see no 'cursor' property,
>     certainly not on the newline that starts the string.  If indeed
>     there is such a property there, please tell how to see it.

I'm sorry, my memory of that code was unclear. `cursor' is currently 
used only under certain rare condition, but see the end of 
`company--replacement-string', where it's applied.

Even if remove the "when" condition to put it on all the time, the 
behavior doesn't change.

By the way, since we're now discussing the changes to Company code, 
let's use the master version from 
git@github.com:company-mode/company-mode.git. Thus, the scenario changes 
from installing it from ELPA, to cloning the repo, adding the dir to 
`load-path' and `(require 'company)'.

>   . In any case, you cannot put the 'cursor' property on a newline and
>     hope it to work: the newline doesn't leave any glyphs on display,
>     certainly not when the cursor is displayed on the fringe.  So the
>     display engine doesn't know you've put the 'cursor' property there.

Which character should I put this property on in this situation, then?

The other odd thing is that the cursor is displayed in the margin 
opposite the last completion (there are 2 or 3 of them), but the overlay 
is actually of fixed size (10). So the cursor margin position doesn't 
correspond to the value of point, nor either of the overlay bounds.

>   . The overlay string generated by Company in this case is
>     problematic: it puts a newline at the end of each screen line, and
>     that removes the last character of each screen line from display.
>     I don't understand why you need to insert newlines when the
>     original text lines were one long continued line.  This is a bug in
>     Company.

It's a workaround I felt forced to take for the bug#18285. Since you've 
vetoed my suggestion to prioritize `invisible' over `display', the 
natural solution is to generally start the overlay a bit earlier.

It also simplifies some logic in the code, because sometimes we have to 
do this anyway (when the popup is displayed below the last line, and it 
has no trailing newline). Although this reason is less important.

> (Any
> idea why Company's post-command-hook calls sit-for, thus forcing
> redisplay?)

The current Company master has no `sit-for' calls, but both problems are 
still there.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/21/2015 04:30 PM, Dmitry Gutov wrote:

> The other odd thing is that the cursor is displayed in the margin
> opposite the last completion (there are 2 or 3 of them), but the overlay
> is actually of fixed size (10).

The overlay height, that is. 10 lines.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 21 Jun 2015 16:30:25 +0300
> 
> >   . There's no 'cursor' property on the overlay string that Company
> >     creates for its "tooltip" of completions.  Or at least I couldn't
> >     find that property: I tried both "M-x describe-text-properties" and
> >     looking at the overlay string in GDB -- I see no 'cursor' property,
> >     certainly not on the newline that starts the string.  If indeed
> >     there is such a property there, please tell how to see it.
> 
> I'm sorry, my memory of that code was unclear. `cursor' is currently 
> used only under certain rare condition, but see the end of 
> `company--replacement-string', where it's applied.
> 
> Even if remove the "when" condition to put it on all the time, the 
> behavior doesn't change.

If the 'cursor' property is put on the newline, then it indeed won't
change.  The support for 'cursor' assumes that the character on which
the property was put produces some glyph on the screen, because the
code which implements that works on screen lines (a.k.a. "glyph rows")
that were already laid out for display.  But the newline doesn't yield
any glyphs in this situation, so the fact that there was a 'cursor'
property there is left unknown to that code.

This cannot be fixed without a complete redesign of how 'cursor' is
supported.

> By the way, since we're now discussing the changes to Company code, 
> let's use the master version from 
> git@github.com:company-mode/company-mode.git. Thus, the scenario changes 
> from installing it from ELPA, to cloning the repo, adding the dir to 
> `load-path' and `(require 'company)'.

Ugh...  Why doesn't ELPA have the latest code?

> >   . In any case, you cannot put the 'cursor' property on a newline and
> >     hope it to work: the newline doesn't leave any glyphs on display,
> >     certainly not when the cursor is displayed on the fringe.  So the
> >     display engine doesn't know you've put the 'cursor' property there.
> 
> Which character should I put this property on in this situation, then?

The one before, I think.  Like I explained later:

> > E.g., perhaps
> > begin the overlay string a few characters earlier, so that it replaces
> > part of buffer text in "hel", and have the 'cursor' property on that
> > part of the string.

> The other odd thing is that the cursor is displayed in the margin 
> opposite the last completion (there are 2 or 3 of them), but the overlay 
> is actually of fixed size (10). So the cursor margin position doesn't 
> correspond to the value of point, nor either of the overlay bounds.

I didn't investigate why this happens.  Is it important?  Suppose I
"fixed" it by having the cursor on the same place in the second
paragraph where it ends up later -- will this be better in some sense?
If so, I will look into it.  (It's probably some redisplay
optimization that needs to be disabled in that case.)

> >   . The overlay string generated by Company in this case is
> >     problematic: it puts a newline at the end of each screen line, and
> >     that removes the last character of each screen line from display.
> >     I don't understand why you need to insert newlines when the
> >     original text lines were one long continued line.  This is a bug in
> >     Company.
> 
> It's a workaround I felt forced to take for the bug#18285. Since you've 
> vetoed my suggestion to prioritize `invisible' over `display', the 
> natural solution is to generally start the overlay a bit earlier.

Sorry, I'm not following, or maybe I don't understand the underlying
issues.  (I've re-read that bug, but I don't think it's relevant to
what I'm asking here.)  My question is about the newlines that get
inserted into the overlay string.  The original buffer text is one
long line, without any newlines, which occupies several screen lines.
Why cannot the overlay string be simply a copy of that text, a single
long string without any newlines?  The display engine will display
them with the continuation indicators on the fringes, exactly like it
does for buffer text, and the user will not "lose" those characters at
the end of each line.

> It also simplifies some logic in the code, because sometimes we have to 
> do this anyway (when the popup is displayed below the last line, and it 
> has no trailing newline).

You have "to do" what sometimes?

> > (Any
> > idea why Company's post-command-hook calls sit-for, thus forcing
> > redisplay?)
> 
> The current Company master has no `sit-for' calls, but both problems are 
> still there.

My last message exonerates these calls, they just got in the way when
I debugged the thing, and frankly surprised me, because having a call
to redisplay inside a post-command-hook that itself causes redisplay
by changing overlays sounds excessive.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/21/2015 07:24 PM, Eli Zaretskii wrote:

> Ugh...  Why doesn't ELPA have the latest code?

Because GitHub provides certain niceties as a project (and code) hosting 
plaform. This has been discussed before. Company is not the only project 
to host on GitHub and only push releases to ELPA.

>> Which character should I put this property on in this situation, then?
>
> The one before, I think.  Like I explained later:

I see. But that's going to look confusing: if the cursor appears 
earlier, the users are going to assume (at least from time to time) that 
they haven't typed the last character yet.

That kind of confusion isn't good for a code completion framework.

> I didn't investigate why this happens.  Is it important?  Suppose I
> "fixed" it by having the cursor on the same place in the second
> paragraph where it ends up later -- will this be better in some sense?
> If so, I will look into it.  (It's probably some redisplay
> optimization that needs to be disabled in that case.)

Not really important. I just thought it might be a detail that would 
lead you to a different explanation for this bug. Same for the extra 
questions in the following email.

> Sorry, I'm not following, or maybe I don't understand the underlying
> issues.  (I've re-read that bug, but I don't think it's relevant to
> what I'm asking here.)  My question is about the newlines that get
> inserted into the overlay string.

We can't really use a different rendering approach for every distinct 
layout of the buffer text: I'll go crazy.

Originally, the overlay (usually) started after the newline, and the 
popup was rendered using an overlay spanning the next 10 lines.

Now that we've found out that the `display' text property beginning 
right after that newline can really screw things (make the display value 
appear twice), the popup overlay begins one character earlier, and 
includes that character in its display string.

If that sounds like an overlay priority war, it pretty much is; except 
one can configure overlay priorities, but they can't do that for 
`invisible' vs `display'.

> The original buffer text is one
> long line, without any newlines, which occupies several screen lines.
> Why cannot the overlay string be simply a copy of that text, a single
> long string without any newlines?  The display engine will display
> them with the continuation indicators on the fringes, exactly like it
> does for buffer text, and the user will not "lose" those characters at
> the end of each line.

Can't say I've considered this before, this sounds more complex that 
what we have now, and depends on lines being wrapped in a certain way.

First, what if visual-line-mode is on? Then we'll have to track the 
presence of it and similar modes.

Second, the continuation characters will look out of place (and we'll 
have to have them for the next 10 lines, right?). Because normally you 
don't see them, but if the overlay rendering switches to wrapped lines 
instead of newlines, they will always be present (or rather blink in and 
out of existence as the popup is displayed or hidden).

>> It also simplifies some logic in the code, because sometimes we have to
>> do this anyway (when the popup is displayed below the last line, and it
>> has no trailing newline).
>
> You have "to do" what sometimes?

Start the popup overlay at the end of the preceding line, and include a 
newline in its display string (because there's no newline in the 
buffer). Admittedly, it's a recent development; previously, Company 
inserted a temporary newline, tracked it, and removed it after 
completion. And caught its share of bugs because of that.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 21 Jun 2015 20:46:59 +0300
> 
> >> Which character should I put this property on in this situation, then?
> >
> > The one before, I think.  Like I explained later:
> 
> I see. But that's going to look confusing: if the cursor appears 
> earlier, the users are going to assume (at least from time to time) that 
> they haven't typed the last character yet.

No, I expect the cursor to be drawn on the fringe, like the user
expects.  (Or maybe I misunderstand what you meant by "earlier".)  You
should give the 'cursor' property an integer value, so that it
"covers" the buffer position of the newline.

> > I didn't investigate why this happens.  Is it important?  Suppose I
> > "fixed" it by having the cursor on the same place in the second
> > paragraph where it ends up later -- will this be better in some sense?
> > If so, I will look into it.  (It's probably some redisplay
> > optimization that needs to be disabled in that case.)
> 
> Not really important. I just thought it might be a detail that would 
> lead you to a different explanation for this bug.

Unlikely.  I've clearly identified the code that moves point, as part
of redisplay, and traced it through the part that decides the cursor
cannot be set on the line with "hel".

> > The original buffer text is one
> > long line, without any newlines, which occupies several screen lines.
> > Why cannot the overlay string be simply a copy of that text, a single
> > long string without any newlines?  The display engine will display
> > them with the continuation indicators on the fringes, exactly like it
> > does for buffer text, and the user will not "lose" those characters at
> > the end of each line.
> 
> Can't say I've considered this before, this sounds more complex that 
> what we have now, and depends on lines being wrapped in a certain way.
> 
> First, what if visual-line-mode is on? Then we'll have to track the 
> presence of it and similar modes.

There are only 2 situations, from your POV: either a line goes all the
way to the window edge, or it doesn't.  The former can happen either
if there's a newline that ends a line, or if word-wrap is in effect.

> Second, the continuation characters will look out of place (and we'll 
> have to have them for the next 10 lines, right?). Because normally you 
> don't see them, but if the overlay rendering switches to wrapped lines 
> instead of newlines, they will always be present (or rather blink in and 
> out of existence as the popup is displayed or hidden).

Yes, there could be such artifacts, but IMO losing some of the text is
worse.  And the problems you describe will be only on the lines
actually used for the popup; the rest of the lines, which just copy
buffer text into an overlay string, will look exactly like they did
before, including the continuation markers.

Anyway, if you think what you have now is justified by the complexity
of the alternatives, fine with me.

> >> It also simplifies some logic in the code, because sometimes we have to
> >> do this anyway (when the popup is displayed below the last line, and it
> >> has no trailing newline).
> >
> > You have "to do" what sometimes?
> 
> Start the popup overlay at the end of the preceding line, and include a 
> newline in its display string (because there's no newline in the 
> buffer).

I had no problems with that newline.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/21/2015 09:09 PM, Eli Zaretskii wrote:

> No, I expect the cursor to be drawn on the fringe, like the user
> expects.  (Or maybe I misunderstand what you meant by "earlier".)  You
> should give the 'cursor' property an integer value, so that it
> "covers" the buffer position of the newline.

Ok, thanks. I forgot that `cursor' can have an integer value.

> There are only 2 situations, from your POV: either a line goes all the
> way to the window edge, or it doesn't.

Could you suggest the best way to check whether the former is the case?

(>= (car (posn-col-row (posn-at-point)))
     (window-width))

 >   The former can happen either if there's a newline that ends a line, 
 > or if word-wrap is in effect.

Does that constitute at least three different cases we'd need to handle?

- Not near the window edge
- near the edge and newline
- near the edge and no newline?

> Yes, there could be such artifacts, but IMO losing some of the text is
> worse.

I'm not sure what you mean by "losing some of the text".

> And the problems you describe will be only on the lines
> actually used for the popup; the rest of the lines, which just copy
> buffer text into an overlay string, will look exactly like they did
> before, including the continuation markers.

Err, why this distinction?

If anything, I guess we could "replace" only the first newline inside 
the overlay display string with wrapping, while keeping the rest as 
newlines, because they are irrelevant to this bug.

>> Start the popup overlay at the end of the preceding line, and include a
>> newline in its display string (because there's no newline in the
>> buffer).
>
> I had no problems with that newline.

Sure. In those circumstances, moving point to the end of the overlay 
would be a no-op anyway.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 21 Jun 2015 23:01:10 +0300
> 
> On 06/21/2015 09:09 PM, Eli Zaretskii wrote:
> 
> > There are only 2 situations, from your POV: either a line goes all the
> > way to the window edge, or it doesn't.
> 
> Could you suggest the best way to check whether the former is the case?
> 
> (>= (car (posn-col-row (posn-at-point)))
>      (window-width))

Something like that, but perhaps using end-of-visual-line as well.

>  >   The former can happen either if there's a newline that ends a line, 
>  > or if word-wrap is in effect.
> 
> Does that constitute at least three different cases we'd need to handle?
> 
> - Not near the window edge
> - near the edge and newline
> - near the edge and no newline?

I'm not sure I understand the meaning of "near" here.  Can you define
what "near the edge" means, in this context?  Does it mean "close", as
in "less than N columns away", or does it mean "flushed all the way to
the edge", i.e. not even a single column between the last character
and the window edge?

> > Yes, there could be such artifacts, but IMO losing some of the text is
> > worse.
> 
> I'm not sure what you mean by "losing some of the text".

The rightmost characters in each continued line are removed from
display because you put a newline there.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/22/2015 05:43 AM, Eli Zaretskii wrote:

> Something like that, but perhaps using end-of-visual-line as well.

Ok, so there's a function for that. Would (eq (point) (save-excursion 
(end-of-visual-line) (point))) not suffice?

>> Does that constitute at least three different cases we'd need to handle?
>>
>> - Not near the window edge
>> - near the edge and newline
>> - near the edge and no newline?
>
> I'm not sure I understand the meaning of "near" here.

Sorry for being unclear. Let's replace that word with "at".

>> I'm not sure what you mean by "losing some of the text".
>
> The rightmost characters in each continued line are removed from
> display because you put a newline there.

Do you have an example? Unless there's a misunderstanding, this would 
seem like a bug to me.

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Eli Zaretskii]

> Cc: 20847@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 22 Jun 2015 13:57:48 +0300
> 
> On 06/22/2015 05:43 AM, Eli Zaretskii wrote:
> 
> > Something like that, but perhaps using end-of-visual-line as well.
> 
> Ok, so there's a function for that. Would (eq (point) (save-excursion 
> (end-of-visual-line) (point))) not suffice?

To determine whether point is at the last position on that line?  Yes.

> >> Does that constitute at least three different cases we'd need to handle?
> >>
> >> - Not near the window edge
> >> - near the edge and newline
> >> - near the edge and no newline?
> >
> > I'm not sure I understand the meaning of "near" here.
> 
> Sorry for being unclear. Let's replace that word with "at".

I see.  Is this something you need to determine in addition to knowing
whether point is at the last position of a screen line?  If so, you
should keep in mind that the last position of a line could in some
rare cases be somewhat farther from the edge -- if the next "thing" to
display is wider than the space left till the edge.

> >> I'm not sure what you mean by "losing some of the text".
> >
> > The rightmost characters in each continued line are removed from
> > display because you put a newline there.
> 
> Do you have an example? Unless there's a misunderstanding, this would 
> seem like a bug to me.

The example I have is the exact recipe with which you started this bug
report.  As soon as you type "el" in step 6, and the "tooltip" with
completions pops up, look at the next paragraph of text below the
"tooltip" -- you will see the rightmost character on every line of
that paragraph disappear.  (For this to show, you need to have both
fringes, but "emacs -Q" should make sure you do.)

bug#20847: [display engine] 25.0.50; company-mode popup makes point jump to an entirely different location

[Dmitry Gutov]

On 06/22/2015 07:23 PM, Eli Zaretskii wrote:

> I see.  Is this something you need to determine in addition to knowing
> whether point is at the last position of a screen line?

I think so. It's only when point is at the last position, and when the 
line continues afterwards, and there's no truncation, or word-wrap, that 
I could omit the newline at the beginning of the completion overlay 
display string. And I've probably missed some possibility.

That's a lot of things to check.

> The example I have is the exact recipe with which you started this bug
> report.  As soon as you type "el" in step 6, and the "tooltip" with
> completions pops up, look at the next paragraph of text below the
> "tooltip" -- you will see the rightmost character on every line of
> that paragraph disappear.  (For this to show, you need to have both
> fringes, but "emacs -Q" should make sure you do.)

Thank you. I've never noticed it myself before. It was an implementation 
quirk, now fixed in master.