bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Alexander Miller]

As requested by Eli on reddit, a proper bug report about the company issue:

Using the new native line numbers in emacs -q
alongside company mode results in line numbers not being rendered in
lines where the company-mode popup is active. This is the same as in
(n)linum-mode, however the company popup, except the first line, is also
shifted to the left into the space line numbers used to occupy:
http://imgur.com/xHf2mlp

It looks like there's a general issue of overlay using packages not
expecting the space now being used by line numbers. For example the
fill-column-indicator package draws its indicator too far to the left
and swerves off with the comments at the top for some reason:
http://imgur.com/WHcrhdC
(Don't know why the company popup looks even worse here, that doesn't
happen when I load my full config).


In GNU Emacs 26.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.15)
of 2017-06-18 built on a-laptop
Repository revision: 7277c0fca7dab9f1b311c3eba5c42fd17acc3593
Windowing system distributor 'The X.Org Foundation

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> From: Alexander Miller <alexanderm@web.de>
> Date: Mon, 19 Jun 2017 18:50:18 +0200
> 
> As requested by Eli on reddit, a proper bug report about the company issue:

Thanks.

> Using the new native line numbers in emacs -q
> alongside company mode results in line numbers not being rendered in
> lines where the company-mode popup is active. This is the same as in
> (n)linum-mode, however the company popup, except the first line, is also
> shifted to the left into the space line numbers used to occupy:
> http://imgur.com/xHf2mlp

Can you show a minimal recipe, assuming company-mode was already
loaded, to reproduce the issue?  Such a recipe will make the job of
looking into the issue much easier, TIA.

> It looks like there's a general issue of overlay using packages not
> expecting the space now being used by line numbers. For example the
> fill-column-indicator package draws its indicator too far to the left
> and swerves off with the comments at the top for some reason:
> http://imgur.com/WHcrhdC
> (Don't know why the company popup looks even worse here, that doesn't
> happen when I load my full config).

It looks like some packages need to revisit their calculations of
horizontal coordinates.  I'm guessing they convert column numbers into
pixels, which will err when line numbers are displayed.  They should
instead use posn-at-point and the likes.  Per Martin's analysis, the
same problems should happen when the buffer has line-prefix defined;
could you try that?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

[Please keep the bug address on the CC line, so this is recorded by
the bug tracker.]

> From: Alexander Miller <alexanderm@web.de>
> Date: Mon, 19 Jun 2017 19:26:23 +0200
> 
> > Can you show a minimal recipe, assuming company-mode was already
> > loaded, to reproduce the issue?  Such a recipe will make the job of
> > looking into the issue much easier, TIA
> It's just the code you see in the pictures, starting from emacs -q.
> 
> Here it is again for easy copy-pasting:

Thanks, I will try to look into this.

> >   Per Martin's analysis, the
> > same problems should happen when the buffer has line-prefix defined;
> > could you try that?
> You're right, exactly the same thing happens.

OK, then may I suggest you raise this issue with the respective
package developers?  They could for now use line-prefix as trigger for
the problem; once they fix that, I think the problem with line numbers
will go away as well.  I will try to help them to pinpoint their
problem using the recipe you provided.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Alexander Miller]

> [Please keep the bug address on the CC line, so this is recorded by
> the bug tracker.]
Sorry about that. I pushed reply out of habit.

> OK, then may I suggest you raise this issue with the respective
> package developers?
Will do. I'll direct them to this ticket as well as the general discussion.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/19/17 8:08 PM, Eli Zaretskii wrote:

> It looks like some packages need to revisit their calculations of
> horizontal coordinates.  I'm guessing they convert column numbers into
> pixels, which will err when line numbers are displayed.  They should
> instead use posn-at-point and the likes.

Use it how?

With (setq line-prefix "..."),

(car (posn-col-row (posn-at-point))) suddenly returns 3 at bol.

> Per Martin's analysis, the
> same problems should happen when the buffer has line-prefix defined;
> could you try that?

Indeed, it does. But it's easy for company-mode to ignore the 
possibility of line-prefix variable being set (it almost never happens).

We do support line-prefix when it's assigned via text property.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 20 Jun 2017 00:03:08 +0300
> 
> With (setq line-prefix "..."),
> 
> (car (posn-col-row (posn-at-point))) suddenly returns 3 at bol.

I guess you think this result is incorrect, but I don't think I
understand why.  posn-col-row returns the horizontal window-relative
coordinate in units of the frame's canonical column width, so it is
expected that its value at BOL will be non-zero when there's a
line-prefix displayed there.

Can you tell how this value is used in company-mode?  I think the key
to my understanding why this change interferes with company-mode is in
that direction.

> > Per Martin's analysis, the
> > same problems should happen when the buffer has line-prefix defined;
> > could you try that?
> 
> Indeed, it does. But it's easy for company-mode to ignore the 
> possibility of line-prefix variable being set (it almost never happens).
> 
> We do support line-prefix when it's assigned via text property.

If you do support the line-prefix property, why are there problems
with line-prefix the variable?  The effect on display is the same,
AFAIU.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/20/17 6:04 PM, Eli Zaretskii wrote:

>> With (setq line-prefix "..."),
>>
>> (car (posn-col-row (posn-at-point))) suddenly returns 3 at bol.
> 
> I guess you think this result is incorrect, but I don't think I
> understand why.  posn-col-row returns the horizontal window-relative
> coordinate in units of the frame's canonical column width, so it is
> expected that its value at BOL will be non-zero when there's a
> line-prefix displayed there.

 From where I'm standing, it points to a design flaw in how line-prefix 
is implemented. Or at least it hints that the native line numbering 
shouldn't use it, and should use fringe or margin (like linum and nlinum 
do), or something like them that's considered to be outside of the 
window by the display engine.

> Can you tell how this value is used in company-mode?  I think the key
> to my understanding why this change interferes with company-mode is in
> that direction.

We've discussed it before, but here's a reminder.

Company draws popup by taking a number of lines that the popup would 
cover, collecting them into strings, modifying those strings to "draw" 
the popup on top of them. These operations use the "current row" and 
"current column" results, respectively.

Then, it (without going into nuance) puts an overlay over the said 
buffer lines, makes it `invisible', and puts a strings consisting of 
concatenation of those modified lines on the overlay's `display' property.

When the position at bol is interpreted as "column 3", all lines of the 
rectangle are rendered starting with the fourth character of each line. 
Somehow, the result turns out to be uneven, and the first line of the 
overlay doesn't get affected by line-prefix (so the popup there ends up 
at the proper horizontal position), but the rest are affected, and 
they're displaced three columns to the right.

>> We do support line-prefix when it's assigned via text property.
> 
> If you do support the line-prefix property, why are there problems
> with line-prefix the variable?  The effect on display is the same,
> AFAIU.

The way we support the property is impossible to translate to the 
variable. For each line, we take the value of `line-prefix', prepend it 
to the line text, and the popup overlay (the one that covers all of the 
affected lines) gets the `line-prefix' property set to "".

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 21 Jun 2017 05:18:38 +0300
> 
> When the position at bol is interpreted as "column 3", all lines of the 
> rectangle are rendered starting with the fourth character of each line. 

So this is the root cause of the problem.  Is there a reason not to
start rendering from the character whose posn-col-row is 3?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/21/17 5:40 AM, Eli Zaretskii wrote:

>> When the position at bol is interpreted as "column 3", all lines of the
>> rectangle are rendered starting with the fourth character of each line.
> 
> So this is the root cause of the problem.  Is there a reason not to
> start rendering from the character whose posn-col-row is 3?

How?

Does that mean I'll have to call posn-at-point twice now? Once for 
point, and once for beginning-of-visual-line?

Even that won't solve all the rendering issues. In my testing, the first 
line of the popup is not at the same column as the rest of them. 
Probably because of the `line-prefix' property on the overlay which I 
explained in the previous email. And we can't stop putting it there 
because we don't have a better solution for the `line-prefix' text property.

I _guess_ I could reorganize the code and track whether the current line 
is the first, and render that line of the popup with a different 
offset... Doesn't sound great, to be honest. That's an extra piece of 
complexity.

Even so, I'm not sure this will be the end of it. Setting line-prefix 
doesn't seem to be an accurate model of the problem company popup has 
with native line numbering:

- The image in the report shows the first line of the popup being 
"rendered" at a higher column than the rest. The rest are basically 
positioned fine.

- In my testing, with (setq line-prefix "..."), it's the opposite. The 
first line of the popup is positioned correctly, while the rest are 
shifted to the right by 3 columns.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Alexander Miller]

> In my testing, with (setq line-prefix "..."), it's the opposite. The 
> first line of the popup is positioned correctly, while the rest are 
> shifted to the right by 3 columns.
You're right about that, it's the same, but opposite problem. I tried 
again with line-numbers and/or line-prefix and the popup is indeed 
shifted around in different directions in both cases. Interestingly 
enough it's possible for the effects of both settings to cancel each 
other out when they take the same amount of space. For me it for example 
happened with (setq display-line-width 5 line-prefix ".......").

Link to pictures: http://imgur.com/a/9QRQs

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 21 Jun 2017 16:04:58 +0300
> 
> On 6/21/17 5:40 AM, Eli Zaretskii wrote:
> 
> >> When the position at bol is interpreted as "column 3", all lines of the
> >> rectangle are rendered starting with the fourth character of each line.
> > 
> > So this is the root cause of the problem.  Is there a reason not to
> > start rendering from the character whose posn-col-row is 3?
> 
> How?
> 
> Does that mean I'll have to call posn-at-point twice now? Once for 
> point, and once for beginning-of-visual-line?

I guess so.  I understand that now you call current-column or
something similar at BOL?  Or do you just assume the column there is
zero?

> Even that won't solve all the rendering issues. In my testing, the first 
> line of the popup is not at the same column as the rest of them. 
> Probably because of the `line-prefix' property on the overlay which I 
> explained in the previous email. And we can't stop putting it there 
> because we don't have a better solution for the `line-prefix' text property.

What I had in mind is to come up with a solution that will work the
same with line-prefix specified in any way we support.  Then you won't
need to put the line-prefix property on the company overlay.

Btw, can you remind me why we don't use pop-up menus for company
popups?

> Even so, I'm not sure this will be the end of it. Setting line-prefix 
> doesn't seem to be an accurate model of the problem company popup has 
> with native line numbering:

Could be.  Can I seduce you to try the line-numbers branch? ;-)

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/21/17 9:15 PM, Eli Zaretskii wrote:
> Or do you just assume the column there is
> zero?

Yep! And there are zero outstanding bug reports related to this.

> What I had in mind is to come up with a solution that will work the
> same with line-prefix specified in any way we support.  Then you won't
> need to put the line-prefix property on the company overlay.

I'm saying it's not easy, and I'm not brimming with ideas.

Can't we put native line numbering outside of the window bounds? Like 
linum and nlinum do.

> Btw, can you remind me why we don't use pop-up menus for company
> popups?

Event handing and theming issues (at least with GUI frames).

- GUI menus have only two colors: background and foreground (and maybe 
"inactive"). Not sure about the terminal mode menus.

- Menus take over the event loop. IIRC you suggested a way to get around 
this, but that doesn't solve the previous problem anyway.

IIUC we've settled on using chromeless frames for the popup. It seems 
Martin is cooking something in this direction (almost ready?), but I 
haven't tried using them. And that would take some work.

Last I checked, Clément was interested, but no results yet.

> Could be.  Can I seduce you to try the line-numbers branch? ;-)

Consider me seduced. I'm seeing the same behavior as what Alexander 
reported. There are two problems:

1. The first visual line containing the popup has the line number at its 
beginning. And as such, the popup line is shifted to the right.

2. The rest don't have the line numbers before them, so they are 
positioned correctly. This may or may not be considered a problem 
(there's probably nothing we can do about the lack of numbers), but the 
inconsistency between the different popup lines seems like it would 
require some special handling in the code.

And whatever that additional code would do, it would have to be 
reconciled with line-prefix as well, so as not to make things worse.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 22 Jun 2017 01:41:15 +0300
> 
> On 6/21/17 9:15 PM, Eli Zaretskii wrote:
> > Or do you just assume the column there is
> > zero?
> 
> Yep! And there are zero outstanding bug reports related to this.

Well, except this one ;-)

> > What I had in mind is to come up with a solution that will work the
> > same with line-prefix specified in any way we support.  Then you won't
> > need to put the line-prefix property on the company overlay.
> 
> I'm saying it's not easy, and I'm not brimming with ideas.

Is it not easy because the assumption about column-zero is hard-coded
in many places?  Or for some other reason?

> Can't we put native line numbering outside of the window bounds? Like 
> linum and nlinum do.

Keeping the numbers out of the margins was my explicit design goal,
because some packages want the margins, and we don't have a good
solution for "sharing" margins.  So from my POV putting the numbers in
the margins would be a step backward.  It will probably also create
major havoc for the few packages that do display in the margins,
because Emacs facilities for layout of text and other stuff there are
exceedingly limited.

> IIUC we've settled on using chromeless frames for the popup. It seems 
> Martin is cooking something in this direction (almost ready?), but I 
> haven't tried using them. And that would take some work.

Not sure what you mean by "chromeless" here, but if I understand you
correctly, Martin's work is already on master.

> 1. The first visual line containing the popup has the line number at its 
> beginning. And as such, the popup line is shifted to the right.

That's the "BOL at non-zero column" issue, right?

> 2. The rest don't have the line numbers before them, so they are 
> positioned correctly. This may or may not be considered a problem 
> (there's probably nothing we can do about the lack of numbers), but the 
> inconsistency between the different popup lines seems like it would 
> require some special handling in the code.

The lack of numbers is a "feature", I think: only "physical" lines in
buffer text are counted, newlines in overlay and display strings do
not count.

Can you artificially offset the beginning of the overlay to account
for the line numbers, and see if that alone solves the problem of the
first lines, and doesn't cause problems in the subsequent lines?

Thanks.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/22/17 5:55 PM, Eli Zaretskii wrote:

>> Yep! And there are zero outstanding bug reports related to this.
> 
> Well, except this one ;-)

But it's not about line-prefix, it's about a new feature in Emacs that 
was just introduced in a way that's different from those that came before.

>> I'm saying it's not easy, and I'm not brimming with ideas.
> 
> Is it not easy because the assumption about column-zero is hard-coded
> in many places?  Or for some other reason?

Because if we're not allowed to use column-zero, the first line of the 
popup and the rest are behaving differently. And, depending on which 
feature is causing the first character of the visual line to be on a 
non-zero column, the popup lines are offset in different directions.

> Keeping the numbers out of the margins was my explicit design goal,
> because some packages want the margins, and we don't have a good
> solution for "sharing" margins.

It's not without problems, though, as it introduces a new, different 
type of margin, doesn't it? Why not put it outside of the window bounds 
next to the "actual" margins? That's one option.

> So from my POV putting the numbers in
> the margins would be a step backward.  It will probably also create
> major havoc for the few packages that do display in the margins,

Not any more havoc than linum or nlinum which many people install and 
use, and are apparently fine with them taking up one of the margins.

> because Emacs facilities for layout of text and other stuff there are
> exceedingly limited.

But of course, it would be ideal if you could also introduce a facility 
that would allow sharing of margins (and hopefully also fringes) between 
different modes.

> Not sure what you mean by "chromeless" here, but if I understand you
> correctly, Martin's work is already on master.

I think so, yes.

>> 1. The first visual line containing the popup has the line number at its
>> beginning. And as such, the popup line is shifted to the right.
> 
> That's the "BOL at non-zero column" issue, right?

Yes, but also the lack of ability to disable the line numbering on a 
line-by-line basis (which could be one solution for this problem).

> The lack of numbers is a "feature", I think: only "physical" lines in
> buffer text are counted, newlines in overlay and display strings do
> not count.

It is indeed a feature from your standpoint, but it's a bug from the 
point of view of the popup's rendering.

Imagine of the line numbers were displayed using line-prefix (I'm not 
really suggesting that, but...). Then the popup could pick them up and 
include in the overlay's `display' property text. The user wouldn't see 
any difference.

> Can you artificially offset the beginning of the overlay to account
> for the line numbers, and see if that alone solves the problem of the
> first lines, and doesn't cause problems in the subsequent lines?

Offset the beginning by how much? By

(- (car (posn-col-row (posn-at-point (point))))
    (car (posn-col-row (posn-at-point (beginning-of-visual-line)))))

? Let's call the value of this expression X.

Let's try a thought experiment.

Suppose X is non-zero because of line numbering. Then adding an offset 
of X to the position of the first popup line should fix it.

What if X is non-zero because of line-prefix, though? If it's because of 
line-prefix variable, okay, we don't support it.

What if X is non-zero because of the line-prefix text property? We'd get 
an "array out of bounds" error somewhere, because the first popup line 
is currently positioned correctly.

So when should we add an X offset to the first popup line's position?

Suppose X is non-nil because of line numbering and the line-prefix text 
property both? Org uses the line-prefix text property in certain 
configurations, so it's a real possibility.

One idea would be to first create an overlay on the first visual line 
which overrides the line-prefix property to zero, *then* measure

(car (posn-col-row (posn-at-point (beginning-of-visual-line))))

, then delete the said overlay and proceed. Doesn't that sound like a 
mess already?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 23 Jun 2017 02:17:24 +0300
> 
> On 6/22/17 5:55 PM, Eli Zaretskii wrote:
> 
> >> Yep! And there are zero outstanding bug reports related to this.
> > 
> > Well, except this one ;-)
> 
> But it's not about line-prefix, it's about a new feature in Emacs that 
> was just introduced in a way that's different from those that came before.

Sorry, I thought that by "related to this" you alluded to the
zero-column-at-BOL assumption.

> > Keeping the numbers out of the margins was my explicit design goal,
> > because some packages want the margins, and we don't have a good
> > solution for "sharing" margins.
> 
> It's not without problems, though, as it introduces a new, different 
> type of margin, doesn't it?

Not really.  The display engine was always inserting stuff at the
beginning of a visual line.  Some features that use this include
line-prefix and wrap-prefix, the left truncation glyphs, and
overlay-arrow.  It's just that AFAIU company-mode has coded
workarounds for each one of them, and now there's one more feature to
deal with.

> Why not put it outside of the window bounds next to the "actual"
> margins? That's one option.

That option requires a thorough redesign of the canvas geometry used
by the display engine: currently there's no area for it to put glyphs
except the 3 existing ones -- the two margins and the text area.

> > So from my POV putting the numbers in
> > the margins would be a step backward.  It will probably also create
> > major havoc for the few packages that do display in the margins,
> 
> Not any more havoc than linum or nlinum which many people install and 
> use, and are apparently fine with them taking up one of the margins.

As long as only one mode uses the margins, no problems indeed.  Once
there are two or more, they need to work around one another, and that
doesn't work very well, sometimes not at all.

> But of course, it would be ideal if you could also introduce a facility 
> that would allow sharing of margins (and hopefully also fringes) between 
> different modes.

Alas, we don't have any consensus for how should this be done.

> > Not sure what you mean by "chromeless" here, but if I understand you
> > correctly, Martin's work is already on master.
> 
> I think so, yes.

I'd urge you to see if company-mode can switch to using these
facilities, as working against the display engine works only up to a
point, and doubtless causes many maintenance headaches.

> >> 1. The first visual line containing the popup has the line number at its
> >> beginning. And as such, the popup line is shifted to the right.
> > 
> > That's the "BOL at non-zero column" issue, right?
> 
> Yes, but also the lack of ability to disable the line numbering on a 
> line-by-line basis (which could be one solution for this problem).

If providing such a facility would solve the problem, it should be
easy to code.  E.g., we could have a display-line-numbers-disable
property which a Lisp program could put on the first visible character
of a visual line, and a non-nil value of that property would signal to
the display engine not to produce the line-number glyphs for that
visual line.  Would that be okay?  (Note that line-number display
produces glyphs even for lines for which no number should be
displayed, such as continuation lines and empty screen lines beyond
EOB, so company-mode would need to decide whether it needs to disable
those as well.  Which is why I wrote "the first visible character of a
visual line" above, emphasis on "visual".)

> Imagine of the line numbers were displayed using line-prefix (I'm not 
> really suggesting that, but...). Then the popup could pick them up and 
> include in the overlay's `display' property text. The user wouldn't see 
> any difference.

You can do similar things with line numbers: Lisp code can count lines
as well as C code, and the fact that the line numbers are being
displayed is easily detected from Lisp.

> Suppose X is non-zero because of line numbering. Then adding an offset 
> of X to the position of the first popup line should fix it.
> 
> What if X is non-zero because of line-prefix, though? If it's because of 
> line-prefix variable, okay, we don't support it.
> 
> What if X is non-zero because of the line-prefix text property? We'd get 
> an "array out of bounds" error somewhere, because the first popup line 
> is currently positioned correctly.
> 
> So when should we add an X offset to the first popup line's position?
> 
> Suppose X is non-nil because of line numbering and the line-prefix text 
> property both? Org uses the line-prefix text property in certain 
> configurations, so it's a real possibility.
> 
> One idea would be to first create an overlay on the first visual line 
> which overrides the line-prefix property to zero, *then* measure
> 
> (car (posn-col-row (posn-at-point (beginning-of-visual-line))))
> 
> , then delete the said overlay and proceed. Doesn't that sound like a 
> mess already?

(And you didn't yet consider the possibility of _both_ line numbers
and line-prefix.  Yes, that should be possible, and I intended that to
be supported.)

Anyway, the mess should be expected in any feature which attempts to
change the display in significant ways, like company-mode does.
Experience shows that this only works up to a point.  Which is why The
Right Way to implement such features is to introduce infrastructure
(on the C level) which will allow the display engine to do the layout
job as those features need.  Native display of line numbers is one
step in that direction, from my POV.  I hope that Martin's work on
child frames will allow company-mode to go in a similar direction as
well.

However, this bug report is about letting the current implementation
of company-mode live in peace with native display of line numbers.  So
if you tell me that the above-mentioned property will allow such a
coexistence, I will code it.

Thanks.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/23/17 12:10 PM, Eli Zaretskii wrote:

> Sorry, I thought that by "related to this" you alluded to the
> zero-column-at-BOL assumption.

I just meant that until now this code coped pretty well with what can be 
found in practice. There are various issues, but none of them were 
zero-column related.

> Not really.  The display engine was always inserting stuff at the
> beginning of a visual line.  Some features that use this include
> line-prefix and wrap-prefix, the left truncation glyphs, and
> overlay-arrow.  It's just that AFAIU company-mode has coded
> workarounds for each one of them, and now there's one more feature to
> deal with.

I don't remember (or see in the code) any workarounds for any of these, 
aside from line-prefix text property. Maybe we're missing certain 
problem situations, of course (examples welcome). Do most of these only 
appear in non-graphical mode?

>> Why not put it outside of the window bounds next to the "actual"
>> margins? That's one option.
> 
> That option requires a thorough redesign of the canvas geometry used
> by the display engine: currently there's no area for it to put glyphs
> except the 3 existing ones -- the two margins and the text area.

In that case, I think we should try harder to make use of the margins.

> As long as only one mode uses the margins, no problems indeed.  Once
> there are two or more, they need to work around one another, and that
> doesn't work very well, sometimes not at all.

Violent agreement, isn't it?

Again, it's a known problem. You're trying to work around it in a way 
that doesn't scale to any potential similar new features, and which 
breaks company-mode popup (and probably some other overlay-wrangling 
code as well, I'm guessing).

I'm saying the users seem willing to wait for a scalable solution, and 
use a margin in the meantime. After all, we have two margins, and two 
fringes as well.

>> But of course, it would be ideal if you could also introduce a facility
>> that would allow sharing of margins (and hopefully also fringes) between
>> different modes.
> 
> Alas, we don't have any consensus for how should this be done.

I can try to outline a possible API. Do we have an existing bug report 
where that discussion would be better placed?

If you have a link to a previous discussion, that would be great as well.

>>> Not sure what you mean by "chromeless" here, but if I understand you
>>> correctly, Martin's work is already on master.
>>
>> I think so, yes.
> 
> I'd urge you to see if company-mode can switch to using these
> facilities, as working against the display engine works only up to a
> point, and doubtless causes many maintenance headaches.

It sounds like a fair amount of work, so it's low on my list. Improving 
ruby-mode and xref seems more important.

>> Yes, but also the lack of ability to disable the line numbering on a
>> line-by-line basis (which could be one solution for this problem).
> 
> If providing such a facility would solve the problem, it should be
> easy to code.  E.g., we could have a display-line-numbers-disable
> property which a Lisp program could put on the first visible character
> of a visual line, and a non-nil value of that property would signal to
> the display engine not to produce the line-number glyphs for that
> visual line.  Would that be okay?

Should be fine, if we find no better choice. As long as that property 
can be set via an overlay.

> (Note that line-number display
> produces glyphs even for lines for which no number should be
> displayed, such as continuation lines and empty screen lines beyond
> EOB, so company-mode would need to decide whether it needs to disable
> those as well.  Which is why I wrote "the first visible character of a
> visual line" above, emphasis on "visual".)

Visual line sounds right.

>> Imagine of the line numbers were displayed using line-prefix (I'm not
>> really suggesting that, but...). Then the popup could pick them up and
>> include in the overlay's `display' property text. The user wouldn't see
>> any difference.
> 
> You can do similar things with line numbers: Lisp code can count lines
> as well as C code, and the fact that the line numbers are being
> displayed is easily detected from Lisp.

I can't get to the actual numbers, though. And isn't counting lines in 
Lisp considered too slow? Or why else would we have this new feature?

> (And you didn't yet consider the possibility of _both_ line numbers
> and line-prefix.  Yes, that should be possible, and I intended that to
> be supported.)

I did, it's in the middle of your quote. ;-)

> Anyway, the mess should be expected in any feature which attempts to
> change the display in significant ways, like company-mode does.
> Experience shows that this only works up to a point.  Which is why The
> Right Way to implement such features is to introduce infrastructure
> (on the C level) which will allow the display engine to do the layout
> job as those features need.  Native display of line numbers is one
> step in that direction, from my POV.  I hope that Martin's work on
> child frames will allow company-mode to go in a similar direction as
> well.

Agreed that it's the right direction (although I wouldn't say no to a 
popup library in the core either ;-). Would it help in non-graphical 
mode, though? Does it support non-maximized frames?

> However, this bug report is about letting the current implementation
> of company-mode live in peace with native display of line numbers.  So
> if you tell me that the above-mentioned property will allow such a
> coexistence, I will code it.

Let's try it, if it's not hard to add. I'll report back as soon as it's 
available.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 24 Jun 2017 02:26:23 +0300
> 
> > Not really.  The display engine was always inserting stuff at the
> > beginning of a visual line.  Some features that use this include
> > line-prefix and wrap-prefix, the left truncation glyphs, and
> > overlay-arrow.  It's just that AFAIU company-mode has coded
> > workarounds for each one of them, and now there's one more feature to
> > deal with.
> 
> I don't remember (or see in the code) any workarounds for any of these, 
> aside from line-prefix text property. Maybe we're missing certain 
> problem situations, of course (examples welcome). Do most of these only 
> appear in non-graphical mode?

Not necessarily.  E.g., disabling the fringes will cause Emacs to
display left-truncation glyphs on GUI frames as well.

I guess you were just lucky, because in all of the other cases the
additional horizontal shift was somehow either accounted for or (in
the case of overlay-arrow) non-existent.  But the principle still
stands: the display engine is allowed to insert glyphs it invents out
of thin air at the edges of the text area, and Emacs always made use
of this.

> Again, [display in margins is] a known problem. You're trying to
> work around it in a way that doesn't scale to any potential similar
> new features, and which breaks company-mode popup (and probably some
> other overlay-wrangling code as well, I'm guessing).

It will only break modes that assume too much about layout.  And given
the popularity of the line numbers, there's no other practical way
except adapt those add-on packages, starting from company-mode.

> I'm saying the users seem willing to wait for a scalable solution, and 
> use a margin in the meantime.

I think you have it backwards: the margin-based solutions were tried
and were found not to be scalable.

> >> But of course, it would be ideal if you could also introduce a facility
> >> that would allow sharing of margins (and hopefully also fringes) between
> >> different modes.
> > 
> > Alas, we don't have any consensus for how should this be done.
> 
> I can try to outline a possible API. Do we have an existing bug report 
> where that discussion would be better placed?

Bug#24193.

> If you have a link to a previous discussion, that would be great as well.

Here's one (I think there were more):

  http://lists.gnu.org/archive/html/emacs-devel/2015-11/msg01171.html

FWIW, I don't recommend going in that direction, as I believe we will
again bump into the same basic disagreements due to conflicting goals.
That's why I decided to stay away of the margins in the first place.

> >> Yes, but also the lack of ability to disable the line numbering on a
> >> line-by-line basis (which could be one solution for this problem).
> > 
> > If providing such a facility would solve the problem, it should be
> > easy to code.  E.g., we could have a display-line-numbers-disable
> > property which a Lisp program could put on the first visible character
> > of a visual line, and a non-nil value of that property would signal to
> > the display engine not to produce the line-number glyphs for that
> > visual line.  Would that be okay?
> 
> Should be fine, if we find no better choice. As long as that property 
> can be set via an overlay.

OK, I will add that.

> >> Imagine of the line numbers were displayed using line-prefix (I'm not
> >> really suggesting that, but...). Then the popup could pick them up and
> >> include in the overlay's `display' property text. The user wouldn't see
> >> any difference.
> > 
> > You can do similar things with line numbers: Lisp code can count lines
> > as well as C code, and the fact that the line numbers are being
> > displayed is easily detected from Lisp.
> 
> I can't get to the actual numbers, though.

count-lines should give them to you, no?

> And isn't counting lines in Lisp considered too slow?

When done as part of routine redisplay, off the post-command-hook,
yes.  But company-mode is not used during routine redisplay, as in
during scrolling through the window, right?

> Or why else would we have this new feature?

Producing line numbers as part of redisplay was indeed intended to
make it faster, since all the line-numbering add-ons basically induce
an immediate additional redisplay cycle, because they change
overlays.  But that was only one of my goals, the other was to do in
the infrastructure something that can be done well only there.

> > I hope that Martin's work on child frames will allow company-mode
> > to go in a similar direction as well.
> 
> Agreed that it's the right direction (although I wouldn't say no to a 
> popup library in the core either ;-). Would it help in non-graphical 
> mode, though? Does it support non-maximized frames?

AFAIU, it supports any kind of frame.

> > However, this bug report is about letting the current implementation
> > of company-mode live in peace with native display of line numbers.  So
> > if you tell me that the above-mentioned property will allow such a
> > coexistence, I will code it.
> 
> Let's try it, if it's not hard to add. I'll report back as soon as it's 
> available.

Thanks.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Sat, 24 Jun 2017 10:47:28 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> 
> > > If providing such a facility would solve the problem, it should be
> > > easy to code.  E.g., we could have a display-line-numbers-disable
> > > property which a Lisp program could put on the first visible character
> > > of a visual line, and a non-nil value of that property would signal to
> > > the display engine not to produce the line-number glyphs for that
> > > visual line.  Would that be okay?
> > 
> > Should be fine, if we find no better choice. As long as that property 
> > can be set via an overlay.
> 
> OK, I will add that.

Now done.  Please see if this allows company-mode to fix its display
when line numbers are displayed.

Thanks.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/24/17 8:28 PM, Eli Zaretskii wrote:

> Now done.  Please see if this allows company-mode to fix its display
> when line numbers are displayed.

Seems to work well locally, aside from the EOB case (*). Will commit 
after you push to master.

BTW, without this property, I now see line numbers beside all the visual 
lines the popup overlay takes up, and the number is the same: N+1, where 
N is the line-at-point. That doesn't look intended.

(*) The case is where the overlay is shown below the last line of the 
buffer. In that case, we display the popup using the `after-string' 
property. The after-EOB glyphs don't seem to be affected by 
`display-line-numbers-disable'.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 304 bytes --]

On 6/24/17 11:43 PM, Dmitry Gutov wrote:

> Seems to work well locally, aside from the EOB case (*). Will commit 
> after you push to master.

A couple of screenshots attached, to illustrate.

The EOB case is a problem, but it seems less grating than before (since 
the popup keeps its shape, at least).

[-- Attachment #2: Screenshot from 2017-06-25 16-47-26.png --]
[-- Type: image/png, Size: 32843 bytes --]

[-- Attachment #3: Screenshot from 2017-06-25 16-47-44.png --]
[-- Type: image/png, Size: 25560 bytes --]

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 24 Jun 2017 23:43:25 +0300
> 
> On 6/24/17 8:28 PM, Eli Zaretskii wrote:
> 
> > Now done.  Please see if this allows company-mode to fix its display
> > when line numbers are displayed.
> 
> Seems to work well locally, aside from the EOB case (*). Will commit 
> after you push to master.

Great, thanks for testing.

> BTW, without this property, I now see line numbers beside all the visual 
> lines the popup overlay takes up, and the number is the same: N+1, where 
> N is the line-at-point. That doesn't look intended.

The displayed line number reflects the line of the buffer positions
corresponding to what's on that screen line.  If none of the buffer
positions appear on that screen line, it's the line of the buffer
position(s) "covered" by the display string/overlay which generates
the display.

If what you see doesn't fit this description, please show a
screenshot, and describe or show the code which puts the overlay that
causes the display.

> (*) The case is where the overlay is shown below the last line of the 
> buffer. In that case, we display the popup using the `after-string' 
> property. The after-EOB glyphs don't seem to be affected by 
> `display-line-numbers-disable'.

I'm not sure I understand: are you saying that you've put the property
in that case and it didn't have the expected effect?  Or are you
saying that you don't have a position to put the property in that
case?  If the former, can you tell on which buffer position you put
the property, and perhaps show a simple reproducer?

Thanks.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/24/17 10:47 AM, Eli Zaretskii wrote:

> I guess you were just lucky, because in all of the other cases the
> additional horizontal shift was somehow either accounted for or (in
> the case of overlay-arrow) non-existent.  But the principle still
> stands: the display engine is allowed to insert glyphs it invents out
> of thin air at the edges of the text area, and Emacs always made use
> of this.

Try as I might, I can't repro those kind of problems. Line wrap 
indicators in the terminal usually come at the right. And I think it's 
aesthetically important that the extra glyphs don't shift line text to 
the right unless absolutely necessary, so those issues don't materialize.

The choice to use one overlay instead of one-overlay-per-line also makes 
some problems easier. Anyway, naturally, a proper popup will be better.

> It will only break modes that assume too much about layout.  And given
> the popularity of the line numbers, there's no other practical way
> except adapt those add-on packages, starting from company-mode.

I think you're trying too hard to stay away from the margins. But 
anyway, we have the alternative solution now.

>> I'm saying the users seem willing to wait for a scalable solution, and
>> use a margin in the meantime.
> 
> I think you have it backwards: the margin-based solutions were tried
> and were found not to be scalable.

I don't see you contradicting me. The _current_ margin-based solutions 
are not scalable in terms of margin-using features, but they can be, 
when we work on that. And the users seem willing to wait for that.

>> I can try to outline a possible API. Do we have an existing bug report
>> where that discussion would be better placed?
> 
> Bug#24193.
> 
>> If you have a link to a previous discussion, that would be great as well.
> 
> Here's one (I think there were more):
> 
>    http://lists.gnu.org/archive/html/emacs-devel/2015-11/msg01171.html
> 
> FWIW, I don't recommend going in that direction, as I believe we will
> again bump into the same basic disagreements due to conflicting goals.
> That's why I decided to stay away of the margins in the first place.

Thanks! I think I have something to contribute. But TBH the use case in 
the emacs-devel thread is ridiculous. We should be able to provide the 
functionality offered in the API of common IDEs, and if that doesn't let 
someone to use the margins as the fill-column, too bad.

It still might, but we can't consider all use cases equally valid.

>> And isn't counting lines in Lisp considered too slow?
> 
> When done as part of routine redisplay, off the post-command-hook,
> yes.  But company-mode is not used during routine redisplay, as in
> during scrolling through the window, right?

It's using post-command-hook, though. Redoing the line numbers rendering 
is not something I'm looking forward to either.

> But that was only one of my goals, the other was to do in
> the infrastructure something that can be done well only there.

Meaning, to use something other than the margins for display?

>> Agreed that it's the right direction (although I wouldn't say no to a
>> popup library in the core either ;-). Would it help in non-graphical
>> mode, though? Does it support non-maximized frames?
> 
> AFAIU, it supports any kind of frame.

I can't see a way to create a less-than-fullscreen frame in terminal 
Emacs (speaking of normal frames here).

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> Date: Sun, 25 Jun 2017 16:51:07 +0300
> 
> A couple of screenshots attached, to illustrate.

Thanks, but as I don't use company-mode, I need explanations to go
with the screenshots ;-)

First, is the display OK from your POV?  If not, what are the problems
you'd prefer to solve?

Next, do those empty numbered lines in the first screenshots indicate
a problem?  Or are there indeed empty lines in the buffer?

Finally, the EOB case is not in these screenshots, is it?  Or is the
last one it?  If the latter, what problems do you see in that
screenshot?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/25/17 5:32 PM, Eli Zaretskii wrote:

> Thanks, but as I don't use company-mode, I need explanations to go
> with the screenshots ;-)

OK.

> First, is the display OK from your POV?  If not, what are the problems
> you'd prefer to solve?

The second screenshot shows the popup rendered at EOB, and it's rendered 
3 columns too far to the right than expected.

> Next, do those empty numbered lines in the first screenshots indicate
> a problem?

Like I described previously, the numbering doesn't know what to do with 
a part of the buffer covered by a multiline `display' string.

> Or are there indeed empty lines in the buffer?

Most lines in this buffer are empty, that doesn't seem to affect the 
presence of the numbers, except for the last line of the buffer.

> Finally, the EOB case is not in these screenshots, is it?  Or is the
> last one it?  If the latter, what problems do you see in that
> screenshot?
The last one, yes. The popup is 3 columns too far to the right.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/25/17 5:13 PM, Eli Zaretskii wrote:

> The displayed line number reflects the line of the buffer positions
> corresponding to what's on that screen line.  If none of the buffer
> positions appear on that screen line, it's the line of the buffer
> position(s) "covered" by the display string/overlay which generates
> the display.

It didn't look like this in the previous version. Might be considered a 
bit annoying, repeating the number several times.

BTW, by that logic, the last empty line should have a number as well: it 
does correspond to a buffer position.

> If what you see doesn't fit this description, please show a
> screenshot, and describe or show the code which puts the overlay that
> causes the display.

It does fit the description.

> I'm not sure I understand: are you saying that you've put the property
> in that case and it didn't have the expected effect?

Yes. Although I half expected it to have no effect in this case.

> Or are you
> saying that you don't have a position to put the property in that
> case? 

Also true, probably.

> If the former, can you tell on which buffer position you put
> the property, and perhaps show a simple reproducer?

(with-current-buffer (get-buffer-create "popup-test.el")
   (setq display-line-numbers t)
   (insert "aaaaaaa
aaaaaaa
aaaaaaa
aaaaaaa
aaaaaaa
")

   (let ((ov (make-overlay (point-max) (point-max))))
     (overlay-put ov 'after-string "bbbbbb\nbbbbbb\n")
     (overlay-put ov 'display-line-numbers-disable t)))

After that, the buffer popup-test.el shows the "bbbbbb" lines prepended 
with the empty line number columns. I'd rather they weren't there.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 25 Jun 2017 17:31:59 +0300
> 
> On 6/24/17 10:47 AM, Eli Zaretskii wrote:
> 
> > I guess you were just lucky, because in all of the other cases the
> > additional horizontal shift was somehow either accounted for or (in
> > the case of overlay-arrow) non-existent.  But the principle still
> > stands: the display engine is allowed to insert glyphs it invents out
> > of thin air at the edges of the text area, and Emacs always made use
> > of this.
> 
> Try as I might, I can't repro those kind of problems. Line wrap 
> indicators in the terminal usually come at the right.

I think I mentioned left-truncation glyphs, so horizontally scrolling
a line should cause them to appear.

> And I think it's aesthetically important that the extra glyphs don't
> shift line text to the right unless absolutely necessary, so those
> issues don't materialize.

That cannot be done in all cases.

> > But that was only one of my goals, the other was to do in
> > the infrastructure something that can be done well only there.
> 
> Meaning, to use something other than the margins for display?

No, I was speaking about the line-number display in general.

> >> Agreed that it's the right direction (although I wouldn't say no to a
> >> popup library in the core either ;-). Would it help in non-graphical
> >> mode, though? Does it support non-maximized frames?
> > 
> > AFAIU, it supports any kind of frame.
> 
> I can't see a way to create a less-than-fullscreen frame in terminal 
> Emacs (speaking of normal frames here).

Maybe I' was wrong.  I hope Martin will chime in and set the record
straight.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 25 Jun 2017 17:46:23 +0300
> 
> On 6/25/17 5:13 PM, Eli Zaretskii wrote:
> 
> > The displayed line number reflects the line of the buffer positions
> > corresponding to what's on that screen line.  If none of the buffer
> > positions appear on that screen line, it's the line of the buffer
> > position(s) "covered" by the display string/overlay which generates
> > the display.
> 
> It didn't look like this in the previous version. Might be considered a 
> bit annoying, repeating the number several times.

It wasn't supposed to be repeated, it was supposed to appear only
once.  Hmm... I think I see what causes this, let me try to come up
with a fix.

> BTW, by that logic, the last empty line should have a number as well: it 
> does correspond to a buffer position.

No, it doesn't correspond to any buffer position: there's never any
character at point-max.

> (with-current-buffer (get-buffer-create "popup-test.el")
>    (setq display-line-numbers t)
>    (insert "aaaaaaa
> aaaaaaa
> aaaaaaa
> aaaaaaa
> aaaaaaa
> ")
> 
>    (let ((ov (make-overlay (point-max) (point-max))))
>      (overlay-put ov 'after-string "bbbbbb\nbbbbbb\n")
>      (overlay-put ov 'display-line-numbers-disable t)))
> 
> After that, the buffer popup-test.el shows the "bbbbbb" lines prepended 
> with the empty line number columns. I'd rather they weren't there.

OK, thanks, I will look into this case.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/25/17 5:54 PM, Eli Zaretskii wrote:

> I think I mentioned left-truncation glyphs, so horizontally scrolling
> a line should cause them to appear.

They are shown on all lines at the same time, and

(car (posn-col-row (posn-at-point)))

returns 0 in the first visible column when horizontally scrolled.

IOW, I don't see any problems when horizontally scrolled either.

>> And I think it's aesthetically important that the extra glyphs don't
>> shift line text to the right unless absolutely necessary, so those
>> issues don't materialize.
> 
> That cannot be done in all cases.

No argument here. And there are other known issues with the popup anyway.

My point is, however, is that you've created a new kind of window area 
that can only used by the native line numbers, and no other features or 
packages. This seems not so great to me.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > Try as I might, I can't repro those kind of problems. Line wrap
 > indicators in the terminal usually come at the right. And I think it's
 > aesthetically important that the extra glyphs don't shift line text to
 > the right unless absolutely necessary, so those issues don't
 > materialize.

(let ((window (split-window-horizontally)))
   (set-window-margins window 0 0)
   (set-window-fringes window 0 0))

Now move the divider between the two windows as far as possible to the
right and in the window on the right move to the end of the first line.
Here a '$' appears at the beginning of each line

 > I think you're trying too hard to stay away from the margins. But
 > anyway, we have the alternative solution now.

Back then I considered Markus' idea to use the margins quite ingenious
(actually, I didn't believe it would work in practice).  Still I always
voted in favor of having line numbers provided by the redisplay engine.
I also voted for a line numbers cache like that of (or even in
combination with) the 'syntax-ppss' cache but that's a different
subject.

Maybe one day Eli will come up with a redesign which would allow to
divide a window into side-by-side subwindows that would share a common
scroll bar so ‘follow-mode’ would be done entirely by the display
engine.  And each such subwindow would have an arbitrary number of
subwindows usable as margins for that window and a line number subwindow
and whatever else we want.

 > I can't see a way to create a less-than-fullscreen frame in terminal
 > Emacs (speaking of normal frames here).

Popup frames (or tooltip frames) on terminals would have to be emulated
as we currently do for menus.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 >> I can't see a way to create a less-than-fullscreen frame in terminal
 >> Emacs (speaking of normal frames here).
 >
 > Maybe I' was wrong.  I hope Martin will chime in and set the record
 > straight.

I hope I've done that already in my previous mail.  Obviously, any such
emulation I mentioned in that mail would result in some sort of overlay
for the buffer text that is displayed there.  While such overlays are
temporarily tolerable for menus I strongly doubt that we would want them
for anything else.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/25/17 6:59 PM, martin rudalics wrote:

> (let ((window (split-window-horizontally)))
>    (set-window-margins window 0 0)
>    (set-window-fringes window 0 0))
> 
> Now move the divider between the two windows as far as possible to the
> right and in the window on the right move to the end of the first line.
> Here a '$' appears at the beginning of each line

Thanks. Still, the popup is positioned correctly here in this scenario.

> Maybe one day Eli will come up with a redesign which would allow to
> divide a window into side-by-side subwindows that would share a common
> scroll bar so ‘follow-mode’ would be done entirely by the display
> engine.  And each such subwindow would have an arbitrary number of
> subwindows usable as margins for that window and a line number subwindow
> and whatever else we want.

I'd rather think in terms of "submargins", one per "category", and 
having an alist of particular properties of each of those categories.

>  > I can't see a way to create a less-than-fullscreen frame in terminal
>  > Emacs (speaking of normal frames here).
> 
> Popup frames (or tooltip frames) on terminals would have to be emulated
> as we currently do for menus.

Thanks for commenting.

Weren't menus in the terminal rendered using a certain special method, 
different from overlays?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Sun, 25 Jun 2017 18:05:54 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> 
> > > The displayed line number reflects the line of the buffer positions
> > > corresponding to what's on that screen line.  If none of the buffer
> > > positions appear on that screen line, it's the line of the buffer
> > > position(s) "covered" by the display string/overlay which generates
> > > the display.
> > 
> > It didn't look like this in the previous version. Might be considered a 
> > bit annoying, repeating the number several times.
> 
> It wasn't supposed to be repeated, it was supposed to appear only
> once.  Hmm... I think I see what causes this, let me try to come up
> with a fix.

Should be fixed now.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > I'd rather think in terms of "submargins", one per "category", and
 > having an alist of particular properties of each of those categories.

Once we do that we should try to accommodate as much as possible in the
concept we provide and I think that having ‘follow-mode’ implemented by
the display engine would be quite valuable.  So I would vote for a more
general solution that encompasses it as well.

 > Weren't menus in the terminal rendered using a certain special method,
 > different from overlays?

I think it boils down to the same underlying concept: Have the display
engine replace one text with another profiting from the restriction that
on terminals all characters are created equal.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Sun, 25 Jun 2017 18:05:54 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> 
> > (with-current-buffer (get-buffer-create "popup-test.el")
> >    (setq display-line-numbers t)
> >    (insert "aaaaaaa
> > aaaaaaa
> > aaaaaaa
> > aaaaaaa
> > aaaaaaa
> > ")
> > 
> >    (let ((ov (make-overlay (point-max) (point-max))))
> >      (overlay-put ov 'after-string "bbbbbb\nbbbbbb\n")
> >      (overlay-put ov 'display-line-numbers-disable t)))
> > 
> > After that, the buffer popup-test.el shows the "bbbbbb" lines prepended 
> > with the empty line number columns. I'd rather they weren't there.
> 
> OK, thanks, I will look into this case.

Should be fixed now.  the fix only works for empty overlays at EOB,
but that should be enough, right?  Or do you see some other way of
specifying this property at EOB?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Sun, 25 Jun 2017 18:12:23 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: alexanderm@web.de, 27427@debbugs.gnu.org
> 
> I hope I've done that already in my previous mail.  Obviously, any such
> emulation I mentioned in that mail would result in some sort of overlay
> for the buffer text that is displayed there.  While such overlays are
> temporarily tolerable for menus I strongly doubt that we would want them
> for anything else.

You are right.  That trick works for menus only because we preempt the
input queue until the menu is exited, exactly what Dmitry wants to get
rid of.

Too bad.  I tried to think about some idea for implementing such
popups in the display engine, but for now came up empty-handed.  I
will think some more.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 25 Jun 2017 19:24:21 +0300
> 
> Weren't menus in the terminal rendered using a certain special method, 
> different from overlays?

Yes.  We simply overwrite the glyph matrices with the menu contents,
and then force a screen update.

But this cannot work for features that want Emacs to return to the
main loop, because anything that triggers any kind of redisplay will
restore portions of the display from buffer text and mess up the menu.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Sun, 25 Jun 2017 19:10:01 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > I'd rather think in terms of "submargins", one per "category", and
>  > having an alist of particular properties of each of those categories.
> 
> Once we do that we should try to accommodate as much as possible in the
> concept we provide and I think that having ‘follow-mode’ implemented by
> the display engine would be quite valuable.  So I would vote for a more
> general solution that encompasses it as well.

I think people who lobby for more extensive and flexible use of the
margins underestimate the amount of work something like that will
take.  The current management of the margin display is exceedingly
rudimentary.  Most of the sophisticated layout stuff we have in the
text area -- truncation and continuation markers, word-wrap, etc. --
all that doesn't exist there.  And the cursor cannot enter the
margins.  Basically, we just put there glyphs until the available
space is exhausted, and that's it.

By contrast, you guys are dreaming about full-fledged additional "text
areas" with all the features we now support in the single one we have.
That's an entirely new ballpark game, although I agree that it's a
natural generalization and extension of what we have.  The problem is
that the knowledge of the basic canvas geometry is hard-coded in many
places in the display code, and all of them will have to be reworked.

I think this would be a very good project and a significant progress
for Emacs, so I'd welcome such a development.  Just don't
underestimate the magnitude of the task.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Sun, 25 Jun 2017 21:36:51 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: alexanderm@web.de, 27427@debbugs.gnu.org, dgutov@yandex.ru
> 
> I think people who lobby for more extensive and flexible use of the
> margins underestimate the amount of work something like that will
> take.  The current management of the margin display is exceedingly
> rudimentary.  Most of the sophisticated layout stuff we have in the
> text area -- truncation and continuation markers, word-wrap, etc. --
> all that doesn't exist there.  And the cursor cannot enter the
> margins.  Basically, we just put there glyphs until the available
> space is exhausted, and that's it.
> 
> By contrast, you guys are dreaming about full-fledged additional "text
> areas" with all the features we now support in the single one we have.
> That's an entirely new ballpark game, although I agree that it's a
> natural generalization and extension of what we have.  The problem is
> that the knowledge of the basic canvas geometry is hard-coded in many
> places in the display code, and all of them will have to be reworked.
> 
> I think this would be a very good project and a significant progress
> for Emacs, so I'd welcome such a development.  Just don't
> underestimate the magnitude of the task.

Btw, it strikes me that if we want something like that, it should be
much easier to provide side-by-side windows that scroll together or
according to some predefined relation.  This at least doesn't need to
redesign the basic display geometry of a window, only to change the
order in which we traverse the window tree -- instead of the current
depth-first order we'd need some more complicated traversal, and
perhaps also some redisplay considerations that look at more than one
window at a time.  Then just removing the scroll bars from all but one
of these "lockstep" windows will get you what you want at a much
smaller effort.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Sun, 25 Jun 2017 17:59:01 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: alexanderm@web.de, 27427@debbugs.gnu.org
> 
> I also voted for a line numbers cache like that of (or even in
> combination with) the 'syntax-ppss' cache but that's a different
> subject.

For the record, IME caches are problematic in the display code.
Various display routines are invoked from semi-random places,
sometimes utterly unrelated to display per se.  As a typical example,
consider vertical-motion.

Also, redisplay optimizations many time perform layout only for small
portions of the window/frame, so you cannot easily rely on the fact
that you will be called for a significant number of screen lines,
something which would favor caching.

Last, but definitely not least, caching gets in the way when the
display engine decides it should abandon some layout attempt, back up,
and continue from some previous state of the layout process.  Many
times, this requires to reset the cache as well, which complicates
management.

So my advice is always to try to code efficient algorithms that can
perform reasonably fast without caching.  It turned out that counting
line (using the same code we always used for line-number display in
the mode line) is just such a fast method.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/25/17 8:57 PM, Eli Zaretskii wrote:

> Should be fixed now.

Thanks, both cases are working well now. See below for the third one 
(though it's relatively rare). ;-(

> the fix only works for empty overlays at EOB,
> but that should be enough, right?  Or do you see some other way of
> specifying this property at EOB?

No, it seems plenty enough for me. I doubt we'd ever going to try using 
text properties for the popup without an overlay.

The third case which disrupts popup positioning: horizontal scrolling, 
which we discussed earlier. With native numbers, it breaks our function 
which calculates the "usable" window body width. It looks like this:

(defun company--window-width ()
   (let ((ww (window-body-width)))
     ;; Account for the line continuation column.
     (when (zerop (cadr (window-fringes)))
       (cl-decf ww))
     (unless (or (display-graphic-p)
                 (version< "24.3.1" emacs-version))
       ;; Emacs 24.3 and earlier included margins
       ;; in window-width when in TTY.
       (cl-decf ww
                (let ((margins (window-margins)))
                  (+ (or (car margins) 0)
                     (or (cdr margins) 0)))))
     (when (and word-wrap
                (version< emacs-version "24.4.51.5"))
       ;; http://debbugs.gnu.org/19300
       (cl-decf ww))
     ;; whitespace-mode with newline-mark
     (when (and buffer-display-table
                (aref buffer-display-table ?\n))
       (cl-decf ww (1- (length (aref buffer-display-table ?\n)))))
     ww))

Without going into details, how do I figure out the width which line 
number glyphs are taking up? Any way to do that without calling 
posn-at-point at the beginning-of-visual-line?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

>> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Sun, 25 Jun 2017 19:24:21 +0300
>>
>> Weren't menus in the terminal rendered using a certain special method,
>> different from overlays?
>
> Yes.  We simply overwrite the glyph matrices with the menu contents,
> and then force a screen update.
>
> But this cannot work for features that want Emacs to return to the
> main loop, because anything that triggers any kind of redisplay will
> restore portions of the display from buffer text and mess up the menu.
>

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > Yes.  We simply overwrite the glyph matrices with the menu contents,
 > and then force a screen update.

We would have to do the same for popups.

 > But this cannot work for features that want Emacs to return to the
 > main loop, because anything that triggers any kind of redisplay will
 > restore portions of the display from buffer text and mess up the menu.

And for popups it would just have to overwrite the glyph matrix again.
Or what am I missing?

Yet I doubt that such popups would be desirable on terminal frames.  For
my taste, they would appear far too obtrusive and distracting.  I think
any such popup should be treated like a tooltip and get displayed in the
echo area.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > By contrast, you guys are dreaming about full-fledged additional "text
 > areas" with all the features we now support in the single one we have.

I suppose that's me only.  IIUC Dmitry would be content with just
getting multiple margins exhibiting the same behavior as they do now.
What I mean is that rather than thinking about how to improve a concept
like that of margins I'd invent a framework where we can provide an
arbitrary amount of arbitrary-sized windows that can be later used for
any purpose by the display engine among them as margins.

 > That's an entirely new ballpark game, although I agree that it's a
 > natural generalization and extension of what we have.  The problem is
 > that the knowledge of the basic canvas geometry is hard-coded in many
 > places in the display code, and all of them will have to be reworked.

Correct.  But the underlying concept exists already---in window.c.  The
hard part is obviously that of synchronizing the display of windows, for
example, to make sure that text laid out in the margins and fringes has
the expected line-to-line coherence with text laid out in the
corresponding text window.

 > I think this would be a very good project and a significant progress
 > for Emacs, so I'd welcome such a development.  Just don't
 > underestimate the magnitude of the task.

I already had my share of this when I attempted to implement horizontal
scroll bars on the top of windows.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > Btw, it strikes me that if we want something like that, it should be
 > much easier to provide side-by-side windows that scroll together or
 > according to some predefined relation.  This at least doesn't need to
 > redesign the basic display geometry of a window, only to change the
 > order in which we traverse the window tree -- instead of the current
 > depth-first order we'd need some more complicated traversal, and
 > perhaps also some redisplay considerations that look at more than one
 > window at a time.  Then just removing the scroll bars from all but one
 > of these "lockstep" windows will get you what you want at a much
 > smaller effort.

This is precisely what I had in mind and why I pleaded for a uniform
handling of windows elsewhere.  The question is whether and how to treat
margins, scroll bars and fringes as first class citizens in this regard.

Our scroll bars are already partly windows partly frames.  Hence
considering them a shared resource like the tool or menu bar
windows/frames should be straightforward.  Margins, fringes and line
numbers are subordinates but I would upgrade them so the display engine
is mostly disconnected from the layout problem.

Now I think we agree that two or more side-by-side windows emulating
‘follow-mode’ should share a single vertical scroll bar window whose
slider size and position would be computed from the amount of text
displayed in all of these windows taken together and the position of
‘point’ in the most recently selected window of them.

I think we could also have these windows share a common fringe and a
line number window.  I'm less sure about the margins.  I think a
vertical divider should be provided separately for each window so the
user can resize them separately with the mouse.  And a horizontal scroll
bar, if wanted, should be provided separately for each of the windows.

There's probably no rule for sharing mode and header lines: A common
mode line seems obviously preferable for ‘follow-mode’, a ruler in the
header line can be hardly shared.

An obvious problem ensuing from such an approach is, for example, that
the display engine may decide that the size of the line number window
must increase because one of the windows now has to display a larger
maximum line number or because ‘point’ has moved from a window with a
lower ‘point’ to a window with a higher one.  This will trigger a check
whether the remaining windows are still large enough and maybe how to
enlarge them.  So what is currently a local decision entirely embedded
in your line numbers code would become a more global decision of window
management.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > So my advice is always to try to code efficient algorithms that can
 > perform reasonably fast without caching.  It turned out that counting
 > line (using the same code we always used for line-number display in
 > the mode line) is just such a fast method.

Yet we have ‘line-number-display-limit’ and
‘line-number-display-limit-width’ ...

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > Without going into details, how do I figure out the width which line
 > number glyphs are taking up? Any way to do that without calling
 > posn-at-point at the beginning-of-visual-line?

You'd probably need a function ‘window-line-number-width’ callable after
the current glyph matrix was calculated.  But if your overlays then mess
with the number of lines displayed in the window you'll get into a loop.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/26/17 11:18 AM, martin rudalics wrote:

> Yet I doubt that such popups would be desirable on terminal frames.  For
> my taste, they would appear far too obtrusive and distracting.  I think
> any such popup should be treated like a tooltip and get displayed in the
> echo area.

We already have a company-mode frontend that uses the echo area to show 
completions (two of them, actually, slightly different from each other).

Problem is, it's hard to fit the same information you would in a popup, 
and the echo area is used for other things (often at the same time): 
e.g. ElDoc, as well as some extra info that company-mode shows for the 
currently selected completion. So the user would have to give up that.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Mon, 26 Jun 2017 10:18:10 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > Yes.  We simply overwrite the glyph matrices with the menu contents,
>  > and then force a screen update.
> 
> We would have to do the same for popups.

Doing that behind the back of the display engine is problematic, see
below.

>  > But this cannot work for features that want Emacs to return to the
>  > main loop, because anything that triggers any kind of redisplay will
>  > restore portions of the display from buffer text and mess up the menu.
> 
> And for popups it would just have to overwrite the glyph matrix again.
> Or what am I missing?

If we let Emacs escape into the main loop, we cannot control when
redisplay kicks in and what effect does it have.  For example, some
timer could display a message that could be longer than a single
screen line, so Emacs will resize the echo area and as result
redisplay the windows above the mode line.  Since the text of the
popup comes from a source about which the display engine knows
absolutely nothing, redisplay will do its job assuming that the screen
shows the contents before we overwrote it, and thus will completely
mess up the display.  Even if the code which displayed the popup gets
control right away (which isn't guaranteed in general, since
company-mode wants to be able to run arbitrary Lisp given user
interaction with the popup), the user will see a momentary flash of
messed-up display.

> Yet I doubt that such popups would be desirable on terminal frames.  For
> my taste, they would appear far too obtrusive and distracting.  I think
> any such popup should be treated like a tooltip and get displayed in the
> echo area.

I don't see how the echo area can fit the job, since the space there
is very small and there's no way to let the user select an alternative
using menu-like interaction.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Mon, 26 Jun 2017 10:21:23 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > So my advice is always to try to code efficient algorithms that can
>  > perform reasonably fast without caching.  It turned out that counting
>  > line (using the same code we always used for line-number display in
>  > the mode line) is just such a fast method.
> 
> Yet we have ‘line-number-display-limit’ and
> ‘line-number-display-limit-width’ ...

Whose default values could probably be enlarged considerably, given
the contemporary CPUs.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Mon, 26 Jun 2017 10:23:44 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > Without going into details, how do I figure out the width which line
>  > number glyphs are taking up? Any way to do that without calling
>  > posn-at-point at the beginning-of-visual-line?
> 
> You'd probably need a function ‘window-line-number-width’ callable after
> the current glyph matrix was calculated.

Or maybe a new value of the argument to window-body-width which would
cause it to return the width excluding the line-number part?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 26 Jun 2017 01:56:28 +0300
> 
> > the fix only works for empty overlays at EOB,
> > but that should be enough, right?  Or do you see some other way of
> > specifying this property at EOB?
> 
> No, it seems plenty enough for me. I doubt we'd ever going to try using 
> text properties for the popup without an overlay.

No, I meant is there any other way of having a property on EOB except
via empty overlays?  get-text-property always returns nil at
point-max.  Is there some creative way of using text properties there?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/26/17 6:22 PM, Eli Zaretskii wrote:

> No, I meant is there any other way of having a property on EOB except
> via empty overlays?  get-text-property always returns nil at
> point-max.  Is there some creative way of using text properties there?

Probably not.

Anyway, from where I'm standing, the new property is more of a temporary 
workaround, so I wouldn't worry about a detail like that.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 26 Jun 2017 18:28:41 +0300
> 
> Anyway, from where I'm standing, the new property is more of a temporary 
> workaround, so I wouldn't worry about a detail like that.

Based on ME, we might be surprised how long this temporary workaround
will be used... ;-)

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/26/17 6:50 PM, Eli Zaretskii wrote:

> Based on ME, we might be surprised how long this temporary workaround
> will be used... ;-)

That happens, sure. I imagine it wouldn't be too hard to add support for 
the additional cases later, though.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Johan Bockgård]

Eli Zaretskii <eliz@gnu.org> writes:

> No, I meant is there any other way of having a property on EOB except
> via empty overlays?  get-text-property always returns nil at
> point-max.  Is there some creative way of using text properties there?

`default-text-properties'

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > If we let Emacs escape into the main loop, we cannot control when
 > redisplay kicks in and what effect does it have.  For example, some
 > timer could display a message that could be longer than a single
 > screen line, so Emacs will resize the echo area and as result
 > redisplay the windows above the mode line.  Since the text of the
 > popup comes from a source about which the display engine knows
 > absolutely nothing, redisplay will do its job assuming that the screen
 > shows the contents before we overwrote it, and thus will completely
 > mess up the display.  Even if the code which displayed the popup gets
 > control right away (which isn't guaranteed in general, since
 > company-mode wants to be able to run arbitrary Lisp given user
 > interaction with the popup), the user will see a momentary flash of
 > messed-up display.

I assume that the buffer position of the popup would have been specified
by company-mode before.  Hence, if on a terminal the character at that
position would get displayed, the display engine would try to display
the overlay for the popup there instead.  On the right and below of that
position if it fits, "anywhere there" otherwise, possibly truncating it
to keep the echo area free as you did for menus.  And that overlay would
stick at that buffer position until company-mode removes it.

So resizing the echo area would, in the worst case, not display the
popup because its buffer position would be temporarily off screen.  But
I can't imagine any mess up here.

 > I don't see how the echo area can fit the job, since the space there
 > is very small and there's no way to let the user select an alternative
 > using menu-like interaction.

The echo area can be resized and can emulate any menu-like interaction.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > Or maybe a new value of the argument to window-body-width which would
 > cause it to return the width excluding the line-number part?

Probably excluding the line- and wrap-prefixes as well.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Tue, 27 Jun 2017 09:06:06 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > If we let Emacs escape into the main loop, we cannot control when
>  > redisplay kicks in and what effect does it have.  For example, some
>  > timer could display a message that could be longer than a single
>  > screen line, so Emacs will resize the echo area and as result
>  > redisplay the windows above the mode line.  Since the text of the
>  > popup comes from a source about which the display engine knows
>  > absolutely nothing, redisplay will do its job assuming that the screen
>  > shows the contents before we overwrote it, and thus will completely
>  > mess up the display.  Even if the code which displayed the popup gets
>  > control right away (which isn't guaranteed in general, since
>  > company-mode wants to be able to run arbitrary Lisp given user
>  > interaction with the popup), the user will see a momentary flash of
>  > messed-up display.
> 
> I assume that the buffer position of the popup would have been specified
> by company-mode before.  Hence, if on a terminal the character at that
> position would get displayed, the display engine would try to display
> the overlay for the popup there instead.  On the right and below of that
> position if it fits, "anywhere there" otherwise, possibly truncating it
> to keep the echo area free as you did for menus.  And that overlay would
> stick at that buffer position until company-mode removes it.
> 
> So resizing the echo area would, in the worst case, not display the
> popup because its buffer position would be temporarily off screen.  But
> I can't imagine any mess up here.

I think we are miscommunicating.  I was describing what would happen
if we try to use the same method as used for TTY menus.  There are no
overlays there, the glyphs are concocted out of thin air by C code and
put into the glyph matrices, overwriting what the display engine
thinks should be there.

By contrast, you are talking about using an overlay, which is what
company-mode is using already, and which is the source of the trouble
Dmitry would like to avoid.  The basic problem here is that the
company-mode popup is multiline, and Emacs cannot display rectangular
regions, only lines.  So company-mode inserts newlines into the
overlay string, and the result is that the popup covers parts of the
display which company-mode doesn't want to conceal.  So it needs to
copy those parts to the overlay string, to make the impression they
weren't covered by the overlay.  It also needs to make all kinds of
layout calculations and decisions.  All that in order to make an
impression of displaying a rectangular region at certain screen
coordinates, something that we ideally should have provided in the
display engine.

>  > I don't see how the echo area can fit the job, since the space there
>  > is very small and there's no way to let the user select an alternative
>  > using menu-like interaction.
> 
> The echo area can be resized and can emulate any menu-like interaction.

Yes, but not conveniently.  And if we are going to emulate, why use
the echo area at all? why not pop up a buffer, like Help mode does?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Tue, 27 Jun 2017 09:06:15 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > Or maybe a new value of the argument to window-body-width which would
>  > cause it to return the width excluding the line-number part?
> 
> Probably excluding the line- and wrap-prefixes as well.

Of course.  Actually, to _not_ exclude them would be extra work.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > I think we are miscommunicating.  I was describing what would happen
 > if we try to use the same method as used for TTY menus.  There are no
 > overlays there, the glyphs are concocted out of thin air by C code and
 > put into the glyph matrices, overwriting what the display engine
 > thinks should be there.
 >
 > By contrast, you are talking about using an overlay,

Sorry for being unclear.  I did mean overlaying text just as the menu
bar code does and did not mean overlays as produced by ‘make-overlay’.

 > which is what
 > company-mode is using already, and which is the source of the trouble
 > Dmitry would like to avoid.  The basic problem here is that the
 > company-mode popup is multiline, and Emacs cannot display rectangular
 > regions, only lines.

But here we talk about the terminal where Emacs can display rectangular
regions because all elements displayed have the same size (the menu bar
code proves it).  So such a region can cover mode and header lines, span
multiple (Emacs) windows---virtually everything that can be found within
the frame edges.

 > So company-mode inserts newlines into the
 > overlay string,

... which is evil because it has to calculate the beginning of the
second overlay (in the sense of being made by ‘make-overlay’) string
which is hardly reliable with proportional fonts, text properties and
all sorts of display artefacts including overlays strings produced by
someone else.

 > and the result is that the popup covers parts of the
 > display which company-mode doesn't want to conceal.  So it needs to
 > copy those parts to the overlay string, to make the impression they
 > weren't covered by the overlay.  It also needs to make all kinds of
 > layout calculations and decisions.  All that in order to make an
 > impression of displaying a rectangular region at certain screen
 > coordinates, something that we ideally should have provided in the
 > display engine.

And I still think we can do that for terminals without any problems
using the same approach you used for menus.  All we need is the buffer
position (whose character represents the preferred upper left corner of
the virtual rectangle) and the text to be shown (with newlines).  Any
clipping or repositioning is up to the display engine as for menus.  And
the lifetime of the rectangular region on screen is that of the popup
frame on a GUI.

 >>   > I don't see how the echo area can fit the job, since the space there
 >>   > is very small and there's no way to let the user select an alternative
 >>   > using menu-like interaction.
 >>
 >> The echo area can be resized and can emulate any menu-like interaction.
 >
 > Yes, but not conveniently.  And if we are going to emulate, why use
 > the echo area at all? why not pop up a buffer, like Help mode does?

Sure.  The echo area is used by tooltip mode and simple questions on
terminals, that's why I mentioned it at all.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Tue, 27 Jun 2017 17:27:45 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
> Sorry for being unclear.  I did mean overlaying text just as the menu
> bar code does and did not mean overlays as produced by ‘make-overlay’.

Then I don't understand what you are suggesting.

> But here we talk about the terminal where Emacs can display rectangular
> regions because all elements displayed have the same size (the menu bar
> code proves it).

Menu code proves that we can do that, but it also proves that we must
prevent Emacs from redisplaying anything for the entire time the menu
is active.  We do that in the menu case by various tricks, but those
are only possible because as long as the menu is active, Emacs is
conceptually stuck in a long input operation, and thus preventing
redisplay raises only minor issues, like display-time stops updating.
But company-mode wants to be able to reflect the current selection
immediately, even before the popup pops down (AFAIU), and in general
wants to be able to run arbitrary Lisp during the popup activation,
and that just cannot live with the tricks used by TTY menus, because
sooner or later redisplay will happen, and will mess up the screen, as
the glyph matrix created from buffer text will overwrite the stuff we
put there.

I hope I explained myself this time.

> So such a region can cover mode and header lines, span multiple
> (Emacs) windows---virtually everything that can be found within the
> frame edges.

It can -- until the first time redisplay kicks in.

> And I still think we can do that for terminals without any problems
> using the same approach you used for menus.  All we need is the buffer
> position (whose character represents the preferred upper left corner of
> the virtual rectangle) and the text to be shown (with newlines).  Any
> clipping or repositioning is up to the display engine as for menus.

But the display engine is not involved in the TTY menus, at least not
in the usual sense.  Please take a look at the second half of
display_menu_bar, and you will see what I mean.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> From: Johan Bockgård <bojohan@gnu.org>
> Cc: Dmitry Gutov <dgutov@yandex.ru>,  alexanderm@web.de,  27427@debbugs.gnu.org
> Date: Mon, 26 Jun 2017 23:30:24 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > No, I meant is there any other way of having a property on EOB except
> > via empty overlays?  get-text-property always returns nil at
> > point-max.  Is there some creative way of using text properties there?
> 
> `default-text-properties'

Thanks, I fixed the code to support that as well.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > But the display engine is not involved in the TTY menus, at least not
 > in the usual sense.  Please take a look at the second half of
 > display_menu_bar, and you will see what I mean.

Isn't display_menu_bar the code for displaying a non-X GUI menu?  I
suppose you allude to the "while (!leave)" part of tty_menu_activate.
Please correct me if I'm wrong.  I don't understand why this should be
relevant for popup frames, probably because I'm too silly.

So let me try again how I think a popup frame could be emulated on a
TTY.  Currently, on a TTY a popup frame is nothing else but a normal
frame because all special frame parameters are inhibited.  Hence, it
will have the same size and position as the frame above which it is
supposed to pop up, which doesn't make any sense.

To specify the position and contents of a popup frame on a TTY I see two
ways: Either in the buffer text via a normal Emacs overlay with a say
'tty-popup' property or via a separate list say ‘tty-popup-frames’.  The
following sketch is based on the former.

An overlay with a 'tty-popup' property would specify a buffer position
and a newline separated text.  The buffer position would implicitly
provide the upper left corner of the popup frame, the text its size.

The display engine would notice the overlay like any other overlay.
However, the 'tty-popup' property would cause it to (1) remember the
current column of the iterator and (2) draw the first line of the
overlay right where the iterator is at this moment, possibly clipping
this line of the overlay text at the frame edge and skipping as many
characters of underlying buffer text as there are on this overlay text
line.

On the next buffer text line the display engine would start to draw the
second line of the overlay text at the column remembered in (1),
clipping and skipping as on the first line.  It would continue to
process the overlay until either its text has been used up or the bottom
of the window or the frame's root window has been reached.

Any redisplay would now redraw the popup frame in the same way, possibly
at a different position or with different size.  The overlay would
probably need a 'window' property to allow showing the same buffer
position in several windows and some convention would be needed for how
to draw overlapping popup frames.  The overlay would be removed from its
buffer by the same function that makes the GUI popup frame invisible or
kills it.  Removing the overlay would cause a redraw just like removing
a GUI popup frame would cause an expose event and a subsequent redraw.

I don't think that the extra check for the column stored in (1) would be
overly expensive.  If a separate ‘tty-popup-frames’ list were used (each
entry containing a buffer position and a newline separated text), then
an approach where the desired glyph matrix is overwritten as with TTY
menus would be used.  This would remove the need for (1) at the expense
of calculating the desired position of the popup in the glyph matrix
from the buffer position and the need to overwrite the relevant portions
of the desired glyph matrix.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Wed, 28 Jun 2017 10:45:26 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > But the display engine is not involved in the TTY menus, at least not
>  > in the usual sense.  Please take a look at the second half of
>  > display_menu_bar, and you will see what I mean.
> 
> Isn't display_menu_bar the code for displaying a non-X GUI menu?  I
> suppose you allude to the "while (!leave)" part of tty_menu_activate.
> Please correct me if I'm wrong.  I don't understand why this should be
> relevant for popup frames, probably because I'm too silly.

Sorry, you are right.  I meant display_tty_menu_item.

> An overlay with a 'tty-popup' property would specify a buffer position
> and a newline separated text.  The buffer position would implicitly
> provide the upper left corner of the popup frame, the text its size.
> 
> The display engine would notice the overlay like any other overlay.
> However, the 'tty-popup' property would cause it to (1) remember the
> current column of the iterator and (2) draw the first line of the
> overlay right where the iterator is at this moment, possibly clipping
> this line of the overlay text at the frame edge and skipping as many
> characters of underlying buffer text as there are on this overlay text
> line.
> 
> On the next buffer text line the display engine would start to draw the
> second line of the overlay text at the column remembered in (1),
> clipping and skipping as on the first line.  It would continue to
> process the overlay until either its text has been used up or the bottom
> of the window or the frame's root window has been reached.

Alas, that is not how redisplay works.  For starters, you seem to
assume that it always traverses all the screen lines of a window (thus
"on the next buffer line" etc.).  But that is only so when a window
needs a complete redisplay, and the display engine tries very hard to
avoid that.  Many times it only redraws some of the lines, sometimes
just one line.  If that line is not the first line of the popup, how
will the display engine know that at some previous buffer position
there is an overlay?

Moreover, the same low-level display code is invoked by the move_it_*
functions which simulate display without drawing anything, and those
are very often invoked to traverse small portions of a buffer,
typically a small number of lines.  These will be in trouble as well,
for the same reason.

So we need a different solution, one that doesn't break due to
redisplay optimizations.

Dmitry, can you tell why the popup overlay is a single overlay with a
single multiline string, and not a series of overlays, one each for
every line shown in the popup?  I assume this caused or could cause
more serious problems than the current implementation, but what
problems were those?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > Alas, that is not how redisplay works.  For starters, you seem to
 > assume that it always traverses all the screen lines of a window (thus
 > "on the next buffer line" etc.).  But that is only so when a window
 > needs a complete redisplay, and the display engine tries very hard to
 > avoid that.  Many times it only redraws some of the lines, sometimes
 > just one line.  If that line is not the first line of the popup, how
 > will the display engine know that at some previous buffer position
 > there is an overlay?

Because an application would include that very line in the 'tty-popup'
overlay BEG...END range and the display engine has to know how to handle
an overlay that spans multiple lines.  Right?

 > Moreover, the same low-level display code is invoked by the move_it_*
 > functions which simulate display without drawing anything, and those
 > are very often invoked to traverse small portions of a buffer,
 > typically a small number of lines.  These will be in trouble as well,
 > for the same reason.

And still would have to be able to handle multiline overlays as
described above.

 > So we need a different solution, one that doesn't break due to
 > redisplay optimizations.

Then consider the ‘tty-popup-frames’ list approach: It would have to
remember the start and end glyph matrix positions of the popup frame as
drawn the last time and if these intersect with what has been redrawn in
an optimized way then it would have to (1) restore the old contents of
the popup frame by the real buffer text, if necessary and (2) rewrite
popup frame parts, if necessary.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Wed, 28 Jun 2017 20:35:50 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > Alas, that is not how redisplay works.  For starters, you seem to
>  > assume that it always traverses all the screen lines of a window (thus
>  > "on the next buffer line" etc.).  But that is only so when a window
>  > needs a complete redisplay, and the display engine tries very hard to
>  > avoid that.  Many times it only redraws some of the lines, sometimes
>  > just one line.  If that line is not the first line of the popup, how
>  > will the display engine know that at some previous buffer position
>  > there is an overlay?
> 
> Because an application would include that very line in the 'tty-popup'
> overlay BEG...END range and the display engine has to know how to handle
> an overlay that spans multiple lines.  Right?

I cannot answer this question, it's too general.  And doesn't
company-mode use before- and after-strings, at least sometimes?  Those
have BEG and END the same value, so you must hit that position to know
something's there.

> Then consider the ‘tty-popup-frames’ list approach: It would have to
> remember the start and end glyph matrix positions of the popup frame as
> drawn the last time and if these intersect with what has been redrawn in
> an optimized way then it would have to (1) restore the old contents of
> the popup frame by the real buffer text, if necessary and (2) rewrite
> popup frame parts, if necessary.

That'll cause flickering, as users will see "incorrect" stuff
momentarily visible until the popup is restored.

Moreover, the "redrawn" part is in dispnew.c routines, called long
after xdisp.c did its job.

I think the only workable idea is to create a special kind of window
that is not part of the usual window tree, and let the display engine
consider those windows after the "normal" ones have been.  Do popup
frames need to support windows, mode lines, etc?  Or are they more
like tooltips -- one window and no decorations?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/28/17 7:48 PM, Eli Zaretskii wrote:

> Dmitry, can you tell why the popup overlay is a single overlay with a
> single multiline string, and not a series of overlays, one each for
> every line shown in the popup?  I assume this caused or could cause
> more serious problems than the current implementation, but what
> problems were those?

Different tradeoffs, some different problems, and a lot of common ones 
(like text scaling, images, character widths, etc).

How would that help with the current issue?

One-line-per-overlay approach will always work worse in display-heavy 
buffers, for instance. Like the 'M-x report-emacs-bug' one. 
auto-complete (with popup.el) use this approach, however. company uses 
the other one.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 >> Because an application would include that very line in the 'tty-popup'
 >> overlay BEG...END range and the display engine has to know how to handle
 >> an overlay that spans multiple lines.  Right?
 >
 > I cannot answer this question, it's too general.  And doesn't
 > company-mode use before- and after-strings, at least sometimes?  Those
 > have BEG and END the same value, so you must hit that position to know
 > something's there.

Obviously, ‘company-mode’ would have to get rid of all earlier
workarounds.  Likely we would encapsulate the TTY/GUI differences in a
separate function or macro.

 > I think the only workable idea is to create a special kind of window
 > that is not part of the usual window tree, and let the display engine
 > consider those windows after the "normal" ones have been.  Do popup
 > frames need to support windows, mode lines, etc?  Or are they more
 > like tooltips -- one window and no decorations?

Note that so far the concept "popup frame" does not exist yet.  If the
window system supports them, we have child frames, undecorated frames,
override redirect frames and some more.  All these are full-fledged
frames but we can easily provide restrictions, single window only, no
echo area, no mode or header lines ... you name it.

But note in this context that a native Emacs tooltip frame is not very
viable.  It works only because it's so ephemeral that before Lisp code
has a chance to lay its hands on it, it has already disappeared.  In
this sense, a tooltip is similar to a TTY menu---an application cannot
disrupt its display.

So the answer to your question is: We can supply any restrictions the
TTY display code would need although I'd prefer to call this a "special
kind of frame" and not a "special kind of window".

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 29 Jun 2017 04:34:16 +0300
> 
> On 6/28/17 7:48 PM, Eli Zaretskii wrote:
> 
> > Dmitry, can you tell why the popup overlay is a single overlay with a
> > single multiline string, and not a series of overlays, one each for
> > every line shown in the popup?  I assume this caused or could cause
> > more serious problems than the current implementation, but what
> > problems were those?
> 
> Different tradeoffs, some different problems, and a lot of common ones 
> (like text scaling, images, character widths, etc).

But all of these are not relevant to TTY frames, right?

> How would that help with the current issue?

Martin is trying very hard to come up with a method to overcome the
fact that Emacs cannot display "rectangular" overlay strings.
Breaking the string into several one-line strings and putting their
overlays at the appropriate buffer positions would solve this problem.

> One-line-per-overlay approach will always work worse in display-heavy 
> buffers, for instance. Like the 'M-x report-emacs-bug' one. 

Why would it work worse in that case?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Thu, 29 Jun 2017 09:17:41 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
> So the answer to your question is: We can supply any restrictions the
> TTY display code would need although I'd prefer to call this a "special
> kind of frame" and not a "special kind of window".

I prefer "window" for the simple reason that doing so will most
probably wreak the least havoc on TTY display, in particular because
it won't invalidate the assumption that only one frame is visible at
any given time.  Also, frames can have decorations, even on a TTY,
which we are better without in this case.  And finally, you most
probably remember that TTY redisplay is frame-based, so having just
one of them would fit better.

Should be a nice project.  Any takers?

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/29/17 7:20 PM, Eli Zaretskii wrote:

>> Different tradeoffs, some different problems, and a lot of common ones
>> (like text scaling, images, character widths, etc).
> 
> But all of these are not relevant to TTY frames, right?

The `display' issue in `M-x report-emacs-bug' should be just as 
relevant. And similar stuff.

Character widths might be relevant as well in some terminals, but that's 
hardly something we could fix in Emacs.

> Martin is trying very hard to come up with a method to overcome the
> fact that Emacs cannot display "rectangular" overlay strings.
> Breaking the string into several one-line strings and putting their
> overlays at the appropriate buffer positions would solve this problem.

Like I said, we have another completion package that does this (but the 
authors refuse to assign copyright).

How will that help with the arithmetics? How is it better than the 
one-overlay approach for the current situation?

>> One-line-per-overlay approach will always work worse in display-heavy
>> buffers, for instance. Like the 'M-x report-emacs-bug' one.
> 
> Why would it work worse in that case?

Imagine that point is above the "If Emacs crashed..." display overlay. 
There is no physical line below it where we can put an overlay with the 
first popup line.

I suppose we could replace (propertize "\n" 'display txt) string that is 
there with a fully made up overlay string, but a) it's less trivial than 
you probably imagined initially, b) the buffer text below it is going to 
jump up and down as the popup is shown and hidden.

With the one-overlay approach, we ignore that `display' property (so 
there's empty space there when the popup is displayed), but preserve the 
height in rows, so the other buffer text is not jumping.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > I prefer "window" for the simple reason that doing so will most
 > probably wreak the least havoc on TTY display, in particular because
 > it won't invalidate the assumption that only one frame is visible at
 > any given time.  Also, frames can have decorations, even on a TTY,
 > which we are better without in this case.  And finally, you most
 > probably remember that TTY redisplay is frame-based, so having just
 > one of them would fit better.

As a window it won't appear on any window list since it's got no frame.
If we made it a window of the frame where we popped it up, then where
would we put it into that frame's window tree?  The window tree code is
not very clean, but changing it for the sake of TTY popups sounds like
very intrusive surgery.  All the (implicit) invariants that window sizes
have to sum up would get invalidated.

Suppose we spliced in some special TTY_popup_window_list slot for every
frame. Then we'd have to invent functions for accessing and changing its
elements.  If such an element were an Emacs window, practically all
functions that access and change windows and their properties would have
to be rewritten in consequence of that.  This is hardly feasible.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Fri, 30 Jun 2017 10:27:50 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
> Suppose we spliced in some special TTY_popup_window_list slot for every
> frame.

That's what I had in mind, yes.

> Then we'd have to invent functions for accessing and changing its
> elements.

We already have functions for accessing elements of lists and changing
lists.

> If such an element were an Emacs window, practically all functions
> that access and change windows and their properties would have to be
> rewritten in consequence of that.

What functions are you alluding to here, specifically?  And what did
you mean by "an Emacs window"?

My idea was that these windows would be "special", in that they could
only display some special buffer or maybe only a string.  About the
only function we'd need is to set the coordinates where such a window
will be displayed.  Everything else, including the dimensions, will be
determined by the text to be displayed there.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > My idea was that these windows would be "special", in that they could
 > only display some special buffer or maybe only a string.

Then we are back at my earlier ‘tty-popup-frames’ proposal with
"windows" substituting "frames".  Please lets call these objects
"popups" for now.

 > About the
 > only function we'd need is to set the coordinates where such a window
 > will be displayed.  Everything else, including the dimensions, will be
 > determined by the text to be displayed there.

We'd have to decide when to call the function that sets the coordinates.
For that purpose each popup would probably have to store the window it
originated from to handle scrolling of that window and any changes to
its buffer.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Sat, 01 Jul 2017 12:31:49 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > My idea was that these windows would be "special", in that they could
>  > only display some special buffer or maybe only a string.
> 
> Then we are back at my earlier ‘tty-popup-frames’ proposal with
> "windows" substituting "frames".  Please lets call these objects
> "popups" for now.

OK.

>  > About the
>  > only function we'd need is to set the coordinates where such a window
>  > will be displayed.  Everything else, including the dimensions, will be
>  > determined by the text to be displayed there.
> 
> We'd have to decide when to call the function that sets the coordinates.

I hoped the application would do that, via some new API.

> For that purpose each popup would probably have to store the window it
> originated from to handle scrolling of that window and any changes to
> its buffer.

That could be one way, but maybe we could use something more direct.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > I hoped the application would do that, via some new API.

We'd at least need something like a ‘TTY-check-popup-functions’ called
immediately before the display engine looks at the popups list: The
application could then check whether its popup is still relevant and
whether is should be redisplayed, possibly at a different location, with
different text.  If the application decides that nothing should be
changed and the desired matrix and the current matrix are the same for a
specific popup, the display engine would probably leave that popup in
the desired matrix alone.

 >> For that purpose each popup would probably have to store the window it
 >> originated from to handle scrolling of that window and any changes to
 >> its buffer.
 >
 > That could be one way, but maybe we could use something more direct.

So far, the implementation of child and undecorated frames is complete.
Hence, for the moment I intend to proceed as follows: People who need
such frames should read the manual and try to provide a GUI solution for
their problem.  As soon as we have a few samples that work
satisfactorily on GUI systems, we can try to implement a fairly useful
workaround for TTY systems.  However, if people say they need to select
such frames or put the cursor into one of their windows, we have to
rethink how to implement a TTY solution if it's viable at all.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Sat, 01 Jul 2017 15:22:43 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > I hoped the application would do that, via some new API.
> 
> We'd at least need something like a ‘TTY-check-popup-functions’ called
> immediately before the display engine looks at the popups list: The
> application could then check whether its popup is still relevant and
> whether is should be redisplayed, possibly at a different location, with
> different text.

Why not the other way around: let the application specify the text and
the position of the popup in dedicated variables, and then redisplay
could just access those as part of its job?

> So far, the implementation of child and undecorated frames is complete.
> Hence, for the moment I intend to proceed as follows: People who need
> such frames should read the manual and try to provide a GUI solution for
> their problem.  As soon as we have a few samples that work
> satisfactorily on GUI systems, we can try to implement a fairly useful
> workaround for TTY systems.  However, if people say they need to select
> such frames or put the cursor into one of their windows, we have to
> rethink how to implement a TTY solution if it's viable at all.

Sounds like a good plan, thanks.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > Why not the other way around: let the application specify the text and
 > the position of the popup in dedicated variables, and then redisplay
 > could just access those as part of its job?

The application has to react when a given buffer position changes its
position on screen.  If ‘window-text-change-functions’ is sufficient to
determine that (I never used it), it could add itself to that hook and
specify the popup accordingly.  Otherwise, the application would have to
track whether the window was scrolled, resized or moved, or its buffer
modified.  Or, for example, whether an overlay before the popup position
changed from visible to invisible.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Sat, 01 Jul 2017 17:20:34 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > Why not the other way around: let the application specify the text and
>  > the position of the popup in dedicated variables, and then redisplay
>  > could just access those as part of its job?
> 
> The application has to react when a given buffer position changes its
> position on screen.

Yes, but I presumed it will do it the same way company-mode reacts to
such changes today.  Currently. company-mode recomputes and moves its
overlay; it will instead recompute the text and its position.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > Yes, but I presumed it will do it the same way company-mode reacts to
 > such changes today.  Currently. company-mode recomputes and moves its
 > overlay; it will instead recompute the text and its position.

To my knowledge ‘company-mode’ and all packages trying to solve similar
display problems are based on idle timers.  The overlay is moved, if
necessry, when the timeout expires.  I'm not sure whether it makes sense
to use a timer-less approach.  If it does, we should allow moving the
overlay right after recalculating the frame's glyph matrix on TTYs.
Whether removing timers makes sense for GUIs is yet another question.

In this context the ‘window-text-change-functions’ I mentioned earlier
seems completely useless.  Here doing for example

(add-hook 'window-text-change-functions 'ignore)

just gets me into a loop and I have to kill Emacs.  Also, it's not clear
to me why this hook is considered abnormal and how to identify from
Elisp the window that is redisplayed at the time the hook is run.  The
fact that the window's buffer is current at that time is hardly useful.
So I think we should either remove that hook or completely redesign it.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Sun, 02 Jul 2017 09:54:12 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > Yes, but I presumed it will do it the same way company-mode reacts to
>  > such changes today.  Currently. company-mode recomputes and moves its
>  > overlay; it will instead recompute the text and its position.
> 
> To my knowledge ‘company-mode’ and all packages trying to solve similar
> display problems are based on idle timers.

I thought they used post-command-hook, but I will let Dmitry comment
on that.

> In this context the ‘window-text-change-functions’ I mentioned earlier
> seems completely useless.  Here doing for example
> 
> (add-hook 'window-text-change-functions 'ignore)
> 
> just gets me into a loop and I have to kill Emacs.  Also, it's not clear
> to me why this hook is considered abnormal and how to identify from
> Elisp the window that is redisplayed at the time the hook is run.  The
> fact that the window's buffer is current at that time is hardly useful.
> So I think we should either remove that hook or completely redesign it.

This hook ws introduced for linum.el, but linum.el as added to Emacs
never used it.  We should delete the hook and any references to it in
the manuals.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 7/2/17 5:10 PM, Eli Zaretskii wrote:

>> To my knowledge ‘company-mode’ and all packages trying to solve similar
>> display problems are based on idle timers.
> 
> I thought they used post-command-hook, but I will let Dmitry comment
> on that.

post-command hooks and non-idle timers. Not that it matters much.

It would be nice if the popup were able to reposition itself when the 
Emacs window is resized, of course (post-command-hook doesn't always run 
in such cases), but we've been living well enough without that.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sun, 2 Jul 2017 17:45:13 +0300
> 
> It would be nice if the popup were able to reposition itself when the 
> Emacs window is resized

We could have a feature whereby the coordinates are determined by a
specific buffer position shown in a "normal" window.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 7/2/17 6:18 PM, Eli Zaretskii wrote:

> We could have a feature whereby the coordinates are determined by a
> specific buffer position shown in a "normal" window.

That's what I was thinking of as well. But that, in turn, might call for 
some extra features:

When there is not enough space below the current line to show the popup, 
we display it above the current line. I'd expect the new popup code 
reposition it like that automatically as well.

But: in company we have feature where, when the popup is displayed above 
the current line, the popup lines are inverted vertically (so that the 
first completion is the closest to the current line visually). I'm not a 
fan, but it's fairly popular.

If the core popup handles repositioning, it would have to handle 
inverting (optionally) as well, or run some sort of hook to require the 
popup items to be recomputed.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: 27427@debbugs.gnu.org, alexanderm@web.de
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 3 Jul 2017 03:22:56 +0300
> 
> On 7/2/17 6:18 PM, Eli Zaretskii wrote:
> 
> > We could have a feature whereby the coordinates are determined by a
> > specific buffer position shown in a "normal" window.
> 
> That's what I was thinking of as well. But that, in turn, might call for 
> some extra features:
> 
> When there is not enough space below the current line to show the popup, 
> we display it above the current line. I'd expect the new popup code 
> reposition it like that automatically as well.
> 
> But: in company we have feature where, when the popup is displayed above 
> the current line, the popup lines are inverted vertically (so that the 
> first completion is the closest to the current line visually). I'm not a 
> fan, but it's fairly popular.
> 
> If the core popup handles repositioning, it would have to handle 
> inverting (optionally) as well, or run some sort of hook to require the 
> popup items to be recomputed.

These are all doable, and AFAIU are needed even if the coordinates are
specified by the application.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

[-- Attachment #1: Type: text/plain, Size: 241 bytes --]

 > This hook ws introduced for linum.el, but linum.el as added to Emacs
 > never used it.  We should delete the hook and any references to it in
 > the manuals.

The attached patch tries to restore the state before its introduction.

martin

[-- Attachment #2: remove-window-text-change-functions.diff --]
[-- Type: text/plain, Size: 3191 bytes --]

diff --git a/doc/lispref/hooks.texi b/doc/lispref/hooks.texi
index 0ac5b08..6443464 100644
--- a/doc/lispref/hooks.texi
+++ b/doc/lispref/hooks.texi
@@ -241,11 +241,6 @@ Standard Hooks
 @itemx window-scroll-functions
 @itemx window-size-change-functions
 @xref{Window Hooks}.
-
-@item window-text-change-functions
-@vindex window-text-change-functions
-Functions to call in redisplay when text in the window might change.
-
 @end table

 @ignore
diff --git a/src/xdisp.c b/src/xdisp.c
index 8bc5d81..bd60ed2 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -16431,9 +16431,8 @@ enum
   eassert (XMARKER (w->start)->buffer == buffer);
   eassert (XMARKER (w->pointm)->buffer == buffer);

-  /* We come here again if we need to run window-text-change-functions
-     below.  */
- restart:
+  specbind (Qinhibit_point_motion_hooks, Qt);
+
   reconsider_clip_changes (w);
   frame_line_height = default_line_pixel_height (w);
   margin = window_scroll_margin (w, MARGIN_IN_LINES);
@@ -16492,6 +16491,10 @@ enum
   /* Really select the buffer, for the sake of buffer-local
      variables.  */
   set_buffer_internal_1 (XBUFFER (w->contents));
+  SET_TEXT_POS (opoint, PT, PT_BYTE);
+
+  beg_unchanged = BEG_UNCHANGED;
+  end_unchanged = END_UNCHANGED;

   current_matrix_up_to_date_p
     = (w->window_end_valid
@@ -16500,23 +16503,6 @@ enum
        && !window_outdated (w)
        && !hscrolling_current_line_p (w));

-  /* Run the window-text-change-functions
-     if it is possible that the text on the screen has changed
-     (either due to modification of the text, or any other reason).  */
-  if (!current_matrix_up_to_date_p
-      && !NILP (Vwindow_text_change_functions))
-    {
-      safe_run_hooks (Qwindow_text_change_functions);
-      goto restart;
-    }
-
-  beg_unchanged = BEG_UNCHANGED;
-  end_unchanged = END_UNCHANGED;
-
-  SET_TEXT_POS (opoint, PT, PT_BYTE);
-
-  specbind (Qinhibit_point_motion_hooks, Qt);
-
   buffer_unchanged_p
     = (w->window_end_valid
        && !current_buffer->clip_changed
@@ -31692,7 +31678,6 @@ A polygon is a cons (poly . [x0 y0 x1 y1 ...]) where each pair in the
   DEFSYM (Qoverriding_terminal_local_map, "overriding-terminal-local-map");
   DEFSYM (Qoverriding_local_map, "overriding-local-map");
   DEFSYM (Qwindow_scroll_functions, "window-scroll-functions");
-  DEFSYM (Qwindow_text_change_functions, "window-text-change-functions");
   DEFSYM (Qredisplay_end_trigger_functions, "redisplay-end-trigger-functions");
   DEFSYM (Qinhibit_point_motion_hooks, "inhibit-point-motion-hooks");
   DEFSYM (Qeval, "eval");
@@ -32016,11 +32001,6 @@ either to point into another buffer (e.g. via `set-window-buffer') or another
 work.  */);
   Vwindow_scroll_functions = Qnil;

-  DEFVAR_LISP ("window-text-change-functions",
-	       Vwindow_text_change_functions,
-    doc: /* Functions to call in redisplay when text in the window might change.  */);
-  Vwindow_text_change_functions = Qnil;
-
   DEFVAR_LISP ("redisplay-end-trigger-functions", Vredisplay_end_trigger_functions,
     doc: /* Functions called when redisplay of a window reaches the end trigger.
 Each function is called with two arguments, the window and the end trigger value.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > post-command hooks and non-idle timers. Not that it matters much.

I don't understand the reasons for this combination.  Inherently this
emulates idle timers as also the variable names (‘company-idle-delay’,
‘company-tooltip-idle-delay’) indicate.

 > It would be nice if the popup were able to reposition itself when the
 > Emacs window is resized, of course (post-command-hook doesn't always
 > run in such cases), but we've been living well enough without that.

Putting the function on ‘window-size-change-functions’ should handle
that now.  For earlier Emacs versions you probably have to resort to
‘window-configuration-change-hook’ as well.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 7/6/17 10:08 AM, martin rudalics wrote:

> I don't understand the reasons for this combination.  Inherently this
> emulates idle timers as also the variable names (‘company-idle-delay’,
> ‘company-tooltip-idle-delay’) indicate.

We need post-command-hook anyway, to add or remove the timer depending 
on the last command. I don't recall the exact reasons for the choice 
between the idle and non-idle timers, but flyspell comes to mind (and 
its use of sit-for).

>  > It would be nice if the popup were able to reposition itself when the
>  > Emacs window is resized, of course (post-command-hook doesn't always
>  > run in such cases), but we've been living well enough without that.
> 
> Putting the function on ‘window-size-change-functions’ should handle
> that now.

Maybe that's enough. What if the user moves the Emacs frame around with 
a mouse, without resizing? Will the tooltip window follow?

> For earlier Emacs versions you probably have to resort to
> ‘window-configuration-change-hook’ as well.

They won't support the new style tooltips anyway.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Date: Thu, 06 Jul 2017 09:08:13 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: dgutov@yandex.ru, alexanderm@web.de, 27427@debbugs.gnu.org
> 
>  > This hook ws introduced for linum.el, but linum.el as added to Emacs
>  > never used it.  We should delete the hook and any references to it in
>  > the manuals.
> 
> The attached patch tries to restore the state before its introduction.

Thanks.

Are the rearrangements necessary, or just out of aesthetic feelings?
If the latter, I'd prefer not to rearrange, as that makes forensics a
tad harder.

Otherwise, fine with me; please push.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > We need post-command-hook anyway, to add or remove the timer depending
 > on the last command. I don't recall the exact reasons for the choice
 > between the idle and non-idle timers, but flyspell comes to mind (and
 > its use of sit-for).

It would be interesting to know these reasons to avoid reinventing that
choice elsewhere.

 > Maybe that's enough. What if the user moves the Emacs frame around
 > with a mouse, without resizing? Will the tooltip window follow?

A child frame automatically moves along with its parent.  Normal frames
must be moved manually.  ‘move-frame-functions’ is the hook provided to
do that.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[martin rudalics]

 > Are the rearrangements necessary, or just out of aesthetic feelings?

Neither.  They were part of the attempt to faithfully restore the state
before that variable was introduced.

 > If the latter, I'd prefer not to rearrange, as that makes forensics a
 > tad harder.

We violently agree.

 > Otherwise, fine with me; please push.

Done.

martin

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 6/26/17 1:56 AM, Dmitry Gutov wrote:
> On 6/25/17 8:57 PM, Eli Zaretskii wrote:
> 
>> Should be fixed now.
> 
> Thanks, both cases are working well now.

...and either something changed before the merge into master, or I 
tested the result poorly. Probably the latter.

What doesn't work well is that the popup "background" gets displaced to 
the left by the width of the numbers column, when rendered using the 
popup overlay. Background meaning the contents of all the lines behind 
the popup. And that happens because the line number glyphs are gone for 
those specific lines, and we can't pad them without knowing the width of 
the numbers column.

Now, I've come upon this idea of a trivial patch. What do you think, 
Eli? It seems to work perfectly, even though I'm struggling to 
understand how this might happen under the covers. Is it brittle?

We bind display-line-numbers to nil around the call to posn-at-point, so 
that it calculates the column value that we want. I imagine there's some 
extra redisplay work involved, but it still turns out to be faster than 
calling posn-at-point twice.

diff --git a/company.el b/company.el
index f361bb7..869c5de 100644
--- a/company.el
+++ b/company.el
@@ -835,7 +835,8 @@ means that `company-mode' is always turned on except 
in `message-mode' buffers."
      (cons (+ col (window-hscroll)) row)))

  (defun company--col-row (&optional pos)
-  (company--posn-col-row (posn-at-point pos)))
+  (let (display-line-numbers)
+    (company--posn-col-row (posn-at-point pos))))

  (defun company--row (&optional pos)
    (cdr (company--col-row pos)))

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> Date: Fri, 14 Jul 2017 02:04:21 +0300
> 
> What doesn't work well is that the popup "background" gets displaced to 
> the left by the width of the numbers column, when rendered using the 
> popup overlay. Background meaning the contents of all the lines behind 
> the popup. And that happens because the line number glyphs are gone for 
> those specific lines, and we can't pad them without knowing the width of 
> the numbers column.

You could use the new function line-number-display-width to obtain
that information.

> Now, I've come upon this idea of a trivial patch. What do you think, 
> Eli? It seems to work perfectly, even though I'm struggling to 
> understand how this might happen under the covers. Is it brittle?

No, it isn't brittle.  What are you struggling to understand?  Maybe I
can help.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 7/14/17 9:11 AM, Eli Zaretskii wrote:

> You could use the new function line-number-display-width to obtain
> that information.

Thanks. The current patch seems preferable, though. And since you are 
saying it's good enough, it seems I have no need for the 
display-line-numbers-disable text property either, sorry.

> No, it isn't brittle.  What are you struggling to understand?  Maybe I
> can help.

To get the value of the current column, Emacs needs to trigger 
redisplay, right? To know how many columns the numbers take up. Then it 
has to redisplay again before the user sees the result.

So I'm surprised Emacs knows it needs to perform two redisplays just 
because I bound display-line-numbers to nil temporarily.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 14 Jul 2017 09:24:58 +0300
> 
> > No, it isn't brittle.  What are you struggling to understand?  Maybe I
> > can help.
> 
> To get the value of the current column, Emacs needs to trigger 
> redisplay, right? To know how many columns the numbers take up. Then it 
> has to redisplay again before the user sees the result.
> 
> So I'm surprised Emacs knows it needs to perform two redisplays just 
> because I bound display-line-numbers to nil temporarily.

That's because posn-at-point "simulates" redisplay: it performs all
the layout calculations exactly like redisplay would, but without
actually displaying anything.  Otherwise, how would posn-at-point be
able to return you the information you request, even when there are no
line numbers?

(If you think posn-at-point takes that information from what is
displayed on the glass, or from some of its internal representation,
then that's not what it does, because the internal representation of
what's on the glass is many times outdated when a Lisp program runs,
so it cannot be trusted.)

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Dmitry Gutov]

On 7/14/17 10:04 AM, Eli Zaretskii wrote:

> That's because posn-at-point "simulates" redisplay: it performs all
> the layout calculations exactly like redisplay would, but without
> actually displaying anything.

Very cool. In that case, let's consider this issue resolved.

I've already pushed this fix and cut a new release now.

> (If you think posn-at-point takes that information from what is
> displayed on the glass, or from some of its internal representation,
> then that's not what it does, because the internal representation of
> what's on the glass is many times outdated when a Lisp program runs,
> so it cannot be trusted.)

I was thinking there was some change canary (or many of them), and the 
"internal representation" is recomputed when any of those values change.

bug#27427: 26.0.50; Native line numbers lead to display error in company-mode popup

[Eli Zaretskii]

> Cc: alexanderm@web.de, 27427-done@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Sat, 15 Jul 2017 20:38:15 +0300
> 
> > (If you think posn-at-point takes that information from what is
> > displayed on the glass, or from some of its internal representation,
> > then that's not what it does, because the internal representation of
> > what's on the glass is many times outdated when a Lisp program runs,
> > so it cannot be trusted.)
> 
> I was thinking there was some change canary (or many of them), and the 
> "internal representation" is recomputed when any of those values change.

The internal representation is recomputed only as part of a redisplay
cycle.  And we cannot start a redisplay cycle before the Lisp
interpreter has done its thing, and we are back in the main loop,
because only then we know all the buffers and other related Lisp data
are in a consistent state that can be reflected on the glass without
risking inconsistencies.