Getting column position accounting for overlays

Getting column position accounting for overlays

[Yuri D'Elia]

Is there a quick way (without iterating through 'display overlays) to
get the current column position for the displayed text?

Re: Getting column position accounting for overlays

[Eli Zaretskii]

> From: Yuri D'Elia <wavexx@thregr.org>
> Date: Tue, 4 Aug 2015 22:01:24 +0200
> 
> Is there a quick way (without iterating through 'display overlays) to
> get the current column position for the displayed text?

What is "the current column position"?  How do you define a "column"
when text can include variable-size fonts and images?

One answer to your question is posn-at-point, but I'm not sure that is
what you want.

Re: Getting column position accounting for overlays

[Yuri D'Elia]

On 05/08/15 04:51, Eli Zaretskii wrote:
>> From: Yuri D'Elia <wavexx@thregr.org>
>> Date: Tue, 4 Aug 2015 22:01:24 +0200
>>
>> Is there a quick way (without iterating through 'display overlays) to
>> get the current column position for the displayed text?
> 
> What is "the current column position"?  How do you define a "column"
> when text can include variable-size fonts and images?
> 
> One answer to your question is posn-at-point, but I'm not sure that is
> what you want.

I expected such question ;)

I'd like the equivalent of (current-column), or character length between
(beginning-of-line) and cursor position [modulus tabs], but using the
actual displayed overlays.

I would almost have expected something to extract the visualized string
between two points, like `buffer-substring' but iterating through
overlays; but couldn't find any (did I miss it?)

(posn-col-row (posn-at-point)) "would" work, but 1) seems inefficient,
especially considering I don't care about variable-pitch fonts and 2)
doesn't work if the point is not visible.

Re: Getting column position accounting for overlays

[Eli Zaretskii]

> From: Yuri D'Elia <wavexx@thregr.org>
> Date: Wed, 5 Aug 2015 11:10:59 +0200
> 
> I'd like the equivalent of (current-column), or character length between
> (beginning-of-line) and cursor position [modulus tabs], but using the
> actual displayed overlays.

I think you want this:

  (/ (car (window-text-pixel-size nil (line-beginning-position) (point)))
     (default-font-width))

But that's a guess.  Please tell more about your use case, because I
don't quite understand why would you need this.  Also note that
default-font-width didn't exist before Emacs 25.

> I would almost have expected something to extract the visualized string
> between two points, like `buffer-substring' but iterating through
> overlays; but couldn't find any (did I miss it?)

The overlay strings are directly accessible via 'overlay-get', so why
would anyone need the feature you expected to exist?

> (posn-col-row (posn-at-point)) "would" work, but 1) seems inefficient,
> especially considering I don't care about variable-pitch fonts and 2)
> doesn't work if the point is not visible.

If you don't care about variable-pitch fonts, how come you do care
about overlay and display strings?  They are conceptually the same:
they extend the width of the text by using data that is not directly
available from buffer text.

Re: Getting column position accounting for overlays

[Yuri D'Elia]

On 05/08/15 17:21, Eli Zaretskii wrote:
>> I would almost have expected something to extract the visualized string
>> between two points, like `buffer-substring' but iterating through
>> overlays; but couldn't find any (did I miss it?)
> 
> The overlay strings are directly accessible via 'overlay-get', so why
> would anyone need the feature you expected to exist?

I don't think we're on the same line of thought.

The usage scenario is pretty simple: assume a terminal display.
I want to determine the current /visual/ column at point.

(current-column) will not account for overlays that have a 'display
property of a different length than the overlay range.

It's really that simple.

The second part is a consequence: if I want to have a string of the
current line as seen by the user, I have to manually iterate through
overlays, handle 'display manually and reassemble the line myself.
There's no buffer-substring equivalent that gives me the string "with
'display overlays" between two points.

Re: Getting column position accounting for overlays

[Eli Zaretskii]

> From: Yuri D'Elia <wavexx@thregr.org>
> Date: Wed, 5 Aug 2015 20:44:19 +0200
> 
> On 05/08/15 17:21, Eli Zaretskii wrote:
> >> I would almost have expected something to extract the visualized string
> >> between two points, like `buffer-substring' but iterating through
> >> overlays; but couldn't find any (did I miss it?)
> > 
> > The overlay strings are directly accessible via 'overlay-get', so why
> > would anyone need the feature you expected to exist?
> 
> I don't think we're on the same line of thought.

That's why I asked to describe your use case.  I'm afraid I still
don't think I understand it.

> The usage scenario is pretty simple: assume a terminal display.
> I want to determine the current /visual/ column at point.

Yes, but why do you need that?  What do you intend to do with the
value?

And does the solution I suggested fulfill your needs?

> The second part is a consequence: if I want to have a string of the
> current line as seen by the user, I have to manually iterate through
> overlays, handle 'display manually and reassemble the line myself.
> There's no buffer-substring equivalent that gives me the string "with
> 'display overlays" between two points.

Indeed, there is not, AFAIK.

But you originally asked only for the dimensions of the text on the
screen, not about the text itself.

Re: Getting column position accounting for overlays

[Yuri D'Elia]

On 05/08/15 21:01, Eli Zaretskii wrote:
> That's why I asked to describe your use case.  I'm afraid I still
> don't think I understand it.

Sometimes it's just curiosity ;)

>> The usage scenario is pretty simple: assume a terminal display.
>> I want to determine the current /visual/ column at point.
> 
> Yes, but why do you need that?  What do you intend to do with the
> value?

For example:

https://github.com/wavexx/rigid-tabs.el

Although this is just one case.

If you're trying to display anything related to the /visual/ appearance,
you'll have to consider the possibility of overlays and thus run into
the same issue.

If I'm not mistaken, there were a couple of other packages that handle
this (highlight-indentation?) if you use pretty-symbols or similar
modes, and the way you have to do that is not pretty.

> Indeed, there is not, AFAIK.
> 
> But you originally asked only for the dimensions of the text on the
> screen, not about the text itself.

Looks orthogonal to me.
I don't have a need for it right now, but somehow seems logical to have.

Re: Getting column position accounting for overlays

[Eli Zaretskii]

> From: Yuri D'Elia <wavexx@thregr.org>
> Date: Fri, 7 Aug 2015 19:13:58 +0200
> 
> On 05/08/15 21:01, Eli Zaretskii wrote:
> > That's why I asked to describe your use case.  I'm afraid I still
> > don't think I understand it.
> 
> Sometimes it's just curiosity ;)

Nothing wrong with that.

> >> The usage scenario is pretty simple: assume a terminal display.
> >> I want to determine the current /visual/ column at point.
> > 
> > Yes, but why do you need that?  What do you intend to do with the
> > value?
> 
> For example:
> 
> https://github.com/wavexx/rigid-tabs.el

If you need to align text on display, you should use the the
(space :align-to POS) display spec, see the node "Specified Space"
in the ELisp manual.  This lets the display engine do the job for
you, so you don't need to figure out the dimensions of the text
on display.

> If you're trying to display anything related to the /visual/ appearance,
> you'll have to consider the possibility of overlays and thus run into
> the same issue.

Can you envision other use cases, besides alignment?

Re: Getting column position accounting for overlays

[Yuri D'Elia]

On 07/08/15 20:19, Eli Zaretskii wrote:
>> https://github.com/wavexx/rigid-tabs.el
> 
> If you need to align text on display, you should use the the
> (space :align-to POS) display spec, see the node "Specified Space"
> in the ELisp manual.  This lets the display engine do the job for
> you, so you don't need to figure out the dimensions of the text
> on display.

Ah yes, but that's precisely the problem: I need to calculate POS
beforehand.

>> If you're trying to display anything related to the /visual/ appearance,
>> you'll have to consider the possibility of overlays and thus run into
>> the same issue.
> 
> Can you envision other use cases, besides alignment?

Looking for solutions, I came across a few snippets in emacswiki that
dump the content of the overlays into another buffer (sort of "save what
you see"). But all other cases were about calculating the current visual
column for alignment purposes.

It's not so trivial, considering that you basically have to handle all
'display cases manually to replicate what emacs does, you might have
overlapping overlays, and so on. I also think that it's bound to break
if 'display gets extended to support new properties.

Re: Getting column position accounting for overlays

[Eli Zaretskii]

> From: Yuri D'Elia <wavexx@thregr.org>
> Date: Sat, 8 Aug 2015 16:31:28 +0200
> 
> On 07/08/15 20:19, Eli Zaretskii wrote:
> >> https://github.com/wavexx/rigid-tabs.el
> > 
> > If you need to align text on display, you should use the the
> > (space :align-to POS) display spec, see the node "Specified Space"
> > in the ELisp manual.  This lets the display engine do the job for
> > you, so you don't need to figure out the dimensions of the text
> > on display.
> 
> Ah yes, but that's precisely the problem: I need to calculate POS
> beforehand.

Yes, of course.  But you will have the same problem if you do get the
text dimensions: you need to compare that with something to do useful
things with the value.  Right?

> >> If you're trying to display anything related to the /visual/ appearance,
> >> you'll have to consider the possibility of overlays and thus run into
> >> the same issue.
> > 
> > Can you envision other use cases, besides alignment?
> 
> Looking for solutions

I actually asked about the problems, not the solutions.

> I came across a few snippets in emacswiki that
> dump the content of the overlays into another buffer (sort of "save what
> you see"). But all other cases were about calculating the current visual
> column for alignment purposes.
> 
> It's not so trivial, considering that you basically have to handle all
> 'display cases manually to replicate what emacs does, you might have
> overlapping overlays, and so on. I also think that it's bound to break
> if 'display gets extended to support new properties.

It's simply impossible, except in a few special cases.  If we need
such a feature that should work everywhere, we will need to ask the
display engine to supply this information.

But I still am unsure when would this be needed.

Re: Getting column position accounting for overlays

[Yuri D'Elia]

On 08/08/15 16:45, Eli Zaretskii wrote:
>> Ah yes, but that's precisely the problem: I need to calculate POS
>> beforehand.
> 
> Yes, of course.  But you will have the same problem if you do get the
> text dimensions: you need to compare that with something to do useful
> things with the value.  Right?

Getting text dimensions was/is sort of a work-around.
If I could get displayed text easily, the iteration over overlays
wouldn't be needed: I could just take the string length.

Again, it would have problems on its own as we discussed, so far none of
the solutions work perfectly.

>> It's not so trivial, considering that you basically have to handle all
>> 'display cases manually to replicate what emacs does, you might have
>> overlapping overlays, and so on. I also think that it's bound to break
>> if 'display gets extended to support new properties.
> 
> It's simply impossible, except in a few special cases.  If we need
> such a feature that should work everywhere, we will need to ask the
> display engine to supply this information.
> 
> But I still am unsure when would this be needed.

To be more concrete, what about a (visual-column) function, that just
gives what's currently needed?

---

(visual-column &optional logical)

Return the horizontal position of point. Beginning of the *visual* line
is column 0. This is calculated by adding together the widths of all the
displayed representations of the characters, *including 'display
overlays*, between the start of the previous *visual* line and point
(e.g., control characters will have a width of 2 or 4, tabs
will have a variable width).

If logical is non-nil, count displayed characters since the beginning of
the current logical line.

(visual-column t) is similar to (current-column), with the difference
that overlays are taken into account.

---

This would be the ideal basis for any visual-alignment function.

Re: Getting column position accounting for overlays

[Eli Zaretskii]

> From: Yuri D'Elia <wavexx@thregr.org>
> Date: Sat, 8 Aug 2015 17:19:02 +0200
> 
> On 08/08/15 16:45, Eli Zaretskii wrote:
> >> Ah yes, but that's precisely the problem: I need to calculate POS
> >> beforehand.
> > 
> > Yes, of course.  But you will have the same problem if you do get the
> > text dimensions: you need to compare that with something to do useful
> > things with the value.  Right?
> 
> Getting text dimensions was/is sort of a work-around.
> If I could get displayed text easily, the iteration over overlays
> wouldn't be needed: I could just take the string length.

Take the string length and do what with it?

And please don't assume that given a string you can compute its
visual length easily: you can't, not with variable-pitch fonts.  And
then you have images, stretches, etc.

> Again, it would have problems on its own as we discussed, so far none of
> the solutions work perfectly.

Solutions to what problem(s)?

> To be more concrete, what about a (visual-column) function, that just
> gives what's currently needed?

I've already shown you a way to compute that, several message ago.
Let me repeat:

  (/ (car (window-text-pixel-size nil (line-beginning-position) (point)))
     (default-font-width))

> (visual-column &optional logical)
> 
> Return the horizontal position of point. Beginning of the *visual* line
> is column 0.                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ^^^^^^^^^^^

This is contrary to current-column conventions: it counts from the
beginning of the physical line, not visual line.  If you want to count
from the beginning of the visual line, replace line-beginning-position
above with something like

  (save-excursion (beginning-of-visual-line) (point))

Re: Getting column position accounting for overlays

[Yuri D'Elia]

On 08/08/15 17:43, Eli Zaretskii wrote:
>> To be more concrete, what about a (visual-column) function, that just
>> gives what's currently needed?
> 
> I've already shown you a way to compute that, several message ago.
> Let me repeat:
> 
>   (/ (car (window-text-pixel-size nil (line-beginning-position) (point)))
>      (default-font-width))

I didn't like this solution precisely because it doesn't rely on
character-cells units, and was explicitly looking for character
lengths/distances between arbitrary points.

If this calculation would be used in rigid-tabs for instance, I would
calculate different amount of spaces whether a variable-pitch font is
used or not, while my intention is much closer to (current-column).

But maybe I'm just arguing about nothing, and this is not really
something that can be generalized.

If you're using a variable pitch font, everything does down the drain
anyway.

Re: Getting column position accounting for overlays

[Eli Zaretskii]

> From: Yuri D'Elia <wavexx@thregr.org>
> Date: Sat, 8 Aug 2015 18:29:18 +0200
> 
> On 08/08/15 17:43, Eli Zaretskii wrote:
> >> To be more concrete, what about a (visual-column) function, that just
> >> gives what's currently needed?
> > 
> > I've already shown you a way to compute that, several message ago.
> > Let me repeat:
> > 
> >   (/ (car (window-text-pixel-size nil (line-beginning-position) (point)))
> >      (default-font-width))
> 
> I didn't like this solution precisely because it doesn't rely on
> character-cells units

Why do you say that?  Did you try it and saw problems?  If so, please
show the test cases where the above doesn't do what you want.

> and was explicitly looking for character lengths/distances between
> arbitrary points.

So you are saying that you want to count a double-width character as a
single character "cell" even though it will take up 2 columns on
display?  And you'd like to count a TAB as a single "cell" even though
it can take 1 to 8 columns on display?  Why would something like that
be useful?  It also goes against current-column, which you said was
the model for what you were looking.

> If this calculation would be used in rigid-tabs for instance, I would
> calculate different amount of spaces whether a variable-pitch font is
> used or not, while my intention is much closer to (current-column).

current-column does know that a TAB doesn't take a single column.  And
you said that variable-pitch fonts are not part of the problem yuou
want to solve anyway.  So now I'm confused regarding what you are
looking for.

> If you're using a variable pitch font, everything does down the drain
> anyway.

Not if you calculate dimensions in pixels, no.

Re: Getting column position accounting for overlays

[Yuri D'Elia]

On 08/08/15 18:48, Eli Zaretskii wrote:
>>>   (/ (car (window-text-pixel-size nil (line-beginning-position) (point)))
>>>      (default-font-width))
>>
>> I didn't like this solution precisely because it doesn't rely on
>> character-cells units
> 
> Why do you say that?  Did you try it and saw problems?  If so, please
> show the test cases where the above doesn't do what you want.

I didn't try indeed. I know it /will/ work for a fixed-pitch font, but I
was searching for alternatives and/or different semantics anyway.

I cannot tell if you're bothered by this (if so, it's not intentional).
I understood perfectly what you wrote/meant so far, so I don't think we
need to go on.