bug#18417: 24.3.93; posn-at-point confused by fill-column-indicator

bug#18417: 24.3.93; posn-at-point confused by fill-column-indicator

[Dmitry]

1. Download, put inside load-path and require fill-column-indicator.el,
from https://github.com/alpaker/Fill-Column-Indicator.
2. Open an empty buffer, add a newline inside it.
3. Toggle fcm-mode on. Notice the indicator at the end of the empty
line.
4. With point at the beginning of that line, evaluate

(posn-col-row (posn-at-point (point)))

and see it evaluate to (71 . 0).

The problem is probably not in posn-col-row, because (posn-x-y
(posn-at-point (point))) also returns a large non-zero value in car.

In GNU Emacs 24.3.93.3 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8)
 of 2014-09-05 on axl
Repository revision: 117482 monnier@iro.umontreal.ca-20140904161426-2072ebabqpyhaadw
Windowing system distributor `The X.Org Foundation', version 11.0.11501000
System Description:	Ubuntu 14.04.1 LTS

bug#18417: 24.3.93; posn-at-point confused by fill-column-indicator

[Eli Zaretskii]

> From: Dmitry <dgutov@yandex.ru>
> Date: Sat, 06 Sep 2014 13:43:00 +0400
> 
> 1. Download, put inside load-path and require fill-column-indicator.el,
> from https://github.com/alpaker/Fill-Column-Indicator.
> 2. Open an empty buffer, add a newline inside it.
> 3. Toggle fcm-mode on. Notice the indicator at the end of the empty
> line.     ^^^^^^^^

I assume you meant "fci-mode".

> 4. With point at the beginning of that line, evaluate
> 
> (posn-col-row (posn-at-point (point)))
> 
> and see it evaluate to (71 . 0).
> 
> The problem is probably not in posn-col-row, because (posn-x-y
> (posn-at-point (point))) also returns a large non-zero value in car.

This was previously reported and discussed here:

  http://lists.gnu.org/archive/html/emacs-devel/2013-07/msg00445.html

My correspondent there expressed interest in fixing this, but I never
heard from him about this since then, although I pointed him at
posn-at-point as the most probable culprit.

Anyway, I'm not going to lose sleep over this, sorry.  The recipe in
effect invokes undefined behavior in posn-at-point, because fci-mode
uses a zero-length (a.k.a. "empty") overlay to place, in a very
convoluted way, a stretch of whitespace followed by an image, before
the newline.  Since "at point" means "before point", and the recipe
asks for coordinates at the newline position, it is not clear whether
the reported coordinates should or shouldn't include the stretch
glyph.  The result you see is the consequence of the implementation
details of the functions that simulate display in order to compute
screen coordinates at a given buffer position.  Since the buffer
position of the newline is not "covered" by the empty overlay, Emacs
happily stops when it reaches the newline, oblivious to the fact that
on the way it produced the stretch glyph of a very large width.

If someone wants to work on this, patches will be welcome, of course.

bug#18417: 24.3.93; posn-at-point confused by fill-column-indicator

[Dmitry Gutov]

On 09/07/2014 09:22 PM, Eli Zaretskii wrote:

> I assume you meant "fci-mode".

Yes, sorry.

> Anyway, I'm not going to lose sleep over this, sorry.

Fair enough. fci-mode does some pretty complex stuff.

> The recipe in
> effect invokes undefined behavior in posn-at-point, because fci-mode
> uses a zero-length (a.k.a. "empty") overlay to place, in a very
> convoluted way, a stretch of whitespace followed by an image, before
> the newline.

If it's a problem, would you maybe suggest a better way to implement 
this feature, for the Fill-Column-Indicator author?

bug#18417: 24.3.93; posn-at-point confused by fill-column-indicator

[Eli Zaretskii]

> Date: Sun, 07 Sep 2014 22:55:34 +0400
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: 18417@debbugs.gnu.org
> 
> > The recipe in
> > effect invokes undefined behavior in posn-at-point, because fci-mode
> > uses a zero-length (a.k.a. "empty") overlay to place, in a very
> > convoluted way, a stretch of whitespace followed by an image, before
> > the newline.
> 
> If it's a problem, would you maybe suggest a better way to implement 
> this feature, for the Fill-Column-Indicator author?

I could try, but right now I don't feel I understand all the subtle
details and reasons that led to such an implementation.

There are 2 special characters, u+e000 and u+e001, each one has a
display-table setting to display it as something else, then there's an
empty overlay with an after-string made of these special characters
that has a display property with :align-to, followed by an image...
And I'm not sure I listed all of the "features".  (The 'cursor'
property I omitted intentionally, because I believe I know why they
are needed.)  Why each one of these is needed?

bug#18417: 24.3.93; posn-at-point confused by fill-column-indicator

[Alp Aker]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

> There are 2 special characters, u+e000 and u+e001, each one has a
> display-table setting to display it as something else, then there's an
> empty overlay with an after-string made of these special characters
> that has a display property with :align-to, followed by an image...
> And I'm not sure I listed all of the "features".  (The 'cursor'
> property I omitted intentionally, because I believe I know why they
> are needed.)  Why each one of these is needed?

At least some of that is cruft, as at one point I was trying to support all
of v22, v23, and v24, and there were various differences in the behavior of
overlay-strings (as well as the 'cursor property) among those versions.
I'll try to simplify the implementation and see if I can come up with
something that doesn't trigger any UB in the display engine.

[-- Attachment #2: Type: text/html, Size: 946 bytes --]

bug#18417: 24.3.93; posn-at-point confused by fill-column-indicator

[Eli Zaretskii]

> Date: Sun, 7 Sep 2014 15:52:34 -0400
> From: Alp Aker <alptekin.aker@gmail.com>
> Cc: Dmitry Gutov <dgutov@yandex.ru>, 18417@debbugs.gnu.org
> 
> > There are 2 special characters, u+e000 and u+e001, each one has a
> > display-table setting to display it as something else, then there's an
> > empty overlay with an after-string made of these special characters
> > that has a display property with :align-to, followed by an image...
> > And I'm not sure I listed all of the "features". (The 'cursor'
> > property I omitted intentionally, because I believe I know why they
> > are needed.) Why each one of these is needed?
> 
> At least some of that is cruft, as at one point I was trying to support all of
> v22, v23, and v24, and there were various differences in the behavior of
> overlay-strings (as well as the 'cursor property) among those versions.

That was my guess, yes.

> I'll try to simplify the implementation and see if I can come up
> with something that doesn't trigger any UB in the display engine.

Thank you.

bug#18417: 24.3.93; posn-at-point confused by fill-column-indicator

[Alp Aker]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

> The recipe in effect invokes undefined behavior in posn-at-point,
> because fci-mode uses a zero-length (a.k.a. "empty") overlay to
> place, in a very convoluted way, a stretch of whitespace followed
> by an image, before the newline.
[snip]
> Since the buffer position of the newline is not "covered" by the
> empty overlay, Emacs happily stops when it reaches the newline,
> oblivious to the fact that on the way it produced the stretch
> glyph of a very large width.

I'm not sure it's due to the overlay having zero length.  Here's a minimal
recipe that provokes the same behavior using a overlay of length 1
(covering the newline):

(progn
  (delete-all-overlays)
  (goto-char (point-min))
  (insert "\n")
  (setq o (make-overlay 1 2)
        s (propertize " "
                      'display '(space :align-to 10)
                      'cursor t))
  (overlay-put o 'before-string s)
  (goto-char (point-min))
  (setq col (car (posn-col-row (posn-at-point (point)))))
  (message (concat "Current column: " (number-to-string col))))

[-- Attachment #2: Type: text/html, Size: 1323 bytes --]

bug#18417: 24.3.93; posn-at-point confused by fill-column-indicator

[Eli Zaretskii]

> Date: Sun, 7 Sep 2014 23:11:35 -0400
> From: Alp Aker <alptekin.aker@gmail.com>
> Cc: Dmitry <dgutov@yandex.ru>, 18417@debbugs.gnu.org
> 
> > The recipe in effect invokes undefined behavior in posn-at-point,
> > because fci-mode uses a zero-length (a.k.a. "empty") overlay to
> > place, in a very convoluted way, a stretch of whitespace followed
> > by an image, before the newline.
> [snip]
> > Since the buffer position of the newline is not "covered" by the
> > empty overlay, Emacs happily stops when it reaches the newline,
> > oblivious to the fact that on the way it produced the stretch
> > glyph of a very large width.
> 
> I'm not sure it's due to the overlay having zero length. Here's a minimal
> recipe that provokes the same behavior using a overlay of length 1 (covering
> the newline):

You are right, the length of the overlay is not the issue here.  I
mentioned that for completeness, but my wording could indeed mislead
into thinking that the zero length is the culprit.

The actual problem is that overlays with before-strings and
after-strings never "cover" (as in "conceal" or "hide") anything.  The
buffer contents is still there, so Emacs gives you the coordinates of
the buffer position on the display, not of the first glyph produced
from the overlay string.

Btw, the same will happen with cursor positioning if you remove the
'cursor' property from the overlay string, and for the same reason.