bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Toby Cubitt]

Two separate issues involving zero-length overlays. By "zero-length", I 
mean an overlay that starts and ends at the same position. (I'm not 
knowledgeable enough about Emacs' c internals to work it out whether 
these issues are fundamentally related or not.)

Strictly speaking, neither of these issues are bugs, since the behaviour 
is consistent with the documentation. But it makes it difficult or 
impossible to do certain things in Emacs that one would expect to be 
able to do.

Both "bugs" have been present since version 21, including the current 
stable version and in recent CVS (20.0.60.1).


1. Overlay keymaps
==================

"Bug" description:
------------------
When a zero-length overlay has a keymap property, that keymap is never 
active, even if when the point is at the overlay's start (and end) 
position. This makes it impossible in Emacs to define a key binding that 
is only active at a single point in the buffer.


Steps to reproduce:
-------------------
(setq test-overlay (make-overlay 1 1))  ; any position will do
(overlay-put test-overlay 'keymap (make-sparse-keymap))
;; any key binding will do
(define-key (overlay-get test-overlay 'keymap) "a" 'end-of-line)

Move point to position 1. Type "a". The character "a" is inserted 
instead of the point moving to the end of the line.


Impact:
-------
Zero-length overlays are probably rarely used. But it is strange to be 
able to define a keybinding in Emacs that is active at two or more 
neighbouring point positions, but not at a single point position. At the 
very least, this is a missing feature.

When this behaviour *is* needed (as in my completion-UI and 
auto-overlays packages), it is extremely difficult to work-around this 
issue. In fact, it requires abandoning overlay keymaps entirely, and 
simulating this behaviour via minor-mode maps and an ugly hack to check 
for overlays, temporarily disable the minor-mode map if no overlay was 
found, look up the keybinding again, call whatever command it is now 
bound to it, then re-enable the minor-mode map.


Suggestions for possible fixes:
-------------------------------
a) Make overlay keymaps active when the point is at the start, end, or 
within the overlay (currently, it is only active when the next character 
is within the overlay). This seems unlikely to have significant impact 
on other parts of Emacs, since zero-length overlays are rarely used.

b) If a) is undesirable, why not make the behaviour depend on the 
overlay's front-advance and read-advance properties? If the zero-length 
overlay has null front-advance and non-null read-advance, then there is 
some logic in making its keymap active when point is at that position, 
since any character inserted at that point will be within the overlay 
after insertion.

c) Add another overlay property to control the keymap behaviour in the 
case of zero-length overlays. (This seems an ugly solution to me, since 
it involves adding a whole new property just to deal with a very rare 
edge-case.)


2. `overlays-at'
================

"Bug" description:
------------------
`overlays-at' never returns zero-length overlays.


Steps to reproduce:
-------------------
(make-overlay 1 1)  ; any position will do
(overlays-at 1)

Returns nil instead of the overlay from 1 to 1.


Impact:
-------
Again, zero-length overlays are probably rarely used. But this makes it 
impossible in Emacs to find a zero-length overlay using `overlays-at'. 
Instead, one has to work-around this using three calls to `overlays-in' 
then filter out overlays that shouldn't be in the list.

Overlays do fairly frequently become zero-length when the text they 
cover is deleted (depending on their front- and rear-advance 
properties), and this "bug" makes them suddenly disappear from the 
return value of `overlays-at' even though they still exist in the buffer.


Suggestions for possible fixes:
-------------------------------
a) Modify (overlays-at pos) to return zero-length overlays that start at 
pos (it already returns all other overlays that start at pos). Again, 
this seems unlikely to have significant impact on other parts of Emacs, 
since zero-length overlays are rarely used.

b) Modify (overlays-at pos) to return zero-length overlays that start at 
pos, and have a null front-advance and non-nil rear-advance property. 
(The logic for this is the same as in option b) for the overlay keymaps 
issue.)

c) Leave `overlays-at' unchanged, and define a new function 
`overlays-at-point' that implements either a) or b).


Toby Cubitt

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Toby Cubitt]

Looking through NEWS.22, I came upon the following:

*** The `keymap' property now also works at the ends of overlays and
text properties, according to their stickiness.  This also means that it
works with empty overlays.  The same hold for the `local-map' property.

So the first issue I reported is already fixed! (In exactly the way I 
suggested, which makes me suspect that future-bug might now have been 
fixed too... :) However, this behaviour isn't documented in the Emacs 
Lisp manual (unless I've overlooked something yet again), which *is* a bug.


I think the second issue still remains, though.

Toby



Toby Cubitt wrote:
[...]
> 1. Overlay keymaps
> ==================
> 
> "Bug" description:
> ------------------
> When a zero-length overlay has a keymap property, that keymap is never 
> active, even if when the point is at the overlay's start (and end) 
> position. This makes it impossible in Emacs to define a key binding that 
> is only active at a single point in the buffer.
[...]
 >
> Suggestions for possible fixes:
> -------------------------------
[...]
> b) If a) is undesirable, why not make the behaviour depend on the 
> overlay's front-advance and read-advance properties? If the zero-length 
> overlay has null front-advance and non-null read-advance, then there is 
> some logic in making its keymap active when point is at that position, 
> since any character inserted at that point will be within the overlay 
> after insertion.
[...]
> 
 >
> 2. `overlays-at'
> ================
> 
> "Bug" description:
> ------------------
> `overlays-at' never returns zero-length overlays.
> 
> 
> Steps to reproduce:
> -------------------
> (make-overlay 1 1)  ; any position will do
> (overlays-at 1)
> 
> Returns nil instead of the overlay from 1 to 1.
> 
> 
> Impact:
> -------
> Again, zero-length overlays are probably rarely used. But this makes it 
> impossible in Emacs to find a zero-length overlay using `overlays-at'. 
> Instead, one has to work-around this using three calls to `overlays-in' 
> then filter out overlays that shouldn't be in the list.
> 
> Overlays do fairly frequently become zero-length when the text they 
> cover is deleted (depending on their front- and rear-advance 
> properties), and this "bug" makes them suddenly disappear from the 
> return value of `overlays-at' even though they still exist in the buffer.
> 
> 
> Suggestions for possible fixes:
> -------------------------------
> a) Modify (overlays-at pos) to return zero-length overlays that start at 
> pos (it already returns all other overlays that start at pos). Again, 
> this seems unlikely to have significant impact on other parts of Emacs, 
> since zero-length overlays are rarely used.
> 
> b) Modify (overlays-at pos) to return zero-length overlays that start at 
> pos, and have a null front-advance and non-nil rear-advance property. 
> (The logic for this is the same as in option b) for the overlay keymaps 
> issue.)
> 
> c) Leave `overlays-at' unchanged, and define a new function 
> `overlays-at-point' that implements either a) or b).
> 
> 
> Toby Cubitt
> 

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Lars Ingebrigtsen]

Toby Cubitt <t.s.cubitt.98@cantab.net> writes:

> "Bug" description:
> ------------------
> `overlays-at' never returns zero-length overlays.
>
> Steps to reproduce:
> -------------------
> (make-overlay 1 1)  ; any position will do
> (overlays-at 1)
>
> Returns nil instead of the overlay from 1 to 1.

(I'm going through old bug reports that unfortunately wasn't resolved at
the time.)

This is still the case in Emacs 28.

> Impact:
> -------
> Again, zero-length overlays are probably rarely used. But this makes
> it impossible in Emacs to find a zero-length overlay using
> `overlays-at'. Instead, one has to work-around this using three calls
> to `overlays-in' then filter out overlays that shouldn't be in the
> list.

Or just `overlay-lists'.

> Overlays do fairly frequently become zero-length when the text they
> cover is deleted (depending on their front- and rear-advance
> properties), and this "bug" makes them suddenly disappear from the
> return value of `overlays-at' even though they still exist in the
> buffer.
>
> Suggestions for possible fixes:
> -------------------------------
> a) Modify (overlays-at pos) to return zero-length overlays that start
> at pos (it already returns all other overlays that start at
> pos). Again, this seems unlikely to have significant impact on other
> parts of Emacs, since zero-length overlays are rarely used.
>
> b) Modify (overlays-at pos) to return zero-length overlays that start
> at pos, and have a null front-advance and non-nil rear-advance
> property. (The logic for this is the same as in option b) for the
> overlay keymaps issue.)
>
> c) Leave `overlays-at' unchanged, and define a new function
> `overlays-at-point' that implements either a) or b).

Hm...  the issue here is that if a zero-length overlay has a
read-advance property, then it can influence what happens when you type
something at that position.  So b) makes sense to me on that level.

But I'd worry that it'd break code that's not aware of zero-length
overlays.  And introducing a new function doesn't seem very attractive.

Does anybody have any opinions here?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 19 Jul 2021 17:26:11 +0200
> Cc: 459@debbugs.gnu.org
> 
> > Suggestions for possible fixes:
> > -------------------------------
> > a) Modify (overlays-at pos) to return zero-length overlays that start
> > at pos (it already returns all other overlays that start at
> > pos). Again, this seems unlikely to have significant impact on other
> > parts of Emacs, since zero-length overlays are rarely used.
> >
> > b) Modify (overlays-at pos) to return zero-length overlays that start
> > at pos, and have a null front-advance and non-nil rear-advance
> > property. (The logic for this is the same as in option b) for the
> > overlay keymaps issue.)
> >
> > c) Leave `overlays-at' unchanged, and define a new function
> > `overlays-at-point' that implements either a) or b).
> 
> Hm...  the issue here is that if a zero-length overlay has a
> read-advance property, then it can influence what happens when you type
> something at that position.  So b) makes sense to me on that level.
> 
> But I'd worry that it'd break code that's not aware of zero-length
> overlays.  And introducing a new function doesn't seem very attractive.

We could add a new optional argument to overlays-at, to make it return
such overlays.  That would avoid the risk of breaking unsuspecting
callers.

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> > a) Modify (overlays-at pos) to return zero-length overlays that start
>> > at pos (it already returns all other overlays that start at
>> > pos). Again, this seems unlikely to have significant impact on other
>> > parts of Emacs, since zero-length overlays are rarely used.
>> >
>> > b) Modify (overlays-at pos) to return zero-length overlays that start
>> > at pos, and have a null front-advance and non-nil rear-advance
>> > property. (The logic for this is the same as in option b) for the
>> > overlay keymaps issue.)
>> >
>> > c) Leave `overlays-at' unchanged, and define a new function
>> > `overlays-at-point' that implements either a) or b).

[...]

> We could add a new optional argument to overlays-at, to make it return
> such overlays.  That would avoid the risk of breaking unsuspecting
> callers.

Yeah, that's true.  Do you have any opinion on whether a) or b) would
make the most sense if given this optional argument?  I'm leaning
towards b)...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: t.s.cubitt.98@cantab.net,  459@debbugs.gnu.org
> Date: Mon, 19 Jul 2021 18:37:07 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > a) Modify (overlays-at pos) to return zero-length overlays that start
> >> > at pos (it already returns all other overlays that start at
> >> > pos). Again, this seems unlikely to have significant impact on other
> >> > parts of Emacs, since zero-length overlays are rarely used.
> >> >
> >> > b) Modify (overlays-at pos) to return zero-length overlays that start
> >> > at pos, and have a null front-advance and non-nil rear-advance
> >> > property. (The logic for this is the same as in option b) for the
> >> > overlay keymaps issue.)
> >> >
> >> > c) Leave `overlays-at' unchanged, and define a new function
> >> > `overlays-at-point' that implements either a) or b).
> 
> [...]
> 
> > We could add a new optional argument to overlays-at, to make it return
> > such overlays.  That would avoid the risk of breaking unsuspecting
> > callers.
> 
> Yeah, that's true.  Do you have any opinion on whether a) or b) would
> make the most sense if given this optional argument?  I'm leaning
> towards b)...

b) is fine by me, but let's hear from Stefan as well.

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Lars Ingebrigtsen [2021-07-19 18:37:07] wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>>> > a) Modify (overlays-at pos) to return zero-length overlays that start
>>> > at pos (it already returns all other overlays that start at
>>> > pos). Again, this seems unlikely to have significant impact on other
>>> > parts of Emacs, since zero-length overlays are rarely used.
>>> >
>>> > b) Modify (overlays-at pos) to return zero-length overlays that start
>>> > at pos, and have a null front-advance and non-nil rear-advance
>>> > property. (The logic for this is the same as in option b) for the
>>> > overlay keymaps issue.)
>>> >
>>> > c) Leave `overlays-at' unchanged, and define a new function
>>> > `overlays-at-point' that implements either a) or b).
>
> [...]
>
>> We could add a new optional argument to overlays-at, to make it return
>> such overlays.  That would avoid the risk of breaking unsuspecting
>> callers.
>
> Yeah, that's true.  Do you have any opinion on whether a) or b) would
> make the most sense if given this optional argument?  I'm leaning
> towards b)...

I vote for a) with or without the requirement of a new optional arg.
The reason I vote for a) rather than b) is because AFAICT `overlays-at`
does not pay currently attention to from/rear advance properties, so it
would be weird to do that just for empty overlays.


        Stefan

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I vote for a) with or without the requirement of a new optional arg.
> The reason I vote for a) rather than b) is because AFAICT `overlays-at`
> does not pay currently attention to from/rear advance properties, so it
> would be weird to do that just for empty overlays.

That's an excellent point.

I'll have a look at doing a) tomorrow, then, unless we (in the mean
time) decide to just include these overlays without adding a new
parameter.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  459@debbugs.gnu.org,
>   t.s.cubitt.98@cantab.net
> Date: Mon, 19 Jul 2021 20:23:39 +0200
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > I vote for a) with or without the requirement of a new optional arg.
> > The reason I vote for a) rather than b) is because AFAICT `overlays-at`
> > does not pay currently attention to from/rear advance properties, so it
> > would be weird to do that just for empty overlays.
> 
> That's an excellent point.
> 
> I'll have a look at doing a) tomorrow, then, unless we (in the mean
> time) decide to just include these overlays without adding a new
> parameter.

No, please don't make that the default behavior.  It makes no sense to
change the default behavior of such a veteran API for the benefit of a
rare use case.  The gains are very small, and don't justify the risks.
An optional argument gets us the best of both worlds, so it's a clear
winner.

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> No, please don't make that the default behavior.  It makes no sense to
> change the default behavior of such a veteran API for the benefit of a
> rare use case.  The gains are very small, and don't justify the risks.
> An optional argument gets us the best of both worlds, so it's a clear
> winner.

I was just hedging in case somebody came up with a brilliant reason for
including them by default.  :-)

However...  after implementing this, I see that `overlays-in' is
something that exists?  And does include zero-length overlays?  *sigh*
So `(overlays-in 1 1)' is the answer to this bug report.

I've included the patch that won't be applied below for reference.
Instead I'll just mention `overlays-in' in the `overlays-at' doc string.

diff --git a/src/buffer.c b/src/buffer.c
index d3a5ffd149..93f3e3dbb6 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -2859,7 +2859,8 @@ DEFUN ("kill-all-local-variables", Fkill_all_local_variables,
 ptrdiff_t
 overlays_at (EMACS_INT pos, bool extend, Lisp_Object **vec_ptr,
 	     ptrdiff_t *len_ptr,
-	     ptrdiff_t *next_ptr, ptrdiff_t *prev_ptr, bool change_req)
+	     ptrdiff_t *next_ptr, ptrdiff_t *prev_ptr, bool change_req,
+	     bool include_zero_length)
 {
   ptrdiff_t idx = 0;
   ptrdiff_t len = *len_ptr;
@@ -2886,7 +2887,8 @@ overlays_at (EMACS_INT pos, bool extend, Lisp_Object **vec_ptr,
 	 so its start counts for PREV_PTR if it's before POS.  */
       if (prev < startpos && startpos < pos)
 	prev = startpos;
-      if (endpos == pos)
+      if (endpos == pos
+	  && (!include_zero_length || endpos != startpos))
 	continue;
       if (startpos <= pos)
 	{
@@ -2928,7 +2930,8 @@ overlays_at (EMACS_INT pos, bool extend, Lisp_Object **vec_ptr,
 	  break;
 	}
       ptrdiff_t endpos = OVERLAY_POSITION (end);
-      if (pos < endpos)
+      if (pos < endpos
+	  || (include_zero_length && pos == endpos && startpos == endpos))
 	{
 	  if (idx == len)
 	    {
@@ -4220,10 +4223,13 @@ DEFUN ("overlay-properties", Foverlay_properties, Soverlay_properties, 1, 1, 0,
 }
 
 \f
-DEFUN ("overlays-at", Foverlays_at, Soverlays_at, 1, 2, 0,
+DEFUN ("overlays-at", Foverlays_at, Soverlays_at, 1, 3, 0,
        doc: /* Return a list of the overlays that contain the character at POS.
-If SORTED is non-nil, then sort them by decreasing priority.  */)
-  (Lisp_Object pos, Lisp_Object sorted)
+If SORTED is non-nil, then sort them by decreasing priority.
+
+If INCLUDE-ZERO-LENGTH is non-nil, also include zero-length overlays
+at POS.  (These are otherwise excluded.)  */)
+  (Lisp_Object pos, Lisp_Object sorted, Lisp_Object include_zero_length)
 {
   ptrdiff_t len, noverlays;
   Lisp_Object *overlay_vec;
@@ -4241,7 +4247,7 @@ DEFUN ("overlays-at", Foverlays_at, Soverlays_at, 1, 2, 0,
   /* Put all the overlays we want in a vector in overlay_vec.
      Store the length in len.  */
   noverlays = overlays_at (XFIXNUM (pos), 1, &overlay_vec, &len,
-			   NULL, NULL, 0);
+			   NULL, NULL, 0, !NILP (include_zero_length));
 
   if (!NILP (sorted))
     noverlays = sort_overlays (overlay_vec, noverlays,
@@ -4317,7 +4323,7 @@ DEFUN ("next-overlay-change", Fnext_overlay_change, Snext_overlay_change,
      Store the length in len.
      endpos gets the position where the next overlay starts.  */
   noverlays = overlays_at (XFIXNUM (pos), 1, &overlay_vec, &len,
-			   &endpos, 0, 1);
+			   &endpos, 0, 1, false);
 
   /* If any of these overlays ends before endpos,
      use its ending point instead.  */
@@ -4364,7 +4370,7 @@ DEFUN ("previous-overlay-change", Fprevious_overlay_change,
      Store the length in len.
      prevpos gets the position of the previous change.  */
   overlays_at (XFIXNUM (pos), 1, &overlay_vec, &len,
-	       0, &prevpos, 1);
+	       0, &prevpos, 1, false);
 
   xfree (overlay_vec);
   return make_fixnum (prevpos);
diff --git a/src/buffer.h b/src/buffer.h
index 24e9c3fcbc..daa666248d 100644
--- a/src/buffer.h
+++ b/src/buffer.h
@@ -1145,7 +1145,8 @@ #define CHECK_FIXNUM_COERCE_MARKER(x) ((x) = make_fixnum (fix_position (x)))
 extern void compact_buffer (struct buffer *);
 extern void evaporate_overlays (ptrdiff_t);
 extern ptrdiff_t overlays_at (EMACS_INT, bool, Lisp_Object **,
-			      ptrdiff_t *, ptrdiff_t *, ptrdiff_t *, bool);
+			      ptrdiff_t *, ptrdiff_t *, ptrdiff_t *, bool,
+			      bool);
 extern ptrdiff_t sort_overlays (Lisp_Object *, ptrdiff_t, struct window *);
 extern void recenter_overlay_lists (struct buffer *, ptrdiff_t);
 extern ptrdiff_t overlay_strings (ptrdiff_t, struct window *, unsigned char **);
@@ -1204,13 +1205,13 @@ #define GET_OVERLAYS_AT(posn, overlays, noverlays, nextp, chrq)		\
     ptrdiff_t maxlen = 40;						\
     SAFE_NALLOCA (overlays, 1, maxlen);					\
     (noverlays) = overlays_at (posn, false, &(overlays), &maxlen,	\
-			       nextp, NULL, chrq);			\
+			       nextp, NULL, chrq, false);		\
     if ((noverlays) > maxlen)						\
       {									\
 	maxlen = noverlays;						\
 	SAFE_NALLOCA (overlays, 1, maxlen);				\
 	(noverlays) = overlays_at (posn, false, &(overlays), &maxlen,	\
-				   nextp, NULL, chrq);			\
+				   nextp, NULL, chrq, false);		\
       }									\
   } while (false)
 
diff --git a/test/src/buffer-tests.el b/test/src/buffer-tests.el
index 2adffc024a..5e83c0beab 100644
--- a/test/src/buffer-tests.el
+++ b/test/src/buffer-tests.el
@@ -1386,4 +1386,17 @@ buffer-tests-inhibit-buffer-hooks-indirect
           (when (buffer-live-p base)
             (kill-buffer base)))))))
 
+(ert-deftest zero-length-overlays-and-not ()
+  (with-temp-buffer
+    (insert "hello")
+    (let ((long-overlay (make-overlay 2 4))
+          (zero-overlay (make-overlay 3 3)))
+      ;; Exclude.
+      (should (= (length (overlays-at 3)) 1))
+      (should (eq (car (overlays-at 3)) long-overlay))
+      ;; Include.
+      (should (= (length (overlays-at 3 nil t)) 2))
+      (should (memq long-overlay (overlays-at 3 nil t)))
+      (should (memq zero-overlay (overlays-at 3 nil t))))))
+
 ;;; buffer-tests.el ends here


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: 459@debbugs.gnu.org,  monnier@iro.umontreal.ca,  t.s.cubitt.98@cantab.net
> Date: Tue, 20 Jul 2021 13:28:10 +0200
> 
> However...  after implementing this, I see that `overlays-in' is
> something that exists?  And does include zero-length overlays?  *sigh*
> So `(overlays-in 1 1)' is the answer to this bug report.

Oops.  For some reason, I was under the impression that the OP tried
overlays-in as well.  I now see that I've just imagined that.  Sorry.

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Toby Cubitt]

On Tue, Jul 20, 2021 at 02:50:20PM +0300, Eli Zaretskii wrote:
> > From: Lars Ingebrigtsen <larsi@gnus.org>
> > Cc: 459@debbugs.gnu.org,  monnier@iro.umontreal.ca,  t.s.cubitt.98@cantab.net
> > Date: Tue, 20 Jul 2021 13:28:10 +0200
> >
> > However...  after implementing this, I see that `overlays-in' is
> > something that exists?  And does include zero-length overlays?  *sigh*
> > So `(overlays-in 1 1)' is the answer to this bug report.
>
> Oops.  For some reason, I was under the impression that the OP tried
> overlays-in as well.  I now see that I've just imagined that.  Sorry.

The OP on this was so much younger when he submitted the bug report, he might as well have been a different person :)

At this length of time, I can't remember for sure. But it looks like I used overlays-in as a workaround in the auto-overlays package (where this bug report originated). So I guess I did try it. In which case, apologies from my 15-years-ago self for not noting this in the bug report. Maybe the bug report was more about the API doing something inconsistent and surprising, which looked like a bug rather than deliberate?

Still seems surprising to me that overlays-at won't return all overlays "at" point, but overlays-in will return overlays that technically aren't "in" the specified region. (The emptyset canonically contains no elements.) I wonder if the proposed change to overlays-at would really break anything? Seems like a case where adding an optional argument or some such, deprecating the current default behaviour, issuing a warning for x decades, then making the change to overlays-in, would clean up the API here.

But I bow to your judgement on the cost-benefit of backwards-compatibility versus API ugliness.

I do remember that the first half of this bug report, about the interacting between overlay keymaps and zero-length overlays, was more significant. I included the overlays-at part of the bug report, as I thought it might be related (as I wrote back then). The fact that keymap properties of zero-length overlays do not apply, irrespective of the front/read-advance properties, means it's impossible to bind a key at a single point location in Emacs. It required an ugly kludge to work around this in the auto-overlays package.

This first bug in the report still seems to be there in current stable Emacs. Is the intention to dismiss that first bug as "won't fix", too? Just flagging, in case that part got missed.

Best wishes,
Toby
--
Dr T. S. Cubitt
Reader (Associate Professor) in Quantum Information
Royal Society University Research Fellow
Department of Computer Science
University College London

email: tsc25@cantab.net
web:   www.dr-qubit.org

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Lars Ingebrigtsen]

Toby Cubitt <toby@dr-qubit.org> writes:

> The OP on this was so much younger when he submitted the bug report,
> he might as well have been a different person :)

:-)

> Still seems surprising to me that overlays-at won't return all
> overlays "at" point, but overlays-in will return overlays that
> technically aren't "in" the specified region.

Yes, it's pretty surprising, so I've noted this in the `overlays-at' doc
string.

> This first bug in the report still seems to be there in current stable
> Emacs. Is the intention to dismiss that first bug as "won't fix", too?
> Just flagging, in case that part got missed.

I didn't check that bit, since you said:

----
Looking through NEWS.22, I came upon the following:

*** The `keymap' property now also works at the ends of overlays and
text properties, according to their stickiness.  This also means that it
works with empty overlays.  The same hold for the `local-map' property.

So the first issue I reported is already fixed! (In exactly the way I
suggested, which makes me suspect that future-bug might now have been
fixed too... :) However, this behaviour isn't documented in the Emacs
Lisp manual (unless I've overlooked something yet again), which *is* a
bug.
----

Oh, I missed the bit about the missing documentation?  Is that what
you're referring to?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Still seems surprising to me that overlays-at won't return all
> overlays "at" point,

FWIW, I see nothing in the code suggesting that this was a deliberate
decision.  It looks much more like an accident (which I hold to be
a misfeature).


        Stefan

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Toby Cubitt]

On Tue, Jul 20, 2021 at 03:46:38PM +0200, Lars Ingebrigtsen wrote:
> > This first bug in the report still seems to be there in current stable
> > Emacs. Is the intention to dismiss that first bug as "won't fix", too?
> > Just flagging, in case that part got missed.
>
> I didn't check that bit, since you said:
>
> ----
> Looking through NEWS.22, I came upon the following:
>
> *** The `keymap' property now also works at the ends of overlays and
> text properties, according to their stickiness.  This also means that it
> works with empty overlays.  The same hold for the `local-map' property.
>
> So the first issue I reported is already fixed! (In exactly the way I
> suggested, which makes me suspect that future-bug might now have been
> fixed too... :) However, this behaviour isn't documented in the Emacs
> Lisp manual (unless I've overlooked something yet again), which *is* a
> bug.
> ----

No, I'd also forgotten that I wrote that :) I did test before emailing, but didn't test with different stickiness. Making it stickiness-dependent makes sense.

> Oh, I missed the bit about the missing documentation?  Is that what
> you're referring to?

I suppose if it was documented, I might have remembered that it was already fixed :) Maybe it is somewhere, but I missed it? The Emacs Lisp manual currently (Emacs stable) has this to say on the overlay keymap property:

"This keymap is used when the character after point is within the overlay"

Which is technically incorrect since that NEWS.22 item, as it doesn't cover zero-length, null front-advance, non-null rear-advance overlays. A concise, technically correct wording that I think covers all cases would be:

"This keymap is used when a character inserted after point would be included within the overlay"

But maybe it would be better to be less concise, and spell out the zero-length overlay special case separately.

Best,
Toby
--
Dr T. S. Cubitt
Reader (Associate Professor) in Quantum Information
Royal Society University Research Fellow
Department of Computer Science
University College London

email: tsc25@cantab.net
web:   www.dr-qubit.org

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Lars Ingebrigtsen]

Toby Cubitt <toby@dr-qubit.org> writes:

> I suppose if it was documented, I might have remembered that it was
> already fixed :) Maybe it is somewhere, but I missed it? The Emacs
> Lisp manual currently (Emacs stable) has this to say on the overlay
> keymap property:
>
> "This keymap is used when the character after point is within the overlay"
>
> Which is technically incorrect since that NEWS.22 item, as it doesn't
> cover zero-length, null front-advance, non-null rear-advance
> overlays. A concise, technically correct wording that I think covers
> all cases would be:
>
> "This keymap is used when a character inserted after point would be
> included within the overlay"
>
> But maybe it would be better to be less concise, and spell out the
> zero-length overlay special case separately.

It's not just zero-length overlays -- any overlay with rear-advance has
this keymap effect on the character after the end of the overlay.  And
in addition, zero-length overlays have to have front-advance nil for
this to happen.  :-/

I've now mentioned that these properties have an effect in the overlay
keymap item, but punted to the "Managing Overlays" section to actually
explain how these overlays behave, because it's too complicated to
repeat in that item.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#459: Zero-length overlays, overlay keymaps, and `overlays-at'

[Chong Yidong]

> `overlays-at' never returns zero-length overlays.

I think the way to fix this is to add an optional argument to
overlays-at telling it to include zero-length overlays.  However, this
will have to wait till after the release.