bug#4854: 23.1.50; before-string overlay and show-paren-mode

bug#4854: 23.1.50; before-string overlay and show-paren-mode

[Stephen Berman]

Start Emacs with -Q, insert the following text into a buffer:

begin
(test)
end

then eval the following:

(defun my-test ()
  ""
  (interactive)
  (widen)
  (goto-char (point-min))
  (let ((begin (search-forward "begin")))
    (search-forward "end")
    (narrow-to-region (1+ begin) (line-beginning-position))
    (goto-char (point-min)))
  (unless (let ((ovlist (overlays-in (1- (point)) (1+ (point)))))
	    (when ovlist (overlay-get (car ovlist) 'before-string)))
    (let ((ov (make-overlay (point) (point) nil t t)))
      (overlay-put ov 'before-string "* "))))

(global-set-key "\C-cy" 'my-test)

and now type `C-c y' in the buffer with the inserted text.  The buffer
nows displays this:

* (test)

As expected, now repeating `C-c y' does not alter the display, i.e. does
not add more overlays.  Now enable show-paren-mode (e.g. by checking
"Paren Match Highlighting" in the Options menu).  Now repeating `C-c y'
display one additional "* " for each repetition (provided point remains
on `(').  Surprisingly, however, typing `M-x my-test' repeatedly does
not add any "* ".  In addition, typing a self-inserting character and
then deleting it has the effect that typing `C-c y' no longer adds "* ";
except after widening and removing the overlays: then repeating `C-c y'
adds "* " once only, as when show-paren-mode is disabled.

If the described behavior after enabling show-paren-mode is not a bug,
can anyone explain it?



In GNU Emacs 23.1.50.1 (i686-pc-linux-gnu, GTK+ Version 2.14.4)
 of 2009-10-27 on escher
Windowing system distributor `The X.Org Foundation', version 11.0.10502000
Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=local
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

bug#4854: 23.1.50; before-string overlay and show-paren-mode

[Stephen Berman]

Here's a slightly simpler test, which has partly different behavior from
the one I previously posted.

1. emacs -Q
2. eval this:
    (defun my-test ()
      (interactive)
      (unless (let ((ovlist (overlays-in (1- (point)) (1+ (point)))))
    	        (when ovlist (overlay-get (car ovlist) 'before-string)))
        (let ((ov (make-overlay (point) (point) nil)))
          (overlay-put ov 'before-string "* "))))
3. eval this:
    (global-set-key "\C-cy" 'my-test)
4. Insert the string "(test)" into a buffer, put the cursor on `('.
5. Type `C-c y' and/or `M-x my-test' any number of times: the overlay is
   added once only due to `unless', i.e. this is displayed:
   * (test)
6. Enable show-paren-mode
7. With the cursor still on `(', type `C-c y' repeatedly
=> each repetition adds an overlay, e.g. after three repetitions this is
   displayed (starting from the preceding display):
   * * * * (test)
8. With the cursor still on `(' and show-paren-mode still enabled, type
   `M-x my-test' repeatedly 
=> no overlays are added, i.e. the display remains unchanged

These two things seem very strange: (i) enabling show-paren-mode and
getting the show-paren overlay somehow nullifies the unless test; (ii)
this effect only happens when the command is invoked via a key sequence.

The result after step 7 seems to be a reproducible heisenbug, in that
stepping through my-test (called by typing `C-c y') with edebug does not
result in more than one overlay being added.  Also, unlike in my OP with
narrowing, with the above my-test, typing and deleting a self-inserting
character does not prevent repetitions of `C-c y' from adding more
overlays.

If anyone has any idea what's going on here, or how to debug it, I'd be
grateful.

Steve Berman

bug#4854: 23.1.50; before-string overlay and show-paren-mode

[npostavs]

severity 4854 minor
tags 4854 notabug
quit

A simpler test is

1. emacs -Q

2. eval

(defun show-overlays ()
  (interactive)
  (message "ovlist %S"
           (mapcar (lambda (o)
                     (cons o (overlay-properties o)))
                   (overlays-in (1- (point)) (1+ (point))))))

(global-set-key "\C-cs" 'show-overlays)

3. M-x show-paren-mode

4. with point on a parenthesis

C-c s gives

ovlist ((#<overlay from 730 to 731 in *scratch*> face show-paren-match priority 1000))

M-x show-overlays gives

ovlist nil

Stephen Berman <stephen.berman@gmx.net> writes:
> These two things seem very strange: (i) enabling show-paren-mode and
> getting the show-paren overlay somehow nullifies the unless test; (ii)
> this effect only happens when the command is invoked via a key
> sequence.

It's because show-paren-mode uses a single (pair of) overlay(s) for all
buffers and moves it to right place during idle time.  When you invoke a
command with M-x the overlay pair gets moved to the minibuffer.  With a
direct keybinding the overlay pair stays in the current buffer.  This
can be demonstrated by adding this advice (additional to the code
evaluated above):

(defun debug-show-paren-mode (&rest _args)
  (message "show-paren-function in buffer %S" (current-buffer)))
(advice-add 'show-paren-function :before #'debug-show-paren-mode)

When using C-c s:

show-paren-function in buffer #<buffer *scratch*> [4 times]
ovlist ((#<overlay from 506 to 507 in *scratch*> face show-paren-match priority 1000))

When using M-x show-overlays:

show-paren-function in buffer #<buffer  *Minibuf-1*> [5 times]
ovlist nil
You can run the command ‘show-overlays’ with C-c s

So I think this is not a bug, it's just how show-paren-mode works.

bug#4854: 23.1.50; before-string overlay and show-paren-mode

[Stephen Berman]

On Fri, 01 Jul 2016 14:42:30 -0400 npostavs@users.sourceforge.net wrote:

[...]
> Stephen Berman <stephen.berman@gmx.net> writes:
>> These two things seem very strange: (i) enabling show-paren-mode and
>> getting the show-paren overlay somehow nullifies the unless test; (ii)
>> this effect only happens when the command is invoked via a key
>> sequence.
>
> It's because show-paren-mode uses a single (pair of) overlay(s) for all
> buffers and moves it to right place during idle time.  When you invoke a
> command with M-x the overlay pair gets moved to the minibuffer.  With a
> direct keybinding the overlay pair stays in the current buffer.

Thanks for the diagnosis, which is convincing.

> So I think this is not a bug, it's just how show-paren-mode works.

I guess you're right; yet this behavior seems to go against the spirit
of the way Emacs is intended to work, as suggested by the Emacs manual,
node `M-x': "Every Emacs command has a name that you can use to run it.
For convenience, many commands also have key bindings.  You can run
those commands by typing the keys, or run them by name.  Most Emacs
commands have no key bindings, so the only way to run them is by
name. [...]  Note that ‘forward-char’ is the same command that you
invoke with the key ‘C-f’.  The existence of a key binding does not stop
you from running the command by name."  Are there any other Emacs
commands that produce different behavior depending on whether they are
invoked with a key binding or by name?  Maybe it would be better to
unify the behavior of show-paren-mode.  (I briefly tried conditioning
the calls to move-overlay on whether the current buffer is a minibuffer,
but my attempt (which I didn't give much thought to) didn't work.)

I guess this bug can be closed, unless leaving it open might spur
someone to try and "improve" show-paren-mode.

Steve Berman

bug#4854: 23.1.50; before-string overlay and show-paren-mode

[npostavs]

Stephen Berman <stephen.berman@gmx.net> writes:

> On Fri, 01 Jul 2016 14:42:30 -0400 npostavs@users.sourceforge.net wrote:
>
> [...]
>> Stephen Berman <stephen.berman@gmx.net> writes:
>>> These two things seem very strange: (i) enabling show-paren-mode and
>>> getting the show-paren overlay somehow nullifies the unless test; (ii)
>>> this effect only happens when the command is invoked via a key
>>> sequence.
>>
>> It's because show-paren-mode uses a single (pair of) overlay(s) for all
>> buffers and moves it to right place during idle time.  When you invoke a
>> command with M-x the overlay pair gets moved to the minibuffer.  With a
>> direct keybinding the overlay pair stays in the current buffer.
>
> Thanks for the diagnosis, which is convincing.
>
>> So I think this is not a bug, it's just how show-paren-mode works.
>
> I guess you're right; yet this behavior seems to go against the spirit
> of the way Emacs is intended to work, as suggested by the Emacs manual,
> node `M-x': "Every Emacs command has a name that you can use to run it.
> For convenience, many commands also have key bindings.  You can run
> those commands by typing the keys, or run them by name.  Most Emacs
> commands have no key bindings, so the only way to run them is by
> name. [...]  Note that ‘forward-char’ is the same command that you
> invoke with the key ‘C-f’.  The existence of a key binding does not stop
> you from running the command by name."  Are there any other Emacs
> commands that produce different behavior depending on whether they are
> invoked with a key binding or by name?

self-insert-command ;)

You can fix your my-test command by changing the condition:

(defun my-test ()
  ...
  (unless (cl-some (lambda (o) (overlay-get o 'before-string))
                   (overlays-in (1- (point)) (1+ (point))))
    ...))

>                                         
>                                         Maybe it would be better to
> unify the behavior of show-paren-mode.  (I briefly tried conditioning
> the calls to move-overlay on whether the current buffer is a minibuffer,
> but my attempt (which I didn't give much thought to) didn't work.)

Well, it's possible to avoid moving overlays to minibuffer, but then
show-paren-mode stops working in the minibuffer, which I don't think is
so great.

diff --git i/lisp/paren.el w/lisp/paren.el
index 53eb500..1e4942f 100644
--- i/lisp/paren.el
+++ w/lisp/paren.el
@@ -236,9 +236,10 @@ show-paren--default
 ;; Find the place to show, if there is one,
 ;; and show it until input arrives.
 (defun show-paren-function ()
-  (let ((data (and show-paren-mode (funcall show-paren-data-function))))
+  (let ((data (and show-paren-mode (not (minibufferp))
+                   (funcall show-paren-data-function))))
     (if (not data)
-        (progn
+        (unless (minibufferp)
           ;; If show-paren-mode is nil in this buffer or if not at a paren that
           ;; has a match, turn off any previous paren highlighting.
           (delete-overlay show-paren--overlay)

With that patch, the original my-test keeps inserting more overlays
whether it's called by keybinding or M-x.

>
> I guess this bug can be closed, unless leaving it open might spur
> someone to try and "improve" show-paren-mode.

Not sure if there's anything to do.

bug#4854: 23.1.50; before-string overlay and show-paren-mode

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Date: Sun, 03 Jul 2016 11:36:01 -0400
> Cc: 4854@debbugs.gnu.org
> 
> Well, it's possible to avoid moving overlays to minibuffer, but then
> show-paren-mode stops working in the minibuffer, which I don't think is
> so great.

To say the least.  Please don't, I edit expressions in the minibuffer
quite a lot, and find show-paren invaluable in general and there in
particular.

bug#4854: 23.1.50; before-string overlay and show-paren-mode

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: npostavs@users.sourceforge.net
>> Date: Sun, 03 Jul 2016 11:36:01 -0400
>> Cc: 4854@debbugs.gnu.org
>> 
>> Well, it's possible to avoid moving overlays to minibuffer, but then
>> show-paren-mode stops working in the minibuffer, which I don't think is
>> so great.
>
> To say the least.  Please don't, I edit expressions in the minibuffer
> quite a lot, and find show-paren invaluable in general and there in
> particular.

Don't worry, I wasn't intending to.  That patch was just for
demonstration.  I suppose it might be possible to have paren overlay
per-buffer, so that the overlay doesn't disappear and reappear all the
time, but I don't think it's worth the complication.  I'm pretty sure it
would remain possible to write a command that's able to observe some
inconsistencies regardless: since show-paren-mode relies on idle-timers,
the behaviour is affected by the timing of the user's keystrokes.  It's
best to write commands that are oblivious to what show-paren-mode does
(e.g., the (cl-some ...) condition I posted in that same message).

bug#4854: 23.1.50; before-string overlay and show-paren-mode

[Stephen Berman]

On Sun, 03 Jul 2016 11:58:16 -0400 npostavs@users.sourceforge.net wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: npostavs@users.sourceforge.net
>>> Date: Sun, 03 Jul 2016 11:36:01 -0400
>>> Cc: 4854@debbugs.gnu.org
>>> 
>>> Well, it's possible to avoid moving overlays to minibuffer, but then
>>> show-paren-mode stops working in the minibuffer, which I don't think is
>>> so great.
>>
>> To say the least.  Please don't, I edit expressions in the minibuffer
>> quite a lot, and find show-paren invaluable in general and there in
>> particular.
>
> Don't worry, I wasn't intending to.  That patch was just for
> demonstration.  

I completely agree that preventing show-paren-mode from working in the
minibuffer is out of the question.

>                 I suppose it might be possible to have paren overlay
> per-buffer, so that the overlay doesn't disappear and reappear all the
> time, but I don't think it's worth the complication.  I'm pretty sure it
> would remain possible to write a command that's able to observe some
> inconsistencies regardless: since show-paren-mode relies on idle-timers,
> the behaviour is affected by the timing of the user's keystrokes.  

It's probably not worth putting much effort into trying to change it,
though I still think it's surprising, and not unlikely an unintended
consequence of the implementation of show-paren-mode, that the effect of
invoking a command that interacts with it differs depending on how it's
invoked (self-insert-command is really a special case, I think).

>                                                                    It's
> best to write commands that are oblivious to what show-paren-mode does
> (e.g., the (cl-some ...) condition I posted in that same message).

This is certainly sound advice, and indeed I applied it long ago in
dealing with the issue (in todo-mode.el) that prompted this bug report.

Again, thanks for the feedback and clarifying the issue.  I'm closing
this bug now.

Steve Berman