bug#49629: 27.2; electric-pair-mode doesn't work for angle brackets in HTML file

[Allen Li <darkfeline@felesatra.moe>]

On Wed, Jul 28, 2021 at 3:42 PM Lars Ingebrigtsen <larsi@gnus.org> wrote:
>
> Allen Li <darkfeline@felesatra.moe> writes:
>
> > I tried some printf debugging and I noticed that the bug goes away
> > when I add (message "%S" (buffer-substring-no-properties (point-min)
> > (point-max))) to the top of electric-pair--balance-info.  Maybe I'm
> > being dense, but I think that expression should not have this kind of
> > side effect.
>
> It should not, but it sounds like whatever's going on might be redisplay
> dependent?  In which case it might be better to say
>
> (push (buffer-string) my-var)
>
> at strategic places in the code instead of messaging, and then examining
> my-var afterwards.

I can't reproduce my previous fix of adding `(message "%S"
(buffer-substring-no-properties (point-min) (point-max)))` to the top
of `electric-pair--balance-info`.  At the time, I had many more
`message`s scattered throughout the function, but I could reliably
make the bug disappear and reappear by adding/removing this particular
`message`.  Of course, I don't remember exactly what `message`s I had
at the time.

Thus, I resorted to actually trying to understand the code.  I've
tracked down the bug to unexpected behavior from the `scan-sexps` call
in this part of `electric-pair--balance-info`:

         (ended-prematurely-fn
          ;; called when `scan-sexps' crashed against a parenthesis
          ;; pointing opposite the direction of travel.  After
          ;; traversing that character, the idea is to travel one sexp
          ;; in the opposite direction looking for a matching
          ;; delimiter.
          #'(lambda ()
              (let* ((pos (point))
                     (matched
                      (save-excursion
                        (cond ((< direction 0)
                               (condition-case err
                                   (eq (char-after pos)
                                       (electric-pair--with-uncached-syntax
                                           (table)
                                         (debug) ;; Added by me
                                         (matching-paren
                                          (char-before
                                           (scan-sexps (point) 1)))))
                                 (scan-error nil)))

The state of my test buffer at this point is `^<p>` where ^ is point.
The buffer starts as `<p^>`, and like the comment states, we move
point past the `<` "pointing opposite the direction of travel" and
look for a match for the opening `<`.

However, the `(scan-sexps (point) 1)` here signals `Scan error:
"Unbalanced parentheses", 1, 4`.  Again, the buffer state here is
`^<p>`.  Since `scan-sexps` is C, I haven't dug into it yet, but it
seems like it should work.  I also checked `(matching-paren ?<)` and
`(matching-paren ?>)` here to see if the syntax table recognizes the
angle brackets and it appears to do so, so I don't know why
`scan-sexps` is complaining.

While I don't know how the `syntax-ppss` cache works, I tried added an
extra `syntax-ppss-flush-cache` call to the start of the
`electric-pair--with-uncached-syntax`, but that doesn't help.

My current hypothesis is some odd behavior in how setting the syntax
table works.