backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[ndame via Users list for the GNU Emacs text editor]

I sometimes run into the situation with huge elisp lists, that I cannot use backward-sexp, because it throws an error, but forward-sexp works on the same list. Then, after a while it fixes itself without changing the buffer and backwards-sexp works too. I suspect scrolling through the file fixes it, so it might be connected to fontification? Though it's just a guess, I don't see how they could be related.

Is there a known situation when the same elisp list can be navigated forward, but not backward?

I'm on Emacs 27.1. Was there a related bug fixed in 28?

I cannot yet replicate the problem knowingly, it just occurs every once in a while.

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[ndame via Users list for the GNU Emacs text editor]

[-- Attachment #1: Type: text/plain, Size: 313 bytes --]

> I suspect scrolling through the file fixes it, so it might be connected to fontification?

Looks like it is connected to fontification, because I browsed through the items of the list and at one point it switches to string syntax for no reason, and I guess that causes the problem.

See the attached screenshot.

[-- Attachment #2: Capture.PNG --]
[-- Type: image/png, Size: 2394 bytes --]

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[Stefan Monnier via Users list for the GNU Emacs text editor]

ndame via Users list for the GNU Emacs text editor [2022-01-12 17:33:12] wrote:
>> I suspect scrolling through the file fixes it, so it might be connected to fontification?
> Looks like it is connected to fontification, because I browsed through the
> items of the list and at one point it switches to string syntax for no
> reason, and I guess that causes the problem.
>
> See the attached screenshot.

Sounds like a bug in the `syntax-propertize-function` of the major mode
you're using (or otherwise a bug elsewhere in that mode where it touches
the `syntax-table` text property).

You might want to open a bug report (against the major mode) and give as
much info as you can (of course a reproducible recipe is ideal, but it
might be tricky to come up with one).


        Stefan

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[ndame via Users list for the GNU Emacs text editor]

> I cannot yet replicate the problem knowingly, it just occurs every once in a while.

Now I can replicate the problem with my own config, but not with -Q

Which is strange, because I don't use any package for changing the font locking for emacs-lisp-mode, so it must be some other package doing this. I'll keep looking and post if I have more info.

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[ndame via Users list for the GNU Emacs text editor]

> I don't use any package for changing the font locking for emacs-lisp-mode, so it must be some other package doing this. I'll keep looking and post if I have more info.

Apparently, aggressive-indent-mode is the culprit. I can replicate the problem with -Q and aggressive-indent-mode turned on. Which is strange, because AFAIK aggressive-indent-mode only uses sexp moving and indent functions.

Should I report this to the mode author or emacs?

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[Emanuel Berg via Users list for the GNU Emacs text editor]

ndame via Users list for the GNU Emacs text editor wrote:

> Should I report this to the mode author or emacs?

If the mode is part of vanilla GNU Emacs (i.e. no packages or
whatever) then use `report-emacs-bug' ... if you happen to
find e-mails to the developers you can CC the bug report to
them, or wait for the bug number and send that to them.

-- 
underground experts united
https://dataswamp.org/~incal

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[ndame via Users list for the GNU Emacs text editor]

> Apparently, aggressive-indent-mode is the culprit. I can replicate the problem with -Q and aggressive-indent-mode turned on.

Reported the problem to aggressive-indent-mode author with steps how to reproduce:

https://github.com/Malabarba/aggressive-indent-mode/issues/153

We'll see what he can say about it.

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[ndame]

>
> Reported the problem to aggressive-indent-mode author with steps how to reproduce:
>
> https://github.com/Malabarba/aggressive-indent-mode/issues/153
>
> We'll see what he can say about it.


The maintainer is not really responsive. I skimmed the code and
it seems it only calls (indent-according-to-mode) and stuff,
and does not do anything crazy:

https://github.com/Malabarba/aggressive-indent-mode/blob/master/aggressive-indent.el

Would you agree that if the code does only automatic indentation
for the user by calling core indent functions then it's likely
that emacs is the culprit for the error and I should submit
a bug report  for emacs?

The problem is reproducible and it only requires installing
aggressive-indent-mode which is in GNU ELPA.

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[Emanuel Berg via Users list for the GNU Emacs text editor]

ndame wrote:

> Would you agree that if the code does only automatic
> indentation for the user by calling core indent functions
> then it's likely that emacs is the culprit for the error and
> I should submit a bug report for emacs?

If so that points in that direction yes but still a lot of
otherwise sound things can be used incorrectly and easily
cause damage that way ...

-- 
underground experts united
https://dataswamp.org/~incal

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[Stefan Monnier via Users list for the GNU Emacs text editor]

> The maintainer is not really responsive. I skimmed the code and
> it seems it only calls (indent-according-to-mode) and stuff,
> and does not do anything crazy:

I haven't looked in detail, but I did notice that it let-binds
`after-change-functions` which can interfere with other code adding
functions to this hook.

Does the quick&dirty patch below help?


        Stefan


diff --git a/aggressive-indent.el b/aggressive-indent.el
index 0234fcf29c..a08a4c241b 100644
--- a/aggressive-indent.el
+++ b/aggressive-indent.el
@@ -393,7 +393,7 @@ or messages."
   "Indent the regions in `aggressive-indent--changed-list'."
   (unless (or (run-hook-wrapped 'aggressive-indent--internal-dont-indent-if #'eval)
               (aggressive-indent--run-user-hooks))
-    (let ((after-change-functions (remove 'aggressive-indent--keep-track-of-changes after-change-functions))
+    (let ((aggressive-indent-mode nil)
           (inhibit-point-motion-hooks t)
           (indent-function
            (if (cl-member-if #'derived-mode-p aggressive-indent-modes-to-prefer-defun)

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[ndame]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

> I haven't looked in detail, but I did notice that it let-binds
> `after-change-functions` which can interfere with other code adding
> functions to this hook.
>
> Does the quick&dirty patch below help?

I started emacs -Q and though I have the mode installed via
package.el, I did not want to setup package stuff, so I just
copied the code from here:

https://raw.githubusercontent.com/Malabarba/aggressive-indent-mode/master/aggressive-indent.el

Turns out with this latest version the problem is not reproducible.

I made a diff compared to my installed version and apparently the
new, working version adds the `after-change-functions` which you wanted
to remove among other things . (diff attached)

So the problem went away, because I upgraded to the latest version,
which is good, but I'm still wondering how the syntax parser got
confused because of indentation stuff.


[-- Attachment #2: diff --]
[-- Type: application/octet-stream, Size: 5962 bytes --]

-;;; aggressive-indent.el --- Minor mode to aggressively keep your code always indented
+;;; aggressive-indent.el --- Minor mode to aggressively keep your code always indented  -*- lexical-binding:t -*-
 
-;; Copyright (C) 2014 Free Software Foundation, Inc.
+;; Copyright (C) 2014-2021 Free Software Foundation, Inc
 
 ;; Author: Artur Malabarba <emacs@endlessparentheses.com>
 ;; URL: https://github.com/Malabarba/aggressive-indent-mode
-;; Package-Version: 20190218.2331
-;; Version: 1.8.4
-;; Package-Requires: ((emacs "24.1") (cl-lib "0.5"))
+;; Version: 1.10.0
+;; Package-Requires: ((emacs "24.3"))
 ;; Keywords: indent lisp maint tools
 ;; Prefix: aggressive-indent
 ;; Separator: -
@@ -71,7 +70,7 @@
 ;;
 ;; This program is free software; you can redistribute it and/or
 ;; modify it under the terms of the GNU General Public License
-;; as published by the Free Software Foundation; either version 2
+;; as published by the Free Software Foundation; either version 3
 ;; of the License, or (at your option) any later version.
 ;;
 ;; This program is distributed in the hope that it will be useful,
@@ -355,8 +354,7 @@
 Call `aggressive-indent-region-function' between L and R, and
 then keep indenting until nothing more happens."
   (interactive "r")
-  (let ((p (point-marker))
-        was-begining-of-line)
+  (let ((p (point-marker)))
     (set-marker-insertion-type p t)
     (unwind-protect
         (progn
@@ -391,11 +389,11 @@
   "List of (left right) limit of regions changed in the last command loop.")
 (make-variable-buffer-local 'aggressive-indent--changed-list)
 
-(defun aggressive-indent--proccess-changed-list-and-indent ()
+(defun aggressive-indent--process-changed-list-and-indent ()
   "Indent the regions in `aggressive-indent--changed-list'."
   (unless (or (run-hook-wrapped 'aggressive-indent--internal-dont-indent-if #'eval)
               (aggressive-indent--run-user-hooks))
-    (let ((inhibit-modification-hooks t)
+    (let ((after-change-functions (remove 'aggressive-indent--keep-track-of-changes after-change-functions))
           (inhibit-point-motion-hooks t)
           (indent-function
            (if (cl-member-if #'derived-mode-p aggressive-indent-modes-to-prefer-defun)
@@ -460,26 +458,31 @@
              nil)
             (t val)))))))
 
-(defun aggressive-indent--indent-if-changed ()
-  "Indent any region that changed in the last command loop."
-  (if (not (buffer-live-p (current-buffer)))
-      (cancel-timer aggressive-indent--idle-timer)
-    (when (and aggressive-indent-mode aggressive-indent--changed-list)
-      (save-excursion
-        (save-selected-window
-          (aggressive-indent--while-no-input
-            (aggressive-indent--proccess-changed-list-and-indent))))
-      (when (timerp aggressive-indent--idle-timer)
-        (cancel-timer aggressive-indent--idle-timer)))))
+(defun aggressive-indent--maybe-cancel-timer ()
+  "Cancel and remove the timer if it is set."
+  (when (timerp aggressive-indent--idle-timer)
+    (cancel-timer aggressive-indent--idle-timer)
+    (setq aggressive-indent--idle-timer nil)))
+
+(defun aggressive-indent--indent-if-changed (buffer)
+  "Indent any region that changed in BUFFER in the last command loop."
+  (if (not (buffer-live-p buffer))
+      (aggressive-indent--maybe-cancel-timer)
+    (with-current-buffer buffer
+      (when (and aggressive-indent-mode aggressive-indent--changed-list)
+        (save-excursion
+          (save-selected-window
+            (aggressive-indent--while-no-input
+              (aggressive-indent--process-changed-list-and-indent))))
+        (aggressive-indent--maybe-cancel-timer)))))
 
 (defun aggressive-indent--keep-track-of-changes (l r &rest _)
   "Store the limits (L and R) of each change in the buffer."
   (when aggressive-indent-mode
     (push (list l r) aggressive-indent--changed-list)
-    (when (timerp aggressive-indent--idle-timer)
-      (cancel-timer aggressive-indent--idle-timer))
+    (aggressive-indent--maybe-cancel-timer)
     (setq aggressive-indent--idle-timer
-          (run-with-idle-timer aggressive-indent-sit-for-time t #'aggressive-indent--indent-if-changed))))
+          (run-with-idle-timer aggressive-indent-sit-for-time t #'aggressive-indent--indent-if-changed (current-buffer)))))
 
 ;;; Minor modes
 ;;;###autoload
@@ -510,14 +513,15 @@
           (aggressive-indent--local-electric t))
         (add-hook 'after-change-functions #'aggressive-indent--keep-track-of-changes nil 'local)
         (add-hook 'after-revert-hook #'aggressive-indent--clear-change-list nil 'local)
-        (add-hook 'before-save-hook #'aggressive-indent--proccess-changed-list-and-indent nil 'local))
+        (add-hook 'before-save-hook #'aggressive-indent--process-changed-list-and-indent nil 'local)
+        (add-hook 'kill-buffer-hook #'aggressive-indent--maybe-cancel-timer nil 'local))
     ;; Clean the hooks
-    (when (timerp aggressive-indent--idle-timer)
-      (cancel-timer aggressive-indent--idle-timer))
+    (aggressive-indent--maybe-cancel-timer)
     (remove-hook 'after-change-functions #'aggressive-indent--keep-track-of-changes 'local)
     (remove-hook 'after-revert-hook #'aggressive-indent--clear-change-list 'local)
-    (remove-hook 'before-save-hook #'aggressive-indent--proccess-changed-list-and-indent 'local)
-    (remove-hook 'post-command-hook #'aggressive-indent--softly-indent-defun 'local)))
+    (remove-hook 'before-save-hook #'aggressive-indent--process-changed-list-and-indent 'local)
+    (remove-hook 'post-command-hook #'aggressive-indent--softly-indent-defun 'local)
+    (remove-hook 'kill-buffer-hook #'aggressive-indent--maybe-cancel-timer 'local)))
 
 (defun aggressive-indent--local-electric (on)
   "Turn variable `electric-indent-mode' on or off locally, as per boolean ON."

Diff finished.  Wed Jan 19 22:15:23 2022

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[Stefan Monnier via Users list for the GNU Emacs text editor]

> I made a diff compared to my installed version and apparently the
> new, working version adds the `after-change-functions` which you wanted
> to remove among other things . (diff attached)

I didn't quite remove it: I replaced it with something that has
a similar effect but without touching that var.

> So the problem went away, because I upgraded to the latest version,
> which is good, but I'm still wondering how the syntax parser got
> confused because of indentation stuff.

Good question.


        Stefan

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[ndame]

>> So the problem went away, because I upgraded to the latest version,
>> which is good, but I'm still wondering how the syntax parser got
>> confused because of indentation stuff.
>
> Good question.

Do you think it's possible that particular version of aggressive-indent-mode
exposed a latent bug in emacs' syntax handler?

Should I open a bug for emacs describing how the issue can be replicated
using that older aggressive-indent-mode version?

Re: backward-sexp throws scan-error "Unbalanced parentheses", but forward-sexp works

[Stefan Monnier via Users list for the GNU Emacs text editor]

> Do you think it's possible that particular version of aggressive-indent-mode
> exposed a latent bug in emacs' syntax handler?

It's possible.

> Should I open a bug for emacs describing how the issue can be replicated
> using that older aggressive-indent-mode version?

I don't think it's worth the trouble, no: it's a kind of bug that can be
fairly hard to track down, and there's a non-negligible chance that in
the end the error is in that old `aggresssive-indent-mode` code, so the
motivation to spend the time on it is quite low.

It's not like we don't have enough other bugs to fix ;-)


        Stefan