all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* [PATCH] Fix python-info-current-defun performance for large defuns
@ 2023-09-04  1:37 Tom Gillespie
  2023-09-04 11:49 ` Eli Zaretskii
  0 siblings, 1 reply; 6+ messages in thread
From: Tom Gillespie @ 2023-09-04  1:37 UTC (permalink / raw)
  To: Emacs developers

[-- Attachment #1: Type: text/plain, Size: 445 bytes --]

Hi,
   Here is a patch to fix a performance issue in
python-info-current-defun that appears when the function
is run on large defuns. The issue can appear to the user as a
noticeable (~500ms) freeze when the point is in a large defun
in a python buffer with which-function-mode enabled. More
details in the commit message and comments in the code.

I made the patch against the emacs-29 branch, but will also
on master without issue.

Best,
Tom

[-- Attachment #2: 0001-Fix-python-info-current-defun-performance-for-large-.patch --]
[-- Type: text/x-patch, Size: 8271 bytes --]

From 523ea2e041d1efdcc6ab6bac637ce4253b856fe2 Mon Sep 17 00:00:00 2001
From: Tom Gillespie <tgbugs@gmail.com>
Date: Sun, 3 Sep 2023 15:09:17 -0700
Subject: [PATCH] Fix python-info-current-defun performance for large defuns

* lisp/progmodes/python.el (python-info-current-defun): Significant
  performance improvement for large functions.
* test/lisp/progmodes/python-tests.el (python-info-current-defun-5):
  Added test for multiply nested non-defun lines.

The previous pervious implementation had two issues. First, it checked
every single intervening defun in the file, even those at the same
nesting level. Second, when it found a putative parent defun it would
call python-nav-end-of-defun which is incredibly slow for large defuns
(e.g. python classes that are many thousands of lines long).

The new implementation avoids these issues by using re-search-backward
to find the containing deindented line. It also handles cases where
there are multiple nested deindented lines that are not defuns.
---
 lisp/progmodes/python.el            | 85 ++++++++++++++++++-----------
 test/lisp/progmodes/python-tests.el | 20 +++++++
 2 files changed, 74 insertions(+), 31 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index d9ca37145e1..1fbd220ddb8 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -5607,42 +5607,31 @@ python-info-current-defun
             (starting-pos (point))
             (first-run t)
             (last-indent)
+            (non-defun-indent (python-info-looking-at-beginning-of-defun))
             (type))
         (catch 'exit
-          (while (python-nav-beginning-of-defun 1)
+          (while (python-nav-beginning-of-defun 1) ; `python-nav-beginning-of-defun' doesn't respect indentation
             (when (save-match-data
                     (and
                      (or (not last-indent)
                          (< (current-indentation) last-indent))
-                     (or
-                      (and first-run
-                           (save-excursion
-                             ;; If this is the first run, we may add
-                             ;; the current defun at point.
-                             (setq first-run nil)
-                             (goto-char starting-pos)
-                             (python-nav-beginning-of-statement)
-                             (beginning-of-line 1)
-                             (looking-at-p
-                              python-nav-beginning-of-defun-regexp)))
-                      (< starting-pos
-                         (save-excursion
-                           (let ((min-indent
-                                  (+ (current-indentation)
-                                     python-indent-offset)))
-                             (if (< starting-indentation  min-indent)
-                                 ;; If the starting indentation is not
-                                 ;; within the min defun indent make the
-                                 ;; check fail.
-                                 starting-pos
-                               ;; Else go to the end of defun and add
-                               ;; up the current indentation to the
-                               ;; ending position.
-                               (python-nav-end-of-defun)
-                               (+ (point)
-                                  (if (>= (current-indentation) min-indent)
-                                      (1+ (current-indentation))
-                                    0)))))))))
+                     (if first-run
+                         (or (save-excursion
+                               ;; If this is the first run, we may add
+                               ;; the current defun at point.
+                               (goto-char starting-pos)
+                               (python-nav-beginning-of-statement)
+                               (beginning-of-line 1)
+                               (looking-at-p
+                                python-nav-beginning-of-defun-regexp))
+                             (< (current-indentation) starting-indentation))
+                       (or
+                        ;; we are at the next enclosing defun
+                        (not non-defun-indent)
+                        ;; we are searching from a not-defun and do not match cases
+                        ;; where a defun is at the same level as the not-defun we
+                        ;; started from
+                        (< (current-indentation) non-defun-indent)))))
               (save-match-data (setq last-indent (current-indentation)))
               (if (or (not include-type) type)
                   (setq names (cons (match-string-no-properties 1) names))
@@ -5650,7 +5639,41 @@ python-info-current-defun
                   (setq type (car match))
                   (setq names (cons (cadr match) names)))))
             ;; Stop searching ASAP.
-            (and (= (current-indentation) 0) (throw 'exit t))))
+            (and (= (current-indentation) 0) (throw 'exit t))
+            (when first-run
+              ;; `python-nav-beginning-of-defun' will go to the previous defun regardless of indentation
+              ;; so on `first-run' we have to reset to `starting-pos' to ensure that `next-ind' will be
+              ;; calculated from the correct starting point
+              (setq first-run nil)
+              (goto-char starting-pos))
+            (let (found-defun next-ind)
+              (while (and (not found-defun) (>= (setq next-ind (- (current-indentation) python-indent-offset)) 0))
+                (progn
+                  ;; skip intervening defuns at the same or greater indentation by
+                  ;; searching backward to find the next line with less indentation
+                  (re-search-backward (format "^[ ]\\{%s\\}[^ \n]" next-ind))
+                  (setq non-defun-indent
+                        ;; we have to record `non-defun-indent' so that
+                        ;; we don't incorrectly match cases where a function
+                        ;; is e.g. defined inside an if statement that is at
+                        ;; the same level as another defun, e.g.
+                        ;;
+                        ;;     def a():
+                        ;;         def b(): return
+                        ;;         if True:
+                        ;;             def c(): return
+                        ;;     if True:
+                        ;;         if True:
+                        ;;             def d(): return
+                        ;;
+                        ;; the while loop is needed to handle the multiply nested case
+                        ;; otherwise it incorrectly matchs a.d when the point is in d
+                        (if (python-info-looking-at-beginning-of-defun)
+                            (progn
+                              (forward-line)
+                              (setq found-defun t)
+                              nil)
+                          (current-indentation))))))))
         (and names
              (concat (and type (format "%s " type))
                      (mapconcat #'identity names ".")))))))
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index e1b4c0a74c0..84ae784bff0 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -5223,6 +5223,26 @@ python-info-current-defun-4
    (should (string= (python-info-current-defun t)
                     "def func"))))
 
+(ert-deftest python-info-current-defun-5 ()
+  "Ensure multiple nested non-defun lines are handle correctly."
+  (python-tests-with-temp-buffer
+   "
+def a():
+    def b(): return
+    if True:
+        def c(): return
+
+if True:
+    if True:
+        def d(): return
+"
+   (python-tests-look-at "def c")
+   (should (string= (python-info-current-defun) "a.c"))
+   (should (string= (python-info-current-defun t) "def a.c"))
+   (python-tests-look-at "def d")
+   (should (string= (python-info-current-defun) "d"))
+   (should (string= (python-info-current-defun t) "def d"))))
+
 (ert-deftest python-info-current-symbol-1 ()
   (python-tests-with-temp-buffer
    "
-- 
2.41.0


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH] Fix python-info-current-defun performance for large defuns
  2023-09-04  1:37 [PATCH] Fix python-info-current-defun performance for large defuns Tom Gillespie
@ 2023-09-04 11:49 ` Eli Zaretskii
  2023-09-05 14:17   ` kobarity
  0 siblings, 1 reply; 6+ messages in thread
From: Eli Zaretskii @ 2023-09-04 11:49 UTC (permalink / raw)
  To: Tom Gillespie, kobarity; +Cc: emacs-devel

> From: Tom Gillespie <tgbugs@gmail.com>
> Date: Sun, 3 Sep 2023 18:37:34 -0700
> 
> Hi,
>    Here is a patch to fix a performance issue in
> python-info-current-defun that appears when the function
> is run on large defuns. The issue can appear to the user as a
> noticeable (~500ms) freeze when the point is in a large defun
> in a python buffer with which-function-mode enabled. More
> details in the commit message and comments in the code.
> 
> I made the patch against the emacs-29 branch, but will also
> on master without issue.

Thanks.

kobarity, any comments?



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] Fix python-info-current-defun performance for large defuns
  2023-09-04 11:49 ` Eli Zaretskii
@ 2023-09-05 14:17   ` kobarity
  2023-09-15  6:56     ` Eli Zaretskii
  0 siblings, 1 reply; 6+ messages in thread
From: kobarity @ 2023-09-05 14:17 UTC (permalink / raw)
  To: Tom Gillespie, Eli Zaretskii; +Cc: emacs-devel


Eli Zaretskii wrote:
> > From: Tom Gillespie <tgbugs@gmail.com>
> > Date: Sun, 3 Sep 2023 18:37:34 -0700
> > 
> > Hi,
> >    Here is a patch to fix a performance issue in
> > python-info-current-defun that appears when the function
> > is run on large defuns. The issue can appear to the user as a
> > noticeable (~500ms) freeze when the point is in a large defun
> > in a python buffer with which-function-mode enabled. More
> > details in the commit message and comments in the code.
> > 
> > I made the patch against the emacs-29 branch, but will also
> > on master without issue.
> 
> Thanks.
> 
> kobarity, any comments?

Hi,
Thank you for the patch.  I have tried it and have confirmed that it
improves performance.  I think that not using the time-consuming
`python-nav-end-of-defun' has helped.

However, I found one unexpected behavior.  The following code is an
example added as an ERT:

def a():
    def b(): return
    if True:
        def c(): return

if True:
    if True:
        def d(): return

In this example, `python-info-current-defun' seems to return "a" if
the point is on the second line from the bottom.  Could you please
take a look?

My personal impression is that 114 characters per line is a bit long.



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] Fix python-info-current-defun performance for large defuns
  2023-09-05 14:17   ` kobarity
@ 2023-09-15  6:56     ` Eli Zaretskii
  2023-09-15 18:20       ` Tom Gillespie
  0 siblings, 1 reply; 6+ messages in thread
From: Eli Zaretskii @ 2023-09-15  6:56 UTC (permalink / raw)
  To: tgbugs, kobarity; +Cc: emacs-devel

Ping!  Tom, could you please respond to the comments, and perhaps post
an updated patch?  I'd like to install this when it's ready.

Thanks.

> Date: Tue, 05 Sep 2023 23:17:48 +0900
> From: kobarity <kobarity@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> 
> Eli Zaretskii wrote:
> > > From: Tom Gillespie <tgbugs@gmail.com>
> > > Date: Sun, 3 Sep 2023 18:37:34 -0700
> > > 
> > > Hi,
> > >    Here is a patch to fix a performance issue in
> > > python-info-current-defun that appears when the function
> > > is run on large defuns. The issue can appear to the user as a
> > > noticeable (~500ms) freeze when the point is in a large defun
> > > in a python buffer with which-function-mode enabled. More
> > > details in the commit message and comments in the code.
> > > 
> > > I made the patch against the emacs-29 branch, but will also
> > > on master without issue.
> > 
> > Thanks.
> > 
> > kobarity, any comments?
> 
> Hi,
> Thank you for the patch.  I have tried it and have confirmed that it
> improves performance.  I think that not using the time-consuming
> `python-nav-end-of-defun' has helped.
> 
> However, I found one unexpected behavior.  The following code is an
> example added as an ERT:
> 
> def a():
>     def b(): return
>     if True:
>         def c(): return
> 
> if True:
>     if True:
>         def d(): return
> 
> In this example, `python-info-current-defun' seems to return "a" if
> the point is on the second line from the bottom.  Could you please
> take a look?
> 
> My personal impression is that 114 characters per line is a bit long.
> 



^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] Fix python-info-current-defun performance for large defuns
  2023-09-15  6:56     ` Eli Zaretskii
@ 2023-09-15 18:20       ` Tom Gillespie
  2023-09-16 14:51         ` kobarity
  0 siblings, 1 reply; 6+ messages in thread
From: Tom Gillespie @ 2023-09-15 18:20 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: kobarity, emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1923 bytes --]

Here's an updated patch with the fix and additional test for the
case kobarity identified. I also shortened the long lines. Best!
Tom

On Thu, Sep 14, 2023 at 11:56 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> Ping!  Tom, could you please respond to the comments, and perhaps post
> an updated patch?  I'd like to install this when it's ready.
>
> Thanks.
>
> > Date: Tue, 05 Sep 2023 23:17:48 +0900
> > From: kobarity <kobarity@gmail.com>
> > Cc: emacs-devel@gnu.org
> >
> >
> > Eli Zaretskii wrote:
> > > > From: Tom Gillespie <tgbugs@gmail.com>
> > > > Date: Sun, 3 Sep 2023 18:37:34 -0700
> > > >
> > > > Hi,
> > > >    Here is a patch to fix a performance issue in
> > > > python-info-current-defun that appears when the function
> > > > is run on large defuns. The issue can appear to the user as a
> > > > noticeable (~500ms) freeze when the point is in a large defun
> > > > in a python buffer with which-function-mode enabled. More
> > > > details in the commit message and comments in the code.
> > > >
> > > > I made the patch against the emacs-29 branch, but will also
> > > > on master without issue.
> > >
> > > Thanks.
> > >
> > > kobarity, any comments?
> >
> > Hi,
> > Thank you for the patch.  I have tried it and have confirmed that it
> > improves performance.  I think that not using the time-consuming
> > `python-nav-end-of-defun' has helped.
> >
> > However, I found one unexpected behavior.  The following code is an
> > example added as an ERT:
> >
> > def a():
> >     def b(): return
> >     if True:
> >         def c(): return
> >
> > if True:
> >     if True:
> >         def d(): return
> >
> > In this example, `python-info-current-defun' seems to return "a" if
> > the point is on the second line from the bottom.  Could you please
> > take a look?
> >
> > My personal impression is that 114 characters per line is a bit long.
> >

[-- Attachment #2: 0001-Fix-python-info-current-defun-performance-for-large-.patch --]
[-- Type: text/x-patch, Size: 9213 bytes --]

From b906381079c4cea5b20986902e82e759756ae17d Mon Sep 17 00:00:00 2001
From: Tom Gillespie <tgbugs@gmail.com>
Date: Sun, 3 Sep 2023 15:09:17 -0700
Subject: [PATCH] Fix python-info-current-defun performance for large defuns

* lisp/progmodes/python.el (python-info-current-defun): Significant
  performance improvement for large functions.
* test/lisp/progmodes/python-tests.el (python-info-current-defun-5):
  Added test for multiply nested non-defun lines.

The previous pervious implementation had two issues. First, it checked
every single intervening defun in the file, even those at the same
nesting level. Second, when it found a putative parent defun it would
call python-nav-end-of-defun which is incredibly slow for large defuns
(e.g. python classes that are many thousands of lines long).

The new implementation avoids these issues by using re-search-backward
to find the containing deindented line. It also handles cases where
there are multiple nested deindented lines that are not defuns.
---
 lisp/progmodes/python.el            | 101 +++++++++++++++++++---------
 test/lisp/progmodes/python-tests.el |  23 +++++++
 2 files changed, 94 insertions(+), 30 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 4b940b3f13b..1ef9dcc90eb 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -5699,42 +5699,44 @@ python-info-current-defun
             (starting-pos (point))
             (first-run t)
             (last-indent)
+            (non-defun-indent (python-info-looking-at-beginning-of-defun))
             (type))
         (catch 'exit
           (while (python-nav-beginning-of-defun 1)
+            ;; `python-nav-beginning-of-defun' doesn't respect indentation
             (when (save-match-data
                     (and
                      (or (not last-indent)
                          (< (current-indentation) last-indent))
-                     (or
-                      (and first-run
-                           (save-excursion
-                             ;; If this is the first run, we may add
-                             ;; the current defun at point.
-                             (setq first-run nil)
-                             (goto-char starting-pos)
-                             (python-nav-beginning-of-statement)
-                             (beginning-of-line 1)
-                             (looking-at-p
-                              python-nav-beginning-of-defun-regexp)))
-                      (< starting-pos
-                         (save-excursion
-                           (let ((min-indent
-                                  (+ (current-indentation)
-                                     python-indent-offset)))
-                             (if (< starting-indentation  min-indent)
-                                 ;; If the starting indentation is not
-                                 ;; within the min defun indent make the
-                                 ;; check fail.
-                                 starting-pos
-                               ;; Else go to the end of defun and add
-                               ;; up the current indentation to the
-                               ;; ending position.
-                               (python-nav-end-of-defun)
-                               (+ (point)
-                                  (if (>= (current-indentation) min-indent)
-                                      (1+ (current-indentation))
-                                    0)))))))))
+                     (if first-run
+                         (or (save-excursion
+                               ;; If this is the first run, we may add
+                               ;; the current defun at point.
+                               (goto-char starting-pos)
+                               (python-nav-beginning-of-statement)
+                               (beginning-of-line 1)
+                               (looking-at-p
+                                python-nav-beginning-of-defun-regexp))
+                             (and
+                              (> starting-indentation 0)
+                              (save-excursion ; ensure no non-defun
+                                ;; deindented lines in between start
+                                ;; and `python-nav-beginning-of-defun'
+                                (goto-char starting-pos)
+                                (re-search-backward
+                                 (format "^[ ]\\{%s\\}[^ \n]"
+                                         (- starting-indentation
+                                            python-indent-offset)))
+                                (looking-at-p
+                                 python-nav-beginning-of-defun-regexp))
+                              (< (current-indentation) starting-indentation)))
+                       (or
+                        ;; we are at the next enclosing defun
+                        (not non-defun-indent)
+                        ;; we are searching from a not-defun and do not match cases
+                        ;; where a defun is at the same level as the not-defun we
+                        ;; started from
+                        (< (current-indentation) non-defun-indent)))))
               (save-match-data (setq last-indent (current-indentation)))
               (if (or (not include-type) type)
                   (setq names (cons (match-string-no-properties 1) names))
@@ -5742,7 +5744,46 @@ python-info-current-defun
                   (setq type (car match))
                   (setq names (cons (cadr match) names)))))
             ;; Stop searching ASAP.
-            (and (= (current-indentation) 0) (throw 'exit t))))
+            (and (= (current-indentation) 0) (throw 'exit t))
+            (when first-run
+              ;; `python-nav-beginning-of-defun' will go to the previous
+              ;; defun regardless of indentation so on `first-run' we
+              ;; have to reset to `starting-pos' to ensure that `next-ind'
+              ;; will be calculated from the correct starting point
+              (setq first-run nil)
+              (goto-char starting-pos))
+            (let (found-defun next-ind)
+              (while (and (not found-defun)
+                          (>= (setq next-ind
+                                    (- (current-indentation)
+                                       python-indent-offset))
+                              0))
+                (progn
+                  ;; search backward to find the next line with less
+                  ;; indentation to skip defuns at same or greater indentation
+                  (re-search-backward (format "^[ ]\\{%s\\}[^ \n]" next-ind))
+                  (setq non-defun-indent
+                        ;; we have to record `non-defun-indent' so that
+                        ;; we don't incorrectly match cases where a function
+                        ;; is e.g. defined inside an if statement that is at
+                        ;; the same level as another defun, e.g.
+                        ;;
+                        ;;     def a():
+                        ;;         def b(): return
+                        ;;         if 1:
+                        ;;             def c(): return
+                        ;;     if 2:
+                        ;;         if 3:
+                        ;;             def d(): return
+                        ;;
+                        ;; while loop needed to handle the multiply nested case
+                        ;; otherwise it incorrectly matchs a.d when the point is in d
+                        (if (python-info-looking-at-beginning-of-defun)
+                            (progn
+                              (forward-line)
+                              (setq found-defun t)
+                              nil)
+                          (current-indentation))))))))
         (and names
              (concat (and type (format "%s " type))
                      (mapconcat #'identity names ".")))))))
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index 9f935f2748c..47391336193 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -5400,6 +5400,29 @@ python-info-current-defun-4
    (should (string= (python-info-current-defun t)
                     "def func"))))
 
+(ert-deftest python-info-current-defun-5 ()
+  "Ensure multiple nested non-defun lines are handle correctly."
+  (python-tests-with-temp-buffer
+   "
+def a():
+    def b(): return
+    if 1:
+        def c(): return
+
+if 2:
+    if 3:
+        def d(): return
+"
+   (python-tests-look-at "def c")
+   (should (string= (python-info-current-defun) "a.c"))
+   (should (string= (python-info-current-defun t) "def a.c"))
+   (python-tests-look-at "    if 3:")
+   (should (not (python-info-current-defun)))
+   (should (not (python-info-current-defun t)))
+   (python-tests-look-at "def d")
+   (should (string= (python-info-current-defun) "d"))
+   (should (string= (python-info-current-defun t) "def d"))))
+
 (ert-deftest python-info-current-symbol-1 ()
   (python-tests-with-temp-buffer
    "
-- 
2.41.0


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH] Fix python-info-current-defun performance for large defuns
  2023-09-15 18:20       ` Tom Gillespie
@ 2023-09-16 14:51         ` kobarity
  0 siblings, 0 replies; 6+ messages in thread
From: kobarity @ 2023-09-16 14:51 UTC (permalink / raw)
  To: Tom Gillespie; +Cc: Eli Zaretskii, emacs-devel


Tom Gillespie wrote:
> Here's an updated patch with the fix and additional test for the
> case kobarity identified. I also shortened the long lines. Best!

Thank you for the updated patch.  However, there still seems to be a
bit of a problem.

def a():
    def b(): return
    if 1:
        def c(): return
        z = 1

When the point is at the "z = 1" line, python-info-current-defun
should return "a", but the new function returns nil.

I also noticed another subtle difference in behavior with the example
added as an ERT:

def a():
    def b(): return
    if 1:
        def c(): return

if 2:
    if 3:
        def d(): return

If the point is on the empty line above "if 2:", the new function
returns "a", while the current function returns nil.



^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2023-09-16 14:51 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-09-04  1:37 [PATCH] Fix python-info-current-defun performance for large defuns Tom Gillespie
2023-09-04 11:49 ` Eli Zaretskii
2023-09-05 14:17   ` kobarity
2023-09-15  6:56     ` Eli Zaretskii
2023-09-15 18:20       ` Tom Gillespie
2023-09-16 14:51         ` kobarity

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.