bug#75387: 29.4; hideshow: hs-hide-level hides multiple top-level blocks as one

bug#75387: 29.4; hideshow: hs-hide-level hides multiple top-level blocks as one

[Christoph Groth]

Hello,

I noticed the following problem when using a homemade function that
allows me to cycle hideshow visibility in a way similar to org-mode.
The issue, however, is independent of any customization (it appears with
emacs -Q as well).

I can reproduce the issue with the following source file

https://gitlab.kwant-project.org/kwant/kwant/-/raw/0d679b3efecd145d13e3174c60d829a7c3cd4374/kwant/builder.py

I tried simplifying that file to isolate the issue, but did not arrive at
something more useful than the original.  To demonstrate the issue:

(1) Download the above file.
(2) Open it with emacs -Q builder.py.  Point is not moved.
(3) M-x hs-minor-mode RET
(4) M-x hs-hide-level RET

The result is that the buffer shows only two hidden blocks, with the
buffer ending in

@total_ordering
class SiteFamily(metaclass=abc.ABCMeta):...

This is incorrect, since there are many other classes and top-level
functions in the file.  Each should appear as a separate top-level
block.

If M-x hs-hide-all is run instead of step (4) the hiding works as
expected, many more top-level hidden blocks appear.  M-x hd-hide-level
also produces the expected effect if the second block created by (4) is
un-hidden in some way.  For example if the following two steps are added
to the above sequence:

(5) M-x hs-show-all RET
(6) M-x hs-hide-level RET

I would expect that hs-hide-level always produces the same result when
called with point at the same position in a buffer.

I hope that this somewhat obscure observation allows to fix some
inconsistency.

Thanks,
Christoph

---

In GNU Emacs 29.4 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.38,
 cairo version 1.16.0) of 2024-12-15, modified by Debian built on sbuild
Windowing system distributor 'The X.Org Foundation', version 11.0.12101007
System Description: Debian GNU/Linux 12 (bookworm)

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/29.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/libexec
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --with-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/29.4/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/29.4/site-lisp:/usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils
 --with-native-compilation --with-cairo --with-x=yes
 --with-x-toolkit=gtk3 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -ffile-prefix-map=/build/reproducible-path/emacs-29.4+1=. -fstack-protector-strong
 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ JPEG
JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM XINPUT2
XPM GTK3 ZLIB

Important settings:
  value of $LC_MESSAGES: en_US.UTF-8
  value of $LANG: en_DK.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
comp comp-cstr warnings icons subr-x rx cl-seq cl-macs gv cl-extra
help-mode bytecomp byte-compile cl-lib sendmail rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils rmc iso-transl tooltip cconv
eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow
isearch easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting cairo
move-toolbar gtk x-toolkit xinput2 x multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 77057 13062)
 (symbols 48 7444 0)
 (strings 32 19535 1545)
 (string-bytes 1 584589)
 (vectors 16 15338)
 (vector-slots 8 328476 18154)
 (floats 8 27 46)
 (intervals 56 275 0)
 (buffers 984 11))

bug#75387: 29.4; hideshow: hs-hide-level hides multiple top-level blocks as one

[Eli Zaretskii]

> From: Christoph Groth <christoph@grothesque.org>
> Date: Sun, 05 Jan 2025 19:53:04 +0100
> 
> I noticed the following problem when using a homemade function that
> allows me to cycle hideshow visibility in a way similar to org-mode.
> The issue, however, is independent of any customization (it appears with
> emacs -Q as well).
> 
> I can reproduce the issue with the following source file
> 
> https://gitlab.kwant-project.org/kwant/kwant/-/raw/0d679b3efecd145d13e3174c60d829a7c3cd4374/kwant/builder.py
> 
> I tried simplifying that file to isolate the issue, but did not arrive at
> something more useful than the original.  To demonstrate the issue:
> 
> (1) Download the above file.
> (2) Open it with emacs -Q builder.py.  Point is not moved.
> (3) M-x hs-minor-mode RET
> (4) M-x hs-hide-level RET
> 
> The result is that the buffer shows only two hidden blocks, with the
> buffer ending in
> 
> @total_ordering
> class SiteFamily(metaclass=abc.ABCMeta):...
> 
> This is incorrect, since there are many other classes and top-level
> functions in the file.  Each should appear as a separate top-level
> block.

Hm... the doc string of hs-hide-level says:

  Hide all blocks ARG levels below this block.

So what is "this block" when point is at position 1 in this buffer?

If I move point to here:

  @total_ordering
  class SiteFamily(metaclass=abc.ABCMeta):

and invoke hs-hide-level, then all the "def" blocks inside this one
are hidden, as expected.  And note that the command only hides blocks
that are inside the current block, not those inside other blocks of
the same level.

So this could be a documentation issue, whereby the doc string doesn't
explain clearly enough what the command does.

kobarity, any comments?

bug#75387: 29.4; hideshow: hs-hide-level hides multiple top-level blocks as one

[kobarity]

Christoph Groth wrote:
> 
> Eli Zaretskii wrote:
> > Hm... the doc string of hs-hide-level says:
> >
> >   Hide all blocks ARG levels below this block.
> >
> > So what is "this block" when point is at position 1 in this buffer?
> 
> In my understanding position 1 corresponds to “outermost level”.  Hiding
> all blocks below this level should give the same result as calling
> hs-hide-all.  And indeed this is what I observe after the initial hiccup
> (e.g. after having invoked hs-hide-all and then hs-show-all).
> 
> > If I move point to here:
> >
> >   @total_ordering
> >   class SiteFamily(metaclass=abc.ABCMeta):
> >
> > and invoke hs-hide-level, then all the "def" blocks inside this one
> > are hidden, as expected.  And note that the command only hides blocks
> > that are inside the current block, not those inside other blocks of
> > the same level.
> 
> Did you try this immediately after opening the file with find-file?  The
> problem I observe only manifests itself until (at most) a full cycle of
> hide-all then show-all.
> 
> If (after freshly loading the file - should there be a buffer visiting
> the file, I first kill it) I move point to the “@” before
> total_ordering, and invoke hs-hide-level, then I see the following (only
> showing the lower part of the buffer where something is hidden):
> 
> ----------------------------------------------------------------
> ################ Sites and site families
> 
> class Site(tuple):...
> 
> 
> @total_ordering
> class SiteFamily(metaclass=abc.ABCMeta):...
> --------------- end of buffer ----------------------------------
> 
> But if I do this after invoking hs-hide-all and then hs-show-all, all
> top-level blocks are visible but hidden (as expected).
> 
> If, after freshly loading the file, I position point on the “c” of
> “class SiteFamily”, and then invoke hs-hide-level, the methods of that
> class are hidden, but also top-level blocks that follow it (for example
> the function validate_hopping).
> 
> On the other hand, when I invoke hs-hide-level at the same place but
> after “cycling” at least once (=hs-hide-all followed by hs-show-all),
> only the methods of the class are hidden, and following top level blocks
> remain unhidden (as I would expect).
> 
> > So this could be a documentation issue, whereby the doc string doesn't
> > explain clearly enough what the command does.
> 
> The behavior I observe depends on whether the file has been freshly
> opened or already “cycled”.  But any correct behavior should not differ
> between a fresh buffer and one where blocks were hidden and then
> completely unhidden.  Therefore I think that there is a problem
> somewhere, at least in Emacs 29.4 that I use.  I have no idea whether
> this is in hideshow.all or in some other component like python.el.

I confirmed the same behavior.  This is not a problem in hideshow.el.
Only immediately after the file is read, `python-nav-end-of-block'
behaves unexpectedly.

1. emacs -Q
2. Find file builder.py
3. Search "class SiteFamily" and go there
4. M-x python-nav-end-of-block  
5. Search "class Symmetry" and go there
6. M-x python-nav-end-of-block

The point moves to the end of the buffer, which is not an expected
behavior.  However, repeating the steps 5 and 6 does not produce this
issue.

This is the direct cause of the problem of `hs-hide-level'.

This problem does not occur when I run `font-lock-debug-fontify' after
reading the file, so it might be related to font-lock code in
python.el, but it seems that disabling font-lock does not solve this
issue.  I will continue to look into this, but it may take some time.

bug#75387: 29.4; hideshow: hs-hide-level hides multiple top-level blocks as one

[kobarity]

[-- Attachment #1: Type: text/plain, Size: 1843 bytes --]

kobarity wrote:
> I confirmed the same behavior.  This is not a problem in hideshow.el.
> Only immediately after the file is read, `python-nav-end-of-block'
> behaves unexpectedly.
> 
> 1. emacs -Q
> 2. Find file builder.py
> 3. Search "class SiteFamily" and go there
> 4. M-x python-nav-end-of-block  
> 5. Search "class Symmetry" and go there
> 6. M-x python-nav-end-of-block
> 
> The point moves to the end of the buffer, which is not an expected
> behavior.  However, repeating the steps 5 and 6 does not produce this
> issue.
> 
> This is the direct cause of the problem of `hs-hide-level'.
> 
> This problem does not occur when I run `font-lock-debug-fontify' after
> reading the file, so it might be related to font-lock code in
> python.el, but it seems that disabling font-lock does not solve this
> issue.  I will continue to look into this, but it may take some time.

I think I've found the root cause.  `python-nav-end-of-statement' may
fail to find the end of a long triple-quoted string when the buffer is
"fresh".  It uses the following code to find the end of a
triple-quoted string:

(re-search-forward (rx (syntax string-delimiter)) nil t)

However, the syntax is not always fully propertized.  If the end of
the long multi-line string and beyond are not yet propertized, the
above search fails and `python-nav-end-of-statement' moves the point
to the end of the buffer.

In case of builder.py, `python-nav-end-of-statement' fails to find the
end of the long docstring.  This results in `python-nav-end-of-block'
moving point to the end of the buffer, which causes `hs-hide-level' to
hide multiple blocks as one.

Attached is a patch to fix this issue.  I changed the above search to
searching for triple-quotes and checking syntax.  Checking syntax
involves `syntax-ppss' and syntax properties up to the point are
updated.

[-- Attachment #2: 0001-Fix-string-end-search-in-python-nav-end-of-statement.patch --]
[-- Type: application/octet-stream, Size: 2739 bytes --]

From 6b3c34d272639f97ecb4043124db54501be20520 Mon Sep 17 00:00:00 2001
From: kobarity <kobarity@gmail.com>
Date: Mon, 13 Jan 2025 22:38:42 +0900
Subject: [PATCH] Fix string end search in python-nav-end-of-statement

* lisp/progmodes/python.el (python-nav-end-of-statement):
Change to look for string delimiter characters and check
syntax, instead of looking for string-delimiter syntax.
* test/lisp/progmodes/python-tests.el
(python-nav-end-of-statement-5): New test.  (Bug#75387)
---
 lisp/progmodes/python.el            |  7 +++++--
 test/lisp/progmodes/python-tests.el | 24 ++++++++++++++++++++++++
 2 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index 16c296a8f86..73eec88977c 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -2328,8 +2328,11 @@ python-nav-end-of-statement
                            (setq
                             last-string-end
                             (or (if (eq t (nth 3 (syntax-ppss)))
-                                    (re-search-forward
-                                     (rx (syntax string-delimiter)) nil t)
+                                    (cl-loop
+                                     while (re-search-forward
+                                            (rx (or "\"\"\"" "'''")) nil t)
+                                     unless (python-syntax-context 'string)
+                                     return (point))
                                   (ignore-error scan-error
                                     (goto-char string-start)
                                     (python-nav--lisp-forward-sexp)
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index ec0a836cb8f..898e2b036e0 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -3204,6 +3204,30 @@ python-nav-end-of-statement-4
                 (python-tests-look-at "c'")
                 (pos-eol))))))
 
+(ert-deftest python-nav-end-of-statement-5 ()
+  "Test long multi-line string (Bug#75387)."
+  (let* ((line (format "%s\n" (make-string 80 ?a)))
+         (lines (apply #'concat (make-list 50 line))))
+    (python-tests-with-temp-buffer
+     (concat
+      "
+s = '''
+"
+      lines
+      "\\'''"
+      lines
+      "'''
+a = 1
+")
+     (python-tests-look-at "s = '''")
+     (should (= (save-excursion
+                  (python-nav-end-of-statement)
+                  (point))
+                (save-excursion
+                  (python-tests-look-at "a = 1")
+                  (forward-line -1)
+                  (pos-eol)))))))
+
 (ert-deftest python-nav-forward-statement-1 ()
   (python-tests-with-temp-buffer
    "
-- 
2.43.0

bug#75387: 29.4; hideshow: hs-hide-level hides multiple top-level blocks as one

[kobarity]

kobarity wrote:
> kobarity wrote:
> > I confirmed the same behavior.  This is not a problem in hideshow.el.
> > Only immediately after the file is read, `python-nav-end-of-block'
> > behaves unexpectedly.
> > 
> > 1. emacs -Q
> > 2. Find file builder.py
> > 3. Search "class SiteFamily" and go there
> > 4. M-x python-nav-end-of-block  
> > 5. Search "class Symmetry" and go there
> > 6. M-x python-nav-end-of-block
> > 
> > The point moves to the end of the buffer, which is not an expected
> > behavior.  However, repeating the steps 5 and 6 does not produce this
> > issue.
> > 
> > This is the direct cause of the problem of `hs-hide-level'.
> > 
> > This problem does not occur when I run `font-lock-debug-fontify' after
> > reading the file, so it might be related to font-lock code in
> > python.el, but it seems that disabling font-lock does not solve this
> > issue.  I will continue to look into this, but it may take some time.
> 
> I think I've found the root cause.  `python-nav-end-of-statement' may
> fail to find the end of a long triple-quoted string when the buffer is
> "fresh".  It uses the following code to find the end of a
> triple-quoted string:
> 
> (re-search-forward (rx (syntax string-delimiter)) nil t)
> 
> However, the syntax is not always fully propertized.  If the end of
> the long multi-line string and beyond are not yet propertized, the
> above search fails and `python-nav-end-of-statement' moves the point
> to the end of the buffer.
> 
> In case of builder.py, `python-nav-end-of-statement' fails to find the
> end of the long docstring.  This results in `python-nav-end-of-block'
> moving point to the end of the buffer, which causes `hs-hide-level' to
> hide multiple blocks as one.
> 
> Attached is a patch to fix this issue.  I changed the above search to
> searching for triple-quotes and checking syntax.  Checking syntax
> involves `syntax-ppss' and syntax properties up to the point are
> updated.

Hi Christoph, could you test my patch?