bug#72080: imenu returns different lists in emacs-news-mode and emacs-news-view-mode

bug#72080: imenu returns different lists in emacs-news-mode and emacs-news-view-mode

[Siyuan Chen]

[-- Attachment #1: Type: text/plain, Size: 1382 bytes --]

Reproduce steps:

1 Clone Emacs source code from https://github.com/emacs-mirror/emacs

2. Emacs -Q (Emacs 29.3 on Windows)

3. Open etc/NEWS.30 from the Emacs source code (by default Emacs will use
emacs-news-mode)

4. M-x imenu and press TAB to complete

Emacs shows 515 possible completions.

6. M-x emacs-news-view-mode

7. M-x imenu and press TAB to complete

Emacs shows 202 possible completions.

This is the problem, i.e. emacs-news-mode and emacs-news-view-mode return
different imenu lists.

The Background:

I am reading Emacs NEWS and using the "imenu-list" package to display the
outline of the NEWS as a side window since NEWS are often very long. It
works well in emacs-news-mode, but lacks many entries in
emacs-news-view-mode. BTW the emacs-news-view-mode is nice because it
allows button click symbol references in the buffer..

In step 4 and step 7, you can do M-x eval-expression

```
(progn
  (require 'imenu)
  (setq message-log-max t)
  (let ((inhibit-read-only t))
    (with-current-buffer "*Messages*" (erase-buffer)))
  (pp (imenu--make-index-alist)))
```
and save the *Messages* buffer to two files respectively, e.g. news-mode.txt
and news-view-mode.txt. You will find that these two files are different.
The news-view-mode.txt lacks many imenu entries.

P.s. This issue only occurs for NEWS.30. It is OK for NEWS.29.

Thanks.

Best regards,
Siyuan Chen

[-- Attachment #2: Type: text/html, Size: 2336 bytes --]

bug#72080: imenu returns different lists in emacs-news-mode and emacs-news-view-mode

[Eli Zaretskii]

bug#72080: imenu returns different lists in emacs-news-mode and emacs-news-view-mode

[Eli Zaretskii]

> From: Siyuan Chen <chansey97@gmail.com>
> Date: Sat, 13 Jul 2024 01:52:11 +0800
> 
> Reproduce steps:
> 
> 1 Clone Emacs source code from https://github.com/emacs-mirror/emacs 
> 
> 2. Emacs -Q (Emacs 29.3 on Windows)
> 
> 3. Open etc/NEWS.30 from the Emacs source code (by default Emacs will use emacs-news-mode)
> 
> 4. M-x imenu and press TAB to complete
> 
> Emacs shows 515 possible completions. 
> 
> 6. M-x emacs-news-view-mode
> 
> 7. M-x imenu and press TAB to complete
> 
> Emacs shows 202 possible completions. 
> 
> This is the problem, i.e. emacs-news-mode and emacs-news-view-mode return different imenu lists.

This happens because emacs-news-view-mode derives from special-mode,
and therefore has a different syntax table.  Which has the effect of
duping imenu--generic-function into thinking some of the NEWS entries
are in comments, and so it ignores them:

		  ;; Insert the item unless it is already present.
		  (unless (or (member item (cdr menu))
                              (and imenu-generic-skip-comments-and-strings
                                   (save-excursion
                                     (goto-char start) (nth 8 (syntax-ppss)))))
		    (setcdr menu
			    (cons item (cdr menu)))))

Note the call to syntax-ppss there: it returns nil in 8th entry in
emacs-news-mode when point is, e.g., on the final ^L in NEWS, but
non-nil in emacs-news-view-mode.

The simple and naïve change below seems to fix that.  The only problem
with it is that one cannot move through the buffer with SPC, DEL, '<'
etc.  For some reason I cannot get emacs-news-view-mode inherit the
keymap of special-mode, nor even use its own keymap I defined in a
defvar emacs-news-view-mode-map.  Stefan, any reason why a simple
definition of emacs-news-view-mode-map doesn't work, as the ELisp
manual promises it should (where it described define-derived-mode)?

Also, any idea why syntax-ppss thinks the final ^L in NEWS, here:

  ---
  *** Emacs on MS-Windows now supports the ':stipple' face attribute.

  ^L
  ----------------------------------------------------------------------
  This file is part of GNU Emacs.

is in a comment when emacs-news-view-mode inherits from special-mode?

Btw, I find our syntax-table introspection features not very helpful
in these situations.  Once I figured out that syntax-ppss produces
different results in these two modes, all the rest was pure guesswork,
with no real way of investigating the differences and their reason(s).
Do we have any ways of comparing two syntax tables and showing the
differences in human-readable form?  Do we have a way of showing the
source(s) f a syntax-table's information, like where it inherited from
another and where it was modified by explicit modify-syntax-entry
calls?  Do we have a way of making one mode use the syntax table of
another mode without having a variable which holds the latter?  I
needed all of these to investigate in this case, and was finally
forced to treat the issue as a kind of "black box", making
more-or-less random changes and looking at the results, which is very
frustrating and a terrible waste of time...


diff --git a/lisp/textmodes/emacs-news-mode.el b/lisp/textmodes/emacs-news-mode.el
index 1dd017a..c87e7fa 100644
--- a/lisp/textmodes/emacs-news-mode.el
+++ b/lisp/textmodes/emacs-news-mode.el
@@ -107,7 +107,7 @@ emacs-news-mode
   (emacs-news--mode-common))
 
 ;;;###autoload
-(define-derived-mode emacs-news-view-mode special-mode "NEWS"
+(define-derived-mode emacs-news-view-mode emacs-news-mode "NEWS"
   "Major mode for viewing the Emacs NEWS file."
   (setq buffer-read-only t)
   (emacs-news--buttonize)

bug#72080: imenu returns different lists in emacs-news-mode and emacs-news-view-mode

[Siyuan Chen]

[-- Attachment #1: Type: text/plain, Size: 4088 bytes --]

I installed the patch and the bug was fixed.

Thanks.

On Sat, Jul 13, 2024 at 5:08 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Siyuan Chen <chansey97@gmail.com>
> > Date: Sat, 13 Jul 2024 01:52:11 +0800
> >
> > Reproduce steps:
> >
> > 1 Clone Emacs source code from https://github.com/emacs-mirror/emacs
> >
> > 2. Emacs -Q (Emacs 29.3 on Windows)
> >
> > 3. Open etc/NEWS.30 from the Emacs source code (by default Emacs will
> use emacs-news-mode)
> >
> > 4. M-x imenu and press TAB to complete
> >
> > Emacs shows 515 possible completions.
> >
> > 6. M-x emacs-news-view-mode
> >
> > 7. M-x imenu and press TAB to complete
> >
> > Emacs shows 202 possible completions.
> >
> > This is the problem, i.e. emacs-news-mode and emacs-news-view-mode
> return different imenu lists.
>
> This happens because emacs-news-view-mode derives from special-mode,
> and therefore has a different syntax table.  Which has the effect of
> duping imenu--generic-function into thinking some of the NEWS entries
> are in comments, and so it ignores them:
>
>                   ;; Insert the item unless it is already present.
>                   (unless (or (member item (cdr menu))
>                               (and imenu-generic-skip-comments-and-strings
>                                    (save-excursion
>                                      (goto-char start) (nth 8
> (syntax-ppss)))))
>                     (setcdr menu
>                             (cons item (cdr menu)))))
>
> Note the call to syntax-ppss there: it returns nil in 8th entry in
> emacs-news-mode when point is, e.g., on the final ^L in NEWS, but
> non-nil in emacs-news-view-mode.
>
> The simple and naïve change below seems to fix that.  The only problem
> with it is that one cannot move through the buffer with SPC, DEL, '<'
> etc.  For some reason I cannot get emacs-news-view-mode inherit the
> keymap of special-mode, nor even use its own keymap I defined in a
> defvar emacs-news-view-mode-map.  Stefan, any reason why a simple
> definition of emacs-news-view-mode-map doesn't work, as the ELisp
> manual promises it should (where it described define-derived-mode)?
>
> Also, any idea why syntax-ppss thinks the final ^L in NEWS, here:
>
>   ---
>   *** Emacs on MS-Windows now supports the ':stipple' face attribute.
>
>   ^L
>   ----------------------------------------------------------------------
>   This file is part of GNU Emacs.
>
> is in a comment when emacs-news-view-mode inherits from special-mode?
>
> Btw, I find our syntax-table introspection features not very helpful
> in these situations.  Once I figured out that syntax-ppss produces
> different results in these two modes, all the rest was pure guesswork,
> with no real way of investigating the differences and their reason(s).
> Do we have any ways of comparing two syntax tables and showing the
> differences in human-readable form?  Do we have a way of showing the
> source(s) f a syntax-table's information, like where it inherited from
> another and where it was modified by explicit modify-syntax-entry
> calls?  Do we have a way of making one mode use the syntax table of
> another mode without having a variable which holds the latter?  I
> needed all of these to investigate in this case, and was finally
> forced to treat the issue as a kind of "black box", making
> more-or-less random changes and looking at the results, which is very
> frustrating and a terrible waste of time...
>
>
> diff --git a/lisp/textmodes/emacs-news-mode.el
> b/lisp/textmodes/emacs-news-mode.el
> index 1dd017a..c87e7fa 100644
> --- a/lisp/textmodes/emacs-news-mode.el
> +++ b/lisp/textmodes/emacs-news-mode.el
> @@ -107,7 +107,7 @@ emacs-news-mode
>    (emacs-news--mode-common))
>
>  ;;;###autoload
> -(define-derived-mode emacs-news-view-mode special-mode "NEWS"
> +(define-derived-mode emacs-news-view-mode emacs-news-mode "NEWS"
>    "Major mode for viewing the Emacs NEWS file."
>    (setq buffer-read-only t)
>    (emacs-news--buttonize)
>

[-- Attachment #2: Type: text/html, Size: 4980 bytes --]

bug#72080: imenu returns different lists in emacs-news-mode and emacs-news-view-mode

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> For some reason I cannot get emacs-news-view-mode inherit the
> keymap of special-mode, nor even use its own keymap I defined in a
> defvar emacs-news-view-mode-map.  Stefan, any reason why a simple
> definition of emacs-news-view-mode-map doesn't work, as the ELisp
> manual promises it should (where it described define-derived-mode)?

Can't think of a reason, no.

> Also, any idea why syntax-ppss thinks the final ^L in NEWS, here:
>
>   ---
>   *** Emacs on MS-Windows now supports the ':stipple' face attribute.
>
>   ^L
>   ----------------------------------------------------------------------
>   This file is part of GNU Emacs.
>
> is in a comment when emacs-news-view-mode inherits from special-mode?

No, sorry.

> Do we have a way of making one mode use the syntax table of another
> mode without having a variable which holds the latter?

`C-h o` says:

    define-derived-mode is an autoloaded Lisp macro in ‘derived.el’.
    [...]
    KEYWORD-ARGS:
    [...]
               :syntax-table TABLE
                       Use TABLE instead of the default
                       (CHILD-syntax-table).  A nil value means to
                       simply use the same syntax-table as the parent.

so `:syntax-table` should be able to do that.


        Stefan

bug#72080: imenu returns different lists in emacs-news-mode and emacs-news-view-mode

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Siyuan Chen <chansey97@gmail.com>,  72080@debbugs.gnu.org
> Date: Sun, 14 Jul 2024 08:57:37 -0400
> 
> > For some reason I cannot get emacs-news-view-mode inherit the
> > keymap of special-mode, nor even use its own keymap I defined in a
> > defvar emacs-news-view-mode-map.  Stefan, any reason why a simple
> > definition of emacs-news-view-mode-map doesn't work, as the ELisp
> > manual promises it should (where it described define-derived-mode)?
> 
> Can't think of a reason, no.

But it doesn't work.  I'm puzzled why.  Any suggestions where to look?

> > Do we have a way of making one mode use the syntax table of another
> > mode without having a variable which holds the latter?
> 
> `C-h o` says:
> 
>     define-derived-mode is an autoloaded Lisp macro in ‘derived.el’.
>     [...]
>     KEYWORD-ARGS:
>     [...]
>                :syntax-table TABLE
>                        Use TABLE instead of the default
>                        (CHILD-syntax-table).  A nil value means to
>                        simply use the same syntax-table as the parent.
> 
> so `:syntax-table` should be able to do that.

Sure, but TABLE should be a variable here.  What I wanted was to tell
define-derived-mode to use the syntax table of a given other mode
which has no such variable.  Would something like this:

  :syntax-table (with-temp-buffer
                  (emacs-news-mode)
		  (syntax-table))

have worked?

bug#72080: imenu returns different lists in emacs-news-mode and emacs-news-view-mode

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Siyuan Chen <chansey97@gmail.com>,  72080@debbugs.gnu.org
> Date: Sun, 14 Jul 2024 08:57:37 -0400
> 
> > For some reason I cannot get emacs-news-view-mode inherit the
> > keymap of special-mode, nor even use its own keymap I defined in a
> > defvar emacs-news-view-mode-map.  Stefan, any reason why a simple
> > definition of emacs-news-view-mode-map doesn't work, as the ELisp
> > manual promises it should (where it described define-derived-mode)?
> 
> Can't think of a reason, no.
> 
> > Also, any idea why syntax-ppss thinks the final ^L in NEWS, here:
> >
> >   ---
> >   *** Emacs on MS-Windows now supports the ':stipple' face attribute.
> >
> >   ^L
> >   ----------------------------------------------------------------------
> >   This file is part of GNU Emacs.
> >
> > is in a comment when emacs-news-view-mode inherits from special-mode?
> 
> No, sorry.

OK, so I fixed that on the emacs-30 branch as best I could, and I'm
therefore closing this bug.