bug#73117: 30.0.90; Imenu missing entries when flattening by group

bug#73117: 30.0.90; Imenu missing entries when flattening by group

[Troy Brown]

This issue appears to be similar to the issue reported in 70846, but
this is specifically regarding when Imenu is configured to flatten
into "groups" (as opposed to "annotation" as was reported there).
When "imenu-flatten" is set to "group", I see an issue where nested
entries, with the same name but belonging to different parents, aren't
all displayed.

I've included an example below (based on the example menu
configuration described in 70846).  This example cycles through
flattening based on "index", "group" and "annotation" with the example
menu configuration.

For "prefix" and "annotation" configurations, it appears to work
correctly, as pressing "M-<down>" when the menu prompt is displayed, I
can see both entries identified in the "*Completions*" buffer.

However, when I do this with "imenu-flatten" set to "group" and press
"M-<down>" to display the completions window, the window indicates "2
possible completions" but only one is actually displayed and
selectable (i.e., the one under "Bar").  The menu entry "Foo" under
"Baz" is not displayed at all and it appears there is no way to select
it.

```
;; begin
(progn
  (require 'imenu)
  (dolist (flatten '(prefix group annotation))
    (setq imenu-flatten flatten)

    (imenu-choose-buffer-index (format "(%s) Index item: " flatten)
                               `(("Bar" . (("Foo" . ,(point-min-marker))))
                                 ("Baz" . (("Foo" . ,(point-max-marker))))))))
;; end
```

bug#73117: 30.0.90; Imenu missing entries when flattening by group

[Juri Linkov]

> This issue appears to be similar to the issue reported in 70846, but
> this is specifically regarding when Imenu is configured to flatten
> into "groups" (as opposed to "annotation" as was reported there).
> When "imenu-flatten" is set to "group", I see an issue where nested
> entries, with the same name but belonging to different parents, aren't
> all displayed.
>
> I've included an example below (based on the example menu
> configuration described in 70846).  This example cycles through
> flattening based on "index", "group" and "annotation" with the example
> menu configuration.
>
> For "prefix" and "annotation" configurations, it appears to work
> correctly, as pressing "M-<down>" when the menu prompt is displayed, I
> can see both entries identified in the "*Completions*" buffer.
>
> However, when I do this with "imenu-flatten" set to "group" and press
> "M-<down>" to display the completions window, the window indicates "2
> possible completions" but only one is actually displayed and
> selectable (i.e., the one under "Bar").  The menu entry "Foo" under
> "Baz" is not displayed at all and it appears there is no way to select
> it.
>
> ```
> ;; begin
> (progn
>   (require 'imenu)
>   (dolist (flatten '(prefix group annotation))
>     (setq imenu-flatten flatten)
>
>     (imenu-choose-buffer-index (format "(%s) Index item: " flatten)
>                                `(("Bar" . (("Foo" . ,(point-min-marker))))
>                                  ("Baz" . (("Foo" . ,(point-max-marker))))))))
> ;; end
> ```

Sorry for leaving out of documentation an unapparent mention
of `completions-group`.  We are discussing this currently at
https://mail.gnu.org/archive/html/emacs-devel/2024-08/msg00241.html
So a prerequisite would be to use `(setopt completions-group t)`.
But currently this should be mentioned in the docstring.

Also in the same discussion we came to conclusion that
`M-<down>` can't be used to select imenu items for
`annotation` and `group`.  So this limitation was
documented recently in the docstring of `imenu-flatten`:

  @@ -158,6 +158,9 @@ imenu-flatten
   with a suffix that is the section name to which it belongs.
   If the value is `group', split completion candidates into groups
   according to the sections.
  +Since the values `annotation' and `group' rely on text properties,
  +you can use them only by selecting candidates from the completions
  +buffer, not by typing in the minibuffer.

Otherwise, `group` should work nicely when using `<down>`
with `minibuffer-visible-completions`.

bug#73117: 30.0.90; Imenu missing entries when flattening by group

[Troy Brown]

On Sun, Sep 8, 2024 at 2:57 AM Juri Linkov <juri@linkov.net> wrote:
>
> > ```
> > ;; begin
> > (progn
> >   (require 'imenu)
> >   (dolist (flatten '(prefix group annotation))
> >     (setq imenu-flatten flatten)
> >
> >     (imenu-choose-buffer-index (format "(%s) Index item: " flatten)
> >                                `(("Bar" . (("Foo" . ,(point-min-marker))))
> >                                  ("Baz" . (("Foo" . ,(point-max-marker))))))))
> > ;; end
> > ```
>
> Sorry for leaving out of documentation an unapparent mention
> of `completions-group`.  We are discussing this currently at
> https://mail.gnu.org/archive/html/emacs-devel/2024-08/msg00241.html
> So a prerequisite would be to use `(setopt completions-group t)`.
> But currently this should be mentioned in the docstring.
>
> Also in the same discussion we came to conclusion that
> `M-<down>` can't be used to select imenu items for
> `annotation` and `group`.  So this limitation was
> documented recently in the docstring of `imenu-flatten`:
>
>   @@ -158,6 +158,9 @@ imenu-flatten
>    with a suffix that is the section name to which it belongs.
>    If the value is `group', split completion candidates into groups
>    according to the sections.
>   +Since the values `annotation' and `group' rely on text properties,
>   +you can use them only by selecting candidates from the completions
>   +buffer, not by typing in the minibuffer.
>
> Otherwise, `group` should work nicely when using `<down>`
> with `minibuffer-visible-completions`.

Hi Juri,

I did see the text in the diff above.  I assumed that was indicating
that you need to use the completions buffer to distinguish between the
identically named items (although after re-reading it, maybe it
affects the selection too).  I was using M-<down> to pop-up the
completions buffer so that I could see those distinctions.  Maybe
using `(setopt completion-auto-help t)` and hitting TAB is a better
method.  Either way, I'm just trying to display the completions
buffer.  This does work for the "annotations" setting as I see both
entries with the suffix annotation in the completions buffer.  Maybe
there is a different issue using M-<down> for the selection, but my
issue is that I can't get the completions buffer to even display all
of the items.

Just to be clear I'm running "emacs -Q" on the latest in the emacs-30
branch as of yesterday.  I ran the code snippet I posted in the
original email by running `eval-last-sexp` in the scratch buffer.

I did try adding your suggested settings to the above code snippet,
but they did not seem to improve anything with regards to what shows
up in the completions buffer:
```
(setopt completions-group t)
(setopt minibuffer-visible-completions t)
```

The following is what I see in the "*Completions*" buffer for all
three settings of `imenu-flatten` (i.e., "prefix", "group" and
"annotation").  Notice that the second output (which is the "group"
setting) only shows a single entry, the one for Bar/Foo.  The "Bar"
group is displayed and the "Foo" entry under it is also displayed, but
the "Baz" group is completely missing (even though it says "2 possible
completions").  This is the issue I'm having.

--8<---------------cut here---------------start------------->8---
Click or type RET on a completion to select it.
Type <right>, <left>, <down>, <up> to move point between completions.

2 possible completions:
Bar:Foo
Baz:Foo
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
Click or type RET on a completion to select it.
Type <right>, <left>, <down>, <up> to move point between completions.

2 possible completions:
     Bar
Foo
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
Click or type RET on a completion to select it.
Type <right>, <left>, <down>, <up> to move point between completions.

2 possible completions:
Foo (Bar)
Foo (Baz)
--8<---------------cut here---------------end--------------->8---

bug#73117: 30.0.90; Imenu missing entries when flattening by group

[Juri Linkov]

> Maybe using `(setopt completion-auto-help t)`
> and hitting TAB is a better method.

Or just using `(setopt imenu-eager-completion-buffer nil)`
because otherwise with its default value you need to hit
'?' (minibuffer-completion-help) instead of TAB
to show the completions buffer with two identical
completions in different groups.

> I did try adding your suggested settings to the above code snippet,
> but they did not seem to improve anything with regards to what shows
> up in the completions buffer:
>
> ```
> (setopt completions-group t)
> (setopt minibuffer-visible-completions t)
> ```

I'm so sorry, I forgot to check with `emacs -Q`, and there is
an additional important setting that affects the groups:

```
(setopt completions-group t)
(setopt completions-format 'vertical)
```

The default value of 'completions-format' is 'horizontal'
that doesn't support groups, and I don't know why.

Maybe Daniel (Cc:ed) could help us understand
why 'completion--insert-horizontal' doesn't display groups.

bug#73117: 30.0.90; Imenu missing entries when flattening by group

[Troy Brown]

On Sun, Sep 8, 2024 at 12:42 PM Juri Linkov <juri@linkov.net> wrote:
>
> > Maybe using `(setopt completion-auto-help t)`
> > and hitting TAB is a better method.
>
> Or just using `(setopt imenu-eager-completion-buffer nil)`
> because otherwise with its default value you need to hit
> '?' (minibuffer-completion-help) instead of TAB
> to show the completions buffer with two identical
> completions in different groups.
>

Ah, thanks!  Yes, I'm used to using Vertico so I'm not very familiar
with the default settings/bindings.  In fact that's how I first ran
into this issue using Vertico and Imenu with `imenu-flatten` set to
"group".

> I'm so sorry, I forgot to check with `emacs -Q`, and there is
> an additional important setting that affects the groups:
>
> ```
> (setopt completions-group t)
> (setopt completions-format 'vertical)
> ```
>
> The default value of 'completions-format' is 'horizontal'
> that doesn't support groups, and I don't know why.
>
> Maybe Daniel (Cc:ed) could help us understand
> why 'completion--insert-horizontal' doesn't display groups.

No worries, at least I know I'm not going crazy, lol.

Indeed, when I set `completions-format` to "vertical" it does show all
entries.  I then expanded this example to see what happens with 3
entries with both "horizontal" and "vertical" values for
`completions-format`.  With a three section menu (see example below),
and format set to "horizontal", it does show multiple groups
vertically.  However, even under this condition, the groups which had
identical entries only show up once (which seems like a bug).

The following example configures `imenu-flatten" to "group" and then
shows the menu, first with `completions-format` set to "horizontal"
and then "vertical".

```elisp
(progn
  (setopt completions-group t)
  (setopt imenu-eager-completion-buffer nil)
  (setopt minibuffer-visible-completions t)
  (setopt imenu-flatten 'group)

  (require 'imenu)

  (dolist (format '(horizontal vertical))
    (setopt completions-format format)

    (imenu-choose-buffer-index (format "(%s) Index item: " format)
                               `(("XYZ" . (("123" . ,(point-min-marker))))
                                 ("Bar" . (("Foo" . ,(point-min-marker))))
                                 ("Baz" . (("Foo" . ,(point-max-marker))))))))
```

--8<---------------cut here---------------start------------->8---
Click or type RET on a completion to select it.
Type <right>, <left>, <down>, <up> to move point between completions.

3 possible completions:
     XYZ
123
     Bar
Foo
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
Click or type RET on a completion to select it.
Type <right>, <left>, <down>, <up> to move point between completions.

3 possible completions:
     XYZ
123
     Bar
Foo
     Baz
Foo
--8<---------------cut here---------------end--------------->8---

bug#73117: 30.0.90; Imenu missing entries when flattening by group

[Daniel Mendler via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Juri Linkov <juri@linkov.net> writes:

> Maybe Daniel (Cc:ed) could help us understand
> why 'completion--insert-horizontal' doesn't display groups.

If I remember correctly, I didn't add group titles to the horizontal
formatting in order to keep the display more dense, with the goal of
fitting as many candidates on the screen as possible. In my Vertico
package, I am using a similar strategy, where group titles are not shown
for the flat and grid display modes. In any case, there shouldn't be a
technical reason which prevents us from showing the titles also for the
horizontal display.

Daniel

bug#73117: 30.0.90; Imenu missing entries when flattening by group

[Troy Brown]

On Sun, Sep 8, 2024 at 2:28 PM Daniel Mendler <mail@daniel-mendler.de> wrote:
>
> Juri Linkov <juri@linkov.net> writes:
>
> > Maybe Daniel (Cc:ed) could help us understand
> > why 'completion--insert-horizontal' doesn't display groups.
>
> If I remember correctly, I didn't add group titles to the horizontal
> formatting in order to keep the display more dense, with the goal of
> fitting as many candidates on the screen as possible. In my Vertico
> package, I am using a similar strategy, where group titles are not shown
> for the flat and grid display modes. In any case, there shouldn't be a
> technical reason which prevents us from showing the titles also for the
> horizontal display.

It appears to be this line in `completion--insert-horizontal` which is
causing all the trouble.  When I comment that out, I get the output
I'm expecting.  Looking at the difference between
`completion--insert-horizontal` and `completion--insert-vertical`,
that check happens after the check for `group-fun`.  Maybe
`completion--insert-horizontal` should be restructured similarly?

```elisp
      (unless (equal last-string str) ; Remove (consecutive) duplicates.
```

bug#73117: 30.0.90; Imenu missing entries when flattening by group

[Troy Brown]

On Sun, Sep 8, 2024 at 3:25 PM Troy Brown <brownts@troybrown.dev> wrote:
>
> It appears to be this line in `completion--insert-horizontal` which is
> causing all the trouble.  When I comment that out, I get the output
> I'm expecting.  Looking at the difference between
> `completion--insert-horizontal` and `completion--insert-vertical`,
> that check happens after the check for `group-fun`.  Maybe
> `completion--insert-horizontal` should be restructured similarly?
>
> ```elisp
>       (unless (equal last-string str) ; Remove (consecutive) duplicates.
> ```

Daniel, any thoughts on my comment above?

It appears completion--insert-horizontal incorrectly filters
consecutive strings when they belong to different groups.  From what I
can tell, it appears completion--insert-horizontal does support
groups, but just displays them vertically.  However, it does contain
the above problem.  I also notice this issue coming through in my use
of consult-imenu.

bug#73117: 30.0.90; Imenu missing entries when flattening by group

[Juri Linkov]

> The following example configures `imenu-flatten" to "group" and then
> shows the menu, first with `completions-format` set to "horizontal"
> and then "vertical".
>
> ```elisp
> (progn
>   (setopt completions-group t)
>   (setopt imenu-eager-completion-buffer nil)
>   (setopt minibuffer-visible-completions t)
>   (setopt imenu-flatten 'group)
>
>   (require 'imenu)
>
>   (dolist (format '(horizontal vertical))
>     (setopt completions-format format)

Any fixes for completion--insert-horizontal should be pushed
only to master, not to the release branch.

But for the release branch we should push the documentation fixes
that will mention all prerequisites for this feature to work
correctly:

diff --git a/lisp/imenu.el b/lisp/imenu.el
index 8f1b1f22a67..13440cfb626 100644
--- a/lisp/imenu.el
+++ b/lisp/imenu.el
@@ -158,9 +158,18 @@ imenu-flatten
 with a suffix that is the section name to which it belongs.
 If the value is `group', split completion candidates into groups
 according to the sections.
+
 Since the values `annotation' and `group' rely on text properties,
 you can use them only by selecting candidates from the completions
-buffer, not by typing in the minibuffer.
+buffer, not by typing in the minibuffer.  This also means that
+if you use `minibuffer-next-completion' (`M-<down>') to select
+a completion while point stays in the minibuffer, you need
+to customize `minibuffer-completion-auto-choose' to nil that
+doesn't insert completion candidates to the minibuffer.
+Also note that for using the value `group' you need to customize
+`completions-group' to the value t, and `completions-format'
+to the value `vertical'.
+
 Any other value is treated as `prefix'.
 
 The value of `imenu-level-separator', a string, is used to separate