[FR] Allow flattened imenu index (was: [PATCH] Add new option 'org-imenu-flatten')

[FR] Allow flattened imenu index (was: [PATCH] Add new option 'org-imenu-flatten')

[Ihor Radchenko]

Morgan Smith <Morgan.J.Smith@outlook.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> Have you considered adding a "flatten" option to imenu itself?
>> That way, you could automatically get the functionality for free
>> everywhere, not just in Org mode.
>
> I have considered that but gave up with minimal investigation because it
> seemed harder then this solution.  It's possible imenu did actually have
> this functionality sometime before 1998 (see commit
> fe2908be7b09f4c765ebdaf16fe07b0a77f78ba8).
>
> The doc-view imenu-flatten stuff was added 2022-09-28 (see commit
> fe002cc8ce38efb256a2a60660ee626c2b2cdf81).  This makes me feel like
> maybe that person thought adding it to imenu directly would be hard.
>
> I might at some point investigate doing that but likely not soon.  Also
> if that feature was ever added, it would still be compatible with the
> patch I sent.  For those reasons, I advocate my patch should still be
> applied even though it is clear that it is a sub-optimal solution.

I'd prefer to ask Emacs upstream first.

We are discussing adding a new feature to Org imenu - an option to
flatten the menu, so that all the nested index entries are displayed at
top level.

This feature is also present in doc-view via `doc-view-imenu-flatten',
and in python.el via `python-imenu-create-flat-index'

I am wondering if it makes more sense to add this "flatten" option
globally into imenu instead.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [FR] Allow flattened imenu index (was: [PATCH] Add new option 'org-imenu-flatten')

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: emacs-orgmode@gnu.org, 58131@debbugs.gnu.org, emacs-devel@gnu.org 
> Date: Sat, 09 Dec 2023 10:57:15 +0000
> 
> Morgan Smith <Morgan.J.Smith@outlook.com> writes:
> 
> > Ihor Radchenko <yantar92@posteo.net> writes:
> >
> >> Have you considered adding a "flatten" option to imenu itself?
> >> That way, you could automatically get the functionality for free
> >> everywhere, not just in Org mode.
> >
> > I have considered that but gave up with minimal investigation because it
> > seemed harder then this solution.  It's possible imenu did actually have
> > this functionality sometime before 1998 (see commit
> > fe2908be7b09f4c765ebdaf16fe07b0a77f78ba8).
> >
> > The doc-view imenu-flatten stuff was added 2022-09-28 (see commit
> > fe002cc8ce38efb256a2a60660ee626c2b2cdf81).  This makes me feel like
> > maybe that person thought adding it to imenu directly would be hard.
> >
> > I might at some point investigate doing that but likely not soon.  Also
> > if that feature was ever added, it would still be compatible with the
> > patch I sent.  For those reasons, I advocate my patch should still be
> > applied even though it is clear that it is a sub-optimal solution.
> 
> I'd prefer to ask Emacs upstream first.
> 
> We are discussing adding a new feature to Org imenu - an option to
> flatten the menu, so that all the nested index entries are displayed at
> top level.
> 
> This feature is also present in doc-view via `doc-view-imenu-flatten',
> and in python.el via `python-imenu-create-flat-index'
> 
> I am wondering if it makes more sense to add this "flatten" option
> globally into imenu instead.

I'm not sure I understand what you are asking.  As we don't seem to
have an active maintainer of imenu on board, more details are needed
to understand the request.  Of course, patches are even more welcome.

Also, should this be a new bug report? the one mentioned in the CC is
already closed and archived.

Thanks.

Re: [FR] Allow flattened imenu index (was: [PATCH] Add new option 'org-imenu-flatten')

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I am wondering if it makes more sense to add this "flatten" option
>> globally into imenu instead.
>
> I'm not sure I understand what you are asking.  As we don't seem to
> have an active maintainer of imenu on board, more details are needed
> to understand the request.  Of course, patches are even more welcome.

Normally, `imenu' supports nested menus, when users select the target
location in steps, like item3 <RET> -> item3.1 <RET> -> ...

What is proposed is to list all the sub-menus together, as an option, so
that the users can choose, for example, item.3.1 directly, without going
through parent item3.

> Also, should this be a new bug report? the one mentioned in the CC is
> already closed and archived.

Maybe. I was just not sure how I can create a new bug report, while
still replying to Org mode feature request.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [FR] Allow flattened imenu index

[Juri Linkov]

> Normally, `imenu' supports nested menus, when users select the target
> location in steps, like item3 <RET> -> item3.1 <RET> -> ...
>
> What is proposed is to list all the sub-menus together, as an option, so
> that the users can choose, for example, item.3.1 directly, without going
> through parent item3.

Shouldn't each sub-item keep the parent menu item as a prefix?
So a nested menu like

  menu1
    sub-item1
    sub-item2

could be flattened to

  menu1 -> sub-item1
  menu1 -> sub-item2

Or this should be optional?

Re: [FR] Allow flattened imenu index

[João Távora]

On Sat, Dec 9, 2023 at 5:39 PM Juri Linkov <juri@linkov.net> wrote:

>   menu1
>     sub-item1
>     sub-item2
>
> could be flattened to
>
>   menu1 -> sub-item1
>   menu1 -> sub-item2


By the way, this seems to be exactly what the breadcrumb-jump command
in my breadcrumb.el package does.  Goes reasonably well with a flex/fuzzy
completion style.

You can look at it for an implementation idea.  Just be sure to do this
flattening at the presentation level (i.e. M-x imenu), not at the
internal representation level.

It shouldn't be needed here at all, but in case anyone's thinking
about it, please avoid messing with imenu's internal representation of
hierarchies as that structure is relied upon by many extensions (not just
mine, but several others).  Even certain things supported by certain
imenu-presenting frontends (like "special elements") are not supported by
other frontends.  It's a bit of a mess.  The symbols holding/describing
this representation (imenu--index-alist, maybe others) are incorrectly
named '--' but they are most definitely externally visible and used
customization points.

João

Re: [FR] Allow flattened imenu index

[Juri Linkov]

>>   menu1
>>     sub-item1
>>     sub-item2
>>
>> could be flattened to
>>
>>   menu1 -> sub-item1
>>   menu1 -> sub-item2
>
> By the way, this seems to be exactly what the breadcrumb-jump command
> in my breadcrumb.el package does.  Goes reasonably well with a flex/fuzzy
> completion style.

It would be great to have some form of breadcrumb-jump in imenu.el
since it's useful on its own even for someone who doesn't use breadcrumbs.

> You can look at it for an implementation idea.  Just be sure to do this
> flattening at the presentation level (i.e. M-x imenu), not at the
> internal representation level.

Here are some observations while testing on
emacs/test/manual/etags/ruby-src/test.rb.

Both ruby-mode and ruby-ts-mode provide a list that is already flat:

ruby-mode:
  ModuleExample
  ModuleExample#ModuleExample.module_class_method
  ModuleExample#module_instance_method
  ModuleExample::ClassExample
  ModuleExample::ClassExample#+
  ModuleExample::ClassExample#ClassExample.class_method
  ModuleExample::ClassExample#instance_method

ruby-ts-mode:
  ModuleExample
  ModuleExample#module_instance_method
  ModuleExample.module_class_method
  ModuleExample::ClassExample
  ModuleExample::ClassExample#+
  ModuleExample::ClassExample#instance_method
  ModuleExample::ClassExample.class_method

When eglot is enabled then imenu-create-index-function returns a tree.

What is interesting is that breadcrumb-jump already correctly handles both
a flat list and a tree:

a flat list in breadcrumb-jump completions is exactly the same as in 'imenu':
  ModuleExample
  ModuleExample#module_instance_method
  ModuleExample.module_class_method
  ModuleExample::ClassExample
  ModuleExample::ClassExample#+
  ModuleExample::ClassExample#instance_method
  ModuleExample::ClassExample.class_method

a tree from eglot in breadcrumb-jump completions:
  Class > ModuleExample > ClassExample
  Method > ClassExample > class_method
  Method > ModuleExample > module_class_method
  Method > ModuleExample > module_instance_method
  Method > ModuleExample::ClassExample > +
  Method > ModuleExample::ClassExample > instance_method
  Module >  > ModuleExample

is still usable even without special characters like "#".

> It shouldn't be needed here at all, but in case anyone's thinking
> about it, please avoid messing with imenu's internal representation of
> hierarchies as that structure is relied upon by many extensions (not just
> mine, but several others).  Even certain things supported by certain
> imenu-presenting frontends (like "special elements") are not supported by
> other frontends.  It's a bit of a mess.  The symbols holding/describing
> this representation (imenu--index-alist, maybe others) are incorrectly
> named '--' but they are most definitely externally visible and used
> customization points.

Indeed, it's unfortunate that imenu--make-index-alist and imenu--index-alist
are named as internal.

Re: [FR] Allow flattened imenu index

[João Távora]

On Mon, Dec 11, 2023 at 5:21 PM Juri Linkov <juri@linkov.net> wrote:

> > By the way, this seems to be exactly what the breadcrumb-jump command
> > in my breadcrumb.el package does.  Goes reasonably well with a flex/fuzzy
> > completion style.
>
> It would be great to have some form of breadcrumb-jump in imenu.el
> since it's useful on its own even for someone who doesn't use breadcrumbs.

You can copy it over, it  doesn't really need breadcrumbs, though,
it just needs imenu.

The only things I'm seeing in its implementation in a penchant
for a particular text property `breadcrumb-region`, which
has information that imenu's structure didn't have a good
place to place.  But it's not required, as you noticed, I  suppose
it just gives it increased accuracy.

> When eglot is enabled then imenu-create-index-function returns a tree.
>
> What is interesting is that breadcrumb-jump already correctly handles both
> a flat list and a tree:

Yes, that's the point.  Though when using Eglot, the exact tree
structure -- of any -- depends on the language server.

João

Re: [FR] Allow flattened imenu index

[Dmitry Gutov]

On 11/12/2023 19:20, Juri Linkov wrote:
> a tree from eglot in breadcrumb-jump completions:
>    Class > ModuleExample > ClassExample
>    Method > ClassExample > class_method
>    Method > ModuleExample > module_class_method
>    Method > ModuleExample > module_instance_method
>    Method > ModuleExample::ClassExample > +
>    Method > ModuleExample::ClassExample > instance_method
>    Module >  > ModuleExample
> 
> is still usable even without special characters like "#".

Until a class contains both an instance and a class method with the same 
name, I suppose.

Re: [FR] Allow flattened imenu index

[João Távora]

On Mon, Dec 11, 2023 at 5:43 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > is still usable even without special characters like "#".
>
> Until a class contains both an instance and a class method with the same
> name, I suppose.

If that happens, the LS will have to provide to distinct paths
to these two distinct items and breadcrumb will have two distinct
strings, so no need to invent #-things.

Re: [FR] Allow flattened imenu index

[Dmitry Gutov]

On 11/12/2023 20:00, João Távora wrote:
> On Mon, Dec 11, 2023 at 5:43 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
> 
>>> is still usable even without special characters like "#".
>> Until a class contains both an instance and a class method with the same
>> name, I suppose.
> If that happens, the LS will have to provide to distinct paths
> to these two distinct items and breadcrumb will have two distinct
> strings, so no need to invent #-things.

It's no invention, it's docstring syntax for reference to a method.

But if the LS will produce distinct strings, good.

Re: [FR] Allow flattened imenu index

[João Távora]

On Mon, Dec 11, 2023 at 7:24 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> It's no invention, it's docstring syntax for reference to a method.

In Ruby right?

> But if the LS will produce distinct strings, good.

All imenu backends, at least all the ones I've seen, produce
trees, not strings.  If you collect all the paths from the root to
all the nodes into lists and make strings thereof, the resulting
set will always be a distinct strings.  So I don't understand the
problem you were surfacing: any particular imenu backend in mind?

João

Re: [FR] Allow flattened imenu index

[Dmitry Gutov]

On 12/12/2023 01:10, João Távora wrote:
> On Mon, Dec 11, 2023 at 7:24 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
>> It's no invention, it's docstring syntax for reference to a method.
> In Ruby right?

Yes, in RDoc. Or Javadoc, for that matter. Probably some other languages 
as well.

>> But if the LS will produce distinct strings, good.
> All imenu backends, at least all the ones I've seen, produce
> trees, not strings.

There are exceptions, like the previously mentioned one.

> If you collect all the paths from the root to
> all the nodes into lists and make strings thereof, the resulting
> set will always be a distinct strings.  So I don't understand the
> problem you were surfacing: any particular imenu backend in mind?

Do you have an example of such tree produced by Eglot when a class 
contains an instance and a singleton method with the same name?

Re: [FR] Allow flattened imenu index

[João Távora]

On Mon, Dec 11, 2023 at 11:23 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> >> But if the LS will produce distinct strings, good.
> > All imenu backends, at least all the ones I've seen, produce
> > trees, not strings.
>
> There are exceptions, like the previously mentioned one.

What is the imenu backend that produces strings?  A list of
strings is still a tree, albeit very flat.  Some LS's produce
such things.  Are there imenu backends that produce lists of
strings with duplicates?   Where, when, how is that not a
problem with the existing imenu UI already, and have we heard
of it all these years?

> > If you collect all the paths from the root to
> > all the nodes into lists and make strings thereof, the resulting
> > set will always be a distinct strings.  So I don't understand the
> > problem you were surfacing: any particular imenu backend in mind?
>
> Do you have an example of such tree produced by Eglot when a class
> contains an instance and a singleton method with the same name?

No.  I thought you did, else why would you bring it up?  Grab a
Ruby LS and try it.  Or maybe say what the ruby-mode imenu backend
does? No rush, but as usual I think there's no point discussing
odd conjectures without something palpable in front.

Thanks,
João

Re: [FR] Allow flattened imenu index

[Dmitry Gutov]

On 12/12/2023 01:35, João Távora wrote:
> Or maybe say what the ruby-mode imenu backend
> does?

Juri just posted a long example of what ruby-mode does (and ruby-ts-mode 
too) in his last message in this thread.

Re: [FR] Allow flattened imenu index

[João Távora]

On Mon, Dec 11, 2023 at 11:41 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 12/12/2023 01:35, João Távora wrote:
> > Or maybe say what the ruby-mode imenu backend
> > does?
>
> Juri just posted a long example of what ruby-mode does (and ruby-ts-mode
> too) in his last message in this thread.

OK, and doesn't it answer _your_ question?  With Eglot

Method > ModuleExample::ClassExample > instance_method
Method > ClassExample > class_method

If both methods were named "foo" no problem, right?
When run without Eglot, also no problem, no duplicate strings,
right?

??

Re: [FR] Allow flattened imenu index

[Dmitry Gutov]

On 12/12/2023 01:48, João Távora wrote:
> On Mon, Dec 11, 2023 at 11:41 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
>> On 12/12/2023 01:35, João Távora wrote:
>>> Or maybe say what the ruby-mode imenu backend
>>> does?
>> Juri just posted a long example of what ruby-mode does (and ruby-ts-mode
>> too) in his last message in this thread.
> OK, and doesn't it answer_your_  question?  With Eglot
> 
> Method > ModuleExample::ClassExample > instance_method
> Method > ClassExample > class_method
> 
> If both methods were named "foo" no problem, right?

But this doesn't look right: if 'class_method' is in a class called 
ModuleExample::ClassExample, shouldn't its entry look like

   Method > ModuleExample::ClassExample > class_method

as well?

And then, if both 'instance_method' and 'class_method' actually are 
called 'foo', then we'd have a problem.

This entry from his message also looks odd:

   Module >  > ModuleExample

Re: [FR] Allow flattened imenu index

[João Távora]

On Mon, Dec 11, 2023 at 11:54 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
> On 12/12/2023 01:48, João Távora wrote:
> > On Mon, Dec 11, 2023 at 11:41 PM Dmitry Gutov<dmitry@gutov.dev>  wrote:
> >> On 12/12/2023 01:35, João Távora wrote:
> >>> Or maybe say what the ruby-mode imenu backend
> >>> does?
> >> Juri just posted a long example of what ruby-mode does (and ruby-ts-mode
> >> too) in his last message in this thread.
> > OK, and doesn't it answer_your_  question?  With Eglot
> >
> > Method > ModuleExample::ClassExample > instance_method
> > Method > ClassExample > class_method
> >
> > If both methods were named "foo" no problem, right?
>
> But this doesn't look right: if 'class_method' is in a class called
> ModuleExample::ClassExample, shouldn't its entry look like
>
>    Method > ModuleExample::ClassExample > class_method
>
> as well?

Take it up with the LS.  I didn't write it.

> And then, if both 'instance_method' and 'class_method' actually are
> called 'foo', then we'd have a problem.

Bizarrely, you wouldn't have a tree to start with, which probably
explains why the devs of the LS Juri is using opted for that
layout.

> This entry from his message also looks odd:
>
>    Module >  > ModuleExample

If you can reproduce it, show the repro.

João

Re: [FR] Allow flattened imenu index

[Ihor Radchenko]

Juri Linkov <juri@linkov.net> writes:

>> It shouldn't be needed here at all, but in case anyone's thinking
>> about it, please avoid messing with imenu's internal representation of
>> hierarchies as that structure is relied upon by many extensions (not just
>> mine, but several others).  Even certain things supported by certain
>> imenu-presenting frontends (like "special elements") are not supported by
>> other frontends.  It's a bit of a mess.  The symbols holding/describing
>> this representation (imenu--index-alist, maybe others) are incorrectly
>> named '--' but they are most definitely externally visible and used
>> customization points.
>
> Indeed, it's unfortunate that imenu--make-index-alist and imenu--index-alist
> are named as internal.

Should they be renamed to `imenu-make-index-alist' and `imenu-index-alist' then?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [FR] Allow flattened imenu index

[João Távora]

On Mon, Dec 11, 2023 at 7:27 PM Ihor Radchenko <yantar92@posteo.net> wrote:

> > Indeed, it's unfortunate that imenu--make-index-alist and imenu--index-alist
> > are named as internal.
>
> Should they be renamed to `imenu-make-index-alist' and `imenu-index-alist' then?

Maybe, but it's much more urgent IMO to review the representation
they describe, since it's all over the place.  Ideally do this
before any rename.

Incidentally, I just remembered why breadcrumb-jump uses the
'breadcrumb-region' text property cookie: it's because without it
the current imenu tree structure has no way to for backends to
store locations to non-leaf nodes, which are often among the places
you want to jump to.

text-properties + standard cons trees + "special elements" is a
bit of a ugly mess, but OTOH it works and is backwards compatible
to existing frontends and backends.

So one of the possibilities that whoever conducts this review
may consider is an 'imenu-region' text property similar to
'breadcrumb-region', describe it imenu--index-alist, and
make M-x imenu take advantage of it.

João

Re: [FR] Allow flattened imenu index

[Spencer Baugh]

Ihor Radchenko <yantar92@posteo.net> writes:
> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> I am wondering if it makes more sense to add this "flatten" option
>>> globally into imenu instead.
>>
>> I'm not sure I understand what you are asking.  As we don't seem to
>> have an active maintainer of imenu on board, more details are needed
>> to understand the request.  Of course, patches are even more welcome.
>
> Normally, `imenu' supports nested menus, when users select the target
> location in steps, like item3 <RET> -> item3.1 <RET> -> ...
>
> What is proposed is to list all the sub-menus together, as an option, so
> that the users can choose, for example, item.3.1 directly, without going
> through parent item3.

I would love to have this feature.  I have wanted this for OCaml, where
the default imenu index produced by the "merlin" tool is annoyingly
deeply nested, which makes it awkward to use.

I have an alternative proposal though, which might be better: We could
enhance the imenu completion table to understand completion boundaries.
Then the imenu hierarchy could be completed over in a single
completing-read instead of a sequence of multiple completing-reads.  It
would work the same way as read-file-name.

So, for example, if the completion boundary was /, and we had a hierarchy
containing these elements:
aaa/bbb/ddd
aaa/bbb/eee
aaa/ccc/eee
aaa/ccc/fff

You could choose aaa/bbb/ddd with this sequence of minibuffer contents:
(input in [brackets], point at |)

[a]
a|
[<TAB>]
aaa/| ; completion now shows bbb and ccc as options
[b]
aaa/b/d|
[TAB]
aaa/bbb/ddd|
[RET]

Alternatively, you could choose aaa/bbb/eee with this sequence:

[*/*/e]
*/*/e|
[TAB]
aaa/|/eee ; completion now shows bbb and ccc as options, point is at |
[b]
aaa/b|/eee
[TAB]
aaa/bbb/eee|
[RET]

To be very clear, these completion features already exist (and are
enabled by default!), all we would need to do is enhance the imenu
completion table to understand completion boundaries, and then imenu
would have access to these features too.

This new completion table could be turned on by default, since it would
be strictly more powerful than the current imenu UI.

Plus, if some alternative completion UI wanted to flatten the hierarchy
anyway, this more advanced imenu completion table would allow that.

Re: [FR] Allow flattened imenu index

[Dmitry Gutov]

On 15/12/2023 01:11, Spencer Baugh wrote:
> I have an alternative proposal though, which might be better: We could
> enhance the imenu completion table to understand completion boundaries.
> Then the imenu hierarchy could be completed over in a single
> completing-read instead of a sequence of multiple completing-reads.  It
> would work the same way as read-file-name.
> 
> So, for example, if the completion boundary was /, and we had a hierarchy
> containing these elements:
> aaa/bbb/ddd
> aaa/bbb/eee
> aaa/ccc/eee
> aaa/ccc/fff
> 
> You could choose aaa/bbb/ddd with this sequence of minibuffer contents:
> (input in [brackets], point at |)
> 
> [a]
> a|
> [<TAB>]
> aaa/| ; completion now shows bbb and ccc as options
> [b]
> aaa/b/d|
> [TAB]
> aaa/bbb/ddd|
> [RET]
> 
> Alternatively, you could choose aaa/bbb/eee with this sequence:
> 
> [*/*/e]
> */*/e|
> [TAB]
> aaa/|/eee ; completion now shows bbb and ccc as options, point is at |
> [b]
> aaa/b|/eee
> [TAB]
> aaa/bbb/eee|
> [RET]
> 
> To be very clear, these completion features already exist (and are
> enabled by default!), all we would need to do is enhance the imenu
> completion table to understand completion boundaries, and then imenu
> would have access to these features too.
> 
> This new completion table could be turned on by default, since it would
> be strictly more powerful than the current imenu UI.
> 
> Plus, if some alternative completion UI wanted to flatten the hierarchy
> anyway, this more advanced imenu completion table would allow that.

"Flattening" a completion table with boundaries is not too easy, though. 
The code that wants to do that (e.g. for vertico or Ido to show all 
entries flat right away) would need to perform a similar "traversal of 
the tree" to what they do now, only with some more advanced logic of 
examining c-t boundaries.

Even so, your proposal would be an improvement for the default UI - one 
continued completion instead of a loop of completing-read's.