Possible minibuffer completion enhancements

Possible minibuffer completion enhancements

[Eshel Yaron]

Hello Emacs,

I have a few enhancements in my working branch that revolve around
inspecting and modifying minibuffer completion settings interactively.
These additions (described below) are backward compatible, documented,
and, at times, even useful.  So I propose adding them to upstream.

You can try out and examine these changes by cloning from
https://git.sr.ht/~eshel/emacs and building Emacs as usual.

[ git.sr.ht has been having some connectivity issues in the past few
  days, to put it mildly, so in case it doesn't work, you can also find
  the code at git://git.eshelyaron.com/emacs.git ]

If any of this looks like it might be appropriate for upstream Emacs,
I'll format and submit the relevant parts as individual patches.

There are five additions that can be considered, together or separately:

1. Interactively narrow (restrict) the list of minibuffer completions

   This lets you filter the list of possible completions.  A new Info
   node "Completions Narrowing" describes this feature in full:

     When there are many possible completion candidates, you may want to
     narrow the selection to a smaller subset by filtering out some of the
     options.  You can use the commands ‘C-x n n’ and ‘C-x n m’ in the
     minibuffer to restrict the completions list.  Narrowing the list of
     minibuffer completions is different from narrowing buffers (*note
     Narrowing::), although the two are conceptually related since both allow
     you to focus on some part of a larger whole.

     ‘C-x n n’ (‘minibuffer-narrow-completions-to-current’) restricts the
     list of possible completions to only include candidates that match the
     current minibuffer input.  This command clears the minibuffer so you can
     type another (partial) input and complete it with completion candidates
     that also match the previous input.  If you call this command with a
     negative prefix argument (‘C-- C-x n n’), it instead excludes all
     matches for the current input from subsequent completions.

     ‘C-x n m’ (‘minibuffer-narrow-completions’) is similar to ‘C-x n n’,
     but more versatile.  This command restricts the list of possible
     completions in different ways, depending on what kind of completion
     candidates you're dealing with.  For example, commands that read a
     symbol name from the minibuffer, such as ‘C-h f’ and ‘C-h o’ (*note Name
     Help::), let you restrict the completions list with ‘C-x n m’ to show
     only symbols with a given property.  *Note (elisp)Symbol Properties::.
     Similarly, commands that read a buffer name, such as ‘C-x b’ (*note
     Select Buffer::), make ‘C-x n m’ restrict the completions list by
     candidate buffer major mode.  When the command does not provide a
     specific way of restricting completion candidates, ‘C-x n m’ prompts you
     for a regular expression and narrows the completions list to only
     include candidates which match that regular expression.  *Note
     Regexps::.

     When you narrow the completions list with ‘C-x n n’ or with ‘C-x n m’,
     Emacs extends the completions heading line with a description of the
     restriction that is currently in effect (*note Completions Heading
     Line::).  The mode line of the ‘*Completions*’ buffer also indicates the
     restriction with the text ‘CompsNarrow’.  You can apply multiple
     restrictions one after the other to narrow the completions list
     incrementally.  For example, typing ‘M-x C-x n m foo <RET> C-x n m bar
     <RET>’ shows only commands that match both ‘foo’ and ‘bar’ in the
     completions list.

     Use ‘C-x n w’ (‘minibuffer-widen-completions’) in the minibuffer to
     remove the restrictions on the list of possible completions that you set
     with ‘C-x n n’ or with ‘C-x n m’.  ‘C-x n w’ prompts you for the
     description of a current completions restriction, and removes the
     corresponding restriction.  The default candidate is the most recent
     restriction, and you can use completion to select other restriction
     descriptions.  You can even specify multiple restrictions to remove at
     once, by separating their descriptions with commas in the minibuffer.
     If there is only one restriction to begin with, ‘C-x n w’ removes it
     without prompting.  If you invoke this command with a prefix argument
     (‘C-u C-x n w’), it removes all restrictions without prompting,
     regardless of how many there are.

2. Interactively sort the list of minibuffer completions

   This lets you change the order of the completions list by invoking a
   command in the minibuffer.  Here's the gist of it:

     ‘C-x C-v’ (‘minibuffer-sort-completions’) changes the order of the
     completions list.  By default, Emacs sorts the list of possible
     completion candidates in the order that you specify in user option
     ‘completions-sort’ (*note Completion Options::).  This command lets you
     change the order of the current completions list interactively.  You can
     invoke it with a negative prefix argument (‘C-- C-x C-v’) to reverse the
     current order.  The user option ‘minibuffer-completions-sort-orders’
     determines which orders this command suggests for sorting the
     completions list.  By default, this includes alphabetical sorting,
     sorting by candidate position in the minibuffer history, and no sorting
     at all.  Some commands that use minibuffer completion also provide
     additional sorting options that are specifically useful with their
     completion candidates.  For example, during file name completion, as in
     ‘C-x C-f’ (*note Visiting::), you can use ‘C-x C-v’ to sort candidate
     file names chronologically by their last modified time.

   Another possibly interesting detail is that the completions heading
   line indicates a non-default sorting order; for example, it might say
   "70 possible completions, sorted alphabetically...".  This is
   customizable via the existing `completions-header-format` option,
   which is extended accordingly (in a backward compatible manner).

3. Interactively set the completion styles for the current minibuffer

   This lets you modify the completion styles that the current
   minibuffer makes use of, on the fly:

     ‘C-x /’ (minibuffer-set-completion-styles) lets you set the
     completion styles for the current minibuffer.  *Note Completion
     Styles::.  This command prompts you for a list of completion styles, and
     sets that list as the effective completion styles for following
     completion operations in the current minibuffer.  With a plain prefix
     argument (‘C-u C-x /’), it instead discards all changes that you made to
     the current completion styles.  With a zero numeric prefix argument
     (‘C-0 C-x /’), it keeps all current completion styles except the style
     that produced that current completions list.  Conversely, a numeric
     prefix argument of one (‘C-1 C-x /’) says to keep only the completion
     style that produced the current completions list, disabling other
     completion styles for the current minibuffer.

     ...

     The mode line of the ‘*Completions*’ buffer indicates which
     completion style produced the listed completion candidates, by showing
     the name of that style.  (For example, the mode line says
     ‘Completions[basic]’ when the ‘basic’ completion style is in effect.)
     You can hover over the mode line style indicator with the mouse to see
     its full description.

4. Interactively replace the separator in `completing-read-multiple`

   This lets you examine and change the regular expression that matches
   input separators in a `completing-read-multiple` minibuffer:

     When displaying the completions list for ‘completing-read-multiple’,
     the mode line of the ‘*Completions*’ buffer includes an indicator that
     says ‘Multi’.  Hovering over that indicator with the mouse shows
     help about the current input separator.

     ...

     [‘crm-change-separator’], bound to ‘C-x ,’ in the minibuffer during
     ‘completing-read-multiple’, changes the current input separator.
     It prompts for a new separator regular expression, and sets the
     local value of ‘crm-separator’ to that regular expression.  With a
     prefix argument, this command also prompts for a replacement string
     (that should match the new separator) and replaces all of the
     existing separators in the minibuffer with that replacement string.

5. Cycling completions convenience commands

   This adds a dedicated command for cycling completions so you can make
   use of both cycling and the regular completion behavior at any time.
   Another new command lets you restore the minibuffer contents to the
   partial input you had when you started cycling:

     ‘C-o’ (‘minibuffer-cycle-completion’) cycles among the list of
     possible completions.  The first time you hit ‘C-o’, it expands your
     partial input in the minibuffer to the first matching completion
     candidate.  Another ‘C-o’ replaces the minibuffer contents with the next
     completion candidate, and repeating ‘C-o’ lets you cycle among all
     completions for your initial input, wrapping around when you reach the
     end of the list.  While you're cycling, Emacs remembers the initial
     partial input you started with, and the corresponding set of completion
     candidates.  If you edit a completion candidate in the minibuffer after
     cycling to it, that tells Emacs to forget about your previous partial
     input and compute a new set of completion candidates based on your new
     input the next time you hit ‘C-o’.  You can invoke ‘C-o’ with a numeric
     prefix argument N to cycle N candidates forward at once.  A negative N
     cycles backward instead.  A prefix argument of zero (‘C-0 C-o’) switches
     the cycling direction, so the next ‘C-o’ presses cycle backward.

     ‘C-l’ (‘minibuffer-restore-completion-input’) restores the minibuffer
     contents to the (partial) input that you last used for completion in the
     current minibuffer.  Commands that complete your input, such as ‘<TAB>’
     and ‘C-o’, record the partial input that you provide them for you to
     later retrieve it with ‘C-l’.  For example, if you type ‘M-x bar’ and
     start cycling with ‘C-o’, only to realize that you want a candidate that
     matches ‘baz’ and not ‘bar’, then you can type ‘C-l’ to restore the
     minibuffer input to ‘bar’, change it to ‘baz’ and complete again.


The overall theme is to provide more control over minibuffer completions
while you're using them.  This is inspired by Icicles by Drew Adams,
although Icicles covers a much wider ground, and my changes aim for
compatibility and integration with "vanilla" Emacs completions first.
I've tried to pick key bindings that are convenient and don't currently
do something particularly useful in the minibuffer, but I'm open for
suggestions on this regard.  How do people feel about adding something
along these lines to Emacs?

In general, I'm happy to upstream anything that seems useful/desirable
in my working branch, even if I haven't mentioned it here specifically.


Best,

Eshel

Re: Possible minibuffer completion enhancements

[Eli Zaretskii]

> From: Eshel Yaron <me@eshelyaron.com>
> Date: Sun, 14 Jan 2024 18:28:27 +0100
> 
> I have a few enhancements in my working branch that revolve around
> inspecting and modifying minibuffer completion settings interactively.
> These additions (described below) are backward compatible, documented,
> and, at times, even useful.  So I propose adding them to upstream.
> 
> You can try out and examine these changes by cloning from
> https://git.sr.ht/~eshel/emacs and building Emacs as usual.
> 
> [ git.sr.ht has been having some connectivity issues in the past few
>   days, to put it mildly, so in case it doesn't work, you can also find
>   the code at git://git.eshelyaron.com/emacs.git ]
> 
> If any of this looks like it might be appropriate for upstream Emacs,
> I'll format and submit the relevant parts as individual patches.

Thanks.  Maybe we should begin by installing this as a feature branch
and ask people to try it and report feedback.

Re: Possible minibuffer completion enhancements

[Stefan Kangas]

Eshel Yaron <me@eshelyaron.com> writes:

> In general, I'm happy to upstream anything that seems useful/desirable
> in my working branch, even if I haven't mentioned it here specifically.

Thanks.  Could I ask that you send patches to us for the changes you
think might be generally useful?

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Hi,

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eshel Yaron <me@eshelyaron.com>
>> Date: Sun, 14 Jan 2024 18:28:27 +0100
>>
>> I have a few enhancements in my working branch that revolve around
>> inspecting and modifying minibuffer completion settings interactively.
>> These additions (described below) are backward compatible, documented,
>> and, at times, even useful.  So I propose adding them to upstream.
>>
>> You can try out and examine these changes by cloning from
>> https://git.sr.ht/~eshel/emacs and building Emacs as usual.
>>
>> [ git.sr.ht has been having some connectivity issues in the past few
>>   days, to put it mildly, so in case it doesn't work, you can also find
>>   the code at git://git.eshelyaron.com/emacs.git ]
>>
>> If any of this looks like it might be appropriate for upstream Emacs,
>> I'll format and submit the relevant parts as individual patches.
>
> Thanks.  Maybe we should begin by installing this as a feature branch
> and ask people to try it and report feedback.

If that'd make it easier for people, sure.  I don't have access to push
to emacs.git though, I can open a request for access on Savannah, or we
could set up some mirror.  WDYT?

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Hi,

Stefan Kangas <stefankangas@gmail.com> writes:

> Eshel Yaron <me@eshelyaron.com> writes:
>
>> In general, I'm happy to upstream anything that seems useful/desirable
>> in my working branch, even if I haven't mentioned it here specifically.
>
> Thanks.  Could I ask that you send patches to us for the changes you
> think might be generally useful?

Sure, I've just sent one in Bug#68508.  Do you feel that any of the
minibuffer commands I suggested should be sent for review as well?


Thanks,

Eshel

Re: Possible minibuffer completion enhancements

[Eli Zaretskii]

> From: Eshel Yaron <me@eshelyaron.com>
> Cc: emacs-devel@gnu.org
> Date: Tue, 16 Jan 2024 14:47:09 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> If any of this looks like it might be appropriate for upstream Emacs,
> >> I'll format and submit the relevant parts as individual patches.
> >
> > Thanks.  Maybe we should begin by installing this as a feature branch
> > and ask people to try it and report feedback.
> 
> If that'd make it easier for people, sure.  I don't have access to push
> to emacs.git though, I can open a request for access on Savannah, or we
> could set up some mirror.  WDYT?

Are you still making frequent changes to the code?  Or is it
relatively stable?

Re: Possible minibuffer completion enhancements

[Stefan Kangas]

Eshel Yaron <me@eshelyaron.com> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
>> Eshel Yaron <me@eshelyaron.com> writes:
>>
>>> In general, I'm happy to upstream anything that seems useful/desirable
>>> in my working branch, even if I haven't mentioned it here specifically.
>>
>> Thanks.  Could I ask that you send patches to us for the changes you
>> think might be generally useful?
>
> Sure, I've just sent one in Bug#68508.

Thank you.  I also saw some doc fixes that might be worth considering,
and maybe there's more?

It takes extra work to pick out changes in an external branch where your
changes are mixed in with the ones from the official tree.  So the more
you can send, the more we are likely to integrate, I think.

> Do you feel that any of the minibuffer commands I suggested should be
> sent for review as well?

I don't use the default completion, but based on their descriptions I
don't see why we shouldn't want to consider all of them.

If the changes are orthogonal, it might make sense with one bug report
per patch, otherwise it might make more sense with a feature branch.

It really is whichever makes it easier to test and review the changes,
and probably you are in a the best position to say what you think about
that.  Then we'll take it from there.

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Hi,

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eshel Yaron <me@eshelyaron.com>
>>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> >> If any of this looks like it might be appropriate for upstream Emacs,
>> >> I'll format and submit the relevant parts as individual patches.
>> >
>> > Thanks.  Maybe we should begin by installing this as a feature branch
>> > and ask people to try it and report feedback.
>>
>> If that'd make it easier for people, sure.  I don't have access to push
>> to emacs.git though, I can open a request for access on Savannah, or we
>> could set up some mirror.  WDYT?
>
> Are you still making frequent changes to the code?  Or is it
> relatively stable?

I think it's likely there'll be a few more changes to make.  The
commands I listed here work pretty well, but there are a many completion
tables/styles/settings that I haven't tested yet, and some might require
further tweaks.


Regards,

Eshel

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Stefan Kangas <stefankangas@gmail.com> writes:

> Eshel Yaron <me@eshelyaron.com> writes:
>
>> Stefan Kangas <stefankangas@gmail.com> writes:
>>
>>> Eshel Yaron <me@eshelyaron.com> writes:
>>>
>>>> In general, I'm happy to upstream anything that seems useful/desirable
>>>> in my working branch, even if I haven't mentioned it here specifically.
>>>
>>> Thanks.  Could I ask that you send patches to us for the changes you
>>> think might be generally useful?
>>
>> Sure, I've just sent one in Bug#68508.
>
> Thank you.  I also saw some doc fixes that might be worth considering,
> and maybe there's more?
>
> It takes extra work to pick out changes in an external branch where your
> changes are mixed in with the ones from the official tree.  So the more
> you can send, the more we are likely to integrate, I think.

Yeah, that makes sense.  There's a small bugfix that I'll try to send
later today, and I'll check the doc fixes to see if they're relevant.

>> Do you feel that any of the minibuffer commands I suggested should be
>> sent for review as well?
>
> I don't use the default completion, but based on their descriptions I
> don't see why we shouldn't want to consider all of them.
>
> If the changes are orthogonal, it might make sense with one bug report
> per patch, otherwise it might make more sense with a feature branch.
>
> It really is whichever makes it easier to test and review the changes,
> and probably you are in a the best position to say what you think about
> that.  Then we'll take it from there.

I think a feature branch could be nice.  The changes are conceptually
orthogonal in the sense that each enhancement can be useful by itself,
but the code changes are not separated as some of them touch the same
areas of minibuffer.el.

I can pick the relevant changes and create a tidy feature branch with
just these new minibuffer commands added on top of master, if that helps.


Eshel

Re: Possible minibuffer completion enhancements

[Stefan Kangas]

Eshel Yaron <me@eshelyaron.com> writes:

> Yeah, that makes sense.  There's a small bugfix that I'll try to send
> later today, and I'll check the doc fixes to see if they're relevant.

Sounds good.  Thanks in advance for your contributions.

> I think a feature branch could be nice.  The changes are conceptually
> orthogonal in the sense that each enhancement can be useful by itself,
> but the code changes are not separated as some of them touch the same
> areas of minibuffer.el.
>
> I can pick the relevant changes and create a tidy feature branch with
> just these new minibuffer commands added on top of master, if that helps.

That sounds like a good way to experiment with this feature, yes.

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Stefan Kangas <stefankangas@gmail.com> writes:

> Eshel Yaron <me@eshelyaron.com> writes:
>
>> Yeah, that makes sense.  There's a small bugfix that I'll try to send
>> later today, and I'll check the doc fixes to see if they're relevant.
>
> Sounds good.  Thanks in advance for your contributions.
>
>> I think a feature branch could be nice.  The changes are conceptually
>> orthogonal in the sense that each enhancement can be useful by itself,
>> but the code changes are not separated as some of them touch the same
>> areas of minibuffer.el.
>>
>> I can pick the relevant changes and create a tidy feature branch with
>> just these new minibuffer commands added on top of master, if that helps.
>
> That sounds like a good way to experiment with this feature, yes.

All right, I've created a new feature branch called
feature/minibuffer-completion-enhancements that you can find at:

--8<---------------cut here---------------start------------->8---
git clone -b feature/minibuffer-completion-enhancements https://git.sr.ht/~eshel/emacs
--8<---------------cut here---------------end--------------->8---

I've also requested access to push this branch to emacs.git on Savannah,
so people won't need to fetch from another Git remote to test it.  If
someone else wants to grab that branch and push it to emacs.git, that's
also fine by me.


Best,

Eshel

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Eshel Yaron <me@eshelyaron.com> writes:

> All right, I've created a new feature branch called
> feature/minibuffer-completion-enhancements that you can find at:
>
> git clone -b feature/minibuffer-completion-enhancements https://git.sr.ht/~eshel/emacs
>
> I've also requested access to push this branch to emacs.git on Savannah,
> so people won't need to fetch from another Git remote to test it.

I've rebased the feature/minibuffer-completion-enhancements branch on
top of the current master, and pushed it to emacs.git.  Looking forward
for your thoughts and feedback!


Thanks,

Eshel

Re: Possible minibuffer completion enhancements

[Daniel Mendler via Emacs development discussions.]

Eshel Yaron <me@eshelyaron.com> writes:

> Eshel Yaron <me@eshelyaron.com> writes:
>
>> All right, I've created a new feature branch called
>> feature/minibuffer-completion-enhancements that you can find at:
>>
>> git clone -b feature/minibuffer-completion-enhancements https://git.sr.ht/~eshel/emacs
>>
>> I've also requested access to push this branch to emacs.git on Savannah,
>> so people won't need to fetch from another Git remote to test it.
>
> I've rebased the feature/minibuffer-completion-enhancements branch on
> top of the current master, and pushed it to emacs.git.  Looking forward
> for your thoughts and feedback!

Thanks. I took a look at your enhancements and I thought one could even
go a step further (for the narrowing and sorting) by adding a
generalized and unified completion candidate :property-function.

This function takes a candidate (or a list of candidates as argument)
and returns a list of properties, characterizing the candidate. For
files the property function could for example return mode, owner, group,
size and last modification date. Similarly, for buffers possible
properties would be the modification status, the size and the mode.

Then these properties could be used for:

- Sorting: Add a command to sort the candidates by property field. A
  comparator function is needed for this to work, maybe based on the
  property type and cl-defgeneric.

- Annotations: Each of the properties could be displayed in a column
  view as annotations. A mechanism (again possibly via cl-defgeneric) is
  needed to transform a property value to its string representation.

- Narrowing/Filtering: A completion style could take the properties and
  handle a query language with space-separated input, similar to the
  Orderless completion style, e.g., @mode=lisp. Alternatively a modified
  minibuffer-completion-predicate could be installed, similar to your
  narrow-completions-function.

Generally I find it a bit difficult to browse through your changes,
since they are slightly mixed. There are many ideas here and this may
make it harder to discuss the changes. Maybe you could reorder the
commits more logically or create separate branches for separate
additions.

A few of the changes go in the direction of technical changes unrelated
to the UI changes, which in my opinion would be good additions:

- completion-table-with-metadata: I've written (or seen) similar helpers
  many times in packages, so the addition of such helpers would reduce
  code duplication and simplify completion table construction. A while
  ago I had also proposed an even simpler addition of a
  completion-table-with-category helper which makes it possible to only
  override the completion category. This functionality is useful given
  that one often wants to specify the candidate category, in particular
  given the recent enhancements by Juri, which added category-based
  overrides via completion-category-overrides.
 
- The idea to rewrite the CRM functionality looks sound to me. I always
  wondered why CRM is not going through plain completing-read. Were
  there specific reasons for the current approach? Maybe Stefan Monnier
  has some insights on this?

Daniel

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Daniel Mendler <mail@daniel-mendler.de> writes:

> Eshel Yaron <me@eshelyaron.com> writes:
>>
>> I've rebased the feature/minibuffer-completion-enhancements branch on
>> top of the current master, and pushed it to emacs.git.  Looking forward
>> for your thoughts and feedback!
>
> Thanks. I took a look at your enhancements

Thank you!

> and I thought one could even go a step further (for the narrowing and
> sorting) by adding a generalized and unified completion candidate
> :property-function.
>
> This function takes a candidate (or a list of candidates as argument)
> and returns a list of properties, characterizing the candidate. For
> files the property function could for example return mode, owner,
> group, size and last modification date. Similarly, for buffers
> possible properties would be the modification status, the size and the
> mode.
>
> Then these properties could be used for:
>
> - Sorting: Add a command to sort the candidates by property field. A
>   comparator function is needed for this to work, maybe based on the
>   property type and cl-defgeneric.
>
> - Annotations: Each of the properties could be displayed in a column
>   view as annotations. A mechanism (again possibly via cl-defgeneric) is
>   needed to transform a property value to its string representation.
>
> - Narrowing/Filtering: A completion style could take the properties and
>   handle a query language with space-separated input, similar to the
>   Orderless completion style, e.g., @mode=lisp. Alternatively a modified
>   minibuffer-completion-predicate could be installed, similar to your
>   narrow-completions-function.

I had similar thoughts while working on some of these things, and I
agree that a standard way for the UI to obtain a set of relevant
properties/attributes of the completion candidates would be an
interesting next step.  My changes already allow you to sort and filter
by properties that are particular to some set of completions candidates
(for example you can sort file completions by last modified time with
`C-x C-v m` in the minibuffer), but a standard way to inspect candidate
attributes would let us provide many more such specialized sorting and
filtering options in many more cases.

> Generally I find it a bit difficult to browse through your changes,
> since they are slightly mixed. There are many ideas here and this may
> make it harder to discuss the changes. Maybe you could reorder the
> commits more logically or create separate branches for separate
> additions.

Are there any parts that you found particularly difficult to review?
Perhaps I could provide more insight.  I've made an effort to provide
some reasoning behind every change in the commit messages, and I also
tried to reorder some commits when I created this branch, but rewriting
history with many overlapping changes proved quite challenging, and I
didn't think it was cost-efficient.

> A few of the changes go in the direction of technical changes unrelated
> to the UI changes, which in my opinion would be good additions:
>
> - completion-table-with-metadata: I've written (or seen) similar helpers
>   many times in packages, so the addition of such helpers would reduce
>   code duplication and simplify completion table construction.

Yes, that's really handy.

>   A while ago I had also proposed an even simpler addition of a
>   completion-table-with-category helper which makes it possible to
>   only override the completion category. This functionality is useful
>   given that one often wants to specify the candidate category, in
>   particular given the recent enhancements by Juri, which added
>   category-based overrides via completion-category-overrides.

Indeed, with `completion-table-with-metadata` this is reduced to:

--8<---------------cut here---------------start------------->8---
(completion-table-with-metadata table '((category . completion-style)))
--8<---------------cut here---------------end--------------->8---

> - The idea to rewrite the CRM functionality looks sound to me. I always
>   wondered why CRM is not going through plain completing-read. Were
>   there specific reasons for the current approach? Maybe Stefan Monnier
>   has some insights on this?

I don't know, I'd appreciate any input from Stefan, of course.


Thanks again for reviewing,

Eshel

Re: Possible minibuffer completion enhancements

[Juri Linkov]

> I've rebased the feature/minibuffer-completion-enhancements branch on
> top of the current master, and pushed it to emacs.git.  Looking forward
> for your thoughts and feedback!

Thanks, I tried 'C-x C-v' (minibuffer-sort-completions),
and it works nicely.  But I can't imagine how the whole
branch could be merged to master when it contains
so diverse set of different features.

For example, probably everyone would agree that
'completion-table-with-metadata' could be installed
in master immediately (if you present a separate patch).
But other feature might require more discussion.

Re: Possible minibuffer completion enhancements

[Daniel Mendler via Emacs development discussions.]

Juri Linkov <juri@linkov.net> writes:

>> I've rebased the feature/minibuffer-completion-enhancements branch on
>> top of the current master, and pushed it to emacs.git.  Looking forward
>> for your thoughts and feedback!
>
> Thanks, I tried 'C-x C-v' (minibuffer-sort-completions),
> and it works nicely.  But I can't imagine how the whole
> branch could be merged to master when it contains
> so diverse set of different features.

I agree. I suggest to create a set of separate patches, e.g., as
follows:

1. Helper functions, e.g., completion-table-with-metadata
2. CRM refactoring
3. Cycle commands for sorting
4. Cycle commands for completion styles
5. Annotation functions
6. Narrowing commands

Regarding 5 and 6 it would be great to explore an alternative approach
with a :property-function because of the additional flexibility and
extensibility. We would get a lot of functionality for free.

Iirc Stefan Monnier had plans to change the implementation of the
completion machinery, rebasing it on a cl-defgeneric mechanism, where
completion tables provide cl-defmethods. If more and more additional
completion candidate metadata is supplied, looking into this again may
be worthwhile.

> For example, probably everyone would agree that
> 'completion-table-with-metadata' could be installed
> in master immediately (if you present a separate patch).
> But other feature might require more discussion.

Indeed. However the implementation of completion-table-with-metadata
I've seen in Eshel's branch did not seem correct. The function should be
simpler:

(defun completion-table-with-metadata (table metadata)
  (lambda (str pred action)
    (if (eq action 'metadata)
        `(metadata ,@metadata
                   ,@(and (functionp table)
                          (cdr (funcall table str pred action))))
      (complete-with-action action table str pred))))

Daniel

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Daniel Mendler <mail@daniel-mendler.de> writes:

> Juri Linkov <juri@linkov.net> writes:
>
>>> I've rebased the feature/minibuffer-completion-enhancements branch on
>>> top of the current master, and pushed it to emacs.git.  Looking forward
>>> for your thoughts and feedback!
>>
>> Thanks, I tried 'C-x C-v' (minibuffer-sort-completions),
>> and it works nicely.

Great, thanks Juri!

>> But I can't imagine how the whole branch could be merged to master
>> when it contains so diverse set of different features.
>
> I agree. I suggest to create a set of separate patches, e.g., as
> follows:
>
> 1. Helper functions, e.g., completion-table-with-metadata
> 2. CRM refactoring
> 3. Cycle commands for sorting
> 4. Cycle commands for completion styles
> 5. Annotation functions
> 6. Narrowing commands

I'm not sure I understand the use of the word "Cycle" in (3) and (4).
My changes provide new commands that pertain to cycling (`C-o`, `C-l`),
and other commands for sorting (`C-x C-v`) and setting completion styles
(`C-x /`) which are not necessarily related to cycling.  Did you mean to
include "Cycle commands" as a separate bullet, perhaps?

Regarding breaking up the changes into multiple patches, I'm definitely
not opposed, but as I mentioned in another reply, it requires some
effort to do that cleanly.  Would any of you be willing to lend a hand
with that endeavor?  I'd be happy to coordinate, could be a good
opportunity to discuss some of this stuff in more detail as well.

> Regarding 5 and 6 it would be great to explore an alternative approach
> with a :property-function because of the additional flexibility and
> extensibility. We would get a lot of functionality for free.

I think such an approach can leverage the infrastructure that my branch
implements, for example with a `narrow-completions-function` that uses
the properties you get from such a `property-function`, no?  Also, I
agree that the approach you brought up could provide a lot of
functionality, but how would it be more flexible than the proposed
`narrow-completions-function` mechanism, where the completion table can
provide an arbitrary function?

> Iirc Stefan Monnier had plans to change the implementation of the
> completion machinery, rebasing it on a cl-defgeneric mechanism, where
> completion tables provide cl-defmethods. If more and more additional
> completion candidate metadata is supplied, looking into this again may
> be worthwhile.
>
>> For example, probably everyone would agree that
>> 'completion-table-with-metadata' could be installed
>> in master immediately (if you present a separate patch).
>> But other feature might require more discussion.
>
> Indeed. However the implementation of completion-table-with-metadata
> I've seen in Eshel's branch did not seem correct. The function should be
> simpler:

Hmm, define "correct" and "should" :)
My implementation is indeed more sophisticated, since it does more.  In
particular, it lets you override some metadata while preserving other
metadata that come from the original table.  Could you elaborate about
how this simplified version is preferable in your opinion?

> (defun completion-table-with-metadata (table metadata)
>   (lambda (str pred action)
>     (if (eq action 'metadata)
>         `(metadata ,@metadata
>                    ,@(and (functionp table)
>                           (cdr (funcall table str pred action))))
>       (complete-with-action action table str pred))))


Cheers,

Eshel

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Eshel Yaron <me@eshelyaron.com> writes:

> Daniel Mendler <mail@daniel-mendler.de> writes:
>
>> ...the implementation of completion-table-with-metadata I've seen in
>> Eshel's branch did not seem correct. The function should be simpler:
>
> Hmm, define "correct" and "should" :)
> My implementation is indeed more sophisticated, since it does more.  In
> particular, it lets you override some metadata while preserving other
> metadata that come from the original table.  Could you elaborate about
> how this simplified version is preferable in your opinion?
>
>> (defun completion-table-with-metadata (table metadata)
>>   (lambda (str pred action)
>>     (if (eq action 'metadata)
>>         `(metadata ,@metadata
>>                    ,@(and (functionp table)
>>                           (cdr (funcall table str pred action))))
>>       (complete-with-action action table str pred))))

On second thought, your version should indeed work just as well as mine,
expect that my implementation also allows the extra metadata to be
computed just in time.  In the common case where the metadata is
computed ahead of time, both versions basically behave the same, right?

Re: Possible minibuffer completion enhancements

[Daniel Mendler via Emacs development discussions.]

Eshel Yaron <me@eshelyaron.com> writes:

> Daniel Mendler <mail@daniel-mendler.de> writes:
>
>> Juri Linkov <juri@linkov.net> writes:
>> I agree. I suggest to create a set of separate patches, e.g., as
>> follows:
>>
>> 1. Helper functions, e.g., completion-table-with-metadata
>> 2. CRM refactoring
>> 3. Cycle commands for sorting
>> 4. Cycle commands for completion styles
>> 5. Annotation functions
>> 6. Narrowing commands
>
> I'm not sure I understand the use of the word "Cycle" in (3) and (4).
> My changes provide new commands that pertain to cycling (`C-o`, `C-l`),
> and other commands for sorting (`C-x C-v`) and setting completion styles
> (`C-x /`) which are not necessarily related to cycling.  Did you mean to
> include "Cycle commands" as a separate bullet, perhaps?

I assumed that you also provide commands to cycle between sort functions
and completion styles, instead of selection commands only. Iirc Icicles
provides such commands to cycle between various modes. Replace "Cycle
commands" with "Select and/or cycle commands" above.

>> Regarding 5 and 6 it would be great to explore an alternative approach
>> with a :property-function because of the additional flexibility and
>> extensibility. We would get a lot of functionality for free.
>
> I think such an approach can leverage the infrastructure that my branch
> implements, for example with a `narrow-completions-function` that uses
> the properties you get from such a `property-function`, no?  Also, I
> agree that the approach you brought up could provide a lot of
> functionality, but how would it be more flexible than the proposed
> `narrow-completions-function` mechanism, where the completion table can
> provide an arbitrary function?

You are right that a narrow-completions-function could offer narrowing
by property, and itself access the properties via a property-function.
However a property-function is more generic since it will allow us to
build more functionality in to the frontend without further extending
the completion tables.

I try to explain the idea a bit better. Instead of providing an
annotation-function, a display-sort-function, a
narrow-completions-function, a completion table could only provide a
lower level property-function which provides metadata for each
candidate. The completion tables would not have to provide the other
functions anymore.

The UI can query the properties of each candidate and format
annotations, offer sorting and narrowing by properties. This leads to a
separation of data backend (completion table) and presentation by the
UI. The completion table itself is only responsible to provide the data
(candidates and its properties), while the UI is responsible for
presentation features, e.g., sorting or formatting and aligning the
properties as annotations, etc.

This means that a large fraction of the functionality will be part of
the UI and has to be implemented there only once. The completion tables
itself would be reduced to data backends, and we still get annotations,
sorting, narrowing, and possible more property-dependent functionality.

Daniel

Re: Possible minibuffer completion enhancements

[Philip Kaludercic]

[-- Attachment #1: Type: text/plain, Size: 934 bytes --]

Eshel Yaron <me@eshelyaron.com> writes:

> Eshel Yaron <me@eshelyaron.com> writes:
>
>> All right, I've created a new feature branch called
>> feature/minibuffer-completion-enhancements that you can find at:
>>
>> git clone -b feature/minibuffer-completion-enhancements https://git.sr.ht/~eshel/emacs
>>
>> I've also requested access to push this branch to emacs.git on Savannah,
>> so people won't need to fetch from another Git remote to test it.
>
> I've rebased the feature/minibuffer-completion-enhancements branch on
> top of the current master, and pushed it to emacs.git.  Looking forward
> for your thoughts and feedback!

I like the improvements, they are not invasive but useful.  C-o is
particularly nice.

One annoyance I had with `completions-auto-update-mode' was that it
didn't update the buffer on deleting a character.  This should fix it,
but perhaps it can be optimised further by using `after-change-functions'?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: pch.diff --]
[-- Type: text/x-diff, Size: 731 bytes --]

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 0f503523817..6b949d67ec1 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -5804,12 +5804,12 @@ completions-auto-update-start-timer
 
 (defun completions-auto-update-setup ()
   "Prepare for updating *Completions* as you type in the minibuffer."
-  (add-hook 'post-self-insert-hook
+  (add-hook 'post-command-hook
             #'completions-auto-update-start-timer nil t))
 
 (defun completions-auto-update-exit ()
   "Stop updating *Completions* as you type in the minibuffer."
-  (remove-hook 'post-self-insert-hook
+  (remove-hook 'post-command-hook
                #'completions-auto-update-start-timer t))
 
 (define-minor-mode completions-auto-update-mode

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Daniel Mendler <mail@daniel-mendler.de> writes:

> Eshel Yaron <me@eshelyaron.com> writes:
>
>> Daniel Mendler <mail@daniel-mendler.de> writes:
>>
>>> ...I suggest to create a set of separate patches, e.g., as
>>> follows:
>>>
>>> 1. Helper functions, e.g., completion-table-with-metadata
>>> 2. CRM refactoring
>>> 3. Cycle commands for sorting
>>> 4. Cycle commands for completion styles
>>> 5. Annotation functions
>>> 6. Narrowing commands
>>
>> I'm not sure I understand the use of the word "Cycle" in (3) and (4).
>> My changes provide new commands that pertain to cycling (`C-o`, `C-l`),
>> and other commands for sorting (`C-x C-v`) and setting completion styles
>> (`C-x /`) which are not necessarily related to cycling.  Did you mean to
>> include "Cycle commands" as a separate bullet, perhaps?
>
> I assumed that you also provide commands to cycle between sort functions
> and completion styles, instead of selection commands only. Iirc Icicles
> provides such commands to cycle between various modes. Replace "Cycle
> commands" with "Select and/or cycle commands" above.

Got it.

>>> Regarding 5 and 6 it would be great to explore an alternative approach
>>> with a :property-function because of the additional flexibility and
>>> extensibility. We would get a lot of functionality for free.
>>
>> I think such an approach can leverage the infrastructure that my branch
>> implements, for example with a `narrow-completions-function` that uses
>> the properties you get from such a `property-function`, no?  Also, I
>> agree that the approach you brought up could provide a lot of
>> functionality, but how would it be more flexible than the proposed
>> `narrow-completions-function` mechanism, where the completion table can
>> provide an arbitrary function?
>
> You are right that a narrow-completions-function could offer narrowing
> by property, and itself access the properties via a property-function.
> However a property-function is more generic since it will allow us to
> build more functionality in to the frontend without further extending
> the completion tables.
>
> I try to explain the idea a bit better. Instead of providing an
> annotation-function, a display-sort-function, a
> narrow-completions-function, a completion table could only provide a
> lower level property-function which provides metadata for each
> candidate. The completion tables would not have to provide the other
> functions anymore.
>
> The UI can query the properties of each candidate and format
> annotations, offer sorting and narrowing by properties. This leads to a
> separation of data backend (completion table) and presentation by the
> UI. The completion table itself is only responsible to provide the data
> (candidates and its properties), while the UI is responsible for
> presentation features, e.g., sorting or formatting and aligning the
> properties as annotations, etc.
>
> This means that a large fraction of the functionality will be part of
> the UI and has to be implemented there only once. The completion tables
> itself would be reduced to data backends, and we still get annotations,
> sorting, narrowing, and possible more property-dependent functionality.

Thank you for the additional explanation!  I find this approach
appealing, I just think it is not mutually exclusive with my changes:
even if completion tables would provide such a candidate-property
inspection facility, they'd still need to provide also the existing
completion metadata in favor of UIs that currently leverage that, right?
Also the new commands from my branch extend the vanilla completion UI,
and those would remain useful, hopefully, even if we later change how
they obtain the data they need from the backend (completion table).

Anyway, if you or anybody else comes up with a (rough) sketch of this
alternative, I'd be glad to examine it and make a concrete comparison.


Best,

Eshel

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Hi Philip,

Philip Kaludercic <philipk@posteo.net> writes:

> Eshel Yaron <me@eshelyaron.com> writes:
>
>> I've rebased the feature/minibuffer-completion-enhancements branch on
>> top of the current master, and pushed it to emacs.git.  Looking forward
>> for your thoughts and feedback!
>
> I like the improvements, they are not invasive but useful.  C-o is
> particularly nice.

Thanks!  I find that one quite handy, too.

> One annoyance I had with `completions-auto-update-mode' was that it
> didn't update the buffer on deleting a character.

Yes, the `completions-auto-update-mode` I've included in this branch can
definitely be improved with that regards.  TBH this is not really the
focal point of these changes as far as I'm concerned, but I included it
since I think it complements the new commands nicely.

I've implemented this `completions-auto-update-mode` after Spencer
brought up the idea a few months ago, and I've been using it since.
Juri also had some interesting thoughts on the subject, IIRC.  See the
discussion here https://yhetil.org/emacs/87bkd3z9bi.fsf@catern.com/

> This should fix it, but perhaps it can be optimised further by using
> `after-change-functions'?

Unfortunately, neither `post-command-hook` (as you propose in the patch)
nor `after-change-functions` do the right thing in this case, because
there are commands and changes to the minibuffer contents that should
_not_ update *Completions*.  For example, when you move between
completion candidates with M-<down> in the minibuffer.  In fact,
`post-self-insert-hook` covers most of the cases in which we do want to
update *Completions* IME, except that it doesn't cover plain deletion,
as you noticed.

An alternative to using a different hook would be to arrange for your
deletion command to update *Completions* when appropriate.  Juri and
Spencer probably have more useful insights on that matter, though.


Best,

Eshel

Re: Possible minibuffer completion enhancements

[Daniel Mendler via Emacs development discussions.]

Eshel Yaron <me@eshelyaron.com> writes:

> Daniel Mendler <mail@daniel-mendler.de> writes:
>>>> Regarding 5 and 6 it would be great to explore an alternative approach
>>>> with a :property-function because of the additional flexibility and
>>>> extensibility. We would get a lot of functionality for free.
>>>
>>> I think such an approach can leverage the infrastructure that my branch
>>> implements, for example with a `narrow-completions-function` that uses
>>> the properties you get from such a `property-function`, no?  Also, I
>>> agree that the approach you brought up could provide a lot of
>>> functionality, but how would it be more flexible than the proposed
>>> `narrow-completions-function` mechanism, where the completion table can
>>> provide an arbitrary function?
>>
>> You are right that a narrow-completions-function could offer narrowing
>> by property, and itself access the properties via a property-function.
>> However a property-function is more generic since it will allow us to
>> build more functionality in to the frontend without further extending
>> the completion tables.
>>
>> I try to explain the idea a bit better. Instead of providing an
>> annotation-function, a display-sort-function, a
>> narrow-completions-function, a completion table could only provide a
>> lower level property-function which provides metadata for each
>> candidate. The completion tables would not have to provide the other
>> functions anymore.
>>
>> The UI can query the properties of each candidate and format
>> annotations, offer sorting and narrowing by properties. This leads to a
>> separation of data backend (completion table) and presentation by the
>> UI. The completion table itself is only responsible to provide the data
>> (candidates and its properties), while the UI is responsible for
>> presentation features, e.g., sorting or formatting and aligning the
>> properties as annotations, etc.
>>
>> This means that a large fraction of the functionality will be part of
>> the UI and has to be implemented there only once. The completion tables
>> itself would be reduced to data backends, and we still get annotations,
>> sorting, narrowing, and possible more property-dependent functionality.
>
> Thank you for the additional explanation!  I find this approach
> appealing, I just think it is not mutually exclusive with my changes:
> even if completion tables would provide such a candidate-property
> inspection facility, they'd still need to provide also the existing
> completion metadata in favor of UIs that currently leverage that, right?
> Also the new commands from my branch extend the vanilla completion UI,
> and those would remain useful, hopefully, even if we later change how
> they obtain the data they need from the backend (completion table).

Absolutely. The changes are not mutually exclusive and the value of the
UI enhancements can be evaluated independently.

Nevertheless I think on the side of the completion table backends we
should explore an approach where the completion tables provides data
only and the UI reuses this data to implement various functions. A few
more points to consider:

1. Changes to completion tables are pervasive, since many completion
   tables will be affected and have to be adapted if we extend them with
   new functionality. We can potentially save complexity if we chose the
   completion table extensions well.
2. If we add metadata extensions with overlapping purpose, e.g., a
   narrowing functions and after that some property function we end up
   with redundant code.
3. If the added functionality to completion tables is generic and not
   too tightly coupled with the UI, new possibilities open up. I had
   mentioned the Orderless completion style which could rely on a
   property-function to implement narrowing by property (not sure if
   this would also work with your narrow function). Also having the
   ability to create sorting functions and annotations from a single
   backend property function seems appealing, if it works.

In your narrow-completions-function proposal, the narrowing
functionality is a UI feature. The narrowing-completions-function even
calls completing-read itself and I think we could do better if we make
sure that the completion tables do not get extended with UI
functionality, but stay lower level, only providing data. In this
respect your narrowing-completions-function differs from existing
metadata extensions, which are lower level and mostly pure
non-interactive functions (for example the annotation-function or
display-sort-function).

> Anyway, if you or anybody else comes up with a (rough) sketch of this
> alternative, I'd be glad to examine it and make a concrete comparison.

Sure. However I am not interested in creating competing implementations
which we then compare and ultimately scrap. I'd thought we could maybe
consider a property function as an alternative (lower level, flexible)
completion table extension and if possible, collaborate on such a design
and rebase your UI enhancements on top of that, if you are open to that.
I am not asking you to do all this work or throw yours away. I am rather
asking if you would be interested in helping to explore such an
alternative (for the completion table backends only), or if you think
that a property-function is a flawed idea to begin with.

Daniel

Re: Possible minibuffer completion enhancements

[Daniel Mendler via Emacs development discussions.]

Eshel Yaron <me@eshelyaron.com> writes:

> Eshel Yaron <me@eshelyaron.com> writes:
>
>> Daniel Mendler <mail@daniel-mendler.de> writes:
>>
>>> ...the implementation of completion-table-with-metadata I've seen in
>>> Eshel's branch did not seem correct. The function should be simpler:
>>
>> Hmm, define "correct" and "should" :)
>> My implementation is indeed more sophisticated, since it does more.  In
>> particular, it lets you override some metadata while preserving other
>> metadata that come from the original table.  Could you elaborate about
>> how this simplified version is preferable in your opinion?
>>
>>> (defun completion-table-with-metadata (table metadata)
>>>   (lambda (str pred action)
>>>     (if (eq action 'metadata)
>>>         `(metadata ,@metadata
>>>                    ,@(and (functionp table)
>>>                           (cdr (funcall table str pred action))))
>>>       (complete-with-action action table str pred))))
>
> On second thought, your version should indeed work just as well as mine,
> expect that my implementation also allows the extra metadata to be
> computed just in time.  In the common case where the metadata is
> computed ahead of time, both versions basically behave the same, right?

Indeed. Your `completion-table-with-metadata' function allows to compute
the metadata on the fly. Are there use cases for this? From my
experience the completion table metadata is mostly static, except in
cases where completion table boundaries are used. But in that case a
simple helper like `completion-table-with-metadata' seems to too limited
anyway?

The above function can be adapted as follows if a dynamic
recomputation feature is desirable:

(defun completion-table-with-metadata (table metadata)
  (lambda (str pred action)
    (if (eq action 'metadata)
        `(metadata ,@(if (functionp metadata)
                         (funcall metadata str pred)
                       metadata)
                   ,@(and (functionp table)
                          (cdr (funcall table str pred action))))
      (complete-with-action action table str pred))))

Daniel

Re: Possible minibuffer completion enhancements

[Eshel Yaron]

Daniel Mendler <mail@daniel-mendler.de> writes:

> Eshel Yaron <me@eshelyaron.com> writes:
>
>> Daniel Mendler <mail@daniel-mendler.de> writes:
>>>
>>> I try to explain the idea a bit better.

[...20 lines elided...]

>> Thank you for the additional explanation!  I find this approach
>> appealing, I just think it is not mutually exclusive with my changes:
>> even if completion tables would provide such a candidate-property
>> inspection facility, they'd still need to provide also the existing
>> completion metadata in favor of UIs that currently leverage that, right?
>> Also the new commands from my branch extend the vanilla completion UI,
>> and those would remain useful, hopefully, even if we later change how
>> they obtain the data they need from the backend (completion table).
>
> Absolutely. The changes are not mutually exclusive and the value of the
> UI enhancements can be evaluated independently.
>
> Nevertheless I think on the side of the completion table backends we
> should explore an approach where the completion tables provides data
> only and the UI reuses this data to implement various functions. A few
> more points to consider:
>
> 1. Changes to completion tables are pervasive, since many completion
>    tables will be affected and have to be adapted if we extend them with
>    new functionality. We can potentially save complexity if we chose the
>    completion table extensions well.
> 2. If we add metadata extensions with overlapping purpose, e.g., a
>    narrowing functions and after that some property function we end up
>    with redundant code.
> 3. If the added functionality to completion tables is generic and not
>    too tightly coupled with the UI, new possibilities open up. I had
>    mentioned the Orderless completion style which could rely on a
>    property-function to implement narrowing by property (not sure if
>    this would also work with your narrow function). Also having the
>    ability to create sorting functions and annotations from a single
>    backend property function seems appealing, if it works.

Right.

> In your narrow-completions-function proposal, the narrowing
> functionality is a UI feature. The narrowing-completions-function even
> calls completing-read itself and I think we could do better if we make
> sure that the completion tables do not get extended with UI
> functionality, but stay lower level, only providing data. In this
> respect your narrowing-completions-function differs from existing
> metadata extensions, which are lower level and mostly pure
> non-interactive functions (for example the annotation-function or
> display-sort-function).

I agree it may be preferable to let the UI concoct completion predicates,
while my current `narrow-completions-function` proposal puts that weight
on the backend.  For the UI to suggest relevant predicates, it needs
access to relevant completion candidate properties/attributes.  So that
makes sense, thanks.

>> Anyway, if you or anybody else comes up with a (rough) sketch of this
>> alternative, I'd be glad to examine it and make a concrete comparison.
>
> Sure. However I am not interested in creating competing implementations
> which we then compare and ultimately scrap. I'd thought we could maybe
> consider a property function as an alternative (lower level, flexible)
> completion table extension and if possible, collaborate on such a design
> and rebase your UI enhancements on top of that, if you are open to that.

Got it, I think that's worth a try.

> I am not asking you to do all this work or throw yours away. I am rather
> asking if you would be interested in helping to explore such an
> alternative (for the completion table backends only), or if you think
> that a property-function is a flawed idea to begin with.

I think it's a good idea.  In the feature branch I've added a
`narrow-completions-function` to `help--symbol-completion-table`, that
lets you narrow completions to symbols that have some symbol property.
That might be an interesting test case for an alternative approach.  A
challenge, I think, is to cooperate well enough with the UI to not apply
a completion predicate but also clearly communicate that to the user.
With my current `narrow-completions-function`, each predicate that you
get comes with a description (string) that is then shown in the
*Completions* heading line in the default UI.  Users can also use that
description to refer to the predicate if they want to remove it later
with `C-x n w` when there are multiple predicates in place...  Do you
have any thoughts about how this could be achieved alternatively?


Best,

Eshel

Re: Possible minibuffer completion enhancements

[Daniel Mendler via Emacs development discussions.]

Eshel Yaron <me@eshelyaron.com> writes:

> Daniel Mendler <mail@daniel-mendler.de> writes:
>
>> Eshel Yaron <me@eshelyaron.com> writes:
>>
>>> Daniel Mendler <mail@daniel-mendler.de> writes:
>>>>
>>>> I try to explain the idea a bit better.

[...]

> I think it's a good idea.  In the feature branch I've added a
> `narrow-completions-function` to `help--symbol-completion-table`, that
> lets you narrow completions to symbols that have some symbol property.
> That might be an interesting test case for an alternative approach.  A
> challenge, I think, is to cooperate well enough with the UI to not apply
> a completion predicate but also clearly communicate that to the user.
> With my current `narrow-completions-function`, each predicate that you
> get comes with a description (string) that is then shown in the
> *Completions* heading line in the default UI.  Users can also use that
> description to refer to the predicate if they want to remove it later
> with `C-x n w` when there are multiple predicates in place...  Do you
> have any thoughts about how this could be achieved alternatively?

Translating from the low level candidate properties to the UI
presentation is indeed a difficulty. In the case of the
narrow-completions-function the backend and frontend are stronger
coupled, as you say, the function already provides presentation
information.

In the simplest scenario, the UI could present a list of of active
property restrictions (candidate property name and the expected value or
range), to which the user can add and from which the user can delete. If
one wants to go with a more sophisticated approach we could even
introduce a query language. In order to make progress, I'll have to
implement a small prototype as a demonstrator.

The idea to attach more flexible properties has come up a few times
before. When the affixation-function was introduced we had discussed a
decoration-function which specifies multiple fields which are
formatted/aligned by the UI. In contrast, the
annotation-/affixation-function themselves are responsible of
formatting, which puts in my opinion too much burden on the backends. In
an unpublished snippet, I experimented with the addition of properties
for narrowing/restricting the set of candidates. For each candidate
category I defined functions generating a list of properties for
candidates which can then be used for regex-based filtering. In that
case the predicate was build up by parsing a part of the minibuffer
input.

However so far I've not tried to unify the concepts. I'd find it pretty
great if such a more far reaching, unified solution would work out -
presenting candidate properties as annotations, restricting/narrowing by
property, and for sorting by property. I hope this explains my
motivation. A difficulty is certainly to not end up with an
over-engineered solution. I would find it particularly important that
complexity is kept out of the backends.

Daniel

Re: Possible minibuffer completion enhancements

[Madhu]

* Daniel Mendler  <87a5ox7png.fsf@daniel-mendler.de> :
Wrote on Mon, 22 Jan 2024 09:12:19 +0100:
> Iirc Stefan Monnier had plans to change the implementation of the
> completion machinery, rebasing it on a cl-defgeneric mechanism, where
> completion tables provide cl-defmethods. If more and more additional
> completion candidate metadata is supplied, looking into this again may
> be worthwhile.

Does this mean I have to restart emacs every time one has to undefine a
method?  elisp has already disclaimed support any form of class
redefinition.

This is a disturbing trend that has been shepherded into emacs by Eli in
other areas too.

There was a comment in another thread about how COND* was a NIH reaction
to PCASE. I haven't seen stallman's response but I think problem is that
COND* was not invented first and is a shoving down on elisp of
constructs and patterns in languages that fundamentally are hostile to
lisp, and now elisp is being dragged down the path again.


Elisp implementations of cl-defclass and cl-defgeneric are deficient in
well understood ways.  the campaign seems to drag the language of elisp
away from the freedom of lisp culture and replace that the culture with
a culture which shapes programmers who have been conditioned for a
different type of market which exploits that lack of freedom (through
derivative investments)