bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Drew Adams]

It seems to me that the doc, including in the Elisp manual, doesn't make
clear that, by default, `completing-read' automatically adds the list of
all completions provided by the completion table to the list of
defaults, just after the default value.  That is, by default, it calls
`minibuffer-default-add-completions'.

This is not obvious from reading the docs.  For example, it's not
obvious that if you use `C-h v' and then `M-p', repeating `M-p, you get
the symbols that are variables, one by one, inserted into the
minibuffer.

How so?  Because by default variable `minibuffer-default-add-function'
is `minibuffer-default-add-completions'.  Again, none of this is
obvious.  To find it out, a user needs to check what `M-p' is bound to,
then check the source code for that function, and then the source code
or the doc string of function `goto-history-element', which it calls.

In sum, important user-visible behavior isn't described in the manual or
the "top-level", most-relevant doc strings (e.g. `completing-read').
This is a request to, somehow, make this default behavior more apparent
in the doc.  Thx.

In GNU Emacs 29.0.91 (build 2, x86_64-w64-mingw32) of 2023-05-14 built
 on AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.19045
System Description: Microsoft Windows 10 Pro (v10.0.2009.19045.3086)

Configured using:
 'configure --with-modules --without-dbus --with-native-compilation
 --without-compress-install --with-tree-sitter CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

(NATIVE_COMP present but libgccjit not available)

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> Date: Sat, 15 Jul 2023 23:35:20 +0000
> 
> It seems to me that the doc, including in the Elisp manual, doesn't make
> clear that, by default, `completing-read' automatically adds the list of
> all completions provided by the completion table to the list of
> defaults, just after the default value.  That is, by default, it calls
> `minibuffer-default-add-completions'.
> 
> This is not obvious from reading the docs.  For example, it's not
> obvious that if you use `C-h v' and then `M-p', repeating `M-p, you get
> the symbols that are variables, one by one, inserted into the
> minibuffer.
> 
> How so?  Because by default variable `minibuffer-default-add-function'
> is `minibuffer-default-add-completions'.  Again, none of this is
> obvious.  To find it out, a user needs to check what `M-p' is bound to,
> then check the source code for that function, and then the source code
> or the doc string of function `goto-history-element', which it calls.
> 
> In sum, important user-visible behavior isn't described in the manual or
> the "top-level", most-relevant doc strings (e.g. `completing-read').

You have described various aspects of the implementation, but no
"user-visible behavior" and no reason why it would be interesting, let
alone important, to have that in the manual.  Please consider changing
the POV of your description so that it will be clear what important
information is missing and why.  The main purpose of API descriptions
in the ELisp manual is to explain to Lisp programmers how to achieve
this or that behavior, and I cannot bridge the gap between that goal
and what you wrote.

For starters, this:

  It seems to me that the doc, including in the Elisp manual, doesn't make
  clear that, by default, `completing-read' automatically adds the list of
  all completions provided by the completion table to the list of
  defaults, just after the default value.  That is, by default, it calls
  `minibuffer-default-add-completions'.

is a huge turn-off, because it talks about what the code does.  After
reading this, I have no idea why I would need to know these details.
Why do I care that the list of all completions is added to the list of
defaults? why do I care that the code calls
minibuffer-default-add-completions?  Etc. etc.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Drew Adams]

> > In sum, important user-visible behavior isn't described in the manual or
> > the "top-level", most-relevant doc strings (e.g. `completing-read').
> 
> You have described various aspects of the implementation, but no
> "user-visible behavior" and no reason why it would be interesting, let
> alone important, to have that in the manual.  Please consider changing
> the POV of your description so that it will be clear what important
> information is missing and why.  The main purpose of API descriptions
> in the ELisp manual is to explain to Lisp programmers how to achieve
> this or that behavior, and I cannot bridge the gap between that goal
> and what you wrote.

There are two kinds of doc changes I'd like to see:

1. Elisp manual and Lisp description in doc strings
   (e.g. `completing-read', `*-history-element').
   The behavior isn't documented, AFAICT.  It would
   also make sense, I think, to say something about
   the "future history" in node (elisp) `Minibuffer
   History' - how it gets filled automatically with
   all of the initial completion candidates, unless
   you bind `minibuffer-default-add-function' to
   nil (or another non-function value).

2. Emacs manual and user/interactive description
   in doc strings.

I guess I'd like to at least see #1.  It isn't at
all obvious what's going on.

Functions such as `completing-read' give you a way
to control the default or defaults provided for
`M-n', but they DON'T give you a way to prevent
the automatic addition of ALL of the initial
completion candidates.  To me that doesn't make
much sense, but Lispers should at least be told
this up front.  Let's at least tell them they can
prevent this automatic addition by binding
`minibuffer-default-add-function' to nil _around_
a function such as `completing-read'.

#2 is presumably less important, as long as we don't
expose variable `minibuffer-default-add-function'
(or similar) as a user option.  E.g., currently
there's no way for a user to turn off this automatic
addition to the `M-n' defaults.

The user visible behavior is that `M-n' inserts a
default value specific to the command, OR, if var
`minibuffer-default-add-function' is non-nil, a
"default" value which can be ANY value from the
initial completion "domain".

E.g., for `C-h v' that means the name of ANY variable
(source is obarray filtered for vars).  And there's
no user or Lisp control over the order of those
additional defaults.  E.g., `C-h v' adds at most a
single "real" default for `M-n', and it adds ALL vars
(from obarray).

For the Emacs manual, node `Minibuffer History' is
a good place to mention this automatic behavior
that's governed by a variable.  That's where "future
history" is introduced for `M-n'.  (FWIW, I'm not a
fan of the name "future history", instead of what
it used to be called: just "defaults".)

There we mention where the defaults added to "future 
history" come from, but only for file-name input.
And in that case we go into detail, mentioning how
to turn off this automatic addition (the variable
for this is even a user option, unlike the general
case, with `minibuffer-default-add-function').

> it talks about what the code does.  After
> reading this, I have no idea why I would need to know these details.
> Why do I care that the list of all completions is added to the list of
> defaults? why do I care that the code calls
> minibuffer-default-add-completions?

If you want to understand the Lisp code then you
need this info.  I guess that's my main concern.

If, as a user, you want to get an idea what the
`M-n' values are with, e.g., `C-h v', after seeing
them present themselves in a seemingly random order,
then you'd maybe like to know something about what
they are and where they come from.  You might just
wonder if you can turn off their inclusion, since
they kinda represent noise (zillions of variables
in obarray, presented in no order understandable to
users).

Hope this clarifies the request/bug report.
___

Beyond this doc concern, I'll maybe also request
(separately, in an enhancement request) that
Lispers have some way to control at least the order
of these "defaults" that get automatically added
for `M-n'.  E.g., be able to sort the obarray or
list.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "64656@debbugs.gnu.org" <64656@debbugs.gnu.org>
> Date: Sun, 16 Jul 2023 14:34:28 +0000
> 
> Hope this clarifies the request/bug report.

It doesn't.  Sorry, I guess I'm too stupid to understand what you are
asking.

I asked specific questions, and I don't see answers to those
questions.

How about if you start from some specific behavior of completing-read
when called in a specific way, and then point out which aspects of
this behavior are not documented?

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Drew Adams]

> > Hope this clarifies the request/bug report.
> 
> It doesn't.  Sorry, I guess I'm too stupid
> to understand what you are asking.

Don't be silly or sarcastic, please.  If
you don't understand it then I wasn't clear
enough or perhaps you haven't yet tried hard
enough.  Let's try one more time.

> I asked specific questions, and I don't see
> answers to those questions.
> 
> How about if you start from some specific
> behavior of completing-read when called in a
> specific way, and then point out which aspects of
> this behavior are not documented?

These are the only questions I find in your replies:

 > Why do I care that the list of all completions
 > is added to the list of defaults?
 >
 > why do I care that the code calls
 > minibuffer-default-add-completions?
 >
 > Etc. etc.

I don't find those so specific.  And I
think my post provides answers to both
of them. But let me try again, keeping
those questions in mind.

A user cares what's available from `M-n'.
(I don't expect another "Why?" for that,
but it comes, I'll try to answer that too.)

Why does it matter that all initial (i.e.,
prior to any user input in the minibuffer)
completions are put into the `M-n' queue?

Because that set of candidates is often
huge.  And because its order isn't designed
for `M-n' or (especially) for the particular
act of input reading.  Often its order has
no special reason.

And the case given as an example, `C-h v',
illustrates that well: (1) zillions of vars, 
(2) in no user-expectable/understandable/
useful order - the order of obarray!

This makes `M-n` useless (beyond the first
variable, at most).  And confusing.  The
natural question is "What's this?  What's
it for?  What can I do with it?"  Answer:
It's for nothing (that I can see).  It's
useless.  It's an antifeature - at least
in the case of `C-h v'.  And in other cases
(but maybe not all).

Why does a user care about
`minibuffer-default-add-completions'?

She doesn't.  I didn't say she does.  (But
an _Elisp_ user _might_ care.)

What an Elisp user (not an end user of a
command) really needs to care about is var
`minibuffer-default-add-function', not the
particular function that's its default value.

Forget for a moment about what various
function values for that variable might do.
The most important thing about that var is
that if nil then the domain of completions
isn't added to the `M-n' queue at all.  IOW,
that _turns off_ the automatic filling of
the `M-n' queue.

An Elisp user needs to know that fact, if
s?he uses `completing-read' and s?he wants
to prevent the kind of confusing overkill
exhibited by `C-h v'.  (She then needs to
bind the var to nil around the call to
`completing-read').

And an Elisp user needs to know about that
variable if s?he has a small completion
domain (unlike that of `describe-variable'),
and so s?he wants automatic inclusion of
completions, and s?he wants to control the
order in which they're presented by `M-n'.
`M-n' and `M-p' are inherently all about
_order_ - order matters.  Different contexts
can call for different orders.
	
The best subset of the completion domain for
a given command's `M-n' use, and the best
`M-n' order for it, is up to the definer of
that command (more precisely, of each use of
`completing-read').

Elisp users thus need to know that to define
the subset and its order for `M-n' they can
bind var `minibuffer-default-add-function'
to a function that returns such a list.
This isn't obvious.  You won't find it by
reading the `completing-read' doc, at least,
though it's just as important to controlling
the behavior as the args to that function. 

I mentioned that I think it would help to
make some changes to both the Elisp doc and
the user doc.  Doc strings and manuals.  But
I also mentioned that I think it's the Lisp
doc that's most in need of some explanation
about this.

Because blind automatic inclusion of all
domain completions occurs, end users need,
unfortunately, to be made aware of the
behavior (at least how to inhibit it).

IMO it's a bug that something like `C-h v'
stuffs all variables into `M-n' - in a 
perceptibly random order, no less.  But
given that bug (is it by design?), a user
(yes, and end user) needs to be given a
heads-up as to what's happening.

In effect, the heads-up tells a user that
when prompted for input with completion,
in some cases the "future history" of
defaults is effectively useless.  And it
doesn't hurt to tell users why: _all_
possible domain completions are included,
possibly in a meaningless order.

To be clear: There's a world of difference
between (1) a command that provides a list
of likely useful default values in arg DEF
and (2) a command that provides a single
such DEF value.

And that difference is felt harder when
the completion domain is large, and even
harder when it's large and in a useless
order.

By tacking - systematically - all domain
completions onto the end of the carefully
defined DEFaults, we now, in effect,
pollute `M-n', reducing or eliminating what
value it had.  And in particular, we've now
worked against the feature of letting DEF
provide multiple values ("future history").

The request mitigates some of the damage
done by the blind and systematic addition
to `M-n' of all completions in the domain.
If we fix that design bug in some way then
the need for the doc requested here might
be less critical.

If you still don't understand the problem,
let me ask you to simply try this:

`C-h v'
`M-n'
`M-n'
`M-n'
`M-n'
`M-n'
`M-n'
`M-n'
...

What's happening?  What determines which
symbols get inserted in the minibuffer,
and in what order?  Answer: All symbols
in the obarray when `completing-read'
is called are filtered by a predicate
that tests for a defvar or a variable
that's buffer-local variable in the
buffer where you invoked `C-h v'.

There are thousands of variables in that
future history.  Good luck getting to
any of them that might be useful (after
the first one, the provided DEF) in any
given context.

And your minibuffer input can't help you.
Try this:

`C-h v org TAB'
`M-n'
`M-n'
`M-n'
`M-n'
`M-n'
`M-n'
`M-n'
...

Of course the zillions of variables
stuffed into the `M-n' queue don't
reflect your minibuffer input ("org").
They don't reflect the state of
completion at all.  So this attempt
to guess at what's happening, and how
you might control it a bit, is a red
herring.

That's OK.  But what on earth do the
`M-n' values correspond to?  What's
their relevance/significance/use?
Answer: None - irrelevant, useless.
April fool!

The behavior is not useful, rational,
expected, or understandable.  With a
little aid from the doc it could at
least become expected and, to some
extent, understandable.  Let's at
least provide that bit of help.

HTH.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

> Try this:
>
> `C-h v org TAB'
> `M-n'
> `M-n'
> `M-n'
> `M-n'
> `M-n'
> `M-n'
> `M-n'
> ...

Why candidates are inserted in a random order?
It would make sense to insert them in the same
order as they are sorted in the *Completions* buffer.
But then `M-n' will become an equivalent of `M-down'.
So maybe better to sort candidates by recency/frequency?
But then how `M-n' will be different from `M-p'?

I think ideally it should use the same order as used
by icomplete-mode.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Drew Adams]

> > Try this:
> > `C-h v org TAB'
> > `M-n'
> > `M-n'
> > ...
> 
> Why candidates are inserted in a random order?
> It would make sense to insert them in the same
> order as they are sorted in the *Completions* buffer.

That's one reasonable possibility.
It's not the only one.

The fact is that the candidates are in
a useless order, particularly when the
completion table is just obarray or an
unsorted, filtered subset of obarray.

Why are all candidates inserted into the 
`M-n' queue at all?  And why no ability
to filter them or sort them - during
completion (i.e., taking the current
completion state into account).

> But then `M-n' will become an equivalent of `M-down'.
> So maybe better to sort candidates by recency/frequency?
> But then how `M-n' will be different from `M-p'?
> 
> I think ideally it should use the same order as used
> by icomplete-mode.

Ideally it should (1) be off by default,
not on, (2) be totally open to program
control, based on current completion state
(e.g. current set of input matches) and
completion history and input history,
(3) filterable, truncatable, etc., and
(4) should have reasonable doc.

A global function-valued variable is maybe
not the best way to add such a feature to
`completing-read' (& compagnie).  Seems
like maybe this feature wasn't well thought
out - the `C-h v' behavior says a lot about
it, I think.

Fundamentally, I think this works _against_
the purpose and idea behind DEF (explicit,
coder-defined list of default values).  It
represents noise pollution, IMHO.

At a bare minimum, the default value of the
function-valued variable should be nil (or
a no-op function), i.e., this new feature
should be OFF by default.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

>> > Try this:
>> > `C-h v org TAB'
>> > `M-n'
>> > `M-n'
>> > ...
>>
>> Why candidates are inserted in a random order?
>> It would make sense to insert them in the same
>> order as they are sorted in the *Completions* buffer.
>
> That's one reasonable possibility.
> It's not the only one.

I tried to use 'completions-sort' to sort 'C-h v M-n' default values.
By default, it sorts alphabetically that makes sense for 'C-h v M-n'.
But this broke many other completions.

For example, currently 'C-x b M-n M-n ...'
provides the default values sorted by the order
of recently used buffers that is very useful.
It keeps the order of 'buffer-alist'
in 'internal-complete-buffer'.

Another example is 'C-x p p M-n M-n ...'
that currently uses the order of recently
accessed projects from 'project--list'.

This means that we can't change this default behavior.

So currently there are three different sorting orders
used by default:
1. 'TAB' uses the alphabetical order;
1. 'M-p' uses the historical order;
2. 'M-n' is unsorted and follows the order of the caller.

> The fact is that the candidates are in
> a useless order, particularly when the
> completion table is just obarray or an
> unsorted, filtered subset of obarray.

This means that the caller should take care about
sorting completions is a meaningful order.
But then a new metadata type similar to
'display-sort-function' should be added
such as 'minibuffer-default-sort-function'
that might be a hassle.

So maybe this could be improved with a simper fix?
This is why I added such condition below:
(eq minibuffer-completion-table 'help--symbol-completion-table)

Please try this modified function, it should work with 'C-h v M-n':

#+begin_src emacs-lisp
(defun minibuffer-default-add-completions ()
  "Return a list of all completions without the default value.
This function is used to add all elements of the completion table to
the end of the list of defaults just after the default value."
  (let ((def minibuffer-default)
	(all (all-completions ""
			      minibuffer-completion-table
			      minibuffer-completion-predicate)))
    (when (eq minibuffer-completion-table 'help--symbol-completion-table)
      (setq all (pcase completions-sort
        	  ('nil all)
        	  ('alphabetical (sort all #'string-lessp))
        	  (_ (funcall completions-sort all)))))
    (if (listp def)
	(append def all)
      (cons def (delete def all)))))
#+end_src

> Why are all candidates inserted into the
> `M-n' queue at all?  And why no ability
> to filter them or sort them - during
> completion (i.e., taking the current
> completion state into account).

Because 'M-n' is not completion.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Drew Adams]

> >> > Try this:
> >> > `C-h v org TAB'
> >> > `M-n'
> >> > `M-n'
> >> > ...
> >>
> >> Why candidates are inserted in a random order?
> >> It would make sense to insert them in the same
> >> order as they are sorted in the *Completions* buffer.
> >
> > That's one reasonable possibility.
> > It's not the only one.

> So currently there are three different sorting orders
> used by default:
> 1. 'TAB' uses the alphabetical order;
> 1. 'M-p' uses the historical order;
> 2. 'M-n' is unsorted and follows the order of the caller.

No, `M-n' is just like `M-p' for previous
inputs.  And for DEFAULTS it's in the order
of DEFAULTS - which has nothing to do with
the set of initial completion candidates or
their order.

The misfeature that introduced all of the
problems you're considering now was to
stuff all of the _completion_ candidates
(the completion domain) into the `M-n'
(and thus `M-p') queue.  That was a huge
mistake, IMHO.

> > The fact is that the candidates are in
> > a useless order, particularly when the
> > completion table is just obarray or an
> > unsorted, filtered subset of obarray.
> 
> This means that the caller should take care about
> sorting completions is a meaningful order.
> But then a new metadata type similar to
> 'display-sort-function' should be added
> such as 'minibuffer-default-sort-function'
> that might be a hassle.

No, just revert the misfeature of adding
the completion domain to the `M-n' queue.
End of story - no problem.

> (defun minibuffer-default-add-completions ()
>   "Return a list of all completions without the default value.
> This function is used to add all elements of the completion table to
> the end of the list of defaults just after the default value."

Just get rid of that function altogether.
Pull this misfeature out of Emacs; it
never should have been added (and IIRC
that was done without any discussion?).

> > Why are all candidates inserted into the
> > `M-n' queue at all?  And why no ability
> > to filter them or sort them - during
> > completion (i.e., taking the current
> > completion state into account).
> 
> Because 'M-n' is not completion.

Precisely!

Minibuffer input, and its history and its
defaults, is not related to the COLLECTION
used for completing.  Lax completion doesn't
even require that your minibuffer input be
any COLLECTION element.

Completion candidates have no business being
automatically injected into the `M-n'/`M-p'
queue.  That's for DEFAULTS and HISTORY.

Arg DEFAULTS has a raison d'etre; it's not
the same as arg COLLECTION.

You're going farther down that dead-end
rabbit hole now.  Back out and plug that
misdirected hole.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

> Pull this misfeature out of Emacs; it
> never should have been added (and IIRC
> that was done without any discussion?).

This feature was removed by bug#5214 and bug#5364,
then later added back by the request in bug#12490
where you participated and raised no objections.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Drew Adams]

> > Pull this misfeature out of Emacs; it
> > never should have been added (and IIRC
> > that was done without any discussion?).
> 
> This feature was removed by bug#5214 and bug#5364,
> then later added back by the request in bug#12490
> where you participated and raised no objections.

Seriously?

The only "participation" I had in bug #12490
was to say that "We should never bind `C-s'
in the minibuffer to anything.  We shouldn't
remove the ability to use Isearch there (using
the global `C-s' binding)."

I said nothing there about any question of
jamming the completion domain (initial
candidates) into the `M-n' queue.  I had no idea
that was going on.

That wasn't the subject of bug #12490, at all!

And looking now, I really don't see that it has
anything to do with that thread - or with bugs
#5214 and #5364.  So I don't see what you're on
about, here.

That someone might have abused bug report #12490
to do what you say is news to me (and it's not
kosher IMO).  I no doubt did not follow that
particular bug thread with a fine-tooth comb.
If that was done then it snuck by me, I guess.

But I certainly have voiced my opposition to
stuffing the `M-n' queue with the completion
domain.  Multiple times, and likely in multiple
threads.

I wasn't aware that this purposeful regression
was reverted in bug threads #5214 and #5364.
I didn't participate in either of those threads,
and was no doubt unaware that they had anything
to do with this problem.

And I don't even see that what you say happened.

In bug #12490 I see you say this, at the outset,
in reply to OP's complaint that he could no
longer _search in the minibuffer_:

 > I want to report, that most used function stop working in Emacs 24
 > (tested 2 versions 24.1.1 and latest 24.2.1). I use Search in
 > Mini buffer a lot (don't know if people use it or not). If you call
 > search C-s when inside minibuffer like C-h f C-s you can search the
 > content of Minibuffer like search for functions or for buffer name to
 > switch like C-x b C-s (better then ido mode). And in Emacs 24 M-x C-s
 > stop working so I you can't search for interactive function to execute
 > anymore. It was working in 23.3 provided by Ubuntu (package
 > 23.3+1-1ubuntu4).

 Sorry, this feature has been removed by http://debbugs.gnu.org/5214
 and http://debbugs.gnu.org/5364

 But it's easy to restore it with a simple patch that prepends the
 current default value (a command at point) to the sorted list of
 all available command names:

And even that doesn't seem to have much, if
anything, to do with adding all of the initial
completions to the `M-n' queue.

So I really don't follow you, here.

To be very clear, I'm opposed to the misfeature
of automatically jamming the initial completions
onto the `M-n' queue.  We have arg DEFAULTS for
that.  Callers of `completing-read' etc. can
provide exactly the list of DEFAULTS they want
to prepend to the `M-n' queue.

Don't remove programmer (and user) control by
smothering `M-n' with the completion candidates.
Please pull this misfeature out.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

>  But it's easy to restore it with a simple patch that prepends the
>  current default value (a command at point) to the sorted list of
>  all available command names:
>
> And even that doesn't seem to have much, if
> anything, to do with adding all of the initial
> completions to the `M-n' queue.
> So I really don't follow you, here.

All available command names mentioned above
are extracted from initial completions.

> To be very clear, I'm opposed to the misfeature
> of automatically jamming the initial completions
> onto the `M-n' queue.  We have arg DEFAULTS for
> that.  Callers of `completing-read' etc. can
> provide exactly the list of DEFAULTS they want
> to prepend to the `M-n' queue.

Indeed, ideally callers of `completing-read' should
provide the exact list of defaults.  The problem
is that it's too late to identify the existing callers
and to add an explicit list of defaults to them.

> Don't remove programmer (and user) control by
> smothering `M-n' with the completion candidates.

This doesn't remove programmer (and user) control
because it's still easy to add own default values
to `M-n' and to remove initial completions from `M-n'.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Drew Adams]

> >  But it's easy to restore it with a simple patch that prepends the
> >  current default value (a command at point) to the sorted list of
> >  all available command names:
> >
> > And even that doesn't seem to have much, if
> > anything, to do with adding all of the initial
> > completions to the `M-n' queue.
> > So I really don't follow you, here.
> 
> All available command names mentioned above
> are extracted from initial completions.

And?  That's the problem.  The completion domain
("initial completions") shouldn't be added to
the `M-n' queue.  At least not by default, and
IMHO, never automatically.  Any programmer is
free to add anything at all to the `M-n' queue,
including all of the initial completions, simply
by adding it to DEFAULTS.  That's what DEFAULTS
is for: to add to the `M-n' queue.

> > To be very clear, I'm opposed to the misfeature
> > of automatically jamming the initial completions
> > onto the `M-n' queue.  We have arg DEFAULTS for
> > that.  Callers of `completing-read' etc. can
> > provide exactly the list of DEFAULTS they want
> > to prepend to the `M-n' queue.
> 
> Indeed, ideally callers of `completing-read' should
> provide the exact list of defaults.

Why only "ideally"?  Anyone is always free to
add whatever they want `M-n' using DEFAULTS in
their call to `completing-read'.

> The problem
> is that it's too late to identify the existing callers
> and to add an explicit list of defaults to them.

_Programmers_ can define DEFAULTS as they like.
No one needs to, or should, try to add anything
automatically to the `M-n' queue, overriding
what a programmer has explicitly decided should
be there using DEFAULTS.

Or perhaps you mean existing `completing-read'
calls in the vanilla Emacs code, not user code?

If so, I'd say don't worry about it.  Don't
second-guess what the `M-n' queue should be for
existing `completing-read' calls.  Or if you
really want to, go ahead, investigate them one
by one.

Each call to `completing-read' deserves its own 
consideration wrt DEFAULTS (the `M-n' queue).
Nothing should ever automatically trounce what
a programmer has explicitly decided should be
in the `M-n' queue (with DEFAULTS).

> > Don't remove programmer (and user) control by
> > smothering `M-n' with the completion candidates.
> 
> This doesn't remove programmer (and user) control
> because it's still easy to add own default values
> to `M-n' and to remove initial completions from `M-n'.

It's not so easy to remove initial completions.
At a minimum, how to do that needs to be added
to the doc (this bug report).  But as Eli said,
it's more important to fix the bug of their
automatic addition than to document a workaround.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Thierry Volpiatto]

Drew Adams <drew.adams@oracle.com> writes:

> And?  That's the problem.  The completion domain
> ("initial completions") shouldn't be added to
> the `M-n' queue.  At least not by default, and
> IMHO, never automatically.  Any programmer is
> free to add anything at all to the `M-n' queue,
> including all of the initial completions, simply
> by adding it to DEFAULTS.  That's what DEFAULTS
> is for: to add to the `M-n' queue.

+1

Please consider reverting this in next release.

Thanks.


-- 
Thierry

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

>> The problem is that it's too late to identify the
>> existing callers and to add an explicit list of defaults to them.
>
> Or perhaps you mean existing `completing-read'
> calls in the vanilla Emacs code, not user code?
>
> If so, I'd say don't worry about it.

We can't help but worry because we can't break
existing `completing-read' calls.

> Don't second-guess what the `M-n' queue should be for
> existing `completing-read' calls.  Or if you
> really want to, go ahead, investigate them one
> by one.

That's the problem: it's a too big task
to check all existing `completing-read' calls
to add explicit DEFAULTS.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

>>> > `C-h v org TAB'
>>> > `M-n'
>>> > `M-n'

I become convinced more and more that the problem lies somewhere else.
Trying to customize 'completions-sort' to the valid option 'nil'
("No sorting") reveals there are some completions where a meaningful
order is still not maintained.  For example, 'C-x v b l TAB'
displays branches in the order reverse from the output of
`git for-each-ref --format="%(refname)"`.  Here is the fix:

#+begin_src patch
diff --git a/lisp/vc/vc-git.el b/lisp/vc/vc-git.el
index 9ec45c59893..7a3eaba78c7 100644
--- a/lisp/vc/vc-git.el
+++ b/lisp/vc/vc-git.el
@@ -1723,7 +1729,7 @@ vc-git-revision-table
                       "^refs/\\(heads\\|tags\\|remotes\\)/\\(.*\\)$")))
         (while (re-search-forward regexp nil t)
           (push (match-string 2) table))))
-    table))
+    (nreverse table)))
 
 (defun vc-git-revision-completion-table (files)
   (letrec ((table (lazy-completion-table
#+end_src

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

>>>> > `C-h v org TAB'
>>>> > `M-n'
>>>> > `M-n'
>
> I become convinced more and more that the problem lies somewhere else.
> Trying to customize 'completions-sort' to the valid option 'nil'
> ("No sorting") reveals there are some completions where a meaningful
> order is still not maintained.  For example, 'C-x v b l TAB'
> displays branches in the order reverse from the output of
> `git for-each-ref --format="%(refname)"`.  Here is the fix:
>
> #+begin_src patch
> diff --git a/lisp/vc/vc-git.el b/lisp/vc/vc-git.el
> index 9ec45c59893..7a3eaba78c7 100644
> --- a/lisp/vc/vc-git.el
> +++ b/lisp/vc/vc-git.el
> @@ -1723,7 +1729,7 @@ vc-git-revision-table
>                        "^refs/\\(heads\\|tags\\|remotes\\)/\\(.*\\)$")))
>          (while (re-search-forward regexp nil t)
>            (push (match-string 2) table))))
> -    table))
> +    (nreverse table)))
>  
>  (defun vc-git-revision-completion-table (files)
>    (letrec ((table (lazy-completion-table
> #+end_src

Pushed.  Customizing 'completions-sort' to the option 'nil' ("No sorting")
reveals more interesting things.  For example, when vc completes a backend,
it turns out that the order is manually crafted:

  (defcustom vc-handled-backends '(RCS CVS SVN SCCS SRC Bzr Git Hg)
    ;; RCS, CVS, SVN, SCCS, and SRC come first because they are per-dir
    ;; rather than per-tree.  RCS comes first because of the multibackend
    ;; support intended to use RCS for local commits (with a remote CVS server).

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

> Customizing 'completions-sort' to the option 'nil' ("No sorting")
> reveals more interesting things.  For example, when vc completes a backend,
> it turns out that the order is manually crafted:
>
>   (defcustom vc-handled-backends '(RCS CVS SVN SCCS SRC Bzr Git Hg)
>     ;; RCS, CVS, SVN, SCCS, and SRC come first because they are per-dir
>     ;; rather than per-tree.  RCS comes first because of the multibackend
>     ;; support intended to use RCS for local commits (with a remote CVS server).

I guess not many users customize 'completions-sort' to nil.
Otherwise someone may notice that the order of unsorted files
is reversed from the output of `ls -U`.  This could be fixed by:

diff --git a/src/dired.c b/src/dired.c
index c10531cdb16..0f527a801f7 100644
--- a/src/dired.c
+++ b/src/dired.c
@@ -846,7 +846,7 @@ DEFUN ("file-name-all-completions", Ffile_name_all_completions,
   bestmatch = unbind_to (count, bestmatch);
 
   if (all_flag || NILP (bestmatch))
-    return bestmatch;
+    return Fnreverse (bestmatch);
   /* Return t if the supplied string is an exact match (counting case);
      it does not require any change to be made.  */
   if (matchcount == 1 && !NILP (Fequal (bestmatch, file)))

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Eli Zaretskii]

> Cc: 64656@debbugs.gnu.org
> From: Juri Linkov <juri@linkov.net>
> Date: Thu, 09 Nov 2023 18:34:52 +0200
> 
> I guess not many users customize 'completions-sort' to nil.
> Otherwise someone may notice that the order of unsorted files
> is reversed from the output of `ls -U`.  This could be fixed by:

The order is not reverse here, AFAICT, so please show a reproducible
recipe to make sure we are on the same page regarding this change.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

>> I guess not many users customize 'completions-sort' to nil.
>> Otherwise someone may notice that the order of unsorted files
>> is reversed from the output of `ls -U`.  This could be fixed by:
>
> The order is not reverse here, AFAICT, so please show a reproducible
> recipe to make sure we are on the same page regarding this change.

A recipe is to customize `completions-sort' to nil ("No sorting"),
then first call `M-& ls -U RET' and afterwards `C-x C-f TAB TAB'
and compare the contents of two buffers *Async Shell Command*
and *Completions*.  The order of files is reversed.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: drew.adams@oracle.com,  64656@debbugs.gnu.org
> Date: Thu, 09 Nov 2023 19:03:49 +0200
> 
> >> I guess not many users customize 'completions-sort' to nil.
> >> Otherwise someone may notice that the order of unsorted files
> >> is reversed from the output of `ls -U`.  This could be fixed by:
> >
> > The order is not reverse here, AFAICT, so please show a reproducible
> > recipe to make sure we are on the same page regarding this change.
> 
> A recipe is to customize `completions-sort' to nil ("No sorting"),
> then first call `M-& ls -U RET' and afterwards `C-x C-f TAB TAB'
> and compare the contents of two buffers *Async Shell Command*
> and *Completions*.  The order of files is reversed.

OK, I see it now, thanks.

But IMO this raises several issues:

  . completions-sort affects all completions, not just completions of
    file names, right?  So why the change only for file names?
  . who said that the order we get file names from readdir is the
    "unsorted order", and not its reverse?
  . in any case, I think we should reverse only when completions-sort
    is nil, because otherwise we could adversely affect the sorting
    performed on the results

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

>> A recipe is to customize `completions-sort' to nil ("No sorting"),
>> then first call `M-& ls -U RET' and afterwards `C-x C-f TAB TAB'
>> and compare the contents of two buffers *Async Shell Command*
>> and *Completions*.  The order of files is reversed.
>
> OK, I see it now, thanks.
>
> But IMO this raises several issues:
>
>   . completions-sort affects all completions, not just completions of
>     file names, right?  So why the change only for file names?

I'm trying various completions after customizing completions-sort to nil,
so currently noticed a problem in the completions of file names.

>   . who said that the order we get file names from readdir is the
>     "unsorted order", and not its reverse?

'readdir' returns the order of the file system,
and 'file_name_completion' reverses it.

>   . in any case, I think we should reverse only when completions-sort
>     is nil, because otherwise we could adversely affect the sorting
>     performed on the results

This means bringing 'Qcompletions_sort' to 'file_name_completion'?
Probably not worth the trouble.  Better to declare the value nil of
`completions-sort' as unsupported.

Anyway this was just an experiment to see how useful is the
no sorting option for completions.

And the conclusion is that it's useful only for part of completion types,
and not useful for others.  It's useless for obarray and file names.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: drew.adams@oracle.com,  64656@debbugs.gnu.org
> Date: Fri, 10 Nov 2023 09:45:02 +0200
> 
> >> A recipe is to customize `completions-sort' to nil ("No sorting"),
> >> then first call `M-& ls -U RET' and afterwards `C-x C-f TAB TAB'
> >> and compare the contents of two buffers *Async Shell Command*
> >> and *Completions*.  The order of files is reversed.
> >
> > OK, I see it now, thanks.
> >
> > But IMO this raises several issues:
> >
> >   . completions-sort affects all completions, not just completions of
> >     file names, right?  So why the change only for file names?
> 
> I'm trying various completions after customizing completions-sort to nil,
> so currently noticed a problem in the completions of file names.

So you agree that the problem is wider than that?

> >   . who said that the order we get file names from readdir is the
> >     "unsorted order", and not its reverse?
> 
> 'readdir' returns the order of the file system,

That is not true in general.  For example, on MS-Windows, it returns
the file names in alphabetical order.  In general, we don't know what
is the relation between the order in which readdir returns file names
and the order of the file entries in the directory on disk, as that is
an implementation detail.

> >   . in any case, I think we should reverse only when completions-sort
> >     is nil, because otherwise we could adversely affect the sorting
> >     performed on the results
> 
> This means bringing 'Qcompletions_sort' to 'file_name_completion'?

Yes.

> Probably not worth the trouble.

Why not?  It's just a single simple test.

> Better to declare the value nil of `completions-sort' as
> unsupported.

I don't see why.

> Anyway this was just an experiment to see how useful is the
> no sorting option for completions.
> 
> And the conclusion is that it's useful only for part of completion types,
> and not useful for others.  It's useless for obarray and file names.

I added Stefan to this discussion, in case he has an opinion or
comments.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 661 bytes --]

>> I'm trying various completions after customizing completions-sort to nil,
>> so currently noticed a problem in the completions of file names.
>
> So you agree that the problem is wider than that?

The problem reported by this bug report is that the order is random by
`C-h v M-n M-n' because it uses obarray.  There is no such problem for
`C-x C-f M-n M-n' because the list of default values is truncated
explicitly in `read-file-name-default' by

  (minibuffer-with-setup-hook
      (lambda ()
        (setq-local minibuffer-default-add-function
          ...

So maybe to use the same to truncate the list of default values
for `C-h f', `C-h v', `C-h o'?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: minibuffer-default-add-function.patch --]
[-- Type: text/x-diff, Size: 4776 bytes --]

diff --git a/lisp/help-fns.el b/lisp/help-fns.el
index e93c535bbef..4931aeb49cd 100644
--- a/lisp/help-fns.el
+++ b/lisp/help-fns.el
@@ -234,18 +234,21 @@ help-fns--describe-function-or-command-prompt
                                   "Describe function")
                                 fn))
          (enable-recursive-minibuffers t)
-         (val (completing-read
-               prompt
-               #'help--symbol-completion-table
-               (lambda (f) (if want-command
-                          (commandp f)
-                        (or (fboundp f) (get f 'function-documentation))))
-               ;; We used `confirm' for a while because we may want to see the
-               ;; meta-info about a function even if the function itself is not
-               ;; defined, but this use case is too marginal and rarely tested,
-               ;; not worth the trouble (bug#64902).
-               t nil nil
-               (and fn (symbol-name fn)))))
+         (val (minibuffer-with-setup-hook
+                  (lambda ()
+                    (setq-local minibuffer-default-add-function nil))
+                (completing-read
+                 prompt
+                 #'help--symbol-completion-table
+                 (lambda (f) (if want-command
+                                 (commandp f)
+                               (or (fboundp f) (get f 'function-documentation))))
+                 ;; We used `confirm' for a while because we may want to see the
+                 ;; meta-info about a function even if the function itself is not
+                 ;; defined, but this use case is too marginal and rarely tested,
+                 ;; not worth the trouble (bug#64902).
+                 t nil nil
+                 (and fn (symbol-name fn))))))
     (unless (equal val "")
       (setq fn (intern val)))
     ;; These error messages are intended to be less technical for the
@@ -1269,18 +1272,21 @@ describe-variable
 	 (enable-recursive-minibuffers t)
          (orig-buffer (current-buffer))
 	 val)
-     (setq val (completing-read
-                (format-prompt "Describe variable" (and (symbolp v) v))
-                #'help--symbol-completion-table
-                (lambda (vv)
-                  (or (get vv 'variable-documentation)
-                      (and (not (keywordp vv))
-                           ;; Since the variable may only exist in the
-                           ;; original buffer, we have to look for it
-                           ;; there.
-                           (buffer-local-boundp vv orig-buffer))))
-                t nil nil
-                (if (symbolp v) (symbol-name v))))
+     (setq val (minibuffer-with-setup-hook
+                   (lambda ()
+                     (setq-local minibuffer-default-add-function nil))
+                 (completing-read
+                  (format-prompt "Describe variable" (and (symbolp v) v))
+                  #'help--symbol-completion-table
+                  (lambda (vv)
+                    (or (get vv 'variable-documentation)
+                        (and (not (keywordp vv))
+                             ;; Since the variable may only exist in the
+                             ;; original buffer, we have to look for it
+                             ;; there.
+                             (buffer-local-boundp vv orig-buffer))))
+                  t nil nil
+                  (if (symbolp v) (symbol-name v)))))
      (list (if (equal val "")
 	       v (intern val)))))
   (let (file-name
@@ -1876,14 +1882,17 @@ describe-symbol
           (v-or-f (if found v-or-f (function-called-at-point)))
           (found (or found v-or-f))
           (enable-recursive-minibuffers t)
-          (val (completing-read (format-prompt "Describe symbol"
-                                               (and found v-or-f))
-				#'help--symbol-completion-table
-				(lambda (vv)
-                                  (cl-some (lambda (x) (funcall (nth 1 x) vv))
-                                           describe-symbol-backends))
-				t nil nil
-				(if found (symbol-name v-or-f)))))
+          (val (minibuffer-with-setup-hook
+                   (lambda ()
+                     (setq-local minibuffer-default-add-function nil))
+                 (completing-read (format-prompt "Describe symbol"
+                                                 (and found v-or-f))
+				  #'help--symbol-completion-table
+				  (lambda (vv)
+                                    (cl-some (lambda (x) (funcall (nth 1 x) vv))
+                                             describe-symbol-backends))
+				  t nil nil
+				  (if found (symbol-name v-or-f))))))
      (list (if (equal val "")
 	       (or v-or-f "") (intern val)))))
   (let ((help-buffer-under-preparation t))

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

> The problem reported by this bug report is that the order is random by
> `C-h v M-n M-n' because it uses obarray.  There is no such problem for
> `C-x C-f M-n M-n' because the list of default values is truncated
> explicitly in `read-file-name-default' by
>
>   (minibuffer-with-setup-hook
>       (lambda ()
>         (setq-local minibuffer-default-add-function
>           ...
>
> So maybe to use the same to truncate the list of default values
> for `C-h f', `C-h v', `C-h o'?

But alas, this won't work for e.g. `C-h w', `C-h S',
and other obarray completions.

So it's a lose-lose situation.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Drew Adams]

> > The problem reported by this bug report is that the order is random by
> > `C-h v M-n M-n' because it uses obarray.

No.  That is NOT the problem reported.  The problem
reported is:

1. Initial completions are added to `M-n' automatically,
   instead of just by a user adding them explicitly to
   DEFAULTS.

2. Given that BUG, _IF_ it won't be fixed, then the doc
   should at least make users aware of it and mention
   how to work around it.

Fixing #1 is preferable.  Fixing #2 is minimal damage
control, IF you won't fix #1.

> > There is no such problem for
> > `C-x C-f M-n M-n' because the list of default values is truncated
> > explicitly in `read-file-name-default' by
> >
> >   (minibuffer-with-setup-hook
> >       (lambda ()
> >         (setq-local minibuffer-default-add-function
> >           ...
> >
> > So maybe to use the same to truncate the list of default values
> > for `C-h f', `C-h v', `C-h o'?
> 
> But alas, this won't work for e.g. `C-h w', `C-h S',
> and other obarray completions.
> 
> So it's a lose-lose situation.

The loss is the introduction of automatically cramming
the completion domain ("initial completions") onto the
`M-n' queue.

Just let programmers themselves control what goes into
`M-n', using DEFAULTS.  IF anyone really wants to add
the completion domain to `M-n' it's trivial to do that,
by adding it to DEFAULTS.  AFAIK, no one had ever done
that, and no one requested the misfeature of adding the
domain to `M-n' automatically.

We should admit the mistake of introducing this forced
polluting of `M-n' and revert that misfeature.  Give
users back complete control of `M-n' through DEFAULTS -
that's what DEFAULTS is for, and its 100% sufficient
and flexible for doing the job.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

>> > The problem reported by this bug report is that the order is random by
>> > `C-h v M-n M-n' because it uses obarray.
>
> No.  That is NOT the problem reported.  The problem
> reported is:
>
> 1. Initial completions are added to `M-n' automatically,
>    instead of just by a user adding them explicitly to
>    DEFAULTS.
>
> 2. Given that BUG, _IF_ it won't be fixed, then the doc
>    should at least make users aware of it and mention
>    how to work around it.
>
> Fixing #1 is preferable.  Fixing #2 is minimal damage
> control, IF you won't fix #1.

#1 can be fixed only partially for some popular obarray completions.
So here is a patch that fixes #1 partially and #2 completely.

diff --git a/lisp/simple.el b/lisp/simple.el
index f86b3f9e208..02005e3b4f9 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -2989,11 +2989,17 @@ minibuffer-default-add-done
 (defun minibuffer-default-add-completions ()
   "Return a list of all completions without the default value.
 This function is used to add all elements of the completion table to
-the end of the list of defaults just after the default value."
+the end of the list of defaults just after the default value.
+When you don't want to add initial completions to the default value,
+you can use either `minibuffer-setup-hook' or `minibuffer-with-setup-hook'
+to set the value of `minibuffer-default-add-function' to nil."
   (let ((def minibuffer-default)
-	(all (all-completions ""
-			      minibuffer-completion-table
-			      minibuffer-completion-predicate)))
+        ;; Avoid some popular completions with undefined order
+        (all (unless (memq minibuffer-completion-table
+                           `(help--symbol-completion-table ,obarray))
+               (all-completions ""
+                                minibuffer-completion-table
+                                minibuffer-completion-predicate))))
     (if (listp def)
 	(append def all)
       (cons def (delete def all)))))

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Juri Linkov]

close 64656 30.0.50
thanks

>> 1. Initial completions are added to `M-n' automatically,
>>    instead of just by a user adding them explicitly to
>>    DEFAULTS.
>>
>> 2. Given that BUG, _IF_ it won't be fixed, then the doc
>>    should at least make users aware of it and mention
>>    how to work around it.
>>
>> Fixing #1 is preferable.  Fixing #2 is minimal damage
>> control, IF you won't fix #1.
>
> #1 can be fixed only partially for some popular obarray completions.
> So here is a patch that fixes #1 partially and #2 completely.

Now pushed to master, and closed.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>   . completions-sort affects all completions, not just completions of
>     file names, right?  So why the change only for file names?

AFAICT many completion tables completely disregard the order in their
output, presuming that it will be reordered later if/when needed.
Clearly `file-name-all-completions` is among them, which is what the
patch proposes to fix.

>   . who said that the order we get file names from readdir is the
>     "unsorted order", and not its reverse?

Nobody, indeed.  Under GNU/Linux with an `ext4` filesystem, the `ls -U`
order is basically random (ext4's htrees use some a hash-based directory
structure), so reversing it doesn't make much difference (unless you
explicitly compare it with `ls -U`).  But for those filesystems that use
a directory structure where files are naturally sorted in a meaningful
way (like alphabetical), returning the directory's order or the reverse
is a lot more noticeable.

>   . in any case, I think we should reverse only when completions-sort
>     is nil, because otherwise we could adversely affect the sorting
>     performed on the results

I don't see how/why.


        Stefan

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "64656@debbugs.gnu.org" <64656@debbugs.gnu.org>
> Date: Tue, 18 Jul 2023 20:27:32 +0000
> 
> > > Hope this clarifies the request/bug report.
> > 
> > It doesn't.  Sorry, I guess I'm too stupid
> > to understand what you are asking.
> 
> Don't be silly or sarcastic, please.

This isn't sarcasm, this is frustration.

And please avoid ad-hominem if you can, especially when you yourself
use language that can easily be interpreted as sarcasm:

> A user cares what's available from `M-n'.
> (I don't expect another "Why?" for that,
> but it comes, I'll try to answer that too.)

Isn't that sarcasm?

So let's try to drop the attitude and discuss the real problems, okay?

(Adding Stefan, because I think he could have insights in this area
that is nowadays complicated enough to blow my mind.)

> Why does it matter that all initial (i.e.,
> prior to any user input in the minibuffer)
> completions are put into the `M-n' queue?
> 
> Because that set of candidates is often
> huge.  And because its order isn't designed
> for `M-n' or (especially) for the particular
> act of input reading.  Often its order has
> no special reason.
> 
> And the case given as an example, `C-h v',
> illustrates that well: (1) zillions of vars, 
> (2) in no user-expectable/understandable/
> useful order - the order of obarray!

This part of your report seems to be a separate issue -- you seem to
be saying that "C-h v" and similar commands should not add all the
variables to the "future history".  It's possible that you are right,
although it could be useful if M-s and M-r in the minibuffer would
actually search that list -- which they don't currently, due to how
this "add to future history" feature is implemented to add elements
lazily (see goto-history-element).

But that is a separate issue, almost unrelated to the Subject of your
report, which is about documentation.  Whatever problems we have in
this area with "C-h v", they cannot be solved by documentation in the
ELisp manual.

So what is the documentation issue?  You say:

> What an Elisp user (not an end user of a
> command) really needs to care about is var
> `minibuffer-default-add-function', not the
> particular function that's its default value.
> 
> Forget for a moment about what various
> function values for that variable might do.
> The most important thing about that var is
> that if nil then the domain of completions
> isn't added to the `M-n' queue at all.  IOW,
> that _turns off_ the automatic filling of
> the `M-n' queue.
> 
> An Elisp user needs to know that fact, if
> s?he uses `completing-read' and s?he wants
> to prevent the kind of confusing overkill
> exhibited by `C-h v'.  (She then needs to
> bind the var to nil around the call to
> `completing-read').
> [...]
> Elisp users thus need to know that to define
> the subset and its order for `M-n' they can
> bind var `minibuffer-default-add-function'
> to a function that returns such a list.
> This isn't obvious.  You won't find it by
> reading the `completing-read' doc, at least,
> though it's just as important to controlling
> the behavior as the args to that function. 

First, M-n is not about completion, it is about minibuffer history.
Completion functions use the minibuffer, so the minibuffer history
affects them, but they are not the only ones affected.  The
documentation of completing-read and of read-from-minibuffer already
state that DEFAULT is added to the "future history":

     The argument DEFAULT specifies default values to make available
     through the history commands.  It should be a string, a list of
     strings, or ‘nil’.  The string or strings become the minibuffer’s
     “future history”, available to the user with ‘M-n’.

What is missing here, it seems, is the hint that this addition can be
controlled, among other measures, via minibuffer-default-add-function,
and the documentation of that variable where the minibuffer history is
documented.  Is that what you are asking for, or is there anything
else?

> I mentioned that I think it would help to
> make some changes to both the Elisp doc and
> the user doc.

Why would Emacs users need to know about this?  The mechanism to
control what and how is added to minibuffer history is not user-level
information; users cannot use it to their benefit.

> In effect, the heads-up tells a user that
> when prompted for input with completion,
> in some cases the "future history" of
> defaults is effectively useless.  And it
> doesn't hurt to tell users why: _all_
> possible domain completions are included,
> possibly in a meaningless order.

If we think that future history in some case is useless, TRT is to
change the code so that it ceases to be useless, not to document that
it is useless.  IOW, we don't document our own bugs, we prefer to fix
them.

So no, we won't be telling this in user documentation.  If we decide
that this behavior of "C-h v" and similar commands is not useful, we
should change it to be more useful.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Drew Adams]

> > > > Hope this clarifies the request/bug report.
> > >
> > > It doesn't.  Sorry, I guess I'm too stupid
> > > to understand what you are asking.
> >
> > Don't be silly or sarcastic, please.
> 
> This isn't sarcasm, this is frustration.
> And please avoid ad-hominem if you can, especially when you yourself
> use language that can easily be interpreted as sarcasm:
> 
> > A user cares what's available from `M-n'.
> > (I don't expect another "Why?" for that,
> > but it comes, I'll try to answer that too.)
> 
> Isn't that sarcasm?

No, it's not.  At all.  It's legitimate for
someone to ask "Why?" for that too.  I was
guessing that I probably didn't need to
explain that, but I wanted to make clear that
I could, and would, if you wanted a reason
for that too.

> So let's try to drop the attitude and discuss the real problems, okay?

Agreed.
(But I'm not showing any negative "attitude".
Please don't look for that a priori.)

> (Adding Stefan, because I think he could have insights in this area
> that is nowadays complicated enough to blow my mind.)

Good.  I'm like you in that.

> > Why does it matter that all initial (i.e.,
> > prior to any user input in the minibuffer)
> > completions are put into the `M-n' queue?
> >
> > Because that set of candidates is often
> > huge.  And because its order isn't designed
> > for `M-n' or (especially) for the particular
> > act of input reading.  Often its order has
> > no special reason.
> >
> > And the case given as an example, `C-h v',
> > illustrates that well: (1) zillions of vars,
> > (2) in no user-expectable/understandable/
> > useful order - the order of obarray!
> 
> This part of your report seems to be a separate issue -- you seem to
> be saying that "C-h v" and similar commands should not add all the
> variables to the "future history".  It's possible that you are right,
> although it could be useful if M-s and M-r in the minibuffer would
> actually search that list -- which they don't currently, due to how
> this "add to future history" feature is implemented to add elements
> lazily (see goto-history-element).

It may seem to be a separate issue.  Actually,
it's just an extreme case, to make clear what
the general problem is.  `C-h v' is as bad as
it can be.

Yes, such extreme cases could be handled
separately. It would be easier, and saner, to
do that if the default behavior for automatic
addition were OFF, not ON.

But I argue that there isn't any good reason,
even in the case where the search space is
small (small completion domain).  Or at least
I'm asking for a rationale behind such default
behavior.  So far, I think there's no good
reason, but I might just be missing something.
 
> But that is a separate issue, almost unrelated to the Subject of your
> report, which is about documentation.

I don't think so (see above).  Cases such as
`C-h v' just point out the problem starkly.

Yes, large completion domains are especially
problematic for this default behavior, because
of the size of the `M-n' queue.  Yes, obarray
is especially problematic for this, because of
its useless order of symbols.

But I'm asking about the general idea behind
this default behavior: What use case(s) does
it really help with?  Even with a small list
of (empty-input) completions, and even when
those are in some meaningful/useful order,
what's the use case for adding them to `M-n',
which is a carefully designed default or list
of defaults?  Why use the completion domain
as a set of defaults - at all?

> Whatever problems we have in this area with
> "C-h v", they cannot be solved by documentation
> in the ELisp manual.

I spoke to this.  With the assumption that the
behavior won't be changed, the doc can at least
help by pointing out that for such cases (large
completion domain, no useful order), you can at
least mitigate the problem:

* You can bind the variable to nil - no addition.
* You can give the variable a custom function
  value that limits the size or sorts to a useful
  order.

Elisp programmers can do that in code they write
that reads minibuffer input (in particular, with
completion).  Emacs users can do that with a
defvar that defines the default value of
`minibuffer-default-add-function' (e.g. as nil).

Currently the doc doesn't point out the problem
or these possible mitigations.  If the problem
isn't fixed then the doc can avert users to it.
 
> So what is the documentation issue?

See above.  Make users aware of the problem
and how to mitigate it a bit - as long as
it exists.

> You say:
> 
> > What an Elisp user (not an end user of a
> > command) really needs to care about is var
> > `minibuffer-default-add-function', not the
> > particular function that's its default value.
> >
> > Forget for a moment about what various
> > function values for that variable might do.
> > The most important thing about that var is
> > that if nil then the domain of completions
> > isn't added to the `M-n' queue at all.  IOW,
> > that _turns off_ the automatic filling of
> > the `M-n' queue.
> >
> > An Elisp user needs to know that fact, if
> > s?he uses `completing-read' and s?he wants
> > to prevent the kind of confusing overkill
> > exhibited by `C-h v'.  (She then needs to
> > bind the var to nil around the call to
> > `completing-read').
> > [...]
> > Elisp users thus need to know that to define
> > the subset and its order for `M-n' they can
> > bind var `minibuffer-default-add-function'
> > to a function that returns such a list.
> > This isn't obvious.  You won't find it by
> > reading the `completing-read' doc, at least,
> > though it's just as important to controlling
> > the behavior as the args to that function.
> 
> First, M-n is not about completion, it is about minibuffer history.
> Completion functions use the minibuffer, so the minibuffer history
> affects them, but they are not the only ones affected.

That's true.  My concern is mainly with the
completion case.  After all, the default value
of `minibuffer-default-add-function' is a
function that only applies to minibuffer
reading with completion.

And I spoke of `completing-read', as a shortcut
to talking about reading with completion more
generally (`read-extended-command-1',
`read-shell-command', `read-file-name-default',
`minibuffer-complete-defaults',...).

> The documentation of completing-read and of read-from-minibuffer already
> state that DEFAULT is added to the "future history":
> 
>      The argument DEFAULT specifies default values to make available
>      through the history commands.  It should be a string, a list of
>      strings, or ‘nil’.  The string or strings become the minibuffer’s
>      “future history”, available to the user with ‘M-n’.

1. It isn't just added to the "future history".
   It's the base (prefix) of the future history.

2. The problem identified has nothing to do with
   putting DEFAULT (single or list) into the
   future history.  That's by definition always
   useful, because it assumes that the programmer
   provides a reasonable set of such defaults.
   Control by human design.

> What is missing here, it seems, is the hint that this addition can be
> controlled, 

Not if by "this addition" you mean the addition
described in that doc you quoted, which is just
addition of DEFAULT.

But if by "this addition" you also mean the
addition that's not mentioned there - automatic
addition of the completion domain, then OK -
THAT addition can (only?) be controlled by var
`minibuffer-default-add-function'.  That var
has no effect on the addition of DEFAULT.

> among other measures, via minibuffer-default-add-function,
> and the documentation of that variable where the minibuffer history is
> documented.  Is that what you are asking for, or is there anything
> else?

See above.

1. I'm asking whether this feature (addition
   of completion domain automatically)
   shouldn't be revisited, maybe even removed,
   and at least default to OFF.

2. If that revisit is NOT to be, then I'm
   asking that the doc at least (a) point out
   that this automatic behavior can be
   problematic, and (b) tell users how to
   (i) turn it off and (ii) control it a bit
   if not turned off.  That control can include
   limiting the size and sorting the elements
   to be added.

> > I mentioned that I think it would help to
> > make some changes to both the Elisp doc and
> > the user doc.
> 
> Why would Emacs users need to know about this?  The mechanism to
> control what and how is added to minibuffer history is not user-level
> information; users cannot use it to their benefit.

Users can defvar the variable value to nil (or
a different function).  That won't help with
some minibuffer reading that binds its own
function, but it can help in many cases.

Users can at least be aware of what's happening,
so they won't wonder "WTF?" when they see odd
`M-n' insertions.  Knowing that, they'll know
enough to stop repeating `M-n' when they get a
few such odd insertions, instead of maybe
thinking that they were one-offs.  IOW, by
being aware of what's happening they can maybe
give up on meaningless `M-n' input sooner.

> > In effect, the heads-up tells a user that
> > when prompted for input with completion,
> > in some cases the "future history" of
> > defaults is effectively useless.  And it
> > doesn't hurt to tell users why: _all_
> > possible domain completions are included,
> > possibly in a meaningless order.
> 
> If we think that future history in some case is useless, TRT is to
> change the code so that it ceases to be useless,

100% agreement.  That's why that was my first
priority request.  Fix this and there likely
will be no, or little, need to change the doc.

I didn't assume that others would agree that
the behavior is harmful.

I thought too that I might just be missing
something.  The behavior seems so bizarre to
me that I don't understand why it would have
been adopted.  Benefit of the doubt made me
guess I'm maybe just misunderstanding.

> not to document that it is useless.  IOW, we
> don't document our own bugs, we prefer to fix
> them.

Agree fully.  Let's see what Stefan, Juri,
etc. think about the actual problem, leaving
doc out of it till the behavior question is
settled.

> So no, we won't be telling this in user documentation.  If we decide
> that this behavior of "C-h v" and similar commands is not useful, we
> should change it to be more useful.

Great.  SGTM.

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "64656@debbugs.gnu.org" <64656@debbugs.gnu.org>
> Date: Thu, 20 Jul 2023 16:45:26 +0000
> 
> Agree fully.  Let's see what Stefan, Juri,
> etc. think about the actual problem, leaving
> doc out of it till the behavior question is
> settled.

I actually decided to document this already.  This behavior exists
since Emacs 23.1(!!), so it's unlikely to go away soon, and it's high
time for it to be documented.

I leave this bug open for discussing the wisdom of adding all the
completion candidates in the likes of "C-h v".

bug#64656: 29.0.91; Doc of minibuffer histories and completing-read - automatic addition of completions to DEFAULT list

[Drew Adams]

> This is not obvious from reading the docs.  For example, it's not
> obvious that if you use `C-h v' and then `M-p', repeating `M-p, you get
> the symbols that are variables, one by one, inserted into the
> minibuffer.

Sorry.  I meant `M-n', not `M-p'.
This is about automatic adding of the
initial completion candidates (before
any input matching) to the "future
history".

(But you probably understood that
there was a typo.)