bug#19064: bug#17272: bug#19064: bug#17272: bug#19064: 25.0.50; `message' overwrites `y-or-n-p' prompt, so user misses it

[Drew Adams <drew.adams@oracle.com>]

> > >>> I think the posters just confused minibuffer
> > >>> and echo area for the case of y-or-n-p then
> > >>> (at least did I).
> > >>
> > >> That's easy to do.  But the statement wasn't just
> > >> about the minibuffer when echo area was meant, or
> > >> vice versa.  The claim was that _user input_ (not
> > >> a prompt) became permanently hidden.
> > >
> > > I don't know if that was also a mistake or really
> > > meant like that.
> >
> > This is not a mistake.

So you're saying it wasn't a mistake for you to say
that _user input_ (not a prompt) is permanently
hidden?

In that case, what's the scenario where user input
is permanently hidden?

My claim was that I don't see how it can happen that
your input in the minibuffer becomes permanently
hidden.  Michael guessed that you maybe meant that a
prompt, not user input, becomes hidden.  But it seems
that's not the case, so please provide a recipe for
the permanent hiding of your input.

> > Permanently hidden user input is a serious
> > problem and security threat.

I'd agree with that, of course - it can be.

> > Today I started compilation, then in a Dired buffer
> > requested files deletion that displayed the prompt:
> >
> >   Delete D [54 files] (y or n)
> >
> > But before I had a chance to answer the prompt,
> > compilation finished and obscured the prompt 

See, now we're back to talking about the _prompt_,
_not user input_ being obscured, right?

The prompt is sent to the echo area.  At least I
think it is.  I thought it was also logged in
`*Messages*', but I don't see it there.  Perhaps
that happens only when the writing to the echo
area uses an explicit call to `message'?

In any case, I don't think it's in the minibuffer.

If you have input in the minibuffer then it should
still be there, after that prompting.  I'm guessing
that `y-or-n-p' is being used.  That function calls
`read-key', which calls `read-key-sequence-vector',
a C function.  Apparently the prompt doesn't get
logged in `*Messages*' - I don't know why.

> > with this message permanently: Compilation finished

Yes, a subsequent message to the echo area can
wipe out a message (in this case a prompt) that
was there.

I'd hope that both messages (the `y-or-n-p' prompt
and the "Compilation finished" message) would be in
`*Messages*', at least.  Doesn't seem good, to me,
that that doesn't seem to be the case.

Not that it would be enough, to solve the problem
described, just to log both messages in `*Messages*'.
But at least that would mean that the prompt would
not be permanently lost.  It would anyway be lost
from the echo area, however - yes.

> > So I forgot about what was in the prompt :-(

I agree that this is an unfortunate scenario.  I
don't want it to happen any more than you do.

But what exactly is the recipe? How was it that
you "requested files deletion that displayed the
prompt" that you show?

I'm using Emacs 26.3.  `dired-do-flagged-delete'
shows `Delete D [54 files] (yes or no)'.  It sounds
like you've maybe set `dired-deletion-confirmer' to
`y-or-n-p'?  (It's default value is `yes-or-no-p'.)

If you had seen `(yes or no)' then you would be
using the minibuffer.  And if you had started to
type, say, `ye' for `yes', then your minibuffer
input would have been obscured only temporarily by
the compilation message.

And the prompt would not have been lost from the
echo area by replacement by a subsequent message
from the compilation process, because `yes-or-no-p'
prompts in the minibuffer, not in the echo area.

If you're using `y-or-n-p' then the problem has
nothing at all to do with the minibuffer, AFAIK.

Unlike `yes-or-no-p', which uses the minibuffer,
`y-or-n-p' prompts using the echo area.  And in
that case the compilation message wipes out the
`y-or-n-p' prompt.

I would think that the first step toward a fix
would be to make sure that the `y-or-n-p' prompt
at least gets logged in `*Messages*'.  (I thought
it would be - I'm surprised.)

But of course that doesn't solve the real problem.

The problem, IIUC, is that messages to the echo
area can wipe out earlier messages there.  And
`y-or-n-p', which uses the echo area for its
"prompt", is (unfortunately, IMO) being used in
your case for an important operation that destroys
data (deletes files).  The default is `yes-or-no-p'
for `dired-deletion-confirmer' for a good reason.

So what's the solution, for multiple writes to the
echo area, including perhaps by async processes?

Using the echo area for a "prompt" is, IMO, not a
great idea.  It's OK for some operations, but not
for something delicate/critical.  Users and code
should not depend on a `y-or-n-p' interaction for
something important.

It's not just about writing a prompt to the echo
area.  The `y-or-n-p' - or any other `read-key'
interaction, is hardly atomic, i.e., modal.

`yes-or-no-p' and other minibuffer interactions
aren't atomic either, but they really don't have
the problem you raised.

(And that's why I responded as I did.  I thought
you were talking about a use of the minibuffer
and losing minibuffer input.)

I made a suggestion in this thread, to present
the `read-key' prompt, e.g. in the case where
you use `y-or-n-p', not in the echo area (danger),
but in a pop-up window:

  To read a key, the prompt basically _tells_ the
  the user to hit a key.  It's not looking to read
  any input into a buffer - minibuffer or otherwise.

  Why not instead just put the prompt in a temporary
  (unselected) popup window or undecorated frame?

And it's possible to use a modal dialog that
keeps the prompt visible until you end/dismiss
the dialog.

One possibility is to add the prompt to the
window that pops up to list the files to be
deleted.

IIUC, the general problem you present is this:

1. You are prompted in the echo area.
2. An async process writes a message to the
   echo area, wiping out the prompt from #1.

That's to be avoided, first of all, IMO.  That
would be the first "fix" - don't do that.

Maybe dialogs should never prompt in the echo area.
Maybe they should instead open a window for the
needed interaction, including the prompt.  Maybe
they should be modal in some cases.

So what about use of the minibuffer as such a
window for a dialog?  For most uses it's fine.
If it had been used in this case, e.g., if the
default value of `dired-deletion-confirmer'
(`yes-or-no-p') were used, then I don't think
you would have had your problem.

Yes, your minibuffer interaction would have
been interrupted by the echo area, to show the
compilation message.  But the prompt and your
partial input would have been restored after
a brief delay, when the minibuffer was again
shown in place of the echo area.

I said "for most uses it's fine."  That does
not mean that the minibuffer is always the
ideal way to realize a prompt-and-read-reply
dialog.  For something critical a modal dialog
could be appropriate.

What about the compilation-finished message?
Maybe that shouldn't just plop down in the echo
area.  IOW, maybe there's some responsibility
here for async operations that report status,
to use another method - something that won't
interfere with the echo area.

There are no doubt other ways to try to deal
with your problem.  But I disagree with either
of the following approaches.  (It's still not
clear to me whether you're trying to do either
of these, but that's been my impression at some
points of this discussion.)

1. Make `y-or-n-p' use the minibuffer.
2. Make `message' turn into `minibuffer-message'
   whenever the minibuffer is active.

I've spoken to both of those.  I don't want to
bore you by repeating a lot, but if you want to
talk more about those then I will.

Summary about them (my opinion):

1. The minibuffer is generally not the place to
read a single character or a key.  It's a buffer
for editing and entering multi-char input.

`y-or-n-p' and other functions that use `read-key'
and similar have good use cases, based on their
advantages, i.e., on their specific behavior.

They do not involve entering text and sending it
by hitting RET.  They're not associated with any
particular window or buffer, with respect to the
user input.  The input action is to just hit a key.

That's their advantage - and their disadvantage.

And yes, they prompt in the echo area (AFAIK),
which is not so great.  We could consider letting
them, alternatively, prompt somewhere else, such
as in a popup.

2. During a minibuffer interaction (i.e., when
the minibuffer is active) all kinds of things
can go on.  That includes recursive minibuffers,
popping up and selecting other windows, using
the echo area for temporal messages - anything
at all, really.

In particular, `message' can be useful when the
minibuffer is active.  So can `minibuffer-message'.
Both are useful; neither replaces the other. 

> > Since Drew doesn't want to improve safety
> > to cover all such cases,

Right; that's a fair thing to say, Juri.  Drew
doesn't want safety.  Drew wants you to lose
your data.

Sheesh.  Where do you get off saying such things?

I spoke only to #1 and #2 above.  Please do not
make `y-or-n-p' read from the minibuffer, and
please do not, when the minibuffer is active,
force `message' to do what `minibuffer-message'
does.  I don't know if either is something you're
trying to do, but both of those would be super
misguided, IMO.  And unnecessary.

> > we need to address these issues one by one [...]
> 
> That's nearly impossible to do, and once you are done,
> new cases will likely be introduced in the future.
> Drew, how would you address this class of problems?

See above.  Hope it helps.  I don't claim to have
all the answers, and I might not understand the
problem well.  But I'm pretty sure that #1 and #2
above are not good.

* Don't use `y-or-n-p' for something important.

* Async operations should maybe not report simply
  by writing to the echo area, since: (1) it shares
  real estate with the minibuffer, and writing to
  it can interrupt a user interaction there; and 
  (2) it can overwrite other messages to the echo
  area (including from other async operations).

* Maybe some important interactions should be modal,
  i.e., more or less blocking you from doing other
  things till the modal interaction is done; and
  maybe blocking reception of some async report
  messages.  (They shouldn't block `C-g', though.)

I'm not the one changing anything.  If you leave
the default value of `dired-deletion-confirmer' as
`yes-or-no-p' then I don't think you'll have the
problem you reported and are trying to fix.  I'm
not saying that there isn't a general potential
problem, but I don't think it's where you said it is.

I'm not the one changing anything.  But those
who are should maybe come up with suggestions -
for general discussion.

Why would this kind of thing be done in a bug
thread (several bug threads?) but at the same
time try to handle a general problem in a
general way?  Why wouldn't it instead be brought
up at emacs-devel, where lots of informed people
can speak to it?

Emacs 27 isn't released yet.  I have a recent
snapshot now, but it doesn't show lots of things
that have apparently been changed recently.  It's
frustrating to get a new Emacs release and find
that lots of stuff that's always worked is broken
because someone implemented what they thought was
an improvement.  Such things should generally be
added as new, optional features, not just replace
longstanding behavior.

IMHO, the minibuffer ain't broke, and likewise
`message' and the echo area.  Can there be bad
use cases, problems that we can identify?  Sure.
But let's not throw out the baby with the bath
water.  Before changing something like `y-or-n-p'
or `message' (again, I'm not sure that's what
you're doing - not clear to me), why not bring
it up for general discussion?