User interaction from multiple threads

User interaction from multiple threads

[Eli Zaretskii]

For background, see bugs #25214 and #32426.  Some additional
discussion was here:

  http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00607.html

I'm trying to revive those past discussion, because we now have a
serious and useful application that bumped into these problems, where
formerly we only had theoretical discussions and toy programs.

After thinking some more about this, I think we should first pay
attention to conceptual issues related to this, before we start
talking about implementation.  The conceptual problem I see here is
the UX when more than one thread needs to interact with the user.

For the following description, recall that normally only the main
thread will wait on the keyboard descriptor (why that happens was
explained in bug#25214).

Use case #1:

  . The main thread is waiting for user input.  The user didn't yet
    type anything
  . A non-main thread runs Lisp that prompts the user for some input

  In this case, we probably want the following input to go to the
  prompting thread, right?  But it might also be the case that the
  user actually wants the input to go to the main thread, e.g. to
  perform some unrelated command.  Should we allow that?  If yes, how
  should Emacs know which thread should receive what the user types?

Use case #2:

  Same as the previous, but now the user is in the middle of typing a
  key sequence, when the non-main thread prompts.  For example,
  suppose the user has typed "C-x".

  What do we want to happen now?  Do we "preempt" the main thread and
  let the following input to go to the prompting thread?  Or do we let
  the prompting thread wait until the main thread reads a full key
  sequence and runs the command bound to it?  If the former, what to
  do with the partial key sequence ("C-x") that the user typed?  If
  the latter, how do we indicate to the user that there is a prompt
  from another thread?

Use case #3:

  Similar, but now 2 or more non-main threads prompt the user, one
  after the other, in quick succession.  What should happen now, and
  how will the user know there are multiple prompts?

Thoughts?

Re: User interaction from multiple threads

[Michael Welsh Duggan]

Eli Zaretskii <eliz@gnu.org> writes:

> For background, see bugs #25214 and #32426.  Some additional
> discussion was here:
>
>   http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00607.html
>
> I'm trying to revive those past discussion, because we now have a
> serious and useful application that bumped into these problems, where
> formerly we only had theoretical discussions and toy programs.
>
> After thinking some more about this, I think we should first pay
> attention to conceptual issues related to this, before we start
> talking about implementation.  The conceptual problem I see here is
> the UX when more than one thread needs to interact with the user.
>
> For the following description, recall that normally only the main
> thread will wait on the keyboard descriptor (why that happens was
> explained in bug#25214).
>
> Use case #1:
>
>   . The main thread is waiting for user input.  The user didn't yet
>     type anything
>   . A non-main thread runs Lisp that prompts the user for some input
>
>   In this case, we probably want the following input to go to the
>   prompting thread, right?  But it might also be the case that the
>   user actually wants the input to go to the main thread, e.g. to
>   perform some unrelated command.  Should we allow that?  If yes, how
>   should Emacs know which thread should receive what the user types?
>
> Use case #2:
>
>   Same as the previous, but now the user is in the middle of typing a
>   key sequence, when the non-main thread prompts.  For example,
>   suppose the user has typed "C-x".
>
>   What do we want to happen now?  Do we "preempt" the main thread and
>   let the following input to go to the prompting thread?  Or do we let
>   the prompting thread wait until the main thread reads a full key
>   sequence and runs the command bound to it?  If the former, what to
>   do with the partial key sequence ("C-x") that the user typed?  If
>   the latter, how do we indicate to the user that there is a prompt
>   from another thread?
>
> Use case #3:
>
>   Similar, but now 2 or more non-main threads prompt the user, one
>   after the other, in quick succession.  What should happen now, and
>   how will the user know there are multiple prompts?
>
> Thoughts?

Here are nmy answers to
your three use cases:

#1 If "waiting for input" means in read-from-minibuffer or something
   similar, I believe that input should go to the the thread.  The other
   thread will have to wait.  If "waiting for input" means idle, it
   should go the the other thread.

#2 You should neveer break a key sequence.  We do not preempt the main
   thread.  The other thread will have to wait.

#3 Each thread should get to ask its question in turn, with its own
   prompt.  Other threads will block until that question is answered.
   The user will know there are multiple prompts because after they
   answer one, they get asked another.

Without trying to figure out how it would be implemented, I think I'd
want the following:

User input has to have a beginning and an end.  The simple cases are
easy: a single full key sequence or a read-from-minibuffer call.  But in
many cases an application has a series of prompts to be asked and
answered in succession.  So these inputs need to be able to be grouped
somehow along with their prompts.  Then, once a group is executing, any
other input-related groups raised by other threads have to wait for the
current input group to finish.  Then they cay be served in FIFO or
random order.

The easiest was I can think of grouping sets of inputs is as some form
of critical section, maybe implemented by mutexes.  This would be
simplified for the programmer with a convenience wrapper, something like
(with-input-group BODY) or some such.

Unfortunately, this proposal suffers in that it would have to be
retroactively added to existing code.  So for existing code, some form
of heuristic will have to be used instead.  The only one that comes to
mind is this: Once an input sequence begins, input is locked to that
thread until that thread ends, explicitly ends input operations (with
some new directive), or (in the case of the main thread) returns to the
command loop.  Now, I can see problems with that.  What happens when a
thread prompts for input and then sits and blocks waiting for something
else?  Unless we can categorize certain blocking operations as ones that
will release the input mutex.

Sorry, this was very much stream-of-thought.  
-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: User interaction from multiple threads

[Michael Albinus]

Michael Welsh Duggan <mwd@md5i.com> writes:

Hi,

>> Use case #1:
>>
>>   . The main thread is waiting for user input.  The user didn't yet
>>     type anything
>>   . A non-main thread runs Lisp that prompts the user for some input
>>
>>   In this case, we probably want the following input to go to the
>>   prompting thread, right?  But it might also be the case that the
>>   user actually wants the input to go to the main thread, e.g. to
>>   perform some unrelated command.  Should we allow that?  If yes, how
>>   should Emacs know which thread should receive what the user types?
>>
>> Use case #2:
>>
>>   Same as the previous, but now the user is in the middle of typing a
>>   key sequence, when the non-main thread prompts.  For example,
>>   suppose the user has typed "C-x".
>>
>>   What do we want to happen now?  Do we "preempt" the main thread and
>>   let the following input to go to the prompting thread?  Or do we let
>>   the prompting thread wait until the main thread reads a full key
>>   sequence and runs the command bound to it?  If the former, what to
>>   do with the partial key sequence ("C-x") that the user typed?  If
>>   the latter, how do we indicate to the user that there is a prompt
>>   from another thread?
>>
>> Use case #3:
>>
>>   Similar, but now 2 or more non-main threads prompt the user, one
>>   after the other, in quick succession.  What should happen now, and
>>   how will the user know there are multiple prompts?
>
> #1 If "waiting for input" means in read-from-minibuffer or something
>    similar, I believe that input should go to the the thread.  The other
>    thread will have to wait.  If "waiting for input" means idle, it
>    should go the the other thread.

I agree. If there is already a prompt in the minibuffer, the input shall
be related to that prompt. If the user wants to do something else (enter
a command for the main thread), she shall cancel the existing prompt with C-g.

Maybe we shall mark the requesting thread for user input somehow, for
example by a modeline indicator with related help-echo.

> #2 You should neveer break a key sequence.  We do not preempt the main
>    thread.  The other thread will have to wait.

I agree. If a user has started typing input, no other thread shall
interrupt this by showing its own prompt, and reading own input. The
other thread has to wait.

> #3 Each thread should get to ask its question in turn, with its own
>    prompt.  Other threads will block until that question is answered.
>    The user will know there are multiple prompts because after they
>    answer one, they get asked another.

Agreed. The prompts must be designed such a way, that a user knows what
she is responding to. She cannot expect a defined order of the prompts.

For example, when answering a question about a file, the prompt shall
always contain the file name the question is related to.

> User input has to have a beginning and an end.  The simple cases are
> easy: a single full key sequence or a read-from-minibuffer call.  But in
> many cases an application has a series of prompts to be asked and
> answered in succession.  So these inputs need to be able to be grouped
> somehow along with their prompts.  Then, once a group is executing, any
> other input-related groups raised by other threads have to wait for the
> current input group to finish.  Then they cay be served in FIFO or
> random order.
>
> The easiest was I can think of grouping sets of inputs is as some form
> of critical section, maybe implemented by mutexes.  This would be
> simplified for the programmer with a convenience wrapper, something like
> (with-input-group BODY) or some such.

Would be useful for new and adapted code.

> Unfortunately, this proposal suffers in that it would have to be
> retroactively added to existing code.  So for existing code, some form
> of heuristic will have to be used instead.  The only one that comes to
> mind is this: Once an input sequence begins, input is locked to that
> thread until that thread ends, explicitly ends input operations (with
> some new directive), or (in the case of the main thread) returns to the
> command loop.  Now, I can see problems with that.  What happens when a
> thread prompts for input and then sits and blocks waiting for something
> else?  Unless we can categorize certain blocking operations as ones that
> will release the input mutex.

I don't agree with this heuristic. Even if we don't run into deadlocks,
one thread with a simple y-or-n-p question would block all other threads
until it finishes, which could take time. We must accept that for not
adapted code user input is requested from different threads, in random
order. The only guarantee is, that user input for the same thread is in
predefined order.

Over the time, code will be adapted for threads to use the grouping
feature as proposed above.

> Sorry, this was very much stream-of-thought.  

Same here.

Best regards, Michael.

Re: User interaction from multiple threads

[martin rudalics]

 >    . The main thread is waiting for user input.  The user didn't yet
 >      type anything
 >    . A non-main thread runs Lisp that prompts the user for some input
 >
 >    In this case, we probably want the following input to go to the
 >    prompting thread, right?  But it might also be the case that the
 >    user actually wants the input to go to the main thread, e.g. to
 >    perform some unrelated command.  Should we allow that?  If yes, how
 >    should Emacs know which thread should receive what the user types?

If we make a rule that each thread owns a frame with a minibuffer, the
decision which thread receives input on a GUI is left to the windowing
subsystem.  We would have to emulate that behavior on a TTY somehow
but that should be an implentational detail.

 >    Same as the previous, but now the user is in the middle of typing a
 >    key sequence, when the non-main thread prompts.  For example,
 >    suppose the user has typed "C-x".
 >
 >    What do we want to happen now?  Do we "preempt" the main thread and
 >    let the following input to go to the prompting thread?  Or do we let
 >    the prompting thread wait until the main thread reads a full key
 >    sequence and runs the command bound to it?  If the former, what to
 >    do with the partial key sequence ("C-x") that the user typed?

The partial key sequence should be preserved and typing resumed where
it was interrupted when the user switches back to the main thread by
selecting a frame owned by the main frame.

 > If
 >    the latter, how do we indicate to the user that there is a prompt
 >    from another thread?
 >
 > Use case #3:
 >
 >    Similar, but now 2 or more non-main threads prompt the user, one
 >    after the other, in quick succession.  What should happen now, and
 >    how will the user know there are multiple prompts?

If each prompt has its own frame, this is trivial.  The user will
answer the prompt of the frame that happens to be selected and will be
able to switch between prompts by selecting their respective frames.

Obviously, for each frame/thread we would have to record its input
state to be able to resume an interaction where the user left off.

martin

Re: User interaction from multiple threads

[Michael Albinus]

martin rudalics <rudalics@gmx.at> writes:

> If we make a rule that each thread owns a frame with a minibuffer,

I don't believe this is feasible. Imagine you'll call "C-x & C-x C-f
.../emacs/lisp/*.el". This would load all Lisp files of Emacs in
parallel, resulting in 250+ frames.

This example sounds strange, but it is one test case in the
feature/tramp-thread-safe branch I'm running regularly.

> martin

Best regards, Michael.

Re: User interaction from multiple threads

[martin rudalics]

 >> If we make a rule that each thread owns a frame with a minibuffer,
 >
 > I don't believe this is feasible. Imagine you'll call "C-x & C-x C-f
 > .../emacs/lisp/*.el". This would load all Lisp files of Emacs in
 > parallel, resulting in 250+ frames.
 >
 > This example sounds strange, but it is one test case in the
 > feature/tramp-thread-safe branch I'm running regularly.

We can make frame creation lazy: Make a frame when the thread wants to
ask a question.

martin

Re: User interaction from multiple threads

[Michael Albinus]

martin rudalics <rudalics@gmx.at> writes:

>>> If we make a rule that each thread owns a frame with a minibuffer,
>>
>> I don't believe this is feasible. Imagine you'll call "C-x & C-x C-f
>> .../emacs/lisp/*.el". This would load all Lisp files of Emacs in
>> parallel, resulting in 250+ frames.
>>
>> This example sounds strange, but it is one test case in the
>> feature/tramp-thread-safe branch I'm running regularly.
>
> We can make frame creation lazy: Make a frame when the thread wants to
> ask a question.

As user, I would be surprised if a new frame appears suddenly.

> martin

Best regards, Michael.

Re: User interaction from multiple threads

[martin rudalics]

 > As user, I would be surprised if a new frame appears suddenly.

Such a frame is like a dialog box - nothing special.  And as user, I
would never launch 250 threads in parallel.  But YMMV.

I do not use Emacs for copying files because I dislike dired and am
used to Commander-like (two-pane) file management.  The Emacs
commander is too light-weight for my taste.  One typical use case here
goes as follows: I mark a number of files and ask the commander to
copy them to the other pane's directory.  The commander pops up a new
window, asks for confirmation, and, through that window, tells me how
much has been copied and how much is left.

In that window, the commander also gives me the opportunity to, at any
time, pause, resume and cancel copying.  And it allows to do copying
in a non-blocking "background" mode.  That latter option is quite
valuable because I only after some time may detect that copying takes
too long and I would like to perform some other synchronous activity
before it has finished.

In this scenario, the entire copying operation is controlled by the
window dedicated to that copying thread and all my dialog with that
thread goes through that window (and maybe some modal child window
popping up on top of it).  Maybe I'm biased - but I would still
consider this a worthwhile behavior.

martin

Re: User interaction from multiple threads

[Michael Albinus]

martin rudalics <rudalics@gmx.at> writes:

> I do not use Emacs for copying files because I dislike dired and am
> used to Commander-like (two-pane) file management.

For the records, my example hasn't anything to do with dired.

But this wasn't the point. I just wanted to show that the proposed
behavior (new frame for every thread with user input) could result in
surprises.

> martin

Best regards, Michael.

Re: User interaction from multiple threads

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

On Wed, 15 Aug 2018, 21:11 Michael Albinus, <michael.albinus@gmx.de> wrote:

>
> But this wasn't the point. I just wanted to show that the proposed
> behavior (new frame for every thread with user input) could result in
> surprises.
>

But if the user really fires off 250 threads, then it makes sense that they
get that many frames. After all, this is a computer and they are supposed
to do what you tell them to do.

In any case, just magically switching focus to a y-or-n-p that was called
in a different thread sounds like it could be incredibly error prone.

Imagine if you're typing in a buffer and you're writing a word that has the
letter y in it. As you are about to insert the y into the buffer, a
background thread asks "are you sure you want to delete all your files?".

Even if you check the minibuffer before you type every key, the thread
could still steal focus just before you type a key.

I don't care much about how this is implemented, but I certainly would not
want focus to be switched off without warning.

Regards,
Elias

>

[-- Attachment #2: Type: text/html, Size: 1815 bytes --]

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Wed, 15 Aug 2018 10:06:40 +0200
> From: martin rudalics <rudalics@gmx.at>
> 
>  >    . The main thread is waiting for user input.  The user didn't yet
>  >      type anything
>  >    . A non-main thread runs Lisp that prompts the user for some input
>  >
>  >    In this case, we probably want the following input to go to the
>  >    prompting thread, right?  But it might also be the case that the
>  >    user actually wants the input to go to the main thread, e.g. to
>  >    perform some unrelated command.  Should we allow that?  If yes, how
>  >    should Emacs know which thread should receive what the user types?
> 
> If we make a rule that each thread owns a frame with a minibuffer, the
> decision which thread receives input on a GUI is left to the windowing
> subsystem.

As you well know, when the user switches to another frame, we
redisplay the mini-window on the newly selected frame to display the
same prompt/echo there as was displayed on the previous selected
frame.  So what you suggest won't work without very serious surgery to
our handling of the mini-window (which is already quite complex and
fragile).

> We would have to emulate that behavior on a TTY somehow
> but that should be an implentational detail.

And that is another complication, of course.

I think that at least at first we should go for some kind of
serialization of using the mini-window, because having several ones
active at the same time is a much harder trick to pull.

>  >    Same as the previous, but now the user is in the middle of typing a
>  >    key sequence, when the non-main thread prompts.  For example,
>  >    suppose the user has typed "C-x".
>  >
>  >    What do we want to happen now?  Do we "preempt" the main thread and
>  >    let the following input to go to the prompting thread?  Or do we let
>  >    the prompting thread wait until the main thread reads a full key
>  >    sequence and runs the command bound to it?  If the former, what to
>  >    do with the partial key sequence ("C-x") that the user typed?
> 
> The partial key sequence should be preserved and typing resumed where
> it was interrupted when the user switches back to the main thread by
> selecting a frame owned by the main frame.

Would be nice, I agree, but again probably not quite simple.

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Wed, 15 Aug 2018 14:27:58 +0200
> From: martin rudalics <rudalics@gmx.at>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> In that window, the commander also gives me the opportunity to, at any
> time, pause, resume and cancel copying.  And it allows to do copying
> in a non-blocking "background" mode.  That latter option is quite
> valuable because I only after some time may detect that copying takes
> too long and I would like to perform some other synchronous activity
> before it has finished.
> 
> In this scenario, the entire copying operation is controlled by the
> window dedicated to that copying thread and all my dialog with that
> thread goes through that window (and maybe some modal child window
> popping up on top of it).  Maybe I'm biased - but I would still
> consider this a worthwhile behavior.

It is a worthy behavior, but it would require that we rethink,
redesign, and reimplement most of the Emacs keyboard stuff and the
higher-level abstractions that read from the minibuffer.  For
starters, recall that reading from the minibuffer is mostly a trick:
we just enter recursive editing.  The keyboard handling code has
almost no idea that something special is going on, and it doesn't
distinguish between "normal" top-level keyboard input and input in
response to a minibuffer prompt.  Your proposal in affect means that
keyboard input should be sensitive to the current buffer, and do
different low-level things in different buffers, like send input to a
different thread.  This completely inverts how things work now: we
first read the input, insert it into the minibuffer, and then
higher-level code works on the text in the minibuffer.  By contrast,
you want input to be sent to keyboard code from a buffer(??).

Re: User interaction from multiple threads

[Michael Albinus]

Elias Mårtenson <lokedhs@gmail.com> writes:

Hi Elias,

> But if the user really fires off 250 threads, then it makes sense that
> they get that many frames. After all, this is a computer and they are
> supposed to do what you tell them to do. 

Why? I don't see a reason to get a new frame for a thread.

> In any case, just magically switching focus to a y-or-n-p that was
> called in a different thread sounds like it could be incredibly error
> prone. 
>
> Imagine if you're typing in a buffer and you're writing a word that
> has the letter y in it. As you are about to insert the y into the
> buffer, a background thread asks "are you sure you want to delete all
> your files?". 

I haven't proposed ever such a scenario. User input is a critical
section in a thread. Once a prompt appears in a minibuffer, the focus of
user input must be changed to another thread untilthe complete input is
read

> Even if you check the minibuffer before you type every key, the thread
> could still steal focus just before you type a key. 
>
> I don't care much about how this is implemented, but I certainly would
> not want focus to be switched off without warning. 

Nobody wants this. I believe, all of us agree with this.

> Regards, 
> Elias 

Best regards, Michael.

Re: User interaction from multiple threads

[martin rudalics]

 > As you well know, when the user switches to another frame, we
 > redisplay the mini-window on the newly selected frame to display the
 > same prompt/echo there as was displayed on the previous selected
 > frame.  So what you suggest won't work without very serious surgery to
 > our handling of the mini-window (which is already quite complex and
 > fragile).

Agreed when I think of extravaganzas like 'yes-or-no-p' calling
'read-from-minibuffer' and prompting in the minibuffer and 'y-or-n-p'
calling 'read-key-sequence' and prompting in the echo area and using
'cursor-in-echo-area' to make one virtually look like the other.  But
in this thread you wanted to "first pay attention to conceptual issues
related to this, before we start talking about implementation" and
that's what I've tried to do.

 > I think that at least at first we should go for some kind of
 > serialization of using the mini-window, because having several ones
 > active at the same time is a much harder trick to pull.

Isn't serialization the hard trick we're pulling ever since?  How to
maintain the order and duration of objects appearing in the
mini-window, how to avoid that tooltip or eldoc text shadows whatever
goes in there?

martin

Re: User interaction from multiple threads

[martin rudalics]

 > It is a worthy behavior, but it would require that we rethink,
 > redesign, and reimplement most of the Emacs keyboard stuff and the
 > higher-level abstractions that read from the minibuffer.

... or a key sequence ...

 > For
 > starters, recall that reading from the minibuffer is mostly a trick:
 > we just enter recursive editing.  The keyboard handling code has
 > almost no idea that something special is going on, and it doesn't
 > distinguish between "normal" top-level keyboard input and input in
 > response to a minibuffer prompt.  Your proposal in affect means that
 > keyboard input should be sensitive to the current buffer, and do
 > different low-level things in different buffers, like send input to a
 > different thread.  This completely inverts how things work now: we
 > first read the input, insert it into the minibuffer,

... the (currently) active minibuffer ...

 > and then
 > higher-level code works on the text in the minibuffer.  By contrast,
 > you want input to be sent to keyboard code from a buffer(??).

I'd want to relax the organization of minibuffers.  Instead of
maintaining them in form of a stack, allow random access to the
minibuffer list.  Instead of having always only the one on top of the
stack active, allow to make any of them the active one.  And allow any
of them to appear in its own dedicated minibuffer window where the
selectedness of that window's frame indicates the currently active
minibuffer.

martin

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Thu, 16 Aug 2018 14:20:58 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-devel@gnu.org
> 
>  > I think that at least at first we should go for some kind of
>  > serialization of using the mini-window, because having several ones
>  > active at the same time is a much harder trick to pull.
> 
> Isn't serialization the hard trick we're pulling ever since?  How to
> maintain the order and duration of objects appearing in the
> mini-window, how to avoid that tooltip or eldoc text shadows whatever
> goes in there?

The simplest idea is to have every primitive that displays in the echo
area to take a certain mutex before it does so.

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Thu, 16 Aug 2018 14:21:21 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: michael.albinus@gmx.de, emacs-devel@gnu.org
> 
>  > and then
>  > higher-level code works on the text in the minibuffer.  By contrast,
>  > you want input to be sent to keyboard code from a buffer(??).
> 
> I'd want to relax the organization of minibuffers.  Instead of
> maintaining them in form of a stack, allow random access to the
> minibuffer list.  Instead of having always only the one on top of the
> stack active, allow to make any of them the active one.  And allow any
> of them to appear in its own dedicated minibuffer window where the
> selectedness of that window's frame indicates the currently active
> minibuffer.

That is a fine plan, but how does it make the issue at hand easier to
solve?

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Michael Welsh Duggan <mwd@md5i.com>
> Date: Tue, 14 Aug 2018 12:42:04 -0400
> 
> #1 If "waiting for input" means in read-from-minibuffer or something
>    similar, I believe that input should go to the the thread.  The other
>    thread will have to wait.  If "waiting for input" means idle, it
>    should go the the other thread.

"Waiting for input" means waiting for user to type something.  That is
normal Emacs situation when the user does nothing, so unconditionally
sending input to the main thread and letting the other thread wait
would mean that other thread might wait indefinitely.

> #2 You should neveer break a key sequence.  We do not preempt the main
>    thread.  The other thread will have to wait.

Do we need to tell the user the other thread wants to prompt?

> User input has to have a beginning and an end.  The simple cases are
> easy: a single full key sequence or a read-from-minibuffer call.  But in
> many cases an application has a series of prompts to be asked and
> answered in succession.  So these inputs need to be able to be grouped
> somehow along with their prompts.  Then, once a group is executing, any
> other input-related groups raised by other threads have to wait for the
> current input group to finish.  Then they cay be served in FIFO or
> random order.
> 
> The easiest was I can think of grouping sets of inputs is as some form
> of critical section, maybe implemented by mutexes.  This would be
> simplified for the programmer with a convenience wrapper, something like
> (with-input-group BODY) or some such.
> 
> Unfortunately, this proposal suffers in that it would have to be
> retroactively added to existing code.

Not necessarily, it could be added to the corresponding primitives.

Thanks.

Re: User interaction from multiple threads

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> The simplest idea is to have every primitive that displays in the echo
> area to take a certain mutex before it does so.

Agreed. Some days ago, I've tried to implement such an approach in the
feature/tramp-thread-safe branch. This failed miserably, due to my lack
of understanding how echo area, minibuffer and keyboard events are
handled in Emacs. Some help would be appreciated for a proof of concept.

(Maybe it is too early to ask for, since we are still collecting ideas)

Best regards, Michael.

Re: User interaction from multiple threads

[Michael Welsh Duggan]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Michael Welsh Duggan <mwd@md5i.com>
>> Date: Tue, 14 Aug 2018 12:42:04 -0400
>> 
>> #1 If "waiting for input" means in read-from-minibuffer or something
>>    similar, I believe that input should go to the the thread.  The other
>>    thread will have to wait.  If "waiting for input" means idle, it
>>    should go the the other thread.
>
> "Waiting for input" means waiting for user to type something.  That is
> normal Emacs situation when the user does nothing, so unconditionally
> sending input to the main thread and letting the other thread wait
> would mean that other thread might wait indefinitely.

To clarify, I believe the command-loop (main-thread), aside from a large
amount of maintainance, is fundamentally a loop around
read-key-sequence, lookup-key, and command-execute.  *This*
read-key-sequence needs to be interruptable by other threads as long as
no keys have yet been entered into the sequence.  This is the point I
considered to be "idling."  If any other thread, or even the main
thread, calls read-key-sequence outside of the main command-loop, that
one should get the input mutex immediately.

You probably understand what I mean at this point, but to clarify, in
case: Another thread should not be blocked for input because the main
thread is waiting for an event, unless the main thread is waiting for
that event because it explicitly called read-key-sequence or some other
higher-level input function (such as read-from-minibuffer) from anything
other than the command loop.  But the instant the command loop's
read-key-sequence reads a single key, it must have the input mutex until
it has finished reading a key sequence (or is quit).

Now, another command -- say, read-from-minibuffer -- could end up
calling the command loop, which will look for events.  That command
should have set the input mutex beforehand, and, to be clear, this
should be a recursive mutex, so it won't be interrupted in this context.
A call to recursive-edit, however, would not (I think) grab the input
mutex, and as such would still yield to an input request from another
thread.

I have no idea what would be caused by a non-main thread calling
recursive-edit.  That way lies madness.  But someone should consider
what it might mean.


>> #2 You should neveer break a key sequence.  We do not preempt the main
>>    thread.  The other thread will have to wait.
>
> Do we need to tell the user the other thread wants to prompt?

No.  Just wait for the current input sequence to finish, then allow the
prompt.  I suppose there could be indicators in the mode line, or
something like that, but I don't think we need that initialy, and I have
no idea whether that would actually be desirable anyway.

>> User input has to have a beginning and an end.  The simple cases are
>> easy: a single full key sequence or a read-from-minibuffer call.  But in
>> many cases an application has a series of prompts to be asked and
>> answered in succession.  So these inputs need to be able to be grouped
>> somehow along with their prompts.  Then, once a group is executing, any
>> other input-related groups raised by other threads have to wait for the
>> current input group to finish.  Then they cay be served in FIFO or
>> random order.
>> 
>> The easiest was I can think of grouping sets of inputs is as some form
>> of critical section, maybe implemented by mutexes.  This would be
>> simplified for the programmer with a convenience wrapper, something like
>> (with-input-group BODY) or some such.
>> 
>> Unfortunately, this proposal suffers in that it would have to be
>> retroactively added to existing code.
>
> Not necessarily, it could be added to the corresponding primitives.
>
> Thanks.

No problem.  As my employer makes it unfeasibly difficult to contribute
code to the FSF (would have to have a separate disclaimer for every
feature), participating in discussions like this is the least I can do.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: User interaction from multiple threads

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 2068 bytes --]

On Thu, 16 Aug 2018 at 02:04, Michael Albinus <michael.albinus@gmx.de>
wrote:

> Elias Mårtenson <lokedhs@gmail.com> writes:
>
> Hi Elias,
>
> > But if the user really fires off 250 threads, then it makes sense that
> > they get that many frames. After all, this is a computer and they are
> > supposed to do what you tell them to do.
>
> Why? I don't see a reason to get a new frame for a thread.
>

I don't think we're in disagreement actually. I was referring to the
specific case where a thread explicitly creates a frame. If you were to
fire off 250 of ‘create-a-new-frame’ threads, you'd expect to have 250
frames created.

But who would run 250 ‘create-new-frame’ calls at all, threads or no
threads?


> > Imagine if you're typing in a buffer and you're writing a word that
> > has the letter y in it. As you are about to insert the y into the
> > buffer, a background thread asks "are you sure you want to delete all
> > your files?".
>
> I haven't proposed ever such a scenario. User input is a critical
> section in a thread. Once a prompt appears in a minibuffer, the focus of
> user input must be changed to another thread untilthe complete input is
> read
>

You mean, the focus must be manually shifted from the active buffer to the
minibuffer? If so, then that sounds good.

This reminds me of a book I recently read, “The Apollo Guidance Computer —
Architecture and Operation”. In it, they describe how the user interface
worked on the limited hardware which had 3 main 50digit numeric input
fields, a few smaller numeric fields and 14 or so buttons. They had an
indicator light labelled KEY REL (“key release”) that indicated that a
background job required operator attention. Pressing a similarly labelled
button switched the display to that of the waiting task).

It seems as though some of these same questions were being asked back in
the 60's. :-)

Here's a picture of the display unit for anyone who is curious:
https://www.ibiblio.org/apollo/RealDSKY.jpg

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 2838 bytes --]

Re: User interaction from multiple threads

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 212 bytes --]

On Fri, 17 Aug 2018 at 12:26, Elias Mårtenson <lokedhs@gmail.com> wrote:

3 main 50digit numeric input fields, a few smaller numeric fields and 14 or
> so buttons
>

Oops, I mean 5-digit. Not 50 digit…

[-- Attachment #2: Type: text/html, Size: 563 bytes --]

Re: User interaction from multiple threads

[martin rudalics]

 >> I'd want to relax the organization of minibuffers.  Instead of
 >> maintaining them in form of a stack, allow random access to the
 >> minibuffer list.  Instead of having always only the one on top of the
 >> stack active, allow to make any of them the active one.  And allow any
 >> of them to appear in its own dedicated minibuffer window where the
 >> selectedness of that window's frame indicates the currently active
 >> minibuffer.
 >
 > That is a fine plan, but how does it make the issue at hand easier to
 > solve?

Calling it a "fine plan" probably means that you doubt it can be
implemented.  Maybe you're right.

Anyway, the issues you raised when you started this thread were:

 > Use case #1:
 >
 >   . The main thread is waiting for user input.  The user didn't yet
 >     type anything
 >   . A non-main thread runs Lisp that prompts the user for some input
 >
 >   In this case, we probably want the following input to go to the
 >   prompting thread, right?  But it might also be the case that the
 >   user actually wants the input to go to the main thread, e.g. to
 >   perform some unrelated command.  Should we allow that?  If yes, how
 >   should Emacs know which thread should receive what the user types?

The last question you ask here would be resolved because the user types
"into a window" and that window is unambiguously owned by the thread
that should receive what the user types.

 > Use case #2:
 >
 >   Same as the previous, but now the user is in the middle of typing a
 >   key sequence, when the non-main thread prompts.  For example,
 >   suppose the user has typed "C-x".
 >
 >   What do we want to happen now?  Do we "preempt" the main thread and
 >   let the following input to go to the prompting thread?  Or do we let
 >   the prompting thread wait until the main thread reads a full key
 >   sequence and runs the command bound to it?  If the former, what to
 >   do with the partial key sequence ("C-x") that the user typed?  If
 >   the latter, how do we indicate to the user that there is a prompt
 >   from another thread?

The non-main thread would prompt in another frame and the user can
choose whether to continue with her input or have a look at the
prompting frame and answer that prompt.

 > Use case #3:
 >
 >   Similar, but now 2 or more non-main threads prompt the user, one
 >   after the other, in quick succession.  What should happen now, and
 >   how will the user know there are multiple prompts?

Each prompt has its own frame and the user can choose the order in
which to answer prompts by switching to the corresponding frame first.

martin

Re: User interaction from multiple threads

[Michael Albinus]

Elias Mårtenson <lokedhs@gmail.com> writes:

Hi Elias,

> I don't think we're in disagreement actually. I was referring to the
> specific case where a thread explicitly creates a frame. If you were
> to fire off 250 of ‘create-a-new-frame’ threads, you'd expect to have
> 250 frames created.

Well, we aren't in disagreement. I spoke about the case of "C-x & C-x
C-f .../emacs/lisp/*.el". This opens 250 buffers, all in the same frame.

> But who would run 250 ‘create-new-frame’ calls at all, threads or no
> threads?

Nobody. At least not me :-)

> This reminds me of a book I recently read, “The Apollo Guidance
> Computer — Architecture and Operation”. In it, they describe how the
> user interface worked on the limited hardware which had 3 main 50digit
> numeric input fields, a few smaller numeric fields and 14 or so
> buttons. They had an indicator light labelled KEY REL (“key release”)
> that indicated that a background job required operator attention.
> Pressing a similarly labelled button switched the display to that of
> the waiting task).
>
> It seems as though some of these same questions were being asked back
> in the 60's. :-)
>
> Here's a picture of the display unit for anyone who is curious:
> https://www.ibiblio.org/apollo/RealDSKY.jpg

Nice. Does it run Emacs? :-)

> Regards,
> Elias

Best regards, Michael.

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Fri, 17 Aug 2018 09:24:31 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: michael.albinus@gmx.de, emacs-devel@gnu.org
> 
>  >> I'd want to relax the organization of minibuffers.  Instead of
>  >> maintaining them in form of a stack, allow random access to the
>  >> minibuffer list.  Instead of having always only the one on top of the
>  >> stack active, allow to make any of them the active one.  And allow any
>  >> of them to appear in its own dedicated minibuffer window where the
>  >> selectedness of that window's frame indicates the currently active
>  >> minibuffer.
>  >
>  > That is a fine plan, but how does it make the issue at hand easier to
>  > solve?
> 
> Calling it a "fine plan" probably means that you doubt it can be
> implemented.  Maybe you're right.

No, I actually meant "fine" literally.  I didn't think about it enough
to reach any conclusions, but I certainly don't see up front why
wouldn't it be doable.

>  > Use case #1:
>  >
>  >   . The main thread is waiting for user input.  The user didn't yet
>  >     type anything
>  >   . A non-main thread runs Lisp that prompts the user for some input
>  >
>  >   In this case, we probably want the following input to go to the
>  >   prompting thread, right?  But it might also be the case that the
>  >   user actually wants the input to go to the main thread, e.g. to
>  >   perform some unrelated command.  Should we allow that?  If yes, how
>  >   should Emacs know which thread should receive what the user types?
> 
> The last question you ask here would be resolved because the user types
> "into a window" and that window is unambiguously owned by the thread
> that should receive what the user types.

You might be thinking about some mechanism we don't have, and didn't
discuss yet, because currently windows and buffers are not "owned" by
threads.  Any thread can modify any buffer and call any functions that
manipulate windows.

>  > Use case #2:
>  >
>  >   Same as the previous, but now the user is in the middle of typing a
>  >   key sequence, when the non-main thread prompts.  For example,
>  >   suppose the user has typed "C-x".
>  >
>  >   What do we want to happen now?  Do we "preempt" the main thread and
>  >   let the following input to go to the prompting thread?  Or do we let
>  >   the prompting thread wait until the main thread reads a full key
>  >   sequence and runs the command bound to it?  If the former, what to
>  >   do with the partial key sequence ("C-x") that the user typed?  If
>  >   the latter, how do we indicate to the user that there is a prompt
>  >   from another thread?
> 
> The non-main thread would prompt in another frame and the user can
> choose whether to continue with her input or have a look at the
> prompting frame and answer that prompt.

That just postpones the question, because now we need to have some
infrastructure in place that "knows" which frame's input goes to what
thread.  And it leaves intact the fundamental implementation-level
difficulty, about which I will write shortly: namely, that we have
only one input "pipeline", which needs to somehow be enhanced to
multiplex between several threads.  The fact that the prompt is shown
in another frame doesn't resolve the difficulty with channeling the
input in response to that prompt, especially when multiple different
prompts are being displayed at the same time.

>  > Use case #3:
>  >
>  >   Similar, but now 2 or more non-main threads prompt the user, one
>  >   after the other, in quick succession.  What should happen now, and
>  >   how will the user know there are multiple prompts?
> 
> Each prompt has its own frame and the user can choose the order in
> which to answer prompts by switching to the corresponding frame first.

See above.

Thanks.

Re: User interaction from multiple threads

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1727 bytes --]

On Fri, 17 Aug 2018 at 15:33, Michael Albinus <michael.albinus@gmx.de>
wrote:

Hi Michael,


> Well, we aren't in disagreement. I spoke about the case of "C-x & C-x
> C-f .../emacs/lisp/*.el". This opens 250 buffers, all in the same frame.
>

But that happens without threading as well. Or did you mean “all in their
own frames”? That would indeed be terrible.

Since I'm already posting to the list, I just want to raise a point that I
have been thinking about as I have been reading this discussion.
Personally, I don't see why most (any?) commands needs C-x &. Most of the
time, the user wouldn't know that a certain operation is going to take time
and therefore needs C-x &. Someone else mentioned on this list that a
better solution would be to simply open a new buffer, but letting its
contents say “loading…” if it takes too long. The same could be the case
for things like Gnus or any other long-running command.

Killing the buffer would kill the process, in case it's hung.

Perhaps I didn't understand the issues that were raised in this thread, but
it wasn't clear to me why C-x & would ever be needed (or rather, why its
absence would ever be needed).


> > It seems as though some of these same questions were being asked back
> > in the 60's. :-)
> >
> > Here's a picture of the display unit for anyone who is curious:
> > https://www.ibiblio.org/apollo/RealDSKY.jpg
>
> Nice. Does it run Emacs? :-)
>

It is a 15-bit machine with 48 kwords of memory. 2 of those kwords were RAM
and the rest ROM. The CPU was 2 MHz, with a very cumbersome instruction set.

So yes, I'm pretty sure you could get some incarnation of Emacs running on
it. :-)

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 2447 bytes --]

Re: User interaction from multiple threads

[Michael Albinus]

Elias Mårtenson <lokedhs@gmail.com> writes:

> Hi Michael,

Hi Elias,

>     Well, we aren't in disagreement. I spoke about the case of "C-x &
>     C-x C-f .../emacs/lisp/*.el". This opens 250 buffers, all in the
>     same frame.
>
> But that happens without threading as well. Or did you mean “all in
> their own frames”? That would indeed be terrible.

Sure, this happens also w/o threads. The problem comes, when some of the
visited files need user input while loading into the buffer. W/o
threads, this happens one after the other. With threads, such user input
requests from different buffers could be mixed, which is a problem.

See <https://debbugs.gnu.org/32426>, where I have described such a scenario.

> Since I'm already posting to the list, I just want to raise a point
> that I have been thinking about as I have been reading this
> discussion. Personally, I don't see why most (any?) commands needs C-x
> &. Most of the time, the user wouldn't know that a certain operation
> is going to take time and therefore needs C-x &. Someone else
> mentioned on this list that a better solution would be to simply open
> a new buffer, but letting its contents say “loading…” if it takes too
> long. The same could be the case for things like Gnus or any other
> long-running command.
>
> Killing the buffer would kill the process, in case it's hung.
>
> Perhaps I didn't understand the issues that were raised in this
> thread, but it wasn't clear to me why C-x & would ever be needed (or
> rather, why its absence would ever be needed).

Your proposal means that we always visit files asynchronously. Do we
want this unconditionally?

Of course you could configure Emacs for a similar behavior in your own
stanza. Set execute-file-commands-asynchronously to t (in the
feature/tramp-thread-safe branch):

--8<---------------cut here---------------start------------->8---
Non-nil means visit file asynchronously when called interactively.
If it is a regular expression, it must match the file name to be
visited.  This behavior is toggled by C-x &
prior the command invocation.
--8<---------------cut here---------------end--------------->8---

> Regards,
> Elias

Best regards, Michael.

Re: User interaction from multiple threads

[martin rudalics]

 > You might be thinking about some mechanism we don't have, and didn't
 > discuss yet, because currently windows and buffers are not "owned" by
 > threads.

That's what I meant in my first post of this thread with "If we make a
rule that each thread owns a frame with a minibuffer,".

 > Any thread can modify any buffer and call any functions that
 > manipulate windows.

A frame owned by a thread can be manipulated only by that thread.
Some mechanism may be needed to delete such frames when a thread dies
and is no more able to do that itself.

 >> The non-main thread would prompt in another frame and the user can
 >> choose whether to continue with her input or have a look at the
 >> prompting frame and answer that prompt.
 >
 > That just postpones the question, because now we need to have some
 > infrastructure in place that "knows" which frame's input goes to what
 > thread.

To the thread that owns that frame.

 > And it leaves intact the fundamental implementation-level
 > difficulty, about which I will write shortly: namely, that we have
 > only one input "pipeline", which needs to somehow be enhanced to
 > multiplex between several threads.  The fact that the prompt is shown
 > in another frame doesn't resolve the difficulty with channeling the
 > input in response to that prompt, especially when multiple different
 > prompts are being displayed at the same time.

There would still be only one input "pipeline" at any time, namely
that of the selected frame.  Just that switching frames now would call
for "saving" the input pipeline of the frame switched from and
"restoring" any input pipeline of the frame switched to.  IIUC we are
able to save and restore an input pipeline when we enter and leave a
recursive minibuffer so the mechanism for doing such switches seems to
exist for minibuffer reading.  And reading a key sequence through the
echo area is IIUC synchronous anyway so a user cannot practically
switch frames during that.

martin

Re: User interaction from multiple threads

[Eric Abrahamsen]

martin rudalics <rudalics@gmx.at> writes:

>> You might be thinking about some mechanism we don't have, and didn't
>> discuss yet, because currently windows and buffers are not "owned" by
>> threads.
>
> That's what I meant in my first post of this thread with "If we make a
> rule that each thread owns a frame with a minibuffer,".

I wonder if child frames could be somehow extended for this purpose.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Michael Welsh Duggan <mwd@md5i.com>
> Cc: Michael Welsh Duggan <mwd@md5i.com>,  emacs-devel@gnu.org
> Date: Thu, 16 Aug 2018 22:59:57 -0400

Let me start responding from the end:

> No problem.  As my employer makes it unfeasibly difficult to contribute
> code to the FSF (would have to have a separate disclaimer for every
> feature), participating in discussions like this is the least I can do.

Does your contract allow you to share ideas?  How about share ideas
that are later attributed to you when those ideas are implemented in
code?  If you can do that, then describing an idea of the
implementation in sufficient detail will allow someone else to write
the code relatively easily, and AFAIU won't be copyrightable or
subject to IP restrictions.  (But IANAL.)

> >> #1 If "waiting for input" means in read-from-minibuffer or something
> >>    similar, I believe that input should go to the the thread.  The other
> >>    thread will have to wait.  If "waiting for input" means idle, it
> >>    should go the the other thread.
> >
> > "Waiting for input" means waiting for user to type something.  That is
> > normal Emacs situation when the user does nothing, so unconditionally
> > sending input to the main thread and letting the other thread wait
> > would mean that other thread might wait indefinitely.
> 
> To clarify, I believe the command-loop (main-thread), aside from a large
> amount of maintainance, is fundamentally a loop around
> read-key-sequence, lookup-key, and command-execute.  *This*
> read-key-sequence needs to be interruptable by other threads as long as
> no keys have yet been entered into the sequence.  This is the point I
> considered to be "idling."  If any other thread, or even the main
> thread, calls read-key-sequence outside of the main command-loop, that
> one should get the input mutex immediately.

I deliberately avoided saying anything about being "idle", because in
the context of this discussion, the meaning is overloaded several
times, and is likely to cause confusion.  So let me now describe the
low-level details which are IMO pertinent to the issue at hand.  (I
suggest to review the relevant code parts: thread_select and the
functions it calls, and wait_reading_process_output.)

When any thread in Emacs is about to become "idle", i.e. has nothing
else to do except wait for input (in particular, but not necessarily,
because it needs the user to respond to some prompt), it releases the
global lock and calls pselect.  The descriptors on which pselect will
wait are computed by compute_input_wait_mask, which skips any
descriptors already being waited for by some other thread; this has
the side effect of having only the main thread wait on keyboard input,
at least in most cases.

The command loop causes pselect to be called with a very large timeout
(unless there are active timers, in which case the timeout is computed
to end when the first timer expires).  So the "idling" thread will be
parked inside the pselect call until that timeout expires.  And since
the global lock is released and up for grabs, another thread may take
the lock and begin/continue running during that "idle wait".  When
that other thread attempts to read from the minibuffer (a.k.a. "become
idle"), it will also release the global lock and call pselect.  What
will happen then depends on whether or not the main thread still
waits:

   . If the main thread still waits, the non-main thread will NOT wait
     on keyboard input, and we now have 2 threads parked inside
     pselect; user input will go to the main thread.
   . If the main thread's timeout expired, it is waiting to take the
     global lock, so it will now take it and loop back to the next
     pselect call.  We therefore have a race between the main and
     non-main thread: there's a small probability that the non-main
     thread will call compute_input_wait_mask first, in which case it
     will wait on the keyboard input.  (This situation could be
     triggered with timers that fire at high frequency, I think.)

Therefore, your idea of "interrupting" an "idling" main thread would
need to somehow cause it exit the pselect call, before the non-main
thread is about to prompt the user, and also prevent it from calling
another read_key_sequence until the non-main thread finishes its
interaction with the user.  Interrupting the pselect call could be
emulated by using a very short timeout, but that is unfriendly to
laptops when Emacs is _really_ idle.  Another way to interrupt pselect
is with some signal and setjmp/longjmp.  We then need a mutex to
prevent the read_key_sequence call during user interaction, so that
the interacting thread could arrange for waiting on the keyboard
descriptor.

Does this make sense, given the details in the related code?

> You probably understand what I mean at this point, but to clarify, in
> case: Another thread should not be blocked for input because the main
> thread is waiting for an event, unless the main thread is waiting for
> that event because it explicitly called read-key-sequence or some other
> higher-level input function (such as read-from-minibuffer) from anything
> other than the command loop.  But the instant the command loop's
> read-key-sequence reads a single key, it must have the input mutex until
> it has finished reading a key sequence (or is quit).

The last part will be non-trivial to translate into code, because I'm
not sure the related code is aware of where we are in the process of
reading a key sequence, and there are many different paths through
that code.  But I may be wrong.

> Now, another command -- say, read-from-minibuffer -- could end up
> calling the command loop, which will look for events.  That command
> should have set the input mutex beforehand, and, to be clear, this
> should be a recursive mutex, so it won't be interrupted in this context.

We will need to enhance the mutexes we use, because currently I think
we use non-recursive mutexes on Posix-ish platforms, and recursive
ones on MS-Windows.

> A call to recursive-edit, however, would not (I think) grab the input
> mutex, and as such would still yield to an input request from another
> thread.

Not sure about that, we need to review the various uses of
recursive-edit first.  For example, read-from-minibuffer uses
recursive-edit, and I think we do need that to lock out input from
other threads.

> I have no idea what would be caused by a non-main thread calling
> recursive-edit.  That way lies madness.  But someone should consider
> what it might mean.

Well, read-from-minibuffer calls (a subroutine of) recursive-edit, so
we should definitely support that madness...

> >> #2 You should neveer break a key sequence.  We do not preempt the main
> >>    thread.  The other thread will have to wait.
> >
> > Do we need to tell the user the other thread wants to prompt?
> 
> No.  Just wait for the current input sequence to finish, then allow the
> prompt.  I suppose there could be indicators in the mode line, or
> something like that, but I don't think we need that initialy, and I have
> no idea whether that would actually be desirable anyway.

I think an indication is extremely desirable, but I agree it could be
tricky to produce, even on the mode line.

Thanks.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Fri, 17 Aug 2018 15:48:26 +0800
> Cc: martin rudalics <rudalics@gmx.at>, Eli Zaretskii <eliz@gnu.org>, emacs-devel <emacs-devel@gnu.org>
> 
> It is a 15-bit machine with 48 kwords of memory. 2 of those kwords were RAM and the rest ROM. The CPU
> was 2 MHz, with a very cumbersome instruction set.
> 
> So yes, I'm pretty sure you could get some incarnation of Emacs running on it. :-)

We could probably extend the MS-DOS port of Emacs to do that.  After
all, what's the difference between 15-bit words and 16-bit words? ;-)

Re: User interaction from multiple threads

[Michael Albinus]

martin rudalics <rudalics@gmx.at> writes:

> That's what I meant in my first post of this thread with "If we make a
> rule that each thread owns a frame with a minibuffer,".

Again, with that rule we can forget visiting files with wildcards. IMHO.

What I could imagine is, that if a non-main thread requires user input,
it creates something like a dialog box temporarily. This doesn't need to
have all minibuffer functionality.

> martin

Best regards, Michael.

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Fri, 17 Aug 2018 10:34:54 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: michael.albinus@gmx.de, emacs-devel@gnu.org
> 
>  > That just postpones the question, because now we need to have some
>  > infrastructure in place that "knows" which frame's input goes to what
>  > thread.
> 
> To the thread that owns that frame.
> 
>  > And it leaves intact the fundamental implementation-level
>  > difficulty, about which I will write shortly: namely, that we have
>  > only one input "pipeline", which needs to somehow be enhanced to
>  > multiplex between several threads.  The fact that the prompt is shown
>  > in another frame doesn't resolve the difficulty with channeling the
>  > input in response to that prompt, especially when multiple different
>  > prompts are being displayed at the same time.
> 
> There would still be only one input "pipeline" at any time, namely
> that of the selected frame.  Just that switching frames now would call
> for "saving" the input pipeline of the frame switched from and
> "restoring" any input pipeline of the frame switched to.

Tricky at best, IMO.  Frame switch is just another input event.  I
guess you are thinking about adding calls to thread-yield into input
processing code?

Also, does it mean that every call to make-thread will now pop up a
new frame?

> IIUC we are able to save and restore an input pipeline when we enter
> and leave a recursive minibuffer so the mechanism for doing such
> switches seems to exist for minibuffer reading.

What will appear in the minibuffer window of the previously-selected
frame while we interact with another thread's frame?

> And reading a key sequence through the echo area is IIUC synchronous
> anyway so a user cannot practically switch frames during that.

Are you sure they cannot switch frames in the middle of reading from
the minibuffer?

Re: User interaction from multiple threads

[martin rudalics]

 > Tricky at best, IMO.

Porting prompts from one frame's minibuffer window to another is just
as tricky IMO.

 > Frame switch is just another input event.  I
 > guess you are thinking about adding calls to thread-yield into input
 > processing code?

When switching to a frame I would resume any pending input for the
frame switched to.  This has, a priori, nothing to do with threads.
Input would be processed by the main thread as now.  If a thread was
blocked because it was waiting for input to complete, the main thread
would send a 'thread-signal' to the blocked thread to unblock it as
soon as input is complete.

 > Also, does it mean that every call to make-thread will now pop up a
 > new frame?

When the thread requests it.  The idea is that a user should not be
pushed to react immediately to the prompt or information provided by
the thread and that none of these get lost in the noise of other
prompts and messages.  Whether such frames are created when the thread
is started or lazily when, for example, the question whether copying a
file should overwrite an existing one with the same name arises, is
left to the thread's writer.  If a thread does not ask for its own
frame, the main thread's mini-window will be used, possibly disrupting
the user's workflow.

 > What will appear in the minibuffer window of the previously-selected
 > frame while we interact with another thread's frame?

If the previously-selected frame was that of the main thread, whatever
the main thread currently does.  Though the main thread's mini-window
would never show contents aimed for the mini-window of a separate
thread provided that thread owns a frame.

If the previously-selected frame was owned by a separate thread, it
would show information pertaining to that thread and only to that
thread.

 >> And reading a key sequence through the echo area is IIUC synchronous
 >> anyway so a user cannot practically switch frames during that.
 >
 > Are you sure they cannot switch frames in the middle of reading from
 > the minibuffer?

You mean when SWITCH-FRAME-OK is set in 'read-key-sequence'?  Maybe
that would require special processing - I haven't looked into it.

martin

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Sat, 18 Aug 2018 10:31:06 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: michael.albinus@gmx.de, emacs-devel@gnu.org
> 
>  > Tricky at best, IMO.
> 
> Porting prompts from one frame's minibuffer window to another is just
> as tricky IMO.

Except that it doesn't involve threads.

>  > Frame switch is just another input event.  I
>  > guess you are thinking about adding calls to thread-yield into input
>  > processing code?
> 
> When switching to a frame I would resume any pending input for the
> frame switched to.  This has, a priori, nothing to do with threads.

Unless I'm misunderstanding you, this has everything to do with
threads, because the input from the newly-selected frame should be
processed by a different thread than the one which was "current"
before the switch.

> Input would be processed by the main thread as now.  If a thread was
> blocked because it was waiting for input to complete, the main thread
> would send a 'thread-signal' to the blocked thread to unblock it as
> soon as input is complete.

thread-signal signals an error (like 'signal' does), so the signaled
thread will exit as soon as it resumes running, which is not what you
want.  That's why I mentioned thread-yield.

>  > Also, does it mean that every call to make-thread will now pop up a
>  > new frame?
> 
> When the thread requests it.  The idea is that a user should not be
> pushed to react immediately to the prompt or information provided by
> the thread and that none of these get lost in the noise of other
> prompts and messages.  Whether such frames are created when the thread
> is started or lazily when, for example, the question whether copying a
> file should overwrite an existing one with the same name arises, is
> left to the thread's writer.

What does "left to the thread's writer" mean in this context?  Given
the low-level details involved, which cannot be done from Lisp, how
can we leave this to the application?  I'm probably missing something.

>  > What will appear in the minibuffer window of the previously-selected
>  > frame while we interact with another thread's frame?
> 
> If the previously-selected frame was that of the main thread, whatever
> the main thread currently does.

I don't think this can fly with how redisplay works, can it?

Re: User interaction from multiple threads

[martin rudalics]

 > Unless I'm misunderstanding you, this has everything to do with
 > threads, because the input from the newly-selected frame should be
 > processed by a different thread than the one which was "current"
 > before the switch.

By design, reading input separately for a specific frame would not be
necessarily tied to a thread.  Once complete, the input could be
stored somwhere and later be processed by the main thread.  Only if a
frame is actually owned by a thread, its input would eventually be
passed to that thread.

 > thread-signal signals an error (like 'signal' does), so the signaled
 > thread will exit as soon as it resumes running, which is not what you
 > want.  That's why I mentioned thread-yield.

Then it's probably 'thread-yield' as you mentioned.  I thought
'thread-signal' would allow to resume a specific blocked thread.
I see now that I was wrong.

 > What does "left to the thread's writer" mean in this context?  Given
 > the low-level details involved, which cannot be done from Lisp, how
 > can we leave this to the application?  I'm probably missing something.

When someone like Michael fires off 250 threads at the same time, we
probably need some way to turn off eager frame creation or turn off
frame creation at all by setting some buffer display option.  By
default I would create frames lazily, with frame creation turned off
and eager frame creation as thread specific options.  Once a setting
has been made for a specific thread, the low-level routines have to
act as prescribed by that setting.

 >>   > What will appear in the minibuffer window of the previously-selected
 >>   > frame while we interact with another thread's frame?
 >>
 >> If the previously-selected frame was that of the main thread, whatever
 >> the main thread currently does.
 >
 > I don't think this can fly with how redisplay works, can it?

If the main thread's frame is not selected, we do not care much, do
we?  If it's selected, we do whatever we already do now.  Why do you
think redisplay enters here?  What am I missing?

martin

Re: User interaction from multiple threads

[Gemini Lasswell]

Hello,

With the benefit of having read everyone else's suggestions to date,
here are my answers to your use cases:

Eli Zaretskii <eliz@gnu.org> writes:

> Use case #1:
>
>   . The main thread is waiting for user input.  The user didn't yet
>     type anything
>   . A non-main thread runs Lisp that prompts the user for some input
>
>  In this case, we probably want the following input to go to the
>  prompting thread, right?  But it might also be the case that the
>  user actually wants the input to go to the main thread, e.g. to
>  perform some unrelated command.  Should we allow that?  If yes, how
>  should Emacs know which thread should receive what the user types?

If a user sees the cursor in a text buffer and types a character and
RET, that should never be interpreted as the response to a prompt that
appeared just as she started typing and which she didn't notice in time
to stop.  This means that a prompt from a non-main thread should become
visible but not active, and the user should have to switch to the window
containing it to respond.

The user should, as now, be able to type C-x C-o or C-x 5 o during
read-from-minibuffer, and perform unrelated commands on another buffer.
The user shouldn't have to have a mental model of what threads are in
order to work with Emacs, and the simpler we can keep the Lisp
programmer's mental model of threads, the better off we will be.  So
those unrelated commands should be run in the main thread.

Perhaps the way to accomplish this is to always read keyboard input in
the main thread.  So when read-from-minibuffer, for example, is called
from a non-main thread, it would put the prompt on a queue for the main
thread to process, and then block until the response is ready.

> Use case #2:
>
>   Same as the previous, but now the user is in the middle of typing a
>   key sequence, when the non-main thread prompts.  For example,
>   suppose the user has typed "C-x".>

>   What do we want to happen now?  Do we "preempt" the main thread and
>   let the following input to go to the prompting thread?  Or do we let
>   the prompting thread wait until the main thread reads a full key
>   sequence and runs the command bound to it?  If the former, what to
>   do with the partial key sequence ("C-x") that the user typed?  If
>   the latter, how do we indicate to the user that there is a prompt
>   from another thread?

I don't think we should ever interrupt a key sequence.  The prompting
thread will have to wait.  To indicate to the user that there is a
waiting prompt, use a numeric indicator at the beginning of the echo
area or upper left corner of the minibuffer, like the recursive
minibuffer indicator.  In a graphical environment a white number on a
red ellipse would be a familiar clue to many users that there are
messages waiting.

> Use case #3:
>
>   Similar, but now 2 or more non-main threads prompt the user, one
>   after the other, in quick succession.  What should happen now, and
>   how will the user know there are multiple prompts?

The first thread should get to show its prompt and wait for a response,
and then show followup prompts if it can present them immediately after
the first is answered.  Other threads will block until the threads ahead
of them are done prompting.  The same numeric indicator as above could
indicate to the user how many other prompts are waiting.  A new
keybinding could allow navigation forward and backward through the
waiting prompts.  (This is inspired by Martin's minibuffer list
suggestion in
http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00516.html)
Another new keybinding might send keyboard-quit to all waiting prompts,
which I would like to have available in the event that I do something
that unexpectedly creates 250 of them.

Re: User interaction from multiple threads

[Phil Sainty]

On 2018-08-20 11:08, Gemini Lasswell wrote:
> Perhaps the way to accomplish this is to always read keyboard input in
> the main thread.  So when read-from-minibuffer, for example, is called
> from a non-main thread, it would put the prompt on a queue for the main
> thread to process, and then block until the response is ready.

>> Use case #3:
>> 
>>   Similar, but now 2 or more non-main threads prompt the user, one
>>   after the other, in quick succession.  What should happen now, and
>>   how will the user know there are multiple prompts?
> 
> The first thread should get to show its prompt and wait for a response,
> and then show followup prompts if it can present them immediately after
> the first is answered.  Other threads will block until the threads 
> ahead
> of them are done prompting.


I'm pondering the details of that "and then show followup prompts if it
can present them immediately" part...

Say a thread has pushed its first prompt onto a queue, and then blocked.
At this point there is no knowledge of whether or not that thread may
have follow-up prompts to issue once it is unblocked.

In the meantime, additional threads may also push prompts onto the 
queue.

Once it's safe for the user to be prompted, the first thread's prompt is
processed in the main thread, and the result is established and passed
back to the calling thread, which then resumes processing.

So long as that thread doesn't voluntarily yield, it would be able to
generate a follow-up prompt before any of the other pending prompts from
other threads had a chance to be processed.  However, it's also going to
need to jump the queue.

If the main thread is managing the queue of pending prompts, is it
sufficient for the main thread to track the single most-recent thread
with which the user interacted, and to always prioritise queue items
from that thread over any other queue items?


-Phil

Re: User interaction from multiple threads

[Gemini Lasswell]

Phil Sainty <psainty@orcon.net.nz> writes:

> I'm pondering the details of that "and then show followup prompts if it
> can present them immediately" part...
>
> Say a thread has pushed its first prompt onto a queue, and then blocked.
> At this point there is no knowledge of whether or not that thread may
> have follow-up prompts to issue once it is unblocked.
>
> In the meantime, additional threads may also push prompts onto the
> queue.
>
> Once it's safe for the user to be prompted, the first thread's prompt is
> processed in the main thread, and the result is established and passed
> back to the calling thread, which then resumes processing.
>
> So long as that thread doesn't voluntarily yield, it would be able to
> generate a follow-up prompt before any of the other pending prompts from
> other threads had a chance to be processed.  However, it's also going to
> need to jump the queue.
>
> If the main thread is managing the queue of pending prompts, is it
> sufficient for the main thread to track the single most-recent thread
> with which the user interacted, and to always prioritise queue items
> from that thread over any other queue items?

That's what I was thinking, but you bring up a good question to which I
don't know the answer.  After the main thread processes the first
thread's prompt and stores the user's response, it needs to yield so the
first thread can resume execution.  Is it guaranteed that the thread
scheduler will allow all other ready threads a turn to execute during
the main thread's thread-yield?

If not, maybe a good-enough-most-of-the-time solution would be for the
main thread to do a sleep-for for an amount of time that won't bother
the user, like 0.1s, and if the most-recent thread hasn't added a prompt
to the queue by then, then go on to the next thread's prompt.

Re: User interaction from multiple threads

[Gemini Lasswell]

Eli Zaretskii <eliz@gnu.org> writes:

> As you well know, when the user switches to another frame, we
> redisplay the mini-window on the newly selected frame to display the
> same prompt/echo there as was displayed on the previous selected
> frame.

This isn't the way it works when I try it.  With 26.1.50, if I do:

M-x make-frame RET
C-x C-f C-x 5 o

Then the cursor moves to the new frame but the find-file prompt stays in
the original frame.  Is this a bug?

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Gemini Lasswell <gazally@runbox.com>
> Cc: martin rudalics <rudalics@gmx.at>,  emacs-devel@gnu.org
> Date: Mon, 20 Aug 2018 07:36:20 -0700
> 
> M-x make-frame RET
> C-x C-f C-x 5 o
> 
> Then the cursor moves to the new frame but the find-file prompt stays in
> the original frame.  Is this a bug?

No, but try the same with (message "foo") instead.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Gemini Lasswell <gazally@runbox.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 19 Aug 2018 16:08:38 -0700
> 
> Perhaps the way to accomplish this is to always read keyboard input in
> the main thread.  So when read-from-minibuffer, for example, is called
> from a non-main thread, it would put the prompt on a queue for the main
> thread to process, and then block until the response is ready.

I have difficulty understanding how this "queue" model will work.
Suppose the main thread has some way of figuring out when to show the
first prompt on that "queue", then what? how (with what API) does it
show the prompt, and how does it read the response?  Remember: the
prompting thread called some function, like read-file-name, and that
function has its own ideas for how to interact with the user -- how
will the prompting thread tell the main thread all those details?
Some of the details reference variables that are local to the
prompting thread's current-buffer, about which the main thread knows
nothing.  The main thread will need to thoroughly "emulate" the
prompting thread to DTRT here.

Then there's the issue of passing the result to the prompting thread.
Unless you want to ask the programmer of that thread's function to
write specialized code for user interaction, the prompting thread
expects to get the keyboard input through the normal input machinery,
like read_key_sequence.  How will that be possible with the "queue"
model?

> If a user sees the cursor in a text buffer and types a character and
> RET, that should never be interpreted as the response to a prompt that
> appeared just as she started typing and which she didn't notice in time
> to stop.

What about input functions like read-char, which read a character
without placing the cursor in the minibuffer?  How will we ensure your
principle above is honored?

> The user should, as now, be able to type C-x C-o or C-x 5 o during
> read-from-minibuffer, and perform unrelated commands on another buffer.

This means some of the user input should go to the prompting thread,
while some other part ("C-x C-o" in your example) should go to the
main thread, so that it switches to another frame.  How will Emacs
know which part of user input is intended to what thread?

Thanks.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Gemini Lasswell <gazally@runbox.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Emacs-devel <emacs-devel-bounces+psainty=orcon.net.nz@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 20 Aug 2018 07:15:05 -0700
> 
> Is it guaranteed that the thread scheduler will allow all other
> ready threads a turn to execute during the main thread's
> thread-yield?

No.  The "scheduler" we have is actually a race to grab the global
lock; the first thread to succeed starts running, the others wait.

> If not, maybe a good-enough-most-of-the-time solution would be for the
> main thread to do a sleep-for for an amount of time that won't bother
> the user, like 0.1s, and if the most-recent thread hasn't added a prompt
> to the queue by then, then go on to the next thread's prompt.

When the main thread calls sleep-for, it enters the "scheduler", so
some other thread will start running, and there's no guarantee that
the main thread will resume after 0.1 sec: it might be much later, if
that other thread has something to do that takes a long time.

Re: User interaction from multiple threads

[Phil Sainty]

On 2018-08-21 05:03, Eli Zaretskii wrote:
> Remember: the prompting thread called some function, like
> read-file-name, and that function has its own ideas for how to
> interact with the user -- how will the prompting thread tell the
> main thread all those details?  Some of the details reference
> variables that are local to the prompting thread's current-buffer,
> about which the main thread knows nothing.  The main thread will
> need to thoroughly "emulate" the prompting thread to DTRT here.

I was wondering about this too, and concluded that the main thread
probably can't read the input (for the reasons you've outlined), but
that the more general approach of the main thread *coordinating* the
reading of input based on information from the other threads might
still be viable and useful.

Instead of the prompting thread pushing its prompt into a queue for
the main thread to somehow execute, the prompting thread could instead
simply push onto a queue a *notification* that it needs to perform
some user interaction.

At that point, as in the prior suggestion, it blocks until the main
thread decides that the user-interaction is safe.  But instead of the
main thread reading the input itself and then passing control (and the
result) back, it simply passes control back to the prompting thread --
which will read the input for itself and continue processing.

I think that's the same outcome as the prior suggestion, but without
the magic of one thread knowing all about another thread.

The queue is then essentially the list of threads which are awaiting
the opportunity to perform user-interaction.  When a non-main thread
attempts to read input from the user, it is automatically blocked and
pushed onto the queue.  Any thread which is blocked on this account is
only allowed to resume when Emacs decides that it is safe for user
interaction for a non-main thread to occur.  And my thought about
queue-jumping procedures would still work the same as before, so that
a single thread could read a series of user inputs without other
threads interjecting with their own reads.


-Phil

p.s. I'm afraid I've not been following the whole discussion, so
apologies if I'm just covering old ground here, or ignoring other
critical elements.

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Tue, 21 Aug 2018 10:38:59 +1200
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: Gemini Lasswell <gazally@runbox.com>, emacs-devel@gnu.org, Emacs-devel
>  <emacs-devel-bounces+psainty=orcon.net.nz@gnu.org>
> 
> Instead of the prompting thread pushing its prompt into a queue for
> the main thread to somehow execute, the prompting thread could instead
> simply push onto a queue a *notification* that it needs to perform
> some user interaction.

This must happen before the prompt is displayed, so it means we will
need to identify all the functions used for prompting (or channel them
all into a single low-level function), and do this enqueuing there.

> The queue is then essentially the list of threads which are awaiting
> the opportunity to perform user-interaction.  When a non-main thread
> attempts to read input from the user, it is automatically blocked and
> pushed onto the queue.

Not "read input", but show a prompt, right?  We don't want to show a
prompt to which the user cannot respond, right?

> Any thread which is blocked on this account is only allowed to
> resume when Emacs decides that it is safe for user interaction for a
> non-main thread to occur.

Who is "Emacs" in this sentence? the main thread?  For that to
happen, the main thread needs to resume, and for that, some other
thread should awaken it.

So we need some additional synchronization facilities for this scheme
to work, and we also need a way to stop the normal "scheduling", so
that we could make sure only the thread that is currently interacting
with the user can run, even if it calls an API that normally will
allow rescheduling.

Re: User interaction from multiple threads

[Gemini Lasswell]

Eli Zaretskii <eliz@gnu.org> writes:

> Then there's the issue of passing the result to the prompting thread.
> Unless you want to ask the programmer of that thread's function to
> write specialized code for user interaction, the prompting thread
> expects to get the keyboard input through the normal input machinery,
> like read_key_sequence.  How will that be possible with the "queue"
> model?

Thanks, I understand the difficulties better now.

>> The user should, as now, be able to type C-x C-o or C-x 5 o during
>> read-from-minibuffer, and perform unrelated commands on another buffer.
>
> This means some of the user input should go to the prompting thread,
> while some other part ("C-x C-o" in your example) should go to the
> main thread, so that it switches to another frame.  How will Emacs
> know which part of user input is intended to what thread?

I don't know, but I frequently leave minibuffer prompts, usually with my
keybinding for avy-find-char-timer, but sometimes using C-x o or C-x 5 o,
and return to them later.  I also have enable-recursive-minibuffers set.

So from my point of view as an Emacs user, I want to be able to continue
to do this, whether the minibuffer prompt I'm looking at came from the
main thread or some other thread.

Re: User interaction from multiple threads

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Is it necessary to support user interaction from multiple threads
on any given terminal?  If not necessary, is it worth the trouble?

If it is ok not to support that, maybe we should take a different
approach, such as assigning any given terminal to one thread.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: User interaction from multiple threads

[martin rudalics]

 > This must happen before the prompt is displayed, so it means we will
 > need to identify all the functions used for prompting (or channel them
 > all into a single low-level function), and do this enqueuing there.

Earlier you said:

 >> The simplest idea is to have every primitive that displays in the echo
 >> area to take a certain mutex before it does so.

Doesn't that boil down to something very similar, namely that "we will
need to identify all the functions that display in the echo area"?

martin

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: psainty@orcon.net.nz, gazally@runbox.com,
> 	emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,
> 	emacs-devel@gnu.org
> Date: Wed, 22 Aug 2018 00:05:58 -0400
> 
> Is it necessary to support user interaction from multiple threads
> on any given terminal?  If not necessary, is it worth the trouble?

IMO, it is worth the trouble.  If we don't provide a way of coping
with that, the concurrency feature will be much less useful than it
could have been, because only a very limited class of Lisp programs
will be able to run in non-main threads.

The case in point is Michael's work on the tramp-thread-safe branch:
there' visiting files and other file-related commands run in another
thread (useful for commands that take a lot of time to complete, such
as accessing remote files via Tramp).  The code does it by running
normal file primitives in separate threads.  However, some primitives
and subroutines need to ask the user something, such as whether to
apply file-local variables that might be unsafe, and need the user to
respond.  If we decide that user interaction from other threads is
"verboten", it means all such mundane operations, of which we have
gazillions in Emacs, suddenly become impossible to use in a program
that could run in another thread, and then what do we expect Lisp
programmers to do about that? invent a whole new set of parallel
functions that somehow avoid asking such questions or prompting the
user when they need to?

IOW, limiting user interaction to a single thread will in practice
allow running in other threads only programs that never interact with
the user, such as some batch-style mathematical calculation or text
processing.  Most Emacs Lisp programs are not like that.

The whole point of introducing threads was to allow running code
asynchronously with minimum fuss.  If it turns out that the amount of
fuss is much larger than that, then we gained nothing, because we can
already run "asynchronous" code from timers and process filters, given
enough effort to make a program abide by the rules necessary to work
in such situations.

> If it is ok not to support that, maybe we should take a different
> approach, such as assigning any given terminal to one thread.

Unless I misunderstand what you mean by "given terminal", doing that
won't solve the hard part of the problem: how to multiplex input
events between different threads which require user attention
simultaneously.  That's because, AFAIR, even with multiple displays,
we still serialize input: once I start typing a key sequence on one
display, the others won't accept any input until I finish.

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Wed, 22 Aug 2018 09:18:48 +0200
> From: martin rudalics <rudalics@gmx.at>
> Cc: gazally@runbox.com, emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,
> 	emacs-devel@gnu.org
> 
>  > This must happen before the prompt is displayed, so it means we will
>  > need to identify all the functions used for prompting (or channel them
>  > all into a single low-level function), and do this enqueuing there.
> 
> Earlier you said:
> 
>  >> The simplest idea is to have every primitive that displays in the echo
>  >> area to take a certain mutex before it does so.
> 
> Doesn't that boil down to something very similar, namely that "we will
> need to identify all the functions that display in the echo area"?

Yes, it does.  But the important part of the "This must happen" text
is the "before the prompt" part.  It was written in response to a
suggestion to have the prompting thread block when it attempts to read
the response to the prompt, which is _after_ the prompt was already
shown.

Re: User interaction from multiple threads

[hw]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> Cc: psainty@orcon.net.nz, gazally@runbox.com,
>> 	emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,
>> 	emacs-devel@gnu.org
>> Date: Wed, 22 Aug 2018 00:05:58 -0400
>> 
>> Is it necessary to support user interaction from multiple threads
>> on any given terminal?  If not necessary, is it worth the trouble?
>
> IMO, it is worth the trouble.  If we don't provide a way of coping
> with that, the concurrency feature will be much less useful than it
> could have been, because only a very limited class of Lisp programs
> will be able to run in non-main threads.
>
> The case in point is Michael's work on the tramp-thread-safe branch:
> there' visiting files and other file-related commands run in another
> thread (useful for commands that take a lot of time to complete, such
> as accessing remote files via Tramp).  The code does it by running
> normal file primitives in separate threads.  However, some primitives
> and subroutines need to ask the user something, such as whether to
> apply file-local variables that might be unsafe, and need the user to
> respond.  If we decide that user interaction from other threads is
> "verboten", it means all such mundane operations, of which we have
> gazillions in Emacs, suddenly become impossible to use in a program
> that could run in another thread, and then what do we expect Lisp
> programmers to do about that? invent a whole new set of parallel
> functions that somehow avoid asking such questions or prompting the
> user when they need to?
>
> IOW, limiting user interaction to a single thread will in practice
> allow running in other threads only programs that never interact with
> the user, such as some batch-style mathematical calculation or text
> processing.  Most Emacs Lisp programs are not like that.
>
> The whole point of introducing threads was to allow running code
> asynchronously with minimum fuss.  If it turns out that the amount of
> fuss is much larger than that, then we gained nothing, because we can
> already run "asynchronous" code from timers and process filters, given
> enough effort to make a program abide by the rules necessary to work
> in such situations.
>
>> If it is ok not to support that, maybe we should take a different
>> approach, such as assigning any given terminal to one thread.
>
> Unless I misunderstand what you mean by "given terminal", doing that
> won't solve the hard part of the problem: how to multiplex input
> events between different threads which require user attention
> simultaneously.  That's because, AFAIR, even with multiple displays,
> we still serialize input: once I start typing a key sequence on one
> display, the others won't accept any input until I finish.

Assuming that

+ for a number of reasons, users can not deal with multiple
  requests for input (RFIs) at the same time anyway,

+ that inevitably Emacs will create multiple RFIs at the same time when
  using multiple threads

+ and that it can be agreed upon that users must not be interrupted by
  RFIs and should not be confused by them

what other choice is there than queueing up the RFIs and leaving it to
the users to work their way through the queue as they see fit?

The only RFIs users probably want to deal with immediately are the ones
directly related to what they are currently doing, like being prompted
for a file name when they want to visit one.

All others need to be queued.  With time, further options may come up,
like classifying RFIs and options to decide which instances the users
want to be brought to their attention right away, and multiple RFI
queues for different classes and/or with different priorities to them
assigned by the users simply through the order in which they choose to
go through them.


For example, when desktop-mode loads a session which contains buffers
that make Emacs prompt for unsafe variables, the RFI should be queued,
multithreading or not, so Emacs can proceed visiting files.  During the
current session, I might never touch the buffer which is asking about
unsafe variables, so why should I have to deal with the RFI at all?
Only if I want to do something with this buffer, I should be presented
with it.

From a useability point of view, it doesn´t make any sense that Emacs
grinds to a halt during restoring a session when restoring it creates an
RFI and tries to force me to deal with it.  That´s not what I want, so
in practise, I have to make sure not to have files restored that cause
RFIs.

Re: User interaction from multiple threads

[Tom Tromey]

>>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:

Richard> If it is ok not to support that, maybe we should take a different
Richard> approach, such as assigning any given terminal to one thread.

This was what I'd planned to do in the past but never got around to.  A
first step would be to make a new thread for each terminal and arrange
for the terminal's file descriptor to be thread-locked.  This didn't
seem very hard.  Once that's done I guess higher-level problems could be
looked into; I don't really know this area well though.

Tom

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: hw <hw@adminart.net>
> Cc: rms@gnu.org,  psainty@orcon.net.nz,  gazally@runbox.com,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
> Date: Wed, 22 Aug 2018 17:48:18 +0200
> 
> what other choice is there than queueing up the RFIs and leaving it to
> the users to work their way through the queue as they see fit?

We all kinda agree about that, the question is how to do that in
practice with as little disruption of existing code as possible.

> >From a useability point of view, it doesn´t make any sense that Emacs
> grinds to a halt during restoring a session when restoring it creates an
> RFI and tries to force me to deal with it.

But that's exactly what happens today with a single thread.  So it's
an unrelated issue.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Tom Tromey <tom@tromey.com>
> Date: Wed, 22 Aug 2018 10:24:49 -0600
> Cc: psainty@orcon.net.nz, gazally@runbox.com, Eli Zaretskii <eliz@gnu.org>,
> 	emacs-devel-bounces+psainty=orcon.net.nz@gnu.org, emacs-devel@gnu.org
> 
> >>>>> "Richard" == Richard Stallman <rms@gnu.org> writes:
> 
> Richard> If it is ok not to support that, maybe we should take a different
> Richard> approach, such as assigning any given terminal to one thread.
> 
> This was what I'd planned to do in the past but never got around to.  A
> first step would be to make a new thread for each terminal and arrange
> for the terminal's file descriptor to be thread-locked.  This didn't
> seem very hard.  Once that's done I guess higher-level problems could be
> looked into; I don't really know this area well though.

Please describe the proposed solution in more detail, because I'm not
sure I understand it.  By "new thread" do you mean a Lisp thread of
the kind made by make-thread, which will run only after it succeeds in
grabbing the global lock?  Or do you mean some other kind of thread?

Thanks.

Re: User interaction from multiple threads

[hw]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hw <hw@adminart.net>
>> Cc: rms@gnu.org,  psainty@orcon.net.nz,  gazally@runbox.com,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
>> Date: Wed, 22 Aug 2018 17:48:18 +0200
>> 
>> what other choice is there than queueing up the RFIs and leaving it to
>> the users to work their way through the queue as they see fit?
>
> We all kinda agree about that, the question is how to do that in
> practice with as little disruption of existing code as possible.

ok

>> >From a useability point of view, it doesn´t make any sense that Emacs
>> grinds to a halt during restoring a session when restoring it creates an
>> RFI and tries to force me to deal with it.
>
> But that's exactly what happens today with a single thread.  So it's
> an unrelated issue.

It means that the issue is already there, regardless.

Re: User interaction from multiple threads

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

With multiple threads asynchonously requesting input, I have a feeling
that users will get confused in complicated situations as a result of
not knowing which thread the input is for.

Perhaps the minibuffer should indicate which thread is making the request
when that isn't the current thread.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: User interaction from multiple threads

[John Wiegley]

>>>>> "RS" == Richard Stallman <rms@gnu.org> writes:

RS> Perhaps the minibuffer should indicate which thread is making the request
RS> when that isn't the current thread.

Or, when there is more than 1 pending input, switch to using a buffer
containing multiple prompts, since it's quite likely the user won't want to
answer them in the same order as we would present them serially in the
minibuffer.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, psainty@orcon.net.nz, gazally@runbox.com,
> 	emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,
> 	emacs-devel@gnu.org
> Date: Wed, 22 Aug 2018 19:58:55 -0400
> 
> With multiple threads asynchonously requesting input, I have a feeling
> that users will get confused in complicated situations as a result of
> not knowing which thread the input is for.

Yes, and I think we all agreed that some sort of serialization is
needed.  The question is how to do that without a complete rewrite of
the low-level input code.

Tom seems to have a different idea, but I don't think I understand it
well enough yet.

> Perhaps the minibuffer should indicate which thread is making the request
> when that isn't the current thread.

Why is that important?  Wouldn't the prompt itself be informative
enough?  And how (in what terms) would you suggest to indicate the
thread ID in this case?

Re: User interaction from multiple threads

[Phil Sainty]

On 2018-08-24 02:08, Eli Zaretskii wrote:
>> From: Richard Stallman <rms@gnu.org>
>> Perhaps the minibuffer should indicate which thread is making the
>> request when that isn't the current thread.
> 
> Why is that important?

Because otherwise it won't necessarily be obvious to the user that a
succession of prompts are unrelated?  They may or may not be able to
figure it out relatively easily, but that depends somewhat on how the
specific prompts in question happen to have been written.

Some kind of identifier for the source of the message seems very
sensible to me.  If nothing else, it immediately tells the user that
the question they are being asked *may* not be related to the thing
they were just doing, which I think is a useful piece of information
to lead with, as it prevents the confusion before it can occur.


> Wouldn't the prompt itself be informative enough?

Undoubtedly that would often be true, but are all prompts written in
such a specific fashion to guarantee they would be unambiguous within
any arbitrary arrangement of asynchronous prompting activity?  I would
be very surprised if that turned out to be true.  At minimum, few if
any of the existing prompts (many of which may now be triggered in
thread contexts), will have been written with this particular scenario
in mind.


> And how (in what terms) would you suggest to indicate the thread ID
> in this case?

`make-thread' takes an optional NAME argument, so perhaps that can be
used when it's set?  Otherwise some kind of internal ID would probably
have to suffice?  If NAME is not generally suitable, then perhaps we
need something additional as a human-readable identifier?


-Phil

Re: User interaction from multiple threads

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Or, when there is more than 1 pending input, switch to using a buffer
  > containing multiple prompts, since it's quite likely the user won't want to
  > answer them in the same order as we would present them serially in the
  > minibuffer.

That could be good.  But we could also make a list to indicate what
the current requests are, so users' Lisp code could take charge of how
users can provide input to them.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: User interaction from multiple threads

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > With multiple threads asynchonously requesting input, I have a feeling
  > > that users will get confused in complicated situations as a result of
  > > not knowing which thread the input is for.

  > Yes, and I think we all agreed that some sort of serialization is
  > needed.

We agree about the serialization, but the issue I am not talking about
now is different.  I am talking about managing the various input
requests -- choosing the order of the requests, and identifying which
request is being made now and on behalf of which activity.

  > > Perhaps the minibuffer should indicate which thread is making the request
  > > when that isn't the current thread.

  > Why is that important?  Wouldn't the prompt itself be informative
  > enough?

We can hardly assume that the prompt shows which activity is asking
for the input.  What if two different threads ask for a Password?
The prompt might say, "Password: ".  Which password should it be?

  >   And how (in what terms) would you suggest to indicate the
  > thread ID in this case?

I don't know enough about the thread feature to figure that out.
Perhaps based on the thread's top-level function call?
Do threads have names?


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: User interaction from multiple threads

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1522 bytes --]

On Fri, 24 Aug 2018 at 10:02, Richard Stallman <rms@gnu.org> wrote:

  > > With multiple threads asynchonously requesting input, I have a feeling
>   > > that users will get confused in complicated situations as a result of
>   > > not knowing which thread the input is for.
>
>   > Yes, and I think we all agreed that some sort of serialization is
>   > needed.
>
> We agree about the serialization, but the issue I am not talking about
> now is different.  I am talking about managing the various input
> requests -- choosing the order of the requests, and identifying which
> request is being made now and on behalf of which activity.
>

I think this is a very important point. We've seen with the development of
numerous different Emacs libraries that people will come up with vastly
improved variations of standard Emacs functionality if they are given the
necessary tools to make it happen.

Providing a comprehensive API to for controlling the queued interaction
requests is probably more important than desiging the perfect UI for it.


>   >   And how (in what terms) would you suggest to indicate the
>   > thread ID in this case?
>
> I don't know enough about the thread feature to figure that out.
> Perhaps based on the thread's top-level function call?
> Do threads have names?
>

Threads do have names, but providing one is optional. The function
signature is (FUNCTION &optional NAME).

As for givng the toplevel function name, I'd expect most toplevel functions
are actually LAMBDA forms.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 2124 bytes --]

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Fri, 24 Aug 2018 10:34:07 +1200
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: rms@gnu.org, hw@adminart.net, emacs-devel@gnu.org,
>  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org, gazally@runbox.com
> 
> On 2018-08-24 02:08, Eli Zaretskii wrote:
> >> From: Richard Stallman <rms@gnu.org>
> >> Perhaps the minibuffer should indicate which thread is making the
> >> request when that isn't the current thread.
> > 
> > Why is that important?
> 
> Because otherwise it won't necessarily be obvious to the user that a
> succession of prompts are unrelated?  They may or may not be able to
> figure it out relatively easily, but that depends somewhat on how the
> specific prompts in question happen to have been written.

Indeed, so maybe those prompts which don't explain themselves (I
expect them to be a small minority) should be improved.

> Some kind of identifier for the source of the message seems very
> sensible to me.

The identifier must be very informative and specific, and I expect
that to be difficult, given the small screen estate we devote to
prompts (of which the ID will have to be a small part).

> > Wouldn't the prompt itself be informative enough?
> 
> Undoubtedly that would often be true, but are all prompts written in
> such a specific fashion to guarantee they would be unambiguous within
> any arbitrary arrangement of asynchronous prompting activity?  I would
> be very surprised if that turned out to be true.  At minimum, few if
> any of the existing prompts (many of which may now be triggered in
> thread contexts), will have been written with this particular scenario
> in mind.

We seem to have different expectations about the current state of
affairs in this aspect, so maybe a small survey should help us make
the proverbial reality check?

> > And how (in what terms) would you suggest to indicate the thread ID
> > in this case?
> 
> `make-thread' takes an optional NAME argument, so perhaps that can be
> used when it's set?  Otherwise some kind of internal ID would probably
> have to suffice?  If NAME is not generally suitable, then perhaps we
> need something additional as a human-readable identifier?

I don't expect anyone to remember which name corresponds to what job
that is running in the background, when there are enough of them.
That name will have to come from some Lisp written by the same
programmers whom you don't trust to provide self-explanatory prompts,
so we are back to the same problem (except that a thread name will
normally be shorter, so less informative and less specific).

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: psainty@orcon.net.nz, hw@adminart.net, emacs-devel@gnu.org,
> 	emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,
> 	gazally@runbox.com
> Date: Thu, 23 Aug 2018 22:00:03 -0400
> 
>   > > With multiple threads asynchonously requesting input, I have a feeling
>   > > that users will get confused in complicated situations as a result of
>   > > not knowing which thread the input is for.
> 
>   > Yes, and I think we all agreed that some sort of serialization is
>   > needed.
> 
> We agree about the serialization, but the issue I am not talking about
> now is different.  I am talking about managing the various input
> requests -- choosing the order of the requests, and identifying which
> request is being made now and on behalf of which activity.

The idea expressed by several people is that once you start
interacting with some thread's prompt, the other threads are locked
out of interaction, until the interacting thread is done with the
series of prompts that allow it to go on with its business.

>   > Why is that important?  Wouldn't the prompt itself be informative
>   > enough?
> 
> We can hardly assume that the prompt shows which activity is asking
> for the input.  What if two different threads ask for a Password?
> The prompt might say, "Password: ".  Which password should it be?

If that's the first prompt in a series of prompts, we should fix the
prompt to tell what password does it ask for.  But it is okay, I
think, to have a series of prompts from a single thread which go like
this:

  Host to logon:
  Password:
  Initial directory:

As long as the user knows this is a coherent series of prompts from
the same thread, she will have no difficulty filling in the blanks,
exactly like she does that now.

>   >   And how (in what terms) would you suggest to indicate the
>   > thread ID in this case?
> 
> I don't know enough about the thread feature to figure that out.
> Perhaps based on the thread's top-level function call?
> Do threads have names?

They do, but the name is optional.  If the Lisp program that starts a
thread doesn't provide a name, Emacs will identify it as something
like #<thread 0x01a2d468>, which is not helpful in the situations we
are discussing.

And even if we do require each thread to have a name, what guarantees
do we have that the name will be more informative than the prompts
about which you were worried?

Re: User interaction from multiple threads

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> The idea expressed by several people is that once you start
> interacting with some thread's prompt, the other threads are locked
> out of interaction, until the interacting thread is done with the
> series of prompts that allow it to go on with its business.

Yes. The simplest locking mechanism I could imagine is a mutex locking
such interactions. This must be used for every
echo-a-prompt-and-read-the-answer interaction by default (we shall
determine all low-level cases). And it shall be a predefined, documented
mutex, like `interaction-mutex'.

> If that's the first prompt in a series of prompts, we should fix the
> prompt to tell what password does it ask for.  But it is okay, I
> think, to have a series of prompts from a single thread which go like
> this:
>
>   Host to logon:
>   Password:
>   Initial directory:
>
> As long as the user knows this is a coherent series of prompts from
> the same thread, she will have no difficulty filling in the blanks,
> exactly like she does that now.

If this series of prompts happen in a thread, the implementation of this
prompt series shall protect them by the same `interaction-mutex'. It
doesn't hurt for the prompts themselves, because their locking with
`interaction-mutex' belongs to the same thread, and mutexes are
recursive (meaning that a thread can re-acquire a mutex it owns any
number of times). All other threads are locked when they try to acquire
the `interaction-mutex'.

>>>   And how (in what terms) would you suggest to indicate the
>>> thread ID in this case?
>> 
>> I don't know enough about the thread feature to figure that out.
>> Perhaps based on the thread's top-level function call?
>> Do threads have names?
>
> They do, but the name is optional.  If the Lisp program that starts a
> thread doesn't provide a name, Emacs will identify it as something
> like #<thread 0x01a2d468>, which is not helpful in the situations we
> are discussing.
>
> And even if we do require each thread to have a name, what guarantees
> do we have that the name will be more informative than the prompts
> about which you were worried?

Since there are no rules for thread names, I use the visiting function
name and the name of the visited file in the feature/tramp-thread-safe
branch. Thread names are here like #<thread find-file
/ssh:user@host:/usr/local/src/emacs/CONTRIBUTE>, which could be used
hardly for indication with limited space in the echo area. The only
place it could be used is a tooltip of the prompt, I believe.

Best regards, Michael.

Re: User interaction from multiple threads

[Phil Sainty]

On 2018-08-24 20:51, Eli Zaretskii wrote:
> I don't expect anyone to remember which name corresponds to what job
> that is running in the background, when there are enough of them.

I concur that this would sometimes happen.  Perhaps commonly.  I also
think that other times it would be entirely apparent to the user, and
therefore that facilitating the latter is significantly better than
showing nothing at all on the basis of the former.


> That name will have to come from some Lisp written by the same
> programmers whom you don't trust to provide self-explanatory prompts,

No so -- the programmer who wrote the (new) threaded code which has
triggered the prompt is not necessarily the same programmer who wrote
the (old) prompt code which has been triggered (perhaps by some
unexpected side-effect) by the new code.

Hopefully most prompts *will* still end up being clear in these
situations, but to me it doesn't seem sensible to simply trust that
what is self-explanatory in a single-threaded situation will remain so
in a multi-threaded situation.


> If the Lisp program that starts a thread doesn't provide a name,
> Emacs will identify it as something like #<thread 0x01a2d468>, which
> is not helpful in the situations we are discussing.

Personally I would still vote for display that over displaying
nothing.


> And even if we do require each thread to have a name, what
> guarantees do we have that the name will be more informative than
> the prompts about which you were worried?

Obviously "guarantee" is a strong term, but the thread name would
(or at least *could*) provide a context which is entirely independent
of the prompt.  As I understand it, the prompt code may have no
particular connection to the multi-threaded library which caused it
to appear, and therefore have no way of phrasing itself in a way
which makes that connection apparent to the user.


-Phil

Re: User interaction from multiple threads

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

On Fri, 24 Aug 2018 at 19:46, Michael Albinus <michael.albinus@gmx.de>
wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> > The idea expressed by several people is that once you start
> > interacting with some thread's prompt, the other threads are locked
> > out of interaction, until the interacting thread is done with the
> > series of prompts that allow it to go on with its business.
>
> Yes. The simplest locking mechanism I could imagine is a mutex locking
> such interactions. This must be used for every
> echo-a-prompt-and-read-the-answer interaction by default (we shall
> determine all low-level cases). And it shall be a predefined, documented
> mutex, like `interaction-mutex'.
>

May I propose that the murex be hidden behind a function and a macro? The
purpose of this would be to allow for the function to adviced by libraries
that want to do something special. Something like this:

(defmacro with-prompt-interaction (&body body)
  (call-with-prompt-interaction `(lambda ()  ,@body)))

(defun call-with-prompt-interaction (fn)
  (with-mutex (interaction-mutex)
    (funcall fn)))

This would allow advices to be applied to the call-with-prompt-interaction
function.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 1722 bytes --]

Re: User interaction from multiple threads

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Undoubtedly that would often be true, but are all prompts written in
  > such a specific fashion to guarantee they would be unambiguous within
  > any arbitrary arrangement of asynchronous prompting activity?  I would
  > be very surprised if that turned out to be true.  At minimum, few if
  > any of the existing prompts (many of which may now be triggered in
  > thread contexts), will have been written with this particular scenario
  > in mind.

I think most prompts are written under the assumption that the user
knows which command perse just typed and therefore knows the context
fully.  So far, that's normally so -- but it won't be so with threads
asking for input in parallel.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: User interaction from multiple threads

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The idea expressed by several people is that once you start
  > interacting with some thread's prompt, the other threads are locked
  > out of interaction, until the interacting thread is done with the
  > series of prompts that allow it to go on with its business.

This might be ok if the thread's further prompts arrive in short
succession.  But what if it runs for 5 minutes and then asks
for input?  Should other threads be blocked for those 5 minutes?

I think the point of multiple threads is to avoid that.

Would a thread explicitly seize the terminal and hold it and
later release it?  Or would asking for input hold the terminal
until the thread exits?

  > If that's the first prompt in a series of prompts, we should fix the
  > prompt to tell what password does it ask for.

I don't think that is practical to do in all cases.  In some, yes.
But what if two threads run the same function?  The prompt could be
identical.

  > And even if we do require each thread to have a name, what guarantees
  > do we have that the name will be more informative than the prompts
  > about which you were worried?

If we tell people, "Give your thread a meaningful name if it is going
to ask for input", and we explain why, they will mostly do it.
When they don't, we can fix it.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: User interaction from multiple threads

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > That name will have to come from some Lisp written by the same
  > programmers whom you don't trust to provide self-explanatory prompts,

It is not a matter of "trusting" anyone.  It has nothing to do with
the skill of those people.  That is a red herring.

The existing prompts may be adequate in the current context of use:
after the user runs a command that asks for input.  The command just
entered provides context.

However, when the same prompts appear in the middle of an asynchronous
process, the user doesn't have that helpful context.  Asynchronous
operation requires better prompts, which we didn't need in the past.

Furthermore, if two threads are running the same code on different
data, the prompt may be the same ib both.  That problem doesn't exist
now.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: psainty@orcon.net.nz, hw@adminart.net, gazally@runbox.com,
> 	emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,
> 	emacs-devel@gnu.org
> Date: Fri, 24 Aug 2018 20:20:48 -0400
> 
>   > The idea expressed by several people is that once you start
>   > interacting with some thread's prompt, the other threads are locked
>   > out of interaction, until the interacting thread is done with the
>   > series of prompts that allow it to go on with its business.
> 
> This might be ok if the thread's further prompts arrive in short
> succession.  But what if it runs for 5 minutes and then asks
> for input?  Should other threads be blocked for those 5 minutes?

If that thread runs (i.e. does some Lisp and never calls
wait_reading_process_output), all the other threads will be blocked
anyway, because the global lock is taken by the running thread.

> I think the point of multiple threads is to avoid that.

To some degree.

> Would a thread explicitly seize the terminal and hold it and
> later release it?  Or would asking for input hold the terminal

My conclusion from the discussion is that when a thread asks for
input, it might be too late, because the prompt was already issued.
But maybe we will be able to do something in the redisplay code to
prevent the prompt from showing, in which case the terminal could
indeed be seized when asking for input.  If not, it will have to be
seized just before issuing the prompt.

> until the thread exits?

Not until it exits, until it no longer needs to interact with the
user, for the current transaction (whatever that means).

>   > If that's the first prompt in a series of prompts, we should fix the
>   > prompt to tell what password does it ask for.
> 
> I don't think that is practical to do in all cases.  In some, yes.
> But what if two threads run the same function?  The prompt could be
> identical.

I'd need a practical example to discuss this in more detail.
Theoretically, you are right, but in practice it could be that the
responses are identical (in which case it doesn't matter what thread
asks the question).  Or the prompts will have to be upgraded to speak
for themselves, for example:

  Host for accessing file FOO:
  Password for accessing file FOO:

etc.

>   > And even if we do require each thread to have a name, what guarantees
>   > do we have that the name will be more informative than the prompts
>   > about which you were worried?
> 
> If we tell people, "Give your thread a meaningful name if it is going
> to ask for input", and we explain why, they will mostly do it.
> When they don't, we can fix it.

I'm saying that fixing the prompt itself is a better alternative,
because we will always give the prompt more screen estate than to the
thread ID.  Also, the thread ID is fixed for each thread, whereas we
could add different text to each prompt to make each one more
self-describing.

But in general, I see no significant difference between these two
alternatives, because they both add some text to the prompt.

Re: User interaction from multiple threads

[Phil Sainty]

On 25/08/18 19:51, Eli Zaretskii wrote:
> I'm saying that fixing the prompt itself is a better alternative,
> because we will always give the prompt more screen estate than to
> the thread ID.

I'm a big fan of how eldoc support was added to `eval-expression'
such that when you enter elisp into the minibuffer, the mode line
is used to display the eldoc information.

It occurs to me that a similar thing could be done to display the
thread identifiers for prompts without taking real-estate away from
the prompt itself, by placing that contextual information in the
mode line.

This would mean that the thread identifier does not appear to be
a part of the prompt itself (which I think is good, because it's
not actually part of the prompt, but rather the context in which
the prompt is happening).

This should also help alleviate related concerns about the thread
identifiers needing to be exceptionally short.

Perhaps this approach to the idea makes it more appealing?


> Also, the thread ID is fixed for each thread, whereas we
> could add different text to each prompt to make each one more
> self-describing.
>
> But in general, I see no significant difference between these
> two alternatives, because they both add some text to the prompt.

Personally I feel the differences are significant because they're
each adding distinctly different text.

The thread identifier/context is adding text which the prompt is
incapable of adding for itself, because it has no knowledge of
whether or not it is being executed in the main thread or not.

That aside, it also provides an indicator to the user that the
prompt in question has been triggered by asynchronous activity,
which I think would be helpful information even *without* a
specific ID.

Adding a thread identifier and improving the prompt text also
don't seem to me like mutually exclusive enhancements.  If a
given prompt can be clarified without it seeming excessive in
synchronous situations, that's still a good improvement even
when a thread identifier is also present.


-Phil

Re: User interaction from multiple threads

[Michael Albinus]

Elias Mårtenson <lokedhs@gmail.com> writes:

Hi Elias,

> May I propose that the murex be hidden behind a function and a macro?
> The purpose of this would be to allow for the function to adviced by
> libraries that want to do something special. Something like this:
>
> (defmacro with-prompt-interaction (&body body)
>   (call-with-prompt-interaction `(lambda ()  ,@body)))
>
> (defun call-with-prompt-interaction (fn)
>   (with-mutex (interaction-mutex)
>     (funcall fn)))
>
> This would allow advices to be applied to the
> call-with-prompt-interaction function.

But then you must force everybody to use the
`call-with-prompt-interaction' function. Doesn't it suffice to advice
`mutex-lock' and `mutex-unlock'? In your advice code you could check,
whether MUTEX is equal to `interaction-mutex'.

> Regards,
> Elias

Best regards, Michael.

Re: User interaction from multiple threads

[Tom Tromey]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

Tom> This was what I'd planned to do in the past but never got around to.  A
Tom> first step would be to make a new thread for each terminal and arrange
Tom> for the terminal's file descriptor to be thread-locked.  This didn't
Tom> seem very hard.  Once that's done I guess higher-level problems could be
Tom> looked into; I don't really know this area well though.

Eli> Please describe the proposed solution in more detail, because I'm not
Eli> sure I understand it.  By "new thread" do you mean a Lisp thread of
Eli> the kind made by make-thread, which will run only after it succeeds in
Eli> grabbing the global lock?  Or do you mean some other kind of thread?

Yes, the first step is to make a lisp thread for a new tty terminal, and
change init_tty and/or add_keyboard_wait_descriptor to thread-lock the
new file descriptor.  I don't know what comes after this, since I don't
know much about how the top level works.

I think the GIL doesn't matter much here (though I also think Emacs
should revisit the idea of periodic GIL releases) because waiting for
I/O releases it.  It seems to me that different terminals ought to be
independent, and if I'm, say, in the middle of entering a key sequence
in an X frame, that shouldn't block any sort of action in a tty frame.

Tom

Re: User interaction from multiple threads

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > My conclusion from the discussion is that when a thread asks for
  > input, it might be too late, because the prompt was already issued.

In Emacs, prompts are usually output by a function
whose main purpose is to ask for input.  Is this no longer true?
So in practice it is easy to tell what is a prompt that asks for input.

  > > until the thread exits?

  > Not until it exits, until it no longer needs to interact with the
  > user, for the current transaction (whatever that means).

How does a thread say that it no longer needs to interact...?

  > I'm saying that fixing the prompt itself is a better alternative,
  > because we will always give the prompt more screen estate than to the
  > thread ID.

Emacs could add the thread ID at the start of a prompt
when there are multiple threads that could be asking for input,
such that there is a possibility of confusion.

That could avoid lengthening the prompts in the usual cases.
 


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: psainty@orcon.net.nz, hw@adminart.net, gazally@runbox.com,
> 	emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,
> 	emacs-devel@gnu.org
> Date: Sat, 25 Aug 2018 15:11:10 -0400
> 
>   > My conclusion from the discussion is that when a thread asks for
>   > input, it might be too late, because the prompt was already issued.
> 
> In Emacs, prompts are usually output by a function
> whose main purpose is to ask for input.  Is this no longer true?
> So in practice it is easy to tell what is a prompt that asks for input.

I think we are miscommunicating regarding what exactly constitutes
"asking for input".  I assumed you meant low-level functions such as
wait_reading_process_output, but you seem to be talking about much
higher-level functions, like read-from-minibuffer.

>   > > until the thread exits?
> 
>   > Not until it exits, until it no longer needs to interact with the
>   > user, for the current transaction (whatever that means).
> 
> How does a thread say that it no longer needs to interact...?

We'd need to provide a new API for that.

>   > I'm saying that fixing the prompt itself is a better alternative,
>   > because we will always give the prompt more screen estate than to the
>   > thread ID.
> 
> Emacs could add the thread ID at the start of a prompt
> when there are multiple threads that could be asking for input,
> such that there is a possibility of confusion.
> 
> That could avoid lengthening the prompts in the usual cases.

I don't see how this would solve the problem, since thread IDs will
always be quite short.

Re: User interaction from multiple threads

[hw]

Eli Zaretskii <eliz@gnu.org> writes:


> [...]
>> 
>> Emacs could add the thread ID at the start of a prompt
>> when there are multiple threads that could be asking for input,
>> such that there is a possibility of confusion.
>> 
>> That could avoid lengthening the prompts in the usual cases.
>
> I don't see how this would solve the problem, since thread IDs will
> always be quite short.

I don´t understand this.  I thought requests for input would be queued
in such a way that the users can look at the queue when ever they see
fit, and the information in the queue along with the prompts would make
it obvious to the users what each request is about.

Wouldn't that be much better than having requests for input and the
threads that created them fight over the mini-buffer and interrupt the
user once they can hijack the mini-buffer?

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: hw <hw@adminart.net>
> Cc: rms@gnu.org,  psainty@orcon.net.nz,  emacs-devel@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  gazally@runbox.com
> Date: Sun, 26 Aug 2018 14:52:41 +0200
> 
> > I don't see how this would solve the problem, since thread IDs will
> > always be quite short.
> 
> I don´t understand this.  I thought requests for input would be queued
> in such a way that the users can look at the queue when ever they see
> fit, and the information in the queue along with the prompts would make
> it obvious to the users what each request is about.
> 
> Wouldn't that be much better than having requests for input and the
> threads that created them fight over the mini-buffer and interrupt the
> user once they can hijack the mini-buffer?

You need to catch up with the discussion ;-)  We all but agreed that
threads should not "fight" one another.  Still, when Emacs decides
that a certain thread is granted access to the minibuffer and can
interact with the user, how will the user know which of the N threads
asks them the question, and for what purpose?  The correct answer
could well depend on that.  As a trivial example, consider 2 threads,
one copying files, the other removing files.  How do you know whether
to confirm or not when the prompt says just "FOO (y or n)?"

Re: User interaction from multiple threads

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think we are miscommunicating regarding what exactly constitutes
  > "asking for input".  I assumed you meant low-level functions such as
  > wait_reading_process_output, but you seem to be talking about much
  > higher-level functions, like read-from-minibuffer.

Right.  Because those are the functions that prompt.
Doing it at that level is a possible solution.

I don't insist it is the best solution -- I am not sure what others
there are -- but I think it is worth thinking about.

  > > Emacs could add the thread ID at the start of a prompt
  > > when there are multiple threads that could be asking for input,
  > > such that there is a possibility of confusion.
  > > 
  > > That could avoid lengthening the prompts in the usual cases.

  > I don't see how this would solve the problem, since thread IDs will
  > always be quite short.

It will still be clutter -- helpful in complex situations, unnecessary
in many simple situations.  If it is possible to recognize a common
case and omit the thread IDs, I think lots of users who don't use
threads will appreciate that.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: User interaction from multiple threads

[hw]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hw <hw@adminart.net>
>> Cc: rms@gnu.org, psainty@orcon.net.nz, emacs-devel@gnu.org,
>> emacs-devel-bounces+psainty=orcon.net.nz@gnu.org, gazally@runbox.com
>> Date: Sun, 26 Aug 2018 14:52:41 +0200
>> 
>> > I don't see how this would solve the problem, since thread IDs will
>> > always be quite short.
>> 
>> I don´t understand this.  I thought requests for input would be queued
>> in such a way that the users can look at the queue when ever they see
>> fit, and the information in the queue along with the prompts would make
>> it obvious to the users what each request is about.
>> 
>> Wouldn't that be much better than having requests for input and the
>> threads that created them fight over the mini-buffer and interrupt the
>> user once they can hijack the mini-buffer?
>
> You need to catch up with the discussion ;-)

I read it as far as it has come.  Maybe I'm missing something, or am
just misunderstanding.

> We all but agreed that threads should not "fight" one another.

When you don't want threads to have to fight over the mini-buffer, each
thread needs its own.  I thought that is not really wanted because it is
so difficult to do?

If that would be done, each mini-buffer would need its own history so
users can look at it to remember what they were doing --- I think that
was mentioned.

Do you want to do that?

> Still, when Emacs decides that a certain thread is granted access to
> the minibuffer and can interact with the user, how will the user know
> which of the N threads asks them the question, and for what purpose?

How about: Emacs does not make this decision and does not let threads
interact with users in this manner.  It puts all requests for input onto
the queue unless it can clearly determine that the request is directly
related to what the user is doing or working with.

Since threads are not spawned out of nothing for no reason and with no
purpose and nothing they work with --- at least I imagine so --- and
since Emacs creates and manages the threads, it should be able to put
information into the queue along with the request that tells the user
what the request is about.

> The correct answer could well depend on that.  As a trivial example,
> consider 2 threads, one copying files, the other removing files.  How
> do you know whether to confirm or not when the prompt says just "FOO
> (y or n)?"

Simple: Either both threads are in the background because meanwhile, I'm
doing something else, or one of the threads is in the background and the
other one is in the foreground.

The one in the foreground is the one Emacs would direct my input to if I
were to make any.  What that is must not change by itself.  In case
Emacs can not determine where to direct input to (because there are
multiple simultaneous inputs [1]), all requests must be queued.

Threads working in the background must not interrupt me, hence their
requests for input are put onto the queue and do not appear in the
mini-buffer before I attend to them. --- Only threads that execute in
the foreground are allowed to interrupt me (unless I disable that).

In any case I know that requests appearing in the mini-buffer always
refer to what is in the foreground, so in all cases, I know what the
prompt is about[2], unless I can't remember what I was doing in the
foreground.


What you seem to think you must do --- grant an arbitrary thread access
to the mini-buffer --- is what you *must not* do.  And I think you don't
want to do that.  A queue would cover it, that's why I don't understand
what the problem is.

However, a situation like in [1] requires multiple foregrounds.  What
would that bring about?


[1]: I can ssh into a machine running an Emacs server from many remote
     machines as the same user and have a person at each remote machine
     who uses emacsclient with the same server to edit each a different
     file.  How does Emacs figure out which input goes where and what to
     display in which mini-buffer?

[2]: What happens when I press C-g before a prompt appears, or when one
     has appeared?  How do I interrupt a particular operation a thread
     is or multiple threads are performing, possibly in the background,
     or partly in the background, when there is no list of operations I
     can turn to and interrupt the operation from there?  If there is
     such a list, use it to queue the requests related to each
     operation.  (Who says that one operation is done by only one thread
     each?  Does that require a mini-buffer or queue for each operation,
     for each thread or for both?)

Re: User interaction from multiple threads

[Michael Albinus]

hw <hw@adminart.net> writes:

> Simple: Either both threads are in the background because meanwhile, I'm
> doing something else, or one of the threads is in the background and the
> other one is in the foreground.

Emacs does not support parallel execution of threads. There's only one
thread which is active, no other thread "in the background" is working
the same time. They are either blocked, or waiting that control is
given to them.

Best regards, Michael.

Re: User interaction from multiple threads

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

On Sat, 25 Aug 2018 at 19:58, Michael Albinus <michael.albinus@gmx.de>
wrote:

> Elias Mårtenson <lokedhs@gmail.com> writes:
>
> Hi Elias,
>
> > May I propose that the murex be hidden behind a function and a macro?
> > The purpose of this would be to allow for the function to adviced by
> > libraries that want to do something special. Something like this:
> >
> > (defmacro with-prompt-interaction (&body body)
> >   (call-with-prompt-interaction `(lambda ()  ,@body)))
> >
> > (defun call-with-prompt-interaction (fn)
> >   (with-mutex (interaction-mutex)
> >     (funcall fn)))
> >
> > This would allow advices to be applied to the
> > call-with-prompt-interaction function.
>
> But then you must force everybody to use the
> `call-with-prompt-interaction' function.


Well, yes. Although I'd expect new code to use ‘with-prompt-interaction’.


> Doesn't it suffice to advice
> `mutex-lock' and `mutex-unlock'? In your advice code you could check,
> whether MUTEX is equal to `interaction-mutex'.
>

That would be possible, but looks rather ugly to me. Also, the fact that a
mutex is used should be an implementation detail not visible to the user of
the API. Having a dedicated function/macro pair for this is much cleaner.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 1888 bytes --]

Re: User interaction from multiple threads

[hw]

Michael Albinus <michael.albinus@gmx.de> writes:

> hw <hw@adminart.net> writes:
>
>> Simple: Either both threads are in the background because meanwhile, I'm
>> doing something else, or one of the threads is in the background and the
>> other one is in the foreground.
>
> Emacs does not support parallel execution of threads. There's only one
> thread which is active, no other thread "in the background" is working
> the same time. They are either blocked, or waiting that control is
> given to them.

This is what I have been missing.

It probably means you need to queue all requests for input because there
is no other way for the user to know what a request is about but
checking the queue.  You could give the information along with the
prompt, but I wouldn't want to be interrupted and forced to attend to an
unrelated prompt while doing something --- and I also don't want to have
to maintain the queue when I, for example, want to visit a file and am
prompted for the file name.

Using one thread per terminal/frame may bring about problems I am not
knowledgeable enough to anticipate.  I would imagine each terminal/frame
would need to have it's own mini-buffer.

Why doesn't Emacs support executing multiple threads in parallel?

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: hw <hw@adminart.net>
> Cc: psainty@orcon.net.nz,  gazally@runbox.com,  rms@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 27 Aug 2018 06:33:39 +0200
> 
> > We all but agreed that threads should not "fight" one another.
> 
> When you don't want threads to have to fight over the mini-buffer, each
> thread needs its own.

Not necessarily: we could serialize the interactions.

> If that would be done, each mini-buffer would need its own history so
> users can look at it to remember what they were doing --- I think that
> was mentioned.
> 
> Do you want to do that?

It doesn't seem to solve the problem: Emacs still has to decide which
minibuffer to use when.  And if you want to put that onus on the user,
then that same user could instead switch to thread N in the single
minibuffer we have now.

> How about: Emacs does not make this decision and does not let threads
> interact with users in this manner.  It puts all requests for input onto
> the queue unless it can clearly determine that the request is directly
> related to what the user is doing or working with.

Then the problem becomes how to manage that queue, and which part of
Emacs will do that.  We are back to the same issue, just in a slightly
different form.

> What you seem to think you must do --- grant an arbitrary thread access
> to the mini-buffer --- is what you *must not* do.

But in that case everything becomes sequential again, and we gained
nothing by introducing concurrency into Emacs.  The only threads that
will be able to run non-sequentially are those that never need to tell
the users or ask them anything, and that makes this feature rather
limited, I think.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: hw <hw@adminart.net>
> Cc: gazally@runbox.com,  rms@gnu.org,  psainty@orcon.net.nz,  emacs-devel@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  Eli Zaretskii <eliz@gnu.org>
> Date: Mon, 27 Aug 2018 16:35:17 +0200
> 
> Why doesn't Emacs support executing multiple threads in parallel?

Because it's much harder, what with all the global resources an
editor has.  E.g., consider editing a buffer while another thread
decides to modify it or even delete it.

Re: User interaction from multiple threads

[hw]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hw <hw@adminart.net>
>> Cc: gazally@runbox.com, rms@gnu.org, psainty@orcon.net.nz,
>> emacs-devel@gnu.org,
>> emacs-devel-bounces+psainty=orcon.net.nz@gnu.org, Eli Zaretskii
>> <eliz@gnu.org>
>> Date: Mon, 27 Aug 2018 16:35:17 +0200
>> 
>> Why doesn't Emacs support executing multiple threads in parallel?
>
> Because it's much harder, what with all the global resources an
> editor has.  E.g., consider editing a buffer while another thread
> decides to modify it or even delete it.

When multiple threads are running serialized, they still could decide
--- or already have decided --- to do stuff that is no longer wanted or
possible by the time they get to run (again).  Or not?

Re: User interaction from multiple threads

[hw]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hw <hw@adminart.net>
>> Cc: psainty@orcon.net.nz,  gazally@runbox.com,  rms@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
>> Date: Mon, 27 Aug 2018 06:33:39 +0200
>> 
>> > We all but agreed that threads should not "fight" one another.
>> 
>> When you don't want threads to have to fight over the mini-buffer, each
>> thread needs its own.
>
> Not necessarily: we could serialize the interactions.

Users still require sequentiality to be able to deal with them.

>> If that would be done, each mini-buffer would need its own history so
>> users can look at it to remember what they were doing --- I think that
>> was mentioned.
>> 
>> Do you want to do that?
>
> It doesn't seem to solve the problem: Emacs still has to decide which
> minibuffer to use when.  And if you want to put that onus on the user,
> then that same user could instead switch to thread N in the single
> minibuffer we have now.

true

And if you want neither Emacs to decide which prompts to interrupt the
user with, nor let the user decide which prompts to deal with, you're
kinda out of options.

>> How about: Emacs does not make this decision and does not let threads
>> interact with users in this manner.  It puts all requests for input onto
>> the queue unless it can clearly determine that the request is directly
>> related to what the user is doing or working with.
>
> Then the problem becomes how to manage that queue, and which part of
> Emacs will do that.  We are back to the same issue, just in a slightly
> different form.

Aren't the threads already managed in some way?  The queue would provide
the historical context, and the user dealing with queue-entry X would
implicitly and transparently switch to thread N.

>> What you seem to think you must do --- grant an arbitrary thread access
>> to the mini-buffer --- is what you *must not* do.
>
> But in that case everything becomes sequential again, and we gained
> nothing by introducing concurrency into Emacs.

The queue circumvents this: it would allow to present sequentiality (by
historical context and perhaps by grouping all requests by the operation
they are involved with) to the user while the execution of threads can
remain serialized.

The disadvantage is that when there goes something wrong with a sequence
(bad example: visiting a file before being prompted for a file name),
the user may have a hard to time to notice that there is something
wrong (unless the grouping by operation makes the queue entry appear in
the wrong historical context maybe).

> The only threads that will be able to run non-sequentially are those
> that never need to tell the users or ask them anything, and that makes
> this feature rather limited, I think.

All threads can run in any arbitrary order as long as they queue their
requests.

The alternative is interrupting the users with prompts out of sequence
so they do not know what Emacs wants, prompts serialized or not.

Emacs could even assume the possible answers to a prompt and perform the
actions resulting from the answers in some cases so that when the user
makes a decision, the result is already available and the one not needed
can be discarded :)

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: hw <hw@adminart.net>
> Cc: psainty@orcon.net.nz,  gazally@runbox.com,  rms@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 27 Aug 2018 21:46:30 +0200
> 
> >> When you don't want threads to have to fight over the mini-buffer, each
> >> thread needs its own.
> >
> > Not necessarily: we could serialize the interactions.
> 
> Users still require sequentiality to be able to deal with them.

Yes, but there's no way around that.

> > It doesn't seem to solve the problem: Emacs still has to decide which
> > minibuffer to use when.  And if you want to put that onus on the user,
> > then that same user could instead switch to thread N in the single
> > minibuffer we have now.
> 
> true
> 
> And if you want neither Emacs to decide which prompts to interrupt the
> user with, nor let the user decide which prompts to deal with, you're
> kinda out of options.

I'd prefer Emacs to decide that.  My point above was that having
several minibuffers doesn't solve the basic problem, which is: how
should Emacs decide which of the threads' prompts gets submitted to
the user at any given time.  When we solve that basic problem, we
could talk about the details like whether this happens in the same
minibuffer or in several.

> > Then the problem becomes how to manage that queue, and which part of
> > Emacs will do that.  We are back to the same issue, just in a slightly
> > different form.
> 
> Aren't the threads already managed in some way?

No, not really.  There's a single global lock and a race to grab it
when it becomes released by whatever thread was holding it.

> The queue would provide the historical context, and the user dealing
> with queue-entry X would implicitly and transparently switch to
> thread N.

Placing that onus on the user is indeed one solution, but I still hope
we could do better.  There are still problems even with such "manual"
switching: e.g., a background thread will typically run between 2 key
strokes of user's typing, which could be typing at the prompt of
another thread.  So a background prompt may request to be serviced in
the middle of typing a key sequence, and it isn't clear what to do
then.

> >> What you seem to think you must do --- grant an arbitrary thread access
> >> to the mini-buffer --- is what you *must not* do.
> >
> > But in that case everything becomes sequential again, and we gained
> > nothing by introducing concurrency into Emacs.
> 
> The queue circumvents this: it would allow to present sequentiality (by
> historical context and perhaps by grouping all requests by the operation
> they are involved with) to the user while the execution of threads can
> remain serialized.

Maybe I misunderstood what you proposed, but I thought that a request
gets removed from the queue when its thread finishes its job and
exits.  If this is indeed what you meant, then we would have lost
concurrency.

> Emacs could even assume the possible answers to a prompt and perform the
> actions resulting from the answers in some cases so that when the user
> makes a decision, the result is already available and the one not needed
> can be discarded :)

I don't think this is possible in practice.  We usually ask the user
questions for which only the user knows the answers.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: hw <hw@adminart.net>
> Cc: gazally@runbox.com,  rms@gnu.org,  psainty@orcon.net.nz,  emacs-devel@gnu.org,  michael.albinus@gmx.de,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org
> Date: Mon, 27 Aug 2018 20:43:08 +0200
> 
> >> Why doesn't Emacs support executing multiple threads in parallel?
> >
> > Because it's much harder, what with all the global resources an
> > editor has.  E.g., consider editing a buffer while another thread
> > decides to modify it or even delete it.
> 
> When multiple threads are running serialized, they still could decide
> --- or already have decided --- to do stuff that is no longer wanted or
> possible by the time they get to run (again).  Or not?

How is this different from a single-threaded session?  There's always
the danger that an operation becomes irrelevant by the time it is
carried out, but the danger should normally be very low.

Re: User interaction from multiple threads

[Phil Sainty]

On 28/08/18 17:38, Eli Zaretskii wrote:
> several minibuffers doesn't solve the basic problem, which is: how
> should Emacs decide which of the threads' prompts gets submitted to
> the user at any given time.  When we solve that basic problem, we
> could talk about the details like whether this happens in the same
> minibuffer or in several.

Backing up a little, the notion of multiple minibuffers seems like it
could be relevant to the question of whether we are dealing with
low-level user interactions, or with the higher-level functions used
for asking the user questions.

Is that still an open question?

IIUC the main concern about dealing with user interaction at a low-level
was the point about the prompt already having been generated by the time
the interaction takes place.

At face value, it seems to me that if we had per-thread minibuffers
then perhaps that concern goes away?  I.e. it wouldn't matter that a
given thread had written a prompt to the minibuffer before the
low-level interaction was requested, if the minibuffer it had written
to wasn't being used by the main thread and/or other threads?

(I do recall some earlier discussion about the possibility of thread-
specific frames being created.  I didn't understand the purpose of
that at the time, but perhaps it was similar?)

If handling the user interactions at a low level is desirable, and
multiple minibuffers could enable that, then perhaps the practicality
(or lack thereof, as the case may be) of thread-specific minibuffers
is more than just a "detail", and might factor into the overall
approach that is taken?


-Phil

Re: User interaction from multiple threads

[Eli Zaretskii]

> Cc: hw <hw@adminart.net>, gazally@runbox.com, rms@gnu.org, emacs-devel@gnu.org
> From: Phil Sainty <psainty@orcon.net.nz>
> Date: Wed, 29 Aug 2018 01:05:35 +1200
> 
> On 28/08/18 17:38, Eli Zaretskii wrote:
> > several minibuffers doesn't solve the basic problem, which is: how
> > should Emacs decide which of the threads' prompts gets submitted to
> > the user at any given time.  When we solve that basic problem, we
> > could talk about the details like whether this happens in the same
> > minibuffer or in several.
> 
> Backing up a little, the notion of multiple minibuffers seems like it
> could be relevant to the question of whether we are dealing with
> low-level user interactions, or with the higher-level functions used
> for asking the user questions.
> 
> Is that still an open question?

It certainly is, because when Emacs is idle at top level, and waiting
for the user to type the next command, the minibuffer is implicitly
taken, but without any function like read-from-minibuffer being
called.  So we still need some way of telling the main thread to stop
waiting and yield the minibuffer.

> IIUC the main concern about dealing with user interaction at a low-level
> was the point about the prompt already having been generated by the time
> the interaction takes place.

That was one concern.  The other is what I described above.

> At face value, it seems to me that if we had per-thread minibuffers
> then perhaps that concern goes away?

Not entirely, because there's still a problem of how and when to
display the relevant minibuffer in the echo area.  And the issue with
directing the input the user types to the right thread also stays.

Re: User interaction from multiple threads

[hw]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hw <hw@adminart.net>
>> Cc: gazally@runbox.com, rms@gnu.org, psainty@orcon.net.nz,
>> emacs-devel@gnu.org, michael.albinus@gmx.de,
>> emacs-devel-bounces+psainty=orcon.net.nz@gnu.org
>> Date: Mon, 27 Aug 2018 20:43:08 +0200
>> 
>> >> Why doesn't Emacs support executing multiple threads in parallel?
>> >
>> > Because it's much harder, what with all the global resources an
>> > editor has.  E.g., consider editing a buffer while another thread
>> > decides to modify it or even delete it.
>> 
>> When multiple threads are running serialized, they still could decide
>> --- or already have decided --- to do stuff that is no longer wanted or
>> possible by the time they get to run (again).  Or not?
>
> How is this different from a single-threaded session?  There's always
> the danger that an operation becomes irrelevant by the time it is
> carried out, but the danger should normally be very low.

When another thread can always modify or delete a buffer while the
buffer is being edited, then this possibility doesn't speak against
running threads in parallel rather than serialized.

Re: User interaction from multiple threads

[hw]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hw <hw@adminart.net>
>> Cc: psainty@orcon.net.nz,  gazally@runbox.com,  rms@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
>> Date: Mon, 27 Aug 2018 21:46:30 +0200
>> 
>> >> When you don't want threads to have to fight over the mini-buffer, each
>> >> thread needs its own.
>> >
>> > Not necessarily: we could serialize the interactions.
>> 
>> Users still require sequentiality to be able to deal with them.
>
> Yes, but there's no way around that.
>
>> > It doesn't seem to solve the problem: Emacs still has to decide which
>> > minibuffer to use when.  And if you want to put that onus on the user,
>> > then that same user could instead switch to thread N in the single
>> > minibuffer we have now.
>> 
>> true
>> 
>> And if you want neither Emacs to decide which prompts to interrupt the
>> user with, nor let the user decide which prompts to deal with, you're
>> kinda out of options.
>
> I'd prefer Emacs to decide that.

I think Emacs might need at least some help from the user to make the
right decision.

Imagine serializing requests and presenting them to users with the right
amount of sequentiality and historical context was already solved.  What
is Emacs supposed to do with requests after I start using it for
programming and tell it that I am in do-not-disturb-mode?

That mode means that I do not want to be disturbed by any prompt which
is not directly related to what I'm currently doing, i. e. it's ok for
Emacs to ask me for a file name right away when I press C-x C-f.  It is
definitely not ok to prompt me "FOO (y or n)" because Emacs is still
copying or deleting files and wants to know something.  I'd look into
the progress of that some time later, or the next day maybe.

It is also not ok for gnus to change the size of the mini-buffer or to
ask me for a password when it automatically checks for new mail every
now and then.

Since there is no such thing as "a foreground", Emacs has no idea what
I'm doing and no way to decide which requests it may allow to reach me
and which ones not.  It might help if I could qualify them so I can tell
Emacs which classes of requests to allow and which ones not.

Or it needs to queue them all.

> My point above was that having several minibuffers doesn't solve the
> basic problem,

yes, I agree

> which is: how should Emacs decide which of the threads' prompts gets
> submitted to the user at any given time.  When we solve that basic
> problem, we could talk about the details like whether this happens in
> the same minibuffer or in several.

If Emacs shall be allowed to control the users in such a manner, it
should always forward those requests the users want to currently deal
with and withhold the ones they do not want to see :)

How does Emacs figure out what a user currently wants to deal with?  The
user could help Emacs by going into the do-not-disturb-mode and giving
it an adjustment period of, say, 10 seconds.  During these 10 seconds,
the user does nothing, and Emacs does not create any new threads and
does not forward any requests.

After that, Emacs forwards only those requests that have come up after
the adjustment period and have been created by threads that were created
after the adjustment period and, if they were created by other threads,
the threads creating them must not have been created before the
beginning of the adjustment period.

This is probably a retarded idea which sucks, but it is a kind of way,
time based, to figure out what the user is currently doing.  But it
would allow Emacs to prompt me right away for a file name when I want to
visit one, which is what I would want.

While the user is not in do-not-disturb-mode, it doesn't matter which
prompt is forwarded first.  It could be the oldest one, a random one or
the newest one, maybe depending on which the user prefers.

>> > Then the problem becomes how to manage that queue, and which part of
>> > Emacs will do that.  We are back to the same issue, just in a slightly
>> > different form.
>> 
>> Aren't the threads already managed in some way?
>
> No, not really.  There's a single global lock and a race to grab it
> when it becomes released by whatever thread was holding it.

How is being decided for how long the most a thread can stall the
others?  What information is available when a thread is created?

>> The queue would provide the historical context, and the user dealing
>> with queue-entry X would implicitly and transparently switch to
>> thread N.
>
> Placing that onus on the user is indeed one solution, but I still hope
> we could do better.  There are still problems even with such "manual"
> switching: e.g., a background thread will typically run between 2 key
> strokes of user's typing, which could be typing at the prompt of
> another thread.  So a background prompt may request to be serviced in
> the middle of typing a key sequence, and it isn't clear what to do
> then.

hm

Yeah it would suck if was prompted for FOO instead of for a file name
when I wanted to visit a file.

>> >> What you seem to think you must do --- grant an arbitrary thread access
>> >> to the mini-buffer --- is what you *must not* do.
>> >
>> > But in that case everything becomes sequential again, and we gained
>> > nothing by introducing concurrency into Emacs.
>> 
>> The queue circumvents this: it would allow to present sequentiality (by
>> historical context and perhaps by grouping all requests by the operation
>> they are involved with) to the user while the execution of threads can
>> remain serialized.
>
> Maybe I misunderstood what you proposed, but I thought that a request
> gets removed from the queue when its thread finishes its job and
> exits.  If this is indeed what you meant, then we would have lost
> concurrency.

A request would be removed from the queue when it has been served
(answered) by the user.  How would a thread that needs to wait for the
answer finish its job before it has been given the very answer it needs
to finish?

There wouldn't be much point in allowing threads that need to wait for
an answer to stall others.  The others could still run.

>> Emacs could even assume the possible answers to a prompt and perform the
>> actions resulting from the answers in some cases so that when the user
>> makes a decision, the result is already available and the one not needed
>> can be discarded :)
>
> I don't think this is possible in practice.  We usually ask the user
> questions for which only the user knows the answers.

While (query-replace) is waiting for the user to answer its prompt,
Emacs could anticipate the possibility that the user is going to perform
all the possible replacements (after verifying only a few) and create a
version of the buffer in which all replacements have been performed. Now
when the user actually does what Emacs anticipated, it switches the
buffers so the user doesn't need to wait for the replacements to be
performed (assuming that would be faster than doing the replacements,
for the sake of this example).

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: hw <hw@adminart.net>
> Cc: psainty@orcon.net.nz,  gazally@runbox.com,  rms@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
> Date: Wed, 29 Aug 2018 00:05:08 +0200
> 
> Imagine serializing requests and presenting them to users with the right
> amount of sequentiality and historical context was already solved.  What
> is Emacs supposed to do with requests after I start using it for
> programming and tell it that I am in do-not-disturb-mode?
> 
> That mode means that I do not want to be disturbed by any prompt which
> is not directly related to what I'm currently doing, i. e. it's ok for
> Emacs to ask me for a file name right away when I press C-x C-f.  It is
> definitely not ok to prompt me "FOO (y or n)" because Emacs is still
> copying or deleting files and wants to know something.  I'd look into
> the progress of that some time later, or the next day maybe.

Such a mode could be a useful feature, but we cannot assume all the
users will want to activate it.  We are looking for a way to allow
them to "be disturbed" by the background threads, but in a way that
will still allow to do something useful in the foreground.

> It is also not ok for gnus to change the size of the mini-buffer or to
> ask me for a password when it automatically checks for new mail every
> now and then.

Letting Gnus running in the background ask me for a password is
actually something we'd like to support.

> Since there is no such thing as "a foreground", Emacs has no idea what
> I'm doing and no way to decide which requests it may allow to reach me
> and which ones not.

The Lisp program with which you are interacting could help Emacs have
some idea about these things.

> While the user is not in do-not-disturb-mode, it doesn't matter which
> prompt is forwarded first.

I don't think the order matters.  I think the fundamental problems are
when it is okay to prompt the user, and how to indicate to threads
it's okay.

> >> Aren't the threads already managed in some way?
> >
> > No, not really.  There's a single global lock and a race to grab it
> > when it becomes released by whatever thread was holding it.
> 
> How is being decided for how long the most a thread can stall the
> others?

It can stall them indefinitely.  As long as the thread runs some Lisp
and doesn't call any of the "yielding" APIs, no other thread can run.

> What information is available when a thread is created?

See 'make-thread' for the gory details, but in a nutshell, just the
function which the thread will run and the optional name.
> >> Emacs could even assume the possible answers to a prompt and perform the
> >> actions resulting from the answers in some cases so that when the user
> >> makes a decision, the result is already available and the one not needed
> >> can be discarded :)
> >
> > I don't think this is possible in practice.  We usually ask the user
> > questions for which only the user knows the answers.
> 
> While (query-replace) is waiting for the user to answer its prompt,
> Emacs could anticipate the possibility that the user is going to perform
> all the possible replacements (after verifying only a few) and create a
> version of the buffer in which all replacements have been performed. Now
> when the user actually does what Emacs anticipated, it switches the
> buffers so the user doesn't need to wait for the replacements to be
> performed (assuming that would be faster than doing the replacements,
> for the sake of this example).

I think this strategy will fail most of the time: if the user wants to
confirm all the replacements, she will press '!' early on.  If they
keep pressing 'y' or SPC, my guess is that they want to skip some of
the matches.

Re: User interaction from multiple threads

[John Shahid]

I had some thoughts recently on this issue.  I see three things being
discussed in this thread:

1. How to implement it.

2. How does this implementation interact with current user input, for
example if the user is in the middle of a key sequence.

3. How does the change affect the APIs and how to request multiple inputs.

4. How to indicate which thread will the input go to.

Regarding the implementation, I'm proposing to use the current
primitives and to implement reading key sequences over IPC/RPC.  The
child thread can request the main thread to display a prompt and return
the value that the user entered.  This is somewhat similar to the idea
of having queues, but I believe has some nice side effects related to
the `2'.

In fact, this IPC/RPC doesn't have to be limited to prompts.  It could
be a general macro that takes any body and return its value, e.g.:

   (on-main-thread
    (read-file-name "source: "))

Since all interactions between the child thread and the main thread go
over IPC, where the main thread act like a server, those prompts will
only be displayed if the main thread was idle and waiting for processes
input.  I believe this answer the second question.

In fact, this email thread made me think about asynchronous processes
and how they interact with the main thread.  A thread is not any
different from an asynchronous process.  An asynchronous process could
(through the process filter) cause user prompt just like threads do.
The exception being, process input plays nice with Emacs' input
handling.  We could make all thread interactions with the display go
over IPC and handled the same way like processes.

Regarding the API, I'm proposing a new lower level `on-main-thread`
macro that can run arbitrary code on the main thread returning the value
of the body.  This can be used to implement multiple prompts, e.g.:

  (on-main-thread
    (let ((src  (read-file-name "source: "))
          (dest (read-file-name "dest: ")))
      `(,src ,dest)))

I also propose that all high level prompt related functions be modified
to do that automatically, e.g.:

   (defun read-file-name (prompt ...)
    (if (not main-thread)
      (on-main-thread (read-file-prompt prompt ...))
     ;; current implementation of read-file-prompt
      ))

This can be used for other display related functions, such as changing
the current buffer or opening a new file, preferring communication with
the main thread as the synchronization mechanism instead of using low
level mutexes.

As to how to indicate which thread the input will go to, I don't
currently have good ideas, other than what have been proposed already in
this email thread.  I also think this is the same problem as an
asynchronous process requesting input.

I apologize for any ignorance regarding the internals of Emacs and how
this proposal might interact with those internals.  I am hoping your
responses will enlighten me and teach me a few things.

Thanks,

-js

Re: User interaction from multiple threads

[hw]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hw <hw@adminart.net>
>> Cc: psainty@orcon.net.nz,  gazally@runbox.com,  rms@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
>> Date: Wed, 29 Aug 2018 00:05:08 +0200
>> 
>> Imagine serializing requests and presenting them to users with the right
>> amount of sequentiality and historical context was already solved.  What
>> is Emacs supposed to do with requests after I start using it for
>> programming and tell it that I am in do-not-disturb-mode?
>> 
>> That mode means that I do not want to be disturbed by any prompt which
>> is not directly related to what I'm currently doing, i. e. it's ok for
>> Emacs to ask me for a file name right away when I press C-x C-f.  It is
>> definitely not ok to prompt me "FOO (y or n)" because Emacs is still
>> copying or deleting files and wants to know something.  I'd look into
>> the progress of that some time later, or the next day maybe.
>
> Such a mode could be a useful feature, but we cannot assume all the
> users will want to activate it.  We are looking for a way to allow
> them to "be disturbed" by the background threads, but in a way that
> will still allow to do something useful in the foreground.

Maybe not all users will activate it, and I probably wouldn't activate
it all the time.  How to make Emacs causing disturbances from the
background is a technical problem; disturbing users in what they are
doing is a usability problem.

If Emacs was disturbing me while I'm programming, I would have no choice
but to switch to a different editor.  Letting aside the loss of time and
the annoyance, the rate of errors goes up tremendously when I'm being
disturbed, and fixing them also costs time.  It is devastating for
productivity.

>> It is also not ok for gnus to change the size of the mini-buffer or to
>> ask me for a password when it automatically checks for new mail every
>> now and then.
>
> Letting Gnus running in the background ask me for a password is
> actually something we'd like to support.

That already works since a long time.

>> Since there is no such thing as "a foreground", Emacs has no idea what
>> I'm doing and no way to decide which requests it may allow to reach me
>> and which ones not.
>
> The Lisp program with which you are interacting could help Emacs have
> some idea about these things.

That would be good.

> [...]
>> >> Aren't the threads already managed in some way?
>> >
>> > No, not really.  There's a single global lock and a race to grab it
>> > when it becomes released by whatever thread was holding it.
>> 
>> How is being decided for how long the most a thread can stall the
>> others?
>
> It can stall them indefinitely.  As long as the thread runs some Lisp
> and doesn't call any of the "yielding" APIs, no other thread can run.

That's bad.  Last week I had tramp block me because it couldn't create a
backup file, and it was impossible to interrupt that.

>> What information is available when a thread is created?
>
> See 'make-thread' for the gory details, but in a nutshell, just the
> function which the thread will run and the optional name.

Hm.  I wish the description would also say how much overhead is involved
in creating a thread to give us some idea about when to start a thread
and when not.

Knowing only the name of a function and maybe a name for the thread is
not much.  What if the function calls other functions which prompt the
user?  Is it possible to show a backtrace so the user can get some idea
about what might have caused the prompt to appear?  What if the name is
designed maliciously?

Also, the user goes like "I want to copy files" rather than "I want to
call this function".  I call functions when I can't remember what
keystrokes to use but have an idea of how the function may be called.
How is the function called that deletes files?  Does that depend on
whether the files are remote or local?  What do I do when I can't figure
out what Emacs wants to know when it shows a prompt?

>> >> Emacs could even assume the possible answers to a prompt and perform the
>> >> actions resulting from the answers in some cases so that when the user
>> >> makes a decision, the result is already available and the one not needed
>> >> can be discarded :)
>> >
>> > I don't think this is possible in practice.  We usually ask the user
>> > questions for which only the user knows the answers.
>> 
>> While (query-replace) is waiting for the user to answer its prompt,
>> Emacs could anticipate the possibility that the user is going to perform
>> all the possible replacements (after verifying only a few) and create a
>> version of the buffer in which all replacements have been performed. Now
>> when the user actually does what Emacs anticipated, it switches the
>> buffers so the user doesn't need to wait for the replacements to be
>> performed (assuming that would be faster than doing the replacements,
>> for the sake of this example).
>
> I think this strategy will fail most of the time: if the user wants to
> confirm all the replacements, she will press '!' early on.  If they
> keep pressing 'y' or SPC, my guess is that they want to skip some of
> the matches.

The strategy would have been successful when the user pressed '!' and
had Emacs perform all replacements so far.  It doesn't matter when she
does that unless she does it when it takes longer to switch buffers than
it takes to do the rest of the replacements.

If Emacs learns that she usually does not press '!', it could change its
strategy :)  But when it can prepare stuff using another thread, what
does it matter when it's wrong?

Re: User interaction from multiple threads

[Phil Sainty]

On 30/08/18 05:36, John Shahid wrote:
> The child thread can request the main thread to display a prompt and
> return the value that the user entered.

The notion of user interaction all happening on the main thread was
proposed earlier, and it was pointed out that it's very difficult to
do that, because the user interaction may involve arbitrary data which
is local to the originating thread.

See Eli's comments here:
https://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00676.html

Re: User interaction from multiple threads

[Uday Reddy]

Sorry for getting into this discussion a bit late:

> Use case #1:
> 
>   . The main thread is waiting for user input.  The user didn't yet
>     type anything
>   . A non-main thread runs Lisp that prompts the user for some input

I guess it is obvious that the non-main thread shouldn't interfere until the
main thread's interaction is completed.

> Use case #2:
> 
>   Same as the previous, but now the user is in the middle of typing a
>   key sequence, when the non-main thread prompts.  For example,
>   suppose the user has typed "C-x".

Again, the non-main thread shouldn't interfere.

> Use case #3:
> 
>   Similar, but now 2 or more non-main threads prompt the user, one
>   after the other, in quick succession.  What should happen now, and
>   how will the user know there are multiple prompts?

I don't see "quick succession" as being an issue. The user is going to take
his/her own time to answer. So, it can't happen too quickly.

But the problem in my mind is that the main thread could be prompting for
multiple things, e.g., think of M-x ediff-files. The non-main thread
shouldn't interference until all of those are completed.

Probably in many cases, the mutliple prompts all come from the same
(interactive...) form, in which case all of them need to complete before
another thread can start prompting.

Now suppose multiple related prompts come from separate calls in the main
thread. Then there must be a way to tie all those calls together, in the
manner of a database transaction, and another thread be allowed to prompt
only after the transaction completes.

I think the database transaction model is a good one to follow here.

Cheers,
Uday

Re: User interaction from multiple threads

[Uday S Reddy]

Eli Zaretskii writes:

> As a trivial example, consider 2 threads, one copying files, the other
> removing files. How do you know whether to confirm or not when the prompt
> says just "FOO (y or n)?"

At a minimum, the elisp coder would have to give fuller prompts, e.g.,

  copy FOO (y or n)?
  remove FOO (y or n)?

But still it is dangerous. If a "remove FOO?" prompt comes in the middle of
a series of "copy FOO?" prompts, the user is not going to notice that the
context has changed. The best solution is to lock the user interaction until
all the prompts of one kind are done.

> But in that case everything becomes sequential again, and we gained
> nothing by introducing concurrency into Emacs.  The only threads that
> will be able to run non-sequentially are those that never need to tell
> the users or ask them anything, and that makes this feature rather
> limited, I think.

In my view, the non-main threads asking for user interaction should be a
rarity. Ideally, the main thread should do all the interaction that is
necessary, and then spawn the sub-threads with all the information they
need. 

It is easy for us to generate a hundred threads. But it is not possible for
the user to create a hundred threads in their own head. So, yes, the user is
going to cut down the concurrency.

Cheers,
Uday

Re: User interaction from multiple threads

[John Shahid]

Phil Sainty <psainty@orcon.net.nz> writes:

> On 30/08/18 05:36, John Shahid wrote:
>> The child thread can request the main thread to display a prompt and
>> return the value that the user entered.
>
> The notion of user interaction all happening on the main thread was
> proposed earlier, and it was pointed out that it's very difficult to
> do that, because the user interaction may involve arbitrary data which
> is local to the originating thread.

I thought closures would solve such a problem by encapsulating the
context.  I imagine the main thread receiving forms from other threads
evaluating them and returning the results back to the thread which
becomes the value of the `(on-main-thread ...)`.  What did I miss ?

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Thu, 30 Aug 2018 01:21:51 +0100
> From: Uday Reddy <usr.vm.rocks@gmail.com>
> 
> >   . The main thread is waiting for user input.  The user didn't yet
> >     type anything
> >   . A non-main thread runs Lisp that prompts the user for some input
> 
> I guess it is obvious that the non-main thread shouldn't interfere until the
> main thread's interaction is completed.

When the main thread is waiting for input, the interaction didn't yet
start, so I don't think "until completed" is well-defined in this
case.  This use case is about the main thread being idle waiting for
the user to type something in order _to_begin_ the interaction.

> >   Similar, but now 2 or more non-main threads prompt the user, one
> >   after the other, in quick succession.  What should happen now, and
> >   how will the user know there are multiple prompts?
> 
> I don't see "quick succession" as being an issue. The user is going to take
> his/her own time to answer. So, it can't happen too quickly.

"Quick succession" describes what the background threads do, not how
the user will react.  It is quite possible for several threads to want
to interact with the user almost simultaneously.

> Now suppose multiple related prompts come from separate calls in the main
> thread. Then there must be a way to tie all those calls together, in the
> manner of a database transaction, and another thread be allowed to prompt
> only after the transaction completes.

Yes, we arrived to the same conclusion, more or less.

The problem that remains is how to mange the beginning of some
interaction, how to tell the main thread "to get lost" for a while,
and how to control which thread gets the interaction permission when
the first one is done with its.

Thanks.

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Thu, 30 Aug 2018 02:16:40 +0100
> From: Uday S Reddy <usr.vm.rocks@gmail.com>
> Cc: emacs-devel@gnu.org,
> 
> In my view, the non-main threads asking for user interaction should be a
> rarity.

This would be an unfortunate limitation, since it means people will
need to write code intended to run in non-main threads specially.  As
a corollary, many existing Lisp programs will have to be significantly
rewritten to fit this scheme, before they can be safely run
concurrently with some foreground job done by the main thread.  The
proverbial example is to be able to run Gnus in a non-main thread so
that its slow reading of the newsgroups would not block Emacs.

> Ideally, the main thread should do all the interaction that is
> necessary, and then spawn the sub-threads with all the information
> they need.

I think this will be hard to do in Emacs, given the abundance of
global state.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: John Shahid <jvshahid@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, Gemini Lasswell <gazally@runbox.com>, emacs-devel@gnu.org
> Date: Wed, 29 Aug 2018 22:34:02 -0400
> 
> > The notion of user interaction all happening on the main thread was
> > proposed earlier, and it was pointed out that it's very difficult to
> > do that, because the user interaction may involve arbitrary data which
> > is local to the originating thread.
> 
> I thought closures would solve such a problem by encapsulating the
> context.  I imagine the main thread receiving forms from other threads
> evaluating them and returning the results back to the thread which
> becomes the value of the `(on-main-thread ...)`.  What did I miss ?

I'm afraid there's too much to encapsulate.  E.g., every buffer-local
variable in every buffer to be used by the thread will need to be
encapsulated; and how will the main thread know in advance all that?

And I think your mental model of how threads work in Emacs is
inaccurate: the main thread doesn't dispatch the other threads, in the
sense of controlling them and interacting with them.  It creates a
thread and then the thread competes for the global lock as all the
rest; once a thread grabs the lock, it basically runs disregarding the
other threads (except if the Lisp program explicitly calls APIs that
synchronize with other threads).

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: hw <hw@adminart.net>
> Cc: psainty@orcon.net.nz,  gazally@runbox.com,  rms@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
> Date: Wed, 29 Aug 2018 22:00:52 +0200
> 
> >> What information is available when a thread is created?
> >
> > See 'make-thread' for the gory details, but in a nutshell, just the
> > function which the thread will run and the optional name.
> 
> Hm.  I wish the description would also say how much overhead is involved
> in creating a thread to give us some idea about when to start a thread
> and when not.

There's very little overhead, but if you are interested in the
details, I suggest to read thread.c.  It's not a lot of code, and the
ideas are quite simple.

> Knowing only the name of a function and maybe a name for the thread is
> not much.  What if the function calls other functions which prompt the
> user?  Is it possible to show a backtrace so the user can get some idea
> about what might have caused the prompt to appear?

Someone wrote a mode where you can list threads and display their
backtraces.

> What if the name is designed maliciously?

Why would you run malicious code?  This is orthogonal to the issue
being discussed here.

> Also, the user goes like "I want to copy files" rather than "I want to
> call this function".  I call functions when I can't remember what
> keystrokes to use but have an idea of how the function may be called.
> How is the function called that deletes files?  Does that depend on
> whether the files are remote or local?  What do I do when I can't figure
> out what Emacs wants to know when it shows a prompt?

These questions again are orthogonal to the issue at hand, I think.

> If Emacs learns that she usually does not press '!', it could change its
> strategy :)  But when it can prepare stuff using another thread, what
> does it matter when it's wrong?

So we will need to have machine learning in Emacs for this to work
satisfactorily?

Re: User interaction from multiple threads

[John Shahid]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: John Shahid <jvshahid@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>, Gemini Lasswell <gazally@runbox.com>, emacs-devel@gnu.org
>> Date: Wed, 29 Aug 2018 22:34:02 -0400
>>
>> > The notion of user interaction all happening on the main thread was
>> > proposed earlier, and it was pointed out that it's very difficult to
>> > do that, because the user interaction may involve arbitrary data which
>> > is local to the originating thread.
>>
>> I thought closures would solve such a problem by encapsulating the
>> context.  I imagine the main thread receiving forms from other threads
>> evaluating them and returning the results back to the thread which
>> becomes the value of the `(on-main-thread ...)`.  What did I miss ?
>
> I'm afraid there's too much to encapsulate.  E.g., every buffer-local
> variable in every buffer to be used by the thread will need to be
> encapsulated; and how will the main thread know in advance all that?

Why do we need buffer local variables for all the buffers ? Isn't the
current buffer during the prompt the only relevant buffer. I was
thinking that the `(on-main-thread ...)` macro (or whatever it ends up
being called) can expand to something like the following:

    (execute-on-main-thread
      (let ((buf (current-buffer)))
        (lambda ()
          (with-current-buffer buf
            body))))

> And I think your mental model of how threads work in Emacs is
> inaccurate: the main thread doesn't dispatch the other threads, in the
> sense of controlling them and interacting with them.

I know, I am proposing to change that but not in a very intrusive way.
My idea, is that each thread is started with a pipe which is used to
interact with the main thread when it needs to prompt the user.  The
child thread write the lambda that needs to run on the main thread then
wait for main thread response using `accept-process-output'.  The main
thread handles the request in the process filter displaying the prompt
and writing the result back to the pipe. Since the child and main thread
are sharing the same memory, the communication can simply be the symbol
name that holds the closure and the response, respectively.

Hope this clarifies my idea. Happy to hear your thoughts.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: John Shahid <jvshahid@gmail.com>
> Cc: psainty@orcon.net.nz, gazally@runbox.com, emacs-devel@gnu.org
> Date: Thu, 30 Aug 2018 12:08:32 -0400
> 
> > I'm afraid there's too much to encapsulate.  E.g., every buffer-local
> > variable in every buffer to be used by the thread will need to be
> > encapsulated; and how will the main thread know in advance all that?
> 
> Why do we need buffer local variables for all the buffers ?

Not all the buffers, only those which the thread function will make
current at some point.  Surely, you've seen and written code that
switches buffers as part of doing whatever it needs.

> Isn't the current buffer during the prompt the only relevant buffer.

No, it is not.  There are also various other parts of the global state
that should be specific to a thread, like the list of condition-case
handlers; see the members of 'struct thread_state' as it is defined
now, for more examples.

> My idea, is that each thread is started with a pipe which is used to
> interact with the main thread when it needs to prompt the user.

If you mean literally a pipe, then does this mean you suggest that
Emacs forks itself and runs each thread in a separate process, talking
via pipes with other processes?  That's a completely different
architecture from what we have now, and it wasn't my intent to discuss
such a complete rewrite of the concurrency code.

If this is not what you mean, please clarify what you mean by "pipe"
in this context.

> The child thread write the lambda that needs to run on the main
> thread then wait for main thread response using
> `accept-process-output'.  The main thread handles the request in the
> process filter displaying the prompt and writing the result back to
> the pipe. Since the child and main thread are sharing the same
> memory, the communication can simply be the symbol name that holds
> the closure and the response, respectively.

We already have the async package written by John Wiegley; it sounds
like you are describing the idea which he already implemented.

The limitation of this is that you need to communicate to much of a
state to the subordinate Emacs, and some stuff is very hard, sometimes
impossible, to share this way.

In any case, let's stay in the framework of the current basic design
of how threads work in Emacs.

Thanks.

Re: User interaction from multiple threads

[John Shahid]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: John Shahid <jvshahid@gmail.com>
>> Cc: psainty@orcon.net.nz, gazally@runbox.com, emacs-devel@gnu.org
>> Date: Thu, 30 Aug 2018 12:08:32 -0400
>>
>> > I'm afraid there's too much to encapsulate.  E.g., every buffer-local
>> > variable in every buffer to be used by the thread will need to be
>> > encapsulated; and how will the main thread know in advance all that?
>>
>> Why do we need buffer local variables for all the buffers ?
>
> Not all the buffers, only those which the thread function will make
> current at some point.  Surely, you've seen and written code that
> switches buffers as part of doing whatever it needs.

Yeah, but I'm not sure why the portion that is prompting the user should
be switching buffers.  Even in that case the code will look like the
following where all the necessary variables are captured by the closure:

          (on-main-thread
            (read-file-name ...)
            (with-current-buffer other-buffer ;; captured by the closure
              (do-something)
              (read-file-name ..)))


>> Isn't the current buffer during the prompt the only relevant buffer.
>
> No, it is not.  There are also various other parts of the global state
> that should be specific to a thread, like the list of condition-case
> handlers; see the members of 'struct thread_state' as it is defined
> now, for more examples.

I am not sure I totally understand the impact of every field in that
struct has on my proposal.  Speaking of signals, I think those should be
re-emitted on the child thread stack.  For example, the response of the
main thread can indicate the error symbol and data, which can be used in
the child thread to raise/signal again.

>> My idea, is that each thread is started with a pipe which is used to
>> interact with the main thread when it needs to prompt the user.
>
> If you mean literally a pipe, then does this mean you suggest that
> Emacs forks itself and runs each thread in a separate process, talking
> via pipes with other processes?  That's a completely different
> architecture from what we have now, and it wasn't my intent to discuss
> such a complete rewrite of the concurrency code.

No, I am not proposing changing the architecture.  I was proposing to
literally use pipes to communicate between the different threads.  In
fact that doesn't have to be a pipe it could any implementation of a
queue, as was proposed earlier.

> If this is not what you mean, please clarify what you mean by "pipe"
> in this context.
>
>> The child thread write the lambda that needs to run on the main
>> thread then wait for main thread response using
>> `accept-process-output'.  The main thread handles the request in the
>> process filter displaying the prompt and writing the result back to
>> the pipe. Since the child and main thread are sharing the same
>> memory, the communication can simply be the symbol name that holds
>> the closure and the response, respectively.
>
> We already have the async package written by John Wiegley; it sounds
> like you are describing the idea which he already implemented.
>
> The limitation of this is that you need to communicate to much of a
> state to the subordinate Emacs, and some stuff is very hard, sometimes
> impossible, to share this way.

I am proposing something similar but taking advantage of the fact that
threads are sharing the same memory space.  Similar in the sense of
preferring using the existing communication primitives
(e.g. `make-pipe-process' and `accept-process-output').

> In any case, let's stay in the framework of the current basic design
> of how threads work in Emacs.

I sincerely apologize if what I'm being off topic or not helping this
conversation.  This is either due to my inability to explain my thoughts
clearly or my flawed understanding of Emacs internals.  May be trying to
hack together the solution I have in mind will educate me and prove test
the feasibility of this solution.

Thanks,

-js

Re: User interaction from multiple threads

[Uday Reddy]

Eli Zaretskii writes:

> This would be an unfortunate limitation, since it means people will
> need to write code intended to run in non-main threads specially.  As
> a corollary, many existing Lisp programs will have to be significantly
> rewritten to fit this scheme, before they can be safely run
> concurrently with some foreground job done by the main thread.  The
> proverbial example is to be able to run Gnus in a non-main thread so
> that its slow reading of the newsgroups would not block Emacs.

Put the entire Gnus into a thread? That sounds rather ambitious.

But, more to the point, I don't see what purpose it serves. That Gnus thread
will hang while it is downloading news and the user is left twiddling his
thumbs. Threads can only be useful if Gnus is *modified* to relegate the
network activity to background threads, so that Gnus itself remains
responsive. That can only be achieved if Gnus is purposely rewritten for
that purpose. No free lunch.

The effect of a "Gnus thread" can already be achieved by just starting a
separate Emacs session for Gnus. Why bother with threads just to replicate
the same behaviour?

> > Ideally, the main thread should do all the interaction that is
> > necessary, and then spawn the sub-threads with all the information
> > they need.
> 
> I think this will be hard to do in Emacs, given the abundance of
> global state.

Think of ELPA. You fill out a huge form telling it everything you want it to
do. Then it goes off and does it. No quesitons asked.

Let us call these kind of things "background threads". They will have to be
designed carefully to touch as little global state as possible and there are
arrangements in place for correct sharing when they do touch global state. 

You seem to want "foreground threads", which handle buffers and frames etc.
to do one kind of activity the user is wanting to do. If that is what you
want, then I don't see what escape there is from giving each thread its own
minibuffer.

Cheers,
Uday

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: John Shahid <jvshahid@gmail.com>
> Date: Thu, 30 Aug 2018 16:15:57 -0400
> Cc: psainty@orcon.net.nz, gazally@runbox.com, emacs-devel@gnu.org
> 
> >> Why do we need buffer local variables for all the buffers ?
> >
> > Not all the buffers, only those which the thread function will make
> > current at some point.  Surely, you've seen and written code that
> > switches buffers as part of doing whatever it needs.
> 
> Yeah, but I'm not sure why the portion that is prompting the user should
> be switching buffers.

Well, for starters, every interaction with the user that involves the
minibuffer changes the current buffer to the minibuffer.  And there
are other, more complex situations.

> I am not sure I totally understand the impact of every field in that
> struct has on my proposal.  Speaking of signals, I think those should be
> re-emitted on the child thread stack.  For example, the response of the
> main thread can indicate the error symbol and data, which can be used in
> the child thread to raise/signal again.

We could do that, but this would make programs running in other
threads much more complicated to write and debug, I think.  The
advantage of the current implementation is its simplicity; I wouldn't
suggest to give up that simplicity without very good reasons.

> No, I am not proposing changing the architecture.  I was proposing to
> literally use pipes to communicate between the different threads.  In
> fact that doesn't have to be a pipe it could any implementation of a
> queue, as was proposed earlier.

We could emulate a queue in any number of Lisp data structures, but
that is not the central problem being discussed in this thread.  The
problem is not how to queue requests, the problem is how to handle
them, and how to arrange interaction with the user, i.e. the part
where the user types the responses.

> I am proposing something similar but taking advantage of the fact that
> threads are sharing the same memory space.  Similar in the sense of
> preferring using the existing communication primitives
> (e.g. `make-pipe-process' and `accept-process-output').

Those primitives are implemented around features that are not useful
with threads, I think.

> I sincerely apologize if what I'm being off topic or not helping this
> conversation.

No need to apologize, your messages certainly weren't off topic.

I do suggest to read the related code, so you understand better how
this stuff works.

Re: User interaction from multiple threads

[Eli Zaretskii]

> Date: Thu, 30 Aug 2018 22:12:35 +0100
> From: Uday Reddy <usr.vm.rocks@gmail.com>
> Cc: Uday S Reddy <usr.vm.rocks@gmail.com>,
>     emacs-devel@gnu.org
> 
> Put the entire Gnus into a thread? That sounds rather ambitious.

And yet this is one of the first things people tried as soon as
concurrency landed on the master branch.

> But, more to the point, I don't see what purpose it serves. That Gnus thread
> will hang while it is downloading news and the user is left twiddling his
> thumbs. Threads can only be useful if Gnus is *modified* to relegate the
> network activity to background threads, so that Gnus itself remains
> responsive. That can only be achieved if Gnus is purposely rewritten for
> that purpose. No free lunch.

Not so: since Gnus reads the newsgroups by using APIs like
accept-process-output, the user could still do something useful while
Gnus waits for the server to respond.

IOW, reading newsgroups is a series of calls that accept stuff from
the server, and the thread infrastructure lets the main thread kick in
between the calls.

> The effect of a "Gnus thread" can already be achieved by just starting a
> separate Emacs session for Gnus. Why bother with threads just to replicate
> the same behaviour?

Because (a) you don't need another process, and (b) you can start
reading the new messages before Gnus is done downloading everything.
IOW, exactly for the same reasons it is better to run "M-x compile"
asynchronously instead of in another Emacs session.

> Think of ELPA. You fill out a huge form telling it everything you want it to
> do. Then it goes off and does it. No quesitons asked.

I think this will be a small subset of what people want to do without
locking up Emacs.  Many Lisp programs do most of its job silently, but
do need to ask a question or two from time to time.  There's no need
to give up on running such jobs on non-main threads.

> You seem to want "foreground threads", which handle buffers and frames etc.
> to do one kind of activity the user is wanting to do.

Correction: it's not me, it's Emacs users and programmers.  Case in
point: the feature/tramp-thread-safe branch.  It is not a coincidence
that the very first serious attempt to use the concurrency feature
bumped at these issues almost immediately.

Re: User interaction from multiple threads

[Gemini Lasswell]

John Shahid <jvshahid@gmail.com> writes:

>> No, it is not.  There are also various other parts of the global state
>> that should be specific to a thread, like the list of condition-case
>> handlers; see the members of 'struct thread_state' as it is defined
>> now, for more examples.
>
> I am not sure I totally understand the impact of every field in that
> struct has on my proposal.

I'm still developing my understanding of all this as well, and one
important point that wasn't obvious to me at first is that each thread
has its own special binding stack.  This is the m_specpdl field of
thread_state, and is described a bit in lisp.h.  In addition to
unwind-protects and the Lisp backtrace, it keeps track of
dynamically-bound variables, the values of which are local to each
thread.

The implication of this for the scenario of having a non-main thread ask
the main thread to prompt on its behalf is that not just buffer-local
variables but other global variables, and anything cl-letf can bind
(including function definitions) would have to be communicated between
the threads.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: Gemini Lasswell <gazally@runbox.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  psainty@orcon.net.nz,  emacs-devel@gnu.org
> Date: Fri, 31 Aug 2018 14:37:44 -0700
> 
> I'm still developing my understanding of all this as well, and one
> important point that wasn't obvious to me at first is that each thread
> has its own special binding stack.  This is the m_specpdl field of
> thread_state, and is described a bit in lisp.h.  In addition to
> unwind-protects and the Lisp backtrace, it keeps track of
> dynamically-bound variables, the values of which are local to each
> thread.
> 
> The implication of this for the scenario of having a non-main thread ask
> the main thread to prompt on its behalf is that not just buffer-local
> variables but other global variables, and anything cl-letf can bind
> (including function definitions) would have to be communicated between
> the threads.

Indeed, and that makes this proposal a very complicated one at best.

Re: User interaction from multiple threads

[hw]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hw <hw@adminart.net>
>> Cc: psainty@orcon.net.nz,  gazally@runbox.com,  rms@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
>> Date: Wed, 29 Aug 2018 22:00:52 +0200
>> 
>> >> What information is available when a thread is created?
>> >
>> > See 'make-thread' for the gory details, but in a nutshell, just the
>> > function which the thread will run and the optional name.
>> 
>> Hm.  I wish the description would also say how much overhead is involved
>> in creating a thread to give us some idea about when to start a thread
>> and when not.
>
> There's very little overhead, but if you are interested in the
> details, I suggest to read thread.c.  It's not a lot of code, and the
> ideas are quite simple.

I wish I had at least 50 lifetimes so I could do more of what I would
like to do.  Would you say that when a user is writing a function that
does some string replacements within texts that are usually between 16
and 64kB in size should use multiple threads to replace different
strings or not?

I would expect it to be slower because of the overhead unless multiple
threads could run at the same time.  Since they can't, what would be an
advantage of using multiple threads?

>> Knowing only the name of a function and maybe a name for the thread is
>> not much.  What if the function calls other functions which prompt the
>> user?  Is it possible to show a backtrace so the user can get some idea
>> about what might have caused the prompt to appear?
>
> Someone wrote a mode where you can list threads and display their
> backtraces.
>
>> What if the name is designed maliciously?
>
> Why would you run malicious code?  This is orthogonal to the issue
> being discussed here.

Because the names of the threads hide the fact that it is malicious so I
don't know that it is.

>> Also, the user goes like "I want to copy files" rather than "I want to
>> call this function".  I call functions when I can't remember what
>> keystrokes to use but have an idea of how the function may be called.
>> How is the function called that deletes files?  Does that depend on
>> whether the files are remote or local?  What do I do when I can't figure
>> out what Emacs wants to know when it shows a prompt?
>
> These questions again are orthogonal to the issue at hand, I think.

Considering how the users can figure out what prompts mean and that they
sometimes may not want to be disturbed by unrelated prompts means
approaching the issue from a different angle while the technical problem
of how to prompt them can reasonably not be solved without such
considerations.

>> If Emacs learns that she usually does not press '!', it could change its
>> strategy :)  But when it can prepare stuff using another thread, what
>> does it matter when it's wrong?
>
> So we will need to have machine learning in Emacs for this to work
> satisfactorily?

That would depend on what one is satisfied with.  Generally, well, why
shouldn't an editor or other software learn about the preferences and
the behaviour of its user and thus become able to yield a better usage
experience and better support with the work the user is using the
software to do?

Thinking of it, I much prefer software that kinda does that already;
only it doesn't do that by itself, so I have to do it --- by setting
preferences, writing my own functions, defining my own key bindings, for
example.  I wouldn't need to do all that if the software could figure it
out by itself.

Re: User interaction from multiple threads

[hw]

Uday S Reddy <usr.vm.rocks@gmail.com> writes:

> Eli Zaretskii writes:
>
>> As a trivial example, consider 2 threads, one copying files, the other
>> removing files. How do you know whether to confirm or not when the prompt
>> says just "FOO (y or n)?"
>
> At a minimum, the elisp coder would have to give fuller prompts, e.g.,
>
>   copy FOO (y or n)?
>   remove FOO (y or n)?
>
> But still it is dangerous. If a "remove FOO?" prompt comes in the middle of
> a series of "copy FOO?" prompts, the user is not going to notice that the
> context has changed.

right

> The best solution is to lock the user interaction until all the
> prompts of one kind are done.

Then how do I save my work before dealing with the prompts so at least
that is save when something crashes while I am being forced to answer
them? :~Z  'C-x s' tends to prompt me ...

Not that I would let Emacs force me, it's just for the sake of the
argument.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: hw <hw@adminart.net>
> Cc: psainty@orcon.net.nz,  gazally@runbox.com,  rms@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
> Date: Sat, 01 Sep 2018 19:32:07 +0200
> 
> > There's very little overhead, but if you are interested in the
> > details, I suggest to read thread.c.  It's not a lot of code, and the
> > ideas are quite simple.
> 
> I wish I had at least 50 lifetimes so I could do more of what I would
> like to do.

Same here.

> Would you say that when a user is writing a function that does some
> string replacements within texts that are usually between 16 and
> 64kB in size should use multiple threads to replace different
> strings or not?

Could be, if the code is written to yield from time to time.

> I would expect it to be slower because of the overhead unless multiple
> threads could run at the same time.  Since they can't, what would be an
> advantage of using multiple threads?

It is advantageous if some of the threads call yielding APIs, since
then other threads could run while they are waiting.

RE: User interaction from multiple threads

[Drew Adams]

> > So we will need to have machine learning in Emacs for this to work
> > satisfactorily?
> 
> That would depend on what one is satisfied with.  Generally, well, why
> shouldn't an editor or other software learn about the preferences and
> the behaviour of its user and thus become able to yield a better usage
> experience and better support with the work the user is using the
> software to do?
> 
> Thinking of it, I much prefer software that kinda does that already;
> only it doesn't do that by itself, so I have to do it --- by setting
> preferences, writing my own functions, defining my own key bindings, for
> example.  I wouldn't need to do all that if the software could figure it
> out by itself.

Do you imagine that there is a policy here to exclude what you're asking
for? On the contrary. Patches for such enhancements are welcome, I'm
sure. Emacs development is by volunteers, such as yourself. You've
helped already by offering thoughts about various things. There are lots
of ways to help. Contributing ideas or code for the kind of thing you're
asking for would be great.

Re: User interaction from multiple threads

[hw]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: hw <hw@adminart.net>
>> Cc: psainty@orcon.net.nz,  gazally@runbox.com,  rms@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
>> Date: Sat, 01 Sep 2018 19:32:07 +0200
>> 
>> > There's very little overhead, but if you are interested in the
>> > details, I suggest to read thread.c.  It's not a lot of code, and the
>> > ideas are quite simple.
>> 
>> I wish I had at least 50 lifetimes so I could do more of what I would
>> like to do.
>
> Same here.
>
>> Would you say that when a user is writing a function that does some
>> string replacements within texts that are usually between 16 and
>> 64kB in size should use multiple threads to replace different
>> strings or not?
>
> Could be, if the code is written to yield from time to time.

Does that mean functions like (replace-string) should now be rewritten
to yield time?  Or are they already written like that?  Will or did that
slow them down?

Or does it mean someone writing a function should always write it so
that it yields time, accepting that it might be slowed down?

>> I would expect it to be slower because of the overhead unless multiple
>> threads could run at the same time.  Since they can't, what would be an
>> advantage of using multiple threads?
>
> It is advantageous if some of the threads call yielding APIs, since
> then other threads could run while they are waiting.

Shouldn't threads that are just waiting yield time to others
automatically?

When I use Term::ReadKey in perl to read a key, my program can be
waiting for a key to be pressed without using CPU time.  It does that
automatically.

Re: User interaction from multiple threads

[hw]

Drew Adams <drew.adams@oracle.com> writes:

>> > So we will need to have machine learning in Emacs for this to work
>> > satisfactorily?
>> 
>> That would depend on what one is satisfied with.  Generally, well, why
>> shouldn't an editor or other software learn about the preferences and
>> the behaviour of its user and thus become able to yield a better usage
>> experience and better support with the work the user is using the
>> software to do?
>> 
>> Thinking of it, I much prefer software that kinda does that already;
>> only it doesn't do that by itself, so I have to do it --- by setting
>> preferences, writing my own functions, defining my own key bindings, for
>> example.  I wouldn't need to do all that if the software could figure it
>> out by itself.
>
> Do you imagine that there is a policy here to exclude what you're asking
> for? On the contrary. Patches for such enhancements are welcome, I'm
> sure. Emacs development is by volunteers, such as yourself. You've
> helped already by offering thoughts about various things. There are lots
> of ways to help. Contributing ideas or code for the kind of thing you're
> asking for would be great.

I'd find it wonderful if I could provide such enhancements.  I provided
an enhanced version of hi-lock-mode years ago and never heard back about
it.  I think my version stopped working some time after that, but since
I didn't have time to look into it and didn't use it as much anymore, I
never found out what happened.

Re: User interaction from multiple threads

[Eli Zaretskii]

> From: hw <hw@adminart.net>
> Cc: psainty@orcon.net.nz,  gazally@runbox.com,  rms@gnu.org,  emacs-devel-bounces+psainty=orcon.net.nz@gnu.org,  emacs-devel@gnu.org
> Date: Sun, 02 Sep 2018 18:37:15 +0200
> 
> >> Would you say that when a user is writing a function that does some
> >> string replacements within texts that are usually between 16 and
> >> 64kB in size should use multiple threads to replace different
> >> strings or not?
> >
> > Could be, if the code is written to yield from time to time.
> 
> Does that mean functions like (replace-string) should now be rewritten
> to yield time?

If you want such functions to yield, yes, you need to insert a call to
some yielding API from time to time.

> Or are they already written like that?

No.

> Or does it mean someone writing a function should always write it so
> that it yields time, accepting that it might be slowed down?

Yes, if the function doesn't ask the user questions (in which case it
will yield automatically while it waits for response), like
query-replace.

> > It is advantageous if some of the threads call yielding APIs, since
> > then other threads could run while they are waiting.
> 
> Shouldn't threads that are just waiting yield time to others
> automatically?

They should and they do.  Calling yielding APIs doesn't mean the
programmer needs to inject stuff like thread-yield, it could mean use
of read-key-sequence or accept-process-output or sit-for, to name a
few.

> When I use Term::ReadKey in perl to read a key, my program can be
> waiting for a key to be pressed without using CPU time.  It does that
> automatically.

Emacs does that as well.