A UI approach for making synchronous commands asynchronous

A UI approach for making synchronous commands asynchronous

[Spencer Baugh]

Basic well known facts:
- Sometimes Emacs blocks for a long time while performing a command
- This is annoying because I don't want to sit and wait
- Making it so I don't have to sit and wait requires two things:
  Step 1. Changing the implementation of the command
     (to be capable of running while Emacs handles user input)
  Step 2. Changing the user interface of the command
     (e.g. to display a buffer or message when the command is done)

Step 1 is difficult on its own and is specific to individual commands.
I'm not going to talk about step 1 at all.  (We've talked about it a lot
recently in the "Concurrency via isolated process/thread" thread)

Instead, I have an idea for step 2!

Even once step 1 is already done, step 2 is still a bunch of work: we
have to design a non-blocking UI for the command.  For a lot of
commands, this is difficult.  Changing the UI to be non-blocking is a
big compatibility break, and can confuse and annoy users, and can be
just plain worse in the common case of a short-running command.

We could make blocking vs non-blocking configurable for each command,
but that adds more annoying configuration overhead.

Instead, perhaps we could add a new basic Emacs feature,
"backgrounding", inspired by job control (C-z) in Unix shells.

Commands supporting this feature could start out with a blocking UI.
When such a command is run, Emacs just blocks and doesn't respond to
user input.  The user can wait for as long as they like, and can
interrupt it with C-g.  This is how things are today.

The new feature is that if they get annoyed with how long a command is
taking, they can hit C-M-z to make the command "go into the background":
convert to some kind of non-blocking UI, like displaying a buffer with
progress or messaging when the task is done, as appropriate for the
individual command.

We could also have a prefix (perhaps C-M-&) to run a command in the
"background" - that is, in a non-blocking way - from the beginning.

This UI approach:
- Doesn't break UI compatibility
- Doesn't require any configuration
- Lets users decide on what they want on a case-by-case basis
- Is already familiar to anyone who uses job control in Unix shells

In particular, I was thinking about this for file operations in dired.
Many dired file operations can take a long time, during which they block
Emacs.  Actually changing their implementation to run during user input
handling (step 1) is a separate matter (and I have separate ideas about
how to do that), but I think the UI aspect (step 2) would be well served
by this "backgrounding" approach.  I think this approach would also work
well for some other commands.

Thoughts?

Re: A UI approach for making synchronous commands asynchronous

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Wed, 26 Jul 2023 15:07:39 -0400
> 
> 
> Basic well known facts:
> - Sometimes Emacs blocks for a long time while performing a command
> - This is annoying because I don't want to sit and wait
> - Making it so I don't have to sit and wait requires two things:
>   Step 1. Changing the implementation of the command
>      (to be capable of running while Emacs handles user input)
>   Step 2. Changing the user interface of the command
>      (e.g. to display a buffer or message when the command is done)
> 
> Step 1 is difficult on its own and is specific to individual commands.
> I'm not going to talk about step 1 at all.  (We've talked about it a lot
> recently in the "Concurrency via isolated process/thread" thread)
> 
> Instead, I have an idea for step 2!
> 
> Even once step 1 is already done, step 2 is still a bunch of work: we
> have to design a non-blocking UI for the command.

You have lost me right here.  In Emacs, commands don't have a UI: the
job of displaying the buffer modified by a command is not a job
commands must do.  Instead, the display engine kicks in and does that.

This feature is one of the basic design features of Emacs.  So to talk
about this, we must be on the same page regarding how this stuff works
in Emacs.

In addition, it is not clear what hides behind the "UI" notion that
you have in mind.  There are two aspects to UI: input and display.
Which one are you talking about?  Only the display or also about input
(via the keyboard, mouse, window-system events etc.)?

> The new feature is that if they get annoyed with how long a command is
> taking, they can hit C-M-z to make the command "go into the background":
> convert to some kind of non-blocking UI, like displaying a buffer with
> progress or messaging when the task is done, as appropriate for the
> individual command.
> 
> We could also have a prefix (perhaps C-M-&) to run a command in the
> "background" - that is, in a non-blocking way - from the beginning.

How is this different from your Step 1, which is to change the command
to run asynchronously (a.k.a. "in the background")?  The main problem
in what you describe is not delaying the redisplay (although it is
also not trivial), it is the fact that while the command runs,
something else, like user input, can change the buffer or some other
aspect of the global state behind the command's back.  Dealing with
this is not a UI problem, it's a problem for the command.

> In particular, I was thinking about this for file operations in dired.
> Many dired file operations can take a long time, during which they block
> Emacs.  Actually changing their implementation to run during user input
> handling (step 1) is a separate matter (and I have separate ideas about
> how to do that), but I think the UI aspect (step 2) would be well served
> by this "backgrounding" approach.

Please describe in more details how this would work for some file
operation in Dired.  Suppose I mark some files and then run some
command which searches them or renames them or deletes them -- how
would this work under your hypothetical "backgrounding"?

Re: A UI approach for making synchronous commands asynchronous

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Date: Wed, 26 Jul 2023 15:07:39 -0400
>> 
>> 
>> Basic well known facts:
>> - Sometimes Emacs blocks for a long time while performing a command
>> - This is annoying because I don't want to sit and wait
>> - Making it so I don't have to sit and wait requires two things:
>>   Step 1. Changing the implementation of the command
>>      (to be capable of running while Emacs handles user input)
>>   Step 2. Changing the user interface of the command
>>      (e.g. to display a buffer or message when the command is done)
>> 
>> Step 1 is difficult on its own and is specific to individual commands.
>> I'm not going to talk about step 1 at all.  (We've talked about it a lot
>> recently in the "Concurrency via isolated process/thread" thread)
>> 
>> Instead, I have an idea for step 2!
>> 
>> Even once step 1 is already done, step 2 is still a bunch of work: we
>> have to design a non-blocking UI for the command.
>
> You have lost me right here.  In Emacs, commands don't have a UI: the
> job of displaying the buffer modified by a command is not a job
> commands must do.  Instead, the display engine kicks in and does that.
>
> This feature is one of the basic design features of Emacs.  So to talk
> about this, we must be on the same page regarding how this stuff works
> in Emacs.
>
> In addition, it is not clear what hides behind the "UI" notion that
> you have in mind.  There are two aspects to UI: input and display.
> Which one are you talking about?  Only the display or also about input
> (via the keyboard, mouse, window-system events etc.)?

By "UI" I'm referring to the category of things which includes questions
like:
- should I call display-buffer or message here?
- what keybindings should I use for this set of commands?
- should I provide a prefix argument for so-and-so behavior, or
  put it in a different command, or make it customizable?

As distinguished from the implementation.  The implementation might be
complex, but we might hide that complexity from the user.  (as I believe
you have said before to me)

This is a meaningful category because it is all things we decide on
primarily based on what's best for users, and many different choices are
possible and easy to program.  Whereas the implementation is determined
by the details of the problem and the platforms we run on, and the
possibilities can be heavily limited.

I don't care about the terminology, so what would you call this category
of questions?  I'll use that term from now on.

I did not mean anything to do with the display engine, which I would
class as "implementation", since it's a fundamental hard problem where
we need to make compromises and hard choices, although of course it's a
bit fuzzy.

>> The new feature is that if they get annoyed with how long a command is
>> taking, they can hit C-M-z to make the command "go into the background":
>> convert to some kind of non-blocking UI, like displaying a buffer with
>> progress or messaging when the task is done, as appropriate for the
>> individual command.
>> 
>> We could also have a prefix (perhaps C-M-&) to run a command in the
>> "background" - that is, in a non-blocking way - from the beginning.
>
> How is this different from your Step 1, which is to change the command
> to run asynchronously (a.k.a. "in the background")?  The main problem
> in what you describe is not delaying the redisplay (although it is
> also not trivial), it is the fact that while the command runs,
> something else, like user input, can change the buffer or some other
> aspect of the global state behind the command's back.  Dealing with
> this is not a UI problem, it's a problem for the command.

Yes.  I agree that is a problem for the command, and something which the
command needs to solve.  I call that the "implementation" of the
command.  The "UI" of the command is different.  Perhaps my explanation
of what I meant by "UI" clarifies this.

>> In particular, I was thinking about this for file operations in dired.
>> Many dired file operations can take a long time, during which they block
>> Emacs.  Actually changing their implementation to run during user input
>> handling (step 1) is a separate matter (and I have separate ideas about
>> how to do that), but I think the UI aspect (step 2) would be well served
>> by this "backgrounding" approach.
>
> Please describe in more details how this would work for some file
> operation in Dired.  Suppose I mark some files and then run some
> command which searches them or renames them or deletes them -- how
> would this work under your hypothetical "backgrounding"?

1. I dired-do-rename to move directory "foo" from /home/sbaugh to
/mnt/sbaugh

2. Emacs messages:
  Renaming ~/foo to /mnt/sbaugh/foo...
And stops responding.

3. After about a second I realize that this is a cross-device move which
will take several minutes (a common occurence for me).
I decide I don't want to wait, so I hit C-M-& and Emacs messages:
  Renaming ~/foo to /mnt/sbaugh/foo... backgrounded.
I switch to some other buffers and do some work.

4. Several minutes later, Emacs messages:
  Renaming ~/foo to /mnt/sbaugh/foo... done.



Alternatively, if it was a fast operation:

1. I dired-do-rename to move file "bar.txt" from /home/sbaugh to
/home/sbaugh/projects

2. Emacs messages:
  Renaming ~/bar.txt to ~/projects/bar.txt...
and stops responding.

3. The operation completes after 100ms, before I have time to get
impatient, and Emacs messages:
  Renaming ~/bar.txt to ~/projects/bar.txt... done.

Re: A UI approach for making synchronous commands asynchronous

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Thu, 27 Jul 2023 10:22:17 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > In addition, it is not clear what hides behind the "UI" notion that
> > you have in mind.  There are two aspects to UI: input and display.
> > Which one are you talking about?  Only the display or also about input
> > (via the keyboard, mouse, window-system events etc.)?
> 
> By "UI" I'm referring to the category of things which includes questions
> like:
> - should I call display-buffer or message here?
> - what keybindings should I use for this set of commands?
> - should I provide a prefix argument for so-and-so behavior, or
>   put it in a different command, or make it customizable?

And you wanted _those_ to be asynchronous?  What would that buy us??
And when I asked a question about a specific command, you describe,
below, something very different...

So now I'm completely confused regarding what you have in mind.  In
particular, since both the prefix argument and the key sequence that
invokes the command are completely detached from the command itself,
and are basically processed by the Emacs main loop.

> I don't care about the terminology, so what would you call this category
> of questions?

I have no idea.  Moreover, I'm not even sure why we should find some
terminology, because the collection of things you call "UI" is not
meaningful for the issue at hand, which is how to make Emacs "less
blocking".

> > Please describe in more details how this would work for some file
> > operation in Dired.  Suppose I mark some files and then run some
> > command which searches them or renames them or deletes them -- how
> > would this work under your hypothetical "backgrounding"?
> 
> 1. I dired-do-rename to move directory "foo" from /home/sbaugh to
> /mnt/sbaugh
> 
> 2. Emacs messages:
>   Renaming ~/foo to /mnt/sbaugh/foo...
> And stops responding.
> 
> 3. After about a second I realize that this is a cross-device move which
> will take several minutes (a common occurence for me).
> I decide I don't want to wait, so I hit C-M-& and Emacs messages:
>   Renaming ~/foo to /mnt/sbaugh/foo... backgrounded.
> I switch to some other buffers and do some work.
> 
> 4. Several minutes later, Emacs messages:
>   Renaming ~/foo to /mnt/sbaugh/foo... done.

What happens if during the time the command runs "in the background",
you go in the Dired display to one of the files that have been renamed
and press RET?  Or some command you invoke while the "backgrounded"
one runs wants to access or visit one of the renamed files?

More importantly, why do you think this kind of "backgrounding" will
be possible without deep changes to the Emacs main loop and to the
implementation of the commands as well?  You are basically describing
an Emacs that can run several Lisp threads concurrently, and we just
had (or are still having) a long discussion outlining the problems and
very non-trivial solutions for that.

Re: A UI approach for making synchronous commands asynchronous

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Date: Thu, 27 Jul 2023 10:22:17 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> > In addition, it is not clear what hides behind the "UI" notion that
>> > you have in mind.  There are two aspects to UI: input and display.
>> > Which one are you talking about?  Only the display or also about input
>> > (via the keyboard, mouse, window-system events etc.)?
>> 
>> By "UI" I'm referring to the category of things which includes questions
>> like:
>> - should I call display-buffer or message here?
>> - what keybindings should I use for this set of commands?
>> - should I provide a prefix argument for so-and-so behavior, or
>>   put it in a different command, or make it customizable?
>
> And you wanted _those_ to be asynchronous?  What would that buy us??

No, I don't want those to be asynchronous.  I'm explaining what "UI"
means, since you asked.

> And when I asked a question about a specific command, you describe,
> below, something very different...
>
> So now I'm completely confused regarding what you have in mind.  In
> particular, since both the prefix argument and the key sequence that
> invokes the command are completely detached from the command itself,
> and are basically processed by the Emacs main loop.
>
>> I don't care about the terminology, so what would you call this category
>> of questions?
>
> I have no idea.  Moreover, I'm not even sure why we should find some
> terminology, because the collection of things you call "UI" is not
> meaningful for the issue at hand, which is how to make Emacs "less
> blocking".
>
>> > Please describe in more details how this would work for some file
>> > operation in Dired.  Suppose I mark some files and then run some
>> > command which searches them or renames them or deletes them -- how
>> > would this work under your hypothetical "backgrounding"?
>> 
>> 1. I dired-do-rename to move directory "foo" from /home/sbaugh to
>> /mnt/sbaugh
>> 
>> 2. Emacs messages:
>>   Renaming ~/foo to /mnt/sbaugh/foo...
>> And stops responding.
>> 
>> 3. After about a second I realize that this is a cross-device move which
>> will take several minutes (a common occurence for me).
>> I decide I don't want to wait, so I hit C-M-& and Emacs messages:
>>   Renaming ~/foo to /mnt/sbaugh/foo... backgrounded.
>> I switch to some other buffers and do some work.
>> 
>> 4. Several minutes later, Emacs messages:
>>   Renaming ~/foo to /mnt/sbaugh/foo... done.
>
> What happens if during the time the command runs "in the background",
> you go in the Dired display to one of the files that have been renamed
> and press RET?  Or some command you invoke while the "backgrounded"
> one runs wants to access or visit one of the renamed files?

Good question.  I'd say, let's follow what the shell does.  If you run
"mv ~/foo /mnt/sbaugh/foo &" and then run ls, you'll see some of the
files still in the source, some of the files in the destination.  If you
try to access the former, you might get failures.

The user already can go do other things that interact with those files,
while Emacs is moving them, which won't behave perfectly.  So I don't
think we should take special care around them.

> More importantly, why do you think this kind of "backgrounding" will
> be possible without deep changes to the Emacs main loop and to the
> implementation of the commands as well?  You are basically describing
> an Emacs that can run several Lisp threads concurrently, and we just
> had (or are still having) a long discussion outlining the problems and
> very non-trivial solutions for that.

Again, I'm talking here about what would be a nice *experience for the
user*.  Perhaps "user experience" is a better word to use here.

I don't care about what the implementation is.  I have thoughts about
how we could implement this, but I am not going to discuss it in this
thread, as I said in my initial email, since we already have a very long
thread about that.

I just want to discuss what a good user experience would be, without
talking about implementation.  This is necessarily somewhat speculative,
since we could describe designs which are hard to implement.  That's
nevertheless what I want to do in this thread, since I think it's
useful.

Re: A UI approach for making synchronous commands asynchronous

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Thu, 27 Jul 2023 11:32:33 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> By "UI" I'm referring to the category of things which includes questions
> >> like:
> >> - should I call display-buffer or message here?
> >> - what keybindings should I use for this set of commands?
> >> - should I provide a prefix argument for so-and-so behavior, or
> >>   put it in a different command, or make it customizable?
> >
> > And you wanted _those_ to be asynchronous?  What would that buy us??
> 
> No, I don't want those to be asynchronous.  I'm explaining what "UI"
> means, since you asked.

There must be some kind of misunderstanding.  In your OP, you said:

    Step 1. Changing the implementation of the command
       (to be capable of running while Emacs handles user input)
    Step 2. Changing the user interface of the command
       (e.g. to display a buffer or message when the command is done)

  Step 1 is difficult on its own and is specific to individual commands.
  I'm not going to talk about step 1 at all.  (We've talked about it a lot
  recently in the "Concurrency via isolated process/thread" thread)

  Instead, I have an idea for step 2!

My interpretation of that is that you are suggesting to change the
stuff you call "UI of the command" such that the user will not need to
wait until the command finishes.  Is that understanding correct?

You then say that this "UI" consists of:

  By "UI" I'm referring to the category of things which includes questions
  like:
  - should I call display-buffer or message here?
  - what keybindings should I use for this set of commands?
  - should I provide a prefix argument for so-and-so behavior, or
    put it in a different command, or make it customizable?

My interpretation of this was that you want to be able to use this
stuff while one or more commands still run.  Is that correct?

If both of the above are correct, then you are, indeed, looking for
making these aspects asynchronous.

> > What happens if during the time the command runs "in the background",
> > you go in the Dired display to one of the files that have been renamed
> > and press RET?  Or some command you invoke while the "backgrounded"
> > one runs wants to access or visit one of the renamed files?
> 
> Good question.  I'd say, let's follow what the shell does.  If you run
> "mv ~/foo /mnt/sbaugh/foo &" and then run ls, you'll see some of the
> files still in the source, some of the files in the destination.  If you
> try to access the former, you might get failures.

Emacs does not work like a shell.  For example, when a Dired command
runs, it often changes the contents of the Dired buffer as it
proceeds: e.g., renaming files modifies the Dired buffer to replace
their old names with new ones.  When other commands or you as the user
want to access that buffer, what do you want to happen, and what do
you want to see on display?

A shell doesn't have these problems, it just writes stuff to the
display and forgets about it.

> The user already can go do other things that interact with those files,
> while Emacs is moving them, which won't behave perfectly.  So I don't
> think we should take special care around them.

I actually think this is a tip of a giant iceberg, even in this very
simple example.  Ignoring these aspects is not recommended, if we ever
want something practical to come out of this.

> I don't care about what the implementation is.  I have thoughts about
> how we could implement this, but I am not going to discuss it in this
> thread, as I said in my initial email, since we already have a very long
> thread about that.
> 
> I just want to discuss what a good user experience would be, without
> talking about implementation.  This is necessarily somewhat speculative,
> since we could describe designs which are hard to implement.  That's
> nevertheless what I want to do in this thread, since I think it's
> useful.

Talking about UX without also discussing the implementation of that in
Emacs is not useful.  We all know how it looks and feels to interact
with a program where the UI, both input and output, are detached from
the processing engine and can work in parallel with it.  So I see no
reason to discuss how this should look, because we already know that,
and hardly disagree.  What exactly do you expect to achieve by
describing how C-z works in a Posix shell? are there any people around
here who don't already know that?

The problem is how to implement something like that in Emacs.  IOW,
the important part, and the only one worth discussing if we want
practical results, is how to implement such features without rewriting
all of Emacs.

Re: A UI approach for making synchronous commands asynchronous

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> > What happens if during the time the command runs "in the background",
>> > you go in the Dired display to one of the files that have been renamed
>> > and press RET?  Or some command you invoke while the "backgrounded"
>> > one runs wants to access or visit one of the renamed files?
>> 
>> Good question.  I'd say, let's follow what the shell does.  If you run
>> "mv ~/foo /mnt/sbaugh/foo &" and then run ls, you'll see some of the
>> files still in the source, some of the files in the destination.  If you
>> try to access the former, you might get failures.
>
> Emacs does not work like a shell.  For example, when a Dired command
> runs, it often changes the contents of the Dired buffer as it
> proceeds: e.g., renaming files modifies the Dired buffer to replace
> their old names with new ones.  When other commands or you as the user
> want to access that buffer, what do you want to happen, and what do
> you want to see on display?
>
> A shell doesn't have these problems, it just writes stuff to the
> display and forgets about it.

Ideally the buffer would update incrementally with the new or removed
names as they happen, and be fully updated once the rename is finished.

That can be difficult to implement, though.  And also, for some kinds of
operations, it's not clear what the buffer should look like while the
command is half-done.

So here's another idea that would help with that: maybe we could have a
kind of buffer-specific blocking.  A "blocking" buffer would refuse all
input and commands while it's "blocking", and it wouldn't update, but
the user can switch to other buffers and edit them without a problem.
So, buffer-specific commands wouldn't work, but commands like C-x b and
C-x o would work.  It might be kind of like how term-mode works today.

So in that world, the user would execute a dired rename operation, and
then execute C-M-z to background it, and that would cause that dired
buffer to stop responding while the rename is proceeding, while other
buffers continue to work.

One question is what happens to the user's input when the buffer
"blocks".  Today when Emacs as a whole is blocking, key input gets
queued up and executed when Emacs resumes.  Should the same happen for
blocking buffers?  Or maybe any key input should just immediately result
in errors being printed?  The latter seems preferable, and it wouldn't
be a compatibility break because the user would have to run C-M-z to
trigger such behavior.

Re: A UI approach for making synchronous commands asynchronous

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Thu, 27 Jul 2023 14:22:04 -0400
> 
> Ideally the buffer would update incrementally with the new or removed
> names as they happen, and be fully updated once the rename is finished.
> 
> That can be difficult to implement, though.  And also, for some kinds of
> operations, it's not clear what the buffer should look like while the
> command is half-done.
> 
> So here's another idea that would help with that: maybe we could have a
> kind of buffer-specific blocking.  A "blocking" buffer would refuse all
> input and commands while it's "blocking", and it wouldn't update, but
> the user can switch to other buffers and edit them without a problem.
> So, buffer-specific commands wouldn't work, but commands like C-x b and
> C-x o would work.  It might be kind of like how term-mode works today.

Here you already describe the same "blocking" or "locking" that was
discussed at length in the other thread about concurrency.  Which I
think means that you are basically thinking about the same ideas with
the same issues and possible solutions.  It is not a different set of
ideas.

> So in that world, the user would execute a dired rename operation, and
> then execute C-M-z to background it, and that would cause that dired
> buffer to stop responding while the rename is proceeding, while other
> buffers continue to work.
> 
> One question is what happens to the user's input when the buffer
> "blocks".  Today when Emacs as a whole is blocking, key input gets
> queued up and executed when Emacs resumes.  Should the same happen for
> blocking buffers?  Or maybe any key input should just immediately result
> in errors being printed?  The latter seems preferable, and it wouldn't
> be a compatibility break because the user would have to run C-M-z to
> trigger such behavior.

Queuing input means blocking UI, so if we lock various parts of Emacs
while a command runs, we will have achieved nothing.

Re: A UI approach for making synchronous commands asynchronous

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2340 bytes --]

On Thu, Jul 27, 2023 at 10:32:34PM +0300, Eli Zaretskii wrote:
> > From: Spencer Baugh <sbaugh@janestreet.com>
> > Date: Thu, 27 Jul 2023 14:22:04 -0400
> > 
> > Ideally the buffer would update incrementally with the new or removed
> > names as they happen, and be fully updated once the rename is finished.
> > 
> > That can be difficult to implement, though.  And also, for some kinds of
> > operations, it's not clear what the buffer should look like while the
> > command is half-done.
> > 
> > So here's another idea that would help with that: maybe we could have a
> > kind of buffer-specific blocking.  A "blocking" buffer would refuse all
> > input and commands while it's "blocking", and it wouldn't update, but
> > the user can switch to other buffers and edit them without a problem.
> > So, buffer-specific commands wouldn't work, but commands like C-x b and
> > C-x o would work.  It might be kind of like how term-mode works today.
> 
> Here you already describe the same "blocking" or "locking" that was
> discussed at length in the other thread about concurrency.  Which I
> think means that you are basically thinking about the same ideas with
> the same issues and possible solutions.  It is not a different set of
> ideas.

I think Spencer is onto something here. The really "fun" part in all
this multiprocessing part is the interaction with the user (as an
independent entity).

Spencer's idea is just one possibility. Another would be to just lock
"part of a buffer" (whatever that means,see below): for example, in
Org-Babel, where a snippet of the buffer is passed for evaluation to
some external process and the results (think stdout) are inserted in
a specific region of the buffer, you only have to keep a "lock" on
this.

Now, what happens if the user deletes a whole region containing that
"part of the buffer"? How does Emacs signal to the user that this "part"
is currently "moving"? Should that "part" be intangible? How does Emacs
signal that it's "done"?

I think Spencer's original mail was a proposal to think about such
things and to try to discuss and develop "language elements" (not
in the sense of "computer language", more of "user interaction
language"), that's how I read his "user interface" term there.

Spencer: am I right?

thanks
-- 
tomás

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: A UI approach for making synchronous commands asynchronous

[Eli Zaretskii]

> Date: Fri, 28 Jul 2023 06:32:26 +0200
> Cc: Spencer Baugh <sbaugh@janestreet.com>, emacs-devel@gnu.org
> From: <tomas@tuxteam.de>
> 
> > Here you already describe the same "blocking" or "locking" that was
> > discussed at length in the other thread about concurrency.  Which I
> > think means that you are basically thinking about the same ideas with
> > the same issues and possible solutions.  It is not a different set of
> > ideas.
> 
> I think Spencer is onto something here. The really "fun" part in all
> this multiprocessing part is the interaction with the user (as an
> independent entity).

I'm not saying this doesn't deserve being discussed and considered.
I'm saying that without some kind of plan for _implementing_ the
features such a MO will need, discussing the UI aspects of this is not
just premature, it could be simply a waste of everyone's time.
Because if we conclude that making this happen in Emacs means
redesigning and reimplementing most of Emacs, then those UI aspects
are no longer relevant, since that "New Emacs", if someone will ever
undertake the job of coding it, will look and feel differently anyway.

> Spencer's idea is just one possibility. Another would be to just lock
> "part of a buffer" (whatever that means,see below): for example, in
> Org-Babel, where a snippet of the buffer is passed for evaluation to
> some external process and the results (think stdout) are inserted in
> a specific region of the buffer, you only have to keep a "lock" on
> this.

Currently, the "lock" is implicit, because Emacs simply waits for the
process to finish its work, and then inserts the output into the
buffer, before it proceeds to processing any further input events.

> Now, what happens if the user deletes a whole region containing that
> "part of the buffer"? How does Emacs signal to the user that this "part"
> is currently "moving"? Should that "part" be intangible? How does Emacs
> signal that it's "done"?

These questions are currently irrelevant, because the problems cannot
happen.

Figuring how to allow Emacs do something while another thread runs a
program which could affect buffers or other globally-visible data
structures, is the main difficulty, and was/is discussed at length in
the "concurrency" thread.

> I think Spencer's original mail was a proposal to think about such
> things and to try to discuss and develop "language elements" (not
> in the sense of "computer language", more of "user interaction
> language"), that's how I read his "user interface" term there.

I understand, but I don't believe we can develop such "language
elements" without having some idea about the implementation aspects
that would enable such behavior.  Without such an idea, discussing the
UI aspects has no basis on which to build it and on which to accept or
reject proposals.

Re: A UI approach for making synchronous commands asynchronous

[Spencer Baugh]

Spencer Baugh <sbaugh@janestreet.com> writes:
> Commands supporting this feature could start out with a blocking UI.
> When such a command is run, Emacs just blocks and doesn't respond to
> user input.  The user can wait for as long as they like, and can
> interrupt it with C-g.  This is how things are today.
>
> The new feature is that if they get annoyed with how long a command is
> taking, they can hit C-M-z to make the command "go into the background":
> convert to some kind of non-blocking UI, like displaying a buffer with
> progress or messaging when the task is done, as appropriate for the
> individual command.

vc operations in particular would benefit this approach is vc
operations.  In particular, see bug#21969

That bug happened after vc-diff was made async: now diffs would pop up a
window which asynchronously would have the diff inserted into it.  But a
user complained that this was a worse user experience when the diff was
both empty and fast to generate: in that case, simply synchronously
running the diff and messaging "No diff" is nicer.

So the command was made async, but then reverted to sync because of UI
issues.

So as I said elsewhere, in bug#21969 the asynchronous implementation
*already* exists.  What was missing was a good user experience for the
asynchronous implementation.

I think my "backgrounding" approach would be a perfect solution to this:
start out blocking the UI waiting for the diff, but let the user hit a
key to "background" it, to immediately create the window which will
receive the diff asynchronously.  That works great both in fast and slow
diff-generation scenarios.

Re: A UI approach for making synchronous commands asynchronous

[Helmut Eller]

[-- Attachment #1: Type: text/plain, Size: 952 bytes --]

On Tue, Aug 01 2023, Spencer Baugh wrote:

> I think my "backgrounding" approach would be a perfect solution to this:
> start out blocking the UI waiting for the diff, but let the user hit a
> key to "background" it, to immediately create the window which will
> receive the diff asynchronously.  That works great both in fast and slow
> diff-generation scenarios.

I was wondering what would happen if Emacs would adopt a
one-thread-per-command strategy, i.e. every command starts out in a new
background thread.

Just for fun I came up with the patch from below.

Surprisingly, quite a few things worked. Like moving around in a
buffer, switching buffers, M-x list-threads. Even C-x C-c. Not
surprisingly, many things don't work. Like killing buffers.

Anyway, just wanted to share this idea, in case somebody wants to write
some actual code to give us some form of "background commands". It might
not even require a total rewrite of Emacs.

Helmut


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: simple.el.patch --]
[-- Type: text/x-diff, Size: 1376 bytes --]

diff --git a/lisp/simple.el b/lisp/simple.el
index 6dc08ff0eb0..de3152fde2d 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -2752,6 +2752,7 @@ command-completion--command-for-this-buffer-function
   (let ((if (cconv--interactive-helper--if f)))
     `(interactive ,(if (functionp if) `(funcall ',if) if))))
 
+(defvar background-thread nil)
 (defun command-execute (cmd &optional record-flag keys special)
   ;; BEWARE: Called directly from the C code.
   "Execute CMD as an editor command.
@@ -2803,7 +2804,17 @@ command-execute
           (execute-kbd-macro final prefixarg))
          (t
           ;; Pass `cmd' rather than `final', for the backtrace's sake.
-          (prog1 (call-interactively cmd record-flag keys)
+          (prog1
+              (let ((f (lambda () (call-interactively cmd record-flag keys))))
+                (cond ((or background-thread t)
+                       (funcall f))
+                      (t
+                       (thread-join
+                        (make-thread
+                         (lambda ()
+                           (let ((background-thread (current-thread)))
+                             (funcall f)))
+                         (format "command: %S" cmd))))))
             (when-let ((info
                         (and (symbolp cmd)
                              (not (get cmd 'command-execute-obsolete-warned))

Re: A UI approach for making synchronous commands asynchronous

[Eli Zaretskii]

> From: Helmut Eller <eller.helmut@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Tue, 01 Aug 2023 18:53:12 +0200
> 
> I was wondering what would happen if Emacs would adopt a
> one-thread-per-command strategy, i.e. every command starts out in a new
> background thread.

Based on past discussions, some of the problems with this are:

  . how to handle user interaction from such "background" commands
    (e.g., many commands ask the user for confirmation or prompt the
    user for input of some other kind, like file names)
  . how to trigger redisplay (and remember that some features, like
    messages in echo-area, are actually calls to redisplay in
    disguise)
  . how to handle timers started by some thread
  . how to handle output from subprocesses (a subprocess is by default
    locked to the thread which started it, but interesting stuff
    happens if that thread exits or dies)

But we definitely need to try to use threads more than we do, because
if not, we will never acquire solutions to these problems.