Introducing thread-safe Tramp

Introducing thread-safe Tramp

[Michael Albinus]

Hi,

I have created a new branch feature/tramp-thread-safe. The major change
is making Tramp thread-safe, that means, several basic file operations
can run concurrently now.

Being a library, Tramp does not create new threads on its own. This must
be performed by the callers.

I have changed the `find-file' family of commands to raise their
underlying file operations asynchronously if indicated. Every visiting
of a file happens in its own thread then. If wildcards are used, one
thread per involved file is created.

In order to enable this, the respective command must be called with a
prefix argument, like "C-u C-x C-f ...".  Emacs will remain responsive,
you can continue with whatever operation. This will be best seen with
remote files, because they need more time to load into a buffer. Emacs
is not as responsive yet as I hoped to achieve; further optimization is
needed.

As second change, I have modified `vc-refresh-state' to run always in
its own thread. This is because I have observed, that for git
repositories, loading a remote file is often much faster than the
following update of its state in the modeline. For the time being this
behaviour is hard-coded, I haven't seen any drawback for this even for
local files. But we could trigger it also via a user option.

@Dmitry?

Please give it as much testing as possible. I cannot promise Emacs won't
crash ever with these changes, so it is not time for production
systems. But all bug reports and feedbacks will help to improve this new
feature.

And, if there are not too serious complaints, I will merge it into
master after a while :-) You are warned!

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Mon, 23 Jul 2018 17:58:39 +0200
> Cc: Dmitry Gutov <dgutov@yandex.ru>
> 
> I have created a new branch feature/tramp-thread-safe. The major change
> is making Tramp thread-safe, that means, several basic file operations
> can run concurrently now.

Thank you!

> I have changed the `find-file' family of commands to raise their
> underlying file operations asynchronously if indicated. Every visiting
> of a file happens in its own thread then. If wildcards are used, one
> thread per involved file is created.
> 
> In order to enable this, the respective command must be called with a
> prefix argument, like "C-u C-x C-f ...".  Emacs will remain responsive,
> you can continue with whatever operation.

What happens if find-file or its subroutine ask a question that the
user must answer?

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> In order to enable this, the respective command must be called with a
>> prefix argument, like "C-u C-x C-f ...".  Emacs will remain responsive,
>> you can continue with whatever operation.
>
> What happens if find-file or its subroutine ask a question that the
> user must answer?

It's presented in the minibuffer, and you can answer. But perhaps, it is
hidden by other messages. One of my open points.

"Release early. Release often." (ESR)

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: emacs-devel@gnu.org,  dgutov@yandex.ru
> Date: Mon, 23 Jul 2018 18:11:59 +0200
> 
> > What happens if find-file or its subroutine ask a question that the
> > user must answer?
> 
> It's presented in the minibuffer, and you can answer. But perhaps, it is
> hidden by other messages. One of my open points.

What bothers me more is that it sounds like you trigger redisplay from
a non-main thread (because 'message' triggers a special type of
redisplay).  Moreover, what happens if the main thread is in the
middle of prompting the user by a command that runs in the main
thread?

I suggest to test these situations and see what happens and whether
something (perhaps some infrastructure) is needed to DTRT in those
cases.

P.S. I'm glad to finally see a major Emacs package going threaded.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> What bothers me more is that it sounds like you trigger redisplay from
> a non-main thread (because 'message' triggers a special type of
> redisplay).  Moreover, what happens if the main thread is in the
> middle of prompting the user by a command that runs in the main
> thread?

I believe that message and other minibuffer related functions need to
check whether they are in the main thread. If not, they need a changed
behaviour.

> I suggest to test these situations and see what happens and whether
> something (perhaps some infrastructure) is needed to DTRT in those
> cases.

Likely yes.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Ken Raeburn]

On Jul 23, 2018, at 11:58, Michael Albinus <michael.albinus@gmx.de> wrote:
> 
> Hi,
> 
> I have created a new branch feature/tramp-thread-safe. The major change
> is making Tramp thread-safe, that means, several basic file operations
> can run concurrently now.

This sounds great!
I haven’t had much time for Emacs work recently but I’ll try to try this out soon.

The changes you describe to the find-file interfaces relate to another idea I’ve been thinking of for a while, though I wanted to wait until I had time to look into it a bit more myself. Namely, make some of the C-level local file system interactions release the global lock, and use additional threads for invoking some of those operations.

So, for example, while loading a large file from a slow NFS server, we can also be processing subprocess output, running garbage collection, etc. If the NFS server hangs, ^G should interrupt the main UI thread that’s sitting waiting for a notification from the file I/O thread (which is hung until the server comes back or the access times out; however, other file I/O threads could be used to access other files or file systems), break it out of the wait, and let the user go do something else. In auto-save-mode or auto-revert-mode, read/write/stat calls shouldn’t cause delays in the UI just because a file server is slow.

There are plenty of details I haven’t worked out, like what to do about these background threads acting on a buffer the user is actively modifying at the same time. But one thing that had occurred to me was that as far as the user interaction is concerned, a multithreaded Tramp would be just the way to try out some of these sorts of changes.

Ken

Re: Introducing thread-safe Tramp

[Michael Albinus]

Ken Raeburn <raeburn@raeburn.org> writes:

Hi Ken,

> The changes you describe to the find-file interfaces relate to another
> idea I’ve been thinking of for a while, though I wanted to wait until
> I had time to look into it a bit more myself. Namely, make some of the
> C-level local file system interactions release the global lock, and
> use additional threads for invoking some of those operations.

Yes. But it might be better to use an own mutex for that.

In Tramp, I have introduced one mutex per connection. Whenever any Tramp
file name operation is called, it goes via `tramp-file-name-handler'. Here
I determine the responsible mutex (according to the connection
characteristics), and I lock this mutex. Once the operation has
finished, at the end of `tramp-file-name-handler', the mutex is
unlocked. (To be honest, I simply use `with-mutex'.)

By this, only one file operation per connection can run at a given
time. This is still better than w/o threads, because if you access
concurrently "/ssh:..." and "/sudo:...", the operations could run in
parallel. And, due to the threads created by find-file, Tramp operations
do not block global editing functions.

Of course, this is a coarse approach. On my todo list is a change, to
apply conection-oriented mutexes not on the file operations level, but
on process related level (sending commands remotely, and retrieve the
output). By this, more concurrency shall be possible, with (hopefully)
better performance. Similar to what you have in mind.

> So, for example, while loading a large file from a slow NFS server, we
> can also be processing subprocess output, running garbage collection,
> etc. If the NFS server hangs, ^G should interrupt the main UI thread
> that’s sitting waiting for a notification from the file I/O thread
> (which is hung until the server comes back or the access times out;
> however, other file I/O threads could be used to access other files or
> file systems), break it out of the wait, and let the user go do
> something else. In auto-save-mode or auto-revert-mode, read/write/stat
> calls shouldn’t cause delays in the UI just because a file server is
> slow.

Same would be true for Tramp then.

> There are plenty of details I haven’t worked out, like what to do
> about these background threads acting on a buffer the user is actively
> modifying at the same time. But one thing that had occurred to me was
> that as far as the user interaction is concerned, a multithreaded
> Tramp would be just the way to try out some of these sorts of changes.

Yep. Currently I'm sitting on the problem of handling the minibuffer
properly, when several find-file operations are in progress in
parallel. Something like confirmation of risky local variables in a
file, and alike.

Another problem will be cached values. Tramp caches a lot of file
information, in order to reduce roundtrips. Those cached values might
become invalid due to operations in another thread.

So there's still a lot to do. I hope to get much feedback from you and
other people.

> Ken

Best regards, Michael.

Re: Introducing thread-safe Tramp

[martin rudalics]

 > Yep. Currently I'm sitting on the problem of handling the minibuffer
 > properly, when several find-file operations are in progress in
 > parallel. Something like confirmation of risky local variables in a
 > file, and alike.

I would try mapping each operation to a separate minibuffer-only
frame.

martin

Re: Introducing thread-safe Tramp

[Michael Albinus]

martin rudalics <rudalics@gmx.at> writes:

Hi Martin,

>> Yep. Currently I'm sitting on the problem of handling the minibuffer
>> properly, when several find-file operations are in progress in
>> parallel. Something like confirmation of risky local variables in a
>> file, and alike.
>
> I would try mapping each operation to a separate minibuffer-only
> frame.

Might work. But threads are not bound to frames, and I doubt that it
shall be the general solution. What about Emacs in a terminal window,
w/o frames?

I believe we'd rather go into locking the minibuffer / echo area while
reading. Somewhere in the read_minibuf function of minibuf.c. But I
couldn't nail it down until now.

> martin

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Wed, 25 Jul 2018 11:51:49 +0200
> Cc: Ken Raeburn <raeburn@raeburn.org>, emacs-devel@gnu.org,
> 	Dmitry Gutov <dgutov@yandex.ru>
> 
> I believe we'd rather go into locking the minibuffer / echo area while
> reading.

What do you mean by "locking" in this context?  Ask a thread to take a
mutex before it can use the minibuffer?

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> Date: Wed, 25 Jul 2018 17:44:58 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: rudalics@gmx.at, raeburn@raeburn.org, dgutov@yandex.ru, emacs-devel@gnu.org
> 
> > From: Michael Albinus <michael.albinus@gmx.de>
> > Date: Wed, 25 Jul 2018 11:51:49 +0200
> > Cc: Ken Raeburn <raeburn@raeburn.org>, emacs-devel@gnu.org,
> > 	Dmitry Gutov <dgutov@yandex.ru>
> > 
> > I believe we'd rather go into locking the minibuffer / echo area while
> > reading.
> 
> What do you mean by "locking" in this context?  Ask a thread to take a
> mutex before it can use the minibuffer?

For the messages part, how about if we create a buffer named
*Messages-for-threadHHHHH*, where HHHHH is the thread ID, and put the
echo-area messages from that thread in that buffer?  We could later
invent a machinery to display that buffer automatically when some
event happens, like when the thread exits?

We could do something similar with echo-area buffers: create
additional buffers when threads are created.

WDYT?

Re: Introducing thread-safe Tramp

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 1378 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> > I believe we'd rather go into locking the minibuffer / echo area while
>> > reading.
>> 
>> What do you mean by "locking" in this context?  Ask a thread to take a
>> mutex before it can use the minibuffer?

Yes, I believe that's what we shall do when input is read from the
minibuffer. There shall be one common minibuffer mutex, and it is locked
when input has to be dealt from the minibuffer. Something like the
appended patch, but it doesn't work yet. Likely, due to the messages
arriving the echo area in parallel.

> For the messages part, how about if we create a buffer named
> *Messages-for-threadHHHHH*, where HHHHH is the thread ID, and put the
> echo-area messages from that thread in that buffer?  We could later
> invent a machinery to display that buffer automatically when some
> event happens, like when the thread exits?
>
> We could do something similar with echo-area buffers: create
> additional buffers when threads are created.
>
> WDYT?

Could work. In general, I don't believe it is a problem when messages
arrive the echo area from different threads in parallel. The problem is
when they disturb input reading. While input is taken, messages must be
blocked writing to the minibuffer being their echo area. With the
proposal you have done, or looking for the mutex, or something else.

Best regards, Michael.


[-- Attachment #2: Type: text/plain, Size: 1271 bytes --]

diff --git a/src/minibuf.c b/src/minibuf.c
index abc4866380..b25565d051 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -49,6 +49,10 @@ static Lisp_Object minibuf_save_list;
 
 EMACS_INT minibuf_level;
 
+/* A mutex to lock reading from the minibuffer.  */
+
+Lisp_Object minibuf_mutex;
+
 /* Fread_minibuffer leaves the input here as a string.  */
 
 Lisp_Object last_minibuf_string;
@@ -489,6 +493,7 @@ read_minibuf (Lisp_Object map, Lisp_Object initial, Lisp_Object prompt,
 
   record_unwind_protect_void (read_minibuf_unwind);
   minibuf_level++;
+  Fmutex_lock (minibuf_mutex);
   /* We are exiting the minibuffer one way or the other, so run the hook.
      It should be run before unwinding the minibuf settings.  Do it
      separately from read_minibuf_unwind because we need to make sure that
@@ -778,6 +783,7 @@ read_minibuf_unwind (void)
 
   /* If this was a recursive minibuffer,
      tie the minibuffer window back to the outer level minibuffer buffer.  */
+  Fmutex_unlock (minibuf_mutex);
   minibuf_level--;
 
   window = minibuf_window;
@@ -1877,6 +1883,10 @@ void
 syms_of_minibuf (void)
 {
   minibuf_level = 0;
+
+  minibuf_mutex = Fmake_mutex (Qnil);
+  staticpro (&minibuf_mutex);
+
   minibuf_prompt = Qnil;
   staticpro (&minibuf_prompt);
 

Re: Introducing thread-safe Tramp

[martin rudalics]

 > For the messages part, how about if we create a buffer named
 > *Messages-for-threadHHHHH*, where HHHHH is the thread ID, and put the
 > echo-area messages from that thread in that buffer?  We could later
 > invent a machinery to display that buffer automatically when some
 > event happens, like when the thread exits?
 >
 > We could do something similar with echo-area buffers: create
                                      ^^^^^^^^^^
Did you mean 'minibuffer' here?

martin

Re: Introducing thread-safe Tramp

[Michael Albinus]

martin rudalics <rudalics@gmx.at> writes:

>> For the messages part, how about if we create a buffer named
>> *Messages-for-threadHHHHH*, where HHHHH is the thread ID, and put the
>> echo-area messages from that thread in that buffer?  We could later
>> invent a machinery to display that buffer automatically when some
>> event happens, like when the thread exits?
>>
>> We could do something similar with echo-area buffers: create
>                                      ^^^^^^^^^^
> Did you mean 'minibuffer' here?

Yes, that's the minibuffer. I should have said "echo area" only; I've
meant messages sent to the echo area and appearing in the minibuffer.

> martin

Best regards, Michael.

Re: Introducing thread-safe Tramp

[martin rudalics]

 >>> For the messages part, how about if we create a buffer named
 >>> *Messages-for-threadHHHHH*, where HHHHH is the thread ID, and put the
 >>> echo-area messages from that thread in that buffer?  We could later
 >>> invent a machinery to display that buffer automatically when some
 >>> event happens, like when the thread exits?
 >>>
 >>> We could do something similar with echo-area buffers: create
 >>                                       ^^^^^^^^^^
 >> Did you mean 'minibuffer' here?
 >
 > Yes, that's the minibuffer. I should have said "echo area" only; I've
 > meant messages sent to the echo area and appearing in the minibuffer.

This question was addressed to Eli.  I meant whether he intended
reading from the minibuffer here (which is what I would expect) rather
than displaying a message in the minibuffer window.

martin

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> Date: Thu, 26 Jul 2018 09:56:14 +0200
> From: martin rudalics <rudalics@gmx.at>
> CC: raeburn@raeburn.org, emacs-devel@gnu.org, dgutov@yandex.ru
> 
>  > For the messages part, how about if we create a buffer named
>  > *Messages-for-threadHHHHH*, where HHHHH is the thread ID, and put the
>  > echo-area messages from that thread in that buffer?  We could later
>  > invent a machinery to display that buffer automatically when some
>  > event happens, like when the thread exits?
>  >
>  > We could do something similar with echo-area buffers: create
>                                       ^^^^^^^^^^
> Did you mean 'minibuffer' here?

No, I meant echo-area buffers used to display the echo-area messages.

RE: Introducing thread-safe Tramp

[Drew Adams]

This sounds like a good feature.  I haven't looked at
the code, so I don't really know how it's implemented.
I have one objection to what I think you're proposing
for the design, so far at least.

It's true that `find-file' etc. are commands.  They
are also functions that users sometimes call from Lisp.

My a priori objection is the use of a prefix arg to
indicate that you want a separate thread.  I would
prefer that users specify this intention in some other
way, and that we reserve the use of a prefix arg for
something else (future).

How else might a user specify use of a separate thread?
Possibilities include:

* Use a different function/command.  This is the
  usual way Emacs handles such things: new function.
  It's what we do for the choice of same/other window,
  for example - we have two commands, which can be
  bound to two different keys.

* Bind a defvar variable.  This is what we do for
  respect of `find-file-wildcards', for example.

I'd be OK with either of those approaches, and perhaps
with others.  I doubt that I can be in favor of the
prefix-arg approach.

If it were I, I would probably do something like this:

1. Add another optional arg (Boolean), to determine
   the behavior.  I haven't seen your code, but this
   is probably what you've done already.

2. In the body, use that optional arg to override a
   defvar variable, which otherwise determines the
   behavior (choice).

3. Define additional commands for convenience, which
   pass the optional arg (non-nil) to give the
   separate-thread behavior.

Any user who wants to combine such a separate command
with the corresponding usual command, and who wants
to use a prefix arg to make the choice, can easily do
so.  But Emacs itself will have reserved the ability
to use a prefix arg for something else, and it will
have provided new commands that users can bind to
other keys.

Just one opinion, and open to change.  Thanks again
for this feature.  Let me know, if I misunderstand
something.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

> My a priori objection is the use of a prefix arg to
> indicate that you want a separate thread.  I would
> prefer that users specify this intention in some other
> way, and that we reserve the use of a prefix arg for
> something else (future).

I don't understand your reservation. 40+ years of Emacs, and there was
no need yet for a prefix argument of find-file. I would be surprised if
it happens next time.

Running the commands asynchronously is very tight bound to the usual
invocation of the commands, so I don't see that something else, which
might asks for a prefix argument of the commands, will be better suited.

> How else might a user specify use of a separate thread?
> Possibilities include:
>
> * Use a different function/command.  This is the
>   usual way Emacs handles such things: new function.
>   It's what we do for the choice of same/other window,
>   for example - we have two commands, which can be
>   bound to two different keys.

It isn't about command names. It is about key bindings. We have all
these bindings 'C-x C-f', 'C-x C-r', 'C-x C-v', 'C-x 4 f', 'C-x 5 f'.
If we don't allow a prefix argument, we need complete new bindings. My
memory muscle tells me I'm too old to learn them.

> * Bind a defvar variable.  This is what we do for
>   respect of `find-file-wildcards', for example.

That's a good idea to visit files asynchronously by default. Maybe we
shall add such user option.

But I don't believe people will always visit files asynchronously. They
will do it for remote files, they will do it for huge files. But for my
daily work with local small files, this would be overkill.

> I'd be OK with either of those approaches, and perhaps
> with others.  I doubt that I can be in favor of the
> prefix-arg approach.
>
> If it were I, I would probably do something like this:
>
> 1. Add another optional arg (Boolean), to determine
>    the behavior.  I haven't seen your code, but this
>    is probably what you've done already.

All these functions have a new optional argument `threads'.

> 2. In the body, use that optional arg to override a
>    defvar variable, which otherwise determines the
>    behavior (choice).
>
> 3. Define additional commands for convenience, which
>    pass the optional arg (non-nil) to give the
>    separate-thread behavior.

These are not needed I believe for the reasons above.

> Any user who wants to combine such a separate command
> with the corresponding usual command, and who wants
> to use a prefix arg to make the choice, can easily do
> so.  But Emacs itself will have reserved the ability
> to use a prefix arg for something else, and it will
> have provided new commands that users can bind to
> other keys.

Again, this is based on the assumption we'll need prefix arguments for
something else. I doubt, "something else" will hit us, and if it
happens, that it is better suited to occupy the prefix arg.

> Just one opinion, and open to change.  Thanks again
> for this feature.  Let me know, if I misunderstand
> something.

Same for me. Just my opinion. Let's see what other people say, and what
the maintainers decide.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Phil Sainty]

On 25/07/18 21:46, Michael Albinus wrote:
> Drew Adams <drew.adams@oracle.com> writes:
>> My a priori objection is the use of a prefix arg to indicate that
>> you want a separate thread.  I would prefer that users specify this
>> intention in some other way, and that we reserve the use of a
>> prefix arg for something else (future).
>
> I don't understand your reservation. 40+ years of Emacs, and there
> was no need yet for a prefix argument of find-file. I would be
> surprised if it happens next time.

At first glance I would lean towards Drew's viewpoint on the basis
that it seems to me that there may be a benefit in having a *common*
interface for saying "do a thing asynchronously" -- for all manner of
things which support that (now or in the future).  find-file may never
have acquired a prefix arg, but other commands which may support
asynchronous execution in the future might already use prefix args,
in which case there would end up being inconsistencies in how the
user invoked async behaviour for any given command.

The analog that springs to my mind is the command `append-next-kill'
(bound to C-M-w), which can be invoked in order to "Cause following
command, if it kills, to add to previous kill."

Acting asynchronously might be such a generic requirement that maybe
it makes sense to implement some similar facility to say "Cause the
following command, if it can act asynchronously, to do so."

We might have a user option meaning "commands should act asynchronously
if possible" which, at least in the early days, might be set to nil by
default; but invoking the "run the next command asynchronously" command
could bind that variable as non-nil for the duration of the following
command.

Then if it was a convention that functions implementing optional async
behaviour did so conditional on this variable, we could gain the
desired outcome not only for `find-file' but for anything else in the
future which implemented such support.

Individual libraries might implement their own additional user options
to enable users to specify default async behaviour for particular sets
of commands implemented by the library in question (e.g. `find-file'
and friends), and/or users could use advice to bind the "use async
behaviour" variable for specific commands if they wanted it for some
things but not others.

Ultimately I can envisage that async-supporting commands which are
initially synchronous-by-default may in the future end up being
asynchronous-by-default once this functionality is better established,
and so tying prefix arguments to such functionality now may end up
seeming short-sighted in the future?


-Phil

Re: Introducing thread-safe Tramp

[Andreas Schwab]

On Jul 25 2018, Phil Sainty <psainty@orcon.net.nz> wrote:

> The analog that springs to my mind is the command `append-next-kill'
> (bound to C-M-w), which can be invoked in order to "Cause following
> command, if it kills, to add to previous kill."

Or like C-x C-m c (universal-coding-system-argument).

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Introducing thread-safe Tramp

[Michael Albinus]

Phil Sainty <psainty@orcon.net.nz> writes:

Hi Phil,

> Ultimately I can envisage that async-supporting commands which are
> initially synchronous-by-default may in the future end up being
> asynchronous-by-default once this functionality is better established,
> and so tying prefix arguments to such functionality now may end up
> seeming short-sighted in the future?

You have a point. But I still believe that synchronous visiting of files
will still be used, especially for small local files. Se we could have a
user option which toggles synchronous-by-default or asynchronous-by-default,
and we should have an indication do-it-the-other-way. Then people could
decide their default, and they could overwrite this default for single files.

The variable I could add easily. But which general indication do we want
to use, if it isn't prefix arg?

> -Phil

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Robert Pluim]

Michael Albinus <michael.albinus@gmx.de> writes:

> Phil Sainty <psainty@orcon.net.nz> writes:
>
> Hi Phil,
>
>> Ultimately I can envisage that async-supporting commands which are
>> initially synchronous-by-default may in the future end up being
>> asynchronous-by-default once this functionality is better established,
>> and so tying prefix arguments to such functionality now may end up
>> seeming short-sighted in the future?
>
> You have a point. But I still believe that synchronous visiting of files
> will still be used, especially for small local files. Se we could have a
> user option which toggles synchronous-by-default or asynchronous-by-default,
> and we should have an indication do-it-the-other-way. Then people could
> decide their default, and they could overwrite this default for single files.
>
> The variable I could add easily. But which general indication do we want
> to use, if it isn't prefix arg?

Given that this is all highly experimental, I wouldn't mind a 'be 100%
asynchronous' variable in order to ensure maximal coverage (plus I
donʼt want the cognitive load of having to think "do I want this
operation to be asynchronous"). Once we have more experience with that
we can decide which if any operations should default to asynchronous.

Robert

Re: Introducing thread-safe Tramp

[Michael Albinus]

Robert Pluim <rpluim@gmail.com> writes:

Hi Robert,

> Given that this is all highly experimental, I wouldn't mind a 'be 100%
> asynchronous' variable in order to ensure maximal coverage (plus I
> donʼt want the cognitive load of having to think "do I want this
> operation to be asynchronous"). Once we have more experience with that
> we can decide which if any operations should default to asynchronous.

For the time being, I have added

--8<---------------cut here---------------start------------->8---
find-file-asynchronously is a variable defined in ‘files.el’.
Its value is nil

Documentation:
Non-nil means visit file asynchronously when called interactively.
This behavior is toggled by a prefix argument to the interactive call.
--8<---------------cut here---------------end--------------->8---

If there are better proposals to the option's name, and to the key
binding for the toggle behavior, I'm all open.

> Robert

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Robert Pluim]

Robert Pluim <rpluim@gmail.com> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> Robert Pluim <rpluim@gmail.com> writes:
>>
>> Hi Robert,
>>
>>> Given that this is all highly experimental, I wouldn't mind a 'be 100%
>>> asynchronous' variable in order to ensure maximal coverage (plus I
>>> donʼt want the cognitive load of having to think "do I want this
>>> operation to be asynchronous"). Once we have more experience with that
>>> we can decide which if any operations should default to asynchronous.
>>
>> For the time being, I have added
>>
>> find-file-asynchronously is a variable defined in ‘files.el’.
>> Its value is nil
>>
>> Documentation:
>> Non-nil means visit file asynchronously when called interactively.
>> This behavior is toggled by a prefix argument to the interactive call.
>>
>> If there are better proposals to the option's name, and to the key
>> binding for the toggle behavior, I'm all open.

So Iʼve been running with find-file-asynchronously t for the past 5
days, and I can't say Iʼve noticed anything going wrong. Having said
that Iʼm not a particularly heavy user of tramp, but so far itʼs all
looking good.

Robert

Re: Introducing thread-safe Tramp

[Michael Albinus]

Robert Pluim <rpluim@gmail.com> writes:

Hi Robert,

> So Iʼve been running with find-file-asynchronously t for the past 5
> days, and I can't say Iʼve noticed anything going wrong. Having said
> that Iʼm not a particularly heavy user of tramp, but so far itʼs all
> looking good.

If you set find-file-asynchronously to t, also local files will be
visited asynchronously. This makes your test even more valuable :-)

> Robert

Best regards, Michael.

PS: Will continue to work on the branch when I'm back from vacation,
next week.

Re: Introducing thread-safe Tramp

[Filipp Gunbin]

On 25/07/2018 11:46 +0200, Michael Albinus wrote:

> Running the commands asynchronously is very tight bound to the usual
> invocation of the commands, so I don't see that something else, which
> might asks for a prefix argument of the commands, will be better suited.

I don't like prefix arg, too.  It feels like prefix arg should change
the user-visible behaviour of the command, and running asynchronously
feels more like a technical detail, though important one.

Maybe there could be rules for find-file to use async behavior by
default, like:

- When a non-local method is explicitly used (that is, not /sudo)

- When default-directory is on a remote host

- When a file being visited is greater than some customizable size

- When a special command was given before (analogy with `C-x RET c' was
already mentioned here).  That special command by default could "toggle
default".  It could accept prefix arg, say C-u for "force sync" and C-u
C-u for "force async".  That way sync/async preference will be decoupled
from the actual command, and could be added to any other command in the
future.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

> I don't like prefix arg, too.  It feels like prefix arg should change
> the user-visible behaviour of the command, and running asynchronously
> feels more like a technical detail, though important one.

It is user-visible. If you visit a file asynchronously, Emacs is still
responsive, and you can run other commands. For example, during
asynchronous loading of /ssh::~/src/emacs/admin/* I was still able to
call gnus and read my emails.

> Maybe there could be rules for find-file to use async behavior by
> default, like:
>
> - When a non-local method is explicitly used (that is, not /sudo)
>
> - When default-directory is on a remote host
>
> - When a file being visited is greater than some customizable size

I was thinking already about. find-file-asynchronously could be a regexp
or a function, determining whether a file should be visited
asynchronously, or not.

> - When a special command was given before (analogy with `C-x RET c' was
> already mentioned here).  That special command by default could "toggle
> default".  It could accept prefix arg, say C-u for "force sync" and C-u
> C-u for "force async".  That way sync/async preference will be decoupled
> from the actual command, and could be added to any other command in the
> future.

I will play with these proposals.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Filipp Gunbin]

On 27/07/2018 21:42 +0200, Michael Albinus wrote:

> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>
>> I don't like prefix arg, too.  It feels like prefix arg should change
>> the user-visible behaviour of the command, and running asynchronously
>> feels more like a technical detail, though important one.
>
> It is user-visible. If you visit a file asynchronously, Emacs is still
> responsive, and you can run other commands. For example, during
> asynchronous loading of /ssh::~/src/emacs/admin/* I was still able to
> call gnus and read my emails.

Yes, it is.  But.. it changes how the command runs, not what it does
(the result is the same, whether sync or async).  So it looks like a
user should be able to both change command behavior and specify
sync/async preference.  That's where decoupling helps.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Date: Sat, 28 Jul 2018 03:14:39 +0300
> Cc: Drew Adams <drew.adams@oracle.com>, emacs-devel@gnu.org
> 
> > It is user-visible. If you visit a file asynchronously, Emacs is still
> > responsive, and you can run other commands. For example, during
> > asynchronous loading of /ssh::~/src/emacs/admin/* I was still able to
> > call gnus and read my emails.
> 
> Yes, it is.  But.. it changes how the command runs, not what it does
> (the result is the same, whether sync or async).

The result is the same in most cases, but the user-facing behavior is
very different.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Michael Albinus <michael.albinus@gmx.de> writes:

>> - When a special command was given before (analogy with `C-x RET c' was
>> already mentioned here).  That special command by default could "toggle
>> default".  It could accept prefix arg, say C-u for "force sync" and C-u
>> C-u for "force async".  That way sync/async preference will be decoupled
>> from the actual command, and could be added to any other command in the
>> future.
>
> I will play with these proposals.

As you have seen the other mail, I'm using now C-u to toggle the default
value of find-file-asynchronously. I'm open for any other key binding,
but C-u has the advantage that it works for both "C-u C-x C-f ..." and
"C-u M-x find-file ..." and friends.

Best regards, Michael.

RE: Introducing thread-safe Tramp

[Drew Adams]

> >> - When a special command was given before (analogy with `C-x RET c' was
> >> already mentioned here).  That special command by default could "toggle
> >> default".  It could accept prefix arg, say C-u for "force sync" and C-u
> >> C-u for "force async".  That way sync/async preference will be decoupled
> >> from the actual command, and could be added to any other command in
> the
> >> future.
> >
> > I will play with these proposals.
> 
> As you have seen the other mail, I'm using now C-u to toggle the default
> value of find-file-asynchronously. I'm open for any other key binding,
> but C-u has the advantage that it works for both "C-u C-x C-f ..." and
> "C-u M-x find-file ..." and friends.

(Don't you mean that `C-u' simply toggles `find-file-asynchronously' - its current value, not its default value?)

As mentioned, I don't think it's a good idea to spend/waste the use of a prefix arg on this. (Just one opinion.) To me, it doesn't matter so much that such commands are longstanding and we haven't yet seen competing requests for prefix-arg behavior.

But if that's the way this will be done (`C-u' toggles ` find-file-asynchronously') then please consider this instead: Don't co-opt the use of an arbitrary prefix arg; instead, do what you propose only for `C-u C-u' (double plain prefix arg). That will make it much easier to accommodate other uses of (other) prefix args in the future.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

>> As you have seen the other mail, I'm using now C-u to toggle the default
>> value of find-file-asynchronously. I'm open for any other key binding,
>> but C-u has the advantage that it works for both "C-u C-x C-f ..." and
>> "C-u M-x find-file ..." and friends.
>
> (Don't you mean that `C-u' simply toggles `find-file-asynchronously' -
> its current value, not its default value?)

Of course.

> As mentioned, I don't think it's a good idea to spend/waste the use of
> a prefix arg on this. (Just one opinion.) To me, it doesn't matter so
> much that such commands are longstanding and we haven't yet seen
> competing requests for prefix-arg behavior.
>
> But if that's the way this will be done (`C-u' toggles `
> find-file-asynchronously') then please consider this instead: Don't
> co-opt the use of an arbitrary prefix arg; instead, do what you
> propose only for `C-u C-u' (double plain prefix arg). That will make
> it much easier to accommodate other uses of (other) prefix args in the
> future.

If nobody objects, I could do this. However, during the discussion it
was proposed to find a common indication for asynchronous file
operations.

`C-u C-u' will work for the `find-file' family of commands, but it
doesn't work for `save-buffer', which uses already `C-u', `C-u C-u',
`C-u C-u C-u' and `C-u 0'.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Mon, 30 Jul 2018 10:44:07 +0200
> Cc: Filipp Gunbin <fgunbin@fastmail.fm>, emacs-devel@gnu.org
> 
> If nobody objects, I could do this. However, during the discussion it
> was proposed to find a common indication for asynchronous file
> operations.
> 
> `C-u C-u' will work for the `find-file' family of commands, but it
> doesn't work for `save-buffer', which uses already `C-u', `C-u C-u',
> `C-u C-u C-u' and `C-u 0'.

Indeed.  So I think the case for having a new prefix command akin to
"C-x RET c" becomes stronger and stronger.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> If nobody objects, I could do this. However, during the discussion it
>> was proposed to find a common indication for asynchronous file
>> operations.
>> 
>> `C-u C-u' will work for the `find-file' family of commands, but it
>> doesn't work for `save-buffer', which uses already `C-u', `C-u C-u',
>> `C-u C-u C-u' and `C-u 0'.
>
> Indeed.  So I think the case for having a new prefix command akin to
> "C-x RET c" becomes stronger and stronger.

What about the appended patch for the feature/tramp-thread-safe branch?
It introduces "C-x &" as prefix argument, toggling the meaning of
the `find-file-asynchronously' variable.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Michael Albinus]

[-- Attachment #1: Type: text/plain, Size: 277 bytes --]

Michael Albinus <michael.albinus@gmx.de> writes:

> What about the appended patch for the feature/tramp-thread-safe branch?
> It introduces "C-x &" as prefix argument, toggling the meaning of
> the `find-file-asynchronously' variable.

Now with patch.

Best regards, Michael.


[-- Attachment #2: Type: text/plain, Size: 5404 bytes --]

diff --git a/doc/emacs/files.texi b/doc/emacs/files.texi
index 46e64c2a27..e4a1369a74 100644
--- a/doc/emacs/files.texi
+++ b/doc/emacs/files.texi
@@ -250,7 +250,7 @@ Visiting
 @end group
 @end example
 
-  With a prefix argument @kbd{C-u}, the meaning of
+  With a prefix argument @kbd{C-x &}, the meaning of
 @code{find-file-asynchronously} will be reverted.  If this user option
 is @code{nil}, visiting a file is performed asynchronously.  Contrary,
 if this user option is non-@code{nil}, visiting a file is performed
diff --git a/doc/lispref/files.texi b/doc/lispref/files.texi
index 9faa47b455..54737ebe64 100644
--- a/doc/lispref/files.texi
+++ b/doc/lispref/files.texi
@@ -119,7 +119,7 @@ Visiting Functions
 If @var{async} is non-@code{nil}, the file will be loaded into the
 buffer asynchronously.  Interactively, this is indicated by either
 setting user option @code{find-file-asynchronously} to non-@code{nil},
-or by a prefix argument.
+or by a prefix argument @kbd{C-x &}.
 
 When @code{find-file} is called interactively, it prompts for
 @var{filename} in the minibuffer.
@@ -232,8 +232,8 @@ Visiting Functions
 If this variable is non-@code{nil}, a file will be visited
 asynchronously when called interactively.  If it is a regular
 expression, it must match the file name to be visited.  This behavior
-is toggled by a prefix argument to the interactive call of the file
-visiting command.
+is toggled by a prefix argument @kbd{C-x &} to the interactive call of
+the file visiting command.
 @end defopt
 
 @defopt find-file-hook
diff --git a/lisp/files.el b/lisp/files.el
index 195694a7b6..ccdbaeb315 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -1576,14 +1576,56 @@ minibuffer-with-setup-hook
              ,@body)
          (remove-hook 'minibuffer-setup-hook ,hook)))))
 
+(defun universal-file-visiting-argument (async)
+  "Execute a file visiting or saving command using the ASYNC argument.
+See `find-file-asynchronously' for valid values of ASYNC."
+  (interactive
+   (list (and (featurep 'threads) (not find-file-asynchronously))))
+  (let* ((find-file-asynchronously async)
+         (keyseq (read-key-sequence (format "Command to visit or save file:")))
+	 (cmd (key-binding keyseq))
+	 prefix)
+    ;; `read-key-sequence' ignores quit, so make an explicit check.
+    ;; Like many places, this assumes quit == C-g, but it need not be.
+    (if (equal last-input-event ?\C-g)
+	(keyboard-quit))
+    (when (memq cmd '(universal-argument digit-argument))
+      (call-interactively cmd)
+
+      ;; Process keys bound in `universal-argument-map'.
+      (while (progn
+	       (setq keyseq (read-key-sequence nil t)
+		     cmd (key-binding keyseq t))
+	       (not (eq cmd 'universal-argument-other-key)))
+	(let ((current-prefix-arg prefix-arg)
+	      ;; Have to bind `last-command-event' here so that
+	      ;; `digit-argument', for instance, can compute the
+	      ;; prefix arg.
+	      (last-command-event (aref keyseq 0)))
+	  (call-interactively cmd)))
+
+      ;; This is the final call to `universal-argument-other-key', which
+      ;; set's the final `prefix-arg.
+      (let ((current-prefix-arg prefix-arg))
+	(call-interactively cmd))
+
+      ;; Read the command to execute with the given prefix arg.
+      (setq prefix prefix-arg
+	    keyseq (read-key-sequence nil t)
+	    cmd (key-binding keyseq)))
+
+    (let ((current-prefix-arg prefix))
+      (message "")
+      (call-interactively cmd))))
+
+(define-key ctl-x-map "&" 'universal-file-visiting-argument)
+
 (defun find-file-read-args (prompt mustmatch &optional wildcards)
   "Return the interactive spec (<filename> <async>).
 If WILDCARDS is non-nil, return the spec (<filename> t <async>)."
   (let ((filename (read-file-name prompt nil default-directory mustmatch))
-        (async (and (xor find-file-asynchronously current-prefix-arg)
-                    (featurep 'threads))))
-    (when (and async (stringp find-file-asynchronously))
-      (setq async (string-match-p find-file-asynchronously filename)))
+        (async (and (featurep 'threads) find-file-asynchronously)))
+    (when (stringp async) (setq async (string-match-p async filename)))
     (if wildcards `(,filename t ,async) `(,filename ,async))))
 
 (defmacro find-file-with-threads (filename async &rest body)
@@ -1794,8 +1836,8 @@ find-alternate-file-other-window
        (list (read-file-name
 	      "Find alternate file: " file-dir nil
               (confirm-nonexistent-file-or-buffer) file-name)
-	     t (and (xor find-file-asynchronously current-prefix-arg)
-                    (featurep 'threads))))))
+	     t (and (featurep 'threads) find-file-asynchronously)))))
+  (when (stringp async) (setq async (string-match-p async filename)))
   (if (one-window-p)
       (find-file-other-window filename wildcards async)
     (save-selected-window
@@ -1839,8 +1881,8 @@ find-alternate-file
      (list (read-file-name
 	    "Find alternate file: " file-dir nil
             (confirm-nonexistent-file-or-buffer) file-name)
-	   t (and (xor find-file-asynchronously current-prefix-arg)
-                  (featurep 'threads)))))
+	   t (and (featurep 'threads) find-file-asynchronously))))
+  (when (stringp async) (setq async (string-match-p async filename)))
   (unless (run-hook-with-args-until-failure 'kill-buffer-query-functions)
     (user-error "Aborted"))
   (and (buffer-modified-p) buffer-file-name

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: drew.adams@oracle.com,  fgunbin@fastmail.fm,  emacs-devel@gnu.org
> Date: Sat, 04 Aug 2018 14:49:35 +0200
> 
> Michael Albinus <michael.albinus@gmx.de> writes:
> 
> > What about the appended patch for the feature/tramp-thread-safe branch?
> > It introduces "C-x &" as prefix argument, toggling the meaning of
> > the `find-file-asynchronously' variable.
> 
> Now with patch.

The command's name should be more general, as I believe this was
envisaged to eventually become effective for more commands than those
which visit files?

> +    ;; `read-key-sequence' ignores quit, so make an explicit check.
> +    ;; Like many places, this assumes quit == C-g, but it need not be.
> +    (if (equal last-input-event ?\C-g)

Why not use current-input-mode here?

> +      ;; This is the final call to `universal-argument-other-key', which
> +      ;; set's the final `prefix-arg.
            ^^^^^
"sets"

Thanks.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> What about the appended patch for the feature/tramp-thread-safe branch?
>> It introduces "C-x &" as prefix argument, toggling the meaning of
>> the `find-file-asynchronously' variable.
>
> The command's name should be more general, as I believe this was
> envisaged to eventually become effective for more commands than those
> which visit files?

I'm bad in finding good names. What is the superset of "file visiting
and saving"?

>> +    ;; `read-key-sequence' ignores quit, so make an explicit check.
>> +    ;; Like many places, this assumes quit == C-g, but it need not be.
>> +    (if (equal last-input-event ?\C-g)
>
> Why not use current-input-mode here?

Well, I've stolen the code from `universal-coding-system-argument' :-)
Will adapt it in my patch, and also in `universal-coding-system-argument'
in master.

>> +      ;; This is the final call to `universal-argument-other-key', which
>> +      ;; set's the final `prefix-arg.
>             ^^^^^
> "sets"

Ditto.

> Thanks.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: drew.adams@oracle.com,  fgunbin@fastmail.fm,  emacs-devel@gnu.org
> Date: Sat, 04 Aug 2018 17:12:50 +0200
> 
> >> What about the appended patch for the feature/tramp-thread-safe branch?
> >> It introduces "C-x &" as prefix argument, toggling the meaning of
> >> the `find-file-asynchronously' variable.
> >
> > The command's name should be more general, as I believe this was
> > envisaged to eventually become effective for more commands than those
> > which visit files?
> 
> I'm bad in finding good names. What is the superset of "file visiting
> and saving"?

I was actually thinking about something like universal-async-argument,
entirely unrelated to files etc.

Thanks.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> > The command's name should be more general, as I believe this was
>> > envisaged to eventually become effective for more commands than those
>> > which visit files?
>> 
>> I'm bad in finding good names. What is the superset of "file visiting
>> and saving"?
>
> I was actually thinking about something like universal-async-argument,
> entirely unrelated to files etc.

OK, But I also miss a better name for the `find-file-asynchronously'
user option.

> Thanks.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: drew.adams@oracle.com,  fgunbin@fastmail.fm,  emacs-devel@gnu.org
> Date: Sat, 04 Aug 2018 17:58:37 +0200
> 
> >> I'm bad in finding good names. What is the superset of "file visiting
> >> and saving"?
> >
> > I was actually thinking about something like universal-async-argument,
> > entirely unrelated to files etc.
> 
> OK, But I also miss a better name for the `find-file-asynchronously'
> user option.

execute-command-asynchronously?

(I'm also bad in coming up with good names.)

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> OK, But I also miss a better name for the `find-file-asynchronously'
>> user option.
>
> execute-command-asynchronously?

Sounds too general to me, because it is about *file* visiting and saving.

execute-file-command-asynchronously?

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: drew.adams@oracle.com,  fgunbin@fastmail.fm,  emacs-devel@gnu.org
> Date: Sat, 04 Aug 2018 18:18:06 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> OK, But I also miss a better name for the `find-file-asynchronously'
> >> user option.
> >
> > execute-command-asynchronously?
> 
> Sounds too general to me, because it is about *file* visiting and saving.
> 
> execute-file-command-asynchronously?

But I _wanted_ it to be general, far beyond just file-related
commands.  I though that was what was being discussed: to have a
convenient way of running a command, any command that supported it,
asynchronously.  Was I mistaken?

In any case, why should there be a separate command to force just the
file-related commands work asynchronously?

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> OK, But I also miss a better name for the `find-file-asynchronously'
>> >> user option.
>> >
>> > execute-command-asynchronously?
>> 
>> Sounds too general to me, because it is about *file* visiting and saving.
>> 
>> execute-file-command-asynchronously?
>
> But I _wanted_ it to be general, far beyond just file-related
> commands.  I though that was what was being discussed: to have a
> convenient way of running a command, any command that supported it,
> asynchronously.  Was I mistaken?

find-file-asynchronously is defined to be file-related (as of now):

--8<---------------cut here---------------start------------->8---
Documentation:
Non-nil means visit file asynchronously when called interactively.
If it is a regular expression, it must match the file name to be
visited.  This behavior is toggled by a prefix argument to the
interactive call.
--8<---------------cut here---------------end--------------->8---

> In any case, why should there be a separate command to force just the
> file-related commands work asynchronously?

I don't speak a bout a separate command. I speak about the user option,
which I have introduced for the find-file family of commands. I believe
the name find-file-asynchronously is too tight (doesn't cover save-buffer
and friends), and execute-command-asynchronously is too wide (because it
shall configure file-related command behavior, with the regexp as value).

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: drew.adams@oracle.com,  fgunbin@fastmail.fm,  emacs-devel@gnu.org
> Date: Sat, 04 Aug 2018 19:29:50 +0200
> 
> find-file-asynchronously is defined to be file-related (as of now):
> 
> --8<---------------cut here---------------start------------->8---
> Documentation:
> Non-nil means visit file asynchronously when called interactively.
> If it is a regular expression, it must match the file name to be
> visited.  This behavior is toggled by a prefix argument to the
> interactive call.
> --8<---------------cut here---------------end--------------->8---

It's okay for the variable to accept a regexp value, because it allows
to customize the variable once and then benefit from the value for
many invocations.

But "C-x &" affects just the next command, so all you need is for it
to produce a boolean value, and have the next command work
asynchronously or not according to that value.  Which probably means
"C-x &" should not bind find-file-asynchronously, but some other
variable, and that variable had better had a name that is not limited
to file operations, for us to be able to use it with other commands.
Because once we extend "C-x &" to non-file commands, it will not
"know" what command will be invoked after it, so it won't be able to
know it should bind find-file-asynchronously or something else.  It
should therefore bind a variable which affects any command.

> > In any case, why should there be a separate command to force just the
> > file-related commands work asynchronously?
> 
> I don't speak a bout a separate command. I speak about the user option,
> which I have introduced for the find-file family of commands. I believe
> the name find-file-asynchronously is too tight (doesn't cover save-buffer
> and friends), and execute-command-asynchronously is too wide (because it
> shall configure file-related command behavior, with the regexp as value).

For the variable which tailors find-file, save-buffer, etc., I agree
that it should not just say "find-file".
execute-file-commands-asynchronously sounds okay to me.  But that's
just part of the broader issue of invoking commands asynchronously.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> But "C-x &" affects just the next command, so all you need is for it
> to produce a boolean value, and have the next command work
> asynchronously or not according to that value.  Which probably means
> "C-x &" should not bind find-file-asynchronously, but some other
> variable, and that variable had better had a name that is not limited
> to file operations, for us to be able to use it with other commands.
> Because once we extend "C-x &" to non-file commands, it will not
> "know" what command will be invoked after it, so it won't be able to
> know it should bind find-file-asynchronously or something else.  It
> should therefore bind a variable which affects any command.

Nobody hinders us to bind several variables in
universal-async-argument. For the time being it toggles the value of
execute-file-commands-asynchronously, but this could be extended.

> For the variable which tailors find-file, save-buffer, etc., I agree
> that it should not just say "find-file".
> execute-file-commands-asynchronously sounds okay to me.  But that's
> just part of the broader issue of invoking commands asynchronously.

I've changed it accordingly. Pushed to the feature/tramp-thread-safe
branch, in order to have the same code basis we're speaking about.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: drew.adams@oracle.com,  fgunbin@fastmail.fm,  emacs-devel@gnu.org
> Date: Sun, 05 Aug 2018 11:51:34 +0200
> 
> > But "C-x &" affects just the next command, so all you need is for it
> > to produce a boolean value, and have the next command work
> > asynchronously or not according to that value.  Which probably means
> > "C-x &" should not bind find-file-asynchronously, but some other
> > variable, and that variable had better had a name that is not limited
> > to file operations, for us to be able to use it with other commands.
> > Because once we extend "C-x &" to non-file commands, it will not
> > "know" what command will be invoked after it, so it won't be able to
> > know it should bind find-file-asynchronously or something else.  It
> > should therefore bind a variable which affects any command.
> 
> Nobody hinders us to bind several variables in
> universal-async-argument.

If that produces expected results in practice, that'd be fine, yes.

> > For the variable which tailors find-file, save-buffer, etc., I agree
> > that it should not just say "find-file".
> > execute-file-commands-asynchronously sounds okay to me.  But that's
> > just part of the broader issue of invoking commands asynchronously.
> 
> I've changed it accordingly. Pushed to the feature/tramp-thread-safe
> branch, in order to have the same code basis we're speaking about.

Thanks.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> > But "C-x &" affects just the next command, so all you need is for it
>> > to produce a boolean value, and have the next command work
>> > asynchronously or not according to that value.  Which probably means
>> > "C-x &" should not bind find-file-asynchronously, but some other
>> > variable, and that variable had better had a name that is not limited
>> > to file operations, for us to be able to use it with other commands.
>> > Because once we extend "C-x &" to non-file commands, it will not
>> > "know" what command will be invoked after it, so it won't be able to
>> > know it should bind find-file-asynchronously or something else.  It
>> > should therefore bind a variable which affects any command.
>> 
>> Nobody hinders us to bind several variables in
>> universal-async-argument.
>
> If that produces expected results in practice, that'd be fine, yes.

I've slept another night over this, and I believe you are right. I've
pushed a patch to the branch, that the universal-async-argument function
toggles the universal-async-argument variable, and binds it before
calling the following command. This is more general, and other commands
but the file visiting ones would profit also from this mechanism w/o
touching universal-async-argument, again.

This needs to be documented better in the Elisp manual, but first we
shall agree how to call this technique. "Prefix argument" is out of the
game. Maybe "prefix command"? Or "prefix sequence", as proposed in this
thread?

Best regards, Michael.

RE: Introducing thread-safe Tramp

[Drew Adams]

> This needs to be documented better in the Elisp manual, but first we
> shall agree how to call this technique. "Prefix argument" is out of the
> game. Maybe "prefix command"? Or "prefix sequence", as proposed in this
> thread?

As Richard suggested, don't call it anything. At least do not call it any of those things (it's not a command or a prefix key (sequence)).

We can, if you want, have a separate discussion, taking our time, to decide whether and what to name this.

C-x & (was: Introducing thread-safe Tramp)

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

>> This needs to be documented better in the Elisp manual, but first we
>> shall agree how to call this technique. "Prefix argument" is out of the
>> game. Maybe "prefix command"? Or "prefix sequence", as proposed in this
>> thread?
>
> As Richard suggested, don't call it anything. At least do not call it
> any of those things (it's not a command or a prefix key (sequence)).

I'm all for an agreement. But until now I hear only how to NOT call
it. I'd prefer to see a counter proposal.

And of course, "C-x &" invokes a command. This is equivalent to "M-x
universal-async-argument". So I don't see what's wrong with "prefix
command" - it is a command which precedes another command.

> We can, if you want, have a separate discussion, taking our time, to
> decide whether and what to name this.

Subject adapted.

Best regards, Michael.

RE: C-x & (was: Introducing thread-safe Tramp)

[Drew Adams]

> And of course, "C-x &" invokes a command. 

Yes. But invoking a command is not being a command.

> This is equivalent to "M-x universal-async-argument". 

Presumably you mean that using key sequence `C-x &' is equivalent to using `M-x universal-async-argument'. That's another way of saying that `C-x &' invokes command `universal-async-argument'.

If you wanted to call command `universal-async-argument' a "prefix command" you could do so. But `C-x &' is not a prefix command or any other kind of command.

> So I don't see what's wrong with "prefix command" - it is a command which precedes another command.

Command `universal-async-argument', when invoked, precedes invocation of the command bound to the key sequence read by `universal-asyn-argument'. Yes.

And using `C-x &' precedes reading of that second key sequence. But neither of those things make `C-x &' into a command, prefix, postfix, or *fix.

> > We can, if you want, have a separate discussion, taking our time, to
> > decide whether and what to name this.
> 
> Subject adapted.

Thx.

Re: C-x &

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

>> And of course, "C-x &" invokes a command. 
>
> Yes. But invoking a command is not being a command.

Yes. I haven't said otherwise, at least in my last email.

>> This is equivalent to "M-x universal-async-argument". 
>
> Presumably you mean that using key sequence `C-x &' is equivalent to
> using `M-x universal-async-argument'. That's another way of saying
> that `C-x &' invokes command `universal-async-argument'.
>
> If you wanted to call command `universal-async-argument' a "prefix
> command" you could do so. But `C-x &' is not a prefix command or any
> other kind of command.

I haven't said that `C-x &' is any kind of command, at least in my last
email.

>> So I don't see what's wrong with "prefix command" - it is a command
>> which precedes another command.
>
> Command `universal-async-argument', when invoked, precedes invocation
> of the command bound to the key sequence read by
> `universal-asyn-argument'. Yes.
>
> And using `C-x &' precedes reading of that second key sequence. But
> neither of those things make `C-x &' into a command, prefix, postfix,
> or *fix.

Again, I haven't said that `C-x &' is a command, in my last email.

Let's agree the obvious: `universal-async-argument' is a command, by
default bound to the key sequences `C-x &' and `M-x  u n i v e r s a l -
a s y n c - a r g u m e n t RET.'

The only raison d'être of `universal-async-argument' is to read another
key sequence, which invokes the command bound to that key sequence. This
means, the command `universal-async-argument' always precedes another
command, which I call a prefix in my understanding. Therefore, I've
proposed to call `universal-async-argument' a "prefix command".

The Emacs manual distinguishes between commands and the key (sequence) a
command is bound to. But it explicitely says, that both could be used
equal:

--8<---------------cut here---------------start------------->8---
   In this manual, we will often speak of keys like ‘C-n’ as commands,
even though strictly speaking the key is bound to a command.
--8<---------------cut here---------------end--------------->8---

See (info "(emacs) Commands")

So it would be also allowed to speak about the key sequence `C-x &' as a
command, but I don't insist on this if it makes you feel bad.

Best regards, Michael.

RE: C-x &

[Drew Adams]

> Yes. I haven't said otherwise, at least in my last email.
> 
> > If you wanted to call command `universal-async-argument' a "prefix
> > command" you could do so. But `C-x &' is not a prefix command or any
> > other kind of command.
> 
> I haven't said that `C-x &' is any kind of command, at least in my last
> email.

Right. You didn't. But some others did. And it wasn't obvious (to me) that you didn't mean to say that `C-x &' is a "prefix command". If you meant only that `universal-sync-argument' is a prefix command then, as I said, that's OK by me. (The doc should then define "prefix command".)

> >> So I don't see what's wrong with "prefix command" - it is a command
> >> which precedes another command.
> >
> > Command `universal-async-argument', when invoked, precedes invocation
> > of the command bound to the key sequence read by
> > `universal-asyn-argument'. Yes.
> >
> > And using `C-x &' precedes reading of that second key sequence. But
> > neither of those things make `C-x &' into a command, prefix, postfix,
> > or *fix.
> 
> Again, I haven't said that `C-x &' is a command, in my last email.
> 
> Let's agree the obvious: `universal-async-argument' is a command, by
> default bound to the key sequences `C-x &' and `M-x  u n i v e r s a l -
> a s y n c - a r g u m e n t RET.'

Agreed.

(Although I wouldn't say that the command is bound to `M-x...'. That sequence of keys invokes `u-a-a', but the command is not bound to it. We just don't say that kind of thing in Emacs. The key sequences bound to a command are well defined, and they don't include `M-x' followed by keys spelling out the command name. Similarly, the command is not bound to `M-x uni-a-arg TAB RET' etc.)

> The only raison d'être of `universal-async-argument' is to read another
> key sequence, which invokes the command bound to that key sequence. This
> means, the command `universal-async-argument' always precedes another
> command, which I call a prefix in my understanding. 

Agreed. (Although the command that is preceded by `u-a-a' is, if anything, a suffix of it, not a prefix. I suspect you just misspoke in the last phrase.)

> Therefore, I've proposed to call `universal-async-argument' a "prefix command".

OK by me (where "it" is the command, not `C-x &'). "Prefix command" then needs to be described/defined/explained in the manual.

> The Emacs manual distinguishes between commands and the key (sequence) a
> command is bound to. But it explicitely says, that both could be used
> equal:
> 
> --8<---------------cut here---------------start------------->8---
>    In this manual, we will often speak of keys like ‘C-n’ as commands,
> even though strictly speaking the key is bound to a command.
> --8<---------------cut here---------------end--------------->8---
> 
> See (info "(emacs) Commands")

Yeah. I was afraid someone might quote that passage. ;-) 

I don't have a problem with that text, because its purpose is to state that we sometimes speak that way even though it is not really correct. It's not defining a key sequence as the command that it's bound to - quite the opposite. It both (1) points out that a key sequence is not the same thing as the command that it's bound to and (2) mentions that sometimes we gloss over that difference, for brevity and when the context makes the meaning clear.

> So it would be also allowed to speak about the key sequence `C-x &' as a
> command, but I don't insist on this if it makes you feel bad.

It would be allowed to say "command `C-x &'", yes, for brevity. That's a far cry from defining a new term "prefix command" and applying it to `C-x &'. (Especially since there is also a notion of prefix key, which is something quite different.)

In sum, I don't have a problem with calling `universal-async-argument' a "prefix command", if that term is defined in the manual.

----

About the command name (FWIW):

I'm not crazy about the name "universal-async-argument'. I realize that it's modeled on `universal-argument', and I can live with it. But I wish that both of those commands were named differently. (No, I'm not proposing a name change for `u-a' at this late date.)

`u-a-a' doesn't necessarily provide a (non-default) prefix arg to a command. It invokes a command (bound to a key sequence that it reads).

It's somewhat akin to `execute-extended-command' or `comint-redirect-send-command'. Those read a command name and not a key sequence, so in that respect they differ. But I think it might be better for `u-a-a' to be named `<something>-command', where <something> does not suggest "universal argument".

Unlike `u-a', you need not provide a prefix arg to `u-a-a'. It's not really the same kind of command. `u-a-a' does accept and take into account a prefix arg, if you provide one; but it's not required (IIUC). (Maybe I'm wrong about this. What prefix arg does bare `C-x &' provide? Does it provide `(4)', as `C-u' does?

No, I don't feel strongly about the command name. Yes, I understand why you proposed "universal-async-argument". But I think its association with `universal-argument' confuses more than clarifies. No, I don't have a great suggestion for the command name. Ideally it should convey "read a key sequence and invoke the command bound to it" (and the doc string would explain that it will also read a prefix arg and pass that to the command). Maybe `invoke-command-bound-to' or some such.

RE: C-x &

[Drew Adams]

> No, I don't feel strongly about the command name. Yes, I understand why
> you proposed "universal-async-argument". But I think its association with
> `universal-argument' confuses more than clarifies. No, I don't have a great
> suggestion for the command name. Ideally it should convey "read a key
> sequence and invoke the command bound to it" (and the doc string would
> explain that it will also read a prefix arg and pass that to the command).
> Maybe `invoke-command-bound-to' or some such.

Sorry; I meant to mention asynchronous execution.

> Ideally it should convey "read a key
> sequence and invoke the command bound to it

, in an asynchronous way"

> (and the doc string would
> explain that it will also read a prefix arg and pass that to the command).
> Maybe `invoke-command-bound-to' or some such.

But the command name should have `async' in it, somewhere.

Re: C-x &

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

>> No, I don't feel strongly about the command name. Yes, I understand why
>> you proposed "universal-async-argument". But I think its association with
>> `universal-argument' confuses more than clarifies. No, I don't have a great
>> suggestion for the command name. Ideally it should convey "read a key
>> sequence and invoke the command bound to it" (and the doc string would
>> explain that it will also read a prefix arg and pass that to the command).
>> Maybe `invoke-command-bound-to' or some such.
>
> Sorry; I meant to mention asynchronous execution.

The name "universal-async-argument" was inherited from
"universal-coding-system-argument", which implements a similar technique.

>> Ideally it should convey "read a key
>> sequence and invoke the command bound to it
>
> , in an asynchronous way"
>
>> (and the doc string would
>> explain that it will also read a prefix arg and pass that to the command).
>> Maybe `invoke-command-bound-to' or some such.
>
> But the command name should have `async' in it, somewhere.

"universal-async-argument" has it. And until we have a better name, I'd
like to keep it. It will be easy to replace the name by another one if
needed, and it isn't user visible in general.

Best regards, Michael.

Re: C-x &

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

>> I haven't said that `C-x &' is any kind of command, at least in my last
>> email.
>
> Right. You didn't. But some others did. And it wasn't obvious (to me)
> that you didn't mean to say that `C-x &' is a "prefix command". If you
> meant only that `universal-sync-argument' is a prefix command then, as
> I said, that's OK by me. (The doc should then define "prefix
> command".)
>
>> Again, I haven't said that `C-x &' is a command, in my last email.
>> 
>> Let's agree the obvious: `universal-async-argument' is a command, by
>> default bound to the key sequences `C-x &' and `M-x  u n i v e r s a l -
>> a s y n c - a r g u m e n t RET.'
>
> Agreed.
>
> (Although I wouldn't say that the command is bound to `M-x...'. That
> sequence of keys invokes `u-a-a', but the command is not bound to
> it. We just don't say that kind of thing in Emacs. The key sequences
> bound to a command are well defined, and they don't include `M-x'
> followed by keys spelling out the command name. Similarly, the command
> is not bound to `M-x uni-a-arg TAB RET' etc.)
>
>> The only raison d'être of `universal-async-argument' is to read another
>> key sequence, which invokes the command bound to that key sequence. This
>> means, the command `universal-async-argument' always precedes another
>> command, which I call a prefix in my understanding. 
>
> Agreed. (Although the command that is preceded by `u-a-a' is, if
> anything, a suffix of it, not a prefix. I suspect you just misspoke in
> the last phrase.)
>
>> Therefore, I've proposed to call `universal-async-argument' a
>> "prefix command".
>
> OK by me (where "it" is the command, not `C-x &'). "Prefix command"
> then needs to be described/defined/explained in the manual.

OMG: we are in violent agreement :-)

>> The Emacs manual distinguishes between commands and the key (sequence) a
>> command is bound to. But it explicitely says, that both could be used
>> equal:
>> 
>> --8<---------------cut here---------------start------------->8---
>>    In this manual, we will often speak of keys like ‘C-n’ as commands,
>> even though strictly speaking the key is bound to a command.
>> --8<---------------cut here---------------end--------------->8---
>> 
>> See (info "(emacs) Commands")
>
> Yeah. I was afraid someone might quote that passage. ;-) 

That's because of *you* - you have forced me to consult the manual about.

> I don't have a problem with that text, because its purpose is to state
> that we sometimes speak that way even though it is not really
> correct. It's not defining a key sequence as the command that it's
> bound to - quite the opposite. It both (1) points out that a key
> sequence is not the same thing as the command that it's bound to and
> (2) mentions that sometimes we gloss over that difference, for brevity
> and when the context makes the meaning clear.
>
>> So it would be also allowed to speak about the key sequence `C-x &' as a
>> command, but I don't insist on this if it makes you feel bad.
>
> It would be allowed to say "command `C-x &'", yes, for brevity. That's
> a far cry from defining a new term "prefix command" and applying it to
> `C-x &'. (Especially since there is also a notion of prefix key, which
> is something quite different.)
>
> In sum, I don't have a problem with calling `universal-async-argument'
> a "prefix command", if that term is defined in the manual.

Agreed.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Stefan Monnier]

> As Richard suggested, don't call it anything. At least do not call it any of
> those things (it's not a command or a prefix key (sequence)).

It's definitely a command and it is used before another, so "prefix
command" sounds about right.


        Stefan

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: drew.adams@oracle.com,  fgunbin@fastmail.fm,  emacs-devel@gnu.org
> Date: Mon, 06 Aug 2018 12:54:07 +0200
> 
> This needs to be documented better in the Elisp manual, but first we
> shall agree how to call this technique. "Prefix argument" is out of the
> game. Maybe "prefix command"? Or "prefix sequence", as proposed in this
> thread?

My favorite so far is "prefix command".

Thanks.

C-x & (was: Introducing thread-safe Tramp)

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> This needs to be documented better in the Elisp manual, but first we
>> shall agree how to call this technique. "Prefix argument" is out of the
>> game. Maybe "prefix command"? Or "prefix sequence", as proposed in this
>> thread?
>
> My favorite so far is "prefix command".

So we are two - see my other message I've just sent, before reading your
message.

> Thanks.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Filipp Gunbin]

Thanks for your work on the thread-safe Tramp and async feature!

Minor comments for the current branch:

files.texi:

+If this option is @code{nil} (the default), visiting a file is
+performed synchronously.  A regexp value let files, which name matches
                                          ^^^
                                          lets

NEWS:

+If the new user option 'execute-file-commands-asynchronously' has a
+proper non-nil value, interactive file visiting commands load the file
+asynchronously into the respective buffer.  I.e., Emacs is still

It'd be useful to elaborate a bit on what is 'proper'.  The case of
regexp value is particularly important for users to know, I think.  This
is present in variable docstring, yes, but not here.


lisp/simple.el:

+(defvar universal-async-argument nil
+  "Non-nil indicates a command to run asynchronously when called interactively.
                      ^^^^^^^^^^^^
                      that a command should?


+(defun universal-async-argument ()
+  "Execute an interactive command asynchronously."
[..]
+    (when (memq cmd '(universal-argument digit-argument))
+      (call-interactively cmd)

What about negative-argument?


Filipp

Re: Introducing thread-safe Tramp

[Michael Albinus]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

Hi Filipp,

Thanks for your comments!

> files.texi:
>
> +If this option is @code{nil} (the default), visiting a file is
> +performed synchronously.  A regexp value let files, which name matches
>                                           ^^^
>                                           lets

Fixed.

> NEWS:
>
> +If the new user option 'execute-file-commands-asynchronously' has a
> +proper non-nil value, interactive file visiting commands load the file
> +asynchronously into the respective buffer.  I.e., Emacs is still
>
> It'd be useful to elaborate a bit on what is 'proper'.  The case of
> regexp value is particularly important for users to know, I think.  This
> is present in variable docstring, yes, but not here.

Adapted.

> lisp/simple.el:
>
> +(defvar universal-async-argument nil
> +  "Non-nil indicates a command to run asynchronously when called interactively.
>                       ^^^^^^^^^^^^
>                       that a command should?

Not "should". It is just a hint that a user wishes the next command to
run asynchronously. This could be prefixed for *every* command; whether
that command cares about the indication is up to the command itself.

If you mean just better English - OK. But the first sentence of a
docstring must fit into one line.

> +(defun universal-async-argument ()
> +  "Execute an interactive command asynchronously."
> [..]
> +    (when (memq cmd '(universal-argument digit-argument))
> +      (call-interactively cmd)
>
> What about negative-argument?

Good point. Fixed.

> Filipp

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Stephen Berman]

On Tue, 07 Aug 2018 16:20:54 +0200 Michael Albinus <michael.albinus@gmx.de> wrote:

> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>
> Hi Filipp,
>
> Thanks for your comments!
>
>> files.texi:
>>
>> +If this option is @code{nil} (the default), visiting a file is
>> +performed synchronously.  A regexp value let files, which name matches
>>                                           ^^^
>>                                           lets
                                                      ^ and no comma here

>> +(defvar universal-async-argument nil
>> +  "Non-nil indicates a command to run asynchronously when called interactively.
>>                       ^^^^^^^^^^^^
>>                       that a command should?
>
> Not "should". It is just a hint that a user wishes the next command to
> run asynchronously. This could be prefixed for *every* command; whether
> that command cares about the indication is up to the command itself.
>
> If you mean just better English - OK. But the first sentence of a
> docstring must fit into one line.

How about this: 
"Non-nil means the next interactively called command runs asynchronously."
Or since commands are really interactively called functions, just this:
"Non-nil means the next command runs asynchronously."

Steve Berman

Re: Introducing thread-safe Tramp

[Robert Pluim]

Stephen Berman <stephen.berman@gmx.net> writes:

> On Tue, 07 Aug 2018 16:20:54 +0200 Michael Albinus <michael.albinus@gmx.de> wrote:
>
>> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>>
>> Hi Filipp,
>>
>> Thanks for your comments!
>>
>>> files.texi:
>>>
>>> +If this option is @code{nil} (the default), visiting a file is
>>> +performed synchronously.  A regexp value let files, which name matches
>>>                                           ^^^
>>>                                           lets
>                                                       ^ and no comma here
>
>>> +(defvar universal-async-argument nil
>>> +  "Non-nil indicates a command to run asynchronously when called interactively.
>>>                       ^^^^^^^^^^^^
>>>                       that a command should?
>>
>> Not "should". It is just a hint that a user wishes the next command to
>> run asynchronously. This could be prefixed for *every* command; whether
>> that command cares about the indication is up to the command itself.
>>
>> If you mean just better English - OK. But the first sentence of a
>> docstring must fit into one line.
>
> How about this: 
> "Non-nil means the next interactively called command runs asynchronously."
> Or since commands are really interactively called functions, just this:
> "Non-nil means the next command runs asynchronously."

"Non-nil means the next command is requested to run asynchronously."

The command in question might not have async support.

Re: Introducing thread-safe Tramp

[Stephen Berman]

On Tue, 07 Aug 2018 17:02:41 +0200 Robert Pluim <rpluim@gmail.com> wrote:

> Stephen Berman <stephen.berman@gmx.net> writes:
>
>> On Tue, 07 Aug 2018 16:20:54 +0200 Michael Albinus <michael.albinus@gmx.de> wrote:
>>
>>> Filipp Gunbin <fgunbin@fastmail.fm> writes:
>>>
>>> Hi Filipp,
>>>
>>> Thanks for your comments!
>>>
>>>> files.texi:
>>>>
>>>> +If this option is @code{nil} (the default), visiting a file is
>>>> +performed synchronously.  A regexp value let files, which name matches
>>>>                                           ^^^
>>>>                                           lets
>>                                                       ^ and no comma here
>>
>>>> +(defvar universal-async-argument nil
>>>> +  "Non-nil indicates a command to run asynchronously when called interactively.
>>>>                       ^^^^^^^^^^^^
>>>>                       that a command should?
>>>
>>> Not "should". It is just a hint that a user wishes the next command to
>>> run asynchronously. This could be prefixed for *every* command; whether
>>> that command cares about the indication is up to the command itself.
>>>
>>> If you mean just better English - OK. But the first sentence of a
>>> docstring must fit into one line.
>>
>> How about this: 
>> "Non-nil means the next interactively called command runs asynchronously."
>> Or since commands are really interactively called functions, just this:
>> "Non-nil means the next command runs asynchronously."
>
> "Non-nil means the next command is requested to run asynchronously."
>
> The command in question might not have async support.

Good point.  Or, since passive is discouraged in the Emacs doc:

"Non-nil means the next command runs asynchronously, if possible."

Steve Berman

Re: Introducing thread-safe Tramp

[Michael Albinus]

Stephen Berman <stephen.berman@gmx.net> writes:

>> "Non-nil means the next command is requested to run asynchronously."
>>
>> The command in question might not have async support.
>
> Good point.  Or, since passive is discouraged in the Emacs doc:
>
> "Non-nil means the next command runs asynchronously, if possible."

Better. But I prefer to be more precise about the "if possible". It
should be rather something like "if the command cares about asynchronous
behavior" (my not so good English).

> Steve Berman

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Robert Pluim <rpluim@gmail.com> writes:

>> How about this: 
>> "Non-nil means the next interactively called command runs asynchronously."
>> Or since commands are really interactively called functions, just this:
>> "Non-nil means the next command runs asynchronously."
>
> "Non-nil means the next command is requested to run asynchronously."

No for the same reason as said already.

> The command in question might not have async support.

Indeed. And that's why I believe "requested" is the wrong word. I've
tried it with "... is indicated ..."

Best regards, Michael.

RE: Introducing thread-safe Tramp

[Drew Adams]

> How about this:
> "Non-nil means the next interactively called command runs asynchronously."
> Or since commands are really interactively called functions, just this:
> "Non-nil means the next command runs asynchronously."

Non-nil means run the next command asynchronously.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

>> How about this:
>> "Non-nil means the next interactively called command runs asynchronously."
>> Or since commands are really interactively called functions, just this:
>> "Non-nil means the next command runs asynchronously."
>
> Non-nil means run the next command asynchronously.

No. As said, most of the commands will ignore "C-x &".

Best regards, Michael.

RE: Introducing thread-safe Tramp

[Drew Adams]

>> Non-nil means run the next command asynchronously.
>
> No, not every command runs asynchronously. ...
> Every command can decide itself 

Non-nil means run the next command asynchronously, if that command allows it.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

> Non-nil means run the next command asynchronously, if that command allows it.

Thanks, this is as good as the proposal of Eli. Let's wait until
tomorrow, and decide then.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Stephen Berman <stephen.berman@gmx.net> writes:

>>> +If this option is @code{nil} (the default), visiting a file is
>>> +performed synchronously.  A regexp value let files, which name matches
>>>                                           ^^^
>>>                                           lets
>                                                       ^ and no comma here

But shouldn't then both commas be removed?

"A regexp value lets files which name matches the regexp being visited
asynchronously, ..."

Is this really better?

> How about this:
> "Non-nil means the next interactively called command runs asynchronously."
> Or since commands are really interactively called functions, just this:
> "Non-nil means the next command runs asynchronously."

No, not every command runs asynchronously. "C-x & C-x C-b" calls
`list-buffers' ("C-x C-b"), but this isn't performed
asynchronously. Every command can decide itself whether it reacts on the
"C-x &" prefix command; for now these are the file visiting
commands. More shall follow.

(Yes, Drew, I know that the prefix command is `universal-async-argument'.
Don't let us discuss it, again :-)

> Steve Berman

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Filipp Gunbin <fgunbin@fastmail.fm>,  drew.adams@oracle.com,  emacs-devel@gnu.org
> Date: Tue, 07 Aug 2018 17:56:27 +0200
> 
> >>> +If this option is @code{nil} (the default), visiting a file is
> >>> +performed synchronously.  A regexp value let files, which name matches
> >>>                                           ^^^
> >>>                                           lets
> >                                                       ^ and no comma here
> 
> But shouldn't then both commas be removed?

Yes.

> "A regexp value lets files which name matches the regexp being visited
> asynchronously, ..."       ^^^^^

"whose", not "which"

I'd actually reword slightly:

  A regexp value means visit asynchronously any file whose name
  matches the regexp.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> I'd actually reword slightly:
>
>   A regexp value means visit asynchronously any file whose name
>   matches the regexp.

Thanks. I've meanwhile committed Stephen's proposal, which sounds also
good to me.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  drew.adams@oracle.com,  emacs-devel@gnu.org
> Date: Tue, 07 Aug 2018 16:20:54 +0200
> 
> > +(defvar universal-async-argument nil
> > +  "Non-nil indicates a command to run asynchronously when called interactively.
> >                       ^^^^^^^^^^^^
> >                       that a command should?
> 
> Not "should". It is just a hint that a user wishes the next command to
> run asynchronously. This could be prefixed for *every* command; whether
> that command cares about the indication is up to the command itself.
> 
> If you mean just better English - OK. But the first sentence of a
> docstring must fit into one line.

Like this?

      If non-nil, the current command should run asynchronously.
  This variable has effect only when commands are invoked interactively.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> > +(defvar universal-async-argument nil
>> > + "Non-nil indicates a command to run asynchronously when called
>> > interactively.
>> >                       ^^^^^^^^^^^^
>> >                       that a command should?
>> 
>> Not "should". It is just a hint that a user wishes the next command to
>> run asynchronously. This could be prefixed for *every* command; whether
>> that command cares about the indication is up to the command itself.
>> 
>> If you mean just better English - OK. But the first sentence of a
>> docstring must fit into one line.
>
> Like this?
>
>       If non-nil, the current command should run asynchronously.
>   This variable has effect only when commands are invoked interactively.

Sounds good. If nobody comes with a better proposal until tomorrow, I'll
apply this.

Thanks, and best regards, Michael.

RE: Introducing thread-safe Tramp

[Drew Adams]

> >       If non-nil, the current command should run asynchronously.
> >   This variable has effect only when commands are invoked interactively.
> 
> Sounds good. If nobody comes with a better proposal until tomorrow, I'll
> apply this.

  Non-nil means a command invoked interactively can run asynchronously, if the command allows it.

But that's too long, so maybe:

  Non-nil means a command invoked interactively can run asynchronously.
  It is run asynchronously only if the command also allows that.

This doc string shouldn't talk about "current" command. And it shouldn't say "should". It should just say that when this variable is non-nil and a command is invoked interactively, the command can be run asynchronously if it wants to.

RE: Introducing thread-safe Tramp

[Drew Adams]

I wonder about this doc (doc strings, at least) talking about running a command asynchronously.

I was going to suggest that it instead talk about using a separate thread, but you clarified that this is about async more generally, not just threads (in the future, at least).

Still, the current uses of "async" in Emacs generally are about running shell commands, not running Emacs commands. So there's a possible source of confusion here, if we just say "run the command asynchronously".

I'm thinking we should say "Emacs command" up front, not just "command".

Re: Introducing thread-safe Tramp

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

> I'm thinking we should say "Emacs command" up front, not just "command".

Thanks, I will apply it where it isn't obvious.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> Date: Tue, 7 Aug 2018 09:46:33 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: fgunbin@fastmail.fm, emacs-devel@gnu.org
> 
> I wonder about this doc (doc strings, at least) talking about running a command asynchronously.

It's not about _running_ a command asynchronously -- we don't really
have such features.  It's about _asking_ the command _itself_ to do
its job asynchronously.

> I'm thinking we should say "Emacs command" up front, not just "command".

We use "command" in our manuals to mean "Emacs commands".

Re: Introducing thread-safe Tramp

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

>   Non-nil means a command invoked interactively can run asynchronously.
>   It is run asynchronously only if the command also allows that.

Finally, I have decided for this. Thanks again to everybody who has
contributed a proposal.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Filipp Gunbin]

On 07/08/2018 16:20 +0200, Michael Albinus wrote:

>> lisp/simple.el:
>>
>> +(defvar universal-async-argument nil
>> +  "Non-nil indicates a command to run asynchronously when called interactively.
>>                       ^^^^^^^^^^^^
>>                       that a command should?
>
> Not "should". It is just a hint that a user wishes the next command to
> run asynchronously. This could be prefixed for *every* command; whether
> that command cares about the indication is up to the command itself.
>
> If you mean just better English - OK. But the first sentence of a
> docstring must fit into one line.

This was more about English, yes.  The original sentence read as the
setting indicated *a command*, while it actually indicates *that a
command run asynchronously*.

Thanks.

Re: Introducing thread-safe Tramp

[Stefan Monnier]

>>> +(defvar universal-async-argument nil
>>> +  "Non-nil indicates a command to run asynchronously when called interactively.
>>>                       ^^^^^^^^^^^^
>>>                       that a command should?

How 'bout

   "... tells a command ..."


-- Stefan

RE: Introducing thread-safe Tramp

[Drew Adams]

> >>> +(defvar universal-async-argument nil
> >>> +  "Non-nil ...
>
> How 'bout
> 
>    "... tells a command ..."

Yes. But apparently it has to be "asks a command", since it is apparently the command that decides. ;-)

RE: Introducing thread-safe Tramp

[Drew Adams]

Hm; interesting. Not sure I've seen this kind of thing before.

I don't really think we should call `C-x &' here a "prefix argument", in any case. For one thing, that muddies the waters for each other time we talk about "prefix argument". 

Not sure how we should talk about this.  It's kind of like a prefix key, but its binding is not a keymap but a command that reads a key sequence and invokes that key's command with particular variable binding in effect. 

A command that dispatches to other commands, perhaps binding controlling variables is an old idea. But in this case you've made use of reading a second key sequence, with its own prefix arg, and involved that with the dispatching logic.

If we're going to do this then it really invites generalization, or at least copycat uses with other such kinda-prefix-keys. The handling of a subsequent prefix arg (for the followup command), in particular, seems something generally useful.

And then there's the possibility of, say, using a prefix arg with `C-x &' itself - to do whatever (temporarily turn on a mode, enabling a given keymap for control of the followup key reading,...). Lots of crazy possibilities.

What's the real reason for doing this, as the answer to the question of how to provide keys for async file commands? 

We will now have the variable, which users can set to get or not get async behavior for file commands by default. I questioned whether we should "spend" a prefix argument on flipping the variable value during the use of a particular command. And you pointed out that some commands that we want to be able to give asyncness already use a prefix arg for other things. So using a normal prefix arg was apparently dropped. (Thank you for that.)

But is this the best way to offer users commands/keys that provide both sync and async file commands? 

I suggested just duplicating the existing commands, which lets users themselves define any number of commands that combine pairs of sync/async commands, e.g., letting a prefix arg decide the behavior (one way or the other). E.g., a user can define a `my-find-file' command that uses a prefix arg to do sync/async, and s?he can remap `find-file' to that.

If necessary, Emacs could provide such sync/async pair commands for some or all of the file commands, so users don't need to define them (not that that's hard to do). But should such new commands usurp the keys of the standard commands now, or should we just (for now, at least) let users remap as desired?

Not sure what to think about the `C-x &' proposal. It certainly is interesting, though - seems even generally promising. We've never taken such an approach before, have we (dunno)? Is there a new need? Or is this discovery of a helpful, general new feature? Seems like the latter. (Could be both.) Perhaps this deserves some new terminology and general doc.

(Someone will probably point out that this is nothing special/new, and that I just haven't had enough coffee this morning...)

Re: Introducing thread-safe Tramp

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

> What's the real reason for doing this, as the answer to the question
> of how to provide keys for async file commands?
>
> We will now have the variable, which users can set to get or not get
> async behavior for file commands by default. I questioned whether we
> should "spend" a prefix argument on flipping the variable value during
> the use of a particular command. And you pointed out that some
> commands that we want to be able to give asyncness already use a
> prefix arg for other things. So using a normal prefix arg was
> apparently dropped. (Thank you for that.)
>
> But is this the best way to offer users commands/keys that provide
> both sync and async file commands?

The variable sets a default. But sometimes, I want to visit a file sync
or async the other way the default tells. It is a simple approach to
overwrite the default ad-hoc, w/o fiddling with the variable customization.

And it has the advantage that it works both "C-x & C-x C-f ..." and
"C-x & M-x find-file ...". No new key bindings, no new commands. I
regard this as essential advantage, given my poor memory muscle ...

> Not sure what to think about the `C-x &' proposal. It certainly is
> interesting, though - seems even generally promising. We've never
> taken such an approach before, have we (dunno)? Is there a new need?
> Or is this discovery of a helpful, general new feature? Seems like the
> latter. (Could be both.) Perhaps this deserves some new terminology
> and general doc.

I propose we just start with it, and see how it works. Emacs 27 is years
away, there's much time for improvement.

> (Someone will probably point out that this is nothing special/new, and
> that I just haven't had enough coffee this morning...)

Indeed, you miss coffee :-) I have stolen this from
`universal-coding-system-argument' ("C-x <RET> c ..."), as suggested by Eli.

Best regards, Michael.

RE: Introducing thread-safe Tramp

[Drew Adams]

> The variable sets a default. But sometimes, I want to visit a file sync
> or async the other way the default tells. It is a simple approach to
> overwrite the default ad-hoc, w/o fiddling with the variable customization.

I understood that. That was the starting point (with your initial desire to use a prefix arg with the standard file commands), and it's a common use of a prefix arg. I agree with such toggling. (In fact, I'm in favor of even toggling user options this way, which is considered a no-no here.)

> And it has the advantage that it works both "C-x & C-x C-f ..." and
> "C-x & M-x find-file ...". No new key bindings, no new commands. I
> regard this as essential advantage, given my poor memory muscle ...

I understood that.

> > Not sure what to think about the `C-x &' proposal. It certainly is
> > interesting, though - seems even generally promising. We've never
> > taken such an approach before, have we (dunno)? Is there a new need?
> > Or is this discovery of a helpful, general new feature? Seems like the
> > latter. (Could be both.) Perhaps this deserves some new terminology
> > and general doc.
> 
> I propose we just start with it, and see how it works. Emacs 27 is years
> away, there's much time for improvement.

You didn't answer wrt why we shouldn't just define additional commands and let users create async/sync toggle commands using them (and binding them to the same muscle-memory keys), or why we shouldn't also create such toggle commands (toggling with a prefix arg), but not necessarily bind them to the standard keys right away. IOW, what we usually do.

> > (Someone will probably point out that this is nothing special/new, and
> > that I just haven't had enough coffee this morning...)
> 
> Indeed, you miss coffee :-) I have stolen this from
> `universal-coding-system-argument' ("C-x <RET> c ..."), as suggested by Eli.

Right. But this is still a new use case, I think. The `C-x RET c' case is pretty much the only other use of this technique, AFAIK.

We document the use of prefix keys, i.e., binding a key to a keymap. We don't document this technique/cliche. Perhaps we should.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

>> I propose we just start with it, and see how it works. Emacs 27 is years
>> away, there's much time for improvement.
>
> You didn't answer wrt why we shouldn't just define additional commands
> and let users create async/sync toggle commands using them (and
> binding them to the same muscle-memory keys), or why we shouldn't also
> create such toggle commands (toggling with a prefix arg), but not
> necessarily bind them to the standard keys right away. IOW, what we
> usually do.

Because I believe this is more convenient. It is common practice to
change the behavior of a command by a prefix, and this is what "C-x &"
is good for. It *is* a prefix; just the implementation varies.

> Right. But this is still a new use case, I think. The `C-x RET c' case
> is pretty much the only other use of this technique, AFAIK.
>
> We document the use of prefix keys, i.e., binding a key to a
> keymap. We don't document this technique/cliche. Perhaps we should.

Agreed. Now, we have two special cases. I agree also with you to make
this technique more general, but since I'm in the muddy waters of
threads, I'd like to postpone it a bit (or somebody else does). If we
have the more general approach, it deserves documentation.

Best regards, Michael.

RE: Introducing thread-safe Tramp

[Drew Adams]

> > You didn't answer wrt why we shouldn't just define additional commands
> > and let users create async/sync toggle commands using them (and
> > binding them to the same muscle-memory keys), or why we shouldn't also
> > create such toggle commands (toggling with a prefix arg), but not
> > necessarily bind them to the standard keys right away. IOW, what we
> > usually do.
> 
> Because I believe this is more convenient. 

I understand that you feel that. Why not find that out from users, over a period of time? If users end up binding the toggle commands I described, in place of the traditional file-operation bindings, then we'll know just how important and convenient such dispatching is. If only some users do that, and with only one or a couple of the file commands, then we'll have info to direct how best to accommodate that need.

Just a suggestion.

> It is common practice to change the behavior of a command by a prefix, 

What does "a prefix" even mean, here? What's prefixing what?

It's common practice to sometimes change command behavior according to a _prefix argument_. There's no prefix argument involved here (except possibly before the second key sequence - the one that is read by `C-x &'). Certainly `C-x &' is not, itself, a prefix argument. 

Nor is there any _prefix key_ involved here. `C-x &' is not bound to a keymap. I don't see how `C-x &' is a prefix <anything>.

And I don't see any common practice involved here. Rather, there is only one such use case I'm aware of: `C-x RET c'.

> and this is what "C-x &" is good for. It *is* a prefix; 

Again, what does "a prefix" even mean? `C-x &' is a key sequence. What is it a prefix of? It's not a _prefix key_ (it's not bound to a keymap). What can a key sequence be a prefix of, if not of a set of key sequences, if it is not a prefix key? 

`C-x &' is a key sequence that's bound to a command that reads a key sequence and invokes that second key sequence's command. And when reading the key sequence it allows use of prefix argument. But in what way is key sequence `C-x &' a "prefix" of something - what's that something that it prefixes?

If I type the key `a' and then I type the key `b', `a' is not a prefix of `b' in any usual sense. The behavior you have does not fit any of the current uses of the word "prefix" in Emacs jargon/doc, I think.

If you try to think of `C-x &' as some (new) kind of "prefix" then please start by saying what it is a prefix of.

> just the implementation varies.

Implementation of what? The answer, from your statement, can only be "a prefix". But implementation of a prefix of what? What is this "a prefix" you speak of?

> > Right. But this is still a new use case, I think. The `C-x RET c' case
> > is pretty much the only other use of this technique, AFAIK.
> >
> > We document the use of prefix keys, i.e., binding a key to a
> > keymap. We don't document this technique/cliche. Perhaps we should.
> 
> Agreed. Now, we have two special cases. I agree also with you to make
> this technique more general, but since I'm in the muddy waters of
> threads, I'd like to postpone it a bit (or somebody else does). If we
> have the more general approach, it deserves documentation.

Sounds good. Thanks for working on the threading feature. Sounds very good.

I don't think this technique / programming cliche (used for `C-x RET c' and proposed for `C-x &') should be passed off as somehow using a different kind of prefix argument - no connection, IMO. Nor should the key sequence involved be shoehorned into consideration as a prefix key. It's neither.

I don't see any useful use of the word "prefix" here, but if there is one then it needs to be defined explicitly, as some new kind of prefixing (starting by saying what gets prefixed).

I think we agree that this technique/cliche is interesting and could be applied more widely by users (other use cases) or turned into something more general (making it easy for users to take advantage of, for their own key-reading commands, without going through the same implementation cliche explicitly).

One could even imagine (similarly to a prefix key whose keymap binding has one or more prefix keys) applying the technique more than once in succession: a command that reads a key sequence (possibly with prefix arg) being itself read by a command that reads a key sequence (possibly with prefix arg).

Whereas it is simple to move a keymap around, binding it to this or that prefix key (any of which might itself be in a keymap that is bound to a prefix key...), it's a bit more cumbersome (less modular) to combine things and move them around with the technique in question. 

It's about defining a particular kind of command. Perhaps some good new programming constructs can be found, to help out here. Maybe a command-defining macro, to bundle up the technique nicely. Or maybe we're hinting at a variant of or a new form for a keymap. 

(Another thing that comes to (my) mind here, for some reason, are hydras...)

Re: Introducing thread-safe Tramp

[Michael Albinus]

Drew Adams <drew.adams@oracle.com> writes:

Hi Drew,

>> > You didn't answer wrt why we shouldn't just define additional commands
>> > and let users create async/sync toggle commands using them (and
>> > binding them to the same muscle-memory keys), or why we shouldn't also
>> > create such toggle commands (toggling with a prefix arg), but not
>> > necessarily bind them to the standard keys right away. IOW, what we
>> > usually do.
>
> I understand that you feel that. Why not find that out from users,
> over a period of time? If users end up binding the toggle commands I
> described, in place of the traditional file-operation bindings, then
> we'll know just how important and convenient such dispatching is. If
> only some users do that, and with only one or a couple of the file
> commands, then we'll have info to direct how best to accommodate that
> need.
>
> Just a suggestion.

I don't believe it will work in practice to observe users how they use
the new feature. How could we collect data about the usage patterns?

Furthermore, I don't believe we need a new set of file visiting commands
for the asynchronous case. The existing commands work well, customized
by the (now named so) `execute-file-commands-asynchronously' user option.

I don't expect the key sequence "C-x &" to be applied very often. It
exists just in case you want to visit a file synchronously or
asynchronously the other way as specified by that option.

>> It is common practice to change the behavior of a command by a prefix, 
>
> What does "a prefix" even mean, here? What's prefixing what?

[...]

Agreed, "prefix" might be misleading. Until we have a better term, I've
changed that to "key sequence C-x & prior the interactive command
invocation". This doesn't read fluid in the documentation, and I'm
prepared to change it once we have better wording.

> It's about defining a particular kind of command. Perhaps some good
> new programming constructs can be found, to help out here. Maybe a
> command-defining macro, to bundle up the technique nicely. Or maybe
> we're hinting at a variant of or a new form for a keymap.

All agreed, but from my side I'd like to postpone this. Everybody is
invited to contribute here.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> Date: Sat, 4 Aug 2018 08:42:49 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: fgunbin@fastmail.fm, emacs-devel@gnu.org
> 
> I don't really think we should call `C-x &' here a "prefix argument", in any case.

It follows the example of "C-x RET c", which runs
universal-coding-system-argument.  So I see nothing wrong with the
name: we have precedence.

Or is that the "prefix" part that bothers you?

RE: Introducing thread-safe Tramp

[Drew Adams]

> > I don't really think we should call `C-x &' here a "prefix argument", in any
> case.
> 
> It follows the example of "C-x RET c", which runs
> universal-coding-system-argument.  So I see nothing wrong with the
> name: we have precedence.
> 
> Or is that the "prefix" part that bothers you?

I see nothing in any of the doc (doc string, Emacs manual, Elisp manual) where we talk about `C-x RET c' being a "prefix argument". And we should not, IMO. It has nothing to do with a prefix argument, except that it reads a key sequence and handles reading a prefix argument in doing that. It certainly is not a prefix argument for the command it is bound to, nor does it provide a prefix arg for the command it is bound to.

Even `C-u' (`universal-argument') is not a prefix argument. It is a command that begins reading a prefix argument and that, possibly together with other commands (such as `digit-argument') provides that prefix argument to a followup command.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> Date: Sat, 4 Aug 2018 09:27:32 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: fgunbin@fastmail.fm, michael.albinus@gmx.de, emacs-devel@gnu.org
> 
> > Or is that the "prefix" part that bothers you?
> 
> I see nothing in any of the doc (doc string, Emacs manual, Elisp manual) where we talk about `C-x RET c' being a "prefix argument". And we should not, IMO. It has nothing to do with a prefix argument, except that it reads a key sequence and handles reading a prefix argument in doing that. It certainly is not a prefix argument for the command it is bound to, nor does it provide a prefix arg for the command it is bound to.
> 
> Even `C-u' (`universal-argument') is not a prefix argument. It is a command that begins reading a prefix argument and that, possibly together with other commands (such as `digit-argument') provides that prefix argument to a followup command.

I don't think you answered my question, did you?

Re: Introducing thread-safe Tramp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Even `C-u' (`universal-argument') is not a prefix argument. It is a command that begins reading a prefix argument and that, possibly together with other commands (such as `digit-argument') provides that prefix argument to a followup command.

We think of C-u as beginning a prefix argument, without exception.
Sometimes the prefix argument consists of C-u and nothing but C-u.

However, I agree it is clearest NOT to think of C-x RET c
as a prefix argument.  The prefix argument is a specific kind of value
and that is not how C-x RET c works.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Introducing thread-safe Tramp

[Michael Albinus]

Richard Stallman <rms@gnu.org> writes:

> However, I agree it is clearest NOT to think of C-x RET c
> as a prefix argument.  The prefix argument is a specific kind of value
> and that is not how C-x RET c works.

How would you name "C-x RET c" or "C-x &" then? I don't know of a term
which describes that behavior. For the time being I use "key sequence
C-x & prior the interactive command invocation", but this is more a
description than a clear term.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > How would you name "C-x RET c" or "C-x &" then? I don't know of a term
  > which describes that behavior.

"Prefix argument" is clearly wrong.  We're better off with no special
name for this unusual practice, than using a misleading name for it.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

RE: Introducing thread-safe Tramp

[Drew Adams]

>  > Even `C-u' (`universal-argument') is not a prefix argument. It is a command
>  > that begins reading a prefix argument and that, possibly together with other
> > commands (such as `digit-argument') provides that prefix argument to a
> > followup command.
> 
> We think of C-u as beginning a prefix argument, without exception.
> Sometimes the prefix argument consists of C-u and nothing but C-u.

If your point is that my "possibly" needs to be scoped only to the "other commands", and that `C-u' _always_ provides a prefix arg, even when alone, then yes, of course - that's what I meant for the scope of the "possibly".

My point was really that `C-u' is a key sequence, not a prefix argument.

`C-u', along with other prefix-arg forming keys (e.g., digits) provides a prefix argument to a command. I don't see those keys as _being_ a prefix argument. But sure, the two could be identified to some extent, when speaking loosely.

The doc describes/defines "raw prefix argument" and "numeric prefix argument", which is the "numeric value" of the raw prefix argument. Those are Lisp values, which include conses, atoms like `-', and integers. They are not key sequences.

When talking about the key sequences, we (I, at least) distinguish "plain `C-u'", which provides a prefix arg of `(4)', double plain `C-u', which provides an arg of `(16)', and so on. Other key sequences provide different raw prefix args.

I think it could make sense to speak of `C-u', `C-u C-u', `C-u 3 4', `M--', etc. as "prefix-arg key sequences" or "prefix arg-providing key sequences" or some such. But I think it would be misleading (except perhaps in certain contexts where the difference was already made clear) to speak of them as "prefix arguments".

Just one opinion.

> However, I agree it is clearest NOT to think of C-x RET c
> as a prefix argument.  The prefix argument is a specific kind of value
> and that is not how C-x RET c works.

Yes. And in particular (IMO, at least), a prefix arg is a Lisp value - just one of those values documented as such, which doesn't include (so far) any key-sequence Lisp values.

I'm not trying to be pedantic. I think it helps users to keep the terminology clear, here. 

It's OK with me if someone providing help to another says, e.g., "use a plain prefix arg to do XYZ", meaning "use plain `C-u' to do XYZ". That's fine, as long as both understand what is meant. 

But I don't think the Emacs doc should talk that way, in general. It's better for the doc to keep these things separate - key sequences are not prefix args.

And wrt this discussion, `C-x RET c' is neither a prefix argument nor a key sequence that provides a prefix arg. IIUC, it has nothing to do with a prefix arg. It is _not_ like `C-u' or `M--'.

At most, `C-x RET c' could be said to be a key sequence that always precedes (is always followed by) another key sequence, which it reads. 

It's not a _prefix key_, because it's not bound to a keymap. The key sequence that follows it is not looked up in a map that `C-x RET c' is bound to. Instead, that following key sequence is read by the command bound to `C-x RET c', and then invoked. (And `C-x RET c' can set some context for that following command - bind variables or whatever.)

It's the behavior of a key sequence such as `C-x RET c' (or rather its command) that's being discussed here. AFAIK, we don't have a name for such key sequences/commands/behavior. Until now, the only instance of such a thing (AFAIK) was `C-x RET c'. Now there might be a second: `C-x &'.

Is it perhaps time to come up with some terminology/concepts for this, and to document it?

And maybe to come up with some constructs that make defining such key sequences/commands/behavior easier for users?

 We added `define-minor-mode' to capture the programming cliche of defining a minor mode. Perhaps something like that could help here. Dunno whether it would be jumping the gun or overkill to do that now, since there has been so little use of the cliche, so far. (I.e., unlike defining a minor mode, it's not really a cliche yet - just a rarely used technique.)

Re: Introducing thread-safe Tramp

[Howard Melman]

My sense is that the proposed C-x & is a prefix, because it
is used before another key sequence. It is not an argument
because it doesn't affect the functions' argument list the
way C-u does.

Maybe the term should be "prefix sequence"?

Wasn't there are a proposal at one point to do something
similar for C-x 4 and C-x 5 so that there didn't have to be
-other-window and -other-frame variants of so many
commands? 

-- 

Howard

Re: Introducing thread-safe Tramp

[Werner LEMBERG]

> My sense is that the proposed C-x & is a prefix, because it is used before
> another key sequence.  It is not an argument because it doesn't affect the
> functions' argument list the way C-u does.
> 
> Maybe the term should be "prefix sequence"?

What about `multiplexer command', avoiding the perhaps misleading `prefix'
word altogether?


    Werner

Re: Introducing thread-safe Tramp

[Michael Albinus]

Howard Melman <hmelman@gmail.com> writes:

Hi Howard,

> My sense is that the proposed C-x & is a prefix, because it
> is used before another key sequence. It is not an argument
> because it doesn't affect the functions' argument list the
> way C-u does.

I'd agree with this.

> Maybe the term should be "prefix sequence"?

I like this proposal. "Prefix key sequence" would be more precise, but
it doesn't read fluid. If there isn't any objection over the next week,
I'll apply your wording.

> Wasn't there are a proposal at one point to do something
> similar for C-x 4 and C-x 5 so that there didn't have to be
> -other-window and -other-frame variants of so many
> commands? 

I'm not aware of such a proposal (which only means that my memory is
bad), but it makes sense.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Clément Pit-Claudel]

On 2018-08-06 04:07, Michael Albinus wrote:
>> Wasn't there are a proposal at one point to do something
>> similar for C-x 4 and C-x 5 so that there didn't have to be
>> -other-window and -other-frame variants of so many
>> commands? 
>> I'm not aware of such a proposal (which only means that my memory is
> bad), but it makes sense.

Indeed, that would be very nice (I don't remember such a proposal either, though)

RE: Introducing thread-safe Tramp

[Drew Adams]

> >> Wasn't there are a proposal at one point to do something
> >> similar for C-x 4 and C-x 5 so that there didn't have to be
> >> -other-window and -other-frame variants of so many
> >> commands?
> >> I'm not aware of such a proposal (which only means that my memory is
> > bad), but it makes sense.
> 
> Indeed, that would be very nice (I don't remember such a proposal either,
> though)

Let's please not go there, in this thread. There are downsides, as well as upsides, to such a proposal. We've discussed it before. We can discuss it again if needed. But let's not branch this thread for such a discussion.

RE: Introducing thread-safe Tramp

[Drew Adams]

> > My sense is that the proposed C-x & is a prefix, because it
> > is used before another key sequence. It is not an argument
> > because it doesn't affect the functions' argument list the
> > way C-u does.
> 
> I'd agree with this.

What is it a prefix of? A prefix is part of what it is a prefix of.

> > Maybe the term should be "prefix sequence"?

Sequence of what?

> I like this proposal. "Prefix key sequence" would be more precise, but
> it doesn't read fluid. If there isn't any objection over the next week,
> I'll apply your wording.

Please don't.

This is _not_ a prefix key. A key is a key sequence. A prefix key is a key sequence that is a prefix of a larger key sequence. But in Emacs a prefix key also has the specific meaning of a key sequence that is bound to a keymap. `C-x &' is not bound to a keymap, and is thus not a prefix key (sequence).

Until and unless we really do decide to baptize this and document it, please just what RMS suggested: don't call it anything. Certainly please do not call it just a "prefix" or a "prefix key (sequence)".

> > Wasn't there are a proposal at one point to do something
> > similar for C-x 4 and C-x 5 so that there didn't have to be
> > -other-window and -other-frame variants of so many
> > commands?
> 
> I'm not aware of such a proposal (which only means that my memory is
> bad), but it makes sense.

I believe he's thinking of Stefan's wish to have prefix keys `C-x 4' and `C-x 5' work automatically, without defining separate other-window and other-frame commands. That's something completely different. Let's please not let this thread wander farther than it already has.

Re: Introducing thread-safe Tramp

[Howard Melman]

> On Aug 6, 2018, at 10:52 AM, Drew Adams <drew.adams@oracle.com> wrote:
> 
>>> My sense is that the proposed C-x & is a prefix, because it
>>> is used before another key sequence. It is not an argument
>>> because it doesn't affect the functions' argument list the
>>> way C-u does.
>>> Maybe the term should be "prefix sequence"?
> 
>> I like this proposal. "Prefix key sequence" would be more precise, but
>> it doesn't read fluid. If there isn't any objection over the next week,
>> I'll apply your wording.
> 
> Please don't.
> 
> This is _not_ a prefix key. A key is a key sequence. A prefix key is a key sequence that is a prefix of a larger key sequence. But in Emacs a prefix key also has the specific meaning of a key sequence that is bound to a keymap. `C-x &' is not bound to a keymap, and is thus not a prefix key (sequence).
> 
> Until and unless we really do decide to baptize this and document it, please just what RMS suggested: don't call it anything. Certainly please do not call it just a "prefix" or a "prefix key (sequence)".

Having reread the Keys section of the Emacs manual I see your objection. Since "key" is short for "key sequence", then indeed it's not a "prefix key (sequence)" or a "(prefix key) sequence". My intention was to read it as "prefix (key sequence)", as something that comes before a key sequence, though clearly the ambiguity is an issue.

> What is it a prefix of? A prefix is part of what it is a prefix of.

It's a prefix of a key sequence, and it is a key sequence itself ("a sequence of one or more input events is is meaningful as a unit"). It's kind of a "complete key" ("a key sequence [that] invokes a command"), though it's a "complete key" that needs more. Maybe it's a "semi-complete key" as something between the existing terms "complete key" and "prefix key" and the full sequence C-x & C-x C-f is still a complete key sequence.

As you pointed out, technically C-u isn't a prefix key because it calls the command universal-argument, maybe it also is a semi-complete key. (Though clearly it would always be more useful to describe C-u as beginning a prefix argument).

>>> Wasn't there are a proposal at one point to do something
>>> similar for C-x 4 and C-x 5 so that there didn't have to be
>>> -other-window and -other-frame variants of so many
>>> commands?
>> 
>> I'm not aware of such a proposal (which only means that my memory is
>> bad), but it makes sense.
> 
> I believe he's thinking of Stefan's wish to have prefix keys `C-x 4' and `C-x 5' work automatically, without defining separate other-window and other-frame commands. That's something completely different. Let's please not let this thread wander farther than it already has.

Yes, that's what I was thinking of. I'm not sure if there was a proposed implementation or if it was similar to this one, but the usage struck me as similar. Since this is a proposal to name a new kind of thing, it didn't feel like wandering to to test it against other similar kinds of proposals to see if it would cover them as well.

Howard 

Re: Introducing thread-safe Tramp

[Stefan Monnier]

> Wasn't there are a proposal at one point to do something
> similar for C-x 4 and C-x 5 so that there didn't have to be
> -other-window and -other-frame variants of so many
> commands? 

Indeed.  It's implemented in the other-frame-window package in GNU ELPA.
And I still hope it will actually replace C-x 4 and C-x 5, and hopefully
sooner rather than later.


        Stefan

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Filipp Gunbin <fgunbin@fastmail.fm>
> Date: Fri, 27 Jul 2018 22:04:04 +0300
> Cc: Drew Adams <drew.adams@oracle.com>, emacs-devel@gnu.org
> 
> It feels like prefix arg should change the user-visible behaviour of
> the command

We never restrict the use of prefix argument only to some sort of
changes.  Besides, running a command asynchronously does change its
user-visible behavior, and quite significantly so.

> Maybe there could be rules for find-file to use async behavior by
> default, like:
> 
> - When a non-local method is explicitly used (that is, not /sudo)
> 
> - When default-directory is on a remote host
> 
> - When a file being visited is greater than some customizable size

That's even worse, IMO.  We will never be able to second-guess what
users want, and they will want a fire escape even if we succeed in
most cases.

> - When a special command was given before (analogy with `C-x RET c' was
> already mentioned here).

"C-x RET c" runs universal-coding-system-argument, which is akin to
prefix argument, only separate from it.  I do agree that using a
separate argument similar to "C-x RET c" is better, since it binds a
variable to a value, like "C-x RET c" does.

Re: Introducing thread-safe Tramp

[Filipp Gunbin]

On 28/07/2018 00:08 +0300, Eli Zaretskii wrote:

>> From: Filipp Gunbin <fgunbin@fastmail.fm>
>> Date: Fri, 27 Jul 2018 22:04:04 +0300
>> Cc: Drew Adams <drew.adams@oracle.com>, emacs-devel@gnu.org
>>
>> It feels like prefix arg should change the user-visible behaviour of
>> the command
>
> We never restrict the use of prefix argument only to some sort of
> changes.  Besides, running a command asynchronously does change its
> user-visible behavior, and quite significantly so.

Yes, replied about it in another message.  It's more about how the
command does its job, not what the job is - that's why prefix arg may be
needed for something else.

>> Maybe there could be rules for find-file to use async behavior by
>> default, like:
>>
>> - When a non-local method is explicitly used (that is, not /sudo)
>>
>> - When default-directory is on a remote host
>>
>> - When a file being visited is greater than some customizable size
>
> That's even worse, IMO.  We will never be able to second-guess what
> users want, and they will want a fire escape even if we succeed in
> most cases.

I think this is more about reasonable defaults than about guessing for
users..  I'd be happy to have all remote files opened asynchronously by
default - the delay is significant, especially if it's large log file
over ssh.  For local files, I'd rarely notice the difference.  Of
course, there could and should be fire escape.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Filipp Gunbin <fgunbin@fastmail.fm> writes:

>>> Maybe there could be rules for find-file to use async behavior by
>>> default, like:
>>>
>>> - When a non-local method is explicitly used (that is, not /sudo)
>>>
>>> - When default-directory is on a remote host
>>>
>>> - When a file being visited is greater than some customizable size
>>
>> That's even worse, IMO.  We will never be able to second-guess what
>> users want, and they will want a fire escape even if we succeed in
>> most cases.
>
> I think this is more about reasonable defaults than about guessing for
> users..  I'd be happy to have all remote files opened asynchronously by
> default - the delay is significant, especially if it's large log file
> over ssh.  For local files, I'd rarely notice the difference.  Of
> course, there could and should be fire escape.

I've changed find-file-asynchronously to accept also a regexp as
value. If you customize

(customize-set-variable
  'find-file-asynchronously tramp-file-name-regexp
  "Visit remote files asynchronously")

you get the requested behavior. The default shall still be nil, I
believe.

If needed, we could also allow a function as find-file-asynchronously
value. People could add their own checks then, like for file name size.

Re: Introducing thread-safe Tramp

[Dmitry Gutov]

On 7/25/18 12:46 PM, Michael Albinus wrote:
> Running the commands asynchronously is very tight bound to the usual
> invocation of the commands, so I don't see that something else, which
> might asks for a prefix argument of the commands, will be better suited.

What is the point of the user asking for asynchronous execution?

How does an "asynchronous command" work? Will it silently abort if the 
user invokes a different command in the meantime?

Re: Introducing thread-safe Tramp

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

Hi Dmitry,

> What is the point of the user asking for asynchronous execution?

That Emacs is not blocked while loading huge and/or remote files.

> How does an "asynchronous command" work? Will it silently abort if the
> user invokes a different command in the meantime?

No, it continues. It runs in another thread.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Yuri Khan]

On Sat, Jul 28, 2018 at 2:33 AM Michael Albinus <michael.albinus@gmx.de> wrote:

> > How does an "asynchronous command" work? Will it silently abort if the
> > user invokes a different command in the meantime?
>
> No, it continues. It runs in another thread.

What feedback is given to the user after the asynchronous find command
is issued and before the user sees the file content and can edit it?

Re: Introducing thread-safe Tramp

[Michael Albinus]

Yuri Khan <yurivkhan@gmail.com> writes:

>> > How does an "asynchronous command" work? Will it silently abort if the
>> > user invokes a different command in the meantime?
>>
>> No, it continues. It runs in another thread.
>
> What feedback is given to the user after the asynchronous find command
> is issued and before the user sees the file content and can edit it?

Nothing yet. Open point.

What do you propose?

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Yuri Khan]

On Sat, Jul 28, 2018 at 2:43 AM Michael Albinus <michael.albinus@gmx.de> wrote:

> > What feedback is given to the user after the asynchronous find command
> > is issued and before the user sees the file content and can edit it?
>
> Nothing yet. Open point.
>
> What do you propose?

Basically, do what a web browser does when a slow-loading URL is
opened in a new tab.

Create a buffer and display it immediately, either empty or with some
placeholder text, to make it clear to the user that the command was
accepted but content is not final. Replace with actual content when it
is loaded or as it is being loaded.

While the file is being loaded, the user can close the window in which
it would be displayed, or display another buffer in that window, so
that the buffer is not visible when it finishes loading. It might be
useful to give a subtle indication that the asynchronous command
finished, maybe by displaying a message in the echo area. (Browsers do
that by animating a loader icon on the tab and stopping that when
loaded, but Emacs does not by default have a tab bar.)

Also, while the file is being loaded, the user can kill the target
buffer. Ideally that would cancel the asynchronous loading operation,
but acceptable behavior would be to silently abandon the content when
it finishes loading.

Bonus points if buffers that are being loaded asynchronously are
visible as such in Ibuffer.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Yuri Khan <yurivkhan@gmail.com>
> Date: Sat, 28 Jul 2018 03:09:24 +0700
> Cc: Emacs developers <emacs-devel@gnu.org>, Drew Adams <drew.adams@oracle.com>,
> 	Dmitry Gutov <dgutov@yandex.ru>
> 
> It might be useful to give a subtle indication that the asynchronous
> command finished, maybe by displaying a message in the echo
> area.

Not in the echo area, on the mode line.  The echo area is too
ephemeral.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> It might be useful to give a subtle indication that the asynchronous
>> command finished, maybe by displaying a message in the echo
>> area.
>
> Not in the echo area, on the mode line.  The echo area is too
> ephemeral.

Some years ago, I've proposed an extension to the progress reporter for
the modeline,
<http://lists.gnu.org/archive/html/emacs-devel/2011-02/msg00764.html>.
Maybe we shall reanimate the proposal?

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Yuri Khan <yurivkhan@gmail.com>,  emacs-devel@gnu.org,  drew.adams@oracle.com,  dgutov@yandex.ru
> Date: Sat, 28 Jul 2018 11:11:01 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> It might be useful to give a subtle indication that the asynchronous
> >> command finished, maybe by displaying a message in the echo
> >> area.
> >
> > Not in the echo area, on the mode line.  The echo area is too
> > ephemeral.
> 
> Some years ago, I've proposed an extension to the progress reporter for
> the modeline,
> <http://lists.gnu.org/archive/html/emacs-devel/2011-02/msg00764.html>.
> Maybe we shall reanimate the proposal?

Maybe so, but that's largely unrelated: the issue at hand here is how
to indicate the progress/completion of an async file visiting command,
with or without an infrastructure that makes such reporting easier or
nicer.  I think.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> Some years ago, I've proposed an extension to the progress reporter for
>> the modeline,
>> <http://lists.gnu.org/archive/html/emacs-devel/2011-02/msg00764.html>.
>> Maybe we shall reanimate the proposal?
>
> Maybe so, but that's largely unrelated: the issue at hand here is how
> to indicate the progress/completion of an async file visiting command,
> with or without an infrastructure that makes such reporting easier or
> nicer.  I think.

Sure. I've put this indication thingy on my todo list, but with less
priority. First, the minibuffer / echo area problem must be solved
(sweat dropping).

Btw, everybody is invited to contribute code to the
feature/tramp-thread-safe branch!

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Dmitry Gutov]

On 7/27/18 10:32 PM, Michael Albinus wrote:

>> How does an "asynchronous command" work? Will it silently abort if the
>> user invokes a different command in the meantime?
> 
> No, it continues. It runs in another thread.

OK, that makes sense, for this particular command. Together with the 
mode-line idea.

Re: Introducing thread-safe Tramp

[Dmitry Gutov]

On 7/25/18 12:46 PM, Michael Albinus wrote:
> But I don't believe people will always visit files asynchronously. They
> will do it for remote files, they will do it for huge files. But for my
> daily work with local small files, this would be overkill.

Wouldn't a small file just load immediately, thus sparing the user from 
having to look at the "loading" indicator?

Re: Introducing thread-safe Tramp

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 7/25/18 12:46 PM, Michael Albinus wrote:
>> But I don't believe people will always visit files asynchronously. They
>> will do it for remote files, they will do it for huge files. But for my
>> daily work with local small files, this would be overkill.
>
> Wouldn't a small file just load immediately, thus sparing the user
> from having to look at the "loading" indicator?

If it is a remote file, and the connetion has not been established yet,
it would take more time due to the initial hand-shaking.

And git's vc-refresh-state implementation takes more time for small
remote files than loading the file into the buffer, due to the remote
processes being called.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Dmitry Gutov]

On 7/28/18 12:18 PM, Michael Albinus wrote:
> If it is a remote file, and the connetion has not been established yet,
> it would take more time due to the initial hand-shaking.
> 
> And git's vc-refresh-state implementation takes more time for small
> remote files than loading the file into the buffer, due to the remote
> processes being called.

In both of those cases, opening the file asynchronously probably makes 
more sense, IMHO.

So I think a global user option should do fine. Maybe even just a 
boolean by default: the case you described is a remote file, so there's 
still no counter-argument for opening small local files asynchronously.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

> So I think a global user option should do fine. Maybe even just a
> boolean by default: the case you described is a remote file, so
> there's still no counter-argument for opening small local files
> asynchronously.

Yes, that's what find-file-asynchronously is good for. Meanwhile.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Dmitry Gutov]

On 7/23/18 6:58 PM, Michael Albinus wrote:

> As second change, I have modified `vc-refresh-state' to run always in
> its own thread. This is because I have observed, that for git
> repositories, loading a remote file is often much faster than the
> following update of its state in the modeline. For the time being this
> behaviour is hard-coded, I haven't seen any drawback for this even for
> local files. But we could trigger it also via a user option.
> 
> @Dmitry?

No objections from me in principle, but we should probably call 
vc-file-clearprops right away (is there a chance that some other code 
calls vc-state before this thread launches and gets stale status?).

We could also make vc-state use the same mutex, in order for it to wait 
until refresh finished, instead of returning the stale result.

Re: Introducing thread-safe Tramp

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

> No objections from me in principle, but we should probably call
> vc-file-clearprops right away (is there a chance that some other code
> calls vc-state before this thread launches and gets stale status?).

I've scanned the Emacs lisp dir, and there doesn't seem to be any other
call to vc-state. If I'm not mistaken.

> We could also make vc-state use the same mutex, in order for it to
> wait until refresh finished, instead of returning the stale result.

Well, you are much more experienced with vc than I do. Look at the
changes I've done in vc-hooks.el, and adapt.

The only other relevant change I've done is, that I lock the vc-mutex in
find-file-with-threads of files.el. This reorders the threads, all
vc-refresh-state calls are performed after all files (from a wildcard)
are loaded into their respective buffers. This improves early visibility
of the visited files.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Dmitry Gutov]

On 7/28/18 1:13 PM, Michael Albinus wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> No objections from me in principle, but we should probably call
>> vc-file-clearprops right away (is there a chance that some other code
>> calls vc-state before this thread launches and gets stale status?).
> 
> I've scanned the Emacs lisp dir, and there doesn't seem to be any other
> call to vc-state. If I'm not mistaken.

Third-party code can do that, too. Such as diff-hl, and other packages 
not in GNU ELPA. I don't know if any of them call vc-refresh-state, but 
it would be nice if

(progn
   (vc-refresh-state)
   (vc-state))

always returned a predictable, up-to-date result.

>> We could also make vc-state use the same mutex, in order for it to
>> wait until refresh finished, instead of returning the stale result.
> 
> Well, you are much more experienced with vc than I do. Look at the
> changes I've done in vc-hooks.el, and adapt.

I'm not well-versed on the use of Emacs threads, though. Yet.

> The only other relevant change I've done is, that I lock the vc-mutex in
> find-file-with-threads of files.el. This reorders the threads, all
> vc-refresh-state calls are performed after all files (from a wildcard)
> are loaded into their respective buffers. This improves early visibility
> of the visited files.

Do we actually want one global mutex? What if we had one mutex per file? 
Doing those process calls in parallel would be faster, no?

Re: Introducing thread-safe Tramp

[Michael Albinus]

Dmitry Gutov <dgutov@yandex.ru> writes:

Hi Dmitry,

> Third-party code can do that, too. Such as diff-hl, and other packages
> not in GNU ELPA. I don't know if any of them call vc-refresh-state,
> but it would be nice if
>
> (progn
>   (vc-refresh-state)
>   (vc-state))
>
> always returned a predictable, up-to-date result.

vc-refresh-state works over the current buffer, vc-state works over a
file (which is missing in your example). I don't see after a short look
in the sources, how they depend. The result of your code snippet would
always be the result of vc-state, which is not threaded.

>> Well, you are much more experienced with vc than I do. Look at the
>> changes I've done in vc-hooks.el, and adapt.
>
> I'm not well-versed on the use of Emacs threads, though. Yet.

Me too, some weeks ago :-) It isn't that hard.

>> The only other relevant change I've done is, that I lock the vc-mutex in
>> find-file-with-threads of files.el. This reorders the threads, all
>> vc-refresh-state calls are performed after all files (from a wildcard)
>> are loaded into their respective buffers. This improves early visibility
>> of the visited files.
>
> Do we actually want one global mutex? What if we had one mutex per
> file? Doing those process calls in parallel would be faster, no?

One mutex per file isn't necessary; vc-refresh-state is already blocked
per file (buffer), because it is called only at the very end of any file
visiting command.

But you are right, the vc-refresh-state calls shall run in parallel. So
I have changed to lock them via vc-mutex only at thread entry, unlock
happens immediately. This still ensures that all vc-refresh-state
threads run only after all files have been loaded into their buffers,
but then they can run in parallel.

Best regards, Michael.

Re: Introducing thread-safe Tramp

[Dmitry Gutov]

On 7/29/18 12:54 PM, Michael Albinus wrote:

> vc-refresh-state works over the current buffer, vc-state works over a
> file (which is missing in your example).

(vc-state buffer-file-name), then.

We don't have a counter-part for vc-refresh-state that takes a file as 
an argument, so it's to be expected that some clients will use with 
function. E.g. with with-current-buffer.

> I don't see after a short look
> in the sources, how they depend. The result of your code snippet would
> always be the result of vc-state, which is not threaded.

My point is that before your branch, the result would be different. It 
would make sense to preserve the behavior.

>>> Well, you are much more experienced with vc than I do. Look at the
>>> changes I've done in vc-hooks.el, and adapt.
>>
>> I'm not well-versed on the use of Emacs threads, though. Yet.
> 
> Me too, some weeks ago :-) It isn't that hard.

I'll try to make some time.