save-excursion and multi-thread?

save-excursion and multi-thread?

[Qiantan Hong]

I tried
(make-thread
 (lambda ()
   (save-excursion (goto-char (point-min)) (sleep-for 1))))
In *scratch*.

It seems that after the thread yields (because of sleep-for)
it doesn’t restore point position, and after 1 second
cursor is reseted to the original position, discarding
any of my movements during this 1 second.

Is this how save-excursion supposed to work?
It’s inconsistent with the behavior of special variable + dynamic bindings,
which works currently under multi-thread.

What’s the supposed way to have “thread-local” movements?

Best,
Qiantan

Re: save-excursion and multi-thread?

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> Date: Sun, 26 Sep 2021 17:41:17 +0000
> Accept-Language: en-US
> 
> I tried
> (make-thread
>  (lambda ()
>    (save-excursion (goto-char (point-min)) (sleep-for 1))))
> In *scratch*.
> 
> It seems that after the thread yields (because of sleep-for)
> it doesn’t restore point position

It cannot, because the save-excursion form is still running.  The call
to sleep-for doesn't end the save-excursion form immediately, it ends
it only after sleep-for returns

> and after 1 second cursor is reseted to the original position,
> discarding any of my movements during this 1 second.

By "my movements" you mean what you did in another thread?

> Is this how save-excursion supposed to work?

Yes.

> It’s inconsistent with the behavior of special variable + dynamic bindings,
> which works currently under multi-thread.

Inconsistent in what way?

> What’s the supposed way to have “thread-local” movements?

What do you mean by "thread-local" movements?  Buffers are global, and
each buffer has only one point.

Re: save-excursion and multi-thread?

[Qiantan Hong]

>> and after 1 second cursor is reseted to the original position,
>> discarding any of my movements during this 1 second.
> 
> By "my movements" you mean what you did in another thread?
I mean I manually move (C-{fbnp}) in the *scratch* buffer.

>> Is this how save-excursion supposed to work?
> 
> Yes.
> 
>> It’s inconsistent with the behavior of special variable + dynamic bindings,
>> which works currently under multi-thread.
> 
> Inconsistent in what way?
> 
>> What’s the supposed way to have “thread-local” movements?
> 
> What do you mean by "thread-local" movements?  Buffers are global, and
> each buffer has only one point.
Similar thing can be said with special variables — they’re global and has only one value.
But what “global” actually means is dynamic scope. 
The values of special variables are associated with each dynamic scope 
(while values of lexical variables are associated with lexical scope),
and each let binding attaches a value to the current dynamic scope.
(or really, the current continuation).

Under multithread settings, each thread can have their own active dynamic scope (continuation),
therefore values of special variable doesn’t interfere.
I expects save-excursion to do the same.

> by "thread-local" movements?
I mean some thread uses save-excursion around some operations involving navigating/editing
the buffer, and expecting to *not* affect buffer state outside the dynamic scope of such code block.
The current behavior makes practical sense under no circumstances,
because we don’t know what thread will be switched to and what it will be doing
after a thread yield.

Re: save-excursion and multi-thread?

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Sun, 26 Sep 2021 18:28:44 +0000
> 
> >> It’s inconsistent with the behavior of special variable + dynamic bindings,
> >> which works currently under multi-thread.
> > 
> > Inconsistent in what way?
> > 
> >> What’s the supposed way to have “thread-local” movements?
> > 
> > What do you mean by "thread-local" movements?  Buffers are global, and
> > each buffer has only one point.
> Similar thing can be said with special variables — they’re global and has only one value.
> But what “global” actually means is dynamic scope. 
> The values of special variables are associated with each dynamic scope 
> (while values of lexical variables are associated with lexical scope),
> and each let binding attaches a value to the current dynamic scope.
> (or really, the current continuation).
> 
> Under multithread settings, each thread can have their own active dynamic scope (continuation),
> therefore values of special variable doesn’t interfere.
> I expects save-excursion to do the same.

I'm sorry, I don't think I follow.  I don't even understand what you
mean by "special variables" or "active dynamic scope".  Maybe you
could explain it in some other way, perhaps with an example or two?

> > by "thread-local" movements?
> I mean some thread uses save-excursion around some operations involving navigating/editing
> the buffer, and expecting to *not* affect buffer state outside the dynamic scope of such code block.
> The current behavior makes practical sense under no circumstances,
> because we don’t know what thread will be switched to and what it will be doing
> after a thread yield.

The Lisp threads in Emacs were not designed to support several threads
making changes to the same buffer.  On the contrary, the assumption
was that each thread will have its own current buffer, which is why
the current buffer is thread-local -- it can be different in each
thread.

There's very few variables that are thread-local, you can see them in
thread.h.  Most of the thread-local variables are there just to let
threads run Lisp more or less independently, to handle non-local exits
and errors independently, and support the thread-related primitives.
Only a couple of them are exposed to Lisp in normal operations.

Re: save-excursion and multi-thread?

[Qiantan Hong]

> I'm sorry, I don't think I follow.  I don't even understand what you
> mean by "special variables" or "active dynamic scope".  Maybe you
> could explain it in some other way, perhaps with an example or two?
(defvar x 5)
(make-thread
 (lambda ()
   (let ((x 7))
     (sleep-for 1))))
x ;; => still 5 within this 1 second, by C-x C-e (which runs in main thread)

I think maybe the confusion is from different terminology to call special variable (in Common Lisp terms),
maybe you simply call it global variable or dynamic variable?

> 
>>> by "thread-local" movements?
>> I mean some thread uses save-excursion around some operations involving navigating/editing
>> the buffer, and expecting to *not* affect buffer state outside the dynamic scope of such code block.
>> The current behavior makes practical sense under no circumstances,
>> because we don’t know what thread will be switched to and what it will be doing
>> after a thread yield.
> 
> The Lisp threads in Emacs were not designed to support several threads
> making changes to the same buffer.  On the contrary, the assumption
> was that each thread will have its own current buffer, which is why
> the current buffer is thread-local -- it can be different in each
> thread.
> 
> There's very few variables that are thread-local, you can see them in
> thread.h.  Most of the thread-local variables are there just to let
> threads run Lisp more or less independently, to handle non-local exits
> and errors independently, and support the thread-related primitives.
> Only a couple of them are exposed to Lisp in normal operations.
Isn’t all global/dynamic variable thread local, as in the example above?

> Right now, point is a field in the buffer data structure, so there's
> just one per buffer shared between all threads.  I think as long as your
> thread doesn't use `sit-for', `sleep-for', or reads input, you can be
> sure it won't be interrupted.  If it does, you cannot assume point is
> still there where your thread moved it.
I’m using thread exactly because my procedure need to wait and I don’t
want it to block the main thread.

Re: save-excursion and multi-thread?

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: Tassilo Horn <tsdh@gnu.org>, Eli Zaretskii <eliz@gnu.org>
> Date: Sun, 26 Sep 2021 19:09:42 +0000
> 
> (defvar x 5)
> (make-thread
>  (lambda ()
>    (let ((x 7))
>      (sleep-for 1))))
> x ;; => still 5 within this 1 second, by C-x C-e (which runs in main thread)
> 
> I think maybe the confusion is from different terminology to call special variable (in Common Lisp terms),
> maybe you simply call it global variable or dynamic variable?

As documented in the ELisp manual, let-bindings are thread-local.

> Isn’t all global/dynamic variable thread local, as in the example above?

No, only the let-bindings are.  And also the regexp search-related
stuff, so that you could have independent searches in each thread.

Re: save-excursion and multi-thread?

[Qiantan Hong]

> The Lisp threads in Emacs were not designed to support several threads
> making changes to the same buffer.  On the contrary, the assumption
> was that each thread will have its own current buffer, which is why
> the current buffer is thread-local -- it can be different in each
> thread.
I see. Can I fake that with indirect-buffer?

Re: save-excursion and multi-thread?

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Sun, 26 Sep 2021 19:13:03 +0000
> Accept-Language: en-US
> 
> > The Lisp threads in Emacs were not designed to support several threads
> > making changes to the same buffer.  On the contrary, the assumption
> > was that each thread will have its own current buffer, which is why
> > the current buffer is thread-local -- it can be different in each
> > thread.
> I see. Can I fake that with indirect-buffer?

For moving point, I think you can.  But if you change the text, you
will see the changes in the other threads as well.

Re: save-excursion and multi-thread?

[Qiantan Hong]

>>> The Lisp threads in Emacs were not designed to support several threads
>>> making changes to the same buffer.  On the contrary, the assumption
>>> was that each thread will have its own current buffer, which is why
>>> the current buffer is thread-local -- it can be different in each
>>> thread.
>> I see. Can I fake that with indirect-buffer?
> 
> For moving point, I think you can.  But if you change the text, you
> will see the changes in the other threads as well.
That’s intended. 
I simply want point to behave like a proper global variable
and save-excursion to behave like a proper let-binding.
In that sense I don’t want point to be thread local, 
I want save-excursion to be thread local, which I think
is the only sensible concurrent semantics for save-excursion.
Only in that way save-excursion really serves its intended purpose
— to have “local” excursion that doesn’t affect global state
outside the code it surrounds.
Does that convince you?

Re: save-excursion and multi-thread?

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Sun, 26 Sep 2021 19:21:38 +0000
> 
> >> I see. Can I fake that with indirect-buffer?
> > 
> > For moving point, I think you can.  But if you change the text, you
> > will see the changes in the other threads as well.
> That’s intended. 
> I simply want point to behave like a proper global variable

But point is not a variable.  It's a property of a buffer, and all you
have is a (read-only) accessor to that property.

> and save-excursion to behave like a proper let-binding.

You can always use save-excursion in the other thread, no?

> In that sense I don’t want point to be thread local, 
> I want save-excursion to be thread local, which I think
> is the only sensible concurrent semantics for save-excursion.
> Only in that way save-excursion really serves its intended purpose
> — to have “local” excursion that doesn’t affect global state
> outside the code it surrounds.

That's one way of looking at save-excursion.  Another way is to look
at it as a means for a thread to protect its own position of point
from changes by other threads.  So from that POV, it's the other
thread which needs to use save-excursion.

> Does that convince you?

Convince me to do what? redesign and reimplement Lisp threads in
Emacs with very different design goals?

Re: save-excursion and multi-thread?

[Qiantan Hong]

> On Sep 26, 2021, at 12:29 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Qiantan Hong <qhong@mit.edu>
>> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
>> Date: Sun, 26 Sep 2021 19:21:38 +0000
>> 
>>>> I see. Can I fake that with indirect-buffer?
>>> 
>>> For moving point, I think you can.  But if you change the text, you
>>> will see the changes in the other threads as well.
>> That’s intended. 
>> I simply want point to behave like a proper global variable
> 
> But point is not a variable.  It's a property of a buffer, and all you
> have is a (read-only) accessor to that property.
> 
>> and save-excursion to behave like a proper let-binding.
> 
> You can always use save-excursion in the other thread, no?
Doesn’t looks like it’s protected
(progn
  (make-thread
   (lambda ()
     (save-excursion
       (sleep-for 5)
       (print (point)))))
  (make-thread
   (lambda ()
     (save-excursion
       (sleep-for 3)
       (goto-char (point-min))
       (sleep-for 3)))))
This prints 1 instead of the point before the first save-excursion enters.

Moreover, the other thread can be the main UI thread, 
I don’t know how I can protect it (assuming that save-excursion does protect,
although it actually doesn’t).

>> Does that convince you?
> 
> Convince me to do what? redesign and reimplement Lisp threads in
> Emacs with very different design goals?
Not doing anything, let’s figure out what’s correct first.
Whether to do the correct thing comes after that.
I also don’t think a redesign is necessary, we just need to
restore point state during thread switching, but I haven’t look at detail yet.

Re: save-excursion and multi-thread?

[Qiantan Hong]

> redesign and reimplement Lisp threads in
> Emacs with very different design goals?
I don’t think design goal matters.
The current concurrent behavior of save-excursion
makes it basically useless with threads in any sense.

Re: save-excursion and multi-thread?

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Sun, 26 Sep 2021 19:45:24 +0000
> 
> The current concurrent behavior of save-excursion
> makes it basically useless with threads in any sense.

Then don't use save-excursion for this purpose, use something else.

Re: save-excursion and multi-thread?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The current concurrent behavior of save-excursion
  > makes it basically useless with threads in any sense.

I think save-excursion in one thread ought to keep its locally-bound
point and mark within that thread alone.  In other words, it should
work like dynamic variable bindings.

So if a thread does not do save-excursion, it should share the value
of point in each buffer with the other threads.  However, doing
save-excursion in a buffer should "locally bind" point (and mark) in
that buffer in the current thread, and that "local binding" should not
affect other threads or be affected by them.

This does not necessarily require changing the way point is
implemented at low level.  Indeed, I would prefer to avoid such
far-reaching changes if I were implementing it.  It's enough to
make it possible for thread-switching to understand save-excursion
and DTRT.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: save-excursion and multi-thread?

[dick]

I've not chimed in because whether or not I did wouldn't alter the trajectory
of a young programmer's eventual realization that implementing concurrent
editing purely in lisp will result in something quite unusable.  Moreover, he
will discover (in a few years) that programming is essentially a solitary
pursuit that does not benefit commensurately to the complexity concurrent
editing adds.  One need only look at how tasks are divvied up at ACM
competitions to see that programming is really not a team sport.

But this remark struck me:

RS> [Having point behave like a special variable] does not necessarily require
RS> changing the way point is implemented at low level.

EZ was quite clear that point does not live in lisp-space, and thus cannot
leverage the BLV (buffer-local-value) machinery of data.c (which I might add
is broken under threads, bug#48990).  Having it behave in accordance with your
and SM's preference would incur a cost in collateral complexity that, I think,
is best avoided.

Re: save-excursion and multi-thread?

[Qiantan Hong]

> I've not chimed in because whether or not I did wouldn't alter the trajectory
> of a young programmer's eventual realization that implementing concurrent
> editing purely in lisp will result in something quite unusable.  Moreover, he
> will discover (in a few years) that programming is essentially a solitary
> pursuit that does not benefit commensurately to the complexity concurrent
> editing adds.  One need only look at how tasks are divvied up at ACM
> competitions to see that programming is really not a team sport.
Emacs is used for many other stuff other than programming.
I don’t aim to make “real programming” concurrent, it obviously doesn't benefit from it.
I’m rather interested in making concurrent edit programmable.
Most exist collaborative editing applications are non-extensible and special purposed.
You have to rebuild Google Slides after you have Google Docs, and so on.
That’s not to mention a significant portion of web applications nowadays
are really just concurrent editors on replicated objects in disguise.
(Disguise by only exposing a very limited/specialized subset of editing operations)
But it seems that if we have a really collaborative Emacs,
then most existing single-user applications could be effortless upgrade to distributed ones,
without needing to reinvent both the application itself and the collaboration support
over and over again. 
And moreover, those applications on Emacs are always programmable and extensible.

Re: save-excursion and multi-thread?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > RS> [Having point behave like a special variable] does not necessarily require
  > RS> changing the way point is implemented at low level.

  > EZ was quite clear that point does not live in lisp-space, and thus cannot
  > leverage the BLV (buffer-local-value) machinery of data.c (which I might add
  > is broken under threads, bug#48990).

That's true, but I doubt it is an obstacle.  The specpdl mechanism has
room for other specialized ways of swapping values in and out.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: save-excursion and multi-thread?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> Date: Tue, 28 Sep 2021 19:33:37 -0400
> 
> I think save-excursion in one thread ought to keep its locally-bound
> point and mark within that thread alone.

IIUC what you mean by "keep ... point alone", this already happens.
(But not for mark: for that, you need save-mark-and-excursion
nowadays.)  That is, when the save-excursion form is exited, point is
guaranteed to be where it was when we entered save-excursion.  So a
thread that wants to be sure its buffer's point will not move due to
preemption should call save-excursion around any code that could
switch threads.

> In other words, it should work like dynamic variable bindings.
> 
> So if a thread does not do save-excursion, it should share the value
> of point in each buffer with the other threads.  However, doing
> save-excursion in a buffer should "locally bind" point (and mark) in
> that buffer in the current thread, and that "local binding" should not
> affect other threads or be affected by them.
> 
> This does not necessarily require changing the way point is
> implemented at low level.  Indeed, I would prefer to avoid such
> far-reaching changes if I were implementing it.  It's enough to
> make it possible for thread-switching to understand save-excursion
> and DTRT.

Note that this might expose background point movements to the user,
depending on which thread runs redisplay.

But if someone wants to work on this, be my guest.  Based on our
experience with Lisp threads since their introduction, I don't
recommend that.  'Nough said.

Re: save-excursion and multi-thread?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > IIUC what you mean by "keep ... point alone", this already happens.
  > (But not for mark: for that, you need save-mark-and-excursion
  > nowadays.)  That is, when the save-excursion form is exited, point is
  > guaranteed to be where it was when we entered save-excursion.

That's half of it.  The other half of what I had in mind is that
changes in point, in other threads, should not affect point inside the
save-excursion form in this thread.

  > Note that this might expose background point movements to the user,
  > depending on which thread runs redisplay.

I see what you are getting at -- but I think it would be good to
make this controllable.

One idea: perhaps there should be a way to control which thread's
context redisplay should occur in.  Perhaps on a per-window basis.

Perhaps each window should be "owned" by a thread, so that point for
that window will be displayed wherever that thread puts it.  If we can
make this happen by default in a way that is usually right, it might
not take much additional work.  Perhaps the default should be
whichever thread last operated on it, in some sense.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: save-excursion and multi-thread?

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Sun, 26 Sep 2021 19:42:25 +0000
> 
> > You can always use save-excursion in the other thread, no?
> Doesn’t looks like it’s protected
> (progn
>   (make-thread
>    (lambda ()
>      (save-excursion
>        (sleep-for 5)
>        (print (point)))))
>   (make-thread
>    (lambda ()
>      (save-excursion
>        (sleep-for 3)
>        (goto-char (point-min))
>        (sleep-for 3)))))
> This prints 1 instead of the point before the first save-excursion enters.

The print should be outside of save-excursion, of course.

> Moreover, the other thread can be the main UI thread, 
> I don’t know how I can protect it (assuming that save-excursion does protect,
> although it actually doesn’t).

Moving point behind the back of the main thread is a bad idea, you
will expose the movement to the user when redisplay hits.

> >> Does that convince you?
> > 
> > Convince me to do what? redesign and reimplement Lisp threads in
> > Emacs with very different design goals?
> Not doing anything, let’s figure out what’s correct first.

I don't see anything that needs fixing here.  Once again, the basic
design was to allow other threads work on other buffers, which is why
each thread has its own current buffer.  This is how this stuff was
designed, so yes, we are talking about design goals.

> I also don’t think a redesign is necessary, we just need to
> restore point state during thread switching, but I haven’t look at detail yet.

In interactive use, moving point in a thread that doesn't perform
redisplay will surprise the user, and I don't see how you could avoid
that.  So I don't think this idea will fly.

Would you mind explaining in more detail what you want to accomplish?
Perhaps there are better ways of doing that that are already
supported, or could be supported with cleaner changes.  I'm not going
to agree to any changes that would make it easier for 2 threads to
work on the same buffer, that way lies madness: buffer changes use a
lot of global machinery behind the scenes.

Re: save-excursion and multi-thread?

[Qiantan Hong]

> Moving point behind the back of the main thread is a bad idea, you
> will expose the movement to the user when redisplay hits.
> 
>>>> Does that convince you?
>>> 
>>> Convince me to do what? redesign and reimplement Lisp threads in
>>> Emacs with very different design goals?
>> Not doing anything, let’s figure out what’s correct first.
> 
> I don't see anything that needs fixing here.  Once again, the basic
> design was to allow other threads work on other buffers, which is why
> each thread has its own current buffer.  This is how this stuff was
> designed, so yes, we are talking about design goals.
> 
>> I also don’t think a redesign is necessary, we just need to
>> restore point state during thread switching, but I haven’t look at detail yet.
> 
> In interactive use, moving point in a thread that doesn't perform
> redisplay will surprise the user, and I don't see how you could avoid
> that.  So I don't think this idea will fly.
I see, thanks for the clarification.

> Would you mind explaining in more detail what you want to accomplish?
> Perhaps there are better ways of doing that that are already
> supported, or could be supported with cleaner changes.  I'm not going
> to agree to any changes that would make it easier for 2 threads to
> work on the same buffer, that way lies madness: buffer changes use a
> lot of global machinery behind the scenes.
I’m experimenting remote commands in my collaborative environment package.
Normally the client just send a message describing the command invocation,
and the server execute it inside process filter (which runs inside main thread).
Server has to wrap save-mark-and-excursion around it, and move point and mark
to those of the client. Then everything works as intended, if the command run itself
without interruption and return in a short time.

Unless the command calls interactive functions like READ-FROM-MINIBUFFER.
Now we need to do a “recursive” remote procedure call, by overriding such interactive function
to send a message back to client and wait for response. 
Now whatever thread that runs such remote command
can’t do anything but waiting (unless we transform the code of such remote command
to async, or CPS, style, but one of my primary design goal is to be able to let
“stock” emacs code just work). It’s therefore infeasible to immediately run such
remote command in process filter as once it calls interactive function, it blocks
main thread and makes server unresponsive.
I therefore have to spawn a new thread for such remote command.
And I do what used to work — wrap the actual command call with save-mark-and-excursion,
setup mark and point and according to client, but this time in a separate thread.
Boom, it doesn’t work at all! For the reason discussed in previous messages.

I guess indirect-buffer will make it sort of work, although looks like it copies lots of stuff
and might be costly? I definitely want remote command call to be as fast as possible
because it’s a foundational facility for building lots of other stuff...

Re: save-excursion and multi-thread?

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Mon, 27 Sep 2021 05:00:01 +0000
> 
> I’m experimenting remote commands in my collaborative environment package.
> Normally the client just send a message describing the command invocation,
> and the server execute it inside process filter (which runs inside main thread).
> Server has to wrap save-mark-and-excursion around it, and move point and mark
> to those of the client. Then everything works as intended, if the command run itself
> without interruption and return in a short time.
> 
> Unless the command calls interactive functions like READ-FROM-MINIBUFFER.
> Now we need to do a “recursive” remote procedure call, by overriding such interactive function
> to send a message back to client and wait for response. 
> Now whatever thread that runs such remote command
> can’t do anything but waiting (unless we transform the code of such remote command
> to async, or CPS, style, but one of my primary design goal is to be able to let
> “stock” emacs code just work). It’s therefore infeasible to immediately run such
> remote command in process filter as once it calls interactive function, it blocks
> main thread and makes server unresponsive.
> I therefore have to spawn a new thread for such remote command.

Why do you need to start a new thread? why not have the filter
function send the message to the client, and arrange for waiting for
the response to that message via the same filter-function method
(perhaps by changing the filter-function temporarily to some other
function)?

IOW, it sounds to me like you need to suspend the execution of the
command until the client answers the question, and you already have
the machinery to handle bidirectional communications between the
client and the server, so just reuse it for asking a question and
receiving the response, before you perform the command.  Am I missing
something?

> I guess indirect-buffer will make it sort of work, although looks like it copies lots of stuff
> and might be costly?

It's definitely not clean enough, I agree.

Re: save-excursion and multi-thread?

[Qiantan Hong]

> Why do you need to start a new thread? why not have the filter
> function send the message to the client, and arrange for waiting for
> the response to that message via the same filter-function method
> (perhaps by changing the filter-function temporarily to some other
> function)?
> 
> IOW, it sounds to me like you need to suspend the execution of the
> command until the client answers the question, and you already have
> the machinery to handle bidirectional communications between the
> client and the server, so just reuse it for asking a question and
> receiving the response, before you perform the command.  Am I missing
> something?
Exactly, however I want the machinery to work for ordinary Emacs “stock” 
commands with minimal effort.

Suppose we opt to rewrite any Emacs command that would be remote-called,
then of course it’s easy to rewrite them so that they immediately return
when they originally READ-FROM-MINIBUFFER while pushing the rest
of the original command as a continuation on some queue 
(this is in fact just a CPS transform). 
The filter function will then execute continuations on the queue upon
receiving response.

We still need to deal with restoring point/mark/etc when switching continuation
(i.e. we need to save state when the command “immediately return” and
restore state when we run the continuation on the queue), but it’s well in our
control and doable.

However I want to use original commands without modification if possible.

My current trick is to run the whole command in new thread, and
override READ-FROM-MINIBUFFER using advice.
It’s conceptually very similar to the above CPS transform solution
(thread-yield is basically (call/cc (lambda (k) (push k continuation-queue))))
However there’s complication that thread-yield might happen at points
out of our control. We thus cannot rely on manually save/restore state
and have to put our hope on some primitive
(just like in a system that switch thread at well defined point one
can implement thread-local dynamic binding in the library, but
has to rely on primitive provided by runtime if there’s preemptive switching
or real parallelism).
Unfortunately such primitive doesn’t seem to exist for point and mark.

Another trick I could think of is to not start a new thread, but
override READ-FROM-MINIBUFFER such that it starts a recursive edit.
Therefore the unfinished state of the command is still on the stack and
we resume it by EXIT-RECURSIVE-EDIT upon receiving response.
It sounds like more madness… and it won’t quite work if the order of receiving
response is different from the order of sending questions.

Re: save-excursion and multi-thread?

[Eli Zaretskii]

> From: Qiantan Hong <qhong@mit.edu>
> CC: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Mon, 27 Sep 2021 06:14:58 +0000
> 
> > IOW, it sounds to me like you need to suspend the execution of the
> > command until the client answers the question, and you already have
> > the machinery to handle bidirectional communications between the
> > client and the server, so just reuse it for asking a question and
> > receiving the response, before you perform the command.  Am I missing
> > something?
> Exactly, however I want the machinery to work for ordinary Emacs “stock” 
> commands with minimal effort.
> 
> Suppose we opt to rewrite any Emacs command that would be remote-called,
> then of course it’s easy to rewrite them so that they immediately return
> when they originally READ-FROM-MINIBUFFER while pushing the rest
> of the original command as a continuation on some queue 
> (this is in fact just a CPS transform). 
> The filter function will then execute continuations on the queue upon
> receiving response.
> 
> We still need to deal with restoring point/mark/etc when switching continuation
> (i.e. we need to save state when the command “immediately return” and
> restore state when we run the continuation on the queue), but it’s well in our
> control and doable.
> 
> However I want to use original commands without modification if possible.

I see your point and your motivation.  However, I don't think that
using threads will make your job any easier.  Threads that run
interactive code (i.e. code that needs to interact with the user and
perform any form of redisplay) are not easy to get working right, and
in fact I'm not aware of any code which accomplishes that.

So I suggest using the "hard way", without threads, as I think it will
eventually turn out to be simpler and less tricky for you to get
right, let alone won't need any changes to Emacs.

Re: save-excursion and multi-thread?

[Tassilo Horn]

Qiantan Hong <qhong@mit.edu> writes:

> I tried
> (make-thread
>  (lambda ()
>    (save-excursion (goto-char (point-min)) (sleep-for 1))))
> In *scratch*.
>
> It seems that after the thread yields (because of sleep-for) it
> doesn’t restore point position, and after 1 second cursor is reseted
> to the original position,

The `sleep-for' is part of the thread's function, and also contained in
the `save-excursion'.  So how could it be different?

> discarding any of my movements during this 1 second.
>
> Is this how save-excursion supposed to work?
> It’s inconsistent with the behavior of special variable + dynamic
> bindings, which works currently under multi-thread.

How are those related?

> What’s the supposed way to have “thread-local” movements?

The thing is, right now there are threads in emacs but no actual
parallelism.  There's ever only one thread active and emacs switches
between the existing threads at certain conditions, e.g., when one
thread waits for input or a condition variable.  I guess another one
might be `sleep-for' which would obviously make a lot of sense.

Oh, ok, I think now I got your question.  If `point' was a special
variable, and let-bindings of special variables in a thread are local to
this thread, then you would naturally have thread-local movements.

Right now, point is a field in the buffer data structure, so there's
just one per buffer shared between all threads.  I think as long as your
thread doesn't use `sit-for', `sleep-for', or reads input, you can be
sure it won't be interrupted.  If it does, you cannot assume point is
still there where your thread moved it.

Bye,
Tassilo

Re: save-excursion and multi-thread?

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Date: Sun, 26 Sep 2021 20:21:39 +0200
> 
> Qiantan Hong <qhong@mit.edu> writes:
> 
> > I tried
> > (make-thread
> >  (lambda ()
> >    (save-excursion (goto-char (point-min)) (sleep-for 1))))
> > In *scratch*.
> >
> > It seems that after the thread yields (because of sleep-for) it
> > doesn’t restore point position, and after 1 second cursor is reseted
> > to the original position,
> 
> The `sleep-for' is part of the thread's function, and also contained in
> the `save-excursion'.  So how could it be different?
> 
> > discarding any of my movements during this 1 second.
> >
> > Is this how save-excursion supposed to work?
> > It’s inconsistent with the behavior of special variable + dynamic
> > bindings, which works currently under multi-thread.
> 
> How are those related?
> 
> > What’s the supposed way to have “thread-local” movements?
> 
> The thing is, right now there are threads in emacs but no actual
> parallelism.  There's ever only one thread active and emacs switches
> between the existing threads at certain conditions, e.g., when one
> thread waits for input or a condition variable.  I guess another one
> might be `sleep-for' which would obviously make a lot of sense.
> 
> Oh, ok, I think now I got your question.  If `point' was a special
> variable, and let-bindings of special variables in a thread are local to
> this thread, then you would naturally have thread-local movements.
> 
> I think as long as your thread doesn't use `sit-for', `sleep-for',
> or reads input, you can be sure it won't be interrupted.

A thread can also be preempted by one of the following:

  . calls to thread primitives such as thread-yield
  . calls to primitives that read from processes, such as
    accept-process-output, and anything that calls these
  . becoming idle (which is a kind of "reading input", but maybe not
    everyone will realize that
  . waiting for X selections to do their thing

Re: save-excursion and multi-thread?

[Stefan Monnier]

> Is this how save-excursion supposed to work?

Good question.

> It’s inconsistent with the behavior of special variable + dynamic bindings,
> which works currently under multi-thread.

You might be right and maybe we should undo/redo such buffer movements when
context-switching, just like we do for the let bindings.

We already implement such undo/redo of this and similar forms in
a different yet related context: when we evaluate a user-specified
expression in a given stack frame (from the *Backtrace* debugger).
See `backtrace-eval`.

This said, your scenario smells like a race-condition to me, so maybe we
should instead "lock" the buffer's point while we're inside the
`save-excursion`.


        Stefan