call-process blocks the active thread

call-process blocks the active thread

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 437 bytes --]

I was trying to use ‘call-process’ in a thread in order to not have to deal
with process buffers and the like while still being able to run an external
program in the background.

As it turns out, ‘call-process’ hangs all of Emacs while the external
program is run.

To reproduce, simply type M-: (make-thread (lambda () (call-process "sleep"
nil nil nil "2")))

This will hang Emacs for 2 seconds.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 575 bytes --]

Re: call-process blocks the active thread

[Eric Abrahamsen]

Elias Mårtenson <lokedhs@gmail.com> writes:

> I was trying to use ‘call-process’ in a thread in order to not have to deal with process buffers and the like while still being able to run an external program in the
> background.
>
> As it turns out, ‘call-process’ hangs all of Emacs while the external program is run.
>
> To reproduce, simply type M-: (make-thread (lambda () (call-process "sleep" nil nil nil "2")))
>
> This will hang Emacs for 2 seconds.

`call-process' will block Emacs no matter what. Threads are good for
encapsulating environment, but they don't really change the nature of
synchronous/asynchronous external processes. You still have to do:

(let ((proc (start-process <etc>)))
  (while (process-live-p proc)
    (accept-process-output proc)))
    
caveat: I am not a thread expert

Eric

Re: call-process blocks the active thread

[Eli Zaretskii]

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Fri, 8 Sep 2017 11:00:42 +0800
> 
> As it turns out, ‘call-process’ hangs all of Emacs while the external program is run.

No, it hangs the calling thread, and the other threads wait for the
calling thread to yield.  To avoid that, use start-process instead.

This is not a bug.

Re: call-process blocks the active thread

[Stefan Monnier]

>> As it turns out, ‘call-process’ hangs all of Emacs while the external
>> program is run.
> No, it hangs the calling thread, and the other threads wait for the
> calling thread to yield.  To avoid that, use start-process instead.
> This is not a bug.

I think we shouldn't consider this as a bug report but as
a feature request.


        Stefan

Re: call-process blocks the active thread

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 08 Sep 2017 03:45:49 -0400
> 
> I think we shouldn't consider this as a bug report but as
> a feature request.

Proposals for implementing such a feature are welcome (probably to
bug-gnu-emacs, not here).

Re: call-process blocks the active thread

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

On 8 September 2017 at 15:01, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Elias Mårtenson <lokedhs@gmail.com>
> > Date: Fri, 8 Sep 2017 11:00:42 +0800
> >
> > As it turns out, ‘call-process’ hangs all of Emacs while the external
> program is run.
>
> No, it hangs the calling thread, and the other threads wait for the
> calling thread to yield.  To avoid that, use start-process instead.
>
> This is not a bug.
>

I'm not entirely sure why you feel that distinction is important. At the
end of the day, it's ‘start-process’ that doesn't yield while the external
process runs. Is there a reason why it can't be changed so that it does?

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 1166 bytes --]

Re: call-process blocks the active thread

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 1082 bytes --]

Elias Mårtenson <lokedhs@gmail.com> schrieb am Fr., 8. Sep. 2017 um
13:00 Uhr:

> On 8 September 2017 at 15:01, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: Elias Mårtenson <lokedhs@gmail.com>
>> > Date: Fri, 8 Sep 2017 11:00:42 +0800
>> >
>> > As it turns out, ‘call-process’ hangs all of Emacs while the external
>> program is run.
>>
>> No, it hangs the calling thread, and the other threads wait for the
>> calling thread to yield.  To avoid that, use start-process instead.
>>
>> This is not a bug.
>>
>
> I'm not entirely sure why you feel that distinction is important. At the
> end of the day, it's ‘start-process’ that doesn't yield while the external
> process runs. Is there a reason why it can't be changed so that it does?
>

Emacs threads can't run in parallel because they share global mutable state
(buffers etc.) without synchronization. `call-process' is guaranteed to not
affect the global state in "significant" ways, therefore it can't yield to
threads that could change the global state.
It's not possible to fix this.

[-- Attachment #2: Type: text/html, Size: 1799 bytes --]

Re: call-process blocks the active thread

[Eli Zaretskii]

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Fri, 8 Sep 2017 18:36:17 +0800
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
>  This is not a bug.
> 
> I'm not entirely sure why you feel that distinction is important.

I thought you were reporting it as a bug, so it was important for me
to say Emacs was working as designed in this case.

> At the end of the day, it's ‘start-process’ that
> doesn't yield while the external process runs. Is there a reason why it can't be changed so that it does?

AFAIU, it would require significant changes in how call-process works,
both how it waits for the child process to exit and how it reads the
sub-process output.  And start-process already has all that figured
out, so I think it's easier to just use it, especially since threading
already supports start-process, in that the calling thread will yield
automatically while it reads sub-process output and/or waits for it to
exit.

Of course, it's possible I don't see some clever way to make
call-process thread-aware, which is why I said that if someone has
concrete ideas for how to do that, they are encouraged to speak up.

Re: call-process blocks the active thread

[Tom Tromey]

Eli> Of course, it's possible I don't see some clever way to make
Eli> call-process thread-aware, which is why I said that if someone has
Eli> concrete ideas for how to do that, they are encouraged to speak up.

I looked at it briefly.  I have two ideas.

Maybe the simplest idea is to rewrite call-process in Lisp, using
make-process and accept-process-output.  This would automatically allow
other threads to run.  I didn't really investigate this in depth (so for
instance I don't know if this can be done without adding extensions to
make-process), but it seems to me that it would be nice to migrate code
out of C when possible.


Otherwise, I think call_process can be made to yield.  A bit of
background (I know Eli doesn't need this but maybe others do):

Only one thread can run Lisp at a time.  A global lock enforces this.
However, threads can run non-Lisp code while not holding the global lock
-- this is why Emacs can wait for input while Lisp runs in another
thread.

The way this is done is that a thread calls flush_stack_call_func to
save registers on the stack (so that GC will do the right thing); this
calls a provided callback, and the callback handles releasing the global
lock, doing whatever non-Lisp thing it wants to do, and then
re-acquiring the lock before returning.

A good example of this is thread.c:thread_select, which winds up in
thread.c:really_call_select.

So, the idea for call-process would be to modify the inner loop of
call_process to do this.  I think the best spot would be to call
flush_stack_call_func around the call to emacs_read_quit.

However, the tricky thing here is that call-process relies on some
global state.  For example, it assumes that current-buffer will not
change while it is working.  So, after emacs_read_quit returns, this
code would have to be careful to restore the state it needs.  The hard
part of the project is figuring out which bits of state must be
preserved.

It might also be reasonable to wrap the wait_for_termination call in a
lock-releasing function.

Tom

Re: call-process blocks the active thread

[Tom Tromey]

Tom> Maybe the simplest idea is to rewrite call-process in Lisp, using
Tom> make-process and accept-process-output.

I gave this a try, and overall it looks pretty easy.
However, I am going to have to add call-process -like "(:file ...)"
handling to make-process.

Tom