all messages for Emacs-related lists mirrored at yhetil.org
 help / color / mirror / code / Atom feed
* call-process blocks the active thread
@ 2017-09-08  3:00 Elias Mårtenson
  2017-09-08  3:56 ` Eric Abrahamsen
  2017-09-08  7:01 ` Eli Zaretskii
  0 siblings, 2 replies; 10+ messages in thread
From: Elias Mårtenson @ 2017-09-08  3:00 UTC (permalink / raw)
  To: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 437 bytes --]

I was trying to use ‘call-process’ in a thread in order to not have to deal
with process buffers and the like while still being able to run an external
program in the background.

As it turns out, ‘call-process’ hangs all of Emacs while the external
program is run.

To reproduce, simply type M-: (make-thread (lambda () (call-process "sleep"
nil nil nil "2")))

This will hang Emacs for 2 seconds.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 575 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: call-process blocks the active thread
  2017-09-08  3:00 call-process blocks the active thread Elias Mårtenson
@ 2017-09-08  3:56 ` Eric Abrahamsen
  2017-09-08  7:01 ` Eli Zaretskii
  1 sibling, 0 replies; 10+ messages in thread
From: Eric Abrahamsen @ 2017-09-08  3:56 UTC (permalink / raw)
  To: emacs-devel

Elias Mårtenson <lokedhs@gmail.com> writes:

> I was trying to use ‘call-process’ in a thread in order to not have to deal with process buffers and the like while still being able to run an external program in the
> background.
>
> As it turns out, ‘call-process’ hangs all of Emacs while the external program is run.
>
> To reproduce, simply type M-: (make-thread (lambda () (call-process "sleep" nil nil nil "2")))
>
> This will hang Emacs for 2 seconds.

`call-process' will block Emacs no matter what. Threads are good for
encapsulating environment, but they don't really change the nature of
synchronous/asynchronous external processes. You still have to do:

(let ((proc (start-process <etc>)))
  (while (process-live-p proc)
    (accept-process-output proc)))
    
caveat: I am not a thread expert

Eric




^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: call-process blocks the active thread
  2017-09-08  3:00 call-process blocks the active thread Elias Mårtenson
  2017-09-08  3:56 ` Eric Abrahamsen
@ 2017-09-08  7:01 ` Eli Zaretskii
  2017-09-08  7:45   ` Stefan Monnier
  2017-09-08 10:36   ` Elias Mårtenson
  1 sibling, 2 replies; 10+ messages in thread
From: Eli Zaretskii @ 2017-09-08  7:01 UTC (permalink / raw)
  To: Elias Mårtenson; +Cc: emacs-devel

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Fri, 8 Sep 2017 11:00:42 +0800
> 
> As it turns out, ‘call-process’ hangs all of Emacs while the external program is run.

No, it hangs the calling thread, and the other threads wait for the
calling thread to yield.  To avoid that, use start-process instead.

This is not a bug.



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: call-process blocks the active thread
  2017-09-08  7:01 ` Eli Zaretskii
@ 2017-09-08  7:45   ` Stefan Monnier
  2017-09-08  8:06     ` Eli Zaretskii
  2017-09-08 10:36   ` Elias Mårtenson
  1 sibling, 1 reply; 10+ messages in thread
From: Stefan Monnier @ 2017-09-08  7:45 UTC (permalink / raw)
  To: emacs-devel

>> As it turns out, ‘call-process’ hangs all of Emacs while the external
>> program is run.
> No, it hangs the calling thread, and the other threads wait for the
> calling thread to yield.  To avoid that, use start-process instead.
> This is not a bug.

I think we shouldn't consider this as a bug report but as
a feature request.


        Stefan




^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: call-process blocks the active thread
  2017-09-08  7:45   ` Stefan Monnier
@ 2017-09-08  8:06     ` Eli Zaretskii
  0 siblings, 0 replies; 10+ messages in thread
From: Eli Zaretskii @ 2017-09-08  8:06 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: emacs-devel

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 08 Sep 2017 03:45:49 -0400
> 
> I think we shouldn't consider this as a bug report but as
> a feature request.

Proposals for implementing such a feature are welcome (probably to
bug-gnu-emacs, not here).



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: call-process blocks the active thread
  2017-09-08  7:01 ` Eli Zaretskii
  2017-09-08  7:45   ` Stefan Monnier
@ 2017-09-08 10:36   ` Elias Mårtenson
  2017-09-08 11:27     ` Philipp Stephani
  2017-09-08 12:16     ` Eli Zaretskii
  1 sibling, 2 replies; 10+ messages in thread
From: Elias Mårtenson @ 2017-09-08 10:36 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

On 8 September 2017 at 15:01, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Elias Mårtenson <lokedhs@gmail.com>
> > Date: Fri, 8 Sep 2017 11:00:42 +0800
> >
> > As it turns out, ‘call-process’ hangs all of Emacs while the external
> program is run.
>
> No, it hangs the calling thread, and the other threads wait for the
> calling thread to yield.  To avoid that, use start-process instead.
>
> This is not a bug.
>

I'm not entirely sure why you feel that distinction is important. At the
end of the day, it's ‘start-process’ that doesn't yield while the external
process runs. Is there a reason why it can't be changed so that it does?

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 1166 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: call-process blocks the active thread
  2017-09-08 10:36   ` Elias Mårtenson
@ 2017-09-08 11:27     ` Philipp Stephani
  2017-09-08 12:16     ` Eli Zaretskii
  1 sibling, 0 replies; 10+ messages in thread
From: Philipp Stephani @ 2017-09-08 11:27 UTC (permalink / raw)
  To: Elias Mårtenson, Eli Zaretskii; +Cc: emacs-devel

[-- Attachment #1: Type: text/plain, Size: 1082 bytes --]

Elias Mårtenson <lokedhs@gmail.com> schrieb am Fr., 8. Sep. 2017 um
13:00 Uhr:

> On 8 September 2017 at 15:01, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: Elias Mårtenson <lokedhs@gmail.com>
>> > Date: Fri, 8 Sep 2017 11:00:42 +0800
>> >
>> > As it turns out, ‘call-process’ hangs all of Emacs while the external
>> program is run.
>>
>> No, it hangs the calling thread, and the other threads wait for the
>> calling thread to yield.  To avoid that, use start-process instead.
>>
>> This is not a bug.
>>
>
> I'm not entirely sure why you feel that distinction is important. At the
> end of the day, it's ‘start-process’ that doesn't yield while the external
> process runs. Is there a reason why it can't be changed so that it does?
>

Emacs threads can't run in parallel because they share global mutable state
(buffers etc.) without synchronization. `call-process' is guaranteed to not
affect the global state in "significant" ways, therefore it can't yield to
threads that could change the global state.
It's not possible to fix this.

[-- Attachment #2: Type: text/html, Size: 1799 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: call-process blocks the active thread
  2017-09-08 10:36   ` Elias Mårtenson
  2017-09-08 11:27     ` Philipp Stephani
@ 2017-09-08 12:16     ` Eli Zaretskii
  2017-09-08 16:12       ` Tom Tromey
  1 sibling, 1 reply; 10+ messages in thread
From: Eli Zaretskii @ 2017-09-08 12:16 UTC (permalink / raw)
  To: Elias Mårtenson; +Cc: emacs-devel

> From: Elias Mårtenson <lokedhs@gmail.com>
> Date: Fri, 8 Sep 2017 18:36:17 +0800
> Cc: emacs-devel <emacs-devel@gnu.org>
> 
>  This is not a bug.
> 
> I'm not entirely sure why you feel that distinction is important.

I thought you were reporting it as a bug, so it was important for me
to say Emacs was working as designed in this case.

> At the end of the day, it's ‘start-process’ that
> doesn't yield while the external process runs. Is there a reason why it can't be changed so that it does?

AFAIU, it would require significant changes in how call-process works,
both how it waits for the child process to exit and how it reads the
sub-process output.  And start-process already has all that figured
out, so I think it's easier to just use it, especially since threading
already supports start-process, in that the calling thread will yield
automatically while it reads sub-process output and/or waits for it to
exit.

Of course, it's possible I don't see some clever way to make
call-process thread-aware, which is why I said that if someone has
concrete ideas for how to do that, they are encouraged to speak up.



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: call-process blocks the active thread
  2017-09-08 12:16     ` Eli Zaretskii
@ 2017-09-08 16:12       ` Tom Tromey
  2017-09-08 18:43         ` Tom Tromey
  0 siblings, 1 reply; 10+ messages in thread
From: Tom Tromey @ 2017-09-08 16:12 UTC (permalink / raw)
  To: emacs-devel

Eli> Of course, it's possible I don't see some clever way to make
Eli> call-process thread-aware, which is why I said that if someone has
Eli> concrete ideas for how to do that, they are encouraged to speak up.

I looked at it briefly.  I have two ideas.

Maybe the simplest idea is to rewrite call-process in Lisp, using
make-process and accept-process-output.  This would automatically allow
other threads to run.  I didn't really investigate this in depth (so for
instance I don't know if this can be done without adding extensions to
make-process), but it seems to me that it would be nice to migrate code
out of C when possible.


Otherwise, I think call_process can be made to yield.  A bit of
background (I know Eli doesn't need this but maybe others do):

Only one thread can run Lisp at a time.  A global lock enforces this.
However, threads can run non-Lisp code while not holding the global lock
-- this is why Emacs can wait for input while Lisp runs in another
thread.

The way this is done is that a thread calls flush_stack_call_func to
save registers on the stack (so that GC will do the right thing); this
calls a provided callback, and the callback handles releasing the global
lock, doing whatever non-Lisp thing it wants to do, and then
re-acquiring the lock before returning.

A good example of this is thread.c:thread_select, which winds up in
thread.c:really_call_select.

So, the idea for call-process would be to modify the inner loop of
call_process to do this.  I think the best spot would be to call
flush_stack_call_func around the call to emacs_read_quit.

However, the tricky thing here is that call-process relies on some
global state.  For example, it assumes that current-buffer will not
change while it is working.  So, after emacs_read_quit returns, this
code would have to be careful to restore the state it needs.  The hard
part of the project is figuring out which bits of state must be
preserved.

It might also be reasonable to wrap the wait_for_termination call in a
lock-releasing function.

Tom



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: call-process blocks the active thread
  2017-09-08 16:12       ` Tom Tromey
@ 2017-09-08 18:43         ` Tom Tromey
  0 siblings, 0 replies; 10+ messages in thread
From: Tom Tromey @ 2017-09-08 18:43 UTC (permalink / raw)
  To: Tom Tromey; +Cc: emacs-devel

Tom> Maybe the simplest idea is to rewrite call-process in Lisp, using
Tom> make-process and accept-process-output.

I gave this a try, and overall it looks pretty easy.
However, I am going to have to add call-process -like "(:file ...)"
handling to make-process.

Tom



^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2017-09-08 18:43 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-09-08  3:00 call-process blocks the active thread Elias Mårtenson
2017-09-08  3:56 ` Eric Abrahamsen
2017-09-08  7:01 ` Eli Zaretskii
2017-09-08  7:45   ` Stefan Monnier
2017-09-08  8:06     ` Eli Zaretskii
2017-09-08 10:36   ` Elias Mårtenson
2017-09-08 11:27     ` Philipp Stephani
2017-09-08 12:16     ` Eli Zaretskii
2017-09-08 16:12       ` Tom Tromey
2017-09-08 18:43         ` Tom Tromey

Code repositories for project(s) associated with this external index

	https://git.savannah.gnu.org/cgit/emacs.git
	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.