sending data to an asynchronous process

sending data to an asynchronous process

[Nicolas Graner]

I am writing a program that creates audio samples in an emacs
buffer, then sends them to an external program (sox) to play in
the background while I continue working with emacs. Part of the
code is roughly as follows:

(setq process
      (let ((process-connection-type nil))
	(start-process "my-process" nil
		       "sox" "-r" rate "-c" channels "-b" bits "-e" encoding "-q" "-d")))
(process-send-region process start end)
(process-send-eof process)

The sound plays as expected, but process-send-region does not
return until about half a second before the sound finishes
playing. This means that if I send several minutes of audio,
emacs is stuck during all that time, which completely defeats the
purpose of an asynchronous process. Using process-send-string
instead of process-send-region makes no difference.

I suppose it has to do with buffering on the pipe, bit I don't
know if I have any control over this. I tried to use stdbuf, as in:
  (start-process "my-process" nil "stdbuf" "-i10M" "-o10M" "sox" ...
which makes no difference.

One possible approach would be to send the data in small chunks,
arranging for some sort of sentinel to be called when a chunk has
been consumed, to send the nex one. Is there any way to do this?

I could also write the data into a temporary file and let sox
read from it but that seems pretty inefficient and would require
careful cleanup to not clutter the disk.

The solution I found is to use a synchronous process:
  (call-process-region start end "sox" nil 0 nil ...
where the fifth argument 0 means to discard process output and
not wait for completion. This works and does essentially what I
want, except that I don't know the PID of the spawned process so
I cannot communicate with it, e.g. to stop it before the end or
determine whether it's still runnin or not.

I'd appreciate any insights and/or solutions?

FYI, running emacs 28.0.50 on debian 5.5.

Thanks,
Nicolas

Re: sending data to an asynchronous process

[Jean Louis]

* Nicolas Graner <nicolas@graner.name> [2021-04-02 13:21]:
> I am writing a program that creates audio samples in an emacs
> buffer, then sends them to an external program (sox) to play in
> the background while I continue working with emacs. Part of the
> code is roughly as follows:
> 
> (setq process
>       (let ((process-connection-type nil))
> 	(start-process "my-process" nil
> 		       "sox" "-r" rate "-c" channels "-b" bits "-e" encoding "-q" "-d")))
> (process-send-region process start end)
> (process-send-eof process)

(process-id (start-process "sleep" nil "sleep" "100")) → 14296



-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://rms-support-letter.github.io/

Re: sending data to an asynchronous process

[Jean Louis]

* Nicolas Graner <nicolas@graner.name> [2021-04-02 13:21]:
> I am writing a program that creates audio samples in an emacs
> buffer, then sends them to an external program (sox) to play in
> the background while I continue working with emacs. Part of the
> code is roughly as follows:
> 
(setq process
      (let ((process-connection-type nil))
	(start-process "my-process" nil
		       "sox" "-r" rate "-c" channels "-b" bits "-e" encoding "-q" "-d")))
(process-send-region process start end)
(process-send-eof process)

For process-send-region, it says:

If PROCESS is a non-blocking network process that hasn’t been fully
set up yet, this function will block until socket setup has completed.

Maybe that applies.

It would be nice to show all the function you use there.

> The sound plays as expected, but process-send-region does not
> return until about half a second before the sound finishes
> playing. This means that if I send several minutes of audio,
> emacs is stuck during all that time, which completely defeats the
> purpose of an asynchronous process. Using process-send-string
> instead of process-send-region makes no difference.

I would like to know if you play files or you feed data to sox to
play files, where does that come from? Is it maybe an elevator
music from an UDP port?

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://rms-support-letter.github.io/

Re: sending data to an asynchronous process

[Nicolas Graner]

Jean Louis <bugs@gnu.support> wrote on 2021-04-02 20:18:
> * Nicolas Graner <nicolas@graner.name> [2021-04-02 13:21]:
>> I am writing a program that creates audio samples in an emacs
>> buffer, then sends them to an external program (sox) to play in
>> the background while I continue working with emacs. Part of the
>> code is roughly as follows:
>> 
> (setq process
>       (let ((process-connection-type nil))
> 	(start-process "my-process" nil
> 		       "sox" "-r" rate "-c" channels "-b" bits "-e" encoding "-q" "-d")))
> (process-send-region process start end)
> (process-send-eof process)
>
> For process-send-region, it says:
>
> If PROCESS is a non-blocking network process that hasn’t been fully
> set up yet, this function will block until socket setup has completed.
>
> Maybe that applies.

No. No network involved whatsoever. Just running /usr/bin/sox which is a
regular file.

> It would be nice to show all the function you use there.

No other function is needed. If you want to reproduce the problem, just
fill a buffer with 1 MB of random binary data and evaluate this slightly
modified version of the above code:

(setq process
      (let ((process-connection-type nil))
	(start-process "my-process" nil
		       "/usr/bin/sox" "-r" "44100" "-c" "2" "-b" "32"
      "-e" "signed-integer" "-t" "raw" "-" "-q" "-d")))

(process-send-region process (point-min) (point-max))

(process-send-eof process)

If you have sox installed, be prepared to hear some horrible random noise :)
You will notice that the first expression above returns immediately as
expected, but the second (process-send-region ...) does not. That's the
problem I'm trying to solve.

> I would like to know if you play files or you feed data to sox to
> play files, where does that come from? Is it maybe an elevator
> music from an UDP port?

The data originally came from files that I read into a buffer and did
some processing on. At the time of calling the above code, all the data
is sitting quietly in an emacs buffer. No network, disk access or
anything else involved. Just sending the contents of a buffer through a
pipe to a program that was started as an asynchronous process.

Nicolas

Re: sending data to an asynchronous process

[Jean Louis]

* Nicolas Graner <nicolas@graner.name> [2021-04-02 21:37]:
> Jean Louis <bugs@gnu.support> wrote on 2021-04-02 20:18:
> > * Nicolas Graner <nicolas@graner.name> [2021-04-02 13:21]:
> >> I am writing a program that creates audio samples in an emacs
> >> buffer, then sends them to an external program (sox) to play in
> >> the background while I continue working with emacs. Part of the
> >> code is roughly as follows:
> >> 
> > (setq process
> >       (let ((process-connection-type nil))
> > 	(start-process "my-process" nil
> > 		       "sox" "-r" rate "-c" channels "-b" bits "-e" encoding "-q" "-d")))
> > (process-send-region process start end)
> > (process-send-eof process)
> >
> > For process-send-region, it says:
> >
> > If PROCESS is a non-blocking network process that hasn’t been fully
> > set up yet, this function will block until socket setup has completed.
> >
> > Maybe that applies.

(defun rcd-command-output-from-input (program input &rest args)
  "Returns output from PROGRAM INPUT with optional ARGS"
  (let* ((output (with-temp-buffer
		  (insert input)
		  (apply #'call-process-region nil nil program t t nil args)
		  (buffer-string))))
    output))

(rcd-command-output-from-input "sox" (buffer-string)
			       "-r" "44100"
			       "-c" "2"
			       "-b" "32"
			       "-e" "signed-integer"
			       "-t" "raw"
			       "-"
			       "-q"
			       "-d") → ""

I have tried this above, it plays my buffer. But it is not
asynchronous.

There is library async.el, but I am not sure if it can receive from
buffer directly, you can however, make your function to feed the
buffer first into a file.

async-start-process is an autoloaded compiled Lisp function in
‘async.el’.

(async-start-process NAME PROGRAM FINISH-FUNC &rest PROGRAM-ARGS)

Start the executable PROGRAM asynchronously named NAME.  See ‘async-start’.
PROGRAM is passed PROGRAM-ARGS, calling FINISH-FUNC with the
process object when done.  If FINISH-FUNC is nil, the future
object will return the process object when the program is
finished.  Set DEFAULT-DIRECTORY to change PROGRAM’s current
working directory.

async-send is a compiled Lisp function in ‘async.el’.

(async-send &rest ARGS)

Send the given messages to the asychronous Emacs PROCESS.


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://rms-support-letter.github.io/

Re: sending data to an asynchronous process

[Jean Louis]

* Jean Louis <bugs@gnu.support> [2021-04-02 21:58]:
> (defun rcd-command-output-from-input (program input &rest args)
>   "Returns output from PROGRAM INPUT with optional ARGS"
>   (let* ((output (with-temp-buffer
> 		  (insert input)
> 		  (apply #'call-process-region nil nil program t t nil args)
> 		  (buffer-string))))
>     output))
> 
> (rcd-command-output-from-input "sox" (buffer-string)
> 			       "-r" "44100"
> 			       "-c" "2"
> 			       "-b" "32"
> 			       "-e" "signed-integer"
> 			       "-t" "raw"
> 			       "-"
> 			       "-q"
> 			       "-d") → ""

I can imagine that if the above command is added as some hook on
load-file or require, then we could sound-alize loading of Emacs
libraries.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://rms-support-letter.github.io/

Re: sending data to an asynchronous process

[Eli Zaretskii]

> From: Nicolas Graner <nicolas@graner.name>
> Date: Fri, 02 Apr 2021 12:20:26 +0200
> 
> I am writing a program that creates audio samples in an emacs
> buffer, then sends them to an external program (sox) to play in
> the background while I continue working with emacs. Part of the
> code is roughly as follows:
> 
> (setq process
>       (let ((process-connection-type nil))
> 	(start-process "my-process" nil
> 		       "sox" "-r" rate "-c" channels "-b" bits "-e" encoding "-q" "-d")))
> (process-send-region process start end)
> (process-send-eof process)
> 
> The sound plays as expected, but process-send-region does not
> return until about half a second before the sound finishes
> playing. This means that if I send several minutes of audio,
> emacs is stuck during all that time, which completely defeats the
> purpose of an asynchronous process.

Asynchronous subprocesses mean that their processing and reading their
output are asynchronous.  Writing to the subprocess is always
synchronous.  It cannot be any other way, really, unless we start
additional threads to write to the subprocess (which then causes
problems because the Emacs Lisp machine cannot tolerate more than one
thread that has access to Lisp data).  What Emacs does is read output
from subprocesses while it loops sending the data in chunks, so that
at least that part would work -- but that doesn't help in your case.

> The solution I found is to use a synchronous process:
>   (call-process-region start end "sox" nil 0 nil ...
> where the fifth argument 0 means to discard process output and
> not wait for completion.

call-process-region writes the data to a temporary file, and then
invokes the subprocess.  So if you do the same "by hand" and then call
start-process, you will have the best of all worlds.  IOW, the
temporary file solution you didn't like is exactly what
call-process-region does under the hood, and so efficiency is not the
issue here.  As for cleaning up the temporary files, that isn't really
a problem, is it?

Re: sending data to an asynchronous process

[Nicolas Graner]

Eli Zaretskii <eliz@gnu.org> wrote on 2021-04-02 22:15:
> call-process-region writes the data to a temporary file, and then
> invokes the subprocess.  So if you do the same "by hand" and then call
> start-process, you will have the best of all worlds.  IOW, the
> temporary file solution you didn't like is exactly what
> call-process-region does under the hood, and so efficiency is not the
> issue here.  As for cleaning up the temporary files, that isn't really
> a problem, is it?

Great, that's exactly the information I needed. I've rewritten the code
using a temporary file and it works like a charm.
Many thanks!

Nicolas