emacsclient in elisp

emacsclient in elisp

[Daniel Mendler]

I am looking for an elisp implementation of emacsclient. Is such a
functionality already build into Emacs? This would allow to start a
separate Emacs instance which can then be used as an asynchronous worker.

I am starting the server like this:
(call-process "/usr/bin/emacs" nil "*server*" nil "-Q"
"--daemon=myserver" "-l" "myserver.el")

Then jobs can be pushed onto the server:
(call-process "/usr/bin/emacsclient" nil "*server*" nil "-s" "myserver"
"-e" "(...)"))

Instead of starting an emacsclient process, I would like to directly
communicate with the server.

Daniel

Re: emacsclient in elisp

[Stefan Monnier]

> I am looking for an elisp implementation of emacsclient.
> Is such a functionality already build into Emacs?

No.  I think the reason goes as follows: for the intended purpose of
`emacsclient`, the time needed to start Emacs itself makes an ELisp
implementation useless, and for other purposes you can generally have
a separate server process per client process and it's just as easy if
not easier to then use `start-process` and a ad-hoc communication
protocol rather than having to reuse the protocol used for `emacsclient`
(which is neither very convenient nor very extensible).

That also saves you from the problem of choosing a socket name/number
for the communication between the two processes and dealing with the
associated problems of security.


        Stefan

Re: emacsclient in elisp

[Daniel Mendler]

On 5/19/21 3:41 PM, Stefan Monnier wrote:
> No.  I think the reason goes as follows: for the intended purpose of
> `emacsclient`, the time needed to start Emacs itself makes an ELisp
> implementation useless...

Of course.

> and for other purposes you can generally have
> a separate server process per client process and it's just as easy if
> not easier to then use `start-process` and a ad-hoc communication
> protocol...

When I a start an Emacs process in the background, which IPC protocol do
you recommend to communicate with it? Should I start a separate custom
TCP server on the daemon and then communicate with it using a TCP client
(inventing my own protocol and with the implied security issues)?

I cannot start the Emacs in the background and communicate with it by
sending/receiving data via stdin/stdout, since Emacs does not offer an
asynchronous facility to read from stdin. Asynchronous reading would be
needed on the daemon. If I am calling `read-from-minibuffer` there it
reads from stdin but blocks the process. Is this correct?

For my use case it is necessary that I start an Emacs process in the
background, which stays alive for some time, maintaining some state.
Then this process should be controlled asynchronously.

There exists the async package, which starts a new Emacs process per
future if I understood correctly. This is not sufficient for what I am
trying.

> rather than having to reuse the protocol used for `emacsclient`
> (which is neither very convenient nor very extensible).

I am not sure I agree. I can send sexps and return sexps which I
consider a sufficiently extensible protocol. But from what I've seen the
client is buggy - it is for example not possible to send strings
containing \0 bytes.

Daniel

Re: emacsclient in elisp

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1997 bytes --]

Daniel Mendler <mail@daniel-mendler.de> writes:


take a look at epc --- > On 5/19/21 3:41 PM, Stefan Monnier wrote:
>> No.  I think the reason goes as follows: for the intended purpose of
>> `emacsclient`, the time needed to start Emacs itself makes an ELisp
>> implementation useless...
>
> Of course.
>
>> and for other purposes you can generally have
>> a separate server process per client process and it's just as easy if
>> not easier to then use `start-process` and a ad-hoc communication
>> protocol...
>
> When I a start an Emacs process in the background, which IPC protocol do
> you recommend to communicate with it? Should I start a separate custom
> TCP server on the daemon and then communicate with it using a TCP client
> (inventing my own protocol and with the implied security issues)?
>
> I cannot start the Emacs in the background and communicate with it by
> sending/receiving data via stdin/stdout, since Emacs does not offer an
> asynchronous facility to read from stdin. Asynchronous reading would be
> needed on the daemon. If I am calling `read-from-minibuffer` there it
> reads from stdin but blocks the process. Is this correct?
>
> For my use case it is necessary that I start an Emacs process in the
> background, which stays alive for some time, maintaining some state.
> Then this process should be controlled asynchronously.
>
> There exists the async package, which starts a new Emacs process per
> future if I understood correctly. This is not sufficient for what I am
> trying.
>
>> rather than having to reuse the protocol used for `emacsclient`
>> (which is neither very convenient nor very extensible).
>
> I am not sure I agree. I can send sexps and return sexps which I
> consider a sufficiently extensible protocol. But from what I've seen the
> client is buggy - it is for example not possible to send strings
> containing \0 bytes.
>
> Daniel
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: emacsclient in elisp

[Daniel Mendler]

On 5/19/21 2:50 PM, Daniel Mendler wrote:
> I am looking for an elisp implementation of emacsclient. Is such a
> functionality already build into Emacs? This would allow to start a
> separate Emacs instance which can then be used as an asynchronous worker.
Turns out such an `emacsclient` is rather easy to build using
`make-network-process`. This is probably neither robust nor platform
independent due to the use of unix sockets. Nevertheless such a client
is useful to communicate with asynchronous Emacs workers. With some
effort it is possible to replace the current `emacsclient` protocol with
something more robust. I assume there is no guarantee for backward
compatibility in this protocol?

Daniel

~~~~

(defun emacsclient (name expr callback)
  (let* ((result)
         (proc (make-network-process
                :name name
                :noquery t
                :sentinel
                (lambda (_proc _event)
                  (funcall callback (and result (read result))))
                :filter
                (lambda (_proc out)
                  (dolist (line (split-string out "\n"))
                    (cond
                     ((string-prefix-p "-print " line)
                      (setq result (server-unquote-arg
                                    (string-remove-prefix "-print " line))))
                     ((string-prefix-p "-print-nonl " line)
                      (setq result
                            (concat
                             result
                             (server-unquote-arg
                              (string-remove-prefix "-printnonl "
line))))))))
                :coding 'raw-text-unix
                :family 'local
                :service (expand-file-name name server-socket-dir))))
    (process-send-string
     proc
     (format "-eval %s \n" (server-quote-arg (prin1-to-string expr))))
    proc))

Re: emacsclient in elisp

[chad]

[-- Attachment #1: Type: text/plain, Size: 2143 bytes --]

Are you perhaps trying to build emacs-async?
  https://github.com/jwiegley/emacs-async
~Chad


On Wed, May 19, 2021 at 2:57 PM Daniel Mendler <mail@daniel-mendler.de>
wrote:

> On 5/19/21 2:50 PM, Daniel Mendler wrote:
> > I am looking for an elisp implementation of emacsclient. Is such a
> > functionality already build into Emacs? This would allow to start a
> > separate Emacs instance which can then be used as an asynchronous worker.
> Turns out such an `emacsclient` is rather easy to build using
> `make-network-process`. This is probably neither robust nor platform
> independent due to the use of unix sockets. Nevertheless such a client
> is useful to communicate with asynchronous Emacs workers. With some
> effort it is possible to replace the current `emacsclient` protocol with
> something more robust. I assume there is no guarantee for backward
> compatibility in this protocol?
>
> Daniel
>
> ~~~~
>
> (defun emacsclient (name expr callback)
>   (let* ((result)
>          (proc (make-network-process
>                 :name name
>                 :noquery t
>                 :sentinel
>                 (lambda (_proc _event)
>                   (funcall callback (and result (read result))))
>                 :filter
>                 (lambda (_proc out)
>                   (dolist (line (split-string out "\n"))
>                     (cond
>                      ((string-prefix-p "-print " line)
>                       (setq result (server-unquote-arg
>                                     (string-remove-prefix "-print "
> line))))
>                      ((string-prefix-p "-print-nonl " line)
>                       (setq result
>                             (concat
>                              result
>                              (server-unquote-arg
>                               (string-remove-prefix "-printnonl "
> line))))))))
>                 :coding 'raw-text-unix
>                 :family 'local
>                 :service (expand-file-name name server-socket-dir))))
>     (process-send-string
>      proc
>      (format "-eval %s \n" (server-quote-arg (prin1-to-string expr))))
>     proc))
>
>

[-- Attachment #2: Type: text/html, Size: 2982 bytes --]

Re: emacsclient in elisp

[Daniel Mendler]

On 5/20/21 12:43 AM, chad wrote:
> Are you perhaps trying to build emacs-async?
>   https://github.com/jwiegley/emacs-async

Of course I am aware of emacs-async. If I understood correctly, it does
not provide the functionality I am looking for. For every future a new
Emacs process is started. In contrast, I want to start a separate Emacs
process and then communicate with it while it is running.

Extending the emacsclient protocol is possible and quite easy. One can
use process-send-string in the backend Emacs process to send data to the
frontend Emacs process.

See here for my current experiment:

affe.el - Asynchronous Fuzzy Finder for Emacs
https://github.com/minad/affe

Daniel

Re: emacsclient in elisp

[Jean Louis]

* Daniel Mendler <mail@daniel-mendler.de> [2021-05-19 17:14]:
> I cannot start the Emacs in the background and communicate with it by
> sending/receiving data via stdin/stdout, since Emacs does not offer an
> asynchronous facility to read from stdin. Asynchronous reading would be
> needed on the daemon. If I am calling `read-from-minibuffer` there it
> reads from stdin but blocks the process. Is this correct?

It would be useful to have Emacs that can read stdin. 

To remedy that, I have to make such Common Lisp scripts that read from
STDIN, a wrapper that will provide file to Emacs.

Can you maybe improve Emacs to read from STDIN?


#!/home/data1/protected/bin/lisp
(defparameter memdir "/dev/shm/")
(defparameter tmp-file "rcd-wrs-XXXXXX")
(defparameter tmp-org (format nil "~a~a.org" memdir tmp-file))
(defparameter tmp-md (format nil "~a~a.md" memdir tmp-file))
;;(delete-file tmp-org)
;;(delete-file tmp-md)
(load "/home/data1/protected/Programming/lib/lisp/streamtools.lisp")
;; (require "syscalls")

(defparameter org-autoloads.el
  (concatenate
   'string
   (namestring (car (directory "/home/data1/protected/.emacs.d/elpa/org-20??????/")))
   "org-autoloads.el"))

(defun main ()
  (let ((input '())
        (output '()))
    (with-open-file (out tmp-org :direction :output :external-format "utf-8" :if-exists :supersede)
      (loop for line = (read-line *standard-input* nil nil)
         while line do
           (write-line line out)))
    (shell (format nil "emacs -Q -l ~a \"~a\" --batch -f org-mode -f org-md-export-to-markdown --kill" org-autoloads.el tmp-org))
    (setf output
            (with-open-file (stream tmp-md :direction :input :external-format "utf-8")
              (loop for line = (read-line stream nil nil)
                 while line collect line)))
    (setf output (format nil "~{~a~^~%~}" output))
    (setf output (slurp-stream-io-command "markdown -F 0x4" output))
    (princ output))
  (exit))

;; ;; (saveinitmem "org2html" :quiet t :init-function 'main :verbose nil :norc t :documentation "Converts Org standard input into markdown" :executable t)

(main)

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

Re: emacsclient in elisp

[Eli Zaretskii]

> Date: Thu, 20 May 2021 17:31:55 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> It would be useful to have Emacs that can read stdin. 

Emacs already is capable of doing that, see the node "Input Streams"
in the ELisp manual.

Re: emacsclient in elisp

[Daniel Mendler]

On 5/20/21 5:10 PM, Eli Zaretskii wrote:
>> It would be useful to have Emacs that can read stdin. 
> 
> Emacs already is capable of doing that, see the node "Input Streams"
> in the ELisp manual.

Yes, synchronously reading from stdin is supported. But it is not
possible to do this asynchronously, or is it? This was specifically the
functionality I have been looking for.

Daniel

Re: emacsclient in elisp

[Arthur Miller]

Daniel Mendler <mail@daniel-mendler.de> writes:

> On 5/19/21 2:50 PM, Daniel Mendler wrote:
>> I am looking for an elisp implementation of emacsclient. Is such a
>> functionality already build into Emacs? This would allow to start a
>> separate Emacs instance which can then be used as an asynchronous worker.
> Turns out such an `emacsclient` is rather easy to build using
> `make-network-process`. This is probably neither robust nor platform
> independent due to the use of unix sockets. Nevertheless such a client
> is useful to communicate with asynchronous Emacs workers. With some
> effort it is possible to replace the current `emacsclient` protocol with
> something more robust. I assume there is no guarantee for backward
> compatibility in this protocol?
>
> Daniel
>
> ~~~~
>
> (defun emacsclient (name expr callback)
>   (let* ((result)
>          (proc (make-network-process
>                 :name name
>                 :noquery t
>                 :sentinel
>                 (lambda (_proc _event)
>                   (funcall callback (and result (read result))))
>                 :filter
>                 (lambda (_proc out)
>                   (dolist (line (split-string out "\n"))
>                     (cond
>                      ((string-prefix-p "-print " line)
>                       (setq result (server-unquote-arg
>                                     (string-remove-prefix "-print " line))))
>                      ((string-prefix-p "-print-nonl " line)
>                       (setq result
>                             (concat
>                              result
>                              (server-unquote-arg
>                               (string-remove-prefix "-printnonl "
> line))))))))
>                 :coding 'raw-text-unix
>                 :family 'local
>                 :service (expand-file-name name server-socket-dir))))
>     (process-send-string
>      proc
>      (format "-eval %s \n" (server-quote-arg (prin1-to-string expr))))
>     proc))

Isn't that what 'server-eval-at' already does?

emacs --daemon=foo

(server-eval-at "foo" '(some-complex-eval))

> This is probably neither robust nor platform
> independent due to the use of unix sockets.

Docs says tcp sockets are used where local sockets are not avialable and:

https://www.gnu.org/software/emacs/manual/html_node/emacs/TCP-Emacs-server.html

Re: emacsclient in elisp

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Thu, 20 May 2021 17:52:31 +0200
> 
> On 5/20/21 5:10 PM, Eli Zaretskii wrote:
> >> It would be useful to have Emacs that can read stdin. 
> > 
> > Emacs already is capable of doing that, see the node "Input Streams"
> > in the ELisp manual.
> 
> Yes, synchronously reading from stdin is supported. But it is not
> possible to do this asynchronously, or is it? This was specifically the
> functionality I have been looking for.

What do you mean by that? how can Emacs read input _asynchronously_
when reading is inherently a synchronous operation?

And what would be on the other end of such asynchronous input,
producing the stuff Emacs is supposed to read?

Re: emacsclient in elisp

[Stefan Monnier]

>> Yes, synchronously reading from stdin is supported. But it is not
>> possible to do this asynchronously, or is it? This was specifically the
>> functionality I have been looking for.
> What do you mean by that? how can Emacs read input _asynchronously_
> when reading is inherently a synchronous operation?

I think the problem is that in batch mode, `read-from-minibuffer` is not
only synchronous (which is fine) but it doesn't perform any async
processing, like running process filters and timers while waiting for
the input from stdin.


        Stefan

Re: emacsclient in elisp

[Daniel Mendler]

On 5/20/21 5:59 PM, Arthur Miller wrote:
> Isn't that what 'server-eval-at' already does?
> 
> emacs --daemon=foo
> 
> (server-eval-at "foo" '(some-complex-eval))

Yes, `server-eval-at` fulfills a similar purpose. But it is not
asynchronous. My function has a callback argument.

>> This is probably neither robust nor platform
>> independent due to the use of unix sockets.
> 
> Docs says tcp sockets are used where local sockets are not avialable and:
> 
> https://www.gnu.org/software/emacs/manual/html_node/emacs/TCP-Emacs-server.html

That's right. The `emacsclient` implementation I've written above should
also handle TCP sockets as a fallback. But for now I am happy with only
handling Unix sockets (see https://github.com/minad/affe).

Daniel

Re: emacsclient in elisp

[Daniel Mendler]

On 5/20/21 6:40 PM, Stefan Monnier wrote:
>>> Yes, synchronously reading from stdin is supported. But it is not
>>> possible to do this asynchronously, or is it? This was specifically the
>>> functionality I have been looking for.
>> What do you mean by that? how can Emacs read input _asynchronously_
>> when reading is inherently a synchronous operation?
> 
> I think the problem is that in batch mode, `read-from-minibuffer` is not
> only synchronous (which is fine) but it doesn't perform any async
> processing, like running process filters and timers while waiting for
> the input from stdin.

Exactly. Reading from stdin hangs up the whole process and I would like
it to handle the other process filters in the background while Emacs is
waiting on stdin for input. Reading from stdin should not get a special
treatment in terms of scheduling. There should be a possibility to read
from stdin using a similar API as `make-network-process` with a filter
and sentinel for EOF.

Daniel

Re: emacsclient in elisp

[Eli Zaretskii]

> Cc: bugs@gnu.support, emacs-devel@gnu.org
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Thu, 20 May 2021 19:49:38 +0200
> 
> On 5/20/21 6:40 PM, Stefan Monnier wrote:
> >>> Yes, synchronously reading from stdin is supported. But it is not
> >>> possible to do this asynchronously, or is it? This was specifically the
> >>> functionality I have been looking for.
> >> What do you mean by that? how can Emacs read input _asynchronously_
> >> when reading is inherently a synchronous operation?
> > 
> > I think the problem is that in batch mode, `read-from-minibuffer` is not
> > only synchronous (which is fine) but it doesn't perform any async
> > processing, like running process filters and timers while waiting for
> > the input from stdin.
> 
> Exactly. Reading from stdin hangs up the whole process and I would like
> it to handle the other process filters in the background while Emacs is
> waiting on stdin for input. Reading from stdin should not get a special
> treatment in terms of scheduling. There should be a possibility to read
> from stdin using a similar API as `make-network-process` with a filter
> and sentinel for EOF.

You didn't answer part of my question:

> And what would be on the other end of such asynchronous input,
> producing the stuff Emacs is supposed to read?

Without that, I don't think I understand the use case you want to
support in sufficient detail.

Re: emacsclient in elisp

[Daniel Mendler]

On 5/20/21 8:59 PM, Eli Zaretskii wrote:
> You didn't answer part of my question:
> 
>> And what would be on the other end of such asynchronous input,
>> producing the stuff Emacs is supposed to read?
> 
> Without that, I don't think I understand the use case you want to
> support in sufficient detail.

Of course, there is another Emacs on the other side of the pipe. It is
Emacs all the way down ;)

Seriously though, it does not matter what sits on the other end. What
matters is that the Emacs reading from stdin can still run the process
filters of asynchronous child processes. My current use case is this:

affe.el - Asynchronous Fuzzy Finder for Emacs
https://github.com/minad/affe

But I imagine there are many more interesting use cases for Emacs
processes communicating asynchronously.

Daniel

Re: emacsclient in elisp

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, bugs@gnu.support, emacs-devel@gnu.org
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Thu, 20 May 2021 21:10:37 +0200
> 
> Of course, there is another Emacs on the other side of the pipe. It is
> Emacs all the way down ;)

But then I don't understand the importance of stdin in this context.
If you want to read output from another program, just run that
program, in this case another Emacs, as a subprocess.  Then anything
that other program writes to its stdout will be read as part of normal
machinery of reading sub-process output, including filters, sentinels,
and all that.

So what am I missing here?

> Seriously though, it does not matter what sits on the other end.

It really does matter, because without understanding these details we
cannot make sure we are on the same page discussing these matters.  I
understand that you have something specific in mind, but if you don't
describe that something in enough detail for me to understand what you
have in mind, we will be forever mis-communicating.

Re: emacsclient in elisp

[Daniel Mendler]

On 5/20/21 9:45 PM, Eli Zaretskii wrote:
> But then I don't understand the importance of stdin in this context.
> If you want to read output from another program, just run that
> program, in this case another Emacs, as a subprocess.  Then anything
> that other program writes to its stdout will be read as part of normal
> machinery of reading sub-process output, including filters, sentinels,
> and all that.

Yes, that's right. I have two Emacs processes A and B. The process A
starts process B as subprocess and communicates with it via the pipe.
The process B reads from the stdin using `(read t)` or
`(read-from-minibuffer..)`. Now process B starts another subprocess C,
which has a filter and a sentinel. While B reads from stdin, B hangs and
the C-filter is not executed.

Alternatively one can use network streams. A creates a server, B creates
a client and then they communicate asynchronously, B can start a
subprocess C and the AB-network communication does not impede the
subprocess C.

> It really does matter, because without understanding these details we
> cannot make sure we are on the same page discussing these matters.  I
> understand that you have something specific in mind, but if you don't
> describe that something in enough detail for me to understand what you
> have in mind, we will be forever mis-communicating.

I would like to offload work from Emacs to sub-Emacs processes which are
connected via some IPC protocol. I am currently using the emacsclient
protocol for this (with some custom extensions). Stefan suggested that
it may be better if the Emacs processes communicate directly using
pipes. But given the stdin blocking problem, described above, this is
not possible.

Actually I am happy with using the emacsclient protocol now. It is a bit
of a hack, but it works for the simple use case I am currently exploring
(https://github.com/minad/affe). There I have Emacs A (frontend)
starting an Emacs B (backend) which starts grep as a subprocess C,
generating output. Then A sends filter regular expressions to B, B
filters the output of C and answers with the matches.

Daniel

Re: emacsclient in elisp

[Stefan Monnier]

> Without that, I don't think I understand the use case you want to
> support in sufficient detail.

I'd expect in the server a loop like:

    (while t
      (let ((cmd (read-from-minibuffer "")))
        (process-command cmd)))

where `process-command` may launch async subprocesses or initiate
network connections or send commands to network connections, and the
above loop will not always immediately send replies but instead they may
be sent from the process filters and/or sentinels.

With the current implementation of `read-from-minibuffer` this above
loop won't work because there is no time where process filters and
timers are processed, including when we're waiting in
`read-from-minibuffer`.


        Stefan

Re: emacsclient in elisp

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, bugs@gnu.support, emacs-devel@gnu.org
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Thu, 20 May 2021 22:54:45 +0200
> 
> On 5/20/21 9:45 PM, Eli Zaretskii wrote:
> > But then I don't understand the importance of stdin in this context.
> > If you want to read output from another program, just run that
> > program, in this case another Emacs, as a subprocess.  Then anything
> > that other program writes to its stdout will be read as part of normal
> > machinery of reading sub-process output, including filters, sentinels,
> > and all that.
> 
> Yes, that's right. I have two Emacs processes A and B. The process A
> starts process B as subprocess and communicates with it via the pipe.
> The process B reads from the stdin using `(read t)` or
> `(read-from-minibuffer..)`. Now process B starts another subprocess C,
> which has a filter and a sentinel. While B reads from stdin, B hangs and
> the C-filter is not executed.

The same will happen if B runs some Lisp function that takes a long
time.

> Alternatively one can use network streams. A creates a server, B creates
> a client and then they communicate asynchronously, B can start a
> subprocess C and the AB-network communication does not impede the
> subprocess C.

So you already have a solution for such use cases, right?  Then what
would be the advantage of doing the same via stdin?

> I would like to offload work from Emacs to sub-Emacs processes which are
> connected via some IPC protocol. I am currently using the emacsclient
> protocol for this (with some custom extensions). Stefan suggested that
> it may be better if the Emacs processes communicate directly using
> pipes. But given the stdin blocking problem, described above, this is
> not possible.

Then I guess it is Stefan who should answer the question above.

> Actually I am happy with using the emacsclient protocol now. It is a bit
> of a hack, but it works for the simple use case I am currently exploring

I don't see why it would be considered a hack.  Using the emacsclient
_protocol_ might be considered a kludge, but you can use your own
protocol, and then I see no kludge at all in this design.

> (https://github.com/minad/affe). There I have Emacs A (frontend)
> starting an Emacs B (backend) which starts grep as a subprocess C,
> generating output. Then A sends filter regular expressions to B, B
> filters the output of C and answers with the matches.

Not sure why you need Grep in this equation at all: Emacs is perfectly
capable of searching files entirely on its own, and since your design
is asynchronous, speed should not matter too much anyway.

Re: emacsclient in elisp

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Daniel Mendler <mail@daniel-mendler.de>,  bugs@gnu.support,
>   emacs-devel@gnu.org
> Date: Thu, 20 May 2021 17:21:01 -0400
> 
> I'd expect in the server a loop like:
> 
>     (while t
>       (let ((cmd (read-from-minibuffer "")))
>         (process-command cmd)))
> 
> where `process-command` may launch async subprocesses or initiate
> network connections or send commands to network connections, and the
> above loop will not always immediately send replies but instead they may
> be sent from the process filters and/or sentinels.
> 
> With the current implementation of `read-from-minibuffer` this above
> loop won't work because there is no time where process filters and
> timers are processed, including when we're waiting in
> `read-from-minibuffer`.

But only if the above runs in batch mode, right?

And as I wrote elsewhere, I don't understand why being able to read
from stdin like it's a sub-process would be so much better than
actually using a network process.  We could introduce such a feature
for the batch mode, but it would mean serious complications, and I
just don't see the justification for that.

Re: emacsclient in elisp

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2021-05-20 18:11]:
> > Date: Thu, 20 May 2021 17:31:55 +0300
> > From: Jean Louis <bugs@gnu.support>
> > Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
> >  "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> > 
> > It would be useful to have Emacs that can read stdin. 
> 
> Emacs already is capable of doing that, see the node "Input Streams"
> in the ELisp manual.

emacs -Q -l process-org.el -l ~/.emacs.d/elpa/org-20201216/org-autoloads.el my-file.org --batch -f org-mode -f process-org

I have tried this way, it seem to be waiting for lines, but then it becomes error. I was expecting that `read-string' gets its stream reading. Maybe it works, I am not sure. But `read' cannot read strings, it is for expressions, the `read' is the only function described to receive input streams in that section of manual.

process-org.el:

(defun process-org () 
  (let ((line))
    (with-temp-buffer 
      (while (not (string= (setq line (read-string "")) "\x04"))
	(insert line)
	(insert "\n"))
      (org-md-export-to-markdown))))

(defun tell-me ()
  (read-from-string ""))

And if I use this function:

(defun process-org () 
  (let ((line))
    (with-temp-buffer 
      (while (not (string= (setq line (read t)) "\x04"))
	(insert line)
	(insert "\n"))
      (org-md-export-to-markdown))))

Then some lines will pass, some will not pass, as it prioritizes Lisp
expressions, not strings primarily.

Yes, Emacs can read stdin, only now it can read expressions. It would
be good that it can read anything. When reading standard input, one
need no input prompts. So I think a designated function is needed to
read standard input, that we may do something like:

cat file | emacs ...

as that way it becomes the tool to process text on command line.

Thank you,
Jean Louis

Re: emacsclient in elisp

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2021-05-20 19:41]:
> >> Yes, synchronously reading from stdin is supported. But it is not
> >> possible to do this asynchronously, or is it? This was specifically the
> >> functionality I have been looking for.
> > What do you mean by that? how can Emacs read input _asynchronously_
> > when reading is inherently a synchronous operation?
> 
> I think the problem is that in batch mode, `read-from-minibuffer` is not
> only synchronous (which is fine) but it doesn't perform any async
> processing, like running process filters and timers while waiting for
> the input from stdin.

Please send a reference how read-from-minibuffer reads standard input,
as I don't get it to work.

(defun tell-me ()
  (read-from-minibuffer "")))

$ emacs -Q -l process-org.el --batch -f tell-me 

Debugger entered--Lisp error: (end-of-file)
  read-from-string("")
  tell-me()
  command-line-1(("-l" "process-org.el" "-f" "tell-me"))
  command-line()
  normal-top-level()


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

Re: emacsclient in elisp

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2021-05-20 22:46]:
> > Cc: monnier@iro.umontreal.ca, bugs@gnu.support, emacs-devel@gnu.org
> > From: Daniel Mendler <mail@daniel-mendler.de>
> > Date: Thu, 20 May 2021 21:10:37 +0200
> > 
> > Of course, there is another Emacs on the other side of the pipe. It is
> > Emacs all the way down ;)
> 
> But then I don't understand the importance of stdin in this context.
> If you want to read output from another program, just run that
> program, in this case another Emacs, as a subprocess.  Then anything
> that other program writes to its stdout will be read as part of normal
> machinery of reading sub-process output, including filters, sentinels,
> and all that.

Side note to your comment, I find it somehow good. So I could just
fake the STDIN reading by using for example `cat`. But it does not
work. Please if you have some reference to demonstration of the above,
let me know.

My expectation is for following to work:

process-org.el:

(defun cat ()
  (princ (shell-command-to-string "cat"))) ;; later call-process to be used

then on command line:

$ emacs -Q -l process-org.el --batch -f cat 

So that does not work, it exits immediately, it is not waiting for
stdin

And within Emacs (shell-command-to-string "cat") will also not wait,
including with call process, it will not wait.

But would it wait for STDIN just as `cat` does wait, then it would be
easy to use such external wrapper to receive STDIN and process it with
Emacs.

Thank you,
Jean Louis

Re: emacsclient in elisp

[Jean Louis]

* Daniel Mendler <mail@daniel-mendler.de> [2021-05-20 23:55]:
> Yes, that's right. I have two Emacs processes A and B. The process A
> starts process B as subprocess and communicates with it via the pipe.
> The process B reads from the stdin using `(read t)` or
> `(read-from-minibuffer..)`.

Please send example how you do that with batch processing to
read-from-minibuffer from STDIN?


-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/
https://rms-support-letter.github.io/

Re: emacsclient in elisp

[Eli Zaretskii]

> Date: Fri, 21 May 2021 09:27:14 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: Eli Zaretskii <eliz@gnu.org>,
>   Daniel Mendler <mail@daniel-mendler.de>, emacs-devel@gnu.org
> 
> (defun tell-me ()
>   (read-from-minibuffer "")))
> 
> $ emacs -Q -l process-org.el --batch -f tell-me 
> 
> Debugger entered--Lisp error: (end-of-file)
>   read-from-string("")
>   tell-me()
>   command-line-1(("-l" "process-org.el" "-f" "tell-me"))
>   command-line()
>   normal-top-level()

Works for me.  And note that your backtrace says read-from-string, not
read-from-minibuffer.

Re: emacsclient in elisp

[Eli Zaretskii]

> Date: Fri, 21 May 2021 09:21:35 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: mail@daniel-mendler.de, monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> 
> * Eli Zaretskii <eliz@gnu.org> [2021-05-20 18:11]:
> > > Date: Thu, 20 May 2021 17:31:55 +0300
> > > From: Jean Louis <bugs@gnu.support>
> > > Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
> > >  "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> > > 
> > > It would be useful to have Emacs that can read stdin. 
> > 
> > Emacs already is capable of doing that, see the node "Input Streams"
> > in the ELisp manual.
> 
> emacs -Q -l process-org.el -l ~/.emacs.d/elpa/org-20201216/org-autoloads.el my-file.org --batch -f org-mode -f process-org
> 
> I have tried this way, it seem to be waiting for lines, but then it becomes error. I was expecting that `read-string' gets its stream reading. Maybe it works, I am not sure. But `read' cannot read strings, it is for expressions, the `read' is the only function described to receive input streams in that section of manual.

Use read-from-minibuffer, it can read anything.

Re: emacsclient in elisp

[Eli Zaretskii]

> Date: Fri, 21 May 2021 09:36:38 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: Daniel Mendler <mail@daniel-mendler.de>,
>   monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > But then I don't understand the importance of stdin in this context.
> > If you want to read output from another program, just run that
> > program, in this case another Emacs, as a subprocess.  Then anything
> > that other program writes to its stdout will be read as part of normal
> > machinery of reading sub-process output, including filters, sentinels,
> > and all that.
> 
> Side note to your comment, I find it somehow good. So I could just
> fake the STDIN reading by using for example `cat`. But it does not
> work. Please if you have some reference to demonstration of the above,
> let me know.
> 
> My expectation is for following to work:
> 
> process-org.el:
> 
> (defun cat ()
>   (princ (shell-command-to-string "cat"))) ;; later call-process to be used
> 
> then on command line:
> 
> $ emacs -Q -l process-org.el --batch -f cat 
> 
> So that does not work, it exits immediately, it is not waiting for
> stdin

I don't understand what "doesn't work" mean in this case.  The code
you show isn't supposed to wait for anything, and it doesn't use
start-process to start async subprocesses.

Re: emacsclient in elisp

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2021-05-21 10:08]:
> > Date: Fri, 21 May 2021 09:27:14 +0300
> > From: Jean Louis <bugs@gnu.support>
> > Cc: Eli Zaretskii <eliz@gnu.org>,
> >   Daniel Mendler <mail@daniel-mendler.de>, emacs-devel@gnu.org
> > 
> > (defun tell-me ()
> >   (read-from-minibuffer "")))
> > 
> > $ emacs -Q -l process-org.el --batch -f tell-me 
> > 
> > Debugger entered--Lisp error: (end-of-file)
> >   read-from-string("")
> >   tell-me()
> >   command-line-1(("-l" "process-org.el" "-f" "tell-me"))
> >   command-line()
> >   normal-top-level()
> 
> Works for me.  And note that your backtrace says read-from-string, not
> read-from-minibuffer.

OK thank you, I got it to work as by reading examples on:
https://superuser.com/questions/31404/how-to-make-emacs-read-buffer-from-stdin-on-start

So I made this pure Emacs command line converter from Org to Markdown
and that works well. 

#!/home/data1/protected/bin/emacs --script 

(require 'org)
(require 'ox)
(require 'ox-md)

(defun org-to-markdown ()
  (let* ((org (with-temp-buffer
		(progn
		  (condition-case nil
		      (let (line)
			(while (setq line (read-from-minibuffer ""))
			  (insert line)
			  (insert "\n")))
		    (error nil))
		  (buffer-string))))
	 (markdown (with-temp-buffer
		     (insert org)
		     (org-export-to-buffer 'md (current-buffer) nil nil nil t)
		     (buffer-string))))
    (prin1 markdown)))

(org-to-markdown)



$ ./process-org.el
#+TITLE: org

** Hello
"
# Table of Contents

1.  [Hello](#orgf318453)



<a id=\"orgf318453\"></a>

# Hello

"

Re: emacsclient in elisp

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2021-05-21 10:09]:

> Use read-from-minibuffer, it can read anything.

Now I use, though it is not documented in manual.

Re: emacsclient in elisp

[Eli Zaretskii]

> Date: Fri, 21 May 2021 10:26:43 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: mail@daniel-mendler.de, monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> 
> * Eli Zaretskii <eliz@gnu.org> [2021-05-21 10:09]:
> 
> > Use read-from-minibuffer, it can read anything.
> 
> Now I use, though it is not documented in manual.

Of course, it's documented: see the  nodes "Input Streams" and "Intro
to Minibuffers" of the ELisp manual.

Re: emacsclient in elisp

[Daniel Mendler]

On 5/21/21 8:16 AM, Eli Zaretskii wrote:
>> Yes, that's right. I have two Emacs processes A and B. The process A
>> starts process B as subprocess and communicates with it via the pipe.
>> The process B reads from the stdin using `(read t)` or
>> `(read-from-minibuffer..)`. Now process B starts another subprocess C,
>> which has a filter and a sentinel. While B reads from stdin, B hangs and
>> the C-filter is not executed.
> 
> The same will happen if B runs some Lisp function that takes a long
> time.

Of course, but IO-bound waiting is fundamentally different from
CPU-bound computations.

>> Actually I am happy with using the emacsclient protocol now. It is a bit
>> of a hack, but it works for the simple use case I am currently exploring
> 
> I don't see why it would be considered a hack.  Using the emacsclient
> _protocol_ might be considered a kludge, but you can use your own
> protocol, and then I see no kludge at all in this design.

You are right about that. I think Stefan should elaborate. What I am
doing now - I am "upgrading" the emacsclient protocol on the server by
replacing the `server-process` filter with my own version. This works
well and is also clean. I am actually never using the emacsclient
protocol now, I am only reusing the Emacs infrastructure to set up the
server and to daemonize.

>> (https://github.com/minad/affe). There I have Emacs A (frontend)
>> starting an Emacs B (backend) which starts grep as a subprocess C,
>> generating output. Then A sends filter regular expressions to B, B
>> filters the output of C and answers with the matches.
> 
> Not sure why you need Grep in this equation at all: Emacs is perfectly
> capable of searching files entirely on its own, and since your design
> is asynchronous, speed should not matter too much anyway.

There is a point in doing what I am doing but it strongly depends on the
UI you are using. Do you know the command line fuzzy finder fzf. My
program basically emulates that working mode in Emacs. The grep
functionality offered by Emacs is not related to that, but is fine by
itself. There is no "need" for my package, it is just an experiment to
see if I can retrofit certain interaction models onto Emacs.

Daniel

Re: emacsclient in elisp

[Eli Zaretskii]

> Date: Fri, 21 May 2021 09:18:51 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: mail@daniel-mendler.de, bugs@gnu.support, emacs-devel@gnu.org
> 
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: Daniel Mendler <mail@daniel-mendler.de>,  bugs@gnu.support,
> >   emacs-devel@gnu.org
> > Date: Thu, 20 May 2021 17:21:01 -0400
> > 
> > I'd expect in the server a loop like:
> > 
> >     (while t
> >       (let ((cmd (read-from-minibuffer "")))
> >         (process-command cmd)))
> > 
> > where `process-command` may launch async subprocesses or initiate
> > network connections or send commands to network connections, and the
> > above loop will not always immediately send replies but instead they may
> > be sent from the process filters and/or sentinels.
> > 
> > With the current implementation of `read-from-minibuffer` this above
> > loop won't work because there is no time where process filters and
> > timers are processed, including when we're waiting in
> > `read-from-minibuffer`.
> 
> But only if the above runs in batch mode, right?

Btw: in batch mode, read-from-minibuffer returns when it sees a
newline, so you could still use the above if you'd process each
incoming line, and then issued another read-from-minibuffer call only
when the processing was done.

So the situation we have doesn't look too bad to me.

Re: emacsclient in elisp

[Eli Zaretskii]

> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Fri, 21 May 2021 12:33:12 +0200
> Cc: monnier@iro.umontreal.ca, bugs@gnu.support, emacs-devel@gnu.org
> 
> On 5/21/21 8:16 AM, Eli Zaretskii wrote:
> >> Yes, that's right. I have two Emacs processes A and B. The process A
> >> starts process B as subprocess and communicates with it via the pipe.
> >> The process B reads from the stdin using `(read t)` or
> >> `(read-from-minibuffer..)`. Now process B starts another subprocess C,
> >> which has a filter and a sentinel. While B reads from stdin, B hangs and
> >> the C-filter is not executed.
> > 
> > The same will happen if B runs some Lisp function that takes a long
> > time.
> 
> Of course, but IO-bound waiting is fundamentally different from
> CPU-bound computations.

With read-from-minibuffer, you only wait until the first newline.  So
I'm not sure I understand the big deal about that.

Re: emacsclient in elisp

[Stefan Monnier]

> And as I wrote elsewhere, I don't understand why being able to read
> from stdin like it's a sub-process would be so much better than
> actually using a network process.

When you start such an Emacs process as a subprocess, it saves you from
having to setup a separate communication channel, which requires
agreeing on a channel "name", making sure that name is not already in
use, and can require extra care to avoid security problems.

IOW, I think using stdin/stdout this way is many cases would be The
Right Way and using a separate network/socket/namedpipe would be
a workaround.

> We could introduce such a feature for the batch mode, but it would
> mean serious complications, and I just don't see the justification
> for that.

I'm not volunteering to do the job, admittedly.

Maybe the easiest way to solve this would be to provide a function
`make-stdin-process` which returns a process-object associated to
stdin/stdout (instead of trying to change `read-from-minibuffer`).


        Stefan

Re: emacsclient in elisp

[Stefan Monnier]

> Btw: in batch mode, read-from-minibuffer returns when it sees a
> newline, so you could still use the above if you'd process each
> incoming line, and then issued another read-from-minibuffer call only
> when the processing was done.

The next line may not come from another half-hour, or even "for ever" if
the other side is waiting for our (asynchronous) answer before sending the
next command.

> So the situation we have doesn't look too bad to me.

There are cases where it's usable, but there are cases where it's
a non-starter.


        Stefan

Re: emacsclient in elisp

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: mail@daniel-mendler.de,  bugs@gnu.support,  emacs-devel@gnu.org
> Date: Fri, 21 May 2021 10:51:51 -0400
> 
> Maybe the easiest way to solve this would be to provide a function
> `make-stdin-process` which returns a process-object associated to
> stdin/stdout (instead of trying to change `read-from-minibuffer`).

In batch mode only or in other modes as well?

And how would you use such a process?

Re: emacsclient in elisp

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: mail@daniel-mendler.de,  bugs@gnu.support,  emacs-devel@gnu.org
> Date: Fri, 21 May 2021 10:55:02 -0400
> 
> > Btw: in batch mode, read-from-minibuffer returns when it sees a
> > newline, so you could still use the above if you'd process each
> > incoming line, and then issued another read-from-minibuffer call only
> > when the processing was done.
> 
> The next line may not come from another half-hour, or even "for ever" if
> the other side is waiting for our (asynchronous) answer before sending the
> next command.

So don't call read-from-minibuffer until you have processed the
previous job and sent out the results.

> > So the situation we have doesn't look too bad to me.
> 
> There are cases where it's usable, but there are cases where it's
> a non-starter.

Why focus on the latter?

Re: emacsclient in elisp

[Stefan Monnier]

>> Maybe the easiest way to solve this would be to provide a function
>> `make-stdin-process` which returns a process-object associated to
>> stdin/stdout (instead of trying to change `read-from-minibuffer`).
>
> In batch mode only or in other modes as well?

I think the only important case is batch mode, so the interactive case
is not terribly important.

> And how would you use such a process?

You'd replace the loop I suggested with something like

    (let ((proc (make-stdio-process)))
      (set-process-filter proc #'process-command)
      (while t (sleep 86400)))


-- Stefan

Re: emacsclient in elisp

[Stefan Monnier]

>> The next line may not come from another half-hour, or even "for ever" if
>> the other side is waiting for our (asynchronous) answer before sending the
>> next command.
> So don't call read-from-minibuffer until you have processed the
> previous job and sent out the results.

That can prevent "streaming" like uses where you can receive several
requests before you have the info needed to answer the first requests.
Or other situations where you want to reply to requests out of order.

>> There are cases where it's usable, but there are cases where it's
>> a non-starter.
> Why focus on the latter?

Because it's the situation that Daniel is into, AFAIK (and because the
other situations are already taken care of by the infrastructure we
have, so we (Emacs developers working on the infrastructure for
developers of third party packages) don't need to worry about those
other cases).


        Stefan

Re: emacsclient in elisp

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2021-05-21 13:32]:
> > Date: Fri, 21 May 2021 10:26:43 +0300
> > From: Jean Louis <bugs@gnu.support>
> > Cc: mail@daniel-mendler.de, monnier@iro.umontreal.ca,
> >   emacs-devel@gnu.org
> > 
> > * Eli Zaretskii <eliz@gnu.org> [2021-05-21 10:09]:
> > 
> > > Use read-from-minibuffer, it can read anything.
> > 
> > Now I use, though it is not documented in manual.
> 
> Of course, it's documented: see the  nodes "Input Streams" and "Intro
> to Minibuffers" of the ELisp manual.

In this page: (info "(elisp) Input Functions") there is nothing about
`read-from-minibuffer' and in the description of the function there is
nothing about STREAM neither that it can read from standard input. I
hope you can see how it lacks better references

On this page (info "(elisp) Intro to Minibuffers") I see:

   When Emacs is running in batch mode, any request to read from the
minibuffer actually reads a line from the standard input descriptor that
was supplied when Emacs was started.  This supports only basic input:
none of the special minibuffer features (history, completion, etc.) are
available in batch mode.

That is great though that is not the only place to explain what the
function does. As on this page: (info "(elisp) Text from Minibuffer")
there is description of `read-from-minibuffer' without mentioning
anything about standard input, there is not even a word "standard", so
it is missing in the manual. I cannot write it myself, I can just
point out to it, I can give a try:

- include the similar paragraph as above on the page (info "(elisp)
  Text from Minibuffer") in the description of the function, something
  like: 

   When Emacs is running in batch mode, any request to read from the
minibuffer actually reads a line from the standard input descriptor that
was supplied when Emacs was started.  This supports only basic input:
none of the special minibuffer features (history, completion, etc.) are
available in batch mode.

And same should be placed in the docstring or at least a link to
explanation. 

You see I found the answer on Internet, I could not find it, not so
easy, in the manual. First thing I do is C-h f but there was nothing
there. I am explaining my flow of actions here. Then when I found
STREAM for `read' the same is not mentioned for `read-from-minibuffer'
and nothing in the docstring, so without asking here, programmer would
could either search Internet or fail. 

If `read-from-string' also reading from standard input?

Thank you,
Jean

Re: emacsclient in elisp

[Daniel Mendler]

On 5/21/21 4:51 PM, Stefan Monnier wrote:
>> We could introduce such a feature for the batch mode, but it would
>> mean serious complications, and I just don't see the justification
>> for that.
> 
> I'm not volunteering to do the job, admittedly.
> 
> Maybe the easiest way to solve this would be to provide a function
> `make-stdin-process` which returns a process-object associated to
> stdin/stdout (instead of trying to change `read-from-minibuffer`).

Yes, something like this is what I had in mind for this feature. Or
adapt the network process API, adding a magic stdin socket name/type.

Daniel

Re: emacsclient in elisp

[Jean Louis]

* Daniel Mendler <mail@daniel-mendler.de> [2021-05-21 13:33]:
> There is a point in doing what I am doing but it strongly depends on the
> UI you are using. Do you know the command line fuzzy finder fzf. My
> program basically emulates that working mode in Emacs. The grep
> functionality offered by Emacs is not related to that, but is fine by
> itself. There is no "need" for my package, it is just an experiment to
> see if I can retrofit certain interaction models onto Emacs.

I have tried using it, here is user experience:

- great name, and I would propose changes instead of:

Possible completions are:
affe--find-history 	affe--grep-history 	affe-backend--head
affe-backend--tail 	affe-count 	affe-find-command
affe-grep-command 	affe-highlight-function 	affe-regexp-function

to be:

affe-banana--find-history 	affe-banana--grep-history 	affe-banana-backend--head
affe-banana-backend--tail 	affe-banana-count 	affe-banana-find-command
affe-banana-grep-command 	affe-banana-highlight-function 	affe-banana-regexp-function

As it is most logical to feed banana variables to affe.

Now...

- affe-find, does not work. Even in a directory with few files, there
  is nothing taking place, let me try again in ~/tmp/mozilla_user0/ so
  nothing is happening there, let us say I write some string, nothing,
  no reaction, if I press enter, the new empty buffer appears. If
  purpose of affe is to open up found file, so you should maybe check
  if the file written corresponds to file on the disk, as opening new
  empty buffer is not helpful when finding a file.

- affe-grep, I would remain to `affe-grape' even better
  `affe-grapefruit'; now that one actually works, it will find
  something yet not so usable, it is not nearly usable as M-x grep;
  and when I tried it in a different directory, even repeated C-g
  could not help and `rg' was running in the background in such way
  that it blocked all the X Window input. I pressed Ctrl-Alt-F4 to
  switch in a waiting manner to console which is more responsive, to
  open up `top' and find out which process to kill;

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/

Re: emacsclient in elisp

[Jean Louis]

* Stefan Monnier <monnier@iro.umontreal.ca> [2021-05-21 17:52]:
> Maybe the easiest way to solve this would be to provide a function
> `make-stdin-process` which returns a process-object associated to
> stdin/stdout (instead of trying to change `read-from-minibuffer`).

Possible use cases on my side for such would be:

- when generating many HTML pages that require Emacs Lisp for
  processing, then it would go faster as there would be no startup or
  loading time;

- Emacs could run in memory and produce HTML pages through a proxy
  server such as Nginx; it would be similar to fastcgi in that case; 

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

Sign an open letter in support of Richard M. Stallman
https://stallmansupport.org/

Re: emacsclient in elisp

[Eli Zaretskii]

> Date: Fri, 21 May 2021 19:07:59 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: mail@daniel-mendler.de, monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> 
> > Of course, it's documented: see the  nodes "Input Streams" and "Intro
> > to Minibuffers" of the ELisp manual.
> 
> In this page: (info "(elisp) Input Functions") there is nothing about
> `read-from-minibuffer' and in the description of the function there is
> nothing about STREAM neither that it can read from standard input.

Really?

> On this page (info "(elisp) Intro to Minibuffers") I see:
> 
>    When Emacs is running in batch mode, any request to read from the
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> minibuffer actually reads a line from the standard input descriptor that
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> was supplied when Emacs was started.  This supports only basic input:
> none of the special minibuffer features (history, completion, etc.) are
> available in batch mode.

If this isn't what you were looking for, then I don't know how to say
it more clearly.

Re: emacsclient in elisp

[Eli Zaretskii]

> Date: Fri, 21 May 2021 19:07:59 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: mail@daniel-mendler.de, monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> 
> If `read-from-string' also reading from standard input?

The doc string says:

  Read a string from the minibuffer, prompting with string PROMPT.

Given that you just read what happens with reading from minibuffer in
batch mode, what do you think is the answer to that question?

Re: emacsclient in elisp

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: mail@daniel-mendler.de,  bugs@gnu.support,  emacs-devel@gnu.org
> Date: Fri, 21 May 2021 11:29:29 -0400
> 
> >> Maybe the easiest way to solve this would be to provide a function
> >> `make-stdin-process` which returns a process-object associated to
> >> stdin/stdout (instead of trying to change `read-from-minibuffer`).
> >
> > In batch mode only or in other modes as well?
> 
> I think the only important case is batch mode, so the interactive case
> is not terribly important.
> 
> > And how would you use such a process?
> 
> You'd replace the loop I suggested with something like
> 
>     (let ((proc (make-stdio-process)))
>       (set-process-filter proc #'process-command)
>       (while t (sleep 86400)))

This would probably mean we need to reimplement reading from
minibuffer in batch mode.

Re: emacsclient in elisp

[Arthur Miller]

Daniel Mendler <mail@daniel-mendler.de> writes:

> On 5/20/21 5:59 PM, Arthur Miller wrote:
>> Isn't that what 'server-eval-at' already does?
>> 
>> emacs --daemon=foo
>> 
>> (server-eval-at "foo" '(some-complex-eval))
>
> Yes, `server-eval-at` fulfills a similar purpose. But it is not
> asynchronous. My function has a callback argument.

Ok, you already looked at it :).

>>> This is probably neither robust nor platform
>>> independent due to the use of unix sockets.
>> 
>> Docs says tcp sockets are used where local sockets are not avialable and:
>> 
>> https://www.gnu.org/software/emacs/manual/html_node/emacs/TCP-Emacs-server.html
>
> That's right. The `emacsclient` implementation I've written above should
> also handle TCP sockets as a fallback. But for now I am happy with only
> handling Unix sockets (see https://github.com/minad/affe).

Oki. There was some guy on GH doing something simnilar via TCP socks,
don't know if you are aware/wish to look at it.

https://gist.github.com/jclosure/cb34dbd813c6bd1e3c4e128ad87d69c7

Re: emacsclient in elisp

[Stefan Monnier]

>>     (let ((proc (make-stdio-process)))
>>       (set-process-filter proc #'process-command)
>>       (while t (sleep 86400)))
>
> This would probably mean we need to reimplement reading from
> minibuffer in batch mode.

I don't think so.  The only problem I could see is interaction between
the stdin/stdout code of `read-from-minibuffer` (in batch mode) and
`make-stdio-process`, but we could just decide that calling
`make-stdio-process` makes future `read-from-minibuffer` signal
an error (i.e. you can use either one or the other but not both).


        Stefan

Re: emacsclient in elisp

[Jean Louis]

* Eli Zaretskii <eliz@gnu.org> [2021-05-21 20:44]:
> > Date: Fri, 21 May 2021 19:07:59 +0300
> > From: Jean Louis <bugs@gnu.support>
> > Cc: mail@daniel-mendler.de, monnier@iro.umontreal.ca,
> >   emacs-devel@gnu.org
> > 
> > If `read-from-string' also reading from standard input?
> 
> The doc string says:
> 
>   Read a string from the minibuffer, prompting with string PROMPT.
> 
> Given that you just read what happens with reading from minibuffer in
> batch mode, what do you think is the answer to that question?

The answer is: maze in documentation. If I am not listening to you I
would not get it from manual.

Re: emacsclient in elisp

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: mail@daniel-mendler.de,  bugs@gnu.support,  emacs-devel@gnu.org
> Date: Fri, 21 May 2021 17:51:02 -0400
> 
> >>     (let ((proc (make-stdio-process)))
> >>       (set-process-filter proc #'process-command)
> >>       (while t (sleep 86400)))
> >
> > This would probably mean we need to reimplement reading from
> > minibuffer in batch mode.
> 
> I don't think so.  The only problem I could see is interaction between
> the stdin/stdout code of `read-from-minibuffer` (in batch mode) and
> `make-stdio-process`, but we could just decide that calling
> `make-stdio-process` makes future `read-from-minibuffer` signal
> an error (i.e. you can use either one or the other but not both).

Doesn't sound like a good idea to me.  There's no real reason for
being so harsh to applications.

Re: emacsclient in elisp

[Eli Zaretskii]

> Date: Sat, 22 May 2021 01:05:33 +0300
> From: Jean Louis <bugs@gnu.support>
> Cc: mail@daniel-mendler.de, monnier@iro.umontreal.ca,
>   emacs-devel@gnu.org
> 
> * Eli Zaretskii <eliz@gnu.org> [2021-05-21 20:44]:
> > > Date: Fri, 21 May 2021 19:07:59 +0300
> > > From: Jean Louis <bugs@gnu.support>
> > > Cc: mail@daniel-mendler.de, monnier@iro.umontreal.ca,
> > >   emacs-devel@gnu.org
> > > 
> > > If `read-from-string' also reading from standard input?
> > 
> > The doc string says:
> > 
> >   Read a string from the minibuffer, prompting with string PROMPT.
> > 
> > Given that you just read what happens with reading from minibuffer in
> > batch mode, what do you think is the answer to that question?
> 
> The answer is: maze in documentation. If I am not listening to you I
> would not get it from manual.

Don't listen to me, listen to what the manual says in the sections
which you already read.

Re: emacsclient in elisp

[Daniel Mendler]

On 5/22/21 8:11 AM, Eli Zaretskii wrote:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> I don't think so.  The only problem I could see is interaction between
>> the stdin/stdout code of `read-from-minibuffer` (in batch mode) and
>> `make-stdio-process`, but we could just decide that calling
>> `make-stdio-process` makes future `read-from-minibuffer` signal
>> an error (i.e. you can use either one or the other but not both).
> 
> Doesn't sound like a good idea to me.  There's no real reason for
> being so harsh to applications.

For my use case this would not be harsh. A running Emacs will explicitly
opt-in to use the special make-stdio-process to read from stdin. This
special process will only be used by external worker Emacsen controlled
from the main Emacs. In that scenario the data flowing over stdin-stdout
is tightly controlled (IPC protocol). There hopefully won't be spurious
read-from-minibuffer calls.

Daniel

Re: emacsclient in elisp

[Eli Zaretskii]

> Cc: bugs@gnu.support, emacs-devel@gnu.org
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Sat, 22 May 2021 12:09:24 +0200
> 
> On 5/22/21 8:11 AM, Eli Zaretskii wrote:
> >> From: Stefan Monnier <monnier@iro.umontreal.ca>
> >> I don't think so.  The only problem I could see is interaction between
> >> the stdin/stdout code of `read-from-minibuffer` (in batch mode) and
> >> `make-stdio-process`, but we could just decide that calling
> >> `make-stdio-process` makes future `read-from-minibuffer` signal
> >> an error (i.e. you can use either one or the other but not both).
> > 
> > Doesn't sound like a good idea to me.  There's no real reason for
> > being so harsh to applications.
> 
> For my use case this would not be harsh. A running Emacs will explicitly
> opt-in to use the special make-stdio-process to read from stdin. This
> special process will only be used by external worker Emacsen controlled
> from the main Emacs. In that scenario the data flowing over stdin-stdout
> is tightly controlled (IPC protocol). There hopefully won't be spurious
> read-from-minibuffer calls.

The last assumption is questionable: even if you only talk about batch
mode, it would prevent or complicate using Emacs as a simple filter,
and will get in the way of asking the user any questions (which should
be possible in batch mode).

Re: emacsclient in elisp

[Eli Zaretskii]

> Date: Sat, 22 May 2021 13:19:21 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: monnier@iro.umontreal.ca, bugs@gnu.support, emacs-devel@gnu.org
> 
> > For my use case this would not be harsh. A running Emacs will explicitly
> > opt-in to use the special make-stdio-process to read from stdin. This
> > special process will only be used by external worker Emacsen controlled
> > from the main Emacs. In that scenario the data flowing over stdin-stdout
> > is tightly controlled (IPC protocol). There hopefully won't be spurious
> > read-from-minibuffer calls.
> 
> The last assumption is questionable

I mean in general, not for the single use case you described.

Re: emacsclient in elisp

[Daniel Mendler]

On 5/22/21 12:29 PM, Eli Zaretskii wrote:
>> Date: Sat, 22 May 2021 13:19:21 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: monnier@iro.umontreal.ca, bugs@gnu.support, emacs-devel@gnu.org
>>
>>> For my use case this would not be harsh. A running Emacs will explicitly
>>> opt-in to use the special make-stdio-process to read from stdin. This
>>> special process will only be used by external worker Emacsen controlled
>>> from the main Emacs. In that scenario the data flowing over stdin-stdout
>>> is tightly controlled (IPC protocol). There hopefully won't be spurious
>>> read-from-minibuffer calls.
>>
>> The last assumption is questionable
> 
> I mean in general, not for the single use case you described.

I agree that it is questionable in general. However given that the use
of `make-stdio-process` is opt-in, the change in behavior would not be
severe. One could also keep the current behavior of
`read-from-minibuffer` as is, even if `make-stdio-process` is used. The
question is if that would shadow some bugs, since you probably don't
want to use both of them at the same time. I assume this is what Stefan
meant.

Daniel

Re: emacsclient in elisp

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Sat, 22 May 2021 13:49:20 +0200
> 
> >>> For my use case this would not be harsh. A running Emacs will explicitly
> >>> opt-in to use the special make-stdio-process to read from stdin. This
> >>> special process will only be used by external worker Emacsen controlled
> >>> from the main Emacs. In that scenario the data flowing over stdin-stdout
> >>> is tightly controlled (IPC protocol). There hopefully won't be spurious
> >>> read-from-minibuffer calls.
> >>
> >> The last assumption is questionable
> > 
> > I mean in general, not for the single use case you described.
> 
> I agree that it is questionable in general. However given that the use
> of `make-stdio-process` is opt-in, the change in behavior would not be
> severe.

No one said that by opting-in to this behavior a Lisp program gives up
the ability to ask user questions via the minibuffer.  I can easily
imagine programs that would like to do both.

> One could also keep the current behavior of `read-from-minibuffer`
> as is, even if `make-stdio-process` is used.

??? How do you know which stdin input should go one way and which
should go the other way?

> The question is if that would shadow some bugs, since you probably
> don't want to use both of them at the same time.

What do you mean by "at the same time"?  If I have a stdin-process
active, we watch it via pselect and read from it using the machinery
in wait_reading_process_output and accept-process-output.  During all
the time such a process is active, I see no way to also read from the
same descriptor bypassing the above mechanisms, in a way that won't
cause trouble on the Lisp level, due toe the basic inability to decide
which input should go what way.

Re: emacsclient in elisp

[Stefan Monnier]

> What do you mean by "at the same time"?  If I have a stdin-process
> active, we watch it via pselect and read from it using the machinery
> in wait_reading_process_output and accept-process-output.  During all
> the time such a process is active, I see no way to also read from the
> same descriptor bypassing the above mechanisms, in a way that won't
> cause trouble on the Lisp level, due toe the basic inability to decide
> which input should go what way.

AFAIK we watch this description only when we're inside
"wait_reading_process_output and accept-process-output".  If we're
inside `read-from-minibuffer` then we're not inside
"wait_reading_process_output and accept-process-output" (and
vice-versa), so I don't think there should be much trouble (at least for
my understanding of POSIX file descriptors, which is admittedly somewhat
limited; for Windows, I have no idea).


        Stefan

Re: emacsclient in elisp

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Daniel Mendler <mail@daniel-mendler.de>,  emacs-devel@gnu.org
> Date: Sat, 22 May 2021 09:55:20 -0400
> 
> > What do you mean by "at the same time"?  If I have a stdin-process
> > active, we watch it via pselect and read from it using the machinery
> > in wait_reading_process_output and accept-process-output.  During all
> > the time such a process is active, I see no way to also read from the
> > same descriptor bypassing the above mechanisms, in a way that won't
> > cause trouble on the Lisp level, due toe the basic inability to decide
> > which input should go what way.
> 
> AFAIK we watch this description only when we're inside
> "wait_reading_process_output and accept-process-output".  If we're
> inside `read-from-minibuffer` then we're not inside
> "wait_reading_process_output and accept-process-output" (and
> vice-versa), so I don't think there should be much trouble (at least for
> my understanding of POSIX file descriptors, which is admittedly somewhat
> limited; for Windows, I have no idea).

So we will enter read-from-minibuffer, and read the stuff from stdin
till the first newline.  How do we know what's there is for us, not
for the stdin-process?

The scenario is that we have the process for reading from stdin, but
some function we called decides to ask the user.

Re: emacsclient in elisp

[Stefan Monnier]

> So we will enter read-from-minibuffer, and read the stuff from stdin
> till the first newline.  How do we know what's there is for us, not
> for the stdin-process?

This is a question to ask the programmers who decided to use both at the
same time: it'll be their responsibility to arrange for the calls to
those functions to happen at the right time in the right order and for
the communication protocol to be designed accordingly.  I don't think
there's much we can do on our side.

> The scenario is that we have the process for reading from stdin, but
> some function we called decides to ask the user.

BTW, a plain (while <read-from-minibuffer> <process-command>) setup we
can have the same problem: if one of the functions called by
<process-command> decides unexpectedly to call `read-from-minibuffer`,
it can wreak havoc in your communication protocol.

Again, there's nothing we can do about it on our side: it's the
responsibility of the programmers writing that code to make sure it
doesn't happen.


        Stefan


PS: Admittedly, using a process object will make things a bit worse,
  since, contrary to `read-from-minibuffer`, the programmer has no real
  control over the amount of data read by `accept-process-output`
  before it's passed to the process filter, which is why I said above
  that it's also a question of protocol-design to make sure that if
  there's data meant for `read-from-minibuffer` some mechanism in the
  protocol should make sure it won't be sent while Emacs is inside
  `accept-process-output`.