How to run shell command with stream input, to get string output

How to run shell command with stream input, to get string output

[Jean Louis]

Hello,

I would like to know how to make in Emacs Lisp the
equivalent function to what I have here below in
Common Lisp.

I wish to feed a string as stream to a command and
to get the string output.

For example, I would like to do something like
this:

(setq output
      (shell-command-feed-string "markdown" "Hello\n=====\n\n"))

output would then be something like "<h1>Hello</h1>"

so to receive the string back as result from
markdown parsing.

This type of function then I would use on various
commands, not just markdown, but that maybe one
good example.

I have tried searching for similar in Emacs Lisp
manual, could not find it.

With CLISP Common Lisp implementation:

(defun slurp-stream-io-command2 (command string)
  "Returns the output of a command to which string
has been fed, very usable for markdown, emacs Org
mode processing and similar"
  (let* ((stream (make-pipe-io-stream command :external-format "utf-8"))
         (in (two-way-stream-input-stream stream))
         (out (two-way-stream-output-stream stream))
         (result ""))
    (princ string out)
    (finish-output out)
    (close out)
    (setf result (with-output-to-string (var)
		   (loop for c = (read-char in nil)
		      while c
		      do (format var "~A" c))))
    (finish-output in)
    (close in)
    (close stream)
    result))

Thank you,
Jean

Re: How to run shell command with stream input, to get string output

[Marcin Borkowski]

On 2019-07-01, at 00:32, Jean Louis <bugs@gnu.support> wrote:

> Hello,
>
> I would like to know how to make in Emacs Lisp the
> equivalent function to what I have here below in
> Common Lisp.
>
> I wish to feed a string as stream to a command and
> to get the string output.

How about `shell-command-to-string'?

Hth,

-- 
Marcin Borkowski
http://mbork.pl

Re: How to run shell command with stream input, to get string output

[Jean Louis]

* Marcin Borkowski <mbork@mbork.pl> [2019-07-01 09:22]:
> 
> On 2019-07-01, at 00:32, Jean Louis <bugs@gnu.support> wrote:
> 
> > Hello,
> >
> > I would like to know how to make in Emacs Lisp the
> > equivalent function to what I have here below in
> > Common Lisp.
> >
> > I wish to feed a string as stream to a command and
> > to get the string output.
> 
> How about `shell-command-to-string'?

That one runs only shell command, without the
input.

I am looking to something equivalent to shell

cat | markdown
Hello
=====

## Hello
CTRL-D here
<h1>Hello</h1>

<h2>Hello</h2>

I have to process thousands of files, I would not
like writing to hard disk the feed data for
processing to spare the hard disk.

As I cannot find solution yet in emacs lisp how to
feed some string as input to the shell command, I
am using now the virtual memory in /run/user/$UID
as this way the hard disk is spared of writing
files.

It would be ideal if I do not write anything to
file system.

(defun command-stream-in-out (command string &rest args)
  (let* ((uid (number-to-string (user-uid)))
	 (xdg-runtime-dir (getenv "XDG_RUNTIME_DIR"))
	 (runtime-dir (concat "/run/user/" uid))
	 (runtime-dir (if xdg-runtime-dir xdg-runtime-dir runtime-dir))
	 (infile (concat (slash-add runtime-dir) "markdown-input")))
    (string-to-file-force string infile)
    (with-temp-buffer
      (apply 'call-process command infile (current-buffer) nil args)
      (buffer-string))))

Then I can do something like this:

(defun markdown (string)
  (command-stream-in-out "markdown" string))

(defun pandoc-markdown (string)
  (command-stream-in-out "pandoc" string "-f" "markdown" "-t" "html"))

(markdown "Hello\n=============")

"<h1>Hello</h1>
"

(pandoc-markdown "Hello\n=============")

"<h1 id=\"hello\">Hello</h1>
"

I have solved my problem. But if somebody knows
how to feed the string to command as its input,
without writing to file system, let me know.

Jean

Re: How to run shell command with stream input, to get string output

[tomas]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

On Mon, Jul 01, 2019 at 10:17:16AM +0200, Jean Louis wrote:

[...]

> That one runs only shell command, without the
> input.
> 
> I am looking to something equivalent to shell
> 
> cat | markdown

[...]

Perhaps call-process-region is for you: it passes the current
region as stdin to the invoked process. Cf [1]. If you need
more control (or perhaps an asynchronous process which you can
feed input spoonwise), perhaps [2] is it. But that takes some
"bricolage" :-)

Cheers

[1] Elisp manual 38.3 Creating a Synchronous Process
[2] Elisp manual 38.7 Sending Input to Processes
-- tomás

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: How to run shell command with stream input, to get string output

[Jean Louis]

* tomas@tuxteam.de <tomas@tuxteam.de> [2019-07-01 10:39]:
> Perhaps call-process-region is for you: it passes the current
> region as stdin to the invoked process. Cf [1]. If you need
> more control (or perhaps an asynchronous process which you can
> feed input spoonwise), perhaps [2] is it. But that takes some
> "bricolage" :-)
> 
> Cheers
> 
> [1] Elisp manual 38.3 Creating a Synchronous Process
> [2] Elisp manual 38.7 Sending Input to Processes
> -- tomás

Thank you, I found this one, now just to figure
out how to find process name.

 -- Function: process-send-string process string
     This function sends PROCESS the contents of STRING as standard
     input.  It returns ‘nil’.  For example, to make a Shell buffer list
     files:

          (process-send-string "shell<1>" "ls\n")
               ⇒ nil

Re: How to run shell command with stream input, to get string output

[Marcin Borkowski]

On 2019-07-01, at 10:17, Jean Louis <bugs@gnu.support> wrote:

> * Marcin Borkowski <mbork@mbork.pl> [2019-07-01 09:22]:
>>
>> On 2019-07-01, at 00:32, Jean Louis <bugs@gnu.support> wrote:
>>
>> > Hello,
>> >
>> > I would like to know how to make in Emacs Lisp the
>> > equivalent function to what I have here below in
>> > Common Lisp.
>> >
>> > I wish to feed a string as stream to a command and
>> > to get the string output.
>>
>> How about `shell-command-to-string'?
>
> That one runs only shell command, without the
> input.
>
> I am looking to something equivalent to shell
>
> cat | markdown
> Hello
> =====
>
> ## Hello
> CTRL-D here
> <h1>Hello</h1>
>
> <h2>Hello</h2>
>
> I have to process thousands of files, I would not
> like writing to hard disk the feed data for
> processing to spare the hard disk.

I'm not sure I get it.

If you have your data in _files_, why don't you

(shell-command-to-string "cat <filename> | ...")

?

Even if you only have it as string within Emacs, you could probably try
(format "echo '%s'" ...), though this might involve escaping problems.

> As I cannot find solution yet in emacs lisp how to
> feed some string as input to the shell command, I
> am using now the virtual memory in /run/user/$UID
> as this way the hard disk is spared of writing
> files.

> [...]

> I have solved my problem. But if somebody knows
> how to feed the string to command as its input,
> without writing to file system, let me know.

Frankly, I'd also like to learn how to solve this.  The above
suggestions are just my 2cents, which probably do not answer your
question exactly, but might inspire someone to get a proper solution...

Best,

--
Marcin Borkowski
http://mbork.pl

Re: How to run shell command with stream input, to get string output

[Jean Louis]

* Marcin Borkowski <mbork@mbork.pl> [2019-07-01 11:14]:
> I'm not sure I get it.
> 
> If you have your data in _files_, why don't you
> 
> (shell-command-to-string "cat <filename> | ...")
> 
> ?

The data is not in files, it is in the database.

My Website Revision System is producing static
HTML pages. Those updates to multiple websites
have thousands and thousands of pages, and if I
would be creating files for each invocation, that
would shorten the life time of a hard disk and
make it slower.

And such easy invokation of shell command with
some input is useful for various other utilities,
for example conversion of coordinates, like here.

(defun proj/arc1960-wgs84 (latitude longitude &optional time height)
  "Converts single coordinates in DD format from
ARC1960 to WGS84 with default height, see
https://github.com/OSGeo/proj.4/issues/1110 and
https://earth-info.nga.mil/GandG/coordsys/onlinedatum/CountryAfricaTable.html"
  (let* ((lat-lon (proj/convert-arc1960-to-wgs84 latitude longitude))
	 (lat-lon (string-trim lat-lon))
	 (lat-lon (split-string lat-lon))
	 (latitude (first lat-lon))
	 (longitude (second lat-lon))
    	 (height (third lat-lon)))
    (list latitude longitude height)))

;; original point -1.47666 34.56861
;; geotrans point -1.47927 34.56933
;; (proj/arc1960-wgs84 -1.47666 34.56861)

(defun proj/convert-arc1960-to-wgs84 (latitude longitude)
  (let ((string (format "%s %s\n" latitude longitude)))
    (command-stream-in-out "cs2cs" string "-f" "%.5f" "Arc 1960" "WGS84")))

;; (proj/arc1960-wgs84 -1.47666 34.56861)

With result being ("-1.47926" "34.56938" "0.00000")

Imagine having thousands of geographic locations
in the database that require conversion, then for
each would be created one file on hard disk.

Until I learn how to use process-send-string, I am
using memory files in /run/user/$UID

Jean

Re: How to run shell command with stream input, to get string output

[Robert Pluim]

>>>>> On Mon, 1 Jul 2019 11:06:23 +0200, Jean Louis <bugs@gnu.support> said:

    Jean> * tomas@tuxteam.de <tomas@tuxteam.de> [2019-07-01 10:39]:
    >> Perhaps call-process-region is for you: it passes the current
    >> region as stdin to the invoked process. Cf [1]. If you need
    >> more control (or perhaps an asynchronous process which you can
    >> feed input spoonwise), perhaps [2] is it. But that takes some
    >> "bricolage" :-)
    >> 
    >> Cheers
    >> 
    >> [1] Elisp manual 38.3 Creating a Synchronous Process
    >> [2] Elisp manual 38.7 Sending Input to Processes
    >> -- tomás

    Jean> Thank you, I found this one, now just to figure
    Jean> out how to find process name.

Another option would be using start-process, process-send-region,
and process-send-eof (all described in the manual).

Robert

Re: How to run shell command with stream input, to get string output

[Jean Louis]

* Robert Pluim <rpluim@gmail.com> [2019-07-01 11:39]:
>     Jean> Thank you, I found this one, now just to figure
>     Jean> out how to find process name.
> 
> Another option would be using start-process, process-send-region,
> and process-send-eof (all described in the
> manual).

(with-temp-buffer
  (let* ((process (start-process "NEW" (current-buffer) "cat")))
    (process-send-string process "Hello")
    (process-send-eof process)
    (kill-process process)
    (buffer-string)))

Something like that, but it does not work as
output of process is not written in the buffer.

If you know how, let me know.

Jean

Re: How to run shell command with stream input, to get string output

[Noam Postavsky]

On Mon, 1 Jul 2019 at 05:45, Jean Louis <bugs@gnu.support> wrote:

> Something like that, but it does not work as
> output of process is not written in the buffer.

When input is not newline terminated, cat requires two EOFs. I find
this is the case when running outside of Emacs as well.

(with-temp-buffer
  (let* ((process (start-process "NEW" (current-buffer) "cat")))
    ;; Don't include "Process NEW finished\n".
    (set-process-sentinel process #'ignore)
    (process-send-string process "Hello")
    (process-send-eof process)
    (process-send-eof process)
    (while (accept-process-output process))
    (buffer-string)))

But if you don't need asynchoronous input, call-process-region is simpler:

(with-temp-buffer
  (insert "Hello")
  (call-process-region (point-min) (point-max)
               "tr" t '(t t) nil "a-z" "A-Z")
  (buffer-string))

Re: How to run shell command with stream input, to get string output

[Jean Louis]

* Noam Postavsky <npostavs@gmail.com> [2019-07-01 13:13]:
> On Mon, 1 Jul 2019 at 05:45, Jean Louis <bugs@gnu.support> wrote:
> 
> > Something like that, but it does not work as
> > output of process is not written in the buffer.
> 
> When input is not newline terminated, cat requires two EOFs. I find
> this is the case when running outside of Emacs as well.
>

Excellent! Thank you.

I see it can work as below, and even markdown needs two times EOF.

(with-temp-buffer
  (let* ((process (make-process :name "NEW" 
                                :buffer (current-buffer)
				:command '("markdown")
				:sentinel #'ignore)))
    (process-send-string process "Hello\n============")
    (process-send-eof process)
    (process-send-eof process)
    (while (accept-process-output process))
    (buffer-string)))

Maybe process-send-eof is making sure of one blank line on the end, as this works without 2 EOF:

(with-temp-buffer
  (let* ((process (make-process :name "NEW" 
                                :buffer (current-buffer)
				:command '("markdown")
				:sentinel #'ignore)))
    (process-send-string process "Hello\n============\n")
    (process-send-eof process)
    (while (accept-process-output process))
    (buffer-string)))

And this works too with proj software to convert coordinates, so I will use your example.

(with-temp-buffer
  (let* ((process (start-process "NEW" (current-buffer) "cs2cs" "-f" "%.5f" "Arc 1960" "WGS84")))
    (set-process-sentinel process #'ignore)
    (process-send-string process "-1.47666 34.56861")
    (process-send-eof process)
    (while (accept-process-output process))
    (buffer-string)))


Thank you,
Jean

Re: How to run shell command with stream input, to get string output

[Marcin Borkowski]

On 2019-07-01, at 11:24, Jean Louis <bugs@gnu.support> wrote:

> * Marcin Borkowski <mbork@mbork.pl> [2019-07-01 11:14]:
>> I'm not sure I get it.
>> 
>> If you have your data in _files_, why don't you
>> 
>> (shell-command-to-string "cat <filename> | ...")
>> 
>> ?
>
> The data is not in files, it is in the database.

Ah, now that finally makes sense to me.

Unfortunately, I can't help with this - but thanks for your patience,
I didn't get the problem earlier.

Best,

-- 
Marcin Borkowski
http://mbork.pl

Re: How to run shell command with stream input, to get string output

[Jean Louis]

* Marcin Borkowski <mbork@mbork.pl> [2019-07-04 21:19]:
> > The data is not in files, it is in the database.
> 
> Ah, now that finally makes sense to me.
> 
> Unfortunately, I can't help with this - but thanks for your patience,
> I didn't get the problem earlier.

Majority of Website Revision Systems require
some templating mechanism, so I am just moving
from Common Lisp CL-EMB to doing the same in emacs.

Reference to Website Revision System:
https://www.gnu.org/philosophy/words-to-avoid.html#Content

I started with this one, it works reasonably well
for faxing from Emacs. Imagine multiple templates,
multiple faxes, variables like first name, last
name, fax numbers, need to be replaced in the
templates. 

(defun rcd/expand-@template@ (template plist)
  "Expands the template that contains @vars@ with plist member values from plist"
  (let ((rx (rx "@" (group (one-or-more (or (any alpha) "_"))) "@")))
    (with-temp-buffer
      (insert template)
      (goto-char 0)
      (while (re-search-forward rx nil t)
	(let* ((var-found (match-string 1))
	       (var-plist (or (plist-get plist (intern var-found)) ""))
	       (var-plist (if (numberp var-plist) (number-to-string var-plist) var-plist)))
	  (replace-match var-plist nil nil)))
      (buffer-string))))

(defun rcd/xml-escape (any)
  (let ((type (type-of any)))
    (cond ((eq 'string type) (xml-escape-string any))
	  ((eq 'integer type) (number-to-string any)))))

This one works better than xml-escape-string for
my purposes, maybe it is not efficient, but the
other one is not working in my templates.

(defun xml-escape (string)
  "Escape XML without matching problems"
  (let ((chars (string-to-list string))
	(nlist '()))
    (dolist (c chars (list-of-strings-to-string (reverse nlist)))
      (let ((char (char-to-string c)))
	(push 
	 (cond ((string= ">" char) "&gt;")
	       ((string= "<" char) "&lt;")
	       ((string= "\"" char) "&quot;")
	       ((string= "'" char) "&#39;")
	       ((string= "&" char) "&amp;")
               (t char))
	 nlist)))))

(defun rcd/expand-<%template%>-plist (template plist)
  "Expands the template that contains <% var %> with plist member values from plist"
    (let ((rx (rx "<%" (one-or-more (or blank "\n")) (group (minimal-match (one-or-more (or (any alpha) "-" "_")))) (one-or-more (or blank "\n")) "-escape" (one-or-more (or blank "\n")) "%>")))
    (with-temp-buffer
      (insert template)
      (goto-char (point-min))
      (while (re-search-forward rx nil t)
	(let* ((var-found (match-string 1))
	       (value-found (or (plist-get plist (intern var-found)) nil))
	       (var-plist (if value-found (plist-get plist (intern var-found)) nil)))
	  (if (and var-found value-found var-plist)
	      (replace-match (xml-escape var-plist) nil nil)
	    (replace-match (concat "<% " var-found " %>") nil nil))))
      (let ((rx2 (rx "<%" (one-or-more (or blank "\n")) (group (minimal-match (one-or-more (or (any alpha) "-" "_")))) (one-or-more (or blank "\n")) "%>")))
	(goto-char (point-min))
	(while (re-search-forward rx2 nil t)
	  (let* ((var-found (match-string 1))
		 (value-found (or (plist-get plist (intern var-found)) nil))
		 (var-plist (if value-found (or (plist-get plist (intern var-found)) (concat "<% " var-found " %>")))))
	    (if (and var-found value-found)
		(replace-match var-plist nil nil)
	      (replace-match (concat "<% " var-found " %>" nil nil)))))
      (buffer-string)))))
	  
(defun rcd/expand-<%template%>-eval (template)
    (let ((rx (rx "<%" (one-or-more (or blank "\n")) (group (minimal-match (one-or-more anything))) (one-or-more (or blank "\n")) "%>")))
      (with-temp-buffer
	(insert template)
	(goto-char (point-min))
	(while (re-search-forward rx nil t)
	  (let* ((eval-found (match-string 1))
		 (eval-value (condition-case nil (eval (car (read-from-string eval-found))) (error "")))
		 (eval-value (if eval-value (format "%s" eval-value) "")))
	    (if eval-found
		(replace-match eval-value nil nil)
	      (replace-match "" nil nil))))
		(buffer-string))))

(defun rcd/expand-<%template%> (template plist)
  "Expands <% templates %> with plist, global variables and any functions"
  (let* ((plist (rcd/expand-<%template%>-plist template plist))
	 (eval (rcd/expand-<%template%>-eval plist)))
    eval))

(defun gold-price-kg ()
      45553.12)
      
And here is working example.

;; (rcd/expand-<%template%> "<title><% (gold-price-kg) %></title> {<% title-some -escape %>} {<% title-some %>}" '(title-some "<<SOMETHING>>"))

If somebody wish to propose better regex let me
know.

Jean

Re: How to run shell command with stream input, to get string output

[Jean Louis]

This is version that I am using now.

(defun command-stream (command string &rest args)
"Feeds string as input to command"
  (with-temp-buffer
    (let* ((process (apply 'start-process "PROCESS" (current-buffer) command args)))
      (set-process-sentinel process #'ignore)
      (process-send-string process string)
      (process-send-eof process)
      (process-send-eof process)
      (while (accept-process-output process))
      (buffer-string))))

(defun markdown (string)
  "This is for discount markdown"
  (command-stream "markdown" string))

(markdown "## Hello")

(defun pandoc-markdown (string)
  (command-stream "pandoc" string "-f" "markdown" "-t" "html"))
  
(pandoc-markdown "## Hello")

(defun proj/convert-arc1960-to-wgs84 (latitude longitude)
  (let ((string (format "%s %s\n" latitude longitude)))
    (command-stream "cs2cs" string "-f" "%.5f" "Arc 1960" "WGS84")))

(defun proj/arc1960-wgs84 (latitude longitude &optional time height)
  "Converts single coordinates in DD format from
ARC1960 to WGS84 with default height, see
https://github.com/OSGeo/proj.4/issues/1110 and
https://earth-info.nga.mil/GandG/coordsys/onlinedatum/CountryAfricaTable.html"
  (let* ((lat-lon (proj/convert-arc1960-to-wgs84 latitude longitude))
	 (lat-lon (string-trim lat-lon))
	 (lat-lon (split-string lat-lon))
	 (latitude (first lat-lon))
	 (longitude (second lat-lon))
    	 (height (third lat-lon)))
    (list latitude longitude height)))

;; original point -1.47666 34.56861
;; geotrans point -1.47927 34.56933
;; (proj/arc1960-wgs84 -1.47666 34.56861)

I call it command-stream, probably is wrong.

Jean