[Proposal] Why not add a "shell" procedure?

[Proposal] Why not add a "shell" procedure?

[Nala Ginrut]

hi folks!
Sometimes we need to run shell script and get the result as string type.
Say, in Ruby:
irb: `ls`
==> "ice-9\nlanguage\nMakefile.am\nMakefile.am~\nMakefile.in\noop\nrnrs\nrnrs.scm\nscripts\nsrfi\nstatprof.scm\nsxml\nsystem\ntexinfo\ntexinfo.scm\nweb\n"

* Note: "system" lib function is useless for this, because "system"
can't return the result as string, but just the retval.

This feature is very easy to implement in Guile, but I recommend to
add a global env-var %current-shell to point any shell-interpreter,
like csh/bash/sh/..., or modify it as user wish.
The "shell" implementation maybe like this:
-------------code----------------
(define %current-shell (getenv "SHELL"))
(use-modules (ice-9 popen) (rnrs io ports))
(define shell
   (lambda (cmd)
       (let ((str (string-append %current-shell " -c " cmd)))
          (get-string-all (open-pipe str OPEN_READ)))))
-------------end----------------

and use it like regular shell:
(shell "sed -i \"s:guile/Guile/g" somefile")

Moreover, we can implement "pwd" with this "shell" easily:
----------code----------
(use-module (ice-9 rdelim))
(define (pwd)
  (call-with-input-string (shell "pwd")
                                     (lambda (port) (read-line port))))
-----------end-----------

(pwd)
==> "/home/nalaginrut/Project/gnulib-20100109+stable"

Any comment?

Re: [Proposal] Why not add a "shell" procedure?

[Nala Ginrut]

Well, sed is an improper example, I should have deleted "-i":
 (shell "sed \"s:guile/Guile/g" somefile")

On Sat, May 12, 2012 at 8:30 PM, Nala Ginrut <nalaginrut@gmail.com> wrote:
> hi folks!
> Sometimes we need to run shell script and get the result as string type.
> Say, in Ruby:
> irb: `ls`
> ==> "ice-9\nlanguage\nMakefile.am\nMakefile.am~\nMakefile.in\noop\nrnrs\nrnrs.scm\nscripts\nsrfi\nstatprof.scm\nsxml\nsystem\ntexinfo\ntexinfo.scm\nweb\n"
>
> * Note: "system" lib function is useless for this, because "system"
> can't return the result as string, but just the retval.
>
> This feature is very easy to implement in Guile, but I recommend to
> add a global env-var %current-shell to point any shell-interpreter,
> like csh/bash/sh/..., or modify it as user wish.
> The "shell" implementation maybe like this:
> -------------code----------------
> (define %current-shell (getenv "SHELL"))
> (use-modules (ice-9 popen) (rnrs io ports))
> (define shell
>   (lambda (cmd)
>       (let ((str (string-append %current-shell " -c " cmd)))
>          (get-string-all (open-pipe str OPEN_READ)))))
> -------------end----------------
>
> and use it like regular shell:
> (shell "sed -i \"s:guile/Guile/g" somefile")
>
> Moreover, we can implement "pwd" with this "shell" easily:
> ----------code----------
> (use-module (ice-9 rdelim))
> (define (pwd)
>  (call-with-input-string (shell "pwd")
>                                     (lambda (port) (read-line port))))
> -----------end-----------
>
> (pwd)
> ==> "/home/nalaginrut/Project/gnulib-20100109+stable"
>
> Any comment?

Re: [Proposal] Why not add a "shell" procedure?

[Thien-Thi Nguyen]

() Nala Ginrut <nalaginrut@gmail.com>
() Sat, 12 May 2012 20:30:21 +0800

   (pwd)
   ==> "/home/nalaginrut/Project/gnulib-20100109+stable"

   Any comment?

For this particular case, you can use instead ‘getcwd’, which
is builtin.  See also ‘shell-command->string’ and friends:

 http://www.gnuvola.org/software/ttn-do/ttn-do.html.gz

Tangentially, Guile 1.4.x has "guile-tools gxsed", a sed
implementation in Scheme:

 http://www.gnuvola.org/software/guile/doc/guile_002dtools-gxsed.html

It needs some work to be compatible w/ Guile 1.8 and 2.0
(i.e., to use 2-arg ‘eval’), but the core transform is sound.
IIRC it PASSed around 60% of the GNU sed 4.2 "make check".

For Emacsoid in-process editing, see also module (ttn-do mogrify),
also in ttn-do.  I (and my daemons) use it every day w/ Guile 1.8.

Re: [Proposal] Why not add a "shell" procedure?

[Nelson H. F. Beebe]

Nala Ginrut <nalaginrut@gmail.com> writes today about adding a
function to return a string from a command executed by a shell,
because (system "...") only returns the exit status of the last
command.

The one point that I think is important is the handling of whitespace.

The Unix shells in their backquoted `cmd` and $(cmd) syntax normalize
the output of cmd to replace runs of whitespace [ \n\r\v\t]+ by single
spaces.  

For shell use, that is convenient.  However, inside guile, it may be
more useful to preserve the whitespace as is, and particularly the
line breaks.  

A user who wants whitespace collapsed to single spaces can easily do
so with a regexp substitution, and that could be facilitated by
providing a standard guile library function to do the job
consistently.

-------------------------------------------------------------------------------
- Nelson H. F. Beebe                    Tel: +1 801 581 5254                  -
- University of Utah                    FAX: +1 801 581 4148                  -
- Department of Mathematics, 110 LCB    Internet e-mail: beebe@math.utah.edu  -
- 155 S 1400 E RM 233                       beebe@acm.org  beebe@computer.org -
- Salt Lake City, UT 84112-0090, USA    URL: http://www.math.utah.edu/~beebe/ -
-------------------------------------------------------------------------------

Re: [Proposal] Why not add a "shell" procedure?

[Nala Ginrut]

Hmm..."pwd" and "sed" are just examples I showed how convenient this
"shell" could be.
But I missed "pwd" should be implemented by "getcwd".
Anyway, my aim is not to implement "pwd" or "sed", but the "shell"
alike in Ruby.

On Sat, May 12, 2012 at 10:22 PM, Thien-Thi Nguyen <ttn@gnuvola.org> wrote:
> () Nala Ginrut <nalaginrut@gmail.com>
> () Sat, 12 May 2012 20:30:21 +0800
>
>   (pwd)
>   ==> "/home/nalaginrut/Project/gnulib-20100109+stable"
>
>   Any comment?
>
> For this particular case, you can use instead ‘getcwd’, which
> is builtin.  See also ‘shell-command->string’ and friends:
>
>  http://www.gnuvola.org/software/ttn-do/ttn-do.html.gz
>
> Tangentially, Guile 1.4.x has "guile-tools gxsed", a sed
> implementation in Scheme:
>
>  http://www.gnuvola.org/software/guile/doc/guile_002dtools-gxsed.html
>
> It needs some work to be compatible w/ Guile 1.8 and 2.0
> (i.e., to use 2-arg ‘eval’), but the core transform is sound.
> IIRC it PASSed around 60% of the GNU sed 4.2 "make check".
>
> For Emacsoid in-process editing, see also module (ttn-do mogrify),
> also in ttn-do.  I (and my daemons) use it every day w/ Guile 1.8.

Re: [Proposal] Why not add a "shell" procedure?

[Nala Ginrut]

On Sat, May 12, 2012 at 10:37 PM, Nelson H. F. Beebe
<beebe@math.utah.edu> wrote:
> Nala Ginrut <nalaginrut@gmail.com> writes today about adding a
> function to return a string from a command executed by a shell,
> because (system "...") only returns the exit status of the last
> command.
>
> The one point that I think is important is the handling of whitespace.
>
> The Unix shells in their backquoted `cmd` and $(cmd) syntax normalize
> the output of cmd to replace runs of whitespace [ \n\r\v\t]+ by single
> spaces.
>
> For shell use, that is convenient.  However, inside guile, it may be
> more useful to preserve the whitespace as is, and particularly the
> line breaks.
>

Yes, I don't think we should handle whitespace for users either. And
the "shell" in Ruby dosen't too.
I think my implementation reflects this.

> A user who wants whitespace collapsed to single spaces can easily do
> so with a regexp substitution, and that could be facilitated by
> providing a standard guile library function to do the job
> consistently.
>

well, this could be interesting.
Anyway, I think we should have an inner way to run shell first.

> -------------------------------------------------------------------------------
> - Nelson H. F. Beebe                    Tel: +1 801 581 5254                  -
> - University of Utah                    FAX: +1 801 581 4148                  -
> - Department of Mathematics, 110 LCB    Internet e-mail: beebe@math.utah.edu  -
> - 155 S 1400 E RM 233                       beebe@acm.org  beebe@computer.org -
> - Salt Lake City, UT 84112-0090, USA    URL: http://www.math.utah.edu/~beebe/ -
> -------------------------------------------------------------------------------
>

Re: [Proposal] Why not add a "shell" procedure?

[Mark H Weaver]

Hi Nala,

Nala Ginrut <nalaginrut@gmail.com> writes:
> (define %current-shell (getenv "SHELL"))
> (use-modules (ice-9 popen) (rnrs io ports))
> (define shell
>    (lambda (cmd)
>        (let ((str (string-append %current-shell " -c " cmd)))
>           (get-string-all (open-pipe str OPEN_READ)))))

(open-pipe <STRING> ...) already does '/bin/sh -c <STRING>', so (shell
<STRING>) does '/bin/sh -c "<SHELL> -c <STRING>"', i.e. it launches a
shell within a shell.  This is wasteful, and might also exacerbate
problems when shell metacharacters are present in <STRING> or <SHELL>.

Therefore, better do (open-pipe* OPEN_READ %current-shell "-c" cmd)
instead.

Also, I recommend making '%current-shell' a fluid, and perhaps adding a
keyword argument to 'shell' to specify the shell directly, so that
'%current-shell' is only used as the default for the keyword argument.

Note that although it is convenient to pass strings directly to the
shell, it is fraught with security risks due to the complexity of
escaping shell metacharacters properly, especially given the diversity
of shells and shell configurations.  Therefore, it is better not to
encourage this way of doing things.  It is generally better to use
'open-pipe*' or 'system*' directly, to avoid the shell entirely.

     Thanks,
       Mark

Re: [Proposal] Why not add a "shell" procedure?

[Nala Ginrut]

hi Mark! Thanks for reply!
One reason to proposal this "shell" is to handle these metacharacters
in programming way. I think it's easier than input them with keyboard.
So my vote is to provide a more safer "shell" and encourage it rather
than using raw "open-pipe*".
What do you think?


On Mon, May 14, 2012 at 2:29 AM, Mark H Weaver <mhw@netris.org> wrote:
> Hi Nala,
>
> Nala Ginrut <nalaginrut@gmail.com> writes:
>> (define %current-shell (getenv "SHELL"))
>> (use-modules (ice-9 popen) (rnrs io ports))
>> (define shell
>>    (lambda (cmd)
>>        (let ((str (string-append %current-shell " -c " cmd)))
>>           (get-string-all (open-pipe str OPEN_READ)))))
>
> (open-pipe <STRING> ...) already does '/bin/sh -c <STRING>', so (shell
> <STRING>) does '/bin/sh -c "<SHELL> -c <STRING>"', i.e. it launches a
> shell within a shell.  This is wasteful, and might also exacerbate
> problems when shell metacharacters are present in <STRING> or <SHELL>.
>
> Therefore, better do (open-pipe* OPEN_READ %current-shell "-c" cmd)
> instead.
>
> Also, I recommend making '%current-shell' a fluid, and perhaps adding a
> keyword argument to 'shell' to specify the shell directly, so that
> '%current-shell' is only used as the default for the keyword argument.
>
> Note that although it is convenient to pass strings directly to the
> shell, it is fraught with security risks due to the complexity of
> escaping shell metacharacters properly, especially given the diversity
> of shells and shell configurations.  Therefore, it is better not to
> encourage this way of doing things.  It is generally better to use
> 'open-pipe*' or 'system*' directly, to avoid the shell entirely.
>
>     Thanks,
>       Mark