Re: bug#1212: 23.0.60; split-string-and-unquote problems

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: 1212@emacsbugs.donarmstrong.com, emacs-pretest-bug@gnu.org
> Date: Mon, 20 Oct 2008 23:28:21 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andreas Schwab <schwab@suse.de>
> >> Cc: 1212@emacsbugs.donarmstrong.com, emacs-pretest-bug@gnu.org
> >> Date: Mon, 20 Oct 2008 19:08:35 +0200
> >> 
> >> Why do you think that "Emacs Lisp quoting" has anything to do with
> >> "shell command quoting"?
> >
> > Because I grep'ped Emacs sources for its users.
> 
> That does not make it more suitable.

You know, since you evidently know better, how about if you explained
what that pair of functions is supposed to do, exactly, and what are
their intended uses?  The doc strings fall short by a large margin,
and NEWS had this to say about these functions:

    *** `split-string-and-unquote' does (what?)

    *** `combine-and-quote-strings' does (what?)

If I can understand the explanation, maybe I could produce from it a
suitable section of the ELisp manual, which is the reason why I
started to play with split-string-and-unquote.  Failing that, I will
stick to my interpretation.

Here's what I wrote in the ELisp manual about these two functions;
comments and corrections, if someone has them, are greatly
appreciated:

  @cindex quoting and unquoting shell command line
    The following two functions help creating shell commands from
  individual argument strings and taking shell command lines apart into
  individual arguments.

  @defun split-string-and-unquote string &optional separators
  This function splits @var{string} into substrings at matches for the
  regular expression @var{separators}, like @code{split-string} does
  (@pxref{Creating Strings}), but it additionally removes quoting from
  the substrings.  It then makes a list of the substrings and returns
  it.

  If @var{separators} is omitted or nil, it defaults to @code{"\\s-+"},
  which is a regular expression that matches one or more characters with
  whitespace syntax (@pxref{Syntax Class Table}).

  The quoting this function supports is of 2 styles: by enclosing a
  whole string in double quotes @code{"@dots{}"}, or by quoting
  individual characters with a backslash escape @samp{\}.  The latter is
  also used in Lisp strings, so this function can handle those as well.
  @end defun

  @defun combine-and-quote-strings list-of-strings &optional separator
  This function concatenates @var{list-of-strings} into a single string,
  quoting each string in the list that needs quoting as it goes.  It
  also sticks the @var{separator} string in between each pair of strings
  in the result, and returns that result.  If @var{separator} is omitted
  or @code{nil}, it defaults to a blank @code{" "}.

  The strings in @var{list-of-strings} that need quoting are those that
  include @var{separator} as their substring.  Quoting a string encloses
  it in double quotes @code{"@dots{}"}.  In the simplest case, if you
  are consing a shell command from the individual command-line
  arguments, every argument that includes embedded blanks will be
  quoted.
  @end defun

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Stefan Monnier]

>     *** `split-string-and-unquote' does (what?)
>     *** `combine-and-quote-strings' does (what?)

AFAIK, these two functions are meant to:

- provide a way to specify any list of strings within a single string
  (i.e. it needs to provide some separators and some way to quote the
  separators), using a format that's simple to type for the end user.
- be inverse of each other; more precisely
  (equal STR (split-string-and-unquote (combine-and-quote-strings STR)))
  this is also stipulated in the docstring of split-string-and-unquote.

As the docstring of split-string-and-unquote explains, the quoting used
is the same as the one used for ELisp strings.

I could have used sh-style quoting or csh-style, or any other style, but
csh-style is nasty, sh-style is too powerful (what should we do with
backquotes and $(...)), and ELisp is what Emacs uses already and it's
reasonably simple and we know it handles all cases reasonably well, so
it seemed like an OK choice.


        Stefan

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Andreas Schwab <schwab@suse.de>,  emacs-pretest-bug@gnu.org,  1212@emacsbugs.donarmstrong.com,  emacs-devel@gnu.org
> Date: Mon, 20 Oct 2008 21:11:10 -0400
> 
> >     *** `split-string-and-unquote' does (what?)
> >     *** `combine-and-quote-strings' does (what?)
> 
> AFAIK, these two functions are meant to:
> 
> - provide a way to specify any list of strings within a single string
>   (i.e. it needs to provide some separators and some way to quote the
>   separators), using a format that's simple to type for the end user.

Thanks.

However, this describes roughly what the code does (which I kinda
understood myself by reading it ;-), but does not explain under what
circumstances would those functions be useful.  By looking at the uses
of split-string-and-unquote, I deduced that the intended use is for
taking apart shell command lines.  If that is not the intent, then
what is it?

> - be inverse of each other; more precisely
>   (equal STR (split-string-and-unquote (combine-and-quote-strings STR)))
>   this is also stipulated in the docstring of split-string-and-unquote.

The doc strings make very clear that the functions are reversible, but
that in itself does not yet mean they are useful, or for what purpose
exactly.

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Stefan Monnier]

> However, this describes roughly what the code does (which I kinda
> understood myself by reading it ;-), but does not explain under what
> circumstances would those functions be useful.  By looking at the uses
> of split-string-and-unquote, I deduced that the intended use is for
> taking apart shell command lines.  If that is not the intent, then
> what is it?

The intent is for the user to be able to enter in a minibuffer a list of
strings, where most of them should usually not need any quoting.
The most common use is to enter a list of arguments to pass to
a command.  Note that "arguments to pass to a command" (even if the
first element could be the command itself) is not the same as "shell
command line", since it does not include operators such as |, &, ;, $,
and `.


        Stefan

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Tue, 21 Oct 2008 11:43:25 -0400
> 
> > However, this describes roughly what the code does (which I kinda
> > understood myself by reading it ;-), but does not explain under what
> > circumstances would those functions be useful.  By looking at the uses
> > of split-string-and-unquote, I deduced that the intended use is for
> > taking apart shell command lines.  If that is not the intent, then
> > what is it?
> 
> The intent is for the user to be able to enter in a minibuffer a list of
> strings, where most of them should usually not need any quoting.
> The most common use is to enter a list of arguments to pass to
> a command.  Note that "arguments to pass to a command" (even if the
> first element could be the command itself) is not the same as "shell
> command line", since it does not include operators such as |, &, ;, $,
> and `.

Well, in that case my understanding of the intent of these functions
was quite accurate.  Which also means that split-string-and-unquote
should, indeed, handle quoting with a backslash and ".." quoting in
the middle of a word, as in `foo" "bar'.  Right?

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Stefan Monnier]

>> > However, this describes roughly what the code does (which I kinda
>> > understood myself by reading it ;-), but does not explain under what
>> > circumstances would those functions be useful.  By looking at the uses
>> > of split-string-and-unquote, I deduced that the intended use is for
>> > taking apart shell command lines.  If that is not the intent, then
>> > what is it?
>> 
>> The intent is for the user to be able to enter in a minibuffer a list of
>> strings, where most of them should usually not need any quoting.
>> The most common use is to enter a list of arguments to pass to
>> a command.  Note that "arguments to pass to a command" (even if the
>> first element could be the command itself) is not the same as "shell
>> command line", since it does not include operators such as |, &, ;, $,
>> and `.

> Well, in that case my understanding of the intent of these functions
> was quite accurate.  Which also means that split-string-and-unquote
> should, indeed, handle quoting with a backslash and ".." quoting in
> the middle of a word, as in `foo" "bar'.  Right?

No.  Just because the arguments are passed to commands doesn't mean they
should use sh-style quoting.  Proof is, neither csh nor Tcl obey sh-style
quoting for the arguments to commands they run.
The list of strings will be passed to call-process or start-process
which ultimately will pass them to execv or somesuch: no shell in sight.


        Stefan

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Tue, 21 Oct 2008 22:04:51 -0400
> 
> The list of strings will be passed to call-process or start-process
> which ultimately will pass them to execv or somesuch: no shell in sight.

But the original string could have been a properly quoted shell
command, and those do use sh-style quoting.  The function needs to
remove that quoting.

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Stefan Monnier]

>> The list of strings will be passed to call-process or start-process
>> which ultimately will pass them to execv or somesuch: no shell in sight.
> But the original string could have been a properly quoted shell
> command, and those do use sh-style quoting.

I do not understand: what makes you think it could be a properly quoted
shell command?

/We/ define what it can accept.  And as of now, we (well, admittedly,
IIUC, it's mostly just myself) decided that it accepts a syntax derived
from Elisp string quoting, so if you feed it sh-style quoted strings, it
won't work in general (tho it will if you stick to the common subset, of
course).


        Stefan

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Wed, 22 Oct 2008 10:58:49 -0400
> 
> >> The list of strings will be passed to call-process or start-process
> >> which ultimately will pass them to execv or somesuch: no shell in sight.
> > But the original string could have been a properly quoted shell
> > command, and those do use sh-style quoting.
> 
> I do not understand: what makes you think it could be a properly quoted
> shell command?

Because you said that working with shell commands was why these
functions were invented in the first place.  And that is how they are
used in Emacs as of now.

> /We/ define what it can accept.

But hopefully, /we/ decide that to accomplish some specific practical
goal, not just to craft a function that accepts a small subset of that
goal.  Right?  So what class of _practical_ problems do these two
functions solve?  After all, using Lisp syntax in shell commands is
not an interesting use-case, is it?

> And as of now, we (well, admittedly,
> IIUC, it's mostly just myself) decided that it accepts a syntax derived
> from Elisp string quoting, so if you feed it sh-style quoted strings, it
> won't work in general (tho it will if you stick to the common subset, of
> course).

Are you saying that modes that work with shell commands, such as GUD,
should not use these functions, because they don't generally support
the full syntax of quoted shell commands?

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Stefan Monnier]

>> >> The list of strings will be passed to call-process or start-process
>> >> which ultimately will pass them to execv or somesuch: no shell in sight.
>> > But the original string could have been a properly quoted shell
>> > command, and those do use sh-style quoting.
>> 
>> I do not understand: what makes you think it could be a properly quoted
>> shell command?

> Because you said that working with shell commands was why these
> functions were invented in the first place.  And that is how they are
> used in Emacs as of now.

No, they're used to run commands.  Without going through a shell.

> Are you saying that modes that work with shell commands, such as GUD,
> should not use these functions, because they don't generally support
> the full syntax of quoted shell commands?

IIUC GUD does not use a shell, so it can use this just fine.


        Stefan

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Wed, 22 Oct 2008 17:42:36 -0400
> >> I do not understand: what makes you think it could be a properly quoted
> >> shell command?
> 
> > Because you said that working with shell commands was why these
> > functions were invented in the first place.  And that is how they are
> > used in Emacs as of now.
> 
> No, they're used to run commands.  Without going through a shell.

Then why would one need quoting?

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Stefan Monnier]

>> >> I do not understand: what makes you think it could be a properly quoted
>> >> shell command?
>> > Because you said that working with shell commands was why these
>> > functions were invented in the first place.  And that is how they are
>> > used in Emacs as of now.
>> No, they're used to run commands.  Without going through a shell.
> Then why would one need quoting?

Because when you run a command with start-process or call-process, you
need to specify a list of strings (each being an argument), but the user
usually wants to enter the list within a single minibuffer.  So from the
string returned by the read-string (or equivalent) function, we need to
get a list of strings.


        Stefan

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Wed, 22 Oct 2008 21:18:20 -0400
> 
> >> >> I do not understand: what makes you think it could be a properly quoted
> >> >> shell command?
> >> > Because you said that working with shell commands was why these
> >> > functions were invented in the first place.  And that is how they are
> >> > used in Emacs as of now.
> >> No, they're used to run commands.  Without going through a shell.
> > Then why would one need quoting?
> 
> Because when you run a command with start-process or call-process, you
> need to specify a list of strings (each being an argument), but the user
> usually wants to enter the list within a single minibuffer.  So from the
> string returned by the read-string (or equivalent) function, we need to
> get a list of strings.

If I am prompted for a command in the minibuffer, as in "M-x compile",
I will type a command that uses shell quoting (assuming I need
quoting).  That is the natural thing to do, so I expect most users to
do the same.  I don't expect many of them to use Lisp string quoting
for commands.

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Miles Bader]

Eli Zaretskii <eliz@gnu.org> writes:
>> Because when you run a command with start-process or call-process, you
>> need to specify a list of strings (each being an argument), but the user
>> usually wants to enter the list within a single minibuffer.  So from the
>> string returned by the read-string (or equivalent) function, we need to
>> get a list of strings.
>
> If I am prompted for a command in the minibuffer, as in "M-x compile",
> I will type a command that uses shell quoting (assuming I need
> quoting).  That is the natural thing to do, so I expect most users to
> do the same.  I don't expect many of them to use Lisp string quoting
> for commands.

Yup, I agree... it seems likely to cause confusion...

-Miles

-- 
Politics, n. A strife of interests masquerading as a contest of
principles. The conduct of public affairs for private advantage.

Re: bug#1212: 23.0.60; split-string-and-unquote problems

[Stefan Monnier]

>> >> >> I do not understand: what makes you think it could be a properly quoted
>> >> >> shell command?
>> >> > Because you said that working with shell commands was why these
>> >> > functions were invented in the first place.  And that is how they are
>> >> > used in Emacs as of now.
>> >> No, they're used to run commands.  Without going through a shell.
>> > Then why would one need quoting?
>> 
>> Because when you run a command with start-process or call-process, you
>> need to specify a list of strings (each being an argument), but the user
>> usually wants to enter the list within a single minibuffer.  So from the
>> string returned by the read-string (or equivalent) function, we need to
>> get a list of strings.

> If I am prompted for a command in the minibuffer, as in "M-x compile",
> I will type a command that uses shell quoting (assuming I need
> quoting).  That is the natural thing to do, so I expect most users to
> do the same.  I don't expect many of them to use Lisp string quoting
> for commands.

Indeed, `compile' uses a shell and hence obeys shell quoting.


        Stefan