Question about start-process and argument list

Question about start-process and argument list

[Eric Abrahamsen]

Hi!

I recently got a bug report about the Mairix search engine for Gnus --
long story short, the gnus-search.el code calls all command-line mail
indexers (notmuch, mairix, namazu, etc) like this:

(apply #'start-process (format "search-%s" server)
			buffer program cp-list)

Where `program' is the indexer, and `cp-list' is a list of command-line
flags. The important thing is that the last element of `cp-list' is the
search-query string itself, but this string has not been split on
spaces. Meaning that cp-list could look like:

'("--rcfile" config-file "-r" "another-arg" "from:bob subject:lunch")

Other search engines seem to accept this fine, but Mairix fails unless
the search query is also split on spaces:

'("--rcfile" config-file "-r" "another-arg" "from:bob" "subject:lunch")

It occurred to me that this is probably the way it should be done, even
if the other engines don't happen to be choking on the format. I have
two questions:

1. What does the program actually see, in this case? Is the first
   example above, is it the equivalent of (on the command line) wrapping
   the query in double quotes?
2. Should I be using `start-process-shell-command' instead? The docs
   mention that the main difference is that it will use shell features,
   which aren't really relevant here, but maybe it's the right thing to
   do, semantically.

Any comments very welcome,
Eric

Re: Question about start-process and argument list

[Jean Louis]

* Eric Abrahamsen <eric@ericabrahamsen.net> [2023-02-03 09:37]:
> I recently got a bug report about the Mairix search engine for Gnus --
> long story short, the gnus-search.el code calls all command-line mail
> indexers (notmuch, mairix, namazu, etc) like this:
> 
> (apply #'start-process (format "search-%s" server)
> 			buffer program cp-list)
> 
> Where `program' is the indexer, and `cp-list' is a list of command-line
> flags. The important thing is that the last element of `cp-list' is the
> search-query string itself, but this string has not been split on
> spaces. Meaning that cp-list could look like:

I would not try to generalize the command for all search engines
unless they really have compatible command lie.

> '("--rcfile" config-file "-r" "another-arg" "from:bob subject:lunch")

Which is OK in itself, as it is safer to run program without using
external shell.

> Other search engines seem to accept this fine, but Mairix fails unless
> the search query is also split on spaces:
> 
> '("--rcfile" config-file "-r" "another-arg" "from:bob" "subject:lunch")
> 
> It occurred to me that this is probably the way it should be done, even
> if the other engines don't happen to be choking on the format. I have
> two questions:
> 
> 1. What does the program actually see, in this case? Is the first
>    example above, is it the equivalent of (on the command line) wrapping
>    the query in double quotes?

It has same effect.

> 2. Should I be using `start-process-shell-command' instead? The docs
>    mention that the main difference is that it will use shell features,
>    which aren't really relevant here, but maybe it's the right thing to
>    do, semantically.

Rather no, as that becomes unsafe. Yes, it would invoke shell and
some environment and the program line. But for programing purposes is
unsafe. 

I learned it from experience ahat `start-process' and `call-process'
are safe way and more conclusive for programming purposes.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Question about start-process and argument list

[Bruno Barbier]

Jean Louis <bugs@gnu.support> writes:

> * Eric Abrahamsen <eric@ericabrahamsen.net> [2023-02-03 09:37]:
>> 
>> 1. What does the program actually see, in this case? Is the first
>>    example above, is it the equivalent of (on the command line) wrapping
>>    the query in double quotes?
>
> It has same effect.

Maybe I didn't understand the question, but I don't think it has the
same effect nor that it should.

Here is what I'm seeing on my POSIX system:

  ("--rcfile" config-file "-r" "another-arg" "from:bob subject:lunch")
   1: '--rcfile'
   2: 'config-file'
   3: '-r'
   4: 'another-arg'
   5: 'from:bob subject:lunch'

  ("--rcfile" config-file "-r" "another-arg" "from:bob" "subject:lunch")
   1: '--rcfile'
   2: 'config-file'
   3: '-r'
   4: 'another-arg'
   5: 'from:bob'
   6: 'subject:lunch'

   showargs --rcfile config-file -r  another-arg "from:bob subject:lunch"
   1: '--rcfile'
   2: 'config-file'
   3: '-r'
   4: 'another-arg'
   5: 'from:bob subject:lunch'
   
   showargs --rcfile config-file -r  another-arg from:bob subject:lunch
   1: '--rcfile'
   2: 'config-file'
   3: '-r'
   4: 'another-arg'
   5: 'from:bob'
   6: 'subject:lunch'


>> 2. Should I be using `start-process-shell-command' instead? The docs
>>    mention that the main difference is that it will use shell features,
>>    which aren't really relevant here, but maybe it's the right thing to
>>    do, semantically.
>
> Rather no, as that becomes unsafe. Yes, it would invoke shell and
> some environment and the program line. But for programing purposes is
> unsafe. 

Right, I would advise you, like Jean, to stay with `start-process'.

And, the search query should probably be only one argument (except if
you accept to restrict you query syntax to something that's compatible
with any possible way to quote and parse arguments on a command line on
any possible OS, including sign like '-', '"', ')',  etc.).

Just my 2c.

Bruno


> -- 
> Jean
>
> Take action in Free Software Foundation campaigns:
> https://www.fsf.org/campaigns
>
> In support of Richard M. Stallman
> https://stallmansupport.org/

Re: Question about start-process and argument list

[Eric Abrahamsen]

Bruno Barbier <brubar.cs@gmail.com> writes:

> Jean Louis <bugs@gnu.support> writes:

Thanks to both of you for the responses.

>> * Eric Abrahamsen <eric@ericabrahamsen.net> [2023-02-03 09:37]:
>>> 
>>> 1. What does the program actually see, in this case? Is the first
>>>    example above, is it the equivalent of (on the command line) wrapping
>>>    the query in double quotes?
>>
>> It has same effect.
>
> Maybe I didn't understand the question, but I don't think it has the
> same effect nor that it should.

[...]

Right, and experience demonstrates that it doesn't have the same effect,
because one version works (with mairix) and the other doesn't.

>>> 2. Should I be using `start-process-shell-command' instead? The docs
>>>    mention that the main difference is that it will use shell features,
>>>    which aren't really relevant here, but maybe it's the right thing to
>>>    do, semantically.
>>
>> Rather no, as that becomes unsafe. Yes, it would invoke shell and
>> some environment and the program line. But for programing purposes is
>> unsafe. 
>
> Right, I would advise you, like Jean, to stay with `start-process'.

Thanks, I will do that.

> And, the search query should probably be only one argument (except if
> you accept to restrict you query syntax to something that's compatible
> with any possible way to quote and parse arguments on a command line on
> any possible OS, including sign like '-', '"', ')',  etc.).

Well, as noted above, it doesn't work when it's all one argument. At
least with one of the possible commands -- I can "fix it" for mairix in
particular, my main question is if I should be "fixing it" for all
search commands.

Not to be contrarian, but this:

> compatible with any possible way to quote and parse arguments on a
> command line on any possible OS

Kind of sounds like I should be using `start-process-shell-command'
again! As I understand it, this is its main selling point.

Re: Question about start-process and argument list

[chad]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

On Fri, Feb 3, 2023 at 4:23 PM Eric Abrahamsen <eric@ericabrahamsen.net>
wrote:

>
> Kind of sounds like I should be using `start-process-shell-command'
> again! As I understand it, this is its main selling point.
>

Have you by chance looked at shell-quote-argument or
split-string-shell-command?  It's been a while since I had to deal with
this, but I recall the former being helpful then, and it looks like it may
be meant to handle your situation.

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 859 bytes --]

Re: Question about start-process and argument list

[Eric Abrahamsen]

chad <yandros@gmail.com> writes:

> On Fri, Feb 3, 2023 at 4:23 PM Eric Abrahamsen <eric@ericabrahamsen.net>
> wrote:
>
>>
>> Kind of sounds like I should be using `start-process-shell-command'
>> again! As I understand it, this is its main selling point.
>>
>
> Have you by chance looked at shell-quote-argument or
> split-string-shell-command?  It's been a while since I had to deal with
> this, but I recall the former being helpful then, and it looks like it may
> be meant to handle your situation.

But, to be clear, my problem doesn't seem to be escaping/unescaping
shell-specific characters, as far as I can see that isn't causing the
trouble. It's passing in the search query as a single argument, when it
contains spaces. It works fine when it's a single string with no spaces
in it.

Re: Question about start-process and argument list

[Bruno Barbier]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> chad <yandros@gmail.com> writes:
>
>> On Fri, Feb 3, 2023 at 4:23 PM Eric Abrahamsen <eric@ericabrahamsen.net>
>> wrote:
>>
>>>
>>> Kind of sounds like I should be using `start-process-shell-command'
>>> again! As I understand it, this is its main selling point.
>>>
>>
>> Have you by chance looked at shell-quote-argument or
>> split-string-shell-command?  It's been a while since I had to deal with
>> this, but I recall the former being helpful then, and it looks like it may
>> be meant to handle your situation.
>
> But, to be clear, my problem doesn't seem to be escaping/unescaping
> shell-specific characters, as far as I can see that isn't causing the
> trouble. It's passing in the search query as a single argument, when it
> contains spaces. It works fine when it's a single string with no spaces
> in it.

But, if you go the `start-process-shell-command' way, shell escaping is
going to be *your* problem (for any possible meaning of shell).

And, in this case, you should really consider using
`shell-quote-argument'.


Bruno

Re: Question about start-process and argument list

[Eric Abrahamsen]

Bruno Barbier <brubar.cs@gmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> chad <yandros@gmail.com> writes:
>>
>>> On Fri, Feb 3, 2023 at 4:23 PM Eric Abrahamsen <eric@ericabrahamsen.net>
>>> wrote:
>>>
>>>>
>>>> Kind of sounds like I should be using `start-process-shell-command'
>>>> again! As I understand it, this is its main selling point.
>>>>
>>>
>>> Have you by chance looked at shell-quote-argument or
>>> split-string-shell-command?  It's been a while since I had to deal with
>>> this, but I recall the former being helpful then, and it looks like it may
>>> be meant to handle your situation.
>>
>> But, to be clear, my problem doesn't seem to be escaping/unescaping
>> shell-specific characters, as far as I can see that isn't causing the
>> trouble. It's passing in the search query as a single argument, when it
>> contains spaces. It works fine when it's a single string with no spaces
>> in it.
>
> But, if you go the `start-process-shell-command' way, shell escaping is
> going to be *your* problem (for any possible meaning of shell).
>
> And, in this case, you should really consider using
> `shell-quote-argument'.

Okay, thanks. Sounds like some testing is in order!

Re: Question about start-process and argument list

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

On Sat, Feb 04, 2023 at 12:02:35AM +0100, Bruno Barbier wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
> 
> > chad <yandros@gmail.com> writes:
> >
> >> On Fri, Feb 3, 2023 at 4:23 PM Eric Abrahamsen <eric@ericabrahamsen.net>
> >> wrote:
> >>
> >>>
> >>> Kind of sounds like I should be using `start-process-shell-command'
> >>> again! As I understand it, this is its main selling point.
> >>>
> >>
> >> Have you by chance looked at shell-quote-argument or
> >> split-string-shell-command?  It's been a while since I had to deal with
> >> this, but I recall the former being helpful then, and it looks like it may
> >> be meant to handle your situation.
> >
> > But, to be clear, my problem doesn't seem to be escaping/unescaping
> > shell-specific characters, as far as I can see that isn't causing the
> > trouble. It's passing in the search query as a single argument, when it
> > contains spaces. It works fine when it's a single string with no spaces
> > in it.
> 
> But, if you go the `start-process-shell-command' way, shell escaping is
> going to be *your* problem (for any possible meaning of shell).
> 
> And, in this case, you should really consider using
> `shell-quote-argument'.

It /does/ help with whitespaces:

  (shell-quote-argument "foo bar baz")
  => "foo\\ bar\\ baz"

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Question about start-process and argument list

[Jean Louis]

Now I think that reason why one program accepts separate arguments for
query, and other needs single argument, is the program
design. 

Programmer decided to accept any follow up arguments, that is why you
can split it and feed it.

While in other program only single argument is accepted.

That is why it is not good generalizing function for different
programs.

It is maybe better to generalize one function, but have it branch to
different other functions for different programs.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Question about start-process and argument list

[Jean Louis]

> > compatible with any possible way to quote and parse arguments on a
> > command line on any possible OS

I do not consider above statement logically related to your statement
below. Maybe I have taken it out of context, but I consider that above
statement may apply to `start-process' but not to
`start-process-shell-command' exactly to avoid problems with quoting.

> Kind of sounds like I should be using `start-process-shell-command'
> again! As I understand it, this is its main selling point.

That one has problems with quoting!

Use `start-process' or `call-process'

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Question about start-process and argument list

[Eric Abrahamsen]

Jean Louis <bugs@gnu.support> writes:

> Now I think that reason why one program accepts separate arguments for
> query, and other needs single argument, is the program
> design. 
>
> Programmer decided to accept any follow up arguments, that is why you
> can split it and feed it.
>
> While in other program only single argument is accepted.
>
> That is why it is not good generalizing function for different
> programs.
>
> It is maybe better to generalize one function, but have it branch to
> different other functions for different programs.

Yup, that's how it works.

Re: Question about start-process and argument list

[Eric Abrahamsen]

Jean Louis <bugs@gnu.support> writes:

>> > compatible with any possible way to quote and parse arguments on a
>> > command line on any possible OS
>
> I do not consider above statement logically related to your statement
> below. Maybe I have taken it out of context, but I consider that above
> statement may apply to `start-process' but not to
> `start-process-shell-command' exactly to avoid problems with quoting.
>
>> Kind of sounds like I should be using `start-process-shell-command'
>> again! As I understand it, this is its main selling point.
>
> That one has problems with quoting!
>
> Use `start-process' or `call-process'

Okay I finally understood why I was confused about this. These programs
essentially don't interact with the shell at all. File globbing is
irrelevant, as is the brace notation, I think even $PWD doesn't matter.
So there's no advantage to running in a shell, and I should treat it as
much like a system call as possible.

Thanks to all for helping me get my brain on front-to-back...