[PATCH] Add shell-quasiquote.

[PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

[-- Attachment #1: Type: text/plain, Size: 19 bytes --]

This is for ELPA.


[-- Attachment #2: 0001-Add-shell-quasiquote.patch --]
[-- Type: text/x-diff, Size: 6151 bytes --]

From acd5cfc5fc57a07cccf233021c23db53cfa24ca7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 <taylanbayirli@gmail.com>
Date: Sat, 17 Oct 2015 18:32:22 +0200
Subject: [PATCH] Add shell-quasiquote.

---
 packages/shell-quasiquote/shell-quasiquote.el | 149 ++++++++++++++++++++++++++
 1 file changed, 149 insertions(+)
 create mode 100644 packages/shell-quasiquote/shell-quasiquote.el

diff --git a/packages/shell-quasiquote/shell-quasiquote.el b/packages/shell-quasiquote/shell-quasiquote.el
new file mode 100644
index 0000000..deb58c2
--- /dev/null
+++ b/packages/shell-quasiquote/shell-quasiquote.el
@@ -0,0 +1,149 @@
+;;; shell-quasiquote.el --- Turn s-expressions into shell command strings.
+
+;; Copyright (C) 2015  Free Software Foundation, Inc.
+
+;; Author: Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
+;; Keywords: extensions, unix
+;; URL: https://github.com/TaylanUB/emacs-shell-quasiquote
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; "Shell quasiquote" -- turn s-expressions into shell command strings.
+;;
+;; Quoting is automatic for POSIX shells.
+;;
+;;   (let ((file1 "file one")
+;;         (file2 "file two"))
+;;     (shqq (cp -r ,file1 ,file2 "My Files")))
+;;       => "cp -r 'file one' 'file two' 'My Files'"
+;;
+;; You can splice many arguments into place with ,@foo.
+;;
+;;   (let ((files (list "file one" "file two")))
+;;     (shqq (cp -r ,@files "My Files")))
+;;       => "cp -r 'file one' 'file two' 'My Files'"
+;;
+;; Note that the quoting disables a variety of shell expansions like ~/foo,
+;; $ENV_VAR, and e.g. {x..y} in GNU Bash.
+;;
+;; You can use ,,foo to escape the quoting.
+;;
+;;   (let ((files "file1 file2"))
+;;     (shqq (cp -r ,,files "My Files")))
+;;       => "cp -r file1 file2 'My Files'"
+;;
+;; And ,,@foo to splice and escape quoting.
+;;
+;;   (let* ((arglist '("-x 'foo bar' -y baz"))
+;;          (arglist (append arglist '("-z 'qux fux'"))))
+;;     (shqq (command ,,@arglist)))
+;;       => "command -x 'foo bar' -y baz -z 'qux fux'"
+;;
+;; Neat, eh?
+
+\f
+;;; Code:
+
+;;; Like `shell-quote-argument', but much simpler in implementation.
+(defun shqq--quote-string (string)
+  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))
+
+(defun shqq--atom-to-string (atom)
+  (cond
+   ((symbolp atom) (symbol-name atom))
+   ((stringp atom) atom)
+   ((numberp atom) (number-to-string atom))
+   (t (error "Bad shqq atom: %S" atom))))
+
+(defun shqq--quote-atom (atom)
+  (shqq--quote-string (shqq--atom-to-string atom)))
+
+(defun shqq--match-comma (form)
+  "Matches FORM against ,foo i.e. (\, foo) and returns foo.
+Returns nil if FORM didn't match.  You can't disambiguate between
+FORM matching ,nil and not matching."
+  (if (and (consp form)
+           (eq '\, (car form))
+           (consp (cdr form))
+           (null (cddr form)))
+      (cadr form)))
+
+(defun shqq--match-comma2 (form)
+  "Matches FORM against ,,foo i.e. (\, (\, foo)) and returns foo.
+Returns nil if FORM didn't match.  You can't disambiguate between
+FORM matching ,,nil and not matching."
+  (if (and (consp form)
+           (eq '\, (car form))
+           (consp (cdr form))
+           (null (cddr form)))
+      (shqq--match-comma (cadr form))))
+
+\f
+(defmacro shqq (parts)
+  "First, PARTS is turned into a list of strings.  For this,
+every element of PARTS must be one of:
+
+- a symbol, evaluating to its name,
+
+- a string, evaluating to itself,
+
+- a number, evaluating to its decimal representation,
+
+- \",expr\", where EXPR must evaluate to an atom that will be
+  interpreted according to the previous rules,
+
+- \",@list-expr\", where LIST-EXPR must evaluate to a list whose
+  elements will each be interpreted like the EXPR in an \",EXPR\"
+  form, and spliced into the list of strings,
+
+- \",,expr\", where EXPR is interpreted like in \",expr\",
+
+- or \",,@expr\", where EXPR is interpreted like in \",@expr\".
+
+In the resulting list of strings, all elements except the ones
+resulting from \",,expr\" and \",,@expr\" forms are quoted for
+shell grammar.
+
+Finally, the resulting list of strings is concatenated with
+separating spaces."
+  (let ((parts
+         (mapcar
+          (lambda (part)
+            (cond
+             ((atom part) (shqq--quote-atom part))
+             ;; We use the match-comma helpers because pcase can't match ,foo.
+             (t (pcase part
+                  ;; ,,foo i.e. (, (, foo))
+                  ((pred shqq--match-comma2)
+                   (shqq--match-comma2 part))
+                  ;; ,,@foo i.e. (, (,@ foo))
+                  ((and (pred shqq--match-comma)
+                        (let `,@,form (shqq--match-comma part)))
+                   `(mapconcat #'identity ,form " "))
+                  ;; ,foo
+                  ;; Insert redundant 'and x' to work around debbugs#18554.
+                  ((and x (pred shqq--match-comma))
+                   `(shqq--quote-atom ,(shqq--match-comma part)))
+                  ;; ,@foo
+                  (`,@,form
+                   `(mapconcat #'shqq--quote-atom ,form " "))
+                  (_
+                   (error "Bad shqq part: %S" part))))))
+          parts)))
+    `(mapconcat #'identity (list ,@parts) " ")))
+
+(provide 'shell-quasiquote)
+;;; shell-quasiquote.el ends here
-- 
2.5.0

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Date: Sat, 17 Oct 2015 18:33:01 +0200
> 
> >From acd5cfc5fc57a07cccf233021c23db53cfa24ca7 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
>  <taylanbayirli@gmail.com>
> Date: Sat, 17 Oct 2015 18:32:22 +0200
> Subject: [PATCH] Add shell-quasiquote.

Thanks.

> +;;; Like `shell-quote-argument', but much simpler in implementation.
> +(defun shqq--quote-string (string)
> +  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))

It might be simpler, but it's wrong, because the result is only
correct for Posix shells.

Please do use shell-quote-argument instead.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> +;;; Like `shell-quote-argument', but much simpler in implementation.
>> +(defun shqq--quote-string (string)
>> +  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))
>
> It might be simpler, but it's wrong, because the result is only
> correct for Posix shells.
>
> Please do use shell-quote-argument instead.

Hmm, I don't really want to take responsibility of my library being used
with shells other than POSIX shells.  (The library could make that
clearer and error on other systems.)

How much can I rely on shell-quote-argument?  Can one fully rely on it
being safe against code injection?

Taylan

Re: [PATCH] Add shell-quasiquote.

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 319 bytes --]

+  (if (and (consp form)
+           (eq '\, (car form))
+           (consp (cdr form))
+           (null (cddr form)))
+      (cadr form)))

There's nothing wrong with this code. But, just FYI, here's what it would
look like with pcase (assuming I didn't mess up writing this on my phone).

(pcase form
  ((`\, x) x))

[-- Attachment #2: Type: text/html, Size: 454 bytes --]

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: emacs-devel@gnu.org
> Date: Sat, 17 Oct 2015 19:14:16 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> +;;; Like `shell-quote-argument', but much simpler in implementation.
> >> +(defun shqq--quote-string (string)
> >> +  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))
> >
> > It might be simpler, but it's wrong, because the result is only
> > correct for Posix shells.
> >
> > Please do use shell-quote-argument instead.
> 
> Hmm, I don't really want to take responsibility of my library being used
> with shells other than POSIX shells.  (The library could make that
> clearer and error on other systems.)

I don't think we'd like to have packages limited in that way.  AFAIK,
we didn't until now, at least not consciously.

And it really isn't a big deal.  Emacs already has all the
infrastructure for portable handling of shell commands.

> How much can I rely on shell-quote-argument?

You can rely on it.  Emacs uses it in umpteen important places.

> Can one fully rely on it being safe against code injection?

I don't think I understand what code injection you had in mind.
Please elaborate.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> + (if (and (consp form) 
> + (eq '\, (car form)) 
> + (consp (cdr form)) 
> + (null (cddr form))) 
> + (cadr form)))
>
> There's nothing wrong with this code. But, just FYI, here's what it
> would look like with pcase (assuming I didn't mess up writing this on
> my phone).
>
> (pcase form
> ((`\, x) x))

As pointed out somewhere else in the code:

    ;; We use the match-comma helpers because pcase can't match ,foo.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 17 Oct 2015 19:14:16 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> +;;; Like `shell-quote-argument', but much simpler in implementation.
>> >> +(defun shqq--quote-string (string)
>> >> +  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))
>> >
>> > It might be simpler, but it's wrong, because the result is only
>> > correct for Posix shells.
>> >
>> > Please do use shell-quote-argument instead.
>> 
>> Hmm, I don't really want to take responsibility of my library being used
>> with shells other than POSIX shells.  (The library could make that
>> clearer and error on other systems.)
>
> I don't think we'd like to have packages limited in that way.  AFAIK,
> we didn't until now, at least not consciously.

Quoting RMS, coincidentally from a couple days ago:

    The policy is non-GNU systems are secondary, and lower priority than
    the GNU system, but we are glad to include support for them in GNU
    packages if users contribute the necessary code -- provided that
    code isn't a maintenance problem for us.

    The maintenainers of any particular package are the ones who judge
    whether that code is a maintenance problem, since they are the ones
    it would be a problem for.

(That mentality made sense to me even before I learned it's GNU policy.)

I generally don't want to take responsibility of my code being used on
non-GNU/non-POSIX systems, but if I can share the responsibility then
that's fine.

> And it really isn't a big deal.  Emacs already has all the
> infrastructure for portable handling of shell commands.
>
>> How much can I rely on shell-quote-argument?
>
> You can rely on it.  Emacs uses it in umpteen important places.
>
>> Can one fully rely on it being safe against code injection?
>
> I don't think I understand what code injection you had in mind.
> Please elaborate.

(let ((file-list (read where-ever)))
  (shqq (cp -- ,@file-list some-place)))

That code is *guaranteed* to either copy the files in file-list to
some-place, or error, so long as the argument quoting by shqq works
well.  If it has a bug, then malicious input from where-ever may be able
to execute arbitrary shell commands.

Is shell-quote-argument safe against such a thing?  My shqq-quote-string
isn't exactly formally proven to be safe either, but its implementation
is so simple it's fairly obvious that it doesn't contain bugs.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 254 bytes --]

On 17 Oct 2015 7:11 pm, "Taylan Ulrich Bayırlı/Kammer" <
taylanbayirli@gmail.com> wrote:
>
> As pointed out somewhere else in the code:
>
>     ;; We use the match-comma helpers because pcase can't match ,foo.

Clearly, I was reading too fast.

[-- Attachment #2: Type: text/html, Size: 392 bytes --]

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: emacs-devel@gnu.org
> Date: Sat, 17 Oct 2015 20:23:17 +0200
> 
> > I don't think we'd like to have packages limited in that way.  AFAIK,
> > we didn't until now, at least not consciously.
> 
> Quoting RMS, coincidentally from a couple days ago:
> 
>     The policy is non-GNU systems are secondary, and lower priority than
>     the GNU system, but we are glad to include support for them in GNU
>     packages if users contribute the necessary code -- provided that
>     code isn't a maintenance problem for us.
> 
>     The maintenainers of any particular package are the ones who judge
>     whether that code is a maintenance problem, since they are the ones
>     it would be a problem for.

I don't see how this is relevant for the issue at hand, since the
necessary code (the shell-quote-argument function) was already
contributed to Emacs years ago, and is used in many places in core
Emacs.  There's no extra effort needed to support more platforms, just
replace one function with another.

> I generally don't want to take responsibility of my code being used on
> non-GNU/non-POSIX systems, but if I can share the responsibility then
> that's fine.

You are sharing the responsibility with a long line of Emacs
developers, all of whom use this function.  I don't see anything you
should worry about, really.

> > And it really isn't a big deal.  Emacs already has all the
> > infrastructure for portable handling of shell commands.
> >
> >> How much can I rely on shell-quote-argument?
> >
> > You can rely on it.  Emacs uses it in umpteen important places.
> >
> >> Can one fully rely on it being safe against code injection?
> >
> > I don't think I understand what code injection you had in mind.
> > Please elaborate.
> 
> (let ((file-list (read where-ever)))
>   (shqq (cp -- ,@file-list some-place)))
> 
> That code is *guaranteed* to either copy the files in file-list to
> some-place, or error, so long as the argument quoting by shqq works
> well.  If it has a bug, then malicious input from where-ever may be able
> to execute arbitrary shell commands.
> 
> Is shell-quote-argument safe against such a thing?  My shqq-quote-string
> isn't exactly formally proven to be safe either, but its implementation
> is so simple it's fairly obvious that it doesn't contain bugs.

Please take a look at the implementation of shell-quote-argument.  It
uses the same interfaces as your implementation, no more, no less.  If
your implementation is safe, then so is shell-quote-argument.

Re: [PATCH] Add shell-quasiquote.

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> +;;; Like `shell-quote-argument', but much simpler in implementation.
>> +(defun shqq--quote-string (string)
>> +  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))
>
> It might be simpler, but it's wrong, because the result is only
> correct for Posix shells.
>
> Please do use shell-quote-argument instead.

It's also simpler than the POSIX section of shell-quote-argument.

For reference:

(defun shell-quote-argument (argument)
  [...] (cond [...] (t
    (if (equal argument "")
        "''"
      ;; Quote everything except POSIX filename characters.
      ;; This should be safe enough even for really weird shells.
      (replace-regexp-in-string
       "\n" "'\n'"
       (replace-regexp-in-string "[^-0-9a-zA-Z_./\n]" "\\\\\\&" argument))))))

I wonder what "really weird shells" this refers to? Certainly not csh,
the mechanism it uses for newlines doesn't work there.

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Sat, 17 Oct 2015 15:14:26 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> >> +;;; Like `shell-quote-argument', but much simpler in implementation.
> >> +(defun shqq--quote-string (string)
> >> +  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))
> >
> > It might be simpler, but it's wrong, because the result is only
> > correct for Posix shells.
> >
> > Please do use shell-quote-argument instead.
> 
> It's also simpler than the POSIX section of shell-quote-argument.

Simpler doesn't mean correct.

> (defun shell-quote-argument (argument)
>   [...] (cond [...] (t
>     (if (equal argument "")
>         "''"
>       ;; Quote everything except POSIX filename characters.
>       ;; This should be safe enough even for really weird shells.
>       (replace-regexp-in-string
>        "\n" "'\n'"
>        (replace-regexp-in-string "[^-0-9a-zA-Z_./\n]" "\\\\\\&" argument))))))
> 
> I wonder what "really weird shells" this refers to?

The set of characters special to an arbitrary shell is not known in
advance.

> Certainly not csh, the mechanism it uses for newlines doesn't work
> there.

What did you try that didn't work with csh?

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Cc: emacs-devel@gnu.org
>> Date: Sat, 17 Oct 2015 20:23:17 +0200
>> 
>> > I don't think we'd like to have packages limited in that way.  AFAIK,
>> > we didn't until now, at least not consciously.
>> 
>> Quoting RMS, coincidentally from a couple days ago:
>> 
>>     The policy is non-GNU systems are secondary, and lower priority than
>>     the GNU system, but we are glad to include support for them in GNU
>>     packages if users contribute the necessary code -- provided that
>>     code isn't a maintenance problem for us.
>> 
>>     The maintenainers of any particular package are the ones who judge
>>     whether that code is a maintenance problem, since they are the ones
>>     it would be a problem for.
>
> I don't see how this is relevant for the issue at hand, since the
> necessary code (the shell-quote-argument function) was already
> contributed to Emacs years ago, and is used in many places in core
> Emacs.  There's no extra effort needed to support more platforms, just
> replace one function with another.

You seem to be implying that using shell-quote-argument will uphold the
invariant that the code is safe against injection.  I'm asking for
explicit confirmation of that.  Once I have confirmation of that, sure,
I will use it and thus make my code portable.  As it stands, I don't
know whether doing that change would really make my code portable with
the same safety guarantees, or weaken the safety guarantees.

>> I generally don't want to take responsibility of my code being used on
>> non-GNU/non-POSIX systems, but if I can share the responsibility then
>> that's fine.
>
> You are sharing the responsibility with a long line of Emacs
> developers, all of whom use this function.  I don't see anything you
> should worry about, really.

I can't have responsibility over every single Elisp function in Emacs,
as no developer can.  In particular I *can't* take responsibility over
shell-quote-argument because I don't know any shell syntax other than
POSIX.

And I surely do worry whether users of my library will be subject to
arbitrary code injection.

>> > And it really isn't a big deal.  Emacs already has all the
>> > infrastructure for portable handling of shell commands.
>> >
>> >> How much can I rely on shell-quote-argument?
>> >
>> > You can rely on it.  Emacs uses it in umpteen important places.
>> >
>> >> Can one fully rely on it being safe against code injection?
>> >
>> > I don't think I understand what code injection you had in mind.
>> > Please elaborate.
>> 
>> (let ((file-list (read where-ever)))
>>   (shqq (cp -- ,@file-list some-place)))
>> 
>> That code is *guaranteed* to either copy the files in file-list to
>> some-place, or error, so long as the argument quoting by shqq works
>> well.  If it has a bug, then malicious input from where-ever may be able
>> to execute arbitrary shell commands.
>> 
>> Is shell-quote-argument safe against such a thing?  My shqq-quote-string
>> isn't exactly formally proven to be safe either, but its implementation
>> is so simple it's fairly obvious that it doesn't contain bugs.
>
> Please take a look at the implementation of shell-quote-argument.  It
> uses the same interfaces as your implementation, no more, no less.  If
> your implementation is safe, then so is shell-quote-argument.

I have taken a look.  It doesn't use the same strategy even for POSIX
shells, and I can't be as sure of its safety as I am of the safety of my
implementation.  When it comes to non-POSIX shells, I have no clue.

If someone explicitly confirms to me that the function is very obviously
safe against injection on all shells it supports, then I will use it.
So far, seeing things like

      ;; This should be safe enough even for really weird shells.

and the implementation complexity for the ms-dos and windows-nt variants
(though as I said I have no clue about those) doesn't exactly inspire
confidence.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Random832 <random832@fastmail.com>
>> Date: Sat, 17 Oct 2015 15:14:26 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> >> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> >> +;;; Like `shell-quote-argument', but much simpler in implementation.
>> >> +(defun shqq--quote-string (string)
>> >> +  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))
>> >
>> > It might be simpler, but it's wrong, because the result is only
>> > correct for Posix shells.
>> >
>> > Please do use shell-quote-argument instead.
>> 
>> It's also simpler than the POSIX section of shell-quote-argument.
>
> Simpler doesn't mean correct.

It doesn't, but anyone who knows POSIX shell grammar well should know
that wrapping a string in '', and escaping literal "'"s by turning them
into "'\''" (close so-far single quote, add backslash-escaped single
quote, reopen single quote) is absolutely safe, because absolutely no
character within two single quotes has a special meaning (one cannot
even escape a single quote within two single quotes, hence the
close/insert/reopen).

>> (defun shell-quote-argument (argument)
>>   [...] (cond [...] (t
>>     (if (equal argument "")
>>         "''"
>>       ;; Quote everything except POSIX filename characters.
>>       ;; This should be safe enough even for really weird shells.
>>       (replace-regexp-in-string
>>        "\n" "'\n'"
>>        (replace-regexp-in-string "[^-0-9a-zA-Z_./\n]" "\\\\\\&" argument))))))
>> 
>> I wonder what "really weird shells" this refers to?
>
> The set of characters special to an arbitrary shell is not known in
> advance.

Yet that piece of code assumes a certain semantics for foo\<newline>bar
(which is that the newline after the backslash is removed instead of
interpreted literally) and special-handles that case, but doesn't
attempt to handle other possible special meanings.

Of course, I'm being pedantic here.  This implementation is probably as
safe as mine.  I'm more worried about the non-POSIX variants because I
have no clue of any non-POSIX shell syntax.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Dmitry Gutov]

On 10/17/2015 11:28 PM, Taylan Ulrich Bayırlı/Kammer wrote:

> I have taken a look.  It doesn't use the same strategy even for POSIX
> shells, and I can't be as sure of its safety as I am of the safety of my
> implementation.  When it comes to non-POSIX shells, I have no clue.

If you know of a real problem scenario reproducible with 
shell-quote-argument, please file a bug. Then we'll fix it.

Either way, please avoid reinventing the wheel.

Re: [PATCH] Add shell-quasiquote.

[Paul Eggert]

Taylan Ulrich Bayırlı/Kammer wrote:
> You seem to be implying that using shell-quote-argument will uphold the
> invariant that the code is safe against injection.  I'm asking for
> explicit confirmation of that.

Yes, it's safe. In contrast, the version you proposed is not safe for really 
weird csh-like shells, where it can mishandle '!'.

Re: [PATCH] Add shell-quasiquote.

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Random832 <random832@fastmail.com>
>> It's also simpler than the POSIX section of shell-quote-argument.
>
> Simpler doesn't mean correct.

But it is correct.

>> I wonder what "really weird shells" this refers to?
>
> The set of characters special to an arbitrary shell is not known in
> advance.

And neither is its quoting mechanism. That tells me that this section of
the function is not, cannot be, and should not be in the business of
supporting "arbitrary shells". I was simply wondering what shells of the
set that it can reasonably support (e.g. POSIX shells, or at the
outside, shells that are likely to be used as "sh" on a unix-like
system) it is referring to as "really weird".

>> Certainly not csh, the mechanism it uses for newlines doesn't work
>> there.
>
> What did you try that didn't work with csh?

It's not a matter of trying anything. For all I know, maybe Emacs will
never try to execute a command with csh even if it is your SHELL...
that's certainly what I'd do. The point is I know that csh doesn't
accept this syntax, and will result in an error of "Unmatched '." But,
fine, if you want a test case:

(call-process "csh" nil t "csh" "-c"
              (concat "echo " (shell-quote-argument "hello\nworld")))
Unmatched '.
Unmatched '.
1

Re: [PATCH] Add shell-quasiquote.

[Random832]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Taylan Ulrich Bayırlı/Kammer wrote:
>> You seem to be implying that using shell-quote-argument will uphold the
>> invariant that the code is safe against injection.  I'm asking for
>> explicit confirmation of that.
>
> Yes, it's safe. In contrast, the version you proposed is not safe for
> really weird csh-like shells, where it can mishandle '!'.

If supporting csh-like shells is a concern, I'll point out that the
newline mishandling I noted in another post allows one to, at least,
inject an arbitrary command with no arguments:

(call-process "csh" nil t "csh" "-c"
              (concat "echo " (shell-quote-argument
              "\nevil-command\n")))
Unmatched '.
evil-command: Command not found.
Unmatched '.
1

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/17/2015 11:28 PM, Taylan Ulrich Bayırlı/Kammer wrote:
>
>> I have taken a look.  It doesn't use the same strategy even for POSIX
>> shells, and I can't be as sure of its safety as I am of the safety of my
>> implementation.  When it comes to non-POSIX shells, I have no clue.
>
> If you know of a real problem scenario reproducible with
> shell-quote-argument, please file a bug. Then we'll fix it.

Not knowing that there are bugs is not proof that there are no bugs.

> Either way, please avoid reinventing the wheel.

It's not a reinvention because it has very strict semantics with regard
to safety guarantees, which shell-quote-argument apparently doesn't.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Taylan Ulrich Bayırlı/Kammer wrote:
>> You seem to be implying that using shell-quote-argument will uphold the
>> invariant that the code is safe against injection.  I'm asking for
>> explicit confirmation of that.
>
> Yes, it's safe. In contrast, the version you proposed is not safe for
> really weird csh-like shells, where it can mishandle '!'.

It doesn't attempt to work for any shells other than POSIX.

Not mentioning this strongly enough in the documentation is a bug; I'll
fix it.  Determining it automatically is probably impossible I'm afraid,
since even on a foreign system one might have a Unix shell installed
through Cygwin, if I'm not mistaken.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Dmitry Gutov]

On 10/18/2015 12:25 AM, Taylan Ulrich Bayırlı/Kammer wrote:

> Not knowing that there are bugs is not proof that there are no bugs.

If you can't point out a bug, you have no justification to not use the 
standard function.

>> Either way, please avoid reinventing the wheel.
>
> It's not a reinvention because it has very strict semantics with regard
> to safety guarantees, which shell-quote-argument apparently doesn't.

shell-quote-argument doesn't guarantee quoting the argument?

Re: [PATCH] Add shell-quasiquote.

[Paul Eggert]

Random832 wrote:
> If supporting csh-like shells is a concern

Emacs cannot support all weird shells in all cases, since their syntax is 
unknown. But it can support csh-like shells when '!' is used. If you can think 
of a good way to also support newline with these weird shells, please feel to 
add that. I'd make it low priority, though, as the problem is unlikely in 
ordinary use, and people worried about code injection should not be using weird 
shells anyway (obviously).

Re: [PATCH] Add shell-quasiquote.

[Paul Eggert]

Taylan Ulrich Bayırlı/Kammer wrote:
> Not mentioning this strongly enough in the documentation is a bug; I'll
> fix it.

Generally speaking it's better to fix a limitation than to document it, if 
fixing it is easy (as is the case here).

On POSIX shells, shell-quote-argument is just as safe as shqq--quote-string, and 
on non-POSIX shells it works better. So it's a win, in both readability and in 
portability, to use shell-quote-argument.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

[-- Attachment #1: Type: text/plain, Size: 1369 bytes --]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/18/2015 12:25 AM, Taylan Ulrich Bayırlı/Kammer wrote:
>
>> Not knowing that there are bugs is not proof that there are no bugs.
>
> If you can't point out a bug, you have no justification to not use the
> standard function.

No, I will *not* let users of my code potentially suffer from arbitrary
code injection attacks, thank you very much.

>>> Either way, please avoid reinventing the wheel.
>>
>> It's not a reinvention because it has very strict semantics with regard
>> to safety guarantees, which shell-quote-argument apparently doesn't.
>
> shell-quote-argument doesn't guarantee quoting the argument?

Apparently, it doesn't.  See Random832's demonstration of an injection
attack on shell-quote-argument when it's used with csh.


My function has a clearly defined domain, and operates correctly within
that domain.  Shell-quote-argument apparently does not even have a
clearly defined domain in which it's supposed to work.

If anyone has a shell argument quoting function that expands the domain
of supported shells in a well-defined manner, without weakening the
safety guarantees, please let me know.  Until then, please accept this
modified patch which clarifies that the library is not supposed to work
with any shells other than ones conforming to POSIX sh.

Taylan


[-- Attachment #2: 0001-Add-shell-quasiquote.patch --]
[-- Type: text/x-diff, Size: 6310 bytes --]

From f8bee6aeed124f0e14fbc55509597cddba439cbd Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 <taylanbayirli@gmail.com>
Date: Sat, 17 Oct 2015 18:32:22 +0200
Subject: [PATCH] Add shell-quasiquote.

---
 packages/shell-quasiquote/shell-quasiquote.el | 152 ++++++++++++++++++++++++++
 1 file changed, 152 insertions(+)
 create mode 100644 packages/shell-quasiquote/shell-quasiquote.el

diff --git a/packages/shell-quasiquote/shell-quasiquote.el b/packages/shell-quasiquote/shell-quasiquote.el
new file mode 100644
index 0000000..7f20afd
--- /dev/null
+++ b/packages/shell-quasiquote/shell-quasiquote.el
@@ -0,0 +1,152 @@
+;;; shell-quasiquote.el --- Turn s-expressions into shell command strings.
+
+;; Copyright (C) 2015  Free Software Foundation, Inc.
+
+;; Author: Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
+;; Keywords: extensions, unix
+;; URL: https://github.com/TaylanUB/emacs-shell-quasiquote
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; "Shell quasiquote" -- turn s-expressions into POSIX shell command strings.
+;;
+;; Shells other than POSIX sh are not supported and WILL make your code subject
+;; to arbitrary code injection attacks.  Otherwise...
+;;
+;; Quoting is automatic and safe against injection.
+;;
+;;   (let ((file1 "file one")
+;;         (file2 "file two"))
+;;     (shqq (cp -r ,file1 ,file2 "My Files")))
+;;       => "cp -r 'file one' 'file two' 'My Files'"
+;;
+;; You can splice many arguments into place with ,@foo.
+;;
+;;   (let ((files (list "file one" "file two")))
+;;     (shqq (cp -r ,@files "My Files")))
+;;       => "cp -r 'file one' 'file two' 'My Files'"
+;;
+;; Note that the quoting disables a variety of shell expansions like ~/foo,
+;; $ENV_VAR, and e.g. {x..y} in GNU Bash.
+;;
+;; You can use ,,foo to escape the quoting.
+;;
+;;   (let ((files "file1 file2"))
+;;     (shqq (cp -r ,,files "My Files")))
+;;       => "cp -r file1 file2 'My Files'"
+;;
+;; And ,,@foo to splice and escape quoting.
+;;
+;;   (let* ((arglist '("-x 'foo bar' -y baz"))
+;;          (arglist (append arglist '("-z 'qux fux'"))))
+;;     (shqq (command ,,@arglist)))
+;;       => "command -x 'foo bar' -y baz -z 'qux fux'"
+;;
+;; Neat, eh?
+
+\f
+;;; Code:
+
+;;; Like `shell-quote-argument', but much simpler in implementation.
+(defun shqq--quote-string (string)
+  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))
+
+(defun shqq--atom-to-string (atom)
+  (cond
+   ((symbolp atom) (symbol-name atom))
+   ((stringp atom) atom)
+   ((numberp atom) (number-to-string atom))
+   (t (error "Bad shqq atom: %S" atom))))
+
+(defun shqq--quote-atom (atom)
+  (shqq--quote-string (shqq--atom-to-string atom)))
+
+(defun shqq--match-comma (form)
+  "Matches FORM against ,foo i.e. (\, foo) and returns foo.
+Returns nil if FORM didn't match.  You can't disambiguate between
+FORM matching ,nil and not matching."
+  (if (and (consp form)
+           (eq '\, (car form))
+           (consp (cdr form))
+           (null (cddr form)))
+      (cadr form)))
+
+(defun shqq--match-comma2 (form)
+  "Matches FORM against ,,foo i.e. (\, (\, foo)) and returns foo.
+Returns nil if FORM didn't match.  You can't disambiguate between
+FORM matching ,,nil and not matching."
+  (if (and (consp form)
+           (eq '\, (car form))
+           (consp (cdr form))
+           (null (cddr form)))
+      (shqq--match-comma (cadr form))))
+
+\f
+(defmacro shqq (parts)
+  "First, PARTS is turned into a list of strings.  For this,
+every element of PARTS must be one of:
+
+- a symbol, evaluating to its name,
+
+- a string, evaluating to itself,
+
+- a number, evaluating to its decimal representation,
+
+- \",expr\", where EXPR must evaluate to an atom that will be
+  interpreted according to the previous rules,
+
+- \",@list-expr\", where LIST-EXPR must evaluate to a list whose
+  elements will each be interpreted like the EXPR in an \",EXPR\"
+  form, and spliced into the list of strings,
+
+- \",,expr\", where EXPR is interpreted like in \",expr\",
+
+- or \",,@expr\", where EXPR is interpreted like in \",@expr\".
+
+In the resulting list of strings, all elements except the ones
+resulting from \",,expr\" and \",,@expr\" forms are quoted for
+shell grammar.
+
+Finally, the resulting list of strings is concatenated with
+separating spaces."
+  (let ((parts
+         (mapcar
+          (lambda (part)
+            (cond
+             ((atom part) (shqq--quote-atom part))
+             ;; We use the match-comma helpers because pcase can't match ,foo.
+             (t (pcase part
+                  ;; ,,foo i.e. (, (, foo))
+                  ((pred shqq--match-comma2)
+                   (shqq--match-comma2 part))
+                  ;; ,,@foo i.e. (, (,@ foo))
+                  ((and (pred shqq--match-comma)
+                        (let `,@,form (shqq--match-comma part)))
+                   `(mapconcat #'identity ,form " "))
+                  ;; ,foo
+                  ;; Insert redundant 'and x' to work around debbugs#18554.
+                  ((and x (pred shqq--match-comma))
+                   `(shqq--quote-atom ,(shqq--match-comma part)))
+                  ;; ,@foo
+                  (`,@,form
+                   `(mapconcat #'shqq--quote-atom ,form " "))
+                  (_
+                   (error "Bad shqq part: %S" part))))))
+          parts)))
+    `(mapconcat #'identity (list ,@parts) " ")))
+
+(provide 'shell-quasiquote)
+;;; shell-quasiquote.el ends here
-- 
2.5.0

Re: [PATCH] Add shell-quasiquote.

[Random832]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>> If you know of a real problem scenario reproducible with
>> shell-quote-argument, please file a bug. Then we'll fix it.
>
> Not knowing that there are bugs is not proof that there are no bugs.

Why aren't you as sure of its safety, regarding the POSIX section, as you
are of the safety of your implementation?

>> Either way, please avoid reinventing the wheel.
>
> It's not a reinvention because it has very strict semantics with regard
> to safety guarantees, which shell-quote-argument apparently doesn't.

Out of curiosity, how are you guaranteeing that the result will be
executed by a POSIX shell? Passing a string quoted by your function to
MS Windows' cmd.exe (or, to that matter, to csh - even worse than the
existing version) would be an absolute disaster as far as injection
vulnerabilities go.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Taylan Ulrich Bayırlı/Kammer wrote:
>> Not mentioning this strongly enough in the documentation is a bug; I'll
>> fix it.
>
> Generally speaking it's better to fix a limitation than to document
> it, if fixing it is easy (as is the case here).
>
> On POSIX shells, shell-quote-argument is just as safe as
> shqq--quote-string, and on non-POSIX shells it works better. So it's a
> win, in both readability and in portability, to use
> shell-quote-argument.

Fixing it does not seem easy at all given I can't trust
shell-quote-argument.

Please tell me which shells shell-quote-argument is guaranteed to work
safely on, so I can document it for users of my library.  Apparently csh
is not one of them for instance, despite the function trying to
accommodate for non-POSIX Unix shells.  And please be realistic in the
amount of trust we can put on the complicated implementations for
non-Unix shells.  I can't judge them myself since I don't know the
syntax of those shells at all.  Does anyone here know their syntax
comprehensively, or checked the implementation against the documentation
of those shells?  Alone the documentation on MSDN (linked in the
comments of shell-quote-argument) is giving me a headache.  (Not that
POSIX sh isn't a big headache inducer, but at least I know its
single-quote semantics very well, since they're so incredibly simple.)

(Actually, those things should be documented for shell-quote-argument,
and my documentation would point there.)

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Random832 <random832@fastmail.com> writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>>> If you know of a real problem scenario reproducible with
>>> shell-quote-argument, please file a bug. Then we'll fix it.
>>
>> Not knowing that there are bugs is not proof that there are no bugs.
>
> Why aren't you as sure of its safety, regarding the POSIX section, as you
> are of the safety of your implementation?

I was probably being overly pedantic on that one, but if \<newline> has
a semantics other than resulting in a literal newline, I thought maybe
some other \<character> sequences might also have different semantics.
With different kinds of whitespace and all.

>>> Either way, please avoid reinventing the wheel.
>>
>> It's not a reinvention because it has very strict semantics with regard
>> to safety guarantees, which shell-quote-argument apparently doesn't.
>
> Out of curiosity, how are you guaranteeing that the result will be
> executed by a POSIX shell? Passing a string quoted by your function to
> MS Windows' cmd.exe (or, to that matter, to csh - even worse than the
> existing version) would be an absolute disaster as far as injection
> vulnerabilities go.

I'm afraid there's no way for my library to mechanically prevent that,
since the library only outputs commands as strings.  (So the user can
pass it to shell-command, async-shell-command, shell-command-on-region,
or anything else.)

Though it would have been best to be able to prevent such mistakes
mechanically, I hope having it in the first two sentences of the
documentation is good enough. :-)

Taylan

Re: [PATCH] Add shell-quasiquote.

[Paul Eggert]

Taylan Ulrich Bayırlı/Kammer wrote:
> Please tell me which shells shell-quote-argument is guaranteed to work
> safely on

Nobody can tell you that. What we can tell you is that shell-quote-argument 
works on a superset of uses that shqq--quote-string works on. The trust-based 
arguments against using shell-quote-argument all apply, with greater force, 
against using shqq--quote-string. For example, shqq--quote-string is more 
vulnerable to code-injection attacks than shell-quote-argument is.

I am not a fan of non-POSIX shells. They are a hassle to deal with and can cause 
significant problems in Emacs maintenance. In areas where they are a significant 
problem, we don't need to support them. But this particular instance is not a 
significant problem. Emacs already has a portable, tested, easy-to-use function 
to quote shell arguments, and there's good reason to use it here.

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Date: Sun, 18 Oct 2015 00:22:33 +0200
> Cc: emacs-devel@gnu.org
> 
> > On POSIX shells, shell-quote-argument is just as safe as
> > shqq--quote-string, and on non-POSIX shells it works better. So it's a
> > win, in both readability and in portability, to use
> > shell-quote-argument.
> 
> Fixing it does not seem easy at all given I can't trust
> shell-quote-argument.

You can trust it.

> And please be realistic in the amount of trust we can put on the
> complicated implementations for non-Unix shells.  I can't judge them
> myself since I don't know the syntax of those shells at all.  Does
> anyone here know their syntax comprehensively, or checked the
> implementation against the documentation of those shells?

Yes, we do.  Yes, we have.

> (Actually, those things should be documented for shell-quote-argument,
> and my documentation would point there.)

What do you think should be documented there that is missing?

Re: [PATCH] Add shell-quasiquote.

[Michael Albinus]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> On 10/18/2015 12:25 AM, Taylan Ulrich Bayırlı/Kammer wrote:
>>
>>> Not knowing that there are bugs is not proof that there are no bugs.
>>
>> If you can't point out a bug, you have no justification to not use the
>> standard function.
>
> No, I will *not* let users of my code potentially suffer from arbitrary
> code injection attacks, thank you very much.

If this is important for you, I recommend stop using Tramp. It makes
heavy use of (a slightly modified version of) `shell-quote-argument'.

> Taylan

Best regards, Michael.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Taylan Ulrich Bayırlı/Kammer wrote:
>> Please tell me which shells shell-quote-argument is guaranteed to work
>> safely on
>
> Nobody can tell you that. What we can tell you is that
> shell-quote-argument works on a superset of uses that
> shqq--quote-string works on. The trust-based arguments against using
> shell-quote-argument all apply, with greater force, against using
> shqq--quote-string. For example, shqq--quote-string is more vulnerable
> to code-injection attacks than shell-quote-argument is.

The domain of a function is part of its semantics, even if in Lisp we
have no way to formalize it other than through documentation.

The domain of shqq--quote-string is arguments to POSIX shell commands.
It's safe within that domain, i.e. its whole domain, meaning in short
"it's safe."

The domain of shell-quote-argument is unknown, so it's unknown whether
it's safe.  (If we include csh in its domain, it's known to be unsafe.)

Saying shqq--quote-string is more vulnerable is plain wrong.  It's
either as safe as, or safer, than shell-quote-argument.

That may sound like "semantics," but it carries over to practice very
simply: if I can't tell my users what shells shqq is safe for (or worse,
imply to them that they can use it with just any shell), there's a good
chance they'll use it for shells its unsafe for, exposing themselves to
vulnerabilities.  (Or if they're smarter than that, they will see that
my library is entirely useless for arbitrary input.)

Of course, I could use shell-quote-argument, but still document that
shqq is safe only for POSIX shells, no matter what shell-quote-argument
seems to try to accommodate for.  I think that's an unnecessary
complication, but if it's going to satisfy others for whatever reason
then I'm not opposed to it because it's at least harmless.  (I'll first
investigate further on possible breakage with shell-quote-argument's
quoting strategy for POSIX though.)

> I am not a fan of non-POSIX shells. They are a hassle to deal with and
> can cause significant problems in Emacs maintenance. In areas where
> they are a significant problem, we don't need to support them. But
> this particular instance is not a significant problem. Emacs already
> has a portable, tested, easy-to-use function to quote shell arguments,
> and there's good reason to use it here.

Arbitrary code injection is a very significant problem, and it has been
demonstrated in this thread that shell-quote-argument is vulnerable
against it.


Let's please all be more rigorous about such things in the future and
not pretend that problems are known not to exist when they're merely not
known to exist, let alone pretending that they don't exist shortly after
they've been demonstrated to exist.

I'll file a bug report about shell-quote-argument shortly, where we can
decide on more precise semantics for it (even if still open-ended) and
clearly document its safety guarantees.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Michael Albinus <michael.albinus@gmx.de> writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>>
>>> On 10/18/2015 12:25 AM, Taylan Ulrich Bayırlı/Kammer wrote:
>>>
>>>> Not knowing that there are bugs is not proof that there are no bugs.
>>>
>>> If you can't point out a bug, you have no justification to not use the
>>> standard function.
>>
>> No, I will *not* let users of my code potentially suffer from arbitrary
>> code injection attacks, thank you very much.
>
> If this is important for you, I recommend stop using Tramp. It makes
> heavy use of (a slightly modified version of) `shell-quote-argument'.

TRAMP doesn't read shell commands from arbitrary input sources...

I hope! :-)

Can a remote host arrange for TRAMP to use shell-quote-argument on
arbitrary strings and pass these to a shell that could potentially be
csh, or any shell we don't know shell-quote-argument to be safe for?  If
so, that might be a *very* serious issue and you should not be telling
*me* to stop using TRAMP but rather to the whole Emacs user-base.  I
mean it.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Michael Albinus]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> Can a remote host arrange for TRAMP to use shell-quote-argument on
> arbitrary strings and pass these to a shell that could potentially be
> csh, or any shell we don't know shell-quote-argument to be safe for?

Tramp uses `shell-quote-argument' on strings it has been constructed
itself. But those strings contain file names Tramp has read on the
remote side. No check what's the contents of such file names.

There is no special check on a remote shell being csh. But most of the
shell commands Tramp emits require a bournish shell. Otherwise, there
would be syntax errors soon, and Tramp would cease to continue on that
host.

In theory, anything could go with unknown file name strings. But I'm not
aware how one could exploit it. If you could show me a real exploit, I
will react.

> Taylan

Best regards, Michael.

PS: I'm working as Security Consultant, and so I am paranoid per
definition. But I'm not *such* paranoid until I see there are good
reasons for.

Re: [PATCH] Add shell-quasiquote.

[Random832]

Michael Albinus <michael.albinus@gmx.de> writes:
> PS: I'm working as Security Consultant, and so I am paranoid per
> definition. But I'm not *such* paranoid until I see there are good
> reasons for.

I do think it's disappointing that people are having such a cavalier
attitude about this...

The documentation does say:

| Precisely what this function does depends on your operating
| system. The function is designed to work with the syntax of your
| system’s standard shell; if you use an unusual shell, you will
| need to redefine this function.

But it doesn't bother explaining what operating systems it works on,
what is an unusual shell, or that _not_ having it defined in a way
consistent with the shell has security implications.

I think this has contributed to Taylan having a "gut feeling" that
it may not be secure on Windows, because it is difficult to
understand the implementation and is not well-documented and the
attitude is not a good sign. For example, ^-quoting is only applied
if [%!"] are present, but is applied to [%!()"<>&|^]. Why? Who
knows? The linked documentation for CommandLineToArgV provides no
insight about this second level of quoting. Why does ms-dos have
separate logic from nt?

And I know there's nothing to be done for it, but the fact that it
does not have any way to escape wildcards is concerning. I think it
would be reasonable for it to be an error if a character that it
doesn't know how to handle or can't handle is present, rather than
just muddle through. The whole point of having a function is to get
it right; if you don't care about that then (format "command \"%s\""
filename) is good enough for 95% of usage.


Speaking of Tramp, what if the local shell is not the same as the
remote shell? And I don't see how the commands it runs "require a
bournish shell" at all. they require that the commands themselves
exist, but that's nothing to do with the shell.

Tramp also (as of Emacs 24.5) wraps shell-quote-argument in its own
logic which fixes a newline handling bug that is no longer present.
Which also violates the "don't reinvent the wheel" policy - the fix
should have been submitted to shell-quote-argument itself (as it
ultimately was), and should never have been included in a version of
tramp that shipped with Emacs.

It even has a TODO item:

;; * Rewrite `tramp-shell-quote-argument' to abstain from using
;;   `shell-quote-argument'.

So much for not reinventing the wheel.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Date: Sun, 18 Oct 2015 00:22:33 +0200
>> Cc: emacs-devel@gnu.org
>> 
>> > On POSIX shells, shell-quote-argument is just as safe as
>> > shqq--quote-string, and on non-POSIX shells it works better. So it's a
>> > win, in both readability and in portability, to use
>> > shell-quote-argument.
>> 
>> Fixing it does not seem easy at all given I can't trust
>> shell-quote-argument.
>
> You can trust it.
>
>> And please be realistic in the amount of trust we can put on the
>> complicated implementations for non-Unix shells.  I can't judge them
>> myself since I don't know the syntax of those shells at all.  Does
>> anyone here know their syntax comprehensively, or checked the
>> implementation against the documentation of those shells?
>
> Yes, we do.  Yes, we have.
>
>> (Actually, those things should be documented for shell-quote-argument,
>> and my documentation would point there.)
>
> What do you think should be documented there that is missing?

I just filed bug #21702 about this.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21702

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Random832 <random832@fastmail.com> writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>> PS: I'm working as Security Consultant, and so I am paranoid per
>> definition. But I'm not *such* paranoid until I see there are good
>> reasons for.
>
> I do think it's disappointing that people are having such a cavalier
> attitude about this...
>
> The documentation does say:
>
> | Precisely what this function does depends on your operating
> | system. The function is designed to work with the syntax of your
> | system’s standard shell; if you use an unusual shell, you will
> | need to redefine this function.

Oh!  I had not looked at the Info manual at all.  As you say though, it
doesn't go into much more detail on the exact semantics anyway, so no
improvement there.

> But it doesn't bother explaining what operating systems it works on,
> what is an unusual shell, or that _not_ having it defined in a way
> consistent with the shell has security implications.
>
> I think this has contributed to Taylan having a "gut feeling" that
> it may not be secure on Windows, because it is difficult to
> understand the implementation and is not well-documented and the
> attitude is not a good sign. For example, ^-quoting is only applied
> if [%!"] are present, but is applied to [%!()"<>&|^]. Why? Who
> knows? The linked documentation for CommandLineToArgV provides no
> insight about this second level of quoting. Why does ms-dos have
> separate logic from nt?
>
> And I know there's nothing to be done for it, but the fact that it
> does not have any way to escape wildcards is concerning. I think it
> would be reasonable for it to be an error if a character that it
> doesn't know how to handle or can't handle is present, rather than
> just muddle through. The whole point of having a function is to get
> it right; if you don't care about that then (format "command \"%s\""
> filename) is good enough for 95% of usage.
>
>
> Speaking of Tramp, what if the local shell is not the same as the
> remote shell? And I don't see how the commands it runs "require a
> bournish shell" at all. they require that the commands themselves
> exist, but that's nothing to do with the shell.
>
> Tramp also (as of Emacs 24.5) wraps shell-quote-argument in its own
> logic which fixes a newline handling bug that is no longer present.
> Which also violates the "don't reinvent the wheel" policy - the fix
> should have been submitted to shell-quote-argument itself (as it
> ultimately was), and should never have been included in a version of
> tramp that shipped with Emacs.
>
> It even has a TODO item:
>
> ;; * Rewrite `tramp-shell-quote-argument' to abstain from using
> ;;   `shell-quote-argument'.
>
> So much for not reinventing the wheel.

Thank you. :-)

Taylan

Re: [PATCH] Add shell-quasiquote.

[Michael Albinus]

Random832 <random832@fastmail.com> writes:

> Speaking of Tramp, what if the local shell is not the same as the
> remote shell? And I don't see how the commands it runs "require a
> bournish shell" at all. they require that the commands themselves
> exist, but that's nothing to do with the shell.

In Tramp, it is the default case that the local shell is different from
the remote shell. The local shell could be even w32's cmd.exe. Tramp
handles this.

> Tramp also (as of Emacs 24.5) wraps shell-quote-argument in its own
> logic which fixes a newline handling bug that is no longer present.
> Which also violates the "don't reinvent the wheel" policy - the fix
> should have been submitted to shell-quote-argument itself (as it
> ultimately was), and should never have been included in a version of
> tramp that shipped with Emacs.

You know, that Tramp supports older Emacsen as well as XEmacs 21.4+?

And `tramp-shell-quote-argument' binds also `system-type' in order to
get a proper result for the remote bournish shell. Admitted, this is not
a documented way to do this ...

> It even has a TODO item:
>
> ;; * Rewrite `tramp-shell-quote-argument' to abstain from using
> ;;   `shell-quote-argument'.

This is an obsolete comment, older than 10+ years. It should have been
removed long ago; thanks for the heads up.

> So much for not reinventing the wheel.

You are invited to support me as co-maintainer of Tramp. Seriously.
Writing compatibility code is one of the major tasks.

Best regards, Michael.

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 17 Oct 2015 19:40:21 -0700
> Cc: emacs-devel@gnu.org
> 
> Taylan Ulrich Bayırlı/Kammer wrote:
> > Please tell me which shells shell-quote-argument is guaranteed to work
> > safely on
> 
> Nobody can tell you that. What we can tell you is that shell-quote-argument 
> works on a superset of uses that shqq--quote-string works on. The trust-based 
> arguments against using shell-quote-argument all apply, with greater force, 
> against using shqq--quote-string. For example, shqq--quote-string is more 
> vulnerable to code-injection attacks than shell-quote-argument is.
> 
> I am not a fan of non-POSIX shells. They are a hassle to deal with and can cause 
> significant problems in Emacs maintenance. In areas where they are a significant 
> problem, we don't need to support them. But this particular instance is not a 
> significant problem. Emacs already has a portable, tested, easy-to-use function 
> to quote shell arguments, and there's good reason to use it here.

I completely agree with everything Paul wrote here.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Sat, 17 Oct 2015 19:40:21 -0700
>> Cc: emacs-devel@gnu.org
>> 
>> Taylan Ulrich Bayırlı/Kammer wrote:
>> > Please tell me which shells shell-quote-argument is guaranteed to work
>> > safely on
>> 
>> Nobody can tell you that. What we can tell you is that shell-quote-argument 
>> works on a superset of uses that shqq--quote-string works on. The trust-based 
>> arguments against using shell-quote-argument all apply, with greater force, 
>> against using shqq--quote-string. For example, shqq--quote-string is more 
>> vulnerable to code-injection attacks than shell-quote-argument is.
>> 
>> I am not a fan of non-POSIX shells. They are a hassle to deal with and can cause 
>> significant problems in Emacs maintenance. In areas where they are a significant 
>> problem, we don't need to support them. But this particular instance is not a 
>> significant problem. Emacs already has a portable, tested, easy-to-use function 
>> to quote shell arguments, and there's good reason to use it here.
>
> I completely agree with everything Paul wrote here.

And as I already said, code injection is far from "not a significant
problem."  I hope everyone here agrees with that.

But anyway, we should discuss this on the bug report ML.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Sun, 18 Oct 2015 08:59:32 -0400
> 
> | Precisely what this function does depends on your operating
> | system. The function is designed to work with the syntax of your
> | system’s standard shell; if you use an unusual shell, you will
> | need to redefine this function.
> 
> But it doesn't bother explaining what operating systems it works on,

If the doc string doesn't say anything about limitations or special
considerations for specific OSes, it means the function works on all
supported systems.  That is the convention in Emacs (and elsewhere, I
believe): only state limitations, not the lack thereof.  Explicitly
mentioning the lack of limitations will unduly bloat the
documentation.

> what is an unusual shell

Any shell that is not the "system's standard shell" is unusual.  I
thought the text made that clear; if not, please suggest how to
clarify it (without having an exhaustive list of shells, which would
be a maintenance burden).

> or that _not_ having it defined in a way consistent with the shell
> has security implications.

Any undefined behavior can have security implications.  I hope that
once the domain of the function is now explained in the doc string,
users will be able to realize that for this particular function, if
they need to use a non-standard shell.

> I think this has contributed to Taylan having a "gut feeling" that
> it may not be secure on Windows, because it is difficult to
> understand the implementation and is not well-documented and the
> attitude is not a good sign. For example, ^-quoting is only applied
> if [%!"] are present, but is applied to [%!()"<>&|^]. Why? Who
> knows?

We do.  See the discussion that led to that code; it started here:

  http://lists.gnu.org/archive/html/emacs-devel/2011-04/msg00717.html

> Why does ms-dos have separate logic from nt?

Because the "standard shell" is different (command.cm instead of
cmd.exe), and the way to invoke inferior process via functions like
'system' differs.  The details are too many to describe here; if you
are interested, please read the sources of the DJGPP libc, available
from http://www.delorie.com/djgpp/cvs.html, and the accompanying
documentation here:

  http://www.delorie.com/djgpp/doc/libc-2.02/

> And I know there's nothing to be done for it, but the fact that it
> does not have any way to escape wildcards is concerning.

Sorry, I don't follow: in what situation do you think the wildcards
cannot be escaped?  Are you still talking about MS-Windows?

Re: [PATCH] Add shell-quasiquote.

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> I am not a fan of non-POSIX shells. They are a hassle to deal with and can
>> cause significant problems in Emacs maintenance. In areas where they are a
>> significant problem, we don't need to support them. But this particular
>> instance is not a significant problem. Emacs already has a portable,
>> tested, easy-to-use function to quote shell arguments, and there's good
>> reason to use it here.

> I completely agree with everything Paul wrote here.

Taylan,

I hope these comments will be viewed as an effort to be constructive, rather
than dismissive of your work or effort in any way.

We have good reason to suspect functions that might duplicate multiple decades
of prior art. Until now, shell quoting has been working nicely, so you can
imagine we are a touch skeptical of the need to rewrite it at this stage.

However, if you can produce some test cases that show a deficiency in
shell-quote-argument, you'll have done several useful things for us:

  1. Produced a test case covering an untested area;
  2. Helped us understand a deficiency in code we had completely trusted;
  3. Given us an opportunity to fix that code, benefiting all its users;
  4. Allowed us to provide you with a working shell-quote-argument, so you
     needn't waste further time at that level.

This is our hope; it is not meant to discourage you, or emphasize who is wrong
about what details. Your energy and enthusiasm are of great value.

John

Re: [PATCH] Add shell-quasiquote.

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Random832 <random832@fastmail.com>
>> what is an unusual shell
>
> Any shell that is not the "system's standard shell" is unusual.  I
> thought the text made that clear; if not, please suggest how to
> clarify it (without having an exhaustive list of shells, which would
> be a maintenance burden).

It works on shells that share quoting rules with POSIX, and it works
on MS-DOS, and it works on MS Windows with the assumption that
cmdproxy or any shell listed in w32-system-shells uses the same
rules as cmd.exe and that any other shell uses POSIX semantics.

This gives people a good starting point for understanding what is
"unusual" and what is not, (i.e. bash, ksh, and zsh are not unusual,
even if they are not the system shell) without requiring an
exhaustive list.

>> And I know there's nothing to be done for it, but the fact that it
>> does not have any way to escape wildcards is concerning.
>
> Sorry, I don't follow: in what situation do you think the wildcards
> cannot be escaped?  Are you still talking about MS-Windows?

Yes, sorry. A typical Windows program (at least, one compiled with
MSVC's setargv.obj) will try to interpret wildcards in any part of
CommandLineToArgv's result which contains a ? or * character, with
no provision to prevent it from doing so. (In particular, double
quotes have no effect).

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Sun, 18 Oct 2015 15:17:53 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> From: Random832 <random832@fastmail.com>
> >> what is an unusual shell
> >
> > Any shell that is not the "system's standard shell" is unusual.  I
> > thought the text made that clear; if not, please suggest how to
> > clarify it (without having an exhaustive list of shells, which would
> > be a maintenance burden).
> 
> It works on shells that share quoting rules with POSIX, and it works
> on MS-DOS, and it works on MS Windows with the assumption that
> cmdproxy or any shell listed in w32-system-shells uses the same
> rules as cmd.exe and that any other shell uses POSIX semantics.

Yes, that's a reasonable interpretation of "standard shell" on each of
these operating systems.

> >> And I know there's nothing to be done for it, but the fact that it
> >> does not have any way to escape wildcards is concerning.
> >
> > Sorry, I don't follow: in what situation do you think the wildcards
> > cannot be escaped?  Are you still talking about MS-Windows?
> 
> Yes, sorry. A typical Windows program (at least, one compiled with
> MSVC's setargv.obj) will try to interpret wildcards in any part of
> CommandLineToArgv's result which contains a ? or * character, with
> no provision to prevent it from doing so. (In particular, double
> quotes have no effect).

This actually depends on the startup code.  The latest release of
mingw.org's MinGW runtime does allow you to quote wildcard characters.
And on Windows XP and older even the other runtimes allow that.

In any case, this is not an Emacs problem.

Re: [PATCH] Add shell-quasiquote.

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > Random832 writes:

 > > Yes, sorry. A typical Windows program (at least, one compiled with
 > > MSVC's setargv.obj) will try to interpret wildcards in any part of
 > > CommandLineToArgv's result which contains a ? or * character, with
 > > no provision to prevent it from doing so. (In particular, double
 > > quotes have no effect).
 > 
 > This actually depends on the startup code.  The latest release of
 > mingw.org's MinGW runtime does allow you to quote wildcard characters.
 > And on Windows XP and older even the other runtimes allow that.
 > 
 > In any case, this is not an Emacs problem.

Of course it is, in a security context.  I don't think it matters
anywhere near as much as code injection, but if Emacs is built with
one of those runtimes that doesn't allow wildcards to be disabled, its
users will be affected.

I think it probably can be immediately judged irrelevant (and perhaps
that's what you meant) if Emacs is normally built with a runtime that
doesn't interpret quoted wildcards, and the runtimes that always
interpret wildcards are not supported.  But if Emacs is to meet modern
security standards, that kind of thing needs to be considered and
confirmed, and to that extent it *is* Emacs's problem.  Clearly some
developers of Emacs Lisp applications want Emacs to meet those
standards.  YMMV, and mine does:

IMHO Emacs is unlikely to meet modern security standards in my
lifetime.  I am discouraged from even thinking about it when the
advocates of security are passing strings to an unknown shell program
and then complaining that Emacs's quoting function may be insecure.
Putting a shell in the loop is already saying "Security?  What, me
worry??"  After all, even if you check for POSIX, it might be a
slightly dated installation of GNU Bash. :-(

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> Date: Mon, 19 Oct 2015 13:32:51 +0900
> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Random832 <random832@fastmail.com>,
>     emacs-devel@gnu.org
> 
> Eli Zaretskii writes:
>  > Random832 writes:
> 
>  > > Yes, sorry. A typical Windows program (at least, one compiled with
>  > > MSVC's setargv.obj) will try to interpret wildcards in any part of
>  > > CommandLineToArgv's result which contains a ? or * character, with
>  > > no provision to prevent it from doing so. (In particular, double
>  > > quotes have no effect).
>  > 
>  > This actually depends on the startup code.  The latest release of
>  > mingw.org's MinGW runtime does allow you to quote wildcard characters.
>  > And on Windows XP and older even the other runtimes allow that.
>  > 
>  > In any case, this is not an Emacs problem.
> 
> Of course it is, in a security context.  I don't think it matters
> anywhere near as much as code injection, but if Emacs is built with
> one of those runtimes that doesn't allow wildcards to be disabled, its
> users will be affected.
> 
> I think it probably can be immediately judged irrelevant (and perhaps
> that's what you meant) if Emacs is normally built with a runtime that
> doesn't interpret quoted wildcards, and the runtimes that always
> interpret wildcards are not supported.

That's a misunderstanding: the runtime in question is the one used
with the program that Emacs invokes, not the one used to run Emacs
itself.  On MS-Windows, the expansion of wildcards on the command line
is done by the application which accepts the command line (in its
startup code, before the main function is invoked), so that's what
determines whether a quoted "*" will or will not be expanded.  The
invoking Emacs cannot control or affect that in any way.

Re: [PATCH] Add shell-quasiquote.

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2009 bytes --]

On 10/18/2015 10:15 PM, Eli Zaretskii wrote:
>> Date: Mon, 19 Oct 2015 13:32:51 +0900
>> From: "Stephen J. Turnbull" <stephen@xemacs.org>
>> Cc: Random832 <random832@fastmail.com>,
>>     emacs-devel@gnu.org
>>
>> Eli Zaretskii writes:
>>  > Random832 writes:
>>
>>  > > Yes, sorry. A typical Windows program (at least, one compiled with
>>  > > MSVC's setargv.obj) will try to interpret wildcards in any part of
>>  > > CommandLineToArgv's result which contains a ? or * character, with
>>  > > no provision to prevent it from doing so. (In particular, double
>>  > > quotes have no effect).
>>  > 
>>  > This actually depends on the startup code.  The latest release of
>>  > mingw.org's MinGW runtime does allow you to quote wildcard characters.
>>  > And on Windows XP and older even the other runtimes allow that.
>>  > 
>>  > In any case, this is not an Emacs problem.
>>
>> Of course it is, in a security context.  I don't think it matters
>> anywhere near as much as code injection, but if Emacs is built with
>> one of those runtimes that doesn't allow wildcards to be disabled, its
>> users will be affected.
>>
>> I think it probably can be immediately judged irrelevant (and perhaps
>> that's what you meant) if Emacs is normally built with a runtime that
>> doesn't interpret quoted wildcards, and the runtimes that always
>> interpret wildcards are not supported.
> 
> That's a misunderstanding: the runtime in question is the one used
> with the program that Emacs invokes, not the one used to run Emacs
> itself.  On MS-Windows, the expansion of wildcards on the command line
> is done by the application which accepts the command line (in its
> startup code, before the main function is invoked), so that's what
> determines whether a quoted "*" will or will not be expanded.  The
> invoking Emacs cannot control or affect that in any way.

But maybe we can make the argument-quoting style a particular program
expects a user-customizable variable.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> Cc: random832@fastmail.com, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Sun, 18 Oct 2015 22:19:51 -0700
> 
> On 10/18/2015 10:15 PM, Eli Zaretskii wrote:
> >> Date: Mon, 19 Oct 2015 13:32:51 +0900
> >> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> >> Cc: Random832 <random832@fastmail.com>,
> >>     emacs-devel@gnu.org
> >>
> >> Eli Zaretskii writes:
> >>  > Random832 writes:
> >>
> >>  > > Yes, sorry. A typical Windows program (at least, one compiled with
> >>  > > MSVC's setargv.obj) will try to interpret wildcards in any part of
> >>  > > CommandLineToArgv's result which contains a ? or * character, with
> >>  > > no provision to prevent it from doing so. (In particular, double
> >>  > > quotes have no effect).
> >>  > 
> >>  > This actually depends on the startup code.  The latest release of
> >>  > mingw.org's MinGW runtime does allow you to quote wildcard characters.
> >>  > And on Windows XP and older even the other runtimes allow that.
> >>  > 
> >>  > In any case, this is not an Emacs problem.
> >>
> >> Of course it is, in a security context.  I don't think it matters
> >> anywhere near as much as code injection, but if Emacs is built with
> >> one of those runtimes that doesn't allow wildcards to be disabled, its
> >> users will be affected.
> >>
> >> I think it probably can be immediately judged irrelevant (and perhaps
> >> that's what you meant) if Emacs is normally built with a runtime that
> >> doesn't interpret quoted wildcards, and the runtimes that always
> >> interpret wildcards are not supported.
> > 
> > That's a misunderstanding: the runtime in question is the one used
> > with the program that Emacs invokes, not the one used to run Emacs
> > itself.  On MS-Windows, the expansion of wildcards on the command line
> > is done by the application which accepts the command line (in its
> > startup code, before the main function is invoked), so that's what
> > determines whether a quoted "*" will or will not be expanded.  The
> > invoking Emacs cannot control or affect that in any way.
> 
> But maybe we can make the argument-quoting style a particular program
> expects a user-customizable variable.

Unless I misunderstand you, this isn't possible.  Or maybe there's
some subtle trick that I'm not aware of.

Here are the details for those who don't already know them.

There are only 2 styles of quoting used by native Windows programs,
and a 3rd style used by ported Posix shells.  Emacs already supports
the 3rd style automatically (by examining the program it is about to
invoke).  As to the other 2 styles, they are the "cmd style" and the
style of MS runtime's setargv.  The "cmd style" is AFAIK supported
only by cmd itself and compatible shells, which leaves us with the
setargv style.

The problem which started this sub-thread is that Microsoft changed
the behavior of setargv, which lives in the system shared library
every C program links against, starting with Windows Vista (or maybe
Windows 7, I never used Vista).  Before the change, quoting a wildcard
"like this" would disable its globbing; after the change, quoted
wildcards are globbed regardless, and there's no alternative way of
protecting wildcards from globbing, AFAIK, except make your program
avoid calling setargv, and instead call your own globbing function
(which is what the latest MinGW runtime does).

The upshot of this is that a program will behave differently depending
on the Windows version it runs on, unless it was linked with the
latest MinGW runtime.

So given all this mess, how would you suggest to allow customization
of the wildcard quoting?

(Btw, in general, I'd prefer Emacs to DTRT by default, without asking
the user to customize anything.  If possible, of course.)

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> IMHO Emacs is unlikely to meet modern security standards in my
> lifetime.  I am discouraged from even thinking about it when the
> advocates of security are passing strings to an unknown shell program
> and then complaining that Emacs's quoting function may be insecure.
> Putting a shell in the loop is already saying "Security?  What, me
> worry??"  After all, even if you check for POSIX, it might be a
> slightly dated installation of GNU Bash. :-(

I have to confess that's a good point.  Maybe it's silly to even ask for
security, in general, when it comes to generating shell commands.  Then
again consider a fairly simple but still pretty useful example like:

  (shqq (cp -- ,@files target))

When the resulting string is passed verbatim as a command to a POSIX
shell, there's really no place for error there, so long as it's ensured
that each member of 'files' will be inserted verbatim into the ARGV of
cp(1).

Most commands will look more or less like that...


On a tangentially related topic, I just discovered there's more semantic
differences between using shqq--quote-string and shell-quote-argument.
The former quotes *everything*, e.g. "if" becomes "'if'", meaning you
cannot use shell keywords.  After a bit of pondering, I would say that's
a feature.  (Try constructing an if statement with shqq even when it
uses shell-quote-argument.  You can't (without the "double-unquote")
because you can't insert a bare newline or semicolon anyway.)

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

[-- Attachment #1: Type: text/plain, Size: 469 bytes --]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> This is for ELPA.
>
> [...]

Here's a version with slightly improved documentation and a clearer
reasoning for why shell-quote-argument is avoided.

Eli refused to clarify the safety guarantees of shell-quote-argument in
bug#21702, and after all it has significantly different semantics anyway
(it doesn't quote shell keywords like 'if'), so I will not be using
shell-quote-argument.


[-- Attachment #2: 0001-Add-shell-quasiquote.patch --]
[-- Type: text/x-diff, Size: 6233 bytes --]

From 276e3adc61b2f083b0348fd231a97feaa7017e36 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 <taylanbayirli@gmail.com>
Date: Sat, 17 Oct 2015 18:32:22 +0200
Subject: [PATCH 1/3] Add shell-quasiquote.

---
 packages/shell-quasiquote/shell-quasiquote.el | 151 ++++++++++++++++++++++++++
 1 file changed, 151 insertions(+)
 create mode 100644 packages/shell-quasiquote/shell-quasiquote.el

diff --git a/packages/shell-quasiquote/shell-quasiquote.el b/packages/shell-quasiquote/shell-quasiquote.el
new file mode 100644
index 0000000..1f18862
--- /dev/null
+++ b/packages/shell-quasiquote/shell-quasiquote.el
@@ -0,0 +1,151 @@
+;;; shell-quasiquote.el --- Turn s-expressions into shell command strings.
+
+;; Copyright (C) 2015  Free Software Foundation, Inc.
+
+;; Author: Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
+;; Keywords: extensions, unix
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; "Shell quasiquote" -- turn s-expressions into POSIX shell command strings.
+;;
+;; Shells other than POSIX sh are not supported.
+;;
+;; Quoting is automatic and safe against injection.
+;;
+;;   (let ((file1 "file one")
+;;         (file2 "file two"))
+;;     (shqq (cp -r ,file1 ,file2 "My Files")))
+;;       => "cp -r 'file one' 'file two' 'My Files'"
+;;
+;; You can splice many arguments into place with ,@foo.
+;;
+;;   (let ((files (list "file one" "file two")))
+;;     (shqq (cp -r ,@files "My Files")))
+;;       => "cp -r 'file one' 'file two' 'My Files'"
+;;
+;; Note that the quoting disables a variety of shell expansions like ~/foo,
+;; $ENV_VAR, and e.g. {x..y} in GNU Bash.
+;;
+;; You can use ,,foo to escape the quoting.
+;;
+;;   (let ((files "file1 file2"))
+;;     (shqq (cp -r ,,files "My Files")))
+;;       => "cp -r file1 file2 'My Files'"
+;;
+;; And ,,@foo to splice and escape quoting.
+;;
+;;   (let* ((arglist '("-x 'foo bar' -y baz"))
+;;          (arglist (append arglist '("-z 'qux fux'"))))
+;;     (shqq (command ,,@arglist)))
+;;       => "command -x 'foo bar' -y baz -z 'qux fux'"
+;;
+;; Neat, eh?
+
+\f
+;;; Code:
+
+;;; We don't use `shell-quote-argument' because it doesn't provide any safety
+;;; guarantees, and this quotes shell keywords as well.
+(defun shqq--quote-string (string)
+  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))
+
+(defun shqq--atom-to-string (atom)
+  (cond
+   ((symbolp atom) (symbol-name atom))
+   ((stringp atom) atom)
+   ((numberp atom) (number-to-string atom))
+   (t (error "Bad shqq atom: %S" atom))))
+
+(defun shqq--quote-atom (atom)
+  (shqq--quote-string (shqq--atom-to-string atom)))
+
+(defun shqq--match-comma (form)
+  "Matches FORM against ,foo i.e. (\, foo) and returns foo.
+Returns nil if FORM didn't match.  You can't disambiguate between
+FORM matching ,nil and not matching."
+  (if (and (consp form)
+           (eq '\, (car form))
+           (consp (cdr form))
+           (null (cddr form)))
+      (cadr form)))
+
+(defun shqq--match-comma2 (form)
+  "Matches FORM against ,,foo i.e. (\, (\, foo)) and returns foo.
+Returns nil if FORM didn't match.  You can't disambiguate between
+FORM matching ,,nil and not matching."
+  (if (and (consp form)
+           (eq '\, (car form))
+           (consp (cdr form))
+           (null (cddr form)))
+      (shqq--match-comma (cadr form))))
+
+\f
+(defmacro shqq (parts)
+  "First, PARTS is turned into a list of strings.  For this,
+every element of PARTS must be one of:
+
+- a symbol, evaluating to its name,
+
+- a string, evaluating to itself,
+
+- a number, evaluating to its decimal representation,
+
+- \",expr\", where EXPR must evaluate to an atom that will be
+  interpreted according to the previous rules,
+
+- \",@list-expr\", where LIST-EXPR must evaluate to a list whose
+  elements will each be interpreted like the EXPR in an \",EXPR\"
+  form, and spliced into the list of strings,
+
+- \",,expr\", where EXPR is interpreted like in \",expr\",
+
+- or \",,@expr\", where EXPR is interpreted like in \",@expr\".
+
+In the resulting list of strings, all elements except the ones
+resulting from \",,expr\" and \",,@expr\" forms are quoted for
+shell grammar.
+
+Finally, the resulting list of strings is concatenated with
+separating spaces."
+  (let ((parts
+         (mapcar
+          (lambda (part)
+            (cond
+             ((atom part) (shqq--quote-atom part))
+             ;; We use the match-comma helpers because pcase can't match ,foo.
+             (t (pcase part
+                  ;; ,,foo i.e. (, (, foo))
+                  ((pred shqq--match-comma2)
+                   (shqq--match-comma2 part))
+                  ;; ,,@foo i.e. (, (,@ foo))
+                  ((and (pred shqq--match-comma)
+                        (let `,@,form (shqq--match-comma part)))
+                   `(mapconcat #'identity ,form " "))
+                  ;; ,foo
+                  ;; Insert redundant 'and x' to work around debbugs#18554.
+                  ((and x (pred shqq--match-comma))
+                   `(shqq--quote-atom ,(shqq--match-comma part)))
+                  ;; ,@foo
+                  (`,@,form
+                   `(mapconcat #'shqq--quote-atom ,form " "))
+                  (_
+                   (error "Bad shqq part: %S" part))))))
+          parts)))
+    `(mapconcat #'identity (list ,@parts) " ")))
+
+(provide 'shell-quasiquote)
+;;; shell-quasiquote.el ends here
-- 
2.5.0

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> This is for ELPA.
>>
>> [...]
>
> Here's a version with slightly improved documentation and a clearer
> reasoning for why shell-quote-argument is avoided.
>
> Eli refused to clarify the safety guarantees of shell-quote-argument in
> bug#21702, and after all it has significantly different semantics anyway
> (it doesn't quote shell keywords like 'if'),

Uh, why should it?

echo if

works just fine. So does

/bin/echo if

The only position where "if" is somewhat special is in command position
and that's not what shell-quote-argument is for.

Seriously, try getting used to the idea that Emacs has not been written
by idiots who would not manage to encounter or analyze basic bugs in
10 years of usage of a very frequently encountered function and that
you'll whip up something better within a minute.

That's just "not invented here" syndrome.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

David Kastrup <dak@gnu.org> writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>>
>>> This is for ELPA.
>>>
>>> [...]
>>
>> Here's a version with slightly improved documentation and a clearer
>> reasoning for why shell-quote-argument is avoided.
>>
>> Eli refused to clarify the safety guarantees of shell-quote-argument in
>> bug#21702, and after all it has significantly different semantics anyway
>> (it doesn't quote shell keywords like 'if'),
>
> Uh, why should it?
>
> echo if
>
> works just fine. So does
>
> /bin/echo if
>
> The only position where "if" is somewhat special is in command position
> and that's not what shell-quote-argument is for.

It was not criticism of shell-quote-argument (those are separate).
Indeed it quotes arguments.  My variant also quotes things that may be
the name of the command and not an argument.

> Seriously, try getting used to the idea that Emacs has not been written
> by idiots who would not manage to encounter or analyze basic bugs in
> 10 years of usage of a very frequently encountered function and that
> you'll whip up something better within a minute.
>
> That's just "not invented here" syndrome.

Don't put words in my mouth please.  The topic has been discussed in
length on the bug report page anyway, I'm not continuing that.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Date: Mon, 19 Oct 2015 14:35:26 +0200
> 
> Eli refused to clarify the safety guarantees of shell-quote-argument in
> bug#21702

No, Eli didn't refuse.  Eli pushed a change that clarified which
shells shell-quote-argument was designed to support.

> and after all it has significantly different semantics anyway
> (it doesn't quote shell keywords like 'if'), so I will not be using
> shell-quote-argument.

Please do use shell-quote-argument.  Fixing any problems in
shell-quote-argument is orthogonal to this package.  Inventing your
own functions for that is not the right way.

Thanks.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Date: Mon, 19 Oct 2015 14:35:26 +0200
>> 
>> Eli refused to clarify the safety guarantees of shell-quote-argument in
>> bug#21702
>
> No, Eli didn't refuse.  Eli pushed a change that clarified which
> shells shell-quote-argument was designed to support.
>
>> and after all it has significantly different semantics anyway
>> (it doesn't quote shell keywords like 'if'), so I will not be using
>> shell-quote-argument.
>
> Please do use shell-quote-argument.  Fixing any problems in
> shell-quote-argument is orthogonal to this package.  Inventing your
> own functions for that is not the right way.

I won't use shell-quote-argument so long as it doesn't provide the same
semantics.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

2015-10-19 14:22 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
>
> > and after all it has significantly different semantics anyway
> > (it doesn't quote shell keywords like 'if'), so I will not be using
> > shell-quote-argument.
>
> Please do use shell-quote-argument.  Fixing any problems in
> shell-quote-argument is orthogonal to this package.  Inventing your
> own functions for that is not the right way.
>

Yes, please do use it.
Elpa is part of Emacs. Having some feature reimplemented there is like
having the same function implemented twice inside Emacs.
As much as possible, the code on the archive should extend the code
released with the Emacs tarballs, not repeat it.

[-- Attachment #2: Type: text/html, Size: 1066 bytes --]

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> 2015-10-19 14:22 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
>
>     > and after all it has significantly different semantics anyway
>     > (it doesn't quote shell keywords like 'if'), so I will not be
>     using
>     > shell-quote-argument.
>     
>     Please do use shell-quote-argument. Fixing any problems in
>     shell-quote-argument is orthogonal to this package. Inventing your
>     own functions for that is not the right way.
>     
>
> Yes, please do use it. 
> Elpa is part of Emacs. Having some feature reimplemented there is like
> having the same function implemented twice inside Emacs.
> As much as possible, the code on the archive should extend the code
> released with the Emacs tarballs, not repeat it.

I went over the reasons extensively in the bug report.

I won't be using shell-quote-argument because it doesn't provide the
needed semantics.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Random832]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
> It was not criticism of shell-quote-argument (those are separate).
> Indeed it quotes arguments.  My variant also quotes things that may be
> the name of the command and not an argument.

But why does it *need* to?

Do you realize that you are now suggesting an injection scenario whereby
the attacker is _legitimately_ permitted to supply an arbitrary string
for an ordinary command to be executed, but somehow letting them execute
"if" [which will be a syntax error anyway since they can't supply the
then/fi as separate statements] becomes a security hole?

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Random832 <random832@fastmail.com> writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>> It was not criticism of shell-quote-argument (those are separate).
>> Indeed it quotes arguments.  My variant also quotes things that may be
>> the name of the command and not an argument.
>
> But why does it *need* to?
>
> Do you realize that you are now suggesting an injection scenario whereby
> the attacker is _legitimately_ permitted to supply an arbitrary string
> for an ordinary command to be executed, but somehow letting them execute
> "if" [which will be a syntax error anyway since they can't supply the
> then/fi as separate statements] becomes a security hole?

It's mostly just a side-effect of the simpler implementation.  If
there's a /bin/if on the system, (shqq (if blah blah)) will call it.
Not very useful, but consistent.

It isn't necessary for shell-quote-argument to do something like that
for me to decide to use it, only the safety guarantees are necessary.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Dmitry Gutov]

On 10/19/2015 04:43 PM, Taylan Ulrich Bayırlı/Kammer wrote:

> I went over the reasons extensively in the bug report.
>
> I won't be using shell-quote-argument because it doesn't provide the
> needed semantics.

As long as your approach doesn't fit our collective standards, I'm sure 
we can live without this contribution.

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: emacs-devel@gnu.org
> Date: Mon, 19 Oct 2015 15:36:56 +0200
> 
> > Please do use shell-quote-argument.  Fixing any problems in
> > shell-quote-argument is orthogonal to this package.  Inventing your
> > own functions for that is not the right way.
> 
> I won't use shell-quote-argument so long as it doesn't provide the same
> semantics.

IMO (and that of others), it provides the correct semantics for this
kind of functionality.

Please feel free to convince us what different semantics it should
provide.  If you succeed, shell-quote-argument will be changed as
needed.

But if you don't succeed, then as long as you want your package to be
part of ELPA, please accept the opinions of the Emacs maintenance
team, even if you don't necessarily agree.  That's the nature of
community projects such as Emacs and ELPA.  The goal is to provide a
coherent set of packages that all adhere to some common standard,
which users can rely upon.  If we all start rolling our own
replacements for APIs we think should work slightly differently, the
result will be chaos: a set of packages that only don't really work
together.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/19/2015 04:43 PM, Taylan Ulrich Bayırlı/Kammer wrote:
>
>> I went over the reasons extensively in the bug report.
>>
>> I won't be using shell-quote-argument because it doesn't provide the
>> needed semantics.
>
> As long as your approach doesn't fit our collective standards, I'm
> sure we can live without this contribution.

And I can live without the emacs-devel tyrants accepting my
contributions, but the situation is so outrageous right now that I'm
seriously considering pulling back all my current and future
contributions, until the development team changes (in spirit or
persons).

(Except for possible work on GuileEmacs, which I'd work on as part of
the Guile community.)

I've contributed a library that does one very well-defined thing and
does it fully correctly.  You're free to reject it, or take it and do
whatever you want with it.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Paul Eggert]

On 10/19/2015 06:53 AM, Taylan Ulrich Bayırlı/Kammer wrote:
> If there's a /bin/if on the system, (shqq (if blah blah)) will call it.
> Not very useful, but consistent.

No, it's not consistent. If there's a file named "sh" in the working 
directory, shqq will not attempt to call it unless "." precedes "/bin" 
and "/usr/bin" in PATH (which is not a good idea for other reasons).  
More amusingly, although there is a file named "." in the working 
directory, shqq will not attempt to call it, and instead will invoke a 
builtin shell command that allows injection of arbitrary shell code.

To quote a file name that one wants to execute one must also prefix it 
with './', unless the unquoted name starts with '/'. That would be more 
consistent. Of course, it would also be incorrect for strings intended 
to be the command's arguments. This means that neither 
shqq--quote-string nor shell-quote-argument should be used blindly to 
quote command names. In this sense, shell-quote-argument has a better 
name than shqq--quote-string does, since the word "argument" means the 
function is intended for command arguments, as opposed to arbitrary strings.

All in all there does not seem to be a good reason to have a separate 
function shqq--quote-string. Any improvement that it has over 
shell-quote-argument should be folded into shell-quote-argument.

Re: [PATCH] Add shell-quasiquote.

[Dmitry Gutov]

On 10/19/2015 05:09 PM, Taylan Ulrich Bayırlı/Kammer wrote:

> And I can live without the emacs-devel tyrants accepting my
> contributions, but the situation is so outrageous right now that I'm
> seriously considering pulling back all my current and future
> contributions, until the development team changes (in spirit or
> persons).

But that is the main point of contributing a package to GNU ELPA: the 
emacs-devel "tyrants" review it, and then share (at least to some 
degree) the responsibility of supporting it.

In general, that means accepting the collective opinions of emacs-devel. 
Or the head maintainer (not sure if John has accepted that position yet).

You can also distribute the package through MELPA.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Paul Eggert <eggert@cs.ucla.edu> writes:

> On 10/19/2015 06:53 AM, Taylan Ulrich Bayırlı/Kammer wrote:
>> If there's a /bin/if on the system, (shqq (if blah blah)) will call it.
>> Not very useful, but consistent.
>
> No, it's not consistent. If there's a file named "sh" in the working
> directory, shqq will not attempt to call it unless "." precedes "/bin"
> and "/usr/bin" in PATH (which is not a good idea for other reasons).
> More amusingly, although there is a file named "." in the working
> directory, shqq will not attempt to call it, and instead will invoke a
> builtin shell command that allows injection of arbitrary shell code.
>
> To quote a file name that one wants to execute one must also prefix it
> with './', unless the unquoted name starts with '/'. That would be
> more consistent. Of course, it would also be incorrect for strings
> intended to be the command's arguments. This means that neither
> shqq--quote-string nor shell-quote-argument should be used blindly to
> quote command names. In this sense, shell-quote-argument has a better
> name than shqq--quote-string does, since the word "argument" means the
> function is intended for command arguments, as opposed to arbitrary
> strings.
>
> All in all there does not seem to be a good reason to have a separate
> function shqq--quote-string. Any improvement that it has over
> shell-quote-argument should be folded into shell-quote-argument.

I have honestly no idea what you're talking about at this point.

Who ever talked about running files in the current directory?

No, the shell does not try to run the . file as a script, no matter what
PATH is.  (How is PATH even relevant to the topic?)


I'm not motivated to waste any more time on this discussion.  People
seem to be bringing up even weirder and weirder pseudo-reasons to reject
perfectly working code with well-defined semantics.  If you don't want
it, fine.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/19/2015 05:09 PM, Taylan Ulrich Bayırlı/Kammer wrote:
>
>> And I can live without the emacs-devel tyrants accepting my
>> contributions, but the situation is so outrageous right now that I'm
>> seriously considering pulling back all my current and future
>> contributions, until the development team changes (in spirit or
>> persons).
>
> But that is the main point of contributing a package to GNU ELPA: the
> emacs-devel "tyrants" review it, and then share (at least to some
> degree) the responsibility of supporting it.
>
> In general, that means accepting the collective opinions of
> emacs-devel. Or the head maintainer (not sure if John has accepted
> that position yet).
>
> You can also distribute the package through MELPA.

The collective "opinion" here seems to be that it's a good idea to
reject perfectly working code with well-defined semantics for made up
reasons.

I'm not interested in contributing to such a development community.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Dmitry Gutov]

On 10/19/2015 08:08 PM, Taylan Ulrich Bayırlı/Kammer wrote:

> The collective "opinion" here seems to be that it's a good idea to
> reject perfectly working code with well-defined semantics for made up
> reasons.

The community values maintainability and, as such, lack of code 
duplication. These concepts aren't particularly novel, or unusual.

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: Artur Malabarba <bruce.connor.am@gmail.com>,  Eli Zaretskii <eliz@gnu.org>,  emacs-devel <emacs-devel@gnu.org>
> Date: Mon, 19 Oct 2015 19:08:32 +0200
> 
> > But that is the main point of contributing a package to GNU ELPA: the
> > emacs-devel "tyrants" review it, and then share (at least to some
> > degree) the responsibility of supporting it.
> >
> > In general, that means accepting the collective opinions of
> > emacs-devel. Or the head maintainer (not sure if John has accepted
> > that position yet).
> >
> > You can also distribute the package through MELPA.
> 
> The collective "opinion" here seems to be that it's a good idea to
> reject perfectly working code with well-defined semantics for made up
> reasons.

Your code was not rejected, far from it.  We just asked for a small
change in it, that's all.

> I'm not interested in contributing to such a development community.

I'm very sorry to hear that.  I urge you to reconsider.

Re: [PATCH] Add shell-quasiquote.

[Paul Eggert]

Taylan Ulrich Bayırlı/Kammer wrote:

> Who ever talked about running files in the current directory?

It's normal to run files in the current directory. Any function that wants to 
run arbitrary shell commands needs to be able to do that. Concerns were raised 
about not being able to run arbitrary commands, so the topic is relevant.

> the shell does not try to run the . file as a script, no matter what
> PATH is.

No, the shell command ". ." tries to run the . file as a script, assuming that 
PATH starts with ".:".

 > How is PATH even relevant to the topic?)

PATH can be relevant if a shell command name lacks slashes.

Admittedly it's weird to try to run "." as a shell script, but what can I say? 
This thread already went down that rabbit hole when the idea of running a 
command named "if" was raised. And "." is not a unique case here; for example, 
shqq won't run an executable named "break" either.

Re: [PATCH] Add shell-quasiquote.

[Stephen J. Turnbull]

(Sorry if anyone gets dupes, I horked my VM and it improperly handled
Taylan's display name, which caused bounces from at least two MTAs.)

Taylan Ulrich Bayırlı/Kammer writes:

 > The collective "opinion" here seems to be that it's a good idea to
 > reject perfectly working code with well-defined semantics for made up
 > reasons.

The reasons aren't made-up.  Duplication of needed functionality is
excessive complexity.  Introduction of unneeded functionality is
excessive complexity.  Excessive complexity makes it harder to learn
Emacs Lisp and/or harder to understand new code.  Compare the horrible
hash that is Emacs Lisp with a cleaner language like Guile[1], and
the costs become clear.  As an exercise, try getting shell-quasiquote
into Guile and see if you fare better there.  (Maybe you already have,
in which case just say so, and what the result was. ;-)[2]

Nor is it obvious to me that shell-quasiquote is "perfectly working".
Shell quoting is a very fragile thing.  If something is quoted that
shouldn't be quoted, that is a bug.  Sure, you provide ",,", but
that's punting the problem back to the user.  Whether it's better to
punt on quoting or unquoting a given construct is a question of
balance -- calling it "perfect" is a judgment that depends on the
application and on the user.  You have every right to say that for
yourself, but not for Emacs, not even in your intended application.
Different arguments apply to whether `shqq' *always* quotes enough to
prevent code injection.

None of that means `shqq' won't turn out to be right thing for many
Emacs applications in the end.  But your argument so far reduces to
"it won't hurt and it works for me".  That is inappropriate on both
counts when requesting maintenance and distribution by the Emacs
Project (which is what inclusion in GNU ELPA means, even if you are
offering to do the maintenance yourself -- you could spend that time
on other services to the project).

If upon consideration you don't like that much better, then you should
find an alternative way to contribute, or an alternative community to
contribute to.  But you seem to misunderstand what the needs of the
Emacs Project are, and how that affects the community's evaluation of
your proposed contribution.  With a better understanding you may be
more willing to contribute on the community's terms (and make no
mistake, every community has its own terms).


Footnotes: 
[1]  Not entirely a fair comparison, of course, because much of the
yuckiness of Emacs is in the bundled applications.  But consider how
much functionality that is already in Emacs Lisp is duplicated by
seq.el, and you see the point.  There are reasons why seq.el cleared
the bar, but the duplication is a wart, even many warts, and it needed
to be justified.

[2]  Note that "quasiquote" already has a somewhat different meaning
in Emacs: "backtick quoting".

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> (Sorry if anyone gets dupes, I horked my VM and it improperly handled
> Taylan's display name, which caused bounces from at least two MTAs.)
>
> Taylan Ulrich Bayırlı/Kammer writes:
>
>  > The collective "opinion" here seems to be that it's a good idea to
>  > reject perfectly working code with well-defined semantics for made up
>  > reasons.
>
> The reasons aren't made-up.  Duplication of needed functionality is
> excessive complexity.  Introduction of unneeded functionality is
> excessive complexity.  Excessive complexity makes it harder to learn
> Emacs Lisp and/or harder to understand new code.  Compare the horrible
> hash that is Emacs Lisp with a cleaner language like Guile[1], and
> the costs become clear.  As an exercise, try getting shell-quasiquote
> into Guile and see if you fare better there.  (Maybe you already have,
> in which case just say so, and what the result was. ;-)[2]

There is no duplication of work because the two functions have differing
semantics.  I already pointed this out.

I don't see shell-quasiquote fitting into Guile very well (there aren't
all the different kinds of shell-command functions).  Guile also doesn't
have a direct analogue to ELPA.

> Nor is it obvious to me that shell-quasiquote is "perfectly working".
> Shell quoting is a very fragile thing.  If something is quoted that
> shouldn't be quoted, that is a bug.  Sure, you provide ",,", but
> that's punting the problem back to the user.  Whether it's better to
> punt on quoting or unquoting a given construct is a question of
> balance -- calling it "perfect" is a judgment that depends on the
> application and on the user.  You have every right to say that for
> yourself, but not for Emacs, not even in your intended application.
> Different arguments apply to whether `shqq' *always* quotes enough to
> prevent code injection.

It's documented precisely what shqq does, and it does that correctly.
(If not, please point it out.)

If anyone finds the semantics of shqq to be "overall wonky" (because it
lies in a funny place between `shell-command' and `call-process' so to
say, in terms of the semantics of the generated command) that would be a
different topic, which nobody brought up yet, and they brought up all
sorts of stuff.  Ironically, it would be a criticism I could actually
understand.

> None of that means `shqq' won't turn out to be right thing for many
> Emacs applications in the end.  But your argument so far reduces to
> "it won't hurt and it works for me".  That is inappropriate on both
> counts when requesting maintenance and distribution by the Emacs
> Project (which is what inclusion in GNU ELPA means, even if you are
> offering to do the maintenance yourself -- you could spend that time
> on other services to the project).
>
> If upon consideration you don't like that much better, then you should
> find an alternative way to contribute, or an alternative community to
> contribute to.  But you seem to misunderstand what the needs of the
> Emacs Project are, and how that affects the community's evaluation of
> your proposed contribution.  With a better understanding you may be
> more willing to contribute on the community's terms (and make no
> mistake, every community has its own terms).

Thanks for the friendliness but I wasted an absurd amount of time and
nerves on this trivial topic (a one-line function).  One could still
accept my patch to shell-quote-argument so that I can use that, or one
could accept the variant using the one-line alternative with correct
semantics, and I've explained in painfully excruciating detail why
either is necessary, but so far people were unwilling to accept that.
I'm not going to work on a third way unless someone has a very, very
good reason.

This tells me that this development community (or at least certain
people in it) don't want my contribution for inexplicable reasons, in
particular not technical reasons.  I do not see any possible sensible
definition of "the needs of the Emacs Project" that would lead to the
behavior I've seen from certain people here, so I disagree absolutely
that it could possibly be a misunderstanding on my side.


Moreover, I've been warned about emacs-devel by multiple people before,
and about a certain member of it in particular, and I did not believe it
could possibly be this bad.  Lest you think this is an isolated case.
Maybe the previous victims were less outspoken so the majority of the
community still doesn't see that there's a serious problem.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Taylan Ulrich Bayırlı/Kammer wrote:
>
>> Who ever talked about running files in the current directory?
>
> It's normal to run files in the current directory. Any function that
> wants to run arbitrary shell commands needs to be able to do
> that. Concerns were raised about not being able to run arbitrary
> commands, so the topic is relevant.
>
>> the shell does not try to run the . file as a script, no matter what
>> PATH is.
>
> No, the shell command ". ." tries to run the . file as a script,
> assuming that PATH starts with ".:".
>
>> How is PATH even relevant to the topic?)
>
> PATH can be relevant if a shell command name lacks slashes.
>
> Admittedly it's weird to try to run "." as a shell script, but what
> can I say? This thread already went down that rabbit hole when the
> idea of running a command named "if" was raised. And "." is not a
> unique case here; for example, shqq won't run an executable named
> "break" either.

(shell-command (shqq (./foo bar baz)))
/bin/bash: ./foo: No such file or directory

I don't see a problem.

(In real code, I would use the string "./foo" instead because I feel
uneasy relying on ./foo being a valid symbol.)

I said (shqq (if ...)) will try to call e.g. /bin/if.  I said that this
is maybe slightly more useful and consistent.  I believe I was fairly
clear in the implication that this is a trivial matter, and pointed out
explicitly that I will use shell-quote-argument once the real problem
with it is fixed.  I never said that any (shqq (x ...)) will try to look
up x in PATH no matter what x is, or any other things you seem to have
taken from what I said.


Apologies if my previous answer was too hasty or unclear, but please
understand that I'm already very irritated because of the non-resolution
of the bug report despite providing a patch solving the exact problem,
and because of the continued unwillingness to accept the code refusing
to use shell-quote-argument despite that the reasons for refusing to use
it were explained in detail.

Taylan

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> This tells me that this development community (or at least certain
> people in it) don't want my contribution for inexplicable reasons, in
> particular not technical reasons.  I do not see any possible sensible
> definition of "the needs of the Emacs Project" that would lead to the
> behavior I've seen from certain people here, so I disagree absolutely
> that it could possibly be a misunderstanding on my side.
>
>
> Moreover, I've been warned about emacs-devel by multiple people before,
> and about a certain member of it in particular, and I did not believe it
> could possibly be this bad.

You don't need to speak in riddles.  I am quite used to seeing my name
explicitly written in such contexts.

However, if you take the time and sort the replies according to their
authors and look at what each individual actually has been writing
rather than your current recollection, you'll find that your impression
of an ongoing attack is simply untenable.  Even if you focus on my
contributions to that thread (and I'm not really much of a regular or
typical here even though I have tried providing missing perspectives to
several recent discussions recently with, I hope, mostly constructive
results).

The impression of a concerted attack that you fancy comes more about by
several people sharing the same opinion and experience than by an
insider/outsider setting and your recollection gets carried away with
what you want to remember about the discussion.

Really, reread the stuff and in particular find the parts that you
remember as being particularly egregious.  You'll find that your
recollection is playing games with you more than the people on this list
are.

> Lest you think this is an isolated case.  Maybe the previous victims
> were less outspoken so the majority of the community still doesn't see
> that there's a serious problem.

If everybody tells you the same, chances are that "everybody is wrong
then" is not the whole truth.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[John Wiegley]

>>>>> David Kastrup <dak@gnu.org> writes:

> You don't need to speak in riddles. I am quite used to seeing my name
> explicitly written in such contexts.

I've found your contributions to be quite helpful on the whole, David.

Lately I've heard and read many things about emacs-devel's "culture" and how
it stifles newcomers. This is something to take seriously, but I don't think
the issue should be over-simplified just to find a place to put blame.

We're a lot of people. We have a lot of experiences. This is no one's full-
time job. We all communicate differently.

Given those truths: as soon as the number of people involved becomes >large,
any perception you choose to adopt of such a group will generally be true in
some ways, and false in several other ways.

Some of the concrete problems I've heard about that could be meaningfully
addressed are:

 1. Some patches die in the bug tracker. They get submitted; the authors
    respond to the criticism; but there is no closure. This gives people the
    impression that their efforts are being wasted on Emacs development, so
    they move elsewhere.

 2. Sometimes people can be abrasive. This isn't something you can solve by
    mandate, or by posting a code of conduct. It requires a willingness on
    the part of participants to assume the best of others, and not expect
    them to do all the work revealing it.

    There could be things we might do here, like making the list passively
    moderated so we can silence egregious posters. But I haven't seen
    anything yet to warrant this type of response.

 3. Newcomers don't understand our culture. If you've grown up in the fast-
    paced GitHub world of one button PRs and brief discussions on Twitter,
    the culture and pace of emacs-devel may well shock you. Some of us are
    OLD, and we like our lawns kid-free a goodly part of the time.

    Now that is no excuse for bad manners, but it does mean we don't just
    "hop to it" when a shiny toy comes along. Be patient, give us time. And
    maybe, if your patch is withering on the vine, remind someone?

I think we have good people, who pay attention to meaningful issues. Not
everything we do needs to be instantly appealing to those unfamiliar with our
history of development. But if it's needlessly off-putting, that should be
brought up and remedied too.

John

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

David Kastrup <dak@gnu.org> writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> This tells me that this development community (or at least certain
>> people in it) don't want my contribution for inexplicable reasons, in
>> particular not technical reasons.  I do not see any possible sensible
>> definition of "the needs of the Emacs Project" that would lead to the
>> behavior I've seen from certain people here, so I disagree absolutely
>> that it could possibly be a misunderstanding on my side.
>>
>>
>> Moreover, I've been warned about emacs-devel by multiple people before,
>> and about a certain member of it in particular, and I did not believe it
>> could possibly be this bad.
>
> You don't need to speak in riddles.  I am quite used to seeing my name
> explicitly written in such contexts.

This had nothing to do with you.  The bug report discussion hopefully
makes it clear who I'm talking about.  Sorry about the misunderstanding,
really.

> However, if you take the time and sort the replies according to their
> authors and look at what each individual actually has been writing
> rather than your current recollection, you'll find that your impression
> of an ongoing attack is simply untenable.  Even if you focus on my
> contributions to that thread (and I'm not really much of a regular or
> typical here even though I have tried providing missing perspectives to
> several recent discussions recently with, I hope, mostly constructive
> results).
>
> The impression of a concerted attack that you fancy comes more about by
> several people sharing the same opinion and experience than by an
> insider/outsider setting and your recollection gets carried away with
> what you want to remember about the discussion.
>
> Really, reread the stuff and in particular find the parts that you
> remember as being particularly egregious.  You'll find that your
> recollection is playing games with you more than the people on this list
> are.
>
>> Lest you think this is an isolated case.  Maybe the previous victims
>> were less outspoken so the majority of the community still doesn't see
>> that there's a serious problem.
>
> If everybody tells you the same, chances are that "everybody is wrong
> then" is not the whole truth.

Really sorry that I had you waste time writing all this, it really had
nothing to do with you. :-)

I could have been clearer in who I meant.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> David Kastrup <dak@gnu.org> writes:
>
>> You don't need to speak in riddles. I am quite used to seeing my name
>> explicitly written in such contexts.
>
> I've found your contributions to be quite helpful on the whole, David.
>
> Lately I've heard and read many things about emacs-devel's "culture" and how
> it stifles newcomers. This is something to take seriously, but I don't think
> the issue should be over-simplified just to find a place to put blame.
>
> We're a lot of people. We have a lot of experiences. This is no one's full-
> time job. We all communicate differently.
>
> Given those truths: as soon as the number of people involved becomes >large,
> any perception you choose to adopt of such a group will generally be true in
> some ways, and false in several other ways.
>
> Some of the concrete problems I've heard about that could be meaningfully
> addressed are:
>
>  1. Some patches die in the bug tracker. They get submitted; the authors
>     respond to the criticism; but there is no closure. This gives people the
>     impression that their efforts are being wasted on Emacs development, so
>     they move elsewhere.
>
>  2. Sometimes people can be abrasive. This isn't something you can solve by
>     mandate, or by posting a code of conduct. It requires a willingness on
>     the part of participants to assume the best of others, and not expect
>     them to do all the work revealing it.
>
>     There could be things we might do here, like making the list passively
>     moderated so we can silence egregious posters. But I haven't seen
>     anything yet to warrant this type of response.
>
>  3. Newcomers don't understand our culture. If you've grown up in the fast-
>     paced GitHub world of one button PRs and brief discussions on Twitter,
>     the culture and pace of emacs-devel may well shock you. Some of us are
>     OLD, and we like our lawns kid-free a goodly part of the time.
>
>     Now that is no excuse for bad manners, but it does mean we don't just
>     "hop to it" when a shiny toy comes along. Be patient, give us time. And
>     maybe, if your patch is withering on the vine, remind someone?
>
> I think we have good people, who pay attention to meaningful issues. Not
> everything we do needs to be instantly appealing to those unfamiliar with our
> history of development. But if it's needlessly off-putting, that should be
> brought up and remedied too.
>
> John

For those interested in this topic, please also read my response to
another mail by John on the bug#21702 thread.  I'm afraid there was a
big misunderstanding in, at least, the reasons for my frustration.

I don't know whether the points mentioned above (people being used to
*faster* paced communication than e-d) apply to other cases of
contributor frustration, but I suspect that it's a red herring.

Taylan

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> David Kastrup <dak@gnu.org> writes:
>
>> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>>
>>> This tells me that this development community (or at least certain
>>> people in it) don't want my contribution for inexplicable reasons, in
>>> particular not technical reasons.  I do not see any possible sensible
>>> definition of "the needs of the Emacs Project" that would lead to the
>>> behavior I've seen from certain people here, so I disagree absolutely
>>> that it could possibly be a misunderstanding on my side.
>>>
>>>
>>> Moreover, I've been warned about emacs-devel by multiple people before,
>>> and about a certain member of it in particular, and I did not believe it
>>> could possibly be this bad.
>>
>> You don't need to speak in riddles.  I am quite used to seeing my name
>> explicitly written in such contexts.
>
> This had nothing to do with you.  The bug report discussion hopefully
> makes it clear who I'm talking about.  Sorry about the misunderstanding,
> really.

Well, presumably Eli then.  If you digged through the Emacs developer
archives of the last 20 years or so, I think you'll easily find about
3000 mails from Eli amounting to "please don't break the MSDOS port of
Emacs gratuitously" and probably 1000 mails amounting to "please don't
sabotage right-to-left typesetting", many in threads of the "why should
we even care about _that_" variety.

In the end, Emacs is better in some respects due to at last someone
taking responsibility for caring about minority interests.  And since
Emacs serves so many interests, it has become second nature to most
developers when such objections are raised to just try addressing them
like everybody else does.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

David Kastrup <dak@gnu.org> writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> David Kastrup <dak@gnu.org> writes:
>>
>>> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>>>
>>>> This tells me that this development community (or at least certain
>>>> people in it) don't want my contribution for inexplicable reasons, in
>>>> particular not technical reasons.  I do not see any possible sensible
>>>> definition of "the needs of the Emacs Project" that would lead to the
>>>> behavior I've seen from certain people here, so I disagree absolutely
>>>> that it could possibly be a misunderstanding on my side.
>>>>
>>>>
>>>> Moreover, I've been warned about emacs-devel by multiple people before,
>>>> and about a certain member of it in particular, and I did not believe it
>>>> could possibly be this bad.
>>>
>>> You don't need to speak in riddles.  I am quite used to seeing my name
>>> explicitly written in such contexts.
>>
>> This had nothing to do with you.  The bug report discussion hopefully
>> makes it clear who I'm talking about.  Sorry about the misunderstanding,
>> really.
>
> Well, presumably Eli then.  If you digged through the Emacs developer
> archives of the last 20 years or so, I think you'll easily find about
> 3000 mails from Eli amounting to "please don't break the MSDOS port of
> Emacs gratuitously" and probably 1000 mails amounting to "please don't
> sabotage right-to-left typesetting", many in threads of the "why should
> we even care about _that_" variety.
>
> In the end, Emacs is better in some respects due to at last someone
> taking responsibility for caring about minority interests.  And since
> Emacs serves so many interests, it has become second nature to most
> developers when such objections are raised to just try addressing them
> like everybody else does.

I didn't break anything, explained why I could not support non-POSIX
without Eli's help, asked for proper help, drafted a patch where he
could have just filled the blanks in documentation to declare the
required safety guarantees for non-POSIX systems, but nope...

Re: [PATCH] Add shell-quasiquote.

[Nicolas Richard]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
> Apologies if my previous answer was too hasty or unclear, but please
> understand that I'm already very irritated because of the non-resolution
> of the bug report despite providing a patch solving the exact problem,
> and because of the continued unwillingness to accept the code refusing
> to use shell-quote-argument despite that the reasons for refusing to use
> it were explained in detail.

I have no plan in entering the discussion, just wanted to write down
explicitly this suggestion: I think

(defun shqq--quote-string (string)
  "Quote argument for any POSIX sh-compliant shell."
  (shell-quote-argument string))

would be good enough to please most people wrt this discussion. If that
includes yourself (I hope), perhaps we can do that and move on ?

Thanks,

-- 
Nico.

Becoming an Emacs contributor (was: [PATCH] Add shell-quasiquote.)

[Óscar Fuentes]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> David Kastrup <dak@gnu.org> writes:
>
>> You don't need to speak in riddles. I am quite used to seeing my name
>> explicitly written in such contexts.
>
> I've found your contributions to be quite helpful on the whole, David.
>
> Lately I've heard and read many things about emacs-devel's "culture" and how
> it stifles newcomers. This is something to take seriously, but I don't think
> the issue should be over-simplified just to find a place to put blame.
>
> We're a lot of people. We have a lot of experiences. This is no one's full-
> time job. We all communicate differently.
>
> Given those truths: as soon as the number of people involved becomes >large,
> any perception you choose to adopt of such a group will generally be true in
> some ways, and false in several other ways.
>
> Some of the concrete problems I've heard about that could be meaningfully
> addressed are:
>
>  1. Some patches die in the bug tracker. They get submitted; the authors
>     respond to the criticism; but there is no closure. This gives people the
>     impression that their efforts are being wasted on Emacs development, so
>     they move elsewhere.
>
>  2. Sometimes people can be abrasive. This isn't something you can solve by
>     mandate, or by posting a code of conduct. It requires a willingness on
>     the part of participants to assume the best of others, and not expect
>     them to do all the work revealing it.
>
>     There could be things we might do here, like making the list passively
>     moderated so we can silence egregious posters. But I haven't seen
>     anything yet to warrant this type of response.
>
>  3. Newcomers don't understand our culture. If you've grown up in the fast-
>     paced GitHub world of one button PRs and brief discussions on Twitter,
>     the culture and pace of emacs-devel may well shock you. Some of us are
>     OLD, and we like our lawns kid-free a goodly part of the time.
>
>     Now that is no excuse for bad manners, but it does mean we don't just
>     "hop to it" when a shiny toy comes along. Be patient, give us time. And
>     maybe, if your patch is withering on the vine, remind someone?
>
> I think we have good people, who pay attention to meaningful issues. Not
> everything we do needs to be instantly appealing to those unfamiliar with our
> history of development. But if it's needlessly off-putting, that should be
> brought up and remedied too.

You forgot *the* problem newcomers face with emacs-devel: bikeshedding.
Even the most trivial contribution can bring huge amounts of discussion,
mostly improductive. And what is productive, often has little real
value. The contributor is overwhelmed by minutia, hypothetical
(unspecified) corner cases, requests for extended features "because we
should completely cover what the user might need", complains about the
code doing too much (at the same time of the previous item.) And
misunderstandings, lots of misunderstandings, which is a huge problem
because some well-meaned top hackers here are overly argumentative. (See
how often emacs-devel or emacs-bugs hosts threads with hundreds of
messages.)

I've made just a few contributions to Emacs and my experience says that
it can be an exasperating process, draining lots of energy. Once you got
commit access and you are trusted to not ask for permission for
operating on certain areas, things turn to be much better, but even then
you confront discussions with other hackers about matters where no clear
criteria exists for setting the matter.

Emacs would benefit from a process that avoids those repetitive,
unproductive discussions that only end when one part resigns by
exhaustion, bringing in frustration.

I think that Stefan tried to do something about this, by encouraging
early inclussion of code, as soon as there was clear that the code is an
improvement for Emacs. In lots of cases, it was obvious that the code
was far from the optimum solution, but it was a positive trade-off.

We could create the figure of mentor, who takes care of a contribution
(singular) and advices the contributor until the code is good enough,
and then he makes sure it is committed. Other people could chime in on
the technical discussion, but the contributor only listens to the
mentor.

BTW, this has nothing to do with the parent thread, which I haven't
followed.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> "John Wiegley" <johnw@newartisans.com> writes:
>
> [...]
>
> For those interested in this topic, please also read my response to
> another mail by John on the bug#21702 thread.  I'm afraid there was a
> big misunderstanding in, at least, the reasons for my frustration.
>
> I don't know whether the points mentioned above (people being used to
> *faster* paced communication than e-d) apply to other cases of
> contributor frustration, but I suspect that it's a red herring.
>
> Taylan

Err, silliness, John took that discussion off the bug ML (sensibly).

Long story short, the problem is far from unresponsiveness.  The problem
is giving me the feeling that no matter how many times I raise the same
point on why I need a given change in shell-quote-argument or else
cannot use it, no matter how clearly I strain myself to explain the
logic, and even provide patches that address the issue, my points get
outright ignored, misunderstood (willfully?), irrelevant points raised
out of nowhere to argue against, the patches of course not used, and so
on and so forth.  All the while I get bluntly commanded to make a change
to my code that goes directly against the point I'm trying to raise the
whole time.

There is not a dismissive attitude towards my work altogether, but to my
very words and ideas.  I'm not a brainless code-editing machine to
follow the orders of emacs-devel, so if I point out a reason why I don't
want to make a given change, listen to it.  If I even propose a solution
that will make me able to make the given change, and even offer a patch,
actually consider it.


To elaborate...

In the first mail in this thread already, a function of mine which did
precisely what the documentation implied, and did that correctly, has
been outright called "wrong," and I was more or less commanded make a
change to my code for which there was a comment indicating that I
already considered the change and intentionally decided against it.  You
would expect some respect to one's intellect, therefore a simple request
of clarification or such if the reason for my decision is unclear, but
nope:

>> +;;; Like `shell-quote-argument', but much simpler in implementation.
>> +(defun shqq--quote-string (string)
>> +  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))
>
> It might be simpler, but it's wrong, because the result is only
> correct for Posix shells.
>
> Please do use shell-quote-argument instead.

(The documentation for the whole library mentioned that only POSIX is
supported, although not that comment.)

Not thinking much about it (this level of unintentional impoliteness is
daily course), I ignored that mistake in attitude, and briefly explained
the reason for not making the change outright:

> Hmm, I don't really want to take responsibility of my library being used
> with shells other than POSIX shells.  (The library could make that
> clearer and error on other systems.)

> How much can I rely on shell-quote-argument?  Can one fully rely on it
> being safe against code injection?

After that I was asked what sort of code injection I mean, which I
clarified.  I also clarified that I don't want to take responsibility of
my code being used on other systems, but that it's no problem if the
responsibility can be shared:

> I generally don't want to take responsibility of my code being used on
> non-GNU/non-POSIX systems, but if I can share the responsibility then
> that's fine.

> (let ((file-list (read where-ever)))
>   (shqq (cp -- ,@file-list some-place)))
> 
> That code is *guaranteed* to either copy the files in file-list to
> some-place, or error, so long as the argument quoting by shqq works
> well.  If it has a bug, then malicious input from where-ever may be able
> to execute arbitrary shell commands.
> 
> Is shell-quote-argument safe against such a thing?  My shqq-quote-string
> isn't exactly formally proven to be safe either, but its implementation
> is so simple it's fairly obvious that it doesn't contain bugs.

This was responded to with an assertion that I somehow share
responsibility over the whole Emacs code-base.  And more or less an
outright dismissal of the problem I explained:

> Please take a look at the implementation of shell-quote-argument.  It
> uses the same interfaces as your implementation, no more, no less.  If
> your implementation is safe, then so is shell-quote-argument.

(Which "interface"?  The two implementations differ entirely.  Was
"interface" meant as in function signature?  How is that relevant to
implementation quality, and what's the problem with clearly documenting
the safety guarantees offered by the interface?)


I could go on, but you're probably bored.  After that point, more people
join in with careless assertions that shell-quote-argument is surely
safe and can be relied on, shortly after which someone demonstrates an
injection attack on it when used with csh.  Even that doesn't convince
our folks, and an abrasive and dismissive attitude towards the problem
I'm pointing out continues.  The same thing repeats/continues on the bug
mailing list.

I feel kind of silly writing this mail, because it doesn't amount to
much more than rehashing what's already found few mails up in the
archive.  I can do little better than pointing and asking "don't you see
the problem here?!"

But maybe some people will look harder now, I hope.


And before someone thinks of making a nasty remark about over
sensitivity, feelings of entitlement, or else, I'd like to point out
that my frustration is rooted in part in people's unwillingness to
accept the importance of a code injection vulnerability.  And that's a
technical point.  It's precisely because I *don't* believe that Emacs
developers are idiots (as was suggested at some point) that I believe
the problem must instead have some social/behavioral aspect to it, like
a fundamental lack of belief in newcomers' ability of insight, or lack
of care in explaining why you think a mentioned problem is in fact not a
problem, or something of that sort.

Taylan

Re: Becoming an Emacs contributor

[Taylan Ulrich Bayırlı/Kammer]

Óscar Fuentes <ofv@wanadoo.es> writes:

> "John Wiegley" <johnw@newartisans.com> writes:
>
>>>>>>> David Kastrup <dak@gnu.org> writes:
>>
>>> You don't need to speak in riddles. I am quite used to seeing my name
>>> explicitly written in such contexts.
>>
>> I've found your contributions to be quite helpful on the whole, David.
>>
>> Lately I've heard and read many things about emacs-devel's "culture" and how
>> it stifles newcomers. This is something to take seriously, but I don't think
>> the issue should be over-simplified just to find a place to put blame.
>>
>> We're a lot of people. We have a lot of experiences. This is no one's full-
>> time job. We all communicate differently.
>>
>> Given those truths: as soon as the number of people involved becomes >large,
>> any perception you choose to adopt of such a group will generally be true in
>> some ways, and false in several other ways.
>>
>> Some of the concrete problems I've heard about that could be meaningfully
>> addressed are:
>>
>>  1. Some patches die in the bug tracker. They get submitted; the authors
>>     respond to the criticism; but there is no closure. This gives people the
>>     impression that their efforts are being wasted on Emacs development, so
>>     they move elsewhere.
>>
>>  2. Sometimes people can be abrasive. This isn't something you can solve by
>>     mandate, or by posting a code of conduct. It requires a willingness on
>>     the part of participants to assume the best of others, and not expect
>>     them to do all the work revealing it.
>>
>>     There could be things we might do here, like making the list passively
>>     moderated so we can silence egregious posters. But I haven't seen
>>     anything yet to warrant this type of response.
>>
>>  3. Newcomers don't understand our culture. If you've grown up in the fast-
>>     paced GitHub world of one button PRs and brief discussions on Twitter,
>>     the culture and pace of emacs-devel may well shock you. Some of us are
>>     OLD, and we like our lawns kid-free a goodly part of the time.
>>
>>     Now that is no excuse for bad manners, but it does mean we don't just
>>     "hop to it" when a shiny toy comes along. Be patient, give us time. And
>>     maybe, if your patch is withering on the vine, remind someone?
>>
>> I think we have good people, who pay attention to meaningful issues. Not
>> everything we do needs to be instantly appealing to those unfamiliar with our
>> history of development. But if it's needlessly off-putting, that should be
>> brought up and remedied too.
>
> You forgot *the* problem newcomers face with emacs-devel: bikeshedding.
> Even the most trivial contribution can bring huge amounts of discussion,
> mostly improductive. And what is productive, often has little real
> value. The contributor is overwhelmed by minutia, hypothetical
> (unspecified) corner cases, requests for extended features "because we
> should completely cover what the user might need", complains about the
> code doing too much (at the same time of the previous item.) And
> misunderstandings, lots of misunderstandings, which is a huge problem
> because some well-meaned top hackers here are overly argumentative. (See
> how often emacs-devel or emacs-bugs hosts threads with hundreds of
> messages.)
>
> I've made just a few contributions to Emacs and my experience says that
> it can be an exasperating process, draining lots of energy. Once you got
> commit access and you are trusted to not ask for permission for
> operating on certain areas, things turn to be much better, but even then
> you confront discussions with other hackers about matters where no clear
> criteria exists for setting the matter.
>
> Emacs would benefit from a process that avoids those repetitive,
> unproductive discussions that only end when one part resigns by
> exhaustion, bringing in frustration.
>
> I think that Stefan tried to do something about this, by encouraging
> early inclussion of code, as soon as there was clear that the code is an
> improvement for Emacs. In lots of cases, it was obvious that the code
> was far from the optimum solution, but it was a positive trade-off.
>
> We could create the figure of mentor, who takes care of a contribution
> (singular) and advices the contributor until the code is good enough,
> and then he makes sure it is committed. Other people could chime in on
> the technical discussion, but the contributor only listens to the
> mentor.
>
> BTW, this has nothing to do with the parent thread, which I haven't
> followed.

This actually sounds pretty similar to what happened in this thread in
some ways, although it differs in other ways, so thanks for the input!

Taylan

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Date: Tue, 20 Oct 2015 10:54:51 +0200
> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>,
> 	emacs-devel <emacs-devel@gnu.org>
> 
> I didn't break anything, explained why I could not support non-POSIX
> without Eli's help, asked for proper help, drafted a patch where he
> could have just filled the blanks in documentation to declare the
> required safety guarantees for non-POSIX systems, but nope...

This is not about non-Posix shells (although that aspect did start
this thread).  This is about using project-wide APIs for certain
standard jobs.  That should have been a no-brainer, because it makes
no sense to have several functions doing the same job in subtly
different ways.

So you were politely requested to call that function for quoting shell
arguments in your package.  Doing that is the only thing that stands
in the way of accepting your package on ELPA.  AFAIK, there were no
other comments.

If you think shell-quote-argument should be changed, feel free to
submit a patch proposal to that effect, and state there your reasons
for the changes.  If they are accepted, all Lisp programs in Emacs
that need to quote command arguments will work that way, and everybody
will win by having a better Emacs.

I cannot understand why you insist on tying your contribution with two
orthogonal issues: what and how should be quoted, and what should be
in the doc string.  By doing this, you prevent acceptance of your
package, which IMO is a pity.

Re: [PATCH] Add shell-quasiquote.

[Dmitry Gutov]

On 10/20/2015 01:16 PM, Nicolas Richard wrote:

> (defun shqq--quote-string (string)
>    "Quote argument for any POSIX sh-compliant shell."
>    (shell-quote-argument string))
>
> would be good enough to please most people wrt this discussion. If that
> includes yourself (I hope), perhaps we can do that and move on ?

Nice. :)

Re: [PATCH] Add shell-quasiquote.

[Paul Eggert]

Taylan Ulrich Bayırlı/Kammer wrote:
> (shell-command (shqq (./foo bar baz)))
> /bin/bash: ./foo: No such file or directory
>
> I don't see a problem.

Nor do I. And you can use the same technique to run "./break" or "/bin/break", 
should anyone be crazy enough to create a file with either name. The point is 
that shqq's quoting mechanism doesn't work any better than shell-quote-argument 
does for contrived examples like "break".

> I will use shell-quote-argument once the real problem with it is fixed.

No case has been presented that there's a significant problem with it. The 
argument that one might want to run an executable named "if" is, as you say, a 
trivial one, i.e., not significant.

Yes, there are problems if you use shell-quote-argument with weird shells that 
don't use standard quoting conventions, but that's also true for shqq's or for 
any other quoting function, so it's not a knock on shell-quote-argument's 
implementation.

Re: Becoming an Emacs contributor

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Date: Tue, 20 Oct 2015 14:56:08 +0200
> Cc: emacs-devel@gnu.org
> 
> This actually sounds pretty similar to what happened in this thread in
> some ways, although it differs in other ways, so thanks for the input!

You were given a single comment that required a one-line change in the
code.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Date: Tue, 20 Oct 2015 10:54:51 +0200
>> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>,
>> 	emacs-devel <emacs-devel@gnu.org>
>> 
>> I didn't break anything, explained why I could not support non-POSIX
>> without Eli's help, asked for proper help, drafted a patch where he
>> could have just filled the blanks in documentation to declare the
>> required safety guarantees for non-POSIX systems, but nope...
>
> This is not about non-Posix shells (although that aspect did start
> this thread).  This is about using project-wide APIs for certain
> standard jobs.  That should have been a no-brainer, because it makes
> no sense to have several functions doing the same job in subtly
> different ways.
>
> So you were politely requested to call that function for quoting shell
> arguments in your package.  Doing that is the only thing that stands
> in the way of accepting your package on ELPA.  AFAIK, there were no
> other comments.
>
> If you think shell-quote-argument should be changed, feel free to
> submit a patch proposal to that effect, and state there your reasons
> for the changes.  If they are accepted, all Lisp programs in Emacs
> that need to quote command arguments will work that way, and everybody
> will win by having a better Emacs.

I've already provided an extensive explanation of the problem with
shell-quote-argument, what the solution to that problem is, and provided
a patch to apply that solution.

The patch was turned down.  (By you.)

> I cannot understand why you insist on tying your contribution with two
> orthogonal issues: what and how should be quoted, and what should be
> in the doc string.  By doing this, you prevent acceptance of your
> package, which IMO is a pity.

I don't know what you mean with "what and how should be quoted."

The doc string can serve as a clear declaration of strict safety
guarantees that will make the function appropriate for my use case.
Until that's done, the function is not appropriate for my use case
because it does not declare the guarantees necessary for my use case.

In practical terms, as explained before, this means that someone editing
the function in the future may insert bugs into it which, from what I've
gathered from other posts in the thread, it indeed also had in the past.

It has also not been verified whether it's void of such bugs for systems
other than POSIX, which is why I left declaring that to you; my patches
were only adding the declaration of safety for POSIX, which I've made
very sure of and gladly take responsibility for.

All of these things I've already said before, multiple times, with
different wording, every time as clearly as I could.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/20/2015 01:16 PM, Nicolas Richard wrote:
>
>> (defun shqq--quote-string (string)
>>    "Quote argument for any POSIX sh-compliant shell."
>>    (shell-quote-argument string))
>>
>> would be good enough to please most people wrt this discussion. If that
>> includes yourself (I hope), perhaps we can do that and move on ?
>
> Nice. :)

Sorry for not replying to that earlier.

I'm afraid it's effectively the same thing as using shell-quote-argument
directly in my code.  It puts the responsibility on me, because if
shell-quote-argument breaks and I don't react fast enough to change
shqq--quote-string, it's my fault for having used a definition of
shqq--quote-string that was prone to breakage.


To clarify, all this "responsibility" stuff I'm talking about is not
about putting blame on people *after* a serious bug has happened.  It's
about APIs declaring their semantics (responsibilities) very precisely,
and programmers accordingly choosing the right API for their task, so
the responsibility of upholding a certain invariant is offloaded to the
implementation of that API.

Just a more pedantic way to look at the basic idea of abstraction
really, which is necessary here because of security concerns.

Responsibility of and blame on programmers serves as a less boring way
to explain it. :-)

Taylan

Re: Becoming an Emacs contributor (was: [PATCH] Add shell-quasiquote.)

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 5711 bytes --]

On the contrary, I had a very pleasant experience making my first
contribution to emacs core very recently.

As Emacs is used by people from varied fields of interest, a corner case
for one person might be the only a major use case for another.

This first contribution wasn't one-shot. I did not email a patch that got
committed right away. I had CC'd the primary committer of the package to
which I was contributing to. He automatically became my mentor and guided
me through multiple versions of my patch, while making it more robust and
meet every corner case we could collectively think of.

For newbie committers, I think that we need to be patient and listen to
what more experienced coders have to teach.

The code robustness needs to be checked not from just one person's
perspective.

TL;DR: My first contribution to emacs was a pleasant experience and I will
contribute more.





--
Kaushal Modi

On Tue, Oct 20, 2015 at 7:45 AM, Óscar Fuentes <ofv@wanadoo.es> wrote:

> "John Wiegley" <johnw@newartisans.com> writes:
>
> >>>>>> David Kastrup <dak@gnu.org> writes:
> >
> >> You don't need to speak in riddles. I am quite used to seeing my name
> >> explicitly written in such contexts.
> >
> > I've found your contributions to be quite helpful on the whole, David.
> >
> > Lately I've heard and read many things about emacs-devel's "culture" and
> how
> > it stifles newcomers. This is something to take seriously, but I don't
> think
> > the issue should be over-simplified just to find a place to put blame.
> >
> > We're a lot of people. We have a lot of experiences. This is no one's
> full-
> > time job. We all communicate differently.
> >
> > Given those truths: as soon as the number of people involved becomes
> >large,
> > any perception you choose to adopt of such a group will generally be
> true in
> > some ways, and false in several other ways.
> >
> > Some of the concrete problems I've heard about that could be meaningfully
> > addressed are:
> >
> >  1. Some patches die in the bug tracker. They get submitted; the authors
> >     respond to the criticism; but there is no closure. This gives people
> the
> >     impression that their efforts are being wasted on Emacs development,
> so
> >     they move elsewhere.
> >
> >  2. Sometimes people can be abrasive. This isn't something you can solve
> by
> >     mandate, or by posting a code of conduct. It requires a willingness
> on
> >     the part of participants to assume the best of others, and not expect
> >     them to do all the work revealing it.
> >
> >     There could be things we might do here, like making the list
> passively
> >     moderated so we can silence egregious posters. But I haven't seen
> >     anything yet to warrant this type of response.
> >
> >  3. Newcomers don't understand our culture. If you've grown up in the
> fast-
> >     paced GitHub world of one button PRs and brief discussions on
> Twitter,
> >     the culture and pace of emacs-devel may well shock you. Some of us
> are
> >     OLD, and we like our lawns kid-free a goodly part of the time.
> >
> >     Now that is no excuse for bad manners, but it does mean we don't just
> >     "hop to it" when a shiny toy comes along. Be patient, give us time.
> And
> >     maybe, if your patch is withering on the vine, remind someone?
> >
> > I think we have good people, who pay attention to meaningful issues. Not
> > everything we do needs to be instantly appealing to those unfamiliar
> with our
> > history of development. But if it's needlessly off-putting, that should
> be
> > brought up and remedied too.
>
> You forgot *the* problem newcomers face with emacs-devel: bikeshedding.
> Even the most trivial contribution can bring huge amounts of discussion,
> mostly improductive. And what is productive, often has little real
> value. The contributor is overwhelmed by minutia, hypothetical
> (unspecified) corner cases, requests for extended features "because we
> should completely cover what the user might need", complains about the
> code doing too much (at the same time of the previous item.) And
> misunderstandings, lots of misunderstandings, which is a huge problem
> because some well-meaned top hackers here are overly argumentative. (See
> how often emacs-devel or emacs-bugs hosts threads with hundreds of
> messages.)
>
> I've made just a few contributions to Emacs and my experience says that
> it can be an exasperating process, draining lots of energy. Once you got
> commit access and you are trusted to not ask for permission for
> operating on certain areas, things turn to be much better, but even then
> you confront discussions with other hackers about matters where no clear
> criteria exists for setting the matter.
>
> Emacs would benefit from a process that avoids those repetitive,
> unproductive discussions that only end when one part resigns by
> exhaustion, bringing in frustration.
>
> I think that Stefan tried to do something about this, by encouraging
> early inclussion of code, as soon as there was clear that the code is an
> improvement for Emacs. In lots of cases, it was obvious that the code
> was far from the optimum solution, but it was a positive trade-off.
>
> We could create the figure of mentor, who takes care of a contribution
> (singular) and advices the contributor until the code is good enough,
> and then he makes sure it is committed. Other people could chime in on
> the technical discussion, but the contributor only listens to the
> mentor.
>
> BTW, this has nothing to do with the parent thread, which I haven't
> followed.
>
>
>

[-- Attachment #2: Type: text/html, Size: 7970 bytes --]

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: emacs-devel@gnu.org
> Date: Tue, 20 Oct 2015 18:31:00 +0200
> 
> > This is not about non-Posix shells (although that aspect did start
> > this thread).  This is about using project-wide APIs for certain
> > standard jobs.  That should have been a no-brainer, because it makes
> > no sense to have several functions doing the same job in subtly
> > different ways.
> >
> > So you were politely requested to call that function for quoting shell
> > arguments in your package.  Doing that is the only thing that stands
> > in the way of accepting your package on ELPA.  AFAIK, there were no
> > other comments.
> >
> > If you think shell-quote-argument should be changed, feel free to
> > submit a patch proposal to that effect, and state there your reasons
> > for the changes.  If they are accepted, all Lisp programs in Emacs
> > that need to quote command arguments will work that way, and everybody
> > will win by having a better Emacs.
> 
> I've already provided an extensive explanation of the problem with
> shell-quote-argument, what the solution to that problem is, and provided
> a patch to apply that solution.
> 
> The patch was turned down.  (By you.)

That was about the documentation.  I understood, perhaps incorrectly,
that you also thought the code of the function needed some changes.
At least your alternative implementation quotes slightly differently,
e.g. it quotes even if the string doesn't need any quoting (because it
includes only characters from the Posix portable set).  It also quotes
'like this', whereas shell-quote-argument uses backslashes.

> > I cannot understand why you insist on tying your contribution with two
> > orthogonal issues: what and how should be quoted, and what should be
> > in the doc string.  By doing this, you prevent acceptance of your
> > package, which IMO is a pity.
> 
> I don't know what you mean with "what and how should be quoted."

See above; I hope now what I meant is clear.

> The doc string can serve as a clear declaration of strict safety
> guarantees that will make the function appropriate for my use case.
> Until that's done, the function is not appropriate for my use case
> because it does not declare the guarantees necessary for my use case.

Documentation cannot change what the code does.  If the function will
be appropriate after changing its doc string, it is appropriate now.

> In practical terms, as explained before, this means that someone editing
> the function in the future may insert bugs into it which, from what I've
> gathered from other posts in the thread, it indeed also had in the past.

Documentation cannot prevent such incidents.  However, we can make
this harder by adding tests for this function.  There already are 3
tests that use it, and we can add more if needed.  Patches to add such
tests are welcome.

The upshot is that it's a pity to hold off a package for reasons that
can and should be taken care of independently and orthogonally.

> It has also not been verified whether it's void of such bugs for systems
> other than POSIX, which is why I left declaring that to you; my patches
> were only adding the declaration of safety for POSIX, which I've made
> very sure of and gladly take responsibility for.

The test suite runs on all supported systems, including MS-Windows,
and the tests all succeed.  This function is quite old, so any
problems with it should have been reported long ago.  The issue with
an embedded newline doesn't exist on MS-Windows at all.

> All of these things I've already said before, multiple times, with
> different wording, every time as clearly as I could.

I'm still hoping you will change your mind.

Re: [PATCH] Add shell-quasiquote.

[Dmitry Gutov]

On 10/20/2015 07:41 PM, Taylan Ulrich Bayırlı/Kammer wrote:

> I'm afraid it's effectively the same thing as using shell-quote-argument
> directly in my code.  It puts the responsibility on me, because if
> shell-quote-argument breaks and I don't react fast enough to change
> shqq--quote-string, it's my fault for having used a definition of
> shqq--quote-string that was prone to breakage.

So it would be okay if shell-quote-argument breaks and thus makes major 
functionality in Emacs vulnerable, but your tiny function in its small 
package is safe and sound? That's a nice set of priorities.

Regarding responsibility, I repeat: contributing package to ELPA means 
that that the developers here share some of it.

And your function, speaking in security terms, adds to the attack 
surface, not subtracts from it.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Taylan Ulrich Bayırlı/Kammer wrote:
>> (shell-command (shqq (./foo bar baz)))
>> /bin/bash: ./foo: No such file or directory
>>
>> I don't see a problem.
>
> Nor do I. And you can use the same technique to run "./break" or
> "/bin/break", should anyone be crazy enough to create a file with
> either name. The point is that shqq's quoting mechanism doesn't work
> any better than shell-quote-argument does for contrived examples like
> "break".

I didn't say it would work better in that case.

>> I will use shell-quote-argument once the real problem with it is fixed.
>
> No case has been presented that there's a significant problem with
> it. The argument that one might want to run an executable named "if"
> is, as you say, a trivial one, i.e., not significant.

Yes, a serious case has been presented multiple times.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Paul Eggert]

Taylan Ulrich Bayırlı/Kammer wrote:
>> >No case has been presented that there's a significant problem with
>> >it. The argument that one might want to run an executable named "if"
>> >is, as you say, a trivial one, i.e., not significant.
> Yes, a serious case has been presented multiple times.

I must have missed it then, because all I remember are the cases (1) of running 
/bin/if (which is trivial and is not a realistic example), and (2) of 
installations with nonstandard shells (a problem that shqq--quote-string does 
not fix). It has been a long thread; quite possibly I missed something.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 20 Oct 2015 18:31:00 +0200
>> 
>> > This is not about non-Posix shells (although that aspect did start
>> > this thread).  This is about using project-wide APIs for certain
>> > standard jobs.  That should have been a no-brainer, because it makes
>> > no sense to have several functions doing the same job in subtly
>> > different ways.
>> >
>> > So you were politely requested to call that function for quoting shell
>> > arguments in your package.  Doing that is the only thing that stands
>> > in the way of accepting your package on ELPA.  AFAIK, there were no
>> > other comments.
>> >
>> > If you think shell-quote-argument should be changed, feel free to
>> > submit a patch proposal to that effect, and state there your reasons
>> > for the changes.  If they are accepted, all Lisp programs in Emacs
>> > that need to quote command arguments will work that way, and everybody
>> > will win by having a better Emacs.
>> 
>> I've already provided an extensive explanation of the problem with
>> shell-quote-argument, what the solution to that problem is, and provided
>> a patch to apply that solution.
>> 
>> The patch was turned down.  (By you.)
>
> That was about the documentation.  I understood, perhaps incorrectly,
> that you also thought the code of the function needed some changes.
> At least your alternative implementation quotes slightly differently,
> e.g. it quotes even if the string doesn't need any quoting (because it
> includes only characters from the Posix portable set).  It also quotes
> 'like this', whereas shell-quote-argument uses backslashes.

The bug report explicitly stated that the current implementation of
shell-quote-argument passes the strictest criterion of correctness, and
included a horrible stress-test script to demonstrate it, merely so I
could justify my documentation patch.

I think I'm trying to be much more cooperative than what you seem to
believe.

Please read my mails more carefully.  I usually spend a lot of time on
them to make them as clear and comprehensive as possible.  Tell me if
they get too "comprehensive," as in burdensome to read.

>> > I cannot understand why you insist on tying your contribution with two
>> > orthogonal issues: what and how should be quoted, and what should be
>> > in the doc string.  By doing this, you prevent acceptance of your
>> > package, which IMO is a pity.
>> 
>> I don't know what you mean with "what and how should be quoted."
>
> See above; I hope now what I meant is clear.

I see.  No, I never said the difference between how shell-quote-argument
and shqq--quote-string do things was significant.  When I found out
about the difference, I explicitly stated it was insignificant.

>> The doc string can serve as a clear declaration of strict safety
>> guarantees that will make the function appropriate for my use case.
>> Until that's done, the function is not appropriate for my use case
>> because it does not declare the guarantees necessary for my use case.
>
> Documentation cannot change what the code does.  If the function will
> be appropriate after changing its doc string, it is appropriate now.

Documentation can change what associated code will do in the future.
Sometimes, ensuring that code will remain appropriate in the future is
very important, as in the cases I explained.

>> In practical terms, as explained before, this means that someone editing
>> the function in the future may insert bugs into it which, from what I've
>> gathered from other posts in the thread, it indeed also had in the past.
>
> Documentation cannot prevent such incidents.  However, we can make
> this harder by adding tests for this function.  There already are 3
> tests that use it, and we can add more if needed.  Patches to add such
> tests are welcome.

Documentation can very well prevent such incidents, because in the case
where the semantics of a function is wonky, as is the case in quoting
strings for shell commands, it can be the documentation that makes the
difference of how carefully future programmers think of what changes
they're making to the code.

Indeed, tests allow mechanically verifiable invariants, which is better.
I might write such tests for shell-quote-argument in the future, but the
lack of a bigger improvement does not justify the rejection of a smaller
improvement.

> The upshot is that it's a pity to hold off a package for reasons that
> can and should be taken care of independently and orthogonally.
>
>> It has also not been verified whether it's void of such bugs for systems
>> other than POSIX, which is why I left declaring that to you; my patches
>> were only adding the declaration of safety for POSIX, which I've made
>> very sure of and gladly take responsibility for.
>
> The test suite runs on all supported systems, including MS-Windows,
> and the tests all succeed.  This function is quite old, so any
> problems with it should have been reported long ago.  The issue with
> an embedded newline doesn't exist on MS-Windows at all.

For how long did the embedded newline problem remain in the POSIX
variant, which is probably used by ten times as many people?

>> All of these things I've already said before, multiple times, with
>> different wording, every time as clearly as I could.
>
> I'm still hoping you will change your mind.

I have not been given any reason to change my mind.  Shoop da woop and
we're talking about unit testing all of a sudden, and my actual points
have once more been ignored.  Congratulations!

Taylan

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/20/2015 07:41 PM, Taylan Ulrich Bayırlı/Kammer wrote:
>
>> I'm afraid it's effectively the same thing as using shell-quote-argument
>> directly in my code.  It puts the responsibility on me, because if
>> shell-quote-argument breaks and I don't react fast enough to change
>> shqq--quote-string, it's my fault for having used a definition of
>> shqq--quote-string that was prone to breakage.
>
> So it would be okay if shell-quote-argument breaks and thus makes
> major functionality in Emacs vulnerable, but your tiny function in its
> small package is safe and sound? That's a nice set of priorities.

I've stress-tested shell-quote-argument to hell and back for POSIX
systems, and provided a documentation patch to hopefully prevent some
future bugs to it.

So what did you say again?

Re: Becoming an Emacs contributor

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Date: Tue, 20 Oct 2015 14:56:08 +0200
>> Cc: emacs-devel@gnu.org
>> 
>> This actually sounds pretty similar to what happened in this thread in
>> some ways, although it differs in other ways, so thanks for the input!
>
> You were given a single comment that required a one-line change in the
> code.

You were provided a patch that you only needed to apply.

I win!

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Taylan Ulrich Bayırlı/Kammer wrote:
>>> >No case has been presented that there's a significant problem with
>>> >it. The argument that one might want to run an executable named "if"
>>> >is, as you say, a trivial one, i.e., not significant.
>> Yes, a serious case has been presented multiple times.
>
> I must have missed it then, because all I remember are the cases (1)
> of running /bin/if (which is trivial and is not a realistic example),
> and (2) of installations with nonstandard shells (a problem that
> shqq--quote-string does not fix). It has been a long thread; quite
> possibly I missed something.

Yeah, you missed the part about risk of code injection. :-)

Taylan

Re: Becoming an Emacs contributor

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: ofv@wanadoo.es,  emacs-devel@gnu.org
> Date: Tue, 20 Oct 2015 19:32:40 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> >> Date: Tue, 20 Oct 2015 14:56:08 +0200
> >> Cc: emacs-devel@gnu.org
> >> 
> >> This actually sounds pretty similar to what happened in this thread in
> >> some ways, although it differs in other ways, so thanks for the input!
> >
> > You were given a single comment that required a one-line change in the
> > code.
> 
> You were provided a patch that you only needed to apply.

I applied the part of it that I thought was in order.

Re: [PATCH] Add shell-quasiquote.

[Dmitry Gutov]

On 10/20/2015 08:32 PM, Taylan Ulrich Bayırlı/Kammer wrote:

> I've stress-tested shell-quote-argument to hell and back for POSIX
> systems,

Do use it, then.

> So what did you say again?

Whether the documentation patch is accepted or not, you should really be 
worried more about shell-quote-argument being secure (now and in the 
future) than about one little package not being at risk.

Re: Becoming an Emacs contributor

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Cc: ofv@wanadoo.es,  emacs-devel@gnu.org
>> Date: Tue, 20 Oct 2015 19:32:40 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> >> Date: Tue, 20 Oct 2015 14:56:08 +0200
>> >> Cc: emacs-devel@gnu.org
>> >> 
>> >> This actually sounds pretty similar to what happened in this thread in
>> >> some ways, although it differs in other ways, so thanks for the input!
>> >
>> > You were given a single comment that required a one-line change in the
>> > code.
>> 
>> You were provided a patch that you only needed to apply.
>
> I applied the part of it that I thought was in order.

If you mean your own version of the patch then sorry but no, that was
missing the crucial bit as I tried to explain.

Taylan

Re: Becoming an Emacs contributor

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>>> Date: Tue, 20 Oct 2015 14:56:08 +0200
>>> Cc: emacs-devel@gnu.org
>>> 
>>> This actually sounds pretty similar to what happened in this thread in
>>> some ways, although it differs in other ways, so thanks for the input!
>>
>> You were given a single comment that required a one-line change in the
>> code.
>
> You were provided a patch that you only needed to apply.
>
> I win!

Mistaking this for a competition might explain some of this thread.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/20/2015 08:32 PM, Taylan Ulrich Bayırlı/Kammer wrote:
>
>> I've stress-tested shell-quote-argument to hell and back for POSIX
>> systems,
>
> Do use it, then.
>
>> So what did you say again?
>
> Whether the documentation patch is accepted or not, you should really
> be worried more about shell-quote-argument being secure (now and in
> the future) than about one little package not being at risk.

And that's *precisely* what the patch was trying to fix.  How about you
support me in getting it applied instead of searching for reasons to
blame me?

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Date: Tue, 20 Oct 2015 19:28:56 +0200
> Cc: emacs-devel@gnu.org
> 
> > I'm still hoping you will change your mind.
> 
> I have not been given any reason to change my mind.

I guess this about sums up this thread.  Too bad.

Re: [PATCH] Add shell-quasiquote.

[Dmitry Gutov]

On 10/20/2015 08:58 PM, Taylan Ulrich Bayırlı/Kammer wrote:

> And that's *precisely* what the patch was trying to fix.  How about you
> support me in getting it applied instead of searching for reasons to
> blame me?

I honestly don't have an opinion on the difference between your and 
Eli's versions. But if forced to choose, I would pick Eli's, purely on 
merits of reputation.

My point, though, is that issue is orthogonal to reusing the standard 
function.

If you re-read the discussion, you'll see that many others have 
expressed the same opinion. Of course, with the explicit goal to spite 
you personally, because that's important to us as a community.

Re: [PATCH] Add shell-quasiquote.

[Paul Eggert]

Taylan Ulrich Bayırlı/Kammer wrote:
>> I must have missed it then, because all I remember are the cases (1)
>> >of running /bin/if (which is trivial and is not a realistic example),
>> >and (2) of installations with nonstandard shells (a problem that
>> >shqq--quote-string does not fix). It has been a long thread; quite
>> >possibly I missed something.
> Yeah, you missed the part about risk of code injection.:-)

Code injection occurs because of (2), right? So it's not a risk that 
shqq--quote-string would put much of a dent in.

I thought the complaint was about shell-quote-argument's implementation. But if 
it's merely about its documentation, then perhaps we can reword it to address 
your concerns. I briefly looked at your most recent docstring proposal in 
Bug#21702 and I'm afraid it is is pretty wordy and is not technically correct. 
For example, (shell-quote-argument "\0") does not produce a string that will be 
parsed as one token whose value will be exactly that of shell-quote-argument's 
argument in any POSIX-conforming shell. This is because you can't put NUL 
characters into a command argument in POSIX.

It'd be better to have docstring wording that is shorter and conveys the gist of 
what shell-quote-argument is for, without going into a lot of technical detail 
that will bog down the reader and may well be wrong anyway. Details about what 
is "safe" and what "safe" means can go into the manual.

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Tue, 20 Oct 2015 21:11:05 +0300
> Cc: Paul Eggert <eggert@cs.ucla.edu>,
> 	Nicolas Richard <youngfrog@members.fsf.org>, emacs-devel@gnu.org
> 
> My point, though, is that issue is orthogonal to reusing the standard 
> function.

That's also my point, exactly.  Any code in Emacs should use standard
APIs, and if those APIs need to be fixed, they should be fixed
regardless.  But the need to be fixed does not mean the APIs should be
bypassed.

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 20 Oct 2015 11:12:23 -0700
> Cc: emacs-devel@gnu.org
> 
> It'd be better to have docstring wording that is shorter and conveys the gist of 
> what shell-quote-argument is for, without going into a lot of technical detail 
> that will bog down the reader and may well be wrong anyway. Details about what 
> is "safe" and what "safe" means can go into the manual.

Please feel free to suggest (or push) such changes, and thanks.

Re: Becoming an Emacs contributor

[Taylan Ulrich Bayırlı/Kammer]

David Kastrup <dak@gnu.org> writes:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>>>> Date: Tue, 20 Oct 2015 14:56:08 +0200
>>>> Cc: emacs-devel@gnu.org
>>>> 
>>>> This actually sounds pretty similar to what happened in this thread in
>>>> some ways, although it differs in other ways, so thanks for the input!
>>>
>>> You were given a single comment that required a one-line change in the
>>> code.
>>
>> You were provided a patch that you only needed to apply.
>>
>> I win!
>
> Mistaking this for a competition might explain some of this thread.

Please don't try to reach serious conclusions from a silly joke.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Taylan Ulrich Bayırlı/Kammer wrote:
>>> I must have missed it then, because all I remember are the cases (1)
>>> >of running /bin/if (which is trivial and is not a realistic example),
>>> >and (2) of installations with nonstandard shells (a problem that
>>> >shqq--quote-string does not fix). It has been a long thread; quite
>>> >possibly I missed something.
>> Yeah, you missed the part about risk of code injection.:-)
>
> Code injection occurs because of (2), right? So it's not a risk that
> shqq--quote-string would put much of a dent in.

Sorry, no, that's not the problem.


So after four days of incredibly tiresome repetition, people still don't
understand the basic issue.

I can't go on like this.

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/20/2015 08:58 PM, Taylan Ulrich Bayırlı/Kammer wrote:
>
>> And that's *precisely* what the patch was trying to fix.  How about you
>> support me in getting it applied instead of searching for reasons to
>> blame me?
>
> I honestly don't have an opinion on the difference between your and
> Eli's versions. But if forced to choose, I would pick Eli's, purely on
> merits of reputation.

Eli's patch does not fix the problem.

I'll leave it to you to guess what the problem is.  I've explained it in
detail multiple times over the last couple days here and in the bug
report.

Anyway, let's drop the topic.  I'll just stop trying to contribute to
Emacs through e-d and everything will be fine.

Taylan

Re: Becoming an Emacs contributor

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> David Kastrup <dak@gnu.org> writes:
>
>> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>>
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>>>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>>>>> Date: Tue, 20 Oct 2015 14:56:08 +0200
>>>>> Cc: emacs-devel@gnu.org
>>>>> 
>>>>> This actually sounds pretty similar to what happened in this
>>>>> thread in some ways, although it differs in other ways, so thanks
>>>>> for the input!
>>>>
>>>> You were given a single comment that required a one-line change in
>>>> the code.
>>>
>>> You were provided a patch that you only needed to apply.
>>>
>>> I win!
>>
>> Mistaking this for a competition might explain some of this thread.
>
> Please don't try to reach serious conclusions from a silly joke.

It wouldn't be a joke without a reference point.  And seriously, I don't
get the reference.  That you consider it obvious enough to let it stand
on its own means that you perceive this communication differently than
I do.

That does not necessarily mean that Eli doesn't get it.  But for one
thing, he _is_ similarly thick-headed as I am, and for another, if
I don't get it, it might mean that my fate is shared if not by Eli still
by a number of others who consequently will form an opinion of this
conversation that may differ from the one you consider self-evident.

Make of that what you will.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[Werner LEMBERG]

> I'll leave it to you to guess what the problem is.  I've explained
> it in detail multiple times over the last couple days here and in
> the bug report.

Well, I'm a complete noob regarding this topic, but I think there is a
problem with your perspective.

There are a couple of *very* experienced developers on this list,
having great a knowledge of not only Emacs but also of other Unix
tools, in particular problems with the various shells.  Especially
Paul is quite active here, providing many patches for gnulib,
coreutils, autoconf, and many other projects.

Your writing insinuates that that they all intentionally don't want to
understand what you are trying to say.  Has it ever come to your mind
that it is probably *still you* who failed to express the issue
clearly enough, inspite of your many attempts?  Or may there is an
`agree to disagree' situation you have to live with but obviously
cannot.


    Werner

Re: [PATCH] Add shell-quasiquote.

[Taylan Ulrich Bayırlı/Kammer]

Werner LEMBERG <wl@gnu.org> writes:

>> I'll leave it to you to guess what the problem is.  I've explained
>> it in detail multiple times over the last couple days here and in
>> the bug report.
>
> Well, I'm a complete noob regarding this topic, but I think there is a
> problem with your perspective.
>
> There are a couple of *very* experienced developers on this list,
> having great a knowledge of not only Emacs but also of other Unix
> tools, in particular problems with the various shells.  Especially
> Paul is quite active here, providing many patches for gnulib,
> coreutils, autoconf, and many other projects.
>
> Your writing insinuates that that they all intentionally don't want to
> understand what you are trying to say.  Has it ever come to your mind
> that it is probably *still you* who failed to express the issue
> clearly enough, inspite of your many attempts?  Or may there is an
> `agree to disagree' situation you have to live with but obviously
> cannot.

Thanks for your concern, and I agree it seems weird to think that the
problem is in such experienced developers, but (genuinely without any
intention of sarcasm in this sentence) I know that I'm not delusional or
anything of that sort, and a fairly simple and well-defined change I
requested seems to not have been understood after having explained it
many times, often with very careful and detailed wording, and the only
thing I can conclude from that is that I'm simply not being listened to.
(Keyword is "safety guarantees", if you're curious.)

What also gives me a bit of confidence is that I know at least two also
*very* experienced Emacs/Elisp developers, long-timers of the GNU and
Emacs communities, who have express a deep disdain towards emacs-devel.
And at least one other newcomer who I know also left e-d in a fit of
rage a while ago.  (Back then I read through the discussion that enraged
them and to be honest didn't understand it myself.  I think it was a
more involved topic than a mere safety guarantee, but maybe also the
nature of the over-arching problem here is such that it's mostly only
the "victim" who even sees the problem?  Not a psychologist here.)

Worse, there seems to be a big split in the Emacs community, and among
some parts of it, emacs-devel is subject to regular ridicule, which I
find disheartening, since it's *the* place in which the development
happens.  When I came here a couple days ago I tried to be as optimistic
as I could, and I'm generally pretty thick-skinned too, but my
experience in this thread felt like a sort of psychological torture. :-)
After that I stopped finding it funny to be jokingly called insane by
someone for even *wanting* my Elisp packages in GNU ELPA.


Among the swathes of people who feel disdain towards emacs-devel, I'm
probably just one of the few who are stupid enough to spend such amounts
of time and energy arguing, and/or blunt enough to just go forth and
claim that there is a very serious behavioral problem with some people.
I've probably also made an ass of myself in some ways, and most people
want to avoid that of course.  If it weren't for these reasons, one
would probably see threads like this happening more often on this list.

(I will strain myself not to argue any further by the way.  No need to
respond to this mail if you don't want.  I would rather stop now.)

Taylan

Re: [PATCH] Add shell-quasiquote.

[Werner LEMBERG]

> Thanks for your concern, and I agree it seems weird to think that
> the problem is in such experienced developers, but [...]

My advice is to put the issue aside, looking again in, say, a month or
two.  At least for me this helps a lot :-)


    Werner

Contributors and maintainers (Was: [PATCH] Add shell-quasiquote.)

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> That's also my point, exactly. Any code in Emacs should use standard APIs,
> and if those APIs need to be fixed, they should be fixed regardless. But the
> need to be fixed does not mean the APIs should be bypassed.

This message is not just to Taylan, but to others who wish to contribute to
Emacs in the future.

The argument transpiring between Taylan and Eli has attempted to paint a
picture wherein the proposed change is "obviously" right (as seen by the
submitter) and "obviously" unacceptable (as seen by the maintainer). This
clash has led to much heated debate.

Part of the debate seems to be a lack of appreciation of the difference
between contributors and maintainers. You see, it is not sufficient to have a
good idea, no matter how clear it is to its submitter. *We* maintain Emacs,
and so the change must satisfy *us*, no matter how thick our skulls may be. If
we ask for clarification that Wednesday follows Tuesday, either you provide us
with that clarification, or the change doesn't go in. Period.

Our work is done on a volunteer basis, and so we choose what we want to
support in the future, and what we don't. Like it or not, Eli is 100% correct
and right, as maintainer, to ask for clarifications how and when he sees fit
-- and to expect those clarification in a format he wants to see them in! This
will remain true until he steps down as maintainer, or someone else fills his
shoes.

No submitter can brow-beat us into accepting a patch because they think it is
"clear" or "right" or "obvious". This isn't how collaboration works in the
free software world. We decide who has commit rights, and we reserve the right
to reject and revert commits.

If anyone does not like this, be forewarned. Otherwise, please show Eli and
the other developers here the respect and deference they deserve, especially
in light of *how much time* they have given freely to the Emacs project. Are
they ideal individuals who always express themselves perfectly? Probably not.
But they are our maintainers, and if you can't respect them, you shouldn't be
contributing here. It will only frustrate you.

Lastly, if anyone is having persistent, negative experiences with some aspect
of the Emacs developer community, please approach me directly. My e-mail is
johnw@newartisans.com. It is my vehement interest that we find a successful
path for all involved, and I will work with anyone to help make this possible.

John

Re: [PATCH] Add shell-quasiquote.

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
>
> That's also my point, exactly.  Any code in Emacs should use standard
> APIs, and if those APIs need to be fixed, they should be fixed
> regardless.  But the need to be fixed does not mean the APIs should be
> bypassed.

Strictly speaking, Emacs doesn't *have* an API for "quote for POSIX
shells", so it's not being bypassed. The objection to the feature of
only supporting POSIX shells is substantive but has little to do with
bypassing an API.

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

Random832 <random832@fastmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> That's also my point, exactly.  Any code in Emacs should use standard
>> APIs, and if those APIs need to be fixed, they should be fixed
>> regardless.  But the need to be fixed does not mean the APIs should be
>> bypassed.
>
> Strictly speaking, Emacs doesn't *have* an API for "quote for POSIX
> shells", so it's not being bypassed. The objection to the feature of
> only supporting POSIX shells is substantive but has little to do with
> bypassing an API.

I don't call it a good-weather feature of a car when the wind-screen
wipers are broken.

-- 
David Kastrup

Re: Contributors and maintainers

[Taylan Ulrich Bayırlı/Kammer]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>> That's also my point, exactly. Any code in Emacs should use standard APIs,
>> and if those APIs need to be fixed, they should be fixed regardless. But the
>> need to be fixed does not mean the APIs should be bypassed.
>
> This message is not just to Taylan, but to others who wish to contribute to
> Emacs in the future.
>
> The argument transpiring between Taylan and Eli has attempted to paint a
> picture wherein the proposed change is "obviously" right (as seen by the
> submitter) and "obviously" unacceptable (as seen by the maintainer). This
> clash has led to much heated debate.
>
> Part of the debate seems to be a lack of appreciation of the difference
> between contributors and maintainers. You see, it is not sufficient to have a
> good idea, no matter how clear it is to its submitter. *We* maintain Emacs,
> and so the change must satisfy *us*, no matter how thick our skulls may be. If
> we ask for clarification that Wednesday follows Tuesday, either you provide us
> with that clarification, or the change doesn't go in. Period.
>
> Our work is done on a volunteer basis, and so we choose what we want to
> support in the future, and what we don't. Like it or not, Eli is 100% correct
> and right, as maintainer, to ask for clarifications how and when he sees fit
> -- and to expect those clarification in a format he wants to see them in! This
> will remain true until he steps down as maintainer, or someone else fills his
> shoes.
>
> No submitter can brow-beat us into accepting a patch because they think it is
> "clear" or "right" or "obvious". This isn't how collaboration works in the
> free software world. We decide who has commit rights, and we reserve the right
> to reject and revert commits.

I provided clarification several times.  It was ignored.

Let me list some different mails in which I repeated more or less the
same explanation with different wording:
https://lists.gnu.org/archive/html/emacs-devel/2015-10/msg01392.html
https://lists.gnu.org/archive/html/emacs-devel/2015-10/msg01401.html
https://lists.gnu.org/archive/html/emacs-devel/2015-10/msg01409.html
https://lists.gnu.org/archive/html/emacs-devel/2015-10/msg01415.html
https://lists.gnu.org/archive/html/emacs-devel/2015-10/msg01448.html
(maybe more, I didn't went through all)

One person got it and also repeated it in their words:
https://lists.gnu.org/archive/html/emacs-devel/2015-10/msg01464.html

And me again on the bug discussion:
http://lists.gnu.org/archive/html/bug-gnu-emacs/2015-10/msg00676.html
http://lists.gnu.org/archive/html/bug-gnu-emacs/2015-10/msg00698.html

That makes at least 7 times in which I repeated the same thing, and it
was ignored every time.

Some of those mails contain very detailed, careful explanations of the
issue, which I spent a lot of time on.  At least 2 or 3 are of that
nature.

Some other mails by me were also hinging on the same issue, mentioning
it implicitly if not explicitly.

I also provided a pair of patches (the second more elaborate) to solve
this problem I explained.  The nature of the patches (their contrast to
Eli's patch) should further explain the problem.

I hope this makes it clear why I'm outraged.  When I say something like
"I repeated myself a dozen times and was ignored every time," the
"dozen" in that sentence is, by now, actually literal.  That's absurd.

> If anyone does not like this, be forewarned. Otherwise, please show Eli and
> the other developers here the respect and deference they deserve, especially
> in light of *how much time* they have given freely to the Emacs project. Are
> they ideal individuals who always express themselves perfectly? Probably not.
> But they are our maintainers, and if you can't respect them, you shouldn't be
> contributing here. It will only frustrate you.

What I gather from being persistently ignored is that I'm receiving
absolutely *no* respect *at all* from most people here.  That is the one
and only reason I would start losing respect towards others.  The
detailed and polite explanations of my problem listed above hopefully
give a hint on which way the lack of respect primarily goes.

The lack of respect I'm receiving is *not* of the kind where someone is
being actively nasty, insulting, etc.  It's a kind where a person's very
voice is being denied, not even countered.  That's pretty grave.

> Lastly, if anyone is having persistent, negative experiences with some aspect
> of the Emacs developer community, please approach me directly. My e-mail is
> johnw@newartisans.com. It is my vehement interest that we find a successful
> path for all involved, and I will work with anyone to help make this possible.

I doubt most people who come to contribute code have much motivation to
work out basic social issues.  My feedback is probably the best you will
get, and I'm not saying it's good at all.

Most others will just leave the place immediately, or not even try
because they already saw in the archive or heard from others enough
horrible things about emacs-devel.

I hope this helps.

> John

Taylan

Re: Contributors and maintainers

[Werner LEMBERG]

>> No submitter can brow-beat us into accepting a patch because they
>> think it is "clear" or "right" or "obvious".  This isn't how
>> collaboration works in the free software world.  We decide who has
>> commit rights, and we reserve the right to reject and revert
>> commits.
> 
> I provided clarification several times.  It was ignored.

Certainly not.  You got *a lot* of replies.

> Let me list some different mails in which I repeated more or less
> the same explanation with different wording:

This is exactly the `agree to disagree' situation I've mentioned in a
previous e-mail.

> What I gather from being persistently ignored is that I'm receiving
> absolutely *no* respect *at all* from most people here.

Hey!  Reality check!  You *do* get replies!  And all of them are
polite – so no disrespect at all.  However, those replies are
obviously not what you want to hear.


    Werner

Re: Contributors and maintainers

[David Kastrup]

Werner LEMBERG <wl@gnu.org> writes:

>>> No submitter can brow-beat us into accepting a patch because they
>>> think it is "clear" or "right" or "obvious".  This isn't how
>>> collaboration works in the free software world.  We decide who has
>>> commit rights, and we reserve the right to reject and revert
>>> commits.
>> 
>> I provided clarification several times.  It was ignored.
>
> Certainly not.  You got *a lot* of replies.
>
>> Let me list some different mails in which I repeated more or less
>> the same explanation with different wording:
>
> This is exactly the `agree to disagree' situation I've mentioned in a
> previous e-mail.

Well, these days generally "discussion" is understood as everybody
repeating his opinion until most drop out, maybe a trickling down from
the culture of political debate, with a focus on scoring points rather
than extending one's views.

This mode of discussion tends to work rather bad in a closed round of
experts.  Repeating your point on the assumption that your opponent was
just too dumb to get it the first time gets old rather fast.  Instead of
making the same point over and over again and riling everybody including
yourself up in the process, you better try bringing up new facts or
considerations.  Everything else is only likely to affect the emotional
but not the factual result of the discussion.

While "everybody's glad that this is over and one will not meet ever
again" may be a somewhat emotionally conclusive resolution in substitute
for a convergence to factual agreement, it's not much of a basis for
ongoing work.

-- 
David Kastrup

Re: Contributors and maintainers

[Taylan Ulrich Bayırlı/Kammer]

[-- Attachment #1: Type: text/plain, Size: 4999 bytes --]

David Kastrup <dak@gnu.org> writes:

> Werner LEMBERG <wl@gnu.org> writes:
>
>>>> No submitter can brow-beat us into accepting a patch because they
>>>> think it is "clear" or "right" or "obvious".  This isn't how
>>>> collaboration works in the free software world.  We decide who has
>>>> commit rights, and we reserve the right to reject and revert
>>>> commits.
>>> 
>>> I provided clarification several times.  It was ignored.
>>
>> Certainly not.  You got *a lot* of replies.
>>
>>> Let me list some different mails in which I repeated more or less
>>> the same explanation with different wording:
>>
>> This is exactly the `agree to disagree' situation I've mentioned in a
>> previous e-mail.
>
> Well, these days generally "discussion" is understood as everybody
> repeating his opinion until most drop out, maybe a trickling down from
> the culture of political debate, with a focus on scoring points rather
> than extending one's views.
>
> This mode of discussion tends to work rather bad in a closed round of
> experts.  Repeating your point on the assumption that your opponent was
> just too dumb to get it the first time gets old rather fast.  Instead of
> making the same point over and over again and riling everybody including
> yourself up in the process, you better try bringing up new facts or
> considerations.  Everything else is only likely to affect the emotional
> but not the factual result of the discussion.
>
> While "everybody's glad that this is over and one will not meet ever
> again" may be a somewhat emotionally conclusive resolution in substitute
> for a convergence to factual agreement, it's not much of a basis for
> ongoing work.

That sounds like saying a discussion where people stick to a clear,
thoroughly and carefully explained point, and don't move on until it's
addressed are silly political discussions, and a discussion between
experts should rather look like one where everyone just jumps in and
brings up another unique perspective which fails to address what was
shortly brought up.

I have to disagree, and offer an alternative analysis by someone who's
not me and is quite a bit better at social issues than me.

    The usual approach on emacs-devel when dealing with something they
    don't like is to come up with random arguments, ignore
    counter-arguments, and move goalposts around because getting
    convinced by arguments is not something you do on the internet.

    From the glimpse I took, that's roughly what's happening there:
    They don't like the idea because gut feeling, so they nitpick
    irrelevancies and go off on tangents to support their gut feeling.

    ...

    [Them] Taylan, could you summarize your core issue here? Not what
    e-d is discussing, but what would you want to happen as the ideal
    outcome?

    [Me] Shell-quote-argument should cleanly document its semantics in
    a way that gives a security-aware programmer confidence that they
    can use the function without worrying about injection.

    [Them] That was the purpose of your [PATCH] thread?

    [Me] That's the summary of the shell-quote-argument bug report.
    The original [PATCH] thread just wants to get shqq into ELPA.

    [Them] Is shqq getting into ELPA?

    [Me] They could have taken it as-is and worked on the "duplication
    of code" (one line of code) later.  They want it neither with the
    one line of duplicated effort, nor do they want to address the
    shell-quote-argument bug report, and I don't want it in ELPA with
    potentially broken semantics.

    [Them] Now that does sound like a more appropriate summary of the
    problem there.  "The thread is about getting shqq into GNU ELPA,
    but it is being rejected on spurious grounds of a single line of
    code supposedly duplicating the intent of some existing piece of
    code."  The latter is the derail.

Indeed, the whole absurdity of the thread is perhaps best summarized by
the fact that it boils down to one line of code (and alternatively, a
single documentation string).


How about, *first* of all, the latest version of my ELPA patch gets
applied, so there is an *immediate* benefit to Emacs users.  Claiming
that a single line of duplicated code outweighs that would be absurd.

After that, emacs-devel can make whatever change they want to the
package.  My opinion is that it's unethical to whitewash potential
security issues, so I beg you to think hard about them and do what's
necessary to eliminate their possibility in shell-quote-argument.  The
best suggestion I've heard was improving unit tests for that, although a
stricter documentation would also be helpful in my opinion.  You're free
to ignore these suggestions an opinions, in which case I'm not
responsible even if you change shqq to use shell-quote-argument.

In any case, please accept the ELPA patch.  Holding it off any longer
would be absurd unless there's a *serious* issue with it.

Here it is so you don't need to dig it out from the older mails.

Thanks.


[-- Attachment #2: 0001-Add-shell-quasiquote.patch --]
[-- Type: text/x-diff, Size: 6233 bytes --]

From 276e3adc61b2f083b0348fd231a97feaa7017e36 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 <taylanbayirli@gmail.com>
Date: Sat, 17 Oct 2015 18:32:22 +0200
Subject: [PATCH 1/3] Add shell-quasiquote.

---
 packages/shell-quasiquote/shell-quasiquote.el | 151 ++++++++++++++++++++++++++
 1 file changed, 151 insertions(+)
 create mode 100644 packages/shell-quasiquote/shell-quasiquote.el

diff --git a/packages/shell-quasiquote/shell-quasiquote.el b/packages/shell-quasiquote/shell-quasiquote.el
new file mode 100644
index 0000000..1f18862
--- /dev/null
+++ b/packages/shell-quasiquote/shell-quasiquote.el
@@ -0,0 +1,151 @@
+;;; shell-quasiquote.el --- Turn s-expressions into shell command strings.
+
+;; Copyright (C) 2015  Free Software Foundation, Inc.
+
+;; Author: Taylan Ulrich Bayırlı/Kammer <taylanbayirli@gmail.com>
+;; Keywords: extensions, unix
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; "Shell quasiquote" -- turn s-expressions into POSIX shell command strings.
+;;
+;; Shells other than POSIX sh are not supported.
+;;
+;; Quoting is automatic and safe against injection.
+;;
+;;   (let ((file1 "file one")
+;;         (file2 "file two"))
+;;     (shqq (cp -r ,file1 ,file2 "My Files")))
+;;       => "cp -r 'file one' 'file two' 'My Files'"
+;;
+;; You can splice many arguments into place with ,@foo.
+;;
+;;   (let ((files (list "file one" "file two")))
+;;     (shqq (cp -r ,@files "My Files")))
+;;       => "cp -r 'file one' 'file two' 'My Files'"
+;;
+;; Note that the quoting disables a variety of shell expansions like ~/foo,
+;; $ENV_VAR, and e.g. {x..y} in GNU Bash.
+;;
+;; You can use ,,foo to escape the quoting.
+;;
+;;   (let ((files "file1 file2"))
+;;     (shqq (cp -r ,,files "My Files")))
+;;       => "cp -r file1 file2 'My Files'"
+;;
+;; And ,,@foo to splice and escape quoting.
+;;
+;;   (let* ((arglist '("-x 'foo bar' -y baz"))
+;;          (arglist (append arglist '("-z 'qux fux'"))))
+;;     (shqq (command ,,@arglist)))
+;;       => "command -x 'foo bar' -y baz -z 'qux fux'"
+;;
+;; Neat, eh?
+
+\f
+;;; Code:
+
+;;; We don't use `shell-quote-argument' because it doesn't provide any safety
+;;; guarantees, and this quotes shell keywords as well.
+(defun shqq--quote-string (string)
+  (concat "'" (replace-regexp-in-string "'" "'\\\\''" string) "'"))
+
+(defun shqq--atom-to-string (atom)
+  (cond
+   ((symbolp atom) (symbol-name atom))
+   ((stringp atom) atom)
+   ((numberp atom) (number-to-string atom))
+   (t (error "Bad shqq atom: %S" atom))))
+
+(defun shqq--quote-atom (atom)
+  (shqq--quote-string (shqq--atom-to-string atom)))
+
+(defun shqq--match-comma (form)
+  "Matches FORM against ,foo i.e. (\, foo) and returns foo.
+Returns nil if FORM didn't match.  You can't disambiguate between
+FORM matching ,nil and not matching."
+  (if (and (consp form)
+           (eq '\, (car form))
+           (consp (cdr form))
+           (null (cddr form)))
+      (cadr form)))
+
+(defun shqq--match-comma2 (form)
+  "Matches FORM against ,,foo i.e. (\, (\, foo)) and returns foo.
+Returns nil if FORM didn't match.  You can't disambiguate between
+FORM matching ,,nil and not matching."
+  (if (and (consp form)
+           (eq '\, (car form))
+           (consp (cdr form))
+           (null (cddr form)))
+      (shqq--match-comma (cadr form))))
+
+\f
+(defmacro shqq (parts)
+  "First, PARTS is turned into a list of strings.  For this,
+every element of PARTS must be one of:
+
+- a symbol, evaluating to its name,
+
+- a string, evaluating to itself,
+
+- a number, evaluating to its decimal representation,
+
+- \",expr\", where EXPR must evaluate to an atom that will be
+  interpreted according to the previous rules,
+
+- \",@list-expr\", where LIST-EXPR must evaluate to a list whose
+  elements will each be interpreted like the EXPR in an \",EXPR\"
+  form, and spliced into the list of strings,
+
+- \",,expr\", where EXPR is interpreted like in \",expr\",
+
+- or \",,@expr\", where EXPR is interpreted like in \",@expr\".
+
+In the resulting list of strings, all elements except the ones
+resulting from \",,expr\" and \",,@expr\" forms are quoted for
+shell grammar.
+
+Finally, the resulting list of strings is concatenated with
+separating spaces."
+  (let ((parts
+         (mapcar
+          (lambda (part)
+            (cond
+             ((atom part) (shqq--quote-atom part))
+             ;; We use the match-comma helpers because pcase can't match ,foo.
+             (t (pcase part
+                  ;; ,,foo i.e. (, (, foo))
+                  ((pred shqq--match-comma2)
+                   (shqq--match-comma2 part))
+                  ;; ,,@foo i.e. (, (,@ foo))
+                  ((and (pred shqq--match-comma)
+                        (let `,@,form (shqq--match-comma part)))
+                   `(mapconcat #'identity ,form " "))
+                  ;; ,foo
+                  ;; Insert redundant 'and x' to work around debbugs#18554.
+                  ((and x (pred shqq--match-comma))
+                   `(shqq--quote-atom ,(shqq--match-comma part)))
+                  ;; ,@foo
+                  (`,@,form
+                   `(mapconcat #'shqq--quote-atom ,form " "))
+                  (_
+                   (error "Bad shqq part: %S" part))))))
+          parts)))
+    `(mapconcat #'identity (list ,@parts) " ")))
+
+(provide 'shell-quasiquote)
+;;; shell-quasiquote.el ends here
-- 
2.5.0

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Tue, 20 Oct 2015 23:25:43 -0400
> 
> > That's also my point, exactly.  Any code in Emacs should use standard
> > APIs, and if those APIs need to be fixed, they should be fixed
> > regardless.  But the need to be fixed does not mean the APIs should be
> > bypassed.
> 
> Strictly speaking, Emacs doesn't *have* an API for "quote for POSIX
> shells"

Of course it does: it's called "shell-quote-argument".  It does
support Posix shells.

> The objection to the feature of only supporting POSIX shells is
> substantive but has little to do with bypassing an API.

That objection is not the primary consideration, it was just the first
thing that caught my eye (because a comment in the code explicitly
said so).  The primary consideration is the need to use established
interfaces for standard jobs.

Re: Contributors and maintainers

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Date: Wed, 21 Oct 2015 09:29:13 +0200
> 
> I provided clarification several times.  It was ignored.

No, it was not ignored.  It was disagreed with, which is something
entirely different.

> One person got it and also repeated it in their words:
> https://lists.gnu.org/archive/html/emacs-devel/2015-10/msg01464.html

After which I pushed a change that took care of the missing
information.

> And me again on the bug discussion:
> http://lists.gnu.org/archive/html/bug-gnu-emacs/2015-10/msg00676.html
> http://lists.gnu.org/archive/html/bug-gnu-emacs/2015-10/msg00698.html
> 
> That makes at least 7 times in which I repeated the same thing, and it
> was ignored every time.

No, it was not ignored.  It was repeatedly read, considered, and
_disagreed_ with.

> Some of those mails contain very detailed, careful explanations of the
> issue, which I spent a lot of time on.  At least 2 or 3 are of that
> nature.

And it took me a similarly considerable amount of time to re-read the
same explanations over and over again, and then provide a polite
response.  All that because you completely refused to accept a simple
comment that required you to make a change in a single line of code,
so that your package will use a standard Emacs API.

> I hope this makes it clear why I'm outraged.  When I say something like
> "I repeated myself a dozen times and was ignored every time," the
> "dozen" in that sentence is, by now, actually literal.  That's absurd.

You were NOT ignored.

> What I gather from being persistently ignored is that I'm receiving
> absolutely *no* respect *at all* from most people here.  That is the one
> and only reason I would start losing respect towards others.  The
> detailed and polite explanations of my problem listed above hopefully
> give a hint on which way the lack of respect primarily goes.

There's no disrespect, there never was.  Respecting an opinion does
not mean it must be accepted.  Rejecting an opinion or a patch doesn't
mean disrespect, it just means disagreement, in this case on purely
technical grounds.

> The lack of respect I'm receiving is *not* of the kind where someone is
> being actively nasty, insulting, etc.  It's a kind where a person's very
> voice is being denied, not even countered.  That's pretty grave.

We should be allowed to disagree and reject patches even if there are
no insults or obnoxious behavior on the part of the person who offers
an opinion or a patch.  Patches and opinions can be rejected on purely
technical grounds, not only on the grounds of nasty conduct.

IOW, we are not obliged to automatically accept patches just because
their submitter is well behaved.  We actually try to ignore his/her
behavior as best as we can, and consider the patches only on technical
merits.

> I doubt most people who come to contribute code have much motivation to
> work out basic social issues.  My feedback is probably the best you will
> get, and I'm not saying it's good at all.

You are wrong.  People do provide useful feedback here about these
issues.  Just yesterday we had such feedback from Óscar Fuentes.

> Most others will just leave the place immediately, or not even try
> because they already saw in the archive or heard from others enough
> horrible things about emacs-devel.

From whom did you hear horrible things about emacs-devel?  What
horrible things?

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Date: Tue, 20 Oct 2015 22:47:26 +0200
> Cc: emacs-devel@gnu.org
> 
> Thanks for your concern, and I agree it seems weird to think that the
> problem is in such experienced developers, but (genuinely without any
> intention of sarcasm in this sentence) I know that I'm not delusional or
> anything of that sort, and a fairly simple and well-defined change I
> requested seems to not have been understood after having explained it
> many times, often with very careful and detailed wording, and the only
> thing I can conclude from that is that I'm simply not being listened to.

There's no reason for you to assume such ill will around here.  A much
simpler explanation of what happened is that you were listened to and
heard, and we simply disagreed with your arguments, due to differences
in experience and perspective.

> What also gives me a bit of confidence is that I know at least two also
> *very* experienced Emacs/Elisp developers, long-timers of the GNU and
> Emacs communities, who have express a deep disdain towards emacs-devel.

If there are indeed such people, I encourage them to come here and
speak up for themselves.  Given the history of this thread, I'm not at
all sure we should hear their concerns through intermediaries.
Changes in leadership are a good time to try to make our procedures
and conduct better, so now is a good time for them to speak up.

Re: [PATCH] Add shell-quasiquote.

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Random832 <random832@fastmail.com>
>> Strictly speaking, Emacs doesn't *have* an API for "quote for POSIX
>> shells"
>
> Of course it does: it's called "shell-quote-argument".  It does
> support Posix shells.

It doesn't have a documented way for the caller to insist that the
string be quoted for a POSIX shell rather than the user's shell. In
particular, the (undocumented) mechanism that Tramp uses to get this
behavior will obviously break in the presence of (as recommended by the
documentation) an advised or completely overridden version of the
function put in place by a user who has an unusual shell.

Maybe what needs to be done is to have separate functions
shell-quote-argument-msdos, shell-quote-argument-nt,
shell-quote-argument-posix, and then have shell-quote-argument call
those based on the user's shell.

Re: Contributors and maintainers

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Date: Wed, 21 Oct 2015 14:03:33 +0200
> Cc: emacs-devel@gnu.org
> 
> I have to disagree, and offer an alternative analysis by someone who's
> not me and is quite a bit better at social issues than me.
> 
>     The usual approach on emacs-devel when dealing with something they
>     don't like is to come up with random arguments, ignore
>     counter-arguments, and move goalposts around because getting
>     convinced by arguments is not something you do on the internet.
> 
>     From the glimpse I took, that's roughly what's happening there:
>     They don't like the idea because gut feeling, so they nitpick
>     irrelevancies and go off on tangents to support their gut feeling.

I don't think things happen like that around here.  But the
description is so vague and devoid of any specific details that it's
easy to misinterpret.  I would first and foremost suspect some
misunderstanding.  After all, for most people here, myself included,
English is not their first language, so nuances could sometimes lead
to misunderstandings.  Can we have the person(s) who came up with this
description please speak up and point to specific discussions and
specific messages that could lead to such conclusions, and perhaps
suggest ways for changing the dynamics here away of that?

> How about, *first* of all, the latest version of my ELPA patch gets
> applied, so there is an *immediate* benefit to Emacs users.  Claiming
> that a single line of duplicated code outweighs that would be absurd.
> 
> After that, emacs-devel can make whatever change they want to the
> package.

Is that what this is about? that you don't want to make that change
yourself, but agree to someone else making it?  If so, then I think we
will gladly provide that service, and there are no more obstacles for
admitting the package.

Re: Contributors and maintainers

[Tassilo Horn]

Hi Taylan,

I haven't read the complete thread but as far as I understand, your
point is that you don't trust `shell-quote-argument' in that it doesn't
allow for running arbitrary code when given thoroughly prepared
arguments.  But you are confident that your shell-quasiquote.el package
doesn't allow for that in POSIX-compliant shells.  Is that correct?

But wouldn't it be better to contribute a set of ERT tests for
`shell-quote-argument' in order to increase the trust we can have in it
and possibly find and fix problems it might actually have?

After all, although you say that shell-quasiquote.el is only intended to
be used with POSIX shells, we/you have no control on where it'll
actually be used.

Bye,
Tassilo

Re: Contributors and maintainers

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Date: Wed, 21 Oct 2015 09:29:13 +0200
>> 
>> I provided clarification several times.  It was ignored.
>
> No, it was not ignored.  It was disagreed with, which is something
> entirely different.
>
>> One person got it and also repeated it in their words:
>> https://lists.gnu.org/archive/html/emacs-devel/2015-10/msg01464.html
>
> After which I pushed a change that took care of the missing
> information.
>
>> And me again on the bug discussion:
>> http://lists.gnu.org/archive/html/bug-gnu-emacs/2015-10/msg00676.html
>> http://lists.gnu.org/archive/html/bug-gnu-emacs/2015-10/msg00698.html
>> 
>> That makes at least 7 times in which I repeated the same thing, and it
>> was ignored every time.
>
> No, it was not ignored.  It was repeatedly read, considered, and
> _disagreed_ with.
>
>> Some of those mails contain very detailed, careful explanations of the
>> issue, which I spent a lot of time on.  At least 2 or 3 are of that
>> nature.
>
> And it took me a similarly considerable amount of time to re-read the
> same explanations over and over again, and then provide a polite
> response.  All that because you completely refused to accept a simple
> comment that required you to make a change in a single line of code,
> so that your package will use a standard Emacs API.
>
>> I hope this makes it clear why I'm outraged.  When I say something like
>> "I repeated myself a dozen times and was ignored every time," the
>> "dozen" in that sentence is, by now, actually literal.  That's absurd.
>
> You were NOT ignored.
>
>> What I gather from being persistently ignored is that I'm receiving
>> absolutely *no* respect *at all* from most people here.  That is the one
>> and only reason I would start losing respect towards others.  The
>> detailed and polite explanations of my problem listed above hopefully
>> give a hint on which way the lack of respect primarily goes.
>
> There's no disrespect, there never was.  Respecting an opinion does
> not mean it must be accepted.  Rejecting an opinion or a patch doesn't
> mean disrespect, it just means disagreement, in this case on purely
> technical grounds.
>
>> The lack of respect I'm receiving is *not* of the kind where someone is
>> being actively nasty, insulting, etc.  It's a kind where a person's very
>> voice is being denied, not even countered.  That's pretty grave.
>
> We should be allowed to disagree and reject patches even if there are
> no insults or obnoxious behavior on the part of the person who offers
> an opinion or a patch.  Patches and opinions can be rejected on purely
> technical grounds, not only on the grounds of nasty conduct.
>
> IOW, we are not obliged to automatically accept patches just because
> their submitter is well behaved.  We actually try to ignore his/her
> behavior as best as we can, and consider the patches only on technical
> merits.

(Trying to respond to all of the above as briefly as I can.)

Can you please show a previous quote by you which serves to show that
you understood the reason I did not want to use shell-quote-argument,
and where you directly addressed that exact reason (either with a change
to shell-quote-argument, *or* an explanation of why you disagree with
that exact reason)?

When you show such a quote, then maybe we can look at it and see how it
could be that you feel my concern has been addressed, yet I don't.

>> I doubt most people who come to contribute code have much motivation to
>> work out basic social issues.  My feedback is probably the best you will
>> get, and I'm not saying it's good at all.
>
> You are wrong.  People do provide useful feedback here about these
> issues.  Just yesterday we had such feedback from Óscar Fuentes.

Point.  That was pretty good feedback.

>> Most others will just leave the place immediately, or not even try
>> because they already saw in the archive or heard from others enough
>> horrible things about emacs-devel.
>
> From whom did you hear horrible things about emacs-devel?  What
> horrible things?

It would be bad to name them, but emacs-devel has become a running joke
among some groups which contain experienced programmers and long-timers
of GNU and Emacs, as well as young folks who could be future Emacs
developers.

I would love it if this could change, and if both those experienced
people and potential future developers could partake in emacs-devel.

Taylan

Re: [PATCH] Add shell-quasiquote.

[Michael Albinus]

Random832 <random832@fastmail.com> writes:

> It doesn't have a documented way for the caller to insist that the
> string be quoted for a POSIX shell rather than the user's shell. In
> particular, the (undocumented) mechanism that Tramp uses to get this
> behavior will obviously break in the presence of (as recommended by the
> documentation) an advised or completely overridden version of the
> function put in place by a user who has an unusual shell.
>
> Maybe what needs to be done is to have separate functions
> shell-quote-argument-msdos, shell-quote-argument-nt,
> shell-quote-argument-posix, and then have shell-quote-argument call
> those based on the user's shell.

I'm in favor of this proposal. Could be also an optional argument to
`shell-quote-argument', instead of four functions.

Tramp has to handle backwards compatibility then, but hey!, this is
daily business :-)

Best regards, Michael.

Re: Contributors and maintainers

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Date: Wed, 21 Oct 2015 14:03:33 +0200
>> Cc: emacs-devel@gnu.org
>> 
>> I have to disagree, and offer an alternative analysis by someone who's
>> not me and is quite a bit better at social issues than me.
>> 
>>     The usual approach on emacs-devel when dealing with something they
>>     don't like is to come up with random arguments, ignore
>>     counter-arguments, and move goalposts around because getting
>>     convinced by arguments is not something you do on the internet.
>> 
>>     From the glimpse I took, that's roughly what's happening there:
>>     They don't like the idea because gut feeling, so they nitpick
>>     irrelevancies and go off on tangents to support their gut feeling.
>
> I don't think things happen like that around here.

Well, I thought that the problem perceived here would have rather been
that the goalposts wouldn't budge than that they'd be moving around...

-- 
David Kastrup

Re: Contributors and maintainers

[Jay Belanger]

> There's no disrespect, there never was.  Respecting an opinion does
> not mean it must be accepted.  Rejecting an opinion or a patch doesn't
> mean disrespect, it just means disagreement, in this case on purely
> technical grounds.

There have been plenty of disagreements on emacs-devel, but I've never
seen an Emacs maintainer or Emacs bigwig (such as Eli) show any
disrespect to someone else on the list.  I have seen cases where others
showed disrespect to the bigwigs, but (impressively) it wasn't
returned.

Re: Contributors and maintainers

[David Kastrup]

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>
>>> Most others will just leave the place immediately, or not even try
>>> because they already saw in the archive or heard from others enough
>>> horrible things about emacs-devel.
>>
>> From whom did you hear horrible things about emacs-devel?  What
>> horrible things?
>
> It would be bad to name them,

Obviously.

> but emacs-devel has become a running joke among some groups which
> contain experienced programmers and long-timers of GNU and Emacs, as
> well as young folks who could be future Emacs developers.
>
> I would love it if this could change, and if both those experienced
> people and potential future developers could partake in emacs-devel.

I think you are setting your aspirations too high about reforming coding
standards as well as social interactions in Emacs development from the
ground up.  Even the acting Emacs maintainer tends not throw his weight
around in the manner you do, particularly not with matters of marginal
importance.  When working on a common project, one needs to pick one's
fights.

Instead of ELPA where packages are reviewed by Emacs developers, their
copyright assigned to the FSF and the Emacs developers are the fallback
for any maintenance issues, there is also the package archive MELPA with
a more customary submission procedure where the author retains copyright
and responsibility for packages of his.  It would seem that placing your
packages there might better correspond to the strong feelings you have
about keeping your code in exactly the style and state you desire it to
be without interference.

-- 
David Kastrup

Re: Contributors and maintainers

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Date: Wed, 21 Oct 2015 14:03:33 +0200
>> Cc: emacs-devel@gnu.org
>> 
>> I have to disagree, and offer an alternative analysis by someone who's
>> not me and is quite a bit better at social issues than me.
>> 
>>     The usual approach on emacs-devel when dealing with something they
>>     don't like is to come up with random arguments, ignore
>>     counter-arguments, and move goalposts around because getting
>>     convinced by arguments is not something you do on the internet.
>> 
>>     From the glimpse I took, that's roughly what's happening there:
>>     They don't like the idea because gut feeling, so they nitpick
>>     irrelevancies and go off on tangents to support their gut feeling.
>
> I don't think things happen like that around here.  But the
> description is so vague and devoid of any specific details that it's
> easy to misinterpret.  I would first and foremost suspect some
> misunderstanding.  After all, for most people here, myself included,
> English is not their first language, so nuances could sometimes lead
> to misunderstandings.  Can we have the person(s) who came up with this
> description please speak up and point to specific discussions and
> specific messages that could lead to such conclusions, and perhaps
> suggest ways for changing the dynamics here away of that?

I'm very sure they don't want to deal with this.

I can ask them if you want absolute confirmation.

>> How about, *first* of all, the latest version of my ELPA patch gets
>> applied, so there is an *immediate* benefit to Emacs users.  Claiming
>> that a single line of duplicated code outweighs that would be absurd.
>> 
>> After that, emacs-devel can make whatever change they want to the
>> package.
>
> Is that what this is about? that you don't want to make that change
> yourself, but agree to someone else making it?  If so, then I think we
> will gladly provide that service, and there are no more obstacles for
> admitting the package.

No it's not "what this is about."  Given the current lack of safety
guarantees in shell-quote-argument, I actively do not want it to be used
in shqq, or any other place where it may receive data from an untrusted
input source.

(The patch I provided for shell-quote-argument changes that, since it
formalizes safety guarantees at least in documentation.  Although, a
stricter solution like comprehensive unit tests would in turn be much
better than my mere documentation patch.)

But since we're in a deadlock regarding that topic, I say take my code
as is, first of all.

Then, having read all my reasoning and attempts to convince that
shell-quote-argument should not be used as is in places where it may
receive data from untrusted input sources, you're free to make any
changes to it.  I gave my suggestion on what change to avoid; if that
suggestion is disagreed with despite my best efforts to convince of it,
then obviously there's nothing more I can do.

Long story short, it's not that I "don't want to make that change
myself, but agree to someone else making it"; it's rather so that I
"disapprove of the change, thus won't do it myself, but ran out of
energy in trying to convince others not to do it."


One way or another, please as a first step apply the patch, since that
has clearly positive utility.  Maybe emacs-devel should indeed follow
Stefan's advice to merge first, then fix, unless someone insists that
there is a *serious* problem.  That might be a very good policy for
emacs-devel.  (With an obvious exception for a few initial feedback
round-trips.)

Taylan

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Wed, 21 Oct 2015 10:18:49 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> From: Random832 <random832@fastmail.com>
> >> Strictly speaking, Emacs doesn't *have* an API for "quote for POSIX
> >> shells"
> >
> > Of course it does: it's called "shell-quote-argument".  It does
> > support Posix shells.
> 
> It doesn't have a documented way for the caller to insist that the
> string be quoted for a POSIX shell rather than the user's shell.

On what OS would that distinction be important, and why?

> In particular, the (undocumented) mechanism that Tramp uses to get
> this behavior will obviously break in the presence of (as
> recommended by the documentation) an advised or completely
> overridden version of the function put in place by a user who has an
> unusual shell.

Maybe we should mention this subtlety with Tramp in the documentation
(I presume you mean the ELisp manual, because the doc string doesn't
mention overriding the function).

> Maybe what needs to be done is to have separate functions
> shell-quote-argument-msdos, shell-quote-argument-nt,
> shell-quote-argument-posix, and then have shell-quote-argument call
> those based on the user's shell.

I don't think anyone asked for exposing these internals.  I also have
trouble imagining situations when a user on a non-Posix platform would
need to call these functions directly: most users on those platforms
aren't even aware we go through cmdproxy, and only futz with with
their shell when they want the Cygwin Bash to be it, a situation Emacs
detects automatically.

But I won't object to this suggestion, if someone sends patches.  I'd
be interested to hear from others, though.

Re: Contributors and maintainers

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: emacs-devel@gnu.org
> Date: Wed, 21 Oct 2015 16:36:08 +0200
> 
> Can you please show a previous quote by you which serves to show that
> you understood the reason I did not want to use shell-quote-argument,
> and where you directly addressed that exact reason (either with a change
> to shell-quote-argument, *or* an explanation of why you disagree with
> that exact reason)?

Everything I wrote serves to show that.  I always responded to your
arguments.

> When you show such a quote, then maybe we can look at it and see how it
> could be that you feel my concern has been addressed, yet I don't.

Not only I feel that your concerns has been addressed, everyone else
here does.

> >> Most others will just leave the place immediately, or not even try
> >> because they already saw in the archive or heard from others enough
> >> horrible things about emacs-devel.
> >
> > From whom did you hear horrible things about emacs-devel?  What
> > horrible things?
> 
> It would be bad to name them, but emacs-devel has become a running joke
> among some groups which contain experienced programmers and long-timers
> of GNU and Emacs, as well as young folks who could be future Emacs
> developers.

They should come here and talk, if they want any change.

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> From: Random832 <random832@fastmail.com>

>> Maybe what needs to be done is to have separate functions
>> shell-quote-argument-msdos, shell-quote-argument-nt,
>> shell-quote-argument-posix, and then have shell-quote-argument call
>> those based on the user's shell.
>
> I don't think anyone asked for exposing these internals.  I also have
> trouble imagining situations when a user on a non-Posix platform would
> need to call these functions directly: most users on those platforms
> aren't even aware we go through cmdproxy, and only futz with with
> their shell when they want the Cygwin Bash to be it, a situation Emacs
> detects automatically.
>
> But I won't object to this suggestion, if someone sends patches.  I'd
> be interested to hear from others, though.

I don't think package authors as a rule _ever_ bother a lot about what
kind of shell might run their commands.  Most external utilities and
Free Software programs of some relevance, including but not restricted
to GNU utilities, are available on a wide variety of both UNIX-like and
unalike systems.

More often than not shell-quote-argument will be used _exactly_ in order
to save the package author from having to bother about the actual shell
in question.  The best way to do that would be a quoting strategy that
works on all systems, the next best autodetection, possibly with some
optionable aid when the system in question is not necessary the local
one (I think we had some expedient for executing helper commands like
decompressors remotely but currently can't find anything.  At any rate,
specifying a local cwd for expanding a shell argument suitable for
execution in the target location or let-binding default-directory
suitably around the call of shell-quote-argument might be appropriate,
though the latter, not being expected behavior for shell-quote-argument
currently, might be problematic).

At any rate, I'm afraid that providing _separate_ shell-quote-argument
functions will encourage people to create programs that stop working
when tramping/communicating/ftping/whatever to a system of different
kind.

-- 
David Kastrup

Re: Contributors and maintainers

[John Wiegley]

>>>>> Tassilo Horn <tsdh@gnu.org> writes:

> But wouldn't it be better to contribute a set of ERT tests for
> `shell-quote-argument' in order to increase the trust we can have in it and
> possibly find and fix problems it might actually have?

I've asked for this several times. I'm still waiting on a response, even if
it's just, "I don't want to." Which is odd given that one of the central
complaints is that the submitter feels ignored.

I'll ask again: Taylan, would you be willing to provide us with tests to
demonstrate the flaws in `shell-quote-argument'?

John

Re: Contributors and maintainers

[John Wiegley]

>>>>> Taylan Ulrich "Bayırlı/Kammer" <taylanbayirli@gmail.com> writes:

> The lack of respect I'm receiving is *not* of the kind where someone is
> being actively nasty, insulting, etc. It's a kind where a person's very
> voice is being denied, not even countered. That's pretty grave.

Taylan, I don't understand. Looking at emacs-devel and the bug tracker, I see
that many people have responded to you. Here are the reply counts:

    26 Eli Zaretskii
     9 Paul Eggert
     9 Dmitry Gutov
     7 Random832
     3 Michael Albinu
     3 John Wiegley
     3 David Kastrup
     3 Artur Malabarb
     1 Stephen J. Tur
     1 Nicolas Richar
     1 Daniel Colasci

Eli has taken the time to write back to you 26 times! This doesn't even count
the sub-threads where we discuss meta issues not directly related to your
issue.

Can you share with me what your real concern is? Do you worry that Emacs is
not secure enough? Is their an active threat of some kind in your environment?
Perhaps there's something about your use case we're not seeing, that would
explain the greater importance of this issue to you.

We've been living with the current shell-quote-argument for literally
*decades*, which might explain why we're not instantly ready to make changes
-- even though Eli has made a change to the docstring at your request. How
does that constitute "no response"? I am confused.

Also, it does not help to reiterate how clear and cogent your arguments have
been. Until we both agree, "clarity" and "cogency" have not been achieved.
These attributes must exist *between* disputants; they cannot be determined by
one side alone. We have all been working to achieve clarity, but I fear this
has been misunderstood as a stubborn rejection of your ideas.

John

Re: [PATCH] Add shell-quasiquote.

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Random832 <random832@fastmail.com>
>> It doesn't have a documented way for the caller to insist that the
>> string be quoted for a POSIX shell rather than the user's shell.
>
> On what OS would that distinction be important, and why?

Any OS which may have both a POSIX shell that a script may want to
execute and a non-POSIX shell that is the user's shell. So basically all
of them, especially if support for more non-POSIX shells such as csh,
rc, scsh, fish, tclsh, is added in the future - or if a user's
configuration supports them in the present by replacing or advising the
function.

It would mainly be useful in the presence of a broader mechanism, which
doesn't exist yet, for executing POSIX shell scripts regardless of the
user's interactive shell.

> (I presume you mean the ELisp manual, because the doc string doesn't
> mention overriding the function).

Yes, I meant (info "(ELisp)Shell Arguments"), sorry.

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Random832 <random832@fastmail.com>,  emacs-devel@gnu.org
> Date: Wed, 21 Oct 2015 18:37:16 +0200
> 
> I don't think package authors as a rule _ever_ bother a lot about what
> kind of shell might run their commands.  Most external utilities and
> Free Software programs of some relevance, including but not restricted
> to GNU utilities, are available on a wide variety of both UNIX-like and
> unalike systems.

I agree.

> More often than not shell-quote-argument will be used _exactly_ in order
> to save the package author from having to bother about the actual shell
> in question.  The best way to do that would be a quoting strategy that
> works on all systems, the next best autodetection, possibly with some
> optionable aid when the system in question is not necessary the local
> one (I think we had some expedient for executing helper commands like
> decompressors remotely but currently can't find anything.

The function shell-quote-argument will not go away under this
suggestion (IIUC), it will still do what it does today.  The
suggestion is to provide a few separate functions that are tailored to
specific shells, for those use cases that might need that.  One such
use case is Tramp, which must work with shells that are not
necessarily identical to the system shell of the locals OS.

> At any rate, I'm afraid that providing _separate_ shell-quote-argument
> functions will encourage people to create programs that stop working
> when tramping/communicating/ftping/whatever to a system of different
> kind.

That's a danger, yes.  We will have to consider it, and stand guard if
needed.

Re: Contributors and maintainers

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Cc: emacs-devel@gnu.org
>> Date: Wed, 21 Oct 2015 16:36:08 +0200
>> 
>> Can you please show a previous quote by you which serves to show that
>> you understood the reason I did not want to use shell-quote-argument,
>> and where you directly addressed that exact reason (either with a change
>> to shell-quote-argument, *or* an explanation of why you disagree with
>> that exact reason)?
>
> Everything I wrote serves to show that.  I always responded to your
> arguments.
>
>> When you show such a quote, then maybe we can look at it and see how it
>> could be that you feel my concern has been addressed, yet I don't.
>
> Not only I feel that your concerns has been addressed, everyone else
> here does.

It would be great if you could provide a quote like I mentioned.
Diverting the topic is not addressing one's concerns.

Taylan

Re: Contributors and maintainers

[Taylan Ulrich Bayırlı/Kammer]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Tassilo Horn <tsdh@gnu.org> writes:
>
>> But wouldn't it be better to contribute a set of ERT tests for
>> `shell-quote-argument' in order to increase the trust we can have in it and
>> possibly find and fix problems it might actually have?
>
> I've asked for this several times. I'm still waiting on a response, even if
> it's just, "I don't want to." Which is odd given that one of the central
> complaints is that the submitter feels ignored.
>
> I'll ask again: Taylan, would you be willing to provide us with tests to
> demonstrate the flaws in `shell-quote-argument'?

Sorry that I'm not answering everything in this utter chaos, I genuinely
forget.

I may work on that in the future.  Currently I'm way too agitated to
want to spend any more effort on the code, although sorting out the
social problems might be very beneficial, but that doesn't seem to go
anywhere either...

Taylan

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Wed, 21 Oct 2015 13:06:46 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> From: Random832 <random832@fastmail.com>
> >> It doesn't have a documented way for the caller to insist that the
> >> string be quoted for a POSIX shell rather than the user's shell.
> >
> > On what OS would that distinction be important, and why?
> 
> Any OS which may have both a POSIX shell that a script may want to
> execute and a non-POSIX shell that is the user's shell. So basically all
> of them, especially if support for more non-POSIX shells such as csh,
> rc, scsh, fish, tclsh, is added in the future - or if a user's
> configuration supports them in the present by replacing or advising the
> function.

First, do csh and the rest really non-Posix?  I wonder.  I always
understood "Posix shells" as a short for "any shell on a Posix host".
Is that incorrect?  In what way are those "non-Posix"?

Next, I could see why users on a Posix host might want to execute some
commands with a particular non-default shell.  I don't see why Emacs
packages, perhaps with a sole exception of Tramp, would need that.

On MS-Windows, using a Posix shell needs to customize variables like
explicit-shell-file-name, and Emacs detects that automatically.

> It would mainly be useful in the presence of a broader mechanism, which
> doesn't exist yet, for executing POSIX shell scripts regardless of the
> user's interactive shell.

On Posix hosts?  I thought that was automatic, since each script says
what interpreter should run it in its "shebang" line.  Right?

On MS-Windows, we would need to write code that parses the shebang
line, and then looks for an appropriate interpreter, probably on
PATH.  But that'd be a w32-only feature.

Re: Contributors and maintainers

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: emacs-devel@gnu.org
> Date: Wed, 21 Oct 2015 19:22:10 +0200
> 
> >> Can you please show a previous quote by you which serves to show that
> >> you understood the reason I did not want to use shell-quote-argument,
> >> and where you directly addressed that exact reason (either with a change
> >> to shell-quote-argument, *or* an explanation of why you disagree with
> >> that exact reason)?
> >
> > Everything I wrote serves to show that.  I always responded to your
> > arguments.
> >
> >> When you show such a quote, then maybe we can look at it and see how it
> >> could be that you feel my concern has been addressed, yet I don't.
> >
> > Not only I feel that your concerns has been addressed, everyone else
> > here does.
> 
> It would be great if you could provide a quote like I mentioned.

OK.  Here are some:

Quote #1:

  > > On POSIX shells, shell-quote-argument is just as safe as
  > > shqq--quote-string, and on non-POSIX shells it works better. So it's a
  > > win, in both readability and in portability, to use
  > > shell-quote-argument.
  > 
  > Fixing it does not seem easy at all given I can't trust
  > shell-quote-argument.

  You can trust it.

  > And please be realistic in the amount of trust we can put on the
  > complicated implementations for non-Unix shells.  I can't judge them
  > myself since I don't know the syntax of those shells at all.  Does
  > anyone here know their syntax comprehensively, or checked the
  > implementation against the documentation of those shells?

  Yes, we do.  Yes, we have.

Quote #2:

  > > It might be simpler, but it's wrong, because the result is only
  > > correct for Posix shells.
  > >
  > > Please do use shell-quote-argument instead.
  > 
  > It's also simpler than the POSIX section of shell-quote-argument.

  Simpler doesn't mean correct.

  > (defun shell-quote-argument (argument)
  >   [...] (cond [...] (t
  >     (if (equal argument "")
  >         "''"
  >       ;; Quote everything except POSIX filename characters.
  >       ;; This should be safe enough even for really weird shells.
  >       (replace-regexp-in-string
  >        "\n" "'\n'"
  >        (replace-regexp-in-string "[^-0-9a-zA-Z_./\n]" "\\\\\\&" argument))))))
  > 
  > I wonder what "really weird shells" this refers to?

  The set of characters special to an arbitrary shell is not known in
  advance.

  > Certainly not csh, the mechanism it uses for newlines doesn't work
  > there.

  What did you try that didn't work with csh?

Quote #3:

  > Quoting RMS, coincidentally from a couple days ago:
  > 
  >     The policy is non-GNU systems are secondary, and lower priority than
  >     the GNU system, but we are glad to include support for them in GNU
  >     packages if users contribute the necessary code -- provided that
  >     code isn't a maintenance problem for us.
  > 
  >     The maintenainers of any particular package are the ones who judge
  >     whether that code is a maintenance problem, since they are the ones
  >     it would be a problem for.

  I don't see how this is relevant for the issue at hand, since the
  necessary code (the shell-quote-argument function) was already
  contributed to Emacs years ago, and is used in many places in core
  Emacs.  There's no extra effort needed to support more platforms, just
  replace one function with another.

  > I generally don't want to take responsibility of my code being used on
  > non-GNU/non-POSIX systems, but if I can share the responsibility then
  > that's fine.

  You are sharing the responsibility with a long line of Emacs
  developers, all of whom use this function.  I don't see anything you
  should worry about, really.

As you see, each response is directly related to your text that I
cite.  I cannot prove to you that I understood what you were saying,
but you can ask any neutral person to read this and tell you what they
think about that.  From my side, I can assure you I completely
understood everything that you said.

Re: Contributors and maintainers

[Taylan Ulrich Bayırlı/Kammer]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Taylan Ulrich "Bayırlı/Kammer" <taylanbayirli@gmail.com> writes:
>
>> The lack of respect I'm receiving is *not* of the kind where someone is
>> being actively nasty, insulting, etc. It's a kind where a person's very
>> voice is being denied, not even countered. That's pretty grave.
>
> Taylan, I don't understand. Looking at emacs-devel and the bug tracker, I see
> that many people have responded to you. Here are the reply counts:
>
>     26 Eli Zaretskii
>      9 Paul Eggert
>      9 Dmitry Gutov
>      7 Random832
>      3 Michael Albinu
>      3 John Wiegley
>      3 David Kastrup
>      3 Artur Malabarb
>      1 Stephen J. Tur
>      1 Nicolas Richar
>      1 Daniel Colasci
>
> Eli has taken the time to write back to you 26 times! This doesn't even count
> the sub-threads where we discuss meta issues not directly related to your
> issue.
>
> Can you share with me what your real concern is? Do you worry that Emacs is
> not secure enough? Is their an active threat of some kind in your environment?
> Perhaps there's something about your use case we're not seeing, that would
> explain the greater importance of this issue to you.

John, I'm afraid you're looking at things too superficially.  I honestly
don't remember a single response that even seemed to acknowledge the
concern I've explained multiple times in detail, except for responses by
Random832.  Having responded at all is not proof of having addressed the
concerns; rather most of those responses were diverting away the topic
while ignoring my actual concern.

Straight out remaining silent would not have irritated me nearly as
much, if at all.  One usually can't draw any clear conclusions from
silence.  But being rapidly responded to, each time with a different
diversion from the main concern, is eventually very agitating.

> We've been living with the current shell-quote-argument for literally
> *decades*, which might explain why we're not instantly ready to make changes
> -- even though Eli has made a change to the docstring at your request. How
> does that constitute "no response"? I am confused.

It made a change that was neither asked for (by me at least), nor
addressed my concern.  Think about how irritating it must be to spend a
lot of effort to explain a concern, *also* provide a patch which solves
that concern, and then have the maintainer reject your patch and instead
apply one of their own which *doesn't* address your concern.

> Also, it does not help to reiterate how clear and cogent your arguments have
> been. Until we both agree, "clarity" and "cogency" have not been achieved.
> These attributes must exist *between* disputants; they cannot be determined by
> one side alone. We have all been working to achieve clarity, but I fear this
> has been misunderstood as a stubborn rejection of your ideas.

The problem is that the concern was not even acknowledged, let alone
being shown the courtesy to be openly disagreed with.

And all the while reiterating my main concern that remained unaddressed,
I *did* try to address many of the counter-concerns that were raised,
although in the grand scheme of things they only served to divert
attention away from my concern.


All of this may not be easy to see to an outside observer of the topic.
Of course, I know what my main concern is very well, but if the thread
contains an equal or greater number of mails talking about other
concerns than mine, then to an outside observer my concern will simply
become less visible, and appear like any of the arbitrary concerns that
were raised and discussed, whichever of them ultimately addressed...

All that might make it very hard to understand why such a level of
irritation would happen in first place, which is why I'm trying to take
a sort of empirical approach to the problem, which is to enumerate the
mails in which I explain the same concern, and ask for mails in which
that concern is clearly acknowledged, and responded to with explicit
disagreement or a solution; anything but bringing up a "related" topic.
If there is a failure to find such mails in the discussion, despite the
spurious amount of mails reiterating the main concern, I hope that's a
clear, somewhat empiric indication of the social problem.

I wish we had a professional psychologist or sociologist as part of the
maintainer team. :-)

Taylan

Re: [PATCH] Add shell-quasiquote.

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > First, do csh and the rest really non-Posix?  I wonder.

Yes.  There is a definition of the syntax of the sh language in the
POSIX standard, and csh and friends do not conform.  Bash and Zsh
don't conform exactly (eg, they provide extended globbing), but they
are nearly upward compatible with minimal sh.

 > I always understood "Posix shells" as a short for "any shell on a
 > Posix host".  Is that incorrect?  In what way are those
 > "non-Posix"?

The syntax of scripts is different from the language defined in POSIX.
csh uses C-like braces for grouping, where sh uses additional keywords
like "then" and "fi" to mark stanzas.  tclsh uses Tcl syntax, of
course.  And so on....

 > Next, I could see why users on a Posix host might want to execute
 > some commands with a particular non-default shell.  I don't see why
 > Emacs packages, perhaps with a sole exception of Tramp, would need
 > that.

Sometimes it pays to be precise.  All of the OS distributions gave up
on bash because it didn't quite conform to POSIX (even when invoked in
POSIX compatibility mode), and that caused bugs in package
installation and management for packages that used sh scripts rather
than perl or python.  That's why shells like ash and dash exist.

 > > It would mainly be useful in the presence of a broader mechanism, which
 > > doesn't exist yet, for executing POSIX shell scripts regardless of the
 > > user's interactive shell.
 > 
 > On Posix hosts?  I thought that was automatic, since each script says
 > what interpreter should run it in its "shebang" line.  Right?

No.  Unfortunately, there's a long history of systems (hel-lo, GNU!)
with /bin/sh linked to a non-conforming program.  Weeding out all the
bashisms in Debian package management scripts took quite a bit of pain
and effort.

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Random832 <random832@fastmail.com>
>> Date: Wed, 21 Oct 2015 13:06:46 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> >> From: Random832 <random832@fastmail.com>
>> >> It doesn't have a documented way for the caller to insist that the
>> >> string be quoted for a POSIX shell rather than the user's shell.
>> >
>> > On what OS would that distinction be important, and why?
>> 
>> Any OS which may have both a POSIX shell that a script may want to
>> execute and a non-POSIX shell that is the user's shell. So basically all
>> of them, especially if support for more non-POSIX shells such as csh,
>> rc, scsh, fish, tclsh, is added in the future - or if a user's
>> configuration supports them in the present by replacing or advising the
>> function.
>
> First, do csh and the rest really non-Posix?  I wonder.  I always
> understood "Posix shells" as a short for "any shell on a Posix host".

I think it's more like "any shell trying to obey an (ex-)POSIX
standard".  Which would be
<URL:http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html>
and would not cover a csh.  Users may set a different login and/or
interactive shell.  That's strictly speaking no longer "POSIX" if it
does not try to be Bourne shell compatible in the main regards.

> Is that incorrect?  In what way are those "non-Posix"?

Not trying to meet an (ex-)POSIX standard I think.

>> It would mainly be useful in the presence of a broader mechanism,
>> which doesn't exist yet, for executing POSIX shell scripts regardless
>> of the user's interactive shell.
>
> On Posix hosts?  I thought that was automatic, since each script says
> what interpreter should run it in its "shebang" line.  Right?

Not necessarily.  You can leave off the shebang line and scripts will be
directly called by a fork (or something programmed to be equivalent) of
your shell which might be more efficient than going through shebang.  To
avoid a csh executing a Bourne-only script, you can place a single : in
the first line.  That's a noop for a Bourne shell, and it tells csh and
its ilk that it should not even try interpreting that script itself but
rather leave that to /bin/sh or similar.

These days, there are very few shell scripts _not_ started with a
shebang line.  But it's a definite possibility, even though then calling
such scripts _not_ from a shell command line might fail.  According to
the man pages for GNU/Linux I have here, system(3) uses /bin/sh and
should likely work whereas execve(2) requires the shebang line in order
to pick an interpreter rather than loading a binary.

Haven't tried this for a long time.

-- 
David Kastrup

Re: Contributors and maintainers

[John Wiegley]

>>>>> Taylan Ulrich "Bayırlı/Kammer" <taylanbayirli@gmail.com> writes:

> And all the while reiterating my main concern that remained unaddressed, I
> *did* try to address many of the counter-concerns that were raised, although
> in the grand scheme of things they only served to divert attention away from
> my concern.

What is your main concern, if I may ask? Even after reading everything on this
topic, I am still not clear.

John

Re: Contributors and maintainers

[Stephen J. Turnbull]

Taylan Ulrich Bayırlı/Kammer writes:

 > Given the current lack of safety guarantees in
 > shell-quote-argument, I actively do not want it to be used in shqq,
 > or any other place where it may receive data from an untrusted
 > input source.

I still have no idea why you are so adamant on this point.  This
function is useless unless you're invoking a shell!  Why worry about
Emacs's `shell-quote-argument' when you are assuming the presence of
an attack surface the size of the Great Red Spot?  In other words,
what safety guarantees are you getting from the shell?

 > Maybe emacs-devel should indeed follow Stefan's advice to merge
 > first, then fix,

N.B.  Stefan's advice is double-edged, you know.  It could be directed
at any maintainer of code, including you!  Ie, you could merge Eli's
suggestion into the GELPA version of your package, and fix
`shell-quote-argument' and/or its documentation and test suite later.

 > unless someone insists that there is a *serious*
 > problem.  That might be a very good policy for emacs-devel.

"emacs-devel" manages the core as well as GELPA.  No large, mature
project can follow that policy in the form you propose for the core
(and if Stefan really meant to apply it to shell-quasiquote, I have to
disagree with him -- see below).  Changes to the core have to have
working consensus.  The maintainers -- developers who have
demonstrated commitment to the project as well as the officially
anointed ones -- are quite likely going to end up taking care of your
code.  To enable that without too much effort, they will want
cooperation up front on practices like using existing APIs --
`shell-quote-argument' -- and following existing style -- the
documentation, see Paul Eggert's post for why Eli's approach conforms
and yours doesn't.

For a well-known committer with history in the project, in practice
"working consensus" is often just that one developer.  But because you
are new and unknown, and apparently very young, you will have people
reviewing your code, and somebody else will have to commit it.  There
will need to be a non-trivial consensus to get it in.  N.B.  The point
of that "very young" is that your job status is likely to change and
you may disappear except for a "Thanks for all the fish".  You may say
that won't happen; unfortunately, history shows that a lot of people
who make such promises are unable to keep them.  From experience, we
can't assign a probability as high as 1/2 that it won't happen to you.
But the probability that any of Eli, David[1], Paul, and the cast
of dozens who disagree with you now, and who *will* take
responsibility for your code if you cannot at some later date, will go
away soon is quite low.

Yes, GNU ELPA is probably a different context.  But one unfortunate
fact is that GELPA is at present neither fish nor fowl.  It is part of
GNU Emacs; it says so on the label, and it follows the same policies
with respect to copyright assignment and so on as the core does.  But
most likely the commitment of the maintainers (as defined above) is
much weaker with respect to GELPA than for the core.  If it is
sufficiently weak, "commit, then fix" becomes a very plausible
strategy.  But GELPA may be merely a device to allow modules with low
coupling to the core to have different development cycles, and. really
"it's all just Emacs" (IMO this is likely the way Richard Stallman
would like it to be, for software freedom reasons).  In that case I'd
have to say to be conservative, because package repositories tend to
grow exponentially, and will quickly exceed the ability of maintainers
to keep up.  In the meantime, it *may* be the latter, and (for now)
conservatism is indicated.

Third-party developers want to get their packages out to Emacs users,
of course.  If they don't like the review and bureaucracy and
compromise currently associated with GELPA, they have alternative
efficient ways to do that: github and MELPA.  All three suffer about
equally in terms of discoverability by users compared to the core
(strictly defined).  You lose some True Believers who refuse to get
github accounts if you use github, and users are less likely to have
configured MELPA than GELPA, but the word does get out.  There are a
fair number of popular packages that don't even live in any ELPA, but
are distributed on the EmacsWiki or personal websites.

Footnotes: 
[1]  David will disclaim maintainership even in the informal sense, I
expect.  He's currently on sabbatical, with occasional visits as
gadfly. :-)

Re: Contributors and maintainers

[Stephen J. Turnbull]

John Wiegley writes:

 > Also, it does not help to reiterate how clear and cogent your
 > arguments have been. Until we both agree, "clarity" and "cogency"
 > have not been achieved.

N.B.  Here I hope that by "agree" John means "agree that we both have
the same understanding of the issue".  Sometimes different value
systems means that although we do understand the issue the same way,
we take different positions, and will still not agree on what to do.
But that's rarely a complete blocker to cooperation.

One of the nice properties of free software is that when agreeing to
disagree, we can both have the bikeshed painted in our favorite color.

Re: Contributors and maintainers

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Date: Wed, 21 Oct 2015 19:46:04 +0200
> 
> I honestly don't remember a single response that even seemed to
> acknowledge the concern I've explained multiple times in detail,
> except for responses by Random832.  Having responded at all is not
> proof of having addressed the
> [...]
> > We've been living with the current shell-quote-argument for literally
> > *decades*, which might explain why we're not instantly ready to make changes
> > -- even though Eli has made a change to the docstring at your request. How
> > does that constitute "no response"? I am confused.
> 
> It made a change that was neither asked for (by me at least), nor
> addressed my concern.  Think about how irritating it must be to spend a
> lot of effort to explain a concern, *also* provide a patch which solves
> that concern, and then have the maintainer reject your patch and instead
> apply one of their own which *doesn't* address your concern.

How can installing a change triggered by your bug report be _anything_
_but_ A RESPONSE to your concerns?  It was not the patch you proposed,
that's true, but it was my response to the less-than-ideal situation
that _you_ described.

That's _exactly_ what we are here for: to listen to you and others,
make our own analysis of the situation, and act accordingly.  You will
have to accept a trivial thing -- that the way we act on your reports
and proposals will not always exactly match what you proposed, because
we have different perspectives and different experiences.  When you
will become an Emacs maintainer, a day that I hope will come, you will
have the same prerogative and the same responsibilities.

> > Also, it does not help to reiterate how clear and cogent your arguments have
> > been. Until we both agree, "clarity" and "cogency" have not been achieved.
> > These attributes must exist *between* disputants; they cannot be determined by
> > one side alone. We have all been working to achieve clarity, but I fear this
> > has been misunderstood as a stubborn rejection of your ideas.
> 
> The problem is that the concern was not even acknowledged, let alone
> being shown the courtesy to be openly disagreed with.

What else can possibly acknowledge your concern, if not making changes
to resolve your concerns??

> And all the while reiterating my main concern that remained unaddressed,
> I *did* try to address many of the counter-concerns that were raised,
> although in the grand scheme of things they only served to divert
> attention away from my concern.

That's your lopsided POV, nothing else.  You only consider your
concerns addressed if they and your suggested solutions
are_absolutely_ accepted, without any changes.  You don't allow anyone
to deviate even an inch from your proposals, and if they dare, they
are "not addressing" and "not acknowledging" your concerns.  Really,
this is beyond 1984's Newspeak.

> All of this may not be easy to see to an outside observer of the topic.

No, it's actually VERY easy to see.  Fact is, several people here
independently told you exactly the same: you need to accept the
judgment of the project maintenance team, and you should respect their
decisions even if you disagree with them.

> All that might make it very hard to understand why such a level of
> irritation would happen in first place, which is why I'm trying to take
> a sort of empirical approach to the problem, which is to enumerate the
> mails in which I explain the same concern, and ask for mails in which
> that concern is clearly acknowledged, and responded to with explicit
> disagreement or a solution; anything but bringing up a "related" topic.

Your concerns were acknowledged, but your proposed solutions were
disagreed to.  That's all that happened.  You need to learn to accept
that, because this is how any Free Software community works.

> I wish we had a professional psychologist or sociologist as part of the
> maintainer team. :-)

She wasn't needed until now.

Re: [PATCH] Add shell-quasiquote.

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> If there are indeed such people, I encourage them to come here and speak up
> for themselves. Given the history of this thread, I'm not at all sure we
> should hear their concerns through intermediaries. Changes in leadership are
> a good time to try to make our procedures and conduct better, so now is a
> good time for them to speak up.

I could agree more, Eli. Now is the best time for those who have been
disaffected in the past to come forward, as this is a perfect time to
re-examine old wounds and make changes for the better.

John

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Eli Zaretskii writes:
>
>  > Next, I could see why users on a Posix host might want to execute
>  > some commands with a particular non-default shell.  I don't see why
>  > Emacs packages, perhaps with a sole exception of Tramp, would need
>  > that.
>
> Sometimes it pays to be precise.  All of the OS distributions gave up
> on bash because it didn't quite conform to POSIX (even when invoked in
> POSIX compatibility mode), and that caused bugs in package
> installation and management for packages that used sh scripts rather
> than perl or python.  That's why shells like ash and dash exist.

I think it was more a question of size and startup speed.  bash has a
lot of features beyond those in the POSIX specs, and the main arguments
I saw for replacing bash with dash on systems such as Ubuntu were based
on benchmarks.

This choice led to a lot of fun with people having to clean their shell
scripts from "bashisms", constructs not available in POSIX standards but
provided by bash.  The "function" keyword for shell functions is one of
the most frequent offenders.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[Wolfgang Jenkner]

On Thu, Oct 22 2015, Stephen J. Turnbull wrote:

> No.  Unfortunately, there's a long history of systems (hel-lo, GNU!)
> with /bin/sh linked to a non-conforming program.  Weeding out all the
> bashisms in Debian package management scripts took quite a bit of pain
> and effort.

Also, a fact that is somewhat relevant to the present discussion is that
emacs doesn't use /bin/sh by default:

$ echo $SHELL
/usr/local/bin/bash
$ emacs --batch -Q --eval '(message "%s" shell-file-name)'
/usr/local/bin/bash
$ ls -l /bin/sh
-r-xr-xr-x  1 root  wheel  172733  1 Jul 15:42 /bin/sh*
$ 

Re: Contributors and maintainers

[John Wiegley]

>>>>> Taylan Ulrich "Bayırlı/Kammer" <taylanbayirli@gmail.com> writes:

> No it's not "what this is about." Given the current lack of safety
> guarantees in shell-quote-argument, I actively do not want it to be used in
> shqq, or any other place where it may receive data from an untrusted input
> source.

Is this really the core issue we're debating? Then let me respond to it: **We
do not agree with your assessment of shell-quote-argument's lack of safety,
and require tests from you to demonstrate this is the case**. The ball is now
officially in your court.

> But since we're in a deadlock regarding that topic, I say take my code as
> is, first of all.

And my response is: No. We will not take your code, because we do not want it
in its present form.

> One way or another, please as a first step apply the patch, since that has
> clearly positive utility.

It is not clearly positive to *us*. "Clearly" is a subjective assessment.

> Maybe emacs-devel should indeed follow Stefan's advice to merge first, then
> fix, unless someone insists that there is a *serious* problem. That might be
> a very good policy for emacs-devel.

Absolutely not. We do not have time to play catch up, and to retroactively fix
bugs we allow in because submitters want their code committed right away. We
try to filter what we accept *before* it goes in, so we can move on to other
things.

"Check in first, fix later" is a policy I have personally seen destroy code
bases at professional organizations, where people were paid 40 hours a week to
keep the code maintained. Now imagine this at Emacs-scale, and you will see
it's untenable as a strategy.

If you're upset because we don't want your submission in its current form,
there isn't much I can do about that. You know what needs to be done to fix
it. If you don't want to fix it, or don't believe it needs fixing, use MELPA.

There is really nothing more to say on this subject, so I would appreciate
closing the matter until new code is produced.

John

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Date: Wed, 21 Oct 2015 20:24:50 +0200
> Cc: Random832 <random832@fastmail.com>, Eli Zaretskii <eliz@gnu.org>,
> 	emacs-devel@gnu.org
> 
> On Thu, Oct 22 2015, Stephen J. Turnbull wrote:
> 
> > No.  Unfortunately, there's a long history of systems (hel-lo, GNU!)
> > with /bin/sh linked to a non-conforming program.  Weeding out all the
> > bashisms in Debian package management scripts took quite a bit of pain
> > and effort.
> 
> Also, a fact that is somewhat relevant to the present discussion is that
> emacs doesn't use /bin/sh by default:
> 
> $ echo $SHELL
> /usr/local/bin/bash
> $ emacs --batch -Q --eval '(message "%s" shell-file-name)'
> /usr/local/bin/bash

That's the documented behavior, see the User Manual.

Re: Contributors and maintainers

[John Wiegley]

>>>>> Taylan Ulrich "Bayırlı/Kammer" <taylanbayirli@gmail.com> writes:

> How about, *first* of all, the latest version of my ELPA patch gets applied,
> so there is an *immediate* benefit to Emacs users. Claiming that a single
> line of duplicated code outweighs that would be absurd.

It is not absurd to us, who maintain Emacs. Telling us that our concerns are
absurd is inconsiderate.

Your current patch has been rejected. You should be working with us to
formulate an acceptable patch, rather than constantly arguing about why your
patch should be accepted, or inciting the venom of those disaffected with
emacs-devel to gain moral support for your position.

You underestimate our desire for a correctly behaving `shell-quote-argument'.
In that, I believe we're in perfect agreement. What we want is a test to make
these weaknesses clear to us, not only so that the improvement may be
technically grounded, but as documentation and evidence that it will never
break in future.

Instead, the way I feel right now is that you want to ram your code down our
throats, and are castigating us for not elevating your concern to same degree
of importance you feel it is. This is just one issue out of hundreds we have
on our plate. To be an effective contributor means working with us, not
against us, to improve Emacs.

If our concerns lead to make-work or inefficiencies sometimes, consider that
an opportunity to demonstrate your willingness to be a good citizen. That
would buy you a lot more social credit when it comes to the next patch, than
fighting for the specific outcome you want.

John

Re: [PATCH] Add shell-quasiquote.

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
> First, do csh and the rest really non-Posix?

As far as I know, all of the ones I named have _some_ quirk in the
quoting mechanism that makes it not work. See my earlier messages about
newline mishandling for how csh fails. More broadly, a sh script and a
csh script are obviously in two completely different languages, and see
below.

Doing anything interesting (where "interesting" is defined as almost
anything that you shouldn't be using call-process for anyway) with a
shell command requires knowing what kind of shell it is, which is why I
am somewhat sympathetic to Taylan's "only support POSIX" position.

> I wonder.  I always
> understood "Posix shells" as a short for "any shell on a Posix host".
> Is that incorrect?  In what way are those "non-Posix"?

POSIX defines an entire shell command language, which is a subset of
[most of] ksh88. Bash, among others, supports a superset of this
language [many of its additional features, incidentally, are shared with
ksh93] and has a --posix mode to support it more precisely (documented
at http://tiswww.case.edu/php/chet/bash/POSIX).

> Next, I could see why users on a Posix host might want to execute some
> commands with a particular non-default shell.  I don't see why Emacs
> packages, perhaps with a sole exception of Tramp, would need that.
>
> On MS-Windows, using a Posix shell needs to customize variables like
> explicit-shell-file-name, and Emacs detects that automatically.
>
>> It would mainly be useful in the presence of a broader mechanism, which
>> doesn't exist yet, for executing POSIX shell scripts regardless of the
>> user's interactive shell.
>
> On Posix hosts?  I thought that was automatic, since each script says
> what interpreter should run it in its "shebang" line.  Right?

I meant a script as a string, not a file. Maybe call it a command, but
I'm specifically referring to things more complicated than a single
utility and a list of arguments (which should use call-process).

Re: Contributors and maintainers

[John Wiegley]

>>>>> Stephen J Turnbull <setphen@xemacs.org> writes:

> N.B. Here I hope that by "agree" John means "agree that we both have the
> same understanding of the issue". Sometimes different value systems means
> that although we do understand the issue the same way, we take different
> positions, and will still not agree on what to do. But that's rarely a
> complete blocker to cooperation.

Thank you for that clarification, Stephen. "Agree" here means that we both
agree the bike is blue, not that we both agree it should be.

You remind me of a funny story. Once, early in my marriage, I had this green
shirt that was somewhat bluish. To my eyes, it was clearly green. To my wife's
eyes, it was clearly blue. We both stood there, saying to one another, "It's
blue! It's green! It's blue! It's green!" :)

We at least agreed it was either blue or green, or ambiguously blue-green,
that much was clear. But the exact shade was not clear, even though it was
*crystal* clear to each of us alone.

John

Re: [PATCH] Add shell-quasiquote.

[Wolfgang Jenkner]

On Wed, Oct 21 2015, Eli Zaretskii wrote:

> That's the documented behavior, see the User Manual.

I know.  But you said in http://permalink.gmane.org/gmane.emacs.bugs/107739

    The standard shell of each
    OS is well defined and known to the users of the respective systems.
    Moreover, Emacs by default uses that shell automatically.

I'd say the standard shell is /bin/sh, which may be different from the
(default) login shell (which is the value of SHELL, usually).

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Wed, 21 Oct 2015 14:49:43 -0400
> 
> > On Posix hosts?  I thought that was automatic, since each script says
> > what interpreter should run it in its "shebang" line.  Right?
> 
> I meant a script as a string, not a file. Maybe call it a command, but
> I'm specifically referring to things more complicated than a single
> utility and a list of arguments (which should use call-process).

You mean, have a shell-like interpreter in Emacs?  Doesn't Eshell do
that already?

Re: [PATCH] Add shell-quasiquote.

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
> You mean, have a shell-like interpreter in Emacs?  Doesn't Eshell do
> that already?

No, I'm talking about running shell commands - the same way you do now,
only with the ability to actually *know* it will be run in a POSIX shell
rather than guessing, so that your code can run unmodified even on other
people's machines who use some other shell as their login/interactive
shell.

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Cc: random832@fastmail.com,  stephen@xemacs.org,  emacs-devel@gnu.org
> Date: Wed, 21 Oct 2015 20:57:21 +0200
> 
> On Wed, Oct 21 2015, Eli Zaretskii wrote:
> 
> > That's the documented behavior, see the User Manual.
> 
> I know.  But you said in http://permalink.gmane.org/gmane.emacs.bugs/107739
> 
>     The standard shell of each
>     OS is well defined and known to the users of the respective systems.
>     Moreover, Emacs by default uses that shell automatically.
> 
> I'd say the standard shell is /bin/sh, which may be different from the
> (default) login shell (which is the value of SHELL, usually).

Yes, all true.  Which means we have a potential problem, because the
shell that Emacs uses to run shell commands might not be "the standard
shell", but some "unusual shell" for which shell-quote-argument might
(albeit in rare cases) produce incorrect results.  And that's what the
updated doc string says, in effect.

I guess patches to extend shell-quote-argument to cover more shells on
Posix systems should be welcome, and should get higher priority than
we thought.

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Wed, 21 Oct 2015 15:10:35 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > You mean, have a shell-like interpreter in Emacs?  Doesn't Eshell do
> > that already?
> 
> No, I'm talking about running shell commands - the same way you do now,
> only with the ability to actually *know* it will be run in a POSIX shell
> rather than guessing, so that your code can run unmodified even on other
> people's machines who use some other shell as their login/interactive
> shell.

Sorry, you lost me.  If it's not Emacs who will interpret the command,
then it must be some real shell.  So are you talking about forcing
Emacs to run /bin/sh and nothing else?  If so, you can simply bind
shell-file-name to that (and maybe also bind shell-command-switch
accordingly).  Or am I (again) missing something?

Re: [PATCH] Add shell-quasiquote.

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> I guess patches to extend shell-quote-argument to cover more shells on Posix
> systems should be welcome, and should get higher priority than we thought.

Agreed. Now if only there were someone willing to write tests for those
systems, so we wouldn't have to actually try it out everywhere to know if it's
correct... :)

John

Re: [PATCH] Add shell-quasiquote.

[Random832]

Eli Zaretskii <eliz@gnu.org> writes:
> Sorry, you lost me.  If it's not Emacs who will interpret the command,
> then it must be some real shell.  So are you talking about forcing
> Emacs to run /bin/sh and nothing else?  If so, you can simply bind
> shell-file-name to that (and maybe also bind shell-command-switch
> accordingly).  Or am I (again) missing something?

/bin/sh might not be a POSIX shell. on Solaris you need
/usr/xpg4/bin/sh. And you might need environment variables set such as
POSIXLY_CORRECT or PATH to get POSIX behavior from all utilities.

Re: Contributors and maintainers

[Taylan Ulrich Bayırlı/Kammer]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>> Cc: emacs-devel@gnu.org
>> Date: Wed, 21 Oct 2015 19:22:10 +0200
>> 
>> >> Can you please show a previous quote by you which serves to show that
>> >> you understood the reason I did not want to use shell-quote-argument,
>> >> and where you directly addressed that exact reason (either with a change
>> >> to shell-quote-argument, *or* an explanation of why you disagree with
>> >> that exact reason)?
>> >
>> > Everything I wrote serves to show that.  I always responded to your
>> > arguments.
>> >
>> >> When you show such a quote, then maybe we can look at it and see how it
>> >> could be that you feel my concern has been addressed, yet I don't.
>> >
>> > Not only I feel that your concerns has been addressed, everyone else
>> > here does.
>> 
>> It would be great if you could provide a quote like I mentioned.
>
> OK.  Here are some:

Thanks. :-)

> Quote #1:
>
>   > > On POSIX shells, shell-quote-argument is just as safe as
>   > > shqq--quote-string, and on non-POSIX shells it works better. So it's a
>   > > win, in both readability and in portability, to use
>   > > shell-quote-argument.
>   > 
>   > Fixing it does not seem easy at all given I can't trust
>   > shell-quote-argument.
>
>   You can trust it.
>
>   > And please be realistic in the amount of trust we can put on the
>   > complicated implementations for non-Unix shells.  I can't judge them
>   > myself since I don't know the syntax of those shells at all.  Does
>   > anyone here know their syntax comprehensively, or checked the
>   > implementation against the documentation of those shells?
>
>   Yes, we do.  Yes, we have.

This coincided with Random832's demonstration of it being broken for
csh, and mentioning that the semantics are questionable on Windows if I
got it right, as well as a mention of a past bug about newline handling
for POSIX shells.  So I'm afraid you were short of lying to me.  And
please don't get me wrong but the annoyed and dismissive tone really
made it obvious that not much thought was put into the response in first
place, so the factual inaccuracy was not exactly a surprise.

> Quote #2:
>
>   > > It might be simpler, but it's wrong, because the result is only
>   > > correct for Posix shells.
>   > >
>   > > Please do use shell-quote-argument instead.
>   > 
>   > It's also simpler than the POSIX section of shell-quote-argument.
>
>   Simpler doesn't mean correct.
>
>   > (defun shell-quote-argument (argument)
>   >   [...] (cond [...] (t
>   >     (if (equal argument "")
>   >         "''"
>   >       ;; Quote everything except POSIX filename characters.
>   >       ;; This should be safe enough even for really weird shells.
>   >       (replace-regexp-in-string
>   >        "\n" "'\n'"
>   >        (replace-regexp-in-string "[^-0-9a-zA-Z_./\n]" "\\\\\\&" argument))))))
>   > 
>   > I wonder what "really weird shells" this refers to?
>
>   The set of characters special to an arbitrary shell is not known in
>   advance.
>
>   > Certainly not csh, the mechanism it uses for newlines doesn't work
>   > there.
>
>   What did you try that didn't work with csh?

This is unrelated to the main concern of lacking safety guarantees.

It's also not a response to me.

> Quote #3:
>
>   > Quoting RMS, coincidentally from a couple days ago:
>   > 
>   >     The policy is non-GNU systems are secondary, and lower priority than
>   >     the GNU system, but we are glad to include support for them in GNU
>   >     packages if users contribute the necessary code -- provided that
>   >     code isn't a maintenance problem for us.
>   > 
>   >     The maintenainers of any particular package are the ones who judge
>   >     whether that code is a maintenance problem, since they are the ones
>   >     it would be a problem for.
>
>   I don't see how this is relevant for the issue at hand, since the
>   necessary code (the shell-quote-argument function) was already
>   contributed to Emacs years ago, and is used in many places in core
>   Emacs.  There's no extra effort needed to support more platforms, just
>   replace one function with another.
>
>   > I generally don't want to take responsibility of my code being used on
>   > non-GNU/non-POSIX systems, but if I can share the responsibility then
>   > that's fine.
>
>   You are sharing the responsibility with a long line of Emacs
>   developers, all of whom use this function.  I don't see anything you
>   should worry about, really.

This tries to push on me a responsibility I cannot take because I don't
use MS Windows and don't know its shell syntax, and asserts that I
should stop worrying, i.e. flat-out dismissing my concern.

> As you see, each response is directly related to your text that I
> cite.  I cannot prove to you that I understood what you were saying,
> but you can ask any neutral person to read this and tell you what they
> think about that.  From my side, I can assure you I completely
> understood everything that you said.

It seems that we cannot even agree on whether an English sentence does
or doesn't address a concern raised in another.

English is not my mother-tongue either, but I think both our English is
perfectly good.  Given that, I genuinely have no idea how we can be this
badly unable to communicate.

I rest my case.

Taylan

Re: Contributors and maintainers

[John Wiegley]

>>>>> Taylan Ulrich "Bayırlı/Kammer" <taylanbayirli@gmail.com> writes:

> I rest my case.

You are the only person in this entire conversation (i.e., among those on
emacs-devel) who believes his case has been demonstrated and proven. Doesn't
this suggest to you that you should re-examine your position and attitude?

John

Re: Contributors and maintainers

[David Kastrup]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Taylan Ulrich "Bayırlı/Kammer" <taylanbayirli@gmail.com> writes:
>
>> I rest my case.
>
> You are the only person in this entire conversation (i.e., among those on
> emacs-devel) who believes his case has been demonstrated and proven. Doesn't
> this suggest to you that you should re-examine your position and attitude?

This thread will not find a resolution.  By now people have lost all
memory of the texts of the original exchange and only have an incomplete
recollection of their feelings at the time, amplified through the
feeling of having made their viewpoint abundantly clear so often that
everybody must be resonating with it and the absurdity of anything else.

Repeating one another's disbelief over and over will only make it easier
to talk past each other.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Wed, 21 Oct 2015 15:50:32 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > Sorry, you lost me.  If it's not Emacs who will interpret the command,
> > then it must be some real shell.  So are you talking about forcing
> > Emacs to run /bin/sh and nothing else?  If so, you can simply bind
> > shell-file-name to that (and maybe also bind shell-command-switch
> > accordingly).  Or am I (again) missing something?
> 
> /bin/sh might not be a POSIX shell. on Solaris you need
> /usr/xpg4/bin/sh. And you might need environment variables set such as
> POSIXLY_CORRECT or PATH to get POSIX behavior from all utilities.

OK, but that still boils down to binding some more variables.  If we
want to help users with these factoids, we could have a small database
of the known Posix shells and their requirements.

Re: [PATCH] Add shell-quasiquote.

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1240 bytes --]

On 10/20/2015 11:55 AM, Taylan Ulrich Bayırlı/Kammer wrote:
> Paul Eggert<eggert@cs.ucla.edu>  writes:
>
>> >Taylan Ulrich Bayırlı/Kammer wrote:
>>>> >>>I must have missed it then, because all I remember are the cases (1)
>>>>> >>> >of running /bin/if (which is trivial and is not a realistic example),
>>>>> >>> >and (2) of installations with nonstandard shells (a problem that
>>>>> >>> >shqq--quote-string does not fix). It has been a long thread; quite
>>>>> >>> >possibly I missed something.
>>> >>Yeah, you missed the part about risk of code injection.:-)
>> >
>> >Code injection occurs because of (2), right? So it's not a risk that
>> >shqq--quote-string would put much of a dent in.
> Sorry, no, that's not the problem.
>
>
> So after four days of incredibly tiresome repetition, people still don't
> understand the basic issue.

That's right, we don't. At least, I don't, and your recent responses 
haven't clarified things for me. That being said, I guessed as to what 
you're driving at, and installed the attached patch into the Emacs 
master. Although the new documentation section is pretty sketchy, 
perhaps it can be fleshed out by people who have more time to worry 
about this sort of thing.

[-- Attachment #2: 0001-New-lispref-section-Security-Considerations.patch --]
[-- Type: text/x-patch, Size: 9080 bytes --]

From 71de784f1a14200b83049e359a9b009eddbceb94 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Wed, 21 Oct 2015 20:22:34 -0700
Subject: [PATCH 1/2] =?UTF-8?q?New=20lispref=20section=20=E2=80=9CSecurity?=
 =?UTF-8?q?=20Considerations=E2=80=9D?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This attempts to document some of the issues recently discussed
on emacs-devel, and to indicate other such issues.  The section
could be a lot longer.
* doc/lispref/os.texi (Security Considerations):
New node.
* doc/lispref/elisp.texi (Top):
* doc/lispref/processes.texi (Shell Arguments):
* lisp/subr.el (shell-quote-argument):
* src/callproc.c (syms_of_callproc):
Reference it.
---
 doc/lispref/elisp.texi     |   1 +
 doc/lispref/os.texi        | 104 +++++++++++++++++++++++++++++++++++++++++++++
 doc/lispref/processes.texi |   2 +-
 lisp/subr.el               |   3 +-
 src/callproc.c             |   2 +-
 5 files changed, 109 insertions(+), 3 deletions(-)

diff --git a/doc/lispref/elisp.texi b/doc/lispref/elisp.texi
index 5ca518e..2d3548f 100644
--- a/doc/lispref/elisp.texi
+++ b/doc/lispref/elisp.texi
@@ -1487,6 +1487,7 @@ Top
 * Desktop Notifications::   Desktop notifications.
 * File Notifications::      File notifications.
 * Dynamic Libraries::       On-demand loading of support libraries.
+* Security Considerations:: Running Emacs in an unfriendly environment.
 
 Starting Up Emacs
 
diff --git a/doc/lispref/os.texi b/doc/lispref/os.texi
index 204055d..1925bd5 100644
--- a/doc/lispref/os.texi
+++ b/doc/lispref/os.texi
@@ -37,6 +37,7 @@ System Interface
 * Desktop Notifications:: Desktop notifications.
 * File Notifications::  File notifications.
 * Dynamic Libraries::   On-demand loading of support libraries.
+* Security Considerations:: Running Emacs in an unfriendly environment.
 @end menu
 
 @node Starting Up
@@ -2760,3 +2761,106 @@ Dynamic Libraries
 This variable is ignored if the given @var{library} is statically
 linked into Emacs.
 @end defvar
+
+@node Security Considerations
+@section Security Considerations
+@cindex security
+@cindex hardening
+
+Like any application, Emacs can be run in a secure environment, where
+the operating system enforces rules about access and the like.  With
+some care, Emacs-based applications can also be part of a security
+perimeter that checks such rules.  Although the default settings for
+Emacs work well for a typical software development environment, they
+may require adjustment in environments containing untrusted users that
+may include attackers.  Here is a compendium of security issues that
+may be helpful if you are developing such applications.  It is by no
+means complete; it is intended to give you an idea of the security
+issues involved, rather than to be a security checklist.
+
+@table @asis
+@item Access control
+Although Emacs normally respects access permissions of the underlying
+operating system, in some cases it handles accesses specially.  For
+example, file names can have handlers that treat the files specially,
+with their own access checking.  @xref{Magic File Names}.  Also, a
+buffer can be read-only even if the corresponding file is writeable,
+and vice versa, which can result in messages such as @samp{File passwd
+is write-protected; try to save anyway? (yes or no)}.  @xref{Read Only
+Buffers}.
+
+@item Authentication
+Emacs has several functions that deal with passwords, e.g.,
+@code{password-read}.  Although these functions do not attempt to
+broadcast passwords to the world, their implementations are not proof
+against determined attackers with access to Emacs internals.  For
+example, even if Elisp code attempts to scrub a password from
+its memory after using it, remnants of the password may still reside
+in the garbage-collected free list.
+
+@item Code injection
+Emacs can send commands to many other applications, and applications
+should take care that strings sent as operands of these commands are
+not misinterpreted as directives.  For example, when sending a shell
+command to rename a file @var{a} to @var{b}, do not simply use the
+string @code{mv @var{a} @var{b}}, because either file name might start
+with @samp{-}, or might contain shell metacharacters like @samp{;}.
+Although functions like @code{shell-quote-argument} can help avoid
+this sort of problem, they are not panaceas; for example, on a POSIX
+platform @code{shell-quote-argument} quotes shell metacharacters but
+not leading @samp{-}.  @xref{Shell Arguments}.
+
+@item Coding systems
+Emacs attempts to infer the coding systems of the files and network
+connections it accesses.  If it makes a mistake, or if the other
+parties to the network connection disagree with Emacs's deductions,
+the resulting system could be unreliable.  Also, even when it infers
+correctly, Emacs often can use bytes that other programs cannot.  For
+example, although to Emacs the NUL (all bits zero) byte is just a
+character like any other, many other applications treat it as a string
+terminator and mishandle strings or files containing NUL bytes.
+
+@item Environment and configuration variables
+POSIX specifies several environment variables that can affect how
+Emacs behaves.  Any environment variable whose name consists entirely
+of uppercase ASCII letters, digits, and the underscore may affect the
+internal behavior of Emacs.  Emacs uses several such variables, e.g.,
+@env{EMACSLOADPATH}.  @xref{Library Search}.  On some platforms some
+environment variables (e.g., @env{PATH}, @env{POSIXLY_CORRECT},
+@env{SHELL}, @env{TMPDIR}) need to have properly-configured values in
+order to get standard behavior for any utility Emacs might invoke.
+Even seemingly-benign variables like @env{TZ} may have security
+implications.
+
+Emacs has customization and other variables with similar
+considerations.  For example, if the variable @code{shell-file-name}
+specifies a shell with nonstandard behavior, an Emacs-based
+application may misbehave.
+
+@item Installation
+When Emacs is installed, if the installation directory hierarchy can
+be modified by untrusted users, the application cannot be trusted.
+This applies also to the directory hierarchies of the programs that
+Emacs uses, and of the files that Emacs reads and writes.
+
+@item Network access
+Emacs often accesses the network, and you may want to configure it to
+avoid network accesses that it would normally do.  For example, unless
+you set @code{tramp-mode} to @code{nil}, file names using a certain
+syntax are interpreted as being network files, and are retrieved
+across the network.  @xref{Top, The Tramp Manual,, tramp, The Tramp
+Manual}.
+
+@item Race conditions
+Emacs applications have the same sort of race-condition issues that
+other applications do.  For example, even when
+@code{(file-readable-p "foo.txt")} returns @code{t}, it could be that
+@file{foo.txt} is unreadable because some other program changed the
+file's permissions between the call to @code{file-readable-p} and now.
+
+@item Resource limits
+When Emacs exhausts memory or other operating system resources, its
+behavior can be less reliable, in that computations that ordinarily
+run to completion may abort back to the top level.  This may cause
+Emacs to neglect operations that it normally would have done.
+@end table
diff --git a/doc/lispref/processes.texi b/doc/lispref/processes.texi
index 196cb7c..0ce696a 100644
--- a/doc/lispref/processes.texi
+++ b/doc/lispref/processes.texi
@@ -180,7 +180,7 @@ Shell Arguments
 Precisely what this function does depends on your operating system.  The
 function is designed to work with the syntax of your system's standard
 shell; if you use an unusual shell, you will need to redefine this
-function.
+function.  @xref{Security Considerations}.
 
 @example
 ;; @r{This example shows the behavior on GNU and Unix systems.}
diff --git a/lisp/subr.el b/lisp/subr.el
index c903ee3..ea926ae 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -2714,7 +2714,8 @@ shell-quote-argument
   "Quote ARGUMENT for passing as argument to an inferior shell.
 
 This function is designed to work with the syntax of your system's
-standard shell, and might produce incorrect results with unusual shells."
+standard shell, and might produce incorrect results with unusual shells.
+See Info node `(elisp)Security Considerations'."
   (cond
    ((eq system-type 'ms-dos)
     ;; Quote using double quotes, but escape any existing quotes in
diff --git a/src/callproc.c b/src/callproc.c
index eafd621..bb21c35 100644
--- a/src/callproc.c
+++ b/src/callproc.c
@@ -1660,7 +1660,7 @@ syms_of_callproc (void)
   DEFVAR_LISP ("shell-file-name", Vshell_file_name,
 	       doc: /* File name to load inferior shells from.
 Initialized from the SHELL environment variable, or to a system-dependent
-default if SHELL is not set.  */);
+default if SHELL is unset.  See Info node `(elisp)Security Considerations'.  */);
 
   DEFVAR_LISP ("exec-path", Vexec_path,
 	       doc: /* List of directories to search programs to run in subprocesses.
-- 
2.1.0

Maintainers and contributors (was: Contributors and maintainers)

[John Wiegley]

>>>>> John Wiegley <johnw@newartisans.com> writes:

> Part of the debate seems to be a lack of appreciation of the difference
> between contributors and maintainers. You see, it is not sufficient to have
> a good idea, no matter how clear it is to its submitter. *We* maintain
> Emacs, and so the change must satisfy *us*, no matter how thick our skulls
> may be. If we ask for clarification that Wednesday follows Tuesday, either
> you provide us with that clarification, or the change doesn't go in. Period.

After I sent this message, it occurred to me that while I had strongly
represented one side of this equation -- the needs of maintainers -- I had
omitted to stress the importance of contributors. Thanks to RMS for reminding
me of this discrepancy. Now let me expound on that side of things.

While it's true that maintainers must agree to what they will maintain, and
that contributors must -- to play in our sandbox -- reach some accord with the
maintainers, this may have made it sound like maintainers are all important,
and contributors are only incidental to Emacs' success. But that is only half
of the story.

What moderates the maintainers use of authority is the fact that contributors
are our *number one* most valued resource. As good, energetic and diligent as
a maintainer may be, he or she cannot perform the work of a hundred, of a
thousand people. Contributors can do this. The role of a good maintainer is to
enable and coordinate this massive sea of energy toward the best end results
for our project.

This typically leads to an easy and natural balance: Contributors respect
maintainers, so their code is accepted; and maintainers respect contributors,
so they generate more code. As long as this synergy is maintained, we all
become more potent than we could ever be alone.

So I'd like to emphasize, to our maintainers, that we must strive to make
patience, kindness, and consideration, our primary attributes when dealing
with contributors -- especially those new to the process, who are developing
their first impressions, and based upon which will either continue to help us,
or walk away frustrated.

[This is not a reflection of malfeasance on anyone's part this week; simply it
is a reminder that our contributors are the lifeblood of our project, and will
someday soon be taking our place.]

John

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Random832 <random832@fastmail.com>
>> Date: Wed, 21 Oct 2015 15:50:32 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> > Sorry, you lost me.  If it's not Emacs who will interpret the command,
>> > then it must be some real shell.  So are you talking about forcing
>> > Emacs to run /bin/sh and nothing else?  If so, you can simply bind
>> > shell-file-name to that (and maybe also bind shell-command-switch
>> > accordingly).  Or am I (again) missing something?
>> 
>> /bin/sh might not be a POSIX shell. on Solaris you need
>> /usr/xpg4/bin/sh. And you might need environment variables set such as
>> POSIXLY_CORRECT or PATH to get POSIX behavior from all utilities.
>
> OK, but that still boils down to binding some more variables.  If we
> want to help users with these factoids, we could have a small database
> of the known Posix shells and their requirements.

I think that's overdoing it with regard to shell-quote-argument and
friends.  We don't need a full POSIX shell, just something with the most
basic quoting conventions of it.  /bin/sh should be fine here.

That's all the guarantee you get for calling commands/scripts with
`system'.  I don't think we should require more than that or try
providing some guarantees in that regard.

-- 
David Kastrup

Re: Maintainers and contributors

[David Kastrup]

John Wiegley <johnw@newartisans.com> writes:

> So I'd like to emphasize, to our maintainers, that we must strive to
> make patience, kindness, and consideration, our primary attributes
> when dealing with contributors -- especially those new to the process,
> who are developing their first impressions, and based upon which will
> either continue to help us, or walk away frustrated.
>
> [This is not a reflection of malfeasance on anyone's part this week;
> simply it is a reminder that our contributors are the lifeblood of our
> project, and will someday soon be taking our place.]

Hopefully.  With regard to that, it's also important that our code is
friendly and welcoming.  It has the potential to annoy people long after
we are gone.  It's a real cost of structuring code based on
legal/licensing considerations rather than technical necessity since
most of the actual coders are more interested in technical than legal
achievements even if the latter may keep their options open better in
the long haul.

-- 
David Kastrup

Re: Maintainers and contributors (was: Contributors and maintainers)

[Artur Malabarba]

On 22 Oct 2015 6:40 am, "John Wiegley" <johnw@newartisans.com> wrote:
> So I'd like to emphasize, to our maintainers, that we must strive to make
> patience, kindness, and consideration, our primary attributes when dealing
> with contributors -- especially those new to the process, who are developing
> their first impressions, and based upon which will either continue to help us,
> or walk away frustrated.

Yes. It's very easy (and only natural) for maintainers to feel at home
here in the list, and it's ok for the maintainers to treat each other
casually.
However, this is not home, it's more like our office. So when someone
new shows up at reception, it's best to treat them with an extra notch
of politeness and respect.

I should say now: I don't mean that the maintainers are unpolite (not
even a little bit). I'm just saying that we don't know the person when
they first submit a contribution, and they don't know us. So it's best
to be extra nice, because that's just how the world works. Some people
have grown in more aggressive areas of the Internet (or the world),
and they'll get defensive _very_ quickly. Other people have grown in
extremely polite areas of the world, and they may feel unwanted if the
same politeness is not extended to them here.

And what do I mean by extra politeness?
1. Start the conversation by saying "Hi ..., thanks for submitting this";
2. When making suggestions that are not essential, say something like
"Your code looks good to apply, but I have a couple of suggestions if
you'd like to improve it further".
3. At random points during a conversation, show small signs of
politeness _especially_ if your email is turning out rather long
(e.g., "Sorry for taking so long", "We're almost getting there now!").

I know, this takes patience.
If we don't think the team has the patience to behave like this, then
one or two who do should be appointed for the job of making first
contact (as sort of embassadors).
(In fact, this might be a good idea even if the team _can_ manage to
be extra polite).

And then there's a fourth point, which is a little harder, but it
would really help demonstrate organization and respect.

4. If the same point goes back and forth twice between you and the
contributor, then stop arguing about it. Bring it up in a separate
place just amongst the maintainers, and then come back and say "Hi
___, I brought this up with X, Y, and Z yesterday, and decided that
___ because ___.".

Even if the decision is against what the contributor wants, this sort
of attitude shows that at least some thought was given to their
arguments.
More importantly, it shows respect, organization, and professionalism.
This may occasionally give off the impression that emacs-devel is
super picky about the code it lets in, but that's better than giving
off the impression that it is a messy team with no structure and that
submitting code is akin to roullete.

Re: [PATCH] Add shell-quasiquote.

[Wolfgang Jenkner]

On Wed, Oct 21 2015, Eli Zaretskii wrote:

> I guess patches to extend shell-quote-argument to cover more shells on
> Posix systems should be welcome, and should get higher priority than
> we thought.

I beg to differ: IMHO, on Unix-like systems, there's no point for
`shell-command' and friends to support anything but /bin/sh, which is
certainly a Bourne shell and hopefully reasonably POSIX compliant.

Of course, things like `shell-mode' should work with any old shell.

Re: Maintainers and contributors

[David Kastrup]

Artur Malabarba <bruce.connor.am@gmail.com> writes:

> And then there's a fourth point, which is a little harder, but it
> would really help demonstrate organization and respect.
>
> 4. If the same point goes back and forth twice between you and the
> contributor, then stop arguing about it. Bring it up in a separate
> place just amongst the maintainers, and then come back and say "Hi
> ___, I brought this up with X, Y, and Z yesterday, and decided that
> ___ because ___.".

I'm not sure about that.  I react _really_ _really_ allergic to people
making decisions involving me in some more or less formal group behind
my back about me in settings that are supposed to constitute a team or
community.

It establishes that I am not considered a member on equal terms with
other members, since a group of members not including myself is supposed
to speak and decide for the group.

> More importantly, it shows respect, organization, and professionalism.

It shows a hierarchy of authority and does not give me an opportunity to
speak for myself.

> This may occasionally give off the impression that emacs-devel is
> super picky about the code it lets in, but that's better than giving
> off the impression that it is a messy team with no structure and that
> submitting code is akin to roullete.

It is.  Problems may or may not be seen in informal review.  There is no
guarantee that anybody may look at code, so some code will tend to get
more scrutiny than other code by mere chance.  If you are hoping for
less scrutiny, that should make you think about what you are actually
doing.  If you are hoping for more, you might be out of luck, but it may
well be worth pointing out that you don't feel particularly secure about
parts of your code in order to get more people take a look at it.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[Jeff Clough]

Wolfgang Jenkner <wjenkner@inode.at> writes:

> On Wed, Oct 21 2015, Eli Zaretskii wrote:
>
>> I guess patches to extend shell-quote-argument to cover more shells on
>> Posix systems should be welcome, and should get higher priority than
>> we thought.
>
> I beg to differ: IMHO, on Unix-like systems, there's no point for
> `shell-command' and friends to support anything but /bin/sh, which is
> certainly a Bourne shell and hopefully reasonably POSIX compliant.

I think this might not be true, depending on what "a Bourne shell" means
to some users. On many systems, /bin/sh is a symlink to whichever shell
floats the vendor's goat.

For example, on the Mint distribution of GNU/Linux, it's a symlink to
dash. While dash is a variant of a variant of the Bourne shell, can we
say with reasonable certainty that those variations don't impact
shell-quote-argument? (I don't know what they are myself, just posing
the question.)

Jeff

Re: Maintainers and contributors

[Artur Malabarba]

2015-10-22 12:08 GMT+01:00 David Kastrup <dak@gnu.org>:
> Artur Malabarba <bruce.connor.am@gmail.com> writes:
>
>> And then there's a fourth point, which is a little harder, but it
>> would really help demonstrate organization and respect.
>>
>> 4. If the same point goes back and forth twice between you and the
>> contributor, then stop arguing about it. Bring it up in a separate
>> place just amongst the maintainers, and then come back and say "Hi
>> ___, I brought this up with X, Y, and Z yesterday, and decided that
>> ___ because ___.".
>
> I'm not sure about that.  I react _really_ _really_ allergic to people
> making decisions involving me in some more or less formal group behind
> my back about me in settings that are supposed to constitute a team or
> community.
>
> It establishes that I am not considered a member on equal terms with
> other members, since a group of members not including myself is supposed
> to speak and decide for the group.
>
>> More importantly, it shows respect, organization, and professionalism.
>
> It shows a hierarchy of authority and does not give me an opportunity to
> speak for myself.

If others agree with that, we can drop the 4th item, I think the first
3 would already be a nice improvement.

Or we can rephrase it to make it sounds less like a decision was made,
and more like we're summarizing the progress so far (e.g. "we
understand that [[A]] is important to you, but we all agree that [[B]]
is a higher priority to us. If you have further concerns besides [[A]]
or if you think we misunderstood [[A]], then do let us know").

The problem is that too often these conversations rotate around the
same point without going anywhere. This message would be a way to make
sure the conversation is progressing, and not an attempt to put a full
stop on it (I see the previous version didn't communicate this well).
Note that this would only be done if the same point has already gone
back and forth twice.
At this point, you _have_ spoken for yourself (twice already), so we
should just ensure that any further communication is to raise new
points, not repeat what has already been repeated once.

Re: Maintainers and contributors

[Dmitry Gutov]

On 10/22/2015 02:55 PM, Artur Malabarba wrote:

> The problem is that too often these conversations rotate around the
> same point without going anywhere. This message would be a way to make
> sure the conversation is progressing, and not an attempt to put a full
> stop on it (I see the previous version didn't communicate this well).
> Note that this would only be done if the same point has already gone
> back and forth twice.

I don't think number 4 would have helped in the latest incident. If the 
submitter is dead-set on an idea and doesn't want to heed, "we all agree 
on disagreeing with you" would probably spark the same reactions that 
we've already seen. The reviewers are tyrants, the mailing list has 
problems and needs a psychologist.

So I rather also put a nice list of rules somewhere that makes it clear 
that at some point you listen to the reviewers, or go away. Maybe with a 
nice of explanation of why that's important.

The amount of time wasted on that recent thread has been staggering.

Re: Maintainers and contributors

[David Kastrup]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 10/22/2015 02:55 PM, Artur Malabarba wrote:
>
>> The problem is that too often these conversations rotate around the
>> same point without going anywhere. This message would be a way to
>> make sure the conversation is progressing, and not an attempt to put
>> a full stop on it (I see the previous version didn't communicate this
>> well).  Note that this would only be done if the same point has
>> already gone back and forth twice.
>
> I don't think number 4 would have helped in the latest incident. If
> the submitter is dead-set on an idea and doesn't want to heed, "we all
> agree on disagreeing with you" would probably spark the same reactions
> that we've already seen. The reviewers are tyrants, the mailing list
> has problems and needs a psychologist.
>
> So I rather also put a nice list of rules somewhere that makes it
> clear that at some point you listen to the reviewers, or go
> away. Maybe with a nice of explanation of why that's important.

Again, this is somewhat complicated by the reviewers not having a
formally different standing from the submitter.  Commit access is a
technical detail, not a ranking.  So in the end it will always boil down
to the ability of working towards a consensus, and part of the consensus
forming does rely on established relations of trust in others'
competence established over longer amounts of time.

So yes, newcomers tend to have a harder stand.  I don't see that as
being specific to Emacs.  Even if Emacs has a comparatively large number
of "ancient" developers that were there from before the beginning of
kernel and are still somewhat active as well as several at least around
for decades.

Several other old projects "suffer" from only a single person of
significant authority and thus may appear to have a smaller barrier of
entry.

I'm not sure whether it would have helped to let Taylan shout it out
with Eli alone.  Maybe it would have helped if Eli had handed off the
discussion completely to someone else at a point of time when it became
obvious that things were going nowhere.

It might have better conveyed it emotionally that this is not about
"winning" anything but getting a contribution in the shape and form that
we keep Emacs in.

> The amount of time wasted on that recent thread has been staggering.

You bet.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

Jeff Clough <jvc@ijmp.net> writes:

> Wolfgang Jenkner <wjenkner@inode.at> writes:
>
>> On Wed, Oct 21 2015, Eli Zaretskii wrote:
>>
>>> I guess patches to extend shell-quote-argument to cover more shells on
>>> Posix systems should be welcome, and should get higher priority than
>>> we thought.
>>
>> I beg to differ: IMHO, on Unix-like systems, there's no point for
>> `shell-command' and friends to support anything but /bin/sh, which is
>> certainly a Bourne shell and hopefully reasonably POSIX compliant.
>
> I think this might not be true, depending on what "a Bourne shell" means
> to some users. On many systems, /bin/sh is a symlink to whichever shell
> floats the vendor's goat.
>
> For example, on the Mint distribution of GNU/Linux, it's a symlink to
> dash. While dash is a variant of a variant of the Bourne shell, can we
> say with reasonable certainty that those variations don't impact
> shell-quote-argument? (I don't know what they are myself, just posing
> the question.)

If something is linked to /bin/sh, it's close enough to a Bourne shell
that its quoting semantics are very clear.  Too many reasonably portable
software depends on that.  This is what "system" is supposed to run.

Details may differ, but shell-quote-argument should not be affected.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[Wolfgang Jenkner]

On Thu, Oct 22 2015, Jeff Clough wrote:

> While dash is a variant of a variant of the Bourne shell, can we
> say with reasonable certainty that those variations don't impact
> shell-quote-argument?

That a backslash quotes any following character (except newlines) seems
to be true since the Thompson shell, cf.

http://v6shell.org/

I've played around a bit with (the FreeBSD port of) sh6 from the site
above using

(defun test--shell-quote-argument (arg &optional shell)
  (let* ((shell-file-name (or shell shell-file-name))
	 (qarg (shell-quote-argument arg))
	 (cmd (format "printf %%s %s" qarg))
	 (res (shell-command-to-string cmd)))
    (list (string-equal arg res) arg qarg res)))


The only problem seems to be with newlines, unsurprisingly:

(test--shell-quote-argument "\n" "sh6")

==> (nil "
" "'
'" "syntax error
")

Re: [PATCH] Add shell-quasiquote.

[Random832]

David Kastrup <dak@gnu.org> writes:
> Eli Zaretskii <eliz@gnu.org> writes:
>> OK, but that still boils down to binding some more variables.  If we
>> want to help users with these factoids, we could have a small database
>> of the known Posix shells and their requirements.

I couldn't find an easy way to do let-style binding of environment
variables, though I didn't look very hard. And the point of what I was
suggesting is to let *packages* do this, not *users*, so the "database"
would have to be part of code that decides what to actually do.

> I think that's overdoing it with regard to shell-quote-argument and
> friends.  We don't need a full POSIX shell, just something with the most
> basic quoting conventions of it.  /bin/sh should be fine here.

I think there might be a basic miscommunication here. If you don't care
about being able to execute arbitrary POSIX shell commands as well as is
possible on a given system, what's even the point of *having* functions
like shell-quote-argument, shell-command, etc, instead of call-process?

In these discussions I've been starting from the assumption that these
functions actually have a point and that being able to execute complex
shell commands (which may use advanced redirection, if/case/for/etc,
parameter expansion, and so forth) from within elisp scripts is a
desirable feature.

> That's all the guarantee you get for calling commands/scripts with
> `system'.  I don't think we should require more than that or try
> providing some guarantees in that regard.

On an actual POSIX system, it's guaranteed that "sh" from PATH (not
necessarily /bin/sh) will be a POSIX shell, and that 'system' will
execute it and not some other shell. (AIUI this is generally still
implemented by a hardcoded path, just one which is not necessarily
/bin/sh)

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

Random832 <random832@fastmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>> Eli Zaretskii <eliz@gnu.org> writes:
>>> OK, but that still boils down to binding some more variables.  If we
>>> want to help users with these factoids, we could have a small database
>>> of the known Posix shells and their requirements.
>
> I couldn't find an easy way to do let-style binding of environment
> variables, though I didn't look very hard. And the point of what I was
> suggesting is to let *packages* do this, not *users*, so the "database"
> would have to be part of code that decides what to actually do.
>
>> I think that's overdoing it with regard to shell-quote-argument and
>> friends.  We don't need a full POSIX shell, just something with the most
>> basic quoting conventions of it.  /bin/sh should be fine here.
>
> I think there might be a basic miscommunication here. If you don't
> care about being able to execute arbitrary POSIX shell commands as
> well as is possible on a given system, what's even the point of
> *having* functions like shell-quote-argument, shell-command, etc,
> instead of call-process?

Executing _basic_ POSIX shell commands with arbitrary correctly quoted
arguments?  "arbitrary POSIX shell commands" is just too ambitious of a
goal.

Please read
<URL:https://www.gnu.org/software/autoconf/manual/autoconf-2.64/html_node/Portable-Shell.html#Portable-Shell>
or
<URL:info:autoconf#Portable%20Shell>
M-x (info "(autoconf) Portable Shell")
for an overview of the terror awaiting you.

> In these discussions I've been starting from the assumption that these
> functions actually have a point and that being able to execute complex
> shell commands (which may use advanced redirection, if/case/for/etc,
> parameter expansion, and so forth) from within elisp scripts is a
> desirable feature.

This bears absolutely no relation to shell-quote-argument as
shell-quote-argument has only a very small well-described job.

>> That's all the guarantee you get for calling commands/scripts with
>> `system'.  I don't think we should require more than that or try
>> providing some guarantees in that regard.
>
> On an actual POSIX system, it's guaranteed that "sh" from PATH (not
> necessarily /bin/sh) will be a POSIX shell, and that 'system' will
> execute it and not some other shell. (AIUI this is generally still
> implemented by a hardcoded path, just one which is not necessarily
> /bin/sh)

Except that "actual POSIX systems" don't actually exist a whole lot.
It's all approximations.  And it's totally irrelevant regarding the job
shell-quote-argument has to do.

-- 
David Kastrup

Re: Contributors and maintainers

[Eli Zaretskii]

> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
> Cc: emacs-devel@gnu.org
> Date: Wed, 21 Oct 2015 21:58:31 +0200
> 
> > Quote #1:
> >
> >   > > On POSIX shells, shell-quote-argument is just as safe as
> >   > > shqq--quote-string, and on non-POSIX shells it works better. So it's a
> >   > > win, in both readability and in portability, to use
> >   > > shell-quote-argument.
> >   > 
> >   > Fixing it does not seem easy at all given I can't trust
> >   > shell-quote-argument.
> >
> >   You can trust it.
> >
> >   > And please be realistic in the amount of trust we can put on the
> >   > complicated implementations for non-Unix shells.  I can't judge them
> >   > myself since I don't know the syntax of those shells at all.  Does
> >   > anyone here know their syntax comprehensively, or checked the
> >   > implementation against the documentation of those shells?
> >
> >   Yes, we do.  Yes, we have.
> 
> This coincided with Random832's demonstration of it being broken for
> csh, and mentioning that the semantics are questionable on Windows if I
> got it right, as well as a mention of a past bug about newline handling
> for POSIX shells.  So I'm afraid you were short of lying to me.  And
> please don't get me wrong but the annoyed and dismissive tone really
> made it obvious that not much thought was put into the response in first
> place, so the factual inaccuracy was not exactly a surprise.

You are changing the subject.  You asked for examples of addressing
your concerns, and I gave you some.  Now it turns out that your real
problem is not with the fact that they were addressed, but with the
fact that the responses disagreed with you.  Which is really what this
is all about: you don't want to accept disagreement.  Disagreement
with you makes you, quite irrationally, agitated, and that prompts you
into looking for rationalizations of your irrational behavior.  So now
I'm suddenly "dismissive", I'm a "liar", and it is "obvious" to you
that the response was "not well thought out".  Of course: if I'd take
your arguments seriously, if I were not a "dismissive liar", then I'd
surely agree with you, right?  Only a "dismissive liar" who doesn't
put much thought into his responses could possibly disagree with you,
right?

> > Quote #2:
> >
> >   > > It might be simpler, but it's wrong, because the result is only
> >   > > correct for Posix shells.
> >   > >
> >   > > Please do use shell-quote-argument instead.
> >   > 
> >   > It's also simpler than the POSIX section of shell-quote-argument.
> >
> >   Simpler doesn't mean correct.
> >
> >   > (defun shell-quote-argument (argument)
> >   >   [...] (cond [...] (t
> >   >     (if (equal argument "")
> >   >         "''"
> >   >       ;; Quote everything except POSIX filename characters.
> >   >       ;; This should be safe enough even for really weird shells.
> >   >       (replace-regexp-in-string
> >   >        "\n" "'\n'"
> >   >        (replace-regexp-in-string "[^-0-9a-zA-Z_./\n]" "\\\\\\&" 
> > argument))))))
> >   > 
> >   > I wonder what "really weird shells" this refers to?
> >
> >   The set of characters special to an arbitrary shell is not known in
> >   advance.
> >
> >   > Certainly not csh, the mechanism it uses for newlines doesn't work
> >   > there.
> >
> >   What did you try that didn't work with csh?
> 
> This is unrelated to the main concern of lacking safety guarantees.

It is related to another "concern" of yours.  See your bug report,
where you echoed this.

> It's also not a response to me.

So what?  It's a concern you expressed, and was part of the thread.

Again, you are changing the subject from the fact of your concerns
being addressed to your refusal to accept our dissenting responses.

> > Quote #3:
> >
> >   > Quoting RMS, coincidentally from a couple days ago:
> >   > 
> >   >     The policy is non-GNU systems are secondary, and lower priority than
> >   >     the GNU system, but we are glad to include support for them in GNU
> >   >     packages if users contribute the necessary code -- provided that
> >   >     code isn't a maintenance problem for us.
> >   > 
> >   >     The maintenainers of any particular package are the ones who judge
> >   >     whether that code is a maintenance problem, since they are the ones
> >   >     it would be a problem for.
> >
> >   I don't see how this is relevant for the issue at hand, since the
> >   necessary code (the shell-quote-argument function) was already
> >   contributed to Emacs years ago, and is used in many places in core
> >   Emacs.  There's no extra effort needed to support more platforms, just
> >   replace one function with another.
> >
> >   > I generally don't want to take responsibility of my code being used on
> >   > non-GNU/non-POSIX systems, but if I can share the responsibility then
> >   > that's fine.
> >
> >   You are sharing the responsibility with a long line of Emacs
> >   developers, all of whom use this function.  I don't see anything you
> >   should worry about, really.
> 
> This tries to push on me a responsibility I cannot take because I don't
> use MS Windows and don't know its shell syntax, and asserts that I
> should stop worrying, i.e. flat-out dismissing my concern.

Once again, a change of subject: I show a direct response to your
concern, and you tell why you didn't like the response.  The fact that
there _was_ a response, i.e. your concern _was_ addressed, is suddenly
no longer of interest to you.

> > As you see, each response is directly related to your text that I
> > cite.  I cannot prove to you that I understood what you were saying,
> > but you can ask any neutral person to read this and tell you what they
> > think about that.  From my side, I can assure you I completely
> > understood everything that you said.
> 
> It seems that we cannot even agree on whether an English sentence does
> or doesn't address a concern raised in another.

Oh yes, we can: you did acknowledge above, albeit implicitly, that
those were responses to your concerns (there were dozens more).  You
just didn't like those responses.  And, for some reason I cannot
imagine, you don't like me personally.  That is all there is to it.

> English is not my mother-tongue either, but I think both our English is
> perfectly good.  Given that, I genuinely have no idea how we can be this
> badly unable to communicate.

It should be clear to any unbiased person who can read English.
(Quite a few of them here have already spoken, but that evidently
wasn't enough for you, probably again because they all said you were
simply wrong.)  There's no problem with communications here, only a
problem with your refusal to accept disagreement and dissent when
submitting patches to a project.

Re: [PATCH] Add shell-quasiquote.

[Random832]

David Kastrup <dak@gnu.org> writes:
> This bears absolutely no relation to shell-quote-argument as
> shell-quote-argument has only a very small well-described job.

shell-quote-argument has *no job at all* without shell-command. Its
purpose is a helper function for building strings to pass to
shell-command. And I am becoming increasingly confused as to what the
purpose of shell-command is.

And anyway I obviously wasn't even talking about shell-quote-argument
anymore, I was talking about the idea of adding a "posix-shell-command"
function [which would naturally need a posix-shell-quote-argument], so
saying "it's irrelevant to shell-quote-argument" is unreasonable.

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

Random832 <random832@fastmail.com> writes:

> David Kastrup <dak@gnu.org> writes:
>> This bears absolutely no relation to shell-quote-argument as
>> shell-quote-argument has only a very small well-described job.
>
> shell-quote-argument has *no job at all* without shell-command.

Disagree.  There are lots of ways for constructing commands and
arguments for shells that don't work via shell-command.  Take M-x shell
RET alone and its TAB completion.  Or the TAB completion for commands
like M-x grep.  Or the operation of Tramp.

> Its purpose is a helper function for building strings to pass to
> shell-command.

To pass to some command executing a shell in its bowels.

> And I am becoming increasingly confused as to what the purpose of
> shell-command is.
>
> And anyway I obviously wasn't even talking about shell-quote-argument
> anymore, I was talking about the idea of adding a
> "posix-shell-command" function [which would naturally need a
> posix-shell-quote-argument], so saying "it's irrelevant to
> shell-quote-argument" is unreasonable.

The problem of "posix-shell-command" is that POSIX shells don't fall
from the tree.  It is something outside of the power of Emacs to provide
or guarantee.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Cc: random832@fastmail.com,  stephen@xemacs.org,  emacs-devel@gnu.org
> Date: Thu, 22 Oct 2015 12:54:51 +0200
> 
> > I guess patches to extend shell-quote-argument to cover more shells on
> > Posix systems should be welcome, and should get higher priority than
> > we thought.
> 
> I beg to differ: IMHO, on Unix-like systems, there's no point for
> `shell-command' and friends to support anything but /bin/sh, which is
> certainly a Bourne shell and hopefully reasonably POSIX compliant.

I see your point.  However, this is contrary to a very old and
documented behavior.  Would such a change be acceptable?

Re: Maintainers and contributors

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Thu, 22 Oct 2015 14:32:02 +0200
> Cc: bruce.connor.am@gmail.com, emacs-devel <emacs-devel@gnu.org>
> 
> Maybe it would have helped if Eli had handed off the discussion
> completely to someone else at a point of time when it became obvious
> that things were going nowhere.

I would consider it unfair to hand off such a "discussion" -- unfair
to the person to whom I'm handing it off.  If someone wants that crown
of thorns, they should volunteer.

And even then, certain attacks of personal nature cannot be ignored or
left without a proper response.

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  random832@fastmail.com,  stephen@xemacs.org,  emacs-devel@gnu.org
> Date: Thu, 22 Oct 2015 14:47:26 +0200
> 
> [/bin/sh] is what "system" is supposed to run.

Maybe I'm wrong, but my reading of Posix indicates that 'system' runs
"sh" in an unspecified directory.

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Wolfgang Jenkner <wjenkner@inode.at>
>> Cc: random832@fastmail.com,  stephen@xemacs.org,  emacs-devel@gnu.org
>> Date: Thu, 22 Oct 2015 12:54:51 +0200
>> 
>> > I guess patches to extend shell-quote-argument to cover more shells on
>> > Posix systems should be welcome, and should get higher priority than
>> > we thought.
>> 
>> I beg to differ: IMHO, on Unix-like systems, there's no point for
>> `shell-command' and friends to support anything but /bin/sh, which is
>> certainly a Bourne shell and hopefully reasonably POSIX compliant.
>
> I see your point.  However, this is contrary to a very old and
> documented behavior.  Would such a change be acceptable?

I think that M-x shell RET (namely interactive shells) should also cover
csh and their ilk, and I consider it quite likely that
shell-quote-argument will be called when preparing arguments to paste
into the comint window.

So I suspect that things are not as easy as that.

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: Random832 <random832@fastmail.com>
> Date: Thu, 22 Oct 2015 09:41:20 -0400
> 
> David Kastrup <dak@gnu.org> writes:
> > Eli Zaretskii <eliz@gnu.org> writes:
> >> OK, but that still boils down to binding some more variables.  If we
> >> want to help users with these factoids, we could have a small database
> >> of the known Posix shells and their requirements.
> 
> I couldn't find an easy way to do let-style binding of environment
> variables, though I didn't look very hard.

Something like this:

  (let ((process-environment process-environment))
    (setq process-environment (cons "FOO=bar" process-environment))
    ...

or like this:

  (let ((process-environment
         (nconc (list "FOO=bar" "OTHER=no") process-environment)))
    ...

Re: [PATCH] Add shell-quasiquote.

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>, random832@fastmail.com,
>> stephen@xemacs.org, emacs-devel@gnu.org
>> Date: Thu, 22 Oct 2015 14:47:26 +0200
>> 
>> [/bin/sh] is what "system" is supposed to run.
>
> Maybe I'm wrong, but my reading of Posix indicates that 'system' runs
> "sh" in an unspecified directory.

<URL:http://pubs.opengroup.org/onlinepubs/9699919799/>

    [CX] [Option Start] The system() function shall behave as if a child
    process were created using fork(), and the child process invoked the
    sh utility using execl() as follows:

    execl(<shell path>, "sh", "-c", command, (char *)0);

    where <shell path> is an unspecified pathname for the sh utility.
    [...]

Well, you're quite right.  It's not even necessary that the executable
file in the unspecified directory is actually called "sh" as long as it
is called with argv[0] being "sh".

-- 
David Kastrup

Re: [PATCH] Add shell-quasiquote.

[Paul Eggert]

On 10/22/2015 08:03 AM, Eli Zaretskii wrote:
>> I beg to differ: IMHO, on Unix-like systems, there's no point for
>> >`shell-command' and friends to support anything but /bin/sh, which is
>> >certainly a Bourne shell and hopefully reasonably POSIX compliant.
> I see your point.  However, this is contrary to a very old and
> documented behavior.  Would such a change be acceptable?
>

I suppose it would be an incompatible change, but I think it'd be a good 
idea. Emacs already distinguishes between interactive shells and shells 
invoked as utilities, and uses explicit-shell-file-name for the former 
but shell-file-name for the latter. We could put into the documentation 
that the latter should be a POSIX-syntax shell (not "POSIX-conforming" 
because strict POSIX conformance is relatively rare). People writing 
Elisp code shouldn't have to worry about the syntax of the C shell, or 
of some MS-Windows shell.

Re: [PATCH] Add shell-quasiquote.

[Andreas Schwab]

David Kastrup <dak@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: David Kastrup <dak@gnu.org>
>>> Cc: Eli Zaretskii <eliz@gnu.org>, random832@fastmail.com,
>>> stephen@xemacs.org, emacs-devel@gnu.org
>>> Date: Thu, 22 Oct 2015 14:47:26 +0200
>>> 
>>> [/bin/sh] is what "system" is supposed to run.
>>
>> Maybe I'm wrong, but my reading of Posix indicates that 'system' runs
>> "sh" in an unspecified directory.
>
> <URL:http://pubs.opengroup.org/onlinepubs/9699919799/>
>
>     [CX] [Option Start] The system() function shall behave as if a child
>     process were created using fork(), and the child process invoked the
>     sh utility using execl() as follows:
>
>     execl(<shell path>, "sh", "-c", command, (char *)0);
>
>     where <shell path> is an unspecified pathname for the sh utility.
>     [...]
>
> Well, you're quite right.  It's not even necessary that the executable
> file in the unspecified directory is actually called "sh" as long as it
> is called with argv[0] being "sh".

That's because file system hierarchy is outside the scope of POSIX
(execept for a few items as specified in 10.1 Directory Structure and
Files).  In Unix, <shell path> is always "/bin/sh".

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> Cc: random832@fastmail.com, stephen@xemacs.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Thu, 22 Oct 2015 08:41:19 -0700
> 
> On 10/22/2015 08:03 AM, Eli Zaretskii wrote:
> >> I beg to differ: IMHO, on Unix-like systems, there's no point for
> >> >`shell-command' and friends to support anything but /bin/sh, which is
> >> >certainly a Bourne shell and hopefully reasonably POSIX compliant.
> > I see your point.  However, this is contrary to a very old and
> > documented behavior.  Would such a change be acceptable?
> >
> 
> I suppose it would be an incompatible change, but I think it'd be a good 
> idea.

Then maybe we should do this now.

> People writing Elisp code shouldn't have to worry about the syntax
> of the C shell, or of some MS-Windows shell.

The change I had in mind was one limited to Posix platforms.

Commands that should run on all supported platforms cannot assume a
Posix shell on all of them.  The issue at hand is unrelated to
MS-Windows, since $SHELL is not set there unless the user sets it, and
Emacs on MS-Windows overrides it (by pointing to cmdproxy) even if it
is set.  And since there are no good native ports of a Posix shell to
MS-Windows, we cannot ask our users to install such a shell as a
prerequisite for Emacs shell-command and its ilk.

Re: [PATCH] Add shell-quasiquote.

[Stephen J. Turnbull]

Random832 writes:

 > And I am becoming increasingly confused as to what the purpose of
 > shell-command is.

As a command, it's a convenience for users who want to give commands
to a shell, rather than write a Lisp script, to interact with the
underlying system.  As a function, it's an API for embedding the same
functionality in a Lisp program.

Re: [PATCH] Add shell-quasiquote.

[Wolfgang Jenkner]

On Thu, Oct 22 2015, Eli Zaretskii wrote:

>> I beg to differ: IMHO, on Unix-like systems, there's no point for
>> `shell-command' and friends to support anything but /bin/sh, which is
>> certainly a Bourne shell and hopefully reasonably POSIX compliant.
>
> I see your point.  However, this is contrary to a very old and
> documented behavior.  Would such a change be acceptable?

It would break shell functions and other stuff defined in some of the
init files belonging to $SHELL (even if sh is a symlink to bash).

So... the status quo is fine, perhaps after all?

In any case, I still think that on Unix-like systems there's little
point in supporting shells with different quoting conventions.

Re: Maintainers and contributors

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> I would consider it unfair to hand off such a "discussion" -- unfair to the
> person to whom I'm handing it off. If someone wants that crown of thorns,
> they should volunteer.

I'm happy to have that crown of thorns, as you call it. It would make you more
productive, Eli, by not draining away your energy in these sorts of disputes.
On the other hand, I have a personal need to improve my ability to negotiate
these types of scenarios (outside of Emacs), so it gives me a chance to hone
my fledgling skills.

I think Artur's suggestion of shifting the discussion after it bounces back
twice on the same point has merit. Not in the shadowy-cabal-backroom sort of
way that David is opposed to, but in the "We hear your concern and are going
to escalate its importance" sense. If I were arguing a point with a single
maintainer, and that maintainer said, "Clearly this issue is of importance to
you, I'm going to bring another maintainer into this discussion", I would feel
very listened to.

As it stands, the falling out with Taylan was not entirely technical. I've
spoken to Taylan on IRC, and he is actually a very reasonable fellow. Mainly,
there was a difference between his desire, and his position, that we missed:

  Desire: Avoid security vulnerabilities in his code.

  Position: `shell-quote-argument' violates this desire, and should not be
          used. Since emacs-devel probably can't fix `shell-quote-argument'
          today, rewrite it until it is fixed.

Had the discussion been about this desire, we could have talked about whether
he should bother worrying about security in the context of Emacs, since we
generally don't put much focus there. Eli did start to mention this, but I
think it was lost in the storm, or seen as a dodge.

Because the `shell-quote-argument' position was stated early in the bug
thread, the discussion devolved into a "hard bargaining" scenario, where
Taylan could not accept using `shell-quote-argument' as it stood, and we could
not accept his re-implementing it. From that moment on there was really no
agreement possible, not without sacrifice. This is when things started to get
nasty, because the submitter thought we were completely ignoring his primary
issue.

Here's what our side looked like:

    Desire: Make Emacs as easy to maintain as possible.

    Position: Re-implementing `shell-quote-argument' is unnecessary; if it
            has problems, we should fix it, rather than increasing our code
            surface.

Was there a solution to resolve both of these desires? I bet you there was.
Was there one to unite the two positions? Doubtful; at least, not in a manner
satisfying to both parties.

I'm not asking the maintainers on emacs-devel to become negotiators who must
worry about layers of meaning: only that Artur's suggestion of escalating
disputatious issues -- so we can step back together and reassess the needs of
the submitter -- could have made it possible to avoid all that was lost in the
past week.

John

Re: Maintainers and contributors

[Jay Belanger]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Thu, 22 Oct 2015 14:32:02 +0200
>> Cc: bruce.connor.am@gmail.com, emacs-devel <emacs-devel@gnu.org>
>> 
>> Maybe it would have helped if Eli had handed off the discussion
>> completely to someone else at a point of time when it became obvious
>> that things were going nowhere.
>
> I would consider it unfair to hand off such a "discussion" -- unfair
> to the person to whom I'm handing it off.  If someone wants that crown
> of thorns, they should volunteer.

Even then, if this became some sort of policy it seems that it might
encourage potential contributors to avoid compromise and hope that
someone else down the line will cede to the potential contributor's
demands. 

> And even then, certain attacks of personal nature cannot be ignored or
> left without a proper response.

Re: Maintainers and contributors

[Dmitry Gutov]

On 10/22/2015 09:27 PM, John Wiegley wrote:

> As it stands, the falling out with Taylan was not entirely technical. I've
> spoken to Taylan on IRC, and he is actually a very reasonable fellow. Mainly,
> there was a difference between his desire, and his position, that we missed:

He indeed gives that impression, most of the time.

>    Desire: Avoid security vulnerabilities in his code.

That desire itself ("in his code" vs "in Emacs") doesn't make much 
sense, because you cannot use his code without using Emacs. And Emacs 
uses shell-quote-argument in many places. If that function is 
vulnerable, you're most likely screwed anyway.

>    Position: `shell-quote-argument' violates this desire, and should not be
>            used. Since emacs-devel probably can't fix `shell-quote-argument'
>            today, rewrite it until it is fixed.
>
> Had the discussion been about this desire, we could have talked about whether
> he should bother worrying about security in the context of Emacs, since we
> generally don't put much focus there. Eli did start to mention this, but I
> think it was lost in the storm, or seen as a dodge.

This point and others have been made. Like Eli said in another email, 
some people just can't accept different views.

Re: Maintainers and contributors

[John Wiegley]

> This point and others have been made. Like Eli said in another email, some people just can't accept different views.

I actually think we don't differ that much in our interests. Everyone cares about security to some extent, and everyone wants to do the least work for the greatest gain. 

My point is that we could have done a better job at addressing the underlying concern, rather than the line that ended up being drawn in the sand.

John (from my iPhone)

Re: Maintainers and contributors

[Jay Belanger]

John Wiegley <jwiegley@gmail.com> writes:

> My point is that we could have done a better job at addressing the
> underlying concern, rather than the line that ended up being drawn in
> the sand.

Who drew a line in the sand?   I didn't see any maintainers draw a line
in the sand; on the contrary, Eli went well out of his way to be
accommodating.

Re: Becoming an Emacs contributor

[Nix]

On 20 Oct 2015, David Kastrup uttered the following:

> taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: taylanbayirli@gmail.com (Taylan Ulrich Bayırlı/Kammer)
>>>> Date: Tue, 20 Oct 2015 14:56:08 +0200
>>>> Cc: emacs-devel@gnu.org
>>>> 
>>>> This actually sounds pretty similar to what happened in this thread in
>>>> some ways, although it differs in other ways, so thanks for the input!
>>>
>>> You were given a single comment that required a one-line change in the
>>> code.
>>
>> You were provided a patch that you only needed to apply.
>>
>> I win!
>
> Mistaking this for a competition might explain some of this thread.

Quite. The purpose of a maintainer is not to apply changes without
consideration or adjustment. (That would be the job of a committing
robot.)

-- 
NULL && (void)

Re: [PATCH] Add shell-quasiquote.

[Steinar Bang]

>>>>> David Kastrup <dak@gnu.org>:

> "Stephen J. Turnbull" <stephen@xemacs.org> writes:

>> Sometimes it pays to be precise.  All of the OS distributions gave up
>> on bash because it didn't quite conform to POSIX (even when invoked in
>> POSIX compatibility mode), and that caused bugs in package
>> installation and management for packages that used sh scripts rather
>> than perl or python.  That's why shells like ash and dash exist.

> I think it was more a question of size and startup speed.

I thought the change to dash was to avoid vulnerabilities like
shellshock...?  https://en.wikipedia.org/wiki/Shellshock_(software_bug)

(I only found out about the existence of dash when I checked my debian
machines for shellshock vulnerability (and to my relief found that
/bin/sh wasn't bash at all...))

Re: [PATCH] Add shell-quasiquote.

[Kai Großjohann]

Michael Albinus <michael.albinus@gmx.de> writes:

> There is no special check on a remote shell being csh. But most of the
> shell commands Tramp emits require a bournish shell. Otherwise, there
> would be syntax errors soon, and Tramp would cease to continue on that
> host.

I thought Tramp does "exec /bin/sh" soon after connecting to the remote
host.  Evil remote systems might arrange that this starts csh, I'm not
clear what to do about that, though.

Kai

Re: [PATCH] Add shell-quasiquote.

[Kai Großjohann]

Random832 <random832@fastmail.com> writes:

> Speaking of Tramp, what if the local shell is not the same as the
> remote shell? And I don't see how the commands it runs "require a
> bournish shell" at all. they require that the commands themselves
> exist, but that's nothing to do with the shell.

In tramp-do-directory-files-and-attributes-with-stat, it uses "2>/dev/null"
which I believe does not work in csh.  I think there are other similar
subtleties.

A more obvious thing is that Tramp sets PS1 and csh doesn't know about PS1.

> It even has a TODO item:
>
> ;; * Rewrite `tramp-shell-quote-argument' to abstain from using
> ;;   `shell-quote-argument'.
>
> So much for not reinventing the wheel.

tramp-shell-quote-argument let-binds system-type to make
shell-quote-argument do the intended thing.  This is iffy.
shell-quote-argument is documented to do the right thing for the local
system, and who knows if tomorrow it still uses system-type to perform
that check?

Kai

Re: [PATCH] Add shell-quasiquote.

[Michael Albinus]

kai@emptydomain.de (Kai Großjohann) writes:

> Michael Albinus <michael.albinus@gmx.de> writes:
>
>> There is no special check on a remote shell being csh. But most of the
>> shell commands Tramp emits require a bournish shell. Otherwise, there
>> would be syntax errors soon, and Tramp would cease to continue on that
>> host.
>
> I thought Tramp does "exec /bin/sh" soon after connecting to the remote
> host.

Yes. But sometimes, this points to zsh. Sometimes, it points to rbash
(restricted bash). Sometimes, it points to busybox. And so on.

All of this infinite fun for the maintainer. Well, not because of shell
argument quoting, at least.

> Kai

Best regards, Michael.

Re: [PATCH] Add shell-quasiquote.

[Kai Großjohann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Random832 <random832@fastmail.com>
>> Date: Wed, 21 Oct 2015 10:18:49 -0400
>> 
>> It doesn't have a documented way for the caller to insist that the
>> string be quoted for a POSIX shell rather than the user's shell.
>
> On what OS would that distinction be important, and why?

Tramp is an example: it runs a Bourne-ish shell on the remote host, and
thus it would like to quote shell arguments for that shell.

Kai

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: kgrossjo@emptydomain.de (Kai Großjohann)
> Date: Sun, 01 Nov 2015 19:39:16 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Random832 <random832@fastmail.com>
> >> Date: Wed, 21 Oct 2015 10:18:49 -0400
> >> 
> >> It doesn't have a documented way for the caller to insist that the
> >> string be quoted for a POSIX shell rather than the user's shell.
> >
> > On what OS would that distinction be important, and why?
> 
> Tramp is an example: it runs a Bourne-ish shell on the remote host, and
> thus it would like to quote shell arguments for that shell.

That's easy to fix by adding an optional 2nd argument, I think.

Re: [PATCH] Add shell-quasiquote.

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> It doesn't have a documented way for the caller to insist that the
>> >> string be quoted for a POSIX shell rather than the user's shell.
>> >
>> > On what OS would that distinction be important, and why?
>> 
>> Tramp is an example: it runs a Bourne-ish shell on the remote host, and
>> thus it would like to quote shell arguments for that shell.
>
> That's easy to fix by adding an optional 2nd argument, I think.

If Tramp would be the only package addressed by this change, it's not
necessary. Tramp knows how to handle the situation, and it wouldn't give
up its own solution for a long time due to backward compatibility.

But feel free to install such a change if you think it is worth. And
this optional argument shall also say the type of shell quoting
(bournish, windows-like, ...). In tramp-smb.el we have exact the reverse
situation: The local shell is likely to be bournish, but the remote host
(accessed via smb-client) expects a windows-like argument quoting.

Best regards, Michael.

Re: [PATCH] Add shell-quasiquote.

[Kai Großjohann]

David Kastrup <dak@gnu.org> writes:

> I think that M-x shell RET (namely interactive shells) should also cover
> csh and their ilk, and I consider it quite likely that
> shell-quote-argument will be called when preparing arguments to paste
> into the comint window.

One thing that could be improved, imho, is that shell-quote-argument
looks at the system it's running on and quotes for that.  But actually
it should quote for the shell that receives the result, and it's not easy
to know what that shell is.

For example, especially on Windows it's conceivable that some folks
would like to quote for cmd.exe, others would like to quote for
Powershell, and still others are Cygwin aficionados and would like to
quote for bash or zsh...

One idea might be that shell-quote-argument could look at shell-file-name.
I have no idea whether that would fly.

Kai

Re: [PATCH] Add shell-quasiquote.

[Eli Zaretskii]

> From: kgrossjo@emptydomain.de (Kai Großjohann)
> Date: Sat, 07 Nov 2015 00:35:35 +0100
> 
> One thing that could be improved, imho, is that shell-quote-argument
> looks at the system it's running on and quotes for that.  But actually
> it should quote for the shell that receives the result, and it's not easy
> to know what that shell is.
> 
> For example, especially on Windows it's conceivable that some folks
> would like to quote for cmd.exe, others would like to quote for
> Powershell, and still others are Cygwin aficionados and would like to
> quote for bash or zsh...
> 
> One idea might be that shell-quote-argument could look at shell-file-name.
> I have no idea whether that would fly.

This already works as you describe in the Windows build, see
w32-shell-dos-semantics.