bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Da Zhang]

[-- Attachment #1: Type: text/html, Size: 16841 bytes --]

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Eli Zaretskii]

> Date: Fri, 05 Jun 2015 12:33:59 -0400
> From: Da Zhang <zhangda82@gmail.com>
> 
> I use python mode (python.el) for some python programming. When I invoke
> the command python-shell-send-buffer, my Emacs always reports "Searching
> for program: no such file or directory,
> c\:/Anaconda3/Scripts/ipython.exe".
> 
> I traced down the cause of this problem, and found the following lines
> in python.el:
> (defun python-shell-parse-command () ;FIXME: why name it "parse"?
> "Calculate the string used to execute the inferior Python process."
> ;; FIXME: process-environment doesn't seem to be used anywhere within
> ;; this let.
> (let ((process-environment (python-shell-calculate-process-environment))
> (exec-path (python-shell-calculate-exec-path)))
> (format "%s %s"
> ;; FIXME: Why executable-find?
> (shell-quote-argument (executable-find python-shell-interpreter))
> python-shell-interpreter-args)))
> 
> The function call 
> (shell-quote-argument (executable-find python-shell-interpreter))
> changed the path of my python interpreter from
> "c:/Anaconda3/Scripts/ipython.exe"
> to
> "c\\:/Anaconda3/Scripts/ipython.exe".
> 
> I worked around this problem by replacing the function call
> (shell-quote-argument (executable-find python-shell-interpreter))
> with
> (executable-find python-shell-interpreter) ;; edited by DZ on
> 2015/06/05: (shell-quote-argument) caused the problem of
> "c\\:/Anaconda3/Scripts/ipython.exe"
> 
> To further analyze the problem, I found the function
> shell-quote-argument defined in subr.el does not deal with the case when
> bash from cygwin is used.

For the record, this is bug #20237.

> I added the following lines in the cond statement, to solve the problem.
> ((and (eq system-type 'windows-nt) (if (string-match ".*cygwin.*"
> (w32-shell-name)) t nil)) ;; DZ's edit on 6/5/2015
> (if (equal argument "")
> "''"
> ;; Quote everything except POSIX filename characters.
> ;; This should be safe enough even for really weird shells.
> (replace-regexp-in-string
> "\n" "'\n'"
> (replace-regexp-in-string "[^-0-9a-zA-Z_./\n:]" "\\\\\\&" argument))))

I'm sorry, but I don't see how we can accept such a change: it means
that when the Cygwin Bash is used as the shell, the ':' character will
not be quoted, which I think could cause trouble in some cases.

Maybe some Bash expert could chime in and tell whether problems are
possible with Bash in this case.

Is there any way you could avoid using the Cygwin Bash as the shell
when running the native Windows build of Emacs?  That would make the
problem go away without any need for changes.

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Da Zhang]

[-- Attachment #1: Type: text/html, Size: 4910 bytes --]

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Da Zhang]

[-- Attachment #1: Type: text/html, Size: 3613 bytes --]

[-- Attachment #2: gaefedad.png --]
[-- Type: image/png, Size: 18480 bytes --]

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Glenn Morris]

Eli Zaretskii wrote:

> I'm sorry, but I don't see how we can accept such a change: it means
> that when the Cygwin Bash is used as the shell, the ':' character will
> not be quoted, which I think could cause trouble in some cases.
>
> Maybe some Bash expert could chime in and tell whether problems are
> possible with Bash in this case.

Not an expert (and didn't read the rest of the report), but I can't
think of a reason why ':' would need quoting in Bash (unlike eg ';' or '&').
What kind of thing are you thinking of?

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Da Zhang <zhangda82@gmail.com>,  20744@debbugs.gnu.org
> Date: Sun, 07 Jun 2015 14:01:31 -0400
> 
> Eli Zaretskii wrote:
> 
> > I'm sorry, but I don't see how we can accept such a change: it means
> > that when the Cygwin Bash is used as the shell, the ':' character will
> > not be quoted, which I think could cause trouble in some cases.
> >
> > Maybe some Bash expert could chime in and tell whether problems are
> > possible with Bash in this case.
> 
> Not an expert (and didn't read the rest of the report), but I can't
> think of a reason why ':' would need quoting in Bash (unlike eg ';' or '&').
> What kind of thing are you thinking of?

PATH-style lists, maybe.

Anyway, if we think ':' doesn't need to be quoted, how about removing
it from the Posix branch of shell-quote-argument?  Then we won't need
ugly hacks that try to detect Cygwin Bash used from a native Windows
Emacs.

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Glenn Morris]

Eli Zaretskii wrote:

> Anyway, if we think ':' doesn't need to be quoted, how about removing
> it from the Posix branch of shell-quote-argument?

The comment there says "This should be safe enough even for really weird
shells." That's a much broader question than just Bash! :)

But I thought of an example that affects Bash:

scp localfile remotehost:

If "localfile" contains ":", things will get confused.
So it's probably best to leave it as-is.

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Glenn Morris]

Glenn Morris wrote:

> scp localfile remotehost:

Well, that's not a shell issue, though. It's an scp issue.
So it depends exactly what shell-quote-argument is for...

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Glenn Morris]

I think the actual answer for Bash is the "metacharacters" list in the
Bash manual, plus the "!" character (for history), as per:

http://www.gnu.org/software/bash/manual/html_node/Quoting.html

That doesn't include ":", but then there's those "really weird shells",
whatever they may be...

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: 20744@debbugs.gnu.org,  zhangda82@gmail.com
> Date: Sun, 07 Jun 2015 20:47:09 -0400
> 
> 
> I think the actual answer for Bash is the "metacharacters" list in the
> Bash manual, plus the "!" character (for history), as per:
> 
> http://www.gnu.org/software/bash/manual/html_node/Quoting.html
> 
> That doesn't include ":", but then there's those "really weird shells",
> whatever they may be...

Perhaps we could have a branch for Bash only.

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> That doesn't include ":", but then there's those "really weird shells",
>> whatever they may be...
>
> Perhaps we could have a branch for Bash only.

Tramp would happily join that branch, for "really weird shells on a
remote host".

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: Glenn Morris <rgm@gnu.org>,  20744@debbugs.gnu.org,  zhangda82@gmail.com
> Date: Mon, 08 Jun 2015 09:43:32 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> That doesn't include ":", but then there's those "really weird shells",
> >> whatever they may be...
> >
> > Perhaps we could have a branch for Bash only.
> 
> Tramp would happily join that branch, for "really weird shells on a
> remote host".

Meaning you are for such a change, or against it?  (Sorry, it wasn't
clear from what you said.)

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> That doesn't include ":", but then there's those "really weird shells",
>> >> whatever they may be...
>> >
>> > Perhaps we could have a branch for Bash only.
>> 
>> Tramp would happily join that branch, for "really weird shells on a
>> remote host".
>
> Meaning you are for such a change, or against it?  (Sorry, it wasn't
> clear from what you said.)

Bash is always a pain to handle, there are exceptions in Tramp's code
for that. But having a branch for this is not realistic for Tramp,
because Tramp doesn't distinguish remote shells, it only expects that
they speak a bourne-shell dialect.

My message was rather a sigh than a request.

Best regards, Michael.

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: rgm@gnu.org,  20744@debbugs.gnu.org,  zhangda82@gmail.com
> Date: Mon, 08 Jun 2015 17:14:33 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> >> That doesn't include ":", but then there's those "really weird shells",
> >> >> whatever they may be...
> >> >
> >> > Perhaps we could have a branch for Bash only.
> >> 
> >> Tramp would happily join that branch, for "really weird shells on a
> >> remote host".
> >
> > Meaning you are for such a change, or against it?  (Sorry, it wasn't
> > clear from what you said.)
> 
> Bash is always a pain to handle, there are exceptions in Tramp's code
> for that. But having a branch for this is not realistic for Tramp,
> because Tramp doesn't distinguish remote shells, it only expects that
> they speak a bourne-shell dialect.
> 
> My message was rather a sigh than a request.

How about Stefan's suggestion to stop quoting the colon and see if
anybody hollers?  Are you okay with that, or will you holler?

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> How about Stefan's suggestion to stop quoting the colon and see if
> anybody hollers?  Are you okay with that, or will you holler?

That would be OK. We'll hear it, when Tramp users start to holler.

Best regards, Michael.

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Stefan Monnier]

>> Anyway, if we think ':' doesn't need to be quoted, how about removing
>> it from the Posix branch of shell-quote-argument?
> The comment there says "This should be safe enough even for really weird
> shells." That's a much broader question than just Bash! :)

Let's stop escaping the : and if/when someone comes up with a very weird
shell where this is a problem, we'll see what can be done.


        Stefan

bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[npostavs]

retitle 20744 python.el tries to undo shell quoting with split-string-and-unquote
tags 20744 confirmed
found 20744 25.1
quit

Da Zhang <zhangda82@gmail.com> writes:

> From: Da Zhang <zhangda82@gmail.com>
> To: bug-gnu-emacs@gnu.org
> Subject: 24.5; shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly
> --text follows this line--
>
> Dear Emacs developers/maintainers,
>
> I am writing this email to report an issue that has been bothering me
> for a while.
>
> I use python mode (python.el) for some python programming. When I invoke
> the command python-shell-send-buffer, my Emacs always reports "Searching
> for program: no such file or directory,
> c\:/Anaconda3/Scripts/ipython.exe".
>
[...]
>
> The function call 
> (shell-quote-argument (executable-find python-shell-interpreter))
> changed the path of my python interpreter from
> "c:/Anaconda3/Scripts/ipython.exe"
> to
> "c\\:/Anaconda3/Scripts/ipython.exe".
>
> I worked around this problem by replacing the function call
> (shell-quote-argument (executable-find python-shell-interpreter))
> with
> (executable-find python-shell-interpreter) ;; edited by DZ on
> 2015/06/05: (shell-quote-argument) caused the problem of
> "c\\:/Anaconda3/Scripts/ipython.exe"
>
> To further analyze the problem, I found the function
> shell-quote-argument defined in subr.el does not deal with the case when
> bash from cygwin is used.
>
[...]

Actually, bash, regardless of being on cygwin or not, should have no
problem with an escaped ":".  The actual problem here is that python.el
is using shell-quote-argument but then not using a shell.  The result of
python-shell-calculate-command (formerly known as
python-shell-parse-command) gets passed to python-shell-make-comint as
the CMD argument:

(defun python-shell-make-comint (cmd proc-name &optional show internal)
  [...]
          (let* ((cmdlist (split-string-and-unquote cmd))
                 (buffer (apply #'make-comint-in-buffer proc-name proc-buffer-name
                                interpreter nil args))

make-comint-in-buffer uses start-file-process, i.e., no shell is called.
So python.el is trying to undo shell-quote-argument with
split-string-and-unquote, this is the bug.

To see this on any system, compare

    C-u M-x run-python RET 'python' -i

this fails, while

    C-u M-x run-python RET "python" -i

this succeeds (assuming you have python in exec-path).