bug#36270: executable-find does not find shell commands on MSYS2

bug#36270: executable-find does not find shell commands on MSYS2

[Juanma Barranquero]

c:\Tools\msys64 is the root of my MSYS2 / MingW64 installation.

From the mingw64 console:

Juanma@ODIEFAST MINGW64 /d/Devel/emacs/repo/trunk
$ mount
C:/Tools/msys64 on / type ntfs (binary,noacl,auto)
C:/Tools/msys64/usr/bin on /bin type ntfs (binary,noacl,auto)
C: on /c type ntfs (binary,noacl,posix=0,user,noumount,auto)

Juanma@ODIEFAST MINGW64 /d/Devel/emacs/repo/trunk
$ which ci
/usr/bin/ci

Juanma@ODIEFAST MINGW64 /d/Devel/emacs/repo/trunk
$ ls -l /usr/bin/ci
-rwxr-xr-x 1 Juanma Juanma 806 Jun  1  2017 /usr/bin/ci

Juanma@ODIEFAST MINGW64 /d/Devel/emacs/repo/trunk
$ file /usr/bin/ci
/usr/bin/ci: POSIX shell script, ASCII text executable

Juanma@ODIEFAST MINGW64 /d/Devel/emacs/repo/trunk
$ ci --version
ci (GNU RCS) 5.9.4
Copyright (C) 2010-2017 Thien-Thi Nguyen
Copyright (C) 1990-1995 Paul Eggert
Copyright (C) 1982,1988,1989 Walter F. Tichy, Purdue CS
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.



Then, from Emacs (master or release branch):

ELISP> (setq shell-file-name "c:/Tools/msys64/usr/bin/bash.exe")
"c:/Tools/msys64/usr/bin/bash.exe"
ELISP> (shell-command-to-string "ci --version")
"ci (GNU RCS) 5.9.4
Copyright (C) 2010-2017 Thien-Thi Nguyen
Copyright (C) 1990-1995 Paul Eggert
Copyright (C) 1982,1988,1989 Walter F. Tichy, Purdue CS
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
"
ELISP> (executable-find "ci")
nil
ELISP> (locate-file "ci" exec-path exec-suffixes)
"c:/Tools/msys64/usr/bin/ci"
ELISP> (locate-file "ci" exec-path exec-suffixes 1)
nil


executable-find passes 1 to locate-file:

    ;; Use 1 rather than file-executable-p to better match the
    ;; behavior of call-process.
    (let ((default-directory (file-name-quote default-directory 'top)))
      (locate-file command exec-path exec-suffixes 1))))


Now, it is true that call-process will not find the shell command:

ELISP> (call-process "c:/Tools/msys64/usr/bin/ci" nil t)
*** Eval error ***  Searching for program: No such file or directory,
c:/Tools/msys64/usr/bin/ci


but 'executable-find' is not defined in terms of call-process, but
finding commands in the exec-path:

(executable-find COMMAND &optional REMOTE)

Search for COMMAND in ‘exec-path’ and return the absolute file name.
Return nil if COMMAND is not found anywhere in ‘exec-path’.


so there's at least some tension between these two interpretations,
IMO. It's weird that you can make shell-command to execute a program
(with a relative, not absolute, name, like "ci"), and yet
executable-find fails to find it in the exec-path, where it, in fact,
is located.

bug#36270: executable-find does not find shell commands on MSYS2

[Glenn Morris]

https://debbugs.gnu.org/7784 ?

bug#36270: executable-find does not find shell commands on MSYS2

[Juanma Barranquero]

merge 7784 36270
quit


 I hadn't found that one. Thanks.

And yes, it's obviously mostly the same issue. Merged.

Still, there's nothing magic about call-process. Is the native way,
but not the only way to execute programs on Emacs. Why should
file-executable-p and executable-find privilege native vs shell
commands, in contexts where you can run the shell command easily?

(let ((shell-file-name "bash"))
  (shell-command "ci --version"))

=> 0

As I said above, there's tension between two meanings of "executable"
here. Both are meaningful.

bug#36270: executable-find does not find shell commands on MSYS2

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Tue, 18 Jun 2019 02:57:59 +0200
> Cc: 36270@debbugs.gnu.org
> 
> Still, there's nothing magic about call-process. Is the native way,
> but not the only way to execute programs on Emacs. Why should
> file-executable-p and executable-find privilege native vs shell
> commands, in contexts where you can run the shell command easily?

Because otherwise we would have inconsistencies: executable-find will
find an "executable" which Emacs may or may not be able to run.  We
want to declare a file "executable" only if it can be invoked by _all_
possible means of running an executable file.

This is a Windows-specific subtlety, because the mechanism for running
arbitrary scripts is different from that of Posix systems.  We could
make the situation a tad better if we add support for the PATHEXT
environment variable, as suggested in bug#7784, patches welcome.  But
this won't solve the problem entirely, when the user has some
interpreter installed which doesn't register itself in PATHEXT.

> (let ((shell-file-name "bash"))
>   (shell-command "ci --version"))
> 
> => 0
> 
> As I said above, there's tension between two meanings of "executable"
> here. Both are meaningful.

I might be okay with adding a new function, shell-executable-find,
say.  But we must first define what that means, exactly, and whether
it's feasible to implement something like that with reasonable effort
and complexity.

Meanwhile, just set up trivial batch file wrappers for your shell
scripts, and Bob's your uncle.

bug#36270: executable-find does not find shell commands on MSYS2

[Juanma Barranquero]

On Tue, Jun 18, 2019 at 6:51 PM Eli Zaretskii <eliz@gnu.org> wrote:

> We want to declare a file "executable" only if it can be invoked by _all_
> possible means of running an executable file.

Sure. But let's perhaps agree that there's something to be said for
files that are "almost executable". shell-command and variants are
used (counting *very* roughly) about 150 times in the elisp sources,
vs. ~300 invocations of call-process.

> We could
> make the situation a tad better if we add support for the PATHEXT
> environment variable, as suggested in bug#7784, patches welcome.

I've read the bug thread. What exactly does mean to "add support for
the PATHEXT environment variable"? Making file-executable-p take it
into account? Are files in the path with one of PATHEXT extensions
runnable with call-process?

> I might be okay with adding a new function, shell-executable-find,
> say.  But we must first define what that means, exactly, and whether
> it's feasible to implement something like that with reasonable effort
> and complexity.

Yeah, no rush.

> Meanwhile, just set up trivial batch file wrappers for your shell
> scripts, and Bob's your uncle.

Sure. My shell scripts weren't the problem, anyhow. I know how to work
around the issue.

I stumbled upon this because I was trying to understand failures in
vc-tests.el on Windows, and the reason is that it calls call-process
to run some shell scripts, like ci. I'd like to fix that and run the
tests, but I haven't really stared at it long enough to know if
there's any non-ugly fix.

bug#36270: executable-find does not find shell commands on MSYS2

[Eli Zaretskii]

> From: Juanma Barranquero <lekktu@gmail.com>
> Date: Tue, 18 Jun 2019 22:49:33 +0200
> Cc: Glenn Morris <rgm@gnu.org>, 36270@debbugs.gnu.org
> 
> On Tue, Jun 18, 2019 at 6:51 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > We want to declare a file "executable" only if it can be invoked by _all_
> > possible means of running an executable file.
> 
> Sure. But let's perhaps agree that there's something to be said for
> files that are "almost executable".

Kids down here have a saying: "almost doesn't count" ;-)

> shell-command and variants are used (counting *very* roughly) about
> 150 times in the elisp sources, vs. ~300 invocations of
> call-process.

Exactly.  So what would be the point of having a general-purpose API
return files that can be useful only with some methods of running a
program?  It will only cause confusion and bug reports.

> > We could
> > make the situation a tad better if we add support for the PATHEXT
> > environment variable, as suggested in bug#7784, patches welcome.
> 
> I've read the bug thread. What exactly does mean to "add support for
> the PATHEXT environment variable"? Making file-executable-p take it
> into account?

That's one part, but it isn't the most important (nor the hardest)
part.

> Are files in the path with one of PATHEXT extensions runnable with
> call-process?

No, PATHEXT is consulted by cmd.exe and AFAIK also the ShellExecute
API.  So to support that, we will need to change sys_spawnve and/or
cmdproxy to detect an extension that appears in PATHEXT, and then
either invoke through cmdproxy or use ShellExecute instead of
CreateProcess.  And this is the more complex part of the job.

> I stumbled upon this because I was trying to understand failures in
> vc-tests.el on Windows, and the reason is that it calls call-process
> to run some shell scripts, like ci. I'd like to fix that and run the
> tests, but I haven't really stared at it long enough to know if
> there's any non-ugly fix.

One non-ugly fix is to use RCS from the ezwinports site.  The binaries
there are 32-bit, but I don't think it will make any difference.

bug#36270: executable-find does not find shell commands on MSYS2

[Andy Moreton]

On Wed 19 Jun 2019, Eli Zaretskii wrote:
>> From: Juanma Barranquero <lekktu@gmail.com>
>> I stumbled upon this because I was trying to understand failures in
>> vc-tests.el on Windows, and the reason is that it calls call-process
>> to run some shell scripts, like ci. I'd like to fix that and run the
>> tests, but I haven't really stared at it long enough to know if
>> there's any non-ugly fix.
>
> One non-ugly fix is to use RCS from the ezwinports site.  The binaries
> there are 32-bit, but I don't think it will make any difference.

I use the following advice in a Windows build of emacs to get shell
scripts working for Cygwin. MSYS2 should be similar, with a suitable
setting for `shell-file-name':

  (defun call-process:filter-args (args)
    "Ensure native Windows emacs can run shell script programs."
    (let ((program (nth 0 args)))
      (if (executable-find program)
          args
        (list shell-file-name (nth 1 args) (nth 2 args) (nth 3 args)
              shell-command-switch
              (mapconcat #'shell-quote-argument
                         (append (list program) (nthcdr 4 args)) " ")))))
  (advice-add 'call-process :filter-args #'call-process:filter-args)

  (defun make-process:filter-args (args)
    "Ensure native Windows emacs can run shell script programs."
    (let* ((command (plist-get args :command)))
      (if (executable-find (car command))
          args
        (plist-put args :command
                   (list shell-file-name shell-command-switch
                         (mapconcat #'shell-quote-argument command " "))))))
  (advice-add 'make-process :filter-args #'make-process:filter-args))


Perhaps something similar could be added for Windows builds.

    AndyM

bug#36270: executable-find does not find shell commands on MSYS2

[Juanma Barranquero]

On Wed, Jun 19, 2019 at 6:24 PM Eli Zaretskii <eliz@gnu.org> wrote:

> Kids down here have a saying: "almost doesn't count" ;-)

Err... Don't want to think about the context for *that* ;-)

> Exactly.  So what would be the point of having a general-purpose API
> return files that can be useful only with some methods of running a
> program?  It will only cause confusion and bug reports.

That's an argument about the general-purpose API return these files,
not about having a special-purpose API, isn't it?

> So to support that, we will need to change sys_spawnve and/or
> cmdproxy to detect an extension that appears in PATHEXT, and then
> either invoke through cmdproxy or use ShellExecute instead of
> CreateProcess.  And this is the more complex part of the job.

Ok, now I understand. I will look into it, but no promises and of
course no hurry.

> One non-ugly fix is to use RCS from the ezwinports site.  The binaries
> there are 32-bit, but I don't think it will make any difference.

With your binaries installed the test will pass (haven't tested yet,
but seems likely). But that does not solve the issue that someone with
a MSYS2 build environment has a working rcs installation, and yet the
vc-tests fail. Of course, the thing is that, from Emacs POV, that's
not a "working rcs installation" at all...

bug#36270: executable-find does not find shell commands on MSYS2

[Juanma Barranquero]

On Wed, Jun 19, 2019 at 7:44 PM Andy Moreton <andrewjmoreton@gmail.com> wrote:

> I use the following advice in a Windows build of emacs to get shell
> scripts working for Cygwin. MSYS2 should be similar, with a suitable
> setting for `shell-file-name':

That seems useful, thanks.

Though there's no error checking at all, isn't it? You try to execute
a command, and if it is not executable from `executable-find' POV, off
to the shell it goes...