bug#71655: Eshell external commands do not work under GNU Emacs for Windows

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[James Hilling via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 5473 bytes --]

Hi all,

There appears to be a potential bug with Eshell when running external commands on GNU Emacs for Windows, i.e. not WSL/WSL2/Cygwin.

To reproduce:

Start Emacs with "-Q", open Eshell with `M-x eshell`, run `winget.exe --help`.

(Eshell) $ winget --help

Opening input file: Invalid argument, C:/Users/MyUser/AppData/Local/Microsoft/WindowsApps/winget.exe

For some reason external commands such as `winget.exe` do not appear to be working properly.

I thought that Eshell may be having issues when reading external command arguments, but running `winget.exe` on its own (with no arguments) also returns the same error.

What works:

Running `winget.exe --help` from the inferior shell works.

Running `cmd.exe /c winget.exe --help` from Eshell also works.

Is this a path related issue?

(Eshell) $ which winget.exe

C:/Users/MyUser/AppData/Local/Microsoft/WindowsApps/winget.exe

(Eshell) $ addpath

c:/Windows/system32
C:/Windows
C:/Windows/System32/Wbem
C:/Windows/System32/WindowsPowerShell/v1.0/
C:/Windows/System32/OpenSSH/
C:/Program Files (x86)/Gpg4win/../GnuPG/bin
C:/Program Files/PuTTY/
C:/Users/MyUser/AppData/Local/Microsoft/WindowsApps
.
C:/Users/MyUser/AppData/Local/Programs/oh-my-posh/bin

From the above output, Eshell appears to be aware of the path and therefore should also be able to execute commands such as `winget.exe`.

I tried launching other executables from `C:/Users/MyUser/AppData/Local/Microsoft/WindowsApps/` but they also fail in the exact same way.

Oddly enough, I can execute external commands within Eshell for executables under `c:/Windows/system32/`, e.g. I can execute `ipconfig.exe` from Eshell and it will work just fine. So this bug does not affect all external commands.

(Eshell) $ which ipconfig.exe

c:/Windows/system32/ipconfig.exe

I then tried running Emacs as an administrator (for troubleshooting purposes) but this also gave me the same error when reproducing.

Does anyone know of any Emacs configuration or variables that will fix this issue? I cannot seem to find any information online nor any hints from the docs that would help me solve this problem. There also doesn't appear to be an existing bug on this issue.

Kind regards,

James

Build:

In GNU Emacs 29.3 (build 2, x86_64-w64-mingw32) of 2024-03-24 built on
AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.22631
System Description: Microsoft Windows 10 Pro (v10.0.2009.22631.3737)

Configured using:
'configure --with-modules --without-dbus --with-native-compilation=aot
--without-compress-install --with-sqlite3 --with-tree-sitter
CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

(NATIVE_COMP present but libgccjit not available)

Important settings:
value of $LANG: ENG
locale-coding-system: cp1252

Major mode: Eshell

Minor modes in effect:
shell-dirtrack-mode: t
eshell-prompt-mode: t
eshell-hist-mode: t
eshell-pred-mode: t
eshell-cmpl-mode: t
eshell-proc-mode: t
eshell-arg-mode: t
tooltip-mode: t
global-eldoc-mode: t
show-paren-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
tool-bar-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
line-number-mode: t
indent-tabs-mode: t
transient-mark-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils time-date cl-seq
em-unix em-term term shell subr-x ehelp em-script em-prompt em-ls
em-hist em-pred em-glob em-extpipe em-cmpl em-dirs esh-var pcomplete
comint ansi-osc ansi-color ring em-basic em-banner em-alias esh-mode
eshell esh-cmd generator cl-loaddefs cl-lib esh-ext esh-opt esh-proc
esh-io esh-arg esh-module esh-groups esh-util files-x rmc iso-transl
tooltip cconv eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel dos-w32 ls-lisp disp-table
term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 77916 10347)
(symbols 48 7258 0)
(strings 32 22552 1535)
(string-bytes 1 692077)
(vectors 16 14796)
(vector-slots 8 338764 15212)
(floats 8 41 32)
(intervals 56 355 0)
(buffers 984 11))

[-- Attachment #2: Type: text/html, Size: 9995 bytes --]

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[Eli Zaretskii]

> Date: Wed, 19 Jun 2024 15:53:14 +0000
> From:  James Hilling via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> There appears to be a potential bug with Eshell when running external commands on GNU Emacs for
> Windows, i.e. not WSL/WSL2/Cygwin.

I don't think this has anything to do with Eshell.  Or Emacs, for that
matter.  See below.

> To reproduce:
> 
> Start Emacs with "-Q", open Eshell with `M-x eshell`, run `winget.exe --help`.
> 
> (Eshell) $ winget --help
> 
> Opening input file: Invalid argument, C:/Users/MyUser/AppData/Local/Microsoft/WindowsApps/winget.exe
> 
> For some reason external commands such as `winget.exe` do not appear to be working properly.

Try

  (Eshell) $ ls -l C:/Users/MyUser/AppData/Local/Microsoft/WindowsApps/winget.exe

What do you see?  Does what you see explain the error?

I think this page explains what is going on:

  https://stackoverflow.com/questions/58296925/what-is-zero-byte-executable-files-in-windows

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[Eli Zaretskii]

> Cc: 71655@debbugs.gnu.org
> Date: Wed, 19 Jun 2024 22:12:34 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
>   (Eshell) $ ls -l C:/Users/MyUser/AppData/Local/Microsoft/WindowsApps/winget.exe
> 
> What do you see?  Does what you see explain the error?
> 
> I think this page explains what is going on:
> 
>   https://stackoverflow.com/questions/58296925/what-is-zero-byte-executable-files-in-windows

That being said, both M-! and call-process succeed in invoking this
"program" okay, so there's something Eshell does that gets in the way.
Here's the backtrace from the error:

  Debugger entered--Lisp error: (file-error "Opening input file" "Invalid argument" "C:/Users/EliZ/AppData/Local/Microsoft/WindowsApps/winget.exe")
    insert-file-contents("C:/Users/EliZ/AppData/Local/Microsoft/WindowsApps/winget.exe" nil 0 256 nil)
    insert-file-contents-literally("C:/Users/EliZ/AppData/Local/Microsoft/WindowsApps/winget.exe" nil 0 256)
    eshell-script-interpreter("C:/Users/EliZ/AppData/Local/Microsoft/WindowsApps/winget.exe")
    eshell-find-interpreter("winget" ("--help") nil)
    eshell-connection-local-command("winget" ("--help"))
    eshell-external-command("winget" ("--help"))
    eshell-plain-command("winget" ("--help"))
    eshell-named-command("winget" ("--help"))
    eval((eshell-named-command '"winget" '("--help")))
    eshell-do-eval((eshell-named-command '"winget" '("--help")) nil)
    eshell-do-eval((unwind-protect (eshell-named-command '"winget" '("--help")) (mapc #'funcall eshell-this-command-hook)) nil)
    #f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)()
    funcall(#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))
    (let ((eshell-this-command-hook '(ignore))) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)))
    eval((let ((eshell-this-command-hook '(ignore))) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))))
    eshell-do-eval((let ((eshell-this-command-hook '(ignore))) (unwind-protect (eshell-named-command '"winget" '("--help")) (mapc #'funcall eshell-this-command-hook))) nil)
    (condition-case err (eshell-do-eval '(let ((eshell-this-command-hook '(ignore))) (unwind-protect (eshell-named-command '"winget" '("--help")) (mapc #'funcall eshell-this-command-hook))) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1)))
    eval((condition-case err (eshell-do-eval '(let ((eshell-this-command-hook '...)) (unwind-protect (eshell-named-command '"winget" '...) (mapc #'funcall eshell-this-command-hook))) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1))))
    eshell-do-eval((condition-case err (eshell-do-eval '(let ((eshell-this-command-hook '...)) (unwind-protect (eshell-named-command '"winget" '...) (mapc #'funcall eshell-this-command-hook))) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1))) nil)
    eshell-do-eval((condition-case err (eshell-do-eval '(let ((eshell-this-command-hook '...)) (unwind-protect (eshell-named-command '"winget" '...) (mapc #'funcall eshell-this-command-hook))) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1))) nil)
    #f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)()
    funcall(#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))
    (let ((eshell-current-handles '[nil (((t) . 2) t) (((t) . 2) t)])) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)))
    eval((let ((eshell-current-handles '[nil ((... . 2) t) ((... . 2) t)])) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))))
    eshell-do-eval((let ((eshell-current-handles '[nil ((... . 2) t) ((... . 2) t)])) (condition-case err (eshell-do-eval '(let ((eshell-this-command-hook ...)) (unwind-protect (eshell-named-command ... ...) (mapc ... eshell-this-command-hook))) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1)))) nil)
    eshell-do-eval((progn (let ((eshell-current-handles '[nil (... t) (... t)])) (condition-case err (eshell-do-eval '(let (...) (unwind-protect ... ...)) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1))))) nil)
    eshell-do-eval((unwind-protect (progn (let ((eshell-current-handles '[nil ... ...])) (condition-case err (eshell-do-eval '(let ... ...) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1))))) (run-hooks 'eshell-post-command-hook)) nil)
    eshell-do-eval((progn 'nil (unwind-protect (progn (let ((eshell-current-handles '...)) (condition-case err (eshell-do-eval '... nil) ((debug error) (eshell-errorn ...) (eshell-close-handles 1))))) (run-hooks 'eshell-post-command-hook))) nil)
    #f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)()
    funcall(#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))
    (let ((eshell-current-handles '[nil (((t) . 2) t) (((t) . 2) t)]) (eshell-current-subjob-p 'nil)) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)))
    eval((let ((eshell-current-handles '[nil ((... . 2) t) ((... . 2) t)]) (eshell-current-subjob-p 'nil)) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))))
    eshell-do-eval((let ((eshell-current-handles '[nil ((... . 2) t) ((... . 2) t)]) eshell-current-subjob-p) (progn 'nil (unwind-protect (progn (let ((eshell-current-handles ...)) (condition-case err (eshell-do-eval ... nil) (... ... ...)))) (run-hooks 'eshell-post-command-hook)))))
    eshell-resume-eval((nil (let ((eshell-current-handles '[nil (... t) (... t)]) eshell-current-subjob-p) (progn 'nil (unwind-protect (progn (let (...) (condition-case err ... ...))) (run-hooks 'eshell-post-command-hook)))) nil))
    eshell-eval-command((let ((eshell-current-handles '[nil ((... . 2) t) ((... . 2) t)]) eshell-current-subjob-p) (progn 'nil (unwind-protect (progn (let ((eshell-current-handles ...)) (condition-case err (eshell-do-eval ... nil) (... ... ...)))) (run-hooks 'eshell-post-command-hook)))) "winget --help")
    eshell-send-input(nil)
    funcall-interactively(eshell-send-input nil)
    call-interactively(eshell-send-input nil nil)
    command-execute(eshell-send-input)

Jim, why does Eshell want to read the executable file winget.exe?  If
that's because it wants to find the signature by which it will deduce
the interpreter, then doing that for zero-size files is not useful,
and should probably be skipped?

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[Eli Zaretskii]

> Cc: 71655@debbugs.gnu.org
> Date: Wed, 19 Jun 2024 22:12:34 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
>   (Eshell) $ ls -l C:/Users/MyUser/AppData/Local/Microsoft/WindowsApps/winget.exe
> 
> What do you see?  Does what you see explain the error?
> 
> I think this page explains what is going on:
> 
>   https://stackoverflow.com/questions/58296925/what-is-zero-byte-executable-files-in-windows

That being said, both M-! and call-process succeed in invoking this
"program" okay, so there's something Eshell does that gets in the way.
Here's the backtrace from the error:

  Debugger entered--Lisp error: (file-error "Opening input file" "Invalid argument" "C:/Users/EliZ/AppData/Local/Microsoft/WindowsApps/winget.exe")
    insert-file-contents("C:/Users/EliZ/AppData/Local/Microsoft/WindowsApps/winget.exe" nil 0 256 nil)
    insert-file-contents-literally("C:/Users/EliZ/AppData/Local/Microsoft/WindowsApps/winget.exe" nil 0 256)
    eshell-script-interpreter("C:/Users/EliZ/AppData/Local/Microsoft/WindowsApps/winget.exe")
    eshell-find-interpreter("winget" ("--help") nil)
    eshell-connection-local-command("winget" ("--help"))
    eshell-external-command("winget" ("--help"))
    eshell-plain-command("winget" ("--help"))
    eshell-named-command("winget" ("--help"))
    eval((eshell-named-command '"winget" '("--help")))
    eshell-do-eval((eshell-named-command '"winget" '("--help")) nil)
    eshell-do-eval((unwind-protect (eshell-named-command '"winget" '("--help")) (mapc #'funcall eshell-this-command-hook)) nil)
    #f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)()
    funcall(#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))
    (let ((eshell-this-command-hook '(ignore))) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)))
    eval((let ((eshell-this-command-hook '(ignore))) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))))
    eshell-do-eval((let ((eshell-this-command-hook '(ignore))) (unwind-protect (eshell-named-command '"winget" '("--help")) (mapc #'funcall eshell-this-command-hook))) nil)
    (condition-case err (eshell-do-eval '(let ((eshell-this-command-hook '(ignore))) (unwind-protect (eshell-named-command '"winget" '("--help")) (mapc #'funcall eshell-this-command-hook))) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1)))
    eval((condition-case err (eshell-do-eval '(let ((eshell-this-command-hook '...)) (unwind-protect (eshell-named-command '"winget" '...) (mapc #'funcall eshell-this-command-hook))) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1))))
    eshell-do-eval((condition-case err (eshell-do-eval '(let ((eshell-this-command-hook '...)) (unwind-protect (eshell-named-command '"winget" '...) (mapc #'funcall eshell-this-command-hook))) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1))) nil)
    eshell-do-eval((condition-case err (eshell-do-eval '(let ((eshell-this-command-hook '...)) (unwind-protect (eshell-named-command '"winget" '...) (mapc #'funcall eshell-this-command-hook))) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1))) nil)
    #f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)()
    funcall(#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))
    (let ((eshell-current-handles '[nil (((t) . 2) t) (((t) . 2) t)])) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)))
    eval((let ((eshell-current-handles '[nil ((... . 2) t) ((... . 2) t)])) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))))
    eshell-do-eval((let ((eshell-current-handles '[nil ((... . 2) t) ((... . 2) t)])) (condition-case err (eshell-do-eval '(let ((eshell-this-command-hook ...)) (unwind-protect (eshell-named-command ... ...) (mapc ... eshell-this-command-hook))) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1)))) nil)
    eshell-do-eval((progn (let ((eshell-current-handles '[nil (... t) (... t)])) (condition-case err (eshell-do-eval '(let (...) (unwind-protect ... ...)) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1))))) nil)
    eshell-do-eval((unwind-protect (progn (let ((eshell-current-handles '[nil ... ...])) (condition-case err (eshell-do-eval '(let ... ...) nil) ((debug error) (eshell-errorn (error-message-string err)) (eshell-close-handles 1))))) (run-hooks 'eshell-post-command-hook)) nil)
    eshell-do-eval((progn 'nil (unwind-protect (progn (let ((eshell-current-handles '...)) (condition-case err (eshell-do-eval '... nil) ((debug error) (eshell-errorn ...) (eshell-close-handles 1))))) (run-hooks 'eshell-post-command-hook))) nil)
    #f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)()
    funcall(#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))
    (let ((eshell-current-handles '[nil (((t) . 2) t) (((t) . 2) t)]) (eshell-current-subjob-p 'nil)) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>)))
    eval((let ((eshell-current-handles '[nil ((... . 2) t) ((... . 2) t)]) (eshell-current-subjob-p 'nil)) (funcall '#f(compiled-function () #<bytecode -0x165be7c9ce9ef886>))))
    eshell-do-eval((let ((eshell-current-handles '[nil ((... . 2) t) ((... . 2) t)]) eshell-current-subjob-p) (progn 'nil (unwind-protect (progn (let ((eshell-current-handles ...)) (condition-case err (eshell-do-eval ... nil) (... ... ...)))) (run-hooks 'eshell-post-command-hook)))))
    eshell-resume-eval((nil (let ((eshell-current-handles '[nil (... t) (... t)]) eshell-current-subjob-p) (progn 'nil (unwind-protect (progn (let (...) (condition-case err ... ...))) (run-hooks 'eshell-post-command-hook)))) nil))
    eshell-eval-command((let ((eshell-current-handles '[nil ((... . 2) t) ((... . 2) t)]) eshell-current-subjob-p) (progn 'nil (unwind-protect (progn (let ((eshell-current-handles ...)) (condition-case err (eshell-do-eval ... nil) (... ... ...)))) (run-hooks 'eshell-post-command-hook)))) "winget --help")
    eshell-send-input(nil)
    funcall-interactively(eshell-send-input nil)
    call-interactively(eshell-send-input nil nil)
    command-execute(eshell-send-input)

Jim, why does Eshell want to read the executable file winget.exe?  If
that's because it wants to find the signature by which it will deduce
the interpreter, then doing that for zero-size files is not useful,
and should probably be skipped?

This naïve patch fixes the problem:

diff --git a/lisp/eshell/esh-ext.el b/lisp/eshell/esh-ext.el
index 3c4deb3..d9631be 100644
--- a/lisp/eshell/esh-ext.el
+++ b/lisp/eshell/esh-ext.el
@@ -247,7 +247,8 @@ eshell-connection-local-command
 		 ;; know the interpreter in that case, therefore the
 		 ;; check is suppressed.
 		 (or (and (stringp command) (file-remote-p command))
-		     (file-remote-p default-directory)))))
+		     (file-remote-p default-directory)
+                     (zerop (file-attribute-size (file-attributes (executable-find command))))))))
     (cl-assert interp)
     (if (functionp (car interp))
 	(apply (car interp) (append (cdr interp) args))

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[Jim Porter]

On 6/19/2024 12:22 PM, Eli Zaretskii wrote:
> Jim, why does Eshell want to read the executable file winget.exe?  If
> that's because it wants to find the signature by which it will deduce
> the interpreter, then doing that for zero-size files is not useful,
> and should probably be skipped?

It's trying to find a shebang, which I guess(?) is so that Eshell can 
support shebangs on MS-Windows. What's strange is that 'file-readable-p' 
is non-nil, but 'insert-file-contents-literally' fails.

As far as I understand things, winget.exe isn't exactly a zero-byte 
file. They're reparse points that point to a real executable living in 
some locked-down folder, so they're like something symlinks I think?

It seems like there's a small bug somewhere in 
'insert-file-contents-literally'. On MS-Windows, "cat 
C:\Users\...\winget.exe" outputs the (binary) contents of winget.exe 
just fine (this is using the MSYS2 build of cat). So I think the real 
winget.exe file truly is readable. I don't know why 
'insert-file-contents-literally' has a problem with it though.

It'd be nice to figure out why that fails and fix it at the source, but 
on the other hand, maybe this only comes up when trying to read these 
.exe files? A more-targeted fix could be to just ignore errors in 
'eshell-script-interpreter': if we can't insert the file, assume it 
doesn't have a shebang and try to run it like a normal program (which 
works fine in Emacs).

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[Eli Zaretskii]

> Date: Wed, 19 Jun 2024 12:40:12 -0700
> Cc: 71655@debbugs.gnu.org, james@literate-devops.io
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 6/19/2024 12:22 PM, Eli Zaretskii wrote:
> > Jim, why does Eshell want to read the executable file winget.exe?  If
> > that's because it wants to find the signature by which it will deduce
> > the interpreter, then doing that for zero-size files is not useful,
> > and should probably be skipped?
> 
> It's trying to find a shebang, which I guess(?) is so that Eshell can 
> support shebangs on MS-Windows. What's strange is that 'file-readable-p' 
> is non-nil, but 'insert-file-contents-literally' fails.

It fails because winget.exe is not a regular file, and
insert-file-contents barely supports non-regular files (and even that
almost exclusively on Posix systems).

> As far as I understand things, winget.exe isn't exactly a zero-byte 
> file. They're reparse points that point to a real executable living in 
> some locked-down folder, so they're like something symlinks I think?

It's a reparse point, but not a symlink.  Symlinks are also
implemented on Windows as reparse points, but this one is a reparse
point of a different kind, because Emacs does support symlinks on
MS-Windows, and yet doesn't recognize this file as a symlink.

> It seems like there's a small bug somewhere in 
> 'insert-file-contents-literally'. On MS-Windows, "cat 
> C:\Users\...\winget.exe" outputs the (binary) contents of winget.exe 
> just fine (this is using the MSYS2 build of cat).

Not here.  The native cat.exe says "Invalid argument", just like
Emacs, and the one from MSYS says "Permission denied".  I get similar
errors from other utilities, for example wc.  And MSYS ls shows it as
a regular file of size zero.

So I think what we see in Emacs is the same issue with these special
"executables" they cannot be easily treated as regular files or links
to regular files.

> So I think the real winget.exe file truly is readable. I don't know
> why 'insert-file-contents-literally' has a problem with it though.

See above: I hope I explained that now.

> It'd be nice to figure out why that fails and fix it at the source, but 
> on the other hand, maybe this only comes up when trying to read these 
> .exe files? A more-targeted fix could be to just ignore errors in 
> 'eshell-script-interpreter': if we can't insert the file, assume it 
> doesn't have a shebang and try to run it like a normal program (which 
> works fine in Emacs).

I'm asking why it even makes sense to try to read these files?  If a
file is not a symlink and its size is zero, what useful things could
possibly happen by trying to read it?  Suppose we add to Emacs support
for these special reparse points -- what do you expect the target to
be if the name ends with .exe? what kind of "interpreter" will we
glean from that?

So my opinion on this is that Eshell should really skip reading files
whose size is zero when it looks for an interpreter, since we will
never find anything useful that way.

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[Jim Porter]

On 6/19/2024 9:53 PM, Eli Zaretskii wrote:
>> Date: Wed, 19 Jun 2024 12:40:12 -0700
>> Cc: 71655@debbugs.gnu.org, james@literate-devops.io
>> From: Jim Porter <jporterbugs@gmail.com>
>>
>> It's trying to find a shebang, which I guess(?) is so that Eshell can
>> support shebangs on MS-Windows. What's strange is that 'file-readable-p'
>> is non-nil, but 'insert-file-contents-literally' fails.
> 
> It fails because winget.exe is not a regular file, and
> insert-file-contents barely supports non-regular files (and even that
> almost exclusively on Posix systems).

'file-regular-p' for that file is also non-nil. Should we change that?

> Not here.  The native cat.exe says "Invalid argument", just like
> Emacs, and the one from MSYS says "Permission denied".  I get similar
> errors from other utilities, for example wc.  And MSYS ls shows it as
> a regular file of size zero.

My version of ls reports it as a symlink, interestingly enough. I'm 
using the MSYS2 binaries that come with Git for Windows to test this. I 
think they apply some additional patches on top so maybe the versions I 
have include some special support for reparse points like this?

> So my opinion on this is that Eshell should really skip reading files
> whose size is zero when it looks for an interpreter, since we will
> never find anything useful that way.

Well, I don't think size=0 is the only way that we could skip over files 
like this. If 'file-regular-p' or 'file-readable-p' returned nil, we 
could use that to skip it. We could also skip files ending in ".exe". We 
could skip files that signal from 'insert-file-contents-literally'.

I don't mind checking for size=0 if that's what we decide, but my 
reading of the existing 'eshell-script-interpreter' suggests that it 
should have already worked. If there's a bug in 'file-regular-p' (or 
some other function Eshell uses here), I think it's work fixing it at 
the source so that we squash the bug once and for all. Otherwise, some 
other Lisp code might try to do something similar one day (maybe a Lisp 
version of file(1)?) and get bitten by this bug.

If that's not convincing, then I'll just add the size=0 check.

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[Eli Zaretskii]

> Date: Wed, 19 Jun 2024 22:34:02 -0700
> Cc: 71655@debbugs.gnu.org, james@literate-devops.io
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 6/19/2024 9:53 PM, Eli Zaretskii wrote:
> >> Date: Wed, 19 Jun 2024 12:40:12 -0700
> >> Cc: 71655@debbugs.gnu.org, james@literate-devops.io
> >> From: Jim Porter <jporterbugs@gmail.com>
> >>
> >> It's trying to find a shebang, which I guess(?) is so that Eshell can
> >> support shebangs on MS-Windows. What's strange is that 'file-readable-p'
> >> is non-nil, but 'insert-file-contents-literally' fails.
> > 
> > It fails because winget.exe is not a regular file, and
> > insert-file-contents barely supports non-regular files (and even that
> > almost exclusively on Posix systems).
> 
> 'file-regular-p' for that file is also non-nil. Should we change that?

Only if that is useful.  For now, I'm not sure I see a reason to do
that, since the code to support that will not be trivial, and will
have to include full support for these files in file-attributes and
similar APIs as well.

> > Not here.  The native cat.exe says "Invalid argument", just like
> > Emacs, and the one from MSYS says "Permission denied".  I get similar
> > errors from other utilities, for example wc.  And MSYS ls shows it as
> > a regular file of size zero.
> 
> My version of ls reports it as a symlink, interestingly enough.

It isn't a symlink.  It is a reparse point of type APPEXECLINK, which
has different attributes and different data structure describing the
target.  We could represent it as a symlink (since Posix has no direct
equivalent), but the implementation under the hood will need to be
different.

> > So my opinion on this is that Eshell should really skip reading files
> > whose size is zero when it looks for an interpreter, since we will
> > never find anything useful that way.
> 
> Well, I don't think size=0 is the only way that we could skip over files 
> like this. If 'file-regular-p' or 'file-readable-p' returned nil, we 
> could use that to skip it. We could also skip files ending in ".exe". We 
> could skip files that signal from 'insert-file-contents-literally'.

I agree that all those other conditions (including the .exe test) seem
to be reasonable, in addition to zero-size.

> I don't mind checking for size=0 if that's what we decide, but my 
> reading of the existing 'eshell-script-interpreter' suggests that it 
> should have already worked. If there's a bug in 'file-regular-p' (or 
> some other function Eshell uses here), I think it's work fixing it at 
> the source so that we squash the bug once and for all.

Fixing file-regular-p (and all the related APIs) for this purpose
sounds like a lot of work for little or no gain.  But if someone wants
to work on that and provide a clean patch, I don't mind.

> Otherwise, some other Lisp code might try to do something similar
> one day (maybe a Lisp version of file(1)?) and get bitten by this
> bug.

When they do, we'll have another situation to consider.

In general, you must understand that the depth and breadth of
emulating Posix assumptions and concepts on MS-Windows are driven by
practical needs, not by theoretical possibilities and potential
breakage that _might_ happen in some hypothetical Lisp.  Especially as
I don't quite see people with such patches lining up...

So if a simpler change in the (so far) single application which bumped
into this could fix the problem, I'm all for it.

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[Jim Porter]

[-- Attachment #1: Type: text/plain, Size: 1399 bytes --]

On 6/20/2024 12:45 AM, Eli Zaretskii wrote:
>> Date: Wed, 19 Jun 2024 22:34:02 -0700
>> Cc: 71655@debbugs.gnu.org, james@literate-devops.io
>> From: Jim Porter <jporterbugs@gmail.com>
>>
>> My version of ls reports it as a symlink, interestingly enough.
> 
> It isn't a symlink.  It is a reparse point of type APPEXECLINK, which
> has different attributes and different data structure describing the
> target.  We could represent it as a symlink (since Posix has no direct
> equivalent), but the implementation under the hood will need to be
> different.

Right. This was just (what I thought was) an interesting observation 
about how other POSIX-based tools treat these reparse points.

> I agree that all those other conditions (including the .exe test) seem
> to be reasonable, in addition to zero-size.

Do you have a preference between either of these patches? They either 
check for zero-size or ignore file errors when trying to insert.

I don't have a strong preference myself, but the latter seems 
ever-so-slightly safer to me. This bug happened because we can't read 
the file when trying to insert it, so ignoring file errors would cover 
any other situations we haven't predicted. On the other hand, maybe 
there's a case where we *want* the 'insert-file-contents-literally' 
error to signal so that we don't try to execute the file normally (I 
can't think of any such cases, though).

[-- Attachment #2: 0001-Use-file-attribute-size.patch --]
[-- Type: text/plain, Size: 1605 bytes --]

From 4521f82dcef0253564574012c571564111e06c4a Mon Sep 17 00:00:00 2001
From: Jim Porter <jporterbugs@gmail.com>
Date: Sat, 22 Jun 2024 12:45:19 -0700
Subject: [PATCH] Fix execution of MS-Windows app execution aliases in Eshell

* lisp/eshell/esh-ext.el (eshell-script-interpreter): Check for 0-size
files.
---
 lisp/eshell/esh-ext.el | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/lisp/eshell/esh-ext.el b/lisp/eshell/esh-ext.el
index 3c4deb32601..cf93d2904da 100644
--- a/lisp/eshell/esh-ext.el
+++ b/lisp/eshell/esh-ext.el
@@ -301,7 +301,17 @@ eshell-script-interpreter
   (INTERPRETER [ARGS] FILE)"
   (let ((maxlen eshell-command-interpreter-max-length))
     (if (and (file-readable-p file)
-	     (file-regular-p file))
+	     (file-regular-p file)
+             ;; If the file is zero bytes, it can't possibly have a
+             ;; shebang.  This check may seem redundant, but we can
+             ;; encounter files that Emacs considers both readable and
+             ;; regular, but which aren't *actually* readable.  This can
+             ;; happen, for example, with certain kinds of reparse
+             ;; points like APPEXECLINK on NTFS filesystems (MS-Windows
+             ;; uses these for "app execution aliases").  In these
+             ;; cases, the file size is 0, so this check protects us
+             ;; from errors.
+             (> (file-attribute-size (file-attributes file)) 0))
 	(with-temp-buffer
 	  (insert-file-contents-literally file nil 0 maxlen)
 	  (if (looking-at "#![ \t]*\\([^ \r\t\n]+\\)\\([ \t]+\\(.+\\)\\)?")
-- 
2.25.1


[-- Attachment #3: 0001-Use-ignore-error.patch --]
[-- Type: text/plain, Size: 2091 bytes --]

From 55cf3088d0075f7d453eecb1e03683de4444eb4f Mon Sep 17 00:00:00 2001
From: Jim Porter <jporterbugs@gmail.com>
Date: Sat, 22 Jun 2024 12:41:25 -0700
Subject: [PATCH] Fix execution of MS-Windows app execution aliases in Eshell

* lisp/eshell/esh-ext.el (eshell-script-interpreter): Return nil if we
can't actually read the file.
---
 lisp/eshell/esh-ext.el | 29 +++++++++++++++++------------
 1 file changed, 17 insertions(+), 12 deletions(-)

diff --git a/lisp/eshell/esh-ext.el b/lisp/eshell/esh-ext.el
index 3c4deb32601..6fb0579ed14 100644
--- a/lisp/eshell/esh-ext.el
+++ b/lisp/eshell/esh-ext.el
@@ -299,18 +299,23 @@ eshell-script-interpreter
 Return nil, or a list of the form:
 
   (INTERPRETER [ARGS] FILE)"
-  (let ((maxlen eshell-command-interpreter-max-length))
-    (if (and (file-readable-p file)
-	     (file-regular-p file))
-	(with-temp-buffer
-	  (insert-file-contents-literally file nil 0 maxlen)
-	  (if (looking-at "#![ \t]*\\([^ \r\t\n]+\\)\\([ \t]+\\(.+\\)\\)?")
-	      (if (match-string 3)
-		  (list (match-string 1)
-			(match-string 3)
-			file)
-		(list (match-string 1)
-		      file)))))))
+  (when (and (file-readable-p file)
+             (file-regular-p file))
+    (let ((maxlen eshell-command-interpreter-max-length))
+      ;; If we encounter a file error, assume that FILE doesn't have a
+      ;; shebang.  This can happen, for example, with certain kinds of
+      ;; reparse points like APPEXECLINK on NTFS filesystems (MS-Windows
+      ;; uses these for "app execution aliases").
+      (ignore-error 'file-error
+        (with-temp-buffer
+          (insert-file-contents-literally file nil 0 maxlen)
+          (if (looking-at "#![ \t]*\\([^ \r\t\n]+\\)\\([ \t]+\\(.+\\)\\)?")
+              (if (match-string 3)
+                  (list (match-string 1)
+                        (match-string 3)
+                        file)
+                (list (match-string 1)
+                      file))))))))
 
 (defun eshell-find-interpreter (file args &optional no-examine-p)
   "Find the command interpreter with which to execute FILE.
-- 
2.25.1

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[Eli Zaretskii]

> Date: Sat, 22 Jun 2024 12:55:32 -0700
> Cc: 71655@debbugs.gnu.org, james@literate-devops.io
> From: Jim Porter <jporterbugs@gmail.com>
> 
> > I agree that all those other conditions (including the .exe test) seem
> > to be reasonable, in addition to zero-size.
> 
> Do you have a preference between either of these patches? They either 
> check for zero-size or ignore file errors when trying to insert.
> 
> I don't have a strong preference myself, but the latter seems 
> ever-so-slightly safer to me. This bug happened because we can't read 
> the file when trying to insert it, so ignoring file errors would cover 
> any other situations we haven't predicted. On the other hand, maybe 
> there's a case where we *want* the 'insert-file-contents-literally' 
> error to signal so that we don't try to execute the file normally (I 
> can't think of any such cases, though).

Why not do both?  If the file has zero size, reading it is pointless,
and if reading it signals an error, we cannot examine it for the
interpreter signature.

Thanks.

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[Jim Porter]

On 6/22/2024 9:36 PM, Eli Zaretskii wrote:
>> Date: Sat, 22 Jun 2024 12:55:32 -0700
>> Cc: 71655@debbugs.gnu.org, james@literate-devops.io
>> From: Jim Porter <jporterbugs@gmail.com>
>>
>> I don't have a strong preference myself, but the latter seems
>> ever-so-slightly safer to me. This bug happened because we can't read
>> the file when trying to insert it, so ignoring file errors would cover
>> any other situations we haven't predicted. On the other hand, maybe
>> there's a case where we *want* the 'insert-file-contents-literally'
>> error to signal so that we don't try to execute the file normally (I
>> can't think of any such cases, though).
> 
> Why not do both?  If the file has zero size, reading it is pointless,
> and if reading it signals an error, we cannot examine it for the
> interpreter signature.

That could work, though thinking about this some more, I think there's a 
benefit to being careful about how we add checks here. For Tramp files, 
we should probably try to keep the number of calls that need to touch 
the remote filesystem to a minimum.

I'll think about this some more and see if we can get all the checks we 
want without making the code slower over Tramp. (Maybe Tramp caches 
enough that this isn't a problem, but I'm not certain yet.)

bug#71655: Eshell external commands do not work under GNU Emacs for Windows

[Michael Albinus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Jim Porter <jporterbugs@gmail.com> writes:

Hi Jim,

> That could work, though thinking about this some more, I think there's
> a benefit to being careful about how we add checks here. For Tramp
> files, we should probably try to keep the number of calls that need to
> touch the remote filesystem to a minimum.
>
> I'll think about this some more and see if we can get all the checks
> we want without making the code slower over Tramp. (Maybe Tramp caches
> enough that this isn't a problem, but I'm not certain yet.)

If Eshell calls `process-file', it shall bind `process-file-side-effects'
to nil if appropriate for the given command.

Furthermore, Eshell might bind `remote-file-name-inhibit-cache' to
something which helps. The default value (10) seems to be very conservative.

In case Eshell needs to clear the cache for a given directory, there is
`dired-uncache'. Contrary to its name, it is not restricted to dired,
but of general purpose.

Best regards, Michael.