bug#66902: 30.0.50; Recognize env -S/--split-string in shebangs

bug#66902: 30.0.50; Recognize env -S/--split-string in shebangs

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 830 bytes --]

Hello,

While Emacs correctly picks makefile-gmake-mode when visiting a file
with the following shebang:

  #!/usr/bin/make -f

It fails to do so for this shebang:

  #!/usr/bin/env -S make -f

env(1) suggests -S is the idiomatic way to pass arguments to programs in
shebangs:

>    -S/--split-string usage in scripts
>        The -S option allows specifying multiple parameters in a script.
>        Running a script named 1.pl containing the following first line:
> 
>               #!/usr/bin/env -S perl -w -T
>               ...
> 
>        Will execute perl -w -T 1.pl .
> 
>        Without the '-S' parameter the script will likely fail with:
> 
>               /usr/bin/env: 'perl -w -T': No such file or directory

I've poked at lisp/files.el; the attached diff seems sufficient to make
Emacs pick makefile-gmake-mode .


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: files.diff --]
[-- Type: text/x-diff, Size: 568 bytes --]

diff --git a/lisp/files.el b/lisp/files.el
index 3d838cd3b8c..97594ff8a13 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -3246,7 +3246,8 @@ inhibit-local-variables-p
 
 (defvar auto-mode-interpreter-regexp
   (purecopy "#![ \t]?\\([^ \t\n]*\
-/bin/env[ \t]\\)?\\([^ \t\n]+\\)")
+/bin/env[ \t]\\(?:\\(?:-S\\|--split-string\\)[ \t]\\)?\\)?\
+\\([^ \t\n]+\\)")
   "Regexp matching interpreters, for file mode determination.
 This regular expression is matched against the first line of a file
 to determine the file's mode in `set-auto-mode'.  If it matches, the file

[-- Attachment #3: Type: text/plain, Size: 1550 bytes --]


Questions before proceeding to ChangeLog entries & regression tests:

1. Is this something we would like Emacs to recognize out of the box, or
is it too niche?

2. What about the more general forms shown in (info "(coreutils) env
invocation")?

  #!/usr/bin/env -[v]S[OPTION]... [NAME=VALUE]... COMMAND [ARGS]...

3. Assuming we do want to amend that regexp, would it be possible to use
rx here?  OT1H guessing "no" because files.el is pre-reloaded, whereas
rx.el is not; OTOH I see that files.el requires easy-mmode at
compile-time, and that package does not show up in loadup.el, so…
settling for "maybe?"

WDYT?


In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.37, cairo version 1.16.0) of 2023-08-20 built on hirondell
Repository revision: 652e45b70d82e6f615febe00553dbded80557845
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101007
System Description: Debian GNU/Linux 12 (bookworm)

Configured using:
 'configure
 --cache-file=/home/peniblec/.cache/emacs/config,src,emacs,master
 --with-cairo --with-gconf --with-sqlite3 --with-xinput2'

Configured features:
ACL CAIRO DBUS FREETYPE GCONF GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ
JPEG JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

bug#66902: 30.0.50; Recognize env -S/--split-string in shebangs

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 1449 bytes --]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> Questions before proceeding to ChangeLog entries & regression tests:

For better or worse, I ended up proceeding to both these things, and
then some.  Let me know if the attached patches make sense; tested with

  make -j8 bootstrap && make -C test files-tests


Tentative answers to my questions:

> 1. Is this something we would like Emacs to recognize out of the box, or
> is it too niche?

Assuming yes.

> 2. What about the more general forms shown in (info "(coreutils) env
> invocation")?
>
>   #!/usr/bin/env -[v]S[OPTION]... [NAME=VALUE]... COMMAND [ARGS]...

Didn't go as far as handling -v nor NAME=VALUE pairs, but that could be
added later if we ever feel like it.

> 3. Assuming we do want to amend that regexp, would it be possible to use
> rx here?  OT1H guessing "no" because files.el is pre-reloaded, whereas
> rx.el is not; OTOH I see that files.el requires easy-mmode at
> compile-time, and that package does not show up in loadup.el, so…
> settling for "maybe?"

Figured rx was similar to pcase in that regard:

* They need to be required explicitly despite their macros being
  "autoloaded", because files.el is loaded during bootstrap before
  autoloading is set up.

* Somehow that does not cause them to be preloaded?  At least going by
  emacs -Q,
  * featurep returns nil,
  * preloaded-file-list does not include them.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-basic-tests-for-interpreter-mode-alist.patch --]
[-- Type: text/x-patch, Size: 2584 bytes --]

From 8ee71e0c70fa5c16cb802722e8de15af0932773d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?K=C3=A9vin=20Le=20Gouguec?= <kevin.legouguec@gmail.com>
Date: Sun, 12 Nov 2023 10:55:24 +0100
Subject: [PATCH 1/3] Add basic tests for interpreter-mode-alist

* test/lisp/files-tests.el (files-tests--check-shebang): New helper to
generate a temporary file with a given interpreter line, and assert
that the mode picked by 'set-auto-mode' is derived from an expected
mode.  Write the 'should' form so that failure reports include useful
context; for example:

    (ert-test-failed
     ((should
       (equal (list shebang actual-mode) (list shebang expected-mode)))
      :form
      (equal ("#!/usr/bin/env -S make -f" fundamental-mode)
	     ("#!/usr/bin/env -S make -f" makefile-mode))
      :value nil :explanation
      (list-elt 1 (different-atoms fundamental-mode makefile-mode))))

(files-tests-auto-mode-interpreter): New test; exercise some aspects
of interpreter-mode-alist.
---
 test/lisp/files-tests.el | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/test/lisp/files-tests.el b/test/lisp/files-tests.el
index 3492bd701b2..233efded945 100644
--- a/test/lisp/files-tests.el
+++ b/test/lisp/files-tests.el
@@ -1656,6 +1656,29 @@ files-tests-file-name-base
   (should (equal (file-name-base "foo") "foo"))
   (should (equal (file-name-base "foo/bar") "bar")))
 
+(defun files-tests--check-shebang (shebang expected-mode)
+  "Assert that mode for SHEBANG derives from EXPECTED-MODE."
+  (let ((actual-mode
+         (ert-with-temp-file script-file
+           :text shebang
+           (find-file script-file)
+           (if (derived-mode-p expected-mode)
+               expected-mode
+             major-mode))))
+    ;; Tuck all the information we need in the `should' form: input
+    ;; shebang, expected mode vs actual.
+    (should
+     (equal (list shebang actual-mode)
+            (list shebang expected-mode)))))
+
+(ert-deftest files-tests-auto-mode-interpreter ()
+  "Test that `set-auto-mode' deduces correct modes from shebangs."
+  (files-tests--check-shebang "#!/bin/bash" 'sh-mode)
+  (files-tests--check-shebang "#!/usr/bin/env bash" 'sh-mode)
+  (files-tests--check-shebang "#!/usr/bin/env python" 'python-base-mode)
+  (files-tests--check-shebang "#!/usr/bin/env python3" 'python-base-mode)
+  (files-tests--check-shebang "#!/usr/bin/make -f" 'makefile-mode))
+
 (ert-deftest files-test-dir-locals-auto-mode-alist ()
   "Test an `auto-mode-alist' entry in `.dir-locals.el'"
   (find-file (ert-resource-file "whatever.quux"))
-- 
2.42.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Convert-auto-mode-interpreter-regexp-to-an-rx-form.patch --]
[-- Type: text/x-patch, Size: 1702 bytes --]

From d730ee2108e3bd4d641bce2cb50f61e8fbdfcd09 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?K=C3=A9vin=20Le=20Gouguec?= <kevin.legouguec@gmail.com>
Date: Sun, 12 Nov 2023 16:51:04 +0100
Subject: [PATCH 2/3] Convert auto-mode-interpreter-regexp to an rx form

* lisp/files.el: explicitly require rx even though the macros are
autoloaded, since files.el is loaded during bootstrap.
(auto-mode-interpreter-regexp): re-write using rx.  A subsequent patch
will add support for env's -S/--split-string argument, which will
complicate the pattern past my personal threshold for bare regexps.
Allow multiple spaces between #!, interpreter and first argument:
empirically, Linux's execve allows it.
---
 lisp/files.el | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/lisp/files.el b/lisp/files.el
index 3d838cd3b8c..dc301bea3c5 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -30,6 +30,7 @@
 
 (eval-when-compile
   (require 'pcase)
+  (require 'rx)
   (require 'easy-mmode)) ; For `define-minor-mode'.
 
 (defvar font-lock-keywords)
@@ -3245,8 +3246,14 @@ inhibit-local-variables-p
     temp))
 
 (defvar auto-mode-interpreter-regexp
-  (purecopy "#![ \t]?\\([^ \t\n]*\
-/bin/env[ \t]\\)?\\([^ \t\n]+\\)")
+  (purecopy
+   (rx-let ((ascii-blank (any " \t"))
+            (non-blank (not (any " \t\n"))))
+     (rx "#!"
+         (* ascii-blank)
+         (? (group (* non-blank) "/bin/env"
+                   (* ascii-blank)))
+         (group (+ non-blank)))))
   "Regexp matching interpreters, for file mode determination.
 This regular expression is matched against the first line of a file
 to determine the file's mode in `set-auto-mode'.  If it matches, the file
-- 
2.42.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-Recognize-shebang-lines-that-pass-S-split-string-to-.patch --]
[-- Type: text/x-patch, Size: 2049 bytes --]

From 0287f84a3ab6b767cc99b91356a96f2162c6a099 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?K=C3=A9vin=20Le=20Gouguec?= <kevin.legouguec@gmail.com>
Date: Sun, 12 Nov 2023 17:46:34 +0100
Subject: [PATCH 3/3] Recognize shebang lines that pass -S/--split-string to
 env

* lisp/files.el (auto-mode-interpreter-regexp): Add optional -S switch
to the ignored group capturing the env invocation.
* test/lisp/files-tests.el (files-test-auto-mode-interpreter): Add a
couple of testcases; one from (info "(coreutils) env invocation"), the
other from a personal project.
---
 lisp/files.el            | 4 +++-
 test/lisp/files-tests.el | 2 ++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/lisp/files.el b/lisp/files.el
index dc301bea3c5..56bdcf9d08b 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -3252,7 +3252,9 @@ auto-mode-interpreter-regexp
      (rx "#!"
          (* ascii-blank)
          (? (group (* non-blank) "/bin/env"
-                   (* ascii-blank)))
+                   (* ascii-blank)
+                   (? (or (: "-S" (* ascii-blank))
+                          (: "--split-string" (or ?= (* ascii-blank)))))))
          (group (+ non-blank)))))
   "Regexp matching interpreters, for file mode determination.
 This regular expression is matched against the first line of a file
diff --git a/test/lisp/files-tests.el b/test/lisp/files-tests.el
index 233efded945..3e499fff468 100644
--- a/test/lisp/files-tests.el
+++ b/test/lisp/files-tests.el
@@ -1677,6 +1677,8 @@ files-tests-auto-mode-interpreter
   (files-tests--check-shebang "#!/usr/bin/env bash" 'sh-mode)
   (files-tests--check-shebang "#!/usr/bin/env python" 'python-base-mode)
   (files-tests--check-shebang "#!/usr/bin/env python3" 'python-base-mode)
+  (files-tests--check-shebang "#!/usr/bin/env -S awk -v FS=\"\\t\" -v OFS=\"\\t\" -f" 'awk-mode)
+  (files-tests--check-shebang "#!/usr/bin/env -S make -f" 'makefile-mode)
   (files-tests--check-shebang "#!/usr/bin/make -f" 'makefile-mode))
 
 (ert-deftest files-test-dir-locals-auto-mode-alist ()
-- 
2.42.1

bug#66902: 30.0.50; Recognize env -S/--split-string in shebangs

[Eli Zaretskii]

> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
> Date: Sun, 12 Nov 2023 18:53:40 +0100
> 
> > 3. Assuming we do want to amend that regexp, would it be possible to use
> > rx here?  OT1H guessing "no" because files.el is pre-reloaded, whereas
> > rx.el is not; OTOH I see that files.el requires easy-mmode at
> > compile-time, and that package does not show up in loadup.el, so…
> > settling for "maybe?"
> 
> Figured rx was similar to pcase in that regard:
> 
> * They need to be required explicitly despite their macros being
>   "autoloaded", because files.el is loaded during bootstrap before
>   autoloading is set up.
> 
> * Somehow that does not cause them to be preloaded?  At least going by
>   emacs -Q,
>   * featurep returns nil,
>   * preloaded-file-list does not include them.

I'd prefer not to have rx required in files.el, so could you please
rewrite those parts of your patch and resubmit?  Also, please add a
NEWS entry about the change.  I think otherwise your patch is ready to
go in.

Thanks.

bug#66902: 30.0.50; Recognize env -S/--split-string in shebangs

[Kévin Le Gouguec]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
>> Date: Sun, 12 Nov 2023 18:53:40 +0100
>> 
>> > 3. Assuming we do want to amend that regexp, would it be possible to use
>> > rx here?  OT1H guessing "no" because files.el is pre-reloaded, whereas
>> > rx.el is not; OTOH I see that files.el requires easy-mmode at
>> > compile-time, and that package does not show up in loadup.el, so…
>> > settling for "maybe?"
>> 
>> Figured rx was similar to pcase in that regard:
>> 
>> * They need to be required explicitly despite their macros being
>>   "autoloaded", because files.el is loaded during bootstrap before
>>   autoloading is set up.
>> 
>> * Somehow that does not cause them to be preloaded?  At least going by
>>   emacs -Q,
>>   * featurep returns nil,
>>   * preloaded-file-list does not include them.
>
> I'd prefer not to have rx required in files.el, so could you please
> rewrite those parts of your patch and resubmit?  Also, please add a
> NEWS entry about the change.  

ACK; will get to it in the coming days.

>                               I think otherwise your patch is ready to
> go in.
>
> Thanks.

Thank you for the review!

bug#66902: 30.0.50; Recognize env -S/--split-string in shebangs

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> I'd prefer not to have rx required in files.el, so could you please
>> rewrite those parts of your patch and resubmit?  Also, please add a
>> NEWS entry about the change.  
>
> ACK; will get to it in the coming days.

s/days/hours/

I left a 'concat' in, because (a) it lets us interleave comments (b) the
byte-compiler seems to smartly condense it all to one big string literal
anyway.  (Though if files.el is preloaded, everything happens at
build-time and the .elc does not matter much, IIUC?)

Let me know if we would prefer a plain raw string literal.

Added a NEWS entry (under § 'Changes in Emacs 30.1 / Miscellaneous',
assuming 'master'); added a bug reference; squashed it all.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Recognize-shebang-lines-that-pass-S-split-string-to-.patch --]
[-- Type: text/x-patch, Size: 4499 bytes --]

From 95068836b5970c1aebb088e987741ad316007b79 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?K=C3=A9vin=20Le=20Gouguec?= <kevin.legouguec@gmail.com>
Date: Sun, 12 Nov 2023 10:55:24 +0100
Subject: [PATCH] Recognize shebang lines that pass -S/--split-string to env
 (bug#66902)

* etc/NEWS: announce the change.

* lisp/files.el (auto-mode-interpreter-regexp): Add optional -S switch
to the ignored group capturing the env invocation.
Allow multiple spaces between #!, interpreter and first argument:
empirically, Linux's execve accepts that.

* test/lisp/files-tests.el (files-tests--check-shebang): New helper to
generate a temporary file with a given interpreter line, and assert
that the mode picked by 'set-auto-mode' is derived from an expected
mode.  Write the 'should' form so that failure reports include useful
context; for example:

    (ert-test-failed
     ((should
       (equal (list shebang actual-mode) (list shebang expected-mode)))
      :form
      (equal ("#!/usr/bin/env -S make -f" fundamental-mode)
	     ("#!/usr/bin/env -S make -f" makefile-mode))
      :value nil :explanation
      (list-elt 1 (different-atoms fundamental-mode makefile-mode))))

(files-tests-auto-mode-interpreter): New test; exercise some aspects
of interpreter-mode-alist.
---
 etc/NEWS                 |  6 ++++++
 lisp/files.el            | 12 ++++++++++--
 test/lisp/files-tests.el | 25 +++++++++++++++++++++++++
 3 files changed, 41 insertions(+), 2 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 12ae8058cb1..b9ee3747040 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -233,6 +233,12 @@ to enter the file you want to modify.
 It can be used to customize the look of the appointment notification
 displayed on the mode line when 'appt-display-mode-line' is non-nil.
 
+---
+*** Emacs now recognizes shebang lines that pass -S/--split-string to env.
+When visiting a script that invokes 'env -S INTERPRETER ARGS...' in
+its shebang line, Emacs will now skip over 'env -S' and deduce the
+major mode based on the interpreter.
+
 ** Emacs Server and Client
 
 ---
diff --git a/lisp/files.el b/lisp/files.el
index d729bdf8c25..1cdcec23b11 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -3245,8 +3245,16 @@ inhibit-local-variables-p
     temp))
 
 (defvar auto-mode-interpreter-regexp
-  (purecopy "#![ \t]?\\([^ \t\n]*\
-/bin/env[ \t]\\)?\\([^ \t\n]+\\)")
+  (purecopy
+   (concat
+    "#![ \t]*"
+    ;; Optional group 1: env(1) invocation.
+    "\\("
+    "[^ \t\n]*/bin/env[ \t]*"
+    "\\(?:-S[ \t]*\\|--split-string\\(?:=\\|[ \t]*\\)\\)?"
+    "\\)?"
+    ;; Group 2: interpreter.
+    "\\([^ \t\n]+\\)"))
   "Regexp matching interpreters, for file mode determination.
 This regular expression is matched against the first line of a file
 to determine the file's mode in `set-auto-mode'.  If it matches, the file
diff --git a/test/lisp/files-tests.el b/test/lisp/files-tests.el
index 3492bd701b2..3e499fff468 100644
--- a/test/lisp/files-tests.el
+++ b/test/lisp/files-tests.el
@@ -1656,6 +1656,31 @@ files-tests-file-name-base
   (should (equal (file-name-base "foo") "foo"))
   (should (equal (file-name-base "foo/bar") "bar")))
 
+(defun files-tests--check-shebang (shebang expected-mode)
+  "Assert that mode for SHEBANG derives from EXPECTED-MODE."
+  (let ((actual-mode
+         (ert-with-temp-file script-file
+           :text shebang
+           (find-file script-file)
+           (if (derived-mode-p expected-mode)
+               expected-mode
+             major-mode))))
+    ;; Tuck all the information we need in the `should' form: input
+    ;; shebang, expected mode vs actual.
+    (should
+     (equal (list shebang actual-mode)
+            (list shebang expected-mode)))))
+
+(ert-deftest files-tests-auto-mode-interpreter ()
+  "Test that `set-auto-mode' deduces correct modes from shebangs."
+  (files-tests--check-shebang "#!/bin/bash" 'sh-mode)
+  (files-tests--check-shebang "#!/usr/bin/env bash" 'sh-mode)
+  (files-tests--check-shebang "#!/usr/bin/env python" 'python-base-mode)
+  (files-tests--check-shebang "#!/usr/bin/env python3" 'python-base-mode)
+  (files-tests--check-shebang "#!/usr/bin/env -S awk -v FS=\"\\t\" -v OFS=\"\\t\" -f" 'awk-mode)
+  (files-tests--check-shebang "#!/usr/bin/env -S make -f" 'makefile-mode)
+  (files-tests--check-shebang "#!/usr/bin/make -f" 'makefile-mode))
+
 (ert-deftest files-test-dir-locals-auto-mode-alist ()
   "Test an `auto-mode-alist' entry in `.dir-locals.el'"
   (find-file (ert-resource-file "whatever.quux"))
-- 
2.42.1

bug#66902: 30.0.50; Recognize env -S/--split-string in shebangs

[Eli Zaretskii]

> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
> Cc: 66902@debbugs.gnu.org
> Date: Sat, 18 Nov 2023 18:44:06 +0100
> 
> Kévin Le Gouguec <kevin.legouguec@gmail.com> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> I'd prefer not to have rx required in files.el, so could you please
> >> rewrite those parts of your patch and resubmit?  Also, please add a
> >> NEWS entry about the change.  
> >
> > ACK; will get to it in the coming days.
> 
> s/days/hours/
> 
> I left a 'concat' in, because (a) it lets us interleave comments (b) the
> byte-compiler seems to smartly condense it all to one big string literal
> anyway.  (Though if files.el is preloaded, everything happens at
> build-time and the .elc does not matter much, IIUC?)
> 
> Let me know if we would prefer a plain raw string literal.
> 
> Added a NEWS entry (under § 'Changes in Emacs 30.1 / Miscellaneous',
> assuming 'master'); added a bug reference; squashed it all.

Thanks, installed on the master branch, and closing the bug.

bug#66902: 30.0.50; Recognize env -S/--split-string in shebangs

[Kévin Le Gouguec]

Eli Zaretskii <eliz@gnu.org> writes:

> Thanks, installed on the master branch, and closing the bug.

Thanks for all that, and the copyedit 🙏