bug#67321: 29.1.90; Different parsing rules for -*- lexical-binding:t; -*- in different circumstances

[Sebastian Miele <iota@whxvd.name>]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Tue, 2023-11-21 11:46 +0100
>
>>   #!/bin/sh
>>   : ; exec emacs --script "$0" -- "$@" #; -*- lexical-binding: t; mode: emacs-lisp; -*-
>>
>>   (defmacro lexical-binding-p ()
>>     '(let* ((x t)
>>             (f (lambda () x))
>>             (x nil))
>>        (funcall f)))
>>
>>   (message "%s %s" lexical-binding (lexical-binding-p))
>
> Can I ask why that idiom is used?

The idiom allows arbitrary shell processing before, and possibly even
after, actually running Emacs in the script.

See https://github.com/doomemacs/doomemacs/blob/master/bin/doom for an
example including both pre- and postprocessing, and lots of comments
about what is done there, and why.

Another case that recently popped up on emacs-devel is that it allows to
insert the "--" for cleanly separating the script commandline options
from options that Emacs may interpret, see
https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg00896.html.

I do not know anymore why I considered the trick in the past.  I do not
want to spend the time to look into my scripts to find that bits that
would become cumbersome without such tricks.  Because of the problem
with lexical-binding not being picked up, and because I, for now, have
the luxury of only using Linux, I can use another trick.  But being able
to mangle the command line of Emacs before running Emacs definitely is a
useful thing.  The portable alternative would be to have one separate
(shell) wrapper around every Emacs script that needs such mangling.

What follows is the beginning of
https://github.com/doomemacs/doomemacs/blob/master/bin/doom for those
who do not visit GitHub:

#!/usr/bin/env sh
:; # -*- mode: emacs-lisp; lexical-binding: t -*-
:; case "$EMACS" in *term*) EMACS=emacs ;; *) EMACS="${EMACS:-emacs}" ;; esac
:; emacs="$EMACS ${DEBUG:+--debug-init} -q --no-site-file --batch"
:; tmpdir=`$emacs --eval '(princ (temporary-file-directory))' 2>/dev/null`
:; [ -z "$tmpdir" ] && { >&2 echo "Error: failed to run Emacs with command '$EMACS'"; >&2 echo; >&2 echo "Are you sure Emacs is installed and in your \$PATH?"; exit 1; }
:; export __DOOMPID="${__DOOMPID:-$$}"
:; export __DOOMSTEP="${__DOOMSTEP:-0}"
:; export __DOOMGEOM="${__DOOMGEOM:-`tput cols 2>/dev/null`x`tput lines 2>/dev/null`}"
:; export __DOOMGPIPE=${__DOOMGPIPE:-$__DOOMPIPE}
:; export __DOOMPIPE=; [ -t 0 ] || __DOOMPIPE="${__DOOMPIPE}0"; [ -t 1 ] || __DOOMPIPE="${__DOOMPIPE}1"
:; $emacs --load "$0" -- "$@" || exit=$?
:; [ "${exit:-0}" -eq 254 ] && { sh "${tmpdir}/doom.${__DOOMPID}.${__DOOMSTEP}.sh" "$0" "$@" && true; exit="$?"; }
:; exit $exit

;; This magical mess of a shebang is necessary for any script that relies on
;; Doom's CLI framework, because Emacs' tty libraries and capabilities are too
;; immature (borderline non-existent) at the time of writing (28.1). This
;; shebang sets out to accomplish these three goals:
;;
;; 1. To produce a more helpful error if Emacs isn't installed or broken. It
;;    must do so without assuming whether $EMACS is a shell command (e.g. 'snap
;;    run emacs') or an absolute path (to an emacs executable). I've avoided
;;    'command -v $EMACS' for this reason.
;;
;; 2. To allow this Emacs session to "exit into" a child process (since Elisp
;;    lacks an analogue for exec system calls) by calling an auto-generated and
;;    self-destructing "exit script" if the parent Emacs process exits with code
;;    254. It takes care to prevent nested child instances from clobbering the
;;    exit script.
;;
;; 3. To expose some information about the terminal and session:
;;    - $__DOOMGEOM holds the dimensions of the terminal (W . H).
;;    - $__DOOMPIPE indicates whether the script has been piped (in and/or out).
;;    - $__DOOMGPIPE indicates whether one of this process' parent has been
;;      piped to/from.
;;    - $__DOOMPID is a unique identifier for the parent script, so
;;      child processes can identify which persistent data files (like logs) it
;;      has access to.
;;    - $__DOOMSTEP counts how many levels deep we are in the dream (appending
;;      this to the exit script's filename avoids child processes clobbering the
;;      same exit script and causing read errors).
;;    - $TMPDIR (or $TEMP and $TMP on Windows) aren't guaranteed to have values,
;;      and mktemp isn't available on all systems, but you know what is? Emacs!
;;      So I use it to print `temporary-file-directory'. And it seconds as a
;;      quick sanity check for Emacs' existence (for goal #1).
;;
;; Other weird facts about this shebang line:
;;
;; - The :; hack exploits properties of : and ; in shell scripting and elisp to
;;   allow shell script and elisp to coexist in the same file without either's
;;   interpreter throwing foreign syntax errors:
;;
;;   - In elisp, ":" is a valid keyword symbol literal; it evaluates to itself
;;     and has no side-effect.
;;   - In the shell, ":" is a valid command that does nothing and ignores its
;;     arguments.
;;   - In elisp, ";" begins a comment. I.e. the interpreter ignores everything
;;     after it.
;;   - In the shell, ";" is a command separator.
;;
;;   Put together, plus a strategically placed exit call, the shell will read
;;   one part of this file and ignore the rest, while the elisp interpreter will
;;   do the opposite.
;; - I intentionally avoid loading site files, so lisp/doom-cli.el can load them
;;   by hand later. There, I can suppress and deal with unhelpful warnings (e.g.
;;   "package cl is deprecated"), "Loading X...DONE" spam, and any other
;;   disasterous side-effects.
;;
;;   But be careful not to use -Q! It implies --no-site-lisp, which omits the
;;   site-lisp directory from `load-path'.
;; - POSIX-compliancy is paramount: there's no guarantee what /bin/sh will be
;;   symlinked to in the esoteric OSes/distros Emacs users use.
;; - The user may have a noexec flag set on /tmp, so pass the exit script to
;;   /bin/sh rather than executing them directly.