bug#67321: 29.1.90; Different parsing rules for -*- lexical-binding:t; -*- in different circumstances

[Sebastian Miele <iota@whxvd.name>]

Put the following into a file named, e.g., ‘test-script’, and ‘chmod +x’
it.

  #!/bin/sh
  : ; exec emacs --script "$0" -- "$@" #; -*- lexical-binding: t; mode: emacs-lisp; -*-

  (defmacro lexical-binding-p ()
    '(let* ((x t)
            (f (lambda () x))
            (x nil))
       (funcall f)))

  (message "%s %s" lexical-binding (lexical-binding-p))

When the script is run, the output is "nil nil", signifying that lexical
binding is not enabled.

Then find the file in an interactive Emacs session, and interactively
evaluate (C-x C-e) the two expressions.  The output now is "t t", i.e.,
lexical binding is in use.

https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg01041.html
contains more details, which are repeated here:

> From: Jens Schmidt <jschmidt4gnu@vodafonemail.de>
> Date: Mon, 2023-11-20 21:10 +0100
>
> I tried byte-compiling something similar yesterday, which also
> indicated that the byte-compiler compiles with lexical bindings.  Only
> the scripting machinery sees dynamical bindings.
>
> […]  It seems that the scripting machinery expects a semicolon in the
> very first column, without that the lexical-binding line is not
> recognized.  Even a space before the semicolon breaks the recognition.
>
> The problem is in function `lisp_file_lexically_bound_p' from lread.c,
> which is indeed much more strict in its recognition of the -*- ... -*-
> stanza than the functions `set-auto-mode-1' and
> `hack-local-variables-prop-line' from files.el.  The Emacs manual
> ((emacs) Specifying File Variables) only mentions that the stanza has
> to be in the first line (or the second one if the first is taken by a
> she-bang), without any restriction where the comment has to start.

((emacs) Specifying File Variables) states no restrictions on what must
(not) precede or follow a "-*- … -*-" on the first or second line of a
file.

Expected: At least consistent behavior.  Ideally, lexical binding
should be enabled in all cases.

In GNU Emacs 29.1.90 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.38, cairo version 1.18.0) of 2023-11-17 built on blatt
Repository revision: b759f9b508da20649a0741bcfad466e6111edb4e
Repository branch: pgtk-hyper
System Description: Arch Linux

Configured using:
 'configure --with-pgtk --with-native-compilation --with-tree-sitter
 --with-json --with-mailutils --with-imagemagick
 'CFLAGS=-march=ivybridge -mtune=ivybridge''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ
IMAGEMAGICK JPEG JSON LCMS2 LIBOTF LIBSYSTEMD LIBXML2 MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XIM GTK3 ZLIB

Important settings:
  value of $LANG: C.UTF-8
  locale-coding-system: utf-8-unix

Major mode: mu4e:view

Minor modes in effect:
  mu4e-search-minor-mode: t
  mu4e-context-minor-mode: t
  envrc-global-mode: t
  envrc-mode: t
  editorconfig-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  marginalia-mode: t
  corfu-popupinfo-mode: t
  corfu-history-mode: t
  corfu-candidate-overlay-mode: t
  global-corfu-mode: t
  corfu-mode: t
  info-rename-buffer-mode: t
  savehist-mode: t
  server-mode: t
  recentf-mode: t
  vertico-mode: t
  shell-dirtrack-mode: t
  minibuffer-depth-indicate-mode: t
  global-so-long-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  buffer-read-only: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t

Load-path shadows:
/home/w/dot/emacs/.init.d/usr/elpa/modus-themes-20231115.1302/theme-loaddefs hides /home/w/usr/emacs/0/29/0/lisp/theme-loaddefs
/home/w/dot/emacs/.init.d/usr/elpa/transient-20231112.923/transient hides /home/w/usr/emacs/0/29/0/lisp/transient

Features:
(shadow emacsbug sort smiley gnus-cite mail-extr textsec uni-scripts
idna-mapping ucs-normalize uni-confusable textsec-check macrostep-c
cmacexp make-mode tabify tramp-cmds misearch multi-isearch goto-addr
vc-hg vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs log-view
emacsql-sqlite-builtin sqlite face-remap cal-move oc-basic ol-eww
ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect ol-docview doc-view
jka-compr image-mode exif ol-bibtex bibtex ol-bbdb ol-w3m ol-doi
org-link-doi cus-start whx-init-later whx-tangled-mu mu4e mu4e-org
mu4e-notification notifications mu4e-main mu4e-view gnus-art mm-uu
mml2015 mm-view mml-smime smime dig gnus-sum gnus-group gnus-undo
gnus-start gnus-dbus dbus gnus-cloud nnimap nnmail mail-source utf7 nnoo
gnus-spec gnus-int gnus-range gnus-win mu4e-headers mu4e-compose
mu4e-draft mu4e-actions smtpmail mu4e-search mu4e-lists mu4e-bookmarks
mu4e-mark mu4e-message flow-fill mule-util hl-line mu4e-contacts
mu4e-update mu4e-folders mu4e-context mu4e-query-items mu4e-server
mu4e-modeline mu4e-vars mu4e-helpers mu4e-config mu4e-window ido
mu4e-obsolete whx-tangled-envrc eglot external-completion array jsonrpc
ert ewoc flymake-proc flymake envrc inheritenv whx-tangled-org ob-python
ob-shell whx-custom org-superstar ol-eshell ol-man whx-password
whx-pcase whx-walk ox-odt rng-loc rng-uri rng-parse rng-match rng-dt
rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util ox-latex
ox-icalendar org-agenda ox-html table ox-ascii ox-publish ox whx-etc-org
whx-tangled-0 web-mode advice disp-table css-mode sgml-mode facemenu eww
url-queue shr pixel-fill kinsoku url-file mm-url gnus nnheader range
docker-compose-mode dockerfile-mode apache-mode systemd cmake-font-lock
cmake-mode rst python sh-script smie executable lua-mode json-mode
json-snatcher js c-ts-common cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine yaml-mode vcard tex-mode compile
consult-register vc-git consult-xref dumb-jump popup xref project
gptel-transient gptel gptel-openai editorconfig editorconfig-core
editorconfig-core-handle editorconfig-fnmatch ace-window avy wdired
dired-filter dired-hacks-utils dired-aux dired-x diary-lib
diary-loaddefs cal-iso forge-list forge-commands forge-semi
forge-bitbucket buck forge-gogs gogs forge-gitea gtea forge-gitlab glab
forge-github ghub-graphql treepy gsexp ghub url-http url-gw nsm url-auth
let-alist gnutls forge-notify forge-revnote forge-pullreq forge-issue
forge-topic yaml bug-reference forge-post markdown-mode forge-repo forge
forge-core forge-db closql emacsql-sqlite-common emacsql
emacsql-compiler eieio-base magit-patch magit-extras magit-bookmark
magit-submodule magit-blame magit-stash magit-reflog magit-bisect
magit-push magit-pull magit-fetch magit-clone magit-remote magit-commit
magit-sequence magit-notes magit-worktree magit-tag magit-merge
magit-branch magit-reset magit-files magit-refs magit-status magit
magit-repos magit-apply magit-wip magit-log which-func magit-diff
smerge-mode diff diff-mode easy-mmode git-commit log-edit message
sendmail yank-media puny rfc822 mml mml-sec epa derived epg rfc6068
epg-config gnus-util mm-decode mm-bodies mm-encode mail-parse rfc2231
rfc2047 rfc2045 mm-util ietf-drums mail-prsvr mailabbrev mail-utils
gmm-utils mailheader pcvs-util add-log magit-core magit-autorevert
autorevert filenotify magit-margin magit-transient magit-process
with-editor magit-mode magit-git magit-base magit-section cursor-sensor
git-modes gitignore-mode gitconfig-mode conf-mode gitattributes-mode
em-hist em-pred esh-mode eshell esh-cmd esh-ext esh-opt esh-proc esh-io
esh-arg esh-module esh-groups esh-util marginalia vertico-quick
corfu-popupinfo corfu-info kind-icon svg-lib color svg dom xml
corfu-history corfu-quick corfu-candidate-overlay corfu orderless crm
man info-rename-buffer helpful cc-langs cc-vars cc-defs imenu trace
edebug info-look f f-shortdoc elisp-refs help-fns radix-tree embark-org
org-element org-persist xdg org-id org-refile avl-tree generator
embark-consult embark ffap thingatpt ispell savehist tramp-cache
time-stamp expreg treesit cap-words superword subword rect whx-cl server
consult-dir consult recentf tree-widget bookmark text-property-search
vertico-directory vertico tramp-sh tramp-rclone tramp-fuse tramp
tramp-loaddefs trampver tramp-integration files-x tramp-compat shell
parse-time iso8601 mb-depth whx-safe-local whx-etc-2 whx-transient comp
comp-cstr warnings whx-arg whx-meta transient cl-extra edmacro kmacro
compat whx-etc-elisp whx-bind whx-elisp whx-feature whx-string s
whx-error debug backtrace help-mode macrostep whx-etc-look dired
dired-loaddefs org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro
org-src ob-comint org-pcomplete pcomplete comint ansi-osc ansi-color
ring org-list org-footnote org-faces org-entities time-date noutline
outline ob-emacs-lisp ob-core ob-eval org-cycle org-table ol org-fold
org-fold-core org-keys oc org-loaddefs find-func cal-menu calendar
cal-loaddefs org-version org-compat org-macs format-spec whx-face
whx-alist whx-plist whx-list dash modus-operandi-theme modus-themes
diminish whx-etc-packages--core finder-inf ace-window-autoloads
apache-mode-autoloads async-autoloads avy-autoloads
cmake-font-lock-autoloads cmake-mode-autoloads consult-dir-autoloads
consult-project-extra-autoloads corfu-candidate-overlay-autoloads
corfu-autoloads debbugs-autoloads diminish-autoloads
dired-collapse-autoloads dired-filter-autoloads dired-subtree-autoloads
dired-hacks-utils-autoloads docker-compose-mode-autoloads
dockerfile-mode-autoloads dumb-jump-autoloads editorconfig-autoloads
eglot-tempel-autoloads embark-consult-autoloads consult-autoloads
embark-autoloads envrc-autoloads expand-region-autoloads
expreg-autoloads forge-autoloads closql-autoloads emacsql-autoloads
geiser-guile-autoloads geiser-autoloads ghub-autoloads
git-modes-autoloads gptel-autoloads helpful-autoloads
elisp-refs-autoloads f-autoloads i3wm-config-mode-autoloads
info-rename-buffer-autoloads inheritenv-autoloads json-mode-autoloads
json-snatcher-autoloads kbd-mode-autoloads keycast-autoloads
kind-icon-autoloads lua-mode-autoloads macrostep-autoloads
magit-autoloads git-commit-autoloads magit-section-autoloads
dash-autoloads marginalia-autoloads markdown-mode-autoloads
modus-themes-autoloads mu4e-autoloads orderless-autoloads
org-superstar-autoloads pkgbuild-mode-autoloads popup-autoloads
quelpa-autoloads rainbow-mode-autoloads s-autoloads svg-lib-autoloads
systemd-autoloads tempel-collection-autoloads tempel-autoloads
transient-autoloads treepy-autoloads vcard-autoloads vertico-autoloads
vimrc-mode-autoloads web-mode-autoloads wgrep-autoloads
with-editor-autoloads info compat-autoloads yaml-autoloads
yaml-mode-autoloads whx-package--core package-vc vc vc-dispatcher
lisp-mnt package browse-url url url-proxy url-privacy url-expand
url-methods url-history url-cookie generate-lisp-file url-domsuf
url-util mailcap url-handlers url-parse auth-source eieio eieio-core
password-cache json map byte-opt bytecomp byte-compile url-vars
whx-etc-1--core so-long whx-etc-buffer--core ibuffer ibuffer-loaddefs
whx-etc-elisp--core whx-elisp--core rx whx-etc-gc--core whx-bind--core
whx-init-early whx-boot whx-etc-0--core cus-edit pp cus-load icons
wid-edit whx-etc-native-comp--core whx-etc-litter--core no-littering
whx-etc-fs-path--core whx-fs-path--core cl-seq whx-string--core
whx-log--core whx-0--core whx-let-set--core whx-symbol--core
whx-list--core subr-x whx-meta--core whx-pcase--core pcase
whx-error--core whx-binary-unit-prefix--core whx-feature--core cl-macs
gv cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/pgtk-win pgtk-win term/common-win pgtk-dnd tool-bar dnd fontset
image regexp-opt fringe tabulated-list replace newcomment text-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo gtk pgtk
lcms2 multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 1580890 49378)
 (symbols 48 74132 9)
 (strings 32 392309 15713)
 (string-bytes 1 11053956)
 (vectors 16 155349)
 (vector-slots 8 3761081 202439)
 (floats 8 807 1807)
 (intervals 56 61711 3388)
 (buffers 984 46))