bug#67321: 29.1.90; Different parsing rules for -*- lexical-binding:t; -*- in different circumstances

[Eli Zaretskii <eliz@gnu.org>]

merge 67321 64272
thanks

> From: Sebastian Miele <iota@whxvd.name>
> Date: Tue, 21 Nov 2023 09:16:13 +0100
> 
> Put the following into a file named, e.g., ‘test-script’, and ‘chmod +x’
> it.
> 
>   #!/bin/sh
>   : ; exec emacs --script "$0" -- "$@" #; -*- lexical-binding: t; mode: emacs-lisp; -*-
> 
>   (defmacro lexical-binding-p ()
>     '(let* ((x t)
>             (f (lambda () x))
>             (x nil))
>        (funcall f)))
> 
>   (message "%s %s" lexical-binding (lexical-binding-p))
> 
> When the script is run, the output is "nil nil", signifying that lexical
> binding is not enabled.
> 
> Then find the file in an interactive Emacs session, and interactively
> evaluate (C-x C-e) the two expressions.  The output now is "t t", i.e.,
> lexical binding is in use.
> 
> https://lists.gnu.org/archive/html/emacs-devel/2023-11/msg01041.html
> contains more details, which are repeated here:
> 
> > From: Jens Schmidt <jschmidt4gnu@vodafonemail.de>
> > Date: Mon, 2023-11-20 21:10 +0100
> >
> > I tried byte-compiling something similar yesterday, which also
> > indicated that the byte-compiler compiles with lexical bindings.  Only
> > the scripting machinery sees dynamical bindings.
> >
> > […]  It seems that the scripting machinery expects a semicolon in the
> > very first column, without that the lexical-binding line is not
> > recognized.  Even a space before the semicolon breaks the recognition.
> >
> > The problem is in function `lisp_file_lexically_bound_p' from lread.c,
> > which is indeed much more strict in its recognition of the -*- ... -*-
> > stanza than the functions `set-auto-mode-1' and
> > `hack-local-variables-prop-line' from files.el.  The Emacs manual
> > ((emacs) Specifying File Variables) only mentions that the stanza has
> > to be in the first line (or the second one if the first is taken by a
> > she-bang), without any restriction where the comment has to start.
> 
> ((emacs) Specifying File Variables) states no restrictions on what must
> (not) precede or follow a "-*- … -*-" on the first or second line of a
> file.
> 
> Expected: At least consistent behavior.  Ideally, lexical binding
> should be enabled in all cases.

There's more here than meets the eye, see bug#64272.