missing lexical-binding cookie warning when loading .el files

missing lexical-binding cookie warning when loading .el files

[Mattias Engdegård]

There is now a warning about missing -*-lexical-binding:...-*- cookie in .el files being loaded, since not all Lisp code is compiled. The idea is that dealing with the odd warnings now is better than being surprised by sudden change of behaviour in Emacs 31, when lexical-binding will be enabled by default.

It should be a net benefit, but no doubt some people we get more warnings than others. If that turns out to be very common then we may reconsider, but the warning can always be suppressed.

Re: missing lexical-binding cookie warning when loading .el files

[Eli Zaretskii]

> From: Mattias Engdegård <mattias.engdegard@gmail.com>
> Date: Fri, 3 May 2024 19:28:02 +0200
> 
> There is now a warning about missing -*-lexical-binding:...-*- cookie in .el files being loaded, since not all Lisp code is compiled. The idea is that dealing with the odd warnings now is better than being surprised by sudden change of behaviour in Emacs 31, when lexical-binding will be enabled by default.

Please announce this in NEWS, if you didn't already.

Re: missing lexical-binding cookie warning when loading .el files

[Mattias Engdegård]

3 maj 2024 kl. 19.59 skrev Eli Zaretskii <eliz@gnu.org>:

> Please announce this in NEWS, if you didn't already.

Yes, it's there.

Re: missing lexical-binding cookie warning when loading .el files

[Po Lu]

Mattias Engdegård <mattias.engdegard@gmail.com> writes:

> 3 maj 2024 kl. 19.59 skrev Eli Zaretskii <eliz@gnu.org>:
>
>> Please announce this in NEWS, if you didn't already.
>
> Yes, it's there.

And furthermore, please exempt all generated files that are generated
without lexical-binding cookies from such warnings.  The init file is
one, the custom file is another, which I've not yet decided how to
exempt, and which at any rate is indifferent to the state of lexical
binding.

Re: missing lexical-binding cookie warning when loading .el files

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> And furthermore, please exempt all generated files that are generated
> without lexical-binding cookies from such warnings.  The init file is
> one, the custom file is another, which I've not yet decided how to
> exempt, and which at any rate is indifferent to the state of lexical
> binding.

Not to mention .newsrc, Custom-theme.el etc.  It's a fair bet that
generated data files which behave identically in both binding modes are
never byte-compiled, from which it follows that this change, as it
stands, is more disruptive than it is helpful, except in relation to the
few _code_ files that are frequently loaded without being byte-compiled.

Presumably, users who fall victim to unmarked files do so through
require, not plain Fload, which is more the domain of packages that load
data files, such as custom, desktop, and Gnus.  If there are no serious
objections, I will modify the criteria for the warning accordingly, to
only emit them when the file concerned is loaded as a library.

It could subsequently be arranged that what few files are often loaded
without lexical binding cookies and otherwise than by require prompt
these warnings unconditionally.  Still, I'm not entirely satisfied with
the practice of emitting warnings in one release for prospective changes
in the next: it might reflect negatively on us if we should decide
against them, and how is it productive to enjoin users to update their
code years in advance, for a change that is far from a done deal, and
for an Emacs release that it is every right of theirs not to install?

One example: it's been 4 years since configure started to state that Xft
would be removed "in the next release of Emacs", yet two releases have
since run their course, with no end to Xft support in sight.  Nor will
it be removed, so long as I'm still around and the Cairo build continues
to generate problems in situations other than it was designed to solve.

Re: missing lexical-binding cookie warning when loading .el files

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> Not to mention .newsrc, Custom-theme.el etc.

Add to this list recentf.  I really wish authors of such changes with
far-reaching consequences would make some rudimentary effort to use
their changes before installing them, because otherwise those of us this
responsibility next devolves on are placed in a very unpleasant
position.  There are numerous past instances of changes, having only
this one property in common, to wit, that they were motivated by some
abstract imperfection in the Lisp interpreter, being installed, only to
break the bootstrapping process or to cause users intense frustration,
which in one extreme case might actually have incurred (insignificant,
but still measurable) loss of money.

Thanks.

Re: missing lexical-binding cookie warning when loading .el files

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Sat, 04 May 2024 12:40:11 +0800
> 
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Not to mention .newsrc, Custom-theme.el etc.
> 
> Add to this list recentf.

And abbrev_defs.

Basically, anything below user-emacs-directory, I think?

Re: missing lexical-binding cookie warning when loading .el files

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> And abbrev_defs.
>
> Basically, anything below user-emacs-directory, I think?

Yes, but not exclusively so, and also excepting those authored by the
user, e.g. early-init.el and init.el.

Re: missing lexical-binding cookie warning when loading .el files

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: mattias.engdegard@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 04 May 2024 16:10:28 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And abbrev_defs.
> >
> > Basically, anything below user-emacs-directory, I think?
> 
> Yes, but not exclusively so

Why not?

> and also excepting those authored by the user, e.g. early-init.el
> and init.el.

Those are under user-emacs-directory, right?

Re: missing lexical-binding cookie warning when loading .el files

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Po Lu <luangruo@yahoo.com>
>> Cc: mattias.engdegard@gmail.com,  emacs-devel@gnu.org
>> Date: Sat, 04 May 2024 16:10:28 +0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > And abbrev_defs.
>> >
>> > Basically, anything below user-emacs-directory, I think?
>> 
>> Yes, but not exclusively so
>
> Why not?

Gnus generated files are saved in the home directory, and the Custom
file can be configured to reside in any directory.

> Those are under user-emacs-directory, right?

Correct.  My point was that presumably the warning is appropriate for at
least the early-init file, and possibly also for the init file, when it
is not generated by such mechanisms as Custom.

Re: missing lexical-binding cookie warning when loading .el files

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: mattias.engdegard@gmail.com,  emacs-devel@gnu.org
> Date: Sat, 04 May 2024 16:23:09 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Those are under user-emacs-directory, right?
> 
> Correct.  My point was that presumably the warning is appropriate for at
> least the early-init file, and possibly also for the init file, when it
> is not generated by such mechanisms as Custom.

I disagree.  I think we shouldn't bug users due to their init files,
not at all.

So I think at this point we could either (a) exempt all files below
user-emacs-directory, (b) add a database of basenames of files that
should be exempt from these warnings, or (c) a combination of those
two.

Any other suggestions?

Re: missing lexical-binding cookie warning when loading .el files

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> I disagree.  I think we shouldn't bug users due to their init files,
> not at all.

My thoughts are actually with you--I was only attempting to apply
Mattias's reasoning, that's all.

> So I think at this point we could either (a) exempt all files below
> user-emacs-directory, (b) add a database of basenames of files that
> should be exempt from these warnings, or (c) a combination of those
> two.
>
> Any other suggestions?

I think we should exempt all files that are not `require'd, because
interpreted Lisp libraries that are neither lexically bound nor marked
as such are the targets of this warning.  It's virtually a godsent
discriminator, as generated files are never required, while libraries
should always be.

Re: missing lexical-binding cookie warning when loading .el files

[Eli Zaretskii]

> Date: Sat, 04 May 2024 12:05:04 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: mattias.engdegard@gmail.com, emacs-devel@gnu.org
> 
> > From: Po Lu <luangruo@yahoo.com>
> > Cc: mattias.engdegard@gmail.com,  emacs-devel@gnu.org
> > Date: Sat, 04 May 2024 16:23:09 +0800
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > > Those are under user-emacs-directory, right?
> > 
> > Correct.  My point was that presumably the warning is appropriate for at
> > least the early-init file, and possibly also for the init file, when it
> > is not generated by such mechanisms as Custom.
> 
> I disagree.  I think we shouldn't bug users due to their init files,
> not at all.
> 
> So I think at this point we could either (a) exempt all files below
> user-emacs-directory, (b) add a database of basenames of files that
> should be exempt from these warnings, or (c) a combination of those
> two.
> 
> Any other suggestions?

I see Mattias already installed something, without waiting for the
discussion to come to conclusions.

But IMO the installed change is incorrect/incomplete:

  . it hard-codes ".emacs" as the file for which we don't warn, thus
    leaving the warning in effect for early-init files
  . the init file can be named init.el or _emacs (with or without
    .el), and the change doesn't handle that
  . there are files like recentf-save-file (default: ".recentf") or
    abbrev-file-name (default: "abbrev_defs") and others, which should
    also be exempt of this

I wish people would discuss such changes before committing them.  This
is a complex issue, with many aspects that are not really obvious, as
Emacs loads files in many contexts and for many different purposes.
We should be sure we understand all the aspects before we design and
implement solutions.

Re: missing lexical-binding cookie warning when loading .el files

[Mattias Engdegård]

4 maj 2024 kl. 09.32 skrev Eli Zaretskii <eliz@gnu.org>:

> And abbrev_defs.
> 
> Basically, anything below user-emacs-directory, I think?

Thanks for raising this concern. For now I settled on warning about *.el and .emacs only, on the grounds that it would eliminate at least some of the false positives.

I do think it is appropriate to warn about the user init files because they tend to gather a lot of actual Lisp code, not just data. (New Emacs-generated init files should have a lexical cookie.)

Not sure how common pure data-as-code files are in reality; most Lisp data files aren't evaluated. Those that are are probably just sequences of function calls with constant arguments and thus rarely affected by lexical-binding.

Now if data-as-code files turn out to be just too common and detecting them systematically too difficult, then I'd rather get rid of the warning altogether; it's not important enough to build an elaborate suppression system for, much less have endless discussions about.

It's quite common to see other actual code files under ~/.emacs.d/, but maybe they are not worth warning about if it means too many false positives?

Some options:

A. Add a suppression system. We could add the variable `lisp-source-load-missing-lexical-binding-declaration-warning-suppression-regexp-list`.
B. Strong-arm package maintainers into writing a lexical cookie when saving data-as-code.
C. Don't warn about ~/.emacs.d/**/*.el, except init.el.
Z. Remove the warning.

Re: missing lexical-binding cookie warning when loading .el files

[Po Lu]

Mattias Engdegård <mattias.engdegard@gmail.com> writes:

> Thanks for raising this concern. For now I settled on warning about
> *.el and .emacs only, on the grounds that it would eliminate at least
> some of the false positives.

But not all, not even all the packages in core, let alone ELPA.  It's
really not acceptable for Emacs to become a patchwork of warring
features, where no sooner does one feature save a file than does the
next feature proceed to pester the user to delete it.

> I do think it is appropriate to warn about the user init files because
> they tend to gather a lot of actual Lisp code, not just data. (New
> Emacs-generated init files should have a lexical cookie.)

That's a matter of opinion.

> Not sure how common pure data-as-code files are in reality; most Lisp
> data files aren't evaluated. Those that are are probably just
> sequences of function calls with constant arguments and thus rarely
> affected by lexical-binding.

newsrc.eld, desktop files (which do bear lexical-binding cookies), the
URL cookie list, generated custom themes, and the like.

> Now if data-as-code files turn out to be just too common and detecting
> them systematically too difficult, then I'd rather get rid of the
> warning altogether; it's not important enough to build an elaborate
> suppression system for, much less have endless discussions about.
>
> It's quite common to see other actual code files under ~/.emacs.d/,
> but maybe they are not worth warning about if it means too many false
> positives?
>
> Some options:
>
> A. Add a suppression system. We could add the variable
> `lisp-source-load-missing-lexical-binding-declaration-warning-suppression-regexp-list`.

I hope this is in jest.

> B. Strong-arm package maintainers into writing a lexical cookie when
> saving data-as-code.

Strong-arming users is hardly a laudable action on our part, or that of
any software developers at that.

> C. Don't warn about ~/.emacs.d/**/*.el, except init.el.

I thought the purpose of this warning was to detect bodies of Lisp code
of which the user might be unaware, to give him an opportunity to
futureproof them in time for Emacs 31.

> Z. Remove the warning.

Not the worst proposal I've heard.

Re: missing lexical-binding cookie warning when loading .el files

[Mattias Engdegård]

4 maj 2024 kl. 11.24 skrev Eli Zaretskii <eliz@gnu.org>:

> I see Mattias already installed something, without waiting for the
> discussion to come to conclusions.

Sorry, other people had started interfering in the code. I will implement the consensus, but it's much easier if nobody else carries out pet changes before even basic adjustments can be made.

>  . it hard-codes ".emacs" as the file for which we don't warn, thus
>    leaving the warning in effect for early-init files
>  . the init file can be named init.el or _emacs (with or without
>    .el), and the change doesn't handle that

Actually it hard-codes .emacs as a file for which we do warn, but I didn't think of _emacs -- sorry.

Anyway, my assumption was that we should warn about the user init file(s) and you seem to disagree. Here is my reasoning:

+ Init files often contain substantial amounts of hand-written code for which lexical-binding matters.
+ New Emacs-generated init files always include a lexical cookie.
- Old Emacs-generated init files did not include a lexical cookie when created.
- Some init files don't contain much actual Lisp code.

Not sure what the balance is. Which would you prefer, and why? I'm not deeply invested in either choice.

>  . there are files like recentf-save-file (default: ".recentf") or
>    abbrev-file-name (default: "abbrev_defs") and others, which should
>    also be exempt of this

Yes, any file not ending in .el are now exempt.

> I wish people would discuss such changes before committing them.

Sorry about that, and I will make sure it's something that we (you specifically included!) are satisfied with, even if it means removing the warning entirely.

Re: missing lexical-binding cookie warning when loading .el files

[Eli Zaretskii]

> From: Mattias Engdegård <mattias.engdegard@gmail.com>
> Date: Sat, 4 May 2024 12:31:38 +0200
> Cc: luangruo@yahoo.com,
>  emacs-devel@gnu.org
> 
> 4 maj 2024 kl. 11.24 skrev Eli Zaretskii <eliz@gnu.org>:
> 
> >  . it hard-codes ".emacs" as the file for which we don't warn, thus
> >    leaving the warning in effect for early-init files
> >  . the init file can be named init.el or _emacs (with or without
> >    .el), and the change doesn't handle that
> 
> Actually it hard-codes .emacs as a file for which we do warn, but I didn't think of _emacs -- sorry.
> 
> Anyway, my assumption was that we should warn about the user init file(s) and you seem to disagree. Here is my reasoning:
> 
> + Init files often contain substantial amounts of hand-written code for which lexical-binding matters.
> + New Emacs-generated init files always include a lexical cookie.
> - Old Emacs-generated init files did not include a lexical cookie when created.
> - Some init files don't contain much actual Lisp code.
> 
> Not sure what the balance is. Which would you prefer, and why? I'm not deeply invested in either choice.

I'd like to hear from others as well, so let's give them some time to
chime in.

My current personal opinion is that we should move in small steps
here: first warn only about Lisp packages, which means exempt all the
user init files, as well as files not ending in ".el".

Not sure what this means for files that user init files load via an
explicit 'load' call -- how many people split their init files into
several portions, and also call those portions SOMETHING.el?  If in
doubt, we could suppress the warning for the entire duration of
startup--load-user-init-file.

Re: missing lexical-binding cookie warning when loading .el files

[Mattias Engdegård]

4 maj 2024 kl. 14.59 skrev Eli Zaretskii <eliz@gnu.org>:

> My current personal opinion is that we should move in small steps
> here: first warn only about Lisp packages, which means exempt all the
> user init files, as well as files not ending in ".el".

Yes, we can do that. There are pragmatic reasons: almost every user will have an init file and it will rarely have a lexical cookie because it isn't thought of as a Lisp program at all, just a settings file. In many cases the user will have copied it from somewhere else and has no idea what it contains, or it was generated by an older Emacs.

> Not sure what this means for files that user init files load via an
> explicit 'load' call -- how many people split their init files into
> several portions, and also call those portions SOMETHING.el?  If in
> doubt, we could suppress the warning for the entire duration of
> startup--load-user-init-file.

I think that files that a user loads from the init file should not get special treatment, because this is more likely done intentionally by someone who knows what he or she is doing (we hope).

RE: [External] : Re: missing lexical-binding cookie warning when loading .el files

[Drew Adams]

> Z. Remove the warning.

+1.


A **WARNING** just because a file doesn't
turn on lexical-binding by default?

"DANGER, ALL USERS!!"  You might just end
up _learning_ about lexical binding.

Re: missing lexical-binding cookie warning when loading .el files

[Mattias Engdegård]

I'm back-pedalling on this warning: in particular I underestimated the amount of little glue files (autoloads, subdirs.el, etc) that users have. Almost all of these, and most of the other files, will likely work without change when the default is changed to lexical-binding, so let users worry about that then.

(And Stefan gets to say 'told you so'.)

Re: missing lexical-binding cookie warning when loading .el files

[Arsen Arsenović]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

Hi Mattias,

Mattias Engdegård <mattias.engdegard@gmail.com> writes:

> I'm back-pedalling on this warning: in particular I underestimated the
> amount of little glue files (autoloads, subdirs.el, etc) that users
> have. Almost all of these, and most of the other files, will likely
> work without change when the default is changed to lexical-binding, so
> let users worry about that then.

May I propose emitting it only when there are variables whose scoping
would be changed by it?  IMO, this warning is a reasonable one, but
there's also lots of glue files where it doesn't matter.

IIUC, the heuristic for that shouldn't be very hard to get right (but I
am also not fully aware of the lexical-binding semantics so I'm not
certain).

Have a lovely day.
-- 
Arsen Arsenović

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

Re: missing lexical-binding cookie warning when loading .el files

[Po Lu]

Mattias Engdegård <mattias.engdegard@gmail.com> writes:

> I'm back-pedalling on this warning: in particular I underestimated the
> amount of little glue files (autoloads, subdirs.el, etc) that users
> have. Almost all of these, and most of the other files, will likely
> work without change when the default is changed to lexical-binding, so
> let users worry about that then.
>
> (And Stefan gets to say 'told you so'.)

I'm glad to see that wiser counsel has prevailed.  Thanks.

Re: missing lexical-binding cookie warning when loading .el files

[Mattias Engdegård]

5 maj 2024 kl. 16.28 skrev Arsen Arsenović <arsen@aarsen.me>:

> May I propose emitting it only when there are variables whose scoping
> would be changed by it?

That would be a good idea, but it is not easy to do so. Example:

(defun f (x)
  (g))

If this code is compiled without lexical-binding enabled, then `x` will be bound locally during the call to `g`, so we would have to know what that function is doing, and everything that it calls, and so on.

Most likely it doesn't make a difference in this case, and in our experience converting code to lexical binding, usually nothing or very little needs changing.

The byte-compiler will nag if the lexical-binding declaration is missing, and that has proven quite useful.

RE: [External] : Re: missing lexical-binding cookie warning when loading .el files

[Drew Adams]

> (defun f (x) (g))
> 
> If this code is compiled without lexical-binding enabled, then `x` will
> be bound locally during the call to `g`

Huh?  In what universe?