Why is lexical-binding's global value ignored?

Why is lexical-binding's global value ignored?

[abq]

If somebody does (setq-default lexical-binding t), the intention is 
obvious. But in current versions of Emacs, when loading a file that 
doesn't specify a local value for lexical-binding, the global value is 
ignored. Why?

IOW, what would break if the global value were honored? It defaults to 
nil, which is already how files are loaded if no local value is 
specified, so honoring the global wouldn't change anything by default.

Re: Why is lexical-binding's global value ignored?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On Fri, Jan 27, 2023 at 08:01:47PM -0700, abq@bitrot.link wrote:
> If somebody does (setq-default lexical-binding t), the intention is obvious.
> But in current versions of Emacs, when loading a file that doesn't specify a
> local value for lexical-binding, the global value is ignored. Why?
> 
> IOW, what would break if the global value were honored? It defaults to nil,
> which is already how files are loaded if no local value is specified, so
> honoring the global wouldn't change anything by default.

I think it's backward compatibility, mainly. Files with no explicit
lexical binding setting (may) contain code which expects dynamic
binding and would break.

It's not about the code distributed with Emacs, but about the code you
have been using for 15 years which would fail in obscure ways.

IOW, "not having a binding declaration in a file means dynamic binding"
has become a contract :-)

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Why is lexical-binding's global value ignored?

[Jean Louis]

* tomas@tuxteam.de <tomas@tuxteam.de> [2023-01-28 10:12]:
> On Fri, Jan 27, 2023 at 08:01:47PM -0700, abq@bitrot.link wrote:
> > If somebody does (setq-default lexical-binding t), the intention is obvious.
> > But in current versions of Emacs, when loading a file that doesn't specify a
> > local value for lexical-binding, the global value is ignored. Why?
> > 
> > IOW, what would break if the global value were honored? It defaults to nil,
> > which is already how files are loaded if no local value is specified, so
> > honoring the global wouldn't change anything by default.
> 
> I think it's backward compatibility, mainly. Files with no explicit
> lexical binding setting (may) contain code which expects dynamic
> binding and would break.

Some programs cannot work with lexical binding turned on.

Such is the RCD Template Interpolation System for Emacs:
https://hyperscope.link/3/7/1/3/3/RCD-Template-Interpolation-System-for-Emacs.html

I use that package for text based spreadsheet-like calculations.

Demonstration here:
https://gnu.support/images/2022/11/2022-11-11/2022-11-11-11:11:11.ogv

Calling it from lexical bindings works only with modified `rcd-dlet'
function, how it was in it's original. 

Calling is one aspect of it.

Other application that does not work with lexical binding is `eev' package.

-- 
Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

Re: Why is lexical-binding's global value ignored?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Some programs cannot work with lexical binding turned on.

  > Such is the RCD Template Interpolation System for Emacs:
  > https://hyperscope.link/3/7/1/3/3/RCD-Template-Interpolation-System-for-Emacs.html

Sooner or later we should call on the user community to fix programs
to work with lexical binding if possible, and to explicitly specify
which type of binding.

After a year or two of that, I suggest making the next Emacs release
so that it warns when it loads a file that doesn't explcitly specify
lexical or dynamic.

A year or two after that. all packages people use will explicitly
specify lexical or not.  At that point, we could safely change the
default, if we want to.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Why is lexical-binding's global value ignored?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: tomas@tuxteam.de, emacs-devel@gnu.org
> Date: Mon, 30 Jan 2023 23:19:25 -0500
> 
>   > Some programs cannot work with lexical binding turned on.
> 
>   > Such is the RCD Template Interpolation System for Emacs:
>   > https://hyperscope.link/3/7/1/3/3/RCD-Template-Interpolation-System-for-Emacs.html
> 
> Sooner or later we should call on the user community to fix programs
> to work with lexical binding if possible, and to explicitly specify
> which type of binding.
> 
> After a year or two of that, I suggest making the next Emacs release
> so that it warns when it loads a file that doesn't explcitly specify
> lexical or dynamic.
> 
> A year or two after that. all packages people use will explicitly
> specify lexical or not.  At that point, we could safely change the
> default, if we want to.

That is basically the plan, but we are still some years before the end
of this journey.  Also, people who are interested in this development
should bring it up far enough from the beginning of a release cycle,
for us to make such a breaking change with enough time for the
community to adapt.

Re: Why is lexical-binding's global value ignored?

[abq]

tomas wrote:
> I think it's backward compatibility, mainly. Files with no explicit
> lexical binding setting (may) contain code which expects dynamic
> binding and would break.

As I mentioned, the global value of lexical-binding defaults to nil. 
This is the case in Emacs 24 through 28, and obviously will be in 29 
too. Therefore, even if the global value were honored in 29 and future 
versions, files with no explicit setting would continue to get dynamic 
binding, as they expect. They won't break.

In order to break anything, you would have to do so explicitly, via 
(setq-default lexical-binding t).

Re: Why is lexical-binding's global value ignored?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 958 bytes --]

On Sat, Jan 28, 2023 at 09:11:41AM -0700, abq@bitrot.link wrote:
> tomas wrote:
> > I think it's backward compatibility, mainly. Files with no explicit
> > lexical binding setting (may) contain code which expects dynamic
> > binding and would break.
> 
> As I mentioned, the global value of lexical-binding defaults to nil.

Yes, you did mention that.

>                                                                     This is
> the case in Emacs 24 through 28, and obviously will be in 29 too. Therefore,
> even if the global value were honored in 29 and future versions, files with
> no explicit setting would continue to get dynamic binding, as they expect.
> They won't break.
> 
> In order to break anything, you would have to do so explicitly, via
> (setq-default lexical-binding t).

The docs only talk about a buffer-local variable, so I'd guess that the
global variable of the same name is simply ignored.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Why is lexical-binding's global value ignored?

[abq]

tomas wrote:
> The docs only talk about a buffer-local variable, so I'd guess that
> the global variable of the same name is simply ignored.

Yes, and my question is: Why not honor it, instead of ignoring it?

Re: Why is lexical-binding's global value ignored?

[Eli Zaretskii]

> Date: Sat, 28 Jan 2023 15:34:28 -0700
> From: abq@bitrot.link
> 
> tomas wrote:
> > The docs only talk about a buffer-local variable, so I'd guess that
> > the global variable of the same name is simply ignored.
> 
> Yes, and my question is: Why not honor it, instead of ignoring it?

Wasn't that already explained?  What happens if you say

  (setq-default lexical-binding t)

and then load a Lisp file that expects dynamic-binding by default?

IOW, the lack of lexical-binding: cookie in a file is interpreted as
having an explicit "lexical-binding: nil" in that file, for the
reasons Tomas explained.

Re: Why is lexical-binding's global value ignored?

[abq]

On 2023-01-28 23:54, Eli Zaretskii wrote:
> Wasn't that already explained?  What happens if you say
> 
>   (setq-default lexical-binding t)
> 
> and then load a Lisp file that expects dynamic-binding by default?

Then it breaks! As I wrote in my response to tomas: “In order to break 
anything, you would have to do so explicitly, via (setq-default 
lexical-binding t).”

Of course, nobody actually does that. Therefore, it would be safe to 
honor the global value in Emacs 29.

And the reason nobody sets the global value is that currently it would 
be pointless, because it isn't honored in Emacs 24 through 28.

There's no danger of anybody accidentally setting it with plain setq, 
since it automatically becomes buffer local when set. You have to 
purposefully shoot yourself in the foot with setq-default.


> IOW, the lack of lexical-binding: cookie in a file is interpreted as
> having an explicit "lexical-binding: nil" in that file, for the
> reasons Tomas explained.

But of course, dynamic binding isn't always used intentionally. It's 
also often used in cases where the programmer gave no consideration to 
the difference (and often, unconsciously intended lexical binding). 
Since the longstanding, widespread computer science consensus is to 
program using lexical binding except in special cases where dynamic 
binding is intentionally chosen, it's sensible to facilitate a bias in 
that direction, at least optionally.

If you use lots of historical code written with no consideration of the 
difference, one way to test the code would be to sprinkle 
lexical-binding: t across the tops of all your files, and keep track of 
which ones have it set because they actually expect it vs. which ones 
have it set just so you can test them. Which, ironically, is a bit like 
using a purely functional language with no dynamic binding or global 
variables, so you're forced to modify all your function signatures and 
calls to explicitly pass global state as an argument.

Or, a more sensible way to test your historical code would be to simply 
do (setq-default lexical-binding t) in your init file. But you can't do 
that in Emacs 24 through 28, because the global value is ignored.

If it were honored in Emacs 29, then a news item could be added, 
suggesting everybody try (setq-default lexical-binding t) in the init 
file, to test all the packages they use, then add explicit 
lexical-binding: nil (or defvar) in the cases where dynamic binding is 
intentional. People who think this is nonsense can harmlessly ignore the 
suggestion.

Re: Why is lexical-binding's global value ignored?

[Eli Zaretskii]

> Date: Sun, 29 Jan 2023 20:53:30 -0700
> From: abq@bitrot.link
> Cc: emacs-devel@gnu.org
> 
> On 2023-01-28 23:54, Eli Zaretskii wrote:
> > Wasn't that already explained?  What happens if you say
> > 
> >   (setq-default lexical-binding t)
> > 
> > and then load a Lisp file that expects dynamic-binding by default?
> 
> Then it breaks!

We want to avoid such a breakage.  It is year 2023 out there, but
there's still gobs of code that assumes dynamic binding.

> As I wrote in my response to tomas: “In order to break 
> anything, you would have to do so explicitly, via (setq-default 
> lexical-binding t).”

We decided that was not explicit enough, because many people will not
understand the disastrous consequences.  E.g., even the user's init
file can fail to load.

> Of course, nobody actually does that. Therefore, it would be safe to 
> honor the global value in Emacs 29.

Sorry, we won't, not in Emacs 29.  And that's final.  Please accept
that, even if you disagree.

Re: Why is lexical-binding's global value ignored?

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

On Mon, Jan 30, 2023 at 02:32:58PM +0200, Eli Zaretskii wrote:
> > Date: Sun, 29 Jan 2023 20:53:30 -0700
> > From: abq@bitrot.link
> > Cc: emacs-devel@gnu.org
> > 
> > On 2023-01-28 23:54, Eli Zaretskii wrote:
> > > Wasn't that already explained?  What happens if you say
> > > 
> > >   (setq-default lexical-binding t)
> > > 
> > > and then load a Lisp file that expects dynamic-binding by default?
> > 
> > Then it breaks!
> 
> We want to avoid such a breakage.  It is year 2023 out there, but
> there's still gobs of code that assumes dynamic binding.

Moreover, I still fail to understand why this is useful. Files
with no binding declaration have most probably been writen at
a time where dynamic binding was the only option [1]. So it does
make sense to treat them as if there were a "lexical-binding: f"
declaration in them. Everything else would be breaking the
interface contract. You better not do that (this would be the
same as changing the meaning of `+' for old programs: what
for?)

This means that the variable `lexical-binding' is always locally
bound, so its global binding is meaningless.

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Why is lexical-binding's global value ignored?

[Po Lu]

abq@bitrot.link writes:

> Then it breaks! As I wrote in my response to tomas: “In order to break
> anything, you would have to do so explicitly, via (setq-default
> lexical-binding t).”
>
> Of course, nobody actually does that. Therefore, it would be safe to
> honor the global value in Emacs 29.
>
> And the reason nobody sets the global value is that currently it would
> be pointless, because it isn't honored in Emacs 24 through 28.
>
> There's no danger of anybody accidentally setting it with plain setq,
> since it automatically becomes buffer local when set. You have to
> purposefully shoot yourself in the foot with setq-default.

[...]

> But of course, dynamic binding isn't always used intentionally. It's
> also often used in cases where the programmer gave no consideration to
> the difference (and often, unconsciously intended lexical
> binding). Since the longstanding, widespread computer science
> consensus is to program using lexical binding except in special cases
> where dynamic binding is intentionally chosen, it's sensible to
> facilitate a bias in that direction, at least optionally.
>
> If you use lots of historical code written with no consideration of
> the difference, one way to test the code would be to sprinkle
> lexical-binding: t across the tops of all your files, and keep track
> of which ones have it set because they actually expect it vs. which
> ones have it set just so you can test them. Which, ironically, is a
> bit like using a purely functional language with no dynamic binding or
> global variables, so you're forced to modify all your function
> signatures and calls to explicitly pass global state as an argument.
>
> Or, a more sensible way to test your historical code would be to
> simply do (setq-default lexical-binding t) in your init file. But you
> can't do that in Emacs 24 through 28, because the global value is
> ignored.
>
> If it were honored in Emacs 29, then a news item could be added,
> suggesting everybody try (setq-default lexical-binding t) in the init
> file, to test all the packages they use, then add explicit
> lexical-binding: nil (or defvar) in the cases where dynamic binding is
> intentional. People who think this is nonsense can harmlessly ignore
> the suggestion.

And why would you want to break it?
I think most of Drew Adam's packages do not use lexical binding.

Neither does much of the code which comes with other GNU software,
such as the major mode GMP provides to edit its special assembler
macros.

We must not deliberately break such software!

Re: Why is lexical-binding's global value ignored?

[abq]

Richard Stallman wrote:
> A year or two after that. all packages people use will explicitly
> specify lexical or not.  At that point, we could safely change the
> default, if we want to.

Yes, and a great way to change the default would be specifically:
First, start honoring the global value of the lexical-binding variable. 
At first, keep it nil by default, as it is now, so no behavior actually 
changes, and nothing breaks. But announce that the default will change 
in a future version. Of course, buffer local lexical-binding continues 
to work as it does now.

Second, brave users can optionally setq-default lexical-binding t in the 
init file to get the new behavior now, and report breakage they discover 
in packages they use. Users who want the old behavior, or don't care 
about the issue and have no desire to learn about it, can ignore it, and 
nothing breaks. The default remains nil.

Third, as announced long in advance, change the default to t in a future 
version, and announce that users can setq-default lexical-binding nil in 
the init file if they still want the old behavior.

The key here is honoring the global value of lexical-binding. This makes 
the transition as smooth as possible, and makes it trivially easy for 
users to opt out even after the transition is complete and the default 
is t.

The current problem is just that Emacs 24 through 28 ignores the global 
value. When loading a file, Emacs pretends that the global value is 
always nil even when it's actually t. (But of course, this entire 
discussion is irrelevant whenever lexical-binding is set buffer 
locally.)

Honoring the global value is arguably just a bug fix.


Or did you have another mechanism in mind for changing the default? 
Another possibility would be to still ignore the global value, but start 
pretending that it's always t even when it's actually nil. But surely 
nobody is advocating that Emacs ever do that. It makes much more sense 
to simply honor the global value (which could start right now, with no 
harm), then change the global default to t sometime in the future.

There is a potential problem: Eli worries that “many people will not 
understand the disastrous consequences” of doing (setq-default 
lexical-binding t), and therefore he decided to delay honoring the 
global value. But a prominent warning DO NOT TOUCH THIS GLOBAL VARIABLE 
UNLESS YOU'RE AN EXPERT would be just as effective for Emacs 29 as it 
would be for Emacs 30 or even 42; the delay accomplishes nothing. And 
people who don't read the news at all, and just blindly upgrade and 
continue using their init file and packages unmodified, will experience 
no breakage, because the default remains nil. No adaptation is required, 
by anybody.

There is another theoretical problem. Maybe somebody, somewhere, 
irrationally already does (setq-default lexical-binding t) in his init 
file, to no effect, and then (even more irrationally) subsequently 
relies on dynamic binding despite explicitly trying to set lexical 
binding. Any future version of Emacs that starts honoring the global 
value will break his config. But abstaining on his behalf is as 
unreasonable as total abstinence from ever adding any new global 
variable to any future version of Emacs ever again, due to the risk that 
the variable will conflict with somebody's private usage of it.

Eli's decision is final, but it's worthwhile to make the situation 
clear.