Most of Elisp lacks lexical-binding: t

Most of Elisp lacks lexical-binding: t

[Simon Reiser]

[-- Attachment #1: Type: text/html, Size: 687 bytes --]

Re: Most of Elisp lacks lexical-binding: t

[Alan Mackenzie]

Hello, Simon.

On Sat, Feb 02, 2019 at 13:35:03 +0100, Simon Reiser wrote:
>    Is there a reason, that out of ~1800 elisp files only 65 of them enable
>    lexical-bindings?

Yes, indeed.  Originally, Emacs Lisp only had dynamic binding, and
lexical binding is a relatively recent innovation.  It is work to
convert files to use lexical bindings, and so far nobody has put in this
work for most of the Lisp sources.

But by my count there are 551 out of 1528 files in the master branch
which have lexical binding enabled.  That's something over a third.

>    AFAIK lexical-binding allows for faster byte-code to be generated.
>    Maybe it would be a good idea to enable it in most of them and
>    document those files where it cannot be enabled.

My feeling is that most files could just be changed without problems,
but there will be enough exceptions that it needs to be done carefully.

Maybe it would be a good idea, but somebody(R) has got to step forward
and volunteer to do the work.  

>    Cheers,

>    Simon

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Most of Elisp lacks lexical-binding: t

[Stefan Monnier]

> My feeling is that most files could just be changed without problems,
> but there will be enough exceptions that it needs to be done carefully.

FWIW, my local Emacs build is hacked so that it always compiles with
lexical-binding enabled.  In practice, it seems that most packages work
just fine with it, but there are indeed very definite exceptions, and
even for those packages that appear to work, the change may introduce
subtle regressions.

So, yes, it has to be done "by hand" and takes time.  Help is
most welcome.  The best is to do it with the packages that you use, so
you can more easily test the result.


        Stefan

Re: Most of Elisp lacks lexical-binding: t

[T.V Raman]

Stefan Monnier <monnier@iro.umontreal.ca> writes:


I did this for emacspeak a few years ago --- and most issues were caught
by byte-compiler warnings. That said, the emacspeak codebase over the
years has used (cl-declare (special ...)) for special variables, which
likely helped with managing spurious byte-compiler warnings.
>> My feeling is that most files could just be changed without problems,
>> but there will be enough exceptions that it needs to be done carefully.
>
> FWIW, my local Emacs build is hacked so that it always compiles with
> lexical-binding enabled.  In practice, it seems that most packages work
> just fine with it, but there are indeed very definite exceptions, and
> even for those packages that appear to work, the change may introduce
> subtle regressions.
>
> So, yes, it has to be done "by hand" and takes time.  Help is
> most welcome.  The best is to do it with the packages that you use, so
> you can more easily test the result.
>
>
>         Stefan
>
>

-- 

Re: Most of Elisp lacks lexical-binding: t

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> My feeling is that most files could just be changed without problems,
>> but there will be enough exceptions that it needs to be done carefully.
>
> FWIW, my local Emacs build is hacked so that it always compiles with
> lexical-binding enabled.  In practice, it seems that most packages work
> just fine with it, but there are indeed very definite exceptions, and
> even for those packages that appear to work, the change may introduce
> subtle regressions.
>
> So, yes, it has to be done "by hand" and takes time.  Help is
> most welcome.  The best is to do it with the packages that you use, so
> you can more easily test the result.


The question is, though, at what point does it become the default?

One answer would be, support a "dynamic-binding" cookie line for
Emacs-27 (which does nothing by default), incorporate your
lexical-binding hack as an option. Then, go to lexical by default for
Emacs-28.

Phil

Re: Most of Elisp lacks lexical-binding: t

[Stefan Monnier]

> The question is, though, at what point does it become the default?

IIUC Eli is not in a hurry to see it become the default.

> One answer would be, support a "dynamic-binding" cookie line for
> Emacs-27 (which does nothing by default),

You can already use

    -*- lexical-binding:nil -*-

for that.

> incorporate your lexical-binding hack as an option.
> Then, go to lexical by default for Emacs-28.

This sounds rather optimistic to me.  We should at least start by giving
some warnings from the byte-compiler when compiling am Elisp file without
lexical-binding.  Maybe same thing when visiting an Elisp file.

There are also all the calls to `eval` which currently tend to mostly
still use dynamic binding (even in files labeled with
lexical-binding:t).  E.g. we could advertise `eval` as taking a minimum
of 2 args so the byte-compiler would start warning about calls that
don't explicitly request lexical or dynamic binding (and hence fall
back to the default dynamic binding).


        Stefan

Re: Most of Elisp lacks lexical-binding: t

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Date: Mon, 04 Feb 2019 16:54:04 -0500
> Cc: emacs-devel@gnu.org
> 
> > The question is, though, at what point does it become the default?
> 
> IIUC Eli is not in a hurry to see it become the default.

I don't see how we could make this the default before we complete
converting all the Lisp files to lexical binding (or, unlikely, decide
that some of them need to use dynamic binding).  It would mean risking
a lot of breakage.

Re: Most of Elisp lacks lexical-binding: t

[Phil Sainty]

On 2019-02-05 16:31, Eli Zaretskii wrote:
> I don't see how we could make this the default before we complete
> converting all the Lisp files to lexical binding (or, unlikely, decide
> that some of them need to use dynamic binding).  It would mean risking
> a lot of breakage.

And that's not just for Emacs core -- there is a massive quantity of
elisp in the ecosystem at large which was not written for lexical
binding, and still works absolutely fine in current versions of Emacs.
Such libraries might well no longer have a maintainer (but at present
not be in any need of one).

Releasing a "lexical-binding by default" version of Emacs would surely
create a substantial risk of many user's configs failing on them in a
variety of ways when they upgrade, and suddenly needing a lot of
effort to get working (which for some users will be outside of their
skill set).

The risk seems rather high to me.  I'm not sure the reward would be
worth it?  (When the reward would seem to be nothing more than library
authors being able to omit -*- lexical-binding: t; -*- which, even if
it looks untidy, isn't exactly an onerous requirement).


-Phil

Re: Most of Elisp lacks lexical-binding: t

[Stefan Monnier]

> And that's not just for Emacs core -- there is a massive quantity of
> Elisp in the ecosystem at large which was not written for lexical
> binding, and still works absolutely fine in current versions of Emacs.
> Such libraries might well no longer have a maintainer (but at present
> not be in any need of one).

Yup.  The path to getting rid of lexical-binding=nil will be long.
Also, in my experience, the older the package the more likely it is to
rely in tricky ways on dynamic scoping (by which I mean that some uses
won't be caught by the compiler warnings).

This needs to be done step by step and there will be many steps.
I proposed some steps a year ago, BTW.  See bug#30078.

> The risk seems rather high to me.  I'm not sure the reward would be
> worth it?  (When the reward would seem to be nothing more than library
> authors being able to omit -*- lexical-binding: t; -*- which, even if
> it looks untidy, isn't exactly an onerous requirement).

For me the issue is that those two settings define two slightly
different languages, and that increases complexity for newcomers (and
for old hats as well: I still regularly find myself writing code that
assumes lexical-binding is non-nil and using it in a context where that
assumption is not valid).


        Stefan

Re: Most of Elisp lacks lexical-binding: t

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> And that's not just for Emacs core -- there is a massive quantity of
>> Elisp in the ecosystem at large which was not written for lexical
>> binding, and still works absolutely fine in current versions of Emacs.
>> Such libraries might well no longer have a maintainer (but at present
>> not be in any need of one).
>
> Yup.  The path to getting rid of lexical-binding=nil will be long.

Longer still if it is not started.

> Also, in my experience, the older the package the more likely it is to
> rely in tricky ways on dynamic scoping (by which I mean that some uses
> won't be caught by the compiler warnings).

Put code in so that if copyright year < 2018 then use dynamic by
default, otherwise use lexical?

(This is not a serious suggestion, just in case there is any doubt).

> This needs to be done step by step and there will be many steps.
> I proposed some steps a year ago, BTW.  See bug#30078.
>
>> The risk seems rather high to me.  I'm not sure the reward would be
>> worth it?  (When the reward would seem to be nothing more than library
>> authors being able to omit -*- lexical-binding: t; -*- which, even if
>> it looks untidy, isn't exactly an onerous requirement).
>
> For me the issue is that those two settings define two slightly
> different languages, and that increases complexity for newcomers (and
> for old hats as well: I still regularly find myself writing code that
> assumes lexical-binding is non-nil and using it in a context where that
> assumption is not valid).

I think it also limits the uptake of lexical-binding and the ability to
depend on it, because it requires some additional work.

It will break things, no doubt about it. I suspect it will break things
whenever it happens, though, so this isn't a reason for not doing it.

Phil

Re: Most of Elisp lacks lexical-binding: t

[Stefan Monnier]

>> Yup.  The path to getting rid of lexical-binding=nil will be long.
> Longer still if it is not started.

Indeed.  Maybe you want to weigh in at bug#30078.

> Put code in so that if copyright year < 2018 then use dynamic by
> default, otherwise use lexical?

Good idea.  Tho I think we might be better off looking at the file's
timestamp instead, since many files don't have a reliably recognizable
copyright year.

> (This is not a serious suggestion, just in case there is any doubt).

This ship has sailed.

> It will break things, no doubt about it. I suspect it will break things
> whenever it happens, though, so this isn't a reason for not doing it.

Breakage will be inevitable, indeed, but we can spread the pain over
some number of years.


        Stefan

Re: Most of Elisp lacks lexical-binding: t

[Paul Eggert]

On 2/6/19 2:47 PM, Stefan Monnier wrote:
> Breakage will be inevitable, indeed, but we can spread the pain over
> some number of years.

Fast bandaid removal causes significantly less pain than slow. See:

Furyk JS, O'Kane CJ, Aitken PJ, Banks CJ, Kault DA. Fast versus slow 
bandaid removal: a randomised trial. Med J Aust. 2009 Dec 
7-21;191(11-12):682-3. https://www.ncbi.nlm.nih.gov/pubmed/20028307 
https://doi.org/10.5694/j.1326-5377.2009.tb03379.x

Re: Most of Elisp lacks lexical-binding: t

[Stefan Monnier]

>> Breakage will be inevitable, indeed, but we can spread the pain over
>> some number of years.
> Fast bandaid removal causes significantly less pain than slow. See:

I don't think the analogy holds.

E.g. one of the main motivations to "spread the pain" is that when
someone bumps into something that breaks because of the new use of
lexical-binding in one context, it may motivate him to proactively fix
the problem in other contexts (where the problem doesn't appear yet but
would appear at one of the next steps in the transition).  So I think
the overall pain is reduced.

This said, I do believe that we could speed this up by making
"development builds" (i.e. builds where emacs-version has 4 elements)
a lot more aggressive wrt to dropping backward compatibility: the
average user relying on normal releases would not be affected, but those
living on the edge would be more often motivated to update the packages
they maintain or use.


        Stefan

Re: Most of Elisp lacks lexical-binding: t

[Stefan Monnier]

>> > The question is, though, at what point does it become the default?
>> IIUC Eli is not in a hurry to see it become the default.
> I don't see how we could make this the default before we complete
> converting all the Lisp files to lexical binding (or, unlikely, decide
> that some of them need to use dynamic binding).

Indeed.


        Stefan

Re: Most of Elisp lacks lexical-binding: t

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   E.g. we could advertise `eval` as taking a minimum
  > of 2 args so the byte-compiler would start warning about calls that
  > don't explicitly request lexical or dynamic binding (and hence fall
  > back to the default dynamic binding).

Perhaps it would be better to define a new function, perhaps
`eval-lexically', for specifying a lexical context.
If these calls need to be changed anyway, changing the function name
is no extra work.  And the changed function name will make it
easy to search for what has or has not been changed.

We also need to keep track of which calls to eval should not be
changed.  There are various ways to do that -- for instance, we could
label files that contain calls to eval when all those calls have been
checked.  Or we could define a new name, `eval-dynamically', to use in
calls which people have verified should not be changed.

Let's choose a way and start using it.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Most of Elisp lacks lexical-binding: t

[Stefan Monnier]

> We also need to keep track of which calls to eval should not be
> changed.

I'm locally using

    (set-advertised-calling-convention 'eval '(form lexical-environment) "future")

which means the byte-compiler emits a warning whenever `eval` is called
with a single argument.  With this arrangement, you can write (eval foo
nil) when you really want to use dynamic binding (tho this has never
happened to me yet, and I'm not sure it should ever happen).


        Stefan