bug#74145: 31.0.50; Default lexical-binding to t

bug#74145: 31.0.50; Default lexical-binding to t

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 151 bytes --]

Package: Emacs
Version: 31.0.50


I believe the time has come to change the default dialect.
I attached a suggested patch to do that.


        Stefan

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Change-the-default-value-of-lexical-binding.patch --]
[-- Type: text/x-diff, Size: 13805 bytes --]

From 973939b9dd79735f051c99f1a20d74aabad931ff Mon Sep 17 00:00:00 2001
From: Stefan Monnier <monnier@iro.umontreal.ca>
Date: Thu, 31 Oct 2024 16:00:10 -0400
Subject: [PATCH] Change the default value of `lexical-binding`

* src/lread.c (syms_of_lread): Set `Vlexical_binding` to Qt.
(Fload, Feval_buffer): Don't assume the default value of
`lexical-binding` is nil.
* lisp/loadup.el: Don't assume the default value of `lexical-binding`
is nil.

* doc/misc/cl.texi (Porting Common Lisp):
* doc/lispref/variables.texi (Variable Scoping)
(Selecting Lisp Dialect): Adjust to new default of `lexical-binding`.

* doc/lispref/edebug.texi (Edebug Eval): Remove out of date paragraph.

* doc/lispintro/emacs-lisp-intro.texi (How let Binds Variables)
(Lexical vs. Dynamic Binding Example, defvar): Only consider the
lexical dialect of ELisp.
---
 doc/lispintro/emacs-lisp-intro.texi | 26 ++++++++----------------
 doc/lispref/edebug.texi             |  7 -------
 doc/lispref/variables.texi          | 31 +++++++++++++++--------------
 doc/misc/cl.texi                    |  9 +++++----
 etc/NEWS                            |  6 ++++++
 lisp/loadup.el                      |  4 ++--
 src/lread.c                         | 14 +++++++++----
 7 files changed, 47 insertions(+), 50 deletions(-)

diff --git a/doc/lispintro/emacs-lisp-intro.texi b/doc/lispintro/emacs-lisp-intro.texi
index 49916235fbf..6cc7a5b3cf2 100644
--- a/doc/lispintro/emacs-lisp-intro.texi
+++ b/doc/lispintro/emacs-lisp-intro.texi
@@ -3754,21 +3754,12 @@ How let Binds Variables
 
 Emacs Lisp supports two different ways of binding variable names to
 their values.  These ways affect the parts of your program where a
-particular binding is valid.  For historical reasons, Emacs Lisp uses
-a form of variable binding called @dfn{dynamic binding} by default.
-However, in this manual we discuss the preferred form of binding,
-called @dfn{lexical binding}, unless otherwise noted (in the future,
-the Emacs maintainers plan to change the default to lexical binding).
+particular binding is valid.
+In this manual we discuss the preferred form of binding,
+called @dfn{lexical binding}, unless otherwise noted.
 If you have programmed in other languages before, you're likely
 already familiar with how lexical binding behaves.
 
-In order to use lexical binding in a program, you should add this to
-the first line of your Emacs Lisp file:
-
-@example
-;;; -*- lexical-binding: t -*-
-@end example
-
 For more information about this, @pxref{Variable Scoping, , ,
 elisp, The Emacs Lisp Reference Manual}.
 
@@ -3818,8 +3809,6 @@ Lexical vs. Dynamic Binding Example
 For example, see what happens in this code under lexical binding:
 
 @example
-;;; -*- lexical-binding: t -*-
-
 (setq x 0)
 
 (defun getx ()
@@ -3845,7 +3834,7 @@ Lexical vs. Dynamic Binding Example
 If we use dynamic binding instead, the behavior is different:
 
 @example
-;;; -*- lexical-binding: nil -*-
+(defvar x) ;; Use dynamic binding for 'x'.
 
 (setq x 0)
 
@@ -3867,8 +3856,9 @@ Lexical vs. Dynamic Binding Example
 @code{x}, since its binding is below the one from our @code{let}
 expression in the stack of bindings.
 
-(Some variables are also ``special'', and they are always dynamically
-bound even when @code{lexical-binding} is @code{t}.  @xref{defvar, ,
+(The @code{defvar} declaration above is said to make the variable
+``special'', which causes it to obey the dynamic binding rules instead of
+the default lexical binding rules.  @xref{defvar, ,
 Initializing a Variable with @code{defvar}}.)
 
 @node if
@@ -9229,7 +9219,7 @@ defvar
 The @code{defvar} special form is similar to @code{setq} in that it
 sets the value of a variable.  It is unlike @code{setq} in three ways:
 first, it marks the variable as ``special'' so that it is always
-dynamically bound, even when @code{lexical-binding} is @code{t}
+dynamically bound
 (@pxref{How let Binds Variables}).  Second, it only sets the value of
 the variable if the variable does not already have a value.  If the
 variable already has a value, @code{defvar} does not override the
diff --git a/doc/lispref/edebug.texi b/doc/lispref/edebug.texi
index 03fae67e528..fe617550b09 100644
--- a/doc/lispref/edebug.texi
+++ b/doc/lispref/edebug.texi
@@ -722,13 +722,6 @@ Edebug Eval
 pretty-printed in a separate buffer.
 @end table
 
-@cindex lexical binding (Edebug)
-  Edebug supports evaluation of expressions containing references to
-lexically bound symbols created by the following constructs in
-@file{cl.el}: @code{lexical-let}, @code{macrolet}, and
-@code{symbol-macrolet}.
-@c FIXME?  What about lexical-binding = t?
-
 @node Eval List
 @subsection Evaluation List Buffer
 
diff --git a/doc/lispref/variables.texi b/doc/lispref/variables.texi
index af11e498d86..fa5f792b40f 100644
--- a/doc/lispref/variables.texi
+++ b/doc/lispref/variables.texi
@@ -285,7 +285,7 @@ Local Variables
 @defspec letrec (bindings@dots{}) forms@dots{}
 This special form is like @code{let*}, but all the variables are bound
 before any of the local values are computed.  The values are then
-assigned to the locally bound variables.  This is only useful when
+assigned to the locally bound variables.  This is useful only when
 lexical binding is in effect, and you want to create closures that
 refer to bindings that would otherwise not yet be in effect when using
 @code{let*}.
@@ -351,7 +351,7 @@ Local Variables
 done so that the value returned by the call is the value of @var{body}
 itself, as is the case in the recursive call to @code{sum} above.
 
-@code{named-let} can only be used when lexical-binding is enabled.
+@code{named-let} can be used only when lexical-binding is enabled.
 @xref{Lexical Binding}.
 @end defspec
 
@@ -392,7 +392,7 @@ Void Variables
 to evaluate the variable signals a @code{void-variable} error, instead
 of returning a value.
 
-  Under the optional lexical scoping rule, the value cell only holds
+  Under the optional lexical scoping rule, the value cell holds only
 the variable's global value---the value outside of any lexical binding
 construct.  When a variable is lexically bound, the local value is
 determined by the lexical environment; hence, variables can have local
@@ -514,7 +514,7 @@ Defining Variables
 
 Note that specifying a value, even @code{nil}, marks the variable as
 special permanently.  Whereas if @var{value} is omitted then the
-variable is only marked special locally (i.e.@: within the current
+variable is marked special only locally (i.e.@: within the current
 lexical scope, or file if at the top-level).  This can be useful for
 suppressing byte compilation warnings, see @ref{Compiler Errors}.
 
@@ -706,8 +706,8 @@ Accessing Variables
   The usual way to reference a variable is to write the symbol which
 names it.  @xref{Symbol Forms}.
 
-  Occasionally, you may want to reference a variable which is only
-determined at run time.  In that case, you cannot specify the variable
+  Occasionally, you may want to reference a variable which is determined
+only at run time.  In that case, you cannot specify the variable
 name in the text of the program.  You can use the @code{symbol-value}
 function to extract the value.
 
@@ -1003,14 +1003,15 @@ Variable Scoping
 while the binding construct (such as the body of a @code{let} form) is
 being executed.
 
-  The old dynamic-only Emacs Lisp dialect is still the default in code
+  The new lexically scoped Emacs Lisp dialect is now the default in code
 loaded or evaluated from Lisp files that lack a dialect declaration.
-Eventually the modern dialect will be made the default.
-All Lisp files should declare the dialect used to ensure that they
-keep working correctly in the future.
+Lisp files which still need the old dynamically scoped dialect need to declare
+it explicitly to ensure that they keep working correctly, and should
+eventually be converted to use the new dialect to
+ensure that they keep working correctly in the future.
 
   The following subsections describe lexical binding and dynamic
-binding in greater detail, and how to enable lexical binding in Emacs
+binding in greater detail, and how to disable lexical binding in Emacs
 Lisp programs.
 
 @menu
@@ -1319,8 +1320,8 @@ Selecting Lisp Dialect
 
 @noindent
 for the old dynamic-only dialect.  When no declaration is present the
-old dialect is used, but this may change in a future release.
-The compiler will warn if no declaration is present.
+new dialect is used.  Support for the old dialect may be removed in
+a future release.
 
 When evaluating Emacs Lisp code directly using an @code{eval} call,
 lexical binding is enabled if the @var{lexical} argument to
@@ -1980,8 +1981,8 @@ File Local Variables
 
 @defvar permanently-enabled-local-variables
 Some local variable settings will, by default, be heeded even if
-@code{enable-local-variables} is @code{nil}.  By default, this is only
-the case for the @code{lexical-binding} local variable setting, but
+@code{enable-local-variables} is @code{nil}.  By default, this is the
+case only for the @code{lexical-binding} local variable setting, but
 this can be controlled by using this variable, which is a list of
 symbols.
 @end defvar
diff --git a/doc/misc/cl.texi b/doc/misc/cl.texi
index 113029700ec..395725944f0 100644
--- a/doc/misc/cl.texi
+++ b/doc/misc/cl.texi
@@ -4758,12 +4758,13 @@ Porting Common Lisp
 @item
 Lexical scoping.  In Common Lisp, function arguments and @code{let}
 bindings apply only to references physically within their bodies (or
-within macro expansions in their bodies).  Traditionally, Emacs Lisp
-uses @dfn{dynamic scoping} wherein a binding to a variable is visible
+within macro expansions in their bodies).  The is also the case in
+modern Emacs Lisp, but historically, Emacs Lisp
+used @dfn{dynamic scoping} wherein a binding to a variable is visible
 even inside functions called from the body.
 @xref{Dynamic Binding,,,elisp,GNU Emacs Lisp Reference Manual}.
-Lexical binding is available since Emacs 24.1, so be sure to set
-@code{lexical-binding} to @code{t} if you need to emulate this aspect
+Lexical binding is available since Emacs 24.1, but be sure not to set
+@code{lexical-binding} to @code{nil} if you need to emulate this aspect
 of Common Lisp.  @xref{Lexical Binding,,,elisp,GNU Emacs Lisp Reference Manual}.
 
 Here is an example of a Common Lisp code fragment that would fail in
diff --git a/etc/NEWS b/etc/NEWS
index d1c7303f976..5a480c5f68a 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -636,6 +636,12 @@ the directory into which the repository was cloned.
 \f
 * Incompatible Lisp Changes in Emacs 31.1
 
++++
+** 'lexical-binding' now defaults to t.
+The default ELisp dialect is now the lexically scoped dialect.
+Any code which has not yet been adapted should either be
+adapted or set 'lexical-binding' to nil explicitly.
+
 ** Nested backquotes are not supported any more in Pcase patterns.
 
 ** The 'rx' category name 'chinese-two-byte' must now be spelled correctly.
diff --git a/lisp/loadup.el b/lisp/loadup.el
index bd74a9d6aff..e676c338c14 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -654,9 +654,9 @@ comp-subr-arities-h
           (unwind-protect
               (let ((tmp-dump-mode dump-mode)
                     (dump-mode nil)
-                    ;; Set `lexical-binding' to nil by default
+                    ;; Set `lexical-binding' to its default value
                     ;; in the dumped Emacs.
-                    (lexical-binding nil))
+                    (lexical-binding (default-toplevel-value 'lexical-binding)))
                 (if (member tmp-dump-mode '("pdump" "pbootstrap"))
                     (dump-emacs-portable (expand-file-name output invocation-directory))
                   (dump-emacs output (if (eq system-type 'ms-dos)
diff --git a/src/lread.c b/src/lread.c
index ea0398196e3..9f56880e551 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -1522,7 +1522,7 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
      otherwise using a file-variable in the first line.  This is bound here
      so that it takes effect whether or not we use
      Vload_source_file_function.  */
-  specbind (Qlexical_binding, Qnil);
+  specbind (Qlexical_binding, Fdefault_toplevel_value (Qlexical_binding));
 
   Lisp_Object found_eff =
     is_native_elisp
@@ -1720,8 +1720,11 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
     }
   else
     {
-      if (lisp_file_lexical_cookie (Qget_file_char) == Cookie_Lex)
+      lexical_cookie_t lexc = lisp_file_lexical_cookie (Qget_file_char);
+      if (lexc == Cookie_Lex)
         Fset (Qlexical_binding, Qt);
+      else if (lexc == Cookie_Dyn)
+        Fset (Qlexical_binding, Qnil);
 
       if (! version || version >= 22)
         readevalloop (Qget_file_char, &input, hist_file_name,
@@ -2606,8 +2609,11 @@ DEFUN ("eval-buffer", Feval_buffer, Seval_buffer, 0, 5, "",
   specbind (Qstandard_output, tem);
   record_unwind_protect_excursion ();
   BUF_TEMP_SET_PT (XBUFFER (buf), BUF_BEGV (XBUFFER (buf)));
+  lexical_cookie_t lexc = lisp_file_lexical_cookie (buf);
   specbind (Qlexical_binding,
-	    lisp_file_lexical_cookie (buf) == Cookie_Lex ? Qt : Qnil);
+	    lexc == Cookie_Lex ? Qt
+	    : lexc == Cookie_Dyn ? Qnil
+	    : Fdefault_toplevel_value (Qlexical_binding));
   BUF_TEMP_SET_PT (XBUFFER (buf), BUF_BEGV (XBUFFER (buf)));
   readevalloop (buf, 0, filename,
 		!NILP (printflag), unibyte, Qnil, Qnil, Qnil);
@@ -6056,7 +6062,7 @@ syms_of_lread (void)
 This variable is automatically set from the file variables of an
 interpreted Lisp file read using `load'.  Unlike other file local
 variables, this must be set in the first line of a file.  */);
-  Vlexical_binding = Qnil;
+  Vlexical_binding = Qt;
   Fmake_variable_buffer_local (Qlexical_binding);
 
   DEFVAR_LISP ("eval-buffer-list", Veval_buffer_list,
-- 
2.39.5

bug#74145: 31.0.50; Default lexical-binding to t

[Stefan Kangas]

Stefan Monnier via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> I believe the time has come to change the default dialect.
> I attached a suggested patch to do that.

Sounds good to me.  Legacy code can still run with a lexical-binding:nil
cookie, so I think we're good in the backwards-compatibility department.
Andrea and Eli?

Some minor comments and questions on your patch below.

> @@ -392,7 +392,7 @@ Void Variables
>  to evaluate the variable signals a @code{void-variable} error, instead
>  of returning a value.
>
> -  Under the optional lexical scoping rule, the value cell only holds
> +  Under the optional lexical scoping rule, the value cell holds only
>  the variable's global value---the value outside of any lexical binding
>  construct.  When a variable is lexically bound, the local value is
>  determined by the lexical environment; hence, variables can have local

I think this section might need more work.  It's a bit confusing to talk
about "default dynamic scoping" and "optional lexical scoping" now.
Shouldn't we talk about "normal variables with lexical scoping" and
"special variables with dynamic scoping" or something along those lines?

> diff --git a/lisp/loadup.el b/lisp/loadup.el
> index bd74a9d6aff..e676c338c14 100644
> --- a/lisp/loadup.el
> +++ b/lisp/loadup.el
> @@ -654,9 +654,9 @@ comp-subr-arities-h
>            (unwind-protect
>                (let ((tmp-dump-mode dump-mode)
>                      (dump-mode nil)
> -                    ;; Set `lexical-binding' to nil by default
> +                    ;; Set `lexical-binding' to its default value
>                      ;; in the dumped Emacs.
> -                    (lexical-binding nil))
> +                    (lexical-binding (default-toplevel-value 'lexical-binding)))
>                  (if (member tmp-dump-mode '("pdump" "pbootstrap"))
>                      (dump-emacs-portable (expand-file-name output invocation-directory))
>                    (dump-emacs output (if (eq system-type 'ms-dos)

Why not just use `t` here instead of its `default-toplevel-value`?

> diff --git a/src/lread.c b/src/lread.c
> index ea0398196e3..9f56880e551 100644
> --- a/src/lread.c
> +++ b/src/lread.c
> @@ -1522,7 +1522,7 @@ DEFUN ("load", Fload, Sload, 1, 5, 0,
>       otherwise using a file-variable in the first line.  This is bound here
>       so that it takes effect whether or not we use
>       Vload_source_file_function.  */
> -  specbind (Qlexical_binding, Qnil);
> +  specbind (Qlexical_binding, Fdefault_toplevel_value (Qlexical_binding));
>
>    Lisp_Object found_eff =
>      is_native_elisp

And here?

> @@ -2606,8 +2609,11 @@ DEFUN ("eval-buffer", Feval_buffer, Seval_buffer, 0, 5, "",
>    specbind (Qstandard_output, tem);
>    record_unwind_protect_excursion ();
>    BUF_TEMP_SET_PT (XBUFFER (buf), BUF_BEGV (XBUFFER (buf)));
> +  lexical_cookie_t lexc = lisp_file_lexical_cookie (buf);
>    specbind (Qlexical_binding,
> -	    lisp_file_lexical_cookie (buf) == Cookie_Lex ? Qt : Qnil);
> +	    lexc == Cookie_Lex ? Qt
> +	    : lexc == Cookie_Dyn ? Qnil
> +	    : Fdefault_toplevel_value (Qlexical_binding));
>    BUF_TEMP_SET_PT (XBUFFER (buf), BUF_BEGV (XBUFFER (buf)));
>    readevalloop (buf, 0, filename,
>  		!NILP (printflag), unibyte, Qnil, Qnil, Qnil);

And here?

bug#74145: 31.0.50; Default lexical-binding to t

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Stefan,

>> @@ -392,7 +392,7 @@ Void Variables
>>  to evaluate the variable signals a @code{void-variable} error, instead
>>  of returning a value.
>>
>> -  Under the optional lexical scoping rule, the value cell only holds
>> +  Under the optional lexical scoping rule, the value cell holds only
>>  the variable's global value---the value outside of any lexical binding
>>  construct.  When a variable is lexically bound, the local value is
>>  determined by the lexical environment; hence, variables can have local
>
> I think this section might need more work.  It's a bit confusing to talk
> about "default dynamic scoping" and "optional lexical scoping" now.
> Shouldn't we talk about "normal variables with lexical scoping" and
> "special variables with dynamic scoping" or something along those lines?

Very good point, will take a closer look.

>> +                    (lexical-binding (default-toplevel-value 'lexical-binding)))
> Why not just use `t` here instead of its `default-toplevel-value`?

I "fixed" the code such that the default is set at exactly one place,
i.e. in `syms_of_lread`, but indeed it's not indispensable.

It's not super important, but it seemed cleaner.
It also makes it easy for users to "go back" to the nil default,
e.g. for testing purposes, or as a quick fix until you can track down
the `.el` files that need to be adjusted.


        Stefan

bug#74145: 31.0.50; Default lexical-binding to t

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I think this section might need more work.  It's a bit confusing to talk
>> about "default dynamic scoping" and "optional lexical scoping" now.
>> Shouldn't we talk about "normal variables with lexical scoping" and
>> "special variables with dynamic scoping" or something along those lines?
>
> Very good point, will take a closer look.

Great, thanks.

Meanwhile, here's two more things I thought of:

- I think there's some text in doc/lispref/tips.texi that will need
  updating too.

- Should we provide any advice somewhere for Emacs Lisp authors that
  want to support Emacs 30 or older?  For example, they will want to
  keep the lexical-binding:t cookie.