News from the lexical world

News from the lexical world

[Stefan Monnier]

Sorry to interrupt your great keybindings discussion, but I'll just
point out that we have now converted to lexical-binding a bit over 80%
of Emacs's ELisp files.

The breakdown of the remaining work is:

      2 test/manual/cedet/
      3 lisp/org/
      4 lisp/cedet/semantic/analyze/
      4 test/manual/
      8 lisp/vc/
      9 lisp/cedet/
      9 lisp/cedet/semantic/bovine/
      9 lisp/cedet/semantic/wisent/
      9 lisp/emulation/
     15 lisp/net/
     18 lisp/mail/
     21 lisp/textmodes/
     22 lisp/cedet/srecode/
     25 lisp/erc/
     25 lisp/mh-e/
     26 lisp/cedet/ede/
     37 lisp/progmodes/
     45 lisp/cedet/semantic/
     49 lisp/obsolete/
     59 lisp/

    399 / 2003 = 19.920119820269594%

I don't expect we'll reach 100% soon but 90% seems quite feasible for
the actual Emacs-28 release.


        Stefan

Re: News from the lexical world

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Sorry to interrupt your great keybindings discussion, but I'll just
> point out that we have now converted to lexical-binding a bit over 80%
> of Emacs's ELisp files.
[...]
>     399 / 2003 = 19.920119820269594%

For comparison, the same figure on the emacs-27 branch is:

    1055 / 1871 = 56.38695884553715 %

IOW, we have already made some good progress for Emacs 28.  But there is
no reason to slow down now -- let's get it done.

If anyone is willing to help convert a couple of files, for example ones
that they are already familiar with, it would help us progress even
faster here.

Re: News from the lexical world

[Lars Ingebrigtsen]

Stefan Kangas <stefankangas@gmail.com> writes:

>> Sorry to interrupt your great keybindings discussion, but I'll just
>> point out that we have now converted to lexical-binding a bit over 80%
>> of Emacs's ELisp files.
> [...]
>>     399 / 2003 = 19.920119820269594%
>
> For comparison, the same figure on the emacs-27 branch is:
>
>     1055 / 1871 = 56.38695884553715 %

That's awesome -- then it sounds like Emacs will be at 100% in Emacs 29
sometime, I guess?  (Or in Emacs 28 if that drags on way too long.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: News from the lexical world

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Sorry to interrupt your great keybindings discussion, but I'll just
> point out that we have now converted to lexical-binding a bit over 80%
> of Emacs's ELisp files.

This is great news!

Thanks to everyone involved into this.

  Andrea

Re: News from the lexical world

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Sorry to interrupt your great keybindings discussion, but I'll just
> point out that we have now converted to lexical-binding a bit over 80%
> of Emacs's ELisp files.


That's an amazing job!

Phil

Re: News from the lexical world

[Stefan Monnier]

Stefan Monnier [2021-02-12 11:23:13] wrote:
> Sorry to interrupt your great keybindings discussion, but I'll just
> point out that we have now converted to lexical-binding a bit over 80%
> of Emacs's ELisp files.

Things have progressed a bit since.  We're now down to:

      1 lisp/org/
      1 lisp/progmodes/

    2 / 2019 = 0.099058940069341253%

I.e. there are only 2 files left in Emacs's ELisp codebase which come
without the `lexical-binding:t` cookie.  These are `org-agenda.el`
(which has already been converted upstream, so we're just waiting for
that code to make its way to an Org release and then to Emacs's
`master`) and `verilog-mode.el` (where the situation is a good bit more
delicate: verilog-mode can be used with Verilog files written
specifically for verilog-mode, with embedded ELisp snippets run via
`eval`, and those snippets tend to use "arbitrary" variables which
happened to be bound at that time, so we're still working on
a satisfactory solution, which will likely involve documenting which of
those variables can be used that way, along with what they mean).

This doesn't quite mean that all of Emacs's ELisp code is now using the
new dialect.  The old non-lexbind dialect is still used in a few cases:

- There are still a fair number of calls to `eval` with a nil (or, more
  often, absent) second argument.

- There is also still some code which builds lambda expressions "by
  hand" (typically via backquote or otherwise embedded inside a larger
  quoted piece of data) and without passing the resulting "source code
  expression" to `eval` (or to the byte-compiler) to turn it into
  a "function value".  In the dynbind world those two were one and the
  same, but not so in the lexbind world, and a value that starts with
  a `lambda` will see its body evaluated using the old ELisp dialect.

I welcome help updating those cases.


        Stefan

Re: News from the lexical world

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Things have progressed a bit since.  We're now down to:
>
>       1 lisp/org/
>       1 lisp/progmodes/
>
>     2 / 2019 = 0.099058940069341253%

Wow, that's amazing.  Thank you very much, Stefan & Stefan.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: News from the lexical world

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Stefan Monnier [2021-02-12 11:23:13] wrote:
>> Sorry to interrupt your great keybindings discussion, but I'll just
>> point out that we have now converted to lexical-binding a bit over 80%
>> of Emacs's ELisp files.
>
> Things have progressed a bit since.  We're now down to:
>
>       1 lisp/org/
>       1 lisp/progmodes/
>
>     2 / 2019 = 0.099058940069341253%

Super! Thanks Stefan(s) for the effort!

  Andrea

Re: News from the lexical world

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 772 bytes --]

Andrea Corallo via "Emacs development discussions."
<emacs-devel@gnu.org> writes:


Between native compilation and lexical scoping updates, emacs 28 is
looking to be a truly momentous released!
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> Stefan Monnier [2021-02-12 11:23:13] wrote:
>>> Sorry to interrupt your great keybindings discussion, but I'll just
>>> point out that we have now converted to lexical-binding a bit over 80%
>>> of Emacs's ELisp files.
>>
>> Things have progressed a bit since.  We're now down to:
>>
>>       1 lisp/org/
>>       1 lisp/progmodes/
>>
>>     2 / 2019 = 0.099058940069341253%
>
> Super! Thanks Stefan(s) for the effort!
>
>   Andrea
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: News from the lexical world

[Stefan Monnier]

> Between native compilation and lexical scoping updates, emacs 28 is
> looking to be a truly momentous released!

Although the fact that bundled ELisp files use lexical-binding should
have no visible impact on Emacs users (not even tho who develop
packages).


        Stefan

Re: News from the lexical world

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 498 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:


Agreed re end-user visibility, but it's still a watershed moment for
emacs.
>> Between native compilation and lexical scoping updates, emacs 28 is
>> looking to be a truly momentous released!
>
> Although the fact that bundled ELisp files use lexical-binding should
> have no visible impact on Emacs users (not even tho who develop
> packages).
>
>
>         Stefan
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: News from the lexical world

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 11 Apr 2021 10:39:03 -0400
> Cc: Andrea Corallo <akrl@sdf.org>,
>  "Andrea Corallo via Emacs development discussions." <emacs-devel@gnu.org>
> 
> > Between native compilation and lexical scoping updates, emacs 28 is
> > looking to be a truly momentous released!
> 
> Although the fact that bundled ELisp files use lexical-binding should
> have no visible impact on Emacs users

Nor native compilation, for that matter: it doesn't add functionality,
"only" speeds Emacs up.

Re: News from the lexical world

[Stefan Monnier]

>> > Between native compilation and lexical scoping updates, emacs 28 is
>> > looking to be a truly momentous released!
>> Although the fact that bundled ELisp files use lexical-binding should
>> have no visible impact on Emacs users
> Nor native compilation, for that matter: it doesn't add functionality,
> "only" speeds Emacs up.

Indeed (tho it should be somewhat visible because of the extra files
and the occasional difference in performance: use of lexical-binding
can't even claim such minor impacts).


        Stefan

Re: News from the lexical world

[Andrea Corallo via Emacs development discussions.]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> > Between native compilation and lexical scoping updates, emacs 28 is
>>> > looking to be a truly momentous released!
>>> Although the fact that bundled ELisp files use lexical-binding should
>>> have no visible impact on Emacs users
>> Nor native compilation, for that matter: it doesn't add functionality,
>> "only" speeds Emacs up.
>
> Indeed (tho it should be somewhat visible because of the extra files
> and the occasional difference in performance: use of lexical-binding
> can't even claim such minor impacts).

But we can say that the native compiler works better on lexical code as
on dynamic one just gives-up on any Lisp optimization :)

  Andrea

Re: News from the lexical world

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, raman@google.com, emacs-devel@gnu.org
> Date: Mon, 12 Apr 2021 08:34:37 +0000
> 
> But we can say that the native compiler works better on lexical code as
> on dynamic one just gives-up on any Lisp optimization :)

Yes, this should definitely be documented.

Re: News from the lexical world

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>, raman@google.com, emacs-devel@gnu.org
>> Date: Mon, 12 Apr 2021 08:34:37 +0000
>> 
>> But we can say that the native compiler works better on lexical code as
>> on dynamic one just gives-up on any Lisp optimization :)
>
> Yes, this should definitely be documented.

Where do you think would be the best place to document it?

Thanks

  Andrea

Re: News from the lexical world

[Stefan Kangas]

Andrea Corallo via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

>>> But we can say that the native compiler works better on lexical code as
>>> on dynamic one just gives-up on any Lisp optimization :)
>>
>> Yes, this should definitely be documented.
>
> Where do you think would be the best place to document it?

This provides a real incentive to convert code to lexical-binding.

My vote would be for the ELisp manual, to give it as high visibility as
possible.  Even better if we could also mention it in NEWS.

Re: News from the lexical world

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: monnier@iro.umontreal.ca, raman@google.com, emacs-devel@gnu.org
> Date: Mon, 12 Apr 2021 12:58:04 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andrea Corallo <akrl@sdf.org>
> >> Cc: Eli Zaretskii <eliz@gnu.org>, raman@google.com, emacs-devel@gnu.org
> >> Date: Mon, 12 Apr 2021 08:34:37 +0000
> >> 
> >> But we can say that the native compiler works better on lexical code as
> >> on dynamic one just gives-up on any Lisp optimization :)
> >
> > Yes, this should definitely be documented.
> 
> Where do you think would be the best place to document it?

Where we describe lexical-binding in the ELisp manual, I think.

Re: News from the lexical world

[Andrew Cohen]

Haven't had time to update emacs in quite some time. Updated today and
ran into a problem with supercite.el:

When trying to reply to an article and including the original I get an error

run-hooks: Symbol’s value as variable is void: attribution

which arises in 'sc-select-attribution. I haven't had a chance to find
the actual bug, but removing lexical binding fixes the problem.

-- 
Andrew Cohen

Re: News from the lexical world

[Stefan Kangas]

Andrew Cohen <acohen@ust.hk> writes:

> Haven't had time to update emacs in quite some time. Updated today and
> ran into a problem with supercite.el:
>
> When trying to reply to an article and including the original I get an error
>
> run-hooks: Symbol’s value as variable is void: attribution
>
> which arises in 'sc-select-attribution. I haven't had a chance to find
> the actual bug, but removing lexical binding fixes the problem.

Could you run `M-x toggle-debug-on-error', redo the steps that led to
this error and post the resulting backtrace?

Re: News from the lexical world

[Andrew Cohen]

I have found the problem (but haven't had the time to fix it :()

In supercite  'sc-select-attribution the variables 'citation and
'attribution are bound and then the hook 'sc-attribs-postselect-hook  is
run. I use bbdb which adds to  that hook an anonymous defun:

(add-hook 'sc-attribs-postselect-hook
            (lambda ()
              (setq bbdb-sc-last-attrib
                    (if sc-downcase-p
                        (downcase attribution)
                      attribution))))

and with lexical binding in supercite this of course fails.




-- 
Andrew Cohen

Re: News from the lexical world

[Stefan Monnier]

> I have found the problem (but haven't had the time to fix it :()
>
> In supercite  'sc-select-attribution the variables 'citation and
> 'attribution are bound and then the hook 'sc-attribs-postselect-hook  is
> run. I use bbdb which adds to  that hook an anonymous defun:
>
> (add-hook 'sc-attribs-postselect-hook
>             (lambda ()
>               (setq bbdb-sc-last-attrib
>                     (if sc-downcase-p
>                         (downcase attribution)
>                       attribution))))

Hmm... I can't find this code in the BBDB copy I have (from elpa.git).
Which version of BBDB do you have?


        Stefan

Re: News from the lexical world

[Andrew Cohen]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

[...]

    >> (add-hook 'sc-attribs-postselect-hook (lambda () (setq
    >> bbdb-sc-last-attrib (if sc-downcase-p (downcase attribution)
    >> attribution))))

    SM> Hmm... I can't find this code in the BBDB copy I have (from
    SM> elpa.git).  Which version of BBDB do you have?

I have the git version from
https://git.savannah.nongnu.org/cgit/bbdb.git/
The code is in bbdb-sc.el 

;; Insert our hooks

;; Dammit, supercite!  It runs `sc-attribs-postselect-hook' in an
;; environment with the local variable `attribution' that we rely on.
(with-no-warnings (defvar attribution))

;;;###autoload
(defun bbdb-insinuate-sc ()
  "Hook BBDB into Supercite.
Do not call this in your init file.  Use `bbdb-initialize'.
However, this is not the full story.  See bbdb-sc.el for how to fully hook
BBDB into Supercite."
  (add-hook 'sc-post-hook 'bbdb-sc-set-attrib)
  (add-hook 'sc-attribs-postselect-hook
            (lambda ()
              (setq bbdb-sc-last-attrib
                    (if sc-downcase-p
                        (downcase attribution)
                      attribution)))))

But I guess this is a more generic problem? Any function added to
sc-attribs-postselect-hook will need access to 'attribution and/or
'citation which are not in scope.

Best,
Andy

-- 
Andrew Cohen

Re: News from the lexical world

[Stefan Monnier]

> But I guess this is a more generic problem? Any function added to
> sc-attribs-postselect-hook will need access to 'attribution and/or
> 'citation which are not in scope.

Indeed, I see now that it's documented in the docstring of
`sc-select-attribution`.

I installed the patch below which should fix it.  Please confirm.


        Stefan


diff --git a/lisp/mail/supercite.el b/lisp/mail/supercite.el
index dc1c641052b..d545b0c3f15 100644
--- a/lisp/mail/supercite.el
+++ b/lisp/mail/supercite.el
@@ -1128,6 +1128,8 @@ sc-select-attribution
 auto-selected citation string and the variable `attribution' is bound
 to the auto-selected attribution string."
   (run-hooks 'sc-attribs-preselect-hook)
+  (with-suppressed-warnings ((lexical citation attribution))
+    (defvar citation) (defvar attribution))
   (let ((query-p sc-confirm-always-p)
 	attribution citation
 	(attriblist sc-preferred-attribution-list))

Re: News from the lexical world

[Andrew Cohen]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

[...]

    SM> I installed the patch below which should fix it.  Please
    SM> confirm.

Yes, that patch fixes it.

Best,
Andy


-- 
Andrew Cohen