cl-block less lexical than expected

cl-block less lexical than expected

[Robin Tarsiger]

Greetings.

Today I was poking around with elisp and decided to look at the
definitions of cl-block and cl-return-from. These are predictably
described in the elisp manual as using lexical tags along the lines
of how actual-CL block and return-from work. However, the way the
macros are implemented doesn't line up with this; they do a static
transformation on the tag name and then intern the result for use
with catch/throw, and that yields dynamic semantics:

   (defun tmp/cl-block-test-1 ()
     (cl-block foo
       (cl-return-from foo 'ok)
       'not-reached))

   (defun tmp/cl-block-test-2 ()
     (cl-return-from foo 'bad))

   (defun tmp/cl-block-test-3 ()
     (cl-block foo
       (tmp/cl-block-test-2)
       ;; Necessary to avoid the catch being discarded.
       (cl-return-from foo 'correct)))

   (tmp/cl-block-test-1)
   ==> ok ; This is fine.

   (tmp/cl-block-test-2)
   ==> <signals error> ; Also okay.

   (tmp/cl-block-test-3)
   ==> bad ; But the foo block isn't actually lexical!

I have to think that since catch and throw both take evaluated
expressions for their tags, a more suitable implementation might
do a static transformation to choose an out-of-the-way lexical
_variable_ name which would then be let-bound to a gensym during
the body. cl-return-from could then try to evaluate the variable
to get the tag; the warnings/errors for bad tags would be awkward
to read, but at least the semantics would be correct. An unbound
tag variable would also be warned about during byte-compilation.

Thoughts?

-RTT

Re: cl-block less lexical than expected

[Robin Tarsiger]

Robin Tarsiger wrote:
> cl-return-from could then try to evaluate the variable
> to get the tag; the warnings/errors for bad tags would be awkward
> to read, but at least the semantics would be correct.

Slight correction---they would be correct _provided that_ lexical-binding
were in use for the let. However, I gather having it on almost all the time
is becoming the norm (and maybe the macros could warn if it's off).

-RTT

Re: cl-block less lexical than expected

[Stefan Monnier]

> I have to think that since catch and throw both take evaluated
> expressions for their tags, a more suitable implementation might
> do a static transformation to choose an out-of-the-way lexical
> _variable_ name which would then be let-bound to a gensym during
> the body. cl-return-from could then try to evaluate the variable
> to get the tag;

Sounds good.  As you noted, this was not an option when these macros
were written because of the need for statically scoped variables.

> the warnings/errors for bad tags would be awkward to read, but at
> least the semantics would be correct.

By carefully choosing the names we use that shouldn't be too bad.


        Stefan

Re: cl-block less lexical than expected

[Robin Tarsiger]

I wrote:
 > I have to think that since catch and throw both take evaluated
 > expressions for their tags, a more suitable implementation might
 > do a static transformation to choose an out-of-the-way lexical
 > _variable_ name which would then be let-bound to a gensym during
 > the body. cl-return-from could then try to evaluate the variable
 > to get the tag;

Stefan Monnier wrote:
> Sounds good.  As you noted, this was not an option when these macros
> were written because of the need for statically scoped variables.

So I've been experimenting a little with modifying cl-macs.el to make
this happen. I think I have it about right, but here's what I've found
that's tricky:

   - cl--block-wrapper currently assumes it's receiving exactly a form
     produced by cl--block and invasively unwraps it, so they have to
     be kept in sync. This isn't too bad when it's just a catch form,
     but the new form is much more highly nested.

     I imagine having cl--block-wrapper do most of the expansion itself
     and be implemented as macro+cmacro rather than passthrough+cmacro
     would be cleaner here?

   - The new form is more highly nested not just because of the let to
     keep the tag in a variable, but because of an unwind-protect used
     to clear out the variable when the body is exited. This is so that
     lambdas that cl-return-from the enclosing block function properly
     when called while the block is executing, but can _not_ erroneously
     return from a separate activation of the same block if called after
     the original activation of the block has exited. Instead, they will
     wind up throwing to a dummy tag and signaling a recognizable error
     thereby.

     Naturally, the existing code misbehaves in this scenario as well.

   - I currently have cl-block warn if lexical-binding is nil. I did a
     quick pass over the Lisp files included with Emacs to make sure
     all files with cl-block-alikes used lexical-binding, but I don't
     know what impact this would have on external packages.

   - It would be nice to have cl-return-from warn if it doesn't see the
     name in the list of extant blocks that the cl--block-wrapper cmacro
     already uses to optimize away the catch, but that's not reliable
     enough, right? It doesn't play nicely with macrostep, for instance.
     Is there a better way to do this?

   - Documenting the new behavior is a bit weird due to the potential
     interaction with lexical-binding. I have a few paragraphs in the
     docstrings Texinfo manual indicating that cl-return-from may
     behave erratically if there are any non-lexical-binding cl-blocks
     in effect.

     The existing documentation is not quite correct anyway.

More general questions for the list:

   - Is this a worthwhile thing to fix in Emacs? I would say yes, else
     I wouldn't be trying it, but I don't know if other people agree.

     The complexity increase in documentation from interactions with
     non-lexical-binding code is awkward, especially. How much do we
     need to worry about that? Is the general trend for lexical-binding
     to be strongly recommended enough that not giving details of
     what happens if it's off (other than that the correct use cases
     should still work) is okay?

   - If so, do I sound reasonably on the right track above? Would it be
     helpful to present a draft patch?

   - The reason I have not already presented a draft patch is because in
     futures where the code is integrated into Emacs, I would have to
     perform a copyright assignment. I do not object to this, but I also
     do not use my government-recognized name as my social name in this
     context. I am okay with the FSF keeping a record of my government
     name for legal purposes, but I would rather it not be included in
     publications anyplace too obvious. Is this a request that I can
     expect to be honored? What would be the impact of this on the
     assignment procedure? I am a US citizen.

-RTT

Re: cl-block less lexical than expected

[Eli Zaretskii]

> Date: Tue, 4 Jan 2022 13:29:23 -0600
> From: Robin Tarsiger <rtt@dasyatidae.com>
> 
>    - The reason I have not already presented a draft patch is because in
>      futures where the code is integrated into Emacs, I would have to
>      perform a copyright assignment. I do not object to this, but I also
>      do not use my government-recognized name as my social name in this
>      context. I am okay with the FSF keeping a record of my government
>      name for legal purposes, but I would rather it not be included in
>      publications anyplace too obvious. Is this a request that I can
>      expect to be honored? What would be the impact of this on the
>      assignment procedure? I am a US citizen.

There should be no problems with that, the FSF staff know how to
handle these cases.  Your legal name will not appear in public
records, only the pseudonym you choose.  Just tell the copyright clerk
about this when you email your paperwork, and they will take it from
there.

Thank you for your interest in Emacs.