Re: lexbind ready for merge

[Daniel Colascione <dan.colascione@gmail.com>]

On 3/29/2011 6:22 PM, Stefan Monnier wrote:
>> - apply-partially should have a compiler-macro now that we can implement it
>> very efficiently; also, funcall-partially.
>
> The current definition of apply-partially is not too
> inefficient, actually.  I don't know what funcall-partially would
> be like.  And I don't like either of them: apply-partially is just a way
> to make it easy to build closures by hand without resorting to
> lexical-let and independently from lexical-binding, but once you use
> lexical-binding you're usually better off using a plain closure.

apply-partially at least involves less typing lexical-let, which (as I 
explain below) is required in some cases even with lexbind.

As for funcall-partially: I never really liked how apply-partially 
differs from apply.  The latter treats its last argument specially, 
while the former does not.  If I had my way, I'd rename the current 
apply-partially to funcall-partially (or partial-funcall?) and create a 
new apply-partially that unpacks its last argument.  But it probably 
can't be changed now...

>> - It might be a good idea to remove the "Once Emacs 19 becomes standard..."
>> comment from cl.el
>
> Feel free to do that on the trunk, I don't think it's really related
> to lexbind.

If I could, I would. :-)

>> - Can lexical-let and lexical-let* be made a no-op when compiling lexbound
>> code?  Looking at cl.el, it appears they're still up their usual
>> dirty tricks.
>
> They can *almost* be turned into `let' and `let*', except that
> (lexical-let ((buffer-file-name 3)) ...) will bind buffer-file-name
> lexically whereas `let' will always bind it dynamically.  We could
> either ignore those issues or try to handle them, but I'd rather just
> mark lexical-let obsolete.

I'd prefer to ignore the issues for now and transform lexical-let to let 
when lexical-binding is on, generating a compiler error if we're trying 
to lexically bind a special variable.  I don't think many people try to 
do that.  A macro that uses a lexical binding in its generated code 
still needs to use lexical-let in order for its generated form to work 
properly in either environment, and even outside macros, making 
lexical-let cheap in the lexbound case gives us a way to create 
backward-compatible code that automatically becomes more efficient in 
Emacs 24.

> (Of course, there's also the difficulty for the macro to reliably
> determine whether the expansion will be run in lexical-binding or
> dynamic-binding mode).

Wouldn't inspecting the value of lexical-binding work well enough?  For 
evaluated code, the macro (regardless of how or whether it was compiled) 
will be called directly from the evaluator with the appropriate value of 
lexical-binding set.  For compiled code, the macro will be called by the 
compiler with lexical-binding set to the value the compiler is using to 
compile the resulting form.

The value of lexical-let at macro-run time and result-evaluation time 
*could* differ if the resulting form is squirreled away somewhere and 
reused later --- but I can't think of a case where that happens without 
the code being wrapped in some kind of function object that would 
capture the current value of lexical-let.

Of course, you can lie:

(defmacro* with-broken-code (&body body)
   (let (lexical-binding)
     (macroexpand-all `(progn ,@body)))

That kind of thing is firmly in "It hurts when I do this" territory though.

>
>> - lexical-binding only applies to code evaluated by `eval-buffer' and
>> eval-region'?! So I can't make code evaluated by M-: lexbound?
>
> ?!?  AFAIK, M-: uses lexical or dynamic scoping according to the value
> of lexical-binding in the current buffer.

It does, but the documentation string still gives me the impression that 
it wouldn't be.  Why isn't lexical-binding respected for all evaluation?

>> - It'd be nice to be able to write small pieces of lexical code in
>> non-lexbound code, e.g., in the expansion of a macro that gets used by both
>> lexbound and non-lexbound.
>
> Yes, it'd be nice.
>
>> What's the best way to do that?
>
> Currently, I think the best way to do that is to add the feature to the
> byte-compiler.  The most promising avenue for it might be to use code
> of the form ((closure ()<formalargs>  <lexbindcode>)<actualargs>) and
> compile efficiently (I think currently such code will either result in
> a complaint about a malformed function, or else will leave the function
> uncompiled).

Ah, I see what you mean.  Would a with-lexical-scope form suffice?

;; with-lexical-scope is itself compiled with lexical-binding t
(defmacro* with-lexical-scope (&body body)
   (let* ((vars (remove-if (lambda (var)
                             (or (special-variable-p var)
                                 (not (boundp var))))
                           (find-free-vars `(progn ,@body))))
          (closure `(closure (t) ; no environment
                   ,@args ,@body)))
     `(funcall ,closure ,@vars)))

(defvar b 42)
(defun some-dynamic-function (a)
   (with-lexical-scope
     (lambda (c)
       (+ a b c)))

;; or even

(with-lexical-scope
   (defun foo (arg) ...))

That's not the same as (defun foo (arg) (with-lexical-scope ...))) 
because the argument binding strategies differ.

>> - The documentation claims that defun doesn't capture its lexical scope. In
>> interpreted code, it does.
>
>> (require 'cl)
>> (let ((bar2 5))
>>    (defun foo ()
>>      (incf bar2)
>>      (message "hi: %s" bar2)))
>
>> In compiled code, we do not capture the variable and instead warn about it.
>> Instead, we should capture the variable.
>
> Yup.
>
>> Common Lisp explicitly allows this use, and it's convenient in
>> some cases.
>
> If you need it you can use
>
>    (let ((bar2 5))
>      (defalias 'foo (lambda ()
>        "Foo."
>        (incf bar2)
>        (message "hi: %s" bar2))))
>
> IIRC, the reason why defun doesn't work for it is fundamentally linked
> to some silly technicality, but I justify it for myself by the fact that
> all the "defun within a non-empty context" I've seen were bogus, so I'm
> not strongly motivated to handle it right.

Even if it's not particularly common, being consistent with Common Lisp 
and having fewer special cases are good things.  Some people use 
constructs like this to create module-private variables (which is a bad 
idea, but that doesn't stop people doing it.)

>> - Disassembling a closure reports closed-over variables as constants;
>> they're not.
>
> They are.

Err, yes.  They are.

>> - Do we really use a whole cons cell for each closed-over variable, even in
>> compiled code?
>
> Yes, tho only for variables which are mutated (yup, `setq' is costly).

Couldn't we create a box type, which would would be like a cons except 
that it'd hold just one value?  (The #<foo> read-syntax is available.)