Re: Is add-to-list supposed to work when lexical-binding is t?

[Kelly Dean <kellydeanch@yahoo.com>]

Stefan Monnier wrote:
>In order to do what you'd want them to do, they'd have to look at the
>value of variables in the scope of their caller.  This is exactly what
>dynamic scoping provides, and is what lexical scoping prevents.

But unconditionally preventing it with lexical scoping is the problem. If lexical quoting weren't useful, then why would C and your new gv-ref/deref enable it?

>   (let ((funs (mapcar (lambda (x) (lambda (y) (+ x y)) '(1 2 3))))
>     (mapcar (lambda (f) (funcall f 3)) funs))
>
>this should return (4 5 6).  Between the two mapcars, we have stored in
>`funs' 3 functions (more specifically closures), each one of them of the
>kind (lambda (y) (+ x y)).  They each have their own copy of `x', so all
>three copies of `x' exist at the same time in `funs'.

I was sloppy in my description. When eval sees the symbol x, and is going to interpret it as a lexical variable, as it does in your example, it looks up, in the current lexical environment, the address p_x of the memory cell that's the current instance of x, then reads from the cell at p_x the address of the object that's the current Lisp-level value of x, or writes some different object address into the cell at p_x if you do (setq x ...). So of course, at this point, eval knows what p_x is.

If you do
(defun foo-incf (x y) (setf (gv-deref x) (+ (gv-deref x) y)))
(let ((funs (mapcar (lambda (x)
		      (lambda (y) (foo-incf (gv-ref x) y) (* x 2)))
		    '(1 2 3))))
  (mapcar (lambda (f) (funcall f 3)) funs)) -> (8 10 12)
then in each of the 3 closures in the list funs, there's a call to foo-incf, with the first argument being a cons cell (with a pair of closures) returned by the expanded form of (gv-ref x).
What I'm proposing is that, instead of using (gv-ref x), have the special form (quote-lex x) return p_x, and use this as the first argument to foo-incf. Each of the 3 times the function (lambda (y) (foo-incf (quote-lex x) y) (* x 2)) is called, there will be a different current environment with its own instance of x, but eval knows which environment is current, so it does know what the correct p_x is, so it knows what to evaluate (quote-lex x) to.
p_x is then passed to foo-incf, which can use symbol-value (modified to accept p_x, i.e. a lexical instance reference) instead of using gv-deref. So foo-incf, add-to-list, etc will work regardless of whether lexical binding is enabled.

>We can definitely make add-to-list work for
>
>   (let ((x '(a))) (add-to-list (gv-ref x) 'b) x)   ===>   (b a)
>
>That's easy and would work fine.  But (gv-ref x) is not the same as 'x

Indeed not the same: I get "Lisp error: (wrong-type-argument symbolp ((closure ..." because gv-ref returns a pair of closures that just imitate a lexical instance reference, so to make it work, I have to modify add-to-list and replace all the calls to symbol-value by calls to gv-deref, and change "set list-var" to "setf (gv-deref list-var)"; this solution is essentially the same as the add-to-list-lexable solution using wrap-/get-/set-passed-lexical in my original message, so it has the same problems: it's incompatible with current use of add-to-list (857 occurrences in Emacs 24.3 el and texi files) and any other functions that use set and symbol-value on an argument, it requires lexical binding, and it's inefficient, with the inefficiency causing macros instead of functions to be necessar
 y as a workaround. Even if you modify set and symbol-value to accept the output of gv-ref, they can't catch type errors; they can distinguish a symbol from a non-symbol
 but can't distinguish the output of gv-ref from other structures of the same form. Real lexical quoting would solve all those problems.

(eq (quote x) (quote x)) -> t (equal addresses)
(equal (gv-ref x) (gv-ref x)) -> t (equal pairs of closures)
(eq (gv-ref x) (gv-ref x)) -> nil (but separate pairs of closures)
In contrast, "&x == &x" is true in C, even for lexical x.
C gives you true equality. Lisp only gives you separate-but-equal. My proposal is:
(eq (quote-lex x) (quote-lex x)) -> t (equal addresses, for either global or lexical x)
quote-lex wouldn't have to make a cons cell and return a pair of closures, and dereferencing one pointer is more efficient than the car and funcall that gv-deref has to do.

Here's another perspective: you pointed out in your original reply, "a variable is not the same thing as a symbol." In
(letrec ((mylen (lambda (x) (if x (1+ (funcall mylen (cdr x))) 0)))
	 (x '(a b x)))
  (funcall mylen x))
the symbol x occurs 5 times as 2 different variables (the first bound to 4 different instances at runtime, the second bound to 1) and occurs once not as a variable.
When you do (setq cursor-type 'bar), bar is just a symbol, so it's correct to use (quote bar) to get that symbol. But when you do (setq indent-line-function 'my-indent-func), my-indent-func is a symbol that's interpreted by funcall in indent-according-to-mode and indent-for-tab-command as a variable (ignoring the Lisp-2 issue), so using (quote my-indent-func), which just returns a symbol, is a pretense that a symbol _is_ the same thing as a variable. That's a type error, but Lisp can't catch it. Since you want the global variable, and there's only one that the symbol can be interpreted as, the pretense isn't a problem. But when you do
(let ((x '(a))) (add-to-list 'x 'b) x)
it fails because the first parameter to add-to-list is a variable, but you pass just the symbol x, and this time the type error bites you, since add-to-list interprets the symbol x as the wrong variable. So the problem isn't really lack of _lexical_ variable quoting; the problem is that in Lisp, you can't quote variables at all. You can only quote symbols, and pretend that symbols and variables are the same thing. So quote-lex would more appropriately be called "quote-var". Sorry for the name change; obviously I didn't think this through enough.
A lexical instance reference is in effect a symbol plus a reference to the environment in which to interpret that symbol as a variable, so modifying symbol-value to accept such references just makes it stop assuming that you always mean the global environment.

For completeness, either (setq indent-line-function 'my-indent-func) should signal a type error, or "'x" should be read as (quote-var x), and (setq cursor-type 'bar) should signal a type error (so use (setq cursor-type :bar) instead), but this would break everything. But just adding quote-var, and continuing to allow symbol<->variable conflation for global variables, wouldn't break anything.

>and trying to magically turn one into the other, while feasible in a few
>particular cases is simply impossible in general.

I don't see cases where quote-var would fail to do what's intended.

>Argument-mutating functions are relatively rare and using macros tends
>to be a lot more efficient than using gv-ref, so in most cases using
>macros makes more sense, despite their well known disadvantages.

If they're rare then I guess my proposal is pointless. But for the ones that do exist, I think quote-var would be suitable.

BTW sorry about my mis-threaded replies, but I read from lists.gnu.org/archive/html/ (since I don't subscribe to lists) and it omits Message-ID: headers, so I have nothing to put in a References: header.