Is add-to-list supposed to work when lexical-binding is t?

Is add-to-list supposed to work when lexical-binding is t?

[Kelly Dean]

Section 11.9.3 (Lexical Binding) in the manual says "functions like `symbol-value', `boundp', and `set' only retrieve or modify a variable's dynamic binding". Why? It causes chaos:
(setq lexical-binding nil)
(let ((x '(a))) (add-to-list 'x 'b) x) -> (b a)
(setq lexical-binding t)
(let ((x '(a))) (add-to-list 'x 'b) x) -> Lisp error: (void-variable x)

(setq lexical-binding nil)
(defun foo (var) (set var 'b))
(let ((x 'a)) (foo 'x) x) -> b
(setq lexical-binding t)
(let ((x 'a)) (foo 'x) x) -> a (also does global (set 'x 'b))

Neither foo nor the let form references any free variables. Ironic that (let ((x 'a)) (foo 'x)) without lexical binding doesn't (permanently) set global x, but with it does. Lexical binding is supposed to reduce such accidents, not cause them.

I get the same result in SBCL, so it isn't just an elisp problem. C can do it:
char x='z'; 
void foo (char* var) {*var='b';}
int main () { {
      char x='a'; foo(&x);
      printf ("Local x: %c, ", x);}
    printf ("Global x: %c\n", x);} -> "Local x: b, Global x: z"

Considering that (intern "x") returns global x, and locals are not eq to globals, it appears (quote x) also returns global x, not lexical x:
(setq lexical-binding t)
(let ((x 'a)) (eq 'x (intern "x"))) -> t
which means the problem of set and symbol-value only accessing global variables is moot, since you can't even pass lexical variables in the first place. (foo 'x) can't work with lexical x, for any foo. In Lisp, it appears you can't even say "&x".

I could avoid the problem this way:
(require 'cl)
(defmacro add-to-list (list-var element &optional append compare-fn)
  "Like function add-to-list in subr.el, but works even when `lexical-binding' is t."
  (let ((list (if (eq (car-safe list-var) 'quote)
		  (cadr list-var)
		`(symbol-value ,list-var))))
    `(if (member* ,element ,list :test (or ,compare-fn 'equal))
	 ,list
       (setf ,list
	     (if ,append
		 (append ,list (list ,element))
	       (cons ,element ,list))))))
which is gross. Or just take a list directly, instead of a list-var, but that's incompatible. Either way, converting every argument-mutating function into a macro obviously isn't the right thing to do. What is the right thing?

Maybe this:
(require 'cl)
(setq lexical-binding t)
(defmacro wrap-lexical (x)
  `(list (lambda () ,x)
	 (lambda (val) (setq ,x val))))
(defun get-passed-lexical (x)
  (funcall (car x)))
(defun set-passed-lexical (x val)
 (funcall (cadr x) val))
(defun add-to-list-lexable (wrapped-list element &optional append compare-fn)
  (let ((list (get-passed-lexical wrapped-list)))
    (if (member* element list :test (or compare-fn 'equal))
	 list
       (set-passed-lexical wrapped-list
	     (if append
		 (append list (list element))
	       (cons element list))))))
(let ((x '(a))) (add-to-list-lexable (wrap-lexical x) 'b) x) -> (b a)
but that's both gross and incompatible.

Re: Is add-to-list supposed to work when lexical-binding is t?

[Stefan Monnier]

> Section 11.9.3 (Lexical Binding) in the manual says "functions like
> `symbol-value', `boundp', and `set' only retrieve or modify
> a variable's dynamic binding".  Why?

Because a variable is not the same thing as a symbol.

For dynamic binding, you can somewhat blur the difference and use the
symbol's value cell as "the content of the variable of that name"
because let-binding just temporarily changes the only global value.

With lexical scoping, a given variable name can have many different
values at the same time so this is not an option.

> (let ((x '(a))) (add-to-list 'x 'b) x) -> (b a)

Yup, this is asking for trouble.  Use `push' or `cl-pushnew' instead.


        Stefan

Re: Is add-to-list supposed to work when lexical-binding is t?

[PJ Weisberg]

On Mon, Jun 3, 2013 at 6:49 PM, Stefan Monnier <monnier@iro.umontreal.ca>wrote:

>
> > (let ((x '(a))) (add-to-list 'x 'b) x) -> (b a)
>
> Yup, this is asking for trouble.  Use `push' or `cl-pushnew' instead.
>
>
Unless you're knee-deep in the internals of the language, the difference
between `cl-pushnew' and `add-to-list' is so obscure as to be
incomprehensible.

-PJ

Gehm's Corollary to Clark's Law: Any technology distinguishable from
magic is insufficiently advanced.

Re: Is add-to-list supposed to work when lexical-binding is t?

[Stefan Monnier]

>> > (let ((x '(a))) (add-to-list 'x 'b) x) -> (b a)
>> Yup, this is asking for trouble.  Use `push' or `cl-pushnew' instead.
> Unless you're knee-deep in the internals of the language, the difference
> between `cl-pushnew' and `add-to-list' is so obscure as to be
> incomprehensible.

Agreed.  I never liked add-to-list, and now I really know why.

To give you an idea of why `add-to-list' is a problem:

   (add-to-list 'o 'b)

does the same as

   (add-to-list (intern (substring "ouch" 0 1)) 'b)

so other than by ad-hoc handling the case where the first arg is passed
via quoting, the byte-compiler would have to handle both cases, and the
second case is pretty hopeless.

   
        Stefan

Re: Is add-to-list supposed to work when lexical-binding is t?

[Kelly Dean]

Stefan Monnier wrote:
>> Section 11.9.3 (Lexical Binding) in the manual says "functions like
>> `symbol-value', `boundp', and `set' only retrieve or modify
>> a variable's dynamic binding".  Why?
>
>Because a variable is not the same thing as a symbol.
>
>For dynamic binding, you can somewhat blur the difference and use the
>symbol's value cell as "the content of the variable of that name"
>because let-binding just temporarily changes the only global value.
>
>With lexical scoping, a given variable name can have many different
>values at the same time so this is not an option.

I read the last bit as, "with lexical scoping, a given symbol (symbolic name) can be interpreted as many different variables in different scopes (and each variable can have multiple instances) ...", to synchronize terminology. And I call the things that variables are bound to "instances", not "bindings" like the docs do.

In the scope of (let ((x 'a)) ...) with lexical binding enabled, since the symbol x is interpreted as a lexical variable rather than as the global variable (the latter bound to the global instance, the value cell for the symbol x), I propose a quote-lex special form, where (quote-lex x), with e.g. "&x" or maybe "^x" as readable syntax, returns a reference to the current instance (current in time and recursion level) of the lexical variable, like "&x" does in C, instead of returning the symbol x (which, without context, is a reference to the global instance). Also change the set function to accept not only symbols, but also lexical instance references, and change symbol-value to either return the value of a symbol's value cell as usual if given the symbol, or return the value of a lexical i
 nstance if given a reference to the latter.

Then you could use argument-mutating functions, including the standard add-to-list function, without having to convert them into macros, and do e.g.:
(let ((x '(a))) (add-to-list &x 'b) x) -> (b a)

The pair of closures produced by wrap-lexical in my previous message (but the first closure should more simply be just the plain value; oops) imitates what I propose; a special form would be needed to actually return a reference to the lexical instance itself. My set-passed-lexical and get-passed-lexical imitate what I propose for set and symbol-value.

Furthermore, instead of having a separate quote-lex, maybe overload quote, and return the symbol as usual where it's used as the global variable, or return a reference to the current lexical instance where the symbol is used as a lexical variable. Then even current code that uses argument-mutating functions could be used unmodified, instead of having to convert their relevant uses of quote to quote-lex. And lexical binding won't cause the accidents I described in my previous message.

If overloading quote is a bad idea, then quote-lex should return a reference to the current instance of the given variable regardless of whether it's given a lexical or global variable; that means in the latter case, it does the same thing as quote, which means quote-lex can be used instead of quote in all places, except where a symbol needs to be returned despite that symbol also serving as a lexical variable in the same scope, because the symbol will be used as the global variable or as something other than a variable.

>> (let ((x '(a))) (add-to-list 'x 'b) x) -> (b a)
>
>Yup, this is asking for trouble.  Use `push' or `cl-pushnew' instead.

So, converting every argument-mutating function into a macro actually _is_ the right thing to do? That seems unnecessarily complicated, just a way of working around the lack of lexical quoting.

Re: Is add-to-list supposed to work when lexical-binding is t?

[Stefan Monnier]

>>> Section 11.9.3 (Lexical Binding) in the manual says "functions like
>>> `symbol-value', `boundp', and `set' only retrieve or modify
>>> a variable's dynamic binding".  Why?

Actually, the answer is much simpler: all the above are *functions* (as
opposed to macros or special forms), so they receive values as arguments
and know very little from their caller other than the values it passed
as arguments.

In order to do what you'd want them to do, they'd have to look at the
value of variables in the scope of their caller.  This is exactly what
dynamic scoping provides, and is what lexical scoping prevents.

>> With lexical scoping, a given variable name can have many different
>> values at the same time so this is not an option.

> I read the last bit as, "with lexical scoping, a given symbol (symbolic
> name) can be interpreted as many different variables in different scopes
> (and each variable can have multiple instances) ...", to synchronize
> terminology. And I call the things that variables are bound to "instances",
> not "bindings" like the docs do.

*at the same time* is the important part.  With dynamic scoping, only
one of the various scopes exists at a given time.  But with lexical
scoping it's a bit more tricky:

   (let ((funs (mapcar (lambda (x) (lambda (y) (+ x y)) '(1 2 3))))
     (mapcar (lambda (f) (funcall f 3)) funs))

this should return (4 5 6).  Between the two mapcars, we have stored in
`funs' 3 functions (more specifically closures), each one of them of the
kind (lambda (y) (+ x y)).  They each have their own copy of `x', so all
three copies of `x' exist at the same time in `funs'.
     
> In the scope of (let ((x 'a)) ...) with lexical binding enabled, since the
> symbol x is interpreted as a lexical variable rather than as the global
> variable (the latter bound to the global instance, the value cell for the
> symbol x), I propose a quote-lex special form, where (quote-lex x), with
> e.g. "&x" or maybe "^x" as readable syntax, returns a reference to the
> current instance

You can definitely do that.  In the current Emacs sources it's provided
under the name `gv-ref', which indeed works like C's &.

> Then you could use argument-mutating functions, including the standard
> add-to-list function, without having to convert them into macros, and do
> e.g.:
> (let ((x '(a))) (add-to-list &x 'b) x) -> (b a)

We can definitely make add-to-list work for

   (let ((x '(a))) (add-to-list (gv-ref x) 'b) x)   ===>   (b a)

That's easy and would work fine.  But (gv-ref x) is not the same as 'x
and trying to magically turn one into the other, while feasible in a few
particular cases is simply impossible in general.

>>> (let ((x '(a))) (add-to-list 'x 'b) x) -> (b a)
>> Yup, this is asking for trouble.  Use `push' or `cl-pushnew' instead.
> So, converting every argument-mutating function into a macro actually _is_
> the right thing to do?

Yes, pretty much.

> That seems unnecessarily complicated, just a way of working around the
> lack of lexical quoting.

There is no "lack of lexical quoting", as seen above.

Argument-mutating functions are relatively rare and using macros tends
to be a lot more efficient than using gv-ref, so in most cases using
macros makes more sense, despite their well known disadvantages.


        Stefan

Re: Is add-to-list supposed to work when lexical-binding is t?

[Kelly Dean]

Stefan Monnier wrote:
>In order to do what you'd want them to do, they'd have to look at the
>value of variables in the scope of their caller.  This is exactly what
>dynamic scoping provides, and is what lexical scoping prevents.

But unconditionally preventing it with lexical scoping is the problem. If lexical quoting weren't useful, then why would C and your new gv-ref/deref enable it?

>   (let ((funs (mapcar (lambda (x) (lambda (y) (+ x y)) '(1 2 3))))
>     (mapcar (lambda (f) (funcall f 3)) funs))
>
>this should return (4 5 6).  Between the two mapcars, we have stored in
>`funs' 3 functions (more specifically closures), each one of them of the
>kind (lambda (y) (+ x y)).  They each have their own copy of `x', so all
>three copies of `x' exist at the same time in `funs'.

I was sloppy in my description. When eval sees the symbol x, and is going to interpret it as a lexical variable, as it does in your example, it looks up, in the current lexical environment, the address p_x of the memory cell that's the current instance of x, then reads from the cell at p_x the address of the object that's the current Lisp-level value of x, or writes some different object address into the cell at p_x if you do (setq x ...). So of course, at this point, eval knows what p_x is.

If you do
(defun foo-incf (x y) (setf (gv-deref x) (+ (gv-deref x) y)))
(let ((funs (mapcar (lambda (x)
		      (lambda (y) (foo-incf (gv-ref x) y) (* x 2)))
		    '(1 2 3))))
  (mapcar (lambda (f) (funcall f 3)) funs)) -> (8 10 12)
then in each of the 3 closures in the list funs, there's a call to foo-incf, with the first argument being a cons cell (with a pair of closures) returned by the expanded form of (gv-ref x).
What I'm proposing is that, instead of using (gv-ref x), have the special form (quote-lex x) return p_x, and use this as the first argument to foo-incf. Each of the 3 times the function (lambda (y) (foo-incf (quote-lex x) y) (* x 2)) is called, there will be a different current environment with its own instance of x, but eval knows which environment is current, so it does know what the correct p_x is, so it knows what to evaluate (quote-lex x) to.
p_x is then passed to foo-incf, which can use symbol-value (modified to accept p_x, i.e. a lexical instance reference) instead of using gv-deref. So foo-incf, add-to-list, etc will work regardless of whether lexical binding is enabled.

>We can definitely make add-to-list work for
>
>   (let ((x '(a))) (add-to-list (gv-ref x) 'b) x)   ===>   (b a)
>
>That's easy and would work fine.  But (gv-ref x) is not the same as 'x

Indeed not the same: I get "Lisp error: (wrong-type-argument symbolp ((closure ..." because gv-ref returns a pair of closures that just imitate a lexical instance reference, so to make it work, I have to modify add-to-list and replace all the calls to symbol-value by calls to gv-deref, and change "set list-var" to "setf (gv-deref list-var)"; this solution is essentially the same as the add-to-list-lexable solution using wrap-/get-/set-passed-lexical in my original message, so it has the same problems: it's incompatible with current use of add-to-list (857 occurrences in Emacs 24.3 el and texi files) and any other functions that use set and symbol-value on an argument, it requires lexical binding, and it's inefficient, with the inefficiency causing macros instead of functions to be necessar
 y as a workaround. Even if you modify set and symbol-value to accept the output of gv-ref, they can't catch type errors; they can distinguish a symbol from a non-symbol
 but can't distinguish the output of gv-ref from other structures of the same form. Real lexical quoting would solve all those problems.

(eq (quote x) (quote x)) -> t (equal addresses)
(equal (gv-ref x) (gv-ref x)) -> t (equal pairs of closures)
(eq (gv-ref x) (gv-ref x)) -> nil (but separate pairs of closures)
In contrast, "&x == &x" is true in C, even for lexical x.
C gives you true equality. Lisp only gives you separate-but-equal. My proposal is:
(eq (quote-lex x) (quote-lex x)) -> t (equal addresses, for either global or lexical x)
quote-lex wouldn't have to make a cons cell and return a pair of closures, and dereferencing one pointer is more efficient than the car and funcall that gv-deref has to do.

Here's another perspective: you pointed out in your original reply, "a variable is not the same thing as a symbol." In
(letrec ((mylen (lambda (x) (if x (1+ (funcall mylen (cdr x))) 0)))
	 (x '(a b x)))
  (funcall mylen x))
the symbol x occurs 5 times as 2 different variables (the first bound to 4 different instances at runtime, the second bound to 1) and occurs once not as a variable.
When you do (setq cursor-type 'bar), bar is just a symbol, so it's correct to use (quote bar) to get that symbol. But when you do (setq indent-line-function 'my-indent-func), my-indent-func is a symbol that's interpreted by funcall in indent-according-to-mode and indent-for-tab-command as a variable (ignoring the Lisp-2 issue), so using (quote my-indent-func), which just returns a symbol, is a pretense that a symbol _is_ the same thing as a variable. That's a type error, but Lisp can't catch it. Since you want the global variable, and there's only one that the symbol can be interpreted as, the pretense isn't a problem. But when you do
(let ((x '(a))) (add-to-list 'x 'b) x)
it fails because the first parameter to add-to-list is a variable, but you pass just the symbol x, and this time the type error bites you, since add-to-list interprets the symbol x as the wrong variable. So the problem isn't really lack of _lexical_ variable quoting; the problem is that in Lisp, you can't quote variables at all. You can only quote symbols, and pretend that symbols and variables are the same thing. So quote-lex would more appropriately be called "quote-var". Sorry for the name change; obviously I didn't think this through enough.
A lexical instance reference is in effect a symbol plus a reference to the environment in which to interpret that symbol as a variable, so modifying symbol-value to accept such references just makes it stop assuming that you always mean the global environment.

For completeness, either (setq indent-line-function 'my-indent-func) should signal a type error, or "'x" should be read as (quote-var x), and (setq cursor-type 'bar) should signal a type error (so use (setq cursor-type :bar) instead), but this would break everything. But just adding quote-var, and continuing to allow symbol<->variable conflation for global variables, wouldn't break anything.

>and trying to magically turn one into the other, while feasible in a few
>particular cases is simply impossible in general.

I don't see cases where quote-var would fail to do what's intended.

>Argument-mutating functions are relatively rare and using macros tends
>to be a lot more efficient than using gv-ref, so in most cases using
>macros makes more sense, despite their well known disadvantages.

If they're rare then I guess my proposal is pointless. But for the ones that do exist, I think quote-var would be suitable.

BTW sorry about my mis-threaded replies, but I read from lists.gnu.org/archive/html/ (since I don't subscribe to lists) and it omits Message-ID: headers, so I have nothing to put in a References: header.

Re: Is add-to-list supposed to work when lexical-binding is t?

[Stefan Monnier]

> I don't see cases where quote-var would fail to do what's intended.

We can have gv-ref/quote-var/quote-lex/younameit, but what we can't do
is merge it with quote, because their semantics are incompatible:

        (eq (let ((x 1)) (quote-var x)) (let ((x 1)) (quote-var x)))

should return nil, whereas

        (eq (let ((x 1)) (quote x)) (let ((x 1)) (quote x)))

should return t.


        Stefan

Re: Is add-to-list supposed to work when lexical-binding is t?

[Barry Margolin]

In article <mailman.1311.1370828650.22516.help-gnu-emacs@gnu.org>,
 Kelly Dean <kellydeanch@yahoo.com> wrote:

> When eval sees the symbol x, and is going to interpret it as a lexical 
> variable, as it does in your example, it looks up, in the current lexical 
> environment

Since eval is just an ordinary function, it doesn't have access to the 
lexical environment.

-- 
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Re: Is add-to-list supposed to work when lexical-binding is t?

[Kelly Dean]

Stefan Monnier wrote:
>> I don't see cases where quote-var would fail to do what's intended.
>We can have gv-ref/quote-var/quote-lex/younameit, but what we can't do
>is merge it with quote, because their semantics are incompatible:
>        (eq (let ((x 1)) (quote-var x)) (let ((x 1)) (quote-var x)))
>should return nil, whereas
>        (eq (let ((x 1)) (quote x)) (let ((x 1)) (quote x)))
>should return t.

True, and that's a good way to explain the difference between a variable and a symbol. On June 5 I wrote, "If overloading quote is a bad idea, then quote-lex should return a reference to the current instance of the given variable regardless of whether it's given a lexical or global variable", which means
(defvar x 1)
(eq (quote-var x) (quote-var x))
should return t. But
(eq (gv-ref x) (gv-ref x))
returns nil, so gv-ref isn't a suitable substitute for quote-var.

I further wrote, "that means in the latter case, it does the same thing as quote", which means I was confusing variables with symbols. :-)

And further, "which means quote-lex can be used instead of quote in all places, except where a symbol needs to be returned despite that symbol also serving as a lexical variable in the same scope, because the symbol will be used as the global variable or as something other than a variable."
Which now I think should be: in order to avoid a type error, quote-var _must_ be used, unless the symbol will be used as something other than a variable. But then there are a couple problems: 1, Lisp can't detect the type error anyway, so what's the point? And 2, if you actually do need to quote a global variable that's shadowed by a lexical variable, there's no way to do it except (quote global-var), yet that's a type error. I suppose the answer to the second problem is: don't shadow the global; use a different name for the lexical.

To solve the first problem, I think set and symbol-value would have to signal an error if given a non-variable symbol, and eq would have to distinguish variables and symbols:
(setq x 1)
(symbol-value (quote-var x)) -> 1
(symbol-value (quote x)) -> type error
(eq (quote x) (quote-var x)) -> type error
which also means symbol-value would more appropriately be called "variable-value", which seems reasonable anyway, since non-variable (well, and non-constant) symbols don't have values.
But it's just an academic argument, since signalling such errors would break everything.

In my previous message I wrote:
>For completeness, either (setq indent-line-function 'my-indent-func) should 
>signal a type error, or "'x" should be read as (quote-var x), and (setq 
>cursor-type 'bar) should signal a type error

But the type error can't be detected until indent-line-function or cursor-type is used. Oops.
This concludes my public conversation with myself for today.

Your example above explains symbol vs. variable. To explain variable vs. instance:
(defun foo (&optional p_x)
  (let ((x))
    (if p_x (eq p_x (quote-var x))
      (foo (quote-var x)))))
(foo) -> nil
which also means maybe "quote-var" isn't a good name either; "younameit" might be just as good!

Barry Margolin wrote:
>> When eval sees the symbol x, and is going to interpret it as a lexical 
>> variable, as it does in your example, it looks up, in the current lexical 
>> environment
>Since eval is just an ordinary function, it doesn't have access to the 
>lexical environment.

The elisp manual, section 11.9.3 Lexical Binding, says:
"When the Lisp evaluator wants the
current value of a variable, it looks first in the lexical environment;
if the variable is not specified in there, it looks in the symbol's
value cell, where the dynamic value is stored."

Did I miss something?

Re: Is add-to-list supposed to work when lexical-binding is t?

[Stefan Monnier]

>>> I don't see cases where quote-var would fail to do what's intended.
>> We can have gv-ref/quote-var/quote-lex/younameit, but what we can't do
>> is merge it with quote, because their semantics are incompatible:
>> (eq (let ((x 1)) (quote-var x)) (let ((x 1)) (quote-var x)))
>> should return nil, whereas
>> (eq (let ((x 1)) (quote x)) (let ((x 1)) (quote x)))
>> should return t.
> True, and that's a good way to explain the difference between a variable
> and a symbol.

And more importantly, it explains why we can't just blindly replace 'x
with (quote-lex x).  So there's no way to automatically and reliably
make things like add-to-list/set/run-hooks/symbol-value/... work.

BTW, I started to use a local hack which compiles all files in
lexical-binding mode (hence bumping into all those problems).
Some problematic cases are even more impossible for the compiler to
auto-handle (see for example in auth-source.el):

  (let* ((base-required '(host user port secret))
    [...]
    (dolist (br base-required)
      (when (symbol-value br)

So now you'd need a quote-lex which doesn't work only to quote symbols
but even lists of symbols (and I wouldn't be surprised to find places
where a vector is used instead, or where the list is itselfs generated).


        Stefan