Is add-to-list supposed to work when lexical-binding is t?

[Kelly Dean <kellydeanch@yahoo.com>]

Section 11.9.3 (Lexical Binding) in the manual says "functions like `symbol-value', `boundp', and `set' only retrieve or modify a variable's dynamic binding". Why? It causes chaos:
(setq lexical-binding nil)
(let ((x '(a))) (add-to-list 'x 'b) x) -> (b a)
(setq lexical-binding t)
(let ((x '(a))) (add-to-list 'x 'b) x) -> Lisp error: (void-variable x)

(setq lexical-binding nil)
(defun foo (var) (set var 'b))
(let ((x 'a)) (foo 'x) x) -> b
(setq lexical-binding t)
(let ((x 'a)) (foo 'x) x) -> a (also does global (set 'x 'b))

Neither foo nor the let form references any free variables. Ironic that (let ((x 'a)) (foo 'x)) without lexical binding doesn't (permanently) set global x, but with it does. Lexical binding is supposed to reduce such accidents, not cause them.

I get the same result in SBCL, so it isn't just an elisp problem. C can do it:
char x='z'; 
void foo (char* var) {*var='b';}
int main () { {
      char x='a'; foo(&x);
      printf ("Local x: %c, ", x);}
    printf ("Global x: %c\n", x);} -> "Local x: b, Global x: z"

Considering that (intern "x") returns global x, and locals are not eq to globals, it appears (quote x) also returns global x, not lexical x:
(setq lexical-binding t)
(let ((x 'a)) (eq 'x (intern "x"))) -> t
which means the problem of set and symbol-value only accessing global variables is moot, since you can't even pass lexical variables in the first place. (foo 'x) can't work with lexical x, for any foo. In Lisp, it appears you can't even say "&x".

I could avoid the problem this way:
(require 'cl)
(defmacro add-to-list (list-var element &optional append compare-fn)
  "Like function add-to-list in subr.el, but works even when `lexical-binding' is t."
  (let ((list (if (eq (car-safe list-var) 'quote)
		  (cadr list-var)
		`(symbol-value ,list-var))))
    `(if (member* ,element ,list :test (or ,compare-fn 'equal))
	 ,list
       (setf ,list
	     (if ,append
		 (append ,list (list ,element))
	       (cons ,element ,list))))))
which is gross. Or just take a list directly, instead of a list-var, but that's incompatible. Either way, converting every argument-mutating function into a macro obviously isn't the right thing to do. What is the right thing?

Maybe this:
(require 'cl)
(setq lexical-binding t)
(defmacro wrap-lexical (x)
  `(list (lambda () ,x)
	 (lambda (val) (setq ,x val))))
(defun get-passed-lexical (x)
  (funcall (car x)))
(defun set-passed-lexical (x val)
 (funcall (cadr x) val))
(defun add-to-list-lexable (wrapped-list element &optional append compare-fn)
  (let ((list (get-passed-lexical wrapped-list)))
    (if (member* element list :test (or compare-fn 'equal))
	 list
       (set-passed-lexical wrapped-list
	     (if append
		 (append list (list element))
	       (cons element list))))))
(let ((x '(a))) (add-to-list-lexable (wrap-lexical x) 'b) x) -> (b a)
but that's both gross and incompatible.