add-to-list with lexical variables

add-to-list with lexical variables

[Hongxu Chen]

Hi list,

  I am writing a snippet to add element into environment variables, and
  it is written as below:

#+BEGIN_SRC elisp  
(defun no-dup-add-env-ele (env env-ele-string)
   (let* ((env-separator (if (string-equal system-type "windows-nt") ";" ":"))
     (env-list (split-string (getenv env) env-separator)))
     (if (string-match-p env-separator env-ele-string)
        (dolist (env-ele (split-string env-ele-string env-separator))
                (add-to-list 'env-list env-ele)) 
     (add-to-list 'env-list env-ele-string))
     (setenv env (mapconcat 'identity env-list ":"))))
#+END_SRC

1. when I set `lexical-binding' to t and byte-compile the file, it
would report this error:

    add-to-list cannot use lexical var `env-list'

2. And when I using `lexical-let*' instead, there would be an warning:

    Warning: assignment to free variable `env-list'

3. However after resetting `lexical-binding' to nil, byte-compiles well.

So what are the differences?

Thanks,
Hongxu Chen

Re: add-to-list with lexical variables

[Pascal J. Bourguignon]

Hongxu Chen <leftcopy.chx@gmail.com> writes:

> Hi list,
>
>   I am writing a snippet to add element into environment variables, and
>   it is written as below:
>
> #+BEGIN_SRC elisp  
> (defun no-dup-add-env-ele (env env-ele-string)
>    (let* ((env-separator (if (string-equal system-type "windows-nt") ";" ":"))
>      (env-list (split-string (getenv env) env-separator)))
>      (if (string-match-p env-separator env-ele-string)
>         (dolist (env-ele (split-string env-ele-string env-separator))
>                 (add-to-list 'env-list env-ele)) 
>      (add-to-list 'env-list env-ele-string))
>      (setenv env (mapconcat 'identity env-list ":"))))
> #+END_SRC
>
> 1. when I set `lexical-binding' to t and byte-compile the file, it
> would report this error:
>
>     add-to-list cannot use lexical var `env-list'
>
> 2. And when I using `lexical-let*' instead, there would be an warning:
>
>     Warning: assignment to free variable `env-list'
>
> 3. However after resetting `lexical-binding' to nil, byte-compiles well.
>
> So what are the differences?

The difference is not.

    (not t) --> nil 
    (not nil) --> t

or:

    not lexical binding is dynamic binding.
    not dynamic binding is lexical binding.


Now, to add new elements to a list bound to some place, there's the
pushnew cl operator (a macro).  push can be used to push
unconditionnaly, and pop to remove the first element from a list bound
to a place.  That's how things have to be done with lexical binding.


(require 'cl)

(defun* pushnew/envvar (env-ele-string env &key (test (function equal)))
  (let* ((env-separator (if (member system-type '(windows-nt ms-dos)) ";" ":"))
         (env-list      (split-string (or (getenv env) "") env-separator t)))
    (dolist (env-ele (split-string env-ele-string env-separator))
      (pushnew env-ele env-list :test test))
    (setenv env (mapconcat 'identity env-list env-separator))))

(defun test/pushnew/envvar ()
  (setenv "TEST" nil)
  (assert (equal (pushnew/envvar "A:B:C" "TEST")
                 "C:B:A"))
  (assert (equal (getenv "TEST")
                 "C:B:A"))
  (assert (equal (pushnew/envvar "D:E:F" "TEST")
                 "F:E:D:C:B:A"))
  (assert (equal (getenv "TEST")
                 "F:E:D:C:B:A"))
  (assert (equal (pushnew/envvar "G" "TEST")
                 "G:F:E:D:C:B:A"))
  (assert (equal (getenv "TEST")
                 "G:F:E:D:C:B:A"))
  (assert (equal (pushnew/envvar "G:F:E:D:C:B:A" "TEST")
                 "G:F:E:D:C:B:A"))
  (assert (equal (pushnew/envvar "X:F:Y:D:Z:B" "TEST")
                 "Z:Y:X:G:F:E:D:C:B:A"))
  (assert (equal (pushnew/envvar "a:b:c:d" "TEST")
                 "d:c:b:a:Z:Y:X:G:F:E:D:C:B:A"))
  (assert (equal (pushnew/envvar "x:y:z" "TEST" :test (function equalp))
                 "d:c:b:a:Z:Y:X:G:F:E:D:C:B:A"))
  :success)

(test/pushnew/envvar)
--> :success


You have to add a test argument, since for some environment variables,
and for some values, you may want to do case insensitive, or more
complex comparison.

For example, if you mount a MS-DOS file system on a case sensitive unix file
system, itself mounted a HFS+ case insensitive file system, you will
have to compare parts of the path case sensitively, and parts case
insensitively:


mount /dev/disk1s1 /Volumes/case-sensitive
mount /dev/disk2s1 /Volumes/case-sensitive/mnt/ms-dos

Now, /VOLUMES/case-sensitive/mnt/ms-dos/DESCENT
is the same path as:
     /Volumes/case-sensitive/mnt/ms-dos/Descent
but not the same as:
     /Volumes/case-sensitive/MNT/ms-dos/descent
Perhaps there are both
     /Volumes/case-sensitive/MNT 
and:
     /Volumes/case-sensitive/mnt 
on the case sensitive file system!

so you will have to write:

(pushnew/envvar "/Volumes/case-sensitive/mnt/ms-dos/Descent"
                "PATH"
                :test (function file-system-sensitive-path-equal-p))

with:

(defun file-system-sensitive-path-equal-p (a b)
  (labels ((compare-path (curdir ac bc)
             (cond
               ((null ac) (null bc))
               ((null bc) nil)
               (t
                (and (funcall (if (case-sensitive-file-system-at-path-p curdir)
                                  (function equal)
                                  (function equalp))
                              (car ac) (car ab))
                     (compare-path (path-append curdir (car ac))
                                   (cdr ac) (cdr bc)))))))
    (compare-path "/"
                  (split-path (expand-file-name a))
                  (split-path (expand-file-name b)))))


So that if there's already

    /Volumes/case-sensitive/mnt/ms-dos/DESCENT

on PATH,

    (pushnew/envvar "/VOLUMES/case-sensitive/mnt/ms-dos/Descent"
                    "PATH"
                    :test (function file-system-sensitive-path-equal-p))

won't add it, but:

    (pushnew/envvar "/Volumes/case-sensitive/mnt/MS-DOS/DESCENT"
                    "PATH"
                    :test (function file-system-sensitive-path-equal-p))

will.


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.
You can take the lisper out of the lisp job, but you can't take the lisp out
of the lisper (; -- antifuchs

Re: add-to-list with lexical variables

[Hongxu Chen]

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> Hongxu Chen <leftcopy.chx@gmail.com> writes:
>
>> Hi list,
>>
>>   I am writing a snippet to add element into environment variables, and
>>   it is written as below:
>>
>> #+BEGIN_SRC elisp  
>> (defun no-dup-add-env-ele (env env-ele-string)
>>    (let* ((env-separator (if (string-equal system-type "windows-nt") ";" ":"))
>>      (env-list (split-string (getenv env) env-separator)))
>>      (if (string-match-p env-separator env-ele-string)
>>         (dolist (env-ele (split-string env-ele-string env-separator))
>>                 (add-to-list 'env-list env-ele)) 
>>      (add-to-list 'env-list env-ele-string))
>>      (setenv env (mapconcat 'identity env-list ":"))))
>> #+END_SRC
>>
>> 1. when I set `lexical-binding' to t and byte-compile the file, it
>> would report this error:
>>
>>     add-to-list cannot use lexical var `env-list'
>>
>> 2. And when I using `lexical-let*' instead, there would be an warning:
>>
>>     Warning: assignment to free variable `env-list'
>>
>> 3. However after resetting `lexical-binding' to nil, byte-compiles well.
>>
>> So what are the differences?
>
> The difference is not.
>
>     (not t) --> nil 
>     (not nil) --> t
>
> or:
>
>     not lexical binding is dynamic binding.
>     not dynamic binding is lexical binding.
>
>
> Now, to add new elements to a list bound to some place, there's the
> pushnew cl operator (a macro).  push can be used to push
> unconditionnaly, and pop to remove the first element from a list bound
> to a place.  That's how things have to be done with lexical binding.
>
>
> (require 'cl)
>
> (defun* pushnew/envvar (env-ele-string env &key (test (function equal)))
>   (let* ((env-separator (if (member system-type '(windows-nt ms-dos)) ";" ":"))
>          (env-list      (split-string (or (getenv env) "") env-separator t)))
>     (dolist (env-ele (split-string env-ele-string env-separator))
>       (pushnew env-ele env-list :test test))
>     (setenv env (mapconcat 'identity env-list env-separator))))
>
> (defun test/pushnew/envvar ()
>   (setenv "TEST" nil)
>   (assert (equal (pushnew/envvar "A:B:C" "TEST")
>                  "C:B:A"))
>   (assert (equal (getenv "TEST")
>                  "C:B:A"))
>   (assert (equal (pushnew/envvar "D:E:F" "TEST")
>                  "F:E:D:C:B:A"))
>   (assert (equal (getenv "TEST")
>                  "F:E:D:C:B:A"))
>   (assert (equal (pushnew/envvar "G" "TEST")
>                  "G:F:E:D:C:B:A"))
>   (assert (equal (getenv "TEST")
>                  "G:F:E:D:C:B:A"))
>   (assert (equal (pushnew/envvar "G:F:E:D:C:B:A" "TEST")
>                  "G:F:E:D:C:B:A"))
>   (assert (equal (pushnew/envvar "X:F:Y:D:Z:B" "TEST")
>                  "Z:Y:X:G:F:E:D:C:B:A"))
>   (assert (equal (pushnew/envvar "a:b:c:d" "TEST")
>                  "d:c:b:a:Z:Y:X:G:F:E:D:C:B:A"))
>   (assert (equal (pushnew/envvar "x:y:z" "TEST" :test (function equalp))
>                  "d:c:b:a:Z:Y:X:G:F:E:D:C:B:A"))
>   :success)
>
> (test/pushnew/envvar)
> --> :success
>
>
> You have to add a test argument, since for some environment variables,
> and for some values, you may want to do case insensitive, or more
> complex comparison.
>
> For example, if you mount a MS-DOS file system on a case sensitive unix file
> system, itself mounted a HFS+ case insensitive file system, you will
> have to compare parts of the path case sensitively, and parts case
> insensitively:
>
>
> mount /dev/disk1s1 /Volumes/case-sensitive
> mount /dev/disk2s1 /Volumes/case-sensitive/mnt/ms-dos
>
> Now, /VOLUMES/case-sensitive/mnt/ms-dos/DESCENT
> is the same path as:
>      /Volumes/case-sensitive/mnt/ms-dos/Descent
> but not the same as:
>      /Volumes/case-sensitive/MNT/ms-dos/descent
> Perhaps there are both
>      /Volumes/case-sensitive/MNT 
> and:
>      /Volumes/case-sensitive/mnt 
> on the case sensitive file system!
>
> so you will have to write:
>
> (pushnew/envvar "/Volumes/case-sensitive/mnt/ms-dos/Descent"
>                 "PATH"
>                 :test (function file-system-sensitive-path-equal-p))
>
> with:
>
> (defun file-system-sensitive-path-equal-p (a b)
>   (labels ((compare-path (curdir ac bc)
>              (cond
>                ((null ac) (null bc))
>                ((null bc) nil)
>                (t
>                 (and (funcall (if (case-sensitive-file-system-at-path-p curdir)
>                                   (function equal)
>                                   (function equalp))
>                               (car ac) (car ab))
>                      (compare-path (path-append curdir (car ac))
>                                    (cdr ac) (cdr bc)))))))
>     (compare-path "/"
>                   (split-path (expand-file-name a))
>                   (split-path (expand-file-name b)))))
>
>
> So that if there's already
>
>     /Volumes/case-sensitive/mnt/ms-dos/DESCENT
>
> on PATH,
>
>     (pushnew/envvar "/VOLUMES/case-sensitive/mnt/ms-dos/Descent"
>                     "PATH"
>                     :test (function file-system-sensitive-path-equal-p))
>
> won't add it, but:
>
>     (pushnew/envvar "/Volumes/case-sensitive/mnt/MS-DOS/DESCENT"
>                     "PATH"
>                     :test (function file-system-sensitive-path-equal-p))
>
> will.

Thanks so much for your instruction. I now feel that there are really so
many flaws in my snippet. 

However I still don't know the difference `lexical-binding' and
`lexical-let' brings. Are there some authoritative introductions/tutorials?

Regards,
Hongxu Chen

Re: add-to-list with lexical variables

[Stefan Monnier]

>>> 1. when I set `lexical-binding' to t and byte-compile the file, it
>>> would report this error:
>>> 
>>> add-to-list cannot use lexical var `env-list'

add-to-list is a function, and functions have access to the
dynamically-bound vars of its caller, but not to the lexically-bound
vars of its caller.

The warning was added because add-to-list is a frequent use-case where
Elisp code presumes dynamic binding.

>>> 2. And when I using `lexical-let*' instead, there would be an warning:
>>> 
>>> Warning: assignment to free variable `env-list'

lexical-let is an old way to get lexical binding.  It was added long
before add-to-list was invented, and it was only used at those few
places where people really wanted lexical binding, and those people
generally knew that add-to-list couldn't possibly work there, so the use
of add-to-list on a var bound with lexical-let was so rare that it
didn't occur to anybody to try and provide an ad-hoc check and warning.
instead it just notices that the call to `add-to-list' uses a (dynamic)
variable `env-list' which is not declared (i.e. is "free").

> However I still don't know the difference `lexical-binding' and
> `lexical-let' brings. Are there some authoritative introductions/tutorials?

lexical-let is an old hack made obsolete by lexical-binding.


        Stefan

Re: add-to-list with lexical variables

[Pascal J. Bourguignon]

Hongxu Chen <leftcopy.chx@gmail.com> writes:

> However I still don't know the difference `lexical-binding' and
> `lexical-let' brings. Are there some authoritative introductions/tutorials?

http://www.emacswiki.org/emacs/DynamicBindingVsLexicalBinding

Also, any Common Lisp tutorial covers them.  


You may be interested by my explanations in the context of Common Lisp:
https://groups.google.com/group/comp.lang.lisp/msg/58f1c7ed53d0c0d6


But basically, it's a question of time vs. space.


Dynamic is change over time.  So dynamic binding will be concerned by
the time, by WHEN things are bound and referenced.


Lexical is textual.  So lexical binding will be concerned by space, by
WHERE in the text things are bound and referenced.


With dynamic binding, things occur at certain times:
#+BEGIN_SRC elisp  
;; with lexical-binding set to nil:
(let ((list (list 1 2 3)))
  (add-to-list 'list 0))
#+END_SRC

time
----

 1.  the function list is called with arguments 1 2 and 3.

 2.  it returns a list: (1 2 3).

 3.  let creates a variable, it saves any dynamic binding the symbol
     list may have, and it names the variable with the symbol list, and
     it binds it (dynamically, at this time 3.) to the list (1 2 3).

 4.  (quote list)  is evaluated.

 5.  it returns the symbol list.

 6.  0 is evaluated.

 7.  it returns 0

 8.  the function add-to-list is called with as arguments, the symbol
     list and the integer 0.

 9.  the function changes the dynamic binding of the symbol list.
     remember at this time, while the function add-to-list is being
     called, there symbol list is dynamically bound to the list (1 2 3),
     since the time 3.

10.  it returns, and the dynamic binding to the variable list has been
     changed.

11.  let destroys the dynamic binding of list, and destroys the variable
     named list.  Whatever dynamic binding list had before is restored.

 

With lexical binding, what matters is the space, the place where
expressions are:
#+BEGIN_SRC elisp  
;; with lexical-binding set to t:
(let ((list (list 1 2 3)))
  (add-to-list 'list 0))
#+END_SRC

(let ((list (list 1 2 3)))
  ;; From HERE
  (add-to-list 'list 0)
  ;; To HERE, there is a lexical variable named list.
  )

(defun add-to-list (variable value)
  ;; From HERE
  …
  … code of add-to-list
  …
  ;; To HERE, we're outside the lexical scope of the let form above. 
  ;; There is no lexical variable named list here!  Only the variables
  ;; and parameters of add-to-list.
  )

So there's no way for add-to-list to access directly the lexical
variable list defined in the let form.  

It could access it indirectly, if you passed it a closure.  See for
example:
http://www.informatimago.com/articles/usenet.html#C-like-pointers-in-Lisp
(those CL examples should run the same in emacs lisp with
lexical-binding set to t).


But since add-to-list is predefined not to use a closure, but to modify
a dynamic binding, it won't change a lexical binding.  It's not it's
purpose, it's not what it's designed to do.


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.
You can take the lisper out of the lisp job, but you can't take the lisp out
of the lisper (; -- antifuchs

Re: add-to-list with lexical variables

[Hongxu Chen]

Thanks Pascal and Stefan! It's so kind of you to tell me so much
about the scope/binding.

It's much clear to me now(although I think I need to read your words and
do some practice to fully grasp it).

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> Hongxu Chen <leftcopy.chx@gmail.com> writes:
>
>> However I still don't know the difference `lexical-binding' and
>> `lexical-let' brings. Are there some authoritative introductions/tutorials?
>
> http://www.emacswiki.org/emacs/DynamicBindingVsLexicalBinding
>
> Also, any Common Lisp tutorial covers them.  
>
>
> You may be interested by my explanations in the context of Common Lisp:
> https://groups.google.com/group/comp.lang.lisp/msg/58f1c7ed53d0c0d6
>
>
> But basically, it's a question of time vs. space.
>
>
> Dynamic is change over time.  So dynamic binding will be concerned by
> the time, by WHEN things are bound and referenced.
>
>
> Lexical is textual.  So lexical binding will be concerned by space, by
> WHERE in the text things are bound and referenced.
>
>
> With dynamic binding, things occur at certain times:
> #+BEGIN_SRC elisp  
> ;; with lexical-binding set to nil:
> (let ((list (list 1 2 3)))
>   (add-to-list 'list 0))
> #+END_SRC
>
> time
> ----
>
>  1.  the function list is called with arguments 1 2 and 3.
>
>  2.  it returns a list: (1 2 3).
>
>  3.  let creates a variable, it saves any dynamic binding the symbol
>      list may have, and it names the variable with the symbol list, and
>      it binds it (dynamically, at this time 3.) to the list (1 2 3).
>
>  4.  (quote list)  is evaluated.
>
>  5.  it returns the symbol list.
>
>  6.  0 is evaluated.
>
>  7.  it returns 0
>
>  8.  the function add-to-list is called with as arguments, the symbol
>      list and the integer 0.
>
>  9.  the function changes the dynamic binding of the symbol list.
>      remember at this time, while the function add-to-list is being
>      called, there symbol list is dynamically bound to the list (1 2 3),
>      since the time 3.
>
> 10.  it returns, and the dynamic binding to the variable list has been
>      changed.
>
> 11.  let destroys the dynamic binding of list, and destroys the variable
>      named list.  Whatever dynamic binding list had before is restored.
>
>  
>
> With lexical binding, what matters is the space, the place where
> expressions are:
> #+BEGIN_SRC elisp  
> ;; with lexical-binding set to t:
> (let ((list (list 1 2 3)))
>   (add-to-list 'list 0))
> #+END_SRC
>
> (let ((list (list 1 2 3)))
>   ;; From HERE
>   (add-to-list 'list 0)
>   ;; To HERE, there is a lexical variable named list.
>   )
>
> (defun add-to-list (variable value)
>   ;; From HERE
>   …
>   … code of add-to-list
>   …
>   ;; To HERE, we're outside the lexical scope of the let form above. 
>   ;; There is no lexical variable named list here!  Only the variables
>   ;; and parameters of add-to-list.
>   )
>
> So there's no way for add-to-list to access directly the lexical
> variable list defined in the let form.  
>
> It could access it indirectly, if you passed it a closure.  See for
> example:
> http://www.informatimago.com/articles/usenet.html#C-like-pointers-in-Lisp
> (those CL examples should run the same in emacs lisp with
> lexical-binding set to t).
>
>
> But since add-to-list is predefined not to use a closure, but to modify
> a dynamic binding, it won't change a lexical binding.  It's not it's
> purpose, it's not what it's designed to do.

-- 
Regards,
Hongxu Chen