bug#9469: buffer-local variables seem to remember previous values

bug#9469: buffer-local variables seem to remember previous values

[Le Wang]

I've reproed in both Emacs 23.3 and bzr build from a few days ago both
on windows.

Steps:

1. emacs -Q

2. eval this region:

(setq buf-a (create-file-buffer "a"))

(setq foo nil)
(make-variable-buffer-local 'foo)

(defun test1 ()
  (interactive)
  (let (alist)
    (push '(:var . 0) alist)
    (with-current-buffer buf-a
      (setq foo alist))))

(defun test2 ()
  (interactive)
  (with-current-buffer buf-a
    (setcdr (assq :var foo) 20)))

(defun show ()
  (interactive)
  (with-current-buffer buf-a
    (format "    ; foo in 'a' is %s" foo)))

(defun test3 ()
  (interactive)
  (let (alist)
    (push `(:var . ,(+ 0)) alist)
    (with-current-buffer buf-a
      (setq foo alist))))


(test1)
(test2)
(test1)
(insert (show))
(test3)
(insert (show))

Note results on both `insert' lines should be identical but the first
insert some how remembers a previous value.  I find it surprising that
no one has ever come across this before.

-- 
Le

bug#9469: buffer-local variables seem to remember previous values

[Drew Adams]

> 1. emacs -Q
> 2. eval this region:
> (setq buf-a (create-file-buffer "a"))
> (setq foo nil)
> (make-variable-buffer-local 'foo)
> (defun test1 ()
>   (interactive)
>   (let (alist)
>     (push '(:var . 0) alist)
>     (with-current-buffer buf-a
>       (setq foo alist))))
> (defun test2 ()
>   (interactive)
>   (with-current-buffer buf-a
>     (setcdr (assq :var foo) 20)))
> (defun show ()
>   (interactive)
>   (with-current-buffer buf-a
>     (format "    ; foo in 'a' is %s" foo)))
> (defun test3 ()
>   (interactive)
>   (let (alist)
>     (push `(:var . ,(+ 0)) alist)
>     (with-current-buffer buf-a
>       (setq foo alist))))
> 
> (test1)
> (test2)
> (test1)
> (insert (show))
> (test3)
> (insert (show))
> 
> Note results on both `insert' lines should be identical but the first
> insert some how remembers a previous value.  I find it surprising that
> no one has ever come across this before.

No, they should not be identical.  This is a classic Lisp gotcha.

(test1) sets buffer-local var `foo' to a new alist ((:var . 0)).
(test2) sets the cdr of the single element of that alist to 20.
That means that `foo' in buf-a is now ((:var . 20)).
(test1) then creates a new alist and pushes the _same_ cons, (:var . 20) onto
it.
And it sets `foo' to this new alist.

If you were to use (cons :var 0) instead of '(:var . 0) then you would not be
reusing the same cons cell.

Note that different Lisps (and different implementations of the same Lisp) can
treat a sexp such as '(a b) differently - they might or might not create a new
list each time it is read or eval'd.

To be sure to get what you expect in situations like this, do not use '(...).
Use `cons' or `list' or equivalent backquote syntax.  Do not expect '(...) to
create new list structure each time it is read/eval'd.

bug#9469: buffer-local variables seem to remember previous values

[Le Wang]

On Sun, Sep 11, 2011 at 1:44 AM, Drew Adams <drew.adams@oracle.com> wrote:
> No, they should not be identical.  This is a classic Lisp gotcha.

Thanks for the pointer, Drew.

I was unable to google up any information regarding this as being a
classic gotcha.  Do you have any references?

If this is in fact "by design", then the manual should definitely say
so in the quoting section.  Do I have to follow a different procedure
to file a documentation bug?

-- 
Le

bug#9469: buffer-local variables seem to remember previous values

[Drew Adams]

> > No, they should not be identical.

BTW, I should have said "It is not the case that they should be identical."  Not
quite the same thing.  There is no "should" here.  AFAIK, most Lisps do not
prescribe the behavior in this regard - it is up to the implementation.  But I'm
no expert on just what various Lisps define wrt this.

> > This is a classic Lisp gotcha.
> Thanks for the pointer, Drew.
> 
> I was unable to google up any information regarding this as being a
> classic gotcha.  Do you have any references?

No, I would have to google also.  I said it's "classic", which probably really
means that I was bitten by the same gotcha many, many moon ago, and I learned my
lesson then, in discussion with people who knew Lisp well.

Well, a quick google of "lisp quote list" brings me to this:
http://stackoverflow.com/questions/134887/when-to-use-quote-in-lisp

Search that page for "DO NOT USE QUOTE TO CREATE LISTS THAT YOU WILL LATER
MODIFY".  It says, "The spec allows the compiler to treat quoted lists as
constants."  I guess it  might be referring to the Common Lisp spec; dunno.

Scroll down further to the explanation by Matthias Benkard.  He points out that
when you use (quote (a b c)) the list (a b c) is created (the first time) by the
Lisp loader or reader, not by `quote'.  `quote' just returns that already
existing list.  `quote' does not create new list structure (or new symbols or
new strings or...).

You can probably find other, similar explanations by googling.

The real lesson, I think, is what I said earlier, which works for all Lisps: Do
not expect `quote' to create new list structure.

> If this is in fact "by design", then the manual should definitely say
> so in the quoting section.  Do I have to follow a different procedure
> to file a documentation bug?

I don't think the Elisp manual should say anything about it, but I'll let others
decide that.

The takeaway is, I think, that you should not depend on '(...) to create new
list structure each time it is evaluated.  I suppose the manual could say that.
Or it could perhaps clarify that `quote' does not create new Lisp objects; it
just returns the object that is its argument, unevaluated.  What can confuse
people is that the object already exists; it is provided by the reader (or the
loader).

I guess you could say that the reader treats list notation it encounters
similarly to what it does for symbol notation: if a symbol with the encountered
name already exists then it uses ("returns") that symbol.  Otherwise, it first
creates a symbol with that name.  IOW, the reader `intern's names it encounters.
Similarly, it handles list notation it encounters by either creating a new list
or returning an existing one.

Again, I'm no expert on this.  Perhaps someone else can explain it better.

bug#9469: buffer-local variables seem to remember previous values

[Andreas Schwab]

This has nothing to do with the use of quote, but with the use of
destructive functions (setcdr) on a shared data structure.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#9469: buffer-local variables seem to remember previous values

[Le Wang]

On Mon, Sep 12, 2011 at 2:08 AM, Andreas Schwab <schwab@linux-m68k.org> wrote:
> This has nothing to do with the use of quote, but with the use of
> destructive functions (setcdr) on a shared data structure.

Of course you're technically right, but should the manual mention in
the "quoting" section that using quote the way I did results in a
shared data structure?  This part was surprising to me.

And I think it will be very surprising to others new to lisp.
Specifically, if you read the stackexchange thread Drew pointed to:
http://stackoverflow.com/questions/134887/when-to-use-quote-in-lisp

It seems I'm not the only one who didn't realize that quote does not cons.

> Andreas.
>
> --
> Andreas Schwab, schwab@linux-m68k.org
> GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
> "And now for something completely different."
>

-- 
Le

bug#9469: buffer-local variables seem to remember previous values

[Lars Magne Ingebrigtsen]

Le Wang <l26wang@gmail.com> writes:

>> This has nothing to do with the use of quote, but with the use of
>> destructive functions (setcdr) on a shared data structure.
>
> Of course you're technically right, but should the manual mention in
> the "quoting" section that using quote the way I did results in a
> shared data structure?  This part was surprising to me.

If I read your code correctly, Andreas is correct and precise as usual,
if a bit terse.  :-)

The thing you're seeing has absolutely nothing to do with the use of
quote.

You're destructively altering a single list and expecting to see the
list be different in various buffers.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#9469: buffer-local variables seem to remember previous values

[Le Wang]

On Mon, Sep 12, 2011 at 2:54 AM, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> If I read your code correctly, Andreas is correct and precise as usual,
> if a bit terse.  :-)

Ok, that was my fault.  I didn't know how to make heads or tails of
the situation, so included some red herrings in my original snippet.
This should clarify my exact point of confusion:

(setq foo-alist nil)
	;; => nil

(defun reset-var-in-foo-alist ()
  (push '(:var . 0) foo-alist))
	;; => reset-var-in-foo-alist

(defun increment-var-in-foo-alist ()
  (let ((var-cons (assq :var foo-alist)))
    (setcdr var-cons (1+ (cdr var-cons)))))
	;; => increment-var-in-foo-alist


(reset-var-in-foo-alist)
	;; => ((:var . 0))
(increment-var-in-foo-alist)
	;; => 1
(reset-var-in-foo-alist)
	;; => ((:var . 1) (:var . 1))


Again, my misunderstanding was that in `reset-var-in-foo-alist' quote
does not cons, I assumed it did.

-- 
Le

bug#9469: buffer-local variables seem to remember previous values

[Helmut Eller]

* Lars Magne Ingebrigtsen [2011-09-11 18:54] writes:

> Le Wang <l26wang@gmail.com> writes:
>
>>> This has nothing to do with the use of quote, but with the use of
>>> destructive functions (setcdr) on a shared data structure.
>>
>> Of course you're technically right, but should the manual mention in
>> the "quoting" section that using quote the way I did results in a
>> shared data structure?  This part was surprising to me.
>
> If I read your code correctly, Andreas is correct and precise as usual,
> if a bit terse.  :-)
>
> The thing you're seeing has absolutely nothing to do with the use of
> quote.
>
> You're destructively altering a single list and expecting to see the
> list be different in various buffers.

Well, quote is tricky!  It wouldn't hurt to point out that modifying a
quoted list often means modifying your code.  E.g.

(defun foo () 
  (let ((list '(1 2 3)))
    (setcdr list (cddr list))))

(equal (copy-tree (symbol-function 'foo))
       (progn
	 (foo)
	 (symbol-function 'foo)))

returns nil.  I think few beginners will see that using setcdr here not
only modifies the value stored in the variable list but also modifies
the s-exp stored in 'foo.

Helmut

bug#9469: buffer-local variables seem to remember previous values

[Andreas Schwab]

Le Wang <l26wang@gmail.com> writes:

> It seems I'm not the only one who didn't realize that quote does not cons.

 -- Special Form: quote object
     This special form returns OBJECT, without evaluating it.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#9469: buffer-local variables seem to remember previous values

[Lars Magne Ingebrigtsen]

Le Wang <l26wang@gmail.com> writes:

> Again, my misunderstanding was that in `reset-var-in-foo-alist' quote
> does not cons, I assumed it did.

Ok.  So I'm closing this report.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#9469: buffer-local variables seem to remember previous values

[Le Wang]

On Mon, Sep 12, 2011 at 3:54 AM, Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
>
> Ok.  So I'm closing this report.

First of all, it's entirely possible that in order to preserve my self
perception that "I am smart", my ego won't let me admit that I just
wasn't smart enough parse the manual properly.

However, I still think it's not spelled out clearly enough.  It needs
an example to clearly show the trap.

"The special form quote returns its single argument, as written,
without evaluating it. This provides a way to include constant symbols
and lists"

Isn't the first sentence misleading?  It does not *always* return a
single argument as written.  It does the first time, and memoizes
subsequently.  Where is the mention of that?

The second sentence clarifies that quote should be used for "constant
symbols and lists".

Again, pedantically speaking, the information is there.  It's just not
clear enough to prevent other people falling into this trap in the
future.  I'm happy to leave this bug closed, or work with someone
specifically responsible for the Emacs Lisp Manual to come up with a
clear small example.


>
> --
> (domestic pets only, the antidote for overdose, milk.)
>  bloggy blog http://lars.ingebrigtsen.no/
>



-- 
Le

bug#9469: buffer-local variables seem to remember previous values

[Andreas Schwab]

Le Wang <l26wang@gmail.com> writes:

> "The special form quote returns its single argument, as written,
> without evaluating it. This provides a way to include constant symbols
> and lists"
>
> Isn't the first sentence misleading?  It does not *always* return a
> single argument as written.  It does the first time, and memoizes
> subsequently.  Where is the mention of that?

Since the argument does not change, how is that wrong?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#9469: buffer-local variables seem to remember previous values

[Le Wang]

On Mon, Sep 12, 2011 at 3:57 PM, Andreas Schwab <schwab@linux-m68k.org> wrote:
> Since the argument does not change, how is that wrong?

Yes, I understand from a functional programming perspective, it's
perfectly fine.  You're right.  but you're still being pedantic.
Maybe setcdr the way I did was not very lisp-y.  I willingly concede
I'm a lisp novice, and I'm used to thinking in a more procedural way.
My point is that the manual would be more helpful to someone in my
position if it had an example saying don't modify quoted objects, and
why.

Anyway, if it's the opinion of the maintainers that the manual is fine
as it is, it's fine with me.

Thanks you for looking into this issue, and working on Emacs!

-- 
Le

bug#9469: buffer-local variables seem to remember previous values

[Andreas Schwab]

Le Wang <l26wang@gmail.com> writes:

> My point is that the manual would be more helpful to someone in my
> position if it had an example saying don't modify quoted objects, and
> why.

This has nothing at all to do with quoting.  Don't use destructive
functions if you don't know what you are doing.  If you store through a
pointer in C you also have to take care of side effects.  There is
nothing new here.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#9469: buffer-local variables seem to remember previous values

[Drew Adams]

> > My point is that the manual would be more helpful to someone in my
> > position if it had an example saying don't modify quoted 
> > objects, and why.
> 
> This has nothing at all to do with quoting.  Don't use destructive
> functions if you don't know what you are doing.

Everything that people have said in the thread is true.  That doesn't mean that
we can't help users more in this regard.

What could perhaps be made clearer in the manual - and perhaps in the Lisp Intro
manual (e.g., with pedagogic examples), is the role of the Lisp reader in
creating list structure, symbols, etc.  It is ignorance of or forgetting about
this feature of Lisp that confuses users (esp. newbies) when it comes to `quote'
etc.

Explanation of Lisp reading is far removed in the manual from the description of
`quote' (and they are not directly related, so there is nothing wrong with
this).  It might help to (a) develop the reader-creates-objects subject in the
Lisp Intro manual and (b) put a reminder about this in the section about
`quote', with a caveat about modifying etc.

In sum, it's not about whether you or the manual or Le is right or wrong.  It's
about helping users.  And yes, this is a potential point of confusion.  The
consequences of the Lisp reader creating things are not necessarily obvious to
newbies.

bug#9469: buffer-local variables seem to remember previous values

[Le Wang]

On Mon, Sep 12, 2011 at 4:35 PM, Andreas Schwab <schwab@linux-m68k.org> wrote:
> This has nothing at all to do with quoting.  Don't use destructive
> functions if you don't know what you are doing.  If you store through a
> pointer in C you also have to take care of side effects.  There is
> nothing new here.

This discussion has turned in a different direction.  I've filed doc
bug #9482, with some more details.  The maintainers can decide what to
do with it.

-- 
Le

bug#9469: buffer-local variables seem to remember previous values

[Stefan Monnier]

> "The special form quote returns its single argument, as written,
> without evaluating it. This provides a way to include constant symbols
> and lists"

> Isn't the first sentence misleading?  It does not *always* return a
> single argument as written.  It does the first time, and memoizes
> subsequently.  Where is the mention of that?

There is no memoization: the quote does not *construct* the object, it
just returns it.

The problem is similar/identical to what happens with immediate strings in C:

char *foo (void)
{
    char *res = "toto\n";
    res[1]++;
    return res;
}

[ modern C tries to fix this problem by the use of "const char*". ]

> Again, pedantically speaking, the information is there.  It's just not
> clear enough to prevent other people falling into this trap in the
> future.

I'd be happy to try and make it more clear to people who aren't familiar
with it (after all, that's the main target audience here), but I'm not
sure how.

Do you have some suggestion about what text to use to make it clear to you?


        Stefan

bug#9469: buffer-local variables seem to remember previous values

[Le Wang]

On Tue, Sep 13, 2011 at 8:54 PM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> Do you have some suggestion about what text to use to make it clear to you?

I filed documentation bug 9482, I I will suggest specific text there.

-- 
Le

bug#9469: buffer-local variables seem to remember previous values

[Helmut Eller]

* Stefan Monnier [2011-09-13 12:54] writes:

> I'd be happy to try and make it more clear to people who aren't familiar
> with it (after all, that's the main target audience here), but I'm not
> sure how.

I think a section "Lisp pitfalls" in the manual would be about right for
this.  That could contain examples and explanations like modifying
quoted lists or using #'delete without storing the result with setq and
other such misunderstandings.  Then add references from the description
for quote to the pitfalls section.

Helmut

bug#9469: buffer-local variables seem to remember previous values

[Johan Bockgård]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> "The special form quote returns its single argument, as written,
>> without evaluating it. This provides a way to include constant symbols
>> and lists"
>
>> Isn't the first sentence misleading?  It does not *always* return a
>> single argument as written.  It does the first time, and memoizes
>> subsequently.  Where is the mention of that?
>
> There is no memoization: the quote does not *construct* the object, it
> just returns it.

I think the "as written" bit is mildly confusing, and nearly falls into
this trap:

    In other languages, an expression is text; it has no other form. In
    Lisp, an expression is primarily a Lisp object and only secondarily
    the text that is the object's read syntax. Often there is no need to
    emphasize this distinction, but you must keep it in the back of your
    mind, or you will occasionally be very confused.

        -- (info "(elisp) Printed Representation")

bug#9469: buffer-local variables seem to remember previous values

[Drew Adams]

> I think the "as written" bit is mildly confusing, and nearly 
> falls into this trap:
> 
>     In other languages, an expression is text; it has no 
>     other form. In Lisp, an expression is primarily a Lisp
>     object

Indeed.  And I repeat that the misunderstanding comes from not knowing or not
sufficiently appreciating where those objects come from: what creates them.

That is what it would help to repeat/emphasize in the doc (e.g., in the context
of things like `quote', at least via a cross-reference).

Users need of course to understand that `quote' just returns its argument, but
that is not really the problem here, since they can misunderstand "its argument"
per the confusion that Johan referred to (text vs Lisp object).

What users might not get is that the argument that `quote' receives is a Lisp
object that has already been created.  That might not be obvious to someone new
to Lisp.  Especially when written as ', it is easy to misunderstand `quote' as
applying to the textual sexp that follows it, and not to a Lisp object.

The Lisp reader is kind of in the background of user awareness, so the question
of where the Lisp object comes from can mistakenly be answered by thinking that
it is `quote' that creates it based on the text/sexp that follows the '.  That,
I think, is the gotcha that tripped up Le (and he's not alone).

In the case of a literal list argument to `quote', e.g. (quote (a b c)), users
need to understand that the Lisp reader (or loader) creates the list (a b c).

Given that understanding, what can also be missing is that, depending on the
Lisp, the Lisp reader might not create a _new_ list each time it encounters the
sexp "(a b c)".  And in Emacs Lisp it in fact does not create a new list; it
reuses a previously created list, if available.  (Is that always true for Elisp?
What about gc?).

This too is not obvious.  And we can point out that literal strings are treated
differently - the reader does create a new, `eq'-unique string each time it sees
"abc".  The bottom line is that we should communicate the lesson that you should
not depend on the Lisp reader creating a new object each time it reads a sexp.

It would be helpful to mention this and explain that the reader's treatment of
list sexps is similar to its treatment of symbol sexps (names).  The treatment
of symbols is perhaps a bit easier to understand because we introduce the notion
of obarray, and because a symbol itself has an explicit string representation
(its name).

At a minimum, then, the manual's treatment of `quote' (and perhaps other things)
should reference some doc that explains the Lisp reader behavior well.

bug#9469: buffer-local variables seem to remember previous values

[Helmut Eller]

* Drew Adams [2011-09-13 18:52] writes:

[...]
> The Lisp reader is kind of in the background of user awareness, so the
> question of where the Lisp object comes from can mistakenly be
> answered by thinking that it is `quote' that creates it based on the
> text/sexp that follows the '.  That, I think, is the gotcha that
> tripped up Le (and he's not alone).

Yes, that's crucial.  Beginners need to understand that read and eval
are separate and what the purpose of each is: read turns text into
s-expressions.  eval "executes" s-expressions; eval is described in
terms of s-expressions and not text.  Also that the reader is not the
only way to construct s-expressions for eval.

[...]
> Given that understanding, what can also be missing is that, depending
> on the Lisp, the Lisp reader might not create a _new_ list each time
> it encounters the sexp "(a b c)".  And in Emacs Lisp it in fact does
> not create a new list; it reuses a previously created list, if
> available.  (Is that always true for Elisp?  What about gc?).

I've never heard of a reader that does what you describe here.  The
reader creates a new list for "'foo" just as for "(quote foo)" or for
"(bar foo)".

(eq (read "'a") (read "'a")) returns nil; all the time.

Just like
(let ((sexp (read "('a 'a)")))
  (eq (car sexp) 
      (cadr sexp)))
is always nil.

The reader interns symbols but lists and vectors are freshly created
(ignoring the #1# syntax for now).  The compiler may coalesce constant
lists that are equal; but don't let us confuse the reader with the
compiler.

Helmut

bug#9469: buffer-local variables seem to remember previous values

[Le Wang]

On Wed, Sep 14, 2011 at 2:02 AM, Johan Bockgård <bojohan@gnu.org> wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> There is no memoization: the quote does not *construct* the object, it
>> just returns it.
>
> I think the "as written" bit is mildly confusing, and nearly falls into
> this trap:
>
>    In other languages, an expression is text; it has no other form. In
>    Lisp, an expression is primarily a Lisp object and only secondarily
>    the text that is the object's read syntax. Often there is no need to
>    emphasize this distinction, but you must keep it in the back of your
>    mind, or you will occasionally be very confused.
>
>        -- (info "(elisp) Printed Representation")

This is kind of funny.  Stefan and Andreas basically kept saying quote
just returns the thing, but doesn't construct it.  My mind couldn't
grasp what that means.

The reader is the chasm that I just couldn't cross.  In my mind a
program describes how something happens, it isn't the thing that
happens.  "(quote (a b))", `quote' is a function that returns a
reference to something.  And for it to return the same reference, then
it must be memoizing based on the inputs.

The passage Johan posted along with Drew's explanation has enlightened
me.  Thank you all.

-- 
Le