lexbind ready for merge

lexbind ready for merge

[Stefan Monnier]

OK, so AFAIK the lexbind branch is pretty much ready for merge.
The only issue is that I'm not sure it has seen as much testing as
I'd like, but then that's largely because it hasn't been merged yet.
So if I don't hear any screams until then, I plan to merge it into trunk
over the week-end or nearby.


        Stefan

Re: lexbind ready for merge

[Daniel Colascione]

On 3/29/2011 2:44 PM, Stefan Monnier wrote:
 > OK, so AFAIK the lexbind branch is pretty much ready for merge.
 > The only issue is that I'm not sure it has seen as much testing as
 > I'd like, but then that's largely because it hasn't been merged yet.
 > So if I don't hear any screams until then, I plan to merge it into trunk
 > over the week-end or nearby.

Thanks.  I've been looking forward to this for a while.

Just a few comments (I might send patches for some of these things when 
I have time):

- The elisp manual still claims in a variety of places that Emacs does 
not support closures

- apply-partially should have a compiler-macro now that we can implement 
it very efficiently; also, funcall-partially.

- It might be a good idea to remove the "Once Emacs 19 becomes 
standard..." comment from cl.el

- Can lexical-let and lexical-let* be made a no-op when compiling 
lexbound code?  Looking at cl.el, it appears they're still up their 
usual dirty tricks.

- lexical-binding only applies to code evaluated by `eval-buffer' and 
`eval-region'?! So I can't make code evaluated by M-: lexbound?

- It'd be nice to be able to write small pieces of lexical code in 
non-lexbound code, e.g., in the expansion of a macro that gets used by 
both lexbound and non-lexbound. What's the best way to do that?

- The documentation claims that defun doesn't capture its lexical scope. 
  In interpreted code, it does.

(require 'cl)
(let ((bar2 5))
   (defun foo ()
     (incf bar2)
     (message "hi: %s" bar2)))

In compiled code, we do not capture the variable and instead warn about 
it.  Instead, we should capture the variable.  Common Lisp explicitly 
allows this use, and it's convenient in some cases.

- Disassembling a closure reports closed-over variables as constants; 
they're not.

- Do we really use a whole cons cell for each closed-over variable, even 
in compiled code?

Re: lexbind ready for merge

[Stefan Monnier]

> - The elisp manual still claims in a variety of places that Emacs does not
> support closures

Pointers to offenders welcome.

> - apply-partially should have a compiler-macro now that we can implement it
> very efficiently; also, funcall-partially.

The current definition of apply-partially is not too
inefficient, actually.  I don't know what funcall-partially would
be like.  And I don't like either of them: apply-partially is just a way
to make it easy to build closures by hand without resorting to
lexical-let and independently from lexical-binding, but once you use
lexical-binding you're usually better off using a plain closure.

> - It might be a good idea to remove the "Once Emacs 19 becomes standard..."
> comment from cl.el

Feel free to do that on the trunk, I don't think it's really related
to lexbind.

> - Can lexical-let and lexical-let* be made a no-op when compiling lexbound
> code?  Looking at cl.el, it appears they're still up their usual
> dirty tricks.

They can *almost* be turned into `let' and `let*', except that
(lexical-let ((buffer-file-name 3)) ...) will bind buffer-file-name
lexically whereas `let' will always bind it dynamically.  We could
either ignore those issues or try to handle them, but I'd rather just
mark lexical-let obsolete.
(Of course, there's also the difficulty for the macro to reliably
determine whether the expansion will be run in lexical-binding or
dynamic-binding mode).

> - lexical-binding only applies to code evaluated by `eval-buffer' and
> eval-region'?! So I can't make code evaluated by M-: lexbound?

?!?  AFAIK, M-: uses lexical or dynamic scoping according to the value
of lexical-binding in the current buffer.

> - It'd be nice to be able to write small pieces of lexical code in
> non-lexbound code, e.g., in the expansion of a macro that gets used by both
> lexbound and non-lexbound.

Yes, it'd be nice.

> What's the best way to do that?

Currently, I think the best way to do that is to add the feature to the
byte-compiler.  The most promising avenue for it might be to use code
of the form ((closure () <formalargs> <lexbindcode>) <actualargs>) and
compile efficiently (I think currently such code will either result in
a complaint about a malformed function, or else will leave the function
uncompiled).

> - The documentation claims that defun doesn't capture its lexical scope. In
> interpreted code, it does.

> (require 'cl)
> (let ((bar2 5))
>   (defun foo ()
>     (incf bar2)
>     (message "hi: %s" bar2)))

> In compiled code, we do not capture the variable and instead warn about it.
> Instead, we should capture the variable.

Yup.

> Common Lisp explicitly allows this use, and it's convenient in
> some cases.

If you need it you can use

  (let ((bar2 5))
    (defalias 'foo (lambda ()
      "Foo."
      (incf bar2)
      (message "hi: %s" bar2))))

IIRC, the reason why defun doesn't work for it is fundamentally linked
to some silly technicality, but I justify it for myself by the fact that
all the "defun within a non-empty context" I've seen were bogus, so I'm
not strongly motivated to handle it right.

> - Disassembling a closure reports closed-over variables as constants;
> they're not.

They are.

> - Do we really use a whole cons cell for each closed-over variable, even in
> compiled code?

Yes, tho only for variables which are mutated (yup, `setq' is costly).


        Stefan

Re: lexbind ready for merge

[Daniel Colascione]

On 3/29/2011 6:22 PM, Stefan Monnier wrote:
>> - apply-partially should have a compiler-macro now that we can implement it
>> very efficiently; also, funcall-partially.
>
> The current definition of apply-partially is not too
> inefficient, actually.  I don't know what funcall-partially would
> be like.  And I don't like either of them: apply-partially is just a way
> to make it easy to build closures by hand without resorting to
> lexical-let and independently from lexical-binding, but once you use
> lexical-binding you're usually better off using a plain closure.

apply-partially at least involves less typing lexical-let, which (as I 
explain below) is required in some cases even with lexbind.

As for funcall-partially: I never really liked how apply-partially 
differs from apply.  The latter treats its last argument specially, 
while the former does not.  If I had my way, I'd rename the current 
apply-partially to funcall-partially (or partial-funcall?) and create a 
new apply-partially that unpacks its last argument.  But it probably 
can't be changed now...

>> - It might be a good idea to remove the "Once Emacs 19 becomes standard..."
>> comment from cl.el
>
> Feel free to do that on the trunk, I don't think it's really related
> to lexbind.

If I could, I would. :-)

>> - Can lexical-let and lexical-let* be made a no-op when compiling lexbound
>> code?  Looking at cl.el, it appears they're still up their usual
>> dirty tricks.
>
> They can *almost* be turned into `let' and `let*', except that
> (lexical-let ((buffer-file-name 3)) ...) will bind buffer-file-name
> lexically whereas `let' will always bind it dynamically.  We could
> either ignore those issues or try to handle them, but I'd rather just
> mark lexical-let obsolete.

I'd prefer to ignore the issues for now and transform lexical-let to let 
when lexical-binding is on, generating a compiler error if we're trying 
to lexically bind a special variable.  I don't think many people try to 
do that.  A macro that uses a lexical binding in its generated code 
still needs to use lexical-let in order for its generated form to work 
properly in either environment, and even outside macros, making 
lexical-let cheap in the lexbound case gives us a way to create 
backward-compatible code that automatically becomes more efficient in 
Emacs 24.

> (Of course, there's also the difficulty for the macro to reliably
> determine whether the expansion will be run in lexical-binding or
> dynamic-binding mode).

Wouldn't inspecting the value of lexical-binding work well enough?  For 
evaluated code, the macro (regardless of how or whether it was compiled) 
will be called directly from the evaluator with the appropriate value of 
lexical-binding set.  For compiled code, the macro will be called by the 
compiler with lexical-binding set to the value the compiler is using to 
compile the resulting form.

The value of lexical-let at macro-run time and result-evaluation time 
*could* differ if the resulting form is squirreled away somewhere and 
reused later --- but I can't think of a case where that happens without 
the code being wrapped in some kind of function object that would 
capture the current value of lexical-let.

Of course, you can lie:

(defmacro* with-broken-code (&body body)
   (let (lexical-binding)
     (macroexpand-all `(progn ,@body)))

That kind of thing is firmly in "It hurts when I do this" territory though.

>
>> - lexical-binding only applies to code evaluated by `eval-buffer' and
>> eval-region'?! So I can't make code evaluated by M-: lexbound?
>
> ?!?  AFAIK, M-: uses lexical or dynamic scoping according to the value
> of lexical-binding in the current buffer.

It does, but the documentation string still gives me the impression that 
it wouldn't be.  Why isn't lexical-binding respected for all evaluation?

>> - It'd be nice to be able to write small pieces of lexical code in
>> non-lexbound code, e.g., in the expansion of a macro that gets used by both
>> lexbound and non-lexbound.
>
> Yes, it'd be nice.
>
>> What's the best way to do that?
>
> Currently, I think the best way to do that is to add the feature to the
> byte-compiler.  The most promising avenue for it might be to use code
> of the form ((closure ()<formalargs>  <lexbindcode>)<actualargs>) and
> compile efficiently (I think currently such code will either result in
> a complaint about a malformed function, or else will leave the function
> uncompiled).

Ah, I see what you mean.  Would a with-lexical-scope form suffice?

;; with-lexical-scope is itself compiled with lexical-binding t
(defmacro* with-lexical-scope (&body body)
   (let* ((vars (remove-if (lambda (var)
                             (or (special-variable-p var)
                                 (not (boundp var))))
                           (find-free-vars `(progn ,@body))))
          (closure `(closure (t) ; no environment
                   ,@args ,@body)))
     `(funcall ,closure ,@vars)))

(defvar b 42)
(defun some-dynamic-function (a)
   (with-lexical-scope
     (lambda (c)
       (+ a b c)))

;; or even

(with-lexical-scope
   (defun foo (arg) ...))

That's not the same as (defun foo (arg) (with-lexical-scope ...))) 
because the argument binding strategies differ.

>> - The documentation claims that defun doesn't capture its lexical scope. In
>> interpreted code, it does.
>
>> (require 'cl)
>> (let ((bar2 5))
>>    (defun foo ()
>>      (incf bar2)
>>      (message "hi: %s" bar2)))
>
>> In compiled code, we do not capture the variable and instead warn about it.
>> Instead, we should capture the variable.
>
> Yup.
>
>> Common Lisp explicitly allows this use, and it's convenient in
>> some cases.
>
> If you need it you can use
>
>    (let ((bar2 5))
>      (defalias 'foo (lambda ()
>        "Foo."
>        (incf bar2)
>        (message "hi: %s" bar2))))
>
> IIRC, the reason why defun doesn't work for it is fundamentally linked
> to some silly technicality, but I justify it for myself by the fact that
> all the "defun within a non-empty context" I've seen were bogus, so I'm
> not strongly motivated to handle it right.

Even if it's not particularly common, being consistent with Common Lisp 
and having fewer special cases are good things.  Some people use 
constructs like this to create module-private variables (which is a bad 
idea, but that doesn't stop people doing it.)

>> - Disassembling a closure reports closed-over variables as constants;
>> they're not.
>
> They are.

Err, yes.  They are.

>> - Do we really use a whole cons cell for each closed-over variable, even in
>> compiled code?
>
> Yes, tho only for variables which are mutated (yup, `setq' is costly).

Couldn't we create a box type, which would would be like a cons except 
that it'd hold just one value?  (The #<foo> read-syntax is available.)

Re: lexbind ready for merge

[Tassilo Horn]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> OK, so AFAIK the lexbind branch is pretty much ready for merge.  The
> only issue is that I'm not sure it has seen as much testing as I'd
> like, but then that's largely because it hasn't been merged yet.  So
> if I don't hear any screams until then, I plan to merge it into trunk
> over the week-end or nearby.

Please go for it.  I'm using it since about a week, and I didn't run on
errors (that haven't been solved until now)...

...except that the current git version of Gnus doesn't build with
lexbind emacs:

--8<---------------cut here---------------start------------->8---
% make
cd lisp && make EMACS="emacs" lispdir="/usr/share/emacs/site-lisp/gnus" all
make[1]: Entering directory `/home/horn/repos/el/gnus/lisp'
rm -f *.elc gnus-load.el auto-autoloads.* custom-load.*
URLDIR="/usr/share/emacs/24.0.50/lisp/url/" W3DIR="no" lispdir="/usr/share/emacs/site-lisp/gnus" srcdir=. emacs -batch -q -no-site-file -l ./dgnushack.el -f dgnushack-make-cus-load .
Debugger entered--Lisp error: (wrong-number-of-arguments (1 . 1) 2)
  byte-compile-from-buffer(#<buffer  *temp*> "foo.el")
  (progn (insert "(defcustom foo (1+ (random)) \"\" :group 'emacs)\n") (byte-compile-from-buffer (current-buffer) "foo.el"))
  (unwind-protect (progn (insert "(defcustom foo (1+ (random)) \"\" :group 'emacs)\n") (byte-compile-from-buffer (current-buffer) "foo.el")) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert "(defcustom foo (1+ (random)) \"\" :group 'emacs)\n") (byte-compile-from-buffer (current-buffer) "foo.el")) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (with-current-buffer temp-buffer (unwind-protect (progn (insert "(defcustom foo (1+ (random)) \"\" :group 'emacs)\n") (byte-compile-from-buffer (current-buffer) "foo.el")) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (with-current-buffer temp-buffer (unwind-protect (progn (insert "(defcustom foo (1+ (random)) \"\" :group 'emacs)\n") (byte-compile-from-buffer (current-buffer) "foo.el")) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  (with-temp-buffer (insert "(defcustom foo (1+ (random)) \"\" :group 'emacs)\n") (byte-compile-from-buffer (current-buffer) "foo.el"))
  (let ((outbuf (with-temp-buffer (insert "(defcustom foo (1+ (random)) \"\" :group 'emacs)\n") (byte-compile-from-buffer (current-buffer) "foo.el")))) (when outbuf (prog1 (with-current-buffer outbuf (goto-char (point-min)) (search-forward " 'foo '(byte-code " nil t)) (kill-buffer outbuf))))
  dgnushack-emacs-compile-defcustom-p()
  (if (dgnushack-emacs-compile-defcustom-p) (progn (maybe-fbind (quote (defined-colors face-attribute))) (maybe-bind (quote (idna-program installation-directory)))))
  (when (dgnushack-emacs-compile-defcustom-p) (maybe-fbind (quote (defined-colors face-attribute))) (maybe-bind (quote (idna-program installation-directory))))
  eval-buffer(#<buffer  *load*> nil "/home/horn/repos/el/gnus/lisp/dgnushack.el" nil t)  ; Reading at buffer position 9629
  load-with-code-conversion("/home/horn/repos/el/gnus/lisp/dgnushack.el" "/home/horn/repos/el/gnus/lisp/dgnushack.el" nil t)
  load("/home/horn/repos/el/gnus/lisp/dgnushack.el" nil t)
  command-line-1(("-l" "./dgnushack.el" "-f" "dgnushack-make-cus-load" "."))
  command-line()
  normal-top-level()

make[1]: *** [gnus-load.el] Error 255
make[1]: Leaving directory `/home/horn/repos/el/gnus/lisp'
make: *** [lick] Error 2
--8<---------------cut here---------------end--------------->8---

The reason is that in trunk, that function has the signature

  byte-compile-from-buffer (bytecomp-inbuffer &optional bytecomp-filename)

but in the lexbind branch, the optional bytecomp-filename has been
removed:

  byte-compile-from-buffer (inbuffer)

AFAIKS, gnus doesn't use the foo.el file anyway, so basically that arg
can be removed.  But maybe the `byte-compile-from-buffer' definition in
[[S]X]Emacs requires exactly 2 arguments?  I don't know...

Bye,
Tassilo

Re: lexbind ready for merge

[Eli Zaretskii]

> From: Tassilo Horn <tassilo@member.fsf.org>
> Date: Wed, 30 Mar 2011 09:28:57 +0200
> Cc: emacs-devel@gnu.org
> 
> Please go for it.
> [...]
> ...except that the current git version of Gnus doesn't build with
> lexbind emacs:

So how is it a good idea to "go for it"?  Gnus is resync'ed with Emacs
practically every day, so chances are they will resync before Stefan
gets around to merging the branch.

Re: lexbind ready for merge

[Juanma Barranquero]

On Wed, Mar 30, 2011 at 06:10, Daniel Colascione
<dan.colascione@gmail.com> wrote:

> Even if it's not particularly common, being consistent with Common Lisp and
> having fewer special cases are good things.  Some people use constructs like
> this to create module-private variables (which is a bad idea, but that
> doesn't stop people doing it.)

Why a bad idea? It's a common idiom for private, persistent variables.

    Juanma

Re: lexbind ready for merge

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> Please go for it.
>> [...]
>> ...except that the current git version of Gnus doesn't build with
>> lexbind emacs:
>
> So how is it a good idea to "go for it"?  Gnus is resync'ed with Emacs
> practically every day, so chances are they will resync before Stefan
> gets around to merging the branch.

The problematic file dgnushack.el is not included in emacs.  It's only
used by the standalone gnus version.  I've already reported the issue at
the Gnus mailinglist, so I'm pretty confident that it will be taken care
of even before the weekend.

Bye,
Tassilo

Re: lexbind ready for merge

[Ted Zlatanov]

On Wed, 30 Mar 2011 07:30:53 -0400 Eli Zaretskii <eliz@gnu.org> wrote: 

>> From: Tassilo Horn <tassilo@member.fsf.org>
>> Date: Wed, 30 Mar 2011 09:28:57 +0200
>> Cc: emacs-devel@gnu.org
>> 
>> Please go for it.
>> [...]
>> ...except that the current git version of Gnus doesn't build with
>> lexbind emacs:

EZ> So how is it a good idea to "go for it"?  Gnus is resync'ed with Emacs
EZ> practically every day, so chances are they will resync before Stefan
EZ> gets around to merging the branch.

Gnus is synchronized with Emacs manually by Katsumi Yamaoka.  He can
abstain from running the sync if Stefan or someone else asks.

Ted

lexbind: how to replace lexical-let approach to hide secrets (was: lexbind ready for merge)

[Ted Zlatanov]

On Wed, 30 Mar 2011 13:35:56 +0200 Juanma Barranquero <lekktu@gmail.com> wrote: 

JB> On Wed, Mar 30, 2011 at 06:10, Daniel Colascione
JB> <dan.colascione@gmail.com> wrote:

>> Even if it's not particularly common, being consistent with Common Lisp and
>> having fewer special cases are good things.  Some people use constructs like
>> this to create module-private variables (which is a bad idea, but that
>> doesn't stop people doing it.)

JB> Why a bad idea? It's a common idiom for private, persistent variables.

Slightly related: auth-source.el uses `lexical-let' to define lambda
accessors for secrets (so printing them, for instance, won't show a
password).  Will the secrets still be hidden in the lexbind branch?  And
is there a smarter way to do it?

Sample code:

(let ((data "my secret"))
  (lexical-let ((data data)) (lambda () data)))

Thanks
Ted

Re: lexbind ready for merge

[Stefan Monnier]

>> be like.  And I don't like either of them: apply-partially is just a way
>> to make it easy to build closures by hand without resorting to
[...]
> apply-partially at least involves less typing lexical-let, which (as

Yes, I see we agree.

> As for funcall-partially: I never really liked how apply-partially differs
> from apply.  The latter treats its last argument specially, while the former
> does not.  If I had my way, I'd rename the current apply-partially to
> funcall-partially (or partial-funcall?) and create a new apply-partially
> that unpacks its last argument.  But it probably can't be changed now...

I see your point.  I chose the name `apply-partially' because in the
functional programming community this is generally called a "partial
application", but indeed it clashes somewhat with the use of `apply' in
Lisp which doesn't just mean "function application".
I think it's too late to fix it, tho.

>>> - It might be a good idea to remove the "Once Emacs 19 becomes standard..."
>>> comment from cl.el
>> Feel free to do that on the trunk, I don't think it's really related
>> to lexbind.
> If I could, I would. :-)

Why don't you request write access via Savannah?  That would also make
it easier for you to maintain js.el.

>> They can *almost* be turned into `let' and `let*', except that
>> (lexical-let ((buffer-file-name 3)) ...) will bind buffer-file-name
>> lexically whereas `let' will always bind it dynamically.  We could
>> either ignore those issues or try to handle them, but I'd rather just
>> mark lexical-let obsolete.
> I'd prefer to ignore the issues for now and transform lexical-let to let
> when lexical-binding is on, generating a compiler error if we're trying to
> lexically bind a special variable.  I don't think many people try to do
> that.  A macro that uses a lexical binding in its generated code still needs
> to use lexical-let in order for its generated form to work properly in
> either environment, and even outside macros, making lexical-let cheap in the
> lexbound case gives us a way to create backward-compatible code that
> automatically becomes more efficient in Emacs 24.

I'd tend to agree, but see below.

>> (Of course, there's also the difficulty for the macro to reliably
>> determine whether the expansion will be run in lexical-binding or
>> dynamic-binding mode).
> Wouldn't inspecting the value of lexical-binding work well enough?

Sadly, that is not reliable: it only indicates whether the code found in
the current buffer uses lexical-binding, but the macro-call might be in
a function defined in some other file/buffer.

>>> - lexical-binding only applies to code evaluated by `eval-buffer' and
>>> eval-region'?! So I can't make code evaluated by M-: lexbound?
>> ?!?  AFAIK, M-: uses lexical or dynamic scoping according to the value
>> of lexical-binding in the current buffer.
> It does, but the documentation string still gives me the impression that it
> wouldn't be.

What part of its docstring gives you this impression?

> Why isn't lexical-binding respected for all evaluation?

I tried to patch all places where it needs to be used.  If I missed
some, please point them out.

>> Currently, I think the best way to do that is to add the feature to the
>> byte-compiler.  The most promising avenue for it might be to use code
>> of the form ((closure ()<formalargs>  <lexbindcode>)<actualargs>) and
>> compile efficiently (I think currently such code will either result in
>> a complaint about a malformed function, or else will leave the function
>> uncompiled).

> Ah, I see what you mean.  Would a with-lexical-scope form suffice?

> ;; with-lexical-scope is itself compiled with lexical-binding t
> (defmacro* with-lexical-scope (&body body)
>   (let* ((vars (remove-if (lambda (var)
>                             (or (special-variable-p var)
>                                 (not (boundp var))))
>                           (find-free-vars `(progn ,@body))))
>          (closure `(closure (t) ; no environment
>                   ,@args ,@body)))
>     `(funcall ,closure ,@vars)))

I think I'd rather just do:

(defmacro with-lexical-scope (&rest body)
  `((closure (t) () ,@body)))

That avoids several problems:
- no need to implement find-free-vars (tho it's already in cconv.el).
- no need to macroexpand-all and traverse `body' to find its free vars.
- `boundp' won't DTRT in the byte-compiler (you'd want to consult
  byte-compile-bound-variables instead).
- the semantics of capturing dynbound variables as if they were lexical
  vars sounds nasty.
- you copy the value of the free vars, which will only do the right
  thing if those vars aren't mutated.
  
Of course, to make the above work, you still need to teach the
byte-compiler how to handle such code, which shouldn't be too hard.

>> IIRC, the reason why defun doesn't work for it is fundamentally linked
>> to some silly technicality, but I justify it for myself by the fact that
>> all the "defun within a non-empty context" I've seen were bogus, so I'm
>> not strongly motivated to handle it right.
> Even if it's not particularly common, being consistent with Common Lisp and
> having fewer special cases are good things.  Some people use constructs like
> this to create module-private variables (which is a bad idea, but that
> doesn't stop people doing it.)

The right way to fix it is to make defun (and defmacro) a macro, and
that would be a good thing it its own right.  When I tried that the only
problem I bumped into is that we want to dump "#$" in the .elc file to
refer to load-file-name for on-demand-loading of docstrings, so as soon
as someone comes up with a clean/neat way to get `pr in1' to output
a "#$", we'll turn defun and defmacro into macros and this issue
will disappear.

>>> - Disassembling a closure reports closed-over variables as constants;
>>> they're not.
>> They are.
> Err, yes.  They are.

Thank you ;-)

>>> - Do we really use a whole cons cell for each closed-over variable, even in
>>> compiled code?
>> Yes, tho only for variables which are mutated (yup, `setq' is costly).
> Couldn't we create a box type, which would would be like a cons except that
> it'd hold just one value?  (The #<foo> read-syntax is available.)

What for?  We currently use the lower 3 bits for tag, so objects have to
be aligned on a multiple of 8, so on 32bit systems the smallest objects
we can handle use up 64bits; and since in such systems cons cells use up
65bits, they're pretty close to optimal.  Also cons cells have their tag
in the immediate 3bit tag which is good for efficiency, but if we want
to add another small object we wouldn't be able to use one of those 3bit
tag values for it because they're all used already, so we'd have to
treat it as a special kind of vector or misc type but all of those
objects already need more than 64bits... nah, cons cells work just fine.

OTOH we do need to reduce the number of cases where we have to convert
variables to cons-cells, and that mostly means "use fewer closures".
This is actually "easy" to do right now: change the (byte-code/compiler)
implementation of condition-case, unwind-protect, and catch so they
don't require closures, since they're the main source of closures.


        Stefan

Re: lexbind ready for merge

[Stefan Monnier]

> The reason is that in trunk, that function has the signature
>   byte-compile-from-buffer (bytecomp-inbuffer &optional bytecomp-filename)
> but in the lexbind branch, the optional bytecomp-filename has been
> removed:

>   byte-compile-from-buffer (inbuffer)

> AFAIKS, gnus doesn't use the foo.el file anyway, so basically that arg
> can be removed.  But maybe the `byte-compile-from-buffer' definition in
> [[S]X]Emacs requires exactly 2 arguments?  I don't know...

Why on earth does Gnus need to call byte-compile-from-buffer?


        Stefan

Re: lexbind ready for merge

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The reason is that in trunk, that function has the signature
>>   byte-compile-from-buffer (bytecomp-inbuffer &optional bytecomp-filename)
>> but in the lexbind branch, the optional bytecomp-filename has been
>> removed:
>
>>   byte-compile-from-buffer (inbuffer)
>
>> AFAIKS, gnus doesn't use the foo.el file anyway, so basically that arg
>> can be removed.  But maybe the `byte-compile-from-buffer' definition in
>> [[S]X]Emacs requires exactly 2 arguments?  I don't know...
>
> Why on earth does Gnus need to call byte-compile-from-buffer?

The purpose of the whole dgnushack.el is to figure out the correct
load-path when byte-compiling.  But I'm the wrong person to ask why that
messy dance has to be danced...  I added the Gnus list to the Cc.

The function definition is

(defun dgnushack-emacs-compile-defcustom-p ()
  "Return non-nil if Emacs byte compiles `defcustom' forms.
Those Emacsen will warn against undefined variables and functions used
in `defcustom' forms."
  (let ((outbuf (with-temp-buffer
		  (insert "(defcustom foo (1+ (random)) \"\" :group 'emacs)\n")
		  (byte-compile-from-buffer (current-buffer) "foo.el"))))
    (when outbuf
      (prog1
	  (with-current-buffer outbuf
	    (goto-char (point-min))
	    (search-forward " 'foo '(byte-code " nil t))
	(kill-buffer outbuf)))))

and its use is in dgnushack's form

(when (dgnushack-emacs-compile-defcustom-p)
  (maybe-fbind '(defined-colors face-attribute))
  (maybe-bind '(idna-program installation-directory)))

where maybe-[f]bind is declared in lpath.el, whose top level comment is

;; Shut up.

;-)

Bye,
Tassilo

Re: lexbind ready for merge

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

On 3/30/11 4:35 AM, Juanma Barranquero wrote:
> On Wed, Mar 30, 2011 at 06:10, Daniel Colascione
> <dan.colascione@gmail.com> wrote:
> 
>> Even if it's not particularly common, being consistent with Common Lisp and
>> having fewer special cases are good things.  Some people use constructs like
>> this to create module-private variables (which is a bad idea, but that
>> doesn't stop people doing it.)
> 
> Why a bad idea? It's a common idiom for private, persistent variables.

Well, it's a great idea if you're into that kind of thing.  Personally,
I feel that having the ability to peer inside a module's state and see
what's wrong is helpful, and truly private variables don't add much over
having a naming convention that discourages casual modification.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: lexbind ready for merge

[Juanma Barranquero]

On Wed, Mar 30, 2011 at 17:09, Daniel Colascione
<dan.colascione@gmail.com> wrote:

> Well, it's a great idea if you're into that kind of thing.

I suppose yes, I'm "into that kind of thing", then. What can you
expect from a guy whose languages of choice are Ada and Common Lisp?

> Personally,
> I feel that having the ability to peer inside a module's state and see
> what's wrong is helpful, and truly private variables don't add much over
> having a naming convention that discourages casual modification.

Having private variables does not preclude peering inside, you just
need accessors :-)

As per naming conventions discouraging modification... Some
users/programmers might see that as a challenge to overcome ;-)

But anyway, my question was because your "bad idea" comment seemed to
imply that private lexical variables were technically unsound, while
is just that you don't like the style. Fair enough.

    Juanma

Re: lexbind ready for merge

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Why on earth does Gnus need to call byte-compile-from-buffer?

It seems to be code dealing with something on XEmacs introduced in
2007.  I've now disabled the code when running under Emacs.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: lexbind ready for merge

[Tassilo Horn]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> Why on earth does Gnus need to call byte-compile-from-buffer?
>
> It seems to be code dealing with something on XEmacs introduced in
> 2007.  I've now disabled the code when running under Emacs.

Yup, I can confirm that git Gnus builds fine with lexbind emacs now.

Bye,
Tassilo

Re: lexbind: how to replace lexical-let approach to hide secrets

[Stefan Monnier]

> Slightly related: auth-source.el uses `lexical-let' to define lambda
> accessors for secrets (so printing them, for instance, won't show a
> password).  Will the secrets still be hidden in the lexbind branch?

Using lexbind, yes.  Using `let', no.

> And
> is there a smarter way to do it?

> Sample code:

> (let ((data "my secret"))
>   (lexical-let ((data data)) (lambda () data)))

When lexical-binding is set:

 (let ((data "my secret"))
   (lambda () data))

returns something like (closure ((data . "my secret") t) () data).
If you wan to hide the value, then use:

  (let ((data (let ((sym (make-symbol "foo")))
                (set sym "secret")
                sym)))
    (lambda () (symbol-value data)))

which is similar to the what lexical-let ends up doing.


        Stefan

Re: lexbind: how to replace lexical-let approach to hide secrets

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> When lexical-binding is set:
>
>  (let ((data "my secret"))
>    (lambda () data))
>
> returns something like (closure ((data . "my secret") t) () data).
> If you wan to hide the value, then use:
>
>   (let ((data (let ((sym (make-symbol "foo")))
>                 (set sym "secret")
>                 sym)))
>     (lambda () (symbol-value data)))
>
> which is similar to the what lexical-let ends up doing.

All of which, by necessity, contain the necessary material to get at the
secret in the bytecode/closure/funcell.  Where, of course, it is
reasonable easy to retrieve it.  And that is actually overkill, since
you can always just call that closure.

Anybody care to tell me what the point of this exercise is actually
supposed to be?

-- 
David Kastrup

Re: lexbind: how to replace lexical-let approach to hide secrets

[Daniel Colascione]

On 3/30/2011 2:56 PM, David Kastrup wrote:
> Stefan Monnier<monnier@iro.umontreal.ca>  writes:
>
>> When lexical-binding is set:
>>
>>   (let ((data "my secret"))
>>     (lambda () data))
>>
>> returns something like (closure ((data . "my secret") t) () data).
>> If you wan to hide the value, then use:
>>
>>    (let ((data (let ((sym (make-symbol "foo")))
>>                  (set sym "secret")
>>                  sym)))
>>      (lambda () (symbol-value data)))
>>
>> which is similar to the what lexical-let ends up doing.
>
> All of which, by necessity, contain the necessary material to get at the
> secret in the bytecode/closure/funcell.  Where, of course, it is
> reasonable easy to retrieve it.  And that is actually overkill, since
> you can always just call that closure.
>
> Anybody care to tell me what the point of this exercise is actually
> supposed to be?

The idea is to prevent secrets being accidentally revealed in 
backtraces, lisp evaluation results, and so on.

By the way: I didn't notice any secret erasure code in auth-source.el. 
Shouldn't someone somewhere do something like this?  I didn't see 
anything in alloc.c that looks like it'd clear an object's bytes upon 
deallocation.

(defun secure-erase-secret (my-secret)
   (dotimes (i (length my-secret)) (setf (aref my-secret i) 0)))

Re: lexbind ready for merge

[Stephen J. Turnbull]

Tassilo Horn writes:

 > >> AFAIKS, gnus doesn't use the foo.el file anyway, so basically that arg
 > >> can be removed.  But maybe the `byte-compile-from-buffer' definition in
 > >> [[S]X]Emacs requires exactly 2 arguments?  I don't know...

(defun byte-compile-from-buffer (byte-compile-inbuffer filename &optional eval)
  ;; buffer --> output-buffer, or buffer --> eval form, return nil

I don't know why FILENAME is not optional in XEmacs; it can be nil.
It can be used in non-trivial ways in that function.  Among others, it
is used in an error message, which would be confusing if the value
were nil.  For that reason, I personally would tend to oppose changing
the signature, to remind callers that null FILENAME is not without
drawbacks.

It's not clear to me what the semantics of the other uses of FILENAME
are, but since it's explicitly tested as a boolean at least once,
XEmacs is apparently prepared for it to be nil.

 > > Why on earth does Gnus need to call byte-compile-from-buffer?

"The Gnus that is the true Gnus calls byte-compiler-from-buffer.
Otherwise it would not be Gnus."

                                    -- larsi-tse, "Gnus Te Ching"

<wink />

Re: lexbind: how to replace lexical-let approach to hide secrets

[Ted Zlatanov]

On Wed, 30 Mar 2011 17:12:37 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> Slightly related: auth-source.el uses `lexical-let' to define lambda
>> accessors for secrets (so printing them, for instance, won't show a
>> password).  Will the secrets still be hidden in the lexbind branch?

SM> If you wan to hide the value, then use:

SM>   (let ((data (let ((sym (make-symbol "foo")))
SM>                 (set sym "secret")
SM>                 sym)))
SM>     (lambda () (symbol-value data)))

SM> which is similar to the what lexical-let ends up doing.

I wonder if the above approach can simply replace the `lexical-let'
macro in the lexbind branch.

On Wed, 30 Mar 2011 23:56:59 +0200 David Kastrup <dak@gnu.org> wrote: 

DK> All of which, by necessity, contain the necessary material to get at the
DK> secret in the bytecode/closure/funcell.  Where, of course, it is
DK> reasonable easy to retrieve it.  And that is actually overkill, since
DK> you can always just call that closure.

DK> Anybody care to tell me what the point of this exercise is actually
DK> supposed to be?

1) hide secrets so they are not printed when the auth-source return data
structure is returned (as Daniel says below)

2) provide some abstraction for secrets so we can use something more
secure when it's available in Emacs

On Wed, 30 Mar 2011 15:29:12 -0700 Daniel Colascione <dan.colascione@gmail.com> wrote: 

DC> The idea is to prevent secrets being accidentally revealed in
DC> backtraces, lisp evaluation results, and so on.

DC> By the way: I didn't notice any secret erasure code in
DC> auth-source.el. Shouldn't someone somewhere do something like this?  I
DC> didn't see anything in alloc.c that looks like it'd clear an object's
DC> bytes upon deallocation.

DC> (defun secure-erase-secret (my-secret)
DC>   (dotimes (i (length my-secret)) (setf (aref my-secret i) 0)))

password-cache.el does something like this in ELisp.  

IMHO this should be done by Emacs; the core should provide a way to tag
strings as "secret" so they are wiped on deallocation.  I think this
property should propagate when the string is copied.  

We can probably simulate some of this with closures but I think it would
be 100 times cleaner at the C level.  That way the consumer doesn't have
to wipe the secret, just release it (but he should still be able to
trigger the wipe explicitly if he knows he's done with the secret).
Also the strings can then be stored in an encoded representation so it's
not trivial to find them in memory.

Ted

Re: lexbind: how to replace lexical-let approach to hide secrets

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > IMHO this should be done by Emacs; the core should provide a way to tag
 > strings as "secret" so they are wiped on deallocation.

I don't see why this is better than the method already used, since you
would have to use a different call to make such strings.  In the end
it's up to the application to manage these secrets.

 > I think this property should propagate when the string is copied.

But what about the storage the string is copied from?  Really, keeping
secrets is up to the application.  I think this is overkill, and won't
really help naive users keep their secrets.

secret strings (was: lexbind: how to replace lexical-let approach to hide secrets)

[Ted Zlatanov]

On Fri, 01 Apr 2011 10:31:01 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> IMHO this should be done by Emacs; the core should provide a way to tag
>> strings as "secret" so they are wiped on deallocation.

SJT> I don't see why this is better than the method already used, since you
SJT> would have to use a different call to make such strings.

The consumer wouldn't have to explicitly track and clear such strings.
I think this is better and *safer* than asking the consumer to do it.

These strings would be made differently by the producer (the auth-source
API in this case) using some Emacs core functionality.  The consumer
doesn't know they are different from ordinary strings.  They print like
normal strings.  They simply get explicitly wiped on deallocation and
*maybe* get stored in an obfuscated way.

SJT> In the end it's up to the application to manage these secrets.

I strongly disagree that the consumer should have to wipe secrets when
done with them.  That simply shifts the burden of managing secrets
without easing it.

>> I think this property should propagate when the string is copied.

SJT> But what about the storage the string is copied from?

Obviously data has to come from somewhere.  It can come from the
environment and from files and from IPC and from process pipes.  Emacs
can provide functions that read directly from those sources into a
secret string.

Again, this is to manage wipe on deallocation and maybe provide
obfuscated storage, not a comprehensive security solution.  I have no
illusions that Emacs Lisp can ever be made secure and no desire to see
that, either.

SJT> I think this is overkill, and won't really help naive users keep
SJT> their secrets.

I'm not sure where "naive users" came into the picture and how we're
supposed to help them with this proposal.  It only benefits the
consumer, which usually is an application, and the producer, which in my
case is the auth-source API.  Users should not see a difference or care.

Hiding secrets from backtraces and printing is another matter.  That we
can do with `lexical-let' or the approach Stefan showed so I think it's
a solved problem.  I've changed the subject to reflect we're discussing
"secret strings" now, though the name is not very good.

Ted

secret strings (was: lexbind: how to replace lexical-let approach to hide secrets)

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > SJT> In the end it's up to the application to manage these secrets.
 > 
 > I strongly disagree that the consumer should have to wipe secrets when
 > done with them.  That simply shifts the burden of managing secrets
 > without easing it.

(defmacro with-secret-strings (variable-list &rest body)
  `(unwind-protect (progn ,@body)
     (mapc #'wipe-secret-string ,variable-list)))

Was that so hard?

 > Obviously data has to come from somewhere.  It can come from the
 > environment and from files and from IPC and from process pipes.  Emacs
 > can provide functions that read directly from those sources into a
 > secret string.

Sure.  One for every such source ....

 > Hiding secrets from backtraces and printing is another matter.  That we
 > can do with `lexical-let' or the approach Stefan showed so I think it's
 > a solved problem.  I've changed the subject to reflect we're discussing
 > "secret strings" now, though the name is not very good.

Well, I don't care about the name, but I don't see a use case where
the users are really protected.

Re: secret strings

[Ted Zlatanov]

On Fri, 01 Apr 2011 14:52:05 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
SJT> In the end it's up to the application to manage these secrets.
>> 
>> I strongly disagree that the consumer should have to wipe secrets when
>> done with them.  That simply shifts the burden of managing secrets
>> without easing it.

SJT> (defmacro with-secret-strings (variable-list &rest body)
SJT>   `(unwind-protect (progn ,@body)
SJT>      (mapc #'wipe-secret-string ,variable-list)))

SJT> Was that so hard?

I don't think that's the same thing.  We want to pass the
producer-generated data around and wipe it when the garbage collection
deallocates the memory.  But maybe I misunderstand something.

>> Hiding secrets from backtraces and printing is another matter.  That we
>> can do with `lexical-let' or the approach Stefan showed so I think it's
>> a solved problem.  I've changed the subject to reflect we're discussing
>> "secret strings" now, though the name is not very good.

SJT> Well, I don't care about the name, but I don't see a use case where
SJT> the users are really protected.

It's a *convenience* so the consumer doesn't have to wipe the secret
strings explicitly.  I'm not proposing a security model; the user
protection is only that there's a smaller chance an attacker would see
the secret strings in a memory image of the Emacs process.  If the
secret strings are stored in an encrypted or obfuscated way, the chance
becomes even smaller.

The alternative way to do the above would be at the Lisp level.  I think
that would be slower, less convenient (requiring timers or manual wipe
calls), and the chance of exposure would be greater.  But it's certainly
possible.  It's how password-cache.el does it.

Ted

Re: secret strings

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > I'm not proposing a security model; the user protection is only
 > that there's a smaller chance an attacker would see the secret
 > strings in a memory image of the Emacs process.

My point is, if you have no security model, why bother?

It is very unlikely that an attack on Emacs memory would reveal
"secret strings".  If somebody cares about that small chance, they're
either kidding themselves, or they have a security model that will
tell them to *ignore* the autowiping GC, and wipe themselves.

Cleanliness-is-next-to-***liness-ly y'rs,

Re: secret strings

[Stefan Monnier]

> I don't think that's the same thing.  We want to pass the
> producer-generated data around and wipe it when the garbage collection
> deallocates the memory.  But maybe I misunderstand something.

I don't think that's what you want: work done by finalizers should never
be significant (e.g. it's a bad idea to use finalizers to close
file-handles, or to wipe sensitive data).  If you want to wipe that
data, then do it explicitly with `clear-string', since the GC might
never collect it.


        Stefan

Re: secret strings

[Ted Zlatanov]

On Fri, 01 Apr 2011 23:38:20 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> I'm not proposing a security model; the user protection is only
>> that there's a smaller chance an attacker would see the secret
>> strings in a memory image of the Emacs process.

SJT> My point is, if you have no security model, why bother?

SJT> It is very unlikely that an attack on Emacs memory would reveal
SJT> "secret strings".  If somebody cares about that small chance, they're
SJT> either kidding themselves, or they have a security model that will
SJT> tell them to *ignore* the autowiping GC, and wipe themselves.

OK.  I'll buy that.  So how, then, does the the producer, the
auth-source API, encourage consumers to wipe their secrets?  Should it
set a timer (for a duration specified by the consumer) after which the
secret gets wiped and 'wiped is returned instead?  How can Emacs Lisp
and maybe the new lexbind features help make this as seamless as
possible for the consumer?

On Fri, 01 Apr 2011 10:59:20 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> I don't think that's the same thing.  We want to pass the
>> producer-generated data around and wipe it when the garbage collection
>> deallocates the memory.  But maybe I misunderstand something.

SM> I don't think that's what you want: work done by finalizers should never
SM> be significant (e.g. it's a bad idea to use finalizers to close
SM> file-handles, or to wipe sensitive data).  If you want to wipe that
SM> data, then do it explicitly with `clear-string', since the GC might
SM> never collect it.

OK, I understand.  See my question above.

Ted

Re: secret strings

[Stephen J. Turnbull]

Ted Zlatanov writes:

 > OK.  I'll buy that.  So how, then, does the the producer, the
 > auth-source API, encourage consumers to wipe their secrets?

That depends on the security model, it seems to me.  For some
purposes, ROT13, with no secret at all, is sufficient "security".  In
other cases, the user is given a secret to be used once (eg, a
temporary password).  In other cases, the user may never see the
secret at all (public key methods).

The problem, as I see it, is that the auth-source doesn't know what
the consumer is going to do with it, or how long the secret will
remain valid.  I don't really see how this is the auth-source's
business.

The `with-secret-strings' macro I suggested is the only fairly generic
kind of thing I can think of, but it's not really very general.

Re: secret strings

[Ted Zlatanov]

On Sat, 02 Apr 2011 01:14:16 +0900 "Stephen J. Turnbull" <stephen@xemacs.org> wrote: 

SJT> Ted Zlatanov writes:
>> OK.  I'll buy that.  So how, then, does the the producer, the
>> auth-source API, encourage consumers to wipe their secrets?

SJT> That depends on the security model, it seems to me.  For some
SJT> purposes, ROT13, with no secret at all, is sufficient "security".  In
SJT> other cases, the user is given a secret to be used once (eg, a
SJT> temporary password).  In other cases, the user may never see the
SJT> secret at all (public key methods).

OK.  All the code is already written to hide it in a lexical closure.  I
think it's possible at least to encode the secret and decode it on the
funcall.  It will, obviously, still be somewhere in memory but at least
not as visibly.  Then the consumer can use `with-secret-strings' in
their local scope.

SJT> The problem, as I see it, is that the auth-source doesn't know what
SJT> the consumer is going to do with it, or how long the secret will
SJT> remain valid.  I don't really see how this is the auth-source's
SJT> business.

It should at least try to hide secret data and help the consumer protect
the secrets from accidental revealing (thus the secrets closure it uses
currently).  Good neighbors can warn you if you leave your door open,
even if it's not strictly their business.

SJT> The `with-secret-strings' macro I suggested is the only fairly generic
SJT> kind of thing I can think of, but it's not really very general.

I'll put it in auth-source.el and suggest it in auth.texi; I'll also
crawl through the places that use auth-source and rewrite them to use
the macro.  So it will be useful--thank you for the suggestion.

Ted

Re: secret strings

[Stefan Monnier]

> think it's possible at least to encode the secret and decode it on the
> funcall.  It will, obviously, still be somewhere in memory but at least

Encoding/decoding is silly: it just hides the data, without storing it
securely, so it's no better than the current "hide it behind
a symbol&closure".  I.e. more work wasted for no benefit.  There are so
many ways the data gets copied in memory during evaluation that encoding
one string doesn't give you much guarantee that there isn't any copy of
its content still lying around somewhere (for the same reason I consider
clear-string to be only for use by the deluded).

> It should at least try to hide secret data and help the consumer protect
> the secrets from accidental revealing (thus the secrets closure it uses
> currently).  Good neighbors can warn you if you leave your door open,
> even if it's not strictly their business.

Note that instead of a closure, it can store the data in a symbol.
That provides the same kind of protection (printing just prints the
symbol), but is much more lightweight.


        Stefan

Re: secret strings

[Ted Zlatanov]

On Fri, 01 Apr 2011 16:34:23 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>> think it's possible at least to encode the secret and decode it on the
>> funcall.  It will, obviously, still be somewhere in memory but at least

SM> Encoding/decoding is silly: it just hides the data, without storing it
SM> securely, so it's no better than the current "hide it behind
SM> a symbol&closure".  I.e. more work wasted for no benefit.  There are so
SM> many ways the data gets copied in memory during evaluation that encoding
SM> one string doesn't give you much guarantee that there isn't any copy of
SM> its content still lying around somewhere (for the same reason I consider
SM> clear-string to be only for use by the deluded).

OK, if I can't store the data securely and Emacs can't help me do it,
then I guess I'll note it in the auth.texi manual and move on.

I hope you and the other Emacs developers will consider secret strings
in the future in the context of a larger discussion about security in Emacs.

>> It should at least try to hide secret data and help the consumer protect
>> the secrets from accidental revealing (thus the secrets closure it uses
>> currently).  Good neighbors can warn you if you leave your door open,
>> even if it's not strictly their business.

SM> Note that instead of a closure, it can store the data in a symbol.
SM> That provides the same kind of protection (printing just prints the
SM> symbol), but is much more lightweight.

True, but I think it's not a big difference (these are rarely created).
The closure has the nice property that it has to be called and thus we
can attach more behavior to it without affecting the consumer.

Ted