Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Alan Mackenzie]

Hello, Emacs.

I'm in Emacs 25.1.

I want to M-x compile-defun the function `edebug'.  When I attempt this,
I get the "warning" messages:

    Warning: Unused lexical variable `edebug-break'
    Warning: Unused lexical variable `edebug-global-break'
    Warning: Unused lexical variable `edebug-break-condition'
    Warning: Unused lexical variable `edebug-break-data'
    Warning: Unused lexical variable `edebug-breakpoints'

.  These variables are dynamic ones, declared earlier on in the file
like this:

    (defvar edebug-breakpoints)

, i.e. without initialising expressions.  I tried C-x C-e on each such
declaration, then repeating the compile-defun, to no avail.
lexical-binding is set in edebug.el.

Emacs should be able to compile successfully in such circumstances,
surely?.

What am I doing wrong, here?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Stefan Monnier]

> I want to M-x compile-defun the function `edebug'.

Looks like `compile-defun` is another function that needs to be changed
to use `eval-sexp-add-defvars`.


        Stefan

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Alan Mackenzie]

Hello, Stefan.

On Tue, Jan 03, 2017 at 01:35:19PM -0500, Stefan Monnier wrote:
> > I want to M-x compile-defun the function `edebug'.

> Looks like `compile-defun` is another function that needs to be changed
> to use `eval-sexp-add-defvars`.

It looks like you've already done that, possibly on 2015-10-29.  Looking
at the source (now I know what to look for), eval-sexp-add-defvars is
also in Emacs 25.1's source for compile-defun.  So, something else has
gone wrong.....

I think the something else might be in Fdefvar.  At its outermost level
Fdefvar contains:

###############################################
  (Lisp_Object args)
{
  Lisp_Object sym, tem, tail;

  sym = XCAR (args);
  tail = XCDR (args);

  if (CONSP (tail))
    {
......
      /* Do it before evaluating the initial value, for self-references.  */
      XSYMBOL (sym)->declared_special = 1;
......
    }
###############################################

, so the question becomes why is setting the declared_special flag done
inside the "if (CONSP (tail))" rather than outside?  I.e. why is it only
done when a variable has an initialisation?

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Stefan Monnier]

> It looks like you've already done that, possibly on 2015-10-29.  Looking
> at the source (now I know what to look for), eval-sexp-add-defvars is
> also in Emacs 25.1's source for compile-defun.

Oh, indeed.  Not sure why you're seeing what you're seeing, then.
At least on Emacs's master branch I can't seem to reproduce your problem
(don't have a fresh emacs-25 build to try it right now).

> , so the question becomes why is setting the declared_special flag done
> inside the "if (CONSP (tail))" rather than outside?  I.e. why is it only
> done when a variable has an initialisation?

Because this code is only relevant when you evaluate (defvar <foo>),
whereas here we're not evaluating it, we're only processing it for the
byte-compiler.

There's also the important difference that a (defvar <foo>) only
has effect for the code in the same file rather than having a global
effect, so you can do (defvar toto) and then use `toto` as
a dynamically-scoped variable in your file without wreaking havoc in all
other files which happen to also use `toto` as a (lexical) variable.


        Stefan

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Alan Mackenzie]

Hello, Stefan.

On Tue, Jan 03, 2017 at 04:48:15PM -0500, Stefan Monnier wrote:
> > It looks like you've already done that, possibly on 2015-10-29.  Looking
> > at the source (now I know what to look for), eval-sexp-add-defvars is
> > also in Emacs 25.1's source for compile-defun.

> Oh, indeed.  Not sure why you're seeing what you're seeing, then.
> At least on Emacs's master branch I can't seem to reproduce your problem
> (don't have a fresh emacs-25 build to try it right now).

For what it's worth, I can't reproduce the problem any more.  I must
have got my Emacs 25.1 into a strange state, somehow.

> > , so the question becomes why is setting the declared_special flag done
> > inside the "if (CONSP (tail))" rather than outside?  I.e. why is it only
> > done when a variable has an initialisation?

> Because this code is only relevant when you evaluate (defvar <foo>),
> whereas here we're not evaluating it, we're only processing it for the
> byte-compiler.

Understood.  Thanks.

> There's also the important difference that a (defvar <foo>) only
> has effect for the code in the same file rather than having a global
> effect, so you can do (defvar toto) and then use `toto` as
> a dynamically-scoped variable in your file without wreaking havoc in all
> other files which happen to also use `toto` as a (lexical) variable.

Ouch!  There seems to be a clash between symbols (which are global,
dynamic) and lexical variables (which are local and "invisible"): the
only way to mark a variable as lexical is at the global level.

However, I'm seeing something else strange (still on 25.1).  I do this:

(i) M-x load-library edebug.
(ii) visit edebug.el.
(iii) instrument `edebug' for edebugging (yes, I know ;-).
(iv) M-: (edebug).
(v) step through the function to just after the let bindings.
(vi) e (special-variable-p 'edebug-breakpoints).

This last returns nil.  This suggests edebug-breakpoints has been bound
as a lexical variable, rather than a dynamic one.  (There is a defvar
for it earlier in the file.)  This is surely not right.

What is going on here?  Is it, perhaps, a bug?

However, edebug.elc still seems to work, though I haven't successfully
mananged to run `edebug' recently.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Stefan Monnier]

> Ouch!  There seems to be a clash between symbols (which are global,
> dynamic) and lexical variables (which are local and "invisible"): the
> only way to mark a variable as lexical is at the global level.

Yes, there are some subtleties there.  The main issue is that every
occurrence of an identifier, whether a binding occurrence or a "use" can
be lexical or dynamic, but we don't want the coders to have to
constantly specify which uses and which bindings are lexical and which
are dynamic.  So we offer ways to "mark" symbols are "this one uses
dynamic scoping".  There are 2 such ways:
- *Evaluation* of (defvar VAR VAL . REST) marks this variable as being
  dynamically scoped in all the code that will be compiled in this session.
- *Compilation* of (defvar VAR . REST) marks this variable as being
  dynamically scoped in the current compilation unit.

> (vi) e (special-variable-p 'edebug-breakpoints).
>
> This last returns nil.  This suggests edebug-breakpoints has been bound
> as a lexical variable, rather than a dynamic one.  (There is a defvar
> for it earlier in the file.)  This is surely not right.

special-variable-p only indicates if (defvar VAR VAL . REST) was evaluated.


        Stefan

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Alan Mackenzie]

Hello, Stefan

On Wed, Jan 04, 2017 at 10:23:16AM -0500, Stefan Monnier wrote:
> > Ouch!  There seems to be a clash between symbols (which are global,
> > dynamic) and lexical variables (which are local and "invisible"): the
> > only way to mark a variable as lexical is at the global level.

> Yes, there are some subtleties there.  The main issue is that every
> occurrence of an identifier, whether a binding occurrence or a "use" can
> be lexical or dynamic, but we don't want the coders to have to
> constantly specify which uses and which bindings are lexical and which
> are dynamic.  So we offer ways to "mark" symbols are "this one uses
> dynamic scoping".  There are 2 such ways:
> - *Evaluation* of (defvar VAR VAL . REST) marks this variable as being
>   dynamically scoped in all the code that will be compiled in this session.
> - *Compilation* of (defvar VAR . REST) marks this variable as being
>   dynamically scoped in the current compilation unit.

> > (vi) e (special-variable-p 'edebug-breakpoints).

> > This last returns nil.  This suggests edebug-breakpoints has been bound
> > as a lexical variable, rather than a dynamic one.  (There is a defvar
> > for it earlier in the file.)  This is surely not right.

> special-variable-p only indicates if (defvar VAR VAL . REST) was evaluated.

So it would seem.  There is a bug in the elisp manual, which says that a
variable being declared by defvar will cause special-variable-p to
return t for it.   The doc string looks right, though far from helpful
for anybody who doesn't already know variable binding inside out, and
even a bit cryptic for those who do.

special-variable-p would appear to be a near useless function, since it
doesn't do what it's name says.

"Defining Variables" in the elisp manual states that "(defvar foo)" makes
foo a special variable; yet "(special-variable-p 'foo)" returns nil.  This
has got to be a bug, of some sort.

There appears to be no way of checking whether a variable's binding is
(or will be) lexical.

I can foresee quite a bit of confusion happening over all this, if it
hasn't happened already.

Am I missing something, or is this all really as incoherent as it
appears to me?

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Stefan Monnier]

> special-variable-p would appear to be a near useless function, since it
> doesn't do what it's name says.

It's useful for the compiler, but it's mostly internal, indeed.

> There appears to be no way of checking whether a variable's binding is
> (or will be) lexical.

In which context do you need/want to do that (I ask because how to do
it (and even if it can be done) depends on the details)?


        Stefan

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Alan Mackenzie]

Hello, Stefan.

On Wed, Jan 04, 2017 at 04:49:15PM -0500, Stefan Monnier wrote:
> > special-variable-p would appear to be a near useless function, since it
> > doesn't do what it's name says.

> It's useful for the compiler, but it's mostly internal, indeed.

OK.

> > There appears to be no way of checking whether a variable's binding is
> > (or will be) lexical.

> In which context do you need/want to do that (I ask because how to do
> it (and even if it can be done) depends on the details)?

I honestly don't know.  I presume that there will be code (other than
the byte compiler) which will want to make a distinction.  Maybe there's
not.  After (defvar foo), the byte compiler seems able to handle foo as
a dynamic variable.  I'm still trying to figure out how, given that that
flag denoting a special variable in the struct symbol doesn't get set.

I'm just thoroughly confused about the whole business of whether (defvar
foo) creates a special variable or not.  At the moment, I'm not even
quite sure any more exactly what a special variable is.  I knew this
morning.  :-(

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Stefan Monnier]

>> In which context do you need/want to do that (I ask because how to do
>> it (and even if it can be done) depends on the details)?
> I honestly don't know.  I presume that there will be code (other than
> the byte compiler) which will want to make a distinction.

There might, indeed, but it's very rare: in most cases rather than
determining which kind of binding will happen, you want to decide/impose
which binding will happen.

I'm not sure how Common-Lip handles it, but AFAICT there is no
equivalent to special-variable-p there, so they don't seem to offer
a way to find out whether a binding will be lexical or dynamic.

> After (defvar foo), the byte compiler seems able to handle foo as
> a dynamic variable.  I'm still trying to figure out how,

The byte-compiler *sees* the defvar, which lets it keep a note
internally (in a data-structure which keeps track of the current
context, which also includes information about which vars that are
let-bound in the surrounding code were bound lexically, so as to know
when we see a reference to var `foo' whether we should look for `foo' in
the dynamic context or in the lexical context (and if so, where in that
context)).


        Stefan

RE: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Drew Adams]

I _almost_ wanted to offer this, as a way to check dynamic vs lexical:

`symbol-value', wrapping with `condition-case', checking for
`void-variable' error and returning `nil' in that case.

But if there are a dynamic var and a lexical var with the same name,
`symbol-value' just sees the former - no real way to test the latter.

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Alan Mackenzie]

Hello, Stefan.

On Wed, Jan 04, 2017 at 05:26:53PM -0500, Stefan Monnier wrote:
> >> In which context do you need/want to do that (I ask because how to do
> >> it (and even if it can be done) depends on the details)?
> > I honestly don't know.  I presume that there will be code (other than
> > the byte compiler) which will want to make a distinction.

> There might, indeed, but it's very rare: in most cases rather than
> determining which kind of binding will happen, you want to decide/impose
> which binding will happen.

OK.

> I'm not sure how Common-Lip handles it, but AFAICT there is no
> equivalent to special-variable-p there, so they don't seem to offer
> a way to find out whether a binding will be lexical or dynamic.

> > After (defvar foo), the byte compiler seems able to handle foo as
> > a dynamic variable.  I'm still trying to figure out how,

> The byte-compiler *sees* the defvar, which lets it keep a note
> internally (in a data-structure which keeps track of the current
> context, which also includes information about which vars that are
> let-bound in the surrounding code were bound lexically, so as to know
> when we see a reference to var `foo' whether we should look for `foo' in
> the dynamic context or in the lexical context (and if so, where in that
> context)).

Thanks, I'll have a look at that sometime.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

RE: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Drew Adams]

Coming back to this old thread (2017).

sm> > special-variable-p only indicates if
sm> > (defvar VAR VAL . REST) was evaluated.

And not (defvar VAR), apparently?

am> So it would seem.  There is a bug in the elisp manual, which says that a
am> variable being declared by defvar will cause special-variable-p to
am> return t for it.   The doc string looks right, though far from helpful
am> for anybody who doesn't already know variable binding inside out, and
am> even a bit cryptic for those who do.
am> 
am> special-variable-p would appear to be a near useless function, since it
am> doesn't do what it's name says.
am> 
am> "Defining Variables" in the elisp manual states that "(defvar foo)" makes
am> foo a special variable; yet "(special-variable-p 'foo)" returns nil.  This
am> has got to be a bug, of some sort.
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
am> There appears to be no way of checking whether a variable's binding is
am> (or will be) lexical.
am> 
am> I can foresee quite a bit of confusion happening over all this, if it
am> hasn't happened already.
am> 
am> Am I missing something, or is this all really as incoherent as it
am> appears to me?

sm> It's useful for the compiler, but it's mostly internal, indeed.

Reading the doc, a user will have no idea that this
is "internal" mostly or not.  And s?he will have no
idea why (defvar foo) (special-variable-p 'foo)
returns nil.

Can we please fix the doc, to make things clear?

If this function is really (or "mostly"?) internal 
why document it in this way?  Neither the manual
nor the doc string gives a clue about this.

Can we please get the doc to not mislead about this,
at least in some way?

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Stefan Monnier]

> Can we please fix the doc, to make things clear?

I added:

    Note that since this is a function, it can only return
    non-@code{nil} for variables which are permanently special, but not
    for those that are only special in the current lexical scope.


-- Stefan

RE: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Drew Adams]

> > Can we please fix the doc, to make things clear?
> 
> I added:
> 
>   Note that since this is a function, it can only return
>   non-@code{nil} for variables which are permanently special, but not
>   for those that are only special in the current lexical scope.

Thanks very much for working on this.

But I'm afraid I don't know what a permanent versus
temporary special variable is.  And I haven't found
anything in the doc that has helped me with that.

I looked in Elisp nodes `Using Lexical Binding',
where `special-variable-p' is documented, and
`Defining Variables', where `defvar' is doc'd.
"special" variable seems to be defined as a term
in each of those nodes, BTW.  (I also looked in
node `Variable Scoping'.)

Node `Defining Variables' seems to say that using
`defvar' is enough to make a variable "always"
dynamic:

  The variable is marked as "special", meaning
  that it should always be dynamically bound
                 ^^^^^^
  (*note Variable Scoping::).

Is "always" something different from "permanent"?

Sorry, but I really don't understand what is meant
by what you wrote.  I also don't understand the
implication, "since this is a function".
              ^^^^^
But maybe that's because I don't understand
"permanently special" and "only special in the
current lexical scope".

AFAIK, in Common Lisp a variable is either special
or it's not.  If it is then its binding by `let'
is dynamic, and if it's not then its let-binding
is lexical. (No?)

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Stefan Monnier]

> But I'm afraid I don't know what a permanent versus
> temporary special variable is.
> And I haven't found
> anything in the doc that has helped me with that.

It's the term used in the part that describes `defvar`:

    Note that specifying a value, even @code{nil}, marks the variable as
    special permanently.  Whereas if @var{value} is omitted then the
    variable is only marked special locally (i.e.@: within the current
    lexical scope, or file if at the top-level).  This can be useful for
    suppressing byte compilation warnings, see @ref{Compiler Errors}.

> AFAIK, in Common Lisp a variable is either special
> or it's not.  If it is then its binding by `let'
> is dynamic, and if it's not then its let-binding
> is lexical. (No?)

Same for us.
Does Common Lisp offer something like `special-variable-p`?


        Stefan

RE: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Drew Adams]

> > But I'm afraid I don't know what a permanent versus
> > temporary special variable is.  And I haven't found
> > anything in the doc that has helped me with that.
> 
> It's the term used in the part that describes `defvar`:
> 
>     Note that specifying a value, even @code{nil},
>     marks the variable as special permanently.

Thanks.

[FWIW, I'm no expert in any of this.  I'm just trying
(1) to understand what the Emacs story is about this
and (2) to reconcile that with what I think about the
Common Lisp story about it.  Limited understanding of
both, no doubt.  Also, I'm not arguing for any change,
including a realignment with CL, if there is in fact a
difference.  I just want the doc to be make sense and
be clear about what Emacs does.  So far, it doesn't
seem clear to me.]

____

Permanently is not the opposite of locally.  Instead
of "permanently", shouldn't that text say "globally"?

I think what you mean there is what Common Lisp calls
"indefinite scope" - there are no limits on the scope
of a special variable, i.e., _where_ it can be
referenced.

(As contrasted with "indefinite extent", which is what
lexical bindings provide - no limit on _when_ the var
can be referenced - the binding can last as long as
there is still some reference to the variable.)

If you said "globally" then that text would no longer
seem to directly contradict the statement that "The
variable is marked as 'special', meaning that it
should _always_ be dynamically bound", which is in
the very same node, `Defining Variables'.

But although not seeming to directly contradict, even
that wouldn't really be right, I think.  More correct
would be to say "'special', meaning that it should be
bound dynamically _everywhere_".  Everywhere, not
always.

Dynamic binding, i.e., the binding of a special var,
is indefinite with respect to _where_ (it's global) -
what CL calls "indefinite scope".  And it's dynamic
with respect to _when_ (how long) - what CL calls
"dynamic extent".

Continuing...

>     if @var{value} is omitted then the variable is only
>     marked special locally (i.e.@: within the current
>     lexical scope, or file if at the top-level).

This feature doesn't exist in Common Lisp, AFAIK.    

In CL, `(defvar foo)' _proclaims_ `foo' to be special,
just like `(proclaim (special foo))', and that means
that it's special _everywhere_.

In CL, `defvar' always proclaims the variable special,
whether or not an initial value is provided.  Without
an initial value the variable has no global value and
it's called "unbound".

> > AFAIK, in Common Lisp a variable is either special
> > or it's not.  If it is then its binding by `let'
> > is dynamic, and if it's not then its let-binding
> > is lexical. (No?)
> 
> Same for us.

I don't see that, so far.  I think maybe you're saying
(and perhaps the doc is saying) that a var that is not
bound lexically in some particular context is special
_in that context_ (not everywhere), and that the same
variable can be bound either lexically or dynamically.

And I think that for CL, a variable that is ever
special is always and everywhere special, i.e., in
all contexts.  It's always bound dynamically.

When you say "Same for us", I'm guessing maybe you're
interpreting what I wrote with "special" having your
special ;-) interpretation as being context-dependent:

  "If it is [special] then its binding by `let'
   is dynamic, and if it's not [special] then its
   let-binding is lexical."

Am I guessing right wrt your interpretation of
"special"?  For you (and thus Emacs terminology),
can a var be sometimes special and sometimes not?

I don't think that's the case for the way CL uses
"special variable".

However, CL does say that about a given _name_ of
a variable.  The same name can refer in some places
to a special var and in other places to a lexical
var.  This is what CLTL says:

  At any given time either or both kinds of variable
  with the same name may have a current value.
  Which of the two kinds of variable is referred to
  when a symbol is evaluated depends on the context
  of the evaluation.

  The general rule is that if the symbol occurs
  textually within a program construct that creates
  a _binding_ for a variable of the same name, then
  the reference is to the variable specified by the
  binding; if no such program construct textually
  contains the reference, then it is taken to refer
  to the special variable of that name.

IOW, there can be a special variable `foo' and one
or more lexical variables `foo'.  They have the same
name but they're different variables.

I think the terminology you're using would say that
it's the same variable, and it's sometimes, or in
some places, special and at other times, or in other
places, lexical.

> Does Common Lisp offer something like `special-variable-p`?

Not that I know of.  But it also doesn't use `defvar'
with and without second arg to declare specialness.
`defvar' always declares specialness (proclaims,
actually).

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Stefan Monnier]

> Permanently is not the opposite of locally.  Instead
> of "permanently", shouldn't that text say "globally"?

Feel free to provide a patch that tries to clarify.  Part of the issue
is that (defvar VAR VAL) is just like (defvar VAR) when it's "processed"
by the byte-compiler: it won't affect bindings of VAR everywhere but
only within the corresponding lexical scope.  But when (defvar VAR VAL)
is *executed* then it subsequently will affect all interpreted code and
all the code that is subsequently compiled in that same Emacs session,
whereas *executing* (defvar VAR VAL) only affects the code subsequently
interpreted, and only within the current lexical scope.

> I think what you mean there is what Common Lisp calls
> "indefinite scope" - there are no limits on the scope
> of a special variable, i.e., _where_ it can be
> referenced.

That sounds about right, yes.

> ... the statement that "The variable is marked as 'special', meaning
> that it should _always_ be dynamically bound"

I'm not very happy with this statement, indeed: "always" isn't
quite right.

>>     if @var{value} is omitted then the variable is only
>>     marked special locally (i.e.@: within the current
>>     lexical scope, or file if at the top-level).
> This feature doesn't exist in Common Lisp, AFAIK.

AFAIK, in CL you can get something similar with (declare (special foo)).

> In CL, `defvar' always proclaims the variable special,
> whether or not an initial value is provided.

Indeed, Elisp is slightly different in this respect.

> And I think that for CL, a variable that is ever
> special is always and everywhere special, i.e., in
> all contexts.  It's always bound dynamically.

In Common Lisp

   (list
     (lambda (x)
       (let ((y x))
         (declare (special y))
         (lambda (z) (+ y z))))
     (lambda (x)
       (let ((y x))
         (lambda (z) (+ y z)))))

gives you two functions that don't behave the same because the `y`
binding in the first is dynamically scoped whereas that same `y` binding
is statically scoped in the second.

> When you say "Same for us", I'm guessing maybe you're interpreting
> what I wrote with "special" having your special ;-) interpretation as
> being context-dependent:

No, I'm saying that there's more to it than just either always or
never.  E.g. in Common Lisp after the system has *compiled* a chunk of
code with (defvar VAR ...), VAR may or may not be special (i.e. the
`defvar` must have affected the compiled code, but not necessarily the
state of the system that compiled the file).
[ In Elisp, the situation is somewhat similar, tho a bit
  less undefined.  ]

>> Does Common Lisp offer something like `special-variable-p`?
> Not that I know of.

Indeed.  And for good reasons: being a function it's hard to make it
provide the kind of semantic you expect from it, given the complexity
and scope-sensitivity of "specialness".

> But it also doesn't use `defvar' with and without second arg to
> declare specialness.

That's just a superficial syntactic difference.

Maybe the slightly deeper difference is that Common Lisp doesn't have
a way to say "all bindings of VAR under the current scope should be
dynamic", like (defvar VAR) does.  Instead it has (declare (special VAR))
which only talks about one particular binding.  But both Elisp's
`defvar` and CL's `declare` introduce the same issue that specialness is
not all-or-nothing and that a plain function like `special-variable-p`
can't give quite the answer we'd ideally want.


        Stefan

RE: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Drew Adams]

> > Permanently is not the opposite of locally.  Instead
> > of "permanently", shouldn't that text say "globally"?
> 
> Feel free to provide a patch that tries to clarify.

That will have to come from someone who understands
this better than I.  I'm hoping to understand it a
bit better from this thread, and then, hopefully,
from an updated doc.

> Part of the issue
> is that (defvar VAR VAL) is just like (defvar VAR) when it's "processed"
> by the byte-compiler: it won't affect bindings of VAR everywhere but
> only within the corresponding lexical scope.  But when (defvar VAR VAL)
> is *executed* then it subsequently will affect all interpreted code and
> all the code that is subsequently compiled in that same Emacs session,
> whereas *executing* (defvar VAR VAL) only affects the code subsequently
> interpreted, and only within the current lexical scope.

I'm not sure how that relates to permanently vs globally,
but OK, thanks.

> >>     if @var{value} is omitted then the variable is only
> >>     marked special locally (i.e.@: within the current
> >>     lexical scope, or file if at the top-level).
> > This feature doesn't exist in Common Lisp, AFAIK.
> 
> AFAIK, in CL you can get something similar with
> (declare (special foo)).

Yes, similar, but not exactly the same.  In CL, such
a declaration can only be used, I think, within
(certain) special forms.

In particular, I don't think there's any equivalent
for Emacs's file-local scope (other than putting the
declaration in a top-level `let', or similar - a
top-level `(locally (declare (special foo)))' perhaps).
But maybe I'm wrong about this.

The real point was about `defvar'.  CL's `defvar' is
not Emacs's, AFAIU.

> > In CL, `defvar' always proclaims the variable special,
> > whether or not an initial value is provided.
> 
> Indeed, Elisp is slightly different in this respect.

The key word there is "proclaim".  Function `proclaim':

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node104.html.

  "The function `proclaim' takes a DECL-SPEC as its
   argument and puts it into effect globally."

  "As a special case (so to speak), `proclaim' treats a
   `special' DECL-SPEC as applying to all bindings as
   well as to all references of the mentioned variables."

  "For example, after `(proclaim '(special x))' in a
   function definition such as `(defun example (x) ...)'
   the parameter X will be bound as a special (dynamic)
   variable rather than as a lexical (static) variable."

That's what CL `defvar' does - it declares the var to
be special, i.e., dynamic, everywhere.  There's no
shadowing (unlike the case for other proclamations).

  "`(defvar variable)' proclaims variable to be
   special (see `proclaim'), and may perform other
   system-dependent bookkeeping actions."

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node67.html

> > And I think that for CL, a variable that is ever
> > special is always and everywhere special, i.e., in
> > all contexts.  It's always bound dynamically.
> 
> In Common Lisp
> 
>    (list
>      (lambda (x)
>        (let ((y x))
>          (declare (special y))
>          (lambda (z) (+ y z))))
>      (lambda (x)
>        (let ((y x))
>          (lambda (z) (+ y z)))))
> 
> gives you two functions that don't behave the same because the `y`
> binding in the first is dynamically scoped whereas that same `y`
> binding is statically scoped in the second.

In CL parlance, those are not only not the same binding,
they are not the same variable `y'.  That's the point.

The same symbol (name) is used for those two different
`y' variables, but they're different vars (for CL).

The first variable named `y' is special, which means
that it's special everywhere (indefinite scope), and
it's duration is the duration of the executing code
that uses it (dynamic extent).

> I'm saying that there's more to it than just either always or
> never.  E.g. in Common Lisp after the system has *compiled* a chunk of
> code with (defvar VAR ...), VAR may or may not be special (i.e. the
> `defvar` must have affected the compiled code, but not necessarily the
> state of the system that compiled the file).

Again, I think maybe you're thinking of VAR being
occurrences of VAR as a symbol, as opposed to the
variable that's declared ("created" in a sense, but
not necessarily assigned a value - it might have
no value, i.e., in CL-speak, be "unbound").

The effect on the compiled code gets to the use
of the actual variable that was declared special.
And yes, occurrences of other, different variables
with the same name are not special.  Not only
"might not be special", but cannot be special -
there is only ever one special variable with a
given name.

(Again, this is my understanding.  It may be wrong.)

> [ In Elisp, the situation is somewhat similar, tho a bit
>   less undefined. ]
> 
> >> Does Common Lisp offer something like `special-variable-p`?
> > Not that I know of.
> 
> Indeed.  And for good reasons: being a function it's hard to make it
> provide the kind of semantic you expect from it, given the complexity
> and scope-sensitivity of "specialness".

I fully agree about that (as far as my understanding goes).
 
> > But it also doesn't use `defvar' with and without second arg to
> > declare specialness.
> 
> That's just a superficial syntactic difference.
> 
> Maybe the slightly deeper difference is that Common Lisp doesn't have
> a way to say "all bindings of VAR under the current scope should be
> dynamic", like (defvar VAR) does.  Instead it has (declare (special VAR))
> which only talks about one particular binding.

I don't think that's right, but perhaps I don't
follow you correctly.

CL's `defvar' _proclaims_ the variable to be
special (dynamic) - and the scope of that is
indefinite - unlimited.  It's not a lexical
scope (this binding or this file).

> But both Elisp's `defvar` and CL's `declare` introduce
> the same issue that specialness is not all-or-nothing

Dunno what that means, but I'm guessing that the
difference in how we're talking about this is
maybe the difference between how CL talks about
a "variable".

For the CL doc, a special variable is special
everywhere - no lexical scoping for it at all,
and its extent (lifetime) is dynamic.  And as a
consequence, there is only ever a single special
variable with any given name (symbol).  Its
bindings are always dynamic. Its scope remains
indefinite.

> and that a plain function like `special-variable-p`
> can't give quite the answer we'd ideally want.

On that we agree.
___


Which brings us back to the beginning: 

Maybe remove `special-variable-p' altogether?
Or if, as is apparently the case, it's needed for
the byte-compiler implementation, at least rename
it to indicate that it's internal, and remove it
from the manual?

I'm not trying to remake the Emacs design of
lexical scoping - or anything like that.  I'm
not even trying to change how it's presented in
the doc (though maybe from this thread you'll
have some slight changes you'd like to make).

This is really only about `special-variable-p'.
Both its name and its doc are currently quite
misleading.  And it doesn't seem like something
that users will ever use or should ever try to
use.  I've asked about that use case...

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Stefan Monnier]

> The real point was about `defvar'.  CL's `defvar' is
> not Emacs's, AFAIU.

That's a given.  But I thought the point was about Emacs's doc.

>> In Common Lisp
>> 
>>    (list
>>      (lambda (x)
>>        (let ((y x))
>>          (declare (special y))
>>          (lambda (z) (+ y z))))
>>      (lambda (x)
>>        (let ((y x))
>>          (lambda (z) (+ y z)))))
>> 
>> gives you two functions that don't behave the same because the `y`
>> binding in the first is dynamically scoped whereas that same `y`
>> binding is statically scoped in the second.
>
> In CL parlance, those are not only not the same binding,
> they are not the same variable `y'.  That's the point.

Not really, no: what the user wants to know is "when I write (let ((y
...)) ...), when will binding be dynamic or lexical".
Whether or not it's "the same variable" as some other
is irrelevant, or more specifically, it just shifts the problem.

> The first variable named `y' is special, which means
> that it's special everywhere (indefinite scope), and
> it's duration is the duration of the executing code
> that uses it (dynamic extent).

E.g. here it just shifts the problem from "which y will be lexical and
which will be dynamic" to "which y corresponds to this one-and-only
special var and which doesn't".


        Stefan

RE: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Drew Adams]

ping.

> Which brings us back to the beginning:
> 
> Maybe remove `special-variable-p' altogether?
> Or if, as is apparently the case, it's needed for
> the byte-compiler implementation, at least rename
> it to indicate that it's internal, and remove it
> from the manual?
> 
> I'm not trying to remake the Emacs design of
> lexical scoping - or anything like that.  I'm
> not even trying to change how it's presented in
> the doc (though maybe from this thread you'll
> have some slight changes you'd like to make).
> 
> This is really only about `special-variable-p'.
> Both its name and its doc are currently quite
> misleading.  And it doesn't seem like something
> that users will ever use or should ever try to
> use.  I've asked about that use case...

Re: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Stefan Monnier]

> Sorry, but I really don't understand what is meant
> by what you wrote.  I also don't understand the
> implication, "since this is a function".
>               ^^^^^

A function cannot access the lexical scope of its caller, so a function
cannot know if variable FOO is special in the caller's scope.


        Stefan

RE: Can't M-x compile-defun `edebug' because dynamic variables are falsely taken as lexical.

[Drew Adams]

> > Sorry, but I really don't understand what is meant
> > by what you wrote.  I also don't understand the
> > implication, "since this is a function".
> >               ^^^^^
> 
> A function cannot access the lexical scope of its caller, so a function
> cannot know if variable FOO is special in the caller's scope.

Right. (See my other reply, for how CL would
say that: it can't know whether the symbol
FOO refers to the special var FOO or a lexical
var FOO.)

I think it would be clearer to just say that
`special-variable-p' can't know whether its
arg refers to a special variable.

But I think that brings us back to what Alan
said: it's apparently useless, since that's
what it claims to do (even by just its name).

To which you replied that the function is
"mostly internal" (which is mostly vague).

So far, it sounds to me like this function
(1) should have an "internal"-advertising
name, (double hyphens) if any function should,
and (2) it shouldn't even be mentioned in the
manual.  Any mention of it seems like an
invitation to use it and be confused in its
use.

Or if it should be mentioned, it's not yet
clear to me what its semi-internal use case
is, and why that should be communicated to
users in the manual.  What will they (users)
do with it?