bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Alan Mackenzie]

Hello, Emacs.

With an up to date master version, start emacs -Q.

(i) Visit bug-compile-defvar.el, which looks like this:

#########################################################################
;; -*- lexical-binding: t -*-

(eval-and-compile
  (defun version-check ()
    (>= emacs-major-version 28)))

(defmacro acm-defvar (var)
  `(eval-when-compile
;     (when (version-check)
       (defvar ,var)
       (setq ,var emacs-major-version)
;       )
     ))

(acm-defvar l-s-p)

(eval-when-compile
  (message
   "\nl-s-p is %sin byte-compile-bound-variables"
   (if (memq 'l-s-p byte-compile-bound-variables)
       "" "not "))
  (message
   "l-s-p is %sbound"
   (if (boundp 'l-s-p) "" "not ")))

(defun acm-bind-l-s-p ()
 (let ((l-s-p emacs-major-version))
   (message "Nothing much\n")))
#########################################################################

The idea here is that the macro acm-defvar itself calls defvar, creating
a variable.

(ii) Do M-x byte-compile-file RET ~/bug-compile-defvar.el.
(iii) Note that this works properly, giving out the two messages, the
  first of which confirms that the new variable l-s-p has been entered
  into the list byte-compile-bound-variables.

(iv) Edit the buffer, removing both single semicolon comment markers, and
  save the file.
(v) Do M-x byte-compile-file RET ~/bug-compile-defvar.el again.
(vi) Note that now the new variable is NOT in
  byte-compile-bound-variables.  This is a bug.
(vii) Note also that there is a spurious compiler warning about l-s-p
  being an unused lexical variable in function acm-bind-l-s-p.  This is
  also a bug.

This bug doesn't occur on Emacs 28.  I have a feeling it was introduced
into Emacs 29 very recently.

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> (defmacro acm-defvar (var)
>   `(eval-when-compile
> ;     (when (version-check)
>        (defvar ,var)
>        (setq ,var emacs-major-version)
> ;       )
>      ))

(defvar VAR) does not "create a variable".  It just "locally" declares
that this identifier should use dynamic scoping when binding a variable.

Before the commit you mentioned, the above (defvar ,var) would sometimes
incorrectly escape its `when` subexpression and affect subsequent code,
which is why in the past the presence of `when` did not make much
difference whereas it now does.

Note also that for the same reason the above should arguably never work
at all since the effect of (defvar ,var) should not escape the
`eval-when-compile` subexpression.  A lot of code relies on this
misbehavior, tho, so we still support it.  But the better way to write
the above code is:

    (defmacro acm-defvar (var)
      `(progn
         (defvar ,var)
         (eval-when-compile
           (when (version-check)
             (setq ,var emacs-major-version)))))

> This bug doesn't occur on Emacs 28.
> I have a feeling it was introduced into Emacs 29 very recently.

The Emacs-28 behavior was a misbehavior, which got fixed accidentally
recently :-)


        Stefan

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Alan Mackenzie]

Hello, Stefan.

On Mon, Nov 07, 2022 at 08:07:28 -0500, Stefan Monnier wrote:
> > (defmacro acm-defvar (var)
> >   `(eval-when-compile
> > ;     (when (version-check)
> >        (defvar ,var)
> >        (setq ,var emacs-major-version)
> > ;       )
> >      ))

> (defvar VAR) does not "create a variable".

According to both its doc string and the Elisp manual it does.

> It just "locally" declares that this identifier should use dynamic
> scoping when binding a variable.

Why "locally"?  There's just one obarray involved, which is global.  The
manual says that, with defun, "The variable is marked as "special",
meaning that it should always be dynamically bound.".  In my example
program, l-s-p was created with defvar, yet later in the example, the
compiler bound it lexically.  There is a bug here, either in the code or
in the documentation.

> Before the commit you mentioned, the above (defvar ,var) would sometimes
> incorrectly escape its `when` subexpression and affect subsequent code,
> which is why in the past the presence of `when` did not make much
> difference whereas it now does.

What was incorrect about this "escaping"?  It would appear to match the
elisp manual, and its effect on subsequent code would be positive,
rather than negative, surely?

Why does the `when' now prevent l-s-p being marked as global?  In
particular, why is it not entered into the list
byte-compile-bound-variables?

> Note also that for the same reason the above should arguably never work
> at all since the effect of (defvar ,var) should not escape the
> `eval-when-compile` subexpression.

Why shouldn't it?  The defvar is evaluated at compilation time, should
create l-s-p, and should mark it as dynamically bound.  It does in fact
create the variable (see `message' output), but doesn't mark it as
dynamic when the evaluation is inside a conditional.

> A lot of code relies on this misbehavior, tho, so we still support it.

By what criterion is it "misbehaviour"?  It would appear to be what
defvar's doc string and manual entry say.

> But the better way to write the above code is:

>     (defmacro acm-defvar (var)
>       `(progn
>          (defvar ,var)
>          (eval-when-compile
>            (when (version-check)
>              (setq ,var emacs-major-version)))))

There are workarounds, yes.  But surely it would be better to fix the
bug.

> > This bug doesn't occur on Emacs 28.
> > I have a feeling it was introduced into Emacs 29 very recently.

> The Emacs-28 behavior was a misbehavior, which got fixed accidentally
> recently :-)

I feel I'm beginning to get repetitive here.  ;-)  But what was
incorrect about the Emacs 28 behaviour?  The current behaviour causes
confusing, incorrect warning messages to get output.  It cost me many
hours of labour till I finally found out the warnings came from a change
in the byte compiler.  Surely either the code or the documentation
should be fixed, so that other people don't have to go through the same.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> (defvar VAR) does not "create a variable".
>
> According to both its doc string and the Elisp manual it does.

Where?  I'm talking specifically about (defvar VAR) not about
`defvar` in general.

>> It just "locally" declares that this identifier should use dynamic
>> scoping when binding a variable.
> Why "locally"?  There's just one obarray involved, which is global.

No, we want to allow `defvar` to have a lexically-local effect.

>> But the better way to write the above code is:
>
>>     (defmacro acm-defvar (var)
>>       `(progn
>>          (defvar ,var)
>>          (eval-when-compile
>>            (when (version-check)
>>              (setq ,var emacs-major-version)))))
>
> There are workarounds, yes.  But surely it would be better to fix the
> bug.

It's not a workaround.  It's how the programmers say whether they want
the (defvar ,var) declaration to affect all the rest of the code or only
the code within the `when`.

Why put the (defvar ,var) within the `when` if not to limit
its scope?


        Stefan

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> (defvar VAR) does not "create a variable".
> According to both its doc string and the Elisp manual it does.

BTW, what I'm describing is how things have worked in `lexical-binding`
since Emacs-24, basically.  A few corner cases didn't quite obey it
(and I'm sure there are still a few remaining ones), but by and large
this is how it has worked for the last 10 years.


        Stefan

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Alan Mackenzie]

Hello, Stefan.

On Mon, Nov 07, 2022 at 16:29:15 -0500, Stefan Monnier wrote:
> >> (defvar VAR) does not "create a variable".

> > According to both its doc string and the Elisp manual it does.

> Where?  I'm talking specifically about (defvar VAR) not about
> `defvar` in general.

The entry in the Elisp manual starts off:

     This special form defines SYMBOL as a variable.  Note that SYMBOL
     is not evaluated; the symbol to be defined should appear explicitly
     in the `defvar' form.  The variable is marked as "special", meaning
     that it should always be dynamically bound (see Variable
     Scoping).

"This special form defines SYMBOL as a variable.".  There are no ifs,
buts, whens, or whenevers in that sentence.  This is, at best, somewhat
confusing.

Note also: "... that it should ALWAYS be dynamically bound ...".

I suppose you're now going to say that "defining" a variable is
different from "creating" it.  And that you're going to say "always"
doesn't mean always.

From defvar's doc string:

    The `defvar' form also declares the variable as "special",
    so that it is ALWAYS dynamically bound even if `lexical-binding' is t.

Here, again, you're going to say that "always" only means "sometimes", I
think.

> >> It just "locally" declares that this identifier should use dynamic
> >> scoping when binding a variable.
> > Why "locally"?  There's just one obarray involved, which is global.

> No, we want to allow `defvar` to have a lexically-local effect.

Why?  What's the use case?  This, again, is contradictory: a defvar with
a value has global effect, but without a value has only lexical scope.
This doesn't make sense.  Why should dynamically bound variables have
lexical scope?  Could it be the enthusiasm for lexical binding has burst
beyond its natural limits?

> >> But the better way to write the above code is:

> >>     (defmacro acm-defvar (var)
> >>       `(progn
> >>          (defvar ,var)
> >>          (eval-when-compile
> >>            (when (version-check)
> >>              (setq ,var emacs-major-version)))))

> > There are workarounds, yes.  But surely it would be better to fix the
> > bug.

> It's not a workaround.  It's how the programmers say whether they want
> the (defvar ,var) declaration to affect all the rest of the code or only
> the code within the `when`.

I think programmers just want defvar to behave sensibly and
consistently, and in accordance with the documentation.  The recently
introduced special handling of defvar without a value is not welcome.
Why should anybody want a dynamic variable later to be a lexically
scoped variable?.  

> Why put the (defvar ,var) within the `when` if not to limit
> its scope?

To limit when it will create the variable, according to the value of the
enclosing `if' form.  Just like if you do a (setq my-global some-value),
that has effect outside of the lexical scope.

In my test program, l-s-p is created with a dynamic binding, but later
on it got a confusing warning about it being unused as a lexically bound
variable.  Nobody wants this, surely?

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> The recently introduced special handling of defvar without a value is
> not welcome.

`defvar` without a value has been treated specially (more specifically
as a pure compiler-directive with no run-time effect) since at least
Emacs-19.34, so I'm not sure "recently" can be applied here.

> Why should anybody want a dynamic variable later to be
> a lexically scoped variable?.  

E.g. for historical reasons `calendar.el` needs to bind dynamically the
variable `date` around some calls to `eval` and to `run-hooks` because
the code evaluated therein has been defined and documented for may years
before to have access to important info via those dynbound vars.

Yet, we don't want `date` to be globally declared as being
always dynbound just because of that package's mishap, do we?

Similar situations affect other innocent-looking vars in other packages.


        Stefan

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Alan Mackenzie]

Hello, Stefan.

On Mon, Nov 07, 2022 at 16:31:32 -0500, Stefan Monnier wrote:
> >> (defvar VAR) does not "create a variable".
> > According to both its doc string and the Elisp manual it does.

> BTW, what I'm describing is how things have worked in `lexical-binding`
> since Emacs-24, basically.  A few corner cases didn't quite obey it
> (and I'm sure there are still a few remaining ones), but by and large
> this is how it has worked for the last 10 years.

OK, I see that now, having checked git logs and git blame.  There was a
bug report in 2018 about the documentation, which Noam Postavsky fixed.

The basic problem is that defvar is a dog's breakfast of a function.  In
fact, it's two distinct functions sharing a name.  These two functions
have little else to do with eachother.

And defvar appears to be the only function in Emacs where supplying nil
as an &optional argument has an effect different from omitting it.  This
is bad for anybody trying to learn Emacs Lisp.

Because defvar isn't a single function, it's impossible to document it
coherently without explicitly saying it's two functions.  I doubt the
powers that be would countenance such explicitness.  So the
documentation will continue to be confusing, and people like me will
continue to lose time sorting out what this hybrid function does.

I will alter the code in CC Mode which gave rise to this bug report.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> The basic problem is that defvar is a dog's breakfast of a function.  In
> fact, it's two distinct functions sharing a name.  These two functions
> have little else to do with eachother.

Yup.

> And defvar appears to be the only function in Emacs where supplying nil
> as an &optional argument has an effect different from omitting it.  This
> is bad for anybody trying to learn Emacs Lisp.

Not the only one, sadly (`declare-function` comes to mind :-)), but yes,
it's unusual.


        Stefan

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Stefan Kangas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The basic problem is that defvar is a dog's breakfast of a function.  In
>> fact, it's two distinct functions sharing a name.  These two functions
>> have little else to do with eachother.
>
> Yup.
>
>> And defvar appears to be the only function in Emacs where supplying nil
>> as an &optional argument has an effect different from omitting it.  This
>> is bad for anybody trying to learn Emacs Lisp.
>
> Not the only one, sadly (`declare-function` comes to mind :-)), but yes,
> it's unusual.

I guess defvar will have to stay as it is.  But would it make sense to
add a new `declare-variable' function, and then document the form

    (defvar <foo>)

to be considered deprecated?

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Eli Zaretskii]

> Cc: Alan Mackenzie <acm@muc.de>, 59057@debbugs.gnu.org
> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Thu, 24 Nov 2022 11:29:32 -0800
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> The basic problem is that defvar is a dog's breakfast of a function.  In
> >> fact, it's two distinct functions sharing a name.  These two functions
> >> have little else to do with eachother.
> >
> > Yup.
> >
> >> And defvar appears to be the only function in Emacs where supplying nil
> >> as an &optional argument has an effect different from omitting it.  This
> >> is bad for anybody trying to learn Emacs Lisp.
> >
> > Not the only one, sadly (`declare-function` comes to mind :-)), but yes,
> > it's unusual.
> 
> I guess defvar will have to stay as it is.  But would it make sense to
> add a new `declare-variable' function, and then document the form
> 
>     (defvar <foo>)
> 
> to be considered deprecated?

It makes no sense to me to deprecate (defvar FOO).  There are only
disadvantages down that path (massive annoyance or breakage of third-party
code, if nothing else).  Please don't.

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> I guess defvar will have to stay as it is.  But would it make sense to
> add a new `declare-variable' function, and then document the form
>
>     (defvar <foo>)
>
> to be considered deprecated?

I don't see any concrete benefit.  It's only a problem of documentation
and/or teaching programmers.  If we could go back to 1984 maybe it would
be worth it to make a different decision, but the cost of the change
doesn't seem worth the very minor (and hypothetical) benefit.


        Stefan

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Stefan Kangas]

> I don't see any concrete benefit.  It's only a problem of documentation
> and/or teaching programmers.  If we could go back to 1984 maybe it would
> be worth it to make a different decision, but the cost of the change
> doesn't seem worth the very minor (and hypothetical) benefit.

So should this bug report be closed?

bug#59057: Emacs 29. Byte compiler sometimes forgets about a defvar.

[Alan Mackenzie]

Hello, Stefan.

On Thu, Nov 24, 2022 at 22:12:07 +0100, Stefan Kangas wrote:
> > I don't see any concrete benefit.  It's only a problem of documentation
> > and/or teaching programmers.  If we could go back to 1984 maybe it would
> > be worth it to make a different decision, but the cost of the change
> > doesn't seem worth the very minor (and hypothetical) benefit.

> So should this bug report be closed?

The action of defvar can't really be changed any more, but the
documentation, which is rather confusing, could be improved.  I would
suggest leaving the bug open until this has been done.  Unfortunately, I
won't have much time to look at this in the next few days.

-- 
Alan Mackenzie (Nuremberg, Germany).