Has eval-and-compile changed in emacs 27?

Has eval-and-compile changed in emacs 27?

[Leo Liu]

Hi there,

A file
https://github.com/erlang/otp/blob/master/lib/tools/emacs/erldoc.el
suddenly has a compiler warning under emacs 27, which has no warnings
before.

    erldoc.el:535:1:Warning: the function ‘json-encode’ might not be
    defined at runtime.

I have (eval-and-compile (require 'json)) right before using json-encode
in a function body. Has anything changed in emacs 27 regarding
eval-and-compile? I checked NEWS but couldn't find anything.

Thanks,
Leo

Re: Has eval-and-compile changed in emacs 27?

[Eli Zaretskii]

> From: Leo Liu <sdl.web@gmail.com>
> Date: Thu, 25 Feb 2021 21:06:15 +0800
> 
>     erldoc.el:535:1:Warning: the function ‘json-encode’ might not be
>     defined at runtime.
> 
> I have (eval-and-compile (require 'json)) right before using json-encode
> in a function body. Has anything changed in emacs 27 regarding
> eval-and-compile? I checked NEWS but couldn't find anything.

eval-and-compile doesn't make sure the function is known at run time,
it only makes sure it is known at compile time.

(And yes, I think we modified the diagnostics there lately.)

Re: Has eval-and-compile changed in emacs 27?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> eval-and-compile doesn't make sure the function is known at run time,
> it only makes sure it is known at compile time.

That's `eval-when-compile', surely...  `eval-and-compile' also evaluates
at load time.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Has eval-and-compile changed in emacs 27?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 25 Feb 2021 16:06:45 +0100
> Cc: Leo Liu <sdl.web@gmail.com>, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > eval-and-compile doesn't make sure the function is known at run time,
> > it only makes sure it is known at compile time.
> 
> That's `eval-when-compile', surely...  `eval-and-compile' also evaluates
> at load time.

I guess my crystal ball said that Leo was using the former...

Sorry if my crystal ball is not clear enough today.

Re: Has eval-and-compile changed in emacs 27?

[Leo Liu]

On 2021-02-25 17:23 +0200, Eli Zaretskii wrote:
> I guess my crystal ball said that Leo was using the former...
> Sorry if my crystal ball is not clear enough today.

No problem at all ;)

Re: Has eval-and-compile changed in emacs 27?

[Basil L. Contovounesios]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> eval-and-compile doesn't make sure the function is known at run time,
>> it only makes sure it is known at compile time.
>
> That's `eval-when-compile', surely...  `eval-and-compile' also evaluates
> at load time.

Yes, but non-top-level requires don't usually make functions known at
run time; for that one needs autoload/declare-function.

Here's how I'd write this particular code, for example:

  (autoload 'json-encode "json")

  (defun
    ...
    (if (not json)
        (pp table (current-buffer))
      (defvar json-encoding-pretty-print)
      (let ((json-encoding-pretty-print t))
        (insert (json-encode table))))
    ...)

or alternatively:

  (defun
    ...
    (if (not json)
        (pp table (current-buffer))
      (require 'json)
      (defvar json-encoding-pretty-print)
      (declare-function json-encode "json" (object))
      (let ((json-encoding-pretty-print t))
        (insert (json-encode table))))
    ...

-- 
Basil

Re: Has eval-and-compile changed in emacs 27?

[Stefan Monnier]

> https://github.com/erlang/otp/blob/master/lib/tools/emacs/erldoc.el
> suddenly has a compiler warning under emacs 27, which has no warnings
> before.
>
>     erldoc.el:535:1:Warning: the function ‘json-encode’ might not be
>     defined at runtime.

That's weird.  I haven't downloaded the whole thing, but I tried to
reproduce it by byte-compiling a file containing:

    (defun sm-foo-from-erldoc (json table output)
      (with-temp-buffer
        (if (not json)
            (pp table (current-buffer))
          (eval-and-compile (require 'json))
          (let ((json-encoding-pretty-print t))
            (insert (json-encode table))))
        (unless (file-directory-p (file-name-directory output))
          (make-directory (file-name-directory output) t))
        (write-region nil nil output nil nil nil 'ask)))

and that doesn't give me any such warning, neither with Debian's 27.1 nor
with `master`.


        Stefan

Re: Has eval-and-compile changed in emacs 27?

[Stefan Monnier]

>>     erldoc.el:535:1:Warning: the function ‘json-encode’ might not be
>>     defined at runtime.
> That's weird.

Oh, I see: it's the (eval-when-compile (require 'url-parse)) because
apparently `url-parse` now ends up loading `json` somehow.


        Stefan

Re: Has eval-and-compile changed in emacs 27?

[Leo Liu]

On 2021-02-25 11:28 -0500, Stefan Monnier wrote:
> Oh, I see: it's the (eval-when-compile (require 'url-parse)) because
> apparently `url-parse` now ends up loading `json` somehow.

I was trying to fix some elisp warnings in
https://github.com/erlang/otp/pull/4542 but there was failed tests:

../../../otp/lib/erlang/lib/tools-3.5/emacs/erldoc.el:534:1:Error: the following functions might not be defined at runtime: json-encode, cl-reduce, cl-mapcan

Somehow I stumbled upon a fix by moving (eval-when-compile (require
'url-parse)) to after (require 'cl-lib) to get rid of the cl-reduce,
cl-mapcan warnings in Emacs 27. But I never understand why. Any ideas?

Re: Has eval-and-compile changed in emacs 27?

[Stefan Monnier]

> Somehow I stumbled upon a fix by moving (eval-when-compile (require
> 'url-parse)) to after (require 'cl-lib) to get rid of the cl-reduce,
> cl-mapcan warnings in Emacs 27. But I never understand why. Any ideas?

That one's easy: the way the warning works is that when we process
a `eval-when-compile` we look at the `load-history` to see the functions
that have been defined during execution of its body, and then we remember
those as "only available now but maybe not at runtime".

If that loaded `cl-lib`, then a subsequent (require 'cl-lib) will be
a no-op and won't "unremember" the corresponding functions.


        Stefan

Re: Has eval-and-compile changed in emacs 27?

[Leo Liu]

On 2021-02-25 17:29 -0500, Stefan Monnier wrote:
> That one's easy: the way the warning works is that when we process
> a `eval-when-compile` we look at the `load-history` to see the functions
> that have been defined during execution of its body, and then we remember
> those as "only available now but maybe not at runtime".
>
> If that loaded `cl-lib`, then a subsequent (require 'cl-lib) will be
> a no-op and won't "unremember" the corresponding functions.

This sounds like nightmare :(

Is this new in Emacs 27? What prompted the change?

I think when managing dependencies I want to be able to look locally at
what I put in a elisp file and be done with it.

The new behaviour means I also need to chase dependencies and their
dependencies...

Another point is subsequent (require 'cl-lib) provides stronger
guarantee but is ignored which produces spurious warnings.

Re: Has eval-and-compile changed in emacs 27?

[Stefan Monnier]

>> That one's easy: the way the warning works is that when we process
>> a `eval-when-compile` we look at the `load-history` to see the functions
>> that have been defined during execution of its body, and then we remember
>> those as "only available now but maybe not at runtime".
>>
>> If that loaded `cl-lib`, then a subsequent (require 'cl-lib) will be
>> a no-op and won't "unremember" the corresponding functions.
>
> This sounds like nightmare :(
>
> Is this new in Emacs 27?

No, it was new back when we added the "not known to exist at run-time",
i.e. a long time ago.

> What prompted the change?

We found it useful to try and warn the users when they did
(eval-when-compile (require FOO)) but FOO was also needed at run time.


        Stefan

Re: Has eval-and-compile changed in emacs 27?

[Leo Liu]

On 2021-02-25 22:54 -0500, Stefan Monnier wrote:
> No, it was new back when we added the "not known to exist at run-time",
> i.e. a long time ago.

But the erldoc.el has been compiling without warnings since 24.5 and it
suddenly changes in 27.

>> What prompted the change?
>
> We found it useful to try and warn the users when they did
> (eval-when-compile (require FOO)) but FOO was also needed at run time.

But it's been suggesting cl-reduce, cl-mapcan not defined at runtime
when I explicitly have (require 'cl-lib) in the file which seems
contradictory.

Re: Has eval-and-compile changed in emacs 27?

[Basil L. Contovounesios]

Leo Liu <sdl.web@gmail.com> writes:

> On 2021-02-25 22:54 -0500, Stefan Monnier wrote:
>> No, it was new back when we added the "not known to exist at run-time",
>> i.e. a long time ago.
>
> But the erldoc.el has been compiling without warnings since 24.5 and it
> suddenly changes in 27.
>
>>> What prompted the change?
>>
>> We found it useful to try and warn the users when they did
>> (eval-when-compile (require FOO)) but FOO was also needed at run time.
>
> But it's been suggesting cl-reduce, cl-mapcan not defined at runtime
> when I explicitly have (require 'cl-lib) in the file which seems
> contradictory.

What's new in Emacs 27 is auth-source now does

  (require 'cl-lib)

instead of

  (eval-when-compile (require 'cl-lib))

lisp/auth-source.el: Depend on cl-lib unconditionally
90a7cd073b 2019-11-26 14:00:25 +0100
https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=90a7cd073bfc7461e0bc824e9883499fe9026727

In turn, url-parse does

  (require 'auth-source)

and erldoc does

  (eval-when-compile (require 'url-parse))
  (require 'cl-lib)

This combination seems to confuse the byte-compiler.

-- 
Basil

Re: Has eval-and-compile changed in emacs 27?

[Basil L. Contovounesios]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> and erldoc does
>
>   (eval-when-compile (require 'url-parse))
>   (require 'cl-lib)
>
> This combination seems to confuse the byte-compiler.

One solution is to transpose those lines and load cl-lib before
url-parse.

-- 
Basil

Re: Has eval-and-compile changed in emacs 27?

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Date: Fri, 26 Feb 2021 09:33:13 +0000
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> >   (eval-when-compile (require 'url-parse))
> >   (require 'cl-lib)
> >
> > This combination seems to confuse the byte-compiler.
> 
> One solution is to transpose those lines and load cl-lib before
> url-parse.

With a comment about the importance of the order, please.

Re: Has eval-and-compile changed in emacs 27?

[Leo Liu]

On 2021-02-26 09:21 +0000, Basil L. Contovounesios wrote:
> What's new in Emacs 27 is auth-source now does
>
>   (require 'cl-lib)
>
> instead of
>
>   (eval-when-compile (require 'cl-lib))
>
> lisp/auth-source.el: Depend on cl-lib unconditionally
> 90a7cd073b 2019-11-26 14:00:25 +0100
> https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=90a7cd073bfc7461e0bc824e9883499fe9026727
>
> In turn, url-parse does
>
>   (require 'auth-source)
>
> and erldoc does
>
>   (eval-when-compile (require 'url-parse))
>   (require 'cl-lib)
>
> This combination seems to confuse the byte-compiler.

Great discovery. It is sad that this is happening. Will wait to see if
others get annoyed by this inadequacy.

Re: Has eval-and-compile changed in emacs 27?

[Stefan Monnier]

> Great discovery. It is sad that this is happening. Will wait to see if
> others get annoyed by this inadequacy.

I doubt waiting will make much of a difference: this problem
exists since commit ea4b0ca303f8ac30f87fd2ee905a6db53951c636
21 years ago.


        Stefan

Re: Has eval-and-compile changed in emacs 27?

[Stefan Monnier]

>> No, it was new back when we added the "not known to exist at run-time",
>> i.e. a long time ago.
> But the erldoc.el has been compiling without warnings since 24.5 and it
> suddenly changes in 27.

That's apparently a consequence of auth-source (`require`d by
`url-parse`) now `require`ing `json`.

>> We found it useful to try and warn the users when they did
>> (eval-when-compile (require FOO)) but FOO was also needed at run time.
> But it's been suggesting cl-reduce, cl-mapcan not defined at runtime
> when I explicitly have (require 'cl-lib) in the file which seems
> contradictory.

Yes, obviously the technique currently used doesn't always get it right.
Patches welcome ;-)


        Stefan

Re: Has eval-and-compile changed in emacs 27?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Oh, I see: it's the (eval-when-compile (require 'url-parse)) because
  > apparently `url-parse` now ends up loading `json` somehow.

It's not a good thing for a package to load more other packages
unnecessarily.  If it happens just once, it is not a big deal.
But if many libraries do it, it could lead to a cascade of bloat.

Does url-parse need json all the time, or only in rare special cases?

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Has eval-and-compile changed in emacs 27?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Fri, 26 Feb 2021 01:36:52 -0500
> Cc: sdl.web@gmail.com, emacs-devel@gnu.org
> 
> It's not a good thing for a package to load more other packages
> unnecessarily.  If it happens just once, it is not a big deal.
> But if many libraries do it, it could lead to a cascade of bloat.
> 
> Does url-parse need json all the time, or only in rare special cases?

I don't see json loaded by any file in the lisp/url directory, so it
must be some indirect load.  The easiest way of finding out who loads
it would be to run Emacs under GDB with a breakpoint on Fload.

Re: Has eval-and-compile changed in emacs 27?

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Richard Stallman <rms@gnu.org>
>> Date: Fri, 26 Feb 2021 01:36:52 -0500
>> Cc: sdl.web@gmail.com, emacs-devel@gnu.org
>> 
>> It's not a good thing for a package to load more other packages
>> unnecessarily.  If it happens just once, it is not a big deal.
>> But if many libraries do it, it could lead to a cascade of bloat.
>> 
>> Does url-parse need json all the time, or only in rare special cases?
>
> I don't see json loaded by any file in the lisp/url directory, so it
> must be some indirect load.  The easiest way of finding out who loads
> it would be to run Emacs under GDB with a breakpoint on Fload.

url-parse.el has only three requires:

  (require 'url-vars)
  (require 'auth-source)
  (eval-when-compile (require 'cl-lib))

auth-source.el has a known JSON backend, and indeed it loads:

  (require 'json)
  (require 'password-cache)

  (require 'cl-lib)
  (require 'eieio)

-- 
Basil

Re: Has eval-and-compile changed in emacs 27?

[Tomas Hlavaty]

On Fri 26 Feb 2021 at 09:54, Eli Zaretskii <eliz@gnu.org> wrote:
> The easiest way of finding out who loads it would be to run Emacs
> under GDB with a breakpoint on Fload.

does trace-function not work on C functions?

Re: Has eval-and-compile changed in emacs 27?

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: emacs-devel@gnu.org
> Date: Sat, 27 Feb 2021 01:34:11 +0100
> 
> On Fri 26 Feb 2021 at 09:54, Eli Zaretskii <eliz@gnu.org> wrote:
> > The easiest way of finding out who loads it would be to run Emacs
> > under GDB with a breakpoint on Fload.
> 
> does trace-function not work on C functions?

It's limited for tracing those; see "Restrictions" in the trace.el's
commentary.

More importantly, it is hard or even impossible to use this when
looking for loads that happen at startup time, which is what this
discussion is about.  (Also, trace.el is much heavier than a single
breakpoint.)