Byte compiler and eval-when-compile

Byte compiler and eval-when-compile

[David Engster]

Somehow the byte compiler got smart during the last days (I think it's
the change in rev. 110510). I now get a lot more 'function X might not
be defined at runtime' warnings than before. The reason seems to be that
it better checks if `eval-when-compile' was used properly. Which is fine
and all, but I now have the following problem:

In CEDET, we often use `require' statements in function bodies, like
this:

(defun test()
  (require 'eldoc)
  (message "%s" (eldoc-function-argstring '("foo" "bar"))))

The reason is simply to only do the require when it is actually needed,
so that startup time is reduced. Still, if you byte-compile the above,
you'll get a 'might not be defined at runtime' warning for
`eldoc-function-argstring'. I used to circumvent that problem by simply
doing

(eval-when-compile
  (require 'eldoc))

Before you scream at me: I *know* this is not what `eval-when-compile'
is for, but it has worked until a few days ago. It seems the
byte-compiler now sees that I'm actually using a function from eldoc,
but he still doesn't see that I'm requiring the package it in the
function body. Do I now really have to use `declare-function' for all
those cases?

-David

Re: Byte compiler and eval-when-compile

[Stefan Monnier]

> Somehow the byte compiler got smart during the last days (I think it's
> the change in rev. 110510).

Actually, it just recovered the smartness I added many years ago and
which got broken years ago as well.

> I now get a lot more 'function X might not be defined at runtime'
> warnings than before.

That's expected.

> In CEDET, we often use `require' statements in function bodies, like
> this:

> (defun test()
>   (require 'eldoc)
>   (message "%s" (eldoc-function-argstring '("foo" "bar"))))

> The reason is simply to only do the require when it is actually needed,
> so that startup time is reduced.
> Still, if you byte-compile the above, you'll get a 'might not be
> defined at runtime' warning for `eldoc-function-argstring'.

Yes, that's an annoying case.

> I used to circumvent that problem by simply doing
> (eval-when-compile
>   (require 'eldoc))
> Before you scream at me: I *know* this is not what `eval-when-compile'
> is for, but it has worked until a few days ago.  It seems the
> byte-compiler now sees that I'm actually using a function from eldoc,
> but he still doesn't see that I'm requiring the package it in the
> function body.

If you both do the (eval-when-compile (require 'eldoc)) and the
`require', the byte-compiler could be smarter indeed: when it sees the
inner `require' call, it could check load-history and mark all functions
defined therein as being fine for the current scope.

> Do I now really have to use `declare-function' for all those cases?

Currently, yes (or use an fboundp test, which you probably won't like
any better).

I think a good solution to such cases would be to add a `lazy-require':
when interpreted, it works like `require', but the byte-compiler will
turn it into a bunch of autoloads.

A simpler solution might be to provide a new (funcall-require PACKAGE
FUNCTION &rest ARGS), so you'd do

   (funcall-require 'eldoc #'eldoc-function-argstring '("foo" "bar"))

but the compiler could still be taught to check that
eldoc-function-argstring indeed exists in eldoc and accepts being called
with a single argument.

   
-- Stefan

Re: Byte compiler and eval-when-compile

[David Engster]

Stefan Monnier writes:
>> Somehow the byte compiler got smart during the last days (I think it's
>> the change in rev. 110510).
>
> Actually, it just recovered the smartness I added many years ago and
> which got broken years ago as well.

Just to make sure I understand this right: the byte-compiler will now
bark if you're using functions (instead of just macros and defsubsts)
from eval-when-compile'd required packages?

>> I used to circumvent that problem by simply doing
>> (eval-when-compile
>>   (require 'eldoc))
>> Before you scream at me: I *know* this is not what `eval-when-compile'
>> is for, but it has worked until a few days ago.  It seems the
>> byte-compiler now sees that I'm actually using a function from eldoc,
>> but he still doesn't see that I'm requiring the package it in the
>> function body.
>
> If you both do the (eval-when-compile (require 'eldoc)) and the
> `require', the byte-compiler could be smarter indeed: when it sees the
> inner `require' call, it could check load-history and mark all functions
> defined therein as being fine for the current scope.

Well, that would be more or less a legalization of the little hack I did
so far.

>> Do I now really have to use `declare-function' for all those cases?
>
> Currently, yes (or use an fboundp test, which you probably won't like
> any better).

Indeed. I will just declare the functions. :-)

> I think a good solution to such cases would be to add a `lazy-require':
> when interpreted, it works like `require', but the byte-compiler will
> turn it into a bunch of autoloads.

That would be neat, yes.

> A simpler solution might be to provide a new (funcall-require PACKAGE
> FUNCTION &rest ARGS), so you'd do
>
>    (funcall-require 'eldoc #'eldoc-function-argstring '("foo" "bar"))
>
> but the compiler could still be taught to check that
> eldoc-function-argstring indeed exists in eldoc and accepts being called
> with a single argument.

That's pretty much a combination of declare-function and require, isn't
it? At least it wouldn't save me any typing. :-)

-David

Re: Byte compiler and eval-when-compile

[Stefan Monnier]

>> Actually, it just recovered the smartness I added many years ago and
>> which got broken years ago as well.
> Just to make sure I understand this right: the byte-compiler will now
> bark if you're using functions (instead of just macros and defsubsts)
> from eval-when-compile'd required packages?

Yes, at least that's the intention: IIRC the implementation is pretty
approximate, so it may very well miss some such functions and complain
about unjustified ones.

>> If you both do the (eval-when-compile (require 'eldoc)) and the
>> `require', the byte-compiler could be smarter indeed: when it sees the
>> inner `require' call, it could check load-history and mark all functions
>> defined therein as being fine for the current scope.
> Well, that would be more or less a legalization of the little hack I did
> so far.

Your hack so far was not and isn't illegal.  It's simply not understood
by the byte-compiler, currently.

>> A simpler solution might be to provide a new (funcall-require PACKAGE
>> FUNCTION &rest ARGS), so you'd do
>> (funcall-require 'eldoc #'eldoc-function-argstring '("foo" "bar"))
>> but the compiler could still be taught to check that
>> eldoc-function-argstring indeed exists in eldoc and accepts being called
>> with a single argument.
> That's pretty much a combination of declare-function and require, isn't
> it?

It's a combination of require, declare-function, and funcall, so I think
it does save typing.


        Stefan

Re: Byte compiler and eval-when-compile

[David Engster]

Stefan Monnier writes:
>>> Actually, it just recovered the smartness I added many years ago and
>>> which got broken years ago as well.
>> Just to make sure I understand this right: the byte-compiler will now
>> bark if you're using functions (instead of just macros and defsubsts)
>> from eval-when-compile'd required packages?
>
> Yes, at least that's the intention: IIRC the implementation is pretty
> approximate, so it may very well miss some such functions and complain
> about unjustified ones.

Could you maybe take a look at the end of eieio.el? I don't understand
why the compiler warns that `eieio-update-lisp-imenu-expression' is
undefined (he actually always did that, not only since your last
change). I'm also not sure what's the fix here - why did you wrap the
check for (boundp 'lisp-imenu-generic-expression) in an
`eval-when-compile'?

[It's not really important, but I'm curious. :-) ]

-David

Re: Byte compiler and eval-when-compile

[Glenn Morris]

David Engster wrote:

> Could you maybe take a look at the end of eieio.el? I don't understand
> why the compiler warns that `eieio-update-lisp-imenu-expression' is
> undefined (he actually always did that, not only since your last
> change). I'm also not sure what's the fix here - why did you wrap the
> check for (boundp 'lisp-imenu-generic-expression) in an
> `eval-when-compile'?

The byte-compiler is not that smart and doesn't fully recognize defuns
that are not at top-level. This one is "hidden" inside an if.
You could eval-and-compile the whole thing (defun and all) to make the
warning go away. But lisp-imenu-generic-expression is defined in every
Emacs since at least 21.1, so I don't see the point of the if, unless
it's an XEmacs thing, in which case featurep 'xemacs can be cleaner.

Re: Byte compiler and eval-when-compile

[David Engster]

Glenn Morris writes:
> The byte-compiler is not that smart and doesn't fully recognize defuns
> that are not at top-level. This one is "hidden" inside an if.
> You could eval-and-compile the whole thing (defun and all) to make the
> warning go away. But lisp-imenu-generic-expression is defined in every
> Emacs since at least 21.1, so I don't see the point of the if, unless
> it's an XEmacs thing, in which case featurep 'xemacs can be cleaner.

Actually, I first wanted to remove that test, but then bzr anotate
showed his name for that line, which is why I thought there *must* be
some very smart reason that test is there. Now I see that he only fixed
a typo there recently, so I guess I can safely remove it. :-)

Sorry for the noise,
-David

Re: Byte compiler and eval-when-compile

[Glenn Morris]

It might be better to simply move the boundp test inside
eieio-update-lisp-imenu-expression and always define the function.

Actually, the whole thing looks both pointless and bogus anyway. The
right thing in Emacs would be to modify the definition of
lisp-imenu-generic-expression in lisp-mode.el directly to add defmethod,
and this has aleady been done. And the regexp optimization means that
"|advice\\" does not appear in lisp-imenu-generic-expression anyway,
since at least Emacs 22.1.

Re: Byte compiler and eval-when-compile

[David Engster]

Glenn Morris writes:
> Actually, the whole thing looks both pointless and bogus anyway.

You are right. I will remove it.

-David

Re: Byte compiler and eval-when-compile

[Stefan Monnier]

> Could you maybe take a look at the end of eieio.el? I don't understand
> why the compiler warns that `eieio-update-lisp-imenu-expression' is
> undefined (he actually always did that, not only since your last
> change).

Because there's a call to eieio-update-lisp-imenu-expression even tho
the function is not always defined (it's only defined within an `if').

Admittedly, it's obvious to the reader that the call can only happen
when the function is defined, but the byte-compiler is pretty dumb in
this respect and doesn't see the connection.

> I'm also not sure what's the fix here - why did you wrap the
> check for (boundp 'lisp-imenu-generic-expression) in an
> `eval-when-compile'?

I didn't wrap it in eval-when-compile.  The only change I installed was
to fix a typo ("list"->"lisp").


        Stefan