cl-byte-compile-compiler-macro

cl-byte-compile-compiler-macro

[Kim F. Storm]

>From time to time, I see this error pop up in a "*Compile Log*" buffer:

Error: Symbol's function definition is void: cl-byte-compile-compiler-macro


Latest when I did C-x v g on an emacs source file.

Have anybody else seen this?

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: cl-byte-compile-compiler-macro

[Glenn Morris]

Kim F. Storm wrote:

> From time to time, I see this error pop up in a "*Compile Log*" buffer:
>
> Error: Symbol's function definition is void: cl-byte-compile-compiler-macro
>
>
> Latest when I did C-x v g on an emacs source file.
>
> Have anybody else seen this?

Yes, I see this the first time (in any emacs session) I use C-x v = on
a file under CVS control. I don't see it with `emacs -q' though.

Re: cl-byte-compile-compiler-macro

[Richard Stallman]

    Yes, I see this the first time (in any emacs session) I use C-x v = on
    a file under CVS control. I don't see it with `emacs -q' though.

Can you track down which customization causes it?

Re: cl-byte-compile-compiler-macro

[Glenn Morris]

Richard Stallman wrote:

>     Yes, I see this the first time (in any emacs session) I use C-x v = on
>     a file under CVS control. I don't see it with `emacs -q' though.
>
> Can you track down which customization causes it?

For me, commenting out the line "(require 'tramp)" in my .emacs stops
this error appearing on my first call to vc-diff. If I start emacs
with -q, require tramp, and then use vc-diff, I don't see the error,
however; so it's not quite that simple. tramp.elc does at least
contain "cl-byte-compile-compiler-macro".

Re: cl-byte-compile-compiler-macro

[Glenn Morris]

I have reduced it to this (confusing) recipe:

  emacs -q --no-site-file -l foo.el changed_file_under_CVS
  C-x v =

where foo.el contains:

  (add-to-list 'load-path "/usr/share/emacs21/site-lisp")
  (add-to-list 'load-path "/usr/share/emacs21/site-lisp/auctex")
  (require 'font-latex)
  (require 'tramp)

This is on an up-to-date Debian testing system, where font-latex is
provided by the Debian auctex package.

Re: cl-byte-compile-compiler-macro

[Andreas Schwab]

Glenn Morris <rgm@gnu.org> writes:

> Richard Stallman wrote:
>
>>     Yes, I see this the first time (in any emacs session) I use C-x v = on
>>     a file under CVS control. I don't see it with `emacs -q' though.
>>
>> Can you track down which customization causes it?
>
> For me, commenting out the line "(require 'tramp)" in my .emacs stops
> this error appearing on my first call to vc-diff. If I start emacs
> with -q, require tramp, and then use vc-diff, I don't see the error,
> however; so it's not quite that simple. tramp.elc does at least
> contain "cl-byte-compile-compiler-macro".

I think the problem is that (defstruct tramp-file-name) needs to be moved
before its first use (in with-parsed-tramp-file-name).

Can you still reproduce it when you apply this patch?

--- lisp/net/tramp.el.~1.89.~	2006-02-13 11:08:07.000000000 +0100
+++ lisp/net/tramp.el	2006-03-25 15:17:34.000000000 +0100
@@ -1995,6 +1995,8 @@ own implementation."
    ((fboundp 'point-at-eol) 	 (funcall (symbol-function 'point-at-eol)))
    (t (save-excursion (end-of-line) (point)))))
 
+(defstruct tramp-file-name multi-method method user host localname)
+
 (defmacro with-parsed-tramp-file-name (filename var &rest body)
   "Parse a Tramp filename and make components available in the body.
 
@@ -6866,8 +6868,6 @@ Not actually used.  Use `(format \"%o\" 
 ;; internal data structure.  Convenience functions for internal
 ;; data structure.
 
-(defstruct tramp-file-name multi-method method user host localname)
-
 (defun tramp-tramp-file-p (name)
   "Return t iff NAME is a tramp file."
   (save-match-data

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: cl-byte-compile-compiler-macro

[Richard Stallman]

    If I start emacs
    with -q, require tramp, and then use vc-diff, I don't see the error,
    however; so it's not quite that simple. tramp.elc does at least
    contain "cl-byte-compile-compiler-macro".

That sounds like a bug in tramp--or in CL.  Can you see what generates
the call to it?  I think cl-byte-compile-compiler-macro is supposed
to be called during byte compilation.  I don't think a call to it
should be in tramp.elc.

Re: cl-byte-compile-compiler-macro

[Kim F. Storm]

Richard Stallman <rms@gnu.org> writes:

>     If I start emacs
>     with -q, require tramp, and then use vc-diff, I don't see the error,
>     however; so it's not quite that simple. tramp.elc does at least
>     contain "cl-byte-compile-compiler-macro".
>
> That sounds like a bug in tramp--or in CL.  Can you see what generates
> the call to it?  I think cl-byte-compile-compiler-macro is supposed
> to be called during byte compilation.  I don't think a call to it
> should be in tramp.elc.
>

If cl-byte-compile-compiler-macro shouldn't be in tramp.elc, it
probably shouldn't be in any other *.elc files either.

However, grepping for cl-byte-compile-compiler-macro in all *.elc
files yields this result:

Binary file ./gnus/nnmaildir.elc matches
Binary file ./erc/erc-backend.elc matches
Binary file ./erc/erc.elc matches
Binary file ./emacs-lisp/bytecomp.elc matches
Binary file ./emacs-lisp/cl-macs.elc matches
Binary file ./emacs-lisp/ewoc.elc matches
Binary file ./net/tramp.elc matches
Binary file ./progmodes/ebrowse.elc matches
Binary file ./progmodes/flymake.elc matches
Binary file ./textmodes/sgml-mode.elc matches
Binary file ./textmodes/tex-mode.elc matches
Binary file ./cvs-status.elc matches
Binary file ./pcvs-info.elc matches
Binary file ./pcvs-util.elc matches
Binary file ./uniquify.elc matches

Except for bytecomp.elc and cl-macs.elc, this looks suspicious.

-- 
Kim F. Storm <storm@cua.dk> http://www.cua.dk

Re: cl-byte-compile-compiler-macro

[Richard Stallman]

    However, grepping for cl-byte-compile-compiler-macro in all *.elc
    files yields this result:

I agree it is suspicious.  On the other hand, it occurs to me that
maybe it is not a bug--maybe this is how defstruct accessor macros
are implemented.  Could that be it?

If so, then (1) that needs to be documented, and (2) maybe the
function needs to be moved to subr.el.

Re: cl-byte-compile-compiler-macro

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

>     However, grepping for cl-byte-compile-compiler-macro in all *.elc
>     files yields this result:
>
> I agree it is suspicious.  On the other hand, it occurs to me that
> maybe it is not a bug--maybe this is how defstruct accessor macros
> are implemented.  Could that be it?

The symbol is only used as a constant.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: cl-byte-compile-compiler-macro

[Stefan Monnier]

>> However, grepping for cl-byte-compile-compiler-macro in all *.elc
>> files yields this result:
>> 
>> I agree it is suspicious.  On the other hand, it occurs to me that
>> maybe it is not a bug--maybe this is how defstruct accessor macros
>> are implemented.  Could that be it?

> The symbol is only used as a constant.

AFAIK the symbol is a used as a symbol property.  It tells the byte-compiler
that a function call to the function bound to the corresponding symbol can
be optimized in some cases and that the way to optimize it to to pass it to
the cl-byte-compile-compiler-macro property value (it's a so-called
"compiler macro").

It's used for example to turn a call to (list* a b c) into (cons a (cons
b c)), i.e. in this case a mix of inlining and unrolling.
And yes, it's quite likely that defstruct accessors define compiler macros
(I believe that CL's defsubst* are implemented using this same mechanism).


        Stefan

Re: cl-byte-compile-compiler-macro

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> AFAIK the symbol is a used as a symbol property.  It tells the byte-compiler
> that a function call to the function bound to the corresponding symbol can
> be optimized in some cases and that the way to optimize it to to pass it to
> the cl-byte-compile-compiler-macro property value (it's a so-called
> "compiler macro").

As I already explained, there is a used-before-defined issue in tramp.el
that appears to trigger the byte-compiler during runtime.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: cl-byte-compile-compiler-macro

[Richard Stallman]

    AFAIK the symbol is a used as a symbol property.  It tells the byte-compiler
    that a function call to the function bound to the corresponding symbol can
    be optimized in some cases and that the way to optimize it to to pass it to
    the cl-byte-compile-compiler-macro property value (it's a so-called
    "compiler macro").

Right.  So compiling a call to that macro will crash unless cl is
loaded.

I think we should move compiler-macroexpand and
cl-byte-compile-compiler-macro from cl-macs.el into subr.el.  It looks
like that will make the compilation of these macro calls work properly,
and it is rather painless.

But these functions need to get doc strings that explain what they do,
and perhaps some comments as well.

Re: cl-byte-compile-compiler-macro

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

>     AFAIK the symbol is a used as a symbol property.  It tells the byte-compiler
>     that a function call to the function bound to the corresponding symbol can
>     be optimized in some cases and that the way to optimize it to to pass it to
>     the cl-byte-compile-compiler-macro property value (it's a so-called
>     "compiler macro").
>
> Right.  So compiling a call to that macro will crash unless cl is
> loaded.

You need to load cl anyway when byte compiling a call to defstruct, and
then cl-macs (which defines byte-compile-compiler-macro) is autoloaded.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: cl-byte-compile-compiler-macro

[Glenn Morris]

Andreas Schwab wrote:

> I think the problem is that (defstruct tramp-file-name) needs to be moved
> before its first use (in with-parsed-tramp-file-name).
>
> Can you still reproduce it when you apply this patch?

I'm afraid that yes, I can (ie the patch seems to have no effect).

Re: cl-byte-compile-compiler-macro

[Glenn Morris]

Richard Stallman wrote:

> Right.  So compiling a call to that macro will crash unless cl is
> loaded.

When I explicitly (require 'cl), there is no error.

> I think we should move compiler-macroexpand and
> cl-byte-compile-compiler-macro from cl-macs.el into subr.el.  It looks
> like that will make the compilation of these macro calls work properly,
> and it is rather painless.

I tried that. I got a new Compile-Log error, this time about
cl-defsubst-expand. So I moved that too. Then I got an error about
cl-simple-exprs-p. Then I gave up. :)

Re: cl-byte-compile-compiler-macro

[Richard Stallman]

    You need to load cl anyway when byte compiling a call to defstruct, and
    then cl-macs (which defines byte-compile-compiler-macro) is autoloaded.

It could be a problem that compiling a call to these macros loads CL.
I am not sure; it is on the borderline.

Does interpreting a call to these defstruct accessor macros also load CL?
That would definitely be a problem.

Re: cl-byte-compile-compiler-macro

[Richard Stallman]

[I sent this message a week ago but did not get a response.
Could we get the discussion moving again?]

    You need to load cl anyway when byte compiling a call to defstruct, and
    then cl-macs (which defines byte-compile-compiler-macro) is autoloaded.

It could be a problem that compiling a call to these macros loads CL.
I am not sure; it is on the borderline.

Does interpreting a call to these defstruct accessor macros also load CL?
That would definitely be a problem.

Re: cl-byte-compile-compiler-macro

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

> It could be a problem that compiling a call to these macros loads CL.
> I am not sure; it is on the borderline.
>
> Does interpreting a call to these defstruct accessor macros also load CL?

defstruct is defined in cl-macs, so you can't use it without loading cl.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: cl-byte-compile-compiler-macro

[Stefan Monnier]

> Does interpreting a call to these defstruct accessor macros also load CL?

It definitely doesn't load CL.  Some uses may require CL to be loaded for
them to work, but I think all such uses are pretty obvious (e.g. when using
`setf').


        Stefan

Re: cl-byte-compile-compiler-macro

[Richard Stallman]

    > It could be a problem that compiling a call to these macros loads CL.
    > I am not sure; it is on the borderline.
    >
    > Does interpreting a call to these defstruct accessor macros also load CL?

    defstruct is defined in cl-macs, so you can't use it without loading cl.

We are miscommunicating.

We have a file foo.el that uses defstruct and defines an accessor,
foo-bar.  Loading foo.elc will load the compiled definition of
foo-bar.

If another file, quux.el, does (require 'foo), it will load the
compiled definition of foo-bar.  So there are two questions:

When I load quux.el, and run it interpreted, does it need to load
CL in order to process the call to foo-bar?  (Stefan says no.)

When I compile quux.el, does that need CL to be loaded?
(Assume quux.el does not load CL itself.)

Re: cl-byte-compile-compiler-macro

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

> When I load quux.el, and run it interpreted, does it need to load
> CL in order to process the call to foo-bar?  (Stefan says no.)

I agree with Stefan.

> When I compile quux.el, does that need CL to be loaded?
> (Assume quux.el does not load CL itself.)

I think it does, since the accessor function has a byte-compile property
that points to function in cl-macs (cl-byte-compile-compiler-macro).

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: cl-byte-compile-compiler-macro

[Stefan Monnier]

>> When I compile quux.el, does that need CL to be loaded?
>> (Assume quux.el does not load CL itself.)

> I think it does, since the accessor function has a byte-compile property
> that points to function in cl-macs (cl-byte-compile-compiler-macro).

In that case it's a bug in CL (either in one of the cl*.el files or in the
bytecomp.el file where we have the hook for CL's compiler-macros).


        Stefan

Re: cl-byte-compile-compiler-macro

[Richard Stallman]

    > When I compile quux.el, does that need CL to be loaded?
    > (Assume quux.el does not load CL itself.)

    I think it does, since the accessor function has a byte-compile property
    that points to function in cl-macs (cl-byte-compile-compiler-macro).

I think it should be possible to compile quux.el without loading CL.

I proposed moving that function and its subroutine to subr.el.
Will that achieve the goal?

Re: cl-byte-compile-compiler-macro

[Stefan Monnier]

>> When I compile quux.el, does that need CL to be loaded?
>> (Assume quux.el does not load CL itself.)

>     I think it does, since the accessor function has a byte-compile property
>     that points to function in cl-macs (cl-byte-compile-compiler-macro).

> I think it should be possible to compile quux.el without loading CL.

> I proposed moving that function and its subroutine to subr.el.
> Will that achieve the goal?

An alternative solution might be the following which just ignores the
cl-byte-compile-compiler-macro value unless that function is indeed defined.
Since compiler-macros are supposed to only provide optimizations but no
change in semantics, no running them should always be harmless.
Please install it if you think it's right.


        Stefan


Index: bytecomp.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/emacs-lisp/bytecomp.el,v
retrieving revision 2.181
diff -u -u -b -r2.181 bytecomp.el
--- bytecomp.el	13 Jan 2006 09:54:33 -0000	2.181
+++ bytecomp.el	5 Apr 2006 19:30:00 -0000
@@ -429,8 +429,7 @@
 ;; 			       (apply 'byte-compiler-options-handler forms)))
     (eval-when-compile . (lambda (&rest body)
 			   (list 'quote
-				 (byte-compile-eval (byte-compile-top-level
-						     (cons 'progn body))))))
+				 (byte-compile-eval (cons 'progn body)))))
     (eval-and-compile . (lambda (&rest body)
 			  (byte-compile-eval-before-compile (cons 'progn body))
 			  (cons 'progn body))))
@@ -2765,6 +2764,11 @@
 		(byte-compile-warn "`%s' used from Lisp code\n\
 That command is designed for interactive use only" fn))
 	   (if (and handler
+                    ;; Make sure that function exists.  This is important
+                    ;; for CL compiler macros since the symbol may be
+                    ;; `cl-byte-compile-compiler-macro' but if CL isn't
+                    ;; loaded, this function doesn't exist.
+                    (fboundp handler)
 		    (or (not (byte-compile-version-cond
 			      byte-compile-compatibility))
 			(not (get (get fn 'byte-opcode) 'emacs19-opcode))))

Re: cl-byte-compile-compiler-macro

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> When I compile quux.el, does that need CL to be loaded?
>>> (Assume quux.el does not load CL itself.)
>
>>     I think it does, since the accessor function has a byte-compile property
>>     that points to function in cl-macs (cl-byte-compile-compiler-macro).
>
>> I think it should be possible to compile quux.el without loading CL.
>
>> I proposed moving that function and its subroutine to subr.el.
>> Will that achieve the goal?
>
> An alternative solution might be the following which just ignores the
> cl-byte-compile-compiler-macro value unless that function is indeed defined.
> Since compiler-macros are supposed to only provide optimizations but no
> change in semantics, no running them should always be harmless.
> Please install it if you think it's right.

Uh, what?  Am I just confused about the terminology you use?  defmacro
most certainly does not provide optional stuff to the compiler.
defsubst would, but is the result of that called a "compiler macro"
while the result of "defmacro" isn't?

Anyway, I don't even think that a defsubst which might or might not be
executed is a good idea: if the programmer wants to achieve a certain
effect with an optimization, failing to do so should not go unnoticed.

It appears that the current behavior makes it hard to track down the
problem.  I think the solution should be to create better diagnostics
for pinpointing the problem instead of removing them completely.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: cl-byte-compile-compiler-macro

[Stefan Monnier]

> Uh, what?  Am I just confused about the terminology you use?

Yes.

> defmacro most certainly does not provide optional stuff to the compiler.

We're not talking about macros but about compiler-macros.

> defsubst would, but is the result of that called a "compiler macro" while
> the result of "defmacro" isn't?

Yes.  A compiler-macro, contrary to a macro, is only expanded by the
compiler but not by the interpreter, so you can't rely on it
being expanded.  Also it's not expanded if the function is
called indirectly.

> It appears that the current behavior makes it hard to track down the
> problem.  I think the solution should be to create better diagnostics
> for pinpointing the problem instead of removing them completely.

OK, here's the same but with a diagnostic.


        Stefan


Index: bytecomp.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/emacs-lisp/bytecomp.el,v
retrieving revision 2.181
diff -u -u -b -r2.181 bytecomp.el
--- bytecomp.el	13 Jan 2006 09:54:33 -0000	2.181
+++ bytecomp.el	5 Apr 2006 20:11:50 -0000
@@ -2765,6 +2764,14 @@
 		(byte-compile-warn "`%s' used from Lisp code\n\
 That command is designed for interactive use only" fn))
 	   (if (and handler
+                    ;; Make sure that function exists.  This is important
+                    ;; for CL compiler macros since the symbol may be
+                    ;; `cl-byte-compile-compiler-macro' but if CL isn't
+                    ;; loaded, this function doesn't exist.
+                    (or (fboundp handler)
+                        (progn (byte-compile-warn
+                                "Ignoring byte-compiler handler `%S'" handler)
+                               nil))
 		    (or (not (byte-compile-version-cond
 			      byte-compile-compatibility))
 			(not (get (get fn 'byte-opcode) 'emacs19-opcode))))

Re: cl-byte-compile-compiler-macro

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

> I proposed moving that function and its subroutine to subr.el.
> Will that achieve the goal?

cl-byte-compile-compiler-macro provides only the hook into the byte
compiler.  The actual macro expansion is performed by applying the
cl-compiler-macro property.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: cl-byte-compile-compiler-macro

[Richard Stallman]

	 (eval-when-compile . (lambda (&rest body)
			       (list 'quote
    -				 (byte-compile-eval (byte-compile-top-level
    -						     (cons 'progn body))))))
    +				 (byte-compile-eval (cons 'progn body)))))

What is the reason for removing byte-compile-top-level?

Was the call superfluous before?

    +                    (or (fboundp handler)
    +                        (progn (byte-compile-warn
    +                                "Ignoring byte-compiler handler `%S'" handler)
    +                               nil))

I don't think this should warn.  This warning would be generated
whenever you compile calls to these functions and CL is not loaded.
The warning isn't useful, because it describes a circumstance which
does not mean anything is wrong.

So please don't add the warning.

Re: cl-byte-compile-compiler-macro

[Stefan Monnier]

> 	 (eval-when-compile . (lambda (&rest body)
> 			       (list 'quote
>     -				 (byte-compile-eval (byte-compile-top-level
>     -						     (cons 'progn body))))))
>     +				 (byte-compile-eval (cons 'progn body)))))

> What is the reason for removing byte-compile-top-level?

Sorry, this part of the patch was a mistake (it's a local change that
I failed to remove before sending the email).

> Was the call superfluous before?

Yes, I think it's superfluous, and more importantly it can introduce "bugs"
if the code is too large to be byte-compiled (causes too deep a recursion).
I bumped into this at some point where I had a defconst whose (constant)
value was built with a deep backquoted expression: the byte-compiler would
choke on it, so I figured "well it's a constant anyway, so let's eval it
before byte-compiling", but wrapping it in eval-when-compile didn't make
a difference because the content is byte-compiled anyway.

>     +                    (or (fboundp handler)
>     +                        (progn (byte-compile-warn
>     +                                "Ignoring byte-compiler handler `%S'" handler)
>     +                               nil))

> I don't think this should warn.  This warning would be generated
> whenever you compile calls to these functions and CL is not loaded.
> The warning isn't useful, because it describes a circumstance which
> does not mean anything is wrong.

> So please don't add the warning.

OK,


        Stefan