Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Stefan Monnier]

>   Correctly treat progn contents as toplevel forms when byte compiling

Your commit messages should be copies of the ChangeLog entry.

Could you describe the case(s) that this fixes?

> +  ;; Macroexpand (not macroexpand-all!)

That could be a problem.

> form at toplevel in case it
> +  ;; expands into a toplevel-equivalent `progn'.  See CLHS section
> +  ;; 3.2.3.1, "Processing of Top Level Forms".

Note that Elisp is not Common-Lisp, so we don't always follow
Common-Lisp's design decisions (although, we often do, as well).

> The semantics are very
> +  ;; subtle: see test/automated/bytecomp-tests.el for interesting
> +  ;; cases.
> +  (setf form (macroexpand form byte-compile-macro-environment))
> +  (if (eq (car-safe form) 'progn)
> +      (cons 'progn
> +            (mapcar (lambda (subform)
> +                      (byte-compile-recurse-toplevel
> +                       subform non-toplevel-case))
> +                    (cdr form)))
> +    (funcall non-toplevel-case form)))

`non-toplevel-case' is declared as optional, but here you call it
without ensuring it's non-nil.  IOW it shouldn't be optional.

>  (defconst byte-compile-initial-macro-environment
>    '(
>      ;; (byte-compiler-options . (lambda (&rest forms)
>      ;;     		       (apply 'byte-compiler-options-handler forms)))
>      (declare-function . byte-compile-macroexpand-declare-function)
>      (eval-when-compile . (lambda (&rest body)

Oops, we have a bug here.  We should be using `(... ,(lambda ...


        Stefan

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2161 bytes --]

On 04/21/2014 08:09 AM, Stefan Monnier wrote:
>>   Correctly treat progn contents as toplevel forms when byte compiling
> 
> Your commit messages should be copies of the ChangeLog entry.
> 
> Could you describe the case(s) that this fixes?

See the testcases. Generally speaking, this change fixes situations
where the byte-compiler miscompiles code that contains (or generates)
top-level progns that define macros, then use them.

>> +  ;; Macroexpand (not macroexpand-all!)
> 
> That could be a problem.

Why? We macroexpand-all forms later when we actually compile or eval them.

>> form at toplevel in case it
>> +  ;; expands into a toplevel-equivalent `progn'.  See CLHS section
>> +  ;; 3.2.3.1, "Processing of Top Level Forms".
> 
> Note that Elisp is not Common-Lisp, so we don't always follow
> Common-Lisp's design decisions (although, we often do, as well).

Unless there's a good reason to depart from CL, we should follow CL's
approach to things. CL in a good sane default, and in this case, CL
specifies exactly the right behavior.

>> The semantics are very
>> +  ;; subtle: see test/automated/bytecomp-tests.el for interesting
>> +  ;; cases.
>> +  (setf form (macroexpand form byte-compile-macro-environment))
>> +  (if (eq (car-safe form) 'progn)
>> +      (cons 'progn
>> +            (mapcar (lambda (subform)
>> +                      (byte-compile-recurse-toplevel
>> +                       subform non-toplevel-case))
>> +                    (cdr form)))
>> +    (funcall non-toplevel-case form)))
> 
> `non-toplevel-case' is declared as optional, but here you call it
> without ensuring it's non-nil.  IOW it shouldn't be optional.

Err, yes.

>>  (defconst byte-compile-initial-macro-environment
>>    '(
>>      ;; (byte-compiler-options . (lambda (&rest forms)
>>      ;;     		       (apply 'byte-compiler-options-handler forms)))
>>      (declare-function . byte-compile-macroexpand-declare-function)
>>      (eval-when-compile . (lambda (&rest body)
> 
> Oops, we have a bug here.  We should be using `(... ,(lambda ...

Well, a minor performance bug. Still worth fixing.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Stefan Monnier]

>>> Correctly treat progn contents as toplevel forms when byte compiling
>> Your commit messages should be copies of the ChangeLog entry.
>> Could you describe the case(s) that this fixes?
> See the testcases. Generally speaking, this change fixes situations
> where the byte-compiler miscompiles code that contains (or generates)
> top-level progns that define macros, then use them.

Can you show an example of a macro that does that?

>>> +  ;; Macroexpand (not macroexpand-all!)
>> That could be a problem.
> Why? We macroexpand-all forms later when we actually compile or eval them.

Not sure why, exactly.  It just feels like it could be a problem.
Usually, we assume it's safe to use macroexpand-all, and it's not 100%
crystal clear why we could be sure that macroexpand-all won't be used.

> Unless there's a good reason to depart from CL, we should follow CL's
> approach to things. CL in a good sane default, and in this case, CL
> specifies exactly the right behavior.

Yes.  But Elisp's design constraints, especially w.r.t macro expansion,
are slightly different, partly for historical reasons, partly for
technical reasons.  It might not be relevant here, but I just want to
make double sure.


        Stefan

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2054 bytes --]

On 04/21/2014 03:09 PM, Stefan Monnier wrote:
>>>> Correctly treat progn contents as toplevel forms when byte compiling
>>> Your commit messages should be copies of the ChangeLog entry.
>>> Could you describe the case(s) that this fixes?
>> See the testcases. Generally speaking, this change fixes situations
>> where the byte-compiler miscompiles code that contains (or generates)
>> top-level progns that define macros, then use them.
> 
> Can you show an example of a macro that does that?

Any macro that expands to something like this:

(progn (defmacro abc (arg) 1) (defun def () (abc 2)))

Some of the iface stuff I was working on used this functionality. The
comments in bytecomp.el suggest that EIEIO has similar stuff. In any
case, it's a reasonable thing to want to do, so we should support it.

>>>> +  ;; Macroexpand (not macroexpand-all!)
>>> That could be a problem.
>> Why? We macroexpand-all forms later when we actually compile or eval them.
> 
> Not sure why, exactly.  It just feels like it could be a problem.
> Usually, we assume it's safe to use macroexpand-all, and it's not 100%
> crystal clear why we could be sure that macroexpand-all won't be used.

We only want to expand the top-level macro to see whether it expands
into a progn. If it does, we treat each form in the progn as its own
top-level form, recursively. If we called macroexpand-all, we'd expand
the subforms too soon, before we'd compiled defmacro forms that might be
inside the progn body.

>> Unless there's a good reason to depart from CL, we should follow CL's
>> approach to things. CL in a good sane default, and in this case, CL
>> specifies exactly the right behavior.
> 
> Yes.  But Elisp's design constraints, especially w.r.t macro expansion,
> are slightly different, partly for historical reasons, partly for
> technical reasons.  It might not be relevant here, but I just want to
> make double sure.

It's not relevant here. I also still want to make those macro-expansion
changes I proposed last year.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Stefan Monnier]

> Any macro that expands to something like this:
> (progn (defmacro abc (arg) 1) (defun def () (abc 2)))

That's not the kind of example I was thinking if.  Do you have a real
example, maybe?

> Some of the iface stuff I was working on used this functionality.

> The comments in bytecomp.el suggest that EIEIO has similar stuff.

No, that comment is about eieio.el using eval-and-compile over large
swaths of code, which is very unusual, so I made eval-and-compile behave
more like the usual top-level (before that eval-and-compile expanded
all its macros in a single pass).

> In any case, it's a reasonable thing to want to do, so we should
> support it.

I'm not sure I want to impose myself this level of detail.
Especially since it'll also have to be ported to the eager
macro-expansion code.

And OTOH your macro could expand calls to `abc' "manually" in the code
it returns, and hence avoid this need.


        Stefan

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2392 bytes --]

On 04/21/2014 07:09 PM, Stefan Monnier wrote:
>> Any macro that expands to something like this:
>> (progn (defmacro abc (arg) 1) (defun def () (abc 2)))
> 
> That's not the kind of example I was thinking if.  Do you have a real
> example, maybe?

Try writing a macro that emits a defstruct, then a function that uses
cl-typep for that struct, all wrapped in the same toplevel progn.

>> Some of the iface stuff I was working on used this functionality.
> 
>> The comments in bytecomp.el suggest that EIEIO has similar stuff.
> 
> No, that comment is about eieio.el using eval-and-compile over large
> swaths of code, which is very unusual, so I made eval-and-compile behave
> more like the usual top-level (before that eval-and-compile expanded
> all its macros in a single pass).

Yes, eieio uses eval-and-compile, but there's no reason the same
toplevel semantics shouldn't apply to regular progn.

>> In any case, it's a reasonable thing to want to do, so we should
>> support it.
> 
> I'm not sure I want to impose myself this level of detail.
> Especially since it'll also have to be ported to the eager
> macro-expansion code.

Other lisps provide the same functionality that my change does, and for
good reason. It's a simple, useful concept: macros can only generate one
form, so if they generate a progn, we treat each form in the progn as a
toplevel form. Not supporting this mode of operation makes elisp much
less useful.

The reason we have automated tests is to make sure we can maintain this
"level of detail". That it's not immediately useful to you isn't a
reason not to include it. I can't believe this issue is even
contentious: the current behavior is a clear bug.

> And OTOH your macro could expand calls to `abc' "manually" in the code
> it returns, and hence avoid this need.

How? By locally defining those macros in a temporary macrolet (which
we'd macroexpand-all?) in addition to emiting defmacros? What if the
macro generation is implicit, as it is with defstruct, so that the
top-level form doesn't even realize it has to employ this awful hack?

The need for this ridiculous and needless hack can be avoided if we just
have sane toplevel macro expansion semantics. Forcing people to rely on
tricks like what you propose instead of providing sane macro expansion
semantics will waste a lot of people's time.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Stefan Monnier]

>> That's not the kind of example I was thinking if.  Do you have a real
>> example, maybe?
> Try writing a macro that emits a defstruct, then a function that uses
> cl-typep for that struct, all wrapped in the same toplevel progn.

That's getting closer to an actual example.  I still can't think of
a case where you'd want to use cl-typep in this way, tho.

> The reason we have automated tests is to make sure we can maintain this
> "level of detail". That it's not immediately useful to you isn't a
> reason not to include it. I can't believe this issue is even
> contentious: the current behavior is a clear bug.

It's a clear bug if we assume Common-Lisp semantics.  But in many cases,
Elisp chooses to provide simpler semantics, to allow a simpler and/or
more naive implementation.

I'm not yet convinced either way, but claiming it's a bug won't sway me.

We've lived with a "simpler" semantics for 30 years and you're the first
to complain about this limitation, so I'd really want to see clear
evidence that it's worth adding this complexity.

> we'd macroexpand-all?) in addition to emiting defmacros? What if the
> macro generation is implicit, as it is with defstruct, so that the
> top-level form doesn't even realize it has to employ this awful hack?

Yes, the defstruct+typep example above sounds like a good example where
my suggestion is not really workable.


        Stefan

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 4286 bytes --]

On 04/21/2014 09:25 PM, Stefan Monnier wrote:
>>> That's not the kind of example I was thinking if.  Do you have a real
>>> example, maybe?
>> Try writing a macro that emits a defstruct, then a function that uses
>> cl-typep for that struct, all wrapped in the same toplevel progn.
> 
> That's getting closer to an actual example.  I still can't think of
> a case where you'd want to use cl-typep in this way, tho.

cl-struct-slot-value uses typep  It makes sense to do so given that
cl-struct-slot-value knows the name of the struct, but not necessarily
the name of its predicate function --- which might not even exist.

The chunk below is from the iface code I'm working on. This code isn't
even close to done --- it's just here to give you a sense of the
problem. You see that we declare a structure, then emit some functions
that use the structure. Some of these functions go on to do things with
values that are allegedly of the structure type, and they use typep to
check that these values are actually of the struct type. This approach
is a perfectly logical way to go about writing code, and we should
support it. Forcing users to do something else is especially pernicious
because the problem we're discussing only shows up when compiling (or
loading) files --- in normal interactive development, everything seems fine!

(eval-and-compile
  (defun iface--make-member-wrapper (iface-name member)
    "Create a lisp form defining an interface accessor.
IFACE-NAME is a symbol giving the name of the interface being
defined.  MEMBER is an element of the MEMBERS parameter to
`iface-declare'."
    (cl-destructuring-bind ((member-name inst-arg &rest xarglist)
                            &optional doc) member
      (unless (and inst-arg
                   (symbolp inst-arg)
                   (not (eql (aref (symbol-name inst-arg) 0) ?\&)))
        (error (concat "Interface members must accept an instance as"
                       " their first argument")))
      `(defun ,member-name (,inst-arg &rest xargs)
         ,doc
         (declare (advertised-calling-convention (,inst-arg ,@xarglist)
                                                 ,emacs-version))
         (apply (cl-struct-slot-value ',iface-name ',member-name ,inst-arg)
                ,inst-arg xargs)))))

(cl-defmacro iface-declare ((name base) &rest members)
  "Create a new interface called NAME, inheriting from BASE.
MEMBERS is a list of member specifiers.  Each is a list of the form
 ((NAME . ARGUMENTS) DOCUMENTATION), where NAME is a symbol
naming the interface member, ARGUMENTS is a cl-lib-style argument
list, and DOCUMENTATION is documentation to attach to that
function."
  (let ((doc (and (stringp (car members)) (pop members)))
        (prefix (concat (symbol-name name) "--iface-")))
    `(progn
       (cl-defstruct (,name (:include ,base)
                            (:conc-name ,(intern prefix))
                            (:copier nil))
         ,@(if doc (list doc))
         ,@(mapcar #'caar members))
       ,@(mapcar (lambda (member)
                   (iface--make-member-wrapper name member))
                 members)
       ',name)))

>> The reason we have automated tests is to make sure we can maintain this
>> "level of detail". That it's not immediately useful to you isn't a
>> reason not to include it. I can't believe this issue is even
>> contentious: the current behavior is a clear bug.
> 
> It's a clear bug if we assume Common-Lisp semantics.  But in many cases,
> Elisp chooses to provide simpler semantics, to allow a simpler and/or
> more naive implementation.

I don't care whether we diverge from CL in sets of supported features
and names of functions, but we should try hard to match CL's fundamental
execution semantics. Practically every possible bad lisp idea was tried
before Common Lisp came around. The CL people knew about the precise
problem we're discussing on this thread and codified the best solution
in the language. There are solutions in CL for problems that we haven't
even encountered yet. The less closely we follow CL, the less we benefit
from the experience baked into the specification and the harder it is to
fix problems that eventually do come up.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Stefan Monnier]

> (eval-and-compile
>   (defun iface--make-member-wrapper (iface-name member)
>     "Create a lisp form defining an interface accessor.
> IFACE-NAME is a symbol giving the name of the interface being
> defined.  MEMBER is an element of the MEMBERS parameter to
> `iface-declare'."
>     (cl-destructuring-bind ((member-name inst-arg &rest xarglist)
>                             &optional doc) member
>       (unless (and inst-arg
>                    (symbolp inst-arg)
>                    (not (eql (aref (symbol-name inst-arg) 0) ?\&)))
>         (error (concat "Interface members must accept an instance as"
>                        " their first argument")))
>       `(defun ,member-name (,inst-arg &rest xargs)
>          ,doc
>          (declare (advertised-calling-convention (,inst-arg ,@xarglist)
>                                                  ,emacs-version))
>          (apply (cl-struct-slot-value ',iface-name ',member-name ,inst-arg)
>                 ,inst-arg xargs)))))

> (cl-defmacro iface-declare ((name base) &rest members)
>   "Create a new interface called NAME, inheriting from BASE.
> MEMBERS is a list of member specifiers.  Each is a list of the form
>  ((NAME . ARGUMENTS) DOCUMENTATION), where NAME is a symbol
> naming the interface member, ARGUMENTS is a cl-lib-style argument
> list, and DOCUMENTATION is documentation to attach to that
> function."
>   (let ((doc (and (stringp (car members)) (pop members)))
>         (prefix (concat (symbol-name name) "--iface-")))
>     `(progn
>        (cl-defstruct (,name (:include ,base)
>                             (:conc-name ,(intern prefix))
>                             (:copier nil))
>          ,@(if doc (list doc))
>          ,@(mapcar #'caar members))
>        ,@(mapcar (lambda (member)
>                    (iface--make-member-wrapper name member))
>                  members)
>        ',name)))

But here you really don't need cl-struct-slot-value since you know very
well the name of the accessor functions.

And if you insist on doing it this way, you can wrap the cl-defstruct
within an eval-and-compile.

The problem with relying on CL semantics, is that it means that your
iface-declare will only work at top-level, e.g. not within a `cl-letf'
or a `cl-flet' or ...

So while the current semantics bites you, CL's semantics will also bite
you in other cases.  IOW They just try to avoid biting you in a couple
more cases, but they can't avoid the fundamental problem.

The above example doesn't convince me yet that the convenience of hiding
the fundamental problem in those few cases is worth the trouble.

Maybe to solve this problem right, we'd need to do it in
macroexpand-all: detect a "defmacro" and add the corresponding macro to
macroexpand-all-environment for the rest of the expansion.  Not sure
it'd be worth the trouble, but at least it would fix this "toplevel
special case".


        Stefan

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Stefan Monnier]

>       `(defun ,member-name (,inst-arg &rest xargs)
>          ,doc
>          (declare (advertised-calling-convention (,inst-arg ,@xarglist)
>                                                  ,emacs-version))
>          (apply (cl-struct-slot-value ',iface-name ',member-name ,inst-arg)
>                 ,inst-arg xargs)))))

I think we'd be just as happy, with something like

   (defun method-call (method inst &rest args)
     (apply (funcall method inst) inst args))

instead of defining umpteen "method call wrappers".


        Stefan

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 911 bytes --]

On 04/22/2014 08:20 AM, Stefan Monnier wrote:
>>       `(defun ,member-name (,inst-arg &rest xargs)
>>          ,doc
>>          (declare (advertised-calling-convention (,inst-arg ,@xarglist)
>>                                                  ,emacs-version))
>>          (apply (cl-struct-slot-value ',iface-name ',member-name ,inst-arg)
>>                 ,inst-arg xargs)))))
> 
> I think we'd be just as happy, with something like
> 
>    (defun method-call (method inst &rest args)
>      (apply (funcall method inst) inst args))
> 
> instead of defining umpteen "method call wrappers".

That way, you don't get docstrings, parameter-number checking, eldoc,
and so on. And you have to be aware that you're using this odd calling
convention instead of just calling a function like you would any other.
I'd much rather have the wrappers, which we can inline and make
practically free.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]

On 04/22/2014 08:06 AM, Stefan Monnier wrote:
> And if you insist on doing it this way, you can wrap the cl-defstruct
> within an eval-and-compile.
> 
> The problem with relying on CL semantics, is that it means that your
> iface-declare will only work at top-level, e.g. not within a `cl-letf'
> or a `cl-flet' or ...
> a
> So while the current semantics bites you, CL's semantics will also bite
> you in other cases.  IOW They just try to avoid biting you in a couple
> more cases, but they can't avoid the fundamental problem.
>
> The above example doesn't convince me yet that the convenience of hiding
> the fundamental problem in those few cases is worth the trouble.
>
> Maybe to solve this problem right, we'd need to do it in
> macroexpand-all: detect a "defmacro" and add the corresponding macro to
> macroexpand-all-environment for the rest of the expansion.  Not sure
> it'd be worth the trouble, but at least it would fix this "toplevel
> special case".

That's fine too, provided we handle the toplevel case correctly. Would
you accept a change that did that?



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1943 bytes --]

On 04/22/2014 10:22 AM, Daniel Colascione wrote:
> On 04/22/2014 08:06 AM, Stefan Monnier wrote:
>> And if you insist on doing it this way, you can wrap the cl-defstruct
>> within an eval-and-compile.
>>
>> The problem with relying on CL semantics, is that it means that your
>> iface-declare will only work at top-level, e.g. not within a `cl-letf'
>> or a `cl-flet' or ...
>> a
>> So while the current semantics bites you, CL's semantics will also bite
>> you in other cases.  IOW They just try to avoid biting you in a couple
>> more cases, but they can't avoid the fundamental problem.
>>
>> The above example doesn't convince me yet that the convenience of hiding
>> the fundamental problem in those few cases is worth the trouble.
>>
>> Maybe to solve this problem right, we'd need to do it in
>> macroexpand-all: detect a "defmacro" and add the corresponding macro to
>> macroexpand-all-environment for the rest of the expansion.  Not sure
>> it'd be worth the trouble, but at least it would fix this "toplevel
>> special case".
> 
> That's fine too, provided we handle the toplevel case correctly. Would
> you accept a change that did that?

That said, the CL way is much better. It's conceptually simpler to say
that defmacro at toplevel is well-defined (progn contents being
considered "toplevel") and everything else is undefined than to try to
support this defmacro stuff in *all* contexts and run into the exciting
corner inherent in your approach: cases involving nested macroexpand-all
calls in particular would be interesting. You worried about the level of
"detail" required to maintain this stuff, yet you're talking about
greatly increasing, relative to my change, the number of situations in
which we have to define macro expansion semantics. The CL toplevel
semantics are required to support real code: yours are not. Nobody is
going to write a defmacro at toplevel inside a cl-letf.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Stefan Monnier]

> That's fine too, provided we handle the toplevel case correctly.
> Would you accept a change that did that?

Depends on what ends up being needed for it.  But if it's not too ugly, yes.


        Stefan

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Stefan Monnier]

> Nobody is going to write a defmacro at toplevel inside a cl-letf.

So far, nobody has written the kind of code you say you want to write.
In both cases it's trying to handle hypothetical cases, so indeed,
I prefer neither overall.

I'm OK with adding a bit of complexity if it removes special cases
(which my proposal might do, tho I have no experience with it, so maybe
it'll need horrendous hacks to implement and additionally it will
introduce new special cases), but the CL semantics adds complexity
without really removing special cases: it just moves the "special case
boundary" elsewhere.  And I don't like it for that reason.
It's a question of taste, to a large extent, hence the bikeshedding.

I prefer you adding `eval-and-compile' at a few places to work around
the "buggy Elisp semantics" over imposing the CL semantics.


        Stefan

RE: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Drew Adams]

> That said, the CL way is much better. It's conceptually simpler to say
> that defmacro at toplevel is well-defined (progn contents being
> considered "toplevel") and everything else is undefined than to try to
> support this defmacro stuff in *all* contexts and run into the exciting
> corner inherent in your approach...

Are you saying that in CL if defmacro is used elsewhere than at top level
it is not well-defined?  That doesn't sound right to me.

(Apologies if I misunderstand.  If so, or if irrelevant, please ignore.)

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2479 bytes --]

On 04/22/2014 11:37 AM, Stefan Monnier wrote:
>> Nobody is going to write a defmacro at toplevel inside a cl-letf.
> I'm OK with adding a bit of complexity if it removes special cases
> (which my proposal might do, tho I have no experience with it, so maybe
> it'll need horrendous hacks to implement and additionally it will
> introduce new special cases), 

I think it will need horrendous hacks (but maybe I'm wrong). We'd have
to expand macros step-by-step and check recursively for macro-defining
macros, and probably introduce some kind of special alternative defmacro
in the environment, since the normal `defmacro' is itself a macro and
would be hidden by macroexpand. (Imagine a macro that just expands to a
single defmacro --- we'd still want to add that macro to our temporary
environment.) To address this problem, we could merge macroexpand-1
instead, but that's an even bigger change. You also have to worry about
nested macroexpand-all calls, leakage between defuns, and so on. There
are interesting corner cases too: what about (progn (and nil (defmacro
foo () ...)) (foo))?

The current code works, solves a real problem, matches many other lisp
implementations, and has automated tests. It doesn't affect (and slow)
every macro expansion everywhere.

> but the CL semantics adds complexity
> without really removing special cases: it just moves the "special case
> boundary" elsewhere.  And I don't like it for that reason.
> It's a question of taste, to a large extent, hence the bikeshedding.

Your proposal is equivalent to sucking all the forms "following" (in
some sense) a defmacro into an implicit macrolet that defines the same
macro that the defmacro does. Fine. But how do you decide how much code
is included in this implicit macrolet? All the remaining forms in the
current call to macroexpand-all? That's too broad: a macro defined in
one defun shouldn't affect forms in a completely different defun. The
forms following the defmacro in the same progn? That's too narrow ---
there's still a "special case boundary". I think that no matter where we
choose to put the implicit macrolet, it'll surprise someone. I'd rather
not try. Instead, we should address the case that's come up in practice
(and that came up 20 years ago in Common Lisp).

> I prefer you adding `eval-and-compile' at a few places to work around
> the "buggy Elisp semantics" over imposing the CL semantics.

I'm not going to do that.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

On 04/22/2014 11:44 AM, Drew Adams wrote:
>> That said, the CL way is much better. It's conceptually simpler to say
>> that defmacro at toplevel is well-defined (progn contents being
>> considered "toplevel") and everything else is undefined than to try to
>> support this defmacro stuff in *all* contexts and run into the exciting
>> corner inherent in your approach...
> 
> Are you saying that in CL if defmacro is used elsewhere than at top level
> it is not well-defined?  That doesn't sound right to me.
> 
> (Apologies if I misunderstand.  If so, or if irrelevant, please ignore.)

Compilers appear to differ. Consider the code below:

(defun foo ()
  (defmacro bar () 1)
  (bar))

(format t "~s" (foo))

SBCL and ECL both signal an error. CLISP compiles the file, substituting
(bar) with 1, then complains when calling foo that bar is redefined. The
SBCL and ECL behavior is what I'd expect from reading the spec, but
maybe I misunderstood something.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

RE: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Drew Adams]

> >> That said, the CL way is much better. It's conceptually simpler to say
> >> that defmacro at toplevel is well-defined (progn contents being
> >> considered "toplevel") and everything else is undefined than to try to
> >> support this defmacro stuff in *all* contexts and run into the exciting
> >> corner inherent in your approach...
> >
> > Are you saying that in CL if defmacro is used elsewhere than at top level
> > it is not well-defined?  That doesn't sound right to me.
> >
> > (Apologies if I misunderstand.  If so, or if irrelevant, please ignore.)
> 
> Compilers appear to differ. Consider the code below:
> 
> (defun foo ()
>   (defmacro bar () 1)
>   (bar))
> 
> (format t "~s" (foo))
> 
> SBCL and ECL both signal an error. CLISP compiles the file, substituting
> (bar) with 1, then complains when calling foo that bar is redefined.

I see.  But that does not imply that the standard says that the behavior
is undefined for `defmacro' at other than top level.  That's what I
thought you were saying.  Implementations do not always implement what
a standard says. ;-)

> The SBCL and ECL behavior is what I'd expect from reading the spec, but
> maybe I misunderstood something.

Hm.  What part of the spec do you think gives the impression that
`defmacro' behavior is defined only at top level?

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 1879 bytes --]

On 04/22/2014 12:59 PM, Drew Adams wrote:
>> The SBCL and ECL behavior is what I'd expect from reading the spec, but
>> maybe I misunderstood something.
> 
> Hm.  What part of the spec do you think gives the impression that
> `defmacro' behavior is defined only at top level?

defmacro is specced to "define[] name as a macro by associating a macro
function with that name in the global environment". Like any form, it's
supposed to do that when it's evaluated, and forms inside defuns aren't
evaluated when they're loaded or compiled: they're evaluated when the
defun is called. So you should expect a defmacro inside a defun to have
no effect unti lthe defun is called.

Likewise, if a defmacro appears at top level, it should ordinarily be
evaluated only when the file that contains it is loaded, just like any
other toplevel form. So, without special provisions, defmacro should not
be expanded inside forms that happen to be in the same file.

But this behavior isn't very useful, so as a special case, CL specifies
that "if a defmacro form appears as a top level form, the compiler must
store the macro definition at compile time, so that occurrences of the
macro later on in the file can be expanded correctly. Users must ensure
that the body of the macro can be evaluated at compile time if it is
referenced within the file being compiled."

It's because of this special case that a normal defmacro, not wrapped in
an eval-when, has any effect on compilation of forms in the same file. A
defmacro inside a defun isn't covered by this special case, so its
behavior should revert to the normal behavior for forms. SBCL and ECL
implement this model of evaluation. CLISP's behavior appears to be
incorrect here.

The behavior I'm proposing for Emacs is identical to what SBCL and ECL
do. Stefan is proposing that we adopt CLISP behavior.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

RE: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Drew Adams]

> >> The SBCL and ECL behavior is what I'd expect from reading the spec, but
> >> maybe I misunderstood something.
> >
> > Hm.  What part of the spec do you think gives the impression that
> > `defmacro' behavior is defined only at top level?
> 
> defmacro is specced to "define[] name as a macro by associating a macro
> function with that name in the global environment". 

> Like any form, it's supposed to do that when it's evaluated,

Yes, anywhere and anywhen it's evaluated, including from a file at any
level.  And including when invoked from running code.  And including
`defmacro' forms that are generated and eval'd on the fly.

> and forms inside defuns aren't evaluated when they're loaded or
> compiled: they're evaluated when the defun is called. So you should
> expect a defmacro inside a defun to have no effect unti lthe defun is
> called.

Agreed, but when the defun is called and the `defmacro' is evaluated,
its behavior should be well-defined - it is not just up for grabs or
up to the implementation.

But I do see now that you allowed for `progn' contexts, at least.
I guess you meant any context, like progn, which evaluates the
`defmacro' at load or compile time.

> Likewise, if a defmacro appears at top level, it should ordinarily be
> evaluated only when the file that contains it is loaded, just like any
> other toplevel form. So, without special provisions, defmacro should not
> be expanded inside forms that happen to be in the same file.
> 
> But this behavior isn't very useful, so as a special case, CL specifies
> that "if a defmacro form appears as a top level form, the compiler must
> store the macro definition at compile time, so that occurrences of the
> macro later on in the file can be expanded correctly. Users must ensure
> that the body of the macro can be evaluated at compile time if it is
> referenced within the file being compiled."

That was precisely the passage I was thinking of, BTW.

It does not say anything about `defmacro' inside a defun signaling a
compile-time error, as opposed to doing what you write above: "have no
effect until the defun is called".  A priori, when the defun is called
the `defmacro' inside it should be evaluated, defining the macro
normally, no?

It says clearly that IF you expect the macro to be available to code
in the same file at compile time for that file, THEN you must ensure
that it gets evaluated at compile time.  If not, then not.

> It's because of this special case that a normal defmacro, not wrapped in
> an eval-when, has any effect on compilation of forms in the same file. A
> defmacro inside a defun isn't covered by this special case, so its
> behavior should revert to the normal behavior for forms.

Which is what?  Why isn't it that the embedded `defmacro' would, as you
said "have no effect until the defun is called"?

> SBCL and ECL implement this model of evaluation. CLISP's behavior
> appears to be incorrect here.

I don't see that either of those behaviors realizes the exclusive truth
here.  All I see that passage saying is that you must not expect a
`defvar' embedded in a defun to have an effect, at compile time, on
subsequent code located in the same file.  I don't see that as a call
to raise a compile-time error.

> The behavior I'm proposing for Emacs is identical to what SBCL and ECL
> do. Stefan is proposing that we adopt CLISP behavior.

I see.  Anyway, I misunderstood your original comment about `defmacro'
behavior being undefined if not at top level - sorry.

(And I have no special opinion about the question you are deciding.
A priori, I prefer that Emacs Lisp be closer to CL, but I have no
opinion about this particular issue.)

Re: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2674 bytes --]

On 04/22/2014 01:41 PM, Drew Adams wrote:
>>>> The SBCL and ECL behavior is what I'd expect from reading the spec, but
>>>> maybe I misunderstood something.
>>>
>>> Hm.  What part of the spec do you think gives the impression that
>>> `defmacro' behavior is defined only at top level?
>>
>> defmacro is specced to "define[] name as a macro by associating a macro
>> function with that name in the global environment". 
> 
>> Like any form, it's supposed to do that when it's evaluated,
> 
> Yes, anywhere and anywhen it's evaluated, including from a file at any
> level.  And including when invoked from running code.  And including
> `defmacro' forms that are generated and eval'd on the fly.

That goes without saying.

> But I do see now that you allowed for `progn' contexts, at least.
> I guess you meant any context, like progn, which evaluates the
> `defmacro' at load or compile time.
> 
> It does not say anything about `defmacro' inside a defun signaling a
> compile-time error, as opposed to doing what you write above: "have no
> effect until the defun is called".  A priori, when the defun is called
> the `defmacro' inside it should be evaluated, defining the macro
> normally, no?

Neither did I. I was imprecise: the code I mentioned compiles without
error. It signals an error at runtime because there is no function
called `bar'. The function is trying to call `bar' because the macro
`bar' wasn't expanded during compilation.

> It says clearly that IF you expect the macro to be available to code
> in the same file at compile time for that file, THEN you must ensure
> that it gets evaluated at compile time.  If not, then not.
> 
>> It's because of this special case that a normal defmacro, not wrapped in
>> an eval-when, has any effect on compilation of forms in the same file. A
>> defmacro inside a defun isn't covered by this special case, so its
>> behavior should revert to the normal behavior for forms.
> 
> Which is what?  Why isn't it that the embedded `defmacro' would, as you
> said "have no effect until the defun is called"?

"No effect until the defun is called" is the correct behavior.

>> SBCL and ECL implement this model of evaluation. CLISP's behavior
>> appears to be incorrect here.
> 
> I don't see that either of those behaviors realizes the exclusive truth
> here.  All I see that passage saying is that you must not expect a
> `defvar' embedded in a defun to have an effect, at compile time, on
> subsequent code located in the same file.  I don't see that as a call
> to raise a compile-time error.

There is no compile-time error; sorry for the misunderstanding.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

RE: [Emacs-diffs] trunk r117002: Correctly treat progn contents as toplevel forms when byte compiling

[Stephen J. Turnbull]

Drew Adams writes:

 > Yes, anywhere and anywhen it's evaluated, including from a file at
 > any level.  And including when invoked from running code.  And
 > including `defmacro' forms that are generated and eval'd on the
 > fly.

The point is that the list of cases you have above is correct
*because* it does not include "when compiling".  The compiler does not
evaluate the forms it processes, with a very few exceptions.  AFAIK
the only forms in the compiled text that cause evaluation at compile
time are `eval-when' and friends.

 > But I do see now that you allowed for `progn' contexts, at least.
 > I guess you meant any context, like progn, which evaluates the
 > `defmacro' at load or compile time.

`progn' does not evaluate anything at compile time.