Strange change in backquote expansion

Strange change in backquote expansion

[Sascha Wilde]

Hi *,

I'm not quite sure what to think about it, but something changed in
expansion of macro expansion, more precisely in expansion of
backquotes:

Old Emacs (including 22.0):
(defmacro debug-macro-test (arg)
  ``(,,@(list (list arg))))

(macroexpand '(debug-macro-test foo))
=> (list (foo))

Current HEAD:
(macroexpand '(debug-macro-test foo))
=> (list ((foo)))

note the extra parenthesis.

Please don't ask, what the code is supposed to do, this is a extremely
boiled down version of the problem.  I stumbled across this problem
when I noticed that DVC stopped working in HEAD recently.  (They are
using a somewhat esoteric macro, which suffers from this change in
behavior.)

cheers
sascha
-- 
Sascha Wilde
Well, *my* brain likes to think it's vastly more powerful than any
finite Turing machine but it hasn't proven that to me...
  -- Christopher Koppler in comp.lang.lisp

Re: Strange change in backquote expansion

[Richard Stallman]

      ``(,,@(list (list arg))))

I don't think that is a meaningful expression.  ,@ is supposed to be
used inside of a list, to cause appending into that list.  Using it
after a comma means it isn't in any visible list.

I've tried to figure out what the first backquote ought to expand
into, and I can't.

Re: Strange change in backquote expansion

[Andreas Schwab]

Richard Stallman <rms@gnu.org> writes:

>       ``(,,@(list (list arg))))
>
> I don't think that is a meaningful expression.  ,@ is supposed to be
> used inside of a list, to cause appending into that list.  Using it
> after a comma means it isn't in any visible list.

If you consider ,a <=> (\, a) then it actually makes sense.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Strange change in backquote expansion

[Johan Bockgård]

Richard Stallman <rms@gnu.org> writes:

>       ``(,,@(list (list arg))))
>
> I don't think that is a meaningful expression.  ,@ is supposed to be
> used inside of a list, to cause appending into that list.  Using it
> after a comma means it isn't in any visible list.


$ sbcl
This is SBCL 1.0, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
* (defmacro debug-macro-test (arg)
  ``(,,@(list (list arg))))

DEBUG-MACRO-TEST
* (macroexpand '(debug-macro-test foo))

`(,(FOO))
T
* (debug-macro-test *)

(1)



      So ,,@q means ``the value of q is a list of forms; splice the list
      of values of the elements of the value of q.''

[ Steele, Common Lisp the Language, 2nd edition, Appendix C. Backquote,
  http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node367.html ]

-- 
Johan Bockgård

Re: Strange change in backquote expansion

[Stefan Monnier]

> I'm not quite sure what to think about it, but something changed in
> expansion of macro expansion, more precisely in expansion of
> backquotes:

> Old Emacs (including 22.0):
> (defmacro debug-macro-test (arg)
>   ``(,,@(list (list arg))))

> (macroexpand '(debug-macro-test foo))
> => (list (foo))

> Current HEAD:
> (macroexpand '(debug-macro-test foo))
> => (list ((foo)))

> note the extra parenthesis.

I made the change.  Will look into it, thanks,


        Stefan

Re: Strange change in backquote expansion

[Richard Stallman]

On a related issue, the Lisp manual says that the old syntax with an
extra level of parens around the args of backquote and comma is still
supported for compatibility with versions before 19.29.

Since 19.29 was 12 years ago, I think it is time to delete the
documentation of that compatibility feature.  Does anyone object?

Re: Strange change in backquote expansion

[Richard Stallman]

    * (defmacro debug-macro-test (arg)
      ``(,,@(list (list arg))))

    DEBUG-MACRO-TEST
    * (macroexpand '(debug-macro-test foo))

    `(,(FOO))

Ok, that makes sense.

So this seems to be a real bug that we should fix.

Re: Strange change in backquote expansion

[Stefan Monnier]

> On a related issue, the Lisp manual says that the old syntax with an
> extra level of parens around the args of backquote and comma is still
> supported for compatibility with versions before 19.29.

> Since 19.29 was 12 years ago, I think it is time to delete the
> documentation of that compatibility feature.  Does anyone object?

I believe the old syntax is still used in some of the bundled packages, not
to mention the unbundled ones.

I think if we want to ever get rid of the old syntax, we'll have to do
something stronger than just declare it obsolete.  Maybe breaking it is the
solution, but maybe something halfway would be preferable.


        Stefan

Re: Strange change in backquote expansion

[Richard Stallman]

    > Since 19.29 was 12 years ago, I think it is time to delete the
    > documentation of that compatibility feature.  Does anyone object?

    I believe the old syntax is still used in some of the bundled packages, not
    to mention the unbundled ones.

Can you arrange to issue a warning when reading code that uses the old
syntax?  There could be a flag to control the warning, and for the moment
we could make only the byte compiler turn on the flag.  That would make
it convenient if the warnings come thru the compiler warning mechanism.

In Emacs 24 we could make the old syntax an error.

Re: Strange change in backquote expansion

[David Kastrup]

Richard Stallman <rms@gnu.org> writes:

>     > Since 19.29 was 12 years ago, I think it is time to delete the
>     > documentation of that compatibility feature.  Does anyone object?
>
>     I believe the old syntax is still used in some of the bundled
>     packages, not to mention the unbundled ones.
>
> Can you arrange to issue a warning when reading code that uses the old
> syntax?  There could be a flag to control the warning, and for the moment
> we could make only the byte compiler turn on the flag.  That would make
> it convenient if the warnings come thru the compiler warning mechanism.
>
> In Emacs 24 we could make the old syntax an error.

Where do we put plans scheduled for Emacs 24?  TODO in the multi-tty
branch?

We probably should also put "Make sure that the Y2038 bug does not
bite us" there.

-- 
David Kastrup

Re: Strange change in backquote expansion

[Glenn Morris]

David Kastrup wrote:

> Where do we put plans scheduled for Emacs 24?  TODO in the multi-tty
> branch?
>
> We probably should also put "Make sure that the Y2038 bug does not
> bite us" there.

Good news! I guess you must have missed these two emails somehow:

http://lists.gnu.org/archive/html/emacs-devel/2007-08/msg00824.html
http://lists.gnu.org/archive/html/emacs-devel/2007-07/msg01503.html

So you should feel free to merge multi-tty to the trunk right now.

Re: Strange change in backquote expansion

[Stefan Monnier]

> Old Emacs (including 22.0):
> (defmacro debug-macro-test (arg)
>   ``(,,@(list (list arg))))

> (macroexpand '(debug-macro-test foo))
> => (list (foo))

I believe the patch below fixes it (installed on the 22 branch).


        Stefan


--- orig/lisp/emacs-lisp/backquote.el
+++ mod/lisp/emacs-lisp/backquote.el
@@ -110,9 +110,8 @@
 (defun backquote-delay-process (s level)
   "Process a (un|back|splice)quote inside a backquote.
 This simply recurses through the body."
-  (let ((exp (backquote-listify (list (backquote-process (nth 1 s) level)
-                                      (cons 0 (list 'quote (car s))))
-                                '(0))))
+  (let ((exp (backquote-listify (list (cons 0 (list 'quote (car s))))
+                                (backquote-process (cdr s) level))))
     (if (eq (car-safe exp) 'quote)
         (cons 0 (list 'quote s))
       (cons 1 exp))))

Re: Strange change in backquote expansion

[Stefan Monnier]

>> Since 19.29 was 12 years ago, I think it is time to delete the
>> documentation of that compatibility feature.  Does anyone object?

>     I believe the old syntax is still used in some of the bundled
>     packages, not to mention the unbundled ones.

> Can you arrange to issue a warning when reading code that uses the old
> syntax?  There could be a flag to control the warning, and for the moment
> we could make only the byte compiler turn on the flag.  That would make
> it convenient if the warnings come thru the compiler warning mechanism.

How 'bout the following:

- add a new variable no-old-style-backquotes, with 2 values:
  t   = treat all backquotes and commas as new style
  nil = behave like now (i.e. try to tell which is which)

- add a new variable new-old-style-backquotes.
  When no-old-style-backquotes is t and we encounter an old-style backquote,
  remember the fact in this variable (could be just a boolean flag or a list
  of the affected sexps).
  I.e. this keeps track of the backquotes that used to be treated as
  old-style but have been instead treated as new-style.

Then default no-old-style-backquotes to nil, but bind it to t in the
byte-compiler (and at the end of byte-compilation of a file, check
new-old-style-backquotes to see if there's been something fishy and warn
about it if so).


        Stefan

Re: Strange change in backquote expansion

[Andreas Schwab]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Old Emacs (including 22.0):
>> (defmacro debug-macro-test (arg)
>>   ``(,,@(list (list arg))))
>
>> (macroexpand '(debug-macro-test foo))
>> => (list (foo))
>
> I believe the patch below fixes it (installed on the 22 branch).

Btw: what testcase did you actually fix with the original bug fix?

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Strange change in backquote expansion

[Richard Stallman]

    - add a new variable no-old-style-backquotes, with 2 values:
      t   = treat all backquotes and commas as new style
      nil = behave like now (i.e. try to tell which is which)

That would actually break the old code.  I don't want to do
it so abruptly.

For now I just want to warn about them the old-style code,
and handle it as it was intended.

Re: Strange change in backquote expansion

[Richard Stallman]

    Where do we put plans scheduled for Emacs 24?  TODO in the multi-tty
    branch?

TODO in the trunk will do.

Would someone like to implement this?

    > Can you arrange to issue a warning when reading code that uses the old
    > syntax?  There could be a flag to control the warning, and for the moment
    > we could make only the byte compiler turn on the flag.  That would make
    > it convenient if the warnings come thru the compiler warning mechanism.

Re: Strange change in backquote expansion

[Stefan Monnier]

>>> Old Emacs (including 22.0):
>>> (defmacro debug-macro-test (arg)
>>> ``(,,@(list (list arg))))
>> 
>>> (macroexpand '(debug-macro-test foo))
>>> => (list (foo))
>> 
>> I believe the patch below fixes it (installed on the 22 branch).

> Btw: what testcase did you actually fix with the original bug fix?

The font-lock-keywords in log-view.el.


        Stefan

Re: Strange change in backquote expansion

[Stefan Monnier]

>     - add a new variable no-old-style-backquotes, with 2 values:
>       t   = treat all backquotes and commas as new style
>       nil = behave like now (i.e. try to tell which is which)

> That would actually break the old code.  I don't want to do
> it so abruptly.

Old code can still be used as is uncompiled or by setting the
no-old-style-backquotes variable.

> For now I just want to warn about them the old-style code,
> and handle it as it was intended.

OK, I've committed a change on the 22 branch that makes the byte-compiler
display a big fat warning if the file uses old-style backquotes.


        Stefan

Re: Strange change in backquote expansion

[Andreas Schwab]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> Btw: what testcase did you actually fix with the original bug fix?
>
> The font-lock-keywords in log-view.el.

But that does not use nested backquotes, and the source of the problem
was only how backquote templates are formatted by the lisp printer.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Strange change in backquote expansion

[Stefan Monnier]

>>> Btw: what testcase did you actually fix with the original bug fix?
>> The font-lock-keywords in log-view.el.

> But that does not use nested backquotes, and the source of the problem
> was only how backquote templates are formatted by the lisp printer.

Yes, there was a problem when printing, but I also found a problem with the
handling of nested backquotes while trying to debug the problem.

Maybe it occurred when I tried to change the ' into a ` around the whole
expression to see if the bug was linked to the handling of backuotes inside
quotes, as suggested in the original bug report.

Maybe I simply got confused and there was no problem at all, but at least
the original code didn't make much sense to me.  Looking at it again, I see
that indeed the addition of the `level' stuff was not necessary, only the
change in the treatment of a , or ` in the middle of a list was needed.

In hindsight the old way to handle nested backquotes was maybe more elegant
(counting levels implicitly), although it had the disadvantage of traversing
the term N times for a nesting of N.  Feel free to revert that part (but
then plase add a comment next to the double call of backquote-process to
explain why it makes counting levels unnecessary).


        Stefan

Re: Strange change in backquote expansion

[Sascha Wilde]

Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> Old Emacs (including 22.0):
>> (defmacro debug-macro-test (arg)
>>   ``(,,@(list (list arg))))
>
>> (macroexpand '(debug-macro-test foo))
>> => (list (foo))
>
> I believe the patch below fixes it (installed on the 22 branch).

I can confirm that your patch fixes the problem for my example code as
well as for the original code.  Can it be installed to trunk to?

cheers
sascha
-- 
Sascha Wilde
"The primary difference [...] is that the Java program will reliably
and obviously crash, whereas the C program will do something obscure."  
   -- Java Language Tutorial

Re: Strange change in backquote expansion

[Stefan Monnier]

> I can confirm that your patch fixes the problem for my example code as
> well as for the original code.  Can it be installed to trunk to?

It'll get there via the Milesomatic merge from the 22 branch to the trunk,


        Stefan

Re: Strange change in backquote expansion

[Michael Olson]

[-- Attachment #1.1: Type: text/plain, Size: 711 bytes --]

Sascha Wilde <wilde@sha-bang.de> writes:

> Please don't ask, what the code is supposed to do, this is a extremely
> boiled down version of the problem.  I stumbled across this problem
> when I noticed that DVC stopped working in HEAD recently.  (They are
> using a somewhat esoteric macro, which suffers from this change in
> behavior.)

For the record, I modified the esoteric macro in upstream DVC to be less
esoteric and more readable.

-- 
       Michael Olson -- FSF Associate Member #652     |
 http://mwolson.org/ -- Jabber: mwolson_at_hcoop.net  |  /` |\ | | |
            Sysadmin -- Hobbies: Lisp, GP2X, HCoop    | |_] | \| |_|
Projects: Emacs, Muse, ERC, EMMS, ErBot, DVC, Planner |

[-- Attachment #1.2: Type: application/pgp-signature, Size: 188 bytes --]

[-- Attachment #2: Type: text/plain, Size: 142 bytes --]

_______________________________________________
Emacs-devel mailing list
Emacs-devel@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-devel