Double unquote/unquote-splicing

Double unquote/unquote-splicing

[Nathan Trapuzzano]

This was a subject of discussion not too long ago.  Someone wrote to say
how unquote and unquote-splicing don't pile up properly (as in ,,@FORM
or ,@,@FORM).  In fact, they do work properly in the case that the inner
unquote-splicing expands to just one form, but this is hardly a useful
case (it's easily re-writable in other terms).  On the other hand, there
are useful cases of ,,@ and ,@,@ that cannot be written without
rendering the code virtually unreadable.  Consider:

(defmacro once-only (names &rest body)
  (let ((gensyms (loop for n in names collect (gensym))))
    `(let (,@(loop for g in gensyms collect `(,g (gensym))))
      `(let (,,@(loop for g in gensyms for n in names collect ``(,,g ,,n)))
        ,(let (,@(loop for n in names for g in gensyms collect `(,n ,g)))
           ,@body)))))

(defmacro once-only (names &rest body)
  (let ((gensyms (loop for n in names collect (gensym))))
    `(let (,@(loop for g in gensyms collect `(,g (gensym))))
       (append (list `let)
               (list (append
                      (list ,@(loop for g in gensyms for n in names
                                    collect `(append (list ,g) (list ,n))))))
               (list
                (let (,@(loop for n in names for g in gensyms
                              collect `(,n ,g)))
                       ,@body))))))

These two macros are semantically equivalent, but one is infinitely more
easy to read thanks to ,,@.  However, only the latter form works in
Emacs when NAMES has length > 1.

For the sake of readability, I'd like to propose properly implementing
,,@ and ,@,@ etc.  Is this doable without major changes to the current
backquote implementation?  If so, is there some other reason why this
should _not_ be done?

Nathan

Re: Double unquote/unquote-splicing

[Stefan Monnier]

> (defmacro once-only (names &rest body)
>   (let ((gensyms (loop for n in names collect (gensym))))
>     `(let (,@(loop for g in gensyms collect `(,g (gensym))))
>       `(let (,,@(loop for g in gensyms for n in names collect ``(,,g ,,n)))
>         ,(let (,@(loop for n in names for g in gensyms collect `(,n ,g)))
>            ,@body)))))

There are two reasons why I'm resisting it:
1- the above code looks clean, but I can't understand it at all.
   Your alternative code is less clean and not easy to understand, but
   I *can* understand it.
2- Since we don't allow (\` (a (\, 1 2 3))), it's weird to allow
   ``(a ,,@x) since one possible expansion for it when x=(1 2 3) is
   (\` (a (\, 1 2 3))).
   Another way to say it is that we should allow (\, 1 2 3) and (\,@
   1 2 3), but that can't be used with the ,e and ,@e syntax, so it'll
   stay as a second-rate citizen.

Isn't the above the same as

   (defmacro once-only (names &rest body)
     (let ((gensyms (loop for n in names collect (gensym))))
       `(let (,@(loop for g in gensyms collect `(,g (gensym))))
         `(let (,@(list ,@(loop for g in gensyms for n in names collect ``(,,g ,,n))))
           ,(let (,@(loop for n in names for g in gensyms collect `(,n ,g)))
              ,@body)))))


-- Stefan

Re: Double unquote/unquote-splicing

[Nathan Trapuzzano]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> There are two reasons why I'm resisting it:
> 1- the above code looks clean, but I can't understand it at all.
>    Your alternative code is less clean and not easy to understand, but
>    I *can* understand it.

We'll just have to disagree there.  But even granting that the latter is
easier to understand, I can't imagine anyone figuring out how to write
it from scratch.

> 2- Since we don't allow (\` (a (\, 1 2 3))), it's weird to allow
>    ``(a ,,@x) since one possible expansion for it when x=(1 2 3) is
>    (\` (a (\, 1 2 3))).
>    Another way to say it is that we should allow (\, 1 2 3) and (\,@
>    1 2 3), but that can't be used with the ,e and ,@e syntax, so it'll
>    stay as a second-rate citizen.

This is incorrect.  ``(a ,,@x) where x=(1 2 3) would evaluate to
(\` (a (\, 1) (\, 2) (\, 3))).  The first comma in ,,@ has the effect of
being applied member-wise to each element spliced out of ,@.  Cf. CLHS
`Backquote' (http://www.lispworks.com/documentation/HyperSpec/Body/02_df.htm).

> Isn't the above the same as
>
>    (defmacro once-only (names &rest body)
>      (let ((gensyms (loop for n in names collect (gensym))))
>        `(let (,@(loop for g in gensyms collect `(,g (gensym))))
>          `(let (,@(list ,@(loop for g in gensyms for n in names collect ``(,,g ,,n))))
>            ,(let (,@(loop for n in names for g in gensyms collect `(,n ,g)))
>               ,@body)))))

I believe so.  (FWIW, I don't think it's that easy to tell, though I'm
not great at this.  I derived the second form I presented from the first
form step-by-step using the rules in CLHS.)

The bottom line to me is that the behavior of Elisp differs from both CL
and Scheme, and there doesn't seem to be a good reason for it.

In further support of my argument :)

"The backquote syntax was particularly powerful when nested. This
occurred primarily within macro-defining macros; because such were coded
primarily by wizards, the ability to write and interpret nested
backquote expressions was soon surrounded by a certain mystique. Alan
Bawden of MIT acquired a particular reputation as backquote-meister in
the early days of the Lisp Machine." - "The Evolution of Lisp", Gabriel,
Steele.

I lifted that quote from a StackOverflow thread and have no idea whether
it's authentic.  But I think it's true.

Re: Double unquote/unquote-splicing

[Nathan Trapuzzano]

Nathan Trapuzzano <nbtrap@nbtrap.com> writes:

> This is incorrect.  ``(a ,,@x) where x=(1 2 3) would evaluate to
> (\` (a (\, 1) (\, 2) (\, 3))).  The first comma in ,,@ has the effect of
> being applied member-wise to each element spliced out of ,@.  Cf. CLHS
> `Backquote' (http://www.lispworks.com/documentation/HyperSpec/Body/02_df.htm).

By the way, like in Common Lisp, what backquote/unquote/unquote-splicing
expand to is not defined.  What's defined is the effect of their
evaluation, so what I presented here doesn't _have_ to be the way ,,@
works.  For example, in Elisp, the expression in question may very well
expand to (\` (a (\, 1 2 3))) (as you say) so long as it is semantically
equivalent to (\` (a (\, 1) (\, 2) (\, 3))).

Re: Double unquote/unquote-splicing

[Stefan Monnier]

>> 2- Since we don't allow (\` (a (\, 1 2 3))), it's weird to allow
>> ``(a ,,@x) since one possible expansion for it when x=(1 2 3) is
>> (\` (a (\, 1 2 3))).
>> Another way to say it is that we should allow (\, 1 2 3) and (\,@
>> 1 2 3), but that can't be used with the ,e and ,@e syntax, so it'll
>> stay as a second-rate citizen.

> This is incorrect.  ``(a ,,@x) where x=(1 2 3) would evaluate to
> (\` (a (\, 1) (\, 2) (\, 3))).  The first comma in ,,@ has the effect of
> being applied member-wise to each element spliced out of ,@.  Cf. CLHS
> `Backquote' (http://www.lispworks.com/documentation/HyperSpec/Body/02_df.htm).

This is even worse since it means that the expansion of ,@foo depends on
the context.

> I believe so.  (FWIW, I don't think it's that easy to tell, though I'm
> not great at this.

Back to problem nb 1, eh?

> "The backquote syntax was particularly powerful when nested. This
> occurred primarily within macro-defining macros; because such were coded
> primarily by wizards, the ability to write and interpret nested
> backquote expressions was soon surrounded by a certain mystique. Alan
> Bawden of MIT acquired a particular reputation as backquote-meister in
> the early days of the Lisp Machine." - "The Evolution of Lisp", Gabriel,
> Steele.

That sounds about right: it's only for wizards.

Nested backquotes were largely broken in Elisp and it took many years
for someone to notice.  I do use them occasionally, but only in fairly
simple ways.  The resulting code is largely impenetrable, so I don't
want to encourage it.


        Stefan

RE: Double unquote/unquote-splicing

[Drew Adams]

> > "The backquote syntax was particularly powerful when nested. This
> > occurred primarily within macro-defining macros; because such were
> > coded primarily by wizards, the ability to write and interpret nested
> > backquote expressions was soon surrounded by a certain mystique.
> > Alan Bawden of MIT acquired a particular reputation as backquote-
> > meister in the early days of the Lisp Machine." - "The Evolution of
> > Lisp", Gabriel, Steele.
> 
> That sounds about right: it's only for wizards.

No, that is exactly backwards.  Please note: "The backquote syntax was
__particularly powerful when nested__."  That's about power simplifying
understanding by humans.

Read the article for explanation and good examples.  The point of that
passage is that _without_ backquote syntax the kinds of things you can
do using nested backquotes are *really* complicated without them.  The
point is that backquote syntax is _especially_ good at simplifying in
the case of nested backquotes.

What is for wizards is the need to write code of this kind: the kinds
of things that nested backquotes simplify.  Those use cases are not so
common, but if you have such a use case then you _really_ want to have
nested backquote syntax.  That's the point of the quoted passage.

The full article is here:
http://extravagaria.com/Files/HOPL2-Uncut.pdf

> Nested backquotes were largely broken in Elisp and it took many
> years for someone to notice.  I do use them occasionally, but only in
> fairly simple ways.  The resulting code is largely impenetrable, so
> I don't want to encourage it.

Nested backquote syntax should be neither encouraged nor discouraged.
The point is that compared to the alternative it is a lot simpler.

Re: Double unquote/unquote-splicing

[Nathan Trapuzzano]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> This is incorrect.  ``(a ,,@x) where x=(1 2 3) would evaluate to
>> (\` (a (\, 1) (\, 2) (\, 3))).  The first comma in ,,@ has the effect of
>> being applied member-wise to each element spliced out of ,@.  Cf. CLHS
>> `Backquote' (http://www.lispworks.com/documentation/HyperSpec/Body/02_df.htm).
>
> This is even worse since it means that the expansion of ,@foo depends on
> the context.

Not necessarily.  As I said, you could simply define \, and \,@ with
more than one argument to behave the same as a series of \, or \,@s with
one argument.

In fact, I just built from trunk with two lines of code from
backquote.el changed (using your helpful comments), and it seems to work
precisely like that.  In other words, this doesn't seem to require any
extra work.

>> "The backquote syntax was particularly powerful when nested. This
>> occurred primarily within macro-defining macros; because such were coded
>> primarily by wizards, the ability to write and interpret nested
>> backquote expressions was soon surrounded by a certain mystique. Alan
>> Bawden of MIT acquired a particular reputation as backquote-meister in
>> the early days of the Lisp Machine." - "The Evolution of Lisp", Gabriel,
>> Steele.
>
> That sounds about right: it's only for wizards.

On second thought, that quote doesn't really support my argument.  It's
rather neutral.

However, the fact is that nested backquotes _are_ for wizards.  The
problem is, Emacs' nested backquotes aren't fully functional, the result
being that the code is harder for us non-wizards to read.

In summary, this change would

1. break nothing,
2. require virtually no work to be done,
3. render macro-writing macros easier to read and write; and
4. bring Elisp behavior right in line with CL and Scheme.

Even if you disagree with (3), this seems like a win all around.

Re: Double unquote/unquote-splicing

[Stefan Monnier]

> In fact, I just built from trunk with two lines of code from
> backquote.el changed (using your helpful comments), and it seems to work
> precisely like that.  In other words, this doesn't seem to require any
> extra work.

I know (obviously), but I rather dislike this weird intermediate state,
which doesn't fit cleanly in the usual semantics of unquote (whose
special syntax makes it clear that it's supposed to have just one
argument).

The fact that Common-Lisp found a way to extend the semantics in a way
that makes some sense is fine for Common-Lips, but not enough to
convince me.

Your example was the first concrete example I've seen where such
a ,,@foo construct was used and as you've seen it's trivially replaced
by ,@(list ,@foo) (which is true because ,foo is equivalent to ,@(list
foo).

So ,,@ is not a very compelling use.  Maybe ,@,@ would be more
compelling, but I haven't seen any concrete use for it yet, so
I can't comment.


        Stefan

Re: Double unquote/unquote-splicing

[Stephen J. Turnbull]

Nathan Trapuzzano writes:

 > However, the fact is that nested backquotes _are_ for wizards.  The
 > problem is, Emacs' nested backquotes aren't fully functional, the result
 > being that the code is harder for us non-wizards to read.

Like Stefan, I couldn't understand what your example is supposed to do
*without reading the expansion*.  That I do understand, though it took
some head-scratching.

"It's pretty, so it must be right" is a fallacy.  The temptation to
write what really should be a cond with a large number of exceptional
cases as a backquote construct looping over applications of a general
function is probably not good for Emacs.  Emacs is about user
interface: its beauty mostly is, and generally should be, "skin deep".

 > In summary, this change would
 > 
 > 1. break nothing,
 > 2. require virtually no work to be done,
 > 3. render macro-writing macros easier to read and write; and
 > 4. bring Elisp behavior right in line with CL and Scheme.
 > 
 > Even if you disagree with (3), this seems like a win all around.

AFAICS, (1) hasn't been properly checked, and one can also disagree
with (4), as phrased.  There will still be plenty of differences in
behavior.  Only nested backquotes (an infrequently used facility) will
be synchronized.

Note also that "powerful" (== expressive) syntax is not necessarily
*efficient* syntax.  This is especially so for Emacs Lisp.

I don't really see why people want to write wizardly code in Emacs
Lisp.  Why not use a language designed for that purpose instead?

Emacs Lisp is a slow implementation (though with lexical scope it can
get a lot faster, it is unlikely to approach the performance of
well-tuned Common Lisp implementations, let alone the faster Scheme
implementations) with idiosyncratic, unstandardized semantics that
change at the whim of the maintainers.  This doesn't matter much in
its primary role as a user interface (nothing in the computing
environment is as slow as the user ;-).  But UIs are typically a
matter of attention to detail and context, rather than complex
algorithms.

I'm not really against this, but if it makes Stefan uneasy, I'll side
with Stefan's ambiguous intuition against the rather tenuous claims in
favor.

Re: Double unquote/unquote-splicing

[Nathan Trapuzzano]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I know (obviously), but I rather dislike this weird intermediate state,
> which doesn't fit cleanly in the usual semantics of unquote (whose
> special syntax makes it clear that it's supposed to have just one
> argument).

I don't think the speical syntax does make that clear.  The point is
that the intermediate state (the state of the object that gets passed
from the reader to the evaluator) is itself not defined.  And even if it
were defined (as in Scheme), I don't see why \, and \@, couldn't take
more than one argument (again, as with Scheme's unquote and
unquote-splicing special operators).

> Your example was the first concrete example I've seen where such
> a ,,@foo construct was used and as you've seen it's trivially replaced
> by ,@(list ,@foo) (which is true because ,foo is equivalent to ,@(list
> foo).

Hmm, I'm actually not sure that will always work.  My hunch is that it
only works consistenly at the most deeply nested level.  (I say so
because I know CL's expansion rules are only guaranteed to be
semantically equivalent when the most deeply nested levels are expanded
first and then the rest of the expansion proceeds outwards.)

> So ,,@ is not a very compelling use.  Maybe ,@,@ would be more
> compelling, but I haven't seen any concrete use for it yet, so I can't
> comment.

Can't say I know of a good use of ,@,@.  That said, I think it would be
silly for this proposal to stand or fall on that basis.

Re: Double unquote/unquote-splicing

[Nathan Trapuzzano]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> "It's pretty, so it must be right" is a fallacy.

I haven't argued in these terms at all.

>  > In summary, this change would
>  > 
>  > 1. break nothing,
>  > 2. require virtually no work to be done,
>  > 3. render macro-writing macros easier to read and write; and
>  > 4. bring Elisp behavior right in line with CL and Scheme.
>  > 
>  > Even if you disagree with (3), this seems like a win all around.
>
> AFAICS, (1) hasn't been properly checked, and one can also disagree
> with (4), as phrased.  There will still be plenty of differences in
> behavior.  Only nested backquotes (an infrequently used facility) will
> be synchronized.

I suspect Stefan would agree with me about (1).  This proposal would
allow syntax that currently signals an error.  If your code doesn't
already throw said error, this won't change anything.

As for (4), of course I only meant _in this instance_ Elisp semantics
would be equivalent to CL's and Scheme's.

> I'm not really against this, but if it makes Stefan uneasy, I'll side
> with Stefan's ambiguous intuition against the rather tenuous claims in
> favor.

I think the only claim that can be legitimately said to be "tenuous" is
(3), given its subjective nature.  It seems to me that the other points
are simply true.

Re: Double unquote/unquote-splicing

[Stefan Monnier]

> I don't think the special syntax does make that clear.

You can't use that special syntax for multi-arg unquote and
unquote/unquote-splicing.  So the special syntax makes it clear that
those are expected to take a single arg.  And indeed, so far, they
can only take a single arg.

>> Your example was the first concrete example I've seen where such
>> a ,,@foo construct was used and as you've seen it's trivially replaced
>> by ,@(list ,@foo) (which is true because ,foo is equivalent to ,@(list
>> foo).
> Hmm, I'm actually not sure that will always work.

Goes to show again that it's too complex for our own good.

>> So ,,@ is not a very compelling use.  Maybe ,@,@ would be more
>> compelling, but I haven't seen any concrete use for it yet, so I can't
>> comment.
> Can't say I know of a good use of ,@,@.  That said, I think it would be
> silly for this proposal to stand or fall on that basis.

There's a straightforward rewriting of ",,@", and there's no known use
case for ",@,@" (which AFAIK can be rewritten ",@(append ,@" anyway if
you really need it).  It probably doesn't break any code, but I prefer
to keep the rule that unquote/unquote-splicing takes a single argument,
since that's true 99.99% of the time, so while people are likely to
understand the single-arg case, and they're unlikely to understand the
multi-arg case.


        Stefan

Re: Double unquote/unquote-splicing

[Stephen J. Turnbull]

Nathan Trapuzzano writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > 
 > > "It's pretty, so it must be right" is a fallacy.
 > 
 > I haven't argued in these terms at all.

That is true, and it doesn't matter.  It's an easy trap for others to
fall into, and it's a particularly dangerous one for Emacs, which is
intentionally permissive toward irregular hacks.

 > I think the only claim that can be legitimately said to be
 > "tenuous" is (3), given its subjective nature.  It seems to me that
 > the other points are simply true.

You've already admitted that they're not.  The best you can say is
that in your opinion the probability of doing any harm is negligible.