defmacro with built-in gensym declaration and initialization

defmacro with built-in gensym declaration and initialization

[akater]

[-- Attachment #1: Type: text/plain, Size: 4223 bytes --]

I suggest extending ~defmacro~ to support ~&gensym~ keyword in its
lambda list, for convenient declaration and initialization of gensyms.

Below there's a link to working implementation, and to specification.

A realistic example from popular package =dash=

#+begin_example emacs-lisp
(defmacro --partition-by (form list)
  "Anaphoric form of `-partition-by'."
  (declare (debug (form form)))
  (let ((r (make-symbol "result"))
        (s (make-symbol "sublist"))
        (v (make-symbol "value"))
        (n (make-symbol "new-value"))
        (l (make-symbol "list")))
    `(let ((,l ,list))
       (when ,l
         (let* ((,r nil)
                (it (car ,l))
                (,s (list it))
                (,v ,form)
                (,l (cdr ,l)))
           (while ,l
             (let* ((it (car ,l))
                    (,n ,form))
               (unless (equal ,v ,n)
                 (!cons (nreverse ,s) ,r)
                 (setq ,s nil)
                 (setq ,v ,n))
               (!cons it ,s)
               (!cdr ,l)))
           (!cons (nreverse ,s) ,r)
           (nreverse ,r))))))
#+end_example

could then be rewritten as

#+begin_example emacs-lisp
(defmacro --partition-by ( form list
                           &gensym result sublist value new-value list)
  "Anaphoric form of `-partition-by'."
  (declare (debug (form form)))
  `(when ,list
     (let* ((,result nil)
            (it (car ,list))
            (,sublist (list it))
            (,value ,form)
            (,list (cdr ,list)))
       (while ,list
         (let* ((it (car ,list))
                (,new-value ,form))
           (unless (equal ,value ,new-value)
             (!cons (nreverse ,sublist) ,result)
             (setq ,sublist nil)
             (setq ,value ,new-value))
           (!cons it ,sublist)
           (!cdr ,list)))
       (!cons (nreverse ,sublist) ,result)
       (nreverse ,result))))
#+end_example

Lambda list keyword ~&gensym~, as implemented below, also provides
~once-only~ functionality.  For example, the definition

#+begin_example emacs-lisp
(defmacro with-file-buffer (filename &rest body)
  "Visit FILENAME unless already visited.  Set the buffer as current,
evaluate BODY forms.  Kill the buffer if it did not exist initially."
  (declare (indent 1))
  (let ((o-o-filename (gensym "filename-"))
        (exists-g (gensym "exists-"))
        (buffer-g (gensym "buffer-")))
    `(let* ((,o-o-filename ,filename)
            (,exists-g (get-file-buffer ,o-o-filename))
            (,buffer-g (or ,exists-g (find-file-noselect ,o-o-filename))))
       (unwind-protect (with-current-buffer ,buffer-g ,@body)
         (unless ,exists-g (kill-buffer ,buffer-g))))))
#+end_example

could be rewritten as

#+begin_example emacs-lisp
(defmacro with-file-buffer ( filename &rest body
                             &gensym
                             filename
                             (exists (get-file-buffer filename))
                             (buffer (or exists
                                         (find-file-noselect filename))))
  "Visit FILENAME unless already visited.  Set the buffer as current,
evaluate BODY forms.  Kill the buffer if it did not exist initially."
  (declare (indent 1))
  `(unwind-protect (with-current-buffer ,buffer ,@body)
     (unless ,exists (kill-buffer ,buffer))))
#+end_example

This ~&gensym~ facility eliminates the need for ~with-gensyms~ and
~once-only~ in cases when gensyms are created unconditionally by macro
function (in Common Lisp parlance).

An implementation ~defmacro/&gensym~ is accessible at
git@gitlab.com:akater/defmacro-gensym.git

It is a fairly lengthy (26.6k words) Org file with exposition,
motivation, somewhat extensive tests (for the macro, its variations
and essential dependencies), examples, a ~cl-demacro/&gensym~ version
that supports destructuring lambda lists, and variations which expand
constants early.  There are 118 tests and examples for all the
variations in total.  The file can be tangled to a self-contained
elisp file that provides the macros in question.

Patch not provided because my implementation depends significantly on
~cl-symbol-macrolet~ which doesn't look like it could be used in such
a low-level code.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 865 bytes --]

Re: defmacro with built-in gensym declaration and initialization

[Basil L. Contovounesios]

akater <nuclearspace@gmail.com> writes:

> I suggest extending ~defmacro~ to support ~&gensym~ keyword in its
> lambda list, for convenient declaration and initialization of gensyms.

[...]

> This ~&gensym~ facility eliminates the need for ~with-gensyms~ and
> ~once-only~ in cases when gensyms are created unconditionally by macro
> function (in Common Lisp parlance).

IMO, this minor convenience is insufficient motivation for
conflating/complicating a macro's global arglist, i.e. its arity,
calling convention, etc., with utilities for its local body.  Is there
some other motivation?  Am I missing something?

Why not provide handy gensym/once-only local conveniences for macro
authors instead (some of which already exist in one form or another,
e.g. macroexp-let2, inline-letevals, and org-with-gensyms)?

Thanks,

-- 
Basil

Re: defmacro with built-in gensym declaration and initialization

[Stefan Monnier]

> IMO, this minor convenience is insufficient motivation for
> conflating/complicating a macro's global arglist, i.e. its arity,
> calling convention, etc., with utilities for its local body.  Is there
> some other motivation?  Am I missing something?

Agreed.  This is like the hideous `&aux` of Common Lisp.

> Why not provide handy gensym/once-only local conveniences for macro
> authors instead (some of which already exist in one form or another,
> e.g. macroexp-let2, inline-letevals, and org-with-gensyms)?

Indeed, `macroexp-let2` was designed for similar situations.
I'm not completely happy with it either, so if someone can design
something better, that'd be welcome,


        Stefan

Re: defmacro with built-in gensym declaration and initialization

[Robin Tarsiger]

Stefan Monnier wrote:
> Indeed, `macroexp-let2` was designed for similar situations.
> I'm not completely happy with it either, so if someone can design
> something better, that'd be welcome,

FWIW, I'm generally happy with the conveniences the Alexandria library
provides for in CL: https://common-lisp.net/project/alexandria/draft/alexandria.html#Macro-Writing

once-only is similar to macroexp-let2* but cleaner and more convenient
(though it also acts like let rather than let*). Ad-hoc example:

  (defmacro strange-hypot (x y)
    (once-only (x y (z '(current-z-difference)))
      `(+ (* ,x ,x) (* ,y ,y) (* ,z ,z)))

  (macroexpand '(strange-hypot (+ x dx) 42))
  ==>
  (LET ((#:X634 (+ X DX)) (#:Y635 42) (#:Z636 (CURRENT-Z-DIFFERENCE)))
    (+ (* #:X634 #:X634) (* #:Y635 #:Y635) (* #:Z636 #:Z636)))

Note that the "single evaluation as expanded of one macro argument"
case is very convenient, which is a big step up from Emacs's macroexp-
forms.

with-gensyms is closer to what akater was describing earlier, and is
a "lower-level" helper around let and gensym. Ad-hoc example:

  (defmacro vec3+nums-macro (v1 x2 y2 z2)
    (with-gensyms (v1-var x-var y-var z-var)
      `(let* ((,v1-var ,v1)
              (,x-var (+ ,x2 (elt ,v1-var 0)))
              (,y-var (+ ,y2 (elt ,v1-var 1)))
              (,z-var (+ ,z2 (elt ,v1-var 2))))
         (vec3 ,x-var ,y-var ,z-var))))

  (macroexpand '(vec3+nums-macro (acquire-vector (somehow))
                                 (+ previous-x dx) y -1.0))
  ==>
  (LET* ((#:V1-VAR637 (ACQUIRE-VECTOR (SOMEHOW)))
         (#:X-VAR638 (+ (+ PREVIOUS-X DX) (ELT #:V1-VAR637 0)))
         (#:Y-VAR639 (+ Y (ELT #:V1-VAR637 1)))
         (#:Z-VAR640 (+ -1.0 (ELT #:V1-VAR637 2))))
    (VEC3 #:X-VAR638 #:Y-VAR639 #:Z-VAR640))

Note that the let* form is not convenient in once-only, and
the Alexandria helpers don't do any of the constant-related
optimization that Emacs's do AFAIK, but the possible slight
adjustments there are fairly obvious.

-RTT

Re: defmacro with built-in gensym declaration and initialization

[akater]

[-- Attachment #1: Type: text/plain, Size: 2026 bytes --]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> IMO, this minor convenience is insufficient motivation for
> conflating/complicating a macro's global arglist, i.e. its arity,
> calling convention, etc., with utilities for its local body.  Is there
> some other motivation?  Am I missing something?

My motivation mostly comes from CL where complicated lambda lists are
not considered an issue, and it didn't come to my mind it could be an
issue for Emacs Lisp users.  In particular, &gensym does indeed follow
the design of &aux, described below as “hideous” (another surprise).  I
use &aux regularly, and I find it a nice addition as it reduces the
nesting depth of defun and defmacro forms.  &gensym does that and saves
some boilerplate on top of it, I tried it in real life macros, found it
useful, and that's pretty much it.

> conflating/complicating a macro's global arglist, i.e. its arity,

What's the arity of defmacro?  This form already allows any number of
arguments so as far as I can see adding another keyword does not change
arity.  Calling convention is backwards compatible.  Again, Common Lisp
implementations are allowed to introduce their own lambda list keywords
so I didn't (and still don't) think extending lambda-list keywords set
is a big deal.

> Why not provide handy gensym/once-only local conveniences for macro
> authors instead (some of which already exist in one form or another,
> e.g. macroexp-let2, inline-letevals, and org-with-gensyms)?

I don't have much experience with macroexp-let2 which on the first
glance looks like an overcomplicated once-only that is mostly relevant
for functions that are meant to be byte-compiled.  Since we're moving to
natively compiled Elisp, I was thinking it's going to become less
relevant in near future, and &gensym covers most use cases of once-only.

Again, org-with-gensyms is precisely something that's being replaced
here with an alternative that is less verbose and has a decreased
nesting depth.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 865 bytes --]

Re: defmacro with built-in gensym declaration and initialization

[Basil L. Contovounesios]

akater <nuclearspace@gmail.com> writes:

> "Basil L. Contovounesios" <contovob@tcd.ie> writes:
>
>> IMO, this minor convenience is insufficient motivation for
>> conflating/complicating a macro's global arglist, i.e. its arity,
>> calling convention, etc., with utilities for its local body.  Is there
>> some other motivation?  Am I missing something?
>
> My motivation mostly comes from CL where complicated lambda lists are
> not considered an issue, and it didn't come to my mind it could be an
> issue for Emacs Lisp users.  In particular, &gensym does indeed follow
> the design of &aux, described below as “hideous” (another surprise).

I'd seen &aux before but didn't look up what it does until now, and I
must say I share Stefan's sentiment :).

> I use &aux regularly, and I find it a nice addition as it reduces the
> nesting depth of defun and defmacro forms.  &gensym does that and saves
> some boilerplate on top of it, I tried it in real life macros, found it
> useful, and that's pretty much it.

Let's just say I would sooner see native arglists gain support for
keyword arguments ;).

For non-native arglists, we could always extend cl-defmacro or some
other definition definer.

>> conflating/complicating a macro's global arglist, i.e. its arity,
>
> What's the arity of defmacro?  This form already allows any number of
> arguments so as far as I can see adding another keyword does not change
> arity.  Calling convention is backwards compatible.

I was referring to the arity of the macro being defined, not that of
defmacro.

> Again, Common Lisp
> implementations are allowed to introduce their own lambda list keywords
> so I didn't (and still don't) think extending lambda-list keywords set
> is a big deal.

I can't imagine it being as little a deal in Elisp, but that's just my
impression.

>> Why not provide handy gensym/once-only local conveniences for macro
>> authors instead (some of which already exist in one form or another,
>> e.g. macroexp-let2, inline-letevals, and org-with-gensyms)?
>
> I don't have much experience with macroexp-let2 which on the first
> glance looks like an overcomplicated once-only that is mostly relevant
> for functions that are meant to be byte-compiled.

It and its variant macroexp-let2* are relevant wherever the macro author
wants to avoid evaluating an argument more than once, but improvements
are always welcome.

> Since we're moving to
> natively compiled Elisp, I was thinking it's going to become less
> relevant in near future, and &gensym covers most use cases of once-only.

I don't see how native compilation changes how existing and new Elisp
macros ought to be written.

> Again, org-with-gensyms is precisely something that's being replaced
> here with an alternative that is less verbose and has a decreased
> nesting depth.

Just my 2 cents,

-- 
Basil

Re: defmacro with built-in gensym declaration and initialization

[akater]

[-- Attachment #1: Type: text/plain, Size: 1998 bytes --]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> Let's just say I would sooner see native arglists gain support for
> keyword arguments ;).
>
> For non-native arglists, we could always extend cl-defmacro or some
> other definition definer.

Sorry, I suspect I misunderstood.  What's a native arglist?  Doesn't
defmacro already have native support for &optional and &rest keywords?

> I was referring to the arity of the macro being defined, not that of
> defmacro.

I guess I'm confused, again.  The arity of macro being defined remains
exactly the same.  See the --partition-by example in the original
message.  defmacro/&gensym is a drop-in replacement; it doesn't change
any arities.

> It and its variant macroexp-let2* are relevant wherever the macro author
> wants to avoid evaluating an argument more than once, but improvements
> are always welcome.

That's usually called “once-only” but unlike once-only, macroexp-let2
also requires a test function which is exactly why I called it “an
overcomplicated once-only”.  &gensym is also relevant wherever the macro
author wants to avoid evaluating an argument more than once, and where
the argument needs to be evaluated unconditionally, but it's more
concise, both in terms of token count and nesting depth.  One variation
of defmacro/&gensym accessible in the linked repository does perform the
same elimination of bindings as macroexp-let2, only it uses a hardcoded
test function, namely macroexp-const-p.

>> Since we're moving to
>> natively compiled Elisp, I was thinking it's going to become less
>> relevant in near future, and &gensym covers most use cases of once-only.
>
> I don't see how native compilation changes how existing and new Elisp
> macros ought to be written.

What's the purpose of TEST in macroexp-let2, then, other than to
minimise bindings in the expansion?  (Which I presume is only relevant
when Elisp's byte compiler is used to compile the expanded form.)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 865 bytes --]

Re: defmacro with built-in gensym declaration and initialization

[Stefan Monnier]

> What's the purpose of TEST in macroexp-let2, then, other than to
> minimise bindings in the expansion?

The purpose is indeed just one of optimization applied to the
source code.

> (Which I presume is only relevant when Elisp's byte compiler is used
> to compile the expanded form.)

It's relevant both when the code is byte-compiled and when
it's interpreted.  But that depends on implementation details, so indeed
it can become useless in the future if/when we improve the interpreter
and/or compiler.  I don't know if it's still relevant when using the
native compiler, indeed.


        Stefan

Re: defmacro with built-in gensym declaration and initialization

[Basil L. Contovounesios]

akater <nuclearspace@gmail.com> writes:

> "Basil L. Contovounesios" <contovob@tcd.ie> writes:
>
>> Let's just say I would sooner see native arglists gain support for
>> keyword arguments ;).
>>
>> For non-native arglists, we could always extend cl-defmacro or some
>> other definition definer.
>
> Sorry, I suspect I misunderstood.  What's a native arglist?  Doesn't
> defmacro already have native support for &optional and &rest keywords?

Yes, and that's what I mean by native arglists - the ones possessed by
built-in C objects such as subroutines, lambda expressions, compiled
code, dynamic module functions, etc. and used by built-in C functions
such as funcall.

By contrast, cl-defmacro, other CL compatibility definitions, etc. have
to parse a plain Elisp arglist for CL-specific features like &aux.

>> I was referring to the arity of the macro being defined, not that of
>> defmacro.
>
> I guess I'm confused, again.  The arity of macro being defined remains
> exactly the same.  See the --partition-by example in the original
> message.  defmacro/&gensym is a drop-in replacement; it doesn't change
> any arities.

Yes, I understood that.  Here's what I said:

> IMO, this minor convenience is insufficient motivation for
> conflating/complicating a macro's global arglist, i.e. its arity,
  ^^^^^^^^^^^^^^^^^^^^^^^
> calling convention, etc., with utilities for its local body.

The bottom line is that argument lists are for arguments, i.e. the
public contract between the caller and the callee.  Local variables are
not part of this contract, so putting them there is misguided, to put it
politely, especially if the only motivation is to save a few keystrokes.

I suspect that adding support for something like &gensym to native Elisp
arglists would amount to a very intrusive change for a very minor
convenience (and IMO very unaesthetic anti-pattern).

Adding support for something like &gensym to non-native arglists such as
those of pcase-lambda, cl-defmacro, etc. would probably be more
feasible, but that doesn't necessarily mean we should do it either ;).

>> It and its variant macroexp-let2* are relevant wherever the macro author
>> wants to avoid evaluating an argument more than once, but improvements
>> are always welcome.
>
> That's usually called “once-only” but unlike once-only, macroexp-let2
> also requires a test function which is exactly why I called it “an
> overcomplicated once-only”.

Then why not improve macroexp-let2 and/or provide once-only, like I
suggested in my first message, without the need for &gensym?

> &gensym is also relevant wherever the macro
> author wants to avoid evaluating an argument more than once, and where
> the argument needs to be evaluated unconditionally, but it's more
> concise, both in terms of token count and nesting depth.  One variation
> of defmacro/&gensym accessible in the linked repository does perform the
> same elimination of bindings as macroexp-let2, only it uses a hardcoded
> test function, namely macroexp-const-p.
>
>>> Since we're moving to
>>> natively compiled Elisp, I was thinking it's going to become less
>>> relevant in near future, and &gensym covers most use cases of once-only.
>>
>> I don't see how native compilation changes how existing and new Elisp
>> macros ought to be written.
>
> What's the purpose of TEST in macroexp-let2, then, other than to
> minimise bindings in the expansion?  (Which I presume is only relevant
> when Elisp's byte compiler is used to compile the expanded form.)

As you say, it's relevant whenever you want to minimise bindings in the
expansion.  I still don't see the relation with native compilation.

-- 
Basil

Re: defmacro with built-in gensym declaration and initialization

[Stefan Monnier]

>> Sorry, I suspect I misunderstood.  What's a native arglist?  Doesn't
>> defmacro already have native support for &optional and &rest keywords?
> Yes, and that's what I mean by native arglists - the ones possessed by
> built-in C objects such as subroutines, lambda expressions, compiled
> code, dynamic module functions, etc. and used by built-in C functions
> such as funcall.
> By contrast, cl-defmacro, other CL compatibility definitions, etc. have
> to parse a plain Elisp arglist for CL-specific features like &aux.

FWIW, `defmacro` is itself a macro, just like `cl-defmacro`, so it
wouldn't be that terribly hard (the only problem is that it needs to be
implemented without itself using macros, more or less).

For me it's more a question that whatever something like `&gensym` can't
use values that are computed during the macro expansion, , so

    (defmacro M (ARGS &gensym FOO)
      ...)

is equivalent to something like:

    (defmacro M (ARGS)
      (with-gensyms (FOO)
        ...))

except:
- it's a bit more concise
- it exposes internal implementation details into the arglist, which is
  usually considered as part of the signature.
- it doesn't cover the cases where you need `with-gensyms` deeper than
  at the toplevel of the macro's body.

I find the added concision doesn't pay for the other two downsides.


        Stefan

Re: defmacro with built-in gensym declaration and initialization

[Basil L. Contovounesios]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Sorry, I suspect I misunderstood.  What's a native arglist?  Doesn't
>>> defmacro already have native support for &optional and &rest keywords?
>> Yes, and that's what I mean by native arglists - the ones possessed by
>> built-in C objects such as subroutines, lambda expressions, compiled
>> code, dynamic module functions, etc. and used by built-in C functions
>> such as funcall.
>> By contrast, cl-defmacro, other CL compatibility definitions, etc. have
>> to parse a plain Elisp arglist for CL-specific features like &aux.
>
> FWIW, `defmacro` is itself a macro, just like `cl-defmacro`, so it
> wouldn't be that terribly hard (the only problem is that it needs to be
> implemented without itself using macros, more or less).

Oh, right.

> For me it's more a question that whatever something like `&gensym` can't
> use values that are computed during the macro expansion, , so
>
>     (defmacro M (ARGS &gensym FOO)
>       ...)
>
> is equivalent to something like:
>
>     (defmacro M (ARGS)
>       (with-gensyms (FOO)
>         ...))
>
> except:
> - it's a bit more concise
> - it exposes internal implementation details into the arglist, which is
>   usually considered as part of the signature.
> - it doesn't cover the cases where you need `with-gensyms` deeper than
>   at the toplevel of the macro's body.
>
> I find the added concision doesn't pay for the other two downsides.

Agreed,

-- 
Basil