A macro containing a mini-macro?

A macro containing a mini-macro?

[HiPhish]

Hello Schemers,

I have written a small macro for writing test specifications:

    (define-syntax test-cases
      (syntax-rules ()
        ((_ title
           (given (byte byte* ...))
           ...)
         (begin
           (test-begin title)
           (call-with-values (λ () (open-bytevector-output-port))
             (λ (out get-bv)
               (pack given out)
               (let ((received (get-bv))
                     (expected (u8-list->bytevector '(byte byte* ...))))
                 (test-assert (bytevector=? received expected)))))
           ...
           (test-end title)))))

The idea is that I can specify a series of test cases where each case consists 
of an object and a sequence of bytes which this object is to be serialized to:

    (test-cases "Single precision floating point numbers"
      (+3.1415927410125732 (#xCA #b01000000 #b01001001 #b00001111 #b11011011))
      (-3.1415927410125732 (#xCA #b11000000 #b01001001 #b00001111 
#b11011011)))

This works fine, but sometimes there is a sequence of the same bytes and it 
would be more readable if I could write something like this:

    ((make-vector 16 0) (#xDC (16 #x00)))

instead of writing out 16 times `#x00`. This would require being able to make 
a distinction in the pattern whether `byte` is of the pattern

    byte

or

    (count byte)

and if it's the latter construct a list of `count` `byte`s (via `(make-list 
count byte)` for example) and splice it in. This distinction needs to be made 
for each byte specification because I want to mix actual bytes and these "RLE-
encoded" byte specifications.

So I guess what I'm looking for is to have a `syntax-rules` inside a `syntax-
rules` in a way. Can this be done?

Re: A macro containing a mini-macro?

[rain1]

On 2018-09-13 23:04, HiPhish wrote:
> Hello Schemers,
> 
> I have written a small macro for writing test specifications:
> 
>     (define-syntax test-cases
>       (syntax-rules ()
>         ((_ title
>            (given (byte byte* ...))
>            ...)
>          (begin
>            (test-begin title)
>            (call-with-values (λ () (open-bytevector-output-port))
>              (λ (out get-bv)
>                (pack given out)
>                (let ((received (get-bv))
>                      (expected (u8-list->bytevector '(byte byte* 
> ...))))
>                  (test-assert (bytevector=? received expected)))))
>            ...
>            (test-end title)))))
> 
> The idea is that I can specify a series of test cases where each case 
> consists
> of an object and a sequence of bytes which this object is to be 
> serialized to:
> 
>     (test-cases "Single precision floating point numbers"
>       (+3.1415927410125732 (#xCA #b01000000 #b01001001 #b00001111 
> #b11011011))
>       (-3.1415927410125732 (#xCA #b11000000 #b01001001 #b00001111
> #b11011011)))
> 
> This works fine, but sometimes there is a sequence of the same bytes 
> and it
> would be more readable if I could write something like this:
> 
>     ((make-vector 16 0) (#xDC (16 #x00)))
> 
> instead of writing out 16 times `#x00`. This would require being able 
> to make
> a distinction in the pattern whether `byte` is of the pattern
> 
>     byte
> 
> or
> 
>     (count byte)
> 
> and if it's the latter construct a list of `count` `byte`s (via 
> `(make-list
> count byte)` for example) and splice it in. This distinction needs to 
> be made
> for each byte specification because I want to mix actual bytes and 
> these "RLE-
> encoded" byte specifications.
> 
> So I guess what I'm looking for is to have a `syntax-rules` inside a 
> `syntax-
> rules` in a way. Can this be done?

You can implement the DSL that transforms bytevector descriptions like 
(#xDC (16 #x00)) into a bytevector as a procedure, suppose we call it 
byte-dsl. Then you only need to change u8-list->bytevector with 
byte-dsl. This lets you do what you wanted without the difficult task of 
macros inside macros.

Re: A macro containing a mini-macro?

[HiPhish]

I don't quite follow; having a macro for the byte DSL is simple:

    (define-syntax byte-dsl
       (syntax-rules ()
        ((_ byte) (list byte))
        ((_ count byte) (make-list count byte byte))))

But this requires every byte specification to be written as `(byte-dsl 0x00)`, 
which I want to avoid in my test-case DSL. I want to be able to write

    (test-case 13 (#x01 #x02 (4 #xAB) #xFF))

This requires being able to match either `byte` or `(amount byte)` inside the 
test-case pattern.

rain1@airmail.cc wrote:
> You can implement the DSL that transforms bytevector descriptions like
> (#xDC (16 #x00)) into a bytevector as a procedure, suppose we call it
> byte-dsl. Then you only need to change u8-list->bytevector with
> byte-dsl. This lets you do what you wanted without the difficult task of
> macros inside macros.

Re: A macro containing a mini-macro?

[Mark H Weaver]

Hi,

HiPhish <hiphish@posteo.de> writes:

> Hello Schemers,
>
> I have written a small macro for writing test specifications:
>
>     (define-syntax test-cases
>       (syntax-rules ()
>         ((_ title
>            (given (byte byte* ...))
>            ...)
>          (begin
>            (test-begin title)
>            (call-with-values (λ () (open-bytevector-output-port))
>              (λ (out get-bv)
>                (pack given out)
>                (let ((received (get-bv))
>                      (expected (u8-list->bytevector '(byte byte* ...))))
>                  (test-assert (bytevector=? received expected)))))
>            ...
>            (test-end title)))))
>
> The idea is that I can specify a series of test cases where each case consists 
> of an object and a sequence of bytes which this object is to be serialized to:
>
>     (test-cases "Single precision floating point numbers"
>       (+3.1415927410125732 (#xCA #b01000000 #b01001001 #b00001111 #b11011011))
>       (-3.1415927410125732 (#xCA #b11000000 #b01001001 #b00001111 
> #b11011011)))
>
> This works fine, but sometimes there is a sequence of the same bytes and it 
> would be more readable if I could write something like this:
>
>     ((make-vector 16 0) (#xDC (16 #x00)))
>
> instead of writing out 16 times `#x00`. This would require being able to make 
> a distinction in the pattern whether `byte` is of the pattern
>
>     byte
>
> or
>
>     (count byte)
>
> and if it's the latter construct a list of `count` `byte`s (via `(make-list 
> count byte)` for example) and splice it in. This distinction needs to be made 
> for each byte specification because I want to mix actual bytes and these "RLE-
> encoded" byte specifications.
>
> So I guess what I'm looking for is to have a `syntax-rules` inside a `syntax-
> rules` in a way. Can this be done?

It cannot be done with pure 'syntax-rules' macros, but it can certainly
be done with procedural macros, sometimes called 'syntax-case' macros.
Procedural macros are quite general, allowing you to write arbitrary
Scheme code that runs at compile time to inspect the macro operands and
generate arbitrary code.

I'll describe how to do this with macros further down, but let me begin
with the simple approach.

As rain1 suggested, this can be accomplished most easily by writing a
normal procedure to convert your compact bytevector notation into a
bytevector, and then having your macro expand into code that calls that
procedure.  Here's working code to do that:

--8<---------------cut here---------------start------------->8---
(use-modules (ice-9 match)
             (srfi srfi-1)
             (rnrs bytevectors))

(define (compact-bytevector segments)
  (u8-list->bytevector
   (append-map (match-lambda
                 ((count byte)  (make-list count byte))
                 (byte          (list byte)))
               segments)))

(define-syntax test-cases
  (syntax-rules ()
    ((_ title
        (given (seg ...))
        ...)
     (begin
       (test-begin title)
       (call-with-values (λ () (open-bytevector-output-port))
         (λ (out get-bv)
           (pack given out)
           (let ((received (get-bv))
                 (expected (compact-bytevector '(seg ...))))
             (test-assert (bytevector=? received expected)))))
       ...
       (test-end title)))))


scheme@(guile-user)> (compact-bytevector '(#xDC (16 #x00)))
$2 = #vu8(220 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)
scheme@(guile-user)> ,expand (test-cases "Single precision floating point numbers"
                                         ((make-vector 16 0) (#xDC (16 #x00))))
$3 = (begin
       (test-begin
         "Single precision floating point numbers")
       (call-with-values
         (lambda () (open-bytevector-output-port))
         (lambda (out get-bv)
           (pack (make-vector 16 0) out)
           (let ((received (get-bv))
                 (expected (compact-bytevector '(220 (16 0)))))
             (test-assert (bytevector=? received expected)))))
       (test-end
         "Single precision floating point numbers"))
scheme@(guile-user)> 
--8<---------------cut here---------------end--------------->8---

Now, suppose it was important to do more of this work at macro expansion
time.  For example, if efficiency was a concern, it might not be
acceptable to postpone the conversion of '(#xDC (16 #x00)) into
#vu8(220 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0) until run time.

It turns out that pure 'syntax-rules' macros are turing complete, but
they are limited in the ways that they can inspect the syntax objects
given to them as operands.  In particular, they cannot inspect atomic
expressions, except to compare them with the finite set of literals in
the first operand to 'syntax-rules'.  This is not sufficient to
interpret an arbitrary integer literal.  It could only be done with
'syntax-rules' macros if the 'count' field were represented using a
finite set of literals and/or list structure.  E.g. it could be done if
the count were represented as a list of decimal digits like (1 4 2) for
142.

In cases like this, we would normally turn to procedural macros,
e.g. 'syntax-case' macros.  Here's a straightforward approach, reusing
the 'compact-bytevector' procedure given above, but calling it at
compile time instead of at run time:

--8<---------------cut here---------------start------------->8---
(use-modules (ice-9 match)
             (srfi srfi-1)
             (rnrs bytevectors))

(define (compact-bytevector segments)
  (u8-list->bytevector
   (append-map (match-lambda
                 ((count byte)  (make-list count byte))
                 (byte          (list byte)))
               segments)))

(define-syntax compact-bytevector-literal
  (lambda (stx)
    (syntax-case stx ()
      ((_ (seg ...))
       (compact-bytevector (syntax->datum #'(seg ...)))))))

(define-syntax test-cases
  (syntax-rules ()
    ((_ title
        (given (seg ...))
        ...)
     (begin
       (test-begin title)
       (call-with-values (λ () (open-bytevector-output-port))
         (λ (out get-bv)
           (pack given out)
           (let ((received (get-bv))
                 (expected (compact-bytevector-literal (seg ...))))
             (test-assert (bytevector=? received expected)))))
       ...
       (test-end title)))))
--8<---------------cut here---------------end--------------->8---

Here, instead of having 'test-cases' expand into a procedure call to
'compact-bytevector', it expands into a *macro* call to
'compact-bytevector-literal'.  The latter is a procedural macro, which
calls 'compact-bytevector' at compile time.

This approach is sufficient in this case, but I sense in your question a
desire to be able to perform more general inspection on the macro
operands and generation of the resulting code.  This particular example
is not ideally suited for this task, but the following example code
comes a bit closer:

--8<---------------cut here---------------start------------->8---
(define (segment-syntax->u8-list stx)
  (syntax-case stx ()
    ((count byte)
     (every number? (syntax->datum #'(count byte)))  ;optional guard
     (make-list (syntax->datum #'count)
                (syntax->datum #'byte)))
    (byte
     (number? (syntax->datum #'byte))  ;optional guard
     (list (syntax->datum #'byte)))))

(define (compact-bytevector-syntax->bytevector stx)
  (syntax-case stx ()
    ((seg ...)
     (u8-list->bytevector
      (append-map segment-syntax->u8-list
                  #'(seg ...))))))

(define-syntax compact-bytevector-literal
  (lambda (stx)
    (syntax-case stx ()
      ((_ (seg ...))
       (compact-bytevector-syntax->bytevector #'(seg ...))))))
--8<---------------cut here---------------end--------------->8---

I've omitted the 'test-cases' macro here because it's unchanged from the
previous example.  Here we have two normal procedures that use
'syntax-case', which might be a bit confusing.  These are procedures
that accept syntax objects as arguments, and return normal data
structures.

In contrast to the previous example, which used 'syntax->datum' on the
entire compact-bytevector-literal, in this example we inspect and
destruct the syntax object itself using 'syntax-case'.  This would be
needed in the more general case where identifiers (i.e. variable
references) might occur in the syntax objects.

Hopefully this gives you some idea of what can be done, but I don't
think this is the best example to explore these possibilities, since in
this case the normal procedural approach in my first code excerpt above
is simplest and most likely sufficient.

     Regards,
       Mark

Re: A macro containing a mini-macro?

[Mark H Weaver]

I wrote:
> It turns out that pure 'syntax-rules' macros are turing complete, but
> they are limited in the ways that they can inspect the syntax objects
> given to them as operands.  In particular, they cannot inspect atomic
> expressions, except to compare them with the finite set of literals in
> the first operand to 'syntax-rules'.  This is not sufficient to
> interpret an arbitrary integer literal.  It could only be done with
> 'syntax-rules' macros if the 'count' field were represented using a
> finite set of literals and/or list structure.  E.g. it could be done if
> the count were represented as a list of decimal digits like (1 4 2) for
> 142.

Correction: the finite set of literals that can be recognized by
'syntax-rules' macros must be identifiers, so a representation with bare
decimal digits like (1 4 2) would not work.  Something like (_1 _4 _2)
would be possible, though.

       Mark

Re: A macro containing a mini-macro?

[Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 515 bytes --]

Hi Mark,

> In contrast to the previous example, which used 'syntax->datum' on the
> entire compact-bytevector-literal, in this example we inspect and
> destruct the syntax object itself using 'syntax-case'.  This would be
> needed in the more general case where identifiers (i.e. variable
> references) might occur in the syntax objects.

Practical examples like these are direly missing. Could you blog your
text?

Best wishes,
Arne

-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1076 bytes --]

Re: A macro containing a mini-macro?

[HiPhish]

Thank you very much for the in-depth explanation, and sorry for answering only 
now. I haven't been able to get around to it until now, and this is exactly 
what I was looking for. One more question if you feel like answering: where 
can I learn properly about Scheme macros? Metaprogramming is one of the most 
powerful features in Lisp, but I haven't come across a really good explanation 
of all its facets.

On Saturday, 29 September 2018 02:28:37 CET Mark H Weaver wrote:
> Hi,
> 
> HiPhish <hiphish@posteo.de> writes:
> > Hello Schemers,
> > 
> > I have written a small macro for writing test specifications:
> >     (define-syntax test-cases
> >     
> >       (syntax-rules ()
> >       
> >         ((_ title
> >         
> >            (given (byte byte* ...))
> >            ...)
> >          
> >          (begin
> >          
> >            (test-begin title)
> >            (call-with-values (λ () (open-bytevector-output-port))
> >            
> >              (λ (out get-bv)
> >              
> >                (pack given out)
> >                (let ((received (get-bv))
> >                
> >                      (expected (u8-list->bytevector '(byte byte* ...))))
> >                  
> >                  (test-assert (bytevector=? received expected)))))
> >            
> >            ...
> >            (test-end title)))))
> > 
> > The idea is that I can specify a series of test cases where each case
> > consists> 
> > of an object and a sequence of bytes which this object is to be serialized 
to:
> >     (test-cases "Single precision floating point numbers"
> >     
> >       (+3.1415927410125732 (#xCA #b01000000 #b01001001 #b00001111
> >       #b11011011))
> >       (-3.1415927410125732 (#xCA #b11000000 #b01001001 #b00001111
> > 
> > #b11011011)))
> > 
> > This works fine, but sometimes there is a sequence of the same bytes and
> > it
> > 
> > would be more readable if I could write something like this:
> >     ((make-vector 16 0) (#xDC (16 #x00)))
> > 
> > instead of writing out 16 times `#x00`. This would require being able to
> > make a distinction in the pattern whether `byte` is of the pattern
> > 
> >     byte
> > 
> > or
> > 
> >     (count byte)
> > 
> > and if it's the latter construct a list of `count` `byte`s (via
> > `(make-list
> > count byte)` for example) and splice it in. This distinction needs to be
> > made for each byte specification because I want to mix actual bytes and
> > these "RLE- encoded" byte specifications.
> > 
> > So I guess what I'm looking for is to have a `syntax-rules` inside a
> > `syntax- rules` in a way. Can this be done?
> 
> It cannot be done with pure 'syntax-rules' macros, but it can certainly
> be done with procedural macros, sometimes called 'syntax-case' macros.
> Procedural macros are quite general, allowing you to write arbitrary
> Scheme code that runs at compile time to inspect the macro operands and
> generate arbitrary code.
> 
> I'll describe how to do this with macros further down, but let me begin
> with the simple approach.
> 
> As rain1 suggested, this can be accomplished most easily by writing a
> normal procedure to convert your compact bytevector notation into a
> bytevector, and then having your macro expand into code that calls that
> procedure.  Here's working code to do that:
> 
> --8<---------------cut here---------------start------------->8---
> (use-modules (ice-9 match)
>              (srfi srfi-1)
>              (rnrs bytevectors))
> 
> (define (compact-bytevector segments)
>   (u8-list->bytevector
>    (append-map (match-lambda
>                  ((count byte)  (make-list count byte))
>                  (byte          (list byte)))
>                segments)))
> 
> (define-syntax test-cases
>   (syntax-rules ()
>     ((_ title
>         (given (seg ...))
>         ...)
>      (begin
>        (test-begin title)
>        (call-with-values (λ () (open-bytevector-output-port))
>          (λ (out get-bv)
>            (pack given out)
>            (let ((received (get-bv))
>                  (expected (compact-bytevector '(seg ...))))
>              (test-assert (bytevector=? received expected)))))
>        ...
>        (test-end title)))))
> 
> 
> scheme@(guile-user)> (compact-bytevector '(#xDC (16 #x00)))
> $2 = #vu8(220 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)
> scheme@(guile-user)> ,expand (test-cases "Single precision floating point
> numbers" ((make-vector 16 0) (#xDC (16 #x00)))) $3 = (begin
>        (test-begin
>          "Single precision floating point numbers")
>        (call-with-values
>          (lambda () (open-bytevector-output-port))
>          (lambda (out get-bv)
>            (pack (make-vector 16 0) out)
>            (let ((received (get-bv))
>                  (expected (compact-bytevector '(220 (16 0)))))
>              (test-assert (bytevector=? received expected)))))
>        (test-end
>          "Single precision floating point numbers"))
> scheme@(guile-user)>
> --8<---------------cut here---------------end--------------->8---
> 
> Now, suppose it was important to do more of this work at macro expansion
> time.  For example, if efficiency was a concern, it might not be
> acceptable to postpone the conversion of '(#xDC (16 #x00)) into
> #vu8(220 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0) until run time.
> 
> It turns out that pure 'syntax-rules' macros are turing complete, but
> they are limited in the ways that they can inspect the syntax objects
> given to them as operands.  In particular, they cannot inspect atomic
> expressions, except to compare them with the finite set of literals in
> the first operand to 'syntax-rules'.  This is not sufficient to
> interpret an arbitrary integer literal.  It could only be done with
> 'syntax-rules' macros if the 'count' field were represented using a
> finite set of literals and/or list structure.  E.g. it could be done if
> the count were represented as a list of decimal digits like (1 4 2) for
> 142.
> 
> In cases like this, we would normally turn to procedural macros,
> e.g. 'syntax-case' macros.  Here's a straightforward approach, reusing
> the 'compact-bytevector' procedure given above, but calling it at
> compile time instead of at run time:
> 
> --8<---------------cut here---------------start------------->8---
> (use-modules (ice-9 match)
>              (srfi srfi-1)
>              (rnrs bytevectors))
> 
> (define (compact-bytevector segments)
>   (u8-list->bytevector
>    (append-map (match-lambda
>                  ((count byte)  (make-list count byte))
>                  (byte          (list byte)))
>                segments)))
> 
> (define-syntax compact-bytevector-literal
>   (lambda (stx)
>     (syntax-case stx ()
>       ((_ (seg ...))
>        (compact-bytevector (syntax->datum #'(seg ...)))))))
> 
> (define-syntax test-cases
>   (syntax-rules ()
>     ((_ title
>         (given (seg ...))
>         ...)
>      (begin
>        (test-begin title)
>        (call-with-values (λ () (open-bytevector-output-port))
>          (λ (out get-bv)
>            (pack given out)
>            (let ((received (get-bv))
>                  (expected (compact-bytevector-literal (seg ...))))
>              (test-assert (bytevector=? received expected)))))
>        ...
>        (test-end title)))))
> --8<---------------cut here---------------end--------------->8---
> 
> Here, instead of having 'test-cases' expand into a procedure call to
> 'compact-bytevector', it expands into a *macro* call to
> 'compact-bytevector-literal'.  The latter is a procedural macro, which
> calls 'compact-bytevector' at compile time.
> 
> This approach is sufficient in this case, but I sense in your question a
> desire to be able to perform more general inspection on the macro
> operands and generation of the resulting code.  This particular example
> is not ideally suited for this task, but the following example code
> comes a bit closer:
> 
> --8<---------------cut here---------------start------------->8---
> (define (segment-syntax->u8-list stx)
>   (syntax-case stx ()
>     ((count byte)
>      (every number? (syntax->datum #'(count byte)))  ;optional guard
>      (make-list (syntax->datum #'count)
>                 (syntax->datum #'byte)))
>     (byte
>      (number? (syntax->datum #'byte))  ;optional guard
>      (list (syntax->datum #'byte)))))
> 
> (define (compact-bytevector-syntax->bytevector stx)
>   (syntax-case stx ()
>     ((seg ...)
>      (u8-list->bytevector
>       (append-map segment-syntax->u8-list
>                   #'(seg ...))))))
> 
> (define-syntax compact-bytevector-literal
>   (lambda (stx)
>     (syntax-case stx ()
>       ((_ (seg ...))
>        (compact-bytevector-syntax->bytevector #'(seg ...))))))
> --8<---------------cut here---------------end--------------->8---
> 
> I've omitted the 'test-cases' macro here because it's unchanged from the
> previous example.  Here we have two normal procedures that use
> 'syntax-case', which might be a bit confusing.  These are procedures
> that accept syntax objects as arguments, and return normal data
> structures.
> 
> In contrast to the previous example, which used 'syntax->datum' on the
> entire compact-bytevector-literal, in this example we inspect and
> destruct the syntax object itself using 'syntax-case'.  This would be
> needed in the more general case where identifiers (i.e. variable
> references) might occur in the syntax objects.
> 
> Hopefully this gives you some idea of what can be done, but I don't
> think this is the best example to explore these possibilities, since in
> this case the normal procedural approach in my first code excerpt above
> is simplest and most likely sufficient.
> 
>      Regards,
>        Mark