Re: A macro containing a mini-macro?

[HiPhish <hiphish@posteo.de>]

Thank you very much for the in-depth explanation, and sorry for answering only 
now. I haven't been able to get around to it until now, and this is exactly 
what I was looking for. One more question if you feel like answering: where 
can I learn properly about Scheme macros? Metaprogramming is one of the most 
powerful features in Lisp, but I haven't come across a really good explanation 
of all its facets.

On Saturday, 29 September 2018 02:28:37 CET Mark H Weaver wrote:
> Hi,
> 
> HiPhish <hiphish@posteo.de> writes:
> > Hello Schemers,
> > 
> > I have written a small macro for writing test specifications:
> >     (define-syntax test-cases
> >     
> >       (syntax-rules ()
> >       
> >         ((_ title
> >         
> >            (given (byte byte* ...))
> >            ...)
> >          
> >          (begin
> >          
> >            (test-begin title)
> >            (call-with-values (λ () (open-bytevector-output-port))
> >            
> >              (λ (out get-bv)
> >              
> >                (pack given out)
> >                (let ((received (get-bv))
> >                
> >                      (expected (u8-list->bytevector '(byte byte* ...))))
> >                  
> >                  (test-assert (bytevector=? received expected)))))
> >            
> >            ...
> >            (test-end title)))))
> > 
> > The idea is that I can specify a series of test cases where each case
> > consists> 
> > of an object and a sequence of bytes which this object is to be serialized 
to:
> >     (test-cases "Single precision floating point numbers"
> >     
> >       (+3.1415927410125732 (#xCA #b01000000 #b01001001 #b00001111
> >       #b11011011))
> >       (-3.1415927410125732 (#xCA #b11000000 #b01001001 #b00001111
> > 
> > #b11011011)))
> > 
> > This works fine, but sometimes there is a sequence of the same bytes and
> > it
> > 
> > would be more readable if I could write something like this:
> >     ((make-vector 16 0) (#xDC (16 #x00)))
> > 
> > instead of writing out 16 times `#x00`. This would require being able to
> > make a distinction in the pattern whether `byte` is of the pattern
> > 
> >     byte
> > 
> > or
> > 
> >     (count byte)
> > 
> > and if it's the latter construct a list of `count` `byte`s (via
> > `(make-list
> > count byte)` for example) and splice it in. This distinction needs to be
> > made for each byte specification because I want to mix actual bytes and
> > these "RLE- encoded" byte specifications.
> > 
> > So I guess what I'm looking for is to have a `syntax-rules` inside a
> > `syntax- rules` in a way. Can this be done?
> 
> It cannot be done with pure 'syntax-rules' macros, but it can certainly
> be done with procedural macros, sometimes called 'syntax-case' macros.
> Procedural macros are quite general, allowing you to write arbitrary
> Scheme code that runs at compile time to inspect the macro operands and
> generate arbitrary code.
> 
> I'll describe how to do this with macros further down, but let me begin
> with the simple approach.
> 
> As rain1 suggested, this can be accomplished most easily by writing a
> normal procedure to convert your compact bytevector notation into a
> bytevector, and then having your macro expand into code that calls that
> procedure.  Here's working code to do that:
> 
> --8<---------------cut here---------------start------------->8---
> (use-modules (ice-9 match)
>              (srfi srfi-1)
>              (rnrs bytevectors))
> 
> (define (compact-bytevector segments)
>   (u8-list->bytevector
>    (append-map (match-lambda
>                  ((count byte)  (make-list count byte))
>                  (byte          (list byte)))
>                segments)))
> 
> (define-syntax test-cases
>   (syntax-rules ()
>     ((_ title
>         (given (seg ...))
>         ...)
>      (begin
>        (test-begin title)
>        (call-with-values (λ () (open-bytevector-output-port))
>          (λ (out get-bv)
>            (pack given out)
>            (let ((received (get-bv))
>                  (expected (compact-bytevector '(seg ...))))
>              (test-assert (bytevector=? received expected)))))
>        ...
>        (test-end title)))))
> 
> 
> scheme@(guile-user)> (compact-bytevector '(#xDC (16 #x00)))
> $2 = #vu8(220 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0)
> scheme@(guile-user)> ,expand (test-cases "Single precision floating point
> numbers" ((make-vector 16 0) (#xDC (16 #x00)))) $3 = (begin
>        (test-begin
>          "Single precision floating point numbers")
>        (call-with-values
>          (lambda () (open-bytevector-output-port))
>          (lambda (out get-bv)
>            (pack (make-vector 16 0) out)
>            (let ((received (get-bv))
>                  (expected (compact-bytevector '(220 (16 0)))))
>              (test-assert (bytevector=? received expected)))))
>        (test-end
>          "Single precision floating point numbers"))
> scheme@(guile-user)>
> --8<---------------cut here---------------end--------------->8---
> 
> Now, suppose it was important to do more of this work at macro expansion
> time.  For example, if efficiency was a concern, it might not be
> acceptable to postpone the conversion of '(#xDC (16 #x00)) into
> #vu8(220 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0) until run time.
> 
> It turns out that pure 'syntax-rules' macros are turing complete, but
> they are limited in the ways that they can inspect the syntax objects
> given to them as operands.  In particular, they cannot inspect atomic
> expressions, except to compare them with the finite set of literals in
> the first operand to 'syntax-rules'.  This is not sufficient to
> interpret an arbitrary integer literal.  It could only be done with
> 'syntax-rules' macros if the 'count' field were represented using a
> finite set of literals and/or list structure.  E.g. it could be done if
> the count were represented as a list of decimal digits like (1 4 2) for
> 142.
> 
> In cases like this, we would normally turn to procedural macros,
> e.g. 'syntax-case' macros.  Here's a straightforward approach, reusing
> the 'compact-bytevector' procedure given above, but calling it at
> compile time instead of at run time:
> 
> --8<---------------cut here---------------start------------->8---
> (use-modules (ice-9 match)
>              (srfi srfi-1)
>              (rnrs bytevectors))
> 
> (define (compact-bytevector segments)
>   (u8-list->bytevector
>    (append-map (match-lambda
>                  ((count byte)  (make-list count byte))
>                  (byte          (list byte)))
>                segments)))
> 
> (define-syntax compact-bytevector-literal
>   (lambda (stx)
>     (syntax-case stx ()
>       ((_ (seg ...))
>        (compact-bytevector (syntax->datum #'(seg ...)))))))
> 
> (define-syntax test-cases
>   (syntax-rules ()
>     ((_ title
>         (given (seg ...))
>         ...)
>      (begin
>        (test-begin title)
>        (call-with-values (λ () (open-bytevector-output-port))
>          (λ (out get-bv)
>            (pack given out)
>            (let ((received (get-bv))
>                  (expected (compact-bytevector-literal (seg ...))))
>              (test-assert (bytevector=? received expected)))))
>        ...
>        (test-end title)))))
> --8<---------------cut here---------------end--------------->8---
> 
> Here, instead of having 'test-cases' expand into a procedure call to
> 'compact-bytevector', it expands into a *macro* call to
> 'compact-bytevector-literal'.  The latter is a procedural macro, which
> calls 'compact-bytevector' at compile time.
> 
> This approach is sufficient in this case, but I sense in your question a
> desire to be able to perform more general inspection on the macro
> operands and generation of the resulting code.  This particular example
> is not ideally suited for this task, but the following example code
> comes a bit closer:
> 
> --8<---------------cut here---------------start------------->8---
> (define (segment-syntax->u8-list stx)
>   (syntax-case stx ()
>     ((count byte)
>      (every number? (syntax->datum #'(count byte)))  ;optional guard
>      (make-list (syntax->datum #'count)
>                 (syntax->datum #'byte)))
>     (byte
>      (number? (syntax->datum #'byte))  ;optional guard
>      (list (syntax->datum #'byte)))))
> 
> (define (compact-bytevector-syntax->bytevector stx)
>   (syntax-case stx ()
>     ((seg ...)
>      (u8-list->bytevector
>       (append-map segment-syntax->u8-list
>                   #'(seg ...))))))
> 
> (define-syntax compact-bytevector-literal
>   (lambda (stx)
>     (syntax-case stx ()
>       ((_ (seg ...))
>        (compact-bytevector-syntax->bytevector #'(seg ...))))))
> --8<---------------cut here---------------end--------------->8---
> 
> I've omitted the 'test-cases' macro here because it's unchanged from the
> previous example.  Here we have two normal procedures that use
> 'syntax-case', which might be a bit confusing.  These are procedures
> that accept syntax objects as arguments, and return normal data
> structures.
> 
> In contrast to the previous example, which used 'syntax->datum' on the
> entire compact-bytevector-literal, in this example we inspect and
> destruct the syntax object itself using 'syntax-case'.  This would be
> needed in the more general case where identifiers (i.e. variable
> references) might occur in the syntax objects.
> 
> Hopefully this gives you some idea of what can be done, but I don't
> think this is the best example to explore these possibilities, since in
> this case the normal procedural approach in my first code excerpt above
> is simplest and most likely sufficient.
> 
>      Regards,
>        Mark