RE: Control the order of expansion of syntax-case macros

[Maxime Devos via General Guile related discussions <guile-user@gnu.org>]

>syntax-case macros can have cool side effects at expansion time.
>However, they are still draped in a veil of mystery to me.

> Consider this code:

(define-syntax hello
  (lambda (stx)
    (syntax-case stx ()
      (_
       (begin
         (format (current-error-port) "Hello!\n")
         (datum->syntax #f "hello"))))))

You can simplify this to:

(define-syntax hello
  (lambda (stx)
     (format (current-error-port) "Hello!\n")
     (datum->syntax #f "hello")))

I think this simplified version makes it a little easier to think about how Scheme macros work.

(define-syntax world
  (lambda (stx)
    (syntax-case stx ()
      (_
       (begin
         (format (current-error-port) "World!\n")
         (datum->syntax #f "world"))))))

(define-syntax hello-world
  (lambda (stx)
    (syntax-case stx ()
      (_
       #`(string-append #,hello " " #,world)))))

(Likewise here)

,expand hello-world


>Running it gives me:
>
>Hello!
>World!
>$1 = (string-append "hello" " " "world")

I wonder, did you copy-paste all four blocks (both the define-syntax forms and the ,expand) at the same time, or one-by-one? I don’t have a Guile at hand to test, but IIUC if you paste the following in the REPL (without any ,expand!)

(define-syntax hello-world
  (lambda (stx)
    (syntax-case stx ()
      (_
       #`(string-append #,hello " " #,world)))))

you’ll get “Hello!<newline>World!” in the REPL, and  ‘,expand hello-world’ will only give you the ‘$1 = […]’ (_without_ the current-error-port output).

What happens here, is that Guile sees your ‘hello-world’ definition and expands it (not the macro that it defines, but rather the ‘lambda’ and its interior) (macros are usually defined with other macros – syntax-case is a macro!). But, it’s not just ‘hello-world’ that’s a macro, hello and world are macros too. Since they aren’t quoted, they are expanded.  So, during the expansion of the ‘define-syntax’ form above, you will see “Hello!” and “World!”. Can we choose which order ‘hello’ and ‘world’ are expanded, while keeping everything else the same? I don’t know.

(‘hello’ and ‘world’ expand to strings, which can be spliced into syntax. If they expanded to, say, a procedure (not just a lambda form, but rather the evaluated lambda), it is invalid (not just a type error for string-append, but invalid).)

>Cool! Now, I suspect there are no clear rules to guess the order of
expansion (whether Hello! is printed before World! or after). I would
very much like to twist that order and reliably get World! and then
Hello!. How can I achieve that?

(define-syntax hello-world
  (lambda (stx)
    (syntax-case stx ()
      (_
       (let* ((w #'world)
              (h #'hello))
         #`(string-append #,h " " #,w))))))

Wait, you are modifying more here!!! In this particular (artificial) example, it probably doesn’t matter, but in whatever the ‘real’ code will eventually be, it might. When expanding the above ‘define-syntax’ form, Guile doesn’t expand ‘world’ and ‘hello’, since they are quoted. If you paste the above definition in the REPL, you won’t get any “Hello! World!” or “World! Hello” messages. No side-effects happen from the expansion of the above define-syntax form, only once a hello-world form is expanded you’ll get the side-effects.

(Depending on what you’re doing, this change might be better or worse.)

>With this modification, I get the same order:

>Hello!
>World!
>$1 = (string-append "hello" " " "world")

>So my guess is that syntax objects are expanded lazily.

Not lazy. If you splice the same syntax into multiple locations, it will be expanded per-location. (I would call it ‘delayed’ if it weren’t for the ‘delay’ macro and ’force’ procedure that implement laziness.)

;; This one is a workaholic! Lots of World! World! World! …
(define-syntax a
  (lambda (stx)
    (let ((a #'world)) #`(list #,a #,a #,a #,a))))

For testing, consider replacing it by

(define-syntax hello-world
  (lambda (stx)
    (syntax-case stx ()
      (_
       (let* ((w #'world)
              (h #'hello))
         (pk 'Post-Quote w h)
         #`(string-append #,h " " #,w))))))
;; no information on stderr yet on stderr
,expand hello-world
;; now we see (post-quote #<syntax world …> #<syntax hello …>) Hello! World!

If you simply write #'world, then no expansion happens yet (except maybe in the sense that `(proc) ‘expands too’ (list 'proc) instead of running the procedure ‘proc’).

During #`(string-append #,h “ “ #,w), no expansion happens yet either. You are simply making a larger piece of code from some smaller pieces. Rather, it’s when you write ‘hello-world’ in some piece of code that Guile applies the lambda (lambda (stx) stuff) to #’hello-world (in realistic settings it would be #'(hello-world arg arg2)), and replaces the code fragment ‘hello-world’ by the output of the lambda. E.g., if the lambda returns #'(not-implemented), then ‘(stuff hello-world more-stuff)’ becomes ‘(stuff (not-implemented) more-stuff).

I think the best way to think of syntax, is as code with some lexical context and line numbers, and ‘quote’ (') is the equivalent of ‘syntax-quote’ (#') (or was it quote-syntax?).

>Is there something I can do to get #'world expanded before #'hello?

Rewritten question:  “Is there something I can do to get ‘world’ expanded before ‘hello’?”

I imagine the following would do it:

(define-syntax hello-world
  (lambda (stx)
    ;; let / let* wouldn’t make a difference, since it’s expansion order we’re concerned
    ;; with, not evaluation order, and the former is equally as unspecified(?) with ‘let’ as
    ;; with ‘let*’
    (let ((w world) (h hello))
      #`(string-append #,hello " " #,world)))))

(but keep in mind the possibly unintended time of expansion).

Perhaps another option would be to use the module reflection API and macro-binding or macro-transformer to extract the (lambda (stx) …) and now use that lambda, or syntax-local-bindings and do the same.

For (well-defined) methods of controlling the evaluation order,  I imagine let-syntax might be able to accomplish something, but I’m not sure.

Another option is to just let ‘hello’ and ‘world’ be procedure (that happen to accept syntax and produce syntax), and simply invoke them in hello-world (and use #, to insert them in the produced code). In this particular case, they could just be thunks. (If you do this, and also use the hello-world form in the same module as where it is defined (not just mentioned somewhere in another macro, but rather when the non-define-syntax forms are expanded, then need the transformer of the hello-world form to run), then make sure to add some eval-when where appropriate.)

Best regards,
Maxime Devos