How to use-modules within macro?

How to use-modules within macro?

[pelzflorian (Florian Pelz)]

Hello,

I am writing a Guile macro to manipulate Scheme code and am stuck on
what I hope is a simple problem and it would be nice if you could
explain.  I try:

(define-syntax O
  (lambda (x)
    (syntax-case x ()
      ((_)
       #`(begin (use-modules (ice-9 local-eval))
                (local-eval 42 (the-environment)))))))
(pk (O))

But it does not work; it cannot resolve local-eval and
the-environment.  Why?  When I write it in the interpreter, a second
call to (pk (O)) works and prints 42.

Using (@ (ice-9 local-eval) …) does not work for looking up
the-environment.

Regards,
Florian

Re: How to use-modules within macro?

[Mark H Weaver]

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> writes:

> I am writing a Guile macro to manipulate Scheme code and am stuck on
> what I hope is a simple problem and it would be nice if you could
> explain.  I try:
>
> (define-syntax O
>   (lambda (x)
>     (syntax-case x ()
>       ((_)
>        #`(begin (use-modules (ice-9 local-eval))
>                 (local-eval 42 (the-environment)))))))
> (pk (O))

This approach is misguided and unnecessary.  You don't need to include
'use-modules' in your macro expansion, and it's best avoided.

The references to 'local-eval' and 'the-environment' in the macro
template above will refer to bindings present in the module where 'O' is
defined, *not* the module where 'O' is used.  This is part of what it
means to say that 'O' is a "hygienic" macro.

Therefore, all you need to do is make sure (ice-9 local-eval) is
imported in the module where 'O' is defined, like this:

--8<---------------cut here---------------start------------->8---
(define-module (my-module-that-exports-O)
  #:use-module (ice-9 local-eval)
  #:export (O))

(define-syntax O
  (lambda (x)
    (syntax-case x ()
      ((_)
       #`(local-eval 42 (the-environment))))))
--8<---------------cut here---------------end--------------->8---

Does that work for you?

FYI, the way this works internally is that the macro expander operates
on "syntax objects" instead of plain S-expressions.  The main difference
is that "syntax objects" keep additional information about the lexical
environments where the embedded identifiers were originally found.  So
when a use of (O) expands into (local-eval 42 (the-environment)), the
identifiers 'local-eval' and 'the-environment' are looked up in the
proper environment.

By the way, another consequence of hygiene, which you probably don't
want here, is that the (the-environment) above will capture the lexical
environment where 'O' was *defined*, instead of the environment where
(O) is used.  In other words, in (let ((x 5)) (O)), the captured lexical
environment will not include 'x'.

I should also mention that using (the-environment) will pretty much
disable most compiler optimizations that would otherwise occur with that
top-level form.  That entire mechanism is best avoided if at all
possible.

Can you tell me more broadly what you are trying to accomplish here?
I may be able to suggest an alternate approach.

     Best,
      Mark

Re: How to use-modules within macro?

[pelzflorian (Florian Pelz)]

On Thu, Aug 29, 2019 at 07:04:07PM -0400, Mark H Weaver wrote:
> This approach is misguided and unnecessary.  You don't need to include
> 'use-modules' in your macro expansion, and it's best avoided.
> 

:) I suspected I was doing something very wrong.



> FYI, the way this works internally is that the macro expander operates
> on "syntax objects" instead of plain S-expressions.  The main difference
> is that "syntax objects" keep additional information about the lexical
> environments where the embedded identifiers were originally found.  So
> when a use of (O) expands into (local-eval 42 (the-environment)), the
> identifiers 'local-eval' and 'the-environment' are looked up in the
> proper environment.
>

I had read but not understood that syntax carries references, but of
course it does because of hygiene.  Thank you!



> By the way, another consequence of hygiene, which you probably don't
> want here, is that the (the-environment) above will capture the lexical
> environment where 'O' was *defined*, instead of the environment where
> (O) is used.  In other words, in (let ((x 5)) (O)), the captured lexical
> environment will not include 'x'.
> 
> I should also mention that using (the-environment) will pretty much
> disable most compiler optimizations that would otherwise occur with that
> top-level form.  That entire mechanism is best avoided if at all
> possible.
> 
> Can you tell me more broadly what you are trying to accomplish here?
> I may be able to suggest an alternate approach.
> 
>      Best,
>       Mark


What I am actually trying to do is rewriting S-expressions based on a
translated gettext MO file for localization.  I have a gettext POT
file which contains:

#: apps/packages/templates/components.scm:104$
msgid " issue"
msgid_plural " issues"
msgstr[0] ""
msgstr[1] ""



(define (sngettext x)
  "After choosing an identifier for behavior similar to ngettext:1,2,
make it usable like (define-syntax N_ sngettext).  sngettext takes
into account that not all languages have only singular and plural
forms."
  (syntax-case x ()
    ((_ msgid1 msgid2 n) ;three sexps
     ;; sexp->msgid computes a gettext msgid for lookup with gettext
     (let* ((msgstr1 (sexp->msgid (syntax->datum #'msgid1)))
            (msgstr2 (sexp->msgid (syntax->datum #'msgid2))))
       #`(begin
           (use-modules (ice-9 local-eval))
           (local-eval (let ((applicable (if (= #'n 1) #'msgid1 #'msgid2)))
                         ;; deconstruct builds a new sexp from either
                         ;; msgid1 or msgid2 and the translation from
                         ;; the gettext MO file, i.e. it deconstructs
                         ;; the translation to an sexp.
                         (deconstruct (syntax->datum applicable)
                                      (ngettext #,msgstr1 #,msgstr2 n)))
                       (the-environment)))))))


The first argument of sexp->msgid and deconstruct must be the msgid,
which can be a complicated nested sexp.  Macros must not have been
expanded there.  For ngettext, it is important that the n s-expression
be evaluated by ngettext in the evaluation phase and not at expansion
time.

The corresponding sgettext works fine with deconstruct moved to macro
expansion time, therefore there is no need for local-eval.  Moving
deconstruct to macro expansion time would mean I would need to
deconstruct all singular, plural etc. forms, because I do not yet know
the value of n.  I could do this by evaluating the Plural-Forms line
from the MO file, I just wanted to avoid this and let ngettext do the
work.


Thank you!!  I now understand macros better.

Regards,
Florian

Re: How to use-modules within macro?

[pelzflorian (Florian Pelz)]

On Thu, Aug 29, 2019 at 07:04:07PM -0400, Mark H Weaver wrote:
> Hi Florian,
> 
> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> writes:
> 
> > I am writing a Guile macro to manipulate Scheme code and am stuck on
> > what I hope is a simple problem and it would be nice if you could
> > explain.  I try:
> >
> > (define-syntax O
> >   (lambda (x)
> >     (syntax-case x ()
> >       ((_)
> >        #`(begin (use-modules (ice-9 local-eval))
> >                 (local-eval 42 (the-environment)))))))
> > (pk (O))
> 
> This approach is misguided and unnecessary.  You don't need to include
> 'use-modules' in your macro expansion, and it's best avoided.
> 
> […]

Your explanation helped a lot.

To retain unhygienic references, I am now using datum->syntax instead
of local-eval.  It is much better.  For example, to make a macro that
increments all numbers in a given program by one:

(use-modules (ice-9 match))
(define-syntax one-more
  (lambda (x)
    (syntax-case x ()
      ((_ exp)
       (datum->syntax
        #'exp
        (let loop ((y (syntax->datum #'exp)))
          (match y
            ((? number?) (1+ y))
            ((? list?) (map loop y))
            (else y))))))))

(let ((four 4))
  (one-more (* 2 3 four)))

Yields 48.  I hope this is the right approach for rewriting programs.

Regards,
Florian

Re: How to use-modules within macro?

[Mark H Weaver]

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> writes:

> To retain unhygienic references, I am now using datum->syntax instead
> of local-eval.  It is much better.  For example, to make a macro that
> increments all numbers in a given program by one:
>
> (use-modules (ice-9 match))
> (define-syntax one-more
>   (lambda (x)
>     (syntax-case x ()
>       ((_ exp)
>        (datum->syntax
>         #'exp
>         (let loop ((y (syntax->datum #'exp)))
>           (match y
>             ((? number?) (1+ y))
>             ((? list?) (map loop y))
>             (else y))))))))
>
> (let ((four 4))
>   (one-more (* 2 3 four)))
>
> Yields 48.  I hope this is the right approach for rewriting programs.

There are some problems above:

(1) The first argument to 'datum->syntax' must be an identifier, which
    is the syntax object corresponding to a symbol.  Here, you are
    passing an entire expression, and in the example usage above, #'exp
    will be the syntax object corresponding to (* 2 3 hour).  Guile
    should ideally raise an error in this case.

(2) The way you are doing things here destroys hygiene within the
    expression that you are rewriting.  You convert the entire
    expression with 'syntax->datum', process the datum, and then convert
    the rewritten expression using 'datum->syntax'.  The problem here is
    that 'syntax->datum' discards all of the extra information about
    lexical environments of identifiers that were kept in the syntax
    object.  This will cause severe problems when 'one-more' is used in
    combination with other macros, including unintended variable
    capture.

To do this properly, you must do the rewriting on the syntax objects
themselves.  It's okay to convert a syntax object to a datum to test
whether it's a literal number, but the important thing is that all
*identifiers* in the rewritten code should be preserved.

So, instead of using 'match' on the result of 'syntax->datum', you
should instead use 'syntax-case' on the syntax object itself, like this
(untested):

  (let loop ((e #'exp))
    (syntax-case e ()
      (num
       (number? (syntax->datum #'num))
       #'(1+ num))
      ((x ...)
       (map loop #'(x ...)))
      (y
       #'y)))

Finally, I should mention that macro expansion is always done from the
outside in, meaning that when 'one-more' is expanded, its operand will
not yet have been expanded.  In general, this means that it's impossible
to comprehend the code within a macro's operands unless the parsing code
knows about every macro that might be used within the operands.  It's
not even possible to know which subparts are expressions and which are
other things like variable binding lists.

For this reason, I think it's generally a mistake to try to parse code
within a macro's operands.  It normally only makes sense for macros to
inspect the parts of operands that are considered part of the macro's
syntax.  For example, it makes sense for a 'let' macro to parse its
binding list, or for a 'match' macro to parse its patterns and
templates, but it does *not* make sense for a macro to try to parse
general subexpressions passed to the macro.

If you could give me a birds-eye view of what you're trying to do here,
I might be able to suggest other approaches.

       Best,
        Mark

Re: How to use-modules within macro?

[Mark H Weaver]

Hello again,

I wrote earlier:
> So, instead of using 'match' on the result of 'syntax->datum', you
> should instead use 'syntax-case' on the syntax object itself, like this
> (untested):
>
>   (let loop ((e #'exp))
>     (syntax-case e ()
>       (num
>        (number? (syntax->datum #'num))
>        #'(1+ num))
>       ((x ...)
>        (map loop #'(x ...)))
>       (y
>        #'y)))

I should mention that the use of 'map' directly on a syntax object is
only allowable in certain cases.  Here, we assume that the syntax object
produced by #'(x ...) is a normal Scheme list.  That is _not_ the case
for arbitrary syntax objects that correspond to a list.  For example, it
would _not_ be safe to pass 'e' as the list argument to 'map', although
'e' is in some sense equivalent to #(x ...) in the second clause above.

The reason is that syntax objects contain information about the
associated lexical environment which starts at the top of the expression
tree, and is *lazily* pushed down into the subexpressions as the tree is
taken apart using 'syntax-case' and put back together using 'syntax'
(a.k.a. "#'").

As a result, there are only a few cases when you can safely assume that
the top structure of a syntax object is a normal list or pair, and they
are spelled out in the documentation for 'syntax' in the R6RS Standard
Libraries specification:

  http://www.r6rs.org/final/html/r6rs-lib/r6rs-lib-Z-H-13.html#node_sec_12.4

Here's the relevant excerpt:

  The output produced by syntax is wrapped or unwrapped according to the
  following rules.

  * the copy of (<t1> . <t2>) is a pair if <t1> or <t2> contain any
    pattern variables,

  * the copy of (<t> <ellipsis>) is a list if <t> contains any pattern
    variables,

  * the copy of #(<t1> ... <tn>) is a vector if any of <t1>, ..., <tn>
    contain any pattern variables, and

  * the copy of any portion of <t> not containing any pattern variables
    is a wrapped syntax object.

A "wrapped syntax object" is one where the lexical environment
information has not yet been pushed down into the subexpressions.  It is
a special kind of object that you can only take apart using
'syntax-case'.

So, in the clause above where 'map' is used, 'e' might be a "wrapped
syntax object", but when the elements are extracted from it using the
'syntax-case' pattern (x ...) and then put back together using
#'(x ...), you can then assume that the resulting syntax object is
a normal Scheme list of syntax objects, and therefore it is safe to
use 'map' on it.

       Mark

Re: How to use-modules within macro?

[pelzflorian (Florian Pelz)]

Thank you for giving my approach a kind of “trial by fire”.
Considering what you wrote, I still believe in my datum->syntax
approach in my special case.  After all, datum->syntax exists for a
reason, I suppose.  I am interested in your further comments.  They
have been very helpful.

On Wed, Sep 04, 2019 at 03:54:13PM -0400, Mark H Weaver wrote:
> If you could give me a birds-eye view of what you're trying to do here,
> I might be able to suggest other approaches.
> 

I am trying to i18n the Guix website by writing a gettext that accepts
s-expresions.

For example, the about page of Guix <https://guix.gnu.org/about/>
would use this SHTML:

      ;; TRANSLATORS: Features and Defining Packages are section names
      ;; in the English (en) manual.
      ,(G_
        `(p
          "GNU Guix provides "
          ,(G_ (manual-href "state-of-the-art package management features"
                            (G_ "en")
                            (G_ "Features.html")))
          " such as transactional upgrades and roll-backs, reproducible
            build environments, unprivileged package management, and
            per-user profiles.  It uses low-level mechanisms from the "
          ,(G_ `(a (@ (href "https://nixos.org/nix/")) "Nix"))
          " package manager, but packages are "
          ,(G_ (manual-href "defined" (G_ "en") (G_ "Defining-Packages.html")))
          " as native "
          ,(G_ `(a (@ (href ,(gnu-url "software/guile"))) "Guile"))
          " modules, using extensions to the "
          ,(G_ `(a (@ (href "http://schemers.org")) "Scheme"))
          " language—which makes it nicely hackable."))


Another program I wrote converts it to a POT file from which I make
this German translation PO file:


#. TRANSLATORS: Features and Defining Packages are section names
#. in the English (en) manual.
#: apps/base/templates/about.scm:64
msgid ""
"GNU Guix provides <1>state-of-the-art package management "
"features<1.1>en</1.1><1.2>Features.html</1.2></1> such as transactional "
"upgrades and roll-backs, reproducible\n"
"            build environments, unprivileged package management, and\n"
"            per-user profiles.  It uses low-level mechanisms from the "
"<2>Nix</2> package manager, but packages are "
"<3>defined<3.1>en</3.1><3.2>Defining-Packages.html</3.2></3> as native "
"<4>Guile</4> modules, using extensions to the <5>Scheme</5> language—which "
"makes it nicely hackable."
msgstr ""
"GNU Guix bietet <1>Paketverwaltungsfunktionalitäten auf dem Stand der "
"Technik<1.1>de</1.1><1.2>Funktionalitaten.html</1.2></1>, wie etwa "
"transaktionelle Aktualisierungen und Rücksetzungen, reproduzierbare "
"Erstellungsumgebungen, eine „unprivilegierte“ Paketverwaltung für Nutzer "
"ohne besondere Berechtigungen sowie ein eigenes Paketprofil für jeden "
"Nutzer. Dazu verwendet es dieselben Mechanismen, die dem "
"Paketverwaltungsprogramm <2>Nix</2> zu Grunde liegen, jedoch werden Pakete "
"als reine <4>Guile</4>-Module <3>definiert<3.1>de</3.1><3.2>Pakete-"
"definieren.html</3.2></3>. Dazu erweitert Guix die <5>Scheme</5>-"
"Programmiersprache, wodurch es leicht ist, selbst an diesen zu hacken."


Then the G_ macro takes apart the original SHTML and, using the
translation, reorders and recombines it automatically to this (I
manually added the indentation now):

(quasiquote
 (p "GNU Guix bietet "
    (unquote
     (manual-href "Paketverwaltungsfunktionalitäten auf dem Stand der Technik"
                  "de"
                  ""
                  "Funktionalitaten.html"
                  ""))
    ", wie etwa transaktionelle Aktualisierungen und Rücksetzungen, reproduzierbare Erstellungsumgebungen, eine „unprivilegierte“ Paketverwaltung für Nutzer ohne besondere Berechtigungen sowie ein eigenes Paketprofil für jeden Nutzer. Dazu verwendet es dieselben Mechanismen, die dem Paketverwaltungsprogramm "
    (unquote (quasiquote (a (@ (href "https://nixos.org/nix/"))
                            "Nix")))
    " zu Grunde liegen, jedoch werden Pakete als reine "
    (unquote (quasiquote (a (@ (href (unquote (gnu-url "software/guile"))))
                            "Guile")))
    "-Module "
    (unquote
     (manual-href "definiert"
                  "de"
                  ""
                  "Pakete-definieren.html"
                  ""))
    ". Dazu erweitert Guix die "
    (unquote (quasiquote (a (@ (href "http://schemers.org"))
                            "Scheme")))
    "-Programmiersprache, wodurch es leicht ist, selbst an diesen zu hacken."))



The reason I argued for such a gettext-like approach is that it is
less effort to mark existing SHTML for translation with G_ than if we
were to rewrite everything using something like sirgazil’s format
strings
<https://gitlab.com/sirgazil/guile-lab/blob/master/glab/i18n.scm>, I
think, and that the resulting msgid text in the POT file seems easier
to work with for translators.  Ricardo Wurmus proposed manual XML
without macros that would be easy for translators too, but a G_ macro
is still easier for SHTML authors, I think.  I believe this gettext
approach could be used easily for very many webpages and on other
websites; currently I have a not-yet-submitted completed
internationalization for the entire Guix website but without its very
many blog posts.


Now let me look at your comments:

On Wed, Sep 04, 2019 at 03:54:13PM -0400, Mark H Weaver wrote:
> Hi Florian,
> 
> "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> writes:
> 
> > […]
> 
> There are some problems above:
> 
> (1) The first argument to 'datum->syntax' must be an identifier, which
>     is the syntax object corresponding to a symbol.  Here, you are
>     passing an entire expression, and in the example usage above, #'exp
>     will be the syntax object corresponding to (* 2 3 hour).  Guile
>     should ideally raise an error in this case.
>

Ahh so in the example above it should be

(define-syntax one-more
  (lambda (x)
    (syntax-case x ()
      ((id exp)
       (datum->syntax
        #'id

instead of

> > (define-syntax one-more
> >   (lambda (x)
> >     (syntax-case x ()
> >       ((_ exp)
> >        (datum->syntax
> >         #'exp

I will change this.



> (2) The way you are doing things here destroys hygiene within the
>     expression that you are rewriting.  You convert the entire
>     expression with 'syntax->datum', process the datum, and then convert
>     the rewritten expression using 'datum->syntax'.  The problem here is
>     that 'syntax->datum' discards all of the extra information about
>     lexical environments of identifiers that were kept in the syntax
>     object.  This will cause severe problems when 'one-more' is used in
>     combination with other macros, including unintended variable
>     capture.
>

Yes, but I do not think capturing everything named G_ is a problem for
this special use case.  If someone were to use the ordinary gettext
for pure strings (let’s say its keyword is G_), it would be just as
much of a problem to use the chosen gettext keyword G_ for anything
other than gettext anywhere in the program, because xgettext would
include that other thing in the POT file for translators.

I acknowledge that error messages for errors within the marked
s-expression may be less precise, I suppose.




> To do this properly, you must do the rewriting on the syntax objects
> themselves.  It's okay to convert a syntax object to a datum to test
> whether it's a literal number, but the important thing is that all
> *identifiers* in the rewritten code should be preserved.
> 
> So, instead of using 'match' on the result of 'syntax->datum', you
> should instead use 'syntax-case' on the syntax object itself, like this
> (untested):
> 
>   (let loop ((e #'exp))
>     (syntax-case e ()
>       (num
>        (number? (syntax->datum #'num))
>        #'(1+ num))
>       ((x ...)
>        (map loop #'(x ...)))
>       (y
>        #'y)))
>

When I look at your second mail:

On Wed, Sep 04, 2019 at 05:55:58PM -0400, Mark H Weaver wrote:
> […]
> I should mention that the use of 'map' directly on a syntax object is
> only allowable in certain cases.
> […]
> As a result, there are only a few cases when you can safely assume that
> the top structure of a syntax object is a normal list or pair, and they
> are spelled out in the documentation for 'syntax' in the R6RS Standard
> Libraries specification:
> […]
>   * the copy of (<t> <ellipsis>) is a list if <t> contains any pattern
>     variables,
> […]

I come to believe that the recommended matching functionality of
syntax-case is not powerful enough for what I want the G_ macro to do.

Another consideration is what you mention last in your first mail:

> Finally, I should mention that macro expansion is always done from the
> outside in, meaning that when 'one-more' is expanded, its operand will
> not yet have been expanded.

This outside-in aspect is vital for what I want to do, and it is the
reason I use macros and not procedures, which I would understand much
better.  For my example above, the inner G_ “macros” within the outer
G_ s-expressions are removed from the output by the outer G_ on
purpose and are only used to distinguish sub-s-expressions with
translatable text from sub-s-expressions without translatable text.
If the inner G_ were evaluated before the outer G_ is evaluated, no
distinction would be possible.




> In general, this means that it's impossible
> to comprehend the code within a macro's operands unless the parsing code
> knows about every macro that might be used within the operands.  It's
> not even possible to know which subparts are expressions and which are
> other things like variable binding lists.
> 
> For this reason, I think it's generally a mistake to try to parse code
> within a macro's operands.  It normally only makes sense for macros to
> inspect the parts of operands that are considered part of the macro's
> syntax.  For example, it makes sense for a 'let' macro to parse its
> binding list, or for a 'match' macro to parse its patterns and
> templates, but it does *not* make sense for a macro to try to parse
> general subexpressions passed to the macro.
> 
> If you could give me a birds-eye view of what you're trying to do here,
> I might be able to suggest other approaches.
> 
>        Best,
>         Mark


I will add a comment to my code that my use of datum->syntax is a very
special use case warranting its use, and that in most cases, one
should not try this at home.

Regards,
Florian