Breaking hygiene with syntax-rules?

Breaking hygiene with syntax-rules?

[Walter Lewis via General Guile related discussions]

Hi guile,

Consider this program:

(define-syntax unhygienic
   (syntax-rules ()
     ((_ the-pair fetch)
      (begin
        (define the-head (car the-pair))
        (define (fetch) the-head)))))

(unhygienic '(1) fetch1)
(unhygienic '(2) fetch2)

(fetch1)

;; => 2

It seems that the second call to unhygienic shadows the previous 
definition of the-head. I expected syntax-rules to generate fresh names 
for it each time.

Is this expected, and if so, is there any way to generate an internal 
definition using syntax-rules? For my case, I was writing a macro to 
generate SRFI-9 record definitions, but I noticed some of the getters 
and setters which I intended to be private were shadowing each other.

Best,

Walter

Re: Breaking hygiene with syntax-rules?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 2960 bytes --]


Walter Lewis via General Guile related discussions <guile-user@gnu.org> writes:

> (define-syntax unhygienic
>   (syntax-rules ()
>     ((_ the-pair fetch)
>      (begin
>        (define the-head (car the-pair))
>        (define (fetch) the-head)))))
>
> (unhygienic '(1) fetch1)
> (unhygienic '(2) fetch2)
>
> (fetch1)
>
> ;; => 2

I just tested this and see the same problem:

(unhygienic '(a) name1)
(unhygienic '(b) name2)
(name1) ;; => b
(name2) ;; => b

So it’s reproduced. Thank you for the minimal example!

> It seems that the second call to unhygienic shadows the previous
> definition of the-head. I expected syntax-rules to generate fresh
> names for it each time.
>
> Is this expected, and if so, is there any way to generate an internal
> definition using syntax-rules? For my case, I was writing a macro to
> generate SRFI-9 record definitions, but I noticed some of the getters
> and setters which I intended to be private were shadowing each other.

I did not expect this.

To track this:

(define-syntax unhygienic
  (syntax-rules ()
    ((_ the-pair fetch)
     (begin
       (define the-head (car the-pair))
       (define (the-proc) the-head)
       (define (fetch) the-head)
       (display the-proc)))))

(display the-head)
;; => error Unbound variable: the-head (step out of the debugger with C-d)
(unhygienic '(a) name1)
;; => #<procedure the-proc-1d2f8a1cb6ff0af6 ()> (the proc has a long suffix, looks good)
(display the-head)
;; => error: Unbound variable: the-head (looks good?)
(name1) ;; => a
(unhygienic '(b) name2)
;; #<procedure the-proc-1d2f8a1cb6ff0af6 ()> (the suffix of the proc is the same as for the one in a?)
(name1) ;; => b

Maybe this is just, because the procedure is *the same*?
Let’s use fetch in the-proc:

(define-syntax unhygienic
  (syntax-rules ()
    ((_ the-pair fetch)
     (begin
       (define the-head (car the-pair))
       (define (the-proc) fetch)   
       (define (fetch) the-head)
       (display the-proc)))))
(unhygienic '(a) name1)
;; => #<procedure the-proc-7f18b8db60e30e7 ()>
(unhygienic '(b) name2)
;; => #<procedure the-proc-ca383a2334aaa56 ()>
(name1)
;; => b


Now the proc is no longer the same, but the variable still collides.

Auto-completion in the REPL
the-head-<TAB>
;; => the-head-3f6c11022fffe02

the-head also has a suffix, as it should have, but there is only one.

Does the counter go wrong?

One more check:

(define-syntax unhygienic
  (syntax-rules ()
    ((_ the-pair fetch)
     (begin
       (define the-head fetch)
       (define (the-proc) fetch)   
       (define (fetch) the-head)
       (display the-proc)))))

Now using fetch in the-head I get two different variables in the shell.

Maybe detection of external data is missing that the-pair is external?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Breaking hygiene with syntax-rules?

[Jean Abou Samra]

[-- Attachment #1: Type: text/plain, Size: 445 bytes --]

Walter,

This is a known limitation in Guile. Please read
https://www.gnu.org/software/guile/manual/html_node/Hygiene-and-the-Top_002dLevel.html

I expect that you would be able to fix your macro by using

(define binding-you-want-to-report
  (let ((private-binding ...) ...)
   expr...))

or

(define-values (binding-to-export-1 binding-to-export-2 ...)
  (let ((private-binding ...) ...)
    (values expr1 expr2 ...)))


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Breaking hygiene with syntax-rules?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]


Jean Abou Samra <jean@abou-samra.fr> writes:

> This is a known limitation in Guile. Please read
> https://www.gnu.org/software/guile/manual/html_node/Hygiene-and-the-Top_002dLevel.html

I would not expect that effect from the description, because in

(define-syntax unhygienic
  (syntax-rules ()
    ((_ the-pair fetch)
     (begin
       (define the-head (car the-pair))
       (define (the-proc) the-head)
       (define (fetch) the-head)))))

the-head depends on (car the-pair) and the-pair is a variable passed to
the macro. So I would expect this to create different identifiers for
different values of the-pair, just like it creates different identifiers
when instead of (car the-pair) I use fetch (also passed from outside).

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Breaking hygiene with syntax-rules?

[Walter Lewis via General Guile related discussions]

Hi Jean,

On 8/11/23 4:00 AM, Jean Abou Samra wrote:
> This is a known limitation in Guile. Please read
> https://www.gnu.org/software/guile/manual/html_node/Hygiene-and-the-Top_002dLevel.html
>
> I expect that you would be able to fix your macro by using
>
> (define binding-you-want-to-report
>   (let ((private-binding ...) ...)
>    expr...))
>
> or
>
> (define-values (binding-to-export-1 binding-to-export-2 ...)
>   (let ((private-binding ...) ...)
>     (values expr1 expr2 ...)))

Unfortunately this won't work for my use case, since I am writing a 
helper to generate record type definitions. The code is something like 
this (slightly simplified for now):

(use-modules (srfi srfi-9) (system foreign))

(define-syntax define-ffi-wrapper
   ((_ size
       name wrap unwrap predicate
       to-pointer from-pointer)
    (begin
      (define-record-type name
       (wrap bv pointer)
       predicate
       (pointer get-pointer set-pointer!))
      (define (to-pointer x)
        (or (get-pointer x)
            (let ((pointer (bytevector->pointer x)))
              (set-pointer! x pointer)
              pointer)))
      (define (from-pointer pointer)
        (wrap (pointer->bytevector pointer size)
              #f)))))

Basically it is a wrapper similar to define-wrapped-pointer-type, except 
when you cast it to a pointer it memoizes the pointer for later. (It's 
actually inspired by the vec2 implementation in Dave Thompson's 
Chickadee.) I wanted to avoid writing all this boilerplate by hand for 
each wrapper type, but I ran into this shadowing issue with the internal 
definitions such as get-pointer and set-pointer!.

Re: Breaking hygiene with syntax-rules?

[Walter Lewis via General Guile related discussions]

Sorry I typed that code offhand and it had some mistakes; should be:

(use-modules (srfi srfi-9) (system foreign))

(define-syntax define-ffi-wrapper
   (syntax-rules ()
     ((_ size
         name wrap unwrap predicate
         to-pointer from-pointer)
      (begin
        (define-record-type name
          (wrap bv pointer)
          predicate
          (bv unwrap)
          (pointer pointer-ref pointer-set!))
        (define (to-pointer x)
          (or (pointer-ref x)
              (let ((pointer (bytevector->pointer x)))
                (pointer-set! x pointer)
                pointer)))
        (define (from-pointer pointer)
          (wrap (pointer->bytevector pointer size)
                pointer))))))

Re: Breaking hygiene with syntax-rules?

[Jean Abou Samra]

[-- Attachment #1: Type: text/plain, Size: 2209 bytes --]

Le vendredi 11 août 2023 à 18:50 -0400, Walter Lewis via General Guile related discussions a écrit :

> Sorry I typed that code offhand and it had some mistakes; should be:
> 
> ```
> (use-modules (srfi srfi-9) (system foreign))
> 
> (define-syntax define-ffi-wrapper
>    (syntax-rules ()
>      ((_ size
>          name wrap unwrap predicate
>          to-pointer from-pointer)
>       (begin
>         (define-record-type name
>           (wrap bv pointer)
>           predicate
>           (bv unwrap)
>           (pointer pointer-ref pointer-set!))
>         (define (to-pointer x)
>           (or (pointer-ref x)
>               (let ((pointer (bytevector->pointer x)))
>                 (pointer-set! x pointer)
>                 pointer)))
>         (define (from-pointer pointer)
>           (wrap (pointer->bytevector pointer size)
>                 pointer))))))
> ```


You could just turn that into a syntax-case macro, use gensym to generate the names, and insert them using with-syntax or quasisyntax. For example (untested):

```
(use-modules (srfi srfi-9) (system foreign))

(define-syntax define-ffi-wrapper
   (lambda (sintax)
     (syntax-case sintax ()
       ((_ size
           name wrap unwrap predicate
           to-pointer from-pointer)
        (with-syntax ((pointer-ref (datum->syntax sintax (gensym "pointer-ref")))
                      (pointer-set! (datum->syntax sintax (gensym "pointer-set!"))))
          #'(begin
              (define-record-type name
                (wrap bv pointer)
                predicate
                (bv unwrap)
                (pointer pointer-ref pointer-set!))
              (define (to-pointer x)
                (or (pointer-ref x)
                    (let ((pointer (bytevector->pointer x)))
                      (pointer-set! x pointer)
                      pointer)))
              (define (from-pointer pointer)
                (wrap (pointer->bytevector pointer size)
                      pointer))))))))
```

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Breaking hygiene with syntax-rules?

[Jean Abou Samra]

[-- Attachment #1: Type: text/plain, Size: 324 bytes --]

By the way, I'm rather confused as to why you deem this caching useful. A
priori, I would expect a simple bytevector->pointer call would be just as fast
as a to-pointer call. Do you somehow create lots of pointers to the contents of
the same bytevector so that weak references they hold incur a noticeable GC
overhead?

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Breaking hygiene with syntax-rules?

[Jean Abou Samra]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]

Le vendredi 11 août 2023 à 10:33 +0200, Dr. Arne Babenhauserheide a écrit :
> 
> Jean Abou Samra <jean@abou-samra.fr> writes:
> 
> > This is a known limitation in Guile. Please read
> > https://www.gnu.org/software/guile/manual/html_node/Hygiene-and-the-Top_002dLevel.html
> 
> I would not expect that effect from the description, because in
> 
> (define-syntax unhygienic
>   (syntax-rules ()
>     ((_ the-pair fetch)
>      (begin
>        (define the-head (car the-pair))
>        (define (the-proc) the-head)
>        (define (fetch) the-head)))))
> 
> the-head depends on (car the-pair) and the-pair is a variable passed to
> the macro. So I would expect this to create different identifiers for
> different values of the-pair, just like it creates different identifiers
> when instead of (car the-pair) I use fetch (also passed from outside).


You're right, that's confusing.

I don't have the time to investigate, though.



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Breaking hygiene with syntax-rules?

[Walter Lewis]

Hi Jean,

> You could just turn that into a syntax-case macro, use gensym to 
> generate the names, and insert them using with-syntax or quasisyntax.

Yes, this is what I ended up doing before I saw your other reply.

> By the way, I'm rather confused as to why you deem this caching 
> useful. A priori, I would expect a simple bytevector->pointer call 
> would be just as fast as a to-pointer call. Do you somehow create lots 
> of pointers to the contents of the same bytevector so that weak 
> references they hold incur a noticeable GC overhead?

To be honest I don't know enough about C to know the performance of 
bytevector->pointer, so I was assuming Chickadee's approach was done for 
a reason. But if you think it's not a big deal then I'm happy to 
simplify things! I think I will remove this caching for now.

Thanks for your help, and Arne as well for digging into the issue.

Best,

Walter

Re: Breaking hygiene with syntax-rules?

[Jean Abou Samra]

[-- Attachment #1: Type: text/plain, Size: 2131 bytes --]

Le vendredi 11 août 2023 à 20:19 -0400, Walter Lewis a écrit :
> To be honest I don't know enough about C to know the performance of 
> bytevector->pointer, so I was assuming Chickadee's approach was done for 
> a reason.


Below is the definition of bytevector->pointer in libguile/foreign.c:


SCM_DEFINE (scm_bytevector_to_pointer, "bytevector->pointer", 1, 1, 0,
	    (SCM bv, SCM offset),
	    "Return a pointer pointer aliasing the memory pointed to by\n"
            "@var{bv} or @var{offset} bytes after @var{bv} when @var{offset}\n"
	    "is passed.")
#define FUNC_NAME s_scm_bytevector_to_pointer
{
  SCM ret;
  signed char *ptr;
  size_t boffset;

  SCM_VALIDATE_BYTEVECTOR (1, bv);
  ptr = SCM_BYTEVECTOR_CONTENTS (bv);

  if (SCM_UNBNDP (offset))
    boffset = 0;
  else
    boffset = scm_to_unsigned_integer (offset, 0,
                                       SCM_BYTEVECTOR_LENGTH (bv) - 1);

  ret = scm_from_pointer (ptr + boffset, NULL);
  register_weak_reference (ret, bv);
  return ret;
}
#undef FUNC_NAME


In plain English, this is basically just checking that the first argument
is a bytevector and the second argument (if provided, defaults to 0)
is an integer that is in bounds for the bytevector, then it calls


SCM
scm_from_pointer (void *ptr, scm_t_pointer_finalizer finalizer)
{
  SCM ret;

  if (ptr == NULL && finalizer == NULL)
    ret = null_pointer;
  else
    {
      ret = scm_cell (scm_tc7_pointer, (scm_t_bits) ptr);

      if (finalizer)
        scm_i_set_finalizer (SCM2PTR (ret), pointer_finalizer_trampoline,
                             finalizer);
    }

  return ret;
}


which just wraps the pointer into a (small and cheap) Guile object,
and that's basically it. So, yet, it's very cheap.

The only possible cost that I can see is memory and GC footprint,
e.g., it's like creating many SRFI-111 boxes to the same object
vs. one box, that kind of overhead. The objects themselves are
very lightweight. You would really have to go out of your way and
create an awful lot of them for this to make any difference.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Breaking hygiene with syntax-rules?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 520 bytes --]


Jean Abou Samra <jean@abou-samra.fr> writes:

>  the macro. So I would expect this to create different identifiers for
>  different values of the-pair, just like it creates different identifiers
>  when instead of (car the-pair) I use fetch (also passed from outside).
>
> You're right, that's confusing.
>
> I don't have the time to investigate, though.

No problem — thank you for your support so far!

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Breaking hygiene with syntax-rules?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]


Walter Lewis <wklew@mailbox.org> writes:

>> By the way, I'm rather confused as to why you deem this caching
>> useful. A priori, I would expect a simple bytevector->pointer call
>> would be just as fast as a to-pointer call. Do you somehow create
>> lots of pointers to the contents of the same bytevector so that weak
>> references they hold incur a noticeable GC overhead?
>
> To be honest I don't know enough about C to know the performance of
> bytevector->pointer, so I was assuming Chickadee's approach was done
> for a reason.

Chickadee is pretty heavily optimized (it’s by dthompson who AFAIR once
showed millions of interacting points with Guile 3). I would expect that
the to-pointer becomes fully inlined, so it’s optimized away.

> But if you think it's not a big deal then I'm happy to
> simplify things! I think I will remove this caching for now.
>
> Thanks for your help, and Arne as well for digging into the issue.

Glad to help :-) — though I cannot yet help beyond "this looks like a
bug to me".

Best wishes,
Arne

PS: I just realized that GNU is turning 40 on Sep. 30 this year … 
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: Breaking hygiene with syntax-rules?

[Thompson, David]

Hi Walter and Arne,

On Sat, Aug 12, 2023 at 2:09 AM Dr. Arne Babenhauserheide
<arne_bab@web.de> wrote:
>
>
> Walter Lewis <wklew@mailbox.org> writes:
>
> >> By the way, I'm rather confused as to why you deem this caching
> >> useful. A priori, I would expect a simple bytevector->pointer call
> >> would be just as fast as a to-pointer call. Do you somehow create
> >> lots of pointers to the contents of the same bytevector so that weak
> >> references they hold incur a noticeable GC overhead?
> >
> > To be honest I don't know enough about C to know the performance of
> > bytevector->pointer, so I was assuming Chickadee's approach was done
> > for a reason.
>
> Chickadee is pretty heavily optimized (it’s by dthompson who AFAIR once
> showed millions of interacting points with Guile 3). I would expect that
> the to-pointer becomes fully inlined, so it’s optimized away.

The reason the pointers are cached in Chickadee's 2d/3d vector types
is because it's fairly common to need to pass the contents of a
vec2/vec3 to some OpenGL FFI binding. Most OpenGL calls happen while
rendering a frame, so calling bytevector->pointer around 60 times per
second for each vector that needs to be sent to OpenGL generates a lot
of garbage. I have to avoid generating garbage wherever possible in
Chickadee to reduce noticeable GC pauses in the rendered game. If
you're not dealing with these types of performance constraints then it
is definitely simpler to not cache pointers.

- Dave