bug#27476: libguile/memoize.c is not thread safe, so syntax parameter expansion is not thread-safe

[Andy Wingo <wingo@igalia.com>]

Hi!

On Wed 06 Feb 2019 15:48, Ludovic Courtès <ludo@gnu.org> writes:

> I drew the conclusion that our syntax parameter is redefined when we
> compile or when we load (guix monads), so there’s a chance that we get
> to see the wrong value when we expand (guix monads) (I’m not entirely
> sure about the exact sequence of events.)

You are a wizard!!!!

To be clear, here's the series of events.  Firstly, know that defining a
syntax parameter is like:

  (define name
    (make-syntax-transformer 'name 'syntax-parameter (list f)))

So at the top level you end up with an association between a name and a
"syntax transformer" object (see macros.[ch]).  The syntax transformer
object itself consists of its name (for debugging), its syntax type, and
its syntax binding.

For syntax parameters, the binding is a list containing a single
element, the syntax transformer.  This list is later used as a key into
a compile-time environment, as it's a unique object associated with the
syntax parameter.

When (syntax-parameterize ((name f*)) exp) is seen, psyntax will look up
`name' in the current expansion-time environment.  It asserts that the
name is bound to a syntax transformer and that the syntax transformer is
indeed a syntax parameter, and extracts the associated binding `b'.
Keep in bind that `b' is the single-element list containing the
"default" syntax transformer `f'.

syntax-parameterize then does something weird: it adds an association
between the binding value `b' and `f*' to the expand-time environment.
It does this because the `b' is just a fresh object, so it's a unique
key that's usable for associations.  (The way this works is my fault
FWIW.)  To be clear, it doesn't add a new definition of `name'; it
instead establishes a new lexical binding for the unique object `b'.

Then when a use of `name' is seen within `exp', Guile finds that `name'
is a syntax parameter, extracts the binding from the syntax transformer
object, then does a second lookup of that binding.  If it finds
something bound, it uses that, otherwise it uses the default binding.

I think you see the race here.  For an initial state of

        (define P (stx-param (list F)))

we have:

           thread A                       thread B
                            time
    resolve P                 |
    extract B                 |
    associate B and F*        |
                              |   define P (stx-param (list F**))
    resolve P                 |
    extract B (!)             |
    resolve B (!)             |
    see F** instead of F* (!) |
                              v

> So I came up with ‘define-syntax-parameter-once’, which is like
> ‘define-once’ but for syntax parameters (note that we can’t use
> ‘define-once’ in ‘define-syntax-parameter-once’ because it expands to a
> reference to NAME, which doesn’t work for a macro):

Your fix is good!  But, it prevents redefinition of syntax parameters.

I would like to work on a solution that instead of using this
double-lookup, simply adds an association between P and F* in the
environment, instead of doing the double-lookup thing.  Probably that
will be 3.0-only.

For 2.2, we can probably update the compiler to trampoline through some
kind of "redefine-syntax" or something that will do (set-car! B F**)
instead of (define P (stx-param B*)).  I.e. redefinition keeps the
unique key there.

Andy