Proposal: Identifier properties for Guile

Proposal: Identifier properties for Guile

[Daphne Preston-Kendal]

Hi!

I want to implement identifier properties a la SRFI 213 in Guile’s
psyntax.
<https://srfi.schemers.org/srfi-213/srfi-213.html>
These will be in R7RS Large (in the fascicle on macros coming out Real
Soon Now), and are in fact the only major macro feature Guile is
missing from R7 Large.

I want to gather feedback on my proposed strategy to implement these
into Guile before starting work. Along the way, I’ll explain some
nuances of the semantics which the SRFI explains very badly (the
fascicle on macros completely redid the text).

First of all, why would you want identifier properties? As an
admittedly pretty advanced macrologist, I’m convinced these are the
best things since sliced bread. I’ve gathered some example use cases
here:
<https://codeberg.org/scheme/r7rs/wiki/Identifier-property-use-cases>
and in fact the second use case is no longer a hypothetical as it was
when I wrote that page, because I have actually implemented it:
<https://codeberg.org/dpk/extensible-match>
Getting that library working in Guile and Hoot is my main personal
motivation for wanting this feature – apart from wanting to see an
implementation support the R7-large spec which I wrote ;-)

Identifier properties are fundamentally very similar to transformer
bindings, and pose some of the same challenges. The main difference is
that identifier properties are attached in addition to a binding,
rather than changing the regular Scheme semantics of the binding.
But like transformer bindings, it makes sense to divide them into
top-level (module-level, in Guile) property definitions and local
definitions.

The latter are easier to deal with because they don’t need to touch
the module system. The expander gets rid of them in the same way it
gets rid of let-syntax; expanded code doesn’t need to care about them
any more.

Implementing this means changing the expander’s idea of a wrap so that
it contains property bindings, much as has been done in Chez’s version
of psyntax:
<https://github.com/cisco/ChezScheme/blob/658e0b152abe3bc4ba889883d4dc218fef093aef/s/syntax.ss#L892>
This is a non-obtrusive change entirely localized to the expander as
it exists in a live Guile system. I don’t see any major issues here or
have any questions.

When we get to module-level property bindings, things get trickier.
I’d like a bit of guidance here. First, to clarify the SRFI:
identifier properties, as the name suggests, are attached to
identifiers in the sense of define/import/export. If you import an
identifier from module A which the unrelated module B has attached a
property to, you won’t see that property – unless you *also* re-import
the same identifier from module B. Identifiers are organized, though,
by keys, which themselves look like identifiers but those identifiers
are actually used only for their bindings in the sanse of
‘free-identifier=?’. (Keys have to be defined – the fallback to
symbolic comparison doesn’t apply here.)

To support this I will have to extend the definition of a module in
boot-9.scm to add, effectively, a second obarray – this one mapping
variables to mappings of bindings to property values, rather than
variables to values directly. However, it looks to me like there is
some duplicate definition between C and Scheme here, and I would also
need to update libguile/modules.c to at least be aware of the new
structure, and probably write the property mapping code in C there as
well to make it work as I describe below. Is this a correct
assumption?

Looking at how syntax transformers work in the current code, the
‘define-property’ form will probably have to be considered primitive
down to a fairly low level. As I understand it from looking at
expansion and disassembly, (define-syntax x y) first expands into
(define x (make-syntax-transformer something something y)), then on
the way from Tree-IL to bytecode it becomes an intrinsic call which is
effectively (define! (current-module) 'x) followed by a (set! 'x
(make-syntax-transformer something something y)). It looks like there
might be double definition here too: once in a direct route from
Tree-IL to bytecode, another with the same source and target but via
CPS.

In any case, based on this, I will add a <toplevel-define-property>
Tree-IL node which compiles to a new define-property! intrinsic the
same way <toplevel-define> compiles to a define!. That primitive will
add the var -> binding -> value mapping to the module. Since there is
no set! operation on properties, I think everything can be done in one
intrinsic. One minor question I will have to solve is how to represent
the binding used for the key down at this level. I am sure there is a
simple answer to this – possibly just a pair:
(<original-module> . <var-name>)

There is an alternative approach which looks easier, which would be to
have define-property at the top level expand directly into a call to a
new ‘module-define-property!’ procedure. But I assume there is a
reason macro transformers and other definitions were done the way they
are done …

Also, because this reaches so far down into the internals of Guile, I
assume it will be necessary to adapt some of this specially for Hoot
as well. I haven’t looked at the Hoot sources yet – it might be an
idea to wait until Andy Wingo has finished porting psyntax to Hoot
before trying anything there.

In any case, there is then one last step, which is to add code
adding/merging properties when modules are imported into other
modules. This should be a fairly simple hash table/alist/whatever
union operation, but I haven’t looked into it yet.

Since this does touch nearly every level of Guile’s compiler, I would
appreciate some feedback and ideally the opportunity to consult
occasionally while I’m working with someone who is more familiar with
all of this code than I am.

Many thanks in advance,

-- 
dpk (Daphne Preston-Kendal) ·· 12103 Berlin, Germany ·· http://dpk.io/
  One Thing to name them all, One Thing to define them,
  One Thing to place them in environments and bind them,
  In the Lambda Order they are all first class.               — R2RS

Re: Proposal: Identifier properties for Guile

[Mikael Djurfeldt]

Dear Daphne,

What you propose sounds exciting! I also think that you have the right
background to pull this off in a nice way.

Obviously, Andy Wingo should give his view on this, but I thought that
your message should have some response. :-)

Best regards,
Mikael Djurfeldt

On Tue, Sep 10, 2024 at 11:08 PM Daphne Preston-Kendal
<dpk@nonceword.org> wrote:
>
> Hi!
>
> I want to implement identifier properties a la SRFI 213 in Guile’s
> psyntax.
> <https://srfi.schemers.org/srfi-213/srfi-213.html>
> These will be in R7RS Large (in the fascicle on macros coming out Real
> Soon Now), and are in fact the only major macro feature Guile is
> missing from R7 Large.
>
> I want to gather feedback on my proposed strategy to implement these
> into Guile before starting work. Along the way, I’ll explain some
> nuances of the semantics which the SRFI explains very badly (the
> fascicle on macros completely redid the text).
>
> First of all, why would you want identifier properties? As an
> admittedly pretty advanced macrologist, I’m convinced these are the
> best things since sliced bread. I’ve gathered some example use cases
> here:
> <https://codeberg.org/scheme/r7rs/wiki/Identifier-property-use-cases>
> and in fact the second use case is no longer a hypothetical as it was
> when I wrote that page, because I have actually implemented it:
> <https://codeberg.org/dpk/extensible-match>
> Getting that library working in Guile and Hoot is my main personal
> motivation for wanting this feature – apart from wanting to see an
> implementation support the R7-large spec which I wrote ;-)
>
> Identifier properties are fundamentally very similar to transformer
> bindings, and pose some of the same challenges. The main difference is
> that identifier properties are attached in addition to a binding,
> rather than changing the regular Scheme semantics of the binding.
> But like transformer bindings, it makes sense to divide them into
> top-level (module-level, in Guile) property definitions and local
> definitions.
>
> The latter are easier to deal with because they don’t need to touch
> the module system. The expander gets rid of them in the same way it
> gets rid of let-syntax; expanded code doesn’t need to care about them
> any more.
>
> Implementing this means changing the expander’s idea of a wrap so that
> it contains property bindings, much as has been done in Chez’s version
> of psyntax:
> <https://github.com/cisco/ChezScheme/blob/658e0b152abe3bc4ba889883d4dc218fef093aef/s/syntax.ss#L892>
> This is a non-obtrusive change entirely localized to the expander as
> it exists in a live Guile system. I don’t see any major issues here or
> have any questions.
>
> When we get to module-level property bindings, things get trickier.
> I’d like a bit of guidance here. First, to clarify the SRFI:
> identifier properties, as the name suggests, are attached to
> identifiers in the sense of define/import/export. If you import an
> identifier from module A which the unrelated module B has attached a
> property to, you won’t see that property – unless you *also* re-import
> the same identifier from module B. Identifiers are organized, though,
> by keys, which themselves look like identifiers but those identifiers
> are actually used only for their bindings in the sanse of
> ‘free-identifier=?’. (Keys have to be defined – the fallback to
> symbolic comparison doesn’t apply here.)
>
> To support this I will have to extend the definition of a module in
> boot-9.scm to add, effectively, a second obarray – this one mapping
> variables to mappings of bindings to property values, rather than
> variables to values directly. However, it looks to me like there is
> some duplicate definition between C and Scheme here, and I would also
> need to update libguile/modules.c to at least be aware of the new
> structure, and probably write the property mapping code in C there as
> well to make it work as I describe below. Is this a correct
> assumption?
>
> Looking at how syntax transformers work in the current code, the
> ‘define-property’ form will probably have to be considered primitive
> down to a fairly low level. As I understand it from looking at
> expansion and disassembly, (define-syntax x y) first expands into
> (define x (make-syntax-transformer something something y)), then on
> the way from Tree-IL to bytecode it becomes an intrinsic call which is
> effectively (define! (current-module) 'x) followed by a (set! 'x
> (make-syntax-transformer something something y)). It looks like there
> might be double definition here too: once in a direct route from
> Tree-IL to bytecode, another with the same source and target but via
> CPS.
>
> In any case, based on this, I will add a <toplevel-define-property>
> Tree-IL node which compiles to a new define-property! intrinsic the
> same way <toplevel-define> compiles to a define!. That primitive will
> add the var -> binding -> value mapping to the module. Since there is
> no set! operation on properties, I think everything can be done in one
> intrinsic. One minor question I will have to solve is how to represent
> the binding used for the key down at this level. I am sure there is a
> simple answer to this – possibly just a pair:
> (<original-module> . <var-name>)
>
> There is an alternative approach which looks easier, which would be to
> have define-property at the top level expand directly into a call to a
> new ‘module-define-property!’ procedure. But I assume there is a
> reason macro transformers and other definitions were done the way they
> are done …
>
> Also, because this reaches so far down into the internals of Guile, I
> assume it will be necessary to adapt some of this specially for Hoot
> as well. I haven’t looked at the Hoot sources yet – it might be an
> idea to wait until Andy Wingo has finished porting psyntax to Hoot
> before trying anything there.
>
> In any case, there is then one last step, which is to add code
> adding/merging properties when modules are imported into other
> modules. This should be a fairly simple hash table/alist/whatever
> union operation, but I haven’t looked into it yet.
>
> Since this does touch nearly every level of Guile’s compiler, I would
> appreciate some feedback and ideally the opportunity to consult
> occasionally while I’m working with someone who is more familiar with
> all of this code than I am.
>
> Many thanks in advance,
>
> --
> dpk (Daphne Preston-Kendal) ·· 12103 Berlin, Germany ·· http://dpk.io/
>   One Thing to name them all, One Thing to define them,
>   One Thing to place them in environments and bind them,
>   In the Lambda Order they are all first class.               — R2RS
>
>