Mutating public bindings of a declarative module

Mutating public bindings of a declarative module

[Ludovic Courtès]

Hello!

It seems that if you ‘set!’ a public variable of a declarative module,
the change is visible to all the module users, but it’s not necessarily
visible to procedures within that module, presumably because they use an
inlined or specialized variant of that thing.

I would have imagined that public bindings are considered mutable and
thus not subject to inlining; OTOH, that would obviously be a loss, so
the current approach makes sense.

Anyway, it complicates a use case for me.  In Guix, we “mock” bindings
like so:

  (define-syntax-rule (mock (module proc replacement) body ...)
    "Within BODY, replace the definition of PROC from MODULE with the definition
  given by REPLACEMENT."
    (let* ((m (resolve-interface 'module))
           (original (module-ref m 'proc)))
      (dynamic-wind
        (lambda () (module-set! m 'proc replacement))
        (lambda () body ...)
        (lambda () (module-set! m 'proc original)))))

and that allows us to write tests that temporarily modify public (or
private!) bindings.

It seems like this could be addressed by compiling selected modules with
‘user-modules-declarative?’ set to #false, or by avoiding the above hack
altogether when possible, but I thought I’d share my impressions and
listen to what people think.  :-)

Thanks,
Ludo’.

Re: Mutating public bindings of a declarative module

[Amirouche Boubekki]

Le dim. 24 nov. 2019 à 18:54, Ludovic Courtès <ludo@gnu.org> a écrit :
>
> Hello!
>
> It seems that if you ‘set!’ a public variable of a declarative module,
> the change is visible to all the module users, but it’s not necessarily
> visible to procedures within that module, presumably because they use an
> inlined or specialized variant of that thing.
>
> I would have imagined that public bindings are considered mutable and
> thus not subject to inlining; OTOH, that would obviously be a loss, so
> the current approach makes sense.
>
> Anyway, it complicates a use case for me.  In Guix, we “mock” bindings
> like so:
>
>   (define-syntax-rule (mock (module proc replacement) body ...)
>     "Within BODY, replace the definition of PROC from MODULE with the definition
>   given by REPLACEMENT."
>     (let* ((m (resolve-interface 'module))
>            (original (module-ref m 'proc)))
>       (dynamic-wind
>         (lambda () (module-set! m 'proc replacement))
>         (lambda () body ...)
>         (lambda () (module-set! m 'proc original)))))
>
> and that allows us to write tests that temporarily modify public (or
> private!) bindings.
>
> It seems like this could be addressed by compiling selected modules with
> ‘user-modules-declarative?’ set to #false, or by avoiding the above hack
> altogether when possible, but I thought I’d share my impressions and
> listen to what people think.  :-)
>

For what it is worth, in my project I take a different approach to
mock. Some may call it inversion-of-control of something like that.
Basically, everything that must be mocked is passed as a procedure.

For instance, in babelia there is a pool of thread worker with a
fibers mainthread. There is three primitives: initialize the thread
pool, apply a thunk in a worker, and for-each-par-map. During the
tests I can not run the pool of thread worker because of the #:drain
behavior [0], I could workaround it some other way like explained in
the ticket by Wingo.

[0] https://github.com/wingo/fibers/issues/30

Anyway, the other advantage of making the thread pool configurable is
that my OKVS abstraction is not tied to it. The full-text search
abstraction dubbed fts is passed `apply` and `for-each-map` as last
two arguments in the constructor:


  (define-record-type <fts>
    (make-fts engine ustore prefix limit apply for-each-map)
    fts?
    (engine fts-engine)
    (prefix fts-prefix)
    (ustore fts-ustore)
    (limit fts-limit)
    (apply %fts-apply)
    (for-each-map %fts-for-each-map))

Then during the tests or else, I can pass custom implementations:

  (define (for-each-map sproc pproc lst)
    (for-each sproc (map pproc lst)))

  (define fts (make-fts engine
                        ustore
                        '(test-fts-prefix)
                        1
                        (lambda (thunk) (apply thunk '())) ;; fts-apply
                        for-each-map))

Similarly, in OKVS SRFI, see [1], to make database engine swappable, I
rely on a similar pattern that was dubbed typeclass object by SRFI-128
(comparators). In those cases, the record instance contains only
procedures.

[1] https://github.com/scheme-requests-for-implementation/srfi-167/blob/master/srfi/engine.sld#L1

The approach I described, that boils down to passing a wanna be mocked
procedure as argument, can work.

> Thanks,
> Ludo’.

Hope this helps.

Re: Mutating public bindings of a declarative module

[Andy Wingo]

Hi :)

On Sun 24 Nov 2019 18:54, Ludovic Courtès <ludo@gnu.org> writes:

> It seems that if you ‘set!’ a public variable of a declarative module,
> the change is visible to all the module users, but it’s not necessarily
> visible to procedures within that module, presumably because they use an
> inlined or specialized variant of that thing.
>
> I would have imagined that public bindings are considered mutable and
> thus not subject to inlining; OTOH, that would obviously be a loss, so
> the current approach makes sense.

Right, I understand the frustration.  For what it is worth, I think we
have the right default for what it means to be a declarative module, but
I'm definitely open to having that conversation.

> Anyway, it complicates a use case for me.  In Guix, we “mock” bindings
> like so:
>
>   (define-syntax-rule (mock (module proc replacement) body ...)
>     "Within BODY, replace the definition of PROC from MODULE with the definition
>   given by REPLACEMENT."
>     (let* ((m (resolve-interface 'module))
>            (original (module-ref m 'proc)))
>       (dynamic-wind
>         (lambda () (module-set! m 'proc replacement))
>         (lambda () body ...)
>         (lambda () (module-set! m 'proc original)))))
>
> and that allows us to write tests that temporarily modify public (or
> private!) bindings.
>
> It seems like this could be addressed by compiling selected modules with
> ‘user-modules-declarative?’ set to #false, or by avoiding the above hack
> altogether when possible, but I thought I’d share my impressions and
> listen to what people think.  :-)

This works.  (Actually the way I would do it is to pass #:declarative?
#f in the define-module for the modules in question.)  Marking some
bindings as not declarative also works (e.g. (set! foo foo)).

For me the most robust solution would be to have `mock' verify that the
module it's funging isn't declarative.  We don't currently have a way to
know if an individual module binding is declarative or not (though we
could add this).

Cheers,

Andy

Re: Mutating public bindings of a declarative module

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@igalia.com> skribis:

> On Sun 24 Nov 2019 18:54, Ludovic Courtès <ludo@gnu.org> writes:
>
>> It seems that if you ‘set!’ a public variable of a declarative module,
>> the change is visible to all the module users, but it’s not necessarily
>> visible to procedures within that module, presumably because they use an
>> inlined or specialized variant of that thing.
>>
>> I would have imagined that public bindings are considered mutable and
>> thus not subject to inlining; OTOH, that would obviously be a loss, so
>> the current approach makes sense.
>
> Right, I understand the frustration.  For what it is worth, I think we
> have the right default for what it means to be a declarative module, but
> I'm definitely open to having that conversation.

In C on GNU/Linux, if we have this in the same compilation unit:

  int foo () { … }
  int bar () { foo (); }

then ‘bar’ will not necessarily call the ‘foo’ that’s just above.  It’s
possible to replace ‘foo’ via loader tricks such as ‘LD_PRELOAD’.

That means that (1) ‘foo’ is not subject to inlining, and (2) which
‘foo’ ‘bar’ refers to is determined at load time.

So the semantics we have go somewhat further than this.  Not saying we
should follow C, but that’s what came to mind when thinking about the
pros and cons.

>> Anyway, it complicates a use case for me.  In Guix, we “mock” bindings
>> like so:
>>
>>   (define-syntax-rule (mock (module proc replacement) body ...)
>>     "Within BODY, replace the definition of PROC from MODULE with the definition
>>   given by REPLACEMENT."
>>     (let* ((m (resolve-interface 'module))
>>            (original (module-ref m 'proc)))
>>       (dynamic-wind
>>         (lambda () (module-set! m 'proc replacement))
>>         (lambda () body ...)
>>         (lambda () (module-set! m 'proc original)))))
>>
>> and that allows us to write tests that temporarily modify public (or
>> private!) bindings.
>>
>> It seems like this could be addressed by compiling selected modules with
>> ‘user-modules-declarative?’ set to #false, or by avoiding the above hack
>> altogether when possible, but I thought I’d share my impressions and
>> listen to what people think.  :-)
>
> This works.  (Actually the way I would do it is to pass #:declarative?
> #f in the define-module for the modules in question.)

The problem of #:declarative? is that you can’t wrap it in ‘cond-expand’.
I suppose most users will have a transition period during which both 2.2
and 3.0 are supported, and they cannot rely on 3.0-only forms during
that period.  :-/

> Marking some
> bindings as not declarative also works (e.g. (set! foo foo)).

Yes, I thought about that.  Perhaps we could provide a
‘preserve-binding!’ macro that does exactly that but would look less
weird.  :-)

> For me the most robust solution would be to have `mock' verify that the
> module it's funging isn't declarative.  We don't currently have a way to
> know if an individual module binding is declarative or not (though we
> could add this).

‘mock’ would have to error out if finds out that it cannot fiddle with a
binding, so it wouldn’t make much of a difference, just changing the
error type.

Thanks for your feedback!

Ludo’.