dynamic foreign function interface

dynamic foreign function interface

[Andy Wingo]

Hey all,

I just finished up some work to make a dynamic foreign function
interface. This is on the wip-ffi branch.

By "dynamic", I mean that you don't have to write C and compile it; you
can do everything at runtime from Scheme. You use dynamic-func and
dynamic-link to get the raw function pointer, and make-foreign-function
to turn that function pointer into a Scheme procedure.

The interface is very low-level. Obviously declaring that an arbitrary
symbol resolved via `dlsym' is of a certain function type is an unsafe
operation that can lead to crashes.

Apart from that typing problem, you have pointer and struct types. If
you say that the function takes an int8, Guile will ensure that it can
make an int8; but if you say that the function takes a pointer or a
by-value struct, Guile will only ensure that the arg is a "foreign"
(from foreign.[ch]) pointer, only checking lengths in the case that it's
a struct of known length.

The intention is to provide an expressive Scheme layer, on top of which
any safety constructs can be built as needed.

I would merge it now, except for the fact that it depends on libffi.
Libffi is very portable, and probably exists for all of Guile's
architectures, but it is an extra dependency. Should we require libffi
in Guile 1.9.8? Or should we build the necessary pieces conditionally?

Let me know,

Andy
-- 
http://wingolog.org/

Re: dynamic foreign function interface

[Ludovic Courtès]

Hello!

Andy Wingo <wingo@pobox.com> writes:

> I just finished up some work to make a dynamic foreign function
> interface. This is on the wip-ffi branch.

Nice!

> By "dynamic", I mean that you don't have to write C and compile it; you
> can do everything at runtime from Scheme. You use dynamic-func and
> dynamic-link to get the raw function pointer, and make-foreign-function
> to turn that function pointer into a Scheme procedure.
>
> The interface is very low-level. Obviously declaring that an arbitrary
> symbol resolved via `dlsym' is of a certain function type is an unsafe
> operation that can lead to crashes.

If it’s in a module of its own, then users will (in theory) be able to
prevent its use by “untrusted” code, which would be fine.

> Apart from that typing problem, you have pointer and struct types. If
> you say that the function takes an int8, Guile will ensure that it can
> make an int8; but if you say that the function takes a pointer or a
> by-value struct, Guile will only ensure that the arg is a "foreign"
> (from foreign.[ch]) pointer, only checking lengths in the case that it's
> a struct of known length.
>
> The intention is to provide an expressive Scheme layer, on top of which
> any safety constructs can be built as needed.

Cool.

One thing that would be neat is to integrate nicely with GNU ld’s symbol
versioning (it would work around some of the safety lost by not
compiling actual C code.)  For instance, one would be able to say:

  (dynamic-func "foo" lib "FOO_0.2")

That would use dlvsym() on GNU and ignore the last argument on other
systems.  (Though ideally ltdl would provide a wrapper for dlvsym).

> I would merge it now, except for the fact that it depends on libffi.
> Libffi is very portable, and probably exists for all of Guile's
> architectures, but it is an extra dependency. Should we require libffi
> in Guile 1.9.8? Or should we build the necessary pieces conditionally?

Well, it’s always annoying to add a dependency, but OTOH it was bound to
happen.  So... let’s go?

Thanks,
Ludo’.

Re: dynamic foreign function interface

[Andy Wingo]

Yellow,

On Tue 26 Jan 2010 23:29, ludo@gnu.org (Ludovic Courtès) writes:

>> By "dynamic", I mean that you don't have to write C and compile it; you
>> can do everything at runtime from Scheme. You use dynamic-func and
>> dynamic-link to get the raw function pointer, and make-foreign-function
>> to turn that function pointer into a Scheme procedure.
>>
>> The interface is very low-level. Obviously declaring that an arbitrary
>> symbol resolved via `dlsym' is of a certain function type is an unsafe
>> operation that can lead to crashes.
>
> If it’s in a module of its own, then users will (in theory) be able to
> prevent its use by “untrusted” code, which would be fine.

Well, yes. But anyone who has to make a sandbox should be specifying
the set of bindings that are available, not the set that are restricted.
But yes, I think we are in agreement hee.

> One thing that would be neat is to integrate nicely with GNU ld’s symbol
> versioning (it would work around some of the safety lost by not
> compiling actual C code.)  For instance, one would be able to say:
>
>   (dynamic-func "foo" lib "FOO_0.2")
>
> That would use dlvsym() on GNU and ignore the last argument on other
> systems.  (Though ideally ltdl would provide a wrapper for dlvsym).

Indeed this would be neat. I think ltdl is the right place for this to
go.

>> I would merge it now, except for the fact that it depends on libffi.
>> Libffi is very portable, and probably exists for all of Guile's
>> architectures, but it is an extra dependency. Should we require libffi
>> in Guile 1.9.8? Or should we build the necessary pieces conditionally?
>
> Well, it’s always annoying to add a dependency, but OTOH it was bound to
> happen.  So... let’s go?

Wow, ok. Well yes, we did always think this was going to happen... so
all right. I'll see what it takes, and merge when ready.

Andy
-- 
http://wingolog.org/

Re: dynamic foreign function interface

[Neil Jerram]

Andy Wingo <wingo@pobox.com> writes:

> Wow, ok. Well yes, we did always think this was going to happen... so
> all right. I'll see what it takes, and merge when ready.

FWIW, I'm really pleased to see this so soon.  I've been musing that an
FFI should allow us to co-opt all the binding libraries out there for
other scripting languages, notably Python.  (Of course, an FFI will also
allow us to use the underlying C library directly, but I would guess
that an existing language binding library might have more regular or
more introspectable function call signatures than the underlying
library, and so it would be easier to generate Scheme-level interfaces
automatically.)

Also note that if we had a Python (for example) translator, the FFI
would make the difference between only being able to execute pure Python
code (not using any binding libraries), and being able to execute _all_
Python code.

So, great news!

     Neil