FFI again

FFI again

[Stefan Monnier]

As you may remember, I'd really like for Emacs to grow some kind of FFI.
Last time we discussed it, I mostly remember the following points being made:
- An FFI can be a lot of work, and the benefit is not as obvious as it seems.
- Emacs-Guile would give us an FFI for free.
- The xwidget branch indirectly gives us some limited FFI.

The Emacs-Guile route seems promising, but I'm not sure I want to depend
on its schedule.  The xwidget one seems a bit too limited for my taste.
And I surely don't want to put a lot of work into it.

So I'd like an FFI, but one that's really cheap to design/implement.

The main purpose of an FFI, as far as I'm concerned, it to make it
possible for Emacs to use any .so library it feels, rather than only the
ones that it was compiled with.  More specifically, so that ELPA
packages can use such .so libraries if they feel like.

I think a "cheapish" way to do that is to make it possible for an ELPA
package to come with a .c file that exports a "syms_of_module" function
that can call things like defsubsr.

Installing such an ELPA package would require running a C compiler,
obviously (unless we also provide some pre-compiled versions for
particular target systems?).  And we'd need to add some function that
can load the resulting compiled object (along with the .so libraries it
depends on, since in many cases the whole purpose would be to call
functions in those .so libs).

It's not ideal, but it should be reasonably cheap to implement, and
would let people get access to pretty much any library they want.


        Stefan

Re: FFI again

[joakim]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> As you may remember, I'd really like for Emacs to grow some kind of FFI.
> Last time we discussed it, I mostly remember the following points being made:
> - An FFI can be a lot of work, and the benefit is not as obvious as it seems.
> - Emacs-Guile would give us an FFI for free.
> - The xwidget branch indirectly gives us some limited FFI.
>
> The Emacs-Guile route seems promising, but I'm not sure I want to depend
> on its schedule.  The xwidget one seems a bit too limited for my taste.
> And I surely don't want to put a lot of work into it.
>
> So I'd like an FFI, but one that's really cheap to design/implement.
>
> The main purpose of an FFI, as far as I'm concerned, it to make it
> possible for Emacs to use any .so library it feels, rather than only the
> ones that it was compiled with.  More specifically, so that ELPA
> packages can use such .so libraries if they feel like.
>
> I think a "cheapish" way to do that is to make it possible for an ELPA
> package to come with a .c file that exports a "syms_of_module" function
> that can call things like defsubsr.
>
> Installing such an ELPA package would require running a C compiler,
> obviously (unless we also provide some pre-compiled versions for
> particular target systems?).  And we'd need to add some function that
> can load the resulting compiled object (along with the .so libraries it
> depends on, since in many cases the whole purpose would be to call
> functions in those .so libs).
>
> It's not ideal, but it should be reasonably cheap to implement, and
> would let people get access to pretty much any library they want.

Whats wrong with libffi?
http://sourceware.org/libffi/

Its what everyone else uses, including Guile, Xwidgets, and the existing
Emacs FFI patches.

>
>
>         Stefan
>

-- 
Joakim Verona

Re: FFI again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 05 Oct 2013 12:11:26 -0400
> 
> I think a "cheapish" way to do that is to make it possible for an ELPA
> package to come with a .c file that exports a "syms_of_module" function
> that can call things like defsubsr.
> 
> Installing such an ELPA package would require running a C compiler,
> obviously (unless we also provide some pre-compiled versions for
> particular target systems?).  And we'd need to add some function that
> can load the resulting compiled object (along with the .so libraries it
> depends on, since in many cases the whole purpose would be to call
> functions in those .so libs).

On Windows, having a foreign library call any of the functions
implemented within Emacs will require us to produce an import library,
as part of building Emacs.  It's easy to do that, and GNU Make in its
latest pretest already does something similar, but I thought I'd
mention that anyway.  When the .c file in question is linked into a
shared library, the import library will have to be passed to the
linker as one of the command-line arguments.

Re: FFI again

[Stefan Monnier]

> Whats wrong with libffi?
> http://sourceware.org/libffi/

I don't know.  I can't remember it being mentioned.

> Its what everyone else uses, including Guile, Xwidgets, and the existing
> Emacs FFI patches.

Can't remember those existing FFI patches.  Any pointers?


        Stefan

Re: FFI again

[joakim]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Whats wrong with libffi?
>> http://sourceware.org/libffi/
>
> I don't know.  I can't remember it being mentioned.
>
>> Its what everyone else uses, including Guile, Xwidgets, and the existing
>> Emacs FFI patches.
>
> Can't remember those existing FFI patches.  Any pointers?

The author of the sxemacs libffi interface offered to provide it for
emacs some time ago on this list, as I recall.

If that doesnt pan out I dont think writing an emacs libffi interface
from scratch is too hard.

>
>         Stefan

-- 
Joakim Verona

Re: FFI again

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

On 10/5/13 9:11 AM, Stefan Monnier wrote:
> As you may remember, I'd really like for Emacs to grow some kind of FFI.
> Last time we discussed it, I mostly remember the following points being made:
> - An FFI can be a lot of work, and the benefit is not as obvious as it seems.
> - Emacs-Guile would give us an FFI for free.
> - The xwidget branch indirectly gives us some limited FFI.
> 
> The Emacs-Guile route seems promising, but I'm not sure I want to depend
> on its schedule.  The xwidget one seems a bit too limited for my taste.
> And I surely don't want to put a lot of work into it.
> 
> So I'd like an FFI, but one that's really cheap to design/implement.
> 
> The main purpose of an FFI, as far as I'm concerned, it to make it
> possible for Emacs to use any .so library it feels, rather than only the
> ones that it was compiled with.  More specifically, so that ELPA
> packages can use such .so libraries if they feel like.
> 
> I think a "cheapish" way to do that is to make it possible for an ELPA
> package to come with a .c file that exports a "syms_of_module" function
> that can call things like defsubsr.
> 
> Installing such an ELPA package would require running a C compiler,
> obviously (unless we also provide some pre-compiled versions for
> particular target systems?).  And we'd need to add some function that
> can load the resulting compiled object (along with the .so libraries it
> depends on, since in many cases the whole purpose would be to call
> functions in those .so libs).

I really don't like this idea. You either force users to have the Emacs
headers, Emacs import library, and a C compiler available to install a
package or you provide pre-compiled binaries for popular platforms and
create an ABI versioning nightmare. The routines declared in lisp.h do
not form stable interface. Sure, you could require that libraries export
a version number, then have Emacs refuse to load libraries compiled with
the wrong ABI version, but then you have to ship many different
pre-compiled binaries and take care to bump the ABI version when necessary.

I'd definitely prefer something libffi-based and with a CFFI-like interface.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: FFI again

[Stefan Monnier]

> The author of the sxemacs libffi interface offered to provide it for
> emacs some time ago on this list, as I recall.
> If that doesnt pan out I dont think writing an emacs libffi interface
> from scratch is too hard.

I don't remember it being mentioned.  I do vaguely remember someone
(Lars, maybe) mentioning that it can be a lot easier to write C code
that massages the lib calls to make them into subrs (as is done for
gnutls and libxml) than it is to write code for something like an FFI.

I personally don't have any experience either way.  But I think that
a good benchmark for an FFI is to make our libxml and libgnutls
code use it.

> I really don't like this idea.  You either force users to have the Emacs
> headers, Emacs import library, and a C compiler available to install a
> package or you provide pre-compiled binaries for popular platforms and
> create an ABI versioning nightmare.

The plan would be to require headers and a C compiler.
Precompiled libraries could be considered for the Windows platform, but
only if we can keep the versioning nightmare in check.

> The routines declared in lisp.h do not form stable interface.

Indeed, we'd need to define a slightly more stable API.


        Stefan

Re: FFI again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 06 Oct 2013 12:39:50 -0400
> Cc: emacs-devel@gnu.org
> 
> > I really don't like this idea.  You either force users to have the Emacs
> > headers, Emacs import library, and a C compiler available to install a
> > package or you provide pre-compiled binaries for popular platforms and
> > create an ABI versioning nightmare.
> 
> The plan would be to require headers and a C compiler.
> Precompiled libraries could be considered for the Windows platform, but
> only if we can keep the versioning nightmare in check.

I don't think the nightmare is as bad as it sounds.  Windows users
already keep such nightmares in check, including with Emacs.

Re: FFI again

[Stefan Monnier]

>> > I really don't like this idea.  You either force users to have the Emacs
>> > headers, Emacs import library, and a C compiler available to install a
>> > package or you provide pre-compiled binaries for popular platforms and
>> > create an ABI versioning nightmare.
>> The plan would be to require headers and a C compiler.
>> Precompiled libraries could be considered for the Windows platform, but
>> only if we can keep the versioning nightmare in check.
> I don't think the nightmare is as bad as it sounds.  Windows users
> already keep such nightmares in check, including with Emacs.

There's some hope, indeed, but notice that the problem is a bit more
serious than what we currently have for libgnutls and friends, because
that precompiled code needs to link both with lib<foo> and with Emacs,
and there's a potential version mismatch on both sides.


        Stefan "still hopeful"

Re: FFI again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: joakim@verona.se,  emacs-devel@gnu.org
> Date: Sun, 06 Oct 2013 14:17:30 -0400
> 
> >> > I really don't like this idea.  You either force users to have the Emacs
> >> > headers, Emacs import library, and a C compiler available to install a
> >> > package or you provide pre-compiled binaries for popular platforms and
> >> > create an ABI versioning nightmare.
> >> The plan would be to require headers and a C compiler.
> >> Precompiled libraries could be considered for the Windows platform, but
> >> only if we can keep the versioning nightmare in check.
> > I don't think the nightmare is as bad as it sounds.  Windows users
> > already keep such nightmares in check, including with Emacs.
> 
> There's some hope, indeed, but notice that the problem is a bit more
> serious than what we currently have for libgnutls and friends, because
> that precompiled code needs to link both with lib<foo> and with Emacs,
> and there's a potential version mismatch on both sides.

Just like libgnutls today, which has to link with libnettle,
linhogweed, libgmp, and p11-kit (to say nothing of the ubiquitous
libintl and zlib).

Re: FFI again

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

Adding an FFI is ok, provided we take precautions (as GCC does)
against making nonfree add-ons to Emacs.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: FFI again

[Stephen J. Turnbull]

Stefan Monnier writes:

 > I personally don't have any experience either way.

The SXEmacs people do, though (using libffi).  Their proof-of-concept
is a simple wrapper for libcurl.  If somebody wants to push forward
with this they should ask.  Most of the other scripting languages
(Python for sure has "ctypes", Ruby has something, maybe Guile does
too) have libffi wrappers.  That experience might be useful too.

Me, I'm just going to use the SXEmacs patch (for XEmacs).  If Steve Y
is willing to sign off, that is probably a good route for Emacs to
start with (the DEFUN macro is different and I don't think ours is
FSF-assigned, but the translation is routine).

 > But I think that a good benchmark for an FFI is to make our libxml
 > and libgnutls code use it.

libxml, maybe.  libgnutls I would continue to provide a dedicated
wrapper for.  It's very tricky to get the Lisp-to-C type matching
robust in an FFI, and if there's a problem in the FFI you probably
break all existing modules by fixing it.  In the meantime, if the FFI
doesn't do *all* the checks just right (and that's very hard to do in
standard C because called libraries typically use platform-dependent
types like "int" rather than "int32", and are often pretty cavalier
about casting internally), you have a crash waiting for some
unsuspecting Lisp code.  Nevertheless, there is a lot of pressure to
avoid changing the FFI.  Better to tweak all that stuff in dedicated
code when you're dealing with security code.

 > > The routines declared in lisp.h do not form stable interface.
 > 
 > Indeed, we'd need to define a slightly more stable API.

This is awfully tricky.  The Python guys spent something like ten
years developing their ABI specification, and had the advantage of a
religious mandate for backward compatibility.  Emacs has always been
wild and woolly with respect to APIs and ABIs since they're all
internal as far as Emacs is concerned.  Even in the published Lisp
Reference backward compatibility with 3rd party programs is frequently
(as measured with decade time units) sacrificed to somebody's sense of
symmetry.  Every release of Emacs has new surprises for XEmacs and
SXEmacs.  (This isn't a complaint, it's a statement of fact.  The
point is that these surprises are much more painful for users at the
level of C code because it's not safe and because it's harder for most
of them to change.)

Re: FFI again

[Stefan Monnier]

> libxml, maybe.  libgnutls I would continue to provide a dedicated
> wrapper for.  It's very tricky to get the Lisp-to-C type matching
> robust in an FFI, and if there's a problem in the FFI you probably
> break all existing modules by fixing it.  In the meantime, if the FFI

IIUC this is the kind of problem I was referring to, which would be
mostly avoided by using an FFI system based on hand-written C wrappers:
you don't have to export to Lisp all the C-level details.


        Stefan

Re: FFI again

[Stephen J. Turnbull]

Stefan Monnier writes:
 > > libxml, maybe.  libgnutls I would continue to provide a dedicated
 > > wrapper for.  It's very tricky to get the Lisp-to-C type matching
 > > robust in an FFI, and if there's a problem in the FFI you probably
 > > break all existing modules by fixing it.  In the meantime, if the FFI
 > 
 > IIUC this is the kind of problem I was referring to, which would be
 > mostly avoided by using an FFI system based on hand-written C wrappers:
 > you don't have to export to Lisp all the C-level details.

That's not FFI.  Are you confusing runtime loading of DSOs with a
foreign function interface (FFI)?  FFI means you can write bindings to
non-Lisp libraries in Lisp, rather than providing the ABI Lisp expects
using code in some other language.  The library called need not be
dynamically loaded, but in a dynamic language like Lisp, that's
obviously a very useful property.

XEmacs (and I believe SXEmacs) also has a *separate* facility for
dynamically loading C (and maybe C++?) modules (conventionally given
the extension .ell) with hand-written wrappers, written by J. Johnston
with loads of additional work by Jerry James and a few others.  This
integrates such modules with the feature system (`provide' and
`require').

Both facilities are useful but they are different, serving somewhat
different use cases.

Re: FFI again

[Stefan Monnier]

> That's not FFI.  Are you confusing runtime loading of DSOs with a
> foreign function interface (FFI)?

No, I'm not, but indeed what I propose is a kind of middle ground: in
normal FFI you describe the external library's ABI in some kind of
domain specific language (DSL) that is then interpreted/compiled into
wrappers to massage the data in a form understandable by the
host language.

So what I propose is to use "runtime loading of DSO" as a poor man's FFI
where the DSL of the FFI is C and where the compilation of this "DSL" is
handled by a C compiler.

The only technical difference between what I propose and typical "runtime
loading of DSOs" is that the system also handles running the C compiler,
so you can download the TedZ-OpenPGP package from ELPA and install it
(provided you have a C compiler on your machine).


        Stefan

Re: FFI again

[Andy Moreton]

On Mon 07 Oct 2013, Stefan Monnier wrote:

> So what I propose is to use "runtime loading of DSO" as a poor man's FFI
> where the DSL of the FFI is C and where the compilation of this "DSL" is
> handled by a C compiler.
>
> The only technical difference between what I propose and typical "runtime
> loading of DSOs" is that the system also handles running the C compiler,
> so you can download the TedZ-OpenPGP package from ELPA and install it
> (provided you have a C compiler on your machine).

This will work well enough for POSIX systems, but will fail miserably on
Windows systems. Most Windows users do not develpment tools installed,
and would have a hard time to find a C compiler producing binaries with
the correct ABI, and the other required build tools and dependencies.

Users should not have to deal with any of this in order to use libraries
that emacs uses via an FFI. They might just be able to deal with loading
a lisp package and adding a DLL to somewhere visible on the PATH, but
even that is too complicated for many (they expect installer programs to
do this work automatically).


    AndyM

Re: FFI again

[Stefan Monnier]

> Windows systems. Most Windows users do not develpment tools installed,
> and would have a hard time to find a C compiler producing binaries with
> the correct ABI, and the other required build tools and dependencies.

As mentioned, maybe this could be alleviated by providing precompiled
binaries for Windows users.


        Stefan

Re: FFI again

[Stephen J. Turnbull]

Andy Moreton writes:
 > On Mon 07 Oct 2013, Stefan Monnier wrote:
 > 
 > > So what I propose is to use "runtime loading of DSO" as a poor man's FFI
 > > where the DSL of the FFI is C and where the compilation of this "DSL" is
 > > handled by a C compiler.
 > >
 > > The only technical difference between what I propose and typical "runtime
 > > loading of DSOs" is that the system also handles running the C compiler,
 > > so you can download the TedZ-OpenPGP package from ELPA and install it
 > > (provided you have a C compiler on your machine).
 > 
 > This will work well enough for POSIX systems,

And in XEmacs, it has worked well enough for a decade now.  Windows,
too.  But very few users actually use it as far as I know, on either
type of system.

 > but will fail miserably on Windows systems. Most Windows users do
 > not develpment tools installed, and would have a hard time to find
 > a C compiler producing binaries with the correct ABI, and the other
 > required build tools and dependencies.

You don't need to worry about such users.  The XEmacs runtime DLL
loading system demonstrated optional static linking of the wrapper
code, and load the external DLL as usual at program execution.  The
wrapper code does an Fprovide at Emacs initialization, so the feature
code works as expected.  The price is a somewhat fatter binary and
perhaps loading an external library that would otherwise not take up
any memory.  No DDL wrapper, no problem for users.

Even if the necessary "stuff" is available, such a DLL loading system
really isn't very useful to ordinary users.  Almost nobody except
developers[1] who can save a lot of time by recompiling the module
only and then reloading to test really benefit (having done 3 of these
myself, I can say it *is* a nice feature), vs building the feature
into the Emacs binary.  If you're only occasionally going to rebuild
modules with a C compiler, it's not so much of additional burden to
rebuild Emacs as well.  If the necessary stuff *isn't* available then
you're dependent on somebody else to build an installer.  Again, these
days it may as well be all of Emacs as a single module (and quite
likely the corresponding external DLL).  That way the user doesn't
even need to put the DLLs in the right place.


Footnotes: 
[1]  The others are those who just love BrightShinyThings.

Re: FFI again

[Richard Stallman]

        [ To any NSA and FBI agents reading my email: please consider
        [ whether defending the US Constitution against all enemies,
        [ foreign or domestic, requires you to follow Snowden's example.

Supporting Windows is a low-priority goal for us.
We don't spend time on that; rather, we allow others to do so
if they wish.

In designing a FFI facility, we should not go to great lengths or
contort the design to make it work on Windows.  If the FFI facility
does not work on Windows, that's not a problem for us.



-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use Ekiga or an ordinary phone call.

Re: FFI again

[Stephen J. Turnbull]

Richard Stallman writes:

 > Supporting Windows is a low-priority goal for us.  We don't spend
 > time on that; rather, we allow others to do so if they wish.

Windows is just the most obvious example of a system where few users
will be able to use the on-demand link-loader alone to import non-Lisp
modules from an ELPA.  However, even in *free* systems non-developers
will encounter at least some of the problems Andy describes using
Windows as an example.

FWIW, I expect no need to contort the on-demand link-loader for
Windows.  IIUC, Emacs for Windows already has support for on-demand
link-loading (maybe better called "if-available" as currently
implemented?), and we didn't need to do any contorting of the "ellcc"
infrastructure that makes sure the right compiler is invoked and links
the module to the Lisp feature system.

Re: FFI again

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 07 Oct 2013 18:47:38 -0400
> Cc: emacs-devel@gnu.org
> 
> > Windows systems. Most Windows users do not develpment tools installed,
> > and would have a hard time to find a C compiler producing binaries with
> > the correct ABI, and the other required build tools and dependencies.
> 
> As mentioned, maybe this could be alleviated by providing precompiled
> binaries for Windows users.

Indeed, this is how this problem is usually solved.  E.g., bzr
provides precompiled binaries of the C extensions used by the bzr-git
plugin.

Re: FFI again

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Tue, 08 Oct 2013 14:33:48 +0900
> Cc: Andy Moreton <andrewjmoreton@gmail.com>, emacs-devel@gnu.org
> 
> IIUC, Emacs for Windows already has support for on-demand
> link-loading (maybe better called "if-available" as currently
> implemented?)

Yes, you understand correctly.  In fact, _all_ of the optional
libraries for which Emacs has interfaces and which are supported on
Windows -- all of them are loaded dynamically at run time, if
available, and when the corresponding feature is required/probed for.

> and we didn't need to do any contorting of the
> "ellcc" infrastructure that makes sure the right compiler is invoked
> and links the module to the Lisp feature system.

Right.