FFI in Emacs

FFI in Emacs

[Aurélien Aptel]

Hi all,

I know this subject has already been discussed in the past but the
people involved and their opinions may have changed since then.

For those who don't know, a Foreign Function Interface is basically a
way to use code from one language in another. More on wikipedia [1].
Here, it's calling dynamically C code from Emacs-Lisp. Dynamically,
means without the need to recompile/link Emacs. One could use a
function in a C library from ELisp just by knowing its prototype.

It has already been done in SXEmacs [2] (a fork of a fork of GNU
Emacs) and I think I've heard on IRC that someone ported it but it was
never merged or suffered from bitrot. By the way, SXEmacs also has
some very nice features like raw strings (= more readable regex), new
data structures (skip lists, double linked list, ...) or compiled
regex that could be ported.

All in all, FFI is a really cool feature. The only real problem is
that some people (and particularly Dr. Stallman) think that a FFI
might lead to the development of proprietary plugins. Someone could
write a nice plugin, only give away a compiled version of the plugin
and an elisp wrapper. Not so long ago, there was a similar problem
with GCC that you might have heard about. GCC did not provide any
plugin interface for the same fear of having a proprietary plugin. It
all changed in the 4.5 release (2010) and you can know write a plugin
without recompiling GCC.
I think transition to GPLv3 has a lot to do with this change because
the new license is more secure (legally speaking) and covers more
cases that used to be problematic in v2. Maybe it's time to do the
same for Emacs and start moving forward.

1: http://en.wikipedia.org/wiki/Foreign_function_interface
2: http://www.sxemacs.org/

Re: FFI in Emacs

[Glenn Morris]

Aurélien Aptel wrote:

> All in all, FFI is a really cool feature. The only real problem is
> that some people (and particularly Dr. Stallman) think that a FFI
> might lead to the development of proprietary plugins.

This aspect was resolved years ago.

http://lists.gnu.org/archive/html/emacs-devel/2010-03/msg00240.html

   We gave GCC a plug-in interface last year, after developing a method
   to make it safer.  So now it is ok to do this in Emacs as well.

http://www.gnu.org/prep/standards/html_node/Dynamic-Plug_002dIn-Interfaces.html

Re: FFI in Emacs

[John Wiegley]

>>>>> Glenn Morris <rgm@gnu.org> writes:

> Aurélien Aptel wrote:
>> All in all, FFI is a really cool feature. The only real problem is that
>> some people (and particularly Dr. Stallman) think that a FFI might lead to
>> the development of proprietary plugins.

> This aspect was resolved years ago.

> http://lists.gnu.org/archive/html/emacs-devel/2010-03/msg00240.html

This is fantastic, I must have missed this!

So if we update Dave's patch, it can be committed toward 24.2?

    http://www.loveshack.ukfsn.org/emacs/dynamic-loading/

John

Re: FFI in Emacs

[Glenn Morris]

"John Wiegley" wrote:

> So if we update Dave's patch, it can be committed toward 24.2?
>
>     http://www.loveshack.ukfsn.org/emacs/dynamic-loading/

Obviously there are many factors that go in to answering that question.

One to possibly bear in mind is that in recent years he no longer wants
to assign his copyright to the FSF (IIUC). I don't know if that has any
relevance to code previously written with the obvious intent of being in
Emacs, but never actually committed. I guess not (?).

Re: FFI in Emacs

[Stefan Monnier]

>> So if we update Dave's patch, it can be committed toward 24.2?
>> http://www.loveshack.ukfsn.org/emacs/dynamic-loading/

Possibly, yes.  I haven't looked at the code, so I can't whether I'd
like it or not, and we'd want someone willing to maintain it.

> One to possibly bear in mind is that in recent years he no longer wants
> to assign his copyright to the FSF (IIUC).

AFAIK, it's more subtle than that.  So I think he'd be perfectly OK if
we used his code.  But for some reason, we seem to have trouble agreeing
on maintenance, so we'd need someone to very actively maintain it.


        Stefan

Re: FFI in Emacs

[John Wiegley]

>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:

> AFAIK, it's more subtle than that.  So I think he'd be perfectly OK if we
> used his code.  But for some reason, we seem to have trouble agreeing on
> maintenance, so we'd need someone to very actively maintain it.

This is definitely something I'd be interested in maintaining for OS X, but I
wouldn't be able to assure functionality on other platforms.

John

Re: FFI in Emacs

[joakim]

Glenn Morris <rgm@gnu.org> writes:

> "John Wiegley" wrote:
>
>> So if we update Dave's patch, it can be committed toward 24.2?
>>
>>     http://www.loveshack.ukfsn.org/emacs/dynamic-loading/
>
> Obviously there are many factors that go in to answering that question.
>
> One to possibly bear in mind is that in recent years he no longer wants
> to assign his copyright to the FSF (IIUC). I don't know if that has any
> relevance to code previously written with the obvious intent of being in
> Emacs, but never actually committed. I guess not (?).

If no existing patch can be used, I can write one. (I have experience of
libffi, etc)

-- 
Joakim Verona

Re: FFI in Emacs

[Nix]

On 28 Jun 2012, John Wiegley verbalised:

>>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> AFAIK, it's more subtle than that.  So I think he'd be perfectly OK if we
>> used his code.  But for some reason, we seem to have trouble agreeing on
>> maintenance, so we'd need someone to very actively maintain it.
>
> This is definitely something I'd be interested in maintaining for OS X, but I
> wouldn't be able to assure functionality on other platforms.

This is definitely something I'd be using on x86-64 Linux, so if it
broke I'd be trying to fix it. (Whether I've got the chops to maintain
something like this is another matter.)

-- 
NULL && (void)

Re: FFI in Emacs

[John Wiegley]

>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> So if we update Dave's patch, it can be committed toward 24.2?
>>> http://www.loveshack.ukfsn.org/emacs/dynamic-loading/

> Possibly, yes.  I haven't looked at the code, so I can't whether I'd like it
> or not, and we'd want someone willing to maintain it.

I took a look at Dave's code, and it's not so much an FFI patch as a "writing
Emacs Lisp functions in C" patch.

A real FFI patch would let me dlopen any C library and call functions within
it, such as openssl or zlib.  Dave's patch is more like what Python allows:
writing Emacs Lisp extensions in C that can be loaded as shared libraries.

John

Re: FFI in Emacs

[Helmut Eller]

On Wed, Jul 11 2012, John Wiegley wrote:

> I took a look at Dave's code, and it's not so much an FFI patch as a "writing
> Emacs Lisp functions in C" patch.
>
> A real FFI patch would let me dlopen any C library and call functions within
> it, such as openssl or zlib.  Dave's patch is more like what Python allows:
> writing Emacs Lisp extensions in C that can be loaded as shared libraries.

Well, you'd have a hard time to dlopen C macros.

Helmut

Re: FFI in Emacs

[Michael Albinus]

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> So if we update Dave's patch, it can be committed toward 24.2?
>>>> http://www.loveshack.ukfsn.org/emacs/dynamic-loading/
>
>> Possibly, yes.  I haven't looked at the code, so I can't whether I'd like it
>> or not, and we'd want someone willing to maintain it.
>
> I took a look at Dave's code, and it's not so much an FFI patch as a "writing
> Emacs Lisp functions in C" patch.
>
> A real FFI patch would let me dlopen any C library and call functions within
> it, such as openssl or zlib.  Dave's patch is more like what Python allows:
> writing Emacs Lisp extensions in C that can be loaded as shared libraries.

Maybe it's worth to look on what the Guile folks have done.

<http://www.gnu.org/software/guile/manual/html_node/Foreign-Function-Interface.html#Foreign-Function-Interface>

> John

Best regards, Michael.

Re: FFI in Emacs

[Stephen J. Turnbull]

John Wiegley writes:

 > A real FFI patch

Which both Python (aka the ctypes module) and SXEmacs have.

    http://www.sxemacs.org/

SXEmacs' implementation is about 5 years old, I believe, so the bugs
should be pretty well worked out.  It's not as elaborate as Python's
ctypes, but quite usable.  I don't see why such code would be
particularly SXEmacs-specific (up to the difference in the DEFUN
macro), so it should be fairly easily portable to Emacs on platforms
that have libffi.  It probably doesn't work on Windows (SXEmacs hasn't
supported Windows for quite a while).

I don't know if the authors would assign; it's easy enough to ask
them, though.  The VCS knows who they are.  (It should be quite
reliable as SXEmacs has always used a pull-style workflow based on a
dVCS.  You shouldn't have to worry about "committed on behalf of").

Re: FFI in Emacs

[Michael Albinus]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> SXEmacs' implementation is about 5 years old, I believe, so the bugs
> should be pretty well worked out.  It's not as elaborate as Python's
> ctypes, but quite usable.

When I rewrote dbusbind.c into Lisp, using SXEmacs' FFI interface, I
didn't find it very comfortable. Maybe due to lack of documentation,
maybe (more likely) due to my stupidness. Besides of this, I did succeed.

The SXEmacs guys seem to have replaced (?) this by dynamic modules. But
this new feature doesn't fly yet, as I have experienced recently.
However, looks interesting.

Best regards, Michael.

Re: FFI in Emacs

[joakim]

joakim@verona.se writes:

> Glenn Morris <rgm@gnu.org> writes:
>
>> "John Wiegley" wrote:
>>
>>> So if we update Dave's patch, it can be committed toward 24.2?
>>>
>>>     http://www.loveshack.ukfsn.org/emacs/dynamic-loading/>
>> Obviously there are many factors that go in to answering that question.
>>
>> One to possibly bear in mind is that in recent years he no longer wants
>> to assign his copyright to the FSF (IIUC). I don't know if that has any
>> relevance to code previously written with the obvious intent of being in
>> Emacs, but never actually committed. I guess not (?).
>
> If no existing patch can be used, I can write one. (I have experience of
> libffi, etc)

I'm currently looking at making the xwidget branch support Gobject
Introspection(GIR), to a degree. This is not full FFI support, but it would
have several interesting consequences:

- Many more gtk widgets could be embedded in an emacs buffer(rather than
  the now hardcoded ones)

- A number of interesting non-visual gnome libraries support
  introspection

I think this would be nice and useful. GIR does use libffi under the
hood.

Anyway, that was just a heads up. Currently it's not going too well
because I find the GIR documentation to be not so lucid in some areas.

BTW a user reported success in running the xwidget branch on osx, with
an embedded webkit! This semi-shows that the approach is viable on
several plattforms.

-- 
Joakim Verona

RE: FFI in Emacs

[Burton Samograd]

From: emacs-devel-bounces+burton.samograd=markit.com@gnu.org [mailto:emacs-devel-bounces+burton.samograd=markit.com@gnu.org] On Behalf Of Michael Albinus
Sent: Wednesday, July 11, 2012 12:43 AM
To: emacs-devel@gnu.org
Subject: Re: FFI in Emacs

"John Wiegley" <johnw@newartisans.com> writes:

>>>>>> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> So if we update Dave's patch, it can be committed toward 24.2?
>>>> http://www.loveshack.ukfsn.org/emacs/dynamic-loading/
>
>> Possibly, yes.  I haven't looked at the code, so I can't whether I'd
>> like it or not, and we'd want someone willing to maintain it.
>
> I took a look at Dave's code, and it's not so much an FFI patch as a
> "writing Emacs Lisp functions in C" patch.
>
> A real FFI patch would let me dlopen any C library and call functions
> within it, such as openssl or zlib.  Dave's patch is more like what Python allows:
> writing Emacs Lisp extensions in C that can be loaded as shared libraries.

> Maybe it's worth to look on what the Guile folks have done.

> http://www.gnu.org/software/guile/manual/html_node/Foreign-Function-Interface.html#Foreign-> Function-Interface

I might also suggest looking CFFI which is a wrapper around the mess of individual FFI APIs for the common lisp world.  I would think that it is a good example of 'common denominator' for FFI features:

http://common-lisp.net/project/cffi/

--
Burton Samograd


This e-mail, including accompanying communications and attachments, is strictly confidential and only for the intended recipient. Any retention, use or disclosure not expressly authorised by Markit is prohibited. This email is subject to all waivers and other terms at the following link: http://www.markit.com/en/about/legal/email-disclaimer.page

Please visit http://www.markit.com/en/about/contact/contact-us.page? for contact information on our offices worldwide.

RE: FFI in Emacs

[Burton Samograd]

CFFI Manual:

http://common-lisp.net/project/cffi/manual/html_node/index.html

I think that adopting a perfectly reasonable, well documented standard is preferable to developing yet another FFI API.

--
Burton Samograd

This e-mail, including accompanying communications and attachments, is strictly confidential and only for the intended recipient. Any retention, use or disclosure not expressly authorised by Markit is prohibited. This email is subject to all waivers and other terms at the following link: http://www.markit.com/en/about/legal/email-disclaimer.page

Please visit http://www.markit.com/en/about/contact/contact-us.page? for contact information on our offices worldwide.

Re: FFI in Emacs

[Eli Zaretskii]

> From: "John Wiegley" <johnw@newartisans.com>
> Date: Wed, 11 Jul 2012 00:24:27 -0500
> Cc: emacs-devel@gnu.org
> 
> A real FFI patch would let me dlopen any C library and call functions within
> it, such as openssl or zlib.

What for?

Re: FFI in Emacs

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Date: Wed, 11 Jul 2012 16:14:56 +0900
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> 
> [SXEmacs FFI interface] should be fairly easily portable to Emacs on
> platforms that have libffi.  It probably doesn't work on Windows

Why not?  Not because of libffi, I think.  libffi compiles out of the
box with MinGW; I did that when I needed to build Guile on Windows,
and my notes from that time indicate that the only libffi issue was a
rather minor problem with "make install".  And there are precompiled
Windows binaries available on the net.

Re: FFI in Emacs

[Samuel Bronson]

On Jul 11, 2012, at 12:11 PM, Eli Zaretskii wrote:

>> From: "Stephen J. Turnbull" <stephen@xemacs.org>
>> Date: Wed, 11 Jul 2012 16:14:56 +0900
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
>>
>> [SXEmacs FFI interface] should be fairly easily portable to Emacs on
>> platforms that have libffi.  It probably doesn't work on Windows
>
> Why not?  Not because of libffi, I think.  libffi compiles out of the
> box with MinGW; I did that when I needed to build Guile on Windows,
> and my notes from that time indicate that the only libffi issue was a
> rather minor problem with "make install".  And there are precompiled
> Windows binaries available on the net.


It's wise to assume that code that hasn't been tried on Windows  
recently might need at least some porting, especially when it has  
anything to do with dynamic loading or calling conventions.  Despite  
all the things the NT hackers picked up from *nix, NT does many things  
rather differently, including what they call DLLs, and the APIs for  
loading them dynamically.  (Plus, it turns out libffi has nothing to  
do with loading; evidently, it only does calling.)

Re: FFI in Emacs

[Eli Zaretskii]

> Cc: "Stephen J. Turnbull" <stephen@xemacs.org>,
>  johnw@newartisans.com,
>  monnier@iro.umontreal.ca,
>  emacs-devel@gnu.org
> From: Samuel Bronson <naesten@gmail.com>
> Date: Wed, 11 Jul 2012 13:08:33 -0400
> 
> 
> On Jul 11, 2012, at 12:11 PM, Eli Zaretskii wrote:
> 
> >> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> >> Date: Wed, 11 Jul 2012 16:14:56 +0900
> >> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> >>
> >> [SXEmacs FFI interface] should be fairly easily portable to Emacs on
> >> platforms that have libffi.  It probably doesn't work on Windows
> >
> > Why not?  Not because of libffi, I think.  libffi compiles out of the
> > box with MinGW; I did that when I needed to build Guile on Windows,
> > and my notes from that time indicate that the only libffi issue was a
> > rather minor problem with "make install".  And there are precompiled
> > Windows binaries available on the net.
> 
> 
> It's wise to assume that code that hasn't been tried on Windows  
> recently might need at least some porting, especially when it has  
> anything to do with dynamic loading or calling conventions.

I see no need to assume anything when there are facts available.  I
revealed the facts known to me so that people won't need to assume.

> Despite  
> all the things the NT hackers picked up from *nix, NT does many things  
> rather differently, including what they call DLLs, and the APIs for  
> loading them dynamically.

AFAIK, libltdl is used for actually loading the modules.  libltdl also
works on Windows, with binaries readily available.

So I really don't know what you are talking about.

Re: FFI in Emacs

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > > >> [SXEmacs FFI interface] should be fairly easily portable to Emacs on
 > > >> platforms that have libffi.  It probably doesn't work on Windows
 > > >
 > > > Why not?  Not because of libffi, I think.

Because there is no support for Windows in SXEmacs at all, and last I
looked XEmacsen do not use the same technology for loading DLLs on
Windows that they do on Unix, so there is no provision whatsoever in
the SXEmacs code for actually loading the C code that the Lisp FFI API
allows you to call.

Come to think of it, I am assuming a "native" build with VC; maybe it
would work with the "Unix" code on Cygwin/Mingwin.

 > I see no need to assume anything when there are facts available.  I
 > revealed the facts known to me so that people won't need to assume.

I made no assumptions about how hard it is to build libraries on
Windows, only that as far as I know half the job has never been coded
for Windows, so AFAIK until it is done the SXEmacs code won't work.

Re: FFI in Emacs

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Samuel Bronson <naesten@gmail.com>,
>     johnw@newartisans.com,
>     monnier@iro.umontreal.ca,
>     emacs-devel@gnu.org
> Date: Thu, 12 Jul 2012 03:23:46 +0900
> 
> Eli Zaretskii writes:
> 
>  > > >> [SXEmacs FFI interface] should be fairly easily portable to Emacs on
>  > > >> platforms that have libffi.  It probably doesn't work on Windows
>  > > >
>  > > > Why not?  Not because of libffi, I think.
> 
> Because there is no support for Windows in SXEmacs at all, and last I
> looked XEmacsen do not use the same technology for loading DLLs on
> Windows that they do on Unix, so there is no provision whatsoever in
> the SXEmacs code for actually loading the C code that the Lisp FFI API
> allows you to call.

Well, if they don't use libltdl or something similar for loading
(which is a natural choice when using libffi, but YMMV), then I guess
you are right, it might not work.

> Come to think of it, I am assuming a "native" build with VC; maybe it
> would work with the "Unix" code on Cygwin/Mingwin.

MinGW uses the same APIs as VC, so I doubt that.  Cygwin is another
matter, of course.

Re: FFI in Emacs

[Stefan Monnier]

> A real FFI patch would let me dlopen any C library and call functions within
> it, such as openssl or zlib.  Dave's patch is more like what Python allows:
> writing Emacs Lisp extensions in C that can be loaded as shared libraries.

That sounds right, thank you.

Just for the record: Emacs's FFI should not be able to open just any
random shared library.  Only the ones that come with the needed extra
"I'm compatible with the GPL" branding (as attested by the
presence of the "plugin_is_GPL_compatible" symbol, IIRC).

Hopefully, more and more libraries will come with such branding in
the future.


        Stefan

Re: FFI in Emacs

[Samuel Bronson]

On Jul 12, 2012, at 9:55 AM, Stefan Monnier wrote:

>> A real FFI patch would let me dlopen any C library and call  
>> functions within
>> it, such as openssl or zlib.  Dave's patch is more like what Python  
>> allows:
>> writing Emacs Lisp extensions in C that can be loaded as shared  
>> libraries.
>
> That sounds right, thank you.
>
> Just for the record: Emacs's FFI should not be able to open just any
> random shared library.  Only the ones that come with the needed extra
> "I'm compatible with the GPL" branding (as attested by the
> presence of the "plugin_is_GPL_compatible" symbol, IIRC).
>
> Hopefully, more and more libraries will come with such branding in
> the future.

What is the purpose of such a symbol?  It sounds suspiciously similar  
to <http://segaretro.org/TradeMark_Security_System>...

Re: FFI in Emacs

[John Wiegley]

>>>>> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> Just for the record: Emacs's FFI should not be able to open just any random
> shared library.  Only the ones that come with the needed extra "I'm
> compatible with the GPL" branding (as attested by the presence of the
> "plugin_is_GPL_compatible" symbol, IIRC).

Understood.

John

Re: FFI in Emacs

[Sam Steingold]

> * Stefan Monnier <zbaavre@VEB.HZbagerny.PN> [2012-07-12 09:55:14 -0400]:
>
> Just for the record: Emacs's FFI should not be able to open just any
> random shared library.  Only the ones that come with the needed extra
> "I'm compatible with the GPL" branding (as attested by the
> presence of the "plugin_is_GPL_compatible" symbol, IIRC).

I can load any elisp into my emacs.
Why shouldn't I be able to interact with any shared library?

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 12.04 (precise) X 11.0.11103000
http://www.childpsy.net/ http://think-israel.org http://camera.org
http://memri.org http://jihadwatch.org http://dhimmi.com http://www.memritv.org
Kleptomania: the ability to find stuff even before its owner loses it.

Re: FFI in Emacs

[Óscar Fuentes]

Sam Steingold <sds@gnu.org> writes:

> I can load any elisp into my emacs.
> Why shouldn't I be able to interact with any shared library?

A more correct analogy is: "I can load any elisp *bytecode* in my
emacs." ...

Re: FFI in Emacs

[Stephen J. Turnbull]

Óscar Fuentes writes:
 > Sam Steingold <sds@gnu.org> writes:
 > 
 > > I can load any elisp into my emacs.
 > > Why shouldn't I be able to interact with any shared library?
 > 
 > A more correct analogy is: "I can load any elisp *bytecode* in my
 > emacs." ...

No, Sam S's analogy is as correct as it gets.  The technical issues of
obfuscation and maintainability of the DLL aren't relevant here.

Sam B's point also is as correct as it gets: technologically, this is
the same as a certain license enforcement device.

In both cases, the answer to the question of "why can't I" is:

    *You* may, distributors *must not*.

And *you* *can*, too.  It's a simple matter of changing your Emacs --
but it would be considered a disservice to the community for you to
distribute that Emacs.  (I don't see how it would be a license
violation, though FSF legal may have some reasoning to that effect.)

Note that Linux does something similar for modules that use "internal"
APIs.  The motivation and so the specification is somewhat different
(there's a command-line option to disable the check), but it's not
like this isn't an OSS-useful, as well as FS-useful, feature.

Re: FFI in Emacs

[Aurélien Aptel]

Hi all,

Joakim, thank you for your work! Did you make any progress since then?
I was wondering, is anyone else working on this?

The patch mentioned earlier to load dynamically compiled modules is
not a FFI but it can be useful nonetheless. Why not have both?

Re: FFI in Emacs

[joakim]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> Hi all,
>
> Joakim, thank you for your work! Did you make any progress since then?
> I was wondering, is anyone else working on this?
>
> The patch mentioned earlier to load dynamically compiled modules is
> not a FFI but it can be useful nonetheless. Why not have both?

I commited the GIR patch to the xwidget emacs repo. You can already do
interesting 
ings with it:

- create gnome objects dynamically(previously only hardcoded objects
were possible)

- dynamically call their methods, with an (early) lisp to C object
marshaller

Anyway, a FFI needs type information from somewhere, and with GIR you
get that from 
e library provider. Therefore GIR is IMHO more interesting than  plain
FFI. You can also provide GIR typelib wrappers that would benefit all
GIR implementations, not only the xwgir one in the xwidget branch.

Well maybe that wasnt so clear. Hear is a concrete example:
- you want to call the libmagic file info functions
- wrap libmagic in a tiny wrapper that provides gir typeinfo
- now you can call these libmagic functions from Emacs, and the xwgir
functions can figure out how to convert elisp data to the data needed by
the library.
- but now your friends using gir from python etc can also benefit your
work!

I think this is the right aproach, but it does not exclude a plain FFI
implementation.

--
Joakim Verona

Re: FFI in Emacs

[Ivan Kanis]

joakim@verona.se wrote:

> I commited the GIR patch to the xwidget emacs repo. You can already do
> interesting ings with it:

What's a GIR?
-- 
Ivan Kanis
http://ivan.kanis.fr

A diplomat is a person who can tell you to go to hell in such a way
that you actually look forward to the trip.
    -- Caskie Stinett

I am listening to "Plan B - She Said".

Re: FFI in Emacs

[joakim]

Ivan Kanis <ivan.kanis@googlemail.com> writes:

> joakim@verona.se wrote:
>
>> I commited the GIR patch to the xwidget emacs repo. You can already do
>> interesting ings with it:
>
> What's a GIR?

Gnome Introspection.

https://live.gnome.org/GObjectIntrospection/

Its basically like Corba, Com, Java introspection, or whatever.

Except its better imho, because its pretty lightweight, not obsessed with
network marshalling like corba, not tied to a particular plattform like
com, and not tied to a particular language like java introspection. (you
can poke holes in my statements, but its more or less true). Also GIR is
a Gnome project which is a Gnu project, so a good fit for Emacs, no?

-- 
Joakim Verona

Re: FFI in Emacs

[Aurélien Aptel]

Hi all,

I've been doing some research in the last few weeks since I wasn't
familiar with all the details that goes into implementing a FFI.

First, you need to be able make arbitrary call to any function in a
shared library that your OS supports (.so, .dll, etc), so:

a) load the shared library dynamically
b) get the address of any symbol in it (be it a global var or a function).

Then, to call a function:
c) load the argument according to user-provided type and ABI
d) make the call
e) retrieve eventual return value (also depends on ABI)

Steps a and b can be done portably using libltdl [1]; c,d and e with
libffi [2] or libffcall [3].

So far so good. Once you have that, you have to expose the feature to
Elisp interpreter. You have to take into account that:
a) a function can allocate memory that has to be freed
b) a function can write in its arguments e.g. a buffer address and its
size (like read(2))
c) Lisp_Object have to be converted to native type (and vice versa)
d) a type can also be structs or unions

Still do-able but as you can see it gets tricky. Also:
- native int type can't be stored as a Lisp int because of the type
bits (problem c)
- any mistake using the FFI can lead to a crash (it is C after all)
- performance cost due to the dynamic call and all the conversion

Finally, we have to agree on:
a) the API
b) making it Stallman-compatible (GPL issues, etc) :-)

I'm sure everyone agree that we should implement an existing API, but which one?
Are we sure Stefan's gpl_compatible symbol solution sufficient for problem b?

Writing a complete wrapper for a slightly complex library can be
tedious and probably easier using directly C. I think emacs needs both
a FFI and a way to load dynamically compiled modules.
Although it can be redundant, I see it as complementary. The FFI can
be simple and not handle complex cases, use it if you need to call a
small set of simple functions, one-off jobs, etc. Write a module for
the rest and use tools like SWIG [4] made to generate binding/wrapper
for a library if you think it fits the job.

Dave Love already wrote a patch to write/load compiled modules for
Emacs 22 [5] which I have not been able to apply (or test, obviously)
on the official Emacs 22 source release. I've tried to fix it but I'm
stuck in autoconf hell which I'm not very familiar with. If someone
managed to try it, please post the steps you used. Anyway it can be a
good starting point.

I'm interested in working seriously on this, possibly for the next
Google Summer of Code.

I'm waiting for your suggestions and comments.

1: http://www.gnu.org/software/libtool/manual/html_node/Using-libltdl.html
2: http://sourceware.org/libffi/
3: http://www.gnu.org/software/libffcall/
4: http://www.swig.org/
5: http://www.loveshack.ukfsn.org/emacs/dynamic-loading/

Re: FFI in Emacs

[Eli Zaretskii]

> Date: Tue, 12 Mar 2013 20:16:42 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: emacs-devel@gnu.org
> 
> a) load the shared library dynamically
> b) get the address of any symbol in it (be it a global var or a function).
> 
> Then, to call a function:
> c) load the argument according to user-provided type and ABI
> d) make the call
> e) retrieve eventual return value (also depends on ABI)
> 
> Steps a and b can be done portably using libltdl [1]; c,d and e with
> libffi [2] or libffcall [3].

Do we really need these libraries?  If the Posix hosts can do with
dlopen, dlsym, dlclose, and dlerror, then it's very easy to emulate
that on platforms that don't have these in the system libraries.  What
else is needed, and why?

> a) a function can allocate memory that has to be freed

At least on Windows, this cannot be done safely, so please don't
design the interface based on the assumption this is doable.  If a
shared object allocates memory, it should be responsible for freeing
it, or provide an API for telling it to free it.

Likewise with file descriptors -- they cannot be safely shared across
the interface, because the shared library could have used a different
runtime for opening files.

Re: FFI in Emacs

[Pascal J. Bourguignon]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Tue, 12 Mar 2013 20:16:42 +0100
>> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
>> Cc: emacs-devel@gnu.org
>> 
>> a) load the shared library dynamically
>> b) get the address of any symbol in it (be it a global var or a function).
>> 
>> Then, to call a function:
>> c) load the argument according to user-provided type and ABI
>> d) make the call
>> e) retrieve eventual return value (also depends on ABI)
>> 
>> Steps a and b can be done portably using libltdl [1]; c,d and e with
>> libffi [2] or libffcall [3].
>
> Do we really need these libraries?  If the Posix hosts can do with
> dlopen, dlsym, dlclose, and dlerror, then it's very easy to emulate
> that on platforms that don't have these in the system libraries.  What
> else is needed, and why?
>
>> a) a function can allocate memory that has to be freed
>
> At least on Windows, this cannot be done safely, so please don't
> design the interface based on the assumption this is doable.  If a
> shared object allocates memory, it should be responsible for freeing
> it, or provide an API for telling it to free it.
>
> Likewise with file descriptors -- they cannot be safely shared across
> the interface, because the shared library could have used a different
> runtime for opening files.

Is emacs garbage collector of the copying kind?  If so, then objects
move in memory during garbage collection, and therefore any reference
given to a foreign library become invalid once the garbage collector
runs.  And vice-versa, managing resources allocated in the library (even if
we use library calls to do so) is quite complex, when those resources
have to be fred when the lisp object that has a reference is collected.
(Do we have finalizer in emacs objects?)

Soon the easiest solution is to copy a lot of data, between the lisp and
C side, which is even more costly.

A lot of libraries are written in C++ too, and they're a bitch to write
a FFI for, in general (ie. as soon as they use templates, smartptr<T>
anybody?), not counting the compiler-dependent name mangling which is
relatively a simple problem to solve.


I mean, even if emacs FFI was perfect (the FFI of some CL
implementations are good enough), you would still have those two
problems, of dealing with C++ libraries, and of crashing into C code,
which makes me choose often to just rewrite the thing in lisp, or to
write an external wrapper program in C, communicating with the lisp
process with whatever IPC is practical (files, sockets, etc).

Also, I guess a lot of modern libraries don't need anything fancy from
C/C++ either, so perhaps the best solution would be to have actually a
C/C++ compiler targetting the emacs VM, and instead of doing FFI to an
external binary library, we'd just compile it into the emacs VM.  But
perhaps writing a whole new compiler is more work than hacking half a
FFI.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

Re: FFI in Emacs

[Aurélien Aptel]

On Tue, Mar 12, 2013 at 10:50 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> Do we really need these libraries?  If the Posix hosts can do with
> dlopen, dlsym, dlclose, and dlerror, then it's very easy to emulate
> that on platforms that don't have these in the system libraries.  What
> else is needed, and why?

I agree that libltdl is small enough to be re-implemented but you
didn't say why you don't want to use it.
We could emulate it ourselves but libltdl is a small well maintained
library that already works on various systems.
This is even more true for libffi/fficall which handles architecture
specific stuff with raw assembler. Are the extra dependencies
bothering you?

>> a) a function can allocate memory that has to be freed
>
> At least on Windows, this cannot be done safely, so please don't
> design the interface based on the assumption this is doable.  If a
> shared object allocates memory, it should be responsible for freeing
> it, or provide an API for telling it to free it.

I'm not sure I understand. You're saying that freeing a pointer
returned by a library function which explicitly says you have to free
it yourself is not safe on Windows? That seems strange.
Anyway this is not a fundamental part of the interface design, it's
just something that can happen to keep in mind.

> Likewise with file descriptors -- they cannot be safely shared across
> the interface, because the shared library could have used a different
> runtime for opening files.

If you're talking about read(), it was just an example of a
problematic API (side effects). If you're writing some code to
interface with C I expect you to know what you're doing.

Re: FFI in Emacs

[Daniel Colascione]

[-- Attachment #1: Type: text/plain, Size: 2470 bytes --]

On 3/12/2013 3:53 PM, Aurélien Aptel wrote:
>>> a) a function can allocate memory that has to be freed
>>
>> At least on Windows, this cannot be done safely, so please don't
>> design the interface based on the assumption this is doable.  If a
>> shared object allocates memory, it should be responsible for freeing
>> it, or provide an API for telling it to free it.
> 
> I'm not sure I understand. You're saying that freeing a pointer
> returned by a library function which explicitly says you have to free
> it yourself is not safe on Windows? That seems strange.

Free using which heap? It's common for different modules in a Windows process to
use entirely different C runtime libraries. On Windows, all inter-module
dependencies are (module, symbol) pairs (written "module!symbol"), not just bare
symbol names as on ELF systems. That is, modA!fun1 calling modB!fun2 can exist
in the same process as a modC!fun1 calling modD!fun2. One module's malloc isn't
necessarily (and often isn't) the same as another module's malloc. It's also
common for Windows programs to allocate memory off private heaps, although
that's discouraged these days for performance reasons.

There's nothing stopping a POSIXish program from using multiple heaps either,
but due to tradition and the way ELF dynamically binds symbols, you're likely to
see most parts of a typical more programs tend to just use the system libc. It's
good practice, though, on both Windows and POSIXish systems, to provide a
function resource-specific (or at least module-specific) deallocation functions.
It's just that if you don't follow this advice under Windows, heap corruption
shoots you in the face.

(Incidentally, one of the challenges of writing Python loadable modules for
Windows is that Python is written with the assumption that all parts of the
program share the same libc, so if foo.pyd isn't linked to precisely the same
version of the C runtime that python27.dll is linked to, things go wrong very
quickly.)

Sometimes Windows library specify that memory is allocated on the "process
heap". The process heap is provided by ntdll.dll and functions more like a
Posixish system's libc heap in that it's shared by all parts of a process; see
HeapAlloc and HeapFree. Likewise, you can't share file descriptors or FILE*
pointers between different modules under Windows, but you *can* share HANDLEs,
since HANDLEs are universal and managed by the kernel.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: FFI in Emacs

[Eli Zaretskii]

> Date: Tue, 12 Mar 2013 23:53:28 +0100
> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
> Cc: joakim@verona.se, emacs-devel@gnu.org
> 
> On Tue, Mar 12, 2013 at 10:50 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > Do we really need these libraries?  If the Posix hosts can do with
> > dlopen, dlsym, dlclose, and dlerror, then it's very easy to emulate
> > that on platforms that don't have these in the system libraries.  What
> > else is needed, and why?
> 
> I agree that libltdl is small enough to be re-implemented but you
> didn't say why you don't want to use it.

Because each additional external library required to build Emacs is
more nuisance and complexity for end users, who need to find a
compatible version, install it, and know how to tell the compiler to
find it.

> >> a) a function can allocate memory that has to be freed
> >
> > At least on Windows, this cannot be done safely, so please don't
> > design the interface based on the assumption this is doable.  If a
> > shared object allocates memory, it should be responsible for freeing
> > it, or provide an API for telling it to free it.
> 
> I'm not sure I understand. You're saying that freeing a pointer
> returned by a library function which explicitly says you have to free
> it yourself is not safe on Windows? That seems strange.

Daniel explained why it isn't strange.  Emacs on Windows does define a
separate heap, to emulate the Posix sbrk and its behavior, see
w32heap.c.

> > Likewise with file descriptors -- they cannot be safely shared across
> > the interface, because the shared library could have used a different
> > runtime for opening files.
> 
> If you're talking about read(), it was just an example of a
> problematic API (side effects). If you're writing some code to
> interface with C I expect you to know what you're doing.

Again, Daniel explained that any function that opens a file and any
FILE object cannot be passed safely.

Re: FFI in Emacs

[Andreas Schwab]

"Pascal J. Bourguignon" <pjb@informatimago.com> writes:

> Is emacs garbage collector of the copying kind?

No, it is mark-and-sweep.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: FFI in Emacs

[BT Templeton]

Aurélien Aptel <aurelien.aptel+emacs@gmail.com> writes:

> Hi all,
>
> I've been doing some research in the last few weeks since I wasn't
> familiar with all the details that goes into implementing a FFI.
[...]

Have you considered using Guile-Emacs as the basis for this? You could
then implement an FFI as an Elisp-level library providing access to
Guile's FFI modules. (It might require some improvements to the C-level
guile integration, and perhaps adding GPL-compatibility-checking to
Guile's FFI, but would still be easier than doing everything in C.)

-- 
Inteligenta persono lernas la lingvon Esperanton rapide kaj facile.
Esperanto estas moderna, kultura lingvo por la mondo. Simpla, fleksebla,
belsona, Esperanto estas la praktika solvo de la problemo de universala
interkompreno. Lernu la interlingvon Esperanton!

Re: FFI in Emacs

[Leo Liu]

On 2013-03-23 03:44 +0800, BT Templeton wrote:
> Have you considered using Guile-Emacs as the basis for this?

Sounds exciting. Is there a plan for guile-emacs to be emacs?

Leo

Re: FFI in Emacs

[Noah Lavine]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

Hello,

On Fri, Mar 22, 2013 at 9:51 PM, Leo Liu <sdl.web@gmail.com> wrote:

> On 2013-03-23 03:44 +0800, BT Templeton wrote:
> > Have you considered using Guile-Emacs as the basis for this?
>
> Sounds exciting. Is there a plan for guile-emacs to be emacs?
>

I spend my time on Guile and not Emacs, but I believe the answer is, "in
theory, maybe, at some point. But no concrete plan." That's been the answer
for at least ten years.

However, there has been more progress in the last two years than there has
in a long time, thanks to BT Templeton (and others). I believe there is
even a repository containing a branch of Emacs that uses Guile as its
execution engine. Maybe BT could tell you the status of that branch.

Best,
Noah

[-- Attachment #2: Type: text/html, Size: 1144 bytes --]

Re: FFI in Emacs

[Stefan Monnier]

>> I've been doing some research in the last few weeks since I wasn't
>> familiar with all the details that goes into implementing a FFI.
> [...]
> Have you considered using Guile-Emacs as the basis for this?

Could you give us a description of what is Guile-Emacs nowadays?


        Stefan

Re: FFI in Emacs

[Yagnesh Raghava Yakkala]

Hello Stefan,

On Mar 24 2013, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Could you give us a description of what is Guile-Emacs nowadays?

Sorry for jumping in,

I was kind of  excited to see this message on guile mailing list.

╭─────┤ http://article.gmane.org/gmane.lisp.guile.user/10198 ├─────
│I don't know if lua-zile is better than guile-emacs, but I'm using
│current unstable guile-emacs smoothly and all the plugins works for me,
│though crash sometime.
╰─────





Thanks.,
-- 
ఎందరో మహానుభావులు అందరికి వందనములు.
YYR