Re: FFI in Emacs

["Pascal J. Bourguignon" <pjb@informatimago.com>]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Tue, 12 Mar 2013 20:16:42 +0100
>> From: Aurélien Aptel <aurelien.aptel+emacs@gmail.com>
>> Cc: emacs-devel@gnu.org
>> 
>> a) load the shared library dynamically
>> b) get the address of any symbol in it (be it a global var or a function).
>> 
>> Then, to call a function:
>> c) load the argument according to user-provided type and ABI
>> d) make the call
>> e) retrieve eventual return value (also depends on ABI)
>> 
>> Steps a and b can be done portably using libltdl [1]; c,d and e with
>> libffi [2] or libffcall [3].
>
> Do we really need these libraries?  If the Posix hosts can do with
> dlopen, dlsym, dlclose, and dlerror, then it's very easy to emulate
> that on platforms that don't have these in the system libraries.  What
> else is needed, and why?
>
>> a) a function can allocate memory that has to be freed
>
> At least on Windows, this cannot be done safely, so please don't
> design the interface based on the assumption this is doable.  If a
> shared object allocates memory, it should be responsible for freeing
> it, or provide an API for telling it to free it.
>
> Likewise with file descriptors -- they cannot be safely shared across
> the interface, because the shared library could have used a different
> runtime for opening files.

Is emacs garbage collector of the copying kind?  If so, then objects
move in memory during garbage collection, and therefore any reference
given to a foreign library become invalid once the garbage collector
runs.  And vice-versa, managing resources allocated in the library (even if
we use library calls to do so) is quite complex, when those resources
have to be fred when the lisp object that has a reference is collected.
(Do we have finalizer in emacs objects?)

Soon the easiest solution is to copy a lot of data, between the lisp and
C side, which is even more costly.

A lot of libraries are written in C++ too, and they're a bitch to write
a FFI for, in general (ie. as soon as they use templates, smartptr<T>
anybody?), not counting the compiler-dependent name mangling which is
relatively a simple problem to solve.


I mean, even if emacs FFI was perfect (the FFI of some CL
implementations are good enough), you would still have those two
problems, of dealing with C++ libraries, and of crashing into C code,
which makes me choose often to just rewrite the thing in lisp, or to
write an external wrapper program in C, communicating with the lisp
process with whatever IPC is practical (files, sockets, etc).

Also, I guess a lot of modern libraries don't need anything fancy from
C/C++ either, so perhaps the best solution would be to have actually a
C/C++ compiler targetting the emacs VM, and instead of doing FFI to an
external binary library, we'd just compile it into the emacs VM.  But
perhaps writing a whole new compiler is more work than hacking half a
FFI.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.