DSO-style FFI (was: Emacs crypto use cases)

[Stefan Monnier <monnier@iro.umontreal.ca>]

[ Hi Jerry.  We're interested in adding support for dynamically loading
  C code into Emacs, so I added you on the Cc, hoping you might be able
  to give us some pointers to the XEmacs code, and maybe also some
  guidance of things to watch out for.  ]

> I have no idea how that would work.  Is there an example?

As mentioned, there's the XEmacs implementation as an example.

> And is this anywhere close to usable?

Well, it doesn't exist for Emacs, so you won't be able to use until you
write it.  AFAIK it's been usable for a decade in XEmacs.

> The FFI discussion thread hasn't indicated it.

There's no doubt it would work and it should be fairly flexible and
convenient for the implementer of lib<foo> wrapper.  The main
inconvenients are in the area of imposing the use of a C compiler on the
user, and managing precompiled versions of the code, none of which seem
like reasons to make it "unusable".

> If I'm to be the guinea pig, I'm not excited about it.

No, you'd be the implementor, so you get to make it work well for you.
The guy that everyone after that will say "and finally, thanks to Ted,
we can now link to the new snazzy libquantum and solve all the Elisp
performance problems we had".

> I'll see what I can do with FFI, but this is likely to take a lot of my
> time.  Some help getting started would go a long way.

Start as follows:
- take your lib<foo> wrapper.
- try to compile it outside of Emacs.
- for that you'll need a .h file that declares some functions like
  `defsubr' and probably others to build and analyze Lisp data.
  Ideally, this should not really too much on cpp macros and should not
  make too many assumptions about the existing Emacs implementation, so
  that this .h file doesn't need to change if/when we decide to tweak
  the representation of Elisp data.
- write the .c file that should be linked into Emacs that provides the
  functions declared in the magic .h file.

Now you should be able to build your "Emacs + lib<foo>" by simply
compiling your file and linking it into Emacs (i.e. pretty much like
what you have, except your lib-wrapper doesn't include lisp.h and such
but only the special new .h file you created).

That's step 1.  It should be reasonably easy to do in terms of coding.
The design of the .h file is somewhat delicate, to try and avoid version
problems in the future, but we'll be happy to help along once you have
a first version of a .h.

Step 2: Write code that lets Emacs dynamically load the (compiled)
lib-wrapper code into a running Emacs, rather than having to provide
that lib-wrapper right when you link the `temacs' executable.
That shouldn't be hard either.  One issue will be compatibility of
various kinds of dl_open mechanisms on various OSes, but you should find
lots of help for that here, so the first version can focus on supporting
only GNU/Linux.  The other issue will be to make sure the loaded code is
GPL-compliant; we'll need help here anyway to know exactly what the FSF
wants in this regard but I don't foresee any serious issue.

At that point, you can start distributing your libnettle wrapper outside
of Emacs.  Yay!

Step 3: provide helper infrastructure to distribute packages that use
this new functionality, e.g. teach package.el how to run the C compiler
on that code, stash the resulting compiled object somewhere, fetch
precompiled versions when a C compiler is not available, etc...

As you move forward, chances are more and more people will be motivated
to take on some of the work, so you don't have to do it all yourself.


        Stefan