From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Stefan Monnier Newsgroups: gmane.emacs.devel Subject: DSO-style FFI (was: Emacs crypto use cases) Date: Tue, 08 Oct 2013 12:35:44 -0400 Message-ID: References: <877gdqrc9u.fsf@flea.lifelogs.com> <87mwmmp05f.fsf@flea.lifelogs.com> <87fvsdpato.fsf@flea.lifelogs.com> <8738oc20xk.fsf@flea.lifelogs.com> <87d2ngzlyl.fsf_-_@flea.lifelogs.com> <87siwcxda7.fsf@flea.lifelogs.com> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: ger.gmane.org 1381250166 24566 80.91.229.3 (8 Oct 2013 16:36:06 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 8 Oct 2013 16:36:06 +0000 (UTC) Cc: jerry.james@xemacs.org To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Tue Oct 08 18:36:09 2013 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1VTaGW-0007ue-OC for ged-emacs-devel@m.gmane.org; Tue, 08 Oct 2013 18:36:08 +0200 Original-Received: from localhost ([::1]:37875 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VTaGW-0005sc-BR for ged-emacs-devel@m.gmane.org; Tue, 08 Oct 2013 12:36:08 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:43851) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VTaGL-0005jh-J6 for emacs-devel@gnu.org; Tue, 08 Oct 2013 12:36:04 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VTaGA-0007ye-Ph for emacs-devel@gnu.org; Tue, 08 Oct 2013 12:35:57 -0400 Original-Received: from ironport2-out.teksavvy.com ([206.248.154.182]:62850) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VTaGA-0007yX-LD for emacs-devel@gnu.org; Tue, 08 Oct 2013 12:35:46 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av4EABK/CFHO+K8t/2dsb2JhbABEvw4Xc4IeAQEEAVYjBQs/EhQYDYhCBrEfkA6JT4c7A6R6gV6DEw X-IPAS-Result: Av4EABK/CFHO+K8t/2dsb2JhbABEvw4Xc4IeAQEEAVYjBQs/EhQYDYhCBrEfkA6JT4c7A6R6gV6DEw X-IronPort-AV: E=Sophos;i="4.84,565,1355115600"; d="scan'208";a="35029623" Original-Received: from 206-248-175-45.dsl.teksavvy.com (HELO pastel.home) ([206.248.175.45]) by ironport2-out.teksavvy.com with ESMTP/TLS/ADH-AES256-SHA; 08 Oct 2013 12:32:03 -0400 Original-Received: by pastel.home (Postfix, from userid 20848) id DB67F6028B; Tue, 8 Oct 2013 12:35:44 -0400 (EDT) In-Reply-To: <87siwcxda7.fsf@flea.lifelogs.com> (Ted Zlatanov's message of "Tue, 08 Oct 2013 06:33:36 -0400") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.3.50 (gnu/linux) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 206.248.154.182 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:164008 Archived-At: [ Hi Jerry. We're interested in adding support for dynamically loading C code into Emacs, so I added you on the Cc, hoping you might be able to give us some pointers to the XEmacs code, and maybe also some guidance of things to watch out for. ] > I have no idea how that would work. Is there an example? As mentioned, there's the XEmacs implementation as an example. > And is this anywhere close to usable? Well, it doesn't exist for Emacs, so you won't be able to use until you write it. AFAIK it's been usable for a decade in XEmacs. > The FFI discussion thread hasn't indicated it. There's no doubt it would work and it should be fairly flexible and convenient for the implementer of lib wrapper. The main inconvenients are in the area of imposing the use of a C compiler on the user, and managing precompiled versions of the code, none of which seem like reasons to make it "unusable". > If I'm to be the guinea pig, I'm not excited about it. No, you'd be the implementor, so you get to make it work well for you. The guy that everyone after that will say "and finally, thanks to Ted, we can now link to the new snazzy libquantum and solve all the Elisp performance problems we had". > I'll see what I can do with FFI, but this is likely to take a lot of my > time. Some help getting started would go a long way. Start as follows: - take your lib wrapper. - try to compile it outside of Emacs. - for that you'll need a .h file that declares some functions like `defsubr' and probably others to build and analyze Lisp data. Ideally, this should not really too much on cpp macros and should not make too many assumptions about the existing Emacs implementation, so that this .h file doesn't need to change if/when we decide to tweak the representation of Elisp data. - write the .c file that should be linked into Emacs that provides the functions declared in the magic .h file. Now you should be able to build your "Emacs + lib" by simply compiling your file and linking it into Emacs (i.e. pretty much like what you have, except your lib-wrapper doesn't include lisp.h and such but only the special new .h file you created). That's step 1. It should be reasonably easy to do in terms of coding. The design of the .h file is somewhat delicate, to try and avoid version problems in the future, but we'll be happy to help along once you have a first version of a .h. Step 2: Write code that lets Emacs dynamically load the (compiled) lib-wrapper code into a running Emacs, rather than having to provide that lib-wrapper right when you link the `temacs' executable. That shouldn't be hard either. One issue will be compatibility of various kinds of dl_open mechanisms on various OSes, but you should find lots of help for that here, so the first version can focus on supporting only GNU/Linux. The other issue will be to make sure the loaded code is GPL-compliant; we'll need help here anyway to know exactly what the FSF wants in this regard but I don't foresee any serious issue. At that point, you can start distributing your libnettle wrapper outside of Emacs. Yay! Step 3: provide helper infrastructure to distribute packages that use this new functionality, e.g. teach package.el how to run the C compiler on that code, stash the resulting compiled object somewhere, fetch precompiled versions when a C compiler is not available, etc... As you move forward, chances are more and more people will be motivated to take on some of the work, so you don't have to do it all yourself. Stefan