From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: =?UTF-8?Q?Aur=C3=A9lien_Aptel?= Newsgroups: gmane.emacs.devel Subject: Re: Dynamic loading progress Date: Thu, 9 Oct 2014 19:08:47 +0200 Message-ID: References: <87y4t9m76w.fsf@lifelogs.com> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Trace: ger.gmane.org 1412874543 12670 80.91.229.3 (9 Oct 2014 17:09:03 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Thu, 9 Oct 2014 17:09:03 +0000 (UTC) To: Emacs development discussions Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Thu Oct 09 19:08:59 2014 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1XcHCy-0003ea-5e for ged-emacs-devel@m.gmane.org; Thu, 09 Oct 2014 19:08:56 +0200 Original-Received: from localhost ([::1]:44149 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XcHCx-0001lV-HO for ged-emacs-devel@m.gmane.org; Thu, 09 Oct 2014 13:08:55 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:38930) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XcHCs-0001lO-U5 for emacs-devel@gnu.org; Thu, 09 Oct 2014 13:08:52 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1XcHCr-000614-80 for emacs-devel@gnu.org; Thu, 09 Oct 2014 13:08:50 -0400 Original-Received: from mail-la0-x233.google.com ([2a00:1450:4010:c03::233]:46489) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XcHCq-00060c-Qf for emacs-devel@gnu.org; Thu, 09 Oct 2014 13:08:49 -0400 Original-Received: by mail-la0-f51.google.com with SMTP id ge10so1626747lab.24 for ; Thu, 09 Oct 2014 10:08:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=p15b8Hj4n9jiuODmZUzCu1e3l3nX11bvUpAO4GJL+1M=; b=FSH8IHAEZb2KH7JKoUnKOv1vjCJw6UlvyKSyQdnQS2Zmn2NprM/rgPjKl+dWGrrfLD 3P9ZMddw7UjoKM9HhkfCamhNxsOBAPiLxi4E93UmLVkpXuDSONX4Z9W7Qfi/rrYaQ4EM m0g7SHsdJH6n+ozljQja5CxZ6eNQEhLfyjL6exA6KtuvC/EQv/AIv7wVlQc8wXRSNy41 JBPEycS1iCnAPHOCWYIchdJj/MyGN6G57wCCjzuQD5psUE5lB4v+JkI7qtut1u+Krgdi 1q3AEEQKV3w7De+8rzHfk++UyNgYykJqO6n6fMCgs4zQjX5tdYWvNUrmyxgF8Kt6gXxd Fe+A== X-Received: by 10.112.161.135 with SMTP id xs7mr18802846lbb.13.1412874527082; Thu, 09 Oct 2014 10:08:47 -0700 (PDT) Original-Received: by 10.114.77.131 with HTTP; Thu, 9 Oct 2014 10:08:47 -0700 (PDT) In-Reply-To: X-Google-Sender-Auth: TS0AbSMrNBU02RZiJib0tXqY5AU X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2a00:1450:4010:c03::233 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:175189 Archived-At: Good news everyone! As of my last commit (7a38270cd) dynamic loading is working pretty well. I've written a few toy modules in the "modules" folder. They can be used as examples and cover various use-case. - The `fmod' module simply expose the fmod(3) function from the math lib. - The `elisp' module evaluates code written in Lisp from C. - The `opaque' module show how any C structs/type can be embedded in a Lisp_Object. - The `curl' module can actually be useful as it links against libcurl and lets you fetch any http url supported by curl. You can use the same curl handle for multiple request in order to reuse the connection when possible (faster!). * Usage Dynamic loading can be enabled/disabled via the configure script. libltdl/ltdl.h is all you need to make it work. I've turned the option on by default (--with-ltdl). # build emacs $ git clone https://github.com/aaptel/emacs-dynamic-module.git emacs-dyn $ cd emacs-dyn $ ./autogen.sh $ ./configure $ make # build all the modules $ for i in modules/*; do make -C "$i"; done # run emacs! (no init file) $ ./src/emacs -Q Switch to *scratch* to test interactively the curl module: (add-to-list 'load-path "path/to/emacs/modules/curl") (require 'curl) => t ;; alternatively you can use ;; (load "path/to/curl/curl.so") ;; or ;; (load-module "path/to/curl/curl.so") (defvar c) (setq c (curl-make)) ;; allocate/init a curl handle => # > (curl-fetch-url c "http://example.com") ;; make and store request => t (curl-content c) " Example Domain ..." (curl-free c) ;; free the curl handle => t I've just tested the build procedure above before posting and *of course* for some reason a clean emacs build fail on something completely unrelated. Even at my first commit where I didn't make any change yet. make[1]: *** No rule to make target 'ctags.c', needed by 'ctags'. Stop. I suppose it comes from my setup? Gahh.. Anyway... * Implementation details and potential issues ** Loading The actual loading is done in Fload_module in lread.c. It is called from Fload when appropriate. See my previous emails. When dynamic modules are enabled, the `load-suffixes' variable has the additional .so suffix. This variable is used by `load' which in turn calls openp() to do the actual search in `load-path'. Adapting `load' to support modules made `require' and `provide' work with no additional work. Nothing is preventing a module from being loaded several times apart from the `features' mechanism. This might be problematic. ** Documentation I have updated several documentation related functions in order to support multiple DOCFILE (each module has its own). See my previous emails. The xref on function documentation (go to definition) is currently broken. I have not looked into it (issue #6). I have not fixed the issue (#1) Stefan mentioned earlier about a memory leak in the GC on the subr doc field because I *still* don't get it :( Any help is welcomed. ** Opaque user types Adding a new proper type is too much work for a module. You need to provide a reader, a printer, add a bunch of enums, etc. I've reused the Save_Value type to store arbitrary pointers. The object hierarchy is as follows: Lisp_Object > Lisp_Misc > Lisp_Save_Value > void* pointer A major problem with this is that you can't make type predicates e.g. `curl-handle-p'. A solution would be to make a new generic user type composed of a user pointer and a cookie or some sort of ID we can test. The space has to be large enough to limit collisions. ** Resource management An allocated curl handle has to be explicitly freed with `curl-free'. Could we extend the GC to provide some sort of destructor mechanism? Just a thought. Also, Lisp_Objects contained in a user type are not protected from the GC. This can be a problem. ** Native type size EMACS_INT_MAX is not always equal to INT_MAX depending on the architecture. Libraries can often use special value to signal something (e.g. (unsigned)-1 which is 0xffff..) which might not be fit in a Lisp type. What can we do about it? Is it really a problem? Emacs Lisp provides other way to signal errors and a module doesn't have to expose low-level stuff like this. ** Packaging Right now each module has its own Makefile and is built separately from emacs. We need to think of a way to package modules for distribution that is practical and takes into account what the module needs from the Emacs build (which configure option should be enabled to build module xxx?). ** Portability The .so suffix is hardcoded but via libltdl loading should work on most systems.