From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Daniel Colascione Newsgroups: gmane.emacs.devel Subject: Re: Dynamic loading progress Date: Sun, 15 Feb 2015 12:20:35 -0800 Message-ID: <54E0FF93.2000104@dancol.org> References: <83y4oiiw81.fsf@gnu.org> <838ugdf251.fsf@gnu.org> <87bnl1vmqf.fsf@lifelogs.com> <87vbj8tow4.fsf@lifelogs.com> <87r3twtagf.fsf@lifelogs.com> <85siebl7ws.fsf@stephe-leake.org> <85a90ilwmm.fsf@stephe-leake.org> <83386a6f7z.fsf@gnu.org> <85h9upjz7v.fsf@stephe-leake.org> <83wq3k3kl4.fsf@gnu.org> <85bnkwil1c.fsf@stephe-leake.org> <83pp9cwky8.fsf@gnu.org> <85a90ggf2d.fsf@stephe-leake.org> <54E0A40F.5080603@dancol.org> <83sie7un20.fsf@gnu.org> <54E0D181.2080802@dancol.org> <83r3trulse.fsf@gnu.org> <54E0D7E0.305@[87.69.4.28]> <83h9unukbg.fsf@gnu.org> <54E0DEF8.7020901@dancol> <83egpruiyp.fsf@gnu.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="bKlLuo4BkcDlAibsgjfomvtw1U2OtcGEc" X-Trace: ger.gmane.org 1424031686 14587 80.91.229.3 (15 Feb 2015 20:21:26 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 15 Feb 2015 20:21:26 +0000 (UTC) Cc: stephen_leake@stephe-leake.org, emacs-devel@gnu.org To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sun Feb 15 21:21:18 2015 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1YN5gm-0001mg-MR for ged-emacs-devel@m.gmane.org; Sun, 15 Feb 2015 21:21:12 +0100 Original-Received: from localhost ([::1]:36498 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YN5gl-00071A-VC for ged-emacs-devel@m.gmane.org; Sun, 15 Feb 2015 15:21:11 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:49634) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YN5gP-0006th-Gn for emacs-devel@gnu.org; Sun, 15 Feb 2015 15:20:55 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YN5gJ-0003TP-CG for emacs-devel@gnu.org; Sun, 15 Feb 2015 15:20:49 -0500 Original-Received: from dancol.org ([2600:3c01::f03c:91ff:fedf:adf3]:36945) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YN5gI-0003TD-JE; Sun, 15 Feb 2015 15:20:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=dancol.org; s=x; h=Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=1+ymXYgLbGg0I7EvRykfK4e3BxMd+IySAeoCg1Hq+x8=; b=BdQgAnyVwdXNLfCkpY57rnDNXqCarFJi0y4efgjP/JPbOLb5hnfo00LPUJxkSiUDEszGNVXEtAfeqh23eZGkE46IedyhWvWrDf3tSVO8XkpxiCx8Ammo2xlXK8hENI/nfsP2oy+aJ3wrUqkYpjduYCHHN4p7N7nC1sY2Yp169mcBmMX7ykUI856kRW2DlpRKsysrFZhp9n2u11In1q++OIO+G7A6qh3/KxeQbgIi/58MUmRXgNZzv8gEkhm0tqu1Fpnl/CuSEAZyc1bjI9KwCgxBIX95jqCGKElws8mQV0V7R226O54o6PqgTqxwtHKmGG/qJYn0JPH28KtJvBI/Mg==; Original-Received: from [2620:10d:c081:1101:2ab2:bdff:fe1c:db58] by dancol.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84) (envelope-from ) id 1YN5gH-0002GW-LW; Sun, 15 Feb 2015 12:20:41 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 In-Reply-To: <83egpruiyp.fsf@gnu.org> X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2600:3c01::f03c:91ff:fedf:adf3 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:183119 Archived-At: This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --bKlLuo4BkcDlAibsgjfomvtw1U2OtcGEc Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 02/15/2015 10:29 AM, Eli Zaretskii wrote: >> Date: Sun, 15 Feb 2015 10:01:28 -0800 >> From: Daniel Colascione >> CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org >> >>> Would you like to propose a list of functions that should be in that >>> table? I don't mean function names, I mean a short description of >>> each one of them. >> >> That depends on the supported module use cases, which are being >> discussed on other branches of the thread. >=20 > You can easily look at the modules already on the branch, I think they > will give enough data points to at least start with this. Here's a broad outline of what I have in mind. We want an ABI powerful enough to let C modules interact with Emacs, but decoupled enough to let the Emacs core evolve independently. The best model we have for an interface like that is JNI. See the specification at [1] for comparison. The JNI designers carefully considered issues like error propagation, GC integration, thread safety, type coercion, and so on, so let's depart from the JNI model only where there's a good reason to do so. We don't need all of JNI's complexity, but let's try to provide a simple subset with an option to extend it later. First, let's define a JNI-like entry point. Call it emacs_module_init: struct emacs_runtime; extern int emacs_module_init(struct emacs_runtime* ert); When Emacs loads a module, it uses dlsym (or the platform equivalent) to find this routine and call it. If it returns 0, the module loaded successfully; otherwise, we report an error to the caller. That's the limit of Emacs integration with the system dynamic loader. ERT provides functions the module can use to do everything else. We'll declare struct emacs_runtime ABI-stable; we can add fields to the end of the struct without breaking binary compatibility. Runtime environment ------------------- Let's define the emacs_runtime structure like this: struct emacs_runtime { size_t size; struct emacs_env (*get_environment)(struct emacs_runtime* ert); }; The `size' member tells modules how long the emacs_runtime structure is. (It's better to use size than an explicit version field: this way, =2Esize =3D sizeof(struct emacs_runtime) is always correct.) Modules use `get_environment' to retrieve a pointer to an `emacs_env' structure that lets them do interesting things; Emacs also supplied module-registered callbacks with `emacs_env' pointers on entry. Thread-local environments ------------------------- The `get_environment' member lets us do anything else interesting. As in Java, environments are thread-local. We only support one thread for the moment, so this constraint is easy to enforce. (Just abort if we use an emacs_env off the main thread.) Now we can define emacs_env like this. (Ignore the forward definitions for the moment.) typedef struct emacs_env_25 emacs_env; typedef struct emacs_value_tag* emacs_value; typedef emacs_value (*emacs_subr)( emacs_env* env, int nargs, emacs_value args[]); struct emacs_env { size_t size /* struct size */; emacs_value (*make_global_reference)( emacs_env* env, emacs_value any_reference); void (*free_global_reference)( emacs_env* env, emacs_value global_reference); bool (*error_check)(emacs_env* env); void (*clear_error)(emacs_env* env); bool (*get_error)( emacs_env* env, emacs_value* error_symbol_out, emacs_value* error_data_out); void (*signal_error)( emacs_env* env, emacs_value error_symbol, emacs_value error_data); emacs_value (*make_function)( emacs_env* env, int min_arity, int max_arity, emacs_subr function); emacs_value (*funcall)( emacs_env* env, emacs_value function, int nargs, emacs_value args[]); emacs_value (*intern)( emacs_env* env, const char* symbol_name); emacs_value (*type_of)( emacs_env* env, emacs_value value); int64_t (*fixnum_to_int)( emacs_env* env, emacs_value value); emacs_value (*make_fixnum)( emacs_env* env, int64_t value); double (*float_to_c_double)( emacs_env* env, emacs_value value); emacs_value (*make_float)( emacs_env* env, double value); bool (*copy_string_contents)( emacs_env* env, emacs_value value, char* buffer, size_* length_inout); emacs_value (*make_string)( emacs_env* env, const char* contents); }; That's a pretty bare-boned interface. If we want, we can add more members for efficiency and functionality reasons. (Maybe car and cdr.) emacs_env size -------------- The first member in emacs_env is `size': that's just the size the emacs_env structure provided by emacs to a module. In emacs.h, we'd have something like this: struct emacs_env_25 { ... }; struct emacs_env_26 { ... }; ...; typedef struct emacs_env_26 emacs_env; This way, modules will use the latest version by default, but can check for a lesser version with something like this: if (env->size < sizeof(struct emacs_env_26)) { avoid_thing_that_works_only_in_emacs_26(); } Memory management ----------------- Let's steal from JNI some more. emacs_value (*make_global_reference)( emacs_env* env, emacs_value any_reference); void (*free_global_reference)( emacs_env* env, emacs_value global_reference); We'll represent all Lisp values as an opaque pointer typedef emacs_value. Each emacs_value is either a local or a global reference. Local references are valid only on the current thread and only while the module function Emacs called is on the stack --- think GCPRO. Global references are valid indefinitely: each one is a GC root. Modules can use make_global_reference to allocate a global reference (i.e., a GC root) for any emacs_value; modules must then free these references explicitly. All routines (except make_global_reference) that return emacs_value values return local references. It's up to modules to register long-lived references explicitly. This way, normal module code remains pretty efficient (since it deals only with stack-allocated local references), but can retain arbitrary data. Also, we don't lock ourselves into conservative stack-scanning GC. Error handling -------------- We can't longjmp through arbitrary C code! We have to let modules treat errors more conventionally. Let's use the JNI approach: bool (*error_check)(emacs_env* env); void (*clear_error)(emacs_env* env); bool (*get_error)( emacs_env* env, emacs_value* error_symbol_out, emacs_value* error_data_out); void (*signal_error)( emacs_env* env, emacs_value error_symbol, emacs_value error_data); We'll give each thread (initially, our only thread) a pending-error information. When Emacs calls a module function, the current thread's pending-error flag will be clear. When that module returns to Emacs, if the thread's pending-error flag is set, Emacs signals the condition corresponding to the current thread's error information. When the module calls an Emacs routine that would ordinarily signal, Emacs catches the signal at the stack frame just before control flow would return to the module, sets the pending-error flag, and returns to the module normally. (Functions that would return a value on success instead return a dummy value, like 0 or NULL.) `error_check' just returns true iff an error is pending. `clear_error' removes pending error information. `get_error' retrieves the pending error information, without clearing it, and returns true. If no error is set, it returns false. signal_error sets the current thread's pending-error flag. To simplify the interface, we can treat `catch' and 'throw' as special kinds of error. Like JNI, we can just declare that it's illegal to call all but a few specially-marked functions (like global reference deregistration) with a pending error. Function registration --------------------- typedef emacs_value (*emacs_subr)( emacs_env* env, int nargs, emacs_value args[]); emacs_value (*make_function)( emacs_env* env, int min_arity, int max_arity, emacs_subr function); emacs_value (*funcall)( emacs_env* env, emacs_value function, int nargs, emacs_value args[]); emacs_value (*intern)( emacs_env* env, const char* symbol_name); Modules create function values using make_function; it works like lambda; max_arity =3D=3D -1 indicates a varargs function. Modules can register functions in the global namespace by calling a Lisp-level function; we can also just provide a `defun' API, although strictly speaking, one would be superfluous. When Lisp calls a module-defined function object, Emacs calls the emacs_subr callback with which the function was defined. Modules can call Lisp functions using `funcall', which does the obvious thing. If Lisp signals or throws, `funcall' returns NULL. `intern' also does the obvious thing. Type coercion ------------- emacs_value is not very useful without letting C look inside specific types of value. emacs_value (*type_of)( emacs_env* env, emacs_value value); Like Lisp type-of: returns a symbol. int64_t (*fixnum_to_int)( emacs_env* env, emacs_value value); emacs_value (*make_fixnum)( emacs_env* env, int64_t value); These functions do the obvious thing. They signal error on type mismatch. We use int64_t to handle big-integer Emacs variants on 32-bit platforms. double (*float_to_c_double)( emacs_env* env, emacs_value value); emacs_value (*make_float)( emacs_env* env, double value); These functions do the obvious thing. They signal on type mismatch. bool (*copy_string_contents)( emacs_env* env, emacs_value value, char* buffer, size_* length_inout); emacs_value (*make_string)( emacs_env* env, const char* contents); These functions let C code access Lisp strings. I imagine we'll always produce and consume UTF-8. `copy_string_contents' copies into a caller-allocated buffer instead of returning a char* callers must free() --- this way, modules and the Emacs core don't need to share the same C runtime. We can deal with the buffer-length issue in a number of ways: here, we just accept the destination buffer size in *length_inout and write the total length of the string to *length_inout on normal return. We just truncate if we're given too short a buffer and don't signal an error; this way, callers can loop around and allocate a sufficiently large buffer for a string's contents. Other functionality ------------------- I think the interface above is enough for complete functionality in a module, but for efficiency, we might want to expose additional facilities, like access to a unibyte buffer's raw representation. Convenience library --------------- This kind of interface is adequate, but asking callers to env->intern() every Emacs facility they want to use is inconvenient. It'd be nice to provide a statically-linked convenience library, written in terms of the ABI-stable interface, that made it straightforward to set text properties, visit files, and so on. For example, we could provide a function like this: bool emacs_find_file(emacs_env* env, const char* filename) { emacs_value e_filename =3D env->make_string(env, filename); if(env->error_check(env)) return false; emacs_value e_find_file =3D env->intern(env, "find-file"); if(env->error_check(env)) return false; return env->funcall(env, e_find_file, &e_filename, 1) !=3D NULL; } Questions --------- 1) Do we need JNI-style local variable frames and functions that release local references? 2) Maybe we want a separate, non-emacs_value type for global references? 3) How exactly do we represent catch/throw values? 4) Do we need to use int64_t for integers? 5) Do we need to provide direct access to string contents? [1] http://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/jniTOC.htm= l [2] Let's not support JNI's deprecated implicit registration system, where modules can register functions with the runtime by exporting symbols with certain magical names. --bKlLuo4BkcDlAibsgjfomvtw1U2OtcGEc Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCAAGBQJU4P+TAAoJEN4WImmbpWBl3/MP/3+xT/DryYQ2XpCgQdN9y0T/ bmCeCEhNaRtdwyh0mUxidzmHDWMkjph8TJ6xup5ah2nMBECp/Lmv6JV0cNJ6pdax OOv/OLc1ZmnAHtyjBqns84AmtTVuIwdcvdkJquZWrRRoWxwYjyuN4fp0jwB7HAkr 3R79/nAD4ziZ/VrtmKZKzGp1S6laDTRfuiImgzXtuNjOCcH7oiwjXtchjcm6EEBf 4Um2WTLriD0XaDDSz2+8caw6pNhaY0HZBRrmzphkKDhyQaJhTwNfB9aLYamwXW3z 8MNM7csgEuqWpGdEjCIohVUtxPqqd8LfcwgXc5htoOP/w6jRgFxXFoHCGol+6t9d KgCHApofa4A0lqJBplLz+NNFLg08WRZgtKFRXv4UQGlf4s0SHfBZFLOf5j+tjQab R3GyIxxDSsIGqKgz6CuJW/mhudwTfOc5KqWUnKdJfGP0Lm3KJnIbfPaZmyfSgOm9 zcrDabLz98BLXAlHGcESByU8yvsdB1JD3EuY7QYUUqzpu+0n7/uWrp9epfRrBg4W zndyae+RspcrA7qucDzQreedDppGBpfMwYWQQUwxDlmHt59DNZ3JdH0HKGTHHlEh 652zzaXAo+9Dbof4uz2iDu5Ok1sF6FtlanUIaD+qXgAKuYIKRbT8WVTyHywETeAY hoFHGfpq8H0oYGo9Jt7n =k/Oc -----END PGP SIGNATURE----- --bKlLuo4BkcDlAibsgjfomvtw1U2OtcGEc--