From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: Dynamic loading progress Date: Mon, 16 Feb 2015 17:43:14 +0200 Message-ID: <833865vp4d.fsf@gnu.org> References: <83y4oiiw81.fsf@gnu.org> <838ugdf251.fsf@gnu.org> <87bnl1vmqf.fsf@lifelogs.com> <87vbj8tow4.fsf@lifelogs.com> <87r3twtagf.fsf@lifelogs.com> <85siebl7ws.fsf@stephe-leake.org> <85a90ilwmm.fsf@stephe-leake.org> <83386a6f7z.fsf@gnu.org> <85h9upjz7v.fsf@stephe-leake.org> <83wq3k3kl4.fsf@gnu.org> <85bnkwil1c.fsf@stephe-leake.org> <83pp9cwky8.fsf@gnu.org> <85a90ggf2d.fsf@stephe-leake.org> <54E0A40F.5080603@dancol.org> <83sie7un20.fsf@gnu.org> <54E0D181.2080802@dancol.org> <83r3trulse.fsf@gnu.org> <54E0D7E0.305@[87.69.4.28]> <83h9unukbg.fsf@gnu.org> <54E0DEF8.7020901@dancol> <83egpruiyp.fsf@gnu.org> <54E0FF93.2000104@dancol.org> Reply-To: Eli Zaretskii NNTP-Posting-Host: plane.gmane.org X-Trace: ger.gmane.org 1424101434 5384 80.91.229.3 (16 Feb 2015 15:43:54 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 16 Feb 2015 15:43:54 +0000 (UTC) Cc: stephen_leake@stephe-leake.org, emacs-devel@gnu.org To: Daniel Colascione Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Mon Feb 16 16:43:39 2015 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1YNNpi-0006eb-Oh for ged-emacs-devel@m.gmane.org; Mon, 16 Feb 2015 16:43:39 +0100 Original-Received: from localhost ([::1]:40286 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YNNpi-0002wx-3q for ged-emacs-devel@m.gmane.org; Mon, 16 Feb 2015 10:43:38 -0500 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:50060) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YNNpK-0002wg-Mx for emacs-devel@gnu.org; Mon, 16 Feb 2015 10:43:16 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YNNpG-00044R-Bl for emacs-devel@gnu.org; Mon, 16 Feb 2015 10:43:14 -0500 Original-Received: from mtaout24.012.net.il ([80.179.55.180]:36395) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YNNpF-000442-Pb for emacs-devel@gnu.org; Mon, 16 Feb 2015 10:43:10 -0500 Original-Received: from conversion-daemon.mtaout24.012.net.il by mtaout24.012.net.il (HyperSendmail v2007.08) id <0NJV00800F2B3800@mtaout24.012.net.il> for emacs-devel@gnu.org; Mon, 16 Feb 2015 17:34:50 +0200 (IST) Original-Received: from HOME-C4E4A596F7 ([87.69.4.28]) by mtaout24.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0NJV00OLXFA14Q80@mtaout24.012.net.il>; Mon, 16 Feb 2015 17:34:50 +0200 (IST) In-reply-to: <54E0FF93.2000104@dancol.org> X-012-Sender: halo1@inter.net.il X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 80.179.55.180 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:183167 Archived-At: > Date: Sun, 15 Feb 2015 12:20:35 -0800 > From: Daniel Colascione > CC: stephen_leake@stephe-leake.org, emacs-devel@gnu.org > > Here's a broad outline of what I have in mind. Thanks. I think the next step is for someone to try rewriting a couple of modules on the branch in terms of this specification, and see if something is missing. Volunteers are welcome. Some comments on your write-up: > When Emacs loads a module, it uses dlsym (or the platform equivalent) > to find this routine and call it. If it returns 0, the module loaded > successfully; otherwise, we report an error to the caller. If this is all we need, then there's no need for libltdl, I think. > struct emacs_runtime { > size_t size; > struct emacs_env (*get_environment)(struct emacs_runtime* ert); > }; > > The `size' member tells modules how long the emacs_runtime structure > is. (It's better to use size than an explicit version field: this way, > .size = sizeof(struct emacs_runtime) is always correct.) This approach requires us to change the size each time we change the layout, even if the change itself leaves the size intact. Using a version field doesn't have this disadvantage. > Thread-local environments > ------------------------- > > The `get_environment' member lets us do anything else interesting. As > in Java, environments are thread-local. We only support one thread for > the moment, so this constraint is easy to enforce. (Just abort if we > use an emacs_env off the main thread.) Is it wise to design for threads at this point? The only suggestion on the table for "threads" is a far cry from real threads, and the current Lisp interpreter is otherwise thread-unsafe. Won't the thread infrastructure add a lot of cruft that will go unneeded for the observable future? > We'll represent all Lisp values as an opaque pointer typedef > emacs_value. Each emacs_value is either a local or a global > reference. Local references are valid only on the current thread and > only while the module function Emacs called is on the stack --- think > GCPRO. Global references are valid indefinitely: each one is a GC > root. With massive use of calls to Lisp functions from modules (since we don't provide them with direct access in C to internals of many objects), how can we ensure GC doesn't strike while the function that created an emacs_value is still on the callstack? You say "we don't lock ourselves into conservative stack-scanning GC", which I interpret as saying you don't want to rely on stack scanning to avoid a destructive GC in this case. But if we don't rely on that, where's the guarantee that such emacs_value will survive GC? > We'll represent all Lisp values as an opaque pointer typedef > emacs_value. This doesn't play well with --with-wide-int, where a value can be wider than a pointer. I think we should instead go with intmax_t or inptr_t, whichever is wider on the host. > Function registration > --------------------- > > typedef emacs_value (*emacs_subr)( > emacs_env* env, > int nargs, > emacs_value args[]); > > emacs_value (*make_function)( > emacs_env* env, > int min_arity, > int max_arity, > emacs_subr function); What about the doc string? > emacs_value (*funcall)( > emacs_env* env, > emacs_value function, > int nargs, > emacs_value args[]); Shouldn't funcall use emacs_subr? > Modules can register functions in the global namespace by calling a > Lisp-level function This is unclear, can you elaborate? What happens if a function is not "registered"? what's its status then? > When Lisp calls a module-defined function object, Emacs calls the > emacs_subr callback with which the function was defined. This is a change in the Lisp interpreter, I think. Why do we need this? > If Lisp signals or throws, `funcall' returns NULL. I suggest some other value or indication of that. NULL is a valid return value, so usurping it for errors might be too harsh. Or maybe I don't understand how will Lisp functions return values to the module, under your suggestion. Can you describe that? > `intern' also does the obvious thing. Do we need 'unintern' as well? > emacs_value (*type_of)( > emacs_env* env, > emacs_value value); > > Like Lisp type-of: returns a symbol. What is a "symbol", from the module's C code POV? You show no functions to access attributes of symbols, so it must be either one of the other types, like an integer or a string, or a C primitive data type, like a char * pointer. > int64_t (*fixnum_to_int)( > emacs_env* env, > emacs_value value); > > emacs_value (*make_fixnum)( > emacs_env* env, > int64_t value); > > These functions do the obvious thing. They signal error on type > mismatch. We use int64_t to handle big-integer Emacs variants on > 32-bit platforms. The last bit means we will need a utility function to return the valid range of integers, so that modules can be written to support 32-bit and 64-bit use cases without raising errors. > bool (*copy_string_contents)( > emacs_env* env, > emacs_value value, > char* buffer, > size_* length_inout); > > emacs_value (*make_string)( > emacs_env* env, > const char* contents); > > These functions let C code access Lisp strings. I imagine we'll > always produce and consume UTF-8. Strings in Emacs are of limited usability if you cannot encode and decode them. So this needs to be part of supported functionality, I think. More generally, modules that would like to process buffer or string text will have to be able to deal with Emacs's internal encoding of text, which means macros and functions we use in the core. The alternative of working only on UTF-8 encoded replica means we'd need to encode and decode text across the module boundaries -- that's a lot of consing. > `copy_string_contents' copies into a caller-allocated buffer instead > of returning a char* callers must free() --- this way, modules and the > Emacs core don't need to share the same C runtime. We can deal with > the buffer-length issue in a number of ways: here, we just accept the > destination buffer size in *length_inout and write the total length of > the string to *length_inout on normal return. We just truncate if > we're given too short a buffer and don't signal an error; this way, > callers can loop around and allocate a sufficiently large buffer for a > string's contents. That's an annoyance, IMO; why not provide a function to return the required size? Its implementation is trivial. > I think the interface above is enough for complete functionality in a > module, but for efficiency, we might want to expose additional > facilities, like access to a unibyte buffer's raw representation. I can envision a few additional missing bits: . direct access to buffer text (using buffer-substring means consing a lot of strings) . creation of opaque objects that should be returned to Lisp (like handles to objects managed by modules) . not sure how will a module "provide" its feature . some operations that must be efficient because they are typically done in inner loops, like regexp matching and accessing syntax of characters: doing that via Lisp will probably be horribly slow > Convenience library > --------------- One thing that's inconvenient is the need to drag the environment pointer through all the calls. Why exactly is that needed? > bool > emacs_find_file(emacs_env* env, const char* filename) > { > emacs_value e_filename = env->make_string(env, filename); > if(env->error_check(env)) return false; > emacs_value e_find_file = env->intern(env, "find-file"); > if(env->error_check(env)) return false; > return env->funcall(env, e_find_file, &e_filename, 1) != NULL; > } This kind of code looks tedious to me, no matter if it's in Emacs or in the module. Just an observation. Also, the buffer returned by find-file when it returns normally is lost here, isn't it? Thanks.