From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Akira Kyle Newsgroups: gmane.emacs.devel Subject: Re: "Asynchronous Requests from Emacs Dynamic Modules" Date: Sat, 31 Oct 2020 18:14:32 -0600 Message-ID: <86d00yexw7.fsf@akirakyle.com> References: <86imarfldb.fsf@akirakyle.com> <86y2jn9j70.fsf@163.com> <83pn4y96ut.fsf@gnu.org> <83lffmhi5y.fsf@gnu.org> <83h7qahe03.fsf@gnu.org> <86ft5ufb95.fsf@akirakyle.com> <831rhegnde.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; format=flowed Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="35282"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: mu4e 1.4.13; emacs 28.0.50 Cc: p.stephani2@gmail.com, emacs-devel@gnu.org, yyoncho@gmail.com, monnier@iro.umontreal.ca, all_but_last@163.com To: Eli Zaretskii Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Nov 01 01:16:11 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kZ12Y-00091q-Rm for ged-emacs-devel@m.gmane-mx.org; Sun, 01 Nov 2020 01:16:10 +0100 Original-Received: from localhost ([::1]:45734 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kZ12X-0007jd-T0 for ged-emacs-devel@m.gmane-mx.org; Sat, 31 Oct 2020 20:16:09 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:35902) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kZ115-0006Wk-NB for emacs-devel@gnu.org; Sat, 31 Oct 2020 20:14:39 -0400 Original-Received: from mail-io1-f44.google.com ([209.85.166.44]:35345) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kZ112-0008Co-E1; Sat, 31 Oct 2020 20:14:39 -0400 Original-Received: by mail-io1-f44.google.com with SMTP id k6so11389194ior.2; Sat, 31 Oct 2020 17:14:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:references:user-agent:from:to:cc:subject :in-reply-to:date:message-id:mime-version; bh=4jVm8ArDp12iMm27bCKh3p846wwxNgXZWKk3MDQHVkw=; b=aQ+So1OinKMxYQ75SrjkftsZ5rA3o5PWAveKz4Hheo8PmgHo0cvuzXIjsoBFqHIWCO xZ9J/6emlIkXF1mh3PPfX20N4gYBR2sDaPnU+RFU7wTmLCwF62gb6y2s7yNys9mg0kl/ Lvu2fYaAc81y/hc/2OeDoEB5B5RcmQ3B6ScfDjRmdrJXsGtxAkgnqR6WGWdBxHLoeOI5 9kwVc8EIWRvwUh001R1gG1aYSlgDnJhsCU/L1HrT+xMpAJzAXsu8J6KNYM2SeSDyonLe 77bo7EoJkHQkSQbWL2sz86QJzcXkKm0Uj/+1H9n8CZcLMEsRDaok6usW9uM7+B4LTEv0 nykQ== X-Gm-Message-State: AOAM531j8XIwID/nF4z/tBwpo/Riqj9eXV6xNzvYxYlm+jj/aB4icHCa MAurNFowU9XVqMZYoAy1cFA= X-Google-Smtp-Source: ABdhPJzdsvLu4YEe+6D+5lSJSm6JIoiqUjZgYGO1t6AHNpnz5WNL8H7dyHBr9si2Vsdor8c0Ab99eQ== X-Received: by 2002:a6b:7114:: with SMTP id q20mr1423749iog.16.1604189673608; Sat, 31 Oct 2020 17:14:33 -0700 (PDT) Original-Received: from lore ([2601:281:8080:45f0:67c5:de00:aa30:3678]) by smtp.gmail.com with ESMTPSA id f203sm7383431ioa.23.2020.10.31.17.14.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Oct 2020 17:14:33 -0700 (PDT) In-reply-to: <831rhegnde.fsf@gnu.org> Received-SPF: pass client-ip=209.85.166.44; envelope-from=aikokyle@gmail.com; helo=mail-io1-f44.google.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/10/31 20:14:33 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -13 X-Spam_score: -1.4 X-Spam_bar: - X-Spam_report: (-1.4 / 5.0 requ) BAYES_00=-1.9, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.249, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:258592 Archived-At: Thanks for the feedback. On Sat, Oct 31, 2020 at 02:18 PM, Eli Zaretskii wrote: >> 1) Chris Wellon's solution in the article I linked using the >> SIGUSR1 signal which Emacs handles in the event queue notify >> the >> running Lisp Thread that it should call back into module >> functions. This idea could be expanded to allow modules to >> define >> their own events and lisp handlers that will respond to them. > > A terrible design, if you ask me. Reminds me how timers worked > in > Emacs long ago: we had an external program which would > periodically > deliver a signal to Emacs. We switched off of that to what we > have > now, for good reasons. > > And please keep in mind that doing anything non-trivial from a > signal > handler is inherently unsafe. I agree signal handlers are tricky to get right and probably not the best option here, especially considering the information that can be conveyed through the signal is pretty limited. >> 2) Stefan's and Philipp's proposal of using the process >> interface >> with file descriptors to pipes and perhaps `process-filter` to >> notify the Lisp Thread that it should call back into the module >> functions. > > This is the way to go, IMO. This is how Emacs itself handles > async > events, so the infrastructure for doing this, both in C and in > Lisp, > already exists, and is mature. All you need is use it. I'll explore this option a bit then and see what I can do with sharing a pipe between Lisp and a dynamic module thread. >> 3) yyoncho's proposal to use the lisp threading interface, >> specifically condition variables to allow dynamic modules to >> `condition-notify`. > > This is unworkable within the current design of Lisp threads. > condition-notify releases the global lock, something that under > the > current design cannot be done at an arbitrary time, because it > could > cause two threads run in parallel. > >> I think the most natural solution would be a dynamic module >> interface that allows grabbing and releasing Lisp's global >> thread >> lock. > > I encourage you to study how threading works in Emacs (the code > is in > thread.c). I'm sure you will see right away why this cannot be > done > without a complete redesign of how switching threads works. > There's > no way of interrupting a running Lisp thread in an arbitrary > place and > switching to another thread. It seems the lisp threading functionality is still pretty beta quality. Have the data races that Chris Wellons mentions here [1] ever been addressed? I generally agree with his sentiment about the current lisp threading interface seems like not the right thing for lisp. It looks like it's mostly just exposing the pthread interface. What is the use case for the current lisp threads given the severe limitation that only will ever run at a time and the constraints the cooperative nature puts on when threads can switch? As I've poked around a bit more I see how my previous thought that some module interface for grabbing and releasing the Lisp thread lock would probably really go against the a lot of the current way Emacs handles threads and the emacs_env struct. I was basing that thought on the way python handles this where it's relatively easy to write low level primitives that release the GIL by just surrounding it with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS and it's up to you to ensure you don't call any interpreter functions. I still think something like this is possible (it would probably be completely separate from the lisp threading interface), but it would probably just require a lot of care around ensuring the integrity of the stack and controlling exactly when in the event loop it would be okay for a thread switch to happen. The more I think about it, the more it would end up resembling the polling that's already done for the process interface with `wait_reading_process_output`. [1] https://nullprogram.com/blog/2018/05/31/