From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Spencer Baugh Newsgroups: gmane.emacs.devel Subject: Releasing the thread global_lock from the module API Date: Fri, 01 Mar 2024 09:53:41 -0500 Message-ID: Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="33478"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Philipp Stephani To: emacs-devel@gnu.org Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Mar 01 17:30:06 2024 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rg5lt-0008UW-Bs for ged-emacs-devel@m.gmane-mx.org; Fri, 01 Mar 2024 17:30:05 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rg5l1-0006Oi-9O; Fri, 01 Mar 2024 11:29:11 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rg4Gi-0000el-I6 for emacs-devel@gnu.org; Fri, 01 Mar 2024 09:53:48 -0500 Original-Received: from mxout5.mail.janestreet.com ([64.215.233.18]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rg4Gd-0003Jh-9u for emacs-devel@gnu.org; Fri, 01 Mar 2024 09:53:47 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1709304821; bh=2tdCUislQbLiJrzzucuaocJTbdn/4eL9m+3wo7sEiQU=; h=From:To:Cc:Subject:Date; b=xMpvYPKXqlBi3mRgszj/hevN5KVSmF7PvJxi6Vf4A705I9RtNKd4DNM10PStOTqNd r4qSUrTiVUz6FlsQcXGj9yeYjIm4FVkGtObk8mRW4iDhNDtS7ERzrZvh35vWoxLVEJ JnrFYvQwI85f06SLAhZmRq4Y9016rjvt+8UvbRJBQi7/WMDCVxW+Duh7zVE31XqcGR Uun7Ad4WEQDR/tf0ZjwoB42E/jNPzWLgXk02VYh25K7rAahL62qiJCEFqh8PzrHX/p cR4jc7h7fiLMkXhpN+nbDE0W0ryfWi8ickk1tott5PvdaOMy7Stduuz07C1PhCxSWd 7KbOrVYYvTIOg== Received-SPF: pass client-ip=64.215.233.18; envelope-from=sbaugh@janestreet.com; helo=mxout5.mail.janestreet.com X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Fri, 01 Mar 2024 11:29:08 -0500 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:316661 Archived-At: Lisp threads all take global_lock while they are running Lisp code. This is good and correct. Lisp threads can also call module code. This is also good and correct. In other languages with an FFI and a global interpreter lock taken by threads, such as Python, it's possible for a native function called from the interpreter to release the global lock, and then re-acquire it before calling any interpreter functions (or automatically re-acquire it upon returning to the interpreter). This allows for a limited form of actual parallelism: if a native function is doing work that doesn't involve the Python interpreter, e.g. doing numerical computations in native code, it can release the lock so that it can run in parallel with other threads. If the native function needs to call a function which does involve the Python interpreter, it can re-acquire the global lock around that call. Could the same functionality be added to the Emacs module API? Then module code could release the global lock when doing Emacs-independent computation, and re-acquire the lock when calling into Emacs. This would allow a limited form of parallelism: if a Lisp thread is calling module code which releases the lock, then the module code could run in parallel with the rest of Emacs on that thread. This should be safe, because this is in effect already possibly through thread-yield: that can be called from anywhere in a Lisp thread, and releases the global lock, calls the operating system thread_yield function, and then re-acquires the global lock. If instead of doing thread_yield, it did some computation, e.g. int x = 0 for (int i = 0; i<999999; i++) x += i; that would have the same effect as yielding, but the computation would run in parallel with the main Emacs thread. This is what would happen if module code released the lock, did some Emacs-independent work, and then re-acquired the lock. As an additional bonus, this would allow the module API to extend Lisp threads: if a C library provides some blocking function which does some complicated form of IO, a module providing bindings for that C library can release global_lock before calling that function, and then re-acquire the lock after the function returns. Then Lisp threads calling this module function will not block the main Emacs thread.