From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Lynn Winebarger Newsgroups: gmane.emacs.devel Subject: Re: Blocking calls and threads Date: Fri, 21 Apr 2023 00:39:29 -0400 Message-ID: References: <838ren6mpp.fsf@gnu.org> <83cz3y65ev.fsf@gnu.org> <837cu662oo.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="10133"; mail-complaints-to="usenet@ciao.gmane.io" Cc: emacs-devel@gnu.org To: Eli Zaretskii , Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Fri Apr 21 06:40:00 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ppiYx-0002PK-O6 for ged-emacs-devel@m.gmane-mx.org; Fri, 21 Apr 2023 06:39:59 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ppiY5-00045D-4W; Fri, 21 Apr 2023 00:39:05 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ppiY3-00044m-Qa for emacs-devel@gnu.org; Fri, 21 Apr 2023 00:39:03 -0400 Original-Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ppiY1-0001ro-MI; Fri, 21 Apr 2023 00:39:03 -0400 Original-Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-63b4960b015so1554245b3a.3; Thu, 20 Apr 2023 21:39:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682051939; x=1684643939; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=vJL8zWZAGlPrQerkKhV5ETXKSJsO0HvJOY6T9Xmbfm4=; b=azRd0RHGhjMTT1JpRldvVEjD1gpmFl7pKdaUnY6VhJdf20hqjPlDHYRHm5fogWWLyW Q4AV7PRQrJKjf67XRNH/i4KWHvIdE66S2eVrzjsE362Gqn9D7IN/AmW+piqpd3vRetOA 6b/qTFlpHCag2uXv0zwRzzfHYqE04KnbcJbye+cYsnOSg/3PDClwirbVaPRYALl5oT8i j9C8P7Y2rM9I18hd+gwMtF1LXyPkvjT87Rb8r3TNrCOy0HDqlTWFZmW8ffQJR23T+ufn Oyxg97WoypJnMBdzr88Jtl8C3CUfDFgC3lWBHsMvm42xuA0M3AuPC93NQI3g0yMe4wWB D58g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682051939; x=1684643939; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vJL8zWZAGlPrQerkKhV5ETXKSJsO0HvJOY6T9Xmbfm4=; b=HuCE8klEj+L4rk61OIDbE5rcykiZ158zf/8t1baVgyoW6kmZeHtG5KdrJQtf7y90Zo 8jgXSXZa95Ca+YdQ4O6RD/OUJRGsBq7azPnqHFSydNR5erzwB37Q61YODWL3O0K9i1mv PlfsQLcRdJC3/5CMB5l+FAwSHpVqouMkmpQu9Sjf60Cc6rR3JCxvQcxM0lC0JbieCSGM dz1Y5xb0fQepyTyv1zs/lwwwAsEg0rjT1M34i/iCCyUZiBxBOj41L4jP54OEvS2fWYzf ks+cdbhufgLQ/lTyUAs71mKqfN/lTutFQQzt0wZn/nlVsy0Otyv3cTfHLPJvMEF11Stf p/aQ== X-Gm-Message-State: AAQBX9dAWQPawQY8Goqw5qZgfPr3kkO5fwKGOlmGeH+NQNdW0YGnBylH lDdYE5+HlX4CsuwqRcitTEE6DRMxOfOsgDc9UHtasqmDbtE= X-Google-Smtp-Source: AKy350ZAvbbILLWYsW7GRg5yHi1+VNSVf/QJLuXKZ59uv1T44qBkPRMUSfIw4X9YzgZA6F64rWNy29W8Ju3auqYZopg= X-Received: by 2002:a17:902:d682:b0:1a6:abac:9cc with SMTP id v2-20020a170902d68200b001a6abac09ccmr3461795ply.66.1682051939013; Thu, 20 Apr 2023 21:38:59 -0700 (PDT) In-Reply-To: <837cu662oo.fsf@gnu.org> Received-SPF: pass client-ip=2607:f8b0:4864:20::429; envelope-from=owinebar@gmail.com; helo=mail-pf1-x429.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:305530 Archived-At: I'm not sure what the etiquette is here - I keep referring to Stefan's effort on futur.el, so I've added him explicitly. On Thu, Apr 20, 2023 at 10:36=E2=80=AFAM Eli Zaretskii wrote= : > > > From: Lynn Winebarger > > Date: Thu, 20 Apr 2023 10:19:11 -0400 > > Cc: emacs-devel@gnu.org > > > > > If you yield before issuing the system call, the system call will wai= t > > > until you re-acquire the lock. So how will this help? > > > > You're talking about yielding the system thread, I'm talking about > > yielding the Lisp machine thread. > > No, I'm also talking about the Lisp machine thread. The thread which > calls insert-file-contents and runs the C code of insert-file-contents > and of the subroutines called by insert-file-contents. A lisp thread is the context (lisp machine state) observable (intentionally) by lisp programs. That is conceptually distinct from the context tracked by the OS thread. To paraphrase a great mind, the identification of the two "is an implementation detail". Meaning, it is not normative with respect to the intended semantics of lisp threads. > > Even though Lisp machine threads are implemented by mapping them to > > the underlying system thread, the Lisp machine execution state is > > kept coherent by the global (interpreter) lock. Releasing the lock > > is effectively yielding the Lisp machine thread. The system thread > > will yield implicitly if the read blocks. > > What you say here is not relevant to the issue at hand. I don't know what you think is the issue at hand. My question was about when (or if) the lisp thread could yield during a blocking I/O operation. You appear to have interpreted that question differently than I intended, so I attempted to be more explicit about what I meant by "the lisp thread" and "yielding". > > the read operation should either use some temporary buffer and copy > > into the lisp buffer, or wait for data to be ready, then reacquire > > the GIL before invoking the read syscall with a pointer into the > > Lisp buffer object. > > Yes, and that's exactly where we will lose: most of the heavy > processing cannot be done in a separate temporary buffer, because it > calls functions from the Lisp machine, and those are written assuming > nothing else is running in the Lisp machine concurrently. For > example, take the code which decodes the file's contents we have just > read. I encourage you to take a good look at that code (most of it is > in coding.c) to appreciate the magnitude of the problem. I believe you. I did mention that the global lock could be (evidently *must be*) reacquired before the actual call to "read". There's also Tromey's comment on https://www.reddit.com/r/emacs/comments/utzxir/a_vision_of_a_multithreaded_= emacs/. OTOH, I asked the question in order to understand what it means to give programmers control over asynchronous execution in controlled ways. The most basic kind of control I can think of is whether functions called in that code are expected to behave synchronously or asynchronously. And what is a more basic operation, that could be done synchronously or asynchronously, than reading text from a file. If something like insert-file-contents can't be performed asynchronously (not in parallel, just with other code running while waiting for IO), the scope of what Stefan's effort is going to be very limited. Limited to the point of not being very interesting. It's also possible that something could be built on the existing data structures that doesn't rely on more fine-grained locking. I just happened to be reading https://www.gnu.org/software/emacs/manual/html_node/emacs/VCS-Merging.html today, after seeing this message, and it seemed to me highly relevant to the problem of concurrent work on a text buffer. Particularly the comment "Experience has shown that merging is superior to locking". Maybe the thing to do to enable asynchronous/concurrent/parallel work would be to add a new type of buffer, built on the existing one, that behaves something like a git repo/working copy of the text buffer, that eventually merges edits, say by windows displaying the buffer each pulling updates from all the "repos" with checked-out copies. Then the model for support of asynchronous programming could encapsulate specifying how to merge and/or handle merge failure. Maybe that would be too expensive, but at least at first, these distributed buffers would only be used by programs using explicit asynchronous programming. Maybe that approach would even be helpful in dealing with extremely large files or long lines. We could call it "merge-oriented programming". :-) > So the code which can run in parallel with another Lisp thread will be > able to do only very simple jobs, and will also add overhead due to > the need of copying stuff from temporary buffers to Lisp objects. I'm not talking about running in parallel - there is still just one lisp machine in this hypothetical. > Of course, we could redesign and reimplement this stuff, but that's a > lot of non-trivial work. My assumption was that you are considering > relatively lightweight changes on top of the existing code, not a > complete redesign of how these primitives work. I wasn't considering anything. I asked a very limited question for the purpose of giving some hard thought to the language problem Stefan requested assistance on. I didn't offer these elaborations because I have any plans, only to respond to your question "How could it?". That is a purely hypothetical question. Lynn