From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id IBaMBvRa1mWpAwAAe85BDQ:P1 (envelope-from ) for ; Wed, 21 Feb 2024 21:20:04 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:4876::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id IBaMBvRa1mWpAwAAe85BDQ (envelope-from ) for ; Wed, 21 Feb 2024 21:20:04 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="GLeo/Td6"; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1708529860; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=vzlCVVpd5ikYD/dFTYHKiEwzRYf/G20vr+oATNZbI4o=; b=JeFDgpaS+U/G/+oxuIVwQFBdCTRsHy3/DdNeX9tu01wU7y+iWrgMQ33HIq/HehbiA7qw3L 7sxQPjqQUBiimHcbd6jQfSwVZfeRVeiXpkTsufUIPcxa8bkITX2pRbmc2qSlS+//nqJRgG Cu3Jm09GyM3m6b+SMAexJXR30KfyHDMYI8s8Aq+bnWhku53E4q7VtmaRvb+qOrVhL3IaSF oTEocLbPq0RP0H1nUKw4N83gyJRvGvbiZcqzVvwg/sXLY6E0EfG1vdtAI12mukkB/QiU7b 9oESYODKCjUFIMaoFVLue6INxKFEtDIMfaPWJEoo5kN8f/NpkEFa3q0VNbrAmw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="GLeo/Td6"; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=key1; d=yhetil.org; t=1708529860; a=rsa-sha256; cv=none; b=iUswl6Fg3fC8tvDmYlD5whN3ZjTTzhOabfugSiI7jpbZUh1F2DU9DdXr52XjgA9TYo7HOU O7YRjAt5XRkbx+FV8OINC9F+sC4igXBRAQIycWuw+PUCiteekD/5hOcI05EQnkhWn3+5/8 53ODeaW/SfPd7znIFFD65OhJIGusOqhRBBNd6wJM2MuZN9qpm9lAGMW3gv7y39GNs1FkwZ 8kXc1voo4E/6iBr5LEjq+WFfr5aAq7e3ElnR/sjRM5QFY4HIAAwOPUZMuCFM7jt/VZASXw JkptYdGaJhSl8A16dvnNkvg0l11qq8xSM9BKWHByrKYYx/5D+LhTJC6W05JlBQ== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id BA4737DE18 for ; Wed, 21 Feb 2024 16:37:40 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rcobz-0005Qt-Q1; Wed, 21 Feb 2024 10:34:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rcoVB-0003hS-N5 for emacs-orgmode@gnu.org; Wed, 21 Feb 2024 10:27:25 -0500 Received: from mail-lj1-x230.google.com ([2a00:1450:4864:20::230]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rcoV9-0005P0-HM for emacs-orgmode@gnu.org; Wed, 21 Feb 2024 10:27:17 -0500 Received: by mail-lj1-x230.google.com with SMTP id 38308e7fff4ca-2d180d6bd32so77140511fa.1 for ; Wed, 21 Feb 2024 07:27:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708529232; x=1709134032; darn=gnu.org; h=mime-version:date:references:in-reply-to:subject:cc:to:from :message-id:from:to:cc:subject:date:message-id:reply-to; bh=vzlCVVpd5ikYD/dFTYHKiEwzRYf/G20vr+oATNZbI4o=; b=GLeo/Td62Drl0tgUVzZXZbN8q+0ng8J+tW7+EtQfvPc9aZeJch5J/bXPhn6hzwxsK7 XHChYwfO+H335mV8ahGEW9Tirr6OstjA9rPnhsoJjinkBEhDn646YeZYBefXrFIxTTDs 7NzhFkzGZnddXyS55vsy3RzsDDcaPSieM3f/TC9vVcXBcOS6margkAbzBHOXLHShSrHx PeAMkE31Q3K8DkceBxrUzUsOGwNxO0FG+aldeHMKjc1eBuq2eHvWGxWLiVGd7h5mS3MJ suOJc5ndjlv76UjaWZ/OGucB9iF8rIk3GhUPrUfLJGd254jB5//LJDe64H+E7vBRp1ak 2dFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708529232; x=1709134032; h=mime-version:date:references:in-reply-to:subject:cc:to:from :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vzlCVVpd5ikYD/dFTYHKiEwzRYf/G20vr+oATNZbI4o=; b=w3GxWKcfYxiInI+VaBPas+CNeYHKhbFHz568UhbjHJQjn1S2ijXGhzQ3XOvLis81Rg 0VV36vXTv3O5GC3lIxJjxT5GL8z9MH99tLwVwKH+vbInlwYwHYxNcFc3s6S0C3UmkKE5 S4SakVdHIwZUXXZM2o4wNkDrbvswmrmHJxCwOHDsR8OLvaXDrDHO+HWXSyo0c2ibpcr2 Sx3Wp3mQAA9hv6fji46ZUUN/4j1f7J2G7xld5GKJ+WTBkXyVzwDGs4E/99ZH16aEV736 7lYvil02WOrWzCWVaKcH/BjARMEcaUdJLp9ylC1URznFgcviwqu2ArmjjhWqUFks64IB dY/g== X-Forwarded-Encrypted: i=1; AJvYcCX+aD7lnaGDduR9GyrS2I5qBe9aZbohRLfmte9yNs4S5BS4RihWyW+wqrn1CXRoaaCZbZqHyR+dW1tqBaBJ22RNPzpe9ew= X-Gm-Message-State: AOJu0YxhkLuRlB4wR4EaKvau/oRTSwa19cxIGGH6L6F4IF4ZYL6P/Ui1 voqCiUS1VqwKsYm4SeF9ObPobU2iHPY87jysYdfqSjST5dv356yW X-Google-Smtp-Source: AGHT+IGE7hJdq4Yfhzg//zLt5/ZI+dkT+VtGmp7zH6L/5HFfEulBUWMLpM4H9lUse4Zi7bW8HF6yFw== X-Received: by 2002:a2e:9b52:0:b0:2d2:44cb:b0a3 with SMTP id o18-20020a2e9b52000000b002d244cbb0a3mr3925329ljj.48.1708529231666; Wed, 21 Feb 2024 07:27:11 -0800 (PST) Received: from keynux ([2a01:e0a:505:3460:1c18:688d:ece4:372e]) by smtp.gmail.com with ESMTPSA id h7-20020a056000000700b0033ce214a97csm17192735wrx.17.2024.02.21.07.27.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 07:27:11 -0800 (PST) Message-ID: <65d6164f.050a0220.5a5dc.f7a8@mx.google.com> Received: by keynux (sSMTP sendmail emulation); Wed, 21 Feb 2024 16:27:09 +0100 From: Bruno Barbier To: Matt Cc: Ihor Radchenko , Jack Kamm , emacs-orgmode Subject: Re: Asynchronous blocks for everything (was Re: [BUG] Unexpected result when evaluating python src block asynchronously [9.7-pre (release_9.6.17-1131-gc9ed03.dirty @ /home/yantar92/.emacs.d/straight/build/org/)]) In-Reply-To: <18dbe11968a.12c0800a31425096.5114791462107560324@excalamus.com> References: <87o7d0mm54.fsf@localhost> <65bbb108.050a0220.b60fd.6790@mx.google.com> <87jznm8hcu.fsf@gmail.com> <875xz42rp9.fsf@localhost> <874jen8zec.fsf@gmail.com> <87o7cv9e80.fsf@localhost> <65c2875f.050a0220.caf6d.8291@mx.google.com> <18dae5cab1d.bf1c7563863897.4896289306902277373@excalamus.com> <65cfa0d8.050a0220.cb569.ce34@mx.google.com> <18dbe11968a.12c0800a31425096.5114791462107560324@excalamus.com> Date: Wed, 21 Feb 2024 16:27:09 +0100 MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2a00:1450:4864:20::230; envelope-from=brubar.cs@gmail.com; helo=mail-lj1-x230.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Scanner: mx10.migadu.com X-Migadu-Spam-Score: -6.14 X-Spam-Score: -6.14 X-Migadu-Queue-Id: BA4737DE18 X-TUID: mU2a1OuHPIDs Hi Matt, Thanks for your answer. Matt writes: > ... > If I understand correctly, there are several independent topics the code addresses: > > | topic | manner addressed | > |------------------+------------------------------------------------| > | execution status | using overlays to communicate execution status | > | locating results | using overlays to locate results | > | blocking | making all execution asynchronous | > |------------------+------------------------------------------------| > > I suggest these be discussed in separate threads. Actually my patch is only about reporting execution status and inserting results, in a generic way, in ob-core. I provided some demo code on top of them, just to show how it could be used. Sorry about the confusion. > ... > > Since this thread is dedicated to blocking, let me share my thoughts on that subject. I guess I should start a new thread then :) > ... > > > Executing a shell block requires starting a [[https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html][process]]. > > Processes are synchronous or asynchronous. > > Three primitives exist in Emacs for making processes: > > 1. make-process (asynchronous) > 2. call-process (synchronous) > 3. call-process-region (synchronous) > > There exist several convenience wrappers for these. AFAIK, everything reduces to these three primitives. For example, =async-shell-command= runs =call-process= and prevents blocking by appending a "&" to the command which tells the shell to run the command in the background and return control to the terminal. This background-foreground distinction is called "job control". > > Output from a process typically goes to a buffer. This may be changed and instead handle output with a filter function. =call-process= has an option to directly send output to a file. > I've reached the same conclusion. As I wanted one simple implementation, I was thinking about using 'make-process', but, as we have to deal with the output manually and use a process sentinel, and, as offering a REPL to the user is nice, my personal pick is to use a plain shell with comint. > Subprocesses inherent the =default-directory= and the environment from Emacs. The environment may be changed using =process-environment=. > One wonderful thing about using =default-directory= is that, if done right, tramp will automatically handle executions on one or more remote computers. > There are two types of asynchronous connections: "pty" ("pseudoterminal") and "pipe". The main difference is that "pty" provides a terminal-like connection which allows for things like job control (=C-c=, =C-z=, etc.). I'm personally ignoring the difference between pty and pipe: as I don't use the terminal features, it doesn't really matter to me. > ... > I'm not sure how we could make a persistent, synchronous process. Persistence is achieved, currently, by a process buffer. Is there another way persistence may be achieved? If I understand correcly, "persistent" here means to preserve some state between executions. They are definitely other way to preserve and provide a state. If you focus on processes only, it's probably OK. But, if you use threads or some other kinds of asynchronicity, that might be too limiting to require a buffer and a process. > Of course, this ignores whether a persistent, synchronous process is even desirable. Given reliable asynchronous execution with persistence, I can't think of reason why someone would prefer a blocking operation. > I'm not so sure. If the success of an execution is really important, and, intrinsically unreliable (like a network connection), and, it only takes a few seconds, I'll probably choose synchronous execution, just to see that everything is OK, before moving to my next task. > ... > Attached are two files, ob-blub.el and ob-blub-test.org. Download both to the same directory. Run the first block in ob-blub-test.org. This imports ob-blub, loads it into Babel, and sets up blub to be whatever =shell-file-name= is (for example, bash). If you want to try Python or Ruby, comment out the shell configuration, uncomment the Python or Ruby implementations, and evaluate the block again. Hopefully ob-blub.el is documented sufficiently for you to experiment. Thanks for sharing. I didn't have time to try it yet. But it looks like you could use the 'execute-with' keyword that my patchs provide. That way, you may redirect evaluation to use your "blub" implementation, and still use the real language name for your code blocks (that way, org knows how to fontify them, edit them, etc). And, you will not have to manually modify and reevaluate your elisp when switching languages. > The blub implementation has the same shortcomings, at least for shells, as the current shell implementation. It has a few ideas, such as everything being asynchronous and completely removing the prompt, that may prove useful for improving Babel generally. The blub implementation is also simpler than related parts of Babel and may be useful for figuring out ways to solve the currently known shortcomings. If you run into an error during execution, you will need to call (setq my-org-babel-comint--async-uuid nil). I'll try it. Thanks, > The challenge I've found with Babel is figuring out how to make the changes. My current approach is to address bugs and to make changes that move us toward something like the ob-blub implementation. I wonder if it might help to discuss the core ideas and use a minimal reference implementation that serves as a guide for the actual changes we make. > > Curious to hear other people's thoughts! One think that comes to mind is to check that our testsuite covers everything we might break. If 'execute-with' gets added to org (from my patchs), that may provide an easy way to test/compare several execution engines. I'm thinking about modifying my patch to provide a 'feedbacks-with' keyword so that we can also test/compare different ways to report the execution status and outcome. Bruno