From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id UNK5AL535GMkQgEAbAwnHQ (envelope-from ) for ; Thu, 09 Feb 2023 05:34:06 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id +KJ+O7135GPcPwAAG6o9tA (envelope-from ) for ; Thu, 09 Feb 2023 05:34:05 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8C5B32DDBB for ; Thu, 9 Feb 2023 05:34:05 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pPycc-0002V3-VW; Wed, 08 Feb 2023 23:33:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pPycb-0002Uf-OV for emacs-orgmode@gnu.org; Wed, 08 Feb 2023 23:33:21 -0500 Received: from sender4-op-o12.zoho.com ([136.143.188.12]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pPycZ-0006mM-Od for emacs-orgmode@gnu.org; Wed, 08 Feb 2023 23:33:21 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1675917194; cv=none; d=zohomail.com; s=zohoarc; b=c3Y3+CyDx9VTfwLiy9WQB34hBLVPNIR88rszTL/Uefcz0ajNnWg4xuPfwboM8MVqIMXcdUMt44FgVFb7wW6l7zasqVCBT8DWBGOrlXtE0q8jESG8rqJuzLBQATi9Z0qzhj88VIC3tSibSSUvw30JMfISlPfuqx6wMb9uJxb0TiY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1675917194; h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=9ocJPiJW/Lda7S8GooJO5vWvFai+XI6t/xHXo/+BwKY=; b=FU2gUrZW4EpRptYaY7iR9Y5zXyJh8/2ybnUYG9OtwitwBEGw8OD8Iw8eUIQWn4F5kkL5btGuIMuRxxeE85LVHulSkY02LEWbHOo805trWrXc7C37B8G7+UWvXUeOYK4SkkkcuBDoDCSyf59zJWD34dRDuWWn4T/41ELJfEnIi9s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=excalamus.com; spf=pass smtp.mailfrom=matt@excalamus.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1675917194; s=zmail; d=excalamus.com; i=matt@excalamus.com; h=Date:Date:From:From:To:To:Cc:Cc:Message-Id:Message-Id:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Reply-To; bh=9ocJPiJW/Lda7S8GooJO5vWvFai+XI6t/xHXo/+BwKY=; b=NpZi9qwlY6YzPLpGat8CuT2/sC9k1R9Iph1cBjg8udos1v2dg7wnm4btY8B3Ua0V 9wgRDgZEhqFOi8QAHL+oZnOLORfTs7egomkpK0e5LvOQKfXhk2qQde9zJJE/ovorfBf 4YtOR6qHYjy9/iinljJHmMRfFchqnA7BcvUYqbyQ= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1675917193221687.2099932345619; Wed, 8 Feb 2023 20:33:13 -0800 (PST) Date: Wed, 08 Feb 2023 23:33:13 -0500 From: Matt To: "Ihor Radchenko" Cc: "emacs-orgmode" Message-Id: <1863472efe9.10fdd5ba4258906.5972264927968042941@excalamus.com> In-Reply-To: <87v8kd8zzw.fsf@localhost> References: <186283d230a.129f5feb61660123.3289004102603503414@excalamus.com> <87v8kd8zzw.fsf@localhost> Subject: Re: [PATCH] Async evaluation in ob-shell MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_844366_51253648.1675917193193" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Zoho-Virus-Status: 1 Received-SPF: pass client-ip=136.143.188.12; envelope-from=matt@excalamus.com; helo=sender4-op-o12.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN ARC-Seal: i=2; s=key1; d=yhetil.org; t=1675917245; a=rsa-sha256; cv=pass; b=L4u92MW2wSjjsmnpMdAJV6a8SBvNGLUj7ZNNncMg70C94zM1PTnuMYeHSe0D7WJDVfmXiX WDVn6lm611ve7/wfRLO64DE6on7B7pFLpmOJgaS8uzgNnwck5dxyJlaycJSfPqWGr588gR 5iXdiKOIr/asu1jOH0/LcV5Jp9MhGC/582yOcRlH5d+azLvzinb1p+efhuxQ+KI+KU/vCv IDkbjsc8H+FyIPWiaPktcZkq1znopGrwltdG58VSde7kONpuFTtfvR9I9lrvQVfkWGds2f WvXpCbJm/RdDO7IZkAbV5zFCp4vhkFAGAIrHwkPnbpSx/ZxlEaNDJ7DZ/7sU7w== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=NpZi9qwl; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=none; arc=pass ("zohomail.com:s=zohoarc:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1675917245; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=9ocJPiJW/Lda7S8GooJO5vWvFai+XI6t/xHXo/+BwKY=; b=nOjbOECJ8dVOql2Q29D1yyx2DqIA+rIyJxhTJ2EaBtHXaSVvmM+86xAFgPZ64HYMPc3hX9 H0gEay2nuOMGGKoPIkJk1TpZB49q7DC4oGppdXXX2cZTP63tm7xgaMOd8qZOpvyKUyujtV cLa8k07sfWPniYtbz+Z7Mv1tcL4bmUNOQUn9pSYMqn+lE+5CL7qUS7uoLn+IGi/rOh+yMZ iXGV3KXzsPszJQjzPra6N92yYEbOLZ4hnz1r29EasBHPp1PBcpHCP63aOSBEPkkReY9zyZ JLlu8kVnD5pCL5W2u1yb6j/FNHU7jG/PyuPIWRRPcl+8j6aOAD4axpr4cYQ8mQ== X-Migadu-Queue-Id: 8C5B32DDBB X-Migadu-Scanner: scn1.migadu.com Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=NpZi9qwl; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=none; arc=pass ("zohomail.com:s=zohoarc:i=1") X-Migadu-Spam-Score: -6.82 X-Spam-Score: -6.82 X-TUID: SrvZOQSBBTXm ------=_Part_844366_51253648.1675917193193 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit I've attached two patches which replace the previous. I found cleaning the output was dramatically helped by calling `buffer-substring-no-properties' instead of `buffer-substring' in `org-babel-comint-async-filter'. I'm not sure why `buffer-substring' was originally used. `make test' shows no failures, so I assume it doesn't make a difference...? ---- On Tue, 07 Feb 2023 06:40:51 -0500 Ihor Radchenko wrote --- > That's likely because of the same reasons why prompt did not get cleaned > in synchronous blocks earlier. See `org-babel-comint-with-output'. That, my friend, was a wild ride. I'm curious about people's feelings on `org-babel-comint-with-output'. Personally, I get the heebie-jeebies. I can't shake feeling that there's a better way, especially since `org-babel-comint-async-filter' achieves similar ends. My hunch is that other Babel languages may want async and that now would be a good time to consolidate the common functionalities of `org-babel-comint-with-output' and `org-babel-comint-async-filter' . Maybe even unify the API. So far, `org-babel-comint-with-output' touches 9 languages and `org-babel-comint-async-filter' appears to touch 2 (soon to be 3). I suspect those numbers will only grow. I also can't shake the feeling that I might become ob-comint maintainer at some point...(not yet!). I'm curious what people's thoughts are. ------=_Part_844366_51253648.1675917193193 Content-Type: application/octet-stream; name=01-ob-shell-remove-properties-from-callback-string.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 138952099931940040 Content-Disposition: attachment; filename=01-ob-shell-remove-properties-from-callback-string.patch diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 54bf5127e..679757735 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -233,7 +233,7 @@ STRING contains the output originally inserted into the comint buffer." (goto-char (point-min)) (while (re-search-forward indicator nil t) ;; update dangling - (setq new-dangling (buffer-substring (point) (point-max))) + (setq new-dangling (buffer-substring-no-properties (point) (point-max))) (cond ((equal (match-string 1) "end") ;; save UUID for insertion later (push (match-string 2) uuid-list)) @@ -271,7 +271,7 @@ STRING contains the output originally inserted into the comint buffer." (when (equal (match-string 1) "end") (let* ((uuid (match-string-no-properties 2)) (res-str-raw - (buffer-substring + (buffer-substring-no-properties ;; move point to beginning of indicator (- (match-beginning 0) 1) ;; find the matching start indicator ------=_Part_844366_51253648.1675917193193 Content-Type: application/octet-stream; name=02-ob-shell-async-non-file.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 138952099932000000 Content-Disposition: attachment; filename=02-ob-shell-async-non-file.patch diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el index 9e7b45a89..23cb9683b 100644 --- a/lisp/ob-shell.el +++ b/lisp/ob-shell.el @@ -269,12 +269,18 @@ var of the same value." (set-marker comint-last-output-start (point)) (get-buffer (current-buffer))))))) +(defconst ob-shell-async-indicator "echo 'ob_comint_async_shell_%s_%s'") + +(defun ob-shell-async-chunk-callback (string) + (replace-regexp-in-string (concat org-babel-sh-prompt "*") "" string)) + (defun org-babel-sh-evaluate (session body &optional params stdin cmdline) "Pass BODY to the Shell process in BUFFER. If RESULT-TYPE equals `output' then return a list of the outputs of the statements in BODY, if RESULT-TYPE equals `value' then return the value of the last statement in BODY." (let* ((shebang (cdr (assq :shebang params))) + (async (org-babel-comint-use-async params)) (results-params (cdr (assq :result-params params))) (value-is-exit-status (or (and @@ -305,6 +311,23 @@ return the value of the last statement in BODY." (list shell-command-switch (concat (file-local-name script-file) " " cmdline))))) (buffer-string)))) + (async + (let ((uuid (org-id-uuid))) + (org-babel-comint-async-register + session + (current-buffer) + "ob_comint_async_shell_\\(.+?\\)_\\(.+\\)" + 'ob-shell-async-chunk-callback + nil) + (org-babel-comint-async-delete-dangling-and-eval + session + (insert (format ob-shell-async-indicator "start" uuid)) + (comint-send-input nil t) + (insert (org-trim body)) + (comint-send-input nil t) + (insert (format ob-shell-async-indicator "end" uuid)) + (comint-send-input nil t)) + uuid)) (session ; session evaluation (mapconcat #'org-babel-sh-strip-weird-long-prompt ------=_Part_844366_51253648.1675917193193--