Re: [PATCH] Async evaluation in ob-shell

[Matt <matt@excalamus.com>]

[-- Attachment #1: Type: text/plain, Size: 1387 bytes --]

I've attached two patches which replace the previous.

I found cleaning the output was dramatically helped by calling `buffer-substring-no-properties' instead of `buffer-substring' in `org-babel-comint-async-filter'.  I'm not sure why `buffer-substring' was originally used.  `make test' shows no failures, so I assume it doesn't make a difference...?

 ---- On Tue, 07 Feb 2023 06:40:51 -0500  Ihor Radchenko  wrote --- 
 > That's likely because of the same reasons why prompt did not get cleaned
 > in synchronous blocks earlier. See `org-babel-comint-with-output'.

That, my friend, was a wild ride.

I'm curious about people's feelings on `org-babel-comint-with-output'.

Personally, I get the heebie-jeebies.  I can't shake feeling that there's a better way, especially since `org-babel-comint-async-filter' achieves similar ends.  My hunch is that other Babel languages may want async and that now would be a good time to consolidate the common functionalities of `org-babel-comint-with-output' and  `org-babel-comint-async-filter' .   Maybe even unify the API.  So far, `org-babel-comint-with-output' touches 9 languages and `org-babel-comint-async-filter' appears to touch 2 (soon to be 3).   I suspect those numbers will only grow.

I also can't shake the feeling that I might become ob-comint maintainer at some point...(not yet!).  I'm curious what people's thoughts are.





[-- Attachment #2: 01-ob-shell-remove-properties-from-callback-string.patch --]
[-- Type: application/octet-stream, Size: 997 bytes --]

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 54bf5127e..679757735 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -233,7 +233,7 @@ STRING contains the output originally inserted into the comint buffer."
       (goto-char (point-min))
       (while (re-search-forward indicator nil t)
 	;; update dangling
-	(setq new-dangling (buffer-substring (point) (point-max)))
+	(setq new-dangling (buffer-substring-no-properties (point) (point-max)))
 	(cond ((equal (match-string 1) "end")
 	       ;; save UUID for insertion later
 	       (push (match-string 2) uuid-list))
@@ -271,7 +271,7 @@ STRING contains the output originally inserted into the comint buffer."
 	  (when (equal (match-string 1) "end")
 	    (let* ((uuid (match-string-no-properties 2))
 		   (res-str-raw
-		    (buffer-substring
+		    (buffer-substring-no-properties
 		     ;; move point to beginning of indicator
                      (- (match-beginning 0) 1)
 		     ;; find the matching start indicator

[-- Attachment #3: 02-ob-shell-async-non-file.patch --]
[-- Type: application/octet-stream, Size: 2022 bytes --]

diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 9e7b45a89..23cb9683b 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -269,12 +269,18 @@ var of the same value."
 	    (set-marker comint-last-output-start (point))
 	    (get-buffer (current-buffer)))))))
 
+(defconst ob-shell-async-indicator "echo 'ob_comint_async_shell_%s_%s'")
+
+(defun ob-shell-async-chunk-callback (string)
+  (replace-regexp-in-string (concat org-babel-sh-prompt "*") "" string))
+
 (defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
   "Pass BODY to the Shell process in BUFFER.
 If RESULT-TYPE equals `output' then return a list of the outputs
 of the statements in BODY, if RESULT-TYPE equals `value' then
 return the value of the last statement in BODY."
   (let* ((shebang (cdr (assq :shebang params)))
+         (async (org-babel-comint-use-async params))
 	 (results-params (cdr (assq :result-params params)))
 	 (value-is-exit-status
 	  (or (and
@@ -305,6 +311,23 @@ return the value of the last statement in BODY."
                           (list shell-command-switch
                                 (concat (file-local-name script-file)  " " cmdline)))))
 		(buffer-string))))
+           (async
+            (let ((uuid (org-id-uuid)))
+              (org-babel-comint-async-register
+               session
+               (current-buffer)
+               "ob_comint_async_shell_\\(.+?\\)_\\(.+\\)"
+               'ob-shell-async-chunk-callback
+               nil)
+              (org-babel-comint-async-delete-dangling-and-eval
+                  session
+                (insert (format ob-shell-async-indicator "start" uuid))
+                (comint-send-input nil t)
+                (insert (org-trim body))
+                (comint-send-input nil t)
+                (insert (format ob-shell-async-indicator "end" uuid))
+                (comint-send-input nil t))
+              uuid))
 	   (session			; session evaluation
 	    (mapconcat
 	     #'org-babel-sh-strip-weird-long-prompt