We have asynchronous sessions, why have anything else?

We have asynchronous sessions, why have anything else?

[Ivar Fredholm]

[-- Attachment #1: Type: text/plain, Size: 514 bytes --]

A session-less block can be executed by starting a session with a special name (say "*none") which always gets killed after block execution is completed. For interpreter-less languages, we could use the shell as an interpreter (for instance, if we wanted to execute C, we could just start a shell, and send it the gcc command to compile and execute). Would this not cut down the amount of code that needs to be maintained and uniformize the existing code?

Sent with [Proton Mail](https://proton.me/) secure email.

[-- Attachment #2: Type: text/html, Size: 1035 bytes --]

Re: We have asynchronous sessions, why have anything else?

[Ihor Radchenko]

Ivar Fredholm <freddyholms@protonmail.com> writes:

> A session-less block can be executed by starting a session with a special name (say "*none") which always gets killed after block execution is completed. For interpreter-less languages, we could use the shell as an interpreter (for instance, if we wanted to execute C, we could just start a shell, and send it the gcc command to compile and execute). Would this not cut down the amount of code that needs to be maintained and uniformize the existing code?

Feel free to compare ob-eval.el and ob-comint.el. Their functionality is
not equivalent. In particular ob-eval.el has a better handling of
errors.

If you find a way to unify the two without loosing the functionality, it
will be welcome.

Best,
Ihor

Re: We have asynchronous sessions, why have anything else?

[Ivar Fredholm]

Hi, Ihor,
I believe the two could be unified if we expand the functionality of the async filter to look for 'exception' tags. Then each language implementation must only put the org-babel src block in a try-except type construction and put the error message into the except block. The async filter will then find the error information and report it back to the org-buffer. We could then emulate the synchronous session functionality by adding a filter to the async-filter which blocks until an 'end' or 'exception' tag is detected. This way, all four combinations of session/session-less and synchronous/asynchronous header arguments could be handled by the asynchronous code alone.
Best, Guacho




Sent with Proton Mail secure email.

------- Original Message -------
On Saturday, June 25th, 2022 at 10:28 PM, Ihor Radchenko <yantar92@gmail.com> wrote:


> Ivar Fredholm freddyholms@protonmail.com writes:
>
> > A session-less block can be executed by starting a session with a special name (say "*none") which always gets killed after block execution is completed. For interpreter-less languages, we could use the shell as an interpreter (for instance, if we wanted to execute C, we could just start a shell, and send it the gcc command to compile and execute). Would this not cut down the amount of code that needs to be maintained and uniformize the existing code?
>
>
> Feel free to compare ob-eval.el and ob-comint.el. Their functionality is
> not equivalent. In particular ob-eval.el has a better handling of
> errors.
>
> If you find a way to unify the two without loosing the functionality, it
> will be welcome.
>
> Best,
> Ihor

Re: We have asynchronous sessions, why have anything else?

[Ihor Radchenko]

Ivar Fredholm <freddyholms@protonmail.com> writes:

> I believe the two could be unified if we expand the functionality of
> the async filter to look for 'exception' tags. Then each language
> implementation must only put the org-babel src block in a try-except
> type construction and put the error message into the except block.

I am not even sure if all the babel backends support try-except.
Think about ob-gnuplot or, say, ob-latex.

Best,
Ihor

Re: We have asynchronous sessions, why have anything else?

[Tom Gillespie]

> I am not even sure if all the babel backends support try-except.
> Think about ob-gnuplot or, say, ob-latex.

Indeed many do not. Defining some standard "features"
for org babel language implementations is something that
is definitely of interest so that we can provide clear interfaces
for things like stdio, error handling, return values, async,
file output, remote execution, sessions, return value caching,
module discovery/tangling, execution from file vs stdin, execution
without a file system path, runtime environment specification,
and much more. However, at the moment there is only a preliminary
survey of a subset of these that was put together by Ian Martins.

https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html

> the two could be unified if we expand the functionality of the async filter

While this might be possible, I would definitely hold off on this because
the changes in semantics absolutely will break many users' blocks. We
barely knew what the impact of changing the default return value for shell
blocks would be.

I absolutely look forward to the day when this can be done safely and
with confidence, but I think we need a much stronger handle on babel
interfaces in general before such a change could even be considered.

At the moment each ob lang impl pretty much has to be considered
to be completely unique, even if the text looks like bash for e.g.
shell, comint, and screen. Users are known to rely on undocumented
quirks of the ob lang impls that can differ wildly in their semantics.

Best!
Tom

Re: We have asynchronous sessions, why have anything else?

[Tim Cross]

Tom Gillespie <tgbugs@gmail.com> writes:

>> I am not even sure if all the babel backends support try-except.
>> Think about ob-gnuplot or, say, ob-latex.
>
> Indeed many do not. Defining some standard "features"
> for org babel language implementations is something that
> is definitely of interest so that we can provide clear interfaces
> for things like stdio, error handling, return values, async,
> file output, remote execution, sessions, return value caching,
> module discovery/tangling, execution from file vs stdin, execution
> without a file system path, runtime environment specification,
> and much more. However, at the moment there is only a preliminary
> survey of a subset of these that was put together by Ian Martins.
>
> https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html
>
>> the two could be unified if we expand the functionality of the async filter
>
> While this might be possible, I would definitely hold off on this because
> the changes in semantics absolutely will break many users' blocks. We
> barely knew what the impact of changing the default return value for shell
> blocks would be.
>
> I absolutely look forward to the day when this can be done safely and
> with confidence, but I think we need a much stronger handle on babel
> interfaces in general before such a change could even be considered.
>
> At the moment each ob lang impl pretty much has to be considered
> to be completely unique, even if the text looks like bash for e.g.
> shell, comint, and screen. Users are known to rely on undocumented
> quirks of the ob lang impls that can differ wildly in their semantics.
>

Well said Tom.

As you point out, there are numerous deficiencies with the current
implementation, despite the fact it all sort of works. To get the sort
of improvements and consistency users want, I suspect this needs more
than just small tweaks around the edges.

To some extent, I see the current babel implementation as similar to a
prototype. It has worked well and we have learnt a lot about what people
want to use it for and the type of functionality they are wanting and
what some of the core challenges are. Now comes the next step, which is
definitely non-trivial. We need to take all that knowledge and
consolidate it into a single model from which we can define the
interfaces and associated APIs. A big job which will take considerable
time.

Re: We have asynchronous sessions, why have anything else?

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 3327 bytes --]

The Jupyter project is one approach to this. It currently has dozens of
kernels for different languages, and new kernels can certainly be made. The
emacs-jupyter package provides one implementation of an interface. It is
complex, and relies on a compiled module for the zeromq message passing
library.

I am not advocating for this as the solution for org-babel, but it is an
interesting case study. You can even connect to remote kernels.

I use it a lot.

On Mon, Jun 27, 2022 at 8:56 PM Tim Cross <theophilusx@gmail.com> wrote:

>
> Tom Gillespie <tgbugs@gmail.com> writes:
>
> >> I am not even sure if all the babel backends support try-except.
> >> Think about ob-gnuplot or, say, ob-latex.
> >
> > Indeed many do not. Defining some standard "features"
> > for org babel language implementations is something that
> > is definitely of interest so that we can provide clear interfaces
> > for things like stdio, error handling, return values, async,
> > file output, remote execution, sessions, return value caching,
> > module discovery/tangling, execution from file vs stdin, execution
> > without a file system path, runtime environment specification,
> > and much more. However, at the moment there is only a preliminary
> > survey of a subset of these that was put together by Ian Martins.
> >
> > https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html
> >
> >> the two could be unified if we expand the functionality of the async
> filter
> >
> > While this might be possible, I would definitely hold off on this because
> > the changes in semantics absolutely will break many users' blocks. We
> > barely knew what the impact of changing the default return value for
> shell
> > blocks would be.
> >
> > I absolutely look forward to the day when this can be done safely and
> > with confidence, but I think we need a much stronger handle on babel
> > interfaces in general before such a change could even be considered.
> >
> > At the moment each ob lang impl pretty much has to be considered
> > to be completely unique, even if the text looks like bash for e.g.
> > shell, comint, and screen. Users are known to rely on undocumented
> > quirks of the ob lang impls that can differ wildly in their semantics.
> >
>
> Well said Tom.
>
> As you point out, there are numerous deficiencies with the current
> implementation, despite the fact it all sort of works. To get the sort
> of improvements and consistency users want, I suspect this needs more
> than just small tweaks around the edges.
>
> To some extent, I see the current babel implementation as similar to a
> prototype. It has worked well and we have learnt a lot about what people
> want to use it for and the type of functionality they are wanting and
> what some of the core challenges are. Now comes the next step, which is
> definitely non-trivial. We need to take all that knowledge and
> consolidate it into a single model from which we can define the
> interfaces and associated APIs. A big job which will take considerable
> time.
>
> --
John

-----------------------------------
Professor John Kitchin (he/him/his)
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
https://kitchingroup.cheme.cmu.edu
https://pointbreezepubs.gumroad.com/ pycse bookstore

[-- Attachment #2: Type: text/html, Size: 4533 bytes --]

Re: We have asynchronous sessions, why have anything else?

[Ivar Fredholm]

[-- Attachment #1: Type: text/plain, Size: 898 bytes --]

Hi Ihor, I have a prototype of what I mentioned earlier, at least for python. This supports asynchronous, synchronous, session, and session-less blocks. It's pretty messy but it helps to illustrate what I had in mind. Let me know what you think.




Sent with Proton Mail secure email.

------- Original Message -------
On Monday, June 27th, 2022 at 4:56 AM, Ihor Radchenko <yantar92@gmail.com> wrote:


> Ivar Fredholm freddyholms@protonmail.com writes:
>
> > I believe the two could be unified if we expand the functionality of
> > the async filter to look for 'exception' tags. Then each language
> > implementation must only put the org-babel src block in a try-except
> > type construction and put the error message into the except block.
>
>
> I am not even sure if all the babel backends support try-except.
> Think about ob-gnuplot or, say, ob-latex.
>
> Best,
> Ihor

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: ob-session4.el --]
[-- Type: text/x-emacs-lisp; name=ob-session4.el, Size: 28551 bytes --]

(defun eval-file (file)
  (with-temp-buffer
    (insert-file-contents file)
    (eval-buffer)))
(eval-file "~/new_org/org-mode/lisp/ob-core.el")
(eval-file "~/new_org/org-mode/lisp/ob-comint.el")
(eval-file "~/new_org/org-mode/lisp/ob-python.el")
(eval-file "~/new_org/org-mode/lisp/org-attach.el")

(require 'subr-x)
(require 'eieio)
(require 'cl-lib)


(defvar org-babel-session-list nil
  "List of all sessions")

(defvar org-babel-shell-buffers nil
  "List of interpreter buffers. This gets garbage collected every
  time a source block is run. Any process-less buffer gets deleted.")


(defclass latch ()
  ((process :initform (start-process "latch" nil nil))
   (value :initform nil))
  :documentation "A blocking latch that can be used any number of times.")

(cl-defmethod wait ((latch latch) &optional timeout)
  "Blocking wait on LATCH for a corresponding `notify', returning
the value passed by the notification. Wait at most TIMEOUT
seconds (float allowed), returning nil if the timeout was reached
with no input. The Emacs display will not update during this
period but I/O and timers will continue to run."
  (accept-process-output (slot-value latch 'process) timeout)
  (slot-value latch 'value))

(cl-defmethod notify ((latch latch) &optional value)
  "Release all execution contexts waiting on LATCH, passing them VALUE."
  (setf (slot-value latch 'value) value)
  (process-send-string (slot-value latch 'process) "\n"))

(cl-defmethod destroy ((latch latch))
  "Destroy a latch, since they can't be fully memory managed."
  (ignore-errors
    (delete-process (slot-value latch 'process))))

(defun make-latch ()
  "Make a latch which can be used any number of times. It must be
`destroy'ed when no longer used, because the underlying process
will not be garbage collected."
  (make-instance 'latch))

(defun destroy-all-latches ()
  "Destroy all known latches."
  (cl-loop for process in (process-list)
           when (string-match-p "latch\\(<[0-9]+>\\)?" (process-name process))
           do (delete-process process)))

;; Code for the administration of sessions and their processes.

(defclass org-babel-session ()
  ((name :initarg :name
	 :documentation "Name of the session, should be unique on
	 a per-language basis or 'none' if the associated source
	 block is session-less.")
   (language :initarg :language
	     :documentation "The language for the source block
	     associated to this session.")
   (is-none :initform nil
	    :documentation "Indicates whether we should delete
   the session once it has finished executing its source block.")
   (unique-id :initform ""
	      :documentation "This is the unique process
   identifier for the session.")
   (process :initform nil
	    :documentation "The interpreter or shell for the
   session.")
   (buffer :initform nil
	   :documentation "The buffer associated with process")
   (ready-for-input :initform nil
		    :documentation "A variable indicating whether
   the interpreter is ready to accept more input.")
   (input-latch :initform nil
		:documentation "A latch that blocks execution
   until the interpreter has finished processing the current
   input. This is used to emulate synchronous blocks using the
   asynchronous process filter.")
   (indicator-regexp :initform nil
		     :documentation "Holds the indicator regexp
		     that the async filter will look for in the
		     comint output. The user must define this on
		     a per-language basis by defining a
		     `org-babel-async-indicator:LANG' constant.")
   (org-buffers :initform nil
		:documentation "A list of buffers to look through
		when searching for a place to insert the results
		of a source block.")
   (async :initform nil
	  :documentation "Tell the process whether to notify its
	  latch when ready for input or not")
   (current-dangling :initform ""
		     :documentation "Holds the most recent text
		     provided by the interpreter in case of
		     output buffering."))
  "To implement concrete classes of this class, one must first
  define: a session initializer which launches an interpreter,
  and a method to asynchronously send input to said
  interpreter. Language specific pre-processing of the input code
  and post-processing of the results must be implemented separately.")

;; Making sessions

(cl-defmethod initialize-instance :after ((sess org-babel-session) &rest _)
  (let* ((name (oref sess name))
	 (language (oref sess language))
	 (is-none (if (string= name "none") t))
	 (unique-id (if (string= name "none")
			;; Generate a unique identifier e.g
			;; python81b509c46a9c502da6a7a86299994ca0
			(concat language (md5 (number-to-string (random 100000000))))
		      ;; Otherwise use the session name
		      (concat language name)))
	 (buffer (concat unique-id "-buffer"))
	 ;; For each language, one must implement a
	 ;; org-babel-interpreter-cmd:LANG function which does the
	 ;; necessary verifications and returns a string with the
	 ;; command to run the interpreter in a make-comint-in-buffer
	 ;; function.	 
	 (process (make-comint-in-buffer (concat unique-id "-session")
					 buffer
					 (org-babel-interpreter-cmd sess))))
    ;; Add a sentinel to kill the buffer once the process is killed without interfering with the filter
    (set-process-sentinel  (get-buffer-process process)
			   `(lambda (proc event)						  
    			      (when (string= event "killed\n")
				(setq org-babel-shell-buffers (remove ,buffer org-babel-shell-buffers))
    				(kill-buffer ,buffer))))						  

    ;; Add the buffer to `org-babel-shell-buffers' so it is garbage
    ;; collected after it is killed.
    (setq org-babel-shell-buffers (push buffer org-babel-shell-buffers))
    (setf (slot-value sess 'is-none) is-none)
    (setf (slot-value sess 'unique-id) unique-id)
    (setf (slot-value sess 'process) process)
    (setf (slot-value sess 'buffer) buffer)
    (setf (slot-value sess 'input-latch) (make-latch))))

(defun org-babel-fetch-session (session language &optional location)
  "Creates an org-babel-session object with name SESSION for the
  language LANGUAGE if it does not exist. If SESSION is none, a
  new session is created by default, and instructed to
  self-destruct after processing its input. To handle none
  sessions, if LOCATION is not nil, we return "
  (let* ((sess-id (concat language session)) ;unique id for session
	 (session-in-list (assoc sess-id org-babel-session-list))
	 (session-obj (if session-in-list
			  (cdr session-in-list)
			(funcall (intern (concat "org-babel-session:" language)) :name session))))
    ;; Do not save session-less processes so they are never found
    ;; during the lookup stage and thus we always get a fresh new
    ;; interpreter.
    (unless (string= session "none")
      (push (cons sess-id session-obj) org-babel-session-list))
    session-obj))

;; Destroying sessions

(cl-defmethod org-babel-kill-session-instance ((sess org-babel-session))
  "Destroys the instance by killing its process, buffer, and
  latch, as well as deleting its other fields. This function also
  deletes session from `org-babel-session-list'"
  (if (slot-value sess 'input-latch)
      (destroy (slot-value sess 'input-latch)))
  (unless (slot-value sess 'is-none)
    (setq org-babel-session-list (remove (cons (slot-value sess 'unique-id) sess) org-babel-session-list)))
  (with-current-buffer (slot-value sess 'buffer)
    (let (kill-buffer-hook kill-buffer-query-functions)
      (kill-process (slot-value sess 'process)))))

;; The following methods need to be implemented on a per-language
;; basis.

(cl-defmethod org-babel-session-execute ((sess org-babel-session) body params async)
  "Execute BODY synchronously or asynchronously (depending on
  ASYNC) using SESS's process. For this one must implement a
  concrete subclass of `org-babel-session' with a method that
  sends input asynchronously to process. This is an abstract
  method and must be implemented on a per-language basis."
  (error "No `org-babel-session-execute' method for this language."))

(cl-defmethod org-babel-interpreter-cmd ((sess org-babel-session))
  "Perform necessary checks and return the command to start the
asynchronous interpreter."
  (error "Implement `org-babel-interpreter-cmd' for language %s" (slot-value sess 'language)))

(cl-defmethod org-babel-table-or-string ((sess org-babel-session) string)
  "Convert RESULTS into an appropriate elisp value.
If the results look like a list or tuple, then convert them into
an Emacs-lisp table, otherwise return the results as a string."
  (error "Implement `org-babel-table-or-string' for language %s" (slot-value sess 'language)))

(cl-defmethod org-babel-place-exceptions ((session org-babel-session) exception-text info lang params result-params)
  "Process the exception test according to the langauge. By
default, it just replaces any previous result under the source
block."
  (message "Using default exception insertion, implement
  `org-babel-interpreter-cmd' for language %s" (slot-value session
							   'language))
  (org-babel-remove-result)
  (org-babel-insert-result exception-text)
  )

(defmacro goto-result-with-tag (location in org-buffers &rest body)
  "Go to the source block whose result is tagged with LOCATION in
some buffer in ORG-BUFFERS and fetch info, language, params,
result-params, and session variables from the header before
executing body."
  (declare (indent 1) (debug t))
  `(cl-loop for buf in ,org-buffers
	    until
	    ;; In each buffer...	   
	    (with-current-buffer buf
	      (save-excursion
		(goto-char (point-min))
		;; search for LOCATION
		(when (search-forward ,location nil t)
		  (org-babel-previous-src-block)
		  ;; Remove previous results
		  (org-babel-remove-result) ;; Replace this with a macro?
		  (let* ((info (org-babel-get-src-block-info))
			 (lang (nth 0 info))
			 (params (nth 2 info))
			 (result-params
			  (cdr (assq :result-params params))))
		    ,@body)
		  t)))))

(defun org-babel-maybe-get-cache (info params arg)
  "Return the cached result of the block with info INFO if it
exists."
  (let* ((cache (let ((c (cdr (assq :cache params))))
		  (and (not arg) c (string= "yes" c))))
	 (new-hash (and cache (org-babel-sha1-hash info :eval)))
	 (old-hash (and cache (org-babel-current-result-hash)))
	 (current-cache (and new-hash (equal new-hash old-hash))))    
    (if current-cache
	(save-excursion
	  (goto-char (org-babel-where-is-src-block-result nil info))
	  (forward-line)
	  (skip-chars-forward " \t")
	  (let ((result (org-babel-read-result)))
	    (message (replace-regexp-in-string "%" "%%" (format "%S" result)))
	    result)))))

(defun org-babel-set-default-dir (mkdirp dir)
  "Change default directories to DIR if instructed by header
variables. If MKDIRP then make a new directory."
  (cond
   ((not dir) default-directory)
   ((member mkdirp '("no" "nil" nil))
    (file-name-as-directory (expand-file-name dir)))
   (t
    (let ((d (file-name-as-directory (expand-file-name dir))))
      (make-directory d 'parents)
      d))))

(defun org-babel-run-src-blk (info params)
  "Run the current source block by expanding the code and
registering the session to execute. We wait if no asynchronous
execution is specified and then insert the results of the
computation."
  ;; Parse the header arguments and expand noweb references in
  ;; the block.  After this, `body' must be given to a
  ;; language-specific input pre-processor before it can be
  ;; executed.
  (let* ((lang (nth 0 info))
	 (result-params (cdr (assq :result-params params)))
	 (async (if (assq :async params) t))
	 (body (org-babel--expand-body info))
	 (session (org-babel-fetch-session (cdr (assoc :session params)) lang))
	 (dir (cdr (assq :dir params)))
	 (mkdirp (cdr (assq :mkdirp params)))
	 (default-directory (org-babel-set-default-dir mkdirp dir))
	 (indicator (slot-value session 'indicator-regexp))
	 result)	    
    ;; Let the process know whether it should notify its latch
    ;; once it has finished executing
    (setf (slot-value session 'async) async)
    ;; Reset current-dangling in case errors in previous
    ;; executions polluted the output
    (setf (slot-value session 'current-dangling) "")
    ;; We must register the session to have its own
    ;; async-filter in order to handle session-less blocks.
    (org-babel-comint-async-register session (current-buffer) indicator arg)	    
    ;; Now compute the result, the user has to implement a
    ;; language specific org-babel-session-execute function
    (setq result (org-babel-session-execute session body params))
    ;; Inform user we are executing the code
    (message "executing %s code block%s..."
	     (capitalize lang)
	     (let ((name (nth 4 info)))
	       (if name (format " (%s)" name) "")))
    ;; Insert the tag for the result
    (org-babel-insert-result
     result result-params info nil lang)
    (run-hooks 'org-babel-after-execute-hook)
    ;; Block execution if not asynchronous
    (unless async
      (wait (slot-value session 'input-latch)))
    result))

(defun org-babel-session-execute-src-block (&optional arg info params)
  "Execute the current source code block.
Insert the results of execution into the buffer.  Source code
execution and the collection and formatting of results can be
controlled through a variety of header arguments.

With prefix argument ARG, force re-execution even if an existing
result cached in the buffer would otherwise have been returned.

Optionally supply a value for INFO in the form returned by
`org-babel-get-src-block-info'.

Optionally supply a value for PARAMS which will be merged with
the header arguments specified at the front of the source code
block."
  (interactive)
  ;; Update the current source block location
  (let* ((org-babel-current-src-block-location
	  (or org-babel-current-src-block-location
	      (nth 5 info)
	      (org-babel-where-is-src-block-head)))
	 ;; Extract header information
	 (info (if info (copy-tree info) (org-babel-get-src-block-info))))
    ;; Merge PARAMS with INFO before considering source block
    ;; evaluation since both could disagree.
    (cl-callf org-babel-merge-params (nth 2 info) params)
    (when (org-babel-check-evaluate info)
      (cl-callf org-babel-process-params (nth 2 info))      
      (if-let* ((params (nth 2 info))
		(result (org-babel-maybe-get-cache info params arg)))
	  ;; If the result was already cached, return it
	  result
	;; Otherwise, we must execute the source block
	(org-babel-run-src-blk info params)))))

(defun org-babel-comint-async-register
    (session org-buffer indicator-regexp arg)
  "Set local org-babel-comint-async variables in SESSION.
ORG-BUFFER is added to `org-babel-comint-async-buffers' if not
present.  `org-babel-comint-async-indicator',
`org-babel-comint-async-chunk-callback', and
`org-babel-comint-async-file-callback' are set to
INDICATOR-REGEXP, CHUNK-CALLBACK, and FILE-CALLBACK
respectively."
  (org-babel-comint-in-buffer (slot-value session 'buffer)       
    (unless (memq org-buffer (slot-value session 'org-buffers))
      (setf (slot-value session 'org-buffers)
	    (cons org-buffer (slot-value session 'org-buffers))))
    (add-hook 'comint-output-filter-functions
	      `(lambda (string)
		 (org-babel-comint-async-filter string ,session ,arg)) nil t)))


(defun org-babel-comint-async-filter (string session arg)
  "Captures Babel async output from comint buffer back to Org mode buffers.
This function is added as a hook to `comint-output-filter-functions'.
STRING contains the output originally inserted into the comint buffer."  
  ;; Remove outdated Org mode buffers  
  (setf (slot-value session 'org-buffers)
	(cl-loop for buf in (slot-value session 'org-buffers)
	         if (buffer-live-p buf)
	         collect buf))
  ;; Set local copies of global bookkeeping variables and get the current,
  ;; possibly incomplete output from the process into combined-string
  (filter-output-string string
			(slot-value session 'indicator-regexp)
			session
			(slot-value session 'org-buffers)
			arg))

(defun org-babel-scan-for-results
    (indicator uuid-list org-buffers end-token insertion-func arg)
  "Scan the comint buffer for INDICATOR pairs with uuid's in
UUID-LIST having the end token END-TOKEN (e.g. 'end' or
'exceptend') and insert the text in between into a buffer in
ORG-BUFFERS using the function INSERTION-FUNC, which should take
as arguments a string result to insert, a uuid location,
org-buffers, and the 'arg' variable passed in optionally to
org-babel-execute-src-block.
"  
  (when uuid-list
    ;; Search for results in the comint buffer
    (save-excursion
      (goto-char (point-max))
      (while uuid-list
	(re-search-backward indicator
			    (when (equal (match-string 1) end-token)
			      (let* ((uuid (match-string-no-properties 2))
				     (res-str-raw
				      (buffer-substring
				       ;; move point to beginning of indicator
				       (- (match-beginning 0) 1)
				       ;; find the matching start indicator
				       (cl-loop
					do (re-search-backward indicator)
					until (and (equal (match-string 1) "start")
						   (equal (match-string 2) uuid))
					finally return (+ 1 (match-end 0)))))
				     ;; Apply callback to clean up the result
				     (res-str (org-trim res-str-raw)))
				;; Insert results
				(goto-result-with-tag uuid in org-buffers
						      (funcall insertion-func
							       session
							       res-str
							       info
							       lang
							       params
							       result-params))
				;; Remove uuid from the list to search for
				(setq uuid-list (delete uuid uuid-list)))))))))

(defun org-babel-place-results (session result info lang params result-params)
  (let* ((file (and (member "file" result-params)
		    (cdr (assq :file params))))
	 (post (cdr (assq :post params))))
    ;; With the raw string result, we can start
    ;; post-process it and insert it so long as the
    ;; user did not silence the results.
    (if (member "none" result-params)
	(message "result silenced")
      ;; Transform result to lisp table if indicated by header arguments
      (setq result (org-babel-maybe-make-table result params result-params session))		     
      ;; If non-empty result and :file then write to :file.
      (when file		       
	;; Now return the file rather than the results
	(setq result (org-babel-write-result-to-file file result params result-params))
	(setq result-params (remove "file" result-params)))
      ;; Possibly perform post process provided its
      ;; appropriate.  Dynamically bind "*this*" to the
      ;; actual results of the block.
      (when post
	;; If the user specified :post, embed the result into the
	;; string given in said header argument.
	(setq result (org-babel-maybe-postprocess file result params post)))		     
      ;; Finally insert the results
      (org-babel-insert-result
       result result-params info nil lang))))

(defun filter-output-string (string indicator session org-buffers arg)
  "Captures Babel async output from comint buffer back to Org mode buffers.
This function is added as a hook to `comint-output-filter-functions'.
STRING contains the output originally inserted into the comint buffer."  
  (let* ((combined-string (concat (slot-value session 'current-dangling) string))
	 (new-dangling combined-string)
	 ;; Indicate whether or not we inserted a result to signal any blocked functions
	 ;; that they may resume
	 (resume-exec nil)
	 ;; list of UUID's matched by `org-babel-comint-async-indicator'
	 uuid-list
	 except-uuid-list)
    (with-temp-buffer
      (insert combined-string)
      (goto-char (point-min))
      (while (re-search-forward indicator nil t)
	;; update dangling
	(setq new-dangling (buffer-substring (point) (point-max)))
	(cond ((equal (match-string 1) "end")
	       ;; We finished executing
	       (setq resume-exec t)
	       ;; save UUID for insertion later
	       (push (match-string 2) uuid-list))
	      ((equal (match-string 1) "exceptend")
	       ;; We finished executing
	       (setq resume-exec t)
	       ;; save UUID for insertion later
	       (push (match-string 2) except-uuid-list))
	      ((equal (match-string 1) "file")
	       ;; We finished executing
	       (setq resume-exec t)
	       ;; insert results from tmp-file
	       (let* ((tmp-file (match-string 2))
		      (result (org-babel-eval-read-file tmp-file)))
		 (goto-result-with-tag tmp-file in org-buffers 
				       (org-babel-place-results session result info
								lang params result-params))))))
      ;; Truncate dangling to only the most recent output
      (when (> (length new-dangling) (length string))
	(setq new-dangling string)))
    (setf (slot-value session 'current-dangling) new-dangling)        
    (org-babel-scan-for-results indicator uuid-list        org-buffers "end"       'org-babel-place-result     arg)
    (org-babel-scan-for-results indicator except-uuid-list org-buffers "exceptend" 'org-babel-place-exceptions arg)
    (if resume-exec
	(progn
	  ;; Also notify that we are ready for new input if we are being waited on
	  (unless (slot-value session 'async)
	    (notify (slot-value session 'input-latch)))        	
	  ;; Since we are done, we can safely kill the none session in
	  ;; case it was created.
	  (if (slot-value session 'is-none)
	      (org-babel-kill-session-instance session))))))

(defun org-babel-write-result-to-file (file result params result-params)
  "If `:results' are special types like `link' or `graphics',
don't write result to `:file'.  Only insert a link to
`:file'. Change the file modes as well."
  (unless (or (member "link" result-params)
	      (member "graphics" result-params))
    (with-temp-file file
      (insert (org-babel-format-result
	       result
	       (cdr (assq :sep params)))))
    ;; Set file permissions if header argument
    ;; `:file-mode' is provided.
    (when (assq :file-mode params)
      (set-file-modes file (cdr (assq :file-mode params)))))
  file)

(defun org-babel-maybe-make-table (result params result-params session)
  "Turn the text-output of the process into a lisp form if
required by the header arguments."
  ;; Receive the raw string result and transform it with the
  ;; language specific org-babel-table-or-string:LANG to an elisp
  ;; form, then to a table if instructed by the headers.
  (let ((result (org-babel-reassemble-table
		 (org-babel-result-cond result-params
		   result
		   (org-babel-table-or-string session (org-trim result))) 
		 (org-babel-pick-name (cdr (assq :colname-names params))
				      (cdr (assq :colnames params)))
		 (org-babel-pick-name (cdr (assq :rowname-names params))
				      (cdr (assq :rownames params))))))
    ;; If the user requested the value of the computation in either
    ;; vector or table form, transform it to an equivalent elisp
    ;; expression.
    (if (and (eq (cdr (assq :result-type params)) 'value)
	     (or (member "vector" result-params)
		 (member "table" result-params))
	     (not (listp result)))
	(list (list result))
      result)))

(defun org-babel-maybe-postprocess (file result params post)
  (let ((*this* (if (not file) result
		  (org-babel-result-to-file
		   file
		   (org-babel--file-desc params result)
		   'attachment))))
    (org-babel-ref-resolve post)))


;; Python stuff to be moved out ----------------------------------------
(defclass org-babel-session:python (org-babel-session)
  ((indicator-regexp :initform "ob_comint_async_python_\\(.+\\)_\\(.+\\)"
		     "The regexp for the indicator that delimits new
	     source-block output")
   (language :initform "python")))

(cl-defmethod org-babel-interpreter-cmd ((sess org-babel-session:python))
  "Execute BODY synchronously or asynchronously (depending on
  ASYNC) using SESS's process. For this one must implement a
  concrete subclass of `org-babel-session' with a method that
  sends input asynchronously to process. This is an abstract
  method and must be implemented on a per-language basis."
  "python3")

(defun org-babel-python-format-session-value
    (src-file result-file result-params)
  "Return Python code to evaluate SRC-FILE and write result to RESULT-FILE."
  (format "\
import ast
with open('%s') as __org_babel_python_tmpfile:
    __org_babel_python_ast = ast.parse(__org_babel_python_tmpfile.read())
__org_babel_python_final = __org_babel_python_ast.body[-1]
if isinstance(__org_babel_python_final, ast.Expr):
    __org_babel_python_ast.body = __org_babel_python_ast.body[:-1]
    exec(compile(__org_babel_python_ast, '<string>', 'exec'))
    __org_babel_python_final = eval(compile(ast.Expression(
        __org_babel_python_final.value), '<string>', 'eval'))
    with open('%s', 'w+') as __org_babel_python_tmpfile:
        if %s:
            import pprint
            __org_babel_python_tmpfile.write(pprint.pformat(__org_babel_python_final))
        else:
            __org_babel_python_tmpfile.write(str(__org_babel_python_final))
else:
    exec(compile(__org_babel_python_ast, '<string>', 'exec'))
    __org_babel_python_final = None"
	  (org-babel-process-file-name src-file 'noquote)
	  (org-babel-process-file-name result-file 'noquote)
	  (if (member "pp" result-params) "True" "False")))

(defun insert-token-cmd (cmd loc)
  (insert (format "\tprint ('ob_comint_async_python_{token}_{location}'.format(token='%s', location='%s'))" cmd loc)))

(defun execute-python-cmd-in-shell (proc tmp-cmd-file ret-loc)
  (with-current-buffer proc
    (insert "import traceback") (comint-send-input) (goto-char (point-max))
    (insert "try:") (comint-send-input)  (goto-char (point-max))
    (insert-token-cmd "start" ret-loc) (comint-send-input)  (goto-char (point-max))
    (insert (format "\texec(open('%s').read())" tmp-cmd-file)) (comint-send-input)  (goto-char (point-max))
    (insert-token-cmd "end" ret-loc) (comint-send-input)  (goto-char (point-max))
    (insert "except SyntaxError:") (comint-send-input)  (goto-char (point-max))  
    (insert "\tprint(traceback.format_exc())") (comint-send-input)  (goto-char (point-max))
    (insert-token-cmd "exceptend" ret-loc) (comint-send-input)  (goto-char (point-max))
    (insert "except Exception:") (comint-send-input)  (goto-char (point-max))  
    (insert "\tprint(traceback.format_exc())") (comint-send-input)  (goto-char (point-max))
    (insert-token-cmd "exceptend" ret-loc) (comint-send-input)  (goto-char (point-max)) (comint-send-input)))


(cl-defmethod org-babel-session-execute ((sess org-babel-session:python) body params)
  "Execute BODY synchronously or asynchronously (depending on
  ASYNC) using SESS's process. For this one must implement a
  concrete subclass of `org-babel-session' with a method that
  sends input asynchronously to process. This is an abstract
  method and must be implemented on a per-language basis."
  (let* ((result-params (cdr (assq :result-params params)))
	 (result-type (cdr (assq :result-type params)))
	 (return-val (when (eq result-type 'value)
		       (cdr (assq :return params))))
	 (proc (slot-value session 'process))
	 (full-body
	  (concat
	   (org-babel-expand-body:generic
	    body params
	    (org-babel-variable-assignments:python params))
	   (when return-val
	     (format "\n%s" return-val)))))
    (pcase result-type
      (`output
       (let ((uuid (md5 (number-to-string (random 100000000))))
	     (tmp-cmd-file (org-babel-temp-file "python-")))
         (with-temp-file tmp-cmd-file
           (insert full-body))
	 (execute-python-cmd-in-shell proc tmp-cmd-file uuid)
         uuid))
      (`value
       (let ((tmp-results-file (org-babel-temp-file "python-"))
             (tmp-src-file (org-babel-temp-file "python-"))
	     (tmp-cmd-file (org-babel-temp-file "python-")))
         (with-temp-file tmp-src-file (insert full-body))
         (with-temp-file tmp-cmd-file
           (insert (org-babel-python-format-session-value tmp-src-file tmp-results-file result-params))
           (insert "\n")
           (insert (format "print ('ob_comint_async_python_file_%s')"  tmp-results-file)))
	 (execute-python-cmd-in-shell proc tmp-cmd-file tmp-results-file)
         tmp-results-file)))))

(cl-defmethod org-babel-table-or-string ((sess org-babel-session:python) string)
  "Convert RESULTS into an appropriate elisp value.
If the results look like a list or tuple, then convert them into
an Emacs-lisp table, otherwise return the results as a string."
  (let ((res (org-babel-script-escape string)))
    (if (listp res)
        (mapcar (lambda (el) (if (eq el 'None)
				 org-babel-python-None-to el))
                res)
      res)))

Re: We have asynchronous sessions, why have anything else?

[Ihor Radchenko]

Ivar Fredholm <freddyholms@protonmail.com> writes:

> Hi Ihor, I have a prototype of what I mentioned earlier, at least for python. This supports asynchronous, synchronous, session, and session-less blocks. It's pretty messy but it helps to illustrate what I had in mind. Let me know what you think.

I am not sure how I feel about it.
From cursory look, the idea looks reasonable implementation-wise.

However, there is one big important requirement which does not appear to
be obeyed by your code: We _must_ be backwards-compatible. All the
existing babel backends must not be broken, especially third-party ones.
We must not make breaking changes to non-private function definitions.

Also, I do not see error handling as it is implemented in our current
babel code: Errors should be displayed in a popup buffer.

Best,
Ihor