Async Python src block behavior with :dir header property

Async Python src block behavior with :dir header property

[Nasser Alkmim]

Hi,

I'm in the latest Emacs GUI.
This works, the =return figname= apparently can recognizee that we are in the =:dir=.

#+begin_src python :dir asyncpy :return figname :results file value
import matplotlib.pyplot as plt
plt.figure(figsize=(4*.5, 3*.5))
plt.plot([1, 2])
figname = 'fig.svg'
plt.savefig(figname, transparent=True, bbox_inches='tight')
#+end_src


This does not work.
Even though the python execution is in the right directory and the file is saved correctly.
The =return= does not produces the right file path considering the directory.

#+begin_src python :dir asyncpy :async :session :return figname :results file value
import matplotlib.pyplot as plt
plt.figure(figsize=(4*.5, 3*.5))
plt.plot([1, 2])
figname = 'fig.svg'
plt.savefig(figname, transparent=True, bbox_inches='tight')
#+end_src

If we inform the right path to the return statement, it works.

#+begin_src python :dir asyncpy :async :session :epilogue figdir='asyncpy/'+figname :return figdir :results file value
import matplotlib.pyplot as plt
plt.figure(figsize=(4*.5, 3*.5))
plt.plot([1, 2])
figname = 'fig.svg'
plt.savefig(figname, transparent=True, bbox_inches='tight')
#+end_src

Another alternative, we can use the =os= package to get the current directory.

#+header: :epilogue import os;figdir=os.getcwd()+'/'+figname 
#+begin_src python :dir asyncpy :async :session :return figdir :results file value
import matplotlib.pyplot as plt
plt.figure(figsize=(4*.5, 3*.5))
plt.plot([1, 2])
figname = 'fig.svg'
plt.savefig(figname, transparent=True, bbox_inches='tight')
#+end_src

Is this behavior as expected?
Is there a better way to have the figures from the python block with async execution?

-- 
Nasser Alkmim 
 +43 677 6408 9171

Re: Async Python src block behavior with :dir header property

[Ihor Radchenko]

Nasser Alkmim <nasser.alkmim@gmail.com> writes:

> This does not work.
> Even though the python execution is in the right directory and the file is saved correctly.
> The =return= does not produces the right file path considering the directory.
>
> #+begin_src python :dir asyncpy :async :session :return figname :results file value

These header arguments are not correctly formatted.
Try M-x org-lint

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Async Python src block behavior with :dir header property

[Nasser Alkmim]

Ihor Radchenko <yantar92@posteo.net> writes:

> Nasser Alkmim <nasser.alkmim@gmail.com> writes:
>
>> This does not work.
>> Even though the python execution is in the right directory and the file is saved
>> correctly.
>> The =return= does not produces the right file path considering the directory.
>>
>> #+begin_src python :dir asyncpy :async :session :return figname :results file value
>
> These header arguments are not correctly formatted.
> Try M-x org-lint

I tried org-lint and it says that the header argument is "unknown".
However, I can execute the source blocks.
I tried with 'emacs -Q' and '(require 'ob-python)' and I noticed that I
missed the header argument ':mkdirp "yes"'.
With that I can reproduce the examples.

In any case, thanks for taking a look.

-- 
Nasser Alkmim 

Re: Async Python src block behavior with :dir header property

[Ihor Radchenko]

Nasser Alkmim <nasser.alkmim@gmail.com> writes:

>>> #+begin_src python :dir asyncpy :async :session :return figname :results file value
>>
>> These header arguments are not correctly formatted.
>> Try M-x org-lint
>
> I tried org-lint and it says that the header argument is "unknown".

You should have got
     1 low   Empty value in header argument ":session"
     1 low   Empty value in header argument ":async"
     1 nil   Unknown header argument ":async"

You need to specify (1) :session name; (2) :async yes.

> However, I can execute the source blocks.

There is no guarantee that execution will work as expected in malformed
src blocks like yours.

> I tried with 'emacs -Q' and '(require 'ob-python)' and I noticed that I
> missed the header argument ':mkdirp "yes"'.
> With that I can reproduce the examples.

Do you mean that you have no problem after you add :mkdirp yes?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Async Python src block behavior with :dir header property

[Nasser Alkmim]

Ihor Radchenko <yantar92@posteo.net> writes:

> Do you mean that you have no problem after you add :mkdirp yes?

No. Just that I can reproduce the examples in the first message.

Here is a clearer description of the unwanted behavior:

1. emacs -Q ~/Desktop/testasync.org

Then evaluate (require 'ob-python) and paste this source block:

#+begin_src python :dir otherdir :async yes :session pysession :return figname :results file value :mkdirp yes
import matplotlib.pyplot as plt
plt.figure(figsize=(1, 1))
plt.plot([1, 2])
figname = 'fig.svg'
plt.savefig(figname)
#+end_src

Execute the source block, which results in 

#+RESULTS:
[[file:fig.svg]]

However, the figure is in ~/Desktop/otherdir/fig.svg and the link is
wrong.  If I run the same block without the ':async' and ':session', I
get the right relative path to the figure:

#+RESULTS:
[[file:otherdir/fig.svg]]

Best regards,

-- 
Nasser Alkmim 
 +43 677 6408 9171

Re: Async Python src block behavior with :dir header property

[Ihor Radchenko]

[ CCing ob-python maintainer ]

Nasser Alkmim <nasser.alkmim@gmail.com> writes:

> Here is a clearer description of the unwanted behavior:
>
> 1. emacs -Q ~/Desktop/testasync.org
>
> Then evaluate (require 'ob-python) and paste this source block:
>
> #+begin_src python :dir otherdir :async yes :session pysession :return figname :results file value :mkdirp yes
> import matplotlib.pyplot as plt
> plt.figure(figsize=(1, 1))
> plt.plot([1, 2])
> figname = 'fig.svg'
> plt.savefig(figname)
> #+end_src
>
> Execute the source block, which results in 
>
> #+RESULTS:
> [[file:fig.svg]]

I don't even see this.
Confirmed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Async Python src block behavior with :dir header property

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 976 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> [ CCing ob-python maintainer ]
>
> Nasser Alkmim <nasser.alkmim@gmail.com> writes:
>
>> Here is a clearer description of the unwanted behavior:
>>
>> 1. emacs -Q ~/Desktop/testasync.org
>>
>> Then evaluate (require 'ob-python) and paste this source block:
>>
>> #+begin_src python :dir otherdir :async yes :session pysession :return figname :results file value :mkdirp yes
>> import matplotlib.pyplot as plt
>> plt.figure(figsize=(1, 1))
>> plt.plot([1, 2])
>> figname = 'fig.svg'
>> plt.savefig(figname)
>> #+end_src
>>
>> Execute the source block, which results in 
>>
>> #+RESULTS:
>> [[file:fig.svg]]
>
> I don't even see this.
> Confirmed.

Does the attached patch fix the issue?

It seems the problem is with async sessions generally (not just
ob-python), and happens because `org-babel-comint-async-filter' does not
set `default-directory' before calling `org-babel-insert-result', unlike
`org-babel-execute-src-block'.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-comint-Make-file-results-from-async-sessions-resp.patch --]
[-- Type: text/x-patch, Size: 2644 bytes --]

From 1430a27e4416d5e88094a64360015a6a2ae7315c Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Wed, 31 Jan 2024 20:06:00 -0800
Subject: [PATCH] ob-comint: Make file results from async sessions respect :dir
 header

* lisp/ob-comint.el (org-babel-comint-async-filter): Set
default-directory before calling `org-babel-insert-result'

https://list.orgmode.org/875xz9o4nj.fsf@localhost/T/#t
---
 lisp/ob-comint.el | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 7d258ea0e..ecce18a95 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -248,7 +248,14 @@ (defun org-babel-comint-async-filter (string)
                                 (let* ((info (org-babel-get-src-block-info))
                                        (params (nth 2 info))
                                        (result-params
-                                        (cdr (assq :result-params params))))
+                                        (cdr (assq :result-params params)))
+                                       (tmp-file (expand-file-name tmp-file))
+                                       (dir (cdr (assq :dir params)))
+                                       (default-directory
+                                        (if dir
+                                            (file-name-as-directory
+                                             (expand-file-name dir))
+                                          default-directory)))
                                   (org-babel-insert-result
                                    (funcall file-callback
                                             (nth
@@ -291,7 +298,13 @@ (defun org-babel-comint-async-filter (string)
                                    (let* ((info (org-babel-get-src-block-info))
                                           (params (nth 2 info))
                                           (result-params
-                                           (cdr (assq :result-params params))))
+                                           (cdr (assq :result-params params)))
+                                          (dir (cdr (assq :dir params)))
+                                          (default-directory
+                                           (if dir
+                                               (file-name-as-directory
+                                                (expand-file-name dir))
+                                             default-directory)))
 				     (org-babel-insert-result
                                       res-str result-params info))
 				   t))))
-- 
2.43.0

Re: Async Python src block behavior with :dir header property

[Nasser Alkmim]

Jack Kamm <jackkamm@gmail.com> writes:

>
> Does the attached patch fix the issue?
>

I briefly tested it here and it works.
Thanks for taking a look into it.

-- 
Nasser Alkmim 
 +43 677 6408 9171

Re: Async Python src block behavior with :dir header property

[Ihor Radchenko]

Jack Kamm <jackkamm@gmail.com> writes:

>> Confirmed.
>
> Does the attached patch fix the issue?
>
> It seems the problem is with async sessions generally (not just
> ob-python), and happens because `org-babel-comint-async-filter' does not
> set `default-directory' before calling `org-babel-insert-result', unlike
> `org-babel-execute-src-block'.

The patch generally looks reasonable, although I am slightly concerned
about interaction between :dir and session we describe in the manual:

    When ‘dir’ is used with ‘session’, Org sets the starting directory
    for a new session.  But Org does not alter the directory of an already
    existing session.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Async Python src block behavior with :dir header property

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

> The patch generally looks reasonable, although I am slightly concerned
> about interaction between :dir and session we describe in the manual:
>
>     When ‘dir’ is used with ‘session’, Org sets the starting directory
>     for a new session.  But Org does not alter the directory of an already
>     existing session.

I agree it's a problem -- if there are multiple blocks with the same
session but different ":dir" arguments, then a ":file" result of the
second block will be relative to the wrong :dir.

This seems like a longstanding problem, and affects both async and
non-async session blocks.

Ideally, ":dir" should be set at session level rather than block
level. This could be done via PROPERTY if there is only one session in
the file or subtree, but there isn't a mechanism for this more
generally.

Perhaps the simplest solution is to add a check to org-lint, requiring
all blocks with the same session to have the same :dir.

Re: Async Python src block behavior with :dir header property

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 1296 bytes --]

Jack Kamm <jackkamm@gmail.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> The patch generally looks reasonable, although I am slightly concerned
>> about interaction between :dir and session we describe in the manual:
>>
>>     When ‘dir’ is used with ‘session’, Org sets the starting directory
>>     for a new session.  But Org does not alter the directory of an already
>>     existing session.
>
> I agree it's a problem -- if there are multiple blocks with the same
> session but different ":dir" arguments, then a ":file" result of the
> second block will be relative to the wrong :dir.
>
> This seems like a longstanding problem, and affects both async and
> non-async session blocks.

Maybe something like the attached?


> Ideally, ":dir" should be set at session level rather than block
> level. This could be done via PROPERTY if there is only one session in
> the file or subtree, but there isn't a mechanism for this more
> generally.

Or we can ask babel backends to respect :dir and issue the necessary
"cd" command in the session for the duration of the code block evaluation.

> Perhaps the simplest solution is to add a check to org-lint, requiring
> all blocks with the same session to have the same :dir.

Or this.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-babel-execute-src-block-Force-dir-according-to-l.patch --]
[-- Type: text/x-patch, Size: 1864 bytes --]

From 1e87de13d44a1bb31985622bfc9c9a6df6b7548c Mon Sep 17 00:00:00 2001
Message-ID: <1e87de13d44a1bb31985622bfc9c9a6df6b7548c.1706975370.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sat, 3 Feb 2024 16:47:57 +0100
Subject: [PATCH] org-babel-execute-src-block: Force :dir according to live
 session

* lisp/ob-core.el (org-babel-execute-src-block): Force eval directory
to follow live session buffer, if any.  This is consistent with what
we promise in the manual.

Link: https://orgmode.org/list/87mssi8ht2.fsf@gmail.com
---
 lisp/ob-core.el | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 1de3af6ad..2cd542012 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -840,14 +840,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		 (dir (cdr (assq :dir params)))
 		 (mkdirp (cdr (assq :mkdirp params)))
 		 (default-directory
-		   (cond
-		    ((not dir) default-directory)
-		    ((member mkdirp '("no" "nil" nil))
-		     (file-name-as-directory (expand-file-name dir)))
-		    (t
-		     (let ((d (file-name-as-directory (expand-file-name dir))))
-		       (make-directory d 'parents)
-		       d))))
+		  (cond
+		   ((not dir) default-directory)
+                   ((when-let ((session (assq :session params)))
+                      (when (org-babel-comint-buffer-livep session)
+                        (buffer-local-value 'default-directory (get-buffer session)))))
+		   ((member mkdirp '("no" "nil" nil))
+		    (file-name-as-directory (expand-file-name dir)))
+		   (t
+		    (let ((d (file-name-as-directory (expand-file-name dir))))
+		      (make-directory d 'parents)
+		      d))))
 		 (cmd (intern (concat "org-babel-execute:" lang)))
 		 result exec-start-time)
 	    (unless (fboundp cmd)
-- 
2.43.0


[-- Attachment #3: Type: text/plain, Size: 225 bytes --]



-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Async Python src block behavior with :dir header property

[Jack Kamm]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

>> I agree it's a problem -- if there are multiple blocks with the same
>> session but different ":dir" arguments, then a ":file" result of the
>> second block will be relative to the wrong :dir.
>>
>> This seems like a longstanding problem, and affects both async and
>> non-async session blocks.
>
> Maybe something like the attached?

Nice, that seems like the right way to do it.

I updated the patch for `org-babel-comint-async-filter' to follow the
same approach, setting default-directory based on the session buffer's
value rather than the :dir header arg.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-comint-Make-file-results-from-async-sessions-resp.patch --]
[-- Type: text/x-patch, Size: 2360 bytes --]

From 1a1a22e4f4a12ebe83c3fef26fe727066fb14476 Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Wed, 31 Jan 2024 20:06:00 -0800
Subject: [PATCH] ob-comint: Make file results from async sessions respect :dir
 header

* lisp/ob-comint.el (org-babel-comint-async-filter): Set
default-directory before calling `org-babel-insert-result'

https://list.orgmode.org/875xz9o4nj.fsf@localhost/T/#t
---
 lisp/ob-comint.el | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 7d258ea0e..349524701 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -224,6 +224,8 @@ (defun org-babel-comint-async-filter (string)
 	 (file-callback org-babel-comint-async-file-callback)
 	 (combined-string (concat org-babel-comint-async-dangling string))
 	 (new-dangling combined-string)
+         ;; Assumes comint filter called with session buffer current
+         (session-dir default-directory)
 	 ;; list of UUID's matched by `org-babel-comint-async-indicator'
 	 uuid-list)
     (with-temp-buffer
@@ -248,7 +250,8 @@ (defun org-babel-comint-async-filter (string)
                                 (let* ((info (org-babel-get-src-block-info))
                                        (params (nth 2 info))
                                        (result-params
-                                        (cdr (assq :result-params params))))
+                                        (cdr (assq :result-params params)))
+                                       (default-directory session-dir))
                                   (org-babel-insert-result
                                    (funcall file-callback
                                             (nth
@@ -291,7 +294,8 @@ (defun org-babel-comint-async-filter (string)
                                    (let* ((info (org-babel-get-src-block-info))
                                           (params (nth 2 info))
                                           (result-params
-                                           (cdr (assq :result-params params))))
+                                           (cdr (assq :result-params params)))
+                                          (default-directory session-dir))
 				     (org-babel-insert-result
                                       res-str result-params info))
 				   t))))
-- 
2.43.0

Re: Async Python src block behavior with :dir header property

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 360 bytes --]

Jack Kamm <jackkamm@gmail.com> writes:

> I updated the patch for `org-babel-comint-async-filter' to follow the
> same approach, setting default-directory based on the session buffer's
> value rather than the :dir header arg.

Thanks!
Attaching the two patches combined with some fixed to my patch.

Please check if these two patches solve the discussed bug.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-babel-execute-src-block-Force-dir-according-to-l.patch --]
[-- Type: text/x-patch, Size: 1946 bytes --]

From 66bbac4207d7132ebc9f91f60384809ae556c2ef Mon Sep 17 00:00:00 2001
Message-ID: <66bbac4207d7132ebc9f91f60384809ae556c2ef.1707064153.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sat, 3 Feb 2024 16:47:57 +0100
Subject: [PATCH 1/2] org-babel-execute-src-block: Force :dir according to live
 session

* lisp/ob-core.el (org-babel-execute-src-block): Force eval directory
to follow live session buffer, if any.  This is consistent with what
we promise in the manual section "16.4 Environment of a Code
Block">Choosing a working directory.

Link: https://orgmode.org/list/87mssi8ht2.fsf@gmail.com
---
 lisp/ob-core.el | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 1de3af6ad..fd8d06c5d 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -840,14 +840,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		 (dir (cdr (assq :dir params)))
 		 (mkdirp (cdr (assq :mkdirp params)))
 		 (default-directory
-		   (cond
-		    ((not dir) default-directory)
-		    ((member mkdirp '("no" "nil" nil))
-		     (file-name-as-directory (expand-file-name dir)))
-		    (t
-		     (let ((d (file-name-as-directory (expand-file-name dir))))
-		       (make-directory d 'parents)
-		       d))))
+		  (cond
+		   ((not dir) default-directory)
+                   ((when-let ((session (cdr (assq :session params))))
+                      (when (org-babel-comint-buffer-livep session)
+                        (buffer-local-value 'default-directory (get-buffer session)))))
+		   ((member mkdirp '("no" "nil" nil))
+		    (file-name-as-directory (expand-file-name dir)))
+		   (t
+		    (let ((d (file-name-as-directory (expand-file-name dir))))
+		      (make-directory d 'parents)
+		      d))))
 		 (cmd (intern (concat "org-babel-execute:" lang)))
 		 result exec-start-time)
 	    (unless (fboundp cmd)
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-ob-comint-Make-file-results-from-async-sessions-resp.patch --]
[-- Type: text/x-patch, Size: 2635 bytes --]

From e9765a146bf27e6769a2563862d52b0efb77eb2e Mon Sep 17 00:00:00 2001
Message-ID: <e9765a146bf27e6769a2563862d52b0efb77eb2e.1707064153.git.yantar92@posteo.net>
In-Reply-To: <66bbac4207d7132ebc9f91f60384809ae556c2ef.1707064153.git.yantar92@posteo.net>
References: <66bbac4207d7132ebc9f91f60384809ae556c2ef.1707064153.git.yantar92@posteo.net>
From: Jack Kamm <jackkamm@gmail.com>
Date: Wed, 31 Jan 2024 20:06:00 -0800
Subject: [PATCH 2/2] ob-comint: Make file results from async sessions respect
 :dir header

* lisp/ob-comint.el (org-babel-comint-async-filter): Set
default-directory before calling `org-babel-insert-result'

https://list.orgmode.org/875xz9o4nj.fsf@localhost/T/#t
---
 lisp/ob-comint.el | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 7d258ea0e..349524701 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -224,6 +224,8 @@ (defun org-babel-comint-async-filter (string)
 	 (file-callback org-babel-comint-async-file-callback)
 	 (combined-string (concat org-babel-comint-async-dangling string))
 	 (new-dangling combined-string)
+         ;; Assumes comint filter called with session buffer current
+         (session-dir default-directory)
 	 ;; list of UUID's matched by `org-babel-comint-async-indicator'
 	 uuid-list)
     (with-temp-buffer
@@ -248,7 +250,8 @@ (defun org-babel-comint-async-filter (string)
                                 (let* ((info (org-babel-get-src-block-info))
                                        (params (nth 2 info))
                                        (result-params
-                                        (cdr (assq :result-params params))))
+                                        (cdr (assq :result-params params)))
+                                       (default-directory session-dir))
                                   (org-babel-insert-result
                                    (funcall file-callback
                                             (nth
@@ -291,7 +294,8 @@ (defun org-babel-comint-async-filter (string)
                                    (let* ((info (org-babel-get-src-block-info))
                                           (params (nth 2 info))
                                           (result-params
-                                           (cdr (assq :result-params params))))
+                                           (cdr (assq :result-params params)))
+                                          (default-directory session-dir))
 				     (org-babel-insert-result
                                       res-str result-params info))
 				   t))))
-- 
2.43.0


[-- Attachment #4: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Async Python src block behavior with :dir header property

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

> Thanks!
> Attaching the two patches combined with some fixed to my patch.
>
> Please check if these two patches solve the discussed bug.

The original bug for async sessions looks fixed. But I encountered a
complication regarding the non-async bug of changing :dir. In particular
the following example:

#+begin_src python :dir otherdir :session pysession :return figname :results file value :mkdirp yes
  import matplotlib.pyplot as plt
  plt.figure(figsize=(1, 1))
  plt.plot([1, 2])
  figname = 'fig.svg'
  plt.savefig(figname)
#+end_src

#+RESULTS:
[[file:otherdir/fig.svg]]

#+begin_src python :dir otherdir2 :session pysession :return figname :results file value :mkdirp yes
  import matplotlib.pyplot as plt
  plt.figure(figsize=(1, 1))
  plt.plot([1, 2])
  figname = 'fig5.svg'
  plt.savefig(figname)
#+end_src

#+RESULTS:
[[file:otherdir2/fig5.svg]]

As you can see the second result points to the wrong directory. 

However, if replacing ":session pysession" with ":session *pysession*",
then it works.

It's because ob-python starts the session in buffer "*pysession*" (it
adds earmuffs around the session name when missing). So the following
doesn't find the inferior Python:

> +                   ((when-let ((session (cdr (assq :session params))))
> +                      (when (org-babel-comint-buffer-livep session)

Honestly, I'm not sure why ob-python insists on the session buffer
having the earmuffs, that behavior is from before my time. In
particular, I'm not yet sure if the motivation is just cosmetic (because
of the general convention like *shell*, *Python*, *R*, etc), or whether
it actually affects python.el behavior. But if it's just cosmetic, then
we might consider removing this behavior to simplify things.

Though note that ob-python is not the only one with this sort of
behavior -- looks like ob-lua might behave similarly, and until recently
ob-R also had some surprising behavior depending on whether the session
name had earmuffs.

Re: Async Python src block behavior with :dir header property

[Ihor Radchenko]

Jack Kamm <jackkamm@gmail.com> writes:

> It's because ob-python starts the session in buffer "*pysession*" (it
> adds earmuffs around the session name when missing). So the following
> doesn't find the inferior Python:

Good point. Some backends indeed do not have comint buffer named the
same as :session name.

What we can do is to introduce a new backend template function
org-babel-session-buffer:<lang> that will be passed a session name and
src block params and return the session buffer name.

If such function is not defined, we fall back to assumption that session
buffer is named the same as the session.

WDYT?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Async Python src block behavior with :dir header property

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

> What we can do is to introduce a new backend template function
> org-babel-session-buffer:<lang> that will be passed a session name and
> src block params and return the session buffer name.
>
> If such function is not defined, we fall back to assumption that session
> buffer is named the same as the session.
>
> WDYT?

Sounds good -- I think this is the best solution.

Re: Async Python src block behavior with :dir header property

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 502 bytes --]

Jack Kamm <jackkamm@gmail.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> What we can do is to introduce a new backend template function
>> org-babel-session-buffer:<lang> that will be passed a session name and
>> src block params and return the session buffer name.
>>
>> If such function is not defined, we fall back to assumption that session
>> buffer is named the same as the session.
>>
>> WDYT?
>
> Sounds good -- I think this is the best solution.

See the updated patchset.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-babel-execute-src-block-Force-dir-according-to-l.patch --]
[-- Type: text/x-patch, Size: 1946 bytes --]

From b877a440c65cb83bd29182daf4c13cd9b14ed182 Mon Sep 17 00:00:00 2001
Message-ID: <b877a440c65cb83bd29182daf4c13cd9b14ed182.1707228351.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sat, 3 Feb 2024 16:47:57 +0100
Subject: [PATCH 1/3] org-babel-execute-src-block: Force :dir according to live
 session

* lisp/ob-core.el (org-babel-execute-src-block): Force eval directory
to follow live session buffer, if any.  This is consistent with what
we promise in the manual section "16.4 Environment of a Code
Block">Choosing a working directory.

Link: https://orgmode.org/list/87mssi8ht2.fsf@gmail.com
---
 lisp/ob-core.el | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 1de3af6ad..fd8d06c5d 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -840,14 +840,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		 (dir (cdr (assq :dir params)))
 		 (mkdirp (cdr (assq :mkdirp params)))
 		 (default-directory
-		   (cond
-		    ((not dir) default-directory)
-		    ((member mkdirp '("no" "nil" nil))
-		     (file-name-as-directory (expand-file-name dir)))
-		    (t
-		     (let ((d (file-name-as-directory (expand-file-name dir))))
-		       (make-directory d 'parents)
-		       d))))
+		  (cond
+		   ((not dir) default-directory)
+                   ((when-let ((session (cdr (assq :session params))))
+                      (when (org-babel-comint-buffer-livep session)
+                        (buffer-local-value 'default-directory (get-buffer session)))))
+		   ((member mkdirp '("no" "nil" nil))
+		    (file-name-as-directory (expand-file-name dir)))
+		   (t
+		    (let ((d (file-name-as-directory (expand-file-name dir))))
+		      (make-directory d 'parents)
+		      d))))
 		 (cmd (intern (concat "org-babel-execute:" lang)))
 		 result exec-start-time)
 	    (unless (fboundp cmd)
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-ob-comint-Make-file-results-from-async-sessions-resp.patch --]
[-- Type: text/x-patch, Size: 2635 bytes --]

From a31c7b3f5f730b63d3753e07a7a5bb5e73a97ce4 Mon Sep 17 00:00:00 2001
Message-ID: <a31c7b3f5f730b63d3753e07a7a5bb5e73a97ce4.1707228351.git.yantar92@posteo.net>
In-Reply-To: <b877a440c65cb83bd29182daf4c13cd9b14ed182.1707228351.git.yantar92@posteo.net>
References: <b877a440c65cb83bd29182daf4c13cd9b14ed182.1707228351.git.yantar92@posteo.net>
From: Jack Kamm <jackkamm@gmail.com>
Date: Wed, 31 Jan 2024 20:06:00 -0800
Subject: [PATCH 2/3] ob-comint: Make file results from async sessions respect
 :dir header

* lisp/ob-comint.el (org-babel-comint-async-filter): Set
default-directory before calling `org-babel-insert-result'

https://list.orgmode.org/875xz9o4nj.fsf@localhost/T/#t
---
 lisp/ob-comint.el | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 7d258ea0e..349524701 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -224,6 +224,8 @@ (defun org-babel-comint-async-filter (string)
 	 (file-callback org-babel-comint-async-file-callback)
 	 (combined-string (concat org-babel-comint-async-dangling string))
 	 (new-dangling combined-string)
+         ;; Assumes comint filter called with session buffer current
+         (session-dir default-directory)
 	 ;; list of UUID's matched by `org-babel-comint-async-indicator'
 	 uuid-list)
     (with-temp-buffer
@@ -248,7 +250,8 @@ (defun org-babel-comint-async-filter (string)
                                 (let* ((info (org-babel-get-src-block-info))
                                        (params (nth 2 info))
                                        (result-params
-                                        (cdr (assq :result-params params))))
+                                        (cdr (assq :result-params params)))
+                                       (default-directory session-dir))
                                   (org-babel-insert-result
                                    (funcall file-callback
                                             (nth
@@ -291,7 +294,8 @@ (defun org-babel-comint-async-filter (string)
                                    (let* ((info (org-babel-get-src-block-info))
                                           (params (nth 2 info))
                                           (result-params
-                                           (cdr (assq :result-params params))))
+                                           (cdr (assq :result-params params)))
+                                          (default-directory session-dir))
 				     (org-babel-insert-result
                                       res-str result-params info))
 				   t))))
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-org-babel-New-babel-backend-API-function-org-babel-s.patch --]
[-- Type: text/x-patch, Size: 6415 bytes --]

From 5afbd3b5dce89d3da9f0532c6358392175a93a9f Mon Sep 17 00:00:00 2001
Message-ID: <5afbd3b5dce89d3da9f0532c6358392175a93a9f.1707228351.git.yantar92@posteo.net>
In-Reply-To: <b877a440c65cb83bd29182daf4c13cd9b14ed182.1707228351.git.yantar92@posteo.net>
References: <b877a440c65cb83bd29182daf4c13cd9b14ed182.1707228351.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 6 Feb 2024 15:02:48 +0100
Subject: [PATCH 3/3] org-babel: New babel backend API function
 org-babel-session-buffer:<lang>

* lisp/ob-core.el (org-babel-session-buffer): New API function that
return session buffer, if such buffer exists and is live.
(org-babel-execute-src-block): Use `org-babel-session-buffer'.
* lisp/ob-lua.el (org-babel-session-buffer:lua):
* lisp/ob-python.el (org-babel-session-buffer:python): Provide API to
retrieve session buffer name.
(org-babel-python-initiate-session-by-key): Use
`org-babel-session-buffer:python'.
* etc/ORG-NEWS (Org babel backends are now expected to define an
additional API function ~org-babel-session-buffer:<lang>~): Declare
the API addition.

Link: https://orgmode.org/list/87r0hr9f3b.fsf@localhost
---
 etc/ORG-NEWS      | 11 +++++++++++
 lisp/ob-core.el   | 32 +++++++++++++++++++++++++++-----
 lisp/ob-lua.el    |  5 +++++
 lisp/ob-python.el |  8 ++++++--
 4 files changed, 49 insertions(+), 7 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 965872d23..246d83ffe 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -13,6 +13,17 @@ Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
 * Version 9.7 (not released yet)
 ** Important announcements and breaking changes
+*** Org babel backends are now expected to define an additional API function ~org-babel-session-buffer:<lang>~
+
+Org babel now uses session buffer (if it exists) to retrieve
+~default-directory~ environment during src block evaluation.
+
+By default, buffer named like session is checked.  All the backends
+that create sessions inside buffers named differently should provide a
+function ~org-babel-session-buffer:<lang>~.  The function must accept
+two arguments - session name and info list (as returned by
+~org-babel-get-src-block-info~); and return the session buffer name.
+
 *** Org mode now fontifies whole table lines (including newline) according to ~org-table~ face
 
 Previously, leading indentation and trailing newline in table rows
diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index fd8d06c5d..a812887fa 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -767,8 +767,31 @@ (defun org-babel--file-desc (params result)
     (`(:file-desc) result)
     (`(:file-desc . ,(and (pred stringp) val)) val)))
 
-(defvar *this*) ; Dynamically bound in `org-babel-execute-src-block'
-                ; and `org-babel-read'
+(defvar *this*)
+;; Dynamically bound in `org-babel-execute-src-block'
+;; and `org-babel-read'
+
+(defun org-babel-session-buffer (&optional info)
+  "Return buffer name for session associated with current code block.
+Return nil when no such live buffer with process exists.
+When INFO is non-nil, it should be a list returned by
+`org-babel-get-src-block-info'.
+This function uses org-babel-session-buffer:<lang> function to
+retrieve backend-specific session buffer name."
+  (when-let* ((info (or info (org-babel-get-src-block-info 'no-eval)))
+              (lang (nth 0 info))
+              (session (cdr (assq :session (nth 2 info))))
+              (cmd (intern (concat "org-babel-session-buffer:" lang)))
+              buffer-name)
+    (setq buffer-name
+          (if (fboundp cmd)
+              (funcall cmd session info)
+            ;; Use session name as buffer name by default.
+            session))
+    (when buffer-name
+      (require 'ob-comint)
+      (when (org-babel-comint-buffer-livep buffer-name)
+        buffer-name))))
 
 ;;;###autoload
 (defun org-babel-execute-src-block (&optional arg info params executor-type)
@@ -842,9 +865,8 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		 (default-directory
 		  (cond
 		   ((not dir) default-directory)
-                   ((when-let ((session (cdr (assq :session params))))
-                      (when (org-babel-comint-buffer-livep session)
-                        (buffer-local-value 'default-directory (get-buffer session)))))
+                   ((when-let ((session (org-babel-session-buffer info)))
+                      (buffer-local-value 'default-directory (get-buffer session))))
 		   ((member mkdirp '("no" "nil" nil))
 		    (file-name-as-directory (expand-file-name dir)))
 		   (t
diff --git a/lisp/ob-lua.el b/lisp/ob-lua.el
index a2b830ca3..b241fccdc 100644
--- a/lisp/ob-lua.el
+++ b/lisp/ob-lua.el
@@ -184,6 +184,11 @@ (defun org-babel-lua-with-earmuffs (session)
 	name
       (format "*%s*" name))))
 
+(defun org-babel-session-buffer:lua (session &optional _)
+  "Return session buffer name for SESSION."
+  (or (org-babel-lua-session-buffer session)
+      (org-babel-lua-with-earmuffs session)))
+
 (defun org-babel-lua-without-earmuffs (session)
 "Remove stars around *SESSION*, leaving SESSION."
   (let ((name (if (stringp session) session (format "%s" session))))
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index cfc0fdcb6..89cdf4c47 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -260,6 +260,11 @@ (defun org-babel-python-without-earmuffs (session)
 	(substring name 1 (- (length name) 1))
       name)))
 
+(defun org-babel-session-buffer:python (session &optional _)
+  "Return session buffer name for SESSION."
+  (or (org-babel-python-session-buffer session)
+      (org-babel-python-with-earmuffs session)))
+
 (defun org-babel-python--python-util-comint-end-of-output-p ()
   "Return non-nil if the last prompt matches input prompt.
 Backport of `python-util-comint-end-of-output-p' to emacs28.  To
@@ -302,8 +307,7 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
 initialized session."
   (save-window-excursion
     (let* ((session (if session (intern session) :default))
-           (py-buffer (or (org-babel-python-session-buffer session)
-                          (org-babel-python-with-earmuffs session)))
+           (py-buffer (org-babel-session-buffer:python session))
            (python-shell-buffer-name
 	    (org-babel-python-without-earmuffs py-buffer))
            (existing-session-p (comint-check-proc py-buffer))
-- 
2.43.0


[-- Attachment #5: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Async Python src block behavior with :dir header property

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

> +(defun org-babel-session-buffer (&optional info)
> +  "Return buffer name for session associated with current code block.
> +Return nil when no such live buffer with process exists.
> +When INFO is non-nil, it should be a list returned by
> +`org-babel-get-src-block-info'.
> +This function uses org-babel-session-buffer:<lang> function to
> +retrieve backend-specific session buffer name."
> +  (when-let* ((info (or info (org-babel-get-src-block-info 'no-eval)))
> +              (lang (nth 0 info))
> +              (session (cdr (assq :session (nth 2 info))))
> +              (cmd (intern (concat "org-babel-session-buffer:" lang)))
> +              buffer-name)

On executing any python session block I am getting the following error
which I think is caused by the above:

  Debugger entered--Lisp error: (void-variable buffer-name)

Also, make shows a byte-compiler warning about this:

  Compiling /home/jack/src/org-mode/2024-01-async-file-results-dir/lisp/ob-core.el...
  
  In org-babel-session-buffer:
  ob-core.el:785:15: Warning: reference to free variable ‘buffer-name’

Re: Async Python src block behavior with :dir header property

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 597 bytes --]

Jack Kamm <jackkamm@gmail.com> writes:

> On executing any python session block I am getting the following error
> which I think is caused by the above:
>
>   Debugger entered--Lisp error: (void-variable buffer-name)

That's a mystery.

> Also, make shows a byte-compiler warning about this:
>
>   Compiling /home/jack/src/org-mode/2024-01-async-file-results-dir/lisp/ob-core.el...
>   
>   In org-babel-session-buffer:
>   ob-core.el:785:15: Warning: reference to free variable ‘buffer-name’

But this is a real problem.
See the attached updated version of the patch set.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v2-0001-org-babel-execute-src-block-Force-dir-according-t.patch --]
[-- Type: text/x-patch, Size: 1949 bytes --]

From 1185e7a3569048b465b697fe173ce50a3ab7f0cc Mon Sep 17 00:00:00 2001
Message-ID: <1185e7a3569048b465b697fe173ce50a3ab7f0cc.1707406936.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sat, 3 Feb 2024 16:47:57 +0100
Subject: [PATCH v2 1/3] org-babel-execute-src-block: Force :dir according to
 live session

* lisp/ob-core.el (org-babel-execute-src-block): Force eval directory
to follow live session buffer, if any.  This is consistent with what
we promise in the manual section "16.4 Environment of a Code
Block">Choosing a working directory.

Link: https://orgmode.org/list/87mssi8ht2.fsf@gmail.com
---
 lisp/ob-core.el | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index 1de3af6ad..fd8d06c5d 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -840,14 +840,17 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		 (dir (cdr (assq :dir params)))
 		 (mkdirp (cdr (assq :mkdirp params)))
 		 (default-directory
-		   (cond
-		    ((not dir) default-directory)
-		    ((member mkdirp '("no" "nil" nil))
-		     (file-name-as-directory (expand-file-name dir)))
-		    (t
-		     (let ((d (file-name-as-directory (expand-file-name dir))))
-		       (make-directory d 'parents)
-		       d))))
+		  (cond
+		   ((not dir) default-directory)
+                   ((when-let ((session (cdr (assq :session params))))
+                      (when (org-babel-comint-buffer-livep session)
+                        (buffer-local-value 'default-directory (get-buffer session)))))
+		   ((member mkdirp '("no" "nil" nil))
+		    (file-name-as-directory (expand-file-name dir)))
+		   (t
+		    (let ((d (file-name-as-directory (expand-file-name dir))))
+		      (make-directory d 'parents)
+		      d))))
 		 (cmd (intern (concat "org-babel-execute:" lang)))
 		 result exec-start-time)
 	    (unless (fboundp cmd)
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: v2-0002-ob-comint-Make-file-results-from-async-sessions-r.patch --]
[-- Type: text/x-patch, Size: 2638 bytes --]

From 8799d747ab2983f43dab117713342dbc92c6fc16 Mon Sep 17 00:00:00 2001
Message-ID: <8799d747ab2983f43dab117713342dbc92c6fc16.1707406936.git.yantar92@posteo.net>
In-Reply-To: <1185e7a3569048b465b697fe173ce50a3ab7f0cc.1707406936.git.yantar92@posteo.net>
References: <1185e7a3569048b465b697fe173ce50a3ab7f0cc.1707406936.git.yantar92@posteo.net>
From: Jack Kamm <jackkamm@gmail.com>
Date: Wed, 31 Jan 2024 20:06:00 -0800
Subject: [PATCH v2 2/3] ob-comint: Make file results from async sessions
 respect :dir header

* lisp/ob-comint.el (org-babel-comint-async-filter): Set
default-directory before calling `org-babel-insert-result'

https://list.orgmode.org/875xz9o4nj.fsf@localhost/T/#t
---
 lisp/ob-comint.el | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 7d258ea0e..349524701 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -224,6 +224,8 @@ (defun org-babel-comint-async-filter (string)
 	 (file-callback org-babel-comint-async-file-callback)
 	 (combined-string (concat org-babel-comint-async-dangling string))
 	 (new-dangling combined-string)
+         ;; Assumes comint filter called with session buffer current
+         (session-dir default-directory)
 	 ;; list of UUID's matched by `org-babel-comint-async-indicator'
 	 uuid-list)
     (with-temp-buffer
@@ -248,7 +250,8 @@ (defun org-babel-comint-async-filter (string)
                                 (let* ((info (org-babel-get-src-block-info))
                                        (params (nth 2 info))
                                        (result-params
-                                        (cdr (assq :result-params params))))
+                                        (cdr (assq :result-params params)))
+                                       (default-directory session-dir))
                                   (org-babel-insert-result
                                    (funcall file-callback
                                             (nth
@@ -291,7 +294,8 @@ (defun org-babel-comint-async-filter (string)
                                    (let* ((info (org-babel-get-src-block-info))
                                           (params (nth 2 info))
                                           (result-params
-                                           (cdr (assq :result-params params))))
+                                           (cdr (assq :result-params params)))
+                                          (default-directory session-dir))
 				     (org-babel-insert-result
                                       res-str result-params info))
 				   t))))
-- 
2.43.0


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: v2-0003-org-babel-New-babel-backend-API-function-org-babe.patch --]
[-- Type: text/x-patch, Size: 6458 bytes --]

From 0d496b8fcc205810c994b32b2ab732305a4d9a26 Mon Sep 17 00:00:00 2001
Message-ID: <0d496b8fcc205810c994b32b2ab732305a4d9a26.1707406936.git.yantar92@posteo.net>
In-Reply-To: <1185e7a3569048b465b697fe173ce50a3ab7f0cc.1707406936.git.yantar92@posteo.net>
References: <1185e7a3569048b465b697fe173ce50a3ab7f0cc.1707406936.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 6 Feb 2024 15:02:48 +0100
Subject: [PATCH v2 3/3] org-babel: New babel backend API function
 org-babel-session-buffer:<lang>

* lisp/ob-core.el (org-babel-session-buffer): New API function that
return session buffer, if such buffer exists and is live.
(org-babel-execute-src-block): Use `org-babel-session-buffer'.
* lisp/ob-lua.el (org-babel-session-buffer:lua):
* lisp/ob-python.el (org-babel-session-buffer:python): Provide API to
retrieve session buffer name.
(org-babel-python-initiate-session-by-key): Use
`org-babel-session-buffer:python'.
* etc/ORG-NEWS (Org babel backends are now expected to define an
additional API function ~org-babel-session-buffer:<lang>~): Declare
the API addition.

Link: https://orgmode.org/list/87r0hr9f3b.fsf@localhost
---
 etc/ORG-NEWS      | 11 +++++++++++
 lisp/ob-core.el   | 31 ++++++++++++++++++++++++++-----
 lisp/ob-lua.el    |  5 +++++
 lisp/ob-python.el |  8 ++++++--
 4 files changed, 48 insertions(+), 7 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 92d363b80..7ac56702d 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -13,6 +13,17 @@ Please send Org bug reports to mailto:emacs-orgmode@gnu.org.
 
 * Version 9.7 (not released yet)
 ** Important announcements and breaking changes
+*** Org babel backends are now expected to define an additional API function ~org-babel-session-buffer:<lang>~
+
+Org babel now uses session buffer (if it exists) to retrieve
+~default-directory~ environment during src block evaluation.
+
+By default, buffer named like session is checked.  All the backends
+that create sessions inside buffers named differently should provide a
+function ~org-babel-session-buffer:<lang>~.  The function must accept
+two arguments - session name and info list (as returned by
+~org-babel-get-src-block-info~); and return the session buffer name.
+
 *** Org mode now fontifies whole table lines (including newline) according to ~org-table~ face
 
 Previously, leading indentation and trailing newline in table rows
diff --git a/lisp/ob-core.el b/lisp/ob-core.el
index fd8d06c5d..bfeac257b 100644
--- a/lisp/ob-core.el
+++ b/lisp/ob-core.el
@@ -767,8 +767,30 @@ (defun org-babel--file-desc (params result)
     (`(:file-desc) result)
     (`(:file-desc . ,(and (pred stringp) val)) val)))
 
-(defvar *this*) ; Dynamically bound in `org-babel-execute-src-block'
-                ; and `org-babel-read'
+(defvar *this*)
+;; Dynamically bound in `org-babel-execute-src-block'
+;; and `org-babel-read'
+
+(defun org-babel-session-buffer (&optional info)
+  "Return buffer name for session associated with current code block.
+Return nil when no such live buffer with process exists.
+When INFO is non-nil, it should be a list returned by
+`org-babel-get-src-block-info'.
+This function uses org-babel-session-buffer:<lang> function to
+retrieve backend-specific session buffer name."
+  (declare-function org-babel-comint-buffer-livep "ob-comint" (buffer))
+  (when-let* ((info (or info (org-babel-get-src-block-info 'no-eval)))
+              (lang (nth 0 info))
+              (session (cdr (assq :session (nth 2 info))))
+              (cmd (intern (concat "org-babel-session-buffer:" lang)))
+              (buffer-name
+               (if (fboundp cmd)
+                   (funcall cmd session info)
+                 ;; Use session name as buffer name by default.
+                 session)))
+    (require 'ob-comint)
+    (when (org-babel-comint-buffer-livep buffer-name)
+      buffer-name)))
 
 ;;;###autoload
 (defun org-babel-execute-src-block (&optional arg info params executor-type)
@@ -842,9 +864,8 @@ (defun org-babel-execute-src-block (&optional arg info params executor-type)
 		 (default-directory
 		  (cond
 		   ((not dir) default-directory)
-                   ((when-let ((session (cdr (assq :session params))))
-                      (when (org-babel-comint-buffer-livep session)
-                        (buffer-local-value 'default-directory (get-buffer session)))))
+                   ((when-let ((session (org-babel-session-buffer info)))
+                      (buffer-local-value 'default-directory (get-buffer session))))
 		   ((member mkdirp '("no" "nil" nil))
 		    (file-name-as-directory (expand-file-name dir)))
 		   (t
diff --git a/lisp/ob-lua.el b/lisp/ob-lua.el
index a2b830ca3..b241fccdc 100644
--- a/lisp/ob-lua.el
+++ b/lisp/ob-lua.el
@@ -184,6 +184,11 @@ (defun org-babel-lua-with-earmuffs (session)
 	name
       (format "*%s*" name))))
 
+(defun org-babel-session-buffer:lua (session &optional _)
+  "Return session buffer name for SESSION."
+  (or (org-babel-lua-session-buffer session)
+      (org-babel-lua-with-earmuffs session)))
+
 (defun org-babel-lua-without-earmuffs (session)
 "Remove stars around *SESSION*, leaving SESSION."
   (let ((name (if (stringp session) session (format "%s" session))))
diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index cfc0fdcb6..89cdf4c47 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -260,6 +260,11 @@ (defun org-babel-python-without-earmuffs (session)
 	(substring name 1 (- (length name) 1))
       name)))
 
+(defun org-babel-session-buffer:python (session &optional _)
+  "Return session buffer name for SESSION."
+  (or (org-babel-python-session-buffer session)
+      (org-babel-python-with-earmuffs session)))
+
 (defun org-babel-python--python-util-comint-end-of-output-p ()
   "Return non-nil if the last prompt matches input prompt.
 Backport of `python-util-comint-end-of-output-p' to emacs28.  To
@@ -302,8 +307,7 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
 initialized session."
   (save-window-excursion
     (let* ((session (if session (intern session) :default))
-           (py-buffer (or (org-babel-python-session-buffer session)
-                          (org-babel-python-with-earmuffs session)))
+           (py-buffer (org-babel-session-buffer:python session))
            (python-shell-buffer-name
 	    (org-babel-python-without-earmuffs py-buffer))
            (existing-session-p (comint-check-proc py-buffer))
-- 
2.43.0


[-- Attachment #5: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Async Python src block behavior with :dir header property

[Bruno Barbier]

Ihor Radchenko <yantar92@posteo.net> writes:

> Jack Kamm <jackkamm@gmail.com> writes:
>
>> On executing any python session block I am getting the following error
>> which I think is caused by the above:
>>
>>   Debugger entered--Lisp error: (void-variable buffer-name)
>
> That's a mystery.

It looks like 'when-let*' doesn't accept symbols without values; it
needs real bindings.  The form `let*' assigns nil when there are no
values, but, with `when-let*', assigning nil would make the whole
`when-let' fails.

Here is the expansion when using a symbol without a value, which
explains both the void-variable error and the compiler message:

   #+begin_src elisp :results scalar
     (macroexpand-all '(when-let* (new-var) 7))
   #+end_src

   #+RESULTS:
   : (let* ((new-var (and t new-var))) (if new-var 7))



Bruno

Re: Async Python src block behavior with :dir header property

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

>>   Compiling /home/jack/src/org-mode/2024-01-async-file-results-dir/lisp/ob-core.el...
>>   
>>   In org-babel-session-buffer:
>>   ob-core.el:785:15: Warning: reference to free variable ‘buffer-name’
>
> But this is a real problem.
> See the attached updated version of the patch set.

Looks good, thanks. I tested it out it works without errors/warnings
now.

Re: Async Python src block behavior with :dir header property

[Ihor Radchenko]

Jack Kamm <jackkamm@gmail.com> writes:

>> See the attached updated version of the patch set.
>
> Looks good, thanks. I tested it out it works without errors/warnings
> now.

Applied, onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=13d0f8bf8
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=644bf846d
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=8c7313d39

Fixed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Async Python src block behavior with :dir header property

[Nasser Alkmim]

Excellent job Ihor and Jack.

On the same note, it would be convenient to have a hook to run after the
async execution returns the output.  Similar to
'org-babel-after-execute-hook'.  So I could, for instance, redisplay the
inline image previews.  Is there an easy way to do that?

-- 
Nasser Alkmim 
 +43 677 6408 9171

Re: Async Python src block behavior with :dir header property

[Ihor Radchenko]

Nasser Alkmim <nasser.alkmim@gmail.com> writes:

> On the same note, it would be convenient to have a hook to run after the
> async execution returns the output.  Similar to
> 'org-babel-after-execute-hook'.  So I could, for instance, redisplay the
> inline image previews.  Is there an easy way to do that?

Maybe we can add a hook for `org-babel-insert-result'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>