org-babel-execute-src-block filters characters from :session *shell* output

org-babel-execute-src-block filters characters from :session *shell* output

[Cook, Malcolm]

Hi,

While using:

	Org mode version 9.7.3 (9.7.3-2f1844 @ /home/mec/.dotfiles/emacs/.emacs.d/elpa/org-9.7.3/)
	GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo

Let me walk you through the issue:

#+begin_src sh
printf 'a\nb\nc\n>d\n<e\n'
#+end_src

#+RESULTS:
: a
: b
: c
: >d
: <e

Results look as expected. So far so good.

Now create a shell buffer (which will be named "*shell*")

#+begin_src elisp
(shell)
#+end_src

and try it again

#+begin_src sh
printf 'a\nb\nc\n>d\n<e\n'
#+end_src

#+RESULTS:
: a
: b
: c
: d
: <e

Huh? what happened to the ">" before the "d"?

Let's try it again.

First kill the *shell* buffer:

#+begin_src elisp
(let ((kill-buffer-query-functions nil)) (kill-buffer "*shell*"))
#+end_src

#+RESULTS:
: t


and try it again:

#+begin_src sh
printf 'a\nb\nc\n>d\n<e\n'
#+end_src

#+RESULTS:
: a
: b
: c
: >d
: <e

The ">" is back!

What is going on.

I'm pretty sure that `org-babel-comint--prompt-filter` is implicated which has had some recent changes (https://list.orgmode.org/87ttmn9fg0.fsf@localhost/T/) related to prompt filtering.

Any workarounds / patches most welcome!

Thanks!

~ Malcolm

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

"Cook, Malcolm" <MEC@stowers.org> writes:

> Let me walk you through the issue:
>
> #+begin_src sh
> printf 'a\nb\nc\n>d\n<e\n'
> #+end_src
>
> #+RESULTS:
> : a
> : b
> : c
> : >d
> : <e
>
> Results look as expected. So far so good.
>
> Now create a shell buffer (which will be named "*shell*")
>
> #+begin_src elisp
> (shell)
> #+end_src
>
> and try it again
>
> #+begin_src sh
> printf 'a\nb\nc\n>d\n<e\n'
> #+end_src
>
> #+RESULTS:
> : a
> : b
> : c
> : d
> : <e
>
> Huh? what happened to the ">" before the "d"?

I cannot reproduce.
May it be that you are setting global :session header argument?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Cook, Malcolm]

>"Cook, Malcolm" <mailto:MEC@stowers.org> writes:
>
>> Let me walk you through the issue:
>>
>> #+begin_src sh
>> printf 'a\nb\nc\n>d\n<e\n'
>> #+end_src
>>
>> #+RESULTS:
>> : a
>> : b
>> : c
>> : >d
>> : <e
>>
>> Results look as expected. So far so good.
>>
>> Now create a shell buffer (which will be named "*shell*")
>>
>> #+begin_src elisp
>> (shell)
>> #+end_src
>>
>> and try it again
>>
>> #+begin_src sh
>> printf 'a\nb\nc\n>d\n<e\n'
>> #+end_src
>>
>> #+RESULTS:
>> : a
>> : b
>> : c
>> : d
>> : <e
>>
>> Huh? what happened to the ">" before the "d"?
>
>I cannot reproduce.
>May it be that you are setting global :session header argument?

Yes, of course, so sorry, there is

#+PROPERTY: header-args:sh :session *shell* :results output

You should now be able to reproduce.

Thanks, please let me know if you have a fix, workaround, or lesson for me.

~Malcolm

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

"Cook, Malcolm" <MEC@stowers.org> writes:

>>> #+begin_src sh
>>> printf 'a\nb\nc\n>d\n<e\n'
>>> #+end_src
>>>
>>> #+RESULTS:
>>> : a
>>> : b
>>> : c
>>> : d
>>> : <e
>>>
>>> Huh? what happened to the ">" before the "d"?
>>
>>I cannot reproduce.
>>May it be that you are setting global :session header argument?
>
> Yes, of course, so sorry, there is
>
> #+PROPERTY: header-args:sh :session *shell* :results output
>
> You should now be able to reproduce.
>
> Thanks, please let me know if you have a fix, workaround, or lesson for me.

In theory, we may provide a workaround for this kind of problem. But it
would have pros and cons.

The underlying cause is limitation of Emacs API for interactive shells -
we cannot easily distinguish command output from prompt and other extra
staff your shell/other interactive command spits into the buffer.
So, we have to either filter output the prompts ourselves to get the
command output reliably or redirect output to files, where nothing
litters the actual output with prompts.

Redirecting output works fine when we do not use sessions and do not
care what is displayed in the *shell* buffer for users. For sessions, we
use more complex approach - filter out anything that looks like prompt.

The problem with bash is that prompt is ">", and, as you showed in your
example, having ">" in the commands output breaks everything.

Again, we know this problem, and we normally force the prompt to be
something more unique when creating sessions. But we do not touch the
prompt when you point Org to an existing interactive shell buffer - that
could be a surprise if Org mode changes the prompts by force in the
shells you use interactively.

TL;DR: It is complicated. When we pull one string out, several more
       get entangled.

As a practical workaround, just do not use *shell* session names and
session names that are the same as shell buffers you create manually.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Max Nikulin]

On 15/06/2024 20:19, Ihor Radchenko wrote:
> The underlying cause is limitation of Emacs API for interactive shells -
> we cannot easily distinguish command output from prompt and other extra
> staff your shell/other interactive command spits into the buffer.
> So, we have to either filter output the prompts ourselves to get the
> command output reliably or redirect output to files, where nothing
> litters the actual output with prompts.

Some shells support "semantic shell" that allows terminal applications 
e.g. to copy whole command output. It is based on escape sequences.

- 
https://docs.kde.org/stable5/en/konsole/konsole/semantic-shell-integration.html
- 
https://gitlab.freedesktop.org/Per_Bothner/specifications/blob/master/proposals/semantic-prompts.md
- https://github.com/tmux/tmux/issues/3064

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Cook, Malcolm]

>>>> #+begin_src sh
>>>> printf 'a\nb\nc\n>d\n<e\n'
>>>> #+end_src
>>>>
>>>> #+RESULTS:
>>>> : a
>>>> : b
>>>> : c
>>>> : d
>>>> : <e
>>>>
>>>> Huh? what happened to the ">" before the "d"?
>>>
>>>I cannot reproduce.
>>>May it be that you are setting global :session header argument?
>>
>> Yes, of course, so sorry, there is
>>
>> #+PROPERTY: header-args:sh :session *shell* :results output
>>
>> You should now be able to reproduce.
>>
>> Thanks, please let me know if you have a fix, workaround, or lesson for me.
>
>In theory, we may provide a workaround for this kind of problem. But it
>would have pros and cons.
>
>The underlying cause is limitation of Emacs API for interactive shells -
>we cannot easily distinguish command output from prompt and other extra
>staff your shell/other interactive command spits into the buffer.
>So, we have to either filter output the prompts ourselves to get the
>command output reliably or redirect output to files, where nothing
>litters the actual output with prompts.

Yeah, I've noticed this also especially with Org as it relates to R and ESS.

>
>Redirecting output works fine when we do not use sessions and do not
>care what is displayed in the *shell* buffer for users. For sessions, we
>use more complex approach - filter out anything that looks like prompt.
>
>The problem with bash is that prompt is ">", and, as you showed in your
>example, having ">" in the commands output breaks everything.
>
>Again, we know this problem, and we normally force the prompt to be
>something more unique when creating sessions. But we do not touch the
>prompt when you point Org to an existing interactive shell buffer - that
>could be a surprise if Org mode changes the prompts by force in the
>shells you use interactively.
>
>TL;DR: It is complicated. When we pull one string out, several more
>get entangled.

Might there be a solution wherein both the interactive shell buffer and Org are talking to a common underlying process?  I would expect such to be significantly more complicated, but perhaps better factored?  Not that I'm offering or capable of such a re-write 😉

>As a practical workaround, just do not use *shell* session names and
>session names that are the same as shell buffers you create manually.

Is there perhaps another practical workaround you might suggest to me involving a more intentional setting of the prompt and/or informing Org of my choice of prompt (e.g. perhaps via setting a regexp to detect exactly my prompt, and only when it is anchored to beginning of line)?

Thanks so much for your careful explanation and pointers, and of course all the work you do for Org ecosystem.  Perhaps time for my 1st trip to liberapay... 😉

~ Malcolm

>
>-- 
>Ihor Radchenko // yantar92,
>Org mode contributor,
>Learn more about Org mode at <https://orgmode.org>.
>Support Org development at <https://liberapay.com/org-mode>,
>or support my work at <https://liberapay.com/yantar92>
>

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> Some shells support "semantic shell" that allows terminal applications 
> e.g. to copy whole command output. It is based on escape sequences.
>
> - 
> https://docs.kde.org/stable5/en/konsole/konsole/semantic-shell-integration.html
> - 
> https://gitlab.freedesktop.org/Per_Bothner/specifications/blob/master/proposals/semantic-prompts.md

I am looking at this gitlab link, and they way it is implemented is
simply setting PROMPT in a way that adds these escape sequences to the
original PROMPT value.

In other words, it is a bit more advanced version of our approach with
setting prompt to a unique string - they surround the prompt with unique
chars instead.

In theory, we can do the same thing, adding some kind of invisible
unicode character around the prompt. We may even do it in the existing
prompts. Although one downside will be that the unicode character we add
may be carried along if the user copies the comint buffer contents.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

"Cook, Malcolm" <MEC@stowers.org> writes:

>>TL;DR: It is complicated. When we pull one string out, several more
>>get entangled.
>
> Might there be a solution wherein both the interactive shell buffer and Org are talking to a common underlying process?  I would expect such to be significantly more complicated, but perhaps better factored?  Not that I'm offering or capable of such a re-write 😉

comint buffer is doing exactly this - it is sending input (user
comments) to the underlying shell process and receiving the output from
that process, putting it back into the comit buffer.

Unfortunately, there is no simple way to distinguish real output, shell
echoing the submitted command, and shell prompt - shells do not provide
any such information. The best they can provide is splitting between
stdout and stderr. Alas.

>>As a practical workaround, just do not use *shell* session names and
>>session names that are the same as shell buffers you create manually.
>
> Is there perhaps another practical workaround you might suggest to me involving a more intentional setting of the prompt and/or informing Org of my choice of prompt (e.g. perhaps via setting a regexp to detect exactly my prompt, and only when it is anchored to beginning of line)?

That's also an option.
What you need to fiddle with is `comint-prompt-regexp'.

> Thanks so much for your careful explanation and pointers, and of course all the work you do for Org ecosystem.  Perhaps time for my 1st trip to liberapay... 😉

Thanks for the kind words.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Cook, Malcolm]

>>>TL;DR: It is complicated. When we pull one string out, several more
>>>get entangled.
>>
>> Might there be a solution wherein both the interactive shell buffer and Org are talking to a common underlying process? I would expect such to be significantly more complicated, but perhaps better factored? Not that I'm offering or capable of such a re-write 😉
>
>comint buffer is doing exactly this - it is sending input (user
>comments) to the underlying shell process and receiving the output from
>that process, putting it back into the comit buffer.
>
>Unfortunately, there is no simple way to distinguish real output, shell
>echoing the submitted command, and shell prompt - shells do not provide
>any such information. The best they can provide is splitting between
>stdout and stderr. Alas.
>
>>>As a practical workaround, just do not use *shell* session names and
>>>session names that are the same as shell buffers you create manually.
>>
>> Is there perhaps another practical workaround you might suggest to me involving a more intentional setting of the prompt and/or informing Org of my choice of prompt (e.g. perhaps via setting a regexp to detect exactly my prompt, and only when it is anchored to beginning of line)?
>
>That's also an option.
>What you need to fiddle with is `comint-prompt-regexp'.

This!  

Since my (bash) shell prompt is a (more or less) constant string (e.g. "myname@myhost> ").

So, my workaround is to:

    (setq comint-prompt-regexp "myname@myhost> ")

Then the filtering works perfectly.

Of course if I change my name, this will fail.  Or, more likely, connect to a different host within the shell.

Or if I change PS1 😉

It would be useful to automate this a little.

The variable needs to be set buffer-local to the shell buffer.

And it could possibly somehow ask the process for the value of PS1.

Any more TIPS on doing this?

Or perhaps advice that I shouldn't want to ...??? 😉

Cheers,

~ Malcolm

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Cook, Malcolm]

> So, my workaround is to:
> 
>     (setq comint-prompt-regexp "myname@myhost> ")
> 
> Then the filtering works perfectly.
> 
> Of course if I change my name, this will fail.  Or, more likely, connect to a
> different host within the shell.
> 
> Or if I change PS1 😉
> 
> It would be useful to automate this a little.
> 
> The variable needs to be set buffer-local to the shell buffer.
> 
> And it could possibly somehow ask the process for the value of PS1.

FWIW: the following gets at it...

(with-current-buffer "*shell*"  (setq comint-prompt-regexp   (buffer-substring-no-properties (pos-bol) (pos-eol))))

> 
> Any more TIPS on doing this?
> 
> Or perhaps advice that I shouldn't want to ...??? 😉
> 
> Cheers,
> 
> ~ Malcolm

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

"Cook, Malcolm" <MEC@stowers.org> writes:

> Since my (bash) shell prompt is a (more or less) constant string (e.g. "myname@myhost> ").
>
> So, my workaround is to:
>
>     (setq comint-prompt-regexp "myname@myhost> ")
>
> Then the filtering works perfectly.
>
> Of course if I change my name, this will fail.  Or, more likely, connect to a different host within the shell.
>
> Or if I change PS1 😉
>
> It would be useful to automate this a little.

There is internal buffer-local variable called
`org-babel-comint-prompt-regexp-old'. If you set it to the default
`comint-prompt-regexp', Org will attempt to use it if no prompt is
detected for longer than `org-babel-comint-fallback-regexp-threshold'
seconds.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

>> https://gitlab.freedesktop.org/Per_Bothner/specifications/blob/master/proposals/semantic-prompts.md
>
> I am looking at this gitlab link, and they way it is implemented is
> simply setting PROMPT in a way that adds these escape sequences to the
> original PROMPT value.
>
> In other words, it is a bit more advanced version of our approach with
> setting prompt to a unique string - they surround the prompt with unique
> chars instead.
>
> In theory, we can do the same thing, adding some kind of invisible
> unicode character around the prompt. We may even do it in the existing
> prompts. Although one downside will be that the unicode character we add
> may be carried along if the user copies the comint buffer contents.

Upon looking closer, it won't fly in general. In particular, multi-line
prompts will be a disaster - comint cannot handle them well no matter
what we add there.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 692 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> Again, we know this problem, and we normally force the prompt to be
> something more unique when creating sessions. But we do not touch the
> prompt when you point Org to an existing interactive shell buffer - that
> could be a surprise if Org mode changes the prompts by force in the
> shells you use interactively.

... on the other hand, Org mode simply hanging or emitting garbage is
also not great. So, maybe changing the prompt by force is good?

I am attaching tentative series of patches that will do exactly this -
setup prompt even in the existing shell buffers. I also changed the
babel prompt to be (1) shorter; (2) more "Babel"y.


[-- Attachment #2: 0001-Rename-org-babel-comint-prompt-regexp-old-to-.-fallb.patch --]
[-- Type: text/x-patch, Size: 7003 bytes --]

From aee2dd3f8ad0f1658ba2a222195351da9c9b4bf6 Mon Sep 17 00:00:00 2001
Message-ID: <aee2dd3f8ad0f1658ba2a222195351da9c9b4bf6.1719774270.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sun, 30 Jun 2024 20:36:25 +0200
Subject: [PATCH 1/4] Rename `org-babel-comint-prompt-regexp-old' to
 `...-fallback'

* lisp/ob-comint.el (org-babel-comint-prompt-regexp-fallback): Rename
to more accurate name.  Leave the old name as an alias.
(org-babel-comint--set-fallback-prompt): Use the new name.
(org-babel-comint-with-output):
(org-babel-comint-wait-for-output):
* lisp/ob-clojure.el (ob-clojure-eval-with-inf-clojure):
* lisp/ob-haskell.el (org-babel-interpret-haskell):
* lisp/ob-ruby.el (org-babel-ruby-initiate-session):
* lisp/ob-shell.el (org-babel-sh-initiate-session):
---
 lisp/ob-clojure.el |  4 ++--
 lisp/ob-comint.el  | 19 ++++++++++---------
 lisp/ob-haskell.el |  2 +-
 lisp/ob-ruby.el    |  2 +-
 lisp/ob-shell.el   |  2 +-
 5 files changed, 15 insertions(+), 14 deletions(-)

diff --git a/lisp/ob-clojure.el b/lisp/ob-clojure.el
index c7ebbbb95..98a66d1ec 100644
--- a/lisp/ob-clojure.el
+++ b/lisp/ob-clojure.el
@@ -237,8 +237,8 @@ (defun ob-clojure-eval-with-inf-clojure (expanded params)
 			     "clojure" (format "clojure -A%s" alias)
 			     cmd0)
 		    cmd0)))
-	(setq
-         org-babel-comint-prompt-regexp-old comint-prompt-regexp
+	(setq-local
+         org-babel-comint-prompt-regexp-fallback comint-prompt-regexp
          comint-prompt-regexp inf-clojure-comint-prompt-regexp)
 	(funcall-interactively #'inf-clojure cmd)
 	(goto-char (point-max))))
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 764927af7..4d4c90c5a 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -58,8 +58,9 @@ (defmacro org-babel-comint-in-buffer (buffer &rest body)
 	   (let ((comint-input-filter (lambda (_input) nil)))
 	     ,@body))))))
 
-(defvar-local org-babel-comint-prompt-regexp-old nil
+(defvar org-babel-comint-prompt-regexp-fallback nil
   "Fallback regexp used to detect prompt.")
+(defvaralias 'org-babel-comint-prompt-regexp-old 'org-babel-comint-prompt-regexp-fallback)
 
 (defcustom org-babel-comint-fallback-regexp-threshold 5.0
   "Waiting time until trying to use fallback regexp to detect prompt.
@@ -69,11 +70,11 @@ (defcustom org-babel-comint-fallback-regexp-threshold 5.0
   :package-version '(Org . "9.7"))
 
 (defun org-babel-comint--set-fallback-prompt ()
-  "Swap `comint-prompt-regexp' and `org-babel-comint-prompt-regexp-old'."
-  (when org-babel-comint-prompt-regexp-old
+  "Swap `comint-prompt-regexp' and `org-babel-comint-prompt-regexp-fallback'."
+  (when org-babel-comint-prompt-regexp-fallback
     (let ((tmp comint-prompt-regexp))
-      (setq comint-prompt-regexp org-babel-comint-prompt-regexp-old
-            org-babel-comint-prompt-regexp-old tmp))))
+      (setq comint-prompt-regexp org-babel-comint-prompt-regexp-fallback
+            org-babel-comint-prompt-regexp-fallback tmp))))
 
 (defun org-babel-comint--prompt-filter (string &optional prompt-regexp)
   "Remove PROMPT-REGEXP from STRING.
@@ -144,7 +145,7 @@ (defmacro org-babel-comint-with-output (meta &rest body)
 	     (accept-process-output
               (get-buffer-process (current-buffer))
               org-babel-comint-fallback-regexp-threshold)
-             (when (and org-babel-comint-prompt-regexp-old
+             (when (and org-babel-comint-prompt-regexp-fallback
                         (> (float-time (time-since start-time))
                            org-babel-comint-fallback-regexp-threshold)
                         (progn
@@ -154,7 +155,7 @@ (defmacro org-babel-comint-with-output (meta &rest body)
                              (re-search-forward
 			      (regexp-quote ,eoe-indicator) nil t)
 			     (re-search-forward
-                              org-babel-comint-prompt-regexp-old nil t)))))
+                              org-babel-comint-prompt-regexp-fallback nil t)))))
                (org-babel-comint--set-fallback-prompt))))
 	 ;; replace cut dangling text
 	 (goto-char (process-mark (get-buffer-process (current-buffer))))
@@ -189,14 +190,14 @@ (defun org-babel-comint-wait-for-output (buffer)
         (accept-process-output
          (get-buffer-process buffer)
          org-babel-comint-fallback-regexp-threshold)
-        (when (and org-babel-comint-prompt-regexp-old
+        (when (and org-babel-comint-prompt-regexp-fallback
                    (> (float-time (time-since start-time))
                       org-babel-comint-fallback-regexp-threshold)
                    (progn
 		     (goto-char comint-last-input-end)
 		     (save-excursion
 		       (re-search-forward
-                        org-babel-comint-prompt-regexp-old nil t))))
+                        org-babel-comint-prompt-regexp-fallback nil t))))
           (org-babel-comint--set-fallback-prompt))))))
 
 (defun org-babel-comint-eval-invisibly-and-wait-for-file
diff --git a/lisp/ob-haskell.el b/lisp/ob-haskell.el
index 05f340fa0..a96a28b6b 100644
--- a/lisp/ob-haskell.el
+++ b/lisp/ob-haskell.el
@@ -153,7 +153,7 @@ (defun org-babel-interpret-haskell (body params)
   (add-hook 'inferior-haskell-hook
             (lambda ()
               (setq-local
-               org-babel-comint-prompt-regexp-old comint-prompt-regexp
+               org-babel-comint-prompt-regexp-fallback comint-prompt-regexp
                comint-prompt-regexp
                (concat haskell-prompt-regexp "\\|^λ?> "))))
   (org-babel-haskell-with-session session params
diff --git a/lisp/ob-ruby.el b/lisp/ob-ruby.el
index d920fb585..24b31c18c 100644
--- a/lisp/ob-ruby.el
+++ b/lisp/ob-ruby.el
@@ -192,7 +192,7 @@ (defun org-babel-ruby-initiate-session (&optional session params)
             (when new-session?
               (with-current-buffer session-buffer
                 (setq-local
-                 org-babel-comint-prompt-regexp-old comint-prompt-regexp
+                 org-babel-comint-prompt-regexp-fallback comint-prompt-regexp
                  comint-prompt-regexp (concat "^" org-babel-ruby-prompt))
                 (insert org-babel-ruby-define-prompt ";")
                 (insert "_org_prompt_mode=conf.prompt_mode;conf.prompt_mode=:CUSTOM;")
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 32b029044..9222379a3 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -294,7 +294,7 @@ (defun org-babel-sh-initiate-session (&optional session _params)
                   (alist-get t org-babel-shell-set-prompt-commands))
               org-babel-sh-prompt))
             (setq-local
-             org-babel-comint-prompt-regexp-old comint-prompt-regexp
+             org-babel-comint-prompt-regexp-fallback comint-prompt-regexp
              comint-prompt-regexp
              (concat "^" (regexp-quote org-babel-sh-prompt)
                      " *"))
-- 
2.45.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-ob-shell-Arrange-unique-prompt-in-manually-started-s.patch --]
[-- Type: text/x-patch, Size: 2840 bytes --]

From c950bb79ad515e53bdac3d79da7b4352aa264e8a Mon Sep 17 00:00:00 2001
Message-ID: <c950bb79ad515e53bdac3d79da7b4352aa264e8a.1719774270.git.yantar92@posteo.net>
In-Reply-To: <aee2dd3f8ad0f1658ba2a222195351da9c9b4bf6.1719774270.git.yantar92@posteo.net>
References: <aee2dd3f8ad0f1658ba2a222195351da9c9b4bf6.1719774270.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sun, 30 Jun 2024 20:37:36 +0200
Subject: [PATCH 2/4] ob-shell: Arrange unique prompt in manually started shell
 buffers

* lisp/ob-shell.el (org-babel-sh--prompt-initialized): New variable
flag that indicates whether ob-shell changed the prompt in current
comint buffer.
(org-babel-sh-initiate-session): Set unique prompt in existing
sessions if it is not yet done.

Link: https://orgmode.org/list/87o782gx7o.fsf@localhost
---
 lisp/ob-shell.el | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 9222379a3..7b0d6ddab 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -276,16 +276,25 @@ (defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
 (defvar org-babel-sh-prompt "org_babel_sh_prompt> "
   "String to set prompt in session shell.")
 
+(defvar-local org-babel-sh--prompt-initialized nil
+  "When non-nil, ob-shell already initialized the prompt in current buffer.")
+
 (defalias 'org-babel-shell-initiate-session #'org-babel-sh-initiate-session)
 (defun org-babel-sh-initiate-session (&optional session _params)
   "Initiate a session named SESSION according to PARAMS."
   (when (and session (not (string= session "none")))
     (save-window-excursion
-      (or (org-babel-comint-buffer-livep session)
+      (or (and (org-babel-comint-buffer-livep session)
+               (buffer-local-value
+                'org-babel-sh--prompt-initialized
+                (get-buffer session))
+               session)
           (progn
-	    (shell session)
-            ;; Set unique prompt for easier analysis of the output.
-            (org-babel-comint-wait-for-output (current-buffer))
+            (if (org-babel-comint-buffer-livep session)
+                (set-buffer session)
+	      (shell session)
+              ;; Set unique prompt for easier analysis of the output.
+              (org-babel-comint-wait-for-output (current-buffer)))
             (org-babel-comint-input-command
              (current-buffer)
              (format
@@ -298,6 +307,7 @@ (defun org-babel-sh-initiate-session (&optional session _params)
              comint-prompt-regexp
              (concat "^" (regexp-quote org-babel-sh-prompt)
                      " *"))
+            (setq org-babel-sh--prompt-initialized t)
 	    ;; Needed for Emacs 23 since the marker is initially
 	    ;; undefined and the filter functions try to use it without
 	    ;; checking.
-- 
2.45.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-org-babel-sh-initiate-session-Fix-setting-non-standa.patch --]
[-- Type: text/x-patch, Size: 2243 bytes --]

From 21d9bcdc22c0b601e40316dd6df7394344d4cccf Mon Sep 17 00:00:00 2001
Message-ID: <21d9bcdc22c0b601e40316dd6df7394344d4cccf.1719774270.git.yantar92@posteo.net>
In-Reply-To: <aee2dd3f8ad0f1658ba2a222195351da9c9b4bf6.1719774270.git.yantar92@posteo.net>
References: <aee2dd3f8ad0f1658ba2a222195351da9c9b4bf6.1719774270.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sun, 30 Jun 2024 20:59:22 +0200
Subject: [PATCH 3/4] org-babel-sh-initiate-session: Fix setting non-standard
 prompt

* lisp/ob-shell.el (org-babel-sh-initiate-session): Set
`comint-prompt-regexp' early, _before_ evaluating prompt change.  This
way, we make sure that comint is not stuck trying to search for the
old prompt if the new prompt no longer matches `comint-prompt-regexp'.
---
 lisp/ob-shell.el | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 7b0d6ddab..f148fead7 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -295,6 +295,11 @@ (defun org-babel-sh-initiate-session (&optional session _params)
 	      (shell session)
               ;; Set unique prompt for easier analysis of the output.
               (org-babel-comint-wait-for-output (current-buffer)))
+            (setq-local
+             org-babel-comint-prompt-regexp-fallback comint-prompt-regexp
+             comint-prompt-regexp
+             (concat "^" (regexp-quote org-babel-sh-prompt)
+                     " *"))
             (org-babel-comint-input-command
              (current-buffer)
              (format
@@ -302,11 +307,6 @@ (defun org-babel-sh-initiate-session (&optional session _params)
                               org-babel-shell-set-prompt-commands))
                   (alist-get t org-babel-shell-set-prompt-commands))
               org-babel-sh-prompt))
-            (setq-local
-             org-babel-comint-prompt-regexp-fallback comint-prompt-regexp
-             comint-prompt-regexp
-             (concat "^" (regexp-quote org-babel-sh-prompt)
-                     " *"))
             (setq org-babel-sh--prompt-initialized t)
 	    ;; Needed for Emacs 23 since the marker is initially
 	    ;; undefined and the filter functions try to use it without
-- 
2.45.2


[-- Attachment #5: 0004-org-babel-sh-prompt-Use-cuneiform-break-symbol-as-un.patch --]
[-- Type: text/x-patch, Size: 1581 bytes --]

From 5b98fef45f262e46323241c0482bf81573cdb763 Mon Sep 17 00:00:00 2001
Message-ID: <5b98fef45f262e46323241c0482bf81573cdb763.1719774270.git.yantar92@posteo.net>
In-Reply-To: <aee2dd3f8ad0f1658ba2a222195351da9c9b4bf6.1719774270.git.yantar92@posteo.net>
References: <aee2dd3f8ad0f1658ba2a222195351da9c9b4bf6.1719774270.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sun, 30 Jun 2024 21:00:17 +0200
Subject: [PATCH 4/4] =?UTF-8?q?org-babel-sh-prompt:=20Use=20cuneiform=20"b?=
 =?UTF-8?q?reak"=20=F0=92=86=B8=20symbol=20as=20unique=20prompt?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/ob-shell.el (org-babel-sh-prompt): Change the value of unique
prompt from long prompt string to a much shorter, but still unlikely
to appear in output, single-char prompt.

Cuneiform because Sumerian, Babylon, Tower ob Babel :)
---
 lisp/ob-shell.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index f148fead7..05bded918 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -273,7 +273,7 @@ (defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
   "String to indicate that evaluation has completed.")
 (defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
   "String to indicate that evaluation has completed.")
-(defvar org-babel-sh-prompt "org_babel_sh_prompt> "
+(defvar org-babel-sh-prompt "𒆸 "
   "String to set prompt in session shell.")
 
 (defvar-local org-babel-sh--prompt-initialized nil
-- 
2.45.2


[-- Attachment #6: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Phil]

* [2024-06-16 14:47] Max Nikulin:
>> On 15/06/2024 20:19, Ihor Radchenko wrote:
>>> The underlying cause is a limitation of Emacs API for
>>> interactive shells - we cannot easily distinguish
>>> command output from prompt and other extra stuff
>>> your shell/other interactive command spits into the
>>> buffer.
>> Some shells support "semantic shell" that allows
>> terminal applications e.g. to copy whole command
>> output. It is based on escape sequences.
>> - https://gitlab.freedesktop.org/Per_Bothner/specifications/blob/master/proposals/semantic-prompts.md
>> - http://per.bothner.com/blog/2019/shell-integration-proposal/
> 
> it is implemented by simply setting PROMPT in a way
> that adds these escape sequences to the original
> PROMPT value.
> one downside will be that the unicode character we
> add may be carried along if the user copies the
> comint buffer contents.

Perhaps a minor-mode would get to the rescue
to filter copies?

I'd like to add a few general remarks about *error status*.
I'm starting to notice there are not much subprocesses
that to do get called through =call-process= with
‘(REAL-DESTINATION ERROR-DESTINATION)’ kept as
separate.  You all know how useful stdin/out/err and
the return codes are, but they're absent from Babel.

That's why a failed command will still provide a result
— an empty array for instance, something that looks
legit even when it's not.  I had again this experience
with =ob-http :pretty= and =ob-mongo= lastly: errors are
silenced.

Comint reads and write from/to terminals with all
three standard channels together. It's meant to
provide requirements to communicate with non-POSIX
terminals too. It's a great library but perhaps
there's a need to specialize things further when
dealing with one shell?

So In Emacs, those specs would land either in one precise
(possibly low-level) shell extension or a generic one that
would once more care for the many idiosyncracies every one
of them have.

Something I figured when dealing with ob-sql-session is
that if several commands are given in one batch (what a
babel source block is in many cases), they can either be
all run until the end or be stopped when an error is
met. In both cases, knowing the number of errors met is
possible in one shell that do agree with the convention
of returning a value for every command (or command line).
Otherwise, a terminal "reader" has to reinterpret the
results.

[[info:elisp#Synchronous
  Processes][elisp#Synchronous Processes]] says :

    You can’t directly specify a buffer to put the
    error output in; that is too difficult to
    implement.

Can someone explain why?


Phil

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

Phil <pe@7d.nz> writes:

> I'd like to add a few general remarks about *error status*.
> I'm starting to notice there are not much subprocesses
> that to do get called through =call-process= with
> ‘(REAL-DESTINATION ERROR-DESTINATION)’ kept as
> separate.  You all know how useful stdin/out/err and
> the return codes are, but they're absent from Babel.

They are not.
See `org-babel-eval-error-notify'.
Not for sessions though.

> That's why a failed command will still provide a result
> — an empty array for instance, something that looks
> legit even when it's not.  I had again this experience
> with =ob-http :pretty= and =ob-mongo= lastly: errors are
> silenced.

The API is there, as long as individual backends can separate stderr
from the rest.

> Comint reads and write from/to terminals with all
> three standard channels together. It's meant to
> provide requirements to communicate with non-POSIX
> terminals too. It's a great library but perhaps
> there's a need to specialize things further when
> dealing with one shell?

> So In Emacs, those specs would land either in one precise
> (possibly low-level) shell extension or a generic one that
> would once more care for the many idiosyncracies every one
> of them have.

It would be nice indeed, but it will be a huge work.

> Something I figured when dealing with ob-sql-session is
> that if several commands are given in one batch (what a
> babel source block is in many cases), they can either be
> all run until the end or be stopped when an error is
> met. In both cases, knowing the number of errors met is
> possible in one shell that do agree with the convention
> of returning a value for every command (or command line).

May you elaborate?

> Otherwise, a terminal "reader" has to reinterpret the
> results.
>
> [[info:elisp#Synchronous
>   Processes][elisp#Synchronous Processes]] says :
>
>     You can’t directly specify a buffer to put the
>     error output in; that is too difficult to
>     implement.
>
> Can someone explain why?

No idea. You better ask on emacs-devel.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Phil]

* [2024-07-02 22:05] Ihor Radchenko:
> Phil Estival <pe@7d.nz> writes:
> 
>> I'd like to add a few general remarks about *error status*.
>> I'm starting to notice there are not much subprocesses
>> that to do get called through =call-process= with
>> ‘(REAL-DESTINATION ERROR-DESTINATION)’ kept as
>> separate.  You all know how useful stdin/out/err and
>> the return codes are, but they're absent from Babel.
> 
> They are not.
> See `org-babel-eval-error-notify'.
> Not for sessions though.

That's right. The situation is fine when there is
one synchronous command line. And I admit too much
"interactivity" may put astray from strict reproducible
frames.

>> When several commands are given in one batch (what a
>> babel source block is in many cases), they can either be
>> all run
I should rather say "sent to a compiler or interpreter process"
>> until the end or be stopped when an error is
>> met. In both cases, knowing the number of errors met is
>> possible in one shell that do agree with the convention
>> of returning a value for every command (or command line).
>
> May you elaborate?

Source code is given to a compiler one character at a
time. It's also typed in a terminal one character at
a time, and when <return> is hit, a line can be sent to
the underlying interpreter.  Programming environments
often run a parallel evaluator to provide feedback,
completions, suggestions, and so on, to the programmer,
on every keystroke.

Reproducible research will only provides the source:
a text, made of syntactic atoms.
On the host on which this text is read can coexist
numerous evaluators, and while they can virtually be
anything, the basic environment is a syntax colorator
and a command processor that activates only when it is
asked to.

The error management channel is important.  Only a very
small fraction of errors should be silenced, and we
must not have feedback displaying that a function ran
correctly when it did not.

Let's decouple an interactive shell from its
compiler/interpreter (setting aside a Lisp
evaluator that can dodge most of the problems):

In many programming languages the source code can
indeed be assimilated line by line by the
interpreter through an interactive shell.
- It returns nothing or an error after a definition
   or a silent directive, then the prompt.
- One or more values as a result or errors, then the prompt,
   when calling functions and procedures.
- It can expect more input and more lines before
   terminating a statement, returning a secondary
   prompt until that statement completes or fails.

The terminal display is organized from programming
interactions but the separation of channels is needed
for further processing.

1) We should avoid filtering output with regexp acrobatics.
2) Communication with the process should go
    through a function that can display
    inputs and outputs and errors in three different channels
    before sending it to the process.
    Communicating directly to the process should remain
    possible, but it's a short-circuit.

+ In : code block.
    -  Headers and parameters are given upon session
       initialization.
+ Out : program output and exit code.  This we have.

+ Err : file, line, error code, message,
      including the prompt which goes to stderr.
    + The line number where the error occurs
      related to a source block. There may be extra code
      wrapping a block and noweb may complicate
      things even more.
      The situation to avoid is a line number unrelated
      to the edited source block.
      (Weaving and tangling programs is fine for me
      I'm only thinking out loud here).

Example:
Errors goes in one file buffer
and a filter wraps them in blocks.
file:/tmp/babel-uuid/fiction.err
#+begin_err :lang fiction :err 5 :line 2
Parse error: near "?": syntax error
   ? ;
   ^--- error here
#+end_err


Should all commands in a block be run ?
It depends on the language of course, but usually, no,
the execution stops and drops into a backtrace
or raises an exception pointing to the stack frames.
Dash has =set -e= (that doesn't always behave as expected).
Sqlite has =.bail= for noninteractive sessions.

And now here is my problem.
Am I missing something about carriage return ?
A differentiation problem between what happen
when pressing C-q C-j in a shell, and <return> ?

If ob-sql-session concatenates the SQL
commands on one line, sqlite is able to stop on error.
#+begin_src sql-session :engine sqlite :session A
   select 1;  raise;   select 2;
   select 33;
#+end_src

#+RESULTS:
: 1
: Parse error: near "raise": syntax error
:   raise;   select 2; select 33;
:   ^--- error here

If it does not, a newline terminates the command,
the prompt is displayed and the next line is
an other command. Which is not the desirable effect
as there's no way to halt the remaining commands.

#+RESULTS:
: 1
: Parse error: near "raise": syntax error
:   raise;   select 2;
:   ^--- error here
: 33


At last, I would reformulate the above paragraph dealing
with a "batch of commands" like so:

- getting the number of errors met (on a shell that
   would not stop on error)
- and agreeing with the convention of returning a value
   for every command — or command line or function

means: returning [Result,Error] conveyed to different channels.
Quite naturally an interpreter or an interactive shell
implemented like so also return [R,E] for every definitions,
calls and statements as input on its command line.
The prompt is only meant for interactive display,
can be anything and change during the session.
The indication of a command termination being the value
held in Error provided on stderr. The terminal can then
acknowledge the returned value by incrementing counters
displayed into that prompt.

--
Phil Estival

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

Phil <pe@7d.nz> writes:

> The error management channel is important.  Only a very
> small fraction of errors should be silenced, and we
> must not have feedback displaying that a function ran
> correctly when it did not.

Sure. I completely agree.

> Let's decouple an interactive shell from its
> compiler/interpreter (setting aside a Lisp
> evaluator that can dodge most of the problems):
>
> In many programming languages the source code can
> indeed be assimilated line by line by the
> interpreter through an interactive shell.
> - It returns nothing or an error after a definition
>    or a silent directive, then the prompt.
> - One or more values as a result or errors, then the prompt,
>    when calling functions and procedures.
> - It can expect more input and more lines before
>    terminating a statement, returning a secondary
>    prompt until that statement completes or fails.
>
> The terminal display is organized from programming
> interactions but the separation of channels is needed
> for further processing.
>
> 1) We should avoid filtering output with regexp acrobatics.

It is indeed not ideal. But it is hard to do better.
We simply do not have the means.

Of course, if someone can find a more robust way to decouple errors,
actual evaluation results, prompts (primary, secondary, etc), and the
interpreter echoing the input, it would be a patch we will happily
accept.

> 2) Communication with the process should go
>     through a function that can display
>     inputs and outputs and errors in three different channels
>     before sending it to the process.
>     Communicating directly to the process should remain
>     possible, but it's a short-circuit.

Yes, of course. But how?

> + Err : file, line, error code, message,
>       including the prompt which goes to stderr.
>     + The line number where the error occurs
>       related to a source block. There may be extra code
>       wrapping a block and noweb may complicate
>       things even more.
>       The situation to avoid is a line number unrelated
>       to the edited source block.
>       (Weaving and tangling programs is fine for me
>       I'm only thinking out loud here).

> Example:
> Errors goes in one file buffer
> and a filter wraps them in blocks.
> file:/tmp/babel-uuid/fiction.err
> #+begin_err :lang fiction :err 5 :line 2
> Parse error: near "?": syntax error
>    ? ;
>    ^--- error here
> #+end_err

This would be nice to have for sessions, yes. We already have this for
many non-session code blocks.

> Should all commands in a block be run ?
> It depends on the language of course, but usually, no,
> the execution stops and drops into a backtrace
> or raises an exception pointing to the stack frames.
> Dash has =set -e= (that doesn't always behave as expected).
> Sqlite has =.bail= for noninteractive sessions.

I am not sure if I agree, but your preference is a valid preference some
people might have. Our defaults are to send the whole block, which is an
ok option as well.

> And now here is my problem.
> Am I missing something about carriage return ?
> A differentiation problem between what happen
> when pressing C-q C-j in a shell, and <return> ?
>
> If ob-sql-session concatenates the SQL
> commands on one line, sqlite is able to stop on error.
> #+begin_src sql-session :engine sqlite :session A
>    select 1;  raise;   select 2;
>    select 33;
> #+end_src
>
> #+RESULTS:
> : 1
> : Parse error: near "raise": syntax error
> :   raise;   select 2; select 33;
> :   ^--- error here
>
> If it does not, a newline terminates the command,
> the prompt is displayed and the next line is
> an other command. Which is not the desirable effect
> as there's no way to halt the remaining commands.

This is simply how comint.el buffers work.
C-q C-j will enter literal newline. <return> will send the command
entered so far (including literal newlines) as input stream to the
interpreter. All at once.

The reason why comint does not do line-by-line is simply because it has
limited notion of the interpreter syntax - multiple lines may or may not
be several commands. Think of \ continuation in bash.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Max Nikulin]

On 18/06/2024 00:57, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> Some shells support "semantic shell" that allows terminal applications
>> e.g. to copy whole command output. It is based on escape sequences.
>> https://gitlab.freedesktop.org/Per_Bothner/specifications/blob/master/proposals/semantic-prompts.md
> 
> I am looking at this gitlab link, and they way it is implemented is
> simply setting PROMPT in a way that adds these escape sequences to the
> original PROMPT value.

It does a bit more. There are escape sequences to mark user input, 
program output, and to report program exit status. I had in mind support 
of this protocol in comint, so Org babel session make take advantage of 
it if the feature is available for a specific interpreter.

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Max Nikulin]

On 01/07/2024 02:08, Ihor Radchenko wrote:
> +++ b/lisp/ob-shell.el
> @@ -273,7 +273,7 @@ (defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
>    "String to indicate that evaluation has completed.")
>  (defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
>    "String to indicate that evaluation has completed.")
> -(defvar org-babel-sh-prompt "org_babel_sh_prompt> "
> +(defvar org-babel-sh-prompt "𒆸 "
>    "String to set prompt in session shell.")

I would use a longer prompt to reduce a chance that these pair of 
characters appear in output of some program.

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> +(defvar org-babel-sh-prompt "𒆸 "
>>    "String to set prompt in session shell.")
>
> I would use a longer prompt to reduce a chance that these pair of 
> characters appear in output of some program.

I'd say that the odds that an output contains cuneiform are quite slim.
Especially since they should not just appear in the output, but they
should appear in the output and also at bol.

I was thinking to add some kind of zero-width space character to be even
more sure. I think that should be good enough in practice. (and if not,
we can advise users to change this variable).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>>> Some shells support "semantic shell" that allows terminal applications
>>> e.g. to copy whole command output. It is based on escape sequences.
>>> https://gitlab.freedesktop.org/Per_Bothner/specifications/blob/master/proposals/semantic-prompts.md
>> 
>> I am looking at this gitlab link, and they way it is implemented is
>> simply setting PROMPT in a way that adds these escape sequences to the
>> original PROMPT value.
>
> It does a bit more. There are escape sequences to mark user input, 
> program output, and to report program exit status. I had in mind support 
> of this protocol in comint, so Org babel session make take advantage of 
> it if the feature is available for a specific interpreter.

If comint can support this, Org should follow. But we first need to get
it to comint.el. Will you be willing to?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-babel-execute-src-block filters characters from :session *shell* output

[Phil]

* [2024-07-06 13:36] Max Nikulin:
> On 18/06/2024 00:57, Ihor Radchenko wrote:
>> Max Nikulin writes:
>>
>>> Some shells support "semantic shell" that allows terminal applications
>>> e.g. to copy whole command output. It is based on escape sequences.
>>> https://gitlab.freedesktop.org/Per_Bothner/specifications/blob/master/proposals/semantic-prompts.md
>>
>> I am looking at this gitlab link, and they way it is implemented is
>> simply setting PROMPT in a way that adds these escape sequences to the
>> original PROMPT value.
> 
> It does a bit more. There are escape sequences to mark user input, 
> program output, and to report program exit status. I had in mind support 
> of this protocol in comint, so Org babel session make take advantage of 
> it if the feature is available for a specific interpreter.
> 

I imagine it organized in such a way that it can communicate with a
program able in the first place to talk natively with std in/out/err,
separating input, output, exit/result code and the prompt.  By means
of escape sequences and a filter, a given shell can talk to that other
layer of the protocol. The insertion of escape sequences stays
possible with filters for any given shell.

RE: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:
> ... on the other hand, Org mode simply hanging or emitting garbage is
> also not great. So, maybe changing the prompt by force is good?
>
> I am attaching tentative series of patches that will do exactly this -
> setup prompt even in the existing shell buffers. I also changed the
> babel prompt to be (1) shorter; (2) more "Babel"y.

Applied, onto main, after adding zero width space to improve prompt
uniqueness further.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

RE: org-babel-execute-src-block filters characters from :session *shell* output AND Re: [BUG] conda doesn't work in ob-shell sessions

[Cook, Malcolm]

# -*-  org-confirm-babel-evaluate: nil; -*-

Excuse the top-posting.  I’m reviving two old-ish threads and felt it best.

I have been struggling with interrelated issues raised in

 - https://list.orgmode.org/87jznda90u.fsf@localhost/#t
 - https://list.orgmode.org/87le1bc8j3.fsf@localhost/

I expect I am using all the patches offered in addressing these given
my recent build from main.  However, in my hands, I find they still
easily allow for mistakes identifying prompts in code block results.

In this demonstration, I am extending the approach to inquiry begun by Jack in
https://list.orgmode.org/87ttzn1mai.fsf@gmail.com/

#+begin_src emacs-lisp
   (emacs-version)
#+end_src

#+RESULTS:
: GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.15.12, Xaw3d scroll bars)
:  of 2024-09-04

#+begin_src emacs-lisp
   (org-version nil t)
#+end_src

#+RESULTS:
: Org mode version 9.7.13 (9.7.13-8566bc @ /home/mec/.emacs.d/elpa/org-9.7.13/)



#+begin_src emacs-lisp
  (org-babel-do-load-languages
    'org-babel-load-languages
    '((shell . t)))
#+end_src

#+RESULTS:

Here I define two org code blocks I will use repeatedly below:

#+name:test_filter
#+begin_src shell :session *shell* :results output
  printf "a\nb\nc\n>d\n<e\nf>\nggg ggg>\nhhh hhh+\na\n"
#+end_src

#+name:shell_prompt_info
#+begin_src elisp
  (with-current-buffer "*shell*"
    (format "[comint-prompt-regexp]=[%s]\n[org-babel-comint-prompt-regexp-old]=[%s]" comint-prompt-regexp org-babel-comint-prompt-regexp-old))
#+end_src

#+caption: The results looks good - the output apparently is not confused as being prompt.
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: >d
: <e
: f>
: ggg ggg>
: hhh hhh+
: a

#+caption: take a look at the prompt variables.
#+call:shell_prompt_info()

#+RESULTS:
: [comint-prompt-regexp]=[^org_babel_sh_prompt>  *]
: [org-babel-comint-prompt-regexp-old]=[^[^#$%>
: ]*[#$%>] *]

#+caption: check on conda's availabiity & version
#+begin_src shell :session *shell* :results output
conda --version
#+end_src

#+RESULTS:
: conda 24.7.1

#+begin_src shell :session *shell* :results output
conda create --yes --name myenv python=3.9
#+end_src

#+RESULTS:
#+begin_example
... abbreviated...

To activate this environment, use

conda activate myenv

To deactivate an active environment, use

conda deactivate
#+end_example

#+begin_src shell :session *shell* :results output
  conda activate myenv
#+end_src

#+RESULTS:

#+begin_src shell :session *shell* :results output
which python
#+end_src

#+RESULTS:
: /n/projects/mec/SRSCHPC2/local/inst/Mambaforge/24.3.0-0/envs/myenv/bin/python

#+caption: alas, the output of test_filter is changed.  Some lines are gone missing and some are changed.
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: Observe the prompts have changed.  Perhaps this is related issue?
#+call:shell_prompt_info()

#+RESULTS:
: [comint-prompt-regexp]=[^[^#$%>
: ]*[#$%>] *]
: [org-babel-comint-prompt-regexp-old]=[^org_babel_sh_prompt>  *]

#+caption:  can we restore by deactivating the environment?
#+begin_src shell :session *shell* :results output
  conda deactivate  
#+end_src

#+RESULTS:

#+caption: alas, no:
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: how about by resetting the prompt
#+begin_src shell :session *shell* :results output
  PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2=
#+end_src

#+RESULTS:

#+caption: alas, again, no
#+call: test_filter()
#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: perhaps restoring the prompt variables will recover?
#+begin_src elisp
    (with-current-buffer "*shell*"
      (setq-local comint-prompt-regexp "^org_babel_sh_prompt>  *"
      	      org-babel-comint-prompt-regexp-old "[^[^#$%>
]*[#$%>] *"))
#+end_src

#+RESULTS:
: [^[^#$%>
: ]*[#$%>] *

#+caption: YES!
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: >d
: <e
: f>
: ggg ggg>
: hhh hhh+
: a

I'm unsure what change this argues for, but I think it pretty clearly
demonstrates the ongoing issue.

In the above, I am exclusively allowing org/ob/comint to "own" the shell
buffer, and not interact with it, as recommended earlier by Ivor.

I have tried the above after first calling `(shell)` and find
variations on the above occur.  I would like to be able to 'share' the
*shell* buffer with org/ob/comint but expect resolving the
non-interactive case should possibly lay foundation.

I would additional like to layer in working with remote shells
(e.g. `:dir "/ssh:me@host:~/`) and have tried but this is just
layering in complexity on the localhost case so I'm backing off for
now.

What else can I report or test?

RE: org-babel-execute-src-block filters characters from :session *shell* output AND Re: [BUG] conda doesn't work in ob-shell sessions

[Cook, Malcolm]

I have a new workaround, which is to configure conda to not change the prompt.

    conda config --set changeps1 false

As discussed elsewhere:

	Do not modify prompt by default #10928
	https://github.com/conda/conda/issues/10928

I generally would not want to do this in normal interactive use, but in this context, am going to proceed.

I think it is preferable to trying to improve on prompt tracking.  That appears just to lead to more convolutions.

Thoughts?

> # -*-  org-confirm-babel-evaluate: nil; -*-
> 
> Excuse the top-posting.  I’m reviving two old-ish threads and felt it best.
> 
> I have been struggling with interrelated issues raised in
> 
>  - https://list.orgmode.org/87jznda90u.fsf@localhost/#t
>  - https://list.orgmode.org/87le1bc8j3.fsf@localhost/
> 
> I expect I am using all the patches offered in addressing these given my recent
> build from main.  However, in my hands, I find they still easily allow for
> mistakes identifying prompts in code block results.
> 
> In this demonstration, I am extending the approach to inquiry begun by Jack
> in https://list.orgmode.org/87ttzn1mai.fsf@gmail.com/
> 
> #+begin_src emacs-lisp
>    (emacs-version)
> #+end_src
> 
> #+RESULTS:
> : GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version
> 1.15.12, Xaw3d scroll bars)
> :  of 2024-09-04
> 
> #+begin_src emacs-lisp
>    (org-version nil t)
> #+end_src
> 
> #+RESULTS:
> : Org mode version 9.7.13 (9.7.13-8566bc @ /home/mec/.emacs.d/elpa/org-
> 9.7.13/)
> 
> 
> 
> #+begin_src emacs-lisp
>   (org-babel-do-load-languages
>     'org-babel-load-languages
>     '((shell . t)))
> #+end_src
> 
> #+RESULTS:
> 
> Here I define two org code blocks I will use repeatedly below:
> 
> #+name:test_filter
> #+begin_src shell :session *shell* :results output
>   printf "a\nb\nc\n>d\n<e\nf>\nggg ggg>\nhhh hhh+\na\n"
> #+end_src
> 
> #+name:shell_prompt_info
> #+begin_src elisp
>   (with-current-buffer "*shell*"
>     (format "[comint-prompt-regexp]=[%s]\n[org-babel-comint-prompt-regexp-
> old]=[%s]" comint-prompt-regexp org-babel-comint-prompt-regexp-old))
> #+end_src
> 
> #+caption: The results looks good - the output apparently is not confused as
> being prompt.
> #+call: test_filter()
> 
> #+RESULTS:
> : a
> : b
> : c
> : >d
> : <e
> : f>
> : ggg ggg>
> : hhh hhh+
> : a
> 
> #+caption: take a look at the prompt variables.
> #+call:shell_prompt_info()
> 
> #+RESULTS:
> : [comint-prompt-regexp]=[^org_babel_sh_prompt>  *]
> : [org-babel-comint-prompt-regexp-old]=[^[^#$%>
> : ]*[#$%>] *]
> 
> #+caption: check on conda's availabiity & version #+begin_src shell :session
> *shell* :results output conda --version #+end_src
> 
> #+RESULTS:
> : conda 24.7.1
> 
> #+begin_src shell :session *shell* :results output conda create --yes --name
> myenv python=3.9 #+end_src
> 
> #+RESULTS:
> #+begin_example
> ... abbreviated...
> 
> To activate this environment, use
> 
> conda activate myenv
> 
> To deactivate an active environment, use
> 
> conda deactivate
> #+end_example
> 
> #+begin_src shell :session *shell* :results output
>   conda activate myenv
> #+end_src
> 
> #+RESULTS:
> 
> #+begin_src shell :session *shell* :results output which python #+end_src
> 
> #+RESULTS:
> : /n/projects/mec/SRSCHPC2/local/inst/Mambaforge/24.3.0-
> 0/envs/myenv/bin/python
> 
> #+caption: alas, the output of test_filter is changed.  Some lines are gone
> missing and some are changed.
> #+call: test_filter()
> 
> #+RESULTS:
> : a
> : b
> : c
> : d
> : <e
> :
> : hhh hhh+
> : a
> 
> #+caption: Observe the prompts have changed.  Perhaps this is related issue?
> #+call:shell_prompt_info()
> 
> #+RESULTS:
> : [comint-prompt-regexp]=[^[^#$%>
> : ]*[#$%>] *]
> : [org-babel-comint-prompt-regexp-old]=[^org_babel_sh_prompt>  *]
> 
> #+caption:  can we restore by deactivating the environment?
> #+begin_src shell :session *shell* :results output
>   conda deactivate
> #+end_src
> 
> #+RESULTS:
> 
> #+caption: alas, no:
> #+call: test_filter()
> 
> #+RESULTS:
> : a
> : b
> : c
> : d
> : <e
> :
> : hhh hhh+
> : a
> 
> #+caption: how about by resetting the prompt #+begin_src shell :session
> *shell* :results output
>   PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2= #+end_src
> 
> #+RESULTS:
> 
> #+caption: alas, again, no
> #+call: test_filter()
> #+RESULTS:
> : a
> : b
> : c
> : d
> : <e
> :
> : hhh hhh+
> : a
> 
> #+caption: perhaps restoring the prompt variables will recover?
> #+begin_src elisp
>     (with-current-buffer "*shell*"
>       (setq-local comint-prompt-regexp "^org_babel_sh_prompt>  *"
>       	      org-babel-comint-prompt-regexp-old "[^[^#$%> ]*[#$%>] *"))
> #+end_src
> 
> #+RESULTS:
> : [^[^#$%>
> : ]*[#$%>] *
> 
> #+caption: YES!
> #+call: test_filter()
> 
> #+RESULTS:
> : a
> : b
> : c
> : >d
> : <e
> : f>
> : ggg ggg>
> : hhh hhh+
> : a
> 
> I'm unsure what change this argues for, but I think it pretty clearly
> demonstrates the ongoing issue.
> 
> In the above, I am exclusively allowing org/ob/comint to "own" the shell
> buffer, and not interact with it, as recommended earlier by Ivor.
> 
> I have tried the above after first calling `(shell)` and find variations on the
> above occur.  I would like to be able to 'share' the
> *shell* buffer with org/ob/comint but expect resolving the non-interactive
> case should possibly lay foundation.
> 
> I would additional like to layer in working with remote shells (e.g. `:dir
> "/ssh:me@host:~/`) and have tried but this is just layering in complexity on
> the localhost case so I'm backing off for now.
> 
> What else can I report or test?

RE: org-babel-execute-src-block filters characters from :session *shell* output AND Re: [BUG] conda doesn't work in ob-shell sessions

[Ihor Radchenko]

"Cook, Malcolm" <MEC@stowers.org> writes:

> # -*-  org-confirm-babel-evaluate: nil; -*-
> ...

Duplicate of
https://list.orgmode.org/orgmode/DS7PR20MB4671E61718CEC1FD86DAC722BE4D2@DS7PR20MB4671.namprd20.prod.outlook.com/

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>