[BUG] conda doesn't work in ob-shell sessions

[BUG] conda doesn't work in ob-shell sessions

[Matt]

 ---- On Sun, 12 Feb 2023 22:16:16 -0500  Jack Kamm  wrote --- 
 
 > But I also noticed another prompt-related issue: conda doesn't seem to
 > work in ob-shell sessions anymore. That is a bigger problem for my use
 > case.

Could you elaborate?

It looks like conda has a new init (from what I remember) which requires users to run a `conda init <shell>' command in order to create a new environment.  For example, I needed to run `conda init bash.'  This added garbage to my .bashrc which automatically enabled the base environment.  I was able to do this from ob-shell, however it required me to reload the shell (I reloaded Emacs to be sure I got the latest environment variables).  After this, I was able to create and activate a new conda environment.

1. M-x shell
2. Executing the following in sequence:

#+begin_src emacs-lisp
(org-version nil t)
#+end_src

#+RESULTS:
: Org mode version 9.6.1 (release_9.6.1-250-ge6353d @ /home/user/Projects/org-mode/lisp/)

#+begin_src emacs-lisp
(org-babel-do-load-languages
  'org-babel-load-languages
  '((shell . t)))
#+end_src

#+begin_src sh :session *shell* :results output
conda create --yes --name myenv python=3.9
#+end_src

#+RESULTS:
#+begin_example
<results removed for brevity>

Downloading and Extracting Packages

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

To activate this environment, use

conda activate myenv

To deactivate an active environment, use

conda deactivate
#+end_example

#+begin_src sh :session *shell* :results output
conda activate myenv
#+end_src

#+RESULTS:

#+begin_src sh :session *shell* :results output
which python
#+end_src

#+RESULTS:
: /home/user/miniconda3/envs/myenv/bin/python

The *shell* buffer's prompt also showed the expected [myenv] prefix.

Re: [BUG] conda doesn't work in ob-shell sessions

[Jack Kamm]

Matt <matt@excalamus.com> writes:

> 1. M-x shell

When I follow your example, in particular starting the session with
M-x shell, then conda works as expected, and behavior is same as you
report.

But when I don't start with M-x shell, instead letting ob-shell create
the session, then conda gives the error about needing to run conda init
(despite the fact that conda init had already set up my zshrc).

Example when _NOT_ using the M-x shell:

#+begin_src emacs-lisp
(org-version nil t)
#+end_src

#+RESULTS:
: Org mode version 9.6.1 (release_9.6.1-250-ge6353d @ /home/jack/devel/org-mode/lisp/)

#+begin_src emacs-lisp
(org-babel-do-load-languages
  'org-babel-load-languages
  '((shell . t)))
#+end_src

#+begin_src sh :session *myshell* :results output
conda activate ledger
#+end_src

#+RESULTS:
#+begin_example
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.
#+end_example

Re: [BUG] conda doesn't work in ob-shell sessions

[Ihor Radchenko]

Jack Kamm <jackkamm@tatersworld.org> writes:

>> 1. M-x shell
>
> When I follow your example, in particular starting the session with
> M-x shell, then conda works as expected, and behavior is same as you
> report.
>
> But when I don't start with M-x shell, instead letting ob-shell create
> the session, then conda gives the error about needing to run conda init
> (despite the fact that conda init had already set up my zshrc).

>
> #+begin_src sh :session *myshell* :results output
> conda activate ledger
> #+end_src

> #+RESULTS:
> #+begin_example
> CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
> To initialize your shell, run
>
>     $ conda init <SHELL_NAME>

I believe that is it because you explicitly specified "sh" shell, not
zsh.

When you use M-x shell, Emacs will pick up the system shell. In
contrast, ob-shell will let-bind `shell-file-name' to the shell name you
specify in the src block. The only exception is #+begin_src shell, which
will use the default system shell.

So, I believe that the following should work for you

#+begin_src shell :session *myshell* :results output
conda activate ledger
#+end_src

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] conda doesn't work in ob-shell sessions

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

> So, I believe that the following should work for you
>
> #+begin_src shell :session *myshell* :results output
> conda activate ledger
> #+end_src

No, it is still broken after switching to "shell" block, though the
error is different now. (Though I agree the previous error message was
due to using "sh" block and not ob-shell's fault).

Here is my minimal example. In init.el:

(add-to-list 'load-path "/path/to/org-mode/lisp")
(require 'org)
(org-babel-do-load-languages
   'org-babel-load-languages
   '((emacs-lisp . t)
     (shell . t)))

Then do:
emacs -q --load init.el test.org

With test.org like:

#+begin_src shell :session *shell* :results output
  conda activate some-conda-env
  echo test
#+end_src

Then, on main branch, trying to execute the block hangs. It is because
conda changes the prompt in a way that breaks the new ob-shell
implementation.

On bugfix branch, ob-shell still works with conda. I do observe a
separate bug on the first evaluation ("org-babel-execute:shell: Symbol’s
value as variable is void: org-babel-prompt-command"), but subsequent
evaluations work, and the error seems unrelated to conda.

IMO breaking conda is a pretty major regression, as conda is a critical
tool in many areas of scientific computing, including in my own
workflows.  Although I was never a major ob-shell user, I did
occasionally find it useful albeit buggy. But I've stopped using it
altogether over the past year because of this change.

Re: [BUG] conda doesn't work in ob-shell sessions

[Jack Kamm]

Jack Kamm <jackkamm@tatersworld.org> writes:

> IMO breaking conda is a pretty major regression, as conda is a critical
> tool in many areas of scientific computing, including in my own
> workflows.  Although I was never a major ob-shell user, I did
> occasionally find it useful albeit buggy. But I've stopped using it
> altogether over the past year because of this change.

Sorry, it turns out I was badly misremembering things, and overly
harsh here.

In particular, going back to older commits on 9.5 and 9.6, ob-shell was
always horribly mangled with conda, and not very useable. So I was
wrong to say the current implementation is a regression.

Actually, it seems like current version of ob-shell (on both main and
bugfix) works _better_ with conda than before -- as long as the shell
session is started with "M-x shell", instead of with
`org-babel-execute-src-block'. In particular, Ihor's commit f2949d4d1
was particularly helpful in making ob-shell play more nicely with conda.
That commit was added to bugfix around Org 9.6.4.

The reason conda doesn't work well when session is started via
`org-babel-execute-src-block' is because of the way the prompt is
set. However, when session is started by "M-x shell", ob-shell doesn't
mess with the prompt so much, and conda seems to work smoothly. Using
"M-x shell" to start the session seems like an acceptable workaround to
me, especially considering how conda never worked well with ob-shell
before.

Re: [BUG] conda doesn't work in ob-shell sessions

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 512 bytes --]

Jack Kamm <jackkamm@tatersworld.org> writes:

> The reason conda doesn't work well when session is started via
> `org-babel-execute-src-block' is because of the way the prompt is
> set. However, when session is started by "M-x shell", ob-shell doesn't
> mess with the prompt so much, and conda seems to work smoothly. Using
> "M-x shell" to start the session seems like an acceptable workaround to
> me, especially considering how conda never worked well with ob-shell
> before.

What about the attached patch?


[-- Attachment #2: 0001-lisp-ob-comint.el-Introduce-a-fallback-prompt-regexp.patch --]
[-- Type: text/x-patch, Size: 8797 bytes --]

From 45ddb20665f6124a5b08a9fbfed02ee2961ac9e3 Mon Sep 17 00:00:00 2001
Message-ID: <45ddb20665f6124a5b08a9fbfed02ee2961ac9e3.1705924712.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Mon, 22 Jan 2024 12:55:09 +0100
Subject: [PATCH] lisp/ob-comint.el: Introduce a fallback prompt regexp

* lisp/ob-comint.el (org-babel-comint-prompt-regexp-old): New variable
storing the default value of `comint-prompt-regexp' to be used when
the prompt set by Org mode changes for some reason.
(org-babel-comint-fallback-regexp-threshold): New customization to set
the time Org babel waits for comint command to finish until trying
fallback prompt regexp.
(org-babel-comint--set-fallback-prompt): New internal function that
sets the fallback prompt regexp, if there is any available.
(org-babel-comint-with-output):
(org-babel-comint-wait-for-output): Try fallback prompt regexp when we
cannot find comint command end for too long.
* lisp/ob-haskell.el (org-babel-interpret-haskell):
* lisp/ob-ruby.el (org-babel-ruby-initiate-session):
* lisp/ob-shell.el (org-babel-sh-initiate-session):
* lisp/ob-clojure.el (ob-clojure-eval-with-inf-clojure): Set
`org-babel-comint-prompt-regexp-old' when initializing the inferior
shell.

Reported-by: Jack Kamm <jackkamm@tatersworld.org>
Link: https://orgmode.org/list/87sf2q9ubd.fsf@gmail.com
---
 lisp/ob-clojure.el |  4 ++-
 lisp/ob-comint.el  | 65 ++++++++++++++++++++++++++++++++++++----------
 lisp/ob-haskell.el |  6 +++--
 lisp/ob-ruby.el    |  4 ++-
 lisp/ob-shell.el   |  8 +++---
 5 files changed, 67 insertions(+), 20 deletions(-)

diff --git a/lisp/ob-clojure.el b/lisp/ob-clojure.el
index ddcfcd9fb..4a54acc51 100644
--- a/lisp/ob-clojure.el
+++ b/lisp/ob-clojure.el
@@ -237,7 +237,9 @@ (defun ob-clojure-eval-with-inf-clojure (expanded params)
 			     "clojure" (format "clojure -A%s" alias)
 			     cmd0)
 		    cmd0)))
-	(setq comint-prompt-regexp inf-clojure-comint-prompt-regexp)
+	(setq
+         org-babel-comint-prompt-regexp-old comint-prompt-regexp
+         comint-prompt-regexp inf-clojure-comint-prompt-regexp)
 	(funcall-interactively #'inf-clojure cmd)
 	(goto-char (point-max))))
     (sit-for 1))
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 7d258ea0e..316848b6b 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -58,6 +58,22 @@ (defmacro org-babel-comint-in-buffer (buffer &rest body)
 	   (let ((comint-input-filter (lambda (_input) nil)))
 	     ,@body))))))
 
+(defvar-local org-babel-comint-prompt-regexp-old nil
+  "Fallback regexp used to detect prompt.")
+
+(defcustom org-babel-comint-fallback-regexp-threshold 5.0
+  "Waiting time until trying to use fallback regexp to detect prompt.
+This is useful when prompt unexpectedly changes."
+  :type 'float
+  :group 'org-babel)
+
+(defun org-babel-comint--set-fallback-prompt ()
+  "Swap `comint-prompt-regexp' and `org-babel-comint-prompt-regexp-old'."
+  (when org-babel-comint-prompt-regexp-old
+    (let ((tmp comint-prompt-regexp))
+      (setq comint-prompt-regexp org-babel-comint-prompt-regexp-old
+            org-babel-comint-prompt-regexp-old tmp))))
+
 (defmacro org-babel-comint-with-output (meta &rest body)
   "Evaluate BODY in BUFFER and return process output.
 Will wait until EOE-INDICATOR appears in the output, then return
@@ -96,14 +112,27 @@ (defmacro org-babel-comint-with-output (meta &rest body)
 	 ;; pass FULL-BODY to process
 	 ,@body
 	 ;; wait for end-of-evaluation indicator
-	 (while (progn
-		  (goto-char comint-last-input-end)
-		  (not (save-excursion
-		       (and (re-search-forward
-			     (regexp-quote ,eoe-indicator) nil t)
-			    (re-search-forward
-			     comint-prompt-regexp nil t)))))
-	   (accept-process-output (get-buffer-process (current-buffer))))
+         (let ((start-time (current-time)))
+	   (while (progn
+		    (goto-char comint-last-input-end)
+		    (not (save-excursion
+		         (and (re-search-forward
+			       (regexp-quote ,eoe-indicator) nil t)
+			      (re-search-forward
+			       comint-prompt-regexp nil t)))))
+	     (accept-process-output (get-buffer-process (current-buffer)))
+             (when (and org-babel-comint-prompt-regexp-old
+                        (> (float-time (time-since start-time))
+                           org-babel-comint-fallback-regexp-threshold)
+                        (progn
+		          (goto-char comint-last-input-end)
+		          (save-excursion
+                            (and
+                             (re-search-forward
+			      (regexp-quote ,eoe-indicator) nil t)
+			     (re-search-forward
+                              org-babel-comint-prompt-regexp-old nil t)))))
+               (org-babel-comint--set-fallback-prompt))))
 	 ;; replace cut dangling text
 	 (goto-char (process-mark (get-buffer-process (current-buffer))))
 	 (insert dangling-text)
@@ -148,11 +177,21 @@ (defun org-babel-comint-wait-for-output (buffer)
 Note: this is only safe when waiting for the result of a single
 statement (not large blocks of code)."
   (org-babel-comint-in-buffer buffer
-    (while (progn
-             (goto-char comint-last-input-end)
-             (not (and (re-search-forward comint-prompt-regexp nil t)
-                     (goto-char (match-beginning 0)))))
-      (accept-process-output (get-buffer-process buffer)))))
+    (let ((start-time (current-time)))
+      (while (progn
+               (goto-char comint-last-input-end)
+               (not (and (re-search-forward comint-prompt-regexp nil t)
+                       (goto-char (match-beginning 0)))))
+        (accept-process-output (get-buffer-process buffer))
+        (when (and org-babel-comint-prompt-regexp-old
+                   (> (float-time (time-since start-time))
+                      org-babel-comint-fallback-regexp-threshold)
+                   (progn
+		     (goto-char comint-last-input-end)
+		     (save-excursion
+		       (re-search-forward
+                        org-babel-comint-prompt-regexp-old nil t))))
+          (org-babel-comint--set-fallback-prompt))))))
 
 (defun org-babel-comint-eval-invisibly-and-wait-for-file
     (buffer file string &optional period)
diff --git a/lisp/ob-haskell.el b/lisp/ob-haskell.el
index a9ed772dd..00be6d52c 100644
--- a/lisp/ob-haskell.el
+++ b/lisp/ob-haskell.el
@@ -152,8 +152,10 @@ (defun org-babel-interpret-haskell (body params)
   (org-require-package 'inf-haskell "haskell-mode")
   (add-hook 'inferior-haskell-hook
             (lambda ()
-              (setq-local comint-prompt-regexp
-                          (concat haskell-prompt-regexp "\\|^λ?> "))))
+              (setq-local
+               org-babel-comint-prompt-regexp-old comint-prompt-regexp
+               comint-prompt-regexp
+               (concat haskell-prompt-regexp "\\|^λ?> "))))
   (org-babel-haskell-with-session session params
     (cl-labels
         ((send-txt-to-ghci (txt)
diff --git a/lisp/ob-ruby.el b/lisp/ob-ruby.el
index 79b33940d..d920fb585 100644
--- a/lisp/ob-ruby.el
+++ b/lisp/ob-ruby.el
@@ -191,7 +191,9 @@ (defun org-babel-ruby-initiate-session (&optional session params)
             ;; uniquely by regexp.
             (when new-session?
               (with-current-buffer session-buffer
-                (setq-local comint-prompt-regexp (concat "^" org-babel-ruby-prompt))
+                (setq-local
+                 org-babel-comint-prompt-regexp-old comint-prompt-regexp
+                 comint-prompt-regexp (concat "^" org-babel-ruby-prompt))
                 (insert org-babel-ruby-define-prompt ";")
                 (insert "_org_prompt_mode=conf.prompt_mode;conf.prompt_mode=:CUSTOM;")
                 (insert "conf.echo=false\n")
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 31135b5fb..20d1d5e33 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -265,9 +265,11 @@ (defun org-babel-sh-initiate-session (&optional session _params)
                               org-babel-shell-set-prompt-commands))
                   (alist-get t org-babel-shell-set-prompt-commands))
               org-babel-sh-prompt))
-            (setq-local comint-prompt-regexp
-                        (concat "^" (regexp-quote org-babel-sh-prompt)
-                                " *"))
+            (setq-local
+             org-babel-comint-prompt-regexp-old comint-prompt-regexp
+             comint-prompt-regexp
+             (concat "^" (regexp-quote org-babel-sh-prompt)
+                     " *"))
 	    ;; Needed for Emacs 23 since the marker is initially
 	    ;; undefined and the filter functions try to use it without
 	    ;; checking.
-- 
2.43.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] conda doesn't work in ob-shell sessions

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

> What about the attached patch?

I applied the patch to main, but the behavior was the same as before.

More specifically, I am testing with the following block (with minimal
init file from my previous email):

#+begin_src shell :session *shell* :results output
  conda activate test-env
  echo test
#+end_src

When I execute the block before session exists, emacs hangs. But if I
start session with "M-x shell", and then execute the block, it works.

Here is how the full *shell* buffer looks in the former case that hangs,
after I quit the hanging with C-g:

$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2=
org_babel_sh_prompt> conda activate test-env
echo test
echo 'org_babel_sh_eoe'
(test-env) org_babel_sh_prompt> test
(test-env) org_babel_sh_prompt> org_babel_sh_eoe
(test-env) org_babel_sh_prompt>

And here is how the *shell* buffer looks in the latter case which works,
when session is started with "M-x shell" instead:

$ conda activate test-env
echo test
echo 'org_babel_sh_eoe'
(test-env) $ test
(test-env) $ org_babel_sh_eoe
(test-env) $

Re: [BUG] conda doesn't work in ob-shell sessions

[Suhail Singh]

Jack Kamm <jackkamm@tatersworld.org> writes:

> #+begin_src shell :session *shell* :results output
>   conda activate some-conda-env
>   echo test
> #+end_src
>
> Then, on main branch, trying to execute the block hangs. It is because
> conda changes the prompt in a way that breaks the new ob-shell
> implementation.

This isn't a fix, but a workaround I've used for a while is to define a
function set_PS1 which I invoke after "conda activate" within org-babel
blocks.  set_PS1 (among other things that aren't relevant) compares PS1
against "(${CONDA_DEFAULT_ENV:-base}) org_babel_sh_prompt> ", and in
those cases sets PS1 to "org_babel_sh_prompt> ".  set_PS1 thus defined,
ensures that the follow block doesn't hang on first invocation.

#+begin_src bash :session conda-test :results replace
  conda activate test && set_PS1
  echo "${CONDA_DEFAULT_ENV}"
#+end_src

#+RESULTS:
: test

> ... on the first evaluation ("org-babel-execute:shell: Symbol’s value
> as variable is void: org-babel-prompt-command")

On org-version as recent as 9.6.17, when performing some actions such as
invoking org-metadown on a block with :session defined, I encounter the
same unless I have the below in my init.el:

#+begin_src elisp
  (defvar org-babel-prompt-command nil)
#+end_src

-- 
Suhail

Re: [BUG] conda doesn't work in ob-shell sessions

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 258 bytes --]

Jack Kamm <jackkamm@tatersworld.org> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> What about the attached patch?
>
> I applied the patch to main, but the behavior was the same as before.

What about the attached second version of the patch?


[-- Attachment #2: v2-0001-lisp-ob-comint.el-Introduce-a-fallback-prompt-reg.patch --]
[-- Type: text/x-patch, Size: 8939 bytes --]

From 0d1331c75ad7a09d88bc5bf38d00488980b6a510 Mon Sep 17 00:00:00 2001
Message-ID: <0d1331c75ad7a09d88bc5bf38d00488980b6a510.1706109720.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Mon, 22 Jan 2024 12:55:09 +0100
Subject: [PATCH v2] lisp/ob-comint.el: Introduce a fallback prompt regexp

* lisp/ob-comint.el (org-babel-comint-prompt-regexp-old): New variable
storing the default value of `comint-prompt-regexp' to be used when
the prompt set by Org mode changes for some reason.
(org-babel-comint-fallback-regexp-threshold): New customization to set
the time Org babel waits for comint command to finish until trying
fallback prompt regexp.
(org-babel-comint--set-fallback-prompt): New internal function that
sets the fallback prompt regexp, if there is any available.
(org-babel-comint-with-output):
(org-babel-comint-wait-for-output): Try fallback prompt regexp when we
cannot find comint command end for too long.
* lisp/ob-haskell.el (org-babel-interpret-haskell):
* lisp/ob-ruby.el (org-babel-ruby-initiate-session):
* lisp/ob-shell.el (org-babel-sh-initiate-session):
* lisp/ob-clojure.el (ob-clojure-eval-with-inf-clojure): Set
`org-babel-comint-prompt-regexp-old' when initializing the inferior
shell.

Reported-by: Jack Kamm <jackkamm@tatersworld.org>
Link: https://orgmode.org/list/87sf2q9ubd.fsf@gmail.com
---
 lisp/ob-clojure.el |  4 ++-
 lisp/ob-comint.el  | 69 +++++++++++++++++++++++++++++++++++++---------
 lisp/ob-haskell.el |  6 ++--
 lisp/ob-ruby.el    |  4 ++-
 lisp/ob-shell.el   |  8 ++++--
 5 files changed, 71 insertions(+), 20 deletions(-)

diff --git a/lisp/ob-clojure.el b/lisp/ob-clojure.el
index ddcfcd9fb..4a54acc51 100644
--- a/lisp/ob-clojure.el
+++ b/lisp/ob-clojure.el
@@ -237,7 +237,9 @@ (defun ob-clojure-eval-with-inf-clojure (expanded params)
 			     "clojure" (format "clojure -A%s" alias)
 			     cmd0)
 		    cmd0)))
-	(setq comint-prompt-regexp inf-clojure-comint-prompt-regexp)
+	(setq
+         org-babel-comint-prompt-regexp-old comint-prompt-regexp
+         comint-prompt-regexp inf-clojure-comint-prompt-regexp)
 	(funcall-interactively #'inf-clojure cmd)
 	(goto-char (point-max))))
     (sit-for 1))
diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 7d258ea0e..26f0c3bf7 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -58,6 +58,22 @@ (defmacro org-babel-comint-in-buffer (buffer &rest body)
 	   (let ((comint-input-filter (lambda (_input) nil)))
 	     ,@body))))))
 
+(defvar-local org-babel-comint-prompt-regexp-old nil
+  "Fallback regexp used to detect prompt.")
+
+(defcustom org-babel-comint-fallback-regexp-threshold 5.0
+  "Waiting time until trying to use fallback regexp to detect prompt.
+This is useful when prompt unexpectedly changes."
+  :type 'float
+  :group 'org-babel)
+
+(defun org-babel-comint--set-fallback-prompt ()
+  "Swap `comint-prompt-regexp' and `org-babel-comint-prompt-regexp-old'."
+  (when org-babel-comint-prompt-regexp-old
+    (let ((tmp comint-prompt-regexp))
+      (setq comint-prompt-regexp org-babel-comint-prompt-regexp-old
+            org-babel-comint-prompt-regexp-old tmp))))
+
 (defmacro org-babel-comint-with-output (meta &rest body)
   "Evaluate BODY in BUFFER and return process output.
 Will wait until EOE-INDICATOR appears in the output, then return
@@ -96,14 +112,29 @@ (defmacro org-babel-comint-with-output (meta &rest body)
 	 ;; pass FULL-BODY to process
 	 ,@body
 	 ;; wait for end-of-evaluation indicator
-	 (while (progn
-		  (goto-char comint-last-input-end)
-		  (not (save-excursion
-		       (and (re-search-forward
-			     (regexp-quote ,eoe-indicator) nil t)
-			    (re-search-forward
-			     comint-prompt-regexp nil t)))))
-	   (accept-process-output (get-buffer-process (current-buffer))))
+         (let ((start-time (current-time)))
+	   (while (progn
+		    (goto-char comint-last-input-end)
+		    (not (save-excursion
+		         (and (re-search-forward
+			       (regexp-quote ,eoe-indicator) nil t)
+			      (re-search-forward
+			       comint-prompt-regexp nil t)))))
+	     (accept-process-output
+              (get-buffer-process (current-buffer))
+              org-babel-comint-fallback-regexp-threshold)
+             (when (and org-babel-comint-prompt-regexp-old
+                        (> (float-time (time-since start-time))
+                           org-babel-comint-fallback-regexp-threshold)
+                        (progn
+		          (goto-char comint-last-input-end)
+		          (save-excursion
+                            (and
+                             (re-search-forward
+			      (regexp-quote ,eoe-indicator) nil t)
+			     (re-search-forward
+                              org-babel-comint-prompt-regexp-old nil t)))))
+               (org-babel-comint--set-fallback-prompt))))
 	 ;; replace cut dangling text
 	 (goto-char (process-mark (get-buffer-process (current-buffer))))
 	 (insert dangling-text)
@@ -148,11 +179,23 @@ (defun org-babel-comint-wait-for-output (buffer)
 Note: this is only safe when waiting for the result of a single
 statement (not large blocks of code)."
   (org-babel-comint-in-buffer buffer
-    (while (progn
-             (goto-char comint-last-input-end)
-             (not (and (re-search-forward comint-prompt-regexp nil t)
-                     (goto-char (match-beginning 0)))))
-      (accept-process-output (get-buffer-process buffer)))))
+    (let ((start-time (current-time)))
+      (while (progn
+               (goto-char comint-last-input-end)
+               (not (and (re-search-forward comint-prompt-regexp nil t)
+                       (goto-char (match-beginning 0)))))
+        (accept-process-output
+         (get-buffer-process buffer)
+         org-babel-comint-fallback-regexp-threshold)
+        (when (and org-babel-comint-prompt-regexp-old
+                   (> (float-time (time-since start-time))
+                      org-babel-comint-fallback-regexp-threshold)
+                   (progn
+		     (goto-char comint-last-input-end)
+		     (save-excursion
+		       (re-search-forward
+                        org-babel-comint-prompt-regexp-old nil t))))
+          (org-babel-comint--set-fallback-prompt))))))
 
 (defun org-babel-comint-eval-invisibly-and-wait-for-file
     (buffer file string &optional period)
diff --git a/lisp/ob-haskell.el b/lisp/ob-haskell.el
index a9ed772dd..00be6d52c 100644
--- a/lisp/ob-haskell.el
+++ b/lisp/ob-haskell.el
@@ -152,8 +152,10 @@ (defun org-babel-interpret-haskell (body params)
   (org-require-package 'inf-haskell "haskell-mode")
   (add-hook 'inferior-haskell-hook
             (lambda ()
-              (setq-local comint-prompt-regexp
-                          (concat haskell-prompt-regexp "\\|^λ?> "))))
+              (setq-local
+               org-babel-comint-prompt-regexp-old comint-prompt-regexp
+               comint-prompt-regexp
+               (concat haskell-prompt-regexp "\\|^λ?> "))))
   (org-babel-haskell-with-session session params
     (cl-labels
         ((send-txt-to-ghci (txt)
diff --git a/lisp/ob-ruby.el b/lisp/ob-ruby.el
index 79b33940d..d920fb585 100644
--- a/lisp/ob-ruby.el
+++ b/lisp/ob-ruby.el
@@ -191,7 +191,9 @@ (defun org-babel-ruby-initiate-session (&optional session params)
             ;; uniquely by regexp.
             (when new-session?
               (with-current-buffer session-buffer
-                (setq-local comint-prompt-regexp (concat "^" org-babel-ruby-prompt))
+                (setq-local
+                 org-babel-comint-prompt-regexp-old comint-prompt-regexp
+                 comint-prompt-regexp (concat "^" org-babel-ruby-prompt))
                 (insert org-babel-ruby-define-prompt ";")
                 (insert "_org_prompt_mode=conf.prompt_mode;conf.prompt_mode=:CUSTOM;")
                 (insert "conf.echo=false\n")
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 31135b5fb..20d1d5e33 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -265,9 +265,11 @@ (defun org-babel-sh-initiate-session (&optional session _params)
                               org-babel-shell-set-prompt-commands))
                   (alist-get t org-babel-shell-set-prompt-commands))
               org-babel-sh-prompt))
-            (setq-local comint-prompt-regexp
-                        (concat "^" (regexp-quote org-babel-sh-prompt)
-                                " *"))
+            (setq-local
+             org-babel-comint-prompt-regexp-old comint-prompt-regexp
+             comint-prompt-regexp
+             (concat "^" (regexp-quote org-babel-sh-prompt)
+                     " *"))
 	    ;; Needed for Emacs 23 since the marker is initially
 	    ;; undefined and the filter functions try to use it without
 	    ;; checking.
-- 
2.43.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] conda doesn't work in ob-shell sessions

[Matt]

 ---- On Wed, 24 Jan 2024 16:20:16 +0100  Ihor Radchenko  wrote --- 
 > Jack Kamm jackkamm@tatersworld.org> writes:
 > 
 > > Ihor Radchenko yantar92@posteo.net> writes:
 > >
 > >> What about the attached patch?
 > >
 > > I applied the patch to main, but the behavior was the same as before.
 > 
 > What about the attached second version of the patch?

I spent way too long trying to test it but I'm not sure I applied the patch correctly.

I did the following, then realized that the patch I applied undid changes (I assume) from a previous patch and so it must have been applied on top of another one.  I then wasted a bunch of time trying to apply various combinations of the four patches I found in the thread.  One of the patches has trailing whitespace yet the various --ignore options of git am didn't seem to ignore it.  I guess I could have looked at the timestamps and used git apply in sequence?

Should make autoloads also be run after applying the patches?

Here's what I was able to do:

In a VM, installed conda as described by their page.  Specifically, my .bashrc was modified to activate the base environment.   Closed terminal, opened it into the base environment, and created a new environment, emacs-test.

Downloaded the latest org-mode by git-clone and did make autoloads.  Added the org-mode git repo's lisp directory to my early-init.el.

Applied patch with

  git apply --cache --ignore-space-changes --ignore-whitespace 2-v2-0001-lisp-ob-comint.el-Introduce-a-fallback-prompt-reg.patch

Doing plain git apply failed.

With the changes in the org-mode index, the base environment activated, I run 'emacs' and confirm I'm using the org-mode git with the patch applied (ob-shell doesn't set org-babel-comint-prompt in ).

Then

(org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))

#+begin_src shell :results output :session *shell*
conda activate emacs-test
#+end_src

It hangs.  C-g and switch to shell buffer shows the following (note, this was typed in and not copy-pasted because VM):

(base) user@debian:~$ PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2=
org_babel_sh_prompt> conda activate emacs-test
echo 'org_babel_sh_eoe'
(emacs-test) org_babel_sh_prompt> 'org_babel_sh_eoe'
(emacs-test) org_babel_sh_prompt> 


--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: [BUG] conda doesn't work in ob-shell sessions

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

>  > What about the attached second version of the patch?
>
> I spent way too long trying to test it but I'm not sure I applied the patch correctly.
>
> I did the following, then realized that the patch I applied undid
> changes (I assume) from a previous patch and so it must have been
> applied on top of another one. I then wasted a bunch of time trying to
> apply various combinations of the four patches I found in the thread.
> One of the patches has trailing whitespace yet the various --ignore
> options of git am didn't seem to ignore it. I guess I could have
> looked at the timestamps and used git apply in sequence?

The patch should apply cleanly on the latest main.
No previous patches are needed.

To simplify working with email patches, you may consider
https://docs.kyleam.com/piem/Overview.html (Emacs package)
It can apply patches right from inside Emacs email client.

You can also use magit.

> Should make autoloads also be run after applying the patches?

Yes. Or make. Or you can use make repro to run clean Emacs using the
current git branch.

> Here's what I was able to do:
>
> In a VM, installed conda as described by their page.  Specifically, my .bashrc was modified to activate the base environment.   Closed terminal, opened it into the base environment, and created a new environment, emacs-test.
>
> Downloaded the latest org-mode by git-clone and did make autoloads.  Added the org-mode git repo's lisp directory to my early-init.el.
>
> Applied patch with
>
>   git apply --cache --ignore-space-changes --ignore-whitespace 2-v2-0001-lisp-ob-comint.el-Introduce-a-fallback-prompt-reg.patch
>
> Doing plain git apply failed.
>
> With the changes in the org-mode index, the base environment activated, I run 'emacs' and confirm I'm using the org-mode git with the patch applied (ob-shell doesn't set org-babel-comint-prompt in ).
>
> Then
>
> (org-babel-do-load-languages 'org-babel-load-languages '((shell . t)))
>
> #+begin_src shell :results output :session *shell*
> conda activate emacs-test
> #+end_src
>
> It hangs.  C-g and switch to shell buffer shows the following (note, this was typed in and not copy-pasted because VM):

Note that the patch should only make Emacs unhang after 5 seconds delay.

I tested the patch with the following (conda does not available for my system):

#+begin_src bash :results output :session *shell*
PS1="prompt> "
ls
#+end_src

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] conda doesn't work in ob-shell sessions

[Jack Kamm]

Ihor Radchenko <yantar92@posteo.net> writes:

> Jack Kamm <jackkamm@tatersworld.org> writes:
>
>> Ihor Radchenko <yantar92@posteo.net> writes:
>>
>>> What about the attached patch?
>>
>> I applied the patch to main, but the behavior was the same as before.
>
> What about the attached second version of the patch?

Second version of the patch works on my test example now. The initial
block hangs for a few seconds but then finishes. Subsequent blocks seem
to work without any noticeable hanging.

Thanks!

Re: [BUG] conda doesn't work in ob-shell sessions

[Matt]

 ---- On Fri, 26 Jan 2024 01:42:59 +0100  Jack Kamm  wrote --- 

 > Second version of the patch works on my test example now. The initial
 > block hangs for a few seconds but then finishes. Subsequent blocks seem
 > to work without any noticeable hanging.

I was able to confirm this as well.

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: [BUG] conda doesn't work in ob-shell sessions

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

>  ---- On Fri, 26 Jan 2024 01:42:59 +0100  Jack Kamm  wrote --- 
>
>  > Second version of the patch works on my test example now. The initial
>  > block hangs for a few seconds but then finishes. Subsequent blocks seem
>  > to work without any noticeable hanging.
>
> I was able to confirm this as well.

Thanks for checking!
Applied, onto main.
I also added ORG-NEWS entry about the new custom variable.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=27d6f8305

Fixed.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] conda doesn't work in ob-shell sessions

[Cook, Malcolm]

# -*-  org-confirm-babel-evaluate: nil; -*-

I have been struggling with interrelated issues raised in

 - https://list.orgmode.org/87jznda90u.fsf@localhost/#t
 - https://list.orgmode.org/87le1bc8j3.fsf@localhost/

I expect I am using all the patches offered in addressing these given
my recent build from main.  However, in my hands, I find they still
easily allow for mistakes identifying prompts in code block results.

In this demonstration, I am extending the approach to inquiry begun by Jack in
https://list.orgmode.org/87ttzn1mai.fsf@gmail.com/

#+begin_src emacs-lisp :results raw
   (org-version nil t)
   (emacs-version)
#+end_src

#+RESULTS:
GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.15.12, Xaw3d scroll bars)
 of 2024-09-04
(Org mode version 9.7.10 (release_9.7.10 @ /home/mec/.local/share/emacs/31.0.50/lisp/org/) GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.15.12, Xaw3d scroll bars)
 of 2024-09-04)

#+begin_src emacs-lisp
  (org-babel-do-load-languages
    'org-babel-load-languages
    '((shell . t)))
#+end_src

#+RESULTS:

Here I define two org code blocks I will use repeatedly below:

#+name:test_filter
#+begin_src shell :session *shell* :results output
  printf "a\nb\nc\n>d\n<e\nf>\nggg ggg>\nhhh hhh+\na\n"
#+end_src

#+name:shell_prompt_info
#+begin_src elisp
  (with-current-buffer "*shell*"
    (format "[comint-prompt-regexp]=[%s]\n[org-babel-comint-prompt-regexp-old]=[%s]" comint-prompt-regexp org-babel-comint-prompt-regexp-old))
#+end_src

#+caption: The results looks good - the output apparently is not confused as being prompt.
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: >d
: <e
: f>
: ggg ggg>
: hhh hhh+
: a

#+caption: take a look at the prompt variables.
#+call:shell_prompt_info()

#+RESULTS:
: [comint-prompt-regexp]=[^org_babel_sh_prompt>  *]
: [org-babel-comint-prompt-regexp-old]=[^[^#$%>
: ]*[#$%>] *]

#+caption: check on conda's availabiity & version
#+begin_src shell :session *shell* :results output
conda --version
#+end_src

#+RESULTS:
: conda 24.7.1

#+begin_src shell :session *shell* :results output
conda create --yes --name myenv python=3.9
#+end_src

#+RESULTS:
#+begin_example
... abbreviated...

To activate this environment, use

conda activate myenv

To deactivate an active environment, use

conda deactivate
#+end_example

#+begin_src shell :session *shell* :results output
  conda activate myenv
#+end_src

#+RESULTS:

#+begin_src shell :session *shell* :results output
which python
#+end_src

#+RESULTS:
: /n/projects/mec/SRSCHPC2/local/inst/Mambaforge/24.3.0-0/envs/myenv/bin/python

#+caption: alas, the output of test_filter is changed.  Some lines are gone missing and some are changed.
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: Observe the prompts have changed.  Perhaps this is related issue?
#+call:shell_prompt_info()

#+RESULTS:
: [comint-prompt-regexp]=[^[^#$%>
: ]*[#$%>] *]
: [org-babel-comint-prompt-regexp-old]=[^org_babel_sh_prompt>  *]

#+caption:  can we restore by deactivating the environment?
#+begin_src shell :session *shell* :results output
  conda activate  
#+end_src

#+RESULTS:

#+caption: alas, no:
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: how about by resetting the prompt
#+begin_src shell :session *shell* :results output
  PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2=
#+end_src

#+RESULTS:

#+caption: alas, again, no
#+call: test_filter()
#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: perhaps restoring the prompt variables will recover?
#+begin_src elisp
    (with-current-buffer "*shell*"
      (setq-local comint-prompt-regexp "^org_babel_sh_prompt>  *"
      	      org-babel-comint-prompt-regexp-old "[^[^#$%>
]*[#$%>] *"))
#+end_src

#+RESULTS:
: [^[^#$%>
: ]*[#$%>] *

#+caption: YES!
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: >d
: <e
: f>
: ggg ggg>
: hhh hhh+
: a


In the above, I am exclusively allowing org/ob/comint to "own" the shell
buffer, and not interact with it, as recommended earlier by Ivor.

I have tried the above after first calling `(shell)` and find
variations on the above occur.  I would like to be able to 'share' the
*shell* buffer with org/ob/comint but expect resolving the
non-interactive case should possibly lay foundation.

I would additional like to layer in working with remote shells
(e.g. `:dir "/ssh:me@host:~/`) and have tried but this is just
layering in complexity on the localhost case so I'm backing off for
now.

What else can I report or test?