Re: [BUG] conda doesn't work in ob-shell sessions AND org-babel-execute-src-block filters characters from :session *shell* output

Re: [BUG] conda doesn't work in ob-shell sessions AND org-babel-execute-src-block filters characters from :session *shell* output

[Cook, Malcolm]

# -*-  org-confirm-babel-evaluate: nil; -*-

I have been struggling with interrelated issues raised in

 - https://list.orgmode.org/87jznda90u.fsf@localhost/#t
 - https://list.orgmode.org/87le1bc8j3.fsf@localhost/

I expect I am using all the patches offered in addressing these given
my recent build from main.  However, in my hands, I find they still
easily allow for mistakes identifying prompts in code block results.

In this demonstration, I am extending the approach to inquiry begun by Jack in
https://list.orgmode.org/87ttzn1mai.fsf@gmail.com/

#+begin_src emacs-lisp :results raw
   (org-version nil t)
   (emacs-version)
#+end_src

#+RESULTS:
GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.15.12, Xaw3d scroll bars)
 of 2024-09-04
(Org mode version 9.7.10 (release_9.7.10 @ /home/mec/.local/share/emacs/31.0.50/lisp/org/) GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.15.12, Xaw3d scroll bars)
 of 2024-09-04)

#+begin_src emacs-lisp
  (org-babel-do-load-languages
    'org-babel-load-languages
    '((shell . t)))
#+end_src

#+RESULTS:

Here I define two org code blocks I will use repeatedly below:

#+name:test_filter
#+begin_src shell :session *shell* :results output
  printf "a\nb\nc\n>d\n<e\nf>\nggg ggg>\nhhh hhh+\na\n"
#+end_src

#+name:shell_prompt_info
#+begin_src elisp
  (with-current-buffer "*shell*"
    (format "[comint-prompt-regexp]=[%s]\n[org-babel-comint-prompt-regexp-old]=[%s]" comint-prompt-regexp org-babel-comint-prompt-regexp-old))
#+end_src

#+caption: The results looks good - the output apparently is not confused as being prompt.
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: >d
: <e
: f>
: ggg ggg>
: hhh hhh+
: a

#+caption: take a look at the prompt variables.
#+call:shell_prompt_info()

#+RESULTS:
: [comint-prompt-regexp]=[^org_babel_sh_prompt>  *]
: [org-babel-comint-prompt-regexp-old]=[^[^#$%>
: ]*[#$%>] *]

#+caption: check on conda's availabiity & version
#+begin_src shell :session *shell* :results output
conda --version
#+end_src

#+RESULTS:
: conda 24.7.1

#+begin_src shell :session *shell* :results output
conda create --yes --name myenv python=3.9
#+end_src

#+RESULTS:
#+begin_example
... abbreviated...

To activate this environment, use

conda activate myenv

To deactivate an active environment, use

conda deactivate
#+end_example

#+begin_src shell :session *shell* :results output
  conda activate myenv
#+end_src

#+RESULTS:

#+begin_src shell :session *shell* :results output
which python
#+end_src

#+RESULTS:
: /n/projects/mec/SRSCHPC2/local/inst/Mambaforge/24.3.0-0/envs/myenv/bin/python

#+caption: alas, the output of test_filter is changed.  Some lines are gone missing and some are changed.
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: Observe the prompts have changed.  Perhaps this is related issue?
#+call:shell_prompt_info()

#+RESULTS:
: [comint-prompt-regexp]=[^[^#$%>
: ]*[#$%>] *]
: [org-babel-comint-prompt-regexp-old]=[^org_babel_sh_prompt>  *]

#+caption:  can we restore by deactivating the environment?
#+begin_src shell :session *shell* :results output
  conda activate  
#+end_src

#+RESULTS:

#+caption: alas, no:
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: how about by resetting the prompt
#+begin_src shell :session *shell* :results output
  PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2=
#+end_src

#+RESULTS:

#+caption: alas, again, no
#+call: test_filter()
#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: perhaps restoring the prompt variables will recover?
#+begin_src elisp
    (with-current-buffer "*shell*"
      (setq-local comint-prompt-regexp "^org_babel_sh_prompt>  *"
      	      org-babel-comint-prompt-regexp-old "[^[^#$%>
]*[#$%>] *"))
#+end_src

#+RESULTS:
: [^[^#$%>
: ]*[#$%>] *

#+caption: YES!
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: >d
: <e
: f>
: ggg ggg>
: hhh hhh+
: a


In the above, I am exclusively allowing org/ob/comint to "own" the shell
buffer, and not interact with it, as recommended earlier by Ivor.

I have tried the above after first calling `(shell)` and find
variations on the above occur.  I would like to be able to 'share' the
*shell* buffer with org/ob/comint but expect resolving the
non-interactive case should possibly lay foundation.

I would additional like to layer in working with remote shells
(e.g. `:dir "/ssh:me@host:~/`) and have tried but this is just
layering in complexity on the localhost case so I'm backing off for
now.

What else can I report or test?

Re: [BUG] conda doesn't work in ob-shell sessions AND org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko]

"Cook, Malcolm" <MEC@stowers.org> writes:

> #+begin_src shell :session *shell* :results output
> conda create --yes --name myenv python=3.9
> #+end_src
> ...
> #+begin_src shell :session *shell* :results output
>   conda activate myenv
> #+end_src
>
> #+RESULTS:
>
> #+begin_src shell :session *shell* :results output
> which python
> #+end_src
>
> #+RESULTS:
> : /n/projects/mec/SRSCHPC2/local/inst/Mambaforge/24.3.0-0/envs/myenv/bin/python
>
> #+caption: alas, the output of test_filter is changed.  Some lines are gone missing and some are changed.
> #+call: test_filter()
> ...

This is likely because conda changes the prompt, which Org mode does not
expect.

I am afraid that there is no general magic solution to detecting
prompts. We either try heuristics and hit false-positives (that's what
we did in the past) or set very specific prompt and try to search it
(now). In the latter case, any changes to the prompt will break things.

What can be done is force-setting the prompt every single time babel
sends output to the comint buffer, but it feels wrong.

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>