RE: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko <yantar92@posteo.net>]

"Cook, Malcolm" <MEC@stowers.org> writes:

>>> #+begin_src sh
>>> printf 'a\nb\nc\n>d\n<e\n'
>>> #+end_src
>>>
>>> #+RESULTS:
>>> : a
>>> : b
>>> : c
>>> : d
>>> : <e
>>>
>>> Huh? what happened to the ">" before the "d"?
>>
>>I cannot reproduce.
>>May it be that you are setting global :session header argument?
>
> Yes, of course, so sorry, there is
>
> #+PROPERTY: header-args:sh :session *shell* :results output
>
> You should now be able to reproduce.
>
> Thanks, please let me know if you have a fix, workaround, or lesson for me.

In theory, we may provide a workaround for this kind of problem. But it
would have pros and cons.

The underlying cause is limitation of Emacs API for interactive shells -
we cannot easily distinguish command output from prompt and other extra
staff your shell/other interactive command spits into the buffer.
So, we have to either filter output the prompts ourselves to get the
command output reliably or redirect output to files, where nothing
litters the actual output with prompts.

Redirecting output works fine when we do not use sessions and do not
care what is displayed in the *shell* buffer for users. For sessions, we
use more complex approach - filter out anything that looks like prompt.

The problem with bash is that prompt is ">", and, as you showed in your
example, having ">" in the commands output breaks everything.

Again, we know this problem, and we normally force the prompt to be
something more unique when creating sessions. But we do not touch the
prompt when you point Org to an existing interactive shell buffer - that
could be a surprise if Org mode changes the prompts by force in the
shells you use interactively.

TL;DR: It is complicated. When we pull one string out, several more
       get entangled.

As a practical workaround, just do not use *shell* session names and
session names that are the same as shell buffers you create manually.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>