comint-output-filter-functions and multi-line input

comint-output-filter-functions and multi-line input

[Ihor Radchenko]

Hello,

I am trying to figure out how to programmatically capture output of a
multi-line bash script in comint buffer.

Consider:

[yantar92:/tmp] $ 
> if true
> then
> echo "hello"
> fi
hello

Each line is submitted to shell via comint-send-input and, AFAIK, the
result can be collected using `comint-output-filter-functions'.

I expect the `comint-output-filter-functions' to be called on "hello"
line. However, it does not seem to be the case. Empty "" corresponding
to "> " PS2 prompts upon sending incomplete script lines are also
passed.

Is it possible to distinguish the actual script output, empty lines in
the actual script output, and the incomplete prompts?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: comint-output-filter-functions and multi-line input

[miha]

[-- Attachment #1: Type: text/plain, Size: 1604 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> Hello,
>
> I am trying to figure out how to programmatically capture output of a
> multi-line bash script in comint buffer.
>
> Consider:
>
> [yantar92:/tmp] $ 
>> if true
>> then
>> echo "hello"
>> fi
> hello
>
> Each line is submitted to shell via comint-send-input and, AFAIK, the
> result can be collected using `comint-output-filter-functions'.
>
> I expect the `comint-output-filter-functions' to be called on "hello"
> line. However, it does not seem to be the case. Empty "" corresponding
> to "> " PS2 prompts upon sending incomplete script lines are also
> passed.
>
> Is it possible to distinguish the actual script output, empty lines in
> the actual script output, and the incomplete prompts?

You could prepend your multiline command with a dummy
'echo multiline_starts_here' command have your
'comint-output-filter-functions' discard output that arrives before
"multiline_starts_here".

Or maybe inject something like 'oldps=$PS2; PS2=""' before the command
and 'PS2=$oldps' after the command. But whether this works could depend
on the shell, some shells might have something like $PS3 or $RPS1, for
example.

You could also accept-process-output after sending each individual line
separately, but this depends on the user not customizing PS2="" in his
bashrc.

As far as Emacs is concerned, PS1/PS2 prompts and non-prompt process
output is all data, read from a single file descriptor. I don't think
it's possible to distinguish them reliably without the above tricks or
some other heuristics.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: comint-output-filter-functions and multi-line input

[Ihor Radchenko]

miha@kamnitnik.top writes:

>> Is it possible to distinguish the actual script output, empty lines in
>> the actual script output, and the incomplete prompts?
>
> You could prepend your multiline command with a dummy
> 'echo multiline_starts_here' command have your
> 'comint-output-filter-functions' discard output that arrives before
> "multiline_starts_here".

For context, I am asking because I am trying to figure out how ob-shell
works in Org mode. My multiline command can be arbitrary bash script
containing multiple single- and multi-line commands. It is hard to
figure out which one is which.

> Or maybe inject something like 'oldps=$PS2; PS2=""' before the command
> and 'PS2=$oldps' after the command. But whether this works could depend
> on the shell, some shells might have something like $PS3 or $RPS1, for
> example.

> You could also accept-process-output after sending each individual line
> separately, but this depends on the user not customizing PS2="" in his
> bashrc.

But will accept-process-output correctly work with PS2=""?

Org uses the following black magic to detect that output is actually
produced:

(org-babel-comint-with-output
		   (session org-babel-sh-eoe-output t body)
		 (dolist (line (append (split-string (org-trim body) "\n")
				       (list org-babel-sh-eoe-indicator)))
		   (insert line)
		   (comint-send-input nil t)
		   (while (save-excursion
			    (goto-char comint-last-input-end)
			    (not (re-search-forward
				  comint-prompt-regexp nil t)))
		     (accept-process-output
		      (get-buffer-process (current-buffer))))))

Note that in addition to accept-process-output, we have to match against
comint-prompt-regexp. Not doing so sometimes lead to infinite loops.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: comint-output-filter-functions and multi-line input

[miha]

[-- Attachment #1: Type: text/plain, Size: 1144 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> miha@kamnitnik.top writes:
>
>>> Is it possible to distinguish the actual script output, empty lines in
>>> the actual script output, and the incomplete prompts?
>>
>> You could prepend your multiline command with a dummy
>> 'echo multiline_starts_here' command have your
>> 'comint-output-filter-functions' discard output that arrives before
>> "multiline_starts_here".
>
> For context, I am asking because I am trying to figure out how ob-shell
> works in Org mode. My multiline command can be arbitrary bash script
> containing multiple single- and multi-line commands. It is hard to
> figure out which one is which.

I see. Perhaps you could turn your bash script into a single multi-line
command by encapsulating it into an 'if true' block. So given

    if [ $(( 1 + 1 )) = 2 ]
    then
        echo "hello"
    fi
    echo $(( 3 + 4 ))

you could encapsulate it into something like

    if true; then
        echo "multiline_starts_here"

        if [ $(( 1 + 1 )) = 2 ]
        then
            echo "hello"
        fi
        echo $(( 3 + 4 ))

        echo "multiline_ends_here"
    fi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: comint-output-filter-functions and multi-line input

[Ihor Radchenko]

<miha@kamnitnik.top> writes:

>> For context, I am asking because I am trying to figure out how ob-shell
>> works in Org mode. My multiline command can be arbitrary bash script
>> containing multiple single- and multi-line commands. It is hard to
>> figure out which one is which.
>
> I see. Perhaps you could turn your bash script into a single multi-line
> command by encapsulating it into an 'if true' block. So given
>
>     if [ $(( 1 + 1 )) = 2 ]
>     then
>         echo "hello"
>     fi
>     echo $(( 3 + 4 ))
>
> you could encapsulate it into something like
>
>     if true; then
>         echo "multiline_starts_here"
>
>         if [ $(( 1 + 1 )) = 2 ]
>         then
>             echo "hello"
>         fi
>         echo $(( 3 + 4 ))
>
>         echo "multiline_ends_here"
>     fi

There could be multiple such blocks in the script. And it may not be
bash, but something else (fish, csh, posh, etc). Finding ways to do the
wrapping for all the shell flavours will be maintenance hell.

So, I'd prefer comint-based solution.
Such solution may also benefit other comint uses beyond this specific
issue with Org.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: comint-output-filter-functions and multi-line input

[miha]

[-- Attachment #1: Type: text/plain, Size: 1713 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> There could be multiple such blocks in the script. And it may not be
> bash, but something else (fish, csh, posh, etc). Finding ways to do the
> wrapping for all the shell flavours will be maintenance hell.

Well we already have something like this in org-babel-sh-eoe-indicator.
Having similar variables for ob-fish.el, ob-csh.el and others doesn't
really sound like too much trouble for me, you'd need something like

(defvar org-babel-sh-boe-indicator "if true; then echo org_babel_sh_boe\n")
(defvar org-babel-sh-eoe-indicator "\necho org_babel_sh_eoe; fi")
(defvar org-babel-sh-boe-output "org_babel_sh_boe")
(defvar org-babel-sh-eoe-output "org_babel_sh_eoe")

and similar for other ob-*.el's. Though, thinking about this more, for
the case of ob-python.el, where indentation matters, it does indeed
sound like trouble. In addition to pre-pending an "if True:", we'd also
have to indent the subsequent lines.

> So, I'd prefer comint-based solution.
> Such solution may also benefit other comint uses beyond this specific
> issue with Org.

>> Or maybe inject something like 'oldps=$PS2; PS2=""' before the command
>> and 'PS2=$oldps' after the command. But whether this works could depend
>> on the shell, some shells might have something like $PS3 or $RPS1, for
>> example.

> But will accept-process-output correctly work with PS2=""?

Looking at the code in ob-shell.el, it probably wouldn't just work if
you were to prepend your command with PS2="". You'd probably have to
move accept-process-output at the end outside of the loop. (Inside the
loop, accept-process-output could wait forever, since the process
wouldn't print anything, PS2="" after all.)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: comint-output-filter-functions and multi-line input

[Ihor Radchenko]

<miha@kamnitnik.top> writes:

>> So, I'd prefer comint-based solution.
>> Such solution may also benefit other comint uses beyond this specific
>> issue with Org.
>
>>> Or maybe inject something like 'oldps=$PS2; PS2=""' before the command
>>> and 'PS2=$oldps' after the command. But whether this works could depend
>>> on the shell, some shells might have something like $PS3 or $RPS1, for
>>> example.
>
>> But will accept-process-output correctly work with PS2=""?
>
> Looking at the code in ob-shell.el, it probably wouldn't just work if
> you were to prepend your command with PS2="". You'd probably have to
> move accept-process-output at the end outside of the loop. (Inside the
> loop, accept-process-output could wait forever, since the process
> wouldn't print anything, PS2="" after all.)

At the end, I did go into PS2="", but instead of sending script
line-by-line, I just send it once and for all outside the loop, as you
suggested. It also works and looks less fragile.

Also, I figured something called PROMPT_COMMAND, which can also
interfere with prompt. Just using PS1/2 is not always enough.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: comint-output-filter-functions and multi-line input

[Stefan Monnier]

> There could be multiple such blocks in the script. And it may not be
> bash, but something else (fish, csh, posh, etc). Finding ways to do the
> wrapping for all the shell flavours will be maintenance hell.
>
> So, I'd prefer comint-based solution.
> Such solution may also benefit other comint uses beyond this specific
> issue with Org.

Comint is designed for interactive use.
You're trying to use to get output like you'd get from a batch use.
And you want it to work for "arbitrary" shells.

I think you can get 2 of the above 3 without too much trouble, but
combining all 3 is going to be really hard.


        Stefan

Re: comint-output-filter-functions and multi-line input

[Stefan Monnier]

Ihor Radchenko [2022-10-16 08:48:11] wrote:
> Each line is submitted to shell via comint-send-input and, AFAIK, the
> result can be collected using `comint-output-filter-functions'.

`comint-output-filter-functions` is probably run too late.  It's meant
for code to react to changes in the *buffer*, I think (take with
a heavy dose of salt, because I could never really understand the
design).
I suspect that `comint-preoutput-filter-functions` is closer to what you
need/want.


> I expect the `comint-output-filter-functions' to be called on "hello"
> line. However, it does not seem to be the case. Empty "" corresponding
> to "> " PS2 prompts upon sending incomplete script lines are also
> passed.

I suspect that those "" actually aren't called asynchronously in
response to process output (like the "> " prompts), but rather they're
called synchronously directly from `comint-send-input` (not sure why we
do that either).


        Stefan

Re: comint-output-filter-functions and multi-line input

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Ihor Radchenko [2022-10-16 08:48:11] wrote:
>> Each line is submitted to shell via comint-send-input and, AFAIK, the
>> result can be collected using `comint-output-filter-functions'.
>
> `comint-output-filter-functions` is probably run too late.  It's meant
> for code to react to changes in the *buffer*, I think (take with
> a heavy dose of salt, because I could never really understand the
> design).
> I suspect that `comint-preoutput-filter-functions` is closer to what you
> need/want.

I tried to use the suggested hook, but it is also supplied with
redundant empty output.

For some contents, I am trying to debug

#+begin_src bash :session test :results output
if true
 then
  echo "hello"
fi
#+end_src

#+RESULTS:
: > > > hello
  ^ ^ ^
  excessive empty output lines matching the number of extra lines in the script.

source blocks with multi-line input executed using 'ob-shell.

The main work is done in `org-babel-sh-evaluate' that feeds bash comint
buffer line by line with comint-send-input and read the output using
comint-output-filter-functions (in org-babel-comint-with-output).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: comint-output-filter-functions and multi-line input

[Rudolf Adamkovič]

Ihor Radchenko <yantar92@posteo.net> writes:

> #+RESULTS:
> : > > > hello
>   ^ ^ ^
>   excessive empty output lines [...]

Given ...

#### (comint-run "/bin/bash")

... the following ...

#### (with-current-buffer "*bash*"
####   (insert "if true
#### then
#### echo hello
#### fi")
####   (comint-send-input))

... outputs ...

#### $ if true
#### then
#### echo hello
#### fi
#### > > > hello

Not good.

Similarly, ...

#### (with-current-buffer "*bash*"
####   (dolist (line (list "if true"
####                       "then"
####                       "echo hello"
####                       "fi"))
####     (insert line)
####     (comint-send-input)))

... also gives the same output.

(I took this as a minimal working bug reproducer).

After investigating, I found that ...

#### (with-current-buffer "*bash*"
####   (let ((comint-process-echoes t))
####     (dolist (line (list "if true"
####                         "then"
####                         "echo hello"
####                         "fi"))
####       (insert line)
####       (comint-send-input))))

... outputs (perhaps the desired) ...

$ if true
> then
> echo hello
> fi
hello

Would this help?

Rudy
-- 
"Logic is a science of the necessary laws of thought, without which no
employment of the understanding and the reason takes place."
-- Immanuel Kant, 1785

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: comint-output-filter-functions and multi-line input

[Ihor Radchenko]

Rudolf Adamkovič <salutis@me.com> writes:

> After investigating, I found that ...
>
> #### (with-current-buffer "*bash*"
> ####   (let ((comint-process-echoes t))
> ####     (dolist (line (list "if true"
> ####                         "then"
> ####                         "echo hello"
> ####                         "fi"))
> ####       (insert line)
> ####       (comint-send-input))))
>
> ... outputs (perhaps the desired) ...
>
> $ if true
>> then
>> echo hello
>> fi
> hello
>
> Would this help?

Not always. It may depend on terminal settings. In particular, on the
PS1, PS2, and PROMPT_COMMAND values.

I ended up getting rid of the whole line-by-line loop.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>