bash source code block: problem after ssh commands

bash source code block: problem after ssh commands

[Alain.Cochard]

Hello.

For me, 'C-c C-c' on the following group

   #+begin_src bash :results output
   echo "foo"
   echo "bar"
   #+end_src

works as I expect: 'foo' and 'bar' are echo'ed.  But it fails on this
one:

   #+begin_src bash :results output
   ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
   echo "bar"
   #+end_src

The file 'foo_file' is created on the remote machine, but 'bar' is not
echo'ed.  I have tried to insert other commands between the ssh and
echo commands, or append semicolons at the end of the commands,
without success.

By contrast, it works with this one:

   #+begin_src bash :results output
   ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
   #+end_src

Perhaps it is worth noting that I observe the same behavior if I
simply copy/yank the commands (I mean: the two commands yank'ed at
once) in an emacs terminal (obtained with 'M-x shell').  However,
everything works as I expect if I copy/yank the commands (all at once)
in a "regular" X terminal (xfce in my case).

I wish I could understand what is going on.

Thanks

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Leo Butler]

On Wed, Oct 25 2023, Alain.Cochard@unistra.fr wrote:

> Hello.
>
> For me, 'C-c C-c' on the following group
>
>    #+begin_src bash :results output
>    echo "foo"
>    echo "bar"
>    #+end_src
>
>
> works as I expect: 'foo' and 'bar' are echo'ed.  But it fails on this
> one:
>
>    #+begin_src bash :results output
>    ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
>    echo "bar"
>    #+end_src
>
>
> The file 'foo_file' is created on the remote machine, but 'bar' is not
> echo'ed.  I have tried to insert other commands between the ssh and
> echo commands, or append semicolons at the end of the commands,
> without success.
>
> By contrast, it works with this one:
>
>    #+begin_src bash :results output
>    ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
>    #+end_src
>
> Perhaps it is worth noting that I observe the same behavior if I
> simply copy/yank the commands (I mean: the two commands yank'ed at
> once) in an emacs terminal (obtained with 'M-x shell').  However,
> everything works as I expect if I copy/yank the commands (all at once)
> in a "regular" X terminal (xfce in my case).

It looks like an issue in comint-mode and the way that it is handling
the temporary ssh session. This is not an issue with org-mode.

Here are a couple work-arounds:

- add the -f to the ssh command
- add a semi-colon and line continuation to the first line.

Working examples:

#+begin_src bash :results output
  ssh -f SSH-HOST 'echo foo>/tmp/foo_file'
  echo $(uname -a) |tee /tmp/uname.txt
#+end_src

#+RESULTS:
: Linux t14 6.5.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.3-1 (2023-09-13) x86_64 GNU/Linux

#+begin_src bash :results output
  ssh pearce-120 'echo foo>/tmp/foo_file' &&\
  echo $(uname -a) | tee /tmp/uname.txt
#+end_src

#+RESULTS:
: Linux t14 6.5.0-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.3-1 (2023-09-13) x86_64 GNU/Linux

Not working:

#+begin_src bash :results output
  ssh SSH-HOST 'echo foo>/tmp/foo_file'
  echo $(uname -a) |tee /tmp/uname1.txt
#+end_src

If you look at /tmp after running the third example, you see that that
second command line has not been executed.

Leo

Re: bash source code block: problem after ssh commands

[Alain.Cochard]

Thanks much for the detailed explanations.

 > It looks like an issue in comint-mode and the way that it is
 > handling the temporary ssh session. This is not an issue with
 > org-mode.

So it is an issue with emacs, right?

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Leo Butler]

On Wed, Oct 25 2023, Alain.Cochard@unistra.fr wrote:

> Thanks much for the detailed explanations.
>
>  > It looks like an issue in comint-mode and the way that it is
>  > handling the temporary ssh session. This is not an issue with
>  > org-mode.
>
> So it is an issue with emacs, right?

Yes, that is my understanding. Perhaps you can report it to the
emacs-devel mailing list.

Leo

Re: bash source code block: problem after ssh commands

[yaxp]

> So it is an issue with emacs, right?

comint-mode

-- 
(yaxp me) => t

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Leo Butler <Leo.Butler@umanitoba.ca> writes:

> It looks like an issue in comint-mode and the way that it is handling
> the temporary ssh session. This is not an issue with org-mode.
> ...
> Not working:
>
> #+begin_src bash :results output
>   ssh SSH-HOST 'echo foo>/tmp/foo_file'
>   echo $(uname -a) |tee /tmp/uname1.txt
> #+end_src

By default (no session), the above command should not use comint-mode.
So, something else might be going on here.

I currently do not have a setup to test ssh commands, so I would
appreciate someone trying to dig why `org-babel-eval' apparently not
returning the expected result.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Alain.Cochard]

Ihor Radchenko writes on Thu 26 Oct 2023 08:44:

 > I currently do not have a setup to test ssh commands,

Sorry if that's irrelevant: I realized that one can ssh to the *same*
machine.

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Alain.Cochard@unistra.fr writes:

> Ihor Radchenko writes on Thu 26 Oct 2023 08:44:
>
>  > I currently do not have a setup to test ssh commands,
>
> Sorry if that's irrelevant: I realized that one can ssh to the *same*
> machine.

You are indeed right.
I can now reproduce the problem locally.

It boils down to

(setq exit-status
	  (process-file shell-file-name input-file
			(if error-file
			    (list t error-file)
			  t)
			nil shell-command-switch command))

that is an equivalent of

bash -c bash /path/to/file-containing-the-source-code.sh

----- file-containing-the-source-code.sh ----
 ssh localhost 'echo foo>/tmp/foo_file'
  echo $(uname -a) |tee /tmp/uname1.txt
---------------------------------------------

If one tries to evaluate the above, the second line is not produced. It
has nothing to do with Emacs itself.

However, if I try

(process-file "bash" "/tmp/test.sh"), the /tmp/uname1.txt is not
produced.

This time, it is Emacs problem.
The problem is caused by interactive password prompt displayed by ssh.

I guess we may try to report this upstream, though I am not sure if this
is something that is supported to start with.

From Org perspective, it is not the first time when interactive shell
programs are causing problems. I'd say that we have no technical ability
to support them reliably and one has to use ob-screen or similar
libraries to properly interact with such code blocks.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Russell Adams]

On Wed, Oct 25, 2023 at 01:17:42PM +0200, Alain.Cochard@unistra.fr wrote:
>    #+begin_src bash :results output
>    ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
>    echo "bar"
>    #+end_src

I know that Ihor has already reproduced, but are you using an SSH key
to connect, or entering a password?

If entering a password, I'd expect there is some mixup. That I defer
to Ihor.

If you use an SSH key for passwordless access, try adding -n (ie: "ssh
-n derp@host mycommand"). The "-n" flag helps prevent the remote
command from interfering with your local terminal by redirecting
/dev/null as the remote stdin. I have to use this commonly in systems
administration, scripts, and tools like Ansible.

Thanks.

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com
                                    https://www.adamsinfoserv.com/

Re: bash source code block: problem after ssh commands

[Alain.Cochard]

Russell Adams writes on Thu 26 Oct 2023 16:44:

 > [...] are you using an SSH key to connect, or entering a password?

 > If you use an SSH key for passwordless access, try adding -n (ie:
 > "ssh -n derp@host mycommand"). 

Yes, I use an SSH key, and yes, '-n' saves the day.  Thanks.

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Alain.Cochard]

Ihor Radchenko writes on Thu 26 Oct 2023 13:44:

 > I can now reproduce the problem locally.
 > 
 > It boils down to
 > 
 > (setq exit-status
 > 	  (process-file shell-file-name input-file
 > 			(if error-file
 > 			    (list t error-file)
 > 			  t)
 > 			nil shell-command-switch command))
 > 
 > that is an equivalent of
 > 
 > bash -c bash /path/to/file-containing-the-source-code.sh
 > 
 > ----- file-containing-the-source-code.sh ----
 >  ssh localhost 'echo foo>/tmp/foo_file'
 >   echo $(uname -a) |tee /tmp/uname1.txt
 > ---------------------------------------------

 > If one tries to evaluate the above, the second line is not produced. It
 > has nothing to do with Emacs itself.

I am confused about what you specifically do to "evaluate the above".
To start with, I have to use quotes to make your command be performed:

   bash -c "bash /path/to/file-containing-the-source-code.sh"

Without the quotes, the 1st line is not executed either.  NB: I get
the same behavior with simply

   bash /path/to/file-containing-the-source-code.sh

or, after making the script executable with 'chmod +x':

   /path/to/file-containing-the-source-code.sh

But most importantly, the second line *is* produced, either if I use
an SSH key for passwordless access or if I enter the password
manually.

 > However, if I try
 > 
 > (process-file "bash" "/tmp/test.sh"), the /tmp/uname1.txt is not
 > produced.

Here too, it is not clear to me how you "try".  The way I know is to
highlight

   (process-file "bash" "/path/to/file-containing-the-source-code.sh")

and do

   M-x eval-region

In this case, indeed, the 1st line is executed but not the 2nd one
(again, whether or not I use an SSH key).


-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 28/10/2023 01:26, Alain.Cochard@unistra.fr wrote:
>     bash -c "bash /path/to/file-containing-the-source-code.sh"
> 
> Without the quotes, the 1st line is not executed either.  NB: I get
> the same behavior with simply

Without quotes it means: execute shell script (just "bash" this case) 
with "$0" set to "/path/to/file-containing-the-source-code.sh". 
"/path/..." is ignored because shell script does not contain "$0". It is 
a way to safely pass arguments to shell, however "$0" is confusing:

     sh -c 'echo "arg0: $0"; echo  args: "$@"' arg0 arg1 arg2
     arg0: arg0
     args: arg1 arg2

In this example "arg0: $0"; echo  args: "$@"'  has the same role as 
"bash" in the cited command.

If you need to pass another variable into "sh -c" the use

var="/bin/sh"
sh -c 'ls -l "$1"' argv0-for-sh "$var"

Unsafe variant ("var" contain quotes): sh -c "ls -l '$var'". The same 
warning is applicable to "emacs --eval "(func \"$var\")".

> or, after making the script executable with 'chmod +x':
> 
>     /path/to/file-containing-the-source-code.sh

Always add a shebang ("#!/bin/sh", "#!/bin/bash", etc.) to executable 
scripts. A chance to face a bug is a shell is higher when it is left up 
to the shell and the kernel what to do with files without shebangs.

Sorry, I have not tried to debug the original issue.

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Hi,

Alain.Cochard@unistra.fr writes:

> Ihor Radchenko writes on Thu 26 Oct 2023 13:44:
>
>  > I can now reproduce the problem locally.
>  > 
>  > It boils down to
>  > 
>  > (setq exit-status
>  > 	  (process-file shell-file-name input-file
>  > 			(if error-file
>  > 			    (list t error-file)
>  > 			  t)
>  > 			nil shell-command-switch command))
>  > 
>  > that is an equivalent of
>  > 
>  > bash -c bash /path/to/file-containing-the-source-code.sh
>  > 
...
> I am confused about what you specifically do to "evaluate the above".
> To start with, I have to use quotes to make your command be performed:
>
>    bash -c "bash /path/to/file-containing-the-source-code.sh"
>
...
>
> But most importantly, the second line *is* produced, either if I use
> an SSH key for passwordless access or if I enter the password
> manually.

IIUC, the elisp expression:

    (process-file "bash" "/tmp/test.sh")

is more equivalent to:

    cat /tmp/test.sh | bash

i.e. the shell is getting the commands from stdin.  Thus, any command
that uses stdin might change what gets executed or not.

I'm able to reproduce using the following minimal script, without
passwords nor SSH (where the cryptic first line closes stdin).

    #+begin_src shell :results output
      exec 0>&-
      echo OK
    #+end_src

The result is "OK" only when commenting out the first line; else, the
echo command is not executed (because stdin has been closed).

Here is an other example, where the second echo is eaten by the script
itself:

    #+begin_src shell :results output
      echo 1
      read -p "Next command? " NEXT_COMMAND
      echo 2
      echo 3
    #+end_src

    #+RESULTS:
    : 1
    : 3

Bruno

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

> IIUC, the elisp expression:
>
>     (process-file "bash" "/tmp/test.sh")
>
> is more equivalent to:
>
>     cat /tmp/test.sh | bash
>
> i.e. the shell is getting the commands from stdin.  Thus, any command
> that uses stdin might change what gets executed or not.

Looking at `org-babel-sh-evaluate', it should be enough to specify
:shebang, :cmdline, or :stdin header argument to force using script
rather than channel source block as input to bash:

#+begin_src shell :results output :cmdline bash
exec 0>&-
echo OK
#+end_src

#+RESULTS:
: OK


#+begin_src shell :results output :cmdline bash
echo 1
read -p "Next command? " NEXT_COMMAND
echo 2
echo 3
#+end_src

#+RESULTS:
: 1
: 2
: 3

I am wondering about the possible downsides of using script approach
instead of stdin redirection.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Matt]

 ---- On Fri, 27 Oct 2023 13:47:24 +0200  Alain.Cochard@unistra.fr wrote --- 
 > Russell Adams writes on Thu 26 Oct 2023 16:44:
 >  > [...] are you using an SSH key to connect, or entering a password?
 >  > If you use an SSH key for passwordless access, try adding -n (ie:
 > 
 >  > "ssh -n derp@host mycommand"). 
 > 
 > Yes, I use an SSH key, and yes, '-n' saves the day.  Thanks.
 
Hi Alain,

I'm the supposed maintainer of ob-shell who's been missing in action for this whole discussion.

I want to confirm, are you able to accomplish your task by using '-n'?

Matt

Re: bash source code block: problem after ssh commands

[Matt]

 ---- On Mon, 06 Nov 2023 14:32:16 +0100  Ihor Radchenko  wrote --- 
 > I am wondering about the possible downsides of using script approach
 > instead of stdin redirection.
 
I'm curious to hear more about what you're thinking.

Re: bash source code block: problem after ssh commands

[Alain.Cochard]

Matt writes on Mon  6 Nov 2023 19:01:

 > I want to confirm, are you able to accomplish your task by using
 > '-n'?

Yes.  Evaluating this block (with C-c C-c):

   #+begin_src bash :results output
   ssh -n cochard@fruc.u-strasbg.fr "echo foo>foo_file"
   echo "bar"
   #+end_src

gives

   #+RESULTS:
   : bar

(without the '-n', #+RESULTS: is empty).  Same conclusion if I
copy/yank the 2 bash instructions in an emacs terminal (M-x shell). 

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

>  ---- On Mon, 06 Nov 2023 14:32:16 +0100  Ihor Radchenko  wrote --- 
>  > I am wondering about the possible downsides of using script approach
>  > instead of stdin redirection.
>  
> I'm curious to hear more about what you're thinking.

I am thinking to change the
  (t (org-babel-eval shell-file-name (org-trim body)))
clause in `org-babel-sh-evaluate' to something that uses a script file.

It will clearly solve the discussed problem, possibly at the cost of
small overhead to write the script file to disk.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Matt]

 ---- On Tue, 07 Nov 2023 09:53:46 +0100  Ihor Radchenko  wrote --- 
 > Matt matt@excalamus.com> writes:
 > 
 > >  ---- On Mon, 06 Nov 2023 14:32:16 +0100  Ihor Radchenko  wrote --- 
 > >  > I am wondering about the possible downsides of using script approach
 > >  > instead of stdin redirection.
 > >  
 > > I'm curious to hear more about what you're thinking.
 > 
 > I am thinking to change the
 >   (t (org-babel-eval shell-file-name (org-trim body)))
 > clause in `org-babel-sh-evaluate' to something that uses a script file.
 > 
 > It will clearly solve the discussed problem, possibly at the cost of
 > small overhead to write the script file to disk.
 
Thanks for clarifying.  I've been away from the codebase for a bit and, now that the FSF paperwork is signed (still need to get Craig a copy), I'm reviewing =ob-shell= with an eye for what could be cleaned or improved.   I feel like =org-babel-sh-evaluate= could use some attention.

I'm open to your suggestion.  The good is that using a script is how :shebang and :cmdline are processed currently (like you pointed out) so there's precedence and experience with it.  Also, it would make all non-session execution use the same model (script versus comint).  I like how that would create a clean separation.

For bad, nothing jumps out to me as obviously a problem.  Let me "think out loud" for a moment.  We'd need to write to disk.  Like you say, this incurs overhead opening, writing, and closing the file.  It's not like we'd forget to close it, though.  Nor is running out of space or inodes our problem.  Writing requires permission.  That's not an issue with /tmp.  Then, it needs to execute.  Aside from permission, any code we insert needs to be correct.  For example, a shebang would need to point to the correct application and any arguments would need to correspond to the implementation being called.  I doubt we'd need anything beyond /bin/<shell>.   FWIW, it looks like there's been at least one instance where :shebang's formatting was questioned (https://yhetil.org/orgmode/CA+A2iZZ1vMmKiUf4Fem1AU7CA1m9GQap+BkvrOsz+0BxRt6rRA@mail.gmail.com/).  We'd also need to control for what environment the script runs in.  That was another issue I saw raised in the list (https://yhetil.org/orgmode/87609ug5ae.fsf@luisa.c0t0d0s0.de/).  Of course, we'd need to read the stdout and stderr.  This is handled by =process-file=.  Any step I missed or some kind of failure I didn't consider?

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

> ... Aside from permission, any code we
> insert needs to be correct. For example, a shebang would need to point
> to the correct application and any arguments would need to correspond
> to the implementation being called. I doubt we'd need anything beyond
> /bin/<shell>. FWIW, it looks like there's been at least one instance
> where :shebang's formatting was questioned
> (https://yhetil.org/orgmode/CA+A2iZZ1vMmKiUf4Fem1AU7CA1m9GQap+BkvrOsz+0BxRt6rRA@mail.gmail.com/).

It is a problem, but I am not sure how useful it is to solve the line
numbering there. A bigger problem is that errors *Org Babel Error
Output* are not clickable:

Try
#+begin_src bash
cdf
#+end_src
and you will get
------ *Org Babel Error Output* ---------
bash: line 1: cdf: command not found
[ Babel evaluation exited with code 127 ]
-----------------------------------------

Clicking on "line 1" will yield file opening dialogue.

However, there is yet another problem revealed when we try

#+begin_src bash :cmdline bash
cdf
#+end_src

#+RESULTS:
: /tmp/babel-47DarV/sh-script-vkbRfJ: line 2: cdf: command not found

There is no *Org Babel Error Output* buffer at all.

So, it looks like we may need an alternative `org-babel-eval' function
that works with script files rather than input and arranges
stdout/stderr separation.

> We'd also need to control for what environment the script runs in.
> That was another issue I saw raised in the list
> (https://yhetil.org/orgmode/87609ug5ae.fsf@luisa.c0t0d0s0.de/). Of
> course, we'd need to read the stdout and stderr. This is handled by
> =process-file=. Any step I missed or some kind of failure I didn't
> consider?

Valid concern. We should reuse `process-file' in this scenario as well,
but utilizing its ARGS argument + arrange to generate the temporary
script file in the appropriate context - see `make-nearby-temp-file'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Matt]

 ---- On Thu, 09 Nov 2023 13:13:36 +0100  Ihor Radchenko  wrote --- 

 > A bigger problem is that errors *Org Babel Error
 > Output* are not clickable:
 
Interesting.

 > So, it looks like we may need an alternative `org-babel-eval' function
 > that works with script files rather than input and arranges
 > stdout/stderr separation.

Yes, it seems like a separate handler may be in order.

 > We should reuse `process-file' in this scenario as well,
 > but utilizing its ARGS argument + arrange to generate the temporary
 > script file in the appropriate context - see `make-nearby-temp-file'.

I was unaware of `make-nearby-temp-file'.  Thank you.  It's helpful for this situation.

I'm not in a position to address these points right now.  My main focus is re-familiarizing myself with the =ob-shell= and =ob-comint= source.  When I was implementing async shell evaluatation earlier this year, I remember having the impression that execution paths, especially `org-babel-comint-with-output', could probably be simplified.  It seems to me that everything we're talking about in this thread aligns with that objective.  I've made a note of your points and have every intention to return to them when I get to writing code.  However, don't let me stop you if this is something you were interested in doing.  I'll do what I can to assist.

Re: bash source code block: problem after ssh commands

[Matt]

I was poking around, learning how sessions are started.  Basically, `shell` creates the comint buffer using `make-comint-in-buffer`.  What I find interesting is that `make-comint-in-buffer` can also create a comint buffer from a network stream:

(let ((buff "*localhost-process-buffer*"))
  (switch-to-buffer
   (make-comint-in-buffer
    "localhost-process"
    buff
    "ssh"
    nil
    (format "%s@localhost" (getenv "USER")))))

So, rather than start a comint and then ssh from there, it's possible to let Emacs start the subprocess, manage the ssh connection, and just read that.  Emacs makes a buffer from a network stream by calling start-process` using `shell-file-name`.  `org-babel-shell-initialize` closes around `shell-file-name` with whatever shell language is used.  Therefore, we could provide header arguments to pass parameters and the destination to ssh.  

I haven't made a judgment yet about whether any of this is good or bad.   I thought it was interesting and figured I'd share.

Re: bash source code block: problem after ssh commands

[Matt]

 ---- On Wed, 15 Nov 2023 17:32:19 +0100  Matt  wrote --- 

 > (let ((buff "*localhost-process-buffer*"))
 >   (switch-to-buffer
 >    (make-comint-in-buffer
 >     "localhost-process"
 >     buff
 >     "ssh"
 >     nil
 >     (format "%s@localhost" (getenv "USER")))))

It looks like the way this is called actually uses `start-file-process` rather than `open-network-stream`.  It's not clear to me how to call it so that `open-network-stream` is used.  However, that's more a question for the Emacs mailing list and, honestly, not one that I think I care about right now.  Posting this just to correct what I said before.

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

> So, rather than start a comint and then ssh from there, it's possible to let Emacs start the subprocess, manage the ssh connection, and just read that.  Emacs makes a buffer from a network stream by calling start-process` using `shell-file-name`.  `org-babel-shell-initialize` closes around `shell-file-name` with whatever shell language is used.  Therefore, we could provide header arguments to pass parameters and the destination to ssh.  

Or we can make use of TRAMP as we usually do to access remote
environment. IMHO, it is more reliable as TRAMP takes care about
arranging all the Elisp FS interaction to work on remote system.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Matt]

 ---- On Thu, 16 Nov 2023 10:30:59 +0100  Ihor Radchenko  wrote --- 

 > Or we can make use of TRAMP as we usually do to access remote
 > environment. IMHO, it is more reliable as TRAMP takes care about
 > arranging all the Elisp FS interaction to work on remote system.
 
Agreed.  Also, I wasn't aware TRAMP worked with shell block evaluation.  That's nice.

Sure enough, these both work:

#+begin_src bash :results output :dir /ssh:user@localhost:/home/user :session *remote*
echo foo>foo_file
echo "bar"
#+end_src

#+begin_src bash :results output :dir /ssh:user@localhost:/home/user
echo foo>foo_file
echo "bar"
#+end_src

Of course, both calls happen on the remote.

I wonder, would this have helped Alain?

I'm admittedly a little confused about the original report,  

#+name: original-report
#+begin_src bash :results output
ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
echo "bar"
#+end_src

Clearly, the ssh line is intended to happen on the remote server.  Was the echo "bar" intended to happen locally?

PS:  I've updated the WORG page for ob-shell with a :dir section (https://git.sr.ht/~bzg/worg/commit/0b85a4fcfe2fa5e6c1ac4edd3f664a8cc385fa54)

Re: bash source code block: problem after ssh commands

[Alain.Cochard]

Matt writes on Thu 16 Nov 2023 20:03:
 > 
 >  ---- On Thu, 16 Nov 2023 10:30:59 +0100  Ihor Radchenko  wrote --- 
 > 
 >  > Or we can make use of TRAMP as we usually do to access remote
 >  > environment. IMHO, it is more reliable as TRAMP takes care about
 >  > arranging all the Elisp FS interaction to work on remote system.
 >  
 > Agreed.  Also, I wasn't aware TRAMP worked with shell block
 > evaluation.  That's nice.
 > 
 > Sure enough, these both work:
 > 
 > #+begin_src bash :results output :dir /ssh:user@localhost:/home/user :session *remote*
 > echo foo>foo_file
 > echo "bar"
 > #+end_src
 > 
 > #+begin_src bash :results output :dir /ssh:user@localhost:/home/user
 > echo foo>foo_file
 > echo "bar"
 > #+end_src
 > 
 > Of course, both calls happen on the remote.
 > 
 > I wonder, would this have helped Alain?

No (see below).

 > I'm admittedly a little confused about the original report,  
 > 
 > #+name: original-report
 > #+begin_src bash :results output
 > ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
 > echo "bar"
 > #+end_src
 > 
 > Clearly, the ssh line is intended to happen on the remote server.
 > Was the echo "bar" intended to happen locally?

Yes.  (So I think that explains the 'no' above.)  Then I'm confused
about your confusion...

Anyway, this gives me the opportunity to come back to the question of
whether or not there is a problem with emacs itself (like some people
here thought), and if some message should be sent to some emacs list.
Again my argument was that the 2 commands copy/yank'ed in an emacs
terminal don't work as expected, while they do if the same is done in
an X terminal.

I guess it is not important for Org, because workarounds have be
proposed, but I think if there is a problem with emacs it should be
reported, right?

Thank you.

-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Matt]

alaincochard alain.cochard@unistra.fr writes:

 > Yes.  (So I think that explains the 'no' above.) 

Thank you for clarifying.

 > Anyway, this gives me the opportunity to come back to the question of
 > whether or not there is a problem with emacs itself (like some people
 > here thought), and if some message should be sent to some emacs list.
 > Again my argument was that the 2 commands copy/yank'ed in an emacs
 > terminal don't work as expected, while they do if the same is done in
 > an X terminal.

Okay, I follow you now.

Yes, I agree with what others have said, it's related to Emacs (probably comint-mode).  It happens with M-x shell and *not* with M-x eshell.   It's hard to reproduce with M-x term because term-char-mode doesn't allow copy-paste.  M-x with term-line-mode doesn' t print "bar".

 > I guess it is not important for Org, because workarounds have be
 > proposed, but I think if there is a problem with emacs it should be
 > reported, right?

I think it's worth reporting.  I'm headed to bed and can submit it tomorrow for you if you're not comfortable with doing it or don't have the time.

Here are the steps I followed to reproduce with M-x shell:

1. Copy the following two lines:

ssh $USER@localhost "echo foo>foo_file"
echo "bar"

2. Paste them into a shell (tested with xterm or xfce4-terminal) and press return
3. After entering the correct password, "bar" is echoed
4. Paste them into the buffer associated with M-x shell
5. After entering the correct password, "bar" is NOT echoed

Here are the steps I followed to reproduce with M-x term:

1. M-x term
2. C-c C-j to switch to term-line-mode
3. Copy the following two lines:

ssh $USER@localhost "echo foo>foo_file"
echo "bar"

4. Paste them into the buffer associated with M-x term
5. After entering the correct password, "bar" is NOT echoed

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

> Thank you for clarifying.
>
>  > Anyway, this gives me the opportunity to come back to the question of
>  > whether or not there is a problem with emacs itself (like some people
>  > here thought), and if some message should be sent to some emacs list.
>  > Again my argument was that the 2 commands copy/yank'ed in an emacs
>  > terminal don't work as expected, while they do if the same is done in
>  > an X terminal.
>
> Okay, I follow you now.
>
> Yes, I agree with what others have said, it's related to Emacs (probably comint-mode).  It happens with M-x shell and *not* with M-x eshell.   It's hard to reproduce with M-x term because term-char-mode doesn't allow copy-paste.  M-x with term-line-mode doesn' t print "bar".

I think that I need to clarify here.
We are talking about two different things:
1. Bash src block without session
2. Bash src block with session

The original report used bash src blocks _without_ session.
In such scenario, comint (and M-x shell) is not relevant.
Org simply uses `process-file' with INFILE argument.
This is equivalent of someone opening __X shell__, and literally typing

  ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
  echo "bar"

In the scenario described in the report, when ssh asks for password, it
is equivalent to

$ ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
Password: <typing 'echo "bar"' here>

This has nothing to do with Emacs comint and this is also not a bug in
Emacs - we use INFILE argument, that is equivalent to the above as _per
docstring_. So, Org mode is simply not using `process-file' function as
users expect - instead of executing a bash script, we emulate
interactive user input to bash.

(Side note: for someone aware about these details, it is possible to do
something like

#+begin_src bash
ssh remote-server;
<<super-secret-password-derived-from-library-of-babel-block-stored-in-safe-private-place>>
#+end_src

or, as a demo

#+begin_src bash
read x;
value2
echo "We just read \"$x\"";
#+end_src

#+RESULTS:
: We just read "value2"

<end of side note>)

Another case is (2), when we do use comint is session is what we
discussed in
https://list.orgmode.org/orgmode/CAL1eYuJntGbXY6A794qM7PTbXH3DiU1aF6OayS7CLf3kOBsbig@mail.gmail.com/
I guess we can try to report this as a bug, especially since it also
manifests itself when comint is used interactively.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Alain.Cochard]

Ihor Radchenko writes on Fri 17 Nov 2023 09:22:

 > I think that I need to clarify here.

Thank you.

Well, thank you for _trying_: most of the discussion in this thread is
way beyond my pay grade.

At the most basic user level (i.e., non lisp aware), why is it not
necessarily a bug if "something" does the expected in an X terminal
but not in an emacs terminal?  I think Matt and I (and others) are on
the same page here, and he already essentially drafted the bug report
I would have sent (only better).

What also confuses me is that it seems to me that, in the minimum
working example, you consider the ssh command and the read command as
equivalent.  But I don't even enter the password when using ssh...


-- 
EOST (École et Observatoire des Sciences de la Terre) 
ITE (Institut Terre & Environnement) | alain.cochard@unistra.fr
5 rue René Descartes   [bureau 110]  | Phone: +33 (0)3 68 85 50 44 
F-67084 Strasbourg Cedex, France     | [ slot available for rent ]

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Alain.Cochard@unistra.fr writes:

> Ihor Radchenko writes on Fri 17 Nov 2023 09:22:
>
>  > I think that I need to clarify here.
>
> Thank you.
>
> Well, thank you for _trying_: most of the discussion in this thread is
> way beyond my pay grade.

My reply was to Matthew, the current ob-shell maintainer.
If you want, you need to examine ob-shell (`org-babel-sh-evaluate') and
ob-eval (`org-babel-eval') code, and try to step through the code using
debugger to understand technical details in thread better.

> At the most basic user level (i.e., non lisp aware), why is it not
> necessarily a bug if "something" does the expected in an X terminal
> but not in an emacs terminal?  I think Matt and I (and others) are on
> the same page here, and he already essentially drafted the bug report
> I would have sent (only better).

WRT M-x shell, feel free to submit a bug report. I mostly pointed that
the problem with M-x shell is not the problem you originally ran to. It
is a different problem (also, we ran into it in the past).

> What also confuses me is that it seems to me that, in the minimum
> working example, you consider the ssh command and the read command as
> equivalent.  But I don't even enter the password when using ssh...

I was only able to reproduce your problem with ssh asking a password.
We are discussing the reproduced case.

If you see problems with
   #+begin_src bash :results output
   ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
   echo "bar"
   #+end_src
even when ssh does not ask for a password, please provide more detailed
reproducer that we can replicate locally without guessing your ssh config.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Leo Butler]

On Fri, Nov 17 2023, Ihor Radchenko <yantar92@posteo.net> wrote:

> I was only able to reproduce your problem with ssh asking a password.
> We are discussing the reproduced case.
>
> If you see problems with
>
>    #+begin_src bash :results output
>    ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
>    echo "bar"
>    #+end_src
>
> even when ssh does not ask for a password, please provide more detailed
> reproducer that we can replicate locally without guessing your ssh config.

I think you are confused, Ihor. The bug was confirmed here:

https://list.orgmode.org/orgmode/87bkcmlor9.fsf@t14.reltub.ca/

I mis-identified the culprit; you corrected me by pointing out the bug
is in `process-file'. And you confirmed the bug:

https://list.orgmode.org/orgmode/87fs1xbis1.fsf@localhost/

Russell Adams and I suggested ssh-specific work-arounds that prevent ssh
from grabbing stdin (either -n or -f).

Leo

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Hi Matt, Ihor, Alain,

Ihor Radchenko <yantar92@posteo.net> writes:

> Alain.Cochard@unistra.fr writes:
>
>> At the most basic user level (i.e., non lisp aware), why is it not
>> necessarily a bug if "something" does the expected in an X terminal
>> but not in an emacs terminal?  I think Matt and I (and others) are on
>> the same page here, and he already essentially drafted the bug report
>> I would have sent (only better).
>
> WRT M-x shell, feel free to submit a bug report. I mostly pointed that
> the problem with M-x shell is not the problem you originally ran to. It
> is a different problem (also, we ran into it in the past).

FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
multiple lines, getting the same results as when I type them manually,
or copy them line by line. My xterm doesn't seem to allow me to do that.


>> What also confuses me is that it seems to me that, in the minimum
>> working example, you consider the ssh command and the read command as
>> equivalent.  But I don't even enter the password when using ssh...

I've introduced the 'read' example, as a simpler way to modify the
standard input and get the same kind of "unexpected" results, without
relying on SSH.  It seems that I have only caused confusion, sorry about
that.

IIUC, the OP example is not working because SSH is modifying the
standard input (with or without passwords).

> I was only able to reproduce your problem with ssh asking a password.
> We are discussing the reproduced case.
>
> If you see problems with
>    #+begin_src bash :results output
>    ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
>    echo "bar"
>    #+end_src
>
> even when ssh does not ask for a password, please provide more detailed
> reproducer that we can replicate locally without guessing your ssh config.
>

I'm not the OP, but, my SSH is configured to work without passwords and
SSH is still consuming lines from standard input:

    #+begin_src bash :results output
    ssh phone echo "remote"
    echo "local"
    #+end_src

    #+RESULTS:
    : remote

It looks like it is a known SSH "feature" (see
https://unix.stackexchange.com/a/688024):

    #+begin_src bash :results output
    seq 1000000 | (ssh phone sleep 1; wc -l)
    #+end_src

    #+RESULTS:
    : 675173

Same block, but asking SSH to not use stdin (using '-n' as mentionned in
this thread):

    #+begin_src bash :results output
    seq 1000000 | (ssh -n phone sleep 1; wc -l)
    #+end_src

    #+RESULTS:
    : 1000000

IMHO, what ob-shell is doing today seems a valid way of evaluating
source blocks (and it seems to have been like that for a long time).  It
should probably be documented somewhere, so that users know how to write
their source blocks, or switch to another way, like adding a :cmdline
parameter as mentionned in this thread.


Hoping I didn't increase the confusion again,
:-)


Bruno




> -- 
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Matt]

 ---- On Fri, 17 Nov 2023 10:20:28 +0100  Ihor Radchenko  wrote --- 

 > This has nothing to do with Emacs comint and this is also not a bug in
 > Emacs 

Ihor, there were two claims made in the original report.  I was referring to Claim 2.  That deals with M-x shell and therefore comint-mode.

Regarding Claim 1:

- Can anyone verify Claim 1?
- Is anyone else unable to verify Claim 1 (like me)?
- What versions are people using?
  + M-x org-version
  + M-x emacs-version

I'm running Org mode version 9.7-pre (release_9.6.10-903-g9183e3.dirty @ /home/ahab/.emacs.d/straight/build/org/) on GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37, cairo version 1.16.0).

* The original report has two claims:
** Claim 1.
The following block is expected to write a remote file called "foo_file" with contents "foo" as well as give "bar" as the result.

    #+begin_src bash :results output
    ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file"
    echo "bar"
    #+end_src

The reported behavior is that "foo_file" is created with "foo" (with "foo" is not stated, but implied) and "bar" is *not* given as the result.

** Claim 2.
Copying and pasting the two lines from the first claim into a terminal like xfce4-terminal executes the ssh line as expected and outputs the result of the second line.  It was noted that this does not happen with M-x shell.

* Comments about the claims:
** Comment 1.
tl;dr I can't reproduce the claim that "bar" is *not* the result.  The result is "bar" for me.

The exact "expected behavior" for a shell block is a little fuzzy.  According to my analysis (given below), what Alain reports (remote file and no "bar") is the "expected" behavior.  What I see (no remote file and "bar") is actually "unexpected".

I used the following to test the claim:

    #+begin_src bash :results output
    ssh localhost "echo foo>foo_file"
    echo "bar"
    #+end_src

I am unable to reproduce the reported behavior (of "bar" not returning).  Instead, I get an ssh-askpass permission denied error, foo_file is not created, and "bar" is given as the result.  I do not see anywhere in the thread that the original claim was reproduced.

The thread preceded something like follows.

Leo Butler suggested two work arounds:

- add the -f to the ssh command
- add a semi-colon and line continuation to the first line.

Russell Adams suggested another work around:

- add -n to the ssh command

Ihor identified that a non-session call does something like the following command:

    bash -c bash /tmp/temp-file-with-source-block-code.sh

where

    ----- /tmp/temp-file-with-source-block-code -----
    ssh localhost "echo foo>foo_file"
    echo "bar" | tee /tmp/bar.txt
    -------------------------------------------------

The second line (significantly different from the original report) pipes the echo result to stdout and to a file, bar.txt.  Writing to a file allows us to confirm if that line was executed.

Ihor suggested that

    bash -c bash /tmp/temp-file-with-source-block-code.sh

does not run the second line because an interactive password prompt is displayed by ssh.  The reasoning is that the prompt hangs the process while waiting for input and the second line never runs.  Indeed, running the command does not produce /tmp/bar.txt.

Ihor is correct about prompts messing with shell blocks (this is not the first time he's seen this).  However, the way it's stated does not demonstrate it. This is because Emacs does *not* make a call like

    bash -c bash /tmp/temp-file-with-source-block-code.sh

Alain responded by pointing out that

    bash -c bash /tmp/temp-file-with-source-block-code.sh

does not execute the first line.  This is true.  Consider calling

    bash -c bash /tmp/two-lines.sh

where

    ------ /tmp/two-lines.sh ------
    echo "first" > /tmp/first.txt
    echo "second" > /tmm/second.txt
    -------------------------------

Neither first.txt or second.txt are created.

Max Nikulin interjected with a helpful reminder that Bash scripting is a snakepit of footguns.  (What Max said is more than that and interesting.  I skip it here because it depends on the form of the call.)

Before trying to untangle what a given Bash command does, we need to be sure what command is actually called.  Unfortunately, there's not a clear Bash command corresponding to how Emacs makes the call.

What happens goes something like this:

1. The Lisp function process-file is called with PROGRAM "bash", INFILE a path to a temp file containing the source block code, and ARGS ("-c" "bash")
2. This information is passed to DEFUN ("call-process"), a Lisp object implemented in C
3. DEFUN ("call-process") forwards this information to the C function call_process
4. call_process calls emacs_spawn
5. emacs_spawn creates a subprocess

A lot of cleaning and setup happens which is dependent on the system (GNU, Window, Darwin, etc.).  There's a call to openp which looks for the executable.  An emacs_pipe is set up (I assume for writing to stdout and stderr).  I'm unable to give a definitive answer as to what precisely emacs_spawn calls.  Do any of the args "get quotes?"  I can't say.

Bruno Barbier commented that his understanding of process-file is that it gets commands from stdin.  Maybe that's what the call to emacs_pipe is doing?

He gives the example:

#+begin_quote
(process-file "bash" "/tmp/test.sh")

is more equivalent to:

cat /tmp/test.sh | bash
#+end_quote

He then proposes an experiment to close stdin.  To do this, he calls

    #+begin_src shell :results output
    exec 0>&-
    echo OK
    #+end_src

He claims that "exec 0<&-" closes stdin.  I believe there is a typo.  It's not clear if it has a negative effect, though.  According to the [[https://tldp.org/LDP/abs/html/io-redirection.html][Advanced Bash-Scripting Guide]],

#+begin_quote
Closing File Descriptors

n<&-

    Close input file descriptor n.
0<&-, <&-

    Close stdin.
n>&-

    Close output file descriptor n.
1>&-, >&-

    Close stdout.
#+end_quote

What Bruno writes corresponds to "closing output file descriptor 0".  I honestly don't know what the difference is between an "output file descriptor" and an "input file descriptor".  I had no luck finding this information in man bash or info bash.

Rerunning the experiment according to the [[https://tldp.org/LDP/abs/html/io-redirection.html][Advanced Bash-Scripting Guide]], the result is the same: "OK" is *not* printed.

    #+begin_src shell :results output
    exec 0<&-
    echo OK
    #+end_src

Doing the following echoes OK for either direction of the redirection:

  -- /tmp/exec-OKlt.sh ---
  exec 0<&-
  echo OK
  ----------------------

  #+begin_example
  bash /tmp/exec-OKlt.sh
  #+end_example

  -- /tmp/exec-OKgt.sh ---
  exec 0>&-
  echo OK
  ----------------------

  #+begin_example
  bash /tmp/exec-OKgt.sh
  #+end_example

The INFILE passed to process-file looks like,

    #+begin_src emacs-lisp
    (process-file
       "bash"
       "/tmp/two-lines.sh"
       '(t "/tmp/babel-mS0Yyg/ob-error-AoxNqH")
       nil
       "-c" "bash")
    #+end_src

So, the call Emacs makes is probably more close to:

    #+begin_example
    cat /tmp/two-lines.sh | bash -c bash
    #+end_example

What this exactly does is unclear to me.  It appears to pass the contents of /tmp/two-lines.sh to a subshell process.  That is, it seems to behave like "bash /tmp/two-lines.sh" is run in a subprocess.

Running this in xfce4-terminal, I get what I expect:

    #+begin_example
    cat /tmp/two-lines.sh | bash -c bash
    #+end_example

Each line is echoed to file so nothing is written to the console.  However, both files are created with the expected text.  Both lines executed.

If I update two-lines to output to std,

    ------ /tmp/two-lines-tee.sh ------
    echo "first"  | tee /tmp/first.txt
    echo "second" | tee /tmp/second.txt
    -----------------------------------

I see "first" and "second" echoed to the console:

    ahab@pequod /tmp$ cat two-lines-tee.sh | bash -c bash
    first
    second

Running the following, neither give an output to the console:

#+begin_example
ahab@pequod /tmp$ cat exec-OKlt.sh | bash -c bash
ahab@pequod /tmp$ cat exec-OKgt.sh | bash -c bash
#+end_example

This is what we see in Org.  I'll be honest, though, I don't really know what to expect with exec 0>&- and exec 0<&-.  When I call them in the terminal, it kills the terminal.

The surprising bit is that running this in xfce4-terminal

    ----- /tmp/temp-file-with-source-block-code -----
    ssh localhost "echo foo>foo_file"
    echo "bar" | tee /tmp/bar.txt
    -------------------------------------------------

    #+begin_example
    cat /tmp/temp-file-with-source-block-code.sh | bash -c bash
    #+end_example

does *not* echo bar (and does not create /tmp/bar.txt) yet it creates foo_file.  I get prompted for my password and then the second line doesn't execute.  Nothing prints to the console and no bar.txt is created.

This is the behavior Alain reports happening in Org (that I am unable to reproduce).  That is, the *reported behavior is the expected behavior* (assuming my analysis is correct).  However, according to the behavior I see when I run the block (fails to create the remote file and echoes "bar"), Org does the "wrong thing".  I can't account for this.

Anyway, Ihor's main point stands: a prompt does not work with non-session shell blocks.  The following returns exit code 1 (which means fail):

    #+begin_src bash :results output
    read -p "What? "
    #+end_src

As far as I can tell, though, that's not what prevents "bar" from being returned.  As far as I can reproduce, calling

    #+begin_src bash :results output
    ssh localhost "echo foo>foo_file"
    echo "bar"
    #+end_src

*does* give "bar" for results even though it shouldn't.

** Comment 2.
The second claim has nothing to do with Org Babel.  I was able to confirm it and provide the steps to reproduce.  I think it would make sense to report it upstream and let them decide if it's expected behavior.  I'm still happy to do that, but I need to step away from the keyboard :)

Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Max Nikulin]

On 18/11/2023 05:07, Matt wrote:
> Consider calling
> 
>      bash -c bash /tmp/two-lines.sh

Matt, could you, please, describe what this command should do 
accordingly to your expectations?

 From my point of view it was a plain mistake in attempts to simulate 
the issue outside of Emacs. There is no point to concentrate on this 
command. I tried to explain that it is incorrect usage of "-c" shell 
option and what is the actual effect of this call, but I seems I failed.

Hints:
- Try this command in a regular shell prompt outside of Emacs
- Try "echo $$" before and after this command
- Try it with a script having a clear side effect and check whether it 
is produced, e.g. "date >>/tmp/touch-tmp-file.txt"

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 17/11/2023 17:17, Ihor Radchenko wrote:
> I was only able to reproduce your problem with ssh asking a password.
> We are discussing the reproduced case.

I see bash vs. dash difference with public key authorization, so no need 
for password prompts. I have not figured out how to construct an example 
without ssh since this command *may* read stdin, but does not do it in a 
same way as e.g. cat(1). Perhaps a small program performing single 
non-blocking read will allow it. The following behavior observed for a 
regular shell prompt, Emacs is not involved. Debian 12 bookworm.

cat ssh-script.sh
ssh -p 2222 127.0.0.1 'echo foo>/tmp/foo'
echo done

Read commands from a script file:

dash ssh-script.sh
done

bash ssh-script.sh
done

Read commands from stdin

dash <ssh-script.sh
done

bash <ssh-script.sh
# no output

I have not expected this difference.

dash reads a block from stdin (whole file in this case) and interprets 
commands.

BASH reads just the ssh command and executes it. SSH reads "echo done" 
from stdin, so when control is returned to bash, stdin is exhausted and 
no commands remain to execute by BASH. SSH can not "unread" part of 
input not consumed by the remote command despite it might be possible in 
the case of the regular file as stdin.

Actually bash reads the whole script file as well when called as it is 
shown above, but it calls lseek before executing ssh. To make difference 
more apparent (e.g. for strace), force creation of pipe(7) for which 
lseek is not supported

cat ssh-script.sh | strace -o /tmp/bash.strace bash

I am unsure if POSIX specifies exact behavior of shell when commands are 
read from stdin. I think, the suggested earlier "-n" ssh option (or 
</dev/null) should be used to make intentions clear: ssh should not read 
stdin. There is too much room for heuristics: interactive vs. 
non-interactive shell, a terminal vs. a regular file vs. a pipe as 
standard input. Be explicit to get reliable behavior.

ssh -n user@host 'command'

or

tar cvf - . | ssh user@host 'tar xvf -'

without "-n" when ssh needs stdin and it is explicitly specified.

I do not think it is an Org or an Emacs bug. It is rather POSIX vs. bash 
vs. dash issue.

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 25/10/2023 18:17, Alain.Cochard@unistra.fr wrote:
> By contrast, it works with this one:
> 
>     #+begin_src bash :results output
>     sshcochard@fruc.u-strasbg.fr  "echo foo>foo_file" ; echo "bar"
>     #+end_src

What about

    #+begin_src bash :results output
    ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
    echo more
    #+end_src

?

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Matt]

 ---- On Sat, 18 Nov 2023 04:11:03 +0100  Max Nikulin  wrote --- 

 > >      bash -c bash /tmp/two-lines.sh
 > 
 >  From my point of view it was a plain mistake in attempts to simulate 
 > the issue outside of Emacs. There is no point to concentrate on this 
 > command. I tried to explain that it is incorrect usage of "-c" shell 
 > option and what is the actual effect of this call, but I seems I failed.

As an ob-shell user, my expectation is that execution within Org produces the same behavior as outside of Emacs.  This is why I've focused on the command.  It acts as a guide for what is "correct."  Maybe this is misguided for a reason I don't yet see?

I concluded that the command is not "what Emacs does" and therefore isn't a valid source of "truth".  I agree it's not worth concentrating on it further.

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Hi Matt,

Thanks this summary and for working on this!

Just a few comments/corrections about some specific points, hoping it
might help.

Matt <matt@excalamus.com> writes:

>  ---- On Fri, 17 Nov 2023 10:20:28 +0100  Ihor Radchenko  wrote --- 
>
>  > This has nothing to do with Emacs comint and this is also not a bug in
>  > Emacs 
>
> Ihor, there were two claims made in the original report.  I was referring to Claim 2.  That deals with M-x shell and therefore comint-mode.
>
> Regarding Claim 1:
>
> - Can anyone verify Claim 1?

I do: the file is created and the command "echo bar" is NOT executed.

Here is my code block and its results:

    #+begin_src bash :results output
      ssh phone "echo foo>foo_file"
      echo "bar"
    #+end_src

    #+RESULTS:

No results (the echo command is NOT executed).

The file "foo_file" is created on the remote; its content is "foo".

    #+begin_src bash :results output
      date
      ssh -n phone "ls -alh foo_file"
      ssh -n phone "cat foo_file"
    #+end_src

    #+RESULTS:
    : Sat Nov 18 08:33:59 CET 2023
    : -rw------- 1 u0_a256 u0_a256 4 Nov 18 08:26 foo_file
    : foo



> - What versions are people using?
>   + M-x org-version
>   + M-x emacs-version

    #+begin_src elisp
      (list emacs-version org-version)
    #+end_src

    #+RESULTS:
    | 30.0.50 | 9.7-pre |

    GNU/Linux gentoo
    
> ...

> * Comments about the claims:

> ** Comment 1.
> ...
> I am unable to reproduce the reported behavior (of
> "bar" not returning).  Instead, I get an ssh-askpass permission denied
> error, foo_file is not created, and "bar" is given as the result.  I
> do not see anywhere in the thread that the original claim was
> reproduced.

It seems your SSH failed to connect.  In that case, I cannot swallow the
second command; thus the command "echo bar" is executed.

I can reproduce what you see on my side if I force the connection to fail:

    #+begin_src bash :results output
      ssh WRONG_REMOTE "echo foo>foo_file"
      echo "bar"
    #+end_src

    #+RESULTS:
    : bar


>
> The thread preceded something like follows.
>
> Leo Butler suggested two work arounds:
>
> - add the -f to the ssh command


> - add a semi-colon and line continuation to the first line.
>
> Russell Adams suggested another work around:
>
> - add -n to the ssh command

That's the one I use; the option -n is enough for me ('-n' = Redirects
stdin from /dev/null). The option '-f' means SSH will go to background;
I'm not sure I want that.

> ...

> ... 
> He then proposes an experiment to close stdin.  To do this, he calls
>
>     #+begin_src shell :results output
>     exec 0>&-
>     echo OK
>     #+end_src
>
> He claims that "exec 0<&-" closes stdin.  I believe there is a typo.
> ...

You're right. Good catch, thanks!

Although it seems to work either way on my side.

    #+begin_src shell :results output
      exec 0<&-
      echo OK
    #+end_src

    #+RESULTS:

    #+begin_src shell :results output
      exec 0>&-
      echo OK
    #+end_src

    #+RESULTS:
    

> What Bruno writes corresponds to "closing output file descriptor 0".  I honestly don't know what the difference is between an "output file descriptor" and an "input file descriptor".  I had no luck finding this information in man bash or info bash.
>

My point was: the commands are read the standard input, thus, any
command that modifies that standard input will modify what gets
executed.


> ...
> This is what we see in Org.  I'll be honest, though, I don't
> really know what to expect with exec 0>&- and exec 0<&-.  When I call
> them in the terminal, it kills the terminal.

Let's forget about 'exec 0<&-' (closing the standard input/outputs):
this is bringing other corner cases.  But, yes, I would expect a
terminal to close itself automatically if its input is closed.

> ...
> As far as I can tell, though, that's not what prevents "bar" from being returned.  As far as I can reproduce, calling
>
>     #+begin_src bash :results output
>     ssh localhost "echo foo>foo_file"
>     echo "bar"
>     #+end_src
>
> *does* give "bar" for results even though it shouldn't.

Does it echo bar when the SSH connection succeeds too ?


Thanks again for working on this.


Bruno

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Matt <matt@excalamus.com> writes:

>  ---- On Sat, 18 Nov 2023 04:11:03 +0100  Max Nikulin  wrote --- 
>
>  > >      bash -c bash /tmp/two-lines.sh
>  > 
>  >  From my point of view it was a plain mistake in attempts to simulate 
>  > the issue outside of Emacs. There is no point to concentrate on this 
>  > command. I tried to explain that it is incorrect usage of "-c" shell 
>  > option and what is the actual effect of this call, but I seems I failed.
>
> As an ob-shell user, my expectation is that execution within Org produces the same behavior as outside of Emacs.  This is why I've focused on the command.  It acts as a guide for what is "correct."  Maybe this is misguided for a reason I don't yet see?

IIUC, what Max is saying is that you should not concentrate on
*that specific command* because that command doesn't do what you think
it does.


To reproduce, I'm personally still using:

    cat /tmp/test.sh | bash
    
which is, IIUC, what:

    (process-file "bash" "/tmp/test.sh")
    
is doing, that is:

   The program’s input comes from file INFILE


Bruno

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Max Nikulin <manikulin@gmail.com> writes:

> On 25/10/2023 18:17, Alain.Cochard@unistra.fr wrote:
>> By contrast, it works with this one:
>> 
>>     #+begin_src bash :results output
>>     sshcochard@fruc.u-strasbg.fr  "echo foo>foo_file" ; echo "bar"
>>     #+end_src
>
> What about
>
>     #+begin_src bash :results output
>     ssh cochard@fruc.u-strasbg.fr "echo foo>foo_file" ; echo "bar"
>     echo more
>     #+end_src
>
> ?

For me:

     #+begin_src bash :results output
     ssh phone "echo foo>foo_file" ; echo "bar"
     echo more
     #+end_src

     #+RESULTS:
     : bar

And, telling SSH to not swallow the remaining commands (option '-n'):

     #+begin_src bash :results output
     ssh -n phone "echo foo>foo_file" ; echo "bar"
     echo more
     #+end_src
    
    #+RESULTS:
    : bar
    : more
    

Bruno

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Matt]

 ---- On Sat, 18 Nov 2023 09:29:56 +0100  Bruno Barbier 

 > IIUC, what Max is saying is that you should not concentrate on
 > *that specific command* because that command doesn't do what you think
 > it does.

Cool, it sounds like we're agreed (albeit for different reasons).  

 > To reproduce, I'm personally still using:
 > 
 >     cat /tmp/test.sh | bash
 >     
 > which is, IIUC, what:
 > 
 >     (process-file "bash" "/tmp/test.sh")

Yes, agreed.  I think that's more like what's happening.  

What about the ("-c" "bash") passed into process-file?  

The whole call looks like this:

    (process-file
       "bash"
       "/tmp/two-lines.sh"
       '(t "/tmp/babel-mS0Yyg/ob-error-AoxNqH")
       nil
       "-c" "bash")

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Matt <matt@excalamus.com> writes:

>  ---- On Sat, 18 Nov 2023 09:29:56 +0100  Bruno Barbier 
>
>  > IIUC, what Max is saying is that you should not concentrate on
>  > *that specific command* because that command doesn't do what you think
>  > it does.
>
> Cool, it sounds like we're agreed (albeit for different reasons).

Great :-)


>  > To reproduce, I'm personally still using:
>  > 
>  >     cat /tmp/test.sh | bash
>  >     
>  > which is, IIUC, what:
>  > 
>  >     (process-file "bash" "/tmp/test.sh")
>
> Yes, agreed.  I think that's more like what's happening.  
>
> What about the ("-c" "bash") passed into process-file?  

Useless indirection when the command is "bash" is the same as
'shell-file-name', like in our case ?

Maybe ...

But, you're right.  To be safe, from now on, I'll use:

    cat /tmp/test.sh | bash -c bash


Thanks.

Bruno

Re: bash source code block: problem after ssh commands

[Matt]

To clarify a previous typo I made.  Everything I've written was also done using:

1. emacs -q
2. C-x b "*scratch*"
3. M-x org-mode
4. Execute 

#+begin_src emacs-lisp :results none
(org-babel-do-load-languages
   'org-babel-load-languages
   '((shell . t)))
#+end_src

This corresponds to Org mode version 9.6.6 (release_9.6.6 @ /gnu/store/xjrhyapm3zwgpmq5baz6m9kavz287jjj-emacs-29.1/share/emacs/29.1/lisp/org/)

I had previously, and incorrectly, given my version used with my full init (which uses a different Org version).

 ---- On Sat, 18 Nov 2023 09:19:13 +0100  Bruno Barbier 
 
 > It seems your SSH failed to connect.  In that case, I cannot swallow the
 > second command; thus the command "echo bar" is executed.
 > 
 > I can reproduce what you see on my side if I force the connection to fail:
 > 
 >     #+begin_src bash :results output
 >       ssh WRONG_REMOTE "echo foo>foo_file"
 >       echo "bar"
 >     #+end_src
 > 
 >     #+RESULTS:
 >     : bar
 
Thank you!  That makes sense.   I forgot that without set -e Bash keeps evaluating when errors happen.

Yes, once it connects, I get the same behavior as reported.

Here is how I connected without a password prompt:

1. ssh-keygen with id_babel, <enter>, <enter>
2. cat ~/.ssh/id_babel.pub >> ~/.ssh/authorized_keys
3. chmod og-wx ~/.ssh/authorized_keys
4. ssh-add .ssh/id_babel

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Matt]

 ---- On Sat, 18 Nov 2023 09:54:46 +0100  Bruno Barbier 
 
 > But, you're right.  To be safe, from now on, I'll use:
 > 
 >     cat /tmp/test.sh | bash -c bash

It's still not clear to me if this is "what Emacs does".  However, that's the best I could come up with.  

Evaluating the following 

#+name: /tmp/test.sh
#+begin_src bash :results output
ssh localhost "echo foo>foo_file"
echo "bar" | tee /tmp/bar.txt
#+end_src

does exactly what

    cat /tmp/test.sh | bash -c bash

does.  Both create a file "foo_file" containing "foo" on the remote machine and neither execute the second line.

So, I would say that what happens in Org is the "expected" behavior.

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Matt <matt@excalamus.com> writes:

>  ---- On Sat, 18 Nov 2023 09:54:46 +0100  Bruno Barbier 
>  
> It's still not clear to me if this is "what Emacs does".  However, that's the best I could come up with.  
>
> Evaluating the following 
>
> #+name: /tmp/test.sh
> #+begin_src bash :results output
> ssh localhost "echo foo>foo_file"
> echo "bar" | tee /tmp/bar.txt
> #+end_src
>
> does exactly what
>
>     cat /tmp/test.sh | bash -c bash
>
> does.  Both create a file "foo_file" containing "foo" on the remote machine and neither execute the second line.
>
> So, I would say that what happens in Org is the "expected" behavior.

Agreed.

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Bruno Barbier <brubar.cs@gmail.com> writes:

>> WRT M-x shell, feel free to submit a bug report. I mostly pointed that
>> the problem with M-x shell is not the problem you originally ran to. It
>> is a different problem (also, we ran into it in the past).
>
> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
> multiple lines, getting the same results as when I type them manually,
> or copy them line by line. My xterm doesn't seem to allow me to do that.

The behavior of M-x shell can indeed be made use of.
However, this particular difference with xterm, AFAIU, is not
documented - unaware users may be surprised.
The situation is worse with Org shell blocks - users naturally expect
script-like behavior (even for :session), but run into edge cases like
this and get confused.

We should either document the caveats, or, preferably, make the behavior
more consistent with expectations. At least, by default.

That's why I think that filing a bug report makes sense from Org mode
project point of view.

> It looks like it is a known SSH "feature" (see
> https://unix.stackexchange.com/a/688024):
>
>     #+begin_src bash :results output
>     seq 1000000 | (ssh phone sleep 1; wc -l)
>     #+end_src
>
>     #+RESULTS:
>     : 675173
> ...

> IMHO, what ob-shell is doing today seems a valid way of evaluating
> source blocks (and it seems to have been like that for a long time).  It
> should probably be documented somewhere, so that users know how to write
> their source blocks, or switch to another way, like adding a :cmdline
> parameter as mentionned in this thread.

And, as Max demonstrated, this ssh feature only works with bash, but not
necessarily other shells. Moreover, "-n" option that disables the above
may depend on ssh configuration (see StdinNull in man 5 ssh_config). So,
I believe that it only leads to confusion, even if we try to document
it.

__By default__, Org should produce more expected behavior - what users
would get from running a script file rather than from redirecting stdin.
We can optionally leave the stdin redirection as an option to be used by
the users who understand the peculiarities.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> ...
> I have not expected this difference.
>
> dash reads a block from stdin (whole file in this case) and interprets 
> commands.
>
> BASH reads just the ssh command and executes it. SSH reads "echo done" 
> from stdin, so when control is returned to bash, stdin is exhausted and 
> no commands remain to execute by BASH. SSH can not "unread" part of 
> input not consumed by the remote command despite it might be possible in 
> the case of the regular file as stdin.
>
> Actually bash reads the whole script file as well when called as it is 
> shown above, but it calls lseek before executing ssh. To make difference 
> more apparent (e.g. for strace), force creation of pipe(7) for which 
> lseek is not supported
> ...
> I do not think it is an Org or an Emacs bug. It is rather POSIX vs. bash 
> vs. dash issue.

I still see it as a bug - what Org mode does to run the shell blocks is
not what users expect. _By default_, we _should_ produce more expected
behavior.

The observed inconsistency between different shells just indicates that
our approach with `process-file' should not be used as it leads to
potentially confusing results (they are not confusing only to people who
dig deeply into ssh, bash/dash/etc, and Org mode internals).

I cannot find any clear motivation behind using `process-file' + INFILE
in the git logs. I assume that it was used simply because it is easier
compared to trying to create and run temporary script file on remote
hosts.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Forget about "bash -c bash file.sh" (Re: bash source code block: problem after ssh commands)

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

> Evaluating the following 
>
> #+name: /tmp/test.sh
> #+begin_src bash :results output
> ssh localhost "echo foo>foo_file"
> echo "bar" | tee /tmp/bar.txt
> #+end_src
>
> does exactly what
>
>     cat /tmp/test.sh | bash -c bash
>
> does.  Both create a file "foo_file" containing "foo" on the remote machine and neither execute the second line.
>
> So, I would say that what happens in Org is the "expected" behavior.

Yup, it is expected. But only in a sense that "Emacs does what we asked
for when calling `process-file'". From the user point of view, it is not
expected at all. And, what is worse, may depend on (1) shell used; (2)
ssh config (see StdinNull in man 5 ssh_config).

I believe that we do need to change how we execute shell blocks _by
default_ - to something more predictable, like creating and running a
temporary script file.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Matt]

 ---- On Fri, 17 Nov 2023 23:07:57 +0100  Matt  wrote --- 
 
 > The second claim has nothing to do with Org Babel.  I was able to confirm it and provide the steps to reproduce.  I think it would make sense to report it upstream and let them decide if it's expected behavior.  I'm still happy to do that, but I need to step away from the keyboard :)

Submitted. 

https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-11/msg00976.html

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 18/11/2023 17:43, Ihor Radchenko wrote:
> 
> I still see it as a bug - what Org mode does to run the shell blocks is
> not what users expect. _By default_, we _should_ produce more expected
> behavior.

A shell without unexpected behavior is neither POSIX nor a descendant 
shell like BASH
https://mywiki.wooledge.org/BashPitfalls

Behavior if interactive and non-interactive shells is not the same anyway.

> I cannot find any clear motivation behind using `process-file' + INFILE
> in the git logs. I assume that it was used simply because it is easier
> compared to trying to create and run temporary script file on remote
> hosts.

 From my point of view it is rather natural to use `process-file' since 
this function is available out of the box. I suspect some subtle issues 
may appear even if script file is forced.

On the other hand ob-shell already has a (bit buggy) implementation 
relying on script files. It is used when :stdin or :cmdline is specified.

Non-emacs shell (Re: bash source code block: problem after ssh commands)

[Max Nikulin]

On 17/11/2023 22:47, Bruno Barbier wrote:
> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
> multiple lines, getting the same results as when I type them manually,
> or copy them line by line. My xterm doesn't seem to allow me to do that.

I am unsure what do you expect from xterm, but perhaps it is not 
responsibility of a terminal application.

Multiple lines can be copied to regular BASH prompt (bracketed paste is 
enabled by default nowadays), however it may be inconvenient to edit.

You may use edit-and-execute-command (C-x C-e) (BASH, not Emacs key 
binding) to start an editor and to paste multiple commands there.

See also "fc" BASH built-in for editing and re-executing last commands.

Re: Non-emacs shell (Re: bash source code block: problem after ssh commands)

[Bruno Barbier]

Hi Max,

Max Nikulin <manikulin@gmail.com> writes:

> On 17/11/2023 22:47, Bruno Barbier wrote:
>> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
>> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
>> multiple lines, getting the same results as when I type them manually,
>> or copy them line by line. My xterm doesn't seem to allow me to do that.
>
> I am unsure what do you expect from xterm, but perhaps it is not 
> responsibility of a terminal application.

It has been said in this thread that 'M-x shell' should be fixed to
match the behavior that we see in a plain terminal, when we copy
multiple lines.  I just wanted to point out that I do prefer the way
'M-x shell' handles the copy of multiple lines.


> Multiple lines can be copied to regular BASH prompt (bracketed paste is 
> enabled by default nowadays), however it may be inconvenient to edit.
> You may use edit-and-execute-command (C-x C-e) (BASH, not Emacs key 
> binding) to start an editor and to paste multiple commands there.
> See also "fc" BASH built-in for editing and re-executing last commands.

Thanks Max! I didn't know that.  I should definitely start using this
when I'm stuck in a console, to safely copy/edit my commands using
Emacs.

Thanks,


Bruno

Re: bash source code block: problem after ssh commands

[Bruno Barbier]

Ihor Radchenko <yantar92@posteo.net> writes:

> Bruno Barbier <brubar.cs@gmail.com> writes:
>
>> FWIW, M-x shell differs from what a plain terminal is doing (xterm, in
>> my case), but, I do prefer 'M-x shell' behavior: it allows me to copy
>> multiple lines, getting the same results as when I type them manually,
>> or copy them line by line. My xterm doesn't seem to allow me to do that.
>
> The behavior of M-x shell can indeed be made use of.
> However, this particular difference with xterm, AFAIU, is not
> documented - unaware users may be surprised.
> The situation is worse with Org shell blocks - users naturally expect
> script-like behavior (even for :session), but run into edge cases like
> this and get confused.
>
> We should either document the caveats, or, preferably, make the behavior
> more consistent with expectations. At least, by default.

> That's why I think that filing a bug report makes sense from Org mode
> project point of view.

Thanks for the explanation.  I see you got my point.  We'll see what
Emacs maintainers will say about the current behavior of M-x shell;
filling the bug report was definitely a good idea anyway.



Ihor Radchenko <yantar92@posteo.net> writes:
> __By default__, Org should produce more expected behavior - what users
> would get from running a script file rather than from redirecting stdin.
> We can optionally leave the stdin redirection as an option to be used by
> the users who understand the peculiarities.

I agree that it would be simpler to switch to the script-like behavior
by default on org side.  About the interactive-like behavior, that
would be nice to keep it, if some people rely on it in their existing
documents (I personally don't).

Bruno

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 22/11/2023 02:01, Bruno Barbier wrote:
> Ihor Radchenko writes:
>> __By default__, Org should produce more expected behavior - what users
>> would get from running a script file rather than from redirecting stdin.
>> We can optionally leave the stdin redirection as an option to be used by
>> the users who understand the peculiarities.
> 
> I agree that it would be simpler to switch to the script-like behavior
> by default on org side.  About the interactive-like behavior, that
> would be nice to keep it, if some people rely on it in their existing
> documents (I personally don't).

I do not like recommendations

     curl https://... | sudo bash

but I regularly see them. So some users might expect behavior like 
namely stdin, not like interactive prompt commands.

Perhaps even sessions may be switched to creation of a temporary file 
and executing "source tmpfile.sh". However it may be shell-specific (posh?).

Likely it is possible to implement a header argument to explicitly 
control whether `process-file' should be used or the block should be 
executed as a temporary script file. Currently it is implicitly 
determined from other header arguments.

P.S.

https://mywiki.wooledge.org/BashFAQ/089
"I'm reading a file line by line and running ssh or ffmpeg, only the 
first line gets processed!"

A similar case, but only data is read from stdin, script is a regular 
file, no difference bash vs. dash.

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 18/11/2023 15:04, Max Nikulin wrote:
> On 17/11/2023 17:17, Ihor Radchenko wrote:
> 
> I see bash vs. dash difference with public key authorization, so no need 
> for password prompts. I have not figured out how to construct an example 
> without ssh since this command *may* read stdin, but does not do it in a 
> same way as e.g. cat(1).

I expect the following couple of lines simulates ssh behavior fairly 
well (besides maybe macOS with its old GPLv2 bash).

bash -c 'read -t 0.5 -r line; printf "read: %s\n" "$line"'
printf 'printf\n'

> cat ssh-script.sh
> ssh -p 2222 127.0.0.1 'echo foo>/tmp/foo'
> echo done
[...]
> dash <ssh-script.sh
> done
> 
> bash <ssh-script.sh
> # no output
> 
> I have not expected this difference.
[...]
> I am unsure if POSIX specifies exact behavior of shell when commands are 
> read from stdin.

It is a bug in dash that should be fixed in next version. Behavior of 
bash is correct despite some users may expect "done" printed.

<https://lore.kernel.org/dash/20221213221732.6mvv22u7ktdozrbx@tarta.nabijaczleweli.xyz/t/>
наб <nabijaczleweli@nabijaczleweli.xyz> to dash@vger.kernel.org
[PATCH] input: preadfd: read standard input byte-wise
Tue, 13 Dec 2022 23:17:32 +0100

<https://bugs.debian.org/862907>
"dash: Incorrectly slurps script from stdin (POSIX compliance issue)"

And curiously ssh command was involved as well
<https://michael-prokop.at/blog/2017/05/18/debugging-a-mystery-ssh-causing-strange-exit-codes/>
"Debugging a mystery: ssh causing strange exit codes?"

A citation from POSIX (XCU):

<https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sh.html#tag_20_117_06>
STDIN in /sh - shell, the standard command language interpreter/
> When the shell is using standard input and it invokes a command that
> also uses standard input, the shell shall ensure that the standard
> input file pointer points directly after the command it has read when
> the command begins execution. It shall not read ahead in such a manner
> that any characters intended to be read by the invoked command are
> consumed by the shell (whether interpreted by the shell or not) or
> that characters that are not read by the invoked command are not seen
> by the shell. When the command expecting to read standard input is
> started asynchronously by an interactive shell, it is unspecified
> whether characters are read by the command or interpreted by the
> shell.

I do not think any script should rely on this behavior, it is better to 
explicitly use "here documents"

cat <<EOF
$PATH
EOF

or to prevent various kinds of expansion

cat <<"EOF"
$PATH
EOF

Just a reminder: reading from stdin in shell scripts may interfere with 
various commands

<https://mywiki.wooledge.org/BashFAQ/089>
"I'm reading a file line by line and running ssh or ffmpeg, only the 
first line gets processed!"

So either "ssh -n" or "ssh </dev/null" should be used to avoid the pitfall.

I agree with Ihor that ob-shell should not feed scripts to shell stdin 
(maybe besides the case when it is explicitly requested by the user 
through some header arguments).

It seems, shell sessions
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=67259>
is a different case unrelated to script files.

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 18/11/2023 22:51, Matt wrote:
>   ---- On Fri, 17 Nov 2023 23:07:57 +0100  Matt  wrote ---
>> The second claim has nothing to do with Org Babel.  I was able to
>> confirm it and provide the steps to reproduce.  I think it would make
>> sense to report it upstream and let them decide if it's expected
>> behavior.  I'm still happy to do that, but I need to step away from the
>> keyboard :)

> https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-11/msg00976.html
https://debbugs.gnu.org/67259

Notice that the report is for *interactive* sessions. Org babel sessions 
are not so interactive. They are to maintain state, not to interact with 
users directly. Consider

     fakessh() {
         bash -c 'read -t 10 -r; printf "fakessh read: %s\n" "$REPLY"';
     }

     fakessh
     echo next

If you type them in terminal then result depends on delay between 
"fakessh" and "echo next" commands. SSH is an example of command that 
*may* read stdin while its command is running. To avoid ambiguity you 
have to do either

     fakessh </dev/null
     echo next

or

     fakessh <<"EOF"
     echo next
     EOF

"Reproducing" the issue in xterm-like applications or various emacs 
shells makes things even more complicated. Result depends on TERM value 
and if bracketed paste is enabled. With active bracketed paste shell 
knows if you paste single line or a several lines into *shell prompt*, 
so it does not feed second and following lines to first command stdin. 
In the case of

#+begin_src bash :session *ob-shell*
   bash -c 'read -t 10 -r; printf "read: %s\n" "$REPLY"'
   echo next
#+end_src

comint can not guess if all lines are commands or some of them should be 
considered as input to the previous command. Timing or bracketed paste 
are irrelevant for ob-shell sessions.

Possible voluntary decision is to consider every session source block 
similar to "C-x C-e" in bash prompt, so nothing should become implicit 
stdin of commands. Content of a source block should be saved into a 
temporary file then "source /path/to/block_content" command should be 
send to the session buffer. Likely command depends on shell.

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> https://debbugs.gnu.org/67259
>
> Notice that the report is for *interactive* sessions. Org babel sessions 
> are not so interactive. They are to maintain state, not to interact with 
> users directly...

You are not right. We have legitimate cases when people can switch to
session buffer, enter commands there, switch back to Org buffers,
send code blocks to the same session, and so on.

> ...
> Possible voluntary decision is to consider every session source block 
> similar to "C-x C-e" in bash prompt, so nothing should become implicit 
> stdin of commands. Content of a source block should be saved into a 
> temporary file then "source /path/to/block_content" command should be 
> send to the session buffer. Likely command depends on shell.

This may work, but we should at least echo the original commands into
session buffer, because users may want to examine them.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 30/06/2024 18:06, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> https://debbugs.gnu.org/67259
>>
>> Notice that the report is for *interactive* sessions. Org babel sessions
>> are not so interactive. They are to maintain state, not to interact with
>> users directly...
> 
> You are not right. We have legitimate cases when people can switch to
> session buffer, enter commands there, switch back to Org buffers,
> send code blocks to the same session, and so on.

Timings are still important part of interactive sessions and ob-shell 
has no chance to simulate them having just block of commands.

By the way, dash does not support bracketed paste since it is a minimal 
shell mainly for interpreting scripts. So result of pasting several 
commands at once is different from bash. I do not think it is related to 
dash bugs.

>> Possible voluntary decision is to consider every session source block
>> similar to "C-x C-e" in bash prompt, so nothing should become implicit
>> stdin of commands. Content of a source block should be saved into a
>> temporary file then "source /path/to/block_content" command should be
>> send to the session buffer. Likely command depends on shell.
> 
> This may work, but we should at least echo the original commands into
> session buffer, because users may want to examine them.

I have no idea which way bash or readline function 
edit-and-execute-command is implemented, but commands are echoed when 
editor is closed. Anyway it may greatly vary across shells.

I am unaware if it is possible to detect if bracketed paste is 
implemented and to send input as bracketed paste. Perhaps some escape 
sequences are involved. ob-shell might offer variants for sessions: dumb 
stdin, source a temporary file, or bracketed paste. Anything besides 
stdin (with its confusing differences from interactive sessions) is 
almost certainly requires shell-specific code. And `shell' is likely not 
suitable with its TERM=dumb.

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> ... ob-shell might offer variants for sessions: dumb 
> stdin, source a temporary file, or bracketed paste. Anything besides 
> stdin (with its confusing differences from interactive sessions) is 
> almost certainly requires shell-specific code. And `shell' is likely not 
> suitable with its TERM=dumb.

We use comint buffers for sessions. Nothing else.
It is non-session blocks that have variants of how they are evaluated.

And, since we are using comint, we are limited by what comint can do.
Under the hood, comint uses stdin approach to send user
commands. Handling what is a command, what is not a command, and hot to
edit it is entirely under Emacs control.

I do not see a great benefit of re-implementing session support via
actual terminal. (I think we agree here).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 30/06/2024 22:28, Ihor Radchenko wrote:
> I do not see a great benefit of re-implementing session support via
> actual terminal. (I think we agree here).

`shell' and so comint allocates "actual" pty. I expect that bracketed 
paste is just some escape sequences delimiting pasted text (fcntl calls 
are less probable). The obstacle here is TERM=dumb.

I would be OK with just documenting that ob-shell blocks differ from 
interactive sessions and some care is required. It is you who want to 
provide experience close to interactive sessions.

There is a chance that some shell built-in command allows to get 
behavior close to edit-and-execute-command (C-x C-e) that, unlike 
"source", echoes input.

Currently I have no over ideas how to avoid interpreting some lines as 
input to earlier commands (and at the same time send some lines to stdin 
of commands as it would be in interactive sessions).

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 30/06/2024 22:28, Ihor Radchenko wrote:
>> I do not see a great benefit of re-implementing session support via
>> actual terminal. (I think we agree here).
>
> `shell' and so comint allocates "actual" pty.

Are you sure?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 30/06/2024 23:42, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> On 30/06/2024 22:28, Ihor Radchenko wrote:
>>> I do not see a great benefit of re-implementing session support via
>>> actual terminal. (I think we agree here).
>>
>> `shell' and so comint allocates "actual" pty.
> 
> Are you sure?

ps wf
     PID TTY      STAT   TIME COMMAND
    2269 pts/0    Ss     0:01 /bin/bash
  123454 pts/0    Sl     0:16  \_ emacs -Q
  125104 pts/4    Ss+    0:00      \_ /bin/bash --noediting -i
  129807 pts/5    Ss+    0:00      \_ /bin/bash
  132172 pts/6    Ss+    0:00      \_ /usr/bin/bash --noediting -i

pts/0 is created by konsole, others are `shell' and `term' sessions 
inside Emacs.

My experience is that it is not easy to start a process without a tty in 
Emacs. Eli will object.

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>>>> I do not see a great benefit of re-implementing session support via
>>>> actual terminal. (I think we agree here).
>>>
>>> `shell' and so comint allocates "actual" pty.
>> 
>> Are you sure?
> ...
>   123454 pts/0    Sl     0:16  \_ emacs -Q
>   125104 pts/4    Ss+    0:00      \_ /bin/bash --noediting -i
>   129807 pts/5    Ss+    0:00      \_ /bin/bash
>   132172 pts/6    Ss+    0:00      \_ /usr/bin/bash --noediting -i
>
> pts/0 is created by konsole, others are `shell' and `term' sessions 
> inside Emacs.
>
> My experience is that it is not easy to start a process without a tty in 
> Emacs. Eli will object.

I guess not on Windows (bug#71081). But otherwise I agree.

Although, I am not sure what we need to document and what we just need
to rewrite. In theory, we may work around all the pty-related problems
by simply running a script file (that possibly also echoes its commands
to keep them recorded for user examination).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 01/07/2024 16:51, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> My experience is that it is not easy to start a process without a tty in
>> Emacs. Eli will object.
> 
> I guess not on Windows (bug#71081). But otherwise I agree.

It was for starting mailcap viewer and Windows was not an issue.

> Although, I am not sure what we need to document and what we just need
> to rewrite.

I have posted it to this thread and to https://debbugs.gnu.org/67259

Some commands try to read standard input if it is available.  In 
interactive sessions it may be hidden due to time interval between typed 
commands.  Be careful when they are used in Org source blocks.
[[https://mywiki.wooledge.org/BashFAQ/089][BASH FAQ #89]]
warns concerning =ssh= and =ffmpeg=.  Either explicitly specify
=</dev/null= as input or use document here syntax.

#+begin_example
ssh example.org 'sed -i -e s/foo/bar/ file.txt' </dev/null
ssh example.org 'tee >>file.log' <<"EOF"
Added by org-babel
EOF
#+end_example

(Quotes around "EOF" suppress variable expansion in the text.)  Tools 
may have dedicated options, for example =ssh -n= is a more concise way 
to avoid the pitfall.

> In theory, we may work around all the pty-related problems
> by simply running a script file (that possibly also echoes its commands
> to keep them recorded for user examination).

Ideally some kind of bracketed paste should be added to `shell', 
otherwise it is insecure. However it is not an option for dash. I still 
have slight hope that implementation of =C-x C-e= might give another 
idea. Some subtle issues may arise with "source", but perhaps they are 
more rare than naive ssh usage. If you know how to add text to shell 
buffer without sending it to the process then you may try. I would leave 
an option to switch back to "direct" input.

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> I guess not on Windows (bug#71081). But otherwise I agree.
>
> It was for starting mailcap viewer and Windows was not an issue.

bug#71081 is about shell-command-on-region, not about mailcap.
(and pty may actually not matter there, because Windows' cmd.exe is not
even POSIX-compient...)

> Some commands try to read standard input if it is available.  In 
> interactive sessions it may be hidden due to time interval between typed 
> commands.  Be careful when they are used in Org source blocks.
> [[https://mywiki.wooledge.org/BashFAQ/089][BASH FAQ #89]]
> warns concerning =ssh= and =ffmpeg=.  Either explicitly specify
> =</dev/null= as input or use document here syntax.
>
> #+begin_example
> ssh example.org 'sed -i -e s/foo/bar/ file.txt' </dev/null
> ssh example.org 'tee >>file.log' <<"EOF"
> Added by org-babel
> EOF
> #+end_example
>
> (Quotes around "EOF" suppress variable expansion in the text.)  Tools 
> may have dedicated options, for example =ssh -n= is a more concise way 
> to avoid the pitfall.

May you submit a patch for the WORG docs that documents this caveat?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bash source code block: problem after ssh commands

[Max Nikulin]

On 01/07/2024 23:01, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
> bug#71081 is about shell-command-on-region, not about mailcap.
> (and pty may actually not matter there, because Windows' cmd.exe is not
> even POSIX-compient...)

Of course, there are no pty's on Windows and cmd.exe is a special case. 
I was trying to say that it may be not so easy to start a process 
without tty allocation on Linux.

>> Either explicitly specify
>> =</dev/null= as input or use document here syntax.
[...]
> May you submit a patch for the WORG docs that documents this caveat?

9733cc41 2024-07-03 17:57:15 +0700 ob-doc-shell.org: Warn concerning 
optional stdin

Feel free to rewrite added text or to move it from introduction to some 
other section. HTML page should be updated when somebody with payed 
account will push more changes.

Re: bash source code block: problem after ssh commands

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> May you submit a patch for the WORG docs that documents this caveat?
>
> 9733cc41 2024-07-03 17:57:15 +0700 ob-doc-shell.org: Warn concerning 
> optional stdin

Thanks!

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>