babel output seems to drop anything before % (in session)

babel output seems to drop anything before % (in session)

[Daniele Pizzolli]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]


Hello,

Try to execute a few times the following and see the output corrupted in
the line:

#+BEGIN_EXAMPLE
1 packets transmitted, 1 received, 0% packet loss, time 0ms
#+END_EXAMPLE

#+PROPERTY: header-args:shell
#+PROPERTY: header-args:shell+ :results output verbatim :wrap src text :session test

#+NAME: ping
#+BEGIN_SRC shell
ping -c 1 127.0.0.1
#+END_SRC

I tried to write a test (now with the :expected-result :failed), that
hit the problem both in main and master (applies fine to both).

I was not able to fix it, but hopefully the tests are still useful for
somebody more knowledgeable!

Thanks for you suggestion/fix!

Best,
Daniele


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-disabled-test-for-in-babel-shell-output.patch --]
[-- Type: text/x-diff, Size: 1944 bytes --]

From 897ae50cf055d459d741c90f8de94dd56a8eb3d9 Mon Sep 17 00:00:00 2001
From: Daniele Pizzolli <dan+git@toel.it>
Date: Wed, 5 May 2021 17:00:28 +0200
Subject: [PATCH 1/1] Add disabled test for % in babel shell output

---
 testing/lisp/test-ob-shell.el | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index 0aadb355d..a42cc3d77 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -106,6 +106,35 @@ ob-comint.el, which was not previously tested."
 	   "#+BEGIN_SRC sh :results output :var l='(1 2)\necho ${l}\n#+END_SRC"
 	   (org-trim (org-babel-execute-src-block))))))
 
+(ert-deftest ob-shell/percent-simple ()
+  "Test percent in output."
+  (should
+   (equal "one 0% two\ntree 0% four"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim\necho one 0% two\necho tree 0% four\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block)))))
+  (should
+   (equal "five 0% two\ntree 0% four"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim\necho five 0% two\necho tree 0% four\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block))))))
+
+(ert-deftest ob-shell/percent-session ()
+  "Test percent in output for session.
+
+At the second iteration the string before % is dropped."
+  :expected-result :failed
+  (should
+   (equal "one 0% two\ntree 0% four"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim :session sess\necho one 0% two\necho tree 0% four\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block)))))
+  (should
+   (equal "five 0% six\nseven 0% eight"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim :session sess\necho five 0% six\necho seven 0% eight\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block))))))
+
 (provide 'test-ob-shell)
 
 ;;; test-ob-shell.el ends here
-- 
2.29.2

Re: babel output seems to drop anything before % (in session)

[John Corless]

[-- Attachment #1: Type: text/plain, Size: 1264 bytes --]

Confirmed

Daniele,

I was able to reproduce the behavior you described.  Using the test case...

#+BEGIN_SRC shell script :results output
ping -c 1 127.0.0.1
#+END_SRC

... the results output matches what I get in a bash shell.  But if you add
the :session to the header args like this...

#+BEGIN_SRC shell script :results output :session test
ping -c 1 127.0.0.1
#+END_SRC

... the output line that includes "0% packet loss" gets truncated to after
the "%" sign.

I tested with 9.4.5.

John


On Wed, May 5, 2021 at 10:19 AM Daniele Pizzolli <dan@toel.it> wrote:

>
> Hello,
>
> Try to execute a few times the following and see the output corrupted in
> the line:
>
> #+BEGIN_EXAMPLE
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> #+END_EXAMPLE
>
> #+PROPERTY: header-args:shell
> #+PROPERTY: header-args:shell+ :results output verbatim :wrap src text
> :session test
>
> #+NAME: ping
> #+BEGIN_SRC shell
> ping -c 1 127.0.0.1
> #+END_SRC
>
> I tried to write a test (now with the :expected-result :failed), that
> hit the problem both in main and master (applies fine to both).
>
> I was not able to fix it, but hopefully the tests are still useful for
> somebody more knowledgeable!
>
> Thanks for you suggestion/fix!
>
> Best,
> Daniele
>
>

[-- Attachment #2: Type: text/html, Size: 2875 bytes --]

Re: babel output seems to drop anything before % (in session)

[Nicholas Savage]

[-- Attachment #1: Type: text/plain, Size: 1622 bytes --]

I can confirm this too on the latest master.

I took a quick peek this morning, and my suspicion is that the problem is somewhere within org-babel-comint-with-output in lisp/ob-comint.el, but I'm not positive at this point.

On Wed, May 5, 2021, at 22:35, John Corless wrote:
> Confirmed
> 
> Daniele,
> 
> I was able to reproduce the behavior you described.  Using the test case...
> 
> #+BEGIN_SRC shell script :results output
> ping -c 1 127.0.0.1
> #+END_SRC
> 
> ... the results output matches what I get in a bash shell.  But if you add the :session to the header args like this...
> 
> #+BEGIN_SRC shell script :results output :session test
> ping -c 1 127.0.0.1
> #+END_SRC
> 
> ... the output line that includes "0% packet loss" gets truncated to after the "%" sign.
> 
> I tested with 9.4.5.
> 
> John
> 
> 
> On Wed, May 5, 2021 at 10:19 AM Daniele Pizzolli <dan@toel.it> wrote:
>> 
>> Hello,
>> 
>> Try to execute a few times the following and see the output corrupted in
>> the line:
>> 
>> #+BEGIN_EXAMPLE
>> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
>> #+END_EXAMPLE
>> 
>> #+PROPERTY: header-args:shell
>> #+PROPERTY: header-args:shell+ :results output verbatim :wrap src text :session test
>> 
>> #+NAME: ping
>> #+BEGIN_SRC shell
>> ping -c 1 127.0.0.1
>> #+END_SRC
>> 
>> I tried to write a test (now with the :expected-result :failed), that
>> hit the problem both in main and master (applies fine to both).
>> 
>> I was not able to fix it, but hopefully the tests are still useful for
>> somebody more knowledgeable!
>> 
>> Thanks for you suggestion/fix!
>> 
>> Best,
>> Daniele
>> 

[-- Attachment #2: Type: text/html, Size: 3965 bytes --]

Re: babel output seems to drop anything before % (in session)

[Ihor Radchenko]

"Nicholas Savage" <nick@nicksavage.ca> writes:

> I can confirm this too on the latest master.
>
> I took a quick peek this morning, and my suspicion is that the problem is somewhere within org-babel-comint-with-output in lisp/ob-comint.el, but I'm not positive at this point.

I confirm as well. I also saw an anomaly in the comint buffer. Note that
all the output lines, except "five 0% six" are after the shell prompt.
As I remember, the code expects the result to be exactly at the prompt
line. So, for some reason the first command ("echo five 0% six") of the
second block does not get inserted at the empty line.

echo one 0% two
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ one 0% two
echo tree 0% four
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ tree 0% four
echo 'org_babel_sh_eoe'
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ echo five 0% six
five 0% six
echo seven 0% eight
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ seven 0% eight
echo 'org_babel_sh_eoe'
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe
yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ 

Best,
Ihor

Re: babel output seems to drop anything before % (in session)

[Nick Savage]

[-- Attachment #1: Type: text/plain, Size: 2654 bytes --]

So I have a patch written that doesn't completely fix the problem, but 
maybe makes enough progress that someone else can figure it out.

The issue is that comint-prompt-regexp is reading the "% " as a prompt, 
and taking everyone off before it. I've added another parameter to 
org-babel-comint-with-output in the "meta" to pass our own regexp to 
replace comint-prompt-regexp, which works except when it doesn't. The 
regexp I've added is just "\n" now, so the newline characters are removed.

The tests that Daniele added as a patch don't quite pass though with 
this. The issue is that something in the way the output is posted in the 
output buffer includes the prompt occasionally is included in a line and 
occasionally not. It seems the first time the block of code is executed, 
it is included (and therefore needs to be removed) and each other time 
it is not, so it is only the first time that it is run is not working 
properly.

This is obviously way too fragile to actually merge, but I was hoping 
the work I've done so far is enough to help someone else make progress. 
I'll probably take another stab at this tomorrow (since it's bugging 
me), but thought I'd share what I have for now.

Cheers,
Nick

On 5/6/21 8:24 AM, Ihor Radchenko wrote:
> "Nicholas Savage" <nick@nicksavage.ca> writes:
>
>> I can confirm this too on the latest master.
>>
>> I took a quick peek this morning, and my suspicion is that the problem is somewhere within org-babel-comint-with-output in lisp/ob-comint.el, but I'm not positive at this point.
> I confirm as well. I also saw an anomaly in the comint buffer. Note that
> all the output lines, except "five 0% six" are after the shell prompt.
> As I remember, the code expects the result to be exactly at the prompt
> line. So, for some reason the first command ("echo five 0% six") of the
> second block does not get inserted at the empty line.
>
> echo one 0% two
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ one 0% two
> echo tree 0% four
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ tree 0% four
> echo 'org_babel_sh_eoe'
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ echo five 0% six
> five 0% six
> echo seven 0% eight
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ seven 0% eight
> echo 'org_babel_sh_eoe'
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $ org_babel_sh_eoe
> yantar92@yantar92-laptop ~/.data/1e/90360c-ef36-4d20-8706-990ae2530cbf $
>
> Best,
> Ihor

[-- Attachment #2: 0001-ob-shell.el-Fix-bug-where-shell-output-was-incorrect.patch --]
[-- Type: text/x-patch, Size: 3692 bytes --]

From 6c7d39bfb9be38b54d23fcffbb09f1fcb96751f4 Mon Sep 17 00:00:00 2001
From: Nicholas Savage <nick@nicksavage.ca>
Date: Thu, 6 May 2021 19:17:33 -0400
Subject: [PATCH] ob-shell.el: Fix bug where shell output was incorrectly
 truncated on special characters.

* lisp/ob-comint.el (org-babel-comint-with-output): Add fifth meta
optional argument for providing a custom prompt regexp.
lisp/ob-shell.el (org-babel-sh-evaluate): Implements using new
argument to prevent incorrect truncation on special characters.

If shell output included special characters that also occasionally are
included in shell prompts, such as "#" or "%", a regexp was tripping
up on them and cutting out part of the line. As ob-shell already cuts
out the shell prompts, this was not necessary and instead we can just
use \n as the separator. Original functionality was retained for other
ob-* files in case this was necessary.
---
 lisp/ob-comint.el | 20 ++++++++++++--------
 lisp/ob-shell.el  |  2 +-
 2 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 18d4f3c93..27ad6efd7 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -57,13 +57,14 @@ executed inside the protection of `save-excursion' and
 
 (defmacro org-babel-comint-with-output (meta &rest body)
   "Evaluate BODY in BUFFER and return process output.
-Will wait until EOE-INDICATOR appears in the output, then return
-all process output.  If REMOVE-ECHO and FULL-BODY are present and
-non-nil, then strip echo'd body from the returned output.  META
-should be a list containing the following where the last two
-elements are optional.
+Will wait until EOE-INDICATOR appears in the output, then return all
+process output.  If REMOVE-ECHO and FULL-BODY are present and non-nil,
+then strip echo'd body from the returned output.  PROMPT-REGEXP is a
+filter that, if provided, overrides the default regexp that tries to
+filter out the shell prompt.  META should be a list containing the
+following where the last three elements are optional.
 
- (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY)
+ (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY PROMPT-REGEXP)
 
 This macro ensures that the filter is removed in case of an error
 or user `keyboard-quit' during execution of body."
@@ -71,7 +72,10 @@ or user `keyboard-quit' during execution of body."
   (let ((buffer (nth 0 meta))
 	(eoe-indicator (nth 1 meta))
 	(remove-echo (nth 2 meta))
-	(full-body (nth 3 meta)))
+	(full-body (nth 3 meta))
+        (prompt-regexp (if (nth 4 meta)
+                           (nth 4 meta)
+                         comint-prompt-regexp)))
     `(org-babel-comint-in-buffer ,buffer
        (let* ((string-buffer "")
 	      (comint-output-filter-functions
@@ -111,7 +115,7 @@ or user `keyboard-quit' during execution of body."
 		      "\n" "[\r\n]+" (regexp-quote (or ,full-body "")))
 		     string-buffer))
 	   (setq string-buffer (substring string-buffer (match-end 0))))
-	 (split-string string-buffer comint-prompt-regexp)))))
+	 (split-string string-buffer ,prompt-regexp)))))
 (def-edebug-spec org-babel-comint-with-output (sexp body))
 
 (defun org-babel-comint-input-command (buffer cmd)
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 3eed0c164..9ec0425cb 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -263,7 +263,7 @@ return the value of the last statement in BODY."
 	      #'org-trim
 	      (butlast
 	       (org-babel-comint-with-output
-		   (session org-babel-sh-eoe-output t body)
+		   (session org-babel-sh-eoe-output t body "\n")
 		 (dolist (line (append (split-string (org-trim body) "\n")
 				       (list org-babel-sh-eoe-indicator)))
 		   (insert line)
-- 
2.20.1

Re: babel output seems to drop anything before % (in session)

[Bastien]

Hi Nick,

Nick Savage <nick@nicksavage.ca> writes:

> So I have a patch written that doesn't completely fix the problem, but
> maybe makes enough progress that someone else can figure it out.

I hope you can continue to make progress on this (annoying) one.  

Let us know!

-- 
 Bastien

Re: babel output seems to drop anything before % (in session)

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1023 bytes --]

Today I ended up on possibly a related issue with ob-shell results.

Observations:
1. Issue triggered when using :session
2. Output text is OK for the the first shell src block's output. But for
the second block, the output before '>' gets truncated.

Here's a minimal reproducible example:

=====
#+property: header-args:shell :exports both :results output :session
*ob-shell-session-issue*

#+begin_src shell
echo "1 > 0"
#+end_src

#+RESULTS:
:
: 1 > 0

#+begin_src shell
echo "1 > 0"
#+end_src

#+RESULTS:
:
: 0
=====

The root cause seems to be similar, with detection of shell prompt: '>',
'%', etc. when session is enabled.


--
Kaushal Modi


On Sat, May 15, 2021 at 4:48 PM Bastien <bzg@gnu.org> wrote:

> Hi Nick,
>
> Nick Savage <nick@nicksavage.ca> writes:
>
> > So I have a patch written that doesn't completely fix the problem, but
> > maybe makes enough progress that someone else can figure it out.
>
> I hope you can continue to make progress on this (annoying) one.
>
> Let us know!
>
> --
>  Bastien
>
>

[-- Attachment #2: Type: text/html, Size: 1859 bytes --]

Re: babel output seems to drop anything before % (in session)

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1906 bytes --]

On Thu, May 6, 2021 at 7:45 PM Nick Savage <nick@nicksavage.ca> wrote:

> So I have a patch written that doesn't completely fix the problem, but
> maybe makes enough progress that someone else can figure it out.
>
> The issue is that comint-prompt-regexp is reading the "% " as a prompt,
> and taking everyone off before it. I've added another parameter to
> org-babel-comint-with-output in the "meta" to pass our own regexp to
> replace comint-prompt-regexp, which works except when it doesn't. The
> regexp I've added is just "\n" now, so the newline characters are removed.
>
> The tests that Daniele added as a patch don't quite pass though with
> this. The issue is that something in the way the output is posted in the
> output buffer includes the prompt occasionally is included in a line and
> occasionally not. It seems the first time the block of code is executed,
> it is included (and therefore needs to be removed) and each other time
> it is not, so it is only the first time that it is run is not working
> properly.
>
> This is obviously way too fragile to actually merge, but I was hoping
> the work I've done so far is enough to help someone else make progress.
> I'll probably take another stab at this tomorrow (since it's bugging
> me), but thought I'd share what I have for now.
>
> Cheers,
> Nick
>

Thanks! I tried out your patch and it does fix the truncation of results
output. Though, it retains the shell prompt in the first eval block and
removes it only from the second eval block onwards.

Here's a MWE:

=====
#+property: header-args:shell :exports both :results output :session
*ob-shell-session-issue*

#+begin_src shell
echo "1 > 0"
#+end_src

#+RESULTS:
:
: sh-4.2$ 1 > 0

#+begin_src shell
echo "1 > 0"
#+end_src

#+RESULTS:
:
: 1 > 0

#+begin_src shell
echo "1 > 0"
#+end_src

#+RESULTS:
:
: 1 > 0

#+begin_src shell
echo "1 > 0"
#+end_src

#+RESULTS:
:
: 1 > 0
=====

[-- Attachment #2: Type: text/html, Size: 2623 bytes --]

Re: babel output seems to drop anything before % (in session)

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

Hi Nick,

On Tue, Jun 8, 2021 at 11:54 PM Kaushal Modi <kaushal.modi@gmail.com> wrote:

>
> Thanks! I tried out your patch and it does fix the truncation of results
> output. Though, it retains the shell prompt in the first eval block and
> removes it only from the second eval block onwards.
>

I have one more feedback about the patch. I see that now indentation is
stripped away from ob-shell results.

I am documenting git commands in ob-shell.

In the terminal, `git log' outputs:

=====
commit ba72672c499af8b713437d8c05eb5979969954dd (HEAD -> main)
Author: Foo Bar <foo.bar@fakeemail.com>
Date:   Tue Jun 8 23:56:48 2021 -0400

    first commit
=====

But in shell src block results, I see that that indentation is gone:

=====
#+begin_src shell
git log
#+end_src

#+RESULTS:
:
: commit 34924761f042eab59567ec150a73c730ab2c929c (HEAD -> main)
: Author: Foo Bar <foo.bar@fakeemail.com>
: Date:   Tue Jun 8 23:56:48 2021 -0400
:
: first commit
=====



Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 1747 bytes --]

Re: babel output seems to drop anything before % (in session)

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 283 bytes --]

Hello all,

I am following up on this issue with ob-shell and sessions.

I had posted the recipe to reproduce the issue few months back, and it
still applies; ref:
https://list.orgmode.org/CAFyQvY2AeToQc2G=e+DE4VOTeTFbkvisENipYpW1Y-feu1ov1g@mail.gmail.com/

Thanks!

--
Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 677 bytes --]

Re: babel output seems to drop anything before % (in session)

[Ihor Radchenko]

Kaushal Modi <kaushal.modi@gmail.com> writes:

> Hello all,
>
> I am following up on this issue with ob-shell and sessions.
>
> I had posted the recipe to reproduce the issue few months back, and it
> still applies; ref:
> https://list.orgmode.org/CAFyQvY2AeToQc2G=e+DE4VOTeTFbkvisENipYpW1Y-feu1ov1g@mail.gmail.com/

FYI, I attempted to fix this several times, but the underlying code is
not trivial. Hopefully someone™ more familiar with comint can make a
working patch.

If you encounter this issue frequently, you may be the most motivated
person here to spend time studying the code. Ideas how to approach the
problem would be appreciated.

Best,
Ihor

Re: babel output seems to drop anything before % (in session)

[Felix Freeman via General discussions about Org-mode.]

Today I stumbed upon the same bug.

Sadly I have no idea how to fix it, but thought it would be nice to post
about it here so the bug doesn't get lost in the sands of time.

Felix Freeman

Re: babel output seems to drop anything before % (in session)

[Felix Freeman via General discussions about Org-mode.]

I've been able to bypass the error by adding the following to my .emacs

(require 'ob-shell)

(defun org-babel-sh-strip-weird-long-prompt (string)
  "Remove prompt cruft from a string of shell output."
  (while (string-match shell-prompt-pattern string)
    (setq string (substring string (match-end 0))))
  string)

(setq shell-prompt-pattern "^sh-[[:digit:]].[[:digit:]]\$ ")

Of course, this is is just a dirty hack not suitable for a release.

It seems logical that `org-babel-sh-strip-weird-long-prompt` on
`ob-shell.el` respects `shell-prompt-pattern` from `shell.el`. It's not
like this on the released source code.

But to be honest still the idea ripping a dynamic string from the shell
output seems pretty hackish to me. I'm not really aware of the internals
of `comint.el`, but I think that instead it should set the shell $PS1 to
an empty value and read the thing afterwards, or - if this changes
things too much - at least set $PS1 to a very unique and distinctive
value in order to avoid these kind of bugs.

Felix Freeman

Re: babel output seems to drop anything before % (in session)

[Felix Freeman via General discussions about Org-mode.]

> (setq shell-prompt-pattern "^sh-[[:digit:]].[[:digit:]]\$ ")

Updated this bit for something better written for emacs and suitable for
my use cases: bash and SSH over TRAMP. I'll leave it here in case anyone
else needs it in the future:

(setq shell-prompt-pattern "^\\(sh-[[:digit:]]\\.[[:digit:]]\\|/ssh:[^:]+:[^$]+\\)$ ")

Notice it requires the patch to org-babel-sh-strip-weird-long-prompt
posted earlier.

I hope I got it right this time ^_^'

Felix Freeman

[PATCH] babel output seems to drop anything before % (in session)

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

"Felix Freeman" <libsys@hacktivista.org> writes:

> Today I stumbed upon the same bug.
>
> Sadly I have no idea how to fix it, but thought it would be nice to post
> about it here so the bug doesn't get lost in the sands of time.

I finally managed to get some working fix.
At least it kind of works on my system using bash.
The idea is changing shell prompt to something unique when we initialize
the session.

See the attached patch.

Please test this on "sh" "zsh" "fish" "csh" "ash" "dash" "ksh" "mksh"
and "posh".
Especially on fish, csh, and posh (which I believe refers to
PowerShell).

Also, doing PS1="blabla > " in bash may not always work. At least it
does not work with https://github.com/nojhan/liquidprompt

So, more reliable ways to change prompt are welcome.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-disabled-test-for-in-babel-shell-output.patch --]
[-- Type: text/x-patch, Size: 2022 bytes --]

From 141f810a658d652e1cb3a147c5f71659f62df86f Mon Sep 17 00:00:00 2001
Message-Id: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com>
From: Daniele Pizzolli <dan+git@toel.it>
Date: Wed, 5 May 2021 17:00:28 +0200
Subject: [PATCH 1/2] Add disabled test for % in babel shell output

---
 testing/lisp/test-ob-shell.el | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index 2f346f699..e81058ac7 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -106,6 +106,35 @@ (ert-deftest ob-shell/simple-list ()
 	   "#+BEGIN_SRC sh :results output :var l='(1 2)\necho ${l}\n#+END_SRC"
 	   (org-trim (org-babel-execute-src-block))))))
 
+(ert-deftest ob-shell/percent-simple ()
+  "Test percent in output."
+  (should
+   (equal "one 0% two\ntree 0% four"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim\necho one 0% two\necho tree 0% four\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block)))))
+  (should
+   (equal "five 0% two\ntree 0% four"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim\necho five 0% two\necho tree 0% four\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block))))))
+
+(ert-deftest ob-shell/percent-session ()
+  "Test percent in output for session.
+
+At the second iteration the string before % is dropped."
+  :expected-result :failed
+  (should
+   (equal "one 0% two\ntree 0% four"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim :session sess\necho one 0% two\necho tree 0% four\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block)))))
+  (should
+   (equal "five 0% six\nseven 0% eight"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim :session sess\necho five 0% six\necho seven 0% eight\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block))))))
+
 (provide 'test-ob-shell)
 
 ;;; test-ob-shell.el ends here
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-ob-shell-Fix-output-containing-strings-matching-comi.patch --]
[-- Type: text/x-patch, Size: 6579 bytes --]

From 4f7a1369e72b22836cb74931c7c86e845155a729 Mon Sep 17 00:00:00 2001
Message-Id: <4f7a1369e72b22836cb74931c7c86e845155a729.1655522152.git.yantar92@gmail.com>
In-Reply-To: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com>
References: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sat, 18 Jun 2022 11:11:12 +0800
Subject: [PATCH 2/2] ob-shell: Fix output containing strings matching
 `comint-prompt-regexp'

* lisp/ob-comint.el (org-babel-comint-wait-for-output): Do not rely on
`face-at-point' returning non-nil.
* lisp/ob-shell.el (org-babel-shell-set-prompt-commands): New constant
holding shell-specific commands to change prompt.
(org-babel-prompt-command): New variable holding command to be user to
set distinguishable prompt.
(org-babel-shell-initialize): Set `org-babel-prompt-command' according
to shell name.
(org-babel-sh-prompt): New variable holding default shell prompt.
(org-babel-sh-initiate-session): Change the default prompt to
`org-babel-sh-prompt' and alter `comint-prompt-regexp' to match it
tightly.
* testing/lisp/test-ob-shell.el (ob-shell/percent-session): Do not
expect failure in the fixed test.

Fixes https://list.orgmode.org/CKK9TULBP2BG.2UITT31YJV03J@laptop/T/#mc8e3ca2f5f1b9a94040a68b4c6201234b209041c
---
 lisp/ob-comint.el             |  4 +---
 lisp/ob-shell.el              | 42 ++++++++++++++++++++++++++++++-----
 testing/lisp/test-ob-shell.el |  4 ++--
 3 files changed, 39 insertions(+), 11 deletions(-)

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 427aba341..1d3e24607 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -124,9 +124,7 @@ (defun org-babel-comint-wait-for-output (buffer)
     (while (progn
              (goto-char comint-last-input-end)
              (not (and (re-search-forward comint-prompt-regexp nil t)
-                       (goto-char (match-beginning 0))
-                       (string= (face-name (face-at-point))
-                                "comint-highlight-prompt"))))
+                     (goto-char (match-beginning 0)))))
       (accept-process-output (get-buffer-process buffer)))))
 
 (defun org-babel-comint-eval-invisibly-and-wait-for-file
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index c25941a44..f80783653 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -42,6 +42,23 @@ (declare-function orgtbl-to-generic "org-table" (table params))
 (defvar org-babel-default-header-args:shell '())
 (defvar org-babel-shell-names)
 
+(defconst org-babel-shell-set-prompt-commands
+  '(("fish" . "function fish_prompt\n\techo \"%s\"\nend")
+    ("csh" . "set prompt=\"%s\"")
+    ("posh" . "function prompt { \"%s\" }")
+    (t . "PS1=\"%s\""))
+  "Alist assigning shells with their prompt setting command.
+
+Each element of the alist associates a shell type from
+`org-babel-shell-names' with a template used to create a command to
+change the default prompt.  The template is an argument to `format'
+that will be called with a single additional argument: prompt string.
+
+The fallback association template is defined in (t . \"template\")
+alist element.")
+
+(defvar org-babel-prompt-command)
+
 (defun org-babel-shell-initialize ()
   "Define execution functions associated to shell names.
 This function has to be called whenever `org-babel-shell-names'
@@ -51,7 +68,10 @@ (defun org-babel-shell-initialize ()
     (eval `(defun ,(intern (concat "org-babel-execute:" name))
 	       (body params)
 	     ,(format "Execute a block of %s commands with Babel." name)
-	     (let ((shell-file-name ,name))
+	     (let ((shell-file-name ,name)
+                   (org-babel-prompt-command
+                    (or (alist-get ,name org-babel-shell-set-prompt-commands)
+                        (alist-get t org-babel-shell-set-prompt-commands))))
 	       (org-babel-execute:shell body params))))
     (eval `(defalias ',(intern (concat "org-babel-variable-assignments:" name))
 	     'org-babel-variable-assignments:shell
@@ -206,6 +226,13 @@ (defun org-babel-sh-var-to-string (var &optional sep hline)
       (mapconcat echo-var var "\n"))
      (t (funcall echo-var var)))))
 
+(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
+  "String to indicate that evaluation has completed.")
+(defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
+  "String to indicate that evaluation has completed.")
+(defvar org-babel-sh-prompt "org_babel_sh_prompt> "
+  "String to set prompt in session shell.")
+
 (defun org-babel-sh-initiate-session (&optional session _params)
   "Initiate a session named SESSION according to PARAMS."
   (when (and session (not (string= session "none")))
@@ -213,17 +240,20 @@ (defun org-babel-sh-initiate-session (&optional session _params)
       (or (org-babel-comint-buffer-livep session)
           (progn
 	    (shell session)
+            ;; Set unique prompt for easier analysis of the output.
+            (org-babel-comint-wait-for-output (current-buffer))
+            (org-babel-comint-input-command
+             (current-buffer)
+             (format org-babel-prompt-command org-babel-sh-prompt))
+            (setq-local comint-prompt-regexp
+                        (concat "^" (regexp-quote org-babel-sh-prompt)
+                                " *"))
 	    ;; Needed for Emacs 23 since the marker is initially
 	    ;; undefined and the filter functions try to use it without
 	    ;; checking.
 	    (set-marker comint-last-output-start (point))
 	    (get-buffer (current-buffer)))))))
 
-(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
-  "String to indicate that evaluation has completed.")
-(defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
-  "String to indicate that evaluation has completed.")
-
 (defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
   "Pass BODY to the Shell process in BUFFER.
 If RESULT-TYPE equals `output' then return a list of the outputs
diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index e81058ac7..2bfc29228 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -122,8 +122,8 @@ (ert-deftest ob-shell/percent-simple ()
 (ert-deftest ob-shell/percent-session ()
   "Test percent in output for session.
 
-At the second iteration the string before % is dropped."
-  :expected-result :failed
+Percent is matching the default value of `comint-prompt-regexp', which
+can cause issues with analysis."
   (should
    (equal "one 0% two\ntree 0% four"
 	  (org-test-with-temp-text
-- 
2.35.1

Re: [PATCH] babel output seems to drop anything before % (in session)

[Ihor Radchenko]

Ihor Radchenko <yantar92@gmail.com> writes:

> I finally managed to get some working fix.
> At least it kind of works on my system using bash.
> The idea is changing shell prompt to something unique when we initialize
> the session.
>
> See the attached patch.

Fixed.
Applied onto main via a35d16368.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a35d163685908386833a3d549ed110931bf3915a

I did not do anything with Daniele's patch because it requires FSF
copyright.


-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [PATCH] babel output seems to drop anything before % (in session)

[Ihor Radchenko]

Ihor Radchenko <yantar92@gmail.com> writes:

> Applied onto main via a35d16368.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a35d163685908386833a3d549ed110931bf3915a
>
> I did not do anything with Daniele's patch because it requires FSF
> copyright.

Bastien,

If I recall correctly, our tests are not all contributed by people with
FSF assignments even for non-tiny contributions.

May you clarify if we are following the FSF copyright assignment rules
for test?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

Re: [PATCH] babel output seems to drop anything before % (in session)

[Bastien Guerry]

Ihor Radchenko <yantar92@gmail.com> writes:

> May you clarify if we are following the FSF copyright assignment rules
> for test?

Yes, we should.  Tests are just source code.

-- 
 Bastien

Re: babel output seems to drop anything before % (in session)

[Ihor Radchenko]

Daniele Pizzolli <dan@toel.it> writes:

> From 897ae50cf055d459d741c90f8de94dd56a8eb3d9 Mon Sep 17 00:00:00 2001
> From: Daniele Pizzolli <dan+git@toel.it>
> Date: Wed, 5 May 2021 17:00:28 +0200
> Subject: [PATCH 1/1] Add disabled test for % in babel shell output

Daniele, your patch is over 15LOC. Would you mind doing FSF paperwork
as described in https://orgmode.org/worg/org-contribute.html#copyright ?

Best,
Ihor