[PATCH] babel output seems to drop anything before % (in session)

[Ihor Radchenko <yantar92@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

"Felix Freeman" <libsys@hacktivista.org> writes:

> Today I stumbed upon the same bug.
>
> Sadly I have no idea how to fix it, but thought it would be nice to post
> about it here so the bug doesn't get lost in the sands of time.

I finally managed to get some working fix.
At least it kind of works on my system using bash.
The idea is changing shell prompt to something unique when we initialize
the session.

See the attached patch.

Please test this on "sh" "zsh" "fish" "csh" "ash" "dash" "ksh" "mksh"
and "posh".
Especially on fish, csh, and posh (which I believe refers to
PowerShell).

Also, doing PS1="blabla > " in bash may not always work. At least it
does not work with https://github.com/nojhan/liquidprompt

So, more reliable ways to change prompt are welcome.

Best,
Ihor


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-disabled-test-for-in-babel-shell-output.patch --]
[-- Type: text/x-patch, Size: 2022 bytes --]

From 141f810a658d652e1cb3a147c5f71659f62df86f Mon Sep 17 00:00:00 2001
Message-Id: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com>
From: Daniele Pizzolli <dan+git@toel.it>
Date: Wed, 5 May 2021 17:00:28 +0200
Subject: [PATCH 1/2] Add disabled test for % in babel shell output

---
 testing/lisp/test-ob-shell.el | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index 2f346f699..e81058ac7 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -106,6 +106,35 @@ (ert-deftest ob-shell/simple-list ()
 	   "#+BEGIN_SRC sh :results output :var l='(1 2)\necho ${l}\n#+END_SRC"
 	   (org-trim (org-babel-execute-src-block))))))
 
+(ert-deftest ob-shell/percent-simple ()
+  "Test percent in output."
+  (should
+   (equal "one 0% two\ntree 0% four"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim\necho one 0% two\necho tree 0% four\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block)))))
+  (should
+   (equal "five 0% two\ntree 0% four"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim\necho five 0% two\necho tree 0% four\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block))))))
+
+(ert-deftest ob-shell/percent-session ()
+  "Test percent in output for session.
+
+At the second iteration the string before % is dropped."
+  :expected-result :failed
+  (should
+   (equal "one 0% two\ntree 0% four"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim :session sess\necho one 0% two\necho tree 0% four\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block)))))
+  (should
+   (equal "five 0% six\nseven 0% eight"
+	  (org-test-with-temp-text
+	   "#+BEGIN_SRC bash :results output verbatim :session sess\necho five 0% six\necho seven 0% eight\n#+END_SRC"
+	   (org-trim (org-babel-execute-src-block))))))
+
 (provide 'test-ob-shell)
 
 ;;; test-ob-shell.el ends here
-- 
2.35.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-ob-shell-Fix-output-containing-strings-matching-comi.patch --]
[-- Type: text/x-patch, Size: 6579 bytes --]

From 4f7a1369e72b22836cb74931c7c86e845155a729 Mon Sep 17 00:00:00 2001
Message-Id: <4f7a1369e72b22836cb74931c7c86e845155a729.1655522152.git.yantar92@gmail.com>
In-Reply-To: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com>
References: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sat, 18 Jun 2022 11:11:12 +0800
Subject: [PATCH 2/2] ob-shell: Fix output containing strings matching
 `comint-prompt-regexp'

* lisp/ob-comint.el (org-babel-comint-wait-for-output): Do not rely on
`face-at-point' returning non-nil.
* lisp/ob-shell.el (org-babel-shell-set-prompt-commands): New constant
holding shell-specific commands to change prompt.
(org-babel-prompt-command): New variable holding command to be user to
set distinguishable prompt.
(org-babel-shell-initialize): Set `org-babel-prompt-command' according
to shell name.
(org-babel-sh-prompt): New variable holding default shell prompt.
(org-babel-sh-initiate-session): Change the default prompt to
`org-babel-sh-prompt' and alter `comint-prompt-regexp' to match it
tightly.
* testing/lisp/test-ob-shell.el (ob-shell/percent-session): Do not
expect failure in the fixed test.

Fixes https://list.orgmode.org/CKK9TULBP2BG.2UITT31YJV03J@laptop/T/#mc8e3ca2f5f1b9a94040a68b4c6201234b209041c
---
 lisp/ob-comint.el             |  4 +---
 lisp/ob-shell.el              | 42 ++++++++++++++++++++++++++++++-----
 testing/lisp/test-ob-shell.el |  4 ++--
 3 files changed, 39 insertions(+), 11 deletions(-)

diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el
index 427aba341..1d3e24607 100644
--- a/lisp/ob-comint.el
+++ b/lisp/ob-comint.el
@@ -124,9 +124,7 @@ (defun org-babel-comint-wait-for-output (buffer)
     (while (progn
              (goto-char comint-last-input-end)
              (not (and (re-search-forward comint-prompt-regexp nil t)
-                       (goto-char (match-beginning 0))
-                       (string= (face-name (face-at-point))
-                                "comint-highlight-prompt"))))
+                     (goto-char (match-beginning 0)))))
       (accept-process-output (get-buffer-process buffer)))))
 
 (defun org-babel-comint-eval-invisibly-and-wait-for-file
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index c25941a44..f80783653 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -42,6 +42,23 @@ (declare-function orgtbl-to-generic "org-table" (table params))
 (defvar org-babel-default-header-args:shell '())
 (defvar org-babel-shell-names)
 
+(defconst org-babel-shell-set-prompt-commands
+  '(("fish" . "function fish_prompt\n\techo \"%s\"\nend")
+    ("csh" . "set prompt=\"%s\"")
+    ("posh" . "function prompt { \"%s\" }")
+    (t . "PS1=\"%s\""))
+  "Alist assigning shells with their prompt setting command.
+
+Each element of the alist associates a shell type from
+`org-babel-shell-names' with a template used to create a command to
+change the default prompt.  The template is an argument to `format'
+that will be called with a single additional argument: prompt string.
+
+The fallback association template is defined in (t . \"template\")
+alist element.")
+
+(defvar org-babel-prompt-command)
+
 (defun org-babel-shell-initialize ()
   "Define execution functions associated to shell names.
 This function has to be called whenever `org-babel-shell-names'
@@ -51,7 +68,10 @@ (defun org-babel-shell-initialize ()
     (eval `(defun ,(intern (concat "org-babel-execute:" name))
 	       (body params)
 	     ,(format "Execute a block of %s commands with Babel." name)
-	     (let ((shell-file-name ,name))
+	     (let ((shell-file-name ,name)
+                   (org-babel-prompt-command
+                    (or (alist-get ,name org-babel-shell-set-prompt-commands)
+                        (alist-get t org-babel-shell-set-prompt-commands))))
 	       (org-babel-execute:shell body params))))
     (eval `(defalias ',(intern (concat "org-babel-variable-assignments:" name))
 	     'org-babel-variable-assignments:shell
@@ -206,6 +226,13 @@ (defun org-babel-sh-var-to-string (var &optional sep hline)
       (mapconcat echo-var var "\n"))
      (t (funcall echo-var var)))))
 
+(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
+  "String to indicate that evaluation has completed.")
+(defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
+  "String to indicate that evaluation has completed.")
+(defvar org-babel-sh-prompt "org_babel_sh_prompt> "
+  "String to set prompt in session shell.")
+
 (defun org-babel-sh-initiate-session (&optional session _params)
   "Initiate a session named SESSION according to PARAMS."
   (when (and session (not (string= session "none")))
@@ -213,17 +240,20 @@ (defun org-babel-sh-initiate-session (&optional session _params)
       (or (org-babel-comint-buffer-livep session)
           (progn
 	    (shell session)
+            ;; Set unique prompt for easier analysis of the output.
+            (org-babel-comint-wait-for-output (current-buffer))
+            (org-babel-comint-input-command
+             (current-buffer)
+             (format org-babel-prompt-command org-babel-sh-prompt))
+            (setq-local comint-prompt-regexp
+                        (concat "^" (regexp-quote org-babel-sh-prompt)
+                                " *"))
 	    ;; Needed for Emacs 23 since the marker is initially
 	    ;; undefined and the filter functions try to use it without
 	    ;; checking.
 	    (set-marker comint-last-output-start (point))
 	    (get-buffer (current-buffer)))))))
 
-(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'"
-  "String to indicate that evaluation has completed.")
-(defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
-  "String to indicate that evaluation has completed.")
-
 (defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
   "Pass BODY to the Shell process in BUFFER.
 If RESULT-TYPE equals `output' then return a list of the outputs
diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index e81058ac7..2bfc29228 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -122,8 +122,8 @@ (ert-deftest ob-shell/percent-simple ()
 (ert-deftest ob-shell/percent-session ()
   "Test percent in output for session.
 
-At the second iteration the string before % is dropped."
-  :expected-result :failed
+Percent is matching the default value of `comint-prompt-regexp', which
+can cause issues with analysis."
   (should
    (equal "one 0% two\ntree 0% four"
 	  (org-test-with-temp-text
-- 
2.35.1