Bug: Org-babel output malformed for MATLAB source blocks running in a session [9.3 (release_9.3 @ /usr/share/emacs/27.1/lisp/org/)]

[Karthik Chikmagalur <karthikchikmagalur@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3154 bytes --]


Remember to cover the basics, that is, what you expected to happen and
what in fact did happen.  You don't know how to make a good report?  See

     https://orgmode.org/manual/Feedback.html#Feedback

Your bug report will be posted to the Org mailing list.
------------------------------------------------------------------------
Org-babel's handling of MATLAB block output with the =:results session= argument is broken. Here is some sample code along with the expected result:

#+begin_src matlab :results output :session *MATLAB*
a = 3;
b = 4;
c = a + b
#+end_src

#+RESULTS:
: c =
: 
:      7

However here is the actual result:

#+begin_src matlab :results output :session *MATLAB*
a = 3;
b = 4;
c = a + b
#+end_src

#+RESULTS:
#+begin_example
a = 3;
b = 4;
c = a + b

c =

     7
'org_babel_eoe'

ans =

    'org_babel_eoe'
#+end_example

There are two separate problems:

1. The =org-babel-octave-eoe-indicator= is not being stripped.
2. The comint input is being echoed in the output.

#1 is easy to fix. The problem is in the function =org-babel-octave-evaluate-session=, in the line:

 #+begin_src emacs-lisp
 (cdr (reverse (delq "" (mapcar #'org-strip-quotes
						(mapcar #'org-trim raw)))))
 #+end_src

Where empty lines in the output are being removed with =delq=. =delq= compares with =eq= instead of =equal=, which fails on blank lines. Replacing this with =delete= works fine.

#2 is a much trickier problem to solve. Here is the body of the input as it appears in the MATLAB session:

#+begin_example
>> a = 3;
b = 4;
c = a + b
'org_babel_eoe'
a = 3;
>> b = 4;
>> c = a + b

c =

     7

>> 'org_babel_eoe'

ans =

    'org_babel_eoe'

>> 
#+end_example

Here `>>' is the shell prompt, which is absent from the raw comint output as given to org by comint-mode. =matlab-shell= does not work like other comint shells in that it doesn't echo bulk (/i.e./ multi-line) input all at once. Instead, it echoes each line of input and follows it with its output. This interspersal of input and output is causing =org-babel-comint-with-output=, the function (actually macro) responsible for removing the comint input echo text from the raw comint output, to fail. This macro assumes that the raw comint output looks like 
#+begin_example
<<contiguous comint input from org source block>>

<<contiguous comint output>>
#+end_example

as you can see from this code from =org-babel-comint-with-output=:
#+begin_src emacs-lisp
	 (when (and ,remove-echo ,full-body
		    (string-match
		     (replace-regexp-in-string
		      "\n" "[\r\n]+" (regexp-quote (or ,full-body "")))
		     string-buffer))
	   (setq string-buffer (substring string-buffer (match-end 0))))
#+end_src

To fix this, either =org-babel-octave-evaluate-session= or =org-babel-comint-with-output= needs to be modified. This problem is local to MATLAB, it does not happen with GNU Octave, which shares most of its org-babel code with MATLAB's. So I wrote a patch to the former that does additional line-by-line processing on the raw comint output to detect and remove the echoed input. (Patch is attached.) However while this fixes the problem it's not a robust solution.

-Karthik


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch to ob-octave.el --]
[-- Type: text/x-patch, Size: 1956 bytes --]

--- ob-octave.el	2020-11-23 11:22:01.473682045 -0800
+++ ob-octave-new.el	2020-11-23 11:10:07.961900383 -0800
@@ -187,6 +187,7 @@
 			(org-babel-process-file-name tmp-file 'noquote)))
 	       (org-babel-octave-import-elisp-from-file tmp-file))))))
 
+
 (defun org-babel-octave-evaluate-session
     (session body result-type &optional matlabp)
   "Evaluate BODY in SESSION."
@@ -237,12 +238,31 @@
       (`output
        (setq results
 	     (if matlabp
-		 (cdr (reverse (delq "" (mapcar #'org-strip-quotes
+		 (cdr (reverse (delete "" (mapcar #'org-strip-quotes
 						(mapcar #'org-trim raw)))))
 	       (cdr (member org-babel-octave-eoe-output
 			    (reverse (mapcar #'org-strip-quotes
 					     (mapcar #'org-trim raw)))))))
-       (mapconcat #'identity (reverse results) "\n")))))
+       ;; This kludge is to remove the input lines from the output. Because of
+       ;; the special way that MATLAB processes bulk comint output (the output
+       ;; of each line follows that line) the macro
+       ;; `org-babel-comint-with-output' cannot remove the echoed commands. The
+       ;; following handles this manually, by splitting both the original input
+       ;; (`BODY') and full output (`RESULTS') on newlines, comparing them line
+       ;; by line and removing all lines in BODY from RESULTS. Note that RESULTS
+       ;; is already a list of strings so additional care is needed.
+       (let* ((body-lines (split-string body "\n+"))
+              (result-lines (flatten-list
+                             (mapcar
+                              (lambda (entry) (reverse (split-string entry "\n")))
+                              results))))
+         (mapconcat
+          #'identity
+          (reverse (cl-remove-if
+                    (lambda (line) (member line body-lines))
+                    result-lines)) "\n")
+         )))))
+
 
 (defun org-babel-octave-import-elisp-from-file (file-name)
   "Import data from FILE-NAME.

[-- Attachment #3: Type: text/plain, Size: 3222 bytes --]


Emacs  : GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.22, cairo version 1.17.3)
 of 2020-08-28
Package: Org mode version 9.3 (release_9.3 @ /usr/share/emacs/27.1/lisp/org/)

current state:
==============
(setq
 org-src-mode-hook '(org-src-babel-configure-edit-buffer org-src-mode-configure-edit-buffer)
 org-link-shell-confirm-function 'yes-or-no-p
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
 org-mode-hook '(#[0 "\300\301\302\303\304$\207" [add-hook change-major-mode-hook org-show-all append local] 5]
		 #[0 "\300\301\302\303\304$\207"
		   [add-hook change-major-mode-hook org-babel-show-result-all append local] 5]
		 org-babel-result-hide-spec org-babel-hide-all-hashes)
 org-archive-hook '(org-attach-archive-delete-maybe)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-bibtex-headline-format-function #[257 "\300\236A\207" [:title] 3 "\n\n(fn ENTRY)"]
 org-babel-pre-tangle-hook '(save-buffer)
 org-tab-first-hook '(org-babel-hide-result-toggle-maybe org-babel-header-arg-expand)
 org-occur-hook '(org-first-headline-recenter)
 org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-show-empty-lines
		  org-optimize-window-after-visibility-change)
 org-speed-command-hook '(org-speed-command-activate org-babel-speed-command-activate)
 org-confirm-shell-link-function 'yes-or-no-p
 org-link-parameters '(("attachment" :follow org-attach-open-link :export org-attach-export-link :complete
			org-attach-complete-link)
		       ("id" :follow org-id-open) ("eww" :follow eww :store org-eww-store-link)
		       ("rmail" :follow org-rmail-open :store org-rmail-store-link)
		       ("mhe" :follow org-mhe-open :store org-mhe-store-link)
		       ("irc" :follow org-irc-visit :store org-irc-store-link :export org-irc-export)
		       ("info" :follow org-info-open :export org-info-export :store org-info-store-link)
		       ("gnus" :follow org-gnus-open :store org-gnus-store-link)
		       ("docview" :follow org-docview-open :export org-docview-export :store
			org-docview-store-link)
		       ("bibtex" :follow org-bibtex-open :store org-bibtex-store-link)
		       ("bbdb" :follow org-bbdb-open :export org-bbdb-export :complete org-bbdb-complete-link
			:store org-bbdb-store-link)
		       ("w3m" :store org-w3m-store-link) ("file+sys") ("file+emacs")
		       ("shell" :follow org-link--open-shell)
		       ("news" :follow #[257 "\301\300\302Q!\207" ["news" browse-url ":"] 5 "\n\n(fn URL)"])
		       ("mailto" :follow #[257 "\301\300\302Q!\207" ["mailto" browse-url ":"] 5 "\n\n(fn URL)"])
		       ("https" :follow #[257 "\301\300\302Q!\207" ["https" browse-url ":"] 5 "\n\n(fn URL)"])
		       ("http" :follow #[257 "\301\300\302Q!\207" ["http" browse-url ":"] 5 "\n\n(fn URL)"])
		       ("ftp" :follow #[257 "\301\300\302Q!\207" ["ftp" browse-url ":"] 5 "\n\n(fn URL)"])
		       ("help" :follow org-link--open-help) ("file" :complete org-link-complete-file)
		       ("elisp" :follow org-link--open-elisp) ("doi" :follow org-link--open-doi))
 org-link-elisp-confirm-function 'yes-or-no-p
 )