[BUG] ":sep" header argument doesn't work in ob-shell [9.6.21 ( @ /home/aragaer/.config/emacs/elpa/org-9.6.21/)]

[BUG] ":sep" header argument doesn't work in ob-shell [9.6.21 ( @ /home/aragaer/.config/emacs/elpa/org-9.6.21/)]

[Ilya Konovalov]

On current master using ":sep" header argument does not work as
expected and records are always split on whitespaces. For example:

#+begin_src sh :sep "\t"
  echo hello world
  echo test
#+end_src

#+RESULTS:
| hello | world |
| test  |       |

It looks like the issue is in the org-babel-sh-evaluate function, where
at the end it passes the result to the org-babel-import-elist-from-file
function for processing. Replacing the code with the following seems to
fix the issue:

(let ((tmp-file (org-babel-temp-file "sh-"))
      (sep (or (cdr (assq :sep params)) "\t")))
  (with-temp-file tmp-file (insert results))
  (org-babel-import-elisp-from-file tmp-file sep))

Emacs  : GNU Emacs 29.2 (build 1, x86_64-pc-linux-gnu, X toolkit)
 of 2024-03-02
Package: Org mode version 9.6.21 ( @ /home/aragaer/.config/emacs/elpa/org-9.6.21/)

Re: [BUG] ":sep" header argument doesn't work in ob-shell [9.6.21 ( @ /home/aragaer/.config/emacs/elpa/org-9.6.21/)]

[Ilya Konovalov]

After some usage I found that this exact patch breaks
`org-babel-import-elisp-from-file's own separator detection. It looks
like it's better to leave it as `nil' unless specified.

Ilya Konovalov <aragaer@gmail.com> writes:

> On current master using ":sep" header argument does not work as
> expected and records are always split on whitespaces. For example:
>
> #+begin_src sh :sep "\t"
>   echo hello world
>   echo test
> #+end_src
>
> #+RESULTS:
> | hello | world |
> | test  |       |
>
> It looks like the issue is in the org-babel-sh-evaluate function, where
> at the end it passes the result to the org-babel-import-elist-from-file
> function for processing. Replacing the code with the following seems to
> fix the issue:
>
> (let ((tmp-file (org-babel-temp-file "sh-"))
>       (sep (or (cdr (assq :sep params)) "\t")))
>   (with-temp-file tmp-file (insert results))
>   (org-babel-import-elisp-from-file tmp-file sep))
>
> Emacs  : GNU Emacs 29.2 (build 1, x86_64-pc-linux-gnu, X toolkit)
>  of 2024-03-02
> Package: Org mode version 9.6.21 ( @ /home/aragaer/.config/emacs/elpa/org-9.6.21/)

Re: [BUG] ":sep" header argument doesn't work in ob-shell [9.6.21 ( @ /home/aragaer/.config/emacs/elpa/org-9.6.21/)]

[Matt]

 ---- On Tue, 12 Mar 2024 10:27:46 +0100  Ilya Konovalov  wrote --- 
 > 
 > After some usage I found that this exact patch breaks
 > `org-babel-import-elisp-from-file's own separator detection. It looks
 > like it's better to leave it as `nil' unless specified.
 
Thanks for taking the time to look into it.  According to my notes, :sep and :separator don't work for anything beyond basic table layout.  I'm not familiar with org-table and there isn't much documentation on those keywords in the manual.  The :sep keyword shows up for Texinfo and LaTeX.   It looks like :sep is used for export?  It's not clear to me how they're supposed to work with shell block output, or if that was even a consideration at all.

They were added in https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a60a83fbb65a368bb96ea6aaf1db20706b3e99c5 claiming "better handling of table in shell scripts"  So, it looks like their inclusion was intentional.

Maybe someone has deeper insights?

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode

Re: [BUG] ":sep" header argument doesn't work in ob-shell [9.6.21 ( @ /home/aragaer/.config/emacs/elpa/org-9.6.21/)]

[Ilya Konovalov]

I assumed that https://org-babel.readthedocs.io/ is at least somewhat
related to org documentation. Looks like I was wrong.

The :separator parameter seems to be related to how block's :var
arguments are formatted when passing to the shell script.

In most cases separator is not passed to
`org-babel-import-elisp-from-file' -- looks like only some languages set
it to some fixed value.

Matt <matt@excalamus.com> writes:

>  ---- On Tue, 12 Mar 2024 10:27:46 +0100  Ilya Konovalov  wrote --- 
>  > 
>  > After some usage I found that this exact patch breaks
>  > `org-babel-import-elisp-from-file's own separator detection. It looks
>  > like it's better to leave it as `nil' unless specified.
>  
> Thanks for taking the time to look into it.  According to my notes,
> :sep and :separator don't work for anything beyond basic table layout.
> I'm not familiar with org-table and there isn't much documentation on
> those keywords in the manual. The :sep keyword shows up for Texinfo
> and LaTeX. It looks like :sep is used for export? It's not clear to me
> how they're supposed to work with shell block output, or if that was
> even a consideration at all.
>
> They were added in
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a60a83fbb65a368bb96ea6aaf1db20706b3e99c5
> claiming "better handling of table in shell scripts" So, it looks like
> their inclusion was intentional.
>
> Maybe someone has deeper insights?

Re: [BUG] ":sep" header argument doesn't work in ob-shell [9.6.21 ( @ /home/aragaer/.config/emacs/elpa/org-9.6.21/)]

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

> ...  According to my notes, :sep and :separator don't work for
> anything beyond basic table layout. I'm not familiar with org-table
> and there isn't much documentation on those keywords in the manual.
> The :sep keyword shows up for Texinfo and LaTeX. It looks like :sep is
> used for export? It's not clear to me how they're supposed to work
> with shell block output, or if that was even a consideration at all.
>
> They were added in
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=a60a83fbb65a368bb96ea6aaf1db20706b3e99c5
> claiming "better handling of table in shell scripts" So, it looks like
> their inclusion was intentional.
>
> Maybe someone has deeper insights?

:sep header argument is used to format table result when writing to file:

     By default, Org assumes that a table written to a file has
     TAB-delimited output.  You can choose a different separator with
     the ‘sep’ header argument.

"16.6 Results of Evaluation" section of the manual.
It is only relevant when you have :results file and the evaluation
results are redirected to file instead of Org buffer.

:sep header argument is universal to all babel backends.

:separator argument is used in ob-sqlite and ob-shell.
However, it means different things in ob-sqlite vs. ob-shell.

In ob-sqlite, :separator argument is defining how sqlite formats its
output. AFAIU, it is only needed when sql table data contains "|" - the
default separator.

In ob-shell, :separator argument is _not_ used to format the output.
Instead, it is used to control how ob-shell passes Elisp data shell variables.
However, it is almost never used in practice (also, it is not
documented) - :separator argument only affects how deeply nested Elisp
lists are converted to bash data:

#+begin_src bash :var x='((((1 1)) 3) (4 5)) :results output :cache no :separator "+"
echo keys: ${!x[@]}
echo values: ${x[@]}
#+end_src

#+RESULTS[7de8e7c561edda2dce18a302f4d8beb8a51595df]:
: keys: 4 1+1
: values: 5 3

Normal two-dimentional tables and lists are handled without being
affected by :separator.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ":sep" header argument doesn't work in ob-shell [9.6.21 ( @ /home/aragaer/.config/emacs/elpa/org-9.6.21/)]

[Ihor Radchenko]

Ilya Konovalov <aragaer@gmail.com> writes:

> On current master using ":sep" header argument does not work as
> expected and records are always split on whitespaces. For example:
>
> #+begin_src sh :sep "\t"
>   echo hello world
>   echo test
> #+end_src
>
> #+RESULTS:
> | hello | world |
> | test  |       |

It looks like you are trying to achieve precise control over how
ob-shell interprets the evaluation result.

Note that you can always force :results scalar to interpret the output
as string. Otherwise, Org mode tries to guess the output format.

Since bash does not have a notion of table output, Org's guess is
arbitrary - split multiline result string by line assuming whitespace as
field separator.

So, this is not a bug.
Canceled.

To parse the result as table using custom rules, you can use :post
header argument + custom src block that parser the output string as you
need.

We might consider making ob-shell output handling smarter, as a feature
request.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>