Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[Andras Major]

Hello, here is an example of apparently erroneous behaviour of the
:results switch.  In this example, we use an sh block, but it also
appears to affect other languages (will need more investigation).

Example Org code:

#+begin_src sh :exports output
  echo "Hello World 1"
#+end_src

#+results:
: Hello World 1

#+begin_src sh :exports value
  echo "Hello World 2"
#+end_src

#+results:
: Hello World 2

Am I missing something here or should the second instance really
output something like an integer exit value from echo?

Cheers,

  András


Emacs  : GNU Emacs 23.2.1 (i486-pc-linux-gnu, GTK+ Version 2.20.0)
 of 2010-12-11 on raven, modified by Debian
Package: Org-mode version 7.7 (release_7.7.107.g7a82)

current state:
==============
(setq
 org-export-latex-after-initial-vars-hook '(org-beamer-after-initial-vars)
 org-speed-command-hook '(org-speed-command-default-hook
org-babel-speed-command-hook)
 org-babel-load-languages '((asymptote . t) (ditaa . t) (dot . t)
(gnuplot . t) (haskell . t) (latex . t)
			    (octave . t) (R . t) (ruby . t) (scheme . t) (sh . t))
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-after-todo-state-change-hook '(org-clock-out-if-current)
 org-babel-tangle-lang-exts '(("ruby" . "rb") ("latex" . "tex")
("haskell" . "hs") ("asymptote" . "asy")
			      ("emacs-lisp" . "el"))
 org-export-blocks-postblock-hook '(org-exp-res/src-name-cleanup)
 org-export-latex-format-toc-function 'org-export-latex-format-toc-default
 org-tab-first-hook '(org-hide-block-toggle-maybe
org-src-native-tab-command-maybe
		      org-babel-hide-result-toggle-maybe)
 org-src-mode-hook '(org-src-babel-configure-edit-buffer
org-src-mode-configure-edit-buffer)
 org-confirm-shell-link-function 'yes-or-no-p
 org-export-first-hook '(org-beamer-initialize-open-trackers)
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-babel-pre-tangle-hook '(save-buffer)
 org-cycle-hook '(org-cycle-hide-archived-subtrees
org-cycle-hide-drawers org-cycle-show-empty-lines
		  org-optimize-window-after-visibility-change)
 org-export-preprocess-before-normalizing-links-hook
'(org-remove-file-link-modifiers)
 org-mode-hook '(#[nil "\300\301\302\303\304$\207"
		   [org-add-hook change-major-mode-hook org-show-block-all append local] 5]
		 #[nil "\300\301\302\303\304$\207"
		   [org-add-hook change-major-mode-hook org-babel-show-result-all
append local] 5]
		 org-babel-result-hide-spec org-babel-hide-all-hashes)
 org-ctrl-c-ctrl-c-hook '(org-babel-hash-at-point
org-babel-execute-safely-maybe)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-export-interblocks '((lob org-babel-exp-lob-one-liners) (src
org-babel-exp-inline-src-blocks))
 org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
 org-occur-hook '(org-first-headline-recenter)
 org-from-is-user-regexp nil
 org-export-preprocess-before-selecting-backend-code-hook
'(org-beamer-select-beamer-code)
 org-confirm-babel-evaluate nil
 org-export-latex-final-hook '(org-beamer-amend-header
org-beamer-fix-toc org-beamer-auto-fragile-frames
			       org-beamer-place-default-actions-for-lists)
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-export-blocks '((src org-babel-exp-src-block nil) (comment
org-export-blocks-format-comment t)
		     (ditaa org-export-blocks-format-ditaa nil) (dot
org-export-blocks-format-dot nil))
 )

Re: Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[Eric Schulte]

Andras Major <andras.g.major@gmail.com> writes:

> Hello, here is an example of apparently erroneous behaviour of the
> :results switch.  In this example, we use an sh block, but it also
> appears to affect other languages (will need more investigation).
>
> Example Org code:
>
> #+begin_src sh :exports output
>   echo "Hello World 1"
> #+end_src
>
> #+results:
> : Hello World 1
>
> #+begin_src sh :exports value
>   echo "Hello World 2"
> #+end_src
>
> #+results:
> : Hello World 2
>
> Am I missing something here or should the second instance really
> output something like an integer exit value from echo?
>

Hi András,

If we did return the value of shell scripts then ":results value" would
almost always simply return 0 (or possibly an error message).  For this
reason shell code blocks do not implement value returns, but rather will
always collect results from STDOUT.

Best -- Eric

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[András Major]

Hi Eric,

> If we did return the value of shell scripts then ":results value" would
> almost always simply return 0 (or possibly an error message).  For this
> reason shell code blocks do not implement value returns, but rather will
> always collect results from STDOUT.

I think that this unnecessarily throws away potentially useful
functionality.  Example: I want to fill a table with data such that
the value of a cell depends on whether a file (whose path is specified
by another cell, for instance) exists or not.  This would be most
easily done using an sh block which returns a numeric exit code.  I
don't see a reason for making clandestine exceptions to the rules in
the manual and strongly suggest that the output and value options be
honoured for every language.

In order not to break existing Org files, I would suggest that the
default choice between value and output (when not explicitly
specified) depend on the language.  With this functionality, sh code
blocks that don't specify ":results output" will still work as they
did before.

  András

Re: Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[Achim Gratz]

András Major <andras.g.major@gmail.com> writes:
> This would be most easily done using an sh block which returns a
> numeric exit code.

No one keeps you from using

echo $?

in the last line of that shell block.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[Sebastien Vauban]

Hi András and Eric,

András Major wrote:
>
>> If we did return the value of shell scripts then ":results value" would
>> almost always simply return 0 (or possibly an error message). For this
>> reason shell code blocks do not implement value returns, but rather will
>> always collect results from STDOUT.
>
> I think that this unnecessarily throws away potentially useful
> functionality. Example: I want to fill a table with data such that the value
> of a cell depends on whether a file (whose path is specified by another
> cell, for instance) exists or not. This would be most easily done using an
> sh block which returns a numeric exit code. I don't see a reason for making
> clandestine exceptions to the rules in the manual and strongly suggest that
> the output and value options be honoured for every language.

I must admit that I was not aware of this peculiarity for sh blocks. And I
guess there must be a reason for not honoring value vs output here. If not,
I'd prefer it to exist -- even if Achim showed that both are always possible.

> In order not to break existing Org files, I would suggest that the default
> choice between value and output (when not explicitly specified) depend on
> the language. With this functionality, sh code blocks that don't specify
> ":results output" will still work as they did before.

That possibility already exist. It can be done like this:

#+begin_src emacs-lisp
  (add-to-list 'org-babel-default-header-args:sh '((:results . "value")))
#+end_src

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[Sebastien Vauban]

Hi Achim,

Achim Gratz wrote:
> András Major <andras.g.major-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:
>> This would be most easily done using an sh block which returns a
>> numeric exit code.
>
> No one keeps you from using
>
> echo $?
>
> in the last line of that shell block.

Even if this is a smart workaround, it don't think this is equivalent to
outputting "value" as, here, you'd get the value printed at the end of the
call, but as well all what has been sent to stdout during the execution of the
block.

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[Eric Schulte]

András Major <andras.g.major@gmail.com> writes:

> Hi Eric,
>
>> If we did return the value of shell scripts then ":results value" would
>> almost always simply return 0 (or possibly an error message).  For this
>> reason shell code blocks do not implement value returns, but rather will
>> always collect results from STDOUT.
>
> I think that this unnecessarily throws away potentially useful
> functionality.  Example: I want to fill a table with data such that
> the value of a cell depends on whether a file (whose path is specified
> by another cell, for instance) exists or not.  This would be most
> easily done using an sh block which returns a numeric exit code.  I
> don't see a reason for making clandestine exceptions to the rules in
> the manual and strongly suggest that the output and value options be
> honoured for every language.
>

I do see your point, and I agree that consistent behavior between
languages is of paramount importance.  In fact I began working on
implementing the return of error codes from shell code blocks, however I
ran into the following issue.

For every language, when an error is thrown babel tries to open an error
buffer holding the contents of the error message.  This is very useful
for debugging code which lives inside of a code block -- a process which
can otherwise be difficult because of the extra layer of indirection
babel interposes between the programmer and the codes execution.  In
order to return exit codes from shell blocks babel would have to
silently ignore errors in shell blocks.  I would lean towards thinking
that passing along error messages is more important than returning error
codes, but if the community thinks differently I'm happy to change the
ob-sh behavior.

Unfortunately it seems that in either case the sh code blocks will need
to be different than other languages either in its handling of errors or
of return values.  This is unavoidable due to the overloading of return
values in the shell as error indicators.

>
> In order not to break existing Org files, I would suggest that the
> default choice between value and output (when not explicitly
> specified) depend on the language.  With this functionality, sh code
> blocks that don't specify ":results output" will still work as they
> did before.
>

I agree, if we go this route this is the way to do it.

Best -- Eric

>
>   András
>
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[Michael Brand]

Hi Eric

2011/8/20 Eric Schulte <schulte.eric@gmail.com>:
> [...] I would lean towards thinking
> that passing along error messages is more important than returning error
> codes, but if the community thinks differently I'm happy to change the
> ob-sh behavior.

A non-zero exit status and stderr of a process are not necessarily
related. Because a process may also use
- a non-zero exit status without error situation (e. g. grep, diff)
- stderr for output not related with errors
- stdout for error messages
I would like very much to be able to collect all available feedback
from a process at the same run. Even with an optional indication of
the origin, for ambiguity like the "hello" below or just for clarification.

> Unfortunately it seems that in either case the sh code blocks will need
> to be different than other languages either in its handling of errors or
> of return values.  This is unavoidable due to the overloading of return
> values in the shell as error indicators.

If the shell is a special case for babel anyway, why not something
like the following?

#+begin_src sh :exports stdout stderr exit_status -v
  echo hello
  echo hello >&2
  false
#+end_src

#+results:
: 2: hello
: 1: hello
: exit status: 1

This would have been
- with an option -v for verbosity to prefix
  stdout with "1: ", stderr with "2: " and the exit status
- with the exit status of the last command without the need of an
  extra "echo $?".

My habit as a background info: To learn more from the shell I use
- a shell prompt with the exit status of the last command
- when I sometimes want to visually divide stdout and stderr (bash):
  { { echo hello; echo hello >&2; } 3>&1 1>&2 2>&3 | sed 's/^/2: /'; } \
      3>&1 1>&2 2>&3 | sed 's/^/1: /'; 3>&-
  to output:
  2: hello
  1: hello

Michael

Re: Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[Achim Gratz]

Michael Brand <michael.ch.brand@gmail.com> writes:
> If the shell is a special case for babel anyway, why not something
> like the following?

Ehm, no.  But I think that it would be generally useful (not just for
shell blocks) to be able to capture stderr, either together with stdout
or separately into a result target block and have the return status
available as a variable (although that does not work for consecutive
invocations in a session).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[Eric Schulte]

Achim Gratz <Stromeko@nexgo.de> writes:

> Michael Brand <michael.ch.brand@gmail.com> writes:
>> If the shell is a special case for babel anyway, why not something
>> like the following?
>
> Ehm, no.  But I think that it would be generally useful (not just for
> shell blocks) to be able to capture stderr, either together with stdout
> or separately into a result target block and have the return status
> available as a variable (although that does not work for consecutive
> invocations in a session).
>

Hmm, I do think that a ":results stderr" option could be useful, however
this would not be a trivial implementation as it may require adjustments
to all of the language files.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[Eric Schulte]

Michael Brand <michael.ch.brand@gmail.com> writes:

> Hi Eric
>
> 2011/8/20 Eric Schulte <schulte.eric@gmail.com>:
>> [...] I would lean towards thinking
>> that passing along error messages is more important than returning error
>> codes, but if the community thinks differently I'm happy to change the
>> ob-sh behavior.
>
> A non-zero exit status and stderr of a process are not necessarily
> related. Because a process may also use
> - a non-zero exit status without error situation (e. g. grep, diff)
> - stderr for output not related with errors
> - stdout for error messages
> I would like very much to be able to collect all available feedback
> from a process at the same run. Even with an optional indication of
> the origin, for ambiguity like the "hello" below or just for clarification.
>

I agree that some mechanism for collecting output from STDERR could be
useful, however its implementation would not be trivial.

>
>> Unfortunately it seems that in either case the sh code blocks will need
>> to be different than other languages either in its handling of errors or
>> of return values.  This is unavoidable due to the overloading of return
>> values in the shell as error indicators.
>
> If the shell is a special case for babel anyway, why not something
> like the following?
>
> #+begin_src sh :exports stdout stderr exit_status -v
>   echo hello
>   echo hello >&2
>   false
> #+end_src
>
> #+results:
> : 2: hello
> : 1: hello
> : exit status: 1
>
> This would have been
> - with an option -v for verbosity to prefix
>   stdout with "1: ", stderr with "2: " and the exit status
> - with the exit status of the last command without the need of an
>   extra "echo $?".
>
> My habit as a background info: To learn more from the shell I use
> - a shell prompt with the exit status of the last command
> - when I sometimes want to visually divide stdout and stderr (bash):
>   { { echo hello; echo hello >&2; } 3>&1 1>&2 2>&3 | sed 's/^/2: /'; } \
>       3>&1 1>&2 2>&3 | sed 's/^/1: /'; 3>&-
>   to output:
>   2: hello
>   1: hello
>

I think that in general mixing the output status with STDOUT would not
be a clean solution, as it would require parsing to use.  Also, I don't
think that the example above would help to bring the behavior of sh code
blocks more in-line with other code blocks.

Note that during interactive evaluation if the exit status is non-0 then
STDERR will be dumped into a babel error buffer which will be poped up,
so this information will not be silently discarded.

Best -- Eric

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: Bug: babel: results switch (output vs. value) has no or wrong effect for sh source block [7.7 (release_7.7.107.g7a82)]

[Michael Brand]

Hi Eric

On Tue, Aug 23, 2011 at 19:13, Eric Schulte <schulte.eric@gmail.com> wrote:
> Note that during interactive evaluation if the exit status is non-0 then
> STDERR will be dumped into a babel error buffer which will be poped up,
> so this information will not be silently discarded.

Thanks for emphasizing this. I didn't use code block evaluation much
for ob-sh and haven't yet noticed this rule with the last exit status
that seems very important to know.

Two things I would like to comment referring to the examples 1) to 5)
below:
a) _completeness_ of the process output:
   - When the last exit status is 0, babel silently discards possible
     important information from stderr, see mainly 1), also 2) to 3).
   - When the last exit status is >0, babel pops up the "Org-Babel
     Error Output" and silently discards possible important
     information from stdout, see 4).
b) _decision_ whether the process succeeded or failed:
   - This is made by discriminating the last exit status 0 or not. But
     there are even standard processes with an exit status and stderr
     not simple, see 4) to 5) ("simple": 0 => success and output only
     on stdout, >0 => failure and output only on stderr).

My suggestions for changes (I have understood that this would not be
easy to do, babel grew differently):
a) _completeness_ of the process output:
   - When the last exit status is 0: Include stderr in "#+results:" by
     default. In the minority of use cases where the behavior changes
     I think this is most of the time an improvement.
   - When the last exit status is >0 and babel pops up the "Org-Babel
     Error Output": Include stdout there by default. In the minority
     of use cases where the behavior changes I think this is always an
     improvement.
b) _decision_ whether the process succeeded or failed:
   - My point of view is that babel code block evaluation is not
     always able to make such a decision for two reasons, at least in
     the case of the languages awk, sh and shell: Firstly not
     universally because it would require to consider which process it
     runs, e. g. the exit status 1 means success for diff or grep but
     >0 means failure for most other processes. Secondly not reliably
     because it would require to consider for which purpose this
     process is used during this evaluation. When one wants to
     document the output of a failure then "Org-Babel Error Output"
     does not come in handy.
   - An option to move this decision to the user would help. For this
     I wish to have an additional and somehow more manual operation
     mode that bypasses the interpretation of the last exit status and
     omits "Org-Babel Error Output". Achim already phrased it in this
     thread as:
     - "I think that it would be generally useful (not just for shell
       blocks) to be able to capture stderr, either together with
       stdout or separately into a result target block and have the
       return status available as a variable."
     - I would only add: It would be nice to have both variants of
       "either or" available to choose from. The first for better
       human readability and the second to read stdout without stderr
       into the next chained code block again.

----------------------------------------------------------------

Comments and questions for the examples and workarounds:

Examples 4) to 5) end up with an empty "Org-Babel Error Output" and
"#+results:" is emptied.
- Since they are both empty it is not necessarily clear what happened.
- The Messages buffer says "Babel evaluation exited with code 1" and
  "Code block produced no output.". The second statement is not
  necessarily clear:
  - Is this an _info_ or was output expected and this is an _error_
    that even stopped babel from doing more work?
    - Is the message "Code block evaluation complete." from other
      cases an indication whether babel finished all its work? If this
      is an indication, isn't this message missing in the following
      case?
        #+begin_src sh :results output
          true
        #+end_src

        #+results:

  - Which "output" was meant to be missing?
    - Output missing on stdout (example 5) or was stdout not
      considered due to last exit status >0 (example 4)?
    - Output missing on stderr?

At the end are the workarounds I will use for now for the examples 1)
to 5). Shown is the complete variant when I need to know the last exit
status.
- 2>&1 :: collect stderr
- $?   :: last exit status
- true :: omit any "Org-Babel Error Output"

----------------------------------------------------------------

Examples:

1) command(s) before the last command have error
   - non-existent file myfile, output mixed on stdout and stderr
     - in a terminal (not babel code block evaluation):
       - shell command:
         #+begin_example
           ls myfile
           echo b
         #+end_example

       - output (stderr: ls, stdout: echo, exit status ls: 2, exit
         status echo: 0):
         #+begin_example
           ls: myfile: No such file or directory
           b
         #+end_example

     - with babel:
       #+begin_src sh :results output
         ls myfile
         echo b
       #+end_src

       #+results:
       : b

2) strace (Linux)
   - output on stderr
     - in a terminal (not babel code block evaluation):
       - shell command:
         #+begin_example
           strace true
         #+end_example

       - output (stdout: empty, stderr: syscalls, exit status: 0):
         #+begin_example
           execve("/bin/true", ["true"], [/* 54 vars */]) = 0
           [...]
           exit_group(0)                           = ?
         #+end_example

     - with babel:
       #+begin_src sh :results output
         strace true
       #+end_src

       #+results:

3) truss (Solaris)
   - output on stderr
     - in a terminal (not babel code block evaluation):
       - shell command:
         #+begin_example
           truss true
         #+end_example

       - output (stdout: empty, stderr: syscalls, exit status: 0):
         #+begin_example
           execve("/usr/bin/true", 0xFFBFF0FC, 0xFFBFF104)  argc = 1
           [...]
           _exit(0)
         #+end_example

     - with babel:
       #+begin_src sh :results output
         truss true
       #+end_src

       #+results:

4) diff
   - the exit status as standardized by POSIX/SUS
     http://pubs.opengroup.org/onlinepubs/009695399/utilities/diff.html
     - 0  :: No differences were found.
     - 1  :: Differences were found.
     - >1 :: An error occurred.

   - example with two differing files
     #+begin_src sh :results silent
       echo a > /tmp/a
       echo b > /tmp/b
     #+end_src

     - in a terminal (not babel code block evaluation):
       - shell command:
         #+begin_example
           diff /tmp/a /tmp/b
         #+end_example

       - output (stdout: differences, stderr: empty, exit status: 1):
         #+begin_example
           1c1
           < a
           ---
           > b
         #+end_example

     - with babel:
       #+begin_src sh :results output
         diff /tmp/a /tmp/b
       #+end_src

       #+results:

5) grep
   - the exit status as standardized by POSIX/SUS
     http://pubs.opengroup.org/onlinepubs/009695399/utilities/grep.html
     - 0  :: One or more lines were selected.
     - 1  :: No lines were selected.
     - >1 :: An error occurred.

   - example:
     - in a terminal (not babel code block evaluation):
       - shell command:
         #+begin_example
           echo a | grep b
         #+end_example

       - output (stdout: empty, stderr: empty, exit status: 1):
         #+begin_example
         #+end_example

     - with babel:
       #+begin_src sh :results output
         echo a | grep b
       #+end_src

       #+results:

----------------------------------------------------------------

Workarounds:

1) command(s) before the last command have error
   #+begin_src sh :results output
     ls myfile 2>&1
     echo b 2>&1
     echo "last exit status: $?" 2>&1
     true 2>&1
   #+end_src

   #+results:
   : ls: myfile: No such file or directory
   : b
   : last exit status: 0

2) strace (Linux)
   #+begin_src sh :results output
     strace true b 2>&1
     echo "last exit status: $?" 2>&1
     true 2>&1
   #+end_src

   #+results:
   : execve("/bin/true", ["true", "b"], [/* 104 vars */]) = 0
   : [...]
   : exit_group(0)                           = ?
   : last exit status: 0

3) truss (Solaris)
   #+begin_src sh :results output
     truss true b 2>&1
     echo "last exit status: $?" 2>&1
     true 2>&1
   #+end_src

   #+results:
   : execve("/usr/bin/true", 0xFFBFF0FC, 0xFFBFF104)  argc = 1
   : [...]
   : _exit(0)
   : last exit status: 0

4) diff
   #+begin_src sh :results output
     diff /tmp/a /tmp/b 2>&1
     echo "last exit status: $?" 2>&1
     true 2>&1
   #+end_src

   #+results:
   : 1c1
   : < a
   : ---
   : > b
   : last exit status: 1

5) grep
   #+begin_src sh :results output
     echo a | grep b 2>&1
     echo "last exit status: $?" 2>&1
     true 2>&1
   #+end_src

   #+results:
   : last exit status: 1

----------------------------------------------------------------

Michael