Documentation of hline symbol in source blocks results

Documentation of hline symbol in source blocks results

[Tim Landscheidt]

Hi,

this is just a data point for anyone looking at restructur-
ing the Org documentation.

I wanted an Emacs Lisp source block to produce a table with
a column name and a horizontal line separating the column
name and the data cells.  The data-only source block:

| #+BEGIN_SRC emacs-lisp
|   '((1)
|     (2)
|     (3))
| #+END_SRC

produces the result:

| #+RESULTS:
| | 1 |
| | 2 |
| | 3 |

I wanted to get the result:

| #+RESULTS:
| | Number |
| |--------|
| |      1 |
| |      2 |
| |      3 |

It took me quite a while to figure out that the first row
returned from the source block can be viewed as the column
name, and a horizontal line can be achieved by returning the
symbol 'hline:

| #+BEGIN_SRC emacs-lisp
|   '(("Number")
|     hline
|     (1)
|     (2)
|     (3))
| #+END_SRC

The only reference to this behaviour I could find, is liter-
ally the last (!, :-)) paragraph in the info file:

|    For complicated translations the generic translator function could be
| replaced by a custom translator function.  Such a custom function must
| take two arguments and return a single string containing the formatted
| table.  The first argument is the table whose lines are a list of fields
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| or the symbol ‘hline’.  The second argument is the property list
  ^^^^^^^^^^^^^^^^^^^^^
| consisting of parameters specified in the ‘#+ORGTBL: SEND’ line.  Please
| share your translator functions by posting them to the Org users mailing
| list, at <emacs-orgmode@gnu.org>.

Just for further inspiration how the documentation could be
rewritten, consider the current wording of "Results of Eval-
uation/Collection/value":

|      Default for most Babel libraries(1).  Functional mode.  Org gets
|      the value by wrapping the code in a function definition in the
|      language of the source block.  That is why when using ‘:results
|      value’, code should execute like a function and return a value.
|      For languages like Python, an explicit ‘return’ statement is
|      mandatory when using ‘:results value’.  Result is the value
|      returned by the last statement in the code block.

|      When evaluating the code block in a session (see *note Environment
|      of a Code Block::), Org passes the code to an interpreter running
|      as an interactive Emacs inferior process.  Org gets the value from
|      the source code interpreter’s last statement output.  Org has to
|      use language-specific methods to obtain the value.  For example,
|      from the variable ‘_’ in Ruby, and the value of ‘.Last.value’ in R.

Wrapping the code?  "Code /should/"?  "Like a function"?
Why is the Python requirement stated here?  Why "using
‘:results value’" when the paragraph should (only) document
this?  "Result is the value"?  What kind of value?  Why are
there references to Ruby and R here?  All this confuses me
and does not provide the information I searched for (empha-
sis on me).

I would probably prefer a clean-slate approach that starts
with something along the lines of: "Source blocks produce
results that can be integrated into an Org document and used
as input for other source blocks.  The abstract specifica-
tion for source blocks looks like this ("structured table
data" vs. "text dump", header arguments, export, etc.).
Emacs Lisp source blocks produce these values in this way.
Python that way.  Shell scripts return tables in this way."

Org is extremely powerful, but (I think I wrote it somewhere
before) the documentation reads like users having had an is-
sue ("my Python code does not produce (correct) results")
and then someone feeling the need to document the solution
right then and there.  IMNSHO this leads to documentation
that is not very usable for the general audience.

Tim

Re: Documentation of hline symbol in source blocks results

[Ihor Radchenko]

Tim Landscheidt <tim@tim-landscheidt.de> writes:

> this is just a data point for anyone looking at restructur-
> ing the Org documentation.
>
> I wanted an Emacs Lisp source block to produce a table with
> a column name and a horizontal line separating the column
> name and the data cells.  The data-only source block:
> ...
> I wanted to get the result:
>
> | #+RESULTS:
> | | Number |
> | |--------|
> | |      1 |
> | |      2 |
> | |      3 |
>
> It took me quite a while to figure out that the first row
> returned from the source block can be viewed as the column
> name, and a horizontal line can be achieved by returning the
> symbol 'hline:

AFAIK, producing tables with column names in
emacs-lisp code blocks is not officially supported.
The `hline' is not documented.

> The only reference to this behaviour I could find, is liter-
> ally the last (!, :-)) paragraph in the info file:
>
> |    For complicated translations the generic translator function could be
> | replaced by a custom translator function.  Such a custom function must
> | take two arguments and return a single string containing the formatted
> | table.  The first argument is the table whose lines are a list of fields
>                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> | or the symbol ‘hline’.

This happens to be the machinery Org babel uses to convert tables from
Elisp format to Org mode format. We do not give you any guarantee that
we are going to keep using `orgtbl-to-generic' in future the same way.

> Just for further inspiration how the documentation could be
> rewritten, consider the current wording of "Results of Eval-
> uation/Collection/value":
>
> |      Default for most Babel libraries(1).  Functional mode.  Org gets
> |      the value by wrapping the code in a function definition in the
> |      language of the source block.  That is why when using ‘:results
> |      value’, code should execute like a function and return a value.
> |      For languages like Python, an explicit ‘return’ statement is
> |      mandatory when using ‘:results value’.  Result is the value
> |      returned by the last statement in the code block.
>
> |      When evaluating the code block in a session (see *note Environment
> |      of a Code Block::), Org passes the code to an interpreter running
> |      as an interactive Emacs inferior process.  Org gets the value from
> |      the source code interpreter’s last statement output.  Org has to
> |      use language-specific methods to obtain the value.  For example,
> |      from the variable ‘_’ in Ruby, and the value of ‘.Last.value’ in R.
>
> Wrapping the code?  "Code /should/"?  "Like a function"?
> Why is the Python requirement stated here?

I am sorry, but I do not fully understand the problem.
What we are saying is that the code block should return a value.
And give an example that python blocks, in order to return a value,
should have `return' statement.

> ... Why "using
> ‘:results value’" when the paragraph should (only) document
> this?

Emm.. But we are documenting :results value is this paragraph.
Mentioning it again is a bit redundant, but not the end of the world.
Improvements welcome, of course.

> ... "Result is the value"?  What kind of value?  Why are
> there references to Ruby and R here?  All this confuses me
> and does not provide the information I searched for (empha-
> sis on me).

 Ruby and R have their own syntax to return value, just like "return" in
 python.
 
I hope it is clear now for you.

> I would probably prefer a clean-slate approach that starts
> with something along the lines of: "Source blocks produce
> results that can be integrated into an Org document and used
> as input for other source blocks.  The abstract specifica-
> tion for source blocks looks like this ("structured table
> data" vs. "text dump", header arguments, export, etc.).
> Emacs Lisp source blocks produce these values in this way.
> Python that way.  Shell scripts return tables in this way."
>
> Org is extremely powerful, but (I think I wrote it somewhere
> before) the documentation reads like users having had an is-
> sue ("my Python code does not produce (correct) results")
> and then someone feeling the need to document the solution
> right then and there.  IMNSHO this leads to documentation
> that is not very usable for the general audience.

AFAIK, the particular paragraph you pointed out does not mention Python
just to address someone issue. Python/R/Ruby are just examples used.

Though I do agree that this (and many other) sections of the manual can
be improved. Patches welcome!

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Documentation Results of Evaluation (was Re: Documentation of hline symbol in source blocks results)

[Matt]

Thanks for your message, Tim!  You make many good points and I can
only respond to a few of them.

I'm also pleased to see you've got FSF Assignment:
https://orgmode.org/worg/contributors.html

Do you have time to work together on this?

I have four goals:
1. Understand your perspective
2. Remove confusion
3. Detail specific problems to solve
4. Suggest, or commit, specific changes to the manual which resolve
the specific problems

This email focuses on the first three.  I'll first respond to you, try
to understand your perspective, remove confusion, and highlight
specific problems to solve.  I'll then give some comments on the
specific problems.  My hope is that after several exchanges, we'll
have a list of specific problems that the mailing list can then
discuss for solutions.

 ---- On Mon, 25 Dec 2023 17:37:41 +0100  Tim Landscheidt  wrote ---

 > the documentation reads like users having had an issue...and then
 > someone feeling the need to document the solution right then and
 > there.

What you say likely happened.

 > IMNSHO this leads to documentation that is not very usable
 > for the general audience.

Agreed!

 > Just for further inspiration how the documentation could be
 > rewritten, consider the current wording of "Results of Eval-
 > uation/Collection/value":
 >
 > |      Default for most Babel libraries(1).  Functional mode.  Org gets
 > |      the value by wrapping the code in a function definition in the
 > |      language of the source block.  That is why when using ‘:results
 > |      value’, code should execute like a function and return a value.
 > |      For languages like Python, an explicit ‘return’ statement is
 > |      mandatory when using ‘:results value’.  Result is the value
 > |      returned by the last statement in the code block.
 >
 > |      When evaluating the code block in a session (see *note Environment
 > |      of a Code Block::), Org passes the code to an interpreter running
 > |      as an interactive Emacs inferior process.  Org gets the value from
 > |      the source code interpreter’s last statement output.  Org has to
 > |      use language-specific methods to obtain the value.  For example,
 > |      from the variable ‘_’ in Ruby, and the value of ‘.Last.value’ in R.
 >
 > Wrapping the code?

When you write

#+begin_src C
printf("hello, world!\n");
#+end_src

The code that's executed is actually:

int main() {
printf("hello, world!\n");
return 0;
}

When you write

#+begin_src python
print("hello, world!")
#+end_src

The code that's executed is actually (brace for it...):

def __org_babel_python_format_value(result, result_file, result_params):
    with open(result_file, 'w') as f:
        if 'graphics' in result_params:
            result.savefig(result_file)
        elif 'pp' in result_params:
            import pprint
            f.write(pprint.pformat(result))
        elif 'list' in result_params and isinstance(result, dict):
            f.write(str(['{} :: {}'.format(k, v) for k, v in result.items()]))
        else:
            if not set(result_params).intersection(['scalar', 'verbatim', 'raw']):
                def dict2table(res):
                    if isinstance(res, dict):
                        return [(k, dict2table(v)) for k, v in res.items()]
                    elif isinstance(res, list) or isinstance(res, tuple):
                        return [dict2table(x) for x in res]
                    else:
                        return res
                if 'table' in result_params:
                    result = dict2table(result)
                try:
                    import pandas
                except ImportError:
                    pass
                else:
                    if isinstance(result, pandas.DataFrame) and 'table' in result_params:
                        result = [[result.index.name or ''] + list(result.columns)] + [None] + [[i] + list(row) for i, row in result.iterrows()]
                    elif isinstance(result, pandas.Series) and 'table' in result_params:
                        result = list(result.items())
                try:
                    import numpy
                except ImportError:
                    pass
                else:
                    if isinstance(result, numpy.ndarray):
                        if 'table' in result_params:
                            result = result.tolist()
                        else:
                            result = repr(result)
            f.write(str(result))
def main():
    print("hello, world!")

__org_babel_python_format_value(main(), '/tmp/babel-7Zq1c7/python-e7IyhX', ["replace"])

However, not every language is wrapped.

For bash, when you write

#+begin_src bash
echo "hello, world!"
#+end_src

Something like this is passed to a bash process:

echo "hello, world!"

PROBLEM: "wrapping" is inaccurate

 > "Code /should/"?

I agree, this can improve.  "Should" in this context reads as
"probably" and is non-committal.  Documentation has a duty to be
authoritative.

PROBLEM: "code should execute like a function" is non-committal

 > "Like a function"?

I agree that this is unclear.

I believe what it's trying to address is that it's possible to have
multiple return values from a source block.

For example, Unix shell commands return an exit code indicating
success, failure, and failure type.  They may also write text to
stdout and stderr.  The header arguments ":results value" and
":results output" distinguish these two types of return values.

PROBLEM: terms are not well-defined

 > Why is the Python requirement stated here?

I see several reasons.  The simplest is the piecemeal development
style already mentioned.

PROBLEM: non-specific requirement reference.  What other languages are "like Python"?

 > Why "using ‘:results value’" when the paragraph should (only)
 > document this?

Do I understand correctly that you're saying "using ':results value'"
is redundant?

PROBLEM: redundant words

> "Result is the value"?

What's the problem you see with this?

PROBLEM: change of voice
PROBLEM: singular form used to reference a plural (should we say "result" or "results")

> What kind of value?

What do you mean by "kind"?

> Why are there references to Ruby and R here?

At risk of being pedantic, because of the previous sentence: "Org has
to use language-specific methods to obtain the value."

I'm having difficulty seeing your perspective.

Can you please share more of your thoughts about the confusion for
this sentence?

 > All this confuses me and does not provide the information I
 > searched for (emphasis on me).

I'm sorry for your confusion.  I've been there.  Few things make my
blood boil quite as much as bad documentation.

My understanding is that the information you searched for is the
answer to these questions:

- "How to make a named column table result for a source block?"
- "How to insert a horizontal line between two rows of a table result
   for a source block?"

Is that correct?

If so, I think Ihor addressed this already by saying that, unfortunately,
this functionality isn't currently available (although contributions
are welcome).

 > I would probably prefer a clean-slate approach that starts
 > with something along the lines of: "Source blocks produce
 > results that can be integrated into an Org document and used
 > as input for other source blocks.

This is covered in the "Feature Overview":
https://www.gnu.org/software/emacs/manual/html_node/org/Features-Overview.html

That's not to say that the manual is clear.  Maybe including it at the
very start (in the "Working with Source Code" section) would have
helped? https://www.gnu.org/software/emacs/manual/html_node/org/Working-with-Source-Code.html

-----

A specific section of manual was referenced, "Results of Evaluation":

- https://www.gnu.org/software/emacs/manual/html_node/org/Results-of-Evaluation.html
- https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/org-manual.org#n18460

The specific problems I've identified:

PROBLEM: "wrapping" is inaccurate
PROBLEM: "code should execute like a function" is non-committal
PROBLEM: terms are not well-defined

Different language has been used over the past 10+ years to describe
behavior of source blocks.

The "Results of Evaluation" section mentions "functional mode" and
"scripting mode". Collaboration notes (org-babel.org) were kept in the
early days of Babel.  Eric (the original author of Org Babel) made a
distinction between (what he calls) "functional" and "imperative."
It's not clear to me what's meant by these terms.

Some snapshots of org-babel.org:
- https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/contrib/babel/org-babel.org?id=c0554c775344207e6adaf191258312fb8d1b6a15
- https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/org-babel.org?id=b1c103890c1523f99e380d88ed684454d902414e

Here's how I think of it:

"Session" means an environment is "persistent."  Each call is executed
in the same environment.  State exists between calls.  In the
early-history of Babel, this was called the "imperative" style.

"Non-session" means an environment is "temporary."  Each call is
executed in an independent environment.  State does not exist between
calls.  In the early-history of Babel, this was called the
"functional" style.

All of this relates to the ":results value" and ":results output"
header arguments.  This is not the first time this has been discussed:
https://list.orgmode.org/orgmode/CA+A2iZaziAfMeGpBqL6qGrzrWEVvLvC0DUw++T4gCF3NGuW-DQ@mail.gmail.com/

Before discussing too much, maybe it would help to get a list of
related terms that need clarification?

- "imperative"
- "functional"
- "session"
- "scripting"

PROBLEM: non-specific requirement reference.  What other languages are "like Python"?
PROBLEM: redundant words
PROBLEM: change of voice
PROBLEM: singular form used to reference a plural (should we say "result" or "results")

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode