Documentation Results of Evaluation (was Re: Documentation of hline symbol in source blocks results)

[Matt <matt@excalamus.com>]

Thanks for your message, Tim!  You make many good points and I can
only respond to a few of them.

I'm also pleased to see you've got FSF Assignment:
https://orgmode.org/worg/contributors.html

Do you have time to work together on this?

I have four goals:
1. Understand your perspective
2. Remove confusion
3. Detail specific problems to solve
4. Suggest, or commit, specific changes to the manual which resolve
the specific problems

This email focuses on the first three.  I'll first respond to you, try
to understand your perspective, remove confusion, and highlight
specific problems to solve.  I'll then give some comments on the
specific problems.  My hope is that after several exchanges, we'll
have a list of specific problems that the mailing list can then
discuss for solutions.

 ---- On Mon, 25 Dec 2023 17:37:41 +0100  Tim Landscheidt  wrote ---

 > the documentation reads like users having had an issue...and then
 > someone feeling the need to document the solution right then and
 > there.

What you say likely happened.

 > IMNSHO this leads to documentation that is not very usable
 > for the general audience.

Agreed!

 > Just for further inspiration how the documentation could be
 > rewritten, consider the current wording of "Results of Eval-
 > uation/Collection/value":
 >
 > |      Default for most Babel libraries(1).  Functional mode.  Org gets
 > |      the value by wrapping the code in a function definition in the
 > |      language of the source block.  That is why when using ‘:results
 > |      value’, code should execute like a function and return a value.
 > |      For languages like Python, an explicit ‘return’ statement is
 > |      mandatory when using ‘:results value’.  Result is the value
 > |      returned by the last statement in the code block.
 >
 > |      When evaluating the code block in a session (see *note Environment
 > |      of a Code Block::), Org passes the code to an interpreter running
 > |      as an interactive Emacs inferior process.  Org gets the value from
 > |      the source code interpreter’s last statement output.  Org has to
 > |      use language-specific methods to obtain the value.  For example,
 > |      from the variable ‘_’ in Ruby, and the value of ‘.Last.value’ in R.
 >
 > Wrapping the code?

When you write

#+begin_src C
printf("hello, world!\n");
#+end_src

The code that's executed is actually:

int main() {
printf("hello, world!\n");
return 0;
}

When you write

#+begin_src python
print("hello, world!")
#+end_src

The code that's executed is actually (brace for it...):

def __org_babel_python_format_value(result, result_file, result_params):
    with open(result_file, 'w') as f:
        if 'graphics' in result_params:
            result.savefig(result_file)
        elif 'pp' in result_params:
            import pprint
            f.write(pprint.pformat(result))
        elif 'list' in result_params and isinstance(result, dict):
            f.write(str(['{} :: {}'.format(k, v) for k, v in result.items()]))
        else:
            if not set(result_params).intersection(['scalar', 'verbatim', 'raw']):
                def dict2table(res):
                    if isinstance(res, dict):
                        return [(k, dict2table(v)) for k, v in res.items()]
                    elif isinstance(res, list) or isinstance(res, tuple):
                        return [dict2table(x) for x in res]
                    else:
                        return res
                if 'table' in result_params:
                    result = dict2table(result)
                try:
                    import pandas
                except ImportError:
                    pass
                else:
                    if isinstance(result, pandas.DataFrame) and 'table' in result_params:
                        result = [[result.index.name or ''] + list(result.columns)] + [None] + [[i] + list(row) for i, row in result.iterrows()]
                    elif isinstance(result, pandas.Series) and 'table' in result_params:
                        result = list(result.items())
                try:
                    import numpy
                except ImportError:
                    pass
                else:
                    if isinstance(result, numpy.ndarray):
                        if 'table' in result_params:
                            result = result.tolist()
                        else:
                            result = repr(result)
            f.write(str(result))
def main():
    print("hello, world!")

__org_babel_python_format_value(main(), '/tmp/babel-7Zq1c7/python-e7IyhX', ["replace"])

However, not every language is wrapped.

For bash, when you write

#+begin_src bash
echo "hello, world!"
#+end_src

Something like this is passed to a bash process:

echo "hello, world!"

PROBLEM: "wrapping" is inaccurate

 > "Code /should/"?

I agree, this can improve.  "Should" in this context reads as
"probably" and is non-committal.  Documentation has a duty to be
authoritative.

PROBLEM: "code should execute like a function" is non-committal

 > "Like a function"?

I agree that this is unclear.

I believe what it's trying to address is that it's possible to have
multiple return values from a source block.

For example, Unix shell commands return an exit code indicating
success, failure, and failure type.  They may also write text to
stdout and stderr.  The header arguments ":results value" and
":results output" distinguish these two types of return values.

PROBLEM: terms are not well-defined

 > Why is the Python requirement stated here?

I see several reasons.  The simplest is the piecemeal development
style already mentioned.

PROBLEM: non-specific requirement reference.  What other languages are "like Python"?

 > Why "using ‘:results value’" when the paragraph should (only)
 > document this?

Do I understand correctly that you're saying "using ':results value'"
is redundant?

PROBLEM: redundant words

> "Result is the value"?

What's the problem you see with this?

PROBLEM: change of voice
PROBLEM: singular form used to reference a plural (should we say "result" or "results")

> What kind of value?

What do you mean by "kind"?

> Why are there references to Ruby and R here?

At risk of being pedantic, because of the previous sentence: "Org has
to use language-specific methods to obtain the value."

I'm having difficulty seeing your perspective.

Can you please share more of your thoughts about the confusion for
this sentence?

 > All this confuses me and does not provide the information I
 > searched for (emphasis on me).

I'm sorry for your confusion.  I've been there.  Few things make my
blood boil quite as much as bad documentation.

My understanding is that the information you searched for is the
answer to these questions:

- "How to make a named column table result for a source block?"
- "How to insert a horizontal line between two rows of a table result
   for a source block?"

Is that correct?

If so, I think Ihor addressed this already by saying that, unfortunately,
this functionality isn't currently available (although contributions
are welcome).

 > I would probably prefer a clean-slate approach that starts
 > with something along the lines of: "Source blocks produce
 > results that can be integrated into an Org document and used
 > as input for other source blocks.

This is covered in the "Feature Overview":
https://www.gnu.org/software/emacs/manual/html_node/org/Features-Overview.html

That's not to say that the manual is clear.  Maybe including it at the
very start (in the "Working with Source Code" section) would have
helped? https://www.gnu.org/software/emacs/manual/html_node/org/Working-with-Source-Code.html

-----

A specific section of manual was referenced, "Results of Evaluation":

- https://www.gnu.org/software/emacs/manual/html_node/org/Results-of-Evaluation.html
- https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/org-manual.org#n18460

The specific problems I've identified:

PROBLEM: "wrapping" is inaccurate
PROBLEM: "code should execute like a function" is non-committal
PROBLEM: terms are not well-defined

Different language has been used over the past 10+ years to describe
behavior of source blocks.

The "Results of Evaluation" section mentions "functional mode" and
"scripting mode". Collaboration notes (org-babel.org) were kept in the
early days of Babel.  Eric (the original author of Org Babel) made a
distinction between (what he calls) "functional" and "imperative."
It's not clear to me what's meant by these terms.

Some snapshots of org-babel.org:
- https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/contrib/babel/org-babel.org?id=c0554c775344207e6adaf191258312fb8d1b6a15
- https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/org-babel.org?id=b1c103890c1523f99e380d88ed684454d902414e

Here's how I think of it:

"Session" means an environment is "persistent."  Each call is executed
in the same environment.  State exists between calls.  In the
early-history of Babel, this was called the "imperative" style.

"Non-session" means an environment is "temporary."  Each call is
executed in an independent environment.  State does not exist between
calls.  In the early-history of Babel, this was called the
"functional" style.

All of this relates to the ":results value" and ":results output"
header arguments.  This is not the first time this has been discussed:
https://list.orgmode.org/orgmode/CA+A2iZaziAfMeGpBqL6qGrzrWEVvLvC0DUw++T4gCF3NGuW-DQ@mail.gmail.com/

Before discussing too much, maybe it would help to get a list of
related terms that need clarification?

- "imperative"
- "functional"
- "session"
- "scripting"

PROBLEM: non-specific requirement reference.  What other languages are "like Python"?
PROBLEM: redundant words
PROBLEM: change of voice
PROBLEM: singular form used to reference a plural (should we say "result" or "results")

--
Matt Trzcinski
Emacs Org contributor (ob-shell)
Learn more about Org mode at https://orgmode.org
Support Org development at https://liberapay.com/org-mode