Fix "inverted" behavior of `colnames' header argument

Fix "inverted" behavior of `colnames' header argument

[Rudolf Adamkovič]

When loading Org tables into Pandas data frames, the

  `colnames' does exactly what it should not,

in that "yes" means "no" and vice versa:

  =colnames= set to =yes=:
  
  #+HEADER: :colnames yes
  #+HEADER: :session *Python*
  #+HEADER: :var INPUT = 20a0ea41-3669-482c-989e-d28486d5d400
  #+BEGIN_SRC python
  import pandas as pd
  pd.DataFrame(INPUT)
  #+END_SRC
  
  #+RESULTS:
  :     0   1
  : 0  10  20
  : 1  30  40
  
  =colnames= set to =no=:
  
  #+HEADER: :colnames no
  #+HEADER: :session *Python*
  #+HEADER: :var INPUT = 20a0ea41-3669-482c-989e-d28486d5d400
  #+BEGIN_SRC python
  import pandas as pd
  pd.DataFrame(INPUT)
  #+END_SRC
  
  #+RESULTS:
  :     0   1
  : 0   A   B
  : 1  10  20
  : 2  30  40

Confirmed on Stack Overflow (Sep 1, 2023):

https://stackoverflow.com/a/67091398/1306956

Rudy
-- 
"I love deadlines.  I love the whooshing noise they make as they go by."
--- Douglas Adams, The Salmon of Doubt, 2002

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org

Re: Fix "inverted" behavior of `colnames' header argument

[Ihor Radchenko]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

> Fix "inverted" behavior of `colnames' header argument

Our convention is to use [BUG] in subject like to mark bugs.
That's what M-x org-submit-bug-report does.

> When loading Org tables into Pandas data frames, the
>
>   `colnames' does exactly what it should not,
> ...

What you show looks consistent with "16.4 Environment of a Code Block"
in Org manual.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Fix "inverted" behavior of `colnames' header argument

[Rudolf Adamkovič]

Ihor Radchenko <yantar92@posteo.net> writes:

> Our convention is to use [BUG] in subject like to mark bugs.  That's
> what M-x org-submit-bug-report does.

TIL ... again. :)

> What you show looks consistent with "16.4 Environment of a Code Block"
> in Org manual.

Oh, I see.  Wow, that is /really/ confusing from the user's POV.

For anyone reading in the future, to get column names in Pandas, write
`:colnames no' (!) and `pd.DataFrame(VAR[1:], columns=VAR[0])'.

Using the example above, this would be:

  #+HEADER: :colnames no
  #+HEADER: :session *Python*
  #+HEADER: :var INPUT = 20a0ea41-3669-482c-989e-d28486d5d400
  #+BEGIN_SRC python
  import pandas as pd
  pd.DataFrame(INPUT[1:], columns=INPUT[0])
  #+END_SRC
  
  #+RESULTS:
  :     A   B
  : 0  10  20
  : 1  30  40

Canceled.

Rudy
-- 
"Programming reliably -- must be an activity of an undeniably
mathematical nature […] You see, mathematics is about thinking, and
doing mathematics is always trying to think as well as possible."
--- Edsger W. Dijkstra, 1981

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org

Re: Fix "inverted" behavior of `colnames' header argument

[Rudolf Adamkovič]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

> Oh, I see.  Wow, that is /really/ confusing from the user's POV.

In fact, I do not understand the manual.  It says:

  The default value is ‘nil’: if an input table has column names—because
  the second row is a horizontal rule—then Org removes the column names,
  processes the table, puts back the column names, and then writes the
  table to the results block.

Three questions immediately pop up in my head:

1. Is the manual talking about the input tables, specified with `:var',
   the output tables, when `:result table', or both?

2. Org "removes" the column names, then "processes" the table, and then
   "puts [the column names] back".  Why?  And what does it mean?

3. What if the second row is not a horizontal rule?  The manual says
   nothing about that happens in that case, with `:colnames nil'.

Also, I tried the following with `emacs -Q':

  #+NAME: 568ccf1e-f266-47ab-9402-0d380359d73c
  | 10 |
  |----|
  | 20 |
  | 30 |
  
  With =colnames= unset:
  
  #+HEADER: :var table = 568ccf1e-f266-47ab-9402-0d380359d73c
  #+BEGIN_SRC python
  return table
  #+END_SRC
  
  #+RESULTS:
  | 20 |
  | 30 |
  
  With =colnames= set to =nil=:
  
  #+HEADER: :colnames nil
  #+HEADER: :var table = 568ccf1e-f266-47ab-9402-0d380359d73c
  #+BEGIN_SRC python
  return table
  #+END_SRC
  
  #+RESULTS:
  | 10 |
  |----|
  | 20 |
  | 30 |

Why do the two outputs differ if

  The default value is ‘nil’: [...]

Thank you in advance for clarifications!

Rudy
-- 
"The whole science is nothing more than a refinement of everyday
thinking."  --- Albert Einstein, 1879-1955

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org

Re: Fix "inverted" behavior of `colnames' header argument

[Rudolf Adamkovič]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

> In fact, I do not understand the manual.  It says: [...]

Upon further thinking, I think I now get it.

My understanding:

  - =nil= ::
  
    1. Drop the column names from the input, if tabular.
    2. Run the code block, possibly producing results.
    3. Re-add the column names, if any, to the results, if tabular.
  
    N.B. Column names are row #1 if row #2 is a horizontal rule.
  
  - =yes= ::
  
    1. Drop the first row from the input, if tabular.
    2. Run the code block, possibly producing results.
    3. Re-add the row, if any, to the results, if tabular.
  
  - no ::
  
    Do not pre-process tabular inputs.

> Also, I tried the following with `emacs -Q':

I noticed that if I pass 'nil (with a quote) instead of nil, everything
works well.  Why is that?  And, no matter why that is, why do we say nil
in the manual instead of 'nil, if that is what the user must write?

Thank you!

Rudy
-- 
"I would prefer an intelligent hell to a stupid paradise."
--- Blaise Pascal

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org

Re: Fix "inverted" behavior of `colnames' header argument

[Ihor Radchenko]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

> Upon further thinking, I think I now get it.
>
> My understanding:
>
>   - =nil= ::
>   
>     1. Drop the column names from the input, if tabular.
>     2. Run the code block, possibly producing results.
>     3. Re-add the column names, if any, to the results, if tabular.
>   
>     N.B. Column names are row #1 if row #2 is a horizontal rule.

Sounds about right. See `org-babel-disassemble-tables' and
`org-babel-reassemble-table' for what is actually being done.

I guess "auto" would have been a better name for this value.
   
>> Also, I tried the following with `emacs -Q':
>
> I noticed that if I pass 'nil (with a quote) instead of nil, everything
> works well.  Why is that?  And, no matter why that is, why do we say nil
> in the manual instead of 'nil, if that is what the user must write?

That's because Org treats :foo text as a string or number by
default. Org only parses the value as Elisp when it starts from "(",
"\\='", "\\=`" or a "[". See `org-babel-read'.

This means that :colnames nil actually means :colnames "nil".

I agree that it is confusing. It would be a good idea to treat nil and
"nil" equally. Maybe even rename the default to "auto" to avoid
confusion (in addition to nil/"nil").

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Fix "inverted" behavior of `colnames' header argument

[Ihor Radchenko]

Rudolf Adamkovič <rudolf@adamkovic.org> writes:

>> Our convention is to use [BUG] in subject like to mark bugs.  That's
>> what M-x org-submit-bug-report does.
>
> TIL ... again. :)

Also, see https://tracker.orgmode.org/howto

>> What you show looks consistent with "16.4 Environment of a Code Block"
>> in Org manual.
>
> Oh, I see.  Wow, that is /really/ confusing from the user's POV.

Yeah. It threw me off first time I've read about column names as well.
But it is what we historically have.

We cannot change the existing behavior, but we might try to
introduce some alternative, more intuitive, parameter or introduce more
intuitive values.

If you have some good ideas how to do such thing, they would be welcome.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Fix "inverted" behavior of `colnames' header argument

[Rudolf Adamkovič]

Ihor Radchenko <yantar92@posteo.net> writes:

> If you have some good ideas how to do such thing, they would be
> welcome.

I think your two ideas were spot on:

> It would be a good idea to treat nil and "nil" equally. Maybe even
> rename the default to "auto" to avoid confusion (in addition to
> nil/"nil").

Rudy
-- 
"I love deadlines.  I love the whooshing noise they make as they go by."
--- Douglas Adams, The Salmon of Doubt, 2002

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org