[BUG] text cell contents have extra parentheses when used in formulas [9.6.4 ( @ /home/jet/.config/emacs/elpa/org-9.6.4/)]

[BUG] text cell contents have extra parentheses when used in formulas [9.6.4 ( @ /home/jet/.config/emacs/elpa/org-9.6.4/)]

[Jeff Trull]

[-- Attachment #1: Type: text/plain, Size: 7923 bytes --]

Expectation: when using cell contents as strings in a formula, the
strings should contain exactly the cell's text contents

Actual: the resulting string is wrapped in parentheses

I can find nothing in the documentation that suggests this is expected,
and it is quite unintuitive if you want to (for example) filter on
a column.

#+TITLE: text formula repro

| Key | Value        |
|-----+--------------|
| A   | [40, 65, 41] |
| B   | [40, 66, 41] |
#+TBLFM: $2="$1"

Here you can see that the contents of the Key column were wrapped in
parentheses (ASCII 40 and 41) when turned into strings.

Emacs  : GNU Emacs 28.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version
3.24.33, cairo version 1.16.0)
 of 2022-10-14
Package: Org mode version 9.6.4 ( @ /home/jet/.config/emacs/elpa/org-9.6.4/)

current state:
==============
(setq
 org-link-elisp-confirm-function 'yes-or-no-p
 org-ditaa-jar-path "/usr/share/ditaa/ditaa.jar"
 org-bibtex-headline-format-function #[257 "\300 \236A\207" [:title] 3
"\n\n(fn ENTRY)"]
 org-persist-after-read-hook '(org-element--cache-persist-after-read)
 org-re-reveal-root "https://cdn.jsdelivr.net/npm/reveal.js@3.6.0"
 org-export-before-parsing-hook '(org-attach-expand-links)
 org-cycle-tab-first-hook '(org-babel-hide-result-toggle-maybe
org-babel-header-arg-expand)
 org-archive-hook '(org-attach-archive-delete-maybe)
 org-ascii-format-drawer-function #[771 " \207" [] 4 "\n\n(fn NAME CONTENTS
WIDTH)"]
 org-cycle-hook '(org-cycle-hide-archived-subtrees
org-cycle-show-empty-lines org-cycle-optimize-window-after-visibility-change
                  org-cycle-display-inline-images)
 org-latex-listings-options '(("basicstyle" "\\footnotesize\\ttfamily")
("showstringspaces" "false") ("breaklines" "true"))
 org-persist-before-read-hook '(org-element--cache-persist-before-read)
 org-mode-hook '(#[0 "\301\211 \207" [imenu-create-index-function
org-imenu-get-tree] 2] org-superstar-mode org-tempo-setup
                 (lambda nil (electric-indent-local-mode -1))
                 #[0 "\300\301\302\303\304$\207" [add-hook
change-major-mode-hook org-fold-show-all append local] 5]
                 #[0 "\300\301\302\303\304$\207" [add-hook
change-major-mode-hook org-babel-show-result-all append local] 5]
org-babel-result-hide-spec
                 org-babel-hide-all-hashes org-eldoc-load)
 org-babel-load-languages '((emacs-lisp . t) (dot . t) (ditaa . t) (shell .
t))
 org-latex-format-drawer-function #[514 "\207" [] 3 "\n\n(fn _ CONTENTS)"]
 org-latex-format-headline-function
'org-latex-format-headline-default-function
 org-confirm-shell-link-function 'yes-or-no-p
 org-html-format-drawer-function #[514 "\207" [] 3 "\n\n(fn NAME CONTENTS)"]
 outline-isearch-open-invisible-function 'outline-isearch-open-invisible
 org-latex-classes '(("beamer"
                      "\\documentclass[11pt,aspectratio=169]{beamer}\n
             \\usenavigationsymbolstemplate{} % no navigation bar\n
         \\beamertemplateballitem         % items have little balls"
                      ("\\section{%s}" . "\\section*{%s}")
("\\subsection{%s}" . "\\subsection*{%s}") ("\\subsubsection{%s}" .
"\\subsubsection*{%s}"))
                     ("article" "\\documentclass[11pt]{article}"
("\\section{%s}" . "\\section*{%s}") ("\\subsection{%s}" .
"\\subsection*{%s}")
                      ("\\subsubsection{%s}" . "\\subsubsection*{%s}")
("\\paragraph{%s}" . "\\paragraph*{%s}")
                      ("\\subparagraph{%s}" . "\\subparagraph*{%s}"))
                     ("report" "\\documentclass[11pt]{report}"
("\\part{%s}" . "\\part*{%s}") ("\\chapter{%s}" . "\\chapter*{%s}")
                      ("\\section{%s}" . "\\section*{%s}")
("\\subsection{%s}" . "\\subsection*{%s}") ("\\subsubsection{%s}" .
"\\subsubsection*{%s}"))
                     ("book" "\\documentclass[11pt]{book}" ("\\part{%s}" .
"\\part*{%s}") ("\\chapter{%s}" . "\\chapter*{%s}")
                      ("\\section{%s}" . "\\section*{%s}")
("\\subsection{%s}" . "\\subsection*{%s}") ("\\subsubsection{%s}" .
"\\subsubsection*{%s}"))
                     )
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-latex-packages-alist '(("" "moresize") ("" "fancyvrb") (""
"inconsolata") ("" "tikz") ("" "color") ("" "listings"))
 org-src-mode-hook '(org-src-babel-configure-edit-buffer
org-src-mode-configure-edit-buffer)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-latex-src-block-backend t
 org-ditaa-eps-jar-path
"/home/jet/.config/emacs/elpa/contrib/scripts/DitaaEps.jar"
 org-structure-template-alist '(("n" . "notes") ("a" . "export ascii") ("c"
. "center") ("C" . "comment") ("e" . "example") ("E" . "export")
                                ("h" . "export html") ("l" . "export
latex") ("q" . "quote") ("s" . "src") ("v" . "verse"))
 org-speed-command-hook '(org-speed-command-activate
org-babel-speed-command-activate)
 org-html-format-inlinetask-function
'org-html-format-inlinetask-default-function
 org-ascii-format-inlinetask-function 'org-ascii-format-inlinetask-default
 org-export-backends '(ascii html icalendar latex md confluence re-reveal)
 org-confirm-babel-evaluate nil
 org-fold-core-isearch-open-function 'org-fold--isearch-reveal
 org-re-reveal-title-slide "<h2>%t</h2><h2>%a</h2><h2>%d</h2>"
 org-latex-format-inlinetask-function
'org-latex-format-inlinetask-default-function
 org-persist-before-write-hook '(org-element--cache-persist-before-write)
 org-tab-first-hook '(org-babel-hide-result-toggle-maybe
org-babel-header-arg-expand)
 org-link-shell-confirm-function 'yes-or-no-p
 org-babel-pre-tangle-hook '(save-buffer)
 org-agenda-loop-over-headlines-in-active-region nil
 org-occur-hook '(org-first-headline-recenter)
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-link-parameters '(("attachment" :follow org-attach-follow :complete
org-attach-complete-link) ("eww" :follow org-eww-open :store
org-eww-store-link)
                       ("rmail" :follow org-rmail-open :store
org-rmail-store-link) ("mhe" :follow org-mhe-open :store org-mhe-store-link)
                       ("irc" :follow org-irc-visit :store
org-irc-store-link :export org-irc-export)
                       ("info" :follow org-info-open :export
org-info-export :store org-info-store-link :insert-description
                        org-info-description-as-command)
                       ("gnus" :follow org-gnus-open :store
org-gnus-store-link)
                       ("docview" :follow org-docview-open :export
org-docview-export :store org-docview-store-link)
                       ("bibtex" :follow org-bibtex-open :store
org-bibtex-store-link)
                       ("bbdb" :follow org-bbdb-open :export
org-bbdb-export :complete org-bbdb-complete-link :store org-bbdb-store-link)
                       ("w3m" :store org-w3m-store-link) ("doi" :follow
org-link-doi-open :export org-link-doi-export) ("id" :follow org-id-open)
                       ("file+sys") ("file+emacs") ("shell" :follow
org-link--open-shell)
                       ("news" :follow #[514 "\301\300\302 Q \"\207"
["news" browse-url ":"] 6 "\n\n(fn URL ARG)"])
                       ("mailto" :follow #[514 "\301\300\302 Q \"\207"
["mailto" browse-url ":"] 6 "\n\n(fn URL ARG)"])
                       ("https" :follow #[514 "\301\300\302 Q \"\207"
["https" browse-url ":"] 6 "\n\n(fn URL ARG)"])
                       ("http" :follow #[514 "\301\300\302 Q \"\207"
["http" browse-url ":"] 6 "\n\n(fn URL ARG)"])
                       ("ftp" :follow #[514 "\301\300\302 Q \"\207" ["ftp"
browse-url ":"] 6 "\n\n(fn URL ARG)"])
                       ("help" :follow org-link--open-help :store
org-link--store-help) ("file" :complete org-link-complete-file)
                       ("elisp" :follow org-link--open-elisp))
 org-html-format-headline-function
'org-html-format-headline-default-function
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-tab-before-tab-emulation-hook '(org-tempo-complete-tag)
 )

[-- Attachment #2: Type: text/html, Size: 10439 bytes --]

Re: [BUG] text cell contents have extra parentheses when used in formulas [9.6.4 ( @ /home/jet/.config/emacs/elpa/org-9.6.4/)]

[Ihor Radchenko]

Jeff Trull <edaskel@att.net> writes:

> Expectation: when using cell contents as strings in a formula, the
> strings should contain exactly the cell's text contents
>
> Actual: the resulting string is wrapped in parentheses

This is actually expected, although surprising.

> I can find nothing in the documentation that suggests this is expected,
> and it is quite unintuitive if you want to (for example) filter on
> a column.

In short, do not use ".
For filtering, see 3.5.7 Lookup functions.

> #+TITLE: text formula repro
>
> | Key | Value        |
> |-----+--------------|
> | A   | [40, 65, 41] |
> | B   | [40, 66, 41] |
> #+TBLFM: $2="$1"
>
> Here you can see that the contents of the Key column were wrapped in
> parentheses (ASCII 40 and 41) when turned into strings.

If you turn formula debugger on (M-x org-table-toggle-formula-debugger),
you will see the following (I also added extra explanations):

Substitution history of formula
Orig:   "$1" ;; begin calculation of $1 field value
$xyz->  "$1" ;; after handling relative references
@r$c->  "$1" ;; after handling cell references
$1->    "(A)" ;; after handling column references; the (...) is added
              ;; because of Calc syntax to ensure that values inside
              ;; cell are grouped. There is no notion of strings in GNU
              ;; Calc (AFAIK). So, (...) grouping always represents
              ;; algebraic Calc expression.
Result: [40, 65, 41] ;; (calc-eval "\"(A)\"")
Format: NONE
Final:  [40, 65, 41]

As you can see, we passed the formula to GNU Calc.
However, "foo" in GNU calc is nothing but a vector of characters.
So, $2="$1" is not what you anticipated.

You may refer to 3.5.2 Formula syntax for Calc where we put a reference
to GNU Calc manual. It is out of scope of Org manual to describe how GNU
Calc works.

Do note that an alternative formula syntax is using Elisp (see 3.5.3
Emacs Lisp forms as formulas). You can use that syntax if you prefer it
over GNU Calc.

Canceled.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] text cell contents have extra parentheses when used in formulas [9.6.4 ( @ /home/jet/.config/emacs/elpa/org-9.6.4/)]

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

>> #+TITLE: text formula repro
>>
>> | Key | Value        |
>> |-----+--------------|
>> | A   | [40, 65, 41] |
>> | B   | [40, 66, 41] |
>> #+TBLFM: $2="$1"
>>
>> Here you can see that the contents of the Key column were wrapped in
>> parentheses (ASCII 40 and 41) when turned into strings.
>
> If you turn formula debugger on (M-x org-table-toggle-formula-debugger),
> you will see the following (I also added extra explanations):
> ...
> $1->    "(A)" ;; after handling column references; the (...) is added
>               ;; because of Calc syntax to ensure that values inside
>               ;; cell are grouped. There is no notion of strings in GNU
>               ;; Calc (AFAIK). So, (...) grouping always represents
>               ;; algebraic Calc expression.
> Result: [40, 65, 41] ;; (calc-eval "\"(A)\"")

After looking closer, the code wrapping cell value into (...) is
actually not necessarily correct.

The parenthesis wrapping has been introduced in

e7e5e25100246e5eab3a36034ac2de578dabd02f
Author:     Carsten Dominik <carsten.dominik@gmail.com>
Fix bug with algebraic evaluation in tables.

There was a problem that complex fields need a pair of parentheses
around them.  For example, if one field is "a a" and another field
is "a+b", multiplication of the two gave "a^3+b" instead of "a^3+a^2 b".
Now variable replacement puts extra parenthesis around values.

----

However, in a way, unconditional wrapping of cell value in parentheses is
against how Org table formulas intuitively work - references are
replaced by appropriate cell values.

Wrapping a cell value in (...) only makes sense as long as algebraic
operations are considered. However, when cell value is intentionally
treated as string (explicit vector of character codes in GNU Calc
conventions), extra (...) will introduce the observed 40 ... 41 into the
vector.

It might be tricky to remove the existing behaviour though - removing the
parenthesis will likely break the existing use cases with genuine
algebraic computations. We might, however, use "L" flag, described in
"3.5.2 Formula syntax for Calc":

‘L’
     Literal, for Lisp formulas only.  See the next section.

Now, it is only used for Lisp formulas, but we can make use of it in
Calc formulas as well, preventing this (...) wrap.

Or maybe others have better ideas.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>