[PATCH] Add support for tables in Calc src block :var

[PATCH] Add support for tables in Calc src block :var

[Visuwesh]

[-- Attachment #1: Type: text/plain, Size: 449 bytes --]

Attached patch adds support for passing tables as matrices (or vector)
to a Calc source block through the :var parameter.  There might be a
better way to do it than manually construct the data structure expected
by Calc for a matrix but given that it rarely sees changes in this area,
it shouldn't be a bother.
I also added tests for this feature, and other simple tests to ensure
everything works.  I checked that all the tests passed on my side.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-calc-Support-passing-tables-as-matrices-with-var.patch --]
[-- Type: text/x-diff, Size: 6126 bytes --]

From 5dd444a3dd688b366e1151739f6a8e8088bb0623 Mon Sep 17 00:00:00 2001
From: Visuwesh <visuweshm@gmail.com>
Date: Sat, 16 Mar 2024 09:10:53 +0530
Subject: [PATCH] ob-calc: Support passing tables as matrices with :var

A table with MxN dimensions is converted to a MxN matrix when given in
:var to a Calc source block.  A table with a single row is converted
to a vector (i.e., row vector).

* lisp/ob-calc.el (org-babel-execute-src-block:calc): Construct the
right data structure to pass tables as matrices to Calc.
* testing/examples/ob-calc-test.org:
* testing/lisp/test-ob-calc.el: Add tests for ob-calc, and this new
feature.
---
 lisp/ob-calc.el                   | 10 +++-
 testing/examples/ob-calc-test.org | 57 ++++++++++++++++++++++
 testing/lisp/test-ob-calc.el      | 80 +++++++++++++++++++++++++++++++
 3 files changed, 146 insertions(+), 1 deletion(-)
 create mode 100644 testing/examples/ob-calc-test.org
 create mode 100644 testing/lisp/test-ob-calc.el

diff --git a/lisp/ob-calc.el b/lisp/ob-calc.el
index f36df77ff..c8bbcd16b 100644
--- a/lisp/ob-calc.el
+++ b/lisp/ob-calc.el
@@ -64,7 +64,15 @@
 	 (var-names (mapcar #'symbol-name org--var-syms)))
     (mapc
      (lambda (pair)
-       (calc-push-list (list (cdr pair)))
+       (let ((val (cdr pair)))
+         (calc-push-list
+          (list (if (listp val)
+                    (cons 'vec
+                          (if (null (cdr val))
+                              (car val)
+                            (mapcar (lambda (x) (if (listp x) (cons 'vec x) x))
+                                    val)))
+                  val))))
        (calc-store-into (car pair)))
      vars)
     (mapc
diff --git a/testing/examples/ob-calc-test.org b/testing/examples/ob-calc-test.org
new file mode 100644
index 000000000..6df44c6a4
--- /dev/null
+++ b/testing/examples/ob-calc-test.org
@@ -0,0 +1,57 @@
+#+TITLE: Tests for ob-calc
+#+OPTIONS: ^:nil
+
+* Simple
+:PROPERTIES:
+:ID:       40e4cd26-fe15-45c0-938b-111e021a5a99
+:END:
+
+#+BEGIN_SRC calc :results silent
+	1 * 2
+#+END_SRC
+
+#+BEGIN_SRC calc :results silent
+	12 + 16 - 1
+#+END_SRC
+
+#+BEGIN_SRC calc :results silent
+	inv(a)
+#+END_SRC
+
+* Tables
+:PROPERTIES:
+:ID:       138938e1-f0ba-4c2b-babe-5d20f2b83557
+:END:
+
+#+NAME: ob-calc-table-1
+| 1 |  2 |  3 |
+| 5 |  6 |  7 |
+| 9 | 14 | 11 |
+
+#+NAME: ob-calc-table-2
+| 1 | 2 | 3 | 4 | 5 |
+
+#+NAME: ob-calc-table-3
+| 1 |
+| 2 |
+| 3 |
+
+#+BEGIN_SRC calc :results silent :var a=ob-calc-table-1
+	inv(a)
+#+END_SRC
+
+#+BEGIN_SRC calc :results silent :var a=ob-calc-table-2
+	a*2 - 2
+#+END_SRC
+
+#+BEGIN_SRC calc :results silent :var a=ob-calc-table-2
+	vmean(a)
+#+END_SRC
+
+#+BEGIN_SRC calc :results silent :var a=ob-calc-table-3
+	a
+#+END_SRC
+
+#+BEGIN_SRC calc :results silent :var a=ob-calc-table-2
+	a
+#+END_SRC
diff --git a/testing/lisp/test-ob-calc.el b/testing/lisp/test-ob-calc.el
new file mode 100644
index 000000000..443ccd64b
--- /dev/null
+++ b/testing/lisp/test-ob-calc.el
@@ -0,0 +1,80 @@
+;;; test-ob-calc.el --- tests for ob-calc.el         -*- lexical-binding: t; -*-
+
+;; Copyright (C) 2024  Visuwesh
+
+;; Author: Visuwesh <visuweshm@gmail.com>
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Code:
+(require 'ob-calc)
+
+(unless (featurep 'ob-calc)
+  (signal 'missing-test-dependency "Support for Calc code blocks"))
+
+(ert-deftest ob-calc/simple-program-mult ()
+  "Test of simple multiplication."
+  (org-test-at-id "40e4cd26-fe15-45c0-938b-111e021a5a99"
+    (org-babel-next-src-block)
+    (should (equal "2" (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/simple-program-arith ()
+  "Test of simple arithmetic."
+  (org-test-at-id "40e4cd26-fe15-45c0-938b-111e021a5a99"
+    (org-babel-next-src-block 2)
+    (should (equal "27" (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/simple-program-symbolic ()
+  "Test of simple symbolic algebra."
+  (org-test-at-id "40e4cd26-fe15-45c0-938b-111e021a5a99"
+    (org-babel-next-src-block 3)
+    (should (equal "1 / a" (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/matrix-inversion ()
+  "Test of a matrix inversion."
+  (org-test-at-id "138938e1-f0ba-4c2b-babe-5d20f2b83557"
+    (org-babel-next-src-block)
+    (should (equal "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]"
+                   (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/matrix-algebra ()
+  "Test of simple matrix algebra."
+  (org-test-at-id "138938e1-f0ba-4c2b-babe-5d20f2b83557"
+    (org-babel-next-src-block 2)
+    (should (equal "[0, 2, 4, 6, 8]"
+                   (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/matrix-mean ()
+  "Test of simple mean of a vector."
+  (org-test-at-id "138938e1-f0ba-4c2b-babe-5d20f2b83557"
+    (org-babel-next-src-block 3)
+    (should (equal "3"
+                   (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/matrix-correct-conv-column ()
+  "Test of conversion of column table to Calc format."
+  (org-test-at-id "138938e1-f0ba-4c2b-babe-5d20f2b83557"
+    (org-babel-next-src-block 4)
+    (should (equal "[[1], [2], [3]]"
+                   (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/matrix-correct-conv-row ()
+  "Test of conversion of row table to Calc format."
+  (org-test-at-id "138938e1-f0ba-4c2b-babe-5d20f2b83557"
+    (org-babel-next-src-block 5)
+    (should (equal "[1, 2, 3, 4, 5]"
+                   (org-babel-execute-src-block)))))
+
+(provide 'test-ob-calc)
+;;; test-ob-calc.el ends here
-- 
2.43.0


[-- Attachment #3: Type: text/plain, Size: 68 bytes --]


P.S. Please keep me in the CCs as I am not subscribed to the list.

Re: [PATCH] Add support for tables in Calc src block :var

[Ihor Radchenko]

Visuwesh <visuweshm@gmail.com> writes:

> Attached patch adds support for passing tables as matrices (or vector)
> to a Calc source block through the :var parameter.  There might be a
> better way to do it than manually construct the data structure expected
> by Calc for a matrix but given that it rarely sees changes in this area,
> it shouldn't be a bother.
> I also added tests for this feature, and other simple tests to ensure
> everything works.  I checked that all the tests passed on my side.

Thanks!
Since you are adding a new feature, may you also add an entry to etc/ORG-NEWS?

> -       (calc-push-list (list (cdr pair)))
> +       (let ((val (cdr pair)))
> +         (calc-push-list
> +          (list (if (listp val)
> +                    (cons 'vec
> +                          (if (null (cdr val))
> +                              (car val)
> +                            (mapcar (lambda (x) (if (listp x) (cons 'vec x) x))
> +                                    val)))
> +                  val))))

It would be nice to add code comments explaining the Calc's internal
format. Ideally, with references to Calc's manual or source code.
Otherwise, this code looks like black magic :)

> +++ b/testing/examples/ob-calc-test.org
> @@ -0,0 +1,57 @@
> +#+TITLE: Tests for ob-calc
> +#+OPTIONS: ^:nil

Thanks a lot for adding many tests!
Would you mind using `org-test-with-temp-text' instead of
`org-test-at-id' as much as possible? Otherwise, looking at tests like

> +(ert-deftest ob-calc/simple-program-mult ()
> +  "Test of simple multiplication."
> +  (org-test-at-id "40e4cd26-fe15-45c0-938b-111e021a5a99"
> +    (org-babel-next-src-block)
> +    (should (equal "2" (org-babel-execute-src-block)))))

it is very difficult to understand what exactly the tests is checking.
`org-test-with-temp-text' would make things a lot more readable as we
would see the calc code being tested right inside the test code.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] Add support for tables in Calc src block :var

[Visuwesh]

[-- Attachment #1: Type: text/plain, Size: 1883 bytes --]

[சனி மார்ச் 16, 2024] Ihor Radchenko wrote:

> Visuwesh <visuweshm@gmail.com> writes:
>
>> Attached patch adds support for passing tables as matrices (or vector)
>> to a Calc source block through the :var parameter.  There might be a
>> better way to do it than manually construct the data structure expected
>> by Calc for a matrix but given that it rarely sees changes in this area,
>> it shouldn't be a bother.
>> I also added tests for this feature, and other simple tests to ensure
>> everything works.  I checked that all the tests passed on my side.
>
> Thanks!
> Since you are adding a new feature, may you also add an entry to etc/ORG-NEWS?

Now done.

>> -       (calc-push-list (list (cdr pair)))
>> +       (let ((val (cdr pair)))
>> +         (calc-push-list
>> +          (list (if (listp val)
>> +                    (cons 'vec
>> +                          (if (null (cdr val))
>> +                              (car val)
>> +                            (mapcar (lambda (x) (if (listp x) (cons 'vec x) x))
>> +                                    val)))
>> +                  val))))
>
> It would be nice to add code comments explaining the Calc's internal
> format. Ideally, with references to Calc's manual or source code.
> Otherwise, this code looks like black magic :)

I hope the comment I added in the attached patch is clear enough.

>> +++ b/testing/examples/ob-calc-test.org
>> @@ -0,0 +1,57 @@
>> +#+TITLE: Tests for ob-calc
>> +#+OPTIONS: ^:nil
>
> Thanks a lot for adding many tests!
> Would you mind using `org-test-with-temp-text' instead of
> `org-test-at-id' as much as possible? Otherwise, looking at tests like

OK, I wasn't sure what test style to use since the README in testing/
does not talk about writing new tests.  Now I have adapted the tests to
use org-test-with-temp-text.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-calc.el-Add-support-for-tables-in-Calc-source-blo.patch --]
[-- Type: text/x-diff, Size: 6241 bytes --]

From 29962bc3ec33e1e25f83f153b681d49182368592 Mon Sep 17 00:00:00 2001
From: Visuwesh <visuweshm@gmail.com>
Date: Sat, 16 Mar 2024 17:04:14 +0530
Subject: [PATCH] ob-calc.el: Add support for tables in Calc source block :var

A table with MxN dimensions is converted to a MxN matrix when given in
:var to a Calc source block.  A table with a single row is converted
to a vector (i.e., row vector).

* lisp/ob-calc.el (org-babel-execute-src-block:calc): Construct the
right data structure to pass tables as matrices to Calc.
* testing/lisp/test-ob-calc.el: Add tests for ob-calc, and this new
feature.
* etc/ORG-NEWS: Announce the feature.
---
 etc/ORG-NEWS                 |   7 +++
 lisp/ob-calc.el              |  14 ++++-
 testing/lisp/test-ob-calc.el | 115 +++++++++++++++++++++++++++++++++++
 3 files changed, 135 insertions(+), 1 deletion(-)
 create mode 100644 testing/lisp/test-ob-calc.el

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index ca73f06e7..197d7503d 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -1121,6 +1121,13 @@ Maxima's graphics packages (~draw~ or ~plot~); the default remains
 ~plot~.  The graphics terminal is now determined from the file-ending
 of the file-name set in the ~:file~ header argument.
 
+*** =ob-calc.el=: Support for tables in ~:var~
+
+=ob-calc= now supports tables in ~:var~.  They are converted to a
+matrix or a vector depending on the dimensionality of the table.  A
+table with a single row is converted to a vector, the rest are
+converted to a matrix.
+
 *** Images and files in clipboard can be pasted
 
 Org asks the user what must be done when pasting images and files
diff --git a/lisp/ob-calc.el b/lisp/ob-calc.el
index f36df77ff..810ed1735 100644
--- a/lisp/ob-calc.el
+++ b/lisp/ob-calc.el
@@ -64,7 +64,19 @@
 	 (var-names (mapcar #'symbol-name org--var-syms)))
     (mapc
      (lambda (pair)
-       (calc-push-list (list (cdr pair)))
+       (let ((val (cdr pair)))
+         (calc-push-list
+          ;; For a vector, Calc follows the format (vec 1 2 3 ...)  so
+          ;; a matrix becomes (vec (vec 1 2 3) (vec 4 5 6) ...).  See
+          ;; the comments in "Arithmetic routines." section of
+          ;; calc.el.
+          (list (if (listp val)
+                    (cons 'vec
+                          (if (null (cdr val))
+                              (car val)
+                            (mapcar (lambda (x) (if (listp x) (cons 'vec x) x))
+                                    val)))
+                  val))))
        (calc-store-into (car pair)))
      vars)
     (mapc
diff --git a/testing/lisp/test-ob-calc.el b/testing/lisp/test-ob-calc.el
new file mode 100644
index 000000000..6d6ca104d
--- /dev/null
+++ b/testing/lisp/test-ob-calc.el
@@ -0,0 +1,115 @@
+;;; test-ob-calc.el --- tests for ob-calc.el         -*- lexical-binding: t; -*-
+
+;; Copyright (C) 2024  Visuwesh
+
+;; Author: Visuwesh <visuweshm@gmail.com>
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Code:
+(require 'ob-calc)
+
+(unless (featurep 'ob-calc)
+  (signal 'missing-test-dependency "Support for Calc code blocks"))
+
+(ert-deftest ob-calc/simple-program-mult ()
+  "Test of simple multiplication."
+  (org-test-with-temp-text "\
+#+BEGIN_SRC calc :results silent
+	1 * 2
+#+END_SRC"
+    (should (equal "2" (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/simple-program-arith ()
+  "Test of simple arithmetic."
+  (org-test-with-temp-text "\
+#+BEGIN_SRC calc :results silent
+	12 + 16 - 1
+#+END_SRC"
+    (should (equal "27" (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/simple-program-symbolic ()
+  "Test of simple symbolic algebra."
+  (org-test-with-temp-text "\
+#+BEGIN_SRC calc :results silent
+	inv(a)
+#+END_SRC"
+    (should (equal "1 / a" (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/matrix-inversion ()
+  "Test of a matrix inversion."
+  (org-test-with-temp-text "\
+#+NAME: ob-calc-table-1
+| 1 |  2 |  3 |
+| 5 |  6 |  7 |
+| 9 | 14 | 11 |
+
+<point>#+BEGIN_SRC calc :results silent :var a=ob-calc-table-1
+	inv(a)
+#+END_SRC "
+    (should (equal "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]"
+                   (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/matrix-algebra ()
+  "Test of simple matrix algebra."
+  (org-test-with-temp-text "\
+#+NAME: ob-calc-table-2
+| 1 | 2 | 3 | 4 | 5 |
+
+<point>#+BEGIN_SRC calc :results silent :var a=ob-calc-table-2
+	a*2 - 2
+#+END_SRC"
+    (should (equal "[0, 2, 4, 6, 8]"
+                   (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/matrix-mean ()
+  "Test of simple mean of a vector."
+  (org-test-with-temp-text "\
+#+NAME: ob-calc-table-2
+| 1 | 2 | 3 | 4 | 5 |
+
+<point>#+BEGIN_SRC calc :results silent :var a=ob-calc-table-2
+	vmean(a)
+#+END_SRC"
+    (should (equal "3"
+                   (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/matrix-correct-conv-column ()
+  "Test of conversion of column table to Calc format."
+  (org-test-with-temp-text "\
+#+NAME: ob-calc-table-3
+| 1 |
+| 2 |
+| 3 |
+
+<point>#+BEGIN_SRC calc :results silent :var a=ob-calc-table-3
+	a
+#+END_SRC"
+    (should (equal "[[1], [2], [3]]"
+                   (org-babel-execute-src-block)))))
+
+(ert-deftest ob-calc/matrix-correct-conv-row ()
+  "Test of conversion of row table to Calc format."
+  (org-test-with-temp-text "\
+#+NAME: ob-calc-table-2
+| 1 | 2 | 3 | 4 | 5 |
+
+<point>#+BEGIN_SRC calc :results silent :var a=ob-calc-table-2
+	a
+#+END_SRC"
+    (should (equal "[1, 2, 3, 4, 5]"
+                   (org-babel-execute-src-block)))))
+
+(provide 'test-ob-calc)
+;;; test-ob-calc.el ends here
-- 
2.43.0

Re: [PATCH] Add support for tables in Calc src block :var

[Ihor Radchenko]

Visuwesh <visuweshm@gmail.com> writes:

> Subject: [PATCH] ob-calc.el: Add support for tables in Calc source block :var

Thanks!
Applied, onto main.
I modified the patch to enable ob-calc tests by default.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=89b0773c3

>> Would you mind using `org-test-with-temp-text' instead of
>> `org-test-at-id' as much as possible? Otherwise, looking at tests like
>
> OK, I wasn't sure what test style to use since the README in testing/
> does not talk about writing new tests.  Now I have adapted the tests to
> use org-test-with-temp-text.

Yeah. We do not have a dedicated style document for Org mode tests.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [PATCH] Add support for tables in Calc src block :var

[Visuwesh]

[சனி மார்ச் 16, 2024] Ihor Radchenko wrote:

> Visuwesh <visuweshm@gmail.com> writes:
>
>> Subject: [PATCH] ob-calc.el: Add support for tables in Calc source block :var
>
> Thanks!
> Applied, onto main.
> I modified the patch to enable ob-calc tests by default.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=89b0773c3

Thanks!

>>> Would you mind using `org-test-with-temp-text' instead of
>>> `org-test-at-id' as much as possible? Otherwise, looking at tests like
>>
>> OK, I wasn't sure what test style to use since the README in testing/
>> does not talk about writing new tests.  Now I have adapted the tests to
>> use org-test-with-temp-text.
>
> Yeah. We do not have a dedicated style document for Org mode tests.

Re: [PATCH] Add support for tables in Calc src block :var

[Fraga, Eric]

Thank you for this.  Potentially very useful.

As you have managed to understand calc internals (to a much greater
degrees than I have ever managed), do you know if there is any way to go
the other way?  Specifically, I would love to make reference to calc
variables, especially those defined using embedded calc, in org tables.

Thanks again,
eric

-- 
Eric S Fraga, https://profiles.ucl.ac.uk/5958-eric-fraga

Re: [PATCH] Add support for tables in Calc src block :var

[Visuwesh]

[திங்கள் மார்ச் 18, 2024] Fraga, Eric wrote:

> Thank you for this.  Potentially very useful.
>
> As you have managed to understand calc internals (to a much greater
> degrees than I have ever managed), do you know if there is any way to go
> the other way?  Specifically, I would love to make reference to calc
> variables, especially those defined using embedded calc, in org tables.

You give me far too credit: I merely placed an edebug trigger for
calc-push-list and used a simple-minded ' [1,2,3;4,5,6] RET to figure
out the vector format (then later I found the commentary).

If I get the time, I will try to look into your request.  Unfortunately,
I cannot promise anything since I am near the end of my semester making
me annoyingly busy.
[ I would like to make ob-calc turn matrices into tables in the result:
  like what ob-fortran and others already do.  ]

> Thanks again,
> eric

Re: [PATCH] Add support for tables in Calc src block :var

[Fraga, Eric]

On Monday, 18 Mar 2024 at 20:59, Visuwesh wrote:
> You give me far too credit: I merely placed an edebug trigger for
> calc-push-list and used a simple-minded ' [1,2,3;4,5,6] RET to figure
> out the vector format (then later I found the commentary).

Cute!

> If I get the time, I will try to look into your request.  Unfortunately,
> I cannot promise anything since I am near the end of my semester making
> me annoyingly busy.

That's perfectly fine!  I'm in no rush; it's just an itch that needs
scratching. :-)

Thank you,
eric

-- 
Eric S Fraga, https://profiles.ucl.ac.uk/5958-eric-fraga

Re: [PATCH] Add support for tables in Calc src block :var

[Visuwesh]

[திங்கள் மார்ச் 18, 2024] Fraga, Eric wrote:

Good day, Eric!

> Thank you for this.  Potentially very useful.
>
> As you have managed to understand calc internals (to a much greater
> degrees than I have ever managed), do you know if there is any way to go
> the other way?  Specifically, I would love to make reference to calc
> variables, especially those defined using embedded calc, in org tables.

Can you please provide a minimal example for me to play around with?  I
realise I would like something like this too [*] but I don't know
concretely what this would/should look like.



[*] More specifically, I want to pass around src block evaluation
    results in a table formula.

Re: [PATCH] Add support for tables in Calc src block :var

[Fraga, Eric]

On Saturday, 30 Mar 2024 at 18:34, Visuwesh wrote:
>> Specifically, I would love to make reference to calc variables,
>> especially those defined using embedded calc, in org tables.
>
> Can you please provide a minimal example for me to play around with?  I
> realise I would like something like this too [*] but I don't know
> concretely what this would/should look like.

So, embedded calc processes expressions in any buffer, including org
mode, which might look like this:

x := 3

y := 5

z := 3 x - y => 4

where, in this case, the value of z has been determined by calc and the
answer given after the => in the line.  The beauty of embedded calc is
you can change the value of x and the subsequent expressions will be
updated automatically (well, with C-x * u).

I would then love to be able to have a table that would allow me to
include the value of any variable, e.g. z above, something like

| var | value |
|-----+-------|
| x   |     3 |
| z   |     4 |

where the values in the second column are obtained by querying Calc.

-- 
: Eric S Fraga, with org release_9.6.23-1314-g945046 in Emacs 30.0.50

Re: [PATCH] Add support for tables in Calc src block :var

[Visuwesh]

Sorry for getting back after so long!

[புதன் ஏப்ரல் 03, 2024] Fraga, Eric wrote:

> On Saturday, 30 Mar 2024 at 18:34, Visuwesh wrote:
>>> Specifically, I would love to make reference to calc variables,
>>> especially those defined using embedded calc, in org tables.
>>
>> Can you please provide a minimal example for me to play around with?  I
>> realise I would like something like this too [*] but I don't know
>> concretely what this would/should look like.
>
> So, embedded calc processes expressions in any buffer, including org
> mode, which might look like this:
>
> x := 3
>
> y := 5
>
> z := 3 x - y => 4
>
> where, in this case, the value of z has been determined by calc and the
> answer given after the => in the line.  The beauty of embedded calc is
> you can change the value of x and the subsequent expressions will be
> updated automatically (well, with C-x * u).
>
> I would then love to be able to have a table that would allow me to
> include the value of any variable, e.g. z above, something like
>
> | var | value |
> |-----+-------|
> | x   |     3 |
> | z   |     4 |
>
> where the values in the second column are obtained by querying Calc.

Here's a hack I cooked up:

    (defun vz/calc-embedded-get-var (var)
      "Return the value of active `calc-embedded' VAR in current buffer."
      (let* ((info (cdr (assq (current-buffer) calc-embedded-active)))
             (var-info
              (seq-find
               (lambda (x)
                 ;; 9 is the variable name: (var XXX var-XXX)
                 (eq var (nth 1 (aref x 9))))
               info))
             old-val)
        (when (and info var-info)
          ;; This is called the `old-val' in `calc-embedded-update'.
          ;; This can be nil when the formula isn't evaled I think?
          ;; (aref VAR-INFO 8) is again repeated in 11th slot when the
          ;; variable is simply as assignment.
          (or (if (consp (setq old-val (aref var-info 11)))
                  (car (last old-val))
                old-val)
              ;; 8 is the eval form: (calcFun-evalto ...) or
              ;; (calcFun-assign ...)
              (car (last (aref var-info 8)))))))

    (define-advice org-table-get-constant (:around (oldfun name) vz/consider-calc-embedded-var)
      "Check if NAME is a `calc-embedded' at last."
      (let ((val (funcall oldfun name)))
        (if (equal val "#UNDEFINED_NAME")
            (or (number-to-string (vz/calc-embedded-get-var (intern name)))
                val)
          val)))

Some caveats:

  1. You need to ensure that all the calc-embedded variables that you
     use in the formula need to be active and evaluated beforehand.
  2. The calc-embedded var is considered at last after everything else
     in org-table-get-constant in the advice.  This would be the best
     way to go forward.

It would be nice to lift the restrict in (1) but I think it would be
better to leave it to the user to ensure everything stays updated since
the user may want to use the old value.  However, ensuring all the vars
are active and eval-ed would be a royal pain when you're quickly
evaluating a table formula.

Here's the case I used to test this hack:

    x := 3

    y := 5

    z := 5 x - y => 10


    | 1 | 20 |
    #+TBLFM: $2=$z*2

Re: [PATCH] Add support for tables in Calc src block :var

[Visuwesh]

[வெள்ளி டிசம்பர் 13, 2024] Visuwesh wrote:

> [...]
> Some caveats:
>
>   1. You need to ensure that all the calc-embedded variables that you
>      use in the formula need to be active and evaluated beforehand.
>   2. The calc-embedded var is considered at last after everything else
>      in org-table-get-constant in the advice.  This would be the best
>      way to go forward.
>
> It would be nice to lift the restrict in (1) but I think it would be
> better to leave it to the user to ensure everything stays updated since
> the user may want to use the old value.  However, ensuring all the vars
> are active and eval-ed would be a royal pain when you're quickly
> evaluating a table formula.
>
> Here's the case I used to test this hack:
>
>     x := 3
>
>     y := 5
>
>     z := 5 x - y => 10
>
>
>     | 1 | 20 |
>
>     #+TBLFM: $2=$z*2

OK, this was not so simple as I thought!  Consider the following
example:

    x := 3

    a := 3 x => 9

    y := 5

    x := 2

    z := 5.2 x - y => 5.4

Update x := 3 and y first.  Then update `a' and `z'.  `z' will be 10.6.
Now update x := 2, and update `z' and `a'.  `z' updates to 5.4 as shown
above but `a' remains 9!  Calc looks for the nearest value of the
variable `x' from the definition of `z'.  The Elisp variable that stores
the var information looks like this:

    (cdar calc-embedded-active)

    ([#<buffer scratch.org> #<buffer *Calculator*> #<marker at 289961 in
           scratch.org> #<marker at 289967 in scratch.org> #<marker at
           289960 in scratch.org> #<marker at 289968 in scratch.org>
           #("x := 3" 0 6 (fontified t)) nil
           (calcFunc-assign (var x var-x) 3) (var x var-x) nil
           (calcFunc-assign (var x var-x) 3) nil nil nil nil]
     [#<buffer scratch.org> #<buffer *Calculator*> #<marker at 289969 in
           scratch.org> #<marker at 289982 in scratch.org> #<marker at
           289968 in scratch.org> #<marker at 289983 in scratch.org>
           #("a := 3 x => 9" 0 13 (fontified t)) nil
           (calcFunc-evalto
            (calcFunc-assign (var a var-a) (* 3 (var x var-x))) 9)
           (var a var-a) (((var x var-x))) 9 nil nil nil nil]
     [#<buffer scratch.org> #<buffer *Calculator*> #<marker at 289984 in
           scratch.org> #<marker at 289990 in scratch.org> #<marker at
           289983 in scratch.org> #<marker at 289991 in scratch.org>
           #("y := 5" 0 6 (fontified t)) nil
           (calcFunc-assign (var y var-y) 5) (var y var-y) nil
           (calcFunc-assign (var y var-y) 5) nil nil nil nil]
     [#<buffer scratch.org> #<buffer *Calculator*> #<marker at 289992 in
           scratch.org> #<marker at 289998 in scratch.org> #<marker at
           289991 in scratch.org> #<marker at 289999 in scratch.org>
           #("x := 2" 0 6 (fontified t)) nil
           (calcFunc-assign (var x var-x) 2) (var x var-x) nil
           (calcFunc-assign (var x var-x) 2) nil nil nil nil]
     [#<buffer scratch.org> #<buffer *Calculator*> #<marker at 290000 in
           scratch.org> #<marker at 290021 in scratch.org> #<marker at
           289999 in scratch.org> #<marker at 290022 in scratch.org>
           "z := 5.2 x - y => 5.4" nil
           (calcFunc-evalto
            (calcFunc-assign (var z var-z)
                     (- (* (float 52 -1) (var x var-x))
                    (var y var-y)))
            (float 54 -1))
           (var z var-z) (((var y var-y)) ((var x var-x)))
           (float 54 -1) nil nil nil nil])

Notice how there are two entries for `x'.  Worse still, change x := 3 to
x := 3.3 and update it.  Evaluating the same expression again now yields
three entries for `x'!

    ([#<buffer scratch.org> #<buffer *Calculator*> #<marker at 289961 in
           scratch.org> #<marker at 289967 in scratch.org> #<marker at
           289960 in scratch.org> #<marker at 289970 in scratch.org>
           #("x := 3" 0 6 (fontified t)) nil
           (calcFunc-assign (var x var-x) 3) (var x var-x) nil
           (calcFunc-assign (var x var-x) 3) nil nil nil nil]
     [#<buffer scratch.org> #<buffer *Calculator*> #<marker at 289961 in
           scratch.org> #<marker at 289969 in scratch.org> #<marker at
           289960 in scratch.org> #<marker at 289970 in scratch.org>
           #("x := 3.3" 0 8 (fontified t)) nil
           (calcFunc-assign (var x var-x) (float 33 -1)) (var x var-x)
           nil (calcFunc-assign (var x var-x) (float 33 -1)) nil nil
           nil nil]
     ...
     ...
     [#<buffer scratch.org> #<buffer *Calculator*> #<marker at 289996 in
           scratch.org> #<marker at 290002 in scratch.org> #<marker at
           289995 in scratch.org> #<marker at 290003 in scratch.org>
           #("x := 2" 0 6 (fontified t)) nil
           (calcFunc-assign (var x var-x) 2) (var x var-x) nil
           (calcFunc-assign (var x var-x) 2) nil nil nil nil]
     ...)

Which begs the question: what should be `x'?

In any case, the code I posted above had a problem with decimals.  The
code still picks the first value of VAR that shows up in the list above.

    (defun vz/calc-embedded-get-var (var)
      "Return the value of active `calc-embedded' VAR in current buffer."
      (let* ((info (cdr (assq (current-buffer) calc-embedded-active)))
             (var-info
              (seq-find
               (lambda (x)
                 ;; 9 is the variable name: (var XXX var-XXX)
                 (eq var (nth 1 (aref x 9))))
               info))
             old-val)
        (when (and info var-info)
          ;; This is called the `old-val' in `calc-embedded-update'.
          ;; This can be nil when the formula isn't evaled I think?
          ;; (aref VAR-INFO 8) is again repeated in 11th slot when the
          ;; variable is simply as assignment.
          (let ((calc-line-numbering))
            ;; For the below trick, see `calc-embedded-update' again.
            (math-format-stack-value
             (list (or (if (and (consp (setq old-val (aref var-info 11)))
                                (eq (car old-val) 'calcFun-assign))
                           (car (last old-val))
                         old-val)
                       ;; 8 is the eval form: (calcFun-evalto ...) or
                       ;; (calcFun-assign ...)
                       (car (last (aref var-info 8))))
                   1 nil))))))

    (define-advice org-table-get-constant (:around (oldfun name) vz/consider-calc-embedded-var)
      "Check if NAME is a `calc-embedded' at last."
      (let ((val (funcall oldfun name)))
        (if (equal val "#UNDEFINED_NAME")
            (or (vz/calc-embedded-get-var (intern name))
                val)
          val)))

Re: [PATCH] Add support for tables in Calc src block :var

[Fraga, Eric]

Response below/inline for email Visuwesh wrote:
> (original email sent 13 Dec 2024 at 14:32)
> 
> Update x := 3 and y first.  Then update `a' and `z'.  `z' will be 10.6.
> Now update x := 2, and update `z' and `a'.  `z' updates to 5.4 as shown
> above but `a' remains 9!  

Yes, this is expected behaviour.  Calc assumes a sequential evaluation,
especially when the same variable is given a value more than once.  It's
not like most modelling languages which look at the ensemble of
equations simultaneously.

[...]

> Which begs the question: what should be `x'?

Ideally, it would follow the Calc convention and would be the value last
given (before point, where the table is), I guess.

But, at least, for simple cases where variables are not overwritten,
what you have done would be sufficient (and a really nice addition to
the Calc package!).

I intend to use it for sure.

Thank you,
eric

-- 
: Eric S Fraga, with org 9.7.17-3ae179 in Emacs 31.0.50

Re: [PATCH] Add support for tables in Calc src block :var

[Visuwesh]

[வெள்ளி டிசம்பர் 13, 2024] Fraga, Eric wrote:

> Response below/inline for email Visuwesh wrote:
>> (original email sent 13 Dec 2024 at 14:32)
>> 
>> Update x := 3 and y first.  Then update `a' and `z'.  `z' will be 10.6.
>> Now update x := 2, and update `z' and `a'.  `z' updates to 5.4 as shown
>> above but `a' remains 9!  
>
> Yes, this is expected behaviour.  Calc assumes a sequential evaluation,
> especially when the same variable is given a value more than once.  It's
> not like most modelling languages which look at the ensemble of
> equations simultaneously.

I assumed this was the case.  It does make sense when you take the whole
picture together.  You are likely to use the same variable name (such as
T for temperature) everywhere so it would be best to use the nearest
one.

> [...]
>
>> Which begs the question: what should be `x'?
>
> Ideally, it would follow the Calc convention and would be the value last
> given (before point, where the table is), I guess.

When we have

    x := 3

    | 1 | 3 |
    #+TBLFM:$2=$x

    x := 6

    | 1 | 6 |
    #+TBLFM:$2=$x

value of `x' for the first table should be 3, and for the second 6.
However, when we have something like

    x := 3
    a := 2 * x
    x := 2
    z := 0.5 * x

    | 1 | |
    #+TBLFM:$2=$x*$z

what should $2 evaluate to?  $z is straight-forward: $z=1.  But what
about $x?  Should it be 2 or 3?  If we go by Calc's convention, it
should be 3.  Do you agree?

> But, at least, for simple cases where variables are not overwritten,
> what you have done would be sufficient (and a really nice addition to
> the Calc package!).
>
> I intend to use it for sure.

I am glad to be of help, and hope the quick hack actually works
reliably.

Re: [PATCH] Add support for tables in Calc src block :var

[Fraga, Eric]

Response below/inline for email Visuwesh wrote:
> (original email sent 13 Dec 2024 at 16:36)
> 
> However, when we have something like
>
>     x := 3
>     a := 2 * x
>     x := 2
>     z := 0.5 * x
>
>     | 1 | |
>     #+TBLFM:$2=$x*$z
>
> what should $2 evaluate to?  $z is straight-forward: $z=1.  But what
> about $x?  Should it be 2 or 3?  If we go by Calc's convention, it
> should be 3.  Do you agree?

I agree: given the location of the table, I would assume that x would be
2 when referenced in the table.

-- 
: Eric S Fraga, with org 9.7.17-3ae179 in Emacs 31.0.50