columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

Hello Org experts,

it seems that the time summing behaviour of columnview dynamic blocks is
different for CLOCKSUM than for EFFORT columns with respect to how the
contributions from sub-headlines are handled. When summing up CLOCKSUM
columns, a parent headline can have its own clocked time, which gets
added to the sum of its sub-items' clocked times to produce its CLOCKSUM
value. When summing up EFFORT columns, any effort a parent headline may
have been manually assigned gets overwritten with the sum of its
sub-items' efforts, however. In the example at the end of this message,
compare the results for tasks A and D. If you change the effort for
either task B or C, and then update the dynamic block, the EFFORT in the
property drawer of task A will get overwritten with the new sum of B's
and C's efforts.

I'd have two questions regarding this:

Does anyone recall the rationale for this different behaviour?

Is there any way to change the summation behaviour for either or both
column types?


I have no preference for either behaviour; they both will have their
merits and applications, depending on how one organises things. I do
think that being able to get the same behaviour would seem advantageous,
however.


Many thanks and looking forward to your thoughts,

  --alexander


---------------------------- Begin Quote -----------------------------
* Example Project
#+BEGIN: columnview :maxlevel 4 :skip-empty-rows t :indent t :format "%70ITEM(Task) %17Effort(Estimated){:} %17CLOCKSUM(Clocked){:}" :id local
| Task            | Estimated | Clocked  |
|-----------------+-----------+----------|
| Example Project |   6d 0:00 | 15d 0:00 |
| \_  Task A      |   3d 0:00 | 9d 0:00  |
| \_    Task B    |        1d | 3d 0:00  |
| \_    Task C    |        2d | 3d 0:00  |
| \_  Task D      |   3d 0:00 | 6d 0:00  |
| \_    Task E    |        1d | 3d 0:00  |
| \_    Task F    |        2d | 3d 0:00  |
#+END:
** Task A
:PROPERTIES:
:EFFORT:   3d 0:00
:END:
:LOGBOOK:
CLOCK: [2023-12-08 Fri 15:51 +0100]--[2023-12-09 Sat 01:50 +0100] =>  9:59
CLOCK: [2022-01-07 Fri 18:35 +0100]--[2022-01-08 Sat 08:36 +0100] => 14:10
:END:
*** Task B
:PROPERTIES:
:Effort:   1d
:END:
:LOGBOOK:
CLOCK: [2023-12-08 Fri 15:51 +0100]--[2023-12-09 Sat 01:50 +0100] =>  9:59
CLOCK: [2022-01-07 Fri 18:35 +0100]--[2022-01-08 Sat 08:36 +0100] => 14:10
:END:
*** Task C
:PROPERTIES:
:Effort:   2d
:END:
:LOGBOOK:
CLOCK: [2023-12-08 Fri 15:51 +0100]--[2023-12-09 Sat 01:50 +0100] =>  9:59
CLOCK: [2022-01-07 Fri 18:35 +0100]--[2022-01-08 Sat 08:36 +0100] => 14:10
:END:
** Task D
*** Task E
:PROPERTIES:
:Effort:   1d
:END:
:LOGBOOK:
CLOCK: [2023-12-08 Fri 15:51 +0100]--[2023-12-09 Sat 01:50 +0100] =>  9:59
CLOCK: [2022-01-07 Fri 18:35 +0100]--[2022-01-08 Sat 08:36 +0100] => 14:10
:END:
*** Task F
:PROPERTIES:
:Effort:   2d
:END:
:LOGBOOK:
CLOCK: [2023-12-08 Fri 15:51 +0100]--[2023-12-09 Sat 01:50 +0100] =>  9:59
CLOCK: [2022-01-07 Fri 18:35 +0100]--[2022-01-08 Sat 08:36 +0100] => 14:10
:END:
----------------------------- End Quote ------------------------------

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> it seems that the time summing behaviour of columnview dynamic blocks is
> different for CLOCKSUM than for EFFORT columns with respect to how the
> contributions from sub-headlines are handled. When summing up CLOCKSUM
> columns, a parent headline can have its own clocked time, which gets
> added to the sum of its sub-items' clocked times to produce its CLOCKSUM
> value. When summing up EFFORT columns, any effort a parent headline may
> have been manually assigned gets overwritten with the sum of its
> sub-items' efforts, however. In the example at the end of this message,
> compare the results for tasks A and D. If you change the effort for
> either task B or C, and then update the dynamic block, the EFFORT in the
> property drawer of task A will get overwritten with the new sum of B's
> and C's efforts.
>
> I'd have two questions regarding this:
>
> Does anyone recall the rationale for this different behaviour?

The "default" behaviour is to store summary of all the child property
value in each parent.

For example, starting from

#+COLUMNS: %ITEM%EFFORT{:}
* 1
:PROPERTIES:
:EFFORT:   0:00
:END:
** 1.1
:PROPERTIES:
:EFFORT:   0:20
:END:
** 1.2
:PROPERTIES:
:EFFORT:   0:20
:END:

after generating column view, you will get

* 1
:PROPERTIES:
:EFFORT:   0:40
:END:
...

This is, however, not possible for CLOCKSUM because clocksum is not an
actual property, but a "special" one - it is derived from logbook data.
That's why its behavior is different.

In fact, CLOCKSUM property does not support custom summaries.

> Is there any way to change the summation behaviour for either or both
> column types?

It is currently hard-coded. (Although, it is not too hard add some kind
of switch).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

Hello Ihor,

Many thanks for your swift response.

Ihor Radchenko <yantar92@posteo.net> writes:

> [...]
>> Does anyone recall the rationale for this different behaviour?
>
> The "default" behaviour is to store summary of all the child property
> value in each parent.
> [...]
> This is, however, not possible for CLOCKSUM because clocksum is not an
> actual property, but a "special" one - it is derived from logbook data.
> That's why its behavior is different.

I see; thanks for explaining the rationale.

Practically this seems to imply that for doing estimates, and to
minimise my own confusion, I should probably use a strategy where I put
all the effort properties in a subtree at the same level (for instance
leaf-nodes-only, or top-nodes-only). And additionally, I should never
modify effort properties by hand, but using the columnview overlay only
(since that will not allow me to modify efforts computed from child
nodes).

> In fact, CLOCKSUM property does not support custom summaries.

???

>> Is there any way to change the summation behaviour for either or both
>> column types?
>
> It is currently hard-coded. (Although, it is not too hard add some kind
> of switch).
> [...]

I think that instead of a switch, I would prefer the columnview dblock
to get a :formatter added. Knowing how the values have been computed in
the dblock's write function, I can re-calculate whatever data I need in
the formatter.

I am already using this approach successfully with the clocktable dblock
to generate invoices for me.

Compared to a new switch, it would seem to me that adding a :formatter
to the columnview dblock has several advantages:

- it would likely be a smaller code change;

- instead of implementing a single, new behaviour (activated by switch)
  it would give users the flexibility to implement any new behaviour
  they might want (user-supplied :formatter function).


Thus, from my point of view, having a :formatter for the columnview
dblock would be quite fabulous. 🦄😜


Cheers, and looking forward to your thoughts,

  --alexander

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

>> In fact, CLOCKSUM property does not support custom summaries.
>
> ???

AFAIU, you cannot do %CLOCKSUM{max}.

>>> Is there any way to change the summation behaviour for either or both
>>> column types?
>>
>> It is currently hard-coded. (Although, it is not too hard add some kind
>> of switch).
>> [...]
>
> I think that instead of a switch, I would prefer the columnview dblock
> to get a :formatter added. Knowing how the values have been computed in
> the dblock's write function, I can re-calculate whatever data I need in
> the formatter.
>
> I am already using this approach successfully with the clocktable dblock
> to generate invoices for me.
>
> Compared to a new switch, it would seem to me that adding a :formatter
> to the columnview dblock has several advantages:
>
> - it would likely be a smaller code change;
>
> - instead of implementing a single, new behaviour (activated by switch)
>   it would give users the flexibility to implement any new behaviour
>   they might want (user-supplied :formatter function).
>
>
> Thus, from my point of view, having a :formatter for the columnview
> dblock would be quite fabulous. 🦄😜

This will not prevent the property values from being changed by column
view.

In any case, making column views more flexible is welcome.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

Ihor Radchenko <yantar92@posteo.net> writes:

> Alexander Adolf <alexander.adolf@condition-alpha.com> writes:
>
>>> In fact, CLOCKSUM property does not support custom summaries.
>>
>> ???
>
> AFAIU, you cannot do %CLOCKSUM{max}.

Ah, I see; that's what you meant.

>> [...]
>> Thus, from my point of view, having a :formatter for the columnview
>> dblock would be quite fabulous. 🦄😜
>
> This will not prevent the property values from being changed by column
> view.

Correct. But I don't see that as a problem, as this has been the
behaviour of effort properties since - always.

> In any case, making column views more flexible is welcome.
> [...]

😇 Lure me into contributing a patch? fair enough; I might just as well
give that a try. I presume I'd get some support here on this list?
And/or on IRC (e.g. libera.chat#org-mode)?

I have signed FSF copyright assignment for all of Emacs, so I guess
nothing would need to be done at the paperwork level?


Cheers,

  --alexander

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

>> In any case, making column views more flexible is welcome.
>> [...]
>
> 😇 Lure me into contributing a patch? fair enough; I might just as well
> give that a try. I presume I'd get some support here on this list?
> And/or on IRC (e.g. libera.chat#org-mode)?

Yup. You are free to ask questions on the mailing list.
You may also try to ask questions in IRC or Matrix:
- https://web.libera.chat/#org-mode
- https://matrix.to/#/%23org-mode:matrix.org
Or even live, during online meetup:
- https://list.orgmode.org/18ed37bf7da.110d936ff448640.1616496967933735750@excalamus.com/T/#t

See https://orgmode.org/worg/org-contribute.html for more instructions.

> I have signed FSF copyright assignment for all of Emacs, so I guess
> nothing would need to be done at the paperwork level?

Org mode is a part of Emacs. So, your copyright assignment for Emacs is
enough.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

[-- Attachment #1: Type: text/plain, Size: 3663 bytes --]

Many thanks for the pointers, Ihor!

The instructions were clear and concise, so the patch creation went
smoothly. Results attached below.

I cloned the Org repository from git://git.sv.gnu.org/emacs/org-mode.git
and followed the instructions at
https://orgmode.org/worg/org-contribute.html#first-patch. I hope I
succeeded for most part. ;-)

`make compile` didn't complain at all, and `make test` ended with the
following:

---------------------------- Begin Quote -----------------------------
Ran 1254 tests, 1238 results as expected, 4 unexpected, 12 skipped (2024-04-15 18:39:50+0200, 57.383197 sec)
2 expected failures

4 unexpected results:
   FAILED  ob-calc/matrix-inversion  ((should (equal "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]" (org-babel-execute-src-block))) :form (equal "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]" "[[-1, 625e-3, -125e-3], [250e-3, -500e-3, 250e-3], [500e-3, 125e-3, -125e-3]]") :value nil :explanation (arrays-of-different-length 63 77 "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]" "[[-1, 625e-3, -125e-3], [250e-3, -500e-3, 250e-3], [500e-3, 125e-3, -125e-3]]" first-mismatch-at 6))
   FAILED  test-ob-shell/bash-uses-assoc-arrays  ((should (equal "two" (org-trim (org-babel-execute-src-block)))) :form (equal "two" "three") :value nil :explanation (arrays-of-different-length 3 5 "two" "three" first-mismatch-at 1))
   FAILED  test-ob-shell/bash-uses-assoc-arrays-with-lists  ((should (equal "20 cm" (org-trim (org-babel-execute-src-block)))) :form (equal "20 cm" "50 dl") :value nil :explanation (array-elt 0 (different-atoms (50 "#x32" "?2") (53 "#x35" "?5"))))
   FAILED  test-org-table/sort-lines  ((should (equal "| a | x |\n| B | 4 |\n| c | 3 |\n" (org-test-with-temp-text "| <point>a | x |\n| c | 3 |\n| B | 4 |\n" (org-table-sort-lines nil 97) (buffer-string)))) :form (equal "| a | x |\n| B | 4 |\n| c | 3 |\n" #("| B | 4 |\n| a | x |\n| c | 3 |\n" 0 9 (face org-table) 9 10 (face org-table-row) 10 19 (face org-table) 19 20 (face org-table-row) 20 29 (face org-table) 29 30 (face org-table-row))) :value nil :explanation (array-elt 2 (different-atoms (97 "#x61" "?a") (66 "#x42" "?B"))))

12 skipped results:
  SKIPPED  org-missing-dependency/test-ob-R  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-clojure  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-haskell-ghci  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-julia  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-lua  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-maxima  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-octave  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-python  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-ruby  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-scheme  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-org-attach-git  ((skip-unless nil) :form nil :value nil)
  SKIPPED  test-org-fold/org-fold-display-inline-images  ((skip-unless (not noninteractive)) :form (not t) :value nil)

make: *** [test] Error 1
----------------------------- End Quote ------------------------------

At first glance, nothing seems to point into a direction suggesting that
I had broken anything (i.e. nothing about any breakage relating to
columnview)?


Many thanks and looking forward to your thoughts,

  --alexander



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-lisp-org-colview.el-add-formatter-parameter-to-colvi.patch --]
[-- Type: text/x-patch, Size: 6568 bytes --]

From c21b18475dd154eed8d3f2489daa5a468531dc82 Mon Sep 17 00:00:00 2001
From: Alexander Adolf <alexander.adolf@condition-alpha.com>
Date: Sun, 14 Apr 2024 18:14:05 +0200
Subject: [PATCH 1/2] lisp/org-colview.el: add formatter parameter to colview
 dynamic block

* lisp/org-colview.el (org-dblock-write:column view): Factor out the
existing formatting code to new function
org-columns-dblock-write-default, and honour new dblock parameter
:formatter for specifying a different formatting function.
(org-columns-dblock-write-default): New function with current
formatting code.
(org-columns--capture-view): Amend docstring to better explain the
format of the data being passed to the formatting function.
(org-clock-clocktable-formatter): New option to define a global
default formatting function, defaulting to the current behaviour.
* doc/org-manual.org (Capturing column view): Describe new :formatter
parameter.
* etc/ORG-NEWS (New option ~org-columns-dblock-formatter~): Announce
new option.
(=colview= dynamic block supports custom formatting function):
Describe new custom formatting function feature.
---
 doc/org-manual.org  |  6 ++++++
 etc/ORG-NEWS        | 26 ++++++++++++++++++++++++++
 lisp/org-colview.el | 38 ++++++++++++++++++++++++++++++++------
 3 files changed, 64 insertions(+), 6 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 6c699ae1e..e66e48cda 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -6012,6 +6012,12 @@ This dynamic block has the following parameters:
   Specify a column attribute (see [[*Column attributes]]) for the dynamic
   block.
 
+- =:formatter= ::
+
+  A function for formatting the data in the dynamic block, overriding
+  the default formatting function set in
+  ~org-columns-dblock-formatter~.
+
 The following commands insert or update the dynamic block:
 
 - ~org-columns-insert-dblock~ ::
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index e61bd6988..eab181bc3 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -999,6 +999,16 @@ even though it does not have its own ID.  By giving files top-level id
 properties, links to headlines in the file can also be made more
 robust by using the file id instead of the file path.
 
+*** New option ~org-columns-dblock-formatter~
+
+=colview= dynamic blocks now understand a new ~:formatter~ parameter
+to use a specific function for formatting the contents of the dynamic
+block. This new option can be used to set the global default
+formatting function that will be used for =colview= dynamic blocks
+that do not specify any ~:formatter~ parameter. Its default value (the
+new function ~org-columns-dblock-write-default~) yields the previous
+(fixed) formatting behaviour.
+
 ** New features
 *** =colview= dynamic block now writes column width specifications
 
@@ -1030,6 +1040,22 @@ Example:
 : | PROYECTO EMACS |
 : #+END:
 
+*** =colview= dynamic block supports custom formatting function
+
+The =colview= dynamic block understands a new ~:formatter~ parameter,
+which specifies a user-supplied function to format the data in the
+dynamic block.
+
+A global default formatting function for =colview= dynamic blocks can
+be set via the new option ~org-columns-dblock-formatter~ which
+defaults to the new function ~org-columns-dblock-write-default~, that
+implements the previous (fixed) formatting behaviour. Hence, the
+default behaviour is identical to previous versions.
+
+The global default function can be overridden for any given =colview=
+dynamic block individually by specifying a custom formatter function
+using the new ~:formatter~ parameter on the block's =BEGIN= line.
+
 *** =ob-tangle.el=: New flag to remove tangle targets before writing
 
 When ~org-babel-tangle-remove-file-before-write~ is set to ~t~ the
diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index 4718d1db7..5c908affe 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -123,6 +123,13 @@ in `org-columns-summary-types-default', which see."
 			      (function :tag "Summarize")
 			      (function :tag "Collect")))))
 
+(defcustom org-columns-dblock-formatter #'org-columns-dblock-write-default
+  "Function to format data in column view dynamic blocks.
+For more information, see `org-columns-dblock-write-default'."
+  :group 'org-properties
+  :version "30.0"
+  :package-version '(Org . "9.7")
+  :type 'function)
 
 \f
 ;;; Column View
@@ -1442,9 +1449,13 @@ that will be excluded from the resulting view.  FORMAT is a
 format string for columns, or nil.  When LOCAL is non-nil, only
 capture headings in current subtree.
 
-This function returns a list containing the title row and all
-other rows.  Each row is a list of fields, as strings, or
-`hline'."
+This function returns a list containing the title row and all other
+rows.  Each row is either a list, or the symbol `hline'.  The first list
+is the heading row as a list of strings with the column titles according
+to FORMAT.  All subsequent lists each represent a body row as a list
+whose first element is an integer indicating the outline level of the
+entry, and whose remaining elements are strings with the contents for
+the columns according to FORMAT."
   (org-columns (not local) format)
   (goto-char org-columns-top-level-marker)
   (let ((columns (length org-columns-current-fmt-compiled))
@@ -1545,8 +1556,13 @@ PARAMS is a property list of parameters:
 `:vlines'
 
     When non-nil, make each column a column group to enforce
-    vertical lines."
-  (let ((table
+    vertical lines.
+
+`:formatter'
+
+    A function to format the data and insert it into the buffer."
+  (let ((insert-pos (point))
+        (table
 	 (let ((id (plist-get params :id))
 	       view-file view-pos)
 	   (pcase id
@@ -1573,7 +1589,17 @@ PARAMS is a property list of parameters:
 					 (plist-get params :exclude-tags)
 					 (plist-get params :format)
 					 view-pos)))))
-        (width-specs
+        (formatter (or (plist-get params :formatter)
+                       org-columns-dblock-formatter
+                       #'org-columns-dblock-write-default)))
+    (funcall formatter insert-pos table params)))
+
+(defun org-columns-dblock-write-default (ipos table params)
+  "Write out a columnview table at position IPOS in the current buffer.
+TABLE is a table with data as produced by `org-columns--capture-view'.
+PARAMS is the parameter property list obtained from the dynamic block
+definition."
+  (let ((width-specs
          (mapcar (lambda (spec) (nth 2 spec))
                  org-columns-current-fmt-compiled)))
     (when table
-- 
2.39.3 (Apple Git-146)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-lisp-org-colview.el-add-link-parameter-to-colview-dy.patch --]
[-- Type: text/x-patch, Size: 4981 bytes --]

From 4d69c3407c7c00fe0a1f52c383ced572f3524076 Mon Sep 17 00:00:00 2001
From: Alexander Adolf <alexander.adolf@condition-alpha.com>
Date: Mon, 15 Apr 2024 18:01:40 +0200
Subject: [PATCH 2/2] lisp/org-colview.el: add link parameter to colview
 dynamic block

* lisp/org-colview.el (org-columns--capture-view): Add new link
parameter, which when non-nil causes ITEM headlines to be linked to
their origins.
(org-dblock-write:columnview): Pass new link parameter to
org-columns--capture-view, and explain its use in the docstring.
* doc/org-manual.org (Capturing column view): Describe new :link
parameter.
* etc/ORG-NEWS (=colview= dynamic block can link to headlines):
Describe new link feature.
---
 doc/org-manual.org  |  5 +++++
 etc/ORG-NEWS        |  6 ++++++
 lisp/org-colview.el | 34 ++++++++++++++++++++++++----------
 3 files changed, 35 insertions(+), 10 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index e66e48cda..5e2549f49 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -6007,6 +6007,11 @@ This dynamic block has the following parameters:
 
   When non-~nil~, indent each =ITEM= field according to its level.
 
+- =:link= ::
+
+  When non-~nil~, link the =ITEM= headlines in the table to their
+  origins.
+
 - =:format= ::
 
   Specify a column attribute (see [[*Column attributes]]) for the dynamic
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index eab181bc3..449487ac0 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -1056,6 +1056,12 @@ The global default function can be overridden for any given =colview=
 dynamic block individually by specifying a custom formatter function
 using the new ~:formatter~ parameter on the block's =BEGIN= line.
 
+*** =colview= dynamic block can link to headlines
+
+The =colview= dynamic block understands a new ~:link~ parameter, which
+when non-~nil~ causes =ITEM= headlines in the table to be linked to
+their origins.
+
 *** =ob-tangle.el=: New flag to remove tangle targets before writing
 
 When ~org-babel-tangle-remove-file-before-write~ is set to ~t~ the
diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index 5c908affe..1771283f2 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -1439,15 +1439,15 @@ and variances (respectively) of the individual estimates."
 \f
 ;;; Dynamic block for Column view
 
-(defun org-columns--capture-view (maxlevel match skip-empty exclude-tags format local)
+(defun org-columns--capture-view (maxlevel match skip-empty exclude-tags link format local)
   "Get the column view of the current buffer.
 
-MAXLEVEL sets the level limit.  SKIP-EMPTY tells whether to skip
-empty rows, an empty row being one where all the column view
-specifiers but ITEM are empty.  EXCLUDE-TAGS is a list of tags
-that will be excluded from the resulting view.  FORMAT is a
-format string for columns, or nil.  When LOCAL is non-nil, only
-capture headings in current subtree.
+MAXLEVEL sets the level limit.  SKIP-EMPTY tells whether to skip empty
+rows, an empty row being one where all the column view specifiers but
+ITEM are empty.  EXCLUDE-TAGS is a list of tags that will be excluded
+from the resulting view.  When LINK is non-nil, item headlines will be
+linked to their origins.  FORMAT is a format string for columns, or nil.
+When LOCAL is non-nil, only capture headings in current subtree.
 
 This function returns a list containing the title row and all other
 rows.  Each row is either a list, or the symbol `hline'.  The first list
@@ -1467,12 +1467,21 @@ the columns according to FORMAT."
 	 (let (row)
 	   (dotimes (i columns)
 	     (let* ((col (+ (line-beginning-position) i))
-		    (p (get-char-property col 'org-columns-key)))
-	       (push (org-quote-vert
+		    (p (get-char-property col 'org-columns-key))
+		    (cell-content
+		     (org-quote-vert
 		      (get-char-property col
 					 (if (string= p "ITEM")
 					     'org-columns-value
-					   'org-columns-value-modified)))
+					   'org-columns-value-modified)))))
+	       (push (if (and link (string= p "ITEM"))
+			 (let ((search (org-link-heading-search-string
+					cell-content)))
+			   (org-link-make-string
+			    (if (not (buffer-file-name)) search
+			      (format "file:%s::%s" (buffer-file-name) search))
+			    cell-content))
+		       cell-content)
 		     row)))
 	   (unless (or
 		    (and skip-empty
@@ -1558,6 +1567,10 @@ PARAMS is a property list of parameters:
     When non-nil, make each column a column group to enforce
     vertical lines.
 
+`:link'
+
+    Link the item headlines in the table to their origins.
+
 `:formatter'
 
     A function to format the data and insert it into the buffer."
@@ -1587,6 +1600,7 @@ PARAMS is a property list of parameters:
 					 (plist-get params :match)
 					 (plist-get params :skip-empty-rows)
 					 (plist-get params :exclude-tags)
+					 (plist-get params :link)
 					 (plist-get params :format)
 					 view-pos)))))
         (formatter (or (plist-get params :formatter)
-- 
2.39.3 (Apple Git-146)

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:


> Subject: [PATCH 1/2] lisp/org-colview.el: add formatter parameter to colview
>  dynamic block

Thanks for the patches!
See my comments below.

> * lisp/org-colview.el (org-dblock-write:column view): Factor out the
> existing formatting code to new function
> org-columns-dblock-write-default, and honour new dblock parameter
> :formatter for specifying a different formatting function.

In the changelog entries, we quote the symbol and variable names like
`this'.

> +- =:formatter= ::
> +
> +  A function for formatting the data in the dynamic block, overriding
> +  the default formatting function set in
> +  ~org-columns-dblock-formatter~.

You can also mention that the function also inserts the data. Something
similar to what we do when describe the equivalent option for clock tables:

    - =:formatter= ::
    
      A function to format clock data and insert it into the buffer.

Also, if you mention a variable in the manual, please add #+vindex:
entry. Maybe even #+cindex: entry for "formatter", to make the parameter
more discoverable.

> +*** New option ~org-columns-dblock-formatter~
> +
> +=colview= dynamic blocks now understand a new ~:formatter~ parameter
> +to use a specific function for formatting the contents of the dynamic
> +block. This new option can be used to set the global default

... and inserting its contents.

It would also be helpful to refer to the existing :formatter parameter
in clock tables.

> +*** =colview= dynamic block supports custom formatting function
> +
> +The =colview= dynamic block understands a new ~:formatter~ parameter,
> +which specifies a user-supplied function to format the data in the

... and insert the data

> +(defcustom org-columns-dblock-formatter #'org-columns-dblock-write-default
> +  "Function to format data in column view dynamic blocks.
> +For more information, see `org-columns-dblock-write-default'."
> +  :group 'org-properties
> +  :version "30.0"

We do not need :version tag when there is :package-version.

> +(defun org-columns-dblock-write-default (ipos table params)
> +  "Write out a columnview table at position IPOS in the current buffer.
> +TABLE is a table with data as produced by `org-columns--capture-view'.
> +PARAMS is the parameter property list obtained from the dynamic block
> +definition."
> +  (let ((width-specs
>           (mapcar (lambda (spec) (nth 2 spec))
>                   org-columns-current-fmt-compiled)))
>      (when table

This new function ignores IPOS parameter. It should not.
 
> From 4d69c3407c7c00fe0a1f52c383ced572f3524076 Mon Sep 17 00:00:00 2001
> From: Alexander Adolf <alexander.adolf@condition-alpha.com>
> Date: Mon, 15 Apr 2024 18:01:40 +0200
> Subject: [PATCH 2/2] lisp/org-colview.el: add link parameter to colview
>  dynamic block

*Add
(capitalize)

> * lisp/org-colview.el (org-columns--capture-view): Add new link
> parameter, which when non-nil causes ITEM headlines to be linked to
> their origins.
> (org-dblock-write:columnview): Pass new link parameter to
> org-columns--capture-view, and explain its use in the docstring.

`org-columns--capture-view'

> -(defun org-columns--capture-view (maxlevel match skip-empty exclude-tags format local)
> +(defun org-columns--capture-view (maxlevel match skip-empty exclude-tags link format local)

While this is an internal function and we are free to change it as we
need, it is generally more robust to not make changes in the order of
function arguments. I recommend

(defun org-columns--capture-view (maxlevel match skip-empty exclude-tags format local &optional link)

> +	       (push (if (and link (string= p "ITEM"))
> +			 (let ((search (org-link-heading-search-string
> +					cell-content)))
> +			   (org-link-make-string
> +			    (if (not (buffer-file-name)) search
> +			      (format "file:%s::%s" (buffer-file-name) search))
> +			    cell-content))

In org-clock, we do

(org-link-make-string
			     (if (not (buffer-file-name)) search
			       (format "file:%s::%s" (buffer-file-name) search))
			     ;; Prune statistics cookies.  Replace
			     ;; links with their description, or
			     ;; a plain link if there is none.
			     (org-trim
			      (org-link-display-format
			       (replace-regexp-in-string
				"\\[[0-9]*\\(?:%\\|/[0-9]*\\)\\]" ""
				headline))))

Is there any reason why you did not remove the statistics cookies here
as well?

> `make compile` didn't complain at all, and `make test` ended with the
> following:
> ...
> 4 unexpected results:
>    FAILED  ob-calc/matrix-inversion
>    FAILED  test-ob-shell/bash-uses-assoc-arrays
>    FAILED  test-ob-shell/bash-uses-assoc-arrays-with-lists
>    FAILED  test-org-table/sort-lines

MacOS? There are known issues with locale rules in MacOS that may cause
test failures.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

Ihor Radchenko <yantar92@posteo.net> writes:

> Alexander Adolf <alexander.adolf@condition-alpha.com> writes:
>
>> Subject: [PATCH 1/2] lisp/org-colview.el: add formatter parameter to colview
>>  dynamic block
>
> Thanks for the patches!
> See my comments below.

Thanks for your swift review, and most helpful comments!

While I'm implementing these, reactions of my own on a select, few
comments of yours:

> [...]
>> +- =:formatter= ::
>> +
>> +  A function for formatting the data in the dynamic block, overriding
>> +  the default formatting function set in
>> +  ~org-columns-dblock-formatter~.
>
> You can also mention that the function also inserts the data. Something
> similar to what we do when describe the equivalent option for clock tables:
>
>     - =:formatter= ::
>
>       A function to format clock data and insert it into the buffer.
>
> Also, if you mention a variable in the manual, please add #+vindex:
> entry. Maybe even #+cindex: entry for "formatter", to make the parameter
> more discoverable.

I kept it to the format of the existing parameter descriptions, which
don't create index entries. Happy to add one though. #+cindex would seem
more appropriate, as it's not a variable?

On a loosely related note: the description of the :formatter parameter
of the clock table does not have and index entry either. Should it get
one too, then?

Btw, I will also move the half-sentence about overriding the default
formatting function from the manual to the docstring of
org-dblock-write:columnview, where the :formatter parameter is
explained, too. It somehow seems more appropriate there, since a user
who is looking into implementing a formatting function will most likely
be accessing that docstring anyway (so will find the information),
whereas the information about the customization variable is likely
adding more confusion than it tries to remove for the rest of the Org
users (who will likely consult the manual only).

> [...]
> (defun org-columns--capture-view (maxlevel match skip-empty exclude-tags format local &optional link)
>
>> +	       (push (if (and link (string= p "ITEM"))
>> +			 (let ((search (org-link-heading-search-string
>> +					cell-content)))
>> +			   (org-link-make-string
>> +			    (if (not (buffer-file-name)) search
>> +			      (format "file:%s::%s" (buffer-file-name) search))
>> +			    cell-content))
>
> In org-clock, we do
>
> (org-link-make-string
> 			     (if (not (buffer-file-name)) search
> 			       (format "file:%s::%s" (buffer-file-name) search))
> 			     ;; Prune statistics cookies.  Replace
> 			     ;; links with their description, or
> 			     ;; a plain link if there is none.
> 			     (org-trim
> 			      (org-link-display-format
> 			       (replace-regexp-in-string
> 				"\\[[0-9]*\\(?:%\\|/[0-9]*\\)\\]" ""
> 				headline))))
>
> Is there any reason why you did not remove the statistics cookies here
> as well?
> [...]

Somehow (how?) the statistics cookies get removed in my current
implementation. org-link-make-string does not remove them (I double
checked). I would thus speculate that perhaps the overlay creation (to
show description only) removes them? OTOH, I'm happy to add the
org-trim part to make things more robust.


Will email updated patches when I will have addressed all your comments.


Many thanks and cheers,

  --alexander

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

>> Also, if you mention a variable in the manual, please add #+vindex:
>> entry. Maybe even #+cindex: entry for "formatter", to make the parameter
>> more discoverable.
>
> I kept it to the format of the existing parameter descriptions, which
> don't create index entries. Happy to add one though. #+cindex would seem
> more appropriate, as it's not a variable?

When suggesting #+vindex, I was referring to
org-columns-dblock-formatter variable.

> On a loosely related note: the description of the :formatter parameter
> of the clock table does not have and index entry either. Should it get
> one too, then?

Within the scope of this patch, it is enough to add the index entry to
the newly added parameter.

More generally, we do want index entries for various parameters in
dynamic blocks and clock tables. As we do for header arguments:

#+cindex: @samp{file}, header argument

But that should be a separate patch.

For colview dynamic blocks, cindex entry may look like

#+cindex: @samp{formatter}, dynamic block parameter

>> Is there any reason why you did not remove the statistics cookies here
>> as well?
>> [...]
>
> Somehow (how?) the statistics cookies get removed in my current
> implementation. org-link-make-string does not remove them (I double
> checked). I would thus speculate that perhaps the overlay creation (to
> show description only) removes them? OTOH, I'm happy to add the
> org-trim part to make things more robust.

I see how. It is because CELL-CONTENT is not the original heading. It is
the heading name processed with `org-columns--clean-item'.

`org-column--clean-item' removes statistics cookies among other things.
It actually removes more, leading to some edge cases in your patch:

** TODO Foo

** TODO src_elisp{"Hello"} world


#+begin: columnview :id global :link t
| <25>  |      | <3>      |      |
| ITEM  | TODO | PRIORITY | TAGS |
|-------+------+----------+------|
| [[file:/tmp/test.org::*Foo][Foo]]   | TODO | B        |      |
| [[file:/tmp/test.org::*src_elisp{"Hello"} world][world]] | TODO | B        |      |
#+end:

Note how inline src block is stripped from the link description.

We should probably also change org-clock to use
`org-columns--clean-item'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

Ihor Radchenko <yantar92@posteo.net> writes:

> [...]
>> `make compile` didn't complain at all, and `make test` ended with the
>> following:
>> ...
>> 4 unexpected results:
>>    FAILED  ob-calc/matrix-inversion
>>    FAILED  test-ob-shell/bash-uses-assoc-arrays
>>    FAILED  test-ob-shell/bash-uses-assoc-arrays-with-lists
>>    FAILED  test-org-table/sort-lines
>
> MacOS? There are known issues with locale rules in MacOS that may cause
> test failures.
> [...]

Yes, macOS. So I won't be worried about these.


  --alexander

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

Many thanks for your swift response!

Ihor Radchenko <yantar92@posteo.net> writes:

> [...]
> When suggesting #+vindex, I was referring to
> org-columns-dblock-formatter variable.
> [...]
> #+cindex: @samp{formatter}, dynamic block parameter

I see; thanks for explaining. I've added both:

---------------------------- Begin Quote -----------------------------
- =:formatter= ::

  #+cindex: @samp{formatter}, dynamic block parameter
  #+vindex: org-columns-dblock-formatter
  A function to format column view data and insert it into the buffer.
  See the option ~org-columns-dblock-formatter~.
----------------------------- End Quote ------------------------------


> [...]
> I see how. It is because CELL-CONTENT is not the original heading. It is
> the heading name processed with `org-columns--clean-item'.
>
> `org-column--clean-item' removes statistics cookies among other things.
> It actually removes more, leading to some edge cases in your patch:
>
> ** TODO Foo
>
> ** TODO src_elisp{"Hello"} world
>
>
> #+begin: columnview :id global :link t
> | <25>  |      | <3>      |      |
> | ITEM  | TODO | PRIORITY | TAGS |
> |-------+------+----------+------|
> | [[file:/tmp/test.org::*Foo][Foo]]   | TODO | B        |      |
> | [[file:/tmp/test.org::*src_elisp{"Hello"} world][world]] | TODO | B        |      |
> #+end:
>
> Note how inline src block is stripped from the link description.

Ah, `org-column--clean-item'; well spotted!

Two observations:

1) As is now, I'm generating the links in the data collection function
   `org-columns--capture-view'. As `org-column--clean-item' is called
   from code that runs after the data collection,
   `org-column--clean-item' was probably never designed to be able to
   handle strings containing links. That it still did sort of "the right
   thing" seems more luck than anything else?

2) Considering what happens when I do `org-store-link' and
   `org-insert-link', it would seem more advisable to:

   a) move the link generation to the new formatting function
      (re-removing it from `org-columns--capture-view');

   b) pass the "cleaned" string to `org-link-make-string' as both, the
      link and the description parameter.

> We should probably also change org-clock to use
> `org-columns--clean-item'.
> [...]

As a separate patch, or as a third commit to the patch we are discussing
now?


Many thanks and looking forward to your thoughts,

  --alexander

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> Ah, `org-column--clean-item'; well spotted!
>
> Two observations:
>
> 1) As is now, I'm generating the links in the data collection function
>    `org-columns--capture-view'. As `org-column--clean-item' is called
>    from code that runs after the data collection,
>    `org-column--clean-item' was probably never designed to be able to
>    handle strings containing links. That it still did sort of "the right
>    thing" seems more luck than anything else?

There is a difference between what `org-columns--clean-item' does and
stripping link paths.

`org-columns--clean-item' is cleaning up the results for insertion into
a table specifically. This is only meaningful in dynamic block, but not
in the traditional column view where nothing is actually written to
buffer - the columns are displayed on top of the existing headings.
Calling `org-columns--clean-item' is a must to create a valid table.

In contrast, the purpose of `org-link-display-format' is purely visual -
to not make the collected titles way too long. Also, it has no effect
when there is a custom `org-columns-modify-value-for-display-function'
(see `org-columns--displayed-value').

> 2) Considering what happens when I do `org-store-link' and
>    `org-insert-link', it would seem more advisable to:
>
>    a) move the link generation to the new formatting function
>       (re-removing it from `org-columns--capture-view');
>
>    b) pass the "cleaned" string to `org-link-make-string' as both, the
>       link and the description parameter.

I do not recall seeing `org-store-link' in the patches you submitted.
So, I am not sure what you are talking about. May you elaborate?

>> We should probably also change org-clock to use
>> `org-columns--clean-item'.
>> [...]
>
> As a separate patch, or as a third commit to the patch we are discussing
> now?

Up to you.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

[-- Attachment #1: Type: text/plain, Size: 1704 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> [...]
> Calling `org-columns--clean-item' is a must to create a valid table.

True.

Additionally, it would seem advisable to call `org-quote-vert' on the
data, too, as `org-columns--clean-item' does not take care of vertical
bars? This is done in a previous step in `org-columns--capture-view',
however, so that the vertical bars get converted to "\vert" before the
formatting function gets called.

`org-link-heading-search-string', and `org-link-make-string' (both
called from the formatting function _after_ `org-columns--clean-item')
OTOH take care of the link's path and description parts being
appropriate for a link.

Thus, we have both layers of safeguarding: "link-safe" data is generated
from "table-safe" data.

> [...]
>> 2) Considering what happens when I do `org-store-link' and
>>    `org-insert-link', it would seem more advisable to:
>> [...]
>
> I do not recall seeing `org-store-link' in the patches you submitted.
> So, I am not sure what you are talking about. May you elaborate?
> [...]

Apologies for not having been clear enough. With "when I do" I wanted to
refer to the "cleanup" of headline text when using `org-store-link' and
`org-insert-link' interactively.

I have moved the link generation to the formatting function as
mentioned. This also removes the need for any new function parameters,
since the formatting function receives the dynamic block's parameter
list, which contains a :link property when set in the #+BEGIN line.

Kindly find updated patches below. I hope to have succeeded in
addressing all your comments; that was my intention at least.


Many thanks and looking forward to your thoughts,

  --alexander



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-lisp-org-colview.el-Add-formatter-parameter-to-colvi.patch --]
[-- Type: text/x-patch, Size: 7333 bytes --]

From a36dba46a63d7d879bf2faf3c4c885b9ea3e28ad Mon Sep 17 00:00:00 2001
From: Alexander Adolf <alexander.adolf@condition-alpha.com>
Date: Sun, 14 Apr 2024 18:14:05 +0200
Subject: [PATCH 1/2] lisp/org-colview.el: Add formatter parameter to colview
 dynamic block

* lisp/org-colview.el (org-dblock-write:column view): Factor out the
existing formatting code to new function
`org-columns-dblock-write-default', and honour new dblock parameter
:formatter for specifying a different formatting function.
(org-columns-dblock-write-default): New function with current
formatting code.
(org-columns--capture-view): Amend docstring to better explain the
format of the data being passed to the formatting function.
(org-clock-clocktable-formatter): New option to define a global
default formatting function, defaulting to the current behaviour.
* doc/org-manual.org (Capturing column view): Describe new :formatter
parameter.
* etc/ORG-NEWS (New option ~org-columns-dblock-formatter~): Announce
new option.
(=colview= dynamic block supports custom formatting function):
Describe new custom formatting function feature.
---
 doc/org-manual.org  |  7 +++++++
 etc/ORG-NEWS        | 29 +++++++++++++++++++++++++++++
 lisp/org-colview.el | 41 +++++++++++++++++++++++++++++++++++------
 3 files changed, 71 insertions(+), 6 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index d24230568..000783772 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -6014,6 +6014,13 @@ This dynamic block has the following parameters:
   Specify a column attribute (see [[*Column attributes]]) for the dynamic
   block.
 
+- =:formatter= ::
+
+  #+cindex: @samp{formatter}, dynamic block parameter
+  #+vindex: org-columns-dblock-formatter
+  A function to format column view data and insert it into the buffer.
+  See the option ~org-columns-dblock-formatter~.
+
 The following commands insert or update the dynamic block:
 
 - ~org-columns-insert-dblock~ ::
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index dbf849422..84ea7cde9 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -999,6 +999,16 @@ even though it does not have its own ID.  By giving files top-level id
 properties, links to headlines in the file can also be made more
 robust by using the file id instead of the file path.
 
+*** New option ~org-columns-dblock-formatter~
+
+=colview= dynamic blocks now understand a new ~:formatter~ parameter
+to use a specific function for formatting and inserting the contents
+of the dynamic block. This new option can be used to set the global
+default formatting function that will be used for =colview= dynamic
+blocks that do not specify any ~:formatter~ parameter. Its default
+value (the new function ~org-columns-dblock-write-default~) yields the
+previous (fixed) formatting behaviour.
+
 ** New features
 *** ~org-paste-subtree~ now handles =C-u= and =C-u C-u= prefix arguments specially
 
@@ -1035,6 +1045,25 @@ Example:
 : | PROYECTO EMACS |
 : #+END:
 
+*** =colview= dynamic block supports custom formatting function
+
+The =colview= dynamic block understands a new ~:formatter~ parameter,
+which specifies a user-supplied function to format and insert the data
+in the dynamic block.
+
+A global default formatting function for =colview= dynamic blocks can
+be set via the new option ~org-columns-dblock-formatter~ which
+defaults to the new function ~org-columns-dblock-write-default~, that
+implements the previous (fixed) formatting behaviour. Hence, the
+default behaviour is identical to previous versions.
+
+The global default function can be overridden for any given =colview=
+dynamic block individually by specifying a custom formatter function
+using the new ~:formatter~ parameter on the block's =BEGIN= line.
+
+This new feature replicates the ~:formatter~ option already available
+for =clocktable= dynamic blocks.
+
 *** =ob-tangle.el=: New flag to remove tangle targets before writing
 
 When ~org-babel-tangle-remove-file-before-write~ is set to ~t~ the
diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index 4718d1db7..7a6471377 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -123,6 +123,12 @@ in `org-columns-summary-types-default', which see."
 			      (function :tag "Summarize")
 			      (function :tag "Collect")))))
 
+(defcustom org-columns-dblock-formatter #'org-columns-dblock-write-default
+  "Function to format data in column view dynamic blocks.
+For more information, see `org-columns-dblock-write-default'."
+  :group 'org-properties
+  :package-version '(Org . "9.7")
+  :type 'function)
 
 \f
 ;;; Column View
@@ -1442,9 +1448,13 @@ that will be excluded from the resulting view.  FORMAT is a
 format string for columns, or nil.  When LOCAL is non-nil, only
 capture headings in current subtree.
 
-This function returns a list containing the title row and all
-other rows.  Each row is a list of fields, as strings, or
-`hline'."
+This function returns a list containing the title row and all other
+rows.  Each row is either a list, or the symbol `hline'.  The first list
+is the heading row as a list of strings with the column titles according
+to FORMAT.  All subsequent lists each represent a body row as a list
+whose first element is an integer indicating the outline level of the
+entry, and whose remaining elements are strings with the contents for
+the columns according to FORMAT."
   (org-columns (not local) format)
   (goto-char org-columns-top-level-marker)
   (let ((columns (length org-columns-current-fmt-compiled))
@@ -1545,8 +1555,15 @@ PARAMS is a property list of parameters:
 `:vlines'
 
     When non-nil, make each column a column group to enforce
-    vertical lines."
-  (let ((table
+    vertical lines.
+
+`:formatter'
+
+    A function to format the data and insert it into the
+    buffer.  Overrides the default formatting function set in
+    `org-columns-dblock-formatter'."
+  (let ((insert-pos (point))
+        (table
 	 (let ((id (plist-get params :id))
 	       view-file view-pos)
 	   (pcase id
@@ -1573,7 +1590,17 @@ PARAMS is a property list of parameters:
 					 (plist-get params :exclude-tags)
 					 (plist-get params :format)
 					 view-pos)))))
-        (width-specs
+        (formatter (or (plist-get params :formatter)
+                       org-columns-dblock-formatter
+                       #'org-columns-dblock-write-default)))
+    (funcall formatter insert-pos table params)))
+
+(defun org-columns-dblock-write-default (ipos table params)
+  "Write out a columnview table at position IPOS in the current buffer.
+TABLE is a table with data as produced by `org-columns--capture-view'.
+PARAMS is the parameter property list obtained from the dynamic block
+definition."
+  (let ((width-specs
          (mapcar (lambda (spec) (nth 2 spec))
                  org-columns-current-fmt-compiled)))
     (when table
@@ -1616,6 +1643,8 @@ PARAMS is a property list of parameters:
         ;; to the resulting table, adding alignment field as the first
         ;; row.
         (push (mapcar (lambda (width) (when width (format "<%d>" width))) width-specs) table))
+      ;; now insert the table into the buffer
+      (goto-char ipos)
       (let ((content-lines (org-split-string (plist-get params :content) "\n"))
 	    recalc)
 	;; Insert affiliated keywords before the table.
-- 
2.39.3 (Apple Git-146)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-lisp-org-colview.el-Add-link-parameter-to-colview-dy.patch --]
[-- Type: text/x-patch, Size: 4950 bytes --]

From 6c3fd68419ac2be705f29bd56dcd23b01b6da86b Mon Sep 17 00:00:00 2001
From: Alexander Adolf <alexander.adolf@condition-alpha.com>
Date: Mon, 15 Apr 2024 18:01:40 +0200
Subject: [PATCH 2/2] lisp/org-colview.el: Add link parameter to colview
 dynamic block

* lisp/org-colview.el (org-columns--capture-view): Add new link
parameter, which when non-nil causes ITEM headlines to be linked to
their origins.
(org-dblock-write:columnview): Pass new link parameter to
`org-columns--capture-view', and explain its use in the docstring.
* doc/org-manual.org (Capturing column view): Describe new :link
parameter.
* etc/ORG-NEWS (=colview= dynamic block can link to headlines):
Describe new link feature.
---
 doc/org-manual.org  |  5 +++++
 etc/ORG-NEWS        |  6 ++++++
 lisp/org-colview.el | 33 +++++++++++++++++++++++----------
 3 files changed, 34 insertions(+), 10 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 000783772..a77492971 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -6009,6 +6009,11 @@ This dynamic block has the following parameters:
 
   When non-~nil~, indent each =ITEM= field according to its level.
 
+- =:link= ::
+
+  When non-~nil~, link the =ITEM= headlines in the table to their
+  origins.
+
 - =:format= ::
 
   Specify a column attribute (see [[*Column attributes]]) for the dynamic
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 84ea7cde9..b4aea10b8 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -1064,6 +1064,12 @@ using the new ~:formatter~ parameter on the block's =BEGIN= line.
 This new feature replicates the ~:formatter~ option already available
 for =clocktable= dynamic blocks.
 
+*** =colview= dynamic block can link to headlines
+
+The =colview= dynamic block understands a new ~:link~ parameter, which
+when non-~nil~ causes =ITEM= headlines in the table to be linked to
+their origins.
+
 *** =ob-tangle.el=: New flag to remove tangle targets before writing
 
 When ~org-babel-tangle-remove-file-before-write~ is set to ~t~ the
diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index 7a6471377..74f7a8f58 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -1441,12 +1441,12 @@ and variances (respectively) of the individual estimates."
 (defun org-columns--capture-view (maxlevel match skip-empty exclude-tags format local)
   "Get the column view of the current buffer.
 
-MAXLEVEL sets the level limit.  SKIP-EMPTY tells whether to skip
-empty rows, an empty row being one where all the column view
-specifiers but ITEM are empty.  EXCLUDE-TAGS is a list of tags
-that will be excluded from the resulting view.  FORMAT is a
-format string for columns, or nil.  When LOCAL is non-nil, only
-capture headings in current subtree.
+MAXLEVEL sets the level limit.  SKIP-EMPTY tells whether to skip empty
+rows, an empty row being one where all the column view specifiers but
+ITEM are empty.  EXCLUDE-TAGS is a list of tags that will be excluded
+from the resulting view.  FORMAT is a format string for columns, or nil.
+When LOCAL is non-nil, only capture headings in current subtree.  When
+LINK is non-nil, item headlines will be linked to their origins.
 
 This function returns a list containing the title row and all other
 rows.  Each row is either a list, or the symbol `hline'.  The first list
@@ -1557,6 +1557,10 @@ PARAMS is a property list of parameters:
     When non-nil, make each column a column group to enforce
     vertical lines.
 
+`:link'
+
+    Link the item headlines in the table to their origins.
+
 `:formatter'
 
     A function to format the data and insert it into the
@@ -1600,9 +1604,10 @@ PARAMS is a property list of parameters:
 TABLE is a table with data as produced by `org-columns--capture-view'.
 PARAMS is the parameter property list obtained from the dynamic block
 definition."
-  (let ((width-specs
-         (mapcar (lambda (spec) (nth 2 spec))
-                 org-columns-current-fmt-compiled)))
+  (let ((link (plist-get params :link))
+	(width-specs
+	 (mapcar (lambda (spec) (nth 2 spec))
+		 org-columns-current-fmt-compiled)))
     (when table
       ;; Prune level information from the table.  Also normalize
       ;; headings: remove stars, add indentation entities, if
@@ -1626,7 +1631,15 @@ definition."
 			   (and (numberp hlines) (<= level hlines))))
 	      (push 'hline new-table))
 	    (when item-index
-	      (let ((item (org-columns--clean-item (nth item-index (cdr row)))))
+	      (let* ((raw (nth item-index (cdr row)))
+		     (cleaned (org-columns--clean-item raw))
+		     (item (if link
+			       (let ((search (org-link-heading-search-string raw)))
+				 (org-link-make-string
+				  (if (not (buffer-file-name)) search
+				    (format "file:%s::%s" (buffer-file-name) search))
+				  cleaned))
+			     cleaned)))
 		(setf (nth item-index (cdr row))
 		      (if (and indent (> level 1))
 			  (concat "\\_" (make-string (* 2 (1- level)) ?\s) item)
-- 
2.39.3 (Apple Git-146)

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 2082 bytes --]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> [...]
>> Calling `org-columns--clean-item' is a must to create a valid table.
>
> True.
>
> Additionally, it would seem advisable to call `org-quote-vert' on the
> data, too, as `org-columns--clean-item' does not take care of vertical
> bars? This is done in a previous step in `org-columns--capture-view',
> however, so that the vertical bars get converted to "\vert" before the
> formatting function gets called.
>
> `org-link-heading-search-string', and `org-link-make-string' (both
> called from the formatting function _after_ `org-columns--clean-item')
> OTOH take care of the link's path and description parts being
> appropriate for a link.

It would make sense then to include `org-quote-vert' call into
`org-columns--clean-item' then.

> Kindly find updated patches below. I hope to have succeeded in
> addressing all your comments; that was my intention at least.

Thanks!

>  (defun org-columns--capture-view (maxlevel match skip-empty exclude-tags format local)
>    "Get the column view of the current buffer.
>...
> +When LOCAL is non-nil, only capture headings in current subtree.  When
> +LINK is non-nil, item headlines will be linked to their origins.

Looks like you removed the LINK parameter, but forgot to remove its
description from the docstring.
  
> +			       (let ((search (org-link-heading-search-string raw)))
> +				 (org-link-make-string
> +				  (if (not (buffer-file-name)) search
> +				    (format "file:%s::%s" (buffer-file-name) search))
> +				  cleaned))

This will unconditionally generate file: links, even when the dynamic
block only refers to headings in the same buffer. The clock tables do
use internal links when appropriate (see `org-clock-get-table-data').

I am attaching a patch containing test case making sure that internal
links are generated when appropriate. The test case is failing with
the latest version of your patch.

If you can, please add some more tests like mine checking
`org-columns--clean-item'.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-colview-Add-test-for-the-new-link-parameter.patch --]
[-- Type: text/x-patch, Size: 1352 bytes --]

From 3961072d80883aef5da21d8d6ba10213778ff32f Mon Sep 17 00:00:00 2001
Message-ID: <3961072d80883aef5da21d8d6ba10213778ff32f.1713871396.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Tue, 23 Apr 2024 14:22:44 +0300
Subject: [PATCH] org-colview: Add test for the new :link parameter

* testing/lisp/test-org-colview.el (test-org-colview/dblock): New
test case.
---
 testing/lisp/test-org-colview.el | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/testing/lisp/test-org-colview.el b/testing/lisp/test-org-colview.el
index 7f0aa763e..872a61753 100644
--- a/testing/lisp/test-org-colview.el
+++ b/testing/lisp/test-org-colview.el
@@ -1422,6 +1422,18 @@ (ert-deftest test-org-colview/dblock ()
         "* H\n:PROPERTIES:\n:A: 1\n:END:\n<point>#+BEGIN: columnview\n#+END:"
       (let ((org-columns-default-format "%ITEM %A")) (org-update-dblock))
       (buffer-substring-no-properties (point) (point-max)))))
+  ;; Test `:link' parameter.
+  (should
+   (equal
+    "#+BEGIN: columnview
+| ITEM |
+|------|
+| [[*H][H]]    |
+#+END:"
+    (org-test-with-temp-text
+        "* H\n<point>#+BEGIN: columnview\n#+END:"
+      (let ((org-columns-default-format "%ITEM")) (org-update-dblock))
+      (buffer-substring-no-properties (point) (point-max)))))
   ;; Test column widths.
   (should
    (equal
-- 
2.44.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

Ihor Radchenko <yantar92@posteo.net> writes:

> [...]
> It would make sense then to include `org-quote-vert' call into
> `org-columns--clean-item' then.

Good point. I'll move the call to `org-quote-vert' there.

> [...]
>>  (defun org-columns--capture-view (maxlevel match skip-empty exclude-tags format local)
>>    "Get the column view of the current buffer.
>>...
>> +When LOCAL is non-nil, only capture headings in current subtree.  When
>> +LINK is non-nil, item headlines will be linked to their origins.
>
> Looks like you removed the LINK parameter, but forgot to remove its
> description from the docstring.

Ah, ashes to my head for overlooking this. Well spotted!

>> +			       (let ((search (org-link-heading-search-string raw)))
>> +				 (org-link-make-string
>> +				  (if (not (buffer-file-name)) search
>> +				    (format "file:%s::%s" (buffer-file-name) search))
>> +				  cleaned))
>
> This will unconditionally generate file: links, even when the dynamic
> block only refers to headings in the same buffer. The clock tables do
> use internal links when appropriate (see `org-clock-get-table-data').

Um, actually it does exactly the same as `org-clock-get-table-data'
(from where I borrowed the code snippet): it generates file: links in
buffers visiting files, and local links in buffers not visiting any
file.

Perhaps you were looking at a test case in which
`org-clock-get-table-data' gets called in an Org buffer that is not
visiting any file?

> [...]
> I am attaching a patch containing test case making sure that internal
> links are generated when appropriate. The test case is failing with
> the latest version of your patch.
>
> If you can, please add some more tests like mine checking
> `org-columns--clean-item'.
> [...]

Thanks for the springboard hint to get started with adding tests, which
I'm happy to do, of course.

Is there any way for me to run a specific subset of the tests only, for
instance "make test colview", or similar?


Many thanks and cheers,

  --alexander

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

>>> +			       (let ((search (org-link-heading-search-string raw)))
>>> +				 (org-link-make-string
>>> +				  (if (not (buffer-file-name)) search
>>> +				    (format "file:%s::%s" (buffer-file-name) search))
>>> +				  cleaned))
>>
>> This will unconditionally generate file: links, even when the dynamic
>> block only refers to headings in the same buffer. The clock tables do
>> use internal links when appropriate (see `org-clock-get-table-data').
>
> Um, actually it does exactly the same as `org-clock-get-table-data'
> (from where I borrowed the code snippet): it generates file: links in
> buffers visiting files, and local links in buffers not visiting any
> file.
>
> Perhaps you were looking at a test case in which
> `org-clock-get-table-data' gets called in an Org buffer that is not
> visiting any file?

Nope. I was writing a test case (the one I shared) and noticed that
file: links are generated. Then, I misread what org-clock does.
You don't need to do anything other than adjusting the test example I
shared to use file: links.

>> If you can, please add some more tests like mine checking
>> `org-columns--clean-item'.
>> [...]
>
> Thanks for the springboard hint to get started with adding tests, which
> I'm happy to do, of course.
>
> Is there any way for me to run a specific subset of the tests only, for
> instance "make test colview", or similar?

make test BTEST_RE="test-org-colview/dblock"
# the same, but do not re-compile Org mode
make test-dirty BTEST_RE="test-org-colview/dblock"

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

FAILED test-ob-shell/bash-uses-assoc-arrays

[Max Nikulin]

On 15/04/2024 23:46, Alexander Adolf wrote:
>     FAILED  test-ob-shell/bash-uses-assoc-arrays  ((should (equal "two"
> (org-trim (org-babel-execute-src-block)))) :form (equal "two" "three")
> :value nil :explanation (arrays-of-different-length 3 5 "two" "three"
> first-mismatch-at 1))
>     FAILED  test-ob-shell/bash-uses-assoc-arrays-with-lists  ((should
> (equal "20 cm" (org-trim (org-babel-execute-src-block)))) :form (equal
> "20 cm" "50 dl") :value nil :explanation (array-elt 0 (different-atoms
> (50 "#x32" "?2") (53 "#x35" "?5"))))

My guess is that GPLv2 BASH on macOS does not support associative 
arrays. Perhaps these tests should be skipped if BASH_VERSION is not 
fresh enough (not supplied by Apple).

Re: FAILED test-ob-shell/bash-uses-assoc-arrays

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 15/04/2024 23:46, Alexander Adolf wrote:
>>     FAILED  test-ob-shell/bash-uses-assoc-arrays  ((should (equal "two"
>> (org-trim (org-babel-execute-src-block)))) :form (equal "two" "three")
>> :value nil :explanation (arrays-of-different-length 3 5 "two" "three"
>> first-mismatch-at 1))
>>     FAILED  test-ob-shell/bash-uses-assoc-arrays-with-lists  ((should
>> (equal "20 cm" (org-trim (org-babel-execute-src-block)))) :form (equal
>> "20 cm" "50 dl") :value nil :explanation (array-elt 0 (different-atoms
>> (50 "#x32" "?2") (53 "#x35" "?5"))))
>
> My guess is that GPLv2 BASH on macOS does not support associative 
> arrays. Perhaps these tests should be skipped if BASH_VERSION is not 
> fresh enough (not supplied by Apple).

I guess so. Which bash versions should we cut off?


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: FAILED test-ob-shell/bash-uses-assoc-arrays

[Max Nikulin]

On 24/04/2024 19:54, Ihor Radchenko wrote:
> Max Nikulin <manikulin@gmail.com> writes:
> 
>> On 15/04/2024 23:46, Alexander Adolf wrote:
>>>      FAILED  test-ob-shell/bash-uses-assoc-arrays  ((should (equal "two"
>>> (org-trim (org-babel-execute-src-block)))) :form (equal "two" "three")
>>> :value nil :explanation (arrays-of-different-length 3 5 "two" "three"
>>> first-mismatch-at 1))
>>>      FAILED  test-ob-shell/bash-uses-assoc-arrays-with-lists  ((should
>>> (equal "20 cm" (org-trim (org-babel-execute-src-block)))) :form (equal
>>> "20 cm" "50 dl") :value nil :explanation (array-elt 0 (different-atoms
>>> (50 "#x32" "?2") (53 "#x35" "?5"))))
>>
>> My guess is that GPLv2 BASH on macOS does not support associative
>> arrays. Perhaps these tests should be skipped if BASH_VERSION is not
>> fresh enough (not supplied by Apple).
> 
> I guess so. Which bash versions should we cut off?

/usr/share/doc/bash/NEWS.gz
> This is a terse description of the new features added to bash-4.0 since
> the release of bash-3.2.  As always, the manual page (doc/bash.1) is
> the place to look for complete descriptions.
[...]
> ii. The shell provides associative array variables, with the appropriate
>     support to create, delete, assign values to, and expand them.

it is consistent with license change discussions
https://news.ycombinator.com/item?id=20102597

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

[-- Attachment #1: Type: text/plain, Size: 1093 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> [...]
>>> If you can, please add some more tests like mine checking
>>> `org-columns--clean-item'.
> [...]

I have added one test for each new feature (headline linkification, and
formatting function). Also, there's a third patch, which moves the call
to `org-quote-vert' to the cleanup function `org-columns--clean-item' as
you suggested.

Updated patches below.


One more thing caught my eye in `org-dblock-write:columnview' though.
When running the currently (i.e. before my patches) last test in
test-org-colview.el, the one with `org-columns-default-format' set to
"%ITEM %DEADLINE(d) %SCHEDULED(s) %TIMESTAMP(t)", `org-with-wide-buffer'
appears to modify the buffer and advances point by one (before the call
point is at 97, after the call at 98). In all other tests,
`org-with-wide-buffer' does not move point. In
`org-dblock-write:columnview' I have thus moved the sampling of point
for passing to the formatting function to after the call to
`org-with-wide-buffer'.


Many thanks and looking forward to your thoughts,

  --alexander



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-lisp-org-colview.el-Add-formatter-parameter-to-colvi.patch --]
[-- Type: text/x-patch, Size: 8958 bytes --]

From 2bb19d9000f29d56295b70754f89f927e7a5740c Mon Sep 17 00:00:00 2001
From: Alexander Adolf <alexander.adolf@condition-alpha.com>
Date: Sun, 14 Apr 2024 18:14:05 +0200
Subject: [PATCH 1/3] lisp/org-colview.el: Add formatter parameter to colview
 dynamic block

* lisp/org-colview.el (org-dblock-write:column view): Factor out the
existing formatting code to new function
`org-columns-dblock-write-default', and honour new dblock parameter
:formatter for specifying a different formatting function.
(org-columns-dblock-write-default): New function with current
formatting code.
(org-columns--capture-view): Amend docstring to better explain the
format of the data being passed to the formatting function.
(org-clock-clocktable-formatter): New option to define a global
default formatting function, defaulting to the current behaviour.
* testing/lisp/test-org-colview.el (test-org-colview/dblock): New test
for formatting function.
(test-org-colview/dblock-formatter): New function used in formatting
test.
* doc/org-manual.org (Capturing column view): Describe new :formatter
parameter.
* etc/ORG-NEWS (New option ~org-columns-dblock-formatter~): Announce
new option.
(=colview= dynamic block supports custom formatting function):
Describe new custom formatting function feature.
---
 doc/org-manual.org               |  7 ++++++
 etc/ORG-NEWS                     | 29 ++++++++++++++++++++++++
 lisp/org-colview.el              | 38 +++++++++++++++++++++++++++-----
 testing/lisp/test-org-colview.el | 20 +++++++++++++++++
 4 files changed, 89 insertions(+), 5 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index d24230568..000783772 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -6014,6 +6014,13 @@ This dynamic block has the following parameters:
   Specify a column attribute (see [[*Column attributes]]) for the dynamic
   block.
 
+- =:formatter= ::
+
+  #+cindex: @samp{formatter}, dynamic block parameter
+  #+vindex: org-columns-dblock-formatter
+  A function to format column view data and insert it into the buffer.
+  See the option ~org-columns-dblock-formatter~.
+
 The following commands insert or update the dynamic block:
 
 - ~org-columns-insert-dblock~ ::
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index dbf849422..84ea7cde9 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -999,6 +999,16 @@ even though it does not have its own ID.  By giving files top-level id
 properties, links to headlines in the file can also be made more
 robust by using the file id instead of the file path.
 
+*** New option ~org-columns-dblock-formatter~
+
+=colview= dynamic blocks now understand a new ~:formatter~ parameter
+to use a specific function for formatting and inserting the contents
+of the dynamic block. This new option can be used to set the global
+default formatting function that will be used for =colview= dynamic
+blocks that do not specify any ~:formatter~ parameter. Its default
+value (the new function ~org-columns-dblock-write-default~) yields the
+previous (fixed) formatting behaviour.
+
 ** New features
 *** ~org-paste-subtree~ now handles =C-u= and =C-u C-u= prefix arguments specially
 
@@ -1035,6 +1045,25 @@ Example:
 : | PROYECTO EMACS |
 : #+END:
 
+*** =colview= dynamic block supports custom formatting function
+
+The =colview= dynamic block understands a new ~:formatter~ parameter,
+which specifies a user-supplied function to format and insert the data
+in the dynamic block.
+
+A global default formatting function for =colview= dynamic blocks can
+be set via the new option ~org-columns-dblock-formatter~ which
+defaults to the new function ~org-columns-dblock-write-default~, that
+implements the previous (fixed) formatting behaviour. Hence, the
+default behaviour is identical to previous versions.
+
+The global default function can be overridden for any given =colview=
+dynamic block individually by specifying a custom formatter function
+using the new ~:formatter~ parameter on the block's =BEGIN= line.
+
+This new feature replicates the ~:formatter~ option already available
+for =clocktable= dynamic blocks.
+
 *** =ob-tangle.el=: New flag to remove tangle targets before writing
 
 When ~org-babel-tangle-remove-file-before-write~ is set to ~t~ the
diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index 4718d1db7..6241fa3ba 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -123,6 +123,12 @@ in `org-columns-summary-types-default', which see."
 			      (function :tag "Summarize")
 			      (function :tag "Collect")))))
 
+(defcustom org-columns-dblock-formatter #'org-columns-dblock-write-default
+  "Function to format data in column view dynamic blocks.
+For more information, see `org-columns-dblock-write-default'."
+  :group 'org-properties
+  :package-version '(Org . "9.7")
+  :type 'function)
 
 \f
 ;;; Column View
@@ -1442,9 +1448,13 @@ that will be excluded from the resulting view.  FORMAT is a
 format string for columns, or nil.  When LOCAL is non-nil, only
 capture headings in current subtree.
 
-This function returns a list containing the title row and all
-other rows.  Each row is a list of fields, as strings, or
-`hline'."
+This function returns a list containing the title row and all other
+rows.  Each row is either a list, or the symbol `hline'.  The first list
+is the heading row as a list of strings with the column titles according
+to FORMAT.  All subsequent lists each represent a body row as a list
+whose first element is an integer indicating the outline level of the
+entry, and whose remaining elements are strings with the contents for
+the columns according to FORMAT."
   (org-columns (not local) format)
   (goto-char org-columns-top-level-marker)
   (let ((columns (length org-columns-current-fmt-compiled))
@@ -1545,7 +1555,13 @@ PARAMS is a property list of parameters:
 `:vlines'
 
     When non-nil, make each column a column group to enforce
-    vertical lines."
+    vertical lines.
+
+`:formatter'
+
+    A function to format the data and insert it into the
+    buffer.  Overrides the default formatting function set in
+    `org-columns-dblock-formatter'."
   (let ((table
 	 (let ((id (plist-get params :id))
 	       view-file view-pos)
@@ -1573,7 +1589,17 @@ PARAMS is a property list of parameters:
 					 (plist-get params :exclude-tags)
 					 (plist-get params :format)
 					 view-pos)))))
-        (width-specs
+        (formatter (or (plist-get params :formatter)
+                       org-columns-dblock-formatter
+                       #'org-columns-dblock-write-default)))
+    (funcall formatter (point) table params)))
+
+(defun org-columns-dblock-write-default (ipos table params)
+  "Write out a columnview table at position IPOS in the current buffer.
+TABLE is a table with data as produced by `org-columns--capture-view'.
+PARAMS is the parameter property list obtained from the dynamic block
+definition."
+  (let ((width-specs
          (mapcar (lambda (spec) (nth 2 spec))
                  org-columns-current-fmt-compiled)))
     (when table
@@ -1616,6 +1642,8 @@ PARAMS is a property list of parameters:
         ;; to the resulting table, adding alignment field as the first
         ;; row.
         (push (mapcar (lambda (width) (when width (format "<%d>" width))) width-specs) table))
+      ;; now insert the table into the buffer
+      (goto-char ipos)
       (let ((content-lines (org-split-string (plist-get params :content) "\n"))
 	    recalc)
 	;; Insert affiliated keywords before the table.
diff --git a/testing/lisp/test-org-colview.el b/testing/lisp/test-org-colview.el
index 7f0aa763e..bcf4e897f 100644
--- a/testing/lisp/test-org-colview.el
+++ b/testing/lisp/test-org-colview.el
@@ -1398,6 +1398,13 @@
 \f
 ;;; Dynamic block
 
+(defun test-org-colview/dblock-formatter (ipos table params)
+  "User-defined columnview dblock formatting function."
+  (goto-char ipos)
+  (insert-before-markers "Hello columnview!" "\n")
+  (insert-before-markers (format "table has %d rows" (length table)) "\n")
+  (insert-before-markers (format "there are %d parameters" (/ (length params) 2))))
+
 (ert-deftest test-org-colview/dblock ()
   "Test the column view table."
   (should
@@ -1703,6 +1710,19 @@ SCHEDULED: <2020-05-11 Mon> DEADLINE: <2020-05-14 Thu>
      (let ((org-columns-default-format
 	    "%ITEM %DEADLINE(d) %SCHEDULED(s) %TIMESTAMP(t)"))
        (org-update-dblock))
+     (buffer-substring-no-properties (point) (point-max)))))
+  ;; custom formatting function
+  (should
+   (equal
+    "#+BEGIN: columnview :formatter test-org-colview/dblock-formatter
+Hello columnview!
+table has 3 rows
+there are 4 parameters
+#+END:"
+    (org-test-with-temp-text
+     "* H\n<point>#+BEGIN: columnview :formatter test-org-colview/dblock-formatter\n#+END:"
+     (let ((org-columns-default-format "%ITEM"))
+       (org-update-dblock))
      (buffer-substring-no-properties (point) (point-max))))))
 
 (provide 'test-org-colview)
-- 
2.39.3 (Apple Git-146)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-lisp-org-colview.el-Add-link-parameter-to-colview-dy.patch --]
[-- Type: text/x-patch, Size: 4827 bytes --]

From eb7c61bea2c644fba79aff2d71e93a1a3004f0b9 Mon Sep 17 00:00:00 2001
From: Alexander Adolf <alexander.adolf@condition-alpha.com>
Date: Mon, 15 Apr 2024 18:01:40 +0200
Subject: [PATCH 2/3] lisp/org-colview.el: Add link parameter to colview
 dynamic block

* lisp/org-colview.el (org-columns--capture-view): Add new link
parameter, which when non-nil causes ITEM headlines to be linked to
their origins.
(org-dblock-write:columnview): Pass new link parameter to
`org-columns--capture-view', and explain its use in the docstring.
* testing/lisp/test-org-colview.el (test-org-colview/dblock): add
new test for link feature
* doc/org-manual.org (Capturing column view): Describe new :link
parameter.
* etc/ORG-NEWS (=colview= dynamic block can link to headlines):
Describe new link feature.
---
 doc/org-manual.org               |  5 +++++
 etc/ORG-NEWS                     |  6 ++++++
 lisp/org-colview.el              | 20 ++++++++++++++++----
 testing/lisp/test-org-colview.el | 12 ++++++++++++
 4 files changed, 39 insertions(+), 4 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 000783772..a77492971 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -6009,6 +6009,11 @@ This dynamic block has the following parameters:
 
   When non-~nil~, indent each =ITEM= field according to its level.
 
+- =:link= ::
+
+  When non-~nil~, link the =ITEM= headlines in the table to their
+  origins.
+
 - =:format= ::
 
   Specify a column attribute (see [[*Column attributes]]) for the dynamic
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 84ea7cde9..b4aea10b8 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -1064,6 +1064,12 @@ using the new ~:formatter~ parameter on the block's =BEGIN= line.
 This new feature replicates the ~:formatter~ option already available
 for =clocktable= dynamic blocks.
 
+*** =colview= dynamic block can link to headlines
+
+The =colview= dynamic block understands a new ~:link~ parameter, which
+when non-~nil~ causes =ITEM= headlines in the table to be linked to
+their origins.
+
 *** =ob-tangle.el=: New flag to remove tangle targets before writing
 
 When ~org-babel-tangle-remove-file-before-write~ is set to ~t~ the
diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index 6241fa3ba..64ed8c16a 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -1557,6 +1557,10 @@ PARAMS is a property list of parameters:
     When non-nil, make each column a column group to enforce
     vertical lines.
 
+`:link'
+
+    Link the item headlines in the table to their origins.
+
 `:formatter'
 
     A function to format the data and insert it into the
@@ -1599,9 +1603,10 @@ PARAMS is a property list of parameters:
 TABLE is a table with data as produced by `org-columns--capture-view'.
 PARAMS is the parameter property list obtained from the dynamic block
 definition."
-  (let ((width-specs
-         (mapcar (lambda (spec) (nth 2 spec))
-                 org-columns-current-fmt-compiled)))
+  (let ((link (plist-get params :link))
+	(width-specs
+	 (mapcar (lambda (spec) (nth 2 spec))
+		 org-columns-current-fmt-compiled)))
     (when table
       ;; Prune level information from the table.  Also normalize
       ;; headings: remove stars, add indentation entities, if
@@ -1625,7 +1630,14 @@ definition."
 			   (and (numberp hlines) (<= level hlines))))
 	      (push 'hline new-table))
 	    (when item-index
-	      (let ((item (org-columns--clean-item (nth item-index (cdr row)))))
+	      (let* ((raw (nth item-index (cdr row)))
+		     (cleaned (org-columns--clean-item raw))
+		     (item (if (not link) cleaned
+			     (let ((search (org-link-heading-search-string raw)))
+			       (org-link-make-string
+				(if (not (buffer-file-name)) search
+				  (format "file:%s::%s" (buffer-file-name) search))
+				cleaned)))))
 		(setf (nth item-index (cdr row))
 		      (if (and indent (> level 1))
 			  (concat "\\_" (make-string (* 2 (1- level)) ?\s) item)
diff --git a/testing/lisp/test-org-colview.el b/testing/lisp/test-org-colview.el
index bcf4e897f..6b603c31b 100644
--- a/testing/lisp/test-org-colview.el
+++ b/testing/lisp/test-org-colview.el
@@ -1723,6 +1723,18 @@ there are 4 parameters
      "* H\n<point>#+BEGIN: columnview :formatter test-org-colview/dblock-formatter\n#+END:"
      (let ((org-columns-default-format "%ITEM"))
        (org-update-dblock))
+     (buffer-substring-no-properties (point) (point-max)))))
+  ;; test headline linkification
+  (should
+   (equal
+    "#+BEGIN: columnview :link t
+| ITEM |
+|------|
+| [[*H][H]]    |
+#+END:"
+    (org-test-with-temp-text
+     "* H\n<point>#+BEGIN: columnview :link t\n#+END:"
+     (let ((org-columns-default-format "%ITEM")) (org-update-dblock))
      (buffer-substring-no-properties (point) (point-max))))))
 
 (provide 'test-org-colview)
-- 
2.39.3 (Apple Git-146)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0003-lisp-org-colview.el-Move-cleanup-code-to-cleanup-fun.patch --]
[-- Type: text/x-patch, Size: 1895 bytes --]

From 03334c493fec07d0571dc9298ce11d96fd8ba185 Mon Sep 17 00:00:00 2001
From: Alexander Adolf <alexander.adolf@condition-alpha.com>
Date: Tue, 23 Apr 2024 17:43:08 +0200
Subject: [PATCH 3/3] lisp/org-colview.el: Move cleanup code to cleanup
 function

* lisp/org-colview.el (org-columns--clean-item): Additionally process
the result with `org-quote-vert'.
(org-columns--capture-view): Remove call to `org-quote-vert'.

`org-columns--clean-item' is used when formatting and inserting column
view dynamic blocks. By moving the call to `org-quote-vert' to this
function, it can be used by all formatting functions, including
user-supplied ones, to format content to be safe for inclusion in a
table.
---
 lisp/org-colview.el | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index 64ed8c16a..e934ae67a 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -1467,11 +1467,10 @@ the columns according to FORMAT."
 	   (dotimes (i columns)
 	     (let* ((col (+ (line-beginning-position) i))
 		    (p (get-char-property col 'org-columns-key)))
-	       (push (org-quote-vert
-		      (get-char-property col
-					 (if (string= p "ITEM")
-					     'org-columns-value
-					   'org-columns-value-modified)))
+	       (push (get-char-property col
+					(if (string= p "ITEM")
+					    'org-columns-value
+					  'org-columns-value-modified))
 		     row)))
 	   (unless (or
 		    (and skip-empty
@@ -1503,7 +1502,9 @@ an inline src-block."
 	'(footnote-reference inline-babel-call inline-src-block target
 			     radio-target statistics-cookie)
       #'org-element-extract)
-    (org-no-properties (org-element-interpret-data data))))
+    (org-quote-vert
+     (org-no-properties
+      (org-element-interpret-data data)))))
 
 ;;;###autoload
 (defun org-dblock-write:columnview (params)
-- 
2.39.3 (Apple Git-146)

Re: FAILED test-ob-shell/bash-uses-assoc-arrays

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>>> My guess is that GPLv2 BASH on macOS does not support associative
>>> arrays. Perhaps these tests should be skipped if BASH_VERSION is not
>>> fresh enough (not supplied by Apple).
>> 
>> I guess so. Which bash versions should we cut off?
>
> /usr/share/doc/bash/NEWS.gz
>> This is a terse description of the new features added to bash-4.0 since
>> the release of bash-3.2.  As always, the manual page (doc/bash.1) is
>> the place to look for complete descriptions.

Done, on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e4ab416fc

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> [...]
>>>> If you can, please add some more tests like mine checking
>>>> `org-columns--clean-item'.
>> [...]
>
> I have added one test for each new feature (headline linkification, and
> formatting function). Also, there's a third patch, which moves the call
> to `org-quote-vert' to the cleanup function `org-columns--clean-item' as
> you suggested.
>
> Updated patches below.

Thanks!
The patch is ready to be merged.
Bastien, may you please confirm the FSF records?
Alexander should have the copyright signed.

> One more thing caught my eye in `org-dblock-write:columnview' though.
> When running the currently (i.e. before my patches) last test in
> test-org-colview.el, the one with `org-columns-default-format' set to
> "%ITEM %DEADLINE(d) %SCHEDULED(s) %TIMESTAMP(t)", `org-with-wide-buffer'
> appears to modify the buffer and advances point by one (before the call
> point is at 97, after the call at 98). In all other tests,
> `org-with-wide-buffer' does not move point. In
> `org-dblock-write:columnview' I have thus moved the sampling of point
> for passing to the formatting function to after the call to
> `org-with-wide-buffer'.

This is expected - `org-dblock-write:columnview' uses
`org-columns--capture-view', which, in turn, calls the `org-columns' -
function used to create an interactive column view.

For interactive column view, we must have at least a single space for
each column. Otherwise, we cannot display the columns overlay. So,
`org-columns' always add spaces to headlines, when there are not enough
of them to fit all the requested columns.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Bastien Guerry]

Ihor Radchenko <yantar92@posteo.net> writes:

> Thanks!
> The patch is ready to be merged.
> Bastien, may you please confirm the FSF records?
> Alexander should have the copyright signed.

Yes, he has, thanks!

-- 
 Bastien Guerry

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

Bastien Guerry <bzg@gnu.org> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> Thanks!
>> The patch is ready to be merged.
>> Bastien, may you please confirm the FSF records?
>> Alexander should have the copyright signed.
>
> Yes, he has, thanks!

Applied, onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=5a98b4c56
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=4e6fa96e2
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=fbf613ece

Thanks for your contribution!

Your name have been added to our contributor list:
https://git.sr.ht/~bzg/worg/commit/73d38812

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

Ihor Radchenko <yantar92@posteo.net> writes:

> Bastien Guerry <bzg@gnu.org> writes:
>
>> Ihor Radchenko <yantar92@posteo.net> writes:
>>
>>> Thanks!
>>> The patch is ready to be merged.
>>> Bastien, may you please confirm the FSF records?
>>> Alexander should have the copyright signed.
>>
>> Yes, he has, thanks!
>
> Applied, onto main.
> [...]
> Thanks for your contribution!

Many thanks for your kind support in getting it in shape; much
appreciated. It was my pleasure working with you!

One last question though: I'm developing an add-on package which will
use the new features. I guess in "Package-Requires:" it should say
(org "9.7") ?

> [...]
> Your name have been added to our contributor list:
> https://git.sr.ht/~bzg/worg/commit/73d38812
> [...]

More than I deserve...


Many thanks and cheers,

  --alexander

Re: FAILED test-ob-shell/bash-uses-assoc-arrays

[Max Nikulin]

On 26/04/2024 18:08, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>>>> My guess is that GPLv2 BASH on macOS does not support associative
>>>> arrays. Perhaps these tests should be skipped if BASH_VERSION is not
>>>> fresh enough (not supplied by Apple).
> 
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e4ab416fc

Feature detection should be more reliable.

;; Old GPLv2 BASH in macOSX does not support associative arrays.
(if-let ((bash (executable-find "bash")))
     (eq 0 (process-file
	   bash nil nil nil
	   "-c" "declare -A assoc_array")))

Even version check may be performed by shell

(if-let ((bash (executable-find "bash")))
     (eq 0 (process-file
	   bash nil nil nil
	   "-c" "[ ${BASH_VERSINFO[0]} -ge 4 ]")))

Re: FAILED test-ob-shell/bash-uses-assoc-arrays

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>>>>> My guess is that GPLv2 BASH on macOS does not support associative
>>>>> arrays. Perhaps these tests should be skipped if BASH_VERSION is not
>>>>> fresh enough (not supplied by Apple).
>> 
>> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e4ab416fc
>
> Feature detection should be more reliable.
>
> ;; Old GPLv2 BASH in macOSX does not support associative arrays.
> (if-let ((bash (executable-find "bash")))
>      (eq 0 (process-file
> 	   bash nil nil nil
> 	   "-c" "declare -A assoc_array")))
>
> Even version check may be performed by shell
>
> (if-let ((bash (executable-find "bash")))
>      (eq 0 (process-file
> 	   bash nil nil nil
> 	   "-c" "[ ${BASH_VERSINFO[0]} -ge 4 ]")))

Feel free to submit a patch.
It is a marginal improvement I will be happy to merge, if a patch is
provided.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> One last question though: I'm developing an add-on package which will
> use the new features. I guess in "Package-Requires:" it should say
> (org "9.7") ?

Yes.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

[PATCH] test-ob-shell.el: Skip based on feature detection

[Max Nikulin]

[-- Attachment #1: Type: text/plain, Size: 273 bytes --]

On 28/04/2024 20:11, Ihor Radchenko wrote:
>>> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e4ab416fc

> Max Nikulin <manikulin@gmail.com> writes:
>> Feature detection should be more reliable.
[...]
> 
> Feel free to submit a patch.

See the attachment.

[-- Attachment #2: 0001-test-ob-shell.el-Skip-based-on-feature-detection.patch --]
[-- Type: text/x-patch, Size: 2781 bytes --]

From 46db53cebef5644be4abd8dcb18ca777ec0c3be3 Mon Sep 17 00:00:00 2001
From: Max Nikulin <manikulin@gmail.com>
Date: Thu, 2 May 2024 17:09:10 +0700
Subject: [PATCH] test-ob-shell.el: Skip based on feature detection

* testing/lisp/test-ob-shell.el (test-ob-shell/bash-uses-assoc-arrays)
(test-ob-shell/bash-uses-assoc-arrays-with-lists): Use feature detection
shell command instead of version comparison to skip tests for
associative arrays.

It affects macOS since Apple ships BASH 3.2 licensed as GPLv2.
---
 testing/lisp/test-ob-shell.el | 20 +++++++-------------
 1 file changed, 7 insertions(+), 13 deletions(-)

diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el
index f58e85815..b760031b4 100644
--- a/testing/lisp/test-ob-shell.el
+++ b/testing/lisp/test-ob-shell.el
@@ -230,6 +230,10 @@ (ert-deftest test-ob-shell/bash-uses-assoc-arrays ()
 Bash will see a table that contains the first column as the
 'index' of the associative array, and the second column as the
 value. "
+  (skip-unless
+   ;; Old GPLv2 BASH in macOSX does not support associative arrays.
+   (if-let ((bash (executable-find "bash")))
+       (eq 0 (process-file bash nil nil nil "-c" "declare -A assoc_array"))))
   (org-test-with-temp-text
       "#+NAME: sample_mapping_table
 | first  | one   |
@@ -240,13 +244,6 @@ (ert-deftest test-ob-shell/bash-uses-assoc-arrays ()
 echo ${table[second]}
 <point>
 #+end_src "
-    (skip-unless (executable-find "bash"))
-    (skip-unless
-     (let* ((version-string (shell-command-to-string "bash -c 'echo $BASH_VERSION'"))
-            (major-version (and (string-match "^\\([0-9]+\\)\\." version-string)
-			        (string-to-number (match-string 1 version-string)))))
-       ;; Bash 4.0 introduced associative arrays support.
-       (>= major-version 4)))
     (should
      (equal "two"
             (org-trim (org-babel-execute-src-block))))))
@@ -256,13 +253,10 @@ (ert-deftest test-ob-shell/bash-uses-assoc-arrays-with-lists ()
 
 Bash will see an associative array that contains each row as a single
 string. Bash cannot handle lists in associative arrays."
-  (skip-unless (executable-find "bash"))
   (skip-unless
-   (let* ((version-string (shell-command-to-string "bash -c 'echo $BASH_VERSION'"))
-          (major-version (and (string-match "^\\([0-9]+\\)\\." version-string)
-			      (string-to-number (match-string 1 version-string)))))
-     ;; Bash 4.0 introduced associative arrays support.
-     (>= major-version 4)))
+   ;; Old GPLv2 BASH in macOSX does not support associative arrays.
+   (if-let ((bash (executable-find "bash")))
+       (eq 0 (process-file bash nil nil nil "-c" "declare -A assoc_array"))))
   (org-test-with-temp-text
       "#+NAME: sample_big_table
 | bread     |  2 | kg |
-- 
2.39.2

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 686 bytes --]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

> 4 unexpected results:
>    FAILED  ob-calc/matrix-inversion  ((should (equal "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]" (org-babel-execute-src-block))) :form (equal "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]" "[[-1, 625e-3, -125e-3], [250e-3, -500e-3, 250e-3], [500e-3, 125e-3, -125e-3]]") :value nil :explanation (arrays-of-different-length 63 77 "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]" "[[-1, 625e-3, -125e-3], [250e-3, -500e-3, 250e-3], [500e-3, 125e-3, -125e-3]]" first-mismatch-at 6))

Will the failure disappear if you try the attached patch?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-calc-matrix-inversion-Force-calc-float-format-in-.patch --]
[-- Type: text/x-patch, Size: 1720 bytes --]

From fb317be318aaf32104e9334d34813681c26ac240 Mon Sep 17 00:00:00 2001
Message-ID: <fb317be318aaf32104e9334d34813681c26ac240.1714651740.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Thu, 2 May 2024 15:07:12 +0300
Subject: [PATCH] ob-calc/matrix-inversion: Force `calc-float-format' in the
 test

* testing/lisp/test-ob-calc.el (ob-calc/matrix-inversion): When
testing, make sure that calc output follows the expected format.
This is needed on MacOS where Emacs defaults are not exactly the same.

Reported-by: Alexander Adolf <alexander.adolf@condition-alpha.com>
Link: https://list.orgmode.org/orgmode/91dfeb0fed1a1fe0564e5eb9b95a409d@condition-alpha.com/
---
 testing/lisp/test-ob-calc.el | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/testing/lisp/test-ob-calc.el b/testing/lisp/test-ob-calc.el
index 6d6ca104d..e5f2cee0b 100644
--- a/testing/lisp/test-ob-calc.el
+++ b/testing/lisp/test-ob-calc.el
@@ -59,7 +59,14 @@ (ert-deftest ob-calc/matrix-inversion ()
 	inv(a)
 #+END_SRC "
     (should (equal "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]"
-                   (org-babel-execute-src-block)))))
+                   (let ((calc-float-format '(float 0)))
+                     ;; ;; Make sure that older Calc buffers are not present.
+                     (save-current-buffer
+                       (when (ignore-errors (calc-select-buffer))
+                         (kill-buffer)))
+                     ;; Now, let-bound `calc-float-format' will take
+                     ;; effect.
+                     (org-babel-execute-src-block))))))
 
 (ert-deftest ob-calc/matrix-algebra ()
   "Test of simple matrix algebra."
-- 
2.44.0


[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Alexander Adolf]

Ihor Radchenko <yantar92@posteo.net> writes:

> Alexander Adolf <alexander.adolf@condition-alpha.com> writes:
>
>> 4 unexpected results:
>>    FAILED  ob-calc/matrix-inversion  ((should (equal "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]" (org-babel-execute-src-block))) :form (equal "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]" "[[-1, 625e-3, -125e-3], [250e-3, -500e-3, 250e-3], [500e-3, 125e-3, -125e-3]]") :value nil :explanation (arrays-of-different-length 63 77 "[[-1, 0.625, -0.125], [0.25, -0.5, 0.25], [0.5, 0.125, -0.125]]" "[[-1, 625e-3, -125e-3], [250e-3, -500e-3, 250e-3], [500e-3, 125e-3, -125e-3]]" first-mismatch-at 6))
>
> Will the failure disappear if you try the attached patch?
> [...]

Yes, your patch fixes that test failure on macOS.


Hoping to have helped,

  --alexander

Re: [PATCH] test-ob-shell.el: Skip based on feature detection

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> Feel free to submit a patch.
>
> See the attachment.

Applied, onto main.
Thanks!
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=ac7c5c5d8

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: columnview dynamic block - different time summing behaviour for EFFORT and CLOCKSUM

[Ihor Radchenko]

Alexander Adolf <alexander.adolf@condition-alpha.com> writes:

>> Will the failure disappear if you try the attached patch?
>> [...]
>
> Yes, your patch fixes that test failure on macOS.

Thanks for checking!
Applied, onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=d8ac95883

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>