[BUG] Error in data input and output format for org-columns--summary-estimate

[BUG] Error in data input and output format for org-columns--summary-estimate

[andrea]

[-- Attachment #1: Type: text/plain, Size: 2136 bytes --]


   Howdy!

   This has been tested to happen with org 9.5.5, a few of the

   following, and on the HEAD version on git (9.6.7+); it's a source

   matter, afflicting org independently of the running platform (GNU/Linux,

   Windows, etc.)

   The matter is related to the fact that org-columns--summary-estimate, from org-colview.el, uses function string-to-number to take value on ranges; acting this way

   the unit is removed making impossible to distinguish between 1y and 1min . Similarly, on output, the two margins are produced a (format "%.0f" <value>), which --again-- is generally wrong as it does not tell anything about the utilized time unit.

   Both issues can be very simply addressed by adoption of org-duration-to-minutes as

   input adapted and org-duration-from-minutes as output adapter.
   A similar concern affects also the simpler case of a single value instead of a range (second pcase branch)

   Here is the patched code with the wrong code lines commented out.

   (defun org-columns--summary-estimate (estimates _)

     "Combine a list of estimates, using mean and variance.

   The mean and variance of the result will be the sum of the means

   and variances (respectively) of the individual estimates."

     (let ((mean 0)

           (var 0))

       (dolist (e estimates)

   ;;      (pcase (mapcar #'string-to-number (split-string e "-"))

         (pcase (mapcar #'org-duration-to-minutes (split-string e "-"))

           (`(,low ,high)

            (let ((m (/ (+ low high) 2.0)))

              (cl-incf mean m)

              (cl-incf var (- (/ (+ (* low low) (* high high)) 2.0) (* m m)))))

   ;;        (`(,value) (cl-incf mean value))))

           (`(,value) (cl-incf mean (org-duration-to-minutes value)))))

       (let ((sd (sqrt var)))

         (format "%s-%s"

   ;;              (format "%.0f" (- mean sd))

   ;;              (format "%.0f" (+ mean sd))))))

                 (org-duration-from-minutes (- mean sd))

                 (org-duration-from-minutes (+ mean sd))))))

   Cheers!
     Andrea Fedeli.

[-- Attachment #2: Type: text/html, Size: 3347 bytes --]

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

andrea@fedeli.eu writes:

>    The matter is related to the fact that org-columns--summary-estimate, from org-colview.el, uses function string-to-number to take value on ranges; acting this way
>
>    the unit is removed making impossible to distinguish between 1y and 1min . Similarly, on output, the two margins are produced a (format "%.0f" <value>), which --again-- is generally wrong as it does not tell anything about the utilized time unit.

May you please provide a concrete example demonstrating the problem?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[andrea]

[-- Attachment #1: Type: text/plain, Size: 1581 bytes --]


   #+PROPERTY: Effort_ALL 0 0:10 0:30 1:00 2:00 3:00 4:00 5:00 6:00 7:00

   #+COLUMNS: %30ITEM(Task) %30Effort(Estimated Effort){est+} %CLOCKSUM

   * efforts

   ** task 1

   :PROPERTIES:

   :Effort:   3d

   :END:

   ** task 2

   :PROPERTIES:

   :Effort:   3m-4m

   :END:

   Originally produces
   image.png
   That shows that days and months are wrongly counted the same
   With the suggested patch produces
   image.png
   Showing that day and months are correctly handled, and reported.
   Cheers,
     Andrea.

   Da emacs-orgmode-bounces+andrea=fedeli.eu@gnu.org
   A andrea@fedeli.eu
   Cc emacs-orgmode@gnu.org
   Data Sun, 09 Jul 2023 14:17:59 +0000
   Oggetto Re: [BUG] Error in data input and output format for org-columns--summary-estimate
   andrea@fedeli.eu writes:

   > The matter is related to the fact that org-columns--summary-estimate, from org-colview.el, uses function string-to-number to take value on ranges; acting this way
   >
   > the unit is removed making impossible to distinguish between 1y and 1min . Similarly, on output, the two margins are produced a (format "%.0f" <value>), which --again-- is generally wrong as it does not tell anything about the utilized time unit.

   May you please provide a concrete example demonstrating the problem?

   --
   Ihor Radchenko // yantar92,
   Org mode contributor,
   Learn more about Org mode at <https://orgmode.org/>.
   Support Org development at <https://liberapay.com/org-mode>,
   or support my work at <https://liberapay.com/yantar92>

[-- Attachment #2.1: Type: text/html, Size: 3410 bytes --]

[-- Attachment #2.2: 00000CSP.png --]
[-- Type: image/png, Size: 14303 bytes --]

[-- Attachment #2.3: 00100CSP.png --]
[-- Type: image/png, Size: 14498 bytes --]

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]

andrea@fedeli.eu writes:

>    #+PROPERTY: Effort_ALL 0 0:10 0:30 1:00 2:00 3:00 4:00 5:00 6:00 7:00
>
>    #+COLUMNS: %30ITEM(Task) %30Effort(Estimated Effort){est+} %CLOCKSUM
>...
>    Originally produces
>    image.png
>    That shows that days and months are wrongly counted the same

Thanks!
Confirmed.

>    > The matter is related to the fact that org-columns--summary-estimate, from org-colview.el, uses function string-to-number to take value on ranges; acting this way

>   Both issues can be very simply addressed by adoption of org-duration-to-minutes as
>
>   input adapted and org-duration-from-minutes as output adapter.
>   A similar concern affects also the simpler case of a single value instead of a range (second pcase branch)

This will not be backwards-compatible.
Some people may abuse this summary type to count something that is not
time, like cost.

We rather need something like the attached diff.
However, this will use `org-duration-format', which may not always be
desirable - the default value will force days even when all the estimates
are months:

3m
3m-4m
---
180d 0:00 - 210d 0:00


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: org-columns-est.diff --]
[-- Type: text/x-patch, Size: 1487 bytes --]

diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index 7aa5ef645..da9a11104 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -1345,18 +1345,32 @@ (defun org-columns--summary-estimate (estimates _)
 The mean and variance of the result will be the sum of the means
 and variances (respectively) of the individual estimates."
   (let ((mean 0)
-        (var 0))
+        (var 0)
+        (durationp
+         (catch :no-match
+           (dolist (e estimates)
+             (dolist (val (split-string e "-"))
+               (unless (org-duration-p val) (throw :no-match nil))))
+           'all-values-are-durations)))
     (dolist (e estimates)
-      (pcase (mapcar #'string-to-number (split-string e "-"))
+      (pcase (mapcar
+              (if durationp
+                  #'org-duration-to-minutes
+                #'string-to-number)
+              (split-string e "-"))
 	(`(,low ,high)
 	 (let ((m (/ (+ low high) 2.0)))
 	   (cl-incf mean m)
 	   (cl-incf var (- (/ (+ (* low low) (* high high)) 2.0) (* m m)))))
 	(`(,value) (cl-incf mean value))))
     (let ((sd (sqrt var)))
-      (format "%s-%s"
-	      (format "%.0f" (- mean sd))
-	      (format "%.0f" (+ mean sd))))))
+      (if durationp
+          (format "%s - %s"
+                  (org-duration-from-minutes (- mean sd))
+                  (org-duration-from-minutes (+ mean sd)))
+        (format "%s-%s"
+	        (format "%.0f" (- mean sd))
+	        (format "%.0f" (+ mean sd)))))))
 
 
 \f

[-- Attachment #3: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

[ Adding Org ML back to CC. Please use "reply all" to reply on the
mailing list ]

"andrea@fedeli.eu" <andrea@fedeli.eu> writes:

>     I do apologize, I noticed only now the patch content: the output
>     format of duration-from-minutes Is controlled by the
>     org-duration-format variable, so if the user wants results in
>     months (s)he can easily get It that way.

`org-duration-format' is controlling many other aspects of formatting.
Customizing, for example, agenda should probably not affect column views.
 
>     About possibile abuses, org documentation, to date, clearly tells
>     org estimate utilizes times.

May you please elaborate?
 
>     Similarly, I'd not spend too much code to check the format; i
>     understand the intent to preserve backward compatibility, bit if
>     that format adaptation mistake slipped forward to these days I
>     doubt that nice feature was used much so far...

Yes, but it is easy to have a fallback, so why not.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[andrea]

[-- Attachment #1: Type: text/plain, Size: 4122 bytes --]


   Howdy!

   Da "Ihor Radchenko" yantar92@posteo.net
   A andrea@fedeli.eu
   Cc emacs-orgmode@gnu.org
   Data Fri, 14 Jul 2023 09:02:04 +0000
   Oggetto Re: [BUG] Error in data input and output format for org-columns--summary-estimate
   [ Adding Org ML back to CC. Please use "reply all" to reply on the
   mailing list ]

   "andrea@fedeli.eu" <andrea@fedeli.eu> writes:

   > I do apologize, I noticed only now the patch content: the output
   > format of duration-from-minutes Is controlled by the
   > org-duration-format variable, so if the user wants results in
   > months (s)he can easily get It that way.

   `org-duration-format' is controlling many other aspects of formatting.
   Customizing, for example, agenda should probably not affect column views.
   AF: I think we need to univoquely clarify which unit is to be used in estimation. As shown below org documentation suggests time it to be used for that, from there it cames the idea to turn times into minutes for computation reason; maybe we may introduce a new format variable for estimation results reporting. I thought I might use org-duration-from-minutes for that, as it came very handy.
   > About possibile abuses, org documentation, to date, clearly tells
   > org estimate utilizes times.

   May you please elaborate?
   AF: Sure! Clause 8.5 of current (2023.Jul.14) org documentation, https://orgmode.org/manual/Effort-Estimates.html, refers to effort estimates giving examples that are all appearing to fall in time domain. In particular it refers to org-duration.el for format information. That, IMO, establish that efforts have to refer to org-duration units. Now, the *default* values of org-duration are *clearly* time measures. From there my assumption about the fact that I may use the org-duration-to-minutes and org-duration-from-minutes adapters.
   Clearly, you /may/ decide to completely redefine the org-duration basis, but that would mean to coherently propagate the information change everywhere somebody used org-duration-units which, even though LisP is a very flexible language, is NOT what I'd expect org-duration utilizer always foresaw to have to be flexible in terms of.
   > Similarly, I'd not spend too much code to check the format; i
   > understand the intent to preserve backward compatibility, bit if
   > that format adaptation mistake slipped forward to these days I
   > doubt that nice feature was used much so far...

   Yes, but it is easy to have a fallback, so why not.
   Because this way you persist in allowing a mistaken usage of that function. A number is a number but a duration is NOT just a number, and having something like (just for example) 10kg-0.5ton be taken as 10-0.5 is in my opinion NOT helpful to any user.
   The pcase matches either value pairs or single values, and org-duration-to-minutes can be charged to decide on what to do if values did not match the proper format (with the default assumption, I'd say, that simple integers are minutes).
   In facts org-duration-to-minutes already has a cond classical closing (t ...) branch to deal with that case. I'd leave the rest as I suggested; maybe, if you --as maintainer hence exposed to the global amount of supports request and use cases-- see that as convenient, with a different output format adapter than the one I picked (org-duration-from-minutes without an extra format specification actual argument); already allowing a custom format adapter would introduce an extra flexibility knob BUT consider that org-duration-to-minutes does NOT do the same (it implicitly utilizes the org-duration-format content, and has hardcoded assumption on quantities being representing times, so there too you'd need to change something, if it was not just a matter of definining a different alternative to display result times).
   Cheers!
     Andrea.

   --
   Ihor Radchenko // yantar92,
   Org mode contributor,
   Learn more about Org mode at <https://orgmode.org/>.
   Support Org development at <https://liberapay.com/org-mode>,
   or support my work at <https://liberapay.com/yantar92>

[-- Attachment #2: Type: text/html, Size: 5641 bytes --]

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

andrea@fedeli.eu writes:

>> About possibile abuses, org documentation, to date, clearly tells
>> org estimate utilizes times.
>
>> May you please elaborate?

>     AF: Sure! Clause 8.5 of current
>    (2023.Jul.14) org documentation,
>    https://orgmode.org/manual/Effort-Estimates.html, refers to effort
>    estimates giving examples that are all appearing to fall in time
>    domain.

I see what you mean.
However, est+ column summary does not have to be applied to EFFORT
columns.

One can, for example, use %SCORE{est+} column specification to summarize
values stored in SCORE property. Org has no right to demand SCORE to be
in time units and only time units.

>    > Similarly, I'd not spend too much code to check the format; i
>    > understand the intent to preserve backward compatibility, bit if
>    > that format adaptation mistake slipped forward to these days I
>    > doubt that nice feature was used much so far...
>
>    Yes, but it is easy to have a fallback, so why not.

>    Because this way you persist in allowing a mistaken usage of that
>    function. A number is a number but a duration is NOT just a number,
>    and having something like (just for example) 10kg-0.5ton be taken
>    as 10-0.5 is in my opinion NOT helpful to any user.

We may provide a linter (for M-x org-lint) that will ensure EFFORT
estimates to be in understandable format.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[andrea]

[-- Attachment #1: Type: text/plain, Size: 4233 bytes --]


   Howdy!
   I'm back to a previous element partially discussed as I found other org places where the duration had to be adapted to be able to deal with ranges: org-clock-get-clock-string and
   org-clock-notify-once-if-expired, both in og-clock.el; both get into action if you have a task you estimated and for which you're now tracking development time (quite handy, I have to say, as you're immediately warned you've running beyond estimations. For those two functions, I have introduced a similar change to the one I did originally to go from the basic string-to number on split-string to org-duration to minutes. Thanks, Sant Ignucius, for the debug-on-entry feature :))
   Two considerations here:
   1. I understand the fact that est+ doesn't have to necessarily be associated with effort, but it is quite clear from the docs which is the intent with which it was introduced: the only provided example is on times, and there we have to consider that time is expressed in durations.
   What I mean is that it does NOT make much sense to me to tell users the effort is to be written as 3d if given as a single value, and it has to be rewritten as 3-5 if we want to say "in a fork of 3 to 5 days", especially if somewhere else some other duration unit is used..
   2. Said differently, whether it was practically used also somewhere else, and not for time estimation, I can say nothing about; it is already quite hard to find it in doc, to date, and there  the only example given is about time.
   As without my changes the effort fork would not work properly in org-clock functions, if we really think estimation ranges shall be used also somewhere else I think we need to consider a possible generalization of what a duration is. In facts if we want to utilize it for other kind of measuring units (weight, money, etc.), it might make sense to pair it with a proper translation table like the one of duration that allows to convert days, hours, weeks, etc. into minutes, back and forth; this way we might allow both a proper type check according to the utilized measure unit, and would be able to avoid having to misleadlingly allow to sum tons to kg, for instance. (Recall: the ending letter today is just discarded hence 1kg-1ton is today taken as 1-1).
   Cheers,
     Andrea.

   Da "Ihor Radchenko" yantar92@posteo.net
   A andrea@fedeli.eu
   Cc emacs-orgmode@gnu.org
   Data Tue, 18 Jul 2023 09:10:33 +0000
   Oggetto Re: [BUG] Error in data input and output format for org-columns--summary-estimate
   andrea@fedeli.eu writes:

   >> About possibile abuses, org documentation, to date, clearly tells
   >> org estimate utilizes times.
   >
   >> May you please elaborate?

   > AF: Sure! Clause 8.5 of current
   > (2023.Jul.14) org documentation,
   > https://orgmode.org/manual/Effort-Estimates.html, refers to effort
   > estimates giving examples that are all appearing to fall in time
   > domain.

   I see what you mean.
   However, est+ column summary does not have to be applied to EFFORT
   columns.

   One can, for example, use %SCORE{est+} column specification to summarize
   values stored in SCORE property. Org has no right to demand SCORE to be
   in time units and only time units.

   > > Similarly, I'd not spend too much code to check the format; i
   > > understand the intent to preserve backward compatibility, bit if
   > > that format adaptation mistake slipped forward to these days I
   > > doubt that nice feature was used much so far...
   >
   > Yes, but it is easy to have a fallback, so why not.

   > Because this way you persist in allowing a mistaken usage of that
   > function. A number is a number but a duration is NOT just a number,
   > and having something like (just for example) 10kg-0.5ton be taken
   > as 10-0.5 is in my opinion NOT helpful to any user.

   We may provide a linter (for M-x org-lint) that will ensure EFFORT
   estimates to be in understandable format.

   --
   Ihor Radchenko // yantar92,
   Org mode contributor,
   Learn more about Org mode at <https://orgmode.org/>.
   Support Org development at <https://liberapay.com/org-mode>,
   or support my work at <https://liberapay.com/yantar92>

[-- Attachment #2: Type: text/html, Size: 5974 bytes --]

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

andrea@fedeli.eu writes:

>    Howdy!
>    I'm back to a previous element partially discussed as I found other org places where the duration had to be adapted to be able to deal with ranges: org-clock-get-clock-string and
>    org-clock-notify-once-if-expired, both in og-clock.el; both get into action if you have a task you estimated and for which you're now tracking development time (quite handy, I have to say, as you're immediately warned you've running beyond estimations. For those two functions, I have introduced a similar change to the one I did originally to go from the basic string-to number on split-string to org-duration to minutes. Thanks, Sant Ignucius, for the debug-on-entry feature :))

May you share your changes? I am not sure if I fully understand what and
why you did without seeing the diff.

>    Two considerations here:
>    1. I understand the fact that est+ doesn't have to necessarily be associated with effort, but it is quite clear from the docs which is the intent with which it was introduced: the only provided example is on times, and there we have to consider that time is expressed in durations.
>    What I mean is that it does NOT make much sense to me to tell users the effort is to be written as 3d if given as a single value, and it has to be rewritten as 3-5 if we want to say "in a fork of 3 to 5 days", especially if somewhere else some other duration unit is used..

It should not. The reason `org-columns--summary-estimate' uses
split-string is because it may have to work with recursively calculated
estimates from subtrees. EFFORT property itself does not officially
support ranges.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[andrea]

[-- Attachment #1: Type: text/plain, Size: 3305 bytes --]


   IR > May you share your changes?
   Sure!
   Here they are: In these slices I take the upper part of the fork (where in case, assuming a small-big usage convention ;)) as that is the value that surely testify the effort estimation overrun. Being so, at the time of this writing I just realized the use of pcase is likely replaceable by a simpler (car (last (mapcar...))) call :).
   Cheers,
     Andrea

   Job: diff -bwi org-clock.el{.old,}

   720c720,723

   < (let* ((effort-in-minutes (org-duration-to-minutes org-clock-effort))

   ---

   >         (let* ((effort-in-minutes

   >                 (pcase (mapcar #'org-duration-to-minutes (split-string org-clock-effort "-"))

   >                   (`(,_ ,value) value)

   >                   (`(,value) value)))

   828c831,833

   <     (let ((effort-in-minutes (org-duration-to-minutes org-clock-effort))

   ---

   >     (let ((effort-in-minutes (pcase (mapcar #'org-duration-to-minutes (split-string org-clock-effort "-"))

   >                                (`(,_ ,value) value)

   >                                (`(,value) value)))

   Da emacs-orgmode-bounces+andrea=fedeli.eu@gnu.org
   A andrea@fedeli.eu
   Cc emacs-orgmode@gnu.org
   Data Wed, 16 Aug 2023 10:15:13 +0000
   Oggetto Re: [BUG] Error in data input and output format for org-columns--summary-estimate
   andrea@fedeli.eu writes:

   > Howdy!
   > I'm back to a previous element partially discussed as I found other org places where the duration had to be adapted to be able to deal with ranges: org-clock-get-clock-string and
   > org-clock-notify-once-if-expired, both in og-clock.el; both get into action if you have a task you estimated and for which you're now tracking development time (quite handy, I have to say, as you're immediately warned you've running beyond estimations. For those two functions, I have introduced a similar change to the one I did originally to go from the basic string-to number on split-string to org-duration to minutes. Thanks, Sant Ignucius, for the debug-on-entry feature :))

   May you share your changes? I am not sure if I fully understand what and
   why you did without seeing the diff.

   > Two considerations here:
   > 1. I understand the fact that est+ doesn't have to necessarily be associated with effort, but it is quite clear from the docs which is the intent with which it was introduced: the only provided example is on times, and there we have to consider that time is expressed in durations.
   > What I mean is that it does NOT make much sense to me to tell users the effort is to be written as 3d if given as a single value, and it has to be rewritten as 3-5 if we want to say "in a fork of 3 to 5 days", especially if somewhere else some other duration unit is used..

   It should not. The reason `org-columns--summary-estimate' uses
   split-string is because it may have to work with recursively calculated
   estimates from subtrees. EFFORT property itself does not officially
   support ranges.

   --
   Ihor Radchenko // yantar92,
   Org mode contributor,
   Learn more about Org mode at <https://orgmode.org/>.
   Support Org development at <https://liberapay.com/org-mode>,
   or support my work at <https://liberapay.com/yantar92>

[-- Attachment #2: Type: text/html, Size: 4874 bytes --]

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

andrea@fedeli.eu writes:

>    IR > May you share your changes?
>    Sure!
>    Here they are: In these slices I take the upper part of the fork (where in case, assuming a small-big usage convention ;)) as that is the value that surely testify the effort estimation overrun. Being so, at the time of this writing I just realized the use of pcase is likely replaceable by a simpler (car (last (mapcar...))) call :).
>    >         (let* ((effort-in-minutes
>
>    >                 (pcase (mapcar #'org-duration-to-minutes (split-string org-clock-effort "-"))
>
>    >                   (`(,_ ,value) value)
>
>    >                   (`(,value) value)))

I see.
As I said, Org does not currently support EFFORT to be a range.
Ranges only appear in column est+ summaries.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>