[BUG] Future repeated tasks marked done in Org Agenda don't show as done [9.5 (9.5-g0a86ad @ /home/gustavo/.emacs.d/elpa/org-9.5/)]

[BUG] Future repeated tasks marked done in Org Agenda don't show as done [9.5 (9.5-g0a86ad @ /home/gustavo/.emacs.d/elpa/org-9.5/)]

[Gustavo Barros]

Hi All,

I've been meeting a small glitch on the Agenda, which had been eluding 
me for some time, as it "sometimes works, sometimes doesn't", and I 
wasn't being able to recognize the rule for it.  So I started keeping 
track of it a while, and I was thus able to come up with a ECM.  I'm not 
sure the "rule" is the one I came up with, since it was based on 
inference, but I hope the ECM is reproducible.

The glitch is that some repeated tasks, when marked done in the Agenda, 
show no visual feedback that the action has taken place, as usual, and 
if you refresh the Agenda, they just vanish, which demonstrates the 
action had indeed taken place in the agenda file, just not shown in the 
Agenda buffer itself.  And, as far as I can tell, this happens to 
repeated tasks, scheduled in future.  For tasks scheduled for today or 
in the past, they appear to be done as expected.

The ECM runs as follows.  Start ~emacs -Q~ and setup current Org and the 
agenda files:

#+begin_src emacs-lisp
(add-to-list 'load-path "~/.emacs.d/elpa/org-9.5")
(setq org-agenda-files '("~/test/agenda.org"))
#+end_src

Where =~/test/agenda.org= contains:

#+begin_src org
,* TODO Foo (scheduled for yesterday)
SCHEDULED: <2021-10-28 Thu +1w>

,* TODO Bar (scheduled for today)
SCHEDULED: <2021-10-29 Fri +1w>

,* TODO Baz (scheduled for tomorrow)
SCHEDULED: <2021-10-30 Sat +1w>
#+end_src

And note the dates relative to today are critical for the ECM to 
demonstrate what it intends to.  So, if you try this out another day, 
shift the schedules as appropriate.

Now, run ~M-x org-agenda~, then "a" to get to the default agenda.  And 
mark each of these tasks done with "t".

"Foo" and "Bar" appear as "DONE", as usual and expected.  "Baz" does 
not.

Refresh the agenda, and see they are indeed all gone from this week's 
view (hence marked done in the actual file).  And indeed, with "f" we 
see they are there next week.

Best regards,
Gustavo.




Emacs  : GNU Emacs 27.2 (build 1, x86_64-pc-linux-gnu, GTK+ Version 
3.24.20, cairo version 1.16.0)
 of 2021-03-25
Package: Org mode version 9.5 (9.5-g0a86ad @ 
/home/gustavo/.emacs.d/elpa/org-9.5/)

current state:
==============
(setq
 org-src-mode-hook '(org-src-babel-configure-edit-buffer 
 org-src-mode-configure-edit-buffer)
 org-link-shell-confirm-function 'yes-or-no-p
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-agenda-files '("~/test/agenda.org")
 org-mode-hook '(#[0 "\300\301\302\303\304$\207" [add-hook 
 change-major-mode-hook org-show-all append local] 5]
		 #[0 "\300\301\302\303\304$\207"
		   [add-hook change-major-mode-hook 
		   org-babel-show-result-all append local] 5]
		 org-babel-result-hide-spec org-babel-hide-all-hashes)
 org-archive-hook '(org-attach-archive-delete-maybe)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-bibtex-headline-format-function #[257 "\300\236A\207" [:title] 3 
 "\n\n(fn ENTRY)"]
 org-babel-pre-tangle-hook '(save-buffer)
 org-tab-first-hook '(org-babel-hide-result-toggle-maybe 
 org-babel-header-arg-expand)
 org-agenda-loop-over-headlines-in-active-region nil
 org-occur-hook '(org-first-headline-recenter)
 org-cycle-hook '(org-cycle-hide-archived-subtrees 
 org-cycle-hide-drawers org-cycle-show-empty-lines
		  org-optimize-window-after-visibility-change)
 org-speed-command-hook '(org-speed-command-activate 
 org-babel-speed-command-activate)
 org-export-before-parsing-hook '(org-attach-expand-links)
 org-confirm-shell-link-function 'yes-or-no-p
 org-link-parameters '(("attachment" :follow org-attach-follow :complete 
 org-attach-complete-link)
		       ("id" :follow org-id-open) ("eww" :follow 
		       org-eww-open :store org-eww-store-link)
		       ("rmail" :follow org-rmail-open :store 
		       org-rmail-store-link)
		       ("mhe" :follow org-mhe-open :store 
		       org-mhe-store-link)
		       ("irc" :follow org-irc-visit :store 
		       org-irc-store-link :export org-irc-export)
		       ("info" :follow org-info-open :export 
		       org-info-export :store org-info-store-link)
		       ("gnus" :follow org-gnus-open :store 
		       org-gnus-store-link)
		       ("docview" :follow org-docview-open :export 
		       org-docview-export :store
			org-docview-store-link)
		       ("bibtex" :follow org-bibtex-open :store 
		       org-bibtex-store-link)
		       ("bbdb" :follow org-bbdb-open :export 
		       org-bbdb-export :complete org-bbdb-complete-link
			:store org-bbdb-store-link)
		       ("w3m" :store org-w3m-store-link)
		       ("doi" :follow org-link-doi-open :export 
		       org-link-doi-export) ("file+sys") ("file+emacs")
		       ("shell" :follow org-link--open-shell)
		       ("news" :follow
			#[514 "\301\300\302Q\"\207" ["news" browse-url 
                         ":"] 6 "\n\n(fn URL ARG)"])
		       ("mailto" :follow
			#[514 "\301\300\302Q\"\207" ["mailto" browse-url 
                         ":"] 6 "\n\n(fn URL ARG)"])
		       ("https" :follow
			#[514 "\301\300\302Q\"\207" ["https" browse-url 
                         ":"] 6 "\n\n(fn URL ARG)"])
		       ("http" :follow
			#[514 "\301\300\302Q\"\207" ["http" browse-url 
                         ":"] 6 "\n\n(fn URL ARG)"])
		       ("ftp" :follow
			#[514 "\301\300\302Q\"\207" ["ftp" browse-url 
                         ":"] 6 "\n\n(fn URL ARG)"])
		       ("help" :follow org-link--open-help :store 
		       org-link--store-help)
		       ("file" :complete org-link-complete-file) 
		       ("elisp" :follow org-link--open-elisp))
 org-link-elisp-confirm-function 'yes-or-no-p
 )

Re: [BUG] Future repeated tasks marked done in Org Agenda don't show as done [9.5 (9.5-g0a86ad @ /home/gustavo/.emacs.d/elpa/org-9.5/)]

[Ihor Radchenko]

Gustavo Barros <gusbrs.2016@gmail.com> writes:

> The glitch is that some repeated tasks, when marked done in the Agenda, 
> show no visual feedback that the action has taken place, as usual, and 
> if you refresh the Agenda, they just vanish, which demonstrates the 
> action had indeed taken place in the agenda file, just not shown in the 
> Agenda buffer itself.  And, as far as I can tell, this happens to 
> repeated tasks, scheduled in future.  For tasks scheduled for today or 
> in the past, they appear to be done as expected.

Confirmed

Best,
Ihor

Re: [BUG] Future repeated tasks marked done in Org Agenda don't show as done [9.5 (9.5-g0a86ad @ /home/gustavo/.emacs.d/elpa/org-9.5/)]

[Gustavo Barros]

Hi Ihor,

On Fri, 29 Oct 2021 at 21:40, Ihor Radchenko <yantar92@gmail.com> wrote:
>
> Confirmed
>

Thanks for checking and marking.

Best,
Gustavo.

Re: [BUG] Future repeated tasks marked done in Org Agenda don't show as done [9.5 (9.5-g0a86ad @ /home/gustavo/.emacs.d/elpa/org-9.5/)]

[Gustavo Barros]

Hi All,

On Fri, 29 Oct 2021 at 21:40, Ihor Radchenko <yantar92@gmail.com> wrote:

> Gustavo Barros <gusbrs.2016@gmail.com> writes:
>
>> The glitch is that some repeated tasks, when marked done in the 
>> Agenda, 
>> show no visual feedback that the action has taken place, as usual, 
>> and 
>> if you refresh the Agenda, they just vanish, which demonstrates the 
>> action had indeed taken place in the agenda file, just not shown in 
>> the 
>> Agenda buffer itself.  And, as far as I can tell, this happens to 
>> repeated tasks, scheduled in future.  For tasks scheduled for today 
>> or 
>> in the past, they appear to be done as expected.
>
> Confirmed
>
> Best,
> Ihor

This is a respectful bump on this one.

But not to bump empty handed, I did some investigation on this, and I 
think I know why the problem happens.

At `org-agenda-todo' when a task is a repeating one, the value of 
`org-agenda-headline-snapshot-before-repeat' stored at `org-todo' may or 
may not replace `newhead' depending on some conditions, which are:

#+begin_src emacs-lisp
(when (and org-agenda-headline-snapshot-before-repeat
	   (not (equal org-agenda-headline-snapshot-before-repeat
		       newhead))
	   todayp)
  (setq newhead org-agenda-headline-snapshot-before-repeat
	just-one t))
#+end_src

So that `newhead' is set to `org-agenda-headline-snapshot-before-repeat' 
only if `todayp' is non-nil.  And, indeed, this seems to be the 
condition which results in the missing visual feedback reported here. 
I've tried without it, and it works.  (I'm currently using built-in 
9.5.2, but I think there's no change in the function to current release 
9.5.4 and also, light testing with the latter suggests no change in the 
reported issue).

What I'm not sure is why this condition is there in the first place. 
That's the only place where the let-bound `todayp' is used in the 
function, so I may be missing why it exists and the purpose of this 
condition.  But one side-effect of it is that, if you happen to do a 
repeating task ahead of schedule, you won't see the change of todo state 
in the agenda.

Best regards,
Gustavo.

Re: [BUG] Future repeated tasks marked done in Org Agenda don't show as done [9.5 (9.5-g0a86ad @ /home/gustavo/.emacs.d/elpa/org-9.5/)]

[Ihor Radchenko]

Gustavo Barros <gusbrs.2016@gmail.com> writes:

> But not to bump empty handed, I did some investigation on this, and I 
> think I know why the problem happens.

Thanks for the detailed analysis!

> What I'm not sure is why this condition is there in the first place. 
> That's the only place where the let-bound `todayp' is used in the 
> function, so I may be missing why it exists and the purpose of this 
> condition.  But one side-effect of it is that, if you happen to do a 
> repeating task ahead of schedule, you won't see the change of todo state 
> in the agenda.

I dug through the old commits and found where this behaviour has been
introduced:

Commit 0bbf3a9bd message details the current behaviour and its caveats:

>> When marking a repeated entry DONE in the daily or weekly agenda, that
>> task would previously still be shown as TODO, because the repeater
>> immediately restores the TODO state after moving the time stamp.  This
>> is bad feedback.
>> 
>> This problem was hard to fix.  Because the same line may be present in
>> other lines in the same weekly agenda, we cannot simply update all
>> lines related to this entry.
>> 
>> What we do now is this:  Before the repeater does its work in shifting
>> the time stamp and resetting the TODO keyword, we take a snapshot of
>> the headline as it looks then.  And then, when we update the agenda
>> view, we change only the line at the cursor instead of all lines
>> related to this entry.  We also make sure that this is only so if the
>> cursor is in a daily/weekly agenda, on TODAY's date.
>> 
>> There still remain possible inconsistencies.  For example, if you have
>> a daily repeating task in the weekly agenda, and you move the cursor a
>> few days into the future and mark it DONE there, the entry will
>> actually be marked DONE for today, but still show up in today's task
>> list as TODO.  refreshing the agenda will fix the display in such an
>> unlikely case.
>> 
>> Thanks to Jack ??? for noticing and reporting this issue.

As you can see, the todayp condition is avoiding issues with weekly
agenda when the same habit is displayed multiple times.

The problem you observed is also noted and left unresolved.

Ideas how to deal with the described are welcome!

Best,
Ihor

Re: [BUG] Future repeated tasks marked done in Org Agenda don't show as done [9.5 (9.5-g0a86ad @ /home/gustavo/.emacs.d/elpa/org-9.5/)]

[Gustavo Barros]

Hi Ihor,

On Tue, 12 Jul 2022 at 10:46, Ihor Radchenko <yantar92@gmail.com> wrote:

> Thanks for the detailed analysis!

I thank you again for your continued interest in this little report.

> I dug through the old commits and found where this behaviour has been
> introduced:
>
> Commit 0bbf3a9bd message details the current behaviour and its 
> caveats:

Ah! this definetely clears up the intended purpose of the condition. 
Great dig!

> As you can see, the todayp condition is avoiding issues with weekly
> agenda when the same habit is displayed multiple times.
>
> The problem you observed is also noted and left unresolved.
>
> Ideas how to deal with the described are welcome!

I can try to think this trough with you, if you'd like.  Since I'm the 
reporter and bumper, it is fair that I start and try to build a base for 
it.

The TL;DR for what follows is that I think `todayp' is ultimately the 
"wrong" condition to apply, but is a good *proxy*.  Perhaps there's a 
chance to get to a more correct condition, but I'm not sure.  But, even 
if not, I'd like to argue that the "occurrence at point" may be a better 
proxy, which would be the condition applied (as far as I can tell, which 
see) if the `todayp' condition were dropped.

The long version:

First a preliminary observation.  I think the case "noted and left 
unresolved" in the commit message is somewhat different than the one 
reported here.  Related, of course, but different.

Let's consider a hypothetical agenda with the following characteristics: 
a weekly agenda starting on Monday (fixed), today is Tuesday.  Unless 
stated otherwise, this is the scenario for the examples that follow.

A daily repeating task, scheduled for today will appear in the agenda 
from Tuesday to Sunday.  If we move point to the occurrence of that task 
on, say, Thursday, and mark it done then, we would have the case 
described in the commit message.  I'm not sure it is "unlikely", but we 
could argue that this is "dubious user input".

Now consider the case of a weekly repeating task scheduled for Thursday. 
This is the case reported here.  And I think marking this entry done 
"ahead of schedule" is arguably more legitimate user input. For once, 
this entry only appears once in the given agenda view, there is no 
option to use any other.


That said, let's try to be systematic. There are a number of reasons an 
entry may appear multiple times in an agenda view:

A1) The entry is scheduled in a past date, and this past date is also 
visible in the agenda view. In the example agenda a task scheduled for 
Monday would appear both Monday, and today, Tuesday.

A2) The entry has a deadline in a future date, this future date is 
visible in the agenda view, and the deadline warning settings are enough 
to be shown today as well.  A task with a deadline for Thursday would 
appear today, Tuesday, and Thursday.

B) The entry is scheduled (deadlined?) to a range of dates.  For 
example, a task scheduled to a range from Thursday to Saturday this week 
would appear four times in the agenda view, once for the regular 
schedule and thrice for the range "(N/3)".

C) The entry has a repeater whose frequency is higher than the span of 
the agenda view.  A daily task on a weekly view, a weekly task on a 
monthly view, etc.

Of course, a given entry may appear in the agenda multiple times for 
multiple of these reasons.

That's all I can think of.  Do you see any other cases?  This is a 
critical question, because the soundness of the argument depends on this 
list being exhaustive.  Anyway, pending on that, let me go on.

Now, this bug only affects repeating tasks, because the problem arises 
only for them because their state in the underlying buffer does not 
correspond to the "todo change the user has just applied".  Indeed, 
`org-agenda-headline-snapshot-before-repeat' is correspondingly just 
stored for them, as the name implies.

Furthermore, reasons A1, A2 and B, are not specific to repeating tasks, 
though they affect them too, of course.  Reason C is the only one 
specific to repeating tasks, and is really the only reason I think 
grants for the case considered in the commit message:

>> Because the same line may be present in
>> other lines in the same weekly agenda, we cannot simply update all
>> lines related to this entry.

Indeed, a non-repeating task which appears multiple times in the agenda 
view (A1, A2, or B), when marked done, is visually changed as such in 
all occurrences.  The same does not happen for a repeating entry 
because, well, "there might be C (as well?)...".

That's the nature of the problem, as far as I can see.  And a real one 
at that.  I don't know enough of the agenda machinery to know if among 
the metadata stored as text properties we would be able to distinguish 
"C" from the other reasons for a given occurrence of a given entry.  It 
is probably fair to presume it is not possible to distinguish, otherwise 
Carsten might have leveraged that information.

That given, `todayp' does get us a good approximation of the case which 
appears to have been focused in the commit: a daily task on a weekly 
agenda view.  However, it fails to get any occurrence in the case 
reported here: a weekly task on a weekly agenda scheduled in the future 
and marked done ahead of schedule.  This discussion was enough for me to 
conceive another failing case: a weekly task scheduled in the past, and 
marked done in the past occurrence (say the task scheduled to yesterday, 
Monday, will be visually marked correctly if marked done in the today 
occurrence, but not in this is done from the past one).

Some ways which could improve things.  First, we could rule out the "C" 
case for repeated entries whose repeating frequency is not higher than 
the span of the agenda view.  We know that an entry with a weekly 
repeater will not appear more than once in a weekly agenda view.  Well, 
at least not for reason "C", it may appear multiple times for other 
reasons, in which case it is not a problem, and we can apply the state 
change to all occurrences, as is done for a similar entry which does not 
repeat (that, is visually mark all occurrences with the new state). 
(The frequency may have to be stored for that, but it is likely viable 
and not too complicated, in practice as long as we can distinguish the 
case, we could not set `just-one' to `t' for it).

Second, we may wish to choose a better proxy for the remainder cases. 
One could argue for the "first occurrence" or "the occurrence at point". 
Actually, the current state of things is "the occurrence at point, if 
today".  I've argued above that there exist cases when the `todayp' 
condition fails to catch the intended case.  It is also a double 
condition trying to exclude visual state change of "unintended cases".

`todayp' is not even sufficient to ensure correctness of the case in 
focus at the commit message ("correctness" here meaning a precise 
correspondence between the operation visualized in the agenda and the 
operation actually carried out in the underlying buffer).  Suppose, for 
example, a daily repeating task on a weekly view ("+1d" repeater), but 
which is a day late.  In our example agenda, scheduled for Monday.  It 
appears every day in our weekly view.  If we go to today and mark the 
task done, it will be visually marked as such. But it was actually 
marked done "yesterday", and if we now refresh the agenda, the task is 
back today as scheduled (and correctly so).

But, if the user asked an entry to be marked as done at a certain point 
in the agenda buffer, is it really so bad to visually mark the 
occurrence at point (regardless of whether it is in today's date), even 
if the operation in the underlying buffer happened in a different day? 
In other words, why not drop the `todayp' condition?  If I read the code 
correctly, the `just-one' argument passed to 
`org-agenda-change-all-lines' already restricts the visual change to the 
occurrence at point.

WDYT?

Best,
Gustavo.

Re: [BUG] Future repeated tasks marked done in Org Agenda don't show as done [9.5 (9.5-g0a86ad @ /home/gustavo/.emacs.d/elpa/org-9.5/)]

[Ihor Radchenko]

Gustavo Barros <gusbrs.2016@gmail.com> writes:

> ... (lots of detailed analysis with various cases)
> WDYT?

I feel that you are overcomplicating things a bit.
What if we simply change all the agenda lines if and only if their date
in agenda is earlier or equal to the next scheduled time (after repeater
is triggered)?

Best,
Ihor

Re: [BUG] Future repeated tasks marked done in Org Agenda don't show as done [9.5 (9.5-g0a86ad @ /home/gustavo/.emacs.d/elpa/org-9.5/)]

[Gustavo Barros]

Hi Ihor,

On Mon, 18 Jul 2022 at 14:28, Ihor Radchenko <yantar92@gmail.com> wrote:

> I feel that you are overcomplicating things a bit.

Well, the most important objective of the analysis was to try to figure 
out if the `todayp' condition was too strict or not.  Since your 
suggested fix implies removing it as well, I take I have you convinced? 
;)

> What if we simply change all the agenda lines if and only if their 
> date
> in agenda is earlier or equal to the next scheduled time (after 
> repeater
> is triggered)?

I think this is a good condition, better than the one I had devised 
(considering the span of the agenda and the repetition frequency).  As 
far as I can tell, it will also require parsing the planning info of the 
entry and consider the relevant cases, but perhaps you are seeing more 
than I can.  And I presume you refer only to repeating tasks, so that 
the rest of the current conditional (except `todayp') would remain in 
place.  The only case I can think of which might trip a little this 
condition is a date range, but "date range with repeater" seems corner 
enough for us not to sweat much over it.

One thing to consider is the timing of things and how it affects the 
conditional.  You said "earlier or equal to the next scheduled time 
(after repeater is triggered)".  We are looking at things in 
`org-agenda-todo' after `org-todo' has done its job, in other words, 
after the repeater has "stepped". 
`org-agenda-headline-snapshot-before-repeat' is stored in `org-todo' 
because there would be no longer any way to retrieve it at this point, 
but we don't store anything else, as far as I can tell.  So, at this 
point, the current scheduled (dedlined, etc.) time would be "the 
next/first instance which is *not* done", and hence wouldn't it be 
"earlier, but not equal, to the next scheduled time"?

Another thing for which you might have something in mind, but I don't 
see how to handle it is that `org-agenda-change-all-lines' can receive 
the `just-this' argument and that decides whether just the current line 
gets changed or if all entries with the same marker are looped over.  So 
it's either "this" or "all". Perhaps the version of the conditional you 
suggested will require change in `org-agenda-change-all-lines' so that 
it can change "some, according to a given predicate".

In sum, all in all, the conditional you suggested sounds very good to 
me, and a clear improvement relative to the current state of things. 
And regardless of anything else, it seems we agree that `todayp' is too 
strict and removing it would be the low hanging fruit in all this.

Best,
Gustavo.