tags-todo org-agenda-custom-command weirdness

tags-todo org-agenda-custom-command weirdness

[Stig Brautaset]

Hi,

I'm seeing some weirdness with tags-todo vs tags custom agenda commands.
Below is the smallest case I've managed to narrow it down to. I would
*expect* that all the a1-4 commands return the same results, and that
the b1-4 return the same results. That is not what I'm seeing, however.

(setq-default org-agenda-custom-commands
	      '(("a1" "A1" tags-todo "-PROJ/TODO")
                ("a2" "A2" tags "-PROJ/TODO")
		("a3" "A3" ((tags-todo "-PROJ/TODO")))
		("a4" "A4" ((tags "-PROJ/TODO")))
		
		("b1" "B1" tags-todo "-PROJ/DONE")
		("b2" "B2" tags "-PROJ/DONE")
		("b3" "B3" ((tags-todo "-PROJ/DONE")))
		("b4" "B4" ((tags "-PROJ/DONE")))))


The effect of a1 & a2 appears identical to eachother, and a3 & a4 too,
but a1-2 and a3-4 differ (in that the former seems to include scheduled
items, but the latter does not.)

What I'm seeing for b1-4 is even weirder. Here I observe b1, b2 and b4
have identical behaviour[*], but b3 is the odd one out: while the other
three return a list of DONE non-project tasks, b3 doesn't find anything.

[*] I suspect the reason b2 and b4 are identical is that I have no
scheduled DONE tasks.

I also see this with =m= vs =M= in the org-agenda menu. I see it even
when starting emacs with -q, so it appears to not be some spectactular
breakage on my part. Is anyone else seing this?


Stig

Re: tags-todo org-agenda-custom-command weirdness

[Stig Brautaset]

Stig Brautaset <stig@brautaset.org> writes:

> Hi,
>
> I'm seeing some weirdness with tags-todo vs tags custom agenda commands.

I forgot to mention my Emacs / Org versions:

- GNU Emacs 28.0.50 (build 1, x86_64-apple-darwin19.4.0, NS appkit-1894.40 Version 10.15.4 (Build 19E287)) of 2020-04-18
- Org mode version 9.3.7 (9.3.7-2-g706970-elpaplus @ /Users/stig/.emacs.d/elpa/org-plus-contrib-20200608/)

Stig

Re: tags-todo org-agenda-custom-command weirdness

[Kyle Meyer]

Stig Brautaset writes:

> I'm seeing some weirdness with tags-todo vs tags custom agenda commands.
> Below is the smallest case I've managed to narrow it down to. I would
> *expect* that all the a1-4 commands return the same results, and that
> the b1-4 return the same results. That is not what I'm seeing, however.
>
> (setq-default org-agenda-custom-commands
>             '(("a1" "A1" tags-todo "-PROJ/TODO")
>               ("a2" "A2" tags "-PROJ/TODO")
>               ("a3" "A3" ((tags-todo "-PROJ/TODO")))
>               ("a4" "A4" ((tags "-PROJ/TODO")))
>               
>               ("b1" "B1" tags-todo "-PROJ/DONE")
>               ("b2" "B2" tags "-PROJ/DONE")
>               ("b3" "B3" ((tags-todo "-PROJ/DONE")))
>               ("b4" "B4" ((tags "-PROJ/DONE")))))
>

I agree with your expectation that a1-4 should be the same.  Why do you
expect that b1-4 would be?  Shouldn't tags-todo consider only none-DONE
todo items, in the same spirit as m versus M?  In other words, I'd
expect b1 and b3 to be empty by definition.

> The effect of a1 & a2 appears identical to eachother, and a3 & a4 too,
> but a1-2 and a3-4 differ (in that the former seems to include scheduled
> items, but the latter does not.)

It's helpful if you provide a minimal test file.  I've tried to come up
with one that I think should capture what you're describing.

--8<---------------cut here---------------start------------->8---
* TODO h1                                                               :PROJ:
* TODO h2
* DONE h3                                                               :PROJ:
* DONE h4
* TODO h5                                                               :PROJ:
SCHEDULED: <2020-06-14 Sun>
* TODO h6
SCHEDULED: <2020-06-14 Sun>
* DONE h7                                                               :PROJ:
SCHEDULED: <2020-06-14 Sun>
* DONE h8
SCHEDULED: <2020-06-14 Sun>
--8<---------------cut here---------------end--------------->8---

With that, a1-4 all show:

    scratch:    TODO h2
    scratch:    TODO h6

That doesn't match what you're seeing.  I'm testing with 706970 checked
out, the commit you reported in your follow-up message.

> What I'm seeing for b1-4 is even weirder. Here I observe b1, b2 and b4
> have identical behaviour[*], but b3 is the odd one out: while the other
> three return a list of DONE non-project tasks, b3 doesn't find anything.
>
> [*] I suspect the reason b2 and b4 are identical is that I have no
> scheduled DONE tasks.

For b2 and b4, I see

    scratch:    DONE h4
    scratch:    DONE h8

b1 and b3 are empty, which I think is expected given the tags-todo type.

Re: tags-todo org-agenda-custom-command weirdness

[Stig Brautaset]

Kyle Meyer <kyle@kyleam.com> writes:

> Stig Brautaset writes:
>
>> I'm seeing some weirdness with tags-todo vs tags custom agenda commands.
>> Below is the smallest case I've managed to narrow it down to. I would
>> *expect* that all the a1-4 commands return the same results, and that
>> the b1-4 return the same results. That is not what I'm seeing, however.
>>
>> (setq-default org-agenda-custom-commands
>>             '(("a1" "A1" tags-todo "-PROJ/TODO")
>>               ("a2" "A2" tags "-PROJ/TODO")
>>               ("a3" "A3" ((tags-todo "-PROJ/TODO")))
>>               ("a4" "A4" ((tags "-PROJ/TODO")))
>>               
>>               ("b1" "B1" tags-todo "-PROJ/DONE")
>>               ("b2" "B2" tags "-PROJ/DONE")
>>               ("b3" "B3" ((tags-todo "-PROJ/DONE")))
>>               ("b4" "B4" ((tags "-PROJ/DONE")))))
>>
>
> I agree with your expectation that a1-4 should be the same.  Why do you
> expect that b1-4 would be?  Shouldn't tags-todo consider only none-DONE
> todo items, in the same spirit as m versus M?  In other words, I'd
> expect b1 and b3 to be empty by definition.

I expected b1-4 to all yield the same result because I expect
`tags-todo` to search all TODO-items, not just TODO items where the
state is "TODO".


>> The effect of a1 & a2 appears identical to eachother, and a3 & a4 too,
>> but a1-2 and a3-4 differ (in that the former seems to include scheduled
>> items, but the latter does not.)
>
> It's helpful if you provide a minimal test file.  I've tried to come up
> with one that I think should capture what you're describing.

Mea culpa, I should have done this. 

>
> --8<---------------cut here---------------start------------->8---
> * TODO h1                                                               :PROJ:
> * TODO h2
> * DONE h3                                                               :PROJ:
> * DONE h4
> * TODO h5                                                               :PROJ:
> SCHEDULED: <2020-06-14 Sun>
> * TODO h6
> SCHEDULED: <2020-06-14 Sun>
> * DONE h7                                                               :PROJ:
> SCHEDULED: <2020-06-14 Sun>
> * DONE h8
> SCHEDULED: <2020-06-14 Sun>
> --8<---------------cut here---------------end--------------->8---
>
> With that, a1-4 all show:
>
>     scratch:    TODO h2
>     scratch:    TODO h6
>
> That doesn't match what you're seeing.  I'm testing with 706970 checked
> out, the commit you reported in your follow-up message.

For me a2 and a4 returns the same, but a1 and a3 just returns TODO h2.
However, I can reproduce what you're seeing by setting this:

(setq org-agenda-todo-ignore-scheduled nil)


>> What I'm seeing for b1-4 is even weirder. Here I observe b1, b2 and b4
>> have identical behaviour[*], but b3 is the odd one out: while the other
>> three return a list of DONE non-project tasks, b3 doesn't find anything.
>>
>> [*] I suspect the reason b2 and b4 are identical is that I have no
>> scheduled DONE tasks.
>
> For b2 and b4, I see
>
>     scratch:    DONE h4
>     scratch:    DONE h8
>
> b1 and b3 are empty, which I think is expected given the tags-todo type.

Well, I disagree :-)

This behaviour of `tags-todo` seems inconsistent to me. If `todo` can
find DONE items, why shouldn't `tags-todo` do the same?

Stig

Re: tags-todo org-agenda-custom-command weirdness

[Kyle Meyer]

Stig Brautaset writes:

> For me a2 and a4 returns the same, but a1 and a3 just returns TODO h2.
> However, I can reproduce what you're seeing by setting this:
>
> (setq org-agenda-todo-ignore-scheduled nil)

So, provided you've configured org-agenda-todo-ignore-scheduled and
org-agenda-tags-todo-honor-ignore-options away from their default values
of nil, the a1-4 behavior you described is as advertised, correct?

>> b1 and b3 are empty, which I think is expected given the tags-todo type.
>
> Well, I disagree :-)
>
> This behaviour of `tags-todo` seems inconsistent to me. If `todo` can
> find DONE items, why shouldn't `tags-todo` do the same?

Perhaps it should.  The behavior has been that way for a long time, and
the code makes it look very deliberate.  That of course is not an
argument that it should be that way, just one basis for expectations
(and a reason to be wary of breaking workflows).

And it looks like it was actually supposed to change to your preference
in 2017.  There was a report [0] that essentially boils down to what
you're saying, I think.  In response, 942b6267a (org-agenda: `tags-todo'
command type includes DONE keywords, 2017-04-18) was applied, but then
reverted for reasons not related to the intended change in behavior [1].

There was then a follow-up in 2fb129b5c (`org-scan-tags' retrieve all
TODO keywords, not only not-done ones, 2017-08-17).  As far as I can
tell, that was supposed to achieve the behavior you're after but didn't.
I don't have time to dig much at the moment, but quickly stepping
through org-scan-tags, I think the issue is that the MATCHER argument
still filters out done states.


[0] https://orgmode.org/list/CAF96XX0XxhPKjAxy0dQmOiy3rnT+DUok4p1y71F1AwyJaNL-MA@mail.gmail.com/
[1] https://orgmode.org/list/874lt89fi2.fsf@free.fr/

Re: tags-todo org-agenda-custom-command weirdness

[Stig Brautaset]

Kyle Meyer <kyle@kyleam.com> writes:

> Stig Brautaset writes:
>
>> For me a2 and a4 returns the same, but a1 and a3 just returns TODO h2.
>> However, I can reproduce what you're seeing by setting this:
>>
>> (setq org-agenda-todo-ignore-scheduled nil)
>
> So, provided you've configured org-agenda-todo-ignore-scheduled and
> org-agenda-tags-todo-honor-ignore-options away from their default values
> of nil, the a1-4 behavior you described is as advertised, correct?

I'm not sure. I would have expected that my configured values of
org-agenda-todo-ignore-scheduled and
org-agenda-tags-todo-honor-ignore-options would apply for all a1-4, so
whatever I configure those to the result of a1-4 should be consistent.
It looks like `tags-todo` takes my configured values into account, but
`tags` does not.

>>> b1 and b3 are empty, which I think is expected given the tags-todo type.
>>
>> Well, I disagree :-)
>>
>> This behaviour of `tags-todo` seems inconsistent to me. If `todo` can
>> find DONE items, why shouldn't `tags-todo` do the same?
>
> Perhaps it should.  The behavior has been that way for a long time, and
> the code makes it look very deliberate.  That of course is not an
> argument that it should be that way, just one basis for expectations
> (and a reason to be wary of breaking workflows).
>
> And it looks like it was actually supposed to change to your preference
> in 2017.  There was a report [0] that essentially boils down to what
> you're saying, I think.  In response, 942b6267a (org-agenda: `tags-todo'
> command type includes DONE keywords, 2017-04-18) was applied, but then
> reverted for reasons not related to the intended change in behavior [1].
>
> There was then a follow-up in 2fb129b5c (`org-scan-tags' retrieve all
> TODO keywords, not only not-done ones, 2017-08-17).  As far as I can
> tell, that was supposed to achieve the behavior you're after but didn't.
> I don't have time to dig much at the moment, but quickly stepping
> through org-scan-tags, I think the issue is that the MATCHER argument
> still filters out done states.
>
>
> [0] https://orgmode.org/list/CAF96XX0XxhPKjAxy0dQmOiy3rnT+DUok4p1y71F1AwyJaNL-MA@mail.gmail.com/
> [1] https://orgmode.org/list/874lt89fi2.fsf@free.fr/

Thank you for looking into this. I'm going to try to come up with some
tests for the behaviour, and with the help of your references see if I
can get those tests to pass.

Stig

Re: tags-todo org-agenda-custom-command weirdness

[Bastien]

Hi Stig,

Stig Brautaset <stig@brautaset.org> writes:

> Thank you for looking into this. I'm going to try to come up with some
> tests for the behaviour, and with the help of your references see if I
> can get those tests to pass.

did you manage to find the time to look into this?

This fix will potentially change the behavior of many agenda views, so
we'd better advertize it in a major release.  If not 9.4, a later one,
there really no hurry.

Thanks,

-- 
 Bastien

Re: tags-todo org-agenda-custom-command weirdness

[Stig Brautaset]

Bastien <bzg@gnu.org> writes:
> Stig Brautaset <stig@brautaset.org> writes:
>> Thank you for looking into this. I'm going to try to come up with some
>> tests for the behaviour, and with the help of your references see if I
>> can get those tests to pass.
>
> did you manage to find the time to look into this?

I didn't manage to make the time, no. I still may, but if someone else
beats me to it I would certainly welcome it! I'm finding it hard to set
aside time to dig into it given I have a functioning workaround.

Stig

Re: tags-todo org-agenda-custom-command weirdness

[Bastien]

Stig Brautaset <stig@brautaset.org> writes:

> I didn't manage to make the time, no. I still may, but if someone else
> beats me to it I would certainly welcome it! I'm finding it hard to set
> aside time to dig into it given I have a functioning workaround.

Quite understandable, no problem.

-- 
 Bastien

tags-todo agenda shoud not ignore DONE items (was: tags-todo org-agenda-custom-command weirdness)

[Bastien]

Kyle Meyer <kyle@kyleam.com> writes:

>> This behaviour of `tags-todo` seems inconsistent to me. If `todo` can
>> find DONE items, why shouldn't `tags-todo` do the same?
>
> Perhaps it should.  The behavior has been that way for a long time, and
> the code makes it look very deliberate.  That of course is not an
> argument that it should be that way, just one basis for expectations
> (and a reason to be wary of breaking workflows).
>
> And it looks like it was actually supposed to change to your preference
> in 2017.  There was a report [0] that essentially boils down to what
> you're saying, I think.  In response, 942b6267a (org-agenda: `tags-todo'
> command type includes DONE keywords, 2017-04-18) was applied, but then
> reverted for reasons not related to the intended change in behavior [1].
>
> There was then a follow-up in 2fb129b5c (`org-scan-tags' retrieve all
> TODO keywords, not only not-done ones, 2017-08-17).  As far as I can
> tell, that was supposed to achieve the behavior you're after but didn't.
> I don't have time to dig much at the moment, but quickly stepping
> through org-scan-tags, I think the issue is that the MATCHER argument
> still filters out done states.
>
>
> [0] https://orgmode.org/list/CAF96XX0XxhPKjAxy0dQmOiy3rnT+DUok4p1y71F1AwyJaNL-MA@mail.gmail.com/
> [1] https://orgmode.org/list/874lt89fi2.fsf@free.fr/

Confirming this as an issue, if someone wants to fix it.

-- 
 Bastien

Re: tags-todo agenda shoud not ignore DONE items

[Bastien]

Bastien <bzg@gnu.org> writes:

> Confirming this as an issue, if someone wants to fix it.

This should be fixed now with 823f9744e in maint, tags-todo should now
include DONE headings.

-- 
 Bastien

Re: tags-todo agenda shoud not ignore DONE items

[Kyle Meyer]

Bastien writes:

> Bastien <bzg@gnu.org> writes:
>
>> Confirming this as an issue, if someone wants to fix it.
>
> This should be fixed now with 823f9744e in maint, tags-todo should now
> include DONE headings.

823f9744e looks like a regression because it removes the distinction
between `tags' and `tags-todo'.  Consider the following file

--8<---------------cut here---------------start------------->8---
* h1                                                                   :atag:
* TODO h2                                                              :atag:
* DONE h3                                                              :atag:
* h4
* TODO h5
* DONE h6
--8<---------------cut here---------------end--------------->8---

and the following configuration

  (setq org-agenda-custom-commands
        '(("1" tags "atag")
          ("2" tags-todo "atag")))

Before the above commit, 1 should show

  scratch:    h1                                                                          :atag:
  scratch:    TODO h2                                                                     :atag:
  scratch:    DONE h3                                                                     :atag:

and 2

  scratch:    TODO h2                                                                     :atag:

That matches my expectations, though the request in this thread is that
2 includes "DONE h3" as well.

With 823f9744e, both 1 and 2 show

  scratch:    h1                                                                          :atag:
  scratch:    TODO h2                                                                     :atag:
  scratch:    DONE h3                                                                     :atag:

Note the inclusion of a non-TODO entry for 2 (tags-todo).

---

James Cash sent a followup patch to this in a detached thread:

  https://orgmode.org/list/87tuvyaopv.fsf@gmail.com

As I mentioned in that thread (<87361d7p3q.fsf@kyleam.com>) and
suggested in this thread (<87d061auiw.fsf@kyleam.com>), I think tags and
tags-todo should stay aligned with agenda's m/M, with tags-todo
excluding DONE items just as M does.

Re: tags-todo agenda shoud not ignore DONE items

[Bastien]

Kyle Meyer <kyle@kyleam.com> writes:

> 823f9744e looks like a regression because it removes the distinction
> between `tags' and `tags-todo'.

I reverted this commit.

> James Cash sent a followup patch to this in a detached thread:
>
>   https://orgmode.org/list/87tuvyaopv.fsf@gmail.com
>
> As I mentioned in that thread (<87361d7p3q.fsf@kyleam.com>) and
> suggested in this thread (<87d061auiw.fsf@kyleam.com>), I think tags and
> tags-todo should stay aligned with agenda's m/M, with tags-todo
> excluding DONE items just as M does.

Sorry I missed these replies, I see now.  If you think there are
places in the documentation that needs some fixing or clarification
please go ahead as you see fit.

Thanks,

-- 
 Bastien

Re: tags-todo agenda shoud not ignore DONE items

[James N V Cash]

[-- Attachment #1: Type: text/plain, Size: 380 bytes --]


Bastien <bzg@gnu.org> writes:

> Confirming this as an issue, if someone wants to fix it.

I've tried taking a shot at addressing this issue via the patch attached.

As mentioned in the thread, might potentially cause issues by changing how it matches -- should there be an option? Or is the idea just that if you really do want to exclude done tags, you'd query like "/!DONE"?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: matcher allow all tags --]
[-- Type: text/x-diff, Size: 551 bytes --]

diff --git a/lisp/org.el b/lisp/org.el
index 020cfc873..3db1add8b 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -11691,7 +11691,7 @@ See also `org-scan-tags'."
 		       `(and ,tagsmatcher ,todomatcher)
 		     (or tagsmatcher todomatcher t))))
       (when org--matcher-tags-todo-only
-	(setq matcher `(and (member todo org-not-done-keywords) ,matcher)))
+	(setq matcher `(and (member todo org-todo-keywords-1) ,matcher)))
       (cons match0 `(lambda (todo tags-list level) ,matcher)))))
 
 (defun org--tags-expand-group (group tag-groups expanded)

Re: tags-todo agenda shoud not ignore DONE items

[Kyle Meyer]

[ Sorry for the slow reply here, and thank you for looking into this. ]

James N V Cash writes:

> Bastien <bzg@gnu.org> writes:
>
>> Confirming this as an issue, if someone wants to fix it.
>
> I've tried taking a shot at addressing this issue via the patch attached.
>
> As mentioned in the thread, might potentially cause issues by changing
> how it matches -- should there be an option? Or is the idea just that
> if you really do want to exclude done tags, you'd query like "/!DONE"?

Yep, that's the idea, I think.  It's mentioned in both 942b6267a
(org-agenda: `tags-todo' command type includes DONE keywords,
2017-04-18) and 2fb129b5c (`org-scan-tags' retrieve all TODO keywords,
not only not-done ones, 2017-08-17).

>
> diff --git a/lisp/org.el b/lisp/org.el
> index 020cfc873..3db1add8b 100644
> --- a/lisp/org.el
> +++ b/lisp/org.el
> @@ -11691,7 +11691,7 @@ See also `org-scan-tags'."
>  		       `(and ,tagsmatcher ,todomatcher)
>  		     (or tagsmatcher todomatcher t))))
>        (when org--matcher-tags-todo-only
> -	(setq matcher `(and (member todo org-not-done-keywords) ,matcher)))
> +	(setq matcher `(and (member todo org-todo-keywords-1) ,matcher)))
>        (cons match0 `(lambda (todo tags-list level) ,matcher)))))
>  
>  (defun org--tags-expand-group (group tag-groups expanded)

This does resolve the reported issue with the b tags-todo set reported
in <https://orgmode.org/list/m2ftazm3l1.fsf@brautaset.org>.  However, it
breaks the org-tags-view todo-only operation (agenda's M): it no longer
filters out todos in a DONE state.  This behavior is explicitly
specified in the manual:

  Using ‘M-x org-agenda M’ or ‘/!’ does not match TODO keywords in a
  DONE state.

In <https://orgmode.org/list/87d061auiw.fsf@kyleam.com>, I asked,
"shouldn't tags-todo consider only none-DONE todo items, in the same
spirit as m versus M?"  And, looking at this again, it's not only in
spirit; the tags-todo agenda view and org-tags-view's todo-only are
really directly linked:

   ((eq type 'tags-todo)
    (org-let lprops '(org-tags-view '(4) org-match)))

After rereading the thread at m2ftazm3l1.fsf@brautaset.org, I'm still
not convinced we should break that link.

Re: tags-todo agenda shoud not ignore DONE items

[Bastien]

Kyle Meyer <kyle@kyleam.com> writes:

>> As mentioned in the thread, might potentially cause issues by changing
>> how it matches -- should there be an option? Or is the idea just that
>> if you really do want to exclude done tags, you'd query like "/!DONE"?
>
> Yep, that's the idea, I think.

Closing this issue right now -- feel free to report again if if still
needs to be resolved.