org-agenda queries for absent properties

org-agenda queries for absent properties

[Jens Schmidt]

The following was initially meant as bug report until I understood that
it's actually some sort of feature ...

Consider the following todo list:

-------------------- todos.org --------------------
* TODO relax

* TODO do this
:PROPERTIES:
:patchday: 202302
:END:

* TODO do that
:PROPERTIES:
:patchday: 202305
:END:

* TODO work harder
:PROPERTIES:
:patchday: 202308
:END:
-------------------- todos.org --------------------

When I use an agenda query

   patchday>=202305

on that I get the result:

   Headlines with TAGS match: patchday>=202305
   Press ‘C-u r’ to search again
     todos:      TODO do that
     todos:      TODO work harder

However for the following:

   patchday<=202305

I get

   Headlines with TAGS match: patchday<=202305
   Press ‘C-u r’ to search again
     todos:      TODO relax
     todos:      TODO do this
     todos:      TODO do that

since the absent property "patchday" on the "relax" todo entry is
defaulted to value zero, obviously.

So I would like to have something easier to type (and remember! why not
"!=" or "/="?) then

   patchday<>0&patchday<=202305

How about starred agenda property operators that match only if the
operand property is actually present, which would result in this query:

   patchday<=*202305

Section <info:org#Matching tags and properties> does not seem to provide
any information on "definedness" expressions for properties.

Thanks.

Re: org-agenda queries for absent properties

[Ihor Radchenko]

Jens Schmidt <jschmidt4gnu@vodafonemail.de> writes:

> The following was initially meant as bug report until I understood that
> it's actually some sort of feature ...
>
> since the absent property "patchday" on the "relax" todo entry is
> defaulted to value zero, obviously.

This is a side effect of handling comparison operators in tag matcher.
The property value is processed as (string-to-number (or value "")). And
since (string-to-number "") => 0, you get what you get.

This code is out there since the beginning of git history.

> So I would like to have something easier to type (and remember! why not
> "!=" or "/="?) then

Not "!=" and "/=" is because parsing of match string is too dumb. See
`org-make-tags-matcher'.

>    patchday<>0&patchday<=202305
>
> How about starred agenda property operators that match only if the
> operand property is actually present, which would result in this query:
>
>    patchday<=*202305
>
> Section <info:org#Matching tags and properties> does not seem to provide
> any information on "definedness" expressions for properties.

If you are up to a job of adding this to `org-make-tags-matcher' and not
breaking things, you can try. But we should really provide more readable
match syntax, similar to org-ql; sooner or later.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-agenda queries for absent properties

[Jens Schmidt]

On 2023-07-31  08:45, Ihor Radchenko wrote:

> If you are up to a job of adding this to `org-make-tags-matcher' and
> not breaking things, you can try.

Depends a bit on the required collateral changes: Updating ORG-NEWS and
doc/org-guide.org should be no problem.  Doing a naive grep-based search
I haven't found tests on `org-tags-view' or `org-make-tags-matcher', so
I hopefully wouldn't need to add anything new in that direction.

Finally, the first step probably would be to make that humongous regexp
in `org-make-tags-matcher' a bit more readable.  What would you prefer?
`rx' or sth long the following lines (not checked for equality yet):

   (re (concat
        ;; AND operator (OR is done by global splitting)
        "^&?"
        ;; 1: exclusion and inclusion (the latter being
        ;; implicit)
        "\\([-+:]\\)?"
        ;; 2: query expression
        "\\("
	   ;; regular expression matching tag
	   "{[^}]+}\\|"
	   ;; LEVEL special property match (3: op, 4: level
	   ;; value)
	   "LEVEL\\([<=>]\\{1,2\\}\\)\\([0-9]+\\)\\|"
	   ;; property match
	   "\\(?:"
	       ;; 5: property name (with backslash escaping minus)
	       "\\(\\(?:[[:alnum:]_]+\\(?:\\\\-\\)*\\)+\\)"
	       ;; 6: property operator
	       "\\([<>=]\\{1,2\\}\\)"
	       ;; 7: property value/operand
	       "\\("
		   ;; regular expression
		   "{[^}]+}\\|"
		   ;; string
		   "\"[^\"]*\"\\|"
		   ;; number
		   "-?[.0-9]+\\(?:[eE][-+]?[0-9]+\\)?"
	       "\\)"
	   "\\)\\|"
	   ;; tag match
	   org-tag-re
        "\\)"))

 > But we should really provide more
 > readable match syntax, similar to org-ql; sooner or later.

Agreed on that if the new syntax will allow for equally succinct
queries.  Plus a very cursory, doc-only review of org-ql hasn't
revealed to me how I'd do numeric property comparisons with it.

Re: org-agenda queries for absent properties

[Ihor Radchenko]

Jens Schmidt <jschmidt4gnu@vodafonemail.de> writes:

> On 2023-07-31  08:45, Ihor Radchenko wrote:
>
>> If you are up to a job of adding this to `org-make-tags-matcher' and
>> not breaking things, you can try.
>
> Depends a bit on the required collateral changes: Updating ORG-NEWS and
> doc/org-guide.org should be no problem.  Doing a naive grep-based search
> I haven't found tests on `org-tags-view' or `org-make-tags-matcher', so
> I hopefully wouldn't need to add anything new in that direction.

There are some tests in testing/lisp/test-org.el:test-org/map-entries
If we add new feature, we should better add test coverage for it as
well. Should not be hard.

> Finally, the first step probably would be to make that humongous regexp
> in `org-make-tags-matcher' a bit more readable.

Agree.

> ... What would you prefer?
> `rx' or sth long the following lines (not checked for equality yet):
>    (re (concat
>         ;; AND operator (OR is done by global splitting)
> ...

`rx' would be great.
But even adding comments like in your example would be an improvement.

>  > But we should really provide more
>  > readable match syntax, similar to org-ql; sooner or later.
>
> Agreed on that if the new syntax will allow for equally succinct
> queries.  Plus a very cursory, doc-only review of org-ql hasn't
> revealed to me how I'd do numeric property comparisons with it.

Similar does not mean the same :)
I am thinking about something like
https://github.com/alphapapa/org-ql#non-sexp-query-syntax

For example, org-ql has

(src :lang "elisp" :regexps ("defun"))
and
src:defun,lang=elisp

We may as well allow >,<,... in addition to =.

I had this in mind for a wile, but I am still hoping that we can
eventually (when it is added to Emacs) rely upon peg.el for parsing.
https://yhetil.org/emacs-devel/875yvtbbn3.fsf@ericabrahamsen.net/

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-agenda queries for absent properties

[Jens Schmidt]

[-- Attachment #1: Type: text/plain, Size: 3181 bytes --]

On 2023-08-02  08:45, Ihor Radchenko wrote:

 > `rx' would be great.
 > But even adding comments like in your example would be an improvement.

Since the future of this code snippet seems to be uncertain I went for
comments only.

And I thought I was pretty much done when I noticed at least one major
issues in the existing code, so I decided to go with a prerelease first
plus some notes and questions.

So there will be a follow up to the attached patch, and I leave it to
you whether you give it already a review or not.  But I'd ask you for
your opinion on the following notes, where the first few should be
uncritical:

- I used "\(?NUM: ... \)" constructs to explicitly number the subres.
   Hope this is OK w.r.t. style and backward-compatibility.

- I fixed the operator-matching subre to also include `==', `!=', `/='
   but exclude `<<' and the like which currently give void-function
   errors.

- I did not fix some "a[^b]*b"-style subres to use non-greedy variants
   since these are strictly speaking not identical.  Even though newline
   characters shouldn't play a big role here ...

- I likewise did not fix the number-matching subre allowing for numbers
   like "1.2.3" to keep things short at least there.  `string-to-number'
   silently takes care of these, even if an exponent gets lost that way.

But from here it gets more intersting:

- The code uses subre "\\\\-" in property names to (supposedly) allow
   for inclusion of minus characters in property names, which (probably)
   could be confused with term negation.

- It also unquotes these minus characters for {tag regexps}:

     (tag (save-match-data
            (replace-regexp-in-string
             "\\\\-" "-" (match-string 2 term))))

   But it never unquotes them in property names.  That missing unquoting
   could be easily amended, but:

- The other issue is: Why do we need "\\\\-" for both property names and
   {tag regexps}?  This forces us to do queries like:

     {[a\\-z]}|foo\\-bar="baz"

   where in my opinion

     {[a\-z]}|foo\-bar="baz"

   should be sufficient.

- Even more, IMO one could do away completely with the minus-quoting and
   unquoting, since the overall regexp should allow for unambiguously
   matching minus characters both

   + in {tag regexps} (because of "{[^}]+}" gobbling them) and

   + in property names (because a property name must always be followed
     by some operator)

   *without* them getting confused with term negation.

   Or do I miss something here?  A cursory test with sth like

     +foo-bar="xxx"-patchday=202302

   seems to work fine.

- However, removing the unquoting of {tag regexps} would be a breaking
   change.  Even though I doubt anybody has ever used it, the more it is
   not mentioned in the documentation.

 > I had this in mind for a wile, but I am still hoping that we can
 > eventually (when it is added to Emacs) rely upon peg.el for parsing.

Given the fact that we have to discuss issues like those above, I
heartily agree.

 > https://yhetil.org/emacs-devel/875yvtbbn3.fsf@ericabrahamsen.net/

Arthouse thread: Interesting plot, surprising sidelines, not everything
comprehensible, (unfortunately) open end.

[-- Attachment #2: 0001-org-make-tags-matcher-Add-starred-property-operators.patch --]
[-- Type: text/x-patch, Size: 11088 bytes --]

From 1765e91d2f7875b321703afe34e32754a022bef4 Mon Sep 17 00:00:00 2001
From: Jens Schmidt <jschmidt4gnu@vodafonemail.de>
Date: Thu, 3 Aug 2023 22:34:56 +0200
Subject: [PATCH] org-make-tags-matcher: Add starred property operators, more
 operator synonyms

* lisp/org.el (org-make-tags-matcher): Add starred property operators.
Recognize additional operators "==", "!=", "/=".  Clean up and
document match term parsing.
(org-op-to-function): Recognize additional inequality operator "/=".

* doc/org-manual.org (Matching tags and properties):
* etc/ORG-NEWS: (~org-tags-view~ supports more property operators):
* testing/lisp/test-org.el (test-org/map-entries): Add documentation,
announcement, and tests on starred and additional operators.
---
 doc/org-manual.org       | 20 ++++++++-
 etc/ORG-NEWS             | 10 ++++-
 lisp/org.el              | 96 +++++++++++++++++++++++++++++-----------
 testing/lisp/test-org.el | 33 ++++++++++++++
 4 files changed, 130 insertions(+), 29 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 16fbb268f..27051fbfc 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -9246,16 +9246,25 @@ When matching properties, a number of different operators can be used
 to test the value of a property.  Here is a complex example:
 
 #+begin_example
-+work-boss+PRIORITY="A"+Coffee="unlimited"+Effort<2
++work-boss+PRIORITY="A"+Coffee="unlimited"+Effort<*2
          +With={Sarah\|Denny}+SCHEDULED>="<2008-10-11>"
 #+end_example
 
+#+cindex: operator, for property search
 #+texinfo: @noindent
 The type of comparison depends on how the comparison value is written:
 
 - If the comparison value is a plain number, a numerical comparison is
   done, and the allowed operators are =<=, ===, =>=, =<==, =>==, and
-  =<>=.
+  =<>=.  As synonym for the equality operator there is also ====, as
+  synonyms for the inequality operator there are =!== and =/==.
+
+- All operators may be optionally followed by an asterisk =*=, like in
+  =<*=, =!=*=, etc.  Such /starred operators/ work like their regular,
+  unstarred counterparts except that they match only headlines where
+  the tested property is actually present.  This is most useful for
+  search terms that logically exclude results, like the inequality
+  operator.
 
 - If the comparison value is enclosed in double-quotes, a string
   comparison is done, and the same operators are allowed.
@@ -9280,6 +9289,13 @@ smaller than 2, a =With= property that is matched by the regular
 expression =Sarah\|Denny=, and that are scheduled on or after October
 11, 2008.
 
+Note that the test on the =EFFORT= property uses operator =<*=, so
+that the search result will include only entries that actually have an
+=EFFORT= property defined and with numerical value smaller than 2.
+With the regular =<= operator, the search would handle entries without
+an =EFFORT= property as having a zero effort and would include them in
+the result as well.
+
 You can configure Org mode to use property inheritance during
 a search, but beware that this can slow down searches considerably.
 See [[*Property Inheritance]], for details.
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 4f16eda24..10c51e354 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -125,7 +125,7 @@ New functions to retrieve and set (via ~setf~) commonly used element properties:
 - =:contents-post-affiliated= :: ~org-element-post-affiliated~
 - =:contents-post-blank= :: ~org-element-post-blank~
 - =:parent= :: ~org-element-parent~
- 
+
 ***** New macro ~org-element-with-enabled-cache~
 
 The macro arranges the element cache to be active during =BODY= execution.
@@ -558,6 +558,14 @@ special repeaters ~++~ and ~.+~ are skipped.
 A capture template can target ~(here)~ which is the equivalent of
 invoking a capture template with a zero prefix.
 
+*** ~org-tags-view~ supports more property operators
+
+It supports inequality operators ~!=~ and ~/=~ in addition to the less
+common (BASIC?  Pascal?  SQL?) ~<>~.  And it supports starred versions
+of all relational operators (~<*~, ~=*~, ~!=*~, etc.) that work like
+the regular, unstarred operators but match a headline only if the
+tested property is actually present.
+
 ** New functions and changes in function arguments
 *** =TYPES= argument in ~org-element-lineage~ can now be a symbol
 
diff --git a/lisp/org.el b/lisp/org.el
index 1ac912e61..a1f4c1c53 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -11306,11 +11306,33 @@ See also `org-scan-tags'."
 
   (let ((match0 match)
 	(re (concat
-	     "^&?\\([-+:]\\)?\\({[^}]+}\\|LEVEL\\([<=>]\\{1,2\\}\\)"
-	     "\\([0-9]+\\)\\|\\(\\(?:[[:alnum:]_]+\\(?:\\\\-\\)*\\)+\\)"
-	     "\\([<>=]\\{1,2\\}\\)"
-	     "\\({[^}]+}\\|\"[^\"]*\"\\|-?[.0-9]+\\(?:[eE][-+]?[0-9]+\\)?\\)"
-	     "\\|" org-tag-re "\\)"))
+	     "^"
+	     ;; implicit AND operator (OR is done by global splitting)
+	     "&?"
+	     ;; exclusion and inclusion (the latter being implicit)
+	     "\\(?1:[-+:]\\)?"
+	     ;; query term
+	     "\\(?2:"
+	         ;; tag regexp match
+	         "{[^}]+}\\|"
+		 ;; LEVEL property match
+		 "LEVEL\\(?3:[<=>]=?\\|[!/]=\\|<>\\)\\(?4:[0-9]+\\)\\|"
+		 ;; regular property match
+		 "\\(?:"
+		     ;; property name
+		     "\\(?5:\\(?:[[:alnum:]_]+\\(?:\\\\-\\)*\\)+\\)"
+		     ;; operator, optionally starred
+		     "\\(?6:[<=>]=?\\|[!/]=\\|<>\\)\\(?7:\\*\\)?"
+		     ;; operand
+		     "\\(?8:"
+		         "{[^}]+}\\|"
+			 "\"[^\"]*\"\\|"
+			 "-?[.0-9]+\\(?:[eE][-+]?[0-9]+\\)?"
+		     "\\)"
+		 "\\)\\|"
+		 ;; exact tag match
+		 org-tag-re
+	     "\\)"))
 	(start 0)
 	tagsmatch todomatch tagsmatcher todomatcher)
 
@@ -11352,6 +11374,11 @@ See also `org-scan-tags'."
 	    (let* ((rest (substring term (match-end 0)))
 		   (minus (and (match-end 1)
 			       (equal (match-string 1 term) "-")))
+		   ;; Bind the whole term to `tag' and use that
+		   ;; variable for a tag regexp match in (1) or as an
+		   ;; exact tag match in (2).  Unquote quoted minus
+		   ;; characters, which would be actually required
+		   ;; only for the former case.
 		   (tag (save-match-data
 			  (replace-regexp-in-string
 			   "\\\\-" "-" (match-string 2 term))))
@@ -11360,7 +11387,7 @@ See also `org-scan-tags'."
 		   (propp (match-end 5))
 		   (mm
 		    (cond
-		     (regexp
+		     (regexp			; (1)
                       `(with-syntax-table org-mode-tags-syntax-table
                          (org-match-any-p ,(substring tag 1 -1) tags-list)))
 		     (levelp
@@ -11368,28 +11395,45 @@ See also `org-scan-tags'."
 			level
 			,(string-to-number (match-string 4 term))))
 		     (propp
-		      (let* ((gv (pcase (upcase (match-string 5 term))
+		      (let* (;; Convert property name to an Elisp
+			     ;; accessor for that property.
+			     (gv (pcase (upcase (match-string 5 term))
 				   ("CATEGORY"
 				    '(org-get-category (point)))
 				   ("TODO" 'todo)
 				   (p `(org-cached-entry-get nil ,p))))
-			     (pv (match-string 7 term))
+			     ;; Determine operand (aka. property
+			     ;; value)
+			     (pv (match-string 8 term))
+			     ;; Determine type of operand.  Note that
+			     ;; these are not exclusive: Any TIMEP is
+			     ;; also STRP.
 			     (regexp (eq (string-to-char pv) ?{))
 			     (strp (eq (string-to-char pv) ?\"))
 			     (timep (string-match-p "^\"[[<]\\(?:[0-9]+\\|now\\|today\\|tomorrow\\|[+-][0-9]+[dmwy]\\).*[]>]\"$" pv))
+			     ;; Massage operand.  TIMEP must come
+			     ;; before STRP.
+			     (pv (cond (regexp (substring pv 1 -1))
+				       (timep  (org-matcher-time
+						(substring pv 1 -1)))
+				       (strp   (substring pv 1 -1))
+				       (t      pv)))
+			     ;; Convert operator to Elisp.
 			     (po (org-op-to-function (match-string 6 term)
-						     (if timep 'time strp))))
-			(setq pv (if (or regexp strp) (substring pv 1 -1) pv))
-			(when timep (setq pv (org-matcher-time pv)))
-			(cond ((and regexp (eq po '/=))
-			       `(not (string-match ,pv (or ,gv ""))))
-			      (regexp `(string-match ,pv (or ,gv "")))
-			      (strp `(,po (or ,gv "") ,pv))
-			      (t
-			       `(,po
-				 (string-to-number (or ,gv ""))
-				 ,(string-to-number pv))))))
-		     (t `(member ,tag tags-list)))))
+						     (if timep 'time strp)))
+			     ;; Convert whole term to Elisp.
+			     (pt (cond ((and regexp (eq po '/=))
+					`(not (string-match ,pv (or ,gv ""))))
+				       (regexp `(string-match ,pv (or ,gv "")))
+				       (strp `(,po (or ,gv "") ,pv))
+				       (t
+					`(,po
+					  (string-to-number (or ,gv ""))
+					  ,(string-to-number pv)))))
+			     ;; Respect the star after the operand.
+			     (pt (if (match-end 7) `(and ,gv ,pt) pt)))
+			pt))
+		     (t `(member ,tag tags-list))))) ; (2)
 	      (push (if minus `(not ,mm) mm) tagsmatcher)
 	      (setq term rest)))
 	  (push `(and ,@tagsmatcher) orlist)
@@ -11520,12 +11564,12 @@ the list of tags in this group."
   "Turn an operator into the appropriate function."
   (setq op
 	(cond
-	 ((equal  op   "<"       ) '(<     org-string<  org-time<))
-	 ((equal  op   ">"       ) '(>     org-string>  org-time>))
-	 ((member op '("<=" "=<")) '(<=    org-string<= org-time<=))
-	 ((member op '(">=" "=>")) '(>=    org-string>= org-time>=))
-	 ((member op '("="  "==")) '(=     string=      org-time=))
-	 ((member op '("<>" "!=")) '(/=    org-string<> org-time<>))))
+	 ((equal  op   "<"            ) '(<     org-string<  org-time<))
+	 ((equal  op   ">"            ) '(>     org-string>  org-time>))
+	 ((member op '("<=" "=<"     )) '(<=    org-string<= org-time<=))
+	 ((member op '(">=" "=>"     )) '(>=    org-string>= org-time>=))
+	 ((member op '("="  "=="     )) '(=     string=      org-time=))
+	 ((member op '("<>" "!=" "/=")) '(/=    org-string<> org-time<>))))
   (nth (if (eq stringp 'time) 2 (if stringp 1 0)) op))
 
 (defvar org-add-colon-after-tag-completion nil)  ;; dynamically scoped param
diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el
index 890ea6a8c..ef52b95c7 100644
--- a/testing/lisp/test-org.el
+++ b/testing/lisp/test-org.el
@@ -2851,6 +2851,11 @@ test <point>
      (equal '(1)
 	    (org-test-with-temp-text "* [#A] H1\n* [#B] H2"
 	      (org-map-entries #'point "PRIORITY=\"A\""))))
+    ;; Negative priority match.
+    (should
+     (equal '(11)
+	    (org-test-with-temp-text "* [#A] H1\n* [#B] H2"
+	      (org-map-entries #'point "PRIORITY/=\"A\""))))
     ;; Date match.
     (should
      (equal '(36)
@@ -2881,6 +2886,34 @@ SCHEDULED: <2014-03-04 tue.>"
 :TEST: 2
 :END:"
 	      (org-map-entries #'point "TEST=1"))))
+    ;; Negative regular property match.
+    (should
+     (equal '(35 68)
+	    (org-test-with-temp-text "
+* H1
+:PROPERTIES:
+:TEST: 1
+:END:
+* H2
+:PROPERTIES:
+:TEST: 2
+:END:
+* H3"
+	      (org-map-entries #'point "TEST!=1"))))
+    ;; Starred negative regular property match.
+    (should
+     (equal '(35)
+	    (org-test-with-temp-text "
+* H1
+:PROPERTIES:
+:TEST: 1
+:END:
+* H2
+:PROPERTIES:
+:TEST: 2
+:END:
+* H3"
+	      (org-map-entries #'point "TEST!=*1"))))
     ;; Multiple criteria.
     (should
      (equal '(23)
-- 
2.30.2

Re: org-agenda queries for absent properties

[Ihor Radchenko]

Jens Schmidt <jschmidt4gnu@vodafonemail.de> writes:

> - I used "\(?NUM: ... \)" constructs to explicitly number the subres.
>    Hope this is OK w.r.t. style and backward-compatibility.

Yes, it is ok.

> - I fixed the operator-matching subre to also include `==', `!=', `/='
>    but exclude `<<' and the like which currently give void-function
>    errors.

Sounds reasonable.

> But from here it gets more intersting:
>
> - The code uses subre "\\\\-" in property names to (supposedly) allow
>    for inclusion of minus characters in property names, which (probably)
>    could be confused with term negation.

Not probably. It is known to be confused.
https://orgmode.org/list/87jzv67k3p.fsf@localhost

> - It also unquotes these minus characters for {tag regexps}:
>
>      (tag (save-match-data
>             (replace-regexp-in-string
>              "\\\\-" "-" (match-string 2 term))))
>
>    But it never unquotes them in property names.  That missing unquoting
>    could be easily amended, but:
>
> - The other issue is: Why do we need "\\\\-" for both property names and
>    {tag regexps}?  This forces us to do queries like:
>
>      {[a\\-z]}|foo\\-bar="baz"
>
>    where in my opinion
>
>      {[a\-z]}|foo\-bar="baz"
>
>    should be sufficient.

Ideally, we should have no need to quote "-" inside regexp terms.
The need to do it is not documented either.

> - Even more, IMO one could do away completely with the minus-quoting and
>    unquoting, since the overall regexp should allow for unambiguously
>    matching minus characters both
>
>    + in {tag regexps} (because of "{[^}]+}" gobbling them) and
>
>    + in property names (because a property name must always be followed
>      by some operator)
>
>    *without* them getting confused with term negation.
>
>    Or do I miss something here?  A cursory test with sth like
>
>      +foo-bar="xxx"-patchday=202302
>
>    seems to work fine.

Agree.

> - However, removing the unquoting of {tag regexps} would be a breaking
>    change.  Even though I doubt anybody has ever used it, the more it is
>    not mentioned in the documentation.

Unquoting in {tag regexps} was never intentional.
The commit that introduced this piece of code is the following:

19b0e03f32c6032a60150fc6cb07c6f766cb3f6c
Author:     Carsten Dominik <carsten.dominik@gmail.com>
Make backslash escape "-" in property matches

* lisp/org.el (org-make-tags-matcher): Read "\\-" as "-" in
the tags/property matcher.

Ilya Shlyakhter writes:

 When doing an agenda tags match for tags or properties with dashes in
 their name, the dashes become negation operators: "my-prop>0" means
 "entries that have the tag 'my' and do not have a positive property
 'prop'", rather than "entries that have a positive property
 'my-prop'".  Is there a way to escape the dashes to get the latter
 meaning?

> +		 ;; LEVEL property match
> +		 "LEVEL\\(?3:[<=>]=?\\|[!/]=\\|<>\\)\\(?4:[0-9]+\\)\\|"
> +		 ;; regular property match
> +		 "\\(?:"
> +		     ;; property name
> +		     "\\(?5:\\(?:[[:alnum:]_]+\\(?:\\\\-\\)*\\)+\\)"
> +		     ;; operator, optionally starred
> +		     "\\(?6:[<=>]=?\\|[!/]=\\|<>\\)\\(?7:\\*\\)?"

?3 and ?6 duplicate a part of regexp. It would make sense to let-bind
the common part to avoid accidental typos in future.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-agenda queries for absent properties

[Jens Schmidt]

On 2023-08-06  09:55, Ihor Radchenko wrote:

>> But from here it gets more intersting:
>>
>> - The code uses subre "\\\\-" in property names to (supposedly) allow
>>     for inclusion of minus characters in property names, which (probably)
>>     could be confused with term negation.
> 
> Not probably. It is known to be confused.
> https://orgmode.org/list/87jzv67k3p.fsf@localhost

Right.  But the issue back then appeared because the subre for property
names does not match minus characters, just alnums and underscores:

   \\([[:alnum:]_]+\\)

What I'm trying to convey is: If we match plain minus characters in
above subre, these will be greedily goobled up including any following
operator, which makes the whole thing uniquely an operator term.  If
there is no following operator, but something else, the whole regexp
backtracks and (hopefully) matches a following negative term.

>> - Even more, IMO one could do away completely with the minus-quoting and
>>     unquoting, since the overall regexp should allow for unambiguously
>>     matching minus characters both
>>
>>     + in {tag regexps} (because of "{[^}]+}" gobbling them) and
>>
>>     + in property names (because a property name must always be followed
>>       by some operator)
>>
>>     *without* them getting confused with term negation.
>>
>>     Or do I miss something here?  A cursory test with sth like
>>
>>       +foo-bar="xxx"-patchday=202302
>>
>>     seems to work fine.
> 
> Agree.

Ah, OK.  I thought you wouldn't.  Let me redo this thing as I think it
is correct (including let-binding common subres as you suggested).  I
will also add tests for matching property names having minuses and then
we can see how it goes.

Re: org-agenda queries for absent properties

[Jens Schmidt]

[-- Attachment #1: Type: text/plain, Size: 959 bytes --]

On 2023-08-06  11:19, Jens Schmidt wrote:

> I
> will also add tests for matching property names having minuses and then
> we can see how it goes.

Went well.  With a property name regexp

   "\\(?5:[[:alnum:]_-]+\\)"

and the following Org file

--------------------------------------------------
* H1 :BAR:
:PROPERTIES:
:TEST-FOO: 1
:END:
* H2 :FOO:
:PROPERTIES:
:TEST-FOO: 2
:END:
* H3 :BAR:
:PROPERTIES:
:-FOO: 1
:END:
* H4 :FOO:
:PROPERTIES:
:-FOO: 2
:END:
* H5
--------------------------------------------------

the search expressions "TEST-FOO!=*0-FOO" and "-FOO+TEST-FOO!=*0" work
as expected, finding only the first heading.

The expression "-FOO!=*0-FOO" does *not* work, though, since the leading
minus in the property name is gobbled by the greedy inclusion/exclusion
subexp "\\(?1:[-+:]\\)?".  However, the remedy is simple: Add an
explicit plus sign: "+-FOO!=*0-FOO".

Documented that in the Org manual, added tests, looks good.  Please
check.

[-- Attachment #2: 0001-org-make-tags-matcher-Add-starred-property-operators.patch --]
[-- Type: text/x-patch, Size: 15123 bytes --]

From cb3268c8b1c69371589a9aa44a96d33e29aedbb1 Mon Sep 17 00:00:00 2001
From: Jens Schmidt <jschmidt4gnu@vodafonemail.de>
Date: Sun, 6 Aug 2023 16:38:04 +0200
Subject: [PATCH] org-make-tags-matcher: Add starred property operators, fix
 quoting

* lisp/org.el (org-make-tags-matcher): Add starred property operators.
Recognize additional operators "==", "!=", "/=".  Clean up and
document match term parsing.  Remove needless and buggy unquoting of
minus characters in property and tag names.
(org-op-to-function): Recognize additional inequality operator "/=".

* doc/org-manual.org (Matching tags and properties): Add documentation
on starred and additional operators.  Document allowed characters in
property names and handling of minus characters in property names.

* testing/lisp/test-org.el (test-org/map-entries): Add tests for
starred and additional operators.  Add tests for property names
containing minus characters.

* etc/ORG-NEWS: (~org-tags-view~ supports more property operators):
Add announcement on starred and additional operators.

Link: https://orgmode.org/list/9132e58f-d89e-f7df-bbe4-43d53a2367d2@vodafonemail.de
---
 doc/org-manual.org       |  35 +++++++++++-
 etc/ORG-NEWS             |  10 +++-
 lisp/org.el              | 120 ++++++++++++++++++++++++++++-----------
 testing/lisp/test-org.el |  64 ++++++++++++++++++++-
 4 files changed, 192 insertions(+), 37 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index e59efc417..4252d7ac3 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -9246,16 +9246,18 @@ When matching properties, a number of different operators can be used
 to test the value of a property.  Here is a complex example:
 
 #+begin_example
-+work-boss+PRIORITY="A"+Coffee="unlimited"+Effort<2
++work-boss+PRIORITY="A"+Coffee="unlimited"+Effort<*2
          +With={Sarah\|Denny}+SCHEDULED>="<2008-10-11>"
 #+end_example
 
+#+cindex: operator, for property search
 #+texinfo: @noindent
 The type of comparison depends on how the comparison value is written:
 
 - If the comparison value is a plain number, a numerical comparison is
   done, and the allowed operators are =<=, ===, =>=, =<==, =>==, and
-  =<>=.
+  =<>=.  As synonym for the equality operator there is also ====, as
+  synonyms for the inequality operator there are =!== and =/==.
 
 - If the comparison value is enclosed in double-quotes, a string
   comparison is done, and the same operators are allowed.
@@ -9273,6 +9275,13 @@ The type of comparison depends on how the comparison value is written:
   is performed, with === meaning that the regexp matches the property
   value, and =<>= meaning that it does not match.
 
+- All operators may be optionally followed by an asterisk =*=, like in
+  =<*=, =!=*=, etc.  Such /starred operators/ work like their regular,
+  unstarred counterparts except that they match only headlines where
+  the tested property is actually present.  This is most useful for
+  search terms that logically exclude results, like the inequality
+  operator.
+
 So the search string in the example finds entries tagged =work= but
 not =boss=, which also have a priority value =A=, a =Coffee= property
 with the value =unlimited=, an =EFFORT= property that is numerically
@@ -9280,6 +9289,28 @@ smaller than 2, a =With= property that is matched by the regular
 expression =Sarah\|Denny=, and that are scheduled on or after October
 11, 2008.
 
+Note that the test on the =EFFORT= property uses operator =<*=, so
+that the search result will include only entries that actually have an
+=EFFORT= property defined and with numerical value smaller than 2.
+With the regular =<= operator, the search would handle entries without
+an =EFFORT= property as having a zero effort and would include them in
+the result as well.
+
+Currently, you can use only property names including alphanumeric
+characters, underscores, and minus characters in search strings.  In
+addition, if you want to search for a property whose name starts with
+a minus character, you have to "quote" that leading minus character
+with an explicit positive selection plus character, like this:
+
+#+begin_example
++-long-and-twisted-property-name-="foo"
+#+end_example
+
+#+texinfo: @noindent
+Without that extra plus character, the minus character would be taken
+to indicate a negative selection on search term
+=long-and-twisted-property-name-​="foo"=.
+
 You can configure Org mode to use property inheritance during
 a search, but beware that this can slow down searches considerably.
 See [[*Property Inheritance]], for details.
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 4f16eda24..10c51e354 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -125,7 +125,7 @@ New functions to retrieve and set (via ~setf~) commonly used element properties:
 - =:contents-post-affiliated= :: ~org-element-post-affiliated~
 - =:contents-post-blank= :: ~org-element-post-blank~
 - =:parent= :: ~org-element-parent~
- 
+
 ***** New macro ~org-element-with-enabled-cache~
 
 The macro arranges the element cache to be active during =BODY= execution.
@@ -558,6 +558,14 @@ special repeaters ~++~ and ~.+~ are skipped.
 A capture template can target ~(here)~ which is the equivalent of
 invoking a capture template with a zero prefix.
 
+*** ~org-tags-view~ supports more property operators
+
+It supports inequality operators ~!=~ and ~/=~ in addition to the less
+common (BASIC?  Pascal?  SQL?) ~<>~.  And it supports starred versions
+of all relational operators (~<*~, ~=*~, ~!=*~, etc.) that work like
+the regular, unstarred operators but match a headline only if the
+tested property is actually present.
+
 ** New functions and changes in function arguments
 *** =TYPES= argument in ~org-element-lineage~ can now be a symbol
 
diff --git a/lisp/org.el b/lisp/org.el
index ed75f3edb..9ee3f98ba 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -11304,15 +11304,50 @@ See also `org-scan-tags'."
 	     "Match: "
 	     'org-tags-completion-function nil nil nil 'org-tags-history))))
 
-  (let ((match0 match)
-	(re (concat
-	     "^&?\\([-+:]\\)?\\({[^}]+}\\|LEVEL\\([<=>]\\{1,2\\}\\)"
-	     "\\([0-9]+\\)\\|\\(\\(?:[[:alnum:]_]+\\(?:\\\\-\\)*\\)+\\)"
-	     "\\([<>=]\\{1,2\\}\\)"
-	     "\\({[^}]+}\\|\"[^\"]*\"\\|-?[.0-9]+\\(?:[eE][-+]?[0-9]+\\)?\\)"
-	     "\\|" org-tag-re "\\)"))
-	(start 0)
-	tagsmatch todomatch tagsmatcher todomatcher)
+  (let* ((match0 match)
+         (opre "[<=>]=?\\|[!/]=\\|<>")
+         (re (concat
+              "^"
+              ;; implicit AND operator (OR is done by global splitting)
+              "&?"
+              ;; exclusion and inclusion (the latter being implicit)
+              "\\(?1:[-+:]\\)?"
+              ;; query term
+              "\\(?2:"
+                  ;; tag regexp match
+                  "{[^}]+}\\|"
+                  ;; LEVEL property match.  For sake of consistency,
+                  ;; recognize starred operators here as well.  We do
+                  ;; not need to process them below, however, since
+                  ;; the LEVEL property is always present.
+                  "LEVEL\\(?3:" opre "\\)\\*?\\(?4:[0-9]+\\)\\|"
+                  ;; regular property match
+                  "\\(?:"
+                      ;; property name [1]
+                      "\\(?5:[[:alnum:]_-]+\\)"
+                      ;; operator, optionally starred
+                      "\\(?6:" opre "\\)\\(?7:\\*\\)?"
+                      ;; operand (regexp, double-quoted string,
+                      ;; number)
+                      "\\(?8:"
+                          "{[^}]+}\\|"
+                          "\"[^\"]*\"\\|"
+                          "-?[.0-9]+\\(?:[eE][-+]?[0-9]+\\)?"
+                      "\\)"
+                  "\\)\\|"
+                  ;; exact tag match
+                  org-tag-re
+              "\\)"))
+         (start 0)
+         tagsmatch todomatch tagsmatcher todomatcher)
+
+    ;; [1] The minus characters in property names do *not* conflict
+    ;; with the exclusion operator above, since the mandatory
+    ;; following operator distinguishes these both cases.
+    ;; Accordingly, minus characters do not need any special quoting,
+    ;; even if https://orgmode.org/list/87jzv67k3p.fsf@localhost and
+    ;; commit 19b0e03f32c6032a60150fc6cb07c6f766cb3f6c suggest
+    ;; otherwise.
 
     ;; Expand group tags.
     (setq match (org-tags-expand match))
@@ -11352,15 +11387,16 @@ See also `org-scan-tags'."
 	    (let* ((rest (substring term (match-end 0)))
 		   (minus (and (match-end 1)
 			       (equal (match-string 1 term) "-")))
-		   (tag (save-match-data
-			  (replace-regexp-in-string
-			   "\\\\-" "-" (match-string 2 term))))
+		   ;; Bind the whole query term to `tag' and use that
+		   ;; variable for a tag regexp match in [2] or as an
+		   ;; exact tag match in [3].
+		   (tag (match-string 2 term))
 		   (regexp (eq (string-to-char tag) ?{))
 		   (levelp (match-end 4))
 		   (propp (match-end 5))
 		   (mm
 		    (cond
-		     (regexp
+		     (regexp			; [2]
                       `(with-syntax-table org-mode-tags-syntax-table
                          (org-match-any-p ,(substring tag 1 -1) tags-list)))
 		     (levelp
@@ -11368,28 +11404,46 @@ See also `org-scan-tags'."
 			level
 			,(string-to-number (match-string 4 term))))
 		     (propp
-		      (let* ((gv (pcase (upcase (match-string 5 term))
+		      (let* (;; Convert property name to an Elisp
+			     ;; accessor for that property (aka. as
+			     ;; getter value?).
+			     (gv (pcase (upcase (match-string 5 term))
 				   ("CATEGORY"
 				    '(org-get-category (point)))
 				   ("TODO" 'todo)
 				   (p `(org-cached-entry-get nil ,p))))
-			     (pv (match-string 7 term))
+			     ;; Determine operand (aka. property
+			     ;; value).
+			     (pv (match-string 8 term))
+			     ;; Determine type of operand.  Note that
+			     ;; these are not exclusive: Any TIMEP is
+			     ;; also STRP.
 			     (regexp (eq (string-to-char pv) ?{))
 			     (strp (eq (string-to-char pv) ?\"))
 			     (timep (string-match-p "^\"[[<]\\(?:[0-9]+\\|now\\|today\\|tomorrow\\|[+-][0-9]+[dmwy]\\).*[]>]\"$" pv))
+			     ;; Massage operand.  TIMEP must come
+			     ;; before STRP.
+			     (pv (cond (regexp (substring pv 1 -1))
+				       (timep  (org-matcher-time
+						(substring pv 1 -1)))
+				       (strp   (substring pv 1 -1))
+				       (t      pv)))
+			     ;; Convert operator to Elisp.
 			     (po (org-op-to-function (match-string 6 term)
-						     (if timep 'time strp))))
-			(setq pv (if (or regexp strp) (substring pv 1 -1) pv))
-			(when timep (setq pv (org-matcher-time pv)))
-			(cond ((and regexp (eq po '/=))
-			       `(not (string-match ,pv (or ,gv ""))))
-			      (regexp `(string-match ,pv (or ,gv "")))
-			      (strp `(,po (or ,gv "") ,pv))
-			      (t
-			       `(,po
-				 (string-to-number (or ,gv ""))
-				 ,(string-to-number pv))))))
-		     (t `(member ,tag tags-list)))))
+						     (if timep 'time strp)))
+			     ;; Convert whole property term to Elisp.
+			     (pt (cond ((and regexp (eq po '/=))
+					`(not (string-match ,pv (or ,gv ""))))
+				       (regexp `(string-match ,pv (or ,gv "")))
+				       (strp `(,po (or ,gv "") ,pv))
+				       (t
+					`(,po
+					  (string-to-number (or ,gv ""))
+					  ,(string-to-number pv)))))
+			     ;; Respect the star after the operand.
+			     (pt (if (match-end 7) `(and ,gv ,pt) pt)))
+			pt))
+		     (t `(member ,tag tags-list))))) ; [3]
 	      (push (if minus `(not ,mm) mm) tagsmatcher)
 	      (setq term rest)))
 	  (push `(and ,@tagsmatcher) orlist)
@@ -11520,12 +11574,12 @@ the list of tags in this group."
   "Turn an operator into the appropriate function."
   (setq op
 	(cond
-	 ((equal  op   "<"       ) '(<     org-string<  org-time<))
-	 ((equal  op   ">"       ) '(>     org-string>  org-time>))
-	 ((member op '("<=" "=<")) '(<=    org-string<= org-time<=))
-	 ((member op '(">=" "=>")) '(>=    org-string>= org-time>=))
-	 ((member op '("="  "==")) '(=     string=      org-time=))
-	 ((member op '("<>" "!=")) '(/=    org-string<> org-time<>))))
+	 ((equal  op   "<"            ) '(<     org-string<  org-time<))
+	 ((equal  op   ">"            ) '(>     org-string>  org-time>))
+	 ((member op '("<=" "=<"     )) '(<=    org-string<= org-time<=))
+	 ((member op '(">=" "=>"     )) '(>=    org-string>= org-time>=))
+	 ((member op '("="  "=="     )) '(=     string=      org-time=))
+	 ((member op '("<>" "!=" "/=")) '(/=    org-string<> org-time<>))))
   (nth (if (eq stringp 'time) 2 (if stringp 1 0)) op))
 
 (defvar org-add-colon-after-tag-completion nil)  ;; dynamically scoped param
diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el
index 890ea6a8c..7c85da9d5 100644
--- a/testing/lisp/test-org.el
+++ b/testing/lisp/test-org.el
@@ -2833,6 +2833,11 @@ test <point>
      (equal '(11)
 	    (org-test-with-temp-text "* Level 1\n** Level 2"
 	      (let (org-odd-levels-only) (org-map-entries #'point "LEVEL>1")))))
+    ;; Level match with (ignored) starred operator.
+    (should
+     (equal '(11)
+	    (org-test-with-temp-text "* Level 1\n** Level 2"
+	      (let (org-odd-levels-only) (org-map-entries #'point "LEVEL>*1")))))
     ;; Tag match.
     (should
      (equal '(11)
@@ -2845,12 +2850,17 @@ test <point>
     (should
      (equal '(11 23)
 	    (org-test-with-temp-text "* H1 :no:\n* H2 :yes1:\n* H3 :yes2:"
-	      (org-map-entries #'point "{yes?}"))))
+	      (org-map-entries #'point "{yes.?}"))))
     ;; Priority match.
     (should
      (equal '(1)
 	    (org-test-with-temp-text "* [#A] H1\n* [#B] H2"
 	      (org-map-entries #'point "PRIORITY=\"A\""))))
+    ;; Negative priority match.
+    (should
+     (equal '(11)
+	    (org-test-with-temp-text "* [#A] H1\n* [#B] H2"
+	      (org-map-entries #'point "PRIORITY/=\"A\""))))
     ;; Date match.
     (should
      (equal '(36)
@@ -2881,6 +2891,58 @@ SCHEDULED: <2014-03-04 tue.>"
 :TEST: 2
 :END:"
 	      (org-map-entries #'point "TEST=1"))))
+    ;; Regular negative property match.
+    (should
+     (equal '(35 68)
+	    (org-test-with-temp-text "
+* H1
+:PROPERTIES:
+:TEST: 1
+:END:
+* H2
+:PROPERTIES:
+:TEST: 2
+:END:
+* H3"
+	      (org-map-entries #'point "TEST!=1"))))
+    ;; Starred negative property match.
+    (should
+     (equal '(35)
+	    (org-test-with-temp-text "
+* H1
+:PROPERTIES:
+:TEST: 1
+:END:
+* H2
+:PROPERTIES:
+:TEST: 2
+:END:
+* H3"
+	      (org-map-entries #'point "TEST!=*1"))))
+    ;; Property matches on names including minus characters.
+    (org-test-with-temp-text
+     "
+* H1 :BAR:
+:PROPERTIES:
+:TEST-FOO: 1
+:END:
+* H2 :FOO:
+:PROPERTIES:
+:TEST-FOO: 2
+:END:
+* H3 :BAR:
+:PROPERTIES:
+:-FOO: 1
+:END:
+* H4 :FOO:
+:PROPERTIES:
+:-FOO: 2
+:END:
+* H5"
+     (should (equal '(2) (org-map-entries #'point "TEST-FOO!=*0-FOO")))
+     (should (equal '(2) (org-map-entries #'point "-FOO+TEST-FOO!=*0")))
+     (should (equal '(88) (org-map-entries #'point "+-FOO!=*0-FOO")))
+     (should (equal '(88) (org-map-entries #'point "-FOO+-FOO!=*0"))))
     ;; Multiple criteria.
     (should
      (equal '(23)
-- 
2.30.2

Re: org-agenda queries for absent properties

[Ihor Radchenko]

Jens Schmidt <jschmidt4gnu@vodafonemail.de> writes:

> Documented that in the Org manual, added tests, looks good.  Please
> check.

Thanks!
Two minor comments.

>  - If the comparison value is a plain number, a numerical comparison is
>    done, and the allowed operators are =<=, ===, =>=, =<==, =>==, and
> -  =<>=.
> +  =<>=.  As synonym for the equality operator there is also ====, as
> +  synonyms for the inequality operator there are =!== and =/==.

I'd mention which operator is "equality" and "inequality" to avoid
ambiguity:

As a synonym for the equality operator ===, there is also ====; =!== and
=/== are synonyms of inequality operator =<>=.

> -		      (let* ((gv (pcase (upcase (match-string 5 term))
> +		      (let* (;; Convert property name to an Elisp
> +			     ;; accessor for that property (aka. as
> +			     ;; getter value?).

Is there any specific reason why you put question mark in the comment here?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: org-agenda queries for absent properties

[Jens Schmidt]

[-- Attachment #1: Type: text/plain, Size: 1153 bytes --]

On 2023-08-07  13:53, Ihor Radchenko wrote:

Thanks for reviewing.

>>   - If the comparison value is a plain number, a numerical comparison is
>>     done, and the allowed operators are =<=, ===, =>=, =<==, =>==, and
>> -  =<>=.
>> +  =<>=.  As synonym for the equality operator there is also ====, as
>> +  synonyms for the inequality operator there are =!== and =/==.
> 
> I'd mention which operator is "equality" and "inequality" to avoid
> ambiguity:
> 
> As a synonym for the equality operator ===, there is also ====; =!== and
> =/== are synonyms of inequality operator =<>=.

Done.

>> -		      (let* ((gv (pcase (upcase (match-string 5 term))
>> +		      (let* (;; Convert property name to an Elisp
>> +			     ;; accessor for that property (aka. as
>> +			     ;; getter value?).
 >
 > Is there any specific reason why you put question mark in the comment
 > here?

I was trying to decipher and document the acronyms used for the
let-bindings in that `let*' sexp.  I think I got them right but was
unsure about the "gv".  Removed the question mark, let's just assume
that "gv" means "getter value" and be done with it.

Updated patch attached.

[-- Attachment #2: 0001-org-make-tags-matcher-Add-starred-property-operators.patch --]
[-- Type: text/x-patch, Size: 15124 bytes --]

From 6e98356dfaf3466288398ff4ecee7fd147c32a20 Mon Sep 17 00:00:00 2001
From: Jens Schmidt <jschmidt4gnu@vodafonemail.de>
Date: Sun, 6 Aug 2023 16:38:04 +0200
Subject: [PATCH] org-make-tags-matcher: Add starred property operators, fix
 quoting

* lisp/org.el (org-make-tags-matcher): Add starred property operators.
Recognize additional operators "==", "!=", "/=".  Clean up and
document match term parsing.  Remove needless and buggy unquoting of
minus characters in property and tag names.
(org-op-to-function): Recognize additional inequality operator "/=".

* doc/org-manual.org (Matching tags and properties): Add documentation
on starred and additional operators.  Document allowed characters in
property names and handling of minus characters in property names.

* testing/lisp/test-org.el (test-org/map-entries): Add tests for
starred and additional operators.  Add tests for property names
containing minus characters.

* etc/ORG-NEWS: (~org-tags-view~ supports more property operators):
Add announcement on starred and additional operators.

Link: https://orgmode.org/list/9132e58f-d89e-f7df-bbe4-43d53a2367d2@vodafonemail.de
---
 doc/org-manual.org       |  35 +++++++++++-
 etc/ORG-NEWS             |  10 +++-
 lisp/org.el              | 120 ++++++++++++++++++++++++++++-----------
 testing/lisp/test-org.el |  64 ++++++++++++++++++++-
 4 files changed, 192 insertions(+), 37 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index e59efc417..dc9f552e5 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -9246,16 +9246,18 @@ When matching properties, a number of different operators can be used
 to test the value of a property.  Here is a complex example:
 
 #+begin_example
-+work-boss+PRIORITY="A"+Coffee="unlimited"+Effort<2
++work-boss+PRIORITY="A"+Coffee="unlimited"+Effort<*2
          +With={Sarah\|Denny}+SCHEDULED>="<2008-10-11>"
 #+end_example
 
+#+cindex: operator, for property search
 #+texinfo: @noindent
 The type of comparison depends on how the comparison value is written:
 
 - If the comparison value is a plain number, a numerical comparison is
   done, and the allowed operators are =<=, ===, =>=, =<==, =>==, and
-  =<>=.
+  =<>=.  As a synonym for the equality operator ===, there is also
+  ====; =!== and =/== are synonyms of the inequality operator =<>=.
 
 - If the comparison value is enclosed in double-quotes, a string
   comparison is done, and the same operators are allowed.
@@ -9273,6 +9275,13 @@ The type of comparison depends on how the comparison value is written:
   is performed, with === meaning that the regexp matches the property
   value, and =<>= meaning that it does not match.
 
+- All operators may be optionally followed by an asterisk =*=, like in
+  =<*=, =!=*=, etc.  Such /starred operators/ work like their regular,
+  unstarred counterparts except that they match only headlines where
+  the tested property is actually present.  This is most useful for
+  search terms that logically exclude results, like the inequality
+  operator.
+
 So the search string in the example finds entries tagged =work= but
 not =boss=, which also have a priority value =A=, a =Coffee= property
 with the value =unlimited=, an =EFFORT= property that is numerically
@@ -9280,6 +9289,28 @@ smaller than 2, a =With= property that is matched by the regular
 expression =Sarah\|Denny=, and that are scheduled on or after October
 11, 2008.
 
+Note that the test on the =EFFORT= property uses operator =<*=, so
+that the search result will include only entries that actually have an
+=EFFORT= property defined and with numerical value smaller than 2.
+With the regular =<= operator, the search would handle entries without
+an =EFFORT= property as having a zero effort and would include them in
+the result as well.
+
+Currently, you can use only property names including alphanumeric
+characters, underscores, and minus characters in search strings.  In
+addition, if you want to search for a property whose name starts with
+a minus character, you have to "quote" that leading minus character
+with an explicit positive selection plus character, like this:
+
+#+begin_example
++-long-and-twisted-property-name-="foo"
+#+end_example
+
+#+texinfo: @noindent
+Without that extra plus character, the minus character would be taken
+to indicate a negative selection on search term
+=long-and-twisted-property-name-​="foo"=.
+
 You can configure Org mode to use property inheritance during
 a search, but beware that this can slow down searches considerably.
 See [[*Property Inheritance]], for details.
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 4f16eda24..10c51e354 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -125,7 +125,7 @@ New functions to retrieve and set (via ~setf~) commonly used element properties:
 - =:contents-post-affiliated= :: ~org-element-post-affiliated~
 - =:contents-post-blank= :: ~org-element-post-blank~
 - =:parent= :: ~org-element-parent~
- 
+
 ***** New macro ~org-element-with-enabled-cache~
 
 The macro arranges the element cache to be active during =BODY= execution.
@@ -558,6 +558,14 @@ special repeaters ~++~ and ~.+~ are skipped.
 A capture template can target ~(here)~ which is the equivalent of
 invoking a capture template with a zero prefix.
 
+*** ~org-tags-view~ supports more property operators
+
+It supports inequality operators ~!=~ and ~/=~ in addition to the less
+common (BASIC?  Pascal?  SQL?) ~<>~.  And it supports starred versions
+of all relational operators (~<*~, ~=*~, ~!=*~, etc.) that work like
+the regular, unstarred operators but match a headline only if the
+tested property is actually present.
+
 ** New functions and changes in function arguments
 *** =TYPES= argument in ~org-element-lineage~ can now be a symbol
 
diff --git a/lisp/org.el b/lisp/org.el
index ed75f3edb..c037b3ee0 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -11304,15 +11304,50 @@ See also `org-scan-tags'."
 	     "Match: "
 	     'org-tags-completion-function nil nil nil 'org-tags-history))))
 
-  (let ((match0 match)
-	(re (concat
-	     "^&?\\([-+:]\\)?\\({[^}]+}\\|LEVEL\\([<=>]\\{1,2\\}\\)"
-	     "\\([0-9]+\\)\\|\\(\\(?:[[:alnum:]_]+\\(?:\\\\-\\)*\\)+\\)"
-	     "\\([<>=]\\{1,2\\}\\)"
-	     "\\({[^}]+}\\|\"[^\"]*\"\\|-?[.0-9]+\\(?:[eE][-+]?[0-9]+\\)?\\)"
-	     "\\|" org-tag-re "\\)"))
-	(start 0)
-	tagsmatch todomatch tagsmatcher todomatcher)
+  (let* ((match0 match)
+         (opre "[<=>]=?\\|[!/]=\\|<>")
+         (re (concat
+              "^"
+              ;; implicit AND operator (OR is done by global splitting)
+              "&?"
+              ;; exclusion and inclusion (the latter being implicit)
+              "\\(?1:[-+:]\\)?"
+              ;; query term
+              "\\(?2:"
+                  ;; tag regexp match
+                  "{[^}]+}\\|"
+                  ;; LEVEL property match.  For sake of consistency,
+                  ;; recognize starred operators here as well.  We do
+                  ;; not need to process them below, however, since
+                  ;; the LEVEL property is always present.
+                  "LEVEL\\(?3:" opre "\\)\\*?\\(?4:[0-9]+\\)\\|"
+                  ;; regular property match
+                  "\\(?:"
+                      ;; property name [1]
+                      "\\(?5:[[:alnum:]_-]+\\)"
+                      ;; operator, optionally starred
+                      "\\(?6:" opre "\\)\\(?7:\\*\\)?"
+                      ;; operand (regexp, double-quoted string,
+                      ;; number)
+                      "\\(?8:"
+                          "{[^}]+}\\|"
+                          "\"[^\"]*\"\\|"
+                          "-?[.0-9]+\\(?:[eE][-+]?[0-9]+\\)?"
+                      "\\)"
+                  "\\)\\|"
+                  ;; exact tag match
+                  org-tag-re
+              "\\)"))
+         (start 0)
+         tagsmatch todomatch tagsmatcher todomatcher)
+
+    ;; [1] The minus characters in property names do *not* conflict
+    ;; with the exclusion operator above, since the mandatory
+    ;; following operator distinguishes these both cases.
+    ;; Accordingly, minus characters do not need any special quoting,
+    ;; even if https://orgmode.org/list/87jzv67k3p.fsf@localhost and
+    ;; commit 19b0e03f32c6032a60150fc6cb07c6f766cb3f6c suggest
+    ;; otherwise.
 
     ;; Expand group tags.
     (setq match (org-tags-expand match))
@@ -11352,15 +11387,16 @@ See also `org-scan-tags'."
 	    (let* ((rest (substring term (match-end 0)))
 		   (minus (and (match-end 1)
 			       (equal (match-string 1 term) "-")))
-		   (tag (save-match-data
-			  (replace-regexp-in-string
-			   "\\\\-" "-" (match-string 2 term))))
+		   ;; Bind the whole query term to `tag' and use that
+		   ;; variable for a tag regexp match in [2] or as an
+		   ;; exact tag match in [3].
+		   (tag (match-string 2 term))
 		   (regexp (eq (string-to-char tag) ?{))
 		   (levelp (match-end 4))
 		   (propp (match-end 5))
 		   (mm
 		    (cond
-		     (regexp
+		     (regexp			; [2]
                       `(with-syntax-table org-mode-tags-syntax-table
                          (org-match-any-p ,(substring tag 1 -1) tags-list)))
 		     (levelp
@@ -11368,28 +11404,46 @@ See also `org-scan-tags'."
 			level
 			,(string-to-number (match-string 4 term))))
 		     (propp
-		      (let* ((gv (pcase (upcase (match-string 5 term))
+		      (let* (;; Convert property name to an Elisp
+			     ;; accessor for that property (aka. as
+			     ;; getter value).
+			     (gv (pcase (upcase (match-string 5 term))
 				   ("CATEGORY"
 				    '(org-get-category (point)))
 				   ("TODO" 'todo)
 				   (p `(org-cached-entry-get nil ,p))))
-			     (pv (match-string 7 term))
+			     ;; Determine operand (aka. property
+			     ;; value).
+			     (pv (match-string 8 term))
+			     ;; Determine type of operand.  Note that
+			     ;; these are not exclusive: Any TIMEP is
+			     ;; also STRP.
 			     (regexp (eq (string-to-char pv) ?{))
 			     (strp (eq (string-to-char pv) ?\"))
 			     (timep (string-match-p "^\"[[<]\\(?:[0-9]+\\|now\\|today\\|tomorrow\\|[+-][0-9]+[dmwy]\\).*[]>]\"$" pv))
+			     ;; Massage operand.  TIMEP must come
+			     ;; before STRP.
+			     (pv (cond (regexp (substring pv 1 -1))
+				       (timep  (org-matcher-time
+						(substring pv 1 -1)))
+				       (strp   (substring pv 1 -1))
+				       (t      pv)))
+			     ;; Convert operator to Elisp.
 			     (po (org-op-to-function (match-string 6 term)
-						     (if timep 'time strp))))
-			(setq pv (if (or regexp strp) (substring pv 1 -1) pv))
-			(when timep (setq pv (org-matcher-time pv)))
-			(cond ((and regexp (eq po '/=))
-			       `(not (string-match ,pv (or ,gv ""))))
-			      (regexp `(string-match ,pv (or ,gv "")))
-			      (strp `(,po (or ,gv "") ,pv))
-			      (t
-			       `(,po
-				 (string-to-number (or ,gv ""))
-				 ,(string-to-number pv))))))
-		     (t `(member ,tag tags-list)))))
+						     (if timep 'time strp)))
+			     ;; Convert whole property term to Elisp.
+			     (pt (cond ((and regexp (eq po '/=))
+					`(not (string-match ,pv (or ,gv ""))))
+				       (regexp `(string-match ,pv (or ,gv "")))
+				       (strp `(,po (or ,gv "") ,pv))
+				       (t
+					`(,po
+					  (string-to-number (or ,gv ""))
+					  ,(string-to-number pv)))))
+			     ;; Respect the star after the operand.
+			     (pt (if (match-end 7) `(and ,gv ,pt) pt)))
+			pt))
+		     (t `(member ,tag tags-list))))) ; [3]
 	      (push (if minus `(not ,mm) mm) tagsmatcher)
 	      (setq term rest)))
 	  (push `(and ,@tagsmatcher) orlist)
@@ -11520,12 +11574,12 @@ the list of tags in this group."
   "Turn an operator into the appropriate function."
   (setq op
 	(cond
-	 ((equal  op   "<"       ) '(<     org-string<  org-time<))
-	 ((equal  op   ">"       ) '(>     org-string>  org-time>))
-	 ((member op '("<=" "=<")) '(<=    org-string<= org-time<=))
-	 ((member op '(">=" "=>")) '(>=    org-string>= org-time>=))
-	 ((member op '("="  "==")) '(=     string=      org-time=))
-	 ((member op '("<>" "!=")) '(/=    org-string<> org-time<>))))
+	 ((equal  op   "<"            ) '(<     org-string<  org-time<))
+	 ((equal  op   ">"            ) '(>     org-string>  org-time>))
+	 ((member op '("<=" "=<"     )) '(<=    org-string<= org-time<=))
+	 ((member op '(">=" "=>"     )) '(>=    org-string>= org-time>=))
+	 ((member op '("="  "=="     )) '(=     string=      org-time=))
+	 ((member op '("<>" "!=" "/=")) '(/=    org-string<> org-time<>))))
   (nth (if (eq stringp 'time) 2 (if stringp 1 0)) op))
 
 (defvar org-add-colon-after-tag-completion nil)  ;; dynamically scoped param
diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el
index 890ea6a8c..7c85da9d5 100644
--- a/testing/lisp/test-org.el
+++ b/testing/lisp/test-org.el
@@ -2833,6 +2833,11 @@ test <point>
      (equal '(11)
 	    (org-test-with-temp-text "* Level 1\n** Level 2"
 	      (let (org-odd-levels-only) (org-map-entries #'point "LEVEL>1")))))
+    ;; Level match with (ignored) starred operator.
+    (should
+     (equal '(11)
+	    (org-test-with-temp-text "* Level 1\n** Level 2"
+	      (let (org-odd-levels-only) (org-map-entries #'point "LEVEL>*1")))))
     ;; Tag match.
     (should
      (equal '(11)
@@ -2845,12 +2850,17 @@ test <point>
     (should
      (equal '(11 23)
 	    (org-test-with-temp-text "* H1 :no:\n* H2 :yes1:\n* H3 :yes2:"
-	      (org-map-entries #'point "{yes?}"))))
+	      (org-map-entries #'point "{yes.?}"))))
     ;; Priority match.
     (should
      (equal '(1)
 	    (org-test-with-temp-text "* [#A] H1\n* [#B] H2"
 	      (org-map-entries #'point "PRIORITY=\"A\""))))
+    ;; Negative priority match.
+    (should
+     (equal '(11)
+	    (org-test-with-temp-text "* [#A] H1\n* [#B] H2"
+	      (org-map-entries #'point "PRIORITY/=\"A\""))))
     ;; Date match.
     (should
      (equal '(36)
@@ -2881,6 +2891,58 @@ SCHEDULED: <2014-03-04 tue.>"
 :TEST: 2
 :END:"
 	      (org-map-entries #'point "TEST=1"))))
+    ;; Regular negative property match.
+    (should
+     (equal '(35 68)
+	    (org-test-with-temp-text "
+* H1
+:PROPERTIES:
+:TEST: 1
+:END:
+* H2
+:PROPERTIES:
+:TEST: 2
+:END:
+* H3"
+	      (org-map-entries #'point "TEST!=1"))))
+    ;; Starred negative property match.
+    (should
+     (equal '(35)
+	    (org-test-with-temp-text "
+* H1
+:PROPERTIES:
+:TEST: 1
+:END:
+* H2
+:PROPERTIES:
+:TEST: 2
+:END:
+* H3"
+	      (org-map-entries #'point "TEST!=*1"))))
+    ;; Property matches on names including minus characters.
+    (org-test-with-temp-text
+     "
+* H1 :BAR:
+:PROPERTIES:
+:TEST-FOO: 1
+:END:
+* H2 :FOO:
+:PROPERTIES:
+:TEST-FOO: 2
+:END:
+* H3 :BAR:
+:PROPERTIES:
+:-FOO: 1
+:END:
+* H4 :FOO:
+:PROPERTIES:
+:-FOO: 2
+:END:
+* H5"
+     (should (equal '(2) (org-map-entries #'point "TEST-FOO!=*0-FOO")))
+     (should (equal '(2) (org-map-entries #'point "-FOO+TEST-FOO!=*0")))
+     (should (equal '(88) (org-map-entries #'point "+-FOO!=*0-FOO")))
+     (should (equal '(88) (org-map-entries #'point "-FOO+-FOO!=*0"))))
     ;; Multiple criteria.
     (should
      (equal '(23)
-- 
2.30.2

Re: org-agenda queries for absent properties

[Ihor Radchenko]

Jens Schmidt <jschmidt4gnu@vodafonemail.de> writes:

> I was trying to decipher and document the acronyms used for the
> let-bindings in that `let*' sexp.  I think I got them right but was
> unsure about the "gv".  Removed the question mark, let's just assume
> that "gv" means "getter value" and be done with it.

Yeah. I also think that it should be something like "getter value".

> Updated patch attached.

Applied, onto main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=f689eb44f
Thanks for your contribution!

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>