Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Ihor Radchenko]

[ Branching to emacs-devel for further input from Emacs devs ]

Jim Porter <jporterbugs@gmail.com> writes:

> This is similar to Emacs bug#66752[1]. It would be nice if 
> "(thing-at-point 'url)" would return the URL when point is over an Org 
> link. With this, it's easier to write a function that copies (or browses 
> to) the URL at point without coding so many special cases.
> ...
> +(defun org--url-at-point ()
> +  "`thing-at-point' provider function."
> +  (org-element-property :raw-link (org-element-context)))

Supporting thingatpt.el is certainly welcome. However, I have some
doubts about how mature thingatpt.el is.

In particular, I am concerned whether `thing-at-point-provider-alist' is
reliable enough in non-trivial scenarios like when given URL string is
not matching some generic URL regexp.

Looking into the source code of `bounds-of-thing-at-point', I see that
for standard "things" (like url),
`thing-at-point-bounds-of-url-at-point' is used unconditionally. In the
case of Org links, we may have something like [<point>[https://orgmode.org]]
that will not match default URL regexp as is. AFAIU, there is no
documented way to customize the behaviour of `bounds-of-thing-at-point'
and `forward-thing'.

I also have concerns about Org-specific part of the patch, but the above
is far more important, and we need to discuss it before starting to
consider anything for Org mode.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Jim Porter]

On 11/6/2023 12:11 PM, Ihor Radchenko wrote:
> [ Branching to emacs-devel for further input from Emacs devs ]
> 
> Jim Porter <jporterbugs@gmail.com> writes:
> 
>> This is similar to Emacs bug#66752[1]. It would be nice if
>> "(thing-at-point 'url)" would return the URL when point is over an Org
>> link. With this, it's easier to write a function that copies (or browses
>> to) the URL at point without coding so many special cases.
>> ...
>> +(defun org--url-at-point ()
>> +  "`thing-at-point' provider function."
>> +  (org-element-property :raw-link (org-element-context)))
> 
> Supporting thingatpt.el is certainly welcome. However, I have some
> doubts about how mature thingatpt.el is.
> 
> In particular, I am concerned whether `thing-at-point-provider-alist' is
> reliable enough in non-trivial scenarios like when given URL string is
> not matching some generic URL regexp.

The nice thing about 'thing-at-point-provider-alist' is that your 
provider has absolute control over what to return, so Org's URL provider 
could do whatever it wants. As far as I can tell, this code path 
completely avoids calling 'bounds-of-thing-at-point' ('botap'). However, 
it *would* call 'botap' if point wasn't on an Org link, since it would 
fall back to the last condition in 'thing-at-point'. Still, this is what 
happens today with no provider, so it's not really any worse than before...

Maybe it would make sense for 'thing-at-point' to have a "(catch 
'not-found ...)" form around the loop over 
'thing-at-point-provider-alist'. Then Org could definitively say, 
"There's no URL at point, no matter what anyone else says".

> Looking into the source code of `bounds-of-thing-at-point', I see that
> for standard "things" (like url),
> `thing-at-point-bounds-of-url-at-point' is used unconditionally. In the
> case of Org links, we may have something like [<point>[https://orgmode.org]]
> that will not match default URL regexp as is. AFAIU, there is no
> documented way to customize the behaviour of `bounds-of-thing-at-point'
> and `forward-thing'.

I think it would make sense to add some sort of 
'bounds-of-thing-at-point-provider-alist' (that's a mouthful!) that 
would let modes override the behavior of 'botap', but I don't think 
that's necessary for the narrower purpose of asking, "I want the value 
of THING at point, if any."

> I also have concerns about Org-specific part of the patch, but the above
> is far more important, and we need to discuss it before starting to
> consider anything for Org mode.

For better or worse, I mostly modeled this patch on how EWW integrates 
with thing-at-point, since that's the only place I saw in the Emacs tree 
that did this already.

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Ihor Radchenko]

Jim Porter <jporterbugs@gmail.com> writes:

>> Looking into the source code of `bounds-of-thing-at-point', I see that
>> for standard "things" (like url),
>> `thing-at-point-bounds-of-url-at-point' is used unconditionally. In the
>> case of Org links, we may have something like [<point>[https://orgmode.org]]
>> that will not match default URL regexp as is. AFAIU, there is no
>> documented way to customize the behaviour of `bounds-of-thing-at-point'
>> and `forward-thing'.
>
> I think it would make sense to add some sort of 
> 'bounds-of-thing-at-point-provider-alist' (that's a mouthful!) that 
> would let modes override the behavior of 'botap', but I don't think 
> that's necessary for the narrower purpose of asking, "I want the value 
> of THING at point, if any."

It would make sense to add a number of alists:
- bounds-of-thing-at-point-provider-alist
- same for 'forward-op, 'beginning-op, 'end-op.

After Emacs have those, we can add Org mode support.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Jim Porter]

On 2/5/2024 7:07 AM, Ihor Radchenko wrote:
> It would make sense to add a number of alists:
> - bounds-of-thing-at-point-provider-alist
> - same for 'forward-op, 'beginning-op, 'end-op.
> 
> After Emacs have those, we can add Org mode support.

That sounds reasonable enough to me; does anyone else have opinions on 
this? Otherwise, I'll get to work on a patch (though probably not for a 
couple weeks).

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Ihor Radchenko]

Jim Porter <jporterbugs@gmail.com> writes:

> On 2/5/2024 7:07 AM, Ihor Radchenko wrote:
>> It would make sense to add a number of alists:
>> - bounds-of-thing-at-point-provider-alist
>> - same for 'forward-op, 'beginning-op, 'end-op.
>> 
>> After Emacs have those, we can add Org mode support.
>
> That sounds reasonable enough to me; does anyone else have opinions on 
> this? Otherwise, I'll get to work on a patch (though probably not for a 
> couple weeks).

CCing Stefan and Eli.
Please, let us know if the above is something not wanted upstream.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: emacs-devel@gnu.org, emacs-orgmode@gnu.org
> Date: Mon, 05 Feb 2024 22:56:05 +0000
> 
> Jim Porter <jporterbugs@gmail.com> writes:
> 
> > On 2/5/2024 7:07 AM, Ihor Radchenko wrote:
> >> It would make sense to add a number of alists:
> >> - bounds-of-thing-at-point-provider-alist
> >> - same for 'forward-op, 'beginning-op, 'end-op.
> >> 
> >> After Emacs have those, we can add Org mode support.
> >
> > That sounds reasonable enough to me; does anyone else have opinions on 
> > this? Otherwise, I'll get to work on a patch (though probably not for a 
> > couple weeks).
> 
> CCing Stefan and Eli.
> Please, let us know if the above is something not wanted upstream.

I think we do want to allow extending of this, but doesn't
thingatpt.el already provide such capabilities?  For example, I see
this in bounds-of-thing-at-point:

  (defun bounds-of-thing-at-point (thing)
    "Determine the start and end buffer locations for the THING at point.
  THING should be a symbol specifying a type of syntactic entity.
  Possibilities include `symbol', `list', `sexp', `defun', `number',
  `filename', `url', `email', `uuid', `word', `sentence', `whitespace',
  `line', and `page'.

  See the file `thingatpt.el' for documentation on how to define a
  valid THING.

  Return a cons cell (START . END) giving the start and end
  positions of the thing found."
    (cond
     ((get thing 'bounds-of-thing-at-point)  <<<<<<<<<<<<<<<<<<<<<<<<
      (funcall (get thing 'bounds-of-thing-at-point)))

Doesn't this provide the extension capabilities you are looking for?
If not, why not?

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

> I think we do want to allow extending of this, but doesn't
> thingatpt.el already provide such capabilities?  For example, I see
> this in bounds-of-thing-at-point:
> ...
>     (cond
>      ((get thing 'bounds-of-thing-at-point)  <<<<<<<<<<<<<<<<<<<<<<<<
>       (funcall (get thing 'bounds-of-thing-at-point)))
>
> Doesn't this provide the extension capabilities you are looking for?
> If not, why not?

Unlike `thing-at-point-provider-alist', which can be buffer-local,
symbol property is always global and setting it would override other
thing providers.

Note how `thing-at-point' uses

(cond
          ((let ((alist thing-at-point-provider-alist)
                 elt result)
             (while (and alist (null result))
               (setq elt (car alist)
                     alist (cdr alist))
               (and (eq (car elt) thing)
                    (setq result (funcall (cdr elt)))))
             result))
          ((get thing 'thing-at-point)
           (funcall (get thing 'thing-at-point)))

checking `thing-at-point-provider-alist' and only then falling back to
`get'. What I am proposing is to add the equivalent alists for other
operators used by thingatpt.el.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: jporterbugs@gmail.com, stefankangas@gmail.com, emacs-devel@gnu.org,
>  emacs-orgmode@gnu.org
> Date: Tue, 06 Feb 2024 12:38:19 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think we do want to allow extending of this, but doesn't
> > thingatpt.el already provide such capabilities?  For example, I see
> > this in bounds-of-thing-at-point:
> > ...
> >     (cond
> >      ((get thing 'bounds-of-thing-at-point)  <<<<<<<<<<<<<<<<<<<<<<<<
> >       (funcall (get thing 'bounds-of-thing-at-point)))
> >
> > Doesn't this provide the extension capabilities you are looking for?
> > If not, why not?
> 
> Unlike `thing-at-point-provider-alist', which can be buffer-local,
> symbol property is always global and setting it would override other
> thing providers.
> 
> Note how `thing-at-point' uses
> 
> (cond
>           ((let ((alist thing-at-point-provider-alist)
>                  elt result)
>              (while (and alist (null result))
>                (setq elt (car alist)
>                      alist (cdr alist))
>                (and (eq (car elt) thing)
>                     (setq result (funcall (cdr elt)))))
>              result))
>           ((get thing 'thing-at-point)
>            (funcall (get thing 'thing-at-point)))
> 
> checking `thing-at-point-provider-alist' and only then falling back to
> `get'. What I am proposing is to add the equivalent alists for other
> operators used by thingatpt.el.

I guess it's fine, then.

But we probably should have such alists in all the other thingatpt
methods as well.

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Ihor Radchenko]

Jim Porter <jporterbugs@gmail.com> writes:

> On 2/5/2024 7:07 AM, Ihor Radchenko wrote:
>> It would make sense to add a number of alists:
>> - bounds-of-thing-at-point-provider-alist
>> - same for 'forward-op, 'beginning-op, 'end-op.
>> 
>> After Emacs have those, we can add Org mode support.
>
> That sounds reasonable enough to me; does anyone else have opinions on 
> this? Otherwise, I'll get to work on a patch (though probably not for a 
> couple weeks).

It has been a while since the last message in this thread.
Jim, may I know if you had a chance to work on the patch?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Jim Porter]

On 4/12/2024 5:41 AM, Ihor Radchenko wrote:
> Jim Porter <jporterbugs@gmail.com> writes:
> 
>> That sounds reasonable enough to me; does anyone else have opinions on
>> this? Otherwise, I'll get to work on a patch (though probably not for a
>> couple weeks).
> 
> It has been a while since the last message in this thread.
> Jim, may I know if you had a chance to work on the patch?

Sorry about that. I'm currently extremely swamped with real life, but 
most of that should be wrapped up by the end of the month, at which 
point I'll be able to devote some time to Emacs again.

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Jim Porter]

[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]

On 4/12/2024 3:30 PM, Jim Porter wrote:
> On 4/12/2024 5:41 AM, Ihor Radchenko wrote:
>> Jim Porter <jporterbugs@gmail.com> writes:
>>
>>> That sounds reasonable enough to me; does anyone else have opinions on
>>> this? Otherwise, I'll get to work on a patch (though probably not for a
>>> couple weeks).
>>
>> It has been a while since the last message in this thread.
>> Jim, may I know if you had a chance to work on the patch?
> 
> Sorry about that. I'm currently extremely swamped with real life, but 
> most of that should be wrapped up by the end of the month, at which 
> point I'll be able to devote some time to Emacs again.

Ihor, Eli: What do you think of the attached patch? I added variables to 
let modes define custom providers for 'bounds-of-thing-at-point' and 
'forward-thing'. (Notably, I avoided adding vars for the 
'beginning-of-thing' and 'end-of-thing' functions, since those just call 
'bounds-of-thing-at-point' anyway.)

If this looks like a reasonable way to go, I'll continue work on this 
patch by adding entries to 'bounds-of-thing-at-point-provider-alist' and 
'forward-thing-provider-alist' in the appropriate places (i.e. wherever 
we already add to 'thing-at-point-provider-alist', like in EWW).

[-- Attachment #2: 0001-Allow-defining-custom-providers-for-more-thingatpt-f.patch --]
[-- Type: text/plain, Size: 6486 bytes --]

From a0ed62aa42fa47043511ba814cf5ce8419e9d03f Mon Sep 17 00:00:00 2001
From: Jim Porter <jporterbugs@gmail.com>
Date: Sun, 28 Apr 2024 21:19:53 -0700
Subject: [PATCH] Allow defining custom providers for more "thingatpt"
 functions

* lisp/thingatpt.el (bounds-of-thing-at-point-provider-alist)
(forward-thing-provider-alist): New variables...
(forward-thing, bounds-of-thing-at-point): ... use them.
* test/lisp/thingatpt-tests.el (thing-at-point-providers)
(forward-thing-providers, bounds-of-thing-at-point-providers): New
tests.

* etc/NEWS: Announce this change.
---
 etc/NEWS                     | 18 +++++++++++++-----
 lisp/thingatpt.el            | 35 ++++++++++++++++++++++++++++++-----
 test/lisp/thingatpt-tests.el | 31 +++++++++++++++++++++++++++++++
 3 files changed, 74 insertions(+), 10 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 7efb4110bcd..2480f0d096d 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1591,19 +1591,27 @@ of the currently existing keyboards macros using the new mode
 duplicating them, deleting them, and editing their counters, formats,
 and keys.
 
-** Miscellaneous
+** thingatpt.el
 
 ---
-*** Webjump now assumes URIs are HTTPS instead of HTTP.
-For links in 'webjump-sites' without an explicit URI scheme, it was
-previously assumed that they should be prefixed with "http://".  Such
-URIs are now prefixed with "https://" instead.
+*** New variables for providing custom thingatpt implementations.
+The new variables 'bounds-of-thing-at-point-provider-alist' and
+'forward-thing-provider-alist' now allow defining custom implementations
+of 'bounds-of-thing-at-point' and 'forward-thing', respectively.
 
 ---
 *** 'bug-reference-mode' now supports 'thing-at-point'.
 Now, calling '(thing-at-point 'url)' when point is on a bug reference
 will return the URL for that bug.
 
+** Miscellaneous
+
+---
+*** Webjump now assumes URIs are HTTPS instead of HTTP.
+For links in 'webjump-sites' without an explicit URI scheme, it was
+previously assumed that they should be prefixed with "http://".  Such
+URIs are now prefixed with "https://" instead.
+
 +++
 *** New user option 'rcirc-log-time-format'
 This allows for rcirc logs to use a custom timestamp format, than the
diff --git a/lisp/thingatpt.el b/lisp/thingatpt.el
index 7896ad984df..d5f71e3c6a8 100644
--- a/lisp/thingatpt.el
+++ b/lisp/thingatpt.el
@@ -75,6 +75,22 @@ thing-at-point-provider-alist
 `existing-filename', `url', `email', `uuid', `word', `sentence',
 `whitespace', `line', `face' and `page'.")
 
+(defvar bounds-of-thing-at-point-provider-alist nil
+  "Alist of providers to return the bounds of a \"thing\" at point.
+This variable can be set globally, or appended to buffer-locally by
+modes, to provide functions that will return the bounds of a \"thing\"
+at point.  The first provider for the \"thing\" that returns a non-nil
+value wins.  You can use this in much the same way as
+`thing-at-point-provider-alist' (which see).")
+
+(defvar forward-thing-provider-alist nil
+  "Alist of providers for moving forward to the end of a \"thing\".
+This variable can be set globally, or appended to buffer-locally by
+modes, to provide functions that will move forward to the end of a
+\"thing\" at point.  The first provider for the \"thing\" that returns a
+non-nil value wins.  You can use this in much the same way as
+`thing-at-point-provider-alist' (which see).")
+
 ;; Basic movement
 
 ;;;###autoload
@@ -84,11 +100,16 @@ forward-thing
 Possibilities include `symbol', `list', `sexp', `defun', `number',
 `filename', `url', `email', `uuid', `word', `sentence', `whitespace',
 `line', and `page'."
-  (let ((forward-op (or (get thing 'forward-op)
-			(intern-soft (format "forward-%s" thing)))))
-    (if (functionp forward-op)
-	(funcall forward-op (or n 1))
-      (error "Can't determine how to move over a %s" thing))))
+  (setq n (or n 1))
+  (or (seq-some (lambda (elt)
+                  (and (eq (car elt) thing)
+                       (funcall (cdr elt) n)))
+                forward-thing-provider-alist)
+      (let ((forward-op (or (get thing 'forward-op)
+			    (intern-soft (format "forward-%s" thing)))))
+        (if (functionp forward-op)
+	    (funcall forward-op n)
+          (error "Can't determine how to move over a %s" thing)))))
 
 ;; General routines
 
@@ -106,6 +127,10 @@ bounds-of-thing-at-point
 Return a cons cell (START . END) giving the start and end
 positions of the thing found."
   (cond
+   ((seq-some (lambda (elt)
+                (and (eq (car elt) thing)
+                     (funcall (cdr elt))))
+                bounds-of-thing-at-point-provider-alist))
    ((get thing 'bounds-of-thing-at-point)
     (funcall (get thing 'bounds-of-thing-at-point)))
    ;; If the buffer is totally empty, give up.
diff --git a/test/lisp/thingatpt-tests.el b/test/lisp/thingatpt-tests.el
index e50738f1122..4aacd776176 100644
--- a/test/lisp/thingatpt-tests.el
+++ b/test/lisp/thingatpt-tests.el
@@ -258,4 +258,35 @@ test-numbers-hex-c
   (should (equal (test--number "0xf00" 2) 3840))
   (should (equal (test--number "0xf00" 3) 3840)))
 
+(ert-deftest thing-at-point-providers ()
+  (with-temp-buffer
+    (setq-local thing-at-point-provider-alist
+                `((url . ,(lambda () "test"))))
+    (insert "hello")
+    (should (equal (thing-at-point 'url) "test"))
+    (should (equal (thing-at-point 'word) "hello"))))
+
+(ert-deftest forward-thing-providers ()
+  (with-temp-buffer
+    (setq-local forward-thing-provider-alist
+                `((url . ,(lambda (n) (goto-char 4)))))
+    (insert "hello there")
+    (goto-char (point-min))
+    (should (eq (save-excursion (forward-thing 'url) (point)) 4))
+    (should (eq (save-excursion (forward-thing 'word) (point)) 6))))
+
+(ert-deftest bounds-of-thing-at-point-providers ()
+  (with-temp-buffer
+    (setq-local bounds-of-thing-at-point-provider-alist
+                `((url . ,(lambda () '(2 . 3)))))
+    (insert "hello")
+    ;; Look for a "URL", using our provider above.
+    (should (equal (bounds-of-thing-at-point 'url) '(2 . 3)))
+    (should (eq (save-excursion (beginning-of-thing 'url)) 2))
+    (should (eq (save-excursion (end-of-thing 'url)) 3))
+    ;; Look for a word, which should *not* use our provider above.
+    (should (equal (bounds-of-thing-at-point 'word) '(1 . 6)))
+    (should (eq (save-excursion (beginning-of-thing 'word)) 1))
+    (should (eq (save-excursion (end-of-thing 'word)) 6))))
+
 ;;; thingatpt-tests.el ends here
-- 
2.25.1

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Ihor Radchenko]

Jim Porter <jporterbugs@gmail.com> writes:

> Ihor, Eli: What do you think of the attached patch? I added variables to 
> let modes define custom providers for 'bounds-of-thing-at-point' and 
> 'forward-thing'. (Notably, I avoided adding vars for the 
> 'beginning-of-thing' and 'end-of-thing' functions, since those just call 
> 'bounds-of-thing-at-point' anyway.)
>
> If this looks like a reasonable way to go, I'll continue work on this 
> patch by adding entries to 'bounds-of-thing-at-point-provider-alist' and 
> 'forward-thing-provider-alist' in the appropriate places (i.e. wherever 
> we already add to 'thing-at-point-provider-alist', like in EWW).

Thanks!
I have a small comment on the docstring of
`forward-thing-provider-alist' - it refers to
`thing-at-point-provider-alist', but the provides here are called with
an argument N, unlike the providers in `thing-at-point-provider-alist'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Jim Porter]

[-- Attachment #1: Type: text/plain, Size: 633 bytes --]

On 4/29/2024 11:14 AM, Ihor Radchenko wrote:
> Thanks!
> I have a small comment on the docstring of
> `forward-thing-provider-alist' - it refers to
> `thing-at-point-provider-alist', but the provides here are called with
> an argument N, unlike the providers in `thing-at-point-provider-alist'.

Fixed.

I've also added some helper functions for 'forward-thing' and 
'bounds-of-thing-at-point' when the "thing" is defined by a text 
property, and then used those helper functions for EWW and 
bug-reference-mode.

I've lightly tested this (and added a few automated regression tests), 
but there could be some bugs lurking in here...

[-- Attachment #2: 0001-Allow-defining-custom-providers-for-more-thingatpt-f.patch --]
[-- Type: text/plain, Size: 12593 bytes --]

From ad8db930907cd760142fd6f035d97ce93ce8d850 Mon Sep 17 00:00:00 2001
From: Jim Porter <jporterbugs@gmail.com>
Date: Sun, 28 Apr 2024 21:19:53 -0700
Subject: [PATCH] Allow defining custom providers for more "thingatpt"
 functions

* lisp/thingatpt.el (forward-thing-provider-alist)
(bounds-of-thing-at-point-provider-alist): New variables...
(forward-thing, bounds-of-thing-at-point): ... use them.
(text-property-search-forward, text-property-search-backward)
(prop-match-beginning, prop-match-end): Declare.
(forward-thing-for-text-property)
(bounds-of-thing-at-point-for-text-property): New functions.

* lisp/net/eww.el (eww--bounds-of-url-at-point, eww--forward-url): New
functions...
(eww-mode): ... use them.

* lisp/progmodes/bug-reference.el
(bug-reference--bounds-of-url-at-point, bug-reference--forward-url): New
functions...
(bug-reference--init): ... use them.

* test/lisp/thingatpt-tests.el (thing-at-point-providers)
(forward-thing-providers, bounds-of-thing-at-point-providers): New
tests.

* etc/NEWS: Announce this change.
---
 etc/NEWS                        | 21 +++++++---
 lisp/net/eww.el                 | 14 +++++++
 lisp/progmodes/bug-reference.el | 22 +++++++++-
 lisp/thingatpt.el               | 71 ++++++++++++++++++++++++++++++---
 test/lisp/thingatpt-tests.el    | 36 +++++++++++++++++
 5 files changed, 153 insertions(+), 11 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 7efb4110bcd..394f75884c1 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1591,19 +1591,30 @@ of the currently existing keyboards macros using the new mode
 duplicating them, deleting them, and editing their counters, formats,
 and keys.
 
-** Miscellaneous
+** thingatpt.el
 
 ---
-*** Webjump now assumes URIs are HTTPS instead of HTTP.
-For links in 'webjump-sites' without an explicit URI scheme, it was
-previously assumed that they should be prefixed with "http://".  Such
-URIs are now prefixed with "https://" instead.
+*** New variables and functions for providing custom thingatpt implementations.
+The new variables 'bounds-of-thing-at-point-provider-alist' and
+'forward-thing-provider-alist' now allow defining custom implementations
+of 'bounds-of-thing-at-point' and 'forward-thing', respectively.  In
+addition, "things" defined by a text property can use the new functions
+'bounds-of-thing-at-point-for-text-property' and
+'forward-thing-for-text-property' to help implement these providers.
 
 ---
 *** 'bug-reference-mode' now supports 'thing-at-point'.
 Now, calling '(thing-at-point 'url)' when point is on a bug reference
 will return the URL for that bug.
 
+** Miscellaneous
+
+---
+*** Webjump now assumes URIs are HTTPS instead of HTTP.
+For links in 'webjump-sites' without an explicit URI scheme, it was
+previously assumed that they should be prefixed with "http://".  Such
+URIs are now prefixed with "https://" instead.
+
 +++
 *** New user option 'rcirc-log-time-format'
 This allows for rcirc logs to use a custom timestamp format, than the
diff --git a/lisp/net/eww.el b/lisp/net/eww.el
index 39ea964d47a..adabd8d8d8b 100644
--- a/lisp/net/eww.el
+++ b/lisp/net/eww.el
@@ -1321,6 +1321,12 @@ eww-mode
   (setq-local thing-at-point-provider-alist
               (append thing-at-point-provider-alist
                       '((url . eww--url-at-point))))
+  (setq-local bounds-of-thing-at-point-provider-alist
+              (append bounds-of-thing-at-point-provider-alist
+                      '((url . eww--bounds-of-url-at-point))))
+  (setq-local forward-thing-provider-alist
+              (append forward-thing-provider-alist
+                      '((url . eww--forward-url))))
   (setq-local bookmark-make-record-function #'eww-bookmark-make-record)
   (buffer-disable-undo)
   (setq-local shr-url-transformer #'eww--transform-url)
@@ -1351,6 +1357,14 @@ eww--url-at-point
   "`thing-at-point' provider function."
   (get-text-property (point) 'shr-url))
 
+(defun eww--bounds-of-url-at-point ()
+  "`bounds-of-thing-at-point' provider function."
+  (bounds-of-thing-at-point-for-text-property 'shr-url))
+
+(defun eww--forward-url (n)
+  "`forward-thing' provider function."
+  (forward-thing-for-text-property 'shr-url n))
+
 ;;;###autoload
 (defun eww-browse-url (url &optional new-window)
   "Ask the EWW browser to load URL.
diff --git a/lisp/progmodes/bug-reference.el b/lisp/progmodes/bug-reference.el
index 977a3d72cb7..bfc22fb10d2 100644
--- a/lisp/progmodes/bug-reference.el
+++ b/lisp/progmodes/bug-reference.el
@@ -660,17 +660,37 @@ bug-reference--url-at-point
   "`thing-at-point' provider function."
   (get-char-property (point) 'bug-reference-url))
 
+(defun bug-reference--bounds-of-url-at-point ()
+  "`bounds-of-thing-at-point' provider function."
+  (bounds-of-thing-at-point-for-text-property 'bug-reference-url))
+
+(defun bug-reference--forward-url (n)
+  "`forward-thing' provider function."
+  (forward-thing-for-text-property 'bug-reference-url n))
+
 (defun bug-reference--init (enable)
   (if enable
       (progn
         (jit-lock-register #'bug-reference-fontify)
         (setq-local thing-at-point-provider-alist
                     (append thing-at-point-provider-alist
-                            '((url . bug-reference--url-at-point)))))
+                            '((url . bug-reference--url-at-point))))
+        (setq-local bounds-of-thing-at-point-provider-alist
+                    (append bounds-of-thing-at-point-provider-alist
+                            '((url . bug-reference--bounds-of-url-at-point))))
+        (setq-local forward-thing-provider-alist
+                    (append forward-thing-provider-alist
+                            '((url . bug-reference--forward-url)))))
     (jit-lock-unregister #'bug-reference-fontify)
     (setq thing-at-point-provider-alist
           (delete '((url . bug-reference--url-at-point))
                   thing-at-point-provider-alist))
+    (setq bounds-of-thing-at-point-provider-alist
+          (delete '((url . bug-reference--bounds-of-url-at-point))
+                  bounds-of-thing-at-point-provider-alist))
+    (setq forward-thing-provider-alist
+          (delete '((url . bug-reference--forward-url))
+                  forward-thing-provider-alist))
     (save-restriction
       (widen)
       (bug-reference-unfontify (point-min) (point-max)))))
diff --git a/lisp/thingatpt.el b/lisp/thingatpt.el
index 7896ad984df..dad71a4ca94 100644
--- a/lisp/thingatpt.el
+++ b/lisp/thingatpt.el
@@ -75,6 +75,27 @@ thing-at-point-provider-alist
 `existing-filename', `url', `email', `uuid', `word', `sentence',
 `whitespace', `line', `face' and `page'.")
 
+(defvar forward-thing-provider-alist nil
+  "Alist of providers for moving forward to the end of a \"thing\".
+This variable can be set globally, or appended to buffer-locally by
+modes, to provide functions that will move forward to the end of a
+\"thing\" at point.  Each function should take a single argument N, the
+number of \"things\" to move forward past.  The first provider for the
+\"thing\" that returns a non-nil value wins.
+
+You can use this variable in much the same way as
+`thing-at-point-provider-alist' (which see).")
+
+(defvar bounds-of-thing-at-point-provider-alist nil
+  "Alist of providers to return the bounds of a \"thing\" at point.
+This variable can be set globally, or appended to buffer-locally by
+modes, to provide functions that will return the bounds of a \"thing\"
+at point.  The first provider for the \"thing\" that returns a non-nil
+value wins.
+
+You can use this variable in much the same way as
+`thing-at-point-provider-alist' (which see).")
+
 ;; Basic movement
 
 ;;;###autoload
@@ -84,11 +105,16 @@ forward-thing
 Possibilities include `symbol', `list', `sexp', `defun', `number',
 `filename', `url', `email', `uuid', `word', `sentence', `whitespace',
 `line', and `page'."
-  (let ((forward-op (or (get thing 'forward-op)
-			(intern-soft (format "forward-%s" thing)))))
-    (if (functionp forward-op)
-	(funcall forward-op (or n 1))
-      (error "Can't determine how to move over a %s" thing))))
+  (setq n (or n 1))
+  (or (seq-some (lambda (elt)
+                  (and (eq (car elt) thing)
+                       (funcall (cdr elt) n)))
+                forward-thing-provider-alist)
+      (let ((forward-op (or (get thing 'forward-op)
+			    (intern-soft (format "forward-%s" thing)))))
+        (if (functionp forward-op)
+	    (funcall forward-op n)
+          (error "Can't determine how to move over a %s" thing)))))
 
 ;; General routines
 
@@ -106,6 +132,10 @@ bounds-of-thing-at-point
 Return a cons cell (START . END) giving the start and end
 positions of the thing found."
   (cond
+   ((seq-some (lambda (elt)
+                (and (eq (car elt) thing)
+                     (funcall (cdr elt))))
+                bounds-of-thing-at-point-provider-alist))
    ((get thing 'bounds-of-thing-at-point)
     (funcall (get thing 'bounds-of-thing-at-point)))
    ;; If the buffer is totally empty, give up.
@@ -775,4 +805,35 @@ list-at-point
       (goto-char (or (nth 8 ppss) (point)))
       (form-at-point 'list 'listp))))
 
+(autoload 'text-property-search-forward "text-property-search")
+(autoload 'text-property-search-backward "text-property-search")
+(autoload 'prop-match-beginning "text-property-search")
+(autoload 'prop-match-end "text-property-search")
+
+(defun forward-thing-for-text-property (property n)
+  "Move forward to the end of the Nth next \"thing\".
+Each \"thing\" is a region of text with the specified text PROPERTY set."
+  (let ((search-func (if (> n 0) #'text-property-search-forward
+                       #'text-property-search-backward))
+        (pos-func (if (> n 0) #'prop-match-end #'prop-match-beginning))
+        (limit (if (> n 0) (point-max) (point-min))))
+    (catch 'done
+      (dotimes (_ (abs n))
+        (if-let ((match (funcall search-func property)))
+            (goto-char (funcall pos-func match))
+          (goto-char limit)
+          (throw 'done t))))
+    ;; Return non-nil.
+    t))
+
+(defun bounds-of-thing-at-point-for-text-property (property)
+  "Determine the start and end buffer locations for the \"thing\" at point.
+The \"thing\" is a region of text with the specified text PROPERTY set."
+  (when (get-text-property (point) property)
+    (cons (or (previous-single-property-change
+               (min (1+ (point)) (point-max)) property)
+              (point-min))
+          (or (next-single-property-change (point) property)
+              (point-max)))))
+
 ;;; thingatpt.el ends here
diff --git a/test/lisp/thingatpt-tests.el b/test/lisp/thingatpt-tests.el
index e50738f1122..26e20f58be7 100644
--- a/test/lisp/thingatpt-tests.el
+++ b/test/lisp/thingatpt-tests.el
@@ -258,4 +258,40 @@ test-numbers-hex-c
   (should (equal (test--number "0xf00" 2) 3840))
   (should (equal (test--number "0xf00" 3) 3840)))
 
+(ert-deftest thing-at-point-providers ()
+  (with-temp-buffer
+    (setq-local thing-at-point-provider-alist
+                `((url . ,(lambda () (get-text-property (point) 'my-url)))))
+    (insert (propertize "hello" 'my-url "test"))
+    (goto-char (point-min))
+    (should (equal (thing-at-point 'url) "test"))
+    (should (equal (thing-at-point 'word) "hello"))))
+
+(ert-deftest forward-thing-providers ()
+  (with-temp-buffer
+    (setq-local forward-thing-provider-alist
+                `((url . ,(lambda (n)
+                            (forward-thing-for-text-property 'my-url n)))))
+    (insert (propertize "foo" 'my-url "test") "bar")
+    (goto-char (point-min))
+    (should (eq (save-excursion (forward-thing 'url) (point)) 4))
+    (should (eq (save-excursion (forward-thing 'word) (point)) 7))))
+
+(ert-deftest bounds-of-thing-at-point-providers ()
+  (with-temp-buffer
+    (setq-local bounds-of-thing-at-point-provider-alist
+                `((url . ,(lambda ()
+                            (bounds-of-thing-at-point-for-text-property
+                             'my-url)))))
+    (insert (propertize "foo" 'my-url "test") "bar")
+    (goto-char (point-min))
+    ;; Look for a "URL", using our provider above.
+    (should (equal (bounds-of-thing-at-point 'url) '(1 . 4)))
+    (should (eq (save-excursion (beginning-of-thing 'url)) 1))
+    (should (eq (save-excursion (end-of-thing 'url)) 4))
+    ;; Look for a word, which should *not* use our provider above.
+    (should (equal (bounds-of-thing-at-point 'word) '(1 . 7)))
+    (should (eq (save-excursion (beginning-of-thing 'word)) 1))
+    (should (eq (save-excursion (end-of-thing 'word)) 7))))
+
 ;;; thingatpt-tests.el ends here
-- 
2.25.1

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Ihor Radchenko]

Jim Porter <jporterbugs@gmail.com> writes:

> +  (setq-local bounds-of-thing-at-point-provider-alist
> +              (append bounds-of-thing-at-point-provider-alist
> +                      '((url . eww--bounds-of-url-at-point))))
> +  (setq-local forward-thing-provider-alist
> +              (append forward-thing-provider-alist
> +                      '((url . eww--forward-url))))
> ...

What happens if you have multiple providers for an URL?
You add the provider to the end, so it will have the lower priority in
this scenario. I guess that you want the opposite - EWW provider to take
precedence. Same for other changes.

> +(ert-deftest thing-at-point-providers ()
> ...
> +(ert-deftest forward-thing-providers ()
> ...
> +(ert-deftest bounds-of-thing-at-point-providers ()
> +  (with-temp-buffer
> +    (setq-local bounds-of-thing-at-point-provider-alist
> +                `((url . ,(lambda ()
> +                            (bounds-of-thing-at-point-for-text-property
> +                             'my-url)))))

It would make sense to add tests for "first wins" behaviour.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Jim Porter]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

On 4/30/2024 4:39 AM, Ihor Radchenko wrote:
> What happens if you have multiple providers for an URL?
> You add the provider to the end, so it will have the lower priority in
> this scenario. I guess that you want the opposite - EWW provider to take
> precedence. Same for other changes.

That's probably reasonable. I was just keeping things the way they were 
historically here, but we might as well fix this now.

> It would make sense to add tests for "first wins" behaviour.

Done.

I've also fixed a bug in EWW and bug-reference-mode where it would 
return nil for (thing-at-point 'url) if point was at the *end* of a URL. 
It's now consistent with how 'thing-at-point' works by default. (If you 
have two consecutive URLs and point is between them - only possible with 
the custom provider function, I think - it'll prefer the second one.)

[-- Attachment #2: 0001-Allow-defining-custom-providers-for-more-thingatpt-f.patch --]
[-- Type: text/plain, Size: 14978 bytes --]

From da26f0160c955f15e123e5b28cf8a9f514395e21 Mon Sep 17 00:00:00 2001
From: Jim Porter <jporterbugs@gmail.com>
Date: Sun, 28 Apr 2024 21:19:53 -0700
Subject: [PATCH] Allow defining custom providers for more "thingatpt"
 functions

This also fixes an issue in EWW and bug-reference-mode where
(thing-at-point 'url) at the end of a URL would return nil.

* lisp/thingatpt.el (forward-thing-provider-alist)
(bounds-of-thing-at-point-provider-alist): New variables...
(forward-thing, bounds-of-thing-at-point): ... use them.
(text-property-search-forward, text-property-search-backward)
(prop-match-beginning, prop-match-end): Declare.
(thing-at-point-for-text-property, forward-thing-for-text-property)
(bounds-of-thing-at-point-for-text-property): New functions.

* lisp/net/eww.el (eww--url-at-point): Use
'thing-at-point-for-text-property'.
(eww--bounds-of-url-at-point, eww--forward-url): New functions...
(eww-mode): ... use them.

* lisp/progmodes/bug-reference.el (bug-reference--url-at-point): Use
'thing-at-point-for-text-property'.
(bug-reference--bounds-of-url-at-point, bug-reference--forward-url): New
functions...
(bug-reference--init): ... use them.

* test/lisp/thingatpt-tests.el (thing-at-point-providers)
(forward-thing-providers, bounds-of-thing-at-point-providers): New
tests.

* etc/NEWS: Announce this change.
---
 etc/NEWS                        | 25 ++++++++--
 lisp/net/eww.el                 | 21 +++++++--
 lisp/progmodes/bug-reference.el | 26 +++++++++--
 lisp/thingatpt.el               | 83 +++++++++++++++++++++++++++++++--
 test/lisp/thingatpt-tests.el    | 59 +++++++++++++++++++++++
 5 files changed, 198 insertions(+), 16 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index 7efb4110bcd..061161bb2fd 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -1591,19 +1591,34 @@ of the currently existing keyboards macros using the new mode
 duplicating them, deleting them, and editing their counters, formats,
 and keys.
 
-** Miscellaneous
+** Thingatpt
 
 ---
-*** Webjump now assumes URIs are HTTPS instead of HTTP.
-For links in 'webjump-sites' without an explicit URI scheme, it was
-previously assumed that they should be prefixed with "http://".  Such
-URIs are now prefixed with "https://" instead.
+*** New variables for providing custom thingatpt implementations.
+The new variables 'bounds-of-thing-at-point-provider-alist' and
+'forward-thing-provider-alist' now allow defining custom implementations
+of 'bounds-of-thing-at-point' and 'forward-thing', respectively.
+
+---
+*** New helper functions for text property-based thingatpt providers.
+The new helper functions 'thing-at-point-for-text-property',
+'bounds-of-thing-at-point-for-text-property', and
+'forward-thing-for-text-property' can help to help implement custom
+thingatpt providers for "things" that are defined by a text property.
 
 ---
 *** 'bug-reference-mode' now supports 'thing-at-point'.
 Now, calling '(thing-at-point 'url)' when point is on a bug reference
 will return the URL for that bug.
 
+** Miscellaneous
+
+---
+*** Webjump now assumes URIs are HTTPS instead of HTTP.
+For links in 'webjump-sites' without an explicit URI scheme, it was
+previously assumed that they should be prefixed with "http://".  Such
+URIs are now prefixed with "https://" instead.
+
 +++
 *** New user option 'rcirc-log-time-format'
 This allows for rcirc logs to use a custom timestamp format, than the
diff --git a/lisp/net/eww.el b/lisp/net/eww.el
index 39ea964d47a..b3997786d9e 100644
--- a/lisp/net/eww.el
+++ b/lisp/net/eww.el
@@ -1318,9 +1318,16 @@ eww-mode
   ;; desktop support
   (setq-local desktop-save-buffer #'eww-desktop-misc-data)
   (setq truncate-lines t)
+  ;; thingatpt support
   (setq-local thing-at-point-provider-alist
-              (append thing-at-point-provider-alist
-                      '((url . eww--url-at-point))))
+              (cons '(url . eww--url-at-point)
+                    thing-at-point-provider-alist))
+  (setq-local forward-thing-provider-alist
+              (cons '(url . eww--forward-url)
+                    forward-thing-provider-alist))
+  (setq-local bounds-of-thing-at-point-provider-alist
+              (cons '(url . eww--bounds-of-url-at-point)
+                    bounds-of-thing-at-point-provider-alist))
   (setq-local bookmark-make-record-function #'eww-bookmark-make-record)
   (buffer-disable-undo)
   (setq-local shr-url-transformer #'eww--transform-url)
@@ -1349,7 +1356,15 @@ eww--rescale-images
 
 (defun eww--url-at-point ()
   "`thing-at-point' provider function."
-  (get-text-property (point) 'shr-url))
+  (thing-at-point-for-text-property 'shr-url))
+
+(defun eww--forward-url (n)
+  "`forward-thing' provider function."
+  (forward-thing-for-text-property 'shr-url n))
+
+(defun eww--bounds-of-url-at-point ()
+  "`bounds-of-thing-at-point' provider function."
+  (bounds-of-thing-at-point-for-text-property 'shr-url))
 
 ;;;###autoload
 (defun eww-browse-url (url &optional new-window)
diff --git a/lisp/progmodes/bug-reference.el b/lisp/progmodes/bug-reference.el
index 977a3d72cb7..be162cf9e11 100644
--- a/lisp/progmodes/bug-reference.el
+++ b/lisp/progmodes/bug-reference.el
@@ -658,19 +658,39 @@ bug-reference--run-auto-setup
 
 (defun bug-reference--url-at-point ()
   "`thing-at-point' provider function."
-  (get-char-property (point) 'bug-reference-url))
+  (thing-at-point-for-text-property 'bug-reference-url))
+
+(defun bug-reference--forward-url (n)
+  "`forward-thing' provider function."
+  (forward-thing-for-text-property 'bug-reference-url n))
+
+(defun bug-reference--bounds-of-url-at-point ()
+  "`bounds-of-thing-at-point' provider function."
+  (bounds-of-thing-at-point-for-text-property 'bug-reference-url))
 
 (defun bug-reference--init (enable)
   (if enable
       (progn
         (jit-lock-register #'bug-reference-fontify)
         (setq-local thing-at-point-provider-alist
-                    (append thing-at-point-provider-alist
-                            '((url . bug-reference--url-at-point)))))
+                    (cons '(url . bug-reference--url-at-point)
+                          thing-at-point-provider-alist))
+        (setq-local forward-thing-provider-alist
+                    (cons '(url . bug-reference--forward-url)
+                          forward-thing-provider-alist))
+        (setq-local bounds-of-thing-at-point-provider-alist
+                    (cons '(url . bug-reference--bounds-of-url-at-point)
+                          bounds-of-thing-at-point-provider-alist)))
     (jit-lock-unregister #'bug-reference-fontify)
     (setq thing-at-point-provider-alist
           (delete '((url . bug-reference--url-at-point))
                   thing-at-point-provider-alist))
+    (setq forward-thing-provider-alist
+          (delete '((url . bug-reference--forward-url))
+                  forward-thing-provider-alist))
+    (setq bounds-of-thing-at-point-provider-alist
+          (delete '((url . bug-reference--bounds-of-url-at-point))
+                  bounds-of-thing-at-point-provider-alist))
     (save-restriction
       (widen)
       (bug-reference-unfontify (point-min) (point-max)))))
diff --git a/lisp/thingatpt.el b/lisp/thingatpt.el
index 7896ad984df..825f49cfab7 100644
--- a/lisp/thingatpt.el
+++ b/lisp/thingatpt.el
@@ -75,6 +75,27 @@ thing-at-point-provider-alist
 `existing-filename', `url', `email', `uuid', `word', `sentence',
 `whitespace', `line', `face' and `page'.")
 
+(defvar forward-thing-provider-alist nil
+  "Alist of providers for moving forward to the end of a \"thing\".
+This variable can be set globally, or appended to buffer-locally by
+modes, to provide functions that will move forward to the end of a
+\"thing\" at point.  Each function should take a single argument N, the
+number of \"things\" to move forward past.  The first provider for the
+\"thing\" that returns a non-nil value wins.
+
+You can use this variable in much the same way as
+`thing-at-point-provider-alist' (which see).")
+
+(defvar bounds-of-thing-at-point-provider-alist nil
+  "Alist of providers to return the bounds of a \"thing\" at point.
+This variable can be set globally, or appended to buffer-locally by
+modes, to provide functions that will return the bounds of a \"thing\"
+at point.  The first provider for the \"thing\" that returns a non-nil
+value wins.
+
+You can use this variable in much the same way as
+`thing-at-point-provider-alist' (which see).")
+
 ;; Basic movement
 
 ;;;###autoload
@@ -84,11 +105,16 @@ forward-thing
 Possibilities include `symbol', `list', `sexp', `defun', `number',
 `filename', `url', `email', `uuid', `word', `sentence', `whitespace',
 `line', and `page'."
-  (let ((forward-op (or (get thing 'forward-op)
-			(intern-soft (format "forward-%s" thing)))))
-    (if (functionp forward-op)
-	(funcall forward-op (or n 1))
-      (error "Can't determine how to move over a %s" thing))))
+  (setq n (or n 1))
+  (or (seq-some (lambda (elt)
+                  (and (eq (car elt) thing)
+                       (funcall (cdr elt) n)))
+                forward-thing-provider-alist)
+      (let ((forward-op (or (get thing 'forward-op)
+			    (intern-soft (format "forward-%s" thing)))))
+        (if (functionp forward-op)
+	    (funcall forward-op n)
+          (error "Can't determine how to move over a %s" thing)))))
 
 ;; General routines
 
@@ -106,6 +132,10 @@ bounds-of-thing-at-point
 Return a cons cell (START . END) giving the start and end
 positions of the thing found."
   (cond
+   ((seq-some (lambda (elt)
+                (and (eq (car elt) thing)
+                     (funcall (cdr elt))))
+                bounds-of-thing-at-point-provider-alist))
    ((get thing 'bounds-of-thing-at-point)
     (funcall (get thing 'bounds-of-thing-at-point)))
    ;; If the buffer is totally empty, give up.
@@ -775,4 +805,47 @@ list-at-point
       (goto-char (or (nth 8 ppss) (point)))
       (form-at-point 'list 'listp))))
 
+;; Provider helper functions
+
+(defun thing-at-point-for-text-property (property)
+  "Return the \"thing\" at point.
+Each \"thing\" is a region of text with the specified text PROPERTY set."
+  (or (get-text-property (point) property)
+      (and (> (point) (point-min))
+           (get-text-property (1- (point)) property))))
+
+(autoload 'text-property-search-forward "text-property-search")
+(autoload 'text-property-search-backward "text-property-search")
+(autoload 'prop-match-beginning "text-property-search")
+(autoload 'prop-match-end "text-property-search")
+
+(defun forward-thing-for-text-property (property n)
+  "Move forward to the end of the Nth next \"thing\".
+Each \"thing\" is a region of text with the specified text PROPERTY set."
+  (let ((search-func (if (> n 0) #'text-property-search-forward
+                       #'text-property-search-backward))
+        (pos-func (if (> n 0) #'prop-match-end #'prop-match-beginning))
+        (limit (if (> n 0) (point-max) (point-min))))
+    (catch 'done
+      (dotimes (_ (abs n))
+        (if-let ((match (funcall search-func property)))
+            (goto-char (funcall pos-func match))
+          (goto-char limit)
+          (throw 'done t))))
+    ;; Return non-nil.
+    t))
+
+(defun bounds-of-thing-at-point-for-text-property (property)
+  "Determine the start and end buffer locations for the \"thing\" at point.
+The \"thing\" is a region of text with the specified text PROPERTY set."
+  (let ((pos (point)))
+    (when (or (get-text-property pos property)
+              (and (> pos (point-min))
+                   (get-text-property (setq pos (1- pos)) property)))
+      (cons (or (previous-single-property-change
+                 (min (1+ pos) (point-max)) property)
+                (point-min))
+            (or (next-single-property-change pos property)
+                (point-max))))))
+
 ;;; thingatpt.el ends here
diff --git a/test/lisp/thingatpt-tests.el b/test/lisp/thingatpt-tests.el
index e50738f1122..88a4bc8a27d 100644
--- a/test/lisp/thingatpt-tests.el
+++ b/test/lisp/thingatpt-tests.el
@@ -258,4 +258,63 @@ test-numbers-hex-c
   (should (equal (test--number "0xf00" 2) 3840))
   (should (equal (test--number "0xf00" 3) 3840)))
 
+(ert-deftest thing-at-point-providers ()
+  (with-temp-buffer
+    (setq-local
+     thing-at-point-provider-alist
+     `((url . ,(lambda () (thing-at-point-for-text-property 'foo-url)))
+       (url . ,(lambda () (thing-at-point-for-text-property 'bar-url)))))
+    (insert (propertize "hello" 'foo-url "foo.com") "\n"
+            (propertize "goodbye" 'bar-url "bar.com"))
+    (goto-char (point-min))
+    ;; Get the URL using the first provider.
+    (should (equal (thing-at-point 'url) "foo.com"))
+    (should (equal (thing-at-point 'word) "hello"))
+    (goto-char (point-max))
+    ;; Get the URL using the second provider.
+    (should (equal (thing-at-point 'url) "bar.com"))))
+
+(ert-deftest forward-thing-providers ()
+  (with-temp-buffer
+    (setq-local
+     forward-thing-provider-alist
+     `((url . ,(lambda (n) (forward-thing-for-text-property 'foo-url n)))
+       (url . ,(lambda (n) (forward-thing-for-text-property 'bar-url n)))))
+    (insert (propertize "hello" 'foo-url "foo.com") "there\n"
+            (propertize "goodbye" 'bar-url "bar.com"))
+    (goto-char (point-min))
+    (save-excursion
+      (forward-thing 'url)              ; Move past the first URL.
+      (should (= (point) 6))
+      (forward-thing 'url)              ; Move past the second URL.
+      (should (= (point) 19)))
+    (goto-char (point-min))             ; Go back to the beginning...
+    (forward-thing 'word)               ; ... and move past the first word.
+    (should (= (point) 11))))
+
+(ert-deftest bounds-of-thing-at-point-providers ()
+  (with-temp-buffer
+    (setq-local
+     bounds-of-thing-at-point-provider-alist
+     `((url . ,(lambda ()
+                 (bounds-of-thing-at-point-for-text-property 'foo-url)))
+       (url . ,(lambda ()
+                 (bounds-of-thing-at-point-for-text-property 'bar-url)))))
+    (insert (propertize "hello" 'foo-url "foo.com") "there\n"
+            (propertize "goodbye" 'bar-url "bar.com"))
+    (goto-char (point-min))
+    ;; Look for a URL, using the first provider above.
+    (should (equal (bounds-of-thing-at-point 'url) '(1 . 6)))
+    (should (eq (save-excursion (beginning-of-thing 'url)) 1))
+    (should (eq (save-excursion (end-of-thing 'url)) 6))
+    ;; Look for a word, which should *not* use our provider above.
+    (should (equal (bounds-of-thing-at-point 'word) '(1 . 11)))
+    (should (eq (save-excursion (beginning-of-thing 'word)) 1))
+    (should (eq (save-excursion (end-of-thing 'word)) 11))
+    (goto-char (point-max))
+    ;; Look for a URL, using the second provider above.
+    (should (equal (bounds-of-thing-at-point 'url) '(12 . 19)))
+    (should (eq (save-excursion (beginning-of-thing 'url)) 12))
+    (should (eq (save-excursion (end-of-thing 'url)) 19))))
+
 ;;; thingatpt-tests.el ends here
-- 
2.25.1

RE: [External] : Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Drew Adams]

> I've also fixed a bug in EWW and bug-reference-mode 
> where it would return nil for (thing-at-point 'url)
> if point was at the *end* of a URL.

By "at the end" I assume you really mean just
_after_ a URL, i.e., no longer on/at the URL.

FWIW, that's actually _superior_ behavior.

Unfortunately however, Emacs has chosen the
behavior you describe here:

> It's now consistent with how 'thing-at-point'
> works by default.

> (If you have two consecutive URLs and point
> is between them...it'll prefer the second one.)

Which is better!  It's what "at point" means.

(Yes, technically point is between the chars.)

And with a block cursor the cursor is on the
second thing, not the first.

And `C-x =' describes the current "cursor
position" (aka point), and it describes - wait
for it - not the char before point but the char
after point, IOW, colloquially the char at point.

And `forward-sexp', `forward-word', `forward-thing',
etc. advance just _past_ the thing.  The cursor
is then _not_ on the thing, and unless the thing
is immediately followed by another thing, there's
_no_ thing at point.

Unfortunately, Emacs maintainers decided that
thingatpt.el isn't useful for anything except
obtaining something to use as a default value for
user input.  The opinion was that no one ever
wants/needs to get nil, telling them that there's
no thing at point.  Better, they think, to always
try to get a thing at point OR at (1- point).

This awful Emacs behavior defeats the successive
use of functions that do something with the next
thing at point, in precisely the case you cited:
when the next thing butts up against the previous
thing.

In particular, these important use cases are
defeated by the behavior chosen for Emacs:

1. To find out _whether there is_, in fact,
   a THING at point.  AT POINT - not point OR
   (point - 1).

2. IF there really is a THING at point, to
   return it (or its bounds).

See bug #9300, " `bounds-of-thing-at-point'
does not return nil when just after THING".
___

Library thingatpt+.el fixes this, providing
more useful behavior for thing-at-point, and
making more use cases possible.

It also provides functions for picking up a
thing that's _near_ point (where "near" can
be specified).

That's what Emacs _should_ do for the only
use case it even cares about, which is trying
to get a thing for use as a default value for
input.  Getting a thing near point is quite
different from getting a thing _at point_.
___

https://www.emacswiki.org/emacs/download/thingatpt%2b.el

Re: RE: [External] : Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Jim Porter]

On 4/30/2024 2:10 PM, Drew Adams wrote:
>> I've also fixed a bug in EWW and bug-reference-mode
>> where it would return nil for (thing-at-point 'url)
>> if point was at the *end* of a URL.
> 
> By "at the end" I assume you really mean just
> _after_ a URL, i.e., no longer on/at the URL.
> 
> FWIW, that's actually _superior_ behavior.
> 
> Unfortunately however, Emacs has chosen the
> behavior you describe here:
> 
>> It's now consistent with how 'thing-at-point'
>> works by default.
> 
>> (If you have two consecutive URLs and point
>> is between them...it'll prefer the second one.)
> 
> Which is better!  It's what "at point" means.
[snip]
> See bug #9300, " `bounds-of-thing-at-point'
> does not return nil when just after THING".

I agree overall that your proposed behavior is more correct, and it's 
probably how I'd have implemented 'thing-at-point' if I were doing it 
from scratch. However, I think an even worse outcome than 
"thing-at-point looks at point or before-point" is "sometimes 
thing-at-point just looks at point, and other times it looks at point or 
before-point" (which is what it does today).

I'd even be open to something like a 'thing-at-point-is-strict' defvar 
that people could let-bind as wanted, but I'm not going to *argue* for 
that myself.

Ultimately though, this patch is really just about providing the 
necessary defcustoms for org-mode to be able to use 'thing-at-point' 
(and for Ihor to feel ok about it ;)). Changing 'thing-at-point's 
behavior should probably be handled separately, especially since there'd 
be an uphill battle to revisit the decision in bug#9300.

RE: RE: [External] : Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Drew Adams]

Thanks for your reply, Jim.

> On 4/30/2024 2:10 PM, Drew Adams wrote:
> >> I've also fixed a bug in EWW and bug-reference-mode
> >> where it would return nil for (thing-at-point 'url)
> >> if point was at the *end* of a URL.
> >
> > By "at the end" I assume you really mean just
> > _after_ a URL, i.e., no longer on/at the URL.
> >
> > FWIW, that's actually _superior_ behavior.
> >
> > Unfortunately however, Emacs has chosen the
> > behavior you describe here:
> >
> >> It's now consistent with how 'thing-at-point'
> >> works by default.
> >
> >> (If you have two consecutive URLs and point
> >> is between them...it'll prefer the second one.)
> >
> > Which is better!  It's what "at point" means.
> >
> [snip]
> >
> > See bug #9300, " `bounds-of-thing-at-point'
> > does not return nil when just after THING".
> 
> I agree overall that your proposed behavior is more correct, and it's
> probably how I'd have implemented 'thing-at-point' if I were doing it
> from scratch. However, I think an even worse outcome than
> "thing-at-point looks at point or before-point" is "sometimes
> thing-at-point just looks at point, and other times it looks at point or
> before-point" (which is what it does today).

Yes, such inconsistency is arguably worse than
consistent bad behavior.  Arguably worse - and
arguably better.  (That's in the nature of
inconsistency - some will see a glass half
full; others half empty.)

> I'd even be open to something like a 'thing-at-point-is-strict' defvar
> that people could let-bind as wanted, but I'm not going to *argue* for
> that myself.
> 
> Ultimately though, this patch is really just about providing the
> necessary defcustoms for org-mode to be able to use 'thing-at-point'
> (and for Ihor to feel ok about it ;)). Changing 'thing-at-point's
> behavior should probably be handled separately, especially since there'd
> be an uphill battle to revisit the decision in bug#9300.

I hear you.  The behavior should be changed so
that, in general, bounds-of-thing-at-point etc.
return nil when there is _no thing at point_,
including when point is after, including just
after, a thing but not on such a thing.

There can be commands (and noncommand fns) that
return things _near_ point, not only at point.
And "near" can be configurable with an argument.

In particular, they can do what the vanilla fns
currently do: return a thing at OR just before
point.  But the "-at-point" functions shouldn't
do that.  They should do what their names say.

It's important to have such functions.  It's
not just about arguing that strictly at-point
is better than at-or-just-after-point.  The
point (sic) is that currently there's _no way
to determine whether_ there's a thing at point.

That's the real problem - no test for a thing
at a given position.  That means that a whole
slew of possible applications of the feature
are impossible to realize.
___

Along with the fix for this bug, there could
also be a defvar, or even an option, that fuses
the two behaviors (at OR just before), i.e.,
that gives the current, misguided behavior, to
allow existing code or habits compatibility.

It's not hard for Emacs to still DTRT.  It just
takes a decision and admission that the behavior
was misguided and unnecessarily limiting (BIG
time).

Re: [External] : Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "eliz@gnu.org" <eliz@gnu.org>,
>         "emacs-devel@gnu.org"
> 	<emacs-devel@gnu.org>,
>         "emacs-orgmode@gnu.org" <emacs-orgmode@gnu.org>
> Date: Tue, 7 May 2024 01:52:13 +0000
> 
> I hear you.  The behavior should be changed so
> that, in general, bounds-of-thing-at-point etc.
> return nil when there is _no thing at point_,
> including when point is after, including just
> after, a thing but not on such a thing.
> 
> There can be commands (and noncommand fns) that
> return things _near_ point, not only at point.
> And "near" can be configurable with an argument.
> 
> In particular, they can do what the vanilla fns
> currently do: return a thing at OR just before
> point.  But the "-at-point" functions shouldn't
> do that.  They should do what their names say.

I disagree.  These functions are nowadays the basis of many
interactive features, and users are usually mightily confused by the
fine print of what "at point" means technically in Emacs.  The current
operation is much easier for users to grasp mentally by observing the
position of the cursor, whether it's on or just after the "thing".

> It's not hard for Emacs to still DTRT.  It just
> takes a decision and admission that the behavior
> was misguided and unnecessarily limiting (BIG
> time).

We made the decision.  It just is not what you think it should be,
because our considerations are different from yours.

RE: [External] : Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Drew Adams]

> > I hear you.  The behavior should be changed so
> > that, in general, bounds-of-thing-at-point etc.
> > return nil when there is _no thing at point_,
> > including when point is after, including just
> > after, a thing but not on such a thing.
> >
> > There can be commands (and noncommand fns) that
> > return things _near_ point, not only at point.
> > And "near" can be configurable with an argument.
> >
> > In particular, they can do what the vanilla fns
> > currently do: return a thing at OR just before
> > point.  But the "-at-point" functions shouldn't
> > do that.  They should do what their names say.
> 
> I disagree.  These functions are nowadays the basis of many
> interactive features, and users are usually mightily confused by the
> fine print of what "at point" means technically in Emacs.  The current
> operation is much easier for users to grasp mentally by observing the
> position of the cursor, whether it's on or just after the "thing".

Of course.  IF the only use case is (as it's limited
to now) trying to get something near point, to use
as a _default value for interactive prompt and input_,
THEN there's no need for an actual at-point semantics.
No need and no advantage.

The point (sic) is that there are important, useful
_additional_ use cases, not supported by the current
ad-hoc semantics.  The idea behind Thing At Point is
much more general than the single use case that the
"many interactive features" you tout boil down to.

And no, legitimate at-point semantics doesn't require
any complex mental grasping.  It's in fact the current
behavior that leads to mental gyrations & puzzling,
because of the inconsistency that Jim pointed out.

The proper semantics is in fact far simpler to "grasp".  
You're just _used_ to grabbing things that aren't at
point, such as a list immediately before point.  Habit
can make you think things are straightforward & simple.
 
> > It's not hard for Emacs to still DTRT.  It just
> > takes a decision and admission that the behavior
> > was misguided and unnecessarily limiting (BIG
> > time).
> 
> We made the decision.

We know.  It's not too late to fix things, and it's
not hard to do.  What's lacking is the volition.

> It just is not what you think it should be, because
> our considerations are different from yours.

Exactly.  Your considerations are limited to maximizing
the possibility of returning a thing near point - more
precisely just before point OR AT point (aka just after
it).  (And as Jim pointed out, even that's not supported
consistently.)

With those blinders on, your decision makes some sense.
Take off the blinders and see there's a whole world
surrounding the narrow slice you saw looking only
straight ahead.

Re: [External] : Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Jim Porter]

On 5/7/2024 5:20 AM, Eli Zaretskii wrote:
> I disagree.  These functions are nowadays the basis of many
> interactive features, and users are usually mightily confused by the
> fine print of what "at point" means technically in Emacs.  The current
> operation is much easier for users to grasp mentally by observing the
> position of the cursor, whether it's on or just after the "thing".

At the risk of veering off-topic (I mainly care about adding 
'bounds-of-thing-at-point-provider-alist' and 
'forward-thing-provider-alist'), would adding a new optional STRICT 
argument to 'thing-at-point' and friends be an ok resolution for 
everyone? This argument would enable Drew's proposed behavior. That way, 
users get all the nice behavior by default just like today, and 
programmers who require strict correctness in their code also have an 
option.

Re: [External] : Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Eli Zaretskii]

> Date: Tue, 7 May 2024 09:10:59 -0700
> Cc: yantar92@posteo.net, emacs-devel@gnu.org, emacs-orgmode@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 5/7/2024 5:20 AM, Eli Zaretskii wrote:
> > I disagree.  These functions are nowadays the basis of many
> > interactive features, and users are usually mightily confused by the
> > fine print of what "at point" means technically in Emacs.  The current
> > operation is much easier for users to grasp mentally by observing the
> > position of the cursor, whether it's on or just after the "thing".
> 
> At the risk of veering off-topic (I mainly care about adding 
> 'bounds-of-thing-at-point-provider-alist' and 
> 'forward-thing-provider-alist'), would adding a new optional STRICT 
> argument to 'thing-at-point' and friends be an ok resolution for 
> everyone? This argument would enable Drew's proposed behavior. That way, 
> users get all the nice behavior by default just like today, and 
> programmers who require strict correctness in their code also have an 
> option.

If enough people want the other behavior, I won't object to supporting
that as well as an option.

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Eli Zaretskii]

> Date: Tue, 30 Apr 2024 11:27:04 -0700
> Cc: eliz@gnu.org, emacs-devel@gnu.org, emacs-orgmode@gnu.org
> From: Jim Porter <jporterbugs@gmail.com>
> 
> On 4/30/2024 4:39 AM, Ihor Radchenko wrote:
> > What happens if you have multiple providers for an URL?
> > You add the provider to the end, so it will have the lower priority in
> > this scenario. I guess that you want the opposite - EWW provider to take
> > precedence. Same for other changes.
> 
> That's probably reasonable. I was just keeping things the way they were 
> historically here, but we might as well fix this now.
> 
> > It would make sense to add tests for "first wins" behaviour.
> 
> Done.
> 
> I've also fixed a bug in EWW and bug-reference-mode where it would 
> return nil for (thing-at-point 'url) if point was at the *end* of a URL. 
> It's now consistent with how 'thing-at-point' works by default. (If you 
> have two consecutive URLs and point is between them - only possible with 
> the custom provider function, I think - it'll prefer the second one.)

I think you can install this now.

Re: Adding custom providers for thingatpt.el (was: [PATCH] Add support for 'thing-at-point' to get URL at point)

[Jim Porter]

On 5/18/2024 1:26 AM, Eli Zaretskii wrote:
> I think you can install this now.

Thanks. Merged as ae9045a8bd8.

Ihor, I'll update the Org-mode part of this next and post the new patch 
for that to the Org list once it's ready.