bug#21391: 24.5; `thing-at-point' should return a string

bug#21391: 24.5; `thing-at-point' should return a string

[Tino Calancha]

Drew Adams <drew.adams@oracle.com> writes:

> `thing-at-point' should always return a string.
> `list-at-point', `form-at-point', etc. are a different
> story - they can return anything.
>
> The problem with the code of `thing-at-point' is not where
> it has been identified so far.  The problem is that if
> THING has property `thing-at-point', and if that function
> returns something that is not a string, then
> `set-text-properties' raises an error.
>
> A proper fix is to convert the result returned by
> (funcall (get thing 'thing-at-point)) to a string.
> For that, you can use (format "%s" thing).

It's natural if `thing-at-point' returns the thing as a string.
When the user don't want a string then s?he can use the specific
functions, like `number-at-point' or `list-at-point'.

Indeed, that is what is specified in the manual:
* doc/lispref/text.texi (Examining Buffer Contents)

Following patch pass all the Emacs tests:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
From ce1ced756c2762b4b0448a0480173e757804f784 Mon Sep 17 00:00:00 2001
From: Tino Calancha <tino.calancha@gmail.com>
Date: Mon, 7 Nov 2016 12:05:28 +0900
Subject: [PATCH] * lisp/thingatpt.el (thing-at-point): Return the thing as a
 string

---
 lisp/thingatpt.el | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lisp/thingatpt.el b/lisp/thingatpt.el
index e423630..fd6b323 100644
--- a/lisp/thingatpt.el
+++ b/lisp/thingatpt.el
@@ -128,7 +128,7 @@ bounds-of-thing-at-point
 
 ;;;###autoload
 (defun thing-at-point (thing &optional no-properties)
-  "Return the THING at point.
+  "Return the THING at point as a string.
 THING should be a symbol specifying a type of syntactic entity.
 Possibilities include `symbol', `list', `sexp', `defun',
 `filename', `url', `email', `word', `sentence', `whitespace',
@@ -145,7 +145,9 @@ thing-at-point
            (let ((bounds (bounds-of-thing-at-point thing)))
              (when bounds
                (buffer-substring (car bounds) (cdr bounds)))))))
-    (when (and text no-properties (sequencep text))
+    (when (and text (not (stringp text)))
+      (setq text (format "%s" text)))
+    (when (and text no-properties)
       (set-text-properties 0 (length text) nil text))
     text))
 
-- 
2.10.1

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
In GNU Emacs 26.0.50.6 (x86_64-pc-linux-gnu, GTK+ Version 3.22.2)
 of 2016-11-06
Repository revision: f1d19d1445a8e7d4ee0d13edb8ed99e222603086

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> CC: Drew Adams <drew.adams@oracle.com>, Tino Calancha <tino.calancha@gmail.com>,Eli Zaretskii <eliz@gnu.org>
> Date: Mon, 07 Nov 2016 12:17:17 +0900
> 
> Drew Adams <drew.adams@oracle.com> writes:
> 
> > A proper fix is to convert the result returned by
> > (funcall (get thing 'thing-at-point)) to a string.
> > For that, you can use (format "%s" thing).
> 
> It's natural if `thing-at-point' returns the thing as a string.
> When the user don't want a string then s?he can use the specific
> functions, like `number-at-point' or `list-at-point'.

Isn't that completely backward-incompatible?  If so, I don't think we
can do that.

I don't really see what is "improper" with the other suggestions to
fix this bug, which simply avoid signaling an error if the "thing at
point" happens to be something other than a string?

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> > > A proper fix is to convert the result returned by
> > > (funcall (get thing 'thing-at-point)) to a string.
> > > For that, you can use (format "%s" thing).
> >
> > It's natural if `thing-at-point' returns the thing as a string.
> > When the user don't want a string then s?he can use the specific
> > functions, like `number-at-point' or `list-at-point'.
> 
> Isn't that completely backward-incompatible?  If so, I don't think
> we can do that.
> 
> I don't really see what is "improper" with the other suggestions to
> fix this bug, which simply avoid signaling an error if the "thing at
> point" happens to be something other than a string?

`thing-at-point' has always allowed, and rightfully so, its default
behavior to be replaced by putting a function on the THING symbol as
property `thing-at-point'.  That function is invoked, and what the
return value was returned by `thing-at-point'.

The two branches of the `if' in `thing-at-point' were thus not
symmetrical (and intentionally so, so that it could return actual
things, not just strings naming things).  One branch could return
anything - a THING at point; the other branch returned a string
naming a thing at point.

This meant that `thing-at-point' could return a list, symbol, etc.
- things that are not strings.

What is new (since Emacs 24.4) is that someone added this at the end
of the definition of `thing-at-point', and made it pertain to _both_
branches of the `if'.  In addition, they named the result of the `if'
by the variable `text'.

(when (and text no-properties)
  (set-text-properties 0 (length text) nil text))

They apparently paid no attention to the important `thing-at-point'
behavior provided by the first `if' clause, and instead just assumed
that the `if' always returned a string.

IIUC, Emacs Dev (not I), has insisted that `thing-at-point' itself
should always return a string, as opposed to what `form-at-point'
(and functions defined using it) returns, which can be a THING (not
just a string naming a thing).  For one thing, this allows code
calling `thing-at-point' to count on the result being a string (if
a thing is found) or nil (if not).

I see three possibly reasonable fixes for the bug introduced in
Emacs 24.4:

1. Make `thing-at-point' always return a thing, as (IIUC) wished
   by Emacs Dev.  Have it convert the result of the first `if'
   clause to a string.  Anyone needing a real (non-string) thing
   can use `form-at-point' to get what they want.

2. Make `thing-at-point', as before, return just what the firat
   `if' clause returns, if that clause is taken.  IOW, move the
   removal of text properties (from non-nil NO-PROPERTIES) into
   the second `if' clause.  Among other things (sic), this allows
   the first `if' clause to return a propertized string.

3. Make `thing-at-point', as before, return just what the firat
   `if' clause returns, if that clause is taken, except that if
   either clause returns a string then strip that string of text
   properties (if NO-PROPERTIES is non-nil).

My understanding is that Emacs Dev has preferred that
`thing-at-point' return a string, and that users should use
`form-at-point' if they want something else.  (I've adapted my
own code to that preference.)

If so, then #1 is probably the option of choice: Convert anything
returned by (funcall (get thing 'thing-at-point)) to a string.

Personally, I do not see why Emacs added arg NO-PROPERTIES, as
anyone could always remove properties on a string result from
`thing-at-point'.  Dunno who added this arg, or why, but I
don't see that it provides any advantage.

(Perhaps the one advantage it has inadvertently provided is to
raise the question again of whether `thing-at-point' should
always return a string.)

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Date: Mon, 7 Nov 2016 08:10:34 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 21391@debbugs.gnu.org, drew.adams@oracle.com
> 
> What is new (since Emacs 24.4) is that someone added this at the end
> of the definition of `thing-at-point', and made it pertain to _both_
> branches of the `if'.  In addition, they named the result of the `if'
> by the variable `text'.
> 
> (when (and text no-properties)
>   (set-text-properties 0 (length text) nil text))
> 
> They apparently paid no attention to the important `thing-at-point'
> behavior provided by the first `if' clause, and instead just assumed
> that the `if' always returned a string.

This is already fixed in the current sources.  We now only remove the
text properties if the object supports that.  Why isn't that enough?

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> This is already fixed in the current sources.  We now only remove
> the text properties if the object supports that.  Why isn't that enough?

My understanding was, as I said, that Emacs Dev wanted `t-a-p' to
always return a string (or nil), and that `form-at-point' is to be
used to return something else.  The advantage of that, I suppose,
was considered to be that callers could count on the return type,
without regard to the THING arg.

I don't have emacs-devel or bug-report messages to hand, to
support my understanding, but I thought that was the point of view
taken by those deciding, so far, wrt thingatpt.el.  I know that I
made an effort to adjust my code, to accommodate that approach.

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 07.11.2016 17:14, Eli Zaretskii wrote:

>> It's natural if `thing-at-point' returns the thing as a string.
>> When the user don't want a string then s?he can use the specific
>> functions, like `number-at-point' or `list-at-point'.
>
> Isn't that completely backward-incompatible?

When thing is a number? Maybe. I don't imagine it's used very often.

> If so, I don't think we
> can do that.

It's pretty clearly a bug: most other "things" behave otherwise (list, 
sexp, symbol). I'd very much rather we didn't retain conceptual problems 
like that just for the sake of compatibility.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Cc: 21391@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 7 Nov 2016 21:12:07 +0200
> 
> On 07.11.2016 17:14, Eli Zaretskii wrote:
> 
> >> It's natural if `thing-at-point' returns the thing as a string.
> >> When the user don't want a string then s?he can use the specific
> >> functions, like `number-at-point' or `list-at-point'.
> >
> > Isn't that completely backward-incompatible?
> 
> When thing is a number? Maybe. I don't imagine it's used very often.

But previously it could be used.  After this change, it would be
impossible.

> > If so, I don't think we
> > can do that.
> 
> It's pretty clearly a bug: most other "things" behave otherwise (list, 
> sexp, symbol). I'd very much rather we didn't retain conceptual problems 
> like that just for the sake of compatibility.

Not sure which behavior you have in mind.  The removal of properties?
Or something else?

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 07.11.2016 21:47, Eli Zaretskii wrote:

>> When thing is a number? Maybe. I don't imagine it's used very often.
>
> But previously it could be used.  After this change, it would be
> impossible.

The callers will need to switch to `number-at-point', probably. The 
resulting code will work with both new and older versions of Emacs.

>> It's pretty clearly a bug: most other "things" behave otherwise (list,
>> sexp, symbol). I'd very much rather we didn't retain conceptual problems
>> like that just for the sake of compatibility.
>
> Not sure which behavior you have in mind.  The removal of properties?
> Or something else?

Returning something other than a string or nil from thing-at-point.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Cc: tino.calancha@gmail.com, 21391@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 7 Nov 2016 21:51:23 +0200
> 
> >> It's pretty clearly a bug: most other "things" behave otherwise (list,
> >> sexp, symbol). I'd very much rather we didn't retain conceptual problems
> >> like that just for the sake of compatibility.
> >
> > Not sure which behavior you have in mind.  The removal of properties?
> > Or something else?
> 
> Returning something other than a string or nil from thing-at-point.

Didn't the function behave that way for quite some time?  If so, how
come it's suddenly a problem?

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 07.11.2016 22:02, Eli Zaretskii wrote:

> Didn't the function behave that way for quite some time?  If so, how
> come it's suddenly a problem?

Someone tried to use it for the first time?

bug#21391: 24.5; `thing-at-point' should return a string

[Andreas Röhler]

On 07.11.2016 21:02, Eli Zaretskii wrote:
>> Cc: tino.calancha@gmail.com, 21391@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Mon, 7 Nov 2016 21:51:23 +0200
>>
>>>> It's pretty clearly a bug: most other "things" behave otherwise (list,
>>>> sexp, symbol). I'd very much rather we didn't retain conceptual problems
>>>> like that just for the sake of compatibility.
>>> Not sure which behavior you have in mind.  The removal of properties?
>>> Or something else?
>> Returning something other than a string or nil from thing-at-point.
> Didn't the function behave that way for quite some time?  If so, how
> come it's suddenly a problem?
>
>
>

Assume a lot of bugs open like #21391.
These library, which is excellent in base, needs a cleanup, resp. 
reconsideration.

bug#21391: 24.5; `thing-at-point' should return a string

[Tino Calancha]

On Mon, 7 Nov 2016, Dmitry Gutov wrote:

> On 07.11.2016 22:02, Eli Zaretskii wrote:
>
>> Didn't the function behave that way for quite some time?  If so, how
>> come it's suddenly a problem?
>
> Someone tried to use it for the first time?
I started to use it ~ 1 year ago.

I want to recall that the manual says that this function
returns _always_ a string:
Return the @var{thing} around or next to point, as a string.
(see at: doc/lispref/text.texi (Buffer Contents)

But as you know, that is not true if
(get thing 'thing-at-point)

returns a non-nil value.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Cc: tino.calancha@gmail.com, 21391@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Mon, 7 Nov 2016 22:03:58 +0200
> 
>     Didn't the function behave that way for quite some time?  If so, how
>     come it's suddenly a problem?
> 
> Someone tried to use it for the first time?

I'm not sure what issue we are discussing and what problem we are
trying to solve.  Let me take a step back and describe the situation
as I see it.

This bug report started because thing-at-point would signal an error
for some of its calls.  That bug is already fixed.  Tino was unhappy
with using sequencep and wanted to replace that with stringp -- fine
with me, I don't object to such a change.

From my POV the issue should be closed once we agree on the predicate
to use when deciding whether or not stripping the text properties is
appropriate.

But then somehow the discussion shifted to be about whether to _force_
thing-at-point value to be a string, even if it isn't for some reason.
I don't understand the rationale for such a change.  Yes,
thing-at-point was most probably always meant to return a string.
Yes, Lisp code that causes it to return some other object is probably
wrong.  However, the change that allowed this was introduced 19 years
ago; who knows what code out there actually uses this loophole?  If
there is such code, why would we want to break it?  To what end?  And
if no code uses this loophole, why do we care that it exists?

IOW, thing-at-point no longer has any known bugs, and we are talking
about forcibly breaking a use case that does no harm to us, and can
only happen if someone abuses the 'thing-at-point' property, which
would make it that someone's bug/misfeature, for them to fix.

So why would we want to make such a change?  What am I missing?

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> I'm not sure what issue we are discussing and what problem we are
> trying to solve.  Let me take a step back and describe the situation
> as I see it.
> 
> This bug report started because thing-at-point would signal an error
> for some of its calls.  That bug is already fixed.  Tino was unhappy
> with using sequencep and wanted to replace that with stringp -- fine
> with me, I don't object to such a change.
> 
> From my POV the issue should be closed once we agree on the
> predicate
> to use when deciding whether or not stripping the text properties is
> appropriate.
> 
> But then somehow the discussion shifted to be about whether to
> _force_ thing-at-point value to be a string, even if it isn't
> for some reason.
>
> I don't understand the rationale for such a change.  Yes,
> thing-at-point was most probably always meant to return a string.
> Yes, Lisp code that causes it to return some other object is
> probably wrong.  However, the change that allowed this was
> introduced 19 years ago; who knows what code out there actually
> uses this loophole? If there is such code, why would we want to
> break it?  To what end? And if no code uses this loophole, why
> do we care that it exists?
> 
> IOW, thing-at-point no longer has any known bugs, and we are talking
> about forcibly breaking a use case that does no harm to us, and can
> only happen if someone abuses the 'thing-at-point' property, which
> would make it that someone's bug/misfeature, for them to fix.
> 
> So why would we want to make such a change?  What am I missing?

FWIW, I agree with you here, in general.

(I do not agree that thingatpt no longer has any known bugs,
however.  See bug #9300, which is a very important bug, to me.)

On the other hand, I'm pretty sure that some Emacs Dev deciders
have previously proclaimed that it is wrong for someone to exploit
this "loophole" and thus have `t-a-p' return something other than
a string.  And, as Tino points out, the doc says clearly that it
returns a string - no ifs, ands, or buts about that.

So yes, enforcing returning a string would be an incompatible
change.  But it would (apparently) do what Emacs Dev has said the
function should do, and it would do what the doc says it does
((elisp) `Buffer Contents').

Yes, deciding to enforce returning a string is outside the error
that was reported in this bug report.  On the other hand, the bug
title is precisely, explicitly this question we are discussing now:
Should `t-a-p' return a string?

I can work with this either way.  Long ago, I adapted my own code
to fit the prescription that `t-a-p' must return a string.  But I
guess I could make an incompatible change to my code, to once again
realign it with allowing a non-string return value.

My preference, at least so far, would be for us to do the following:

1. Enforce a string return value, in the way suggested.

2. CLEARLY point out, in the high-level doc, that returning a
   THING at point can mean two different things:

   * For `thing-at-point' it means return text (a string) that
     names/represents a THING.

   * For `form-at-point' and its callers (e.g., `symbol-at-point',
     `list-at-point', `sexp-at-point', `number-at-point'), it
     means return a THING, not its name.  That is, it returns an
     Emacs-Lisp entity - any kind of value that Lisp can return.

If we do NOT go down this road, but we instead stick with what is
there now (i.e., we do not enforce a string value for `t-a-p')
then I think it is imperative that we point out, in the doc,
that `t-a-p', like `form-at-point' and its callers, can return
a Lisp THING of any kind OR a string naming such a THING, and
that the former case is realized via property `thing-at-point'
on the THING-argument symbol.

Being clear and explicit about what it does and how will help
avoid the kinds of ambiguity that have plagued this function
for a long time now.

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 08.11.2016 17:05, Eli Zaretskii wrote:

> But then somehow the discussion shifted to be about whether to _force_
> thing-at-point value to be a string, even if it isn't for some reason.

I'd suggest trying to fix that from the other end, as one alternative. 
If we agree that the return value of thing-at-point should be a string, 
(get 'number 'thing-at-point) can't return `number-at-point', it should 
return a function that will return the said number as a string.

And of all things enumerated in thing-at-point's docstring, IIUC only 
number has such problem. Which leaves third-party things, but, they will 
either need to be fixed, or people will have to remain content not to 
use thing-at-point with NO-PROPERTIES argument on them.

> I don't understand the rationale for such a change.  Yes,
> thing-at-point was most probably always meant to return a string.
> Yes, Lisp code that causes it to return some other object is probably
> wrong.  However, the change that allowed this was introduced 19 years
> ago; who knows what code out there actually uses this loophole?

I can't imagine that loophole to be too useful. The proposal above 
should leave all such uses of third-party things alone. But yes, anyone 
who uses (thing-at-point 'number) as a (longer, pointless) substitute 
for (number-at-point) will have to change their code.

> If
> there is such code, why would we want to break it?  To what end?  And
> if no code uses this loophole, why do we care that it exists?

To make thing-at-point behavior more consistent. This kind of thing is 
usually performed with future new uses in mind. But it also might help 
with code maintenance a bit, for existing users (not likely to make much 
of a difference, but still; the danger of breakage is not very 
significant either).

> IOW, thing-at-point no longer has any known bugs, and we are talking
> about forcibly breaking a use case that does no harm to us, and can
> only happen if someone abuses the 'thing-at-point' property, which
> would make it that someone's bug/misfeature, for them to fix.

Yes. The fix is very easy, though, for projects that retain at least 
somewhat active maintainer.

bug#21391: 24.5; `thing-at-point' should return a string

[Andreas Röhler]

On 09.11.2016 01:04, Dmitry Gutov wrote:
> On 08.11.2016 17:05, Eli Zaretskii wrote:
>
>> But then somehow the discussion shifted to be about whether to _force_
>> thing-at-point value to be a string, even if it isn't for some reason.
>
> I'd suggest trying to fix that from the other end, as one alternative. 
> If we agree that the return value of thing-at-point should be a 
> string, (get 'number 'thing-at-point) can't return `number-at-point', 
> it should return a function that will return the said number as a string.

It might be worth noticing that result of thing-at-point natively is a 
string. In case these string is not returned for now, additional 
functions are called upon. For example "intern" in case of 
symbol-at-point, which makes me frown. ;)

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Cc: tino.calancha@gmail.com, 21391@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 9 Nov 2016 02:04:20 +0200
> 
> On 08.11.2016 17:05, Eli Zaretskii wrote:
> 
> > But then somehow the discussion shifted to be about whether to _force_
> > thing-at-point value to be a string, even if it isn't for some reason.
> 
> I'd suggest trying to fix that from the other end, as one alternative. 
> If we agree that the return value of thing-at-point should be a string, 
> (get 'number 'thing-at-point) can't return `number-at-point', it should 
> return a function that will return the said number as a string.
> 
> And of all things enumerated in thing-at-point's docstring, IIUC only 
> number has such problem. Which leaves third-party things, but, they will 
> either need to be fixed, or people will have to remain content not to 
> use thing-at-point with NO-PROPERTIES argument on them.

I don't think I understand what you are suggesting.  Can you show a
proposed patch, so I could see the light?

> > If
> > there is such code, why would we want to break it?  To what end?  And
> > if no code uses this loophole, why do we care that it exists?
> 
> To make thing-at-point behavior more consistent.

It is consistent now.  The only way to make it inconsistent is to have
a 'thing-at-point' property that violates that, but we never do that
in Emacs proper, so if someone else does that, it would be their bug.

> > IOW, thing-at-point no longer has any known bugs, and we are talking
> > about forcibly breaking a use case that does no harm to us, and can
> > only happen if someone abuses the 'thing-at-point' property, which
> > would make it that someone's bug/misfeature, for them to fix.
> 
> Yes. The fix is very easy, though, for projects that retain at least 
> somewhat active maintainer.

I might agree when I see a concrete proposal.

Thanks.

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> If we agree that the return value of thing-at-point should be a
> string, (get 'number 'thing-at-point) can't return `number-at-point', it
> should return a function that will return the said number as a string.

Oh, God, no.  That would be among the most misguided proposals
I've heard in a long time.  That truly throws out the baby with
the bathwater.

It is not property `thing-at-point' that should return (a function
that returns) a string; it is function `thing-at-point' that should
return a string - if anything should.

> And of all things enumerated in thing-at-point's docstring,

That list is only an indication of some of what is possible.

> IIUC only number has such problem. 

It has no problem that I'm aware of.  What's the problem?
`number-at-point' returns a number, as it should.
`list-at-point' returns a list, as it should, and so on.

(get 'number 'thing-at-point) returns `number-at-point', which
is 100% reasonable.

> Which leaves third-party things, but, they will either need
> to be fixed,

I get the impression that your idea of what is broken and needs
to be fixed is very different from mine.

> or people will have to remain content not to use thing-at-point
> with NO-PROPERTIES argument on them.

What is "them"?

What Eli has suggested is one reasonable approach: just prevent
raising the stupid error.  I proposed at least two other
reasonable approaches.

In none of those suggestions is there a problem with calling
`thing-at-point' on ANY kind of THING while using non-nil
NO-PROPERTIES.

It is not clear what you are seeing as a problem, for which
you think that requiring or prescribing that property
`thing-at-point' return a (function that returns a) string is
the solution.

> I can't imagine that loophole to be too useful. The proposal above
> should leave all such uses of third-party things alone. But yes,
> anyone who uses (thing-at-point 'number) as a (longer, pointless)
> substitute for (number-at-point) will have to change their code.

Not wanting someone to use (thing-at-point 'number) to return
a number via (get 'number 'thing-at-point) is one thing.
And it is already addressed by what I proposed.

Not wanting to let (get 'number 'thing-at-point) to return a
function that returns a number is quite another thing altogether.
And it is totally uncalled for. 

> To make thing-at-point behavior more consistent. This kind of thing
> is usually performed with future new uses in mind. But it also might
> help with code maintenance a bit, for existing users (not likely to make
> much of a difference, but still; the danger of breakage is not very
> significant either).

I really disagree with what seems to be your view of thingatpt.el
functionality.

I disagreed with it strongly in the context of bug #9300, where you
were the lone voice proclaiming it.  You apparently view the only
use of this functionality as returning a string near (not at) point
that can be used as an input default value.  thingatpt.el is much
more than that.  It is very important that it be able to be used
to test whether there (really) is a given THING _at_ point (not
just somewhere near point).

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1296 bytes --]

On 09.11.2016 17:45, Eli Zaretskii wrote:

> I don't think I understand what you are suggesting.  Can you show a
> proposed patch, so I could see the light?

See the attached patch.

Or to take a step further, we might want to deprecate the 
`thing-at-point' property, and recommend to only use the 
`bounds-of-thing-at-point' property. This way, we get the string-ness 
guarantee automatically, and the bounds-of-thing-at-point function will 
work for all things (it currently fails for `number').

>>> If
>>> there is such code, why would we want to break it?  To what end?  And
>>> if no code uses this loophole, why do we care that it exists?
>>
>> To make thing-at-point behavior more consistent.
>
> It is consistent now.

Put point on a number. Number is a sexp. Not all sexps are strings.

(thing-at-point 'number) => 123
(thing-at-point 'sexp) => "123"

That doesn't looks consistent to me. And there's no way to guess the 
return value type in advance without knowing which exact function is the 
thing's `thing-at-point' property.

> The only way to make it inconsistent is to have
> a 'thing-at-point' property that violates that, but we never do that
> in Emacs proper, so if someone else does that, it would be their bug.

number's `thing-at-point' property is not like the others.

[-- Attachment #2: thingatpt-number.diff --]
[-- Type: text/x-diff, Size: 1579 bytes --]

diff --git a/lisp/thingatpt.el b/lisp/thingatpt.el
index 9920fa0..1c630f7 100644
--- a/lisp/thingatpt.el
+++ b/lisp/thingatpt.el
@@ -128,7 +128,7 @@ bounds-of-thing-at-point
 
 ;;;###autoload
 (defun thing-at-point (thing &optional no-properties)
-  "Return the THING at point.
+  "Return the THING at point as a string.
 THING should be a symbol specifying a type of syntactic entity.
 Possibilities include `symbol', `list', `sexp', `defun',
 `filename', `url', `email', `word', `sentence', `whitespace',
@@ -145,7 +145,7 @@ thing-at-point
            (let ((bounds (bounds-of-thing-at-point thing)))
              (when bounds
                (buffer-substring (car bounds) (cdr bounds)))))))
-    (when (and text no-properties (sequencep text))
+    (when (and text no-properties)
       (set-text-properties 0 (length text) nil text))
     text))
 
@@ -557,6 +557,14 @@ thing-at-point-email-regexp
 (put 'buffer 'end-op (lambda () (goto-char (point-max))))
 (put 'buffer 'beginning-op (lambda () (goto-char (point-min))))
 
+;;  Number
+
+(put 'number 'thing-at-point
+     (lambda ()
+       (let ((thing (thing-at-point 'sexp)))
+         (when (numberp (thing-at-point--read-from-whole-string thing))
+           thing))))
+
 ;;  Aliases
 
 (defun word-at-point ()
@@ -604,7 +612,6 @@ symbol-at-point
 (defun number-at-point ()
   "Return the number at point, or nil if none is found."
   (form-at-point 'sexp 'numberp))
-(put 'number 'thing-at-point 'number-at-point)
 ;;;###autoload
 (defun list-at-point ()
   "Return the Lisp list at point, or nil if none is found."

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 09.11.2016 19:58, Drew Adams wrote:

> It has no problem that I'm aware of.  What's the problem?
> `number-at-point' returns a number, as it should.
> `list-at-point' returns a list, as it should, and so on.

No problem there.

> (get 'number 'thing-at-point) returns `number-at-point', which
> is 100% reasonable.

Not at all. As I've explained previously.

>> or people will have to remain content not to use thing-at-point
>> with NO-PROPERTIES argument on them.
>
> What is "them"?

The third-party things.

> Not wanting someone to use (thing-at-point 'number) to return
> a number via (get 'number 'thing-at-point) is one thing.
> And it is already addressed by what I proposed.

I must have missed that proposal.

> Not wanting to let (get 'number 'thing-at-point) to return a
> function that returns a number is quite another thing altogether.
> And it is totally uncalled for.

Isn't that one and the same?

> I disagreed with it strongly in the context of bug #9300, where you
> were the lone voice proclaiming it.  You apparently view the only
> use of this functionality as returning a string near (not at) point
> that can be used as an input default value.  thingatpt.el is much
> more than that.  It is very important that it be able to be used
> to test whether there (really) is a given THING _at_ point (not
> just somewhere near point).

That seems orthogonal to the current discussion.

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> > (get 'number 'thing-at-point) returns `number-at-point', which
> > is 100% reasonable.
> 
> Not at all. As I've explained previously.

No, you have not.  This is the whole point of symbol property
`thing-at-point', and it always has been.

What do YOU think is the point of putting property `thing-at-point'
on a THING type?

> > Not wanting someone to use (thing-at-point 'number) to return
> > a number via (get 'number 'thing-at-point) is one thing.
> > And it is already addressed by what I proposed.
> 
> I must have missed that proposal.

I showed three possibilities, and I pointed out which one I prefer.
The bug thread is there, in case you really missed it.

> > Not wanting to let (get 'number 'thing-at-point) to return a
> > function that returns a number is quite another thing altogether.
> > And it is totally uncalled for.
> 
> Isn't that one and the same?

Of course not.  The use of (get 'number 'thing-at-point) in the
definition of `thing-at-point' is only _one_ use of it.  That's
apparently what you've missed from the beginning: the point of
symbol property `thing-at-point'.

You will see things like this in thingatpt.el:

(put 'url 'thing-at-point 'thing-at-point-url-at-point)

(put 'email 'thing-at-point
     (lambda ()
       (let ((boundary-pair (bounds-of-thing-at-point 'email)))
         (if boundary-pair
             (buffer-substring-no-properties
              (car boundary-pair) (cdr boundary-pair))))))

(put 'number 'thing-at-point 'number-at-point)

And that's only a few predefined uses of it.  One of the main
points of the library is to provide tools that you can use to
define your functions that implement providing a THING at point.
And by that I do not mean just a string at point that names a
thing - I mean a Lisp THING (list, sexp, number, color name,
file name,... whatever).

It is irrelevant to the design of thingatpt.el, and its intended
use cases, that the THINGs defined for `url' and `email' are
string values.  They are no more "normal" cases for the design
than is `number'.  What is important is that the design provides
this functionality for _any_ kind of THING - any Lisp value.

Examples of other THING-returning definitions:

(put 'list 'thing-at-point 'my-list-at-point)
(put 'unquoted-list 'thing-at-point 'my-unquoted-list-at-point)
(put 'non-nil-symbol-name 'thing-at-point 'my-non-nil-symbol-name-at-point)
(put 'symbol-name 'thing-at-point 'my-symbol-name-at-point)
(put 'region-or-word 'thing-at-point 'my-region-or-word-at-point)
(put 'color 'thing-at-point 'my-color-at-point)
(put 'decimal-number 'thing-at-point 'my-number-at-point-decimal)
(put 'hex-number 'thing-at-point 'my-number-at-point-hex)
(put 'string 'thing-at-point 'my-string-at-point)

Some of those THING types are strings; some are lists; some are
symbols; some are numbers.  The THING possibilities are limitless.

Practically the entire design of thingatpt.el revolves around
putting properties on THING-type symbols and function symbols.
This includes property `thing-at-point'.

> > I disagreed with it strongly in the context of bug #9300, where
> > you were the lone voice proclaiming it.  You apparently view the only
> > use of this functionality as returning a string near (not at)
> > point that can be used as an input default value.  thingatpt.el is much
> > more than that.  It is very important that it be able to be used
> > to test whether there (really) is a given THING _at_ point (not
> > just somewhere near point).
> 
> That seems orthogonal to the current discussion.

It points to the fact that you apparently have only a limited view
of what thingatpt.el is for and what features it provides.  You've
missed two for two, so far - and important ones, at that.

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 10.11.2016 02:40, Drew Adams wrote:

> What do YOU think is the point of putting property `thing-at-point'
> on a THING type?

Being lazy or unable to make up your mind. Or maybe some kind of 
micro-optimization.

> You will see things like this in thingatpt.el:
>
> (put 'url 'thing-at-point 'thing-at-point-url-at-point)
>
> (put 'email 'thing-at-point
>      (lambda ()
>        (let ((boundary-pair (bounds-of-thing-at-point 'email)))
>          (if boundary-pair
>              (buffer-substring-no-properties
>               (car boundary-pair) (cdr boundary-pair))))))

Try removing these two in your local Emacs, save and restart (maybe do a 
rebuild as well, to be sure).

(thing-at-point 'url) and (thing-at-point 'email) will continue to work 
fine.

> (put 'number 'thing-at-point 'number-at-point)
>
> And that's only a few predefined uses of it.  One of the main
> points of the library is to provide tools that you can use to
> define your functions that implement providing a THING at point.

Search my messages for the word "third-party".

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Cc: tino.calancha@gmail.com, 21391@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 10 Nov 2016 01:30:20 +0200
> 
> See the attached patch.
> 
> Or to take a step further, we might want to deprecate the 
> `thing-at-point' property, and recommend to only use the 
> `bounds-of-thing-at-point' property. This way, we get the string-ness 
> guarantee automatically, and the bounds-of-thing-at-point function will 
> work for all things (it currently fails for `number').
> 
> >>> If
> >>> there is such code, why would we want to break it?  To what end?  And
> >>> if no code uses this loophole, why do we care that it exists?
> >>
> >> To make thing-at-point behavior more consistent.
> >
> > It is consistent now.
> 
> Put point on a number. Number is a sexp. Not all sexps are strings.
> 
> (thing-at-point 'number) => 123
> (thing-at-point 'sexp) => "123"
> 
> That doesn't looks consistent to me.

There's a tension here between consistency and backward compatibility.
And since this function was "inconsistent" for a very long time, I'm
not sure losing backward compatibility can be justified by consistency
at this point.

We'd also lose something else: some Lisp objects can be printed, but
their printed representation cannot be read back.  So for some
objects, requiring thing-at-point to return a string would lose
information.

> > The only way to make it inconsistent is to have
> > a 'thing-at-point' property that violates that, but we never do that
> > in Emacs proper, so if someone else does that, it would be their bug.
> 
> number's `thing-at-point' property is not like the others.

Right, I succeeded to forget that since the beginning of this thread.

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 10.11.2016 18:08, Eli Zaretskii wrote:

> There's a tension here between consistency and backward compatibility.
> And since this function was "inconsistent" for a very long time, I'm
> not sure losing backward compatibility can be justified by consistency
> at this point.

I believe I've touched on this already.

And another inconsistency:

(bounds-of-thing-at-point 'number) always returns nil.

> We'd also lose something else: some Lisp objects can be printed, but
> their printed representation cannot be read back.  So for some
> objects, requiring thing-at-point to return a string would lose
> information.

We won't lose that if we go with either of my proposals: instead of 
printing objects inside the thing-at-point function, we would require 
that each returned thing is a string already. Any thing-at-point 
function that returns a non-string will be considered non-conformant.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Cc: tino.calancha@gmail.com, 21391@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 10 Nov 2016 18:19:00 +0200
> 
> > We'd also lose something else: some Lisp objects can be printed, but
> > their printed representation cannot be read back.  So for some
> > objects, requiring thing-at-point to return a string would lose
> > information.
> 
> We won't lose that if we go with either of my proposals: instead of 
> printing objects inside the thing-at-point function, we would require 
> that each returned thing is a string already.

But that's exactly my point: you cannot usefully return some objects
as strings.  Their text representation is much less useful than the
object itself.

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 10.11.2016 19:13, Eli Zaretskii wrote:

> But that's exactly my point: you cannot usefully return some objects
> as strings.  Their text representation is much less useful than the
> object itself.

If the consumer needs the text representation, they will call

(thing-at-point 'foo)

If they need the "real thing", they will call

(foo-at-point)

which we really expect to be defined.

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> And another inconsistency:
> (bounds-of-thing-at-point 'number) always returns nil.

It's not an inconsistency.  It's a bug.

`thingatpt.el' HAS bugs.  I've provided patches and filed bug
reports, pointing to code that fixes things.  All of that has
been ignored.  I've been using the fixed code for years, and
so have others.

In my code I do this, and things work fine:

(put 'number 'tap-bounds-of-thing-at-point 'tap-bounds-of-number-at-point)

(defun tap-bounds-of-number-at-point ()
  "Return the bounds of the number represented by the numeral at point.
Return nil if none is found."
  (and (number-at-point)  (tap-bounds-of-thing-at-point 'sexp)))

`bounds-of-thing-at-point' is broken.  I've provided a fix for it, which
you've rejected summarily, based on your own perceived "need" to force
`thing-at-point' to not respect "at point" but try to return things that
are only near point - things that are either here or there - no "at".

> We won't lose that if we go with either of my proposals: instead of
> printing objects inside the thing-at-point function, we would
> require that each returned thing is a string already. Any thing-at-point
> function that returns a non-string will be considered non-conformant.

You want to throw out the baby with the bathwater.  You have no use
for what is truly useful in thingatpt.  You ignore the design and
insist on your own interpretation of what's "useless" or "inconsistent".

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Cc: tino.calancha@gmail.com, 21391@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 10 Nov 2016 19:28:17 +0200
> 
> On 10.11.2016 19:13, Eli Zaretskii wrote:
> 
> > But that's exactly my point: you cannot usefully return some objects
> > as strings.  Their text representation is much less useful than the
> > object itself.
> 
> If the consumer needs the text representation, they will call
> 
> (thing-at-point 'foo)
> 
> If they need the "real thing", they will call
> 
> (foo-at-point)
> 
> which we really expect to be defined.

Which is again entirely backward incompatible, right?

Sorry, this doesn't sound safe to me.

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 10.11.2016 20:49, Eli Zaretskii wrote:

>> If the consumer needs the text representation, they will call
>>
>> (thing-at-point 'foo)
>>
>> If they need the "real thing", they will call
>>
>> (foo-at-point)
>>
>> which we really expect to be defined.
>
> Which is again entirely backward incompatible, right?

Why do you think so? This is the normal usage of thingatpt.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Cc: tino.calancha@gmail.com, 21391@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Thu, 10 Nov 2016 20:55:44 +0200
> 
> On 10.11.2016 20:49, Eli Zaretskii wrote:
> 
> >> If the consumer needs the text representation, they will call
> >>
> >> (thing-at-point 'foo)
> >>
> >> If they need the "real thing", they will call
> >>
> >> (foo-at-point)
> >>
> >> which we really expect to be defined.
> >
> > Which is again entirely backward incompatible, right?
> 
> Why do you think so?

Because currently they can call thing-at-point without defining
foo-at-point.  AFAIU, you suggest to deny them this possibility, and
force them to define foo-at-point for every foo.

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 10.11.2016 21:59, Eli Zaretskii wrote:

> Because currently they can call thing-at-point without defining
> foo-at-point.  AFAIU, you suggest to deny them this possibility, and
> force them to define foo-at-point for every foo.

They'll really have to do that only for string-unrepresentable foos.

Like currently (thing-at-point 'symbol) returns a string, whereas 
(symbol-at-point) returns an interned symbol. That's nothing new.

bug#21391: 24.5; `thing-at-point' should return a string

[Tino Calancha]

On Thu, 10 Nov 2016, Dmitry Gutov wrote:

> On 10.11.2016 21:59, Eli Zaretskii wrote:
>
>> Because currently they can call thing-at-point without defining
>> foo-at-point.  AFAIU, you suggest to deny them this possibility, and
>> force them to define foo-at-point for every foo.
>
> They'll really have to do that only for string-unrepresentable foos.
>
> Like currently (thing-at-point 'symbol) returns a string, whereas 
> (symbol-at-point) returns an interned symbol. That's nothing new.
Indeed, this sounds the right usage for me, and for an user who has 
read the part of the manual refering to `thing-at-point'.

If i want the list object i call `list-at-point', and if i just want the
text representation i would call (thing-at-point 'list).

The fact that we have that duality in the code, i.e.

I) (thing-at-point 'foo)
II) (foo-at-point)

in some sense is asking for I) and II) returning two different things.
I find it easier if I) always return a string.
Note also that we don't need to define one version II) per each THING: 
just in the few cases where it has sense to return the THING with a 
different type than a string.  We already have defined all II) functions
in `thingatpt.el'.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> Date: Fri, 11 Nov 2016 19:07:15 +0900 (JST)
> cc: Eli Zaretskii <eliz@gnu.org>, tino.calancha@gmail.com, 
>     21391@debbugs.gnu.org
> 
> > On 10.11.2016 21:59, Eli Zaretskii wrote:
> >
> >> Because currently they can call thing-at-point without defining
> >> foo-at-point.  AFAIU, you suggest to deny them this possibility, and
> >> force them to define foo-at-point for every foo.
> >
> > They'll really have to do that only for string-unrepresentable foos.
> >
> > Like currently (thing-at-point 'symbol) returns a string, whereas 
> > (symbol-at-point) returns an interned symbol. That's nothing new.
> Indeed, this sounds the right usage for me, and for an user who has 
> read the part of the manual refering to `thing-at-point'.

I disagree, sorry.

bug#21391: 24.5; `thing-at-point' should return a string

[Tino Calancha]

On Fri, 11 Nov 2016, Eli Zaretskii wrote:

>> From: Tino Calancha <tino.calancha@gmail.com>
>> Date: Fri, 11 Nov 2016 19:07:15 +0900 (JST)
>> cc: Eli Zaretskii <eliz@gnu.org>, tino.calancha@gmail.com,
>>     21391@debbugs.gnu.org
>>
>>> On 10.11.2016 21:59, Eli Zaretskii wrote:
>>>
>>>> Because currently they can call thing-at-point without defining
>>>> foo-at-point.  AFAIU, you suggest to deny them this possibility, and
>>>> force them to define foo-at-point for every foo.
>>>
>>> They'll really have to do that only for string-unrepresentable foos.
>>>
>>> Like currently (thing-at-point 'symbol) returns a string, whereas
>>> (symbol-at-point) returns an interned symbol. That's nothing new.
>> Indeed, this sounds the right usage for me, and for an user who has
>> read the part of the manual refering to `thing-at-point'.
>
> I disagree, sorry.
The current implementation makes the things harder to understand.
Currently one user need to read the source code to know the actual
type this function returns.

Depending of what is foo in:

(thing-at-point 'foo nil)

you could get:

1) a non-string
2) a string with properties
3) a string without properties.

That's true even after you pass nil to the second argument
(see examples below).  Then it's not clear to me the motivation
of the second argument: if non-nil means strip the properties, i guess
a nil value should means do not strip them.

Let's say i want to get some things as strings keeping their
properties (in this example warning face).

;; list: OK
(with-temp-buffer
   (insert (propertize "(foo)" 'face 'warning))
   (goto-char 2)
   (thing-at-point 'list))
=> #("(foo)" 0 5 (face warning))

;; sexp: OK
(with-temp-buffer
   (insert (propertize "(foo)" 'face 'warning))
   (goto-char 2)
   (thing-at-point 'sexp))
=> #("foo" 0 3 (face warning))

;; symbol: OK
(with-temp-buffer
   (insert (propertize "(foo)" 'face 'warning))
   (goto-char 2)
   (thing-at-point 'symbol))
=> #("foo" 0 3 (face warning))

;; filename: OK
(with-temp-buffer
   (insert (propertize "thingatpt.el" 'face 'warning))
   (goto-char 2)
   (thing-at-point 'filename))
=> #("thingatpt.el" 0 12 (face warning))

;; url: Opps! And the properties?
(with-temp-buffer
   (insert (propertize "https://www.example.com" 'face 'warning))
   (goto-char 2)
   (thing-at-point 'url))
=> "https://www.example.com"

;; email: Again No properties
(with-temp-buffer
   (insert (propertize "foo@bar.edu" 'face 'warning))
   (goto-char 2)
   (thing-at-point 'email))
=> "foo@bar.edu"

;; number: Cannot get even a string.
(with-temp-buffer
   (insert (propertize "4" 'face 'warning))
   (goto-char 1)
   (thing-at-point 'number))
=> 4

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> Date: Fri, 11 Nov 2016 23:13:55 +0900 (JST)
> cc: Tino Calancha <tino.calancha@gmail.com>, dgutov@yandex.ru, 
>     21391@debbugs.gnu.org
> 
> The current implementation makes the things harder to understand.
> Currently one user need to read the source code to know the actual
> type this function returns.

If that's a problem, let's improve the doc string (and the manual, if
needed) to make this more clear.

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 11.11.2016 16:35, Eli Zaretskii wrote:

>> The current implementation makes the things harder to understand.
>> Currently one user need to read the source code to know the actual
>> type this function returns.
>
> If that's a problem, let's improve the doc string (and the manual, if
> needed) to make this more clear.

Improve how? By saying that the function behaves in a variety of ways, 
solely depending on which `thing' it is passed?

That won't make it much clearer, or make it easier to follow code that 
uses `thing-at-point'.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Cc: 21391@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 11 Nov 2016 16:46:14 +0200
> 
> On 11.11.2016 16:35, Eli Zaretskii wrote:
> 
> >> The current implementation makes the things harder to understand.
> >> Currently one user need to read the source code to know the actual
> >> type this function returns.
> >
> > If that's a problem, let's improve the doc string (and the manual, if
> > needed) to make this more clear.
> 
> Improve how? By saying that the function behaves in a variety of ways, 
> solely depending on which `thing' it is passed?

Improve it by describing whatever it is that is evident from the code
that is not evident from the doc string.

> That won't make it much clearer, or make it easier to follow code that 
> uses `thing-at-point'.

I don't understand why.  It's not like we don't have other functions
that can return different kinds of objects.

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 11.11.2016 16:51, Eli Zaretskii wrote:

> Improve it by describing whatever it is that is evident from the code
> that is not evident from the doc string.

So we will enumerate how every built-in "thing" behaves? That still 
won't account for the third-party ones.

> I don't understand why.  It's not like we don't have other functions
> that can return different kinds of objects.

Because there is no rhyme or reason in the current behavior, no matter 
how you want to look at it.

E.g. symbols or sexps are not strings, and yet (thing-at-point 'symbol) 
and (thing-at-point 'sexp) return strings (or nil).

bug#21391: 24.5; `thing-at-point' should return a string

[Tino Calancha]

On Fri, 11 Nov 2016, Eli Zaretskii wrote:

>> Cc: 21391@debbugs.gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>> Date: Fri, 11 Nov 2016 16:46:14 +0200
>>
>> On 11.11.2016 16:35, Eli Zaretskii wrote:
>>
>>>> The current implementation makes the things harder to understand.
>>>> Currently one user need to read the source code to know the actual
>>>> type this function returns.
>>>
>>> If that's a problem, let's improve the doc string (and the manual, if
>>> needed) to make this more clear.
>>
>> Improve how? By saying that the function behaves in a variety of ways,
>> solely depending on which `thing' it is passed?
>
> Improve it by describing whatever it is that is evident from the code
> that is not evident from the doc string.
>
>> That won't make it much clearer, or make it easier to follow code that
>> uses `thing-at-point'.
>
> I don't understand why.  It's not like we don't have other functions
> that can return different kinds of objects.
Yes, and it's a good thing when it results in a simpler code.
I don't think in this case it makes a simpler code or a code
easier to follow, kinda the opposite.
I don't think we actually need a polymorphic function here with the 
bizarre behaviour pointed out in my previous email.  Instead of document
its counter-intuitive behaviour, i would go for fixing the function to do
just one obvious thing.  It would save the users time reading the manual.
It's a change for better, a good thing.  We will have a better 
function.  Everyone will be happy with this.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Cc: tino.calancha@gmail.com, 21391@debbugs.gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Fri, 11 Nov 2016 16:57:20 +0200
> 
> On 11.11.2016 16:51, Eli Zaretskii wrote:
> 
> > Improve it by describing whatever it is that is evident from the code
> > that is not evident from the doc string.
> 
> So we will enumerate how every built-in "thing" behaves?

No, we won't, not unless you and Tino say that's what you think should
be there.

> > I don't understand why.  It's not like we don't have other functions
> > that can return different kinds of objects.
> 
> Because there is no rhyme or reason in the current behavior, no matter 
> how you want to look at it.

The reason is that it existed in this form for the last 19 years.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> Date: Sat, 12 Nov 2016 00:05:56 +0900 (JST)
> cc: Dmitry Gutov <dgutov@yandex.ru>, tino.calancha@gmail.com, 
>     21391@debbugs.gnu.org
> 
> > I don't understand why.  It's not like we don't have other functions
> > that can return different kinds of objects.
> Yes, and it's a good thing when it results in a simpler code.
> I don't think in this case it makes a simpler code or a code
> easier to follow, kinda the opposite.

I can envision it being used to simplify code as well, but I don't
think this is the important point.

> I don't think we actually need a polymorphic function here with the 
> bizarre behaviour pointed out in my previous email.  Instead of document
> its counter-intuitive behaviour, i would go for fixing the function to do
> just one obvious thing.  It would save the users time reading the manual.
> It's a change for better, a good thing.  We will have a better 
> function.  Everyone will be happy with this.

I understand, but backward compatibility is more important in my eyes.

I can suggest adding a new function, with the features you mention.
We could even deprecate thing-at-point and advise to use the new one
instead.  But I don't see here a reason good enough to change a
long-standing behavior of thing-at-point in backward-incompatible
ways, sorry.

bug#21391: 24.5; `thing-at-point' should return a string

[Dmitry Gutov]

On 11.11.2016 17:37, Eli Zaretskii wrote:

> I can suggest adding a new function, with the features you mention.
> We could even deprecate thing-at-point and advise to use the new one
> instead.

In this vein, I would propose deprecating `thing-at-point' in favor of 
`bounds-of-thing-at-point', which should provide all the necessary 
information for a `buffer-substring' call anyway (when it works).

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> > I can suggest adding a new function, with the features you
> > mention.   We could even deprecate thing-at-point and advise
> > to use the new one instead.
> 
> In this vein, I would propose deprecating `thing-at-point' in favor
> of `bounds-of-thing-at-point', which should provide all the necessary
> information for a `buffer-substring' call anyway (when it works).

This is really going from bad to worse.  But I can't say I'm
surprised.

Eli suggested to keep the behavior backward-compatible, rather
than ensuring that the return value is a string.  That is a
reasonable approach.  It's OK by me.

It is the 2nd of the 3 approaches I described as reasonable 
(https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21391#52):

d> 2. Make `thing-at-point', as before, return just what the
      firat `if' clause returns, if that clause is taken.
      IOW, move the removal of text properties (from non-nil
      NO-PROPERTIES) into the second `if' clause.

Can we please just do that, and stop f__ing with thingatpt?

I also suggested that we do this, in that case:

d> Point out, in the doc, that `t-a-p', like `form-at-point'
   and its callers, can return a Lisp THING of any kind OR a
   string naming such a THING, and that the former case is
   realized via property `thing-at-point' on the THING-argument
   symbol.

That's important.  The use of symbol property `thing-at-point'
is one of the most important features of library `thingatpt.el'.
Pointing out `thing-at-point' in the doc without pointing out
this feature is letting users down.

Each of the 3 approaches I described is reasonable.  What is
not so reasonable are the kinds of changes you are suggesting
now.

If you are really entertaining removing existing functionality
then I would suggest you remove (deprecate) the NO-PROPERTIES
argument that was added fairly recently.  It is unnecessary,
and its addition was apparently completely gratuitous.
Did that action correspond to a bug fix or a user request?
I cannot imagine that it did.

Are we going to start adding a NO-PROPERTIES arg to _every_
function that can return a string?  If not, why does it make
sense here?  How hard is it to remove the properties of a
string?

bug#21391: 24.5; `thing-at-point' should return a string

[Andreas Röhler]

On 14.11.2016 03:43, Drew Adams wrote:
>>> I can suggest adding a new function, with the features you
>>> mention.   We could even deprecate thing-at-point and advise
>>> to use the new one instead.
>> In this vein, I would propose deprecating `thing-at-point' in favor
>> of `bounds-of-thing-at-point', which should provide all the necessary
>> information for a `buffer-substring' call anyway (when it works).
> This is really going from bad to worse.  But I can't say I'm
> surprised.
>
> Eli suggested to keep the behavior backward-compatible, rather
> than ensuring that the return value is a string.  That is a
> reasonable approach.  It's OK by me.

IMHO the current design is really confusing and shouldn't be kept.

Look at the start of thing-at-point implementation:

if (get thing 'thing-at-point)
              (funcall (get thing 'thing-at-point))

AFAIU an arbitrary function might  be stored here, no real relation to 
thing-at-point at all.

Next clause deals with buffer-substring - which is thing-at-point about 
in my understanding.

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> Look at the start of thing-at-point implementation:
> 
> if (get thing 'thing-at-point)
>    (funcall (get thing 'thing-at-point))
> 
> AFAIU an arbitrary function might  be stored here, no real
> relation to thing-at-point at all.

ANY time you use a function as a parameter (passed as an
argument, obtained as the value of a variable, or picked up
in some other way, such as here) there is little or no control
over the nature of that function in the function that makes
use of it.  It is up to the code that provides the function
to provide an appropriate function (DTRT).

> Next clause deals with buffer-substring - which is
> thing-at-point about in my understanding.

Your understanding is partial, then.  Thing-at-point has
been, from the outset, about actual Lisp THINGS, not just
strings that name such things.  Yes, string-naming-a-thing
is a common use case - the most common, currently.  But it
is only part of the story.

And yes, buffer text is usually examined to come up with
the returned THING.  But the buffer text need not be
examined.  What is most significant is point - the buffer
position.  And even that could be ignored by the function
that comes up with an appropriate THING to return.

thingatpt.el is much more general, and offers many more
possibilities, than just grabbing some text at or near
point.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Date: Sun, 13 Nov 2016 18:43:52 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 21391@debbugs.gnu.org
> 
> d> 2. Make `thing-at-point', as before, return just what the
>       firat `if' clause returns, if that clause is taken.
>       IOW, move the removal of text properties (from non-nil
>       NO-PROPERTIES) into the second `if' clause.

Why would we want to do that?  AFAIU, it would require the function
that is the value of the thing-at-point property to second-guess what
the caller of thing-at-point wants, something it has no means to do.

If thing-at-point is called with NO-PROPERTIES non-nil, and the
thing-at-point property returns a string, then any properties should
be removed from that string, exactly as in the other case.  I don't
see why we should single out that one use case.

I think the only change that makes sense at this point is to replace
sequencep by stringp, as Tino originally proposed.  Other than that,
there are no problems here that we need to solve.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> Date: Mon, 14 Nov 2016 11:26:54 +0100
> 
> IMHO the current design is really confusing and shouldn't be kept.

IMNSHO it would be madness to make incompatible changes 20 years after
the code was admitted into Emacs.

> if (get thing 'thing-at-point)
>               (funcall (get thing 'thing-at-point))
> 
> AFAIU an arbitrary function might  be stored here, no real relation to 
> thing-at-point at all.

Correct.  We have gobs of functions that can return several different
kinds of objects.

> Next clause deals with buffer-substring - which is thing-at-point about 
> in my understanding.

IMO, your understanding is wrong at least since 1997.

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> > d> 2. Make `thing-at-point', as before, return just what the
> >       firat `if' clause returns, if that clause is taken.
> >       IOW, move the removal of text properties (from non-nil
> >       NO-PROPERTIES) into the second `if' clause.
> 
> Why would we want to do that?  AFAIU, it would require the function
> that is the value of the thing-at-point property to second-guess
> what the caller of thing-at-point wants, something it has no means to do.

Quite the contrary.  It gives the function that is the value of
the property the ability to determine the return value.  Which
is precisely the intended behavior of that property, from the
start.

That is the point of allowing such a function: to let you
control the behavior from outside function `thing-at-point'.
This is similar to passing a function-valued optional parameter
and calling that to provide the behavior.

That's the point of the original design of `thing-at-point':
(1) provide a default, string-producing behavior and (2) also
allow for another function to define the behavior for particular
types of THING.

> If thing-at-point is called with NO-PROPERTIES non-nil, and the
> thing-at-point property returns a string, then any properties
> should be removed from that string, exactly as in the other case.

That overrides the intended behavior of the THING function.
It removes its control, which is there by design.

Sure, you can say that if a caller provides NO-PROPERTIES then
s?he always wants a string value to have properties removed.

I agree that this is not a problem (since the arg is optional,
and new).

However, I really see no reason for NO-PROPERTIES to have ever
been added.  What were we thinking, to do such a thing?  I
cannot imagine a good reason for that change, even if the arg
is optional.  We don't add optional args willy nilly just
because they don't do any harm if they're not called.

And we certainly don't add a NO-PROPERTIES arg to every
function that can return a string.  What is so special about
this function, that it needs such a "feature"?  I haven't
seen an answer to this question yet.

I seriously would propose that we deprecate that optional
arg and return to what was there before it was introduced,
along with this bug.  That was a misguided change, IMO.
It is perhaps no accident that whoever added it introduced
this bug at the same time.  I suspect that the change
betrays a poor understanding of thingatpt.el.

> I don't see why we should single out that one use case.
> I think the only change that makes sense at this point is to replace
> sequencep by stringp, as Tino originally proposed.  Other than that,
> there are no problems here that we need to solve.

I'm OK with that, too.  That was my #3 "reasonable solution":

d> 3. Make `thing-at-point', as before, return just what the firat
      `if' clause returns, if that clause is taken, except that if
      either clause returns a string then strip that string of text
      properties (if NO-PROPERTIES is non-nil).

Fine by me.

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Date: Mon, 14 Nov 2016 08:24:30 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: dgutov@yandex.ru, tino.calancha@gmail.com, 21391@debbugs.gnu.org
> 
> > > d> 2. Make `thing-at-point', as before, return just what the
> > >       firat `if' clause returns, if that clause is taken.
> > >       IOW, move the removal of text properties (from non-nil
> > >       NO-PROPERTIES) into the second `if' clause.
> > 
> > Why would we want to do that?  AFAIU, it would require the function
> > that is the value of the thing-at-point property to second-guess
> > what the caller of thing-at-point wants, something it has no means to do.
> 
> Quite the contrary.  It gives the function that is the value of
> the property the ability to determine the return value.  Which
> is precisely the intended behavior of that property, from the
> start.

So you are saying that the function should control the caller, instead
of the other way around?  That makes very little sense to me, because
the function writer has no way of knowing the caller's intent.

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> > > > d> 2. Make `thing-at-point', as before, return just what the
> > > >       firat `if' clause returns, if that clause is taken.
> > > >       IOW, move the removal of text properties (from non-nil
> > > >       NO-PROPERTIES) into the second `if' clause.
> > >
> > > Why would we want to do that?  AFAIU, it would require the
> > > function that is the value of the thing-at-point property to
> > > second-guess what the caller of thing-at-point wants, something
> > > it has no means to do.
> >
> > Quite the contrary.  It gives the function that is the value of
> > the property the ability to determine the return value.  Which
> > is precisely the intended behavior of that property, from the
> > start.
> 
> So you are saying that the function should control the caller,
> instead of the other way around?  That makes very little sense to me,
> because the function writer has no way of knowing the caller's intent.

If the THING has property `thing-at-point' then the function that
is the value of that property controls the behavior of function
`thing-at-point' _for that type of THING_.  Yes.

That is the design.  And of course you can override the behavior
locally or temporarily by removing the symbol property (and then
restoring it).  You can, similarly, change the symbol property,
to locally or temporarily get different behavior.  I don't see
that being done often.  I've never seen it done, in practice -
but it could be done.

All this just means that what you call the "caller", function
`thing-at-point', does not have its behavior decided/defined
once and for all, ahead of time.  Its behavior is controlled
by the symbol property, in addition to the `thing-at-point'
defun.

Again, this is very similar (it is equivalent) to what happens
when you pass a function-valued parameter to a higher-order
function.  In effect, `thing-at-point' is a higher-order function,
because of symbol-property `thing-at-point'.

Being able to put the property on specific THING symbols is a
way of making the higher-order nature more specific (THING
type-specific).

And (except for the fact that the property value could be
changed at runtime) this differs from passing a function-valued
parameter that is computed at runtime: with this design, the
THING-specific behavior is defined at coding time, not at
runtime.

A user or a 3rd-party library can easily define a specialized
(including a personally preferred) behavior for a given THING
type, simply by setting property `thing-at-point'.  No advising
needed.  (You can think of it as a kind of hook, if you like.)

bug#21391: 24.5; `thing-at-point' should return a string

[Eli Zaretskii]

> Date: Mon, 14 Nov 2016 09:33:00 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: dgutov@yandex.ru, tino.calancha@gmail.com, 21391@debbugs.gnu.org
> 
> All this just means that what you call the "caller", function
> `thing-at-point',

No, the caller is whoever calls thing-at-point the function.  Only
that caller knows whether the text properties on any text the function
returns are wanted or not, which is why the caller can use the
NO-PROPERTIES argument to express the intent.

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> > All this just means that what you call the "caller", function
> > `thing-at-point',
> 
> No, the caller is whoever calls thing-at-point the function.  Only
> that caller knows whether the text properties on any text the
> function returns are wanted or not, which is why the caller can use the
> NO-PROPERTIES argument to express the intent.

I misspoke there.  That is what I meant - the calling sequence,
not the defun of `thing-at-point'.

The point of the other approach is that if another function
(on the THING-symbol property) determines the `thing-at-point'
function behavior then that applies to a string value that it
might return, as well.  That enables a THING-specific behavior
that can return a propertized string - yes, overriding callers
that provide non-nil NO-PROPERTIES.

IOW, with the other approach "the behavior is controlled by the
symbol property, in addition to the `thing-at-point' defun."
And that includes the part of the defun that tries to respect
argument NO-PROPERTIES.

It's a design choice.

There is no golden rule about this.  Emacs lets things that are
outside a defun AND things that are outside a calling sequence
change the behavior of a called function.  There are umpteen ways
that this is allowed, from binding variables around code that
eventually calls "the caller" (reaching deep inside) to advising,
to hooks, to, yes, fiddling with symbol properties.

As I said, I'm fine with either approach.  One gives some more
control to the symbol property (which, remember, can be used
by "calling" code and users).  The other gives more control to
a given call.

And note, again, that if we got rid of the ill-designed
NO-PROPERTIES arg then the behavior you prefer would be used:
Calling code would remove properties separately, after the
string is returned.

bug#21391: 24.5; `thing-at-point' should return a string

[Andreas Röhler]

On 14.11.2016 16:36, Eli Zaretskii wrote:
>> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
>> Date: Mon, 14 Nov 2016 11:26:54 +0100
>>
>> IMHO the current design is really confusing and shouldn't be kept.
> IMNSHO it would be madness to make incompatible changes 20 years after
> the code was admitted into Emacs.
>
>> if (get thing 'thing-at-point)
>>                (funcall (get thing 'thing-at-point))
>>
>> AFAIU an arbitrary function might  be stored here, no real relation to
>> thing-at-point at all.
> Correct.  We have gobs of functions that can return several different
> kinds of objects.
>
>> Next clause deals with buffer-substring - which is thing-at-point about
>> in my understanding.
> IMO, your understanding is wrong at least since 1997.

So lets have closer a look at thing-at-point, which under the hood is 
called also by list-at-point, form-at-point:

((text
          (if (get thing 'thing-at-point)
              (funcall (get thing 'thing-at-point))
            (let ((bounds (bounds-of-thing-at-point thing)))
              (when bounds
                (buffer-substring (car bounds) (cdr bounds)))))))

The second clause returns a string. All further deals with this 
string-var text AFAICS.
That's why I'm calling the string the native return type here.

The first clause was discussed above. Seeing just one use-case for now:

(put 'email 'thing-at-point
      (lambda ()
        (let ((boundary-pair (bounds-of-thing-at-point 'email)))
          (if boundary-pair
              (buffer-substring-no-properties
               (car boundary-pair) (cdr boundary-pair))))))


However, also returning a string. And definitely dressed up from behind.

bug#21391: 24.5; `thing-at-point' should return a string

[Andreas Röhler]

On 14.11.2016 16:19, Drew Adams wrote:
>> Look at the start of thing-at-point implementation:
>>
>> if (get thing 'thing-at-point)
>>     (funcall (get thing 'thing-at-point))
>>
>> AFAIU an arbitrary function might  be stored here, no real
>> relation to thing-at-point at all.
> ANY time you use a function as a parameter (passed as an
> argument, obtained as the value of a variable, or picked up
> in some other way, such as here) there is little or no control
> over the nature of that function in the function that makes
> use of it.  It is up to the code that provides the function
> to provide an appropriate function (DTRT).
>
>> Next clause deals with buffer-substring - which is
>> thing-at-point about in my understanding.
> Your understanding is partial, then.  Thing-at-point has
> been, from the outset, about actual Lisp THINGS, not just
> strings that name such things.  Yes, string-naming-a-thing
> is a common use case - the most common, currently.  But it
> is only part of the story.
>
> And yes, buffer text is usually examined to come up with
> the returned THING.  But the buffer text need not be
> examined.  What is most significant is point - the buffer
> position.  And even that could be ignored by the function
> that comes up with an appropriate THING to return.
>
> thingatpt.el is much more general, and offers many more
> possibilities, than just grabbing some text at or near
> point.

Agreed so far, understand that.
My point is to suggest a restriction here, making the use easier.

If a symbol might return an arbitrary thing doing arbitrary actions, why 
it should reside in this library?
Beside this - upholding that in the core, a string is the result in any 
case first.

Follows functions which operate on this string, derive from its 
representation a number, intern it or whatever.
For clarification IMO it would be better to separate these use-cases 
from the core-thing.

These use-cases are plain operations on strings, nothing special to 
thing-at-point.

bug#21391: 24.5; `thing-at-point' should return a string

[Drew Adams]

> > Thing-at-point has been, from the outset, about actual Lisp
> > THINGS, not just strings that name such things.  Yes,
> > string-naming-a-thing is a common use case - the most common,
> > currently.  But it is only part of the story.
> >
> > And yes, buffer text is usually examined to come up with
> > the returned THING.  But the buffer text need not be
> > examined.  What is most significant is point - the buffer
> > position.  And even that could be ignored by the function
> > that comes up with an appropriate THING to return.
> >
> > thingatpt.el is much more general, and offers many more
> > possibilities, than just grabbing some text at or near
> > point.
> 
> Agreed so far, understand that.
> My point is to suggest a restriction here, making the use easier.

What difficulty in use to you see?  What is the difficult-use
problem you are trying to solve?

> If a symbol might return an arbitrary thing doing arbitrary
> actions, why it should reside in this library?

Users of thingatpt.el, including code that uses its features,
will use functions to do THING-related things.  Any "arbitrary"
functions used to come up with an appropriate thing are not, in
practice (i.e., in the real world) not arbitrary.  They are
designed to return a Lisp THING usefully.  And yes, this is all
about obtaining a THING at point.

> Beside this - upholding that in the core, a string is the result in
> any case first.  Follows functions which operate on this string,

It's hard for me to parse what you're saying, but I think your
point is only that buffer text at point is the origin of coming
up with a THING.

That does not at all imply that the THING result is a string.

> derive from its representation a number, intern it or whatever.

What's "its"?  All that serves as input to the function is point
and a buffer (and any global Emacs context - windows etc.).

It is precisely up to the "arbitrary" function to decide just
what part of that (humongous) "its" input it uses as relevant,
and it is up to that function to decide how to use it and what
THING to return.

> For clarification IMO it would be better to separate these
> use-cases from the core-thing.

There is no "core-thing".  The buffer text is only an input.
It is not the THING that is returned.  And even in the case
where a string taken from the buffer text is returned, which
string?  which bit of buffer text?

That's the point.  There are an infinite number of THINGs
that the function could return.  And even picking some buffer
text as the (string) THING to return, there are a myriad (but
in this case not infinite) different string THINGS taken from
buffer text that it could return.  There is NO "core-thing".

It is always up to the function (even in the case of a string
return) to decide what to return and how to obtain it.

> These use-cases are plain operations on strings, nothing
> special to thing-at-point.

There are no "core" or special use cases.  There is nothing
special or more "core" about returning a string value,
whether or not that string is a subsequence of buffer text.

bug#21391: 24.5; `thing-at-point' should return a string

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> IOW, thing-at-point no longer has any known bugs, and we are talking
> about forcibly breaking a use case that does no harm to us, and can
> only happen if someone abuses the 'thing-at-point' property, which
> would make it that someone's bug/misfeature, for them to fix.
>
> So why would we want to make such a change?  What am I missing?

I semi-skimmed this thread, and I think the conclusion here was that 1)
the thingatpt library has functions with non-optimal semantics (I
agree), and 2) changing the return value of thing-at-point in the manner
proposed would likely break stuff.

So it doesn't seem likely that there'll be any further progress in this
bug report, and I'm closing it.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no