bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

ping.

I rediscovered by accident that this bug still exists, because
I had assumed it would be fixed in Emacs 24, and my code was
mistakenly but in-vain-hopefully treating it as (assumed) fixed
in 24.  Since I use my library that doesn't have the bug I did
not run across it myself anymore.

;; The correct code here is (setq beg thg-end).  However, unless you
;; use my library `thingatpt+.el' or unless Emacs bug #9300 is fixed
;; (hopefully in Emacs 24), that will loop forever.  In that case we
;; move forward a char to prevent looping, but that means that the
;; position just after a THING is considered to be covered by the
;; THING (which is incorrect).
;;
(setq beg (if (or (featurep 'thingatpt+)  (> emacs-major-version 23))
              thg-end
            (1+ thg-end)))

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> Sent: Tuesday, July 28, 2015 6:45 PM
> 
> ping.
> 
> I rediscovered by accident that this bug still exists, because
> I had assumed it would be fixed in Emacs 24, and my code was
> mistakenly but in-vain-hopefully treating it as (assumed) fixed
> in 24.  Since I use my library that doesn't have the bug I did
> not run across it myself anymore.
> 
> ;; The correct code here is (setq beg thg-end).  However, unless you
> ;; use my library `thingatpt+.el' or unless Emacs bug #9300 is fixed
> ;; (hopefully in Emacs 24), that will loop forever.  In that case we
> ;; move forward a char to prevent looping, but that means that the
> ;; position just after a THING is considered to be covered by the
> ;; THING (which is incorrect).
> ;;
> (setq beg (if (or (featurep 'thingatpt+)  (> emacs-major-version 23))
>               thg-end
>             (1+ thg-end)))


A user of thingatpt+.el (which has the fix) sent me this today:

 > I miss in bug #9300 someone giving a comment about why the vanilla
 > bounds-of-thing-at-point should behave like that. Maybe a second
 > ping?

ping.  It's a trivial fix to perform.  I hope you will apply it.  Thx.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Dmitry Gutov]

I'm not sure it should be fixed.

Your reasoning seems valid, however by now this behavior is ingrained 
into my expectations of how thing-at-point should behave.

This would be a breaking change. For instance, it will make 
(bounds-of-thing-at-point 'symbol) unsuitable for use in a 
completion-at-point-functions element, to compute the first two values 
of the returned list, because during completion you're most often 
"after" the symbol. And I do use it for that purpose in one third-party 
package.

emacs-eclim also uses it in 
eclim-java-show-documentation-for-current-element. At the very least, 
this will change the existing behavior.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> I'm not sure it should be fixed. Your reasoning seems valid,
> however by now this behavior is ingrained into my expectations
> of how thing-at-point should behave.
> 
> This would be a breaking change. For instance, it will make
> (bounds-of-thing-at-point 'symbol) unsuitable for use in a
> completion-at-point-functions element, 

Why do you think so?  Have you tried it?  It does not affect
the behavior for THING = `symbol' at all.

> to compute the first two values of the returned list, because
> during completion you're most often "after" the symbol.

So?  Not a problem.  Put point after a symbol - you get the
same answer as now.

> And I do use it for that purpose in one third-party package.
> 
> emacs-eclim also uses it in
> eclim-java-show-documentation-for-current-element. At the
> very least, this will change the existing behavior.

This is the design of the thingatpt code, and the reason why
`<=' instead of `<' is a bug: 

  the function that is (get THING 'end-op) moves PAST the THING,
  so that point is not on the THING.  This is true generally, no
  matter the type of THING.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Andreas Röhler]

On 23.02.2016 07:37, Drew Adams wrote:
>> I'm not sure it should be fixed. Your reasoning seems valid,
>> however by now this behavior is ingrained into my expectations
>> of how thing-at-point should behave.
>>
>> This would be a breaking change. For instance, it will make
>> (bounds-of-thing-at-point 'symbol) unsuitable for use in a
>> completion-at-point-functions element,
> Why do you think so?  Have you tried it?  It does not affect
> the behavior for THING = `symbol' at all.
>
>> to compute the first two values of the returned list, because
>> during completion you're most often "after" the symbol.
> So?  Not a problem.  Put point after a symbol - you get the
> same answer as now.
>
>> And I do use it for that purpose in one third-party package.
>>
>> emacs-eclim also uses it in
>> eclim-java-show-documentation-for-current-element. At the
>> very least, this will change the existing behavior.
> This is the design of the thingatpt code, and the reason why
> `<=' instead of `<' is a bug:
>
>    the function that is (get THING 'end-op) moves PAST the THING,
>    so that point is not on the THING.  This is true generally, no
>    matter the type of THING.
>
>
>


That what I changed at

https://github.com/andreas-roehler/thing-at-point-utils

ar-forward-list-atpt would stop at closing paren, not one char after as 
forward-sexp would do.

Char after FORM in not considered inside in sense of at-point.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Dmitry Gutov]

On 02/23/2016 08:37 AM, Drew Adams wrote:

>> This would be a breaking change. For instance, it will make
>> (bounds-of-thing-at-point 'symbol) unsuitable for use in a
>> completion-at-point-functions element,
>
> Why do you think so?  Have you tried it?  It does not affect
> the behavior for THING = `symbol' at all.

Tried it, and yes it does. Otherwise, I wouldn't understand what this 
bug is about.

>> to compute the first two values of the returned list, because
>> during completion you're most often "after" the symbol.
>
> So?  Not a problem.  Put point after a symbol - you get the
> same answer as now.

You're contradicting the very title of this bug report.

> This is the design of the thingatpt code, and the reason why
> `<=' instead of `<' is a bug:
>
>    the function that is (get THING 'end-op) moves PAST the THING,
>    so that point is not on the THING.  This is true generally, no
>    matter the type of THING.

That's not a quote from thingatpt.el.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> Tried it, and yes it does. Otherwise, I wouldn't understand what
> this bug is about.

Sorry, my bad (it's been several years since I filed this).
Yes, it should return nil, as there is NO symbol at point.

> Your reasoning seems valid, however by now this behavior is
> ingrained into my expectations of how thing-at-point should behave.

It is your expectation that is wrong.  There are plenty of uses
of thing-at-point that go far beyond just looking for a default
value of a name near point or trying to complete a name before
(not at) point.

Those other uses include the need to test whether or not there
IS a given THING at point.  The design itself depends on this
difference: Is there a THING at point or not?  The purpose of
thingatpt.el is not only to maximize finding a THING that is
_near_ point.  One purpose is to test whether there IS a THING
AT point.

If your code wants _only_ to complete a name (or other THING)
that is at or _near_ point, then it should temporarily move to
where it really wants to pick up the name (or other THING).
Your code should not depend on an off-by-one bug, however
longstanding.

I am well aware of how useful picking up a name at or _near_
point can be, either to complete it or to use it as a default
value. I see (in emacs-devel) that you are now looking into
picking up a name near point - but you are limiting that to the
same line.

I'm glad someone is finally getting around to this.  FWIW, I
did it in 1996 and have been using it ever since (I proposed
it for Emacs).

But my code does not limit looking to the current line.
Users can control what "near" means using these variables,
which are the max number of chars and lines to search from
point, left-and-right and up-and-down, respectively:

 (tap-)near-point-x-distance - default 50 chars
 (tap-)near-point-y-distance - default 50 lines

These variables are used typically by functions that invoke
`tap-thing/form-nearest-point-with-bounds', and that provide
default text for minibuffer input or text to complete.

`thingatpt+.el' includes these "nearest-point" functions
(with library prefix `tap-'):

 bounds-of-form-nearest-point
 bounds-of-list-nearest-point
 bounds-of-sexp-nearest-point
 bounds-of-symbol-nearest-point
 bounds-of-thing-nearest-point
 color-nearest-point
 color-nearest-point-with-bounds
 find-fn-or-var-nearest-point  (a command)
 form-nearest-point
 form-nearest-point-with-bounds
 list-contents-nearest-point
 list-nearest-point
 list-nearest-point-with-bounds
 list-nearest-point-as-string
 non-nil-symbol-name-nearest-point
 non-nil-symbol-nearest-point
 number-nearest-point
 region-or-word-nearest-point
 region-or-non-nil-symbol-name-nearest-point
 sentence-nearest-point
 sexp-nearest-point
 sexp-nearest-point-with-bounds
 string-contents-nearest-point
 string-nearest-point
 symbol-name-nearest-point
 symbol-nearest-point
 symbol-nearest-point-with-bounds
 thing-nearest-point
 thing-nearest-point-with-bounds
 unquoted-list-nearest-point
 unquoted-list-nearest-point-as-string
 word-nearest-point

So I do have some experience with the kinds of uses of
thing-at-point that you are interested in.  But there are
lots of other uses, as well, which rely on its behavior
being usable in repetitive (iterative, recursive) contexts.

And that means relying on its treating the position after
a thing differently from the position before the thing, wrt
whether or not there is a THING at that position.

If you never make use of thing-at-point, beyond wanting to
pick up a thing at or near point, and you don't care much
about exactly where point is in relation to the thing, then
you won't understand the importance of this bug.

My code that does use thing-at-point repetitively needs the
basic code to distinguish whether there is a thing at point
in a precise way.  It is not enough to just maximize the
possibility of obtaining a thing near point.  The point of
such code is instead to perform operations on occurrences
of a specific THING over a given region of text.

One example is `isearchp-thing', which does incremental
search within THING search contexts.  The contexts are
determined by scanning a region/buffer progressively, and
this depends on basic thing-at-point functions doing the
right thing wrt whether there is a THING at a given point.

The scanning function is `isearchp-thing-scan', and it is
in its code that you will find the comment included in this
bug report:

;; The correct code here is (setq beg thg-end).  However,
;; unless you use my library `thingatpt+.el' or unless Emacs
;; bug #9300 is fixed that will loop forever.  In that case
;; we move forward a char to prevent looping, but that means
;; that the position just after a THING is considered to be
;; covered by the THING (which is incorrect).

(setq beg (if (featurep 'thingatpt+) thg-end (1+ thg-end)))

> This would be a breaking change. For instance, it will make
> (bounds-of-thing-at-point 'symbol) unsuitable for use in a
> completion-at-point-functions element, to compute the first
> two values of the returned list, because during completion 
> you're most often "after" the symbol. And I do use it for
> that purpose in one third-party package.

See previous.  It is the calling code that is wrong, here (and
no need for quoting "after" - you are after, not on, the symbol).
The calling code should instead check `bounds-of-thing-at-point'
at the right position.

There has to be a difference between there being a THING at
point and there not being a THING at point.  And thingatpt.el
is  designed for there not being at THING at point when, well,
there is no THING at point.  At point is not the same as before
point.

(Yes, point is _between_ characters.  But one or the other,
but not both, positions wrt a char that is part of THING needs
to be picked as its start or end.

Especially for uses of thing-at-point that are iterative or
recursive, it is important that either the position after the
last char or the position before the first char not be included
in the THING "at" that position.  This should be obvious, but
is not paid attention to if one's only interest is in grabbing
some text that is _near_ point but might not be exactly _at_
point.

> At the very least, this will change the existing behavior.

Yes - it is a bug fix.  The current behavior is bugged.

> > This is the design of the thingatpt code, and the reason why
> > `<=' instead of `<' is a bug:
> >
> >    the function that is (get THING 'end-op) moves PAST the THING,
> >    so that point is not on the THING.  This is true generally, no
> >    matter the type of THING.
> 
> That's not a quote from thingatpt.el.

It is nevertheless the design (intention), clear from the code.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Dmitry Gutov]

On 02/23/2016 06:15 PM, Drew Adams wrote:

> Yes, it should return nil, as there is NO symbol at point.

If we ask the users, I'm guessing we'll get mixed answers on that, at 
least as a result of this long-standing thing-at-point behavior.

> It is your expectation that is wrong.  There are plenty of uses
> of thing-at-point that go far beyond just looking for a default
> value of a name near point or trying to complete a name before
> (not at) point.

What I'm saying is, "fixing" it will most likely break code in the wild. 
Not just mine.

> Those other uses include the need to test whether or not there
> IS a given THING at point.  The design itself depends on this
> difference: Is there a THING at point or not?

They can call (bounds-of-thing-at-point 'foo), and then compare the cdr 
with the value of point.

> The purpose of
> thingatpt.el is not only to maximize finding a THING that is
> _near_ point.  One purpose is to test whether there IS a THING
> AT point.

We're a long way from maximizing it. To see something closer to the 
other end of the spectrum, see the definition of find-tag-default-bounds 
before 
http://git.savannah.gnu.org/cgit/emacs.git/commit/?h=emacs-25&id=e72a26e00981a508569a0856125061310a3f64ac.

> I see (in emacs-devel) that you are now looking into
> picking up a name near point - but you are limiting that to the
> same line.

Not at all, see above.

>>> This is the design of the thingatpt code, and the reason why
>>> `<=' instead of `<' is a bug:
>>>
>>>    the function that is (get THING 'end-op) moves PAST the THING,
>>>    so that point is not on the THING.  This is true generally, no
>>>    matter the type of THING.
>>
>> That's not a quote from thingatpt.el.
>
> It is nevertheless the design (intention), clear from the code.

I'm not so clear on that. The following comment tells me the opposite 
(the position where a substring ends is normally the one _after_ its 
last character):

     ;; Try a second time, moving backward first and then forward,
     ;; so that we can find a thing that ends at ORIG.

If we didn't need to be able to find a thing that ends just before 
point, I don't think the implementation would need the "Try a second 
time" branch at all: when point if before the last character of a 
symbol, (forward-symbol) still works.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> > Yes, it should return nil, as there is NO symbol at point.
> 
> If we ask the users, I'm guessing we'll get mixed answers on that, at
> least as a result of this long-standing thing-at-point behavior.

You might get opinions, but the fact is that there is no thing at
point in that case.

"At point" can only mean, since point is between characters, either
before point or after point.  It cannot mean both.  Not and keep
a possibility of recursive/iterative use.  Moving forward over a
thing puts point after the thing.  It does not keep point on/at
the thing.

The design must pick one or the other meaning of "at": a character
that belongs to the THING at point is either after or before point.
It cannot reasonably be both.

IMO, and all of the code confirms this (apart from this bug): "at"
in "thing at point" means after point, not before point.

> > It is your expectation that is wrong.  There are plenty of uses
> > of thing-at-point that go far beyond just looking for a default
> > value of a name near point or trying to complete a name before
> > (not at) point.
> 
> What I'm saying is, "fixing" it will most likely break code in the wild.
> Not just mine.

The fix to your code and theirs is trivial.

Bit if you must, rename the current, bugged implementation of
`bounds-of-thing-at-point' to `bounds-of-thing-at-or-after-point'.
Tell such folks to use that.  Likewise, add `thing-at-or-after-point',
if necessary, for any code that current depends on the broken
`thing-at-point'.

If you must, do that plus deprecate the (perfectly good, but not
for this broken code) name `bounds-of-thing-at-point', so any such
3rd-party code makes the change.

And add a function `bounds-of-thing-at-point-strict' that does
what `bounds-of-thing-at-point' should do (= the bug fix).  Change
the Emacs code that uses the broken `bounds-of-thing-at-point' to
use `bounds-of-thing-at-point-strict'.

IOW, wean any code from the broken implementation and use the
fixed implementation.

This is if you are convinced that there are zillions of uses of
the bugged `bounds-of-thing-at-point' that depend on the bugged
behavior.  I'm not convinced of that.

I'd say bite the bullet: fix the bug properly, and when anyone
complains tell them to use `bounds-of-thing-at-or-after-point'
if they really want the bugged behavior.  Better: tell them
to use the fixed `bounds-of-thing-at-point' after backing up
so point is actually on the THING instead of after it.

> > Those other uses include the need to test whether or not there
> > IS a given THING at point.  The design itself depends on this
> > difference: Is there a THING at point or not?
> 
> They can call (bounds-of-thing-at-point 'foo), and then compare
> the cdr with the value of point.

You are missing the point.  I won't repeat myself.  See what
I wrote about use cases.  See the code I referenced, if needed.

> >>> This is the design of the thingatpt code, and the reason why
> >>> `<=' instead of `<' is a bug:
> >>>
> >>>    the function that is (get THING 'end-op) moves PAST the THING,
> >>>    so that point is not on the THING.  This is true generally, no
> >>>    matter the type of THING.
> >>
> >> That's not a quote from thingatpt.el.
> >
> > It is nevertheless the design (intention), clear from the code.
> 
> I'm not so clear on that.

That much is clear.

> The following comment tells me the opposite
> (the position where a substring ends is normally the one _after_ its
> last character):
> 
>      ;; Try a second time, moving backward first and then forward,
>      ;; so that we can find a thing that ends at ORIG.

ORIG is the original position.  A thing that ends at that position
is at point.  A thing that ends before that position is not a thing
at point.

Look at `goto-char' or any other char-counting functions.  If you
move point "to" character 2, point = 2.  The char "at" point is the
char after point - point is before the char that has the same number
as point.  When point = 2 the cursor position (aka point) is between
chars 1 and 2, and we say point is "at" (or "looking at") char 2.

> If we didn't need to be able to find a thing that ends just before
> point,

Before is not at (= after).  Ends at ORIG does not mean ends before
ORIG.

> I don't think the implementation would need the "Try a second
> time" branch at all: when point if before the last character of a
> symbol, (forward-symbol) still works.

Believe me, I've walked through that particular code a hundred
times, in the debugger and without.  The code you are referring
to is needed, and it is not about finding a thing that ends before
point.

But I think you either try to see or you don't.  I cannot make
you see.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Dmitry Gutov]

On 02/24/2016 03:31 AM, Drew Adams wrote:

> The fix to your code and theirs is trivial.

Where's that "think of the poor users" call of yours that accompanies 
any breaking change in Emacs? Users don't like broken code. We can only 
fix code inside Emacs.

> Bit if you must, rename the current, bugged implementation of
> `bounds-of-thing-at-point' to `bounds-of-thing-at-or-after-point'.

A rename will break third-party code just as well.

> Tell such folks to use that.  Likewise, add `thing-at-or-after-point',
> if necessary, for any code that current depends on the broken
> `thing-at-point'.

Won't be usable in packages targeting older versions.

I'm not saying it's entirely out of the question, but personally, I'm 
not convinced.

> If you must, do that plus deprecate the (perfectly good, but not
> for this broken code) name `bounds-of-thing-at-point', so any such
> 3rd-party code makes the change.
>
> And add a function `bounds-of-thing-at-point-strict' that does
> what `bounds-of-thing-at-point' should do (= the bug fix).  Change
> the Emacs code that uses the broken `bounds-of-thing-at-point' to
> use `bounds-of-thing-at-point-strict'.

We can add bounds-of-thing-at-point-strict even without changing the 
existing function. Patch welcome, I think.

> This is if you are convinced that there are zillions of uses of
> the bugged `bounds-of-thing-at-point' that depend on the bugged
> behavior.  I'm not convinced of that.

Maybe there aren't too many. Will you do the research on this?

> I'd say bite the bullet: fix the bug properly, and when anyone
> complains tell them to use `bounds-of-thing-at-or-after-point'
> if they really want the bugged behavior.  Better: tell them
> to use the fixed `bounds-of-thing-at-point' after backing up
> so point is actually on the THING instead of after it.

Any such client would be forced to call bounds-of-thing-at-point-strict 
twice. Which is not particularly ideal.

>> They can call (bounds-of-thing-at-point 'foo), and then compare
>> the cdr with the value of point.
>
> You are missing the point.  I won't repeat myself.

Thanks for that.

> Look at `goto-char' or any other char-counting functions.  If you
> move point "to" character 2, point = 2.  The char "at" point is the
> char after point - point is before the char that has the same number
> as point.  When point = 2 the cursor position (aka point) is between
> chars 1 and 2, and we say point is "at" (or "looking at") char 2.

Yup. But when we say "word X ends at position P", P is after the last 
character of X, not before.

> Before is not at (= after).  Ends at ORIG does not mean ends before
> ORIG.

Think of the semantics of `match-end', or the last argument of `substring'.

> Believe me, I've walked through that particular code a hundred
> times, in the debugger and without.  The code you are referring
> to is needed, and it is not about finding a thing that ends before
> point.

Even though the comment says that.

> But I think you either try to see or you don't.  I cannot make
> you see.

That's a very zen position for someone who just wrote a 2.5 screen 
email. Why don't you present a valid (in your sense) configuration that 
a bounds-of-thing-at-point implementation without the "else" branch will 
return nil in?

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> > The fix to your code and theirs is trivial.
> 
> Where's that "think of the poor users" call of yours that
> accompanies any breaking change in Emacs? Users don't like broken code.
> We can only fix code inside Emacs.

And when we do, 3rd-party code sometimes has to adjust.  I'm forced
to do that all the time - and not for bug fixes, usually.

This fix will give all users better, and consistent behavior.

> > Bit if you must, rename the current, bugged implementation of
> > `bounds-of-thing-at-point' to `bounds-of-thing-at-or-after-point'.

I meant ...-or-before-... not ...-or-after-...

> A rename will break third-party code just as well.

The proper fix for such bad 3rd-party code (it never should have
depended on a bug that nil is not returned when there is no thing
at point) is to move to the position where it _really_ wants to look
for a thing.  Such code is typically just looking for a thing near
point, for completion or to use as a default value.

But we _could_ provide a function that finds the thing at or just
before point.  If you don't want to provide such a function, so
much the better.

> > Tell such folks to use that.  Likewise, add `thing-at-or-after-
> > point',

(Again, I meant ...-or-before-... not ...-or-after-...)

> > if necessary, for any code that current depends on the broken
> > `thing-at-point'.
> 
> Won't be usable in packages targeting older versions.

Right.  Like all 3rd-party code, it will use (if (fboundp...)...).

But the proper fix for 3rd-party code, mentioned above, does not
have any such problem.  It should look for a thing at (1- (point))
if it wants to get a thing that might be just before point but not
at point.

That's what always should have done, and that has always worked
(and hopefully always will).

> > If you must, do that plus deprecate the (perfectly good, but not
> > for this broken code) name `bounds-of-thing-at-point', so any such
> > 3rd-party code makes the change.
> >
> > And add a function `bounds-of-thing-at-point-strict' that does
> > what `bounds-of-thing-at-point' should do (= the bug fix).  Change
> > the Emacs code that uses the broken `bounds-of-thing-at-point' to
> > use `bounds-of-thing-at-point-strict'.
> 
> We can add bounds-of-thing-at-point-strict even without changing the
> existing function. Patch welcome, I think.

It's the same patch.  The proper name for it is
`bounds-of-thing-at-point'.  But if you are stubborn then go for
`bounds-of-thing-at-point-strict'.  But be sure to use it everywhere
in the Emacs code in place of `bounds-of-thing-at-point' - that's the fix.

> > This is if you are convinced that there are zillions of uses of
> > the bugged `bounds-of-thing-at-point' that depend on the bugged
> > behavior.  I'm not convinced of that.
> 
> Maybe there aren't too many. Will you do the research on this?

Nope.  Will you?  Does anyone need to?  You're the one who
mentioned that your code depends on checking for a thing at the
wrong position in order to get a thing at point-minus-one.  And
you mentioned an Eclipse function that acts similarly.  That's two.

> > I'd say bite the bullet: fix the bug properly, and when anyone
> > complains tell them to use `bounds-of-thing-at-or-after-point'

(Again, I meant ...-or-before-... not ...-or-after-...)

> > if they really want the bugged behavior.  Better: tell them
> > to use the fixed `bounds-of-thing-at-point' after backing up
> > so point is actually on the THING instead of after it.
> 
> Any such client would be forced to call bounds-of-thing-at-point-
> strict twice. Which is not particularly ideal.

Not at all.  Why do you say that?  That's the behavior you get
today, and apparently the behavior you still want: ask for a thing
that is either at point or one char before point.

> Yup. But when we say "word X ends at position P", P is after the
> last character of X, not before.
> 
> > Before is not at (= after).  Ends at ORIG does not mean ends
> > before ORIG.
> 
> Think of the semantics of `match-end', or the last argument of
> `substring'.

Think of all the other uses of thing-at-point, and the other THINGs
it finds and where it finds them.

Type (foo bar) at top level, and put point after the ).
M-: (thing-at-point 'list)
What do you get?  id it give you "(foo bar)"?  Or did it give
you nil?  There is no list at point.  Is this a bug?  No; it's TRT.

There is a reason for this behavior.  It is what's needed when you
use `thing-at-point' in combination (e.g. repetitively).  I pointed
you to code that does this.  And yes, it needs to work for all types
of THINGs, including, yes, symbols (words, names,...).

> > But I think you either try to see or you don't.  I cannot make
> > you see.
> 
> That's a very zen position for someone who just wrote a 2.5 screen
> email. Why don't you present a valid (in your sense) configuration
> that a bounds-of-thing-at-point implementation without the "else"
> branch will return nil in?

OK, I give up.  I don't need this bug fix for my own code.  Just
trying to do a good deed for Emacs and its users.  I fixed this
long ago for myself, and I make heavy use of the fix.  You're not
interested in fixing this in Emacs.  So be it.

You said at the outset:

  Your reasoning seems valid, however by now this behavior is
  ingrained into my expectations of how thing-at-point should behave.

It's clearly not about your being unconvinced that the fix is correct.
It's about your not wanting to give up your ingrained expectations
of the incorrect behavior.  Sheesh.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Dmitry Gutov]

On 02/26/2016 03:44 AM, Drew Adams wrote:

> But the proper fix for 3rd-party code, mentioned above, does not
> have any such problem.  It should look for a thing at (1- (point))
> if it wants to get a thing that might be just before point but not
> at point.

If the thing _begins_ at point, and the third-party code in question 
calls (save-excursion (forward-char -1) (thing-at-point 'foo)), they 
will get nil.

>> Maybe there aren't too many. Will you do the research on this?
>
> Does anyone need to?

I imagine so.

> You're the one who
> mentioned that your code depends on checking for a thing at the
> wrong position in order to get a thing at point-minus-one.  And
> you mentioned an Eclipse function that acts similarly.  That's two.

I never mentioned anything Eclipse-related in this bug.

>>> if they really want the bugged behavior.  Better: tell them
>>> to use the fixed `bounds-of-thing-at-point' after backing up
>>> so point is actually on the THING instead of after it.
>>
>> Any such client would be forced to call bounds-of-thing-at-point-
>> strict twice. Which is not particularly ideal.
>
> Not at all.  Why do you say that?

See above.

>> Think of the semantics of `match-end', or the last argument of
>> `substring'.
>
> Think of all the other uses of thing-at-point, and the other THINGs
> it finds and where it finds them.
>
> Type (foo bar) at top level, and put point after the ).
> M-: (thing-at-point 'list)
> What do you get?  id it give you "(foo bar)"?  Or did it give
> you nil?  There is no list at point.  Is this a bug?  No; it's TRT.

If the list is at the end of the buffer, it gives me an empty string, or 
a string of spaces. So yeah, this particular "thing" seems bugged.

> Why don't you present a valid (in your sense) configuration
>> that a bounds-of-thing-at-point implementation without the "else"
>> branch will return nil in?
>
> OK, I give up.

Because there is no such example.

> It's clearly not about your being unconvinced that the fix is correct.
> It's about your not wanting to give up your ingrained expectations
> of the incorrect behavior.

Not just mine. I believe I have demonstrated that the code has been 
written with exactly this expectation in mind. And stayed like that for 
decades.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> > But the proper fix for 3rd-party code, mentioned above, does not
> > have any such problem.  It should look for a thing at (1- (point))
> > if it wants to get a thing that might be just before point but not
> > at point.
> 
> If the thing _begins_ at point, and the third-party code in question
> calls (save-excursion (forward-char -1) (thing-at-point 'foo)), they
> will get nil.

So what?  The point is that if code wants to get a thing at point
OR a thing at point-minus-one, then that's exactly what it should
check - which is, BTW, what the currently bugged code does.

> > You're the one who
> > mentioned that your code depends on checking for a thing at the
> > wrong position in order to get a thing at point-minus-one.  And
> > you mentioned an Eclipse function that acts similarly.  That's
> > two.
> 
> I never mentioned anything Eclipse-related in this bug.

Sorry - "eclim".

> >>> if they really want the bugged behavior.  Better: tell them
> >>> to use the fixed `bounds-of-thing-at-point' after backing up
> >>> so point is actually on the THING instead of after it.
> >>
> >> Any such client would be forced to call bounds-of-thing-at-point-
> >> strict twice. Which is not particularly ideal.
> >
> > Not at all.  Why do you say that?
> 
> See above.

No.  Just use the current (bugged) implementation.  It is a
`bounds-of-thing-at-point-or-at-point-minus-one'.

You might even provide a function that takes an other (e.g.
optional) arg that is the other end of a range of positions
over which to check whether there is a thing.

Currently, you want the behavior of getting a thing at point or
a thing that is point-minus-one.  Add an argument to the new
function that lets it return a thing that is at any of the positions
between point and the new arg (a position).

I only need to test for (and get) the thing at point (really at
point).  But other use cases might want what you want.

> >> Think of the semantics of `match-end', or the last argument of
> >> `substring'.
> >
> > Think of all the other uses of thing-at-point, and the other
> > THINGs it finds and where it finds them.
> >
> > Type (foo bar) at top level, and put point after the ).
> > M-: (thing-at-point 'list)
> > What do you get?  id it give you "(foo bar)"?  Or did it give
> > you nil?  There is no list at point.  Is this a bug?  No; it's
> > TRT.
> 
> If the list is at the end of the buffer, it gives me an empty
> string, or a string of spaces. So yeah, this particular "thing"
> seems bugged.

I have much better-behaving list-at-point code.  It's in the
same file, if you are really interested.  It always returns nil
if the cursor is not on a list (there is no list at point),
including when point is at eob.

The crucial point is that THINGs that end *before* point are not
*at* point.  That's all.  And this applies to all kinds of THINGs.

Whether or not there are bugs for particular kinds of THINGs is
something else, and those should be fixed individually.  Comments
come to mind, IIRC.  I have several such fixes in my code.

But the basic off-by-one bug (this bug) needs to be fixed,
if we expect coherent thing-at-point behavior and flexible,
repeatable use.

> > Why don't you present a valid (in your sense) configuration
> >> that a bounds-of-thing-at-point implementation without the
> >> "else" branch will return nil in?
> >
> > OK, I give up.
> 
> Because there is no such example.

Sigh.

> > It's clearly not about your being unconvinced that the fix is
> > correct.  It's about your not wanting to give up your ingrained
> > expectations of the incorrect behavior.
> 
> Not just mine. I believe I have demonstrated that the code has been
> written with exactly this expectation in mind. And stayed like that
> for decades.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Dmitry Gutov]

On 02/26/2016 04:39 PM, Drew Adams wrote:

>> I never mentioned anything Eclipse-related in this bug.
>
> Sorry - "eclim".

My bad, it is indeed Eclipse-related.

As for the rest, my first comment here still seems to be an appropriate 
response.

Maybe someone else would like to comment as well?

> I have much better-behaving list-at-point code.  It's in the
> same file, if you are really interested.  It always returns nil
> if the cursor is not on a list (there is no list at point),
> including when point is at eob.

I'd be interested in a patch. Especially if it's orthogonal (as it 
should be) to whatever we decide in this bug.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> > I have much better-behaving list-at-point code.  It's in the
> > same file, if you are really interested.  It always returns nil
> > if the cursor is not on a list (there is no list at point),
> > including when point is at eob.
> 
> I'd be interested in a patch. Especially if it's orthogonal
> (as it should be) to whatever we decide in this bug.

I'm interested in _this_ bug being fixed.  Bugs for individual
THINGs are much more minor.  This bug affects *all* THINGs - the
basic behavior of thing at point.

This about the basic behavior.  But this is a bug that someone will
not notice if their only use of thingatpt.el is, as in your case,
to grab some text near point for completion or for use as a default
value.  That is why this bug has gone unnoticed and unreported.

Such uses underestimate the power of thingatpt.el and misunderstand
what it is about.  It is about more than trying to maximize the
possibility of getting something near point.  It is important that
the functions can actually tell you, accurately, _whether_ there is
a THING at point.  If you are looking to grab something near point
then you don't care about this.  But that's not all of what
thingatpt.el is about.

If you want to get a THING that is at OR NEAR point then you should
use code that does that (I have such *_nearest_* code, and you are
beginning to work on some too, it seems).  But thing-at-point (and
bounds) should not be corrupted to return a THING instead of nil
when there is no THING at point.  That is, alas, currently the case.

But as I said, I do not need this bug fix for myself.  I use my own
code, which does not have the bug.  I had to fix it long ago, to
get reasonable behavior for navigating among and parsing multiple
THING occurrences.  The boundary between one THING and another, and
between a THING and non-THING needs to be defined in a way that
is consistent and usable in a general way.  That's not the case
if a THING _before_ point is returned when you try to test for
a THING at point.

Likewise for users of my libraries.  I let them know that the
Isearch+ and Icicles features that act on successive THINGs in a
buffer will not work for some THINGs if they use only vanilla
thingatpt.el.  For consistent and reliable use they will need to
also load thingatpt+.el.  (My libraries do not _require_ it.)
I occasionally get a bug report to which I need to reply that
thingatpt.el has this bug and advise them to use thingatpt+.el
for things to work.

For them to take advantage of the code that really uses THINGs for
more than simply grabbing text near point to use as a default value
or for completion, this bug needs to be fixed.  Or they need to use
thingatpt+.el.  I would prefer that they be able to get the fixed
behavior from vanilla Emacs, but if not, no problem for me.
I filed the bug report for Emacs, not for me.  But you know that.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Lars Ingebrigtsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> I'm not sure it should be fixed.
>
> Your reasoning seems valid, however by now this behavior is ingrained
> into my expectations of how thing-at-point should behave.
>
> This would be a breaking change. For instance, it will make
> (bounds-of-thing-at-point 'symbol) unsuitable for use in a
> completion-at-point-functions element, to compute the first two values
> of the returned list, because during completion you're most often
> "after" the symbol. And I do use it for that purpose in one
> third-party package.

So I think the conclusion here is that we don't want to change the
behaviour, and I'm therefore closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> So I think the conclusion here is that we don't want to change the
> behaviour, and I'm therefore closing this bug report.


This is extremely unfortunate.  Quite misguided.

This completely misunderstands and misses the
point of `thingatpt.el' - in particular of
`bounds-of-thing-at-point'.

The point is NOT simply to try to obtain a
thing (and its bounds) _near_ point.

The aims and use cases (there are two) are:

1. To find out _whether there is_, in fact,
   a THING at point.

   AT POINT - not point OR (point - 1).

2. IF there really is a THING at point, to
   return it (or its bounds).

This bug is about the misguided behavior that
returns the THING at (point - 1) instead of
returning nil, when there's NO THING at point.

This bug renders thingatpt.el useless for any
functionality that needs to know _whether_
there's a THING at point.  And that means a
loss of lots of use cases - down the drain.

The miscoding is motivated by an expectation
that the only thing anyone ever wants to use
thingatpt.el for is to get a THING near point
for use as a default value for reading input.

That's the least useful application of thing
at point.  (It's a common one, admittedly.)
___

It's possible to define other functions that
do have as their only aim to return a THING
that's NEAR point.

As mentioned in the bug thread, it's not hard
to do that , and even to let callers specify
a tolerance that defines NEAR.  Such a function
is really what should be used to get some
default text for reading minibuffer input.
(I've offered this - code.)

But a get-some-THING-near-point functionality
doesn't in any way _replace_ the need for aims
1 and 2 of the thingatpt.el design.  The code
is corrupted, resulting in loss of the raison
d'etre for thingatpt.el.

A new low for Emacs.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

emacs -Q
 
In a Lisp buffer, put point after a list: just after the closing paren.
 
M-: (bounds-of-thing-at-point 'sexp)
 
It should return nil, since there is no sexp at point.  Instead it
returns the bounds of the sexp at the previous char (the closing paren).
 
The fix is to change these two sexps in the code:
 
(<= orig real-end) SHOULD BE (< orig real-end)
(<= orig end)      SHOULD BE (< orig end)
 
The tests for the THING'S beginning are correct - they should be <=, as
they are.  But the tests for the THING's end are incorrect - they should
always be <.
 
The reason is that the function that is (get THING 'end-op) moves PAST
the THING, so that point is not on the THING.  This is true generally,
no matter the type of THING.
 

In GNU Emacs 24.0.50.1 (i386-mingw-nt5.1.2600)
 of 2011-08-08 on 3249CTO
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (4.5) --no-opt'
 

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Tino Calancha]

Consider following code:


emacs -Q:

C-x b foo RET
M-: (insert "(foo bar)")

I)
M-: (bounds-of-thing-at-point 'sexp) RET
=> (1 . 10)
II)
M-: (bounds-of-thing-at-point 'list) RET
=> (10 . 10)
III)
M-: (thing-at-point 'sexp) RET
=> "(foo bar)"
IV)
M-: (thing-at-point 'list) RET
=> ""


V)
M-: (save-excursion (goto-char 1) (bounds-of-thing-at-point 'list)) RET
=> (1 . 10)
VI)
M-: (save-excursion (goto-char 1) (bounds-of-thing-at-point 'sexp)) RET
=> (1 . 10)


* I agree with Drew that there is neither sexp nor list at point,
   so i would expect I), II), III) and IV) returning nil.

* Both function names, i.e., functions at I) and III), and their doc strings
   looks clear: return THING at point (III) or return the locations
   of THING at point.  If there is no such THING at point i would expect
   both return nil: IMO that would be more consistent/intuitive with the 
func.
   names and doc strings.

* I) and II) agree but III) and IV disagree.
   I would expect III) and IV) returning the same value.


In GNU Emacs 25.1.50.1 (x86_64-pc-linux-gnu, GTK+ Version 3.20.6)
Repository revision: 9341142dc876f4d93c442242206a7d2d40fd03af

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Dmitry Gutov]

On 06/20/2016 12:21 PM, Tino Calancha wrote:

> * I agree with Drew that there is neither sexp nor list at point,
>   so i would expect I), II), III) and IV) returning nil.

It's a matter of definition. If we say there is, then there is. We could 
also add a variable, of course.

> * I) and II) agree

Do they?

> but III) and IV disagree.
>   I would expect III) and IV) returning the same value.

Agree. But will you be satisfied if they both return "(foo bar)"?

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Tino Calancha]

> It's a matter of definition. If we say there is, then there is. We could also 
> add a variable, of course.
Sure, definitions are free.  Maybe the doc string is not clear enough: at 
point or right before than at point would be more precise.


>> * I) and II) agree
>
> Do they?
No they don't, sorry for that.
It should read:
V) and VI) agree but I) and II) disagree.  It seems it depends from the 
point of view and how you ask.  It shouldn't: or always there is a list or 
never.

>> but III) and IV disagree.
>>   I would expect III) and IV) returning the same value.
>
> Agree. But will you be satisfied if they both return "(foo bar)"?
I am sure you know my answer :-)

Regards,
Tino

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Eli Zaretskii]

> From: Tino Calancha <tino.calancha@gmail.com>
> Date: Mon, 20 Jun 2016 18:21:09 +0900
> Cc: f92capac@gmail.com, dgutov@yandex.ru
> 
> * I agree with Drew that there is neither sexp nor list at point,
>    so i would expect I), II), III) and IV) returning nil.

FWIW, I agree with Dmitry: this has been a de-facto behavior long
enough to consider it the correct one.  If documentation is confusing
in that it says otherwise, we should fix the documentation.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Tino Calancha]

> FWIW, I agree with Dmitry: this has been a de-facto behavior long
> enough to consider it the correct one.  If documentation is confusing
> in that it says otherwise, we should fix the documentation.
* If the behaviour is obviously correct the funcs. are easier to use.
* If the definition allows for more general uses, as Drew explained, 
much better.
* For code where the existence of the THING (exactly) at point really
     matters, the current behaviour may cause undesirable effects:

     * Karen Lisp, the mother of 8 yr old Emily, is enjoying a family 
cruise.
     * Karen just downloaded in her favourite iTHING one new APP based on
       'thingatpt:
       - It shows an alert whenever there is _no_ a boat at point.
       - The point equals Emily position.
     * The ship is not moving: they throw the anchor during the Hawaiian
       dance show, which Karen is watching while Emily went to play.

     1) First, Emily went to the prow to receive some fresh air.
        => App: no alert
     2) Later, Emily went to the middle of the ship following one cat.
        => App: no alert
     3) Finally, she went to the stern to see the dolphins.
        => App: no alert
     4) While Emily was feeding the dolphins with her cheeseburger,
        she fell down to the water.
        => App: no alert!!!

     I)  Hopefully Emily knows how to swim.

     II) Emily would be OK if
         (thing-at-point THING)
         return nil whenever there is no THING at point.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> > * I agree with Drew that there is neither sexp nor list at point,
> >    so i would expect I), II), III) and IV) returning nil.

Tino agrees because he wants to make use of the difference between
there being a THING at point and there being no THING at point but
there being a THING at (1- (point)).

> FWIW, I agree with Dmitry: this has been a de-facto behavior long
> enough to consider it the correct one. If documentation is confusing
> in that it says otherwise, we should fix the documentation.

I couldn't disagree more.

It is wrong to consider the current behavior "the correct one",
regardless of how long it has been in place.  It is wrong because
you cannot use it in a general and precise way.  It is just broken.
It has been broken for a long time, but it is broken nevertheless.

The proper thing to do is to:

  1. Fix this bug.  It is a real bug.

  2. Add a new function that provides the same behavior as the
     broken behavior that is current, or similar.  And call it
     "-near-point", not "-at-point".

More precisely, for #2 the use case you cite is only to maximize
grabbing a thing at or _near_ point.  In the case of the current
code, that means at point or at (1- (point)).  If all you care
about is grabbing something at either of those positions then the
code works for you.  If you try to use it more generally, it is
broken.  IOW, if you actually care about the difference between
point and (1- (point)) then you are out of luck.

A better, more general, near-point function looks farther from
point, up to some caller-specified distance (horizontally and
vertically).  That's the purpose, in my code, of variables
`tap-near-point-x-distance' & `tap-near-point-y-distance' and
function `tap-bounds-of-thing-nearest-point' (prefix `tap-' for
library `thingatpt+.el').

See the arguments given in the bug thread for why #1 is important -
why `bounds-of-thing-at-point' should be fixed as indicated.

It is important that one be able to use `bounds-of-thing-at-point'
and `thing-at-point' in a way that is accurate, precise, and
general, and not only to try to grab something that is near point.

In particular, this matters when the functions are used
programmatically to handle successive THINGs (of the same type)
in a buffer or region.  For that, there needs to be a clean
boundary between THING and no THING at a given position.  You
need to be able to test whether there is actually a THING at
point.

I've spent a lot of time with this code, and with a fixed version
of it, over the years.  My use cases go beyond just trying to come
up with a default value for a minibuffer read.

That doesn't seem to matter to those in charge here.  Too bad, but
so be it.  I'll continue to use my code (`thingatpt+.el'), so not
fixing this bug doesn't affect me much, directly.

But it does affect me, and it affects others too.  For me, it
means that other code, which makes use of this fix, must also
conditionally handle the case where a user does not have
`thingatpt+.el', even if the result is inadequate.

I recommend to users of my code that makes use of thing-at-point
features that they also use library `thingatpt+.el', but I try to
let that other code have some semi-workable fallback behavior for
the case where they do not use `thingatpt+.el'.

So yes, this is an added (and unnecessary) burden for me and
for users of my code.  But I've been dealing with it for years,
so it's nothing new.

The real loss, if you do not fix this bug, is for Emacs users who
do not use `thingatpt+.el'.  They will be unable to do things with
the `thing-at-point' code that they should be able to do, simply
because it is broken at the end-of-thing boundary.

But it has proven to be impossible to convince you to apply this
simple fix.  So be it.

I can only repeat that the proper solution is to fix this bug
and to also give users a new function that does what you and they
currently expect of `thing-at-point', for the use case of
providing a default value for something: be able to grab a thing
at or near point.

`thingatpt+.el' has existed since 1996, yet you argue that even
though the thingatpt.el code has this bug, it should not be
fixed because users (or you) are used to it.

  "Your reasoning seems valid, however by now this behavior is
   ingrained into my expectations of how thing-at-point should
   behave."

Your expectations come from using the code only to grab a
default value from the text.  You don't care about testing for
a thing at point.  You want grabbing text to work even when
there is no THING at point but there is a THING at (1- (point)).

It would be simple enough to tell users to use the new function
that gives you a thing at or _near_ point, if they want only to
retrieve some text to use as a default value.

You make changes all the time to Emacs code that necessitate
3rd-party code using (if (fboundp 'AAA) BBB CCC) tests.  This
would be no different.  And the function names would be more
correct: "-at-point" would really mean at point, and the
behavior you expect would be properly named "-near-point".

This bug has been declared "minor", but it is not - it makes
the thing-at-point code unusable in a general and precise way.
The fix, however, is trivial.  The pushback from you is major.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Andreas Röhler]

On 20.06.2016 19:50, Drew Adams wrote:
>>> * I agree with Drew that there is neither sexp nor list at point,
>>>     so i would expect I), II), III) and IV) returning nil.
> Tino agrees because he wants to make use of the difference between
> there being a THING at point and there being no THING at point but
> there being a THING at (1- (point)).
>
>> FWIW, I agree with Dmitry: this has been a de-facto behavior long
>> enough to consider it the correct one. If documentation is confusing
>> in that it says otherwise, we should fix the documentation.
> I couldn't disagree more.
>
> It is wrong to consider the current behavior "the correct one",
> regardless of how long it has been in place.  It is wrong because
> you cannot use it in a general and precise way.  It is just broken.
> It has been broken for a long time, but it is broken nevertheless.
>
> The proper thing to do is to:
>
>    1. Fix this bug.  It is a real bug.
>
>    2. Add a new function that provides the same behavior as the
>       broken behavior that is current, or similar.  And call it
>       "-near-point", not "-at-point".
>
> More precisely, for #2 the use case you cite is only to maximize
> grabbing a thing at or _near_ point.  In the case of the current
> code, that means at point or at (1- (point)).  If all you care
> about is grabbing something at either of those positions then the
> code works for you.  If you try to use it more generally, it is
> broken.  IOW, if you actually care about the difference between
> point and (1- (point)) then you are out of luck.
>
> A better, more general, near-point function looks farther from
> point, up to some caller-specified distance (horizontally and
> vertically).  That's the purpose, in my code, of variables
> `tap-near-point-x-distance' & `tap-near-point-y-distance' and
> function `tap-bounds-of-thing-nearest-point' (prefix `tap-' for
> library `thingatpt+.el').
>
> See the arguments given in the bug thread for why #1 is important -
> why `bounds-of-thing-at-point' should be fixed as indicated.
>
> It is important that one be able to use `bounds-of-thing-at-point'
> and `thing-at-point' in a way that is accurate, precise, and
> general, and not only to try to grab something that is near point.
>
> In particular, this matters when the functions are used
> programmatically to handle successive THINGs (of the same type)
> in a buffer or region.  For that, there needs to be a clean
> boundary between THING and no THING at a given position.  You
> need to be able to test whether there is actually a THING at
> point.
>
> I've spent a lot of time with this code, and with a fixed version
> of it, over the years.  My use cases go beyond just trying to come
> up with a default value for a minibuffer read.
>
> That doesn't seem to matter to those in charge here.  Too bad, but
> so be it.  I'll continue to use my code (`thingatpt+.el'), so not
> fixing this bug doesn't affect me much, directly.
>
> But it does affect me, and it affects others too.  For me, it
> means that other code, which makes use of this fix, must also
> conditionally handle the case where a user does not have
> `thingatpt+.el', even if the result is inadequate.
>
> I recommend to users of my code that makes use of thing-at-point
> features that they also use library `thingatpt+.el', but I try to
> let that other code have some semi-workable fallback behavior for
> the case where they do not use `thingatpt+.el'.
>
> So yes, this is an added (and unnecessary) burden for me and
> for users of my code.  But I've been dealing with it for years,
> so it's nothing new.
>
> The real loss, if you do not fix this bug, is for Emacs users who
> do not use `thingatpt+.el'.  They will be unable to do things with
> the `thing-at-point' code that they should be able to do, simply
> because it is broken at the end-of-thing boundary.
>
> But it has proven to be impossible to convince you to apply this
> simple fix.  So be it.
>
> I can only repeat that the proper solution is to fix this bug
> and to also give users a new function that does what you and they
> currently expect of `thing-at-point', for the use case of
> providing a default value for something: be able to grab a thing
> at or near point.
>
> `thingatpt+.el' has existed since 1996, yet you argue that even
> though the thingatpt.el code has this bug, it should not be
> fixed because users (or you) are used to it.
>
>    "Your reasoning seems valid, however by now this behavior is
>     ingrained into my expectations of how thing-at-point should
>     behave."
>
> Your expectations come from using the code only to grab a
> default value from the text.  You don't care about testing for
> a thing at point.  You want grabbing text to work even when
> there is no THING at point but there is a THING at (1- (point)).
>
> It would be simple enough to tell users to use the new function
> that gives you a thing at or _near_ point, if they want only to
> retrieve some text to use as a default value.
>
> You make changes all the time to Emacs code that necessitate
> 3rd-party code using (if (fboundp 'AAA) BBB CCC) tests.  This
> would be no different.  And the function names would be more
> correct: "-at-point" would really mean at point, and the
> behavior you expect would be properly named "-near-point".
>
> This bug has been declared "minor", but it is not - it makes
> the thing-at-point code unusable in a general and precise way.
> The fix, however, is trivial.  The pushback from you is major.
>
>
>

+1

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Eli Zaretskii]

> Date: Mon, 20 Jun 2016 10:50:27 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: f92capac@gmail.com, 9300@debbugs.gnu.org, dgutov@yandex.ru
> 
> > FWIW, I agree with Dmitry: this has been a de-facto behavior long
> > enough to consider it the correct one. If documentation is confusing
> > in that it says otherwise, we should fix the documentation.
> 
> I couldn't disagree more.
> 
> It is wrong to consider the current behavior "the correct one",
> regardless of how long it has been in place.  It is wrong because
> you cannot use it in a general and precise way.  It is just broken.
> It has been broken for a long time, but it is broken nevertheless.

That's immaterial.  It is being used in many places, and it's
obviously useful.

Somewhere in this long discussion there was a suggestion to add new
functions that behave like you want.  I suggest to invest energy in
that direction, instead of more bikeshedding.  That way, everyone is
happy, and you even get to prove you are right, if at some future
point in time we will find that most applications switched to the new
APIs.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Andreas Röhler]

On 20.06.2016 22:04, Eli Zaretskii wrote:
>> Date: Mon, 20 Jun 2016 10:50:27 -0700 (PDT)
>> From: Drew Adams <drew.adams@oracle.com>
>> Cc: f92capac@gmail.com, 9300@debbugs.gnu.org, dgutov@yandex.ru
>>
>>> FWIW, I agree with Dmitry: this has been a de-facto behavior long
>>> enough to consider it the correct one. If documentation is confusing
>>> in that it says otherwise, we should fix the documentation.
>> I couldn't disagree more.
>>
>> It is wrong to consider the current behavior "the correct one",
>> regardless of how long it has been in place.  It is wrong because
>> you cannot use it in a general and precise way.  It is just broken.
>> It has been broken for a long time, but it is broken nevertheless.
> That's immaterial.  It is being used in many places, and it's
> obviously useful.

It is useful, but not in the way of the lemma "at-point". At-point means 
at cursor-position.
What is expected when calling "C-x ="  -- probably not info WRT char 
after, but at cursor position.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Eli Zaretskii]

> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> Date: Tue, 21 Jun 2016 08:14:22 +0200
> 
> On 20.06.2016 22:04, Eli Zaretskii wrote:
> >> Date: Mon, 20 Jun 2016 10:50:27 -0700 (PDT)
> >> From: Drew Adams <drew.adams@oracle.com>
> >> Cc: f92capac@gmail.com, 9300@debbugs.gnu.org, dgutov@yandex.ru
> >>
> >>> FWIW, I agree with Dmitry: this has been a de-facto behavior long
> >>> enough to consider it the correct one. If documentation is confusing
> >>> in that it says otherwise, we should fix the documentation.
> >> I couldn't disagree more.
> >>
> >> It is wrong to consider the current behavior "the correct one",
> >> regardless of how long it has been in place.  It is wrong because
> >> you cannot use it in a general and precise way.  It is just broken.
> >> It has been broken for a long time, but it is broken nevertheless.
> > That's immaterial.  It is being used in many places, and it's
> > obviously useful.
> 
> It is useful, but not in the way of the lemma "at-point". At-point means 
> at cursor-position.

Yes, the de-facto behavior is actually "at or around point".

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Andreas Röhler]

On 21.06.2016 14:50, Eli Zaretskii wrote:
>> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
>> Date: Tue, 21 Jun 2016 08:14:22 +0200
>>
>> On 20.06.2016 22:04, Eli Zaretskii wrote:
>>>> Date: Mon, 20 Jun 2016 10:50:27 -0700 (PDT)
>>>> From: Drew Adams <drew.adams@oracle.com>
>>>> Cc: f92capac@gmail.com, 9300@debbugs.gnu.org, dgutov@yandex.ru
>>>>
>>>>> FWIW, I agree with Dmitry: this has been a de-facto behavior long
>>>>> enough to consider it the correct one. If documentation is confusing
>>>>> in that it says otherwise, we should fix the documentation.
>>>> I couldn't disagree more.
>>>>
>>>> It is wrong to consider the current behavior "the correct one",
>>>> regardless of how long it has been in place.  It is wrong because
>>>> you cannot use it in a general and precise way.  It is just broken.
>>>> It has been broken for a long time, but it is broken nevertheless.
>>> That's immaterial.  It is being used in many places, and it's
>>> obviously useful.
>> It is useful, but not in the way of the lemma "at-point". At-point means
>> at cursor-position.
> Yes, the de-facto behavior is actually "at or around point".

In what programming language users will be satisfied with results which 
are correct or just a little bit false?
Well, assume there are some - so Emacs entered the area of AI :)

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Eli Zaretskii]

> Cc: 9300@debbugs.gnu.org
> From: Andreas Röhler <andreas.roehler@easy-emacs.de>
> Date: Tue, 21 Jun 2016 15:07:52 +0200
> 
> >> It is useful, but not in the way of the lemma "at-point". At-point means
> >> at cursor-position.
> > Yes, the de-facto behavior is actually "at or around point".
> 
> In what programming language users will be satisfied with results which 
> are correct or just a little bit false?

Emacs does this all the time.  It's called DWIM.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> Yes, the de-facto behavior is actually "at or around point".

Around only in the sense of something looser than at.
Not around in the sense of before and after.

To be exact, the current behavior is "at point or at point
minus one".  It is not "at point or at point plus or minus one".

It does not test a _single_ position.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Eli Zaretskii]

> Date: Tue, 21 Jun 2016 06:31:12 -0700 (PDT)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 9300@debbugs.gnu.org
> 
> > Yes, the de-facto behavior is actually "at or around point".
> 
> Around only in the sense of something looser than at.
> Not around in the sense of before and after.

Yes, that's true.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> >>> FWIW, I agree with Dmitry: this has been a de-facto behavior long
> >>> enough to consider it the correct one. If documentation is confusing
> >>> in that it says otherwise, we should fix the documentation.
> >>
> >> I couldn't disagree more.
> >>
> >> It is wrong to consider the current behavior "the correct one",
> >> regardless of how long it has been in place.  It is wrong because
> >> you cannot use it in a general and precise way.  It is just broken.
> >> It has been broken for a long time, but it is broken nevertheless.
> >
> > That's immaterial.  It is being used in many places, and it's
> > obviously useful.
> 
> It is useful, but not in the way of the lemma "at-point". At-point
> means at cursor-position.  What is expected when calling "C-x ="
> -- probably not info WRT char after, but at cursor position.

Yes, but this has all been said before.  Eli knows this, but it does
not sway him.

And at least as important is the fact that "-at-" needs to refer to
only ONE POSITION, not two.

Currently, the function acts the same for both point and point-minus-1.
You cannot tell whether it has determined that the thing it returns
is at point or at point-minus-1.

But this too has all been said before, and Eli knows this too.
It too has not persuaded him.  It seems not to matter whether the
function DTRT.  The only thing that seems to matter to him is that
this broken behavior has been in effect for a while.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> > > FWIW, I agree with Dmitry: this has been a de-facto behavior long
> > > enough to consider it the correct one. If documentation is confusing
> > > in that it says otherwise, we should fix the documentation.
> >
> > I couldn't disagree more.
> >
> > It is wrong to consider the current behavior "the correct one",
> > regardless of how long it has been in place.  It is wrong because
> > you cannot use it in a general and precise way.  It is just broken.
> > It has been broken for a long time, but it is broken nevertheless.
> 
> That's immaterial.  It is being used in many places, and it's
> obviously useful.

It is not being used in _any_ place where it matters whether
there is a thing just before point but not at point.  It cannot
be used in such a place because of this bug.  Can you point to
such a use?

"It is obviously useful" ONLY for cases where you don't really
care _whether_ there is a thing at point and you only want to
get a thing at point or at point-minus-one - you prefer to get a
thing rather than nil, even if the thing is not quite at point.

Sure, such behavior can be useful if that's what one wants, and
"it is being used in many places" - to just grab something to
use as a default value.  But it is not always grabbing a thing
at point.

Just rename this grab-for-defaulting function: "*-near-point"
or "*-at-or-just-before-point".  It is not _at_ point.

> Somewhere in this long discussion there was a suggestion to
> add new functions that behave like you want.

It is not about what I want.  It is what "at point" means.
At_point_or_at_point_minus_one is not the same thing as at_point.
Currently the behavior is the former, not the latter.  That most
people don't notice or care about that is immaterial.

I already provided a correct implementation for at-point behavior.

And I already provided an implementation for near-point behavior,
albeit a better one than just at_point_or_at_point_minus_one.  For
the latter, you already have the current, broken implementation -
just rename it "*-near-point*".

> I suggest to invest energy in that direction, instead of more
> bikeshedding.

I'm not bikeshedding.  And I'll thank you to drop such a
characterization.  This is a real bug.  That you don't recognize
it is too bad.

I already invested energy in providing the function needed,
i.e., in fixing, as well as reporting, this bug.  And I (and
others) have been using the fix for decades.

I pointed you to code that provides not only the needed behavior
for `bounds-of-thing-at-point' but also other improvements for
thingatpt.el.  If you are uninterested, that's too bad.

> That way, everyone is happy, and you even get to prove you
> are right, if at some future point in time we will find that
> most applications switched to the new APIs.

Unlike some, I'm not really interested in proving I'm right.
But if you are interested, the proof is that you cannot use
the current code to distinguish whether there is a thing at
point from whether there is a thing at point-minus-one.

Can you point to a single use of thingatpt.el code that does
more than just use a thing at-or-just-before point as a
default value?  Can you point to a single use that really
cares about whether there actually is a thing at point, and
is not just trying to grab a thing near point?  A use where
a nil value is actually useful and taken into account as
more than simply a lack of a default value?

I don't think you'll find any (other than uses of my code).
This bug prevents using thing-at-point that way (general,
precise).  It confounds a thing at point with a thing at
point-minus-one.

I have what I need, in my own code.  You've heard in this
bug thread from a couple other users as well.  Lousy
bikeshedders too, no doubt.  But one of them has written his
own code that builds on thingatpt.el, and has clearly been
interested in thing-at-point and knowledgable about it for
years.  The other has contributed several uncontroversial
and non-bikeshed bug fixes to Emacs recently.

You will not hear from lots of others about this, naturally.
If one does not try to use thing-at-point to actually see
whether there is a thing at point then one will not even
notice this bug.

But if the bug is fixed then all kinds of possibilities
open up for handling multiple occurrences of a thing etc.,
possibilities that are precluded today, simply because the
code cannot tell the difference between there being a thing
at point and there being a thing at point-minus-one.
Dommage.

And if you fix this bug what happens to those who are using
the code today only to get a default value?  If point is
after a thing, and there is NO thing at point, then they
will get no default value.

If they complain about that in some context, you have only
to point them to your new `*-near-point' function for the
behavior they think they miss.  And for any occurrences in
Emacs code where you think that is the behavior you want,
just use the new function.

It's pretty simple, really.  If you want to improve Emacs
for thing-at-point, apply the one-off-bug fix and also
offer another function that maximizes returning a thing
rather than precisely getting a thing at point or returning
nil if there is none there.

My suggestion for the `*-near-point' function would be to
do something like what I did, letting users and code
control how near "near" is in any given context.  But if
you want to keep it rudimentary, where "near" means only
at point or at point-minus-one, then just rename the code
you have now to `*-near-point'.

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Noam Postavsky]

Isn't the safest and simplest solution to rename the current
*-at-point to *-near-point, keep the *-at-point names as obsolete
aliases, and add new *-precisely-at-point functions with the
definitions from the thingatpt+ library? That way, current users of
*-at-point who happen to be relying the on *-near-point functionality
won't break. Only downside I can see is a slightly longer name for the
*-precisely-at-point callers, but that doesn't seem too bad.

What you do think?

bug#9300: 24.0.50; `bounds-of-thing-at-point' does not return nil when just after THING

[Drew Adams]

> Isn't the safest and simplest solution to rename the current
> *-at-point to *-near-point, keep the *-at-point names as obsolete
> aliases, and add new *-precisely-at-point functions with the
> definitions from the thingatpt+ library? That way, current users of
> *-at-point who happen to be relying the on *-near-point functionality
> won't break. Only downside I can see is a slightly longer name for the
> *-precisely-at-point callers, but that doesn't seem too bad.
> 
> What you do think?

That's possible (and I appreciate your trying to find a
diplomatic way to get this bug fixed), but I don't think
that is the best approach.

We should aim to have a reasonable name, not just something
that doesn't conflict.  There is little sense in abandoning
the most reasonable name for this, IMO.  This is what the
library is intended for: returning a thing at point.

If the name "*-at-point" is kept (for behavior that is really
at point) then the worst that will happen is that some users
might complain that they no longer get a thing that is before
point but not also at point.

And that won't even happen for distributed Emacs code, which
should replace any appropriate calls to *-at by *-near (where
appropriate means that you really do want to retrieve the
thing before point as well as the thing at point).

This is a simple off-by-one bug.  It really should not require
anything to be deprecated.  Just because someone might have
gotten used to the bugged behavior is not a good reason not
to fix this bug.

If going the direction you suggest is the best compromise
that can be had, I'd suggest using the name *-at-pt instead
of *-precisely-at-point.  IOW, just change "point" to "pt".

That's not the fix I prefer, and it hurts discoverability
(matches against "point"), and it doesn't jibe with names
such as `find-file-at-point', but I think it's better than
something as artificial as *-precisely-at-point.  That name
just makes one wonder.