bug#64739: (forward-thing 'number) doesn't work with decimal numbers

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Joseph Turner via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hello,

Given the following number with point at ★:

★12.34

number-at-point correctly returns "12.34".

However,(forward-thing 'number) puts point just before the period:

12★.34

while I'd expect:

12.34★

(get 'number 'forward-op)
     ⇒ forward-word

forward-word doesn't seem to be the correct forward-op for numbers.

Joseph

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Eli Zaretskii]

> Date: Wed, 19 Jul 2023 23:19:27 -0700
> From:  Joseph Turner via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Hello,
> 
> Given the following number with point at ★:
> 
> ★12.34
> 
> number-at-point correctly returns "12.34".
> 
> However,(forward-thing 'number) puts point just before the period:
> 
> 12★.34
> 
> while I'd expect:
> 
> 12.34★
> 
> (get 'number 'forward-op)
>      ⇒ forward-word
> 
> forward-word doesn't seem to be the correct forward-op for numbers.

That depends on the buffer's syntax table, though.

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Roland Winkler]

On Thu, Jul 20 2023, Eli Zaretskii wrote:
>> forward-word doesn't seem to be the correct forward-op for numbers.
>
> That depends on the buffer's syntax table, though.

I am frequently running into similar problems when I deal with numbers,
but the syntax table is not designed for this.  Would it make sense if
(forward-thing 'number) temporarily used a customized synatx table?

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Eli Zaretskii]

> From: Roland Winkler <winkler@gnu.org>
> Cc: Joseph Turner <joseph@breatheoutbreathe.in>,  64739@debbugs.gnu.org
> Date: Fri, 21 Jul 2023 07:28:29 -0500
> 
> On Thu, Jul 20 2023, Eli Zaretskii wrote:
> >> forward-word doesn't seem to be the correct forward-op for numbers.
> >
> > That depends on the buffer's syntax table, though.
> 
> I am frequently running into similar problems when I deal with numbers,
> but the syntax table is not designed for this.  Would it make sense if
> (forward-thing 'number) temporarily used a customized synatx table?

It should use its own function, if you ask me.  forward-word is a
kludge.

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Eli Zaretskii]

> Cc: 64739@debbugs.gnu.org, joseph@breatheoutbreathe.in
> Date: Fri, 21 Jul 2023 15:48:07 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > From: Roland Winkler <winkler@gnu.org>
> > Cc: Joseph Turner <joseph@breatheoutbreathe.in>,  64739@debbugs.gnu.org
> > Date: Fri, 21 Jul 2023 07:28:29 -0500
> > 
> > On Thu, Jul 20 2023, Eli Zaretskii wrote:
> > >> forward-word doesn't seem to be the correct forward-op for numbers.
> > >
> > > That depends on the buffer's syntax table, though.
> > 
> > I am frequently running into similar problems when I deal with numbers,
> > but the syntax table is not designed for this.  Would it make sense if
> > (forward-thing 'number) temporarily used a customized synatx table?
> 
> It should use its own function, if you ask me.  forward-word is a
> kludge.

To clarify: I meant that using forward-word in this case is a kludge.

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Roland Winkler]

On Fri, Jul 21 2023, Eli Zaretskii wrote:
>> It should use its own function, if you ask me.  forward-word is a
>> kludge.
>
> To clarify: I meant that using forward-word in this case is a kludge.

I am not familiar with how forward-thing is implemented.  A dedicated
function has been my personal solution for this problem that has been
serving me well for many years.  So one can rephrase: How common a
problem is this that it may (not?) justify a solution in the standard
emacs code?

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Eli Zaretskii]

> From: Roland Winkler <winkler@gnu.org>
> Cc: 64739@debbugs.gnu.org,  joseph@breatheoutbreathe.in
> Date: Fri, 21 Jul 2023 09:14:48 -0500
> 
> On Fri, Jul 21 2023, Eli Zaretskii wrote:
> >> It should use its own function, if you ask me.  forward-word is a
> >> kludge.
> >
> > To clarify: I meant that using forward-word in this case is a kludge.
> 
> I am not familiar with how forward-thing is implemented.  A dedicated
> function has been my personal solution for this problem that has been
> serving me well for many years.  So one can rephrase: How common a
> problem is this that it may (not?) justify a solution in the standard
> emacs code?

By "its own function" I meant that we implement it in Emacs, not that
you implement it in your personal customizations.

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> > > On Thu, Jul 20 2023, Eli Zaretskii wrote:
> > > >> forward-word doesn't seem to be the correct forward-op for numbers.
> > > >
> > > > That depends on the buffer's syntax table, though.
> > >
> > > I am frequently running into similar problems when I deal with numbers,
> > > but the syntax table is not designed for this.  Would it make sense if
> > > (forward-thing 'number) temporarily used a customized synatx table?
> >
> > It should use its own function, if you ask me.  forward-word is a
> > kludge.
>
> To clarify: I meant that using forward-word in this case is a kludge.

It had been added by Lars when fixing Bug#54555 "number is not an
accepted value for THING in bounds-of-thing-at-point" last year.
Probably just not being aware of this problem with the decimal point.
So: yes, a klugde, it's too simple.


Michael.

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Joseph Turner via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

> By "its own function" I meant that we implement it in Emacs, not that
> you implement it in your personal customizations.

I see that the definition of number-at-point includes two magic regexps,
one matching hex numbers and another matching decimal numbers. Would it
make sense to move those regexps into defconsts, and then create a new
function forward-number which uses those new constants?

Joseph

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Eli Zaretskii]

> From: Joseph Turner <joseph@breatheoutbreathe.in>
> Cc: Roland Winkler <winkler@gnu.org>, 64739@debbugs.gnu.org
> Date: Sat, 22 Jul 2023 20:36:16 -0700
> 
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > By "its own function" I meant that we implement it in Emacs, not that
> > you implement it in your personal customizations.
> 
> I see that the definition of number-at-point includes two magic regexps,
> one matching hex numbers and another matching decimal numbers. Would it
> make sense to move those regexps into defconsts, and then create a new
> function forward-number which uses those new constants?

That's probably one idea, yes.  (But note that those regexps don't
support numbers like "1.0e2".)

Another idea is to define a special-purpose syntax table and use that
with forward-word.

Yet another idea is to use 'read', since the Lisp reader already knows
how to read numbers.

And there probably are other ideas as well.

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Joseph Turner via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Joseph Turner <joseph@breatheoutbreathe.in>
>> Cc: Roland Winkler <winkler@gnu.org>, 64739@debbugs.gnu.org
>> Date: Sat, 22 Jul 2023 20:36:16 -0700
>>
>> > By "its own function" I meant that we implement it in Emacs, not that
>> > you implement it in your personal customizations.
>>
>> I see that the definition of number-at-point includes two magic regexps,
>> one matching hex numbers and another matching decimal numbers. Would it
>> make sense to move those regexps into defconsts, and then create a new
>> function forward-number which uses those new constants?
>
> That's probably one idea, yes.  (But note that those regexps don't
> support numbers like "1.0e2".)

I agree that the regexp solution is not ideal.

> Another idea is to define a special-purpose syntax table and use that
> with forward-word.

Did you have something like this in mind?

(defvar number-syntax-table
  (let ((st (make-syntax-table)))
    (modify-syntax-entry ?. "w " st)
    ;; ?- and ?+ could indicate the sign at the beginning of a number or
    ;; they could indicate the sign of an exponent like 1.5e-6
    (modify-syntax-entry ?- "w " st)
    (modify-syntax-entry ?+ "w " st)
    ;; ?# acts as a prefix for integers in bases other than 10.
    (modify-syntax-entry ?# "w " st)
    st)
  "Syntax table used to detect word boundaries for numbers.")

(defun forward-number (&optional n)
  (interactive)
  (with-syntax-table number-syntax-table
    (forward-word-strictly n)))

(defun backward-number (&optional n)
  (interactive)
  (with-syntax-table number-syntax-table
    (forward-word-strictly n)))

;; We don't need the following two lines, since forward-thing uses
;; intern-soft to get these function names anyway.

(put 'number 'forward-op 'forward-number)
(put 'number 'beginning-op 'backward-number)

What would we use for end-op? Is it necessary?

Also I notice that the current implementation of number-at-point fails
to recognize the following as a single number:

-1.5e+6

Could we do something like:

(defun number-at-point (&optional no-properties)
  (with-syntax-table number-syntax-table
    (word-at-point no-properties)))

(put 'number 'thing-at-point 'number-at-point)

I'd appreciate feedback on the syntax table above. It incorrectly
recognizes some non-number strings as numbers:

a-a+a-
#abc#abc

> Yet another idea is to use 'read', since the Lisp reader already knows
> how to read numbers.

Sounds interesting! I'm not sure how to approach a solution like that.
How would you let the Lisp reader know where to start and stop reading?

Joseph

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Eli Zaretskii]

> From: Joseph Turner <joseph@breatheoutbreathe.in>
> Cc: winkler@gnu.org, 64739@debbugs.gnu.org
> Date: Sun, 23 Jul 2023 12:03:50 -0700
> 
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That's probably one idea, yes.  (But note that those regexps don't
> > support numbers like "1.0e2".)
> 
> I agree that the regexp solution is not ideal.
> 
> > Another idea is to define a special-purpose syntax table and use that
> > with forward-word.
> 
> Did you have something like this in mind?

More or less.  TBH, I didn't think about this too much, so maybe this
idea doesn't "hold water".

> > Yet another idea is to use 'read', since the Lisp reader already knows
> > how to read numbers.
> 
> Sounds interesting! I'm not sure how to approach a solution like that.
> How would you let the Lisp reader know where to start and stop reading?

Start is easy: 'read' always starts at point.  End might give us
trouble, but in general 'read' reads one expression, so it should stop
at the end of the number, no?

Anyway, I was just brain-storming.  A real solution will need more
thought and more testing.  Perhaps Stefan (CC'ed) has some ideas or
inputs.

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> Anyway, I was just brain-storming.  A real solution will need more
> thought and more testing.  Perhaps Stefan (CC'ed) has some ideas or
> inputs.

FWIW, I've always found the `thingatpt.el` library to be ..hmm.. wobbly?
I'd welcome someone investing time to straighten it up (maybe
starting by extending it so it can return "thing around region" and use
that as the basis for "at point").


        Stefan

bug#64739: (forward-thing 'number) doesn't work with decimal numbers

[Joseph Turner via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> FWIW, I've always found the `thingatpt.el` library to be ..hmm.. wobbly?
> I'd welcome someone investing time to straighten it up (maybe
> starting by extending it so it can return "thing around region" and use
> that as the basis for "at point").

Would you elaborate on this design idea?