bug#679: 23.0.60; Function json-read-number does not handle complete Javascript spec

bug#679: 23.0.60; Function json-read-number does not handle complete Javascript spec

[Chong Yidong]

Hi Edward,

Could you take a look at this bug report?

Thanks.

---------------

In json.el, the 'json-read-number' function is intended to read the
textual representation of Javascript numbers.

(defun json-read-number ()
  "Read the JSON number following point.
N.B.: Only numbers which can fit in Emacs Lisp's native number
representation will be parsed correctly."
  (if (char-equal (json-peek) ?-)
      (progn
        (json-advance)
        (- 0 (json-read-number)))
    (if (looking-at "[0-9]+\\([.][0-9]+\\)?\\([eE][+-]?[0-9]+\\)?")
        (progn
          (goto-char (match-end 0))
          (string-to-number (match-string 0)))
      (signal 'json-number-format (list (point))))))

The Javascript spec
(http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Guide:Literals#Floating-Point_Literals)
says the following about floating-point numbers:

| A floating-point literal can have the following parts:
| 
|     * A decimal integer which can be signed (preceded by "+" or "-"),
|     * A decimal point ("."),
|     * A fraction (another decimal number),
|     * An exponent. 
| 
| The exponent part is an "e" or "E" followed by an integer, which can
| be signed (preceded by "+" or "-"). A floating-point literal must have
| at least one digit and either a decimal point or "e" (or "E"). 

This means that

  1.

is a valid floating point-literal per the spec, but the regular
expression above does not match the complete text---the regular
expression requires digits after the decimal point.  This ends up
matched as a simple integer, leaving the decimal point in the input
stream and causing a later syntax error.  Similarly

  .1

is a valid floating point-literal illegal that Emacs rejects, because
the regular expression requires digits before the decimal point.
Testing with the error console in Firefox confirms that the two
examples above are valid syntax.

The parser function needs to be fixed to handle the complete syntax.

bug#679: 23.0.60; Function json-read-number does not handle complete Javascript spec

[Edward O'Connor]

Hi,

> Could you take a look at this bug report?

Sure. I'll have a fix to you in the next few days.


Ted

bug#679: 23.0.60; Function json-read-number does not handle complete Javascript spec

[Kevin Rodgers]

Edward O'Connor wrote:
> Hi,
> 
>> Could you take a look at this bug report?
> 
> Sure. I'll have a fix to you in the next few days.

Here's an improved version: Besides recognizing the D. and .D number
forms mentioned in the original bug report, and the positive signed
forms mentioned in my first response, it also

(a) recognizes hexadecimal and octal integers per 
http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Guide:Literals#Integers

(b) correctly fails to recognize more than 1 leading sign (e.g. --, ++, 
+-, -+)

(c) correctly fails to recognize exponent forms without a preceding number.

I hope you'll use it.

(defun json-read-number (&optional sign base)
   "Read the JSON number following point.
The optional SIGN and BASE arguments are for internal use.

N.B.: Only numbers which can fit in Emacs Lisp's native number
representation will be parsed correctly."
   ;; If SIGN is non-nil, the number is explicitly signed.
   ;; If BASE is null, read a decimal integer or floating point;
   ;; if BASE is 16, read a hexadecimal integer;
   ;; if BASE is 8, read an octal integer.
   (let ((number-regexp
	 (cond ((null base)
		"\\([0-9]+\\)?\\(\\.[0-9]+\\)?\\([Ee][+-]?[0-9]+\\)?")
	       ((equal base 8) "[0-7]+")
	       ((equal base 16) "[0-9A-Fa-f]+"))))
     (cond ((and (null sign) (char-equal (json-peek) ?-))
	   (json-advance)
	   (- (json-read-number t base)))
	  ((and (null sign) (char-equal (json-peek) ?+))
	   (json-advance)
	   (json-read-number t base))
	  ((and (null base) (looking-at "0[Xx]"))
	   (json-advance 2)
	   (json-read-number t 16))
	  ((and (null base) (looking-at "0[0-7]"))
	   (json-advance 1)
	   (json-read-number t 8))
	  ((and (looking-at number-regexp)
		(or base
		    (match-beginning 1)
		    (match-beginning 2)))
	   (goto-char (match-end 0))
	   (string-to-number (match-string 0) base))
	  (t (signal 'json-number-format (list (point)))))))

Thanks,

-- 
Kevin Rodgers
Denver, Colorado, USA

bug#679: 23.0.60; Function json-read-number does not handle complete Javascript spec

[Edward O'Connor]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

> Here's an improved version: Besides recognizing the D. and .D number
> forms mentioned in the original bug report, and the positive signed
> forms mentioned in my first response, it also
>
> (a) recognizes hexadecimal and octal integers per
> http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Guide:Literals#Integers

JSON disallows octal and hexadecimal notation...

> I hope you'll use it.

I've attached a modified version, which disallows octal & hexadecimal.


Thanks,
Ted

[-- Attachment #2: json-number.diff --]
[-- Type: application/octet-stream, Size: 1875 bytes --]

? json-number.diff
? qed.diff
Index: lisp/json.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/json.el,v
retrieving revision 1.5
diff -u -u -r1.5 json.el
--- lisp/json.el	8 May 2008 03:37:07 -0000	1.5
+++ lisp/json.el	19 Aug 2008 05:35:58 -0000
@@ -221,19 +221,27 @@
 
 ;; Number parsing
 
-(defun json-read-number ()
-  "Read the JSON number following point.
+(defun json-read-number (&optional sign)
+ "Read the JSON number following point.
+The optional SIGN  argument is for internal use.
+
 N.B.: Only numbers which can fit in Emacs Lisp's native number
 representation will be parsed correctly."
-  (if (char-equal (json-peek) ?-)
-      (progn
-        (json-advance)
-        (- 0 (json-read-number)))
-    (if (looking-at "[0-9]+\\([.][0-9]+\\)?\\([eE][+-]?[0-9]+\\)?")
-        (progn
+ ;; If SIGN is non-nil, the number is explicitly signed.
+ (let ((number-regexp
+        "\\([0-9]+\\)?\\(\\.[0-9]+\\)?\\([Ee][+-]?[0-9]+\\)?"))
+   (cond ((and (null sign) (char-equal (json-peek) ?-))
+          (json-advance)
+          (- (json-read-number t)))
+         ((and (null sign) (char-equal (json-peek) ?+))
+          (json-advance)
+          (json-read-number t))
+         ((and (looking-at number-regexp)
+               (or (match-beginning 1)
+                   (match-beginning 2)))
           (goto-char (match-end 0))
           (string-to-number (match-string 0)))
-      (signal 'json-number-format (list (point))))))
+         (t (signal 'json-number-format (list (point)))))))
 
 ;; Number encoding
 
@@ -470,7 +478,7 @@
            (?\" json-read-string))))
     (mapc (lambda (char)
             (push (list char 'json-read-number) table))
-          '(?- ?0 ?1 ?2 ?3 ?4 ?5 ?6 ?7 ?8 ?9))
+          '(?- ?+ ?. ?0 ?1 ?2 ?3 ?4 ?5 ?6 ?7 ?8 ?9))
     table)
   "Readtable for JSON reader.")
 

bug#679: 23.0.60; Function json-read-number does not handle complete Javascript spec

[sand]

----

In json.el, the 'json-read-number' function is intended to read the
textual representation of Javascript numbers.

(defun json-read-number ()
  "Read the JSON number following point.
N.B.: Only numbers which can fit in Emacs Lisp's native number
representation will be parsed correctly."
  (if (char-equal (json-peek) ?-)
      (progn
        (json-advance)
        (- 0 (json-read-number)))
    (if (looking-at "[0-9]+\\([.][0-9]+\\)?\\([eE][+-]?[0-9]+\\)?")
        (progn
          (goto-char (match-end 0))
          (string-to-number (match-string 0)))
      (signal 'json-number-format (list (point))))))

The Javascript spec
(http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Guide:Literals#Floating-Point_Literals)
says the following about floating-point numbers:

| A floating-point literal can have the following parts:
| 
|     * A decimal integer which can be signed (preceded by "+" or "-"),
|     * A decimal point ("."),
|     * A fraction (another decimal number),
|     * An exponent. 
| 
| The exponent part is an "e" or "E" followed by an integer, which can
| be signed (preceded by "+" or "-"). A floating-point literal must have
| at least one digit and either a decimal point or "e" (or "E"). 

This means that

  1.

is a valid floating point-literal per the spec, but the regular
expression above does not match the complete text---the regular
expression requires digits after the decimal point.  This ends up
matched as a simple integer, leaving the decimal point in the input
stream and causing a later syntax error.  Similarly

  .1

is a valid floating point-literal illegal that Emacs rejects, because
the regular expression requires digits before the decimal point.
Testing with the error console in Firefox confirms that the two
examples above are valid syntax.

The parser function needs to be fixed to handle the complete syntax.

----


In GNU Emacs 23.0.60.1 (i486-pc-linux-gnu, GTK+ Version 2.12.11)
 of 2008-07-27 on elegiac, modified by Debian
 (emacs-snapshot package, version 1:20080727-1)
Windowing system distributor `The X.Org Foundation', version 11.0.10402000
configured using `configure  '--build' 'i486-linux-gnu' '--host' 'i486-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' '--libexecdir=/usr/lib' '--localstatedir=/var' '--infodir=/usr/share/info' '--mandir=/usr/share/man' '--with-pop=yes' '--enable-locallisppath=/etc/emacs-snapshot:/etc/emacs:/usr/local/share/emacs/23.0.60/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.0.60/site-lisp:/usr/share/emacs/site-lisp' '--with-x=yes' '--with-x-toolkit=gtk' 'build_alias=i486-linux-gnu' 'host_alias=i486-linux-gnu' 'CFLAGS=-DDEBIAN -DSITELOAD_PURESIZE_EXTRA=5000 -g -O2' 'LDFLAGS=-g -Wl,--as-needed' 'CPPFLAGS=''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: POSIX
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default-enable-multibyte-characters: t

Major mode: VM

Minor modes in effect:
  shell-dirtrack-mode: t
  auto-image-file-mode: t
  show-paren-mode: t
  icomplete-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  global-auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

bug#679: 23.0.60; Function json-read-number does not handle complete Javascript spec

[ＯＦＦＩＣＥ ZERO]

no thank you

----- Original Message ----- 
From: <sand@blarg.net>
To: <emacs-pretest-bug@gnu.org>
Cc: <rfrancoise@debian.org>
Sent: Saturday, August 09, 2008 4:05 PM
Subject: bug#679: 23.0.60;Function json-read-number does not handle complete 
Javascript spec


> ----
>
> In json.el, the 'json-read-number' function is intended to read the
> textual representation of Javascript numbers.
>
> (defun json-read-number ()
>  "Read the JSON number following point.
> N.B.: Only numbers which can fit in Emacs Lisp's native number
> representation will be parsed correctly."
>  (if (char-equal (json-peek) ?-)
>      (progn
>        (json-advance)
>        (- 0 (json-read-number)))
>    (if (looking-at "[0-9]+\\([.][0-9]+\\)?\\([eE][+-]?[0-9]+\\)?")
>        (progn
>          (goto-char (match-end 0))
>          (string-to-number (match-string 0)))
>      (signal 'json-number-format (list (point))))))
>
> The Javascript spec
> (http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Guide:Literals#Floating-Point_Literals)
> says the following about floating-point numbers:
>
> | A floating-point literal can have the following parts:
> |
> |     * A decimal integer which can be signed (preceded by "+" or "-"),
> |     * A decimal point ("."),
> |     * A fraction (another decimal number),
> |     * An exponent.
> |
> | The exponent part is an "e" or "E" followed by an integer, which can
> | be signed (preceded by "+" or "-"). A floating-point literal must have
> | at least one digit and either a decimal point or "e" (or "E").
>
> This means that
>
>  1.
>
> is a valid floating point-literal per the spec, but the regular
> expression above does not match the complete text---the regular
> expression requires digits after the decimal point.  This ends up
> matched as a simple integer, leaving the decimal point in the input
> stream and causing a later syntax error.  Similarly
>
>  .1
>
> is a valid floating point-literal illegal that Emacs rejects, because
> the regular expression requires digits before the decimal point.
> Testing with the error console in Firefox confirms that the two
> examples above are valid syntax.
>
> The parser function needs to be fixed to handle the complete syntax.
>
> ----
>
>
> In GNU Emacs 23.0.60.1 (i486-pc-linux-gnu, GTK+ Version 2.12.11)
> of 2008-07-27 on elegiac, modified by Debian
> (emacs-snapshot package, version 1:20080727-1)
> Windowing system distributor `The X.Org Foundation', version 11.0.10402000
> configured using `configure  '--build' 'i486-linux-gnu' '--host' 
> 'i486-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' 
> '--libexecdir=/usr/lib' '--localstatedir=/var' '--infodir=/usr/share/info' 
> '--mandir=/usr/share/man' '--with-pop=yes' 
> '--enable-locallisppath=/etc/emacs-snapshot:/etc/emacs:/usr/local/share/emacs/23.0.60/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.0.60/site-lisp:/usr/share/emacs/site-lisp' 
> '--with-x=yes' '--with-x-toolkit=gtk' 'build_alias=i486-linux-gnu' 
> 'host_alias=i486-linux-gnu' 
> 'CFLAGS=-DDEBIAN -DSITELOAD_PURESIZE_EXTRA=5000 -g -O2' 
> 'LDFLAGS=-g -Wl,--as-needed' 'CPPFLAGS=''
>
> Important settings:
>  value of $LC_ALL: nil
>  value of $LC_COLLATE: POSIX
>  value of $LC_CTYPE: nil
>  value of $LC_MESSAGES: nil
>  value of $LC_MONETARY: nil
>  value of $LC_NUMERIC: nil
>  value of $LC_TIME: nil
>  value of $LANG: en_US.UTF-8
>  value of $XMODIFIERS: nil
>  locale-coding-system: utf-8-unix
>  default-enable-multibyte-characters: t
>
> Major mode: VM
>
> Minor modes in effect:
>  shell-dirtrack-mode: t
>  auto-image-file-mode: t
>  show-paren-mode: t
>  icomplete-mode: t
>  tooltip-mode: t
>  mouse-wheel-mode: t
>  menu-bar-mode: t
>  file-name-shadow-mode: t
>  global-font-lock-mode: t
>  font-lock-mode: t
>  blink-cursor-mode: t
>  global-auto-composition-mode: t
>  auto-encryption-mode: t
>  auto-compression-mode: t
>  line-number-mode: t
>  transient-mark-mode: t
>
>
>
> 

bug#679: 23.0.60; Function json-read-number does not handle complete Javascript spec

[ＯＦＦＩＣＥ ZERO]

Do'nt   send  me mail  !!!
no tkank you!!

----- Original Message ----- 
From: <sand@blarg.net>
To: <emacs-pretest-bug@gnu.org>
Cc: <rfrancoise@debian.org>
Sent: Saturday, August 09, 2008 4:05 PM
Subject: bug#679: 23.0.60;Function json-read-number does not handle complete 
Javascript spec


> ----
>
> In json.el, the 'json-read-number' function is intended to read the
> textual representation of Javascript numbers.
>
> (defun json-read-number ()
>  "Read the JSON number following point.
> N.B.: Only numbers which can fit in Emacs Lisp's native number
> representation will be parsed correctly."
>  (if (char-equal (json-peek) ?-)
>      (progn
>        (json-advance)
>        (- 0 (json-read-number)))
>    (if (looking-at "[0-9]+\\([.][0-9]+\\)?\\([eE][+-]?[0-9]+\\)?")
>        (progn
>          (goto-char (match-end 0))
>          (string-to-number (match-string 0)))
>      (signal 'json-number-format (list (point))))))
>
> The Javascript spec
> (http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Guide:Literals#Floating-Point_Literals)
> says the following about floating-point numbers:
>
> | A floating-point literal can have the following parts:
> |
> |     * A decimal integer which can be signed (preceded by "+" or "-"),
> |     * A decimal point ("."),
> |     * A fraction (another decimal number),
> |     * An exponent.
> |
> | The exponent part is an "e" or "E" followed by an integer, which can
> | be signed (preceded by "+" or "-"). A floating-point literal must have
> | at least one digit and either a decimal point or "e" (or "E").
>
> This means that
>
>  1.
>
> is a valid floating point-literal per the spec, but the regular
> expression above does not match the complete text---the regular
> expression requires digits after the decimal point.  This ends up
> matched as a simple integer, leaving the decimal point in the input
> stream and causing a later syntax error.  Similarly
>
>  .1
>
> is a valid floating point-literal illegal that Emacs rejects, because
> the regular expression requires digits before the decimal point.
> Testing with the error console in Firefox confirms that the two
> examples above are valid syntax.
>
> The parser function needs to be fixed to handle the complete syntax.
>
> ----
>
>
> In GNU Emacs 23.0.60.1 (i486-pc-linux-gnu, GTK+ Version 2.12.11)
> of 2008-07-27 on elegiac, modified by Debian
> (emacs-snapshot package, version 1:20080727-1)
> Windowing system distributor `The X.Org Foundation', version 11.0.10402000
> configured using `configure  '--build' 'i486-linux-gnu' '--host' 
> 'i486-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' 
> '--libexecdir=/usr/lib' '--localstatedir=/var' '--infodir=/usr/share/info' 
> '--mandir=/usr/share/man' '--with-pop=yes' 
> '--enable-locallisppath=/etc/emacs-snapshot:/etc/emacs:/usr/local/share/emacs/23.0.60/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.0.60/site-lisp:/usr/share/emacs/site-lisp' 
> '--with-x=yes' '--with-x-toolkit=gtk' 'build_alias=i486-linux-gnu' 
> 'host_alias=i486-linux-gnu' 
> 'CFLAGS=-DDEBIAN -DSITELOAD_PURESIZE_EXTRA=5000 -g -O2' 
> 'LDFLAGS=-g -Wl,--as-needed' 'CPPFLAGS=''
>
> Important settings:
>  value of $LC_ALL: nil
>  value of $LC_COLLATE: POSIX
>  value of $LC_CTYPE: nil
>  value of $LC_MESSAGES: nil
>  value of $LC_MONETARY: nil
>  value of $LC_NUMERIC: nil
>  value of $LC_TIME: nil
>  value of $LANG: en_US.UTF-8
>  value of $XMODIFIERS: nil
>  locale-coding-system: utf-8-unix
>  default-enable-multibyte-characters: t
>
> Major mode: VM
>
> Minor modes in effect:
>  shell-dirtrack-mode: t
>  auto-image-file-mode: t
>  show-paren-mode: t
>  icomplete-mode: t
>  tooltip-mode: t
>  mouse-wheel-mode: t
>  menu-bar-mode: t
>  file-name-shadow-mode: t
>  global-font-lock-mode: t
>  font-lock-mode: t
>  blink-cursor-mode: t
>  global-auto-composition-mode: t
>  auto-encryption-mode: t
>  auto-compression-mode: t
>  line-number-mode: t
>  transient-mark-mode: t
>
>
>
> 

bug#679: 23.0.60; Function json-read-number does not handle complete Javascript spec

[Kevin Rodgers]

sand@blarg.net wrote:
> In json.el, the 'json-read-number' function is intended to read the
> textual representation of Javascript numbers.
> 
> (defun json-read-number ()
>   "Read the JSON number following point.
> N.B.: Only numbers which can fit in Emacs Lisp's native number
> representation will be parsed correctly."
>   (if (char-equal (json-peek) ?-)
>       (progn
>         (json-advance)
>         (- 0 (json-read-number)))
>     (if (looking-at "[0-9]+\\([.][0-9]+\\)?\\([eE][+-]?[0-9]+\\)?")
>         (progn
>           (goto-char (match-end 0))
>           (string-to-number (match-string 0)))
>       (signal 'json-number-format (list (point))))))
> 
> The Javascript spec
> (http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Guide:Literals#Floating-Point_Literals)
> says the following about floating-point numbers:
> 
> | A floating-point literal can have the following parts:
> | 
> |     * A decimal integer which can be signed (preceded by "+" or "-"),
> |     * A decimal point ("."),
> |     * A fraction (another decimal number),
> |     * An exponent. 
> | 
> | The exponent part is an "e" or "E" followed by an integer, which can
> | be signed (preceded by "+" or "-"). A floating-point literal must have
> | at least one digit and either a decimal point or "e" (or "E"). 
> 
> This means that
> 
>   1.
> 
> is a valid floating point-literal per the spec, but the regular
> expression above does not match the complete text---the regular
> expression requires digits after the decimal point.  This ends up
> matched as a simple integer, leaving the decimal point in the input
> stream and causing a later syntax error.  Similarly
> 
>   .1
> 
> is a valid floating point-literal illegal that Emacs rejects, because
> the regular expression requires digits before the decimal point.
> Testing with the error console in Firefox confirms that the two
> examples above are valid syntax.
> 
> The parser function needs to be fixed to handle the complete syntax.

It also means that "+1" is valid but not recognized.  That's easy to fix:

   (cond ((char-equal (json-peek) ?-)
	 (json-advance)
	 (- (json-read-number)))
	((char-equal (json-peek) ?+)
	 (json-advance)
	 (json-read-number))
	((looking-at "[0-9]+\\([.][0-9]+\\)?\\([eE][+-]?[0-9]+\\)?")
...)
	(t (signal 'json-number-format (list (point)))))

The spec you cite also says:

| More succinctly, the syntax is:
|
| [digits][.digits][(E|e)[(+|-)]digits]

which would be translated literally as

"\\([0-9]+\\)?\\(\\.[0-9]+\\)?\\([Ee][+-]?[0-9]+\\)?"

The only difference between that and the regexp in json-read-number is
the optionality of the leading digits and the representation of the
decimal point.

But the potential problem with changing the regexp like that is that it
now matches the empty string, which the spec does not allow:

| A floating-point literal must have at least one digit and either a 
decimal point or "e" (or "E").

So, does this give the desired results:

(defun json-read-number ()
   "Read the JSON number following point.
N.B.: Only numbers which can fit in Emacs Lisp's native number
representation will be parsed correctly."
   (cond ((char-equal (json-peek) ?-)
	 (json-advance)
	 (- (json-read-number)))
	((char-equal (json-peek) ?+)
	 (json-advance)
	 (json-read-number))
	((and (looking-at "\\([0-9]+\\)?\\(\\.[0-9]+\\)?\\([Ee][+-]?[0-9]+\\)?")
	      (or (match-beginning 1) (match-beginning 2) (match-beginning 3)))
	 (goto-char (match-end 0))
	 (string-to-number (match-string 0)))
	(t (signal 'json-number-format (list (point))))))

-- 
Kevin Rodgers
Denver, Colorado, USA