bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Henry Minsky]

[-- Attachment #1: Type: text/plain, Size: 3850 bytes --]

If you run M-x json-pretty-print on this buffer contents:

{"t": 259}

You get an error
Bad JSON object key: t

It seems like the json parser is converting the string "t" into the
symbol t which is being treated specially?





In GNU Emacs 26.3 (build 1, x86_64-apple-darwin18.2.0, NS appkit-1671.20
Version 10.14.3 (Build 18D109))
 of 2019-09-02 built on builder10-14.porkrind.org
Windowing system distributor 'Apple', version 10.3.2022
Recent messages:
Loading vc-svn...done
Loading autorevert...done
For information about GNU Emacs and the GNU system, type C-h C-a.
Quit
ls does not support --dired; see ‘dired-use-ls-dired’ for more details.
Quit
command-execute: The mark is not set now, so there is no region
Mark set
json-encode-key: Bad JSON object key: t

Configured using:
 'configure --with-ns '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp' --with-modules'

Configured features:
NOTIFY ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES THREADS

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: JavaScript

Minor modes in effect:
  override-global-mode: t
  global-auto-revert-mode: t
  display-time-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr warnings emacsbug message rmc puny rfc822 mml
mml-sec epa derived epg gnus-util rmail rmail-loaddefs mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils js
advice sgml-mode dom json map imenu thingatpt dired dired-loaddefs
elec-pair cl-extra help-mode use-package use-package-ensure
use-package-delight use-package-diminish use-package-bind-key bind-key
easy-mmode use-package-core finder-inf info package epg-config
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp
byte-compile cconv edmacro kmacro format-spec server autorevert
filenotify cus-start cus-load vc-svn bdc-indent cc-mode cc-fonts
easymenu cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars
cc-defs cl-loaddefs cl-lib time time-date tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/ns-win ns-win
ucs-normalize mule-util term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads kqueue cocoa ns multi-tty make-network-process emacs)

Memory information:
((conses 16 459220 15608)
 (symbols 48 31276 1)
 (miscs 40 62 287)
 (strings 32 108877 1948)
 (string-bytes 1 2792952)
 (vectors 16 49371)
 (vector-slots 8 918195 17836)
 (floats 8 64 200)
 (intervals 56 375 1)
 (buffers 992 14))

[-- Attachment #2: Type: text/html, Size: 4241 bytes --]

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Lars Ingebrigtsen]

Henry Minsky <henry.minsky@gmail.com> writes:

> If you run M-x json-pretty-print on this buffer contents:
>
> {"t": 259}
>
> You get an error
> Bad JSON object key: t
>
> It seems like the json parser is converting the string "t" into the
> symbol t which is being treated specially?

It's unfortunate that `json-read' is being ambiguous here.

true
=> t

{"t": 259}
=> ((t . 259))

So I don't know how to fix this.  We could add another kludge -- saying
that a boolean used as an object key "obviously" should be a string
instead (when converting back to JSON)?  Opinions?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Basil L. Contovounesios]

forcemerge 42545 45032
quit

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So I don't know how to fix this.  We could add another kludge -- saying
> that a boolean used as an object key "obviously" should be a string
> instead (when converting back to JSON)?  Opinions?

This is a duplicate of https://debbugs.gnu.org/42545.

I think it's important that native and Elisp JSON serialisation are as
consistent as possible here; please comment on my thoughts at:
https://lists.gnu.org/r/emacs-devel/2020-07/msg00708.html

I might actually have some time to look at this soon.

-- 
Basil

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Philipp Stephani]

Am Sa., 5. Dez. 2020 um 17:26 Uhr schrieb Basil L. Contovounesios
<contovob@tcd.ie>:
>
> forcemerge 42545 45032
> quit
>
> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
> > So I don't know how to fix this.  We could add another kludge -- saying
> > that a boolean used as an object key "obviously" should be a string
> > instead (when converting back to JSON)?  Opinions?
>
> This is a duplicate of https://debbugs.gnu.org/42545.
>
> I think it's important that native and Elisp JSON serialisation are as
> consistent as possible here;

I don't think that's realistic: any change in behavior to either of
these functions would be a breaking change.

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Basil L. Contovounesios]

Philipp Stephani <p.stephani2@gmail.com> writes:

> Am Sa., 5. Dez. 2020 um 17:26 Uhr schrieb Basil L. Contovounesios
> <contovob@tcd.ie>:
>>
>> I think it's important that native and Elisp JSON serialisation are as
>> consistent as possible here;
>
> I don't think that's realistic: any change in behavior to either of
> these functions would be a breaking change.

As possible/reasonable then.

-- 
Basil

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Lars Ingebrigtsen]

Philipp Stephani <p.stephani2@gmail.com> writes:

>> I think it's important that native and Elisp JSON serialisation are as
>> consistent as possible here;
>
> I don't think that's realistic: any change in behavior to either of
> these functions would be a breaking change.

I think we should have JSON/Elisp round trips that are 100%
reproducible.  The current functions certainly aren't.

But we could just leave them as deprecated legacy functions and
introduce new ones that are consistent.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Philipp Stephani]

Am So., 6. Dez. 2020 um 14:37 Uhr schrieb Lars Ingebrigtsen <larsi@gnus.org>:
>
> Philipp Stephani <p.stephani2@gmail.com> writes:
>
> >> I think it's important that native and Elisp JSON serialisation are as
> >> consistent as possible here;
> >
> > I don't think that's realistic: any change in behavior to either of
> > these functions would be a breaking change.
>
> I think we should have JSON/Elisp round trips that are 100%
> reproducible.  The current functions certainly aren't.

I don't understand why that is so important. I designed the C JSON
functions partially because I disagree with some aspects of API design
and behavior of the Elisp functions, so they are pretty much
incompatible on purpose. Trying to make them compatible would make the
C functions worse.
The one thing that I could imagine would be feasible would be:
(1) Document the precise behavior of the C JSON functions, including
all edge cases.
(2) Provide polyfills in Elisp that replicate that exact behavior.
(3) Deprecate the other Elisp functions.

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Lars Ingebrigtsen]

Philipp Stephani <p.stephani2@gmail.com> writes:

>> >> I think it's important that native and Elisp JSON serialisation are as
>> >> consistent as possible here;
>> >
>> > I don't think that's realistic: any change in behavior to either of
>> > these functions would be a breaking change.
>>
>> I think we should have JSON/Elisp round trips that are 100%
>> reproducible.  The current functions certainly aren't.
>
> I don't understand why that is so important. I designed the C JSON
> functions partially because I disagree with some aspects of API design
> and behavior of the Elisp functions, so they are pretty much
> incompatible on purpose. Trying to make them compatible would make the
> C functions worse.

Sorry, I was unclear -- I'm not saying the old and the new functions
should be compatible, only that there should be functions that can round
trip via JSON->Elisp->JSON and get identical results back.

Is that the case today?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Philipp Stephani]

Am So., 6. Dez. 2020 um 18:06 Uhr schrieb Lars Ingebrigtsen <larsi@gnus.org>:
>
> Philipp Stephani <p.stephani2@gmail.com> writes:
>
> >> >> I think it's important that native and Elisp JSON serialisation are as
> >> >> consistent as possible here;
> >> >
> >> > I don't think that's realistic: any change in behavior to either of
> >> > these functions would be a breaking change.
> >>
> >> I think we should have JSON/Elisp round trips that are 100%
> >> reproducible.  The current functions certainly aren't.
> >
> > I don't understand why that is so important. I designed the C JSON
> > functions partially because I disagree with some aspects of API design
> > and behavior of the Elisp functions, so they are pretty much
> > incompatible on purpose. Trying to make them compatible would make the
> > C functions worse.
>
> Sorry, I was unclear -- I'm not saying the old and the new functions
> should be compatible, only that there should be functions that can round
> trip via JSON->Elisp->JSON and get identical results back.
>
> Is that the case today?

You mean something like (json-serialize (json-parse-string ...))? I'd
hope that's indeed the case to the furthest extent possible. There are
cases where roundtripping is impossible (parsing ignores whitespace,
field order, and duplicate keys), but otherwise I'd hope these
functions are inverses of each other. Or is there a case where they
aren't?

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Dmitry Gutov]

On 06.12.2020 19:02, Philipp Stephani wrote:
> I designed the C JSON
> functions partially because I disagree with some aspects of API design
> and behavior of the Elisp functions, so they are pretty much
> incompatible on purpose

So this difference in keywords serialization was by design?

ELISP> (json-serialize '(:a 1 :b 2))
"{\"a\":1,\"b\":2}"
ELISP> (json-serialize '((:a . 1) (:b . 2)))
"{\":a\":1,\":b\":2}"

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Lars Ingebrigtsen]

Philipp Stephani <p.stephani2@gmail.com> writes:

>> Sorry, I was unclear -- I'm not saying the old and the new functions
>> should be compatible, only that there should be functions that can round
>> trip via JSON->Elisp->JSON and get identical results back.
>>
>> Is that the case today?
>
> You mean something like (json-serialize (json-parse-string ...))? I'd
> hope that's indeed the case to the furthest extent possible. There are
> cases where roundtripping is impossible (parsing ignores whitespace,
> field order, and duplicate keys), but otherwise I'd hope these
> functions are inverses of each other. Or is there a case where they
> aren't?

No, the confusion is on my part -- I somehow assumed that
json-pretty-print had been converted to use the new C functions, but
it's using the old, non-consistent json.el functions instead.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Lars Ingebrigtsen]

(Please keep the debbugs address in the CC's.)

Henry Minsky <henry.minsky@gmail.com> writes:

> The use case I had was in using M-x json-pretty-print in a buffer, so
> if that got fixed, it would solve my particular issue.

I guess the main obstacle for doing that is that json.c and json.el have
different interfaces, and we'd want the function to work with and
without built-in json.c support.

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Philipp Stephani]

Am So., 6. Dez. 2020 um 20:26 Uhr schrieb Dmitry Gutov <dgutov@yandex.ru>:
>
> On 06.12.2020 19:02, Philipp Stephani wrote:
> > I designed the C JSON
> > functions partially because I disagree with some aspects of API design
> > and behavior of the Elisp functions, so they are pretty much
> > incompatible on purpose
>
> So this difference in keywords serialization was by design?
>
> ELISP> (json-serialize '(:a 1 :b 2))
> "{\"a\":1,\"b\":2}"
> ELISP> (json-serialize '((:a . 1) (:b . 2)))
> "{\":a\":1,\":b\":2}"

I can't answer that. When I wrote json.c, I had it only support
hashtables, and list support is a later addition.

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Dmitry Gutov]

On 12.12.2020 16:28, Philipp Stephani wrote:
>> So this difference in keywords serialization was by design?
>>
>> ELISP> (json-serialize '(:a 1 :b 2))
>> "{\"a\":1,\"b\":2}"
>> ELISP> (json-serialize '((:a . 1) (:b . 2)))
>> "{\":a\":1,\":b\":2}"
> I can't answer that. When I wrote json.c, I had it only support
> hashtables, and list support is a later addition.

Perhaps we could agree that there _are_ some things that can be fixed in 
json.c's behavior, then?

And use json.el's behavior in those situations as example.

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Philipp Stephani]

Am Sa., 12. Dez. 2020 um 22:25 Uhr schrieb Dmitry Gutov <dgutov@yandex.ru>:
>
> On 12.12.2020 16:28, Philipp Stephani wrote:
> >> So this difference in keywords serialization was by design?
> >>
> >> ELISP> (json-serialize '(:a 1 :b 2))
> >> "{\"a\":1,\"b\":2}"
> >> ELISP> (json-serialize '((:a . 1) (:b . 2)))
> >> "{\":a\":1,\":b\":2}"
> > I can't answer that. When I wrote json.c, I had it only support
> > hashtables, and list support is a later addition.
>
> Perhaps we could agree that there _are_ some things that can be fixed in
> json.c's behavior, then?

It doesn't matter what we can agree on, now that json.c is released
into a stable release, we can't change its behavior any more (unless
you're proposing to add new keyword arguments to json-serialize, which
would still be possible).

bug#45032: 26.3; json-pretty-print of JSON with dict containing 't' as a key causes error

[Dmitry Gutov]

On 13.12.2020 15:19, Philipp Stephani wrote:
> It doesn't matter what we can agree on, now that json.c is released
> into a stable release, we can't change its behavior any more (unless
> you're proposing to add new keyword arguments to json-serialize, which
> would still be possible).

We do change behavior from time to time, especially when we consider it 
a bug.

The inconsistency looks like a bug to me.