help with guile-json inconsistencies

help with guile-json inconsistencies

[Aleix Conchillo Flaqué]

Hi guilers,

a new guile-json issue came to me recently:

https://github.com/aconchillo/guile-json/issues/22

I don't really have a solution yet on how to solve this, so I would
love some advice. The issue is that one can do:

  (scm->json '(1 2 3)) which returns [1,2,3]

however, if you do:

  (scm->json '((1 2 3)) it returns {"1":[2,3]} but one would probably
expect [[1,2,3]]
  (scm->json '((1 2 3) (4 5 6)) it returns {"1":[2,3], "4":[5,6]} but
one would probably expect [[1,2,3], [4,5,6]]

I guess the first question is, does everyone else feel the same way
about the return value?

The problem is that there's a function that checks if something is an
alist by basically checking if it's a pair? recursively and if so, it
builds a json object.

In the past, guile-json had a macro (json) which allowed you to do
(json (array (array...))) but I just realized it has exactly the same
problem. But may be there's some way to fix it.

Any help would be appreciated!

Best,

Aleix

Re: help with guile-json inconsistencies

[Aleix Conchillo Flaqué]

On Sun, Dec 9, 2018 at 2:56 AM Aleix Conchillo Flaqué
<aconchillo@gmail.com> wrote:
>
> Hi guilers,
>
> a new guile-json issue came to me recently:
>
> https://github.com/aconchillo/guile-json/issues/22
>
> I don't really have a solution yet on how to solve this, so I would
> love some advice. The issue is that one can do:
>
>   (scm->json '(1 2 3)) which returns [1,2,3]
>
> however, if you do:
>
>   (scm->json '((1 2 3)) it returns {"1":[2,3]} but one would probably
> expect [[1,2,3]]
>   (scm->json '((1 2 3) (4 5 6)) it returns {"1":[2,3], "4":[5,6]} but
> one would probably expect [[1,2,3], [4,5,6]]
>
> I guess the first question is, does everyone else feel the same way
> about the return value?
>
> The problem is that there's a function that checks if something is an
> alist by basically checking if it's a pair? recursively and if so, it
> builds a json object.
>

Just to clarify, the problem is not when parsing JSON to scheme data
types, this is fine as internally guile-json uses hash tables and
lists properly.

The problem is with going from scheme to json. For convenience,
guile-json doesn't expect a hash table from the user, instead it tries
to guess what's a list and what's an alist however, that seems not
really distinguishable. That is, there's no way to tell if '((1 2)) is
[[1, 2]] or {"1" : [2]} or  {"1" : "2"}.

I guess going back to a macro approach for these type of cases would
help (even though I had the same problem with my initial macro).

Aleix

Re: help with guile-json inconsistencies

[Aleix Conchillo Flaqué]

On Sun, Dec 9, 2018 at 10:33 PM Aleix Conchillo Flaqué
<aconchillo@gmail.com> wrote:
>
> On Sun, Dec 9, 2018 at 2:56 AM Aleix Conchillo Flaqué
> <aconchillo@gmail.com> wrote:
> >
> > Hi guilers,
> >
> > a new guile-json issue came to me recently:
> >
> > https://github.com/aconchillo/guile-json/issues/22
> >
> > I don't really have a solution yet on how to solve this, so I would
> > love some advice. The issue is that one can do:
> >
> >   (scm->json '(1 2 3)) which returns [1,2,3]
> >
> > however, if you do:
> >
> >   (scm->json '((1 2 3)) it returns {"1":[2,3]} but one would probably
> > expect [[1,2,3]]
> >   (scm->json '((1 2 3) (4 5 6)) it returns {"1":[2,3], "4":[5,6]} but
> > one would probably expect [[1,2,3], [4,5,6]]
> >
> > I guess the first question is, does everyone else feel the same way
> > about the return value?
> >
> > The problem is that there's a function that checks if something is an
> > alist by basically checking if it's a pair? recursively and if so, it
> > builds a json object.
> >
>
> Just to clarify, the problem is not when parsing JSON to scheme data
> types, this is fine as internally guile-json uses hash tables and
> lists properly.
>
> The problem is with going from scheme to json. For convenience,
> guile-json doesn't expect a hash table from the user, instead it tries
> to guess what's a list and what's an alist however, that seems not
> really distinguishable. That is, there's no way to tell if '((1 2)) is
> [[1, 2]] or {"1" : [2]} or  {"1" : "2"}.
>
> I guess going back to a macro approach for these type of cases would
> help (even though I had the same problem with my initial macro).
>

Answering myself again. Re-adding the macro fixes the issue, but with
a breaking change.

(scm->json '(1 2 3))
      -> [1,2,3]

(scm->json '((1 2 3) (4 5 6)))
      -> [[1,2,3],[4,5,6]]

(scm->json (json (object ("key1" "value1") ("key2" (1 2 3)))))
      -> {"key2":[1,2,3],"key1":"value1"}

(scm->json `((1 2 3) ,(json (object (4 5)))))
      -> [[1,2,3],{"4":5}]

So, the (json) macro is only needed to specify JSON objects, otherwise
you can use scheme types directly (and lists are always arrays).

Aleix

Re: help with guile-json inconsistencies

[swedebugia]

On 2018-12-10 08:27, Aleix Conchillo Flaqué wrote:
> On Sun, Dec 9, 2018 at 10:33 PM Aleix Conchillo Flaqué
> <aconchillo@gmail.com> wrote:
>>
>> On Sun, Dec 9, 2018 at 2:56 AM Aleix Conchillo Flaqué
>> <aconchillo@gmail.com> wrote:
>>>
>>> Hi guilers,
>>>
>>> a new guile-json issue came to me recently:
>>>
>>> https://github.com/aconchillo/guile-json/issues/22
>>>
>>> I don't really have a solution yet on how to solve this, so I would
>>> love some advice. The issue is that one can do:
>>>
>>>    (scm->json '(1 2 3)) which returns [1,2,3]
>>>
>>> however, if you do:
>>>
>>>    (scm->json '((1 2 3)) it returns {"1":[2,3]} but one would probably
>>> expect [[1,2,3]]
>>>    (scm->json '((1 2 3) (4 5 6)) it returns {"1":[2,3], "4":[5,6]} but
>>> one would probably expect [[1,2,3], [4,5,6]]
>>>
>>> I guess the first question is, does everyone else feel the same way
>>> about the return value?
>>>
>>> The problem is that there's a function that checks if something is an
>>> alist by basically checking if it's a pair? recursively and if so, it
>>> builds a json object.
>>>
>>
>> Just to clarify, the problem is not when parsing JSON to scheme data
>> types, this is fine as internally guile-json uses hash tables and
>> lists properly.
>>
>> The problem is with going from scheme to json. For convenience,
>> guile-json doesn't expect a hash table from the user, instead it tries
>> to guess what's a list and what's an alist however, that seems not
>> really distinguishable. That is, there's no way to tell if '((1 2)) is
>> [[1, 2]] or {"1" : [2]} or  {"1" : "2"}.
>>
>> I guess going back to a macro approach for these type of cases would
>> help (even though I had the same problem with my initial macro).
>>
> 
> Answering myself again. Re-adding the macro fixes the issue, but with
> a breaking change.
> 
> (scm->json '(1 2 3))
>        -> [1,2,3]
> 
> (scm->json '((1 2 3) (4 5 6)))
>        -> [[1,2,3],[4,5,6]]
> 
> (scm->json (json (object ("key1" "value1") ("key2" (1 2 3)))))
>        -> {"key2":[1,2,3],"key1":"value1"}
> 
> (scm->json `((1 2 3) ,(json (object (4 5)))))
>        -> [[1,2,3],{"4":5}]
> 
> So, the (json) macro is only needed to specify JSON objects, otherwise
> you can use scheme types directly (and lists are always arrays).

Nice to hear you solved it :)

Both Guix and guile-wikidata use guile-json but only from json->scm and 
that works flawlessly. Thanks for creating and maintaining this lib.

-- 
Cheers
Swedebugia

Re: help with guile-json inconsistencies

[Alex Vong]

[-- Attachment #1: Type: text/plain, Size: 3383 bytes --]

Hello Aleix,

Aleix Conchillo Flaqué <aconchillo@gmail.com> writes:

> On Sun, Dec 9, 2018 at 10:33 PM Aleix Conchillo Flaqué
> <aconchillo@gmail.com> wrote:
>>
>> On Sun, Dec 9, 2018 at 2:56 AM Aleix Conchillo Flaqué
>> <aconchillo@gmail.com> wrote:
>> >
>> > Hi guilers,
>> >
>> > a new guile-json issue came to me recently:
>> >
>> > https://github.com/aconchillo/guile-json/issues/22
>> >
>> > I don't really have a solution yet on how to solve this, so I would
>> > love some advice. The issue is that one can do:
>> >
>> >   (scm->json '(1 2 3)) which returns [1,2,3]
>> >
>> > however, if you do:
>> >
>> >   (scm->json '((1 2 3)) it returns {"1":[2,3]} but one would probably
>> > expect [[1,2,3]]
>> >   (scm->json '((1 2 3) (4 5 6)) it returns {"1":[2,3], "4":[5,6]} but
>> > one would probably expect [[1,2,3], [4,5,6]]
>> >
>> > I guess the first question is, does everyone else feel the same way
>> > about the return value?
>> >
>> > The problem is that there's a function that checks if something is an
>> > alist by basically checking if it's a pair? recursively and if so, it
>> > builds a json object.
>> >
>>
>> Just to clarify, the problem is not when parsing JSON to scheme data
>> types, this is fine as internally guile-json uses hash tables and
>> lists properly.
>>
>> The problem is with going from scheme to json. For convenience,
>> guile-json doesn't expect a hash table from the user, instead it tries
>> to guess what's a list and what's an alist however, that seems not
>> really distinguishable. That is, there's no way to tell if '((1 2)) is
>> [[1, 2]] or {"1" : [2]} or  {"1" : "2"}.
>>
>> I guess going back to a macro approach for these type of cases would
>> help (even though I had the same problem with my initial macro).
>>
>
> Answering myself again. Re-adding the macro fixes the issue, but with
> a breaking change.
>
> (scm->json '(1 2 3))
>       -> [1,2,3]
>
> (scm->json '((1 2 3) (4 5 6)))
>       -> [[1,2,3],[4,5,6]]
>
> (scm->json (json (object ("key1" "value1") ("key2" (1 2 3)))))
>       -> {"key2":[1,2,3],"key1":"value1"}
>
> (scm->json `((1 2 3) ,(json (object (4 5)))))
>       -> [[1,2,3],{"4":5}]
>
> So, the (json) macro is only needed to specify JSON objects, otherwise
> you can use scheme types directly (and lists are always arrays).
>
Another ambuiguity would be that both [] and {} corresponds to ().

It seems to me that the core of this problem is that the Pair, List and
Alist (which is a List of Pairs) types in scheme are not orthogonal,
i.e. () are both List and Alist, and (1 2 3) are both List and Pair.

Therefore, one fix would be to make Vector instead of List to
corresponds to Json Array, but maybe this change will cause too much
breakage, so I am not sure it is practical.

Another way would be to make some other type other than Alist to
corresponds to Json Object. If we were in Clojure, it would be Hash
Map. I think VHash is a good candidate, but I am not happy with it not
having a read/write syntax (Hash Table has the same problem too) which
makes it not as flexible as Alist.

(Actually, I had been trying to write a reader macro for VHash (w/ or
w/o the Guile Reader library) but I didn't succeed in doing so because
of my limited knowledge in how lexing works in Guile.)

> Aleix

Cheers,
Alex

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: help with guile-json inconsistencies

[John Cowan]

On Mon, Dec 10, 2018 at 3:56 PM Alex Vong <alexvong1995@gmail.com> wrote:

Therefore, one fix would be to make Vector instead of List to
> corresponds to Json Array, but maybe this change will cause too much
> breakage, so I am not sure it is practical.
>

That is the approach used by the Chicken json egg
<http://wiki.call-cc.org/eggref/5/json>.  Note that when writing
you can use either a vector of pairs or a hash table to represent
a JSON object, but when reading it is always re-created as
a vector of pairs.  This is portable and eliminates any problems
with readability from within Scheme.

The json egg depends on the packrat egg at
< http://wiki.call-cc.org/eggref/5/packrat> which has no further
dependencies, but it would be easy to replace the packrat
parser with a hand-rolled parser.

-- 
John Cowan          http://vrici.lojban.org/~cowan        cowan@ccil.org
If I have seen farther than others, it is because I was looking through a
spyglass with my one good eye, with a parrot standing on my shoulder. --"Y"


> Another way would be to make some other type other than Alist to
> corresponds to Json Object. If we were in Clojure, it would be Hash
> Map. I think VHash is a good candidate, but I am not happy with it not
> having a read/write syntax (Hash Table has the same problem too) which
> makes it not as flexible as Alist.
>
> (Actually, I had been trying to write a reader macro for VHash (w/ or
> w/o the Guile Reader library) but I didn't succeed in doing so because
> of my limited knowledge in how lexing works in Guile.)
>
> > Aleix
>
> Cheers,
> Alex
>

Re: help with guile-json inconsistencies

[Zelphir Kaltstahl]

On 10.12.18 23:16, guile-user-request@gnu.org wrote:
> Therefore, one fix would be to make Vector instead of List to
> corresponds to Json Array, but maybe this change will cause too much
> breakage, so I am not sure it is practical.
>
> Another way would be to make some other type other than Alist to
> corresponds to Json Object. If we were in Clojure, it would be Hash
> Map. I think VHash is a good candidate, but I am not happy with it not
> having a read/write syntax (Hash Table has the same problem too) which
> makes it not as flexible as Alist.

I think that is how it is done in Racket, if I am not mistaken.

Re: help with guile-json inconsistencies

[Aleix Conchillo Flaqué]

On Mon, Dec 10, 2018 at 4:38 PM Zelphir Kaltstahl
<zelphirkaltstahl@gmail.com> wrote:
>
> On 10.12.18 23:16, guile-user-request@gnu.org wrote:
> > Therefore, one fix would be to make Vector instead of List to
> > corresponds to Json Array, but maybe this change will cause too much
> > breakage, so I am not sure it is practical.
> >
> > Another way would be to make some other type other than Alist to
> > corresponds to Json Object. If we were in Clojure, it would be Hash
> > Map. I think VHash is a good candidate, but I am not happy with it not
> > having a read/write syntax (Hash Table has the same problem too) which
> > makes it not as flexible as Alist.
>
> I think that is how it is done in Racket, if I am not mistaken.
>
>

This is how it's done in guile-json as well. A json object corresponds
to a hash table in guile-json, otherwise it would not be possible to
work properly internally.

The question was not how to parse json and represent it in guile (that
works well), but how to solve an inconsistency when building a json
object from scheme data types. This was done by converting an alist to
a hash table, but unfortunately it doesn't work well in all cases. So,
the solution is that I will remove the conversion from an alist to an
object, so a user will need to explicitly provide a hash table (e.g.
using alist->hash-table). With this change, the bidirectional mapping
mentioned here will be consistent:

https://github.com/aconchillo/guile-json#usage (without the alist note)

Thanks everyone for all the feedback!

Aleix

Re: help with guile-json inconsistencies

[Zelphir Kaltstahl]

On 11.12.18 06:18, Aleix Conchillo Flaqué wrote:
> On Mon, Dec 10, 2018 at 4:38 PM Zelphir Kaltstahl
> <zelphirkaltstahl@gmail.com> wrote:
>> On 10.12.18 23:16, guile-user-request@gnu.org wrote:
>>> Therefore, one fix would be to make Vector instead of List to
>>> corresponds to Json Array, but maybe this change will cause too much
>>> breakage, so I am not sure it is practical.
>>>
>>> Another way would be to make some other type other than Alist to
>>> corresponds to Json Object. If we were in Clojure, it would be Hash
>>> Map. I think VHash is a good candidate, but I am not happy with it not
>>> having a read/write syntax (Hash Table has the same problem too) which
>>> makes it not as flexible as Alist.
>> I think that is how it is done in Racket, if I am not mistaken.
>>
>>
> This is how it's done in guile-json as well. A json object corresponds
> to a hash table in guile-json, otherwise it would not be possible to
> work properly internally.
>
> The question was not how to parse json and represent it in guile (that
> works well), but how to solve an inconsistency when building a json
> object from scheme data types. This was done by converting an alist to
> a hash table, but unfortunately it doesn't work well in all cases. So,
> the solution is that I will remove the conversion from an alist to an
> object, so a user will need to explicitly provide a hash table (e.g.
> using alist->hash-table). 

I think that is a good idea and don't see a problem with having to do
that, as it makes things explicit and I would have expected, that I have
to provide a hash table to get a JSON object out.

> With this change, the bidirectional mapping
> mentioned here will be consistent:
>
> https://github.com/aconchillo/guile-json#usage (without the alist note)
As long as translating back and forth from JSON and guile still results
in the same structure, this seems fine. Translating JSON to Guile and
then writing it back as JSON again should not result in JSON changes.
> Thanks everyone for all the feedback!
>
> Aleix
Thanks for guile-json!

Re: help with guile-json inconsistencies

[Aleix Conchillo Flaqué]

On Tue, Dec 11, 2018 at 12:31 AM Zelphir Kaltstahl
<zelphirkaltstahl@gmail.com> wrote:
> As long as translating back and forth from JSON and guile still results
> in the same structure, this seems fine. Translating JSON to Guile and
> then writing it back as JSON again should not result in JSON changes.

Yes, that's guaranteed. The only thing that changes is the order of
json objects but, by the spec, objects are unordered so we are good.

> Thanks for guile-json!

My pleasure! Thank you!

Best,

Aleix