Re: User-defined record types

[Ted Zlatanov <tzz@lifelogs.com>]

On Fri, 18 Oct 2013 22:09:39 -0400 Stefan Monnier <monnier@iro.umontreal.ca> wrote: 

>>>> Something that represents JSON and YAML well would be nice.  Currently
>>>> we don't have an ELisp data structure that can preserve all JSON nuances
>>>> without acrobatics (e.g. preserving the difference between "null" and
>>>> "empty list" or the native JSON data types).
SM> I don't understand what you mean here.  It's very easy to have special
SM> values, e.g. (defconst json-null (make-symbol "json-null")).
>> Yes, but it's not something you can communicate externally.  Compare
>> with pure JSON or BSON, which are intended for communication across
>> programs.

SM> You lost me: I though "null" and "empty list" were JSON thingies, so
SM> I just offered you ways to represent them on the Elisp side.  Of course,
SM> when turning those elements into JSON, you'd be careful to map them to
SM> the appropriate JSON elements.

Right.  So instead of a special mapping operation, the underlying C
storage (record) should hold the JSON/YAML/XML types and values
directly.  The ELisp API to that C storage would map things to
ELisp-land, but the original data would be preserved.

>> Sure.  I'm saying a custom data structure would help here, not that it's
>> the only way to accomplish it, and trying to answer your earlier
>> question about custom record types.

SM> I still don't understand in what way a custom data structure would help.

By holding the original data in a way that today's Emacs internals can't.

>>>> a native XML data structure would also be nice.  We have what libxml
>>>> produces, dumped in an awkward tree, but nothing native.
SM> Not sure what "native" can mean in this context: we were talking about
SM> new Lisp-defined types.
>> Native to ELisp, but in a way that represents the original data
>> structure cleanly and transparently.

SM> I still don't see what that means.  In which way would it be cleaner or
SM> more transparent?

>> I'm talking about custom data types that can be efficiently and
>> transparently converted to what the external libraries and protocols
>> expect, and provide a good ELisp interface to their contents.  I think
>> the currently available XML and JSON representation in ELisp don't do
>> both.  Am I misunderstanding the question?

SM> What alternative do you have in mind that would be more efficient and/or
SM> more transparent?

Well, let's look at your original question:

On Sat, 12 Oct 2013 14:58:15 -0400 Stefan Monnier <monnier@IRO.UMontreal.CA> wrote: 

>>> I'm not necessarily opposed to user-defined record types, but if
>>> cl-defstruct is not sufficient and C-level changes are required, then
>>> I'd hope for the new code to provide more exciting features than just
>>> "vectorp returns nil".

Look.  This is JSON (YAML is similar; XML is much more complex but has
similar conversion and implementation issues):

{ "x": 1, "y": [ null, [], true, false ] }

A C structure would represent that as follows:

- the number 1 would be stored as a JSON-spec integer, which is
  essentially a tagged string.  These are not ELisp ints or bignums
  although the implementation can choose to limit them to Javascript
  ints.  See http://www.json.org/ for the details on number formatting.

- the value under the "y" key has four distinct values that should be
  represented and serialized differently, yet provide the expected
  semantics for ELisp access ("null", "[]", and "false" would be exposed
  as `nil').

- stored efficiently

All of this can be done in ELisp with custom symbols and other tricks,
but I think a C implementation would be cleaner by being closer to the
original input data and removing ELisp baggage from a simple data
format.  The C implementation would also be more efficient than an ELisp
implementation if it serializes to and from JSON in C, but that's a
minor concern.  Finally, a C implementation would be able to make null,
false, and [] look like `nil' in boolean or iteration contexts, yet
preserve the original data.  I don't think an ELisp implementation can
do that unless all access goes through a forced API.

So the exciting feature is that you don't have to work around the
absence of these record types in ELisp.  You may consider that an
advantage (less C code = less maintenance) but the original question was
about "exciting features" not maintainability.

Ted