Fast JSON codec based on YAJL

Fast JSON codec based on YAJL

[Julian Scheid]

Hi,

I'm getting close to finishing a first version of an interface to YAJL,
"a small event-driven (SAX-style) JSON parser written in ANSI C, and a
small validating JSON generator."  See http://lloyd.github.com/yajl/

I've started working on this interface because json.el proves to be a
bottleneck in another project of mine.

A few unscientific benchmarks suggest that YAJL (via my interface) is
roughly 5 to 25 times as fast as byte-compiled json.el, on average about
an order of magnitude faster.  Memory usage is about on par.

With a few minor exceptions it's fully compatible to json.el and can be
used as a drop-in replacement in most situations. [*]

The interface also exposes a number of useful YAJL features not found in
json.el, such as optional pretty-printing.

It comes with a comprehensive ERT-based unit test suite, a small
stress-test and benchmarking suite, full TexInfo documentation, and the
usual detection code and switches for the configure script.

I've looked at a number of JSON libraries and chose YAJL mainly because
it's mature, clean and fast, and its event-driven architecture means a
redundant document tree won't have to be kept in memory.

YAJL is available in most major package repositories such as Debian, Red
Hat, Arch Linux, and MacPorts.

Is there interest in adding this interface to Emacs mainline?  If so,
I'd polish it a bit more and send over my code for review soon.

As for the legal side, YAJL (the underlying library, not my interface)
uses the ISC license but is only linked against.  The interface itself
doesn't include any ISC-licensed code, or any third-party code for that
matter.  I'd be happy to sign over copyright.

Julian


[*] The minor incompatibilities boil down to the fact that json.el can
emit invalid JSON in some cases when YAJL refuses to do so.  Example:
(json-encode '((1 . 1))) will yield invalid JSON output.  json.el also
has issues with characters on the astral plane and some to do with NaN
and Inf, which YAJL handles slightly different (and closer to the JSON
specification.)

Re: Fast JSON codec based on YAJL

[Stefan Monnier]

> A few unscientific benchmarks suggest that YAJL (via my interface) is
> roughly 5 to 25 times as fast as byte-compiled json.el, on average about
> an order of magnitude faster.  Memory usage is about on par.

You mean that json.elc is only 5 to 25 times slower than
a C implementation?  I actually expected significantly worse.


        Stefan

Re: Fast JSON codec based on YAJL

[Julian Scheid]

[-- Attachment #1: Type: text/plain, Size: 1395 bytes --]

On Tue, Jan 15, 2013 at 3:26 AM, Stefan Monnier <monnier@iro.umontreal.ca>wrote:

> You mean that json.elc is only 5 to 25 times slower than
> a C implementation?  I actually expected significantly worse.
>

Both share quite a few code paths actually, for example number-to-string
conversion (and vice versa) and multibyte handling. I wrote the code a
little
while ago but IIRC data conversion between Emacs-land and native land
accounts
for a non-negligible share of the runtime, and of course the same goes for
building Emacs data structures from parsed data. (It doesn't help that with
JSON, you don't know how big an array or map will be until you're finished
parsing it.)

Also, although performance is my primary objective, I didn't pick the C
library
on that aspect alone. For example, libjson bills itself as a "JSON reader
and
writer which is super-effiecient [sic] and usually runs circles around other
JSON libraries." But it doesn't support streaming and thus would add a
significant memory overhead for large documents, and also it's not nearly as
portable.

I did spend some time making sure that the interface performs well and
there are
no obvious low-hanging-fruit-type optimizations left, but any help with
making
it even faster is welcome! I think that an order of magnitude is a good
start
though, it certainly takes my other app from "painfully slow" to "perfectly
usable".

[-- Attachment #2: Type: text/html, Size: 2638 bytes --]