Speed of Elisp (was: JSON/YAML/TOML/etc. parsing performance)

[Stefan Monnier <monnier@iro.umontreal.ca>]

[ To clarify up front: I'm in favor of using those libraries.
  The questions below don't mean that I think it's better to speed up
  Elisp than to use a C implementation of those json primitives: In any
  case, it makes sense to use existing C libraries for that, both for
  speed reasons and for maintenance reasons; like we do for XML.
  The choice between C and Elisp would only make sense if we had to
  write&maintain the C code.  ]

> (license: Expat/X11/MIT), resulting in significant speedups using the test
> data from https://github.com/miloyip/nativejson-benchmark: a factor of 3.9
> to 6.4 for parsing,

Very interesting.  The way I read it, it means either that Elisp is not
nearly as slow as we tend to assume, or that the overhead introduced
when turning json-c's output into an Elisp-usable form dwarfs the json-c
parsing itself.

> and a factor of 27 to 67 for serializing.

I'm curious why there is such a wide discrepancy between the speedup for
parsing and that for serializing (sounds like a factor 10 difference).

Is it because parsing with json-c is slowed down by the conversion to
(especially allocation of) Elisp data structures, or is it because the
Elisp implementation of json serialization suffers more from poor
performance (in which case, maybe it could point to a performance issue
in Elisp which we could try to tackle)?


        Stefan