[RFC] Use JSON in emacs interface

[RFC] Use JSON in emacs interface

[Chris Gray]

Hi,

My thinking about this arises from the fact that there is a person on
one of the lists that I read who puts a semicolon after his name.  Of
course, this confuses the regex in notmuch-search-process-filter, which
expects that the first semicolon in the string representing a thread is
after all the authors.

I first thought of changing the regex so that it looked for the last
semicolon in the string or something like that, but that would just move
the problem.  (Semicolons are probably more frequent in subject lines
than in author names.)  So it seems to me that what is needed is for
notmuch and emacs to talk with each other in a format that is
unambiguously parseable.  Since notmuch search already has the option of
outputting to JSON, that seems like a natural fit.

Emacs has an existing JSON parser,
<http://cvs.savannah.gnu.org/viewvc/*checkout*/emacs/lisp/json.el?root=emacs>,
but it doesn't appear that it is able to parse progressively, meaning
that it wouldn't be able to display results as they come in from notmuch
search if used as-is.  My guess is that its parts could be hacked
together to overcome this limitation though.

Anyway, if others think this is a good idea, I'm willing to do the
coding.

Cheers,
Chris

Re: [RFC] Use JSON in emacs interface

[Adam Wolfe Gordon]

Hi Chris,

On Sun, Apr 29, 2012 at 10:22, Chris Gray <chrismgray@gmail.com> wrote:
> I first thought of changing the regex so that it looked for the last
> semicolon in the string or something like that, but that would just move
> the problem.  (Semicolons are probably more frequent in subject lines
> than in author names.)  So it seems to me that what is needed is for
> notmuch and emacs to talk with each other in a format that is
> unambiguously parseable.  Since notmuch search already has the option of
> outputting to JSON, that seems like a natural fit.
>
> Emacs has an existing JSON parser,
> <http://cvs.savannah.gnu.org/viewvc/*checkout*/emacs/lisp/json.el?root=emacs>,
> but it doesn't appear that it is able to parse progressively, meaning
> that it wouldn't be able to display results as they come in from notmuch
> search if used as-is.  My guess is that its parts could be hacked
> together to overcome this limitation though.
>
> Anyway, if others think this is a good idea, I'm willing to do the
> coding.

I think this is a great idea. If you look at notmuch-mua.el and
notmuch-query.el, you'll see that we already use json.el for reply and
parts of show.

As for parsing progressively to show search results as they arrive,
I'd be inclined to instead implement paging, so emacs could ask for
just the first n results, display them, then ask for the next n
results, etc. Or maybe ask for the results grouped, so that there
would be a valid JSON object for the first n results, then another for
the next n, etc. This might be a tricky thing to design and implement,
but I think it's been discussed before as something that would be nice
to have.

Personally, I think it would be fine to implement the JSON search
first, then deal with progressive parsing and/or grouping, but others
may differ here. Either way, I'd be happy to review patches for this
and offer any suggestions.

-- Adam

Re: [RFC] Use JSON in emacs interface

[Austin Clements]

Quoth Chris Gray on Apr 29 at 10:22 am:
> Hi,
> 
> My thinking about this arises from the fact that there is a person on
> one of the lists that I read who puts a semicolon after his name.  Of
> course, this confuses the regex in notmuch-search-process-filter, which
> expects that the first semicolon in the string representing a thread is
> after all the authors.
> 
> I first thought of changing the regex so that it looked for the last
> semicolon in the string or something like that, but that would just move
> the problem.  (Semicolons are probably more frequent in subject lines
> than in author names.)  So it seems to me that what is needed is for
> notmuch and emacs to talk with each other in a format that is
> unambiguously parseable.  Since notmuch search already has the option of
> outputting to JSON, that seems like a natural fit.
> 
> Emacs has an existing JSON parser,
> <http://cvs.savannah.gnu.org/viewvc/*checkout*/emacs/lisp/json.el?root=emacs>,
> but it doesn't appear that it is able to parse progressively, meaning
> that it wouldn't be able to display results as they come in from notmuch
> search if used as-is.  My guess is that its parts could be hacked
> together to overcome this limitation though.
> 
> Anyway, if others think this is a good idea, I'm willing to do the
> coding.

This is definitely a good idea.  In fact, there's been quite a bit of
discussion about it in the past, and even some (very old and
outdated) implementation work:

  id:"878vs28dvo.fsf@praet.org"
    A rather lengthy discussion of another motivation for using JSON
    search.  Includes performance results for using JSON in search.

  id:"20120214152114.GQ27039@mit.edu"
    Discussion of "framed" JSON that would be a valid JSON object, but
    could easily be consumed in a streaming fashion without hacking
    Emacs' JSON parser or resorting to paging.

  id:"1290777202-14040-1-git-send-email-dme@dme.org"
    The implementation from long ago.  Definitely outdated, but could
    be a good starting point.

It looks like most of the discussion about streaming JSON parsing has
been on the IRC channel, rather than the mailing list, but I'd be
happy to summarize it or dig out the IRC logs.  

I agree with Adam that it probably makes the most sense to get an
implementation working without streaming first and then add streaming
support.