unofficial mirror of notmuch@notmuchmail.org
 help / color / mirror / code / Atom feed
From: Austin Clements <amdragon@MIT.EDU>
To: Pieter Praet <pieter@praet.org>
Cc: Notmuch Mail <notmuch@notmuchmail.org>, David Edmondson <dme@dme.org>
Subject: Re: [PATCH v2] emacs: bad regexp @ `notmuch-search-process-filter'
Date: Wed, 13 Jul 2011 14:57:21 -0400	[thread overview]
Message-ID: <20110713185721.GI25558@mit.edu> (raw)
In-Reply-To: <878vs28dvo.fsf@praet.org>

Quoth Pieter Praet on Jul 13 at  4:16 pm:
> On Mon, 11 Jul 2011 17:05:32 -0400, Austin Clements <amdragon@MIT.EDU> wrote:
> > Quoth Pieter Praet on Jul 11 at 10:43 pm:
> > > TL;DR: I can haz regex pl0x?
> > 
> > Oof, what a pain.  I'm happy to change the output format of search; I
> > hadn't realized how difficult it would be to parse.  In fact, I'm not
> > sure it's even parsable by regexp, because the message ID's themselves
> > could contain parens.
> > 
> > So what would be a good format?  One possibility would be to
> > NULL-delimit the query part; as distasteful as I find that, this part
> > of the search output isn't meant for user consumption.  Though I fear
> > this is endemic to the dual role the search output currently plays as
> > both user and computer readable.
> > 
> > I've also got the code to do everything using document ID's instead of
> > message ID's.  As a side-effect, it makes the search output clean and
> > readily parsable since document ID's are just numbers.  Hence, there
> > are no quoting or escaping issues (plus the output is much more
> > compact).  I haven't sent this to the list yet because I haven't had a
> > chance to benchmark it and determine if the performance benefits make
> > exposing document ID's worthwhile.
> 
> Jamie Zawinski once said/wrote [1]:
>   'Some people, when confronted with a problem, think "I know,
>   I'll use regular expressions." Now they have two problems.'
> 
> With this in mind, I set out to get rid of this whole regex mess altogether,
> by populating the search buffer using Notmuch's JSON output instead of doing
> brittle text matching tricks.
> 
> Looking for some documentation, I stumbled upon a long-forgotten gem [2].
> 
> David's already done pretty much all of the work for us!

Yes, similar thoughts were running through my head as I futzed with
the formatting for this.  My concern with moving to JSON for search
buffers is that parsing it is about *30 times slower* than the current
regexp-based approach (0.6 seconds versus 0.02 seconds for a mere 1413
result search buffer).  I think JSON makes a lot of sense for show
buffers because there's generally less data and it has a lot of
complicated structure.  Search results, on the other hand, have a very
simple, regular, and constrained structure, so JSON doesn't buy us
nearly as much.

JSON is hard to parse because, like the text search output, it's
designed for human consumption (of course, unlike the text search
output, it's also designed for computer consumption).  There's
something to be said for the debuggability and generality of this and
JSON is very good for exchanging small objects, but it's a remarkably
inefficient way to exchange large amounts of data between two
programs.

I guess what I'm getting at, though it pains me to say it, is perhaps
search needs a fast, computer-readable interchange format.  The
structure of the data is so simple and constrained that this could be
altogether trivial.

Or maybe I need a faster computer.


If anyone is curious, here's how I timed the parsing.

(defmacro time-it (code)
  `(let ((start-time (get-internal-run-time)))
     ,code
     (float-time (time-subtract (get-internal-run-time) start-time))))

(with-current-buffer "json"
  (goto-char (point-min))
  (time-it (json-read)))

(with-current-buffer "text"
  (goto-char (point-min))
  (time-it
   (while (re-search-forward "^\\(thread:[0-9A-Fa-f]*\\) \\([^][]*\\) \\(\\[[0-9/]*\\]\\) \\([^;]*\\); \\(.*\\) (\\([^()]*\\))$" nil t))))

  parent reply	other threads:[~2011-07-13 18:57 UTC|newest]

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-07-01 16:37 [PATCH 2/2] [RFC] possible solution for "Race condition for '*' command" Austin Clements
2011-07-02 14:20 ` Pieter Praet
2011-07-03 17:17   ` Austin Clements
2011-07-04  6:51     ` [PROTO] " Pieter Praet
2011-07-04  6:51       ` [PATCH 1/5] emacs: add property "matched-msgids" to each search result Pieter Praet
2011-07-04  6:51       ` [PATCH 2/5] emacs: add some functions to fetch the matched-msgids of a (region of) search result(s) Pieter Praet
2011-07-04  6:51       ` [PATCH 3/5] emacs: stashing (a region of) matched-msgids Pieter Praet
2011-07-04  6:51       ` [PATCH 4/5] test: emacs: add/remove tags from all matching messages with `notmuch-search-operate-all' Pieter Praet
2011-07-04  6:51       ` [PATCH 5/5] emacs: make `notmuch-search-operate-all' use matched-msgids instead of the original query string Pieter Praet
2011-07-04 17:56       ` [PROTO] possible solution for "Race condition for '*' command" Austin Clements
2011-07-04 18:48         ` Pieter Praet
2011-07-05 19:04           ` Pieter Praet
2011-07-05 21:42             ` Austin Clements
2011-07-10 14:11               ` [PATCH] emacs: bad regexp @ `notmuch-search-process-filter' Pieter Praet
2011-07-11 20:43               ` [PATCH v2] " Pieter Praet
2011-07-11 21:05                 ` Austin Clements
2011-07-13 14:16                   ` Pieter Praet
2011-07-13 14:47                     ` David Edmondson
2011-07-13 18:57                     ` Austin Clements [this message]
2011-07-16 15:07                       ` Pieter Praet
2011-07-20  4:50                       ` servilio
2011-07-20 20:50                       ` JSON parsing performance (was Re: [PATCH v2] emacs: bad regexp @ `notmuch-search-process-filter') Austin Clements
2011-08-12  8:07                   ` [PATCH v2] emacs: bad regexp @ `notmuch-search-process-filter' Sebastian Spaeth
2011-08-12  8:28                     ` Austin Clements
2011-08-03 20:47               ` [PROTO] possible solution for "Race condition for '*' command" Austin Clements
2011-08-03 21:42                 ` Jameson Graef Rollins
2011-08-03 22:21                   ` Austin Clements

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://notmuchmail.org/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20110713185721.GI25558@mit.edu \
    --to=amdragon@mit.edu \
    --cc=dme@dme.org \
    --cc=notmuch@notmuchmail.org \
    --cc=pieter@praet.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhetil.org/notmuch.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).