The overloading of show (was Re: [PATCH] Output unmodified Content-Type header value for JSON format.)

The overloading of show (was Re: [PATCH] Output unmodified Content-Type header value for JSON format.)

[Austin Clements]

(was in reply to id:87ehv2proa.fsf@praet.org, but I wanted to start a
new top-level thread)

Quoth Pieter Praet on Jan 14 at 10:19 am:
> On Thu, 12 Jan 2012 12:28:40 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > Quoth Pieter Praet on Jan 12 at  6:07 pm:
> > > On Tue, 22 Nov 2011 22:40:21 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> > > > Quoth Jameson Graef Rollins on Nov 20 at 12:10 pm:
> > > > > The open question seems to be how we handle the content encoding
> > > > > parameters.  My argument is that those should either be used by notmuch
> > > > > to properly encode the content for the consumer.  If that's not
> > > > > possible, then just those parameters needed by the consumer to decode
> > > > > the content should be output.
> > > > 
> > > > If notmuch is going to include part content in the JSON output (which
> > > > perhaps it shouldn't, as per recent IRC discussions), then it must
> > > > handle content encodings because JSON must be Unicode and therefore
> > > > the content strings in the JSON must be Unicode.
> > > 
> > > Having missed the IRC discussions: what is the rationale for not
> > > including (specific types of?) part content in the JSON output ?
> > > Eg. how about inline attached text/x-patch ?
> > 
> > Technically the IRC discussion was about not including *any* part
> > content in the JSON output, and always using show --format=raw or
> > similar to retrieve desired parts.  Currently, notmuch includes part
> > content in the JSON only for text/*, *except* when it's text/html.  I
> > assume non-text parts are omitted because binary data is hard to
> > represent in JSON and text/html is omitted because some people don't
> > need it.  However, this leads to some peculiar asymmetry in the Emacs
> > code where sometimes it pulls part content out of the JSON and
> > sometimes it retrieves it using show --format=raw.  This in turn leads
> > to asymmetry in content encoding handling, since notmuch handles
> > content encoding for parts included in the JSON (and there's no good
> > way around that since JSON is Unicode), but not for parts retrieved as
> > raw.
> > 
> > The idea discussed on IRC was to remove all part content from the JSON
> > output and to always use show to retrieve it, possibly beefing up
> > show's support for content decoding (and possibly introducing a way to
> > retrieve multiple raw parts at once to avoid re-parsing).  This would
> > get the JSON format out of the business of guessing what consumers
> > need, simplify the Emacs code, and normalize content encoding
> > handling.
> 
> Full ACK.
> 
> One concern though (IIUC): Due to the prevalence of retarded MUA's, not
> outputting 'text/plain' and/or 'text/html' parts is unfortunately all
> too often equivalent to not outputting anything at all, so wouldn't we,
> in essence, be reducing `show --format=json' to an ever-so-slightly
> augmented `search --format=json' ?

I'm not sure I fully understand what you're saying, but there are
several levels of structure here:

1. Threads (query results)
2. Thread structure
3. Message structure (MIME)
4. Part content

Currently, search returns 1; show --format=json returns 2, 3, and
sometimes 4 (but sometimes not); and show --format=raw returns 4.
Notably, 1 does not require opening message files (neither does 2),
which I consider an important distinction between search and show.

Some of the discussion has been about putting 4 squarely in the realm
of show --format=raw.  One counterargument (which has grown on me
since this discussion) is that the part content included in
--format=json can be thought of as pre-fetching content that clients
are likely to need in order to avoid re-parsing the message in most
circumstances.  I believe this is not the *intent* of the current
code, though without a specification of the JSON format it's hard to
tell.

Other discussion (more interesting, in my mind) has been about
separating retrieving thread structure, 2, from retrieving message
structure, 3.  To me, splitting these feels much more natural than
what we do now, which seems to be inflexibly bound to the specific way
the Emacs show mode currently works.  The thread structure is readily
available from the database, so I think separating these would open up
some new UI opportunities, particularly easy and fast thread outlining
and navigation.  I believe it would also simplify the code and address
some irritating asymmetries in the way notmuch show handles the --part
argument.

Re: The overloading of show (was Re: [PATCH] Output unmodified Content-Type header value for JSON format.)

[David Edmondson]

[-- Attachment #1: Type: text/plain, Size: 2654 bytes --]

On Sat, 14 Jan 2012 19:36:17 -0500, Austin Clements <amdragon@MIT.EDU> wrote:
> ...there are several levels of structure here:
> 
> 1. Threads (query results)
> 2. Thread structure
> 3. Message structure (MIME)
> 4. Part content
> 
> Currently, search returns 1; show --format=json returns 2, 3, and
> sometimes 4 (but sometimes not); and show --format=raw returns 4.
> Notably, 1 does not require opening message files (neither does 2),
> which I consider an important distinction between search and show.
> 
> Some of the discussion has been about putting 4 squarely in the realm
> of show --format=raw.  One counterargument (which has grown on me
> since this discussion) is that the part content included in
> --format=json can be thought of as pre-fetching content that clients
> are likely to need in order to avoid re-parsing the message in most
> circumstances.  I believe this is not the *intent* of the current
> code, though without a specification of the JSON format it's hard to
> tell.

The JSON output included what was considered useful (mostly for the
Emacs UI), but how much deep thought went into 'useful' I couldn't say.

> Other discussion (more interesting, in my mind) has been about
> separating retrieving thread structure, 2, from retrieving message
> structure, 3.  To me, splitting these feels much more natural than
> what we do now, which seems to be inflexibly bound to the specific way
> the Emacs show mode currently works.  The thread structure is readily
> available from the database, so I think separating these would open up
> some new UI opportunities, particularly easy and fast thread outlining
> and navigation.

Given that the current output already includes both 2 and 3, anything
that could be done with 2 can be done with the current output, so
there's no block on the kind of innovation that you describe other than
possibly some performance problems.

notmuch-lkml.el[1] was a quick prototype of an alternative way to find
messages to read based on suggestions from Aneesh. It could have used
the proposed 'thread structure only' output.

The changes you allude to make sense. My only concern would be any
potential impact on the current Emacs UI's use of JSON output. Switching
to a model where a typical 'notmuch-show' buffer requires many calls to
notmuch (and commensurate JSON parsing) may perform significantly worse
than the current approach.

> I believe it would also simplify the code and address some irritating
> asymmetries in the way notmuch show handles the --part argument.

Footnotes: 
[1]  http://dme.org/data/emacs/notmuch-lkml.el


[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]