JSON field names in terminal/pager output

JSON field names in terminal/pager output

[Eric Wong]

Are "f", "s", "t", "c" acceptable field names to show in JSON
output?  (instead of from/subject/to/cc)

The single-chars have been used in the search queries for as
long as we've had search.  And I stole that UI bit from
mairix(1), so there's prior art.

Anyways, the current JSON output looks something like the
following, comments inline

[{
  "blob": "d21717dae7e18dbc9efcd5ca57fe2ee92747bc06",
  "docid": 41946,
  "dt": "2021-01-05T09:24:49Z",

  # dt: is not an exact match for current WWW behavior which
  # needs YYYYMMDDHHMMSS (all digits).  Getting Xapian to parse
  # dates from Perl (w/o custom C++) isn't possible, yet.
  # dt: is the date header, "UTCDate" in JMAP.

  "f": "Eric Wong <e@80x24.org>",

  # "from": might be more obvious, but seeing it thousands of
  # times every message might be too much

  "m": "<20210105092449.GA22853@dcvr>",

  # mid/msgid/message-id

  "rcvd": "2021-01-05T09:24:49Z",
  # IMAP calls this INTERNALDATE, JMAP calls it "receivedAt"

  "references": [
    "<20210105090437.22801-1-e@80x24.org>",
    "<20210105090437.22801-5-e@80x24.org>"
  ],

  # we don't actually support searching on rereference, yet;
  # not sure if it's needed since we already do thread # expansion

  "relevance": 13,

  # relevance is ->get_percent from Xapian

  "s": "JSON pretty-printing [was: [4/4] ... (and maybe lei)]",
  "t": "meta@public-inbox.org"

  # Subject and To: headers
}, {
  # Another message, we do "}, {" to save vertical white space
  rather than "},\n{"

  "blob": "0ee1d7d9ec9b29c1d8f103033ed06e9e48e6ebfe",
  "docid": 41930,

  # side note: it's probably not worth displaying docid for
  # ephemeral search indices like LeiXSearch

  "dt": "2021-01-05T09:04:36Z",
  "f": "Eric Wong <e@80x24.org>",
  "m": "<20210105090437.22801-4-e@80x24.org>",
  "rcvd": "2021-01-05T09:04:38Z",
  "references": [
    "<20210105090437.22801-1-e@80x24.org>"
  ],
  "relevance": 36,
  "s": "[PATCH 3/4] lei: use client env as-is, drop daemon-env command",
  "t": "meta@public-inbox.org"
}, {


...

Re: JSON field names in terminal/pager output

[Kyle Meyer]

Eric Wong writes:

> Are "f", "s", "t", "c" acceptable field names to show in JSON
> output?  (instead of from/subject/to/cc)

In my view they are, and, as you mention next, I like that they align
with the search prefixes.

> Anyways, the current JSON output looks something like the
> following, comments inline
>
> [{
[...]

Nice.

>   "references": [
>     "<20210105090437.22801-1-e@80x24.org>",
>     "<20210105090437.22801-5-e@80x24.org>"
>   ],
>
>   # we don't actually support searching on rereference, yet;
>   # not sure if it's needed since we already do thread # expansion

Is "thread # expansion" the "num -> tid -> nums" mapping done by Over's
expand_thread()?  And, from the CLI, that will be triggered by
`thread|t', right?

Re: JSON field names in terminal/pager output

[Eric Wong]

Kyle Meyer <kyle@kyleam.com> wrote:
> Eric Wong writes:
> 
> > Are "f", "s", "t", "c" acceptable field names to show in JSON
> > output?  (instead of from/subject/to/cc)
> 
> In my view they are, and, as you mention next, I like that they align
> with the search prefixes.
> 
> > Anyways, the current JSON output looks something like the
> > following, comments inline
> >
> > [{
> [...]
> 
> Nice.

Btw, I remember there's several JSON streaming formats.  I think
"JSONL" aka "NDJSON" aka "LDJSON" is the best for interopability
with tools like jq(1) and will be the default format when stdout
is to a pipe or regular file.

"Concatenated JSON" may be nice for pretty-printing

<RS>-delimited JSON is gross, but there's an RFC behind it...

cf.  https://en.wikipedia.org/wiki/JSON_streaming

> 
> >   "references": [
> >     "<20210105090437.22801-1-e@80x24.org>",
> >     "<20210105090437.22801-5-e@80x24.org>"
> >   ],
> >
> >   # we don't actually support searching on rereference, yet;
> >   # not sure if it's needed since we already do thread # expansion
> 
> Is "thread # expansion" the "num -> tid -> nums" mapping done by Over's
> expand_thread()?  And, from the CLI, that will be triggered by
> `thread|t', right?

Yup.  It works well for single sources (and even a single
extindex), but not when combinining multiple inboxes/extindices
since THREADID ({tid}) is per-inbox/extindex.  So extra dedupe
work needs to be done...

So searching across externals needs to be parallelized.  I'm
thinking SOCK_SEQPACKET Unix sockets is ideal for local
externals and hope it's usable in the modern *BSDs.

And SIGPIPE on pager exit needs to get delivered in a timely
fashion, too... (yes, little things like that really bug me :x)