JSON Output Documentation

JSON Output Documentation

[Tim Culverhouse]

[-- Attachment #1.1: Type: text/plain, Size: 629 bytes --]

Hey everyone -

I've scoured the manpages and website for some documentation on the JSON
output format, but haven't found what I'm looking for. For the most
part, the commands are easy to figure out, but `notmuch show` has me
confused on the array nesting. Does anyone know if there is
documentation on this structure?

It seems to be something like:

[ (Array of results)
  [ (Array of thread)
    [  (Another array of thread?)
      {Thread parent object},
      [ (array of thread children objects)]
    ]
  ]
]

I just can't figure out why there are two arrays wrapping the thread?

Thanks!

-- 
Tim

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: JSON Output Documentation

[David Bremner]

"Tim Culverhouse" <tim@timculverhouse.com> writes:

> Hey everyone -
>
> I've scoured the manpages and website for some documentation on the JSON
> output format, but haven't found what I'm looking for. For the most
> part, the commands are easy to figure out, but `notmuch show` has me
> confused on the array nesting. Does anyone know if there is
> documentation on this structure?
>
> It seems to be something like:
>
> [ (Array of results)
>   [ (Array of thread)
>     [  (Another array of thread?)
>       {Thread parent object},
>       [ (array of thread children objects)]
>     ]
>   ]
> ]

We have had some discussions on the pros and cons of more publically
documenting this interface, but for now you can find the documentation
in the source, in devel/schemata.

At this point I'm open to people who either want to convert that
document to rst and integrate it with the published docs, or convince me
it's a bad idea. I think the idea was that we wanted to avoid commiting
to that interface, but since we already version it like a C library, I'm
a bit fuzzy on why that's a problem.

d

Re: JSON Output Documentation

[David Bremner]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

David Bremner <david@tethera.net> writes:

> "Tim Culverhouse" <tim@timculverhouse.com> writes:
>
>> Hey everyone -
>>
>> I've scoured the manpages and website for some documentation on the JSON
>> output format, but haven't found what I'm looking for. For the most
>> part, the commands are easy to figure out, but `notmuch show` has me
>> confused on the array nesting. Does anyone know if there is
>> documentation on this structure?
>>
>> It seems to be something like:
>>
>> [ (Array of results)
>>   [ (Array of thread)
>>     [  (Another array of thread?)
>>       {Thread parent object},
>>       [ (array of thread children objects)]
>>     ]
>>   ]
>> ]
>
> We have had some discussions on the pros and cons of more publically
> documenting this interface, but for now you can find the documentation
> in the source, in devel/schemata.

See attached.


[-- Attachment #2: schemata --]
[-- Type: application/octet-stream, Size: 7452 bytes --]

This file describes the schemata used for notmuch's structured output
format (currently JSON and S-Expressions).

[]'s indicate lists.  List items can be marked with a '?', meaning
they are optional; or a '*', meaning there can be zero or more of that
item.  {}'s indicate an object that maps from field identifiers to
values.  An object field marked '?' is optional; one marked with '*'
can repeat (with a different name). |'s indicate alternates (e.g.,
int|string means something can be an int or a string).

For S-Expression output, lists are printed delimited by () instead of
[]. Objects are printed as p-lists, i.e. lists where the keys and values
are interleaved. Keys are printed as keywords (symbols preceded by a
colon), e.g. (:id "123" :time 54321 :from "foobar"). Null is printed as
nil, true as t and false as nil.

This is version 5 of the structured output format.

Version history
---------------

v1
- First versioned schema release.
- Added part.content-length and part.content-transfer-encoding fields.

v2
- Added the thread_summary.query field.

v3
- Replaced message.filename string with a list of filenames.
- Added part.content-disposition field.

v4
- replace signature error integer bitmask with a set of flags for
  individual errors.
- (notmuch 0.29) added message.crypto to identify overall message
  cryptographic state

v5
- sorting support for notmuch show (no change to actual schema,
  just new command line argument)

Common non-terminals
--------------------

# Number of seconds since the Epoch
unix_time = int

# Thread ID, sans "thread:"
threadid = string

# Message ID, sans "id:"
messageid = string

# E-mail header name, sans trailing colon, like "Subject" or "In-Reply-To"
header_name = string

notmuch show schema
-------------------

# A top-level set of threads (do_show)
# Returned by notmuch show without a --part argument
thread_set = [thread*]

# Top-level messages in a thread (show_messages)
thread = [thread_node*]

# A message and its replies (show_messages)
thread_node = [
    message|null,             # null if not matched and not --entire-thread
    [thread_node*]            # children of message
]

# A message (format_part_sprinter)
message = {
    # (format_message_sprinter)
    id:             messageid,
    match:          bool,
    filename:	    [string*],
    timestamp:      unix_time, # date header as unix time
    date_relative:  string,   # user-friendly timestamp
    tags:           [string*],

    headers:        headers,
    crypto:         crypto,
    duplicate:      integer,
    body?:          [part]    # omitted if --body=false
}

# when showing the message, was any or all of it decrypted?
msgdecstatus: "full"|"partial"

# The overall cryptographic state of the message as a whole:
crypto = {
    signed?:    {
                  status:      sigstatus,
                  # was the set of signatures described under encrypted cover?
                  encrypted:   bool,
                  # which of the headers is covered by sigstatus?
                  headers:     [header_name*]
                },
    decrypted?: {
                  status: msgdecstatus,
                  # map encrypted headers that differed from the outside headers.
                  # the value of each item in the map is what that field showed externally
                  # (maybe null if it was not present in the external headers).
                  header-mask:  { header_name*: string|null }
                }
}

# A MIME part (format_part_sprinter)
part = {
    id:             int|string, # part id (currently DFS part number)

    encstatus?:     encstatus,
    sigstatus?:     sigstatus,

    content-type:   string,
    content-disposition?:       string,
    content-id?:    string,
    # if content-type starts with "multipart/":
    content:        [part*],
    # if content-type is "message/rfc822":
    content:        [{headers: headers, body: [part]}],
    # otherwise (leaf parts):
    filename?:      string,
    content-charset?: string,
    # A leaf part's body content is optional, but may be included if
    # it can be correctly encoded as a string.  Consumers should use
    # this in preference to fetching the part content separately.
    content?:       string,
    # If a leaf part's body content is not included, the length of
    # the encoded content (in bytes) may be given instead.
    content-length?: int,
    # If a leaf part's body content is not included, its transfer encoding
    # may be given.  Using this and the encoded content length, it is
    # possible for the consumer to estimate the decoded content length.
    content-transfer-encoding?: string
}

# The headers of a message or part (format_headers_sprinter with reply = FALSE)
headers = {
    Subject:        string,
    From:           string,
    To?:            string,
    Cc?:            string,
    Bcc?:           string,
    Reply-To?:      string,
    Date:           string,
    extra_header_pair*
}

extra_header_pair=  (header_name: string)
# Encryption status (format_part_sprinter)
encstatus = [{status: "good"|"bad"}]

# Signature status (format_part_sigstatus_sprinter)
sigstatus = [signature*]

signature = {
    # (signature_status_to_string)
    status:         "good"|"bad"|"error"|"unknown",
    # if status is "good":
    fingerprint?:   string,
    created?:       unix_time,
    expires?:       unix_time,
    userid?:        string
    email?:         string
    # if status is not "good":
    keyid?:         string
    errors?: 	    sig_errors
}

sig_errors = {
    key-revoked?: bool,
    key-expired?: bool,
    sig-expired?: bool,
    key-missing?: bool,
    alg-unsupported?: bool,
    crl-missing?: bool,
    crl-too-old?: bool,
    bad-policy?: bool,
    sys-error?: bool,
    tofu-conflict?: bool
}

notmuch search schema
---------------------

# --output=summary
search_summary = [thread_summary*]

# --output=threads
search_threads = [threadid*]

# --output=messages
search_messages = [messageid*]

# --output=files
search_files = [string*]

# --output=tags
search_tags = [string*]

thread_summary = {
    thread:         threadid,
    timestamp:      unix_time,
    date_relative:  string,   # user-friendly timestamp
    matched:        int,      # number of matched messages
    total:          int,      # total messages in thread
    authors:        string,   # comma-separated names with | between
                              # matched and unmatched
    subject:        string,
    tags:           [string*],

    # Two stable query strings identifying exactly the matched and
    # unmatched messages currently in this thread.  The messages
    # matched by these queries will not change even if more messages
    # arrive in the thread.  If there are no matched or unmatched
    # messages, the corresponding query will be null (there is no
    # query that matches nothing).  (Added in schema version 2.)
    query:          [string|null, string|null],
}

notmuch reply schema
--------------------

reply = {
    # The headers of the constructed reply
    reply-headers: reply_headers,

    # As in the show format (format_part_sprinter)
    original: message
}

# Reply headers (format_headers_sprinter with reply = TRUE)
reply_headers = {
    Subject:        string,
    From:           string,
    To?:            string,
    Cc?:            string,
    Bcc?:           string,
    In-reply-to:    string,
    References:     string
}

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: JSON Output Documentation

[Tim Culverhouse]

[-- Attachment #1.1: Type: text/plain, Size: 298 bytes --]

On Wed May 3, 2023 at 2:35 PM CDT, David Bremner wrote:
> > We have had some discussions on the pros and cons of more publically
> > documenting this interface, but for now you can find the documentation
> > in the source, in devel/schemata.
>
> See attached.

Perfect! Thanks!

-- 
Tim

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]