Show email adress in output of `notmuch search --format json`

Show email adress in output of `notmuch search --format json`

[Edgar Hipp]

Hello !

I'm building a tool that combines fzf (a fuzzy finder) and notmuch to
find email more interactively.

For that, it would be great if the --format=json would return not just
the author name, but also the author email.

I run : 

```
notmuch search  --format=json '*' | tail -1
```

And get following output : 

```
{"thread": "0000000000000b72", "timestamp": 1397639242, "date_relative": "2014-04-16", "matched": 1, "total": 1, "authors": "-a l'attention de contact-", "subject": "[SPAM] Lettre d'information", "query": ["id:SKT8635X-JCWL-FS8M-IDH7-WTK8XFCLCQ@idealcoursier.com", null], "tags": ["inbox"]}]
```

I would like to get the real email adress too : 

```
{"thread": "0000000000000b72", "timestamp": 1397639242, "date_relative": "2014-04-16", "matched": 1, "total": 1, "authoremail": "contact@idealcoursier.com", "authors": "-a l'attention de contact-", "subject": "[SPAM] Lettre d'information", "query": ["id:SKT8635X-JCWL-FS8M-IDH7-WTK8XFCLCQ@idealcoursier.com", null], "tags": ["inbox"]}]
```

More generally, it would be great to have a configuration option to
specify which fields you want, like how `git log --format=` works.

What do you think ?

Best,

Edgar

Re: Show email adress in output of `notmuch search --format json`

[David Edmondson]

On Thursday, 2017-06-01 at 08:59:17 +0200, Edgar Hipp wrote:

> Hello !
>
> I'm building a tool that combines fzf (a fuzzy finder) and notmuch to
> find email more interactively.
>
> For that, it would be great if the --format=json would return not just
> the author name, but also the author email.
>
> I run : 
>
> ```
> notmuch search  --format=json '*' | tail -1
> ```
>
> And get following output : 
>
> ```
> {"thread": "0000000000000b72", "timestamp": 1397639242, "date_relative": "2014-04-16", "matched": 1, "total": 1, "authors": "-a l'attention de contact-", "subject": "[SPAM] Lettre d'information", "query": ["id:SKT8635X-JCWL-FS8M-IDH7-WTK8XFCLCQ@idealcoursier.com", null], "tags": ["inbox"]}]
> ```
>
> I would like to get the real email adress too : 
>
> ```
> {"thread": "0000000000000b72", "timestamp": 1397639242, "date_relative": "2014-04-16", "matched": 1, "total": 1, "authoremail": "contact@idealcoursier.com", "authors": "-a l'attention de contact-", "subject": "[SPAM] Lettre d'information", "query": ["id:SKT8635X-JCWL-FS8M-IDH7-WTK8XFCLCQ@idealcoursier.com", null], "tags": ["inbox"]}]
> ```
>
> More generally, it would be great to have a configuration option to
> specify which fields you want, like how `git log --format=` works.
>
> What do you think ?

You're using the output of “search”:

% notmuch search --format=json id:20170601065917.tb7enmxklevslevt@fr-fadpc15.europe.altair.com
[{"thread": "0000000000037667", "timestamp": 1496300357, "date_relative": "Today 07:59", "matched": 1, "total": 1, "authors": "Edgar Hipp", "subject": "Show email adress in output of `notmuch search --format json`", "query": ["id:20170601065917.tb7enmxklevslevt@fr-fadpc15.europe.altair.com", null], "tags": ["inbox4", "list", "notmuch"]}]

Perhaps the output of “show” is more what you're after?

% notmuch show --format=json id:20170601065917.tb7enmxklevslevt@fr-fadpc15.europe.altair.com
[[[{"id": "20170601065917.tb7enmxklevslevt@fr-fadpc15.europe.altair.com", "match": true, "excluded": false, "filename": "/home/dme/Maildir/.d.list.notmuch/cur/1496309415_0.21047.disaster-area,U=12312,FMD5=48696af6d61a012532c1ce55c0592d76:2,S", "timestamp": 1496300357, "date_relative": "Today 07:59", "tags": ["inbox4", "list", "notmuch"], "headers": {"Subject": "Show email adress in output of `notmuch search --format json`", "From": "Edgar Hipp <contact@javascript-ninja.fr>", "To": "notmuch@notmuchmail.org", "Date": "Thu, 01 Jun 2017 08:59:17 +0200"}, "body": [{"id": 1, "content-type": "text/plain", "content": "Hello !\n\nI'm building a tool that combines fzf (a fuzzy finder) and notmuch to\nfind email more interactively.\n\nFor that, it would be great if the --format=json would return not just\nthe author name, but also the author email.\n\nI run : \n\n```\nnotmuch search  --format=json '*' | tail -1\n```\n\nAnd get following output : \n\n```\n{\"thread\": \"0000000000000b72\", \"timestamp\": 1397639242, \"date_relative\": \"2014-04-16\", \"matched\": 1, \"total\": 1, \"authors\": \"-a l'attention de contact-\", \"subject\": \"[SPAM] Lettre d'information\", \"query\": [\"id:SKT8635X-JCWL-FS8M-IDH7-WTK8XFCLCQ@idealcoursier.com\", null], \"tags\": [\"inbox\"]}]\n```\n\nI would like to get the real email adress too : \n\n```\n{\"thread\": \"0000000000000b72\", \"timestamp\": 1397639242, \"date_relative\": \"2014-04-16\", \"matched\": 1, \"total\": 1, \"authoremail\": \"contact@idealcoursier.com\", \"authors\": \"-a l'attention de contact-\", \"subject\": \"[SPAM] Lettre d'information\", \"query\": [\"id:SKT8635X-JCWL-FS8M-IDH7-WTK8XFCLCQ@idealcoursier.com\", null], \"tags\": [\"inbox\"]}]\n```\n\nMore generally, it would be great to have a configuration option to\nspecify which fields you want, like how `git log --format=` works.\n\nWhat do you think ?\n\nBest,\n\nEdgar\n\n_______________________________________________\nnotmuch mailing list\nnotmuch@notmuchmail.org\nhttps://notmuchmail.org/mailman/listinfo/notmuch\n"}]}, []]]]
% 

The “search” output is meant purely for displaying a list of threads, so
the information presented is limited to whatever was considered
necessary for that.

dme.
-- 
And the sign said long haired freaky people need not apply.

Re: Show email adress in output of `notmuch search --format json`

[Edgar Hipp]

On Thu, Jun 01, 2017 at 10:41:02AM +0100, David Edmondson wrote:
> On Thursday, 2017-06-01 at 08:59:17 +0200, Edgar Hipp wrote:
> 
> > Hello !
> >
> > I'm building a tool that combines fzf (a fuzzy finder) and notmuch to
> > find email more interactively.
> >
> > For that, it would be great if the --format=json would return not just
> > the author name, but also the author email.
> >
> > I run : 
> >
> > ```
> > notmuch search  --format=json '*' | tail -1
> > ```
> >
> > And get following output : 
> >
> > ```
> > {"thread": "0000000000000b72", "timestamp": 1397639242, "date_relative": "2014-04-16", "matched": 1, "total": 1, "authors": "-a l'attention de contact-", "subject": "[SPAM] Lettre d'information", "query": ["id:SKT8635X-JCWL-FS8M-IDH7-WTK8XFCLCQ@idealcoursier.com", null], "tags": ["inbox"]}]
> > ```
> >
> > I would like to get the real email adress too : 
> >
> > ```
> > {"thread": "0000000000000b72", "timestamp": 1397639242, "date_relative": "2014-04-16", "matched": 1, "total": 1, "authoremail": "contact@idealcoursier.com", "authors": "-a l'attention de contact-", "subject": "[SPAM] Lettre d'information", "query": ["id:SKT8635X-JCWL-FS8M-IDH7-WTK8XFCLCQ@idealcoursier.com", null], "tags": ["inbox"]}]
> > ```
> >
> > More generally, it would be great to have a configuration option to
> > specify which fields you want, like how `git log --format=` works.
> >
> > What do you think ?
> 
> You're using the output of “search”:
> 
> % notmuch search --format=json id:20170601065917.tb7enmxklevslevt@fr-fadpc15.europe.altair.com
> [{"thread": "0000000000037667", "timestamp": 1496300357, "date_relative": "Today 07:59", "matched": 1, "total": 1, "authors": "Edgar Hipp", "subject": "Show email adress in output of `notmuch search --format json`", "query": ["id:20170601065917.tb7enmxklevslevt@fr-fadpc15.europe.altair.com", null], "tags": ["inbox4", "list", "notmuch"]}]
> 
> Perhaps the output of “show” is more what you're after?
> 
> % notmuch show --format=json id:20170601065917.tb7enmxklevslevt@fr-fadpc15.europe.altair.com
> [[[{"id": "20170601065917.tb7enmxklevslevt@fr-fadpc15.europe.altair.com", "match": true, "excluded": false, "filename": "/home/dme/Maildir/.d.list.notmuch/cur/1496309415_0.21047.disaster-area,U=12312,FMD5=48696af6d61a012532c1ce55c0592d76:2,S", "timestamp": 1496300357, "date_relative": "Today 07:59", "tags": ["inbox4", "list", "notmuch"], "headers": {"Subject": "Show email adress in output of `notmuch search --format json`", "From": "Edgar Hipp <contact@javascript-ninja.fr>", "To": "notmuch@notmuchmail.org", "Date": "Thu, 01 Jun 2017 08:59:17 +0200"}, "body": [{"id": 1, "content-type": "text/plain", "content": "Hello !\n\nI'm building a tool that combines fzf (a fuzzy finder) and notmuch to\nfind email more interactively.\n\nFor that, it would be great if the --format=json would return not just\nthe author name, but also the author email.\n\nI run : \n\n```\nnotmuch search  --format=json '*' | tail -1\n```\n\nAnd get following output : \n\n```\n{\"thread\": \"0000000000000b72\", \"timestamp\": 1397639242, \"date_relative\": \"2014-04-16\", \"matched\": 1, \"total\": 1, \"authors\": \"-a l'attention de contact-\", \"subject\": \"[SPAM] Lettre d'information\", \"query\": [\"id:SKT8635X-JCWL-FS8M-IDH7-WTK8XFCLCQ@idealcoursier.com\", null], \"tags\": [\"inbox\"]}]\n```\n\nI would like to get the real email adress too : \n\n```\n{\"thread\": \"0000000000000b72\", \"timestamp\": 1397639242, \"date_relative\": \"2014-04-16\", \"matched\": 1, \"total\": 1, \"authoremail\": \"contact@idealcoursier.com\", \"authors\": \"-a l'attention de contact-\", \"subject\": \"[SPAM] Lettre d'information\", \"query\": [\"id:SKT8635X-JCWL-FS8M-IDH7-WTK8XFCLCQ@idealcoursier.com\", null], \"tags\": [\"inbox\"]}]\n```\n\nMore generally, it would be great to have a configuration option to\nspecify which fields you want, like how `git log --format=` works.\n\nWhat do you think ?\n\nBest,\n\nEdgar\n\n_______________________________________________\nnotmuch mailing list\nnotmuch@notmuchmail.org\nhttps://notmuchmail.org/mailman/listinfo/notmuch\n"}]}, []]]]
> % 
> 
> The “search” output is meant purely for displaying a list of threads, so
> the information presented is limited to whatever was considered
> necessary for that.
> 
> dme.
> -- 
> And the sign said long haired freaky people need not apply.

Sorry, I wasn't completely clear.

What I'm doing is basically 

notmuch search --format=json '*' | processing_program | fzf

Which makes it possible to search trough messages interactively : fzf is
a fuzzy finder(see the gif on the page https://github.com/junegunn/fzf)

What I would expect is to have the same format as show but for multiple
messages, (in my case for search).

That's why I would find it interesting to be able to customize the
fields shown in search.

Best,

Edgar

Re: Show email adress in output of `notmuch search --format json`

[David Edmondson]

On Friday, 2017-06-02 at 10:35:03 +0200, Edgar Hipp wrote:

> What I'm doing is basically 
>
> notmuch search --format=json '*' | processing_program | fzf
>
> Which makes it possible to search trough messages interactively : fzf is
> a fuzzy finder(see the gif on the page https://github.com/junegunn/fzf)
>
> What I would expect is to have the same format as show but for multiple
> messages, (in my case for search).
>
> That's why I would find it interesting to be able to customize the
> fields shown in search.

Understood. There is no general ability to customise the fields today.

Would:

  notmuch show --body=false --format=json "*" | other_processing | fzf

let you achieve your goal? (The processing would have to change, of
course.)

In general, I wonder whether “search” or “show” for “*” will give you
the performance you want if fzf waits for all of the input before
allowing the user to start matching.

For example, with 0.5 million messages in a database:

% time notmuch search --format=json "*" > /dev/null
notmuch search --format=json "*" > /dev/null  108.14s user 35.55s system 75% cpu 3:10.18 total
% 

(Not a particularly fast machine, but all of the relevant stuff is on
SSD.)

If I had to wait 108 seconds to start matching it would be
unusable. “show” will probably be slower.

dme.
-- 
You know your green from your red.

Re: Show email adress in output of `notmuch search --format json`

[Edgar Hipp]

On Fri, Jun 02, 2017 at 10:08:30AM +0100, David Edmondson wrote:
> On Friday, 2017-06-02 at 10:35:03 +0200, Edgar Hipp wrote:
> 
> > What I'm doing is basically 
> >
> > notmuch search --format=json '*' | processing_program | fzf
> >
> > Which makes it possible to search trough messages interactively : fzf is
> > a fuzzy finder(see the gif on the page https://github.com/junegunn/fzf)
> >
> > What I would expect is to have the same format as show but for multiple
> > messages, (in my case for search).
> >
> > That's why I would find it interesting to be able to customize the
> > fields shown in search.
> 
> Understood. There is no general ability to customise the fields today.
> 
> Would:
> 
>   notmuch show --body=false --format=json "*" | other_processing | fzf
> 
> let you achieve your goal? (The processing would have to change, of
> course.)
> 
> In general, I wonder whether “search” or “show” for “*” will give you
> the performance you want if fzf waits for all of the input before
> allowing the user to start matching.
> 
> For example, with 0.5 million messages in a database:
> 
> % time notmuch search --format=json "*" > /dev/null
> notmuch search --format=json "*" > /dev/null  108.14s user 35.55s system 75% cpu 3:10.18 total
> % 
> 
> (Not a particularly fast machine, but all of the relevant stuff is on
> SSD.)
> 
> If I had to wait 108 seconds to start matching it would be
> unusable. “show” will probably be slower.
> 
> dme.
> -- 
> You know your green from your red.

It seems that my notmuch show does'nt show multiple outputs : 

$ notmuch --version
notmuch 0.24.1

$ notmuch show --body=false --format=json "*" | wc -l
1

$ notmuch search --format=json "*" | wc -l
3481

Thanks in advance,

Edgar

Re: Show email adress in output of `notmuch search --format json`

[Edgar Hipp]

On Fri, Jun 02, 2017 at 11:28:06AM +0200, Edgar Hipp wrote:
> On Fri, Jun 02, 2017 at 10:08:30AM +0100, David Edmondson wrote:
> > On Friday, 2017-06-02 at 10:35:03 +0200, Edgar Hipp wrote:
> > 
> > > What I'm doing is basically 
> > >
> > > notmuch search --format=json '*' | processing_program | fzf
> > >
> > > Which makes it possible to search trough messages interactively : fzf is
> > > a fuzzy finder(see the gif on the page https://github.com/junegunn/fzf)
> > >
> > > What I would expect is to have the same format as show but for multiple
> > > messages, (in my case for search).
> > >
> > > That's why I would find it interesting to be able to customize the
> > > fields shown in search.
> > 
> > Understood. There is no general ability to customise the fields today.
> > 
> > Would:
> > 
> >   notmuch show --body=false --format=json "*" | other_processing | fzf
> > 
> > let you achieve your goal? (The processing would have to change, of
> > course.)
> > 
> > In general, I wonder whether “search” or “show” for “*” will give you
> > the performance you want if fzf waits for all of the input before
> > allowing the user to start matching.
> > 
> > For example, with 0.5 million messages in a database:
> > 
> > % time notmuch search --format=json "*" > /dev/null
> > notmuch search --format=json "*" > /dev/null  108.14s user 35.55s system 75% cpu 3:10.18 total
> > % 
> > 
> > (Not a particularly fast machine, but all of the relevant stuff is on
> > SSD.)
> > 
> > If I had to wait 108 seconds to start matching it would be
> > unusable. “show” will probably be slower.
> > 
> > dme.
> > -- 
> > You know your green from your red.
> 
> It seems that my notmuch show does'nt show multiple outputs : 
> 
> $ notmuch --version
> notmuch 0.24.1
> 
> $ notmuch show --body=false --format=json "*" | wc -l
> 1
> 
> $ notmuch search --format=json "*" | wc -l
> 3481
> 
> Thanks in advance,
> 
> Edgar

fzf starts as soon as one line is in stdin.
I use it a lot for log searching , and also in real time.

When I use my search command, I will probably first filter the results
very grossly (to have less than 1000 mails to fuzzy find).

Re: Show email adress in output of `notmuch search --format json`

[David Edmondson]

On Friday, 2017-06-02 at 11:28:06 +0200, Edgar Hipp wrote:

> It seems that my notmuch show does'nt show multiple outputs : 
>
> $ notmuch --version
> notmuch 0.24.1
>
> $ notmuch show --body=false --format=json "*" | wc -l
> 1
>
> $ notmuch search --format=json "*" | wc -l
> 3481

Ouch. Works for me with 0.23.7.

> fzf starts as soon as one line is in stdin.
> I use it a lot for log searching , and also in real time.

Cool.

> When I use my search command, I will probably first filter the results
> very grossly (to have less than 1000 mails to fuzzy find).

Won't your filter then end up waiting for all of the input?

dme.
-- 
Ah, oh your hair is beautiful.

Re: Show email adress in output of `notmuch search --format json`

[Edgar Hipp]

Indeed, notmuch show works, i just saw one line, but actually, the line is an array of messages `[]`, so I guess I can make it work. 

For the filtering, fzf starts filtering whenever there is some input, and the filter is also applied to new lines. 

I will send you a gif once my script is ready if you're curious