Re: notmuch Digest, Vol 20, Issue 57

Re: notmuch Digest, Vol 20, Issue 57

[Sander Boer]

Hi all,

I was wondering what a "best practice" would be in dealing with a gmail
account and notmuch ?

For instance, is this a possibility sync gmail tags with notmuch tags ?
As fas as I am aware Gmail exposes its tags through imap folders, I am
not too thrilled about the mixing of paradigms (folders vs tags) and it
would be great if it could be aligned.

-- 
Sander Boer

--

"Architecture is the most fragile of all the media --it has a practical application as well as an artistic application... People demolish and change buildings all the time and an architect's work is destroyed, but you wouldn't think about altering a painting or music." --R. Venturi

--

Re: notmuch Digest, Vol 20, Issue 57

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 2738 bytes --]

On Mon, 27 Jun 2011 09:43:41 +0200, Sander Boer <sanderboer@mauc.nl> wrote:
> For instance, is this a possibility sync gmail tags with notmuch tags ?
> As fas as I am aware Gmail exposes its tags through imap folders, I am
> not too thrilled about the mixing of paradigms (folders vs tags) and it
> would be great if it could be aligned.

I'm not a gmail user myself, but I can describe some of the options
available.

First, it's important to understand that any friction here comes from
Gmail exposing its tags as folders, (which in turn could be the lack of
availability of a more tag-aware protocol than imap).

So, let's say you've done an imap-based fetch of all of your mail and
now have it in several folders corresponding to the various tags you
made in the Gmail interface, (and you've run "notmuch new" to create a
notmuch database of all of that).

At that point, you can search (with current notmuch from git---and soon
to be released as notmuch 0.6) on the folder names. This allows you to
do a search based on a Gmail tag. For example, you might have an
"important" tag in Gmail which you could search with:

	notmuch search folder:important

If you wanted to make an actual notmuch tag based on those search
results, you could do that as well:

	notmuch tag +important folder:important

And you could even do that on a regular basis, (for example in some
script that sucks down new email from Gmail). In this case you might
want to avoid re-tagging pre-tagged mail by amending the search terms:

	notmuch tag +important folder:important and not tag:important

(Note that this is an optimization that notmuch could be making
automatically, and that's something I'd love to see happen in notmuch at
some point.)

Hopefully it's clear enough that you could do the above in a script that
loops over all of your existing tags.

And if you were doing a one-time switch from Gmail to notmuch that would
be all you would need.

I don't know if you're looking to also push tags added via some notmuch
interface back to Gmail, (does Gmail even provide a mechanism for doing
that?). If so, then you'd need something that took notmuch tags and made
copies of the message in the appropriate files. That would hopefully be
easy to script based on the output of:

	notmuch search --output=files tag:important

I think my short answer is that it's fairly easy to convert from Gmail
tags to notmuch tags as part of a one-time import. Doing this on a
continual basis might benefit from writing a few scripts, and I don't
know if anyone has written those scripts yet.

Would any Gmail+notmuch users care to add anything to the conversation?

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuch Digest, Vol 20, Issue 57

[Sander Boer]

Carl Worth <cworth@cworth.org> writes:> 

[snip]

> First, it's important to understand that any friction here comes from
> Gmail exposing its tags as folders, (which in turn could be the lack of
> availability of a more tag-aware protocol than imap).
>

I agree that this is the key issue here. 
I was hoping that google somehow was able to expose the tags in the "All
Mail" folder, like the headers that are gmail specific: X-pstn-nxpr and X-pstn-nxp (which contains a
hash) for instance.


> So, let's say you've done an imap-based fetch of all of your mail and
> now have it in several folders corresponding to the various tags you
> made in the Gmail interface, (and you've run "notmuch new" to create a
> notmuch database of all of that).
>
> At that point, you can search (with current notmuch from git---and soon
> to be released as notmuch 0.6) on the folder names. This allows you to
> do a search based on a Gmail tag. For example, you might have an
> "important" tag in Gmail which you could search with:
>
> 	notmuch search folder:important
>
ah, I have the ubuntu version,  will update to git soon.


> If you wanted to make an actual notmuch tag based on those search
> results, you could do that as well:
>
> 	notmuch tag +important folder:important
>
> And you could even do that on a regular basis, (for example in some
> script that sucks down new email from Gmail). In this case you might
> want to avoid re-tagging pre-tagged mail by amending the search terms:
>
> 	notmuch tag +important folder:important and not tag:important
>
> (Note that this is an optimization that notmuch could be making
> automatically, and that's something I'd love to see happen in notmuch at
> some point.)
>
> Hopefully it's clear enough that you could do the above in a script that
> loops over all of your existing tags.
>
> And if you were doing a one-time switch from Gmail to notmuch that would
> be all you would need.
>
> I don't know if you're looking to also push tags added via some notmuch
> interface back to Gmail, (does Gmail even provide a mechanism for doing
> that?). If so, then you'd need something that took notmuch tags and made
> copies of the message in the appropriate files. That would hopefully be
> easy to script based on the output of:
>
> 	notmuch search --output=files tag:important
>
> I think my short answer is that it's fairly easy to convert from Gmail
> tags to notmuch tags as part of a one-time import. Doing this on a
> continual basis might benefit from writing a few scripts, and I don't
> know if anyone has written those scripts yet.
>
> Would any Gmail+notmuch users care to add anything to the conversation?
>
> -Carl


now using gnus still, b/c notmuch does not build for the N900 (bummer)

-- 
Sander Boer

Re: notmuch Digest, Vol 20, Issue 57

[Mark Anderson]

On Tue, 28 Jun 2011 23:43:52 +0200, "Sander Boer" <sanboer@gmail.com> wrote:
> Carl Worth <cworth@cworth.org> writes:> 
> >
> > Hopefully it's clear enough that you could do the above in a script that
> > loops over all of your existing tags.
> >
> > And if you were doing a one-time switch from Gmail to notmuch that would
> > be all you would need.
> >
> > I don't know if you're looking to also push tags added via some notmuch
> > interface back to Gmail, (does Gmail even provide a mechanism for doing
> > that?). If so, then you'd need something that took notmuch tags and made
> > copies of the message in the appropriate files. That would hopefully be
> > easy to script based on the output of:
> >
> > 	notmuch search --output=files tag:important

You'd probably actually want this:

         notmuch search --output=files tag:importand and not folder:important

Although until the folder: tag bug is fixed, it won't be as definitive
as you want, because once the message was in folder:important, it
doesn't really leave.

However, With my recent patch you'll also get more filenames than you
want for this behavior.  If you already have the mail in All Mail,
Inbox, my_special_tag, and not_that_tag, do you want 4 links or copies
of the message placed in the folder for Important?

I had thought of this, because I am a Gmail/notmuch user (well,
somewhat, I have some of the infra in place, but it's not polished, I
usually end up using phone or web)

I had briefly considered adding another output format "file", just to get a
single file for each message in the db, but the file/files distinction
feels a bit niggling.  Perhaps it should be changed to "files" and
"filelists" or something else more clear.

It was nagging me as I implemented the fix I submitted, which looks like
it has been pushed.

Any comments?  I don't think I have time to code any changes to this for
a couple days.

> > I think my short answer is that it's fairly easy to convert from Gmail
> > tags to notmuch tags as part of a one-time import. Doing this on a
> > continual basis might benefit from writing a few scripts, and I don't
> > know if anyone has written those scripts yet.

Yes, you need a label->folder as well as a folder->label part of the
script.  I think of it as:

1. label->folder sync
2. offlineimap (pushes label changes, pulls new mail)
3. folder->label sync

There is certainly room for some conflict, if you use multiple
interfaces.  So watch your head.

-Mark

> >
> > Would any Gmail+notmuch users care to add anything to the conversation?
> >
> > -Carl
> 
> 
> now using gnus still, b/c notmuch does not build for the N900 (bummer)
> 
> -- 
> Sander Boer
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: notmuch Digest, Vol 20, Issue 57

[Mark Anderson]

On Tue, 28 Jun 2011 23:43:52 +0200, "Sander Boer" <sanboer@gmail.com> wrote:
> 
> Carl Worth <cworth@cworth.org> writes:> 
> I was hoping that google somehow was able to expose the tags in the "All
> Mail" folder, like the headers that are gmail specific: X-pstn-nxpr and X-pstn-nxp (which contains a
> hash) for instance.

I don't even see those headers in my email in All Mail.  How are you
fetching your mail?

Or are you hoping for something else to be there that isn't there?

-Mark

Re: notmuch Digest, Vol 20, Issue 57

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 764 bytes --]

On Tue, 28 Jun 2011 16:38:30 -0600, Mark Anderson <ma.skies@gmail.com> wrote:
> I had briefly considered adding another output format "file", just to get a
> single file for each message in the db, but the file/files distinction
> feels a bit niggling.  Perhaps it should be changed to "files" and
> "filelists" or something else more clear.

Another option that would be general to several commands would be:

	notmuch search --output=files --exclude-duplicates

Or alternately, --include-duplicates. That might be more useful for
"notmuch show" which is a case where users have previously asked for the
ability to ask for duplicate messages, (and where the duplicates are
squelched by default).

Thoughts?

-Carl

-- 
carl.d.worth@intel.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuch Digest, Vol 20, Issue 57

[Mark Anderson]

On Tue, 28 Jun 2011 16:53:30 -0700, Carl Worth <cworth@cworth.org> wrote:
> On Tue, 28 Jun 2011 16:38:30 -0600, Mark Anderson <ma.skies@gmail.com> wrote:
> > I had briefly considered adding another output format "file", just to get a
> > single file for each message in the db, but the file/files distinction
> > feels a bit niggling.  Perhaps it should be changed to "files" and
> > "filelists" or something else more clear.
> 
> Another option that would be general to several commands would be:
> 
> 	notmuch search --output=files --exclude-duplicates
> 
> Or alternately, --include-duplicates. That might be more useful for
> "notmuch show" which is a case where users have previously asked for the
> ability to ask for duplicate messages, (and where the duplicates are
> squelched by default).
> 
> Thoughts?

I personally prefer --output=files remain as it was, with one file per
mail (even though I submitted the patch to change it).  I suggest that
we could add another format to supply all files (perhaps
--output=allfiles, or --output=dupfiles).  I don't like my original
suggestion of "filelists" because it implies a list of lists to me.  A
list of lists would correlate better to the number of messages which
match the search terms, but doesn't correlate well to xargs input.

I understand that we could use --include-duplicates, but I don't think
there are currently any output specifers that actually have a plurality
for a single message.  If we had something like --output=from, or some
other message attribute, then I think we would achieve more useful
orthogonality from adding an argument similar to --include-duplicates.

As it stands, it looks better to me to have a different --output
specifier to represent the uncommon case of multiple outputs per search
match.

-Mark

Re: notmuch Digest, Vol 20, Issue 57

[Jameson Graef Rollins]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

On Wed, 29 Jun 2011 14:21:11 -0600, Mark Anderson <ma.skies@gmail.com> wrote:
> I personally prefer --output=files remain as it was, with one file per
> mail (even though I submitted the patch to change it).  I suggest that
> we could add another format to supply all files (perhaps
> --output=allfiles, or --output=dupfiles).  I don't like my original
> suggestion of "filelists" because it implies a list of lists to me.  A
> list of lists would correlate better to the number of messages which
> match the search terms, but doesn't correlate well to xargs input.

What's wrong with just outputting all the files matching the search,
including duplicates?  I can't think of any reason where one would want
to not include all files matching the search.  I would be curious to
hear a use case there.

Since I'm on this kick anyway, I'm going to keep pushing against further
customizations where there really isn't a need.

jamie.

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch Digest, Vol 20, Issue 57

[Mark Anderson]

On Wed, 29 Jun 2011 13:54:40 -0700, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> On Wed, 29 Jun 2011 14:21:11 -0600, Mark Anderson <ma.skies@gmail.com> wrote:
> > I personally prefer --output=files remain as it was, with one file per
> > mail (even though I submitted the patch to change it).  I suggest that
> > we could add another format to supply all files (perhaps
> > --output=allfiles, or --output=dupfiles).  I don't like my original
> > suggestion of "filelists" because it implies a list of lists to me.  A
> > list of lists would correlate better to the number of messages which
> > match the search terms, but doesn't correlate well to xargs input.
> 
> What's wrong with just outputting all the files matching the search,
> including duplicates?  I can't think of any reason where one would want
> to not include all files matching the search.  I would be curious to
> hear a use case there.

For someone who is using gmail + offlineimap, labels in gmail become
folders in maildir.

The maildir structure can have a large number of copies of each email
corresponding to the labels/tags which have been applied.

To add a label/tag that is visible to the gmail interface, one should
copy a file representing the message to the folder representing the
gmail label, which will then sync to gmail.

Copying more than one file for each message being labeled is more
wasteful of time and storage.

With all files returned, a simple xargs script to add a label by copying
files will end up with many copies of the same file in the new
directory.

The consuming script could hunt for message-id's in files and uniquify,
but since notmuch was doing that implicitly before, and it's fairly
natural, it seems not a big deal to add.

> Since I'm on this kick anyway, I'm going to keep pushing against further
> customizations where there really isn't a need.

With a common use case for the biggest email userbase which makes
labels/tags natural, I think it is worth considering seriously.

There are certainly other namesets which could be used to reprecent the
two categories.  I'm happy to use names that makes the 'allfiles' output
the common case and the "one file/message" the longer string, but I
think we should provide the "one file/message" output category.

The 'allfiles' case is great for deleting all copies of an email, so I
definitely want it to continue being available.

-Mark

Re: notmuch Digest, Vol 20, Issue 57

[Pieter Praet]

On Wed, 29 Jun 2011 16:19:44 -0600, Mark Anderson <ma.skies@gmail.com> wrote:
> On Wed, 29 Jun 2011 13:54:40 -0700, Jameson Graef Rollins <jrollins@finestructure.net> wrote:
> > On Wed, 29 Jun 2011 14:21:11 -0600, Mark Anderson <ma.skies@gmail.com> wrote:
> > > I personally prefer --output=files remain as it was, with one file per
> > > mail (even though I submitted the patch to change it).  I suggest that
> > > we could add another format to supply all files (perhaps
> > > --output=allfiles, or --output=dupfiles).  I don't like my original
> > > suggestion of "filelists" because it implies a list of lists to me.  A
> > > list of lists would correlate better to the number of messages which
> > > match the search terms, but doesn't correlate well to xargs input.
> > 
> > What's wrong with just outputting all the files matching the search,
> > including duplicates?  I can't think of any reason where one would want
> > to not include all files matching the search.  I would be curious to
> > hear a use case there.
> 
> For someone who is using gmail + offlineimap, labels in gmail become
> folders in maildir.
> 
> The maildir structure can have a large number of copies of each email
> corresponding to the labels/tags which have been applied.
> 
> To add a label/tag that is visible to the gmail interface, one should
> copy a file representing the message to the folder representing the
> gmail label, which will then sync to gmail.
> 
> Copying more than one file for each message being labeled is more
> wasteful of time and storage.

Hardlinks to the rescue!

> With all files returned, a simple xargs script to add a label by copying
> files will end up with many copies of the same file in the new
> directory.

Shove in a "head -n 1" ?

> The consuming script could hunt for message-id's in files and uniquify,
> but since notmuch was doing that implicitly before, and it's fairly
> natural, it seems not a big deal to add.
> 
> > Since I'm on this kick anyway, I'm going to keep pushing against further
> > customizations where there really isn't a need.
> 
> With a common use case for the biggest email userbase which makes
> labels/tags natural, I think it is worth considering seriously.
> 
> There are certainly other namesets which could be used to reprecent the
> two categories.  I'm happy to use names that makes the 'allfiles' output
> the common case and the "one file/message" the longer string, but I
> think we should provide the "one file/message" output category.
> 
> The 'allfiles' case is great for deleting all copies of an email, so I
> definitely want it to continue being available.
> 
> -Mark
> 
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Peace

-- 
Pieter

Re: notmuch Digest, Vol 20, Issue 57

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]

On Tue, 28 Jun 2011 13:22:23 -0700, Carl Worth <cworth@cworth.org> wrote:
> First, it's important to understand that any friction here comes from
> Gmail exposing its tags as folders, (which in turn could be the lack of
> availability of a more tag-aware protocol than imap).

Even risking to become a bit thread-offtopic: IMAP itself supports tags
just fine and should be able to read/set/search all tags just fine (even
any user defined). My feeling is more that this is a lack of tag-using
IMAP clients to expose existing tag functionality. Thunderbird is doing
fine exposing up to 4 user-defined tags that are synced to the server,
but it's still not doing all it can.

I still believe that it would be possible to eg. sync all our notmuch
tags to the IMAP server, which would help enormously with syncing across
machines. I still have the long-term goal of offlineimap being able to sync
notmuch tags. (very long term, though)

As for Gmail and folders, I think it is an ugly kludge leading to all
kinds of awkward behavior (at least when treating Gmail as an IMAP
server). On the other hand it exposes nice tag behavior to clients that
wouldn't support it.

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuch Digest, Vol 20, Issue 57

[Sander Boer]

[-- Attachment #1: Type: text/plain, Size: 1766 bytes --]

Dear Sebastian,

If IMAP supports tags, is that not a big deal ?
I mean, having a converging point for all tags, is that not like the holy
grail in this field ?

Obviously, there must be a caveat, you mentioned client-support, which is
inconvenient, but of no long term consequence.
Do you know what the status is of *server* support ? Because imo this *is* a
big deal, without real standardized server support an IMAP store for tags is
off the table.

best,
Sander


On Thu, Jun 30, 2011 at 9:24 AM, Sebastian Spaeth <Sebastian@sspaeth.de>wrote:

> On Tue, 28 Jun 2011 13:22:23 -0700, Carl Worth <cworth@cworth.org> wrote:
> > First, it's important to understand that any friction here comes from
> > Gmail exposing its tags as folders, (which in turn could be the lack of
> > availability of a more tag-aware protocol than imap).
>
> Even risking to become a bit thread-offtopic: IMAP itself supports tags
> just fine and should be able to read/set/search all tags just fine (even
> any user defined). My feeling is more that this is a lack of tag-using
> IMAP clients to expose existing tag functionality. Thunderbird is doing
> fine exposing up to 4 user-defined tags that are synced to the server,
> but it's still not doing all it can.
>
> I still believe that it would be possible to eg. sync all our notmuch
> tags to the IMAP server, which would help enormously with syncing across
> machines. I still have the long-term goal of offlineimap being able to sync
> notmuch tags. (very long term, though)
>
> As for Gmail and folders, I think it is an ugly kludge leading to all
> kinds of awkward behavior (at least when treating Gmail as an IMAP
> server). On the other hand it exposes nice tag behavior to clients that
> wouldn't support it.
>
> Sebastian
>

[-- Attachment #2: Type: text/html, Size: 2491 bytes --]

Re: notmuch Digest, Vol 20, Issue 57

[Daniel Schoepe]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

On Wed, 6 Jul 2011 20:31:03 +0200, Sander Boer <sanderboer@mauc.nl> wrote:
> If IMAP supports tags, is that not a big deal ?
> I mean, having a converging point for all tags, is that not like the holy
> grail in this field ?

One problem is that maildir doesn't support tags, so we would have to
switch to a format that does or somehow store them in the maildir, in
which case we would also have to adapt offlineimap or a similar tool to
sync tags as well.

> Do you know what the status is of *server* support ? Because imo this *is* a
> big deal, without real standardized server support an IMAP store for tags is
> off the table.

Support for tags is mentioned in the RFC for IMAP, but it's optional. As
far as I know, must servers today support them though.

Cheers, Daniel

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: notmuch Digest, Vol 20, Issue 57

[Sebastian Spaeth]

[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]

On Wed, 06 Jul 2011 20:46:48 +0200, Daniel Schoepe <daniel.schoepe@googlemail.com> wrote:
> One problem is that maildir doesn't support tags, so we would have to
> switch to a format that does or somehow store them in the maildir, in
> which case we would also have to adapt offlineimap or a similar tool to
> sync tags as well.

Right, Maildirs don't store them. AFAIK dovecot stores them in the
user-defined 'use-1-letter-abbrev-for-up-to-20-tags' notation or similar
for it's maildir. But there is no standard, and maildir is not the
answer. Either notmuch would have to directly interact via IMAP protocol
(outch), or some intermediary fetcher would have to store tags in some
type of database (which is what offlineimap could do at some point in
time).
 
> > Do you know what the status is of *server* support ? Because imo this *is* a
> > big deal, without real standardized server support an IMAP store for tags is
> > off the table.
> 
> Support for tags is mentioned in the RFC for IMAP, but it's optional. As
> far as I know, must servers today support them though.

I can't speak for Gmail, but all major servers, ie Cyrus, Dovecot, and
even Exchange seem to handle tags just fine.

Sebastian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: notmuch Digest, Vol 20, Issue 57

[Daniel Schoepe]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

On Mon, 11 Jul 2011 15:22:24 +0200, Sebastian Spaeth <Sebastian@sspaeth.de> wrote:
> > Support for tags is mentioned in the RFC for IMAP, but it's optional. As
> > far as I know, must servers today support them though.
> 
> I can't speak for Gmail, but all major servers, ie Cyrus, Dovecot, and
> even Exchange seem to handle tags just fine.

I just realized that this was a quite misleading typo, that should have
read "As far as I know, _most_ servers today support them though". 

I'd be very suprised if Gmail didn't support them. And even if not, one
could work around that by having a hypothetical tag synchroniser handle
Gmail differently and use their tagging mechanism:

http://code.google.com/apis/gmail/imap/#x-gm-labels

This would also have the added benefit of notmuch tags showing up in the
other gmail-tag-aware applications, like the web-interface.

Another issue brought up by DraX on IRC is that IMAP keywords don't
handle unicode. (Although the RFC specifies that every character except
some excluded special characters are allowed, so I guess it's
implementation-dependent).

I think the best way to go would be to extend both maildir and
offlineimap somehow to store tags and then store them as IMAP keywords /
gmail-labels in the case of Gmail on the server.

Cheers,
Daniel

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]