[gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

[gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

[Gaute Hope]

[-- Attachment #1: Type: text/plain, Size: 849 bytes --]

Hi,

  'gmailieer' (or 'gmi') is a small program that can pull email and labels
(and changes to labels) from your GMail account and store them locally in a
maildir with the labels synchronized with a notmuch database. The
changes to tags in the notmuch database may be pushed back remotely to
your GMail account.

The initial fetch of all emails takes some time, but synchronizing
labels and tags, and checking for new messages, is usually done in 1-2
seconds.

It requires the most recent notmuch, the python googleapi bindings and
tqdm.

Disclaimer:

  This is still experimental, but it does not have access to delete
  e-mail on your account - only fetch and change labels, so damage
  should be limited.



Instructions and source code can be found here:

  https://github.com/gauteh/gmailieer




Regards, Gaute

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

[David Bremner]

Gaute Hope <eg@gaute.vetsj.com> writes:

> Hi,
>
>   'gmailieer' (or 'gmi') is a small program that can pull email and labels
> (and changes to labels) from your GMail account and store them locally in a
> maildir with the labels synchronized with a notmuch database. The
> changes to tags in the notmuch database may be pushed back remotely to
> your GMail account.

Hi Gaute;

Thanks for writing this. It seems like the right approach for people who
want to sync gmail labels with notmuch.

d

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

[Rafael Avila de Espindola]

That looks really interesting. What is the license? :-)

Cheers,
Rafael

Gaute Hope <eg@gaute.vetsj.com> writes:

> Hi,
>
>   'gmailieer' (or 'gmi') is a small program that can pull email and labels
> (and changes to labels) from your GMail account and store them locally in a
> maildir with the labels synchronized with a notmuch database. The
> changes to tags in the notmuch database may be pushed back remotely to
> your GMail account.
>
> The initial fetch of all emails takes some time, but synchronizing
> labels and tags, and checking for new messages, is usually done in 1-2
> seconds.
>
> It requires the most recent notmuch, the python googleapi bindings and
> tqdm.
>
> Disclaimer:
>
>   This is still experimental, but it does not have access to delete
>   e-mail on your account - only fetch and change labels, so damage
>   should be limited.
>
>
>
> Instructions and source code can be found here:
>
>   https://github.com/gauteh/gmailieer
>
>
>
>
> Regards, Gaute
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

[Gaute Hope]

Rafael Avila de Espindola writes on mars 14, 2017 16:51:
> 
> That looks really interesting. What is the license? :-)

Ay, GPL v3+ it is!

- gaute

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

[Rafael Avila de Espindola]

I currently have a mbsync + notmuch setup and I am just about to give
this a try.

One question is what the notmuch setup should look like. I assume I
should create a new config file pointing to the directory where the
emails will be written to. Do I have to run notmuch setup to create the
db?

Cheers,
Rafael

Gaute Hope <eg@gaute.vetsj.com> writes:

> Rafael Avila de Espindola writes on mars 14, 2017 16:51:
>> 
>> That looks really interesting. What is the license? :-)
>
> Ay, GPL v3+ it is!
>
> - gaute

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

[Gaute Hope]

Rafael Avila de Espindola writes on mars 27, 2017 4:32:
> I currently have a mbsync + notmuch setup and I am just about to give
> this a try.
>
> One question is what the notmuch setup should look like. I assume I
> should create a new config file pointing to the directory where the
> emails will be written to. Do I have to run notmuch setup to create the
> db?
>

Yes, it is a good idea to just back up your entire notmuch db and set up
a new one to test gmailieer with.

gmailieer respects $NOTMUCH_CONFIG and reads it to figure out which
'new' tags to add (no need to use inbox or unread here, that will only
create confusion - I only use 'new' to do tagging).

Assuming you want to set up a separate notmuch database to test you do
the following:

first:
    get a client secret API key for gmailieer [0], you only need this
  during setup or if gmailieer needs to be re-authenticated. so store it
  somewhere safe, say: ~/.secret/client_secret.json. ideally this should
  be unnecessary [1].

# setting up a separate notmuch db
$ mkdir ~/test_mail/  # our new mail root
$ cd ~/test_mail
$ export NOTMUCH_CONFIG=$(pwd)/.notmuch-config
$ mkdir mail          # new mail database root
$ notmuch setup
  * specify ~/test_mail/mail as mail database
  * specify only 'new' as new tag
$ notmuch new         # create db
$ cd mail

NOTE: As long as you are using a non-standard notmuch config location make
sure that environment variable is set when using gmailieer.

# setting up gmailieer
$ mkdir gmail
  * this will be the gmailieer local repository.
  * it should be a sub-dir of the root mail db.
  * you can keep other mail (from other accounts) outside this
    gmailieer local repository.
$ cd gmail
$ gmi auth -c ~/.secret/client_secret.json
  * authenticate gmailieer, choose the correct account, say:
    charlie.root@gmail.com
$ gmi init -a charlie.root@gmail.com # gmailieer should now be set up
$ gmi pull # does the initial pull, fetching all you messages - this
           # could take some time.

When you want to synchronize your account you do:
$ gmi sync    # which effectively does gmi push; gmi pull

this will also update the notmuch database so it should not be necessary
to run 'notmuch new' afterwards.

Regards, Gaute

[0] https://console.developers.google.com/flows/enableapi?apiid=gmail
[1] https://github.com/gauteh/gmailieer/issues/1

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

[Rafael Espíndola]

> Assuming you want to set up a separate notmuch database to test you do
> the following:
>
> first:
>    get a client secret API key for gmailieer [0], you only need this
>  during setup or if gmailieer needs to be re-authenticated. so store it
>  somewhere safe, say: ~/.secret/client_secret.json. ideally this should
>  be unnecessary [1].
>
> # setting up a separate notmuch db
> $ mkdir ~/test_mail/  # our new mail root
> $ cd ~/test_mail
> $ export NOTMUCH_CONFIG=$(pwd)/.notmuch-config
> $ mkdir mail          # new mail database root
> $ notmuch setup
>  * specify ~/test_mail/mail as mail database
>  * specify only 'new' as new tag
> $ notmuch new         # create db
> $ cd mail
>
> NOTE: As long as you are using a non-standard notmuch config location make
> sure that environment variable is set when using gmailieer.
>
> # setting up gmailieer
> $ mkdir gmail
>  * this will be the gmailieer local repository.
>  * it should be a sub-dir of the root mail db.
>  * you can keep other mail (from other accounts) outside this
>    gmailieer local repository.
> $ cd gmail
> $ gmi auth -c ~/.secret/client_secret.json
>  * authenticate gmailieer, choose the correct account, say:
>    charlie.root@gmail.com
> $ gmi init -a charlie.root@gmail.com # gmailieer should now be set up
> $ gmi pull # does the initial pull, fetching all you messages - this
>           # could take some time.

It is currently running. Thank you so much.

The only things I noticed so far:

* running auth before init fails, but running init -c ... -a ... works
* I had to name the auth file client_secret.json or "gmi pull" would
fail trying to read it.

Thanks again for writing this. It seems a much better way to sync with
gmail. My previous setup was to use mbsync. It is really fast for
fetching new emails (--pull-new), but with imap the only way to get
flag changes (including read/unread) seems to be to ask for the flag
of every message!

Cheers,
Rafael

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

[Gaute Hope]

Rafael Espíndola writes on mars 27, 2017 15:23:
>> $ gmi pull # does the initial pull, fetching all you messages - this
>>           # could take some time.
> 
> It is currently running. Thank you so much.
> 
> The only things I noticed so far:
> 
> * running auth before init fails, but running init -c ... -a ... works
> * I had to name the auth file client_secret.json or "gmi pull" would
> fail trying to read it.


Right, that should be the other way around. I'll update the README.

There are two secret files. One is the API client_secret which can be
specified with the '-c' option (also for pull), this one is only needed
initially and can otherwise be stored somewhere else:

When authenticating (either running 'auth' or some operation requiring
authentication: pull/push) a local access token is retrieved, this is
stored in: .credentials.gmailieer.json in the local mail repository.

> gmail. My previous setup was to use mbsync. It is really fast for
> fetching new emails (--pull-new), but with imap the only way to get
> flag changes (including read/unread) seems to be to ask for the flag
> of every message!

Thanks!

- gaute

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

[Rafael Avila de Espindola]

After a few issues with the initial sync this is working perfectly.

Thanks a lot, it is a big improvement over mbsync.

Cheers,
Rafael

Gaute Hope <eg@gaute.vetsj.com> writes:

> Hi,
>
>   'gmailieer' (or 'gmi') is a small program that can pull email and labels
> (and changes to labels) from your GMail account and store them locally in a
> maildir with the labels synchronized with a notmuch database. The
> changes to tags in the notmuch database may be pushed back remotely to
> your GMail account.
>
> The initial fetch of all emails takes some time, but synchronizing
> labels and tags, and checking for new messages, is usually done in 1-2
> seconds.
>
> It requires the most recent notmuch, the python googleapi bindings and
> tqdm.
>
> Disclaimer:
>
>   This is still experimental, but it does not have access to delete
>   e-mail on your account - only fetch and change labels, so damage
>   should be limited.
>
>
>
> Instructions and source code can be found here:
>
>   https://github.com/gauteh/gmailieer
>
>
>
>
> Regards, Gaute
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

[Gaute Hope]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

Rafael Avila de Espindola writes on april 3, 2017 16:35:
> After a few issues with the initial sync this is working perfectly.
>
> Thanks a lot, it is a big improvement over mbsync.
>

Great, thanks for the patches! This made inital sync much more robust.

I have a non-user-api-key version working [0], but I need to figure out if
it is safe first [1].

- Gaute

[0] https://github.com/gauteh/gmailieer/pull/9
[1] http://stackoverflow.com/questions/43173367/is-it-safe-to-distribute-client-id-and-client-secret-for-google-api-for-an-insta

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]