gmail importer script

gmail importer script

[Jason A. Donenfeld]

Hi folks,

I wrote a script that imports emails and tags from gmail. It's
resumable and appears to be working reasonably well. I could use some
experienced eyes looking at it, as my first exposure to notmuch was
just a few hours ago.

http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py

Thanks,
Jason

Re: gmail importer script

[Kushal Kumaran]

"Jason A. Donenfeld" <Jason@zx2c4.com> writes:

> Hi folks,
>
> I wrote a script that imports emails and tags from gmail. It's
> resumable and appears to be working reasonably well. I could use some
> experienced eyes looking at it, as my first exposure to notmuch was
> just a few hours ago.
>
> http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py
>

Not specific to notmuch, but not every gmail user has a @gmail.com or
@googlemail.com email address.  Google Apps users can have arbitrary
email addresses, and still use gmail's imap and web interface.

I finished syncing my gmail locally (using offlineimap), and creating
tags by hand not one day ago, or I would certainly have tried out this
script.

-- 
regards,
kushal

Re: gmail importer script

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/12/12 05:05, Kushal Kumaran wrote:
> "Jason A. Donenfeld" <Jason@zx2c4.com> writes:
> 
>> Hi folks,
>> 
>> I wrote a script that imports emails and tags from gmail. It's resumable and appears to be
>> working reasonably well. I could use some experienced eyes looking at it, as my first
>> exposure to notmuch was just a few hours ago.
>> 
>> http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py
>> 
> 
> Not specific to notmuch, but not every gmail user has a @gmail.com or @googlemail.com email
> address.  Google Apps users can have arbitrary email addresses, and still use gmail's imap and
> web interface.
> 
> I finished syncing my gmail locally (using offlineimap), and creating tags by hand not one day
> ago, or I would certainly have tried out this script.
> 

Sounds interesting.

I don't know much python, so excuse my questions:

1) are multiple downloads of an email done, if the mail has multiple labels?
2) I am downloading with the patched notmuch (to add x-keywords containing the labels) only the
"All Mail" folder - does your script use this information (if present) to tag the emails?
3) are label changes in gmail synched into the local emails?
4) I assume, local changes are not synced to gmail?
5) how are conflicts handled (local tag changes versus label changes on gmail)?

Thanks,

Rainer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk
ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x
=Sx0B
-----END PGP SIGNATURE-----

Re: gmail importer script

[Patrick Totzke]

Is this supposed to happen?

---------------------------------------------
Logging in...
Selecting all mail...
Discovering local messages...
Receiving message list...
Traceback (most recent call last):
  File "./gmail-notmuch.py", line 175, in <module>
    main()
  File "./gmail-notmuch.py", line 58, in main
    new_messages = discover_new_messages(imap, old_messages)
  File "./gmail-notmuch.py", line 80, in discover_new_messages
    typ, data = imap.fetch("1:*", "X-GM-MSGID")
  File "/usr/lib/python2.7/imaplib.py", line 443, in fetch
    typ, dat = self._simple_command(name, message_set, message_parts)
  File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/usr/lib/python2.7/imaplib.py", line 825, in _command
    ', '.join(Commands[name])))
imaplib.error: command FETCH illegal in state AUTH, only allowed in states SELECTED

------------------------------------------------------


In [2]: imaplib.__version__
Out[2]: '2.58'

One thing you also might want to watch out for is
exceptions due to a locked notmuch index.
rock on,
/p

[PATCH] Minor style changes.

[Michal Nazarewicz]

From: Michal Nazarewicz <mina86@mina86.com>

---
 gmail-notmuch.py |   29 +++++++++--------------------
 1 files changed, 9 insertions(+), 20 deletions(-)

On Fri, Dec 07 2012, Jason A. Donenfeld wrote:
> I wrote a script that imports emails and tags from gmail. It's
> resumable and appears to be working reasonably well. I could use some
> experienced eyes looking at it, as my first exposure to notmuch was
> just a few hours ago.

Didn't really use it, but I still have some suggested code changes: ;)

diff --git a/gmail-notmuch.py b/gmail-notmuch.py
index 04a60dc..ef7fa85 100755
--- a/gmail-notmuch.py
+++ b/gmail-notmuch.py
@@ -43,7 +43,7 @@ def main():
 	(options, args) = parser.parse_args()
 	if options.username == None or options.password == None:
 		parser.error("Username and password are required.")
-	if not options.username.lower().endswith("@gmail.com") and not options.username.lower().endswith("@googlemail.com"):
+	if options.username.find("@") == -1:
 		options.username += "@gmail.com"
 	if len(args) == 0:
 		parser.error("Maildir location is required.")
@@ -60,7 +60,7 @@ def main():
 		print("Nothing to do!")
 		logout(imap)
 		sys.exit(0)
-	
+
 	download_new_messages(imap, new_messages, destination_dir)
 
 	logout(imap)
@@ -84,9 +84,8 @@ def discover_new_messages(imap, old_messages):
 	for response in data:
 		imap_seq = response[0:response.find(" ")]
 		gmail_id = response[response.rfind(" ") + 1:len(response) - 1]
-		if gmail_id in old_messages:
-			continue
-		new_messages.append((gmail_id, imap_seq))
+		if gmail_id not in old_messages:
+			new_messages.append((gmail_id, imap_seq))
 	return new_messages
 
 def download_new_messages(imap, messages, destination):
@@ -112,7 +111,7 @@ def download_new_messages(imap, messages, destination):
 		typ, data = imap.fetch(str(imap_seq), "(FLAGS X-GM-LABELS)")
 		if typ != "OK":
 			sys.exit("Failed to download labels gmail-%d/imap-%d" % (gmail_id, imap_seq))
-		
+
 		labels = label_parser.search(data[0]).groups()
 		labels = filter_labels(shlex.split(labels[0], False, True) + labels[1].split(" "))
 
@@ -149,22 +148,12 @@ def filter_labels(labels):
 			"\\Important":	None, # I realize this is controversial, but I hate the priority inbox.
 			"Junk":		"spam",
 			"NonJunk":	None }
-	ret = set()
-	for label in labels:
-		if label in translation:
-			if translation[label] is None:
-				continue
-			ret.add(translation[label])
-		else:
-			ret.add(label)
-	if "!read!" in ret:
-		ret.remove("!read!")
-	else:
+	ret = set(translation.get(label, label) for label in labels)
+	if not ret.pop("!read!", None):
 		ret.add("unread")
-	if "" in ret:
-		ret.remove("")
+	ret.pop(None, None)
+	ret.pop("", None)
 	return ret
-			
 
 def logout(imap):
 	imap.close()
-- 
1.7.7.3

Re: gmail importer script

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 1289 bytes --]

On Fri, Dec 7, 2012 at 9:53 AM, Rainer M Krug <R.M.Krug@gmail.com> wrote:
>
> Sounds interesting.
>
> I don't know much python, so excuse my questions:
>
> 1) are multiple downloads of an email done, if the mail has multiple
> labels?
>

No.


> 2) I am downloading with the patched notmuch (to add x-keywords containing
> the labels) only the
> "All Mail" folder - does your script use this information (if present) to
> tag the emails?
>

What is the patched notmuch? What does that do?


> 3) are label changes in gmail synched into the local emails?
>

It pulls labels from gmail messages into notmuch, with the labels in tact,
once per each email.


> 4) I assume, local changes are not synced to gmail?
>

Correctamundo.



> 5) how are conflicts handled (local tag changes versus label changes on
> gmail)?
>

They aren't. See 3.


>
> Thanks,
>
> Rainer
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk
> ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x
> =Sx0B
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
>

[-- Attachment #2: Type: text/html, Size: 2613 bytes --]

Re: gmail importer script

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 1607 bytes --]

On Fri, Dec 7, 2012 at 10:23 AM, Patrick Totzke <patricktotzke@gmail.com>wrote:

> Is this supposed to happen?
>

Obviously not.



>
> ---------------------------------------------
> Logging in...
> Selecting all mail...
> Discovering local messages...
> Receiving message list...
> Traceback (most recent call last):
>   File "./gmail-notmuch.py", line 175, in <module>
>     main()
>   File "./gmail-notmuch.py", line 58, in main
>     new_messages = discover_new_messages(imap, old_messages)
>   File "./gmail-notmuch.py", line 80, in discover_new_messages
>     typ, data = imap.fetch("1:*", "X-GM-MSGID")
>   File "/usr/lib/python2.7/imaplib.py", line 443, in fetch
>     typ, dat = self._simple_command(name, message_set, message_parts)
>   File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command
>     return self._command_complete(name, self._command(name, *args))
>   File "/usr/lib/python2.7/imaplib.py", line 825, in _command
>     ', '.join(Commands[name])))
> imaplib.error: command FETCH illegal in state AUTH, only allowed in states
> SELECTED
>
>
Not sure what is causing this. My best guess is that your password was
incorrect and that I'm not checking the login return value.



> ------------------------------------------------------
>
>
> In [2]: imaplib.__version__
> Out[2]: '2.58'
>
> One thing you also might want to watch out for is
> exceptions due to a locked notmuch index.
>

How do I do this?



> rock on,
> /p
> _______________________________________________
> notmuch mailing list
> notmuch@notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
>

[-- Attachment #2: Type: text/html, Size: 2726 bytes --]

Re: gmail importer script

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/12/12 14:48, Jason A. Donenfeld wrote:
> 
> 
> On Fri, Dec 7, 2012 at 9:53 AM, Rainer M Krug <R.M.Krug@gmail.com <mailto:R.M.Krug@gmail.com>>
> wrote:
> 
> Sounds interesting.
> 
> I don't know much python, so excuse my questions:
> 
> 1) are multiple downloads of an email done, if the mail has multiple labels?
> 
> 
> No.

Good.

> 
> 
> 2) I am downloading with the patched notmuch (to add x-keywords containing the labels) only
> the "All Mail" folder - does your script use this information (if present) to tag the emails?
> 
> 
> What is the patched notmuch? What does that do?

Sorry - should have been "patched offlineimap" ( https://github.com/aroig/offlineimap
http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970 and
http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )

It is synching the gmail labels into the x-keywords header as a comma separated list. These could
then be used by notmuch to be added to the labels. It also does sync changes from the X-Keywords
to gmail (see http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970 for details)

> 
> 
> 3) are label changes in gmail synched into the local emails?
> 
> 
> It pulls labels from gmail messages into notmuch, with the labels in tact, once per each
> email.

Good.

> 
> 
> 4) I assume, local changes are not synced to gmail?
> 
> 
> Correctamundo.
> 

OK.

> 
> 
> 5) how are conflicts handled (local tag changes versus label changes on gmail)?
> 
> 
> They aren't. See 3.
> 

Thanks,

Rainer


> 
> 
> Thanks,
> 
> Rainer -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG
> with Mozilla - http://www.enigmail.net/
> 
> iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk 
> ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x =Sx0B -----END PGP SIGNATURE-----
> 
> _______________________________________________ notmuch mailing list notmuch@notmuchmail.org
> <mailto:notmuch@notmuchmail.org> http://notmuchmail.org/mailman/listinfo/notmuch
> 
> 


- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys.
(Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlDB9cMACgkQoYgNqgF2egrEkACeLD9p4CUW3EhliFQ44cigpZdF
o8AAnA9M8tmw52W3m62QFJaFuoOLdto2
=6Zrk
-----END PGP SIGNATURE-----

Re: gmail importer script

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 539 bytes --]

On Fri, Dec 7, 2012 at 2:48 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote:

>
>
>
>
>> 3) are label changes in gmail synched into the local emails?
>>
>
> It pulls labels from gmail messages into notmuch, with the labels in tact,
> once per each email.
>

Okay, it's way more robust now. Just updated it. Now it will pull gmail
labels to notmuch, each time, every time. So you can run this thing over
and over, and it will always push gmail into notmuch. It doesn't yet sync
the the other way yet, but at least this is now fairly robust.

[-- Attachment #2: Type: text/html, Size: 1015 bytes --]

Re: gmail importer script

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 1051 bytes --]

On Fri, Dec 7, 2012 at 2:57 PM, Rainer M Krug <r.m.krug@gmail.com> wrote:
>
> >
> >
> > 2) I am downloading with the patched notmuch (to add x-keywords
> containing the labels) only
> > the "All Mail" folder - does your script use this information (if
> present) to tag the emails?
> >
> >
> > What is the patched notmuch? What does that do?
>
> Sorry - should have been "patched offlineimap" (
> https://github.com/aroig/offlineimap
> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970and
> http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )
>
> It is synching the gmail labels into the x-keywords header as a comma
> separated list. These could
> then be used by notmuch to be added to the labels. It also does sync
> changes from the X-Keywords
> to gmail (see
> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970for details)


I see.

I use the All Mail folder and get the tags using the X-GM-LABELS imap
extension. I don't muck with the mail headers in the process, fortunately.

[-- Attachment #2: Type: text/html, Size: 1815 bytes --]

Re: gmail importer script

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 2819 bytes --]

On 7 December 2012 16:32, Jason A. Donenfeld <Jason@zx2c4.com> wrote:

>
>
> On Fri, Dec 7, 2012 at 2:57 PM, Rainer M Krug <r.m.krug@gmail.com> wrote:
>>
>>  >
>> >
>> > 2) I am downloading with the patched notmuch (to add x-keywords
>> containing the labels) only
>> > the "All Mail" folder - does your script use this information (if
>> present) to tag the emails?
>> >
>> >
>> > What is the patched notmuch? What does that do?
>>
>> Sorry - should have been "patched offlineimap" (
>> https://github.com/aroig/offlineimap
>>
>> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970and
>> http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 )
>>
>> It is synching the gmail labels into the x-keywords header as a comma
>> separated list. These could
>> then be used by notmuch to be added to the labels. It also does sync
>> changes from the X-Keywords
>> to gmail (see
>> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970for details)
>
>
> I see.
>
> I use the All Mail folder and get the tags using the X-GM-LABELS imap
> extension. I don't muck with the mail headers in the process, fortunately.
>

I actually prefer this approach, but I think it would be more useful to
leave the syncing of the emails to a different program, and then just
managing the labels / tags with your tool (which is notmuch territory). So
the workflow would be:

Initial or download (only remote changes):

1) Download email ("All Mail" folder) with whatever you are using
(offlinemail, ...)
2) call your script to index the downloaded emails (All Mail" folder) with
notmuch and get the labels and set / remove them in notmuch accordingly.

Upload (only local changes):

1) upload emails using whatever (offlineimap, ...)
2) upload the changed tags and create the needed new labels (and possibly
delete empty ons?)

The tricky part will be the synching when changes on both sides were
observed.
The mail sync is implemented in e.g. offlineimap and works reilably. But I
don't know how one could handle tag / label changes on both sides.

So, implementing the mail sync in this script would, as I see it, kind of
reinventing the wheel. One can leave it in, but I would like to have an
option to leave the synching of the emails to a separate program.

Cheers,

Rainer



-- 
NEW GERMAN FAX NUMBER!!!

Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology,
UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Natural Sciences Building
Office Suite 2039
Stellenbosch University
Main Campus, Merriman Avenue
Stellenbosch
South Africa

Cell:           +27 - (0)83 9479 042
Fax:            +27 - (0)86 516 2782
Fax:            +49 - (0)321 2125 2244
email:          Rainer@krugs.de

Skype:          RMkrug
Google:         R.M.Krug@gmail.com

[-- Attachment #2: Type: text/html, Size: 4137 bytes --]

Re: gmail importer script

[Jason A. Donenfeld]

> I actually prefer this approach, but I think it would be more useful to
> leave the syncing of the emails to a different program, and then just
> managing the labels / tags with your tool (which is notmuch territory). So
> the workflow would be:
> So, implementing the mail sync in this script would, as I see it, kind of
> reinventing the wheel.

You're misunderstanding me. Let me make it very clear what the script
does. Actually why don't you just read the source code? Please? Well,
anyway, here:

- It looks in gmail and asks it which messages gmail has
- It downloads those messages
- It applies gmail's labels to those messages as tags

The end.

Along the way it has some smart things to reduce redundant downloads.

For more information, consult the source code.

Re: gmail importer script

[Patrick Totzke]

[-- Attachment #1: Type: text/plain, Size: 2544 bytes --]

Quoting Jason A. Donenfeld (2012-12-07 13:49:46)
> Not sure what is causing this. My best guess is that your password was
> incorrect and that I'm not checking the login return value.

Yes, you're right, it was an incorrect passwd.

>     One thing you also might want to watch out for is
>     exceptions due to a locked notmuch index.
> 
> 
> How do I do this?

See e.g. here: https://github.com/pazz/alot/blob/master/alot/db/manager.py#L163
I'm sure "afew" (https://github.com/teythoon/afew) does a similar thing.


I have two new errors:

---------------------------------------------------------
./gmail-notmuch.py -u patricktotzke@gmail.com -p mypwd ~/mail/gmail/
Logging in...
Selecting all mail...
Error opening database at /home/pazz/mail/gmail/.notmuch: No such file or directory
Traceback (most recent call last):
  File "./gmail-notmuch.py", line 225, in <module>
    main()
  File "./gmail-notmuch.py", line 56, in main
    database = notmuch.Database(destination, False, notmuch.Database.MODE.READ_WRITE)
  File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py", line 154, in __init__
    self.open(path, mode)
  File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py", line 214, in open
    raise NotmuchError(status)
notmuch.errors.FileError
---------------------------------------------------------

I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
as this is where my all-mail maildir is, and i also tried an absolute path, all the same result.

If I point it to the root of my notmuch directory I get:
---------------------------------------------------------
[~/projects/gmail-notmuch] ./gmail-notmuch.py -u patricktotzke@gmail.com -p mypwd /home/pazz/mail
Logging in...
Selecting all mail...
Traceback (most recent call last):
  File "./gmail-notmuch.py", line 225, in <module>
    main()
  File "./gmail-notmuch.py", line 58, in main
    messages = discover_messages(imap, total)
  File "./gmail-notmuch.py", line 104, in discover_messages
    new_readline.progressbar = create_progressbar("Receiving message list", total).start()
  File "./gmail-notmuch.py", line 147, in create_progressbar
    return ProgressBar(maxval=total, widgets=[text + ": ", SimpleProgress(), Bar(), Percentage(), " ", ETA(), " ", FileTransferSpeed(unit="emails")])
TypeError: __init__() got an unexpected keyword argument 'unit'

BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get that far.


HTH,
/p

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEABECAAYFAlDDaOcACgkQlDQDZ9fWxap7fACeMTapo+x9flWn8lVKRg5/Ap6m
poUAoLIPjuEIGWge29R3i0XceRNfZSNW
=RGmo
-----END PGP SIGNATURE-----

Re: gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke <patricktotzke@gmail.com> wrote:
> I have two new errors:
>
> ---------------------------------------------------------
> ./gmail-notmuch.py -u patricktotzke@gmail.com -p mypwd ~/mail/gmail/
>
> I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
> as this is where my all-mail maildir is, and i also tried an absolute path, all the same result.
>
> If I point it to the root of my notmuch directory I get:

Pointing it at the root directory is what you want. In fact, I'm
removing the argument from the script and having it just get that from
the config. So the correct invocation will be:

./gmail-notmuch.py -u blabla -p password

>   File "./gmail-notmuch.py", line 147, in create_progressbar
>     return ProgressBar(maxval=total, widgets=[text + ": ", SimpleProgress(), Bar(), Percentage(), " ", ETA(), " ", FileTransferSpeed(unit="emails")])
> TypeError: __init__() got an unexpected keyword argument 'unit'

You're probably running with an old version of python-statusbar.

>
> BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get that far.

Really? Are you in the UK? I'll have to do this dynamically using the
XLIST imap extension.

Re: gmail importer script

[Jason A. Donenfeld]

On Sat, Dec 8, 2012 at 6:04 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>> BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get that far.
>
> Really? Are you in the UK? I'll have to do this dynamically using the
> XLIST imap extension.

Okay, I've updated it to dynamically find the All Mail folder.

Re: gmail importer script

[Patrick Totzke]

[-- Attachment #1: Type: text/plain, Size: 3423 bytes --]

Quoting Jason A. Donenfeld (2012-12-08 17:04:48)
> On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke <patricktotzke@gmail.com> wrote:
> > I have two new errors:
> >
> > ---------------------------------------------------------
> > ./gmail-notmuch.py -u patricktotzke@gmail.com -p mypwd ~/mail/gmail/
> >
> > I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail
> > as this is where my all-mail maildir is, and i also tried an absolute path, all the same result.
> >
> > If I point it to the root of my notmuch directory I get:
> 
> Pointing it at the root directory is what you want. In fact, I'm
> removing the argument from the script and having it just get that from
> the config. So the correct invocation will be:
> 
> ./gmail-notmuch.py -u blabla -p password

Okay..
> 
> >   File "./gmail-notmuch.py", line 147, in create_progressbar
> >     return ProgressBar(maxval=total, widgets=[text + ": ", SimpleProgress(), Bar(), Percentage(), " ", ETA(), " ", FileTransferSpeed(unit="emails")])
> > TypeError: __init__() got an unexpected keyword argument 'unit'
> 
> You're probably running with an old version of python-statusbar.

I just installed it from ubuntu 12.04's repository. the version was 2.2-2.
Installed from pip now, and it seems ok. you'll want to make more specific dependencies at some
point..

> >
> > BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get that far.
> 
> Really? Are you in the UK? I'll have to do this dynamically using the
> XLIST imap extension.

Yea, but its a *really* old account I first got from google.de.
Reading this dynamically would def. help.


next try..
-----------------------------------------------------------------

Traceback (most recent call last):
  File "./gmail-notmuch.py", line 246, in <module>
    main()
  File "./gmail-notmuch.py", line 74, in main
    new_messages = retag_old_messages(database, messages, destination)
  File "./gmail-notmuch.py", line 176, in retag_old_messages
    progressbar.start()
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 311, in start
    self.update(0)
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 283, in update
    self.fd.write(self._format_line() + '\r')
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 243, in _format_line
    widgets = ''.join(self._format_widgets())
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 223, in _format_widgets
    widget = format_updatable(widget, self)
  File "/usr/local/lib/python2.7/dist-packages/progressbar/widgets.py", line 38, in format_updatable
    if hasattr(updatable, 'update'): return updatable.update(pbar)
  File "/usr/local/lib/python2.7/dist-packages/progressbar/widgets.py", line 184, in update
    return '%3d%%' % pbar.percentage()
  File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 208, in percentage
    return self.currval * 100.0 / self.maxval
ZeroDivisionError: float division by zero
--------------------------------------------------------------------------



Another feature request:
Could you make the progressbar stuff optinal? in the end one will want to run this via cronjob
and this shouldn't print anything to stdout.
Also, I can pretty much recommend the 'logging' module.

cheers,
/p

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEABECAAYFAlDDfPIACgkQlDQDZ9fWxaoN0wCfSDu1vUlfji9mYFPRdlzM6LFC
k6kAoKDA7JIWG/JMQQn1S8SmyxTfrnco
=NABV
-----END PGP SIGNATURE-----

Re: gmail importer script

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

On Sat, Dec 8, 2012 at 6:46 PM, Patrick Totzke <patricktotzke@gmail.com>wrote:
>
> ZeroDivisionError: float division by zero
>

Fixed it! Sorry about all this... Jeeze Louise.


>
> Another feature request:
> Could you make the progressbar stuff optinal? in the end one will want to
> run this via cronjob
> and this shouldn't print anything to stdout.
> Also, I can pretty much recommend the 'logging' module.
>

Sure, but... >/dev/null doesn't cut it? I can add some more refined logging
capabilities if you want.



>
> cheers,
> /p
>

[-- Attachment #2: Type: text/html, Size: 1171 bytes --]

Re: gmail importer script

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/12/12 14:15, Jason A. Donenfeld wrote:
>> I actually prefer this approach, but I think it would be more useful to leave the syncing of
>> the emails to a different program, and then just managing the labels / tags with your tool
>> (which is notmuch territory). So the workflow would be: So, implementing the mail sync in
>> this script would, as I see it, kind of reinventing the wheel.
> 
> You're misunderstanding me. Let me make it very clear what the script does. Actually why don't
> you just read the source code? Please? Well,

Because I have never used python, and you know what your script is doing. I prefer to know what
the script is doing, before I try it out and it changes up my local emails and my notmuch
databases in unexpected ways, because I did not understand the source code properly?

> anyway, here:
> 
> - It looks in gmail and asks it which messages gmail has

OK.

> - It downloads those messages

Here is my question - see below.

> - It applies gmail's labels to those messages as tags

Perfect.

Now my final question: after reading your info which is useful, I looked in more detail at the
source code and it made now sense to me. My question is:

If I only want to aply the tags and *not download* emails, can I simply remove the line 88:

  download_new_messages(imap, database, new_messages, destination)

Is this correct? In this case, the script would ony tag y messages which are already downloaded?

If this is the case, I would immediately start using the script.

Rainer

> 
> The end.
> 
> Along the way it has some smart things to reduce redundant downloads.
> 
> For more information, consult the source code.
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlDFp5kACgkQoYgNqgF2egqfVACfWckdCl3SOmue6EEHF1cv6d44
IKgAmwV9cQH/VxCmPHmmU6z3akQ9pFZV
=vZxS
-----END PGP SIGNATURE-----

Re: gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-09 23:13:12)
> On Sat, Dec 8, 2012 at 6:46 PM, Patrick Totzke <patricktotzke@gmail.com> wrote:
> 
>     ZeroDivisionError: float division by zero
> 
> 
> Fixed it! Sorry about all this... Jeeze Louise. 

thx

> 
>     Another feature request:
>     Could you make the progressbar stuff optinal? in the end one will want to
>     run this via cronjob
>     and this shouldn't print anything to stdout.
>     Also, I can pretty much recommend the 'logging' module.
> 
> 
> Sure, but... >/dev/null doesn't cut it? I can add some more refined logging
> capabilities if you want.
It would, but its nicer not to load ressources you're not gonna use. Also, progressbar 2.2 seems to be the newest version you see on a reasonably new debian/ubuntu install.. </nitpick>


Next exception.. do you have a bugtracker? (/me <3 github)
----------------------------------------------------
Logging in...
Selecting all mail...
Receiving message list: 24455 of 24455|########################################|100% Time: 0:00:12   2.04 kemails/s
Parsing message list and labels...
Searching for local messages...
Traceback (most recent call last):
  File "./gmail-notmuch.py", line 248, in <module>
    main()
  File "./gmail-notmuch.py", line 75, in main
    if len(new_messages) == 0:
TypeError: object of type 'NoneType' has no len()
----------------------------------------------------


/p

Re: gmail importer script

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke <patricktotzke@gmail.com>wrote:
>
> It would, but its nicer not to load ressources you're not gonna use. Also,
> progressbar 2.2 seems to be the newest version you see on a reasonably new
> debian/ubuntu install.. </nitpick>
>

I'll consider it.


>
>
> Next exception.. do you have a bugtracker? (/me <3 github)
>

To hell with github. Sometimes I do use a bugtracker though. In light of me
being an idiot, maybe I should get one for this script. But this script is
so short, and these errors are really due to me being an idiot. Sit tight a
little longer and things will begin to work. Sorry for the prior impartial
fixes. Once this baby is running you will be quite delighted.


> TypeError: object of type 'NoneType' has no len()


Fixed.
http://git.zx2c4.com/gmail-notmuch/commit/?id=3bf8ce9698ec798a42bc2f92f596243fa8911bf8

[-- Attachment #2: Type: text/html, Size: 1592 bytes --]

Re: gmail importer script

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

On Mon, Dec 10, 2012 at 6:46 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote:

>
>
> On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke <patricktotzke@gmail.com>wrote:
>>
>> It would, but its nicer not to load ressources you're not gonna use.
>> Also, progressbar 2.2 seems to be the newest version you see on a
>> reasonably new debian/ubuntu install.. </nitpick>
>>
>
> I'll consider it.
>

--silent or -s now silences stdout.

[-- Attachment #2: Type: text/html, Size: 996 bytes --]

Re: gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-11 07:06:22)
> On Mon, Dec 10, 2012 at 6:46 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> 
> 
> 
>     On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke <patricktotzke@gmail.com>
>     wrote:
> 
>         It would, but its nicer not to load ressources you're not gonna use.
>         Also, progressbar 2.2 seems to be the newest version you see on a
>         reasonably new debian/ubuntu install.. </nitpick>
> 
> 
>     I'll consider it.
> 
> 
> --silent or -s now silences stdout.

It seems to work, but i'm still waiting for the first run-through.

Downloading messages: 457 of 22831|

This takes ages. I hope it doesn't try to re-download all my messages everytime.

Another thing:
It seems that throughout its run, your script locks the notmuch database. This is particularly
annoying if you want to read/tag mails while waiting for a long sync-process. I think you only
really want to lock the index for a short time in the end.
Do you write to the index directly after a msg is downloaded?

/p

Re: gmail importer script

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]

It seems to work, but i'm still waiting for the first run-through.

>
> Downloading messages: 457 of 22831|
>
> This takes ages. I hope it doesn't try to re-download all my messages
> everytime.
>

Nope. It's very smart about not doing that.


>
> Another thing:
> It seems that throughout its run, your script locks the notmuch database.
> This is particularly
> annoying if you want to read/tag mails while waiting for a long
> sync-process. I think you only
> really want to lock the index for a short time in the end.
> Do you write to the index directly after a msg is downloaded?
>

I tried to solve this by closing and opening the database each time, but it
actually reduced the performance considerably -- even slower than the gmail
download.

[-- Attachment #2: Type: text/html, Size: 1125 bytes --]

Re: gmail importer script

[Patrick Totzke]

Quoting Jason A. Donenfeld (2012-12-11 09:12:56)
> It seems to work, but i'm still waiting for the first run-through.
> 
> 
>     Downloading messages: 457 of 22831|
> 
>     This takes ages. I hope it doesn't try to re-download all my messages
>     everytime.
> 
> 
> Nope. It's very smart about not doing that.
>  

hmmk

I ended up killing the process after i saw that ETA was >3h. This broke my index!
afterwards, a `notmuch new` printed 

A Xapian exception occurred creating a directory: Expected block
485 to be level 1, not 0.

I wasnt able to properly read the index anymore and had to
restore an old index dump. not cool!

/p

Re: gmail importer script

[Jason A. Donenfeld]

Please don't reply offlist. This is message #2 that's been sent offlist.

On Thu, Dec 13, 2012 at 3:22 PM, fREW Schmidt <frioux@gmail.com>
> One thing you might want to try is to only open the notmuch index for
> writing at a specified sync time.

How do I do this? What's the python function? I couldn't find
anything. Would be interested in doing this though if it exists.

> Also I was going to say you might want to catch SIGINT like

The library already does this.

Re: gmail importer script

[fREW Schmidt]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

On Thu, Dec 13, 2012 at 03:30:19PM +0100, Jason A. Donenfeld wrote:
> Please don't reply offlist. This is message #2 that's been sent offlist.

Oh I'm sorry, that was an accident.

> On Thu, Dec 13, 2012 at 3:22 PM, fREW Schmidt <frioux@gmail.com>
> > One thing you might want to try is to only open the notmuch index for
> > writing at a specified sync time.
>
> How do I do this? What's the python function? I couldn't find
> anything. Would be interested in doing this though if it exists.

I just mean a general programming technique.  As you get information
about tags you hold them in memory and then sync it later and clear
your memory.

> > Also I was going to say you might want to catch SIGINT like
>
> The library already does this.

--
fREW Schmidt
http://blog.afoolishmanifesto.com

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: gmail importer script

[Jason A. Donenfeld]

On Thu, Dec 13, 2012 at 3:35 PM, fREW Schmidt <frioux@gmail.com> wrote:
> I just mean a general programming technique.  As you get information
> about tags you hold them in memory and then sync it later and clear
> your memory.

Oh, yea, that's a great idea. I'll play with that. Good thinking.

Re: gmail importer script

[fREW Schmidt]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

On Thu, Dec 13, 2012 at 03:28:30PM +0100, Jason A. Donenfeld wrote:
> On Thu, Dec 13, 2012 at 3:19 PM, fREW Schmidt <frioux@gmail.com> wrote:
> >> Okay, I've updated it to dynamically find the All Mail folder.
> >
> > I can't seem to find it now, but I read that Gmail actually localizes
> > the name of various folders, so I think All Mail will actually be
> > called something else for non-english speakers.  It might be a good
> > idea for you to at least make it a config option.
>
> A) Why are you replying off list?
>
> B) Did you totally miss my message directly above yours where I wrote
> "Okay, I've updated it to dynamically find the All Mail folder"?

Right, I did see that and I did look at the code; all I'm saying is
that Gmail may localize AllMail as I'm fairly sure they localize the
other builtin labels
(http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py#n214)

I could certainly be wrong though, I haven't looked at this at such a
low level before.

--
fREW Schmidt
http://blog.afoolishmanifesto.com

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: gmail importer script

[Jason A. Donenfeld]

\AllMail is the unlocalized generic version that maps to the localized
version, as are all the other back slash situations.

Re: gmail importer script

[Patrick Totzke]

Sorry, I'm misusing this thread as a bugtracker..

------------------------------------------------
Traceback (most recent call last):4|####                                       | 10% ETA:  1:24:41   3.54  emails/s
  File "./gmail-notmuch.py", line 251, in <module>
    main()
  File "./gmail-notmuch.py", line 89, in main
    download_new_messages(imap, database, new_messages, destination)
  File "./gmail-notmuch.py", line 208, in download_new_messages
    tag_message(database, dest, labels)
  File "./gmail-notmuch.py", line 168, in tag_message
    raise e
notmuch.errors.FileNotEmailError
------------------------------------------------


best,
/p

Re: gmail importer script

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 162 bytes --]

On Thu, Dec 13, 2012 at 5:48 PM, Patrick Totzke <patricktotzke@gmail.com>wrote:

>
> notmuch.errors.FileNotEmailError


Delete the file it dies on and try again.

[-- Attachment #2: Type: text/html, Size: 438 bytes --]

Re: gmail importer script

[Patrick Totzke]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

Quoting Jason A. Donenfeld (2012-12-15 08:22:46)
> On Thu, Dec 13, 2012 at 5:48 PM, Patrick Totzke <patricktotzke@gmail.com>
> wrote:
> 
> 
>     notmuch.errors.FileNotEmailError
> 
> 
> Delete the file it dies on and try again.

Well, thats not the point.. the script shouldn't die like this.
I think it's be better if the script caught that exception, deleted the file
and continued..

Thanks,
/p

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEABECAAYFAlDMU/YACgkQlDQDZ9fWxaqk5gCdEcOVSdgOXqsam7objXLMyhPm
kdwAoI92eZD/YAMSMrwn7ejCmmF/db5B
=llhR
-----END PGP SIGNATURE-----

Re: gmail importer script

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 591 bytes --]

On Sat, Dec 15, 2012 at 11:41 AM, Patrick Totzke <patricktotzke@gmail.com>wrote:
>
> Well, thats not the point.. the script shouldn't die like this.
> I think it's be better if the script caught that exception, deleted the
> file
> and continued..


Probably, but I suspect it's related to whatever mystery error you ran into
before with the corruption.

Does deleting that file and trying again fix it?


Anyway, this is extremely stable for me and a few others at this point. I'm
going to wait for other users to report errors. Alternatively, send me
patches if you want things to happen.

[-- Attachment #2: Type: text/html, Size: 945 bytes --]

Re: gmail importer script

[Patrick Totzke]

[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]

Quoting Jason A. Donenfeld (2012-12-16 20:44:04)
> On Sat, Dec 15, 2012 at 11:41 AM, Patrick Totzke <patricktotzke@gmail.com>
> wrote:
> 
>     Well, thats not the point.. the script shouldn't die like this.
>     I think it's be better if the script caught that exception, deleted the
>     file
>     and continued..
> 
> 
> Probably, but I suspect it's related to whatever mystery error you ran into
> before with the corruption.
> 
> Does deleting that file and trying again fix it? 

I don't know. How would I know which file to remove?
I just removed the newest files in new/cur/tmp
but this doesnt solve the issue.
What would really help here is some more informative error message!

I tried with the `-d` option but did not see any local path in the output.

> 
> Anyway, this is extremely stable for me and a few others at this point. I'm
> going to wait for other users to report errors. Alternatively, send me patches
> if you want things to happen.

Personally, I don't really care if 'things are happening', as I'm kind of OK with my current
offlineimap configuration and would seriously consider switching from OI
only if it meant an improvement in speed or stability.

I test this code because I believe that the notmuch ecosystem
would benefit from a working solution and that reviews and bug-reports
are important.

Of course, I don't expect any miracles in terms of speed at this early stage.
But the bug I reported above should be addressed at some point in order not to
scare off potential users.

Have you considered including your code in offlineimap?
I'm asking because OI already solved some of the problems you might face later on
if you want to continue working on this.

* multi-threaded downloads
* the ability to read passwords not as plaintext parameter..

best,
/p

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEABECAAYFAlDOS7oACgkQlDQDZ9fWxarMsACfX/EVgFbyEty3JtK7BDji6oRG
sOQAn0aUMlcf0FIfvAAC0K7PoEJ6yhf6
=Jhvq
-----END PGP SIGNATURE-----

Re: gmail importer script

[Patrick Totzke]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

BTW: I removed the whole maildir contents and restarted the downlowd over night. here's the result:

---------------------------------------------------------------
Traceback (most recent call last):2|####################                                                          | 26% ETA:  1:37:22   3.11  emails/s
  File "./gmail-notmuch.py", line 253, in <module>
    main()
  File "./gmail-notmuch.py", line 89, in main
    download_new_messages(imap, database, new_messages, destination)
  File "./gmail-notmuch.py", line 210, in download_new_messages
    tag_message(database, dest, labels)
  File "./gmail-notmuch.py", line 168, in tag_message
    raise e
notmuch.errors.FileNotEmailError
-----------------------------------------------------------------------

/p

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEABECAAYFAlDO2HYACgkQlDQDZ9fWxaprkQCfat0Rr5KrdpoafQzedwp1+uck
1IsAn3Q7DLNAfMkdh6n4spwL1+K84tk0
=0M77
-----END PGP SIGNATURE-----