* gmail importer script @ 2012-12-07 0:24 Jason A. Donenfeld 2012-12-07 4:05 ` Kushal Kumaran 2012-12-07 11:48 ` [PATCH] Minor style changes Michal Nazarewicz 0 siblings, 2 replies; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-07 0:24 UTC (permalink / raw) To: notmuch Hi folks, I wrote a script that imports emails and tags from gmail. It's resumable and appears to be working reasonably well. I could use some experienced eyes looking at it, as my first exposure to notmuch was just a few hours ago. http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py Thanks, Jason ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-07 0:24 gmail importer script Jason A. Donenfeld @ 2012-12-07 4:05 ` Kushal Kumaran 2012-12-07 8:53 ` Rainer M Krug 2012-12-07 11:48 ` [PATCH] Minor style changes Michal Nazarewicz 1 sibling, 1 reply; 35+ messages in thread From: Kushal Kumaran @ 2012-12-07 4:05 UTC (permalink / raw) To: notmuch "Jason A. Donenfeld" <Jason@zx2c4.com> writes: > Hi folks, > > I wrote a script that imports emails and tags from gmail. It's > resumable and appears to be working reasonably well. I could use some > experienced eyes looking at it, as my first exposure to notmuch was > just a few hours ago. > > http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py > Not specific to notmuch, but not every gmail user has a @gmail.com or @googlemail.com email address. Google Apps users can have arbitrary email addresses, and still use gmail's imap and web interface. I finished syncing my gmail locally (using offlineimap), and creating tags by hand not one day ago, or I would certainly have tried out this script. -- regards, kushal ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-07 4:05 ` Kushal Kumaran @ 2012-12-07 8:53 ` Rainer M Krug 2012-12-07 9:23 ` Patrick Totzke 2012-12-07 13:48 ` Jason A. Donenfeld 0 siblings, 2 replies; 35+ messages in thread From: Rainer M Krug @ 2012-12-07 8:53 UTC (permalink / raw) To: notmuch; +Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/12/12 05:05, Kushal Kumaran wrote: > "Jason A. Donenfeld" <Jason@zx2c4.com> writes: > >> Hi folks, >> >> I wrote a script that imports emails and tags from gmail. It's resumable and appears to be >> working reasonably well. I could use some experienced eyes looking at it, as my first >> exposure to notmuch was just a few hours ago. >> >> http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py >> > > Not specific to notmuch, but not every gmail user has a @gmail.com or @googlemail.com email > address. Google Apps users can have arbitrary email addresses, and still use gmail's imap and > web interface. > > I finished syncing my gmail locally (using offlineimap), and creating tags by hand not one day > ago, or I would certainly have tried out this script. > Sounds interesting. I don't know much python, so excuse my questions: 1) are multiple downloads of an email done, if the mail has multiple labels? 2) I am downloading with the patched notmuch (to add x-keywords containing the labels) only the "All Mail" folder - does your script use this information (if present) to tag the emails? 3) are label changes in gmail synched into the local emails? 4) I assume, local changes are not synced to gmail? 5) how are conflicts handled (local tag changes versus label changes on gmail)? Thanks, Rainer -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x =Sx0B -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-07 8:53 ` Rainer M Krug @ 2012-12-07 9:23 ` Patrick Totzke 2012-12-07 13:49 ` Jason A. Donenfeld 2012-12-07 13:48 ` Jason A. Donenfeld 1 sibling, 1 reply; 35+ messages in thread From: Patrick Totzke @ 2012-12-07 9:23 UTC (permalink / raw) To: Rainer M Krug, notmuch; +Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g Is this supposed to happen? --------------------------------------------- Logging in... Selecting all mail... Discovering local messages... Receiving message list... Traceback (most recent call last): File "./gmail-notmuch.py", line 175, in <module> main() File "./gmail-notmuch.py", line 58, in main new_messages = discover_new_messages(imap, old_messages) File "./gmail-notmuch.py", line 80, in discover_new_messages typ, data = imap.fetch("1:*", "X-GM-MSGID") File "/usr/lib/python2.7/imaplib.py", line 443, in fetch typ, dat = self._simple_command(name, message_set, message_parts) File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command return self._command_complete(name, self._command(name, *args)) File "/usr/lib/python2.7/imaplib.py", line 825, in _command ', '.join(Commands[name]))) imaplib.error: command FETCH illegal in state AUTH, only allowed in states SELECTED ------------------------------------------------------ In [2]: imaplib.__version__ Out[2]: '2.58' One thing you also might want to watch out for is exceptions due to a locked notmuch index. rock on, /p ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-07 9:23 ` Patrick Totzke @ 2012-12-07 13:49 ` Jason A. Donenfeld 2012-12-08 16:20 ` Patrick Totzke 0 siblings, 1 reply; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-07 13:49 UTC (permalink / raw) To: Patrick Totzke Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug [-- Attachment #1: Type: text/plain, Size: 1607 bytes --] On Fri, Dec 7, 2012 at 10:23 AM, Patrick Totzke <patricktotzke@gmail.com>wrote: > Is this supposed to happen? > Obviously not. > > --------------------------------------------- > Logging in... > Selecting all mail... > Discovering local messages... > Receiving message list... > Traceback (most recent call last): > File "./gmail-notmuch.py", line 175, in <module> > main() > File "./gmail-notmuch.py", line 58, in main > new_messages = discover_new_messages(imap, old_messages) > File "./gmail-notmuch.py", line 80, in discover_new_messages > typ, data = imap.fetch("1:*", "X-GM-MSGID") > File "/usr/lib/python2.7/imaplib.py", line 443, in fetch > typ, dat = self._simple_command(name, message_set, message_parts) > File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command > return self._command_complete(name, self._command(name, *args)) > File "/usr/lib/python2.7/imaplib.py", line 825, in _command > ', '.join(Commands[name]))) > imaplib.error: command FETCH illegal in state AUTH, only allowed in states > SELECTED > > Not sure what is causing this. My best guess is that your password was incorrect and that I'm not checking the login return value. > ------------------------------------------------------ > > > In [2]: imaplib.__version__ > Out[2]: '2.58' > > One thing you also might want to watch out for is > exceptions due to a locked notmuch index. > How do I do this? > rock on, > /p > _______________________________________________ > notmuch mailing list > notmuch@notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch > [-- Attachment #2: Type: text/html, Size: 2726 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-07 13:49 ` Jason A. Donenfeld @ 2012-12-08 16:20 ` Patrick Totzke 2012-12-08 17:04 ` Jason A. Donenfeld 0 siblings, 1 reply; 35+ messages in thread From: Patrick Totzke @ 2012-12-08 16:20 UTC (permalink / raw) To: Jason A. Donenfeld Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug [-- Attachment #1: Type: text/plain, Size: 2544 bytes --] Quoting Jason A. Donenfeld (2012-12-07 13:49:46) > Not sure what is causing this. My best guess is that your password was > incorrect and that I'm not checking the login return value. Yes, you're right, it was an incorrect passwd. > One thing you also might want to watch out for is > exceptions due to a locked notmuch index. > > > How do I do this? See e.g. here: https://github.com/pazz/alot/blob/master/alot/db/manager.py#L163 I'm sure "afew" (https://github.com/teythoon/afew) does a similar thing. I have two new errors: --------------------------------------------------------- ./gmail-notmuch.py -u patricktotzke@gmail.com -p mypwd ~/mail/gmail/ Logging in... Selecting all mail... Error opening database at /home/pazz/mail/gmail/.notmuch: No such file or directory Traceback (most recent call last): File "./gmail-notmuch.py", line 225, in <module> main() File "./gmail-notmuch.py", line 56, in main database = notmuch.Database(destination, False, notmuch.Database.MODE.READ_WRITE) File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py", line 154, in __init__ self.open(path, mode) File "/home/pazz/.local/lib/python2.7/site-packages/notmuch/database.py", line 214, in open raise NotmuchError(status) notmuch.errors.FileError --------------------------------------------------------- I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail as this is where my all-mail maildir is, and i also tried an absolute path, all the same result. If I point it to the root of my notmuch directory I get: --------------------------------------------------------- [~/projects/gmail-notmuch] ./gmail-notmuch.py -u patricktotzke@gmail.com -p mypwd /home/pazz/mail Logging in... Selecting all mail... Traceback (most recent call last): File "./gmail-notmuch.py", line 225, in <module> main() File "./gmail-notmuch.py", line 58, in main messages = discover_messages(imap, total) File "./gmail-notmuch.py", line 104, in discover_messages new_readline.progressbar = create_progressbar("Receiving message list", total).start() File "./gmail-notmuch.py", line 147, in create_progressbar return ProgressBar(maxval=total, widgets=[text + ": ", SimpleProgress(), Bar(), Percentage(), " ", ETA(), " ", FileTransferSpeed(unit="emails")]) TypeError: __init__() got an unexpected keyword argument 'unit' BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get that far. HTH, /p [-- Attachment #2: signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEABECAAYFAlDDaOcACgkQlDQDZ9fWxap7fACeMTapo+x9flWn8lVKRg5/Ap6m poUAoLIPjuEIGWge29R3i0XceRNfZSNW =RGmo -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-08 16:20 ` Patrick Totzke @ 2012-12-08 17:04 ` Jason A. Donenfeld 2012-12-08 17:19 ` Jason A. Donenfeld 2012-12-08 17:46 ` Patrick Totzke 0 siblings, 2 replies; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-08 17:04 UTC (permalink / raw) To: Patrick Totzke Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke <patricktotzke@gmail.com> wrote: > I have two new errors: > > --------------------------------------------------------- > ./gmail-notmuch.py -u patricktotzke@gmail.com -p mypwd ~/mail/gmail/ > > I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail > as this is where my all-mail maildir is, and i also tried an absolute path, all the same result. > > If I point it to the root of my notmuch directory I get: Pointing it at the root directory is what you want. In fact, I'm removing the argument from the script and having it just get that from the config. So the correct invocation will be: ./gmail-notmuch.py -u blabla -p password > File "./gmail-notmuch.py", line 147, in create_progressbar > return ProgressBar(maxval=total, widgets=[text + ": ", SimpleProgress(), Bar(), Percentage(), " ", ETA(), " ", FileTransferSpeed(unit="emails")]) > TypeError: __init__() got an unexpected keyword argument 'unit' You're probably running with an old version of python-statusbar. > > BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get that far. Really? Are you in the UK? I'll have to do this dynamically using the XLIST imap extension. ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-08 17:04 ` Jason A. Donenfeld @ 2012-12-08 17:19 ` Jason A. Donenfeld [not found] ` <20121213141917.GD3658@tungsten.lan.mitsi.com> 2012-12-08 17:46 ` Patrick Totzke 1 sibling, 1 reply; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-08 17:19 UTC (permalink / raw) To: Patrick Totzke Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug On Sat, Dec 8, 2012 at 6:04 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote: >> BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get that far. > > Really? Are you in the UK? I'll have to do this dynamically using the > XLIST imap extension. Okay, I've updated it to dynamically find the All Mail folder. ^ permalink raw reply [flat|nested] 35+ messages in thread
[parent not found: <20121213141917.GD3658@tungsten.lan.mitsi.com>]
[parent not found: <CAHmME9op8V+ZJGVwBJPZ653mHYBnMAvxb426LAP-Bu27-rzmYA@mail.gmail.com>]
* Re: gmail importer script [not found] ` <CAHmME9op8V+ZJGVwBJPZ653mHYBnMAvxb426LAP-Bu27-rzmYA@mail.gmail.com> @ 2012-12-13 15:23 ` fREW Schmidt 2012-12-13 15:24 ` Jason A. Donenfeld 0 siblings, 1 reply; 35+ messages in thread From: fREW Schmidt @ 2012-12-13 15:23 UTC (permalink / raw) To: Jason A. Donenfeld; +Cc: notmuch ML [-- Attachment #1: Type: text/plain, Size: 1026 bytes --] On Thu, Dec 13, 2012 at 03:28:30PM +0100, Jason A. Donenfeld wrote: > On Thu, Dec 13, 2012 at 3:19 PM, fREW Schmidt <frioux@gmail.com> wrote: > >> Okay, I've updated it to dynamically find the All Mail folder. > > > > I can't seem to find it now, but I read that Gmail actually localizes > > the name of various folders, so I think All Mail will actually be > > called something else for non-english speakers. It might be a good > > idea for you to at least make it a config option. > > A) Why are you replying off list? > > B) Did you totally miss my message directly above yours where I wrote > "Okay, I've updated it to dynamically find the All Mail folder"? Right, I did see that and I did look at the code; all I'm saying is that Gmail may localize AllMail as I'm fairly sure they localize the other builtin labels (http://git.zx2c4.com/gmail-notmuch/tree/gmail-notmuch.py#n214) I could certainly be wrong though, I haven't looked at this at such a low level before. -- fREW Schmidt http://blog.afoolishmanifesto.com [-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-13 15:23 ` fREW Schmidt @ 2012-12-13 15:24 ` Jason A. Donenfeld 2012-12-13 16:48 ` Patrick Totzke 0 siblings, 1 reply; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-13 15:24 UTC (permalink / raw) To: fREW Schmidt; +Cc: notmuch ML \AllMail is the unlocalized generic version that maps to the localized version, as are all the other back slash situations. ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-13 15:24 ` Jason A. Donenfeld @ 2012-12-13 16:48 ` Patrick Totzke 2012-12-15 8:22 ` Jason A. Donenfeld 0 siblings, 1 reply; 35+ messages in thread From: Patrick Totzke @ 2012-12-13 16:48 UTC (permalink / raw) To: Jason A. Donenfeld, fREW Schmidt; +Cc: notmuch ML Sorry, I'm misusing this thread as a bugtracker.. ------------------------------------------------ Traceback (most recent call last):4|#### | 10% ETA: 1:24:41 3.54 emails/s File "./gmail-notmuch.py", line 251, in <module> main() File "./gmail-notmuch.py", line 89, in main download_new_messages(imap, database, new_messages, destination) File "./gmail-notmuch.py", line 208, in download_new_messages tag_message(database, dest, labels) File "./gmail-notmuch.py", line 168, in tag_message raise e notmuch.errors.FileNotEmailError ------------------------------------------------ best, /p ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-13 16:48 ` Patrick Totzke @ 2012-12-15 8:22 ` Jason A. Donenfeld 2012-12-15 10:41 ` Patrick Totzke 0 siblings, 1 reply; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-15 8:22 UTC (permalink / raw) To: Patrick Totzke; +Cc: notmuch ML [-- Attachment #1: Type: text/plain, Size: 162 bytes --] On Thu, Dec 13, 2012 at 5:48 PM, Patrick Totzke <patricktotzke@gmail.com>wrote: > > notmuch.errors.FileNotEmailError Delete the file it dies on and try again. [-- Attachment #2: Type: text/html, Size: 438 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-15 8:22 ` Jason A. Donenfeld @ 2012-12-15 10:41 ` Patrick Totzke 2012-12-16 20:44 ` Jason A. Donenfeld 0 siblings, 1 reply; 35+ messages in thread From: Patrick Totzke @ 2012-12-15 10:41 UTC (permalink / raw) To: Jason A. Donenfeld; +Cc: notmuch ML [-- Attachment #1: Type: text/plain, Size: 415 bytes --] Quoting Jason A. Donenfeld (2012-12-15 08:22:46) > On Thu, Dec 13, 2012 at 5:48 PM, Patrick Totzke <patricktotzke@gmail.com> > wrote: > > > notmuch.errors.FileNotEmailError > > > Delete the file it dies on and try again. Well, thats not the point.. the script shouldn't die like this. I think it's be better if the script caught that exception, deleted the file and continued.. Thanks, /p [-- Attachment #2: signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEABECAAYFAlDMU/YACgkQlDQDZ9fWxaqk5gCdEcOVSdgOXqsam7objXLMyhPm kdwAoI92eZD/YAMSMrwn7ejCmmF/db5B =llhR -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-15 10:41 ` Patrick Totzke @ 2012-12-16 20:44 ` Jason A. Donenfeld 2012-12-16 22:31 ` Patrick Totzke 0 siblings, 1 reply; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-16 20:44 UTC (permalink / raw) To: Patrick Totzke; +Cc: notmuch ML [-- Attachment #1: Type: text/plain, Size: 591 bytes --] On Sat, Dec 15, 2012 at 11:41 AM, Patrick Totzke <patricktotzke@gmail.com>wrote: > > Well, thats not the point.. the script shouldn't die like this. > I think it's be better if the script caught that exception, deleted the > file > and continued.. Probably, but I suspect it's related to whatever mystery error you ran into before with the corruption. Does deleting that file and trying again fix it? Anyway, this is extremely stable for me and a few others at this point. I'm going to wait for other users to report errors. Alternatively, send me patches if you want things to happen. [-- Attachment #2: Type: text/html, Size: 945 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-16 20:44 ` Jason A. Donenfeld @ 2012-12-16 22:31 ` Patrick Totzke 2012-12-17 8:31 ` Patrick Totzke 0 siblings, 1 reply; 35+ messages in thread From: Patrick Totzke @ 2012-12-16 22:31 UTC (permalink / raw) To: Jason A. Donenfeld; +Cc: notmuch ML [-- Attachment #1: Type: text/plain, Size: 1845 bytes --] Quoting Jason A. Donenfeld (2012-12-16 20:44:04) > On Sat, Dec 15, 2012 at 11:41 AM, Patrick Totzke <patricktotzke@gmail.com> > wrote: > > Well, thats not the point.. the script shouldn't die like this. > I think it's be better if the script caught that exception, deleted the > file > and continued.. > > > Probably, but I suspect it's related to whatever mystery error you ran into > before with the corruption. > > Does deleting that file and trying again fix it? I don't know. How would I know which file to remove? I just removed the newest files in new/cur/tmp but this doesnt solve the issue. What would really help here is some more informative error message! I tried with the `-d` option but did not see any local path in the output. > > Anyway, this is extremely stable for me and a few others at this point. I'm > going to wait for other users to report errors. Alternatively, send me patches > if you want things to happen. Personally, I don't really care if 'things are happening', as I'm kind of OK with my current offlineimap configuration and would seriously consider switching from OI only if it meant an improvement in speed or stability. I test this code because I believe that the notmuch ecosystem would benefit from a working solution and that reviews and bug-reports are important. Of course, I don't expect any miracles in terms of speed at this early stage. But the bug I reported above should be addressed at some point in order not to scare off potential users. Have you considered including your code in offlineimap? I'm asking because OI already solved some of the problems you might face later on if you want to continue working on this. * multi-threaded downloads * the ability to read passwords not as plaintext parameter.. best, /p [-- Attachment #2: signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEABECAAYFAlDOS7oACgkQlDQDZ9fWxarMsACfX/EVgFbyEty3JtK7BDji6oRG sOQAn0aUMlcf0FIfvAAC0K7PoEJ6yhf6 =Jhvq -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-16 22:31 ` Patrick Totzke @ 2012-12-17 8:31 ` Patrick Totzke 0 siblings, 0 replies; 35+ messages in thread From: Patrick Totzke @ 2012-12-17 8:31 UTC (permalink / raw) To: Patrick Totzke, Jason A. Donenfeld; +Cc: notmuch ML [-- Attachment #1: Type: text/plain, Size: 788 bytes --] BTW: I removed the whole maildir contents and restarted the downlowd over night. here's the result: --------------------------------------------------------------- Traceback (most recent call last):2|#################### | 26% ETA: 1:37:22 3.11 emails/s File "./gmail-notmuch.py", line 253, in <module> main() File "./gmail-notmuch.py", line 89, in main download_new_messages(imap, database, new_messages, destination) File "./gmail-notmuch.py", line 210, in download_new_messages tag_message(database, dest, labels) File "./gmail-notmuch.py", line 168, in tag_message raise e notmuch.errors.FileNotEmailError ----------------------------------------------------------------------- /p [-- Attachment #2: signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEABECAAYFAlDO2HYACgkQlDQDZ9fWxaprkQCfat0Rr5KrdpoafQzedwp1+uck 1IsAn3Q7DLNAfMkdh6n4spwL1+K84tk0 =0M77 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-08 17:04 ` Jason A. Donenfeld 2012-12-08 17:19 ` Jason A. Donenfeld @ 2012-12-08 17:46 ` Patrick Totzke 2012-12-09 23:13 ` Jason A. Donenfeld 1 sibling, 1 reply; 35+ messages in thread From: Patrick Totzke @ 2012-12-08 17:46 UTC (permalink / raw) To: Jason A. Donenfeld Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug [-- Attachment #1: Type: text/plain, Size: 3423 bytes --] Quoting Jason A. Donenfeld (2012-12-08 17:04:48) > On Sat, Dec 8, 2012 at 5:20 PM, Patrick Totzke <patricktotzke@gmail.com> wrote: > > I have two new errors: > > > > --------------------------------------------------------- > > ./gmail-notmuch.py -u patricktotzke@gmail.com -p mypwd ~/mail/gmail/ > > > > I also tried with maildir param ~/mail/gmail/\[Google\ Mail\].All\ Mail > > as this is where my all-mail maildir is, and i also tried an absolute path, all the same result. > > > > If I point it to the root of my notmuch directory I get: > > Pointing it at the root directory is what you want. In fact, I'm > removing the argument from the script and having it just get that from > the config. So the correct invocation will be: > > ./gmail-notmuch.py -u blabla -p password Okay.. > > > File "./gmail-notmuch.py", line 147, in create_progressbar > > return ProgressBar(maxval=total, widgets=[text + ": ", SimpleProgress(), Bar(), Percentage(), " ", ETA(), " ", FileTransferSpeed(unit="emails")]) > > TypeError: __init__() got an unexpected keyword argument 'unit' > > You're probably running with an old version of python-statusbar. I just installed it from ubuntu 12.04's repository. the version was 2.2-2. Installed from pip now, and it seems ok. you'll want to make more specific dependencies at some point.. > > > > BTW: I had to change the hardcoded "[Gmail]" to "[Google Mail]" for it to get that far. > > Really? Are you in the UK? I'll have to do this dynamically using the > XLIST imap extension. Yea, but its a *really* old account I first got from google.de. Reading this dynamically would def. help. next try.. ----------------------------------------------------------------- Traceback (most recent call last): File "./gmail-notmuch.py", line 246, in <module> main() File "./gmail-notmuch.py", line 74, in main new_messages = retag_old_messages(database, messages, destination) File "./gmail-notmuch.py", line 176, in retag_old_messages progressbar.start() File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 311, in start self.update(0) File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 283, in update self.fd.write(self._format_line() + '\r') File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 243, in _format_line widgets = ''.join(self._format_widgets()) File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 223, in _format_widgets widget = format_updatable(widget, self) File "/usr/local/lib/python2.7/dist-packages/progressbar/widgets.py", line 38, in format_updatable if hasattr(updatable, 'update'): return updatable.update(pbar) File "/usr/local/lib/python2.7/dist-packages/progressbar/widgets.py", line 184, in update return '%3d%%' % pbar.percentage() File "/usr/local/lib/python2.7/dist-packages/progressbar/__init__.py", line 208, in percentage return self.currval * 100.0 / self.maxval ZeroDivisionError: float division by zero -------------------------------------------------------------------------- Another feature request: Could you make the progressbar stuff optinal? in the end one will want to run this via cronjob and this shouldn't print anything to stdout. Also, I can pretty much recommend the 'logging' module. cheers, /p [-- Attachment #2: signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEABECAAYFAlDDfPIACgkQlDQDZ9fWxaoN0wCfSDu1vUlfji9mYFPRdlzM6LFC k6kAoKDA7JIWG/JMQQn1S8SmyxTfrnco =NABV -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-08 17:46 ` Patrick Totzke @ 2012-12-09 23:13 ` Jason A. Donenfeld 2012-12-10 9:41 ` Patrick Totzke 0 siblings, 1 reply; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-09 23:13 UTC (permalink / raw) To: Patrick Totzke Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug [-- Attachment #1: Type: text/plain, Size: 538 bytes --] On Sat, Dec 8, 2012 at 6:46 PM, Patrick Totzke <patricktotzke@gmail.com>wrote: > > ZeroDivisionError: float division by zero > Fixed it! Sorry about all this... Jeeze Louise. > > Another feature request: > Could you make the progressbar stuff optinal? in the end one will want to > run this via cronjob > and this shouldn't print anything to stdout. > Also, I can pretty much recommend the 'logging' module. > Sure, but... >/dev/null doesn't cut it? I can add some more refined logging capabilities if you want. > > cheers, > /p > [-- Attachment #2: Type: text/html, Size: 1171 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-09 23:13 ` Jason A. Donenfeld @ 2012-12-10 9:41 ` Patrick Totzke 2012-12-10 17:46 ` Jason A. Donenfeld 0 siblings, 1 reply; 35+ messages in thread From: Patrick Totzke @ 2012-12-10 9:41 UTC (permalink / raw) To: Jason A. Donenfeld Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug Quoting Jason A. Donenfeld (2012-12-09 23:13:12) > On Sat, Dec 8, 2012 at 6:46 PM, Patrick Totzke <patricktotzke@gmail.com> wrote: > > ZeroDivisionError: float division by zero > > > Fixed it! Sorry about all this... Jeeze Louise. thx > > Another feature request: > Could you make the progressbar stuff optinal? in the end one will want to > run this via cronjob > and this shouldn't print anything to stdout. > Also, I can pretty much recommend the 'logging' module. > > > Sure, but... >/dev/null doesn't cut it? I can add some more refined logging > capabilities if you want. It would, but its nicer not to load ressources you're not gonna use. Also, progressbar 2.2 seems to be the newest version you see on a reasonably new debian/ubuntu install.. </nitpick> Next exception.. do you have a bugtracker? (/me <3 github) ---------------------------------------------------- Logging in... Selecting all mail... Receiving message list: 24455 of 24455|########################################|100% Time: 0:00:12 2.04 kemails/s Parsing message list and labels... Searching for local messages... Traceback (most recent call last): File "./gmail-notmuch.py", line 248, in <module> main() File "./gmail-notmuch.py", line 75, in main if len(new_messages) == 0: TypeError: object of type 'NoneType' has no len() ---------------------------------------------------- /p ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-10 9:41 ` Patrick Totzke @ 2012-12-10 17:46 ` Jason A. Donenfeld 2012-12-11 7:06 ` Jason A. Donenfeld 0 siblings, 1 reply; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-10 17:46 UTC (permalink / raw) To: Patrick Totzke Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug [-- Attachment #1: Type: text/plain, Size: 879 bytes --] On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke <patricktotzke@gmail.com>wrote: > > It would, but its nicer not to load ressources you're not gonna use. Also, > progressbar 2.2 seems to be the newest version you see on a reasonably new > debian/ubuntu install.. </nitpick> > I'll consider it. > > > Next exception.. do you have a bugtracker? (/me <3 github) > To hell with github. Sometimes I do use a bugtracker though. In light of me being an idiot, maybe I should get one for this script. But this script is so short, and these errors are really due to me being an idiot. Sit tight a little longer and things will begin to work. Sorry for the prior impartial fixes. Once this baby is running you will be quite delighted. > TypeError: object of type 'NoneType' has no len() Fixed. http://git.zx2c4.com/gmail-notmuch/commit/?id=3bf8ce9698ec798a42bc2f92f596243fa8911bf8 [-- Attachment #2: Type: text/html, Size: 1592 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-10 17:46 ` Jason A. Donenfeld @ 2012-12-11 7:06 ` Jason A. Donenfeld 2012-12-11 9:12 ` Patrick Totzke 0 siblings, 1 reply; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-11 7:06 UTC (permalink / raw) To: Patrick Totzke Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug [-- Attachment #1: Type: text/plain, Size: 426 bytes --] On Mon, Dec 10, 2012 at 6:46 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke <patricktotzke@gmail.com>wrote: >> >> It would, but its nicer not to load ressources you're not gonna use. >> Also, progressbar 2.2 seems to be the newest version you see on a >> reasonably new debian/ubuntu install.. </nitpick> >> > > I'll consider it. > --silent or -s now silences stdout. [-- Attachment #2: Type: text/html, Size: 996 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-11 7:06 ` Jason A. Donenfeld @ 2012-12-11 9:12 ` Patrick Totzke 2012-12-11 9:12 ` Jason A. Donenfeld 0 siblings, 1 reply; 35+ messages in thread From: Patrick Totzke @ 2012-12-11 9:12 UTC (permalink / raw) To: Jason A. Donenfeld Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug Quoting Jason A. Donenfeld (2012-12-11 07:06:22) > On Mon, Dec 10, 2012 at 6:46 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > > On Mon, Dec 10, 2012 at 10:41 AM, Patrick Totzke <patricktotzke@gmail.com> > wrote: > > It would, but its nicer not to load ressources you're not gonna use. > Also, progressbar 2.2 seems to be the newest version you see on a > reasonably new debian/ubuntu install.. </nitpick> > > > I'll consider it. > > > --silent or -s now silences stdout. It seems to work, but i'm still waiting for the first run-through. Downloading messages: 457 of 22831| This takes ages. I hope it doesn't try to re-download all my messages everytime. Another thing: It seems that throughout its run, your script locks the notmuch database. This is particularly annoying if you want to read/tag mails while waiting for a long sync-process. I think you only really want to lock the index for a short time in the end. Do you write to the index directly after a msg is downloaded? /p ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-11 9:12 ` Patrick Totzke @ 2012-12-11 9:12 ` Jason A. Donenfeld 2012-12-11 10:04 ` Patrick Totzke [not found] ` <20121213142228.GE3658@tungsten.lan.mitsi.com> 0 siblings, 2 replies; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-11 9:12 UTC (permalink / raw) To: Patrick Totzke Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug [-- Attachment #1: Type: text/plain, Size: 751 bytes --] It seems to work, but i'm still waiting for the first run-through. > > Downloading messages: 457 of 22831| > > This takes ages. I hope it doesn't try to re-download all my messages > everytime. > Nope. It's very smart about not doing that. > > Another thing: > It seems that throughout its run, your script locks the notmuch database. > This is particularly > annoying if you want to read/tag mails while waiting for a long > sync-process. I think you only > really want to lock the index for a short time in the end. > Do you write to the index directly after a msg is downloaded? > I tried to solve this by closing and opening the database each time, but it actually reduced the performance considerably -- even slower than the gmail download. [-- Attachment #2: Type: text/html, Size: 1125 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-11 9:12 ` Jason A. Donenfeld @ 2012-12-11 10:04 ` Patrick Totzke [not found] ` <20121213142228.GE3658@tungsten.lan.mitsi.com> 1 sibling, 0 replies; 35+ messages in thread From: Patrick Totzke @ 2012-12-11 10:04 UTC (permalink / raw) To: Jason A. Donenfeld Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch, Rainer M Krug Quoting Jason A. Donenfeld (2012-12-11 09:12:56) > It seems to work, but i'm still waiting for the first run-through. > > > Downloading messages: 457 of 22831| > > This takes ages. I hope it doesn't try to re-download all my messages > everytime. > > > Nope. It's very smart about not doing that. > hmmk I ended up killing the process after i saw that ETA was >3h. This broke my index! afterwards, a `notmuch new` printed A Xapian exception occurred creating a directory: Expected block 485 to be level 1, not 0. I wasnt able to properly read the index anymore and had to restore an old index dump. not cool! /p ^ permalink raw reply [flat|nested] 35+ messages in thread
[parent not found: <20121213142228.GE3658@tungsten.lan.mitsi.com>]
* Re: gmail importer script [not found] ` <20121213142228.GE3658@tungsten.lan.mitsi.com> @ 2012-12-13 14:30 ` Jason A. Donenfeld 2012-12-13 14:35 ` fREW Schmidt 0 siblings, 1 reply; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-13 14:30 UTC (permalink / raw) To: fREW Schmidt; +Cc: notmuch Please don't reply offlist. This is message #2 that's been sent offlist. On Thu, Dec 13, 2012 at 3:22 PM, fREW Schmidt <frioux@gmail.com> > One thing you might want to try is to only open the notmuch index for > writing at a specified sync time. How do I do this? What's the python function? I couldn't find anything. Would be interested in doing this though if it exists. > Also I was going to say you might want to catch SIGINT like The library already does this. ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-13 14:30 ` Jason A. Donenfeld @ 2012-12-13 14:35 ` fREW Schmidt 2012-12-13 15:15 ` Jason A. Donenfeld 0 siblings, 1 reply; 35+ messages in thread From: fREW Schmidt @ 2012-12-13 14:35 UTC (permalink / raw) To: Jason A. Donenfeld; +Cc: notmuch [-- Attachment #1: Type: text/plain, Size: 795 bytes --] On Thu, Dec 13, 2012 at 03:30:19PM +0100, Jason A. Donenfeld wrote: > Please don't reply offlist. This is message #2 that's been sent offlist. Oh I'm sorry, that was an accident. > On Thu, Dec 13, 2012 at 3:22 PM, fREW Schmidt <frioux@gmail.com> > > One thing you might want to try is to only open the notmuch index for > > writing at a specified sync time. > > How do I do this? What's the python function? I couldn't find > anything. Would be interested in doing this though if it exists. I just mean a general programming technique. As you get information about tags you hold them in memory and then sync it later and clear your memory. > > Also I was going to say you might want to catch SIGINT like > > The library already does this. -- fREW Schmidt http://blog.afoolishmanifesto.com [-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-13 14:35 ` fREW Schmidt @ 2012-12-13 15:15 ` Jason A. Donenfeld 0 siblings, 0 replies; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-13 15:15 UTC (permalink / raw) To: fREW Schmidt; +Cc: notmuch On Thu, Dec 13, 2012 at 3:35 PM, fREW Schmidt <frioux@gmail.com> wrote: > I just mean a general programming technique. As you get information > about tags you hold them in memory and then sync it later and clear > your memory. Oh, yea, that's a great idea. I'll play with that. Good thinking. ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-07 8:53 ` Rainer M Krug 2012-12-07 9:23 ` Patrick Totzke @ 2012-12-07 13:48 ` Jason A. Donenfeld 2012-12-07 13:57 ` Rainer M Krug 2012-12-07 15:30 ` Jason A. Donenfeld 1 sibling, 2 replies; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-07 13:48 UTC (permalink / raw) To: Rainer M Krug; +Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch [-- Attachment #1: Type: text/plain, Size: 1289 bytes --] On Fri, Dec 7, 2012 at 9:53 AM, Rainer M Krug <R.M.Krug@gmail.com> wrote: > > Sounds interesting. > > I don't know much python, so excuse my questions: > > 1) are multiple downloads of an email done, if the mail has multiple > labels? > No. > 2) I am downloading with the patched notmuch (to add x-keywords containing > the labels) only the > "All Mail" folder - does your script use this information (if present) to > tag the emails? > What is the patched notmuch? What does that do? > 3) are label changes in gmail synched into the local emails? > It pulls labels from gmail messages into notmuch, with the labels in tact, once per each email. > 4) I assume, local changes are not synced to gmail? > Correctamundo. > 5) how are conflicts handled (local tag changes versus label changes on > gmail)? > They aren't. See 3. > > Thanks, > > Rainer > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ > > iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk > ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x > =Sx0B > -----END PGP SIGNATURE----- > > _______________________________________________ > notmuch mailing list > notmuch@notmuchmail.org > http://notmuchmail.org/mailman/listinfo/notmuch > [-- Attachment #2: Type: text/html, Size: 2613 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-07 13:48 ` Jason A. Donenfeld @ 2012-12-07 13:57 ` Rainer M Krug 2012-12-07 15:32 ` Jason A. Donenfeld 2012-12-07 15:30 ` Jason A. Donenfeld 1 sibling, 1 reply; 35+ messages in thread From: Rainer M Krug @ 2012-12-07 13:57 UTC (permalink / raw) To: Jason A. Donenfeld; +Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/12/12 14:48, Jason A. Donenfeld wrote: > > > On Fri, Dec 7, 2012 at 9:53 AM, Rainer M Krug <R.M.Krug@gmail.com <mailto:R.M.Krug@gmail.com>> > wrote: > > Sounds interesting. > > I don't know much python, so excuse my questions: > > 1) are multiple downloads of an email done, if the mail has multiple labels? > > > No. Good. > > > 2) I am downloading with the patched notmuch (to add x-keywords containing the labels) only > the "All Mail" folder - does your script use this information (if present) to tag the emails? > > > What is the patched notmuch? What does that do? Sorry - should have been "patched offlineimap" ( https://github.com/aroig/offlineimap http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970 and http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 ) It is synching the gmail labels into the x-keywords header as a comma separated list. These could then be used by notmuch to be added to the labels. It also does sync changes from the X-Keywords to gmail (see http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970 for details) > > > 3) are label changes in gmail synched into the local emails? > > > It pulls labels from gmail messages into notmuch, with the labels in tact, once per each > email. Good. > > > 4) I assume, local changes are not synced to gmail? > > > Correctamundo. > OK. > > > 5) how are conflicts handled (local tag changes versus label changes on gmail)? > > > They aren't. See 3. > Thanks, Rainer > > > Thanks, > > Rainer -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG > with Mozilla - http://www.enigmail.net/ > > iEYEARECAAYFAlDBrqAACgkQoYgNqgF2egqrEwCeOSr5TGZgAIaRvzHCl1Nv+DBk > ko4AnRT0FMBwYNRGzSclXxZXzaH0Q23x =Sx0B -----END PGP SIGNATURE----- > > _______________________________________________ notmuch mailing list notmuch@notmuchmail.org > <mailto:notmuch@notmuchmail.org> http://notmuchmail.org/mailman/listinfo/notmuch > > - -- Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Stellenbosch University South Africa Tel : +33 - (0)9 53 10 27 44 Cell: +33 - (0)6 85 62 59 98 Fax : +33 - (0)9 58 10 27 44 Fax (D): +49 - (0)3 21 21 25 22 44 email: Rainer@krugs.de Skype: RMkrug -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlDB9cMACgkQoYgNqgF2egrEkACeLD9p4CUW3EhliFQ44cigpZdF o8AAnA9M8tmw52W3m62QFJaFuoOLdto2 =6Zrk -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-07 13:57 ` Rainer M Krug @ 2012-12-07 15:32 ` Jason A. Donenfeld 2012-12-08 12:40 ` Rainer M Krug 0 siblings, 1 reply; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-07 15:32 UTC (permalink / raw) To: Rainer; +Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch [-- Attachment #1: Type: text/plain, Size: 1051 bytes --] On Fri, Dec 7, 2012 at 2:57 PM, Rainer M Krug <r.m.krug@gmail.com> wrote: > > > > > > > 2) I am downloading with the patched notmuch (to add x-keywords > containing the labels) only > > the "All Mail" folder - does your script use this information (if > present) to tag the emails? > > > > > > What is the patched notmuch? What does that do? > > Sorry - should have been "patched offlineimap" ( > https://github.com/aroig/offlineimap > http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970and > http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 ) > > It is synching the gmail labels into the x-keywords header as a comma > separated list. These could > then be used by notmuch to be added to the labels. It also does sync > changes from the X-Keywords > to gmail (see > http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970for details) I see. I use the All Mail folder and get the tags using the X-GM-LABELS imap extension. I don't muck with the mail headers in the process, fortunately. [-- Attachment #2: Type: text/html, Size: 1815 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-07 15:32 ` Jason A. Donenfeld @ 2012-12-08 12:40 ` Rainer M Krug 2012-12-08 13:15 ` Jason A. Donenfeld 0 siblings, 1 reply; 35+ messages in thread From: Rainer M Krug @ 2012-12-08 12:40 UTC (permalink / raw) To: Jason A. Donenfeld; +Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch [-- Attachment #1: Type: text/plain, Size: 2819 bytes --] On 7 December 2012 16:32, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > On Fri, Dec 7, 2012 at 2:57 PM, Rainer M Krug <r.m.krug@gmail.com> wrote: >> >> > >> > >> > 2) I am downloading with the patched notmuch (to add x-keywords >> containing the labels) only >> > the "All Mail" folder - does your script use this information (if >> present) to tag the emails? >> > >> > >> > What is the patched notmuch? What does that do? >> >> Sorry - should have been "patched offlineimap" ( >> https://github.com/aroig/offlineimap >> >> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970and >> http://article.gmane.org/gmane.mail.imap.offlineimap.general/5970 ) >> >> It is synching the gmail labels into the x-keywords header as a comma >> separated list. These could >> then be used by notmuch to be added to the labels. It also does sync >> changes from the X-Keywords >> to gmail (see >> http://thread.gmane.org/gmane.mail.imap.offlineimap.general/5943/focus=5970for details) > > > I see. > > I use the All Mail folder and get the tags using the X-GM-LABELS imap > extension. I don't muck with the mail headers in the process, fortunately. > I actually prefer this approach, but I think it would be more useful to leave the syncing of the emails to a different program, and then just managing the labels / tags with your tool (which is notmuch territory). So the workflow would be: Initial or download (only remote changes): 1) Download email ("All Mail" folder) with whatever you are using (offlinemail, ...) 2) call your script to index the downloaded emails (All Mail" folder) with notmuch and get the labels and set / remove them in notmuch accordingly. Upload (only local changes): 1) upload emails using whatever (offlineimap, ...) 2) upload the changed tags and create the needed new labels (and possibly delete empty ons?) The tricky part will be the synching when changes on both sides were observed. The mail sync is implemented in e.g. offlineimap and works reilably. But I don't know how one could handle tag / label changes on both sides. So, implementing the mail sync in this script would, as I see it, kind of reinventing the wheel. One can leave it in, but I would like to have an option to leave the synching of the emails to a separate program. Cheers, Rainer -- NEW GERMAN FAX NUMBER!!! Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany) Centre of Excellence for Invasion Biology Natural Sciences Building Office Suite 2039 Stellenbosch University Main Campus, Merriman Avenue Stellenbosch South Africa Cell: +27 - (0)83 9479 042 Fax: +27 - (0)86 516 2782 Fax: +49 - (0)321 2125 2244 email: Rainer@krugs.de Skype: RMkrug Google: R.M.Krug@gmail.com [-- Attachment #2: Type: text/html, Size: 4137 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-08 12:40 ` Rainer M Krug @ 2012-12-08 13:15 ` Jason A. Donenfeld 2012-12-10 9:12 ` Rainer M Krug 0 siblings, 1 reply; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-08 13:15 UTC (permalink / raw) To: Rainer; +Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch > I actually prefer this approach, but I think it would be more useful to > leave the syncing of the emails to a different program, and then just > managing the labels / tags with your tool (which is notmuch territory). So > the workflow would be: > So, implementing the mail sync in this script would, as I see it, kind of > reinventing the wheel. You're misunderstanding me. Let me make it very clear what the script does. Actually why don't you just read the source code? Please? Well, anyway, here: - It looks in gmail and asks it which messages gmail has - It downloads those messages - It applies gmail's labels to those messages as tags The end. Along the way it has some smart things to reduce redundant downloads. For more information, consult the source code. ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-08 13:15 ` Jason A. Donenfeld @ 2012-12-10 9:12 ` Rainer M Krug 0 siblings, 0 replies; 35+ messages in thread From: Rainer M Krug @ 2012-12-10 9:12 UTC (permalink / raw) To: notmuch Cc: public-public-notmuch-gxuj+Tv9EO5zyzON3hdc1g-wOFGN7rlS/M9smdsby/KFg, public-Rainer-vfylz/Ys1k4, public-notmuch-gxuj+Tv9EO5zyzON3hdc1g -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 08/12/12 14:15, Jason A. Donenfeld wrote: >> I actually prefer this approach, but I think it would be more useful to leave the syncing of >> the emails to a different program, and then just managing the labels / tags with your tool >> (which is notmuch territory). So the workflow would be: So, implementing the mail sync in >> this script would, as I see it, kind of reinventing the wheel. > > You're misunderstanding me. Let me make it very clear what the script does. Actually why don't > you just read the source code? Please? Well, Because I have never used python, and you know what your script is doing. I prefer to know what the script is doing, before I try it out and it changes up my local emails and my notmuch databases in unexpected ways, because I did not understand the source code properly? > anyway, here: > > - It looks in gmail and asks it which messages gmail has OK. > - It downloads those messages Here is my question - see below. > - It applies gmail's labels to those messages as tags Perfect. Now my final question: after reading your info which is useful, I looked in more detail at the source code and it made now sense to me. My question is: If I only want to aply the tags and *not download* emails, can I simply remove the line 88: download_new_messages(imap, database, new_messages, destination) Is this correct? In this case, the script would ony tag y messages which are already downloaded? If this is the case, I would immediately start using the script. Rainer > > The end. > > Along the way it has some smart things to reduce redundant downloads. > > For more information, consult the source code. > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlDFp5kACgkQoYgNqgF2egqfVACfWckdCl3SOmue6EEHF1cv6d44 IKgAmwV9cQH/VxCmPHmmU6z3akQ9pFZV =vZxS -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: gmail importer script 2012-12-07 13:48 ` Jason A. Donenfeld 2012-12-07 13:57 ` Rainer M Krug @ 2012-12-07 15:30 ` Jason A. Donenfeld 1 sibling, 0 replies; 35+ messages in thread From: Jason A. Donenfeld @ 2012-12-07 15:30 UTC (permalink / raw) To: Rainer M Krug; +Cc: public-notmuch-gxuj+Tv9EO5zyzON3hdc1g, notmuch [-- Attachment #1: Type: text/plain, Size: 539 bytes --] On Fri, Dec 7, 2012 at 2:48 PM, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > > >> 3) are label changes in gmail synched into the local emails? >> > > It pulls labels from gmail messages into notmuch, with the labels in tact, > once per each email. > Okay, it's way more robust now. Just updated it. Now it will pull gmail labels to notmuch, each time, every time. So you can run this thing over and over, and it will always push gmail into notmuch. It doesn't yet sync the the other way yet, but at least this is now fairly robust. [-- Attachment #2: Type: text/html, Size: 1015 bytes --] ^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH] Minor style changes. 2012-12-07 0:24 gmail importer script Jason A. Donenfeld 2012-12-07 4:05 ` Kushal Kumaran @ 2012-12-07 11:48 ` Michal Nazarewicz 1 sibling, 0 replies; 35+ messages in thread From: Michal Nazarewicz @ 2012-12-07 11:48 UTC (permalink / raw) To: Jason; +Cc: notmuch From: Michal Nazarewicz <mina86@mina86.com> --- gmail-notmuch.py | 29 +++++++++-------------------- 1 files changed, 9 insertions(+), 20 deletions(-) On Fri, Dec 07 2012, Jason A. Donenfeld wrote: > I wrote a script that imports emails and tags from gmail. It's > resumable and appears to be working reasonably well. I could use some > experienced eyes looking at it, as my first exposure to notmuch was > just a few hours ago. Didn't really use it, but I still have some suggested code changes: ;) diff --git a/gmail-notmuch.py b/gmail-notmuch.py index 04a60dc..ef7fa85 100755 --- a/gmail-notmuch.py +++ b/gmail-notmuch.py @@ -43,7 +43,7 @@ def main(): (options, args) = parser.parse_args() if options.username == None or options.password == None: parser.error("Username and password are required.") - if not options.username.lower().endswith("@gmail.com") and not options.username.lower().endswith("@googlemail.com"): + if options.username.find("@") == -1: options.username += "@gmail.com" if len(args) == 0: parser.error("Maildir location is required.") @@ -60,7 +60,7 @@ def main(): print("Nothing to do!") logout(imap) sys.exit(0) - + download_new_messages(imap, new_messages, destination_dir) logout(imap) @@ -84,9 +84,8 @@ def discover_new_messages(imap, old_messages): for response in data: imap_seq = response[0:response.find(" ")] gmail_id = response[response.rfind(" ") + 1:len(response) - 1] - if gmail_id in old_messages: - continue - new_messages.append((gmail_id, imap_seq)) + if gmail_id not in old_messages: + new_messages.append((gmail_id, imap_seq)) return new_messages def download_new_messages(imap, messages, destination): @@ -112,7 +111,7 @@ def download_new_messages(imap, messages, destination): typ, data = imap.fetch(str(imap_seq), "(FLAGS X-GM-LABELS)") if typ != "OK": sys.exit("Failed to download labels gmail-%d/imap-%d" % (gmail_id, imap_seq)) - + labels = label_parser.search(data[0]).groups() labels = filter_labels(shlex.split(labels[0], False, True) + labels[1].split(" ")) @@ -149,22 +148,12 @@ def filter_labels(labels): "\\Important": None, # I realize this is controversial, but I hate the priority inbox. "Junk": "spam", "NonJunk": None } - ret = set() - for label in labels: - if label in translation: - if translation[label] is None: - continue - ret.add(translation[label]) - else: - ret.add(label) - if "!read!" in ret: - ret.remove("!read!") - else: + ret = set(translation.get(label, label) for label in labels) + if not ret.pop("!read!", None): ret.add("unread") - if "" in ret: - ret.remove("") + ret.pop(None, None) + ret.pop("", None) return ret - def logout(imap): imap.close() -- 1.7.7.3 ^ permalink raw reply related [flat|nested] 35+ messages in thread
end of thread, other threads:[~2012-12-17 8:31 UTC | newest]
Thread overview: 35+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-12-07 0:24 gmail importer script Jason A. Donenfeld
2012-12-07 4:05 ` Kushal Kumaran
2012-12-07 8:53 ` Rainer M Krug
2012-12-07 9:23 ` Patrick Totzke
2012-12-07 13:49 ` Jason A. Donenfeld
2012-12-08 16:20 ` Patrick Totzke
2012-12-08 17:04 ` Jason A. Donenfeld
2012-12-08 17:19 ` Jason A. Donenfeld
[not found] ` <20121213141917.GD3658@tungsten.lan.mitsi.com>
[not found] ` <CAHmME9op8V+ZJGVwBJPZ653mHYBnMAvxb426LAP-Bu27-rzmYA@mail.gmail.com>
2012-12-13 15:23 ` fREW Schmidt
2012-12-13 15:24 ` Jason A. Donenfeld
2012-12-13 16:48 ` Patrick Totzke
2012-12-15 8:22 ` Jason A. Donenfeld
2012-12-15 10:41 ` Patrick Totzke
2012-12-16 20:44 ` Jason A. Donenfeld
2012-12-16 22:31 ` Patrick Totzke
2012-12-17 8:31 ` Patrick Totzke
2012-12-08 17:46 ` Patrick Totzke
2012-12-09 23:13 ` Jason A. Donenfeld
2012-12-10 9:41 ` Patrick Totzke
2012-12-10 17:46 ` Jason A. Donenfeld
2012-12-11 7:06 ` Jason A. Donenfeld
2012-12-11 9:12 ` Patrick Totzke
2012-12-11 9:12 ` Jason A. Donenfeld
2012-12-11 10:04 ` Patrick Totzke
[not found] ` <20121213142228.GE3658@tungsten.lan.mitsi.com>
2012-12-13 14:30 ` Jason A. Donenfeld
2012-12-13 14:35 ` fREW Schmidt
2012-12-13 15:15 ` Jason A. Donenfeld
2012-12-07 13:48 ` Jason A. Donenfeld
2012-12-07 13:57 ` Rainer M Krug
2012-12-07 15:32 ` Jason A. Donenfeld
2012-12-08 12:40 ` Rainer M Krug
2012-12-08 13:15 ` Jason A. Donenfeld
2012-12-10 9:12 ` Rainer M Krug
2012-12-07 15:30 ` Jason A. Donenfeld
2012-12-07 11:48 ` [PATCH] Minor style changes Michal Nazarewicz
Code repositories for project(s) associated with this public inbox https://yhetil.org/notmuch.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).