Can't get notmuch to find mail

Can't get notmuch to find mail

[Stephen Paul Weber]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I've installed notmuch, run the setup to point it at the maildir of my Gmail
"All Mail" folder, and run notmuch new.  It says it found 32981 file, but no
new mail... what?  Then when I run searches, none of them return results.

Help?

- -- 
Stephen Paul Weber, @singpolyma
See <http://singpolyma.net> for how I prefer to be contacted
edition right joseph
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQIcBAEBCAAGBQJMWvr+AAoJENEcKRHOUZzexM4QAJKgXzOdKMVUdn4EQDBB62J4
tzX7rwTXX3/yBmYyrW2ThmkxzcEifpJnLG4ydubSfAy4kBVoMn31q4lK7PCJnLyj
LxcwRKFxxB56aDxCCPwBzIvkzUpQDPZ/lAPMgLidq5pKmrKH5P7Fq6WWm9/lB4ft
rFHEhfhe/4ZQsYtYSbPQBJCOiWjEAX5uxQT+w57m+Qmynqy/FcSgMxD6dslD7ruK
+WihfNTKU0Yn42bWyb7ImGQaItsn9nFLf2EcNaGRAON3ZSBS/6+s+8VCnGa1zoTq
lQlmfwQDRRXdVbMMo0ATgtMslXH7R1mDZe4x82O+bEQNLuxyq34X4VvnfQhcjp19
Brqy414Caq27uqbU0fo3sMpgcWf/lv+JNkQZlG9S/dNl/VQay3M1ewqEt6cwAQG3
iut/cU2J2jkmO8h3nrFU8ymcer4GbnAFzpFJTt9rKaKOkJwjSsUfgNNkP2du1TCS
uwr61OyOJ8L0rFYflolcwaB2MWTKnQK6YDQDIvQ5k0mZQ2EIkhBEn765iLqUqjOQ
cn4JZnQzDpH7c/fJDqHORCJLRUXht9quWbyYdE4xAO2VkqLYz+5F+ageGslnQ3PO
aLdjiYAQNFxokr+YliuP7Dz2oYx5bmMrUQGdABphjo/A5ZorOGvaXPsfJraK659e
EuF2ux8+7No/RYF/+8O0
=Han3
-----END PGP SIGNATURE-----

Re: Can't get notmuch to find mail

[Bryan Hunt]

Files somehow have a timestamp in the future ?


Excerpts from Stephen Paul Weber's message of Thu Aug 05 18:55:11 +0100 2010:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> I've installed notmuch, run the setup to point it at the maildir of my Gmail
> "All Mail" folder, and run notmuch new.  It says it found 32981 file, but no
> new mail... what?  Then when I run searches, none of them return results.
> 
> Help?
> 
> - -- 
> Stephen Paul Weber, @singpolyma
> See <http://singpolyma.net> for how I prefer to be contacted
> edition right joseph
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> 
> iQIcBAEBCAAGBQJMWvr+AAoJENEcKRHOUZzexM4QAJKgXzOdKMVUdn4EQDBB62J4
> tzX7rwTXX3/yBmYyrW2ThmkxzcEifpJnLG4ydubSfAy4kBVoMn31q4lK7PCJnLyj
> LxcwRKFxxB56aDxCCPwBzIvkzUpQDPZ/lAPMgLidq5pKmrKH5P7Fq6WWm9/lB4ft
> rFHEhfhe/4ZQsYtYSbPQBJCOiWjEAX5uxQT+w57m+Qmynqy/FcSgMxD6dslD7ruK
> +WihfNTKU0Yn42bWyb7ImGQaItsn9nFLf2EcNaGRAON3ZSBS/6+s+8VCnGa1zoTq
> lQlmfwQDRRXdVbMMo0ATgtMslXH7R1mDZe4x82O+bEQNLuxyq34X4VvnfQhcjp19
> Brqy414Caq27uqbU0fo3sMpgcWf/lv+JNkQZlG9S/dNl/VQay3M1ewqEt6cwAQG3
> iut/cU2J2jkmO8h3nrFU8ymcer4GbnAFzpFJTt9rKaKOkJwjSsUfgNNkP2du1TCS
> uwr61OyOJ8L0rFYflolcwaB2MWTKnQK6YDQDIvQ5k0mZQ2EIkhBEn765iLqUqjOQ
> cn4JZnQzDpH7c/fJDqHORCJLRUXht9quWbyYdE4xAO2VkqLYz+5F+ageGslnQ3PO
> aLdjiYAQNFxokr+YliuP7Dz2oYx5bmMrUQGdABphjo/A5ZorOGvaXPsfJraK659e
> EuF2ux8+7No/RYF/+8O0
> =Han3
> -----END PGP SIGNATURE-----
-- 
--

Re: Can't get notmuch to find mail

[Michal Sojka]

On Thu, 05 Aug 2010, Stephen Paul Weber wrote:
> I've installed notmuch, run the setup to point it at the maildir of my Gmail
> "All Mail" folder, and run notmuch new.  It says it found 32981 file, but no
> new mail... what?  Then when I run searches, none of them return results.
> 
> Help?

That's really strange. Could you check the database with delve tool
(a part of Xapian), whether you can get similar output as me?

If I run the following command, I get the list of document numbers
(messages) written by me:

$ delve -t XFROMsojkam1 ~/mail/.notmuch/xapian
Posting List for term `XFROMsojkam1' (termfreq 8229, collfreq 8276,
wdf_max 8276): 525 526 527 528 529 530 531 1178 1183 4593 4632 5290 5844
5972 5974 5975 5982 5988 6000 6003 6008 6015 6016 6017 6018 6070 6085
6086 6087 6088 6089 6104 6106 6150 6153 6175 6177 6178 6188 6199 6200
6202 6204 6247 6249 6262 6352 6353 6354 6355 6366 6371 6383 6384 6385
6394 6397 6398 6399 6416 6426 6432 6434...

Then, I can look at what Xapian knows about the document 525:

$ delve -V -r 525 ~/mail/.notmuch/xapian                                                                                                                            
Values for record #525: 0:�\nT� 1:46E0554F.7060608@fel.cvut.cz
Term List for record #525: 000000 1 154 2004 2100 4.01 5_deshpande
802.11 G00000000000001fa Q46E0554F.7060608@fel.cvut.cz Tmail
XFDIRENTRY524:1265611852_0.11574.steelpick,U=64,FMD5=db3af42ba64b595db6ab3ea999fe0bb1:2,S
XFROMcvut XFROMcz XFROMfel XFROMmichal...

Can you see the messages in the database or not?

-Michal

Re: Can't get notmuch to find mail

[Stephen Paul Weber]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Somebody claiming to be Michal Sojka wrote:
> On Thu, 05 Aug 2010, Stephen Paul Weber wrote:
> > I've installed notmuch, run the setup to point it at the maildir of my Gmail
> > "All Mail" folder, and run notmuch new.  It says it found 32981 file, but no
> > new mail... what?  Then when I run searches, none of them return results.
> > 
> > Help?
> 
> $ delve -t XFROMsojkam1 ~/mail/.notmuch/xapian
> Posting List for term `XFROMsojkam1' (termfreq 8229, collfreq 8276,
> wdf_max 8276): 525 526 527 528 529 530 531 1178 1183 4593 4632 5290 5844
> 5972 5974 5975 5982 5988 6000 6003 6008 6015 6016 6017 6018 6070 6085
> 6086 6087 6088 6089 6104 6106 6150 6153 6175 6177 6178 6188 6199 6200
> 6202 6204 6247 6249 6262 6352 6353 6354 6355 6366 6371 6383 6384 6385
> 6394 6397 6398 6399 6416 6426 6432 6434...

For me:

$ delve -t XFROMsingpolyma xapian
term `XFROMsingpolyma' not in database
$ delve -t XFROMdenver xapian
term `XFROMdenver' not in database
$ delve -V -r 1 xapian
Values for record #1: 0:à\Æ.¾p
Term List for record #1: XDIRECTORY
$ delve -V -r 2 xapian
Values for record #2:
Error: DocNotFoundError: Document 2 not found.

- -- 
Stephen Paul Weber, @singpolyma
See <http://singpolyma.net> for how I prefer to be contacted
edition right joseph
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQIbBAEBCAAGBQJMZAscAAoJENEcKRHOUZzeb5IP90HfbGFoACQpZBpYYP9ROjV3
ZIuHUc+sTmg9IgC2Pl4jErN4zpbS8UlIL9PjZoUurQStqG1bCPsUQ2M5THHfO02+
MFoekCrd1/NTfvGz6aVNDme2ShtNBbN41Bv2xXaHx4UOI2746sbuUPidVDKdZSSo
bvPU+0E3Od1kfRZTBoIYT3QQgYoOlklPGWkOAilGhZnvPxIva0eQMZY1HftZoC96
zyuaxiKk7EZejIPsxvTAdoyMTk55l87+SZO2OC8/0mKg0I7N1rDocdbQheYcrj2R
ml8jZAO0pW1AAV98ZmDyZDskd/NoxvURXUS1y84hOxJYC5wnbAv6nLozJqRS1esK
FYa4Q8PoWPMSRXkpp9SD7FDTzfF5aF1uZ07xczpuk7hpT7eCyPSJxQwimUuvpLNA
D5Dns5yk7W6Bwr9g98c2E+T13N3FePWLENgp8gkmyhtq9ffK2Re39LhIn8Xy6FiN
5CuujUh+UjbdU4o4S9C9h/LbNHDz/LLsHL1ZO3LDgUtP1i+wGhZxlOXq0tcJsD87
LOoTPNni9NpbJ2OSlxD7YWDpRuU0WAH8lD2HYN6qJPpFTTv+WT6NT2PjLwuw8o2A
/qvxP0enghXYIgNJuimfkLwnAF8jcthavLAQxPvHspOg4R26YFNxy8hIxN8TTL4D
wddbKrU4lfedUdZPVy4=
=po+9
-----END PGP SIGNATURE-----

Re: Can't get notmuch to find mail

[Michal Sojka]

On Thu, 12 Aug 2010, Stephen Paul Weber wrote:
> For me:
> 
> $ delve -t XFROMsingpolyma xapian
> term `XFROMsingpolyma' not in database
> $ delve -t XFROMdenver xapian
> term `XFROMdenver' not in database
> $ delve -V -r 1 xapian
> Values for record #1: 0:à\Æ.¾p
> Term List for record #1: XDIRECTORY
> $ delve -V -r 2 xapian
> Values for record #2:
> Error: DocNotFoundError: Document 2 not found.

Hmm, so you do not have the messages in the database.

On which platform (and filesystem) you are?

Could you run notmuch new through strace and/or apply the patch bellow
and send the output?

-Michal

diff --git a/notmuch-new.c b/notmuch-new.c
index 8818728..2fef9d1 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -298,6 +298,9 @@ add_files_recursive (notmuch_database_t *notmuch,
        }
 
        next = talloc_asprintf (notmuch, "%s/%s", path, entry->d_name);
+
+       printf ("Pass 1 recursing into : %s = %s\n", entry->d_name, next);
+
        status = add_files_recursive (notmuch, next, state);
        if (status && ret == NOTMUCH_STATUS_SUCCESS)
            ret = status;
@@ -307,8 +310,10 @@ add_files_recursive (notmuch_database_t *notmuch,
 
     /* If this directory hasn't been modified since the last
      * "notmuch new", then we can skip the second pass entirely. */
-    if (fs_mtime <= db_mtime)
+    if (fs_mtime <= db_mtime) {
+       printf ("Skipping pass 2 (old mtime)\n");
        goto DONE;
+    }
 
     /* Pass 2: Scan for new files, removed files, and removed directories. */
     for (i = 0; i < num_fs_entries; i++)
@@ -318,6 +323,8 @@ add_files_recursive (notmuch_database_t *notmuch,
 
         entry = fs_entries[i];
 
+       printf ("Pass 2 looking at: %s\n", entry->d_name);
+
        /* Check if we've walked past any names in db_files or
         * db_subdirs. If so, these have been deleted. */
        while (notmuch_filenames_valid (db_files) &&
@@ -380,6 +387,7 @@ add_files_recursive (notmuch_database_t *notmuch,
        if (notmuch_filenames_valid (db_files) &&
            strcmp (notmuch_filenames_get (db_files), entry->d_name) == 0)
        {
+           printf ("Pass 2 skipping: %s (already added)\n", entry->d_name);
            notmuch_filenames_move_to_next (db_files);
            continue;
        }

Re: Can't get notmuch to find mail

[Michal Sojka]

On 08/12/2010 05:33 PM, Stephen Paul Weber wrote:
> Somebody claiming to be Michal Sojka wrote:
>> Could you run notmuch new through strace and/or apply the patch bellow
>> and send the output?
>
> TRACE file attached

Hmm, I see nothing suspicious in the TRACE (besides that it does not 
index your mails). Are you really using version 0.1 (notmuch --version)?

There was a bug in some pre-0.1 versions which would explain the 
behavior you get - see the commit
c5416b6 notmuch new: Fix to work on filesystems returning DT_UNKNOWN

-Michal

Re: Can't get notmuch to find mail

[Stephen Paul Weber]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Somebody claiming to be Michal Sojka wrote:
> On 08/12/2010 05:33 PM, Stephen Paul Weber wrote:
> >Somebody claiming to be Michal Sojka wrote:
> >>Could you run notmuch new through strace and/or apply the patch bellow
> >>and send the output?
> >
> >TRACE file attached
> 
> Hmm, I see nothing suspicious in the TRACE (besides that it does not
> index your mails). Are you really using version 0.1 (notmuch
> --version)?

Hmm...
My manpage said 0.1 on it, so I didn't question, but my package manager says
0.0+201001211401

I guess I should upgrade.  I suppose the only official way to do that (until
the Ubuntu package updates) is to build from source?

- -- 
Stephen Paul Weber, @singpolyma
See <http://singpolyma.net> for how I prefer to be contacted
edition right joseph
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQIcBAEBCAAGBQJMZGfWAAoJENEcKRHOUZzegTYP/3mFOh0YZr4MMoCbX4WNpykY
tY7hPAqNO6R3KJsSsz+1UFGjZbXdtxoQiFuBxTzpb+d3YzZ3uou6WCCKRVq1fFl+
4Mmr10FOnHVYP76VfsEahKFn2MuzD/hj2F525G9i/u9U9pbfRxb3MUMpYLv9Xn6P
Ggh263/0cqwFoaMbJBLMJnBQpBhy0p7VVxfuER26QxCvJW9xT+SBZwDN5K01fHfR
ZrgRvkNnU9IGnbbkeWJWSRlCcPmCwXn5gahWhMmt5iUZ+IZlIW3RbR3llLNKhGGX
CtSaWiakYGT1uvp32J8hBh24nnxMh02vGVLiotdp4wPx7hAiCpc+toA1GAVh0uTs
2BM+20nHs2gNw5uNx4r6l1GLdg6ctqrtaY9j7A8OtNNbdWqllu527sI/Vw60UlC+
jGxIt7C8FTDJ9hti+6NUt/YGYNiA12CMaCTi4OpQao7dw7IVmW4y1HXjfcqMUYvV
PIs4ELuKrDCOAZHVGUBg3MaYqqn4pXSt1Va3e7Bv5v5tu8AuNSQSIbynbaqj0CDf
bb7P61fL/f5BF/lQoZDpPatl7tYJB5h8sAnmKDlTSN2mwZQ5z5Uf7Dtj11qyiCd0
tsWzvTiCrnntY3X5sw9akRd+HPrhzNhFBnD5ZQx8/X2cDSpM/YHTR3+Nfq5NBqaI
iZwB7TUjkti2Mwld/0m5
=EQBT
-----END PGP SIGNATURE-----

Re: Can't get notmuch to find mail

[Michael Hudson]

On Thu, 12 Aug 2010 16:29:59 -0500, Stephen Paul Weber <singpolyma@singpolyma.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> Somebody claiming to be Michal Sojka wrote:
> > On 08/12/2010 05:33 PM, Stephen Paul Weber wrote:
> > >Somebody claiming to be Michal Sojka wrote:
> > >>Could you run notmuch new through strace and/or apply the patch bellow
> > >>and send the output?
> > >
> > >TRACE file attached
> > 
> > Hmm, I see nothing suspicious in the TRACE (besides that it does not
> > index your mails). Are you really using version 0.1 (notmuch
> > --version)?
> 
> Hmm...
> My manpage said 0.1 on it, so I didn't question, but my package manager says
> 0.0+201001211401
> 
> I guess I should upgrade.  I suppose the only official way to do that (until
> the Ubuntu package updates) is to build from source?

FWIW, I've got a more recent build in my ppa (add-apt-repository
ppa:mwhudson).  Up to you whether to trust a PPA of course :-)

Cheers,
mwh