Bug with commit 2e96464f9705be4ec772280cad71a6c9d5831e6f

Bug with commit 2e96464f9705be4ec772280cad71a6c9d5831e6f

[racin]

Hello,

I just updated notmuch and now notmuch new cannot update my mail anymore... It tells me that there are
700 files found, but tells that there's no new mail.

I did a git bisect, which tells me the first bad commit is commit 2e96464f9705be4ec772280cad71a6c9d5831e6f.

I did not try to use the new xapian database or to update xapian; maybe this is the problem.

I tested with several tools to get mail in the maildir format, including mb2md and getmail, and I always get the problem.

I will try to investigate a bit more.


Matthieu

Re: Bug with commit 2e96464f9705be4ec772280cad71a6c9d5831e6f

[Ali Polatel]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

racin@free.fr yazmış:
> Hello,
> 
> I just updated notmuch and now notmuch new cannot update my mail anymore... It tells me that there are
> 700 files found, but tells that there's no new mail.
> 
> I did a git bisect, which tells me the first bad commit is commit 2e96464f9705be4ec772280cad71a6c9d5831e6f.
> 
> I did not try to use the new xapian database or to update xapian; maybe this is the problem.
> 
> I tested with several tools to get mail in the maildir format, including mb2md and getmail, and I always get the problem.
Same problem here, I tried upgrading xapian to xapian-core-1.1.3_svn13824, the problem persists.
Here's what happens here:
3074 alip@harikalardiyari> rm -fr .maildir/.notmuch
3075 alip@harikalardiyari> notmuch new
Found 210302 total files (that's not much mail).
No new mail.
3076 alip@harikalardiyari> notmuch search from:alip@exherbo.org
3077 alip@harikalardiyari>

> 
> I will try to investigate a bit more.
> 
> 
> Matthieu

-- 
Regards,
Ali Polatel

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Bug with commit 2e96464f9705be4ec772280cad71a6c9d5831e6f

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1785 bytes --]

On Mon, 11 Jan 2010 00:29:05 +0100 (CET), racin@free.fr wrote:
> I just updated notmuch and now notmuch new cannot update my mail anymore... It tells me that there are
> 700 files found, but tells that there's no new mail.

Hi Mathieu,

Thanks for testing the new notmuch. I appreciate that. And I'm sorry to
hear that you're weren't pleasantly rewarded for your efforts. I tried
as well as I could to avoid bugs slipping through, but I also knew I
couldn't test for everything.

> I did a git bisect, which tells me the first bad commit is commit
> 2e96464f9705be4ec772280cad71a6c9d5831e6f.

Can you tell me what you were bisecting? Were you attempting the
database upgrade over again each time? Or just testing whether you could
update a post-upgraded database each time?

That commit looks remarkably benign. It doesn't really change anything
about how the database is interpreted or upgraded, (but merely adds a
list to defer the printing of a few filenames). Could you perform a test
to confirm the results of bisection---such as manually applying and
reverting this patch to the immediately preceding version and check
whether the behavior changes?

> I did not try to use the new xapian database or to update xapian;
> maybe this is the problem.

It shouldn't be. The new notmuch is intended to work just fin with an
old Xapian. And I *think* that I tested that at least a few times.

> I will try to investigate a bit more.

Thanks. I'll look forward to any further information you can provide.

And if it would be practical for you to share your email database with
me (privately, of course) so that I can investigate, I would be glad to
do that. Obviously, I'll understand if that's not possible. Just contact
me off-list if you'd like to arrange something like that.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Bug with commit 2e96464f9705be4ec772280cad71a6c9d5831e6f

[Ali Polatel]

[-- Attachment #1: Type: text/plain, Size: 1227 bytes --]

Ali Polatel yazmış:
> racin@free.fr yazmış:
> > Hello,
> > 
> > I just updated notmuch and now notmuch new cannot update my mail anymore... It tells me that there are
> > 700 files found, but tells that there's no new mail.
> > 
> > I did a git bisect, which tells me the first bad commit is commit 2e96464f9705be4ec772280cad71a6c9d5831e6f.
> > 
> > I did not try to use the new xapian database or to update xapian; maybe this is the problem.
> > 
> > I tested with several tools to get mail in the maildir format, including mb2md and getmail, and I always get the problem.
> Same problem here, I tried upgrading xapian to xapian-core-1.1.3_svn13824, the problem persists.
> Here's what happens here:
> 3074 alip@harikalardiyari> rm -fr .maildir/.notmuch
> 3075 alip@harikalardiyari> notmuch new
> Found 210302 total files (that's not much mail).
> No new mail.
> 3076 alip@harikalardiyari> notmuch search from:alip@exherbo.org
> 3077 alip@harikalardiyari>
> 

After doing git bisect which gave random results and testing notmuch
using cworth's notmuch-test script on another file system I figured out
my problem was due to file system corruption.
Sorry for the noise...

-- 
Regards,
Ali Polatel

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: Bug with commit 2e96464f9705be4ec772280cad71a6c9d5831e6f

[racin]

[-- Attachment #1: Type: text/plain, Size: 2597 bytes --]

I still confirm the bug.

The problem is due to relying on non-standardized fields of directory entries (i.e. d_type), which don't behave the
same on reiserfs than on ext2 (I use reiserfs). 

The following ugly patch "solves" my problem.

diff --git a/notmuch-new.c b/notmuch-new.c
index b740ee2..42d9f89 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -359,7 +359,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 
            if (! S_ISREG (st.st_mode))
                continue;
-       } else if (entry->d_type != DT_REG) {
+       } else if (entry->d_type != DT_REG && entry->d_type != 0) {
            continue;
        }

With this patch, mails are added by the database, but some directories also have d_type set to 0
and can slip through.

It seems that the only portable way to test for directory types is stat, which is expensive... I don't known
what's the best way to go.

Matthieu


----- Mail Original -----
De: "Ali Polatel" <alip@exherbo.org>
À: notmuch@notmuchmail.org
Envoyé: Mercredi 13 Janvier 2010 11h21:52 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne
Objet: Re: [notmuch] Bug with commit 2e96464f9705be4ec772280cad71a6c9d5831e6f

Ali Polatel yazmış:
> racin@free.fr yazmış:
> > Hello,
> > 
> > I just updated notmuch and now notmuch new cannot update my mail anymore... It tells me that there are
> > 700 files found, but tells that there's no new mail.
> > 
> > I did a git bisect, which tells me the first bad commit is commit 2e96464f9705be4ec772280cad71a6c9d5831e6f.
> > 
> > I did not try to use the new xapian database or to update xapian; maybe this is the problem.
> > 
> > I tested with several tools to get mail in the maildir format, including mb2md and getmail, and I always get the problem.
> Same problem here, I tried upgrading xapian to xapian-core-1.1.3_svn13824, the problem persists.
> Here's what happens here:
> 3074 alip@harikalardiyari> rm -fr .maildir/.notmuch
> 3075 alip@harikalardiyari> notmuch new
> Found 210302 total files (that's not much mail).
> No new mail.
> 3076 alip@harikalardiyari> notmuch search from:alip@exherbo.org
> 3077 alip@harikalardiyari>
> 

After doing git bisect which gave random results and testing notmuch
using cworth's notmuch-test script on another file system I figured out
my problem was due to file system corruption.
Sorry for the noise...

-- 
Regards,
Ali Polatel

_______________________________________________
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[-- Attachment #2: patch --]
[-- Type: application/octet-stream, Size: 1076 bytes --]

diff --git a/notmuch-new.c b/notmuch-new.c
index b740ee2..a972390 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -153,7 +153,7 @@ _entries_resemble_maildir (struct dirent **entries, int count)
     int i, found = 0;
 
     for (i = 0; i < count; i++) {
-	if (entries[i]->d_type != DT_DIR)
+	if (entries[i]->d_type != DT_DIR || entries[i]->d_type != DT_UNKNOWN)
 	    continue;
 
 	if (strcmp(entries[i]->d_name, "new") == 0 ||
@@ -273,8 +273,8 @@ add_files_recursive (notmuch_database_t *notmuch,
 
 	entry = fs_entries[i];
 
-	if (entry->d_type != DT_DIR && entry->d_type != DT_LNK)
-	    continue;
+	if (entry->d_type != DT_DIR && entry->d_type != DT_LNK && entry->d_type != DT_UNKNOWN)
+	  continue;
 
 	/* Ignore special directories to avoid infinite recursion.
 	 * Also ignore the .notmuch directory and any "tmp" directory
@@ -359,7 +359,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 
 	    if (! S_ISREG (st.st_mode))
 		continue;
-	} else if (entry->d_type != DT_REG) {
+	} else if (entry->d_type != DT_REG && entry->d_type != 0) {
 	    continue;
 	}
 

Re: Bug with commit 2e96464f9705be4ec772280cad71a6c9d5831e6f

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 597 bytes --]

On Sat, 16 Jan 2010 02:42:39 +0100 (CET), racin@free.fr wrote:
> I still confirm the bug.
> 
> The problem is due to relying on non-standardized fields of directory entries (i.e. d_type), which don't behave the
> same on reiserfs than on ext2 (I use reiserfs). 
> 
> The following ugly patch "solves" my problem.

I just pushed a cleaner version of this patch from a later post. I hope
it helps and that you can have fun with notmuch!

-Carl

PS. Micah and Martin, Stacy says "you're welcome" for this and any
further notmuch work that happens to get done during this vacation. :-)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Bug with commit 2e96464f9705be4ec772280cad71a6c9d5831e6f

[racin]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

Works with the following patch (replace a DT_UKNOWN with DT_UNKNOWN, else it fails to compile)

Matthieu


----- Mail Original -----
De: "Carl Worth" <cworth@cworth.org>
À: racin@free.fr, "Ali Polatel" <alip@exherbo.org>
Cc: notmuch@notmuchmail.org
Envoyé: Samedi 23 Janvier 2010 07h06:12 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne
Objet: Re: [notmuch] Bug with commit 2e96464f9705be4ec772280cad71a6c9d5831e6f

On Sat, 16 Jan 2010 02:42:39 +0100 (CET), racin@free.fr wrote:
> I still confirm the bug.
> 
> The problem is due to relying on non-standardized fields of directory entries (i.e. d_type), which don't behave the
> same on reiserfs than on ext2 (I use reiserfs). 
> 
> The following ugly patch "solves" my problem.

I just pushed a cleaner version of this patch from a later post. I hope
it helps and that you can have fun with notmuch!

-Carl

PS. Micah and Martin, Stacy says "you're welcome" for this and any
further notmuch work that happens to get done during this vacation. :-)

[-- Attachment #2: patch --]
[-- Type: application/octet-stream, Size: 362 bytes --]

diff --git a/notmuch-new.c b/notmuch-new.c
index 124d38d..f25c71f 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -282,7 +282,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 	 * recursion). */
 	if (entry->d_type != DT_DIR &&
 	    entry->d_type != DT_LNK &&
-	    entry->d_type != DT_UKNOWN)
+	    entry->d_type != DT_UNKNOWN)
 	{
 	    continue;
 	}