indexing mail?

indexing mail?

[Arvid Picciani]

Hi,
how do you add new mails to the index?
manual says "notmuch new" should be enough, but it simply says
"No new mail."

on the first run (when no .notmuch is there yet), it finds some 
messages, but doesn't index them either.

$ notmuch search tag:inbox
$

$ notmuch search s
$


-- 
Arvid
Asgaard Technologies

Re: indexing mail?

[Arvid Picciani]

On Thu, 14 Jan 2010 09:38:00 +0100, Arvid Picciani <aep@exys.org> wrote:

> on the first run (when no .notmuch is there yet), it finds some 
> messages, but doesn't index them either.

the offending commit is 2c4555f1a56602ff1dd55a63699810522ba4d91e

from readdir (3):

     "Currently, only some file systems (among them: Btrfs, ext2, ext3,
     and ext4) have full  support  returning  the  file
       type in d_type.  All applications must properly handle a return
     of DT_UNKNOWN."


thanks "kanru" for helping on irc.

Re: indexing mail?

[Adrian Perez de Castro]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

On Thu, 14 Jan 2010 18:13:53 +0100, Arvid wrote:

> On Thu, 14 Jan 2010 09:38:00 +0100, Arvid Picciani <aep@exys.org> wrote:
> 
> > on the first run (when no .notmuch is there yet), it finds some 
> > messages, but doesn't index them either.

Yuk! I logged-in via Gmail's web interface and found that I have some new
messages which are not being picked by Notmuch.

> the offending commit is 2c4555f1a56602ff1dd55a63699810522ba4d91e
> 
> from readdir (3):
> 
>      "Currently, only some file systems (among them: Btrfs, ext2, ext3,
>      and ext4) have full  support  returning  the  file
>        type in d_type.  All applications must properly handle a return
>      of DT_UNKNOWN."

I am using XFS, which always returns DT_UNKNOWN. Taking into account that
there is a good deal of people using filesystems other than the ones you
mention, and that other non-linux filesystems may also return DT_UNKNOWN,
in my opinion there should be a fall-back. I will try to post a patch
Anytime Soon™.

Also, I have the feeling that the "d_type" field from "struct dirent" may
not be available in some OSes because it is a BSD extension.

Cheers,

-- 
Adrian Perez de Castro <aperez@igalia.com>
Igalia - Free Software Engineering

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: indexing mail?

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 2429 bytes --]

On Thu, 14 Jan 2010 18:38:54 +0100, Adrian Perez de Castro <aperez@igalia.com> wrote:
> > the offending commit is 2c4555f1a56602ff1dd55a63699810522ba4d91e
> > 
> > from readdir (3):
> > 
> >      "Currently, only some file systems (among them: Btrfs, ext2, ext3,
> >      and ext4) have full  support  returning  the  file
> >        type in d_type.  All applications must properly handle a return
> >      of DT_UNKNOWN."

Yes. The broken code was my mistake. I clearly didn't read the above
warning closely enough. Sorry about that!

> I am using XFS, which always returns DT_UNKNOWN. Taking into account that
> there is a good deal of people using filesystems other than the ones you
> mention, and that other non-linux filesystems may also return DT_UNKNOWN,
> in my opinion there should be a fall-back. I will try to post a patch
> Anytime Soon™.

We definitely want the fallback. I can attempt to code it, but I don't
have ready access to an afflicted filesystem, so I'd need help testing
anyway.

I'd love to see a patch for this bug soon. Be sure to CC me when the
patch is sent and that will help me commit it sooner.

> Also, I have the feeling that the "d_type" field from "struct dirent" may
> not be available in some OSes because it is a BSD extension.

I'm generally quite bad at determining whether functionality I'm using
in my software is non-portable. As proven in this case, even when the
man page tells me something is not portable I don't always notice, (and
often, the man pages aren't even that useful).

Beyond that, even if something is *known* to be theoretically
non-portable, it can be a waste of time to code compatibility paths that
nobody will be running in practice.

So I've basically gotten to the point where I just code for what works
on my system, (not out of disregard for what other people run---just
that it's impossible for me to know what subset of functionality is
actually relevant). Then, at the same time, I'm quite happy to accept
code to improve the portability when people note that things are broken
on other systems.

See the git history and email archives for examples of how we fixed
strndup and getline portability problems.

I know that "wait for people to notice it's broken" isn't the nicest
thing we could do with our code. But I don't really know a much better
way. I'm happy to entertain suggestions here.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: indexing mail?

[Olly Betts]

On 2010-01-14, Carl Worth wrote:
> On Thu, 14 Jan 2010 18:38:54 +0100, Adrian Perez de Castro <aperez@igalia.com> wrote:
>> I am using XFS, which always returns DT_UNKNOWN. Taking into account that
>> there is a good deal of people using filesystems other than the ones you
>> mention, and that other non-linux filesystems may also return DT_UNKNOWN,
>> in my opinion there should be a fall-back. I will try to post a patch
>> Anytime Soon=E2=84=A2.
>
> We definitely want the fallback. I can attempt to code it, but I don't
> have ready access to an afflicted filesystem, so I'd need help testing
> anyway.
>
> I'd love to see a patch for this bug soon. Be sure to CC me when the
> patch is sent and that will help me commit it sooner.

Not a full patch, but I already posted what this code should look like
to handle both systems without d_type, and those which return DT_UNKNOWN:

http://article.gmane.org/gmane.mail.notmuch.general/1044

Cheers,
    Olly

Re: indexing mail?

[Dirk-Jan C. Binnema]

>>>>> "Olly" == Olly Betts <olly@survex.com> writes:

    Olly> On 2010-01-14, Carl Worth wrote:
    >> On Thu, 14 Jan 2010 18:38:54 +0100, Adrian Perez de Castro <aperez@igalia.com> wrote:
    >>> I am using XFS, which always returns DT_UNKNOWN. Taking into account that
    >>> there is a good deal of people using filesystems other than the ones you
    >>> mention, and that other non-linux filesystems may also return DT_UNKNOWN,
    >>> in my opinion there should be a fall-back. I will try to post a patch
    >>> Anytime Soon=E2=84=A2.
    >> 
    >> We definitely want the fallback. I can attempt to code it, but I don't
    >> have ready access to an afflicted filesystem, so I'd need help testing
    >> anyway.
    >> 
    >> I'd love to see a patch for this bug soon. Be sure to CC me when the
    >> patch is sent and that will help me commit it sooner.

    Olly> Not a full patch, but I already posted what this code should look like
    Olly> to handle both systems without d_type, and those which return DT_UNKNOWN:

    Olly> http://article.gmane.org/gmane.mail.notmuch.general/1044

I take a slighly different approach in mu:

/* if the file system does not support entry->d_type, we add it ourselves
 * this is slower (extra stat) but at least it works
 */
static gboolean
_set_dtype (const char* path, struct dirent *entry)
{
	struct stat statbuf;
	char fullpath[4096];
	
	snprintf (fullpath, sizeof(fullpath), "%s%c%s",
		  path, G_DIR_SEPARATOR, entry->d_name);

	if (stat (fullpath, &statbuf) != 0) {
		g_warning ("stat failed on %s: %s", fullpath,
			   strerror(errno));
		return FALSE;
	}

	/* we only care about dirs, regular files and links */
	if (S_ISREG (statbuf.st_mode))
		entry->d_type = DT_REG;
	else if (S_ISDIR (statbuf.st_mode))
		entry->d_type = DT_DIR;
	else if (S_ISLNK (statbuf.st_mode))
		entry->d_type = DT_LNK;
	
	return TRUE;
}


and then in some other places:

/* handle FSs that don't support entry->d_type */
if (entry->d_type == DT_UNKNOWN) 
	_set_dtype (dirname, entry);

	
Note, that is untested as of yet.

Best wishes,
Dirk.

-- 
Dirk-Jan C. Binnema                  Helsinki, Finland
e:djcb@djcbsoftware.nl           w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

Re: indexing mail?

[Arvid Picciani]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

On Fri, 15 Jan 2010 08:47:10 +0200, Dirk-Jan C. Binnema <djcb.bulk@gmail.com> wrote:

> I take a slighly different approach in mu:


works for me, thanks Dirk.  patch against HEAD is attached ( i hope. i'm
new to emacs mail :D )


It takes around half an hour for my 60K mail on reiserfs, but it did take 10 minutes
already on ext4. I suggest having a different approach to feed new mail
in, such as:

for i in (fetchmail)
do
    notmuch new < $i
done

[-- Attachment #2: notmuch_scandir_stat.patch --]
[-- Type: application/octet-stream, Size: 1440 bytes --]

diff --git a/notmuch-new.c b/notmuch-new.c
index 490101d..d8ac4ba 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -119,6 +119,37 @@ is_maildir (struct dirent **entries, int count)
     return 0;
 }
 
+/*
+  add entry->d_type
+ */
+
+ static gboolean
+ _set_dtype (const char* path, struct dirent *entry)
+ {
+     struct stat statbuf;
+     char fullpath[4096];
+
+     snprintf (fullpath, sizeof(fullpath), "%s%c%s",
+           path, G_DIR_SEPARATOR, entry->d_name);
+
+     if (stat (fullpath, &statbuf) != 0) {
+         g_warning ("stat failed on %s: %s", fullpath,
+                strerror(errno));
+         return FALSE;
+     }
+
+     /* we only care about dirs, regular files and links */
+     if (S_ISREG (statbuf.st_mode))
+         entry->d_type = DT_REG;
+     else if (S_ISDIR (statbuf.st_mode))
+         entry->d_type = DT_DIR;
+     else if (S_ISLNK (statbuf.st_mode))
+         entry->d_type = DT_LNK;
+
+     return TRUE;
+ }
+
+
 /* Examine 'path' recursively as follows:
  *
  *   o Ask the filesystem for the mtime of 'path' (fs_mtime)
@@ -186,6 +217,11 @@ add_files_recursive (notmuch_database_t *notmuch,
 	    break;
 
         entry= namelist[i++];
+        /* handle FSs that don't support entry->d_type */
+        if (entry->d_type == DT_UNKNOWN)
+            _set_dtype (path, entry);
+
+
 
 	/* If this directory hasn't been modified since the last
 	 * add_files, then we only need to look further for

Re: indexing mail?

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 892 bytes --]

On Fri, 15 Jan 2010 08:27:50 +0100, Arvid Picciani <aep@exys.org> wrote:
> works for me, thanks Dirk.  patch against HEAD is attached ( i hope. i'm
> new to emacs mail :D )

Hi Arvid,

Your original patch was sent as "application/octet-stream" which made it
awkward to read, (I would have to manually save it rather than just
being able to read it within emacs with notmuch).

But I've since pushed a separate patch to fix this bug. Please give it a
try and let me know what you think.

> It takes around half an hour for my 60K mail on reiserfs, but it did take 10 minutes
> already on ext4.

What operation is taking that long?

> I suggest having a different approach to feed new mail in, such as:
> 
> for i in (fetchmail)
> do
>     notmuch new < $i
> done

I'm still not sure what is slow for you, but I'm also not sure how the
above would help.

-Carl



[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: indexing mail?

[Jesse Rosenthal]

On Sat, 23 Jan 2010 19:09:35 +1300, Carl Worth <cworth@cworth.org> wrote:
> But I've since pushed a separate patch to fix this bug. Please give it a
> try and let me know what you think.

I just gave it a try, and building failed because of a seeming
misspelling on line 285 (`DT_UKNOWN'), from commit
344c48a47de23cc63f1885d850b82359d1a34064 . Fixing the misspelling fixed
the build.

Thanks, as always, for all your work on this.

Best,
Jesse

Re: indexing mail?

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

On Sat, 23 Jan 2010 01:28:38 -0500, Jesse Rosenthal <jrosenthal@jhu.edu> wrote:
> I just gave it a try, and building failed because of a seeming
> misspelling on line 285 (`DT_UKNOWN'), from commit
> 344c48a47de23cc63f1885d850b82359d1a34064 . Fixing the misspelling fixed
> the build.

That's embarrassing. I've pushed a fix now (and I even compiled it this
time before pushing it!).

> Thanks, as always, for all your work on this.

No problem at all. It's my pleasure.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: indexing mail?

[Arvid Picciani]

On 01/23/2010 07:09 AM, Carl Worth wrote:

> Your original patch was sent as "application/octet-stream" which made it
> awkward to read, (I would have to manually save it rather than just
> being able to read it within emacs with notmuch).

uum yeah thanks. I'll try to figure out how this works.

> But I've since pushed a separate patch to fix this bug. Please give it a
> try and let me know what you think.

will do later.

>> It takes around half an hour for my 60K mail on reiserfs, but it did take 10 minutes
>> already on ext4.
>
> What operation is taking that long?

notmuch new, rescanning my entire 60k mails on every single new message 
i get.

>> I suggest having a different approach to feed new mail in, such as:
>>
>> for i in (fetchmail)
>> do
>>      notmuch new<  $i
>> done
>
> I'm still not sure what is slow for you,

scanning 60k mails. :D  That's not fixable, other then by not doing that.

but I'm also not sure how the above would help.

It doesn't scan all 60K individually but only the single new one.



-- 
Arvid
Asgaard Technologies

Re: indexing mail?

[Arvid Picciani]

On 01/23/2010 03:29 PM, Arvid Picciani wrote:
>>> done
>>
>> I'm still not sure what is slow for you,
>
> scanning 60k mails. :D That's not fixable, other then by not doing that.
>
> but I'm also not sure how the above would help.
>
> It doesn't scan all 60K individually but only the single new one.


Thunderfuck is a pain. it sent the mail with a different quoting then 
showing it, again. Desperatly need to get notmuch working :(

-- 
Arvid
Asgaard Technologies

Re: indexing mail?

[Olly Betts]

On 2010-01-15, Dirk-Jan C  Binnema wrote:
>>>>>> "Olly" == Olly Betts <olly@survex.com> writes:
>    Olly> Not a full patch, but I already posted what this code should look like
>    Olly> to handle both systems without d_type, and those which return DT_UNKNOWN:
>
>    Olly> http://article.gmane.org/gmane.mail.notmuch.general/1044

> static gboolean
> _set_dtype (const char* path, struct dirent *entry)

Underscore prefixed identifiers are reserved by ISO C at file-scope; using them
yourself is undefined behaviour...

> 	/* we only care about dirs, regular files and links */
> 	if (S_ISREG (statbuf.st_mode))
> 		entry->d_type = DT_REG;
> 	else if (S_ISDIR (statbuf.st_mode))
> 		entry->d_type = DT_DIR;
> 	else if (S_ISLNK (statbuf.st_mode))
> 		entry->d_type = DT_LNK;

This addresses the case where the FS returns DT_UNKNOWN for d_type, but doesn't
deal with the case of platforms where struct dirent has no d_type member - from
the Linux readdir man page:

  The only fields in the dirent structure that are mandated by POSIX.1 are:
  d_name[], of unspecified size, with at most NAME_MAX characters preceding
  the terminating null byte; and (as an XSI extension) d_ino.  The other fields
  are unstandardized, and not present on all systems; see NOTES below for some
  further details.

And in NOTES:

  Other than Linux, the d_type field is available mainly only on BSD systems.

Cheers,
    Olly

Re: indexing mail?

[Dirk-Jan C. Binnema]

Hi Olly,

>>>>> "Olly" == Olly Betts <olly@survex.com> writes:

    Olly> On 2010-01-15, Dirk-Jan C  Binnema wrote:
    >>>>>>> "Olly" == Olly Betts <olly@survex.com> writes:
    Olly> Not a full patch, but I already posted what this code should look like
    Olly> to handle both systems without d_type, and those which return DT_UNKNOWN:
    >> 
    Olly> http://article.gmane.org/gmane.mail.notmuch.general/1044

    >> static gboolean
    >> _set_dtype (const char* path, struct dirent *entry)

    Olly> Underscore prefixed identifiers are reserved by ISO C at file-scope;
    Olly> using them yourself is undefined behaviour...

Ah, thanks for reminding, I thought it was __ and _C (capital), but you are
right:

,---- (7.1.3 Reserved identifiers)
|  All identifiers that begin with an underscore and either an uppercase letter
|  or another underscore are always reserved for any use.
| 
| — All identifiers that begin with an underscore are always reserved for use as
| identifiers with file scope in both the ordinary and tag name spaces.
`----

    >> /* we only care about dirs, regular files and links */
    >> if (S_ISREG (statbuf.st_mode))
    entry-> d_type = DT_REG;
    >> else if (S_ISDIR (statbuf.st_mode))
    entry-> d_type = DT_DIR;
    >> else if (S_ISLNK (statbuf.st_mode))
    entry-> d_type = DT_LNK;

    Olly> This addresses the case where the FS returns DT_UNKNOWN for d_type,
    Olly> but doesn't deal with the case of platforms where struct dirent has
    Olly> no d_type member - from the Linux readdir man page:

    Olly>   The only fields in the dirent structure that are mandated by
    Olly>   POSIX.1 are: d_name[], of unspecified size, with at most NAME_MAX
    Olly>   characters preceding the terminating null byte; and (as an XSI
    Olly>   extension) d_ino.  The other fields are unstandardized, and not
    Olly>   present on all systems; see NOTES below for some further details.

    Olly> And in NOTES:

    Olly>   Other than Linux, the d_type field is available mainly only on BSD
    Olly>   systems.

Yes, my patch could me generalized a bit more, just like your patch could not
hardcode the '/'-separator :)

In practice though, what Unices in use today do not support d_type?

Best wishes,
Dirk,

-- 
Dirk-Jan C. Binnema                  Helsinki, Finland
e:djcb@djcbsoftware.nl           w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

Re: indexing mail?

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]

On Fri, 15 Jan 2010 21:57:32 +0200, Dirk-Jan C. Binnema <djcb.bulk@gmail.com> wrote:
>     Olly> Underscore prefixed identifiers are reserved by ISO C at file-scope;
>     Olly> using them yourself is undefined behaviour...
> 
> Ah, thanks for reminding, I thought it was __ and _C (capital), but you are
> right:
> 
> ,---- (7.1.3 Reserved identifiers)
> |  All identifiers that begin with an underscore and either an uppercase letter
> |  or another underscore are always reserved for any use.
> | 
> | — All identifiers that begin with an underscore are always reserved for use as
> | identifiers with file scope in both the ordinary and tag name spaces.
> `----

But please don't be too strict about this.

Please feel very free to use any identifier with a _notmuch prefix.

And really, feel free to use just about any underscore-prefixed
identifier that you want that doesn't clash with anything on your
system.

Then if we do identify an actual clash somewhere then we can fix it.

I think it was stupid of Posix to steal _ and a reserved prefix, and I
really don't have a problem ignoring that. This is like I described in a
recent mail---trying to prevent all portability problems just isn't
worth the effort. It's much easier to fix problems that actually occur
in practice.

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: indexing mail?

[Olly Betts]

On 2010-01-15, Dirk-Jan C  Binnema wrote:
>    Olly>   Other than Linux, the d_type field is available mainly only on BSD
>    Olly>   systems.
>
> Yes, my patch could me generalized a bit more, just like your patch could not
> hardcode the '/'-separator :)

Well, '/' works as a directory separator for all Unix systems and also
for Microsoft Windows at this level.  Is there a system which doesn't
accept '/' in this place which is still relevant?

Personally I don't see the point in aiming for portability to systems like
Mac OS 9, RISC OS, and VMS in 2010...

> In practice though, what Unices in use today do not support d_type?

Solaris 10 doesn't for starters.  I don't have ready access to the other
non-Linux, non-BSD Unix flavours to check those right now.

Cheers,
    Olly

Re: indexing mail?

[Dirk-Jan C. Binnema]

Hi Olly,

>>>>> "OB" == Olly Betts <olly@survex.com> writes:

    OB> On 2010-01-15, Dirk-Jan C  Binnema wrote:
    Olly> Other than Linux, the d_type field is available mainly only on BSD
    Olly> systems.
    >> 
    >> Yes, my patch could me generalized a bit more, just like your patch could not
    >> hardcode the '/'-separator :)

    OB> Well, '/' works as a directory separator for all Unix systems and also
    OB> for Microsoft Windows at this level.  Is there a system which doesn't
    OB> accept '/' in this place which is still relevant?

Note the ':)' This was just point that it's very hard to write software that
does not include *some* degree of platform-preference.

Anyway (going slightly OT here) regarding the '\', I think the one system in
wide use that insists on '\' would be Symbian. Porting Unix stuff is usually
not too hard with the Symbian's compatiblity-libc. (But maybe the
dir-separator is still the least of your concerns).

>> In practice though, what Unices in use today do not support d_type?

    OB> Solaris 10 doesn't for starters.  I don't have ready access to the other
    OB> non-Linux, non-BSD Unix flavours to check those right now.

Ah, thanks. Solaris 10 is fairly important indeed.  Another one is Cygwin (it
added it fairly recently), Linux before 2.6.4. I've added a configure check
for now. Performance must be quite terrible with a working d_type.

Best wishes,
Dirk.

-- 
Dirk-Jan C. Binnema                  Helsinki, Finland
e:djcb@djcbsoftware.nl           w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C