Notmuch doesn't index new mails

Notmuch doesn't index new mails

[Tassilo Horn]

Hi all,

after setting up notmuch, initially indexing all my mail, and removing
the inbox and unread tags thereafter, now I recognize that notmuch
doesn't index new mail.

--8<---------------cut here---------------start------------->8---
% notmuch new --verbose
No new mail---and that's not much.
--8<---------------cut here---------------end--------------->8---

I did check that there's really new mail.

In the man page, it is mentioned, that notmuch skips directories that
are read-only.  So now all my folders are "drwxrwxr-x mail mail", and I
am in the mail group.

But still, notmuch doesn't index new mails.

I assumed that the mail files also have to be writable.  So I changed
them to "-rw-rw-r-- mail mail", but still no luck.

Any ideas what might be the problem?

Bye,
Tassilo

Notmuch doesn't index new mails when mail location contains symlinks (was: Notmuch doesn't index new mails)

[Tassilo Horn]

Tassilo Horn <tassilo@member.fsf.org> writes:

Hi all,

I've investigated a bit further.

> [notmuch doesn't index new mails although all directories and files
> are readable and writable.]

In my config, I have:

--8<---------------cut here---------------start------------->8---
[database]
path=/home/horn/Mail/Dovecot
--8<---------------cut here---------------end--------------->8---

In that directory, there are two symlinks pointing to the real mail
location in /var/spool/mail:

--8<---------------cut here---------------start------------->8---
[horn@localhost][~/Mail/Dovecot][0][5213]
[:)] % ll
total 0
lrwxrwxrwx 1 horn horn 34 Nov 23 14:49 fastmail -> /var/spool/mail/fastmail/mailboxes
lrwxrwxrwx 1 horn horn 29 Nov 23 14:49 uni -> /var/spool/mail/uni/mailboxes
--8<---------------cut here---------------end--------------->8---

Whenever I delete those symlinks and created them anew, the new mails
get indexed with the next "notmuch new".  Of course, I could create a
script that does exactly that, but there should be a better way, right?

Bye,
Tassilo

Re: Notmuch doesn't index new mails when mail location contains symlinks

[Mikhail Gusarov]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]


Twas brillig at 14:55:40 23.11.2009 UTC+01 when tassilo@member.fsf.org did gyre and gimble:

 TH> Whenever I delete those symlinks and created them anew, the new
 TH> mails get indexed with the next "notmuch new".  Of course, I could
 TH> create a script that does exactly that, but there should be a
 TH> better way, right?

Probably mail does not get indexed due to mtime checks. Please try
whether touch'ing directory with mailboxes makes it work.

-- 
  http://fossarchy.blogspot.com/

[-- Attachment #2: Type: application/pgp-signature, Size: 834 bytes --]

Re: Notmuch doesn't index new mails when mail location contains symlinks

[Tassilo Horn]

Mikhail Gusarov <dottedmag@dottedmag.net> writes:

Hi Mikhail,

>  TH> Whenever I delete those symlinks and created them anew, the new
>  TH> mails get indexed with the next "notmuch new".  Of course, I could
>  TH> create a script that does exactly that, but there should be a
>  TH> better way, right?
>
> Probably mail does not get indexed due to mtime checks. Please try
> whether touch'ing directory with mailboxes makes it work.

No, it seems that doesn't help either.

First, I only touched the two symlinks.  This didn't help.  Then I used
"find . -type d | xargs touch" to touch all directories inside the
directories the symlinks point to.  But still no luck.  Finally, I
deleted the symlinks and created them anew, and then it indexed the 12
new mails that arrived in the meantime.

Bye,
Tassilo

Re: Notmuch doesn't index new mails when mail location contains symlinks

[Tassilo Horn]

Tassilo Horn <tassilo@member.fsf.org> writes:

Hi Mikhail,

>>  TH> Whenever I delete those symlinks and created them anew, the new
>>  TH> mails get indexed with the next "notmuch new".  Of course, I could
>>  TH> create a script that does exactly that, but there should be a
>>  TH> better way, right?
>>
>> Probably mail does not get indexed due to mtime checks. Please try
>> whether touch'ing directory with mailboxes makes it work.
>
> No, it seems that doesn't help either.

Ah, I'm stupid!  I don't have to touch the symlinks or the directories
inside the locations the symlinks point to, but instead I have to touch
the top-level directory where the symlinks are contained in.  Then it
works as expected, AFAICT.

Thanks a lot,
Tassilo

Re: Notmuch doesn't index new mails when mail location contains symlinks

[Mikhail Gusarov]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]


Twas brillig at 16:01:41 23.11.2009 UTC+01 when tassilo@member.fsf.org did gyre and gimble:

 >>> Probably mail does not get indexed due to mtime checks. Please try
 >>> whether touch'ing directory with mailboxes makes it work.
 >> No, it seems that doesn't help either.
 TH> Ah, I'm stupid!  I don't have to touch the symlinks or the
 TH> directories inside the locations the symlinks point to, but instead
 TH> I have to touch the top-level directory where the symlinks are
 TH> contained in.  Then it works as expected, AFAICT.

Really odd.

-- 
  http://fossarchy.blogspot.com/

[-- Attachment #2: Type: application/pgp-signature, Size: 834 bytes --]

Re: Notmuch doesn't index new mails when mail location contains symlinks

[Jed Brown]

On Mon, 23 Nov 2009 16:01:41 +0100, Tassilo Horn <tassilo@member.fsf.org> wrote:
> Tassilo Horn <tassilo@member.fsf.org> writes:

> I don't have to touch the symlinks or the directories inside the
> locations the symlinks point to, but instead I have to touch the
> top-level directory where the symlinks are contained in.

Ah, it's slightly more subtle.  Notmuch correctly uses stat() instead of
lstat() to check whether the link target changed.  However, mtime for
your mail root directory (containing the symlinks) does not get updated
when the target of the symlinks is updated.  I think the only way to fix
this is to replace the current search (which skips a directory as soon
as it's mtime is older than the database) with one that enters all
directories so that symlinks are actually followed.

Jed

Re: Notmuch doesn't index new mails when mail location contains symlinks

[Jed Brown]

On Mon, 23 Nov 2009 15:43:41 +0100, Tassilo Horn <tassilo@member.fsf.org> wrote:
> First, I only touched the two symlinks.

Unfortunately, this actually touched the file pointed to by the symlink,
if you stat the symlink you will see that mtime did not change.

> This didn't help.  Then I used
> "find . -type d | xargs touch" to touch all directories inside the
> directories the symlinks point to.

Actually, this would not have followed the symlinks so it does the same
thing as before.  I think it is actually hard (or not possible) to
change mtime on symlinks under Linux.

> But still no luck.  Finally, I deleted the symlinks and created them
> anew, and then it indexed the 12 new mails that arrived in the
> meantime.

If /var is on the same filesystem, you could use hard links instead of
symlinks.  Otherwise I would just add the appropriate ln -sf in the hook
before notmuch new.

The real solution is for notmuch to check mtime of whatever the
symlink's target.

Jed

Re: Notmuch doesn't index new mails when mail location contains symlinks

[Mikhail Gusarov]

[-- Attachment #1: Type: text/plain, Size: 285 bytes --]


Twas brillig at 16:03:00 23.11.2009 UTC+01 when jed@59A2.org did gyre and gimble:

 JB> The real solution is for notmuch to check mtime of whatever the
 JB> symlink's target.

It does exactly this AFACT, stat() everywhere, not lstat().

-- 
  http://fossarchy.blogspot.com/

[-- Attachment #2: Type: application/pgp-signature, Size: 834 bytes --]

Re: Notmuch doesn't index new mails when mail location contains symlinks

[Carl Worth]

On Mon, 23 Nov 2009 16:03:00 +0100, Jed Brown <jed@59A2.org> wrote:
> > But still no luck.  Finally, I deleted the symlinks and created them
> > anew, and then it indexed the 12 new mails that arrived in the
> > meantime.

OK. So that's definitely a bug we need to fix.

> The real solution is for notmuch to check mtime of whatever the
> symlink's target.

I'm a little confused here. Notmuch only uses stat, so it should be
looking at the target's mtime already. It actually takes special effort
(via lstat) to get at the mtime of the link itself.

So why aren't things just working?

-Carl

Re: Notmuch doesn't index new mails when mail location contains symlinks

[Jed Brown]

On Thu, 26 Nov 2009 10:34:12 -0800, Carl Worth <cworth@cworth.org> wrote:
> I'm a little confused here. Notmuch only uses stat, so it should be
> looking at the target's mtime already. It actually takes special effort
> (via lstat) to get at the mtime of the link itself.
> 
> So why aren't things just working?

Because mtime doesn't change on the directory *above* the symlinks.

I think.

Jed

Re: Notmuch doesn't index new mails when mail location contains symlinks

[Carl Worth]

On Thu, 26 Nov 2009 19:39:01 +0100, Jed Brown <jed@59A2.org> wrote:
> On Thu, 26 Nov 2009 10:34:12 -0800, Carl Worth <cworth@cworth.org> wrote:
> > I'm a little confused here. Notmuch only uses stat, so it should be
> > looking at the target's mtime already. It actually takes special effort
> > (via lstat) to get at the mtime of the link itself.
> > 
> > So why aren't things just working?
> 
> Because mtime doesn't change on the directory *above* the symlinks.
> 
> I think.

Then I'm still being really dense here. The non-propagation of mtime is
the with actual directories. And the code is trying to do the right
thing for that.

So I still don't see what's being different with the symlinks. I should
probably stop exposing ignorance here on the list, and just sit down
with symlinks in a mail store and see what's happening.

-Carl

Re: Notmuch doesn't index new mails when mail location contains symlinks

[Jed Brown]

On Thu, 26 Nov 2009 13:14:22 -0800, Carl Worth <cworth@cworth.org> wrote:

> Then I'm still being really dense here. The non-propagation of mtime is
> the with actual directories. And the code is trying to do the right
> thing for that.

The stat() is correct, it's the check for the d_type field coming out of scandir().

20:08 [   jedbrown  ] cworth: I think there are two problems with symlinks.
20:08 [   jedbrown  ] cworth: First, notmuch-new.c:152 checks path_mtime which is mtime for the current directory.
20:09 [   jedbrown  ] cworth: Additionally, won't entry->d_type == DT_LNK for a symlink?
20:14 [     aneesh  ] jedbrown,  i already have a patch posted for that
20:15 [     aneesh  ] 1259125104-18785-1-git-send-email-aneesh.kumar@linux.vnet.ibm.com


Jed