muchsync files renames

muchsync files renames

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 363 bytes --]

Hi,

I am testing muchsync-2 and it looks to me that files names across
machines are different.  Moreover when syncing again after
initialization it seems muchsync is working on something.  I have
canceled this and rerun muchsync.  notmuch reported lots of files
renames on server.  What and why it happens?


Kind regards,

-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: muchsync files renames

[David Mazieres]

Amadeusz Żołnowski <aidecoe@aidecoe.name> writes:

> Hi,
>
> I am testing muchsync-2 and it looks to me that files names across
> machines are different.  Moreover when syncing again after
> initialization it seems muchsync is working on something.  I have
> canceled this and rerun muchsync.  notmuch reported lots of files
> renames on server.  What and why it happens?

What muchsync specifically synchronizes for messages in the mapping:

    (directory, SHA-1-hash, link-count)

So if a directory contains two copies of a file on one machine, it will
end up with two copies on the other machine.  However, the file names
themselves are not the same, but rather are created in accordance with
the maildir spec.  (Note SHA-1 wouldn't be my first choice of hash
function, but notmuch already uses this for messages with long message
IDs, so I figured I'd just be consistent with existing practice.)

In terms of what muchsync is working on, you can run it with "-vvvv" on
both sides to get an idea, as in "muchsync -vvvv server -vvvv".  Better
yet, you can just run it on one side with "muchsync -vvvv".  You'll get
a lot of output, so maybe run it inside the script command to save the
output.maybe run it inside the script command to save the output.  If
you have enabled maildir.synchronize_flags, it could be that notmuch is
initially renaming all of your files, in which case muchsync needs to
re-hash them to make sure they haven't changed.

How did you cancel muchsync?  If you send it a single SIGINT or SIGTERM,
it attempts to clean up after itself.  However, upon multiple signals or
other signals, it immediately exits.  Muchsync is conservative about
updating the database, to avoid missing tags or files that have been
changed.  It always updates the notmuch database first, then its own
sqlite database with a version number.  That means if you kill muchsync,
some number of files may get picked up as changed again even though
really they were just copied from a peer.

To mitigate this problem, the muchsync client syncs the database every
10 seconds, so that in theory you should only get 10 seconds of extra
work from killing the client.  However, the server does not sync
periodically, on the assumption that it is more likely to read an EOF
than get killed, although currently it doesn't appear to commit any
pending transactions to the sqlite database upon EOF, which may be an
oversight.

So to summarize:

  * File names are not the same across machine, only file contents and
    directory structure.

  * Give muchsync lots of "-v" options to see what it is doing.

  * Try to avoid killing muchsync.  Doing so is safe, but likely to
    generate extra work in the form of phantom renames or tag changes
    that get synchronized even though they don't need to be.

  * Possibly the server should handle EOF more gracefully and commit any
    pending transactions, or the client should periodically send a
    commit command to the server.

If you think something is wrong, I can help you figure it out, but I
need to know what maildir.synchronize_flags is set to on each replica,
what you mean by "canceled", and roughly what was happening when you
canceled (uploading or downloading).

David

Re: muchsync files renames

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 1176 bytes --]

Hi David,

Fist of all thank you for such elaborate answer.

I have missed the paragraph about maildir.synchronize_flags somehow.  I
have it enabled.  So this must be source of a problem (?).

Here follows steps I followed:

1. I initialized server locally with muchsync -vv.  My mail is stored in
~/Mail on the server.
2. I run muchsync --init ~/mail SERVER. (Directory names do not need to
be the same, do they?)
3. I run muchsync SERVER.
4. When it lasted much longer then initialization I canceled it by
single SIGINT (^c).
5. I rerun muchsync SERVER and then it notified me that notmuch
identified files names changes - more than 1000.
6. I waited a bit and then I canceled it by SIGINT.
7. I run muchsync --noup SERVER. This took only seconds to finish.

I suspected that muchsync at step 3 and 5 tried to push files renames
back to server.  But now I am not sure what was going on.  Have I
desynchronized file mail flags?  It's hard to say if anything has broken
for me, but I am a bit worried anyway.

If I just disable maildir.synchronize_flags and rerun muchsync, will
everything get synchronized properly?


-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: muchsync files renames

[David Mazieres]

Amadeusz Żołnowski <aidecoe@aidecoe.name> writes:

> Hi David,
>
> Fist of all thank you for such elaborate answer.
>
> I have missed the paragraph about maildir.synchronize_flags somehow.  I
> have it enabled.  So this must be source of a problem (?).

I've only ever tested with mailder.synchronize_flags = true, because I'm
worried that if I disable it I will have a hard time re-enabling it.  I
do think it is a source of inefficiency, but muchsync should still be
usable.

> Here follows steps I followed:
>
> 1. I initialized server locally with muchsync -vv.  My mail is stored in
> ~/Mail on the server.
> 2. I run muchsync --init ~/mail SERVER. (Directory names do not need to
> be the same, do they?)

Confirmed that directory names do not need to be the same.

> 3. I run muchsync SERVER.
> 4. When it lasted much longer then initialization I canceled it by
> single SIGINT (^c).

Interesting.  I wish I knew why this was taking much longer than running
it on the server, and whether the delay was caused by client activity or
server activity.

I don't suppose you'd be willing to make a copy of your mail database to
repeat the experiment without any risk of messing up your real maildir?
Basically what would be interesting to see is (assuming .notmuch-copy on
server is configured for this disposable copy):

    # Log everything for later analysis
    script
    # Use new config file location for disposable copy
    export NOTMUCH_CONFIG=$HOME/.notmuch-copy
    # Set up a new initial database
    muchsync -vvvv --init ~/test-copy SERVER -vvvv --config=.notmuch-copy

    # Now initial copy is made, see if client is slow
    # Is notmuch new itself slow?
    notmuch new
    # Is resynchronizing muchsync with notmuch slow?
    muchsync -vvvv

    # Now see if something weird happened on server to make notmuch new slow
    ssh SERVER notmuch --config=.notmuch-copy new
    # Now see if something weird happened on server to make muchsync slow
    ssh SERVER muchsync -vvvv --config=.notmuch-copy

    # Now finally try resynchronizing to see if this is slow
    muchsync -vvvv SERVER -vvvv --config=.notmuch-copy
    # Save script file
    exit

Does something along these lines make sense?  If you can figure out
which of these is slow (other than --init, which always will be), and
what the verbose chatter is, that would help me narrow down and identify
any problem.

> 5. I rerun muchsync SERVER and then it notified me that notmuch
> identified files names changes - more than 1000.

Were the link changes on the client (sent) or the server (received)
side?

> 6. I waited a bit and then I canceled it by SIGINT.
> 7. I run muchsync --noup SERVER. This took only seconds to finish.

So this suggests the issue is on the client side.  It sounds like a bug.
I wonder if I we can just reproduce this using a public email corpus, so
we don't have to worry about people's private email.

> I suspected that muchsync at step 3 and 5 tried to push files renames
> back to server.  But now I am not sure what was going on.  Have I
> desynchronized file mail flags?  It's hard to say if anything has broken
> for me, but I am a bit worried anyway.
>
> If I just disable maildir.synchronize_flags and rerun muchsync, will
> everything get synchronized properly?

I don't think that will change things.  maildir.synchronized_flags will
make things slower, but it shouldn't affect the SUMMARY numbers you see
at the end of muchsync, other than maybe files moving from .../new to
.../cur.  But presumably most of your mail files were already in cur
directories.

David

Re: muchsync files renames

[David Mazieres]

So just to follow up a bit.  I looked into things a bit further, and
here is what I found with maildir.synchronize_flags set to true.

Initially, when you run "muchsync --init", it copies all the files to
your maildir, and for each file invokes
notmuch_message_tags_to_maildir_flag.  That changes the name of the
file, with the result that the sql database and the actual mail
directory end up out of sync.  That on it's own is not a big deal, but
it means that the next time muchsync, muchsync will have to rescan all
of the files, as their names are no longer correct.  That shouldn't
cause any extra traffic between the two machines, but it will require
time on the client.  That is likely the source of the delay you were
seeing.

However, if you C-c the client during this process, I still don't see
any problems arising that cause more links to be transferred between
machines.  So I'm kind of stumped about that part.

Maybe muchsync should create the original file name based on tags, so as
to avoid having to rescan in the common case.  I wish there were some
way of getting the renamed file after
notmuch_message_tags_to_maildir_flags.

David

Re: muchsync files renames

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 1153 bytes --]

David Mazieres <dm-list-email-notmuch@scs.stanford.edu> writes:
> Initially, when you run "muchsync --init", it copies all the files to
> your maildir, and for each file invokes
> notmuch_message_tags_to_maildir_flag.  That changes the name of the
> file, with the result that the sql database and the actual mail
> directory end up out of sync.  That on it's own is not a big deal, but
> it means that the next time muchsync, muchsync will have to rescan all
> of the files, as their names are no longer correct.  That shouldn't
> cause any extra traffic between the two machines, but it will require
> time on the client.  That is likely the source of the delay you were
> seeing.

Yes, that seems to be probable scenario.

> However, if you C-c the client during this process, I still don't see
> any problems arising that cause more links to be transferred between
> machines.  So I'm kind of stumped about that part.

I don't think that C-c caused more links transfers. It's just about that
on rerun (after C-c) the notmuch on server reported renames which
probably took place in the previous run.

-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: muchsync files renames

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 1892 bytes --]

Hi David,

(Resending, because I forgot to Cc mailing list.)

David Mazieres <dm-list-email-notmuch@scs.stanford.edu> writes:
>> 3. I run muchsync SERVER.
>> 4. When it lasted much longer then initialization I canceled it by
>> single SIGINT (^c).
>
> Interesting.  I wish I knew why this was taking much longer than running
> it on the server, and whether the delay was caused by client activity or
> server activity.

I think there was something happening on server side because with --noup
it has been completed in few seconds.

 > I don't suppose you'd be willing to make a copy of your mail database
 > to repeat the experiment without any risk of messing up your real
 > maildir?

I would try it, but unfortunately I would have to make a bit more space
for having second copy of my mail.  I am testing muchsync on the same
machine between different users home directories, so it already takes
some space.  I'll try the experiment some day this week, I hope.

>> 5. I rerun muchsync SERVER and then it notified me that notmuch
>> identified files names changes - more than 1000.
>
> Were the link changes on the client (sent) or the server (received)
> side?

On the server side.  That's why I am worried.


> I don't think that will change things.  maildir.synchronized_flags
> will make things slower, but it shouldn't affect the SUMMARY numbers
> you see at the end of muchsync, other than maybe files moving from
> .../new to .../cur.  But presumably most of your mail files were
> already in cur directories.

So what would happen on my machine is that first client initialization
took place.  During this stage muchsync moved some files from new/ to
cur/. Later running "muchsync SERVER" tried to reflect client changes on
server by pushing renames requests to server.  Is it what actually could
happend?


-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: muchsync files renames

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 3011 bytes --]

Hi David,

First of all thank you a lot for support.  I am Cc'ing ml because the
last paragraph may be useful hint for other users.


David Mazieres writes:
> So to be clear, you are getting tons of lines that start "[SERVER]
> [notmuch]" and contain the string "Ignoring non-mail file"?  Is the
> "##...##" literal, or is that an ellipsis?

I have just cut off few directories on the path. :-) All of these files
are invalid spam mail, indeed.  I have removed them.  One problem less.


> Also, those file names were not generated were not generated by
> muchsync.  Any mail file created by muchsync will have a file name of
> the form:
>
>     nnn.MnnnPnnnQnRn.machine
>     nnn.MnnnPnnnQnRn.machine:2,

Just to makes things clear (once again? :-)), these file names are
generated only on client side.  Muchsync is not gonna ever to sync file
names to server, is it?


> When you run "notmuch new" on the server, without muchsync, does it
> take forever and print all these message while scanning non mail
> files?

No. Notmuch doesn't print these messages when I just run "notmuch new"
myself.  Anyway there was only around 100 of invalid mail files.


> Okay, this is the interesting part.  It appears that 5775 out of your
> 115877 messages have been moved to a different directory on the
> client.  I notice that the one message you include above has been
> moved to the Spam maildir.

> Is it possible that A) you have some spam filtering on the client that
> is moving things to the Spam folder,

I have a mailfilter rule which moves mails with "X-Spam-Status: Yes" to
Spam directory, but this happens on delivery before notmuch indexing.


> or B) that one of your two machines is using a case-independent file
> system that is causing confusion between "Spam" and "spam"?

I am testing it on single GNU/Linux host between different users.  It is
ext4 fs.


> So... based on all the evidence so fare the culprit seems to be that
> something is moving mail files into your Spam folder on the client.
> If that rings any bells and solves the problem, great.  If not, here
> is what we need to do to track it down further.

I have followed you hints to track down the issue.  All of these
messages are spam. What I suspect follows.

All of these files have been placed to new/ subdir by maildrop and
during posthook (afew) have been stripped of any tags besides 'spam'
tag, in particular 'unread' tag has been removed, but files still remain
in new/ subdir.  So... what had to happen is that during muchsync these
messages have been discovered as already read, so they don't belong to
new/ but must be moved to cur/.  And this is what happened on client
side.  During next muchsync these changes had to be pushed to server,
i.e. move from new/ to cur/.

So if my assumptions are correct, actually there is no issue!  I would
just have to adjust afew filtering to prevent this behaviour.


Thank you,

-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: muchsync files renames

[dm-list-email-notmuch]

Amadeusz Żołnowski <aidecoe@aidecoe.name> writes:

>> So... based on all the evidence so fare the culprit seems to be that
>> something is moving mail files into your Spam folder on the client.
>> If that rings any bells and solves the problem, great.  If not, here
>> is what we need to do to track it down further.
>
> I have followed you hints to track down the issue.  All of these
> messages are spam. What I suspect follows.
>
> All of these files have been placed to new/ subdir by maildrop and
> during posthook (afew) have been stripped of any tags besides 'spam'
> tag, in particular 'unread' tag has been removed, but files still remain
> in new/ subdir.  So... what had to happen is that during muchsync these
> messages have been discovered as already read, so they don't belong to
> new/ but must be moved to cur/.  And this is what happened on client
> side.  During next muchsync these changes had to be pushed to server,
> i.e. move from new/ to cur/.

Right.  Muchsync checks to see if maildir.synchronize_flags is true on
the client.  If it is, then muchsync calls
notmuch_message_tags_to_maildir_flags after setting the flags (which is
the same as what would happen if you set the tags manually with the
"notmuch tag" command).

A maildir file in the new/ directory can't have any tags (except the
implicit unread flag, which is indicated by the absence of "S" in the
end of the filename).  So the notmuch_message_tags_to_maildir_flags()
function is renaming the file to the cur subdirectory, and then
propagating this rename back to the server.

The one thing I'm still unclear on is whether afew is running on the
client of the server.  If you are running it on the client, then this
makes sense.  If you are running it on the server, then somehow afew
must not be respecting the maildir.synchronize_flags setting.
Otherwise, the file should already be moved to the cur directory after
having the unread tag stripped off on the server.  I guess the other
option is that your maildir.synchronize_flags false on the server and
true on the client.

> So if my assumptions are correct, actually there is no issue!  I would
> just have to adjust afew filtering to prevent this behaviour.

Right.  You could have afew preserve the unread flag on spam.
Alternatively, you could just disable maildir.synchronize_flags on both
the client and server.  Finally, you could just accept the performance
penalty, as one would hope that this is a one-time thing and that
usually you don't have 5000 new spam messages every time you synchronize
your mail.

David

Re: muchsync files renames

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]

Hi David,

dm-list-email-notmuch@scs.stanford.edu writes:
> The one thing I'm still unclear on is whether afew is running on the
> client of the server.

It is run as a post-hook, i.e. after "notmuch new", so it's on the
server.

> I guess the other option is that your maildir.synchronize_flags false
> on the server and true on the client.

Both have this set to true.

> If you are running it on the server, then somehow afew must not be
> respecting the maildir.synchronize_flags setting.  Otherwise, the file
> should already be moved to the cur directory after having the unread
> tag stripped off on the server.

Not necessarily. The recommended setup of notmuch for afew is that
"notmuch new" tags messages with "new" tag only. Then afew processes all
messages with "new" tag. So if it is a spam, then it gets "new" removed
and "spam" added. A spam message at any time doesn't have "unread" tag
assigned which should explain this behaviour.  So the problem is to be
fixed on the afew side.

Once again thank you for helping with tracking down the issue.

-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: muchsync files renames

[David Mazieres]

Amadeusz Żołnowski <aidecoe@aidecoe.name> writes:

> Not necessarily. The recommended setup of notmuch for afew is that
> "notmuch new" tags messages with "new" tag only. Then afew processes all
> messages with "new" tag. So if it is a spam, then it gets "new" removed
> and "spam" added. A spam message at any time doesn't have "unread" tag
> assigned which should explain this behaviour.  So the problem is to be
> fixed on the afew side.

Let's just make sure I understand:  Your mail starts out like this:

    Path:  spam/new/nnn.MnnnPnnnQnRn.machine
    Tags:  new

Then you run afew, and afew runs

    notmuch tag -new +spam <message-ID>

You are saying that that even though maildir.synchronize_tags is true,
you end up with:

    Path:  spam/new/nnn.MnnnPnnnQnRn.machine
    Tags:  spam

That's a little surprising, because the next time you run "notmuch new,"
I would have expected it to add the unread flag based on the pathname.
But, I suppose it might make sense for notmuch to special-case that
flag.  In other words, if notmuch new finds a file called:

    spam/new/nnn.MnnnPnnnQnRn.machine:2,

Then it will add the unread tag to the Xapian database.  But maybe if it
finds a file in the new folder it doesn't add the unread flag.

But why does notmuch_message_tags_to_maildir_flag() then feel the need
to rename the file when muchsync calls it.  Muchsync should ideally
behave exactly the same as the notmuch tag command.  Specifically, when
muchsync receives a new file from the server, it does the following:

 1. create file in same directory as the server (presumably spam/new)

 2. Call the following functions on this file:
      notmuch_database_add_message()
      notmuch_message_freeze()
      notmuch_message_remove_all_tags()
      notmuch_message_add_tag() for each tag in new.tags
      if (synchronize_tags) notmuch_message_tags_to_maildir_flag()
      notmuch_message_thaw()

 3. get the current tags of the message from the server (presumably just
    spam)

 4. Call the following functions on the Message-ID:
      notmuch_message_freeze()
      notmuch_message_remove_all_tags()
      notmuch_message_add_tag() for each tag sent *by the server*
      if (synchronize_tags) notmuch_message_tags_to_maildir_flag()
      notmuch_message_thaw()

So what I'm wondering is how this is any different from what is already
happening on the server.  "notmuch new" should be doing what muchsync
does in step 2, and afew (via "notmuch tag") should be doing what
muchsync does in step 4.

Yet somehow you are saying that on the server the file stays in
spam/new/, while on the client muchsync's actions cause the file to move
to spam/cur/?  So that means there's still something I don't really
understand--possibly the series of notmuch library calls happening
server side (which I should then maybe emulate client side).

None of this is super serious, beyond a one-time extra cost, but I like
to understand things thoroughly, particularly when writing software that
manipulates critical data like mail...

It there any possibility that new.tags has a different setting on your
client and server machines?

Thanks,
David

Re: muchsync files renames

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 3023 bytes --]

Hi David,

David Mazieres <dm-list-email-notmuch@scs.stanford.edu> writes:
> Let's just make sure I understand:  Your mail starts out like this:
>
>     Path:  spam/new/nnn.MnnnPnnnQnRn.machine
>     Tags:  new
>
> Then you run afew, and afew runs
>
>     notmuch tag -new +spam <message-ID>
>
> You are saying that that even though maildir.synchronize_tags is true,
> you end up with:
>
>     Path:  spam/new/nnn.MnnnPnnnQnRn.machine
>     Tags:  spam

Yes.


> That's a little surprising, because the next time you run "notmuch new,"
> I would have expected it to add the unread flag based on the pathname.

What's more surprising is that there is a test case in notmuch test
suite which test whether after modifing tag of a mail it is moved from
new/ to cur/. Yes, it should be moved on any tag modification if I
understand correctly. But it seems it does not for my maildirs...

$ notmuch search --output=files thread:00000000000108bf
/home/aidecoe/Mail/aidecoe/2015/new/1441022521.M714465P23412V000000000000FE04I0000000000141A38_0.freja,S=53857
$ notmuch search thread:00000000000108bf
thread:00000000000108bf  Yest. 11:58 [1/1] Somebody; Subject (reklama unread)
$ notmuch tag +hey thread:00000000000108bf
$ notmuch search thread:00000000000108bf
thread:00000000000108bf  Yest. 11:58 [1/1] Somebody; Subject (hey reklama unread)
$ notmuch search --output=files thread:00000000000108bf
/home/aidecoe/Mail/aidecoe/2015/new/1441022521.M714465P23412V000000000000FE04I0000000000141A38_0.freja,S=53857


> Then it will add the unread tag to the Xapian database.  But maybe if it
> finds a file in the new folder it doesn't add the unread flag.

Might be.


> But why does notmuch_message_tags_to_maildir_flag() then feel the need
> to rename the file when muchsync calls it.  Muchsync should ideally
> behave exactly the same as the notmuch tag command.  Specifically, when
> muchsync receives a new file from the server, it does the following:
>
>  1. create file in same directory as the server (presumably spam/new)
>
>  2. Call the following functions on this file:
>       notmuch_database_add_message()
>       notmuch_message_freeze()
>       notmuch_message_remove_all_tags()
>       notmuch_message_add_tag() for each tag in new.tags
>       if (synchronize_tags) notmuch_message_tags_to_maildir_flag()
>       notmuch_message_thaw()
>
>  3. get the current tags of the message from the server (presumably just
>     spam)
>
>  4. Call the following functions on the Message-ID:
>       notmuch_message_freeze()
>       notmuch_message_remove_all_tags()
>       notmuch_message_add_tag() for each tag sent *by the server*
>       if (synchronize_tags) notmuch_message_tags_to_maildir_flag()
>       notmuch_message_thaw()

So for some reason in my maildirs mails are not moved from new/ to cur/
on tag manipulation, but they are on client side by muchsync.  I will
have to investigate why this happens to me.


-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

synchronize_flags leaving files in new (was muchsync files renames)

[dm-list-email-notmuch]

Amadeusz =C5=BBo=C5=82nowski <aidecoe@aidecoe.name> writes:

> What's more surprising is that there is a test case in notmuch test
> suite which test whether after modifing tag of a mail it is moved from
> new/ to cur/. Yes, it should be moved on any tag modification if I
> understand correctly. But it seems it does not for my maildirs...
>
> $ notmuch search --output=3Dfiles thread:00000000000108bf
> /home/aidecoe/Mail/aidecoe/2015/new/1441022521.M714465P23412V000000000000=
FE04I0000000000141A38_0.freja,S=3D53857
> $ notmuch search thread:00000000000108bf
> thread:00000000000108bf  Yest. 11:58 [1/1] Somebody; Subject (reklama unr=
ead)
> $ notmuch tag +hey thread:00000000000108bf
> $ notmuch search thread:00000000000108bf
> thread:00000000000108bf  Yest. 11:58 [1/1] Somebody; Subject (hey reklama=
 unread)
> $ notmuch search --output=3Dfiles thread:00000000000108bf
> /home/aidecoe/Mail/aidecoe/2015/new/1441022521.M714465P23412V000000000000=
FE04I0000000000141A38_0.freja,S=3D53857

First, just to be absolutely sure, doe the file exist?  It's conceivably
that notmuch search is returning stale data until the next time you run
notmuch new.  Not likely, but just trying to rule out all possibilities.

Second, I wonder if the ",S=3D53857" suffix (size) is throwing things off.
Perhaps libnotmuch only expects suffixes when they are of the form
",S=3D53857:2,<flags>."  Since muchsync does not add a size field (in
either the new or cur subdirectory), it could be leading to the
different behavior.  To test this, can you rename the file in the new
directory without the ",S=3D..." and see if the behavior changes?

Thanks,
David

Re: synchronize_flags leaving files in new (was muchsync files renames)

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

David Mazieres expires 2015-11-30 PST
<mazieres-id6wst8pvwmdssdepck8t5fkje@temporary-address.scs.stanford.edu>
writes:
>> $ notmuch tag +hey thread:00000000000108bf
>> $ notmuch search thread:00000000000108bf
>> thread:00000000000108bf  Yest. 11:58 [1/1] Somebody; Subject (hey reklama unread)
>> $ notmuch search --output=files thread:00000000000108bf
>> /home/aidecoe/Mail/aidecoe/2015/new/1441022521.M714465P23412V000000000000FE04I0000000000141A38_0.freja,S=53857
>
> First, just to be absolutely sure, doe the file exist?

100% sure.


> Second, I wonder if the ",S=53857" suffix (size) is throwing things
> off.  Perhaps libnotmuch only expects suffixes when they are of the
> form ",S=53857:2,<flags>."  Since muchsync does not add a size field
> (in either the new or cur subdirectory), it could be leading to the
> different behavior.  To test this, can you rename the file in the new
> directory without the ",S=..." and see if the behavior changes?

,S=... doesn't have any impact. I have tested it.


-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: muchsync files renames

[David Bremner]

Amadeusz Żołnowski <aidecoe@aidecoe.name> writes:

> What's more surprising is that there is a test case in notmuch test
> suite which test whether after modifing tag of a mail it is moved from
> new/ to cur/. Yes, it should be moved on any tag modification if I
> understand correctly. But it seems it does not for my maildirs...
>

If I understand the code correctly, this movement will only happen when
one of the maildir-flag-equivalent tags is changed. I haven't dug ack
through the archives, but I think mutt uses presence in new/ as some
kind of extra unseen state, so people requested not to move files until
needed.

d

Re: muchsync files renames

[dm-list-email-notmuch]

David Bremner <david@tethera.net> writes:

> Amadeusz Żołnowski <aidecoe@aidecoe.name> writes:
>
>> What's more surprising is that there is a test case in notmuch test
>> suite which test whether after modifing tag of a mail it is moved from
>> new/ to cur/. Yes, it should be moved on any tag modification if I
>> understand correctly. But it seems it does not for my maildirs...
>>
>
> If I understand the code correctly, this movement will only happen when
> one of the maildir-flag-equivalent tags is changed. I haven't dug ack
> through the archives, but I think mutt uses presence in new/ as some
> kind of extra unseen state, so people requested not to move files until
> needed.

Can you explain how/where this is implemented?  I would like muchsync to
do exactly what notmuch does, and ideally without replicating its logic,
if I can just have libnotmuch handle this.  Currently, my code looks
something like this:

      notmuch_message_freeze()
      notmuch_message_remove_all_tags()
      notmuch_message_add_tag(); notmuch_message_add_tag(); ...
      if (synchronize_tags)
        notmuch_message_tags_to_maildir_flag()
      notmuch_message_thaw()

And what we're finding is the above code causes the message to move from
new/ to cur/, while the "notmuch tag" command does not, even while
changing between the same before and after tag sets.

Any ideas?

Thanks,
David

Re: muchsync files renames

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 1281 bytes --]

David Bremner <david@tethera.net> writes:
> If I understand the code correctly, this movement will only happen
> when one of the maildir-flag-equivalent tags is changed. I haven't dug
> ack through the archives, but I think mutt uses presence in new/ as
> some kind of extra unseen state, so people requested not to move files
> until needed.

When I have added 'unread' tag the file was still in new/. Only after
removing 'unread' afterwards the file has been moved to cur/. So it
seems you're right, but take a look at the following excerpt from
T340-maildir-sync.sh:

    test_begin_subtest "Message in new with maildir info is moved to cur on any tag change"
    add_message [filename]='message-with-info-to-be-moved-to-cur:2,' [dir]=new
    notmuch tag +anytag id:$gen_msg_id
    output=$(cd "$MAIL_DIR"; ls */message-with-info-to-be-moved-to-cur*)
    test_expect_equal "$output" "cur/message-with-info-to-be-moved-to-cur:2,"

What is different about the test case and my case is that my mail file
doesn't have ":2," suffix. Adding the suffix to file name makes it
working as expect by test case. I see I would have to convert my mail
files names, but I think this inconsistency in notmuch should also take
some attention.

-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]

Re: muchsync files renames

[David Bremner]

Amadeusz Żołnowski <aidecoe@aidecoe.name> writes:

> When I have added 'unread' tag the file was still in new/. Only after
> removing 'unread' afterwards the file has been moved to cur/.

The unread tag corresponds to the *absence* of the ,S flag, so if you
don't add unread at notmuch new, tagging it unread later is effectively
a no-op from the point of view of maildir-flag synching. I guess the
part that is optional is moving from new/foo to cur/foo:2, . I believe
we used to be more aggressive about doing this, but mutt users
complained.

> So it seems you're right, but take a look at the following excerpt
> from T340-maildir-sync.sh:
>
[...]
> What is different about the test case and my case is that my mail file
> doesn't have ":2," suffix. Adding the suffix to file name makes it
> working as expect by test case. I see I would have to convert my mail
> files names, but I think this inconsistency in notmuch should also take
> some attention.

Have a look at

     http://cr.yp.to/proto/maildir.html
     http://www.qmail.org/man/man5/maildir.html
     
I don't think messages in new are supposed to have : in their names. So
this test is dealing with a corner case of some out-of-spec MUA writing
:info onto the filename. So I don't think adding a suffix is the right
thing to do here. It also seems like leaving a message in new/ when
tagging it as unread is a reasonable option.

The gory details (per David's earlier request) are in
_new_maildir_filename in lib/message.cc.


d

Re: muchsync files renames

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]

Thank you David B. for explanation.  I think that everything is clear
now.

David M., what about updating your website on that?  I think it's
important to warn about possible files moves between new/ and cur/.

And what I can do is to prepare patches for afew to handle it
appropriately.

-- 
Amadeusz Żołnowski

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 950 bytes --]