From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 4EB406DE13DB for ; Sun, 23 Aug 2015 13:06:30 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: -2.261 X-Spam-Level: X-Spam-Status: No, score=-2.261 tagged_above=-999 required=5 tests=[AWL=0.590, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-0.55, SPF_PASS=-0.001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AVNoyFJljYQs for ; Sun, 23 Aug 2015 13:06:28 -0700 (PDT) Received: from market.scs.stanford.edu (market.scs.stanford.edu [171.66.3.10]) by arlo.cworth.org (Postfix) with ESMTPS id 5C0D76DE13DA for ; Sun, 23 Aug 2015 13:06:28 -0700 (PDT) Received: from market.scs.stanford.edu (localhost.scs.stanford.edu [127.0.0.1]) by market.scs.stanford.edu (8.14.7/8.14.7) with ESMTP id t7NK6LXZ024280; Sun, 23 Aug 2015 13:06:21 -0700 (PDT) Received: (from dm@localhost) by market.scs.stanford.edu (8.14.7/8.14.7/Submit) id t7NK6KAv003545; Sun, 23 Aug 2015 13:06:20 -0700 (PDT) X-Authentication-Warning: market.scs.stanford.edu: dm set sender to return-jmxwjhzeuidrb5in85caytaxb2@ta.scs.stanford.edu using -f From: David Mazieres To: Amadeusz =?utf-8?B?xbtvxYJub3dza2k=?= , notmuch@notmuchmail.org Subject: Re: muchsync files renames In-Reply-To: <871teu8kdd.fsf@freja.aidecoe.name> References: <878u93ujdo.fsf@freja.aidecoe.name> <876146o920.fsf@ta.scs.stanford.edu> <871teu8kdd.fsf@freja.aidecoe.name> Reply-To: David Mazieres expires 2015-11-21 PST Date: Sun, 23 Aug 2015 13:06:20 -0700 Message-ID: <87oahxojlv.fsf@ta.scs.stanford.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Aug 2015 20:06:30 -0000 Amadeusz =C5=BBo=C5=82nowski writes: > Hi David, > > Fist of all thank you for such elaborate answer. > > I have missed the paragraph about maildir.synchronize_flags somehow. I > have it enabled. So this must be source of a problem (?). I've only ever tested with mailder.synchronize_flags =3D true, because I'm worried that if I disable it I will have a hard time re-enabling it. I do think it is a source of inefficiency, but muchsync should still be usable. > Here follows steps I followed: > > 1. I initialized server locally with muchsync -vv. My mail is stored in > ~/Mail on the server. > 2. I run muchsync --init ~/mail SERVER. (Directory names do not need to > be the same, do they?) Confirmed that directory names do not need to be the same. > 3. I run muchsync SERVER. > 4. When it lasted much longer then initialization I canceled it by > single SIGINT (^c). Interesting. I wish I knew why this was taking much longer than running it on the server, and whether the delay was caused by client activity or server activity. I don't suppose you'd be willing to make a copy of your mail database to repeat the experiment without any risk of messing up your real maildir? Basically what would be interesting to see is (assuming .notmuch-copy on server is configured for this disposable copy): # Log everything for later analysis script # Use new config file location for disposable copy export NOTMUCH_CONFIG=3D$HOME/.notmuch-copy # Set up a new initial database muchsync -vvvv --init ~/test-copy SERVER -vvvv --config=3D.notmuch-copy # Now initial copy is made, see if client is slow # Is notmuch new itself slow? notmuch new # Is resynchronizing muchsync with notmuch slow? muchsync -vvvv # Now see if something weird happened on server to make notmuch new slow ssh SERVER notmuch --config=3D.notmuch-copy new # Now see if something weird happened on server to make muchsync slow ssh SERVER muchsync -vvvv --config=3D.notmuch-copy # Now finally try resynchronizing to see if this is slow muchsync -vvvv SERVER -vvvv --config=3D.notmuch-copy # Save script file exit Does something along these lines make sense? If you can figure out which of these is slow (other than --init, which always will be), and what the verbose chatter is, that would help me narrow down and identify any problem. > 5. I rerun muchsync SERVER and then it notified me that notmuch > identified files names changes - more than 1000. Were the link changes on the client (sent) or the server (received) side? > 6. I waited a bit and then I canceled it by SIGINT. > 7. I run muchsync --noup SERVER. This took only seconds to finish. So this suggests the issue is on the client side. It sounds like a bug. I wonder if I we can just reproduce this using a public email corpus, so we don't have to worry about people's private email. > I suspected that muchsync at step 3 and 5 tried to push files renames > back to server. But now I am not sure what was going on. Have I > desynchronized file mail flags? It's hard to say if anything has broken > for me, but I am a bit worried anyway. > > If I just disable maildir.synchronize_flags and rerun muchsync, will > everything get synchronized properly? I don't think that will change things. maildir.synchronized_flags will make things slower, but it shouldn't affect the SUMMARY numbers you see at the end of muchsync, other than maybe files moving from .../new to .../cur. But presumably most of your mail files were already in cur directories. David