From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id D70476DE0948 for ; Sun, 22 Dec 2019 19:01:51 -0800 (PST) X-Virus-Scanned: Debian amavisd-new at cworth.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_NONE=-0.0001] autolearn=disabled Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hVQVbunLK0sU for ; Sun, 22 Dec 2019 19:01:50 -0800 (PST) X-Greylist: delayed 441 seconds by postgrey-1.36 at arlo; Sun, 22 Dec 2019 19:01:50 PST Received: from h4.fbrelay.privateemail.com. (h4.fbrelay.privateemail.com [131.153.2.45]) by arlo.cworth.org (Postfix) with ESMTPS id 595456DE02C5 for ; Sun, 22 Dec 2019 19:01:50 -0800 (PST) Received: from MTA-10-1.privateemail.com (mta-10.privateemail.com [68.65.122.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by h3.fbrelay.privateemail.com (Postfix) with ESMTPS id E0CD1800BF for ; Sun, 22 Dec 2019 21:54:27 -0500 (EST) Received: from MTA-10.privateemail.com (localhost [127.0.0.1]) by MTA-10.privateemail.com (Postfix) with ESMTP id 16C8860038 for ; Sun, 22 Dec 2019 21:54:25 -0500 (EST) Received: from [10.15.0.4] (unknown [10.20.151.203]) by MTA-10.privateemail.com (Postfix) with ESMTPA id D1C9560033 for ; Mon, 23 Dec 2019 02:54:24 +0000 (UTC) To: notmuch@notmuchmail.org From: Matthew Schauer Subject: Xapian commits unexpectedly slow Message-ID: <4b3b642b-8f5b-4e8c-9f29-76d393d45fd6@e10x.net> Date: Sun, 22 Dec 2019 19:54:24 -0700 User-Agent: Mozilla/5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP X-Mailman-Approved-At: Sat, 28 Dec 2019 05:05:24 -0800 X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Dec 2019 03:01:52 -0000 Greetings, I've been trying to migrate about 25K e-mails to Notmuch, and I'm seeing some frustrating performance characteristics that don't seem to match with the experience others report. I'm dumping messages from Thunderbird in batches and then running `notmuch new` to add each batch to the database. The indexing performance remains okay, at more than 200 per second, but after Notmuch has reported it's finished indexing, it hangs for as much as several minutes before exiting. A stack trace confirms that this is Xapian committing the database, with most of the time seemingly spent in `fdatasync`. The time spent grows with the size of the database, not the number of e-mails being imported, which means this will remain a problem during day-to-day usage. Has nobody else had a problem like this? Is my setup just weird? I'm using Notmuch 0.29.3 from the Arch community repository, with Xapian 1.4.14 also from Arch repositories. I am using a spinning-platter hard disk, but I find it hard to believe that an SSD is required to get Xapian to perform well at this scale. Please let me know if you have any performance pointers or can help me investigate this further. Many thanks! Matthew Schauer