From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id LeXbLYPRoF4aCAAA0tVLHw (envelope-from ) for ; Wed, 22 Apr 2020 23:21:39 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id MAZ6DorRoF42NgAAbx9fmQ (envelope-from ) for ; Wed, 22 Apr 2020 23:21:46 +0000 Received: from arlo.cworth.org (arlo.cworth.org [50.126.95.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 57EF3942190 for ; Wed, 22 Apr 2020 23:21:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 283EC6DE0F37; Wed, 22 Apr 2020 16:21:40 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j1ePu-KsTozi; Wed, 22 Apr 2020 16:21:39 -0700 (PDT) Received: from arlo.cworth.org (localhost [IPv6:::1]) by arlo.cworth.org (Postfix) with ESMTP id 7D0E96DE0F1C; Wed, 22 Apr 2020 16:21:38 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by arlo.cworth.org (Postfix) with ESMTP id 162216DE0F1C for ; Wed, 22 Apr 2020 16:21:37 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at cworth.org Received: from arlo.cworth.org ([127.0.0.1]) by localhost (arlo.cworth.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uLBFY-09lvVn for ; Wed, 22 Apr 2020 16:21:36 -0700 (PDT) Received: from thyestes.tartarus.org (thyestes.tartarus.org [5.196.91.86]) by arlo.cworth.org (Postfix) with ESMTPS id 679256DE0EDC for ; Wed, 22 Apr 2020 16:21:36 -0700 (PDT) Received: from olly by thyestes.tartarus.org with local (Exim 4.92) (envelope-from ) id 1jROgM-0003GN-44; Thu, 23 Apr 2020 00:21:30 +0100 Date: Thu, 23 Apr 2020 00:21:30 +0100 From: Olly Betts To: David Bremner Subject: Re: performance problems with notmuch new Message-ID: <20200422232130.GH28897@survex.com> Mail-Followup-To: David Bremner , Franz Fellner , Don Zickus , notmuch@notmuchmail.org, xapian-discuss@lists.xapian.org References: <20200415150801.h2mazyo37sspvech@redhat.com> <1587211167-ner-6.432@LappyL520> <87imhup6kr.fsf@tethera.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <87imhup6kr.fsf@tethera.net> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: notmuch@notmuchmail.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Use and development of the notmuch mail system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Xapian Discussion Cc: Don Zickus , notmuch@notmuchmail.org, xapian-discuss@lists.xapian.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: notmuch-bounces@notmuchmail.org Sender: "notmuch" X-Scanner: scn0 X-Spam-Score: -1.21 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of notmuch-bounces@notmuchmail.org designates 50.126.95.6 as permitted sender) smtp.mailfrom=notmuch-bounces@notmuchmail.org X-Scan-Result: default: False [-1.21 / 13.00]; HAS_REPLYTO(0.00)[xapian-discuss@lists.xapian.org]; GENERIC_REPUTATION(0.00)[-0.4654990864186]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:c]; IP_REPUTATION_HAM(0.00)[asn: 27017(-0.19), country: US(-0.00), ip: 50.126.95.6(-0.47)]; RCVD_IN_DNSWL_MED(-0.20)[50.126.95.6:from]; MX_GOOD(-0.50)[notmuchmail.org]; MAILLIST(-0.20)[mailman]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:27017, ipnet:50.126.64.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[olly@survex.com,notmuch-bounces@notmuchmail.org]; ARC_NA(0.00)[]; URIBL_BLOCKED(0.00)[notmuchmail.org:email]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MIME_GOOD(-0.10)[text/plain]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[notmuch@notmuchmail.org]; HAS_LIST_UNSUB(-0.01)[]; DMARC_NA(0.00)[survex.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_SEVEN(0.00)[8]; FORGED_SENDER_MAILLIST(0.00)[] X-TUID: BqDYBVO/Njxs On Mon, Apr 20, 2020 at 11:36:36AM -0300, David Bremner wrote: > Franz Fellner writes: > > > I also suffer from bad performance of notmuch new. I used notmuch > > some years ago and notmuch new always felt instantanious. Had to stop > > using it because internet was too slow to sync my mails :/ Now (with > > better internet and a completely new setup using mbsync) indexing one > > mail takes at least 10 seconds, sometimes even more. It can go into > > minutes when I get lots of mail (~30...). First question: what version of Xapian are you using? And second thing to check, are you committing each message separately? The commit operation tries to ensure that the data has actually been written out to disk, so the time to index one message by itself isn't indicative as it'll often mostly just be waiting for fdatasync() or similar to return. If you index 30 messages but commit each separately (i.e. run "notmuch new" 30 times picking up one new message each time) that'll probably scale something like linearly, but indexing a batch of 30 messages should be much quicker per message. > > When I run it after a > > reboot I can have breakfast while notmuch starts up... This is all on > > spinning rust. I thought of getting an SSD but not in the near future. After reboot the disk cache won't have any of the database in, so the first operation will typically be slower, especially with a spinning drive where seeks are relatively slow. > > What I observe during that time: notmuch doesn't really need much CPU. > > iotop shows constant read and write with extremely low rates, under > > 1MB/sec. So I think it might be an issue in xapian? > > Just in case one of the xapian experts can suggest some kind of test for > why you might be seeing this behaviour, I've included the xapian list in > CC. It sounds like you're seek-limited in this "cold cache" phase. That is not necessarily related to the slow indexing, but it could be. I'd check the SMART diagnostics for the drive first (e.g. with smartctl). It's not the most likely cause, but it's quick to check and if the drive is starting to fail it's better to find out sooner rather than later. Then I'd try compacting the database (I think there's a "notmuch compact" subcommand to do this). If that doesn't help, profiling the I/O would probably be my next suggestion - there are some tools in the xapian git repo to help with this (in xapian-maintainer-tools/profiling). Under Linux I'd suggest the strace ones (there's also an LD_PRELOAD library but it may need tweaking for 32 vs 64 bit). Cheers, Olly